2024-04-23T14:13:46.663 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-23T14:13:46.663 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_rishabh@teuthology --archive /home/teuthworker/archive/rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi/7669575 --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} --name rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi -- /home/teuthworker/archive/rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi/7669575/orig.config.yaml 2024-04-23T14:13:46.719 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-23T14:13:46.808 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi/7669575 branch: wip-rishabh-testing-20240416.193735-3 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} email: null first_in_suite: false job_id: '7669575' kernel: client: branch: testing flavor: default kdb: true sha1: 806ec36b21129f113ce356c007dd59694d340b37 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-rishabh-testing-20240416.193735-3 ceph: cephfs: max_mds: 1 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: d44fddd910b5e94da0f2357af4be37d7bd495ae0 subvols: create: 2 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: d44fddd910b5e94da0f2357af4be37d7bd495ae0 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - nowsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-rishabh-testing-20240416.193735-3 sha1: d44fddd910b5e94da0f2357af4be37d7bd495ae0 owner: scheduled_rishabh@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 7954 sha1: d44fddd910b5e94da0f2357af4be37d7bd495ae0 sleep_before_teardown: 0 subset: 364/512 suite: fs suite_branch: wip-rishabh-testing-20240416.193735-3 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d44fddd910b5e94da0f2357af4be37d7bd495ae0/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d44fddd910b5e94da0f2357af4be37d7bd495ae0 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi028.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBUehiPSkozqQnJ9UaPSqzBxGc+A6onb13HqwF8SaFKzIJuHG8vAfiA2efvG52A4du+l9t0tmP/+d8LC/GLF1ls= smithi038.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDEaaRjD218Zv1QKJfZeOdXOvqJctxdXM3ACEaWxTdK0xC2fsrPp874KZQJzxpfucl0yuqsl++K1Sho4lgDl4Mk= smithi105.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKmkDL7EOfk19pPMyDZ6wHoDyMlR57e1ijQcjvaYXN9Cue9hzd2b5xKz07ICDhkEW9gQSsNFfw6uBT8hS6bjaVA= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-23_14:04:26 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-23T14:13:46.808 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d44fddd910b5e94da0f2357af4be37d7bd495ae0/qa; will attempt to use it 2024-04-23T14:13:46.809 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d44fddd910b5e94da0f2357af4be37d7bd495ae0/qa/tasks 2024-04-23T14:13:46.809 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-23T14:13:46.811 INFO:teuthology.task.internal:Checking packages... 2024-04-23T14:13:46.832 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd44fddd910b5e94da0f2357af4be37d7bd495ae0' 2024-04-23T14:13:46.832 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-23T14:13:46.833 INFO:teuthology.packaging:ref: None 2024-04-23T14:13:46.833 INFO:teuthology.packaging:tag: None 2024-04-23T14:13:46.833 INFO:teuthology.packaging:branch: wip-rishabh-testing-20240416.193735-3 2024-04-23T14:13:46.833 INFO:teuthology.packaging:sha1: d44fddd910b5e94da0f2357af4be37d7bd495ae0 2024-04-23T14:13:46.833 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-rishabh-testing-20240416.193735-3 2024-04-23T14:13:46.980 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3126.gd44fddd9 2024-04-23T14:13:46.981 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-23T14:13:46.987 INFO:teuthology.task.internal:no buildpackages task found 2024-04-23T14:13:46.987 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-23T14:13:47.068 INFO:teuthology.task.internal:Saving configuration 2024-04-23T14:13:47.088 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-23T14:13:47.095 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-23T14:13:47.120 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi/7669575', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 14:05:38.351647', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBUehiPSkozqQnJ9UaPSqzBxGc+A6onb13HqwF8SaFKzIJuHG8vAfiA2efvG52A4du+l9t0tmP/+d8LC/GLF1ls='} 2024-04-23T14:13:47.140 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi/7669575', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 14:05:38.350474', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDEaaRjD218Zv1QKJfZeOdXOvqJctxdXM3ACEaWxTdK0xC2fsrPp874KZQJzxpfucl0yuqsl++K1Sho4lgDl4Mk='} 2024-04-23T14:13:47.164 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi105.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi/7669575', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 14:05:38.348737', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKmkDL7EOfk19pPMyDZ6wHoDyMlR57e1ijQcjvaYXN9Cue9hzd2b5xKz07ICDhkEW9gQSsNFfw6uBT8hS6bjaVA='} 2024-04-23T14:13:47.164 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-23T14:13:47.170 INFO:teuthology.task.internal:roles: ubuntu@smithi028.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-23T14:13:47.170 INFO:teuthology.task.internal:roles: ubuntu@smithi038.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-23T14:13:47.170 INFO:teuthology.task.internal:roles: ubuntu@smithi105.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-23T14:13:47.171 INFO:teuthology.run_tasks:Running task console_log... 2024-04-23T14:13:47.267 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8a56989160>, signals=[15]) 2024-04-23T14:13:47.267 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-23T14:13:47.274 INFO:teuthology.task.internal:Opening connections... 2024-04-23T14:13:47.274 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi028.front.sepia.ceph.com 2024-04-23T14:13:47.276 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T14:13:47.353 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi038.front.sepia.ceph.com 2024-04-23T14:13:47.353 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T14:13:47.429 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi105.front.sepia.ceph.com 2024-04-23T14:13:47.430 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T14:13:47.505 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-23T14:13:47.511 DEBUG:teuthology.orchestra.run.smithi028:> uname -m 2024-04-23T14:13:47.529 INFO:teuthology.orchestra.run.smithi028.stdout:x86_64 2024-04-23T14:13:47.530 DEBUG:teuthology.orchestra.run.smithi028:> cat /etc/os-release 2024-04-23T14:13:47.588 INFO:teuthology.orchestra.run.smithi028.stdout:NAME="CentOS Stream" 2024-04-23T14:13:47.588 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION="9" 2024-04-23T14:13:47.588 INFO:teuthology.orchestra.run.smithi028.stdout:ID="centos" 2024-04-23T14:13:47.588 INFO:teuthology.orchestra.run.smithi028.stdout:ID_LIKE="rhel fedora" 2024-04-23T14:13:47.588 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION_ID="9" 2024-04-23T14:13:47.588 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM_ID="platform:el9" 2024-04-23T14:13:47.588 INFO:teuthology.orchestra.run.smithi028.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-23T14:13:47.588 INFO:teuthology.orchestra.run.smithi028.stdout:ANSI_COLOR="0;31" 2024-04-23T14:13:47.588 INFO:teuthology.orchestra.run.smithi028.stdout:LOGO="fedora-logo-icon" 2024-04-23T14:13:47.589 INFO:teuthology.orchestra.run.smithi028.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-23T14:13:47.589 INFO:teuthology.orchestra.run.smithi028.stdout:HOME_URL="https://centos.org/" 2024-04-23T14:13:47.589 INFO:teuthology.orchestra.run.smithi028.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-23T14:13:47.589 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-23T14:13:47.589 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-23T14:13:47.589 INFO:teuthology.lock.ops:Updating smithi028.front.sepia.ceph.com on lock server 2024-04-23T14:13:47.618 DEBUG:teuthology.orchestra.run.smithi038:> uname -m 2024-04-23T14:13:47.637 INFO:teuthology.orchestra.run.smithi038.stdout:x86_64 2024-04-23T14:13:47.637 DEBUG:teuthology.orchestra.run.smithi038:> cat /etc/os-release 2024-04-23T14:13:47.695 INFO:teuthology.orchestra.run.smithi038.stdout:NAME="CentOS Stream" 2024-04-23T14:13:47.695 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION="9" 2024-04-23T14:13:47.695 INFO:teuthology.orchestra.run.smithi038.stdout:ID="centos" 2024-04-23T14:13:47.695 INFO:teuthology.orchestra.run.smithi038.stdout:ID_LIKE="rhel fedora" 2024-04-23T14:13:47.695 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION_ID="9" 2024-04-23T14:13:47.695 INFO:teuthology.orchestra.run.smithi038.stdout:PLATFORM_ID="platform:el9" 2024-04-23T14:13:47.695 INFO:teuthology.orchestra.run.smithi038.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-23T14:13:47.695 INFO:teuthology.orchestra.run.smithi038.stdout:ANSI_COLOR="0;31" 2024-04-23T14:13:47.695 INFO:teuthology.orchestra.run.smithi038.stdout:LOGO="fedora-logo-icon" 2024-04-23T14:13:47.696 INFO:teuthology.orchestra.run.smithi038.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-23T14:13:47.696 INFO:teuthology.orchestra.run.smithi038.stdout:HOME_URL="https://centos.org/" 2024-04-23T14:13:47.696 INFO:teuthology.orchestra.run.smithi038.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-23T14:13:47.696 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-23T14:13:47.696 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-23T14:13:47.696 INFO:teuthology.lock.ops:Updating smithi038.front.sepia.ceph.com on lock server 2024-04-23T14:13:47.718 DEBUG:teuthology.orchestra.run.smithi105:> uname -m 2024-04-23T14:13:47.735 INFO:teuthology.orchestra.run.smithi105.stdout:x86_64 2024-04-23T14:13:47.736 DEBUG:teuthology.orchestra.run.smithi105:> cat /etc/os-release 2024-04-23T14:13:47.792 INFO:teuthology.orchestra.run.smithi105.stdout:NAME="CentOS Stream" 2024-04-23T14:13:47.792 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION="9" 2024-04-23T14:13:47.792 INFO:teuthology.orchestra.run.smithi105.stdout:ID="centos" 2024-04-23T14:13:47.792 INFO:teuthology.orchestra.run.smithi105.stdout:ID_LIKE="rhel fedora" 2024-04-23T14:13:47.792 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION_ID="9" 2024-04-23T14:13:47.792 INFO:teuthology.orchestra.run.smithi105.stdout:PLATFORM_ID="platform:el9" 2024-04-23T14:13:47.792 INFO:teuthology.orchestra.run.smithi105.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-23T14:13:47.792 INFO:teuthology.orchestra.run.smithi105.stdout:ANSI_COLOR="0;31" 2024-04-23T14:13:47.792 INFO:teuthology.orchestra.run.smithi105.stdout:LOGO="fedora-logo-icon" 2024-04-23T14:13:47.792 INFO:teuthology.orchestra.run.smithi105.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-23T14:13:47.793 INFO:teuthology.orchestra.run.smithi105.stdout:HOME_URL="https://centos.org/" 2024-04-23T14:13:47.793 INFO:teuthology.orchestra.run.smithi105.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-23T14:13:47.793 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-23T14:13:47.793 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-23T14:13:47.793 INFO:teuthology.lock.ops:Updating smithi105.front.sepia.ceph.com on lock server 2024-04-23T14:13:47.818 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-23T14:13:47.828 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-23T14:13:47.834 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-23T14:13:47.834 DEBUG:teuthology.orchestra.run.smithi028:> test '!' -e /home/ubuntu/cephtest 2024-04-23T14:13:47.836 DEBUG:teuthology.orchestra.run.smithi038:> test '!' -e /home/ubuntu/cephtest 2024-04-23T14:13:47.839 DEBUG:teuthology.orchestra.run.smithi105:> test '!' -e /home/ubuntu/cephtest 2024-04-23T14:13:47.854 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-23T14:13:47.860 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-23T14:13:47.861 DEBUG:teuthology.orchestra.run.smithi028:> test -z $(ls -A /var/lib/ceph) 2024-04-23T14:13:47.894 DEBUG:teuthology.orchestra.run.smithi038:> test -z $(ls -A /var/lib/ceph) 2024-04-23T14:13:47.896 DEBUG:teuthology.orchestra.run.smithi105:> test -z $(ls -A /var/lib/ceph) 2024-04-23T14:13:47.959 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-23T14:13:48.029 INFO:teuthology.run_tasks:Running task kernel... 2024-04-23T14:13:48.044 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'} 2024-04-23T14:13:48.044 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-04-23T14:13:48.044 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-04-23T14:13:48.044 DEBUG:teuthology.orchestra.run.smithi028:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-23T14:13:48.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T14:13:48.062 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-04-23T14:13:48.232 DEBUG:teuthology.task.kernel:sha1 for client.0 is 806ec36b21129f113ce356c007dd59694d340b37 2024-04-23T14:13:48.232 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "806ec36b21129f113ce356c007dd59694d340b37"... 2024-04-23T14:13:48.233 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-04-23T14:13:48.250 INFO:teuthology.orchestra.run.smithi028.stdout:5.14.0-437.el9.x86_64 2024-04-23T14:13:48.250 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 806ec36b21129f113ce356c007dd59694d340b37 2024-04-23T14:13:48.251 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-23T14:13:48.251 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum upgrade -y linux-firmware 2024-04-23T14:13:52.120 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 3.4 MB/s | 8.0 MB 00:02 2024-04-23T14:13:55.505 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 8.5 MB/s | 19 MB 00:02 2024-04-23T14:14:00.214 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 6.2 MB/s | 6.2 MB 00:00 2024-04-23T14:14:02.500 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 15 kB/s | 16 kB 00:01 2024-04-23T14:14:04.519 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-04-23T14:14:09.733 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 38 kB/s | 1.7 kB 00:00 2024-04-23T14:14:11.778 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-23T14:14:11.779 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-04-23T14:14:11.779 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-23T14:14:11.874 INFO:teuthology.task.kernel:Downloading kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-23T14:14:11.875 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-04-23T14:14:12.025 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ 2024-04-23T14:14:12.025 DEBUG:teuthology.orchestra.run.smithi028:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-23T14:14:12.028 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi028.front.sepia.ceph.com to complete... 2024-04-23T14:20:53.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23 14:20:53 URL:https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm [357367792/357367792] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-23T14:20:53.016 INFO:teuthology.orchestra.run.smithi028.stderr:FINISHED --2024-04-23 14:20:53-- 2024-04-23T14:20:53.016 INFO:teuthology.orchestra.run.smithi028.stderr:Total wall clock time: 6m 41s 2024-04-23T14:20:53.016 INFO:teuthology.orchestra.run.smithi028.stderr:Downloaded: 1 files, 341M in 6m 39s (874 KB/s) 2024-04-23T14:20:53.020 INFO:teuthology.task.kernel:Installing kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-23T14:20:53.020 DEBUG:teuthology.orchestra.run.smithi028:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-23T14:20:55.213 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying... ######################################## 2024-04-23T14:20:55.344 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing... ######################################## 2024-04-23T14:20:55.458 INFO:teuthology.orchestra.run.smithi028.stdout:Updating / installing... 2024-04-23T14:21:03.848 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-6.9.0_rc3_g806ec36b2112-1 ######################################## 2024-04-23T14:21:41.364 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi028.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-23T14:21:41.365 DEBUG:teuthology.orchestra.run.smithi028:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-23T14:21:41.474 INFO:teuthology.orchestra.run.smithi028.stdout:/boot/System.map-6.9.0-rc3-g806ec36b2112 2024-04-23T14:21:41.474 INFO:teuthology.orchestra.run.smithi028.stdout:/boot/config-6.9.0-rc3-g806ec36b2112 2024-04-23T14:21:41.475 INFO:teuthology.orchestra.run.smithi028.stdout:/boot/vmlinuz-6.9.0-rc3-g806ec36b2112 2024-04-23T14:21:41.475 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/System.map 2024-04-23T14:21:41.475 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/config 2024-04-23T14:21:41.475 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel 2024-04-23T14:21:41.475 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch 2024-04-23T14:21:41.475 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86 2024-04-23T14:21:41.475 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto 2024-04-23T14:21:41.475 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-23T14:21:41.475 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-23T14:21:41.475 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-23T14:21:41.475 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-23T14:21:41.475 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-23T14:21:41.475 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-23T14:21:41.475 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-23T14:21:41.476 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-23T14:21:41.476 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-23T14:21:41.476 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-23T14:21:41.476 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-23T14:21:41.476 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-23T14:21:41.476 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-23T14:21:41.476 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-23T14:21:41.476 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-23T14:21:41.476 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-23T14:21:41.476 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-23T14:21:41.476 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-23T14:21:41.476 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-23T14:21:41.476 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-23T14:21:41.476 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel 2024-04-23T14:21:41.477 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu 2024-04-23T14:21:41.477 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce 2024-04-23T14:21:41.477 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-23T14:21:41.477 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm 2024-04-23T14:21:41.477 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-23T14:21:41.477 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-23T14:21:41.477 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm.ko 2024-04-23T14:21:41.477 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto 2024-04-23T14:21:41.477 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ansi_cprng.ko 2024-04-23T14:21:41.477 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/anubis.ko 2024-04-23T14:21:41.477 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/arc4.ko 2024-04-23T14:21:41.477 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx 2024-04-23T14:21:41.477 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_memcpy.ko 2024-04-23T14:21:41.477 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_pq.ko 2024-04-23T14:21:41.478 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-23T14:21:41.478 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_tx.ko 2024-04-23T14:21:41.478 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_xor.ko 2024-04-23T14:21:41.478 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/raid6test.ko 2024-04-23T14:21:41.478 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authenc.ko 2024-04-23T14:21:41.478 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authencesn.ko 2024-04-23T14:21:41.478 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blake2b_generic.ko 2024-04-23T14:21:41.478 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_common.ko 2024-04-23T14:21:41.478 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_generic.ko 2024-04-23T14:21:41.478 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/camellia_generic.ko 2024-04-23T14:21:41.478 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast5_generic.ko 2024-04-23T14:21:41.478 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast6_generic.ko 2024-04-23T14:21:41.478 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast_common.ko 2024-04-23T14:21:41.478 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ccm.ko 2024-04-23T14:21:41.479 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cmac.ko 2024-04-23T14:21:41.479 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crc32_generic.ko 2024-04-23T14:21:41.479 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_engine.ko 2024-04-23T14:21:41.479 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_user.ko 2024-04-23T14:21:41.479 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cts.ko 2024-04-23T14:21:41.479 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/des_generic.ko 2024-04-23T14:21:41.479 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/dh_generic.ko 2024-04-23T14:21:41.479 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/echainiv.ko 2024-04-23T14:21:41.479 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/essiv.ko 2024-04-23T14:21:41.479 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/fcrypt.ko 2024-04-23T14:21:41.479 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/khazad.ko 2024-04-23T14:21:41.479 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4.ko 2024-04-23T14:21:41.479 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4hc.ko 2024-04-23T14:21:41.479 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/md4.ko 2024-04-23T14:21:41.479 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/michael_mic.ko 2024-04-23T14:21:41.480 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcbc.ko 2024-04-23T14:21:41.480 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcrypt.ko 2024-04-23T14:21:41.480 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/rmd160.ko 2024-04-23T14:21:41.480 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/seed.ko 2024-04-23T14:21:41.480 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/serpent_generic.ko 2024-04-23T14:21:41.480 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tcrypt.ko 2024-04-23T14:21:41.480 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tea.ko 2024-04-23T14:21:41.480 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_common.ko 2024-04-23T14:21:41.480 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_generic.ko 2024-04-23T14:21:41.480 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/vmac.ko 2024-04-23T14:21:41.480 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/wp512.ko 2024-04-23T14:21:41.480 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xcbc.ko 2024-04-23T14:21:41.480 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xor.ko 2024-04-23T14:21:41.480 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xxhash_generic.ko 2024-04-23T14:21:41.481 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers 2024-04-23T14:21:41.481 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi 2024-04-23T14:21:41.481 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-23T14:21:41.481 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_pad.ko 2024-04-23T14:21:41.481 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/ec_sys.ko 2024-04-23T14:21:41.481 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/platform_profile.ko 2024-04-23T14:21:41.481 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbs.ko 2024-04-23T14:21:41.481 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbshc.ko 2024-04-23T14:21:41.481 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/video.ko 2024-04-23T14:21:41.481 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata 2024-04-23T14:21:41.481 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/acard-ahci.ko 2024-04-23T14:21:41.481 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ahci_platform.ko 2024-04-23T14:21:41.481 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ata_generic.ko 2024-04-23T14:21:41.481 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/libahci_platform.ko 2024-04-23T14:21:41.481 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_acpi.ko 2024-04-23T14:21:41.482 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ali.ko 2024-04-23T14:21:41.482 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_amd.ko 2024-04-23T14:21:41.482 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_artop.ko 2024-04-23T14:21:41.482 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atiixp.ko 2024-04-23T14:21:41.482 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atp867x.ko 2024-04-23T14:21:41.482 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd640.ko 2024-04-23T14:21:41.482 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd64x.ko 2024-04-23T14:21:41.482 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cypress.ko 2024-04-23T14:21:41.482 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_efar.ko 2024-04-23T14:21:41.482 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt366.ko 2024-04-23T14:21:41.482 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt37x.ko 2024-04-23T14:21:41.482 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-23T14:21:41.482 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-23T14:21:41.482 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it8213.ko 2024-04-23T14:21:41.482 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it821x.ko 2024-04-23T14:21:41.483 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_jmicron.ko 2024-04-23T14:21:41.483 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_marvell.ko 2024-04-23T14:21:41.483 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_mpiix.ko 2024-04-23T14:21:41.483 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_netcell.ko 2024-04-23T14:21:41.483 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ninja32.ko 2024-04-23T14:21:41.483 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87410.ko 2024-04-23T14:21:41.483 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87415.ko 2024-04-23T14:21:41.483 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_oldpiix.ko 2024-04-23T14:21:41.483 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_opti.ko 2024-04-23T14:21:41.483 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_optidma.ko 2024-04-23T14:21:41.483 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pcmcia.ko 2024-04-23T14:21:41.483 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-23T14:21:41.483 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-23T14:21:41.483 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_piccolo.ko 2024-04-23T14:21:41.484 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_rdc.ko 2024-04-23T14:21:41.484 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sch.ko 2024-04-23T14:21:41.484 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_serverworks.ko 2024-04-23T14:21:41.484 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sil680.ko 2024-04-23T14:21:41.484 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sis.ko 2024-04-23T14:21:41.484 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sl82c105.ko 2024-04-23T14:21:41.484 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_triflex.ko 2024-04-23T14:21:41.484 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_via.ko 2024-04-23T14:21:41.484 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pdc_adma.ko 2024-04-23T14:21:41.484 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_inic162x.ko 2024-04-23T14:21:41.484 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_mv.ko 2024-04-23T14:21:41.484 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_nv.ko 2024-04-23T14:21:41.484 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_promise.ko 2024-04-23T14:21:41.484 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_qstor.ko 2024-04-23T14:21:41.484 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil.ko 2024-04-23T14:21:41.485 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil24.ko 2024-04-23T14:21:41.485 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sis.ko 2024-04-23T14:21:41.485 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_svw.ko 2024-04-23T14:21:41.485 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sx4.ko 2024-04-23T14:21:41.485 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_uli.ko 2024-04-23T14:21:41.485 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_via.ko 2024-04-23T14:21:41.485 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_vsc.ko 2024-04-23T14:21:41.485 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay 2024-04-23T14:21:41.485 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay/ks0108.ko 2024-04-23T14:21:41.485 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base 2024-04-23T14:21:41.485 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap 2024-04-23T14:21:41.485 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-23T14:21:41.485 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-23T14:21:41.485 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma 2024-04-23T14:21:41.485 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma/bcma.ko 2024-04-23T14:21:41.486 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block 2024-04-23T14:21:41.486 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe 2024-04-23T14:21:41.486 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe/aoe.ko 2024-04-23T14:21:41.486 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/brd.ko 2024-04-23T14:21:41.486 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd 2024-04-23T14:21:41.486 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd/drbd.ko 2024-04-23T14:21:41.486 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/floppy.ko 2024-04-23T14:21:41.486 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/loop.ko 2024-04-23T14:21:41.486 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx 2024-04-23T14:21:41.486 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-23T14:21:41.486 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/nbd.ko 2024-04-23T14:21:41.486 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk 2024-04-23T14:21:41.486 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk/null_blk.ko 2024-04-23T14:21:41.486 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/pktcdvd.ko 2024-04-23T14:21:41.487 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/rbd.ko 2024-04-23T14:21:41.487 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/virtio_blk.ko 2024-04-23T14:21:41.487 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback 2024-04-23T14:21:41.487 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-23T14:21:41.487 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkfront.ko 2024-04-23T14:21:41.487 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram 2024-04-23T14:21:41.487 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram/zram.ko 2024-04-23T14:21:41.487 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char 2024-04-23T14:21:41.487 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hangcheck-timer.ko 2024-04-23T14:21:41.487 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random 2024-04-23T14:21:41.487 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-23T14:21:41.487 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-23T14:21:41.487 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-23T14:21:41.487 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/via-rng.ko 2024-04-23T14:21:41.488 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-23T14:21:41.488 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi 2024-04-23T14:21:41.488 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-23T14:21:41.488 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-23T14:21:41.488 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-23T14:21:41.488 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-23T14:21:41.488 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-23T14:21:41.488 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-23T14:21:41.488 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/lp.ko 2024-04-23T14:21:41.488 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave 2024-04-23T14:21:41.488 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave/mwave.ko 2024-04-23T14:21:41.488 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ppdev.ko 2024-04-23T14:21:41.488 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tlclk.ko 2024-04-23T14:21:41.488 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm 2024-04-23T14:21:41.488 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm.ko 2024-04-23T14:21:41.489 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-23T14:21:41.489 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-23T14:21:41.489 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-23T14:21:41.489 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-23T14:21:41.489 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-23T14:21:41.489 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/uv_mmtimer.ko 2024-04-23T14:21:41.489 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/virtio_console.ko 2024-04-23T14:21:41.489 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq 2024-04-23T14:21:41.489 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-23T14:21:41.489 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-23T14:21:41.489 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-23T14:21:41.489 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-23T14:21:41.489 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-23T14:21:41.489 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-23T14:21:41.490 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto 2024-04-23T14:21:41.490 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp 2024-04-23T14:21:41.490 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-23T14:21:41.490 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp.ko 2024-04-23T14:21:41.490 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel 2024-04-23T14:21:41.490 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat 2024-04-23T14:21:41.490 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common 2024-04-23T14:21:41.490 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-23T14:21:41.490 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-23T14:21:41.490 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-23T14:21:41.490 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-aes.ko 2024-04-23T14:21:41.490 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-sha.ko 2024-04-23T14:21:41.490 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio 2024-04-23T14:21:41.490 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-23T14:21:41.490 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca 2024-04-23T14:21:41.491 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca/dca.ko 2024-04-23T14:21:41.491 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq 2024-04-23T14:21:41.491 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-23T14:21:41.491 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma 2024-04-23T14:21:41.491 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw 2024-04-23T14:21:41.491 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-23T14:21:41.491 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat 2024-04-23T14:21:41.491 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-23T14:21:41.491 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac 2024-04-23T14:21:41.491 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/amd64_edac.ko 2024-04-23T14:21:41.491 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/e752x_edac.ko 2024-04-23T14:21:41.491 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/edac_mce_amd.ko 2024-04-23T14:21:41.491 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3000_edac.ko 2024-04-23T14:21:41.491 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3200_edac.ko 2024-04-23T14:21:41.492 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5100_edac.ko 2024-04-23T14:21:41.492 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5400_edac.ko 2024-04-23T14:21:41.492 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7300_edac.ko 2024-04-23T14:21:41.492 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7core_edac.ko 2024-04-23T14:21:41.492 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i82975x_edac.ko 2024-04-23T14:21:41.492 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/ie31200_edac.ko 2024-04-23T14:21:41.492 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/sb_edac.ko 2024-04-23T14:21:41.492 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/x38_edac.ko 2024-04-23T14:21:41.492 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware 2024-04-23T14:21:41.492 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/edd.ko 2024-04-23T14:21:41.492 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-23T14:21:41.492 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio 2024-04-23T14:21:41.492 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-ich.ko 2024-04-23T14:21:41.492 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-23T14:21:41.492 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid 2024-04-23T14:21:41.493 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-appleir.ko 2024-04-23T14:21:41.493 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-aureal.ko 2024-04-23T14:21:41.493 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-axff.ko 2024-04-23T14:21:41.493 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-dr.ko 2024-04-23T14:21:41.493 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elecom.ko 2024-04-23T14:21:41.493 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elo.ko 2024-04-23T14:21:41.493 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-emsff.ko 2024-04-23T14:21:41.493 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gaff.ko 2024-04-23T14:21:41.493 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gt683r.ko 2024-04-23T14:21:41.493 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gyration.ko 2024-04-23T14:21:41.493 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-23T14:21:41.493 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-23T14:21:41.493 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtekff.ko 2024-04-23T14:21:41.493 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-hyperv.ko 2024-04-23T14:21:41.493 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-icade.ko 2024-04-23T14:21:41.494 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-keytouch.ko 2024-04-23T14:21:41.494 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-kye.ko 2024-04-23T14:21:41.494 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lcpower.ko 2024-04-23T14:21:41.494 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-led.ko 2024-04-23T14:21:41.494 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lenovo.ko 2024-04-23T14:21:41.494 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-23T14:21:41.494 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-23T14:21:41.494 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-multitouch.ko 2024-04-23T14:21:41.494 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-ortek.ko 2024-04-23T14:21:41.494 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-penmount.ko 2024-04-23T14:21:41.494 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-petalynx.ko 2024-04-23T14:21:41.494 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-picolcd.ko 2024-04-23T14:21:41.494 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-pl.ko 2024-04-23T14:21:41.494 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-plantronics.ko 2024-04-23T14:21:41.495 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-primax.ko 2024-04-23T14:21:41.495 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-rmi.ko 2024-04-23T14:21:41.495 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-23T14:21:41.495 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-common.ko 2024-04-23T14:21:41.495 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-23T14:21:41.495 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-23T14:21:41.495 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-23T14:21:41.495 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-23T14:21:41.495 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-23T14:21:41.495 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-23T14:21:41.495 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-23T14:21:41.495 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-23T14:21:41.495 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-23T14:21:41.495 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat.ko 2024-04-23T14:21:41.495 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-saitek.ko 2024-04-23T14:21:41.496 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-samsung.ko 2024-04-23T14:21:41.496 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-23T14:21:41.496 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sjoy.ko 2024-04-23T14:21:41.496 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sony.ko 2024-04-23T14:21:41.496 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-speedlink.ko 2024-04-23T14:21:41.496 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-steelseries.ko 2024-04-23T14:21:41.496 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sunplus.ko 2024-04-23T14:21:41.496 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-23T14:21:41.496 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tivo.ko 2024-04-23T14:21:41.496 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tmff.ko 2024-04-23T14:21:41.496 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-topseed.ko 2024-04-23T14:21:41.496 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-twinhan.ko 2024-04-23T14:21:41.496 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-uclogic.ko 2024-04-23T14:21:41.496 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-waltop.ko 2024-04-23T14:21:41.496 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-wiimote.ko 2024-04-23T14:21:41.497 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-xinmo.ko 2024-04-23T14:21:41.497 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zpff.ko 2024-04-23T14:21:41.497 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zydacron.ko 2024-04-23T14:21:41.497 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/uhid.ko 2024-04-23T14:21:41.497 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/wacom.ko 2024-04-23T14:21:41.497 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv 2024-04-23T14:21:41.497 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_balloon.ko 2024-04-23T14:21:41.497 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_utils.ko 2024-04-23T14:21:41.497 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_vmbus.ko 2024-04-23T14:21:41.497 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon 2024-04-23T14:21:41.497 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru.ko 2024-04-23T14:21:41.497 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru3.ko 2024-04-23T14:21:41.497 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-23T14:21:41.497 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7414.ko 2024-04-23T14:21:41.497 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7418.ko 2024-04-23T14:21:41.497 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adc128d818.ko 2024-04-23T14:21:41.498 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1025.ko 2024-04-23T14:21:41.498 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1026.ko 2024-04-23T14:21:41.498 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1029.ko 2024-04-23T14:21:41.498 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1031.ko 2024-04-23T14:21:41.498 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm9240.ko 2024-04-23T14:21:41.498 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ads7828.ko 2024-04-23T14:21:41.498 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7410.ko 2024-04-23T14:21:41.498 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7411.ko 2024-04-23T14:21:41.498 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7462.ko 2024-04-23T14:21:41.498 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7470.ko 2024-04-23T14:21:41.498 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7475.ko 2024-04-23T14:21:41.498 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7x10.ko 2024-04-23T14:21:41.498 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/amc6821.ko 2024-04-23T14:21:41.498 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/applesmc.ko 2024-04-23T14:21:41.499 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asb100.ko 2024-04-23T14:21:41.499 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asc7621.ko 2024-04-23T14:21:41.499 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-23T14:21:41.499 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/atxp1.ko 2024-04-23T14:21:41.499 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/coretemp.ko 2024-04-23T14:21:41.499 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-23T14:21:41.499 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dme1737.ko 2024-04-23T14:21:41.499 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds1621.ko 2024-04-23T14:21:41.499 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds620.ko 2024-04-23T14:21:41.499 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc1403.ko 2024-04-23T14:21:41.499 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc6w201.ko 2024-04-23T14:21:41.499 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71805f.ko 2024-04-23T14:21:41.499 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71882fg.ko 2024-04-23T14:21:41.499 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f75375s.ko 2024-04-23T14:21:41.500 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fam15h_power.ko 2024-04-23T14:21:41.500 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fschmd.ko 2024-04-23T14:21:41.500 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g760a.ko 2024-04-23T14:21:41.500 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g762.ko 2024-04-23T14:21:41.500 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl518sm.ko 2024-04-23T14:21:41.500 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl520sm.ko 2024-04-23T14:21:41.500 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-23T14:21:41.500 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5500_temp.ko 2024-04-23T14:21:41.500 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5k_amb.ko 2024-04-23T14:21:41.500 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmaem.ko 2024-04-23T14:21:41.500 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmpex.ko 2024-04-23T14:21:41.500 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina209.ko 2024-04-23T14:21:41.500 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina2xx.ko 2024-04-23T14:21:41.500 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/it87.ko 2024-04-23T14:21:41.500 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k10temp.ko 2024-04-23T14:21:41.501 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k8temp.ko 2024-04-23T14:21:41.501 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lineage-pem.ko 2024-04-23T14:21:41.501 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm63.ko 2024-04-23T14:21:41.501 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm73.ko 2024-04-23T14:21:41.501 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm75.ko 2024-04-23T14:21:41.501 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm77.ko 2024-04-23T14:21:41.501 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm78.ko 2024-04-23T14:21:41.501 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm80.ko 2024-04-23T14:21:41.501 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm83.ko 2024-04-23T14:21:41.501 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm85.ko 2024-04-23T14:21:41.501 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm87.ko 2024-04-23T14:21:41.501 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm90.ko 2024-04-23T14:21:41.501 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm92.ko 2024-04-23T14:21:41.501 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm93.ko 2024-04-23T14:21:41.501 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95234.ko 2024-04-23T14:21:41.502 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95241.ko 2024-04-23T14:21:41.502 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95245.ko 2024-04-23T14:21:41.502 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc2945.ko 2024-04-23T14:21:41.502 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4151.ko 2024-04-23T14:21:41.502 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4215.ko 2024-04-23T14:21:41.502 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4222.ko 2024-04-23T14:21:41.502 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4245.ko 2024-04-23T14:21:41.502 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4260.ko 2024-04-23T14:21:41.502 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4261.ko 2024-04-23T14:21:41.502 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max16065.ko 2024-04-23T14:21:41.502 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1619.ko 2024-04-23T14:21:41.502 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1668.ko 2024-04-23T14:21:41.502 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max197.ko 2024-04-23T14:21:41.502 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6639.ko 2024-04-23T14:21:41.503 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6650.ko 2024-04-23T14:21:41.503 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6697.ko 2024-04-23T14:21:41.503 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/mcp3021.ko 2024-04-23T14:21:41.503 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6683.ko 2024-04-23T14:21:41.503 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775-core.ko 2024-04-23T14:21:41.503 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775.ko 2024-04-23T14:21:41.503 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct7802.ko 2024-04-23T14:21:41.503 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87360.ko 2024-04-23T14:21:41.503 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87427.ko 2024-04-23T14:21:41.503 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pcf8591.ko 2024-04-23T14:21:41.503 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus 2024-04-23T14:21:41.503 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-23T14:21:41.503 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-23T14:21:41.503 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-23T14:21:41.503 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-23T14:21:41.504 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-23T14:21:41.504 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-23T14:21:41.504 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-23T14:21:41.504 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-23T14:21:41.504 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-23T14:21:41.504 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-23T14:21:41.504 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-23T14:21:41.504 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-23T14:21:41.504 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/powr1220.ko 2024-04-23T14:21:41.504 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5627.ko 2024-04-23T14:21:41.504 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5636.ko 2024-04-23T14:21:41.504 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-23T14:21:41.504 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht15.ko 2024-04-23T14:21:41.504 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht21.ko 2024-04-23T14:21:41.504 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/shtc1.ko 2024-04-23T14:21:41.505 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sis5595.ko 2024-04-23T14:21:41.505 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47b397.ko 2024-04-23T14:21:41.505 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m1.ko 2024-04-23T14:21:41.505 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m192.ko 2024-04-23T14:21:41.505 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/thmc50.ko 2024-04-23T14:21:41.505 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp102.ko 2024-04-23T14:21:41.505 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp103.ko 2024-04-23T14:21:41.505 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp401.ko 2024-04-23T14:21:41.505 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp421.ko 2024-04-23T14:21:41.505 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via-cputemp.ko 2024-04-23T14:21:41.505 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via686a.ko 2024-04-23T14:21:41.505 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt1211.ko 2024-04-23T14:21:41.505 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt8231.ko 2024-04-23T14:21:41.505 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627ehf.ko 2024-04-23T14:21:41.505 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627hf.ko 2024-04-23T14:21:41.506 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83781d.ko 2024-04-23T14:21:41.506 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83791d.ko 2024-04-23T14:21:41.506 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83792d.ko 2024-04-23T14:21:41.506 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83793.ko 2024-04-23T14:21:41.506 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83795.ko 2024-04-23T14:21:41.506 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l785ts.ko 2024-04-23T14:21:41.506 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l786ng.ko 2024-04-23T14:21:41.506 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c 2024-04-23T14:21:41.506 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos 2024-04-23T14:21:41.506 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-23T14:21:41.506 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-23T14:21:41.506 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses 2024-04-23T14:21:41.506 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-23T14:21:41.506 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-23T14:21:41.507 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-23T14:21:41.507 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-23T14:21:41.507 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-23T14:21:41.507 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-23T14:21:41.507 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-23T14:21:41.507 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-23T14:21:41.507 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-23T14:21:41.507 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-23T14:21:41.507 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-23T14:21:41.507 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-23T14:21:41.507 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-23T14:21:41.507 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-23T14:21:41.507 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-23T14:21:41.507 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-23T14:21:41.507 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-23T14:21:41.508 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-23T14:21:41.508 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-23T14:21:41.508 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-23T14:21:41.508 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-23T14:21:41.508 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-23T14:21:41.508 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-23T14:21:41.508 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-dev.ko 2024-04-23T14:21:41.508 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-mux.ko 2024-04-23T14:21:41.508 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-23T14:21:41.508 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-smbus.ko 2024-04-23T14:21:41.508 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-stub.ko 2024-04-23T14:21:41.508 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband 2024-04-23T14:21:41.508 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core 2024-04-23T14:21:41.508 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-23T14:21:41.509 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_core.ko 2024-04-23T14:21:41.509 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-23T14:21:41.509 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-23T14:21:41.509 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-23T14:21:41.509 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-23T14:21:41.509 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-23T14:21:41.509 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw 2024-04-23T14:21:41.509 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4 2024-04-23T14:21:41.509 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-23T14:21:41.509 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4 2024-04-23T14:21:41.509 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-23T14:21:41.509 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5 2024-04-23T14:21:41.509 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-23T14:21:41.509 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca 2024-04-23T14:21:41.509 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-23T14:21:41.510 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp 2024-04-23T14:21:41.510 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib 2024-04-23T14:21:41.510 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-23T14:21:41.510 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser 2024-04-23T14:21:41.510 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-23T14:21:41.510 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert 2024-04-23T14:21:41.510 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-23T14:21:41.510 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp 2024-04-23T14:21:41.510 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-23T14:21:41.510 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt 2024-04-23T14:21:41.510 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-23T14:21:41.510 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input 2024-04-23T14:21:41.510 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport 2024-04-23T14:21:41.510 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-23T14:21:41.511 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-23T14:21:41.511 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/gameport.ko 2024-04-23T14:21:41.511 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/joydev.ko 2024-04-23T14:21:41.511 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard 2024-04-23T14:21:41.511 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-23T14:21:41.511 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse 2024-04-23T14:21:41.511 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/appletouch.ko 2024-04-23T14:21:41.511 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/bcm5974.ko 2024-04-23T14:21:41.511 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/cyapatp.ko 2024-04-23T14:21:41.511 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-23T14:21:41.511 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/sermouse.ko 2024-04-23T14:21:41.511 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-23T14:21:41.511 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-23T14:21:41.511 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-23T14:21:41.511 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4 2024-04-23T14:21:41.512 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-23T14:21:41.512 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio 2024-04-23T14:21:41.512 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/altera_ps2.ko 2024-04-23T14:21:41.512 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/arc_ps2.ko 2024-04-23T14:21:41.512 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-23T14:21:41.512 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/serio_raw.ko 2024-04-23T14:21:41.512 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/sparse-keymap.ko 2024-04-23T14:21:41.512 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds 2024-04-23T14:21:41.512 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-blinkm.ko 2024-04-23T14:21:41.512 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lm3530.ko 2024-04-23T14:21:41.512 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lp3944.ko 2024-04-23T14:21:41.512 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-ss4200.ko 2024-04-23T14:21:41.512 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger 2024-04-23T14:21:41.512 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-23T14:21:41.512 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-23T14:21:41.513 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-23T14:21:41.513 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-23T14:21:41.513 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-23T14:21:41.513 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-23T14:21:41.513 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-23T14:21:41.513 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-23T14:21:41.513 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-23T14:21:41.513 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md 2024-04-23T14:21:41.513 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache 2024-04-23T14:21:41.513 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache/bcache.ko 2024-04-23T14:21:41.513 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-bio-prison.ko 2024-04-23T14:21:41.513 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache-smq.ko 2024-04-23T14:21:41.513 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache.ko 2024-04-23T14:21:41.513 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-crypt.ko 2024-04-23T14:21:41.514 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-delay.ko 2024-04-23T14:21:41.514 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-flakey.ko 2024-04-23T14:21:41.514 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-log-userspace.ko 2024-04-23T14:21:41.514 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-multipath.ko 2024-04-23T14:21:41.514 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-queue-length.ko 2024-04-23T14:21:41.514 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-raid.ko 2024-04-23T14:21:41.514 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-round-robin.ko 2024-04-23T14:21:41.514 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-service-time.ko 2024-04-23T14:21:41.514 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-switch.ko 2024-04-23T14:21:41.514 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-thin-pool.ko 2024-04-23T14:21:41.515 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-verity.ko 2024-04-23T14:21:41.515 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data 2024-04-23T14:21:41.515 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-23T14:21:41.515 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid0.ko 2024-04-23T14:21:41.515 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid1.ko 2024-04-23T14:21:41.515 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid10.ko 2024-04-23T14:21:41.515 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid456.ko 2024-04-23T14:21:41.515 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message 2024-04-23T14:21:41.515 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion 2024-04-23T14:21:41.515 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptbase.ko 2024-04-23T14:21:41.515 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptctl.ko 2024-04-23T14:21:41.515 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptfc.ko 2024-04-23T14:21:41.515 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptlan.ko 2024-04-23T14:21:41.515 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptsas.ko 2024-04-23T14:21:41.516 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptscsih.ko 2024-04-23T14:21:41.516 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptspi.ko 2024-04-23T14:21:41.516 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd 2024-04-23T14:21:41.516 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_ich.ko 2024-04-23T14:21:41.516 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_sch.ko 2024-04-23T14:21:41.516 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/mfd-core.ko 2024-04-23T14:21:41.516 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/sm501.ko 2024-04-23T14:21:41.516 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/viperboard.ko 2024-04-23T14:21:41.516 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/vx855.ko 2024-04-23T14:21:41.516 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/wl1273-core.ko 2024-04-23T14:21:41.516 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc 2024-04-23T14:21:41.516 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl 2024-04-23T14:21:41.516 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-23T14:21:41.516 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds9802als.ko 2024-04-23T14:21:41.516 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds990x.ko 2024-04-23T14:21:41.517 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/bh1770glc.ko 2024-04-23T14:21:41.517 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710 2024-04-23T14:21:41.517 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710/cb710.ko 2024-04-23T14:21:41.517 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom 2024-04-23T14:21:41.517 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/at24.ko 2024-04-23T14:21:41.517 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-23T14:21:41.517 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/max6875.ko 2024-04-23T14:21:41.517 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/enclosure.ko 2024-04-23T14:21:41.517 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/hpilo.ko 2024-04-23T14:21:41.517 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29003.ko 2024-04-23T14:21:41.517 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29020.ko 2024-04-23T14:21:41.517 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d 2024-04-23T14:21:41.517 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-23T14:21:41.517 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-23T14:21:41.517 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei 2024-04-23T14:21:41.518 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-me.ko 2024-04-23T14:21:41.518 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-txe.ko 2024-04-23T14:21:41.518 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei.ko 2024-04-23T14:21:41.518 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru 2024-04-23T14:21:41.518 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-23T14:21:41.518 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp 2024-04-23T14:21:41.518 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-23T14:21:41.518 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-23T14:21:41.518 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-23T14:21:41.518 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_7xx1.ko 2024-04-23T14:21:41.518 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_core.ko 2024-04-23T14:21:41.518 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tsl2550.ko 2024-04-23T14:21:41.518 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_balloon.ko 2024-04-23T14:21:41.518 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci 2024-04-23T14:21:41.519 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-23T14:21:41.519 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net 2024-04-23T14:21:41.519 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding 2024-04-23T14:21:41.519 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding/bonding.ko 2024-04-23T14:21:41.519 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa 2024-04-23T14:21:41.519 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53 2024-04-23T14:21:41.519 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-23T14:21:41.519 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-23T14:21:41.519 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-23T14:21:41.519 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx 2024-04-23T14:21:41.519 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-23T14:21:41.519 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dummy.ko 2024-04-23T14:21:41.519 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/eql.ko 2024-04-23T14:21:41.519 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet 2024-04-23T14:21:41.520 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com 2024-04-23T14:21:41.520 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-23T14:21:41.520 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-23T14:21:41.520 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-23T14:21:41.520 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-23T14:21:41.520 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390 2024-04-23T14:21:41.520 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-23T14:21:41.520 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-23T14:21:41.520 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-23T14:21:41.520 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-23T14:21:41.520 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec 2024-04-23T14:21:41.520 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-23T14:21:41.520 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere 2024-04-23T14:21:41.520 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-23T14:21:41.520 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon 2024-04-23T14:21:41.521 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-23T14:21:41.521 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera 2024-04-23T14:21:41.521 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-23T14:21:41.521 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd 2024-04-23T14:21:41.521 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-23T14:21:41.521 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-23T14:21:41.521 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-23T14:21:41.521 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros 2024-04-23T14:21:41.521 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx 2024-04-23T14:21:41.521 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-23T14:21:41.521 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-23T14:21:41.521 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-23T14:21:41.521 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-23T14:21:41.521 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-23T14:21:41.521 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx 2024-04-23T14:21:41.522 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-23T14:21:41.522 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-23T14:21:41.522 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom 2024-04-23T14:21:41.522 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-23T14:21:41.522 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-23T14:21:41.522 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-23T14:21:41.522 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-23T14:21:41.522 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-23T14:21:41.522 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet 2024-04-23T14:21:41.522 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-23T14:21:41.522 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-23T14:21:41.522 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade 2024-04-23T14:21:41.522 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna 2024-04-23T14:21:41.523 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-23T14:21:41.523 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio 2024-04-23T14:21:41.523 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-23T14:21:41.523 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-23T14:21:41.523 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-23T14:21:41.523 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-23T14:21:41.523 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-23T14:21:41.523 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-23T14:21:41.523 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-23T14:21:41.523 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-23T14:21:41.523 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-23T14:21:41.523 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-23T14:21:41.523 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco 2024-04-23T14:21:41.523 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic 2024-04-23T14:21:41.523 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-23T14:21:41.524 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec 2024-04-23T14:21:41.524 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip 2024-04-23T14:21:41.524 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-23T14:21:41.524 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-23T14:21:41.524 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-23T14:21:41.524 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-23T14:21:41.524 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-23T14:21:41.524 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-23T14:21:41.524 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink 2024-04-23T14:21:41.524 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-23T14:21:41.524 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-23T14:21:41.524 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dnet.ko 2024-04-23T14:21:41.524 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex 2024-04-23T14:21:41.524 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet 2024-04-23T14:21:41.524 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-23T14:21:41.525 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ethoc.ko 2024-04-23T14:21:41.525 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/fealnx.ko 2024-04-23T14:21:41.525 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel 2024-04-23T14:21:41.525 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-23T14:21:41.525 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000 2024-04-23T14:21:41.525 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-23T14:21:41.525 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e 2024-04-23T14:21:41.525 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-23T14:21:41.525 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k 2024-04-23T14:21:41.525 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-23T14:21:41.525 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e 2024-04-23T14:21:41.525 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-23T14:21:41.525 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf 2024-04-23T14:21:41.525 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-23T14:21:41.526 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb 2024-04-23T14:21:41.526 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-23T14:21:41.526 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf 2024-04-23T14:21:41.526 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-23T14:21:41.526 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-23T14:21:41.526 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-23T14:21:41.526 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-23T14:21:41.526 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-23T14:21:41.526 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/jme.ko 2024-04-23T14:21:41.526 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell 2024-04-23T14:21:41.526 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-23T14:21:41.526 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-23T14:21:41.526 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-23T14:21:41.526 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox 2024-04-23T14:21:41.527 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-23T14:21:41.527 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-23T14:21:41.527 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-23T14:21:41.527 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-23T14:21:41.527 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-23T14:21:41.527 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-23T14:21:41.527 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel 2024-04-23T14:21:41.527 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-23T14:21:41.527 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom 2024-04-23T14:21:41.527 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-23T14:21:41.527 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-23T14:21:41.527 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi 2024-04-23T14:21:41.527 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-23T14:21:41.527 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-23T14:21:41.528 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion 2024-04-23T14:21:41.528 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-23T14:21:41.528 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia 2024-04-23T14:21:41.528 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-23T14:21:41.528 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines 2024-04-23T14:21:41.528 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-23T14:21:41.528 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-23T14:21:41.528 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic 2024-04-23T14:21:41.528 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-23T14:21:41.528 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-23T14:21:41.528 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-23T14:21:41.528 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-23T14:21:41.528 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-23T14:21:41.528 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc 2024-04-23T14:21:41.528 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-23T14:21:41.529 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek 2024-04-23T14:21:41.529 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-23T14:21:41.529 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-23T14:21:41.529 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-23T14:21:41.529 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-23T14:21:41.529 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker 2024-04-23T14:21:41.529 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-23T14:21:41.529 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc 2024-04-23T14:21:41.529 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-23T14:21:41.529 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan 2024-04-23T14:21:41.529 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-23T14:21:41.529 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis 2024-04-23T14:21:41.529 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-23T14:21:41.530 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-23T14:21:41.530 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc 2024-04-23T14:21:41.530 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-23T14:21:41.530 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-23T14:21:41.530 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-23T14:21:41.530 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-23T14:21:41.530 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro 2024-04-23T14:21:41.530 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-23T14:21:41.530 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-23T14:21:41.530 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-23T14:21:41.530 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun 2024-04-23T14:21:41.530 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-23T14:21:41.530 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-23T14:21:41.530 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-23T14:21:41.530 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-23T14:21:41.531 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti 2024-04-23T14:21:41.531 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-23T14:21:41.531 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti 2024-04-23T14:21:41.531 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-23T14:21:41.531 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via 2024-04-23T14:21:41.531 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-23T14:21:41.531 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-23T14:21:41.531 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet 2024-04-23T14:21:41.531 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-23T14:21:41.531 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-23T14:21:41.531 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom 2024-04-23T14:21:41.531 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-23T14:21:41.531 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/geneve.ko 2024-04-23T14:21:41.531 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv 2024-04-23T14:21:41.532 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-23T14:21:41.532 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154 2024-04-23T14:21:41.532 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-23T14:21:41.532 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ifb.ko 2024-04-23T14:21:41.532 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan 2024-04-23T14:21:41.532 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-23T14:21:41.532 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvlan.ko 2024-04-23T14:21:41.532 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvtap.ko 2024-04-23T14:21:41.532 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio 2024-04-23T14:21:41.532 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio.ko 2024-04-23T14:21:41.532 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-23T14:21:41.532 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-23T14:21:41.532 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-23T14:21:41.532 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mii.ko 2024-04-23T14:21:41.532 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/net_failover.ko 2024-04-23T14:21:41.533 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/netconsole.ko 2024-04-23T14:21:41.533 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/nlmon.ko 2024-04-23T14:21:41.533 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs 2024-04-23T14:21:41.533 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-23T14:21:41.533 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-23T14:21:41.533 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy 2024-04-23T14:21:41.533 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/amd.ko 2024-04-23T14:21:41.533 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-23T14:21:41.533 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-23T14:21:41.533 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-23T14:21:41.533 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm87xx.ko 2024-04-23T14:21:41.533 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/broadcom.ko 2024-04-23T14:21:41.533 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/cicada.ko 2024-04-23T14:21:41.533 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/davicom.ko 2024-04-23T14:21:41.534 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/dp83640.ko 2024-04-23T14:21:41.534 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/et1011c.ko 2024-04-23T14:21:41.534 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/icplus.ko 2024-04-23T14:21:41.534 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/lxt.ko 2024-04-23T14:21:41.534 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/marvell.ko 2024-04-23T14:21:41.534 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/micrel.ko 2024-04-23T14:21:41.534 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/national.ko 2024-04-23T14:21:41.534 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/phylink.ko 2024-04-23T14:21:41.534 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/qsemi.ko 2024-04-23T14:21:41.534 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/realtek.ko 2024-04-23T14:21:41.534 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/smsc.ko 2024-04-23T14:21:41.534 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/ste10Xp.ko 2024-04-23T14:21:41.534 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/vitesse.ko 2024-04-23T14:21:41.534 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp 2024-04-23T14:21:41.535 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-23T14:21:41.535 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_async.ko 2024-04-23T14:21:41.535 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-23T14:21:41.535 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-23T14:21:41.535 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-23T14:21:41.535 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-23T14:21:41.535 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppoe.ko 2024-04-23T14:21:41.535 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppox.ko 2024-04-23T14:21:41.535 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pptp.ko 2024-04-23T14:21:41.535 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip 2024-04-23T14:21:41.535 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slhc.ko 2024-04-23T14:21:41.535 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slip.ko 2024-04-23T14:21:41.535 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/sungem_phy.ko 2024-04-23T14:21:41.535 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tap.ko 2024-04-23T14:21:41.536 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team 2024-04-23T14:21:41.536 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team.ko 2024-04-23T14:21:41.536 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-23T14:21:41.536 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-23T14:21:41.536 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-23T14:21:41.536 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_random.ko 2024-04-23T14:21:41.536 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-23T14:21:41.536 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tun.ko 2024-04-23T14:21:41.536 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/veth.ko 2024-04-23T14:21:41.536 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/virtio_net.ko 2024-04-23T14:21:41.536 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3 2024-04-23T14:21:41.536 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-23T14:21:41.536 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan 2024-04-23T14:21:41.536 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan/vxlan.ko 2024-04-23T14:21:41.537 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback 2024-04-23T14:21:41.537 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-23T14:21:41.537 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netfront.ko 2024-04-23T14:21:41.537 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb 2024-04-23T14:21:41.537 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb/ntb.ko 2024-04-23T14:21:41.537 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme 2024-04-23T14:21:41.537 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host 2024-04-23T14:21:41.537 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme-core.ko 2024-04-23T14:21:41.537 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme.ko 2024-04-23T14:21:41.537 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport 2024-04-23T14:21:41.537 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport.ko 2024-04-23T14:21:41.537 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_cs.ko 2024-04-23T14:21:41.537 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_pc.ko 2024-04-23T14:21:41.537 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_serial.ko 2024-04-23T14:21:41.537 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci 2024-04-23T14:21:41.538 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug 2024-04-23T14:21:41.538 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-23T14:21:41.538 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie 2024-04-23T14:21:41.538 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-23T14:21:41.538 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/xen-pcifront.ko 2024-04-23T14:21:41.538 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia 2024-04-23T14:21:41.538 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/i82092.ko 2024-04-23T14:21:41.538 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/pd6729.ko 2024-04-23T14:21:41.538 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-23T14:21:41.538 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl 2024-04-23T14:21:41.538 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel 2024-04-23T14:21:41.538 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-23T14:21:41.538 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform 2024-04-23T14:21:41.539 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86 2024-04-23T14:21:41.539 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-23T14:21:41.539 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acerhdf.ko 2024-04-23T14:21:41.539 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-23T14:21:41.539 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-23T14:21:41.539 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-23T14:21:41.539 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-23T14:21:41.539 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-23T14:21:41.539 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-23T14:21:41.539 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-23T14:21:41.539 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-23T14:21:41.539 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-23T14:21:41.539 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/hdaps.ko 2024-04-23T14:21:41.540 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel 2024-04-23T14:21:41.540 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-23T14:21:41.540 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel_ips.ko 2024-04-23T14:21:41.540 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-23T14:21:41.540 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-23T14:21:41.540 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-23T14:21:41.540 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-23T14:21:41.540 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-23T14:21:41.540 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-23T14:21:41.540 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-23T14:21:41.540 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-23T14:21:41.540 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-23T14:21:41.540 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi.ko 2024-04-23T14:21:41.540 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap 2024-04-23T14:21:41.540 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-23T14:21:41.541 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-23T14:21:41.541 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps 2024-04-23T14:21:41.541 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients 2024-04-23T14:21:41.541 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-23T14:21:41.541 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-23T14:21:41.541 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps_parport.ko 2024-04-23T14:21:41.541 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/pps_core.ko 2024-04-23T14:21:41.541 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp 2024-04-23T14:21:41.541 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp.ko 2024-04-23T14:21:41.541 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp_kvm.ko 2024-04-23T14:21:41.541 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras 2024-04-23T14:21:41.541 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd 2024-04-23T14:21:41.541 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl 2024-04-23T14:21:41.541 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-23T14:21:41.542 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/fmpm.ko 2024-04-23T14:21:41.542 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc 2024-04-23T14:21:41.542 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-23T14:21:41.542 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-23T14:21:41.542 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-23T14:21:41.542 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-23T14:21:41.542 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-23T14:21:41.542 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-23T14:21:41.542 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-23T14:21:41.542 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-23T14:21:41.542 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-23T14:21:41.542 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-23T14:21:41.542 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-em3027.ko 2024-04-23T14:21:41.542 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-23T14:21:41.543 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-23T14:21:41.543 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-23T14:21:41.543 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-23T14:21:41.543 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-23T14:21:41.543 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-23T14:21:41.543 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-max6900.ko 2024-04-23T14:21:41.543 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-23T14:21:41.543 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-23T14:21:41.543 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-23T14:21:41.543 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-23T14:21:41.543 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-23T14:21:41.543 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-23T14:21:41.543 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-23T14:21:41.543 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-23T14:21:41.543 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-23T14:21:41.543 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-23T14:21:41.544 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-23T14:21:41.544 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-23T14:21:41.544 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-x1205.ko 2024-04-23T14:21:41.544 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi 2024-04-23T14:21:41.544 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-9xxx.ko 2024-04-23T14:21:41.544 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-sas.ko 2024-04-23T14:21:41.544 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-xxxx.ko 2024-04-23T14:21:41.544 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/BusLogic.ko 2024-04-23T14:21:41.544 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/a100u2w.ko 2024-04-23T14:21:41.544 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid 2024-04-23T14:21:41.544 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-23T14:21:41.544 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/advansys.ko 2024-04-23T14:21:41.544 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx 2024-04-23T14:21:41.544 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-23T14:21:41.544 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-23T14:21:41.545 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/am53c974.ko 2024-04-23T14:21:41.545 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr 2024-04-23T14:21:41.545 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-23T14:21:41.545 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/atp870u.ko 2024-04-23T14:21:41.545 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi 2024-04-23T14:21:41.545 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-23T14:21:41.545 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa 2024-04-23T14:21:41.545 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa/bfa.ko 2024-04-23T14:21:41.545 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc 2024-04-23T14:21:41.545 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-23T14:21:41.545 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i 2024-04-23T14:21:41.545 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-23T14:21:41.545 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ch.ko 2024-04-23T14:21:41.545 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor 2024-04-23T14:21:41.545 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-23T14:21:41.546 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi 2024-04-23T14:21:41.546 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-23T14:21:41.546 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-23T14:21:41.546 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-23T14:21:41.546 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-23T14:21:41.546 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-23T14:21:41.546 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dc395x.ko 2024-04-23T14:21:41.546 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler 2024-04-23T14:21:41.546 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-23T14:21:41.546 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-23T14:21:41.546 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-23T14:21:41.546 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-23T14:21:41.546 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dmx3191d.ko 2024-04-23T14:21:41.547 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r 2024-04-23T14:21:41.547 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-23T14:21:41.547 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esp_scsi.ko 2024-04-23T14:21:41.547 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe 2024-04-23T14:21:41.547 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-23T14:21:41.547 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-23T14:21:41.547 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic 2024-04-23T14:21:41.547 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic/fnic.ko 2024-04-23T14:21:41.547 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hpsa.ko 2024-04-23T14:21:41.547 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hptiop.ko 2024-04-23T14:21:41.547 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hv_storvsc.ko 2024-04-23T14:21:41.547 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/initio.ko 2024-04-23T14:21:41.547 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ipr.ko 2024-04-23T14:21:41.547 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ips.ko 2024-04-23T14:21:41.547 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci 2024-04-23T14:21:41.547 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci/isci.ko 2024-04-23T14:21:41.548 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-23T14:21:41.548 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-23T14:21:41.548 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc 2024-04-23T14:21:41.548 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc/libfc.ko 2024-04-23T14:21:41.548 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi.ko 2024-04-23T14:21:41.548 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-23T14:21:41.548 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas 2024-04-23T14:21:41.548 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas/libsas.ko 2024-04-23T14:21:41.548 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc 2024-04-23T14:21:41.548 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-23T14:21:41.548 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid 2024-04-23T14:21:41.548 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid.ko 2024-04-23T14:21:41.548 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-23T14:21:41.548 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-23T14:21:41.549 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-23T14:21:41.549 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas 2024-04-23T14:21:41.549 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-23T14:21:41.549 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas 2024-04-23T14:21:41.549 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-23T14:21:41.549 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvumi.ko 2024-04-23T14:21:41.549 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001 2024-04-23T14:21:41.549 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-23T14:21:41.549 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pmcraid.ko 2024-04-23T14:21:41.549 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla1280.ko 2024-04-23T14:21:41.549 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx 2024-04-23T14:21:41.549 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-23T14:21:41.549 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-23T14:21:41.549 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx 2024-04-23T14:21:41.549 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-23T14:21:41.549 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/raid_class.ko 2024-04-23T14:21:41.550 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_debug.ko 2024-04-23T14:21:41.550 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-23T14:21:41.550 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-23T14:21:41.550 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-23T14:21:41.550 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-23T14:21:41.550 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-23T14:21:41.550 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ses.ko 2024-04-23T14:21:41.550 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/st.ko 2024-04-23T14:21:41.550 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/stex.ko 2024-04-23T14:21:41.550 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2 2024-04-23T14:21:41.550 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-23T14:21:41.550 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/virtio_scsi.ko 2024-04-23T14:21:41.550 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-23T14:21:41.550 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/wd719x.ko 2024-04-23T14:21:41.550 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb 2024-04-23T14:21:41.551 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb/ssb.ko 2024-04-23T14:21:41.551 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target 2024-04-23T14:21:41.551 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi 2024-04-23T14:21:41.551 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-23T14:21:41.551 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback 2024-04-23T14:21:41.551 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-23T14:21:41.551 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_file.ko 2024-04-23T14:21:41.551 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_iblock.ko 2024-04-23T14:21:41.551 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_mod.ko 2024-04-23T14:21:41.551 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_pscsi.ko 2024-04-23T14:21:41.551 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_user.ko 2024-04-23T14:21:41.551 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc 2024-04-23T14:21:41.551 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-23T14:21:41.551 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal 2024-04-23T14:21:41.551 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel 2024-04-23T14:21:41.552 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal 2024-04-23T14:21:41.552 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-23T14:21:41.552 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-23T14:21:41.552 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-23T14:21:41.552 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-23T14:21:41.552 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-23T14:21:41.552 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-23T14:21:41.552 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-23T14:21:41.552 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-23T14:21:41.552 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-23T14:21:41.552 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-23T14:21:41.552 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-23T14:21:41.552 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-23T14:21:41.552 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-23T14:21:41.552 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-23T14:21:41.553 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-23T14:21:41.553 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-23T14:21:41.553 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-23T14:21:41.553 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-23T14:21:41.553 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty 2024-04-23T14:21:41.553 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless 2024-04-23T14:21:41.553 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-23T14:21:41.553 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_gsm.ko 2024-04-23T14:21:41.553 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_hdlc.ko 2024-04-23T14:21:41.553 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/nozomi.ko 2024-04-23T14:21:41.553 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial 2024-04-23T14:21:41.553 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250 2024-04-23T14:21:41.553 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-23T14:21:41.553 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/arc_uart.ko 2024-04-23T14:21:41.553 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm 2024-04-23T14:21:41.554 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-23T14:21:41.554 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs 2024-04-23T14:21:41.554 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core 2024-04-23T14:21:41.554 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-23T14:21:41.554 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host 2024-04-23T14:21:41.554 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-23T14:21:41.554 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio 2024-04-23T14:21:41.554 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio.ko 2024-04-23T14:21:41.554 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_aec.ko 2024-04-23T14:21:41.554 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_cif.ko 2024-04-23T14:21:41.554 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_pci_generic.ko 2024-04-23T14:21:41.554 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_sercos3.ko 2024-04-23T14:21:41.554 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb 2024-04-23T14:21:41.554 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class 2024-04-23T14:21:41.555 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-acm.ko 2024-04-23T14:21:41.555 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-23T14:21:41.555 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usblp.ko 2024-04-23T14:21:41.555 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usbtmc.ko 2024-04-23T14:21:41.555 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host 2024-04-23T14:21:41.555 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-23T14:21:41.555 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image 2024-04-23T14:21:41.555 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/mdc800.ko 2024-04-23T14:21:41.555 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/microtek.ko 2024-04-23T14:21:41.555 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc 2024-04-23T14:21:41.555 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/adutux.ko 2024-04-23T14:21:41.555 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/appledisplay.ko 2024-04-23T14:21:41.555 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi26.ko 2024-04-23T14:21:41.555 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi62.ko 2024-04-23T14:21:41.555 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ezusb.ko 2024-04-23T14:21:41.556 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/idmouse.ko 2024-04-23T14:21:41.556 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/iowarrior.ko 2024-04-23T14:21:41.556 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-23T14:21:41.556 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ldusb.ko 2024-04-23T14:21:41.556 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/legousbtower.ko 2024-04-23T14:21:41.556 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga 2024-04-23T14:21:41.556 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-23T14:21:41.556 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-23T14:21:41.556 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usb3503.ko 2024-04-23T14:21:41.556 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usblcd.ko 2024-04-23T14:21:41.556 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-23T14:21:41.556 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/uss720.ko 2024-04-23T14:21:41.556 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/yurex.ko 2024-04-23T14:21:41.556 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy 2024-04-23T14:21:41.556 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy/phy-generic.ko 2024-04-23T14:21:41.557 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial 2024-04-23T14:21:41.557 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/aircable.ko 2024-04-23T14:21:41.557 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ark3116.ko 2024-04-23T14:21:41.557 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-23T14:21:41.557 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ch341.ko 2024-04-23T14:21:41.557 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cp210x.ko 2024-04-23T14:21:41.557 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cyberjack.ko 2024-04-23T14:21:41.557 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-23T14:21:41.557 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-23T14:21:41.557 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/empeg.ko 2024-04-23T14:21:41.557 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-23T14:21:41.557 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-23T14:21:41.557 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-23T14:21:41.557 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_ti.ko 2024-04-23T14:21:41.558 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipaq.ko 2024-04-23T14:21:41.558 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipw.ko 2024-04-23T14:21:41.558 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ir-usb.ko 2024-04-23T14:21:41.558 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-23T14:21:41.558 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan.ko 2024-04-23T14:21:41.558 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-23T14:21:41.558 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-23T14:21:41.558 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-23T14:21:41.558 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mct_u232.ko 2024-04-23T14:21:41.558 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7720.ko 2024-04-23T14:21:41.558 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7840.ko 2024-04-23T14:21:41.558 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/navman.ko 2024-04-23T14:21:41.558 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/omninet.ko 2024-04-23T14:21:41.558 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/opticon.ko 2024-04-23T14:21:41.559 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/option.ko 2024-04-23T14:21:41.559 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/oti6858.ko 2024-04-23T14:21:41.559 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/pl2303.ko 2024-04-23T14:21:41.559 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcaux.ko 2024-04-23T14:21:41.559 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcserial.ko 2024-04-23T14:21:41.559 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/quatech2.ko 2024-04-23T14:21:41.559 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/safe_serial.ko 2024-04-23T14:21:41.559 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/sierra.ko 2024-04-23T14:21:41.559 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-23T14:21:41.559 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ssu100.ko 2024-04-23T14:21:41.559 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/symbolserial.ko 2024-04-23T14:21:41.559 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-23T14:21:41.559 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-23T14:21:41.559 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_debug.ko 2024-04-23T14:21:41.559 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-23T14:21:41.559 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/visor.ko 2024-04-23T14:21:41.560 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/whiteheat.ko 2024-04-23T14:21:41.560 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-23T14:21:41.560 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage 2024-04-23T14:21:41.560 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/uas.ko 2024-04-23T14:21:41.560 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-23T14:21:41.560 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-23T14:21:41.560 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-23T14:21:41.560 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-23T14:21:41.560 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-23T14:21:41.560 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-23T14:21:41.560 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-23T14:21:41.560 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-karma.ko 2024-04-23T14:21:41.560 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-23T14:21:41.560 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-23T14:21:41.560 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-23T14:21:41.561 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-23T14:21:41.561 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-23T14:21:41.561 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/usb-storage.ko 2024-04-23T14:21:41.561 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip 2024-04-23T14:21:41.561 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-23T14:21:41.561 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-23T14:21:41.561 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-23T14:21:41.561 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio 2024-04-23T14:21:41.561 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci 2024-04-23T14:21:41.561 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-23T14:21:41.561 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-23T14:21:41.561 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio.ko 2024-04-23T14:21:41.561 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-23T14:21:41.561 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost 2024-04-23T14:21:41.561 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost.ko 2024-04-23T14:21:41.562 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-23T14:21:41.562 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_net.ko 2024-04-23T14:21:41.562 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_scsi.ko 2024-04-23T14:21:41.562 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video 2024-04-23T14:21:41.562 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight 2024-04-23T14:21:41.562 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/apple_bl.ko 2024-04-23T14:21:41.562 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/lcd.ko 2024-04-23T14:21:41.562 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-23T14:21:41.562 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio 2024-04-23T14:21:41.562 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio.ko 2024-04-23T14:21:41.562 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_balloon.ko 2024-04-23T14:21:41.562 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_mmio.ko 2024-04-23T14:21:41.562 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci.ko 2024-04-23T14:21:41.562 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-23T14:21:41.562 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-23T14:21:41.563 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_ring.ko 2024-04-23T14:21:41.563 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1 2024-04-23T14:21:41.563 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters 2024-04-23T14:21:41.563 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2482.ko 2024-04-23T14:21:41.563 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2490.ko 2024-04-23T14:21:41.563 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves 2024-04-23T14:21:41.563 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-23T14:21:41.563 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-23T14:21:41.563 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-23T14:21:41.563 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-23T14:21:41.563 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-23T14:21:41.563 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-23T14:21:41.563 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-23T14:21:41.563 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-23T14:21:41.563 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-23T14:21:41.564 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-23T14:21:41.564 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-23T14:21:41.564 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/wire.ko 2024-04-23T14:21:41.564 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog 2024-04-23T14:21:41.564 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-23T14:21:41.564 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-23T14:21:41.564 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-23T14:21:41.564 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/hpwdt.ko 2024-04-23T14:21:41.564 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/i6300esb.ko 2024-04-23T14:21:41.564 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-23T14:21:41.564 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-23T14:21:41.565 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ib700wdt.ko 2024-04-23T14:21:41.565 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ibmasr.ko 2024-04-23T14:21:41.565 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-23T14:21:41.565 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-23T14:21:41.565 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it87_wdt.ko 2024-04-23T14:21:41.565 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/machzwd.ko 2024-04-23T14:21:41.565 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/nv_tco.ko 2024-04-23T14:21:41.565 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-23T14:21:41.565 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-23T14:21:41.565 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-23T14:21:41.565 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-23T14:21:41.565 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/softdog.ko 2024-04-23T14:21:41.565 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-23T14:21:41.565 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/via_wdt.ko 2024-04-23T14:21:41.565 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-23T14:21:41.566 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-23T14:21:41.566 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-23T14:21:41.566 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/wdt_pci.ko 2024-04-23T14:21:41.566 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/xen_wdt.ko 2024-04-23T14:21:41.566 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen 2024-04-23T14:21:41.566 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-23T14:21:41.566 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-evtchn.ko 2024-04-23T14:21:41.566 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntalloc.ko 2024-04-23T14:21:41.566 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntdev.ko 2024-04-23T14:21:41.566 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback 2024-04-23T14:21:41.566 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-23T14:21:41.566 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-privcmd.ko 2024-04-23T14:21:41.566 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs 2024-04-23T14:21:41.566 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-23T14:21:41.566 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs 2024-04-23T14:21:41.567 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p 2024-04-23T14:21:41.567 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p/9p.ko 2024-04-23T14:21:41.567 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs 2024-04-23T14:21:41.567 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs/affs.ko 2024-04-23T14:21:41.567 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs 2024-04-23T14:21:41.567 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs/befs.ko 2024-04-23T14:21:41.567 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/binfmt_misc.ko 2024-04-23T14:21:41.567 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs 2024-04-23T14:21:41.567 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs/btrfs.ko 2024-04-23T14:21:41.567 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles 2024-04-23T14:21:41.567 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles/cachefiles.ko 2024-04-23T14:21:41.567 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph 2024-04-23T14:21:41.567 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph/ceph.ko 2024-04-23T14:21:41.567 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda 2024-04-23T14:21:41.567 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda/coda.ko 2024-04-23T14:21:41.568 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs 2024-04-23T14:21:41.568 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs/cramfs.ko 2024-04-23T14:21:41.568 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm 2024-04-23T14:21:41.568 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm/dlm.ko 2024-04-23T14:21:41.568 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs 2024-04-23T14:21:41.568 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-23T14:21:41.568 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs 2024-04-23T14:21:41.568 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs/f2fs.ko 2024-04-23T14:21:41.568 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat 2024-04-23T14:21:41.568 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/fat.ko 2024-04-23T14:21:41.568 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/msdos.ko 2024-04-23T14:21:41.568 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/vfat.ko 2024-04-23T14:21:41.568 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse 2024-04-23T14:21:41.568 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/cuse.ko 2024-04-23T14:21:41.568 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/fuse.ko 2024-04-23T14:21:41.569 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2 2024-04-23T14:21:41.569 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2/gfs2.ko 2024-04-23T14:21:41.569 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs 2024-04-23T14:21:41.569 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs/hfs.ko 2024-04-23T14:21:41.569 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus 2024-04-23T14:21:41.569 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus/hfsplus.ko 2024-04-23T14:21:41.569 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs 2024-04-23T14:21:41.569 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs/isofs.ko 2024-04-23T14:21:41.569 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs 2024-04-23T14:21:41.569 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs/jfs.ko 2024-04-23T14:21:41.569 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd 2024-04-23T14:21:41.569 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd/lockd.ko 2024-04-23T14:21:41.569 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix 2024-04-23T14:21:41.569 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix/minix.ko 2024-04-23T14:21:41.569 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs 2024-04-23T14:21:41.570 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs/netfs.ko 2024-04-23T14:21:41.570 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs 2024-04-23T14:21:41.570 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout 2024-04-23T14:21:41.570 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-23T14:21:41.570 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout 2024-04-23T14:21:41.570 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-23T14:21:41.570 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout 2024-04-23T14:21:41.570 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-23T14:21:41.570 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfs.ko 2024-04-23T14:21:41.570 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv3.ko 2024-04-23T14:21:41.570 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv4.ko 2024-04-23T14:21:41.570 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common 2024-04-23T14:21:41.570 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/grace.ko 2024-04-23T14:21:41.570 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/nfs_acl.ko 2024-04-23T14:21:41.571 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd 2024-04-23T14:21:41.571 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd/nfsd.ko 2024-04-23T14:21:41.571 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2 2024-04-23T14:21:41.571 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2/nilfs2.ko 2024-04-23T14:21:41.571 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls 2024-04-23T14:21:41.571 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-celtic.ko 2024-04-23T14:21:41.571 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-centeuro.ko 2024-04-23T14:21:41.571 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-croatian.ko 2024-04-23T14:21:41.571 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-cyrillic.ko 2024-04-23T14:21:41.571 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-gaelic.ko 2024-04-23T14:21:41.571 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-greek.ko 2024-04-23T14:21:41.571 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-iceland.ko 2024-04-23T14:21:41.571 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-inuit.ko 2024-04-23T14:21:41.571 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-roman.ko 2024-04-23T14:21:41.571 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-romanian.ko 2024-04-23T14:21:41.572 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-turkish.ko 2024-04-23T14:21:41.572 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1250.ko 2024-04-23T14:21:41.572 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1251.ko 2024-04-23T14:21:41.572 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1255.ko 2024-04-23T14:21:41.572 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp737.ko 2024-04-23T14:21:41.572 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp775.ko 2024-04-23T14:21:41.572 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp850.ko 2024-04-23T14:21:41.572 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp852.ko 2024-04-23T14:21:41.572 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp855.ko 2024-04-23T14:21:41.572 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp857.ko 2024-04-23T14:21:41.572 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp860.ko 2024-04-23T14:21:41.572 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp861.ko 2024-04-23T14:21:41.572 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp862.ko 2024-04-23T14:21:41.572 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp863.ko 2024-04-23T14:21:41.572 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp864.ko 2024-04-23T14:21:41.572 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp865.ko 2024-04-23T14:21:41.573 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp866.ko 2024-04-23T14:21:41.573 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp869.ko 2024-04-23T14:21:41.573 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp874.ko 2024-04-23T14:21:41.573 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp932.ko 2024-04-23T14:21:41.573 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp936.ko 2024-04-23T14:21:41.573 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp949.ko 2024-04-23T14:21:41.573 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp950.ko 2024-04-23T14:21:41.573 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_euc-jp.ko 2024-04-23T14:21:41.573 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-1.ko 2024-04-23T14:21:41.573 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-13.ko 2024-04-23T14:21:41.573 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-14.ko 2024-04-23T14:21:41.573 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-15.ko 2024-04-23T14:21:41.573 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-2.ko 2024-04-23T14:21:41.573 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-3.ko 2024-04-23T14:21:41.574 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-4.ko 2024-04-23T14:21:41.574 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-5.ko 2024-04-23T14:21:41.574 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-6.ko 2024-04-23T14:21:41.574 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-7.ko 2024-04-23T14:21:41.574 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-9.ko 2024-04-23T14:21:41.574 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-r.ko 2024-04-23T14:21:41.574 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-ru.ko 2024-04-23T14:21:41.574 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-u.ko 2024-04-23T14:21:41.574 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-23T14:21:41.574 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_utf8.ko 2024-04-23T14:21:41.574 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2 2024-04-23T14:21:41.574 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster 2024-04-23T14:21:41.574 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-23T14:21:41.574 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm 2024-04-23T14:21:41.574 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-23T14:21:41.575 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs 2024-04-23T14:21:41.575 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-23T14:21:41.575 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2.ko 2024-04-23T14:21:41.575 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-23T14:21:41.575 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-23T14:21:41.575 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-23T14:21:41.575 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs 2024-04-23T14:21:41.575 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs/overlay.ko 2024-04-23T14:21:41.575 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore 2024-04-23T14:21:41.575 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore/ramoops.ko 2024-04-23T14:21:41.575 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs 2024-04-23T14:21:41.575 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs/reiserfs.ko 2024-04-23T14:21:41.575 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs 2024-04-23T14:21:41.575 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs/romfs.ko 2024-04-23T14:21:41.575 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb 2024-04-23T14:21:41.576 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client 2024-04-23T14:21:41.576 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client/cifs.ko 2024-04-23T14:21:41.576 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common 2024-04-23T14:21:41.576 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_arc4.ko 2024-04-23T14:21:41.576 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_md4.ko 2024-04-23T14:21:41.576 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs 2024-04-23T14:21:41.576 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs/squashfs.ko 2024-04-23T14:21:41.576 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv 2024-04-23T14:21:41.576 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv/sysv.ko 2024-04-23T14:21:41.576 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf 2024-04-23T14:21:41.576 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf/udf.ko 2024-04-23T14:21:41.576 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs 2024-04-23T14:21:41.576 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs/ufs.ko 2024-04-23T14:21:41.576 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs 2024-04-23T14:21:41.576 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs/xfs.ko 2024-04-23T14:21:41.577 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel 2024-04-23T14:21:41.577 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu 2024-04-23T14:21:41.577 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu/rcutorture.ko 2024-04-23T14:21:41.577 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/torture.ko 2024-04-23T14:21:41.577 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace 2024-04-23T14:21:41.577 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-23T14:21:41.577 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib 2024-04-23T14:21:41.577 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/asn1_encoder.ko 2024-04-23T14:21:41.577 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc-itu-t.ko 2024-04-23T14:21:41.577 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc8.ko 2024-04-23T14:21:41.577 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto 2024-04-23T14:21:41.577 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libarc4.ko 2024-04-23T14:21:41.577 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libdes.ko 2024-04-23T14:21:41.577 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/libcrc32c.ko 2024-04-23T14:21:41.577 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lru_cache.ko 2024-04-23T14:21:41.578 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4 2024-04-23T14:21:41.578 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4_compress.ko 2024-04-23T14:21:41.578 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4hc_compress.ko 2024-04-23T14:21:41.578 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math 2024-04-23T14:21:41.578 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math/cordic.ko 2024-04-23T14:21:41.578 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6 2024-04-23T14:21:41.578 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6/raid6_pq.ko 2024-04-23T14:21:41.578 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon 2024-04-23T14:21:41.578 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-23T14:21:41.578 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_bm.ko 2024-04-23T14:21:41.578 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_fsm.ko 2024-04-23T14:21:41.578 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_kmp.ko 2024-04-23T14:21:41.578 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd 2024-04-23T14:21:41.578 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd/zstd_compress.ko 2024-04-23T14:21:41.579 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm 2024-04-23T14:21:41.579 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm/hwpoison-inject.ko 2024-04-23T14:21:41.579 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net 2024-04-23T14:21:41.579 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan 2024-04-23T14:21:41.579 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/6lowpan.ko 2024-04-23T14:21:41.579 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_dest.ko 2024-04-23T14:21:41.579 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_fragment.ko 2024-04-23T14:21:41.579 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_hop.ko 2024-04-23T14:21:41.579 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-23T14:21:41.579 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_mobility.ko 2024-04-23T14:21:41.579 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_routing.ko 2024-04-23T14:21:41.579 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_udp.ko 2024-04-23T14:21:41.579 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802 2024-04-23T14:21:41.579 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/garp.ko 2024-04-23T14:21:41.579 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/mrp.ko 2024-04-23T14:21:41.580 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/p8022.ko 2024-04-23T14:21:41.580 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/psnap.ko 2024-04-23T14:21:41.580 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/stp.ko 2024-04-23T14:21:41.580 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q 2024-04-23T14:21:41.580 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q/8021q.ko 2024-04-23T14:21:41.580 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p 2024-04-23T14:21:41.580 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet.ko 2024-04-23T14:21:41.580 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_fd.ko 2024-04-23T14:21:41.580 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_rdma.ko 2024-04-23T14:21:41.580 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_virtio.ko 2024-04-23T14:21:41.580 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv 2024-04-23T14:21:41.580 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv/batman-adv.ko 2024-04-23T14:21:41.580 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge 2024-04-23T14:21:41.580 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/br_netfilter.ko 2024-04-23T14:21:41.580 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/bridge.ko 2024-04-23T14:21:41.581 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter 2024-04-23T14:21:41.581 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-23T14:21:41.581 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-23T14:21:41.581 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-23T14:21:41.581 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-23T14:21:41.581 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-23T14:21:41.581 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-23T14:21:41.581 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-23T14:21:41.581 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-23T14:21:41.581 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-23T14:21:41.581 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-23T14:21:41.581 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-23T14:21:41.581 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-23T14:21:41.581 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-23T14:21:41.581 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-23T14:21:41.582 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-23T14:21:41.582 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-23T14:21:41.582 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-23T14:21:41.582 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-23T14:21:41.582 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-23T14:21:41.582 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-23T14:21:41.582 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtables.ko 2024-04-23T14:21:41.582 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-23T14:21:41.582 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-23T14:21:41.582 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-23T14:21:41.582 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph 2024-04-23T14:21:41.582 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph/libceph.ko 2024-04-23T14:21:41.582 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core 2024-04-23T14:21:41.582 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/failover.ko 2024-04-23T14:21:41.582 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/pktgen.ko 2024-04-23T14:21:41.583 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp 2024-04-23T14:21:41.583 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp.ko 2024-04-23T14:21:41.583 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_diag.ko 2024-04-23T14:21:41.583 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv4.ko 2024-04-23T14:21:41.583 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv6.ko 2024-04-23T14:21:41.583 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver 2024-04-23T14:21:41.583 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver/dns_resolver.ko 2024-04-23T14:21:41.583 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa 2024-04-23T14:21:41.583 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/dsa_core.ko 2024-04-23T14:21:41.583 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_brcm.ko 2024-04-23T14:21:41.583 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_dsa.ko 2024-04-23T14:21:41.583 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_none.ko 2024-04-23T14:21:41.583 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_trailer.ko 2024-04-23T14:21:41.583 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154 2024-04-23T14:21:41.583 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan 2024-04-23T14:21:41.584 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-23T14:21:41.584 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154.ko 2024-04-23T14:21:41.584 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-23T14:21:41.584 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4 2024-04-23T14:21:41.584 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ah4.ko 2024-04-23T14:21:41.584 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/esp4.ko 2024-04-23T14:21:41.584 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/fou.ko 2024-04-23T14:21:41.584 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/gre.ko 2024-04-23T14:21:41.584 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/inet_diag.ko 2024-04-23T14:21:41.584 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_gre.ko 2024-04-23T14:21:41.584 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_tunnel.ko 2024-04-23T14:21:41.584 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_vti.ko 2024-04-23T14:21:41.584 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipcomp.ko 2024-04-23T14:21:41.584 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipip.ko 2024-04-23T14:21:41.584 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter 2024-04-23T14:21:41.585 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-23T14:21:41.585 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-23T14:21:41.585 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-23T14:21:41.585 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-23T14:21:41.585 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-23T14:21:41.585 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-23T14:21:41.585 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-23T14:21:41.585 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-23T14:21:41.585 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-23T14:21:41.585 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-23T14:21:41.585 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-23T14:21:41.585 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-23T14:21:41.585 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-23T14:21:41.585 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-23T14:21:41.585 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-23T14:21:41.586 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-23T14:21:41.586 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-23T14:21:41.586 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-23T14:21:41.586 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-23T14:21:41.586 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-23T14:21:41.586 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-23T14:21:41.586 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_bic.ko 2024-04-23T14:21:41.586 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_dctcp.ko 2024-04-23T14:21:41.586 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_diag.ko 2024-04-23T14:21:41.586 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_highspeed.ko 2024-04-23T14:21:41.586 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_htcp.ko 2024-04-23T14:21:41.586 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_hybla.ko 2024-04-23T14:21:41.586 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_illinois.ko 2024-04-23T14:21:41.586 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_lp.ko 2024-04-23T14:21:41.587 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_scalable.ko 2024-04-23T14:21:41.587 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_vegas.ko 2024-04-23T14:21:41.587 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_veno.ko 2024-04-23T14:21:41.587 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_westwood.ko 2024-04-23T14:21:41.587 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_yeah.ko 2024-04-23T14:21:41.587 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tunnel4.ko 2024-04-23T14:21:41.587 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_diag.ko 2024-04-23T14:21:41.587 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_tunnel.ko 2024-04-23T14:21:41.587 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-23T14:21:41.587 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6 2024-04-23T14:21:41.587 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ah6.ko 2024-04-23T14:21:41.587 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/esp6.ko 2024-04-23T14:21:41.587 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/fou6.ko 2024-04-23T14:21:41.587 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_tunnel.ko 2024-04-23T14:21:41.587 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-23T14:21:41.588 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_vti.ko 2024-04-23T14:21:41.588 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ipcomp6.ko 2024-04-23T14:21:41.588 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter 2024-04-23T14:21:41.588 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-23T14:21:41.588 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-23T14:21:41.588 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-23T14:21:41.588 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-23T14:21:41.588 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-23T14:21:41.588 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-23T14:21:41.588 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-23T14:21:41.588 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-23T14:21:41.588 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-23T14:21:41.588 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-23T14:21:41.588 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-23T14:21:41.588 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-23T14:21:41.589 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-23T14:21:41.589 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-23T14:21:41.589 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-23T14:21:41.589 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-23T14:21:41.589 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-23T14:21:41.589 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-23T14:21:41.589 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-23T14:21:41.589 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-23T14:21:41.589 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-23T14:21:41.589 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-23T14:21:41.589 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-23T14:21:41.589 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-23T14:21:41.589 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/sit.ko 2024-04-23T14:21:41.589 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/tunnel6.ko 2024-04-23T14:21:41.589 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-23T14:21:41.590 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key 2024-04-23T14:21:41.590 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key/af_key.ko 2024-04-23T14:21:41.590 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp 2024-04-23T14:21:41.590 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_core.ko 2024-04-23T14:21:41.590 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-23T14:21:41.590 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_eth.ko 2024-04-23T14:21:41.590 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip.ko 2024-04-23T14:21:41.590 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip6.ko 2024-04-23T14:21:41.590 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_netlink.ko 2024-04-23T14:21:41.590 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ppp.ko 2024-04-23T14:21:41.590 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc 2024-04-23T14:21:41.590 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc/llc.ko 2024-04-23T14:21:41.590 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154 2024-04-23T14:21:41.590 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154/mac802154.ko 2024-04-23T14:21:41.591 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls 2024-04-23T14:21:41.591 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls/mpls_gso.ko 2024-04-23T14:21:41.591 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter 2024-04-23T14:21:41.591 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset 2024-04-23T14:21:41.591 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set.ko 2024-04-23T14:21:41.591 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-23T14:21:41.591 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-23T14:21:41.591 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-23T14:21:41.591 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-23T14:21:41.591 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-23T14:21:41.591 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-23T14:21:41.591 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-23T14:21:41.591 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-23T14:21:41.591 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-23T14:21:41.591 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-23T14:21:41.592 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-23T14:21:41.592 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-23T14:21:41.592 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-23T14:21:41.592 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-23T14:21:41.592 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-23T14:21:41.592 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs 2024-04-23T14:21:41.592 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-23T14:21:41.592 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-23T14:21:41.592 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-23T14:21:41.592 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-23T14:21:41.592 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-23T14:21:41.592 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-23T14:21:41.592 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-23T14:21:41.592 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-23T14:21:41.592 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-23T14:21:41.592 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-23T14:21:41.593 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-23T14:21:41.593 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-23T14:21:41.593 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-23T14:21:41.593 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-23T14:21:41.593 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conncount.ko 2024-04-23T14:21:41.593 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack.ko 2024-04-23T14:21:41.593 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-23T14:21:41.593 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-23T14:21:41.593 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-23T14:21:41.593 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-23T14:21:41.593 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-23T14:21:41.593 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-23T14:21:41.593 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-23T14:21:41.593 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-23T14:21:41.594 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-23T14:21:41.594 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-23T14:21:41.594 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-23T14:21:41.594 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-23T14:21:41.594 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-23T14:21:41.594 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table.ko 2024-04-23T14:21:41.594 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-23T14:21:41.594 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_log_syslog.ko 2024-04-23T14:21:41.594 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat.ko 2024-04-23T14:21:41.594 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-23T14:21:41.594 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-23T14:21:41.594 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_irc.ko 2024-04-23T14:21:41.594 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_sip.ko 2024-04-23T14:21:41.594 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-23T14:21:41.595 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-23T14:21:41.595 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_tables.ko 2024-04-23T14:21:41.595 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink.ko 2024-04-23T14:21:41.595 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-23T14:21:41.595 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_log.ko 2024-04-23T14:21:41.595 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-23T14:21:41.595 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-23T14:21:41.595 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_chain_nat.ko 2024-04-23T14:21:41.595 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_compat.ko 2024-04-23T14:21:41.595 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_ct.ko 2024-04-23T14:21:41.595 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-23T14:21:41.595 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fib.ko 2024-04-23T14:21:41.595 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-23T14:21:41.595 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_hash.ko 2024-04-23T14:21:41.595 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_limit.ko 2024-04-23T14:21:41.596 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_log.ko 2024-04-23T14:21:41.596 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_masq.ko 2024-04-23T14:21:41.596 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_nat.ko 2024-04-23T14:21:41.596 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_numgen.ko 2024-04-23T14:21:41.596 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_queue.ko 2024-04-23T14:21:41.596 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_quota.ko 2024-04-23T14:21:41.596 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_redir.ko 2024-04-23T14:21:41.596 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject.ko 2024-04-23T14:21:41.596 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject_inet.ko 2024-04-23T14:21:41.596 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_socket.ko 2024-04-23T14:21:41.596 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_tunnel.ko 2024-04-23T14:21:41.596 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_AUDIT.ko 2024-04-23T14:21:41.596 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-23T14:21:41.596 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-23T14:21:41.596 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-23T14:21:41.597 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CT.ko 2024-04-23T14:21:41.597 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_DSCP.ko 2024-04-23T14:21:41.597 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HL.ko 2024-04-23T14:21:41.597 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HMARK.ko 2024-04-23T14:21:41.597 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-23T14:21:41.597 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LED.ko 2024-04-23T14:21:41.597 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LOG.ko 2024-04-23T14:21:41.597 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-23T14:21:41.597 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NETMAP.ko 2024-04-23T14:21:41.597 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFLOG.ko 2024-04-23T14:21:41.597 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-23T14:21:41.597 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_RATEEST.ko 2024-04-23T14:21:41.597 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-23T14:21:41.597 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_SECMARK.ko 2024-04-23T14:21:41.597 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-23T14:21:41.598 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-23T14:21:41.598 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TEE.ko 2024-04-23T14:21:41.598 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TPROXY.ko 2024-04-23T14:21:41.598 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TRACE.ko 2024-04-23T14:21:41.598 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_addrtype.ko 2024-04-23T14:21:41.598 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_bpf.ko 2024-04-23T14:21:41.598 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cgroup.ko 2024-04-23T14:21:41.598 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cluster.ko 2024-04-23T14:21:41.598 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_comment.ko 2024-04-23T14:21:41.598 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connbytes.ko 2024-04-23T14:21:41.598 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlabel.ko 2024-04-23T14:21:41.598 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlimit.ko 2024-04-23T14:21:41.598 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connmark.ko 2024-04-23T14:21:41.598 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_conntrack.ko 2024-04-23T14:21:41.598 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cpu.ko 2024-04-23T14:21:41.599 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dccp.ko 2024-04-23T14:21:41.599 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_devgroup.ko 2024-04-23T14:21:41.599 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dscp.ko 2024-04-23T14:21:41.599 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ecn.ko 2024-04-23T14:21:41.599 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_esp.ko 2024-04-23T14:21:41.599 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hashlimit.ko 2024-04-23T14:21:41.599 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_helper.ko 2024-04-23T14:21:41.599 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hl.ko 2024-04-23T14:21:41.599 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipcomp.ko 2024-04-23T14:21:41.599 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_iprange.ko 2024-04-23T14:21:41.599 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipvs.ko 2024-04-23T14:21:41.599 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_l2tp.ko 2024-04-23T14:21:41.599 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_length.ko 2024-04-23T14:21:41.599 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_limit.ko 2024-04-23T14:21:41.599 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mac.ko 2024-04-23T14:21:41.600 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mark.ko 2024-04-23T14:21:41.600 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_multiport.ko 2024-04-23T14:21:41.600 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nat.ko 2024-04-23T14:21:41.600 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nfacct.ko 2024-04-23T14:21:41.600 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_osf.ko 2024-04-23T14:21:41.600 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_owner.ko 2024-04-23T14:21:41.600 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_physdev.ko 2024-04-23T14:21:41.600 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_pkttype.ko 2024-04-23T14:21:41.600 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_policy.ko 2024-04-23T14:21:41.600 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_quota.ko 2024-04-23T14:21:41.600 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_rateest.ko 2024-04-23T14:21:41.600 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_realm.ko 2024-04-23T14:21:41.600 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_recent.ko 2024-04-23T14:21:41.600 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_sctp.ko 2024-04-23T14:21:41.600 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_set.ko 2024-04-23T14:21:41.601 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_socket.ko 2024-04-23T14:21:41.601 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_state.ko 2024-04-23T14:21:41.601 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_statistic.ko 2024-04-23T14:21:41.601 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_string.ko 2024-04-23T14:21:41.601 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_tcpmss.ko 2024-04-23T14:21:41.601 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_time.ko 2024-04-23T14:21:41.601 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_u32.ko 2024-04-23T14:21:41.601 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink 2024-04-23T14:21:41.601 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink/netlink_diag.ko 2024-04-23T14:21:41.601 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh 2024-04-23T14:21:41.601 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh/nsh.ko 2024-04-23T14:21:41.601 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch 2024-04-23T14:21:41.601 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/openvswitch.ko 2024-04-23T14:21:41.601 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-geneve.ko 2024-04-23T14:21:41.601 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-gre.ko 2024-04-23T14:21:41.601 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-vxlan.ko 2024-04-23T14:21:41.602 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet 2024-04-23T14:21:41.602 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet/af_packet_diag.ko 2024-04-23T14:21:41.602 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds 2024-04-23T14:21:41.602 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds.ko 2024-04-23T14:21:41.602 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_rdma.ko 2024-04-23T14:21:41.602 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_tcp.ko 2024-04-23T14:21:41.602 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched 2024-04-23T14:21:41.602 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_csum.ko 2024-04-23T14:21:41.602 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_gact.ko 2024-04-23T14:21:41.602 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_mirred.ko 2024-04-23T14:21:41.602 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_nat.ko 2024-04-23T14:21:41.602 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_pedit.ko 2024-04-23T14:21:41.602 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_police.ko 2024-04-23T14:21:41.602 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_simple.ko 2024-04-23T14:21:41.603 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_skbedit.ko 2024-04-23T14:21:41.603 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_vlan.ko 2024-04-23T14:21:41.603 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_basic.ko 2024-04-23T14:21:41.603 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_bpf.ko 2024-04-23T14:21:41.603 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_flow.ko 2024-04-23T14:21:41.603 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_fw.ko 2024-04-23T14:21:41.603 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_route.ko 2024-04-23T14:21:41.603 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_u32.ko 2024-04-23T14:21:41.603 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_cmp.ko 2024-04-23T14:21:41.603 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_ipset.ko 2024-04-23T14:21:41.603 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_meta.ko 2024-04-23T14:21:41.603 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_nbyte.ko 2024-04-23T14:21:41.603 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_text.ko 2024-04-23T14:21:41.603 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_u32.ko 2024-04-23T14:21:41.603 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_choke.ko 2024-04-23T14:21:41.604 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_codel.ko 2024-04-23T14:21:41.604 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_drr.ko 2024-04-23T14:21:41.604 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_fq.ko 2024-04-23T14:21:41.604 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_gred.ko 2024-04-23T14:21:41.604 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hfsc.ko 2024-04-23T14:21:41.604 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hhf.ko 2024-04-23T14:21:41.604 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_htb.ko 2024-04-23T14:21:41.604 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_ingress.ko 2024-04-23T14:21:41.604 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio.ko 2024-04-23T14:21:41.604 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio_lib.ko 2024-04-23T14:21:41.604 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_multiq.ko 2024-04-23T14:21:41.604 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_netem.ko 2024-04-23T14:21:41.604 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_pie.ko 2024-04-23T14:21:41.604 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_plug.ko 2024-04-23T14:21:41.604 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_prio.ko 2024-04-23T14:21:41.605 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_qfq.ko 2024-04-23T14:21:41.605 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_red.ko 2024-04-23T14:21:41.605 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfb.ko 2024-04-23T14:21:41.605 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfq.ko 2024-04-23T14:21:41.605 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_tbf.ko 2024-04-23T14:21:41.605 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_teql.ko 2024-04-23T14:21:41.605 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp 2024-04-23T14:21:41.605 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp.ko 2024-04-23T14:21:41.605 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp_diag.ko 2024-04-23T14:21:41.605 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc 2024-04-23T14:21:41.605 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss 2024-04-23T14:21:41.605 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-23T14:21:41.605 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-23T14:21:41.605 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/sunrpc.ko 2024-04-23T14:21:41.605 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma 2024-04-23T14:21:41.606 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-23T14:21:41.606 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc 2024-04-23T14:21:41.606 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc.ko 2024-04-23T14:21:41.606 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc_diag.ko 2024-04-23T14:21:41.606 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix 2024-04-23T14:21:41.606 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix/unix_diag.ko 2024-04-23T14:21:41.606 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock 2024-04-23T14:21:41.606 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-23T14:21:41.606 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-23T14:21:41.606 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock.ko 2024-04-23T14:21:41.606 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-23T14:21:41.606 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-23T14:21:41.606 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm 2024-04-23T14:21:41.606 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-23T14:21:41.606 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security 2024-04-23T14:21:41.607 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys 2024-04-23T14:21:41.607 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys 2024-04-23T14:21:41.607 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-23T14:21:41.607 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys 2024-04-23T14:21:41.607 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys/trusted.ko 2024-04-23T14:21:41.607 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias 2024-04-23T14:21:41.607 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias.bin 2024-04-23T14:21:41.607 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin 2024-04-23T14:21:41.607 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.alias.bin 2024-04-23T14:21:41.607 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.bin 2024-04-23T14:21:41.607 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.modinfo 2024-04-23T14:21:41.607 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep 2024-04-23T14:21:41.607 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep.bin 2024-04-23T14:21:41.607 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.devname 2024-04-23T14:21:41.607 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.order 2024-04-23T14:21:41.608 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.softdep 2024-04-23T14:21:41.608 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols 2024-04-23T14:21:41.608 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols.bin 2024-04-23T14:21:41.608 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-23T14:21:41.612 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g806ec36b2112 2024-04-23T14:21:41.613 DEBUG:teuthology.orchestra.run.smithi028:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-23T14:21:41.637 INFO:teuthology.orchestra.run.smithi028.stdout:postinstall scriptlet (using /bin/sh): 2024-04-23T14:21:41.637 INFO:teuthology.orchestra.run.smithi028.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-23T14:21:41.637 INFO:teuthology.orchestra.run.smithi028.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g806ec36b2112 /lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-23T14:21:41.637 INFO:teuthology.orchestra.run.smithi028.stdout:fi 2024-04-23T14:21:41.637 INFO:teuthology.orchestra.run.smithi028.stdout:for file in vmlinuz System.map config; do 2024-04-23T14:21:41.637 INFO:teuthology.orchestra.run.smithi028.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112"; then 2024-04-23T14:21:41.637 INFO:teuthology.orchestra.run.smithi028.stdout: cp "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112" 2024-04-23T14:21:41.638 INFO:teuthology.orchestra.run.smithi028.stdout: fi 2024-04-23T14:21:41.638 INFO:teuthology.orchestra.run.smithi028.stdout:done 2024-04-23T14:21:41.638 INFO:teuthology.orchestra.run.smithi028.stdout:if [ -d "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" ] && \ 2024-04-23T14:21:41.638 INFO:teuthology.orchestra.run.smithi028.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" >/dev/null 2>&1; then 2024-04-23T14:21:41.638 INFO:teuthology.orchestra.run.smithi028.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-23T14:21:41.638 INFO:teuthology.orchestra.run.smithi028.stdout: cp -r "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-23T14:21:41.638 INFO:teuthology.orchestra.run.smithi028.stdout:fi 2024-04-23T14:21:41.638 INFO:teuthology.orchestra.run.smithi028.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep" ]; then 2024-04-23T14:21:41.638 INFO:teuthology.orchestra.run.smithi028.stdout: /usr/sbin/depmod 6.9.0-rc3-g806ec36b2112 2024-04-23T14:21:41.638 INFO:teuthology.orchestra.run.smithi028.stdout:fi 2024-04-23T14:21:41.638 INFO:teuthology.orchestra.run.smithi028.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-23T14:21:41.638 INFO:teuthology.orchestra.run.smithi028.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-23T14:21:41.638 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-install remove 6.9.0-rc3-g806ec36b2112 2024-04-23T14:21:41.638 INFO:teuthology.orchestra.run.smithi028.stdout:fi 2024-04-23T14:21:41.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo rpm -qi grub2-tools 2024-04-23T14:21:41.724 INFO:teuthology.orchestra.run.smithi028.stdout:Name : grub2-tools 2024-04-23T14:21:41.724 INFO:teuthology.orchestra.run.smithi028.stdout:Epoch : 1 2024-04-23T14:21:41.724 INFO:teuthology.orchestra.run.smithi028.stdout:Version : 2.06 2024-04-23T14:21:41.724 INFO:teuthology.orchestra.run.smithi028.stdout:Release : 68.el9 2024-04-23T14:21:41.724 INFO:teuthology.orchestra.run.smithi028.stdout:Architecture: x86_64 2024-04-23T14:21:41.724 INFO:teuthology.orchestra.run.smithi028.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-04-23T14:21:41.724 INFO:teuthology.orchestra.run.smithi028.stdout:Group : Unspecified 2024-04-23T14:21:41.724 INFO:teuthology.orchestra.run.smithi028.stdout:Size : 8264254 2024-04-23T14:21:41.725 INFO:teuthology.orchestra.run.smithi028.stdout:License : GPLv3+ 2024-04-23T14:21:41.725 INFO:teuthology.orchestra.run.smithi028.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-23T14:21:41.725 INFO:teuthology.orchestra.run.smithi028.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-23T14:21:41.725 INFO:teuthology.orchestra.run.smithi028.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-23T14:21:41.725 INFO:teuthology.orchestra.run.smithi028.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-23T14:21:41.725 INFO:teuthology.orchestra.run.smithi028.stdout:Packager : builder@centos.org 2024-04-23T14:21:41.725 INFO:teuthology.orchestra.run.smithi028.stdout:Vendor : CentOS 2024-04-23T14:21:41.725 INFO:teuthology.orchestra.run.smithi028.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-23T14:21:41.725 INFO:teuthology.orchestra.run.smithi028.stdout:Summary : Support tools for GRUB. 2024-04-23T14:21:41.725 INFO:teuthology.orchestra.run.smithi028.stdout:Description : 2024-04-23T14:21:41.725 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:21:41.725 INFO:teuthology.orchestra.run.smithi028.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-23T14:21:41.725 INFO:teuthology.orchestra.run.smithi028.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-23T14:21:41.725 INFO:teuthology.orchestra.run.smithi028.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-23T14:21:41.725 INFO:teuthology.orchestra.run.smithi028.stdout:hardware devices. 2024-04-23T14:21:41.725 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:21:41.726 INFO:teuthology.orchestra.run.smithi028.stdout:This subpackage provides tools for support of all platforms. 2024-04-23T14:21:41.726 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-23T14:21:41.727 INFO:teuthology.task.kernel:Updating grub on smithi028 to boot 6.9.0-rc3-g806ec36b2112 2024-04-23T14:21:41.727 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-23T14:21:42.286 INFO:teuthology.orchestra.run.smithi028.stderr:Generating grub configuration file ... 2024-04-23T14:21:43.762 INFO:teuthology.orchestra.run.smithi028.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-23T14:21:43.779 INFO:teuthology.orchestra.run.smithi028.stderr:done 2024-04-23T14:21:43.781 DEBUG:teuthology.orchestra.run.smithi028:> mktemp 2024-04-23T14:21:43.803 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.Orp3nDBFlI 2024-04-23T14:21:43.803 DEBUG:teuthology.orchestra.run.smithi028:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Orp3nDBFlI 2024-04-23T14:21:43.870 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0666 /tmp/tmp.Orp3nDBFlI 2024-04-23T14:21:44.001 DEBUG:teuthology.orchestra.remote:smithi028:/tmp/tmp.Orp3nDBFlI is 6KB 2024-04-23T14:21:44.050 DEBUG:teuthology.orchestra.run.smithi028:> rm -fr /tmp/tmp.Orp3nDBFlI 2024-04-23T14:21:44.065 DEBUG:teuthology.orchestra.run.smithi028:> sudo /bin/ls /boot/loader/entries || true 2024-04-23T14:21:44.127 INFO:teuthology.orchestra.run.smithi028.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-04-23T14:21:44.127 INFO:teuthology.orchestra.run.smithi028.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-04-23T14:21:44.127 INFO:teuthology.orchestra.run.smithi028.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-04-23T14:21:44.128 INFO:teuthology.orchestra.run.smithi028.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112.conf 2024-04-23T14:21:44.129 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112 2024-04-23T14:21:44.314 DEBUG:teuthology.orchestra.run.smithi028:> sudo shutdown -r now 2024-04-23T14:22:14.317 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-04-23T14:22:14.318 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-04-23T14:22:14.319 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T14:22:32.738 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.28 2024-04-23T14:22:41.739 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-04-23T14:22:41.740 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T14:23:00.126 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.28 2024-04-23T14:23:12.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-04-23T14:23:12.128 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T14:23:15.198 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.28 2024-04-23T14:23:30.199 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-04-23T14:23:30.199 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T14:23:30.563 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-04-23T14:23:31.105 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-04-23T14:23:31.105 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.9.0-rc3-g806ec36b2112"... 2024-04-23T14:23:31.105 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-04-23T14:23:31.145 INFO:teuthology.orchestra.run.smithi028.stdout:6.9.0-rc3-g806ec36b2112 2024-04-23T14:23:31.145 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g806ec36b2112 vs 6.9.0-rc3-g806ec36b2112 2024-04-23T14:23:31.145 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-23T14:23:31.145 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-23T14:23:32.146 DEBUG:teuthology.parallel:result is None 2024-04-23T14:23:32.147 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-23T14:23:32.155 INFO:teuthology.task.internal:Creating test directory... 2024-04-23T14:23:32.155 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-23T14:23:32.161 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-23T14:23:32.164 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-23T14:23:32.207 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-23T14:23:32.215 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-23T14:23:32.229 INFO:teuthology.task.internal:Creating archive directory... 2024-04-23T14:23:32.229 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-23T14:23:32.234 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-23T14:23:32.237 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-23T14:23:32.559 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-23T14:23:32.571 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-23T14:23:32.571 DEBUG:teuthology.orchestra.run.smithi038:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-23T14:23:32.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T14:23:32.588 DEBUG:teuthology.orchestra.run.smithi105:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-23T14:23:32.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T14:23:32.605 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-23T14:23:32.610 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-23T14:23:32.631 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-23T14:23:32.670 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T14:23:32.685 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T14:23:32.697 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T14:23:32.716 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T14:23:32.743 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T14:23:33.032 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T14:23:33.036 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-23T14:23:33.045 INFO:teuthology.task.internal:Configuring sudo... 2024-04-23T14:23:33.045 DEBUG:teuthology.orchestra.run.smithi028:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-23T14:23:33.049 DEBUG:teuthology.orchestra.run.smithi038:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-23T14:23:33.051 DEBUG:teuthology.orchestra.run.smithi105:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-23T14:23:33.159 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-23T14:23:33.171 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-23T14:23:33.171 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-23T14:23:33.177 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-23T14:23:33.180 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-23T14:23:33.251 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T14:23:33.353 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T14:23:33.498 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T14:23:33.607 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T14:23:33.730 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-23T14:23:33.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-23T14:23:33.850 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T14:23:33.896 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T14:23:33.952 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T14:23:34.034 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T14:23:34.082 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-23T14:23:34.082 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-23T14:23:34.150 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T14:23:34.194 DEBUG:teuthology.orchestra.run.smithi105:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T14:23:34.248 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T14:23:34.333 DEBUG:teuthology.orchestra.run.smithi105:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T14:23:34.373 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-23T14:23:34.373 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-23T14:23:34.438 DEBUG:teuthology.orchestra.run.smithi028:> sudo service rsyslog restart 2024-04-23T14:23:34.447 DEBUG:teuthology.orchestra.run.smithi038:> sudo service rsyslog restart 2024-04-23T14:23:34.449 DEBUG:teuthology.orchestra.run.smithi105:> sudo service rsyslog restart 2024-04-23T14:23:34.487 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T14:23:34.521 INFO:teuthology.orchestra.run.smithi105.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T14:23:34.590 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T14:23:34.978 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-23T14:23:34.986 INFO:teuthology.task.internal:Starting timer... 2024-04-23T14:23:34.986 INFO:teuthology.run_tasks:Running task pcp... 2024-04-23T14:23:35.027 INFO:teuthology.run_tasks:Running task selinux... 2024-04-23T14:23:35.039 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-23T14:23:35.040 DEBUG:teuthology.orchestra.run.smithi028:> sudo service auditd rotate 2024-04-23T14:23:35.129 INFO:teuthology.orchestra.run.smithi028.stdout:Rotating logs: 2024-04-23T14:23:35.134 DEBUG:teuthology.orchestra.run.smithi038:> sudo service auditd rotate 2024-04-23T14:23:35.185 INFO:teuthology.orchestra.run.smithi038.stdout:Rotating logs: 2024-04-23T14:23:35.187 DEBUG:teuthology.orchestra.run.smithi105:> sudo service auditd rotate 2024-04-23T14:23:35.252 INFO:teuthology.orchestra.run.smithi105.stdout:Rotating logs: 2024-04-23T14:23:35.254 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-23T14:23:35.254 DEBUG:teuthology.orchestra.run.smithi028:> /usr/sbin/getenforce 2024-04-23T14:23:35.312 INFO:teuthology.orchestra.run.smithi028.stdout:Permissive 2024-04-23T14:23:35.313 DEBUG:teuthology.orchestra.run.smithi038:> /usr/sbin/getenforce 2024-04-23T14:23:35.335 INFO:teuthology.orchestra.run.smithi038.stdout:Permissive 2024-04-23T14:23:35.335 DEBUG:teuthology.orchestra.run.smithi105:> /usr/sbin/getenforce 2024-04-23T14:23:35.355 INFO:teuthology.orchestra.run.smithi105.stdout:Permissive 2024-04-23T14:23:35.355 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi028.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi038.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi105.front.sepia.ceph.com': 'permissive'} 2024-04-23T14:23:35.355 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T14:23:35.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T14:23:35.430 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T14:23:35.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T14:23:35.458 DEBUG:teuthology.orchestra.run.smithi105:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T14:23:35.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T14:23:35.484 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-23T14:23:35.485 DEBUG:teuthology.orchestra.run.smithi028:> sudo /usr/sbin/setenforce permissive 2024-04-23T14:23:35.559 DEBUG:teuthology.orchestra.run.smithi038:> sudo /usr/sbin/setenforce permissive 2024-04-23T14:23:35.585 DEBUG:teuthology.orchestra.run.smithi105:> sudo /usr/sbin/setenforce permissive 2024-04-23T14:23:35.609 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-23T14:23:35.623 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-23T14:23:35.630 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-23T14:23:35.760 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-23T14:23:35.776 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-23T14:23:35.778 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi028.front.sepia.ceph.com,smithi038.front.sepia.ceph.com,smithi105.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-23T14:28:57.820 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi028.front.sepia.ceph.com'), Remote(name='ubuntu@smithi038.front.sepia.ceph.com'), Remote(name='ubuntu@smithi105.front.sepia.ceph.com')] 2024-04-23T14:28:57.822 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-04-23T14:28:57.823 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T14:28:57.940 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-04-23T14:28:58.103 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-04-23T14:28:58.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-04-23T14:28:58.104 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T14:28:58.184 DEBUG:teuthology.orchestra.run.smithi038:> true 2024-04-23T14:28:58.267 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-04-23T14:28:58.268 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-04-23T14:28:58.268 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T14:28:58.353 DEBUG:teuthology.orchestra.run.smithi105:> true 2024-04-23T14:28:58.439 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-04-23T14:28:58.439 INFO:teuthology.run_tasks:Running task clock... 2024-04-23T14:28:58.451 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-23T14:28:58.451 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-23T14:28:58.452 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T14:28:58.460 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-23T14:28:58.460 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T14:28:58.462 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-23T14:28:58.463 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T14:28:58.493 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-23T14:28:58.508 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-23T14:28:58.511 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-23T14:28:58.526 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-23T14:28:58.535 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: ntpd: command not found 2024-04-23T14:28:58.546 INFO:teuthology.orchestra.run.smithi038.stdout:506 Cannot talk to daemon 2024-04-23T14:28:58.551 INFO:teuthology.orchestra.run.smithi105.stderr:sudo: ntpd: command not found 2024-04-23T14:28:58.553 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-23T14:28:58.560 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-23T14:28:58.561 INFO:teuthology.orchestra.run.smithi105.stdout:506 Cannot talk to daemon 2024-04-23T14:28:58.575 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-23T14:28:58.575 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-23T14:28:58.588 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-23T14:28:58.596 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-23T14:28:58.631 INFO:teuthology.orchestra.run.smithi038.stderr:bash: line 1: ntpq: command not found 2024-04-23T14:28:58.635 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T14:28:58.635 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2024-04-23T14:28:58.635 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T14:28:58.635 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T14:28:58.635 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T14:28:58.635 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T14:28:58.645 INFO:teuthology.orchestra.run.smithi105.stderr:bash: line 1: ntpq: command not found 2024-04-23T14:28:58.649 INFO:teuthology.orchestra.run.smithi105.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T14:28:58.649 INFO:teuthology.orchestra.run.smithi105.stdout:=============================================================================== 2024-04-23T14:28:58.649 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T14:28:58.649 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T14:28:58.649 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T14:28:58.649 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T14:28:58.677 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: ntpd: command not found 2024-04-23T14:28:58.712 INFO:teuthology.orchestra.run.smithi028.stdout:506 Cannot talk to daemon 2024-04-23T14:28:58.754 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-23T14:28:58.802 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-23T14:28:58.943 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-04-23T14:28:58.953 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T14:28:58.954 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-04-23T14:28:58.954 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T14:28:58.954 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T14:28:58.954 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T14:28:58.954 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T14:28:58.955 INFO:teuthology.run_tasks:Running task install... 2024-04-23T14:28:58.964 DEBUG:teuthology.task.install:project ceph 2024-04-23T14:28:58.964 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd44fddd910b5e94da0f2357af4be37d7bd495ae0'}} 2024-04-23T14:28:58.964 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd44fddd910b5e94da0f2357af4be37d7bd495ae0'} 2024-04-23T14:28:58.964 INFO:teuthology.task.install:Using flavor: default 2024-04-23T14:28:58.971 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-23T14:28:58.971 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-23T14:28:58.971 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd44fddd910b5e94da0f2357af4be37d7bd495ae0', 'tag': None, 'wait_for_package': False} 2024-04-23T14:28:58.972 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d44fddd910b5e94da0f2357af4be37d7bd495ae0 2024-04-23T14:28:58.974 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd44fddd910b5e94da0f2357af4be37d7bd495ae0', 'tag': None, 'wait_for_package': False} 2024-04-23T14:28:58.974 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d44fddd910b5e94da0f2357af4be37d7bd495ae0 2024-04-23T14:28:58.975 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd44fddd910b5e94da0f2357af4be37d7bd495ae0', 'tag': None, 'wait_for_package': False} 2024-04-23T14:28:58.975 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d44fddd910b5e94da0f2357af4be37d7bd495ae0 2024-04-23T14:28:59.126 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240416.193735-3/d44fddd910b5e94da0f2357af4be37d7bd495ae0/centos/9/flavors/default/ 2024-04-23T14:28:59.126 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3126.gd44fddd9 2024-04-23T14:28:59.130 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240416.193735-3/d44fddd910b5e94da0f2357af4be37d7bd495ae0/centos/9/flavors/default/ 2024-04-23T14:28:59.130 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3126.gd44fddd9 2024-04-23T14:28:59.193 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240416.193735-3/d44fddd910b5e94da0f2357af4be37d7bd495ae0/centos/9/flavors/default/ 2024-04-23T14:28:59.194 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3126.gd44fddd9 2024-04-23T14:28:59.361 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240416.193735-3/d44fddd910b5e94da0f2357af4be37d7bd495ae0/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240416.193735-3/d44fddd910b5e94da0f2357af4be37d7bd495ae0/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240416.193735-3/d44fddd910b5e94da0f2357af4be37d7bd495ae0/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-23T14:28:59.361 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-23T14:28:59.361 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-23T14:28:59.363 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240416.193735-3/d44fddd910b5e94da0f2357af4be37d7bd495ae0/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240416.193735-3/d44fddd910b5e94da0f2357af4be37d7bd495ae0/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240416.193735-3/d44fddd910b5e94da0f2357af4be37d7bd495ae0/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-23T14:28:59.364 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-23T14:28:59.364 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-23T14:28:59.389 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-23T14:28:59.390 DEBUG:teuthology.orchestra.run.smithi105:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d44fddd910b5e94da0f2357af4be37d7bd495ae0/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-23T14:28:59.454 DEBUG:teuthology.orchestra.run.smithi105:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-23T14:28:59.461 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-23T14:28:59.461 DEBUG:teuthology.orchestra.run.smithi028:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d44fddd910b5e94da0f2357af4be37d7bd495ae0/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-23T14:28:59.507 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240416.193735-3/d44fddd910b5e94da0f2357af4be37d7bd495ae0/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240416.193735-3/d44fddd910b5e94da0f2357af4be37d7bd495ae0/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240416.193735-3/d44fddd910b5e94da0f2357af4be37d7bd495ae0/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-23T14:28:59.507 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-23T14:28:59.507 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-23T14:28:59.529 DEBUG:teuthology.orchestra.run.smithi105:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-23T14:28:59.539 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-23T14:28:59.539 DEBUG:teuthology.orchestra.run.smithi038:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d44fddd910b5e94da0f2357af4be37d7bd495ae0/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-23T14:28:59.592 INFO:teuthology.orchestra.run.smithi105.stdout:check_obsoletes = 1 2024-04-23T14:28:59.594 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum clean all 2024-04-23T14:28:59.596 DEBUG:teuthology.orchestra.run.smithi028:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-23T14:28:59.609 DEBUG:teuthology.orchestra.run.smithi038:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-23T14:28:59.688 DEBUG:teuthology.orchestra.run.smithi038:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-23T14:28:59.712 DEBUG:teuthology.orchestra.run.smithi028:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-23T14:28:59.755 INFO:teuthology.orchestra.run.smithi038.stdout:check_obsoletes = 1 2024-04-23T14:28:59.756 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum clean all 2024-04-23T14:28:59.790 INFO:teuthology.orchestra.run.smithi028.stdout:check_obsoletes = 1 2024-04-23T14:28:59.794 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum clean all 2024-04-23T14:28:59.817 INFO:teuthology.orchestra.run.smithi105.stdout:48 files removed 2024-04-23T14:28:59.844 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-23T14:28:59.996 INFO:teuthology.orchestra.run.smithi038.stdout:48 files removed 2024-04-23T14:29:00.023 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-23T14:29:00.129 INFO:teuthology.orchestra.run.smithi028.stdout:48 files removed 2024-04-23T14:29:00.169 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-23T14:29:00.614 INFO:teuthology.orchestra.run.smithi105.stdout:ceph packages for x86_64 163 kB/s | 83 kB 00:00 2024-04-23T14:29:01.008 INFO:teuthology.orchestra.run.smithi038.stdout:ceph packages for x86_64 117 kB/s | 83 kB 00:00 2024-04-23T14:29:01.056 INFO:teuthology.orchestra.run.smithi105.stdout:ceph noarch packages 29 kB/s | 12 kB 00:00 2024-04-23T14:29:01.376 INFO:teuthology.orchestra.run.smithi028.stdout:ceph packages for x86_64 110 kB/s | 83 kB 00:00 2024-04-23T14:29:01.616 INFO:teuthology.orchestra.run.smithi038.stdout:ceph noarch packages 21 kB/s | 12 kB 00:00 2024-04-23T14:29:01.699 INFO:teuthology.orchestra.run.smithi105.stdout:ceph source packages 3.1 kB/s | 1.9 kB 00:00 2024-04-23T14:29:02.167 INFO:teuthology.orchestra.run.smithi038.stdout:ceph source packages 3.7 kB/s | 1.9 kB 00:00 2024-04-23T14:29:02.201 INFO:teuthology.orchestra.run.smithi028.stdout:ceph noarch packages 16 kB/s | 12 kB 00:00 2024-04-23T14:29:02.701 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-04-23T14:29:02.918 INFO:teuthology.orchestra.run.smithi028.stdout:ceph source packages 3.0 kB/s | 1.9 kB 00:00 2024-04-23T14:29:03.967 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 8.4 MB/s | 8.0 MB 00:00 2024-04-23T14:29:07.718 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 7.6 MB/s | 19 MB 00:02 2024-04-23T14:29:11.623 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - BaseOS 833 kB/s | 8.0 MB 00:09 2024-04-23T14:29:12.352 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 9.8 MB/s | 6.2 MB 00:00 2024-04-23T14:29:12.735 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - AppStream 2.1 MB/s | 19 MB 00:08 2024-04-23T14:29:13.432 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-04-23T14:29:14.244 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 16 kB 00:00 2024-04-23T14:29:19.521 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 4.1 MB/s | 21 MB 00:05 2024-04-23T14:29:21.044 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - CRB 1.4 MB/s | 6.2 MB 00:04 2024-04-23T14:29:22.713 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-04-23T14:29:24.568 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-23T14:29:24.899 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - CRB 797 kB/s | 6.2 MB 00:07 2024-04-23T14:29:25.846 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-04-23T14:29:26.283 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - Extras packages 69 kB/s | 16 kB 00:00 2024-04-23T14:29:28.358 INFO:teuthology.orchestra.run.smithi028.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-23T14:29:28.360 INFO:teuthology.orchestra.run.smithi028.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-23T14:29:28.382 INFO:teuthology.orchestra.run.smithi028.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-23T14:29:28.384 INFO:teuthology.orchestra.run.smithi028.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-23T14:29:28.385 INFO:teuthology.orchestra.run.smithi028.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-23T14:29:28.386 INFO:teuthology.orchestra.run.smithi028.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-23T14:29:28.389 INFO:teuthology.orchestra.run.smithi028.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-23T14:29:28.393 INFO:teuthology.orchestra.run.smithi028.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-23T14:29:28.394 INFO:teuthology.orchestra.run.smithi028.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-23T14:29:28.472 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-23T14:29:28.487 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-04-23T14:29:28.487 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-04-23T14:29:28.487 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-04-23T14:29:28.487 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-04-23T14:29:28.487 INFO:teuthology.orchestra.run.smithi028.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-23T14:29:28.487 INFO:teuthology.orchestra.run.smithi028.stdout: ceph x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 6.6 k 2024-04-23T14:29:28.488 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 5.4 M 2024-04-23T14:29:28.488 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 891 k 2024-04-23T14:29:28.488 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 145 k 2024-04-23T14:29:28.488 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 1.5 M 2024-04-23T14:29:28.488 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 142 k 2024-04-23T14:29:28.488 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 3.5 M 2024-04-23T14:29:28.488 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 7.4 M 2024-04-23T14:29:28.488 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 49 k 2024-04-23T14:29:28.488 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 11 M 2024-04-23T14:29:28.488 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 49 M 2024-04-23T14:29:28.488 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 273 k 2024-04-23T14:29:28.488 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 762 k 2024-04-23T14:29:28.488 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-mirror x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 225 k 2024-04-23T14:29:28.488 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-shell x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 27 k 2024-04-23T14:29:28.489 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-top noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 18 k 2024-04-23T14:29:28.489 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-23T14:29:28.489 INFO:teuthology.orchestra.run.smithi028.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-23T14:29:28.489 INFO:teuthology.orchestra.run.smithi028.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-23T14:29:28.489 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 32 k 2024-04-23T14:29:28.489 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2 x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 746 k 2024-04-23T14:29:28.489 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 127 k 2024-04-23T14:29:28.489 INFO:teuthology.orchestra.run.smithi028.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-23T14:29:28.489 INFO:teuthology.orchestra.run.smithi028.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-23T14:29:28.489 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-23T14:29:28.489 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-23T14:29:28.489 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-23T14:29:28.489 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-23T14:29:28.490 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 161 k 2024-04-23T14:29:28.490 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 320 k 2024-04-23T14:29:28.490 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 297 k 2024-04-23T14:29:28.490 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 100 k 2024-04-23T14:29:28.490 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 85 k 2024-04-23T14:29:28.490 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 3.1 M 2024-04-23T14:29:28.490 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 171 k 2024-04-23T14:29:28.490 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-23T14:29:28.490 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrading: 2024-04-23T14:29:28.490 INFO:teuthology.orchestra.run.smithi028.stdout: librados2 x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 3.4 M 2024-04-23T14:29:28.490 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1 x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 3.1 M 2024-04-23T14:29:28.490 INFO:teuthology.orchestra.run.smithi028.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-23T14:29:28.490 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-23T14:29:28.491 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-04-23T14:29:28.491 INFO:teuthology.orchestra.run.smithi028.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-23T14:29:28.491 INFO:teuthology.orchestra.run.smithi028.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-23T14:29:28.491 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-23T14:29:28.491 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 21 M 2024-04-23T14:29:28.491 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 34 k 2024-04-23T14:29:28.491 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 2.4 M 2024-04-23T14:29:28.491 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 249 k 2024-04-23T14:29:28.491 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 4.7 M 2024-04-23T14:29:28.491 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 17 M 2024-04-23T14:29:28.491 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 16 k 2024-04-23T14:29:28.491 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 25 k 2024-04-23T14:29:28.491 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-23T14:29:28.491 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-23T14:29:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-23T14:29:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-23T14:29:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-23T14:29:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-23T14:29:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-23T14:29:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-23T14:29:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 163 k 2024-04-23T14:29:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-23T14:29:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-23T14:29:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-23T14:29:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-23T14:29:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-23T14:29:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1 x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 500 k 2024-04-23T14:29:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-23T14:29:28.493 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2 x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 5.4 M 2024-04-23T14:29:28.493 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-23T14:29:28.493 INFO:teuthology.orchestra.run.smithi028.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-23T14:29:28.493 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-23T14:29:28.493 INFO:teuthology.orchestra.run.smithi028.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-23T14:29:28.493 INFO:teuthology.orchestra.run.smithi028.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-23T14:29:28.493 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-23T14:29:28.493 INFO:teuthology.orchestra.run.smithi028.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-23T14:29:28.493 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-23T14:29:28.493 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-23T14:29:28.493 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-23T14:29:28.493 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-23T14:29:28.493 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-23T14:29:28.493 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-23T14:29:28.494 INFO:teuthology.orchestra.run.smithi028.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-23T14:29:28.494 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-23T14:29:28.494 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-23T14:29:28.494 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-23T14:29:28.494 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 45 k 2024-04-23T14:29:28.494 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 135 k 2024-04-23T14:29:28.494 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-23T14:29:28.494 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-23T14:29:28.494 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-23T14:29:28.494 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-23T14:29:28.494 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-23T14:29:28.494 INFO:teuthology.orchestra.run.smithi028.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-23T14:29:28.494 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-23T14:29:28.495 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-23T14:29:28.495 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-23T14:29:28.495 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-23T14:29:28.495 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-23T14:29:28.495 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-23T14:29:28.495 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-23T14:29:28.495 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-23T14:29:28.495 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-23T14:29:28.495 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-23T14:29:28.495 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-23T14:29:28.495 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-23T14:29:28.495 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-23T14:29:28.495 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-23T14:29:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-23T14:29:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-23T14:29:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-23T14:29:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-23T14:29:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-23T14:29:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-23T14:29:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-23T14:29:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-23T14:29:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-23T14:29:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-23T14:29:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-23T14:29:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-23T14:29:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-23T14:29:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-23T14:29:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-23T14:29:28.497 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-23T14:29:28.497 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-23T14:29:28.497 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-23T14:29:28.497 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-23T14:29:28.497 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-23T14:29:28.497 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-23T14:29:28.497 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-23T14:29:28.497 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-23T14:29:28.497 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-23T14:29:28.497 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-23T14:29:28.497 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-23T14:29:28.497 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-23T14:29:28.497 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-23T14:29:28.497 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-23T14:29:28.498 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-23T14:29:28.498 INFO:teuthology.orchestra.run.smithi028.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-23T14:29:28.498 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-23T14:29:28.498 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-23T14:29:28.498 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-23T14:29:28.498 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-23T14:29:28.498 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-23T14:29:28.498 INFO:teuthology.orchestra.run.smithi028.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-23T14:29:28.498 INFO:teuthology.orchestra.run.smithi028.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-23T14:29:28.498 INFO:teuthology.orchestra.run.smithi028.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-23T14:29:28.498 INFO:teuthology.orchestra.run.smithi028.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-23T14:29:28.498 INFO:teuthology.orchestra.run.smithi028.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-23T14:29:28.498 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-23T14:29:28.499 INFO:teuthology.orchestra.run.smithi028.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-23T14:29:28.499 INFO:teuthology.orchestra.run.smithi028.stdout:Installing weak dependencies: 2024-04-23T14:29:28.499 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-23T14:29:28.499 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:29:28.499 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-04-23T14:29:28.499 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-04-23T14:29:28.499 INFO:teuthology.orchestra.run.smithi028.stdout:Install 148 Packages 2024-04-23T14:29:28.499 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrade 4 Packages 2024-04-23T14:29:28.499 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:29:28.500 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 222 M 2024-04-23T14:29:28.500 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-04-23T14:29:28.537 INFO:teuthology.orchestra.run.smithi105.stdout:Extra Packages for Enterprise Linux 9.8 MB/s | 21 MB 00:02 2024-04-23T14:29:29.564 INFO:teuthology.orchestra.run.smithi028.stdout:(1/152): ceph-19.0.0-3126.gd44fddd9.el9.x86_64. 24 kB/s | 6.6 kB 00:00 2024-04-23T14:29:29.740 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-04-23T14:29:29.772 INFO:teuthology.orchestra.run.smithi028.stdout:(2/152): ceph-fuse-19.0.0-3126.gd44fddd9.el9.x8 4.2 MB/s | 891 kB 00:00 2024-04-23T14:29:29.839 INFO:teuthology.orchestra.run.smithi028.stdout:(3/152): ceph-immutable-object-cache-19.0.0-312 2.2 MB/s | 145 kB 00:00 2024-04-23T14:29:30.115 INFO:teuthology.orchestra.run.smithi028.stdout:(4/152): ceph-mds-19.0.0-3126.gd44fddd9.el9.x86 8.6 MB/s | 2.4 MB 00:00 2024-04-23T14:29:30.373 INFO:teuthology.orchestra.run.smithi028.stdout:(5/152): ceph-mgr-19.0.0-3126.gd44fddd9.el9.x86 5.8 MB/s | 1.5 MB 00:00 2024-04-23T14:29:30.957 INFO:teuthology.orchestra.run.smithi028.stdout:(6/152): ceph-mon-19.0.0-3126.gd44fddd9.el9.x86 8.1 MB/s | 4.7 MB 00:00 2024-04-23T14:29:31.240 INFO:teuthology.orchestra.run.smithi028.stdout:(7/152): ceph-common-19.0.0-3126.gd44fddd9.el9. 11 MB/s | 21 MB 00:01 2024-04-23T14:29:31.365 INFO:teuthology.orchestra.run.smithi028.stdout:(8/152): ceph-base-19.0.0-3126.gd44fddd9.el9.x8 2.6 MB/s | 5.4 MB 00:02 2024-04-23T14:29:31.491 INFO:teuthology.orchestra.run.smithi028.stdout:(9/152): ceph-selinux-19.0.0-3126.gd44fddd9.el9 203 kB/s | 25 kB 00:00 2024-04-23T14:29:31.959 INFO:teuthology.orchestra.run.smithi038.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-23T14:29:31.960 INFO:teuthology.orchestra.run.smithi038.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-23T14:29:31.972 INFO:teuthology.orchestra.run.smithi038.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-23T14:29:31.973 INFO:teuthology.orchestra.run.smithi038.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-23T14:29:31.974 INFO:teuthology.orchestra.run.smithi038.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-23T14:29:31.974 INFO:teuthology.orchestra.run.smithi038.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-23T14:29:31.977 INFO:teuthology.orchestra.run.smithi038.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-23T14:29:31.978 INFO:teuthology.orchestra.run.smithi038.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-23T14:29:31.979 INFO:teuthology.orchestra.run.smithi038.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-23T14:29:32.043 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-23T14:29:32.049 INFO:teuthology.orchestra.run.smithi028.stdout:(10/152): ceph-radosgw-19.0.0-3126.gd44fddd9.el 13 MB/s | 11 MB 00:00 2024-04-23T14:29:32.065 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================= 2024-04-23T14:29:32.065 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-04-23T14:29:32.065 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================= 2024-04-23T14:29:32.065 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-04-23T14:29:32.066 INFO:teuthology.orchestra.run.smithi038.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-23T14:29:32.066 INFO:teuthology.orchestra.run.smithi038.stdout: ceph x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 6.6 k 2024-04-23T14:29:32.066 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 5.4 M 2024-04-23T14:29:32.066 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 891 k 2024-04-23T14:29:32.066 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 145 k 2024-04-23T14:29:32.066 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 1.5 M 2024-04-23T14:29:32.067 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 142 k 2024-04-23T14:29:32.067 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 3.5 M 2024-04-23T14:29:32.067 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 7.4 M 2024-04-23T14:29:32.067 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 49 k 2024-04-23T14:29:32.067 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 11 M 2024-04-23T14:29:32.067 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 49 M 2024-04-23T14:29:32.067 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-volume noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 273 k 2024-04-23T14:29:32.068 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 762 k 2024-04-23T14:29:32.068 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-mirror x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 225 k 2024-04-23T14:29:32.068 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-shell x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 27 k 2024-04-23T14:29:32.068 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-top noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 18 k 2024-04-23T14:29:32.068 INFO:teuthology.orchestra.run.smithi038.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-23T14:29:32.068 INFO:teuthology.orchestra.run.smithi038.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-23T14:29:32.068 INFO:teuthology.orchestra.run.smithi038.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-23T14:29:32.068 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 32 k 2024-04-23T14:29:32.069 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2 x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 746 k 2024-04-23T14:29:32.069 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 127 k 2024-04-23T14:29:32.069 INFO:teuthology.orchestra.run.smithi038.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-23T14:29:32.069 INFO:teuthology.orchestra.run.smithi038.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-23T14:29:32.069 INFO:teuthology.orchestra.run.smithi038.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-23T14:29:32.069 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-23T14:29:32.069 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-23T14:29:32.069 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-23T14:29:32.069 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 161 k 2024-04-23T14:29:32.070 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 320 k 2024-04-23T14:29:32.070 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 297 k 2024-04-23T14:29:32.070 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 100 k 2024-04-23T14:29:32.070 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 85 k 2024-04-23T14:29:32.070 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 3.1 M 2024-04-23T14:29:32.070 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 171 k 2024-04-23T14:29:32.070 INFO:teuthology.orchestra.run.smithi038.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-23T14:29:32.070 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrading: 2024-04-23T14:29:32.071 INFO:teuthology.orchestra.run.smithi038.stdout: librados2 x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 3.4 M 2024-04-23T14:29:32.071 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1 x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 3.1 M 2024-04-23T14:29:32.071 INFO:teuthology.orchestra.run.smithi038.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-23T14:29:32.071 INFO:teuthology.orchestra.run.smithi038.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-23T14:29:32.071 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-04-23T14:29:32.071 INFO:teuthology.orchestra.run.smithi038.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-23T14:29:32.071 INFO:teuthology.orchestra.run.smithi038.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-23T14:29:32.072 INFO:teuthology.orchestra.run.smithi038.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-23T14:29:32.072 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 21 M 2024-04-23T14:29:32.072 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 34 k 2024-04-23T14:29:32.072 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 2.4 M 2024-04-23T14:29:32.072 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 249 k 2024-04-23T14:29:32.072 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 4.7 M 2024-04-23T14:29:32.073 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 17 M 2024-04-23T14:29:32.073 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 16 k 2024-04-23T14:29:32.073 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 25 k 2024-04-23T14:29:32.073 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-23T14:29:32.073 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-23T14:29:32.073 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-23T14:29:32.074 INFO:teuthology.orchestra.run.smithi038.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-23T14:29:32.074 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-23T14:29:32.074 INFO:teuthology.orchestra.run.smithi038.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-23T14:29:32.074 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-23T14:29:32.074 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-23T14:29:32.074 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 163 k 2024-04-23T14:29:32.075 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-23T14:29:32.075 INFO:teuthology.orchestra.run.smithi038.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-23T14:29:32.075 INFO:teuthology.orchestra.run.smithi038.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-23T14:29:32.075 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-23T14:29:32.075 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-23T14:29:32.075 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1 x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 500 k 2024-04-23T14:29:32.076 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-23T14:29:32.076 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2 x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 5.4 M 2024-04-23T14:29:32.076 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-23T14:29:32.076 INFO:teuthology.orchestra.run.smithi038.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-23T14:29:32.076 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-23T14:29:32.076 INFO:teuthology.orchestra.run.smithi038.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-23T14:29:32.077 INFO:teuthology.orchestra.run.smithi038.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-23T14:29:32.077 INFO:teuthology.orchestra.run.smithi038.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-23T14:29:32.077 INFO:teuthology.orchestra.run.smithi038.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-23T14:29:32.077 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-23T14:29:32.077 INFO:teuthology.orchestra.run.smithi038.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-23T14:29:32.077 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-23T14:29:32.078 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-23T14:29:32.078 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-23T14:29:32.078 INFO:teuthology.orchestra.run.smithi038.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-23T14:29:32.079 INFO:teuthology.orchestra.run.smithi038.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-23T14:29:32.079 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-23T14:29:32.079 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-23T14:29:32.079 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-23T14:29:32.079 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 45 k 2024-04-23T14:29:32.079 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 135 k 2024-04-23T14:29:32.079 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-23T14:29:32.079 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-23T14:29:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-23T14:29:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-23T14:29:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-23T14:29:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-23T14:29:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-23T14:29:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-23T14:29:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-23T14:29:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-23T14:29:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-23T14:29:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-23T14:29:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-23T14:29:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-23T14:29:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-23T14:29:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-23T14:29:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-23T14:29:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-23T14:29:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-23T14:29:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-23T14:29:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-23T14:29:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-23T14:29:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-23T14:29:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-23T14:29:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-23T14:29:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-23T14:29:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-23T14:29:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-23T14:29:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-23T14:29:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-23T14:29:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-23T14:29:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-23T14:29:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-23T14:29:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-23T14:29:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-23T14:29:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-23T14:29:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-23T14:29:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-23T14:29:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-23T14:29:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-23T14:29:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-23T14:29:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-23T14:29:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-23T14:29:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-23T14:29:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-23T14:29:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-23T14:29:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-23T14:29:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-23T14:29:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-23T14:29:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-23T14:29:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-23T14:29:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-23T14:29:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-23T14:29:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-23T14:29:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-23T14:29:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-23T14:29:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-23T14:29:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-23T14:29:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-23T14:29:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-23T14:29:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-23T14:29:32.085 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-23T14:29:32.085 INFO:teuthology.orchestra.run.smithi038.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-23T14:29:32.085 INFO:teuthology.orchestra.run.smithi038.stdout:Installing weak dependencies: 2024-04-23T14:29:32.085 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-23T14:29:32.085 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-23T14:29:32.085 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-04-23T14:29:32.085 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================= 2024-04-23T14:29:32.085 INFO:teuthology.orchestra.run.smithi038.stdout:Install 148 Packages 2024-04-23T14:29:32.085 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrade 4 Packages 2024-04-23T14:29:32.085 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-23T14:29:32.086 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 222 M 2024-04-23T14:29:32.086 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-04-23T14:29:32.133 INFO:teuthology.orchestra.run.smithi028.stdout:(11/152): cephfs-mirror-19.0.0-3126.gd44fddd9.e 2.7 MB/s | 225 kB 00:00 2024-04-23T14:29:32.191 INFO:teuthology.orchestra.run.smithi028.stdout:(12/152): cephfs-shell-19.0.0-3126.gd44fddd9.el 471 kB/s | 27 kB 00:00 2024-04-23T14:29:32.251 INFO:teuthology.orchestra.run.smithi028.stdout:(13/152): libcephfs-devel-19.0.0-3126.gd44fddd9 549 kB/s | 32 kB 00:00 2024-04-23T14:29:32.359 INFO:teuthology.orchestra.run.smithi028.stdout:(14/152): libcephfs2-19.0.0-3126.gd44fddd9.el9. 6.8 MB/s | 746 kB 00:00 2024-04-23T14:29:32.671 INFO:teuthology.orchestra.run.smithi028.stdout:(15/152): ceph-osd-19.0.0-3126.gd44fddd9.el9.x8 9.8 MB/s | 17 MB 00:01 2024-04-23T14:29:32.710 INFO:teuthology.orchestra.run.smithi028.stdout:(16/152): libcephsqlite-19.0.0-3126.gd44fddd9.e 467 kB/s | 163 kB 00:00 2024-04-23T14:29:32.743 INFO:teuthology.orchestra.run.smithi028.stdout:(17/152): librados-devel-19.0.0-3126.gd44fddd9. 1.7 MB/s | 127 kB 00:00 2024-04-23T14:29:32.793 INFO:teuthology.orchestra.run.smithi028.stdout:(18/152): libradosstriper1-19.0.0-3126.gd44fddd 5.9 MB/s | 500 kB 00:00 2024-04-23T14:29:32.860 INFO:teuthology.orchestra.run.smithi028.stdout:(19/152): python3-ceph-argparse-19.0.0-3126.gd4 687 kB/s | 45 kB 00:00 2024-04-23T14:29:32.936 INFO:teuthology.orchestra.run.smithi028.stdout:(20/152): python3-ceph-common-19.0.0-3126.gd44f 1.8 MB/s | 135 kB 00:00 2024-04-23T14:29:33.003 INFO:teuthology.orchestra.run.smithi028.stdout:(21/152): python3-cephfs-19.0.0-3126.gd44fddd9. 2.4 MB/s | 161 kB 00:00 2024-04-23T14:29:33.050 INFO:teuthology.orchestra.run.smithi038.stdout:(1/152): ceph-19.0.0-3126.gd44fddd9.el9.x86_64. 25 kB/s | 6.6 kB 00:00 2024-04-23T14:29:33.078 INFO:teuthology.orchestra.run.smithi028.stdout:(22/152): python3-rados-19.0.0-3126.gd44fddd9.e 4.2 MB/s | 320 kB 00:00 2024-04-23T14:29:33.186 INFO:teuthology.orchestra.run.smithi028.stdout:(23/152): librgw2-19.0.0-3126.gd44fddd9.el9.x86 12 MB/s | 5.4 MB 00:00 2024-04-23T14:29:33.228 INFO:teuthology.orchestra.run.smithi028.stdout:(24/152): python3-rbd-19.0.0-3126.gd44fddd9.el9 1.9 MB/s | 297 kB 00:00 2024-04-23T14:29:33.262 INFO:teuthology.orchestra.run.smithi028.stdout:(25/152): python3-rgw-19.0.0-3126.gd44fddd9.el9 1.3 MB/s | 100 kB 00:00 2024-04-23T14:29:33.279 INFO:teuthology.orchestra.run.smithi028.stdout:(26/152): rbd-fuse-19.0.0-3126.gd44fddd9.el9.x8 1.7 MB/s | 85 kB 00:00 2024-04-23T14:29:33.371 INFO:teuthology.orchestra.run.smithi028.stdout:(27/152): rbd-nbd-19.0.0-3126.gd44fddd9.el9.x86 1.8 MB/s | 171 kB 00:00 2024-04-23T14:29:33.451 INFO:teuthology.orchestra.run.smithi105.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-04-23T14:29:33.463 INFO:teuthology.orchestra.run.smithi028.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3126.g 377 kB/s | 34 kB 00:00 2024-04-23T14:29:33.509 INFO:teuthology.orchestra.run.smithi038.stdout:(2/152): ceph-fuse-19.0.0-3126.gd44fddd9.el9.x8 1.9 MB/s | 891 kB 00:00 2024-04-23T14:29:33.563 INFO:teuthology.orchestra.run.smithi028.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3126.gd44fddd 1.4 MB/s | 142 kB 00:00 2024-04-23T14:29:33.647 INFO:teuthology.orchestra.run.smithi028.stdout:(30/152): rbd-mirror-19.0.0-3126.gd44fddd9.el9. 8.1 MB/s | 3.1 MB 00:00 2024-04-23T14:29:33.650 INFO:teuthology.orchestra.run.smithi038.stdout:(3/152): ceph-immutable-object-cache-19.0.0-312 1.0 MB/s | 145 kB 00:00 2024-04-23T14:29:33.776 INFO:teuthology.orchestra.run.smithi038.stdout:(4/152): ceph-base-19.0.0-3126.gd44fddd9.el9.x8 5.4 MB/s | 5.4 MB 00:00 2024-04-23T14:29:33.905 INFO:teuthology.orchestra.run.smithi028.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3126.gd44fd 10 MB/s | 3.5 MB 00:00 2024-04-23T14:29:33.989 INFO:teuthology.orchestra.run.smithi028.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3126.gd4 2.9 MB/s | 249 kB 00:00 2024-04-23T14:29:34.064 INFO:teuthology.orchestra.run.smithi028.stdout:(33/152): ceph-mgr-rook-19.0.0-3126.gd44fddd9.e 667 kB/s | 49 kB 00:00 2024-04-23T14:29:34.126 INFO:teuthology.orchestra.run.smithi038.stdout:(5/152): ceph-mgr-19.0.0-3126.gd44fddd9.el9.x86 4.3 MB/s | 1.5 MB 00:00 2024-04-23T14:29:34.139 INFO:teuthology.orchestra.run.smithi028.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3126.gd 222 kB/s | 16 kB 00:00 2024-04-23T14:29:34.240 INFO:teuthology.orchestra.run.smithi028.stdout:(35/152): ceph-volume-19.0.0-3126.gd44fddd9.el9 2.7 MB/s | 273 kB 00:00 2024-04-23T14:29:34.357 INFO:teuthology.orchestra.run.smithi028.stdout:(36/152): cephadm-19.0.0-3126.gd44fddd9.el9.noa 6.4 MB/s | 762 kB 00:00 2024-04-23T14:29:34.532 INFO:teuthology.orchestra.run.smithi028.stdout:(37/152): ceph-mgr-diskprediction-local-19.0.0- 8.4 MB/s | 7.4 MB 00:00 2024-04-23T14:29:34.574 INFO:teuthology.orchestra.run.smithi028.stdout:(38/152): cephfs-top-19.0.0-3126.gd44fddd9.el9. 84 kB/s | 18 kB 00:00 2024-04-23T14:29:34.682 INFO:teuthology.orchestra.run.smithi028.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 540 kB/s | 80 kB 00:00 2024-04-23T14:29:34.718 INFO:teuthology.orchestra.run.smithi038.stdout:(6/152): ceph-mon-19.0.0-3126.gd44fddd9.el9.x86 7.9 MB/s | 4.7 MB 00:00 2024-04-23T14:29:34.758 INFO:teuthology.orchestra.run.smithi028.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 195 kB 00:00 2024-04-23T14:29:34.768 INFO:teuthology.orchestra.run.smithi038.stdout:(7/152): ceph-mds-19.0.0-3126.gd44fddd9.el9.x86 2.1 MB/s | 2.4 MB 00:01 2024-04-23T14:29:34.791 INFO:teuthology.orchestra.run.smithi028.stdout:(41/152): libgfortran-11.4.1-3.el9.x86_64.rpm 3.6 MB/s | 803 kB 00:00 2024-04-23T14:29:34.833 INFO:teuthology.orchestra.run.smithi028.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.3 MB/s | 253 kB 00:00 2024-04-23T14:29:34.884 INFO:teuthology.orchestra.run.smithi028.stdout:(43/152): python3-cryptography-36.0.1-4.el9.x86 13 MB/s | 1.2 MB 00:00 2024-04-23T14:29:34.909 INFO:teuthology.orchestra.run.smithi028.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 106 kB 00:00 2024-04-23T14:29:34.943 INFO:teuthology.orchestra.run.smithi028.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-04-23T14:29:34.976 INFO:teuthology.orchestra.run.smithi028.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 1.9 MB/s | 125 kB 00:00 2024-04-23T14:29:35.010 INFO:teuthology.orchestra.run.smithi028.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 3.2 MB/s | 215 kB 00:00 2024-04-23T14:29:35.060 INFO:teuthology.orchestra.run.smithi028.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 8.2 MB/s | 689 kB 00:00 2024-04-23T14:29:35.110 INFO:teuthology.orchestra.run.smithi028.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 9.3 MB/s | 943 kB 00:00 2024-04-23T14:29:35.136 INFO:teuthology.orchestra.run.smithi028.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.4 MB/s | 107 kB 00:00 2024-04-23T14:29:35.186 INFO:teuthology.orchestra.run.smithi028.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 288 kB/s | 21 kB 00:00 2024-04-23T14:29:35.220 INFO:teuthology.orchestra.run.smithi028.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 3.7 MB/s | 313 kB 00:00 2024-04-23T14:29:35.270 INFO:teuthology.orchestra.run.smithi028.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 394 kB/s | 33 kB 00:00 2024-04-23T14:29:35.337 INFO:teuthology.orchestra.run.smithi028.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 274 kB/s | 18 kB 00:00 2024-04-23T14:29:35.404 INFO:teuthology.orchestra.run.smithi028.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 169 kB/s | 11 kB 00:00 2024-04-23T14:29:35.488 INFO:teuthology.orchestra.run.smithi028.stdout:(56/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 11 MB/s | 3.0 MB 00:00 2024-04-23T14:29:35.521 INFO:teuthology.orchestra.run.smithi028.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 160 kB 00:00 2024-04-23T14:29:35.527 INFO:teuthology.orchestra.run.smithi105.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-23T14:29:35.528 INFO:teuthology.orchestra.run.smithi105.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-23T14:29:35.538 INFO:teuthology.orchestra.run.smithi105.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-23T14:29:35.539 INFO:teuthology.orchestra.run.smithi105.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-23T14:29:35.540 INFO:teuthology.orchestra.run.smithi105.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-23T14:29:35.541 INFO:teuthology.orchestra.run.smithi105.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-23T14:29:35.542 INFO:teuthology.orchestra.run.smithi105.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-23T14:29:35.544 INFO:teuthology.orchestra.run.smithi105.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-23T14:29:35.545 INFO:teuthology.orchestra.run.smithi105.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-23T14:29:35.572 INFO:teuthology.orchestra.run.smithi028.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 546 kB/s | 45 kB 00:00 2024-04-23T14:29:35.605 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-23T14:29:35.622 INFO:teuthology.orchestra.run.smithi028.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 6.5 MB/s | 662 kB 00:00 2024-04-23T14:29:35.623 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================= 2024-04-23T14:29:35.623 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-04-23T14:29:35.623 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================= 2024-04-23T14:29:35.623 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-04-23T14:29:35.623 INFO:teuthology.orchestra.run.smithi105.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-23T14:29:35.623 INFO:teuthology.orchestra.run.smithi105.stdout: ceph x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 6.6 k 2024-04-23T14:29:35.623 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-base x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 5.4 M 2024-04-23T14:29:35.623 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-fuse x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 891 k 2024-04-23T14:29:35.624 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 145 k 2024-04-23T14:29:35.624 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 1.5 M 2024-04-23T14:29:35.624 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 142 k 2024-04-23T14:29:35.624 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 3.5 M 2024-04-23T14:29:35.624 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 7.4 M 2024-04-23T14:29:35.624 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-rook noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 49 k 2024-04-23T14:29:35.624 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-radosgw x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 11 M 2024-04-23T14:29:35.624 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-test x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 49 M 2024-04-23T14:29:35.624 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-volume noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 273 k 2024-04-23T14:29:35.624 INFO:teuthology.orchestra.run.smithi105.stdout: cephadm noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 762 k 2024-04-23T14:29:35.624 INFO:teuthology.orchestra.run.smithi105.stdout: cephfs-mirror x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 225 k 2024-04-23T14:29:35.624 INFO:teuthology.orchestra.run.smithi105.stdout: cephfs-shell x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 27 k 2024-04-23T14:29:35.624 INFO:teuthology.orchestra.run.smithi105.stdout: cephfs-top noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 18 k 2024-04-23T14:29:35.625 INFO:teuthology.orchestra.run.smithi105.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-23T14:29:35.625 INFO:teuthology.orchestra.run.smithi105.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-23T14:29:35.625 INFO:teuthology.orchestra.run.smithi105.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-23T14:29:35.625 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs-devel x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 32 k 2024-04-23T14:29:35.625 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs2 x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 746 k 2024-04-23T14:29:35.625 INFO:teuthology.orchestra.run.smithi105.stdout: librados-devel x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 127 k 2024-04-23T14:29:35.625 INFO:teuthology.orchestra.run.smithi105.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-23T14:29:35.625 INFO:teuthology.orchestra.run.smithi105.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-23T14:29:35.625 INFO:teuthology.orchestra.run.smithi105.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-23T14:29:35.625 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-23T14:29:35.625 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-23T14:29:35.625 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-23T14:29:35.625 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cephfs x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 161 k 2024-04-23T14:29:35.625 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rados x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 320 k 2024-04-23T14:29:35.626 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rbd x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 297 k 2024-04-23T14:29:35.626 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rgw x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 100 k 2024-04-23T14:29:35.626 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-fuse x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 85 k 2024-04-23T14:29:35.626 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-mirror x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 3.1 M 2024-04-23T14:29:35.626 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-nbd x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 171 k 2024-04-23T14:29:35.626 INFO:teuthology.orchestra.run.smithi105.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-23T14:29:35.626 INFO:teuthology.orchestra.run.smithi105.stdout:Upgrading: 2024-04-23T14:29:35.626 INFO:teuthology.orchestra.run.smithi105.stdout: librados2 x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 3.4 M 2024-04-23T14:29:35.626 INFO:teuthology.orchestra.run.smithi105.stdout: librbd1 x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 3.1 M 2024-04-23T14:29:35.626 INFO:teuthology.orchestra.run.smithi105.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-23T14:29:35.626 INFO:teuthology.orchestra.run.smithi105.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-23T14:29:35.626 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-04-23T14:29:35.626 INFO:teuthology.orchestra.run.smithi105.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-23T14:29:35.626 INFO:teuthology.orchestra.run.smithi105.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-23T14:29:35.627 INFO:teuthology.orchestra.run.smithi105.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-23T14:29:35.627 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-common x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 21 M 2024-04-23T14:29:35.627 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 34 k 2024-04-23T14:29:35.627 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mds x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 2.4 M 2024-04-23T14:29:35.627 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 249 k 2024-04-23T14:29:35.627 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mon x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 4.7 M 2024-04-23T14:29:35.627 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-osd x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 17 M 2024-04-23T14:29:35.627 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3126.gd44fddd9.el9 ceph-noarch 16 k 2024-04-23T14:29:35.627 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-selinux x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 25 k 2024-04-23T14:29:35.627 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-23T14:29:35.627 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-23T14:29:35.627 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-23T14:29:35.627 INFO:teuthology.orchestra.run.smithi105.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-23T14:29:35.627 INFO:teuthology.orchestra.run.smithi105.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-23T14:29:35.628 INFO:teuthology.orchestra.run.smithi105.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-23T14:29:35.628 INFO:teuthology.orchestra.run.smithi105.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-23T14:29:35.628 INFO:teuthology.orchestra.run.smithi105.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-23T14:29:35.628 INFO:teuthology.orchestra.run.smithi105.stdout: libcephsqlite x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 163 k 2024-04-23T14:29:35.628 INFO:teuthology.orchestra.run.smithi105.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-23T14:29:35.628 INFO:teuthology.orchestra.run.smithi105.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-23T14:29:35.628 INFO:teuthology.orchestra.run.smithi105.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-23T14:29:35.628 INFO:teuthology.orchestra.run.smithi105.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-23T14:29:35.628 INFO:teuthology.orchestra.run.smithi105.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-23T14:29:35.628 INFO:teuthology.orchestra.run.smithi105.stdout: libradosstriper1 x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 500 k 2024-04-23T14:29:35.628 INFO:teuthology.orchestra.run.smithi105.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-23T14:29:35.628 INFO:teuthology.orchestra.run.smithi105.stdout: librgw2 x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 5.4 M 2024-04-23T14:29:35.628 INFO:teuthology.orchestra.run.smithi105.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-23T14:29:35.628 INFO:teuthology.orchestra.run.smithi105.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-23T14:29:35.629 INFO:teuthology.orchestra.run.smithi105.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-23T14:29:35.629 INFO:teuthology.orchestra.run.smithi105.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-23T14:29:35.629 INFO:teuthology.orchestra.run.smithi105.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-23T14:29:35.629 INFO:teuthology.orchestra.run.smithi105.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-23T14:29:35.629 INFO:teuthology.orchestra.run.smithi105.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-23T14:29:35.629 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-23T14:29:35.629 INFO:teuthology.orchestra.run.smithi105.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-23T14:29:35.629 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-23T14:29:35.629 INFO:teuthology.orchestra.run.smithi105.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-23T14:29:35.629 INFO:teuthology.orchestra.run.smithi105.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-23T14:29:35.629 INFO:teuthology.orchestra.run.smithi105.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-23T14:29:35.629 INFO:teuthology.orchestra.run.smithi105.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-23T14:29:35.629 INFO:teuthology.orchestra.run.smithi105.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-23T14:29:35.629 INFO:teuthology.orchestra.run.smithi105.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-23T14:29:35.629 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-23T14:29:35.630 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-argparse x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 45 k 2024-04-23T14:29:35.630 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-common x86_64 2:19.0.0-3126.gd44fddd9.el9 ceph 135 k 2024-04-23T14:29:35.630 INFO:teuthology.orchestra.run.smithi105.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-23T14:29:35.630 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-23T14:29:35.630 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-23T14:29:35.630 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-23T14:29:35.630 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-23T14:29:35.630 INFO:teuthology.orchestra.run.smithi105.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-23T14:29:35.630 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-23T14:29:35.630 INFO:teuthology.orchestra.run.smithi105.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-23T14:29:35.630 INFO:teuthology.orchestra.run.smithi105.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-23T14:29:35.630 INFO:teuthology.orchestra.run.smithi105.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-23T14:29:35.630 INFO:teuthology.orchestra.run.smithi105.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-23T14:29:35.630 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-23T14:29:35.631 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-23T14:29:35.631 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-23T14:29:35.631 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-23T14:29:35.631 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-23T14:29:35.631 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-23T14:29:35.631 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-23T14:29:35.631 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-23T14:29:35.631 INFO:teuthology.orchestra.run.smithi105.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-23T14:29:35.631 INFO:teuthology.orchestra.run.smithi105.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-23T14:29:35.631 INFO:teuthology.orchestra.run.smithi105.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-23T14:29:35.631 INFO:teuthology.orchestra.run.smithi105.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-23T14:29:35.631 INFO:teuthology.orchestra.run.smithi105.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-23T14:29:35.631 INFO:teuthology.orchestra.run.smithi105.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-23T14:29:35.631 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-23T14:29:35.632 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-23T14:29:35.632 INFO:teuthology.orchestra.run.smithi105.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-23T14:29:35.632 INFO:teuthology.orchestra.run.smithi105.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-23T14:29:35.632 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-23T14:29:35.632 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-23T14:29:35.632 INFO:teuthology.orchestra.run.smithi105.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-23T14:29:35.632 INFO:teuthology.orchestra.run.smithi105.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-23T14:29:35.632 INFO:teuthology.orchestra.run.smithi105.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-23T14:29:35.632 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-23T14:29:35.632 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-23T14:29:35.632 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-23T14:29:35.632 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-23T14:29:35.632 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-23T14:29:35.632 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-23T14:29:35.633 INFO:teuthology.orchestra.run.smithi105.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-23T14:29:35.633 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-23T14:29:35.633 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-23T14:29:35.633 INFO:teuthology.orchestra.run.smithi105.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-23T14:29:35.633 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-23T14:29:35.633 INFO:teuthology.orchestra.run.smithi105.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-23T14:29:35.633 INFO:teuthology.orchestra.run.smithi105.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-23T14:29:35.633 INFO:teuthology.orchestra.run.smithi105.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-23T14:29:35.633 INFO:teuthology.orchestra.run.smithi105.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-23T14:29:35.633 INFO:teuthology.orchestra.run.smithi105.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-23T14:29:35.633 INFO:teuthology.orchestra.run.smithi105.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-23T14:29:35.633 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-23T14:29:35.633 INFO:teuthology.orchestra.run.smithi105.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-23T14:29:35.633 INFO:teuthology.orchestra.run.smithi105.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-23T14:29:35.633 INFO:teuthology.orchestra.run.smithi105.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-23T14:29:35.634 INFO:teuthology.orchestra.run.smithi105.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-23T14:29:35.634 INFO:teuthology.orchestra.run.smithi105.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-23T14:29:35.634 INFO:teuthology.orchestra.run.smithi105.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-23T14:29:35.634 INFO:teuthology.orchestra.run.smithi105.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-23T14:29:35.634 INFO:teuthology.orchestra.run.smithi105.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-23T14:29:35.634 INFO:teuthology.orchestra.run.smithi105.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-23T14:29:35.634 INFO:teuthology.orchestra.run.smithi105.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-23T14:29:35.634 INFO:teuthology.orchestra.run.smithi105.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-23T14:29:35.634 INFO:teuthology.orchestra.run.smithi105.stdout:Installing weak dependencies: 2024-04-23T14:29:35.634 INFO:teuthology.orchestra.run.smithi105.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-23T14:29:35.634 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-23T14:29:35.634 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-23T14:29:35.634 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================= 2024-04-23T14:29:35.634 INFO:teuthology.orchestra.run.smithi105.stdout:Install 148 Packages 2024-04-23T14:29:35.635 INFO:teuthology.orchestra.run.smithi105.stdout:Upgrade 4 Packages 2024-04-23T14:29:35.635 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-23T14:29:35.635 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 222 M 2024-04-23T14:29:35.635 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-04-23T14:29:35.664 INFO:teuthology.orchestra.run.smithi028.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 6.3 MB/s | 582 kB 00:00 2024-04-23T14:29:35.714 INFO:teuthology.orchestra.run.smithi028.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 279 kB/s | 25 kB 00:00 2024-04-23T14:29:35.764 INFO:teuthology.orchestra.run.smithi028.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 474 kB/s | 47 kB 00:00 2024-04-23T14:29:35.798 INFO:teuthology.orchestra.run.smithi028.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.5 MB/s | 292 kB 00:00 2024-04-23T14:29:35.832 INFO:teuthology.orchestra.run.smithi028.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 2.8 MB/s | 188 kB 00:00 2024-04-23T14:29:35.873 INFO:teuthology.orchestra.run.smithi028.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 475 kB/s | 35 kB 00:00 2024-04-23T14:29:35.977 INFO:teuthology.orchestra.run.smithi038.stdout:(8/152): ceph-common-19.0.0-3126.gd44fddd9.el9. 6.6 MB/s | 21 MB 00:03 2024-04-23T14:29:36.007 INFO:teuthology.orchestra.run.smithi028.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 27 MB/s | 4.7 MB 00:00 2024-04-23T14:29:36.027 INFO:teuthology.orchestra.run.smithi038.stdout:(9/152): ceph-selinux-19.0.0-3126.gd44fddd9.el9 507 kB/s | 25 kB 00:00 2024-04-23T14:29:36.099 INFO:teuthology.orchestra.run.smithi028.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 20 MB/s | 4.4 MB 00:00 2024-04-23T14:29:36.183 INFO:teuthology.orchestra.run.smithi028.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 9.1 MB/s | 1.6 MB 00:00 2024-04-23T14:29:36.233 INFO:teuthology.orchestra.run.smithi028.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 6.7 MB/s | 911 kB 00:00 2024-04-23T14:29:36.258 INFO:teuthology.orchestra.run.smithi028.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.7 MB/s | 133 kB 00:00 2024-04-23T14:29:36.350 INFO:teuthology.orchestra.run.smithi028.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 11 MB/s | 1.0 MB 00:00 2024-04-23T14:29:36.392 INFO:teuthology.orchestra.run.smithi028.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 2.1 MB/s | 87 kB 00:00 2024-04-23T14:29:36.527 INFO:teuthology.orchestra.run.smithi105.stdout:(1/152): ceph-19.0.0-3126.gd44fddd9.el9.x86_64. 21 kB/s | 6.6 kB 00:00 2024-04-23T14:29:36.543 INFO:teuthology.orchestra.run.smithi028.stdout:(73/152): postgresql-server-13.14-1.el9.x86_64. 19 MB/s | 5.8 MB 00:00 2024-04-23T14:29:36.593 INFO:teuthology.orchestra.run.smithi028.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 4.9 MB/s | 244 kB 00:00 2024-04-23T14:29:36.603 INFO:teuthology.orchestra.run.smithi038.stdout:(10/152): ceph-osd-19.0.0-3126.gd44fddd9.el9.x8 8.9 MB/s | 17 MB 00:01 2024-04-23T14:29:36.643 INFO:teuthology.orchestra.run.smithi028.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 4.9 MB/s | 248 kB 00:00 2024-04-23T14:29:36.720 INFO:teuthology.orchestra.run.smithi038.stdout:(11/152): cephfs-mirror-19.0.0-3126.gd44fddd9.e 1.9 MB/s | 225 kB 00:00 2024-04-23T14:29:36.744 INFO:teuthology.orchestra.run.smithi028.stdout:(76/152): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-04-23T14:29:36.770 INFO:teuthology.orchestra.run.smithi028.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 205 kB/s | 26 kB 00:00 2024-04-23T14:29:36.794 INFO:teuthology.orchestra.run.smithi028.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 386 kB/s | 19 kB 00:00 2024-04-23T14:29:36.836 INFO:teuthology.orchestra.run.smithi038.stdout:(12/152): cephfs-shell-19.0.0-3126.gd44fddd9.el 232 kB/s | 27 kB 00:00 2024-04-23T14:29:36.858 INFO:teuthology.orchestra.run.smithi028.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 172 kB 00:00 2024-04-23T14:29:36.887 INFO:teuthology.orchestra.run.smithi028.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 381 kB/s | 35 kB 00:00 2024-04-23T14:29:36.937 INFO:teuthology.orchestra.run.smithi028.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 5.0 MB/s | 250 kB 00:00 2024-04-23T14:29:36.962 INFO:teuthology.orchestra.run.smithi038.stdout:(13/152): libcephfs-devel-19.0.0-3126.gd44fddd9 254 kB/s | 32 kB 00:00 2024-04-23T14:29:36.987 INFO:teuthology.orchestra.run.smithi028.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 4.4 MB/s | 222 kB 00:00 2024-04-23T14:29:37.029 INFO:teuthology.orchestra.run.smithi028.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 1.9 MB/s | 77 kB 00:00 2024-04-23T14:29:37.071 INFO:teuthology.orchestra.run.smithi028.stdout:(84/152): python3-prettytable-0.7.2-27.el9.noar 1.0 MB/s | 42 kB 00:00 2024-04-23T14:29:37.171 INFO:teuthology.orchestra.run.smithi028.stdout:(85/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 5.1 MB 00:00 2024-04-23T14:29:37.213 INFO:teuthology.orchestra.run.smithi028.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 1.9 MB/s | 269 kB 00:00 2024-04-23T14:29:37.247 INFO:teuthology.orchestra.run.smithi028.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-04-23T14:29:37.280 INFO:teuthology.orchestra.run.smithi028.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 279 kB 00:00 2024-04-23T14:29:37.306 INFO:teuthology.orchestra.run.smithi028.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 882 kB/s | 51 kB 00:00 2024-04-23T14:29:37.332 INFO:teuthology.orchestra.run.smithi028.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-04-23T14:29:37.373 INFO:teuthology.orchestra.run.smithi028.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 1.0 MB/s | 42 kB 00:00 2024-04-23T14:29:37.412 INFO:teuthology.orchestra.run.smithi038.stdout:(14/152): libcephfs2-19.0.0-3126.gd44fddd9.el9. 1.6 MB/s | 746 kB 00:00 2024-04-23T14:29:37.415 INFO:teuthology.orchestra.run.smithi028.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.1 MB/s | 44 kB 00:00 2024-04-23T14:29:37.466 INFO:teuthology.orchestra.run.smithi028.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 4.4 MB/s | 221 kB 00:00 2024-04-23T14:29:37.507 INFO:teuthology.orchestra.run.smithi028.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 877 kB/s | 36 kB 00:00 2024-04-23T14:29:37.519 INFO:teuthology.orchestra.run.smithi105.stdout:(2/152): ceph-fuse-19.0.0-3126.gd44fddd9.el9.x8 898 kB/s | 891 kB 00:00 2024-04-23T14:29:37.554 INFO:teuthology.orchestra.run.smithi038.stdout:(15/152): libcephsqlite-19.0.0-3126.gd44fddd9.e 1.1 MB/s | 163 kB 00:00 2024-04-23T14:29:37.641 INFO:teuthology.orchestra.run.smithi028.stdout:(95/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 356 kB/s | 47 kB 00:00 2024-04-23T14:29:37.696 INFO:teuthology.orchestra.run.smithi038.stdout:(16/152): librados-devel-19.0.0-3126.gd44fddd9. 896 kB/s | 127 kB 00:00 2024-04-23T14:29:37.699 INFO:teuthology.orchestra.run.smithi028.stdout:(96/152): socat-1.7.4.1-5.el9.x86_64.rpm 5.2 MB/s | 305 kB 00:00 2024-04-23T14:29:37.728 INFO:teuthology.orchestra.run.smithi105.stdout:(3/152): ceph-immutable-object-cache-19.0.0-312 697 kB/s | 145 kB 00:00 2024-04-23T14:29:37.741 INFO:teuthology.orchestra.run.smithi028.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 1.4 MB/s | 58 kB 00:00 2024-04-23T14:29:37.784 INFO:teuthology.orchestra.run.smithi028.stdout:(98/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 2.2 MB/s | 94 kB 00:00 2024-04-23T14:29:37.825 INFO:teuthology.orchestra.run.smithi028.stdout:(99/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-04-23T14:29:37.868 INFO:teuthology.orchestra.run.smithi028.stdout:(100/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-04-23T14:29:37.905 INFO:teuthology.orchestra.run.smithi038.stdout:(17/152): libradosstriper1-19.0.0-3126.gd44fddd 2.3 MB/s | 500 kB 00:00 2024-04-23T14:29:38.118 INFO:teuthology.orchestra.run.smithi028.stdout:(101/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 20 MB/s | 16 MB 00:00 2024-04-23T14:29:38.301 INFO:teuthology.orchestra.run.smithi028.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 52 kB/s | 22 kB 00:00 2024-04-23T14:29:38.561 INFO:teuthology.orchestra.run.smithi105.stdout:(4/152): ceph-common-19.0.0-3126.gd44fddd9.el9. 9.0 MB/s | 21 MB 00:02 2024-04-23T14:29:38.645 INFO:teuthology.orchestra.run.smithi105.stdout:(5/152): ceph-base-19.0.0-3126.gd44fddd9.el9.x8 2.2 MB/s | 5.4 MB 00:02 2024-04-23T14:29:38.778 INFO:teuthology.orchestra.run.smithi105.stdout:(6/152): ceph-mgr-19.0.0-3126.gd44fddd9.el9.x86 6.9 MB/s | 1.5 MB 00:00 2024-04-23T14:29:39.177 INFO:teuthology.orchestra.run.smithi028.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 816 kB/s | 863 kB 00:01 2024-04-23T14:29:39.280 INFO:teuthology.orchestra.run.smithi038.stdout:(18/152): librgw2-19.0.0-3126.gd44fddd9.el9.x86 3.9 MB/s | 5.4 MB 00:01 2024-04-23T14:29:39.354 INFO:teuthology.orchestra.run.smithi105.stdout:(7/152): ceph-mds-19.0.0-3126.gd44fddd9.el9.x86 1.5 MB/s | 2.4 MB 00:01 2024-04-23T14:29:39.385 INFO:teuthology.orchestra.run.smithi028.stdout:(104/152): abseil-cpp-20211102.0-3.el9.x86_64.r 506 kB/s | 548 kB 00:01 2024-04-23T14:29:39.397 INFO:teuthology.orchestra.run.smithi038.stdout:(19/152): python3-ceph-argparse-19.0.0-3126.gd4 387 kB/s | 45 kB 00:00 2024-04-23T14:29:39.514 INFO:teuthology.orchestra.run.smithi038.stdout:(20/152): python3-ceph-common-19.0.0-3126.gd44f 1.1 MB/s | 135 kB 00:00 2024-04-23T14:29:39.527 INFO:teuthology.orchestra.run.smithi028.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 138 kB/s | 19 kB 00:00 2024-04-23T14:29:39.631 INFO:teuthology.orchestra.run.smithi038.stdout:(21/152): python3-cephfs-19.0.0-3126.gd44fddd9. 1.4 MB/s | 161 kB 00:00 2024-04-23T14:29:39.748 INFO:teuthology.orchestra.run.smithi038.stdout:(22/152): python3-rados-19.0.0-3126.gd44fddd9.e 2.7 MB/s | 320 kB 00:00 2024-04-23T14:29:39.845 INFO:teuthology.orchestra.run.smithi105.stdout:(8/152): ceph-mon-19.0.0-3126.gd44fddd9.el9.x86 3.9 MB/s | 4.7 MB 00:01 2024-04-23T14:29:39.864 INFO:teuthology.orchestra.run.smithi038.stdout:(23/152): python3-rbd-19.0.0-3126.gd44fddd9.el9 2.5 MB/s | 297 kB 00:00 2024-04-23T14:29:39.946 INFO:teuthology.orchestra.run.smithi105.stdout:(9/152): ceph-selinux-19.0.0-3126.gd44fddd9.el9 252 kB/s | 25 kB 00:00 2024-04-23T14:29:39.982 INFO:teuthology.orchestra.run.smithi038.stdout:(24/152): python3-rgw-19.0.0-3126.gd44fddd9.el9 853 kB/s | 100 kB 00:00 2024-04-23T14:29:40.020 INFO:teuthology.orchestra.run.smithi028.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 367 kB/s | 309 kB 00:00 2024-04-23T14:29:40.107 INFO:teuthology.orchestra.run.smithi038.stdout:(25/152): rbd-fuse-19.0.0-3126.gd44fddd9.el9.x8 684 kB/s | 85 kB 00:00 2024-04-23T14:29:40.137 INFO:teuthology.orchestra.run.smithi028.stdout:(107/152): libarrow-9.0.0-10.el9.x86_64.rpm 7.3 MB/s | 4.4 MB 00:00 2024-04-23T14:29:40.162 INFO:teuthology.orchestra.run.smithi028.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 184 kB/s | 26 kB 00:00 2024-04-23T14:29:40.229 INFO:teuthology.orchestra.run.smithi105.stdout:(10/152): ceph-osd-19.0.0-3126.gd44fddd9.el9.x8 12 MB/s | 17 MB 00:01 2024-04-23T14:29:40.279 INFO:teuthology.orchestra.run.smithi028.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 345 kB/s | 49 kB 00:00 2024-04-23T14:29:40.288 INFO:teuthology.orchestra.run.smithi105.stdout:(11/152): cephfs-mirror-19.0.0-3126.gd44fddd9.e 3.8 MB/s | 225 kB 00:00 2024-04-23T14:29:40.312 INFO:teuthology.orchestra.run.smithi028.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 451 kB/s | 67 kB 00:00 2024-04-23T14:29:40.356 INFO:teuthology.orchestra.run.smithi105.stdout:(12/152): cephfs-shell-19.0.0-3126.gd44fddd9.el 404 kB/s | 27 kB 00:00 2024-04-23T14:29:40.414 INFO:teuthology.orchestra.run.smithi105.stdout:(13/152): libcephfs-devel-19.0.0-3126.gd44fddd9 544 kB/s | 32 kB 00:00 2024-04-23T14:29:40.429 INFO:teuthology.orchestra.run.smithi028.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-04-23T14:29:40.507 INFO:teuthology.orchestra.run.smithi105.stdout:(14/152): libcephfs2-19.0.0-3126.gd44fddd9.el9. 8.0 MB/s | 746 kB 00:00 2024-04-23T14:29:40.573 INFO:teuthology.orchestra.run.smithi105.stdout:(15/152): libcephsqlite-19.0.0-3126.gd44fddd9.e 2.4 MB/s | 163 kB 00:00 2024-04-23T14:29:40.604 INFO:teuthology.orchestra.run.smithi028.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 3.1 MB/s | 548 kB 00:00 2024-04-23T14:29:40.631 INFO:teuthology.orchestra.run.smithi105.stdout:(16/152): librados-devel-19.0.0-3126.gd44fddd9. 2.1 MB/s | 127 kB 00:00 2024-04-23T14:29:40.705 INFO:teuthology.orchestra.run.smithi028.stdout:(113/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.1 MB/s | 838 kB 00:00 2024-04-23T14:29:40.715 INFO:teuthology.orchestra.run.smithi105.stdout:(17/152): libradosstriper1-19.0.0-3126.gd44fddd 5.9 MB/s | 500 kB 00:00 2024-04-23T14:29:40.747 INFO:teuthology.orchestra.run.smithi028.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 307 kB/s | 43 kB 00:00 2024-04-23T14:29:40.839 INFO:teuthology.orchestra.run.smithi028.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 241 kB/s | 32 kB 00:00 2024-04-23T14:29:40.881 INFO:teuthology.orchestra.run.smithi028.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 113 kB/s | 15 kB 00:00 2024-04-23T14:29:40.890 INFO:teuthology.orchestra.run.smithi038.stdout:(26/152): rbd-mirror-19.0.0-3126.gd44fddd9.el9. 3.9 MB/s | 3.1 MB 00:00 2024-04-23T14:29:40.989 INFO:teuthology.orchestra.run.smithi028.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.1 MB/s | 172 kB 00:00 2024-04-23T14:29:41.008 INFO:teuthology.orchestra.run.smithi038.stdout:(27/152): ceph-radosgw-19.0.0-3126.gd44fddd9.el 1.7 MB/s | 11 MB 00:06 2024-04-23T14:29:41.023 INFO:teuthology.orchestra.run.smithi028.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 2.5 MB/s | 358 kB 00:00 2024-04-23T14:29:41.033 INFO:teuthology.orchestra.run.smithi038.stdout:(28/152): rbd-nbd-19.0.0-3126.gd44fddd9.el9.x86 1.2 MB/s | 171 kB 00:00 2024-04-23T14:29:41.125 INFO:teuthology.orchestra.run.smithi038.stdout:(29/152): ceph-grafana-dashboards-19.0.0-3126.g 293 kB/s | 34 kB 00:00 2024-04-23T14:29:41.165 INFO:teuthology.orchestra.run.smithi028.stdout:(119/152): python3-colorama-0.4.4-12.el9.noarch 250 kB/s | 35 kB 00:00 2024-04-23T14:29:41.166 INFO:teuthology.orchestra.run.smithi038.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3126.gd44fddd 1.0 MB/s | 142 kB 00:00 2024-04-23T14:29:41.199 INFO:teuthology.orchestra.run.smithi028.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.4 MB/s | 303 kB 00:00 2024-04-23T14:29:41.207 INFO:teuthology.orchestra.run.smithi105.stdout:(18/152): librgw2-19.0.0-3126.gd44fddd9.el9.x86 11 MB/s | 5.4 MB 00:00 2024-04-23T14:29:41.265 INFO:teuthology.orchestra.run.smithi105.stdout:(19/152): python3-ceph-argparse-19.0.0-3126.gd4 779 kB/s | 45 kB 00:00 2024-04-23T14:29:41.324 INFO:teuthology.orchestra.run.smithi028.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 1.3 MB/s | 206 kB 00:00 2024-04-23T14:29:41.341 INFO:teuthology.orchestra.run.smithi105.stdout:(20/152): python3-ceph-common-19.0.0-3126.gd44f 1.8 MB/s | 135 kB 00:00 2024-04-23T14:29:41.410 INFO:teuthology.orchestra.run.smithi105.stdout:(21/152): python3-cephfs-19.0.0-3126.gd44fddd9. 2.3 MB/s | 161 kB 00:00 2024-04-23T14:29:41.477 INFO:teuthology.orchestra.run.smithi105.stdout:(22/152): python3-rados-19.0.0-3126.gd44fddd9.e 4.7 MB/s | 320 kB 00:00 2024-04-23T14:29:41.532 INFO:teuthology.orchestra.run.smithi028.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 696 kB/s | 144 kB 00:00 2024-04-23T14:29:41.535 INFO:teuthology.orchestra.run.smithi105.stdout:(23/152): python3-rbd-19.0.0-3126.gd44fddd9.el9 5.0 MB/s | 297 kB 00:00 2024-04-23T14:29:41.577 INFO:teuthology.orchestra.run.smithi105.stdout:(24/152): python3-rgw-19.0.0-3126.gd44fddd9.el9 2.3 MB/s | 100 kB 00:00 2024-04-23T14:29:41.628 INFO:teuthology.orchestra.run.smithi105.stdout:(25/152): rbd-fuse-19.0.0-3126.gd44fddd9.el9.x8 1.7 MB/s | 85 kB 00:00 2024-04-23T14:29:41.632 INFO:teuthology.orchestra.run.smithi028.stdout:(123/152): python3-grpcio-1.46.7-10.el9.x86_64. 4.7 MB/s | 2.0 MB 00:00 2024-04-23T14:29:41.750 INFO:teuthology.orchestra.run.smithi028.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 49 kB/s | 11 kB 00:00 2024-04-23T14:29:41.808 INFO:teuthology.orchestra.run.smithi028.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 101 kB/s | 18 kB 00:00 2024-04-23T14:29:41.865 INFO:teuthology.orchestra.run.smithi038.stdout:(31/152): ceph-test-19.0.0-3126.gd44fddd9.el9.x 8.5 MB/s | 49 MB 00:05 2024-04-23T14:29:41.878 INFO:teuthology.orchestra.run.smithi105.stdout:(26/152): rbd-mirror-19.0.0-3126.gd44fddd9.el9. 12 MB/s | 3.1 MB 00:00 2024-04-23T14:29:41.901 INFO:teuthology.orchestra.run.smithi038.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3126.gd4 2.9 MB/s | 249 kB 00:00 2024-04-23T14:29:41.917 INFO:teuthology.orchestra.run.smithi028.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 139 kB/s | 23 kB 00:00 2024-04-23T14:29:41.928 INFO:teuthology.orchestra.run.smithi105.stdout:(27/152): rbd-nbd-19.0.0-3126.gd44fddd9.el9.x86 3.4 MB/s | 171 kB 00:00 2024-04-23T14:29:41.943 INFO:teuthology.orchestra.run.smithi038.stdout:(33/152): ceph-mgr-rook-19.0.0-3126.gd44fddd9.e 1.2 MB/s | 49 kB 00:00 2024-04-23T14:29:41.943 INFO:teuthology.orchestra.run.smithi028.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 146 kB/s | 19 kB 00:00 2024-04-23T14:29:41.970 INFO:teuthology.orchestra.run.smithi105.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3126.g 826 kB/s | 34 kB 00:00 2024-04-23T14:29:41.976 INFO:teuthology.orchestra.run.smithi038.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3126.gd 492 kB/s | 16 kB 00:00 2024-04-23T14:29:42.020 INFO:teuthology.orchestra.run.smithi105.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3126.gd44fddd 2.8 MB/s | 142 kB 00:00 2024-04-23T14:29:42.026 INFO:teuthology.orchestra.run.smithi038.stdout:(35/152): ceph-volume-19.0.0-3126.gd44fddd9.el9 5.3 MB/s | 273 kB 00:00 2024-04-23T14:29:42.059 INFO:teuthology.orchestra.run.smithi028.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 139 kB/s | 20 kB 00:00 2024-04-23T14:29:42.168 INFO:teuthology.orchestra.run.smithi028.stdout:(129/152): python3-kubernetes-26.1.0-2.el9.noar 4.5 MB/s | 1.0 MB 00:00 2024-04-23T14:29:42.202 INFO:teuthology.orchestra.run.smithi028.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 328 kB/s | 46 kB 00:00 2024-04-23T14:29:42.202 INFO:teuthology.orchestra.run.smithi038.stdout:(36/152): cephadm-19.0.0-3126.gd44fddd9.el9.noa 4.3 MB/s | 762 kB 00:00 2024-04-23T14:29:42.285 INFO:teuthology.orchestra.run.smithi038.stdout:(37/152): cephfs-top-19.0.0-3126.gd44fddd9.el9. 219 kB/s | 18 kB 00:00 2024-04-23T14:29:42.327 INFO:teuthology.orchestra.run.smithi028.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 499 kB/s | 79 kB 00:00 2024-04-23T14:29:42.337 INFO:teuthology.orchestra.run.smithi105.stdout:(30/152): ceph-mgr-dashboard-19.0.0-3126.gd44fd 11 MB/s | 3.5 MB 00:00 2024-04-23T14:29:42.427 INFO:teuthology.orchestra.run.smithi028.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 257 kB/s | 58 kB 00:00 2024-04-23T14:29:42.502 INFO:teuthology.orchestra.run.smithi028.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 1.5 MB/s | 272 kB 00:00 2024-04-23T14:29:42.536 INFO:teuthology.orchestra.run.smithi038.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 321 kB/s | 80 kB 00:00 2024-04-23T14:29:42.569 INFO:teuthology.orchestra.run.smithi028.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 116 kB/s | 16 kB 00:00 2024-04-23T14:29:42.653 INFO:teuthology.orchestra.run.smithi028.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 601 kB/s | 90 kB 00:00 2024-04-23T14:29:42.720 INFO:teuthology.orchestra.run.smithi028.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 188 kB/s | 28 kB 00:00 2024-04-23T14:29:42.787 INFO:teuthology.orchestra.run.smithi028.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 231 kB/s | 31 kB 00:00 2024-04-23T14:29:42.846 INFO:teuthology.orchestra.run.smithi028.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-04-23T14:29:42.887 INFO:teuthology.orchestra.run.smithi105.stdout:(31/152): ceph-mgr-diskprediction-local-19.0.0- 13 MB/s | 7.4 MB 00:00 2024-04-23T14:29:42.912 INFO:teuthology.orchestra.run.smithi028.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 474 kB/s | 59 kB 00:00 2024-04-23T14:29:42.937 INFO:teuthology.orchestra.run.smithi105.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3126.gd4 4.9 MB/s | 249 kB 00:00 2024-04-23T14:29:42.953 INFO:teuthology.orchestra.run.smithi038.stdout:(39/152): ceph-mgr-diskprediction-local-19.0.0- 4.2 MB/s | 7.4 MB 00:01 2024-04-23T14:29:42.980 INFO:teuthology.orchestra.run.smithi028.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 269 kB/s | 36 kB 00:00 2024-04-23T14:29:42.988 INFO:teuthology.orchestra.run.smithi105.stdout:(33/152): ceph-mgr-rook-19.0.0-3126.gd44fddd9.e 988 kB/s | 49 kB 00:00 2024-04-23T14:29:43.038 INFO:teuthology.orchestra.run.smithi105.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3126.gd 329 kB/s | 16 kB 00:00 2024-04-23T14:29:43.046 INFO:teuthology.orchestra.run.smithi028.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 384 kB/s | 51 kB 00:00 2024-04-23T14:29:43.078 INFO:teuthology.orchestra.run.smithi038.stdout:(40/152): libgfortran-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 803 kB 00:00 2024-04-23T14:29:43.097 INFO:teuthology.orchestra.run.smithi105.stdout:(35/152): ceph-volume-19.0.0-3126.gd44fddd9.el9 4.6 MB/s | 273 kB 00:00 2024-04-23T14:29:43.113 INFO:teuthology.orchestra.run.smithi028.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-04-23T14:29:43.181 INFO:teuthology.orchestra.run.smithi105.stdout:(36/152): cephadm-19.0.0-3126.gd44fddd9.el9.noa 8.9 MB/s | 762 kB 00:00 2024-04-23T14:29:43.202 INFO:teuthology.orchestra.run.smithi028.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 581 kB/s | 90 kB 00:00 2024-04-23T14:29:43.220 INFO:teuthology.orchestra.run.smithi038.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 732 kB/s | 195 kB 00:00 2024-04-23T14:29:43.222 INFO:teuthology.orchestra.run.smithi105.stdout:(37/152): cephfs-top-19.0.0-3126.gd44fddd9.el9. 440 kB/s | 18 kB 00:00 2024-04-23T14:29:43.245 INFO:teuthology.orchestra.run.smithi038.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.5 MB/s | 253 kB 00:00 2024-04-23T14:29:43.255 INFO:teuthology.orchestra.run.smithi028.stdout:(144/152): python3-werkzeug-2.0.3-3.el9.1.noarc 2.9 MB/s | 427 kB 00:00 2024-04-23T14:29:43.313 INFO:teuthology.orchestra.run.smithi038.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-04-23T14:29:43.322 INFO:teuthology.orchestra.run.smithi028.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 166 kB/s | 20 kB 00:00 2024-04-23T14:29:43.381 INFO:teuthology.orchestra.run.smithi028.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2024-04-23T14:29:43.412 INFO:teuthology.orchestra.run.smithi038.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 135 kB 00:00 2024-04-23T14:29:43.423 INFO:teuthology.orchestra.run.smithi105.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 402 kB/s | 80 kB 00:00 2024-04-23T14:29:43.488 INFO:teuthology.orchestra.run.smithi038.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-04-23T14:29:43.557 INFO:teuthology.orchestra.run.smithi028.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 6.8 MB/s | 1.6 MB 00:00 2024-04-23T14:29:43.621 INFO:teuthology.orchestra.run.smithi038.stdout:(46/152): python3-urllib3-1.26.5-5.el9.noarch.r 1.6 MB/s | 215 kB 00:00 2024-04-23T14:29:43.688 INFO:teuthology.orchestra.run.smithi038.stdout:(47/152): ceph-mgr-dashboard-19.0.0-3126.gd44fd 1.4 MB/s | 3.5 MB 00:02 2024-04-23T14:29:43.698 INFO:teuthology.orchestra.run.smithi105.stdout:(39/152): ceph-radosgw-19.0.0-3126.gd44fddd9.el 2.4 MB/s | 11 MB 00:04 2024-04-23T14:29:43.722 INFO:teuthology.orchestra.run.smithi038.stdout:(48/152): python3-cryptography-36.0.1-4.el9.x86 2.4 MB/s | 1.2 MB 00:00 2024-04-23T14:29:43.731 INFO:teuthology.orchestra.run.smithi105.stdout:(40/152): libgfortran-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 803 kB 00:00 2024-04-23T14:29:43.740 INFO:teuthology.orchestra.run.smithi028.stdout:(148/152): librados2-19.0.0-3126.gd44fddd9.el9. 9.5 MB/s | 3.4 MB 00:00 2024-04-23T14:29:43.798 INFO:teuthology.orchestra.run.smithi105.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.7 MB/s | 253 kB 00:00 2024-04-23T14:29:43.924 INFO:teuthology.orchestra.run.smithi105.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 9.8 MB/s | 1.2 MB 00:00 2024-04-23T14:29:43.932 INFO:teuthology.orchestra.run.smithi028.stdout:(149/152): librbd1-19.0.0-3126.gd44fddd9.el9.x8 8.4 MB/s | 3.1 MB 00:00 2024-04-23T14:29:43.940 INFO:teuthology.orchestra.run.smithi105.stdout:(43/152): libquadmath-11.4.1-3.el9.x86_64.rpm 806 kB/s | 195 kB 00:00 2024-04-23T14:29:43.947 INFO:teuthology.orchestra.run.smithi038.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 475 kB/s | 107 kB 00:00 2024-04-23T14:29:43.982 INFO:teuthology.orchestra.run.smithi028.stdout:(150/152): openssl-3.2.1-1.el9.x86_64.rpm 5.6 MB/s | 1.3 MB 00:00 2024-04-23T14:29:43.983 INFO:teuthology.orchestra.run.smithi105.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-04-23T14:29:44.014 INFO:teuthology.orchestra.run.smithi038.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 322 kB/s | 21 kB 00:00 2024-04-23T14:29:44.026 INFO:teuthology.orchestra.run.smithi105.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 1.6 MB/s | 135 kB 00:00 2024-04-23T14:29:44.048 INFO:teuthology.orchestra.run.smithi038.stdout:(51/152): automake-1.16.2-8.el9.noarch.rpm 1.6 MB/s | 689 kB 00:00 2024-04-23T14:29:44.050 INFO:teuthology.orchestra.run.smithi105.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 1.8 MB/s | 125 kB 00:00 2024-04-23T14:29:44.123 INFO:teuthology.orchestra.run.smithi038.stdout:(52/152): bison-3.7.4-5.el9.x86_64.rpm 2.1 MB/s | 943 kB 00:00 2024-04-23T14:29:44.125 INFO:teuthology.orchestra.run.smithi105.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 2.1 MB/s | 215 kB 00:00 2024-04-23T14:29:44.140 INFO:teuthology.orchestra.run.smithi028.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 12 MB/s | 2.4 MB 00:00 2024-04-23T14:29:44.148 INFO:teuthology.orchestra.run.smithi038.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 326 kB/s | 33 kB 00:00 2024-04-23T14:29:44.182 INFO:teuthology.orchestra.run.smithi038.stdout:(54/152): flex-2.6.4-9.el9.x86_64.rpm 1.8 MB/s | 313 kB 00:00 2024-04-23T14:29:44.207 INFO:teuthology.orchestra.run.smithi038.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 311 kB/s | 18 kB 00:00 2024-04-23T14:29:44.232 INFO:teuthology.orchestra.run.smithi038.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 221 kB/s | 11 kB 00:00 2024-04-23T14:29:44.266 INFO:teuthology.orchestra.run.smithi038.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-04-23T14:29:44.291 INFO:teuthology.orchestra.run.smithi038.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 772 kB/s | 45 kB 00:00 2024-04-23T14:29:44.383 INFO:teuthology.orchestra.run.smithi038.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 662 kB 00:00 2024-04-23T14:29:44.434 INFO:teuthology.orchestra.run.smithi038.stdout:(60/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 508 kB/s | 25 kB 00:00 2024-04-23T14:29:44.467 INFO:teuthology.orchestra.run.smithi038.stdout:(61/152): libtool-2.4.6-45.el9.x86_64.rpm 3.2 MB/s | 582 kB 00:00 2024-04-23T14:29:44.493 INFO:teuthology.orchestra.run.smithi038.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 810 kB/s | 47 kB 00:00 2024-04-23T14:29:44.560 INFO:teuthology.orchestra.run.smithi038.stdout:(63/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 6.9 MB/s | 3.0 MB 00:00 2024-04-23T14:29:44.585 INFO:teuthology.orchestra.run.smithi038.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-04-23T14:29:44.610 INFO:teuthology.orchestra.run.smithi038.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 188 kB 00:00 2024-04-23T14:29:44.627 INFO:teuthology.orchestra.run.smithi038.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 524 kB/s | 35 kB 00:00 2024-04-23T14:29:44.786 INFO:teuthology.orchestra.run.smithi038.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-04-23T14:29:44.851 INFO:teuthology.orchestra.run.smithi105.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 861 kB/s | 689 kB 00:00 2024-04-23T14:29:44.886 INFO:teuthology.orchestra.run.smithi038.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 8.9 MB/s | 911 kB 00:00 2024-04-23T14:29:44.937 INFO:teuthology.orchestra.run.smithi038.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 2.6 MB/s | 133 kB 00:00 2024-04-23T14:29:44.959 INFO:teuthology.orchestra.run.smithi105.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 991 kB/s | 107 kB 00:00 2024-04-23T14:29:44.993 INFO:teuthology.orchestra.run.smithi105.stdout:(50/152): bison-3.7.4-5.el9.x86_64.rpm 1.1 MB/s | 943 kB 00:00 2024-04-23T14:29:45.068 INFO:teuthology.orchestra.run.smithi105.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 198 kB/s | 21 kB 00:00 2024-04-23T14:29:45.110 INFO:teuthology.orchestra.run.smithi105.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 2.6 MB/s | 313 kB 00:00 2024-04-23T14:29:45.185 INFO:teuthology.orchestra.run.smithi105.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 279 kB/s | 33 kB 00:00 2024-04-23T14:29:45.195 INFO:teuthology.orchestra.run.smithi038.stdout:(70/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 7.8 MB/s | 4.4 MB 00:00 2024-04-23T14:29:45.294 INFO:teuthology.orchestra.run.smithi105.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 168 kB/s | 18 kB 00:00 2024-04-23T14:29:45.295 INFO:teuthology.orchestra.run.smithi038.stdout:(71/152): openblas-openmp-0.3.21-2.el9.x86_64.r 6.7 MB/s | 4.7 MB 00:00 2024-04-23T14:29:45.354 INFO:teuthology.orchestra.run.smithi038.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-04-23T14:29:45.402 INFO:teuthology.orchestra.run.smithi105.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 103 kB/s | 11 kB 00:00 2024-04-23T14:29:45.479 INFO:teuthology.orchestra.run.smithi038.stdout:(73/152): postgresql-server-13.14-1.el9.x86_64. 11 MB/s | 5.8 MB 00:00 2024-04-23T14:29:45.521 INFO:teuthology.orchestra.run.smithi038.stdout:(74/152): protobuf-3.14.0-13.el9.x86_64.rpm 3.1 MB/s | 1.0 MB 00:00 2024-04-23T14:29:45.527 INFO:teuthology.orchestra.run.smithi105.stdout:(56/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 160 kB 00:00 2024-04-23T14:29:45.546 INFO:teuthology.orchestra.run.smithi038.stdout:(75/152): python3-devel-3.9.18-3.el9.x86_64.rpm 3.6 MB/s | 244 kB 00:00 2024-04-23T14:29:45.580 INFO:teuthology.orchestra.run.smithi038.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 4.1 MB/s | 248 kB 00:00 2024-04-23T14:29:45.605 INFO:teuthology.orchestra.run.smithi038.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 437 kB/s | 26 kB 00:00 2024-04-23T14:29:45.628 INFO:teuthology.orchestra.run.smithi105.stdout:(57/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 5.8 MB/s | 3.0 MB 00:00 2024-04-23T14:29:45.639 INFO:teuthology.orchestra.run.smithi038.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 326 kB/s | 19 kB 00:00 2024-04-23T14:29:45.645 INFO:teuthology.orchestra.run.smithi105.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 386 kB/s | 45 kB 00:00 2024-04-23T14:29:45.666 INFO:teuthology.orchestra.run.smithi038.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-04-23T14:29:45.690 INFO:teuthology.orchestra.run.smithi038.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 691 kB/s | 35 kB 00:00 2024-04-23T14:29:45.748 INFO:teuthology.orchestra.run.smithi038.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.2 MB/s | 250 kB 00:00 2024-04-23T14:29:45.815 INFO:teuthology.orchestra.run.smithi038.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 3.3 MB/s | 222 kB 00:00 2024-04-23T14:29:45.987 INFO:teuthology.orchestra.run.smithi105.stdout:(59/152): libtool-2.4.6-45.el9.x86_64.rpm 1.7 MB/s | 582 kB 00:00 2024-04-23T14:29:46.049 INFO:teuthology.orchestra.run.smithi038.stdout:(83/152): python3-babel-2.9.1-2.el9.noarch.rpm 8.6 MB/s | 6.0 MB 00:00 2024-04-23T14:29:46.074 INFO:teuthology.orchestra.run.smithi038.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 298 kB/s | 77 kB 00:00 2024-04-23T14:29:46.087 INFO:teuthology.orchestra.run.smithi105.stdout:(60/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 255 kB/s | 25 kB 00:00 2024-04-23T14:29:46.100 INFO:teuthology.orchestra.run.smithi038.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 841 kB/s | 42 kB 00:00 2024-04-23T14:29:46.167 INFO:teuthology.orchestra.run.smithi038.stdout:(86/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 10 MB/s | 5.1 MB 00:00 2024-04-23T14:29:46.188 INFO:teuthology.orchestra.run.smithi105.stdout:(61/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 475 kB/s | 47 kB 00:00 2024-04-23T14:29:46.192 INFO:teuthology.orchestra.run.smithi038.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 2.2 MB/s | 269 kB 00:00 2024-04-23T14:29:46.217 INFO:teuthology.orchestra.run.smithi038.stdout:(88/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-04-23T14:29:46.243 INFO:teuthology.orchestra.run.smithi038.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-23T14:29:46.268 INFO:teuthology.orchestra.run.smithi038.stdout:(90/152): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 279 kB 00:00 2024-04-23T14:29:46.293 INFO:teuthology.orchestra.run.smithi038.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 711 kB/s | 54 kB 00:00 2024-04-23T14:29:46.304 INFO:teuthology.orchestra.run.smithi105.stdout:(62/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.5 MB/s | 292 kB 00:00 2024-04-23T14:29:46.343 INFO:teuthology.orchestra.run.smithi038.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 838 kB/s | 42 kB 00:00 2024-04-23T14:29:46.368 INFO:teuthology.orchestra.run.smithi038.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 589 kB/s | 44 kB 00:00 2024-04-23T14:29:46.394 INFO:teuthology.orchestra.run.smithi038.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-04-23T14:29:46.421 INFO:teuthology.orchestra.run.smithi105.stdout:(63/152): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 188 kB 00:00 2024-04-23T14:29:46.427 INFO:teuthology.orchestra.run.smithi038.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 611 kB/s | 36 kB 00:00 2024-04-23T14:29:46.453 INFO:teuthology.orchestra.run.smithi038.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 808 kB/s | 47 kB 00:00 2024-04-23T14:29:46.486 INFO:teuthology.orchestra.run.smithi038.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 305 kB 00:00 2024-04-23T14:29:46.512 INFO:teuthology.orchestra.run.smithi038.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 986 kB/s | 58 kB 00:00 2024-04-23T14:29:46.538 INFO:teuthology.orchestra.run.smithi105.stdout:(64/152): openblas-0.3.21-2.el9.x86_64.rpm 302 kB/s | 35 kB 00:00 2024-04-23T14:29:46.545 INFO:teuthology.orchestra.run.smithi038.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.6 MB/s | 94 kB 00:00 2024-04-23T14:29:46.571 INFO:teuthology.orchestra.run.smithi038.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-23T14:29:46.604 INFO:teuthology.orchestra.run.smithi038.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 768 kB/s | 45 kB 00:00 2024-04-23T14:29:46.639 INFO:teuthology.orchestra.run.smithi105.stdout:(65/152): librdkafka-1.6.1-102.el9.x86_64.rpm 656 kB/s | 662 kB 00:01 2024-04-23T14:29:46.688 INFO:teuthology.orchestra.run.smithi038.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 191 kB/s | 22 kB 00:00 2024-04-23T14:29:47.063 INFO:teuthology.orchestra.run.smithi038.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 1.8 MB/s | 863 kB 00:00 2024-04-23T14:29:47.339 INFO:teuthology.orchestra.run.smithi038.stdout:(104/152): abseil-cpp-20211102.0-3.el9.x86_64.r 843 kB/s | 548 kB 00:00 2024-04-23T14:29:47.372 INFO:teuthology.orchestra.run.smithi105.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 5.7 MB/s | 4.7 MB 00:00 2024-04-23T14:29:47.439 INFO:teuthology.orchestra.run.smithi038.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 194 kB/s | 19 kB 00:00 2024-04-23T14:29:47.572 INFO:teuthology.orchestra.run.smithi038.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 607 kB/s | 309 kB 00:00 2024-04-23T14:29:47.689 INFO:teuthology.orchestra.run.smithi038.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 223 kB/s | 26 kB 00:00 2024-04-23T14:29:47.806 INFO:teuthology.orchestra.run.smithi038.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 416 kB/s | 49 kB 00:00 2024-04-23T14:29:47.865 INFO:teuthology.orchestra.run.smithi038.stdout:(109/152): libarrow-9.0.0-10.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-04-23T14:29:47.899 INFO:teuthology.orchestra.run.smithi038.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 732 kB/s | 67 kB 00:00 2024-04-23T14:29:47.966 INFO:teuthology.orchestra.run.smithi038.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-04-23T14:29:48.083 INFO:teuthology.orchestra.run.smithi038.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 4.6 MB/s | 548 kB 00:00 2024-04-23T14:29:48.308 INFO:teuthology.orchestra.run.smithi038.stdout:(113/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 7.9 MB/s | 16 MB 00:02 2024-04-23T14:29:48.326 INFO:teuthology.orchestra.run.smithi038.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 180 kB/s | 43 kB 00:00 2024-04-23T14:29:48.358 INFO:teuthology.orchestra.run.smithi038.stdout:(115/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.8 MB/s | 838 kB 00:00 2024-04-23T14:29:48.417 INFO:teuthology.orchestra.run.smithi038.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 163 kB/s | 15 kB 00:00 2024-04-23T14:29:48.467 INFO:teuthology.orchestra.run.smithi038.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.5 MB/s | 172 kB 00:00 2024-04-23T14:29:48.543 INFO:teuthology.orchestra.run.smithi038.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 2.8 MB/s | 358 kB 00:00 2024-04-23T14:29:48.585 INFO:teuthology.orchestra.run.smithi038.stdout:(119/152): python3-cachetools-4.2.4-1.el9.noarc 117 kB/s | 32 kB 00:00 2024-04-23T14:29:48.618 INFO:teuthology.orchestra.run.smithi038.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.0 MB/s | 303 kB 00:00 2024-04-23T14:29:48.677 INFO:teuthology.orchestra.run.smithi038.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 263 kB/s | 35 kB 00:00 2024-04-23T14:29:48.802 INFO:teuthology.orchestra.run.smithi038.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-04-23T14:29:48.911 INFO:teuthology.orchestra.run.smithi038.stdout:(123/152): python3-jaraco-8.2.1-3.el9.noarch.rp 98 kB/s | 11 kB 00:00 2024-04-23T14:29:49.011 INFO:teuthology.orchestra.run.smithi038.stdout:(124/152): python3-jaraco-classes-3.2.1-5.el9.n 177 kB/s | 18 kB 00:00 2024-04-23T14:29:49.087 INFO:teuthology.orchestra.run.smithi038.stdout:(125/152): python3-google-auth-2.29.0-1.el9.noa 410 kB/s | 206 kB 00:00 2024-04-23T14:29:49.137 INFO:teuthology.orchestra.run.smithi038.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 184 kB/s | 23 kB 00:00 2024-04-23T14:29:49.287 INFO:teuthology.orchestra.run.smithi038.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 97 kB/s | 19 kB 00:00 2024-04-23T14:29:49.312 INFO:teuthology.orchestra.run.smithi038.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 112 kB/s | 20 kB 00:00 2024-04-23T14:29:49.363 INFO:teuthology.orchestra.run.smithi038.stdout:(129/152): python3-grpcio-1.46.7-10.el9.x86_64. 2.7 MB/s | 2.0 MB 00:00 2024-04-23T14:29:49.430 INFO:teuthology.orchestra.run.smithi038.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 395 kB/s | 46 kB 00:00 2024-04-23T14:29:49.439 INFO:teuthology.orchestra.run.smithi105.stdout:(67/152): ceph-test-19.0.0-3126.gd44fddd9.el9.x 5.2 MB/s | 49 MB 00:09 2024-04-23T14:29:49.463 INFO:teuthology.orchestra.run.smithi038.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 786 kB/s | 79 kB 00:00 2024-04-23T14:29:49.530 INFO:teuthology.orchestra.run.smithi038.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 575 kB/s | 58 kB 00:00 2024-04-23T14:29:49.564 INFO:teuthology.orchestra.run.smithi038.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-04-23T14:29:49.623 INFO:teuthology.orchestra.run.smithi038.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 179 kB/s | 16 kB 00:00 2024-04-23T14:29:49.656 INFO:teuthology.orchestra.run.smithi038.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 977 kB/s | 90 kB 00:00 2024-04-23T14:29:49.715 INFO:teuthology.orchestra.run.smithi038.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 307 kB/s | 28 kB 00:00 2024-04-23T14:29:49.748 INFO:teuthology.orchestra.run.smithi038.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 335 kB/s | 31 kB 00:00 2024-04-23T14:29:49.815 INFO:teuthology.orchestra.run.smithi038.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-04-23T14:29:49.841 INFO:teuthology.orchestra.run.smithi038.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 644 kB/s | 59 kB 00:00 2024-04-23T14:29:49.908 INFO:teuthology.orchestra.run.smithi038.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 389 kB/s | 36 kB 00:00 2024-04-23T14:29:49.933 INFO:teuthology.orchestra.run.smithi038.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 554 kB/s | 51 kB 00:00 2024-04-23T14:29:49.964 INFO:teuthology.orchestra.run.smithi105.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 627 kB/s | 1.6 MB 00:02 2024-04-23T14:29:50.008 INFO:teuthology.orchestra.run.smithi038.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-04-23T14:29:50.034 INFO:teuthology.orchestra.run.smithi038.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 896 kB/s | 90 kB 00:00 2024-04-23T14:29:50.109 INFO:teuthology.orchestra.run.smithi038.stdout:(144/152): python3-werkzeug-2.0.3-3.el9.1.noarc 4.2 MB/s | 427 kB 00:00 2024-04-23T14:29:50.134 INFO:teuthology.orchestra.run.smithi038.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 199 kB/s | 20 kB 00:00 2024-04-23T14:29:50.210 INFO:teuthology.orchestra.run.smithi038.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-04-23T14:29:50.231 INFO:teuthology.orchestra.run.smithi105.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 1.1 MB/s | 911 kB 00:00 2024-04-23T14:29:50.256 INFO:teuthology.orchestra.run.smithi105.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 456 kB/s | 133 kB 00:00 2024-04-23T14:29:50.257 INFO:teuthology.orchestra.run.smithi028.stdout:(152/152): ceph-test-19.0.0-3126.gd44fddd9.el9. 2.6 MB/s | 49 MB 00:18 2024-04-23T14:29:50.268 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-04-23T14:29:50.268 INFO:teuthology.orchestra.run.smithi028.stdout:Total 10 MB/s | 222 MB 00:21 2024-04-23T14:29:50.385 INFO:teuthology.orchestra.run.smithi038.stdout:(147/152): python3-kubernetes-26.1.0-2.el9.noar 953 kB/s | 1.0 MB 00:01 2024-04-23T14:29:50.485 INFO:teuthology.orchestra.run.smithi038.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 4.5 MB/s | 1.6 MB 00:00 2024-04-23T14:29:50.531 INFO:teuthology.orchestra.run.smithi105.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 3.7 MB/s | 1.0 MB 00:00 2024-04-23T14:29:50.648 INFO:teuthology.orchestra.run.smithi105.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 749 kB/s | 87 kB 00:00 2024-04-23T14:29:50.952 INFO:teuthology.orchestra.run.smithi038.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 2.9 MB/s | 1.3 MB 00:00 2024-04-23T14:29:51.002 INFO:teuthology.orchestra.run.smithi038.stdout:(150/152): librbd1-19.0.0-3126.gd44fddd9.el9.x8 5.1 MB/s | 3.1 MB 00:00 2024-04-23T14:29:51.353 INFO:teuthology.orchestra.run.smithi038.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 6.0 MB/s | 2.4 MB 00:00 2024-04-23T14:29:51.595 INFO:teuthology.orchestra.run.smithi038.stdout:(152/152): librados2-19.0.0-3126.gd44fddd9.el9. 2.5 MB/s | 3.4 MB 00:01 2024-04-23T14:29:51.598 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-04-23T14:29:51.598 INFO:teuthology.orchestra.run.smithi038.stdout:Total 11 MB/s | 222 MB 00:19 2024-04-23T14:29:52.689 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-04-23T14:29:52.846 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-04-23T14:29:52.846 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-04-23T14:29:52.965 INFO:teuthology.orchestra.run.smithi105.stdout:(73/152): python3-babel-2.9.1-2.el9.noarch.rpm 2.6 MB/s | 6.0 MB 00:02 2024-04-23T14:29:53.115 INFO:teuthology.orchestra.run.smithi105.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 1.6 MB/s | 244 kB 00:00 2024-04-23T14:29:53.194 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-04-23T14:29:53.455 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-04-23T14:29:53.455 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-04-23T14:29:54.024 INFO:teuthology.orchestra.run.smithi105.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 273 kB/s | 248 kB 00:00 2024-04-23T14:29:54.149 INFO:teuthology.orchestra.run.smithi105.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 205 kB/s | 26 kB 00:00 2024-04-23T14:29:54.275 INFO:teuthology.orchestra.run.smithi105.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 153 kB/s | 19 kB 00:00 2024-04-23T14:29:54.408 INFO:teuthology.orchestra.run.smithi105.stdout:(78/152): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-04-23T14:29:54.559 INFO:teuthology.orchestra.run.smithi105.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 231 kB/s | 35 kB 00:00 2024-04-23T14:29:54.727 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-04-23T14:29:54.727 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-04-23T14:29:56.203 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-04-23T14:29:56.204 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-04-23T14:29:56.500 INFO:teuthology.orchestra.run.smithi105.stdout:(80/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 459 kB/s | 4.4 MB 00:09 2024-04-23T14:29:56.626 INFO:teuthology.orchestra.run.smithi105.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.0 MB/s | 250 kB 00:00 2024-04-23T14:29:56.734 INFO:teuthology.orchestra.run.smithi105.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.0 MB/s | 222 kB 00:00 2024-04-23T14:29:56.851 INFO:teuthology.orchestra.run.smithi105.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 662 kB/s | 77 kB 00:00 2024-04-23T14:29:56.968 INFO:teuthology.orchestra.run.smithi105.stdout:(84/152): python3-prettytable-0.7.2-27.el9.noar 362 kB/s | 42 kB 00:00 2024-04-23T14:29:57.093 INFO:teuthology.orchestra.run.smithi105.stdout:(85/152): python3-protobuf-3.14.0-13.el9.noarch 2.1 MB/s | 269 kB 00:00 2024-04-23T14:29:57.219 INFO:teuthology.orchestra.run.smithi105.stdout:(86/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-04-23T14:29:57.344 INFO:teuthology.orchestra.run.smithi105.stdout:(87/152): python3-pyasn1-modules-0.4.8-6.el9.no 2.2 MB/s | 279 kB 00:00 2024-04-23T14:29:57.477 INFO:teuthology.orchestra.run.smithi105.stdout:(88/152): python3-pytz-2021.1-5.el9.noarch.rpm 383 kB/s | 51 kB 00:00 2024-04-23T14:29:57.603 INFO:teuthology.orchestra.run.smithi105.stdout:(89/152): python3-requests-oauthlib-1.3.0-12.el 429 kB/s | 54 kB 00:00 2024-04-23T14:29:57.670 INFO:teuthology.orchestra.run.smithi105.stdout:(90/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 1.6 MB/s | 5.1 MB 00:03 2024-04-23T14:29:57.786 INFO:teuthology.orchestra.run.smithi105.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 357 kB/s | 42 kB 00:00 2024-04-23T14:29:57.904 INFO:teuthology.orchestra.run.smithi105.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 377 kB/s | 44 kB 00:00 2024-04-23T14:29:58.029 INFO:teuthology.orchestra.run.smithi105.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 1.7 MB/s | 221 kB 00:00 2024-04-23T14:29:58.171 INFO:teuthology.orchestra.run.smithi105.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 253 kB/s | 36 kB 00:00 2024-04-23T14:29:58.296 INFO:teuthology.orchestra.run.smithi105.stdout:(95/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 380 kB/s | 47 kB 00:00 2024-04-23T14:29:58.430 INFO:teuthology.orchestra.run.smithi105.stdout:(96/152): socat-1.7.4.1-5.el9.x86_64.rpm 2.2 MB/s | 305 kB 00:00 2024-04-23T14:29:58.547 INFO:teuthology.orchestra.run.smithi105.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 496 kB/s | 58 kB 00:00 2024-04-23T14:29:58.697 INFO:teuthology.orchestra.run.smithi105.stdout:(98/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 624 kB/s | 94 kB 00:00 2024-04-23T14:29:58.831 INFO:teuthology.orchestra.run.smithi105.stdout:(99/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 478 kB/s | 64 kB 00:00 2024-04-23T14:29:58.964 INFO:teuthology.orchestra.run.smithi105.stdout:(100/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 336 kB/s | 45 kB 00:00 2024-04-23T14:29:59.090 INFO:teuthology.orchestra.run.smithi105.stdout:(101/152): lua-devel-5.4.4-4.el9.x86_64.rpm 178 kB/s | 22 kB 00:00 2024-04-23T14:29:59.432 INFO:teuthology.orchestra.run.smithi105.stdout:(102/152): protobuf-compiler-3.14.0-13.el9.x86_ 2.5 MB/s | 863 kB 00:00 2024-04-23T14:29:59.607 INFO:teuthology.orchestra.run.smithi105.stdout:(103/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 8.1 MB/s | 16 MB 00:02 2024-04-23T14:29:59.824 INFO:teuthology.orchestra.run.smithi105.stdout:(104/152): postgresql-server-13.14-1.el9.x86_64 617 kB/s | 5.8 MB 00:09 2024-04-23T14:29:59.849 INFO:teuthology.orchestra.run.smithi105.stdout:(105/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.3 MB/s | 548 kB 00:00 2024-04-23T14:29:59.874 INFO:teuthology.orchestra.run.smithi105.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 386 kB/s | 19 kB 00:00 2024-04-23T14:29:59.916 INFO:teuthology.orchestra.run.smithi105.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 623 kB/s | 26 kB 00:00 2024-04-23T14:29:59.959 INFO:teuthology.orchestra.run.smithi105.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-04-23T14:30:00.000 INFO:teuthology.orchestra.run.smithi105.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-04-23T14:30:00.026 INFO:teuthology.orchestra.run.smithi105.stdout:(110/152): gperftools-libs-2.9.1-2.el9.x86_64.r 738 kB/s | 309 kB 00:00 2024-04-23T14:30:00.051 INFO:teuthology.orchestra.run.smithi105.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 2.9 MB/s | 151 kB 00:00 2024-04-23T14:30:00.176 INFO:teuthology.orchestra.run.smithi105.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 4.3 MB/s | 548 kB 00:00 2024-04-23T14:30:00.210 INFO:teuthology.orchestra.run.smithi105.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.3 MB/s | 43 kB 00:00 2024-04-23T14:30:00.243 INFO:teuthology.orchestra.run.smithi105.stdout:(114/152): python3-cachetools-4.2.4-1.el9.noarc 970 kB/s | 32 kB 00:00 2024-04-23T14:30:00.285 INFO:teuthology.orchestra.run.smithi105.stdout:(115/152): python3-certifi-2021.10.8-2.el9.noar 359 kB/s | 15 kB 00:00 2024-04-23T14:30:00.344 INFO:teuthology.orchestra.run.smithi105.stdout:(116/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-04-23T14:30:00.429 INFO:teuthology.orchestra.run.smithi105.stdout:(117/152): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-04-23T14:30:00.653 INFO:teuthology.orchestra.run.smithi105.stdout:(118/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.3 MB/s | 303 kB 00:00 2024-04-23T14:30:00.687 INFO:teuthology.orchestra.run.smithi105.stdout:(119/152): python3-colorama-0.4.4-12.el9.noarch 1.0 MB/s | 35 kB 00:00 2024-04-23T14:30:00.720 INFO:teuthology.orchestra.run.smithi105.stdout:(120/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.2 MB/s | 838 kB 00:00 2024-04-23T14:30:00.762 INFO:teuthology.orchestra.run.smithi105.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 2.7 MB/s | 206 kB 00:00 2024-04-23T14:30:00.821 INFO:teuthology.orchestra.run.smithi105.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 2.4 MB/s | 144 kB 00:00 2024-04-23T14:30:00.842 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-04-23T14:30:00.862 INFO:teuthology.orchestra.run.smithi105.stdout:(123/152): python3-jaraco-8.2.1-3.el9.noarch.rp 256 kB/s | 11 kB 00:00 2024-04-23T14:30:00.905 INFO:teuthology.orchestra.run.smithi105.stdout:(124/152): python3-jaraco-classes-3.2.1-5.el9.n 424 kB/s | 18 kB 00:00 2024-04-23T14:30:00.924 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-23T14:30:00.946 INFO:teuthology.orchestra.run.smithi105.stdout:(125/152): python3-jaraco-collections-3.0.0-8.e 556 kB/s | 23 kB 00:00 2024-04-23T14:30:00.963 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-23T14:30:00.988 INFO:teuthology.orchestra.run.smithi105.stdout:(126/152): python3-jaraco-functools-3.5.0-2.el9 467 kB/s | 19 kB 00:00 2024-04-23T14:30:01.017 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-23T14:30:01.039 INFO:teuthology.orchestra.run.smithi105.stdout:(127/152): python3-jaraco-text-3.2.0-6.el9.noar 393 kB/s | 20 kB 00:00 2024-04-23T14:30:01.067 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-23T14:30:01.092 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-23T14:30:01.136 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-23T14:30:01.287 INFO:teuthology.orchestra.run.smithi105.stdout:(128/152): python3-kubernetes-26.1.0-2.el9.noar 4.1 MB/s | 1.0 MB 00:00 2024-04-23T14:30:01.331 INFO:teuthology.orchestra.run.smithi105.stdout:(129/152): python3-logutils-0.3.5-21.el9.noarch 1.0 MB/s | 46 kB 00:00 2024-04-23T14:30:01.373 INFO:teuthology.orchestra.run.smithi105.stdout:(130/152): python3-more-itertools-8.12.0-2.el9. 1.9 MB/s | 79 kB 00:00 2024-04-23T14:30:01.415 INFO:teuthology.orchestra.run.smithi105.stdout:(131/152): python3-natsort-7.1.1-5.el9.noarch.r 1.4 MB/s | 58 kB 00:00 2024-04-23T14:30:01.481 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-23T14:30:01.499 INFO:teuthology.orchestra.run.smithi105.stdout:(132/152): python3-grpcio-1.46.7-10.el9.x86_64. 2.6 MB/s | 2.0 MB 00:00 2024-04-23T14:30:01.524 INFO:teuthology.orchestra.run.smithi105.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.5 MB/s | 272 kB 00:00 2024-04-23T14:30:01.549 INFO:teuthology.orchestra.run.smithi105.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 327 kB/s | 16 kB 00:00 2024-04-23T14:30:01.600 INFO:teuthology.orchestra.run.smithi105.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-04-23T14:30:01.601 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librados2-2:19.0.0-3126.gd44fddd9.el9.x86_64 8/156 2024-04-23T14:30:01.625 INFO:teuthology.orchestra.run.smithi105.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 375 kB/s | 28 kB 00:00 2024-04-23T14:30:01.651 INFO:teuthology.orchestra.run.smithi105.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 613 kB/s | 31 kB 00:00 2024-04-23T14:30:01.692 INFO:teuthology.orchestra.run.smithi105.stdout:(138/152): python3-rsa-4.9-2.el9.noarch.rpm 1.4 MB/s | 59 kB 00:00 2024-04-23T14:30:01.718 INFO:teuthology.orchestra.run.smithi105.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 2.0 MB/s | 188 kB 00:00 2024-04-23T14:30:01.726 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-2:19.0.0-3126.gd44fddd9.el9.x86_64 8/156 2024-04-23T14:30:01.743 INFO:teuthology.orchestra.run.smithi105.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 714 kB/s | 36 kB 00:00 2024-04-23T14:30:01.761 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs2-2:19.0.0-3126.gd44fddd9.el9.x86_64 9/156 2024-04-23T14:30:01.776 INFO:teuthology.orchestra.run.smithi105.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 874 kB/s | 51 kB 00:00 2024-04-23T14:30:01.818 INFO:teuthology.orchestra.run.smithi105.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-04-23T14:30:01.844 INFO:teuthology.orchestra.run.smithi105.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.3 MB/s | 90 kB 00:00 2024-04-23T14:30:01.894 INFO:teuthology.orchestra.run.smithi105.stdout:(144/152): python3-zc-lockfile-2.0-10.el9.noarc 400 kB/s | 20 kB 00:00 2024-04-23T14:30:01.936 INFO:teuthology.orchestra.run.smithi105.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-04-23T14:30:01.970 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephfs2-2:19.0.0-3126.gd44fddd9.el9.x86_64 9/156 2024-04-23T14:30:01.986 INFO:teuthology.orchestra.run.smithi105.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-04-23T14:30:02.007 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rados-2:19.0.0-3126.gd44fddd9.el9.x86_64 10/156 2024-04-23T14:30:02.164 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-23T14:30:02.195 INFO:teuthology.orchestra.run.smithi105.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 6.1 MB/s | 1.6 MB 00:00 2024-04-23T14:30:02.213 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-23T14:30:02.240 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephsqlite-2:19.0.0-3126.gd44fddd9.el9.x86_64 13/156 2024-04-23T14:30:02.253 INFO:teuthology.orchestra.run.smithi105.stdout:(148/152): libarrow-9.0.0-10.el9.x86_64.rpm 1.8 MB/s | 4.4 MB 00:02 2024-04-23T14:30:02.345 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3126.gd44fddd9.el9.x86_64 13/156 2024-04-23T14:30:02.354 INFO:teuthology.orchestra.run.smithi105.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 14 MB/s | 1.3 MB 00:00 2024-04-23T14:30:02.375 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libradosstriper1-2:19.0.0-3126.gd44fddd9.el9.x86 14/156 2024-04-23T14:30:02.429 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3126.gd44fddd9.el9.x86 14/156 2024-04-23T14:30:02.451 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-23T14:30:02.479 INFO:teuthology.orchestra.run.smithi105.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 19 MB/s | 2.4 MB 00:00 2024-04-23T14:30:02.485 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-23T14:30:02.670 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-23T14:30:02.702 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-23T14:30:02.854 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-23T14:30:02.949 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-23T14:30:02.987 INFO:teuthology.orchestra.run.smithi105.stdout:(151/152): librbd1-19.0.0-3126.gd44fddd9.el9.x8 4.0 MB/s | 3.1 MB 00:00 2024-04-23T14:30:02.992 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-23T14:30:03.027 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-23T14:30:03.039 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-04-23T14:30:03.113 INFO:teuthology.orchestra.run.smithi105.stdout:(152/152): librados2-19.0.0-3126.gd44fddd9.el9. 3.0 MB/s | 3.4 MB 00:01 2024-04-23T14:30:03.115 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-23T14:30:03.116 INFO:teuthology.orchestra.run.smithi105.stdout:Total 8.1 MB/s | 222 MB 00:27 2024-04-23T14:30:03.140 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-23T14:30:03.217 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-23T14:30:03.219 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-23T14:30:03.266 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-23T14:30:03.312 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-23T14:30:03.345 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-23T14:30:03.356 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-23T14:30:03.373 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-23T14:30:03.425 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-23T14:30:03.444 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-23T14:30:03.459 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-23T14:30:03.504 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-23T14:30:03.535 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-23T14:30:03.578 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-23T14:30:03.594 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-23T14:30:03.655 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-23T14:30:03.688 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-common-2:19.0.0-3126.gd44fddd9.el9. 31/156 2024-04-23T14:30:03.732 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-argparse-2:19.0.0-3126.gd44fddd9.el 32/156 2024-04-23T14:30:03.760 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cephfs-2:19.0.0-3126.gd44fddd9.el9.x86_6 33/156 2024-04-23T14:30:03.832 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-23T14:30:03.881 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-23T14:30:03.961 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-23T14:30:04.106 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-23T14:30:04.159 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-23T14:30:04.162 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-23T14:30:04.195 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-04-23T14:30:04.264 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librados2-2:19.0.0-3126.gd44fddd9.el9.x86_64 8/156 2024-04-23T14:30:04.334 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-04-23T14:30:04.334 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-04-23T14:30:04.462 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:19.0.0-3126.gd44fddd9.el9.x86_64 8/156 2024-04-23T14:30:04.490 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs2-2:19.0.0-3126.gd44fddd9.el9.x86_64 9/156 2024-04-23T14:30:04.623 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephfs2-2:19.0.0-3126.gd44fddd9.el9.x86_64 9/156 2024-04-23T14:30:04.665 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rados-2:19.0.0-3126.gd44fddd9.el9.x86_64 10/156 2024-04-23T14:30:04.712 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-23T14:30:04.809 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-23T14:30:04.845 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-23T14:30:04.888 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-23T14:30:04.926 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librados-devel-2:19.0.0-3126.gd44fddd9.el9.x86_6 43/156 2024-04-23T14:30:04.968 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-23T14:30:05.020 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-23T14:30:05.077 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-23T14:30:05.102 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephsqlite-2:19.0.0-3126.gd44fddd9.el9.x86_64 13/156 2024-04-23T14:30:05.128 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-23T14:30:05.188 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-23T14:30:05.243 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-23T14:30:05.253 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3126.gd44fddd9.el9.x86_64 13/156 2024-04-23T14:30:05.278 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libradosstriper1-2:19.0.0-3126.gd44fddd9.el9.x86 14/156 2024-04-23T14:30:05.317 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-23T14:30:05.346 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-23T14:30:05.365 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3126.gd44fddd9.el9.x86 14/156 2024-04-23T14:30:05.383 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-23T14:30:05.395 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-23T14:30:05.439 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-23T14:30:05.474 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-23T14:30:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-23T14:30:05.518 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-23T14:30:05.560 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-23T14:30:05.701 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-23T14:30:05.749 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-23T14:30:05.779 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-23T14:30:05.821 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-23T14:30:05.833 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-23T14:30:05.890 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-23T14:30:05.918 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-23T14:30:05.954 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-23T14:30:05.995 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-23T14:30:06.028 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-23T14:30:06.053 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-23T14:30:06.080 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-04-23T14:30:06.081 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-04-23T14:30:06.090 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-23T14:30:06.122 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-23T14:30:06.212 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-23T14:30:06.251 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-23T14:30:06.255 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-23T14:30:06.295 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-23T14:30:06.304 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-23T14:30:06.412 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-23T14:30:06.439 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-23T14:30:06.474 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-23T14:30:06.476 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-23T14:30:06.513 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-23T14:30:06.541 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-23T14:30:06.553 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-23T14:30:06.583 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-23T14:30:06.590 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-23T14:30:06.626 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-23T14:30:06.628 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-23T14:30:06.659 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-23T14:30:06.701 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-23T14:30:06.743 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-23T14:30:06.754 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-23T14:30:06.818 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-23T14:30:06.834 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-23T14:30:06.857 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-23T14:30:06.883 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-23T14:30:06.913 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-23T14:30:06.913 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-23T14:30:06.961 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-23T14:30:07.027 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-23T14:30:07.061 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-common-2:19.0.0-3126.gd44fddd9.el9. 31/156 2024-04-23T14:30:07.072 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-23T14:30:07.106 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-argparse-2:19.0.0-3126.gd44fddd9.el 32/156 2024-04-23T14:30:07.124 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-23T14:30:07.159 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cephfs-2:19.0.0-3126.gd44fddd9.el9.x86_6 33/156 2024-04-23T14:30:07.286 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-23T14:30:07.357 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-23T14:30:07.472 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-23T14:30:07.665 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-23T14:30:07.733 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-23T14:30:07.882 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-23T14:30:07.955 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-23T14:30:07.988 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-23T14:30:08.092 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-23T14:30:08.147 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-23T14:30:08.237 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-23T14:30:08.287 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-23T14:30:08.470 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-23T14:30:08.477 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-23T14:30:08.617 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-23T14:30:08.622 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-23T14:30:08.657 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-23T14:30:08.703 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-23T14:30:08.711 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-23T14:30:08.745 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librados-devel-2:19.0.0-3126.gd44fddd9.el9.x86_6 43/156 2024-04-23T14:30:08.831 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-23T14:30:09.082 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-23T14:30:09.144 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-23T14:30:09.224 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-23T14:30:09.329 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-23T14:30:09.369 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-23T14:30:09.385 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-23T14:30:09.408 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-23T14:30:09.476 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-23T14:30:09.507 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-23T14:30:09.524 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-23T14:30:09.589 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-23T14:30:09.655 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-23T14:30:09.868 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-23T14:30:09.962 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-23T14:30:10.002 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-23T14:30:10.061 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-23T14:30:10.206 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-23T14:30:10.259 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-23T14:30:10.309 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-23T14:30:10.350 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-23T14:30:10.382 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-23T14:30:10.584 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-23T14:30:10.794 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-23T14:30:10.794 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-23T14:30:10.843 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-23T14:30:10.879 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-23T14:30:10.895 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-23T14:30:10.922 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-23T14:30:10.939 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-23T14:30:10.955 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-23T14:30:11.005 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-23T14:30:11.061 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-23T14:30:11.110 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-23T14:30:11.156 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-23T14:30:11.205 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-23T14:30:11.281 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-23T14:30:11.284 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-23T14:30:11.327 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librbd1-2:19.0.0-3126.gd44fddd9.el9.x86_64 99/156 2024-04-23T14:30:11.328 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-23T14:30:11.418 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-2:19.0.0-3126.gd44fddd9.el9.x86_64 99/156 2024-04-23T14:30:11.449 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rbd-2:19.0.0-3126.gd44fddd9.el9.x86_64 100/156 2024-04-23T14:30:11.485 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-23T14:30:11.496 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-23T14:30:11.559 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-23T14:30:11.611 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-23T14:30:11.686 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-23T14:30:11.848 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-23T14:30:11.925 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-23T14:30:11.992 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-23T14:30:12.022 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librgw2-2:19.0.0-3126.gd44fddd9.el9.x86_64 103/156 2024-04-23T14:30:12.093 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librgw2-2:19.0.0-3126.gd44fddd9.el9.x86_64 103/156 2024-04-23T14:30:12.123 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rgw-2:19.0.0-3126.gd44fddd9.el9.x86_64 104/156 2024-04-23T14:30:12.247 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-04-23T14:30:12.342 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-23T14:30:12.379 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-23T14:30:12.433 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-23T14:30:12.467 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-23T14:30:12.501 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-23T14:30:12.543 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-23T14:30:12.866 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-23T14:30:12.933 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-23T14:30:12.985 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : librados2-2:19.0.0-3126.gd44fddd9.el9.x86_64 8/156 2024-04-23T14:30:12.990 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-23T14:30:13.024 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-23T14:30:13.107 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados2-2:19.0.0-3126.gd44fddd9.el9.x86_64 8/156 2024-04-23T14:30:13.136 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephfs2-2:19.0.0-3126.gd44fddd9.el9.x86_64 9/156 2024-04-23T14:30:13.157 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-23T14:30:13.220 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-23T14:30:13.227 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephfs2-2:19.0.0-3126.gd44fddd9.el9.x86_64 9/156 2024-04-23T14:30:13.257 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rados-2:19.0.0-3126.gd44fddd9.el9.x86_64 10/156 2024-04-23T14:30:13.331 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-23T14:30:13.392 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-23T14:30:13.406 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-23T14:30:13.463 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-23T14:30:13.490 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephsqlite-2:19.0.0-3126.gd44fddd9.el9.x86_64 13/156 2024-04-23T14:30:13.594 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3126.gd44fddd9.el9.x86_64 13/156 2024-04-23T14:30:13.617 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libradosstriper1-2:19.0.0-3126.gd44fddd9.el9.x86 14/156 2024-04-23T14:30:13.641 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-23T14:30:13.672 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-23T14:30:13.679 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3126.gd44fddd9.el9.x86 14/156 2024-04-23T14:30:13.701 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-23T14:30:13.721 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-23T14:30:13.774 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-23T14:30:13.917 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-23T14:30:13.950 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-23T14:30:14.057 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-23T14:30:14.152 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-23T14:30:14.185 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-23T14:30:14.217 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-23T14:30:14.282 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-23T14:30:14.325 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-23T14:30:14.368 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-23T14:30:14.401 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-23T14:30:14.429 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-23T14:30:14.497 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-23T14:30:14.505 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:19.0.0-3126.gd44fddd9.el9.x86_64 105/156 2024-04-23T14:30:14.546 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-common-2:19.0.0-3126.gd44fddd9.el9.x86_64 105/156 2024-04-23T14:30:14.551 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-23T14:30:14.582 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-23T14:30:14.617 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-23T14:30:14.666 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-23T14:30:14.693 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-23T14:30:14.728 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ceph-common-2:19.0.0-3126.gd44fddd9.el9. 31/156 2024-04-23T14:30:14.771 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ceph-argparse-2:19.0.0-3126.gd44fddd9.el 32/156 2024-04-23T14:30:14.802 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cephfs-2:19.0.0-3126.gd44fddd9.el9.x86_6 33/156 2024-04-23T14:30:14.810 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-23T14:30:14.869 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-23T14:30:14.914 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-23T14:30:14.989 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-23T14:30:15.115 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:19.0.0-3126.gd44fddd9.el9.x86_64 105/156 2024-04-23T14:30:15.129 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-23T14:30:15.158 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-base-2:19.0.0-3126.gd44fddd9.el9.x86_64 106/156 2024-04-23T14:30:15.184 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-23T14:30:15.249 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-base-2:19.0.0-3126.gd44fddd9.el9.x86_64 106/156 2024-04-23T14:30:15.250 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-23T14:30:15.250 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-23T14:30:15.250 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-23T14:30:15.284 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-selinux-2:19.0.0-3126.gd44fddd9.el9.x86_64 107/156 2024-04-23T14:30:15.711 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-23T14:30:15.787 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-23T14:30:15.816 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-23T14:30:15.851 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-23T14:30:15.881 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librados-devel-2:19.0.0-3126.gd44fddd9.el9.x86_6 43/156 2024-04-23T14:30:15.922 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-23T14:30:16.078 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-23T14:30:16.125 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-23T14:30:16.180 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-23T14:30:16.255 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-23T14:30:16.284 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-23T14:30:16.321 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-23T14:30:16.368 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-23T14:30:16.403 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-23T14:30:16.447 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-23T14:30:16.488 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-23T14:30:16.549 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-23T14:30:16.624 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-23T14:30:16.645 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-23T14:30:16.669 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-23T14:30:16.688 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-23T14:30:16.699 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-23T14:30:16.729 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-23T14:30:16.740 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-23T14:30:16.834 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-23T14:30:16.865 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-23T14:30:16.897 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-23T14:30:16.931 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-23T14:30:16.956 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-23T14:30:17.032 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-23T14:30:17.123 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-23T14:30:17.158 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-23T14:30:17.165 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librbd1-2:19.0.0-3126.gd44fddd9.el9.x86_64 99/156 2024-04-23T14:30:17.188 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-23T14:30:17.216 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-23T14:30:17.250 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-23T14:30:17.288 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:19.0.0-3126.gd44fddd9.el9.x86_64 99/156 2024-04-23T14:30:17.289 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-23T14:30:17.321 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-23T14:30:17.322 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rbd-2:19.0.0-3126.gd44fddd9.el9.x86_64 100/156 2024-04-23T14:30:17.351 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-23T14:30:17.377 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-23T14:30:17.380 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-23T14:30:17.403 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-23T14:30:17.444 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-23T14:30:17.478 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-23T14:30:17.564 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-23T14:30:17.610 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-23T14:30:17.639 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-23T14:30:17.687 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-23T14:30:17.785 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-23T14:30:17.833 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-23T14:30:17.982 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-23T14:30:18.018 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librgw2-2:19.0.0-3126.gd44fddd9.el9.x86_64 103/156 2024-04-23T14:30:18.120 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librgw2-2:19.0.0-3126.gd44fddd9.el9.x86_64 103/156 2024-04-23T14:30:18.146 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rgw-2:19.0.0-3126.gd44fddd9.el9.x86_64 104/156 2024-04-23T14:30:18.545 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-23T14:30:18.596 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-23T14:30:18.630 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-23T14:30:18.731 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-23T14:30:18.788 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-23T14:30:18.876 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-23T14:30:18.916 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-23T14:30:19.092 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-23T14:30:19.121 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-23T14:30:19.199 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-23T14:30:19.846 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-23T14:30:19.953 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-23T14:30:20.934 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:19.0.0-3126.gd44fddd9.el9.x86_64 105/156 2024-04-23T14:30:20.975 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-common-2:19.0.0-3126.gd44fddd9.el9.x86_64 105/156 2024-04-23T14:30:21.168 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-23T14:30:21.250 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-23T14:30:21.303 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-23T14:30:21.328 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-23T14:30:21.630 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-23T14:30:21.666 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:19.0.0-3126.gd44fddd9.el9.x86_64 105/156 2024-04-23T14:30:21.674 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : librbd1-2:19.0.0-3126.gd44fddd9.el9.x86_64 99/156 2024-04-23T14:30:21.720 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-base-2:19.0.0-3126.gd44fddd9.el9.x86_64 106/156 2024-04-23T14:30:21.764 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librbd1-2:19.0.0-3126.gd44fddd9.el9.x86_64 99/156 2024-04-23T14:30:21.796 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rbd-2:19.0.0-3126.gd44fddd9.el9.x86_64 100/156 2024-04-23T14:30:21.831 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-23T14:30:21.878 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-base-2:19.0.0-3126.gd44fddd9.el9.x86_64 106/156 2024-04-23T14:30:21.878 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-23T14:30:21.879 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-23T14:30:21.879 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:30:21.954 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-selinux-2:19.0.0-3126.gd44fddd9.el9.x86_64 107/156 2024-04-23T14:30:22.308 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-23T14:30:22.335 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librgw2-2:19.0.0-3126.gd44fddd9.el9.x86_64 103/156 2024-04-23T14:30:22.398 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librgw2-2:19.0.0-3126.gd44fddd9.el9.x86_64 103/156 2024-04-23T14:30:22.429 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rgw-2:19.0.0-3126.gd44fddd9.el9.x86_64 104/156 2024-04-23T14:30:24.660 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-common-2:19.0.0-3126.gd44fddd9.el9.x86_64 105/156 2024-04-23T14:30:24.700 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-common-2:19.0.0-3126.gd44fddd9.el9.x86_64 105/156 2024-04-23T14:30:25.236 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-common-2:19.0.0-3126.gd44fddd9.el9.x86_64 105/156 2024-04-23T14:30:25.287 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-base-2:19.0.0-3126.gd44fddd9.el9.x86_64 106/156 2024-04-23T14:30:25.377 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-base-2:19.0.0-3126.gd44fddd9.el9.x86_64 106/156 2024-04-23T14:30:25.377 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-23T14:30:25.377 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-23T14:30:25.377 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-23T14:30:25.405 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-selinux-2:19.0.0-3126.gd44fddd9.el9.x86_64 107/156 2024-04-23T14:30:27.217 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3126.gd44fddd9.el9.x86_64 107/156 2024-04-23T14:30:27.217 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /sys 2024-04-23T14:30:27.218 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /proc 2024-04-23T14:30:27.218 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /mnt 2024-04-23T14:30:27.218 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /var/tmp 2024-04-23T14:30:27.218 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /home 2024-04-23T14:30:27.218 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /root 2024-04-23T14:30:27.218 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /tmp 2024-04-23T14:30:27.218 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-23T14:30:27.400 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-volume-2:19.0.0-3126.gd44fddd9.el9.noarch 108/156 2024-04-23T14:30:27.417 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-volume-2:19.0.0-3126.gd44fddd9.el9.noarch 108/156 2024-04-23T14:30:27.417 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:30:27.417 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-23T14:30:27.417 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-23T14:30:28.962 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-osd-2:19.0.0-3126.gd44fddd9.el9.x86_64 109/156 2024-04-23T14:30:28.989 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-osd-2:19.0.0-3126.gd44fddd9.el9.x86_64 109/156 2024-04-23T14:30:28.989 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:30:28.989 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-23T14:30:28.989 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T14:30:28.989 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T14:30:28.990 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-23T14:30:29.230 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mds-2:19.0.0-3126.gd44fddd9.el9.x86_64 110/156 2024-04-23T14:30:29.253 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mds-2:19.0.0-3126.gd44fddd9.el9.x86_64 110/156 2024-04-23T14:30:29.253 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:30:29.254 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-23T14:30:29.254 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T14:30:29.254 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T14:30:29.254 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-23T14:30:29.713 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mon-2:19.0.0-3126.gd44fddd9.el9.x86_64 111/156 2024-04-23T14:30:29.737 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mon-2:19.0.0-3126.gd44fddd9.el9.x86_64 111/156 2024-04-23T14:30:29.738 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:30:29.738 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-23T14:30:29.738 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T14:30:29.738 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T14:30:29.738 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-23T14:30:29.844 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-23T14:30:29.902 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-23T14:30:29.967 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-23T14:30:30.162 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-23T14:30:30.207 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-23T14:30:30.288 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-23T14:30:30.385 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-23T14:30:30.527 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-23T14:30:30.566 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-23T14:30:30.597 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-23T14:30:30.625 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-23T14:30:31.098 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephadm-2:19.0.0-3126.gd44fddd9.el9.noarch 123/156 2024-04-23T14:30:31.199 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephadm-2:19.0.0-3126.gd44fddd9.el9.noarch 123/156 2024-04-23T14:30:31.232 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3126.gd44fddd9.e 124/156 2024-04-23T14:30:31.305 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3126.gd44fddd9. 125/156 2024-04-23T14:30:31.331 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3126.gd44fddd9.el9.noa 126/156 2024-04-23T14:30:32.426 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3126.gd44fddd9.el9.noa 126/156 2024-04-23T14:30:32.466 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3126.gd44fddd9.el9.n 127/156 2024-04-23T14:30:33.481 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3126.gd44fddd9.el9.n 127/156 2024-04-23T14:30:33.508 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3126.gd44 128/156 2024-04-23T14:30:33.615 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3126.gd44 128/156 2024-04-23T14:30:33.787 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3126.gd44fddd9.el 129/156 2024-04-23T14:30:33.818 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-2:19.0.0-3126.gd44fddd9.el9.x86_64 130/156 2024-04-23T14:30:33.842 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3126.gd44fddd9.el9.x86_64 130/156 2024-04-23T14:30:33.842 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:30:33.842 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-23T14:30:33.842 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T14:30:33.843 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T14:30:33.843 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-23T14:30:33.894 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-rook-2:19.0.0-3126.gd44fddd9.el9.noarch 131/156 2024-04-23T14:30:33.905 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3126.gd44fddd9.el9.noarch 131/156 2024-04-23T14:30:34.007 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-2:19.0.0-3126.gd44fddd9.el9.x86_64 132/156 2024-04-23T14:30:34.052 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-fuse-2:19.0.0-3126.gd44fddd9.el9.x86_64 133/156 2024-04-23T14:30:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-23T14:30:35.055 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-radosgw-2:19.0.0-3126.gd44fddd9.el9.x86_64 135/156 2024-04-23T14:30:35.079 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3126.gd44fddd9.el9.x86_64 135/156 2024-04-23T14:30:35.080 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:30:35.080 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-23T14:30:35.080 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T14:30:35.080 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T14:30:35.080 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-23T14:30:35.131 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3126.gd44fd 136/156 2024-04-23T14:30:35.154 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3126.gd44fd 136/156 2024-04-23T14:30:35.154 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:30:35.154 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-23T14:30:35.154 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-23T14:30:35.164 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3126.gd44fddd9.el9.x86_64 107/156 2024-04-23T14:30:35.164 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /sys 2024-04-23T14:30:35.164 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /proc 2024-04-23T14:30:35.164 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /mnt 2024-04-23T14:30:35.164 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /var/tmp 2024-04-23T14:30:35.164 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /home 2024-04-23T14:30:35.164 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /root 2024-04-23T14:30:35.164 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /tmp 2024-04-23T14:30:35.165 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:30:35.199 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephfs-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 137/156 2024-04-23T14:30:35.222 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 137/156 2024-04-23T14:30:35.222 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:30:35.223 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-23T14:30:35.223 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-23T14:30:35.223 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-23T14:30:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-23T14:30:35.427 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-volume-2:19.0.0-3126.gd44fddd9.el9.noarch 108/156 2024-04-23T14:30:35.475 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-volume-2:19.0.0-3126.gd44fddd9.el9.noarch 108/156 2024-04-23T14:30:35.475 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:30:35.476 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-23T14:30:35.476 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:30:35.541 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 138/156 2024-04-23T14:30:35.564 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 138/156 2024-04-23T14:30:35.565 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:30:35.565 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-23T14:30:35.565 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T14:30:35.565 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T14:30:35.565 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-23T14:30:37.018 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3126.gd44fddd9.el9.x86_64 107/156 2024-04-23T14:30:37.019 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /sys 2024-04-23T14:30:37.019 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /proc 2024-04-23T14:30:37.019 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /mnt 2024-04-23T14:30:37.019 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /var/tmp 2024-04-23T14:30:37.019 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /home 2024-04-23T14:30:37.019 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /root 2024-04-23T14:30:37.019 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /tmp 2024-04-23T14:30:37.019 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-23T14:30:37.179 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-osd-2:19.0.0-3126.gd44fddd9.el9.x86_64 109/156 2024-04-23T14:30:37.183 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-volume-2:19.0.0-3126.gd44fddd9.el9.noarch 108/156 2024-04-23T14:30:37.198 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-volume-2:19.0.0-3126.gd44fddd9.el9.noarch 108/156 2024-04-23T14:30:37.198 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:30:37.198 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-23T14:30:37.199 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-23T14:30:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-osd-2:19.0.0-3126.gd44fddd9.el9.x86_64 109/156 2024-04-23T14:30:37.262 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:30:37.262 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-23T14:30:37.263 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T14:30:37.263 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T14:30:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:30:37.529 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mds-2:19.0.0-3126.gd44fddd9.el9.x86_64 110/156 2024-04-23T14:30:37.601 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mds-2:19.0.0-3126.gd44fddd9.el9.x86_64 110/156 2024-04-23T14:30:37.601 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:30:37.601 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-23T14:30:37.601 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T14:30:37.601 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T14:30:37.601 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:30:38.122 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mon-2:19.0.0-3126.gd44fddd9.el9.x86_64 111/156 2024-04-23T14:30:38.194 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mon-2:19.0.0-3126.gd44fddd9.el9.x86_64 111/156 2024-04-23T14:30:38.194 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:30:38.194 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-23T14:30:38.194 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T14:30:38.194 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T14:30:38.195 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:30:38.341 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-23T14:30:38.405 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-23T14:30:38.481 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-23T14:30:38.661 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-osd-2:19.0.0-3126.gd44fddd9.el9.x86_64 109/156 2024-04-23T14:30:38.687 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-osd-2:19.0.0-3126.gd44fddd9.el9.x86_64 109/156 2024-04-23T14:30:38.687 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:30:38.687 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-23T14:30:38.687 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T14:30:38.687 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T14:30:38.687 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-23T14:30:38.728 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-23T14:30:38.794 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-23T14:30:38.884 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-23T14:30:38.912 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mds-2:19.0.0-3126.gd44fddd9.el9.x86_64 110/156 2024-04-23T14:30:38.934 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mds-2:19.0.0-3126.gd44fddd9.el9.x86_64 110/156 2024-04-23T14:30:38.935 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:30:38.935 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-23T14:30:38.935 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T14:30:38.935 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T14:30:38.935 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-23T14:30:39.004 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-23T14:30:39.186 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-23T14:30:39.235 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-23T14:30:39.275 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-23T14:30:39.308 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-23T14:30:39.379 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mon-2:19.0.0-3126.gd44fddd9.el9.x86_64 111/156 2024-04-23T14:30:39.401 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mon-2:19.0.0-3126.gd44fddd9.el9.x86_64 111/156 2024-04-23T14:30:39.402 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:30:39.402 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-23T14:30:39.402 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T14:30:39.402 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T14:30:39.402 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-23T14:30:39.509 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-23T14:30:39.548 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-23T14:30:39.604 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-23T14:30:39.789 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-23T14:30:39.830 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-23T14:30:39.907 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-23T14:30:39.954 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephadm-2:19.0.0-3126.gd44fddd9.el9.noarch 123/156 2024-04-23T14:30:39.994 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-23T14:30:39.995 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephadm-2:19.0.0-3126.gd44fddd9.el9.noarch 123/156 2024-04-23T14:30:40.050 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3126.gd44fddd9.e 124/156 2024-04-23T14:30:40.155 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-23T14:30:40.165 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3126.gd44fddd9. 125/156 2024-04-23T14:30:40.196 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-23T14:30:40.204 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3126.gd44fddd9.el9.noa 126/156 2024-04-23T14:30:40.228 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-23T14:30:40.256 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-23T14:30:40.420 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-test-2:19.0.0-3126.gd44fddd9.el9.x86_64 139/156 2024-04-23T14:30:40.462 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-fuse-2:19.0.0-3126.gd44fddd9.el9.x86_64 140/156 2024-04-23T14:30:40.503 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-nbd-2:19.0.0-3126.gd44fddd9.el9.x86_64 141/156 2024-04-23T14:30:40.534 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-23T14:30:40.562 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-23T14:30:40.641 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: cephadm-2:19.0.0-3126.gd44fddd9.el9.noarch 123/156 2024-04-23T14:30:40.671 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : cephadm-2:19.0.0-3126.gd44fddd9.el9.noarch 123/156 2024-04-23T14:30:40.704 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3126.gd44fddd9.e 124/156 2024-04-23T14:30:40.755 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephfs-shell-2:19.0.0-3126.gd44fddd9.el9.x86_64 144/156 2024-04-23T14:30:40.765 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3126.gd44fddd9. 125/156 2024-04-23T14:30:40.795 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3126.gd44fddd9.el9.noa 126/156 2024-04-23T14:30:40.800 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-23T14:30:40.830 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs-devel-2:19.0.0-3126.gd44fddd9.el9.x86_ 146/156 2024-04-23T14:30:41.498 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T14:30:41.536 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T14:30:41.557 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T14:30:41.591 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3126.gd44fddd9.el9.noa 126/156 2024-04-23T14:30:41.639 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3126.gd44fddd9.el9.n 127/156 2024-04-23T14:30:41.828 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3126.gd44fddd9.el9.noa 126/156 2024-04-23T14:30:41.863 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3126.gd44fddd9.el9.n 127/156 2024-04-23T14:30:42.442 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephfs-top-2:19.0.0-3126.gd44fddd9.el9.noarch 148/156 2024-04-23T14:30:42.552 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-23T14:30:42.594 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-23T14:30:42.679 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-23T14:30:42.714 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-23T14:30:42.715 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-23T14:30:42.799 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-23T14:30:42.811 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3126.gd44fddd9.el9.n 127/156 2024-04-23T14:30:42.823 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3126.gd44fddd9.el9.n 127/156 2024-04-23T14:30:42.839 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3126.gd44 128/156 2024-04-23T14:30:42.840 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-23T14:30:42.840 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-23T14:30:42.855 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3126.gd44 128/156 2024-04-23T14:30:42.929 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-23T14:30:42.955 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3126.gd44 128/156 2024-04-23T14:30:42.994 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3126.gd44 128/156 2024-04-23T14:30:43.000 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T14:30:43.110 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3126.gd44fddd9.el 129/156 2024-04-23T14:30:43.140 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-2:19.0.0-3126.gd44fddd9.el9.x86_64 130/156 2024-04-23T14:30:43.164 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3126.gd44fddd9.el9.x86_64 130/156 2024-04-23T14:30:43.164 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:30:43.164 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-23T14:30:43.164 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T14:30:43.164 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T14:30:43.164 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-23T14:30:43.181 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3126.gd44fddd9.el 129/156 2024-04-23T14:30:43.208 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-rook-2:19.0.0-3126.gd44fddd9.el9.noarch 131/156 2024-04-23T14:30:43.217 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-2:19.0.0-3126.gd44fddd9.el9.x86_64 130/156 2024-04-23T14:30:43.219 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3126.gd44fddd9.el9.noarch 131/156 2024-04-23T14:30:43.288 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3126.gd44fddd9.el9.x86_64 130/156 2024-04-23T14:30:43.288 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:30:43.288 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-23T14:30:43.288 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T14:30:43.288 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T14:30:43.288 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:30:43.316 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-2:19.0.0-3126.gd44fddd9.el9.x86_64 132/156 2024-04-23T14:30:43.351 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-rook-2:19.0.0-3126.gd44fddd9.el9.noarch 131/156 2024-04-23T14:30:43.364 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-fuse-2:19.0.0-3126.gd44fddd9.el9.x86_64 133/156 2024-04-23T14:30:43.390 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3126.gd44fddd9.el9.noarch 131/156 2024-04-23T14:30:43.542 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-2:19.0.0-3126.gd44fddd9.el9.x86_64 132/156 2024-04-23T14:30:43.626 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-fuse-2:19.0.0-3126.gd44fddd9.el9.x86_64 133/156 2024-04-23T14:30:44.294 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-23T14:30:44.327 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-radosgw-2:19.0.0-3126.gd44fddd9.el9.x86_64 135/156 2024-04-23T14:30:44.350 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3126.gd44fddd9.el9.x86_64 135/156 2024-04-23T14:30:44.350 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:30:44.350 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-23T14:30:44.350 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T14:30:44.350 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T14:30:44.350 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-23T14:30:44.394 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3126.gd44fd 136/156 2024-04-23T14:30:44.416 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3126.gd44fd 136/156 2024-04-23T14:30:44.416 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:30:44.416 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-23T14:30:44.416 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-23T14:30:44.462 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : cephfs-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 137/156 2024-04-23T14:30:44.484 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 137/156 2024-04-23T14:30:44.484 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:30:44.484 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-23T14:30:44.485 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-23T14:30:44.485 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-23T14:30:44.485 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-23T14:30:44.709 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-23T14:30:44.745 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-radosgw-2:19.0.0-3126.gd44fddd9.el9.x86_64 135/156 2024-04-23T14:30:44.787 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 138/156 2024-04-23T14:30:44.810 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 138/156 2024-04-23T14:30:44.810 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:30:44.810 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-23T14:30:44.810 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T14:30:44.810 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T14:30:44.810 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-23T14:30:44.828 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3126.gd44fddd9.el9.x86_64 135/156 2024-04-23T14:30:44.828 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:30:44.828 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-23T14:30:44.829 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T14:30:44.829 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T14:30:44.829 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:30:44.879 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3126.gd44fd 136/156 2024-04-23T14:30:44.952 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3126.gd44fd 136/156 2024-04-23T14:30:44.953 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:30:44.953 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-23T14:30:44.953 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:30:45.021 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 137/156 2024-04-23T14:30:45.100 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 137/156 2024-04-23T14:30:45.100 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:30:45.101 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-23T14:30:45.101 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-23T14:30:45.101 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-23T14:30:45.101 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:30:45.472 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 138/156 2024-04-23T14:30:45.543 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 138/156 2024-04-23T14:30:45.544 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T14:30:45.544 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-23T14:30:45.544 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T14:30:45.544 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T14:30:45.544 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:30:48.965 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T14:30:48.965 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-2:19.0.0-3126.gd44fddd9.el9.x86_64 1/156 2024-04-23T14:30:48.965 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-base-2:19.0.0-3126.gd44fddd9.el9.x86_64 2/156 2024-04-23T14:30:48.965 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-common-2:19.0.0-3126.gd44fddd9.el9.x86_64 3/156 2024-04-23T14:30:48.965 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-fuse-2:19.0.0-3126.gd44fddd9.el9.x86_64 4/156 2024-04-23T14:30:48.966 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3126.gd44fd 5/156 2024-04-23T14:30:48.966 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mds-2:19.0.0-3126.gd44fddd9.el9.x86_64 6/156 2024-04-23T14:30:48.966 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-2:19.0.0-3126.gd44fddd9.el9.x86_64 7/156 2024-04-23T14:30:48.966 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mon-2:19.0.0-3126.gd44fddd9.el9.x86_64 8/156 2024-04-23T14:30:48.966 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-osd-2:19.0.0-3126.gd44fddd9.el9.x86_64 9/156 2024-04-23T14:30:48.966 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-radosgw-2:19.0.0-3126.gd44fddd9.el9.x86_64 10/156 2024-04-23T14:30:48.966 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-selinux-2:19.0.0-3126.gd44fddd9.el9.x86_64 11/156 2024-04-23T14:30:48.966 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-test-2:19.0.0-3126.gd44fddd9.el9.x86_64 12/156 2024-04-23T14:30:48.966 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephfs-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 13/156 2024-04-23T14:30:48.966 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephfs-shell-2:19.0.0-3126.gd44fddd9.el9.x86_64 14/156 2024-04-23T14:30:48.966 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs-devel-2:19.0.0-3126.gd44fddd9.el9.x86_ 15/156 2024-04-23T14:30:48.966 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs2-2:19.0.0-3126.gd44fddd9.el9.x86_64 16/156 2024-04-23T14:30:48.966 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephsqlite-2:19.0.0-3126.gd44fddd9.el9.x86_64 17/156 2024-04-23T14:30:48.966 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados-devel-2:19.0.0-3126.gd44fddd9.el9.x86_6 18/156 2024-04-23T14:30:48.967 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libradosstriper1-2:19.0.0-3126.gd44fddd9.el9.x86 19/156 2024-04-23T14:30:48.967 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librgw2-2:19.0.0-3126.gd44fddd9.el9.x86_64 20/156 2024-04-23T14:30:48.967 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3126.gd44fddd9.el 21/156 2024-04-23T14:30:48.967 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-common-2:19.0.0-3126.gd44fddd9.el9. 22/156 2024-04-23T14:30:48.967 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cephfs-2:19.0.0-3126.gd44fddd9.el9.x86_6 23/156 2024-04-23T14:30:48.967 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rados-2:19.0.0-3126.gd44fddd9.el9.x86_64 24/156 2024-04-23T14:30:48.967 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rbd-2:19.0.0-3126.gd44fddd9.el9.x86_64 25/156 2024-04-23T14:30:48.967 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rgw-2:19.0.0-3126.gd44fddd9.el9.x86_64 26/156 2024-04-23T14:30:48.967 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-fuse-2:19.0.0-3126.gd44fddd9.el9.x86_64 27/156 2024-04-23T14:30:48.967 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 28/156 2024-04-23T14:30:48.967 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-nbd-2:19.0.0-3126.gd44fddd9.el9.x86_64 29/156 2024-04-23T14:30:48.967 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3126.gd44fddd9. 30/156 2024-04-23T14:30:48.967 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3126.gd44fddd9.el9.noa 31/156 2024-04-23T14:30:48.967 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3126.gd44fddd9.el9.n 32/156 2024-04-23T14:30:48.968 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3126.gd44 33/156 2024-04-23T14:30:48.968 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3126.gd44fddd9.el 34/156 2024-04-23T14:30:48.968 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3126.gd44fddd9.el9.noarch 35/156 2024-04-23T14:30:48.968 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3126.gd44fddd9.e 36/156 2024-04-23T14:30:48.968 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-volume-2:19.0.0-3126.gd44fddd9.el9.noarch 37/156 2024-04-23T14:30:48.968 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephadm-2:19.0.0-3126.gd44fddd9.el9.noarch 38/156 2024-04-23T14:30:48.968 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephfs-top-2:19.0.0-3126.gd44fddd9.el9.noarch 39/156 2024-04-23T14:30:48.968 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-23T14:30:48.968 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-23T14:30:48.968 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-23T14:30:48.968 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-23T14:30:48.968 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-23T14:30:48.968 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-23T14:30:48.968 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-23T14:30:48.968 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-23T14:30:48.969 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-23T14:30:48.969 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-23T14:30:48.969 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-23T14:30:48.969 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-23T14:30:48.969 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-23T14:30:48.969 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-23T14:30:48.969 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-23T14:30:48.969 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-23T14:30:48.969 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-23T14:30:48.969 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-23T14:30:48.969 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-23T14:30:48.969 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-23T14:30:48.969 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-23T14:30:48.969 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-23T14:30:48.969 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-23T14:30:48.969 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-23T14:30:48.970 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-23T14:30:48.970 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-23T14:30:48.970 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-23T14:30:48.970 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-23T14:30:48.970 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-23T14:30:48.970 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-23T14:30:48.970 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-23T14:30:48.970 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-23T14:30:48.970 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-23T14:30:48.970 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-23T14:30:48.970 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-23T14:30:48.970 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-23T14:30:48.971 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-23T14:30:48.971 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-23T14:30:48.971 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-23T14:30:48.971 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-23T14:30:48.971 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-23T14:30:48.971 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-23T14:30:48.971 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-23T14:30:48.971 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-23T14:30:48.971 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-23T14:30:48.971 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-23T14:30:48.971 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-23T14:30:48.971 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-23T14:30:48.971 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-23T14:30:48.971 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-23T14:30:48.972 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-23T14:30:48.972 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-23T14:30:48.972 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-23T14:30:48.972 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-23T14:30:48.972 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-23T14:30:48.972 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-23T14:30:48.972 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-23T14:30:48.972 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-23T14:30:48.972 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-23T14:30:48.972 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-23T14:30:48.972 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-23T14:30:48.972 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-23T14:30:48.972 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-23T14:30:48.972 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-23T14:30:48.973 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-23T14:30:48.973 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-23T14:30:48.973 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-23T14:30:48.973 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-23T14:30:48.973 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-23T14:30:48.973 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-23T14:30:48.973 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-23T14:30:48.973 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-23T14:30:48.973 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-23T14:30:48.973 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-23T14:30:48.973 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-23T14:30:48.973 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-23T14:30:48.973 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-23T14:30:48.973 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-23T14:30:48.973 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-23T14:30:48.974 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-23T14:30:48.974 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-23T14:30:48.974 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-23T14:30:48.974 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-23T14:30:48.974 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-23T14:30:48.974 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-23T14:30:48.974 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-23T14:30:48.974 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-23T14:30:48.974 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-23T14:30:48.974 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-23T14:30:48.974 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-23T14:30:48.974 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-23T14:30:48.974 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-23T14:30:48.974 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-23T14:30:48.975 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-23T14:30:48.975 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-23T14:30:48.975 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-23T14:30:48.975 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-23T14:30:48.975 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-23T14:30:48.975 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-23T14:30:48.975 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-23T14:30:48.975 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-23T14:30:48.975 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-23T14:30:48.975 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-23T14:30:48.975 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-23T14:30:48.975 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-23T14:30:48.975 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-23T14:30:48.975 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-23T14:30:48.976 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-23T14:30:48.976 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-23T14:30:48.976 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-2:19.0.0-3126.gd44fddd9.el9.x86_64 149/156 2024-04-23T14:30:48.976 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-23T14:30:48.976 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-2:19.0.0-3126.gd44fddd9.el9.x86_64 151/156 2024-04-23T14:30:48.976 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-23T14:30:48.976 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-23T14:30:48.976 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-23T14:30:48.976 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-23T14:30:49.382 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-test-2:19.0.0-3126.gd44fddd9.el9.x86_64 139/156 2024-04-23T14:30:49.424 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-fuse-2:19.0.0-3126.gd44fddd9.el9.x86_64 140/156 2024-04-23T14:30:49.464 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-nbd-2:19.0.0-3126.gd44fddd9.el9.x86_64 141/156 2024-04-23T14:30:49.496 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-23T14:30:49.524 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-23T14:30:49.702 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : cephfs-shell-2:19.0.0-3126.gd44fddd9.el9.x86_64 144/156 2024-04-23T14:30:49.753 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-23T14:30:49.783 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephfs-devel-2:19.0.0-3126.gd44fddd9.el9.x86_ 146/156 2024-04-23T14:30:50.360 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T14:30:50.397 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T14:30:50.417 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T14:30:50.879 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-test-2:19.0.0-3126.gd44fddd9.el9.x86_64 139/156 2024-04-23T14:30:50.934 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-fuse-2:19.0.0-3126.gd44fddd9.el9.x86_64 140/156 2024-04-23T14:30:51.008 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-nbd-2:19.0.0-3126.gd44fddd9.el9.x86_64 141/156 2024-04-23T14:30:51.057 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-23T14:30:51.105 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-23T14:30:51.211 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T14:30:51.211 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-23T14:30:51.211 INFO:teuthology.orchestra.run.smithi038.stdout:Upgraded: 2024-04-23T14:30:51.211 INFO:teuthology.orchestra.run.smithi038.stdout: librados2-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:51.211 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:51.211 INFO:teuthology.orchestra.run.smithi038.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-23T14:30:51.211 INFO:teuthology.orchestra.run.smithi038.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-23T14:30:51.211 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-04-23T14:30:51.211 INFO:teuthology.orchestra.run.smithi038.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-23T14:30:51.212 INFO:teuthology.orchestra.run.smithi038.stdout: automake-1.16.2-8.el9.noarch 2024-04-23T14:30:51.212 INFO:teuthology.orchestra.run.smithi038.stdout: bison-3.7.4-5.el9.x86_64 2024-04-23T14:30:51.212 INFO:teuthology.orchestra.run.smithi038.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-23T14:30:51.212 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:51.212 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:51.212 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:51.212 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:51.212 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:30:51.212 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:51.212 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:51.212 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:51.212 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:30:51.212 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:30:51.212 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:30:51.213 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:30:51.213 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:30:51.213 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:51.213 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:51.213 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:30:51.213 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:51.213 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:51.213 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:51.213 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-volume-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:30:51.213 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:30:51.213 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:51.213 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-shell-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:51.213 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-top-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:30:51.213 INFO:teuthology.orchestra.run.smithi038.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-23T14:30:51.213 INFO:teuthology.orchestra.run.smithi038.stdout: flex-2.6.4-9.el9.x86_64 2024-04-23T14:30:51.213 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-23T14:30:51.214 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-23T14:30:51.214 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-23T14:30:51.214 INFO:teuthology.orchestra.run.smithi038.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-23T14:30:51.214 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-23T14:30:51.214 INFO:teuthology.orchestra.run.smithi038.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-23T14:30:51.214 INFO:teuthology.orchestra.run.smithi038.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-23T14:30:51.214 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-23T14:30:51.214 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-23T14:30:51.214 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:51.214 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:51.214 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:51.214 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-23T14:30:51.214 INFO:teuthology.orchestra.run.smithi038.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-23T14:30:51.215 INFO:teuthology.orchestra.run.smithi038.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-23T14:30:51.215 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-23T14:30:51.215 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-23T14:30:51.215 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:51.215 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:51.215 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-23T14:30:51.215 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:51.215 INFO:teuthology.orchestra.run.smithi038.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-23T14:30:51.215 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-23T14:30:51.215 INFO:teuthology.orchestra.run.smithi038.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-23T14:30:51.215 INFO:teuthology.orchestra.run.smithi038.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-23T14:30:51.215 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-23T14:30:51.215 INFO:teuthology.orchestra.run.smithi038.stdout: lua-5.4.4-4.el9.x86_64 2024-04-23T14:30:51.215 INFO:teuthology.orchestra.run.smithi038.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-23T14:30:51.215 INFO:teuthology.orchestra.run.smithi038.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-23T14:30:51.216 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-23T14:30:51.216 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-23T14:30:51.216 INFO:teuthology.orchestra.run.smithi038.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-23T14:30:51.216 INFO:teuthology.orchestra.run.smithi038.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-23T14:30:51.216 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-13.14-1.el9.x86_64 2024-04-23T14:30:51.216 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-23T14:30:51.216 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-23T14:30:51.216 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-23T14:30:51.216 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-23T14:30:51.216 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-23T14:30:51.216 INFO:teuthology.orchestra.run.smithi038.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-23T14:30:51.216 INFO:teuthology.orchestra.run.smithi038.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-23T14:30:51.216 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-23T14:30:51.216 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-23T14:30:51.216 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-23T14:30:51.217 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:51.217 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:51.217 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:51.217 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-23T14:30:51.217 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-23T14:30:51.217 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-23T14:30:51.217 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-23T14:30:51.217 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-23T14:30:51.217 INFO:teuthology.orchestra.run.smithi038.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-23T14:30:51.217 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-23T14:30:51.217 INFO:teuthology.orchestra.run.smithi038.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-23T14:30:51.217 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-23T14:30:51.217 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-23T14:30:51.217 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-23T14:30:51.217 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-23T14:30:51.217 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-23T14:30:51.218 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-23T14:30:51.218 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-23T14:30:51.218 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-23T14:30:51.218 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-23T14:30:51.218 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-23T14:30:51.218 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-23T14:30:51.218 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-23T14:30:51.218 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-23T14:30:51.218 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-23T14:30:51.218 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-23T14:30:51.218 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-23T14:30:51.218 INFO:teuthology.orchestra.run.smithi038.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-23T14:30:51.218 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-23T14:30:51.218 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-23T14:30:51.219 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-23T14:30:51.219 INFO:teuthology.orchestra.run.smithi038.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-23T14:30:51.219 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-23T14:30:51.219 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ply-3.11-14.el9.noarch 2024-04-23T14:30:51.219 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-23T14:30:51.219 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-23T14:30:51.219 INFO:teuthology.orchestra.run.smithi038.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-23T14:30:51.219 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-23T14:30:51.219 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-23T14:30:51.219 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-23T14:30:51.219 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-23T14:30:51.219 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-23T14:30:51.219 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-23T14:30:51.219 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:51.220 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:51.220 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-23T14:30:51.220 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-23T14:30:51.220 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-23T14:30:51.220 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:51.220 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-23T14:30:51.220 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-23T14:30:51.220 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-23T14:30:51.220 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-23T14:30:51.220 INFO:teuthology.orchestra.run.smithi038.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-23T14:30:51.220 INFO:teuthology.orchestra.run.smithi038.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-23T14:30:51.220 INFO:teuthology.orchestra.run.smithi038.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-23T14:30:51.220 INFO:teuthology.orchestra.run.smithi038.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-23T14:30:51.220 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-23T14:30:51.221 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-23T14:30:51.221 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-23T14:30:51.221 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-23T14:30:51.221 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-23T14:30:51.221 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-23T14:30:51.221 INFO:teuthology.orchestra.run.smithi038.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-23T14:30:51.221 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:51.221 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:51.221 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:51.221 INFO:teuthology.orchestra.run.smithi038.stdout: re2-1:20211101-3.el9.x86_64 2024-04-23T14:30:51.221 INFO:teuthology.orchestra.run.smithi038.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-23T14:30:51.221 INFO:teuthology.orchestra.run.smithi038.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-23T14:30:51.221 INFO:teuthology.orchestra.run.smithi038.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-23T14:30:51.221 INFO:teuthology.orchestra.run.smithi038.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-23T14:30:51.221 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-23T14:30:51.222 INFO:teuthology.orchestra.run.smithi038.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-23T14:30:51.222 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-23T14:30:51.222 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-23T14:30:51.240 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : cephfs-top-2:19.0.0-3126.gd44fddd9.el9.noarch 148/156 2024-04-23T14:30:51.335 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-23T14:30:51.376 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-23T14:30:51.457 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-23T14:30:51.458 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-shell-2:19.0.0-3126.gd44fddd9.el9.x86_64 144/156 2024-04-23T14:30:51.489 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-23T14:30:51.489 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-23T14:30:51.516 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-23T14:30:51.561 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs-devel-2:19.0.0-3126.gd44fddd9.el9.x86_ 146/156 2024-04-23T14:30:51.573 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-23T14:30:51.606 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-23T14:30:51.606 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-23T14:30:51.613 DEBUG:teuthology.parallel:result is None 2024-04-23T14:30:51.695 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-23T14:30:51.771 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T14:30:52.537 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T14:30:52.593 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T14:30:52.652 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T14:30:54.117 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-top-2:19.0.0-3126.gd44fddd9.el9.noarch 148/156 2024-04-23T14:30:54.224 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-23T14:30:54.306 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-23T14:30:54.526 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-23T14:30:54.573 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-23T14:30:54.573 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-23T14:30:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-23T14:30:54.737 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-23T14:30:54.738 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-23T14:30:54.878 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-23T14:30:55.012 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T14:30:55.282 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T14:30:55.282 INFO:teuthology.orchestra.run.smithi028.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-23T14:30:55.282 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:30:57.822 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T14:30:57.822 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-2:19.0.0-3126.gd44fddd9.el9.x86_64 1/156 2024-04-23T14:30:57.822 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-base-2:19.0.0-3126.gd44fddd9.el9.x86_64 2/156 2024-04-23T14:30:57.822 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-common-2:19.0.0-3126.gd44fddd9.el9.x86_64 3/156 2024-04-23T14:30:57.822 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-fuse-2:19.0.0-3126.gd44fddd9.el9.x86_64 4/156 2024-04-23T14:30:57.823 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3126.gd44fd 5/156 2024-04-23T14:30:57.823 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mds-2:19.0.0-3126.gd44fddd9.el9.x86_64 6/156 2024-04-23T14:30:57.823 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-2:19.0.0-3126.gd44fddd9.el9.x86_64 7/156 2024-04-23T14:30:57.823 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mon-2:19.0.0-3126.gd44fddd9.el9.x86_64 8/156 2024-04-23T14:30:57.823 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-osd-2:19.0.0-3126.gd44fddd9.el9.x86_64 9/156 2024-04-23T14:30:57.823 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-radosgw-2:19.0.0-3126.gd44fddd9.el9.x86_64 10/156 2024-04-23T14:30:57.823 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-selinux-2:19.0.0-3126.gd44fddd9.el9.x86_64 11/156 2024-04-23T14:30:57.823 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-test-2:19.0.0-3126.gd44fddd9.el9.x86_64 12/156 2024-04-23T14:30:57.823 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : cephfs-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 13/156 2024-04-23T14:30:57.823 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : cephfs-shell-2:19.0.0-3126.gd44fddd9.el9.x86_64 14/156 2024-04-23T14:30:57.824 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephfs-devel-2:19.0.0-3126.gd44fddd9.el9.x86_ 15/156 2024-04-23T14:30:57.824 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephfs2-2:19.0.0-3126.gd44fddd9.el9.x86_64 16/156 2024-04-23T14:30:57.824 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephsqlite-2:19.0.0-3126.gd44fddd9.el9.x86_64 17/156 2024-04-23T14:30:57.824 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados-devel-2:19.0.0-3126.gd44fddd9.el9.x86_6 18/156 2024-04-23T14:30:57.824 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libradosstriper1-2:19.0.0-3126.gd44fddd9.el9.x86 19/156 2024-04-23T14:30:57.824 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librgw2-2:19.0.0-3126.gd44fddd9.el9.x86_64 20/156 2024-04-23T14:30:57.824 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3126.gd44fddd9.el 21/156 2024-04-23T14:30:57.824 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ceph-common-2:19.0.0-3126.gd44fddd9.el9. 22/156 2024-04-23T14:30:57.824 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cephfs-2:19.0.0-3126.gd44fddd9.el9.x86_6 23/156 2024-04-23T14:30:57.824 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rados-2:19.0.0-3126.gd44fddd9.el9.x86_64 24/156 2024-04-23T14:30:57.824 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rbd-2:19.0.0-3126.gd44fddd9.el9.x86_64 25/156 2024-04-23T14:30:57.825 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rgw-2:19.0.0-3126.gd44fddd9.el9.x86_64 26/156 2024-04-23T14:30:57.825 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-fuse-2:19.0.0-3126.gd44fddd9.el9.x86_64 27/156 2024-04-23T14:30:57.825 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 28/156 2024-04-23T14:30:57.825 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-nbd-2:19.0.0-3126.gd44fddd9.el9.x86_64 29/156 2024-04-23T14:30:57.825 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3126.gd44fddd9. 30/156 2024-04-23T14:30:57.825 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3126.gd44fddd9.el9.noa 31/156 2024-04-23T14:30:57.825 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3126.gd44fddd9.el9.n 32/156 2024-04-23T14:30:57.825 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3126.gd44 33/156 2024-04-23T14:30:57.825 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3126.gd44fddd9.el 34/156 2024-04-23T14:30:57.825 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3126.gd44fddd9.el9.noarch 35/156 2024-04-23T14:30:57.825 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3126.gd44fddd9.e 36/156 2024-04-23T14:30:57.826 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-volume-2:19.0.0-3126.gd44fddd9.el9.noarch 37/156 2024-04-23T14:30:57.826 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : cephadm-2:19.0.0-3126.gd44fddd9.el9.noarch 38/156 2024-04-23T14:30:57.826 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : cephfs-top-2:19.0.0-3126.gd44fddd9.el9.noarch 39/156 2024-04-23T14:30:57.826 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-23T14:30:57.826 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-23T14:30:57.826 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-23T14:30:57.826 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-23T14:30:57.826 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-23T14:30:57.826 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-23T14:30:57.826 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-23T14:30:57.826 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-23T14:30:57.827 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-23T14:30:57.827 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-23T14:30:57.827 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-23T14:30:57.827 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-23T14:30:57.827 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-23T14:30:57.827 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-23T14:30:57.827 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-23T14:30:57.827 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-23T14:30:57.827 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-23T14:30:57.827 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-23T14:30:57.827 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-23T14:30:57.827 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-23T14:30:57.828 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-23T14:30:57.828 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-23T14:30:57.828 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-23T14:30:57.828 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-23T14:30:57.828 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-23T14:30:57.828 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-23T14:30:57.828 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-23T14:30:57.828 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-23T14:30:57.828 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-23T14:30:57.828 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-23T14:30:57.829 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-23T14:30:57.829 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-23T14:30:57.829 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-23T14:30:57.829 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-23T14:30:57.829 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-23T14:30:57.829 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-23T14:30:57.829 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-23T14:30:57.829 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-23T14:30:57.829 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-23T14:30:57.829 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-23T14:30:57.829 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-23T14:30:57.830 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-23T14:30:57.830 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-23T14:30:57.830 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-23T14:30:57.830 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-23T14:30:57.830 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-23T14:30:57.830 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-23T14:30:57.830 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-23T14:30:57.830 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-23T14:30:57.830 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-23T14:30:57.830 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-23T14:30:57.830 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-23T14:30:57.831 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-23T14:30:57.831 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-23T14:30:57.831 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-23T14:30:57.831 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-23T14:30:57.831 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-23T14:30:57.831 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-23T14:30:57.831 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-23T14:30:57.831 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-23T14:30:57.831 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-23T14:30:57.831 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-23T14:30:57.831 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-23T14:30:57.831 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-23T14:30:57.832 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-23T14:30:57.832 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-23T14:30:57.832 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-23T14:30:57.832 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-23T14:30:57.832 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-23T14:30:57.832 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-23T14:30:57.832 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-23T14:30:57.832 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-23T14:30:57.832 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-23T14:30:57.832 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-23T14:30:57.832 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-23T14:30:57.832 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-23T14:30:57.833 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-23T14:30:57.833 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-23T14:30:57.833 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-23T14:30:57.833 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-23T14:30:57.833 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-23T14:30:57.833 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-23T14:30:57.833 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-23T14:30:57.833 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-23T14:30:57.833 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-23T14:30:57.833 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-23T14:30:57.833 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-23T14:30:57.833 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-23T14:30:57.834 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-23T14:30:57.834 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-23T14:30:57.834 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-23T14:30:57.834 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-23T14:30:57.834 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-23T14:30:57.834 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-23T14:30:57.834 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-23T14:30:57.834 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-23T14:30:57.834 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-23T14:30:57.834 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-23T14:30:57.834 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-23T14:30:57.834 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-23T14:30:57.835 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-23T14:30:57.835 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-23T14:30:57.835 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-23T14:30:57.835 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-23T14:30:57.835 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-23T14:30:57.835 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-23T14:30:57.835 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-23T14:30:57.835 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-23T14:30:57.835 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-23T14:30:57.835 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados2-2:19.0.0-3126.gd44fddd9.el9.x86_64 149/156 2024-04-23T14:30:57.835 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-23T14:30:57.835 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librbd1-2:19.0.0-3126.gd44fddd9.el9.x86_64 151/156 2024-04-23T14:30:57.836 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-23T14:30:57.836 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-23T14:30:57.836 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-23T14:30:57.836 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-23T14:30:59.936 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T14:30:59.936 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-23T14:30:59.936 INFO:teuthology.orchestra.run.smithi105.stdout:Upgraded: 2024-04-23T14:30:59.936 INFO:teuthology.orchestra.run.smithi105.stdout: librados2-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:59.936 INFO:teuthology.orchestra.run.smithi105.stdout: librbd1-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:59.936 INFO:teuthology.orchestra.run.smithi105.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-23T14:30:59.936 INFO:teuthology.orchestra.run.smithi105.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-23T14:30:59.936 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-04-23T14:30:59.936 INFO:teuthology.orchestra.run.smithi105.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-23T14:30:59.936 INFO:teuthology.orchestra.run.smithi105.stdout: automake-1.16.2-8.el9.noarch 2024-04-23T14:30:59.937 INFO:teuthology.orchestra.run.smithi105.stdout: bison-3.7.4-5.el9.x86_64 2024-04-23T14:30:59.937 INFO:teuthology.orchestra.run.smithi105.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-23T14:30:59.937 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:59.937 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-base-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:59.937 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-common-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:59.937 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-fuse-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:59.937 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-grafana-dashboards-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:30:59.937 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-immutable-object-cache-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:59.937 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mds-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:59.937 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:59.937 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-cephadm-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:30:59.937 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-dashboard-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:30:59.937 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:30:59.937 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-modules-core-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:30:59.937 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-rook-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:30:59.938 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mon-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:59.938 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-osd-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:59.938 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-prometheus-alerts-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:30:59.938 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-radosgw-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:59.938 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-selinux-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:59.938 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-test-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:59.938 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-volume-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:30:59.938 INFO:teuthology.orchestra.run.smithi105.stdout: cephadm-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:30:59.938 INFO:teuthology.orchestra.run.smithi105.stdout: cephfs-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:59.938 INFO:teuthology.orchestra.run.smithi105.stdout: cephfs-shell-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:59.938 INFO:teuthology.orchestra.run.smithi105.stdout: cephfs-top-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:30:59.938 INFO:teuthology.orchestra.run.smithi105.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-23T14:30:59.939 INFO:teuthology.orchestra.run.smithi105.stdout: flex-2.6.4-9.el9.x86_64 2024-04-23T14:30:59.939 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-23T14:30:59.939 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-23T14:30:59.939 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-23T14:30:59.939 INFO:teuthology.orchestra.run.smithi105.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-23T14:30:59.939 INFO:teuthology.orchestra.run.smithi105.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-23T14:30:59.939 INFO:teuthology.orchestra.run.smithi105.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-23T14:30:59.939 INFO:teuthology.orchestra.run.smithi105.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-23T14:30:59.939 INFO:teuthology.orchestra.run.smithi105.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-23T14:30:59.939 INFO:teuthology.orchestra.run.smithi105.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-23T14:30:59.939 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs-devel-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:59.939 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs2-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:59.939 INFO:teuthology.orchestra.run.smithi105.stdout: libcephsqlite-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:59.940 INFO:teuthology.orchestra.run.smithi105.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-23T14:30:59.940 INFO:teuthology.orchestra.run.smithi105.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-23T14:30:59.940 INFO:teuthology.orchestra.run.smithi105.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-23T14:30:59.940 INFO:teuthology.orchestra.run.smithi105.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-23T14:30:59.940 INFO:teuthology.orchestra.run.smithi105.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-23T14:30:59.940 INFO:teuthology.orchestra.run.smithi105.stdout: librados-devel-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:59.940 INFO:teuthology.orchestra.run.smithi105.stdout: libradosstriper1-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:59.940 INFO:teuthology.orchestra.run.smithi105.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-23T14:30:59.940 INFO:teuthology.orchestra.run.smithi105.stdout: librgw2-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:59.940 INFO:teuthology.orchestra.run.smithi105.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-23T14:30:59.940 INFO:teuthology.orchestra.run.smithi105.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-23T14:30:59.940 INFO:teuthology.orchestra.run.smithi105.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-23T14:30:59.940 INFO:teuthology.orchestra.run.smithi105.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-23T14:30:59.940 INFO:teuthology.orchestra.run.smithi105.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-23T14:30:59.941 INFO:teuthology.orchestra.run.smithi105.stdout: lua-5.4.4-4.el9.x86_64 2024-04-23T14:30:59.941 INFO:teuthology.orchestra.run.smithi105.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-23T14:30:59.941 INFO:teuthology.orchestra.run.smithi105.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-23T14:30:59.941 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-23T14:30:59.941 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-23T14:30:59.941 INFO:teuthology.orchestra.run.smithi105.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-23T14:30:59.941 INFO:teuthology.orchestra.run.smithi105.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-23T14:30:59.941 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-13.14-1.el9.x86_64 2024-04-23T14:30:59.941 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-23T14:30:59.941 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-23T14:30:59.941 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-23T14:30:59.941 INFO:teuthology.orchestra.run.smithi105.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-23T14:30:59.941 INFO:teuthology.orchestra.run.smithi105.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-23T14:30:59.942 INFO:teuthology.orchestra.run.smithi105.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-23T14:30:59.942 INFO:teuthology.orchestra.run.smithi105.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-23T14:30:59.942 INFO:teuthology.orchestra.run.smithi105.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-23T14:30:59.942 INFO:teuthology.orchestra.run.smithi105.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-23T14:30:59.942 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-23T14:30:59.942 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-argparse-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:59.942 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-common-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:59.942 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cephfs-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:59.942 INFO:teuthology.orchestra.run.smithi105.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-23T14:30:59.942 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-23T14:30:59.942 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-23T14:30:59.942 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-23T14:30:59.942 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-23T14:30:59.943 INFO:teuthology.orchestra.run.smithi105.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-23T14:30:59.943 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-23T14:30:59.943 INFO:teuthology.orchestra.run.smithi105.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-23T14:30:59.943 INFO:teuthology.orchestra.run.smithi105.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-23T14:30:59.943 INFO:teuthology.orchestra.run.smithi105.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-23T14:30:59.943 INFO:teuthology.orchestra.run.smithi105.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-23T14:30:59.943 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-23T14:30:59.943 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-23T14:30:59.943 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-23T14:30:59.943 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-23T14:30:59.943 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-23T14:30:59.943 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-23T14:30:59.943 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-23T14:30:59.943 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-23T14:30:59.943 INFO:teuthology.orchestra.run.smithi105.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-23T14:30:59.944 INFO:teuthology.orchestra.run.smithi105.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-23T14:30:59.944 INFO:teuthology.orchestra.run.smithi105.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-23T14:30:59.944 INFO:teuthology.orchestra.run.smithi105.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-23T14:30:59.944 INFO:teuthology.orchestra.run.smithi105.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-23T14:30:59.944 INFO:teuthology.orchestra.run.smithi105.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-23T14:30:59.944 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-23T14:30:59.944 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-23T14:30:59.944 INFO:teuthology.orchestra.run.smithi105.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-23T14:30:59.944 INFO:teuthology.orchestra.run.smithi105.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-23T14:30:59.944 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-23T14:30:59.944 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ply-3.11-14.el9.noarch 2024-04-23T14:30:59.944 INFO:teuthology.orchestra.run.smithi105.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-23T14:30:59.944 INFO:teuthology.orchestra.run.smithi105.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-23T14:30:59.944 INFO:teuthology.orchestra.run.smithi105.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-23T14:30:59.945 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-23T14:30:59.945 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-23T14:30:59.945 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-23T14:30:59.945 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-23T14:30:59.945 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-23T14:30:59.945 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-23T14:30:59.945 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rados-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:59.945 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rbd-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:59.945 INFO:teuthology.orchestra.run.smithi105.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-23T14:30:59.945 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-23T14:30:59.945 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-23T14:30:59.945 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rgw-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:59.945 INFO:teuthology.orchestra.run.smithi105.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-23T14:30:59.946 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-23T14:30:59.946 INFO:teuthology.orchestra.run.smithi105.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-23T14:30:59.946 INFO:teuthology.orchestra.run.smithi105.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-23T14:30:59.946 INFO:teuthology.orchestra.run.smithi105.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-23T14:30:59.946 INFO:teuthology.orchestra.run.smithi105.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-23T14:30:59.946 INFO:teuthology.orchestra.run.smithi105.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-23T14:30:59.946 INFO:teuthology.orchestra.run.smithi105.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-23T14:30:59.946 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-23T14:30:59.946 INFO:teuthology.orchestra.run.smithi105.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-23T14:30:59.946 INFO:teuthology.orchestra.run.smithi105.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-23T14:30:59.946 INFO:teuthology.orchestra.run.smithi105.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-23T14:30:59.946 INFO:teuthology.orchestra.run.smithi105.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-23T14:30:59.946 INFO:teuthology.orchestra.run.smithi105.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-23T14:30:59.946 INFO:teuthology.orchestra.run.smithi105.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-23T14:30:59.947 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-fuse-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:59.947 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:59.947 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-nbd-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:30:59.947 INFO:teuthology.orchestra.run.smithi105.stdout: re2-1:20211101-3.el9.x86_64 2024-04-23T14:30:59.947 INFO:teuthology.orchestra.run.smithi105.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-23T14:30:59.947 INFO:teuthology.orchestra.run.smithi105.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-23T14:30:59.947 INFO:teuthology.orchestra.run.smithi105.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-23T14:30:59.947 INFO:teuthology.orchestra.run.smithi105.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-23T14:30:59.947 INFO:teuthology.orchestra.run.smithi105.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-23T14:30:59.947 INFO:teuthology.orchestra.run.smithi105.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-23T14:30:59.947 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-23T14:30:59.948 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-23T14:31:00.321 DEBUG:teuthology.parallel:result is None 2024-04-23T14:31:01.181 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-2:19.0.0-3126.gd44fddd9.el9.x86_64 1/156 2024-04-23T14:31:01.181 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-base-2:19.0.0-3126.gd44fddd9.el9.x86_64 2/156 2024-04-23T14:31:01.181 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-common-2:19.0.0-3126.gd44fddd9.el9.x86_64 3/156 2024-04-23T14:31:01.181 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-fuse-2:19.0.0-3126.gd44fddd9.el9.x86_64 4/156 2024-04-23T14:31:01.181 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3126.gd44fd 5/156 2024-04-23T14:31:01.181 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mds-2:19.0.0-3126.gd44fddd9.el9.x86_64 6/156 2024-04-23T14:31:01.181 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-2:19.0.0-3126.gd44fddd9.el9.x86_64 7/156 2024-04-23T14:31:01.181 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mon-2:19.0.0-3126.gd44fddd9.el9.x86_64 8/156 2024-04-23T14:31:01.181 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-osd-2:19.0.0-3126.gd44fddd9.el9.x86_64 9/156 2024-04-23T14:31:01.182 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-radosgw-2:19.0.0-3126.gd44fddd9.el9.x86_64 10/156 2024-04-23T14:31:01.182 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-selinux-2:19.0.0-3126.gd44fddd9.el9.x86_64 11/156 2024-04-23T14:31:01.182 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-test-2:19.0.0-3126.gd44fddd9.el9.x86_64 12/156 2024-04-23T14:31:01.182 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 13/156 2024-04-23T14:31:01.182 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-shell-2:19.0.0-3126.gd44fddd9.el9.x86_64 14/156 2024-04-23T14:31:01.182 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs-devel-2:19.0.0-3126.gd44fddd9.el9.x86_ 15/156 2024-04-23T14:31:01.182 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs2-2:19.0.0-3126.gd44fddd9.el9.x86_64 16/156 2024-04-23T14:31:01.182 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephsqlite-2:19.0.0-3126.gd44fddd9.el9.x86_64 17/156 2024-04-23T14:31:01.182 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados-devel-2:19.0.0-3126.gd44fddd9.el9.x86_6 18/156 2024-04-23T14:31:01.182 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libradosstriper1-2:19.0.0-3126.gd44fddd9.el9.x86 19/156 2024-04-23T14:31:01.182 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librgw2-2:19.0.0-3126.gd44fddd9.el9.x86_64 20/156 2024-04-23T14:31:01.182 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3126.gd44fddd9.el 21/156 2024-04-23T14:31:01.182 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-common-2:19.0.0-3126.gd44fddd9.el9. 22/156 2024-04-23T14:31:01.182 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cephfs-2:19.0.0-3126.gd44fddd9.el9.x86_6 23/156 2024-04-23T14:31:01.182 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rados-2:19.0.0-3126.gd44fddd9.el9.x86_64 24/156 2024-04-23T14:31:01.183 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rbd-2:19.0.0-3126.gd44fddd9.el9.x86_64 25/156 2024-04-23T14:31:01.183 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rgw-2:19.0.0-3126.gd44fddd9.el9.x86_64 26/156 2024-04-23T14:31:01.183 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-fuse-2:19.0.0-3126.gd44fddd9.el9.x86_64 27/156 2024-04-23T14:31:01.183 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 28/156 2024-04-23T14:31:01.183 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-nbd-2:19.0.0-3126.gd44fddd9.el9.x86_64 29/156 2024-04-23T14:31:01.183 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3126.gd44fddd9. 30/156 2024-04-23T14:31:01.183 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3126.gd44fddd9.el9.noa 31/156 2024-04-23T14:31:01.183 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3126.gd44fddd9.el9.n 32/156 2024-04-23T14:31:01.183 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3126.gd44 33/156 2024-04-23T14:31:01.183 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3126.gd44fddd9.el 34/156 2024-04-23T14:31:01.183 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3126.gd44fddd9.el9.noarch 35/156 2024-04-23T14:31:01.183 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3126.gd44fddd9.e 36/156 2024-04-23T14:31:01.183 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-volume-2:19.0.0-3126.gd44fddd9.el9.noarch 37/156 2024-04-23T14:31:01.183 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephadm-2:19.0.0-3126.gd44fddd9.el9.noarch 38/156 2024-04-23T14:31:01.184 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-top-2:19.0.0-3126.gd44fddd9.el9.noarch 39/156 2024-04-23T14:31:01.184 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-23T14:31:01.184 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-23T14:31:01.184 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-23T14:31:01.184 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-23T14:31:01.184 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-23T14:31:01.184 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-23T14:31:01.184 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-23T14:31:01.184 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-23T14:31:01.184 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-23T14:31:01.184 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-23T14:31:01.184 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-23T14:31:01.184 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-23T14:31:01.185 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-23T14:31:01.185 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-23T14:31:01.185 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-23T14:31:01.185 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-23T14:31:01.185 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-23T14:31:01.185 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-23T14:31:01.185 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-23T14:31:01.185 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-23T14:31:01.185 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-23T14:31:01.185 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-23T14:31:01.185 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-23T14:31:01.185 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-23T14:31:01.185 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-23T14:31:01.186 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-23T14:31:01.186 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-23T14:31:01.186 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-23T14:31:01.186 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-23T14:31:01.186 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-23T14:31:01.186 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-23T14:31:01.186 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-23T14:31:01.186 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-23T14:31:01.186 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-23T14:31:01.186 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-23T14:31:01.186 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-23T14:31:01.186 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-23T14:31:01.186 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-23T14:31:01.187 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-23T14:31:01.187 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-23T14:31:01.187 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-23T14:31:01.187 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-23T14:31:01.187 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-23T14:31:01.187 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-23T14:31:01.187 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-23T14:31:01.187 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-23T14:31:01.187 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-23T14:31:01.187 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-23T14:31:01.187 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-23T14:31:01.187 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-23T14:31:01.187 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-23T14:31:01.187 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-23T14:31:01.188 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-23T14:31:01.188 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-23T14:31:01.188 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-23T14:31:01.188 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-23T14:31:01.188 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-23T14:31:01.188 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-23T14:31:01.188 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-23T14:31:01.188 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-23T14:31:01.188 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-23T14:31:01.188 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-23T14:31:01.188 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-23T14:31:01.188 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-23T14:31:01.188 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-23T14:31:01.189 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-23T14:31:01.189 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-23T14:31:01.189 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-23T14:31:01.189 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-23T14:31:01.189 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-23T14:31:01.189 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-23T14:31:01.189 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-23T14:31:01.189 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-23T14:31:01.189 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-23T14:31:01.189 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-23T14:31:01.189 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-23T14:31:01.189 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-23T14:31:01.189 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-23T14:31:01.189 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-23T14:31:01.190 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-23T14:31:01.190 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-23T14:31:01.190 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-23T14:31:01.190 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-23T14:31:01.190 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-23T14:31:01.190 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-23T14:31:01.190 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-23T14:31:01.190 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-23T14:31:01.190 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-23T14:31:01.190 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-23T14:31:01.190 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-23T14:31:01.190 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-23T14:31:01.191 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-23T14:31:01.191 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-23T14:31:01.191 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-23T14:31:01.191 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-23T14:31:01.191 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-23T14:31:01.191 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-23T14:31:01.191 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-23T14:31:01.191 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-23T14:31:01.191 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-23T14:31:01.191 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-23T14:31:01.191 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-23T14:31:01.191 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-23T14:31:01.191 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-23T14:31:01.191 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-23T14:31:01.192 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-23T14:31:01.192 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-23T14:31:01.192 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-23T14:31:01.192 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-23T14:31:01.192 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:19.0.0-3126.gd44fddd9.el9.x86_64 149/156 2024-04-23T14:31:01.192 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-23T14:31:01.192 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:19.0.0-3126.gd44fddd9.el9.x86_64 151/156 2024-04-23T14:31:01.192 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-23T14:31:01.192 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-23T14:31:01.192 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-23T14:31:01.192 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-23T14:31:03.792 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T14:31:03.793 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:31:03.793 INFO:teuthology.orchestra.run.smithi028.stdout:Upgraded: 2024-04-23T14:31:03.793 INFO:teuthology.orchestra.run.smithi028.stdout: librados2-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:31:03.793 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:31:03.793 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-23T14:31:03.793 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-23T14:31:03.793 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-04-23T14:31:03.793 INFO:teuthology.orchestra.run.smithi028.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-23T14:31:03.793 INFO:teuthology.orchestra.run.smithi028.stdout: automake-1.16.2-8.el9.noarch 2024-04-23T14:31:03.793 INFO:teuthology.orchestra.run.smithi028.stdout: bison-3.7.4-5.el9.x86_64 2024-04-23T14:31:03.793 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-23T14:31:03.793 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:31:03.793 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:31:03.794 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:31:03.794 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:31:03.794 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:31:03.794 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:31:03.794 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:31:03.794 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:31:03.794 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:31:03.794 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:31:03.794 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:31:03.794 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:31:03.794 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:31:03.794 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:31:03.794 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:31:03.795 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:31:03.795 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:31:03.795 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:31:03.795 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:31:03.795 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:31:03.795 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:31:03.795 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:31:03.795 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-shell-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:31:03.795 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-top-2:19.0.0-3126.gd44fddd9.el9.noarch 2024-04-23T14:31:03.795 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-23T14:31:03.795 INFO:teuthology.orchestra.run.smithi028.stdout: flex-2.6.4-9.el9.x86_64 2024-04-23T14:31:03.795 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-23T14:31:03.795 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-23T14:31:03.796 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-23T14:31:03.796 INFO:teuthology.orchestra.run.smithi028.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-23T14:31:03.796 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-23T14:31:03.796 INFO:teuthology.orchestra.run.smithi028.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-23T14:31:03.796 INFO:teuthology.orchestra.run.smithi028.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-23T14:31:03.797 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-23T14:31:03.797 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-23T14:31:03.797 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:31:03.797 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:31:03.797 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:31:03.797 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-23T14:31:03.797 INFO:teuthology.orchestra.run.smithi028.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-23T14:31:03.797 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-23T14:31:03.797 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-23T14:31:03.797 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-23T14:31:03.797 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:31:03.798 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:31:03.798 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-23T14:31:03.798 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:31:03.798 INFO:teuthology.orchestra.run.smithi028.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-23T14:31:03.798 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-23T14:31:03.798 INFO:teuthology.orchestra.run.smithi028.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-23T14:31:03.798 INFO:teuthology.orchestra.run.smithi028.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-23T14:31:03.798 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-23T14:31:03.798 INFO:teuthology.orchestra.run.smithi028.stdout: lua-5.4.4-4.el9.x86_64 2024-04-23T14:31:03.798 INFO:teuthology.orchestra.run.smithi028.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-23T14:31:03.798 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-23T14:31:03.798 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-23T14:31:03.798 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-23T14:31:03.799 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-23T14:31:03.799 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-23T14:31:03.799 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-13.14-1.el9.x86_64 2024-04-23T14:31:03.799 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-23T14:31:03.799 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-23T14:31:03.799 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-23T14:31:03.799 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-23T14:31:03.799 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-23T14:31:03.799 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-23T14:31:03.799 INFO:teuthology.orchestra.run.smithi028.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-23T14:31:03.799 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-23T14:31:03.799 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-23T14:31:03.799 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-23T14:31:03.800 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:31:03.800 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:31:03.800 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:31:03.800 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-23T14:31:03.800 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-23T14:31:03.800 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-23T14:31:03.800 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-23T14:31:03.800 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-23T14:31:03.800 INFO:teuthology.orchestra.run.smithi028.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-23T14:31:03.800 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-23T14:31:03.800 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-23T14:31:03.800 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-23T14:31:03.800 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-23T14:31:03.801 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-23T14:31:03.801 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-23T14:31:03.801 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-23T14:31:03.801 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-23T14:31:03.801 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-23T14:31:03.801 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-23T14:31:03.801 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-23T14:31:03.801 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-23T14:31:03.801 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-23T14:31:03.801 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-23T14:31:03.801 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-23T14:31:03.801 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-23T14:31:03.802 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-23T14:31:03.802 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-23T14:31:03.802 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-23T14:31:03.802 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-23T14:31:03.802 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-23T14:31:03.802 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-23T14:31:03.802 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-23T14:31:03.802 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-23T14:31:03.802 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply-3.11-14.el9.noarch 2024-04-23T14:31:03.802 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-23T14:31:03.802 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-23T14:31:03.802 INFO:teuthology.orchestra.run.smithi028.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-23T14:31:03.802 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-23T14:31:03.803 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-23T14:31:03.803 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-23T14:31:03.803 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-23T14:31:03.803 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-23T14:31:03.803 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-23T14:31:03.803 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:31:03.803 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:31:03.803 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-23T14:31:03.803 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-23T14:31:03.803 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-23T14:31:03.803 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:31:03.803 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-23T14:31:03.803 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-23T14:31:03.803 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-23T14:31:03.803 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-23T14:31:03.804 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-23T14:31:03.804 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-23T14:31:03.804 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-23T14:31:03.804 INFO:teuthology.orchestra.run.smithi028.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-23T14:31:03.804 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-23T14:31:03.804 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-23T14:31:03.804 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-23T14:31:03.804 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-23T14:31:03.804 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-23T14:31:03.804 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-23T14:31:03.804 INFO:teuthology.orchestra.run.smithi028.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-23T14:31:03.804 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:31:03.804 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:31:03.805 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd-2:19.0.0-3126.gd44fddd9.el9.x86_64 2024-04-23T14:31:03.805 INFO:teuthology.orchestra.run.smithi028.stdout: re2-1:20211101-3.el9.x86_64 2024-04-23T14:31:03.805 INFO:teuthology.orchestra.run.smithi028.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-23T14:31:03.805 INFO:teuthology.orchestra.run.smithi028.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-23T14:31:03.805 INFO:teuthology.orchestra.run.smithi028.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-23T14:31:03.805 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-23T14:31:03.805 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-23T14:31:03.806 INFO:teuthology.orchestra.run.smithi028.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-23T14:31:03.806 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:31:03.806 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-23T14:31:04.372 DEBUG:teuthology.parallel:result is None 2024-04-23T14:31:04.373 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d44fddd910b5e94da0f2357af4be37d7bd495ae0 2024-04-23T14:31:04.775 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-23T14:31:04.832 INFO:teuthology.orchestra.run.smithi028.stdout:19.0.0-3126.gd44fddd9.el9 2024-04-23T14:31:04.833 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3126.gd44fddd9.el9 2024-04-23T14:31:04.833 INFO:teuthology.task.install:The correct ceph version 19.0.0-3126.gd44fddd9 is installed. 2024-04-23T14:31:04.835 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d44fddd910b5e94da0f2357af4be37d7bd495ae0 2024-04-23T14:31:05.193 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-23T14:31:05.219 INFO:teuthology.orchestra.run.smithi038.stdout:19.0.0-3126.gd44fddd9.el9 2024-04-23T14:31:05.220 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3126.gd44fddd9.el9 2024-04-23T14:31:05.220 INFO:teuthology.task.install:The correct ceph version 19.0.0-3126.gd44fddd9 is installed. 2024-04-23T14:31:05.222 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d44fddd910b5e94da0f2357af4be37d7bd495ae0 2024-04-23T14:31:05.446 DEBUG:teuthology.orchestra.run.smithi105:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-23T14:31:05.472 INFO:teuthology.orchestra.run.smithi105.stdout:19.0.0-3126.gd44fddd9.el9 2024-04-23T14:31:05.472 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3126.gd44fddd9.el9 2024-04-23T14:31:05.472 INFO:teuthology.task.install:The correct ceph version 19.0.0-3126.gd44fddd9 is installed. 2024-04-23T14:31:05.474 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-23T14:31:05.474 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-23T14:31:05.474 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-23T14:31:05.542 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-23T14:31:05.542 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-23T14:31:05.575 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-23T14:31:05.575 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-23T14:31:05.606 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-23T14:31:05.606 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-23T14:31:05.606 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/daemon-helper 2024-04-23T14:31:05.667 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-23T14:31:05.768 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-23T14:31:05.768 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/daemon-helper 2024-04-23T14:31:05.803 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-23T14:31:05.870 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-23T14:31:05.870 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/daemon-helper 2024-04-23T14:31:05.899 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-23T14:31:05.966 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-23T14:31:05.966 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-23T14:31:05.966 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-23T14:31:06.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-23T14:31:06.139 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-23T14:31:06.139 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-23T14:31:06.165 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-23T14:31:06.231 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-23T14:31:06.231 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-23T14:31:06.260 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-23T14:31:06.328 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-23T14:31:06.328 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-23T14:31:06.328 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/stdin-killer 2024-04-23T14:31:06.390 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-23T14:31:06.492 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-23T14:31:06.492 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/stdin-killer 2024-04-23T14:31:06.517 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-23T14:31:06.582 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-23T14:31:06.583 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/stdin-killer 2024-04-23T14:31:06.607 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-23T14:31:06.671 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-23T14:31:06.772 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 1, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': 'd44fddd910b5e94da0f2357af4be37d7bd495ae0', 'subvols': {'create': 2}} 2024-04-23T14:31:06.772 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 2024-04-23T14:31:06.773 INFO:tasks.cephadm:Cluster fsid is 1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T14:31:06.773 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-23T14:31:06.773 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.28', 'mon.b': '172.21.15.38', 'mon.c': '172.21.15.105'} 2024-04-23T14:31:06.773 INFO:tasks.cephadm:First mon is mon.a on smithi028 2024-04-23T14:31:06.773 INFO:tasks.cephadm:First mgr is x 2024-04-23T14:31:06.773 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-23T14:31:06.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo hostname $(hostname -s) 2024-04-23T14:31:06.841 DEBUG:teuthology.orchestra.run.smithi038:> sudo hostname $(hostname -s) 2024-04-23T14:31:06.865 DEBUG:teuthology.orchestra.run.smithi105:> sudo hostname $(hostname -s) 2024-04-23T14:31:06.890 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-23T14:31:06.890 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d44fddd910b5e94da0f2357af4be37d7bd495ae0 2024-04-23T14:31:07.032 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'd44fddd910b5e94da0f2357af4be37d7bd495ae0', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78800/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3126-gd44fddd9', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3126.gd44fddd9'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240416.193735-3/d44fddd910b5e94da0f2357af4be37d7bd495ae0/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-22 14:41:17.415668', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-rishabh-testing-20240416.193735-3', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-rishabh-testing-20240416.193735-3/d44fddd910b5e94da0f2357af4be37d7bd495ae0/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-23T14:31:07.176 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-rishabh-testing-20240416.193735-3, sha1 d44fddd910b5e94da0f2357af4be37d7bd495ae0 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=d44fddd910b5e94da0f2357af4be37d7bd495ae0 2024-04-23T14:31:07.178 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240416.193735-3/d44fddd910b5e94da0f2357af4be37d7bd495ae0/centos/9/x86_64/flavors/default/cephadm 2024-04-23T14:31:07.178 DEBUG:teuthology.orchestra.run.smithi028:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240416.193735-3/d44fddd910b5e94da0f2357af4be37d7bd495ae0/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-23T14:31:08.632 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r--. 1 ubuntu ubuntu 781447 Apr 23 14:31 /home/ubuntu/cephtest/cephadm 2024-04-23T14:31:08.632 DEBUG:teuthology.orchestra.run.smithi038:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240416.193735-3/d44fddd910b5e94da0f2357af4be37d7bd495ae0/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-23T14:31:10.526 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-r--r--. 1 ubuntu ubuntu 781447 Apr 23 14:31 /home/ubuntu/cephtest/cephadm 2024-04-23T14:31:10.527 DEBUG:teuthology.orchestra.run.smithi105:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240416.193735-3/d44fddd910b5e94da0f2357af4be37d7bd495ae0/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-23T14:31:11.732 INFO:teuthology.orchestra.run.smithi105.stdout:-rw-r--r--. 1 ubuntu ubuntu 781447 Apr 23 14:31 /home/ubuntu/cephtest/cephadm 2024-04-23T14:31:11.732 DEBUG:teuthology.orchestra.run.smithi028:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-23T14:31:11.781 DEBUG:teuthology.orchestra.run.smithi038:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-23T14:31:11.801 DEBUG:teuthology.orchestra.run.smithi105:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-23T14:31:11.836 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 on all hosts... 2024-04-23T14:31:11.836 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 pull 2024-04-23T14:31:11.841 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 pull 2024-04-23T14:31:11.844 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 pull 2024-04-23T14:31:13.394 INFO:teuthology.orchestra.run.smithi105.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0... 2024-04-23T14:31:13.477 INFO:teuthology.orchestra.run.smithi038.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0... 2024-04-23T14:31:14.980 INFO:teuthology.orchestra.run.smithi028.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0... 2024-04-23T14:31:42.686 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-23T14:31:42.687 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_version": "ceph version 19.0.0-3126-gd44fddd9 (d44fddd910b5e94da0f2357af4be37d7bd495ae0) squid (dev)", 2024-04-23T14:31:42.687 INFO:teuthology.orchestra.run.smithi105.stdout: "image_id": "76e77f280c74312ca1add9b6019a4a921a516f91795d85bae00a0c8ca9b1c62e", 2024-04-23T14:31:42.687 INFO:teuthology.orchestra.run.smithi105.stdout: "repo_digests": [ 2024-04-23T14:31:42.687 INFO:teuthology.orchestra.run.smithi105.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd" 2024-04-23T14:31:42.687 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-23T14:31:42.687 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-23T14:31:42.690 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-23T14:31:42.690 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_version": "ceph version 19.0.0-3126-gd44fddd9 (d44fddd910b5e94da0f2357af4be37d7bd495ae0) squid (dev)", 2024-04-23T14:31:42.690 INFO:teuthology.orchestra.run.smithi038.stdout: "image_id": "76e77f280c74312ca1add9b6019a4a921a516f91795d85bae00a0c8ca9b1c62e", 2024-04-23T14:31:42.690 INFO:teuthology.orchestra.run.smithi038.stdout: "repo_digests": [ 2024-04-23T14:31:42.690 INFO:teuthology.orchestra.run.smithi038.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd" 2024-04-23T14:31:42.690 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-04-23T14:31:42.690 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-23T14:32:11.240 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:32:11.241 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 19.0.0-3126-gd44fddd9 (d44fddd910b5e94da0f2357af4be37d7bd495ae0) squid (dev)", 2024-04-23T14:32:11.241 INFO:teuthology.orchestra.run.smithi028.stdout: "image_id": "76e77f280c74312ca1add9b6019a4a921a516f91795d85bae00a0c8ca9b1c62e", 2024-04-23T14:32:11.241 INFO:teuthology.orchestra.run.smithi028.stdout: "repo_digests": [ 2024-04-23T14:32:11.241 INFO:teuthology.orchestra.run.smithi028.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd" 2024-04-23T14:32:11.241 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-23T14:32:11.241 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:32:11.308 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /etc/ceph 2024-04-23T14:32:11.404 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /etc/ceph 2024-04-23T14:32:11.437 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /etc/ceph 2024-04-23T14:32:11.468 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 777 /etc/ceph 2024-04-23T14:32:11.564 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 777 /etc/ceph 2024-04-23T14:32:11.593 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 777 /etc/ceph 2024-04-23T14:32:11.621 INFO:tasks.cephadm:Writing seed config... 2024-04-23T14:32:11.622 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-23T14:32:11.622 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-23T14:32:11.622 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-23T14:32:11.623 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-23T14:32:11.623 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-23T14:32:11.623 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-23T14:32:11.623 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-23T14:32:11.623 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-23T14:32:11.623 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-23T14:32:11.623 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-23T14:32:11.623 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-23T14:32:11.623 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-23T14:32:11.623 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-23T14:32:11.623 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-23T14:32:11.623 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-23T14:32:11.623 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-23T14:32:11.623 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-23T14:32:11.624 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-23T14:32:11.624 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-23T14:32:11.624 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-23T14:32:11.624 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-23T14:32:11.624 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-23T14:32:11.624 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-23T14:32:11.624 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-23T14:32:11.624 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-23T14:32:11.624 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-23T14:32:11.624 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-23T14:32:11.624 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-23T14:32:11.624 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-23T14:32:11.624 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-23T14:32:11.624 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-04-23T14:32:11.624 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-04-23T14:32:11.624 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-04-23T14:32:11.624 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-23T14:32:11.625 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-23T14:32:11.625 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-04-23T14:32:11.625 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-04-23T14:32:11.625 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-23T14:32:11.625 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-23T14:32:11.625 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-04-23T14:32:11.625 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-23T14:32:11.625 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-23T14:32:11.625 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-23T14:32:11.625 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-23T14:32:11.625 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-23T14:32:11.625 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-23T14:32:11.625 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-23T14:32:11.625 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-23T14:32:11.626 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-23T14:32:11.626 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-23T14:32:11.685 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 1ee6e996-017e-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-23T14:32:11.685 DEBUG:teuthology.orchestra.run.smithi028:mon.a> sudo journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mon.a.service 2024-04-23T14:32:11.733 DEBUG:teuthology.orchestra.run.smithi028:mgr.x> sudo journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mgr.x.service 2024-04-23T14:32:11.779 INFO:tasks.cephadm:Bootstrapping... 2024-04-23T14:32:11.779 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 -v bootstrap --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.28 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-23T14:32:12.066 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-04-23T14:32:12.066 INFO:teuthology.orchestra.run.smithi028.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0', '-v', 'bootstrap', '--fsid', '1ee6e996-017e-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.28', '--skip-admin-label'] 2024-04-23T14:32:12.067 INFO:teuthology.orchestra.run.smithi028.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-23T14:32:12.067 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying podman|docker is present... 2024-04-23T14:32:12.125 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-23T14:32:12.125 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying lvm2 is present... 2024-04-23T14:32:12.125 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying time synchronization is in place... 2024-04-23T14:32:12.142 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-23T14:32:12.142 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-23T14:32:12.158 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-23T14:32:12.158 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-04-23T14:32:12.176 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout enabled 2024-04-23T14:32:12.198 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout active 2024-04-23T14:32:12.199 INFO:teuthology.orchestra.run.smithi028.stdout:Unit chronyd.service is enabled and running 2024-04-23T14:32:12.199 INFO:teuthology.orchestra.run.smithi028.stdout:Repeating the final host check... 2024-04-23T14:32:12.260 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-23T14:32:12.260 INFO:teuthology.orchestra.run.smithi028.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-23T14:32:12.260 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl is present 2024-04-23T14:32:12.260 INFO:teuthology.orchestra.run.smithi028.stdout:lvcreate is present 2024-04-23T14:32:12.278 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-23T14:32:12.278 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-23T14:32:12.298 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-23T14:32:12.298 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-04-23T14:32:12.324 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout enabled 2024-04-23T14:32:12.346 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout active 2024-04-23T14:32:12.346 INFO:teuthology.orchestra.run.smithi028.stdout:Unit chronyd.service is enabled and running 2024-04-23T14:32:12.347 INFO:teuthology.orchestra.run.smithi028.stdout:Host looks OK 2024-04-23T14:32:12.347 INFO:teuthology.orchestra.run.smithi028.stdout:Cluster fsid: 1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T14:32:12.347 INFO:teuthology.orchestra.run.smithi028.stdout:Acquiring lock 140599141334176 on /run/cephadm/1ee6e996-017e-11ef-bc93-c7b262605968.lock 2024-04-23T14:32:12.348 INFO:teuthology.orchestra.run.smithi028.stdout:Lock 140599141334176 acquired on /run/cephadm/1ee6e996-017e-11ef-bc93-c7b262605968.lock 2024-04-23T14:32:12.349 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying IP 172.21.15.28 port 3300 ... 2024-04-23T14:32:12.350 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying IP 172.21.15.28 port 6789 ... 2024-04-23T14:32:12.350 INFO:teuthology.orchestra.run.smithi028.stdout:Base mon IP(s) is [172.21.15.28:3300, 172.21.15.28:6789], mon addrv is [v2:172.21.15.28:3300,v1:172.21.15.28:6789] 2024-04-23T14:32:12.366 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.28 metric 100 2024-04-23T14:32:12.366 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.28 metric 100 2024-04-23T14:32:12.379 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-23T14:32:12.379 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-23T14:32:12.379 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-23T14:32:12.390 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-23T14:32:12.391 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-23T14:32:12.391 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-23T14:32:12.391 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-23T14:32:12.391 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link noprefixroute 2024-04-23T14:32:12.391 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-23T14:32:12.394 INFO:teuthology.orchestra.run.smithi028.stdout:Mon IP `172.21.15.28` is in CIDR network `172.21.0.0/20` 2024-04-23T14:32:12.394 INFO:teuthology.orchestra.run.smithi028.stdout:Mon IP `172.21.15.28` is in CIDR network `172.21.0.0/20` 2024-04-23T14:32:12.394 INFO:teuthology.orchestra.run.smithi028.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-23T14:32:12.397 INFO:teuthology.orchestra.run.smithi028.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-23T14:32:12.400 INFO:teuthology.orchestra.run.smithi028.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0... 2024-04-23T14:32:13.096 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout 76e77f280c74312ca1add9b6019a4a921a516f91795d85bae00a0c8ca9b1c62e 2024-04-23T14:32:13.097 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0... 2024-04-23T14:32:13.097 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Getting image source signatures 2024-04-23T14:32:13.097 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying blob sha256:871ee18b3c7f87d5ff3fbc84e47e46994673cad1275c5c818c1e7168af8e5da2 2024-04-23T14:32:13.097 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying blob sha256:426fede9fffa2c7dfe40a4f9322a34794b72405170134d54d2c30d9cc6c6a5be 2024-04-23T14:32:13.097 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying config sha256:76e77f280c74312ca1add9b6019a4a921a516f91795d85bae00a0c8ca9b1c62e 2024-04-23T14:32:13.097 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-23T14:32:25.146 INFO:teuthology.orchestra.run.smithi028.stdout:ceph: stdout ceph version 19.0.0-3126-gd44fddd9 (d44fddd910b5e94da0f2357af4be37d7bd495ae0) squid (dev) 2024-04-23T14:32:25.146 INFO:teuthology.orchestra.run.smithi028.stdout:Ceph version: ceph version 19.0.0-3126-gd44fddd9 (d44fddd910b5e94da0f2357af4be37d7bd495ae0) squid (dev) 2024-04-23T14:32:25.146 INFO:teuthology.orchestra.run.smithi028.stdout:Extracting ceph user uid/gid from container image... 2024-04-23T14:32:26.260 INFO:teuthology.orchestra.run.smithi028.stdout:stat: stdout 167 167 2024-04-23T14:32:26.260 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial keys... 2024-04-23T14:32:27.432 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQB6xidmzIgCKxAAh4ARohF+0KjIYmufbKegjw== 2024-04-23T14:32:28.501 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQB7xidmex+/NxAAzcW1DmEiuiG6jV7iL0ldug== 2024-04-23T14:32:29.834 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQB8xidmyLFDOBAAy3C7Buq6wtkL6g4bCawzTA== 2024-04-23T14:32:29.835 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial monmap... 2024-04-23T14:32:31.045 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-23T14:32:31.046 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-23T14:32:31.046 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T14:32:31.046 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-23T14:32:31.046 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool for a [v2:172.21.15.28:3300,v1:172.21.15.28:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-23T14:32:31.046 INFO:teuthology.orchestra.run.smithi028.stdout:setting min_mon_release = quincy 2024-04-23T14:32:31.046 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: set fsid to 1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T14:32:31.046 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-23T14:32:31.047 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:32:31.047 INFO:teuthology.orchestra.run.smithi028.stdout:Creating mon... 2024-04-23T14:32:32.167 INFO:teuthology.orchestra.run.smithi028.stdout:create mon.a on 2024-04-23T14:32:32.553 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-23T14:32:32.804 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-23T14:32:33.129 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-1ee6e996-017e-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-1ee6e996-017e-11ef-bc93-c7b262605968.target. 2024-04-23T14:32:33.129 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-1ee6e996-017e-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-1ee6e996-017e-11ef-bc93-c7b262605968.target. 2024-04-23T14:32:33.495 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mon.a 2024-04-23T14:32:33.495 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to reset failed state of unit ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-23T14:32:33.795 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-1ee6e996-017e-11ef-bc93-c7b262605968.target.wants/ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-1ee6e996-017e-11ef-bc93-c7b262605968@.service. 2024-04-23T14:32:34.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:34 smithi028 podman[27426]: 2024-04-23 14:32:34.202534076 +0000 UTC m=+0.134960341 container create b1c4012b52eca4997ecbe5232ef0a767d4a16f089d503bcf29ce87110143a863 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-23T14:32:34.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:34 smithi028 podman[27426]: 2024-04-23 14:32:34.489293158 +0000 UTC m=+0.421719430 container init b1c4012b52eca4997ecbe5232ef0a767d4a16f089d503bcf29ce87110143a863 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mon-a, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-23T14:32:34.601 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-23T14:32:34.601 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-04-23T14:32:34.616 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-23T14:32:34.617 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-04-23T14:32:34.617 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-04-23T14:32:34.617 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to enable service . firewalld.service is not available 2024-04-23T14:32:34.617 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mon to start... 2024-04-23T14:32:34.617 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mon... 2024-04-23T14:32:34.765 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:34 smithi028 podman[27426]: 2024-04-23 14:32:34.500320007 +0000 UTC m=+0.432746245 container start b1c4012b52eca4997ecbe5232ef0a767d4a16f089d503bcf29ce87110143a863 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mon-a, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-23T14:32:34.765 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:34 smithi028 bash[27426]: b1c4012b52eca4997ecbe5232ef0a767d4a16f089d503bcf29ce87110143a863 2024-04-23T14:32:34.765 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:34 smithi028 systemd[1]: Started Ceph mon.a for 1ee6e996-017e-11ef-bc93-c7b262605968. 2024-04-23T14:32:34.765 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:34 smithi028 ceph-mon[27444]: mkfs 1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T14:32:34.765 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:34 smithi028 ceph-mon[27444]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-23T14:32:35.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:35 smithi028 ceph-mon[27444]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-23T14:32:35.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:35 smithi028 ceph-mon[27444]: monmap epoch 1 2024-04-23T14:32:35.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:35 smithi028 ceph-mon[27444]: fsid 1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T14:32:35.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:35 smithi028 ceph-mon[27444]: last_changed 2024-04-23T14:32:30.337765+0000 2024-04-23T14:32:35.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:35 smithi028 ceph-mon[27444]: created 2024-04-23T14:32:30.337765+0000 2024-04-23T14:32:35.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:35 smithi028 ceph-mon[27444]: min_mon_release 19 (squid) 2024-04-23T14:32:35.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:35 smithi028 ceph-mon[27444]: election_strategy: 1 2024-04-23T14:32:35.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:35 smithi028 ceph-mon[27444]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-23T14:32:35.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:35 smithi028 ceph-mon[27444]: fsmap 2024-04-23T14:32:35.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:35 smithi028 ceph-mon[27444]: osdmap e1: 0 total, 0 up, 0 in 2024-04-23T14:32:35.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:35 smithi028 ceph-mon[27444]: mgrmap e1: no daemons active 2024-04-23T14:32:35.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:35 smithi028 ceph-mon[27444]: from='client.? 172.21.15.28:0/2765029308' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-23T14:32:35.913 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout cluster: 2024-04-23T14:32:35.925 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout id: 1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T14:32:35.925 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-23T14:32:35.925 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-23T14:32:35.925 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout services: 2024-04-23T14:32:35.925 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.736051s) 2024-04-23T14:32:35.926 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-23T14:32:35.926 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-23T14:32:35.926 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-23T14:32:35.926 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout data: 2024-04-23T14:32:35.926 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-23T14:32:35.927 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-23T14:32:35.927 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-23T14:32:35.927 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout pgs: 2024-04-23T14:32:35.927 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-23T14:32:35.927 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.220+0000 7f1285000700 1 Processor -- start 2024-04-23T14:32:35.928 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.220+0000 7f1285000700 1 -- start start 2024-04-23T14:32:35.928 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.221+0000 7f1285000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1280106e90 0x7f12801072b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:35.928 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.221+0000 7f1285000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f128007be20 con 0x7f1280106e90 2024-04-23T14:32:35.928 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.221+0000 7f127f400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1280106e90 0x7f12801072b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:35.928 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.222+0000 7f127f400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1280106e90 0x7f12801072b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47284/0 (socket says 172.21.15.28:47284) 2024-04-23T14:32:35.928 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.222+0000 7f127f400700 1 -- 172.21.15.28:0/356706581 learned_addr learned my addr 172.21.15.28:0/356706581 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:32:35.929 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.222+0000 7f127f400700 1 -- 172.21.15.28:0/356706581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12801077f0 con 0x7f1280106e90 2024-04-23T14:32:35.929 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.222+0000 7f127f400700 1 --2- 172.21.15.28:0/356706581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1280106e90 0x7f12801072b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1270009e80 tx=0x7f127000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dba642b012330a51 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:35.929 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.223+0000 7f127ea00700 1 -- 172.21.15.28:0/356706581 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1270017070 con 0x7f1280106e90 2024-04-23T14:32:35.929 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.223+0000 7f127ea00700 1 -- 172.21.15.28:0/356706581 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1270003a20 con 0x7f1280106e90 2024-04-23T14:32:35.929 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.223+0000 7f127ea00700 1 -- 172.21.15.28:0/356706581 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1270003d20 con 0x7f1280106e90 2024-04-23T14:32:35.930 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.223+0000 7f1285000700 1 -- 172.21.15.28:0/356706581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1280106e90 msgr2=0x7f12801072b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:35.930 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.223+0000 7f1285000700 1 --2- 172.21.15.28:0/356706581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1280106e90 0x7f12801072b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1270009e80 tx=0x7f127000b330 comp rx=0 tx=0).stop 2024-04-23T14:32:35.930 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.223+0000 7f1285000700 1 -- 172.21.15.28:0/356706581 shutdown_connections 2024-04-23T14:32:35.930 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.223+0000 7f1285000700 1 --2- 172.21.15.28:0/356706581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1280106e90 0x7f12801072b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:35.930 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.223+0000 7f1285000700 1 -- 172.21.15.28:0/356706581 >> 172.21.15.28:0/356706581 conn(0x7f1280102a70 msgr2=0x7f1280104e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:35.930 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.224+0000 7f1285000700 1 -- 172.21.15.28:0/356706581 shutdown_connections 2024-04-23T14:32:35.931 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.224+0000 7f1285000700 1 -- 172.21.15.28:0/356706581 wait complete. 2024-04-23T14:32:35.931 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.225+0000 7f1285000700 1 Processor -- start 2024-04-23T14:32:35.931 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.225+0000 7f1285000700 1 -- start start 2024-04-23T14:32:35.932 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.225+0000 7f1285000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1280106e90 0x7f128019de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:35.932 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.225+0000 7f1285000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1280107d20 con 0x7f1280106e90 2024-04-23T14:32:35.932 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.225+0000 7f127f400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1280106e90 0x7f128019de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:35.932 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.225+0000 7f127f400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1280106e90 0x7f128019de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47290/0 (socket says 172.21.15.28:47290) 2024-04-23T14:32:35.932 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.225+0000 7f127f400700 1 -- 172.21.15.28:0/2765029308 learned_addr learned my addr 172.21.15.28:0/2765029308 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:32:35.932 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.226+0000 7f127f400700 1 -- 172.21.15.28:0/2765029308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f128019e370 con 0x7f1280106e90 2024-04-23T14:32:35.933 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.226+0000 7f127f400700 1 --2- 172.21.15.28:0/2765029308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1280106e90 0x7f128019de30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f128007b260 tx=0x7f1270004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:35.933 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.226+0000 7f127cc00700 1 -- 172.21.15.28:0/2765029308 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1270022020 con 0x7f1280106e90 2024-04-23T14:32:35.933 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.227+0000 7f127cc00700 1 -- 172.21.15.28:0/2765029308 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1270018040 con 0x7f1280106e90 2024-04-23T14:32:35.933 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.227+0000 7f127cc00700 1 -- 172.21.15.28:0/2765029308 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1270017070 con 0x7f1280106e90 2024-04-23T14:32:35.933 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.227+0000 7f1285000700 1 -- 172.21.15.28:0/2765029308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f128019e600 con 0x7f1280106e90 2024-04-23T14:32:35.933 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.227+0000 7f1285000700 1 -- 172.21.15.28:0/2765029308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12801a12f0 con 0x7f1280106e90 2024-04-23T14:32:35.934 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.227+0000 7f127cc00700 1 -- 172.21.15.28:0/2765029308 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1270025020 con 0x7f1280106e90 2024-04-23T14:32:35.934 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.227+0000 7f127cc00700 1 -- 172.21.15.28:0/2765029308 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f127000f050 con 0x7f1280106e90 2024-04-23T14:32:35.934 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.228+0000 7f1285000700 1 -- 172.21.15.28:0/2765029308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1280064150 con 0x7f1280106e90 2024-04-23T14:32:35.934 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.231+0000 7f127cc00700 1 -- 172.21.15.28:0/2765029308 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f127002c030 con 0x7f1280106e90 2024-04-23T14:32:35.934 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.314+0000 7f1285000700 1 -- 172.21.15.28:0/2765029308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f12801a1760 con 0x7f1280106e90 2024-04-23T14:32:35.935 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.314+0000 7f127cc00700 1 -- 172.21.15.28:0/2765029308 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f127001e480 con 0x7f1280106e90 2024-04-23T14:32:35.935 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.316+0000 7f1285000700 1 -- 172.21.15.28:0/2765029308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1280106e90 msgr2=0x7f128019de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:35.935 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.316+0000 7f1285000700 1 --2- 172.21.15.28:0/2765029308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1280106e90 0x7f128019de30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f128007b260 tx=0x7f1270004ad0 comp rx=0 tx=0).stop 2024-04-23T14:32:35.935 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.317+0000 7f1285000700 1 -- 172.21.15.28:0/2765029308 shutdown_connections 2024-04-23T14:32:35.935 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.317+0000 7f1285000700 1 --2- 172.21.15.28:0/2765029308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1280106e90 0x7f128019de30 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:35.935 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.317+0000 7f1285000700 1 -- 172.21.15.28:0/2765029308 >> 172.21.15.28:0/2765029308 conn(0x7f1280102a70 msgr2=0x7f1280194450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:35.935 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.317+0000 7f1285000700 1 -- 172.21.15.28:0/2765029308 shutdown_connections 2024-04-23T14:32:35.935 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:35.317+0000 7f1285000700 1 -- 172.21.15.28:0/2765029308 wait complete. 2024-04-23T14:32:35.936 INFO:teuthology.orchestra.run.smithi028.stdout:mon is available 2024-04-23T14:32:35.936 INFO:teuthology.orchestra.run.smithi028.stdout:Assimilating anything we can from ceph.conf... 2024-04-23T14:32:36.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:36 smithi028 ceph-mon[27444]: from='client.? 172.21.15.28:0/1663713079' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-23T14:32:36.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:36 smithi028 ceph-mon[27444]: from='client.? 172.21.15.28:0/1663713079' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-23T14:32:37.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-23T14:32:37.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [global] 2024-04-23T14:32:37.326 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout fsid = 1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T14:32:37.326 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.28:3300,v1:172.21.15.28:6789] 2024-04-23T14:32:37.326 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-23T14:32:37.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-23T14:32:37.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-23T14:32:37.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-23T14:32:37.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-23T14:32:37.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-23T14:32:37.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-23T14:32:37.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-23T14:32:37.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [osd] 2024-04-23T14:32:37.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-04-23T14:32:37.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-23T14:32:37.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-23T14:32:37.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-23T14:32:37.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-23T14:32:37.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-23T14:32:37.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.548+0000 7f9f75000700 1 Processor -- start 2024-04-23T14:32:37.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.548+0000 7f9f75000700 1 -- start start 2024-04-23T14:32:37.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.549+0000 7f9f75000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f70108790 0x7f9f70108bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:37.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.549+0000 7f9f75000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9f70109180 con 0x7f9f70108790 2024-04-23T14:32:37.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.549+0000 7f9f6de00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f70108790 0x7f9f70108bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:37.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.549+0000 7f9f6de00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f70108790 0x7f9f70108bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47300/0 (socket says 172.21.15.28:47300) 2024-04-23T14:32:37.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.549+0000 7f9f6de00700 1 -- 172.21.15.28:0/2305632369 learned_addr learned my addr 172.21.15.28:0/2305632369 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:32:37.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.550+0000 7f9f6de00700 1 -- 172.21.15.28:0/2305632369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f70109940 con 0x7f9f70108790 2024-04-23T14:32:37.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.550+0000 7f9f6de00700 1 --2- 172.21.15.28:0/2305632369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f70108790 0x7f9f70108bb0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f9f600098a0 tx=0x7f9f60009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5e37cb57c95b4ba server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:37.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.550+0000 7f9f6ca00700 1 -- 172.21.15.28:0/2305632369 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f60004160 con 0x7f9f70108790 2024-04-23T14:32:37.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.551+0000 7f9f6ca00700 1 -- 172.21.15.28:0/2305632369 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f60004300 con 0x7f9f70108790 2024-04-23T14:32:37.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.551+0000 7f9f6ca00700 1 -- 172.21.15.28:0/2305632369 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f600036a0 con 0x7f9f70108790 2024-04-23T14:32:37.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.551+0000 7f9f75000700 1 -- 172.21.15.28:0/2305632369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f70108790 msgr2=0x7f9f70108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:37.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.551+0000 7f9f75000700 1 --2- 172.21.15.28:0/2305632369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f70108790 0x7f9f70108bb0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f9f600098a0 tx=0x7f9f60009bb0 comp rx=0 tx=0).stop 2024-04-23T14:32:37.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.551+0000 7f9f75000700 1 -- 172.21.15.28:0/2305632369 shutdown_connections 2024-04-23T14:32:37.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.551+0000 7f9f75000700 1 --2- 172.21.15.28:0/2305632369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f70108790 0x7f9f70108bb0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:37.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.551+0000 7f9f75000700 1 -- 172.21.15.28:0/2305632369 >> 172.21.15.28:0/2305632369 conn(0x7f9f70103d70 msgr2=0x7f9f70106190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:37.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.552+0000 7f9f75000700 1 -- 172.21.15.28:0/2305632369 shutdown_connections 2024-04-23T14:32:37.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.552+0000 7f9f75000700 1 -- 172.21.15.28:0/2305632369 wait complete. 2024-04-23T14:32:37.335 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.553+0000 7f9f75000700 1 Processor -- start 2024-04-23T14:32:37.335 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.553+0000 7f9f75000700 1 -- start start 2024-04-23T14:32:37.335 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.553+0000 7f9f75000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f70108790 0x7f9f7019be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:37.335 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.553+0000 7f9f75000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9f70109e70 con 0x7f9f70108790 2024-04-23T14:32:37.337 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.554+0000 7f9f6de00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f70108790 0x7f9f7019be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:37.337 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.554+0000 7f9f6de00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f70108790 0x7f9f7019be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47314/0 (socket says 172.21.15.28:47314) 2024-04-23T14:32:37.337 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.554+0000 7f9f6de00700 1 -- 172.21.15.28:0/1663713079 learned_addr learned my addr 172.21.15.28:0/1663713079 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:32:37.337 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.554+0000 7f9f6de00700 1 -- 172.21.15.28:0/1663713079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f7019c3a0 con 0x7f9f70108790 2024-04-23T14:32:37.337 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.554+0000 7f9f6de00700 1 --2- 172.21.15.28:0/1663713079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f70108790 0x7f9f7019be60 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f9f60000c00 tx=0x7f9f60004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:37.337 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.555+0000 7f9f66a00700 1 -- 172.21.15.28:0/1663713079 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f60003db0 con 0x7f9f70108790 2024-04-23T14:32:37.337 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.555+0000 7f9f66a00700 1 -- 172.21.15.28:0/1663713079 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f60011630 con 0x7f9f70108790 2024-04-23T14:32:37.337 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.555+0000 7f9f66a00700 1 -- 172.21.15.28:0/1663713079 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f60011950 con 0x7f9f70108790 2024-04-23T14:32:37.338 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.555+0000 7f9f75000700 1 -- 172.21.15.28:0/1663713079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f7019c630 con 0x7f9f70108790 2024-04-23T14:32:37.338 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.555+0000 7f9f75000700 1 -- 172.21.15.28:0/1663713079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f7019f320 con 0x7f9f70108790 2024-04-23T14:32:37.338 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.555+0000 7f9f66a00700 1 -- 172.21.15.28:0/1663713079 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9f6001a460 con 0x7f9f70108790 2024-04-23T14:32:37.338 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.555+0000 7f9f66a00700 1 -- 172.21.15.28:0/1663713079 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9f6001baf0 con 0x7f9f70108790 2024-04-23T14:32:37.338 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.556+0000 7f9f75000700 1 -- 172.21.15.28:0/1663713079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f70064150 con 0x7f9f70108790 2024-04-23T14:32:37.338 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.559+0000 7f9f66a00700 1 -- 172.21.15.28:0/1663713079 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f9f60004260 con 0x7f9f70108790 2024-04-23T14:32:37.338 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.638+0000 7f9f75000700 1 -- 172.21.15.28:0/1663713079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f9f7019cb10 con 0x7f9f70108790 2024-04-23T14:32:37.338 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.648+0000 7f9f66a00700 1 -- 172.21.15.28:0/1663713079 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+456 (secure 0 0 0) 0x7f9f7019cb10 con 0x7f9f70108790 2024-04-23T14:32:37.338 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.649+0000 7f9f75000700 1 -- 172.21.15.28:0/1663713079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f70108790 msgr2=0x7f9f7019be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:37.338 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.649+0000 7f9f75000700 1 --2- 172.21.15.28:0/1663713079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f70108790 0x7f9f7019be60 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f9f60000c00 tx=0x7f9f60004750 comp rx=0 tx=0).stop 2024-04-23T14:32:37.338 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.649+0000 7f9f75000700 1 -- 172.21.15.28:0/1663713079 shutdown_connections 2024-04-23T14:32:37.338 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.649+0000 7f9f75000700 1 --2- 172.21.15.28:0/1663713079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f70108790 0x7f9f7019be60 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:37.338 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.649+0000 7f9f75000700 1 -- 172.21.15.28:0/1663713079 >> 172.21.15.28:0/1663713079 conn(0x7f9f70103d70 msgr2=0x7f9f70192a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:37.339 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.650+0000 7f9f75000700 1 -- 172.21.15.28:0/1663713079 shutdown_connections 2024-04-23T14:32:37.339 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:36.650+0000 7f9f75000700 1 -- 172.21.15.28:0/1663713079 wait complete. 2024-04-23T14:32:37.339 INFO:teuthology.orchestra.run.smithi028.stdout:Generating new minimal ceph.conf... 2024-04-23T14:32:38.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:38 smithi028 ceph-mon[27444]: from='client.? 172.21.15.28:0/4066528062' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:32:38.600 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.910+0000 7f83db200700 1 Processor -- start 2024-04-23T14:32:38.600 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.910+0000 7f83db200700 1 -- start start 2024-04-23T14:32:38.600 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.911+0000 7f83db200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83dc10a840 0x7f83dc10ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:38.600 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.911+0000 7f83db200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f83dc10b230 con 0x7f83dc10a840 2024-04-23T14:32:38.601 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.912+0000 7f83d9e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83dc10a840 0x7f83dc10ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:38.601 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.912+0000 7f83d9e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83dc10a840 0x7f83dc10ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55738/0 (socket says 172.21.15.28:55738) 2024-04-23T14:32:38.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.912+0000 7f83d9e00700 1 -- 172.21.15.28:0/2340078638 learned_addr learned my addr 172.21.15.28:0/2340078638 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:32:38.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.912+0000 7f83d9e00700 1 -- 172.21.15.28:0/2340078638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83dc10b9f0 con 0x7f83dc10a840 2024-04-23T14:32:38.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.913+0000 7f83d9e00700 1 --2- 172.21.15.28:0/2340078638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83dc10a840 0x7f83dc10ac60 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f83cc0098a0 tx=0x7f83cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fa567da29f5eb4f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:38.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.913+0000 7f83d8a00700 1 -- 172.21.15.28:0/2340078638 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f83cc004160 con 0x7f83dc10a840 2024-04-23T14:32:38.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.913+0000 7f83d8a00700 1 -- 172.21.15.28:0/2340078638 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f83cc004300 con 0x7f83dc10a840 2024-04-23T14:32:38.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.913+0000 7f83d8a00700 1 -- 172.21.15.28:0/2340078638 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f83cc011b50 con 0x7f83dc10a840 2024-04-23T14:32:38.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.914+0000 7f83db200700 1 -- 172.21.15.28:0/2340078638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83dc10a840 msgr2=0x7f83dc10ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:38.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.914+0000 7f83db200700 1 --2- 172.21.15.28:0/2340078638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83dc10a840 0x7f83dc10ac60 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f83cc0098a0 tx=0x7f83cc009bb0 comp rx=0 tx=0).stop 2024-04-23T14:32:38.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.914+0000 7f83db200700 1 -- 172.21.15.28:0/2340078638 shutdown_connections 2024-04-23T14:32:38.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.914+0000 7f83db200700 1 --2- 172.21.15.28:0/2340078638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83dc10a840 0x7f83dc10ac60 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:38.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.914+0000 7f83db200700 1 -- 172.21.15.28:0/2340078638 >> 172.21.15.28:0/2340078638 conn(0x7f83dc07de00 msgr2=0x7f83dc1084c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:38.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.914+0000 7f83db200700 1 -- 172.21.15.28:0/2340078638 shutdown_connections 2024-04-23T14:32:38.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.914+0000 7f83db200700 1 -- 172.21.15.28:0/2340078638 wait complete. 2024-04-23T14:32:38.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.915+0000 7f83db200700 1 Processor -- start 2024-04-23T14:32:38.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.915+0000 7f83db200700 1 -- start start 2024-04-23T14:32:38.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.916+0000 7f83db200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83dc10a840 0x7f83dc19e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:38.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.916+0000 7f83db200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f83dc10eda0 con 0x7f83dc10a840 2024-04-23T14:32:38.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.916+0000 7f83d9e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83dc10a840 0x7f83dc19e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:38.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.916+0000 7f83d9e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83dc10a840 0x7f83dc19e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55746/0 (socket says 172.21.15.28:55746) 2024-04-23T14:32:38.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.916+0000 7f83d9e00700 1 -- 172.21.15.28:0/4066528062 learned_addr learned my addr 172.21.15.28:0/4066528062 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:32:38.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.916+0000 7f83d9e00700 1 -- 172.21.15.28:0/4066528062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83dc19ec50 con 0x7f83dc10a840 2024-04-23T14:32:38.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.917+0000 7f83d9e00700 1 --2- 172.21.15.28:0/4066528062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83dc10a840 0x7f83dc19e710 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f83cc000c00 tx=0x7f83cc011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:38.606 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.917+0000 7f83d2a00700 1 -- 172.21.15.28:0/4066528062 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f83cc003b20 con 0x7f83dc10a840 2024-04-23T14:32:38.606 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.917+0000 7f83d2a00700 1 -- 172.21.15.28:0/4066528062 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f83cc003cc0 con 0x7f83dc10a840 2024-04-23T14:32:38.606 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.917+0000 7f83db200700 1 -- 172.21.15.28:0/4066528062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83dc19eee0 con 0x7f83dc10a840 2024-04-23T14:32:38.606 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.917+0000 7f83d2a00700 1 -- 172.21.15.28:0/4066528062 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f83cc024910 con 0x7f83dc10a840 2024-04-23T14:32:38.606 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.917+0000 7f83db200700 1 -- 172.21.15.28:0/4066528062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83dc1a1bd0 con 0x7f83dc10a840 2024-04-23T14:32:38.606 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.918+0000 7f83d2a00700 1 -- 172.21.15.28:0/4066528062 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f83cc022020 con 0x7f83dc10a840 2024-04-23T14:32:38.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.918+0000 7f83d2a00700 1 -- 172.21.15.28:0/4066528062 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f83cc011e00 con 0x7f83dc10a840 2024-04-23T14:32:38.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.919+0000 7f83db200700 1 -- 172.21.15.28:0/4066528062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83dc064150 con 0x7f83dc10a840 2024-04-23T14:32:38.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:37.921+0000 7f83d2a00700 1 -- 172.21.15.28:0/4066528062 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f83cc01acd0 con 0x7f83dc10a840 2024-04-23T14:32:38.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:38.002+0000 7f83db200700 1 -- 172.21.15.28:0/4066528062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f83dc1a20d0 con 0x7f83dc10a840 2024-04-23T14:32:38.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:38.003+0000 7f83d2a00700 1 -- 172.21.15.28:0/4066528062 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f83cc032030 con 0x7f83dc10a840 2024-04-23T14:32:38.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:38.004+0000 7f83db200700 1 -- 172.21.15.28:0/4066528062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83dc10a840 msgr2=0x7f83dc19e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:38.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:38.004+0000 7f83db200700 1 --2- 172.21.15.28:0/4066528062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83dc10a840 0x7f83dc19e710 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f83cc000c00 tx=0x7f83cc011c10 comp rx=0 tx=0).stop 2024-04-23T14:32:38.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:38.004+0000 7f83db200700 1 -- 172.21.15.28:0/4066528062 shutdown_connections 2024-04-23T14:32:38.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:38.004+0000 7f83db200700 1 --2- 172.21.15.28:0/4066528062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83dc10a840 0x7f83dc19e710 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:38.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:38.004+0000 7f83db200700 1 -- 172.21.15.28:0/4066528062 >> 172.21.15.28:0/4066528062 conn(0x7f83dc07de00 msgr2=0x7f83dc107be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:38.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:38.004+0000 7f83db200700 1 -- 172.21.15.28:0/4066528062 shutdown_connections 2024-04-23T14:32:38.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:38.004+0000 7f83db200700 1 -- 172.21.15.28:0/4066528062 wait complete. 2024-04-23T14:32:38.608 INFO:teuthology.orchestra.run.smithi028.stdout:Restarting the monitor... 2024-04-23T14:32:38.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:38 smithi028 systemd[1]: Stopping Ceph mon.a for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T14:32:39.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:39 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mon-a[27440]: 2024-04-23T14:32:39.088+0000 7f1602800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T14:32:39.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:39 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mon-a[27440]: 2024-04-23T14:32:39.088+0000 7f1602800700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-23T14:32:39.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:39 smithi028 podman[27685]: 2024-04-23 14:32:39.244241204 +0000 UTC m=+0.288534266 container died b1c4012b52eca4997ecbe5232ef0a767d4a16f089d503bcf29ce87110143a863 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, ceph=True) 2024-04-23T14:32:39.774 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:39 smithi028 podman[27685]: 2024-04-23 14:32:39.6063325 +0000 UTC m=+0.650625568 container cleanup b1c4012b52eca4997ecbe5232ef0a767d4a16f089d503bcf29ce87110143a863 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-23T14:32:39.774 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:39 smithi028 bash[27685]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mon-a 2024-04-23T14:32:40.086 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:39 smithi028 podman[27700]: 2024-04-23 14:32:39.773536137 +0000 UTC m=+0.515881552 container remove b1c4012b52eca4997ecbe5232ef0a767d4a16f089d503bcf29ce87110143a863 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-23T14:32:40.362 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:40 smithi028 systemd[1]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-23T14:32:40.362 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:40 smithi028 systemd[1]: Stopped Ceph mon.a for 1ee6e996-017e-11ef-bc93-c7b262605968. 2024-04-23T14:32:40.362 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:40 smithi028 systemd[1]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mon.a.service: Consumed 1.960s CPU time. 2024-04-23T14:32:40.362 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:40 smithi028 systemd[1]: Starting Ceph mon.a for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T14:32:41.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:40 smithi028 podman[27791]: 2024-04-23 14:32:40.582628418 +0000 UTC m=+0.033761988 image pull 76e77f280c74312ca1add9b6019a4a921a516f91795d85bae00a0c8ca9b1c62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 2024-04-23T14:32:41.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:40 smithi028 podman[27791]: 2024-04-23 14:32:40.690929296 +0000 UTC m=+0.142062917 container create 4e1b4802bb2e60eb95ec341069b34f30d2320e0e2905d0a1c27842cec27769e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240416, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T14:32:41.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 podman[27791]: 2024-04-23 14:32:41.10279871 +0000 UTC m=+0.553932354 container init 4e1b4802bb2e60eb95ec341069b34f30d2320e0e2905d0a1c27842cec27769e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd) 2024-04-23T14:32:41.202 INFO:teuthology.orchestra.run.smithi028.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-23T14:32:41.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 podman[27791]: 2024-04-23 14:32:41.115514359 +0000 UTC m=+0.566647957 container start 4e1b4802bb2e60eb95ec341069b34f30d2320e0e2905d0a1c27842cec27769e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-23T14:32:41.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: set uid:gid to 167:167 (ceph:ceph) 2024-04-23T14:32:41.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: ceph version 19.0.0-3126-gd44fddd9 (d44fddd910b5e94da0f2357af4be37d7bd495ae0) squid (dev), process ceph-mon, pid 2 2024-04-23T14:32:41.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: pidfile_write: ignore empty --pid-file 2024-04-23T14:32:41.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: load: jerasure load: lrc 2024-04-23T14:32:41.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: RocksDB version: 7.9.2 2024-04-23T14:32:41.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Git sha 0 2024-04-23T14:32:41.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Compile date 2024-04-22 13:15:13 2024-04-23T14:32:41.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: DB SUMMARY 2024-04-23T14:32:41.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: DB Session ID: EQSRZE4N7PNOJPC2GVSB 2024-04-23T14:32:41.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: CURRENT file: CURRENT 2024-04-23T14:32:41.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: IDENTITY file: IDENTITY 2024-04-23T14:32:41.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-23T14:32:41.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-23T14:32:41.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105440 ; 2024-04-23T14:32:41.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.error_if_exists: 0 2024-04-23T14:32:41.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.create_if_missing: 0 2024-04-23T14:32:41.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.paranoid_checks: 1 2024-04-23T14:32:41.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-23T14:32:41.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-23T14:32:41.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-23T14:32:41.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.env: 0x56149f4afcc0 2024-04-23T14:32:41.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.fs: PosixFileSystem 2024-04-23T14:32:41.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.info_log: 0x5614affa9c60 2024-04-23T14:32:41.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.max_file_opening_threads: 16 2024-04-23T14:32:41.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.statistics: (nil) 2024-04-23T14:32:41.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.use_fsync: 0 2024-04-23T14:32:41.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.max_log_file_size: 0 2024-04-23T14:32:41.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-23T14:32:41.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-23T14:32:41.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.keep_log_file_num: 1000 2024-04-23T14:32:41.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.recycle_log_file_num: 0 2024-04-23T14:32:41.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.allow_fallocate: 1 2024-04-23T14:32:41.378 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.allow_mmap_reads: 0 2024-04-23T14:32:41.378 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.allow_mmap_writes: 0 2024-04-23T14:32:41.378 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.use_direct_reads: 0 2024-04-23T14:32:41.378 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-23T14:32:41.378 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.create_missing_column_families: 0 2024-04-23T14:32:41.378 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.db_log_dir: 2024-04-23T14:32:41.379 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.wal_dir: 2024-04-23T14:32:41.379 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-23T14:32:41.379 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-23T14:32:41.379 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-23T14:32:41.379 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-23T14:32:41.379 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-23T14:32:41.379 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-23T14:32:41.379 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.advise_random_on_open: 1 2024-04-23T14:32:41.379 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.db_write_buffer_size: 0 2024-04-23T14:32:41.379 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.write_buffer_manager: 0x5614aff781e0 2024-04-23T14:32:41.379 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-23T14:32:41.379 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-23T14:32:41.379 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-23T14:32:41.379 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.rate_limiter: (nil) 2024-04-23T14:32:41.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-23T14:32:41.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.wal_recovery_mode: 2 2024-04-23T14:32:41.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.enable_thread_tracking: 0 2024-04-23T14:32:41.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.enable_pipelined_write: 0 2024-04-23T14:32:41.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.unordered_write: 0 2024-04-23T14:32:41.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-23T14:32:41.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-23T14:32:41.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-23T14:32:41.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-23T14:32:41.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.row_cache: None 2024-04-23T14:32:41.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.wal_filter: None 2024-04-23T14:32:41.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-23T14:32:41.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.allow_ingest_behind: 0 2024-04-23T14:32:41.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.two_write_queues: 0 2024-04-23T14:32:41.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.manual_wal_flush: 0 2024-04-23T14:32:41.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.wal_compression: 0 2024-04-23T14:32:41.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.atomic_flush: 0 2024-04-23T14:32:41.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-23T14:32:41.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-23T14:32:41.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-23T14:32:41.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.log_readahead_size: 0 2024-04-23T14:32:41.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-23T14:32:41.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.best_efforts_recovery: 0 2024-04-23T14:32:41.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-23T14:32:41.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-23T14:32:41.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.allow_data_in_errors: 0 2024-04-23T14:32:41.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.db_host_id: __hostname__ 2024-04-23T14:32:41.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-23T14:32:41.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.max_background_jobs: 2 2024-04-23T14:32:41.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.max_background_compactions: -1 2024-04-23T14:32:41.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.max_subcompactions: 1 2024-04-23T14:32:41.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-23T14:32:41.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-23T14:32:41.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-23T14:32:41.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.max_total_wal_size: 0 2024-04-23T14:32:41.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-23T14:32:41.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-23T14:32:41.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-23T14:32:41.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-23T14:32:41.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.max_open_files: -1 2024-04-23T14:32:41.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.bytes_per_sync: 0 2024-04-23T14:32:41.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-23T14:32:41.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-23T14:32:41.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.compaction_readahead_size: 0 2024-04-23T14:32:41.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.max_background_flushes: -1 2024-04-23T14:32:41.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Compression algorithms supported: 2024-04-23T14:32:41.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-23T14:32:41.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: kZSTD supported: 0 2024-04-23T14:32:41.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: kXpressCompression supported: 0 2024-04-23T14:32:41.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: kLZ4HCCompression supported: 1 2024-04-23T14:32:41.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: kZlibCompression supported: 1 2024-04-23T14:32:41.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: kSnappyCompression supported: 1 2024-04-23T14:32:41.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: kLZ4Compression supported: 1 2024-04-23T14:32:41.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: kBZip2Compression supported: 0 2024-04-23T14:32:41.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-23T14:32:41.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-23T14:32:41.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-23T14:32:41.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-23T14:32:41.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-23T14:32:41.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.merge_operator: 2024-04-23T14:32:41.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.compaction_filter: None 2024-04-23T14:32:41.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.compaction_filter_factory: None 2024-04-23T14:32:41.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.sst_partitioner_factory: None 2024-04-23T14:32:41.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-23T14:32:41.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-23T14:32:41.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5614affa9e20) 2024-04-23T14:32:41.384 INFO:journalctl@ceph.mon.a.smithi028.stdout: cache_index_and_filter_blocks: 1 2024-04-23T14:32:41.384 INFO:journalctl@ceph.mon.a.smithi028.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-23T14:32:41.384 INFO:journalctl@ceph.mon.a.smithi028.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-23T14:32:41.384 INFO:journalctl@ceph.mon.a.smithi028.stdout: pin_top_level_index_and_filter: 1 2024-04-23T14:32:41.384 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_type: 0 2024-04-23T14:32:41.384 INFO:journalctl@ceph.mon.a.smithi028.stdout: data_block_index_type: 0 2024-04-23T14:32:41.384 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_shortening: 1 2024-04-23T14:32:41.384 INFO:journalctl@ceph.mon.a.smithi028.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-23T14:32:41.384 INFO:journalctl@ceph.mon.a.smithi028.stdout: checksum: 4 2024-04-23T14:32:41.384 INFO:journalctl@ceph.mon.a.smithi028.stdout: no_block_cache: 0 2024-04-23T14:32:41.384 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache: 0x5614affab090 2024-04-23T14:32:41.384 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_name: BinnedLRUCache 2024-04-23T14:32:41.384 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_options: 2024-04-23T14:32:41.384 INFO:journalctl@ceph.mon.a.smithi028.stdout: capacity : 536870912 2024-04-23T14:32:41.385 INFO:journalctl@ceph.mon.a.smithi028.stdout: num_shard_bits : 4 2024-04-23T14:32:41.385 INFO:journalctl@ceph.mon.a.smithi028.stdout: strict_capacity_limit : 0 2024-04-23T14:32:41.385 INFO:journalctl@ceph.mon.a.smithi028.stdout: high_pri_pool_ratio: 0.000 2024-04-23T14:32:41.385 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_compressed: (nil) 2024-04-23T14:32:41.385 INFO:journalctl@ceph.mon.a.smithi028.stdout: persistent_cache: (nil) 2024-04-23T14:32:41.385 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_size: 4096 2024-04-23T14:32:41.385 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_size_deviation: 10 2024-04-23T14:32:41.385 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_restart_interval: 16 2024-04-23T14:32:41.385 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_block_restart_interval: 1 2024-04-23T14:32:41.385 INFO:journalctl@ceph.mon.a.smithi028.stdout: metadata_block_size: 4096 2024-04-23T14:32:41.385 INFO:journalctl@ceph.mon.a.smithi028.stdout: partition_filters: 0 2024-04-23T14:32:41.385 INFO:journalctl@ceph.mon.a.smithi028.stdout: use_delta_encoding: 1 2024-04-23T14:32:41.385 INFO:journalctl@ceph.mon.a.smithi028.stdout: filter_policy: bloomfilter 2024-04-23T14:32:41.385 INFO:journalctl@ceph.mon.a.smithi028.stdout: whole_key_filtering: 1 2024-04-23T14:32:41.385 INFO:journalctl@ceph.mon.a.smithi028.stdout: verify_compression: 0 2024-04-23T14:32:41.386 INFO:journalctl@ceph.mon.a.smithi028.stdout: read_amp_bytes_per_bit: 0 2024-04-23T14:32:41.386 INFO:journalctl@ceph.mon.a.smithi028.stdout: format_version: 5 2024-04-23T14:32:41.386 INFO:journalctl@ceph.mon.a.smithi028.stdout: enable_index_compression: 1 2024-04-23T14:32:41.386 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_align: 0 2024-04-23T14:32:41.386 INFO:journalctl@ceph.mon.a.smithi028.stdout: max_auto_readahead_size: 262144 2024-04-23T14:32:41.386 INFO:journalctl@ceph.mon.a.smithi028.stdout: prepopulate_block_cache: 0 2024-04-23T14:32:41.386 INFO:journalctl@ceph.mon.a.smithi028.stdout: initial_auto_readahead_size: 8192 2024-04-23T14:32:41.386 INFO:journalctl@ceph.mon.a.smithi028.stdout: num_file_reads_for_auto_readahead: 2 2024-04-23T14:32:41.386 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.write_buffer_size: 33554432 2024-04-23T14:32:41.386 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.max_write_buffer_number: 2 2024-04-23T14:32:41.386 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.compression: NoCompression 2024-04-23T14:32:41.386 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.bottommost_compression: Disabled 2024-04-23T14:32:41.386 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.prefix_extractor: nullptr 2024-04-23T14:32:41.387 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-23T14:32:41.387 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.num_levels: 7 2024-04-23T14:32:41.387 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-23T14:32:41.387 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-23T14:32:41.387 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-23T14:32:41.387 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-23T14:32:41.387 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-23T14:32:41.387 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-23T14:32:41.387 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-23T14:32:41.387 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-23T14:32:41.387 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-23T14:32:41.388 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-23T14:32:41.388 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-23T14:32:41.388 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-23T14:32:41.388 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-23T14:32:41.388 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.compression_opts.level: 32767 2024-04-23T14:32:41.388 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.compression_opts.strategy: 0 2024-04-23T14:32:41.388 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-23T14:32:41.388 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-23T14:32:41.388 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-23T14:32:41.388 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-23T14:32:41.388 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.compression_opts.enabled: false 2024-04-23T14:32:41.388 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-23T14:32:41.388 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-23T14:32:41.388 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-23T14:32:41.388 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-23T14:32:41.388 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.target_file_size_base: 67108864 2024-04-23T14:32:41.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-23T14:32:41.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-23T14:32:41.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-23T14:32:41.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-23T14:32:41.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-23T14:32:41.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-23T14:32:41.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-23T14:32:41.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-23T14:32:41.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-23T14:32:41.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-23T14:32:41.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-23T14:32:41.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-23T14:32:41.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-23T14:32:41.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-23T14:32:41.390 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.arena_block_size: 1048576 2024-04-23T14:32:41.390 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-23T14:32:41.390 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-23T14:32:41.390 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.disable_auto_compactions: 0 2024-04-23T14:32:41.390 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-23T14:32:41.390 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-23T14:32:41.390 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-23T14:32:41.391 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-23T14:32:41.391 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-23T14:32:41.391 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-23T14:32:41.391 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-23T14:32:41.391 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-23T14:32:41.391 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-23T14:32:41.391 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-23T14:32:41.391 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-23T14:32:41.391 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.inplace_update_support: 0 2024-04-23T14:32:41.391 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-23T14:32:41.391 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-23T14:32:41.391 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-23T14:32:41.391 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-23T14:32:41.391 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.bloom_locality: 0 2024-04-23T14:32:41.391 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.max_successive_merges: 0 2024-04-23T14:32:41.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-23T14:32:41.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.paranoid_file_checks: 0 2024-04-23T14:32:41.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.force_consistency_checks: 1 2024-04-23T14:32:41.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.report_bg_io_stats: 0 2024-04-23T14:32:41.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.ttl: 2592000 2024-04-23T14:32:41.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-23T14:32:41.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-23T14:32:41.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-23T14:32:41.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.enable_blob_files: false 2024-04-23T14:32:41.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.min_blob_size: 0 2024-04-23T14:32:41.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.blob_file_size: 268435456 2024-04-23T14:32:41.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-23T14:32:41.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-23T14:32:41.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-23T14:32:41.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-23T14:32:41.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-23T14:32:41.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.blob_file_starting_level: 0 2024-04-23T14:32:41.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-23T14:32:41.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-23T14:32:41.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-23T14:32:41.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d2063da0-e97c-4a34-9fdf-46719e49a2f0 2024-04-23T14:32:41.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713882761174464, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-23T14:32:41.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-23T14:32:41.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713882761176361, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100521, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 328, "table_properties": {"data_size": 98406, "index_size": 307, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14268, "raw_average_key_size": 49, "raw_value_size": 90687, "raw_average_value_size": 315, "num_data_blocks": 12, "num_entries": 287, "num_filter_entries": 287, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713882761, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d2063da0-e97c-4a34-9fdf-46719e49a2f0", "db_session_id": "EQSRZE4N7PNOJPC2GVSB", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-23T14:32:41.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713882761176537, "job": 1, "event": "recovery_finished"} 2024-04-23T14:32:41.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-23T14:32:41.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-23T14:32:41.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5614b0098000 2024-04-23T14:32:41.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: DB pointer 0x5614b0082000 2024-04-23T14:32:41.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-23T14:32:41.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-23T14:32:41.393 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** DB Stats ** 2024-04-23T14:32:41.394 INFO:journalctl@ceph.mon.a.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T14:32:41.394 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-23T14:32:41.394 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T14:32:41.394 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T14:32:41.394 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-23T14:32:41.394 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T14:32:41.394 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T14:32:41.394 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-04-23T14:32:41.394 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** Compaction Stats [default] ** 2024-04-23T14:32:41.394 INFO:journalctl@ceph.mon.a.smithi028.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-23T14:32:41.394 INFO:journalctl@ceph.mon.a.smithi028.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-23T14:32:41.394 INFO:journalctl@ceph.mon.a.smithi028.stdout: L0 2/0 100.03 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 76.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T14:32:41.394 INFO:journalctl@ceph.mon.a.smithi028.stdout: Sum 2/0 100.03 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 76.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T14:32:41.394 INFO:journalctl@ceph.mon.a.smithi028.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 76.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T14:32:41.394 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-04-23T14:32:41.395 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** Compaction Stats [default] ** 2024-04-23T14:32:41.395 INFO:journalctl@ceph.mon.a.smithi028.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-23T14:32:41.395 INFO:journalctl@ceph.mon.a.smithi028.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-23T14:32:41.395 INFO:journalctl@ceph.mon.a.smithi028.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 76.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T14:32:41.395 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-04-23T14:32:41.395 INFO:journalctl@ceph.mon.a.smithi028.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-23T14:32:41.395 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-04-23T14:32:41.395 INFO:journalctl@ceph.mon.a.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T14:32:41.395 INFO:journalctl@ceph.mon.a.smithi028.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-23T14:32:41.395 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-23T14:32:41.395 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-23T14:32:41.395 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-23T14:32:41.395 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-23T14:32:41.395 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative compaction: 0.00 GB write, 12.90 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T14:32:41.396 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval compaction: 0.00 GB write, 12.90 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T14:32:41.396 INFO:journalctl@ceph.mon.a.smithi028.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-23T14:32:41.396 INFO:journalctl@ceph.mon.a.smithi028.stdout: Block cache BinnedLRUCache@0x5614affab090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 2.3e-05 secs_since: 0 2024-04-23T14:32:41.396 INFO:journalctl@ceph.mon.a.smithi028.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-23T14:32:41.396 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-04-23T14:32:41.396 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-23T14:32:41.396 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: starting mon.a rank 0 at public addrs [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] at bind addrs [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T14:32:41.396 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: mon.a@-1(???) e1 preinit fsid 1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T14:32:41.396 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: mon.a@-1(???).mds e1 new map 2024-04-23T14:32:41.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: mon.a@-1(???).mds e1 print_map 2024-04-23T14:32:41.397 INFO:journalctl@ceph.mon.a.smithi028.stdout: e1 2024-04-23T14:32:41.397 INFO:journalctl@ceph.mon.a.smithi028.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-23T14:32:41.397 INFO:journalctl@ceph.mon.a.smithi028.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-23T14:32:41.397 INFO:journalctl@ceph.mon.a.smithi028.stdout: legacy client fscid: -1 2024-04-23T14:32:41.397 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-04-23T14:32:41.397 INFO:journalctl@ceph.mon.a.smithi028.stdout: No filesystems configured 2024-04-23T14:32:41.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-23T14:32:41.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T14:32:41.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T14:32:41.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T14:32:41.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-23T14:32:41.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 bash[27791]: 4e1b4802bb2e60eb95ec341069b34f30d2320e0e2905d0a1c27842cec27769e9 2024-04-23T14:32:41.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 systemd[1]: Started Ceph mon.a for 1ee6e996-017e-11ef-bc93-c7b262605968. 2024-04-23T14:32:41.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-23T14:32:41.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: monmap epoch 1 2024-04-23T14:32:41.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: fsid 1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T14:32:41.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: last_changed 2024-04-23T14:32:30.337765+0000 2024-04-23T14:32:41.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: created 2024-04-23T14:32:30.337765+0000 2024-04-23T14:32:41.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: min_mon_release 19 (squid) 2024-04-23T14:32:41.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: election_strategy: 1 2024-04-23T14:32:41.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-23T14:32:41.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: fsmap 2024-04-23T14:32:41.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: osdmap e1: 0 total, 0 up, 0 in 2024-04-23T14:32:41.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:41 smithi028 ceph-mon[27810]: mgrmap e1: no daemons active 2024-04-23T14:32:42.521 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.822+0000 7f1680c00700 1 Processor -- start 2024-04-23T14:32:42.521 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.822+0000 7f1680c00700 1 -- start start 2024-04-23T14:32:42.521 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.822+0000 7f1680c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f167c1090c0 0x7f167c10b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:42.522 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.822+0000 7f1680c00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f167c07be00 con 0x7f167c1090c0 2024-04-23T14:32:42.522 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.823+0000 7f167b400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f167c1090c0 0x7f167c10b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:42.522 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.823+0000 7f167b400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f167c1090c0 0x7f167c10b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55756/0 (socket says 172.21.15.28:55756) 2024-04-23T14:32:42.522 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.823+0000 7f167b400700 1 -- 172.21.15.28:0/3661683067 learned_addr learned my addr 172.21.15.28:0/3661683067 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:32:42.522 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.824+0000 7f167b400700 1 -- 172.21.15.28:0/3661683067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f167c10b9f0 con 0x7f167c1090c0 2024-04-23T14:32:42.522 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.824+0000 7f167b400700 1 --2- 172.21.15.28:0/3661683067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f167c1090c0 0x7f167c10b4b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f166c0098a0 tx=0x7f166c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ca07c18e5e8a06 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:42.522 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.824+0000 7f167a000700 1 -- 172.21.15.28:0/3661683067 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f166c004160 con 0x7f167c1090c0 2024-04-23T14:32:42.522 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.824+0000 7f167a000700 1 -- 172.21.15.28:0/3661683067 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f166c004300 con 0x7f167c1090c0 2024-04-23T14:32:42.522 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.824+0000 7f167a000700 1 -- 172.21.15.28:0/3661683067 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f166c011b50 con 0x7f167c1090c0 2024-04-23T14:32:42.522 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.825+0000 7f1680c00700 1 -- 172.21.15.28:0/3661683067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f167c1090c0 msgr2=0x7f167c10b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:42.522 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.825+0000 7f1680c00700 1 --2- 172.21.15.28:0/3661683067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f167c1090c0 0x7f167c10b4b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f166c0098a0 tx=0x7f166c009bb0 comp rx=0 tx=0).stop 2024-04-23T14:32:42.522 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.825+0000 7f1680c00700 1 -- 172.21.15.28:0/3661683067 shutdown_connections 2024-04-23T14:32:42.522 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.825+0000 7f1680c00700 1 --2- 172.21.15.28:0/3661683067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f167c1090c0 0x7f167c10b4b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:42.523 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.825+0000 7f1680c00700 1 -- 172.21.15.28:0/3661683067 >> 172.21.15.28:0/3661683067 conn(0x7f167c102a70 msgr2=0x7f167c104e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:42.523 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.825+0000 7f1680c00700 1 -- 172.21.15.28:0/3661683067 shutdown_connections 2024-04-23T14:32:42.523 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.825+0000 7f1680c00700 1 -- 172.21.15.28:0/3661683067 wait complete. 2024-04-23T14:32:42.523 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.826+0000 7f1680c00700 1 Processor -- start 2024-04-23T14:32:42.523 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.827+0000 7f1680c00700 1 -- start start 2024-04-23T14:32:42.523 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.827+0000 7f1680c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f167c1090c0 0x7f167c19e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:42.523 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.827+0000 7f1680c00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f167c10c8d0 con 0x7f167c1090c0 2024-04-23T14:32:42.523 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.827+0000 7f167b400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f167c1090c0 0x7f167c19e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:42.523 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.827+0000 7f167b400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f167c1090c0 0x7f167c19e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55760/0 (socket says 172.21.15.28:55760) 2024-04-23T14:32:42.523 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.827+0000 7f167b400700 1 -- 172.21.15.28:0/1179760041 learned_addr learned my addr 172.21.15.28:0/1179760041 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:32:42.523 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.828+0000 7f167b400700 1 -- 172.21.15.28:0/1179760041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f167c19ec10 con 0x7f167c1090c0 2024-04-23T14:32:42.523 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.828+0000 7f167b400700 1 --2- 172.21.15.28:0/1179760041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f167c1090c0 0x7f167c19e6d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f166c009fb0 tx=0x7f166c011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:42.523 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.828+0000 7f1673e00700 1 -- 172.21.15.28:0/1179760041 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f166c0038b0 con 0x7f167c1090c0 2024-04-23T14:32:42.523 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.829+0000 7f1673e00700 1 -- 172.21.15.28:0/1179760041 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f166c011e50 con 0x7f167c1090c0 2024-04-23T14:32:42.523 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.829+0000 7f1673e00700 1 -- 172.21.15.28:0/1179760041 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f166c019420 con 0x7f167c1090c0 2024-04-23T14:32:42.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.829+0000 7f1680c00700 1 -- 172.21.15.28:0/1179760041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f167c19eea0 con 0x7f167c1090c0 2024-04-23T14:32:42.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.829+0000 7f1680c00700 1 -- 172.21.15.28:0/1179760041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f167c1a1b90 con 0x7f167c1090c0 2024-04-23T14:32:42.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.830+0000 7f1673e00700 1 -- 172.21.15.28:0/1179760041 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f166c019720 con 0x7f167c1090c0 2024-04-23T14:32:42.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.830+0000 7f1673e00700 1 -- 172.21.15.28:0/1179760041 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f166c02dc70 con 0x7f167c1090c0 2024-04-23T14:32:42.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.830+0000 7f1680c00700 1 -- 172.21.15.28:0/1179760041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f167c064150 con 0x7f167c1090c0 2024-04-23T14:32:42.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.834+0000 7f1673e00700 1 -- 172.21.15.28:0/1179760041 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f166c043e50 con 0x7f167c1090c0 2024-04-23T14:32:42.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.920+0000 7f1680c00700 1 -- 172.21.15.28:0/1179760041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f167c1a1d60 con 0x7f167c1090c0 2024-04-23T14:32:42.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.924+0000 7f1673e00700 1 -- 172.21.15.28:0/1179760041 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f166c032030 con 0x7f167c1090c0 2024-04-23T14:32:42.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.925+0000 7f1680c00700 1 -- 172.21.15.28:0/1179760041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f167c1090c0 msgr2=0x7f167c19e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:42.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.925+0000 7f1680c00700 1 --2- 172.21.15.28:0/1179760041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f167c1090c0 0x7f167c19e6d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f166c009fb0 tx=0x7f166c011bf0 comp rx=0 tx=0).stop 2024-04-23T14:32:42.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.925+0000 7f1680c00700 1 -- 172.21.15.28:0/1179760041 shutdown_connections 2024-04-23T14:32:42.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.925+0000 7f1680c00700 1 --2- 172.21.15.28:0/1179760041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f167c1090c0 0x7f167c19e6d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:42.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.925+0000 7f1680c00700 1 -- 172.21.15.28:0/1179760041 >> 172.21.15.28:0/1179760041 conn(0x7f167c102a70 msgr2=0x7f167c103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:42.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.926+0000 7f1680c00700 1 -- 172.21.15.28:0/1179760041 shutdown_connections 2024-04-23T14:32:42.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:41.926+0000 7f1680c00700 1 -- 172.21.15.28:0/1179760041 wait complete. 2024-04-23T14:32:42.525 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-23T14:32:42.549 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-23T14:32:42.549 INFO:teuthology.orchestra.run.smithi028.stdout:Creating mgr... 2024-04-23T14:32:42.550 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-23T14:32:42.552 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-23T14:32:43.007 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mgr.x 2024-04-23T14:32:43.007 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to reset failed state of unit ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-23T14:32:43.272 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-1ee6e996-017e-11ef-bc93-c7b262605968.target.wants/ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-1ee6e996-017e-11ef-bc93-c7b262605968@.service. 2024-04-23T14:32:43.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:42 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1179760041' entity='client.admin' 2024-04-23T14:32:43.624 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:43 smithi028 systemd[1]: Starting Ceph mgr.x for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T14:32:43.958 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:43 smithi028 podman[28033]: 2024-04-23 14:32:43.700938505 +0000 UTC m=+0.131536804 container create 7f8a92032a8ce13a50ae5482bf17653809d0862599c39f2bed9d08ef99aa24b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T14:32:43.958 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:43 smithi028 podman[28033]: 2024-04-23 14:32:43.60754748 +0000 UTC m=+0.038145766 image pull 76e77f280c74312ca1add9b6019a4a921a516f91795d85bae00a0c8ca9b1c62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 2024-04-23T14:32:43.958 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:43 smithi028 podman[28033]: 2024-04-23 14:32:43.946486578 +0000 UTC m=+0.377084918 container init 7f8a92032a8ce13a50ae5482bf17653809d0862599c39f2bed9d08ef99aa24b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416) 2024-04-23T14:32:43.958 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:43 smithi028 podman[28033]: 2024-04-23 14:32:43.957413939 +0000 UTC m=+0.388012259 container start 7f8a92032a8ce13a50ae5482bf17653809d0862599c39f2bed9d08ef99aa24b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5) 2024-04-23T14:32:44.070 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-23T14:32:44.071 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-04-23T14:32:44.087 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-23T14:32:44.087 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-04-23T14:32:44.087 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-04-23T14:32:44.087 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to enable service . firewalld.service is not available 2024-04-23T14:32:44.107 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-23T14:32:44.107 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-04-23T14:32:44.125 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-23T14:32:44.125 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-04-23T14:32:44.125 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-04-23T14:32:44.125 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-23T14:32:44.125 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr to start... 2024-04-23T14:32:44.125 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr... 2024-04-23T14:32:44.212 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:44 smithi028 bash[28033]: 7f8a92032a8ce13a50ae5482bf17653809d0862599c39f2bed9d08ef99aa24b6 2024-04-23T14:32:44.212 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:44 smithi028 systemd[1]: Started Ceph mgr.x for 1ee6e996-017e-11ef-bc93-c7b262605968. 2024-04-23T14:32:44.466 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:44 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:32:44.209+0000 7f953deca200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T14:32:44.731 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:44 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:32:44.479+0000 7f953deca200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T14:32:44.731 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:44 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:32:44.626+0000 7f953deca200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T14:32:45.052 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:44 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3088624410' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-23T14:32:45.374 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:45 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:32:45.051+0000 7f953deca200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T14:32:45.374 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:45 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:32:45.168+0000 7f953deca200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T14:32:45.479 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-23T14:32:45.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-23T14:32:45.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "1ee6e996-017e-11ef-bc93-c7b262605968", 2024-04-23T14:32:45.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-04-23T14:32:45.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-23T14:32:45.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-23T14:32:45.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-23T14:32:45.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-23T14:32:45.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-23T14:32:45.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-23T14:32:45.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-04-23T14:32:45.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-23T14:32:45.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-23T14:32:45.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-04-23T14:32:45.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-23T14:32:45.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-04-23T14:32:45.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-23T14:32:45.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T14:32:45.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-23T14:32:45.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-23T14:32:45.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-23T14:32:45.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-23T14:32:45.490 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T14:32:45.490 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-23T14:32:45.490 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-23T14:32:45.490 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-23T14:32:45.490 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-23T14:32:45.490 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-23T14:32:45.490 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-23T14:32:45.490 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-23T14:32:45.490 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-23T14:32:45.490 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-23T14:32:45.490 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-23T14:32:45.490 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-23T14:32:45.490 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-23T14:32:45.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-23T14:32:45.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-23T14:32:45.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-23T14:32:45.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-23T14:32:45.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-23T14:32:45.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-23T14:32:45.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T14:32:45.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-23T14:32:45.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-23T14:32:45.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-23T14:32:45.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-23T14:32:45.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-23T14:32:45.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-23T14:32:45.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-23T14:32:45.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-23T14:32:45.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-23T14:32:45.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-04-23T14:32:45.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-23T14:32:45.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T14:32:45.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-23T14:32:45.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-23T14:32:45.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T14:32:45.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-04-23T14:32:34.581743+0000", 2024-04-23T14:32:45.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T14:32:45.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-23T14:32:45.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-23T14:32:45.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-23T14:32:45.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.726+0000 7f57dee00700 1 Processor -- start 2024-04-23T14:32:45.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.726+0000 7f57dee00700 1 -- start start 2024-04-23T14:32:45.493 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.726+0000 7f57dee00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57d8108780 0x7f57d8108ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:45.493 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.726+0000 7f57dee00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f57d8109170 con 0x7f57d8108780 2024-04-23T14:32:45.493 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.727+0000 7f57d7e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57d8108780 0x7f57d8108ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:45.493 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.727+0000 7f57d7e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57d8108780 0x7f57d8108ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55772/0 (socket says 172.21.15.28:55772) 2024-04-23T14:32:45.493 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.727+0000 7f57d7e00700 1 -- 172.21.15.28:0/3296851426 learned_addr learned my addr 172.21.15.28:0/3296851426 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:32:45.493 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.727+0000 7f57d7e00700 1 -- 172.21.15.28:0/3296851426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57d8109930 con 0x7f57d8108780 2024-04-23T14:32:45.493 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.728+0000 7f57d7e00700 1 --2- 172.21.15.28:0/3296851426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57d8108780 0x7f57d8108ba0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f57c80098a0 tx=0x7f57c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7859b622b25bf89 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:45.493 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.728+0000 7f57d7400700 1 -- 172.21.15.28:0/3296851426 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f57c8004160 con 0x7f57d8108780 2024-04-23T14:32:45.493 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.728+0000 7f57d7400700 1 -- 172.21.15.28:0/3296851426 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f57c8004300 con 0x7f57d8108780 2024-04-23T14:32:45.493 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.728+0000 7f57d7400700 1 -- 172.21.15.28:0/3296851426 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f57c8011b50 con 0x7f57d8108780 2024-04-23T14:32:45.493 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.729+0000 7f57dee00700 1 -- 172.21.15.28:0/3296851426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57d8108780 msgr2=0x7f57d8108ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:45.493 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.729+0000 7f57dee00700 1 --2- 172.21.15.28:0/3296851426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57d8108780 0x7f57d8108ba0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f57c80098a0 tx=0x7f57c8009bb0 comp rx=0 tx=0).stop 2024-04-23T14:32:45.493 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.729+0000 7f57dee00700 1 -- 172.21.15.28:0/3296851426 shutdown_connections 2024-04-23T14:32:45.493 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.729+0000 7f57dee00700 1 --2- 172.21.15.28:0/3296851426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57d8108780 0x7f57d8108ba0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:45.494 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.729+0000 7f57dee00700 1 -- 172.21.15.28:0/3296851426 >> 172.21.15.28:0/3296851426 conn(0x7f57d8103da0 msgr2=0x7f57d81061c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:45.494 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.729+0000 7f57dee00700 1 -- 172.21.15.28:0/3296851426 shutdown_connections 2024-04-23T14:32:45.494 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.729+0000 7f57dee00700 1 -- 172.21.15.28:0/3296851426 wait complete. 2024-04-23T14:32:45.494 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.730+0000 7f57dee00700 1 Processor -- start 2024-04-23T14:32:45.494 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.730+0000 7f57dee00700 1 -- start start 2024-04-23T14:32:45.494 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.731+0000 7f57dee00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57d8108780 0x7f57d819c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:45.494 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.731+0000 7f57dee00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f57d810cce0 con 0x7f57d8108780 2024-04-23T14:32:45.494 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.731+0000 7f57d7e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57d8108780 0x7f57d819c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:45.494 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.731+0000 7f57d7e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57d8108780 0x7f57d819c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55776/0 (socket says 172.21.15.28:55776) 2024-04-23T14:32:45.494 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.731+0000 7f57d7e00700 1 -- 172.21.15.28:0/3088624410 learned_addr learned my addr 172.21.15.28:0/3088624410 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:32:45.494 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.731+0000 7f57d7e00700 1 -- 172.21.15.28:0/3088624410 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57d819cbf0 con 0x7f57d8108780 2024-04-23T14:32:45.496 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.732+0000 7f57d7e00700 1 --2- 172.21.15.28:0/3088624410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57d8108780 0x7f57d819c6b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f57c8009fb0 tx=0x7f57c8011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:45.496 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.732+0000 7f57d5600700 1 -- 172.21.15.28:0/3088624410 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f57c80038b0 con 0x7f57d8108780 2024-04-23T14:32:45.496 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.732+0000 7f57d5600700 1 -- 172.21.15.28:0/3088624410 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f57c8011e50 con 0x7f57d8108780 2024-04-23T14:32:45.496 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.732+0000 7f57d5600700 1 -- 172.21.15.28:0/3088624410 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f57c8019550 con 0x7f57d8108780 2024-04-23T14:32:45.496 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.732+0000 7f57dee00700 1 -- 172.21.15.28:0/3088624410 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57d819ce80 con 0x7f57d8108780 2024-04-23T14:32:45.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.733+0000 7f57dee00700 1 -- 172.21.15.28:0/3088624410 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57d819fb70 con 0x7f57d8108780 2024-04-23T14:32:45.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.733+0000 7f57d5600700 1 -- 172.21.15.28:0/3088624410 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f57c80196f0 con 0x7f57d8108780 2024-04-23T14:32:45.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.733+0000 7f57d5600700 1 -- 172.21.15.28:0/3088624410 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f57c802dcc0 con 0x7f57d8108780 2024-04-23T14:32:45.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.734+0000 7f57dee00700 1 -- 172.21.15.28:0/3088624410 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57d8064150 con 0x7f57d8108780 2024-04-23T14:32:45.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.737+0000 7f57d5600700 1 -- 172.21.15.28:0/3088624410 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f57c8043e50 con 0x7f57d8108780 2024-04-23T14:32:45.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.819+0000 7f57dee00700 1 -- 172.21.15.28:0/3088624410 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f57d81a0070 con 0x7f57d8108780 2024-04-23T14:32:45.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.819+0000 7f57d5600700 1 -- 172.21.15.28:0/3088624410 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f57c8032030 con 0x7f57d8108780 2024-04-23T14:32:45.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.821+0000 7f57dee00700 1 -- 172.21.15.28:0/3088624410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57d8108780 msgr2=0x7f57d819c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:45.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.821+0000 7f57dee00700 1 --2- 172.21.15.28:0/3088624410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57d8108780 0x7f57d819c6b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f57c8009fb0 tx=0x7f57c8011bf0 comp rx=0 tx=0).stop 2024-04-23T14:32:45.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.821+0000 7f57dee00700 1 -- 172.21.15.28:0/3088624410 shutdown_connections 2024-04-23T14:32:45.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.821+0000 7f57dee00700 1 --2- 172.21.15.28:0/3088624410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57d8108780 0x7f57d819c6b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:45.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.821+0000 7f57dee00700 1 -- 172.21.15.28:0/3088624410 >> 172.21.15.28:0/3088624410 conn(0x7f57d8103da0 msgr2=0x7f57d8104970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:45.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.822+0000 7f57dee00700 1 -- 172.21.15.28:0/3088624410 shutdown_connections 2024-04-23T14:32:45.498 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:44.822+0000 7f57dee00700 1 -- 172.21.15.28:0/3088624410 wait complete. 2024-04-23T14:32:45.498 INFO:teuthology.orchestra.run.smithi028.stdout:mgr not available, waiting (1/15)... 2024-04-23T14:32:47.124 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:46 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:32:46.813+0000 7f953deca200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T14:32:47.124 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:47 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:32:47.047+0000 7f953deca200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T14:32:47.623 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:47 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:32:47.263+0000 7f953deca200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T14:32:47.963 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:47 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:32:47.658+0000 7f953deca200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T14:32:47.964 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:47 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:32:47.785+0000 7f953deca200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T14:32:48.213 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:48 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:32:48.048+0000 7f953deca200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T14:32:48.616 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:48 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3430720851' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-23T14:32:48.616 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:48 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:32:48.312+0000 7f953deca200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T14:32:48.905 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-23T14:32:48.905 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-23T14:32:48.906 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "1ee6e996-017e-11ef-bc93-c7b262605968", 2024-04-23T14:32:48.906 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-04-23T14:32:48.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-23T14:32:48.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-23T14:32:48.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-23T14:32:48.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-23T14:32:48.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-23T14:32:48.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-23T14:32:48.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-04-23T14:32:48.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-23T14:32:48.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-23T14:32:48.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-04-23T14:32:48.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-23T14:32:48.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-04-23T14:32:48.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-23T14:32:48.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T14:32:48.910 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-23T14:32:48.910 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-23T14:32:48.910 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-23T14:32:48.910 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-23T14:32:48.910 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T14:32:48.910 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-23T14:32:48.910 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-23T14:32:48.910 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-23T14:32:48.914 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-23T14:32:48.914 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-23T14:32:48.914 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-23T14:32:48.914 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-23T14:32:48.914 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-23T14:32:48.914 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-23T14:32:48.914 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-23T14:32:48.914 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-23T14:32:48.914 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-23T14:32:48.914 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-23T14:32:48.914 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-23T14:32:48.914 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-23T14:32:48.914 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-23T14:32:48.915 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-23T14:32:48.915 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-23T14:32:48.915 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T14:32:48.915 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-23T14:32:48.915 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-23T14:32:48.915 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-23T14:32:48.915 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-23T14:32:48.915 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-23T14:32:48.915 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-23T14:32:48.915 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-23T14:32:48.915 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-23T14:32:48.915 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-23T14:32:48.915 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-04-23T14:32:48.915 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-23T14:32:48.915 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T14:32:48.916 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-23T14:32:48.916 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-23T14:32:48.916 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T14:32:48.916 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-04-23T14:32:34.581743+0000", 2024-04-23T14:32:48.916 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T14:32:48.916 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-23T14:32:48.916 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-23T14:32:48.916 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-23T14:32:48.920 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.117+0000 7f8f19400700 1 Processor -- start 2024-04-23T14:32:48.921 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.117+0000 7f8f19400700 1 -- start start 2024-04-23T14:32:48.921 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.118+0000 7f8f19400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f14109020 0x7f8f1410b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:48.921 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.118+0000 7f8f19400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8f1407bea0 con 0x7f8f14109020 2024-04-23T14:32:48.921 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.118+0000 7f8f13e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f14109020 0x7f8f1410b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:48.921 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.118+0000 7f8f13e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f14109020 0x7f8f1410b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35534/0 (socket says 172.21.15.28:35534) 2024-04-23T14:32:48.921 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.118+0000 7f8f13e00700 1 -- 172.21.15.28:0/1711739681 learned_addr learned my addr 172.21.15.28:0/1711739681 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:32:48.921 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.119+0000 7f8f13e00700 1 -- 172.21.15.28:0/1711739681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f1410b950 con 0x7f8f14109020 2024-04-23T14:32:48.921 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.119+0000 7f8f13e00700 1 --2- 172.21.15.28:0/1711739681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f14109020 0x7f8f1410b410 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8f040098a0 tx=0x7f8f04009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c1a075791499cec server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:48.921 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.119+0000 7f8f12a00700 1 -- 172.21.15.28:0/1711739681 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f04004160 con 0x7f8f14109020 2024-04-23T14:32:48.921 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.119+0000 7f8f12a00700 1 -- 172.21.15.28:0/1711739681 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8f04004300 con 0x7f8f14109020 2024-04-23T14:32:48.921 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.119+0000 7f8f12a00700 1 -- 172.21.15.28:0/1711739681 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f04011b50 con 0x7f8f14109020 2024-04-23T14:32:48.921 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.120+0000 7f8f19400700 1 -- 172.21.15.28:0/1711739681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f14109020 msgr2=0x7f8f1410b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:48.922 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.120+0000 7f8f19400700 1 --2- 172.21.15.28:0/1711739681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f14109020 0x7f8f1410b410 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8f040098a0 tx=0x7f8f04009bb0 comp rx=0 tx=0).stop 2024-04-23T14:32:48.922 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.120+0000 7f8f19400700 1 -- 172.21.15.28:0/1711739681 shutdown_connections 2024-04-23T14:32:48.922 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.120+0000 7f8f19400700 1 --2- 172.21.15.28:0/1711739681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f14109020 0x7f8f1410b410 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:48.922 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.120+0000 7f8f19400700 1 -- 172.21.15.28:0/1711739681 >> 172.21.15.28:0/1711739681 conn(0x7f8f14102a70 msgr2=0x7f8f14104e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:48.922 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.120+0000 7f8f19400700 1 -- 172.21.15.28:0/1711739681 shutdown_connections 2024-04-23T14:32:48.922 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.120+0000 7f8f19400700 1 -- 172.21.15.28:0/1711739681 wait complete. 2024-04-23T14:32:48.922 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.121+0000 7f8f19400700 1 Processor -- start 2024-04-23T14:32:48.922 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.122+0000 7f8f19400700 1 -- start start 2024-04-23T14:32:48.922 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.122+0000 7f8f19400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f14109020 0x7f8f141088a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:48.922 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.122+0000 7f8f19400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8f1410c8a0 con 0x7f8f14109020 2024-04-23T14:32:48.922 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.122+0000 7f8f13e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f14109020 0x7f8f141088a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:48.922 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.122+0000 7f8f13e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f14109020 0x7f8f141088a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35538/0 (socket says 172.21.15.28:35538) 2024-04-23T14:32:48.922 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.122+0000 7f8f13e00700 1 -- 172.21.15.28:0/3430720851 learned_addr learned my addr 172.21.15.28:0/3430720851 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:32:48.922 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.123+0000 7f8f13e00700 1 -- 172.21.15.28:0/3430720851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f14108de0 con 0x7f8f14109020 2024-04-23T14:32:48.923 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.123+0000 7f8f13e00700 1 --2- 172.21.15.28:0/3430720851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f14109020 0x7f8f141088a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8f04009fb0 tx=0x7f8f04011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:48.923 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.123+0000 7f8f10c00700 1 -- 172.21.15.28:0/3430720851 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f040038b0 con 0x7f8f14109020 2024-04-23T14:32:48.923 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.123+0000 7f8f10c00700 1 -- 172.21.15.28:0/3430720851 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8f04011e50 con 0x7f8f14109020 2024-04-23T14:32:48.923 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.123+0000 7f8f10c00700 1 -- 172.21.15.28:0/3430720851 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f04019420 con 0x7f8f14109020 2024-04-23T14:32:48.923 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.123+0000 7f8f19400700 1 -- 172.21.15.28:0/3430720851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f14106fd0 con 0x7f8f14109020 2024-04-23T14:32:48.923 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.123+0000 7f8f19400700 1 -- 172.21.15.28:0/3430720851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f14107510 con 0x7f8f14109020 2024-04-23T14:32:48.923 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.124+0000 7f8f10c00700 1 -- 172.21.15.28:0/3430720851 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8f04019a60 con 0x7f8f14109020 2024-04-23T14:32:48.923 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.124+0000 7f8f10c00700 1 -- 172.21.15.28:0/3430720851 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8f04024940 con 0x7f8f14109020 2024-04-23T14:32:48.923 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.124+0000 7f8f19400700 1 -- 172.21.15.28:0/3430720851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f14064150 con 0x7f8f14109020 2024-04-23T14:32:48.923 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.127+0000 7f8f10c00700 1 -- 172.21.15.28:0/3430720851 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f8f04043e50 con 0x7f8f14109020 2024-04-23T14:32:48.923 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.207+0000 7f8f19400700 1 -- 172.21.15.28:0/3430720851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f8f14107a40 con 0x7f8f14109020 2024-04-23T14:32:48.923 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.207+0000 7f8f10c00700 1 -- 172.21.15.28:0/3430720851 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f8f04032020 con 0x7f8f14109020 2024-04-23T14:32:48.923 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.209+0000 7f8f19400700 1 -- 172.21.15.28:0/3430720851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f14109020 msgr2=0x7f8f141088a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:48.923 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.209+0000 7f8f19400700 1 --2- 172.21.15.28:0/3430720851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f14109020 0x7f8f141088a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8f04009fb0 tx=0x7f8f04011bf0 comp rx=0 tx=0).stop 2024-04-23T14:32:48.923 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.209+0000 7f8f19400700 1 -- 172.21.15.28:0/3430720851 shutdown_connections 2024-04-23T14:32:48.923 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.209+0000 7f8f19400700 1 --2- 172.21.15.28:0/3430720851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f14109020 0x7f8f141088a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:48.924 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.209+0000 7f8f19400700 1 -- 172.21.15.28:0/3430720851 >> 172.21.15.28:0/3430720851 conn(0x7f8f14102a70 msgr2=0x7f8f14103600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:48.924 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.209+0000 7f8f19400700 1 -- 172.21.15.28:0/3430720851 shutdown_connections 2024-04-23T14:32:48.924 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:48.209+0000 7f8f19400700 1 -- 172.21.15.28:0/3430720851 wait complete. 2024-04-23T14:32:48.924 INFO:teuthology.orchestra.run.smithi028.stdout:mgr not available, waiting (2/15)... 2024-04-23T14:32:49.374 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:49 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:32:49.133+0000 7f953deca200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T14:32:50.098 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:49 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:32:49.695+0000 7f953deca200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T14:32:50.099 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:49 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:32:49.825+0000 7f953deca200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T14:32:50.099 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:50 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:32:50.098+0000 7f953deca200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T14:32:51.078 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:50 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:32:50.760+0000 7f953deca200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T14:32:51.078 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:50 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:32:50.888+0000 7f953deca200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T14:32:51.374 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:51 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:32:51.223+0000 7f953deca200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T14:32:51.661 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:51 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:32:51.411+0000 7f953deca200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T14:32:51.661 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:51 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:32:51.542+0000 7f953deca200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T14:32:51.974 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:51 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:32:51.707+0000 7f953deca200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T14:32:51.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:51 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1836953106' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-23T14:32:52.262 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:52 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:32:52.081+0000 7f953deca200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T14:32:52.286 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-23T14:32:52.286 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-23T14:32:52.286 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "1ee6e996-017e-11ef-bc93-c7b262605968", 2024-04-23T14:32:52.286 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-04-23T14:32:52.286 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-23T14:32:52.286 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-23T14:32:52.286 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-23T14:32:52.286 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-23T14:32:52.286 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-23T14:32:52.287 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-23T14:32:52.287 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-04-23T14:32:52.287 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-23T14:32:52.287 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-23T14:32:52.287 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-04-23T14:32:52.287 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-23T14:32:52.287 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-04-23T14:32:52.287 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-23T14:32:52.287 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T14:32:52.287 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-23T14:32:52.287 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-23T14:32:52.287 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-23T14:32:52.287 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-23T14:32:52.287 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T14:32:52.287 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-23T14:32:52.288 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-23T14:32:52.288 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-23T14:32:52.288 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-23T14:32:52.288 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-23T14:32:52.288 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-23T14:32:52.288 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-23T14:32:52.288 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-23T14:32:52.288 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-23T14:32:52.288 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-23T14:32:52.288 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-23T14:32:52.288 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-23T14:32:52.288 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-23T14:32:52.288 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-23T14:32:52.288 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-23T14:32:52.289 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-23T14:32:52.289 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-23T14:32:52.289 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-23T14:32:52.289 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T14:32:52.289 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-23T14:32:52.289 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-23T14:32:52.289 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-23T14:32:52.289 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-23T14:32:52.289 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-23T14:32:52.289 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-23T14:32:52.289 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-23T14:32:52.289 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-23T14:32:52.289 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-23T14:32:52.289 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-04-23T14:32:52.289 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-23T14:32:52.290 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T14:32:52.290 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-23T14:32:52.290 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-23T14:32:52.290 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T14:32:52.290 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-04-23T14:32:34.581743+0000", 2024-04-23T14:32:52.290 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T14:32:52.290 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-23T14:32:52.290 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-23T14:32:52.290 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-23T14:32:52.290 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.568+0000 7f7981800700 1 Processor -- start 2024-04-23T14:32:52.290 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.568+0000 7f7981800700 1 -- start start 2024-04-23T14:32:52.290 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.569+0000 7f7981800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f797c108790 0x7f797c108bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:52.290 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.569+0000 7f7981800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f797c109180 con 0x7f797c108790 2024-04-23T14:32:52.290 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.569+0000 7f797aa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f797c108790 0x7f797c108bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:52.291 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.569+0000 7f797aa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f797c108790 0x7f797c108bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35542/0 (socket says 172.21.15.28:35542) 2024-04-23T14:32:52.291 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.569+0000 7f797aa00700 1 -- 172.21.15.28:0/1847323853 learned_addr learned my addr 172.21.15.28:0/1847323853 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:32:52.291 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.570+0000 7f797aa00700 1 -- 172.21.15.28:0/1847323853 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f797c109940 con 0x7f797c108790 2024-04-23T14:32:52.291 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.570+0000 7f797aa00700 1 --2- 172.21.15.28:0/1847323853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f797c108790 0x7f797c108bb0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f796c0098a0 tx=0x7f796c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86d167bfa0a151b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:52.291 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.570+0000 7f7979600700 1 -- 172.21.15.28:0/1847323853 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f796c004160 con 0x7f797c108790 2024-04-23T14:32:52.291 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.570+0000 7f7979600700 1 -- 172.21.15.28:0/1847323853 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f796c004300 con 0x7f797c108790 2024-04-23T14:32:52.291 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.570+0000 7f7979600700 1 -- 172.21.15.28:0/1847323853 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f796c011b50 con 0x7f797c108790 2024-04-23T14:32:52.291 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.571+0000 7f7981800700 1 -- 172.21.15.28:0/1847323853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f797c108790 msgr2=0x7f797c108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:52.291 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.571+0000 7f7981800700 1 --2- 172.21.15.28:0/1847323853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f797c108790 0x7f797c108bb0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f796c0098a0 tx=0x7f796c009bb0 comp rx=0 tx=0).stop 2024-04-23T14:32:52.291 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.571+0000 7f7981800700 1 -- 172.21.15.28:0/1847323853 shutdown_connections 2024-04-23T14:32:52.291 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.571+0000 7f7981800700 1 --2- 172.21.15.28:0/1847323853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f797c108790 0x7f797c108bb0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:52.291 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.571+0000 7f7981800700 1 -- 172.21.15.28:0/1847323853 >> 172.21.15.28:0/1847323853 conn(0x7f797c103d70 msgr2=0x7f797c106190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:52.291 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.571+0000 7f7981800700 1 -- 172.21.15.28:0/1847323853 shutdown_connections 2024-04-23T14:32:52.292 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.572+0000 7f7981800700 1 -- 172.21.15.28:0/1847323853 wait complete. 2024-04-23T14:32:52.292 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.573+0000 7f7981800700 1 Processor -- start 2024-04-23T14:32:52.292 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.573+0000 7f7981800700 1 -- start start 2024-04-23T14:32:52.292 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.573+0000 7f7981800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f797c108790 0x7f797c19e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:52.292 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.573+0000 7f7981800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f797c10ccf0 con 0x7f797c108790 2024-04-23T14:32:52.292 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.573+0000 7f797aa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f797c108790 0x7f797c19e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:52.292 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.574+0000 7f797aa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f797c108790 0x7f797c19e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35546/0 (socket says 172.21.15.28:35546) 2024-04-23T14:32:52.292 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.574+0000 7f797aa00700 1 -- 172.21.15.28:0/1836953106 learned_addr learned my addr 172.21.15.28:0/1836953106 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:32:52.292 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.574+0000 7f797aa00700 1 -- 172.21.15.28:0/1836953106 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f797c19ee30 con 0x7f797c108790 2024-04-23T14:32:52.292 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.574+0000 7f797aa00700 1 --2- 172.21.15.28:0/1836953106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f797c108790 0x7f797c19e8f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f796c011850 tx=0x7f796c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:52.292 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.574+0000 7f7973400700 1 -- 172.21.15.28:0/1836953106 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f796c003af0 con 0x7f797c108790 2024-04-23T14:32:52.292 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.574+0000 7f7973400700 1 -- 172.21.15.28:0/1836953106 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f796c020070 con 0x7f797c108790 2024-04-23T14:32:52.292 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.575+0000 7f7973400700 1 -- 172.21.15.28:0/1836953106 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f796c0195a0 con 0x7f797c108790 2024-04-23T14:32:52.292 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.575+0000 7f7981800700 1 -- 172.21.15.28:0/1836953106 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f797c19f0c0 con 0x7f797c108790 2024-04-23T14:32:52.292 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.575+0000 7f7981800700 1 -- 172.21.15.28:0/1836953106 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f797c1a1db0 con 0x7f797c108790 2024-04-23T14:32:52.293 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.575+0000 7f7973400700 1 -- 172.21.15.28:0/1836953106 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f796c019be0 con 0x7f797c108790 2024-04-23T14:32:52.293 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.575+0000 7f7973400700 1 -- 172.21.15.28:0/1836953106 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f796c011e00 con 0x7f797c108790 2024-04-23T14:32:52.293 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.576+0000 7f7981800700 1 -- 172.21.15.28:0/1836953106 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f797c064150 con 0x7f797c108790 2024-04-23T14:32:52.293 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.579+0000 7f7973400700 1 -- 172.21.15.28:0/1836953106 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f796c003c90 con 0x7f797c108790 2024-04-23T14:32:52.293 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.658+0000 7f7981800700 1 -- 172.21.15.28:0/1836953106 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f797c1a1f80 con 0x7f797c108790 2024-04-23T14:32:52.293 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.659+0000 7f7973400700 1 -- 172.21.15.28:0/1836953106 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7f796c025460 con 0x7f797c108790 2024-04-23T14:32:52.293 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.660+0000 7f7981800700 1 -- 172.21.15.28:0/1836953106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f797c108790 msgr2=0x7f797c19e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:52.293 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.660+0000 7f7981800700 1 --2- 172.21.15.28:0/1836953106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f797c108790 0x7f797c19e8f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f796c011850 tx=0x7f796c011c10 comp rx=0 tx=0).stop 2024-04-23T14:32:52.293 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.661+0000 7f7981800700 1 -- 172.21.15.28:0/1836953106 shutdown_connections 2024-04-23T14:32:52.293 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.661+0000 7f7981800700 1 --2- 172.21.15.28:0/1836953106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f797c108790 0x7f797c19e8f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:52.293 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.661+0000 7f7981800700 1 -- 172.21.15.28:0/1836953106 >> 172.21.15.28:0/1836953106 conn(0x7f797c103d70 msgr2=0x7f797c104830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:52.293 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.661+0000 7f7981800700 1 -- 172.21.15.28:0/1836953106 shutdown_connections 2024-04-23T14:32:52.293 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:51.661+0000 7f7981800700 1 -- 172.21.15.28:0/1836953106 wait complete. 2024-04-23T14:32:52.294 INFO:teuthology.orchestra.run.smithi028.stdout:mgr not available, waiting (3/15)... 2024-04-23T14:32:53.124 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:52 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:32:52.871+0000 7f953deca200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T14:32:53.624 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:53 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:32:53.165+0000 7f953deca200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T14:32:53.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:53 smithi028 ceph-mon[27810]: Activating manager daemon x 2024-04-23T14:32:53.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:53 smithi028 ceph-mon[27810]: mgrmap e2: x(active, starting, since 0.0107269s) 2024-04-23T14:32:53.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:53 smithi028 ceph-mon[27810]: from='mgr.14100 172.21.15.28:0/1000107881' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T14:32:53.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:53 smithi028 ceph-mon[27810]: from='mgr.14100 172.21.15.28:0/1000107881' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T14:32:53.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:53 smithi028 ceph-mon[27810]: from='mgr.14100 172.21.15.28:0/1000107881' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T14:32:53.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:53 smithi028 ceph-mon[27810]: from='mgr.14100 172.21.15.28:0/1000107881' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T14:32:53.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:53 smithi028 ceph-mon[27810]: from='mgr.14100 172.21.15.28:0/1000107881' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T14:32:53.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:53 smithi028 ceph-mon[27810]: Manager daemon x is now available 2024-04-23T14:32:53.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:53 smithi028 ceph-mon[27810]: from='mgr.14100 172.21.15.28:0/1000107881' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T14:32:53.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:53 smithi028 ceph-mon[27810]: from='mgr.14100 172.21.15.28:0/1000107881' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T14:32:53.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:53 smithi028 ceph-mon[27810]: from='mgr.14100 172.21.15.28:0/1000107881' entity='mgr.x' 2024-04-23T14:32:54.569 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:54 smithi028 ceph-mon[27810]: from='mgr.14100 172.21.15.28:0/1000107881' entity='mgr.x' 2024-04-23T14:32:54.569 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:54 smithi028 ceph-mon[27810]: from='mgr.14100 172.21.15.28:0/1000107881' entity='mgr.x' 2024-04-23T14:32:54.569 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:54 smithi028 ceph-mon[27810]: mgrmap e3: x(active, since 1.02042s) 2024-04-23T14:32:55.917 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-23T14:32:55.917 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-23T14:32:55.917 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "1ee6e996-017e-11ef-bc93-c7b262605968", 2024-04-23T14:32:55.917 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-04-23T14:32:55.917 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-23T14:32:55.917 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-23T14:32:55.917 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-23T14:32:55.918 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-23T14:32:55.918 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-23T14:32:55.918 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-23T14:32:55.918 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-04-23T14:32:55.918 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-23T14:32:55.918 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-23T14:32:55.918 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-04-23T14:32:55.918 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-23T14:32:55.918 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-04-23T14:32:55.918 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-23T14:32:55.918 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T14:32:55.918 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-23T14:32:55.918 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-23T14:32:55.918 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-23T14:32:55.919 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-23T14:32:55.919 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T14:32:55.919 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-23T14:32:55.919 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-23T14:32:55.919 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-23T14:32:55.927 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-23T14:32:55.927 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-23T14:32:55.928 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-23T14:32:55.928 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-23T14:32:55.928 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-23T14:32:55.928 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-23T14:32:55.928 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-23T14:32:55.928 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-23T14:32:55.928 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-23T14:32:55.928 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-23T14:32:55.928 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-23T14:32:55.928 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-23T14:32:55.928 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-23T14:32:55.928 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-23T14:32:55.928 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-23T14:32:55.928 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T14:32:55.929 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-23T14:32:55.929 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-23T14:32:55.929 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-23T14:32:55.929 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-23T14:32:55.929 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-23T14:32:55.929 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-23T14:32:55.929 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-23T14:32:55.929 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-23T14:32:55.929 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-23T14:32:55.929 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-04-23T14:32:55.929 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-23T14:32:55.929 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T14:32:55.929 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-23T14:32:55.929 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-23T14:32:55.930 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T14:32:55.930 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-04-23T14:32:34.581743+0000", 2024-04-23T14:32:55.930 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T14:32:55.930 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-23T14:32:55.930 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-23T14:32:55.930 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-23T14:32:55.930 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.922+0000 7fd4cda00700 1 Processor -- start 2024-04-23T14:32:55.930 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.922+0000 7fd4cda00700 1 -- start start 2024-04-23T14:32:55.930 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.923+0000 7fd4cda00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c810a950 0x7fd4c810ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:55.930 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.923+0000 7fd4cda00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c810b340 con 0x7fd4c810a950 2024-04-23T14:32:55.930 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.923+0000 7fd4c6a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c810a950 0x7fd4c810ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:55.930 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.923+0000 7fd4c6a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c810a950 0x7fd4c810ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35628/0 (socket says 172.21.15.28:35628) 2024-04-23T14:32:55.930 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.923+0000 7fd4c6a00700 1 -- 172.21.15.28:0/3112096608 learned_addr learned my addr 172.21.15.28:0/3112096608 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:32:55.931 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.924+0000 7fd4c6a00700 1 -- 172.21.15.28:0/3112096608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4c810bb00 con 0x7fd4c810a950 2024-04-23T14:32:55.931 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.924+0000 7fd4c6a00700 1 --2- 172.21.15.28:0/3112096608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c810a950 0x7fd4c810ad70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd4b80098a0 tx=0x7fd4b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=677a0c365d88dbfe server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:55.931 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.924+0000 7fd4c5600700 1 -- 172.21.15.28:0/3112096608 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd4b8004160 con 0x7fd4c810a950 2024-04-23T14:32:55.931 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.924+0000 7fd4c5600700 1 -- 172.21.15.28:0/3112096608 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd4b8004300 con 0x7fd4c810a950 2024-04-23T14:32:55.931 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.924+0000 7fd4c5600700 1 -- 172.21.15.28:0/3112096608 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd4b8011b50 con 0x7fd4c810a950 2024-04-23T14:32:55.931 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.925+0000 7fd4cda00700 1 -- 172.21.15.28:0/3112096608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c810a950 msgr2=0x7fd4c810ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:55.931 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.925+0000 7fd4cda00700 1 --2- 172.21.15.28:0/3112096608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c810a950 0x7fd4c810ad70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd4b80098a0 tx=0x7fd4b8009bb0 comp rx=0 tx=0).stop 2024-04-23T14:32:55.931 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.925+0000 7fd4cda00700 1 -- 172.21.15.28:0/3112096608 shutdown_connections 2024-04-23T14:32:55.931 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.925+0000 7fd4cda00700 1 --2- 172.21.15.28:0/3112096608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c810a950 0x7fd4c810ad70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:55.931 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.925+0000 7fd4cda00700 1 -- 172.21.15.28:0/3112096608 >> 172.21.15.28:0/3112096608 conn(0x7fd4c807ddd0 msgr2=0x7fd4c81084c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:55.931 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.925+0000 7fd4cda00700 1 -- 172.21.15.28:0/3112096608 shutdown_connections 2024-04-23T14:32:55.931 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.925+0000 7fd4cda00700 1 -- 172.21.15.28:0/3112096608 wait complete. 2024-04-23T14:32:55.931 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.926+0000 7fd4cda00700 1 Processor -- start 2024-04-23T14:32:55.932 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.926+0000 7fd4cda00700 1 -- start start 2024-04-23T14:32:55.932 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.927+0000 7fd4cda00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c810a950 0x7fd4c819e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:55.932 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.927+0000 7fd4cda00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c810eeb0 con 0x7fd4c810a950 2024-04-23T14:32:55.932 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.927+0000 7fd4c6a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c810a950 0x7fd4c819e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:55.932 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.927+0000 7fd4c6a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c810a950 0x7fd4c819e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35644/0 (socket says 172.21.15.28:35644) 2024-04-23T14:32:55.932 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.927+0000 7fd4c6a00700 1 -- 172.21.15.28:0/2362714909 learned_addr learned my addr 172.21.15.28:0/2362714909 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:32:55.932 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.927+0000 7fd4c6a00700 1 -- 172.21.15.28:0/2362714909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4c819ee00 con 0x7fd4c810a950 2024-04-23T14:32:55.932 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.928+0000 7fd4c6a00700 1 --2- 172.21.15.28:0/2362714909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c810a950 0x7fd4c819e8c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd4b8009fb0 tx=0x7fd4b8011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:55.932 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.928+0000 7fd4bf400700 1 -- 172.21.15.28:0/2362714909 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd4b80038b0 con 0x7fd4c810a950 2024-04-23T14:32:55.932 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.928+0000 7fd4bf400700 1 -- 172.21.15.28:0/2362714909 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd4b8011e50 con 0x7fd4c810a950 2024-04-23T14:32:55.932 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.928+0000 7fd4bf400700 1 -- 172.21.15.28:0/2362714909 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd4b8019550 con 0x7fd4c810a950 2024-04-23T14:32:55.932 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.928+0000 7fd4cda00700 1 -- 172.21.15.28:0/2362714909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4c819f090 con 0x7fd4c810a950 2024-04-23T14:32:55.932 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.928+0000 7fd4cda00700 1 -- 172.21.15.28:0/2362714909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4c81a1d80 con 0x7fd4c810a950 2024-04-23T14:32:55.932 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.929+0000 7fd4bf400700 1 -- 172.21.15.28:0/2362714909 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 3) ==== 46441+0+0 (secure 0 0 0) 0x7fd4b8003a50 con 0x7fd4c810a950 2024-04-23T14:32:55.933 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.929+0000 7fd4cda00700 1 -- 172.21.15.28:0/2362714909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4c8064150 con 0x7fd4c810a950 2024-04-23T14:32:55.933 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.930+0000 7fd4bf400700 1 --2- 172.21.15.28:0/2362714909 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7fd4ac039450 0x7fd4ac03b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:55.933 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.930+0000 7fd4bf400700 1 -- 172.21.15.28:0/2362714909 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd4b804d670 con 0x7fd4c810a950 2024-04-23T14:32:55.933 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.930+0000 7fd4c6000700 1 --2- 172.21.15.28:0/2362714909 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7fd4ac039450 0x7fd4ac03b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:55.933 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.931+0000 7fd4c6000700 1 --2- 172.21.15.28:0/2362714909 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7fd4ac039450 0x7fd4ac03b910 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4006fd0 tx=0x7fd4b4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:55.933 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:54.936+0000 7fd4bf400700 1 -- 172.21.15.28:0/2362714909 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd4b8029d50 con 0x7fd4c810a950 2024-04-23T14:32:55.933 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:55.228+0000 7fd4bf400700 1 -- 172.21.15.28:0/2362714909 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7fd4b8023390 con 0x7fd4c810a950 2024-04-23T14:32:55.933 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:55.241+0000 7fd4cda00700 1 -- 172.21.15.28:0/2362714909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fd4c81a1f50 con 0x7fd4c810a950 2024-04-23T14:32:55.933 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:55.241+0000 7fd4bf400700 1 -- 172.21.15.28:0/2362714909 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fd4b8044ab0 con 0x7fd4c810a950 2024-04-23T14:32:55.933 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:55.244+0000 7fd4cda00700 1 -- 172.21.15.28:0/2362714909 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7fd4ac039450 msgr2=0x7fd4ac03b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:55.933 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:55.244+0000 7fd4cda00700 1 --2- 172.21.15.28:0/2362714909 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7fd4ac039450 0x7fd4ac03b910 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4006fd0 tx=0x7fd4b4006e40 comp rx=0 tx=0).stop 2024-04-23T14:32:55.933 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:55.244+0000 7fd4cda00700 1 -- 172.21.15.28:0/2362714909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c810a950 msgr2=0x7fd4c819e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:55.933 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:55.244+0000 7fd4cda00700 1 --2- 172.21.15.28:0/2362714909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c810a950 0x7fd4c819e8c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd4b8009fb0 tx=0x7fd4b8011bf0 comp rx=0 tx=0).stop 2024-04-23T14:32:55.933 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:55.244+0000 7fd4cda00700 1 -- 172.21.15.28:0/2362714909 shutdown_connections 2024-04-23T14:32:55.934 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:55.244+0000 7fd4cda00700 1 --2- 172.21.15.28:0/2362714909 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7fd4ac039450 0x7fd4ac03b910 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:55.934 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:55.244+0000 7fd4cda00700 1 --2- 172.21.15.28:0/2362714909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c810a950 0x7fd4c819e8c0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:55.934 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:55.244+0000 7fd4cda00700 1 -- 172.21.15.28:0/2362714909 >> 172.21.15.28:0/2362714909 conn(0x7fd4c807ddd0 msgr2=0x7fd4c8107c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:55.934 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:55.245+0000 7fd4cda00700 1 -- 172.21.15.28:0/2362714909 shutdown_connections 2024-04-23T14:32:55.934 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:55.245+0000 7fd4cda00700 1 -- 172.21.15.28:0/2362714909 wait complete. 2024-04-23T14:32:55.934 INFO:teuthology.orchestra.run.smithi028.stdout:mgr is available 2024-04-23T14:32:56.525 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:56 smithi028 ceph-mon[27810]: mgrmap e4: x(active, since 2s) 2024-04-23T14:32:56.525 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:56 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2362714909' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-23T14:32:57.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:57 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3845895807' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-23T14:32:57.458 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-23T14:32:57.458 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [global] 2024-04-23T14:32:57.458 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout fsid = 1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T14:32:57.458 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-23T14:32:57.458 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-23T14:32:57.458 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-23T14:32:57.458 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-23T14:32:57.459 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-23T14:32:57.459 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-23T14:32:57.459 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-23T14:32:57.459 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-23T14:32:57.459 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [osd] 2024-04-23T14:32:57.459 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-04-23T14:32:57.459 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-23T14:32:57.459 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-23T14:32:57.459 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-23T14:32:57.459 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-23T14:32:57.459 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-23T14:32:57.459 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.523+0000 7fb9bcc00700 1 Processor -- start 2024-04-23T14:32:57.459 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.524+0000 7fb9bcc00700 1 -- start start 2024-04-23T14:32:57.459 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.524+0000 7fb9bcc00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9b8107000 0x7fb9b8107420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:57.459 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.524+0000 7fb9bcc00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b807be20 con 0x7fb9b8107000 2024-04-23T14:32:57.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.524+0000 7fb9b5e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9b8107000 0x7fb9b8107420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:57.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.525+0000 7fb9b5e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9b8107000 0x7fb9b8107420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35660/0 (socket says 172.21.15.28:35660) 2024-04-23T14:32:57.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.525+0000 7fb9b5e00700 1 -- 172.21.15.28:0/251255159 learned_addr learned my addr 172.21.15.28:0/251255159 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:32:57.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.525+0000 7fb9b5e00700 1 -- 172.21.15.28:0/251255159 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9b8107960 con 0x7fb9b8107000 2024-04-23T14:32:57.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.525+0000 7fb9b5e00700 1 --2- 172.21.15.28:0/251255159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9b8107000 0x7fb9b8107420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb9a80098a0 tx=0x7fb9a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea5b61e7818785f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:57.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.526+0000 7fb9b5400700 1 -- 172.21.15.28:0/251255159 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9a8004160 con 0x7fb9b8107000 2024-04-23T14:32:57.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.526+0000 7fb9b5400700 1 -- 172.21.15.28:0/251255159 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb9a8004300 con 0x7fb9b8107000 2024-04-23T14:32:57.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.526+0000 7fb9b5400700 1 -- 172.21.15.28:0/251255159 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9a8011b50 con 0x7fb9b8107000 2024-04-23T14:32:57.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.526+0000 7fb9bcc00700 1 -- 172.21.15.28:0/251255159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9b8107000 msgr2=0x7fb9b8107420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:57.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.526+0000 7fb9bcc00700 1 --2- 172.21.15.28:0/251255159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9b8107000 0x7fb9b8107420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb9a80098a0 tx=0x7fb9a8009bb0 comp rx=0 tx=0).stop 2024-04-23T14:32:57.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.526+0000 7fb9bcc00700 1 -- 172.21.15.28:0/251255159 shutdown_connections 2024-04-23T14:32:57.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.526+0000 7fb9bcc00700 1 --2- 172.21.15.28:0/251255159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9b8107000 0x7fb9b8107420 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:57.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.526+0000 7fb9bcc00700 1 -- 172.21.15.28:0/251255159 >> 172.21.15.28:0/251255159 conn(0x7fb9b8102c00 msgr2=0x7fb9b8105020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:57.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.527+0000 7fb9bcc00700 1 -- 172.21.15.28:0/251255159 shutdown_connections 2024-04-23T14:32:57.461 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.527+0000 7fb9bcc00700 1 -- 172.21.15.28:0/251255159 wait complete. 2024-04-23T14:32:57.461 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.528+0000 7fb9bcc00700 1 Processor -- start 2024-04-23T14:32:57.461 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.528+0000 7fb9bcc00700 1 -- start start 2024-04-23T14:32:57.461 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.528+0000 7fb9bcc00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9b8107000 0x7fb9b819e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:57.461 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.528+0000 7fb9bcc00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b8108980 con 0x7fb9b8107000 2024-04-23T14:32:57.461 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.529+0000 7fb9b5e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9b8107000 0x7fb9b819e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:57.461 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.529+0000 7fb9b5e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9b8107000 0x7fb9b819e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35674/0 (socket says 172.21.15.28:35674) 2024-04-23T14:32:57.461 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.529+0000 7fb9b5e00700 1 -- 172.21.15.28:0/3845895807 learned_addr learned my addr 172.21.15.28:0/3845895807 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:32:57.461 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.529+0000 7fb9b5e00700 1 -- 172.21.15.28:0/3845895807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9b819eda0 con 0x7fb9b8107000 2024-04-23T14:32:57.461 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.529+0000 7fb9b5e00700 1 --2- 172.21.15.28:0/3845895807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9b8107000 0x7fb9b819e860 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb9a8009fb0 tx=0x7fb9a8011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:57.461 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.530+0000 7fb9aea00700 1 -- 172.21.15.28:0/3845895807 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9a80038b0 con 0x7fb9b8107000 2024-04-23T14:32:57.461 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.530+0000 7fb9aea00700 1 -- 172.21.15.28:0/3845895807 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb9a8011e50 con 0x7fb9b8107000 2024-04-23T14:32:57.461 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.530+0000 7fb9aea00700 1 -- 172.21.15.28:0/3845895807 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9a8019420 con 0x7fb9b8107000 2024-04-23T14:32:57.461 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.530+0000 7fb9bcc00700 1 -- 172.21.15.28:0/3845895807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9b819f030 con 0x7fb9b8107000 2024-04-23T14:32:57.462 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.530+0000 7fb9bcc00700 1 -- 172.21.15.28:0/3845895807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9b81a1d20 con 0x7fb9b8107000 2024-04-23T14:32:57.462 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.531+0000 7fb9aea00700 1 -- 172.21.15.28:0/3845895807 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7fb9a8003a50 con 0x7fb9b8107000 2024-04-23T14:32:57.462 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.531+0000 7fb9bcc00700 1 -- 172.21.15.28:0/3845895807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9b8064150 con 0x7fb9b8107000 2024-04-23T14:32:57.462 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.531+0000 7fb9aea00700 1 --2- 172.21.15.28:0/3845895807 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7fb998039560 0x7fb99803ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:57.462 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.531+0000 7fb9aea00700 1 -- 172.21.15.28:0/3845895807 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb9a804d880 con 0x7fb9b8107000 2024-04-23T14:32:57.462 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.532+0000 7fb9afe00700 1 --2- 172.21.15.28:0/3845895807 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7fb998039560 0x7fb99803ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:57.462 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.532+0000 7fb9afe00700 1 --2- 172.21.15.28:0/3845895807 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7fb998039560 0x7fb99803ba20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb9a0006fd0 tx=0x7fb9a0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:57.462 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.537+0000 7fb9aea00700 1 -- 172.21.15.28:0/3845895807 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb9a80172b0 con 0x7fb9b8107000 2024-04-23T14:32:57.462 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.769+0000 7fb9bcc00700 1 -- 172.21.15.28:0/3845895807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fb9b802e250 con 0x7fb9b8107000 2024-04-23T14:32:57.462 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.772+0000 7fb9aea00700 1 -- 172.21.15.28:0/3845895807 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+400 (secure 0 0 0) 0x7fb9a804d2a0 con 0x7fb9b8107000 2024-04-23T14:32:57.462 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.775+0000 7fb9bcc00700 1 -- 172.21.15.28:0/3845895807 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7fb998039560 msgr2=0x7fb99803ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:57.463 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.775+0000 7fb9bcc00700 1 --2- 172.21.15.28:0/3845895807 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7fb998039560 0x7fb99803ba20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb9a0006fd0 tx=0x7fb9a0006e40 comp rx=0 tx=0).stop 2024-04-23T14:32:57.463 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.775+0000 7fb9bcc00700 1 -- 172.21.15.28:0/3845895807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9b8107000 msgr2=0x7fb9b819e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:57.463 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.775+0000 7fb9bcc00700 1 --2- 172.21.15.28:0/3845895807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9b8107000 0x7fb9b819e860 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb9a8009fb0 tx=0x7fb9a8011bf0 comp rx=0 tx=0).stop 2024-04-23T14:32:57.463 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.775+0000 7fb9bcc00700 1 -- 172.21.15.28:0/3845895807 shutdown_connections 2024-04-23T14:32:57.463 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.775+0000 7fb9bcc00700 1 --2- 172.21.15.28:0/3845895807 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7fb998039560 0x7fb99803ba20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:57.463 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.775+0000 7fb9bcc00700 1 --2- 172.21.15.28:0/3845895807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9b8107000 0x7fb9b819e860 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:57.463 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.775+0000 7fb9bcc00700 1 -- 172.21.15.28:0/3845895807 >> 172.21.15.28:0/3845895807 conn(0x7fb9b8102c00 msgr2=0x7fb9b8195560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:57.463 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.775+0000 7fb9bcc00700 1 -- 172.21.15.28:0/3845895807 shutdown_connections 2024-04-23T14:32:57.463 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:56.775+0000 7fb9bcc00700 1 -- 172.21.15.28:0/3845895807 wait complete. 2024-04-23T14:32:57.464 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling cephadm module... 2024-04-23T14:32:58.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:58 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1328519180' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-23T14:32:58.746 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:58 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: ignoring --setuser ceph since I am not root 2024-04-23T14:32:58.746 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:58 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: ignoring --setgroup ceph since I am not root 2024-04-23T14:32:58.746 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:58 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:32:58.744+0000 7f04f0005200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T14:32:59.124 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:58 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:32:58.991+0000 7f04f0005200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T14:32:59.173 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.079+0000 7f033c000700 1 Processor -- start 2024-04-23T14:32:59.175 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.080+0000 7f033c000700 1 -- start start 2024-04-23T14:32:59.175 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.080+0000 7f033c000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f033410a990 0x7f033410adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:59.175 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.080+0000 7f033c000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f033410b380 con 0x7f033410a990 2024-04-23T14:32:59.175 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.081+0000 7f0339600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f033410a990 0x7f033410adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:59.175 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.081+0000 7f0339600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f033410a990 0x7f033410adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39888/0 (socket says 172.21.15.28:39888) 2024-04-23T14:32:59.175 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.081+0000 7f0339600700 1 -- 172.21.15.28:0/3968443281 learned_addr learned my addr 172.21.15.28:0/3968443281 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:32:59.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.081+0000 7f0339600700 1 -- 172.21.15.28:0/3968443281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f033410bb40 con 0x7f033410a990 2024-04-23T14:32:59.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.081+0000 7f0339600700 1 --2- 172.21.15.28:0/3968443281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f033410a990 0x7f033410adb0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f03280098a0 tx=0x7f0328009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e14329ec7be1dd10 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:59.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.082+0000 7f0333e00700 1 -- 172.21.15.28:0/3968443281 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0328004160 con 0x7f033410a990 2024-04-23T14:32:59.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.082+0000 7f0333e00700 1 -- 172.21.15.28:0/3968443281 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0328004300 con 0x7f033410a990 2024-04-23T14:32:59.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.082+0000 7f0333e00700 1 -- 172.21.15.28:0/3968443281 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0328011b50 con 0x7f033410a990 2024-04-23T14:32:59.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.083+0000 7f033c000700 1 -- 172.21.15.28:0/3968443281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f033410a990 msgr2=0x7f033410adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:59.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.083+0000 7f033c000700 1 --2- 172.21.15.28:0/3968443281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f033410a990 0x7f033410adb0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f03280098a0 tx=0x7f0328009bb0 comp rx=0 tx=0).stop 2024-04-23T14:32:59.177 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.083+0000 7f033c000700 1 -- 172.21.15.28:0/3968443281 shutdown_connections 2024-04-23T14:32:59.177 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.083+0000 7f033c000700 1 --2- 172.21.15.28:0/3968443281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f033410a990 0x7f033410adb0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:59.181 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.083+0000 7f033c000700 1 -- 172.21.15.28:0/3968443281 >> 172.21.15.28:0/3968443281 conn(0x7f033407de00 msgr2=0x7f03341084c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:59.181 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.083+0000 7f033c000700 1 -- 172.21.15.28:0/3968443281 shutdown_connections 2024-04-23T14:32:59.181 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.083+0000 7f033c000700 1 -- 172.21.15.28:0/3968443281 wait complete. 2024-04-23T14:32:59.181 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.084+0000 7f033c000700 1 Processor -- start 2024-04-23T14:32:59.181 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.085+0000 7f033c000700 1 -- start start 2024-04-23T14:32:59.181 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.085+0000 7f033c000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f033410a990 0x7f033419e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:59.181 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.085+0000 7f033c000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f033410eef0 con 0x7f033410a990 2024-04-23T14:32:59.182 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.085+0000 7f0339600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f033410a990 0x7f033419e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:59.182 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.085+0000 7f0339600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f033410a990 0x7f033419e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39902/0 (socket says 172.21.15.28:39902) 2024-04-23T14:32:59.182 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.085+0000 7f0339600700 1 -- 172.21.15.28:0/1328519180 learned_addr learned my addr 172.21.15.28:0/1328519180 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:32:59.182 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.086+0000 7f0339600700 1 -- 172.21.15.28:0/1328519180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f033419ee10 con 0x7f033410a990 2024-04-23T14:32:59.182 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.086+0000 7f0339600700 1 --2- 172.21.15.28:0/1328519180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f033410a990 0x7f033419e8d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0328011850 tx=0x7f0328011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:59.182 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.086+0000 7f0332000700 1 -- 172.21.15.28:0/1328519180 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0328003af0 con 0x7f033410a990 2024-04-23T14:32:59.182 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.086+0000 7f0332000700 1 -- 172.21.15.28:0/1328519180 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0328020070 con 0x7f033410a990 2024-04-23T14:32:59.182 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.086+0000 7f033c000700 1 -- 172.21.15.28:0/1328519180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f033419f0a0 con 0x7f033410a990 2024-04-23T14:32:59.182 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.086+0000 7f0332000700 1 -- 172.21.15.28:0/1328519180 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f03280195a0 con 0x7f033410a990 2024-04-23T14:32:59.182 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.087+0000 7f033c000700 1 -- 172.21.15.28:0/1328519180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03341a1d90 con 0x7f033410a990 2024-04-23T14:32:59.182 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.088+0000 7f0332000700 1 -- 172.21.15.28:0/1328519180 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f0328019850 con 0x7f033410a990 2024-04-23T14:32:59.183 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.088+0000 7f033c000700 1 -- 172.21.15.28:0/1328519180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0334064150 con 0x7f033410a990 2024-04-23T14:32:59.183 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.088+0000 7f0332000700 1 --2- 172.21.15.28:0/1328519180 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7f031c039600 0x7f031c03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:32:59.183 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.088+0000 7f0332000700 1 -- 172.21.15.28:0/1328519180 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0328011e00 con 0x7f033410a990 2024-04-23T14:32:59.183 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.089+0000 7f0338c00700 1 --2- 172.21.15.28:0/1328519180 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7f031c039600 0x7f031c03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:32:59.183 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.089+0000 7f0338c00700 1 --2- 172.21.15.28:0/1328519180 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7f031c039600 0x7f031c03bac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0320006fd0 tx=0x7f0320006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:32:59.183 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.094+0000 7f0332000700 1 -- 172.21.15.28:0/1328519180 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f032800e260 con 0x7f033410a990 2024-04-23T14:32:59.183 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.387+0000 7f033c000700 1 -- 172.21.15.28:0/1328519180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f03341a1f60 con 0x7f033410a990 2024-04-23T14:32:59.183 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.441+0000 7f0332000700 1 -- 172.21.15.28:0/1328519180 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f0328003c90 con 0x7f033410a990 2024-04-23T14:32:59.183 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.446+0000 7f0332000700 1 -- 172.21.15.28:0/1328519180 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f032801a430 con 0x7f033410a990 2024-04-23T14:32:59.183 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.446+0000 7f033c000700 1 -- 172.21.15.28:0/1328519180 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7f031c039600 msgr2=0x7f031c03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:59.183 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.446+0000 7f033c000700 1 --2- 172.21.15.28:0/1328519180 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7f031c039600 0x7f031c03bac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0320006fd0 tx=0x7f0320006e40 comp rx=0 tx=0).stop 2024-04-23T14:32:59.183 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.446+0000 7f033c000700 1 -- 172.21.15.28:0/1328519180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f033410a990 msgr2=0x7f033419e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:32:59.183 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.446+0000 7f033c000700 1 --2- 172.21.15.28:0/1328519180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f033410a990 0x7f033419e8d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0328011850 tx=0x7f0328011c10 comp rx=0 tx=0).stop 2024-04-23T14:32:59.184 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.446+0000 7f033c000700 1 -- 172.21.15.28:0/1328519180 shutdown_connections 2024-04-23T14:32:59.184 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.446+0000 7f033c000700 1 --2- 172.21.15.28:0/1328519180 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7f031c039600 0x7f031c03bac0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:59.184 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.446+0000 7f033c000700 1 --2- 172.21.15.28:0/1328519180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f033410a990 0x7f033419e8d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:32:59.184 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.446+0000 7f033c000700 1 -- 172.21.15.28:0/1328519180 >> 172.21.15.28:0/1328519180 conn(0x7f033407de00 msgr2=0x7f0334107cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:32:59.184 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.447+0000 7f033c000700 1 -- 172.21.15.28:0/1328519180 shutdown_connections 2024-04-23T14:32:59.184 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:58.447+0000 7f033c000700 1 -- 172.21.15.28:0/1328519180 wait complete. 2024-04-23T14:32:59.451 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:59 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:32:59.126+0000 7f04f0005200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T14:32:59.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:59 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1328519180' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-23T14:32:59.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:32:59 smithi028 ceph-mon[27810]: mgrmap e5: x(active, since 5s) 2024-04-23T14:32:59.757 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:59 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:32:59.547+0000 7f04f0005200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T14:32:59.757 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:32:59 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:32:59.663+0000 7f04f0005200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T14:33:00.760 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:00 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3208557322' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-23T14:33:00.771 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-23T14:33:00.772 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-23T14:33:00.772 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-23T14:33:00.772 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-23T14:33:00.773 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-23T14:33:00.773 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-23T14:33:00.773 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.806+0000 7fa9d4a00700 1 Processor -- start 2024-04-23T14:33:00.773 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.807+0000 7fa9d4a00700 1 -- start start 2024-04-23T14:33:00.774 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.807+0000 7fa9d4a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9d01090a0 0x7fa9d010b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:00.774 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.807+0000 7fa9d4a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d007be20 con 0x7fa9d01090a0 2024-04-23T14:33:00.774 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.808+0000 7fa9ce800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9d01090a0 0x7fa9d010b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:00.774 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.808+0000 7fa9ce800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9d01090a0 0x7fa9d010b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39942/0 (socket says 172.21.15.28:39942) 2024-04-23T14:33:00.775 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.808+0000 7fa9ce800700 1 -- 172.21.15.28:0/4250392262 learned_addr learned my addr 172.21.15.28:0/4250392262 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:00.775 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.808+0000 7fa9ce800700 1 -- 172.21.15.28:0/4250392262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9d010b9d0 con 0x7fa9d01090a0 2024-04-23T14:33:00.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.808+0000 7fa9ce800700 1 --2- 172.21.15.28:0/4250392262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9d01090a0 0x7fa9d010b490 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa9c00098a0 tx=0x7fa9c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=817a9834a9552d08 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:00.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.809+0000 7fa9cde00700 1 -- 172.21.15.28:0/4250392262 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa9c0004160 con 0x7fa9d01090a0 2024-04-23T14:33:00.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.809+0000 7fa9cde00700 1 -- 172.21.15.28:0/4250392262 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa9c0004300 con 0x7fa9d01090a0 2024-04-23T14:33:00.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.809+0000 7fa9cde00700 1 -- 172.21.15.28:0/4250392262 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa9c0011b50 con 0x7fa9d01090a0 2024-04-23T14:33:00.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.809+0000 7fa9d4a00700 1 -- 172.21.15.28:0/4250392262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9d01090a0 msgr2=0x7fa9d010b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:00.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.809+0000 7fa9d4a00700 1 --2- 172.21.15.28:0/4250392262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9d01090a0 0x7fa9d010b490 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa9c00098a0 tx=0x7fa9c0009bb0 comp rx=0 tx=0).stop 2024-04-23T14:33:00.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.809+0000 7fa9d4a00700 1 -- 172.21.15.28:0/4250392262 shutdown_connections 2024-04-23T14:33:00.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.809+0000 7fa9d4a00700 1 --2- 172.21.15.28:0/4250392262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9d01090a0 0x7fa9d010b490 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:00.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.809+0000 7fa9d4a00700 1 -- 172.21.15.28:0/4250392262 >> 172.21.15.28:0/4250392262 conn(0x7fa9d0102a70 msgr2=0x7fa9d0104e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:00.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.810+0000 7fa9d4a00700 1 -- 172.21.15.28:0/4250392262 shutdown_connections 2024-04-23T14:33:00.780 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.810+0000 7fa9d4a00700 1 -- 172.21.15.28:0/4250392262 wait complete. 2024-04-23T14:33:00.780 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.810+0000 7fa9d4a00700 1 Processor -- start 2024-04-23T14:33:00.780 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.811+0000 7fa9d4a00700 1 -- start start 2024-04-23T14:33:00.780 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.811+0000 7fa9d4a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9d01090a0 0x7fa9d0108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:00.780 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.811+0000 7fa9d4a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d010c8b0 con 0x7fa9d01090a0 2024-04-23T14:33:00.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.811+0000 7fa9ce800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9d01090a0 0x7fa9d0108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:00.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.811+0000 7fa9ce800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9d01090a0 0x7fa9d0108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39956/0 (socket says 172.21.15.28:39956) 2024-04-23T14:33:00.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.811+0000 7fa9ce800700 1 -- 172.21.15.28:0/3208557322 learned_addr learned my addr 172.21.15.28:0/3208557322 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:00.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.812+0000 7fa9ce800700 1 -- 172.21.15.28:0/3208557322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9d0108e60 con 0x7fa9d01090a0 2024-04-23T14:33:00.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.812+0000 7fa9ce800700 1 --2- 172.21.15.28:0/3208557322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9d01090a0 0x7fa9d0108920 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa9c0009fb0 tx=0x7fa9c0011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:00.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.812+0000 7fa9c7e00700 1 -- 172.21.15.28:0/3208557322 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa9c00038b0 con 0x7fa9d01090a0 2024-04-23T14:33:00.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.812+0000 7fa9c7e00700 1 -- 172.21.15.28:0/3208557322 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa9c0011e50 con 0x7fa9d01090a0 2024-04-23T14:33:00.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.812+0000 7fa9d4a00700 1 -- 172.21.15.28:0/3208557322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9d0107050 con 0x7fa9d01090a0 2024-04-23T14:33:00.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.812+0000 7fa9c7e00700 1 -- 172.21.15.28:0/3208557322 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa9c0019420 con 0x7fa9d01090a0 2024-04-23T14:33:00.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.813+0000 7fa9d4a00700 1 -- 172.21.15.28:0/3208557322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9d0107590 con 0x7fa9d01090a0 2024-04-23T14:33:00.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.814+0000 7fa9c7e00700 1 -- 172.21.15.28:0/3208557322 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7fa9c00196d0 con 0x7fa9d01090a0 2024-04-23T14:33:00.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.814+0000 7fa9d4a00700 1 -- 172.21.15.28:0/3208557322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9d0064150 con 0x7fa9d01090a0 2024-04-23T14:33:00.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.814+0000 7fa9c7e00700 1 --2- 172.21.15.28:0/3208557322 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7fa9b00395b0 0x7fa9b003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:00.782 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.814+0000 7fa9c7e00700 1 -- 172.21.15.28:0/3208557322 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa9c004d970 con 0x7fa9d01090a0 2024-04-23T14:33:00.782 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.814+0000 7fa9c5e00700 1 -- 172.21.15.28:0/3208557322 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7fa9b00395b0 msgr2=0x7fa9b003ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1307025091 2024-04-23T14:33:00.782 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.814+0000 7fa9c5e00700 1 --2- 172.21.15.28:0/3208557322 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7fa9b00395b0 0x7fa9b003ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T14:33:00.782 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:32:59.820+0000 7fa9c7e00700 1 -- 172.21.15.28:0/3208557322 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa9c000bc70 con 0x7fa9d01090a0 2024-04-23T14:33:00.782 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:00.015+0000 7fa9c5e00700 1 -- 172.21.15.28:0/3208557322 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7fa9b00395b0 msgr2=0x7fa9b003ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1307025091 2024-04-23T14:33:00.782 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:00.015+0000 7fa9c5e00700 1 --2- 172.21.15.28:0/3208557322 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7fa9b00395b0 0x7fa9b003ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T14:33:00.782 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:00.108+0000 7fa9d4a00700 1 -- 172.21.15.28:0/3208557322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fa9d0107ec0 con 0x7fa9d01090a0 2024-04-23T14:33:00.782 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:00.109+0000 7fa9c7e00700 1 -- 172.21.15.28:0/3208557322 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7fa9c004d390 con 0x7fa9d01090a0 2024-04-23T14:33:00.782 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:00.112+0000 7fa9d4a00700 1 -- 172.21.15.28:0/3208557322 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7fa9b00395b0 msgr2=0x7fa9b003ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T14:33:00.782 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:00.112+0000 7fa9d4a00700 1 --2- 172.21.15.28:0/3208557322 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7fa9b00395b0 0x7fa9b003ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:00.782 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:00.112+0000 7fa9d4a00700 1 -- 172.21.15.28:0/3208557322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9d01090a0 msgr2=0x7fa9d0108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:00.782 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:00.112+0000 7fa9d4a00700 1 --2- 172.21.15.28:0/3208557322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9d01090a0 0x7fa9d0108920 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa9c0009fb0 tx=0x7fa9c0011bf0 comp rx=0 tx=0).stop 2024-04-23T14:33:00.782 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:00.112+0000 7fa9d4a00700 1 -- 172.21.15.28:0/3208557322 shutdown_connections 2024-04-23T14:33:00.782 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:00.112+0000 7fa9d4a00700 1 --2- 172.21.15.28:0/3208557322 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7fa9b00395b0 0x7fa9b003ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:00.783 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:00.112+0000 7fa9d4a00700 1 --2- 172.21.15.28:0/3208557322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9d01090a0 0x7fa9d0108920 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:00.783 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:00.112+0000 7fa9d4a00700 1 -- 172.21.15.28:0/3208557322 >> 172.21.15.28:0/3208557322 conn(0x7fa9d0102a70 msgr2=0x7fa9d0103620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:00.783 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:00.112+0000 7fa9d4a00700 1 -- 172.21.15.28:0/3208557322 shutdown_connections 2024-04-23T14:33:00.783 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:00.112+0000 7fa9d4a00700 1 -- 172.21.15.28:0/3208557322 wait complete. 2024-04-23T14:33:00.783 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for the mgr to restart... 2024-04-23T14:33:00.783 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr epoch 5... 2024-04-23T14:33:01.373 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:01 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:01.264+0000 7f04f0005200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T14:33:01.721 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:01 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:01.501+0000 7f04f0005200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T14:33:01.721 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:01 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:01.719+0000 7f04f0005200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T14:33:02.238 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:02 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:02.112+0000 7f04f0005200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T14:33:02.238 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:02 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:02.236+0000 7f04f0005200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T14:33:02.754 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:02 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:02.492+0000 7f04f0005200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T14:33:02.755 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:02 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:02.753+0000 7f04f0005200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T14:33:03.873 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:03 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:03.556+0000 7f04f0005200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T14:33:04.389 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:04 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:04.125+0000 7f04f0005200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T14:33:04.389 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:04 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:04.256+0000 7f04f0005200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T14:33:04.873 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:04 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:04.528+0000 7f04f0005200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T14:33:05.624 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:05 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:05.200+0000 7f04f0005200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T14:33:05.624 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:05 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:05.326+0000 7f04f0005200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T14:33:05.974 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:05 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:05.660+0000 7f04f0005200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T14:33:05.975 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:05 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:05.842+0000 7f04f0005200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T14:33:05.975 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:05 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:05.972+0000 7f04f0005200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T14:33:06.373 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:06 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:06.127+0000 7f04f0005200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T14:33:06.874 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:06 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:06.500+0000 7f04f0005200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T14:33:07.574 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:07 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:07.282+0000 7f04f0005200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T14:33:07.874 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:07 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:07.573+0000 7f04f0005200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T14:33:07.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:07 smithi028 ceph-mon[27810]: Active manager daemon x restarted 2024-04-23T14:33:07.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:07 smithi028 ceph-mon[27810]: Activating manager daemon x 2024-04-23T14:33:07.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:07 smithi028 ceph-mon[27810]: osdmap e2: 0 total, 0 up, 0 in 2024-04-23T14:33:07.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:07 smithi028 ceph-mon[27810]: mgrmap e6: x(active, starting, since 0.0103889s) 2024-04-23T14:33:07.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:07 smithi028 ceph-mon[27810]: from='mgr.14120 172.21.15.28:0/2789735782' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T14:33:07.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:07 smithi028 ceph-mon[27810]: from='mgr.14120 172.21.15.28:0/2789735782' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T14:33:07.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:07 smithi028 ceph-mon[27810]: from='mgr.14120 172.21.15.28:0/2789735782' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T14:33:07.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:07 smithi028 ceph-mon[27810]: from='mgr.14120 172.21.15.28:0/2789735782' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T14:33:07.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:07 smithi028 ceph-mon[27810]: from='mgr.14120 172.21.15.28:0/2789735782' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T14:33:07.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:07 smithi028 ceph-mon[27810]: Manager daemon x is now available 2024-04-23T14:33:07.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:07 smithi028 ceph-mon[27810]: from='mgr.14120 172.21.15.28:0/2789735782' entity='mgr.x' 2024-04-23T14:33:08.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:08 smithi028 ceph-mon[27810]: Found migration_current of "None". Setting to last migration. 2024-04-23T14:33:08.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:08 smithi028 ceph-mon[27810]: from='mgr.14120 172.21.15.28:0/2789735782' entity='mgr.x' 2024-04-23T14:33:08.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:08 smithi028 ceph-mon[27810]: from='mgr.14120 172.21.15.28:0/2789735782' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:33:08.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:08 smithi028 ceph-mon[27810]: from='mgr.14120 172.21.15.28:0/2789735782' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:33:08.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:08 smithi028 ceph-mon[27810]: from='mgr.14120 172.21.15.28:0/2789735782' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T14:33:08.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:08 smithi028 ceph-mon[27810]: from='mgr.14120 172.21.15.28:0/2789735782' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T14:33:08.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:08 smithi028 ceph-mon[27810]: from='mgr.14120 172.21.15.28:0/2789735782' entity='mgr.x' 2024-04-23T14:33:08.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:08 smithi028 ceph-mon[27810]: from='mgr.14120 172.21.15.28:0/2789735782' entity='mgr.x' 2024-04-23T14:33:08.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:08 smithi028 ceph-mon[27810]: mgrmap e7: x(active, since 1.01809s) 2024-04-23T14:33:09.349 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-23T14:33:09.349 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-23T14:33:09.349 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-23T14:33:09.349 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-23T14:33:09.349 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.374+0000 7f533f200700 1 Processor -- start 2024-04-23T14:33:09.350 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.374+0000 7f533f200700 1 -- start start 2024-04-23T14:33:09.350 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.374+0000 7f533f200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53401074b0 0x7f53401078d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:09.350 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.374+0000 7f533f200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5340107ea0 con 0x7f53401074b0 2024-04-23T14:33:09.350 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.375+0000 7f533de00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53401074b0 0x7f53401078d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:09.350 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.375+0000 7f533de00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53401074b0 0x7f53401078d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39970/0 (socket says 172.21.15.28:39970) 2024-04-23T14:33:09.350 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.375+0000 7f533de00700 1 -- 172.21.15.28:0/2263270824 learned_addr learned my addr 172.21.15.28:0/2263270824 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:09.350 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.375+0000 7f533de00700 1 -- 172.21.15.28:0/2263270824 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53401086b0 con 0x7f53401074b0 2024-04-23T14:33:09.350 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.376+0000 7f533de00700 1 --2- 172.21.15.28:0/2263270824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53401074b0 0x7f53401078d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f533000bb40 tx=0x7f533000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=366a78d1b69b1dd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:09.350 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.376+0000 7f533ca00700 1 -- 172.21.15.28:0/2263270824 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f533000dbe0 con 0x7f53401074b0 2024-04-23T14:33:09.350 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.376+0000 7f533ca00700 1 -- 172.21.15.28:0/2263270824 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f53300044f0 con 0x7f53401074b0 2024-04-23T14:33:09.350 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.376+0000 7f533ca00700 1 -- 172.21.15.28:0/2263270824 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5330010470 con 0x7f53401074b0 2024-04-23T14:33:09.350 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.377+0000 7f533f200700 1 -- 172.21.15.28:0/2263270824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53401074b0 msgr2=0x7f53401078d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:09.350 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.377+0000 7f533f200700 1 --2- 172.21.15.28:0/2263270824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53401074b0 0x7f53401078d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f533000bb40 tx=0x7f533000be50 comp rx=0 tx=0).stop 2024-04-23T14:33:09.351 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.377+0000 7f533f200700 1 -- 172.21.15.28:0/2263270824 shutdown_connections 2024-04-23T14:33:09.351 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.377+0000 7f533f200700 1 --2- 172.21.15.28:0/2263270824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53401074b0 0x7f53401078d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:09.351 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.377+0000 7f533f200700 1 -- 172.21.15.28:0/2263270824 >> 172.21.15.28:0/2263270824 conn(0x7f5340102a70 msgr2=0x7f5340104e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:09.351 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.377+0000 7f533f200700 1 -- 172.21.15.28:0/2263270824 shutdown_connections 2024-04-23T14:33:09.351 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.377+0000 7f533f200700 1 -- 172.21.15.28:0/2263270824 wait complete. 2024-04-23T14:33:09.351 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.379+0000 7f533f200700 1 Processor -- start 2024-04-23T14:33:09.351 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.379+0000 7f533f200700 1 -- start start 2024-04-23T14:33:09.351 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.379+0000 7f533f200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53401074b0 0x7f534019e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:09.351 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.379+0000 7f533f200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f534007afc0 con 0x7f53401074b0 2024-04-23T14:33:09.351 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.379+0000 7f533de00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53401074b0 0x7f534019e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:09.351 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.379+0000 7f533de00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53401074b0 0x7f534019e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39972/0 (socket says 172.21.15.28:39972) 2024-04-23T14:33:09.351 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.380+0000 7f533de00700 1 -- 172.21.15.28:0/1346655039 learned_addr learned my addr 172.21.15.28:0/1346655039 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:09.351 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.380+0000 7f533de00700 1 -- 172.21.15.28:0/1346655039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f534019ec40 con 0x7f53401074b0 2024-04-23T14:33:09.351 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.380+0000 7f533de00700 1 --2- 172.21.15.28:0/1346655039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53401074b0 0x7f534019e700 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f533000b340 tx=0x7f533000dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:09.351 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.380+0000 7f5336a00700 1 -- 172.21.15.28:0/1346655039 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5330010680 con 0x7f53401074b0 2024-04-23T14:33:09.351 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.380+0000 7f5336a00700 1 -- 172.21.15.28:0/1346655039 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f533001d070 con 0x7f53401074b0 2024-04-23T14:33:09.352 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.380+0000 7f533f200700 1 -- 172.21.15.28:0/1346655039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f534019eed0 con 0x7f53401074b0 2024-04-23T14:33:09.352 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.381+0000 7f5336a00700 1 -- 172.21.15.28:0/1346655039 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5330021af0 con 0x7f53401074b0 2024-04-23T14:33:09.352 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.381+0000 7f533f200700 1 -- 172.21.15.28:0/1346655039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53401a1bc0 con 0x7f53401074b0 2024-04-23T14:33:09.352 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.382+0000 7f5336a00700 1 -- 172.21.15.28:0/1346655039 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f533001f030 con 0x7f53401074b0 2024-04-23T14:33:09.352 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.382+0000 7f5336a00700 1 --2- 172.21.15.28:0/1346655039 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7f5324039600 0x7f532403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:09.352 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.382+0000 7f5336a00700 1 -- 172.21.15.28:0/1346655039 --> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f532403c1d0 con 0x7f5324039600 2024-04-23T14:33:09.352 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.382+0000 7f5336a00700 1 -- 172.21.15.28:0/1346655039 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f533004d5b0 con 0x7f53401074b0 2024-04-23T14:33:09.352 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.382+0000 7f533d400700 1 -- 172.21.15.28:0/1346655039 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7f5324039600 msgr2=0x7f532403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1307025091 2024-04-23T14:33:09.352 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.382+0000 7f533d400700 1 --2- 172.21.15.28:0/1346655039 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7f5324039600 0x7f532403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T14:33:09.352 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.583+0000 7f533d400700 1 -- 172.21.15.28:0/1346655039 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7f5324039600 msgr2=0x7f532403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1307025091 2024-04-23T14:33:09.352 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.583+0000 7f533d400700 1 --2- 172.21.15.28:0/1346655039 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7f5324039600 0x7f532403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T14:33:09.352 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.984+0000 7f533d400700 1 -- 172.21.15.28:0/1346655039 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7f5324039600 msgr2=0x7f532403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1307025091 2024-04-23T14:33:09.352 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:01.984+0000 7f533d400700 1 --2- 172.21.15.28:0/1346655039 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7f5324039600 0x7f532403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-23T14:33:09.352 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:02.785+0000 7f533d400700 1 -- 172.21.15.28:0/1346655039 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7f5324039600 msgr2=0x7f532403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1307025091 2024-04-23T14:33:09.352 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:02.785+0000 7f533d400700 1 --2- 172.21.15.28:0/1346655039 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7f5324039600 0x7f532403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-23T14:33:09.352 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:04.387+0000 7f533d400700 1 -- 172.21.15.28:0/1346655039 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7f5324039600 msgr2=0x7f532403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1307025091 2024-04-23T14:33:09.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:04.387+0000 7f533d400700 1 --2- 172.21.15.28:0/1346655039 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7f5324039600 0x7f532403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-23T14:33:09.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:07.583+0000 7f5336a00700 1 -- 172.21.15.28:0/1346655039 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mgrmap(e 6) ==== 46325+0+0 (secure 0 0 0) 0x7f5330019770 con 0x7f53401074b0 2024-04-23T14:33:09.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:07.583+0000 7f5336a00700 1 -- 172.21.15.28:0/1346655039 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7f5324039600 msgr2=0x7f532403bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T14:33:09.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:07.583+0000 7f5336a00700 1 --2- 172.21.15.28:0/1346655039 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7f5324039600 0x7f532403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:09.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:08.588+0000 7f5336a00700 1 -- 172.21.15.28:0/1346655039 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f533004ef20 con 0x7f53401074b0 2024-04-23T14:33:09.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:08.589+0000 7f5336a00700 1 --2- 172.21.15.28:0/1346655039 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f532403cf40 0x7f532403f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:09.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:08.589+0000 7f5336a00700 1 -- 172.21.15.28:0/1346655039 --> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f533001be50 con 0x7f532403cf40 2024-04-23T14:33:09.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:08.591+0000 7f533d400700 1 --2- 172.21.15.28:0/1346655039 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f532403cf40 0x7f532403f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:09.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:08.593+0000 7f533d400700 1 --2- 172.21.15.28:0/1346655039 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f532403cf40 0x7f532403f330 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f532c003a10 tx=0x7f532c0092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:09.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:08.594+0000 7f5336a00700 1 -- 172.21.15.28:0/1346655039 <== mgr.14120 v2:172.21.15.28:6800/2222038555 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f533001be50 con 0x7f532403cf40 2024-04-23T14:33:09.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:08.603+0000 7f533f200700 1 -- 172.21.15.28:0/1346655039 --> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f5320002630 con 0x7f532403cf40 2024-04-23T14:33:09.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:08.603+0000 7f5336a00700 1 -- 172.21.15.28:0/1346655039 <== mgr.14120 v2:172.21.15.28:6800/2222038555 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f5320002630 con 0x7f532403cf40 2024-04-23T14:33:09.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:08.604+0000 7f533f200700 1 -- 172.21.15.28:0/1346655039 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f532403cf40 msgr2=0x7f532403f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:09.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:08.604+0000 7f533f200700 1 --2- 172.21.15.28:0/1346655039 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f532403cf40 0x7f532403f330 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f532c003a10 tx=0x7f532c0092b0 comp rx=0 tx=0).stop 2024-04-23T14:33:09.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:08.604+0000 7f533f200700 1 -- 172.21.15.28:0/1346655039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53401074b0 msgr2=0x7f534019e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:09.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:08.604+0000 7f533f200700 1 --2- 172.21.15.28:0/1346655039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53401074b0 0x7f534019e700 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f533000b340 tx=0x7f533000dd90 comp rx=0 tx=0).stop 2024-04-23T14:33:09.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:08.604+0000 7f533f200700 1 -- 172.21.15.28:0/1346655039 shutdown_connections 2024-04-23T14:33:09.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:08.604+0000 7f533f200700 1 --2- 172.21.15.28:0/1346655039 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f532403cf40 0x7f532403f330 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:09.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:08.604+0000 7f533f200700 1 --2- 172.21.15.28:0/1346655039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53401074b0 0x7f534019e700 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:09.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:08.604+0000 7f533f200700 1 --2- 172.21.15.28:0/1346655039 >> [v2:172.21.15.28:6800/1307025091,v1:172.21.15.28:6801/1307025091] conn(0x7f5324039600 0x7f532403bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:09.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:08.604+0000 7f533f200700 1 -- 172.21.15.28:0/1346655039 >> 172.21.15.28:0/1346655039 conn(0x7f5340102a70 msgr2=0x7f5340194d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:09.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:08.604+0000 7f533f200700 1 -- 172.21.15.28:0/1346655039 shutdown_connections 2024-04-23T14:33:09.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:08.604+0000 7f533f200700 1 -- 172.21.15.28:0/1346655039 wait complete. 2024-04-23T14:33:09.354 INFO:teuthology.orchestra.run.smithi028.stdout:mgr epoch 5 is available 2024-04-23T14:33:09.354 INFO:teuthology.orchestra.run.smithi028.stdout:Setting orchestrator backend to cephadm... 2024-04-23T14:33:09.893 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:09 smithi028 ceph-mon[27810]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-23T14:33:09.893 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:09 smithi028 ceph-mon[27810]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-23T14:33:09.893 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:09 smithi028 ceph-mon[27810]: [23/Apr/2024:14:33:09] ENGINE Bus STARTING 2024-04-23T14:33:09.893 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:09 smithi028 ceph-mon[27810]: [23/Apr/2024:14:33:09] ENGINE Serving on https://172.21.15.28:7150 2024-04-23T14:33:09.893 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:09 smithi028 ceph-mon[27810]: [23/Apr/2024:14:33:09] ENGINE Serving on http://172.21.15.28:8765 2024-04-23T14:33:09.893 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:09 smithi028 ceph-mon[27810]: [23/Apr/2024:14:33:09] ENGINE Bus STARTED 2024-04-23T14:33:09.894 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:09 smithi028 ceph-mon[27810]: from='mgr.14120 172.21.15.28:0/2789735782' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:33:10.830 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:10 smithi028 ceph-mon[27810]: mgrmap e8: x(active, since 2s) 2024-04-23T14:33:10.830 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:10 smithi028 ceph-mon[27810]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:33:10.830 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:10 smithi028 ceph-mon[27810]: from='mgr.14120 172.21.15.28:0/2789735782' entity='mgr.x' 2024-04-23T14:33:10.830 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:10 smithi028 ceph-mon[27810]: from='mgr.14120 172.21.15.28:0/2789735782' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:33:10.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.975+0000 7fdb60c00700 1 Processor -- start 2024-04-23T14:33:10.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.975+0000 7fdb60c00700 1 -- start start 2024-04-23T14:33:10.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.976+0000 7fdb60c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb5c107060 0x7fdb5c107480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:10.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.976+0000 7fdb60c00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdb5c07be00 con 0x7fdb5c107060 2024-04-23T14:33:10.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.976+0000 7fdb59e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb5c107060 0x7fdb5c107480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:10.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.976+0000 7fdb59e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb5c107060 0x7fdb5c107480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44736/0 (socket says 172.21.15.28:44736) 2024-04-23T14:33:10.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.976+0000 7fdb59e00700 1 -- 172.21.15.28:0/3225627433 learned_addr learned my addr 172.21.15.28:0/3225627433 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:10.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.977+0000 7fdb59e00700 1 -- 172.21.15.28:0/3225627433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb5c1079c0 con 0x7fdb5c107060 2024-04-23T14:33:10.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.977+0000 7fdb59e00700 1 --2- 172.21.15.28:0/3225627433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb5c107060 0x7fdb5c107480 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c0098a0 tx=0x7fdb4c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eba6502eb91ebccb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:10.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.977+0000 7fdb58a00700 1 -- 172.21.15.28:0/3225627433 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb4c004160 con 0x7fdb5c107060 2024-04-23T14:33:10.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.977+0000 7fdb58a00700 1 -- 172.21.15.28:0/3225627433 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fdb4c004300 con 0x7fdb5c107060 2024-04-23T14:33:10.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.977+0000 7fdb58a00700 1 -- 172.21.15.28:0/3225627433 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb4c011b50 con 0x7fdb5c107060 2024-04-23T14:33:10.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.978+0000 7fdb60c00700 1 -- 172.21.15.28:0/3225627433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb5c107060 msgr2=0x7fdb5c107480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:10.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.978+0000 7fdb60c00700 1 --2- 172.21.15.28:0/3225627433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb5c107060 0x7fdb5c107480 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c0098a0 tx=0x7fdb4c009bb0 comp rx=0 tx=0).stop 2024-04-23T14:33:10.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.978+0000 7fdb60c00700 1 -- 172.21.15.28:0/3225627433 shutdown_connections 2024-04-23T14:33:10.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.978+0000 7fdb60c00700 1 --2- 172.21.15.28:0/3225627433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb5c107060 0x7fdb5c107480 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:10.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.978+0000 7fdb60c00700 1 -- 172.21.15.28:0/3225627433 >> 172.21.15.28:0/3225627433 conn(0x7fdb5c102c60 msgr2=0x7fdb5c105080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:10.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.978+0000 7fdb60c00700 1 -- 172.21.15.28:0/3225627433 shutdown_connections 2024-04-23T14:33:10.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.978+0000 7fdb60c00700 1 -- 172.21.15.28:0/3225627433 wait complete. 2024-04-23T14:33:10.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.979+0000 7fdb60c00700 1 Processor -- start 2024-04-23T14:33:10.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.980+0000 7fdb60c00700 1 -- start start 2024-04-23T14:33:10.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.980+0000 7fdb60c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb5c107060 0x7fdb5c19e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:10.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.980+0000 7fdb60c00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdb5c1089e0 con 0x7fdb5c107060 2024-04-23T14:33:10.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.980+0000 7fdb59e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb5c107060 0x7fdb5c19e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:10.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.980+0000 7fdb59e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb5c107060 0x7fdb5c19e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44752/0 (socket says 172.21.15.28:44752) 2024-04-23T14:33:10.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.980+0000 7fdb59e00700 1 -- 172.21.15.28:0/3229526998 learned_addr learned my addr 172.21.15.28:0/3229526998 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:10.853 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.981+0000 7fdb59e00700 1 -- 172.21.15.28:0/3229526998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb5c19edc0 con 0x7fdb5c107060 2024-04-23T14:33:10.853 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.981+0000 7fdb59e00700 1 --2- 172.21.15.28:0/3229526998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb5c107060 0x7fdb5c19e880 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c011b60 tx=0x7fdb4c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:10.853 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.981+0000 7fdb52a00700 1 -- 172.21.15.28:0/3229526998 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb4c003970 con 0x7fdb5c107060 2024-04-23T14:33:10.853 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.981+0000 7fdb52a00700 1 -- 172.21.15.28:0/3229526998 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fdb4c003b10 con 0x7fdb5c107060 2024-04-23T14:33:10.853 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.981+0000 7fdb60c00700 1 -- 172.21.15.28:0/3229526998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb5c19f050 con 0x7fdb5c107060 2024-04-23T14:33:10.853 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.981+0000 7fdb52a00700 1 -- 172.21.15.28:0/3229526998 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb4c024610 con 0x7fdb5c107060 2024-04-23T14:33:10.853 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.981+0000 7fdb60c00700 1 -- 172.21.15.28:0/3229526998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb5c1a1d40 con 0x7fdb5c107060 2024-04-23T14:33:10.853 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.982+0000 7fdb52a00700 1 -- 172.21.15.28:0/3229526998 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fdb4c01a9f0 con 0x7fdb5c107060 2024-04-23T14:33:10.853 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.982+0000 7fdb60c00700 1 -- 172.21.15.28:0/3229526998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb5c064150 con 0x7fdb5c107060 2024-04-23T14:33:10.853 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.983+0000 7fdb52a00700 1 --2- 172.21.15.28:0/3229526998 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7fdb40039600 0x7fdb4003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:10.853 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.983+0000 7fdb52a00700 1 -- 172.21.15.28:0/3229526998 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fdb4c011e00 con 0x7fdb5c107060 2024-04-23T14:33:10.853 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.983+0000 7fdb59400700 1 --2- 172.21.15.28:0/3229526998 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7fdb40039600 0x7fdb4003bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:10.853 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.984+0000 7fdb59400700 1 --2- 172.21.15.28:0/3229526998 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7fdb40039600 0x7fdb4003bac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdb48006fd0 tx=0x7fdb48006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:10.853 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:09.989+0000 7fdb52a00700 1 -- 172.21.15.28:0/3229526998 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdb4c016260 con 0x7fdb5c107060 2024-04-23T14:33:10.853 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:10.237+0000 7fdb60c00700 1 -- 172.21.15.28:0/3229526998 --> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fdb5c02e2c0 con 0x7fdb40039600 2024-04-23T14:33:10.854 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:10.246+0000 7fdb52a00700 1 -- 172.21.15.28:0/3229526998 <== mgr.14120 v2:172.21.15.28:6800/2222038555 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fdb5c02e2c0 con 0x7fdb40039600 2024-04-23T14:33:10.854 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:10.250+0000 7fdb60c00700 1 -- 172.21.15.28:0/3229526998 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7fdb40039600 msgr2=0x7fdb4003bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:10.854 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:10.250+0000 7fdb60c00700 1 --2- 172.21.15.28:0/3229526998 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7fdb40039600 0x7fdb4003bac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdb48006fd0 tx=0x7fdb48006e40 comp rx=0 tx=0).stop 2024-04-23T14:33:10.854 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:10.250+0000 7fdb60c00700 1 -- 172.21.15.28:0/3229526998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb5c107060 msgr2=0x7fdb5c19e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:10.854 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:10.250+0000 7fdb60c00700 1 --2- 172.21.15.28:0/3229526998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb5c107060 0x7fdb5c19e880 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c011b60 tx=0x7fdb4c011c40 comp rx=0 tx=0).stop 2024-04-23T14:33:10.854 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:10.250+0000 7fdb60c00700 1 -- 172.21.15.28:0/3229526998 shutdown_connections 2024-04-23T14:33:10.854 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:10.250+0000 7fdb60c00700 1 --2- 172.21.15.28:0/3229526998 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7fdb40039600 0x7fdb4003bac0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:10.854 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:10.250+0000 7fdb60c00700 1 --2- 172.21.15.28:0/3229526998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb5c107060 0x7fdb5c19e880 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:10.854 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:10.250+0000 7fdb60c00700 1 -- 172.21.15.28:0/3229526998 >> 172.21.15.28:0/3229526998 conn(0x7fdb5c102c60 msgr2=0x7fdb5c195520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:10.854 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:10.251+0000 7fdb60c00700 1 -- 172.21.15.28:0/3229526998 shutdown_connections 2024-04-23T14:33:10.854 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:10.251+0000 7fdb60c00700 1 -- 172.21.15.28:0/3229526998 wait complete. 2024-04-23T14:33:12.486 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-23T14:33:12.486 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.454+0000 7f230be00700 1 Processor -- start 2024-04-23T14:33:12.486 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.454+0000 7f230be00700 1 -- start start 2024-04-23T14:33:12.486 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.455+0000 7f230be00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230c1090c0 0x7f230c10b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:12.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.455+0000 7f230be00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f230c07be00 con 0x7f230c1090c0 2024-04-23T14:33:12.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.456+0000 7f230aa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230c1090c0 0x7f230c10b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:12.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.456+0000 7f230aa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230c1090c0 0x7f230c10b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44762/0 (socket says 172.21.15.28:44762) 2024-04-23T14:33:12.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.456+0000 7f230aa00700 1 -- 172.21.15.28:0/217527745 learned_addr learned my addr 172.21.15.28:0/217527745 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:12.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.456+0000 7f230aa00700 1 -- 172.21.15.28:0/217527745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f230c10b9f0 con 0x7f230c1090c0 2024-04-23T14:33:12.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.456+0000 7f230aa00700 1 --2- 172.21.15.28:0/217527745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230c1090c0 0x7f230c10b4b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f22fc0098a0 tx=0x7f22fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce8ab5c0af1d6f7e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:12.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.457+0000 7f2309600700 1 -- 172.21.15.28:0/217527745 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f22fc004160 con 0x7f230c1090c0 2024-04-23T14:33:12.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.457+0000 7f2309600700 1 -- 172.21.15.28:0/217527745 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f22fc004300 con 0x7f230c1090c0 2024-04-23T14:33:12.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.457+0000 7f2309600700 1 -- 172.21.15.28:0/217527745 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f22fc011b50 con 0x7f230c1090c0 2024-04-23T14:33:12.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.458+0000 7f230be00700 1 -- 172.21.15.28:0/217527745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230c1090c0 msgr2=0x7f230c10b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:12.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.458+0000 7f230be00700 1 --2- 172.21.15.28:0/217527745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230c1090c0 0x7f230c10b4b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f22fc0098a0 tx=0x7f22fc009bb0 comp rx=0 tx=0).stop 2024-04-23T14:33:12.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.458+0000 7f230be00700 1 -- 172.21.15.28:0/217527745 shutdown_connections 2024-04-23T14:33:12.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.458+0000 7f230be00700 1 --2- 172.21.15.28:0/217527745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230c1090c0 0x7f230c10b4b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:12.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.458+0000 7f230be00700 1 -- 172.21.15.28:0/217527745 >> 172.21.15.28:0/217527745 conn(0x7f230c102a70 msgr2=0x7f230c104e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:12.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.458+0000 7f230be00700 1 -- 172.21.15.28:0/217527745 shutdown_connections 2024-04-23T14:33:12.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.458+0000 7f230be00700 1 -- 172.21.15.28:0/217527745 wait complete. 2024-04-23T14:33:12.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.459+0000 7f230be00700 1 Processor -- start 2024-04-23T14:33:12.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.460+0000 7f230be00700 1 -- start start 2024-04-23T14:33:12.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.460+0000 7f230be00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230c1090c0 0x7f230c108940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:12.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.460+0000 7f230be00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f230c10c8d0 con 0x7f230c1090c0 2024-04-23T14:33:12.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.460+0000 7f230aa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230c1090c0 0x7f230c108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:12.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.460+0000 7f230aa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230c1090c0 0x7f230c108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44776/0 (socket says 172.21.15.28:44776) 2024-04-23T14:33:12.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.460+0000 7f230aa00700 1 -- 172.21.15.28:0/3647746507 learned_addr learned my addr 172.21.15.28:0/3647746507 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:12.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.461+0000 7f230aa00700 1 -- 172.21.15.28:0/3647746507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f230c108e80 con 0x7f230c1090c0 2024-04-23T14:33:12.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.461+0000 7f230aa00700 1 --2- 172.21.15.28:0/3647746507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230c1090c0 0x7f230c108940 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f22fc011b60 tx=0x7f22fc011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:12.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.461+0000 7f2303400700 1 -- 172.21.15.28:0/3647746507 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f22fc003970 con 0x7f230c1090c0 2024-04-23T14:33:12.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.461+0000 7f2303400700 1 -- 172.21.15.28:0/3647746507 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f22fc003b10 con 0x7f230c1090c0 2024-04-23T14:33:12.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.461+0000 7f2303400700 1 -- 172.21.15.28:0/3647746507 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f22fc024610 con 0x7f230c1090c0 2024-04-23T14:33:12.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.461+0000 7f230be00700 1 -- 172.21.15.28:0/3647746507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f230c107070 con 0x7f230c1090c0 2024-04-23T14:33:12.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.462+0000 7f230be00700 1 -- 172.21.15.28:0/3647746507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f230c1075b0 con 0x7f230c1090c0 2024-04-23T14:33:12.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.463+0000 7f2303400700 1 -- 172.21.15.28:0/3647746507 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f22fc01a9f0 con 0x7f230c1090c0 2024-04-23T14:33:12.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.463+0000 7f230be00700 1 -- 172.21.15.28:0/3647746507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f230c064150 con 0x7f230c1090c0 2024-04-23T14:33:12.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.464+0000 7f2303400700 1 --2- 172.21.15.28:0/3647746507 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f22f0039600 0x7f22f003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:12.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.464+0000 7f2303400700 1 -- 172.21.15.28:0/3647746507 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f22fc011e00 con 0x7f230c1090c0 2024-04-23T14:33:12.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.464+0000 7f230a000700 1 --2- 172.21.15.28:0/3647746507 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f22f0039600 0x7f22f003bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:12.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.465+0000 7f230a000700 1 --2- 172.21.15.28:0/3647746507 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f22f0039600 0x7f22f003bac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f22f8006fd0 tx=0x7f22f8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:12.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.469+0000 7f2303400700 1 -- 172.21.15.28:0/3647746507 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f22fc016260 con 0x7f230c1090c0 2024-04-23T14:33:12.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.697+0000 7f230be00700 1 -- 172.21.15.28:0/3647746507 --> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f230c103d10 con 0x7f22f0039600 2024-04-23T14:33:12.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.698+0000 7f2303400700 1 -- 172.21.15.28:0/3647746507 <== mgr.14120 v2:172.21.15.28:6800/2222038555 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f230c103d10 con 0x7f22f0039600 2024-04-23T14:33:12.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.701+0000 7f230be00700 1 -- 172.21.15.28:0/3647746507 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f22f0039600 msgr2=0x7f22f003bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:12.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.701+0000 7f230be00700 1 --2- 172.21.15.28:0/3647746507 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f22f0039600 0x7f22f003bac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f22f8006fd0 tx=0x7f22f8006e40 comp rx=0 tx=0).stop 2024-04-23T14:33:12.490 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.701+0000 7f230be00700 1 -- 172.21.15.28:0/3647746507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230c1090c0 msgr2=0x7f230c108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:12.490 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.701+0000 7f230be00700 1 --2- 172.21.15.28:0/3647746507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230c1090c0 0x7f230c108940 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f22fc011b60 tx=0x7f22fc011c40 comp rx=0 tx=0).stop 2024-04-23T14:33:12.490 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.701+0000 7f230be00700 1 -- 172.21.15.28:0/3647746507 shutdown_connections 2024-04-23T14:33:12.490 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.701+0000 7f230be00700 1 --2- 172.21.15.28:0/3647746507 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f22f0039600 0x7f22f003bac0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:12.490 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.701+0000 7f230be00700 1 --2- 172.21.15.28:0/3647746507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230c1090c0 0x7f230c108940 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:12.490 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.701+0000 7f230be00700 1 -- 172.21.15.28:0/3647746507 >> 172.21.15.28:0/3647746507 conn(0x7f230c102a70 msgr2=0x7f230c103600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:12.490 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.701+0000 7f230be00700 1 -- 172.21.15.28:0/3647746507 shutdown_connections 2024-04-23T14:33:12.490 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:11.701+0000 7f230be00700 1 -- 172.21.15.28:0/3647746507 wait complete. 2024-04-23T14:33:12.490 INFO:teuthology.orchestra.run.smithi028.stdout:Generating ssh key... 2024-04-23T14:33:12.795 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:12 smithi028 ceph-mon[27810]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:33:13.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:13 smithi028 ceph-mon[27810]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:33:13.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:13 smithi028 ceph-mon[27810]: Generating ssh key... 2024-04-23T14:33:14.321 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:13 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: Generating public/private rsa key pair. 2024-04-23T14:33:14.321 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:13 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: Your identification has been saved in /tmp/tmpzop8k9uo/key. 2024-04-23T14:33:14.321 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:13 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: Your public key has been saved in /tmp/tmpzop8k9uo/key.pub. 2024-04-23T14:33:14.321 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:13 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: The key fingerprint is: 2024-04-23T14:33:14.322 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:13 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: SHA256:tulouPt6gR4JC6/oQe8b5DfaOogAovMS878fgaLxQRU ceph-1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T14:33:14.322 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:13 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: The key's randomart image is: 2024-04-23T14:33:14.322 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:13 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: +---[RSA 3072]----+ 2024-04-23T14:33:14.322 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:13 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: | E. | 2024-04-23T14:33:14.322 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:13 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: | . | 2024-04-23T14:33:14.322 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:13 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: | . | 2024-04-23T14:33:14.322 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:13 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: |+.. . | 2024-04-23T14:33:14.322 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:13 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: |=+o+.o. S | 2024-04-23T14:33:14.322 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:13 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: |B+*o+ .o o | 2024-04-23T14:33:14.322 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:13 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: |=Oo= =..o | 2024-04-23T14:33:14.322 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:13 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: |=.=.B ++ | 2024-04-23T14:33:14.322 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:13 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: |.o *X@+ . | 2024-04-23T14:33:14.322 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:13 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: +----[SHA256]-----+ 2024-04-23T14:33:14.676 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.089+0000 7f0bf3200700 1 Processor -- start 2024-04-23T14:33:14.677 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.089+0000 7f0bf3200700 1 -- start start 2024-04-23T14:33:14.677 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.089+0000 7f0bf3200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bf410f050 0x7f0bf410f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:14.677 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.089+0000 7f0bf3200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf410fa40 con 0x7f0bf410f050 2024-04-23T14:33:14.677 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.090+0000 7f0bf1e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bf410f050 0x7f0bf410f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:14.678 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.090+0000 7f0bf1e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bf410f050 0x7f0bf410f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44792/0 (socket says 172.21.15.28:44792) 2024-04-23T14:33:14.678 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.090+0000 7f0bf1e00700 1 -- 172.21.15.28:0/3246189866 learned_addr learned my addr 172.21.15.28:0/3246189866 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:14.678 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.090+0000 7f0bf1e00700 1 -- 172.21.15.28:0/3246189866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bf410fd70 con 0x7f0bf410f050 2024-04-23T14:33:14.678 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.091+0000 7f0bf1e00700 1 --2- 172.21.15.28:0/3246189866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bf410f050 0x7f0bf410f470 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0be40098a0 tx=0x7f0be4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb0f27a733afae85 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:14.678 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.091+0000 7f0bf0a00700 1 -- 172.21.15.28:0/3246189866 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0be4004160 con 0x7f0bf410f050 2024-04-23T14:33:14.679 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.091+0000 7f0bf0a00700 1 -- 172.21.15.28:0/3246189866 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0be4004300 con 0x7f0bf410f050 2024-04-23T14:33:14.679 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.091+0000 7f0bf0a00700 1 -- 172.21.15.28:0/3246189866 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0be4011b50 con 0x7f0bf410f050 2024-04-23T14:33:14.679 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.092+0000 7f0bf3200700 1 -- 172.21.15.28:0/3246189866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bf410f050 msgr2=0x7f0bf410f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:14.680 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.092+0000 7f0bf3200700 1 --2- 172.21.15.28:0/3246189866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bf410f050 0x7f0bf410f470 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0be40098a0 tx=0x7f0be4009bb0 comp rx=0 tx=0).stop 2024-04-23T14:33:14.680 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.092+0000 7f0bf3200700 1 -- 172.21.15.28:0/3246189866 shutdown_connections 2024-04-23T14:33:14.680 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.092+0000 7f0bf3200700 1 --2- 172.21.15.28:0/3246189866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bf410f050 0x7f0bf410f470 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:14.680 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.092+0000 7f0bf3200700 1 -- 172.21.15.28:0/3246189866 >> 172.21.15.28:0/3246189866 conn(0x7f0bf407de00 msgr2=0x7f0bf4080240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:14.680 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.092+0000 7f0bf3200700 1 -- 172.21.15.28:0/3246189866 shutdown_connections 2024-04-23T14:33:14.680 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.092+0000 7f0bf3200700 1 -- 172.21.15.28:0/3246189866 wait complete. 2024-04-23T14:33:14.681 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.093+0000 7f0bf3200700 1 Processor -- start 2024-04-23T14:33:14.681 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.094+0000 7f0bf3200700 1 -- start start 2024-04-23T14:33:14.681 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.094+0000 7f0bf3200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bf410f050 0x7f0bf41a2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:14.682 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.094+0000 7f0bf3200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf4113120 con 0x7f0bf410f050 2024-04-23T14:33:14.682 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.094+0000 7f0bf1e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bf410f050 0x7f0bf41a2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:14.682 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.094+0000 7f0bf1e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bf410f050 0x7f0bf41a2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44796/0 (socket says 172.21.15.28:44796) 2024-04-23T14:33:14.682 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.094+0000 7f0bf1e00700 1 -- 172.21.15.28:0/2729670569 learned_addr learned my addr 172.21.15.28:0/2729670569 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:14.683 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.095+0000 7f0bf1e00700 1 -- 172.21.15.28:0/2729670569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bf41a3070 con 0x7f0bf410f050 2024-04-23T14:33:14.683 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.095+0000 7f0bf1e00700 1 --2- 172.21.15.28:0/2729670569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bf410f050 0x7f0bf41a2b30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0be4011850 tx=0x7f0be4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:14.683 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.095+0000 7f0beaa00700 1 -- 172.21.15.28:0/2729670569 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0be4003af0 con 0x7f0bf410f050 2024-04-23T14:33:14.683 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.095+0000 7f0beaa00700 1 -- 172.21.15.28:0/2729670569 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0be4020070 con 0x7f0bf410f050 2024-04-23T14:33:14.684 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.095+0000 7f0beaa00700 1 -- 172.21.15.28:0/2729670569 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0be40195a0 con 0x7f0bf410f050 2024-04-23T14:33:14.684 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.095+0000 7f0bf3200700 1 -- 172.21.15.28:0/2729670569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bf41a3300 con 0x7f0bf410f050 2024-04-23T14:33:14.684 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.095+0000 7f0bf3200700 1 -- 172.21.15.28:0/2729670569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bf41a5ff0 con 0x7f0bf410f050 2024-04-23T14:33:14.684 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.096+0000 7f0beaa00700 1 -- 172.21.15.28:0/2729670569 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f0be4019850 con 0x7f0bf410f050 2024-04-23T14:33:14.685 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.097+0000 7f0bf3200700 1 -- 172.21.15.28:0/2729670569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bf4064150 con 0x7f0bf410f050 2024-04-23T14:33:14.685 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.097+0000 7f0beaa00700 1 --2- 172.21.15.28:0/2729670569 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f0bd8039600 0x7f0bd803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:14.685 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.097+0000 7f0beaa00700 1 -- 172.21.15.28:0/2729670569 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0be4011e00 con 0x7f0bf410f050 2024-04-23T14:33:14.685 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.097+0000 7f0bf1400700 1 --2- 172.21.15.28:0/2729670569 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f0bd8039600 0x7f0bd803bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:14.685 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.098+0000 7f0bf1400700 1 --2- 172.21.15.28:0/2729670569 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f0bd8039600 0x7f0bd803bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0be0006fd0 tx=0x7f0be0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:14.686 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.103+0000 7f0beaa00700 1 -- 172.21.15.28:0/2729670569 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0be400e260 con 0x7f0bf410f050 2024-04-23T14:33:14.686 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.338+0000 7f0bf3200700 1 -- 172.21.15.28:0/2729670569 --> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f0bf40800c0 con 0x7f0bd8039600 2024-04-23T14:33:14.686 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.971+0000 7f0beaa00700 1 -- 172.21.15.28:0/2729670569 <== mgr.14120 v2:172.21.15.28:6800/2222038555 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f0bf40800c0 con 0x7f0bd8039600 2024-04-23T14:33:14.686 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.974+0000 7f0bf3200700 1 -- 172.21.15.28:0/2729670569 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f0bd8039600 msgr2=0x7f0bd803bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:14.687 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.974+0000 7f0bf3200700 1 --2- 172.21.15.28:0/2729670569 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f0bd8039600 0x7f0bd803bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0be0006fd0 tx=0x7f0be0006e40 comp rx=0 tx=0).stop 2024-04-23T14:33:14.687 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.974+0000 7f0bf3200700 1 -- 172.21.15.28:0/2729670569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bf410f050 msgr2=0x7f0bf41a2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:14.687 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.974+0000 7f0bf3200700 1 --2- 172.21.15.28:0/2729670569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bf410f050 0x7f0bf41a2b30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0be4011850 tx=0x7f0be4011c10 comp rx=0 tx=0).stop 2024-04-23T14:33:14.687 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.974+0000 7f0bf3200700 1 -- 172.21.15.28:0/2729670569 shutdown_connections 2024-04-23T14:33:14.687 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.974+0000 7f0bf3200700 1 --2- 172.21.15.28:0/2729670569 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f0bd8039600 0x7f0bd803bac0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:14.688 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.974+0000 7f0bf3200700 1 --2- 172.21.15.28:0/2729670569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bf410f050 0x7f0bf41a2b30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:14.688 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.974+0000 7f0bf3200700 1 -- 172.21.15.28:0/2729670569 >> 172.21.15.28:0/2729670569 conn(0x7f0bf407de00 msgr2=0x7f0bf407f9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:14.688 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.975+0000 7f0bf3200700 1 -- 172.21.15.28:0/2729670569 shutdown_connections 2024-04-23T14:33:14.688 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:13.975+0000 7f0bf3200700 1 -- 172.21.15.28:0/2729670569 wait complete. 2024-04-23T14:33:15.235 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:14 smithi028 ceph-mon[27810]: from='mgr.14120 172.21.15.28:0/2789735782' entity='mgr.x' 2024-04-23T14:33:15.236 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:14 smithi028 ceph-mon[27810]: from='mgr.14120 172.21.15.28:0/2789735782' entity='mgr.x' 2024-04-23T14:33:16.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:15 smithi028 ceph-mon[27810]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:33:16.174 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDePNESHCvfO7qfiUFXQ40tSG3i5sLt0ezAou6+Is/yrbWk91HXm+bgctGhOG+XING3iFd/2T5Jr9HNELbEN+Ci3YMVeqXgHavNh7wmBANjWF8DiRdHwJlwzcUXg9cWOPIFMSpyLNZ3y2Q11jyIt8rvUkPt5vox5mIRx5rf1cf9ldebaXvx+IjRFihaEjGiWQljwXNsEql5HXB2LKh06BXKzMclxg+g8Lf0tEKhIjaGcTJTGx7VbIzifano3jw3vrYEkkqjsOZfaFwCTppkQWNuGhhQTv9T5zTM4iHmQfDfsXKarMIhXilrD2YZGwDrkHb97Zj/xcbnUQUqWjXUYNXm/MR5SNx++wXVRjhiFT78oH2ENfA1TFfz/b+KT1Wb9NKdD9kZ+DuaV4DUGwCqm2U7+TivHfpADPjnqTTwRvXJm6AV3LhdT/jTKk0m8pOGzOP0YJKHYdu9aQOu61bvbqPF1bpQ01DcJMaCwHkdAJhsCmvdN501URzeBb59AAC0uM= ceph-1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T14:33:16.175 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.313+0000 7f553a200700 1 Processor -- start 2024-04-23T14:33:16.175 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.313+0000 7f553a200700 1 -- start start 2024-04-23T14:33:16.175 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.313+0000 7f553a200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5534108750 0x7f5534108b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:16.175 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.313+0000 7f553a200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5534109140 con 0x7f5534108750 2024-04-23T14:33:16.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.314+0000 7f5533400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5534108750 0x7f5534108b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:16.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.314+0000 7f5533400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5534108750 0x7f5534108b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44800/0 (socket says 172.21.15.28:44800) 2024-04-23T14:33:16.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.314+0000 7f5533400700 1 -- 172.21.15.28:0/3299598486 learned_addr learned my addr 172.21.15.28:0/3299598486 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:16.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.314+0000 7f5533400700 1 -- 172.21.15.28:0/3299598486 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5534109900 con 0x7f5534108750 2024-04-23T14:33:16.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.315+0000 7f5533400700 1 --2- 172.21.15.28:0/3299598486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5534108750 0x7f5534108b70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f55240098a0 tx=0x7f5524009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbb85851b87e9d83 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:16.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.315+0000 7f5532000700 1 -- 172.21.15.28:0/3299598486 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5524004160 con 0x7f5534108750 2024-04-23T14:33:16.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.315+0000 7f5532000700 1 -- 172.21.15.28:0/3299598486 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5524004300 con 0x7f5534108750 2024-04-23T14:33:16.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.315+0000 7f5532000700 1 -- 172.21.15.28:0/3299598486 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5524011b50 con 0x7f5534108750 2024-04-23T14:33:16.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.316+0000 7f553a200700 1 -- 172.21.15.28:0/3299598486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5534108750 msgr2=0x7f5534108b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:16.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.316+0000 7f553a200700 1 --2- 172.21.15.28:0/3299598486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5534108750 0x7f5534108b70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f55240098a0 tx=0x7f5524009bb0 comp rx=0 tx=0).stop 2024-04-23T14:33:16.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.316+0000 7f553a200700 1 -- 172.21.15.28:0/3299598486 shutdown_connections 2024-04-23T14:33:16.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.316+0000 7f553a200700 1 --2- 172.21.15.28:0/3299598486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5534108750 0x7f5534108b70 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:16.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.316+0000 7f553a200700 1 -- 172.21.15.28:0/3299598486 >> 172.21.15.28:0/3299598486 conn(0x7f5534103d70 msgr2=0x7f5534106190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:16.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.316+0000 7f553a200700 1 -- 172.21.15.28:0/3299598486 shutdown_connections 2024-04-23T14:33:16.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.316+0000 7f553a200700 1 -- 172.21.15.28:0/3299598486 wait complete. 2024-04-23T14:33:16.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.317+0000 7f553a200700 1 Processor -- start 2024-04-23T14:33:16.180 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.317+0000 7f553a200700 1 -- start start 2024-04-23T14:33:16.181 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.317+0000 7f553a200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5534108750 0x7f553419c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:16.181 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.317+0000 7f553a200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f553410ccb0 con 0x7f5534108750 2024-04-23T14:33:16.181 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.318+0000 7f5533400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5534108750 0x7f553419c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:16.181 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.318+0000 7f5533400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5534108750 0x7f553419c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44816/0 (socket says 172.21.15.28:44816) 2024-04-23T14:33:16.181 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.318+0000 7f5533400700 1 -- 172.21.15.28:0/2780208834 learned_addr learned my addr 172.21.15.28:0/2780208834 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:16.181 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.318+0000 7f5533400700 1 -- 172.21.15.28:0/2780208834 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f553419cbb0 con 0x7f5534108750 2024-04-23T14:33:16.181 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.318+0000 7f5533400700 1 --2- 172.21.15.28:0/2780208834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5534108750 0x7f553419c670 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5524000c00 tx=0x7f5524011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:16.181 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.319+0000 7f552be00700 1 -- 172.21.15.28:0/2780208834 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5524003b20 con 0x7f5534108750 2024-04-23T14:33:16.181 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.319+0000 7f552be00700 1 -- 172.21.15.28:0/2780208834 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5524003cc0 con 0x7f5534108750 2024-04-23T14:33:16.181 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.319+0000 7f552be00700 1 -- 172.21.15.28:0/2780208834 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5524024910 con 0x7f5534108750 2024-04-23T14:33:16.181 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.319+0000 7f553a200700 1 -- 172.21.15.28:0/2780208834 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f553419ce40 con 0x7f5534108750 2024-04-23T14:33:16.181 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.319+0000 7f553a200700 1 -- 172.21.15.28:0/2780208834 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f553419fb30 con 0x7f5534108750 2024-04-23T14:33:16.181 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.320+0000 7f552be00700 1 -- 172.21.15.28:0/2780208834 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f5524022020 con 0x7f5534108750 2024-04-23T14:33:16.181 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.320+0000 7f553a200700 1 -- 172.21.15.28:0/2780208834 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5534064150 con 0x7f5534108750 2024-04-23T14:33:16.182 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.321+0000 7f552be00700 1 --2- 172.21.15.28:0/2780208834 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f5518039600 0x7f551803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:16.182 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.321+0000 7f552be00700 1 -- 172.21.15.28:0/2780208834 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f5524011e00 con 0x7f5534108750 2024-04-23T14:33:16.182 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.321+0000 7f5532a00700 1 --2- 172.21.15.28:0/2780208834 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f5518039600 0x7f551803bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:16.182 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.321+0000 7f5532a00700 1 --2- 172.21.15.28:0/2780208834 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f5518039600 0x7f551803bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5520006fd0 tx=0x7f5520006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:16.182 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.327+0000 7f552be00700 1 -- 172.21.15.28:0/2780208834 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f552401ad80 con 0x7f5534108750 2024-04-23T14:33:16.182 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.559+0000 7f553a200700 1 -- 172.21.15.28:0/2780208834 --> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f5534105030 con 0x7f5518039600 2024-04-23T14:33:16.183 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.560+0000 7f552be00700 1 -- 172.21.15.28:0/2780208834 <== mgr.14120 v2:172.21.15.28:6800/2222038555 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f5534105030 con 0x7f5518039600 2024-04-23T14:33:16.183 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.563+0000 7f553a200700 1 -- 172.21.15.28:0/2780208834 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f5518039600 msgr2=0x7f551803bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:16.184 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.563+0000 7f553a200700 1 --2- 172.21.15.28:0/2780208834 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f5518039600 0x7f551803bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5520006fd0 tx=0x7f5520006e40 comp rx=0 tx=0).stop 2024-04-23T14:33:16.184 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.563+0000 7f553a200700 1 -- 172.21.15.28:0/2780208834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5534108750 msgr2=0x7f553419c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:16.184 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.563+0000 7f553a200700 1 --2- 172.21.15.28:0/2780208834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5534108750 0x7f553419c670 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5524000c00 tx=0x7f5524011c10 comp rx=0 tx=0).stop 2024-04-23T14:33:16.184 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.563+0000 7f553a200700 1 -- 172.21.15.28:0/2780208834 shutdown_connections 2024-04-23T14:33:16.184 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.563+0000 7f553a200700 1 --2- 172.21.15.28:0/2780208834 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f5518039600 0x7f551803bac0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:16.184 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.563+0000 7f553a200700 1 --2- 172.21.15.28:0/2780208834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5534108750 0x7f553419c670 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:16.184 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.563+0000 7f553a200700 1 -- 172.21.15.28:0/2780208834 >> 172.21.15.28:0/2780208834 conn(0x7f5534103d70 msgr2=0x7f5534104920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:16.184 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.564+0000 7f553a200700 1 -- 172.21.15.28:0/2780208834 shutdown_connections 2024-04-23T14:33:16.184 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:15.564+0000 7f553a200700 1 -- 172.21.15.28:0/2780208834 wait complete. 2024-04-23T14:33:16.184 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-23T14:33:16.184 INFO:teuthology.orchestra.run.smithi028.stdout:Adding key to root@localhost authorized_keys... 2024-04-23T14:33:16.188 INFO:teuthology.orchestra.run.smithi028.stdout:Adding host smithi028... 2024-04-23T14:33:17.721 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:17 smithi028 ceph-mon[27810]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi028", "addr": "172.21.15.28", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:33:18.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:18 smithi028 ceph-mon[27810]: Deploying cephadm binary to smithi028 2024-04-23T14:33:24.125 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Added host 'smithi028' with addr '172.21.15.28' 2024-04-23T14:33:24.126 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.790+0000 7f0f99200700 1 Processor -- start 2024-04-23T14:33:24.126 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.790+0000 7f0f99200700 1 -- start start 2024-04-23T14:33:24.131 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.791+0000 7f0f99200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f941092a0 0x7f0f9410b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:24.132 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.791+0000 7f0f99200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0f9407bde0 con 0x7f0f941092a0 2024-04-23T14:33:24.132 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.791+0000 7f0f91e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f941092a0 0x7f0f9410b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:24.132 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.792+0000 7f0f91e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f941092a0 0x7f0f9410b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44820/0 (socket says 172.21.15.28:44820) 2024-04-23T14:33:24.132 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.792+0000 7f0f91e00700 1 -- 172.21.15.28:0/2517168283 learned_addr learned my addr 172.21.15.28:0/2517168283 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:24.132 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.792+0000 7f0f91e00700 1 -- 172.21.15.28:0/2517168283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f9410bbd0 con 0x7f0f941092a0 2024-04-23T14:33:24.132 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.792+0000 7f0f91e00700 1 --2- 172.21.15.28:0/2517168283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f941092a0 0x7f0f9410b690 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0f840098a0 tx=0x7f0f84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c416b2d04c1379c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:24.132 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.793+0000 7f0f90a00700 1 -- 172.21.15.28:0/2517168283 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f84004160 con 0x7f0f941092a0 2024-04-23T14:33:24.132 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.793+0000 7f0f90a00700 1 -- 172.21.15.28:0/2517168283 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0f84004300 con 0x7f0f941092a0 2024-04-23T14:33:24.132 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.793+0000 7f0f90a00700 1 -- 172.21.15.28:0/2517168283 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f84011b50 con 0x7f0f941092a0 2024-04-23T14:33:24.132 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.793+0000 7f0f99200700 1 -- 172.21.15.28:0/2517168283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f941092a0 msgr2=0x7f0f9410b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:24.132 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.793+0000 7f0f99200700 1 --2- 172.21.15.28:0/2517168283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f941092a0 0x7f0f9410b690 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0f840098a0 tx=0x7f0f84009bb0 comp rx=0 tx=0).stop 2024-04-23T14:33:24.132 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.794+0000 7f0f99200700 1 -- 172.21.15.28:0/2517168283 shutdown_connections 2024-04-23T14:33:24.133 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.794+0000 7f0f99200700 1 --2- 172.21.15.28:0/2517168283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f941092a0 0x7f0f9410b690 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:24.133 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.794+0000 7f0f99200700 1 -- 172.21.15.28:0/2517168283 >> 172.21.15.28:0/2517168283 conn(0x7f0f94102cf0 msgr2=0x7f0f94105110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:24.133 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.794+0000 7f0f99200700 1 -- 172.21.15.28:0/2517168283 shutdown_connections 2024-04-23T14:33:24.133 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.794+0000 7f0f99200700 1 -- 172.21.15.28:0/2517168283 wait complete. 2024-04-23T14:33:24.133 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.795+0000 7f0f99200700 1 Processor -- start 2024-04-23T14:33:24.133 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.795+0000 7f0f99200700 1 -- start start 2024-04-23T14:33:24.133 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.795+0000 7f0f99200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f941092a0 0x7f0f94108b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:24.133 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.795+0000 7f0f99200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0f9410cad0 con 0x7f0f941092a0 2024-04-23T14:33:24.133 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.796+0000 7f0f91e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f941092a0 0x7f0f94108b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:24.133 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.796+0000 7f0f91e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f941092a0 0x7f0f94108b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44834/0 (socket says 172.21.15.28:44834) 2024-04-23T14:33:24.133 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.796+0000 7f0f91e00700 1 -- 172.21.15.28:0/394439149 learned_addr learned my addr 172.21.15.28:0/394439149 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:24.133 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.796+0000 7f0f91e00700 1 -- 172.21.15.28:0/394439149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f94109060 con 0x7f0f941092a0 2024-04-23T14:33:24.133 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.796+0000 7f0f91e00700 1 --2- 172.21.15.28:0/394439149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f941092a0 0x7f0f94108b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0f84009fb0 tx=0x7f0f84011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:24.133 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.797+0000 7f0f8aa00700 1 -- 172.21.15.28:0/394439149 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f840038b0 con 0x7f0f941092a0 2024-04-23T14:33:24.134 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.797+0000 7f0f8aa00700 1 -- 172.21.15.28:0/394439149 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0f84011e50 con 0x7f0f941092a0 2024-04-23T14:33:24.134 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.797+0000 7f0f8aa00700 1 -- 172.21.15.28:0/394439149 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f84019420 con 0x7f0f941092a0 2024-04-23T14:33:24.134 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.797+0000 7f0f99200700 1 -- 172.21.15.28:0/394439149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f94107250 con 0x7f0f941092a0 2024-04-23T14:33:24.134 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.797+0000 7f0f99200700 1 -- 172.21.15.28:0/394439149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f94107790 con 0x7f0f941092a0 2024-04-23T14:33:24.134 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.798+0000 7f0f8aa00700 1 -- 172.21.15.28:0/394439149 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f0f84003a50 con 0x7f0f941092a0 2024-04-23T14:33:24.134 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.798+0000 7f0f99200700 1 -- 172.21.15.28:0/394439149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f94064150 con 0x7f0f941092a0 2024-04-23T14:33:24.134 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.799+0000 7f0f8aa00700 1 --2- 172.21.15.28:0/394439149 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f0f780395b0 0x7f0f7803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:24.134 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.799+0000 7f0f8aa00700 1 -- 172.21.15.28:0/394439149 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0f8404d9f0 con 0x7f0f941092a0 2024-04-23T14:33:24.134 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.799+0000 7f0f91400700 1 --2- 172.21.15.28:0/394439149 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f0f780395b0 0x7f0f7803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:24.134 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.800+0000 7f0f91400700 1 --2- 172.21.15.28:0/394439149 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f0f780395b0 0x7f0f7803ba70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0f80006fd0 tx=0x7f0f80006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:24.134 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:16.805+0000 7f0f8aa00700 1 -- 172.21.15.28:0/394439149 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0f8404d340 con 0x7f0f941092a0 2024-04-23T14:33:24.134 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:17.042+0000 7f0f99200700 1 -- 172.21.15.28:0/394439149 --> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi028", "addr": "172.21.15.28", "target": ["mon-mgr", ""]}) -- 0x7f0f94103ef0 con 0x7f0f780395b0 2024-04-23T14:33:24.134 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:23.403+0000 7f0f8aa00700 1 -- 172.21.15.28:0/394439149 <== mgr.14120 v2:172.21.15.28:6800/2222038555 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f0f94103ef0 con 0x7f0f780395b0 2024-04-23T14:33:24.134 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:23.407+0000 7f0f99200700 1 -- 172.21.15.28:0/394439149 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f0f780395b0 msgr2=0x7f0f7803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:24.135 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:23.407+0000 7f0f99200700 1 --2- 172.21.15.28:0/394439149 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f0f780395b0 0x7f0f7803ba70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0f80006fd0 tx=0x7f0f80006e40 comp rx=0 tx=0).stop 2024-04-23T14:33:24.135 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:23.408+0000 7f0f99200700 1 -- 172.21.15.28:0/394439149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f941092a0 msgr2=0x7f0f94108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:24.135 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:23.408+0000 7f0f99200700 1 --2- 172.21.15.28:0/394439149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f941092a0 0x7f0f94108b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0f84009fb0 tx=0x7f0f84011bf0 comp rx=0 tx=0).stop 2024-04-23T14:33:24.135 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:23.408+0000 7f0f99200700 1 -- 172.21.15.28:0/394439149 shutdown_connections 2024-04-23T14:33:24.135 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:23.408+0000 7f0f99200700 1 --2- 172.21.15.28:0/394439149 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f0f780395b0 0x7f0f7803ba70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:24.135 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:23.408+0000 7f0f99200700 1 --2- 172.21.15.28:0/394439149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f941092a0 0x7f0f94108b20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:24.135 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:23.408+0000 7f0f99200700 1 -- 172.21.15.28:0/394439149 >> 172.21.15.28:0/394439149 conn(0x7f0f94102cf0 msgr2=0x7f0f941037e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:24.135 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:23.408+0000 7f0f99200700 1 -- 172.21.15.28:0/394439149 shutdown_connections 2024-04-23T14:33:24.135 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:23.408+0000 7f0f99200700 1 -- 172.21.15.28:0/394439149 wait complete. 2024-04-23T14:33:24.135 INFO:teuthology.orchestra.run.smithi028.stdout:Deploying unmanaged mon service... 2024-04-23T14:33:24.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:24 smithi028 ceph-mon[27810]: from='mgr.14120 172.21.15.28:0/2789735782' entity='mgr.x' 2024-04-23T14:33:24.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:24 smithi028 ceph-mon[27810]: Added host smithi028 2024-04-23T14:33:24.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:24 smithi028 ceph-mon[27810]: from='mgr.14120 172.21.15.28:0/2789735782' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:33:25.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-23T14:33:25.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.816+0000 7fc0ff200700 1 Processor -- start 2024-04-23T14:33:25.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.816+0000 7fc0ff200700 1 -- start start 2024-04-23T14:33:25.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.817+0000 7fc0ff200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0f810f050 0x7fc0f810f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:25.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.817+0000 7fc0ff200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f810fa40 con 0x7fc0f810f050 2024-04-23T14:33:25.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.817+0000 7fc0fde00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0f810f050 0x7fc0f810f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:25.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.817+0000 7fc0fde00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0f810f050 0x7fc0f810f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34920/0 (socket says 172.21.15.28:34920) 2024-04-23T14:33:25.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.817+0000 7fc0fde00700 1 -- 172.21.15.28:0/4048140259 learned_addr learned my addr 172.21.15.28:0/4048140259 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:25.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.818+0000 7fc0fde00700 1 -- 172.21.15.28:0/4048140259 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0f810fd70 con 0x7fc0f810f050 2024-04-23T14:33:25.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.818+0000 7fc0fde00700 1 --2- 172.21.15.28:0/4048140259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0f810f050 0x7fc0f810f470 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec0098a0 tx=0x7fc0ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adf2401e3ed8a249 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:25.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.818+0000 7fc0fca00700 1 -- 172.21.15.28:0/4048140259 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc0ec004160 con 0x7fc0f810f050 2024-04-23T14:33:25.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.818+0000 7fc0fca00700 1 -- 172.21.15.28:0/4048140259 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc0ec004300 con 0x7fc0f810f050 2024-04-23T14:33:25.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.818+0000 7fc0fca00700 1 -- 172.21.15.28:0/4048140259 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc0ec011b50 con 0x7fc0f810f050 2024-04-23T14:33:25.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.819+0000 7fc0ff200700 1 -- 172.21.15.28:0/4048140259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0f810f050 msgr2=0x7fc0f810f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:25.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.819+0000 7fc0ff200700 1 --2- 172.21.15.28:0/4048140259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0f810f050 0x7fc0f810f470 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec0098a0 tx=0x7fc0ec009bb0 comp rx=0 tx=0).stop 2024-04-23T14:33:25.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.819+0000 7fc0ff200700 1 -- 172.21.15.28:0/4048140259 shutdown_connections 2024-04-23T14:33:25.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.819+0000 7fc0ff200700 1 --2- 172.21.15.28:0/4048140259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0f810f050 0x7fc0f810f470 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:25.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.819+0000 7fc0ff200700 1 -- 172.21.15.28:0/4048140259 >> 172.21.15.28:0/4048140259 conn(0x7fc0f807de00 msgr2=0x7fc0f8080240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:25.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.819+0000 7fc0ff200700 1 -- 172.21.15.28:0/4048140259 shutdown_connections 2024-04-23T14:33:25.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.819+0000 7fc0ff200700 1 -- 172.21.15.28:0/4048140259 wait complete. 2024-04-23T14:33:25.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.820+0000 7fc0ff200700 1 Processor -- start 2024-04-23T14:33:25.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.821+0000 7fc0ff200700 1 -- start start 2024-04-23T14:33:25.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.821+0000 7fc0ff200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0f810f050 0x7fc0f81a2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:25.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.821+0000 7fc0ff200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f8113120 con 0x7fc0f810f050 2024-04-23T14:33:25.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.821+0000 7fc0fde00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0f810f050 0x7fc0f81a2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:25.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.821+0000 7fc0fde00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0f810f050 0x7fc0f81a2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34930/0 (socket says 172.21.15.28:34930) 2024-04-23T14:33:25.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.821+0000 7fc0fde00700 1 -- 172.21.15.28:0/3538060239 learned_addr learned my addr 172.21.15.28:0/3538060239 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:25.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.822+0000 7fc0fde00700 1 -- 172.21.15.28:0/3538060239 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0f81a3070 con 0x7fc0f810f050 2024-04-23T14:33:25.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.822+0000 7fc0fde00700 1 --2- 172.21.15.28:0/3538060239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0f810f050 0x7fc0f81a2b30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec011850 tx=0x7fc0ec011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:25.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.822+0000 7fc0f2a00700 1 -- 172.21.15.28:0/3538060239 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc0ec003af0 con 0x7fc0f810f050 2024-04-23T14:33:25.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.822+0000 7fc0f2a00700 1 -- 172.21.15.28:0/3538060239 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc0ec020070 con 0x7fc0f810f050 2024-04-23T14:33:25.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.822+0000 7fc0ff200700 1 -- 172.21.15.28:0/3538060239 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0f81a3300 con 0x7fc0f810f050 2024-04-23T14:33:25.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.822+0000 7fc0f2a00700 1 -- 172.21.15.28:0/3538060239 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc0ec0195a0 con 0x7fc0f810f050 2024-04-23T14:33:25.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.823+0000 7fc0ff200700 1 -- 172.21.15.28:0/3538060239 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0f81a5ff0 con 0x7fc0f810f050 2024-04-23T14:33:25.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.824+0000 7fc0ff200700 1 -- 172.21.15.28:0/3538060239 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0f8064150 con 0x7fc0f810f050 2024-04-23T14:33:25.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.824+0000 7fc0f2a00700 1 -- 172.21.15.28:0/3538060239 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fc0ec019850 con 0x7fc0f810f050 2024-04-23T14:33:25.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.824+0000 7fc0f2a00700 1 --2- 172.21.15.28:0/3538060239 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7fc0e0039600 0x7fc0e003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:25.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.824+0000 7fc0f2a00700 1 -- 172.21.15.28:0/3538060239 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc0ec011e00 con 0x7fc0f810f050 2024-04-23T14:33:25.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.825+0000 7fc0fd400700 1 --2- 172.21.15.28:0/3538060239 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7fc0e0039600 0x7fc0e003bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:25.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.825+0000 7fc0fd400700 1 --2- 172.21.15.28:0/3538060239 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7fc0e0039600 0x7fc0e003bac0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8006fd0 tx=0x7fc0e8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:25.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:24.830+0000 7fc0f2a00700 1 -- 172.21.15.28:0/3538060239 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc0ec00e260 con 0x7fc0f810f050 2024-04-23T14:33:25.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:25.067+0000 7fc0ff200700 1 -- 172.21.15.28:0/3538060239 --> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fc0f80800c0 con 0x7fc0e0039600 2024-04-23T14:33:25.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:25.077+0000 7fc0f2a00700 1 -- 172.21.15.28:0/3538060239 <== mgr.14120 v2:172.21.15.28:6800/2222038555 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fc0f80800c0 con 0x7fc0e0039600 2024-04-23T14:33:25.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:25.079+0000 7fc0ff200700 1 -- 172.21.15.28:0/3538060239 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7fc0e0039600 msgr2=0x7fc0e003bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:25.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:25.079+0000 7fc0ff200700 1 --2- 172.21.15.28:0/3538060239 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7fc0e0039600 0x7fc0e003bac0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8006fd0 tx=0x7fc0e8006e40 comp rx=0 tx=0).stop 2024-04-23T14:33:25.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:25.080+0000 7fc0ff200700 1 -- 172.21.15.28:0/3538060239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0f810f050 msgr2=0x7fc0f81a2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:25.733 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:25.080+0000 7fc0ff200700 1 --2- 172.21.15.28:0/3538060239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0f810f050 0x7fc0f81a2b30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec011850 tx=0x7fc0ec011c10 comp rx=0 tx=0).stop 2024-04-23T14:33:25.733 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:25.080+0000 7fc0ff200700 1 -- 172.21.15.28:0/3538060239 shutdown_connections 2024-04-23T14:33:25.733 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:25.080+0000 7fc0ff200700 1 --2- 172.21.15.28:0/3538060239 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7fc0e0039600 0x7fc0e003bac0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:25.733 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:25.080+0000 7fc0ff200700 1 --2- 172.21.15.28:0/3538060239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0f810f050 0x7fc0f81a2b30 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:25.733 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:25.080+0000 7fc0ff200700 1 -- 172.21.15.28:0/3538060239 >> 172.21.15.28:0/3538060239 conn(0x7fc0f807de00 msgr2=0x7fc0f807f9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:25.733 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:25.080+0000 7fc0ff200700 1 -- 172.21.15.28:0/3538060239 shutdown_connections 2024-04-23T14:33:25.733 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:25.080+0000 7fc0ff200700 1 -- 172.21.15.28:0/3538060239 wait complete. 2024-04-23T14:33:25.733 INFO:teuthology.orchestra.run.smithi028.stdout:Deploying unmanaged mgr service... 2024-04-23T14:33:26.220 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:26 smithi028 ceph-mon[27810]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:33:26.221 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:26 smithi028 ceph-mon[27810]: Saving service mon spec with placement count:5 2024-04-23T14:33:26.221 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:26 smithi028 ceph-mon[27810]: from='mgr.14120 172.21.15.28:0/2789735782' entity='mgr.x' 2024-04-23T14:33:27.931 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-23T14:33:27.931 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.702+0000 7f9a2b600700 1 Processor -- start 2024-04-23T14:33:27.931 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.702+0000 7f9a2b600700 1 -- start start 2024-04-23T14:33:27.931 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.702+0000 7f9a2b600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a24108790 0x7f9a24108bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:27.931 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.702+0000 7f9a2b600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9a24109180 con 0x7f9a24108790 2024-04-23T14:33:27.931 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.703+0000 7f9a28c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a24108790 0x7f9a24108bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:27.931 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.703+0000 7f9a28c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a24108790 0x7f9a24108bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34940/0 (socket says 172.21.15.28:34940) 2024-04-23T14:33:27.931 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.703+0000 7f9a28c00700 1 -- 172.21.15.28:0/416873915 learned_addr learned my addr 172.21.15.28:0/416873915 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:27.931 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.703+0000 7f9a28c00700 1 -- 172.21.15.28:0/416873915 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a24109940 con 0x7f9a24108790 2024-04-23T14:33:27.932 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.704+0000 7f9a28c00700 1 --2- 172.21.15.28:0/416873915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a24108790 0x7f9a24108bb0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9a18009e80 tx=0x7f9a1800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=709c6c5918aa987b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:27.932 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.704+0000 7f9a23400700 1 -- 172.21.15.28:0/416873915 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a18017070 con 0x7f9a24108790 2024-04-23T14:33:27.932 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.704+0000 7f9a23400700 1 -- 172.21.15.28:0/416873915 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9a1800b8c0 con 0x7f9a24108790 2024-04-23T14:33:27.934 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.704+0000 7f9a23400700 1 -- 172.21.15.28:0/416873915 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a180045c0 con 0x7f9a24108790 2024-04-23T14:33:27.934 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.705+0000 7f9a2b600700 1 -- 172.21.15.28:0/416873915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a24108790 msgr2=0x7f9a24108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:27.934 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.705+0000 7f9a2b600700 1 --2- 172.21.15.28:0/416873915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a24108790 0x7f9a24108bb0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9a18009e80 tx=0x7f9a1800b330 comp rx=0 tx=0).stop 2024-04-23T14:33:27.934 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.705+0000 7f9a2b600700 1 -- 172.21.15.28:0/416873915 shutdown_connections 2024-04-23T14:33:27.934 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.705+0000 7f9a2b600700 1 --2- 172.21.15.28:0/416873915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a24108790 0x7f9a24108bb0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:27.934 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.705+0000 7f9a2b600700 1 -- 172.21.15.28:0/416873915 >> 172.21.15.28:0/416873915 conn(0x7f9a24103d70 msgr2=0x7f9a24106190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:27.934 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.705+0000 7f9a2b600700 1 -- 172.21.15.28:0/416873915 shutdown_connections 2024-04-23T14:33:27.935 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.705+0000 7f9a2b600700 1 -- 172.21.15.28:0/416873915 wait complete. 2024-04-23T14:33:27.935 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.706+0000 7f9a2b600700 1 Processor -- start 2024-04-23T14:33:27.935 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.706+0000 7f9a2b600700 1 -- start start 2024-04-23T14:33:27.935 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.707+0000 7f9a2b600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a24108790 0x7f9a2419e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:27.935 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.707+0000 7f9a2b600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9a2410ccf0 con 0x7f9a24108790 2024-04-23T14:33:27.935 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.707+0000 7f9a28c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a24108790 0x7f9a2419e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:27.935 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.707+0000 7f9a28c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a24108790 0x7f9a2419e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34944/0 (socket says 172.21.15.28:34944) 2024-04-23T14:33:27.935 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.707+0000 7f9a28c00700 1 -- 172.21.15.28:0/1125711075 learned_addr learned my addr 172.21.15.28:0/1125711075 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:27.935 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.707+0000 7f9a28c00700 1 -- 172.21.15.28:0/1125711075 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a2419ee10 con 0x7f9a24108790 2024-04-23T14:33:27.935 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.707+0000 7f9a28c00700 1 --2- 172.21.15.28:0/1125711075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a24108790 0x7f9a2419e8d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9a18009e50 tx=0x7f9a1800bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:27.935 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.708+0000 7f9a21600700 1 -- 172.21.15.28:0/1125711075 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a18020020 con 0x7f9a24108790 2024-04-23T14:33:27.935 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.708+0000 7f9a21600700 1 -- 172.21.15.28:0/1125711075 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9a18003b50 con 0x7f9a24108790 2024-04-23T14:33:27.935 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.708+0000 7f9a2b600700 1 -- 172.21.15.28:0/1125711075 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a2419f0a0 con 0x7f9a24108790 2024-04-23T14:33:27.935 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.708+0000 7f9a21600700 1 -- 172.21.15.28:0/1125711075 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a18017040 con 0x7f9a24108790 2024-04-23T14:33:27.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.708+0000 7f9a2b600700 1 -- 172.21.15.28:0/1125711075 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a241a1d90 con 0x7f9a24108790 2024-04-23T14:33:27.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.709+0000 7f9a21600700 1 -- 172.21.15.28:0/1125711075 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f9a18023020 con 0x7f9a24108790 2024-04-23T14:33:27.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.709+0000 7f9a2b600700 1 -- 172.21.15.28:0/1125711075 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a24064150 con 0x7f9a24108790 2024-04-23T14:33:27.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.710+0000 7f9a21600700 1 --2- 172.21.15.28:0/1125711075 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f9a0c0395b0 0x7f9a0c03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:27.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.710+0000 7f9a21600700 1 -- 172.21.15.28:0/1125711075 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9a1800f050 con 0x7f9a24108790 2024-04-23T14:33:27.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.710+0000 7f9a23e00700 1 --2- 172.21.15.28:0/1125711075 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f9a0c0395b0 0x7f9a0c03ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:27.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.711+0000 7f9a23e00700 1 --2- 172.21.15.28:0/1125711075 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f9a0c0395b0 0x7f9a0c03ba70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9a10006fd0 tx=0x7f9a10006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:27.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.716+0000 7f9a21600700 1 -- 172.21.15.28:0/1125711075 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9a1802a020 con 0x7f9a24108790 2024-04-23T14:33:27.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.968+0000 7f9a2b600700 1 -- 172.21.15.28:0/1125711075 --> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f9a24104f00 con 0x7f9a0c0395b0 2024-04-23T14:33:27.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.979+0000 7f9a21600700 1 -- 172.21.15.28:0/1125711075 <== mgr.14120 v2:172.21.15.28:6800/2222038555 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f9a24104f00 con 0x7f9a0c0395b0 2024-04-23T14:33:27.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.982+0000 7f9a2b600700 1 -- 172.21.15.28:0/1125711075 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f9a0c0395b0 msgr2=0x7f9a0c03ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:27.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.982+0000 7f9a2b600700 1 --2- 172.21.15.28:0/1125711075 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f9a0c0395b0 0x7f9a0c03ba70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9a10006fd0 tx=0x7f9a10006e40 comp rx=0 tx=0).stop 2024-04-23T14:33:27.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.982+0000 7f9a2b600700 1 -- 172.21.15.28:0/1125711075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a24108790 msgr2=0x7f9a2419e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:27.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.982+0000 7f9a2b600700 1 --2- 172.21.15.28:0/1125711075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a24108790 0x7f9a2419e8d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9a18009e50 tx=0x7f9a1800bd30 comp rx=0 tx=0).stop 2024-04-23T14:33:27.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.983+0000 7f9a2b600700 1 -- 172.21.15.28:0/1125711075 shutdown_connections 2024-04-23T14:33:27.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.983+0000 7f9a2b600700 1 --2- 172.21.15.28:0/1125711075 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f9a0c0395b0 0x7f9a0c03ba70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:27.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.983+0000 7f9a2b600700 1 --2- 172.21.15.28:0/1125711075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a24108790 0x7f9a2419e8d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:27.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.983+0000 7f9a2b600700 1 -- 172.21.15.28:0/1125711075 >> 172.21.15.28:0/1125711075 conn(0x7f9a24103d70 msgr2=0x7f9a241047f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:27.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.983+0000 7f9a2b600700 1 -- 172.21.15.28:0/1125711075 shutdown_connections 2024-04-23T14:33:27.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:26.983+0000 7f9a2b600700 1 -- 172.21.15.28:0/1125711075 wait complete. 2024-04-23T14:33:28.230 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:27 smithi028 ceph-mon[27810]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:33:28.231 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:27 smithi028 ceph-mon[27810]: Saving service mgr spec with placement count:2 2024-04-23T14:33:28.231 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:27 smithi028 ceph-mon[27810]: from='mgr.14120 172.21.15.28:0/2789735782' entity='mgr.x' 2024-04-23T14:33:28.231 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:27 smithi028 ceph-mon[27810]: from='mgr.14120 172.21.15.28:0/2789735782' entity='mgr.x' 2024-04-23T14:33:29.452 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.527+0000 7f4bef000700 1 Processor -- start 2024-04-23T14:33:29.452 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.527+0000 7f4bef000700 1 -- start start 2024-04-23T14:33:29.452 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.528+0000 7f4bef000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4be8108790 0x7f4be8108bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:29.453 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.528+0000 7f4bef000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4be8109180 con 0x7f4be8108790 2024-04-23T14:33:29.453 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.528+0000 7f4be7e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4be8108790 0x7f4be8108bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:29.453 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.529+0000 7f4be7e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4be8108790 0x7f4be8108bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44068/0 (socket says 172.21.15.28:44068) 2024-04-23T14:33:29.454 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.529+0000 7f4be7e00700 1 -- 172.21.15.28:0/256760281 learned_addr learned my addr 172.21.15.28:0/256760281 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:29.454 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.529+0000 7f4be7e00700 1 -- 172.21.15.28:0/256760281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4be8109940 con 0x7f4be8108790 2024-04-23T14:33:29.454 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.529+0000 7f4be7e00700 1 --2- 172.21.15.28:0/256760281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4be8108790 0x7f4be8108bb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4bd80098a0 tx=0x7f4bd8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bedc27d3d41d5ac server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:29.454 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.530+0000 7f4be6a00700 1 -- 172.21.15.28:0/256760281 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4bd8004160 con 0x7f4be8108790 2024-04-23T14:33:29.454 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.530+0000 7f4be6a00700 1 -- 172.21.15.28:0/256760281 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bd8004300 con 0x7f4be8108790 2024-04-23T14:33:29.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.530+0000 7f4be6a00700 1 -- 172.21.15.28:0/256760281 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4bd8011b20 con 0x7f4be8108790 2024-04-23T14:33:29.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.531+0000 7f4bef000700 1 -- 172.21.15.28:0/256760281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4be8108790 msgr2=0x7f4be8108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:29.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.531+0000 7f4bef000700 1 --2- 172.21.15.28:0/256760281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4be8108790 0x7f4be8108bb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4bd80098a0 tx=0x7f4bd8009bb0 comp rx=0 tx=0).stop 2024-04-23T14:33:29.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.531+0000 7f4bef000700 1 -- 172.21.15.28:0/256760281 shutdown_connections 2024-04-23T14:33:29.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.531+0000 7f4bef000700 1 --2- 172.21.15.28:0/256760281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4be8108790 0x7f4be8108bb0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:29.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.531+0000 7f4bef000700 1 -- 172.21.15.28:0/256760281 >> 172.21.15.28:0/256760281 conn(0x7f4be8103d70 msgr2=0x7f4be8106190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:29.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.531+0000 7f4bef000700 1 -- 172.21.15.28:0/256760281 shutdown_connections 2024-04-23T14:33:29.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.531+0000 7f4bef000700 1 -- 172.21.15.28:0/256760281 wait complete. 2024-04-23T14:33:29.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.532+0000 7f4bef000700 1 Processor -- start 2024-04-23T14:33:29.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.532+0000 7f4bef000700 1 -- start start 2024-04-23T14:33:29.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.533+0000 7f4bef000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4be8108790 0x7f4be807db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:29.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.533+0000 7f4bef000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4be810ccf0 con 0x7f4be8108790 2024-04-23T14:33:29.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.533+0000 7f4be7e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4be8108790 0x7f4be807db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:29.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.533+0000 7f4be7e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4be8108790 0x7f4be807db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44072/0 (socket says 172.21.15.28:44072) 2024-04-23T14:33:29.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.533+0000 7f4be7e00700 1 -- 172.21.15.28:0/683458868 learned_addr learned my addr 172.21.15.28:0/683458868 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:29.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.533+0000 7f4be7e00700 1 -- 172.21.15.28:0/683458868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4be807e0a0 con 0x7f4be8108790 2024-04-23T14:33:29.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.534+0000 7f4be7e00700 1 --2- 172.21.15.28:0/683458868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4be8108790 0x7f4be807db60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4bd8000c00 tx=0x7f4bd8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:29.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.534+0000 7f4be4c00700 1 -- 172.21.15.28:0/683458868 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4bd8003b60 con 0x7f4be8108790 2024-04-23T14:33:29.459 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.534+0000 7f4be4c00700 1 -- 172.21.15.28:0/683458868 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bd801a910 con 0x7f4be8108790 2024-04-23T14:33:29.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.534+0000 7f4be4c00700 1 -- 172.21.15.28:0/683458868 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4bd8024930 con 0x7f4be8108790 2024-04-23T14:33:29.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.534+0000 7f4bef000700 1 -- 172.21.15.28:0/683458868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4be807c290 con 0x7f4be8108790 2024-04-23T14:33:29.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.534+0000 7f4bef000700 1 -- 172.21.15.28:0/683458868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4be807c750 con 0x7f4be8108790 2024-04-23T14:33:29.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.535+0000 7f4bef000700 1 -- 172.21.15.28:0/683458868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4be8064150 con 0x7f4be8108790 2024-04-23T14:33:29.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.536+0000 7f4be4c00700 1 -- 172.21.15.28:0/683458868 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f4bd8022020 con 0x7f4be8108790 2024-04-23T14:33:29.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.536+0000 7f4be4c00700 1 --2- 172.21.15.28:0/683458868 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f4bcc039600 0x7f4bcc03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:29.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.536+0000 7f4be4c00700 1 -- 172.21.15.28:0/683458868 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4bd804d8d0 con 0x7f4be8108790 2024-04-23T14:33:29.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.537+0000 7f4be7400700 1 --2- 172.21.15.28:0/683458868 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f4bcc039600 0x7f4bcc03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:29.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.538+0000 7f4be7400700 1 --2- 172.21.15.28:0/683458868 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f4bcc039600 0x7f4bcc03bac0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4006fd0 tx=0x7f4bd4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:29.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.542+0000 7f4be4c00700 1 -- 172.21.15.28:0/683458868 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4bd800f2e0 con 0x7f4be8108790 2024-04-23T14:33:29.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.772+0000 7f4bef000700 1 -- 172.21.15.28:0/683458868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f4be807cff0 con 0x7f4be8108790 2024-04-23T14:33:29.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.777+0000 7f4be4c00700 1 -- 172.21.15.28:0/683458868 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f4bd801ab70 con 0x7f4be8108790 2024-04-23T14:33:29.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.780+0000 7f4bef000700 1 -- 172.21.15.28:0/683458868 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f4bcc039600 msgr2=0x7f4bcc03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:29.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.780+0000 7f4bef000700 1 --2- 172.21.15.28:0/683458868 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f4bcc039600 0x7f4bcc03bac0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4006fd0 tx=0x7f4bd4006e40 comp rx=0 tx=0).stop 2024-04-23T14:33:29.461 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.780+0000 7f4bef000700 1 -- 172.21.15.28:0/683458868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4be8108790 msgr2=0x7f4be807db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:29.461 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.780+0000 7f4bef000700 1 --2- 172.21.15.28:0/683458868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4be8108790 0x7f4be807db60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4bd8000c00 tx=0x7f4bd8011c10 comp rx=0 tx=0).stop 2024-04-23T14:33:29.461 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.781+0000 7f4bef000700 1 -- 172.21.15.28:0/683458868 shutdown_connections 2024-04-23T14:33:29.461 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.781+0000 7f4bef000700 1 --2- 172.21.15.28:0/683458868 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f4bcc039600 0x7f4bcc03bac0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:29.461 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.781+0000 7f4bef000700 1 --2- 172.21.15.28:0/683458868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4be8108790 0x7f4be807db60 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:29.461 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.781+0000 7f4bef000700 1 -- 172.21.15.28:0/683458868 >> 172.21.15.28:0/683458868 conn(0x7f4be8103d70 msgr2=0x7f4be8104810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:29.461 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.781+0000 7f4bef000700 1 -- 172.21.15.28:0/683458868 shutdown_connections 2024-04-23T14:33:29.461 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:28.782+0000 7f4bef000700 1 -- 172.21.15.28:0/683458868 wait complete. 2024-04-23T14:33:29.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:29 smithi028 ceph-mon[27810]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:33:29.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:29 smithi028 ceph-mon[27810]: from='mgr.14120 172.21.15.28:0/2789735782' entity='mgr.x' 2024-04-23T14:33:29.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:29 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/683458868' entity='client.admin' 2024-04-23T14:33:31.203 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.049+0000 7fb4d5e00700 1 Processor -- start 2024-04-23T14:33:31.204 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.049+0000 7fb4d5e00700 1 -- start start 2024-04-23T14:33:31.204 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.050+0000 7fb4d5e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4d0108790 0x7fb4d0108bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:31.204 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.050+0000 7fb4d5e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d0109180 con 0x7fb4d0108790 2024-04-23T14:33:31.204 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.050+0000 7fb4cea00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4d0108790 0x7fb4d0108bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:31.204 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.050+0000 7fb4cea00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4d0108790 0x7fb4d0108bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44074/0 (socket says 172.21.15.28:44074) 2024-04-23T14:33:31.204 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.050+0000 7fb4cea00700 1 -- 172.21.15.28:0/3978995101 learned_addr learned my addr 172.21.15.28:0/3978995101 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:31.204 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.051+0000 7fb4cea00700 1 -- 172.21.15.28:0/3978995101 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4d0109940 con 0x7fb4d0108790 2024-04-23T14:33:31.204 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.051+0000 7fb4cea00700 1 --2- 172.21.15.28:0/3978995101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4d0108790 0x7fb4d0108bb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0009e80 tx=0x7fb4c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95b433fdfcb46686 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:31.204 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.051+0000 7fb4ce000700 1 -- 172.21.15.28:0/3978995101 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb4c0017070 con 0x7fb4d0108790 2024-04-23T14:33:31.205 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.051+0000 7fb4ce000700 1 -- 172.21.15.28:0/3978995101 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4c000b8c0 con 0x7fb4d0108790 2024-04-23T14:33:31.205 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.051+0000 7fb4ce000700 1 -- 172.21.15.28:0/3978995101 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb4c00045c0 con 0x7fb4d0108790 2024-04-23T14:33:31.205 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.052+0000 7fb4d5e00700 1 -- 172.21.15.28:0/3978995101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4d0108790 msgr2=0x7fb4d0108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:31.205 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.052+0000 7fb4d5e00700 1 --2- 172.21.15.28:0/3978995101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4d0108790 0x7fb4d0108bb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0009e80 tx=0x7fb4c000b330 comp rx=0 tx=0).stop 2024-04-23T14:33:31.205 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.052+0000 7fb4d5e00700 1 -- 172.21.15.28:0/3978995101 shutdown_connections 2024-04-23T14:33:31.205 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.052+0000 7fb4d5e00700 1 --2- 172.21.15.28:0/3978995101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4d0108790 0x7fb4d0108bb0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:31.205 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.052+0000 7fb4d5e00700 1 -- 172.21.15.28:0/3978995101 >> 172.21.15.28:0/3978995101 conn(0x7fb4d0103d70 msgr2=0x7fb4d0106190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:31.205 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.052+0000 7fb4d5e00700 1 -- 172.21.15.28:0/3978995101 shutdown_connections 2024-04-23T14:33:31.205 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.052+0000 7fb4d5e00700 1 -- 172.21.15.28:0/3978995101 wait complete. 2024-04-23T14:33:31.205 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.053+0000 7fb4d5e00700 1 Processor -- start 2024-04-23T14:33:31.205 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.053+0000 7fb4d5e00700 1 -- start start 2024-04-23T14:33:31.205 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.054+0000 7fb4d5e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4d0108790 0x7fb4d019e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:31.206 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.054+0000 7fb4d5e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d010ccf0 con 0x7fb4d0108790 2024-04-23T14:33:31.206 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.054+0000 7fb4cea00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4d0108790 0x7fb4d019e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:31.206 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.054+0000 7fb4cea00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4d0108790 0x7fb4d019e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44082/0 (socket says 172.21.15.28:44082) 2024-04-23T14:33:31.206 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.054+0000 7fb4cea00700 1 -- 172.21.15.28:0/30765982 learned_addr learned my addr 172.21.15.28:0/30765982 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:31.206 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.054+0000 7fb4cea00700 1 -- 172.21.15.28:0/30765982 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4d019edd0 con 0x7fb4d0108790 2024-04-23T14:33:31.206 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.054+0000 7fb4cea00700 1 --2- 172.21.15.28:0/30765982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4d0108790 0x7fb4d019e890 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0009530 tx=0x7fb4c0003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:31.206 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.055+0000 7fb4d4a00700 1 -- 172.21.15.28:0/30765982 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb4c0020020 con 0x7fb4d0108790 2024-04-23T14:33:31.206 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.055+0000 7fb4d4a00700 1 -- 172.21.15.28:0/30765982 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4c000bba0 con 0x7fb4d0108790 2024-04-23T14:33:31.206 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.055+0000 7fb4d5e00700 1 -- 172.21.15.28:0/30765982 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4d019f060 con 0x7fb4d0108790 2024-04-23T14:33:31.206 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.055+0000 7fb4d4a00700 1 -- 172.21.15.28:0/30765982 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb4c0017040 con 0x7fb4d0108790 2024-04-23T14:33:31.206 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.055+0000 7fb4d5e00700 1 -- 172.21.15.28:0/30765982 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4d01a1d50 con 0x7fb4d0108790 2024-04-23T14:33:31.206 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.056+0000 7fb4d5e00700 1 -- 172.21.15.28:0/30765982 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4d0064150 con 0x7fb4d0108790 2024-04-23T14:33:31.207 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.056+0000 7fb4d4a00700 1 -- 172.21.15.28:0/30765982 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fb4c0023020 con 0x7fb4d0108790 2024-04-23T14:33:31.207 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.057+0000 7fb4d4a00700 1 --2- 172.21.15.28:0/30765982 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7fb4b40395b0 0x7fb4b403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:31.207 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.057+0000 7fb4d4a00700 1 -- 172.21.15.28:0/30765982 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb4c000f050 con 0x7fb4d0108790 2024-04-23T14:33:31.207 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.057+0000 7fb4c7e00700 1 --2- 172.21.15.28:0/30765982 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7fb4b40395b0 0x7fb4b403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:31.207 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.058+0000 7fb4c7e00700 1 --2- 172.21.15.28:0/30765982 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7fb4b40395b0 0x7fb4b403ba70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8006fd0 tx=0x7fb4b8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:31.207 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.062+0000 7fb4d4a00700 1 -- 172.21.15.28:0/30765982 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb4c002a020 con 0x7fb4d0108790 2024-04-23T14:33:31.207 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.293+0000 7fb4d5e00700 1 -- 172.21.15.28:0/30765982 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7fb4d01a1f20 con 0x7fb4d0108790 2024-04-23T14:33:31.207 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.296+0000 7fb4d4a00700 1 -- 172.21.15.28:0/30765982 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7fb4c002d350 con 0x7fb4d0108790 2024-04-23T14:33:31.207 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.300+0000 7fb4d5e00700 1 -- 172.21.15.28:0/30765982 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7fb4b40395b0 msgr2=0x7fb4b403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:31.207 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.300+0000 7fb4d5e00700 1 --2- 172.21.15.28:0/30765982 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7fb4b40395b0 0x7fb4b403ba70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8006fd0 tx=0x7fb4b8006e40 comp rx=0 tx=0).stop 2024-04-23T14:33:31.207 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.301+0000 7fb4d5e00700 1 -- 172.21.15.28:0/30765982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4d0108790 msgr2=0x7fb4d019e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:31.208 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.301+0000 7fb4d5e00700 1 --2- 172.21.15.28:0/30765982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4d0108790 0x7fb4d019e890 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0009530 tx=0x7fb4c0003d00 comp rx=0 tx=0).stop 2024-04-23T14:33:31.208 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.301+0000 7fb4d5e00700 1 -- 172.21.15.28:0/30765982 shutdown_connections 2024-04-23T14:33:31.208 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.301+0000 7fb4d5e00700 1 --2- 172.21.15.28:0/30765982 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7fb4b40395b0 0x7fb4b403ba70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:31.208 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.301+0000 7fb4d5e00700 1 --2- 172.21.15.28:0/30765982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4d0108790 0x7fb4d019e890 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:31.208 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.301+0000 7fb4d5e00700 1 -- 172.21.15.28:0/30765982 >> 172.21.15.28:0/30765982 conn(0x7fb4d0103d70 msgr2=0x7fb4d01047d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:31.208 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.301+0000 7fb4d5e00700 1 -- 172.21.15.28:0/30765982 shutdown_connections 2024-04-23T14:33:31.208 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:30.302+0000 7fb4d5e00700 1 -- 172.21.15.28:0/30765982 wait complete. 2024-04-23T14:33:31.208 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling the dashboard module... 2024-04-23T14:33:31.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:31 smithi028 ceph-mon[27810]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:33:31.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:31 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/30765982' entity='client.admin' 2024-04-23T14:33:31.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:31 smithi028 ceph-mon[27810]: from='mgr.14120 172.21.15.28:0/2789735782' entity='mgr.x' 2024-04-23T14:33:32.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:32 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/4108021521' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-23T14:33:32.711 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:32 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: ignoring --setuser ceph since I am not root 2024-04-23T14:33:32.712 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:32 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: ignoring --setgroup ceph since I am not root 2024-04-23T14:33:32.712 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:32 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:32.626+0000 7f22977e2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T14:33:33.020 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:32 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:32.890+0000 7f22977e2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T14:33:33.030 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.793+0000 7f03ec000700 1 Processor -- start 2024-04-23T14:33:33.035 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.793+0000 7f03ec000700 1 -- start start 2024-04-23T14:33:33.035 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.793+0000 7f03ec000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03e41091f0 0x7f03e410b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:33.036 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.793+0000 7f03ec000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f03e407be80 con 0x7f03e41091f0 2024-04-23T14:33:33.036 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.794+0000 7f03e9600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03e41091f0 0x7f03e410b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:33.036 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.794+0000 7f03e9600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03e41091f0 0x7f03e410b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44084/0 (socket says 172.21.15.28:44084) 2024-04-23T14:33:33.036 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.794+0000 7f03e9600700 1 -- 172.21.15.28:0/4222606053 learned_addr learned my addr 172.21.15.28:0/4222606053 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:33.036 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.794+0000 7f03e9600700 1 -- 172.21.15.28:0/4222606053 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03e410bb20 con 0x7f03e41091f0 2024-04-23T14:33:33.036 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.795+0000 7f03e9600700 1 --2- 172.21.15.28:0/4222606053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03e41091f0 0x7f03e410b5e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f03d80098a0 tx=0x7f03d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fed80bd838935cc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:33.036 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.795+0000 7f03e3e00700 1 -- 172.21.15.28:0/4222606053 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f03d8004160 con 0x7f03e41091f0 2024-04-23T14:33:33.036 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.795+0000 7f03e3e00700 1 -- 172.21.15.28:0/4222606053 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03d8004300 con 0x7f03e41091f0 2024-04-23T14:33:33.036 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.795+0000 7f03e3e00700 1 -- 172.21.15.28:0/4222606053 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f03d8011b20 con 0x7f03e41091f0 2024-04-23T14:33:33.036 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.795+0000 7f03ec000700 1 -- 172.21.15.28:0/4222606053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03e41091f0 msgr2=0x7f03e410b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:33.036 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.795+0000 7f03ec000700 1 --2- 172.21.15.28:0/4222606053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03e41091f0 0x7f03e410b5e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f03d80098a0 tx=0x7f03d8009bb0 comp rx=0 tx=0).stop 2024-04-23T14:33:33.036 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.796+0000 7f03ec000700 1 -- 172.21.15.28:0/4222606053 shutdown_connections 2024-04-23T14:33:33.037 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.796+0000 7f03ec000700 1 --2- 172.21.15.28:0/4222606053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03e41091f0 0x7f03e410b5e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:33.037 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.796+0000 7f03ec000700 1 -- 172.21.15.28:0/4222606053 >> 172.21.15.28:0/4222606053 conn(0x7f03e4102c60 msgr2=0x7f03e4105080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:33.037 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.796+0000 7f03ec000700 1 -- 172.21.15.28:0/4222606053 shutdown_connections 2024-04-23T14:33:33.037 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.796+0000 7f03ec000700 1 -- 172.21.15.28:0/4222606053 wait complete. 2024-04-23T14:33:33.037 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.797+0000 7f03ec000700 1 Processor -- start 2024-04-23T14:33:33.037 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.797+0000 7f03ec000700 1 -- start start 2024-04-23T14:33:33.037 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.797+0000 7f03ec000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03e41091f0 0x7f03e419c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:33.037 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.797+0000 7f03ec000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f03e410cb20 con 0x7f03e41091f0 2024-04-23T14:33:33.037 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.798+0000 7f03e9600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03e41091f0 0x7f03e419c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:33.037 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.798+0000 7f03e9600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03e41091f0 0x7f03e419c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44086/0 (socket says 172.21.15.28:44086) 2024-04-23T14:33:33.037 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.798+0000 7f03e9600700 1 -- 172.21.15.28:0/4108021521 learned_addr learned my addr 172.21.15.28:0/4108021521 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:33.037 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.798+0000 7f03e9600700 1 -- 172.21.15.28:0/4108021521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03e419cb50 con 0x7f03e41091f0 2024-04-23T14:33:33.037 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.798+0000 7f03e9600700 1 --2- 172.21.15.28:0/4108021521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03e41091f0 0x7f03e419c610 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f03d8011b60 tx=0x7f03d8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:33.037 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.799+0000 7f03e2000700 1 -- 172.21.15.28:0/4108021521 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f03d8003b60 con 0x7f03e41091f0 2024-04-23T14:33:33.038 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.799+0000 7f03e2000700 1 -- 172.21.15.28:0/4108021521 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03d801a910 con 0x7f03e41091f0 2024-04-23T14:33:33.038 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.799+0000 7f03ec000700 1 -- 172.21.15.28:0/4108021521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03e419ce40 con 0x7f03e41091f0 2024-04-23T14:33:33.038 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.799+0000 7f03e2000700 1 -- 172.21.15.28:0/4108021521 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f03d8024930 con 0x7f03e41091f0 2024-04-23T14:33:33.038 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.799+0000 7f03ec000700 1 -- 172.21.15.28:0/4108021521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03e419fad0 con 0x7f03e41091f0 2024-04-23T14:33:33.038 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.800+0000 7f03ec000700 1 -- 172.21.15.28:0/4108021521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03e4064150 con 0x7f03e41091f0 2024-04-23T14:33:33.038 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.800+0000 7f03e2000700 1 -- 172.21.15.28:0/4108021521 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f03d8022020 con 0x7f03e41091f0 2024-04-23T14:33:33.038 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.801+0000 7f03e2000700 1 --2- 172.21.15.28:0/4108021521 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f03cc039600 0x7f03cc03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:33.038 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.801+0000 7f03e2000700 1 -- 172.21.15.28:0/4108021521 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f03d804da60 con 0x7f03e41091f0 2024-04-23T14:33:33.038 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.801+0000 7f03e8c00700 1 --2- 172.21.15.28:0/4108021521 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f03cc039600 0x7f03cc03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:33.038 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.802+0000 7f03e8c00700 1 --2- 172.21.15.28:0/4108021521 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f03cc039600 0x7f03cc03bac0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f03d0006fd0 tx=0x7f03d0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:33.038 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:31.807+0000 7f03e2000700 1 -- 172.21.15.28:0/4108021521 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f03d804d3b0 con 0x7f03e41091f0 2024-04-23T14:33:33.038 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:32.120+0000 7f03ec000700 1 -- 172.21.15.28:0/4108021521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f03e419fca0 con 0x7f03e41091f0 2024-04-23T14:33:33.038 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:32.304+0000 7f03e2000700 1 -- 172.21.15.28:0/4108021521 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f03d801a4e0 con 0x7f03e41091f0 2024-04-23T14:33:33.038 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:32.307+0000 7f03e2000700 1 -- 172.21.15.28:0/4108021521 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f03d802d3f0 con 0x7f03e41091f0 2024-04-23T14:33:33.038 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:32.309+0000 7f03ec000700 1 -- 172.21.15.28:0/4108021521 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f03cc039600 msgr2=0x7f03cc03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:33.039 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:32.309+0000 7f03ec000700 1 --2- 172.21.15.28:0/4108021521 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f03cc039600 0x7f03cc03bac0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f03d0006fd0 tx=0x7f03d0006e40 comp rx=0 tx=0).stop 2024-04-23T14:33:33.039 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:32.309+0000 7f03ec000700 1 -- 172.21.15.28:0/4108021521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03e41091f0 msgr2=0x7f03e419c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:33.039 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:32.309+0000 7f03ec000700 1 --2- 172.21.15.28:0/4108021521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03e41091f0 0x7f03e419c610 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f03d8011b60 tx=0x7f03d8011c40 comp rx=0 tx=0).stop 2024-04-23T14:33:33.039 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:32.310+0000 7f03ec000700 1 -- 172.21.15.28:0/4108021521 shutdown_connections 2024-04-23T14:33:33.039 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:32.310+0000 7f03ec000700 1 --2- 172.21.15.28:0/4108021521 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7f03cc039600 0x7f03cc03bac0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:33.039 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:32.310+0000 7f03ec000700 1 --2- 172.21.15.28:0/4108021521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03e41091f0 0x7f03e419c610 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:33.039 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:32.310+0000 7f03ec000700 1 -- 172.21.15.28:0/4108021521 >> 172.21.15.28:0/4108021521 conn(0x7f03e4102c60 msgr2=0x7f03e4103700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:33.039 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:32.310+0000 7f03ec000700 1 -- 172.21.15.28:0/4108021521 shutdown_connections 2024-04-23T14:33:33.039 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:32.310+0000 7f03ec000700 1 -- 172.21.15.28:0/4108021521 wait complete. 2024-04-23T14:33:33.307 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:33 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:33.036+0000 7f22977e2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T14:33:33.563 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:33 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:33.455+0000 7f22977e2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T14:33:33.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:33 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/4108021521' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-23T14:33:33.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:33 smithi028 ceph-mon[27810]: mgrmap e9: x(active, since 24s) 2024-04-23T14:33:33.846 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:33 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:33.572+0000 7f22977e2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T14:33:34.567 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:34 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1214823955' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-23T14:33:34.576 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-23T14:33:34.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-23T14:33:34.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-23T14:33:34.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-23T14:33:34.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-23T14:33:34.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-23T14:33:34.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.636+0000 7fe2b6200700 1 Processor -- start 2024-04-23T14:33:34.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.636+0000 7fe2b6200700 1 -- start start 2024-04-23T14:33:34.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.636+0000 7fe2b6200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2b01091f0 0x7fe2b010b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:34.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.636+0000 7fe2b6200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b007be80 con 0x7fe2b01091f0 2024-04-23T14:33:34.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.637+0000 7fe2af400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2b01091f0 0x7fe2b010b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:34.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.637+0000 7fe2af400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2b01091f0 0x7fe2b010b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44110/0 (socket says 172.21.15.28:44110) 2024-04-23T14:33:34.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.637+0000 7fe2af400700 1 -- 172.21.15.28:0/4210624525 learned_addr learned my addr 172.21.15.28:0/4210624525 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:34.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.637+0000 7fe2af400700 1 -- 172.21.15.28:0/4210624525 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2b010bb20 con 0x7fe2b01091f0 2024-04-23T14:33:34.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.637+0000 7fe2af400700 1 --2- 172.21.15.28:0/4210624525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2b01091f0 0x7fe2b010b5e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe2a00098a0 tx=0x7fe2a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75381a8117c96cc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:34.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.638+0000 7fe2ae000700 1 -- 172.21.15.28:0/4210624525 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe2a0004160 con 0x7fe2b01091f0 2024-04-23T14:33:34.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.638+0000 7fe2ae000700 1 -- 172.21.15.28:0/4210624525 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2a0004300 con 0x7fe2b01091f0 2024-04-23T14:33:34.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.638+0000 7fe2ae000700 1 -- 172.21.15.28:0/4210624525 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe2a0011b20 con 0x7fe2b01091f0 2024-04-23T14:33:34.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.638+0000 7fe2b6200700 1 -- 172.21.15.28:0/4210624525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2b01091f0 msgr2=0x7fe2b010b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:34.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.638+0000 7fe2b6200700 1 --2- 172.21.15.28:0/4210624525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2b01091f0 0x7fe2b010b5e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe2a00098a0 tx=0x7fe2a0009bb0 comp rx=0 tx=0).stop 2024-04-23T14:33:34.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.639+0000 7fe2b6200700 1 -- 172.21.15.28:0/4210624525 shutdown_connections 2024-04-23T14:33:34.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.639+0000 7fe2b6200700 1 --2- 172.21.15.28:0/4210624525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2b01091f0 0x7fe2b010b5e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:34.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.639+0000 7fe2b6200700 1 -- 172.21.15.28:0/4210624525 >> 172.21.15.28:0/4210624525 conn(0x7fe2b0102c60 msgr2=0x7fe2b0105080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:34.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.639+0000 7fe2b6200700 1 -- 172.21.15.28:0/4210624525 shutdown_connections 2024-04-23T14:33:34.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.639+0000 7fe2b6200700 1 -- 172.21.15.28:0/4210624525 wait complete. 2024-04-23T14:33:34.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.640+0000 7fe2b6200700 1 Processor -- start 2024-04-23T14:33:34.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.640+0000 7fe2b6200700 1 -- start start 2024-04-23T14:33:34.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.640+0000 7fe2b6200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2b01091f0 0x7fe2b019e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:34.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.640+0000 7fe2b6200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b010cb20 con 0x7fe2b01091f0 2024-04-23T14:33:34.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.640+0000 7fe2af400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2b01091f0 0x7fe2b019e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:34.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.641+0000 7fe2af400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2b01091f0 0x7fe2b019e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44114/0 (socket says 172.21.15.28:44114) 2024-04-23T14:33:34.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.641+0000 7fe2af400700 1 -- 172.21.15.28:0/1214823955 learned_addr learned my addr 172.21.15.28:0/1214823955 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:34.585 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.641+0000 7fe2af400700 1 -- 172.21.15.28:0/1214823955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2b019ed70 con 0x7fe2b01091f0 2024-04-23T14:33:34.585 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.641+0000 7fe2af400700 1 --2- 172.21.15.28:0/1214823955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2b01091f0 0x7fe2b019e830 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0011b60 tx=0x7fe2a0011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:34.585 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.641+0000 7fe2a7e00700 1 -- 172.21.15.28:0/1214823955 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe2a0003b60 con 0x7fe2b01091f0 2024-04-23T14:33:34.585 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.642+0000 7fe2a7e00700 1 -- 172.21.15.28:0/1214823955 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2a001a910 con 0x7fe2b01091f0 2024-04-23T14:33:34.585 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.642+0000 7fe2b6200700 1 -- 172.21.15.28:0/1214823955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2b019f060 con 0x7fe2b01091f0 2024-04-23T14:33:34.585 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.642+0000 7fe2b6200700 1 -- 172.21.15.28:0/1214823955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2b01a1cf0 con 0x7fe2b01091f0 2024-04-23T14:33:34.585 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.642+0000 7fe2a7e00700 1 -- 172.21.15.28:0/1214823955 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe2a0024930 con 0x7fe2b01091f0 2024-04-23T14:33:34.585 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.643+0000 7fe2a7e00700 1 -- 172.21.15.28:0/1214823955 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7fe2a0022020 con 0x7fe2b01091f0 2024-04-23T14:33:34.585 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.643+0000 7fe2b6200700 1 -- 172.21.15.28:0/1214823955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2b0064150 con 0x7fe2b01091f0 2024-04-23T14:33:34.585 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.643+0000 7fe2a7e00700 1 --2- 172.21.15.28:0/1214823955 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7fe294039600 0x7fe29403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:34.585 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.643+0000 7fe2a7e00700 1 -- 172.21.15.28:0/1214823955 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe2a004da70 con 0x7fe2b01091f0 2024-04-23T14:33:34.585 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.644+0000 7fe2aea00700 1 -- 172.21.15.28:0/1214823955 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7fe294039600 msgr2=0x7fe29403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2222038555 2024-04-23T14:33:34.585 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.644+0000 7fe2aea00700 1 --2- 172.21.15.28:0/1214823955 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7fe294039600 0x7fe29403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T14:33:34.586 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.649+0000 7fe2a7e00700 1 -- 172.21.15.28:0/1214823955 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe2a0019c70 con 0x7fe2b01091f0 2024-04-23T14:33:34.586 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.845+0000 7fe2aea00700 1 -- 172.21.15.28:0/1214823955 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7fe294039600 msgr2=0x7fe29403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2222038555 2024-04-23T14:33:34.586 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.845+0000 7fe2aea00700 1 --2- 172.21.15.28:0/1214823955 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7fe294039600 0x7fe29403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T14:33:34.586 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.937+0000 7fe2b6200700 1 -- 172.21.15.28:0/1214823955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fe2b01a1ec0 con 0x7fe2b01091f0 2024-04-23T14:33:34.586 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.938+0000 7fe2a7e00700 1 -- 172.21.15.28:0/1214823955 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fe2a002d910 con 0x7fe2b01091f0 2024-04-23T14:33:34.586 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.941+0000 7fe2b6200700 1 -- 172.21.15.28:0/1214823955 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7fe294039600 msgr2=0x7fe29403bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T14:33:34.586 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.941+0000 7fe2b6200700 1 --2- 172.21.15.28:0/1214823955 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7fe294039600 0x7fe29403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:34.586 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.941+0000 7fe2b6200700 1 -- 172.21.15.28:0/1214823955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2b01091f0 msgr2=0x7fe2b019e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:34.586 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.941+0000 7fe2b6200700 1 --2- 172.21.15.28:0/1214823955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2b01091f0 0x7fe2b019e830 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0011b60 tx=0x7fe2a0011c40 comp rx=0 tx=0).stop 2024-04-23T14:33:34.586 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.941+0000 7fe2b6200700 1 -- 172.21.15.28:0/1214823955 shutdown_connections 2024-04-23T14:33:34.586 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.941+0000 7fe2b6200700 1 --2- 172.21.15.28:0/1214823955 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7fe294039600 0x7fe29403bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:34.586 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.941+0000 7fe2b6200700 1 --2- 172.21.15.28:0/1214823955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2b01091f0 0x7fe2b019e830 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:34.586 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.941+0000 7fe2b6200700 1 -- 172.21.15.28:0/1214823955 >> 172.21.15.28:0/1214823955 conn(0x7fe2b0102c60 msgr2=0x7fe2b0103680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:34.586 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.941+0000 7fe2b6200700 1 -- 172.21.15.28:0/1214823955 shutdown_connections 2024-04-23T14:33:34.586 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:33.941+0000 7fe2b6200700 1 -- 172.21.15.28:0/1214823955 wait complete. 2024-04-23T14:33:34.586 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for the mgr to restart... 2024-04-23T14:33:34.587 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr epoch 9... 2024-04-23T14:33:35.624 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:35 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:35.178+0000 7f22977e2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T14:33:35.624 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:35 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:35.411+0000 7f22977e2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T14:33:36.024 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:35 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:35.631+0000 7f22977e2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T14:33:36.025 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:36 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:36.023+0000 7f22977e2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T14:33:36.374 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:36 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:36.147+0000 7f22977e2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T14:33:36.670 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:36 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:36.405+0000 7f22977e2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T14:33:36.671 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:36 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:36.669+0000 7f22977e2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T14:33:37.873 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:37 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:37.476+0000 7f22977e2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T14:33:38.374 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:38 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:38.044+0000 7f22977e2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T14:33:38.374 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:38 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:38.175+0000 7f22977e2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T14:33:38.874 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:38 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:38.450+0000 7f22977e2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T14:33:39.374 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:39 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:39.116+0000 7f22977e2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T14:33:39.374 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:39 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:39.243+0000 7f22977e2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T14:33:39.874 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:39 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:39.572+0000 7f22977e2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T14:33:39.874 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:39 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:39.757+0000 7f22977e2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T14:33:40.374 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:39 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:39.888+0000 7f22977e2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T14:33:40.374 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:40 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:40.045+0000 7f22977e2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T14:33:40.739 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:40 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:40.422+0000 7f22977e2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T14:33:41.373 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:41 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:41.229+0000 7f22977e2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T14:33:41.679 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:33:41 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:33:41.526+0000 7f22977e2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T14:33:41.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:41 smithi028 ceph-mon[27810]: Active manager daemon x restarted 2024-04-23T14:33:41.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:41 smithi028 ceph-mon[27810]: Activating manager daemon x 2024-04-23T14:33:41.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:41 smithi028 ceph-mon[27810]: osdmap e3: 0 total, 0 up, 0 in 2024-04-23T14:33:41.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:41 smithi028 ceph-mon[27810]: mgrmap e10: x(active, starting, since 0.00631817s) 2024-04-23T14:33:41.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:41 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T14:33:41.680 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:41 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T14:33:41.680 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:41 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T14:33:41.680 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:41 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T14:33:41.680 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:41 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T14:33:41.680 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:41 smithi028 ceph-mon[27810]: Manager daemon x is now available 2024-04-23T14:33:42.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:42 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:33:42.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:42 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T14:33:42.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:42 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T14:33:42.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:42 smithi028 ceph-mon[27810]: [23/Apr/2024:14:33:42] ENGINE Bus STARTING 2024-04-23T14:33:42.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:42 smithi028 ceph-mon[27810]: [23/Apr/2024:14:33:42] ENGINE Serving on https://172.21.15.28:7150 2024-04-23T14:33:42.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:42 smithi028 ceph-mon[27810]: [23/Apr/2024:14:33:42] ENGINE Serving on http://172.21.15.28:8765 2024-04-23T14:33:42.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:42 smithi028 ceph-mon[27810]: [23/Apr/2024:14:33:42] ENGINE Bus STARTED 2024-04-23T14:33:42.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:42 smithi028 ceph-mon[27810]: mgrmap e11: x(active, since 1.00982s) 2024-04-23T14:33:43.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-23T14:33:43.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-23T14:33:43.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-23T14:33:43.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-23T14:33:43.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.204+0000 7ff0af800700 1 Processor -- start 2024-04-23T14:33:43.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.204+0000 7ff0af800700 1 -- start start 2024-04-23T14:33:43.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.205+0000 7ff0af800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0a8108790 0x7ff0a8108bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:43.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.205+0000 7ff0af800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a8109180 con 0x7ff0a8108790 2024-04-23T14:33:43.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.205+0000 7ff0ace00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0a8108790 0x7ff0a8108bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:43.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.205+0000 7ff0ace00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0a8108790 0x7ff0a8108bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44118/0 (socket says 172.21.15.28:44118) 2024-04-23T14:33:43.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.205+0000 7ff0ace00700 1 -- 172.21.15.28:0/366660686 learned_addr learned my addr 172.21.15.28:0/366660686 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:43.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.206+0000 7ff0ace00700 1 -- 172.21.15.28:0/366660686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0a8109940 con 0x7ff0a8108790 2024-04-23T14:33:43.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.206+0000 7ff0ace00700 1 --2- 172.21.15.28:0/366660686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0a8108790 0x7ff0a8108bb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff09c0098a0 tx=0x7ff09c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6afd3ca4033d2965 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:43.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.206+0000 7ff0a7400700 1 -- 172.21.15.28:0/366660686 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff09c004160 con 0x7ff0a8108790 2024-04-23T14:33:43.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.206+0000 7ff0a7400700 1 -- 172.21.15.28:0/366660686 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff09c004300 con 0x7ff0a8108790 2024-04-23T14:33:43.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.206+0000 7ff0a7400700 1 -- 172.21.15.28:0/366660686 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff09c011b20 con 0x7ff0a8108790 2024-04-23T14:33:43.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.207+0000 7ff0af800700 1 -- 172.21.15.28:0/366660686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0a8108790 msgr2=0x7ff0a8108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:43.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.207+0000 7ff0af800700 1 --2- 172.21.15.28:0/366660686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0a8108790 0x7ff0a8108bb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff09c0098a0 tx=0x7ff09c009bb0 comp rx=0 tx=0).stop 2024-04-23T14:33:43.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.207+0000 7ff0af800700 1 -- 172.21.15.28:0/366660686 shutdown_connections 2024-04-23T14:33:43.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.207+0000 7ff0af800700 1 --2- 172.21.15.28:0/366660686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0a8108790 0x7ff0a8108bb0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:43.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.207+0000 7ff0af800700 1 -- 172.21.15.28:0/366660686 >> 172.21.15.28:0/366660686 conn(0x7ff0a8103d70 msgr2=0x7ff0a8106190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:43.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.207+0000 7ff0af800700 1 -- 172.21.15.28:0/366660686 shutdown_connections 2024-04-23T14:33:43.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.207+0000 7ff0af800700 1 -- 172.21.15.28:0/366660686 wait complete. 2024-04-23T14:33:43.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.208+0000 7ff0af800700 1 Processor -- start 2024-04-23T14:33:43.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.209+0000 7ff0af800700 1 -- start start 2024-04-23T14:33:43.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.209+0000 7ff0af800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0a8108790 0x7ff0a807db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:43.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.209+0000 7ff0af800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a810ccf0 con 0x7ff0a8108790 2024-04-23T14:33:43.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.209+0000 7ff0ace00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0a8108790 0x7ff0a807db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:43.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.209+0000 7ff0ace00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0a8108790 0x7ff0a807db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44122/0 (socket says 172.21.15.28:44122) 2024-04-23T14:33:43.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.209+0000 7ff0ace00700 1 -- 172.21.15.28:0/4205659298 learned_addr learned my addr 172.21.15.28:0/4205659298 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:43.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.210+0000 7ff0ace00700 1 -- 172.21.15.28:0/4205659298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0a807e0a0 con 0x7ff0a8108790 2024-04-23T14:33:43.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.210+0000 7ff0ace00700 1 --2- 172.21.15.28:0/4205659298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0a8108790 0x7ff0a807db60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff09c000c00 tx=0x7ff09c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:43.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.210+0000 7ff0a5600700 1 -- 172.21.15.28:0/4205659298 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff09c003b60 con 0x7ff0a8108790 2024-04-23T14:33:43.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.210+0000 7ff0a5600700 1 -- 172.21.15.28:0/4205659298 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff09c01a910 con 0x7ff0a8108790 2024-04-23T14:33:43.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.210+0000 7ff0af800700 1 -- 172.21.15.28:0/4205659298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0a807c290 con 0x7ff0a8108790 2024-04-23T14:33:43.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.210+0000 7ff0a5600700 1 -- 172.21.15.28:0/4205659298 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff09c024930 con 0x7ff0a8108790 2024-04-23T14:33:43.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.211+0000 7ff0af800700 1 -- 172.21.15.28:0/4205659298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0a807c750 con 0x7ff0a8108790 2024-04-23T14:33:43.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.212+0000 7ff0a5600700 1 -- 172.21.15.28:0/4205659298 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7ff09c022020 con 0x7ff0a8108790 2024-04-23T14:33:43.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.212+0000 7ff0a5600700 1 --2- 172.21.15.28:0/4205659298 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7ff090039600 0x7ff09003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:43.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.212+0000 7ff0af800700 1 -- 172.21.15.28:0/4205659298 --> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff0a8064150 con 0x7ff090039600 2024-04-23T14:33:43.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.212+0000 7ff0a5600700 1 -- 172.21.15.28:0/4205659298 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ff09c0515d0 con 0x7ff0a8108790 2024-04-23T14:33:43.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.212+0000 7ff0a7e00700 1 -- 172.21.15.28:0/4205659298 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7ff090039600 msgr2=0x7ff09003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2222038555 2024-04-23T14:33:43.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.212+0000 7ff0a7e00700 1 --2- 172.21.15.28:0/4205659298 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7ff090039600 0x7ff09003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T14:33:43.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.413+0000 7ff0a7e00700 1 -- 172.21.15.28:0/4205659298 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7ff090039600 msgr2=0x7ff09003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2222038555 2024-04-23T14:33:43.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.413+0000 7ff0a7e00700 1 --2- 172.21.15.28:0/4205659298 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7ff090039600 0x7ff09003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T14:33:43.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.814+0000 7ff0a7e00700 1 -- 172.21.15.28:0/4205659298 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7ff090039600 msgr2=0x7ff09003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2222038555 2024-04-23T14:33:43.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:35.814+0000 7ff0a7e00700 1 --2- 172.21.15.28:0/4205659298 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7ff090039600 0x7ff09003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-23T14:33:43.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:36.615+0000 7ff0a7e00700 1 -- 172.21.15.28:0/4205659298 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7ff090039600 msgr2=0x7ff09003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2222038555 2024-04-23T14:33:43.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:36.615+0000 7ff0a7e00700 1 --2- 172.21.15.28:0/4205659298 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7ff090039600 0x7ff09003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-23T14:33:43.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:38.217+0000 7ff0a7e00700 1 -- 172.21.15.28:0/4205659298 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7ff090039600 msgr2=0x7ff09003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2222038555 2024-04-23T14:33:43.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:38.217+0000 7ff0a7e00700 1 --2- 172.21.15.28:0/4205659298 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7ff090039600 0x7ff09003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-23T14:33:43.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:41.420+0000 7ff0a7e00700 1 -- 172.21.15.28:0/4205659298 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7ff090039600 msgr2=0x7ff09003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2222038555 2024-04-23T14:33:43.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:41.420+0000 7ff0a7e00700 1 --2- 172.21.15.28:0/4205659298 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7ff090039600 0x7ff09003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-23T14:33:43.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:41.534+0000 7ff0a5600700 1 -- 172.21.15.28:0/4205659298 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mgrmap(e 10) ==== 46338+0+0 (secure 0 0 0) 0x7ff09c017d50 con 0x7ff0a8108790 2024-04-23T14:33:43.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:41.535+0000 7ff0a5600700 1 -- 172.21.15.28:0/4205659298 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7ff090039600 msgr2=0x7ff09003bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T14:33:43.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:41.535+0000 7ff0a5600700 1 --2- 172.21.15.28:0/4205659298 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7ff090039600 0x7ff09003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:43.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:42.537+0000 7ff0a5600700 1 -- 172.21.15.28:0/4205659298 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7ff09c021d00 con 0x7ff0a8108790 2024-04-23T14:33:43.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:42.538+0000 7ff0a5600700 1 --2- 172.21.15.28:0/4205659298 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ff09003ce90 0x7ff09003f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:43.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:42.538+0000 7ff0a5600700 1 -- 172.21.15.28:0/4205659298 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff09c007da0 con 0x7ff09003ce90 2024-04-23T14:33:43.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:42.538+0000 7ff0a7e00700 1 --2- 172.21.15.28:0/4205659298 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ff09003ce90 0x7ff09003f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:43.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:42.540+0000 7ff0a7e00700 1 --2- 172.21.15.28:0/4205659298 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ff09003ce90 0x7ff09003f280 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff094003a10 tx=0x7ff0940092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:43.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:42.541+0000 7ff0a5600700 1 -- 172.21.15.28:0/4205659298 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7ff09c007da0 con 0x7ff09003ce90 2024-04-23T14:33:43.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:42.555+0000 7ff0af800700 1 -- 172.21.15.28:0/4205659298 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7ff0a8104f70 con 0x7ff09003ce90 2024-04-23T14:33:43.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:42.555+0000 7ff0a5600700 1 -- 172.21.15.28:0/4205659298 <== mgr.14152 v2:172.21.15.28:6800/1997136173 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7ff0a8104f70 con 0x7ff09003ce90 2024-04-23T14:33:43.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:42.556+0000 7ff0af800700 1 -- 172.21.15.28:0/4205659298 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ff09003ce90 msgr2=0x7ff09003f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:43.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:42.556+0000 7ff0af800700 1 --2- 172.21.15.28:0/4205659298 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ff09003ce90 0x7ff09003f280 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff094003a10 tx=0x7ff0940092b0 comp rx=0 tx=0).stop 2024-04-23T14:33:43.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:42.556+0000 7ff0af800700 1 -- 172.21.15.28:0/4205659298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0a8108790 msgr2=0x7ff0a807db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:43.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:42.556+0000 7ff0af800700 1 --2- 172.21.15.28:0/4205659298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0a8108790 0x7ff0a807db60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff09c000c00 tx=0x7ff09c011c10 comp rx=0 tx=0).stop 2024-04-23T14:33:43.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:42.556+0000 7ff0af800700 1 -- 172.21.15.28:0/4205659298 shutdown_connections 2024-04-23T14:33:43.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:42.556+0000 7ff0af800700 1 --2- 172.21.15.28:0/4205659298 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ff09003ce90 0x7ff09003f280 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:43.335 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:42.556+0000 7ff0af800700 1 --2- 172.21.15.28:0/4205659298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0a8108790 0x7ff0a807db60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:43.335 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:42.556+0000 7ff0af800700 1 --2- 172.21.15.28:0/4205659298 >> [v2:172.21.15.28:6800/2222038555,v1:172.21.15.28:6801/2222038555] conn(0x7ff090039600 0x7ff09003bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:43.335 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:42.556+0000 7ff0af800700 1 -- 172.21.15.28:0/4205659298 >> 172.21.15.28:0/4205659298 conn(0x7ff0a8103d70 msgr2=0x7ff0a8104860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:43.335 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:42.556+0000 7ff0af800700 1 -- 172.21.15.28:0/4205659298 shutdown_connections 2024-04-23T14:33:43.335 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:42.556+0000 7ff0af800700 1 -- 172.21.15.28:0/4205659298 wait complete. 2024-04-23T14:33:43.335 INFO:teuthology.orchestra.run.smithi028.stdout:mgr epoch 9 is available 2024-04-23T14:33:43.335 INFO:teuthology.orchestra.run.smithi028.stdout:Generating a dashboard self-signed certificate... 2024-04-23T14:33:43.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:43 smithi028 ceph-mon[27810]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-23T14:33:43.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:43 smithi028 ceph-mon[27810]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-23T14:33:44.766 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:44 smithi028 ceph-mon[27810]: mgrmap e12: x(active, since 2s) 2024-04-23T14:33:44.766 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:44 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:33:45.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-23T14:33:45.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.512+0000 7f139b400700 1 Processor -- start 2024-04-23T14:33:45.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.512+0000 7f139b400700 1 -- start start 2024-04-23T14:33:45.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.513+0000 7f139b400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1394108740 0x7f1394108b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:45.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.513+0000 7f139b400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1394109130 con 0x7f1394108740 2024-04-23T14:33:45.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.513+0000 7f1398a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1394108740 0x7f1394108b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:45.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.513+0000 7f1398a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1394108740 0x7f1394108b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58626/0 (socket says 172.21.15.28:58626) 2024-04-23T14:33:45.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.514+0000 7f1398a00700 1 -- 172.21.15.28:0/3235038744 learned_addr learned my addr 172.21.15.28:0/3235038744 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:45.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.514+0000 7f1398a00700 1 -- 172.21.15.28:0/3235038744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13941098f0 con 0x7f1394108740 2024-04-23T14:33:45.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.514+0000 7f1398a00700 1 --2- 172.21.15.28:0/3235038744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1394108740 0x7f1394108b60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f13880098a0 tx=0x7f1388009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=108902aee3faa633 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:45.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.515+0000 7f1393400700 1 -- 172.21.15.28:0/3235038744 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1388004160 con 0x7f1394108740 2024-04-23T14:33:45.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.515+0000 7f1393400700 1 -- 172.21.15.28:0/3235038744 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1388004300 con 0x7f1394108740 2024-04-23T14:33:45.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.515+0000 7f1393400700 1 -- 172.21.15.28:0/3235038744 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1388011b20 con 0x7f1394108740 2024-04-23T14:33:45.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.515+0000 7f139b400700 1 -- 172.21.15.28:0/3235038744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1394108740 msgr2=0x7f1394108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:45.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.515+0000 7f139b400700 1 --2- 172.21.15.28:0/3235038744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1394108740 0x7f1394108b60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f13880098a0 tx=0x7f1388009bb0 comp rx=0 tx=0).stop 2024-04-23T14:33:45.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.516+0000 7f139b400700 1 -- 172.21.15.28:0/3235038744 shutdown_connections 2024-04-23T14:33:45.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.516+0000 7f139b400700 1 --2- 172.21.15.28:0/3235038744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1394108740 0x7f1394108b60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:45.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.516+0000 7f139b400700 1 -- 172.21.15.28:0/3235038744 >> 172.21.15.28:0/3235038744 conn(0x7f1394103d40 msgr2=0x7f1394106160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:45.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.516+0000 7f139b400700 1 -- 172.21.15.28:0/3235038744 shutdown_connections 2024-04-23T14:33:45.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.516+0000 7f139b400700 1 -- 172.21.15.28:0/3235038744 wait complete. 2024-04-23T14:33:45.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.517+0000 7f139b400700 1 Processor -- start 2024-04-23T14:33:45.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.517+0000 7f139b400700 1 -- start start 2024-04-23T14:33:45.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.517+0000 7f139b400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1394108740 0x7f139407db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:45.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.517+0000 7f139b400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f139410cca0 con 0x7f1394108740 2024-04-23T14:33:45.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.518+0000 7f1398a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1394108740 0x7f139407db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:45.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.518+0000 7f1398a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1394108740 0x7f139407db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58632/0 (socket says 172.21.15.28:58632) 2024-04-23T14:33:45.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.518+0000 7f1398a00700 1 -- 172.21.15.28:0/299479019 learned_addr learned my addr 172.21.15.28:0/299479019 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:45.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.518+0000 7f1398a00700 1 -- 172.21.15.28:0/299479019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f139407e0a0 con 0x7f1394108740 2024-04-23T14:33:45.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.518+0000 7f1398a00700 1 --2- 172.21.15.28:0/299479019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1394108740 0x7f139407db60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f13880116a0 tx=0x7f1388011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:45.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.519+0000 7f1391600700 1 -- 172.21.15.28:0/299479019 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1388003a40 con 0x7f1394108740 2024-04-23T14:33:45.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.519+0000 7f1391600700 1 -- 172.21.15.28:0/299479019 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1388003be0 con 0x7f1394108740 2024-04-23T14:33:45.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.519+0000 7f1391600700 1 -- 172.21.15.28:0/299479019 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f138801a720 con 0x7f1394108740 2024-04-23T14:33:45.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.519+0000 7f139b400700 1 -- 172.21.15.28:0/299479019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f139407c290 con 0x7f1394108740 2024-04-23T14:33:45.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.519+0000 7f139b400700 1 -- 172.21.15.28:0/299479019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f139407c750 con 0x7f1394108740 2024-04-23T14:33:45.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.520+0000 7f139b400700 1 -- 172.21.15.28:0/299479019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1394064150 con 0x7f1394108740 2024-04-23T14:33:45.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.520+0000 7f1391600700 1 -- 172.21.15.28:0/299479019 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f1388024aa0 con 0x7f1394108740 2024-04-23T14:33:45.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.520+0000 7f1391600700 1 --2- 172.21.15.28:0/299479019 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f137c039610 0x7f137c03bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:45.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.521+0000 7f1391600700 1 -- 172.21.15.28:0/299479019 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f138804de00 con 0x7f1394108740 2024-04-23T14:33:45.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.521+0000 7f1393e00700 1 --2- 172.21.15.28:0/299479019 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f137c039610 0x7f137c03bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:45.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.522+0000 7f1393e00700 1 --2- 172.21.15.28:0/299479019 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f137c039610 0x7f137c03bad0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1384006fd0 tx=0x7f1384006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:45.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.526+0000 7f1391600700 1 -- 172.21.15.28:0/299479019 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1388011e20 con 0x7f1394108740 2024-04-23T14:33:45.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.765+0000 7f139b400700 1 -- 172.21.15.28:0/299479019 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f139407ce80 con 0x7f137c039610 2024-04-23T14:33:45.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.936+0000 7f1391600700 1 -- 172.21.15.28:0/299479019 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f139407ce80 con 0x7f137c039610 2024-04-23T14:33:45.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.939+0000 7f139b400700 1 -- 172.21.15.28:0/299479019 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f137c039610 msgr2=0x7f137c03bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:45.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.939+0000 7f139b400700 1 --2- 172.21.15.28:0/299479019 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f137c039610 0x7f137c03bad0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1384006fd0 tx=0x7f1384006e40 comp rx=0 tx=0).stop 2024-04-23T14:33:45.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.939+0000 7f139b400700 1 -- 172.21.15.28:0/299479019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1394108740 msgr2=0x7f139407db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:45.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.939+0000 7f139b400700 1 --2- 172.21.15.28:0/299479019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1394108740 0x7f139407db60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f13880116a0 tx=0x7f1388011ba0 comp rx=0 tx=0).stop 2024-04-23T14:33:45.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.939+0000 7f139b400700 1 -- 172.21.15.28:0/299479019 shutdown_connections 2024-04-23T14:33:45.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.939+0000 7f139b400700 1 --2- 172.21.15.28:0/299479019 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f137c039610 0x7f137c03bad0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:45.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.939+0000 7f139b400700 1 --2- 172.21.15.28:0/299479019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1394108740 0x7f139407db60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:45.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.939+0000 7f139b400700 1 -- 172.21.15.28:0/299479019 >> 172.21.15.28:0/299479019 conn(0x7f1394103d40 msgr2=0x7f13941048a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:45.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.939+0000 7f139b400700 1 -- 172.21.15.28:0/299479019 shutdown_connections 2024-04-23T14:33:45.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:44.939+0000 7f139b400700 1 -- 172.21.15.28:0/299479019 wait complete. 2024-04-23T14:33:45.669 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial admin user... 2024-04-23T14:33:46.250 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:45 smithi028 ceph-mon[27810]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:33:46.251 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:45 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:33:46.251 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:45 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:33:47.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:47 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:33:47.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:47 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:33:47.626 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$Cjf.vElIIIbORZM5j8dxzOslOQkbNkmG/5GTuI4ZTk7YwBq4fcEQy", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713882826, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-23T14:33:47.626 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.358+0000 7fc221800700 1 Processor -- start 2024-04-23T14:33:47.626 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.358+0000 7fc221800700 1 -- start start 2024-04-23T14:33:47.626 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.359+0000 7fc221800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc21c10f0a0 0x7fc21c10f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:47.626 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.359+0000 7fc221800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc21c10fa90 con 0x7fc21c10f0a0 2024-04-23T14:33:47.626 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.359+0000 7fc21aa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc21c10f0a0 0x7fc21c10f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:47.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.359+0000 7fc21aa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc21c10f0a0 0x7fc21c10f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58638/0 (socket says 172.21.15.28:58638) 2024-04-23T14:33:47.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.359+0000 7fc21aa00700 1 -- 172.21.15.28:0/2362904276 learned_addr learned my addr 172.21.15.28:0/2362904276 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:47.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.360+0000 7fc21aa00700 1 -- 172.21.15.28:0/2362904276 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc21c10ff00 con 0x7fc21c10f0a0 2024-04-23T14:33:47.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.360+0000 7fc21aa00700 1 --2- 172.21.15.28:0/2362904276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc21c10f0a0 0x7fc21c10f4c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc20c0098a0 tx=0x7fc20c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74ef0f93a214bb53 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:47.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.360+0000 7fc219600700 1 -- 172.21.15.28:0/2362904276 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc20c004160 con 0x7fc21c10f0a0 2024-04-23T14:33:47.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.360+0000 7fc219600700 1 -- 172.21.15.28:0/2362904276 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc20c004300 con 0x7fc21c10f0a0 2024-04-23T14:33:47.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.360+0000 7fc219600700 1 -- 172.21.15.28:0/2362904276 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc20c011b20 con 0x7fc21c10f0a0 2024-04-23T14:33:47.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.361+0000 7fc221800700 1 -- 172.21.15.28:0/2362904276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc21c10f0a0 msgr2=0x7fc21c10f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:47.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.361+0000 7fc221800700 1 --2- 172.21.15.28:0/2362904276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc21c10f0a0 0x7fc21c10f4c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc20c0098a0 tx=0x7fc20c009bb0 comp rx=0 tx=0).stop 2024-04-23T14:33:47.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.361+0000 7fc221800700 1 -- 172.21.15.28:0/2362904276 shutdown_connections 2024-04-23T14:33:47.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.361+0000 7fc221800700 1 --2- 172.21.15.28:0/2362904276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc21c10f0a0 0x7fc21c10f4c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:47.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.361+0000 7fc221800700 1 -- 172.21.15.28:0/2362904276 >> 172.21.15.28:0/2362904276 conn(0x7fc21c07ddd0 msgr2=0x7fc21c0801f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:47.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.361+0000 7fc221800700 1 -- 172.21.15.28:0/2362904276 shutdown_connections 2024-04-23T14:33:47.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.361+0000 7fc221800700 1 -- 172.21.15.28:0/2362904276 wait complete. 2024-04-23T14:33:47.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.362+0000 7fc221800700 1 Processor -- start 2024-04-23T14:33:47.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.363+0000 7fc221800700 1 -- start start 2024-04-23T14:33:47.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.363+0000 7fc221800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc21c10f0a0 0x7fc21c1a2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:47.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.363+0000 7fc221800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc21c1132b0 con 0x7fc21c10f0a0 2024-04-23T14:33:47.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.363+0000 7fc21aa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc21c10f0a0 0x7fc21c1a2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:47.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.363+0000 7fc21aa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc21c10f0a0 0x7fc21c1a2bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58640/0 (socket says 172.21.15.28:58640) 2024-04-23T14:33:47.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.364+0000 7fc21aa00700 1 -- 172.21.15.28:0/3732668334 learned_addr learned my addr 172.21.15.28:0/3732668334 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:47.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.364+0000 7fc21aa00700 1 -- 172.21.15.28:0/3732668334 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc21c1a3100 con 0x7fc21c10f0a0 2024-04-23T14:33:47.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.364+0000 7fc21aa00700 1 --2- 172.21.15.28:0/3732668334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc21c10f0a0 0x7fc21c1a2bc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc20c0117e0 tx=0x7fc20c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:47.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.364+0000 7fc213400700 1 -- 172.21.15.28:0/3732668334 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc20c003bf0 con 0x7fc21c10f0a0 2024-04-23T14:33:47.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.364+0000 7fc213400700 1 -- 172.21.15.28:0/3732668334 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc20c01a910 con 0x7fc21c10f0a0 2024-04-23T14:33:47.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.364+0000 7fc213400700 1 -- 172.21.15.28:0/3732668334 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc20c024970 con 0x7fc21c10f0a0 2024-04-23T14:33:47.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.365+0000 7fc221800700 1 -- 172.21.15.28:0/3732668334 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc21c1a3390 con 0x7fc21c10f0a0 2024-04-23T14:33:47.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.365+0000 7fc221800700 1 -- 172.21.15.28:0/3732668334 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc21c1a37f0 con 0x7fc21c10f0a0 2024-04-23T14:33:47.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.366+0000 7fc221800700 1 -- 172.21.15.28:0/3732668334 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc21c064150 con 0x7fc21c10f0a0 2024-04-23T14:33:47.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.366+0000 7fc213400700 1 -- 172.21.15.28:0/3732668334 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7fc20c022020 con 0x7fc21c10f0a0 2024-04-23T14:33:47.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.367+0000 7fc213400700 1 --2- 172.21.15.28:0/3732668334 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fc200039660 0x7fc20003bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:47.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.367+0000 7fc213400700 1 -- 172.21.15.28:0/3732668334 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc20c04dc10 con 0x7fc21c10f0a0 2024-04-23T14:33:47.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.367+0000 7fc21a000700 1 --2- 172.21.15.28:0/3732668334 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fc200039660 0x7fc20003bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:47.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.368+0000 7fc21a000700 1 --2- 172.21.15.28:0/3732668334 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fc200039660 0x7fc20003bb20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc208006fd0 tx=0x7fc208006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:47.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.373+0000 7fc213400700 1 -- 172.21.15.28:0/3732668334 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc20c0212c0 con 0x7fc21c10f0a0 2024-04-23T14:33:47.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.627+0000 7fc221800700 1 -- 172.21.15.28:0/3732668334 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7fc21c1a6200 con 0x7fc200039660 2024-04-23T14:33:47.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.899+0000 7fc213400700 1 -- 172.21.15.28:0/3732668334 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7fc21c1a6200 con 0x7fc200039660 2024-04-23T14:33:47.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.902+0000 7fc221800700 1 -- 172.21.15.28:0/3732668334 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fc200039660 msgr2=0x7fc20003bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:47.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.902+0000 7fc221800700 1 --2- 172.21.15.28:0/3732668334 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fc200039660 0x7fc20003bb20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc208006fd0 tx=0x7fc208006e40 comp rx=0 tx=0).stop 2024-04-23T14:33:47.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.902+0000 7fc221800700 1 -- 172.21.15.28:0/3732668334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc21c10f0a0 msgr2=0x7fc21c1a2bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:47.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.902+0000 7fc221800700 1 --2- 172.21.15.28:0/3732668334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc21c10f0a0 0x7fc21c1a2bc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc20c0117e0 tx=0x7fc20c011ba0 comp rx=0 tx=0).stop 2024-04-23T14:33:47.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.902+0000 7fc221800700 1 -- 172.21.15.28:0/3732668334 shutdown_connections 2024-04-23T14:33:47.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.902+0000 7fc221800700 1 --2- 172.21.15.28:0/3732668334 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fc200039660 0x7fc20003bb20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:47.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.902+0000 7fc221800700 1 --2- 172.21.15.28:0/3732668334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc21c10f0a0 0x7fc21c1a2bc0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:47.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.902+0000 7fc221800700 1 -- 172.21.15.28:0/3732668334 >> 172.21.15.28:0/3732668334 conn(0x7fc21c07ddd0 msgr2=0x7fc21c07e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:47.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.903+0000 7fc221800700 1 -- 172.21.15.28:0/3732668334 shutdown_connections 2024-04-23T14:33:47.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:46.903+0000 7fc221800700 1 -- 172.21.15.28:0/3732668334 wait complete. 2024-04-23T14:33:47.633 INFO:teuthology.orchestra.run.smithi028.stdout:Fetching dashboard port number... 2024-04-23T14:33:48.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:48 smithi028 ceph-mon[27810]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:33:49.387 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:49 smithi028 ceph-mon[27810]: mgrmap e13: x(active, since 6s) 2024-04-23T14:33:49.387 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:49 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1856952118' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-23T14:33:49.901 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 8443 2024-04-23T14:33:49.901 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.319+0000 7f1701a00700 1 Processor -- start 2024-04-23T14:33:49.901 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.319+0000 7f1701a00700 1 -- start start 2024-04-23T14:33:49.901 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.320+0000 7f1701a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16fc108790 0x7f16fc108bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:49.901 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.320+0000 7f1701a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16fc109180 con 0x7f16fc108790 2024-04-23T14:33:49.901 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.320+0000 7f16faa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16fc108790 0x7f16fc108bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:49.901 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.320+0000 7f16faa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16fc108790 0x7f16fc108bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39992/0 (socket says 172.21.15.28:39992) 2024-04-23T14:33:49.901 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.320+0000 7f16faa00700 1 -- 172.21.15.28:0/3436458200 learned_addr learned my addr 172.21.15.28:0/3436458200 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:49.901 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.321+0000 7f16faa00700 1 -- 172.21.15.28:0/3436458200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16fc109940 con 0x7f16fc108790 2024-04-23T14:33:49.901 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.321+0000 7f16faa00700 1 --2- 172.21.15.28:0/3436458200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16fc108790 0x7f16fc108bb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f16ec0098a0 tx=0x7f16ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ea17e42fe4c3008 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:49.906 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.321+0000 7f16f9600700 1 -- 172.21.15.28:0/3436458200 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16ec004160 con 0x7f16fc108790 2024-04-23T14:33:49.906 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.321+0000 7f16f9600700 1 -- 172.21.15.28:0/3436458200 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16ec004300 con 0x7f16fc108790 2024-04-23T14:33:49.906 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.321+0000 7f16f9600700 1 -- 172.21.15.28:0/3436458200 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16ec011b20 con 0x7f16fc108790 2024-04-23T14:33:49.906 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.322+0000 7f1701a00700 1 -- 172.21.15.28:0/3436458200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16fc108790 msgr2=0x7f16fc108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:49.906 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.322+0000 7f1701a00700 1 --2- 172.21.15.28:0/3436458200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16fc108790 0x7f16fc108bb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f16ec0098a0 tx=0x7f16ec009bb0 comp rx=0 tx=0).stop 2024-04-23T14:33:49.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.322+0000 7f1701a00700 1 -- 172.21.15.28:0/3436458200 shutdown_connections 2024-04-23T14:33:49.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.322+0000 7f1701a00700 1 --2- 172.21.15.28:0/3436458200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16fc108790 0x7f16fc108bb0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:49.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.322+0000 7f1701a00700 1 -- 172.21.15.28:0/3436458200 >> 172.21.15.28:0/3436458200 conn(0x7f16fc103d70 msgr2=0x7f16fc106190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:49.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.322+0000 7f1701a00700 1 -- 172.21.15.28:0/3436458200 shutdown_connections 2024-04-23T14:33:49.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.322+0000 7f1701a00700 1 -- 172.21.15.28:0/3436458200 wait complete. 2024-04-23T14:33:49.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.323+0000 7f1701a00700 1 Processor -- start 2024-04-23T14:33:49.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.323+0000 7f1701a00700 1 -- start start 2024-04-23T14:33:49.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.324+0000 7f1701a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16fc108790 0x7f16fc19e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:49.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.324+0000 7f1701a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16fc10ccf0 con 0x7f16fc108790 2024-04-23T14:33:49.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.324+0000 7f16faa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16fc108790 0x7f16fc19e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:49.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.324+0000 7f16faa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16fc108790 0x7f16fc19e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39996/0 (socket says 172.21.15.28:39996) 2024-04-23T14:33:49.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.324+0000 7f16faa00700 1 -- 172.21.15.28:0/1856952118 learned_addr learned my addr 172.21.15.28:0/1856952118 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:49.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.324+0000 7f16faa00700 1 -- 172.21.15.28:0/1856952118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16fc19eda0 con 0x7f16fc108790 2024-04-23T14:33:49.908 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.325+0000 7f16faa00700 1 --2- 172.21.15.28:0/1856952118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16fc108790 0x7f16fc19e860 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f16ec000c00 tx=0x7f16ec011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:49.908 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.325+0000 7f16f3400700 1 -- 172.21.15.28:0/1856952118 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16ec003b60 con 0x7f16fc108790 2024-04-23T14:33:49.908 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.325+0000 7f1701a00700 1 -- 172.21.15.28:0/1856952118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16fc19f030 con 0x7f16fc108790 2024-04-23T14:33:49.908 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.325+0000 7f16f3400700 1 -- 172.21.15.28:0/1856952118 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16ec01a910 con 0x7f16fc108790 2024-04-23T14:33:49.908 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.325+0000 7f16f3400700 1 -- 172.21.15.28:0/1856952118 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16ec024930 con 0x7f16fc108790 2024-04-23T14:33:49.908 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.325+0000 7f1701a00700 1 -- 172.21.15.28:0/1856952118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16fc19f490 con 0x7f16fc108790 2024-04-23T14:33:49.908 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.326+0000 7f16f3400700 1 -- 172.21.15.28:0/1856952118 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f16ec022020 con 0x7f16fc108790 2024-04-23T14:33:49.908 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.326+0000 7f1701a00700 1 -- 172.21.15.28:0/1856952118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16fc064150 con 0x7f16fc108790 2024-04-23T14:33:49.908 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.327+0000 7f16f3400700 1 --2- 172.21.15.28:0/1856952118 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f16e0039660 0x7f16e003bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:49.908 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.327+0000 7f16f3400700 1 -- 172.21.15.28:0/1856952118 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f16ec04da50 con 0x7f16fc108790 2024-04-23T14:33:49.908 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.327+0000 7f16fa000700 1 --2- 172.21.15.28:0/1856952118 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f16e0039660 0x7f16e003bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:49.908 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.328+0000 7f16fa000700 1 --2- 172.21.15.28:0/1856952118 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f16e0039660 0x7f16e003bb20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f16e8006fd0 tx=0x7f16e8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:49.908 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.333+0000 7f16f3400700 1 -- 172.21.15.28:0/1856952118 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f16ec00f2e0 con 0x7f16fc108790 2024-04-23T14:33:49.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.381+0000 7f16f3400700 1 -- 172.21.15.28:0/1856952118 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f16ec02d3f0 con 0x7f16fc108790 2024-04-23T14:33:49.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.567+0000 7f1701a00700 1 -- 172.21.15.28:0/1856952118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f16fc1a1e60 con 0x7f16fc108790 2024-04-23T14:33:49.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.568+0000 7f16f3400700 1 -- 172.21.15.28:0/1856952118 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f16ec02be10 con 0x7f16fc108790 2024-04-23T14:33:49.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.571+0000 7f1701a00700 1 -- 172.21.15.28:0/1856952118 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f16e0039660 msgr2=0x7f16e003bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:49.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.571+0000 7f1701a00700 1 --2- 172.21.15.28:0/1856952118 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f16e0039660 0x7f16e003bb20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f16e8006fd0 tx=0x7f16e8006e40 comp rx=0 tx=0).stop 2024-04-23T14:33:49.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.571+0000 7f1701a00700 1 -- 172.21.15.28:0/1856952118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16fc108790 msgr2=0x7f16fc19e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:49.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.571+0000 7f1701a00700 1 --2- 172.21.15.28:0/1856952118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16fc108790 0x7f16fc19e860 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f16ec000c00 tx=0x7f16ec011c10 comp rx=0 tx=0).stop 2024-04-23T14:33:49.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.571+0000 7f1701a00700 1 -- 172.21.15.28:0/1856952118 shutdown_connections 2024-04-23T14:33:49.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.571+0000 7f1701a00700 1 --2- 172.21.15.28:0/1856952118 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f16e0039660 0x7f16e003bb20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:49.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.571+0000 7f1701a00700 1 --2- 172.21.15.28:0/1856952118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16fc108790 0x7f16fc19e860 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:49.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.571+0000 7f1701a00700 1 -- 172.21.15.28:0/1856952118 >> 172.21.15.28:0/1856952118 conn(0x7f16fc103d70 msgr2=0x7f16fc1047d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:49.910 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.571+0000 7f1701a00700 1 -- 172.21.15.28:0/1856952118 shutdown_connections 2024-04-23T14:33:49.910 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:48.571+0000 7f1701a00700 1 -- 172.21.15.28:0/1856952118 wait complete. 2024-04-23T14:33:49.947 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-23T14:33:49.947 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-04-23T14:33:49.975 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-23T14:33:49.975 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-04-23T14:33:49.976 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-04-23T14:33:49.976 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-23T14:33:49.984 INFO:teuthology.orchestra.run.smithi028.stdout:Ceph Dashboard is now available at: 2024-04-23T14:33:49.984 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:33:49.984 INFO:teuthology.orchestra.run.smithi028.stdout: URL: https://smithi028.front.sepia.ceph.com:8443/ 2024-04-23T14:33:49.984 INFO:teuthology.orchestra.run.smithi028.stdout: User: admin 2024-04-23T14:33:49.984 INFO:teuthology.orchestra.run.smithi028.stdout: Password: 9dzk58qwfz 2024-04-23T14:33:49.984 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:33:49.985 INFO:teuthology.orchestra.run.smithi028.stdout:Saving cluster configuration to /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config directory 2024-04-23T14:33:49.987 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling autotune for osd_memory_target 2024-04-23T14:33:51.600 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.583+0000 7f8b57000700 1 Processor -- start 2024-04-23T14:33:51.600 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.583+0000 7f8b57000700 1 -- start start 2024-04-23T14:33:51.600 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.583+0000 7f8b57000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b5010f0a0 0x7f8b5010f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:51.600 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.583+0000 7f8b57000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8b5010fa90 con 0x7f8b5010f0a0 2024-04-23T14:33:51.601 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.584+0000 7f8b4fe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b5010f0a0 0x7f8b5010f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:51.601 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.584+0000 7f8b4fe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b5010f0a0 0x7f8b5010f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40002/0 (socket says 172.21.15.28:40002) 2024-04-23T14:33:51.601 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.584+0000 7f8b4fe00700 1 -- 172.21.15.28:0/1248465623 learned_addr learned my addr 172.21.15.28:0/1248465623 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:51.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.585+0000 7f8b4fe00700 1 -- 172.21.15.28:0/1248465623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b5010fec0 con 0x7f8b5010f0a0 2024-04-23T14:33:51.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.585+0000 7f8b4fe00700 1 --2- 172.21.15.28:0/1248465623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b5010f0a0 0x7f8b5010f4c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8b400098a0 tx=0x7f8b40009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dbadcf8f7191430 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:51.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.585+0000 7f8b4ea00700 1 -- 172.21.15.28:0/1248465623 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b40004160 con 0x7f8b5010f0a0 2024-04-23T14:33:51.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.585+0000 7f8b4ea00700 1 -- 172.21.15.28:0/1248465623 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b40004300 con 0x7f8b5010f0a0 2024-04-23T14:33:51.603 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.585+0000 7f8b4ea00700 1 -- 172.21.15.28:0/1248465623 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b40011b20 con 0x7f8b5010f0a0 2024-04-23T14:33:51.603 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.586+0000 7f8b57000700 1 -- 172.21.15.28:0/1248465623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b5010f0a0 msgr2=0x7f8b5010f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:51.603 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.586+0000 7f8b57000700 1 --2- 172.21.15.28:0/1248465623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b5010f0a0 0x7f8b5010f4c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8b400098a0 tx=0x7f8b40009bb0 comp rx=0 tx=0).stop 2024-04-23T14:33:51.603 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.586+0000 7f8b57000700 1 -- 172.21.15.28:0/1248465623 shutdown_connections 2024-04-23T14:33:51.603 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.586+0000 7f8b57000700 1 --2- 172.21.15.28:0/1248465623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b5010f0a0 0x7f8b5010f4c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:51.603 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.586+0000 7f8b57000700 1 -- 172.21.15.28:0/1248465623 >> 172.21.15.28:0/1248465623 conn(0x7f8b5007ddd0 msgr2=0x7f8b50080210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:51.603 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.587+0000 7f8b57000700 1 -- 172.21.15.28:0/1248465623 shutdown_connections 2024-04-23T14:33:51.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.587+0000 7f8b57000700 1 -- 172.21.15.28:0/1248465623 wait complete. 2024-04-23T14:33:51.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.588+0000 7f8b57000700 1 Processor -- start 2024-04-23T14:33:51.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.588+0000 7f8b57000700 1 -- start start 2024-04-23T14:33:51.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.588+0000 7f8b57000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b5010f0a0 0x7f8b501a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:51.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.588+0000 7f8b57000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8b50113270 con 0x7f8b5010f0a0 2024-04-23T14:33:51.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.589+0000 7f8b4fe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b5010f0a0 0x7f8b501a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:51.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.589+0000 7f8b4fe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b5010f0a0 0x7f8b501a2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40012/0 (socket says 172.21.15.28:40012) 2024-04-23T14:33:51.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.589+0000 7f8b4fe00700 1 -- 172.21.15.28:0/1632698897 learned_addr learned my addr 172.21.15.28:0/1632698897 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:51.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.589+0000 7f8b4fe00700 1 -- 172.21.15.28:0/1632698897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b501a3180 con 0x7f8b5010f0a0 2024-04-23T14:33:51.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.589+0000 7f8b4fe00700 1 --2- 172.21.15.28:0/1632698897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b5010f0a0 0x7f8b501a2c40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8b40000c00 tx=0x7f8b40011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:51.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.590+0000 7f8b4cc00700 1 -- 172.21.15.28:0/1632698897 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b40003b60 con 0x7f8b5010f0a0 2024-04-23T14:33:51.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.590+0000 7f8b4cc00700 1 -- 172.21.15.28:0/1632698897 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b4001a910 con 0x7f8b5010f0a0 2024-04-23T14:33:51.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.590+0000 7f8b57000700 1 -- 172.21.15.28:0/1632698897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b501a3410 con 0x7f8b5010f0a0 2024-04-23T14:33:51.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.590+0000 7f8b4cc00700 1 -- 172.21.15.28:0/1632698897 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b40024930 con 0x7f8b5010f0a0 2024-04-23T14:33:51.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.590+0000 7f8b57000700 1 -- 172.21.15.28:0/1632698897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b501a3870 con 0x7f8b5010f0a0 2024-04-23T14:33:51.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.591+0000 7f8b4cc00700 1 -- 172.21.15.28:0/1632698897 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f8b40022020 con 0x7f8b5010f0a0 2024-04-23T14:33:51.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.591+0000 7f8b57000700 1 -- 172.21.15.28:0/1632698897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b50064150 con 0x7f8b5010f0a0 2024-04-23T14:33:51.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.591+0000 7f8b4cc00700 1 --2- 172.21.15.28:0/1632698897 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f8b34039700 0x7f8b3403bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:51.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.591+0000 7f8b4cc00700 1 -- 172.21.15.28:0/1632698897 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8b4004db00 con 0x7f8b5010f0a0 2024-04-23T14:33:51.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.592+0000 7f8b4f400700 1 --2- 172.21.15.28:0/1632698897 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f8b34039700 0x7f8b3403bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:51.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.592+0000 7f8b4f400700 1 --2- 172.21.15.28:0/1632698897 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f8b34039700 0x7f8b3403bbc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c006fd0 tx=0x7f8b3c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:51.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.597+0000 7f8b4cc00700 1 -- 172.21.15.28:0/1632698897 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8b4004fe20 con 0x7f8b5010f0a0 2024-04-23T14:33:51.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.832+0000 7f8b57000700 1 -- 172.21.15.28:0/1632698897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f8b501a62e0 con 0x7f8b5010f0a0 2024-04-23T14:33:51.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.832+0000 7f8b4cc00700 1 -- 172.21.15.28:0/1632698897 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f8b4002a2a0 con 0x7f8b5010f0a0 2024-04-23T14:33:51.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.837+0000 7f8b57000700 1 -- 172.21.15.28:0/1632698897 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f8b34039700 msgr2=0x7f8b3403bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:51.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.837+0000 7f8b57000700 1 --2- 172.21.15.28:0/1632698897 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f8b34039700 0x7f8b3403bbc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c006fd0 tx=0x7f8b3c006e40 comp rx=0 tx=0).stop 2024-04-23T14:33:51.610 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.837+0000 7f8b57000700 1 -- 172.21.15.28:0/1632698897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b5010f0a0 msgr2=0x7f8b501a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:51.611 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.837+0000 7f8b57000700 1 --2- 172.21.15.28:0/1632698897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b5010f0a0 0x7f8b501a2c40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8b40000c00 tx=0x7f8b40011c10 comp rx=0 tx=0).stop 2024-04-23T14:33:51.611 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.837+0000 7f8b57000700 1 -- 172.21.15.28:0/1632698897 shutdown_connections 2024-04-23T14:33:51.611 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.837+0000 7f8b57000700 1 --2- 172.21.15.28:0/1632698897 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f8b34039700 0x7f8b3403bbc0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:51.611 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.837+0000 7f8b57000700 1 --2- 172.21.15.28:0/1632698897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b5010f0a0 0x7f8b501a2c40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:51.611 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.837+0000 7f8b57000700 1 -- 172.21.15.28:0/1632698897 >> 172.21.15.28:0/1632698897 conn(0x7f8b5007ddd0 msgr2=0x7f8b5007f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:51.611 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.838+0000 7f8b57000700 1 -- 172.21.15.28:0/1632698897 shutdown_connections 2024-04-23T14:33:51.611 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:50.838+0000 7f8b57000700 1 -- 172.21.15.28:0/1632698897 wait complete. 2024-04-23T14:33:53.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.216+0000 7f650d000700 1 Processor -- start 2024-04-23T14:33:53.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.216+0000 7f650d000700 1 -- start start 2024-04-23T14:33:53.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.217+0000 7f650d000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f650810a990 0x7f650810adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:53.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.217+0000 7f650d000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f650810b380 con 0x7f650810a990 2024-04-23T14:33:53.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.217+0000 7f6505e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f650810a990 0x7f650810adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:53.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.217+0000 7f6505e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f650810a990 0x7f650810adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40018/0 (socket says 172.21.15.28:40018) 2024-04-23T14:33:53.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.217+0000 7f6505e00700 1 -- 172.21.15.28:0/3640060771 learned_addr learned my addr 172.21.15.28:0/3640060771 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:53.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.218+0000 7f6505e00700 1 -- 172.21.15.28:0/3640060771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f650810bb40 con 0x7f650810a990 2024-04-23T14:33:53.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.218+0000 7f6505e00700 1 --2- 172.21.15.28:0/3640060771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f650810a990 0x7f650810adb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f64f80098a0 tx=0x7f64f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c9491b549825def server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:53.257 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.218+0000 7f6504a00700 1 -- 172.21.15.28:0/3640060771 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f64f8004160 con 0x7f650810a990 2024-04-23T14:33:53.257 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.219+0000 7f6504a00700 1 -- 172.21.15.28:0/3640060771 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64f8004300 con 0x7f650810a990 2024-04-23T14:33:53.257 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.219+0000 7f6504a00700 1 -- 172.21.15.28:0/3640060771 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f64f8011b20 con 0x7f650810a990 2024-04-23T14:33:53.258 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.219+0000 7f650d000700 1 -- 172.21.15.28:0/3640060771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f650810a990 msgr2=0x7f650810adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:53.258 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.219+0000 7f650d000700 1 --2- 172.21.15.28:0/3640060771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f650810a990 0x7f650810adb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f64f80098a0 tx=0x7f64f8009bb0 comp rx=0 tx=0).stop 2024-04-23T14:33:53.258 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.219+0000 7f650d000700 1 -- 172.21.15.28:0/3640060771 shutdown_connections 2024-04-23T14:33:53.258 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.219+0000 7f650d000700 1 --2- 172.21.15.28:0/3640060771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f650810a990 0x7f650810adb0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:53.258 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.219+0000 7f650d000700 1 -- 172.21.15.28:0/3640060771 >> 172.21.15.28:0/3640060771 conn(0x7f650807ddc0 msgr2=0x7f65081084c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:53.258 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.219+0000 7f650d000700 1 -- 172.21.15.28:0/3640060771 shutdown_connections 2024-04-23T14:33:53.258 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.219+0000 7f650d000700 1 -- 172.21.15.28:0/3640060771 wait complete. 2024-04-23T14:33:53.258 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.221+0000 7f650d000700 1 Processor -- start 2024-04-23T14:33:53.258 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.221+0000 7f650d000700 1 -- start start 2024-04-23T14:33:53.258 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.221+0000 7f650d000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f650810a990 0x7f650819e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:53.258 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.221+0000 7f650d000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f650810eef0 con 0x7f650810a990 2024-04-23T14:33:53.258 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.221+0000 7f6505e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f650810a990 0x7f650819e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:53.258 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.221+0000 7f6505e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f650810a990 0x7f650819e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40022/0 (socket says 172.21.15.28:40022) 2024-04-23T14:33:53.259 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.221+0000 7f6505e00700 1 -- 172.21.15.28:0/744911033 learned_addr learned my addr 172.21.15.28:0/744911033 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:53.259 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.222+0000 7f6505e00700 1 -- 172.21.15.28:0/744911033 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f650819ed70 con 0x7f650810a990 2024-04-23T14:33:53.259 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.222+0000 7f6505e00700 1 --2- 172.21.15.28:0/744911033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f650810a990 0x7f650819e830 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f64f8000c00 tx=0x7f64f8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:53.259 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.222+0000 7f64fea00700 1 -- 172.21.15.28:0/744911033 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f64f8003b60 con 0x7f650810a990 2024-04-23T14:33:53.259 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.223+0000 7f64fea00700 1 -- 172.21.15.28:0/744911033 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64f801a910 con 0x7f650810a990 2024-04-23T14:33:53.259 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.223+0000 7f650d000700 1 -- 172.21.15.28:0/744911033 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f650819f000 con 0x7f650810a990 2024-04-23T14:33:53.259 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.223+0000 7f64fea00700 1 -- 172.21.15.28:0/744911033 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f64f8024930 con 0x7f650810a990 2024-04-23T14:33:53.259 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.223+0000 7f650d000700 1 -- 172.21.15.28:0/744911033 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f650819f460 con 0x7f650810a990 2024-04-23T14:33:53.259 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.224+0000 7f64fea00700 1 -- 172.21.15.28:0/744911033 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f64f8022020 con 0x7f650810a990 2024-04-23T14:33:53.259 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.224+0000 7f650d000700 1 -- 172.21.15.28:0/744911033 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6508064150 con 0x7f650810a990 2024-04-23T14:33:53.259 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.224+0000 7f64fea00700 1 --2- 172.21.15.28:0/744911033 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f64ec0396a0 0x7f64ec03bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:53.259 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.224+0000 7f64fea00700 1 -- 172.21.15.28:0/744911033 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f64f804db10 con 0x7f650810a990 2024-04-23T14:33:53.259 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.225+0000 7f6505400700 1 --2- 172.21.15.28:0/744911033 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f64ec0396a0 0x7f64ec03bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:53.259 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.225+0000 7f6505400700 1 --2- 172.21.15.28:0/744911033 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f64ec0396a0 0x7f64ec03bb60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f64f4006fd0 tx=0x7f64f4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:53.260 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.230+0000 7f64fea00700 1 -- 172.21.15.28:0/744911033 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f64f80212c0 con 0x7f650810a990 2024-04-23T14:33:53.260 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.589+0000 7f650d000700 1 -- 172.21.15.28:0/744911033 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f65081a1e30 con 0x7f650810a990 2024-04-23T14:33:53.260 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.592+0000 7f64fea00700 1 -- 172.21.15.28:0/744911033 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7f64f801ab70 con 0x7f650810a990 2024-04-23T14:33:53.260 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.595+0000 7f650d000700 1 -- 172.21.15.28:0/744911033 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f64ec0396a0 msgr2=0x7f64ec03bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:53.260 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.595+0000 7f650d000700 1 --2- 172.21.15.28:0/744911033 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f64ec0396a0 0x7f64ec03bb60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f64f4006fd0 tx=0x7f64f4006e40 comp rx=0 tx=0).stop 2024-04-23T14:33:53.260 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.595+0000 7f650d000700 1 -- 172.21.15.28:0/744911033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f650810a990 msgr2=0x7f650819e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:53.260 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.595+0000 7f650d000700 1 --2- 172.21.15.28:0/744911033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f650810a990 0x7f650819e830 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f64f8000c00 tx=0x7f64f8011c10 comp rx=0 tx=0).stop 2024-04-23T14:33:53.260 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.596+0000 7f650d000700 1 -- 172.21.15.28:0/744911033 shutdown_connections 2024-04-23T14:33:53.260 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.596+0000 7f650d000700 1 --2- 172.21.15.28:0/744911033 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f64ec0396a0 0x7f64ec03bb60 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:53.260 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.596+0000 7f650d000700 1 --2- 172.21.15.28:0/744911033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f650810a990 0x7f650819e830 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:53.260 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.596+0000 7f650d000700 1 -- 172.21.15.28:0/744911033 >> 172.21.15.28:0/744911033 conn(0x7f650807ddc0 msgr2=0x7f6508107b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:53.260 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.596+0000 7f650d000700 1 -- 172.21.15.28:0/744911033 shutdown_connections 2024-04-23T14:33:53.260 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-23T14:33:52.596+0000 7f650d000700 1 -- 172.21.15.28:0/744911033 wait complete. 2024-04-23T14:33:53.260 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-23T14:33:53.261 INFO:teuthology.orchestra.run.smithi028.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-23T14:33:53.261 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:33:53.261 INFO:teuthology.orchestra.run.smithi028.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-23T14:33:53.261 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:33:53.261 INFO:teuthology.orchestra.run.smithi028.stdout:Or, if you are only running a single cluster on this host: 2024-04-23T14:33:53.262 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:33:53.262 INFO:teuthology.orchestra.run.smithi028.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-23T14:33:53.262 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:33:53.262 INFO:teuthology.orchestra.run.smithi028.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-23T14:33:53.262 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:33:53.262 INFO:teuthology.orchestra.run.smithi028.stdout: ceph telemetry on 2024-04-23T14:33:53.262 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:33:53.262 INFO:teuthology.orchestra.run.smithi028.stdout:For more information see: 2024-04-23T14:33:53.262 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:33:53.262 INFO:teuthology.orchestra.run.smithi028.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-23T14:33:53.262 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:33:53.262 INFO:teuthology.orchestra.run.smithi028.stdout:Bootstrap complete. 2024-04-23T14:33:53.359 INFO:tasks.cephadm:Fetching config... 2024-04-23T14:33:53.359 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-23T14:33:53.359 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-23T14:33:53.413 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-23T14:33:53.413 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-23T14:33:53.413 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-23T14:33:53.462 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-23T14:33:53.462 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-23T14:33:53.462 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-23T14:33:53.570 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-23T14:33:53.570 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-23T14:33:53.571 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-23T14:33:53.668 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-23T14:33:53.669 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDePNESHCvfO7qfiUFXQ40tSG3i5sLt0ezAou6+Is/yrbWk91HXm+bgctGhOG+XING3iFd/2T5Jr9HNELbEN+Ci3YMVeqXgHavNh7wmBANjWF8DiRdHwJlwzcUXg9cWOPIFMSpyLNZ3y2Q11jyIt8rvUkPt5vox5mIRx5rf1cf9ldebaXvx+IjRFihaEjGiWQljwXNsEql5HXB2LKh06BXKzMclxg+g8Lf0tEKhIjaGcTJTGx7VbIzifano3jw3vrYEkkqjsOZfaFwCTppkQWNuGhhQTv9T5zTM4iHmQfDfsXKarMIhXilrD2YZGwDrkHb97Zj/xcbnUQUqWjXUYNXm/MR5SNx++wXVRjhiFT78oH2ENfA1TFfz/b+KT1Wb9NKdD9kZ+DuaV4DUGwCqm2U7+TivHfpADPjnqTTwRvXJm6AV3LhdT/jTKk0m8pOGzOP0YJKHYdu9aQOu61bvbqPF1bpQ01DcJMaCwHkdAJhsCmvdN501URzeBb59AAC0uM= ceph-1ee6e996-017e-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-23T14:33:53.773 INFO:teuthology.orchestra.run.smithi028.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDePNESHCvfO7qfiUFXQ40tSG3i5sLt0ezAou6+Is/yrbWk91HXm+bgctGhOG+XING3iFd/2T5Jr9HNELbEN+Ci3YMVeqXgHavNh7wmBANjWF8DiRdHwJlwzcUXg9cWOPIFMSpyLNZ3y2Q11jyIt8rvUkPt5vox5mIRx5rf1cf9ldebaXvx+IjRFihaEjGiWQljwXNsEql5HXB2LKh06BXKzMclxg+g8Lf0tEKhIjaGcTJTGx7VbIzifano3jw3vrYEkkqjsOZfaFwCTppkQWNuGhhQTv9T5zTM4iHmQfDfsXKarMIhXilrD2YZGwDrkHb97Zj/xcbnUQUqWjXUYNXm/MR5SNx++wXVRjhiFT78oH2ENfA1TFfz/b+KT1Wb9NKdD9kZ+DuaV4DUGwCqm2U7+TivHfpADPjnqTTwRvXJm6AV3LhdT/jTKk0m8pOGzOP0YJKHYdu9aQOu61bvbqPF1bpQ01DcJMaCwHkdAJhsCmvdN501URzeBb59AAC0uM= ceph-1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T14:33:53.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:53 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/744911033' entity='client.admin' 2024-04-23T14:33:53.824 DEBUG:teuthology.orchestra.run.smithi038:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDePNESHCvfO7qfiUFXQ40tSG3i5sLt0ezAou6+Is/yrbWk91HXm+bgctGhOG+XING3iFd/2T5Jr9HNELbEN+Ci3YMVeqXgHavNh7wmBANjWF8DiRdHwJlwzcUXg9cWOPIFMSpyLNZ3y2Q11jyIt8rvUkPt5vox5mIRx5rf1cf9ldebaXvx+IjRFihaEjGiWQljwXNsEql5HXB2LKh06BXKzMclxg+g8Lf0tEKhIjaGcTJTGx7VbIzifano3jw3vrYEkkqjsOZfaFwCTppkQWNuGhhQTv9T5zTM4iHmQfDfsXKarMIhXilrD2YZGwDrkHb97Zj/xcbnUQUqWjXUYNXm/MR5SNx++wXVRjhiFT78oH2ENfA1TFfz/b+KT1Wb9NKdD9kZ+DuaV4DUGwCqm2U7+TivHfpADPjnqTTwRvXJm6AV3LhdT/jTKk0m8pOGzOP0YJKHYdu9aQOu61bvbqPF1bpQ01DcJMaCwHkdAJhsCmvdN501URzeBb59AAC0uM= ceph-1ee6e996-017e-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-23T14:33:53.865 INFO:teuthology.orchestra.run.smithi038.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDePNESHCvfO7qfiUFXQ40tSG3i5sLt0ezAou6+Is/yrbWk91HXm+bgctGhOG+XING3iFd/2T5Jr9HNELbEN+Ci3YMVeqXgHavNh7wmBANjWF8DiRdHwJlwzcUXg9cWOPIFMSpyLNZ3y2Q11jyIt8rvUkPt5vox5mIRx5rf1cf9ldebaXvx+IjRFihaEjGiWQljwXNsEql5HXB2LKh06BXKzMclxg+g8Lf0tEKhIjaGcTJTGx7VbIzifano3jw3vrYEkkqjsOZfaFwCTppkQWNuGhhQTv9T5zTM4iHmQfDfsXKarMIhXilrD2YZGwDrkHb97Zj/xcbnUQUqWjXUYNXm/MR5SNx++wXVRjhiFT78oH2ENfA1TFfz/b+KT1Wb9NKdD9kZ+DuaV4DUGwCqm2U7+TivHfpADPjnqTTwRvXJm6AV3LhdT/jTKk0m8pOGzOP0YJKHYdu9aQOu61bvbqPF1bpQ01DcJMaCwHkdAJhsCmvdN501URzeBb59AAC0uM= ceph-1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T14:33:53.878 DEBUG:teuthology.orchestra.run.smithi105:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDePNESHCvfO7qfiUFXQ40tSG3i5sLt0ezAou6+Is/yrbWk91HXm+bgctGhOG+XING3iFd/2T5Jr9HNELbEN+Ci3YMVeqXgHavNh7wmBANjWF8DiRdHwJlwzcUXg9cWOPIFMSpyLNZ3y2Q11jyIt8rvUkPt5vox5mIRx5rf1cf9ldebaXvx+IjRFihaEjGiWQljwXNsEql5HXB2LKh06BXKzMclxg+g8Lf0tEKhIjaGcTJTGx7VbIzifano3jw3vrYEkkqjsOZfaFwCTppkQWNuGhhQTv9T5zTM4iHmQfDfsXKarMIhXilrD2YZGwDrkHb97Zj/xcbnUQUqWjXUYNXm/MR5SNx++wXVRjhiFT78oH2ENfA1TFfz/b+KT1Wb9NKdD9kZ+DuaV4DUGwCqm2U7+TivHfpADPjnqTTwRvXJm6AV3LhdT/jTKk0m8pOGzOP0YJKHYdu9aQOu61bvbqPF1bpQ01DcJMaCwHkdAJhsCmvdN501URzeBb59AAC0uM= ceph-1ee6e996-017e-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-23T14:33:53.919 INFO:teuthology.orchestra.run.smithi105.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDePNESHCvfO7qfiUFXQ40tSG3i5sLt0ezAou6+Is/yrbWk91HXm+bgctGhOG+XING3iFd/2T5Jr9HNELbEN+Ci3YMVeqXgHavNh7wmBANjWF8DiRdHwJlwzcUXg9cWOPIFMSpyLNZ3y2Q11jyIt8rvUkPt5vox5mIRx5rf1cf9ldebaXvx+IjRFihaEjGiWQljwXNsEql5HXB2LKh06BXKzMclxg+g8Lf0tEKhIjaGcTJTGx7VbIzifano3jw3vrYEkkqjsOZfaFwCTppkQWNuGhhQTv9T5zTM4iHmQfDfsXKarMIhXilrD2YZGwDrkHb97Zj/xcbnUQUqWjXUYNXm/MR5SNx++wXVRjhiFT78oH2ENfA1TFfz/b+KT1Wb9NKdD9kZ+DuaV4DUGwCqm2U7+TivHfpADPjnqTTwRvXJm6AV3LhdT/jTKk0m8pOGzOP0YJKHYdu9aQOu61bvbqPF1bpQ01DcJMaCwHkdAJhsCmvdN501URzeBb59AAC0uM= ceph-1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T14:33:53.930 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-23T14:33:54.371 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:33:57.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:56.759+0000 7fbb90600700 1 -- 172.21.15.28:0/809444943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb88100dd0 msgr2=0x7fbb88103200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:57.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:56.759+0000 7fbb90600700 1 --2- 172.21.15.28:0/809444943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb88100dd0 0x7fbb88103200 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c009910 tx=0x7fbb7c009c20 comp rx=0 tx=0).stop 2024-04-23T14:33:57.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:56.759+0000 7fbb90600700 1 -- 172.21.15.28:0/809444943 shutdown_connections 2024-04-23T14:33:57.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:56.759+0000 7fbb90600700 1 --2- 172.21.15.28:0/809444943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb88100dd0 0x7fbb88103200 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:57.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:56.759+0000 7fbb90600700 1 -- 172.21.15.28:0/809444943 >> 172.21.15.28:0/809444943 conn(0x7fbb880fc9f0 msgr2=0x7fbb880fee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:57.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:56.760+0000 7fbb90600700 1 -- 172.21.15.28:0/809444943 shutdown_connections 2024-04-23T14:33:57.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:56.760+0000 7fbb90600700 1 -- 172.21.15.28:0/809444943 wait complete. 2024-04-23T14:33:57.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:56.761+0000 7fbb90600700 1 Processor -- start 2024-04-23T14:33:57.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:56.761+0000 7fbb90600700 1 -- start start 2024-04-23T14:33:57.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:56.761+0000 7fbb90600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb88100dd0 0x7fbb8819e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:57.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:56.761+0000 7fbb90600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbb88106490 con 0x7fbb88100dd0 2024-04-23T14:33:57.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:56.762+0000 7fbb8dc00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb88100dd0 0x7fbb8819e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:57.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:56.762+0000 7fbb8dc00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb88100dd0 0x7fbb8819e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40036/0 (socket says 172.21.15.28:40036) 2024-04-23T14:33:57.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:56.762+0000 7fbb8dc00700 1 -- 172.21.15.28:0/3003475113 learned_addr learned my addr 172.21.15.28:0/3003475113 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:33:57.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:56.762+0000 7fbb8dc00700 1 -- 172.21.15.28:0/3003475113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb8819e750 con 0x7fbb88100dd0 2024-04-23T14:33:57.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:56.762+0000 7fbb8dc00700 1 --2- 172.21.15.28:0/3003475113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb88100dd0 0x7fbb8819e210 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c005950 tx=0x7fbb7c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:57.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:56.763+0000 7fbb86000700 1 -- 172.21.15.28:0/3003475113 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb7c026070 con 0x7fbb88100dd0 2024-04-23T14:33:57.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:56.763+0000 7fbb86000700 1 -- 172.21.15.28:0/3003475113 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb7c00bc30 con 0x7fbb88100dd0 2024-04-23T14:33:57.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:56.763+0000 7fbb90600700 1 -- 172.21.15.28:0/3003475113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb8819e9e0 con 0x7fbb88100dd0 2024-04-23T14:33:57.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:56.763+0000 7fbb86000700 1 -- 172.21.15.28:0/3003475113 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb7c003b30 con 0x7fbb88100dd0 2024-04-23T14:33:57.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:56.763+0000 7fbb90600700 1 -- 172.21.15.28:0/3003475113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb881a16d0 con 0x7fbb88100dd0 2024-04-23T14:33:57.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:56.764+0000 7fbb86000700 1 -- 172.21.15.28:0/3003475113 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fbb7c029070 con 0x7fbb88100dd0 2024-04-23T14:33:57.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:56.764+0000 7fbb90600700 1 -- 172.21.15.28:0/3003475113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb88106c10 con 0x7fbb88100dd0 2024-04-23T14:33:57.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:56.765+0000 7fbb86000700 1 --2- 172.21.15.28:0/3003475113 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fbb700396b0 0x7fbb7003bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:33:57.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:56.765+0000 7fbb86000700 1 -- 172.21.15.28:0/3003475113 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fbb7c01e070 con 0x7fbb88100dd0 2024-04-23T14:33:57.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:56.765+0000 7fbb8d200700 1 --2- 172.21.15.28:0/3003475113 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fbb700396b0 0x7fbb7003bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:33:57.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:56.766+0000 7fbb8d200700 1 --2- 172.21.15.28:0/3003475113 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fbb700396b0 0x7fbb7003bb70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbb74006fd0 tx=0x7fbb74006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:33:57.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:56.771+0000 7fbb86000700 1 -- 172.21.15.28:0/3003475113 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbb7c013070 con 0x7fbb88100dd0 2024-04-23T14:33:57.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:57.009+0000 7fbb90600700 1 -- 172.21.15.28:0/3003475113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7fbb8802e0f0 con 0x7fbb88100dd0 2024-04-23T14:33:57.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:57.017+0000 7fbb86000700 1 -- 172.21.15.28:0/3003475113 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7fbb7c0300c0 con 0x7fbb88100dd0 2024-04-23T14:33:57.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:57.021+0000 7fbb90600700 1 -- 172.21.15.28:0/3003475113 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fbb700396b0 msgr2=0x7fbb7003bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:57.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:57.021+0000 7fbb90600700 1 --2- 172.21.15.28:0/3003475113 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fbb700396b0 0x7fbb7003bb70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbb74006fd0 tx=0x7fbb74006e40 comp rx=0 tx=0).stop 2024-04-23T14:33:57.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:57.021+0000 7fbb90600700 1 -- 172.21.15.28:0/3003475113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb88100dd0 msgr2=0x7fbb8819e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:33:57.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:57.021+0000 7fbb90600700 1 --2- 172.21.15.28:0/3003475113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb88100dd0 0x7fbb8819e210 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c005950 tx=0x7fbb7c005030 comp rx=0 tx=0).stop 2024-04-23T14:33:57.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:57.022+0000 7fbb90600700 1 -- 172.21.15.28:0/3003475113 shutdown_connections 2024-04-23T14:33:57.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:57.022+0000 7fbb90600700 1 --2- 172.21.15.28:0/3003475113 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fbb700396b0 0x7fbb7003bb70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:57.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:57.022+0000 7fbb90600700 1 --2- 172.21.15.28:0/3003475113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb88100dd0 0x7fbb8819e210 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:33:57.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:57.022+0000 7fbb90600700 1 -- 172.21.15.28:0/3003475113 >> 172.21.15.28:0/3003475113 conn(0x7fbb880fc9f0 msgr2=0x7fbb880fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:33:57.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:57.022+0000 7fbb90600700 1 -- 172.21.15.28:0/3003475113 shutdown_connections 2024-04-23T14:33:57.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:33:57.022+0000 7fbb90600700 1 -- 172.21.15.28:0/3003475113 wait complete. 2024-04-23T14:33:57.777 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-23T14:33:57.778 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-23T14:33:58.224 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:33:58.371 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:58 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:33:58.371 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:58 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:33:58.371 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:58 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3003475113' entity='client.admin' 2024-04-23T14:33:58.371 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:58 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi028", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:33:58.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:58 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:33:58.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:58 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:33:58.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:33:58 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:00.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.113+0000 7f211a400700 1 -- 172.21.15.28:0/3306723337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21141012b0 msgr2=0x7f21141016d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:00.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.113+0000 7f211a400700 1 --2- 172.21.15.28:0/3306723337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21141012b0 0x7f21141016d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2108009910 tx=0x7f2108009c20 comp rx=0 tx=0).stop 2024-04-23T14:34:00.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.114+0000 7f211a400700 1 -- 172.21.15.28:0/3306723337 shutdown_connections 2024-04-23T14:34:00.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.114+0000 7f211a400700 1 --2- 172.21.15.28:0/3306723337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21141012b0 0x7f21141016d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:00.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.114+0000 7f211a400700 1 -- 172.21.15.28:0/3306723337 >> 172.21.15.28:0/3306723337 conn(0x7f21140fc850 msgr2=0x7f21140fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:00.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.114+0000 7f211a400700 1 -- 172.21.15.28:0/3306723337 shutdown_connections 2024-04-23T14:34:00.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.114+0000 7f211a400700 1 -- 172.21.15.28:0/3306723337 wait complete. 2024-04-23T14:34:00.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.115+0000 7f211a400700 1 Processor -- start 2024-04-23T14:34:00.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.115+0000 7f211a400700 1 -- start start 2024-04-23T14:34:00.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.115+0000 7f211a400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21141012b0 0x7f211419e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:00.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.115+0000 7f211a400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2114107340 con 0x7f21141012b0 2024-04-23T14:34:00.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.116+0000 7f2119000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21141012b0 0x7f211419e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:00.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.116+0000 7f2119000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21141012b0 0x7f211419e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59404/0 (socket says 172.21.15.28:59404) 2024-04-23T14:34:00.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.116+0000 7f2119000700 1 -- 172.21.15.28:0/278401085 learned_addr learned my addr 172.21.15.28:0/278401085 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:34:00.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.116+0000 7f2119000700 1 -- 172.21.15.28:0/278401085 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f211419e6b0 con 0x7f21141012b0 2024-04-23T14:34:00.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.116+0000 7f2119000700 1 --2- 172.21.15.28:0/278401085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21141012b0 0x7f211419e170 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2108009fd0 tx=0x7f2108005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:00.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.117+0000 7f2111600700 1 -- 172.21.15.28:0/278401085 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2108027070 con 0x7f21141012b0 2024-04-23T14:34:00.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.117+0000 7f2111600700 1 -- 172.21.15.28:0/278401085 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f210800be40 con 0x7f21141012b0 2024-04-23T14:34:00.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.117+0000 7f2111600700 1 -- 172.21.15.28:0/278401085 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f210800f460 con 0x7f21141012b0 2024-04-23T14:34:00.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.117+0000 7f211a400700 1 -- 172.21.15.28:0/278401085 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f211419e940 con 0x7f21141012b0 2024-04-23T14:34:00.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.117+0000 7f211a400700 1 -- 172.21.15.28:0/278401085 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21141a1630 con 0x7f21141012b0 2024-04-23T14:34:00.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.118+0000 7f2111600700 1 -- 172.21.15.28:0/278401085 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f210802a070 con 0x7f21141012b0 2024-04-23T14:34:00.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.118+0000 7f211a400700 1 -- 172.21.15.28:0/278401085 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2114107ac0 con 0x7f21141012b0 2024-04-23T14:34:00.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.118+0000 7f2111600700 1 --2- 172.21.15.28:0/278401085 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f20fc0396b0 0x7f20fc03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:00.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.118+0000 7f2111600700 1 -- 172.21.15.28:0/278401085 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f210801e030 con 0x7f21141012b0 2024-04-23T14:34:00.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.119+0000 7f2113e00700 1 --2- 172.21.15.28:0/278401085 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f20fc0396b0 0x7f20fc03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:00.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.120+0000 7f2113e00700 1 --2- 172.21.15.28:0/278401085 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f20fc0396b0 0x7f20fc03bb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2100006fd0 tx=0x7f2100006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:00.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.124+0000 7f2111600700 1 -- 172.21.15.28:0/278401085 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2108013070 con 0x7f21141012b0 2024-04-23T14:34:00.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.357+0000 7f211a400700 1 -- 172.21.15.28:0/278401085 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f21140fe0c0 con 0x7f20fc0396b0 2024-04-23T14:34:00.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.363+0000 7f2111600700 1 -- 172.21.15.28:0/278401085 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f21140fe0c0 con 0x7f20fc0396b0 2024-04-23T14:34:00.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.366+0000 7f211a400700 1 -- 172.21.15.28:0/278401085 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f20fc0396b0 msgr2=0x7f20fc03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:00.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.366+0000 7f211a400700 1 --2- 172.21.15.28:0/278401085 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f20fc0396b0 0x7f20fc03bb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2100006fd0 tx=0x7f2100006e40 comp rx=0 tx=0).stop 2024-04-23T14:34:00.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.366+0000 7f211a400700 1 -- 172.21.15.28:0/278401085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21141012b0 msgr2=0x7f211419e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:00.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.366+0000 7f211a400700 1 --2- 172.21.15.28:0/278401085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21141012b0 0x7f211419e170 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2108009fd0 tx=0x7f2108005050 comp rx=0 tx=0).stop 2024-04-23T14:34:00.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.366+0000 7f211a400700 1 -- 172.21.15.28:0/278401085 shutdown_connections 2024-04-23T14:34:00.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.366+0000 7f211a400700 1 --2- 172.21.15.28:0/278401085 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f20fc0396b0 0x7f20fc03bb70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:00.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.366+0000 7f211a400700 1 --2- 172.21.15.28:0/278401085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21141012b0 0x7f211419e170 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:00.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.366+0000 7f211a400700 1 -- 172.21.15.28:0/278401085 >> 172.21.15.28:0/278401085 conn(0x7f21140fc850 msgr2=0x7f21140fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:00.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.367+0000 7f211a400700 1 -- 172.21.15.28:0/278401085 shutdown_connections 2024-04-23T14:34:00.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:00.367+0000 7f211a400700 1 -- 172.21.15.28:0/278401085 wait complete. 2024-04-23T14:34:01.144 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi038 2024-04-23T14:34:01.144 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-23T14:34:01.144 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/etc/ceph/ceph.conf 2024-04-23T14:34:01.163 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-23T14:34:01.163 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-23T14:34:01.221 INFO:tasks.cephadm:Adding host smithi038 to orchestrator... 2024-04-23T14:34:01.221 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph orch host add smithi038 2024-04-23T14:34:01.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:01 smithi028 ceph-mon[27810]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:34:01.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:01 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:01.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:01 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:34:01.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:01 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:34:01.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:01 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:34:01.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:01 smithi028 ceph-mon[27810]: Updating smithi028:/etc/ceph/ceph.conf 2024-04-23T14:34:01.644 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:34:02.620 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:02 smithi028 ceph-mon[27810]: Updating smithi028:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T14:34:03.462 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:03 smithi028 ceph-mon[27810]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:03.462 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:03 smithi028 ceph-mon[27810]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-04-23T14:34:03.462 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:03 smithi028 ceph-mon[27810]: Updating smithi028:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T14:34:03.462 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:03 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:03.462 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:03 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:03.462 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:03 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:03.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:03.638+0000 7fbaba600700 1 -- 172.21.15.28:0/232296612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbab410a710 msgr2=0x7fbab410aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:03.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:03.638+0000 7fbaba600700 1 --2- 172.21.15.28:0/232296612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbab410a710 0x7fbab410aaf0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4009910 tx=0x7fbaa4009c20 comp rx=0 tx=0).stop 2024-04-23T14:34:03.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:03.638+0000 7fbaba600700 1 -- 172.21.15.28:0/232296612 shutdown_connections 2024-04-23T14:34:03.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:03.638+0000 7fbaba600700 1 --2- 172.21.15.28:0/232296612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbab410a710 0x7fbab410aaf0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:03.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:03.638+0000 7fbaba600700 1 -- 172.21.15.28:0/232296612 >> 172.21.15.28:0/232296612 conn(0x7fbab4100240 msgr2=0x7fbab4102660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:03.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:03.638+0000 7fbaba600700 1 -- 172.21.15.28:0/232296612 shutdown_connections 2024-04-23T14:34:03.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:03.638+0000 7fbaba600700 1 -- 172.21.15.28:0/232296612 wait complete. 2024-04-23T14:34:03.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:03.640+0000 7fbaba600700 1 Processor -- start 2024-04-23T14:34:03.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:03.640+0000 7fbaba600700 1 -- start start 2024-04-23T14:34:03.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:03.640+0000 7fbaba600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbab410a710 0x7fbab419a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:03.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:03.640+0000 7fbaba600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbab410ed50 con 0x7fbab410a710 2024-04-23T14:34:03.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:03.641+0000 7fbab3400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbab410a710 0x7fbab419a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:03.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:03.641+0000 7fbab3400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbab410a710 0x7fbab419a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59432/0 (socket says 172.21.15.28:59432) 2024-04-23T14:34:03.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:03.641+0000 7fbab3400700 1 -- 172.21.15.28:0/3338535474 learned_addr learned my addr 172.21.15.28:0/3338535474 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:34:03.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:03.641+0000 7fbab3400700 1 -- 172.21.15.28:0/3338535474 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbab419a7f0 con 0x7fbab410a710 2024-04-23T14:34:03.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:03.641+0000 7fbab3400700 1 --2- 172.21.15.28:0/3338535474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbab410a710 0x7fbab419a2b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4006010 tx=0x7fbaa4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:03.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:03.642+0000 7fbaabe00700 1 -- 172.21.15.28:0/3338535474 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbaa4026070 con 0x7fbab410a710 2024-04-23T14:34:03.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:03.642+0000 7fbaabe00700 1 -- 172.21.15.28:0/3338535474 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbaa400bbd0 con 0x7fbab410a710 2024-04-23T14:34:03.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:03.642+0000 7fbaba600700 1 -- 172.21.15.28:0/3338535474 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbab419aae0 con 0x7fbab410a710 2024-04-23T14:34:03.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:03.642+0000 7fbaabe00700 1 -- 172.21.15.28:0/3338535474 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbaa4003910 con 0x7fbab410a710 2024-04-23T14:34:03.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:03.642+0000 7fbaba600700 1 -- 172.21.15.28:0/3338535474 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbab419eb80 con 0x7fbab410a710 2024-04-23T14:34:03.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:03.643+0000 7fbaabe00700 1 -- 172.21.15.28:0/3338535474 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fbaa4029070 con 0x7fbab410a710 2024-04-23T14:34:03.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:03.643+0000 7fbaba600700 1 -- 172.21.15.28:0/3338535474 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbab4105e50 con 0x7fbab410a710 2024-04-23T14:34:03.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:03.644+0000 7fbaabe00700 1 --2- 172.21.15.28:0/3338535474 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fba98039650 0x7fba9803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:03.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:03.644+0000 7fbaabe00700 1 -- 172.21.15.28:0/3338535474 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fbaa401e070 con 0x7fbab410a710 2024-04-23T14:34:03.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:03.644+0000 7fbab2a00700 1 --2- 172.21.15.28:0/3338535474 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fba98039650 0x7fba9803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:03.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:03.645+0000 7fbab2a00700 1 --2- 172.21.15.28:0/3338535474 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fba98039650 0x7fba9803bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fbaa0006fd0 tx=0x7fbaa0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:03.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:03.650+0000 7fbaabe00700 1 -- 172.21.15.28:0/3338535474 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbaa4013070 con 0x7fbab410a710 2024-04-23T14:34:03.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:03.884+0000 7fbaba600700 1 -- 172.21.15.28:0/3338535474 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi038", "target": ["mon-mgr", ""]}) -- 0x7fbab419b610 con 0x7fba98039650 2024-04-23T14:34:05.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:05 smithi028 ceph-mon[27810]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:05.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:05 smithi028 ceph-mon[27810]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi038", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:34:06.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:06 smithi028 ceph-mon[27810]: Deploying cephadm binary to smithi038 2024-04-23T14:34:07.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:07 smithi028 ceph-mon[27810]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:08.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:08.770+0000 7fbaabe00700 1 -- 172.21.15.28:0/3338535474 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fbab419b610 con 0x7fba98039650 2024-04-23T14:34:08.772 INFO:teuthology.orchestra.run.smithi028.stdout:Added host 'smithi038' with addr '172.21.15.38' 2024-04-23T14:34:08.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:08.774+0000 7fbaba600700 1 -- 172.21.15.28:0/3338535474 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fba98039650 msgr2=0x7fba9803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:08.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:08.774+0000 7fbaba600700 1 --2- 172.21.15.28:0/3338535474 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fba98039650 0x7fba9803bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fbaa0006fd0 tx=0x7fbaa0006e40 comp rx=0 tx=0).stop 2024-04-23T14:34:08.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:08.774+0000 7fbaba600700 1 -- 172.21.15.28:0/3338535474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbab410a710 msgr2=0x7fbab419a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:08.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:08.774+0000 7fbaba600700 1 --2- 172.21.15.28:0/3338535474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbab410a710 0x7fbab419a2b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4006010 tx=0x7fbaa4005030 comp rx=0 tx=0).stop 2024-04-23T14:34:08.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:08.774+0000 7fbaba600700 1 -- 172.21.15.28:0/3338535474 shutdown_connections 2024-04-23T14:34:08.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:08.774+0000 7fbaba600700 1 --2- 172.21.15.28:0/3338535474 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fba98039650 0x7fba9803bb10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:08.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:08.774+0000 7fbaba600700 1 --2- 172.21.15.28:0/3338535474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbab410a710 0x7fbab419a2b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:08.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:08.774+0000 7fbaba600700 1 -- 172.21.15.28:0/3338535474 >> 172.21.15.28:0/3338535474 conn(0x7fbab4100240 msgr2=0x7fbab410c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:08.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:08.775+0000 7fbaba600700 1 -- 172.21.15.28:0/3338535474 shutdown_connections 2024-04-23T14:34:08.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:08.775+0000 7fbaba600700 1 -- 172.21.15.28:0/3338535474 wait complete. 2024-04-23T14:34:09.428 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-23T14:34:09.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:09 smithi028 ceph-mon[27810]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:09.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:09 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:09.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:09 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:34:09.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:09 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:09.876 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:34:10.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:10 smithi028 ceph-mon[27810]: Added host smithi038 2024-04-23T14:34:10.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:10 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:11.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:11 smithi028 ceph-mon[27810]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:11.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:11.807+0000 7f2da2a00700 1 -- 172.21.15.28:0/3909275583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9c1036a0 msgr2=0x7f2d9c103a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:11.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:11.807+0000 7f2da2a00700 1 --2- 172.21.15.28:0/3909275583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9c1036a0 0x7f2d9c103a80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c009910 tx=0x7f2d8c009c20 comp rx=0 tx=0).stop 2024-04-23T14:34:11.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:11.808+0000 7f2da2a00700 1 -- 172.21.15.28:0/3909275583 shutdown_connections 2024-04-23T14:34:11.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:11.808+0000 7f2da2a00700 1 --2- 172.21.15.28:0/3909275583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9c1036a0 0x7f2d9c103a80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:11.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:11.808+0000 7f2da2a00700 1 -- 172.21.15.28:0/3909275583 >> 172.21.15.28:0/3909275583 conn(0x7f2d9c0fcef0 msgr2=0x7f2d9c0ff310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:11.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:11.808+0000 7f2da2a00700 1 -- 172.21.15.28:0/3909275583 shutdown_connections 2024-04-23T14:34:11.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:11.808+0000 7f2da2a00700 1 -- 172.21.15.28:0/3909275583 wait complete. 2024-04-23T14:34:11.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:11.809+0000 7f2da2a00700 1 Processor -- start 2024-04-23T14:34:11.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:11.809+0000 7f2da2a00700 1 -- start start 2024-04-23T14:34:11.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:11.809+0000 7f2da2a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9c1036a0 0x7f2d9c19a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:11.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:11.809+0000 7f2da2a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2d9c10cb90 con 0x7f2d9c1036a0 2024-04-23T14:34:11.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:11.810+0000 7f2d9be00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9c1036a0 0x7f2d9c19a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:11.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:11.810+0000 7f2d9be00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9c1036a0 0x7f2d9c19a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40364/0 (socket says 172.21.15.28:40364) 2024-04-23T14:34:11.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:11.810+0000 7f2d9be00700 1 -- 172.21.15.28:0/804506415 learned_addr learned my addr 172.21.15.28:0/804506415 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:34:11.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:11.810+0000 7f2d9be00700 1 -- 172.21.15.28:0/804506415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d9c19a860 con 0x7f2d9c1036a0 2024-04-23T14:34:11.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:11.810+0000 7f2d9be00700 1 --2- 172.21.15.28:0/804506415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9c1036a0 0x7f2d9c19a320 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c006010 tx=0x7f2d8c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:11.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:11.811+0000 7f2d98c00700 1 -- 172.21.15.28:0/804506415 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d8c026070 con 0x7f2d9c1036a0 2024-04-23T14:34:11.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:11.811+0000 7f2d98c00700 1 -- 172.21.15.28:0/804506415 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d8c00bc30 con 0x7f2d9c1036a0 2024-04-23T14:34:11.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:11.811+0000 7f2d98c00700 1 -- 172.21.15.28:0/804506415 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d8c0039a0 con 0x7f2d9c1036a0 2024-04-23T14:34:11.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:11.811+0000 7f2da2a00700 1 -- 172.21.15.28:0/804506415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d9c19aaf0 con 0x7f2d9c1036a0 2024-04-23T14:34:11.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:11.811+0000 7f2da2a00700 1 -- 172.21.15.28:0/804506415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d9c19af50 con 0x7f2d9c1036a0 2024-04-23T14:34:11.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:11.812+0000 7f2da2a00700 1 -- 172.21.15.28:0/804506415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d9c06b080 con 0x7f2d9c1036a0 2024-04-23T14:34:11.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:11.812+0000 7f2d98c00700 1 -- 172.21.15.28:0/804506415 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f2d8c029070 con 0x7f2d9c1036a0 2024-04-23T14:34:11.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:11.813+0000 7f2d98c00700 1 --2- 172.21.15.28:0/804506415 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2d800396b0 0x7f2d8003bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:11.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:11.813+0000 7f2d98c00700 1 -- 172.21.15.28:0/804506415 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f2d8c01e070 con 0x7f2d9c1036a0 2024-04-23T14:34:11.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:11.813+0000 7f2d9b400700 1 --2- 172.21.15.28:0/804506415 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2d800396b0 0x7f2d8003bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:11.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:11.814+0000 7f2d9b400700 1 --2- 172.21.15.28:0/804506415 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2d800396b0 0x7f2d8003bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2d88006fd0 tx=0x7f2d88006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:11.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:11.819+0000 7f2d98c00700 1 -- 172.21.15.28:0/804506415 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2d8c013070 con 0x7f2d9c1036a0 2024-04-23T14:34:12.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:12.063+0000 7f2da2a00700 1 -- 172.21.15.28:0/804506415 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2d9c19b680 con 0x7f2d800396b0 2024-04-23T14:34:12.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:12.066+0000 7f2d98c00700 1 -- 172.21.15.28:0/804506415 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7f2d9c19b680 con 0x7f2d800396b0 2024-04-23T14:34:12.067 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:34:12.067 INFO:teuthology.orchestra.run.smithi028.stdout:[{"addr": "172.21.15.28", "hostname": "smithi028", "labels": [], "status": ""}, {"addr": "172.21.15.38", "hostname": "smithi038", "labels": [], "status": ""}] 2024-04-23T14:34:12.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:12.068+0000 7f2da2a00700 1 -- 172.21.15.28:0/804506415 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2d800396b0 msgr2=0x7f2d8003bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:12.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:12.068+0000 7f2da2a00700 1 --2- 172.21.15.28:0/804506415 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2d800396b0 0x7f2d8003bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2d88006fd0 tx=0x7f2d88006e40 comp rx=0 tx=0).stop 2024-04-23T14:34:12.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:12.068+0000 7f2da2a00700 1 -- 172.21.15.28:0/804506415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9c1036a0 msgr2=0x7f2d9c19a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:12.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:12.068+0000 7f2da2a00700 1 --2- 172.21.15.28:0/804506415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9c1036a0 0x7f2d9c19a320 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c006010 tx=0x7f2d8c005030 comp rx=0 tx=0).stop 2024-04-23T14:34:12.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:12.068+0000 7f2da2a00700 1 -- 172.21.15.28:0/804506415 shutdown_connections 2024-04-23T14:34:12.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:12.068+0000 7f2da2a00700 1 --2- 172.21.15.28:0/804506415 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2d800396b0 0x7f2d8003bb70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:12.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:12.068+0000 7f2da2a00700 1 --2- 172.21.15.28:0/804506415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9c1036a0 0x7f2d9c19a320 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:12.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:12.068+0000 7f2da2a00700 1 -- 172.21.15.28:0/804506415 >> 172.21.15.28:0/804506415 conn(0x7f2d9c0fcef0 msgr2=0x7f2d9c108860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:12.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:12.069+0000 7f2da2a00700 1 -- 172.21.15.28:0/804506415 shutdown_connections 2024-04-23T14:34:12.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:12.069+0000 7f2da2a00700 1 -- 172.21.15.28:0/804506415 wait complete. 2024-04-23T14:34:12.760 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:12 smithi028 ceph-mon[27810]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:12.760 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:12 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:12.760 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:12 smithi028 ceph-mon[27810]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T14:34:12.826 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi105 2024-04-23T14:34:12.826 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-23T14:34:12.827 DEBUG:teuthology.orchestra.run.smithi105:> dd of=/etc/ceph/ceph.conf 2024-04-23T14:34:12.845 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-23T14:34:12.845 DEBUG:teuthology.orchestra.run.smithi105:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-23T14:34:12.901 INFO:tasks.cephadm:Adding host smithi105 to orchestrator... 2024-04-23T14:34:12.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph orch host add smithi105 2024-04-23T14:34:13.385 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:34:14.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:14 smithi028 ceph-mon[27810]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:15.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:15.533+0000 7fd07d000700 1 -- 172.21.15.28:0/1261205905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd07810d850 msgr2=0x7fd07810dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:15.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:15.533+0000 7fd07d000700 1 --2- 172.21.15.28:0/1261205905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd07810d850 0x7fd07810dc30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd068009910 tx=0x7fd068009c20 comp rx=0 tx=0).stop 2024-04-23T14:34:15.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:15.533+0000 7fd07d000700 1 -- 172.21.15.28:0/1261205905 shutdown_connections 2024-04-23T14:34:15.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:15.533+0000 7fd07d000700 1 --2- 172.21.15.28:0/1261205905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd07810d850 0x7fd07810dc30 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:15.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:15.533+0000 7fd07d000700 1 -- 172.21.15.28:0/1261205905 >> 172.21.15.28:0/1261205905 conn(0x7fd078078100 msgr2=0x7fd07807a520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:15.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:15.533+0000 7fd07d000700 1 -- 172.21.15.28:0/1261205905 shutdown_connections 2024-04-23T14:34:15.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:15.533+0000 7fd07d000700 1 -- 172.21.15.28:0/1261205905 wait complete. 2024-04-23T14:34:15.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:15.534+0000 7fd07d000700 1 Processor -- start 2024-04-23T14:34:15.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:15.534+0000 7fd07d000700 1 -- start start 2024-04-23T14:34:15.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:15.535+0000 7fd07d000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd07810d850 0x7fd07819e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:15.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:15.535+0000 7fd07d000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd078112fa0 con 0x7fd07810d850 2024-04-23T14:34:15.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:15.535+0000 7fd077400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd07810d850 0x7fd07819e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:15.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:15.535+0000 7fd077400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd07810d850 0x7fd07819e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40388/0 (socket says 172.21.15.28:40388) 2024-04-23T14:34:15.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:15.535+0000 7fd077400700 1 -- 172.21.15.28:0/3630520454 learned_addr learned my addr 172.21.15.28:0/3630520454 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:34:15.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:15.536+0000 7fd077400700 1 -- 172.21.15.28:0/3630520454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd07819e930 con 0x7fd07810d850 2024-04-23T14:34:15.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:15.536+0000 7fd077400700 1 --2- 172.21.15.28:0/3630520454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd07810d850 0x7fd07819e3f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd068006010 tx=0x7fd068005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:15.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:15.536+0000 7fd06fe00700 1 -- 172.21.15.28:0/3630520454 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd068026070 con 0x7fd07810d850 2024-04-23T14:34:15.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:15.536+0000 7fd06fe00700 1 -- 172.21.15.28:0/3630520454 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd06800bbd0 con 0x7fd07810d850 2024-04-23T14:34:15.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:15.536+0000 7fd07d000700 1 -- 172.21.15.28:0/3630520454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd07819ec20 con 0x7fd07810d850 2024-04-23T14:34:15.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:15.536+0000 7fd06fe00700 1 -- 172.21.15.28:0/3630520454 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd0680039d0 con 0x7fd07810d850 2024-04-23T14:34:15.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:15.537+0000 7fd07d000700 1 -- 172.21.15.28:0/3630520454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0781a2cc0 con 0x7fd07810d850 2024-04-23T14:34:15.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:15.538+0000 7fd06fe00700 1 -- 172.21.15.28:0/3630520454 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fd068029070 con 0x7fd07810d850 2024-04-23T14:34:15.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:15.538+0000 7fd07d000700 1 -- 172.21.15.28:0/3630520454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd07810a070 con 0x7fd07810d850 2024-04-23T14:34:15.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:15.538+0000 7fd06fe00700 1 --2- 172.21.15.28:0/3630520454 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd05c0396b0 0x7fd05c03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:15.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:15.538+0000 7fd06fe00700 1 -- 172.21.15.28:0/3630520454 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd06801e070 con 0x7fd07810d850 2024-04-23T14:34:15.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:15.539+0000 7fd076a00700 1 --2- 172.21.15.28:0/3630520454 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd05c0396b0 0x7fd05c03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:15.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:15.540+0000 7fd076a00700 1 --2- 172.21.15.28:0/3630520454 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd05c0396b0 0x7fd05c03bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd064006fd0 tx=0x7fd064006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:15.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:15.544+0000 7fd06fe00700 1 -- 172.21.15.28:0/3630520454 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd068013070 con 0x7fd07810d850 2024-04-23T14:34:15.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:15.779+0000 7fd07d000700 1 -- 172.21.15.28:0/3630520454 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi105", "target": ["mon-mgr", ""]}) -- 0x7fd07819f750 con 0x7fd05c0396b0 2024-04-23T14:34:16.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:15 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:16.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:15 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:16.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:15 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:16.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:15 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:16.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:15 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:34:16.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:15 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:34:16.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:15 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:34:16.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:15 smithi028 ceph-mon[27810]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-23T14:34:16.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:15 smithi028 ceph-mon[27810]: Updating smithi038:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T14:34:16.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:15 smithi028 ceph-mon[27810]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-04-23T14:34:17.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:17 smithi028 ceph-mon[27810]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:17.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:17 smithi028 ceph-mon[27810]: Updating smithi038:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T14:34:17.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:17 smithi028 ceph-mon[27810]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi105", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:34:17.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:17 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:17.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:17 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:17.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:17 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:18.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:18 smithi028 ceph-mon[27810]: Deploying cephadm binary to smithi105 2024-04-23T14:34:19.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:19 smithi028 ceph-mon[27810]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:20.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:20.701+0000 7fd06fe00700 1 -- 172.21.15.28:0/3630520454 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fd07819f750 con 0x7fd05c0396b0 2024-04-23T14:34:20.704 INFO:teuthology.orchestra.run.smithi028.stdout:Added host 'smithi105' with addr '172.21.15.105' 2024-04-23T14:34:20.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:20.704+0000 7fd07d000700 1 -- 172.21.15.28:0/3630520454 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd05c0396b0 msgr2=0x7fd05c03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:20.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:20.704+0000 7fd07d000700 1 --2- 172.21.15.28:0/3630520454 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd05c0396b0 0x7fd05c03bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd064006fd0 tx=0x7fd064006e40 comp rx=0 tx=0).stop 2024-04-23T14:34:20.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:20.704+0000 7fd07d000700 1 -- 172.21.15.28:0/3630520454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd07810d850 msgr2=0x7fd07819e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:20.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:20.704+0000 7fd07d000700 1 --2- 172.21.15.28:0/3630520454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd07810d850 0x7fd07819e3f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd068006010 tx=0x7fd068005030 comp rx=0 tx=0).stop 2024-04-23T14:34:20.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:20.705+0000 7fd07d000700 1 -- 172.21.15.28:0/3630520454 shutdown_connections 2024-04-23T14:34:20.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:20.705+0000 7fd07d000700 1 --2- 172.21.15.28:0/3630520454 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd05c0396b0 0x7fd05c03bb70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:20.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:20.705+0000 7fd07d000700 1 --2- 172.21.15.28:0/3630520454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd07810d850 0x7fd07819e3f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:20.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:20.705+0000 7fd07d000700 1 -- 172.21.15.28:0/3630520454 >> 172.21.15.28:0/3630520454 conn(0x7fd078078100 msgr2=0x7fd078110410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:20.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:20.705+0000 7fd07d000700 1 -- 172.21.15.28:0/3630520454 shutdown_connections 2024-04-23T14:34:20.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:20.705+0000 7fd07d000700 1 -- 172.21.15.28:0/3630520454 wait complete. 2024-04-23T14:34:21.297 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:21 smithi028 ceph-mon[27810]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:21.297 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:21 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:21.297 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:21 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:34:21.368 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-23T14:34:21.825 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:34:22.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:22 smithi028 ceph-mon[27810]: Added host smithi105 2024-04-23T14:34:22.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:22 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:22.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:22 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:23.529 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:23 smithi028 ceph-mon[27810]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:23.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:23.810+0000 7ff8c2200700 1 -- 172.21.15.28:0/465148888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8bc074e90 msgr2=0x7ff8bc075270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:23.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:23.810+0000 7ff8c2200700 1 --2- 172.21.15.28:0/465148888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8bc074e90 0x7ff8bc075270 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac009910 tx=0x7ff8ac009c20 comp rx=0 tx=0).stop 2024-04-23T14:34:23.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:23.811+0000 7ff8c2200700 1 -- 172.21.15.28:0/465148888 shutdown_connections 2024-04-23T14:34:23.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:23.811+0000 7ff8c2200700 1 --2- 172.21.15.28:0/465148888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8bc074e90 0x7ff8bc075270 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:23.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:23.811+0000 7ff8c2200700 1 -- 172.21.15.28:0/465148888 >> 172.21.15.28:0/465148888 conn(0x7ff8bc0fe030 msgr2=0x7ff8bc100450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:23.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:23.811+0000 7ff8c2200700 1 -- 172.21.15.28:0/465148888 shutdown_connections 2024-04-23T14:34:23.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:23.811+0000 7ff8c2200700 1 -- 172.21.15.28:0/465148888 wait complete. 2024-04-23T14:34:23.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:23.812+0000 7ff8c2200700 1 Processor -- start 2024-04-23T14:34:23.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:23.812+0000 7ff8c2200700 1 -- start start 2024-04-23T14:34:23.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:23.813+0000 7ff8c2200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8bc074e90 0x7ff8bc19e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:23.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:23.813+0000 7ff8c2200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff8bc1131a0 con 0x7ff8bc074e90 2024-04-23T14:34:23.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:23.813+0000 7ff8bb400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8bc074e90 0x7ff8bc19e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:23.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:23.813+0000 7ff8bb400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8bc074e90 0x7ff8bc19e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39346/0 (socket says 172.21.15.28:39346) 2024-04-23T14:34:23.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:23.813+0000 7ff8bb400700 1 -- 172.21.15.28:0/3881419753 learned_addr learned my addr 172.21.15.28:0/3881419753 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:34:23.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:23.814+0000 7ff8bb400700 1 -- 172.21.15.28:0/3881419753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8bc19ec10 con 0x7ff8bc074e90 2024-04-23T14:34:23.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:23.814+0000 7ff8bb400700 1 --2- 172.21.15.28:0/3881419753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8bc074e90 0x7ff8bc19e6d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac006010 tx=0x7ff8ac005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:23.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:23.814+0000 7ff8b8c00700 1 -- 172.21.15.28:0/3881419753 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff8ac026070 con 0x7ff8bc074e90 2024-04-23T14:34:23.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:23.814+0000 7ff8b8c00700 1 -- 172.21.15.28:0/3881419753 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8ac00bc30 con 0x7ff8bc074e90 2024-04-23T14:34:23.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:23.814+0000 7ff8c2200700 1 -- 172.21.15.28:0/3881419753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8bc19eea0 con 0x7ff8bc074e90 2024-04-23T14:34:23.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:23.814+0000 7ff8b8c00700 1 -- 172.21.15.28:0/3881419753 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff8ac0039a0 con 0x7ff8bc074e90 2024-04-23T14:34:23.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:23.814+0000 7ff8c2200700 1 -- 172.21.15.28:0/3881419753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8bc19f300 con 0x7ff8bc074e90 2024-04-23T14:34:23.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:23.815+0000 7ff8c2200700 1 -- 172.21.15.28:0/3881419753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8bc0765d0 con 0x7ff8bc074e90 2024-04-23T14:34:23.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:23.815+0000 7ff8b8c00700 1 -- 172.21.15.28:0/3881419753 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7ff8ac029070 con 0x7ff8bc074e90 2024-04-23T14:34:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:23.816+0000 7ff8b8c00700 1 --2- 172.21.15.28:0/3881419753 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ff8a00396b0 0x7ff8a003bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:23.816+0000 7ff8b8c00700 1 -- 172.21.15.28:0/3881419753 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff8ac01e070 con 0x7ff8bc074e90 2024-04-23T14:34:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:23.816+0000 7ff8b2a00700 1 --2- 172.21.15.28:0/3881419753 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ff8a00396b0 0x7ff8a003bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:23.817+0000 7ff8b2a00700 1 --2- 172.21.15.28:0/3881419753 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ff8a00396b0 0x7ff8a003bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4006fd0 tx=0x7ff8a4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:23.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:23.822+0000 7ff8b8c00700 1 -- 172.21.15.28:0/3881419753 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff8ac013070 con 0x7ff8bc074e90 2024-04-23T14:34:24.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:24.061+0000 7ff8c2200700 1 -- 172.21.15.28:0/3881419753 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff8bc19fa30 con 0x7ff8a00396b0 2024-04-23T14:34:24.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:24.064+0000 7ff8b8c00700 1 -- 172.21.15.28:0/3881419753 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+239 (secure 0 0 0) 0x7ff8bc19fa30 con 0x7ff8a00396b0 2024-04-23T14:34:24.065 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:34:24.065 INFO:teuthology.orchestra.run.smithi028.stdout:[{"addr": "172.21.15.28", "hostname": "smithi028", "labels": [], "status": ""}, {"addr": "172.21.15.38", "hostname": "smithi038", "labels": [], "status": ""}, {"addr": "172.21.15.105", "hostname": "smithi105", "labels": [], "status": ""}] 2024-04-23T14:34:24.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:24.067+0000 7ff8c2200700 1 -- 172.21.15.28:0/3881419753 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ff8a00396b0 msgr2=0x7ff8a003bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:24.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:24.067+0000 7ff8c2200700 1 --2- 172.21.15.28:0/3881419753 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ff8a00396b0 0x7ff8a003bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4006fd0 tx=0x7ff8a4006e40 comp rx=0 tx=0).stop 2024-04-23T14:34:24.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:24.067+0000 7ff8c2200700 1 -- 172.21.15.28:0/3881419753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8bc074e90 msgr2=0x7ff8bc19e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:24.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:24.067+0000 7ff8c2200700 1 --2- 172.21.15.28:0/3881419753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8bc074e90 0x7ff8bc19e6d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac006010 tx=0x7ff8ac005030 comp rx=0 tx=0).stop 2024-04-23T14:34:24.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:24.067+0000 7ff8c2200700 1 -- 172.21.15.28:0/3881419753 shutdown_connections 2024-04-23T14:34:24.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:24.067+0000 7ff8c2200700 1 --2- 172.21.15.28:0/3881419753 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ff8a00396b0 0x7ff8a003bb70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:24.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:24.067+0000 7ff8c2200700 1 --2- 172.21.15.28:0/3881419753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8bc074e90 0x7ff8bc19e6d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:24.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:24.067+0000 7ff8c2200700 1 -- 172.21.15.28:0/3881419753 >> 172.21.15.28:0/3881419753 conn(0x7ff8bc0fe030 msgr2=0x7ff8bc0ffa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:24.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:24.067+0000 7ff8c2200700 1 -- 172.21.15.28:0/3881419753 shutdown_connections 2024-04-23T14:34:24.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:24.067+0000 7ff8c2200700 1 -- 172.21.15.28:0/3881419753 wait complete. 2024-04-23T14:34:24.749 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:24 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:24.749 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:24 smithi028 ceph-mon[27810]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:24.749 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:24 smithi028 ceph-mon[27810]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T14:34:24.822 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-23T14:34:24.823 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-23T14:34:25.282 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:34:26.772 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:26 smithi028 ceph-mon[27810]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:27.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.244+0000 7f3cd3e00700 1 -- 172.21.15.28:0/380719545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cd41083f0 msgr2=0x7f3cd41087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:27.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.244+0000 7f3cd3e00700 1 --2- 172.21.15.28:0/380719545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cd41083f0 0x7f3cd41087d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4009910 tx=0x7f3cc4009c20 comp rx=0 tx=0).stop 2024-04-23T14:34:27.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.244+0000 7f3cd3e00700 1 -- 172.21.15.28:0/380719545 shutdown_connections 2024-04-23T14:34:27.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.244+0000 7f3cd3e00700 1 --2- 172.21.15.28:0/380719545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cd41083f0 0x7f3cd41087d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:27.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.244+0000 7f3cd3e00700 1 -- 172.21.15.28:0/380719545 >> 172.21.15.28:0/380719545 conn(0x7f3cd4076da0 msgr2=0x7f3cd40771b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:27.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.244+0000 7f3cd3e00700 1 -- 172.21.15.28:0/380719545 shutdown_connections 2024-04-23T14:34:27.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.244+0000 7f3cd3e00700 1 -- 172.21.15.28:0/380719545 wait complete. 2024-04-23T14:34:27.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.245+0000 7f3cd3e00700 1 Processor -- start 2024-04-23T14:34:27.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.245+0000 7f3cd3e00700 1 -- start start 2024-04-23T14:34:27.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.246+0000 7f3cd3e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cd41083f0 0x7f3cd4191ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:27.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.246+0000 7f3cd3e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd410c970 con 0x7f3cd41083f0 2024-04-23T14:34:27.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.246+0000 7f3cd2a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cd41083f0 0x7f3cd4191ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:27.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.246+0000 7f3cd2a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cd41083f0 0x7f3cd4191ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39378/0 (socket says 172.21.15.28:39378) 2024-04-23T14:34:27.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.246+0000 7f3cd2a00700 1 -- 172.21.15.28:0/4100873924 learned_addr learned my addr 172.21.15.28:0/4100873924 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:34:27.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.247+0000 7f3cd2a00700 1 -- 172.21.15.28:0/4100873924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cd4195e30 con 0x7f3cd41083f0 2024-04-23T14:34:27.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.248+0000 7f3cd2a00700 1 --2- 172.21.15.28:0/4100873924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cd41083f0 0x7f3cd4191ed0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4005850 tx=0x7f3cc4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:27.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.248+0000 7f3ccb400700 1 -- 172.21.15.28:0/4100873924 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3cc4026070 con 0x7f3cd41083f0 2024-04-23T14:34:27.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.248+0000 7f3ccb400700 1 -- 172.21.15.28:0/4100873924 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3cc400bbd0 con 0x7f3cd41083f0 2024-04-23T14:34:27.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.248+0000 7f3ccb400700 1 -- 172.21.15.28:0/4100873924 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3cc4003970 con 0x7f3cd41083f0 2024-04-23T14:34:27.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.248+0000 7f3cd3e00700 1 -- 172.21.15.28:0/4100873924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cd4192410 con 0x7f3cd41083f0 2024-04-23T14:34:27.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.248+0000 7f3cd3e00700 1 -- 172.21.15.28:0/4100873924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cd41928d0 con 0x7f3cd41083f0 2024-04-23T14:34:27.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.250+0000 7f3cd3e00700 1 -- 172.21.15.28:0/4100873924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cd4103aa0 con 0x7f3cd41083f0 2024-04-23T14:34:27.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.250+0000 7f3ccb400700 1 -- 172.21.15.28:0/4100873924 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f3cc4029070 con 0x7f3cd41083f0 2024-04-23T14:34:27.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.250+0000 7f3ccb400700 1 --2- 172.21.15.28:0/4100873924 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f3cb80396b0 0x7f3cb803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:27.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.251+0000 7f3ccb400700 1 -- 172.21.15.28:0/4100873924 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3cc401e070 con 0x7f3cd41083f0 2024-04-23T14:34:27.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.251+0000 7f3cd2000700 1 --2- 172.21.15.28:0/4100873924 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f3cb80396b0 0x7f3cb803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:27.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.252+0000 7f3cd2000700 1 --2- 172.21.15.28:0/4100873924 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f3cb80396b0 0x7f3cb803bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3cc0006fd0 tx=0x7f3cc0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:27.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.256+0000 7f3ccb400700 1 -- 172.21.15.28:0/4100873924 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3cc4013070 con 0x7f3cd41083f0 2024-04-23T14:34:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.503+0000 7f3cd3e00700 1 -- 172.21.15.28:0/4100873924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f3cd4064150 con 0x7f3cd41083f0 2024-04-23T14:34:27.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.837+0000 7f3ccb400700 1 -- 172.21.15.28:0/4100873924 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f3cc400fe20 con 0x7f3cd41083f0 2024-04-23T14:34:27.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.842+0000 7f3cd3e00700 1 -- 172.21.15.28:0/4100873924 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f3cb80396b0 msgr2=0x7f3cb803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:27.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.842+0000 7f3cd3e00700 1 --2- 172.21.15.28:0/4100873924 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f3cb80396b0 0x7f3cb803bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3cc0006fd0 tx=0x7f3cc0006e40 comp rx=0 tx=0).stop 2024-04-23T14:34:27.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.842+0000 7f3cd3e00700 1 -- 172.21.15.28:0/4100873924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cd41083f0 msgr2=0x7f3cd4191ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:27.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.842+0000 7f3cd3e00700 1 --2- 172.21.15.28:0/4100873924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cd41083f0 0x7f3cd4191ed0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4005850 tx=0x7f3cc4005030 comp rx=0 tx=0).stop 2024-04-23T14:34:27.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.842+0000 7f3cd3e00700 1 -- 172.21.15.28:0/4100873924 shutdown_connections 2024-04-23T14:34:27.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.842+0000 7f3cd3e00700 1 --2- 172.21.15.28:0/4100873924 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f3cb80396b0 0x7f3cb803bb70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:27.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.842+0000 7f3cd3e00700 1 --2- 172.21.15.28:0/4100873924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cd41083f0 0x7f3cd4191ed0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:27.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.842+0000 7f3cd3e00700 1 -- 172.21.15.28:0/4100873924 >> 172.21.15.28:0/4100873924 conn(0x7f3cd4076da0 msgr2=0x7f3cd410aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:27.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.843+0000 7f3cd3e00700 1 -- 172.21.15.28:0/4100873924 shutdown_connections 2024-04-23T14:34:27.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:27.843+0000 7f3cd3e00700 1 -- 172.21.15.28:0/4100873924 wait complete. 2024-04-23T14:34:27.845 INFO:teuthology.orchestra.run.smithi028.stderr:adjusted tunables profile to default 2024-04-23T14:34:28.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:27 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:28.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:27 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:28.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:27 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:28.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:27 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:28.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:27 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi105", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:34:28.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:27 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:34:28.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:27 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:34:28.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:27 smithi028 ceph-mon[27810]: Updating smithi105:/etc/ceph/ceph.conf 2024-04-23T14:34:28.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:27 smithi028 ceph-mon[27810]: Updating smithi105:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T14:34:28.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:27 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/4100873924' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-23T14:34:28.623 INFO:tasks.cephadm:Adding mon.a on smithi028 2024-04-23T14:34:28.623 INFO:tasks.cephadm:Adding mon.b on smithi038 2024-04-23T14:34:28.623 INFO:tasks.cephadm:Adding mon.c on smithi105 2024-04-23T14:34:28.624 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi028:172.21.15.28=a;smithi038:172.21.15.38=b;smithi105:172.21.15.105=c' 2024-04-23T14:34:28.849 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T14:34:29.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:28 smithi028 ceph-mon[27810]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-04-23T14:34:29.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:28 smithi028 ceph-mon[27810]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:29.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:28 smithi028 ceph-mon[27810]: Updating smithi105:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T14:34:29.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:28 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/4100873924' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-23T14:34:29.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:28 smithi028 ceph-mon[27810]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T14:34:29.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:29.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:29.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:30.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.239+0000 7f78eba8e700 1 -- 172.21.15.105:0/2225857011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78e4107480 msgr2=0x7f78e4107860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:30.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.239+0000 7f78eba8e700 1 --2- 172.21.15.105:0/2225857011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78e4107480 0x7f78e4107860 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f78d8009910 tx=0x7f78d8009c20 comp rx=0 tx=0).stop 2024-04-23T14:34:30.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.239+0000 7f78eba8e700 1 -- 172.21.15.105:0/2225857011 shutdown_connections 2024-04-23T14:34:30.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.239+0000 7f78eba8e700 1 --2- 172.21.15.105:0/2225857011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78e4107480 0x7f78e4107860 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:30.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.239+0000 7f78eba8e700 1 -- 172.21.15.105:0/2225857011 >> 172.21.15.105:0/2225857011 conn(0x7f78e4076f50 msgr2=0x7f78e4077360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:30.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.239+0000 7f78eba8e700 1 -- 172.21.15.105:0/2225857011 shutdown_connections 2024-04-23T14:34:30.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.239+0000 7f78eba8e700 1 -- 172.21.15.105:0/2225857011 wait complete. 2024-04-23T14:34:30.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.239+0000 7f78eba8e700 1 Processor -- start 2024-04-23T14:34:30.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.239+0000 7f78eba8e700 1 -- start start 2024-04-23T14:34:30.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.240+0000 7f78eba8e700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78e4107480 0x7f78e4197000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:30.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.240+0000 7f78eba8e700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f78e410ba30 con 0x7f78e4107480 2024-04-23T14:34:30.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.240+0000 7f78e982a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78e4107480 0x7f78e4197000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:30.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.240+0000 7f78e982a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78e4107480 0x7f78e4197000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.105:52920/0 (socket says 172.21.15.105:52920) 2024-04-23T14:34:30.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.240+0000 7f78e982a700 1 -- 172.21.15.105:0/860443874 learned_addr learned my addr 172.21.15.105:0/860443874 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-23T14:34:30.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.241+0000 7f78e982a700 1 -- 172.21.15.105:0/860443874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78e4197540 con 0x7f78e4107480 2024-04-23T14:34:30.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.241+0000 7f78e982a700 1 --2- 172.21.15.105:0/860443874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78e4107480 0x7f78e4197000 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f78d8005850 tx=0x7f78d8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:30.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.241+0000 7f78d6ffd700 1 -- 172.21.15.105:0/860443874 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f78d8028070 con 0x7f78e4107480 2024-04-23T14:34:30.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.241+0000 7f78eba8e700 1 -- 172.21.15.105:0/860443874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78e41977d0 con 0x7f78e4107480 2024-04-23T14:34:30.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.241+0000 7f78eba8e700 1 -- 172.21.15.105:0/860443874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78e419b8d0 con 0x7f78e4107480 2024-04-23T14:34:30.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.241+0000 7f78d6ffd700 1 -- 172.21.15.105:0/860443874 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78d800b6f0 con 0x7f78e4107480 2024-04-23T14:34:30.244 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.242+0000 7f78d6ffd700 1 -- 172.21.15.105:0/860443874 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f78d8032060 con 0x7f78e4107480 2024-04-23T14:34:30.244 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.242+0000 7f78eba8e700 1 -- 172.21.15.105:0/860443874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78c8005330 con 0x7f78e4107480 2024-04-23T14:34:30.244 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.243+0000 7f78d6ffd700 1 -- 172.21.15.105:0/860443874 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f78d802b070 con 0x7f78e4107480 2024-04-23T14:34:30.245 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.243+0000 7f78d6ffd700 1 --2- 172.21.15.105:0/860443874 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f78d0039650 0x7f78d003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:30.245 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.243+0000 7f78d6ffd700 1 -- 172.21.15.105:0/860443874 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f78d8020070 con 0x7f78e4107480 2024-04-23T14:34:30.245 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.243+0000 7f78e9029700 1 --2- 172.21.15.105:0/860443874 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f78d0039650 0x7f78d003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:30.245 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.244+0000 7f78e9029700 1 --2- 172.21.15.105:0/860443874 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f78d0039650 0x7f78d003bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f78e0006fd0 tx=0x7f78e0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:30.258 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.258+0000 7f78d6ffd700 1 -- 172.21.15.105:0/860443874 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f78d8013070 con 0x7f78e4107480 2024-04-23T14:34:30.473 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.472+0000 7f78eba8e700 1 -- 172.21.15.105:0/860443874 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi038:172.21.15.38=b;smithi105:172.21.15.105=c", "target": ["mon-mgr", ""]}) -- 0x7f78c8000c80 con 0x7f78d0039650 2024-04-23T14:34:30.488 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.488+0000 7f78d6ffd700 1 -- 172.21.15.105:0/860443874 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f78c8000c80 con 0x7f78d0039650 2024-04-23T14:34:30.489 INFO:teuthology.orchestra.run.smithi105.stdout:Scheduled mon update... 2024-04-23T14:34:30.491 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.490+0000 7f78eba8e700 1 -- 172.21.15.105:0/860443874 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f78d0039650 msgr2=0x7f78d003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:30.491 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.490+0000 7f78eba8e700 1 --2- 172.21.15.105:0/860443874 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f78d0039650 0x7f78d003bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f78e0006fd0 tx=0x7f78e0006e40 comp rx=0 tx=0).stop 2024-04-23T14:34:30.491 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.490+0000 7f78eba8e700 1 -- 172.21.15.105:0/860443874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78e4107480 msgr2=0x7f78e4197000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:30.491 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.490+0000 7f78eba8e700 1 --2- 172.21.15.105:0/860443874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78e4107480 0x7f78e4197000 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f78d8005850 tx=0x7f78d8005030 comp rx=0 tx=0).stop 2024-04-23T14:34:30.491 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.490+0000 7f78eba8e700 1 -- 172.21.15.105:0/860443874 shutdown_connections 2024-04-23T14:34:30.491 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.490+0000 7f78eba8e700 1 --2- 172.21.15.105:0/860443874 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f78d0039650 0x7f78d003bb10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:30.491 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.490+0000 7f78eba8e700 1 --2- 172.21.15.105:0/860443874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78e4107480 0x7f78e4197000 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:30.491 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.490+0000 7f78eba8e700 1 -- 172.21.15.105:0/860443874 >> 172.21.15.105:0/860443874 conn(0x7f78e4076f50 msgr2=0x7f78e4102f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:30.491 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.490+0000 7f78eba8e700 1 -- 172.21.15.105:0/860443874 shutdown_connections 2024-04-23T14:34:30.491 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:30.490+0000 7f78eba8e700 1 -- 172.21.15.105:0/860443874 wait complete. 2024-04-23T14:34:31.029 DEBUG:teuthology.orchestra.run.smithi038:mon.b> sudo journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mon.b.service 2024-04-23T14:34:31.032 DEBUG:teuthology.orchestra.run.smithi105:mon.c> sudo journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mon.c.service 2024-04-23T14:34:31.034 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-23T14:34:31.035 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-23T14:34:31.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:30 smithi028 ceph-mon[27810]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:31.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:30 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:31.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:30 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:34:31.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:30 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:34:31.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:30 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:34:31.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:30 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:31.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:30 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T14:34:31.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:30 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:34:32.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:31 smithi028 ceph-mon[27810]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi038:172.21.15.38=b;smithi105:172.21.15.105=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:34:32.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:31 smithi028 ceph-mon[27810]: Saving service mon spec with placement smithi028:172.21.15.28=a;smithi038:172.21.15.38=b;smithi105:172.21.15.105=c;count:3 2024-04-23T14:34:32.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:31 smithi028 ceph-mon[27810]: Deploying daemon mon.c on smithi105 2024-04-23T14:34:32.845 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T14:34:33.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:32 smithi028 ceph-mon[27810]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:34.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:34.984+0000 7fd42fec6700 1 -- 172.21.15.105:0/3437271368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4281074a0 msgr2=0x7fd428107880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:34.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:34.984+0000 7fd42fec6700 1 --2- 172.21.15.105:0/3437271368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4281074a0 0x7fd428107880 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd424009910 tx=0x7fd424009c20 comp rx=0 tx=0).stop 2024-04-23T14:34:34.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:34.984+0000 7fd42fec6700 1 -- 172.21.15.105:0/3437271368 shutdown_connections 2024-04-23T14:34:34.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:34.984+0000 7fd42fec6700 1 --2- 172.21.15.105:0/3437271368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4281074a0 0x7fd428107880 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:34.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:34.984+0000 7fd42fec6700 1 -- 172.21.15.105:0/3437271368 >> 172.21.15.105:0/3437271368 conn(0x7fd4280fcf40 msgr2=0x7fd4280ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:34.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:34.984+0000 7fd42fec6700 1 -- 172.21.15.105:0/3437271368 shutdown_connections 2024-04-23T14:34:34.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:34.984+0000 7fd42fec6700 1 -- 172.21.15.105:0/3437271368 wait complete. 2024-04-23T14:34:34.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:34.985+0000 7fd42fec6700 1 Processor -- start 2024-04-23T14:34:34.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:34.985+0000 7fd42fec6700 1 -- start start 2024-04-23T14:34:34.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:34.985+0000 7fd42fec6700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4281074a0 0x7fd428196c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:34.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:34.985+0000 7fd42fec6700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd42810ba50 con 0x7fd4281074a0 2024-04-23T14:34:34.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:34.986+0000 7fd42dc62700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4281074a0 0x7fd428196c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:34.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:34.986+0000 7fd42dc62700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4281074a0 0x7fd428196c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.105:48266/0 (socket says 172.21.15.105:48266) 2024-04-23T14:34:34.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:34.986+0000 7fd42dc62700 1 -- 172.21.15.105:0/3376060822 learned_addr learned my addr 172.21.15.105:0/3376060822 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-23T14:34:34.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:34.987+0000 7fd42dc62700 1 -- 172.21.15.105:0/3376060822 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd42806b700 con 0x7fd4281074a0 2024-04-23T14:34:34.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:34.987+0000 7fd42dc62700 1 --2- 172.21.15.105:0/3376060822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4281074a0 0x7fd428196c00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd424005850 tx=0x7fd424005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:34.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:34.988+0000 7fd41effd700 1 -- 172.21.15.105:0/3376060822 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd424026070 con 0x7fd4281074a0 2024-04-23T14:34:34.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:34.988+0000 7fd42fec6700 1 -- 172.21.15.105:0/3376060822 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd428197140 con 0x7fd4281074a0 2024-04-23T14:34:34.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:34.988+0000 7fd42fec6700 1 -- 172.21.15.105:0/3376060822 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd428197600 con 0x7fd4281074a0 2024-04-23T14:34:34.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:34.988+0000 7fd41effd700 1 -- 172.21.15.105:0/3376060822 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd42400bbd0 con 0x7fd4281074a0 2024-04-23T14:34:34.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:34.988+0000 7fd41effd700 1 -- 172.21.15.105:0/3376060822 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd4240039b0 con 0x7fd4281074a0 2024-04-23T14:34:34.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:34.989+0000 7fd42fec6700 1 -- 172.21.15.105:0/3376060822 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd40c005360 con 0x7fd4281074a0 2024-04-23T14:34:34.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:34.989+0000 7fd41effd700 1 -- 172.21.15.105:0/3376060822 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fd424029070 con 0x7fd4281074a0 2024-04-23T14:34:34.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:34.989+0000 7fd41effd700 1 --2- 172.21.15.105:0/3376060822 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd414039650 0x7fd41403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:34.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:34.990+0000 7fd41effd700 1 -- 172.21.15.105:0/3376060822 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fd42401e070 con 0x7fd4281074a0 2024-04-23T14:34:34.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:34.990+0000 7fd42d461700 1 --2- 172.21.15.105:0/3376060822 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd414039650 0x7fd41403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:34.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:34.991+0000 7fd42d461700 1 --2- 172.21.15.105:0/3376060822 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd414039650 0x7fd41403bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd418006fd0 tx=0x7fd418006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:35.006 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:35.006+0000 7fd41effd700 1 -- 172.21.15.105:0/3376060822 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd424013070 con 0x7fd4281074a0 2024-04-23T14:34:35.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:34 smithi028 ceph-mon[27810]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:35.301 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:35.299+0000 7fd42fec6700 1 -- 172.21.15.105:0/3376060822 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fd40c005920 con 0x7fd4281074a0 2024-04-23T14:34:35.301 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:35.300+0000 7fd41effd700 1 -- 172.21.15.105:0/3376060822 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7fd424034090 con 0x7fd4281074a0 2024-04-23T14:34:35.301 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-23T14:34:35.301 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":1,"fsid":"1ee6e996-017e-11ef-bc93-c7b262605968","modified":"2024-04-23T14:32:30.337765Z","created":"2024-04-23T14:32:30.337765Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-23T14:34:35.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:35.303+0000 7fd42fec6700 1 -- 172.21.15.105:0/3376060822 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd414039650 msgr2=0x7fd41403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:35.304 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:35.303+0000 7fd42fec6700 1 --2- 172.21.15.105:0/3376060822 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd414039650 0x7fd41403bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd418006fd0 tx=0x7fd418006e40 comp rx=0 tx=0).stop 2024-04-23T14:34:35.304 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:35.303+0000 7fd42fec6700 1 -- 172.21.15.105:0/3376060822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4281074a0 msgr2=0x7fd428196c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:35.304 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:35.303+0000 7fd42fec6700 1 --2- 172.21.15.105:0/3376060822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4281074a0 0x7fd428196c00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd424005850 tx=0x7fd424005030 comp rx=0 tx=0).stop 2024-04-23T14:34:35.304 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:35.303+0000 7fd42fec6700 1 -- 172.21.15.105:0/3376060822 shutdown_connections 2024-04-23T14:34:35.304 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:35.303+0000 7fd42fec6700 1 --2- 172.21.15.105:0/3376060822 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd414039650 0x7fd41403bb10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:35.304 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:35.303+0000 7fd42fec6700 1 --2- 172.21.15.105:0/3376060822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4281074a0 0x7fd428196c00 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:35.304 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:35.303+0000 7fd42fec6700 1 -- 172.21.15.105:0/3376060822 >> 172.21.15.105:0/3376060822 conn(0x7fd4280fcf40 msgr2=0x7fd428108f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:35.304 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:35.303+0000 7fd42fec6700 1 -- 172.21.15.105:0/3376060822 shutdown_connections 2024-04-23T14:34:35.304 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:35.303+0000 7fd42fec6700 1 -- 172.21.15.105:0/3376060822 wait complete. 2024-04-23T14:34:35.304 INFO:teuthology.orchestra.run.smithi105.stderr:dumped monmap epoch 1 2024-04-23T14:34:35.364 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 podman[32515]: 2024-04-23 14:34:35.340411041 +0000 UTC m=+0.352425683 container start 4684ef5a77bf23e76c8c169c169f128211ff4fa4bd86f95bb0d4e0bf9ae042ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mon-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD) 2024-04-23T14:34:35.364 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: set uid:gid to 167:167 (ceph:ceph) 2024-04-23T14:34:35.364 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: ceph version 19.0.0-3126-gd44fddd9 (d44fddd910b5e94da0f2357af4be37d7bd495ae0) squid (dev), process ceph-mon, pid 2 2024-04-23T14:34:35.364 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: pidfile_write: ignore empty --pid-file 2024-04-23T14:34:35.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: load: jerasure load: lrc 2024-04-23T14:34:35.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: RocksDB version: 7.9.2 2024-04-23T14:34:35.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Git sha 0 2024-04-23T14:34:35.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Compile date 2024-04-22 13:15:13 2024-04-23T14:34:35.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: DB SUMMARY 2024-04-23T14:34:35.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: DB Session ID: U7MAXK8BQ3PQ9EDMS7X0 2024-04-23T14:34:35.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: CURRENT file: CURRENT 2024-04-23T14:34:35.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: IDENTITY file: IDENTITY 2024-04-23T14:34:35.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-23T14:34:35.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-23T14:34:35.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-23T14:34:35.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.error_if_exists: 0 2024-04-23T14:34:35.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.create_if_missing: 0 2024-04-23T14:34:35.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.paranoid_checks: 1 2024-04-23T14:34:35.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-23T14:34:35.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-23T14:34:35.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-23T14:34:35.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.env: 0x56419d230cc0 2024-04-23T14:34:35.789 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.fs: PosixFileSystem 2024-04-23T14:34:35.789 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.info_log: 0x56419f27fc20 2024-04-23T14:34:35.789 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.max_file_opening_threads: 16 2024-04-23T14:34:35.789 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.statistics: (nil) 2024-04-23T14:34:35.789 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.use_fsync: 0 2024-04-23T14:34:35.789 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.max_log_file_size: 0 2024-04-23T14:34:35.789 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-23T14:34:35.789 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-23T14:34:35.789 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.keep_log_file_num: 1000 2024-04-23T14:34:35.789 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.recycle_log_file_num: 0 2024-04-23T14:34:35.789 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.allow_fallocate: 1 2024-04-23T14:34:35.789 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.allow_mmap_reads: 0 2024-04-23T14:34:35.790 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.allow_mmap_writes: 0 2024-04-23T14:34:35.790 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.use_direct_reads: 0 2024-04-23T14:34:35.790 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-23T14:34:35.790 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.create_missing_column_families: 0 2024-04-23T14:34:35.790 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.db_log_dir: 2024-04-23T14:34:35.790 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.wal_dir: 2024-04-23T14:34:35.790 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-23T14:34:35.790 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-23T14:34:35.790 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-23T14:34:35.790 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-23T14:34:35.791 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-23T14:34:35.791 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-23T14:34:35.791 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.advise_random_on_open: 1 2024-04-23T14:34:35.791 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.db_write_buffer_size: 0 2024-04-23T14:34:35.791 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.write_buffer_manager: 0x56419f24e1e0 2024-04-23T14:34:35.791 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-23T14:34:35.791 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-23T14:34:35.791 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-23T14:34:35.791 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.rate_limiter: (nil) 2024-04-23T14:34:35.791 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-23T14:34:35.791 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.wal_recovery_mode: 2 2024-04-23T14:34:35.791 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.enable_thread_tracking: 0 2024-04-23T14:34:35.791 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.enable_pipelined_write: 0 2024-04-23T14:34:35.791 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.unordered_write: 0 2024-04-23T14:34:35.792 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-23T14:34:35.792 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-23T14:34:35.792 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-23T14:34:35.792 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-23T14:34:35.792 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.row_cache: None 2024-04-23T14:34:35.792 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.wal_filter: None 2024-04-23T14:34:35.792 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-23T14:34:35.792 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.allow_ingest_behind: 0 2024-04-23T14:34:35.792 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.two_write_queues: 0 2024-04-23T14:34:35.792 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.manual_wal_flush: 0 2024-04-23T14:34:35.792 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.wal_compression: 0 2024-04-23T14:34:35.792 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.atomic_flush: 0 2024-04-23T14:34:35.792 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-23T14:34:35.792 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-23T14:34:35.793 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-23T14:34:35.793 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.log_readahead_size: 0 2024-04-23T14:34:35.793 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-23T14:34:35.793 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.best_efforts_recovery: 0 2024-04-23T14:34:35.793 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-23T14:34:35.793 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-23T14:34:35.793 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.allow_data_in_errors: 0 2024-04-23T14:34:35.793 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.db_host_id: __hostname__ 2024-04-23T14:34:35.793 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-23T14:34:35.793 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.max_background_jobs: 2 2024-04-23T14:34:35.793 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.max_background_compactions: -1 2024-04-23T14:34:35.793 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.max_subcompactions: 1 2024-04-23T14:34:35.795 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-23T14:34:35.795 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-23T14:34:35.795 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-23T14:34:35.795 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.max_total_wal_size: 0 2024-04-23T14:34:35.795 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-23T14:34:35.795 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-23T14:34:35.796 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-23T14:34:35.796 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-23T14:34:35.796 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.max_open_files: -1 2024-04-23T14:34:35.796 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.bytes_per_sync: 0 2024-04-23T14:34:35.796 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-23T14:34:35.796 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-23T14:34:35.796 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.compaction_readahead_size: 0 2024-04-23T14:34:35.796 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.max_background_flushes: -1 2024-04-23T14:34:35.796 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Compression algorithms supported: 2024-04-23T14:34:35.796 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-23T14:34:35.796 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: kZSTD supported: 0 2024-04-23T14:34:35.796 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: kXpressCompression supported: 0 2024-04-23T14:34:35.796 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: kLZ4HCCompression supported: 1 2024-04-23T14:34:35.797 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: kZlibCompression supported: 1 2024-04-23T14:34:35.797 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: kSnappyCompression supported: 1 2024-04-23T14:34:35.797 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: kLZ4Compression supported: 1 2024-04-23T14:34:35.797 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: kBZip2Compression supported: 0 2024-04-23T14:34:35.797 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-23T14:34:35.797 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-23T14:34:35.797 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-04-23T14:34:35.797 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-23T14:34:35.797 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-23T14:34:35.797 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.merge_operator: 2024-04-23T14:34:35.797 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.compaction_filter: None 2024-04-23T14:34:35.798 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.compaction_filter_factory: None 2024-04-23T14:34:35.798 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.sst_partitioner_factory: None 2024-04-23T14:34:35.798 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-23T14:34:35.798 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-23T14:34:35.798 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56419f27fde0) 2024-04-23T14:34:35.798 INFO:journalctl@ceph.mon.c.smithi105.stdout: cache_index_and_filter_blocks: 1 2024-04-23T14:34:35.798 INFO:journalctl@ceph.mon.c.smithi105.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-23T14:34:35.798 INFO:journalctl@ceph.mon.c.smithi105.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-23T14:34:35.798 INFO:journalctl@ceph.mon.c.smithi105.stdout: pin_top_level_index_and_filter: 1 2024-04-23T14:34:35.798 INFO:journalctl@ceph.mon.c.smithi105.stdout: index_type: 0 2024-04-23T14:34:35.798 INFO:journalctl@ceph.mon.c.smithi105.stdout: data_block_index_type: 0 2024-04-23T14:34:35.798 INFO:journalctl@ceph.mon.c.smithi105.stdout: index_shortening: 1 2024-04-23T14:34:35.798 INFO:journalctl@ceph.mon.c.smithi105.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-23T14:34:35.799 INFO:journalctl@ceph.mon.c.smithi105.stdout: checksum: 4 2024-04-23T14:34:35.799 INFO:journalctl@ceph.mon.c.smithi105.stdout: no_block_cache: 0 2024-04-23T14:34:35.799 INFO:journalctl@ceph.mon.c.smithi105.stdout: block_cache: 0x56419f281090 2024-04-23T14:34:35.799 INFO:journalctl@ceph.mon.c.smithi105.stdout: block_cache_name: BinnedLRUCache 2024-04-23T14:34:35.799 INFO:journalctl@ceph.mon.c.smithi105.stdout: block_cache_options: 2024-04-23T14:34:35.799 INFO:journalctl@ceph.mon.c.smithi105.stdout: capacity : 536870912 2024-04-23T14:34:35.799 INFO:journalctl@ceph.mon.c.smithi105.stdout: num_shard_bits : 4 2024-04-23T14:34:35.799 INFO:journalctl@ceph.mon.c.smithi105.stdout: strict_capacity_limit : 0 2024-04-23T14:34:35.799 INFO:journalctl@ceph.mon.c.smithi105.stdout: high_pri_pool_ratio: 0.000 2024-04-23T14:34:35.799 INFO:journalctl@ceph.mon.c.smithi105.stdout: block_cache_compressed: (nil) 2024-04-23T14:34:35.799 INFO:journalctl@ceph.mon.c.smithi105.stdout: persistent_cache: (nil) 2024-04-23T14:34:35.799 INFO:journalctl@ceph.mon.c.smithi105.stdout: block_size: 4096 2024-04-23T14:34:35.799 INFO:journalctl@ceph.mon.c.smithi105.stdout: block_size_deviation: 10 2024-04-23T14:34:35.799 INFO:journalctl@ceph.mon.c.smithi105.stdout: block_restart_interval: 16 2024-04-23T14:34:35.799 INFO:journalctl@ceph.mon.c.smithi105.stdout: index_block_restart_interval: 1 2024-04-23T14:34:35.800 INFO:journalctl@ceph.mon.c.smithi105.stdout: metadata_block_size: 4096 2024-04-23T14:34:35.800 INFO:journalctl@ceph.mon.c.smithi105.stdout: partition_filters: 0 2024-04-23T14:34:35.800 INFO:journalctl@ceph.mon.c.smithi105.stdout: use_delta_encoding: 1 2024-04-23T14:34:35.800 INFO:journalctl@ceph.mon.c.smithi105.stdout: filter_policy: bloomfilter 2024-04-23T14:34:35.800 INFO:journalctl@ceph.mon.c.smithi105.stdout: whole_key_filtering: 1 2024-04-23T14:34:35.800 INFO:journalctl@ceph.mon.c.smithi105.stdout: verify_compression: 0 2024-04-23T14:34:35.800 INFO:journalctl@ceph.mon.c.smithi105.stdout: read_amp_bytes_per_bit: 0 2024-04-23T14:34:35.800 INFO:journalctl@ceph.mon.c.smithi105.stdout: format_version: 5 2024-04-23T14:34:35.800 INFO:journalctl@ceph.mon.c.smithi105.stdout: enable_index_compression: 1 2024-04-23T14:34:35.800 INFO:journalctl@ceph.mon.c.smithi105.stdout: block_align: 0 2024-04-23T14:34:35.800 INFO:journalctl@ceph.mon.c.smithi105.stdout: max_auto_readahead_size: 262144 2024-04-23T14:34:35.800 INFO:journalctl@ceph.mon.c.smithi105.stdout: prepopulate_block_cache: 0 2024-04-23T14:34:35.800 INFO:journalctl@ceph.mon.c.smithi105.stdout: initial_auto_readahead_size: 8192 2024-04-23T14:34:35.800 INFO:journalctl@ceph.mon.c.smithi105.stdout: num_file_reads_for_auto_readahead: 2 2024-04-23T14:34:35.801 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.write_buffer_size: 33554432 2024-04-23T14:34:35.801 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.max_write_buffer_number: 2 2024-04-23T14:34:35.801 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.compression: NoCompression 2024-04-23T14:34:35.801 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.bottommost_compression: Disabled 2024-04-23T14:34:35.801 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.prefix_extractor: nullptr 2024-04-23T14:34:35.801 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-23T14:34:35.801 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.num_levels: 7 2024-04-23T14:34:35.801 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-23T14:34:35.801 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-23T14:34:35.801 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-23T14:34:35.801 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-23T14:34:35.801 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-23T14:34:35.801 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-23T14:34:35.801 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-23T14:34:35.802 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-23T14:34:35.802 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-23T14:34:35.802 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-23T14:34:35.802 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-23T14:34:35.802 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-23T14:34:35.802 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-23T14:34:35.802 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.compression_opts.level: 32767 2024-04-23T14:34:35.802 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.compression_opts.strategy: 0 2024-04-23T14:34:35.802 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-23T14:34:35.802 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-23T14:34:35.802 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-23T14:34:35.802 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-23T14:34:35.802 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.compression_opts.enabled: false 2024-04-23T14:34:35.803 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-23T14:34:35.803 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-23T14:34:35.803 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-23T14:34:35.803 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-23T14:34:35.803 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.target_file_size_base: 67108864 2024-04-23T14:34:35.803 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-23T14:34:35.803 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-23T14:34:35.803 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-23T14:34:35.803 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-23T14:34:35.803 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-23T14:34:35.803 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-23T14:34:35.803 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-23T14:34:35.803 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-23T14:34:35.803 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-23T14:34:35.803 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-23T14:34:35.803 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-23T14:34:35.804 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-23T14:34:35.804 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-23T14:34:35.804 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-23T14:34:35.804 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.arena_block_size: 1048576 2024-04-23T14:34:35.804 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-23T14:34:35.804 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-23T14:34:35.804 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.disable_auto_compactions: 0 2024-04-23T14:34:35.804 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-23T14:34:35.804 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-23T14:34:35.804 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-23T14:34:35.804 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-23T14:34:35.804 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-23T14:34:35.804 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-23T14:34:35.804 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-23T14:34:35.804 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-23T14:34:35.804 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-23T14:34:35.805 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-23T14:34:35.805 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-23T14:34:35.805 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.inplace_update_support: 0 2024-04-23T14:34:35.805 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-23T14:34:35.805 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-23T14:34:35.805 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-23T14:34:35.805 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-23T14:34:35.805 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.bloom_locality: 0 2024-04-23T14:34:35.805 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.max_successive_merges: 0 2024-04-23T14:34:35.805 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-23T14:34:35.805 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.paranoid_file_checks: 0 2024-04-23T14:34:35.805 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.force_consistency_checks: 1 2024-04-23T14:34:35.805 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.report_bg_io_stats: 0 2024-04-23T14:34:35.805 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.ttl: 2592000 2024-04-23T14:34:35.805 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-23T14:34:35.806 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-23T14:34:35.806 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-23T14:34:35.806 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.enable_blob_files: false 2024-04-23T14:34:35.806 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.min_blob_size: 0 2024-04-23T14:34:35.806 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.blob_file_size: 268435456 2024-04-23T14:34:35.806 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-23T14:34:35.806 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-23T14:34:35.806 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-23T14:34:35.806 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-23T14:34:35.806 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-23T14:34:35.806 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.blob_file_starting_level: 0 2024-04-23T14:34:35.806 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-23T14:34:35.806 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-23T14:34:35.807 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-23T14:34:35.807 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 37d7c60b-91ee-4dd3-b7c1-41fc47cc169f 2024-04-23T14:34:35.807 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713882875370767, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-23T14:34:35.807 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-23T14:34:35.807 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713882875371173, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713882875, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "37d7c60b-91ee-4dd3-b7c1-41fc47cc169f", "db_session_id": "U7MAXK8BQ3PQ9EDMS7X0", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-23T14:34:35.807 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713882875371251, "job": 1, "event": "recovery_finished"} 2024-04-23T14:34:35.807 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-23T14:34:35.807 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-23T14:34:35.807 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56419f36e000 2024-04-23T14:34:35.807 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: DB pointer 0x56419f358000 2024-04-23T14:34:35.807 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-23T14:34:35.807 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-23T14:34:35.807 INFO:journalctl@ceph.mon.c.smithi105.stdout: ** DB Stats ** 2024-04-23T14:34:35.808 INFO:journalctl@ceph.mon.c.smithi105.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T14:34:35.808 INFO:journalctl@ceph.mon.c.smithi105.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-23T14:34:35.808 INFO:journalctl@ceph.mon.c.smithi105.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T14:34:35.808 INFO:journalctl@ceph.mon.c.smithi105.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T14:34:35.808 INFO:journalctl@ceph.mon.c.smithi105.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-23T14:34:35.808 INFO:journalctl@ceph.mon.c.smithi105.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T14:34:35.808 INFO:journalctl@ceph.mon.c.smithi105.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T14:34:35.808 INFO:journalctl@ceph.mon.c.smithi105.stdout: 2024-04-23T14:34:35.808 INFO:journalctl@ceph.mon.c.smithi105.stdout: ** Compaction Stats [default] ** 2024-04-23T14:34:35.808 INFO:journalctl@ceph.mon.c.smithi105.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-23T14:34:35.808 INFO:journalctl@ceph.mon.c.smithi105.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-23T14:34:35.808 INFO:journalctl@ceph.mon.c.smithi105.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-23T14:34:35.808 INFO:journalctl@ceph.mon.c.smithi105.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-23T14:34:35.808 INFO:journalctl@ceph.mon.c.smithi105.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-23T14:34:35.809 INFO:journalctl@ceph.mon.c.smithi105.stdout: 2024-04-23T14:34:35.809 INFO:journalctl@ceph.mon.c.smithi105.stdout: ** Compaction Stats [default] ** 2024-04-23T14:34:35.809 INFO:journalctl@ceph.mon.c.smithi105.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-23T14:34:35.809 INFO:journalctl@ceph.mon.c.smithi105.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-23T14:34:35.809 INFO:journalctl@ceph.mon.c.smithi105.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-23T14:34:35.809 INFO:journalctl@ceph.mon.c.smithi105.stdout: 2024-04-23T14:34:35.809 INFO:journalctl@ceph.mon.c.smithi105.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-23T14:34:35.809 INFO:journalctl@ceph.mon.c.smithi105.stdout: 2024-04-23T14:34:35.809 INFO:journalctl@ceph.mon.c.smithi105.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T14:34:35.809 INFO:journalctl@ceph.mon.c.smithi105.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-23T14:34:35.809 INFO:journalctl@ceph.mon.c.smithi105.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-23T14:34:35.809 INFO:journalctl@ceph.mon.c.smithi105.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-23T14:34:35.809 INFO:journalctl@ceph.mon.c.smithi105.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-23T14:34:35.809 INFO:journalctl@ceph.mon.c.smithi105.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-23T14:34:35.809 INFO:journalctl@ceph.mon.c.smithi105.stdout: Cumulative compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T14:34:35.810 INFO:journalctl@ceph.mon.c.smithi105.stdout: Interval compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T14:34:35.810 INFO:journalctl@ceph.mon.c.smithi105.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-23T14:34:35.810 INFO:journalctl@ceph.mon.c.smithi105.stdout: Block cache BinnedLRUCache@0x56419f281090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-23T14:34:35.810 INFO:journalctl@ceph.mon.c.smithi105.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-23T14:34:35.810 INFO:journalctl@ceph.mon.c.smithi105.stdout: 2024-04-23T14:34:35.810 INFO:journalctl@ceph.mon.c.smithi105.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-23T14:34:35.810 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-23T14:34:35.810 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: using public_addr v2:172.21.15.105:0/0 -> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] 2024-04-23T14:34:35.810 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: starting mon.c rank -1 at public addrs [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] at bind addrs [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T14:34:35.810 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: mon.c@-1(???) e0 preinit fsid 1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T14:34:35.810 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: mon.c@-1(synchronizing).mds e1 new map 2024-04-23T14:34:35.810 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-23T14:34:35.810 INFO:journalctl@ceph.mon.c.smithi105.stdout: e1 2024-04-23T14:34:35.810 INFO:journalctl@ceph.mon.c.smithi105.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-23T14:34:35.811 INFO:journalctl@ceph.mon.c.smithi105.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-23T14:34:35.811 INFO:journalctl@ceph.mon.c.smithi105.stdout: legacy client fscid: -1 2024-04-23T14:34:35.811 INFO:journalctl@ceph.mon.c.smithi105.stdout: 2024-04-23T14:34:35.811 INFO:journalctl@ceph.mon.c.smithi105.stdout: No filesystems configured 2024-04-23T14:34:35.811 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-23T14:34:35.811 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-23T14:34:35.811 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-23T14:34:35.811 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-23T14:34:35.811 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-23T14:34:35.811 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-23T14:34:35.811 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-23T14:34:35.811 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T14:34:35.811 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T14:34:35.811 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T14:34:35.811 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:35.812 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:35.812 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:34:35.812 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:35.812 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: Added host smithi038 2024-04-23T14:34:35.812 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:35.812 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:35.812 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:35.812 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:35.812 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T14:34:35.812 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:35.812 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:35.812 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:35.812 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:35.812 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:35.813 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:34:35.813 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:34:35.813 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:34:35.813 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-23T14:34:35.813 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: Updating smithi038:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T14:34:35.813 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-04-23T14:34:35.813 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:35.813 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: Updating smithi038:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T14:34:35.813 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi105", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:34:35.813 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:35.813 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:35.813 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:35.813 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: Deploying cephadm binary to smithi105 2024-04-23T14:34:35.813 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:35.813 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:35.813 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:35.814 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:34:35.814 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: Added host smithi105 2024-04-23T14:34:35.814 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:35.814 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:35.814 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:35.814 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:35.814 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:35.814 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T14:34:35.814 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:35.814 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:35.814 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:35.814 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:35.814 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:35.814 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi105", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:34:35.815 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:34:35.815 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:34:35.815 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: Updating smithi105:/etc/ceph/ceph.conf 2024-04-23T14:34:35.815 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: Updating smithi105:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T14:34:35.815 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/4100873924' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-23T14:34:35.815 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-04-23T14:34:35.815 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:35.815 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: Updating smithi105:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T14:34:35.815 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/4100873924' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-23T14:34:35.815 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T14:34:35.815 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:35.816 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:35.816 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:35.816 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:35.816 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:35.816 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:34:35.816 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:34:35.816 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:34:35.816 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:35.816 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T14:34:35.816 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:34:35.816 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi038:172.21.15.38=b;smithi105:172.21.15.105=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:34:35.816 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: Saving service mon spec with placement smithi028:172.21.15.28=a;smithi038:172.21.15.38=b;smithi105:172.21.15.105=c;count:3 2024-04-23T14:34:35.816 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: Deploying daemon mon.c on smithi105 2024-04-23T14:34:35.816 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:35.817 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:35.817 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 ceph-mon[32555]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-23T14:34:35.817 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 bash[32515]: 4684ef5a77bf23e76c8c169c169f128211ff4fa4bd86f95bb0d4e0bf9ae042ba 2024-04-23T14:34:35.817 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:35 smithi105 systemd[1]: Started Ceph mon.c for 1ee6e996-017e-11ef-bc93-c7b262605968. 2024-04-23T14:34:36.969 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-23T14:34:36.969 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-23T14:34:37.213 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T14:34:40.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:40 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T14:34:40.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:40 smithi105 ceph-mon[32555]: mon.a calling monitor election 2024-04-23T14:34:40.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:40 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:34:40.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:40 smithi105 ceph-mon[32555]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:40.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:40 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:34:40.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:40 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:34:40.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:40 smithi105 ceph-mon[32555]: mon.c calling monitor election 2024-04-23T14:34:40.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:40 smithi105 ceph-mon[32555]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:40.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:40 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:34:40.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:40 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:34:40.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:40 smithi105 ceph-mon[32555]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:40.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:40 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:34:40.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:40 smithi105 ceph-mon[32555]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T14:34:40.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:40 smithi105 ceph-mon[32555]: monmap epoch 2 2024-04-23T14:34:40.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:40 smithi105 ceph-mon[32555]: fsid 1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T14:34:40.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:40 smithi105 ceph-mon[32555]: last_changed 2024-04-23T14:34:35.412286+0000 2024-04-23T14:34:40.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:40 smithi105 ceph-mon[32555]: created 2024-04-23T14:32:30.337765+0000 2024-04-23T14:34:40.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:40 smithi105 ceph-mon[32555]: min_mon_release 19 (squid) 2024-04-23T14:34:40.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:40 smithi105 ceph-mon[32555]: election_strategy: 1 2024-04-23T14:34:40.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:40 smithi105 ceph-mon[32555]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-23T14:34:40.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:40 smithi105 ceph-mon[32555]: 1: [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon.c 2024-04-23T14:34:40.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:40 smithi105 ceph-mon[32555]: fsmap 2024-04-23T14:34:40.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:40 smithi105 ceph-mon[32555]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T14:34:40.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:40 smithi105 ceph-mon[32555]: mgrmap e13: x(active, since 58s) 2024-04-23T14:34:40.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:40 smithi105 ceph-mon[32555]: overall HEALTH_OK 2024-04-23T14:34:40.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:40 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:40.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:40 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:40.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:40 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:40.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:40 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T14:34:40.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:40 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:34:40.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:40 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T14:34:40.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:40 smithi028 ceph-mon[27810]: mon.a calling monitor election 2024-04-23T14:34:40.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:40 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:34:40.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:40 smithi028 ceph-mon[27810]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:40.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:40 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:34:40.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:40 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:34:40.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:40 smithi028 ceph-mon[27810]: mon.c calling monitor election 2024-04-23T14:34:40.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:40 smithi028 ceph-mon[27810]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:40.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:40 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:34:40.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:40 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:34:40.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:40 smithi028 ceph-mon[27810]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:40.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:40 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:34:40.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:40 smithi028 ceph-mon[27810]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T14:34:40.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:40 smithi028 ceph-mon[27810]: monmap epoch 2 2024-04-23T14:34:40.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:40 smithi028 ceph-mon[27810]: fsid 1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T14:34:40.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:40 smithi028 ceph-mon[27810]: last_changed 2024-04-23T14:34:35.412286+0000 2024-04-23T14:34:40.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:40 smithi028 ceph-mon[27810]: created 2024-04-23T14:32:30.337765+0000 2024-04-23T14:34:40.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:40 smithi028 ceph-mon[27810]: min_mon_release 19 (squid) 2024-04-23T14:34:40.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:40 smithi028 ceph-mon[27810]: election_strategy: 1 2024-04-23T14:34:40.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:40 smithi028 ceph-mon[27810]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-23T14:34:40.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:40 smithi028 ceph-mon[27810]: 1: [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon.c 2024-04-23T14:34:40.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:40 smithi028 ceph-mon[27810]: fsmap 2024-04-23T14:34:40.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:40 smithi028 ceph-mon[27810]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T14:34:40.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:40 smithi028 ceph-mon[27810]: mgrmap e13: x(active, since 58s) 2024-04-23T14:34:40.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:40 smithi028 ceph-mon[27810]: overall HEALTH_OK 2024-04-23T14:34:40.877 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:40 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:40.877 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:40 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:40.877 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:40 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:40.877 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:40 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T14:34:40.877 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:40 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:34:41.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.544+0000 7fc49d5b5700 1 -- 172.21.15.105:0/894525226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4800035c0 msgr2=0x7fc480005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:41.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.544+0000 7fc49d5b5700 1 --2- 172.21.15.105:0/894525226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4800035c0 0x7fc480005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc498101450 tx=0x7fc48c00feb0 comp rx=0 tx=0).stop 2024-04-23T14:34:41.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.544+0000 7fc49d5b5700 1 -- 172.21.15.105:0/894525226 shutdown_connections 2024-04-23T14:34:41.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.544+0000 7fc49d5b5700 1 --2- 172.21.15.105:0/894525226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4800035c0 0x7fc480005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:41.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.544+0000 7fc49d5b5700 1 -- 172.21.15.105:0/894525226 >> 172.21.15.105:0/894525226 conn(0x7fc4980fd030 msgr2=0x7fc4980ff450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:41.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.544+0000 7fc49d5b5700 1 -- 172.21.15.105:0/894525226 shutdown_connections 2024-04-23T14:34:41.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.544+0000 7fc49d5b5700 1 -- 172.21.15.105:0/894525226 wait complete. 2024-04-23T14:34:41.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.545+0000 7fc49d5b5700 1 Processor -- start 2024-04-23T14:34:41.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.545+0000 7fc49d5b5700 1 -- start start 2024-04-23T14:34:41.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.545+0000 7fc49d5b5700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4800035c0 0x7fc49819a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:41.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.545+0000 7fc49d5b5700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc49819a8e0 0x7fc49819ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:41.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.545+0000 7fc49d5b5700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc498108510 con 0x7fc4800035c0 2024-04-23T14:34:41.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.545+0000 7fc49d5b5700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc498108390 con 0x7fc49819a8e0 2024-04-23T14:34:41.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.546+0000 7fc4967fc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc49819a8e0 0x7fc49819ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:41.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.546+0000 7fc4967fc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc49819a8e0 0x7fc49819ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:54654/0 (socket says 172.21.15.105:54654) 2024-04-23T14:34:41.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.546+0000 7fc4967fc700 1 -- 172.21.15.105:0/2345276069 learned_addr learned my addr 172.21.15.105:0/2345276069 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-23T14:34:41.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.546+0000 7fc4967fc700 1 -- 172.21.15.105:0/2345276069 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc49819a8e0 msgr2=0x7fc49819ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-23T14:34:41.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.546+0000 7fc4967fc700 1 -- 172.21.15.105:0/2345276069 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc49819a8e0 msgr2=0x7fc49819ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-23T14:34:41.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.546+0000 7fc4967fc700 1 --2- 172.21.15.105:0/2345276069 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc49819a8e0 0x7fc49819ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T14:34:41.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.546+0000 7fc496ffd700 1 --2- 172.21.15.105:0/2345276069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4800035c0 0x7fc49819a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:41.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.546+0000 7fc4967fc700 1 --2- 172.21.15.105:0/2345276069 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc49819a8e0 0x7fc49819ec70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T14:34:41.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.546+0000 7fc496ffd700 1 -- 172.21.15.105:0/2345276069 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc49819a8e0 msgr2=0x7fc49819ec70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T14:34:41.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.546+0000 7fc496ffd700 1 --2- 172.21.15.105:0/2345276069 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc49819a8e0 0x7fc49819ec70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:41.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.546+0000 7fc496ffd700 1 -- 172.21.15.105:0/2345276069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc49819af20 con 0x7fc4800035c0 2024-04-23T14:34:41.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.547+0000 7fc496ffd700 1 --2- 172.21.15.105:0/2345276069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4800035c0 0x7fc49819a3a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc48c010010 tx=0x7fc48c0055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:41.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.547+0000 7fc47ffff700 1 -- 172.21.15.105:0/2345276069 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fc48c004b20 con 0x7fc4800035c0 2024-04-23T14:34:41.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.547+0000 7fc49d5b5700 1 -- 172.21.15.105:0/2345276069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc49819f1b0 con 0x7fc4800035c0 2024-04-23T14:34:41.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.547+0000 7fc49d5b5700 1 -- 172.21.15.105:0/2345276069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc49819f790 con 0x7fc4800035c0 2024-04-23T14:34:41.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.547+0000 7fc47ffff700 1 -- 172.21.15.105:0/2345276069 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc48c013e40 con 0x7fc4800035c0 2024-04-23T14:34:41.550 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.547+0000 7fc47ffff700 1 -- 172.21.15.105:0/2345276069 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fc48c0119c0 con 0x7fc4800035c0 2024-04-23T14:34:41.550 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.548+0000 7fc49d5b5700 1 -- 172.21.15.105:0/2345276069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc480002140 con 0x7fc4800035c0 2024-04-23T14:34:41.550 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.549+0000 7fc47ffff700 1 -- 172.21.15.105:0/2345276069 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fc48c023070 con 0x7fc4800035c0 2024-04-23T14:34:41.550 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.549+0000 7fc47ffff700 1 --2- 172.21.15.105:0/2345276069 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fc484039400 0x7fc48403b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:41.551 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.549+0000 7fc47ffff700 1 -- 172.21.15.105:0/2345276069 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fc48c020050 con 0x7fc4800035c0 2024-04-23T14:34:41.551 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.550+0000 7fc4967fc700 1 --2- 172.21.15.105:0/2345276069 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fc484039400 0x7fc48403b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:41.551 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.550+0000 7fc4967fc700 1 --2- 172.21.15.105:0/2345276069 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fc484039400 0x7fc48403b8c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc488009180 tx=0x7fc488008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:41.568 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.567+0000 7fc47ffff700 1 -- 172.21.15.105:0/2345276069 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc48c028020 con 0x7fc4800035c0 2024-04-23T14:34:41.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:41 smithi105 ceph-mon[32555]: Deploying daemon mon.b on smithi038 2024-04-23T14:34:41.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:41 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:34:41.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.850+0000 7fc49d5b5700 1 -- 172.21.15.105:0/2345276069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fc480002c30 con 0x7fc4800035c0 2024-04-23T14:34:41.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.851+0000 7fc47ffff700 1 -- 172.21.15.105:0/2345276069 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7fc48c01a390 con 0x7fc4800035c0 2024-04-23T14:34:41.852 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-23T14:34:41.852 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":2,"fsid":"1ee6e996-017e-11ef-bc93-c7b262605968","modified":"2024-04-23T14:34:35.412286Z","created":"2024-04-23T14:32:30.337765Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-23T14:34:41.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.853+0000 7fc49d5b5700 1 -- 172.21.15.105:0/2345276069 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fc484039400 msgr2=0x7fc48403b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:41.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.853+0000 7fc49d5b5700 1 --2- 172.21.15.105:0/2345276069 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fc484039400 0x7fc48403b8c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc488009180 tx=0x7fc488008040 comp rx=0 tx=0).stop 2024-04-23T14:34:41.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.853+0000 7fc49d5b5700 1 -- 172.21.15.105:0/2345276069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4800035c0 msgr2=0x7fc49819a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:41.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.853+0000 7fc49d5b5700 1 --2- 172.21.15.105:0/2345276069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4800035c0 0x7fc49819a3a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc48c010010 tx=0x7fc48c0055a0 comp rx=0 tx=0).stop 2024-04-23T14:34:41.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.853+0000 7fc49d5b5700 1 -- 172.21.15.105:0/2345276069 shutdown_connections 2024-04-23T14:34:41.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.853+0000 7fc49d5b5700 1 --2- 172.21.15.105:0/2345276069 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fc484039400 0x7fc48403b8c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:41.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.853+0000 7fc49d5b5700 1 --2- 172.21.15.105:0/2345276069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4800035c0 0x7fc49819a3a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:41.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.853+0000 7fc49d5b5700 1 --2- 172.21.15.105:0/2345276069 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc49819a8e0 0x7fc49819ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:41.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.853+0000 7fc49d5b5700 1 -- 172.21.15.105:0/2345276069 >> 172.21.15.105:0/2345276069 conn(0x7fc4980fd030 msgr2=0x7fc4980fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:41.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.854+0000 7fc49d5b5700 1 -- 172.21.15.105:0/2345276069 shutdown_connections 2024-04-23T14:34:41.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:41.854+0000 7fc49d5b5700 1 -- 172.21.15.105:0/2345276069 wait complete. 2024-04-23T14:34:41.856 INFO:teuthology.orchestra.run.smithi105.stderr:dumped monmap epoch 2 2024-04-23T14:34:41.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:41 smithi028 ceph-mon[27810]: Deploying daemon mon.b on smithi038 2024-04-23T14:34:41.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:41 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:34:42.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:42 smithi105 ceph-mon[32555]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:42.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:42 smithi105 ceph-mon[32555]: from='client.? 172.21.15.105:0/2345276069' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T14:34:42.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:42 smithi028 ceph-mon[27810]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:42.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:42 smithi028 ceph-mon[27810]: from='client.? 172.21.15.105:0/2345276069' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T14:34:43.460 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-23T14:34:43.460 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-23T14:34:43.701 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T14:34:44.216 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 podman[32330]: 2024-04-23 14:34:44.014662931 +0000 UTC m=+0.018080288 image pull 76e77f280c74312ca1add9b6019a4a921a516f91795d85bae00a0c8ca9b1c62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd 2024-04-23T14:34:44.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 podman[32330]: 2024-04-23 14:34:44.295454674 +0000 UTC m=+0.298872014 container init 8a2678fe19a19ff9da076bfa60308e44a1bd96ed446890180ff163e85e29f1c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mon-b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd) 2024-04-23T14:34:44.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 podman[32330]: 2024-04-23 14:34:44.298420504 +0000 UTC m=+0.301837844 container start 8a2678fe19a19ff9da076bfa60308e44a1bd96ed446890180ff163e85e29f1c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mon-b, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.schema-version=1.0) 2024-04-23T14:34:44.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: set uid:gid to 167:167 (ceph:ceph) 2024-04-23T14:34:44.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: ceph version 19.0.0-3126-gd44fddd9 (d44fddd910b5e94da0f2357af4be37d7bd495ae0) squid (dev), process ceph-mon, pid 2 2024-04-23T14:34:44.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: pidfile_write: ignore empty --pid-file 2024-04-23T14:34:44.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: load: jerasure load: lrc 2024-04-23T14:34:44.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: RocksDB version: 7.9.2 2024-04-23T14:34:44.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Git sha 0 2024-04-23T14:34:44.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Compile date 2024-04-22 13:15:13 2024-04-23T14:34:44.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: DB SUMMARY 2024-04-23T14:34:44.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: DB Session ID: 620E9QS0EFPA65KV4CAY 2024-04-23T14:34:44.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: CURRENT file: CURRENT 2024-04-23T14:34:44.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: IDENTITY file: IDENTITY 2024-04-23T14:34:44.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-23T14:34:44.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-23T14:34:44.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-04-23T14:34:44.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.error_if_exists: 0 2024-04-23T14:34:44.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.create_if_missing: 0 2024-04-23T14:34:44.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.paranoid_checks: 1 2024-04-23T14:34:44.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-23T14:34:44.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-23T14:34:44.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-23T14:34:44.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.env: 0x555d47878cc0 2024-04-23T14:34:44.625 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.fs: PosixFileSystem 2024-04-23T14:34:44.625 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.info_log: 0x555d4a0a9c20 2024-04-23T14:34:44.625 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.max_file_opening_threads: 16 2024-04-23T14:34:44.627 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.statistics: (nil) 2024-04-23T14:34:44.627 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.use_fsync: 0 2024-04-23T14:34:44.627 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.max_log_file_size: 0 2024-04-23T14:34:44.627 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-23T14:34:44.627 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-23T14:34:44.627 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.keep_log_file_num: 1000 2024-04-23T14:34:44.627 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.recycle_log_file_num: 0 2024-04-23T14:34:44.627 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.allow_fallocate: 1 2024-04-23T14:34:44.627 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.allow_mmap_reads: 0 2024-04-23T14:34:44.627 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.allow_mmap_writes: 0 2024-04-23T14:34:44.627 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.use_direct_reads: 0 2024-04-23T14:34:44.627 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-23T14:34:44.628 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.create_missing_column_families: 0 2024-04-23T14:34:44.628 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.db_log_dir: 2024-04-23T14:34:44.628 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.wal_dir: 2024-04-23T14:34:44.628 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-23T14:34:44.628 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-23T14:34:44.628 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-23T14:34:44.628 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-23T14:34:44.628 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-23T14:34:44.628 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-23T14:34:44.628 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.advise_random_on_open: 1 2024-04-23T14:34:44.628 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.db_write_buffer_size: 0 2024-04-23T14:34:44.628 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.write_buffer_manager: 0x555d4a0781e0 2024-04-23T14:34:44.628 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-23T14:34:44.628 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-23T14:34:44.629 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-23T14:34:44.629 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.rate_limiter: (nil) 2024-04-23T14:34:44.629 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-23T14:34:44.629 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.wal_recovery_mode: 2 2024-04-23T14:34:44.629 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.enable_thread_tracking: 0 2024-04-23T14:34:44.629 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.enable_pipelined_write: 0 2024-04-23T14:34:44.629 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.unordered_write: 0 2024-04-23T14:34:44.629 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-23T14:34:44.629 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-23T14:34:44.629 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-23T14:34:44.629 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-23T14:34:44.629 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.row_cache: None 2024-04-23T14:34:44.629 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.wal_filter: None 2024-04-23T14:34:44.629 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-23T14:34:44.629 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.allow_ingest_behind: 0 2024-04-23T14:34:44.629 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.two_write_queues: 0 2024-04-23T14:34:44.630 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.manual_wal_flush: 0 2024-04-23T14:34:44.630 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.wal_compression: 0 2024-04-23T14:34:44.630 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.atomic_flush: 0 2024-04-23T14:34:44.630 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-23T14:34:44.630 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-23T14:34:44.630 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-23T14:34:44.630 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.log_readahead_size: 0 2024-04-23T14:34:44.630 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-23T14:34:44.630 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.best_efforts_recovery: 0 2024-04-23T14:34:44.630 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-23T14:34:44.630 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-23T14:34:44.630 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.allow_data_in_errors: 0 2024-04-23T14:34:44.630 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.db_host_id: __hostname__ 2024-04-23T14:34:44.631 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-23T14:34:44.631 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.max_background_jobs: 2 2024-04-23T14:34:44.631 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.max_background_compactions: -1 2024-04-23T14:34:44.631 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.max_subcompactions: 1 2024-04-23T14:34:44.631 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-23T14:34:44.631 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-23T14:34:44.631 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-23T14:34:44.631 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.max_total_wal_size: 0 2024-04-23T14:34:44.631 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-23T14:34:44.631 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-23T14:34:44.631 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-23T14:34:44.631 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-23T14:34:44.631 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.max_open_files: -1 2024-04-23T14:34:44.631 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.bytes_per_sync: 0 2024-04-23T14:34:44.632 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-23T14:34:44.632 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-23T14:34:44.632 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.compaction_readahead_size: 0 2024-04-23T14:34:44.632 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.max_background_flushes: -1 2024-04-23T14:34:44.632 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Compression algorithms supported: 2024-04-23T14:34:44.632 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-23T14:34:44.632 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: kZSTD supported: 0 2024-04-23T14:34:44.632 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: kXpressCompression supported: 0 2024-04-23T14:34:44.632 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: kLZ4HCCompression supported: 1 2024-04-23T14:34:44.632 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: kZlibCompression supported: 1 2024-04-23T14:34:44.632 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: kSnappyCompression supported: 1 2024-04-23T14:34:44.632 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: kLZ4Compression supported: 1 2024-04-23T14:34:44.632 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: kBZip2Compression supported: 0 2024-04-23T14:34:44.632 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-23T14:34:44.633 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-23T14:34:44.633 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-04-23T14:34:44.633 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-23T14:34:44.633 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-23T14:34:44.633 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.merge_operator: 2024-04-23T14:34:44.633 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.compaction_filter: None 2024-04-23T14:34:44.633 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.compaction_filter_factory: None 2024-04-23T14:34:44.633 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.sst_partitioner_factory: None 2024-04-23T14:34:44.633 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-23T14:34:44.633 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-23T14:34:44.633 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555d4a0a9de0) 2024-04-23T14:34:44.633 INFO:journalctl@ceph.mon.b.smithi038.stdout: cache_index_and_filter_blocks: 1 2024-04-23T14:34:44.633 INFO:journalctl@ceph.mon.b.smithi038.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-23T14:34:44.633 INFO:journalctl@ceph.mon.b.smithi038.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-23T14:34:44.633 INFO:journalctl@ceph.mon.b.smithi038.stdout: pin_top_level_index_and_filter: 1 2024-04-23T14:34:44.634 INFO:journalctl@ceph.mon.b.smithi038.stdout: index_type: 0 2024-04-23T14:34:44.634 INFO:journalctl@ceph.mon.b.smithi038.stdout: data_block_index_type: 0 2024-04-23T14:34:44.634 INFO:journalctl@ceph.mon.b.smithi038.stdout: index_shortening: 1 2024-04-23T14:34:44.634 INFO:journalctl@ceph.mon.b.smithi038.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-23T14:34:44.634 INFO:journalctl@ceph.mon.b.smithi038.stdout: checksum: 4 2024-04-23T14:34:44.634 INFO:journalctl@ceph.mon.b.smithi038.stdout: no_block_cache: 0 2024-04-23T14:34:44.634 INFO:journalctl@ceph.mon.b.smithi038.stdout: block_cache: 0x555d4a0ab090 2024-04-23T14:34:44.634 INFO:journalctl@ceph.mon.b.smithi038.stdout: block_cache_name: BinnedLRUCache 2024-04-23T14:34:44.634 INFO:journalctl@ceph.mon.b.smithi038.stdout: block_cache_options: 2024-04-23T14:34:44.634 INFO:journalctl@ceph.mon.b.smithi038.stdout: capacity : 536870912 2024-04-23T14:34:44.634 INFO:journalctl@ceph.mon.b.smithi038.stdout: num_shard_bits : 4 2024-04-23T14:34:44.634 INFO:journalctl@ceph.mon.b.smithi038.stdout: strict_capacity_limit : 0 2024-04-23T14:34:44.634 INFO:journalctl@ceph.mon.b.smithi038.stdout: high_pri_pool_ratio: 0.000 2024-04-23T14:34:44.635 INFO:journalctl@ceph.mon.b.smithi038.stdout: block_cache_compressed: (nil) 2024-04-23T14:34:44.635 INFO:journalctl@ceph.mon.b.smithi038.stdout: persistent_cache: (nil) 2024-04-23T14:34:44.635 INFO:journalctl@ceph.mon.b.smithi038.stdout: block_size: 4096 2024-04-23T14:34:44.635 INFO:journalctl@ceph.mon.b.smithi038.stdout: block_size_deviation: 10 2024-04-23T14:34:44.635 INFO:journalctl@ceph.mon.b.smithi038.stdout: block_restart_interval: 16 2024-04-23T14:34:44.635 INFO:journalctl@ceph.mon.b.smithi038.stdout: index_block_restart_interval: 1 2024-04-23T14:34:44.635 INFO:journalctl@ceph.mon.b.smithi038.stdout: metadata_block_size: 4096 2024-04-23T14:34:44.635 INFO:journalctl@ceph.mon.b.smithi038.stdout: partition_filters: 0 2024-04-23T14:34:44.635 INFO:journalctl@ceph.mon.b.smithi038.stdout: use_delta_encoding: 1 2024-04-23T14:34:44.635 INFO:journalctl@ceph.mon.b.smithi038.stdout: filter_policy: bloomfilter 2024-04-23T14:34:44.635 INFO:journalctl@ceph.mon.b.smithi038.stdout: whole_key_filtering: 1 2024-04-23T14:34:44.635 INFO:journalctl@ceph.mon.b.smithi038.stdout: verify_compression: 0 2024-04-23T14:34:44.635 INFO:journalctl@ceph.mon.b.smithi038.stdout: read_amp_bytes_per_bit: 0 2024-04-23T14:34:44.635 INFO:journalctl@ceph.mon.b.smithi038.stdout: format_version: 5 2024-04-23T14:34:44.636 INFO:journalctl@ceph.mon.b.smithi038.stdout: enable_index_compression: 1 2024-04-23T14:34:44.636 INFO:journalctl@ceph.mon.b.smithi038.stdout: block_align: 0 2024-04-23T14:34:44.636 INFO:journalctl@ceph.mon.b.smithi038.stdout: max_auto_readahead_size: 262144 2024-04-23T14:34:44.636 INFO:journalctl@ceph.mon.b.smithi038.stdout: prepopulate_block_cache: 0 2024-04-23T14:34:44.636 INFO:journalctl@ceph.mon.b.smithi038.stdout: initial_auto_readahead_size: 8192 2024-04-23T14:34:44.636 INFO:journalctl@ceph.mon.b.smithi038.stdout: num_file_reads_for_auto_readahead: 2 2024-04-23T14:34:44.636 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.write_buffer_size: 33554432 2024-04-23T14:34:44.636 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.max_write_buffer_number: 2 2024-04-23T14:34:44.636 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.compression: NoCompression 2024-04-23T14:34:44.636 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.bottommost_compression: Disabled 2024-04-23T14:34:44.636 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.prefix_extractor: nullptr 2024-04-23T14:34:44.636 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-23T14:34:44.636 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.num_levels: 7 2024-04-23T14:34:44.636 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-23T14:34:44.637 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-23T14:34:44.637 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-23T14:34:44.637 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-23T14:34:44.637 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-23T14:34:44.637 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-23T14:34:44.637 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-23T14:34:44.637 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-23T14:34:44.637 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-23T14:34:44.637 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-23T14:34:44.637 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-23T14:34:44.637 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-23T14:34:44.637 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-23T14:34:44.637 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.compression_opts.level: 32767 2024-04-23T14:34:44.638 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.compression_opts.strategy: 0 2024-04-23T14:34:44.638 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-23T14:34:44.638 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-23T14:34:44.638 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-23T14:34:44.638 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-23T14:34:44.638 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.compression_opts.enabled: false 2024-04-23T14:34:44.638 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-23T14:34:44.638 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-23T14:34:44.638 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-23T14:34:44.638 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-23T14:34:44.638 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.target_file_size_base: 67108864 2024-04-23T14:34:44.638 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-23T14:34:44.639 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-23T14:34:44.639 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-23T14:34:44.639 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-23T14:34:44.639 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-23T14:34:44.639 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-23T14:34:44.639 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-23T14:34:44.639 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-23T14:34:44.639 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-23T14:34:44.639 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-23T14:34:44.639 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-23T14:34:44.639 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-23T14:34:44.639 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-23T14:34:44.640 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-23T14:34:44.640 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.arena_block_size: 1048576 2024-04-23T14:34:44.640 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-23T14:34:44.640 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-23T14:34:44.640 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.disable_auto_compactions: 0 2024-04-23T14:34:44.640 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-23T14:34:44.640 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-23T14:34:44.640 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-23T14:34:44.640 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-23T14:34:44.640 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-23T14:34:44.640 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-23T14:34:44.640 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-23T14:34:44.640 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-23T14:34:44.640 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-23T14:34:44.641 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-23T14:34:44.641 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-23T14:34:44.641 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.inplace_update_support: 0 2024-04-23T14:34:44.641 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-23T14:34:44.641 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-23T14:34:44.641 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-23T14:34:44.641 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-23T14:34:44.641 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.bloom_locality: 0 2024-04-23T14:34:44.641 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.max_successive_merges: 0 2024-04-23T14:34:44.641 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-23T14:34:44.641 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.paranoid_file_checks: 0 2024-04-23T14:34:44.641 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.force_consistency_checks: 1 2024-04-23T14:34:44.641 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.report_bg_io_stats: 0 2024-04-23T14:34:44.641 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.ttl: 2592000 2024-04-23T14:34:44.641 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-23T14:34:44.642 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-23T14:34:44.642 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-23T14:34:44.642 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.enable_blob_files: false 2024-04-23T14:34:44.642 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.min_blob_size: 0 2024-04-23T14:34:44.642 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.blob_file_size: 268435456 2024-04-23T14:34:44.642 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-23T14:34:44.642 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-23T14:34:44.642 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-23T14:34:44.642 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-23T14:34:44.642 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-23T14:34:44.642 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.blob_file_starting_level: 0 2024-04-23T14:34:44.642 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-23T14:34:44.642 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-23T14:34:44.642 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-23T14:34:44.642 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b55baf3b-10a8-4773-b90b-9a50c3ac8cf7 2024-04-23T14:34:44.643 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713882884329969, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-23T14:34:44.643 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-23T14:34:44.643 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713882884330566, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713882884, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b55baf3b-10a8-4773-b90b-9a50c3ac8cf7", "db_session_id": "620E9QS0EFPA65KV4CAY", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-23T14:34:44.643 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713882884330670, "job": 1, "event": "recovery_finished"} 2024-04-23T14:34:44.643 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-23T14:34:44.643 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-23T14:34:44.643 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x555d4a198000 2024-04-23T14:34:44.643 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: DB pointer 0x555d4a182000 2024-04-23T14:34:44.643 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-23T14:34:44.643 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-23T14:34:44.643 INFO:journalctl@ceph.mon.b.smithi038.stdout: ** DB Stats ** 2024-04-23T14:34:44.643 INFO:journalctl@ceph.mon.b.smithi038.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T14:34:44.643 INFO:journalctl@ceph.mon.b.smithi038.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-23T14:34:44.643 INFO:journalctl@ceph.mon.b.smithi038.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T14:34:44.643 INFO:journalctl@ceph.mon.b.smithi038.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T14:34:44.643 INFO:journalctl@ceph.mon.b.smithi038.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-23T14:34:44.644 INFO:journalctl@ceph.mon.b.smithi038.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T14:34:44.644 INFO:journalctl@ceph.mon.b.smithi038.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T14:34:44.644 INFO:journalctl@ceph.mon.b.smithi038.stdout: 2024-04-23T14:34:44.644 INFO:journalctl@ceph.mon.b.smithi038.stdout: ** Compaction Stats [default] ** 2024-04-23T14:34:44.644 INFO:journalctl@ceph.mon.b.smithi038.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-23T14:34:44.644 INFO:journalctl@ceph.mon.b.smithi038.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-23T14:34:44.644 INFO:journalctl@ceph.mon.b.smithi038.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T14:34:44.644 INFO:journalctl@ceph.mon.b.smithi038.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T14:34:44.644 INFO:journalctl@ceph.mon.b.smithi038.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T14:34:44.644 INFO:journalctl@ceph.mon.b.smithi038.stdout: 2024-04-23T14:34:44.644 INFO:journalctl@ceph.mon.b.smithi038.stdout: ** Compaction Stats [default] ** 2024-04-23T14:34:44.644 INFO:journalctl@ceph.mon.b.smithi038.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-23T14:34:44.644 INFO:journalctl@ceph.mon.b.smithi038.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-23T14:34:44.644 INFO:journalctl@ceph.mon.b.smithi038.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T14:34:44.644 INFO:journalctl@ceph.mon.b.smithi038.stdout: 2024-04-23T14:34:44.645 INFO:journalctl@ceph.mon.b.smithi038.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-23T14:34:44.645 INFO:journalctl@ceph.mon.b.smithi038.stdout: 2024-04-23T14:34:44.645 INFO:journalctl@ceph.mon.b.smithi038.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T14:34:44.645 INFO:journalctl@ceph.mon.b.smithi038.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-23T14:34:44.645 INFO:journalctl@ceph.mon.b.smithi038.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-23T14:34:44.645 INFO:journalctl@ceph.mon.b.smithi038.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-23T14:34:44.645 INFO:journalctl@ceph.mon.b.smithi038.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-23T14:34:44.645 INFO:journalctl@ceph.mon.b.smithi038.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-23T14:34:44.645 INFO:journalctl@ceph.mon.b.smithi038.stdout: Cumulative compaction: 0.00 GB write, 0.22 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T14:34:44.645 INFO:journalctl@ceph.mon.b.smithi038.stdout: Interval compaction: 0.00 GB write, 0.22 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T14:34:44.645 INFO:journalctl@ceph.mon.b.smithi038.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-23T14:34:44.645 INFO:journalctl@ceph.mon.b.smithi038.stdout: Block cache BinnedLRUCache@0x555d4a0ab090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-23T14:34:44.645 INFO:journalctl@ceph.mon.b.smithi038.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-23T14:34:44.645 INFO:journalctl@ceph.mon.b.smithi038.stdout: 2024-04-23T14:34:44.645 INFO:journalctl@ceph.mon.b.smithi038.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-23T14:34:44.646 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-23T14:34:44.646 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: using public_addr v2:172.21.15.38:0/0 -> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] 2024-04-23T14:34:44.646 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: starting mon.b rank -1 at public addrs [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] at bind addrs [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T14:34:44.646 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: mon.b@-1(???) e0 preinit fsid 1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T14:34:44.646 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: mon.b@-1(synchronizing).mds e1 new map 2024-04-23T14:34:44.646 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-23T14:34:44.646 INFO:journalctl@ceph.mon.b.smithi038.stdout: e1 2024-04-23T14:34:44.646 INFO:journalctl@ceph.mon.b.smithi038.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-23T14:34:44.646 INFO:journalctl@ceph.mon.b.smithi038.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-23T14:34:44.646 INFO:journalctl@ceph.mon.b.smithi038.stdout: legacy client fscid: -1 2024-04-23T14:34:44.646 INFO:journalctl@ceph.mon.b.smithi038.stdout: 2024-04-23T14:34:44.646 INFO:journalctl@ceph.mon.b.smithi038.stdout: No filesystems configured 2024-04-23T14:34:44.647 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-23T14:34:44.647 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-23T14:34:44.647 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-23T14:34:44.647 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-23T14:34:44.647 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-23T14:34:44.647 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-23T14:34:44.647 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-23T14:34:44.647 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T14:34:44.647 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T14:34:44.647 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T14:34:44.647 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:44.647 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:44.647 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:34:44.647 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:44.648 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: Added host smithi038 2024-04-23T14:34:44.648 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:44.648 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:44.648 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:44.648 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:44.648 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T14:34:44.648 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:44.648 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:44.648 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:44.648 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:44.648 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:44.648 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:34:44.648 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:34:44.649 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:34:44.649 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-23T14:34:44.649 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: Updating smithi038:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T14:34:44.649 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-04-23T14:34:44.649 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:44.649 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: Updating smithi038:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T14:34:44.649 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi105", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:34:44.649 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:44.649 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:44.649 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:44.649 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: Deploying cephadm binary to smithi105 2024-04-23T14:34:44.649 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:44.649 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:44.649 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:44.650 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:34:44.650 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: Added host smithi105 2024-04-23T14:34:44.650 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:44.650 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:44.650 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:44.650 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:44.650 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:44.650 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T14:34:44.650 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:44.650 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:44.650 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:44.650 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:44.650 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:44.650 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi105", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:34:44.650 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:34:44.651 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:34:44.651 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: Updating smithi105:/etc/ceph/ceph.conf 2024-04-23T14:34:44.651 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: Updating smithi105:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T14:34:44.651 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/4100873924' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-23T14:34:44.651 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-04-23T14:34:44.651 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:44.651 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: Updating smithi105:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T14:34:44.651 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/4100873924' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-23T14:34:44.651 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T14:34:44.651 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:44.651 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:44.651 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:44.651 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:44.651 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:44.652 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:34:44.652 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:34:44.652 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:34:44.652 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:44.652 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T14:34:44.652 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:34:44.652 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi038:172.21.15.38=b;smithi105:172.21.15.105=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:34:44.652 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: Saving service mon spec with placement smithi028:172.21.15.28=a;smithi038:172.21.15.38=b;smithi105:172.21.15.105=c;count:3 2024-04-23T14:34:44.652 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: Deploying daemon mon.c on smithi105 2024-04-23T14:34:44.652 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:44.652 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:44.652 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T14:34:44.652 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: mon.a calling monitor election 2024-04-23T14:34:44.652 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:34:44.652 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:44.652 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:34:44.653 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:34:44.653 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: mon.c calling monitor election 2024-04-23T14:34:44.653 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:44.653 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:34:44.653 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:34:44.653 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:44.653 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:34:44.653 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T14:34:44.653 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: monmap epoch 2 2024-04-23T14:34:44.653 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: fsid 1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T14:34:44.653 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: last_changed 2024-04-23T14:34:35.412286+0000 2024-04-23T14:34:44.653 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: created 2024-04-23T14:32:30.337765+0000 2024-04-23T14:34:44.653 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: min_mon_release 19 (squid) 2024-04-23T14:34:44.653 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: election_strategy: 1 2024-04-23T14:34:44.653 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-23T14:34:44.654 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: 1: [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon.c 2024-04-23T14:34:44.654 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: fsmap 2024-04-23T14:34:44.654 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T14:34:44.654 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: mgrmap e13: x(active, since 58s) 2024-04-23T14:34:44.654 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: overall HEALTH_OK 2024-04-23T14:34:44.654 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:44.654 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:44.654 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:44.654 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T14:34:44.654 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:34:44.654 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: Deploying daemon mon.b on smithi038 2024-04-23T14:34:44.654 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:34:44.654 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:44.654 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: from='client.? 172.21.15.105:0/2345276069' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T14:34:44.655 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 bash[32330]: 8a2678fe19a19ff9da076bfa60308e44a1bd96ed446890180ff163e85e29f1c8 2024-04-23T14:34:44.655 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 ceph-mon[32348]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-23T14:34:44.655 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:44 smithi038 systemd[1]: Started Ceph mon.b for 1ee6e996-017e-11ef-bc93-c7b262605968. 2024-04-23T14:34:49.617 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.616+0000 7f8c7317d700 1 -- 172.21.15.105:0/4114036855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c540035c0 msgr2=0x7f8c54005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:49.618 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.617+0000 7f8c7317d700 1 --2- 172.21.15.105:0/4114036855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c540035c0 0x7f8c54005a50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c101460 tx=0x7f8c68009e80 comp rx=0 tx=0).stop 2024-04-23T14:34:49.618 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.617+0000 7f8c7317d700 1 -- 172.21.15.105:0/4114036855 shutdown_connections 2024-04-23T14:34:49.618 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.617+0000 7f8c7317d700 1 --2- 172.21.15.105:0/4114036855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c540035c0 0x7f8c54005a50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:49.618 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.617+0000 7f8c7317d700 1 -- 172.21.15.105:0/4114036855 >> 172.21.15.105:0/4114036855 conn(0x7f8c6c0fd040 msgr2=0x7f8c6c0ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:49.618 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.617+0000 7f8c7317d700 1 -- 172.21.15.105:0/4114036855 shutdown_connections 2024-04-23T14:34:49.618 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.617+0000 7f8c7317d700 1 -- 172.21.15.105:0/4114036855 wait complete. 2024-04-23T14:34:49.620 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.617+0000 7f8c7317d700 1 Processor -- start 2024-04-23T14:34:49.620 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.618+0000 7f8c7317d700 1 -- start start 2024-04-23T14:34:49.620 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.618+0000 7f8c7317d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c540035c0 0x7f8c6c1031c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:49.621 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.618+0000 7f8c7317d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c6c103700 0x7f8c6c103b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:49.621 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.618+0000 7f8c7317d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c6c1a43f0 0x7f8c6c1a67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:49.621 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.618+0000 7f8c7317d700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8c6c109580 con 0x7f8c540035c0 2024-04-23T14:34:49.621 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.618+0000 7f8c7317d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8c6c109400 con 0x7f8c6c1a43f0 2024-04-23T14:34:49.621 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.618+0000 7f8c7317d700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8c6c109700 con 0x7f8c6c103700 2024-04-23T14:34:49.621 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.618+0000 7f8c63fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c6c103700 0x7f8c6c103b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:49.621 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.618+0000 7f8c63fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c6c103700 0x7f8c6c103b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:54664/0 (socket says 172.21.15.105:54664) 2024-04-23T14:34:49.621 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.618+0000 7f8c63fff700 1 -- 172.21.15.105:0/2742878504 learned_addr learned my addr 172.21.15.105:0/2742878504 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-23T14:34:49.621 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.618+0000 7f8c7171a700 1 --2- 172.21.15.105:0/2742878504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c6c1a43f0 0x7f8c6c1a67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:49.621 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.618+0000 7f8c63fff700 1 -- 172.21.15.105:0/2742878504 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c6c103700 msgr2=0x7f8c6c103b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-23T14:34:49.621 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.618+0000 7f8c63fff700 1 -- 172.21.15.105:0/2742878504 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c6c103700 msgr2=0x7f8c6c103b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-23T14:34:49.621 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.618+0000 7f8c63fff700 1 --2- 172.21.15.105:0/2742878504 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c6c103700 0x7f8c6c103b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T14:34:49.621 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.618+0000 7f8c63fff700 1 --2- 172.21.15.105:0/2742878504 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c6c103700 0x7f8c6c103b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T14:34:49.622 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.618+0000 7f8c7171a700 1 -- 172.21.15.105:0/2742878504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c6c1a43f0 msgr2=0x7f8c6c1a67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-23T14:34:49.622 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.618+0000 7f8c7171a700 1 -- 172.21.15.105:0/2742878504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c6c1a43f0 msgr2=0x7f8c6c1a67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-23T14:34:49.622 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.618+0000 7f8c7171a700 1 --2- 172.21.15.105:0/2742878504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c6c1a43f0 0x7f8c6c1a67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T14:34:49.622 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.618+0000 7f8c70f19700 1 --2- 172.21.15.105:0/2742878504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c540035c0 0x7f8c6c1031c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:49.622 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.618+0000 7f8c7171a700 1 --2- 172.21.15.105:0/2742878504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c6c1a43f0 0x7f8c6c1a67e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T14:34:49.622 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.619+0000 7f8c70f19700 1 -- 172.21.15.105:0/2742878504 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c6c103700 msgr2=0x7f8c6c103b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T14:34:49.622 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.619+0000 7f8c70f19700 1 --2- 172.21.15.105:0/2742878504 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c6c103700 0x7f8c6c103b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:49.622 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.619+0000 7f8c70f19700 1 -- 172.21.15.105:0/2742878504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c6c1a43f0 msgr2=0x7f8c6c1a67e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T14:34:49.622 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.619+0000 7f8c70f19700 1 --2- 172.21.15.105:0/2742878504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c6c1a43f0 0x7f8c6c1a67e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:49.622 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.619+0000 7f8c70f19700 1 -- 172.21.15.105:0/2742878504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c6c1a6f10 con 0x7f8c540035c0 2024-04-23T14:34:49.622 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.619+0000 7f8c70f19700 1 --2- 172.21.15.105:0/2742878504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c540035c0 0x7f8c6c1031c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8c68002230 tx=0x7f8c6801b450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:49.623 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.620+0000 7f8c61ffb700 1 -- 172.21.15.105:0/2742878504 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c6802f070 con 0x7f8c540035c0 2024-04-23T14:34:49.623 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.620+0000 7f8c7317d700 1 -- 172.21.15.105:0/2742878504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c6c1a71a0 con 0x7f8c540035c0 2024-04-23T14:34:49.624 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.620+0000 7f8c7317d700 1 -- 172.21.15.105:0/2742878504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c6c1a7730 con 0x7f8c540035c0 2024-04-23T14:34:49.624 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.620+0000 7f8c61ffb700 1 -- 172.21.15.105:0/2742878504 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c68003a00 con 0x7f8c540035c0 2024-04-23T14:34:49.624 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.620+0000 7f8c61ffb700 1 -- 172.21.15.105:0/2742878504 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c68022680 con 0x7f8c540035c0 2024-04-23T14:34:49.624 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.621+0000 7f8c7317d700 1 -- 172.21.15.105:0/2742878504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c6c064150 con 0x7f8c540035c0 2024-04-23T14:34:49.624 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.621+0000 7f8c61ffb700 1 -- 172.21.15.105:0/2742878504 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f8c68032070 con 0x7f8c540035c0 2024-04-23T14:34:49.624 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.622+0000 7f8c61ffb700 1 --2- 172.21.15.105:0/2742878504 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f8c58039ae0 0x7f8c5803bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:49.624 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.622+0000 7f8c61ffb700 1 -- 172.21.15.105:0/2742878504 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f8c6804eb20 con 0x7f8c540035c0 2024-04-23T14:34:49.624 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.622+0000 7f8c63fff700 1 --2- 172.21.15.105:0/2742878504 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f8c58039ae0 0x7f8c5803bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:49.624 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.623+0000 7f8c63fff700 1 --2- 172.21.15.105:0/2742878504 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f8c58039ae0 0x7f8c5803bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c009180 tx=0x7f8c5c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:49.638 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.638+0000 7f8c61ffb700 1 -- 172.21.15.105:0/2742878504 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8c68027070 con 0x7f8c540035c0 2024-04-23T14:34:49.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:49 smithi105 ceph-mon[32555]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:49.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:49 smithi105 ceph-mon[32555]: mon.c calling monitor election 2024-04-23T14:34:49.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:49 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T14:34:49.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:49 smithi105 ceph-mon[32555]: mon.a calling monitor election 2024-04-23T14:34:49.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:49 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:34:49.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:49 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:34:49.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:49 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:34:49.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:49 smithi105 ceph-mon[32555]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:49.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:49 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:34:49.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:49 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:34:49.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:49 smithi105 ceph-mon[32555]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:49.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:49 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:34:49.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:49 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:34:49.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:49 smithi105 ceph-mon[32555]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T14:34:49.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:49 smithi105 ceph-mon[32555]: monmap epoch 3 2024-04-23T14:34:49.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:49 smithi105 ceph-mon[32555]: fsid 1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T14:34:49.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:49 smithi105 ceph-mon[32555]: last_changed 2024-04-23T14:34:44.375947+0000 2024-04-23T14:34:49.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:49 smithi105 ceph-mon[32555]: created 2024-04-23T14:32:30.337765+0000 2024-04-23T14:34:49.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:49 smithi105 ceph-mon[32555]: min_mon_release 19 (squid) 2024-04-23T14:34:49.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:49 smithi105 ceph-mon[32555]: election_strategy: 1 2024-04-23T14:34:49.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:49 smithi105 ceph-mon[32555]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-23T14:34:49.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:49 smithi105 ceph-mon[32555]: 1: [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon.c 2024-04-23T14:34:49.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:49 smithi105 ceph-mon[32555]: 2: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.b 2024-04-23T14:34:49.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:49 smithi105 ceph-mon[32555]: fsmap 2024-04-23T14:34:49.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:49 smithi105 ceph-mon[32555]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T14:34:49.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:49 smithi105 ceph-mon[32555]: mgrmap e13: x(active, since 67s) 2024-04-23T14:34:49.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:49 smithi105 ceph-mon[32555]: overall HEALTH_OK 2024-04-23T14:34:49.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:49 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:49.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:49 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:49.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:49 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:49.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:49 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:49.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:49 smithi028 ceph-mon[27810]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:49.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:49 smithi028 ceph-mon[27810]: mon.c calling monitor election 2024-04-23T14:34:49.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:49 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T14:34:49.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:49 smithi028 ceph-mon[27810]: mon.a calling monitor election 2024-04-23T14:34:49.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:49 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:34:49.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:49 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:34:49.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:49 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:34:49.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:49 smithi028 ceph-mon[27810]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:49.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:49 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:34:49.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:49 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:34:49.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:49 smithi028 ceph-mon[27810]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:49.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:49 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:34:49.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:49 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:34:49.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:49 smithi028 ceph-mon[27810]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T14:34:49.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:49 smithi028 ceph-mon[27810]: monmap epoch 3 2024-04-23T14:34:49.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:49 smithi028 ceph-mon[27810]: fsid 1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T14:34:49.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:49 smithi028 ceph-mon[27810]: last_changed 2024-04-23T14:34:44.375947+0000 2024-04-23T14:34:49.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:49 smithi028 ceph-mon[27810]: created 2024-04-23T14:32:30.337765+0000 2024-04-23T14:34:49.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:49 smithi028 ceph-mon[27810]: min_mon_release 19 (squid) 2024-04-23T14:34:49.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:49 smithi028 ceph-mon[27810]: election_strategy: 1 2024-04-23T14:34:49.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:49 smithi028 ceph-mon[27810]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-23T14:34:49.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:49 smithi028 ceph-mon[27810]: 1: [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon.c 2024-04-23T14:34:49.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:49 smithi028 ceph-mon[27810]: 2: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.b 2024-04-23T14:34:49.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:49 smithi028 ceph-mon[27810]: fsmap 2024-04-23T14:34:49.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:49 smithi028 ceph-mon[27810]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T14:34:49.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:49 smithi028 ceph-mon[27810]: mgrmap e13: x(active, since 67s) 2024-04-23T14:34:49.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:49 smithi028 ceph-mon[27810]: overall HEALTH_OK 2024-04-23T14:34:49.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:49 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:49.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:49 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:49.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:49 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:49.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:49 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:49.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.935+0000 7f8c7317d700 1 -- 172.21.15.105:0/2742878504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f8c6c1a79b0 con 0x7f8c540035c0 2024-04-23T14:34:49.937 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.936+0000 7f8c61ffb700 1 -- 172.21.15.105:0/2742878504 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1279 (secure 0 0 0) 0x7f8c6802acd0 con 0x7f8c540035c0 2024-04-23T14:34:49.937 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-23T14:34:49.937 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":3,"fsid":"1ee6e996-017e-11ef-bc93-c7b262605968","modified":"2024-04-23T14:34:44.375947Z","created":"2024-04-23T14:32:30.337765Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-23T14:34:49.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.939+0000 7f8c7317d700 1 -- 172.21.15.105:0/2742878504 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f8c58039ae0 msgr2=0x7f8c5803bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:49.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.939+0000 7f8c7317d700 1 --2- 172.21.15.105:0/2742878504 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f8c58039ae0 0x7f8c5803bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c009180 tx=0x7f8c5c008040 comp rx=0 tx=0).stop 2024-04-23T14:34:49.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.939+0000 7f8c7317d700 1 -- 172.21.15.105:0/2742878504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c540035c0 msgr2=0x7f8c6c1031c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:49.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.939+0000 7f8c7317d700 1 --2- 172.21.15.105:0/2742878504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c540035c0 0x7f8c6c1031c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8c68002230 tx=0x7f8c6801b450 comp rx=0 tx=0).stop 2024-04-23T14:34:49.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.939+0000 7f8c7317d700 1 -- 172.21.15.105:0/2742878504 shutdown_connections 2024-04-23T14:34:49.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.939+0000 7f8c7317d700 1 --2- 172.21.15.105:0/2742878504 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f8c58039ae0 0x7f8c5803bfa0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:49.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.939+0000 7f8c7317d700 1 --2- 172.21.15.105:0/2742878504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c6c1a43f0 0x7f8c6c1a67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:49.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.939+0000 7f8c7317d700 1 --2- 172.21.15.105:0/2742878504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c540035c0 0x7f8c6c1031c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:49.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.939+0000 7f8c7317d700 1 --2- 172.21.15.105:0/2742878504 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c6c103700 0x7f8c6c103b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:49.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.939+0000 7f8c7317d700 1 -- 172.21.15.105:0/2742878504 >> 172.21.15.105:0/2742878504 conn(0x7f8c6c0fd040 msgr2=0x7f8c6c106930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:49.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.939+0000 7f8c7317d700 1 -- 172.21.15.105:0/2742878504 shutdown_connections 2024-04-23T14:34:49.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:49.939+0000 7f8c7317d700 1 -- 172.21.15.105:0/2742878504 wait complete. 2024-04-23T14:34:49.941 INFO:teuthology.orchestra.run.smithi105.stderr:dumped monmap epoch 3 2024-04-23T14:34:50.500 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:50 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:34:50.500 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:50 smithi105 ceph-mon[32555]: from='client.? 172.21.15.105:0/2742878504' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T14:34:50.500 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:50 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:34:50.501 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-23T14:34:50.501 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-23T14:34:50.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:50 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:34:50.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:50 smithi028 ceph-mon[27810]: from='client.? 172.21.15.105:0/2742878504' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T14:34:50.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:50 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:34:50.970 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:34:51.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:51 smithi038 ceph-mon[32348]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:51.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:51 smithi038 ceph-mon[32348]: mon.c calling monitor election 2024-04-23T14:34:51.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:51 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T14:34:51.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:51 smithi038 ceph-mon[32348]: mon.a calling monitor election 2024-04-23T14:34:51.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:51 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:34:51.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:51 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:34:51.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:51 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:34:51.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:51 smithi038 ceph-mon[32348]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:51.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:51 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:34:51.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:51 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:34:51.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:51 smithi038 ceph-mon[32348]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:51.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:51 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:34:51.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:51 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:34:51.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:51 smithi038 ceph-mon[32348]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T14:34:51.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:51 smithi038 ceph-mon[32348]: monmap epoch 3 2024-04-23T14:34:51.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:51 smithi038 ceph-mon[32348]: fsid 1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T14:34:51.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:51 smithi038 ceph-mon[32348]: last_changed 2024-04-23T14:34:44.375947+0000 2024-04-23T14:34:51.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:51 smithi038 ceph-mon[32348]: created 2024-04-23T14:32:30.337765+0000 2024-04-23T14:34:51.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:51 smithi038 ceph-mon[32348]: min_mon_release 19 (squid) 2024-04-23T14:34:51.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:51 smithi038 ceph-mon[32348]: election_strategy: 1 2024-04-23T14:34:51.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:51 smithi038 ceph-mon[32348]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-23T14:34:51.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:51 smithi038 ceph-mon[32348]: 1: [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon.c 2024-04-23T14:34:51.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:51 smithi038 ceph-mon[32348]: 2: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.b 2024-04-23T14:34:51.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:51 smithi038 ceph-mon[32348]: fsmap 2024-04-23T14:34:51.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:51 smithi038 ceph-mon[32348]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T14:34:51.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:51 smithi038 ceph-mon[32348]: mgrmap e13: x(active, since 67s) 2024-04-23T14:34:51.875 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:51 smithi038 ceph-mon[32348]: overall HEALTH_OK 2024-04-23T14:34:51.875 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:51 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:51.875 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:51 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:51.875 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:51 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:51.875 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:51 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:51.875 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:51 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:34:51.875 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:51 smithi038 ceph-mon[32348]: from='client.? 172.21.15.105:0/2742878504' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T14:34:51.875 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:51 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:34:52.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:52 smithi028 ceph-mon[27810]: mon.b calling monitor election 2024-04-23T14:34:52.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:52 smithi028 ceph-mon[27810]: Updating smithi028:/etc/ceph/ceph.conf 2024-04-23T14:34:52.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:52 smithi028 ceph-mon[27810]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-23T14:34:52.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:52 smithi028 ceph-mon[27810]: Updating smithi105:/etc/ceph/ceph.conf 2024-04-23T14:34:52.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:52 smithi028 ceph-mon[27810]: mon.b calling monitor election 2024-04-23T14:34:52.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:52 smithi028 ceph-mon[27810]: mon.c calling monitor election 2024-04-23T14:34:52.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:52 smithi028 ceph-mon[27810]: mon.a calling monitor election 2024-04-23T14:34:52.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:52 smithi028 ceph-mon[27810]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-23T14:34:52.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:52 smithi028 ceph-mon[27810]: monmap epoch 3 2024-04-23T14:34:52.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:52 smithi028 ceph-mon[27810]: fsid 1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T14:34:52.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:52 smithi028 ceph-mon[27810]: last_changed 2024-04-23T14:34:44.375947+0000 2024-04-23T14:34:52.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:52 smithi028 ceph-mon[27810]: created 2024-04-23T14:32:30.337765+0000 2024-04-23T14:34:52.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:52 smithi028 ceph-mon[27810]: min_mon_release 19 (squid) 2024-04-23T14:34:52.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:52 smithi028 ceph-mon[27810]: election_strategy: 1 2024-04-23T14:34:52.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:52 smithi028 ceph-mon[27810]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-23T14:34:52.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:52 smithi028 ceph-mon[27810]: 1: [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon.c 2024-04-23T14:34:52.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:52 smithi028 ceph-mon[27810]: 2: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.b 2024-04-23T14:34:52.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:52 smithi028 ceph-mon[27810]: fsmap 2024-04-23T14:34:52.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:52 smithi028 ceph-mon[27810]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T14:34:52.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:52 smithi028 ceph-mon[27810]: mgrmap e13: x(active, since 69s) 2024-04-23T14:34:52.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:52 smithi028 ceph-mon[27810]: overall HEALTH_OK 2024-04-23T14:34:52.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:52 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:52.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:52 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:52.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:52 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:52.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:52 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:52.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:52 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:52.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:52 smithi105 ceph-mon[32555]: mon.b calling monitor election 2024-04-23T14:34:52.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:52 smithi105 ceph-mon[32555]: Updating smithi028:/etc/ceph/ceph.conf 2024-04-23T14:34:52.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:52 smithi105 ceph-mon[32555]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-23T14:34:52.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:52 smithi105 ceph-mon[32555]: Updating smithi105:/etc/ceph/ceph.conf 2024-04-23T14:34:52.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:52 smithi105 ceph-mon[32555]: mon.b calling monitor election 2024-04-23T14:34:52.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:52 smithi105 ceph-mon[32555]: mon.c calling monitor election 2024-04-23T14:34:52.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:52 smithi105 ceph-mon[32555]: mon.a calling monitor election 2024-04-23T14:34:52.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:52 smithi105 ceph-mon[32555]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-23T14:34:52.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:52 smithi105 ceph-mon[32555]: monmap epoch 3 2024-04-23T14:34:52.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:52 smithi105 ceph-mon[32555]: fsid 1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T14:34:52.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:52 smithi105 ceph-mon[32555]: last_changed 2024-04-23T14:34:44.375947+0000 2024-04-23T14:34:52.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:52 smithi105 ceph-mon[32555]: created 2024-04-23T14:32:30.337765+0000 2024-04-23T14:34:52.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:52 smithi105 ceph-mon[32555]: min_mon_release 19 (squid) 2024-04-23T14:34:52.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:52 smithi105 ceph-mon[32555]: election_strategy: 1 2024-04-23T14:34:52.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:52 smithi105 ceph-mon[32555]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-23T14:34:52.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:52 smithi105 ceph-mon[32555]: 1: [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon.c 2024-04-23T14:34:52.538 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:52 smithi105 ceph-mon[32555]: 2: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.b 2024-04-23T14:34:52.538 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:52 smithi105 ceph-mon[32555]: fsmap 2024-04-23T14:34:52.538 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:52 smithi105 ceph-mon[32555]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T14:34:52.538 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:52 smithi105 ceph-mon[32555]: mgrmap e13: x(active, since 69s) 2024-04-23T14:34:52.538 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:52 smithi105 ceph-mon[32555]: overall HEALTH_OK 2024-04-23T14:34:52.538 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:52 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:52.538 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:52 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:52.538 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:52 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:52.538 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:52 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:52.538 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:52 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:52.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:52 smithi038 ceph-mon[32348]: mon.b calling monitor election 2024-04-23T14:34:52.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:52 smithi038 ceph-mon[32348]: Updating smithi028:/etc/ceph/ceph.conf 2024-04-23T14:34:52.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:52 smithi038 ceph-mon[32348]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-23T14:34:52.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:52 smithi038 ceph-mon[32348]: Updating smithi105:/etc/ceph/ceph.conf 2024-04-23T14:34:52.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:52 smithi038 ceph-mon[32348]: mon.b calling monitor election 2024-04-23T14:34:52.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:52 smithi038 ceph-mon[32348]: mon.c calling monitor election 2024-04-23T14:34:52.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:52 smithi038 ceph-mon[32348]: mon.a calling monitor election 2024-04-23T14:34:52.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:52 smithi038 ceph-mon[32348]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-23T14:34:52.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:52 smithi038 ceph-mon[32348]: monmap epoch 3 2024-04-23T14:34:52.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:52 smithi038 ceph-mon[32348]: fsid 1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T14:34:52.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:52 smithi038 ceph-mon[32348]: last_changed 2024-04-23T14:34:44.375947+0000 2024-04-23T14:34:52.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:52 smithi038 ceph-mon[32348]: created 2024-04-23T14:32:30.337765+0000 2024-04-23T14:34:52.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:52 smithi038 ceph-mon[32348]: min_mon_release 19 (squid) 2024-04-23T14:34:52.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:52 smithi038 ceph-mon[32348]: election_strategy: 1 2024-04-23T14:34:52.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:52 smithi038 ceph-mon[32348]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-23T14:34:52.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:52 smithi038 ceph-mon[32348]: 1: [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon.c 2024-04-23T14:34:52.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:52 smithi038 ceph-mon[32348]: 2: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.b 2024-04-23T14:34:52.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:52 smithi038 ceph-mon[32348]: fsmap 2024-04-23T14:34:52.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:52 smithi038 ceph-mon[32348]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T14:34:52.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:52 smithi038 ceph-mon[32348]: mgrmap e13: x(active, since 69s) 2024-04-23T14:34:52.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:52 smithi038 ceph-mon[32348]: overall HEALTH_OK 2024-04-23T14:34:52.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:52 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:52.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:52 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:52.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:52 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:52.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:52 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:52.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:52 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:53.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.237+0000 7fe002600700 1 -- 172.21.15.28:0/3050015204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdffc107480 msgr2=0x7fdffc107860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:53.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.237+0000 7fe002600700 1 --2- 172.21.15.28:0/3050015204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdffc107480 0x7fdffc107860 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdfec009910 tx=0x7fdfec009c20 comp rx=0 tx=0).stop 2024-04-23T14:34:53.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.237+0000 7fe002600700 1 -- 172.21.15.28:0/3050015204 shutdown_connections 2024-04-23T14:34:53.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.237+0000 7fe002600700 1 --2- 172.21.15.28:0/3050015204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdffc107480 0x7fdffc107860 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:53.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.238+0000 7fe002600700 1 -- 172.21.15.28:0/3050015204 >> 172.21.15.28:0/3050015204 conn(0x7fdffc076f50 msgr2=0x7fdffc077360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:53.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.238+0000 7fe002600700 1 -- 172.21.15.28:0/3050015204 shutdown_connections 2024-04-23T14:34:53.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.238+0000 7fe002600700 1 -- 172.21.15.28:0/3050015204 wait complete. 2024-04-23T14:34:53.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.239+0000 7fe002600700 1 Processor -- start 2024-04-23T14:34:53.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.239+0000 7fe002600700 1 -- start start 2024-04-23T14:34:53.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.239+0000 7fe002600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdffc107480 0x7fdffc19a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:53.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.239+0000 7fe002600700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdffc19a9a0 0x7fdffc19ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:53.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.239+0000 7fe002600700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdffc19f270 0x7fdffc19f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:53.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.239+0000 7fe002600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdffc10b9e0 con 0x7fdffc107480 2024-04-23T14:34:53.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.240+0000 7fe002600700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdffc10b860 con 0x7fdffc19a9a0 2024-04-23T14:34:53.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.240+0000 7fe002600700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdffc10bb60 con 0x7fdffc19f270 2024-04-23T14:34:53.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.240+0000 7fdffbe00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdffc19f270 0x7fdffc19f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:53.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.240+0000 7fdffbe00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdffc19f270 0x7fdffc19f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:40808/0 (socket says 172.21.15.28:40808) 2024-04-23T14:34:53.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.240+0000 7fdffbe00700 1 -- 172.21.15.28:0/3498106438 learned_addr learned my addr 172.21.15.28:0/3498106438 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:34:53.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.240+0000 7fdffaa00700 1 --2- 172.21.15.28:0/3498106438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdffc19a9a0 0x7fdffc19ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:53.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.240+0000 7fdffbe00700 1 -- 172.21.15.28:0/3498106438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdffc19a9a0 msgr2=0x7fdffc19ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:53.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.240+0000 7fdffbe00700 1 --2- 172.21.15.28:0/3498106438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdffc19a9a0 0x7fdffc19ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:53.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.240+0000 7fdffbe00700 1 -- 172.21.15.28:0/3498106438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdffc107480 msgr2=0x7fdffc19a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:53.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.241+0000 7fdffb400700 1 --2- 172.21.15.28:0/3498106438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdffc107480 0x7fdffc19a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:53.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.241+0000 7fdffbe00700 1 --2- 172.21.15.28:0/3498106438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdffc107480 0x7fdffc19a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:53.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.241+0000 7fdffbe00700 1 -- 172.21.15.28:0/3498106438 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdffc1a6fe0 con 0x7fdffc19f270 2024-04-23T14:34:53.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.241+0000 7fdffb400700 1 --2- 172.21.15.28:0/3498106438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdffc107480 0x7fdffc19a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:34:53.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.241+0000 7fdffbe00700 1 --2- 172.21.15.28:0/3498106438 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdffc19f270 0x7fdffc19f720 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdff400ba50 tx=0x7fdff400bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:53.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.241+0000 7fdff3e00700 1 -- 172.21.15.28:0/3498106438 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff4019070 con 0x7fdffc19f270 2024-04-23T14:34:53.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.241+0000 7fdff3e00700 1 -- 172.21.15.28:0/3498106438 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdff4014440 con 0x7fdffc19f270 2024-04-23T14:34:53.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.241+0000 7fdff3e00700 1 -- 172.21.15.28:0/3498106438 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff40075b0 con 0x7fdffc19f270 2024-04-23T14:34:53.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.241+0000 7fe002600700 1 -- 172.21.15.28:0/3498106438 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdffc1a7210 con 0x7fdffc19f270 2024-04-23T14:34:53.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.242+0000 7fe002600700 1 -- 172.21.15.28:0/3498106438 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdffc1a7700 con 0x7fdffc19f270 2024-04-23T14:34:53.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.243+0000 7fe002600700 1 -- 172.21.15.28:0/3498106438 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdffc064150 con 0x7fdffc19f270 2024-04-23T14:34:53.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.243+0000 7fdff3e00700 1 -- 172.21.15.28:0/3498106438 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fdff401c070 con 0x7fdffc19f270 2024-04-23T14:34:53.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.243+0000 7fdff3e00700 1 --2- 172.21.15.28:0/3498106438 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fdfe0039ae0 0x7fdfe003bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:53.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.244+0000 7fdff3e00700 1 -- 172.21.15.28:0/3498106438 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fdff404bf50 con 0x7fdffc19f270 2024-04-23T14:34:53.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.244+0000 7fdffb400700 1 --2- 172.21.15.28:0/3498106438 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fdfe0039ae0 0x7fdfe003bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:53.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.245+0000 7fdffb400700 1 --2- 172.21.15.28:0/3498106438 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fdfe0039ae0 0x7fdfe003bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdfec009750 tx=0x7fdfec00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:53.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.249+0000 7fdff3e00700 1 -- 172.21.15.28:0/3498106438 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdff404b7c0 con 0x7fdffc19f270 2024-04-23T14:34:53.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.507+0000 7fe002600700 1 -- 172.21.15.28:0/3498106438 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fdffc02e1a0 con 0x7fdffc19f270 2024-04-23T14:34:53.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:53 smithi028 ceph-mon[27810]: Updating smithi038:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T14:34:53.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:53 smithi028 ceph-mon[27810]: Updating smithi105:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T14:34:53.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:53 smithi028 ceph-mon[27810]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:53.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:53 smithi028 ceph-mon[27810]: Updating smithi028:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T14:34:53.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:53 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:34:53.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:53 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:53.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:53 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:53.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:53 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:53.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:53 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:53.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:53 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:53.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:53 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:53.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:53 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:53.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:53 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T14:34:53.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:53 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T14:34:53.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:53 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:34:53.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.507+0000 7fdff3e00700 1 -- 172.21.15.28:0/3498106438 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+273 (secure 0 0 0) 0x7fdff400e090 con 0x7fdffc19f270 2024-04-23T14:34:53.510 INFO:teuthology.orchestra.run.smithi028.stdout:# minimal ceph.conf for 1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T14:34:53.510 INFO:teuthology.orchestra.run.smithi028.stdout:[global] 2024-04-23T14:34:53.510 INFO:teuthology.orchestra.run.smithi028.stdout: fsid = 1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T14:34:53.510 INFO:teuthology.orchestra.run.smithi028.stdout: mon_host = [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] 2024-04-23T14:34:53.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.511+0000 7fe002600700 1 -- 172.21.15.28:0/3498106438 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fdfe0039ae0 msgr2=0x7fdfe003bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:53.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.511+0000 7fe002600700 1 --2- 172.21.15.28:0/3498106438 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fdfe0039ae0 0x7fdfe003bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdfec009750 tx=0x7fdfec00b540 comp rx=0 tx=0).stop 2024-04-23T14:34:53.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.511+0000 7fe002600700 1 -- 172.21.15.28:0/3498106438 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdffc19f270 msgr2=0x7fdffc19f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:53.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.511+0000 7fe002600700 1 --2- 172.21.15.28:0/3498106438 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdffc19f270 0x7fdffc19f720 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdff400ba50 tx=0x7fdff400bd60 comp rx=0 tx=0).stop 2024-04-23T14:34:53.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.511+0000 7fe002600700 1 -- 172.21.15.28:0/3498106438 shutdown_connections 2024-04-23T14:34:53.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.511+0000 7fe002600700 1 --2- 172.21.15.28:0/3498106438 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fdfe0039ae0 0x7fdfe003bfa0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:53.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.511+0000 7fe002600700 1 --2- 172.21.15.28:0/3498106438 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdffc19f270 0x7fdffc19f720 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:53.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.511+0000 7fe002600700 1 --2- 172.21.15.28:0/3498106438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdffc107480 0x7fdffc19a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:53.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.511+0000 7fe002600700 1 --2- 172.21.15.28:0/3498106438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdffc19a9a0 0x7fdffc19ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:53.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.511+0000 7fe002600700 1 -- 172.21.15.28:0/3498106438 >> 172.21.15.28:0/3498106438 conn(0x7fdffc076f50 msgr2=0x7fdffc06a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:53.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.511+0000 7fe002600700 1 -- 172.21.15.28:0/3498106438 shutdown_connections 2024-04-23T14:34:53.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:34:53.511+0000 7fe002600700 1 -- 172.21.15.28:0/3498106438 wait complete. 2024-04-23T14:34:53.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:53 smithi105 ceph-mon[32555]: Updating smithi038:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T14:34:53.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:53 smithi105 ceph-mon[32555]: Updating smithi105:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T14:34:53.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:53 smithi105 ceph-mon[32555]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:53.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:53 smithi105 ceph-mon[32555]: Updating smithi028:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T14:34:53.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:53 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:34:53.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:53 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:53.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:53 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:53.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:53 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:53.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:53 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:53.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:53 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:53.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:53 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:53.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:53 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:53.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:53 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T14:34:53.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:53 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T14:34:53.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:53 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:34:53.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:53 smithi038 ceph-mon[32348]: Updating smithi038:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T14:34:53.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:53 smithi038 ceph-mon[32348]: Updating smithi105:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T14:34:53.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:53 smithi038 ceph-mon[32348]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:53.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:53 smithi038 ceph-mon[32348]: Updating smithi028:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T14:34:53.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:53 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:34:53.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:53 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:53.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:53 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:53.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:53 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:53.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:53 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:53.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:53 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:53.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:53 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:53.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:53 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:53.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:53 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T14:34:53.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:53 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T14:34:53.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:53 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:34:54.244 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-23T14:34:54.244 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-23T14:34:54.244 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.conf 2024-04-23T14:34:54.329 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-23T14:34:54.329 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-23T14:34:54.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:54 smithi028 ceph-mon[27810]: Reconfiguring mon.a (unknown last config time)... 2024-04-23T14:34:54.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:54 smithi028 ceph-mon[27810]: Reconfiguring daemon mon.a on smithi028 2024-04-23T14:34:54.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:54 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3498106438' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:34:54.453 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-23T14:34:54.453 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.conf 2024-04-23T14:34:54.486 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-23T14:34:54.486 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-23T14:34:54.551 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:54 smithi038 ceph-mon[32348]: Reconfiguring mon.a (unknown last config time)... 2024-04-23T14:34:54.552 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:54 smithi038 ceph-mon[32348]: Reconfiguring daemon mon.a on smithi028 2024-04-23T14:34:54.552 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:54 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3498106438' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:34:54.556 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-23T14:34:54.556 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/ceph/ceph.conf 2024-04-23T14:34:54.578 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:54 smithi105 ceph-mon[32555]: Reconfiguring mon.a (unknown last config time)... 2024-04-23T14:34:54.578 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:54 smithi105 ceph-mon[32555]: Reconfiguring daemon mon.a on smithi028 2024-04-23T14:34:54.578 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:54 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3498106438' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:34:54.583 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-23T14:34:54.583 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-23T14:34:54.648 INFO:tasks.cephadm:Adding mgr.x on smithi028 2024-04-23T14:34:54.648 INFO:tasks.cephadm:Adding mgr.y on smithi038 2024-04-23T14:34:54.648 INFO:tasks.cephadm:Adding mgr.z on smithi105 2024-04-23T14:34:54.648 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi028=x;smithi038=y;smithi105=z' 2024-04-23T14:34:54.935 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T14:34:55.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:55 smithi105 ceph-mon[32555]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:55.594 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:55 smithi028 ceph-mon[27810]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:55.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:55 smithi038 ceph-mon[32348]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:56.609 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.607+0000 7f3e4e7fc700 1 -- 172.21.15.105:0/3780879271 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e400056d0 con 0x7f3e50102680 2024-04-23T14:34:56.609 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.608+0000 7f3e5612a700 1 -- 172.21.15.105:0/3780879271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e50102680 msgr2=0x7f3e50102aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:56.609 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.608+0000 7f3e5612a700 1 --2- 172.21.15.105:0/3780879271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e50102680 0x7f3e50102aa0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3e40009910 tx=0x7f3e40009c20 comp rx=0 tx=0).stop 2024-04-23T14:34:56.609 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.608+0000 7f3e5612a700 1 -- 172.21.15.105:0/3780879271 shutdown_connections 2024-04-23T14:34:56.609 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.608+0000 7f3e5612a700 1 --2- 172.21.15.105:0/3780879271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e50102680 0x7f3e50102aa0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:56.609 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.608+0000 7f3e5612a700 1 -- 172.21.15.105:0/3780879271 >> 172.21.15.105:0/3780879271 conn(0x7f3e500fdc20 msgr2=0x7f3e50100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:56.610 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.608+0000 7f3e5612a700 1 -- 172.21.15.105:0/3780879271 shutdown_connections 2024-04-23T14:34:56.610 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.608+0000 7f3e5612a700 1 -- 172.21.15.105:0/3780879271 wait complete. 2024-04-23T14:34:56.610 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.609+0000 7f3e5612a700 1 Processor -- start 2024-04-23T14:34:56.612 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.609+0000 7f3e5612a700 1 -- start start 2024-04-23T14:34:56.612 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.609+0000 7f3e5612a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e50102680 0x7f3e50076890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:56.612 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.609+0000 7f3e5612a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e50074ee0 0x7f3e50075360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:56.612 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.609+0000 7f3e5612a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e500758a0 0x7f3e501a1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:56.612 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.609+0000 7f3e5612a700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3e501086c0 con 0x7f3e50074ee0 2024-04-23T14:34:56.612 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.609+0000 7f3e5612a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3e50108540 con 0x7f3e500758a0 2024-04-23T14:34:56.612 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.609+0000 7f3e5612a700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3e50108840 con 0x7f3e50102680 2024-04-23T14:34:56.612 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.609+0000 7f3e4f7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e50102680 0x7f3e50076890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:56.612 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.609+0000 7f3e4f7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e50102680 0x7f3e50076890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:37960/0 (socket says 172.21.15.105:37960) 2024-04-23T14:34:56.612 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.609+0000 7f3e4f7fe700 1 -- 172.21.15.105:0/1550603593 learned_addr learned my addr 172.21.15.105:0/1550603593 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-23T14:34:56.612 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.609+0000 7f3e4ffff700 1 --2- 172.21.15.105:0/1550603593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e500758a0 0x7f3e501a1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:56.613 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.609+0000 7f3e4f7fe700 1 -- 172.21.15.105:0/1550603593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e500758a0 msgr2=0x7f3e501a1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:56.613 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.609+0000 7f3e4f7fe700 1 --2- 172.21.15.105:0/1550603593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e500758a0 0x7f3e501a1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:56.613 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.609+0000 7f3e4f7fe700 1 -- 172.21.15.105:0/1550603593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e50074ee0 msgr2=0x7f3e50075360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:56.613 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.609+0000 7f3e4f7fe700 1 --2- 172.21.15.105:0/1550603593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e50074ee0 0x7f3e50075360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:56.613 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.609+0000 7f3e4f7fe700 1 -- 172.21.15.105:0/1550603593 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e501a24b0 con 0x7f3e50102680 2024-04-23T14:34:56.613 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.610+0000 7f3e4f7fe700 1 --2- 172.21.15.105:0/1550603593 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e50102680 0x7f3e50076890 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3e40000c00 tx=0x7f3e400050b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:56.613 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.610+0000 7f3e4cff9700 1 -- 172.21.15.105:0/1550603593 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e40027070 con 0x7f3e50102680 2024-04-23T14:34:56.613 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.610+0000 7f3e5612a700 1 -- 172.21.15.105:0/1550603593 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e501a2740 con 0x7f3e50102680 2024-04-23T14:34:56.613 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.610+0000 7f3e4cff9700 1 -- 172.21.15.105:0/1550603593 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e40022470 con 0x7f3e50102680 2024-04-23T14:34:56.613 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.610+0000 7f3e4cff9700 1 -- 172.21.15.105:0/1550603593 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e4000f6c0 con 0x7f3e50102680 2024-04-23T14:34:56.613 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.610+0000 7f3e5612a700 1 -- 172.21.15.105:0/1550603593 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e501a2ca0 con 0x7f3e50102680 2024-04-23T14:34:56.613 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.611+0000 7f3e4cff9700 1 -- 172.21.15.105:0/1550603593 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f3e4002a070 con 0x7f3e50102680 2024-04-23T14:34:56.613 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.611+0000 7f3e5612a700 1 -- 172.21.15.105:0/1550603593 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e50064150 con 0x7f3e50102680 2024-04-23T14:34:56.613 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.611+0000 7f3e4cff9700 1 --2- 172.21.15.105:0/1550603593 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f3e38039ae0 0x7f3e3803bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:34:56.613 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.611+0000 7f3e4cff9700 1 -- 172.21.15.105:0/1550603593 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f3e4004da50 con 0x7f3e50102680 2024-04-23T14:34:56.614 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.611+0000 7f3e4effd700 1 --2- 172.21.15.105:0/1550603593 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f3e38039ae0 0x7f3e3803bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:34:56.614 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.612+0000 7f3e4effd700 1 --2- 172.21.15.105:0/1550603593 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f3e38039ae0 0x7f3e3803bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3e440098e0 tx=0x7f3e44006dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:34:56.616 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.616+0000 7f3e4cff9700 1 -- 172.21.15.105:0/1550603593 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3e40013070 con 0x7f3e50102680 2024-04-23T14:34:56.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.826+0000 7f3e5612a700 1 -- 172.21.15.105:0/1550603593 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi038=y;smithi105=z", "target": ["mon-mgr", ""]}) -- 0x7f3e50062fc0 con 0x7f3e38039ae0 2024-04-23T14:34:56.841 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.840+0000 7f3e4cff9700 1 -- 172.21.15.105:0/1550603593 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f3e50062fc0 con 0x7f3e38039ae0 2024-04-23T14:34:56.841 INFO:teuthology.orchestra.run.smithi105.stdout:Scheduled mgr update... 2024-04-23T14:34:56.843 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.842+0000 7f3e5612a700 1 -- 172.21.15.105:0/1550603593 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f3e38039ae0 msgr2=0x7f3e3803bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:56.843 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.842+0000 7f3e5612a700 1 --2- 172.21.15.105:0/1550603593 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f3e38039ae0 0x7f3e3803bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3e440098e0 tx=0x7f3e44006dd0 comp rx=0 tx=0).stop 2024-04-23T14:34:56.843 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.842+0000 7f3e5612a700 1 -- 172.21.15.105:0/1550603593 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e50102680 msgr2=0x7f3e50076890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:34:56.843 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.842+0000 7f3e5612a700 1 --2- 172.21.15.105:0/1550603593 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e50102680 0x7f3e50076890 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3e40000c00 tx=0x7f3e400050b0 comp rx=0 tx=0).stop 2024-04-23T14:34:56.843 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.842+0000 7f3e5612a700 1 -- 172.21.15.105:0/1550603593 shutdown_connections 2024-04-23T14:34:56.843 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.842+0000 7f3e5612a700 1 --2- 172.21.15.105:0/1550603593 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f3e38039ae0 0x7f3e3803bfa0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:56.844 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.842+0000 7f3e5612a700 1 --2- 172.21.15.105:0/1550603593 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e50102680 0x7f3e50076890 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:56.844 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.842+0000 7f3e5612a700 1 --2- 172.21.15.105:0/1550603593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e500758a0 0x7f3e501a1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:56.844 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.842+0000 7f3e5612a700 1 --2- 172.21.15.105:0/1550603593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e50074ee0 0x7f3e50075360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:34:56.844 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.842+0000 7f3e5612a700 1 -- 172.21.15.105:0/1550603593 >> 172.21.15.105:0/1550603593 conn(0x7f3e500fdc20 msgr2=0x7f3e500fe8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:34:56.844 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.842+0000 7f3e5612a700 1 -- 172.21.15.105:0/1550603593 shutdown_connections 2024-04-23T14:34:56.844 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:34:56.842+0000 7f3e5612a700 1 -- 172.21.15.105:0/1550603593 wait complete. 2024-04-23T14:34:57.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:56 smithi105 ceph-mon[32555]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:57.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:56 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:57.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:56 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:57.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:56 smithi105 ceph-mon[32555]: Reconfiguring mon.b (monmap changed)... 2024-04-23T14:34:57.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:56 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T14:34:57.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:56 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T14:34:57.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:56 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:34:57.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:56 smithi105 ceph-mon[32555]: Reconfiguring daemon mon.b on smithi038 2024-04-23T14:34:57.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:56 smithi038 ceph-mon[32348]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:57.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:56 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:57.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:56 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:57.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:56 smithi038 ceph-mon[32348]: Reconfiguring mon.b (monmap changed)... 2024-04-23T14:34:57.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:56 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T14:34:57.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:56 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T14:34:57.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:56 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:34:57.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:56 smithi038 ceph-mon[32348]: Reconfiguring daemon mon.b on smithi038 2024-04-23T14:34:57.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:56 smithi028 ceph-mon[27810]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:57.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:56 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:57.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:56 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:57.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:56 smithi028 ceph-mon[27810]: Reconfiguring mon.b (monmap changed)... 2024-04-23T14:34:57.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:56 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T14:34:57.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:56 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T14:34:57.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:56 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:34:57.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:56 smithi028 ceph-mon[27810]: Reconfiguring daemon mon.b on smithi038 2024-04-23T14:34:57.462 DEBUG:teuthology.orchestra.run.smithi038:mgr.y> sudo journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mgr.y.service 2024-04-23T14:34:57.464 DEBUG:teuthology.orchestra.run.smithi105:mgr.z> sudo journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mgr.z.service 2024-04-23T14:34:57.467 INFO:tasks.cephadm:Deploying OSDs... 2024-04-23T14:34:57.467 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-23T14:34:57.467 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/scratch_devs of=/dev/stdout 2024-04-23T14:34:57.528 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-23T14:34:57.528 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_1 2024-04-23T14:34:57.585 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-23T14:34:57.585 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T14:34:57.585 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 6h/6d Inode: 632 Links: 1 2024-04-23T14:34:57.586 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T14:34:57.586 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T14:34:57.586 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-04-23 14:33:55.145300837 +0000 2024-04-23T14:34:57.586 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-04-23 14:28:29.388966080 +0000 2024-04-23T14:34:57.586 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-04-23 14:28:29.388966080 +0000 2024-04-23T14:34:57.586 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-04-23 14:28:29.388966080 +0000 2024-04-23T14:34:57.586 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-23T14:34:57.671 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-04-23T14:34:57.671 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-04-23T14:34:57.671 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000294702 s, 1.7 MB/s 2024-04-23T14:34:57.675 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-23T14:34:57.737 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_2 2024-04-23T14:34:57.790 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-23T14:34:57.790 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T14:34:57.790 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 6h/6d Inode: 651 Links: 1 2024-04-23T14:34:57.791 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T14:34:57.791 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T14:34:57.791 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-04-23 14:33:55.268297353 +0000 2024-04-23T14:34:57.791 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-04-23 14:28:30.005948736 +0000 2024-04-23T14:34:57.791 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-04-23 14:28:30.005948736 +0000 2024-04-23T14:34:57.791 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-04-23 14:28:30.005948736 +0000 2024-04-23T14:34:57.791 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-23T14:34:57.875 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-04-23T14:34:57.875 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-04-23T14:34:57.875 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000288725 s, 1.8 MB/s 2024-04-23T14:34:57.879 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-23T14:34:57.940 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_3 2024-04-23T14:34:57.993 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-23T14:34:57.994 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T14:34:57.994 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 6h/6d Inode: 671 Links: 1 2024-04-23T14:34:57.994 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T14:34:57.994 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T14:34:57.994 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-04-23 14:33:55.365294605 +0000 2024-04-23T14:34:57.994 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-04-23 14:28:30.653930521 +0000 2024-04-23T14:34:57.994 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-04-23 14:28:30.653930521 +0000 2024-04-23T14:34:57.994 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-04-23 14:28:30.653930521 +0000 2024-04-23T14:34:57.994 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-23T14:34:58.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:57 smithi028 ceph-mon[27810]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi038=y;smithi105=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:34:58.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:57 smithi028 ceph-mon[27810]: Saving service mgr spec with placement smithi028=x;smithi038=y;smithi105=z;count:3 2024-04-23T14:34:58.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:57 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:58.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:57 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:58.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:57 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:58.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:57 smithi028 ceph-mon[27810]: Reconfiguring mon.c (monmap changed)... 2024-04-23T14:34:58.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:57 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T14:34:58.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:57 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T14:34:58.069 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:57 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:34:58.069 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:57 smithi028 ceph-mon[27810]: Reconfiguring daemon mon.c on smithi105 2024-04-23T14:34:58.077 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-04-23T14:34:58.077 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-04-23T14:34:58.077 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000289442 s, 1.8 MB/s 2024-04-23T14:34:58.081 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-23T14:34:58.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:57 smithi038 ceph-mon[32348]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi038=y;smithi105=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:34:58.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:57 smithi038 ceph-mon[32348]: Saving service mgr spec with placement smithi028=x;smithi038=y;smithi105=z;count:3 2024-04-23T14:34:58.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:57 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:58.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:57 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:58.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:57 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:58.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:57 smithi038 ceph-mon[32348]: Reconfiguring mon.c (monmap changed)... 2024-04-23T14:34:58.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:57 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T14:34:58.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:57 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T14:34:58.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:57 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:34:58.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:57 smithi038 ceph-mon[32348]: Reconfiguring daemon mon.c on smithi105 2024-04-23T14:34:58.138 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_4 2024-04-23T14:34:58.177 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:57 smithi105 ceph-mon[32555]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi038=y;smithi105=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:34:58.178 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:57 smithi105 ceph-mon[32555]: Saving service mgr spec with placement smithi028=x;smithi038=y;smithi105=z;count:3 2024-04-23T14:34:58.178 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:57 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:58.178 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:57 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:58.178 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:57 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:58.178 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:57 smithi105 ceph-mon[32555]: Reconfiguring mon.c (monmap changed)... 2024-04-23T14:34:58.178 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:57 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T14:34:58.178 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:57 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T14:34:58.178 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:57 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:34:58.178 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:57 smithi105 ceph-mon[32555]: Reconfiguring daemon mon.c on smithi105 2024-04-23T14:34:58.192 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-23T14:34:58.192 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T14:34:58.192 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 6h/6d Inode: 689 Links: 1 2024-04-23T14:34:58.193 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T14:34:58.193 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T14:34:58.193 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-04-23 14:33:55.457291999 +0000 2024-04-23T14:34:58.193 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-04-23 14:28:31.250913740 +0000 2024-04-23T14:34:58.193 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-04-23 14:28:31.250913740 +0000 2024-04-23T14:34:58.193 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-04-23 14:28:31.250913740 +0000 2024-04-23T14:34:58.193 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-23T14:34:58.280 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-04-23T14:34:58.280 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-04-23T14:34:58.280 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000269115 s, 1.9 MB/s 2024-04-23T14:34:58.284 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-23T14:34:58.341 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-23T14:34:58.341 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/scratch_devs of=/dev/stdout 2024-04-23T14:34:58.361 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-23T14:34:58.361 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_1 2024-04-23T14:34:58.420 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-23T14:34:58.420 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T14:34:58.420 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-23T14:34:58.420 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T14:34:58.420 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T14:34:58.420 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-04-23 14:34:43.757041401 +0000 2024-04-23T14:34:58.420 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-04-23 14:27:31.765271438 +0000 2024-04-23T14:34:58.421 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-04-23 14:27:31.765271438 +0000 2024-04-23T14:34:58.421 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-04-23 14:27:31.765271438 +0000 2024-04-23T14:34:58.421 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-23T14:34:58.488 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-04-23T14:34:58.489 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-04-23T14:34:58.489 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000327496 s, 1.6 MB/s 2024-04-23T14:34:58.490 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-23T14:34:58.550 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_2 2024-04-23T14:34:58.608 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-23T14:34:58.608 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T14:34:58.608 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-23T14:34:58.608 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T14:34:58.608 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T14:34:58.608 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-04-23 14:34:43.757041401 +0000 2024-04-23T14:34:58.608 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-04-23 14:27:32.119264510 +0000 2024-04-23T14:34:58.608 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-04-23 14:27:32.119264510 +0000 2024-04-23T14:34:58.608 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-04-23 14:27:32.119264510 +0000 2024-04-23T14:34:58.609 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-23T14:34:58.676 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-04-23T14:34:58.676 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-04-23T14:34:58.676 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000301354 s, 1.7 MB/s 2024-04-23T14:34:58.677 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-23T14:34:58.736 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_3 2024-04-23T14:34:58.794 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-23T14:34:58.794 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T14:34:58.794 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-23T14:34:58.794 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T14:34:58.794 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T14:34:58.794 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-04-23 14:34:43.758041382 +0000 2024-04-23T14:34:58.794 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-04-23 14:27:32.531256447 +0000 2024-04-23T14:34:58.794 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-04-23 14:27:32.531256447 +0000 2024-04-23T14:34:58.794 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-04-23 14:27:32.531256447 +0000 2024-04-23T14:34:58.794 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-23T14:34:58.860 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-04-23T14:34:58.861 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-04-23T14:34:58.861 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000291334 s, 1.8 MB/s 2024-04-23T14:34:58.862 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-23T14:34:58.921 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_4 2024-04-23T14:34:58.979 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-23T14:34:58.979 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T14:34:58.979 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-23T14:34:58.979 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T14:34:58.979 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T14:34:58.979 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-04-23 14:34:43.758041382 +0000 2024-04-23T14:34:58.979 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-04-23 14:27:32.921248815 +0000 2024-04-23T14:34:58.979 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-04-23 14:27:32.921248815 +0000 2024-04-23T14:34:58.980 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-04-23 14:27:32.921248815 +0000 2024-04-23T14:34:58.980 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-23T14:34:59.048 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-04-23T14:34:59.048 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-04-23T14:34:59.048 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000313238 s, 1.6 MB/s 2024-04-23T14:34:59.049 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-23T14:34:59.109 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-23T14:34:59.109 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/scratch_devs of=/dev/stdout 2024-04-23T14:34:59.124 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-23T14:34:59.125 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_1 2024-04-23T14:34:59.179 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-23T14:34:59.179 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T14:34:59.179 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-23T14:34:59.179 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T14:34:59.179 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T14:34:59.179 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-04-23 14:34:34.113374649 +0000 2024-04-23T14:34:59.179 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-04-23 14:27:14.487908010 +0000 2024-04-23T14:34:59.179 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-04-23 14:27:14.487908010 +0000 2024-04-23T14:34:59.179 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-04-23 14:27:14.487908010 +0000 2024-04-23T14:34:59.180 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-23T14:34:59.243 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-04-23T14:34:59.322 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-04-23T14:34:59.322 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000247264 s, 2.1 MB/s 2024-04-23T14:34:59.323 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-23T14:34:59.379 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_2 2024-04-23T14:34:59.436 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-23T14:34:59.436 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T14:34:59.436 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-23T14:34:59.436 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T14:34:59.436 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T14:34:59.436 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-04-23 14:34:34.114374629 +0000 2024-04-23T14:34:59.436 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-04-23 14:27:14.833901027 +0000 2024-04-23T14:34:59.436 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-04-23 14:27:14.833901027 +0000 2024-04-23T14:34:59.436 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-04-23 14:27:14.833901027 +0000 2024-04-23T14:34:59.436 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-23T14:34:59.495 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:59 smithi105 ceph-mon[32555]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:59.496 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:59 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:59.496 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:59 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:59.496 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:59 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:34:59.496 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:59 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:34:59.496 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:59 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:34:59.496 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:59 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:59.496 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:59 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T14:34:59.496 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:59 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T14:34:59.496 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:59 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T14:34:59.496 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:34:59 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:34:59.498 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-04-23T14:34:59.498 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-04-23T14:34:59.498 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.00026471 s, 1.9 MB/s 2024-04-23T14:34:59.499 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-23T14:34:59.555 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_3 2024-04-23T14:34:59.609 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-23T14:34:59.609 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T14:34:59.609 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-23T14:34:59.609 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T14:34:59.609 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T14:34:59.609 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-04-23 14:34:34.114374629 +0000 2024-04-23T14:34:59.609 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-04-23 14:27:15.177894085 +0000 2024-04-23T14:34:59.609 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-04-23 14:27:15.177894085 +0000 2024-04-23T14:34:59.609 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-04-23 14:27:15.177894085 +0000 2024-04-23T14:34:59.610 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-23T14:34:59.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:59 smithi038 ceph-mon[32348]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:59.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:59 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:59.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:59 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:59.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:59 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:34:59.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:59 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:34:59.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:59 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:34:59.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:59 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:59.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:59 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T14:34:59.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:59 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T14:34:59.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:59 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T14:34:59.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:34:59 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:34:59.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:59 smithi028 ceph-mon[27810]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:34:59.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:59 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:59.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:59 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:59.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:59 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:34:59.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:59 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:34:59.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:59 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:34:59.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:59 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:34:59.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:59 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T14:34:59.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:59 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T14:34:59.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:59 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T14:34:59.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:34:59 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:34:59.672 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-04-23T14:34:59.672 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-04-23T14:34:59.673 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000196947 s, 2.6 MB/s 2024-04-23T14:34:59.674 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-23T14:34:59.730 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_4 2024-04-23T14:34:59.785 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-23T14:34:59.786 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T14:34:59.786 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-23T14:34:59.786 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T14:34:59.786 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T14:34:59.786 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-04-23 14:34:34.114374629 +0000 2024-04-23T14:34:59.786 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-04-23 14:27:15.527887022 +0000 2024-04-23T14:34:59.786 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-04-23 14:27:15.527887022 +0000 2024-04-23T14:34:59.786 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-04-23 14:27:15.527887022 +0000 2024-04-23T14:34:59.786 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-23T14:34:59.848 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-04-23T14:34:59.848 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-04-23T14:34:59.848 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000275713 s, 1.9 MB/s 2024-04-23T14:34:59.849 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-23T14:34:59.904 INFO:tasks.cephadm:Deploying osd.0 on smithi028 with /dev/vg_nvme/lv_4... 2024-04-23T14:34:59.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-23T14:35:00.348 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:35:00.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:00 smithi105 ceph-mon[32555]: Deploying daemon mgr.z on smithi105 2024-04-23T14:35:00.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:00 smithi038 ceph-mon[32348]: Deploying daemon mgr.z on smithi105 2024-04-23T14:35:00.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:00 smithi028 ceph-mon[27810]: Deploying daemon mgr.z on smithi105 2024-04-23T14:35:01.036 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:35:00 smithi105 systemd[1]: Starting Ceph mgr.z for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T14:35:01.327 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:01 smithi105 ceph-mon[32555]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:01.327 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:35:01 smithi105 podman[34410]: 2024-04-23 14:35:00.9642093 +0000 UTC m=+0.017753285 image pull 76e77f280c74312ca1add9b6019a4a921a516f91795d85bae00a0c8ca9b1c62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd 2024-04-23T14:35:01.327 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:35:01 smithi105 podman[34410]: 2024-04-23 14:35:01.06473521 +0000 UTC m=+0.118279236 container create afe82dbc72f2d6902a5c7b41314b235c4ba6c1369ca8c767617b3463dfc4bd29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T14:35:01.328 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:35:01 smithi105 podman[34410]: 2024-04-23 14:35:01.300395671 +0000 UTC m=+0.353939652 container init afe82dbc72f2d6902a5c7b41314b235c4ba6c1369ca8c767617b3463dfc4bd29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T14:35:01.328 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:35:01 smithi105 podman[34410]: 2024-04-23 14:35:01.303008839 +0000 UTC m=+0.356552820 container start afe82dbc72f2d6902a5c7b41314b235c4ba6c1369ca8c767617b3463dfc4bd29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240416, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T14:35:01.328 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:35:01 smithi105 ceph-mgr[34429]: -- 172.21.15.105:0/1390945657 <== mon.1 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e1ada1e5a0 con 0x55e1ad9f2800 2024-04-23T14:35:01.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:01 smithi028 ceph-mon[27810]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:01.558 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:01 smithi038 ceph-mon[32348]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:01.653 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:35:01 smithi105 bash[34410]: afe82dbc72f2d6902a5c7b41314b235c4ba6c1369ca8c767617b3463dfc4bd29 2024-04-23T14:35:01.653 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:35:01 smithi105 systemd[1]: Started Ceph mgr.z for 1ee6e996-017e-11ef-bc93-c7b262605968. 2024-04-23T14:35:01.653 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:35:01 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:35:01.468+0000 7f368c84e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T14:35:01.653 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:35:01 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:35:01.652+0000 7f368c84e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T14:35:02.036 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:35:01 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:35:01.754+0000 7f368c84e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T14:35:02.477 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:35:02 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:35:02.056+0000 7f368c84e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T14:35:02.477 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:35:02 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:35:02.139+0000 7f368c84e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T14:35:02.485 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:02 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:02.485 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:02 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:02.486 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:02 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:02.486 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:02 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T14:35:02.486 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:02 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T14:35:02.486 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:02 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T14:35:02.486 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:02 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:35:02.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:02 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:02.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:02 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:02.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:02 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:02.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:02 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T14:35:02.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:02 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T14:35:02.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:02 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T14:35:02.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:02 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:35:02.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:02 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:02.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:02 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:02.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:02 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:02.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:02 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T14:35:02.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:02 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T14:35:02.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:02 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T14:35:02.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:02 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:35:03.003 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-23T14:35:03.003 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-23T14:35:03.003 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-04-23T14:35:03.003 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-04-23T14:35:03.003 INFO:teuthology.orchestra.run.smithi028.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0372801 s, 281 MB/s 2024-04-23T14:35:03.003 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-04-23T14:35:03.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:03 smithi028 ceph-mon[27810]: Deploying daemon mgr.y on smithi038 2024-04-23T14:35:03.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:03 smithi028 ceph-mon[27810]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:03.650 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:03 smithi105 ceph-mon[32555]: Deploying daemon mgr.y on smithi038 2024-04-23T14:35:03.650 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:03 smithi105 ceph-mon[32555]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:03.651 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:35:03 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:35:03.314+0000 7f368c84e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T14:35:03.651 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:35:03 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:35:03.481+0000 7f368c84e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T14:35:03.651 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:35:03 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:35:03.649+0000 7f368c84e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T14:35:03.758 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_4 2024-04-23T14:35:03.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:03 smithi038 ceph-mon[32348]: Deploying daemon mgr.y on smithi038 2024-04-23T14:35:03.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:03 smithi038 ceph-mon[32348]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:03.873 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:35:03 smithi038 systemd[1]: Starting Ceph mgr.y for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T14:35:04.143 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:35:03 smithi038 podman[33727]: 2024-04-23 14:35:03.776545006 +0000 UTC m=+0.018336264 image pull 76e77f280c74312ca1add9b6019a4a921a516f91795d85bae00a0c8ca9b1c62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd 2024-04-23T14:35:04.143 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:35:03 smithi038 podman[33727]: 2024-04-23 14:35:03.885076955 +0000 UTC m=+0.126868215 container create bf6cd51d0f02f4c6094c6e5dc35c0aa41807ce5e6d44e08e677dc2d374b64c2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T14:35:04.143 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:35:04 smithi038 podman[33727]: 2024-04-23 14:35:04.114696941 +0000 UTC m=+0.356488214 container init bf6cd51d0f02f4c6094c6e5dc35c0aa41807ce5e6d44e08e677dc2d374b64c2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-23T14:35:04.143 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:35:04 smithi038 podman[33727]: 2024-04-23 14:35:04.117919511 +0000 UTC m=+0.359710768 container start bf6cd51d0f02f4c6094c6e5dc35c0aa41807ce5e6d44e08e677dc2d374b64c2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-23T14:35:04.214 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:35:04.225 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:35:03 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:35:03.933+0000 7f368c84e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T14:35:04.225 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:35:04 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:35:04.024+0000 7f368c84e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T14:35:04.225 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:35:04 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:35:04.224+0000 7f368c84e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T14:35:04.468 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:35:04 smithi038 ceph-mgr[33746]: -- 172.21.15.38:0/843614408 <== mon.2 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ceea4cc5a0 con 0x55ceea4a1000 2024-04-23T14:35:04.468 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:35:04 smithi038 bash[33727]: bf6cd51d0f02f4c6094c6e5dc35c0aa41807ce5e6d44e08e677dc2d374b64c2c 2024-04-23T14:35:04.468 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:35:04 smithi038 systemd[1]: Started Ceph mgr.y for 1ee6e996-017e-11ef-bc93-c7b262605968. 2024-04-23T14:35:04.469 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:35:04 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:35:04.273+0000 7f7c972f6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T14:35:04.469 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:35:04 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:35:04.467+0000 7f7c972f6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T14:35:04.536 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:35:04 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:35:04.415+0000 7f368c84e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T14:35:04.872 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:35:04 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:35:04.574+0000 7f7c972f6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T14:35:05.285 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:35:05 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:35:05.030+0000 7f368c84e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T14:35:05.285 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:35:04 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:35:04.906+0000 7f7c972f6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T14:35:05.285 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:35:04 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:35:04.995+0000 7f7c972f6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T14:35:05.299 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:05 smithi028 ceph-mon[27810]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:05.299 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:05 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:05.299 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:05 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:05.299 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:05 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:05.299 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:05 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:05.299 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:05 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:35:05.536 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:35:05 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:35:05.472+0000 7f368c84e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T14:35:05.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:05 smithi105 ceph-mon[32555]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:05.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:05 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:05.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:05 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:05.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:05 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:05.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:05 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:05.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:05 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:35:05.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:05 smithi038 ceph-mon[32348]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:05.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:05 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:05.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:05 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:05.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:05 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:05.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:05 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:05.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:05 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:35:06.036 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:35:05 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:35:05.573+0000 7f368c84e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T14:35:06.036 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:35:05 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:35:05.776+0000 7f368c84e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T14:35:06.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.120+0000 7f6ebd800700 1 -- 172.21.15.28:0/67676564 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6eb8105f10 msgr2=0x7f6eb8108300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:35:06.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.120+0000 7f6ebd800700 1 --2- 172.21.15.28:0/67676564 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6eb8105f10 0x7f6eb8108300 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6ea40098f0 tx=0x7f6ea4009c00 comp rx=0 tx=0).stop 2024-04-23T14:35:06.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.120+0000 7f6ebd800700 1 -- 172.21.15.28:0/67676564 shutdown_connections 2024-04-23T14:35:06.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.120+0000 7f6ebd800700 1 --2- 172.21.15.28:0/67676564 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6eb81035e0 0x7f6eb81059d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:35:06.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.120+0000 7f6ebd800700 1 --2- 172.21.15.28:0/67676564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6eb8108840 0x7f6eb810acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:35:06.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.120+0000 7f6ebd800700 1 --2- 172.21.15.28:0/67676564 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6eb8105f10 0x7f6eb8108300 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:35:06.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.120+0000 7f6ebd800700 1 -- 172.21.15.28:0/67676564 >> 172.21.15.28:0/67676564 conn(0x7f6eb80fcff0 msgr2=0x7f6eb80ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:35:06.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.120+0000 7f6ebd800700 1 -- 172.21.15.28:0/67676564 shutdown_connections 2024-04-23T14:35:06.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.120+0000 7f6ebd800700 1 -- 172.21.15.28:0/67676564 wait complete. 2024-04-23T14:35:06.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.121+0000 7f6ebd800700 1 Processor -- start 2024-04-23T14:35:06.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.121+0000 7f6ebd800700 1 -- start start 2024-04-23T14:35:06.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.122+0000 7f6ebd800700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6eb81035e0 0x7f6eb819b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:35:06.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.122+0000 7f6ebd800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6eb8105f10 0x7f6eb819b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:35:06.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.122+0000 7f6ebd800700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6eb8108840 0x7f6eb81a2620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:35:06.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.122+0000 7f6ebd800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb810e7a0 con 0x7f6eb8105f10 2024-04-23T14:35:06.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.122+0000 7f6ebd800700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb810e620 con 0x7f6eb8108840 2024-04-23T14:35:06.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.122+0000 7f6ebd800700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb810e920 con 0x7f6eb81035e0 2024-04-23T14:35:06.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.122+0000 7f6eb6a00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6eb81035e0 0x7f6eb819b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:35:06.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.122+0000 7f6eb6000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6eb8105f10 0x7f6eb819b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:35:06.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.122+0000 7f6eb7400700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6eb8108840 0x7f6eb81a2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:35:06.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.122+0000 7f6eb6a00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6eb81035e0 0x7f6eb819b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:41814/0 (socket says 172.21.15.28:41814) 2024-04-23T14:35:06.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.122+0000 7f6eb6a00700 1 -- 172.21.15.28:0/886139486 learned_addr learned my addr 172.21.15.28:0/886139486 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:35:06.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.122+0000 7f6eb6a00700 1 -- 172.21.15.28:0/886139486 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6eb8108840 msgr2=0x7f6eb81a2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:35:06.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.122+0000 7f6eb6a00700 1 --2- 172.21.15.28:0/886139486 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6eb8108840 0x7f6eb81a2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:35:06.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.123+0000 7f6eb6a00700 1 -- 172.21.15.28:0/886139486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6eb8105f10 msgr2=0x7f6eb819b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:35:06.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.123+0000 7f6eb6a00700 1 --2- 172.21.15.28:0/886139486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6eb8105f10 0x7f6eb819b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:35:06.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.123+0000 7f6eb6a00700 1 -- 172.21.15.28:0/886139486 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eb81a2cc0 con 0x7f6eb81035e0 2024-04-23T14:35:06.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.123+0000 7f6eb7400700 1 --2- 172.21.15.28:0/886139486 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6eb8108840 0x7f6eb81a2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:35:06.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.123+0000 7f6eb6a00700 1 --2- 172.21.15.28:0/886139486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6eb81035e0 0x7f6eb819b060 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6ea800ea20 tx=0x7f6ea800ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:35:06.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.123+0000 7f6eaf400700 1 -- 172.21.15.28:0/886139486 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea8004a30 con 0x7f6eb81035e0 2024-04-23T14:35:06.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.123+0000 7f6eaf400700 1 -- 172.21.15.28:0/886139486 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ea8005500 con 0x7f6eb81035e0 2024-04-23T14:35:06.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.123+0000 7f6ebd800700 1 -- 172.21.15.28:0/886139486 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6eb81a2fb0 con 0x7f6eb81035e0 2024-04-23T14:35:06.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.123+0000 7f6eaf400700 1 -- 172.21.15.28:0/886139486 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea800ce30 con 0x7f6eb81035e0 2024-04-23T14:35:06.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.123+0000 7f6ebd800700 1 -- 172.21.15.28:0/886139486 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6eb81a3400 con 0x7f6eb81035e0 2024-04-23T14:35:06.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.124+0000 7f6ebd800700 1 -- 172.21.15.28:0/886139486 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6eb80697e0 con 0x7f6eb81035e0 2024-04-23T14:35:06.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.125+0000 7f6eaf400700 1 -- 172.21.15.28:0/886139486 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f6ea80104d0 con 0x7f6eb81035e0 2024-04-23T14:35:06.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.125+0000 7f6eaf400700 1 --2- 172.21.15.28:0/886139486 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f6e9c039890 0x7f6e9c03bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:35:06.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.125+0000 7f6eaf400700 1 -- 172.21.15.28:0/886139486 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f6ea804ced0 con 0x7f6eb81035e0 2024-04-23T14:35:06.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.125+0000 7f6eb6000700 1 --2- 172.21.15.28:0/886139486 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f6e9c039890 0x7f6e9c03bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:35:06.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.126+0000 7f6eb6000700 1 --2- 172.21.15.28:0/886139486 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f6e9c039890 0x7f6e9c03bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4009810 tx=0x7f6ea4019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:35:06.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.130+0000 7f6eaf400700 1 -- 172.21.15.28:0/886139486 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6ea8016bd0 con 0x7f6eb81035e0 2024-04-23T14:35:06.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:06.384+0000 7f6ebd800700 1 -- 172.21.15.28:0/886139486 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f6eb8062f30 con 0x7f6e9c039890 2024-04-23T14:35:06.622 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:35:06 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:35:06.315+0000 7f7c972f6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T14:35:06.622 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:35:06 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:35:06.491+0000 7f7c972f6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T14:35:06.641 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:35:06 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:35:06.309+0000 7f368c84e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T14:35:06.641 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:35:06 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:35:06.401+0000 7f368c84e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T14:35:06.967 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:35:06 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:35:06.668+0000 7f7c972f6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T14:35:06.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:06 smithi038 ceph-mon[32348]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:06.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:06 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:06.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:06 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:06.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:06 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:06.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:06 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:06.967 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:06 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:35:06.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:06 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:35:06.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:06 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:06.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:06 smithi038 ceph-mon[32348]: Reconfiguring mgr.x (unknown last config time)... 2024-04-23T14:35:06.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:06 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T14:35:06.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:06 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T14:35:06.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:06 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:35:06.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:06 smithi038 ceph-mon[32348]: Reconfiguring daemon mgr.x on smithi028 2024-04-23T14:35:06.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:06 smithi038 ceph-mon[32348]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:35:06.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:06 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:35:06.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:06 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:35:06.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:06 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:35:06.968 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:06 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:06.978 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:35:06 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:35:06.640+0000 7f368c84e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T14:35:06.978 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:35:06 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:35:06.771+0000 7f368c84e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T14:35:06.978 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:35:06 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:35:06.864+0000 7f368c84e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T14:35:06.978 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:35:06 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:35:06.977+0000 7f368c84e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T14:35:06.978 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:06 smithi105 ceph-mon[32555]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:06.979 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:06 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:06.979 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:06 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:06.979 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:06 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:06.979 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:06 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:06.979 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:06 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:35:06.979 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:06 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:35:06.979 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:06 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:06.980 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:06 smithi105 ceph-mon[32555]: Reconfiguring mgr.x (unknown last config time)... 2024-04-23T14:35:06.980 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:06 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T14:35:06.980 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:06 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T14:35:06.980 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:06 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:35:06.980 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:06 smithi105 ceph-mon[32555]: Reconfiguring daemon mgr.x on smithi028 2024-04-23T14:35:06.980 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:06 smithi105 ceph-mon[32555]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:35:06.981 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:06 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:35:06.981 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:06 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:35:06.981 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:06 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:35:06.981 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:06 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:07.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:06 smithi028 ceph-mon[27810]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:07.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:06 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:07.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:06 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:07.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:06 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:07.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:06 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:07.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:06 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:35:07.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:06 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:35:07.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:06 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:07.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:06 smithi028 ceph-mon[27810]: Reconfiguring mgr.x (unknown last config time)... 2024-04-23T14:35:07.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:06 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T14:35:07.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:06 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T14:35:07.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:06 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:35:07.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:06 smithi028 ceph-mon[27810]: Reconfiguring daemon mgr.x on smithi028 2024-04-23T14:35:07.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:06 smithi028 ceph-mon[27810]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:35:07.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:06 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:35:07.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:06 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:35:07.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:06 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:35:07.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:06 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:07.272 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:35:06 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:35:06.966+0000 7f7c972f6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T14:35:07.272 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:35:07 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:35:07.061+0000 7f7c972f6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T14:35:07.272 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:35:07 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:35:07.272+0000 7f7c972f6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T14:35:07.536 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:35:07 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:35:07.275+0000 7f368c84e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T14:35:07.622 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:35:07 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:35:07.470+0000 7f7c972f6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T14:35:08.286 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:35:07 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:35:07.869+0000 7f368c84e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T14:35:08.286 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:35:08 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:35:08.080+0000 7f368c84e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T14:35:08.372 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:35:08 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:35:08.096+0000 7f7c972f6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T14:35:08.839 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:35:08 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:35:08.533+0000 7f7c972f6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T14:35:08.839 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:35:08 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:35:08.631+0000 7f7c972f6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T14:35:08.839 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:35:08 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:35:08.838+0000 7f7c972f6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T14:35:08.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:08.871+0000 7f6eaf400700 1 -- 172.21.15.28:0/886139486 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mgrmap(e 14) ==== 92602+0+0 (secure 0 0 0) 0x7f6ea80217c0 con 0x7f6eb81035e0 2024-04-23T14:35:09.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:08 smithi038 ceph-mon[32348]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:09.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:08 smithi038 ceph-mon[32348]: Standby manager daemon z started 2024-04-23T14:35:09.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:08 smithi038 ceph-mon[32348]: from='mgr.? 172.21.15.105:0/1149420372' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-23T14:35:09.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:08 smithi038 ceph-mon[32348]: from='mgr.? 172.21.15.105:0/1149420372' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T14:35:09.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:08 smithi038 ceph-mon[32348]: from='mgr.? 172.21.15.105:0/1149420372' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-23T14:35:09.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:08 smithi038 ceph-mon[32348]: from='mgr.? 172.21.15.105:0/1149420372' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T14:35:09.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:08 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:09.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:08 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:09.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:08 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:35:09.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:08 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:35:09.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:08 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:35:09.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:08 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:09.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:08 smithi028 ceph-mon[27810]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:09.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:08 smithi028 ceph-mon[27810]: Standby manager daemon z started 2024-04-23T14:35:09.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:08 smithi028 ceph-mon[27810]: from='mgr.? 172.21.15.105:0/1149420372' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-23T14:35:09.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:08 smithi028 ceph-mon[27810]: from='mgr.? 172.21.15.105:0/1149420372' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T14:35:09.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:08 smithi028 ceph-mon[27810]: from='mgr.? 172.21.15.105:0/1149420372' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-23T14:35:09.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:08 smithi028 ceph-mon[27810]: from='mgr.? 172.21.15.105:0/1149420372' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T14:35:09.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:08 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:09.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:08 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:09.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:08 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:35:09.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:08 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:35:09.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:08 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:35:09.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:08 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:09.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:08 smithi105 ceph-mon[32555]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:09.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:08 smithi105 ceph-mon[32555]: Standby manager daemon z started 2024-04-23T14:35:09.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:08 smithi105 ceph-mon[32555]: from='mgr.? 172.21.15.105:0/1149420372' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-23T14:35:09.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:08 smithi105 ceph-mon[32555]: from='mgr.? 172.21.15.105:0/1149420372' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T14:35:09.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:08 smithi105 ceph-mon[32555]: from='mgr.? 172.21.15.105:0/1149420372' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-23T14:35:09.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:08 smithi105 ceph-mon[32555]: from='mgr.? 172.21.15.105:0/1149420372' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T14:35:09.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:08 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:09.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:08 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:09.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:08 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:35:09.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:08 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:35:09.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:08 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:35:09.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:08 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:09.730 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:35:09 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:35:09.385+0000 7f7c972f6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T14:35:09.731 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:35:09 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:35:09.479+0000 7f7c972f6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T14:35:10.086 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:09 smithi038 ceph-mon[32348]: mgrmap e14: x(active, since 87s), standbys: z 2024-04-23T14:35:10.086 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:09 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-23T14:35:10.086 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:35:09 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:35:09.730+0000 7f7c972f6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T14:35:10.086 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:35:09 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:35:09.868+0000 7f7c972f6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T14:35:10.086 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:35:09 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:35:09.966+0000 7f7c972f6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T14:35:10.087 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:35:10 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:35:10.085+0000 7f7c972f6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T14:35:10.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:09 smithi105 ceph-mon[32555]: mgrmap e14: x(active, since 87s), standbys: z 2024-04-23T14:35:10.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:09 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-23T14:35:10.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:09 smithi028 ceph-mon[27810]: mgrmap e14: x(active, since 87s), standbys: z 2024-04-23T14:35:10.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:09 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-23T14:35:10.872 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:35:10 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:35:10.398+0000 7f7c972f6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T14:35:11.250 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:10 smithi038 ceph-mon[32348]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:11.250 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:35:11 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:35:11.025+0000 7f7c972f6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T14:35:11.250 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:35:11 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:35:11.249+0000 7f7c972f6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T14:35:11.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:10 smithi105 ceph-mon[32555]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:11.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:10 smithi028 ceph-mon[27810]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:12.102 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:11 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1409604805' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0b427bc-be65-4543-99cc-2f53ad98ddad"}]: dispatch 2024-04-23T14:35:12.102 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:11 smithi028 ceph-mon[27810]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0b427bc-be65-4543-99cc-2f53ad98ddad"}]: dispatch 2024-04-23T14:35:12.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:11 smithi028 ceph-mon[27810]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e0b427bc-be65-4543-99cc-2f53ad98ddad"}]': finished 2024-04-23T14:35:12.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:11 smithi028 ceph-mon[27810]: osdmap e5: 1 total, 0 up, 1 in 2024-04-23T14:35:12.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:11 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T14:35:12.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:11 smithi028 ceph-mon[27810]: from='mgr.? 172.21.15.38:0/592651726' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T14:35:12.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:11 smithi028 ceph-mon[27810]: Standby manager daemon y started 2024-04-23T14:35:12.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:11 smithi028 ceph-mon[27810]: from='mgr.? 172.21.15.38:0/592651726' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T14:35:12.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:11 smithi028 ceph-mon[27810]: from='mgr.? 172.21.15.38:0/592651726' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T14:35:12.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:11 smithi028 ceph-mon[27810]: from='mgr.? 172.21.15.38:0/592651726' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T14:35:12.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:12.146+0000 7f6eaf400700 1 -- 172.21.15.28:0/886139486 <== mon.1 v2:172.21.15.105:3300/0 8 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f6ea801f340 con 0x7f6eb81035e0 2024-04-23T14:35:12.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:11 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1409604805' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0b427bc-be65-4543-99cc-2f53ad98ddad"}]: dispatch 2024-04-23T14:35:12.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:11 smithi105 ceph-mon[32555]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0b427bc-be65-4543-99cc-2f53ad98ddad"}]: dispatch 2024-04-23T14:35:12.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:11 smithi105 ceph-mon[32555]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e0b427bc-be65-4543-99cc-2f53ad98ddad"}]': finished 2024-04-23T14:35:12.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:11 smithi105 ceph-mon[32555]: osdmap e5: 1 total, 0 up, 1 in 2024-04-23T14:35:12.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:11 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T14:35:12.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:11 smithi105 ceph-mon[32555]: from='mgr.? 172.21.15.38:0/592651726' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T14:35:12.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:11 smithi105 ceph-mon[32555]: Standby manager daemon y started 2024-04-23T14:35:12.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:11 smithi105 ceph-mon[32555]: from='mgr.? 172.21.15.38:0/592651726' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T14:35:12.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:11 smithi105 ceph-mon[32555]: from='mgr.? 172.21.15.38:0/592651726' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T14:35:12.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:11 smithi105 ceph-mon[32555]: from='mgr.? 172.21.15.38:0/592651726' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T14:35:12.320 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:11 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1409604805' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0b427bc-be65-4543-99cc-2f53ad98ddad"}]: dispatch 2024-04-23T14:35:12.320 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:11 smithi038 ceph-mon[32348]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0b427bc-be65-4543-99cc-2f53ad98ddad"}]: dispatch 2024-04-23T14:35:12.320 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:11 smithi038 ceph-mon[32348]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e0b427bc-be65-4543-99cc-2f53ad98ddad"}]': finished 2024-04-23T14:35:12.320 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:11 smithi038 ceph-mon[32348]: osdmap e5: 1 total, 0 up, 1 in 2024-04-23T14:35:12.320 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:11 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T14:35:12.320 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:11 smithi038 ceph-mon[32348]: from='mgr.? 172.21.15.38:0/592651726' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T14:35:12.321 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:11 smithi038 ceph-mon[32348]: Standby manager daemon y started 2024-04-23T14:35:12.321 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:11 smithi038 ceph-mon[32348]: from='mgr.? 172.21.15.38:0/592651726' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T14:35:12.321 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:11 smithi038 ceph-mon[32348]: from='mgr.? 172.21.15.38:0/592651726' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T14:35:12.321 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:11 smithi038 ceph-mon[32348]: from='mgr.? 172.21.15.38:0/592651726' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T14:35:13.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:12 smithi105 ceph-mon[32555]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:13.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:12 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1878907890' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:35:13.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:12 smithi105 ceph-mon[32555]: mgrmap e15: x(active, since 90s), standbys: z, y 2024-04-23T14:35:13.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:12 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T14:35:13.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:12 smithi038 ceph-mon[32348]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:13.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:12 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1878907890' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:35:13.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:12 smithi038 ceph-mon[32348]: mgrmap e15: x(active, since 90s), standbys: z, y 2024-04-23T14:35:13.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:12 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T14:35:13.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:12 smithi028 ceph-mon[27810]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:13.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:12 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1878907890' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:35:13.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:12 smithi028 ceph-mon[27810]: mgrmap e15: x(active, since 90s), standbys: z, y 2024-04-23T14:35:13.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:12 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T14:35:15.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:14 smithi105 ceph-mon[32555]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:15.337 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:14 smithi028 ceph-mon[27810]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:15.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:14 smithi038 ceph-mon[32348]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:17.250 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:16 smithi028 ceph-mon[27810]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:17.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:16 smithi105 ceph-mon[32555]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:17.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:16 smithi038 ceph-mon[32348]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:19.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:18 smithi028 ceph-mon[27810]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:19.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:18 smithi105 ceph-mon[32555]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:19.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:18 smithi038 ceph-mon[32348]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:21.233 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:20 smithi028 ceph-mon[27810]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:21.233 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:20 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-23T14:35:21.233 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:20 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:35:21.234 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:20 smithi028 ceph-mon[27810]: Deploying daemon osd.0 on smithi028 2024-04-23T14:35:21.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:20 smithi105 ceph-mon[32555]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:21.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:20 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-23T14:35:21.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:20 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:35:21.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:20 smithi105 ceph-mon[32555]: Deploying daemon osd.0 on smithi028 2024-04-23T14:35:21.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:20 smithi038 ceph-mon[32348]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:21.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:20 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-23T14:35:21.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:20 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:35:21.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:20 smithi038 ceph-mon[32348]: Deploying daemon osd.0 on smithi028 2024-04-23T14:35:23.000 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:22 smithi028 ceph-mon[27810]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:23.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:22 smithi105 ceph-mon[32555]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:23.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:22 smithi038 ceph-mon[32348]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:25.244 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:24 smithi028 ceph-mon[27810]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:25.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:24 smithi105 ceph-mon[32555]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:25.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:24 smithi038 ceph-mon[32348]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:27.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:26 smithi105 ceph-mon[32555]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:27.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:26 smithi038 ceph-mon[32348]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:27.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:26 smithi028 ceph-mon[27810]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:29.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:28 smithi105 ceph-mon[32555]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:29.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:28 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:29.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:28 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:29.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:28 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:35:29.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:28 smithi038 ceph-mon[32348]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:29.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:28 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:29.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:28 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:29.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:28 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:35:29.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:28 smithi028 ceph-mon[27810]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:29.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:29.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:29.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:35:31.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:30 smithi028 ceph-mon[27810]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:31.189 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:30 smithi038 ceph-mon[32348]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:31.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:30 smithi105 ceph-mon[32555]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:32.925 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:32 smithi028 ceph-mon[27810]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:32.925 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:32 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:32.925 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:32 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:32.925 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:32 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:35:32.925 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:32 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:35:32.925 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:32 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:32.925 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:32 smithi028 ceph-mon[27810]: from='osd.0 [v2:172.21.15.28:6802/2993445336,v1:172.21.15.28:6803/2993445336]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-23T14:35:32.926 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:32 smithi028 ceph-mon[27810]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-23T14:35:33.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:32 smithi105 ceph-mon[32555]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:33.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:32 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:33.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:32 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:33.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:32 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:35:33.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:32 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:35:33.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:32 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:33.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:32 smithi105 ceph-mon[32555]: from='osd.0 [v2:172.21.15.28:6802/2993445336,v1:172.21.15.28:6803/2993445336]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-23T14:35:33.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:32 smithi105 ceph-mon[32555]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-23T14:35:33.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:32 smithi038 ceph-mon[32348]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:33.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:32 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:33.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:32 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:33.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:32 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:35:33.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:32 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:35:33.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:32 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:33.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:32 smithi038 ceph-mon[32348]: from='osd.0 [v2:172.21.15.28:6802/2993445336,v1:172.21.15.28:6803/2993445336]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-23T14:35:33.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:32 smithi038 ceph-mon[32348]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-23T14:35:33.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:33.682+0000 7f6eaf400700 1 -- 172.21.15.28:0/886139486 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6eb8062f30 con 0x7f6e9c039890 2024-04-23T14:35:33.684 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 0 on host 'smithi028' 2024-04-23T14:35:33.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:33.686+0000 7f6ebd800700 1 -- 172.21.15.28:0/886139486 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f6e9c039890 msgr2=0x7f6e9c03bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:35:33.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:33.686+0000 7f6ebd800700 1 --2- 172.21.15.28:0/886139486 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f6e9c039890 0x7f6e9c03bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4009810 tx=0x7f6ea4019040 comp rx=0 tx=0).stop 2024-04-23T14:35:33.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:33.686+0000 7f6ebd800700 1 -- 172.21.15.28:0/886139486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6eb81035e0 msgr2=0x7f6eb819b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:35:33.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:33.686+0000 7f6ebd800700 1 --2- 172.21.15.28:0/886139486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6eb81035e0 0x7f6eb819b060 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6ea800ea20 tx=0x7f6ea800ed30 comp rx=0 tx=0).stop 2024-04-23T14:35:33.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:33.686+0000 7f6ebd800700 1 -- 172.21.15.28:0/886139486 shutdown_connections 2024-04-23T14:35:33.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:33.686+0000 7f6ebd800700 1 --2- 172.21.15.28:0/886139486 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f6e9c039890 0x7f6e9c03bd50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:35:33.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:33.686+0000 7f6ebd800700 1 --2- 172.21.15.28:0/886139486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6eb81035e0 0x7f6eb819b060 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:35:33.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:33.686+0000 7f6ebd800700 1 --2- 172.21.15.28:0/886139486 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6eb8108840 0x7f6eb81a2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:35:33.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:33.686+0000 7f6ebd800700 1 --2- 172.21.15.28:0/886139486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6eb8105f10 0x7f6eb819b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:35:33.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:33.686+0000 7f6ebd800700 1 -- 172.21.15.28:0/886139486 >> 172.21.15.28:0/886139486 conn(0x7f6eb80fcff0 msgr2=0x7f6eb8102e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:35:33.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:33.686+0000 7f6ebd800700 1 -- 172.21.15.28:0/886139486 shutdown_connections 2024-04-23T14:35:33.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:35:33.686+0000 7f6ebd800700 1 -- 172.21.15.28:0/886139486 wait complete. 2024-04-23T14:35:34.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:33 smithi105 ceph-mon[32555]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-23T14:35:34.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:33 smithi105 ceph-mon[32555]: from='osd.0 [v2:172.21.15.28:6802/2993445336,v1:172.21.15.28:6803/2993445336]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-23T14:35:34.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:33 smithi105 ceph-mon[32555]: osdmap e6: 1 total, 0 up, 1 in 2024-04-23T14:35:34.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:33 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T14:35:34.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:33 smithi105 ceph-mon[32555]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-23T14:35:34.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:33 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:34.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:33 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:34.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:33 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:35:34.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:33 smithi028 ceph-mon[27810]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-23T14:35:34.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:33 smithi028 ceph-mon[27810]: from='osd.0 [v2:172.21.15.28:6802/2993445336,v1:172.21.15.28:6803/2993445336]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-23T14:35:34.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:33 smithi028 ceph-mon[27810]: osdmap e6: 1 total, 0 up, 1 in 2024-04-23T14:35:34.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:33 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T14:35:34.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:33 smithi028 ceph-mon[27810]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-23T14:35:34.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:33 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:34.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:33 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:34.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:33 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:35:34.372 DEBUG:teuthology.orchestra.run.smithi028:osd.0> sudo journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.0.service 2024-04-23T14:35:34.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:33 smithi038 ceph-mon[32348]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-23T14:35:34.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:33 smithi038 ceph-mon[32348]: from='osd.0 [v2:172.21.15.28:6802/2993445336,v1:172.21.15.28:6803/2993445336]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-23T14:35:34.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:33 smithi038 ceph-mon[32348]: osdmap e6: 1 total, 0 up, 1 in 2024-04-23T14:35:34.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:33 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T14:35:34.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:33 smithi038 ceph-mon[32348]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-23T14:35:34.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:33 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:34.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:33 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:34.374 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:33 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:35:34.378 INFO:tasks.cephadm:Deploying osd.1 on smithi038 with /dev/vg_nvme/lv_4... 2024-04-23T14:35:34.378 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-23T14:35:34.644 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T14:35:35.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:34 smithi028 ceph-mon[27810]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:35.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:34 smithi028 ceph-mon[27810]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-23T14:35:35.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:34 smithi028 ceph-mon[27810]: osdmap e7: 1 total, 0 up, 1 in 2024-04-23T14:35:35.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:34 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T14:35:35.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:34 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T14:35:35.017 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 23 14:35:34 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-0[40196]: 2024-04-23T14:35:34.970+0000 7f767fa00700 -1 osd.0 0 waiting for initial osdmap 2024-04-23T14:35:35.017 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 23 14:35:34 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-0[40196]: 2024-04-23T14:35:34.983+0000 7f7675800700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:35:35.093 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:34 smithi038 ceph-mon[32348]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:35.093 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:34 smithi038 ceph-mon[32348]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-23T14:35:35.093 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:34 smithi038 ceph-mon[32348]: osdmap e7: 1 total, 0 up, 1 in 2024-04-23T14:35:35.093 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:34 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T14:35:35.093 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:34 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T14:35:35.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:34 smithi105 ceph-mon[32555]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:35.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:34 smithi105 ceph-mon[32555]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-23T14:35:35.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:34 smithi105 ceph-mon[32555]: osdmap e7: 1 total, 0 up, 1 in 2024-04-23T14:35:35.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:34 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T14:35:35.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:34 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T14:35:36.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:35 smithi105 ceph-mon[32555]: purged_snaps scrub starts 2024-04-23T14:35:36.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:35 smithi105 ceph-mon[32555]: purged_snaps scrub ok 2024-04-23T14:35:36.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T14:35:36.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:35 smithi105 ceph-mon[32555]: from='osd.0 ' entity='osd.0' 2024-04-23T14:35:36.045 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:35 smithi028 ceph-mon[27810]: purged_snaps scrub starts 2024-04-23T14:35:36.045 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:35 smithi028 ceph-mon[27810]: purged_snaps scrub ok 2024-04-23T14:35:36.045 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:35 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T14:35:36.045 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:35 smithi028 ceph-mon[27810]: from='osd.0 ' entity='osd.0' 2024-04-23T14:35:36.173 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:35 smithi038 ceph-mon[32348]: purged_snaps scrub starts 2024-04-23T14:35:36.173 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:35 smithi038 ceph-mon[32348]: purged_snaps scrub ok 2024-04-23T14:35:36.173 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:35 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T14:35:36.174 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:35 smithi038 ceph-mon[32348]: from='osd.0 ' entity='osd.0' 2024-04-23T14:35:36.562 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-23T14:35:36.562 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-23T14:35:36.562 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-04-23T14:35:36.562 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-04-23T14:35:36.562 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0170621 s, 615 MB/s 2024-04-23T14:35:36.562 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-04-23T14:35:37.185 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:36 smithi038 ceph-mon[32348]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:37.185 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:36 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T14:35:37.185 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:36 smithi038 ceph-mon[32348]: osd.0 [v2:172.21.15.28:6802/2993445336,v1:172.21.15.28:6803/2993445336] boot 2024-04-23T14:35:37.185 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:36 smithi038 ceph-mon[32348]: osdmap e8: 1 total, 1 up, 1 in 2024-04-23T14:35:37.185 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:36 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T14:35:37.216 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi038:vg_nvme/lv_4 2024-04-23T14:35:37.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:36 smithi105 ceph-mon[32555]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:37.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:36 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T14:35:37.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:36 smithi105 ceph-mon[32555]: osd.0 [v2:172.21.15.28:6802/2993445336,v1:172.21.15.28:6803/2993445336] boot 2024-04-23T14:35:37.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:36 smithi105 ceph-mon[32555]: osdmap e8: 1 total, 1 up, 1 in 2024-04-23T14:35:37.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:36 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T14:35:37.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:36 smithi028 ceph-mon[27810]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T14:35:37.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:36 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T14:35:37.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:36 smithi028 ceph-mon[27810]: osd.0 [v2:172.21.15.28:6802/2993445336,v1:172.21.15.28:6803/2993445336] boot 2024-04-23T14:35:37.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:36 smithi028 ceph-mon[27810]: osdmap e8: 1 total, 1 up, 1 in 2024-04-23T14:35:37.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:36 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T14:35:37.461 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T14:35:38.257 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:37 smithi038 ceph-mon[32348]: osdmap e9: 1 total, 1 up, 1 in 2024-04-23T14:35:38.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:37 smithi105 ceph-mon[32555]: osdmap e9: 1 total, 1 up, 1 in 2024-04-23T14:35:38.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:37 smithi028 ceph-mon[27810]: osdmap e9: 1 total, 1 up, 1 in 2024-04-23T14:35:38.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.983+0000 7fc3018d8700 1 -- 172.21.15.38:0/2375086802 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2fc1035e0 msgr2=0x7fc2fc1059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:35:38.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.983+0000 7fc3018d8700 1 --2- 172.21.15.38:0/2375086802 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2fc1035e0 0x7fc2fc1059d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec00b0a0 tx=0x7fc2ec00b3b0 comp rx=0 tx=0).stop 2024-04-23T14:35:38.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.983+0000 7fc3018d8700 1 -- 172.21.15.38:0/2375086802 shutdown_connections 2024-04-23T14:35:38.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.983+0000 7fc3018d8700 1 --2- 172.21.15.38:0/2375086802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2fc108840 0x7fc2fc10acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:35:38.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.983+0000 7fc3018d8700 1 --2- 172.21.15.38:0/2375086802 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2fc1035e0 0x7fc2fc1059d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:35:38.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.983+0000 7fc3018d8700 1 --2- 172.21.15.38:0/2375086802 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2fc105f10 0x7fc2fc108300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:35:38.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.983+0000 7fc3018d8700 1 -- 172.21.15.38:0/2375086802 >> 172.21.15.38:0/2375086802 conn(0x7fc2fc0fcff0 msgr2=0x7fc2fc0ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:35:38.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.983+0000 7fc3018d8700 1 -- 172.21.15.38:0/2375086802 shutdown_connections 2024-04-23T14:35:38.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.983+0000 7fc3018d8700 1 -- 172.21.15.38:0/2375086802 wait complete. 2024-04-23T14:35:38.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.983+0000 7fc3018d8700 1 Processor -- start 2024-04-23T14:35:38.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.984+0000 7fc3018d8700 1 -- start start 2024-04-23T14:35:38.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.984+0000 7fc3018d8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2fc1035e0 0x7fc2fc198f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:35:38.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.984+0000 7fc3018d8700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2fc105f10 0x7fc2fc199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:35:38.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.984+0000 7fc3018d8700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2fc108840 0x7fc2fc1a0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:35:38.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.984+0000 7fc3018d8700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc2fc10e870 con 0x7fc2fc108840 2024-04-23T14:35:38.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.984+0000 7fc3018d8700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc2fc10e6f0 con 0x7fc2fc1035e0 2024-04-23T14:35:38.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.984+0000 7fc3018d8700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc2fc10e9f0 con 0x7fc2fc105f10 2024-04-23T14:35:38.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.984+0000 7fc2faffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2fc1035e0 0x7fc2fc198f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:35:38.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.984+0000 7fc2faffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2fc1035e0 0x7fc2fc198f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37842/0 (socket says 172.21.15.38:37842) 2024-04-23T14:35:38.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.984+0000 7fc2faffd700 1 -- 172.21.15.38:0/4149585361 learned_addr learned my addr 172.21.15.38:0/4149585361 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-23T14:35:38.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.984+0000 7fc2fa7fc700 1 --2- 172.21.15.38:0/4149585361 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2fc105f10 0x7fc2fc199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:35:38.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.984+0000 7fc2faffd700 1 -- 172.21.15.38:0/4149585361 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2fc105f10 msgr2=0x7fc2fc199450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:35:38.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.984+0000 7fc2faffd700 1 --2- 172.21.15.38:0/4149585361 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2fc105f10 0x7fc2fc199450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:35:38.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.984+0000 7fc2faffd700 1 -- 172.21.15.38:0/4149585361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2fc108840 msgr2=0x7fc2fc1a0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:35:38.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.984+0000 7fc2faffd700 1 --2- 172.21.15.38:0/4149585361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2fc108840 0x7fc2fc1a0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:35:38.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.984+0000 7fc2faffd700 1 -- 172.21.15.38:0/4149585361 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2fc1a0b20 con 0x7fc2fc1035e0 2024-04-23T14:35:38.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.984+0000 7fc2faffd700 1 --2- 172.21.15.38:0/4149585361 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2fc1035e0 0x7fc2fc198f10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec00b070 tx=0x7fc2ec007770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:35:38.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.985+0000 7fc3008d6700 1 -- 172.21.15.38:0/4149585361 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec021070 con 0x7fc2fc1035e0 2024-04-23T14:35:38.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.985+0000 7fc3008d6700 1 -- 172.21.15.38:0/4149585361 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2ec01d930 con 0x7fc2fc1035e0 2024-04-23T14:35:38.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.985+0000 7fc3018d8700 1 -- 172.21.15.38:0/4149585361 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2fc1a0db0 con 0x7fc2fc1035e0 2024-04-23T14:35:38.990 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.985+0000 7fc3008d6700 1 -- 172.21.15.38:0/4149585361 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec005710 con 0x7fc2fc1035e0 2024-04-23T14:35:38.990 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.985+0000 7fc3018d8700 1 -- 172.21.15.38:0/4149585361 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2fc1a1230 con 0x7fc2fc1035e0 2024-04-23T14:35:38.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.986+0000 7fc3018d8700 1 -- 172.21.15.38:0/4149585361 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2fc0697e0 con 0x7fc2fc1035e0 2024-04-23T14:35:38.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.987+0000 7fc3008d6700 1 -- 172.21.15.38:0/4149585361 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fc2ec024070 con 0x7fc2fc1035e0 2024-04-23T14:35:38.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.987+0000 7fc3008d6700 1 --2- 172.21.15.38:0/4149585361 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fc2e80a3bb0 0x7fc2e80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:35:38.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.988+0000 7fc3008d6700 1 -- 172.21.15.38:0/4149585361 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7fc2ec00f020 con 0x7fc2fc1035e0 2024-04-23T14:35:38.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.988+0000 7fc2fa7fc700 1 --2- 172.21.15.38:0/4149585361 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fc2e80a3bb0 0x7fc2e80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:35:38.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.989+0000 7fc2fa7fc700 1 --2- 172.21.15.38:0/4149585361 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fc2e80a3bb0 0x7fc2e80a6070 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4006fd0 tx=0x7fc2e4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:35:38.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:38.992+0000 7fc3008d6700 1 -- 172.21.15.38:0/4149585361 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc2ec02b020 con 0x7fc2fc1035e0 2024-04-23T14:35:39.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:39.209+0000 7fc3018d8700 1 -- 172.21.15.38:0/4149585361 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fc2fc062f30 con 0x7fc2e80a3bb0 2024-04-23T14:35:39.211 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:38 smithi038 ceph-mon[32348]: pgmap v57: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:35:39.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:38 smithi105 ceph-mon[32555]: pgmap v57: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:35:39.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:38 smithi028 ceph-mon[27810]: pgmap v57: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:35:40.103 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:40 smithi038 ceph-mon[32348]: Detected new or changed devices on smithi028 2024-04-23T14:35:40.104 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:40 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:40.104 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:40 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:40.104 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:40 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:35:40.104 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:40 smithi038 ceph-mon[32348]: Adjusting osd_memory_target on smithi028 to 17094M 2024-04-23T14:35:40.104 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:40 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:40.104 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:40 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:35:40.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:40 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:35:40.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:40 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:40.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:40 smithi038 ceph-mon[32348]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:35:40.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:40 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:35:40.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:40 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:35:40.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:40 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:35:40.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:40 smithi028 ceph-mon[27810]: Detected new or changed devices on smithi028 2024-04-23T14:35:40.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:40 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:40.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:40 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:40.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:40 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:35:40.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:40 smithi028 ceph-mon[27810]: Adjusting osd_memory_target on smithi028 to 17094M 2024-04-23T14:35:40.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:40 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:40.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:40 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:35:40.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:40 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:35:40.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:40 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:40.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:40 smithi028 ceph-mon[27810]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:35:40.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:40 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:35:40.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:40 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:35:40.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:40 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:35:40.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:40 smithi105 ceph-mon[32555]: Detected new or changed devices on smithi028 2024-04-23T14:35:40.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:40 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:40.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:40 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:40.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:40 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:35:40.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:40 smithi105 ceph-mon[32555]: Adjusting osd_memory_target on smithi028 to 17094M 2024-04-23T14:35:40.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:40 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:40.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:40 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:35:40.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:40 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:35:40.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:40 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:40.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:40 smithi105 ceph-mon[32555]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:35:40.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:40 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:35:40.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:40 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:35:40.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:40 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:35:41.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:41 smithi038 ceph-mon[32348]: pgmap v58: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:35:41.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:41 smithi028 ceph-mon[27810]: pgmap v58: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:35:41.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:41 smithi105 ceph-mon[32555]: pgmap v58: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:35:42.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:42 smithi028 ceph-mon[27810]: from='client.? 172.21.15.38:0/4263839084' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef527ec9-1bf3-4bb0-8df3-f5a015af557c"}]: dispatch 2024-04-23T14:35:42.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:42 smithi028 ceph-mon[27810]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef527ec9-1bf3-4bb0-8df3-f5a015af557c"}]: dispatch 2024-04-23T14:35:42.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:42 smithi028 ceph-mon[27810]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef527ec9-1bf3-4bb0-8df3-f5a015af557c"}]': finished 2024-04-23T14:35:42.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:42 smithi028 ceph-mon[27810]: osdmap e10: 2 total, 1 up, 2 in 2024-04-23T14:35:42.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:42 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T14:35:42.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:42 smithi105 ceph-mon[32555]: from='client.? 172.21.15.38:0/4263839084' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef527ec9-1bf3-4bb0-8df3-f5a015af557c"}]: dispatch 2024-04-23T14:35:42.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:42 smithi105 ceph-mon[32555]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef527ec9-1bf3-4bb0-8df3-f5a015af557c"}]: dispatch 2024-04-23T14:35:42.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:42 smithi105 ceph-mon[32555]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef527ec9-1bf3-4bb0-8df3-f5a015af557c"}]': finished 2024-04-23T14:35:42.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:42 smithi105 ceph-mon[32555]: osdmap e10: 2 total, 1 up, 2 in 2024-04-23T14:35:42.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:42 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T14:35:42.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:42 smithi038 ceph-mon[32348]: from='client.? 172.21.15.38:0/4263839084' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef527ec9-1bf3-4bb0-8df3-f5a015af557c"}]: dispatch 2024-04-23T14:35:42.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:42 smithi038 ceph-mon[32348]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef527ec9-1bf3-4bb0-8df3-f5a015af557c"}]: dispatch 2024-04-23T14:35:42.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:42 smithi038 ceph-mon[32348]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef527ec9-1bf3-4bb0-8df3-f5a015af557c"}]': finished 2024-04-23T14:35:42.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:42 smithi038 ceph-mon[32348]: osdmap e10: 2 total, 1 up, 2 in 2024-04-23T14:35:42.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:42 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T14:35:43.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:43 smithi105 ceph-mon[32555]: pgmap v59: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:35:43.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:43 smithi105 ceph-mon[32555]: from='client.? 172.21.15.38:0/848572801' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:35:43.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:43 smithi038 ceph-mon[32348]: pgmap v59: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:35:43.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:43 smithi038 ceph-mon[32348]: from='client.? 172.21.15.38:0/848572801' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:35:43.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:43 smithi028 ceph-mon[27810]: pgmap v59: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:35:43.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:43 smithi028 ceph-mon[27810]: from='client.? 172.21.15.38:0/848572801' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:35:45.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:45 smithi105 ceph-mon[32555]: pgmap v61: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:35:45.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:45 smithi038 ceph-mon[32348]: pgmap v61: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:35:45.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:45 smithi028 ceph-mon[27810]: pgmap v61: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:35:47.383 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:47 smithi038 ceph-mon[32348]: pgmap v62: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:35:47.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:47 smithi105 ceph-mon[32555]: pgmap v62: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:35:47.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:47 smithi028 ceph-mon[27810]: pgmap v62: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:35:49.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:49 smithi105 ceph-mon[32555]: pgmap v63: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:35:49.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:49 smithi038 ceph-mon[32348]: pgmap v63: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:35:49.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:49 smithi028 ceph-mon[27810]: pgmap v63: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:35:51.298 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:51 smithi038 ceph-mon[32348]: pgmap v64: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:35:51.298 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:51 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-23T14:35:51.298 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:51 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:35:51.298 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:51 smithi038 ceph-mon[32348]: Deploying daemon osd.1 on smithi038 2024-04-23T14:35:51.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:51 smithi105 ceph-mon[32555]: pgmap v64: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:35:51.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:51 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-23T14:35:51.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:51 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:35:51.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:51 smithi105 ceph-mon[32555]: Deploying daemon osd.1 on smithi038 2024-04-23T14:35:51.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:51 smithi028 ceph-mon[27810]: pgmap v64: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:35:51.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:51 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-23T14:35:51.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:51 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:35:51.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:51 smithi028 ceph-mon[27810]: Deploying daemon osd.1 on smithi038 2024-04-23T14:35:53.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:53 smithi038 ceph-mon[32348]: pgmap v65: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:35:53.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:53 smithi105 ceph-mon[32555]: pgmap v65: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:35:53.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:53 smithi028 ceph-mon[27810]: pgmap v65: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:35:55.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:55 smithi105 ceph-mon[32555]: pgmap v66: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:35:55.585 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:55 smithi038 ceph-mon[32348]: pgmap v66: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:35:55.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:55 smithi028 ceph-mon[27810]: pgmap v66: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:35:56.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:56 smithi038 ceph-mon[32348]: pgmap v67: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:35:56.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:56 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:56.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:56 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:56.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:56 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:35:57.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:56 smithi105 ceph-mon[32555]: pgmap v67: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:35:57.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:56 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:57.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:56 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:57.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:56 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:35:57.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:56 smithi028 ceph-mon[27810]: pgmap v67: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:35:57.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:56 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:57.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:56 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:57.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:56 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:35:59.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:59.401+0000 7fc3008d6700 1 -- 172.21.15.38:0/4149585361 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc2fc062f30 con 0x7fc2e80a3bb0 2024-04-23T14:35:59.403 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 1 on host 'smithi038' 2024-04-23T14:35:59.403 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:59 smithi038 ceph-mon[32348]: pgmap v68: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:35:59.403 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:59 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:59.403 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:59 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:59.403 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:59 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:35:59.403 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:59 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:35:59.403 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:35:59 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:59.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:59.404+0000 7fc3018d8700 1 -- 172.21.15.38:0/4149585361 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fc2e80a3bb0 msgr2=0x7fc2e80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:35:59.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:59.404+0000 7fc3018d8700 1 --2- 172.21.15.38:0/4149585361 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fc2e80a3bb0 0x7fc2e80a6070 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4006fd0 tx=0x7fc2e4008040 comp rx=0 tx=0).stop 2024-04-23T14:35:59.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:59.404+0000 7fc3018d8700 1 -- 172.21.15.38:0/4149585361 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2fc1035e0 msgr2=0x7fc2fc198f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:35:59.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:59.404+0000 7fc3018d8700 1 --2- 172.21.15.38:0/4149585361 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2fc1035e0 0x7fc2fc198f10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec00b070 tx=0x7fc2ec007770 comp rx=0 tx=0).stop 2024-04-23T14:35:59.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:59.404+0000 7fc3018d8700 1 -- 172.21.15.38:0/4149585361 shutdown_connections 2024-04-23T14:35:59.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:59.404+0000 7fc3018d8700 1 --2- 172.21.15.38:0/4149585361 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fc2e80a3bb0 0x7fc2e80a6070 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:35:59.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:59.404+0000 7fc3018d8700 1 --2- 172.21.15.38:0/4149585361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2fc108840 0x7fc2fc1a0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:35:59.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:59.404+0000 7fc3018d8700 1 --2- 172.21.15.38:0/4149585361 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2fc1035e0 0x7fc2fc198f10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:35:59.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:59.404+0000 7fc3018d8700 1 --2- 172.21.15.38:0/4149585361 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2fc105f10 0x7fc2fc199450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:35:59.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:59.404+0000 7fc3018d8700 1 -- 172.21.15.38:0/4149585361 >> 172.21.15.38:0/4149585361 conn(0x7fc2fc0fcff0 msgr2=0x7fc2fc103f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:35:59.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:59.404+0000 7fc3018d8700 1 -- 172.21.15.38:0/4149585361 shutdown_connections 2024-04-23T14:35:59.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:35:59.404+0000 7fc3018d8700 1 -- 172.21.15.38:0/4149585361 wait complete. 2024-04-23T14:35:59.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:59 smithi028 ceph-mon[27810]: pgmap v68: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:35:59.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:59 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:59.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:59 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:59.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:59 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:35:59.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:59 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:35:59.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:35:59 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:59.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:59 smithi105 ceph-mon[32555]: pgmap v68: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:35:59.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:59 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:59.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:59 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:35:59.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:59 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:35:59.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:59 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:35:59.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:35:59 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:00.071 DEBUG:teuthology.orchestra.run.smithi038:osd.1> sudo journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.1.service 2024-04-23T14:36:00.074 INFO:tasks.cephadm:Deploying osd.2 on smithi105 with /dev/vg_nvme/lv_4... 2024-04-23T14:36:00.074 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-23T14:36:00.318 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T14:36:00.439 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:00 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:36:00.439 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:00 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:00.440 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:00 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:00.440 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:00 smithi105 ceph-mon[32555]: from='osd.1 [v2:172.21.15.38:6800/4024093383,v1:172.21.15.38:6801/4024093383]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T14:36:00.440 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:00 smithi105 ceph-mon[32555]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T14:36:00.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:00 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:36:00.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:00 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:00.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:00 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:00.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:00 smithi038 ceph-mon[32348]: from='osd.1 [v2:172.21.15.38:6800/4024093383,v1:172.21.15.38:6801/4024093383]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T14:36:00.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:00 smithi038 ceph-mon[32348]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T14:36:00.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:00 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:36:00.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:00 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:00.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:00 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:00.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:00 smithi028 ceph-mon[27810]: from='osd.1 [v2:172.21.15.38:6800/4024093383,v1:172.21.15.38:6801/4024093383]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T14:36:00.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:00 smithi028 ceph-mon[27810]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T14:36:01.586 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:01 smithi105 ceph-mon[32555]: pgmap v69: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:36:01.586 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:01 smithi105 ceph-mon[32555]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-23T14:36:01.586 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:01 smithi105 ceph-mon[32555]: from='osd.1 [v2:172.21.15.38:6800/4024093383,v1:172.21.15.38:6801/4024093383]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-23T14:36:01.586 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:01 smithi105 ceph-mon[32555]: osdmap e11: 2 total, 1 up, 2 in 2024-04-23T14:36:01.586 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:01 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T14:36:01.586 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:01 smithi105 ceph-mon[32555]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-23T14:36:01.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:01 smithi038 ceph-mon[32348]: pgmap v69: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:36:01.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:01 smithi038 ceph-mon[32348]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-23T14:36:01.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:01 smithi038 ceph-mon[32348]: from='osd.1 [v2:172.21.15.38:6800/4024093383,v1:172.21.15.38:6801/4024093383]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-23T14:36:01.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:01 smithi038 ceph-mon[32348]: osdmap e11: 2 total, 1 up, 2 in 2024-04-23T14:36:01.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:01 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T14:36:01.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:01 smithi038 ceph-mon[32348]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-23T14:36:01.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:01 smithi028 ceph-mon[27810]: pgmap v69: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:36:01.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:01 smithi028 ceph-mon[27810]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-23T14:36:01.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:01 smithi028 ceph-mon[27810]: from='osd.1 [v2:172.21.15.38:6800/4024093383,v1:172.21.15.38:6801/4024093383]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-23T14:36:01.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:01 smithi028 ceph-mon[27810]: osdmap e11: 2 total, 1 up, 2 in 2024-04-23T14:36:01.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:01 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T14:36:01.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:01 smithi028 ceph-mon[27810]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-23T14:36:02.137 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-23T14:36:02.137 INFO:teuthology.orchestra.run.smithi105.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-23T14:36:02.137 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10+0 records in 2024-04-23T14:36:02.138 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2024-04-23T14:36:02.138 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114803 s, 913 MB/s 2024-04-23T14:36:02.138 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping successful for: 2024-04-23T14:36:02.606 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:02 smithi105 ceph-mon[32555]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-23T14:36:02.606 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:02 smithi105 ceph-mon[32555]: osdmap e12: 2 total, 1 up, 2 in 2024-04-23T14:36:02.607 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:02 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T14:36:02.607 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:02 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T14:36:02.607 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:02 smithi105 ceph-mon[32555]: from='osd.1 ' entity='osd.1' 2024-04-23T14:36:02.609 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:02 smithi038 ceph-mon[32348]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-23T14:36:02.609 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:02 smithi038 ceph-mon[32348]: osdmap e12: 2 total, 1 up, 2 in 2024-04-23T14:36:02.609 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:02 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T14:36:02.609 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:02 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T14:36:02.610 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:02 smithi038 ceph-mon[32348]: from='osd.1 ' entity='osd.1' 2024-04-23T14:36:02.610 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 23 14:36:02 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-1[36932]: 2024-04-23T14:36:02.363+0000 7fb79e54d700 -1 osd.1 0 waiting for initial osdmap 2024-04-23T14:36:02.610 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 23 14:36:02 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-1[36932]: 2024-04-23T14:36:02.383+0000 7fb796338700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:36:02.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:02 smithi028 ceph-mon[27810]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-23T14:36:02.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:02 smithi028 ceph-mon[27810]: osdmap e12: 2 total, 1 up, 2 in 2024-04-23T14:36:02.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:02 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T14:36:02.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:02 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T14:36:02.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:02 smithi028 ceph-mon[27810]: from='osd.1 ' entity='osd.1' 2024-04-23T14:36:02.900 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi105:vg_nvme/lv_4 2024-04-23T14:36:03.146 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T14:36:03.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:03 smithi105 ceph-mon[32555]: purged_snaps scrub starts 2024-04-23T14:36:03.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:03 smithi105 ceph-mon[32555]: purged_snaps scrub ok 2024-04-23T14:36:03.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:03 smithi105 ceph-mon[32555]: pgmap v72: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:36:03.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:03 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T14:36:03.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:03 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:03.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:03 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:03.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:03 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:36:03.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:03 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:03.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:03 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:36:03.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:03 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:36:03.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:03 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:03.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:03 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T14:36:03.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:03 smithi038 ceph-mon[32348]: purged_snaps scrub starts 2024-04-23T14:36:03.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:03 smithi038 ceph-mon[32348]: purged_snaps scrub ok 2024-04-23T14:36:03.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:03 smithi038 ceph-mon[32348]: pgmap v72: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:36:03.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:03 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T14:36:03.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:03 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:03.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:03 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:03.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:03 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:36:03.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:03 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:03.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:03 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:36:03.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:03 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:36:03.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:03 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:03.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:03 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T14:36:03.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:03 smithi028 ceph-mon[27810]: purged_snaps scrub starts 2024-04-23T14:36:03.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:03 smithi028 ceph-mon[27810]: purged_snaps scrub ok 2024-04-23T14:36:03.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:03 smithi028 ceph-mon[27810]: pgmap v72: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:36:03.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:03 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T14:36:03.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:03 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:03.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:03 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:03.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:03 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:36:03.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:03 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:03.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:03 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:36:03.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:03 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:36:03.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:03 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:03.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:03 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T14:36:04.633 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:04 smithi105 ceph-mon[32555]: Detected new or changed devices on smithi038 2024-04-23T14:36:04.633 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:04 smithi105 ceph-mon[32555]: Adjusting osd_memory_target on smithi038 to 17093M 2024-04-23T14:36:04.633 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:04 smithi105 ceph-mon[32555]: osd.1 [v2:172.21.15.38:6800/4024093383,v1:172.21.15.38:6801/4024093383] boot 2024-04-23T14:36:04.633 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:04 smithi105 ceph-mon[32555]: osdmap e13: 2 total, 2 up, 2 in 2024-04-23T14:36:04.633 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:04 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T14:36:04.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.653+0000 7f35e4af8700 1 -- 172.21.15.105:0/518181674 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35e01035f0 msgr2=0x7f35e0109ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:04.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.653+0000 7f35e4af8700 1 --2- 172.21.15.105:0/518181674 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35e01035f0 0x7f35e0109ea0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f35d800b0a0 tx=0x7f35d800b3b0 comp rx=0 tx=0).stop 2024-04-23T14:36:04.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.653+0000 7f35e4af8700 1 -- 172.21.15.105:0/518181674 shutdown_connections 2024-04-23T14:36:04.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.653+0000 7f35e4af8700 1 --2- 172.21.15.105:0/518181674 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35e01035f0 0x7f35e0109ea0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:04.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.653+0000 7f35e4af8700 1 --2- 172.21.15.105:0/518181674 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35e0101a50 0x7f35e0101e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:04.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.653+0000 7f35e4af8700 1 --2- 172.21.15.105:0/518181674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35e0102c50 0x7f35e01030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:04.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.653+0000 7f35e4af8700 1 -- 172.21.15.105:0/518181674 >> 172.21.15.105:0/518181674 conn(0x7f35e00fcff0 msgr2=0x7f35e00ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:36:04.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.653+0000 7f35e4af8700 1 -- 172.21.15.105:0/518181674 shutdown_connections 2024-04-23T14:36:04.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.653+0000 7f35e4af8700 1 -- 172.21.15.105:0/518181674 wait complete. 2024-04-23T14:36:04.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.654+0000 7f35e4af8700 1 Processor -- start 2024-04-23T14:36:04.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.654+0000 7f35e4af8700 1 -- start start 2024-04-23T14:36:04.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.654+0000 7f35e4af8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35e0101a50 0x7f35e006b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:04.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.654+0000 7f35e4af8700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35e0102c50 0x7f35e019dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:04.657 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.654+0000 7f35e4af8700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35e01035f0 0x7f35e01a0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:04.657 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.654+0000 7f35e4af8700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f35e010d730 con 0x7f35e01035f0 2024-04-23T14:36:04.657 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.654+0000 7f35e4af8700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f35e010d5b0 con 0x7f35e0101a50 2024-04-23T14:36:04.657 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.654+0000 7f35e4af8700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f35e010d8b0 con 0x7f35e0102c50 2024-04-23T14:36:04.657 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.654+0000 7f35ddd9b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35e0102c50 0x7f35e019dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:04.657 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.654+0000 7f35ddd9b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35e0102c50 0x7f35e019dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:44318/0 (socket says 172.21.15.105:44318) 2024-04-23T14:36:04.657 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.654+0000 7f35ddd9b700 1 -- 172.21.15.105:0/439589347 learned_addr learned my addr 172.21.15.105:0/439589347 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-23T14:36:04.657 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.654+0000 7f35de59c700 1 --2- 172.21.15.105:0/439589347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35e0101a50 0x7f35e006b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:04.657 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.654+0000 7f35ddd9b700 1 -- 172.21.15.105:0/439589347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35e0101a50 msgr2=0x7f35e006b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:04.657 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.654+0000 7f35ddd9b700 1 --2- 172.21.15.105:0/439589347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35e0101a50 0x7f35e006b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:04.657 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.654+0000 7f35ddd9b700 1 -- 172.21.15.105:0/439589347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35e01035f0 msgr2=0x7f35e01a0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:04.657 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.654+0000 7f35ddd9b700 1 --2- 172.21.15.105:0/439589347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35e01035f0 0x7f35e01a0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:04.657 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.654+0000 7f35ddd9b700 1 -- 172.21.15.105:0/439589347 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35e01a0b50 con 0x7f35e0102c50 2024-04-23T14:36:04.658 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.655+0000 7f35ddd9b700 1 --2- 172.21.15.105:0/439589347 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35e0102c50 0x7f35e019dc40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f35d400b530 tx=0x7f35d400b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:04.658 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.655+0000 7f35cf7fe700 1 -- 172.21.15.105:0/439589347 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d4015020 con 0x7f35e0102c50 2024-04-23T14:36:04.658 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.655+0000 7f35e4af8700 1 -- 172.21.15.105:0/439589347 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35e01a0e40 con 0x7f35e0102c50 2024-04-23T14:36:04.658 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.655+0000 7f35cf7fe700 1 -- 172.21.15.105:0/439589347 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35d4011d10 con 0x7f35e0102c50 2024-04-23T14:36:04.658 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.655+0000 7f35e4af8700 1 -- 172.21.15.105:0/439589347 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35e01a13c0 con 0x7f35e0102c50 2024-04-23T14:36:04.658 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.655+0000 7f35cf7fe700 1 -- 172.21.15.105:0/439589347 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d400f2e0 con 0x7f35e0102c50 2024-04-23T14:36:04.659 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.656+0000 7f35e4af8700 1 -- 172.21.15.105:0/439589347 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35e00697e0 con 0x7f35e0102c50 2024-04-23T14:36:04.659 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.657+0000 7f35cf7fe700 1 -- 172.21.15.105:0/439589347 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f35d4018020 con 0x7f35e0102c50 2024-04-23T14:36:04.659 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.657+0000 7f35cf7fe700 1 --2- 172.21.15.105:0/439589347 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f35c80a3bb0 0x7f35c80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:04.659 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.658+0000 7f35cf7fe700 1 -- 172.21.15.105:0/439589347 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f35d40ce3a0 con 0x7f35e0102c50 2024-04-23T14:36:04.659 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.658+0000 7f35de59c700 1 --2- 172.21.15.105:0/439589347 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f35c80a3bb0 0x7f35c80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:04.659 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.659+0000 7f35de59c700 1 --2- 172.21.15.105:0/439589347 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f35c80a3bb0 0x7f35c80a6070 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f35e0102ab0 tx=0x7f35d0009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:04.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.662+0000 7f35cf7fe700 1 -- 172.21.15.105:0/439589347 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f35d401f020 con 0x7f35e0102c50 2024-04-23T14:36:04.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:04 smithi038 ceph-mon[32348]: Detected new or changed devices on smithi038 2024-04-23T14:36:04.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:04 smithi038 ceph-mon[32348]: Adjusting osd_memory_target on smithi038 to 17093M 2024-04-23T14:36:04.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:04 smithi038 ceph-mon[32348]: osd.1 [v2:172.21.15.38:6800/4024093383,v1:172.21.15.38:6801/4024093383] boot 2024-04-23T14:36:04.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:04 smithi038 ceph-mon[32348]: osdmap e13: 2 total, 2 up, 2 in 2024-04-23T14:36:04.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:04 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T14:36:04.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:04 smithi028 ceph-mon[27810]: Detected new or changed devices on smithi038 2024-04-23T14:36:04.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:04 smithi028 ceph-mon[27810]: Adjusting osd_memory_target on smithi038 to 17093M 2024-04-23T14:36:04.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:04 smithi028 ceph-mon[27810]: osd.1 [v2:172.21.15.38:6800/4024093383,v1:172.21.15.38:6801/4024093383] boot 2024-04-23T14:36:04.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:04 smithi028 ceph-mon[27810]: osdmap e13: 2 total, 2 up, 2 in 2024-04-23T14:36:04.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:04 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T14:36:04.874 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:04.873+0000 7f35e4af8700 1 -- 172.21.15.105:0/439589347 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f35e0062f30 con 0x7f35c80a3bb0 2024-04-23T14:36:05.702 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:05 smithi105 ceph-mon[32555]: pgmap v74: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:36:05.702 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:05 smithi105 ceph-mon[32555]: osdmap e14: 2 total, 2 up, 2 in 2024-04-23T14:36:05.702 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:05 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:36:05.702 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:05 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:36:05.702 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:05 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:36:05.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:05 smithi038 ceph-mon[32348]: pgmap v74: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:36:05.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:05 smithi038 ceph-mon[32348]: osdmap e14: 2 total, 2 up, 2 in 2024-04-23T14:36:05.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:05 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:36:05.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:05 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:36:05.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:05 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:36:05.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:05 smithi028 ceph-mon[27810]: pgmap v74: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-23T14:36:05.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:05 smithi028 ceph-mon[27810]: osdmap e14: 2 total, 2 up, 2 in 2024-04-23T14:36:05.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:05 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:36:05.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:05 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:36:05.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:05 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:36:06.694 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:06 smithi105 ceph-mon[32555]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:36:06.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:06 smithi038 ceph-mon[32348]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:36:06.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:06 smithi028 ceph-mon[27810]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:36:07.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:07 smithi105 ceph-mon[32555]: pgmap v76: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:07.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:07 smithi028 ceph-mon[27810]: pgmap v76: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:07.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:07 smithi038 ceph-mon[32348]: pgmap v76: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:08.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:08 smithi105 ceph-mon[32555]: from='client.? 172.21.15.105:0/2800347176' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b33b23d9-adce-4c11-b304-2e117bdf6bfe"}]: dispatch 2024-04-23T14:36:08.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:08 smithi105 ceph-mon[32555]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b33b23d9-adce-4c11-b304-2e117bdf6bfe"}]: dispatch 2024-04-23T14:36:08.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:08 smithi105 ceph-mon[32555]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b33b23d9-adce-4c11-b304-2e117bdf6bfe"}]': finished 2024-04-23T14:36:08.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:08 smithi105 ceph-mon[32555]: osdmap e15: 3 total, 2 up, 3 in 2024-04-23T14:36:08.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:08 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T14:36:08.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:08 smithi105 ceph-mon[32555]: from='client.? 172.21.15.105:0/2171056952' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:36:08.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:08 smithi038 ceph-mon[32348]: from='client.? 172.21.15.105:0/2800347176' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b33b23d9-adce-4c11-b304-2e117bdf6bfe"}]: dispatch 2024-04-23T14:36:08.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:08 smithi038 ceph-mon[32348]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b33b23d9-adce-4c11-b304-2e117bdf6bfe"}]: dispatch 2024-04-23T14:36:08.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:08 smithi038 ceph-mon[32348]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b33b23d9-adce-4c11-b304-2e117bdf6bfe"}]': finished 2024-04-23T14:36:08.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:08 smithi038 ceph-mon[32348]: osdmap e15: 3 total, 2 up, 3 in 2024-04-23T14:36:08.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:08 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T14:36:08.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:08 smithi038 ceph-mon[32348]: from='client.? 172.21.15.105:0/2171056952' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:36:08.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:08 smithi028 ceph-mon[27810]: from='client.? 172.21.15.105:0/2800347176' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b33b23d9-adce-4c11-b304-2e117bdf6bfe"}]: dispatch 2024-04-23T14:36:08.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:08 smithi028 ceph-mon[27810]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b33b23d9-adce-4c11-b304-2e117bdf6bfe"}]: dispatch 2024-04-23T14:36:08.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:08 smithi028 ceph-mon[27810]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b33b23d9-adce-4c11-b304-2e117bdf6bfe"}]': finished 2024-04-23T14:36:08.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:08 smithi028 ceph-mon[27810]: osdmap e15: 3 total, 2 up, 3 in 2024-04-23T14:36:08.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:08 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T14:36:08.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:08 smithi028 ceph-mon[27810]: from='client.? 172.21.15.105:0/2171056952' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:36:09.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:09 smithi105 ceph-mon[32555]: pgmap v78: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:09.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:09 smithi038 ceph-mon[32348]: pgmap v78: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:09.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:09 smithi028 ceph-mon[27810]: pgmap v78: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:11.678 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:11 smithi028 ceph-mon[27810]: pgmap v79: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:11.721 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:11 smithi105 ceph-mon[32555]: pgmap v79: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:11.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:11 smithi038 ceph-mon[32348]: pgmap v79: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:13.598 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:13 smithi105 ceph-mon[32555]: pgmap v80: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:13.686 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:13 smithi038 ceph-mon[32348]: pgmap v80: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:13.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:13 smithi028 ceph-mon[27810]: pgmap v80: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:15.678 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:15 smithi105 ceph-mon[32555]: pgmap v81: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:15.678 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:15 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-23T14:36:15.679 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:15 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:36:15.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:15 smithi038 ceph-mon[32348]: pgmap v81: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:15.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:15 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-23T14:36:15.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:15 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:36:15.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:15 smithi028 ceph-mon[27810]: pgmap v81: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:15.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:15 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-23T14:36:15.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:15 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:36:16.767 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:16 smithi105 ceph-mon[32555]: Deploying daemon osd.2 on smithi105 2024-04-23T14:36:16.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:16 smithi038 ceph-mon[32348]: Deploying daemon osd.2 on smithi105 2024-04-23T14:36:16.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:16 smithi028 ceph-mon[27810]: Deploying daemon osd.2 on smithi105 2024-04-23T14:36:17.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:17 smithi105 ceph-mon[32555]: pgmap v82: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:17.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:17 smithi038 ceph-mon[32348]: pgmap v82: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:17.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:17 smithi028 ceph-mon[27810]: pgmap v82: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:19.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:19 smithi105 ceph-mon[32555]: pgmap v83: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:19.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:19 smithi038 ceph-mon[32348]: pgmap v83: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:19.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:19 smithi028 ceph-mon[27810]: pgmap v83: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:21.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:21 smithi105 ceph-mon[32555]: pgmap v84: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:21.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:21 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:21.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:21 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:21.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:21 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:36:21.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:21 smithi038 ceph-mon[32348]: pgmap v84: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:21.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:21 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:21.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:21 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:21.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:21 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:36:21.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:21 smithi028 ceph-mon[27810]: pgmap v84: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:21.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:21 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:21.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:21 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:21.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:21 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:36:23.211 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:23 smithi105 ceph-mon[32555]: pgmap v85: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:23.310 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:23.309+0000 7f35cf7fe700 1 -- 172.21.15.105:0/439589347 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f35e0062f30 con 0x7f35c80a3bb0 2024-04-23T14:36:23.311 INFO:teuthology.orchestra.run.smithi105.stdout:Created osd(s) 2 on host 'smithi105' 2024-04-23T14:36:23.312 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:23.311+0000 7f35e4af8700 1 -- 172.21.15.105:0/439589347 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f35c80a3bb0 msgr2=0x7f35c80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:23.312 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:23.311+0000 7f35e4af8700 1 --2- 172.21.15.105:0/439589347 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f35c80a3bb0 0x7f35c80a6070 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f35e0102ab0 tx=0x7f35d0009190 comp rx=0 tx=0).stop 2024-04-23T14:36:23.312 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:23.311+0000 7f35e4af8700 1 -- 172.21.15.105:0/439589347 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35e0102c50 msgr2=0x7f35e019dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:23.312 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:23.311+0000 7f35e4af8700 1 --2- 172.21.15.105:0/439589347 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35e0102c50 0x7f35e019dc40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f35d400b530 tx=0x7f35d400b840 comp rx=0 tx=0).stop 2024-04-23T14:36:23.312 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:23.311+0000 7f35e4af8700 1 -- 172.21.15.105:0/439589347 shutdown_connections 2024-04-23T14:36:23.312 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:23.311+0000 7f35e4af8700 1 --2- 172.21.15.105:0/439589347 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f35c80a3bb0 0x7f35c80a6070 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:23.313 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:23.311+0000 7f35e4af8700 1 --2- 172.21.15.105:0/439589347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35e01035f0 0x7f35e01a0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:23.313 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:23.311+0000 7f35e4af8700 1 --2- 172.21.15.105:0/439589347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35e0101a50 0x7f35e006b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:23.313 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:23.311+0000 7f35e4af8700 1 --2- 172.21.15.105:0/439589347 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35e0102c50 0x7f35e019dc40 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:23.313 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:23.311+0000 7f35e4af8700 1 -- 172.21.15.105:0/439589347 >> 172.21.15.105:0/439589347 conn(0x7f35e00fcff0 msgr2=0x7f35e0105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:36:23.313 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:23.311+0000 7f35e4af8700 1 -- 172.21.15.105:0/439589347 shutdown_connections 2024-04-23T14:36:23.313 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:36:23.311+0000 7f35e4af8700 1 -- 172.21.15.105:0/439589347 wait complete. 2024-04-23T14:36:23.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:23 smithi038 ceph-mon[32348]: pgmap v85: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:23.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:23 smithi028 ceph-mon[27810]: pgmap v85: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:23.955 DEBUG:teuthology.orchestra.run.smithi105:osd.2> sudo journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.2.service 2024-04-23T14:36:23.957 INFO:tasks.cephadm:Deploying osd.3 on smithi028 with /dev/vg_nvme/lv_3... 2024-04-23T14:36:23.957 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-23T14:36:24.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:24 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:24.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:24 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:24.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:24 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:24.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:24 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:24.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:24 smithi028 ceph-mon[27810]: from='osd.2 [v2:172.21.15.105:6800/570739029,v1:172.21.15.105:6801/570739029]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T14:36:24.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:24 smithi028 ceph-mon[27810]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T14:36:24.429 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:36:24.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:24 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:24.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:24 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:24.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:24 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:24.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:24 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:24.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:24 smithi105 ceph-mon[32555]: from='osd.2 [v2:172.21.15.105:6800/570739029,v1:172.21.15.105:6801/570739029]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T14:36:24.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:24 smithi105 ceph-mon[32555]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T14:36:24.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:24 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:24.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:24 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:24.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:24 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:24.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:24 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:24.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:24 smithi038 ceph-mon[32348]: from='osd.2 [v2:172.21.15.105:6800/570739029,v1:172.21.15.105:6801/570739029]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T14:36:24.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:24 smithi038 ceph-mon[32348]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T14:36:25.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:25 smithi105 ceph-mon[32555]: pgmap v86: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:25.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:25 smithi105 ceph-mon[32555]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-23T14:36:25.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:25 smithi105 ceph-mon[32555]: from='osd.2 [v2:172.21.15.105:6800/570739029,v1:172.21.15.105:6801/570739029]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-23T14:36:25.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:25 smithi105 ceph-mon[32555]: osdmap e16: 3 total, 2 up, 3 in 2024-04-23T14:36:25.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:25 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T14:36:25.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:25 smithi105 ceph-mon[32555]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-23T14:36:25.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:25 smithi038 ceph-mon[32348]: pgmap v86: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:25.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:25 smithi038 ceph-mon[32348]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-23T14:36:25.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:25 smithi038 ceph-mon[32348]: from='osd.2 [v2:172.21.15.105:6800/570739029,v1:172.21.15.105:6801/570739029]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-23T14:36:25.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:25 smithi038 ceph-mon[32348]: osdmap e16: 3 total, 2 up, 3 in 2024-04-23T14:36:25.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:25 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T14:36:25.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:25 smithi038 ceph-mon[32348]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-23T14:36:25.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:25 smithi028 ceph-mon[27810]: pgmap v86: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:25.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:25 smithi028 ceph-mon[27810]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-23T14:36:25.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:25 smithi028 ceph-mon[27810]: from='osd.2 [v2:172.21.15.105:6800/570739029,v1:172.21.15.105:6801/570739029]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-23T14:36:25.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:25 smithi028 ceph-mon[27810]: osdmap e16: 3 total, 2 up, 3 in 2024-04-23T14:36:25.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:25 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T14:36:25.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:25 smithi028 ceph-mon[27810]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-23T14:36:26.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:26 smithi028 ceph-mon[27810]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-04-23T14:36:26.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:26 smithi028 ceph-mon[27810]: osdmap e17: 3 total, 2 up, 3 in 2024-04-23T14:36:26.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:26 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T14:36:26.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:26 smithi028 ceph-mon[27810]: from='osd.2 ' entity='osd.2' 2024-04-23T14:36:26.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:26 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T14:36:26.445 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:26 smithi105 ceph-mon[32555]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-04-23T14:36:26.445 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:26 smithi105 ceph-mon[32555]: osdmap e17: 3 total, 2 up, 3 in 2024-04-23T14:36:26.445 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:26 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T14:36:26.445 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:26 smithi105 ceph-mon[32555]: from='osd.2 ' entity='osd.2' 2024-04-23T14:36:26.446 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:26 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T14:36:26.446 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 23 14:36:26 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-2[37737]: 2024-04-23T14:36:26.221+0000 7f9110bfb700 -1 osd.2 0 waiting for initial osdmap 2024-04-23T14:36:26.446 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 23 14:36:26 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-2[37737]: 2024-04-23T14:36:26.341+0000 7f91089e6700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:36:26.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:26 smithi038 ceph-mon[32348]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-04-23T14:36:26.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:26 smithi038 ceph-mon[32348]: osdmap e17: 3 total, 2 up, 3 in 2024-04-23T14:36:26.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:26 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T14:36:26.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:26 smithi038 ceph-mon[32348]: from='osd.2 ' entity='osd.2' 2024-04-23T14:36:26.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:26 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T14:36:27.281 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-23T14:36:27.281 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-23T14:36:27.281 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-04-23T14:36:27.281 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-04-23T14:36:27.281 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0313743 s, 334 MB/s 2024-04-23T14:36:27.281 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-04-23T14:36:27.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:27 smithi038 ceph-mon[32348]: purged_snaps scrub starts 2024-04-23T14:36:27.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:27 smithi038 ceph-mon[32348]: purged_snaps scrub ok 2024-04-23T14:36:27.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:27 smithi038 ceph-mon[32348]: pgmap v89: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:27.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:27 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:27.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:27 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:27.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:27 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:36:27.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:27 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:27.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:27 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:36:27.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:27 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:36:27.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:27 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:27.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:27 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:36:27.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:27 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:36:27.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:27 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:36:27.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:27 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:27.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:27 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T14:36:27.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:27 smithi028 ceph-mon[27810]: purged_snaps scrub starts 2024-04-23T14:36:27.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:27 smithi028 ceph-mon[27810]: purged_snaps scrub ok 2024-04-23T14:36:27.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:27 smithi028 ceph-mon[27810]: pgmap v89: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:27.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:27 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:27.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:27 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:27.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:27 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:36:27.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:27 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:27.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:27 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:36:27.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:27 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:36:27.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:27 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:27.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:27 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:36:27.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:27 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:36:27.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:27 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:36:27.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:27 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:27.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:27 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T14:36:27.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:27 smithi105 ceph-mon[32555]: purged_snaps scrub starts 2024-04-23T14:36:27.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:27 smithi105 ceph-mon[32555]: purged_snaps scrub ok 2024-04-23T14:36:27.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:27 smithi105 ceph-mon[32555]: pgmap v89: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:27.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:27 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:27.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:27 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:27.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:27 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:36:27.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:27 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:27.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:27 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:36:27.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:27 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:36:27.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:27 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:27.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:27 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:36:27.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:27 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:36:27.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:27 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:36:27.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:27 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:27.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:27 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T14:36:28.112 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_3 2024-04-23T14:36:28.585 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:36:28.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:28 smithi038 ceph-mon[32348]: Detected new or changed devices on smithi105 2024-04-23T14:36:28.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:28 smithi038 ceph-mon[32348]: Adjusting osd_memory_target on smithi105 to 17088M 2024-04-23T14:36:28.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:28 smithi038 ceph-mon[32348]: osd.2 [v2:172.21.15.105:6800/570739029,v1:172.21.15.105:6801/570739029] boot 2024-04-23T14:36:28.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:28 smithi038 ceph-mon[32348]: osdmap e18: 3 total, 3 up, 3 in 2024-04-23T14:36:28.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:28 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T14:36:28.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:28 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-23T14:36:28.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:28 smithi028 ceph-mon[27810]: Detected new or changed devices on smithi105 2024-04-23T14:36:28.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:28 smithi028 ceph-mon[27810]: Adjusting osd_memory_target on smithi105 to 17088M 2024-04-23T14:36:28.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:28 smithi028 ceph-mon[27810]: osd.2 [v2:172.21.15.105:6800/570739029,v1:172.21.15.105:6801/570739029] boot 2024-04-23T14:36:28.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:28 smithi028 ceph-mon[27810]: osdmap e18: 3 total, 3 up, 3 in 2024-04-23T14:36:28.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T14:36:28.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-23T14:36:28.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:28 smithi105 ceph-mon[32555]: Detected new or changed devices on smithi105 2024-04-23T14:36:28.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:28 smithi105 ceph-mon[32555]: Adjusting osd_memory_target on smithi105 to 17088M 2024-04-23T14:36:28.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:28 smithi105 ceph-mon[32555]: osd.2 [v2:172.21.15.105:6800/570739029,v1:172.21.15.105:6801/570739029] boot 2024-04-23T14:36:28.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:28 smithi105 ceph-mon[32555]: osdmap e18: 3 total, 3 up, 3 in 2024-04-23T14:36:28.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:28 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T14:36:28.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:28 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-23T14:36:29.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:29 smithi038 ceph-mon[32348]: pgmap v91: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:29.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:29 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-23T14:36:29.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:29 smithi038 ceph-mon[32348]: osdmap e19: 3 total, 3 up, 3 in 2024-04-23T14:36:29.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:29 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T14:36:29.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:29 smithi028 ceph-mon[27810]: pgmap v91: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:29.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:29 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-23T14:36:29.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:29 smithi028 ceph-mon[27810]: osdmap e19: 3 total, 3 up, 3 in 2024-04-23T14:36:29.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:29 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T14:36:29.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:29 smithi105 ceph-mon[32555]: pgmap v91: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-23T14:36:29.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:29 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-23T14:36:29.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:29 smithi105 ceph-mon[32555]: osdmap e19: 3 total, 3 up, 3 in 2024-04-23T14:36:29.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:29 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T14:36:30.018 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 23 14:36:29 smithi028 sudo[44371]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T14:36:30.018 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 23 14:36:29 smithi028 sudo[44371]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T14:36:30.018 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 23 14:36:29 smithi028 sudo[44371]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T14:36:30.018 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 23 14:36:29 smithi028 sudo[44371]: pam_unix(sudo:session): session closed for user root 2024-04-23T14:36:30.273 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 23 14:36:30 smithi028 sudo[44374]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T14:36:30.273 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 23 14:36:30 smithi028 sudo[44374]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T14:36:30.273 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 23 14:36:30 smithi028 sudo[44374]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T14:36:30.273 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 23 14:36:30 smithi028 sudo[44374]: pam_unix(sudo:session): session closed for user root 2024-04-23T14:36:30.372 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 23 14:36:30 smithi038 sudo[40758]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T14:36:30.372 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 23 14:36:30 smithi038 sudo[40758]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T14:36:30.372 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 23 14:36:30 smithi038 sudo[40758]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T14:36:30.373 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 23 14:36:30 smithi038 sudo[40758]: pam_unix(sudo:session): session closed for user root 2024-04-23T14:36:30.373 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 23 14:36:30 smithi038 sudo[40761]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T14:36:30.373 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 23 14:36:30 smithi038 sudo[40761]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T14:36:30.373 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 23 14:36:30 smithi038 sudo[40761]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T14:36:30.373 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 23 14:36:30 smithi038 sudo[40761]: pam_unix(sudo:session): session closed for user root 2024-04-23T14:36:30.536 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 23 14:36:30 smithi105 sudo[41775]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T14:36:30.536 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 23 14:36:30 smithi105 sudo[41775]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T14:36:30.537 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 23 14:36:30 smithi105 sudo[41775]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T14:36:30.537 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 23 14:36:30 smithi105 sudo[41775]: pam_unix(sudo:session): session closed for user root 2024-04-23T14:36:30.537 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 23 14:36:30 smithi105 sudo[41778]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T14:36:30.537 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 23 14:36:30 smithi105 sudo[41778]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T14:36:30.537 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 23 14:36:30 smithi105 sudo[41778]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T14:36:30.537 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 23 14:36:30 smithi105 sudo[41778]: pam_unix(sudo:session): session closed for user root 2024-04-23T14:36:30.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:30 smithi105 sudo[41781]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T14:36:30.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:30 smithi105 sudo[41781]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T14:36:30.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:30 smithi105 sudo[41781]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T14:36:30.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:30 smithi105 sudo[41781]: pam_unix(sudo:session): session closed for user root 2024-04-23T14:36:30.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:30 smithi105 sudo[41784]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T14:36:30.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:30 smithi105 sudo[41784]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T14:36:30.538 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:30 smithi105 sudo[41784]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T14:36:30.538 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:30 smithi105 sudo[41784]: pam_unix(sudo:session): session closed for user root 2024-04-23T14:36:30.538 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:30 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-23T14:36:30.538 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:30 smithi105 ceph-mon[32555]: osdmap e20: 3 total, 3 up, 3 in 2024-04-23T14:36:30.538 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:30 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:30.538 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:30 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:30.538 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:30 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:30.538 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:30 smithi105 ceph-mon[32555]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T14:36:30.538 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:30 smithi105 ceph-mon[32555]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T14:36:30.538 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:30 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T14:36:30.538 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:30 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:36:30.538 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:30 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:36:30.538 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:30 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T14:36:30.539 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:30 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:36:30.539 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:30 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:36:30.594 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:30 smithi028 sudo[44384]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T14:36:30.594 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:30 smithi028 sudo[44384]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T14:36:30.594 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:30 smithi028 sudo[44384]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T14:36:30.594 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:30 smithi028 sudo[44384]: pam_unix(sudo:session): session closed for user root 2024-04-23T14:36:30.594 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:30 smithi028 sudo[44400]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T14:36:30.594 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:30 smithi028 sudo[44400]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T14:36:30.594 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:30 smithi028 sudo[44400]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T14:36:30.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:30 smithi028 sudo[44400]: pam_unix(sudo:session): session closed for user root 2024-04-23T14:36:30.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:30 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-23T14:36:30.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:30 smithi028 ceph-mon[27810]: osdmap e20: 3 total, 3 up, 3 in 2024-04-23T14:36:30.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:30 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:30.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:30 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:30.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:30 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:30.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:30 smithi028 ceph-mon[27810]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T14:36:30.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:30 smithi028 ceph-mon[27810]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T14:36:30.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:30 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T14:36:30.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:30 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:36:30.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:30 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:36:30.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:30 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T14:36:30.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:30 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:36:30.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:30 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:36:30.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:30 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-23T14:36:30.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:30 smithi038 ceph-mon[32348]: osdmap e20: 3 total, 3 up, 3 in 2024-04-23T14:36:30.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:30 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:30.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:30 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:30.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:30 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:30.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:30 smithi038 ceph-mon[32348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T14:36:30.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:30 smithi038 ceph-mon[32348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T14:36:30.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:30 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T14:36:30.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:30 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:36:30.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:30 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:36:30.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:30 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T14:36:30.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:30 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:36:30.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:30 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:36:30.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:30 smithi038 sudo[40764]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T14:36:30.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:30 smithi038 sudo[40764]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T14:36:30.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:30 smithi038 sudo[40764]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T14:36:30.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:30 smithi038 sudo[40764]: pam_unix(sudo:session): session closed for user root 2024-04-23T14:36:30.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:30 smithi038 sudo[40767]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T14:36:30.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:30 smithi038 sudo[40767]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T14:36:30.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:30 smithi038 sudo[40767]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T14:36:30.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:30 smithi038 sudo[40767]: pam_unix(sudo:session): session closed for user root 2024-04-23T14:36:30.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.903+0000 7fede6800700 1 -- 172.21.15.28:0/1785977425 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fede010f050 msgr2=0x7fede0111440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:30.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.903+0000 7fede6800700 1 --2- 172.21.15.28:0/1785977425 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fede010f050 0x7fede0111440 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fedd8009f90 tx=0x7fedd800b330 comp rx=0 tx=0).stop 2024-04-23T14:36:30.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.904+0000 7fede6800700 1 -- 172.21.15.28:0/1785977425 shutdown_connections 2024-04-23T14:36:30.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.904+0000 7fede6800700 1 --2- 172.21.15.28:0/1785977425 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fede010f050 0x7fede0111440 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:30.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.904+0000 7fede6800700 1 --2- 172.21.15.28:0/1785977425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fede0076b20 0x7fede0074f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:30.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.904+0000 7fede6800700 1 --2- 172.21.15.28:0/1785977425 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fede0075550 0x7fede00759d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:30.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.904+0000 7fede6800700 1 -- 172.21.15.28:0/1785977425 >> 172.21.15.28:0/1785977425 conn(0x7fede00fe030 msgr2=0x7fede0100450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:36:30.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.904+0000 7fede6800700 1 -- 172.21.15.28:0/1785977425 shutdown_connections 2024-04-23T14:36:30.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.904+0000 7fede6800700 1 -- 172.21.15.28:0/1785977425 wait complete. 2024-04-23T14:36:30.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.905+0000 7fede6800700 1 Processor -- start 2024-04-23T14:36:30.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.905+0000 7fede6800700 1 -- start start 2024-04-23T14:36:30.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.905+0000 7fede6800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fede0075550 0x7fede019f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:30.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.905+0000 7fede6800700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fede0076b20 0x7fede019f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:30.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.905+0000 7fede6800700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fede010f050 0x7fede01a6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:30.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.905+0000 7fede6800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fede0114cc0 con 0x7fede0075550 2024-04-23T14:36:30.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.905+0000 7fede6800700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fede0114b40 con 0x7fede0076b20 2024-04-23T14:36:30.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.905+0000 7fede6800700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fede0114e40 con 0x7fede010f050 2024-04-23T14:36:30.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.906+0000 7fede4a00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fede0076b20 0x7fede019f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:30.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.906+0000 7fede4a00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fede0076b20 0x7fede019f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:54472/0 (socket says 172.21.15.28:54472) 2024-04-23T14:36:30.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.906+0000 7fede4a00700 1 -- 172.21.15.28:0/678821640 learned_addr learned my addr 172.21.15.28:0/678821640 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:36:30.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.906+0000 7fede5400700 1 --2- 172.21.15.28:0/678821640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fede0075550 0x7fede019f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:30.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.906+0000 7fede5e00700 1 --2- 172.21.15.28:0/678821640 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fede010f050 0x7fede01a6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:30.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.906+0000 7fede4a00700 1 -- 172.21.15.28:0/678821640 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fede010f050 msgr2=0x7fede01a6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:30.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.906+0000 7fede4a00700 1 --2- 172.21.15.28:0/678821640 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fede010f050 0x7fede01a6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:30.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.906+0000 7fede4a00700 1 -- 172.21.15.28:0/678821640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fede0075550 msgr2=0x7fede019f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:30.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.906+0000 7fede4a00700 1 --2- 172.21.15.28:0/678821640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fede0075550 0x7fede019f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:30.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.906+0000 7fede4a00700 1 -- 172.21.15.28:0/678821640 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fede01a6e20 con 0x7fede0076b20 2024-04-23T14:36:30.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.907+0000 7fede4a00700 1 --2- 172.21.15.28:0/678821640 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fede0076b20 0x7fede019f700 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fedcc00e9f0 tx=0x7fedcc00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:30.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.907+0000 7fedde000700 1 -- 172.21.15.28:0/678821640 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedcc004230 con 0x7fede0076b20 2024-04-23T14:36:30.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.907+0000 7fedde000700 1 -- 172.21.15.28:0/678821640 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedcc0043d0 con 0x7fede0076b20 2024-04-23T14:36:30.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.907+0000 7fede6800700 1 -- 172.21.15.28:0/678821640 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fede01a7110 con 0x7fede0076b20 2024-04-23T14:36:30.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.907+0000 7fedde000700 1 -- 172.21.15.28:0/678821640 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedcc003d80 con 0x7fede0076b20 2024-04-23T14:36:30.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.907+0000 7fede6800700 1 -- 172.21.15.28:0/678821640 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fede01a7650 con 0x7fede0076b20 2024-04-23T14:36:30.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.908+0000 7fede6800700 1 -- 172.21.15.28:0/678821640 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fede00697e0 con 0x7fede0076b20 2024-04-23T14:36:30.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.911+0000 7fedde000700 1 -- 172.21.15.28:0/678821640 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fedcc00c7c0 con 0x7fede0076b20 2024-04-23T14:36:30.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.912+0000 7fedde000700 1 --2- 172.21.15.28:0/678821640 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fedc80a3c30 0x7fedc80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:36:30.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.912+0000 7fedde000700 1 -- 172.21.15.28:0/678821640 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3237+0+0 (secure 0 0 0) 0x7fedcc0cf0d0 con 0x7fede0076b20 2024-04-23T14:36:30.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.912+0000 7fede5400700 1 --2- 172.21.15.28:0/678821640 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fedc80a3c30 0x7fedc80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:36:30.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.913+0000 7fede5400700 1 --2- 172.21.15.28:0/678821640 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fedc80a3c30 0x7fedc80a60f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fede0076f00 tx=0x7fedd400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:36:30.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:30.915+0000 7fedde000700 1 -- 172.21.15.28:0/678821640 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fedcc09cec0 con 0x7fede0076b20 2024-04-23T14:36:31.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:31.156+0000 7fede6800700 1 -- 172.21.15.28:0/678821640 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fede0062f30 con 0x7fedc80a3c30 2024-04-23T14:36:31.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:31 smithi028 ceph-mon[27810]: pgmap v94: 1 pgs: 1 creating+activating; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:31.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:31 smithi028 ceph-mon[27810]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T14:36:31.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:31 smithi028 ceph-mon[27810]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T14:36:31.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:31 smithi028 ceph-mon[27810]: osdmap e21: 3 total, 3 up, 3 in 2024-04-23T14:36:31.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:31 smithi028 ceph-mon[27810]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T14:36:31.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:31 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T14:36:31.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:31 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:36:31.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:31 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:36:31.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:31 smithi028 ceph-mon[27810]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T14:36:31.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:31 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:36:31.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:31 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:36:31.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:31 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:36:31.775 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:31 smithi038 ceph-mon[32348]: pgmap v94: 1 pgs: 1 creating+activating; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:31.775 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:31 smithi038 ceph-mon[32348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T14:36:31.775 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:31 smithi038 ceph-mon[32348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T14:36:31.775 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:31 smithi038 ceph-mon[32348]: osdmap e21: 3 total, 3 up, 3 in 2024-04-23T14:36:31.776 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:31 smithi038 ceph-mon[32348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T14:36:31.776 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:31 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T14:36:31.776 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:31 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:36:31.776 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:31 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:36:31.776 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:31 smithi038 ceph-mon[32348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T14:36:31.776 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:31 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:36:31.776 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:31 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:36:31.776 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:31 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:36:31.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:31 smithi105 ceph-mon[32555]: pgmap v94: 1 pgs: 1 creating+activating; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:31.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:31 smithi105 ceph-mon[32555]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T14:36:31.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:31 smithi105 ceph-mon[32555]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T14:36:31.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:31 smithi105 ceph-mon[32555]: osdmap e21: 3 total, 3 up, 3 in 2024-04-23T14:36:31.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:31 smithi105 ceph-mon[32555]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T14:36:31.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:31 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T14:36:31.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:31 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:36:31.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:31 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:36:31.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:31 smithi105 ceph-mon[32555]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T14:36:31.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:31 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:36:31.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:31 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:36:31.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:31 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:36:32.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:32.404+0000 7fedde000700 1 -- 172.21.15.28:0/678821640 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fedcc098e30 con 0x7fede0076b20 2024-04-23T14:36:32.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:32 smithi028 ceph-mon[27810]: from='client.24197 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:36:32.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:32 smithi105 ceph-mon[32555]: from='client.24197 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:36:32.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:32 smithi038 ceph-mon[32348]: from='client.24197 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:36:33.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:33 smithi028 ceph-mon[27810]: pgmap v96: 1 pgs: 1 creating+activating; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:33.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:33 smithi028 ceph-mon[27810]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-23T14:36:33.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:33 smithi105 ceph-mon[32555]: pgmap v96: 1 pgs: 1 creating+activating; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:33.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:33 smithi105 ceph-mon[32555]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-23T14:36:33.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:33 smithi038 ceph-mon[32348]: pgmap v96: 1 pgs: 1 creating+activating; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:33.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:33 smithi038 ceph-mon[32348]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-23T14:36:35.775 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:35 smithi028 ceph-mon[27810]: pgmap v97: 1 pgs: 1 creating+activating; 0 B data, 102 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:35.775 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:35 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1591344581' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "28bd5608-b426-434e-9db1-bdaa9828bdb9"}]: dispatch 2024-04-23T14:36:35.775 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:35 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1591344581' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "28bd5608-b426-434e-9db1-bdaa9828bdb9"}]': finished 2024-04-23T14:36:35.775 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:35 smithi028 ceph-mon[27810]: osdmap e22: 4 total, 3 up, 4 in 2024-04-23T14:36:35.775 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:35 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T14:36:35.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:35 smithi105 ceph-mon[32555]: pgmap v97: 1 pgs: 1 creating+activating; 0 B data, 102 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:35.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:35 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1591344581' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "28bd5608-b426-434e-9db1-bdaa9828bdb9"}]: dispatch 2024-04-23T14:36:35.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:35 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1591344581' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "28bd5608-b426-434e-9db1-bdaa9828bdb9"}]': finished 2024-04-23T14:36:35.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:35 smithi105 ceph-mon[32555]: osdmap e22: 4 total, 3 up, 4 in 2024-04-23T14:36:35.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T14:36:35.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:35 smithi038 ceph-mon[32348]: pgmap v97: 1 pgs: 1 creating+activating; 0 B data, 102 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:35.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:35 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1591344581' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "28bd5608-b426-434e-9db1-bdaa9828bdb9"}]: dispatch 2024-04-23T14:36:35.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:35 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1591344581' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "28bd5608-b426-434e-9db1-bdaa9828bdb9"}]': finished 2024-04-23T14:36:35.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:35 smithi038 ceph-mon[32348]: osdmap e22: 4 total, 3 up, 4 in 2024-04-23T14:36:35.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:35 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T14:36:36.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:36 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1745988758' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:36:36.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:36 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1745988758' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:36:36.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:36 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1745988758' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:36:37.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:37 smithi105 ceph-mon[32555]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 104 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:37.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:37 smithi038 ceph-mon[32348]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 104 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:37.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:37 smithi028 ceph-mon[27810]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 104 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:39.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:39 smithi105 ceph-mon[32555]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 103 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:39.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:39 smithi038 ceph-mon[32348]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 103 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:39.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:39 smithi028 ceph-mon[27810]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 103 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:41.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:41 smithi028 ceph-mon[27810]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:41.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:41 smithi105 ceph-mon[32555]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:41.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:41 smithi038 ceph-mon[32348]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:43.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:43 smithi105 ceph-mon[32555]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:43.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:43 smithi038 ceph-mon[32348]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:43.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:43 smithi028 ceph-mon[27810]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:44.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:44 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-23T14:36:44.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:44 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:36:44.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:44 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-23T14:36:44.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:44 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:36:44.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-23T14:36:44.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:36:45.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:45 smithi028 ceph-mon[27810]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:45.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:45 smithi028 ceph-mon[27810]: Deploying daemon osd.3 on smithi028 2024-04-23T14:36:45.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:45 smithi105 ceph-mon[32555]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:45.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:45 smithi105 ceph-mon[32555]: Deploying daemon osd.3 on smithi028 2024-04-23T14:36:45.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:45 smithi038 ceph-mon[32348]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:45.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:45 smithi038 ceph-mon[32348]: Deploying daemon osd.3 on smithi028 2024-04-23T14:36:47.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:47 smithi105 ceph-mon[32555]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:47.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:47 smithi028 ceph-mon[27810]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:47.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:47 smithi038 ceph-mon[32348]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:49.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:49 smithi028 ceph-mon[27810]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:49.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:49 smithi105 ceph-mon[32555]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:49.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:49 smithi038 ceph-mon[32348]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:51.550 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:51 smithi028 ceph-mon[27810]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:51.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:51 smithi105 ceph-mon[32555]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:51.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:51 smithi038 ceph-mon[32348]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:53.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:53 smithi028 ceph-mon[27810]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:53.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:53 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:53.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:53 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:53.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:53 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:36:53.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:53 smithi105 ceph-mon[32555]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:53.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:53 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:53.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:53 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:53.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:53 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:36:53.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:53 smithi038 ceph-mon[32348]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:53.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:53 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:53.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:53 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:53.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:53 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:36:55.599 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:55 smithi028 ceph-mon[27810]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:55.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:55 smithi105 ceph-mon[32555]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:55.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:55 smithi038 ceph-mon[32348]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:57.091 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:56 smithi028 ceph-mon[27810]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:57.091 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:56 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:57.091 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:56 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:57.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:56 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:36:57.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:56 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:36:57.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:56 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:57.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:56 smithi028 ceph-mon[27810]: from='osd.3 [v2:172.21.15.28:6810/2480400247,v1:172.21.15.28:6811/2480400247]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-23T14:36:57.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:56 smithi028 ceph-mon[27810]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-23T14:36:57.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:56 smithi038 ceph-mon[32348]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:57.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:56 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:57.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:56 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:57.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:56 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:36:57.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:56 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:36:57.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:56 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:57.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:56 smithi038 ceph-mon[32348]: from='osd.3 [v2:172.21.15.28:6810/2480400247,v1:172.21.15.28:6811/2480400247]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-23T14:36:57.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:56 smithi038 ceph-mon[32348]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-23T14:36:57.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:56 smithi105 ceph-mon[32555]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:57.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:56 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:57.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:56 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:57.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:56 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:36:57.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:56 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:36:57.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:56 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:57.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:56 smithi105 ceph-mon[32555]: from='osd.3 [v2:172.21.15.28:6810/2480400247,v1:172.21.15.28:6811/2480400247]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-23T14:36:57.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:56 smithi105 ceph-mon[32555]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-23T14:36:57.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:57.516+0000 7fedde000700 1 -- 172.21.15.28:0/678821640 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fede0062f30 con 0x7fedc80a3c30 2024-04-23T14:36:57.517 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 3 on host 'smithi028' 2024-04-23T14:36:57.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:57.520+0000 7fede6800700 1 -- 172.21.15.28:0/678821640 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fedc80a3c30 msgr2=0x7fedc80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:57.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:57.520+0000 7fede6800700 1 --2- 172.21.15.28:0/678821640 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fedc80a3c30 0x7fedc80a60f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fede0076f00 tx=0x7fedd400a380 comp rx=0 tx=0).stop 2024-04-23T14:36:57.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:57.520+0000 7fede6800700 1 -- 172.21.15.28:0/678821640 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fede0076b20 msgr2=0x7fede019f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:36:57.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:57.520+0000 7fede6800700 1 --2- 172.21.15.28:0/678821640 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fede0076b20 0x7fede019f700 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fedcc00e9f0 tx=0x7fedcc00ed00 comp rx=0 tx=0).stop 2024-04-23T14:36:57.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:57.520+0000 7fede6800700 1 -- 172.21.15.28:0/678821640 shutdown_connections 2024-04-23T14:36:57.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:57.520+0000 7fede6800700 1 --2- 172.21.15.28:0/678821640 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fedc80a3c30 0x7fedc80a60f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:57.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:57.520+0000 7fede6800700 1 --2- 172.21.15.28:0/678821640 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fede010f050 0x7fede01a6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:57.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:57.520+0000 7fede6800700 1 --2- 172.21.15.28:0/678821640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fede0075550 0x7fede019f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:57.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:57.520+0000 7fede6800700 1 --2- 172.21.15.28:0/678821640 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fede0076b20 0x7fede019f700 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:36:57.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:57.521+0000 7fede6800700 1 -- 172.21.15.28:0/678821640 >> 172.21.15.28:0/678821640 conn(0x7fede00fe030 msgr2=0x7fede01023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:36:57.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:57.521+0000 7fede6800700 1 -- 172.21.15.28:0/678821640 shutdown_connections 2024-04-23T14:36:57.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:36:57.521+0000 7fede6800700 1 -- 172.21.15.28:0/678821640 wait complete. 2024-04-23T14:36:58.252 DEBUG:teuthology.orchestra.run.smithi028:osd.3> sudo journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.3.service 2024-04-23T14:36:58.258 INFO:tasks.cephadm:Deploying osd.4 on smithi038 with /dev/vg_nvme/lv_3... 2024-04-23T14:36:58.258 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-23T14:36:58.513 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T14:36:58.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:58 smithi038 ceph-mon[32348]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-23T14:36:58.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:58 smithi038 ceph-mon[32348]: from='osd.3 [v2:172.21.15.28:6810/2480400247,v1:172.21.15.28:6811/2480400247]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-23T14:36:58.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:58 smithi038 ceph-mon[32348]: osdmap e23: 4 total, 3 up, 4 in 2024-04-23T14:36:58.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:58 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T14:36:58.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:58 smithi038 ceph-mon[32348]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-23T14:36:58.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:58 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:58.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:58 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:58.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:58 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:36:58.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:58 smithi105 ceph-mon[32555]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-23T14:36:58.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:58 smithi105 ceph-mon[32555]: from='osd.3 [v2:172.21.15.28:6810/2480400247,v1:172.21.15.28:6811/2480400247]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-23T14:36:58.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:58 smithi105 ceph-mon[32555]: osdmap e23: 4 total, 3 up, 4 in 2024-04-23T14:36:58.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:58 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T14:36:58.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:58 smithi105 ceph-mon[32555]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-23T14:36:58.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:58 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:58.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:58 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:58.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:58 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:36:58.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:58 smithi028 ceph-mon[27810]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-23T14:36:58.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:58 smithi028 ceph-mon[27810]: from='osd.3 [v2:172.21.15.28:6810/2480400247,v1:172.21.15.28:6811/2480400247]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-23T14:36:58.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:58 smithi028 ceph-mon[27810]: osdmap e23: 4 total, 3 up, 4 in 2024-04-23T14:36:58.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:58 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T14:36:58.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:58 smithi028 ceph-mon[27810]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-23T14:36:58.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:58 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:58.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:58 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:36:58.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:58 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:36:58.874 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 23 14:36:58 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-3[46706]: 2024-04-23T14:36:58.514+0000 7fb378c00700 -1 osd.3 0 waiting for initial osdmap 2024-04-23T14:36:58.875 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 23 14:36:58 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-3[46706]: 2024-04-23T14:36:58.522+0000 7fb373a00700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:36:59.769 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:59 smithi038 ceph-mon[32348]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:59.770 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:59 smithi038 ceph-mon[32348]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-23T14:36:59.770 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:59 smithi038 ceph-mon[32348]: osdmap e24: 4 total, 3 up, 4 in 2024-04-23T14:36:59.770 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:59 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T14:36:59.770 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:59 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T14:36:59.770 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:59 smithi038 ceph-mon[32348]: osd.3 [v2:172.21.15.28:6810/2480400247,v1:172.21.15.28:6811/2480400247] boot 2024-04-23T14:36:59.770 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:59 smithi038 ceph-mon[32348]: osdmap e25: 4 total, 4 up, 4 in 2024-04-23T14:36:59.770 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:36:59 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T14:36:59.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:59 smithi105 ceph-mon[32555]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:59.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:59 smithi105 ceph-mon[32555]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-23T14:36:59.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:59 smithi105 ceph-mon[32555]: osdmap e24: 4 total, 3 up, 4 in 2024-04-23T14:36:59.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:59 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T14:36:59.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:59 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T14:36:59.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:59 smithi105 ceph-mon[32555]: osd.3 [v2:172.21.15.28:6810/2480400247,v1:172.21.15.28:6811/2480400247] boot 2024-04-23T14:36:59.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:59 smithi105 ceph-mon[32555]: osdmap e25: 4 total, 4 up, 4 in 2024-04-23T14:36:59.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:36:59 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T14:36:59.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:59 smithi028 ceph-mon[27810]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-04-23T14:36:59.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:59 smithi028 ceph-mon[27810]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-23T14:36:59.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:59 smithi028 ceph-mon[27810]: osdmap e24: 4 total, 3 up, 4 in 2024-04-23T14:36:59.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:59 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T14:36:59.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:59 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T14:36:59.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:59 smithi028 ceph-mon[27810]: osd.3 [v2:172.21.15.28:6810/2480400247,v1:172.21.15.28:6811/2480400247] boot 2024-04-23T14:36:59.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:59 smithi028 ceph-mon[27810]: osdmap e25: 4 total, 4 up, 4 in 2024-04-23T14:36:59.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:36:59 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T14:37:00.428 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-23T14:37:00.428 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-23T14:37:00.428 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-04-23T14:37:00.428 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-04-23T14:37:00.428 INFO:teuthology.orchestra.run.smithi038.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00760005 s, 1.4 GB/s 2024-04-23T14:37:00.428 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-04-23T14:37:00.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:00 smithi038 ceph-mon[32348]: purged_snaps scrub starts 2024-04-23T14:37:00.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:00 smithi038 ceph-mon[32348]: purged_snaps scrub ok 2024-04-23T14:37:00.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:00 smithi028 ceph-mon[27810]: purged_snaps scrub starts 2024-04-23T14:37:00.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:00 smithi028 ceph-mon[27810]: purged_snaps scrub ok 2024-04-23T14:37:00.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:00 smithi105 ceph-mon[32555]: purged_snaps scrub starts 2024-04-23T14:37:00.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:00 smithi105 ceph-mon[32555]: purged_snaps scrub ok 2024-04-23T14:37:01.034 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi038:vg_nvme/lv_3 2024-04-23T14:37:01.284 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T14:37:01.542 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:01 smithi038 ceph-mon[32348]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-04-23T14:37:01.542 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:01 smithi038 ceph-mon[32348]: osdmap e26: 4 total, 4 up, 4 in 2024-04-23T14:37:01.542 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:01 smithi038 ceph-mon[32348]: osdmap e27: 4 total, 4 up, 4 in 2024-04-23T14:37:01.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:01 smithi028 ceph-mon[27810]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-04-23T14:37:01.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:01 smithi028 ceph-mon[27810]: osdmap e26: 4 total, 4 up, 4 in 2024-04-23T14:37:01.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:01 smithi028 ceph-mon[27810]: osdmap e27: 4 total, 4 up, 4 in 2024-04-23T14:37:01.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:01 smithi105 ceph-mon[32555]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-04-23T14:37:01.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:01 smithi105 ceph-mon[32555]: osdmap e26: 4 total, 4 up, 4 in 2024-04-23T14:37:01.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:01 smithi105 ceph-mon[32555]: osdmap e27: 4 total, 4 up, 4 in 2024-04-23T14:37:02.597 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:02 smithi038 ceph-mon[32348]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-04-23T14:37:02.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.703+0000 7f0abec85700 1 -- 172.21.15.38:0/2468809657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ab8101a70 msgr2=0x7f0ab8101e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:02.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.703+0000 7f0abec85700 1 --2- 172.21.15.38:0/2468809657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ab8101a70 0x7f0ab8101e90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0aac0099c0 tx=0x7f0aac009cd0 comp rx=0 tx=0).stop 2024-04-23T14:37:02.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.704+0000 7f0abec85700 1 -- 172.21.15.38:0/2468809657 shutdown_connections 2024-04-23T14:37:02.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.704+0000 7f0abec85700 1 --2- 172.21.15.38:0/2468809657 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ab8103610 0x7f0ab8109ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:02.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.704+0000 7f0abec85700 1 --2- 172.21.15.38:0/2468809657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ab8101a70 0x7f0ab8101e90 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:02.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.704+0000 7f0abec85700 1 --2- 172.21.15.38:0/2468809657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ab8102c70 0x7f0ab81030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:02.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.704+0000 7f0abec85700 1 -- 172.21.15.38:0/2468809657 >> 172.21.15.38:0/2468809657 conn(0x7f0ab80fd010 msgr2=0x7f0ab80ff450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:02.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.704+0000 7f0abec85700 1 -- 172.21.15.38:0/2468809657 shutdown_connections 2024-04-23T14:37:02.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.704+0000 7f0abec85700 1 -- 172.21.15.38:0/2468809657 wait complete. 2024-04-23T14:37:02.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.704+0000 7f0abec85700 1 Processor -- start 2024-04-23T14:37:02.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.704+0000 7f0abec85700 1 -- start start 2024-04-23T14:37:02.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.704+0000 7f0abec85700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ab8101a70 0x7f0ab8105990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:02.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.704+0000 7f0abec85700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ab8102c70 0x7f0ab8104050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:02.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.704+0000 7f0abec85700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ab8103610 0x7f0ab8104590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:02.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.704+0000 7f0abec85700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ab810d730 con 0x7f0ab8102c70 2024-04-23T14:37:02.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.704+0000 7f0abec85700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0ab810d5b0 con 0x7f0ab8101a70 2024-04-23T14:37:02.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.704+0000 7f0abec85700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0ab810d8b0 con 0x7f0ab8103610 2024-04-23T14:37:02.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.705+0000 7f0abca21700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ab8101a70 0x7f0ab8105990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:02.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.705+0000 7f0abca21700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ab8101a70 0x7f0ab8105990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54892/0 (socket says 172.21.15.38:54892) 2024-04-23T14:37:02.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.705+0000 7f0abca21700 1 -- 172.21.15.38:0/3875099515 learned_addr learned my addr 172.21.15.38:0/3875099515 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-23T14:37:02.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.705+0000 7f0ab7fff700 1 --2- 172.21.15.38:0/3875099515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ab8102c70 0x7f0ab8104050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:02.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.705+0000 7f0abd222700 1 --2- 172.21.15.38:0/3875099515 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ab8103610 0x7f0ab8104590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:02.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.705+0000 7f0abca21700 1 -- 172.21.15.38:0/3875099515 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ab8103610 msgr2=0x7f0ab8104590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:02.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.705+0000 7f0abca21700 1 --2- 172.21.15.38:0/3875099515 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ab8103610 0x7f0ab8104590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:02.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.705+0000 7f0abca21700 1 -- 172.21.15.38:0/3875099515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ab8102c70 msgr2=0x7f0ab8104050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:02.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.705+0000 7f0abca21700 1 --2- 172.21.15.38:0/3875099515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ab8102c70 0x7f0ab8104050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:02.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.705+0000 7f0abca21700 1 -- 172.21.15.38:0/3875099515 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ab8104dd0 con 0x7f0ab8101a70 2024-04-23T14:37:02.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.705+0000 7f0abca21700 1 --2- 172.21.15.38:0/3875099515 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ab8101a70 0x7f0ab8105990 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0aac00c010 tx=0x7f0aac005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:02.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.705+0000 7f0ab5ffb700 1 -- 172.21.15.38:0/3875099515 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aac027070 con 0x7f0ab8101a70 2024-04-23T14:37:02.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.705+0000 7f0ab5ffb700 1 -- 172.21.15.38:0/3875099515 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0aac0041d0 con 0x7f0ab8101a70 2024-04-23T14:37:02.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.705+0000 7f0abec85700 1 -- 172.21.15.38:0/3875099515 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ab81a2fb0 con 0x7f0ab8101a70 2024-04-23T14:37:02.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.705+0000 7f0ab5ffb700 1 -- 172.21.15.38:0/3875099515 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aac022e60 con 0x7f0ab8101a70 2024-04-23T14:37:02.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.705+0000 7f0abec85700 1 -- 172.21.15.38:0/3875099515 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ab81a3420 con 0x7f0ab8101a70 2024-04-23T14:37:02.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.706+0000 7f0abec85700 1 -- 172.21.15.38:0/3875099515 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ab80697e0 con 0x7f0ab8101a70 2024-04-23T14:37:02.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.708+0000 7f0ab5ffb700 1 -- 172.21.15.38:0/3875099515 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0aac02a070 con 0x7f0ab8101a70 2024-04-23T14:37:02.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.708+0000 7f0ab5ffb700 1 --2- 172.21.15.38:0/3875099515 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f0aa80a3c40 0x7f0aa80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:02.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.708+0000 7f0ab5ffb700 1 -- 172.21.15.38:0/3875099515 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3669+0+0 (secure 0 0 0) 0x7f0aac0d0130 con 0x7f0ab8101a70 2024-04-23T14:37:02.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.709+0000 7f0ab7fff700 1 --2- 172.21.15.38:0/3875099515 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f0aa80a3c40 0x7f0aa80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:02.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.709+0000 7f0ab7fff700 1 --2- 172.21.15.38:0/3875099515 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f0aa80a3c40 0x7f0aa80a6100 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4006fd0 tx=0x7f0aa4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:02.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.713+0000 7f0ab5ffb700 1 -- 172.21.15.38:0/3875099515 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0aac013070 con 0x7f0ab8101a70 2024-04-23T14:37:02.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:02 smithi105 ceph-mon[32555]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-04-23T14:37:02.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:02 smithi028 ceph-mon[27810]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-04-23T14:37:02.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:02.930+0000 7f0abec85700 1 -- 172.21.15.38:0/3875099515 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f0ab80ff1f0 con 0x7f0aa80a3c40 2024-04-23T14:37:03.838 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:03 smithi038 ceph-mon[32348]: osdmap e28: 4 total, 4 up, 4 in 2024-04-23T14:37:03.838 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:03 smithi038 ceph-mon[32348]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:37:03.838 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:03 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:37:03.838 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:03 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:37:03.838 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:03 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:37:03.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:03 smithi028 ceph-mon[27810]: osdmap e28: 4 total, 4 up, 4 in 2024-04-23T14:37:03.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:03 smithi028 ceph-mon[27810]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:37:03.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:03 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:37:03.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:03 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:37:03.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:03 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:37:04.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:03 smithi105 ceph-mon[32555]: osdmap e28: 4 total, 4 up, 4 in 2024-04-23T14:37:04.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:03 smithi105 ceph-mon[32555]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:37:04.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:03 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:37:04.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:03 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:37:04.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:03 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:37:04.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:04 smithi028 ceph-mon[27810]: osdmap e29: 4 total, 4 up, 4 in 2024-04-23T14:37:04.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:04 smithi028 ceph-mon[27810]: pgmap v120: 1 pgs: 1 peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-04-23T14:37:04.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:04 smithi028 ceph-mon[27810]: Detected new or changed devices on smithi028 2024-04-23T14:37:04.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:04 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:04.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:04 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:04.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:04 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:37:04.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:04 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:37:04.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:04 smithi028 ceph-mon[27810]: Adjusting osd_memory_target on smithi028 to 8547M 2024-04-23T14:37:04.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:04 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:04.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:04 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:37:04.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:04 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:37:04.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:04 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:04.863 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:04 smithi038 ceph-mon[32348]: osdmap e29: 4 total, 4 up, 4 in 2024-04-23T14:37:04.863 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:04 smithi038 ceph-mon[32348]: pgmap v120: 1 pgs: 1 peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-04-23T14:37:04.863 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:04 smithi038 ceph-mon[32348]: Detected new or changed devices on smithi028 2024-04-23T14:37:04.863 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:04 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:04.863 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:04 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:04.863 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:04 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:37:04.863 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:04 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:37:04.863 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:04 smithi038 ceph-mon[32348]: Adjusting osd_memory_target on smithi028 to 8547M 2024-04-23T14:37:04.863 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:04 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:04.864 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:04 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:37:04.864 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:04 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:37:04.864 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:04 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:05.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:04 smithi105 ceph-mon[32555]: osdmap e29: 4 total, 4 up, 4 in 2024-04-23T14:37:05.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:04 smithi105 ceph-mon[32555]: pgmap v120: 1 pgs: 1 peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-04-23T14:37:05.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:04 smithi105 ceph-mon[32555]: Detected new or changed devices on smithi028 2024-04-23T14:37:05.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:04 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:05.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:04 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:05.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:04 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:37:05.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:04 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:37:05.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:04 smithi105 ceph-mon[32555]: Adjusting osd_memory_target on smithi028 to 8547M 2024-04-23T14:37:05.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:04 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:05.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:04 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:37:05.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:04 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:37:05.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:04 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:07.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:06 smithi105 ceph-mon[32555]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-04-23T14:37:07.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:06 smithi105 ceph-mon[32555]: from='client.? 172.21.15.38:0/4079871369' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5d1c89b-025f-4d57-8807-59d75c787b98"}]: dispatch 2024-04-23T14:37:07.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:06 smithi105 ceph-mon[32555]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5d1c89b-025f-4d57-8807-59d75c787b98"}]: dispatch 2024-04-23T14:37:07.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:06 smithi105 ceph-mon[32555]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e5d1c89b-025f-4d57-8807-59d75c787b98"}]': finished 2024-04-23T14:37:07.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:06 smithi105 ceph-mon[32555]: osdmap e30: 5 total, 4 up, 5 in 2024-04-23T14:37:07.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:06 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T14:37:07.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:06 smithi105 ceph-mon[32555]: from='client.? 172.21.15.38:0/3364546351' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:37:07.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:06 smithi038 ceph-mon[32348]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-04-23T14:37:07.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:06 smithi038 ceph-mon[32348]: from='client.? 172.21.15.38:0/4079871369' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5d1c89b-025f-4d57-8807-59d75c787b98"}]: dispatch 2024-04-23T14:37:07.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:06 smithi038 ceph-mon[32348]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5d1c89b-025f-4d57-8807-59d75c787b98"}]: dispatch 2024-04-23T14:37:07.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:06 smithi038 ceph-mon[32348]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e5d1c89b-025f-4d57-8807-59d75c787b98"}]': finished 2024-04-23T14:37:07.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:06 smithi038 ceph-mon[32348]: osdmap e30: 5 total, 4 up, 5 in 2024-04-23T14:37:07.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:06 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T14:37:07.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:06 smithi038 ceph-mon[32348]: from='client.? 172.21.15.38:0/3364546351' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:37:07.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:06 smithi028 ceph-mon[27810]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-04-23T14:37:07.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:06 smithi028 ceph-mon[27810]: from='client.? 172.21.15.38:0/4079871369' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5d1c89b-025f-4d57-8807-59d75c787b98"}]: dispatch 2024-04-23T14:37:07.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:06 smithi028 ceph-mon[27810]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5d1c89b-025f-4d57-8807-59d75c787b98"}]: dispatch 2024-04-23T14:37:07.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:06 smithi028 ceph-mon[27810]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e5d1c89b-025f-4d57-8807-59d75c787b98"}]': finished 2024-04-23T14:37:07.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:06 smithi028 ceph-mon[27810]: osdmap e30: 5 total, 4 up, 5 in 2024-04-23T14:37:07.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:06 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T14:37:07.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:06 smithi028 ceph-mon[27810]: from='client.? 172.21.15.38:0/3364546351' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:37:09.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:08 smithi105 ceph-mon[32555]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-23T14:37:09.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:08 smithi038 ceph-mon[32348]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-23T14:37:09.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:08 smithi028 ceph-mon[27810]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-23T14:37:11.026 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:10 smithi038 ceph-mon[32348]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-23T14:37:11.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:10 smithi105 ceph-mon[32555]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-23T14:37:11.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:10 smithi028 ceph-mon[27810]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-23T14:37:13.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:12 smithi038 ceph-mon[32348]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-23T14:37:13.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:12 smithi028 ceph-mon[27810]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-23T14:37:13.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:12 smithi105 ceph-mon[32555]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-23T14:37:15.108 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:14 smithi038 ceph-mon[32348]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T14:37:15.108 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:14 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-23T14:37:15.108 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:14 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:37:15.108 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:14 smithi038 ceph-mon[32348]: Deploying daemon osd.4 on smithi038 2024-04-23T14:37:15.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:14 smithi028 ceph-mon[27810]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T14:37:15.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:14 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-23T14:37:15.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:14 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:37:15.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:14 smithi028 ceph-mon[27810]: Deploying daemon osd.4 on smithi038 2024-04-23T14:37:15.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:14 smithi105 ceph-mon[32555]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T14:37:15.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:14 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-23T14:37:15.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:14 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:37:15.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:14 smithi105 ceph-mon[32555]: Deploying daemon osd.4 on smithi038 2024-04-23T14:37:17.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:16 smithi038 ceph-mon[32348]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-04-23T14:37:17.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:16 smithi028 ceph-mon[27810]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-04-23T14:37:17.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:16 smithi105 ceph-mon[32555]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-04-23T14:37:19.029 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:18 smithi038 ceph-mon[32348]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-04-23T14:37:19.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:18 smithi028 ceph-mon[27810]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-04-23T14:37:19.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:18 smithi105 ceph-mon[32555]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-04-23T14:37:20.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:20 smithi038 ceph-mon[32348]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-04-23T14:37:20.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:20 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:20.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:20 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:20.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:20 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:37:20.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:20 smithi028 ceph-mon[27810]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-04-23T14:37:20.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:20 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:20.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:20 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:20.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:20 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:37:21.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:20 smithi105 ceph-mon[32555]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-04-23T14:37:21.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:20 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:21.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:20 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:21.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:20 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:37:22.732 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:22 smithi038 ceph-mon[32348]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-04-23T14:37:22.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:22 smithi028 ceph-mon[27810]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-04-23T14:37:23.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:22 smithi105 ceph-mon[32555]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-04-23T14:37:23.062 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 4 on host 'smithi038' 2024-04-23T14:37:23.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:23.059+0000 7f0ab5ffb700 1 -- 172.21.15.38:0/3875099515 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0ab80ff1f0 con 0x7f0aa80a3c40 2024-04-23T14:37:23.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:23.061+0000 7f0abec85700 1 -- 172.21.15.38:0/3875099515 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f0aa80a3c40 msgr2=0x7f0aa80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:23.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:23.061+0000 7f0abec85700 1 --2- 172.21.15.38:0/3875099515 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f0aa80a3c40 0x7f0aa80a6100 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4006fd0 tx=0x7f0aa4008040 comp rx=0 tx=0).stop 2024-04-23T14:37:23.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:23.061+0000 7f0abec85700 1 -- 172.21.15.38:0/3875099515 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ab8101a70 msgr2=0x7f0ab8105990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:23.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:23.061+0000 7f0abec85700 1 --2- 172.21.15.38:0/3875099515 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ab8101a70 0x7f0ab8105990 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0aac00c010 tx=0x7f0aac005e80 comp rx=0 tx=0).stop 2024-04-23T14:37:23.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:23.061+0000 7f0abec85700 1 -- 172.21.15.38:0/3875099515 shutdown_connections 2024-04-23T14:37:23.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:23.061+0000 7f0abec85700 1 --2- 172.21.15.38:0/3875099515 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f0aa80a3c40 0x7f0aa80a6100 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:23.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:23.061+0000 7f0abec85700 1 --2- 172.21.15.38:0/3875099515 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ab8103610 0x7f0ab8104590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:23.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:23.061+0000 7f0abec85700 1 --2- 172.21.15.38:0/3875099515 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ab8101a70 0x7f0ab8105990 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:23.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:23.061+0000 7f0abec85700 1 --2- 172.21.15.38:0/3875099515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ab8102c70 0x7f0ab8104050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:23.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:23.062+0000 7f0abec85700 1 -- 172.21.15.38:0/3875099515 >> 172.21.15.38:0/3875099515 conn(0x7f0ab80fd010 msgr2=0x7f0ab80feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:23.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:23.062+0000 7f0abec85700 1 -- 172.21.15.38:0/3875099515 shutdown_connections 2024-04-23T14:37:23.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:37:23.062+0000 7f0abec85700 1 -- 172.21.15.38:0/3875099515 wait complete. 2024-04-23T14:37:23.689 DEBUG:teuthology.orchestra.run.smithi038:osd.4> sudo journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.4.service 2024-04-23T14:37:23.692 INFO:tasks.cephadm:Deploying osd.5 on smithi105 with /dev/vg_nvme/lv_3... 2024-04-23T14:37:23.692 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-23T14:37:23.935 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T14:37:24.357 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:24 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:24.358 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:24 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:24.358 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:24 smithi105 ceph-mon[32555]: from='osd.4 [v2:172.21.15.38:6808/3772189246,v1:172.21.15.38:6809/3772189246]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T14:37:24.358 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:24 smithi105 ceph-mon[32555]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T14:37:24.358 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:24 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:24.358 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:24 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:24.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:24 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:24.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:24 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:24.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:24 smithi038 ceph-mon[32348]: from='osd.4 [v2:172.21.15.38:6808/3772189246,v1:172.21.15.38:6809/3772189246]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T14:37:24.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:24 smithi038 ceph-mon[32348]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T14:37:24.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:24 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:24.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:24 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:24.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:24 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:24.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:24 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:24.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:24 smithi028 ceph-mon[27810]: from='osd.4 [v2:172.21.15.38:6808/3772189246,v1:172.21.15.38:6809/3772189246]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T14:37:24.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:24 smithi028 ceph-mon[27810]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T14:37:24.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:24 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:24.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:24 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:25.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:25 smithi038 ceph-mon[32348]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-04-23T14:37:25.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:25 smithi038 ceph-mon[32348]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-23T14:37:25.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:25 smithi038 ceph-mon[32348]: from='osd.4 [v2:172.21.15.38:6808/3772189246,v1:172.21.15.38:6809/3772189246]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-23T14:37:25.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:25 smithi038 ceph-mon[32348]: osdmap e31: 5 total, 4 up, 5 in 2024-04-23T14:37:25.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:25 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T14:37:25.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:25 smithi038 ceph-mon[32348]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-23T14:37:25.310 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:25 smithi105 ceph-mon[32555]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-04-23T14:37:25.310 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:25 smithi105 ceph-mon[32555]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-23T14:37:25.310 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:25 smithi105 ceph-mon[32555]: from='osd.4 [v2:172.21.15.38:6808/3772189246,v1:172.21.15.38:6809/3772189246]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-23T14:37:25.310 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:25 smithi105 ceph-mon[32555]: osdmap e31: 5 total, 4 up, 5 in 2024-04-23T14:37:25.310 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:25 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T14:37:25.310 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:25 smithi105 ceph-mon[32555]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-23T14:37:25.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:25 smithi028 ceph-mon[27810]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-04-23T14:37:25.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:25 smithi028 ceph-mon[27810]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-23T14:37:25.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:25 smithi028 ceph-mon[27810]: from='osd.4 [v2:172.21.15.38:6808/3772189246,v1:172.21.15.38:6809/3772189246]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-23T14:37:25.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:25 smithi028 ceph-mon[27810]: osdmap e31: 5 total, 4 up, 5 in 2024-04-23T14:37:25.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:25 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T14:37:25.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:25 smithi028 ceph-mon[27810]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-23T14:37:25.888 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-23T14:37:25.888 INFO:teuthology.orchestra.run.smithi105.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-23T14:37:25.888 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10+0 records in 2024-04-23T14:37:25.888 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2024-04-23T14:37:25.888 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113111 s, 927 MB/s 2024-04-23T14:37:25.888 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping successful for: 2024-04-23T14:37:26.081 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 23 14:37:25 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-4[43599]: 2024-04-23T14:37:25.879+0000 7f5260b3c700 -1 osd.4 0 waiting for initial osdmap 2024-04-23T14:37:26.082 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 23 14:37:25 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-4[43599]: 2024-04-23T14:37:25.919+0000 7f5258927700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:37:26.235 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:26 smithi105 ceph-mon[32555]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-23T14:37:26.235 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:26 smithi105 ceph-mon[32555]: osdmap e32: 5 total, 4 up, 5 in 2024-04-23T14:37:26.236 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:26 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T14:37:26.236 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:26 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T14:37:26.236 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:26 smithi105 ceph-mon[32555]: from='osd.4 ' entity='osd.4' 2024-04-23T14:37:26.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:26 smithi038 ceph-mon[32348]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-23T14:37:26.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:26 smithi038 ceph-mon[32348]: osdmap e32: 5 total, 4 up, 5 in 2024-04-23T14:37:26.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:26 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T14:37:26.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:26 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T14:37:26.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:26 smithi038 ceph-mon[32348]: from='osd.4 ' entity='osd.4' 2024-04-23T14:37:26.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:26 smithi028 ceph-mon[27810]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-23T14:37:26.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:26 smithi028 ceph-mon[27810]: osdmap e32: 5 total, 4 up, 5 in 2024-04-23T14:37:26.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:26 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T14:37:26.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:26 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T14:37:26.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:26 smithi028 ceph-mon[27810]: from='osd.4 ' entity='osd.4' 2024-04-23T14:37:26.573 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi105:vg_nvme/lv_3 2024-04-23T14:37:26.827 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T14:37:27.173 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:27 smithi105 ceph-mon[32555]: purged_snaps scrub starts 2024-04-23T14:37:27.173 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:27 smithi105 ceph-mon[32555]: purged_snaps scrub ok 2024-04-23T14:37:27.173 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:27 smithi105 ceph-mon[32555]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-04-23T14:37:27.173 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:27 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T14:37:27.173 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:27 smithi105 ceph-mon[32555]: osd.4 [v2:172.21.15.38:6808/3772189246,v1:172.21.15.38:6809/3772189246] boot 2024-04-23T14:37:27.173 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:27 smithi105 ceph-mon[32555]: osdmap e33: 5 total, 5 up, 5 in 2024-04-23T14:37:27.173 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:27 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T14:37:27.191 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:27 smithi038 ceph-mon[32348]: purged_snaps scrub starts 2024-04-23T14:37:27.191 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:27 smithi038 ceph-mon[32348]: purged_snaps scrub ok 2024-04-23T14:37:27.191 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:27 smithi038 ceph-mon[32348]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-04-23T14:37:27.191 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:27 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T14:37:27.191 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:27 smithi038 ceph-mon[32348]: osd.4 [v2:172.21.15.38:6808/3772189246,v1:172.21.15.38:6809/3772189246] boot 2024-04-23T14:37:27.191 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:27 smithi038 ceph-mon[32348]: osdmap e33: 5 total, 5 up, 5 in 2024-04-23T14:37:27.191 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:27 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T14:37:27.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:27 smithi028 ceph-mon[27810]: purged_snaps scrub starts 2024-04-23T14:37:27.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:27 smithi028 ceph-mon[27810]: purged_snaps scrub ok 2024-04-23T14:37:27.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:27 smithi028 ceph-mon[27810]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-04-23T14:37:27.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:27 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T14:37:27.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:27 smithi028 ceph-mon[27810]: osd.4 [v2:172.21.15.38:6808/3772189246,v1:172.21.15.38:6809/3772189246] boot 2024-04-23T14:37:27.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:27 smithi028 ceph-mon[27810]: osdmap e33: 5 total, 5 up, 5 in 2024-04-23T14:37:27.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:27 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T14:37:28.442 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.439+0000 7f046ea23700 1 -- 172.21.15.105:0/1108769548 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0468102c40 msgr2=0x7f04681030a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:28.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.439+0000 7f046ea23700 1 --2- 172.21.15.105:0/1108769548 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0468102c40 0x7f04681030a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f045c009990 tx=0x7f045c009ca0 comp rx=0 tx=0).stop 2024-04-23T14:37:28.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.439+0000 7f046ea23700 1 -- 172.21.15.105:0/1108769548 shutdown_connections 2024-04-23T14:37:28.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.439+0000 7f046ea23700 1 --2- 172.21.15.105:0/1108769548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04681035e0 0x7f0468109e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:28.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.439+0000 7f046ea23700 1 --2- 172.21.15.105:0/1108769548 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0468101a40 0x7f0468101e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:28.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.439+0000 7f046ea23700 1 --2- 172.21.15.105:0/1108769548 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0468102c40 0x7f04681030a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:28.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.439+0000 7f046ea23700 1 -- 172.21.15.105:0/1108769548 >> 172.21.15.105:0/1108769548 conn(0x7f04680fcfe0 msgr2=0x7f04680ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:28.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.439+0000 7f046ea23700 1 -- 172.21.15.105:0/1108769548 shutdown_connections 2024-04-23T14:37:28.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.439+0000 7f046ea23700 1 -- 172.21.15.105:0/1108769548 wait complete. 2024-04-23T14:37:28.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.440+0000 7f046ea23700 1 Processor -- start 2024-04-23T14:37:28.444 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.440+0000 7f046ea23700 1 -- start start 2024-04-23T14:37:28.444 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.440+0000 7f046ea23700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0468101a40 0x7f046806bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:28.444 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.440+0000 7f046ea23700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0468102c40 0x7f046819ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:28.444 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.440+0000 7f046ea23700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04681035e0 0x7f04681a2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:28.444 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.440+0000 7f046ea23700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f046810d6a0 con 0x7f0468101a40 2024-04-23T14:37:28.444 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.440+0000 7f046ea23700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f046810d520 con 0x7f04681035e0 2024-04-23T14:37:28.444 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.440+0000 7f046ea23700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f046810d820 con 0x7f0468102c40 2024-04-23T14:37:28.444 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.440+0000 7f04677fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0468102c40 0x7f046819ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:28.444 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.440+0000 7f04677fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0468102c40 0x7f046819ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40724/0 (socket says 172.21.15.105:40724) 2024-04-23T14:37:28.444 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.440+0000 7f04677fe700 1 -- 172.21.15.105:0/2038316502 learned_addr learned my addr 172.21.15.105:0/2038316502 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-23T14:37:28.444 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.440+0000 7f046cfc0700 1 --2- 172.21.15.105:0/2038316502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04681035e0 0x7f04681a2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:28.445 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.440+0000 7f04677fe700 1 -- 172.21.15.105:0/2038316502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04681035e0 msgr2=0x7f04681a2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:28.445 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.440+0000 7f04677fe700 1 --2- 172.21.15.105:0/2038316502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04681035e0 0x7f04681a2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:28.445 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.440+0000 7f04677fe700 1 -- 172.21.15.105:0/2038316502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0468101a40 msgr2=0x7f046806bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:28.445 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.440+0000 7f04677fe700 1 --2- 172.21.15.105:0/2038316502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0468101a40 0x7f046806bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:28.445 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.440+0000 7f04677fe700 1 -- 172.21.15.105:0/2038316502 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04681a2c40 con 0x7f0468102c40 2024-04-23T14:37:28.445 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.441+0000 7f04677fe700 1 --2- 172.21.15.105:0/2038316502 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0468102c40 0x7f046819ff10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f045c005910 tx=0x7f045c009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:28.445 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.441+0000 7f04657fa700 1 -- 172.21.15.105:0/2038316502 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f045c027070 con 0x7f0468102c40 2024-04-23T14:37:28.445 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.441+0000 7f04657fa700 1 -- 172.21.15.105:0/2038316502 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f045c004150 con 0x7f0468102c40 2024-04-23T14:37:28.445 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.441+0000 7f046ea23700 1 -- 172.21.15.105:0/2038316502 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04681a2ed0 con 0x7f0468102c40 2024-04-23T14:37:28.445 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.441+0000 7f04657fa700 1 -- 172.21.15.105:0/2038316502 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f045c00f460 con 0x7f0468102c40 2024-04-23T14:37:28.445 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.441+0000 7f046ea23700 1 -- 172.21.15.105:0/2038316502 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04681a33a0 con 0x7f0468102c40 2024-04-23T14:37:28.446 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.442+0000 7f046ea23700 1 -- 172.21.15.105:0/2038316502 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04680697e0 con 0x7f0468102c40 2024-04-23T14:37:28.446 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.443+0000 7f04657fa700 1 -- 172.21.15.105:0/2038316502 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f045c02a070 con 0x7f0468102c40 2024-04-23T14:37:28.446 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.443+0000 7f04657fa700 1 --2- 172.21.15.105:0/2038316502 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f04500a3cc0 0x7f04500a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:28.447 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.443+0000 7f04657fa700 1 -- 172.21.15.105:0/2038316502 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7f045c0d01e0 con 0x7f0468102c40 2024-04-23T14:37:28.447 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.444+0000 7f0467fff700 1 --2- 172.21.15.105:0/2038316502 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f04500a3cc0 0x7f04500a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:28.447 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.445+0000 7f0467fff700 1 --2- 172.21.15.105:0/2038316502 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f04500a3cc0 0x7f04500a6180 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0468102aa0 tx=0x7f0458009470 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:28.449 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.448+0000 7f04657fa700 1 -- 172.21.15.105:0/2038316502 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f045c05eff0 con 0x7f0468102c40 2024-04-23T14:37:28.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:28 smithi105 ceph-mon[32555]: Detected new or changed devices on smithi038 2024-04-23T14:37:28.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:28 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:28.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:28 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:28.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:28 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:37:28.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:28 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:37:28.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:28 smithi105 ceph-mon[32555]: Adjusting osd_memory_target on smithi038 to 8546M 2024-04-23T14:37:28.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:28 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:28.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:28 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:37:28.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:28 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:37:28.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:28 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:28.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:28 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:37:28.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:28 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:37:28.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:28 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:37:28.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:28 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:28.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:28 smithi038 ceph-mon[32348]: Detected new or changed devices on smithi038 2024-04-23T14:37:28.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:28 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:28.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:28 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:28.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:28 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:37:28.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:28 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:37:28.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:28 smithi038 ceph-mon[32348]: Adjusting osd_memory_target on smithi038 to 8546M 2024-04-23T14:37:28.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:28 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:28.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:28 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:37:28.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:28 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:37:28.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:28 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:28.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:28 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:37:28.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:28 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:37:28.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:28 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:37:28.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:28 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:28.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:28 smithi028 ceph-mon[27810]: Detected new or changed devices on smithi038 2024-04-23T14:37:28.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:28.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:28.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:37:28.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:37:28.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:28 smithi028 ceph-mon[27810]: Adjusting osd_memory_target on smithi038 to 8546M 2024-04-23T14:37:28.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:28.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:37:28.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:37:28.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:28.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:37:28.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:37:28.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:37:28.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:28.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:28.654+0000 7f046ea23700 1 -- 172.21.15.105:0/2038316502 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f0468062f30 con 0x7f04500a3cc0 2024-04-23T14:37:29.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:29 smithi105 ceph-mon[32555]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:29.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:29 smithi105 ceph-mon[32555]: osdmap e34: 5 total, 5 up, 5 in 2024-04-23T14:37:29.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:29 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:37:29.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:29 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:37:29.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:29 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:37:29.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:29 smithi038 ceph-mon[32348]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:29.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:29 smithi038 ceph-mon[32348]: osdmap e34: 5 total, 5 up, 5 in 2024-04-23T14:37:29.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:29 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:37:29.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:29 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:37:29.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:29 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:37:29.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:29 smithi028 ceph-mon[27810]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:29.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:29 smithi028 ceph-mon[27810]: osdmap e34: 5 total, 5 up, 5 in 2024-04-23T14:37:29.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:29 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:37:29.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:29 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:37:29.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:29 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:37:30.387 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:30 smithi105 ceph-mon[32555]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:37:30.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:30 smithi038 ceph-mon[32348]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:37:30.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:30 smithi028 ceph-mon[27810]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:37:31.577 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:31 smithi105 ceph-mon[32555]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:31.578 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:31 smithi105 ceph-mon[32555]: from='client.? 172.21.15.105:0/3662487007' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cf9cf4c-aafd-4ed3-bfb1-0527710a0292"}]: dispatch 2024-04-23T14:37:31.578 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:31 smithi105 ceph-mon[32555]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cf9cf4c-aafd-4ed3-bfb1-0527710a0292"}]: dispatch 2024-04-23T14:37:31.578 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:31 smithi105 ceph-mon[32555]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9cf9cf4c-aafd-4ed3-bfb1-0527710a0292"}]': finished 2024-04-23T14:37:31.578 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:31 smithi105 ceph-mon[32555]: osdmap e35: 6 total, 5 up, 6 in 2024-04-23T14:37:31.578 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:31 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T14:37:31.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:31 smithi038 ceph-mon[32348]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:31.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:31 smithi038 ceph-mon[32348]: from='client.? 172.21.15.105:0/3662487007' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cf9cf4c-aafd-4ed3-bfb1-0527710a0292"}]: dispatch 2024-04-23T14:37:31.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:31 smithi038 ceph-mon[32348]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cf9cf4c-aafd-4ed3-bfb1-0527710a0292"}]: dispatch 2024-04-23T14:37:31.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:31 smithi038 ceph-mon[32348]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9cf9cf4c-aafd-4ed3-bfb1-0527710a0292"}]': finished 2024-04-23T14:37:31.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:31 smithi038 ceph-mon[32348]: osdmap e35: 6 total, 5 up, 6 in 2024-04-23T14:37:31.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:31 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T14:37:31.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:31 smithi028 ceph-mon[27810]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:31.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:31 smithi028 ceph-mon[27810]: from='client.? 172.21.15.105:0/3662487007' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cf9cf4c-aafd-4ed3-bfb1-0527710a0292"}]: dispatch 2024-04-23T14:37:31.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:31 smithi028 ceph-mon[27810]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cf9cf4c-aafd-4ed3-bfb1-0527710a0292"}]: dispatch 2024-04-23T14:37:31.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:31 smithi028 ceph-mon[27810]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9cf9cf4c-aafd-4ed3-bfb1-0527710a0292"}]': finished 2024-04-23T14:37:31.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:31 smithi028 ceph-mon[27810]: osdmap e35: 6 total, 5 up, 6 in 2024-04-23T14:37:31.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:31 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T14:37:32.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:32 smithi038 ceph-mon[32348]: from='client.? 172.21.15.105:0/2062377216' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:37:32.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:32 smithi028 ceph-mon[27810]: from='client.? 172.21.15.105:0/2062377216' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:37:32.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:32 smithi105 ceph-mon[32555]: from='client.? 172.21.15.105:0/2062377216' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:37:33.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:33 smithi038 ceph-mon[32348]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:33.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:33 smithi028 ceph-mon[27810]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:33.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:33 smithi105 ceph-mon[32555]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:35.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:35 smithi105 ceph-mon[32555]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:35.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:35 smithi038 ceph-mon[32348]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:35.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:35 smithi028 ceph-mon[27810]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:37.556 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:37 smithi105 ceph-mon[32555]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:37.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:37 smithi038 ceph-mon[32348]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:37.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:37 smithi028 ceph-mon[27810]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:39.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:39 smithi105 ceph-mon[32555]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:39.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:39 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-23T14:37:39.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:39 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:37:39.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:39 smithi038 ceph-mon[32348]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:39.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:39 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-23T14:37:39.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:39 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:37:39.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:39 smithi028 ceph-mon[27810]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:39.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:39 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-23T14:37:39.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:39 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:37:40.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:40 smithi038 ceph-mon[32348]: Deploying daemon osd.5 on smithi105 2024-04-23T14:37:40.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:40 smithi028 ceph-mon[27810]: Deploying daemon osd.5 on smithi105 2024-04-23T14:37:40.640 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:40 smithi105 ceph-mon[32555]: Deploying daemon osd.5 on smithi105 2024-04-23T14:37:41.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:41 smithi105 ceph-mon[32555]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:41.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:41 smithi038 ceph-mon[32348]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:41.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:41 smithi028 ceph-mon[27810]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:43.594 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:43 smithi105 ceph-mon[32555]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:43.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:43 smithi038 ceph-mon[32348]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:43.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:43 smithi028 ceph-mon[27810]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:45.503 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:45 smithi105 ceph-mon[32555]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:45.503 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:45 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:45.503 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:45 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:45.503 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:45 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:37:45.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:45 smithi038 ceph-mon[32348]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:45.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:45 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:45.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:45 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:45.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:45 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:37:45.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:45 smithi028 ceph-mon[27810]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:45.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:45 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:45.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:45 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:45.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:45 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:37:47.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:47 smithi105 ceph-mon[32555]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:47.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:47 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:47.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:47 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:47.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:47 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:37:47.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:47 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:37:47.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:47 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:47.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:47 smithi038 ceph-mon[32348]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:47.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:47 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:47.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:47 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:47.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:47 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:37:47.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:47 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:37:47.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:47 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:47.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:47 smithi028 ceph-mon[27810]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:47.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:47 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:47.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:47 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:47.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:47 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:37:47.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:47 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:37:47.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:47 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:47.892 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:47.891+0000 7f04657fa700 1 -- 172.21.15.105:0/2038316502 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0468062f30 con 0x7f04500a3cc0 2024-04-23T14:37:47.892 INFO:teuthology.orchestra.run.smithi105.stdout:Created osd(s) 5 on host 'smithi105' 2024-04-23T14:37:47.894 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:47.893+0000 7f046ea23700 1 -- 172.21.15.105:0/2038316502 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f04500a3cc0 msgr2=0x7f04500a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:47.894 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:47.893+0000 7f046ea23700 1 --2- 172.21.15.105:0/2038316502 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f04500a3cc0 0x7f04500a6180 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0468102aa0 tx=0x7f0458009470 comp rx=0 tx=0).stop 2024-04-23T14:37:47.894 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:47.893+0000 7f046ea23700 1 -- 172.21.15.105:0/2038316502 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0468102c40 msgr2=0x7f046819ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:47.895 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:47.893+0000 7f046ea23700 1 --2- 172.21.15.105:0/2038316502 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0468102c40 0x7f046819ff10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f045c005910 tx=0x7f045c009580 comp rx=0 tx=0).stop 2024-04-23T14:37:47.895 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:47.893+0000 7f046ea23700 1 -- 172.21.15.105:0/2038316502 shutdown_connections 2024-04-23T14:37:47.895 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:47.893+0000 7f046ea23700 1 --2- 172.21.15.105:0/2038316502 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f04500a3cc0 0x7f04500a6180 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:47.895 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:47.893+0000 7f046ea23700 1 --2- 172.21.15.105:0/2038316502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04681035e0 0x7f04681a2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:47.895 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:47.893+0000 7f046ea23700 1 --2- 172.21.15.105:0/2038316502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0468101a40 0x7f046806bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:47.895 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:47.893+0000 7f046ea23700 1 --2- 172.21.15.105:0/2038316502 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0468102c40 0x7f046819ff10 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:47.895 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:47.893+0000 7f046ea23700 1 -- 172.21.15.105:0/2038316502 >> 172.21.15.105:0/2038316502 conn(0x7f04680fcfe0 msgr2=0x7f0468105f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:47.895 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:47.893+0000 7f046ea23700 1 -- 172.21.15.105:0/2038316502 shutdown_connections 2024-04-23T14:37:47.895 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:37:47.893+0000 7f046ea23700 1 -- 172.21.15.105:0/2038316502 wait complete. 2024-04-23T14:37:48.511 DEBUG:teuthology.orchestra.run.smithi105:osd.5> sudo journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.5.service 2024-04-23T14:37:48.514 INFO:tasks.cephadm:Deploying osd.6 on smithi028 with /dev/vg_nvme/lv_2... 2024-04-23T14:37:48.514 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-23T14:37:48.970 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:37:49.011 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:48 smithi105 ceph-mon[32555]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:49.011 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:48 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:49.011 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:48 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:37:49.012 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:48 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:49.012 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:48 smithi105 ceph-mon[32555]: from='osd.5 [v2:172.21.15.105:6808/928755894,v1:172.21.15.105:6809/928755894]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T14:37:49.012 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:48 smithi105 ceph-mon[32555]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T14:37:49.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:48 smithi028 ceph-mon[27810]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:49.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:48 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:49.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:48 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:37:49.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:48 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:49.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:48 smithi028 ceph-mon[27810]: from='osd.5 [v2:172.21.15.105:6808/928755894,v1:172.21.15.105:6809/928755894]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T14:37:49.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:48 smithi028 ceph-mon[27810]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T14:37:49.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:48 smithi038 ceph-mon[32348]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:49.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:48 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:49.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:48 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:37:49.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:48 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:49.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:48 smithi038 ceph-mon[32348]: from='osd.5 [v2:172.21.15.105:6808/928755894,v1:172.21.15.105:6809/928755894]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T14:37:49.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:48 smithi038 ceph-mon[32348]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T14:37:49.982 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:49 smithi105 ceph-mon[32555]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-23T14:37:49.982 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:49 smithi105 ceph-mon[32555]: from='osd.5 [v2:172.21.15.105:6808/928755894,v1:172.21.15.105:6809/928755894]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-23T14:37:49.982 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:49 smithi105 ceph-mon[32555]: osdmap e36: 6 total, 5 up, 6 in 2024-04-23T14:37:49.982 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:49 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T14:37:49.983 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:49 smithi105 ceph-mon[32555]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-23T14:37:49.983 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 14:37:49 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T14:37:49.875+0000 7f7dc3a20700 -1 osd.5 0 waiting for initial osdmap 2024-04-23T14:37:49.983 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 14:37:49 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T14:37:49.881+0000 7f7dbf813700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:37:50.243 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:49 smithi028 ceph-mon[27810]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-23T14:37:50.243 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:49 smithi028 ceph-mon[27810]: from='osd.5 [v2:172.21.15.105:6808/928755894,v1:172.21.15.105:6809/928755894]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-23T14:37:50.243 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:49 smithi028 ceph-mon[27810]: osdmap e36: 6 total, 5 up, 6 in 2024-04-23T14:37:50.243 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:49 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T14:37:50.243 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:49 smithi028 ceph-mon[27810]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-23T14:37:50.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:49 smithi038 ceph-mon[32348]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-23T14:37:50.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:49 smithi038 ceph-mon[32348]: from='osd.5 [v2:172.21.15.105:6808/928755894,v1:172.21.15.105:6809/928755894]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-23T14:37:50.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:49 smithi038 ceph-mon[32348]: osdmap e36: 6 total, 5 up, 6 in 2024-04-23T14:37:50.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:49 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T14:37:50.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:49 smithi038 ceph-mon[32348]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-23T14:37:51.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:50 smithi105 ceph-mon[32555]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:51.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:50 smithi105 ceph-mon[32555]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-04-23T14:37:51.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:50 smithi105 ceph-mon[32555]: osdmap e37: 6 total, 5 up, 6 in 2024-04-23T14:37:51.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:50 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T14:37:51.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:50 smithi105 ceph-mon[32555]: osd.5 [v2:172.21.15.105:6808/928755894,v1:172.21.15.105:6809/928755894] boot 2024-04-23T14:37:51.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:50 smithi038 ceph-mon[32348]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:51.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:50 smithi038 ceph-mon[32348]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-04-23T14:37:51.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:50 smithi038 ceph-mon[32348]: osdmap e37: 6 total, 5 up, 6 in 2024-04-23T14:37:51.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:50 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T14:37:51.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:50 smithi038 ceph-mon[32348]: osd.5 [v2:172.21.15.105:6808/928755894,v1:172.21.15.105:6809/928755894] boot 2024-04-23T14:37:51.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:50 smithi028 ceph-mon[27810]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 159 MiB used, 447 GiB / 447 GiB avail 2024-04-23T14:37:51.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:50 smithi028 ceph-mon[27810]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-04-23T14:37:51.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:50 smithi028 ceph-mon[27810]: osdmap e37: 6 total, 5 up, 6 in 2024-04-23T14:37:51.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:50 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T14:37:51.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:50 smithi028 ceph-mon[27810]: osd.5 [v2:172.21.15.105:6808/928755894,v1:172.21.15.105:6809/928755894] boot 2024-04-23T14:37:51.821 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-23T14:37:51.821 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-23T14:37:51.821 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-04-23T14:37:51.821 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-04-23T14:37:51.821 INFO:teuthology.orchestra.run.smithi028.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0380864 s, 275 MB/s 2024-04-23T14:37:51.822 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-04-23T14:37:52.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:51 smithi028 ceph-mon[27810]: purged_snaps scrub starts 2024-04-23T14:37:52.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:51 smithi028 ceph-mon[27810]: purged_snaps scrub ok 2024-04-23T14:37:52.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:51 smithi028 ceph-mon[27810]: osdmap e38: 6 total, 6 up, 6 in 2024-04-23T14:37:52.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:51 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T14:37:52.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:51 smithi028 ceph-mon[27810]: osdmap e39: 6 total, 6 up, 6 in 2024-04-23T14:37:52.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:51 smithi028 ceph-mon[27810]: Detected new or changed devices on smithi105 2024-04-23T14:37:52.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:51 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:52.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:51 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:52.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:51 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:37:52.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:51 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:37:52.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:51 smithi028 ceph-mon[27810]: Adjusting osd_memory_target on smithi105 to 8544M 2024-04-23T14:37:52.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:51 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:52.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:51 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:37:52.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:51 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:37:52.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:51 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:52.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:51 smithi105 ceph-mon[32555]: purged_snaps scrub starts 2024-04-23T14:37:52.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:51 smithi105 ceph-mon[32555]: purged_snaps scrub ok 2024-04-23T14:37:52.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:51 smithi105 ceph-mon[32555]: osdmap e38: 6 total, 6 up, 6 in 2024-04-23T14:37:52.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:51 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T14:37:52.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:51 smithi105 ceph-mon[32555]: osdmap e39: 6 total, 6 up, 6 in 2024-04-23T14:37:52.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:51 smithi105 ceph-mon[32555]: Detected new or changed devices on smithi105 2024-04-23T14:37:52.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:51 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:52.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:51 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:52.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:51 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:37:52.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:51 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:37:52.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:51 smithi105 ceph-mon[32555]: Adjusting osd_memory_target on smithi105 to 8544M 2024-04-23T14:37:52.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:51 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:52.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:51 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:37:52.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:51 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:37:52.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:51 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:52.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:51 smithi038 ceph-mon[32348]: purged_snaps scrub starts 2024-04-23T14:37:52.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:51 smithi038 ceph-mon[32348]: purged_snaps scrub ok 2024-04-23T14:37:52.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:51 smithi038 ceph-mon[32348]: osdmap e38: 6 total, 6 up, 6 in 2024-04-23T14:37:52.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:51 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T14:37:52.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:51 smithi038 ceph-mon[32348]: osdmap e39: 6 total, 6 up, 6 in 2024-04-23T14:37:52.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:51 smithi038 ceph-mon[32348]: Detected new or changed devices on smithi105 2024-04-23T14:37:52.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:51 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:52.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:51 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:52.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:51 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:37:52.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:51 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:37:52.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:51 smithi038 ceph-mon[32348]: Adjusting osd_memory_target on smithi105 to 8544M 2024-04-23T14:37:52.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:51 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:52.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:51 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:37:52.374 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:51 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:37:52.374 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:51 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:37:52.793 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_2 2024-04-23T14:37:53.245 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:37:53.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:53 smithi105 ceph-mon[32555]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:37:53.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:53 smithi105 ceph-mon[32555]: osdmap e40: 6 total, 6 up, 6 in 2024-04-23T14:37:53.596 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:53 smithi028 ceph-mon[27810]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:37:53.596 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:53 smithi028 ceph-mon[27810]: osdmap e40: 6 total, 6 up, 6 in 2024-04-23T14:37:53.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:53 smithi038 ceph-mon[32348]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:37:53.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:53 smithi038 ceph-mon[32348]: osdmap e40: 6 total, 6 up, 6 in 2024-04-23T14:37:54.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:54 smithi105 ceph-mon[32555]: osdmap e41: 6 total, 6 up, 6 in 2024-04-23T14:37:54.844 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:54 smithi028 ceph-mon[27810]: osdmap e41: 6 total, 6 up, 6 in 2024-04-23T14:37:54.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:54 smithi038 ceph-mon[32348]: osdmap e41: 6 total, 6 up, 6 in 2024-04-23T14:37:55.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.310+0000 7f49d8400700 1 -- 172.21.15.28:0/290350604 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49d0109190 msgr2=0x7f49d01095b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:55.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.310+0000 7f49d8400700 1 --2- 172.21.15.28:0/290350604 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49d0109190 0x7f49d01095b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f49c40099c0 tx=0x7f49c4009cd0 comp rx=0 tx=0).stop 2024-04-23T14:37:55.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.310+0000 7f49d8400700 1 -- 172.21.15.28:0/290350604 shutdown_connections 2024-04-23T14:37:55.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.310+0000 7f49d8400700 1 --2- 172.21.15.28:0/290350604 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f49d010ad30 0x7f49d01115e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:55.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.310+0000 7f49d8400700 1 --2- 172.21.15.28:0/290350604 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49d0109190 0x7f49d01095b0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:55.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.310+0000 7f49d8400700 1 --2- 172.21.15.28:0/290350604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49d010a390 0x7f49d010a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:55.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.310+0000 7f49d8400700 1 -- 172.21.15.28:0/290350604 >> 172.21.15.28:0/290350604 conn(0x7f49d0078160 msgr2=0x7f49d007a5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:37:55.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.311+0000 7f49d8400700 1 -- 172.21.15.28:0/290350604 shutdown_connections 2024-04-23T14:37:55.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.311+0000 7f49d8400700 1 -- 172.21.15.28:0/290350604 wait complete. 2024-04-23T14:37:55.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.312+0000 7f49d8400700 1 Processor -- start 2024-04-23T14:37:55.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.313+0000 7f49d8400700 1 -- start start 2024-04-23T14:37:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.313+0000 7f49d8400700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f49d0109190 0x7f49d019f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.313+0000 7f49d8400700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49d010a390 0x7f49d019f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.313+0000 7f49d8400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49d010ad30 0x7f49d01a6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.313+0000 7f49d8400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f49d0114dc0 con 0x7f49d010ad30 2024-04-23T14:37:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.313+0000 7f49d8400700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f49d0114c40 con 0x7f49d010a390 2024-04-23T14:37:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.313+0000 7f49d8400700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f49d0114f40 con 0x7f49d0109190 2024-04-23T14:37:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.314+0000 7f49d5a00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f49d0109190 0x7f49d019f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.314+0000 7f49d5a00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f49d0109190 0x7f49d019f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:48956/0 (socket says 172.21.15.28:48956) 2024-04-23T14:37:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.314+0000 7f49d5a00700 1 -- 172.21.15.28:0/3216074003 learned_addr learned my addr 172.21.15.28:0/3216074003 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:37:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.314+0000 7f49d5000700 1 --2- 172.21.15.28:0/3216074003 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49d010a390 0x7f49d019f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.314+0000 7f49d6400700 1 --2- 172.21.15.28:0/3216074003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49d010ad30 0x7f49d01a6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:55.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.314+0000 7f49d5a00700 1 -- 172.21.15.28:0/3216074003 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49d010a390 msgr2=0x7f49d019f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:55.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.314+0000 7f49d5a00700 1 --2- 172.21.15.28:0/3216074003 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49d010a390 0x7f49d019f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:55.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.314+0000 7f49d5a00700 1 -- 172.21.15.28:0/3216074003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49d010ad30 msgr2=0x7f49d01a6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:37:55.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.314+0000 7f49d5a00700 1 --2- 172.21.15.28:0/3216074003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49d010ad30 0x7f49d01a6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:37:55.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.314+0000 7f49d5a00700 1 -- 172.21.15.28:0/3216074003 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49d01a7030 con 0x7f49d0109190 2024-04-23T14:37:55.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.314+0000 7f49d5000700 1 --2- 172.21.15.28:0/3216074003 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49d010a390 0x7f49d019f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:37:55.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.315+0000 7f49d5a00700 1 --2- 172.21.15.28:0/3216074003 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f49d0109190 0x7f49d019f3d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f49c400fa00 tx=0x7f49c400fa30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:55.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.315+0000 7f49ce000700 1 -- 172.21.15.28:0/3216074003 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c4027070 con 0x7f49d0109190 2024-04-23T14:37:55.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.315+0000 7f49d8400700 1 -- 172.21.15.28:0/3216074003 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49d01a72c0 con 0x7f49d0109190 2024-04-23T14:37:55.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.315+0000 7f49ce000700 1 -- 172.21.15.28:0/3216074003 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49c4022950 con 0x7f49d0109190 2024-04-23T14:37:55.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.315+0000 7f49ce000700 1 -- 172.21.15.28:0/3216074003 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c400fe60 con 0x7f49d0109190 2024-04-23T14:37:55.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.315+0000 7f49d8400700 1 -- 172.21.15.28:0/3216074003 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49d01a7770 con 0x7f49d0109190 2024-04-23T14:37:55.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.316+0000 7f49d8400700 1 -- 172.21.15.28:0/3216074003 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49d00697e0 con 0x7f49d0109190 2024-04-23T14:37:55.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.318+0000 7f49ce000700 1 -- 172.21.15.28:0/3216074003 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f49c402a070 con 0x7f49d0109190 2024-04-23T14:37:55.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.320+0000 7f49ce000700 1 --2- 172.21.15.28:0/3216074003 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f49b80a3c40 0x7f49b80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:37:55.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.320+0000 7f49ce000700 1 -- 172.21.15.28:0/3216074003 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 4533+0+0 (secure 0 0 0) 0x7f49c40d0290 con 0x7f49d0109190 2024-04-23T14:37:55.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.320+0000 7f49d5000700 1 --2- 172.21.15.28:0/3216074003 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f49b80a3c40 0x7f49b80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:37:55.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.321+0000 7f49d5000700 1 --2- 172.21.15.28:0/3216074003 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f49b80a3c40 0x7f49b80a6100 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f49bc011490 tx=0x7f49bc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:37:55.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.322+0000 7f49ce000700 1 -- 172.21.15.28:0/3216074003 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f49c4013070 con 0x7f49d0109190 2024-04-23T14:37:55.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:37:55.570+0000 7f49d8400700 1 -- 172.21.15.28:0/3216074003 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f49d0062f30 con 0x7f49b80a3c40 2024-04-23T14:37:55.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:55 smithi105 ceph-mon[32555]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:37:55.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:55 smithi105 ceph-mon[32555]: osdmap e42: 6 total, 6 up, 6 in 2024-04-23T14:37:55.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:55 smithi038 ceph-mon[32348]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:37:55.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:55 smithi038 ceph-mon[32348]: osdmap e42: 6 total, 6 up, 6 in 2024-04-23T14:37:55.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:55 smithi028 ceph-mon[27810]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:37:55.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:55 smithi028 ceph-mon[27810]: osdmap e42: 6 total, 6 up, 6 in 2024-04-23T14:37:56.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:56 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:37:56.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:56 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:37:56.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:56 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:37:56.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:56 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:37:56.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:56 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:37:56.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:56 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:37:56.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:56 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:37:56.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:56 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:37:56.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:56 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:37:57.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:57 smithi105 ceph-mon[32555]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:37:57.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:57 smithi105 ceph-mon[32555]: pgmap v159: 1 pgs: 1 peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 133 KiB/s, 0 objects/s recovering 2024-04-23T14:37:57.842 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:57 smithi028 ceph-mon[27810]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:37:57.842 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:57 smithi028 ceph-mon[27810]: pgmap v159: 1 pgs: 1 peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 133 KiB/s, 0 objects/s recovering 2024-04-23T14:37:57.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:57 smithi038 ceph-mon[32348]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:37:57.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:57 smithi038 ceph-mon[32348]: pgmap v159: 1 pgs: 1 peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 133 KiB/s, 0 objects/s recovering 2024-04-23T14:37:59.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:37:59 smithi105 ceph-mon[32555]: pgmap v160: 1 pgs: 1 peering; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-23T14:37:59.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:37:59 smithi038 ceph-mon[32348]: pgmap v160: 1 pgs: 1 peering; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-23T14:37:59.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:37:59 smithi028 ceph-mon[27810]: pgmap v160: 1 pgs: 1 peering; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-23T14:38:00.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:00 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3583416668' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "304db525-d24d-4687-bfa0-f92bdf66d1e1"}]: dispatch 2024-04-23T14:38:00.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:00 smithi028 ceph-mon[27810]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "304db525-d24d-4687-bfa0-f92bdf66d1e1"}]: dispatch 2024-04-23T14:38:00.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:00 smithi028 ceph-mon[27810]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "304db525-d24d-4687-bfa0-f92bdf66d1e1"}]': finished 2024-04-23T14:38:00.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:00 smithi028 ceph-mon[27810]: osdmap e43: 7 total, 6 up, 7 in 2024-04-23T14:38:00.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:00 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T14:38:00.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:00 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3583416668' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "304db525-d24d-4687-bfa0-f92bdf66d1e1"}]: dispatch 2024-04-23T14:38:00.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:00 smithi105 ceph-mon[32555]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "304db525-d24d-4687-bfa0-f92bdf66d1e1"}]: dispatch 2024-04-23T14:38:00.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:00 smithi105 ceph-mon[32555]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "304db525-d24d-4687-bfa0-f92bdf66d1e1"}]': finished 2024-04-23T14:38:00.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:00 smithi105 ceph-mon[32555]: osdmap e43: 7 total, 6 up, 7 in 2024-04-23T14:38:00.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:00 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T14:38:00.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:00 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3583416668' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "304db525-d24d-4687-bfa0-f92bdf66d1e1"}]: dispatch 2024-04-23T14:38:00.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:00 smithi038 ceph-mon[32348]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "304db525-d24d-4687-bfa0-f92bdf66d1e1"}]: dispatch 2024-04-23T14:38:00.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:00 smithi038 ceph-mon[32348]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "304db525-d24d-4687-bfa0-f92bdf66d1e1"}]': finished 2024-04-23T14:38:00.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:00 smithi038 ceph-mon[32348]: osdmap e43: 7 total, 6 up, 7 in 2024-04-23T14:38:00.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:00 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T14:38:01.774 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:01 smithi038 ceph-mon[32348]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-04-23T14:38:01.774 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:01 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1008980543' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:38:01.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:01 smithi105 ceph-mon[32555]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-04-23T14:38:01.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:01 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1008980543' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:38:01.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:01 smithi028 ceph-mon[27810]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-04-23T14:38:01.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:01 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1008980543' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:38:02.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:02 smithi038 ceph-mon[32348]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-23T14:38:02.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:02 smithi028 ceph-mon[27810]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-23T14:38:03.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:02 smithi105 ceph-mon[32555]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-23T14:38:05.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:04 smithi105 ceph-mon[32555]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:38:05.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:04 smithi038 ceph-mon[32348]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:38:05.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:04 smithi028 ceph-mon[27810]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:38:06.913 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:06 smithi028 ceph-mon[27810]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:38:06.944 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:06 smithi105 ceph-mon[32555]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:38:07.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:06 smithi038 ceph-mon[32348]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:38:08.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:08 smithi028 ceph-mon[27810]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:38:09.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:08 smithi105 ceph-mon[32555]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:38:09.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:08 smithi038 ceph-mon[32348]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:38:10.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:09 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-23T14:38:10.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:09 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:38:10.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:09 smithi105 ceph-mon[32555]: Deploying daemon osd.6 on smithi028 2024-04-23T14:38:10.048 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:09 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-23T14:38:10.048 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:09 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:38:10.048 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:09 smithi028 ceph-mon[27810]: Deploying daemon osd.6 on smithi028 2024-04-23T14:38:10.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:09 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-23T14:38:10.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:09 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:38:10.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:09 smithi038 ceph-mon[32348]: Deploying daemon osd.6 on smithi028 2024-04-23T14:38:11.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:10 smithi105 ceph-mon[32555]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:38:11.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:10 smithi028 ceph-mon[27810]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:38:11.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:10 smithi038 ceph-mon[32348]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:38:12.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:12 smithi038 ceph-mon[32348]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:38:13.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:12 smithi105 ceph-mon[32555]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:38:13.044 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:12 smithi028 ceph-mon[27810]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:38:14.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:14 smithi028 ceph-mon[27810]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:38:15.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:14 smithi105 ceph-mon[32555]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:38:15.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:14 smithi038 ceph-mon[32348]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:38:17.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:16 smithi105 ceph-mon[32555]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:38:17.093 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:16 smithi028 ceph-mon[27810]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:38:17.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:16 smithi038 ceph-mon[32348]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:38:17.967 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:17 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:38:17.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:17 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:17.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:17 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:18.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:17 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:38:18.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:17 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:18.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:17 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:18.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:17 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:38:18.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:17 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:18.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:17 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:18.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:18 smithi028 ceph-mon[27810]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:38:19.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:18 smithi105 ceph-mon[32555]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:38:19.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:18 smithi038 ceph-mon[32348]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:38:20.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:20 smithi028 ceph-mon[27810]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:38:21.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:20 smithi105 ceph-mon[32555]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:38:21.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:20 smithi038 ceph-mon[32348]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:38:22.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:38:22.119+0000 7f49ce000700 1 -- 172.21.15.28:0/3216074003 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f49d0062f30 con 0x7f49b80a3c40 2024-04-23T14:38:22.122 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 6 on host 'smithi028' 2024-04-23T14:38:22.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:38:22.123+0000 7f49d8400700 1 -- 172.21.15.28:0/3216074003 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f49b80a3c40 msgr2=0x7f49b80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:22.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:38:22.123+0000 7f49d8400700 1 --2- 172.21.15.28:0/3216074003 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f49b80a3c40 0x7f49b80a6100 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f49bc011490 tx=0x7f49bc009260 comp rx=0 tx=0).stop 2024-04-23T14:38:22.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:38:22.123+0000 7f49d8400700 1 -- 172.21.15.28:0/3216074003 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f49d0109190 msgr2=0x7f49d019f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:22.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:38:22.123+0000 7f49d8400700 1 --2- 172.21.15.28:0/3216074003 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f49d0109190 0x7f49d019f3d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f49c400fa00 tx=0x7f49c400fa30 comp rx=0 tx=0).stop 2024-04-23T14:38:22.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:38:22.123+0000 7f49d8400700 1 -- 172.21.15.28:0/3216074003 shutdown_connections 2024-04-23T14:38:22.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:38:22.123+0000 7f49d8400700 1 --2- 172.21.15.28:0/3216074003 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f49b80a3c40 0x7f49b80a6100 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:22.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:38:22.123+0000 7f49d8400700 1 --2- 172.21.15.28:0/3216074003 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f49d0109190 0x7f49d019f3d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:22.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:38:22.123+0000 7f49d8400700 1 --2- 172.21.15.28:0/3216074003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49d010ad30 0x7f49d01a6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:22.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:38:22.123+0000 7f49d8400700 1 --2- 172.21.15.28:0/3216074003 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49d010a390 0x7f49d019f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:22.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:38:22.123+0000 7f49d8400700 1 -- 172.21.15.28:0/3216074003 >> 172.21.15.28:0/3216074003 conn(0x7f49d0078160 msgr2=0x7f49d010d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:22.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:38:22.124+0000 7f49d8400700 1 -- 172.21.15.28:0/3216074003 shutdown_connections 2024-04-23T14:38:22.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:38:22.124+0000 7f49d8400700 1 -- 172.21.15.28:0/3216074003 wait complete. 2024-04-23T14:38:22.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:21 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:22.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:21 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:22.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:21 smithi105 ceph-mon[32555]: from='osd.6 [v2:172.21.15.28:6818/897979441,v1:172.21.15.28:6819/897979441]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T14:38:22.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:21 smithi105 ceph-mon[32555]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T14:38:22.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:21 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:22.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:21 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:22.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:21 smithi038 ceph-mon[32348]: from='osd.6 [v2:172.21.15.28:6818/897979441,v1:172.21.15.28:6819/897979441]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T14:38:22.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:21 smithi038 ceph-mon[32348]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T14:38:22.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:21 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:22.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:21 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:22.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:21 smithi028 ceph-mon[27810]: from='osd.6 [v2:172.21.15.28:6818/897979441,v1:172.21.15.28:6819/897979441]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T14:38:22.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:21 smithi028 ceph-mon[27810]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T14:38:22.887 DEBUG:teuthology.orchestra.run.smithi028:osd.6> sudo journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.6.service 2024-04-23T14:38:22.895 INFO:tasks.cephadm:Deploying osd.7 on smithi038 with /dev/vg_nvme/lv_2... 2024-04-23T14:38:22.895 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-23T14:38:23.057 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:23 smithi028 ceph-mon[27810]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:38:23.057 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:23 smithi028 ceph-mon[27810]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-23T14:38:23.058 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:23 smithi028 ceph-mon[27810]: osdmap e44: 7 total, 6 up, 7 in 2024-04-23T14:38:23.058 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:23 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T14:38:23.058 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:23 smithi028 ceph-mon[27810]: from='osd.6 [v2:172.21.15.28:6818/897979441,v1:172.21.15.28:6819/897979441]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-23T14:38:23.058 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:23 smithi028 ceph-mon[27810]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-23T14:38:23.058 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:23 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:23.058 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:23 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:23.058 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 14:38:23 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T14:38:23.037+0000 7fe635400700 -1 osd.6 0 waiting for initial osdmap 2024-04-23T14:38:23.058 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 14:38:23 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T14:38:23.055+0000 7fe630200700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:38:23.151 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T14:38:23.275 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:23 smithi038 ceph-mon[32348]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:38:23.275 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:23 smithi038 ceph-mon[32348]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-23T14:38:23.275 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:23 smithi038 ceph-mon[32348]: osdmap e44: 7 total, 6 up, 7 in 2024-04-23T14:38:23.275 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:23 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T14:38:23.275 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:23 smithi038 ceph-mon[32348]: from='osd.6 [v2:172.21.15.28:6818/897979441,v1:172.21.15.28:6819/897979441]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-23T14:38:23.275 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:23 smithi038 ceph-mon[32348]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-23T14:38:23.275 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:23 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:23.275 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:23 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:23.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:23 smithi105 ceph-mon[32555]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:38:23.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:23 smithi105 ceph-mon[32555]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-23T14:38:23.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:23 smithi105 ceph-mon[32555]: osdmap e44: 7 total, 6 up, 7 in 2024-04-23T14:38:23.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:23 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T14:38:23.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:23 smithi105 ceph-mon[32555]: from='osd.6 [v2:172.21.15.28:6818/897979441,v1:172.21.15.28:6819/897979441]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-23T14:38:23.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:23 smithi105 ceph-mon[32555]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-23T14:38:23.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:23 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:23.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:23 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:24.307 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:24 smithi038 ceph-mon[32348]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-23T14:38:24.308 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:24 smithi038 ceph-mon[32348]: osdmap e45: 7 total, 6 up, 7 in 2024-04-23T14:38:24.308 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:24 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T14:38:24.308 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:24 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T14:38:24.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:24 smithi028 ceph-mon[27810]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-23T14:38:24.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:24 smithi028 ceph-mon[27810]: osdmap e45: 7 total, 6 up, 7 in 2024-04-23T14:38:24.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:24 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T14:38:24.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:24 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T14:38:24.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:24 smithi105 ceph-mon[32555]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-23T14:38:24.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:24 smithi105 ceph-mon[32555]: osdmap e45: 7 total, 6 up, 7 in 2024-04-23T14:38:24.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:24 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T14:38:24.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:24 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T14:38:25.131 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:25 smithi038 ceph-mon[32348]: purged_snaps scrub starts 2024-04-23T14:38:25.132 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:25 smithi038 ceph-mon[32348]: purged_snaps scrub ok 2024-04-23T14:38:25.132 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:25 smithi038 ceph-mon[32348]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:38:25.132 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:25 smithi038 ceph-mon[32348]: osd.6 [v2:172.21.15.28:6818/897979441,v1:172.21.15.28:6819/897979441] boot 2024-04-23T14:38:25.132 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:25 smithi038 ceph-mon[32348]: osdmap e46: 7 total, 7 up, 7 in 2024-04-23T14:38:25.132 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:25 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T14:38:25.132 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-23T14:38:25.132 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-23T14:38:25.132 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-04-23T14:38:25.132 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-04-23T14:38:25.132 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0107161 s, 979 MB/s 2024-04-23T14:38:25.133 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-04-23T14:38:25.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:25 smithi028 ceph-mon[27810]: purged_snaps scrub starts 2024-04-23T14:38:25.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:25 smithi028 ceph-mon[27810]: purged_snaps scrub ok 2024-04-23T14:38:25.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:25 smithi028 ceph-mon[27810]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:38:25.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:25 smithi028 ceph-mon[27810]: osd.6 [v2:172.21.15.28:6818/897979441,v1:172.21.15.28:6819/897979441] boot 2024-04-23T14:38:25.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:25 smithi028 ceph-mon[27810]: osdmap e46: 7 total, 7 up, 7 in 2024-04-23T14:38:25.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:25 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T14:38:25.437 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:25 smithi105 ceph-mon[32555]: purged_snaps scrub starts 2024-04-23T14:38:25.437 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:25 smithi105 ceph-mon[32555]: purged_snaps scrub ok 2024-04-23T14:38:25.438 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:25 smithi105 ceph-mon[32555]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-23T14:38:25.438 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:25 smithi105 ceph-mon[32555]: osd.6 [v2:172.21.15.28:6818/897979441,v1:172.21.15.28:6819/897979441] boot 2024-04-23T14:38:25.438 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:25 smithi105 ceph-mon[32555]: osdmap e46: 7 total, 7 up, 7 in 2024-04-23T14:38:25.438 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:25 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T14:38:25.856 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi038:vg_nvme/lv_2 2024-04-23T14:38:26.107 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T14:38:26.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:26 smithi038 ceph-mon[32348]: osdmap e47: 7 total, 7 up, 7 in 2024-04-23T14:38:26.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:26 smithi028 ceph-mon[27810]: osdmap e47: 7 total, 7 up, 7 in 2024-04-23T14:38:26.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:26 smithi105 ceph-mon[32555]: osdmap e47: 7 total, 7 up, 7 in 2024-04-23T14:38:27.275 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:27 smithi038 ceph-mon[32348]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:27.365 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:27 smithi028 ceph-mon[27810]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:27.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:27 smithi105 ceph-mon[32555]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:27.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.755+0000 7f269372a700 1 -- 172.21.15.38:0/643282337 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f268c102b40 msgr2=0x7f268c102f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:27.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.755+0000 7f269372a700 1 --2- 172.21.15.38:0/643282337 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f268c102b40 0x7f268c102f60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f26880099c0 tx=0x7f2688009cd0 comp rx=0 tx=0).stop 2024-04-23T14:38:27.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.755+0000 7f269372a700 1 -- 172.21.15.38:0/643282337 shutdown_connections 2024-04-23T14:38:27.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.755+0000 7f269372a700 1 --2- 172.21.15.38:0/643282337 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f268c1046e0 0x7f268c10af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:27.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.755+0000 7f269372a700 1 --2- 172.21.15.38:0/643282337 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f268c102b40 0x7f268c102f60 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:27.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.755+0000 7f269372a700 1 --2- 172.21.15.38:0/643282337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f268c103d40 0x7f268c1041a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:27.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.755+0000 7f269372a700 1 -- 172.21.15.38:0/643282337 >> 172.21.15.38:0/643282337 conn(0x7f268c0fe0e0 msgr2=0x7f268c100520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:27.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.755+0000 7f269372a700 1 -- 172.21.15.38:0/643282337 shutdown_connections 2024-04-23T14:38:27.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.755+0000 7f269372a700 1 -- 172.21.15.38:0/643282337 wait complete. 2024-04-23T14:38:27.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.756+0000 7f269372a700 1 Processor -- start 2024-04-23T14:38:27.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.756+0000 7f269372a700 1 -- start start 2024-04-23T14:38:27.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.756+0000 7f269372a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f268c102b40 0x7f268c198ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:27.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.756+0000 7f269372a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f268c103d40 0x7f268c1993e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:27.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.756+0000 7f269372a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f268c1046e0 0x7f268c1a0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:27.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.756+0000 7f269372a700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f268c10e800 con 0x7f268c1046e0 2024-04-23T14:38:27.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.756+0000 7f269372a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f268c10e680 con 0x7f268c102b40 2024-04-23T14:38:27.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.756+0000 7f269372a700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f268c10e980 con 0x7f268c103d40 2024-04-23T14:38:27.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.756+0000 7f26914c6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f268c102b40 0x7f268c198ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:27.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.756+0000 7f26914c6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f268c102b40 0x7f268c198ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46010/0 (socket says 172.21.15.38:46010) 2024-04-23T14:38:27.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.756+0000 7f26914c6700 1 -- 172.21.15.38:0/3673938752 learned_addr learned my addr 172.21.15.38:0/3673938752 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-23T14:38:27.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.757+0000 7f2691cc7700 1 --2- 172.21.15.38:0/3673938752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f268c1046e0 0x7f268c1a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:27.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.757+0000 7f2690cc5700 1 --2- 172.21.15.38:0/3673938752 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f268c103d40 0x7f268c1993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:27.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.757+0000 7f26914c6700 1 -- 172.21.15.38:0/3673938752 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f268c103d40 msgr2=0x7f268c1993e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:27.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.757+0000 7f26914c6700 1 --2- 172.21.15.38:0/3673938752 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f268c103d40 0x7f268c1993e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:27.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.757+0000 7f26914c6700 1 -- 172.21.15.38:0/3673938752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f268c1046e0 msgr2=0x7f268c1a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:27.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.757+0000 7f26914c6700 1 --2- 172.21.15.38:0/3673938752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f268c1046e0 0x7f268c1a0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:27.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.757+0000 7f26914c6700 1 -- 172.21.15.38:0/3673938752 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f268c1a0b00 con 0x7f268c102b40 2024-04-23T14:38:27.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.757+0000 7f2691cc7700 1 --2- 172.21.15.38:0/3673938752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f268c1046e0 0x7f268c1a0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:38:27.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.757+0000 7f2690cc5700 1 --2- 172.21.15.38:0/3673938752 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f268c103d40 0x7f268c1993e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:38:27.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.757+0000 7f26914c6700 1 --2- 172.21.15.38:0/3673938752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f268c102b40 0x7f268c198ea0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f268800c010 tx=0x7f2688003680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:27.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.757+0000 7f26827fc700 1 -- 172.21.15.38:0/3673938752 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2688027070 con 0x7f268c102b40 2024-04-23T14:38:27.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.757+0000 7f26827fc700 1 -- 172.21.15.38:0/3673938752 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2688022470 con 0x7f268c102b40 2024-04-23T14:38:27.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.757+0000 7f269372a700 1 -- 172.21.15.38:0/3673938752 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f268c1a0d90 con 0x7f268c102b40 2024-04-23T14:38:27.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.757+0000 7f26827fc700 1 -- 172.21.15.38:0/3673938752 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f268800f6a0 con 0x7f268c102b40 2024-04-23T14:38:27.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.757+0000 7f269372a700 1 -- 172.21.15.38:0/3673938752 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f268c1a1280 con 0x7f268c102b40 2024-04-23T14:38:27.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.758+0000 7f269372a700 1 -- 172.21.15.38:0/3673938752 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f268c0697e0 con 0x7f268c102b40 2024-04-23T14:38:27.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.759+0000 7f26827fc700 1 -- 172.21.15.38:0/3673938752 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f268802a070 con 0x7f268c102b40 2024-04-23T14:38:27.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.760+0000 7f26827fc700 1 --2- 172.21.15.38:0/3673938752 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f26780a3cc0 0x7f26780a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:27.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.760+0000 7f26827fc700 1 -- 172.21.15.38:0/3673938752 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7f26880d0930 con 0x7f268c102b40 2024-04-23T14:38:27.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.760+0000 7f2690cc5700 1 --2- 172.21.15.38:0/3673938752 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f26780a3cc0 0x7f26780a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:27.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.761+0000 7f2690cc5700 1 --2- 172.21.15.38:0/3673938752 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f26780a3cc0 0x7f26780a6180 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f267c007900 tx=0x7f267c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:27.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.764+0000 7f26827fc700 1 -- 172.21.15.38:0/3673938752 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2688013070 con 0x7f268c102b40 2024-04-23T14:38:27.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:27.984+0000 7f269372a700 1 -- 172.21.15.38:0/3673938752 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f268c062f30 con 0x7f26780a3cc0 2024-04-23T14:38:28.940 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:28 smithi038 ceph-mon[32348]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:28.940 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:28 smithi038 ceph-mon[32348]: Detected new or changed devices on smithi028 2024-04-23T14:38:28.940 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:28 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:28.941 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:28 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:28.941 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:28 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:38:28.941 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:28 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:38:28.941 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:28 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:38:28.941 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:28 smithi038 ceph-mon[32348]: Adjusting osd_memory_target on smithi028 to 5698M 2024-04-23T14:38:28.941 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:28 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:28.941 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:28 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:38:28.941 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:28 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:38:28.941 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:28 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:28.941 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:28 smithi038 ceph-mon[32348]: from='client.24299 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:38:28.941 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:28 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:38:28.941 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:28 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:38:28.941 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:28 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:38:28.941 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:28 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:38:28.941 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:28 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:38:28.942 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:28 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:38:28.942 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:28 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:29.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:28 smithi028 ceph-mon[27810]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:29.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:28 smithi028 ceph-mon[27810]: Detected new or changed devices on smithi028 2024-04-23T14:38:29.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:29.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:29.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:38:29.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:38:29.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:38:29.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:28 smithi028 ceph-mon[27810]: Adjusting osd_memory_target on smithi028 to 5698M 2024-04-23T14:38:29.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:29.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:38:29.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:38:29.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:29.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:28 smithi028 ceph-mon[27810]: from='client.24299 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:38:29.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:38:29.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:38:29.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:38:29.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:38:29.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:38:29.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:38:29.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:28 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:29.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:28 smithi105 ceph-mon[32555]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:29.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:28 smithi105 ceph-mon[32555]: Detected new or changed devices on smithi028 2024-04-23T14:38:29.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:28 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:29.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:28 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:29.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:28 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:38:29.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:28 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:38:29.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:28 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:38:29.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:28 smithi105 ceph-mon[32555]: Adjusting osd_memory_target on smithi028 to 5698M 2024-04-23T14:38:29.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:28 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:29.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:28 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:38:29.288 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:28 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:38:29.288 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:28 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:29.288 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:28 smithi105 ceph-mon[32555]: from='client.24299 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:38:29.288 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:28 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:38:29.288 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:28 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:38:29.288 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:28 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:38:29.288 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:28 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:38:29.288 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:28 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:38:29.288 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:28 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:38:29.288 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:28 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:31.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:30 smithi038 ceph-mon[32348]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:31.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:30 smithi038 ceph-mon[32348]: from='client.? 172.21.15.38:0/2130076964' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "08e01d59-03f3-47b1-b45f-35ec5a50992c"}]: dispatch 2024-04-23T14:38:31.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:30 smithi038 ceph-mon[32348]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "08e01d59-03f3-47b1-b45f-35ec5a50992c"}]: dispatch 2024-04-23T14:38:31.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:30 smithi038 ceph-mon[32348]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "08e01d59-03f3-47b1-b45f-35ec5a50992c"}]': finished 2024-04-23T14:38:31.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:30 smithi038 ceph-mon[32348]: osdmap e48: 8 total, 7 up, 8 in 2024-04-23T14:38:31.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:30 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T14:38:31.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:30 smithi028 ceph-mon[27810]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:31.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:30 smithi028 ceph-mon[27810]: from='client.? 172.21.15.38:0/2130076964' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "08e01d59-03f3-47b1-b45f-35ec5a50992c"}]: dispatch 2024-04-23T14:38:31.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:30 smithi028 ceph-mon[27810]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "08e01d59-03f3-47b1-b45f-35ec5a50992c"}]: dispatch 2024-04-23T14:38:31.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:30 smithi028 ceph-mon[27810]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "08e01d59-03f3-47b1-b45f-35ec5a50992c"}]': finished 2024-04-23T14:38:31.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:30 smithi028 ceph-mon[27810]: osdmap e48: 8 total, 7 up, 8 in 2024-04-23T14:38:31.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:30 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T14:38:31.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:30 smithi105 ceph-mon[32555]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:31.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:30 smithi105 ceph-mon[32555]: from='client.? 172.21.15.38:0/2130076964' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "08e01d59-03f3-47b1-b45f-35ec5a50992c"}]: dispatch 2024-04-23T14:38:31.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:30 smithi105 ceph-mon[32555]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "08e01d59-03f3-47b1-b45f-35ec5a50992c"}]: dispatch 2024-04-23T14:38:31.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:30 smithi105 ceph-mon[32555]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "08e01d59-03f3-47b1-b45f-35ec5a50992c"}]': finished 2024-04-23T14:38:31.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:30 smithi105 ceph-mon[32555]: osdmap e48: 8 total, 7 up, 8 in 2024-04-23T14:38:31.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:30 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T14:38:32.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:32 smithi105 ceph-mon[32555]: from='client.? 172.21.15.38:0/2325989694' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:38:32.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:32 smithi038 ceph-mon[32348]: from='client.? 172.21.15.38:0/2325989694' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:38:32.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:32 smithi028 ceph-mon[27810]: from='client.? 172.21.15.38:0/2325989694' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:38:33.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:33 smithi105 ceph-mon[32555]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:33.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:33 smithi038 ceph-mon[32348]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:33.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:33 smithi028 ceph-mon[27810]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:35.474 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:35 smithi038 ceph-mon[32348]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:35.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:35 smithi105 ceph-mon[32555]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:35.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:35 smithi028 ceph-mon[27810]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:37.474 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:37 smithi038 ceph-mon[32348]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:37.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:37 smithi105 ceph-mon[32555]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:37.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:37 smithi028 ceph-mon[27810]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:39.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:39 smithi105 ceph-mon[32555]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:39.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:39 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-23T14:38:39.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:39 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:38:39.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:39 smithi038 ceph-mon[32348]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:39.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:39 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-23T14:38:39.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:39 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:38:39.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:39 smithi028 ceph-mon[27810]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:39.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:39 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-23T14:38:39.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:39 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:38:40.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:40 smithi105 ceph-mon[32555]: Deploying daemon osd.7 on smithi038 2024-04-23T14:38:40.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:40 smithi038 ceph-mon[32348]: Deploying daemon osd.7 on smithi038 2024-04-23T14:38:40.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:40 smithi028 ceph-mon[27810]: Deploying daemon osd.7 on smithi038 2024-04-23T14:38:41.367 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:41 smithi038 ceph-mon[32348]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:41.526 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:41 smithi028 ceph-mon[27810]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:41.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:41 smithi105 ceph-mon[32555]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:43.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:43 smithi038 ceph-mon[32348]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:43.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:43 smithi105 ceph-mon[32555]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:43.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:43 smithi028 ceph-mon[27810]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:45.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:45 smithi038 ceph-mon[32348]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:45.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:45 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:38:45.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:45 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:45.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:45 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:45.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:45 smithi105 ceph-mon[32555]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:45.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:45 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:38:45.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:45 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:45.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:45 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:45.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:45 smithi028 ceph-mon[27810]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:45.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:45 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:38:45.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:45 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:45.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:45 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:47.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:47 smithi038 ceph-mon[32348]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:47.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:47 smithi105 ceph-mon[32555]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:47.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:47 smithi028 ceph-mon[27810]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:47.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:47.762+0000 7f26827fc700 1 -- 172.21.15.38:0/3673938752 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f268c062f30 con 0x7f26780a3cc0 2024-04-23T14:38:47.763 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 7 on host 'smithi038' 2024-04-23T14:38:47.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:47.765+0000 7f269372a700 1 -- 172.21.15.38:0/3673938752 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f26780a3cc0 msgr2=0x7f26780a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:47.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:47.765+0000 7f269372a700 1 --2- 172.21.15.38:0/3673938752 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f26780a3cc0 0x7f26780a6180 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f267c007900 tx=0x7f267c008040 comp rx=0 tx=0).stop 2024-04-23T14:38:47.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:47.765+0000 7f269372a700 1 -- 172.21.15.38:0/3673938752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f268c102b40 msgr2=0x7f268c198ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:47.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:47.765+0000 7f269372a700 1 --2- 172.21.15.38:0/3673938752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f268c102b40 0x7f268c198ea0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f268800c010 tx=0x7f2688003680 comp rx=0 tx=0).stop 2024-04-23T14:38:47.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:47.765+0000 7f269372a700 1 -- 172.21.15.38:0/3673938752 shutdown_connections 2024-04-23T14:38:47.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:47.765+0000 7f269372a700 1 --2- 172.21.15.38:0/3673938752 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f26780a3cc0 0x7f26780a6180 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:47.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:47.765+0000 7f269372a700 1 --2- 172.21.15.38:0/3673938752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f268c1046e0 0x7f268c1a0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:47.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:47.765+0000 7f269372a700 1 --2- 172.21.15.38:0/3673938752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f268c102b40 0x7f268c198ea0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:47.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:47.765+0000 7f269372a700 1 --2- 172.21.15.38:0/3673938752 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f268c103d40 0x7f268c1993e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:47.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:47.765+0000 7f269372a700 1 -- 172.21.15.38:0/3673938752 >> 172.21.15.38:0/3673938752 conn(0x7f268c0fe0e0 msgr2=0x7f268c107150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:47.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:47.765+0000 7f269372a700 1 -- 172.21.15.38:0/3673938752 shutdown_connections 2024-04-23T14:38:47.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:38:47.765+0000 7f269372a700 1 -- 172.21.15.38:0/3673938752 wait complete. 2024-04-23T14:38:48.605 DEBUG:teuthology.orchestra.run.smithi038:osd.7> sudo journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.7.service 2024-04-23T14:38:48.608 INFO:tasks.cephadm:Deploying osd.8 on smithi105 with /dev/vg_nvme/lv_2... 2024-04-23T14:38:48.608 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-23T14:38:48.853 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T14:38:48.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:48 smithi038 ceph-mon[32348]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:48.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:48 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:48.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:48 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:48.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:48 smithi038 ceph-mon[32348]: from='osd.7 [v2:172.21.15.38:6816/2400251786,v1:172.21.15.38:6817/2400251786]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T14:38:48.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:48 smithi038 ceph-mon[32348]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T14:38:48.954 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:48 smithi105 ceph-mon[32555]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:48.954 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:48 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:48.954 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:48 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:48.954 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:48 smithi105 ceph-mon[32555]: from='osd.7 [v2:172.21.15.38:6816/2400251786,v1:172.21.15.38:6817/2400251786]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T14:38:48.955 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:48 smithi105 ceph-mon[32555]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T14:38:49.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:48 smithi028 ceph-mon[27810]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:49.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:48 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:49.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:48 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:49.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:48 smithi028 ceph-mon[27810]: from='osd.7 [v2:172.21.15.38:6816/2400251786,v1:172.21.15.38:6817/2400251786]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T14:38:49.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:48 smithi028 ceph-mon[27810]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T14:38:50.106 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:49 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:50.107 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:49 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:50.107 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:49 smithi105 ceph-mon[32555]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-23T14:38:50.107 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:49 smithi105 ceph-mon[32555]: from='osd.7 [v2:172.21.15.38:6816/2400251786,v1:172.21.15.38:6817/2400251786]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-23T14:38:50.107 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:49 smithi105 ceph-mon[32555]: osdmap e49: 8 total, 7 up, 8 in 2024-04-23T14:38:50.107 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:49 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T14:38:50.107 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:49 smithi105 ceph-mon[32555]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-23T14:38:50.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:49 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:50.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:49 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:50.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:49 smithi038 ceph-mon[32348]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-23T14:38:50.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:49 smithi038 ceph-mon[32348]: from='osd.7 [v2:172.21.15.38:6816/2400251786,v1:172.21.15.38:6817/2400251786]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-23T14:38:50.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:49 smithi038 ceph-mon[32348]: osdmap e49: 8 total, 7 up, 8 in 2024-04-23T14:38:50.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:49 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T14:38:50.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:49 smithi038 ceph-mon[32348]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-23T14:38:50.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:49 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:50.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:49 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:50.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:49 smithi028 ceph-mon[27810]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-23T14:38:50.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:49 smithi028 ceph-mon[27810]: from='osd.7 [v2:172.21.15.38:6816/2400251786,v1:172.21.15.38:6817/2400251786]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-23T14:38:50.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:49 smithi028 ceph-mon[27810]: osdmap e49: 8 total, 7 up, 8 in 2024-04-23T14:38:50.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:49 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T14:38:50.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:49 smithi028 ceph-mon[27810]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-23T14:38:50.790 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-23T14:38:50.790 INFO:teuthology.orchestra.run.smithi105.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-23T14:38:50.791 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10+0 records in 2024-04-23T14:38:50.791 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2024-04-23T14:38:50.791 INFO:teuthology.orchestra.run.smithi105.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0130558 s, 803 MB/s 2024-04-23T14:38:50.791 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping successful for: 2024-04-23T14:38:51.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:51 smithi038 ceph-mon[32348]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:51.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:51 smithi038 ceph-mon[32348]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-23T14:38:51.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:51 smithi038 ceph-mon[32348]: osdmap e50: 8 total, 7 up, 8 in 2024-04-23T14:38:51.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:51 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T14:38:51.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:51 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T14:38:51.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:51 smithi038 ceph-mon[32348]: from='osd.7 ' entity='osd.7' 2024-04-23T14:38:51.373 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 14:38:51 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T14:38:51.018+0000 7f705a3c2700 -1 osd.7 0 waiting for initial osdmap 2024-04-23T14:38:51.374 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 14:38:51 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T14:38:51.059+0000 7f70521ad700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:38:51.400 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:51 smithi105 ceph-mon[32555]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:51.400 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:51 smithi105 ceph-mon[32555]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-23T14:38:51.400 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:51 smithi105 ceph-mon[32555]: osdmap e50: 8 total, 7 up, 8 in 2024-04-23T14:38:51.400 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:51 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T14:38:51.400 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:51 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T14:38:51.400 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:51 smithi105 ceph-mon[32555]: from='osd.7 ' entity='osd.7' 2024-04-23T14:38:51.426 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi105:vg_nvme/lv_2 2024-04-23T14:38:51.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:51 smithi028 ceph-mon[27810]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:51.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:51 smithi028 ceph-mon[27810]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-23T14:38:51.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:51 smithi028 ceph-mon[27810]: osdmap e50: 8 total, 7 up, 8 in 2024-04-23T14:38:51.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:51 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T14:38:51.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:51 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T14:38:51.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:51 smithi028 ceph-mon[27810]: from='osd.7 ' entity='osd.7' 2024-04-23T14:38:51.668 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T14:38:52.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:52 smithi105 ceph-mon[32555]: purged_snaps scrub starts 2024-04-23T14:38:52.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:52 smithi105 ceph-mon[32555]: purged_snaps scrub ok 2024-04-23T14:38:52.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:52 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T14:38:52.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:52 smithi105 ceph-mon[32555]: osd.7 [v2:172.21.15.38:6816/2400251786,v1:172.21.15.38:6817/2400251786] boot 2024-04-23T14:38:52.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:52 smithi105 ceph-mon[32555]: osdmap e51: 8 total, 8 up, 8 in 2024-04-23T14:38:52.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:52 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T14:38:52.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:52 smithi105 ceph-mon[32555]: osdmap e52: 8 total, 8 up, 8 in 2024-04-23T14:38:52.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:52 smithi038 ceph-mon[32348]: purged_snaps scrub starts 2024-04-23T14:38:52.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:52 smithi038 ceph-mon[32348]: purged_snaps scrub ok 2024-04-23T14:38:52.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:52 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T14:38:52.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:52 smithi038 ceph-mon[32348]: osd.7 [v2:172.21.15.38:6816/2400251786,v1:172.21.15.38:6817/2400251786] boot 2024-04-23T14:38:52.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:52 smithi038 ceph-mon[32348]: osdmap e51: 8 total, 8 up, 8 in 2024-04-23T14:38:52.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:52 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T14:38:52.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:52 smithi038 ceph-mon[32348]: osdmap e52: 8 total, 8 up, 8 in 2024-04-23T14:38:52.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:52 smithi028 ceph-mon[27810]: purged_snaps scrub starts 2024-04-23T14:38:52.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:52 smithi028 ceph-mon[27810]: purged_snaps scrub ok 2024-04-23T14:38:52.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:52 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T14:38:52.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:52 smithi028 ceph-mon[27810]: osd.7 [v2:172.21.15.38:6816/2400251786,v1:172.21.15.38:6817/2400251786] boot 2024-04-23T14:38:52.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:52 smithi028 ceph-mon[27810]: osdmap e51: 8 total, 8 up, 8 in 2024-04-23T14:38:52.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:52 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T14:38:52.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:52 smithi028 ceph-mon[27810]: osdmap e52: 8 total, 8 up, 8 in 2024-04-23T14:38:53.180 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.177+0000 7f2a4798c700 1 -- 172.21.15.105:0/4023502350 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2a40103d60 msgr2=0x7f2a401041c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:53.180 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.177+0000 7f2a4798c700 1 --2- 172.21.15.105:0/4023502350 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2a40103d60 0x7f2a401041c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2a30009990 tx=0x7f2a30009ca0 comp rx=0 tx=0).stop 2024-04-23T14:38:53.180 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.177+0000 7f2a4798c700 1 -- 172.21.15.105:0/4023502350 shutdown_connections 2024-04-23T14:38:53.180 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.177+0000 7f2a4798c700 1 --2- 172.21.15.105:0/4023502350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a40104700 0x7f2a4010afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:53.180 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.177+0000 7f2a4798c700 1 --2- 172.21.15.105:0/4023502350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a40102b60 0x7f2a40102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:53.180 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.177+0000 7f2a4798c700 1 --2- 172.21.15.105:0/4023502350 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2a40103d60 0x7f2a401041c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:53.180 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.177+0000 7f2a4798c700 1 -- 172.21.15.105:0/4023502350 >> 172.21.15.105:0/4023502350 conn(0x7f2a400fe100 msgr2=0x7f2a40100540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:38:53.181 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.177+0000 7f2a4798c700 1 -- 172.21.15.105:0/4023502350 shutdown_connections 2024-04-23T14:38:53.181 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.177+0000 7f2a4798c700 1 -- 172.21.15.105:0/4023502350 wait complete. 2024-04-23T14:38:53.181 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.177+0000 7f2a4798c700 1 Processor -- start 2024-04-23T14:38:53.181 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.178+0000 7f2a4798c700 1 -- start start 2024-04-23T14:38:53.181 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.178+0000 7f2a4798c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2a40102b60 0x7f2a40198da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:53.181 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.178+0000 7f2a4798c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a40103d60 0x7f2a401992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:53.181 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.178+0000 7f2a4798c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a40104700 0x7f2a401a0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:53.181 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.178+0000 7f2a4798c700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2a4010e7e0 con 0x7f2a40103d60 2024-04-23T14:38:53.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.178+0000 7f2a4798c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2a4010e660 con 0x7f2a40104700 2024-04-23T14:38:53.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.178+0000 7f2a4798c700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2a4010e960 con 0x7f2a40102b60 2024-04-23T14:38:53.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.178+0000 7f2a45728700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2a40102b60 0x7f2a40198da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:53.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.178+0000 7f2a45728700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2a40102b60 0x7f2a40198da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:58436/0 (socket says 172.21.15.105:58436) 2024-04-23T14:38:53.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.178+0000 7f2a45728700 1 -- 172.21.15.105:0/504628797 learned_addr learned my addr 172.21.15.105:0/504628797 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-23T14:38:53.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.178+0000 7f2a45f29700 1 --2- 172.21.15.105:0/504628797 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a40104700 0x7f2a401a0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:53.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.178+0000 7f2a45728700 1 -- 172.21.15.105:0/504628797 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a40104700 msgr2=0x7f2a401a0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:53.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.178+0000 7f2a45728700 1 --2- 172.21.15.105:0/504628797 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a40104700 0x7f2a401a0360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:53.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.178+0000 7f2a45728700 1 -- 172.21.15.105:0/504628797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a40103d60 msgr2=0x7f2a401992e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:38:53.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.178+0000 7f2a45728700 1 --2- 172.21.15.105:0/504628797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a40103d60 0x7f2a401992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:38:53.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.178+0000 7f2a45728700 1 -- 172.21.15.105:0/504628797 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a401a0a00 con 0x7f2a40102b60 2024-04-23T14:38:53.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.178+0000 7f2a45728700 1 --2- 172.21.15.105:0/504628797 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2a40102b60 0x7f2a40198da0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c00b9f0 tx=0x7f2a3c00bdb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:53.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.178+0000 7f2a367fc700 1 -- 172.21.15.105:0/504628797 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c019070 con 0x7f2a40102b60 2024-04-23T14:38:53.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.178+0000 7f2a367fc700 1 -- 172.21.15.105:0/504628797 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a3c014440 con 0x7f2a40102b60 2024-04-23T14:38:53.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.178+0000 7f2a4798c700 1 -- 172.21.15.105:0/504628797 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a401a0cf0 con 0x7f2a40102b60 2024-04-23T14:38:53.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.179+0000 7f2a367fc700 1 -- 172.21.15.105:0/504628797 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c0125a0 con 0x7f2a40102b60 2024-04-23T14:38:53.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.179+0000 7f2a4798c700 1 -- 172.21.15.105:0/504628797 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a401a1200 con 0x7f2a40102b60 2024-04-23T14:38:53.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.180+0000 7f2a4798c700 1 -- 172.21.15.105:0/504628797 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a4002dfe0 con 0x7f2a40102b60 2024-04-23T14:38:53.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.181+0000 7f2a367fc700 1 -- 172.21.15.105:0/504628797 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2a3c01c070 con 0x7f2a40102b60 2024-04-23T14:38:53.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.181+0000 7f2a367fc700 1 --2- 172.21.15.105:0/504628797 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2a2c0a3c40 0x7f2a2c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:38:53.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.181+0000 7f2a367fc700 1 -- 172.21.15.105:0/504628797 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5426+0+0 (secure 0 0 0) 0x7f2a3c0cf180 con 0x7f2a40102b60 2024-04-23T14:38:53.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.182+0000 7f2a44f27700 1 --2- 172.21.15.105:0/504628797 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2a2c0a3c40 0x7f2a2c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:38:53.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.183+0000 7f2a44f27700 1 --2- 172.21.15.105:0/504628797 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2a2c0a3c40 0x7f2a2c0a6100 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2a30005f50 tx=0x7f2a30005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:38:53.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.185+0000 7f2a367fc700 1 -- 172.21.15.105:0/504628797 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2a3c09c6a0 con 0x7f2a40102b60 2024-04-23T14:38:53.395 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:38:53.394+0000 7f2a4798c700 1 -- 172.21.15.105:0/504628797 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f2a40062f30 con 0x7f2a2c0a3c40 2024-04-23T14:38:53.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:53 smithi105 ceph-mon[32555]: pgmap v197: 1 pgs: 1 unknown; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:53.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:53 smithi105 ceph-mon[32555]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-04-23T14:38:53.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:53 smithi105 ceph-mon[32555]: Detected new or changed devices on smithi038 2024-04-23T14:38:53.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:53 smithi105 ceph-mon[32555]: osdmap e53: 8 total, 8 up, 8 in 2024-04-23T14:38:53.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:53 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:53.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:53 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:53.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:53 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:38:53.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:53 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:38:53.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:53 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:38:53.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:53 smithi105 ceph-mon[32555]: Adjusting osd_memory_target on smithi038 to 5697M 2024-04-23T14:38:53.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:53 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:53.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:53 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:38:53.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:53 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:38:53.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:53 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:53.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:53 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:38:53.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:53 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:38:53.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:53 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:38:53.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:53 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:53.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:53 smithi038 ceph-mon[32348]: pgmap v197: 1 pgs: 1 unknown; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:53.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:53 smithi038 ceph-mon[32348]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-04-23T14:38:53.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:53 smithi038 ceph-mon[32348]: Detected new or changed devices on smithi038 2024-04-23T14:38:53.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:53 smithi038 ceph-mon[32348]: osdmap e53: 8 total, 8 up, 8 in 2024-04-23T14:38:53.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:53 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:53.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:53 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:53.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:53 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:38:53.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:53 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:38:53.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:53 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:38:53.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:53 smithi038 ceph-mon[32348]: Adjusting osd_memory_target on smithi038 to 5697M 2024-04-23T14:38:53.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:53 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:53.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:53 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:38:53.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:53 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:38:53.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:53 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:53.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:53 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:38:53.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:53 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:38:53.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:53 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:38:53.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:53 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:53.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:53 smithi028 ceph-mon[27810]: pgmap v197: 1 pgs: 1 unknown; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-23T14:38:53.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:53 smithi028 ceph-mon[27810]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-04-23T14:38:53.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:53 smithi028 ceph-mon[27810]: Detected new or changed devices on smithi038 2024-04-23T14:38:53.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:53 smithi028 ceph-mon[27810]: osdmap e53: 8 total, 8 up, 8 in 2024-04-23T14:38:53.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:53 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:53.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:53 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:53.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:53 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:38:53.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:53 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:38:53.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:53 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:38:53.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:53 smithi028 ceph-mon[27810]: Adjusting osd_memory_target on smithi038 to 5697M 2024-04-23T14:38:53.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:53 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:53.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:53 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:38:53.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:53 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:38:53.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:53 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:53.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:53 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:38:53.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:53 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:38:53.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:53 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:38:53.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:53 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:38:54.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:54 smithi028 ceph-mon[27810]: osdmap e54: 8 total, 8 up, 8 in 2024-04-23T14:38:54.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:54 smithi028 ceph-mon[27810]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:38:54.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:54 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:38:54.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:54 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:38:54.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:54 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:38:54.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:54 smithi105 ceph-mon[32555]: osdmap e54: 8 total, 8 up, 8 in 2024-04-23T14:38:54.789 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:54 smithi105 ceph-mon[32555]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:38:54.789 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:54 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:38:54.789 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:54 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:38:54.790 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:54 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:38:54.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:54 smithi038 ceph-mon[32348]: osdmap e54: 8 total, 8 up, 8 in 2024-04-23T14:38:54.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:54 smithi038 ceph-mon[32348]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:38:54.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:54 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:38:54.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:54 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:38:54.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:54 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:38:55.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:55 smithi105 ceph-mon[32555]: pgmap v200: 1 pgs: 1 unknown; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:38:55.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:55 smithi105 ceph-mon[32555]: osdmap e55: 8 total, 8 up, 8 in 2024-04-23T14:38:55.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:55 smithi038 ceph-mon[32348]: pgmap v200: 1 pgs: 1 unknown; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:38:55.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:55 smithi038 ceph-mon[32348]: osdmap e55: 8 total, 8 up, 8 in 2024-04-23T14:38:55.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:55 smithi028 ceph-mon[27810]: pgmap v200: 1 pgs: 1 unknown; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:38:55.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:55 smithi028 ceph-mon[27810]: osdmap e55: 8 total, 8 up, 8 in 2024-04-23T14:38:56.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:56 smithi105 ceph-mon[32555]: from='client.? 172.21.15.105:0/3632909644' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "64ee1e95-0756-48b1-9c00-472d0c3cbd29"}]: dispatch 2024-04-23T14:38:56.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:56 smithi105 ceph-mon[32555]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "64ee1e95-0756-48b1-9c00-472d0c3cbd29"}]: dispatch 2024-04-23T14:38:56.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:56 smithi105 ceph-mon[32555]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "64ee1e95-0756-48b1-9c00-472d0c3cbd29"}]': finished 2024-04-23T14:38:56.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:56 smithi105 ceph-mon[32555]: osdmap e56: 9 total, 8 up, 9 in 2024-04-23T14:38:56.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:56 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T14:38:56.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:56 smithi038 ceph-mon[32348]: from='client.? 172.21.15.105:0/3632909644' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "64ee1e95-0756-48b1-9c00-472d0c3cbd29"}]: dispatch 2024-04-23T14:38:56.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:56 smithi038 ceph-mon[32348]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "64ee1e95-0756-48b1-9c00-472d0c3cbd29"}]: dispatch 2024-04-23T14:38:56.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:56 smithi038 ceph-mon[32348]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "64ee1e95-0756-48b1-9c00-472d0c3cbd29"}]': finished 2024-04-23T14:38:56.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:56 smithi038 ceph-mon[32348]: osdmap e56: 9 total, 8 up, 9 in 2024-04-23T14:38:56.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:56 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T14:38:56.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:56 smithi028 ceph-mon[27810]: from='client.? 172.21.15.105:0/3632909644' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "64ee1e95-0756-48b1-9c00-472d0c3cbd29"}]: dispatch 2024-04-23T14:38:56.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:56 smithi028 ceph-mon[27810]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "64ee1e95-0756-48b1-9c00-472d0c3cbd29"}]: dispatch 2024-04-23T14:38:56.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:56 smithi028 ceph-mon[27810]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "64ee1e95-0756-48b1-9c00-472d0c3cbd29"}]': finished 2024-04-23T14:38:56.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:56 smithi028 ceph-mon[27810]: osdmap e56: 9 total, 8 up, 9 in 2024-04-23T14:38:56.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:56 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T14:38:57.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:57 smithi105 ceph-mon[32555]: pgmap v202: 1 pgs: 1 unknown; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:38:57.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:57 smithi105 ceph-mon[32555]: from='client.? 172.21.15.105:0/2673659149' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:38:57.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:57 smithi038 ceph-mon[32348]: pgmap v202: 1 pgs: 1 unknown; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:38:57.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:57 smithi038 ceph-mon[32348]: from='client.? 172.21.15.105:0/2673659149' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:38:57.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:57 smithi028 ceph-mon[27810]: pgmap v202: 1 pgs: 1 unknown; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:38:57.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:57 smithi028 ceph-mon[27810]: from='client.? 172.21.15.105:0/2673659149' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:38:58.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:58 smithi105 ceph-mon[32555]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-04-23T14:38:58.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:58 smithi105 ceph-mon[32555]: Cluster is now healthy 2024-04-23T14:38:58.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:58 smithi038 ceph-mon[32348]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-04-23T14:38:58.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:58 smithi038 ceph-mon[32348]: Cluster is now healthy 2024-04-23T14:38:58.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:58 smithi028 ceph-mon[27810]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-04-23T14:38:58.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:58 smithi028 ceph-mon[27810]: Cluster is now healthy 2024-04-23T14:38:59.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:38:59 smithi105 ceph-mon[32555]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:38:59.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:38:59 smithi038 ceph-mon[32348]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:38:59.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:38:59 smithi028 ceph-mon[27810]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:39:01.636 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:01 smithi105 ceph-mon[32555]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:39:01.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:01 smithi038 ceph-mon[32348]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:39:01.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:01 smithi028 ceph-mon[27810]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:39:03.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:03 smithi105 ceph-mon[32555]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:39:03.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:03 smithi038 ceph-mon[32348]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:39:03.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:03 smithi028 ceph-mon[27810]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:39:04.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:04 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-23T14:39:04.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:04 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:39:04.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:04 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-23T14:39:04.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:04 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:39:04.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:04 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-23T14:39:04.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:04 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:39:05.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:05 smithi105 ceph-mon[32555]: Deploying daemon osd.8 on smithi105 2024-04-23T14:39:05.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:05 smithi105 ceph-mon[32555]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:39:05.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:05 smithi038 ceph-mon[32348]: Deploying daemon osd.8 on smithi105 2024-04-23T14:39:05.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:05 smithi038 ceph-mon[32348]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:39:05.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:05 smithi028 ceph-mon[27810]: Deploying daemon osd.8 on smithi105 2024-04-23T14:39:05.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:05 smithi028 ceph-mon[27810]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:39:07.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:07 smithi105 ceph-mon[32555]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:39:07.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:07 smithi038 ceph-mon[32348]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:39:07.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:07 smithi028 ceph-mon[27810]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:39:09.753 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:09 smithi105 ceph-mon[32555]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:39:09.753 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:09 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:09.753 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:09 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:09.754 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:09 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:39:09.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:09 smithi038 ceph-mon[32348]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:39:09.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:09 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:09.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:09 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:09.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:09 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:39:09.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:09 smithi028 ceph-mon[27810]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:39:09.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:09 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:09.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:09 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:09.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:09 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:39:11.741 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:11 smithi105 ceph-mon[32555]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:39:11.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:11 smithi038 ceph-mon[32348]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:39:11.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:11 smithi028 ceph-mon[27810]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:39:12.788 INFO:teuthology.orchestra.run.smithi105.stdout:Created osd(s) 8 on host 'smithi105' 2024-04-23T14:39:12.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:39:12.787+0000 7f2a367fc700 1 -- 172.21.15.105:0/504628797 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2a40062f30 con 0x7f2a2c0a3c40 2024-04-23T14:39:12.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:39:12.789+0000 7f2a4798c700 1 -- 172.21.15.105:0/504628797 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2a2c0a3c40 msgr2=0x7f2a2c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:12.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:39:12.789+0000 7f2a4798c700 1 --2- 172.21.15.105:0/504628797 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2a2c0a3c40 0x7f2a2c0a6100 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2a30005f50 tx=0x7f2a30005e80 comp rx=0 tx=0).stop 2024-04-23T14:39:12.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:39:12.789+0000 7f2a4798c700 1 -- 172.21.15.105:0/504628797 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2a40102b60 msgr2=0x7f2a40198da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:12.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:39:12.789+0000 7f2a4798c700 1 --2- 172.21.15.105:0/504628797 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2a40102b60 0x7f2a40198da0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c00b9f0 tx=0x7f2a3c00bdb0 comp rx=0 tx=0).stop 2024-04-23T14:39:12.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:39:12.789+0000 7f2a4798c700 1 -- 172.21.15.105:0/504628797 shutdown_connections 2024-04-23T14:39:12.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:39:12.789+0000 7f2a4798c700 1 --2- 172.21.15.105:0/504628797 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2a2c0a3c40 0x7f2a2c0a6100 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:12.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:39:12.789+0000 7f2a4798c700 1 --2- 172.21.15.105:0/504628797 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2a40102b60 0x7f2a40198da0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:12.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:39:12.789+0000 7f2a4798c700 1 --2- 172.21.15.105:0/504628797 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a40104700 0x7f2a401a0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:12.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:39:12.789+0000 7f2a4798c700 1 --2- 172.21.15.105:0/504628797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a40103d60 0x7f2a401992e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:12.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:39:12.789+0000 7f2a4798c700 1 -- 172.21.15.105:0/504628797 >> 172.21.15.105:0/504628797 conn(0x7f2a400fe100 msgr2=0x7f2a40107000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:12.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:39:12.789+0000 7f2a4798c700 1 -- 172.21.15.105:0/504628797 shutdown_connections 2024-04-23T14:39:12.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:39:12.789+0000 7f2a4798c700 1 -- 172.21.15.105:0/504628797 wait complete. 2024-04-23T14:39:13.025 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:12 smithi105 ceph-mon[32555]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:39:13.025 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:12 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:13.025 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:12 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:13.025 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:12 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:39:13.025 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:12 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:39:13.025 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:12 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:13.025 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:12 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:13.025 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:12 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:39:13.025 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:12 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:13.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:12 smithi038 ceph-mon[32348]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:39:13.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:12 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:13.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:12 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:13.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:12 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:39:13.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:12 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:39:13.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:12 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:13.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:12 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:13.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:12 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:39:13.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:12 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:13.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:12 smithi028 ceph-mon[27810]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:39:13.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:12 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:13.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:12 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:13.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:12 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:39:13.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:12 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:39:13.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:12 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:13.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:12 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:13.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:12 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:39:13.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:12 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:13.461 DEBUG:teuthology.orchestra.run.smithi105:osd.8> sudo journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.8.service 2024-04-23T14:39:13.464 INFO:tasks.cephadm:Deploying osd.9 on smithi028 with /dev/vg_nvme/lv_1... 2024-04-23T14:39:13.464 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-23T14:39:13.920 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:39:14.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:13 smithi105 ceph-mon[32555]: from='osd.8 [v2:172.21.15.105:6816/1682312269,v1:172.21.15.105:6817/1682312269]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T14:39:14.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:13 smithi105 ceph-mon[32555]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T14:39:14.232 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:13 smithi028 ceph-mon[27810]: from='osd.8 [v2:172.21.15.105:6816/1682312269,v1:172.21.15.105:6817/1682312269]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T14:39:14.232 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:13 smithi028 ceph-mon[27810]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T14:39:14.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:13 smithi038 ceph-mon[32348]: from='osd.8 [v2:172.21.15.105:6816/1682312269,v1:172.21.15.105:6817/1682312269]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T14:39:14.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:13 smithi038 ceph-mon[32348]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T14:39:15.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:14 smithi105 ceph-mon[32555]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:39:15.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:14 smithi105 ceph-mon[32555]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-23T14:39:15.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:14 smithi105 ceph-mon[32555]: from='osd.8 [v2:172.21.15.105:6816/1682312269,v1:172.21.15.105:6817/1682312269]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-23T14:39:15.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:14 smithi105 ceph-mon[32555]: osdmap e57: 9 total, 8 up, 9 in 2024-04-23T14:39:15.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:14 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T14:39:15.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:14 smithi105 ceph-mon[32555]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-23T14:39:15.287 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 14:39:14 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T14:39:14.968+0000 7feb25d6e700 -1 osd.8 0 waiting for initial osdmap 2024-04-23T14:39:15.287 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 14:39:14 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T14:39:14.979+0000 7feb21b61700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:39:15.335 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:14 smithi028 ceph-mon[27810]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:39:15.335 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:14 smithi028 ceph-mon[27810]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-23T14:39:15.335 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:14 smithi028 ceph-mon[27810]: from='osd.8 [v2:172.21.15.105:6816/1682312269,v1:172.21.15.105:6817/1682312269]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-23T14:39:15.336 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:14 smithi028 ceph-mon[27810]: osdmap e57: 9 total, 8 up, 9 in 2024-04-23T14:39:15.336 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:14 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T14:39:15.336 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:14 smithi028 ceph-mon[27810]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-23T14:39:15.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:14 smithi038 ceph-mon[32348]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:39:15.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:14 smithi038 ceph-mon[32348]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-23T14:39:15.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:14 smithi038 ceph-mon[32348]: from='osd.8 [v2:172.21.15.105:6816/1682312269,v1:172.21.15.105:6817/1682312269]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-23T14:39:15.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:14 smithi038 ceph-mon[32348]: osdmap e57: 9 total, 8 up, 9 in 2024-04-23T14:39:15.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:14 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T14:39:15.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:14 smithi038 ceph-mon[32348]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-23T14:39:16.262 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:15 smithi105 ceph-mon[32555]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-04-23T14:39:16.263 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:15 smithi105 ceph-mon[32555]: osdmap e58: 9 total, 8 up, 9 in 2024-04-23T14:39:16.263 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:15 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T14:39:16.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:15 smithi038 ceph-mon[32348]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-04-23T14:39:16.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:15 smithi038 ceph-mon[32348]: osdmap e58: 9 total, 8 up, 9 in 2024-04-23T14:39:16.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:15 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T14:39:16.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:15 smithi028 ceph-mon[27810]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-04-23T14:39:16.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:15 smithi028 ceph-mon[27810]: osdmap e58: 9 total, 8 up, 9 in 2024-04-23T14:39:16.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:15 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T14:39:16.885 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-23T14:39:16.885 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-23T14:39:16.885 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-04-23T14:39:16.885 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-04-23T14:39:16.886 INFO:teuthology.orchestra.run.smithi028.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0439624 s, 239 MB/s 2024-04-23T14:39:16.886 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-04-23T14:39:17.083 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:16 smithi028 ceph-mon[27810]: purged_snaps scrub starts 2024-04-23T14:39:17.083 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:16 smithi028 ceph-mon[27810]: purged_snaps scrub ok 2024-04-23T14:39:17.083 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:16 smithi028 ceph-mon[27810]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:39:17.083 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:16 smithi028 ceph-mon[27810]: osd.8 [v2:172.21.15.105:6816/1682312269,v1:172.21.15.105:6817/1682312269] boot 2024-04-23T14:39:17.083 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:16 smithi028 ceph-mon[27810]: osdmap e59: 9 total, 9 up, 9 in 2024-04-23T14:39:17.083 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:16 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T14:39:17.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:16 smithi028 ceph-mon[27810]: osdmap e60: 9 total, 9 up, 9 in 2024-04-23T14:39:17.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:16 smithi028 ceph-mon[27810]: Detected new or changed devices on smithi105 2024-04-23T14:39:17.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:16 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:17.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:16 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:17.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:16 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:39:17.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:16 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:39:17.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:16 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:39:17.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:16 smithi028 ceph-mon[27810]: Adjusting osd_memory_target on smithi105 to 5696M 2024-04-23T14:39:17.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:16 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:17.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:16 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:39:17.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:16 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:39:17.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:16 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:17.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:16 smithi105 ceph-mon[32555]: purged_snaps scrub starts 2024-04-23T14:39:17.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:16 smithi105 ceph-mon[32555]: purged_snaps scrub ok 2024-04-23T14:39:17.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:16 smithi105 ceph-mon[32555]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:39:17.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:16 smithi105 ceph-mon[32555]: osd.8 [v2:172.21.15.105:6816/1682312269,v1:172.21.15.105:6817/1682312269] boot 2024-04-23T14:39:17.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:16 smithi105 ceph-mon[32555]: osdmap e59: 9 total, 9 up, 9 in 2024-04-23T14:39:17.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:16 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T14:39:17.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:16 smithi105 ceph-mon[32555]: osdmap e60: 9 total, 9 up, 9 in 2024-04-23T14:39:17.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:16 smithi105 ceph-mon[32555]: Detected new or changed devices on smithi105 2024-04-23T14:39:17.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:16 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:17.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:16 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:17.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:16 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:39:17.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:16 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:39:17.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:16 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:39:17.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:16 smithi105 ceph-mon[32555]: Adjusting osd_memory_target on smithi105 to 5696M 2024-04-23T14:39:17.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:16 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:17.288 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:16 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:39:17.288 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:16 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:39:17.288 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:16 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:17.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:16 smithi038 ceph-mon[32348]: purged_snaps scrub starts 2024-04-23T14:39:17.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:16 smithi038 ceph-mon[32348]: purged_snaps scrub ok 2024-04-23T14:39:17.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:16 smithi038 ceph-mon[32348]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-23T14:39:17.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:16 smithi038 ceph-mon[32348]: osd.8 [v2:172.21.15.105:6816/1682312269,v1:172.21.15.105:6817/1682312269] boot 2024-04-23T14:39:17.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:16 smithi038 ceph-mon[32348]: osdmap e59: 9 total, 9 up, 9 in 2024-04-23T14:39:17.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:16 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T14:39:17.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:16 smithi038 ceph-mon[32348]: osdmap e60: 9 total, 9 up, 9 in 2024-04-23T14:39:17.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:16 smithi038 ceph-mon[32348]: Detected new or changed devices on smithi105 2024-04-23T14:39:17.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:16 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:17.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:16 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:17.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:16 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:39:17.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:16 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:39:17.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:16 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:39:17.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:16 smithi038 ceph-mon[32348]: Adjusting osd_memory_target on smithi105 to 5696M 2024-04-23T14:39:17.374 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:16 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:17.374 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:16 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:39:17.374 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:16 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:39:17.374 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:16 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:17.863 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_1 2024-04-23T14:39:18.312 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:39:18.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:18 smithi038 ceph-mon[32348]: osdmap e61: 9 total, 9 up, 9 in 2024-04-23T14:39:18.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:18 smithi028 ceph-mon[27810]: osdmap e61: 9 total, 9 up, 9 in 2024-04-23T14:39:18.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:18 smithi105 ceph-mon[32555]: osdmap e61: 9 total, 9 up, 9 in 2024-04-23T14:39:19.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:19 smithi038 ceph-mon[32348]: pgmap v219: 1 pgs: 1 remapped+peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:19.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:19 smithi038 ceph-mon[32348]: osdmap e62: 9 total, 9 up, 9 in 2024-04-23T14:39:19.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:19 smithi028 ceph-mon[27810]: pgmap v219: 1 pgs: 1 remapped+peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:19.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:19 smithi028 ceph-mon[27810]: osdmap e62: 9 total, 9 up, 9 in 2024-04-23T14:39:19.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:19 smithi105 ceph-mon[32555]: pgmap v219: 1 pgs: 1 remapped+peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:19.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:19 smithi105 ceph-mon[32555]: osdmap e62: 9 total, 9 up, 9 in 2024-04-23T14:39:20.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.524+0000 7f3deca00700 1 -- 172.21.15.28:0/1332353230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3de8104720 msgr2=0x7f3de810afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:20.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.524+0000 7f3deca00700 1 --2- 172.21.15.28:0/1332353230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3de8104720 0x7f3de810afd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3de000b0d0 tx=0x7f3de000b3e0 comp rx=0 tx=0).stop 2024-04-23T14:39:20.527 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:20 smithi028 ceph-mon[27810]: osdmap e63: 9 total, 9 up, 9 in 2024-04-23T14:39:20.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.524+0000 7f3deca00700 1 -- 172.21.15.28:0/1332353230 shutdown_connections 2024-04-23T14:39:20.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.524+0000 7f3deca00700 1 --2- 172.21.15.28:0/1332353230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3de8104720 0x7f3de810afd0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:20.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.524+0000 7f3deca00700 1 --2- 172.21.15.28:0/1332353230 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3de8102b80 0x7f3de8102fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:20.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.524+0000 7f3deca00700 1 --2- 172.21.15.28:0/1332353230 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3de8103d80 0x7f3de81041e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:20.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.524+0000 7f3deca00700 1 -- 172.21.15.28:0/1332353230 >> 172.21.15.28:0/1332353230 conn(0x7f3de80fe140 msgr2=0x7f3de8100560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:20.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.524+0000 7f3deca00700 1 -- 172.21.15.28:0/1332353230 shutdown_connections 2024-04-23T14:39:20.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.524+0000 7f3deca00700 1 -- 172.21.15.28:0/1332353230 wait complete. 2024-04-23T14:39:20.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.526+0000 7f3deca00700 1 Processor -- start 2024-04-23T14:39:20.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.526+0000 7f3deca00700 1 -- start start 2024-04-23T14:39:20.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.526+0000 7f3deca00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3de8102b80 0x7f3de819aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:20.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.526+0000 7f3deca00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3de8103d80 0x7f3de819b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.526+0000 7f3deca00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3de8104720 0x7f3de81a24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.526+0000 7f3deca00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3de810e7a0 con 0x7f3de8104720 2024-04-23T14:39:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.526+0000 7f3deca00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3de810e620 con 0x7f3de8103d80 2024-04-23T14:39:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.526+0000 7f3deca00700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3de810e920 con 0x7f3de8102b80 2024-04-23T14:39:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.527+0000 7f3ddd400700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3de8103d80 0x7f3de819b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.527+0000 7f3de6800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3de8104720 0x7f3de81a24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.527+0000 7f3ddd400700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3de8103d80 0x7f3de819b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48602/0 (socket says 172.21.15.28:48602) 2024-04-23T14:39:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.527+0000 7f3de5e00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3de8102b80 0x7f3de819aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.527+0000 7f3ddd400700 1 -- 172.21.15.28:0/1878854578 learned_addr learned my addr 172.21.15.28:0/1878854578 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:39:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.527+0000 7f3de6800700 1 -- 172.21.15.28:0/1878854578 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3de8102b80 msgr2=0x7f3de819aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.527+0000 7f3de6800700 1 --2- 172.21.15.28:0/1878854578 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3de8102b80 0x7f3de819aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.527+0000 7f3de6800700 1 -- 172.21.15.28:0/1878854578 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3de8103d80 msgr2=0x7f3de819b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.527+0000 7f3de6800700 1 --2- 172.21.15.28:0/1878854578 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3de8103d80 0x7f3de819b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.527+0000 7f3de6800700 1 -- 172.21.15.28:0/1878854578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3de81a2b40 con 0x7f3de8104720 2024-04-23T14:39:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.527+0000 7f3ddd400700 1 --2- 172.21.15.28:0/1878854578 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3de8103d80 0x7f3de819b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:39:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.528+0000 7f3de6800700 1 --2- 172.21.15.28:0/1878854578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3de8104720 0x7f3de81a24a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3de000b0d0 tx=0x7f3de0009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.528+0000 7f3ddf400700 1 -- 172.21.15.28:0/1878854578 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de0021020 con 0x7f3de8104720 2024-04-23T14:39:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.528+0000 7f3ddf400700 1 -- 172.21.15.28:0/1878854578 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3de0004c50 con 0x7f3de8104720 2024-04-23T14:39:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.528+0000 7f3ddf400700 1 -- 172.21.15.28:0/1878854578 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de00052b0 con 0x7f3de8104720 2024-04-23T14:39:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.528+0000 7f3deca00700 1 -- 172.21.15.28:0/1878854578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3de81a2dd0 con 0x7f3de8104720 2024-04-23T14:39:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.528+0000 7f3deca00700 1 -- 172.21.15.28:0/1878854578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3de81a31d0 con 0x7f3de8104720 2024-04-23T14:39:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.529+0000 7f3deca00700 1 -- 172.21.15.28:0/1878854578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3de81a3400 con 0x7f3de8104720 2024-04-23T14:39:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.531+0000 7f3ddf400700 1 -- 172.21.15.28:0/1878854578 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3de0024020 con 0x7f3de8104720 2024-04-23T14:39:20.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.533+0000 7f3ddf400700 1 --2- 172.21.15.28:0/1878854578 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f3dcc0a3cc0 0x7f3dcc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.533+0000 7f3ddf400700 1 -- 172.21.15.28:0/1878854578 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5829+0+0 (secure 0 0 0) 0x7f3de000f020 con 0x7f3de8104720 2024-04-23T14:39:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.533+0000 7f3de5e00700 1 --2- 172.21.15.28:0/1878854578 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f3dcc0a3cc0 0x7f3dcc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.534+0000 7f3de5e00700 1 --2- 172.21.15.28:0/1878854578 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f3dcc0a3cc0 0x7f3dcc0a6180 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3de8103be0 tx=0x7f3dd8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:20.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.538+0000 7f3ddf400700 1 -- 172.21.15.28:0/1878854578 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3de009dda0 con 0x7f3de8104720 2024-04-23T14:39:20.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:20 smithi038 ceph-mon[32348]: osdmap e63: 9 total, 9 up, 9 in 2024-04-23T14:39:20.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:20.778+0000 7f3deca00700 1 -- 172.21.15.28:0/1878854578 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f3de8076440 con 0x7f3dcc0a3cc0 2024-04-23T14:39:20.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:20 smithi105 ceph-mon[32555]: osdmap e63: 9 total, 9 up, 9 in 2024-04-23T14:39:21.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:21 smithi038 ceph-mon[32348]: pgmap v222: 1 pgs: 1 remapped+peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:21.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:21 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:39:21.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:21 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:39:21.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:21 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:39:21.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:21 smithi028 ceph-mon[27810]: pgmap v222: 1 pgs: 1 remapped+peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:21.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:21 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:39:21.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:21 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:39:21.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:21 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:39:21.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:21 smithi105 ceph-mon[32555]: pgmap v222: 1 pgs: 1 remapped+peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:21.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:21 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:39:21.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:21 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:39:21.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:21 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:39:22.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:22 smithi038 ceph-mon[32348]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:39:22.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:22 smithi028 ceph-mon[27810]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:39:22.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:22 smithi105 ceph-mon[32555]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:39:23.545 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:23 smithi028 ceph-mon[27810]: pgmap v223: 1 pgs: 1 remapped+peering; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:23.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:23 smithi038 ceph-mon[32348]: pgmap v223: 1 pgs: 1 remapped+peering; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:23.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:23 smithi105 ceph-mon[32555]: pgmap v223: 1 pgs: 1 remapped+peering; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:25.559 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:25 smithi028 ceph-mon[27810]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-23T14:39:25.559 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:25 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2030142765' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18f60893-3ffd-48c2-9db5-a67d35611d70"}]: dispatch 2024-04-23T14:39:25.559 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:25 smithi028 ceph-mon[27810]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18f60893-3ffd-48c2-9db5-a67d35611d70"}]: dispatch 2024-04-23T14:39:25.559 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:25 smithi028 ceph-mon[27810]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "18f60893-3ffd-48c2-9db5-a67d35611d70"}]': finished 2024-04-23T14:39:25.559 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:25 smithi028 ceph-mon[27810]: osdmap e64: 10 total, 9 up, 10 in 2024-04-23T14:39:25.559 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:25 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T14:39:25.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:25 smithi038 ceph-mon[32348]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-23T14:39:25.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:25 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2030142765' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18f60893-3ffd-48c2-9db5-a67d35611d70"}]: dispatch 2024-04-23T14:39:25.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:25 smithi038 ceph-mon[32348]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18f60893-3ffd-48c2-9db5-a67d35611d70"}]: dispatch 2024-04-23T14:39:25.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:25 smithi038 ceph-mon[32348]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "18f60893-3ffd-48c2-9db5-a67d35611d70"}]': finished 2024-04-23T14:39:25.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:25 smithi038 ceph-mon[32348]: osdmap e64: 10 total, 9 up, 10 in 2024-04-23T14:39:25.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:25 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T14:39:25.789 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:25 smithi105 ceph-mon[32555]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-23T14:39:25.789 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:25 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2030142765' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18f60893-3ffd-48c2-9db5-a67d35611d70"}]: dispatch 2024-04-23T14:39:25.789 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:25 smithi105 ceph-mon[32555]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18f60893-3ffd-48c2-9db5-a67d35611d70"}]: dispatch 2024-04-23T14:39:25.790 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:25 smithi105 ceph-mon[32555]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "18f60893-3ffd-48c2-9db5-a67d35611d70"}]': finished 2024-04-23T14:39:25.790 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:25 smithi105 ceph-mon[32555]: osdmap e64: 10 total, 9 up, 10 in 2024-04-23T14:39:25.790 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:25 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T14:39:26.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:26 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1826721522' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:39:26.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:26 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1826721522' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:39:26.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:26 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1826721522' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:39:27.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:27 smithi038 ceph-mon[32348]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-04-23T14:39:27.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:27 smithi028 ceph-mon[27810]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-04-23T14:39:27.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:27 smithi105 ceph-mon[32555]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-04-23T14:39:29.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:29 smithi028 ceph-mon[27810]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-23T14:39:29.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:29 smithi105 ceph-mon[32555]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-23T14:39:29.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:29 smithi038 ceph-mon[32348]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-23T14:39:31.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:31 smithi028 ceph-mon[27810]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T14:39:31.675 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:31 smithi038 ceph-mon[32348]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T14:39:31.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:31 smithi105 ceph-mon[32555]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T14:39:33.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:33 smithi105 ceph-mon[32555]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T14:39:33.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:33 smithi038 ceph-mon[32348]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T14:39:33.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:33 smithi028 ceph-mon[27810]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T14:39:34.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:34 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-23T14:39:34.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:34 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:39:34.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:34 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-23T14:39:34.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:34 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:39:34.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:34 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-23T14:39:34.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:34 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:39:35.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:35 smithi028 ceph-mon[27810]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:35.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:35 smithi028 ceph-mon[27810]: Deploying daemon osd.9 on smithi028 2024-04-23T14:39:35.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:35 smithi105 ceph-mon[32555]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:35.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:35 smithi105 ceph-mon[32555]: Deploying daemon osd.9 on smithi028 2024-04-23T14:39:35.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:35 smithi038 ceph-mon[32348]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:35.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:35 smithi038 ceph-mon[32348]: Deploying daemon osd.9 on smithi028 2024-04-23T14:39:37.569 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:37 smithi028 ceph-mon[27810]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:37.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:37 smithi105 ceph-mon[32555]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:37.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:37 smithi038 ceph-mon[32348]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:39.558 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:39 smithi028 ceph-mon[27810]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:39.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:39 smithi105 ceph-mon[32555]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:39.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:39 smithi038 ceph-mon[32348]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:41.528 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:41 smithi028 ceph-mon[27810]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:41.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:41 smithi105 ceph-mon[32555]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:41.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:41 smithi038 ceph-mon[32348]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:43.760 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:43 smithi028 ceph-mon[27810]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:43.761 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:43 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:43.761 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:43 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:43.761 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:43 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:39:43.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:43 smithi105 ceph-mon[32555]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:43.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:43 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:43.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:43 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:43.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:43 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:39:43.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:43 smithi038 ceph-mon[32348]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:43.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:43 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:43.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:43 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:43.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:43 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:39:45.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:45 smithi105 ceph-mon[32555]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:45.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:45 smithi038 ceph-mon[32348]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:45.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:45 smithi028 ceph-mon[27810]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:47.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:47 smithi028 ceph-mon[27810]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:47.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:47 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:47.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:47 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:47.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:47 smithi028 ceph-mon[27810]: from='osd.9 [v2:172.21.15.28:6826/1292942516,v1:172.21.15.28:6827/1292942516]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-23T14:39:47.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:47 smithi028 ceph-mon[27810]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-23T14:39:47.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:47 smithi105 ceph-mon[32555]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:47.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:47 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:47.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:47 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:47.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:47 smithi105 ceph-mon[32555]: from='osd.9 [v2:172.21.15.28:6826/1292942516,v1:172.21.15.28:6827/1292942516]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-23T14:39:47.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:47 smithi105 ceph-mon[32555]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-23T14:39:47.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:47 smithi038 ceph-mon[32348]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:47.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:47 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:47.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:47 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:47.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:47 smithi038 ceph-mon[32348]: from='osd.9 [v2:172.21.15.28:6826/1292942516,v1:172.21.15.28:6827/1292942516]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-23T14:39:47.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:47 smithi038 ceph-mon[32348]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-23T14:39:48.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:48.272+0000 7f3ddf400700 1 -- 172.21.15.28:0/1878854578 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3de8076440 con 0x7f3dcc0a3cc0 2024-04-23T14:39:48.274 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 9 on host 'smithi028' 2024-04-23T14:39:48.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:48.276+0000 7f3deca00700 1 -- 172.21.15.28:0/1878854578 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f3dcc0a3cc0 msgr2=0x7f3dcc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:48.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:48.276+0000 7f3deca00700 1 --2- 172.21.15.28:0/1878854578 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f3dcc0a3cc0 0x7f3dcc0a6180 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3de8103be0 tx=0x7f3dd8009260 comp rx=0 tx=0).stop 2024-04-23T14:39:48.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:48.276+0000 7f3deca00700 1 -- 172.21.15.28:0/1878854578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3de8104720 msgr2=0x7f3de81a24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:48.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:48.276+0000 7f3deca00700 1 --2- 172.21.15.28:0/1878854578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3de8104720 0x7f3de81a24a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3de000b0d0 tx=0x7f3de0009810 comp rx=0 tx=0).stop 2024-04-23T14:39:48.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:48.276+0000 7f3deca00700 1 -- 172.21.15.28:0/1878854578 shutdown_connections 2024-04-23T14:39:48.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:48.276+0000 7f3deca00700 1 --2- 172.21.15.28:0/1878854578 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f3dcc0a3cc0 0x7f3dcc0a6180 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:48.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:48.276+0000 7f3deca00700 1 --2- 172.21.15.28:0/1878854578 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3de8102b80 0x7f3de819aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:48.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:48.276+0000 7f3deca00700 1 --2- 172.21.15.28:0/1878854578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3de8104720 0x7f3de81a24a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:48.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:48.276+0000 7f3deca00700 1 --2- 172.21.15.28:0/1878854578 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3de8103d80 0x7f3de819b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:48.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:48.276+0000 7f3deca00700 1 -- 172.21.15.28:0/1878854578 >> 172.21.15.28:0/1878854578 conn(0x7f3de80fe140 msgr2=0x7f3de8074d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:48.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:48.276+0000 7f3deca00700 1 -- 172.21.15.28:0/1878854578 shutdown_connections 2024-04-23T14:39:48.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:39:48.276+0000 7f3deca00700 1 -- 172.21.15.28:0/1878854578 wait complete. 2024-04-23T14:39:49.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:48 smithi038 ceph-mon[32348]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:49.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:48 smithi038 ceph-mon[32348]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-23T14:39:49.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:48 smithi038 ceph-mon[32348]: osdmap e65: 10 total, 9 up, 10 in 2024-04-23T14:39:49.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:48 smithi038 ceph-mon[32348]: from='osd.9 [v2:172.21.15.28:6826/1292942516,v1:172.21.15.28:6827/1292942516]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-23T14:39:49.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:48 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T14:39:49.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:48 smithi038 ceph-mon[32348]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-23T14:39:49.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:48 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:49.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:48 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:49.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:48 smithi028 ceph-mon[27810]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:49.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:48 smithi028 ceph-mon[27810]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-23T14:39:49.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:48 smithi028 ceph-mon[27810]: osdmap e65: 10 total, 9 up, 10 in 2024-04-23T14:39:49.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:48 smithi028 ceph-mon[27810]: from='osd.9 [v2:172.21.15.28:6826/1292942516,v1:172.21.15.28:6827/1292942516]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-23T14:39:49.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:48 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T14:39:49.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:48 smithi028 ceph-mon[27810]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-23T14:39:49.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:48 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:49.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:48 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:49.181 DEBUG:teuthology.orchestra.run.smithi028:osd.9> sudo journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.9.service 2024-04-23T14:39:49.185 INFO:tasks.cephadm:Deploying osd.10 on smithi038 with /dev/vg_nvme/lv_1... 2024-04-23T14:39:49.185 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-23T14:39:49.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:48 smithi105 ceph-mon[32555]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:49.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:48 smithi105 ceph-mon[32555]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-23T14:39:49.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:48 smithi105 ceph-mon[32555]: osdmap e65: 10 total, 9 up, 10 in 2024-04-23T14:39:49.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:48 smithi105 ceph-mon[32555]: from='osd.9 [v2:172.21.15.28:6826/1292942516,v1:172.21.15.28:6827/1292942516]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-23T14:39:49.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:48 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T14:39:49.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:48 smithi105 ceph-mon[32555]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-23T14:39:49.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:48 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:49.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:48 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:49.447 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T14:39:49.984 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:49 smithi028 ceph-mon[27810]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-23T14:39:49.984 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:49 smithi028 ceph-mon[27810]: osdmap e66: 10 total, 9 up, 10 in 2024-04-23T14:39:49.984 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:49 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T14:39:49.985 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:49 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T14:39:49.985 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:49 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T14:39:49.985 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 14:39:49 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T14:39:49.816+0000 7f4210600700 -1 osd.9 0 waiting for initial osdmap 2024-04-23T14:39:49.985 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 14:39:49 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T14:39:49.875+0000 7f4206400700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:39:50.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:49 smithi038 ceph-mon[32348]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-23T14:39:50.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:49 smithi038 ceph-mon[32348]: osdmap e66: 10 total, 9 up, 10 in 2024-04-23T14:39:50.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:49 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T14:39:50.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:49 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T14:39:50.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:49 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T14:39:50.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:49 smithi105 ceph-mon[32555]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-23T14:39:50.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:49 smithi105 ceph-mon[32555]: osdmap e66: 10 total, 9 up, 10 in 2024-04-23T14:39:50.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:49 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T14:39:50.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:49 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T14:39:50.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:49 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T14:39:50.984 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:50 smithi038 ceph-mon[32348]: purged_snaps scrub starts 2024-04-23T14:39:50.984 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:50 smithi038 ceph-mon[32348]: purged_snaps scrub ok 2024-04-23T14:39:50.984 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:50 smithi038 ceph-mon[32348]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:50.984 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:50 smithi038 ceph-mon[32348]: from='osd.9 ' entity='osd.9' 2024-04-23T14:39:50.984 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:50 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T14:39:51.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:50 smithi028 ceph-mon[27810]: purged_snaps scrub starts 2024-04-23T14:39:51.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:50 smithi028 ceph-mon[27810]: purged_snaps scrub ok 2024-04-23T14:39:51.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:50 smithi028 ceph-mon[27810]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:51.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:50 smithi028 ceph-mon[27810]: from='osd.9 ' entity='osd.9' 2024-04-23T14:39:51.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:50 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T14:39:51.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:50 smithi105 ceph-mon[32555]: purged_snaps scrub starts 2024-04-23T14:39:51.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:50 smithi105 ceph-mon[32555]: purged_snaps scrub ok 2024-04-23T14:39:51.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:50 smithi105 ceph-mon[32555]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:51.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:50 smithi105 ceph-mon[32555]: from='osd.9 ' entity='osd.9' 2024-04-23T14:39:51.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:50 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T14:39:51.562 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-23T14:39:51.562 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-23T14:39:51.563 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-04-23T14:39:51.563 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-04-23T14:39:51.563 INFO:teuthology.orchestra.run.smithi038.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00691262 s, 1.5 GB/s 2024-04-23T14:39:51.563 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-04-23T14:39:52.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:51 smithi038 ceph-mon[32348]: osd.9 [v2:172.21.15.28:6826/1292942516,v1:172.21.15.28:6827/1292942516] boot 2024-04-23T14:39:52.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:51 smithi038 ceph-mon[32348]: osdmap e67: 10 total, 10 up, 10 in 2024-04-23T14:39:52.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:51 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T14:39:52.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:51 smithi028 ceph-mon[27810]: osd.9 [v2:172.21.15.28:6826/1292942516,v1:172.21.15.28:6827/1292942516] boot 2024-04-23T14:39:52.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:51 smithi028 ceph-mon[27810]: osdmap e67: 10 total, 10 up, 10 in 2024-04-23T14:39:52.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:51 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T14:39:52.198 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi038:vg_nvme/lv_1 2024-04-23T14:39:52.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:51 smithi105 ceph-mon[32555]: osd.9 [v2:172.21.15.28:6826/1292942516,v1:172.21.15.28:6827/1292942516] boot 2024-04-23T14:39:52.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:51 smithi105 ceph-mon[32555]: osdmap e67: 10 total, 10 up, 10 in 2024-04-23T14:39:52.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:51 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T14:39:52.447 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T14:39:53.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:52 smithi038 ceph-mon[32348]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:53.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:52 smithi038 ceph-mon[32348]: osdmap e68: 10 total, 10 up, 10 in 2024-04-23T14:39:53.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:52 smithi028 ceph-mon[27810]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:53.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:52 smithi028 ceph-mon[27810]: osdmap e68: 10 total, 10 up, 10 in 2024-04-23T14:39:53.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:52 smithi105 ceph-mon[32555]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T14:39:53.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:52 smithi105 ceph-mon[32555]: osdmap e68: 10 total, 10 up, 10 in 2024-04-23T14:39:53.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.904+0000 7fd9b4c77700 1 -- 172.21.15.38:0/853959155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9b0103600 msgr2=0x7fd9b01059f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:53.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.904+0000 7fd9b4c77700 1 --2- 172.21.15.38:0/853959155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9b0103600 0x7fd9b01059f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd9a0009990 tx=0x7fd9a0009ca0 comp rx=0 tx=0).stop 2024-04-23T14:39:53.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.904+0000 7fd9b4c77700 1 -- 172.21.15.38:0/853959155 shutdown_connections 2024-04-23T14:39:53.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.904+0000 7fd9b4c77700 1 --2- 172.21.15.38:0/853959155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9b0108860 0x7fd9b010acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:53.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.904+0000 7fd9b4c77700 1 --2- 172.21.15.38:0/853959155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9b0103600 0x7fd9b01059f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:53.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.904+0000 7fd9b4c77700 1 --2- 172.21.15.38:0/853959155 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd9b0105f30 0x7fd9b0108320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:53.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.904+0000 7fd9b4c77700 1 -- 172.21.15.38:0/853959155 >> 172.21.15.38:0/853959155 conn(0x7fd9b00fd010 msgr2=0x7fd9b00ff450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:39:53.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.904+0000 7fd9b4c77700 1 -- 172.21.15.38:0/853959155 shutdown_connections 2024-04-23T14:39:53.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.904+0000 7fd9b4c77700 1 -- 172.21.15.38:0/853959155 wait complete. 2024-04-23T14:39:53.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.905+0000 7fd9b4c77700 1 Processor -- start 2024-04-23T14:39:53.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.905+0000 7fd9b4c77700 1 -- start start 2024-04-23T14:39:53.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.905+0000 7fd9b4c77700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd9b0103600 0x7fd9b0198d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:53.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.905+0000 7fd9b4c77700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9b0105f30 0x7fd9b0199290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:53.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.905+0000 7fd9b4c77700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9b0108860 0x7fd9b01a0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:53.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.905+0000 7fd9b4c77700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b010e7c0 con 0x7fd9b0105f30 2024-04-23T14:39:53.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.905+0000 7fd9b4c77700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b010e640 con 0x7fd9b0108860 2024-04-23T14:39:53.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.905+0000 7fd9b4c77700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b010e940 con 0x7fd9b0103600 2024-04-23T14:39:53.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.905+0000 7fd9aed9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9b0108860 0x7fd9b01a0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:53.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.905+0000 7fd9aed9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9b0108860 0x7fd9b01a0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38690/0 (socket says 172.21.15.38:38690) 2024-04-23T14:39:53.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.905+0000 7fd9aed9d700 1 -- 172.21.15.38:0/4078329362 learned_addr learned my addr 172.21.15.38:0/4078329362 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-23T14:39:53.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.905+0000 7fd9ae59c700 1 --2- 172.21.15.38:0/4078329362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd9b0103600 0x7fd9b0198d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:53.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.906+0000 7fd9aed9d700 1 -- 172.21.15.38:0/4078329362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd9b0103600 msgr2=0x7fd9b0198d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:53.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.906+0000 7fd9aed9d700 1 --2- 172.21.15.38:0/4078329362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd9b0103600 0x7fd9b0198d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:53.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.906+0000 7fd9aed9d700 1 -- 172.21.15.38:0/4078329362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9b0105f30 msgr2=0x7fd9b0199290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:39:53.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.906+0000 7fd9add9b700 1 --2- 172.21.15.38:0/4078329362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9b0105f30 0x7fd9b0199290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:53.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.906+0000 7fd9aed9d700 1 --2- 172.21.15.38:0/4078329362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9b0105f30 0x7fd9b0199290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:39:53.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.906+0000 7fd9aed9d700 1 -- 172.21.15.38:0/4078329362 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9b01a09b0 con 0x7fd9b0108860 2024-04-23T14:39:53.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.906+0000 7fd9aed9d700 1 --2- 172.21.15.38:0/4078329362 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9b0108860 0x7fd9b01a0310 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd9a800b9e0 tx=0x7fd9a800bcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:53.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.906+0000 7fd99f7fe700 1 -- 172.21.15.38:0/4078329362 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a8019070 con 0x7fd9b0108860 2024-04-23T14:39:53.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.906+0000 7fd99f7fe700 1 -- 172.21.15.38:0/4078329362 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9a800ce10 con 0x7fd9b0108860 2024-04-23T14:39:53.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.906+0000 7fd9b4c77700 1 -- 172.21.15.38:0/4078329362 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9b01a0ca0 con 0x7fd9b0108860 2024-04-23T14:39:53.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.906+0000 7fd99f7fe700 1 -- 172.21.15.38:0/4078329362 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a80123a0 con 0x7fd9b0108860 2024-04-23T14:39:53.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.906+0000 7fd9b4c77700 1 -- 172.21.15.38:0/4078329362 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9b01a1190 con 0x7fd9b0108860 2024-04-23T14:39:53.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.907+0000 7fd9b4c77700 1 -- 172.21.15.38:0/4078329362 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9b00697e0 con 0x7fd9b0108860 2024-04-23T14:39:53.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.908+0000 7fd99f7fe700 1 -- 172.21.15.38:0/4078329362 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd9a801c070 con 0x7fd9b0108860 2024-04-23T14:39:53.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.909+0000 7fd99f7fe700 1 --2- 172.21.15.38:0/4078329362 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd9980a3c40 0x7fd9980a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:39:53.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.909+0000 7fd99f7fe700 1 -- 172.21.15.38:0/4078329362 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7fd9a80cef00 con 0x7fd9b0108860 2024-04-23T14:39:53.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.909+0000 7fd9ae59c700 1 --2- 172.21.15.38:0/4078329362 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd9980a3c40 0x7fd9980a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:39:53.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.910+0000 7fd9ae59c700 1 --2- 172.21.15.38:0/4078329362 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd9980a3c40 0x7fd9980a6100 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd9a000b5c0 tx=0x7fd9a0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:39:53.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:53.914+0000 7fd99f7fe700 1 -- 172.21.15.38:0/4078329362 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd9a809c0e0 con 0x7fd9b0108860 2024-04-23T14:39:54.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:39:54.132+0000 7fd9b4c77700 1 -- 172.21.15.38:0/4078329362 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fd9b0062f30 con 0x7fd9980a3c40 2024-04-23T14:39:55.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:54 smithi038 ceph-mon[32348]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:39:55.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:54 smithi038 ceph-mon[32348]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:39:55.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:54 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:39:55.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:54 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:39:55.124 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:54 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:39:55.124 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:54 smithi038 ceph-mon[32348]: Detected new or changed devices on smithi028 2024-04-23T14:39:55.124 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:54 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:55.124 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:54 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:55.124 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:54 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:39:55.124 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:54 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:39:55.124 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:54 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:39:55.124 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:54 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:39:55.124 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:54 smithi038 ceph-mon[32348]: Adjusting osd_memory_target on smithi028 to 4273M 2024-04-23T14:39:55.124 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:54 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:55.124 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:54 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:39:55.124 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:54 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:39:55.124 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:54 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:55.124 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:54 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:39:55.125 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:54 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:39:55.125 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:54 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:39:55.125 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:54 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:55.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:54 smithi028 ceph-mon[27810]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:39:55.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:54 smithi028 ceph-mon[27810]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:39:55.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:54 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:39:55.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:54 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:39:55.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:54 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:39:55.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:54 smithi028 ceph-mon[27810]: Detected new or changed devices on smithi028 2024-04-23T14:39:55.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:54 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:55.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:54 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:55.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:54 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:39:55.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:54 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:39:55.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:54 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:39:55.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:54 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:39:55.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:54 smithi028 ceph-mon[27810]: Adjusting osd_memory_target on smithi028 to 4273M 2024-04-23T14:39:55.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:54 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:55.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:54 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:39:55.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:54 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:39:55.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:54 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:55.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:54 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:39:55.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:54 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:39:55.127 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:54 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:39:55.127 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:54 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:55.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:54 smithi105 ceph-mon[32555]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:39:55.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:54 smithi105 ceph-mon[32555]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:39:55.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:54 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:39:55.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:54 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:39:55.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:54 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:39:55.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:54 smithi105 ceph-mon[32555]: Detected new or changed devices on smithi028 2024-04-23T14:39:55.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:54 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:55.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:54 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:55.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:54 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:39:55.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:54 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:39:55.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:54 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:39:55.288 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:54 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:39:55.288 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:54 smithi105 ceph-mon[32555]: Adjusting osd_memory_target on smithi028 to 4273M 2024-04-23T14:39:55.288 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:54 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:55.288 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:54 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:39:55.288 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:54 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:39:55.288 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:54 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:55.288 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:54 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:39:55.288 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:54 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:39:55.288 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:54 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:39:55.288 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:54 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:39:57.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:56 smithi038 ceph-mon[32348]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:39:57.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:56 smithi028 ceph-mon[27810]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:39:57.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:56 smithi105 ceph-mon[32555]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:39:58.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:57 smithi105 ceph-mon[32555]: from='client.? 172.21.15.38:0/275865836' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76c06725-7040-4118-8894-fe808a87f999"}]: dispatch 2024-04-23T14:39:58.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:57 smithi105 ceph-mon[32555]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76c06725-7040-4118-8894-fe808a87f999"}]: dispatch 2024-04-23T14:39:58.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:57 smithi105 ceph-mon[32555]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "76c06725-7040-4118-8894-fe808a87f999"}]': finished 2024-04-23T14:39:58.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:57 smithi105 ceph-mon[32555]: osdmap e69: 11 total, 10 up, 11 in 2024-04-23T14:39:58.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:57 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T14:39:58.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:57 smithi105 ceph-mon[32555]: from='client.? 172.21.15.38:0/1828039803' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:39:58.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:58 smithi038 ceph-mon[32348]: from='client.? 172.21.15.38:0/275865836' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76c06725-7040-4118-8894-fe808a87f999"}]: dispatch 2024-04-23T14:39:58.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:58 smithi038 ceph-mon[32348]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76c06725-7040-4118-8894-fe808a87f999"}]: dispatch 2024-04-23T14:39:58.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:58 smithi038 ceph-mon[32348]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "76c06725-7040-4118-8894-fe808a87f999"}]': finished 2024-04-23T14:39:58.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:58 smithi038 ceph-mon[32348]: osdmap e69: 11 total, 10 up, 11 in 2024-04-23T14:39:58.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:58 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T14:39:58.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:58 smithi038 ceph-mon[32348]: from='client.? 172.21.15.38:0/1828039803' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:39:58.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:57 smithi028 ceph-mon[27810]: from='client.? 172.21.15.38:0/275865836' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76c06725-7040-4118-8894-fe808a87f999"}]: dispatch 2024-04-23T14:39:58.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:57 smithi028 ceph-mon[27810]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76c06725-7040-4118-8894-fe808a87f999"}]: dispatch 2024-04-23T14:39:58.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:57 smithi028 ceph-mon[27810]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "76c06725-7040-4118-8894-fe808a87f999"}]': finished 2024-04-23T14:39:58.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:57 smithi028 ceph-mon[27810]: osdmap e69: 11 total, 10 up, 11 in 2024-04-23T14:39:58.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:57 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T14:39:58.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:57 smithi028 ceph-mon[27810]: from='client.? 172.21.15.38:0/1828039803' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:39:59.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:39:59 smithi038 ceph-mon[32348]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:39:59.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:39:59 smithi028 ceph-mon[27810]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:39:59.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:39:59 smithi105 ceph-mon[32555]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:40:00.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:00 smithi038 ceph-mon[32348]: overall HEALTH_OK 2024-04-23T14:40:00.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:00 smithi028 ceph-mon[27810]: overall HEALTH_OK 2024-04-23T14:40:00.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:00 smithi105 ceph-mon[32555]: overall HEALTH_OK 2024-04-23T14:40:01.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:01 smithi038 ceph-mon[32348]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:40:01.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:01 smithi028 ceph-mon[27810]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:40:01.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:01 smithi105 ceph-mon[32555]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:40:03.607 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:03 smithi038 ceph-mon[32348]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:40:03.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:03 smithi028 ceph-mon[27810]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:40:03.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:03 smithi105 ceph-mon[32555]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:40:05.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:05 smithi038 ceph-mon[32348]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:40:05.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:05 smithi028 ceph-mon[27810]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:40:05.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:05 smithi105 ceph-mon[32555]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:40:06.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:06 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-23T14:40:06.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:06 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:40:06.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:06 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-23T14:40:06.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:06 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:40:06.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:06 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-23T14:40:06.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:06 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:40:07.592 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:07 smithi038 ceph-mon[32348]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:40:07.593 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:07 smithi038 ceph-mon[32348]: Deploying daemon osd.10 on smithi038 2024-04-23T14:40:07.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:07 smithi028 ceph-mon[27810]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:40:07.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:07 smithi028 ceph-mon[27810]: Deploying daemon osd.10 on smithi038 2024-04-23T14:40:07.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:07 smithi105 ceph-mon[32555]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:40:07.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:07 smithi105 ceph-mon[32555]: Deploying daemon osd.10 on smithi038 2024-04-23T14:40:09.599 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:09 smithi038 ceph-mon[32348]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:40:09.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:09 smithi028 ceph-mon[27810]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:40:09.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:09 smithi105 ceph-mon[32555]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:40:11.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:11 smithi038 ceph-mon[32348]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:40:11.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:11 smithi028 ceph-mon[27810]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:40:11.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:11 smithi105 ceph-mon[32555]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:40:12.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:12 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:12.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:12 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:12.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:12 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:40:12.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:12 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:12.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:12 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:12.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:12 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:40:12.808 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:12 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:12.809 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:12 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:12.809 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:12 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:40:13.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:13 smithi105 ceph-mon[32555]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:40:13.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:13 smithi038 ceph-mon[32348]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:40:13.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:13 smithi028 ceph-mon[27810]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:40:15.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:40:15.277+0000 7fd99f7fe700 1 -- 172.21.15.38:0/4078329362 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fd9b0062f30 con 0x7fd9980a3c40 2024-04-23T14:40:15.279 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 10 on host 'smithi038' 2024-04-23T14:40:15.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:40:15.280+0000 7fd9b4c77700 1 -- 172.21.15.38:0/4078329362 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd9980a3c40 msgr2=0x7fd9980a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:15.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:40:15.280+0000 7fd9b4c77700 1 --2- 172.21.15.38:0/4078329362 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd9980a3c40 0x7fd9980a6100 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd9a000b5c0 tx=0x7fd9a0005fd0 comp rx=0 tx=0).stop 2024-04-23T14:40:15.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:40:15.280+0000 7fd9b4c77700 1 -- 172.21.15.38:0/4078329362 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9b0108860 msgr2=0x7fd9b01a0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:15.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:40:15.280+0000 7fd9b4c77700 1 --2- 172.21.15.38:0/4078329362 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9b0108860 0x7fd9b01a0310 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd9a800b9e0 tx=0x7fd9a800bcf0 comp rx=0 tx=0).stop 2024-04-23T14:40:15.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:40:15.280+0000 7fd9b4c77700 1 -- 172.21.15.38:0/4078329362 shutdown_connections 2024-04-23T14:40:15.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:40:15.280+0000 7fd9b4c77700 1 --2- 172.21.15.38:0/4078329362 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd9980a3c40 0x7fd9980a6100 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:15.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:40:15.280+0000 7fd9b4c77700 1 --2- 172.21.15.38:0/4078329362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd9b0103600 0x7fd9b0198d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:15.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:40:15.280+0000 7fd9b4c77700 1 --2- 172.21.15.38:0/4078329362 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9b0108860 0x7fd9b01a0310 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:15.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:40:15.280+0000 7fd9b4c77700 1 --2- 172.21.15.38:0/4078329362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9b0105f30 0x7fd9b0199290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:15.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:40:15.280+0000 7fd9b4c77700 1 -- 172.21.15.38:0/4078329362 >> 172.21.15.38:0/4078329362 conn(0x7fd9b00fd010 msgr2=0x7fd9b0103f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:40:15.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:40:15.280+0000 7fd9b4c77700 1 -- 172.21.15.38:0/4078329362 shutdown_connections 2024-04-23T14:40:15.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-23T14:40:15.280+0000 7fd9b4c77700 1 -- 172.21.15.38:0/4078329362 wait complete. 2024-04-23T14:40:15.581 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:15 smithi038 ceph-mon[32348]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:40:15.581 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:15 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:15.581 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:15 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:15.581 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:15 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:40:15.581 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:15 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:40:15.581 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:15 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:15.581 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:15 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:15.581 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:15 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:15.581 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:15 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:40:15.581 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:15 smithi038 ceph-mon[32348]: from='osd.10 [v2:172.21.15.38:6824/2444616468,v1:172.21.15.38:6825/2444616468]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T14:40:15.582 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:15 smithi038 ceph-mon[32348]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T14:40:15.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:15 smithi028 ceph-mon[27810]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:40:15.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:15 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:15.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:15 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:15.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:15 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:40:15.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:15 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:40:15.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:15 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:15.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:15 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:15.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:15 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:15.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:15 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:40:15.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:15 smithi028 ceph-mon[27810]: from='osd.10 [v2:172.21.15.38:6824/2444616468,v1:172.21.15.38:6825/2444616468]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T14:40:15.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:15 smithi028 ceph-mon[27810]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T14:40:15.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:15 smithi105 ceph-mon[32555]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:40:15.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:15 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:15.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:15 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:15.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:15 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:40:15.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:15 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:40:15.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:15 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:15.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:15 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:15.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:15 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:15.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:15 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:40:15.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:15 smithi105 ceph-mon[32555]: from='osd.10 [v2:172.21.15.38:6824/2444616468,v1:172.21.15.38:6825/2444616468]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T14:40:15.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:15 smithi105 ceph-mon[32555]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T14:40:15.835 DEBUG:teuthology.orchestra.run.smithi038:osd.10> sudo journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.10.service 2024-04-23T14:40:15.837 INFO:tasks.cephadm:Deploying osd.11 on smithi105 with /dev/vg_nvme/lv_1... 2024-04-23T14:40:15.838 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-23T14:40:16.084 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T14:40:17.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:17 smithi105 ceph-mon[32555]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:40:17.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:17 smithi105 ceph-mon[32555]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-23T14:40:17.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:17 smithi105 ceph-mon[32555]: from='osd.10 [v2:172.21.15.38:6824/2444616468,v1:172.21.15.38:6825/2444616468]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-23T14:40:17.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:17 smithi105 ceph-mon[32555]: osdmap e70: 11 total, 10 up, 11 in 2024-04-23T14:40:17.538 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:17 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T14:40:17.538 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:17 smithi105 ceph-mon[32555]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-23T14:40:17.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:17 smithi038 ceph-mon[32348]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:40:17.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:17 smithi038 ceph-mon[32348]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-23T14:40:17.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:17 smithi038 ceph-mon[32348]: from='osd.10 [v2:172.21.15.38:6824/2444616468,v1:172.21.15.38:6825/2444616468]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-23T14:40:17.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:17 smithi038 ceph-mon[32348]: osdmap e70: 11 total, 10 up, 11 in 2024-04-23T14:40:17.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:17 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T14:40:17.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:17 smithi038 ceph-mon[32348]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-23T14:40:17.625 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 14:40:17 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T14:40:17.241+0000 7f708fad5700 -1 osd.10 0 waiting for initial osdmap 2024-04-23T14:40:17.625 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 14:40:17 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T14:40:17.258+0000 7f708b8c8700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:40:17.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:17 smithi028 ceph-mon[27810]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:40:17.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:17 smithi028 ceph-mon[27810]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-23T14:40:17.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:17 smithi028 ceph-mon[27810]: from='osd.10 [v2:172.21.15.38:6824/2444616468,v1:172.21.15.38:6825/2444616468]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-23T14:40:17.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:17 smithi028 ceph-mon[27810]: osdmap e70: 11 total, 10 up, 11 in 2024-04-23T14:40:17.627 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:17 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T14:40:17.627 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:17 smithi028 ceph-mon[27810]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-23T14:40:18.145 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-23T14:40:18.145 INFO:teuthology.orchestra.run.smithi105.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-23T14:40:18.145 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10+0 records in 2024-04-23T14:40:18.145 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2024-04-23T14:40:18.145 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112866 s, 929 MB/s 2024-04-23T14:40:18.146 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping successful for: 2024-04-23T14:40:18.303 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:18 smithi105 ceph-mon[32555]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-23T14:40:18.303 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:18 smithi105 ceph-mon[32555]: osdmap e71: 11 total, 10 up, 11 in 2024-04-23T14:40:18.303 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:18 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T14:40:18.303 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:18 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T14:40:18.365 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:18 smithi038 ceph-mon[32348]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-23T14:40:18.365 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:18 smithi038 ceph-mon[32348]: osdmap e71: 11 total, 10 up, 11 in 2024-04-23T14:40:18.365 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:18 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T14:40:18.365 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:18 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T14:40:18.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:18 smithi028 ceph-mon[27810]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-23T14:40:18.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:18 smithi028 ceph-mon[27810]: osdmap e71: 11 total, 10 up, 11 in 2024-04-23T14:40:18.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:18 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T14:40:18.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:18 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T14:40:18.844 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi105:vg_nvme/lv_1 2024-04-23T14:40:19.087 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T14:40:19.539 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:19 smithi105 ceph-mon[32555]: purged_snaps scrub starts 2024-04-23T14:40:19.539 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:19 smithi105 ceph-mon[32555]: purged_snaps scrub ok 2024-04-23T14:40:19.539 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:19 smithi105 ceph-mon[32555]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:40:19.539 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:19 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T14:40:19.539 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:19 smithi105 ceph-mon[32555]: osd.10 [v2:172.21.15.38:6824/2444616468,v1:172.21.15.38:6825/2444616468] boot 2024-04-23T14:40:19.539 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:19 smithi105 ceph-mon[32555]: osdmap e72: 11 total, 11 up, 11 in 2024-04-23T14:40:19.539 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:19 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T14:40:19.539 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:19 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:19.540 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:19 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:19.540 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:19 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:40:19.540 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:19 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:40:19.540 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:19 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:40:19.540 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:19 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:40:19.540 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:19 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:19.540 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:19 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:40:19.540 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:19 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:40:19.540 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:19 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:19.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:19 smithi038 ceph-mon[32348]: purged_snaps scrub starts 2024-04-23T14:40:19.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:19 smithi038 ceph-mon[32348]: purged_snaps scrub ok 2024-04-23T14:40:19.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:19 smithi038 ceph-mon[32348]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:40:19.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:19 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T14:40:19.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:19 smithi038 ceph-mon[32348]: osd.10 [v2:172.21.15.38:6824/2444616468,v1:172.21.15.38:6825/2444616468] boot 2024-04-23T14:40:19.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:19 smithi038 ceph-mon[32348]: osdmap e72: 11 total, 11 up, 11 in 2024-04-23T14:40:19.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:19 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T14:40:19.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:19 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:19.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:19 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:19.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:19 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:40:19.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:19 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:40:19.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:19 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:40:19.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:19 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:40:19.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:19 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:19.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:19 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:40:19.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:19 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:40:19.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:19 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:19.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:19 smithi028 ceph-mon[27810]: purged_snaps scrub starts 2024-04-23T14:40:19.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:19 smithi028 ceph-mon[27810]: purged_snaps scrub ok 2024-04-23T14:40:19.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:19 smithi028 ceph-mon[27810]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-04-23T14:40:19.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:19 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T14:40:19.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:19 smithi028 ceph-mon[27810]: osd.10 [v2:172.21.15.38:6824/2444616468,v1:172.21.15.38:6825/2444616468] boot 2024-04-23T14:40:19.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:19 smithi028 ceph-mon[27810]: osdmap e72: 11 total, 11 up, 11 in 2024-04-23T14:40:19.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:19 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T14:40:19.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:19 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:19.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:19 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:19.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:19 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:40:19.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:19 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:40:19.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:19 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:40:19.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:19 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:40:19.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:19 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:19.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:19 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:40:19.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:19 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:40:19.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:19 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:20.466 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:20 smithi105 ceph-mon[32555]: Detected new or changed devices on smithi038 2024-04-23T14:40:20.466 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:20 smithi105 ceph-mon[32555]: Adjusting osd_memory_target on smithi038 to 4273M 2024-04-23T14:40:20.466 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:20 smithi105 ceph-mon[32555]: osdmap e73: 11 total, 11 up, 11 in 2024-04-23T14:40:20.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:20 smithi038 ceph-mon[32348]: Detected new or changed devices on smithi038 2024-04-23T14:40:20.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:20 smithi038 ceph-mon[32348]: Adjusting osd_memory_target on smithi038 to 4273M 2024-04-23T14:40:20.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:20 smithi038 ceph-mon[32348]: osdmap e73: 11 total, 11 up, 11 in 2024-04-23T14:40:20.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:20 smithi028 ceph-mon[27810]: Detected new or changed devices on smithi038 2024-04-23T14:40:20.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:20 smithi028 ceph-mon[27810]: Adjusting osd_memory_target on smithi038 to 4273M 2024-04-23T14:40:20.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:20 smithi028 ceph-mon[27810]: osdmap e73: 11 total, 11 up, 11 in 2024-04-23T14:40:20.679 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.677+0000 7f2d78c63700 1 -- 172.21.15.105:0/618433338 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d74101d30 msgr2=0x7f2d74102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:20.680 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.677+0000 7f2d78c63700 1 --2- 172.21.15.105:0/618433338 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d74101d30 0x7f2d74102190 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2d680099c0 tx=0x7f2d68009cd0 comp rx=0 tx=0).stop 2024-04-23T14:40:20.680 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.677+0000 7f2d78c63700 1 -- 172.21.15.105:0/618433338 shutdown_connections 2024-04-23T14:40:20.680 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.677+0000 7f2d78c63700 1 --2- 172.21.15.105:0/618433338 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d741026d0 0x7f2d74109f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:20.680 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.677+0000 7f2d78c63700 1 --2- 172.21.15.105:0/618433338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d74102ff0 0x7f2d74103410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:20.680 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.677+0000 7f2d78c63700 1 --2- 172.21.15.105:0/618433338 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d74101d30 0x7f2d74102190 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:20.680 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.677+0000 7f2d78c63700 1 -- 172.21.15.105:0/618433338 >> 172.21.15.105:0/618433338 conn(0x7f2d740fc760 msgr2=0x7f2d740febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:40:20.680 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.677+0000 7f2d78c63700 1 -- 172.21.15.105:0/618433338 shutdown_connections 2024-04-23T14:40:20.680 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.677+0000 7f2d78c63700 1 -- 172.21.15.105:0/618433338 wait complete. 2024-04-23T14:40:20.680 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.677+0000 7f2d78c63700 1 Processor -- start 2024-04-23T14:40:20.680 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.677+0000 7f2d78c63700 1 -- start start 2024-04-23T14:40:20.680 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.678+0000 7f2d78c63700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d74101d30 0x7f2d74198f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:20.681 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.678+0000 7f2d78c63700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d741026d0 0x7f2d74199490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:20.681 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.678+0000 7f2d78c63700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d74102ff0 0x7f2d741a04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:20.681 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.678+0000 7f2d78c63700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2d7410d740 con 0x7f2d741026d0 2024-04-23T14:40:20.681 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.678+0000 7f2d78c63700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2d7410d5c0 con 0x7f2d74102ff0 2024-04-23T14:40:20.681 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.678+0000 7f2d78c63700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2d7410d8c0 con 0x7f2d74101d30 2024-04-23T14:40:20.681 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.678+0000 7f2d7259c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d74101d30 0x7f2d74198f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:40:20.681 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.678+0000 7f2d7259c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d74101d30 0x7f2d74198f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47232/0 (socket says 172.21.15.105:47232) 2024-04-23T14:40:20.681 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.678+0000 7f2d7259c700 1 -- 172.21.15.105:0/604645611 learned_addr learned my addr 172.21.15.105:0/604645611 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-23T14:40:20.681 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.678+0000 7f2d72d9d700 1 --2- 172.21.15.105:0/604645611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d74102ff0 0x7f2d741a04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:40:20.682 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.678+0000 7f2d7259c700 1 -- 172.21.15.105:0/604645611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d74102ff0 msgr2=0x7f2d741a04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:20.682 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.678+0000 7f2d7259c700 1 --2- 172.21.15.105:0/604645611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d74102ff0 0x7f2d741a04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:20.682 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.678+0000 7f2d7259c700 1 -- 172.21.15.105:0/604645611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d741026d0 msgr2=0x7f2d74199490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:20.682 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.678+0000 7f2d7259c700 1 --2- 172.21.15.105:0/604645611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d741026d0 0x7f2d74199490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:20.682 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.678+0000 7f2d7259c700 1 -- 172.21.15.105:0/604645611 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d741a0b40 con 0x7f2d74101d30 2024-04-23T14:40:20.682 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.678+0000 7f2d72d9d700 1 --2- 172.21.15.105:0/604645611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d74102ff0 0x7f2d741a04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:40:20.682 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.678+0000 7f2d7259c700 1 --2- 172.21.15.105:0/604645611 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d74101d30 0x7f2d74198f50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2d6400b530 tx=0x7f2d6400b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:40:20.682 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.678+0000 7f2d637fe700 1 -- 172.21.15.105:0/604645611 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d64015020 con 0x7f2d74101d30 2024-04-23T14:40:20.682 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.678+0000 7f2d78c63700 1 -- 172.21.15.105:0/604645611 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d741a0e30 con 0x7f2d74101d30 2024-04-23T14:40:20.682 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.678+0000 7f2d637fe700 1 -- 172.21.15.105:0/604645611 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d64011d10 con 0x7f2d74101d30 2024-04-23T14:40:20.682 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.678+0000 7f2d637fe700 1 -- 172.21.15.105:0/604645611 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d6400f2e0 con 0x7f2d74101d30 2024-04-23T14:40:20.682 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.678+0000 7f2d78c63700 1 -- 172.21.15.105:0/604645611 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d741a1300 con 0x7f2d74101d30 2024-04-23T14:40:20.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.679+0000 7f2d78c63700 1 -- 172.21.15.105:0/604645611 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d741a15c0 con 0x7f2d74101d30 2024-04-23T14:40:20.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.680+0000 7f2d637fe700 1 -- 172.21.15.105:0/604645611 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2d64018020 con 0x7f2d74101d30 2024-04-23T14:40:20.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.681+0000 7f2d637fe700 1 --2- 172.21.15.105:0/604645611 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2d5c0a3c40 0x7f2d5c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:20.684 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.681+0000 7f2d637fe700 1 -- 172.21.15.105:0/604645611 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6723+0+0 (secure 0 0 0) 0x7f2d640cfb00 con 0x7f2d74101d30 2024-04-23T14:40:20.684 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.681+0000 7f2d71d9b700 1 --2- 172.21.15.105:0/604645611 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2d5c0a3c40 0x7f2d5c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:40:20.684 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.682+0000 7f2d71d9b700 1 --2- 172.21.15.105:0/604645611 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2d5c0a3c40 0x7f2d5c0a6100 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2d68003890 tx=0x7f2d68000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:40:20.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.685+0000 7f2d637fe700 1 -- 172.21.15.105:0/604645611 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2d6401f020 con 0x7f2d74101d30 2024-04-23T14:40:20.893 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:20.892+0000 7f2d78c63700 1 -- 172.21.15.105:0/604645611 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f2d740fe960 con 0x7f2d5c0a3c40 2024-04-23T14:40:21.534 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:21 smithi105 ceph-mon[32555]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-23T14:40:21.534 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:21 smithi105 ceph-mon[32555]: osdmap e74: 11 total, 11 up, 11 in 2024-04-23T14:40:21.534 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:21 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:40:21.534 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:21 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:40:21.534 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:21 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:40:21.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:21 smithi038 ceph-mon[32348]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-23T14:40:21.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:21 smithi038 ceph-mon[32348]: osdmap e74: 11 total, 11 up, 11 in 2024-04-23T14:40:21.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:21 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:40:21.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:21 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:40:21.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:21 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:40:21.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:21 smithi028 ceph-mon[27810]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-23T14:40:21.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:21 smithi028 ceph-mon[27810]: osdmap e74: 11 total, 11 up, 11 in 2024-04-23T14:40:21.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:21 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T14:40:21.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:21 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T14:40:21.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:21 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:40:22.546 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:22 smithi105 ceph-mon[32555]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:40:22.546 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:22 smithi105 ceph-mon[32555]: osdmap e75: 11 total, 11 up, 11 in 2024-04-23T14:40:22.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:22 smithi038 ceph-mon[32348]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:40:22.667 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:22 smithi038 ceph-mon[32348]: osdmap e75: 11 total, 11 up, 11 in 2024-04-23T14:40:22.668 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:22 smithi028 ceph-mon[27810]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:40:22.668 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:22 smithi028 ceph-mon[27810]: osdmap e75: 11 total, 11 up, 11 in 2024-04-23T14:40:23.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:23 smithi038 ceph-mon[32348]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-23T14:40:23.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:23 smithi038 ceph-mon[32348]: osdmap e76: 11 total, 11 up, 11 in 2024-04-23T14:40:23.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:23 smithi028 ceph-mon[27810]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-23T14:40:23.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:23 smithi028 ceph-mon[27810]: osdmap e76: 11 total, 11 up, 11 in 2024-04-23T14:40:23.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:23 smithi105 ceph-mon[32555]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-23T14:40:23.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:23 smithi105 ceph-mon[32555]: osdmap e76: 11 total, 11 up, 11 in 2024-04-23T14:40:24.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:24 smithi038 ceph-mon[32348]: from='client.? 172.21.15.105:0/2132706146' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8fc9454-5540-4e2e-91bf-3ba251dbd1a8"}]: dispatch 2024-04-23T14:40:24.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:24 smithi038 ceph-mon[32348]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8fc9454-5540-4e2e-91bf-3ba251dbd1a8"}]: dispatch 2024-04-23T14:40:24.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:24 smithi038 ceph-mon[32348]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e8fc9454-5540-4e2e-91bf-3ba251dbd1a8"}]': finished 2024-04-23T14:40:24.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:24 smithi038 ceph-mon[32348]: osdmap e77: 12 total, 11 up, 12 in 2024-04-23T14:40:24.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:24 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T14:40:24.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:24 smithi038 ceph-mon[32348]: from='client.? 172.21.15.105:0/3782048637' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:40:24.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:24 smithi028 ceph-mon[27810]: from='client.? 172.21.15.105:0/2132706146' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8fc9454-5540-4e2e-91bf-3ba251dbd1a8"}]: dispatch 2024-04-23T14:40:24.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:24 smithi028 ceph-mon[27810]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8fc9454-5540-4e2e-91bf-3ba251dbd1a8"}]: dispatch 2024-04-23T14:40:24.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:24 smithi028 ceph-mon[27810]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e8fc9454-5540-4e2e-91bf-3ba251dbd1a8"}]': finished 2024-04-23T14:40:24.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:24 smithi028 ceph-mon[27810]: osdmap e77: 12 total, 11 up, 12 in 2024-04-23T14:40:24.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:24 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T14:40:24.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:24 smithi028 ceph-mon[27810]: from='client.? 172.21.15.105:0/3782048637' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:40:24.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:24 smithi105 ceph-mon[32555]: from='client.? 172.21.15.105:0/2132706146' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8fc9454-5540-4e2e-91bf-3ba251dbd1a8"}]: dispatch 2024-04-23T14:40:24.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:24 smithi105 ceph-mon[32555]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8fc9454-5540-4e2e-91bf-3ba251dbd1a8"}]: dispatch 2024-04-23T14:40:24.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:24 smithi105 ceph-mon[32555]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e8fc9454-5540-4e2e-91bf-3ba251dbd1a8"}]': finished 2024-04-23T14:40:24.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:24 smithi105 ceph-mon[32555]: osdmap e77: 12 total, 11 up, 12 in 2024-04-23T14:40:24.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:24 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T14:40:24.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:24 smithi105 ceph-mon[32555]: from='client.? 172.21.15.105:0/3782048637' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T14:40:25.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:25 smithi038 ceph-mon[32348]: pgmap v268: 1 pgs: 1 peering; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-23T14:40:25.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:25 smithi028 ceph-mon[27810]: pgmap v268: 1 pgs: 1 peering; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-23T14:40:25.688 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:25 smithi105 ceph-mon[32555]: pgmap v268: 1 pgs: 1 peering; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-23T14:40:27.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:27 smithi038 ceph-mon[32348]: pgmap v269: 1 pgs: 1 peering; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-04-23T14:40:27.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:27 smithi028 ceph-mon[27810]: pgmap v269: 1 pgs: 1 peering; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-04-23T14:40:27.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:27 smithi105 ceph-mon[32555]: pgmap v269: 1 pgs: 1 peering; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-04-23T14:40:29.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:29 smithi038 ceph-mon[32348]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-23T14:40:29.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:29 smithi028 ceph-mon[27810]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-23T14:40:29.707 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:29 smithi105 ceph-mon[32555]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-23T14:40:31.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:31 smithi105 ceph-mon[32555]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-23T14:40:31.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:31 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-23T14:40:31.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:31 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:40:31.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:31 smithi038 ceph-mon[32348]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-23T14:40:31.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:31 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-23T14:40:31.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:31 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:40:31.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:31 smithi028 ceph-mon[27810]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-23T14:40:31.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:31 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-23T14:40:31.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:31 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:40:32.535 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:32 smithi105 ceph-mon[32555]: Deploying daemon osd.11 on smithi105 2024-04-23T14:40:32.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:32 smithi038 ceph-mon[32348]: Deploying daemon osd.11 on smithi105 2024-04-23T14:40:32.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:32 smithi028 ceph-mon[27810]: Deploying daemon osd.11 on smithi105 2024-04-23T14:40:33.436 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:33 smithi105 ceph-mon[32555]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-23T14:40:33.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:33 smithi038 ceph-mon[32348]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-23T14:40:33.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:33 smithi028 ceph-mon[27810]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-23T14:40:35.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:35 smithi038 ceph-mon[32348]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-23T14:40:35.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:35 smithi028 ceph-mon[27810]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-23T14:40:35.729 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:35 smithi105 ceph-mon[32555]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-23T14:40:37.431 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:37 smithi105 ceph-mon[32555]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-23T14:40:37.431 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:37 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:37.431 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:37 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:37.431 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:37 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:40:37.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:37 smithi038 ceph-mon[32348]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-23T14:40:37.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:37 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:37.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:37 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:37.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:37 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:40:37.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:37 smithi028 ceph-mon[27810]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-23T14:40:37.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:37 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:37.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:37 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:37.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:37 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:40:39.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:39 smithi038 ceph-mon[32348]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-23T14:40:39.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:39 smithi028 ceph-mon[27810]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-23T14:40:39.663 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:39 smithi105 ceph-mon[32555]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-23T14:40:40.733 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:40.731+0000 7f2d637fe700 1 -- 172.21.15.105:0/604645611 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f2d740fe960 con 0x7f2d5c0a3c40 2024-04-23T14:40:40.733 INFO:teuthology.orchestra.run.smithi105.stdout:Created osd(s) 11 on host 'smithi105' 2024-04-23T14:40:40.735 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:40.733+0000 7f2d78c63700 1 -- 172.21.15.105:0/604645611 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2d5c0a3c40 msgr2=0x7f2d5c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:40.735 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:40.733+0000 7f2d78c63700 1 --2- 172.21.15.105:0/604645611 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2d5c0a3c40 0x7f2d5c0a6100 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2d68003890 tx=0x7f2d68000bc0 comp rx=0 tx=0).stop 2024-04-23T14:40:40.735 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:40.733+0000 7f2d78c63700 1 -- 172.21.15.105:0/604645611 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d74101d30 msgr2=0x7f2d74198f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:40.735 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:40.733+0000 7f2d78c63700 1 --2- 172.21.15.105:0/604645611 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d74101d30 0x7f2d74198f50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2d6400b530 tx=0x7f2d6400b8f0 comp rx=0 tx=0).stop 2024-04-23T14:40:40.735 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:40.733+0000 7f2d78c63700 1 -- 172.21.15.105:0/604645611 shutdown_connections 2024-04-23T14:40:40.736 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:40.733+0000 7f2d78c63700 1 --2- 172.21.15.105:0/604645611 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2d5c0a3c40 0x7f2d5c0a6100 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:40.736 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:40.733+0000 7f2d78c63700 1 --2- 172.21.15.105:0/604645611 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d74101d30 0x7f2d74198f50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:40.736 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:40.733+0000 7f2d78c63700 1 --2- 172.21.15.105:0/604645611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d74102ff0 0x7f2d741a04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:40.736 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:40.733+0000 7f2d78c63700 1 --2- 172.21.15.105:0/604645611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d741026d0 0x7f2d74199490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:40.736 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:40.733+0000 7f2d78c63700 1 -- 172.21.15.105:0/604645611 >> 172.21.15.105:0/604645611 conn(0x7f2d740fc760 msgr2=0x7f2d740fe250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:40:40.736 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:40.733+0000 7f2d78c63700 1 -- 172.21.15.105:0/604645611 shutdown_connections 2024-04-23T14:40:40.736 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:40.733+0000 7f2d78c63700 1 -- 172.21.15.105:0/604645611 wait complete. 2024-04-23T14:40:41.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:40 smithi105 ceph-mon[32555]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-23T14:40:41.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:40 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:41.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:40 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:41.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:40 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:40:41.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:40 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:40:41.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:40 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:41.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:40 smithi105 ceph-mon[32555]: from='osd.11 [v2:172.21.15.105:6824/3400299562,v1:172.21.15.105:6825/3400299562]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T14:40:41.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:40 smithi105 ceph-mon[32555]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T14:40:41.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:40 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:41.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:40 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:41.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:40 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:40:41.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:40 smithi038 ceph-mon[32348]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-23T14:40:41.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:40 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:41.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:40 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:41.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:40 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:40:41.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:40 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:40:41.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:40 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:41.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:40 smithi038 ceph-mon[32348]: from='osd.11 [v2:172.21.15.105:6824/3400299562,v1:172.21.15.105:6825/3400299562]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T14:40:41.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:40 smithi038 ceph-mon[32348]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T14:40:41.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:40 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:41.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:40 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:41.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:40 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:40:41.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:40 smithi028 ceph-mon[27810]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-23T14:40:41.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:40 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:41.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:40 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:41.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:40 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:40:41.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:40 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:40:41.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:40 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:41.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:40 smithi028 ceph-mon[27810]: from='osd.11 [v2:172.21.15.105:6824/3400299562,v1:172.21.15.105:6825/3400299562]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T14:40:41.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:40 smithi028 ceph-mon[27810]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T14:40:41.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:40 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:41.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:40 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:41.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:40 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:40:41.428 DEBUG:teuthology.orchestra.run.smithi105:osd.11> sudo journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.11.service 2024-04-23T14:40:41.431 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-23T14:40:41.431 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-23T14:40:41.926 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:40:42.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:42 smithi038 ceph-mon[32348]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-23T14:40:42.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:42 smithi038 ceph-mon[32348]: osdmap e78: 12 total, 11 up, 12 in 2024-04-23T14:40:42.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:42 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T14:40:42.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:42 smithi038 ceph-mon[32348]: from='osd.11 [v2:172.21.15.105:6824/3400299562,v1:172.21.15.105:6825/3400299562]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-23T14:40:42.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:42 smithi038 ceph-mon[32348]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-23T14:40:42.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:42 smithi028 ceph-mon[27810]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-23T14:40:42.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:42 smithi028 ceph-mon[27810]: osdmap e78: 12 total, 11 up, 12 in 2024-04-23T14:40:42.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:42 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T14:40:42.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:42 smithi028 ceph-mon[27810]: from='osd.11 [v2:172.21.15.105:6824/3400299562,v1:172.21.15.105:6825/3400299562]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-23T14:40:42.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:42 smithi028 ceph-mon[27810]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-23T14:40:42.685 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:42 smithi105 ceph-mon[32555]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-23T14:40:42.685 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:42 smithi105 ceph-mon[32555]: osdmap e78: 12 total, 11 up, 12 in 2024-04-23T14:40:42.685 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:42 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T14:40:42.685 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:42 smithi105 ceph-mon[32555]: from='osd.11 [v2:172.21.15.105:6824/3400299562,v1:172.21.15.105:6825/3400299562]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-23T14:40:42.685 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:42 smithi105 ceph-mon[32555]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-23T14:40:42.686 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 14:40:42 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T14:40:42.369+0000 7f67b4d76700 -1 osd.11 0 waiting for initial osdmap 2024-04-23T14:40:42.686 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 14:40:42 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T14:40:42.386+0000 7f67b0b69700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T14:40:43.748 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:43 smithi105 ceph-mon[32555]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-23T14:40:43.748 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:43 smithi105 ceph-mon[32555]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-04-23T14:40:43.748 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:43 smithi105 ceph-mon[32555]: osdmap e79: 12 total, 11 up, 12 in 2024-04-23T14:40:43.748 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:43 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T14:40:43.748 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:43 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T14:40:43.748 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:43 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T14:40:43.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:43 smithi028 ceph-mon[27810]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-23T14:40:43.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:43 smithi028 ceph-mon[27810]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-04-23T14:40:43.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:43 smithi028 ceph-mon[27810]: osdmap e79: 12 total, 11 up, 12 in 2024-04-23T14:40:43.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:43 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T14:40:43.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:43 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T14:40:43.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:43 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T14:40:43.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:43 smithi038 ceph-mon[32348]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-23T14:40:43.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:43 smithi038 ceph-mon[32348]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-04-23T14:40:43.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:43 smithi038 ceph-mon[32348]: osdmap e79: 12 total, 11 up, 12 in 2024-04-23T14:40:43.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:43 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T14:40:43.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:43 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T14:40:43.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:43 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T14:40:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.415+0000 7f8cdc000700 1 -- 172.21.15.28:0/231337663 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8cd4108840 msgr2=0x7f8cd410acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.415+0000 7f8cdc000700 1 --2- 172.21.15.28:0/231337663 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8cd4108840 0x7f8cd410acb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc00b0a0 tx=0x7f8ccc00b3b0 comp rx=0 tx=0).stop 2024-04-23T14:40:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.415+0000 7f8cdc000700 1 -- 172.21.15.28:0/231337663 shutdown_connections 2024-04-23T14:40:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.415+0000 7f8cdc000700 1 --2- 172.21.15.28:0/231337663 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8cd4108840 0x7f8cd410acb0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.415+0000 7f8cdc000700 1 --2- 172.21.15.28:0/231337663 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cd41035e0 0x7f8cd41059d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.415+0000 7f8cdc000700 1 --2- 172.21.15.28:0/231337663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cd4105f10 0x7f8cd4108300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.415+0000 7f8cdc000700 1 -- 172.21.15.28:0/231337663 >> 172.21.15.28:0/231337663 conn(0x7f8cd40fd010 msgr2=0x7f8cd40ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:40:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.415+0000 7f8cdc000700 1 -- 172.21.15.28:0/231337663 shutdown_connections 2024-04-23T14:40:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.415+0000 7f8cdc000700 1 -- 172.21.15.28:0/231337663 wait complete. 2024-04-23T14:40:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.416+0000 7f8cdc000700 1 Processor -- start 2024-04-23T14:40:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.416+0000 7f8cdc000700 1 -- start start 2024-04-23T14:40:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.417+0000 7f8cdc000700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cd41035e0 0x7f8cd419af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.417+0000 7f8cdc000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cd4105f10 0x7f8cd419b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.417+0000 7f8cdc000700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8cd4108840 0x7f8cd41a2550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.417+0000 7f8cdc000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8cd410e770 con 0x7f8cd4105f10 2024-04-23T14:40:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.417+0000 7f8cdc000700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8cd410e5f0 con 0x7f8cd41035e0 2024-04-23T14:40:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.417+0000 7f8cdc000700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8cd410e8f0 con 0x7f8cd4108840 2024-04-23T14:40:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.417+0000 7f8cd8c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cd4105f10 0x7f8cd419b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:40:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.417+0000 7f8cd8c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cd4105f10 0x7f8cd419b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58544/0 (socket says 172.21.15.28:58544) 2024-04-23T14:40:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.417+0000 7f8cd8c00700 1 -- 172.21.15.28:0/3931844557 learned_addr learned my addr 172.21.15.28:0/3931844557 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:40:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.417+0000 7f8cda000700 1 --2- 172.21.15.28:0/3931844557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8cd4108840 0x7f8cd41a2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:40:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.417+0000 7f8cd9600700 1 --2- 172.21.15.28:0/3931844557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cd41035e0 0x7f8cd419af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:40:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.417+0000 7f8cda000700 1 -- 172.21.15.28:0/3931844557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cd41035e0 msgr2=0x7f8cd419af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.417+0000 7f8cda000700 1 --2- 172.21.15.28:0/3931844557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cd41035e0 0x7f8cd419af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.417+0000 7f8cda000700 1 -- 172.21.15.28:0/3931844557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cd4105f10 msgr2=0x7f8cd419b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.417+0000 7f8cda000700 1 --2- 172.21.15.28:0/3931844557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cd4105f10 0x7f8cd419b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.417+0000 7f8cda000700 1 -- 172.21.15.28:0/3931844557 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cd41a2bf0 con 0x7f8cd4108840 2024-04-23T14:40:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.418+0000 7f8cda000700 1 --2- 172.21.15.28:0/3931844557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8cd4108840 0x7f8cd41a2550 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc007fa0 tx=0x7f8ccc004c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:40:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.418+0000 7f8cd2000700 1 -- 172.21.15.28:0/3931844557 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc021070 con 0x7f8cd4108840 2024-04-23T14:40:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.418+0000 7f8cdc000700 1 -- 172.21.15.28:0/3931844557 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cd41a2e80 con 0x7f8cd4108840 2024-04-23T14:40:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.418+0000 7f8cd2000700 1 -- 172.21.15.28:0/3931844557 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ccc007b80 con 0x7f8cd4108840 2024-04-23T14:40:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.418+0000 7f8cd2000700 1 -- 172.21.15.28:0/3931844557 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc004750 con 0x7f8cd4108840 2024-04-23T14:40:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.418+0000 7f8cdc000700 1 -- 172.21.15.28:0/3931844557 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cd41a32e0 con 0x7f8cd4108840 2024-04-23T14:40:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.420+0000 7f8cdc000700 1 -- 172.21.15.28:0/3931844557 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cd40697e0 con 0x7f8cd4108840 2024-04-23T14:40:44.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.421+0000 7f8cd2000700 1 -- 172.21.15.28:0/3931844557 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8ccc003a00 con 0x7f8cd4108840 2024-04-23T14:40:44.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.423+0000 7f8cd2000700 1 --2- 172.21.15.28:0/3931844557 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f8cbc0a3c40 0x7f8cbc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:44.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.423+0000 7f8cd2000700 1 -- 172.21.15.28:0/3931844557 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f8ccc00f020 con 0x7f8cd4108840 2024-04-23T14:40:44.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.423+0000 7f8cd9600700 1 --2- 172.21.15.28:0/3931844557 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f8cbc0a3c40 0x7f8cbc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:40:44.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.424+0000 7f8cd9600700 1 --2- 172.21.15.28:0/3931844557 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f8cbc0a3c40 0x7f8cbc0a6100 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8009c60 tx=0x7f8cc8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:40:44.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.427+0000 7f8cd2000700 1 -- 172.21.15.28:0/3931844557 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8ccc02a020 con 0x7f8cd4108840 2024-04-23T14:40:44.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:44 smithi028 ceph-mon[27810]: purged_snaps scrub starts 2024-04-23T14:40:44.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:44 smithi028 ceph-mon[27810]: purged_snaps scrub ok 2024-04-23T14:40:44.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:44 smithi028 ceph-mon[27810]: osd.11 [v2:172.21.15.105:6824/3400299562,v1:172.21.15.105:6825/3400299562] boot 2024-04-23T14:40:44.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:44 smithi028 ceph-mon[27810]: osdmap e80: 12 total, 12 up, 12 in 2024-04-23T14:40:44.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:44 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T14:40:44.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.706+0000 7f8cdc000700 1 -- 172.21.15.28:0/3931844557 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f8cd4063700 con 0x7f8cd4108840 2024-04-23T14:40:44.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.707+0000 7f8cd2000700 1 -- 172.21.15.28:0/3931844557 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7f8ccc09e250 con 0x7f8cd4108840 2024-04-23T14:40:44.708 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:40:44.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.711+0000 7f8cdc000700 1 -- 172.21.15.28:0/3931844557 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f8cbc0a3c40 msgr2=0x7f8cbc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:44.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.711+0000 7f8cdc000700 1 --2- 172.21.15.28:0/3931844557 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f8cbc0a3c40 0x7f8cbc0a6100 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8009c60 tx=0x7f8cc8009260 comp rx=0 tx=0).stop 2024-04-23T14:40:44.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.711+0000 7f8cdc000700 1 -- 172.21.15.28:0/3931844557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8cd4108840 msgr2=0x7f8cd41a2550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:44.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.711+0000 7f8cdc000700 1 --2- 172.21.15.28:0/3931844557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8cd4108840 0x7f8cd41a2550 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc007fa0 tx=0x7f8ccc004c80 comp rx=0 tx=0).stop 2024-04-23T14:40:44.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.711+0000 7f8cdc000700 1 -- 172.21.15.28:0/3931844557 shutdown_connections 2024-04-23T14:40:44.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.711+0000 7f8cdc000700 1 --2- 172.21.15.28:0/3931844557 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f8cbc0a3c40 0x7f8cbc0a6100 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:44.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.711+0000 7f8cdc000700 1 --2- 172.21.15.28:0/3931844557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8cd4108840 0x7f8cd41a2550 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:44.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.711+0000 7f8cdc000700 1 --2- 172.21.15.28:0/3931844557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cd41035e0 0x7f8cd419af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:44.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.711+0000 7f8cdc000700 1 --2- 172.21.15.28:0/3931844557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cd4105f10 0x7f8cd419b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:44.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.711+0000 7f8cdc000700 1 -- 172.21.15.28:0/3931844557 >> 172.21.15.28:0/3931844557 conn(0x7f8cd40fd010 msgr2=0x7f8cd4102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:40:44.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.711+0000 7f8cdc000700 1 -- 172.21.15.28:0/3931844557 shutdown_connections 2024-04-23T14:40:44.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:44.711+0000 7f8cdc000700 1 -- 172.21.15.28:0/3931844557 wait complete. 2024-04-23T14:40:44.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:44 smithi105 ceph-mon[32555]: purged_snaps scrub starts 2024-04-23T14:40:44.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:44 smithi105 ceph-mon[32555]: purged_snaps scrub ok 2024-04-23T14:40:44.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:44 smithi105 ceph-mon[32555]: osd.11 [v2:172.21.15.105:6824/3400299562,v1:172.21.15.105:6825/3400299562] boot 2024-04-23T14:40:44.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:44 smithi105 ceph-mon[32555]: osdmap e80: 12 total, 12 up, 12 in 2024-04-23T14:40:44.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:44 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T14:40:44.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:44 smithi038 ceph-mon[32348]: purged_snaps scrub starts 2024-04-23T14:40:44.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:44 smithi038 ceph-mon[32348]: purged_snaps scrub ok 2024-04-23T14:40:44.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:44 smithi038 ceph-mon[32348]: osd.11 [v2:172.21.15.105:6824/3400299562,v1:172.21.15.105:6825/3400299562] boot 2024-04-23T14:40:44.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:44 smithi038 ceph-mon[32348]: osdmap e80: 12 total, 12 up, 12 in 2024-04-23T14:40:44.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:44 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T14:40:45.472 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1713883243,"num_in_osds":12,"osd_in_since":1713883223,"num_remapped_pgs":0} 2024-04-23T14:40:45.473 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-23T14:40:45.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:45 smithi028 ceph-mon[27810]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-23T14:40:45.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:45 smithi028 ceph-mon[27810]: osdmap e81: 12 total, 12 up, 12 in 2024-04-23T14:40:45.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:45 smithi028 ceph-mon[27810]: Detected new or changed devices on smithi105 2024-04-23T14:40:45.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:45 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:45.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:45 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:45.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:45 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:40:45.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:45 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:40:45.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:45 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:40:45.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:45 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:40:45.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:45 smithi028 ceph-mon[27810]: Adjusting osd_memory_target on smithi105 to 4272M 2024-04-23T14:40:45.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:45 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:45.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:45 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:40:45.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:45 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:40:45.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:45 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:45.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:45 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3931844557' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-23T14:40:45.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:45 smithi105 ceph-mon[32555]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-23T14:40:45.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:45 smithi105 ceph-mon[32555]: osdmap e81: 12 total, 12 up, 12 in 2024-04-23T14:40:45.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:45 smithi105 ceph-mon[32555]: Detected new or changed devices on smithi105 2024-04-23T14:40:45.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:45 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:45.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:45 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:45.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:45 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:40:45.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:45 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:40:45.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:45 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:40:45.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:45 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:40:45.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:45 smithi105 ceph-mon[32555]: Adjusting osd_memory_target on smithi105 to 4272M 2024-04-23T14:40:45.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:45 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:45.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:45 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:40:45.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:45 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:40:45.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:45 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:45.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:45 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3931844557' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-23T14:40:45.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:45 smithi038 ceph-mon[32348]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-23T14:40:45.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:45 smithi038 ceph-mon[32348]: osdmap e81: 12 total, 12 up, 12 in 2024-04-23T14:40:45.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:45 smithi038 ceph-mon[32348]: Detected new or changed devices on smithi105 2024-04-23T14:40:45.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:45 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:45.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:45 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:45.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:45 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:40:45.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:45 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:40:45.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:45 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:40:45.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:45 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:40:45.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:45 smithi038 ceph-mon[32348]: Adjusting osd_memory_target on smithi105 to 4272M 2024-04-23T14:40:45.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:45 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:45.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:45 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:40:45.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:45 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:40:45.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:45 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:45.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:45 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3931844557' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-23T14:40:45.898 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:40:47.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:47 smithi028 ceph-mon[27810]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:40:47.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:47 smithi105 ceph-mon[32555]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:40:47.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:47 smithi038 ceph-mon[32348]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:40:48.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.137+0000 7f939f200700 1 -- 172.21.15.28:0/4206077400 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93a0104500 msgr2=0x7f93a010c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:48.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.137+0000 7f939f200700 1 --2- 172.21.15.28:0/4206077400 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93a0104500 0x7f93a010c9f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f938c009990 tx=0x7f938c009ca0 comp rx=0 tx=0).stop 2024-04-23T14:40:48.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.137+0000 7f939f200700 1 -- 172.21.15.28:0/4206077400 shutdown_connections 2024-04-23T14:40:48.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.137+0000 7f939f200700 1 --2- 172.21.15.28:0/4206077400 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f93a010d010 0x7f93a010f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:48.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.137+0000 7f939f200700 1 --2- 172.21.15.28:0/4206077400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93a0103be0 0x7f93a0103fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:48.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.137+0000 7f939f200700 1 --2- 172.21.15.28:0/4206077400 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93a0104500 0x7f93a010c9f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:48.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.137+0000 7f939f200700 1 -- 172.21.15.28:0/4206077400 >> 172.21.15.28:0/4206077400 conn(0x7f93a00fd410 msgr2=0x7f93a00ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:40:48.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.137+0000 7f939f200700 1 -- 172.21.15.28:0/4206077400 shutdown_connections 2024-04-23T14:40:48.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.137+0000 7f939f200700 1 -- 172.21.15.28:0/4206077400 wait complete. 2024-04-23T14:40:48.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.138+0000 7f939f200700 1 Processor -- start 2024-04-23T14:40:48.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.139+0000 7f939f200700 1 -- start start 2024-04-23T14:40:48.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.139+0000 7f939f200700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f93a0103be0 0x7f93a0101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:48.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.139+0000 7f939f200700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93a0104500 0x7f93a0101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:48.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.139+0000 7f939f200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93a010d010 0x7f93a01022c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:48.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.139+0000 7f939f200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f93a0112da0 con 0x7f93a010d010 2024-04-23T14:40:48.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.139+0000 7f939f200700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f93a0112c20 con 0x7f93a0104500 2024-04-23T14:40:48.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.139+0000 7f939f200700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f93a0112f20 con 0x7f93a0103be0 2024-04-23T14:40:48.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.139+0000 7f939e800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93a010d010 0x7f93a01022c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:40:48.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.139+0000 7f939e800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93a010d010 0x7f93a01022c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43106/0 (socket says 172.21.15.28:43106) 2024-04-23T14:40:48.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.139+0000 7f939e800700 1 -- 172.21.15.28:0/3586658077 learned_addr learned my addr 172.21.15.28:0/3586658077 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:40:48.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.139+0000 7f939d400700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93a0104500 0x7f93a0101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:40:48.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.140+0000 7f939de00700 1 --2- 172.21.15.28:0/3586658077 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f93a0103be0 0x7f93a0101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:40:48.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.140+0000 7f939d400700 1 -- 172.21.15.28:0/3586658077 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f93a0103be0 msgr2=0x7f93a0101840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:48.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.140+0000 7f939d400700 1 --2- 172.21.15.28:0/3586658077 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f93a0103be0 0x7f93a0101840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:48.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.140+0000 7f939d400700 1 -- 172.21.15.28:0/3586658077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93a010d010 msgr2=0x7f93a01022c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:48.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.140+0000 7f939d400700 1 --2- 172.21.15.28:0/3586658077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93a010d010 0x7f93a01022c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:48.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.140+0000 7f939d400700 1 -- 172.21.15.28:0/3586658077 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93a01a7770 con 0x7f93a0104500 2024-04-23T14:40:48.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.140+0000 7f939d400700 1 --2- 172.21.15.28:0/3586658077 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93a0104500 0x7f93a0101d80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f938c0098e0 tx=0x7f938c00b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:40:48.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.140+0000 7f9396a00700 1 -- 172.21.15.28:0/3586658077 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f938c027070 con 0x7f93a0104500 2024-04-23T14:40:48.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.141+0000 7f9396a00700 1 -- 172.21.15.28:0/3586658077 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f938c003b60 con 0x7f93a0104500 2024-04-23T14:40:48.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.141+0000 7f939f200700 1 -- 172.21.15.28:0/3586658077 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93a01a7a00 con 0x7f93a0104500 2024-04-23T14:40:48.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.141+0000 7f9396a00700 1 -- 172.21.15.28:0/3586658077 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f938c022d70 con 0x7f93a0104500 2024-04-23T14:40:48.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.141+0000 7f939f200700 1 -- 172.21.15.28:0/3586658077 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93a01a7ee0 con 0x7f93a0104500 2024-04-23T14:40:48.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.142+0000 7f939f200700 1 -- 172.21.15.28:0/3586658077 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93a00697e0 con 0x7f93a0104500 2024-04-23T14:40:48.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.144+0000 7f9396a00700 1 -- 172.21.15.28:0/3586658077 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f938c02a070 con 0x7f93a0104500 2024-04-23T14:40:48.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.145+0000 7f9396a00700 1 --2- 172.21.15.28:0/3586658077 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f93840a3c40 0x7f93840a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:48.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.145+0000 7f9396a00700 1 -- 172.21.15.28:0/3586658077 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f938c0d0d70 con 0x7f93a0104500 2024-04-23T14:40:48.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.145+0000 7f939de00700 1 --2- 172.21.15.28:0/3586658077 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f93840a3c40 0x7f93840a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:40:48.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.146+0000 7f939de00700 1 --2- 172.21.15.28:0/3586658077 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f93840a3c40 0x7f93840a6100 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9390009c60 tx=0x7f9390009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:40:48.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.150+0000 7f9396a00700 1 -- 172.21.15.28:0/3586658077 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f938c05eff0 con 0x7f93a0104500 2024-04-23T14:40:48.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.382+0000 7f939f200700 1 -- 172.21.15.28:0/3586658077 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f93a002dfe0 con 0x7f93a0104500 2024-04-23T14:40:48.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.384+0000 7f9396a00700 1 -- 172.21.15.28:0/3586658077 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18407 (secure 0 0 0) 0x7f938c013070 con 0x7f93a0104500 2024-04-23T14:40:48.387 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:40:48.387 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":81,"fsid":"1ee6e996-017e-11ef-bc93-c7b262605968","created":"2024-04-23T14:32:34.581082+0000","modified":"2024-04-23T14:40:44.389733+0000","last_up_change":"2024-04-23T14:40:43.375846+0000","last_in_change":"2024-04-23T14:40:23.576939+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:36:27.698112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e0b427bc-be65-4543-99cc-2f53ad98ddad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6803","nonce":2993445336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6805","nonce":2993445336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6809","nonce":2993445336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6807","nonce":2993445336}]},"public_addr":"172.21.15.28:6803/2993445336","cluster_addr":"172.21.15.28:6805/2993445336","heartbeat_back_addr":"172.21.15.28:6809/2993445336","heartbeat_front_addr":"172.21.15.28:6807/2993445336","state":["exists","up"]},{"osd":1,"uuid":"ef527ec9-1bf3-4bb0-8df3-f5a015af557c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6801","nonce":4024093383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6803","nonce":4024093383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6807","nonce":4024093383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6805","nonce":4024093383}]},"public_addr":"172.21.15.38:6801/4024093383","cluster_addr":"172.21.15.38:6803/4024093383","heartbeat_back_addr":"172.21.15.38:6807/4024093383","heartbeat_front_addr":"172.21.15.38:6805/4024093383","state":["exists","up"]},{"osd":2,"uuid":"b33b23d9-adce-4c11-b304-2e117bdf6bfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6801","nonce":570739029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6803","nonce":570739029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6807","nonce":570739029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6805","nonce":570739029}]},"public_addr":"172.21.15.105:6801/570739029","cluster_addr":"172.21.15.105:6803/570739029","heartbeat_back_addr":"172.21.15.105:6807/570739029","heartbeat_front_addr":"172.21.15.105:6805/570739029","state":["exists","up"]},{"osd":3,"uuid":"28bd5608-b426-434e-9db1-bdaa9828bdb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6811","nonce":2480400247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6813","nonce":2480400247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6817","nonce":2480400247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6815","nonce":2480400247}]},"public_addr":"172.21.15.28:6811/2480400247","cluster_addr":"172.21.15.28:6813/2480400247","heartbeat_back_addr":"172.21.15.28:6817/2480400247","heartbeat_front_addr":"172.21.15.28:6815/2480400247","state":["exists","up"]},{"osd":4,"uuid":"e5d1c89b-025f-4d57-8807-59d75c787b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6809","nonce":3772189246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6811","nonce":3772189246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6815","nonce":3772189246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6813","nonce":3772189246}]},"public_addr":"172.21.15.38:6809/3772189246","cluster_addr":"172.21.15.38:6811/3772189246","heartbeat_back_addr":"172.21.15.38:6815/3772189246","heartbeat_front_addr":"172.21.15.38:6813/3772189246","state":["exists","up"]},{"osd":5,"uuid":"9cf9cf4c-aafd-4ed3-bfb1-0527710a0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6809","nonce":928755894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6811","nonce":928755894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6815","nonce":928755894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6813","nonce":928755894}]},"public_addr":"172.21.15.105:6809/928755894","cluster_addr":"172.21.15.105:6811/928755894","heartbeat_back_addr":"172.21.15.105:6815/928755894","heartbeat_front_addr":"172.21.15.105:6813/928755894","state":["exists","up"]},{"osd":6,"uuid":"304db525-d24d-4687-bfa0-f92bdf66d1e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6819","nonce":897979441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6821","nonce":897979441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6825","nonce":897979441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6823","nonce":897979441}]},"public_addr":"172.21.15.28:6819/897979441","cluster_addr":"172.21.15.28:6821/897979441","heartbeat_back_addr":"172.21.15.28:6825/897979441","heartbeat_front_addr":"172.21.15.28:6823/897979441","state":["exists","up"]},{"osd":7,"uuid":"08e01d59-03f3-47b1-b45f-35ec5a50992c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6817","nonce":2400251786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6819","nonce":2400251786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6823","nonce":2400251786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6821","nonce":2400251786}]},"public_addr":"172.21.15.38:6817/2400251786","cluster_addr":"172.21.15.38:6819/2400251786","heartbeat_back_addr":"172.21.15.38:6823/2400251786","heartbeat_front_addr":"172.21.15.38:6821/2400251786","state":["exists","up"]},{"osd":8,"uuid":"64ee1e95-0756-48b1-9c00-472d0c3cbd29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6817","nonce":1682312269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6819","nonce":1682312269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6823","nonce":1682312269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6821","nonce":1682312269}]},"public_addr":"172.21.15.105:6817/1682312269","cluster_addr":"172.21.15.105:6819/1682312269","heartbeat_back_addr":"172.21.15.105:6823/1682312269","heartbeat_front_addr":"172.21.15.105:6821/1682312269","state":["exists","up"]},{"osd":9,"uuid":"18f60893-3ffd-48c2-9db5-a67d35611d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6827","nonce":1292942516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6829","nonce":1292942516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6833","nonce":1292942516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6831","nonce":1292942516}]},"public_addr":"172.21.15.28:6827/1292942516","cluster_addr":"172.21.15.28:6829/1292942516","heartbeat_back_addr":"172.21.15.28:6833/1292942516","heartbeat_front_addr":"172.21.15.28:6831/1292942516","state":["exists","up"]},{"osd":10,"uuid":"76c06725-7040-4118-8894-fe808a87f999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6825","nonce":2444616468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6827","nonce":2444616468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6831","nonce":2444616468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6829","nonce":2444616468}]},"public_addr":"172.21.15.38:6825/2444616468","cluster_addr":"172.21.15.38:6827/2444616468","heartbeat_back_addr":"172.21.15.38:6831/2444616468","heartbeat_front_addr":"172.21.15.38:6829/2444616468","state":["exists","up"]},{"osd":11,"uuid":"e8fc9454-5540-4e2e-91bf-3ba251dbd1a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6825","nonce":3400299562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6827","nonce":3400299562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6831","nonce":3400299562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6829","nonce":3400299562}]},"public_addr":"172.21.15.105:6825/3400299562","cluster_addr":"172.21.15.105:6827/3400299562","heartbeat_back_addr":"172.21.15.105:6831/3400299562","heartbeat_front_addr":"172.21.15.105:6829/3400299562","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:35:33.615395+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:00.545293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:24.940211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:57.684887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:24.519061+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:49.010804+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:22.638478+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:49.559104+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:14.011771+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:47.859950+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:16.298445+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:41.619069+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/590495740":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/1163172512":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/4290328460":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/4145020172":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/720626958":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/948450262":"2024-04-24T14:33:07.576363+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:40:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.387+0000 7f939f200700 1 -- 172.21.15.28:0/3586658077 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f93840a3c40 msgr2=0x7f93840a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.387+0000 7f939f200700 1 --2- 172.21.15.28:0/3586658077 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f93840a3c40 0x7f93840a6100 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9390009c60 tx=0x7f9390009260 comp rx=0 tx=0).stop 2024-04-23T14:40:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.387+0000 7f939f200700 1 -- 172.21.15.28:0/3586658077 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93a0104500 msgr2=0x7f93a0101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.387+0000 7f939f200700 1 --2- 172.21.15.28:0/3586658077 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93a0104500 0x7f93a0101d80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f938c0098e0 tx=0x7f938c00b880 comp rx=0 tx=0).stop 2024-04-23T14:40:48.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.387+0000 7f939f200700 1 -- 172.21.15.28:0/3586658077 shutdown_connections 2024-04-23T14:40:48.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.387+0000 7f939f200700 1 --2- 172.21.15.28:0/3586658077 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f93840a3c40 0x7f93840a6100 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:48.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.387+0000 7f939f200700 1 --2- 172.21.15.28:0/3586658077 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f93a0103be0 0x7f93a0101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:48.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.387+0000 7f939f200700 1 --2- 172.21.15.28:0/3586658077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93a010d010 0x7f93a01022c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:48.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.387+0000 7f939f200700 1 --2- 172.21.15.28:0/3586658077 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93a0104500 0x7f93a0101d80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:48.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.387+0000 7f939f200700 1 -- 172.21.15.28:0/3586658077 >> 172.21.15.28:0/3586658077 conn(0x7f93a00fd410 msgr2=0x7f93a00fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:40:48.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.387+0000 7f939f200700 1 -- 172.21.15.28:0/3586658077 shutdown_connections 2024-04-23T14:40:48.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:48.387+0000 7f939f200700 1 -- 172.21.15.28:0/3586658077 wait complete. 2024-04-23T14:40:48.763 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:48 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3586658077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:40:48.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:48 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3586658077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:40:48.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:48 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3586658077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:40:49.152 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T14:36:27.698112+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T14:40:49.152 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-23T14:40:49.610 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:40:49.762 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:49 smithi028 ceph-mon[27810]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:40:49.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:49 smithi105 ceph-mon[32555]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:40:49.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:49 smithi038 ceph-mon[32348]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:40:51.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:51 smithi028 ceph-mon[27810]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:40:51.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:51 smithi105 ceph-mon[32555]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:40:51.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.858+0000 7f41b8600700 1 -- 172.21.15.28:0/3566673683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41b01019f0 msgr2=0x7f41b0101e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:51.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.858+0000 7f41b8600700 1 --2- 172.21.15.28:0/3566673683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41b01019f0 0x7f41b0101e70 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f419c0098f0 tx=0x7f419c009c00 comp rx=0 tx=0).stop 2024-04-23T14:40:51.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.858+0000 7f41b8600700 1 -- 172.21.15.28:0/3566673683 shutdown_connections 2024-04-23T14:40:51.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.858+0000 7f41b8600700 1 --2- 172.21.15.28:0/3566673683 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41b01023b0 0x7f41b010e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:51.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.858+0000 7f41b8600700 1 --2- 172.21.15.28:0/3566673683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41b0107a40 0x7f41b0107e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:51.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.858+0000 7f41b8600700 1 --2- 172.21.15.28:0/3566673683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41b01019f0 0x7f41b0101e70 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:51.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.858+0000 7f41b8600700 1 -- 172.21.15.28:0/3566673683 >> 172.21.15.28:0/3566673683 conn(0x7f41b00789a0 msgr2=0x7f41b0078db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:40:51.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.858+0000 7f41b8600700 1 -- 172.21.15.28:0/3566673683 shutdown_connections 2024-04-23T14:40:51.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.858+0000 7f41b8600700 1 -- 172.21.15.28:0/3566673683 wait complete. 2024-04-23T14:40:51.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.859+0000 7f41b8600700 1 Processor -- start 2024-04-23T14:40:51.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.859+0000 7f41b8600700 1 -- start start 2024-04-23T14:40:51.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.860+0000 7f41b8600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41b01019f0 0x7f41b01035a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:51.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.860+0000 7f41b8600700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41b01023b0 0x7f41b0103ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:51.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.860+0000 7f41b8600700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41b0107a40 0x7f41b0104020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:51.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.860+0000 7f41b8600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f41b0111d40 con 0x7f41b01019f0 2024-04-23T14:40:51.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.860+0000 7f41b8600700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f41b0111bc0 con 0x7f41b01023b0 2024-04-23T14:40:51.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.860+0000 7f41b8600700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f41b0111ec0 con 0x7f41b0107a40 2024-04-23T14:40:51.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.860+0000 7f41b5c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41b01019f0 0x7f41b01035a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:40:51.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.860+0000 7f41b5c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41b01019f0 0x7f41b01035a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43116/0 (socket says 172.21.15.28:43116) 2024-04-23T14:40:51.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.860+0000 7f41b5c00700 1 -- 172.21.15.28:0/1829028120 learned_addr learned my addr 172.21.15.28:0/1829028120 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:40:51.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.860+0000 7f41b5200700 1 --2- 172.21.15.28:0/1829028120 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41b01023b0 0x7f41b0103ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:40:51.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.860+0000 7f41b6600700 1 --2- 172.21.15.28:0/1829028120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41b0107a40 0x7f41b0104020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:40:51.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.861+0000 7f41b5c00700 1 -- 172.21.15.28:0/1829028120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41b0107a40 msgr2=0x7f41b0104020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:51.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.861+0000 7f41b5c00700 1 --2- 172.21.15.28:0/1829028120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41b0107a40 0x7f41b0104020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:51.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.861+0000 7f41b5c00700 1 -- 172.21.15.28:0/1829028120 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41b01023b0 msgr2=0x7f41b0103ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:51.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.861+0000 7f41b5c00700 1 --2- 172.21.15.28:0/1829028120 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41b01023b0 0x7f41b0103ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:51.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.861+0000 7f41b5c00700 1 -- 172.21.15.28:0/1829028120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41b01a77e0 con 0x7f41b01019f0 2024-04-23T14:40:51.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.861+0000 7f41b5c00700 1 --2- 172.21.15.28:0/1829028120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41b01019f0 0x7f41b01035a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f41a400e9c0 tx=0x7f41a400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:40:51.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.861+0000 7f41ae000700 1 -- 172.21.15.28:0/1829028120 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a4004a30 con 0x7f41b01019f0 2024-04-23T14:40:51.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.861+0000 7f41b8600700 1 -- 172.21.15.28:0/1829028120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41b01a7ad0 con 0x7f41b01019f0 2024-04-23T14:40:51.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.861+0000 7f41ae000700 1 -- 172.21.15.28:0/1829028120 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41a400cca0 con 0x7f41b01019f0 2024-04-23T14:40:51.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.861+0000 7f41ae000700 1 -- 172.21.15.28:0/1829028120 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a4005610 con 0x7f41b01019f0 2024-04-23T14:40:51.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.862+0000 7f41b8600700 1 -- 172.21.15.28:0/1829028120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41b01a8030 con 0x7f41b01019f0 2024-04-23T14:40:51.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.863+0000 7f41b8600700 1 -- 172.21.15.28:0/1829028120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41b00697e0 con 0x7f41b01019f0 2024-04-23T14:40:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.865+0000 7f41ae000700 1 -- 172.21.15.28:0/1829028120 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f41a4004cf0 con 0x7f41b01019f0 2024-04-23T14:40:51.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.866+0000 7f41ae000700 1 --2- 172.21.15.28:0/1829028120 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f41980a3c40 0x7f41980a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:51.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.866+0000 7f41ae000700 1 -- 172.21.15.28:0/1829028120 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f41a40d0080 con 0x7f41b01019f0 2024-04-23T14:40:51.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.867+0000 7f41b5200700 1 --2- 172.21.15.28:0/1829028120 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f41980a3c40 0x7f41980a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:40:51.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.867+0000 7f41b5200700 1 --2- 172.21.15.28:0/1829028120 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f41980a3c40 0x7f41980a6100 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f419c009810 tx=0x7f419c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:40:51.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:51 smithi038 ceph-mon[32348]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:40:51.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:51.872+0000 7f41ae000700 1 -- 172.21.15.28:0/1829028120 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f41a409ce60 con 0x7f41b01019f0 2024-04-23T14:40:52.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:52.104+0000 7f41b8600700 1 -- 172.21.15.28:0/1829028120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f41b002dfe0 con 0x7f41b01019f0 2024-04-23T14:40:52.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:52.105+0000 7f41ae000700 1 -- 172.21.15.28:0/1829028120 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f41a409c760 con 0x7f41b01019f0 2024-04-23T14:40:52.107 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-23T14:40:52.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:52.108+0000 7f41b8600700 1 -- 172.21.15.28:0/1829028120 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f41980a3c40 msgr2=0x7f41980a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:52.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:52.108+0000 7f41b8600700 1 --2- 172.21.15.28:0/1829028120 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f41980a3c40 0x7f41980a6100 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f419c009810 tx=0x7f419c019040 comp rx=0 tx=0).stop 2024-04-23T14:40:52.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:52.108+0000 7f41b8600700 1 -- 172.21.15.28:0/1829028120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41b01019f0 msgr2=0x7f41b01035a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:52.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:52.108+0000 7f41b8600700 1 --2- 172.21.15.28:0/1829028120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41b01019f0 0x7f41b01035a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f41a400e9c0 tx=0x7f41a400ecd0 comp rx=0 tx=0).stop 2024-04-23T14:40:52.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:52.109+0000 7f41b8600700 1 -- 172.21.15.28:0/1829028120 shutdown_connections 2024-04-23T14:40:52.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:52.109+0000 7f41b8600700 1 --2- 172.21.15.28:0/1829028120 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f41980a3c40 0x7f41980a6100 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:52.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:52.109+0000 7f41b8600700 1 --2- 172.21.15.28:0/1829028120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41b0107a40 0x7f41b0104020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:52.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:52.109+0000 7f41b8600700 1 --2- 172.21.15.28:0/1829028120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41b01019f0 0x7f41b01035a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:52.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:52.109+0000 7f41b8600700 1 --2- 172.21.15.28:0/1829028120 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41b01023b0 0x7f41b0103ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:52.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:52.109+0000 7f41b8600700 1 -- 172.21.15.28:0/1829028120 >> 172.21.15.28:0/1829028120 conn(0x7f41b00789a0 msgr2=0x7f41b006ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:40:52.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:52.109+0000 7f41b8600700 1 -- 172.21.15.28:0/1829028120 shutdown_connections 2024-04-23T14:40:52.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:52.109+0000 7f41b8600700 1 -- 172.21.15.28:0/1829028120 wait complete. 2024-04-23T14:40:52.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:52 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1829028120' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:40:52.829 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:52 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1829028120' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:40:52.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:52 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1829028120' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:40:52.910 INFO:tasks.cephadm:Adding mds.a on smithi028 2024-04-23T14:40:52.910 INFO:tasks.cephadm:Adding mds.d on smithi028 2024-04-23T14:40:52.910 INFO:tasks.cephadm:Adding mds.g on smithi028 2024-04-23T14:40:52.910 INFO:tasks.cephadm:Adding mds.j on smithi028 2024-04-23T14:40:52.910 INFO:tasks.cephadm:Adding mds.b on smithi038 2024-04-23T14:40:52.910 INFO:tasks.cephadm:Adding mds.e on smithi038 2024-04-23T14:40:52.910 INFO:tasks.cephadm:Adding mds.h on smithi038 2024-04-23T14:40:52.910 INFO:tasks.cephadm:Adding mds.k on smithi038 2024-04-23T14:40:52.910 INFO:tasks.cephadm:Adding mds.c on smithi105 2024-04-23T14:40:52.910 INFO:tasks.cephadm:Adding mds.f on smithi105 2024-04-23T14:40:52.910 INFO:tasks.cephadm:Adding mds.i on smithi105 2024-04-23T14:40:52.910 INFO:tasks.cephadm:Adding mds.l on smithi105 2024-04-23T14:40:52.913 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi105=c;smithi105=f;smithi105=i;smithi105=l' 2024-04-23T14:40:53.159 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T14:40:53.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:53 smithi105 ceph-mon[32555]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:40:53.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:53 smithi038 ceph-mon[32348]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:40:53.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:53 smithi028 ceph-mon[27810]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:40:54.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.754+0000 7fa4c6a32700 1 -- 172.21.15.105:0/1331819003 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4c01047c0 msgr2=0x7fa4c010b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:54.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.754+0000 7fa4c6a32700 1 --2- 172.21.15.105:0/1331819003 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4c01047c0 0x7fa4c010b070 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa4b800b0a0 tx=0x7fa4b800b3b0 comp rx=0 tx=0).stop 2024-04-23T14:40:54.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.754+0000 7fa4c6a32700 1 -- 172.21.15.105:0/1331819003 shutdown_connections 2024-04-23T14:40:54.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.754+0000 7fa4c6a32700 1 --2- 172.21.15.105:0/1331819003 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4c01047c0 0x7fa4c010b070 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:54.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.754+0000 7fa4c6a32700 1 --2- 172.21.15.105:0/1331819003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4c0100e40 0x7fa4c0101260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:54.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.754+0000 7fa4c6a32700 1 --2- 172.21.15.105:0/1331819003 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4c0103e20 0x7fa4c0104280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:54.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.754+0000 7fa4c6a32700 1 -- 172.21.15.105:0/1331819003 >> 172.21.15.105:0/1331819003 conn(0x7fa4c00fc7a0 msgr2=0x7fa4c00febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:40:54.761 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.754+0000 7fa4c6a32700 1 -- 172.21.15.105:0/1331819003 shutdown_connections 2024-04-23T14:40:54.761 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.754+0000 7fa4c6a32700 1 -- 172.21.15.105:0/1331819003 wait complete. 2024-04-23T14:40:54.762 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.755+0000 7fa4c6a32700 1 Processor -- start 2024-04-23T14:40:54.762 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.755+0000 7fa4c6a32700 1 -- start start 2024-04-23T14:40:54.762 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.755+0000 7fa4c6a32700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4c0100e40 0x7fa4c0103780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:54.762 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.755+0000 7fa4c6a32700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4c0103e20 0x7fa4c0101dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:54.762 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.755+0000 7fa4c6a32700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4c01047c0 0x7fa4c0102310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:54.762 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.755+0000 7fa4c6a32700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c010e850 con 0x7fa4c0103e20 2024-04-23T14:40:54.762 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.755+0000 7fa4c6a32700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c010e6d0 con 0x7fa4c0100e40 2024-04-23T14:40:54.762 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.755+0000 7fa4c6a32700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c010e9d0 con 0x7fa4c01047c0 2024-04-23T14:40:54.762 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.755+0000 7fa4c4fcf700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4c01047c0 0x7fa4c0102310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:40:54.762 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.755+0000 7fa4c4fcf700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4c01047c0 0x7fa4c0102310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:50856/0 (socket says 172.21.15.105:50856) 2024-04-23T14:40:54.762 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.755+0000 7fa4c4fcf700 1 -- 172.21.15.105:0/716510056 learned_addr learned my addr 172.21.15.105:0/716510056 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-23T14:40:54.762 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.755+0000 7fa4bffff700 1 --2- 172.21.15.105:0/716510056 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4c0100e40 0x7fa4c0103780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:40:54.762 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.756+0000 7fa4c4fcf700 1 -- 172.21.15.105:0/716510056 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4c0100e40 msgr2=0x7fa4c0103780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:54.763 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.756+0000 7fa4c4fcf700 1 --2- 172.21.15.105:0/716510056 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4c0100e40 0x7fa4c0103780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:54.763 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.756+0000 7fa4c4fcf700 1 -- 172.21.15.105:0/716510056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4c0103e20 msgr2=0x7fa4c0101dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:54.763 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.756+0000 7fa4c4fcf700 1 --2- 172.21.15.105:0/716510056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4c0103e20 0x7fa4c0101dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:54.763 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.756+0000 7fa4c4fcf700 1 -- 172.21.15.105:0/716510056 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4c0102bc0 con 0x7fa4c01047c0 2024-04-23T14:40:54.763 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.756+0000 7fa4bffff700 1 --2- 172.21.15.105:0/716510056 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4c0100e40 0x7fa4c0103780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:40:54.763 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.756+0000 7fa4c4fcf700 1 --2- 172.21.15.105:0/716510056 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4c01047c0 0x7fa4c0102310 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8007f00 tx=0x7fa4b800bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:40:54.763 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.756+0000 7fa4bd7fa700 1 -- 172.21.15.105:0/716510056 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b8021030 con 0x7fa4c01047c0 2024-04-23T14:40:54.763 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.756+0000 7fa4c6a32700 1 -- 172.21.15.105:0/716510056 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4c01a2ed0 con 0x7fa4c01047c0 2024-04-23T14:40:54.763 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.756+0000 7fa4bd7fa700 1 -- 172.21.15.105:0/716510056 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4b8007bb0 con 0x7fa4c01047c0 2024-04-23T14:40:54.763 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.756+0000 7fa4bd7fa700 1 -- 172.21.15.105:0/716510056 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b801d470 con 0x7fa4c01047c0 2024-04-23T14:40:54.763 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.756+0000 7fa4c6a32700 1 -- 172.21.15.105:0/716510056 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4c01a3350 con 0x7fa4c01047c0 2024-04-23T14:40:54.763 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.757+0000 7fa4c6a32700 1 -- 172.21.15.105:0/716510056 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4c00697e0 con 0x7fa4c01047c0 2024-04-23T14:40:54.764 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.758+0000 7fa4bd7fa700 1 -- 172.21.15.105:0/716510056 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa4b8024070 con 0x7fa4c01047c0 2024-04-23T14:40:54.764 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.759+0000 7fa4bd7fa700 1 --2- 172.21.15.105:0/716510056 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fa4ac0a3c40 0x7fa4ac0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:54.764 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.759+0000 7fa4bd7fa700 1 -- 172.21.15.105:0/716510056 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa4b800f020 con 0x7fa4c01047c0 2024-04-23T14:40:54.764 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.759+0000 7fa4bffff700 1 --2- 172.21.15.105:0/716510056 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fa4ac0a3c40 0x7fa4ac0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:40:54.764 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.760+0000 7fa4bffff700 1 --2- 172.21.15.105:0/716510056 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fa4ac0a3c40 0x7fa4ac0a6100 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa4c006b060 tx=0x7fa4a8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:40:54.765 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.762+0000 7fa4bd7fa700 1 -- 172.21.15.105:0/716510056 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa4b809e050 con 0x7fa4c01047c0 2024-04-23T14:40:54.971 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.970+0000 7fa4c6a32700 1 -- 172.21.15.105:0/716510056 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi105=c;smithi105=f;smithi105=i;smithi105=l", "target": ["mon-mgr", ""]}) -- 0x7fa4c00631f0 con 0x7fa4ac0a3c40 2024-04-23T14:40:54.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.987+0000 7fa4bd7fa700 1 -- 172.21.15.105:0/716510056 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fa4c00631f0 con 0x7fa4ac0a3c40 2024-04-23T14:40:54.988 INFO:teuthology.orchestra.run.smithi105.stdout:Scheduled mds.all update... 2024-04-23T14:40:54.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.989+0000 7fa4c6a32700 1 -- 172.21.15.105:0/716510056 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fa4ac0a3c40 msgr2=0x7fa4ac0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:54.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.989+0000 7fa4c6a32700 1 --2- 172.21.15.105:0/716510056 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fa4ac0a3c40 0x7fa4ac0a6100 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa4c006b060 tx=0x7fa4a8008040 comp rx=0 tx=0).stop 2024-04-23T14:40:54.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.989+0000 7fa4c6a32700 1 -- 172.21.15.105:0/716510056 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4c01047c0 msgr2=0x7fa4c0102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:54.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.989+0000 7fa4c6a32700 1 --2- 172.21.15.105:0/716510056 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4c01047c0 0x7fa4c0102310 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8007f00 tx=0x7fa4b800bf10 comp rx=0 tx=0).stop 2024-04-23T14:40:54.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.989+0000 7fa4c6a32700 1 -- 172.21.15.105:0/716510056 shutdown_connections 2024-04-23T14:40:54.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.989+0000 7fa4c6a32700 1 --2- 172.21.15.105:0/716510056 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fa4ac0a3c40 0x7fa4ac0a6100 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:54.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.989+0000 7fa4c6a32700 1 --2- 172.21.15.105:0/716510056 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4c01047c0 0x7fa4c0102310 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:54.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.989+0000 7fa4c6a32700 1 --2- 172.21.15.105:0/716510056 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4c0100e40 0x7fa4c0103780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:54.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.989+0000 7fa4c6a32700 1 --2- 172.21.15.105:0/716510056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4c0103e20 0x7fa4c0101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:54.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.989+0000 7fa4c6a32700 1 -- 172.21.15.105:0/716510056 >> 172.21.15.105:0/716510056 conn(0x7fa4c00fc7a0 msgr2=0x7fa4c0107050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:40:54.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.989+0000 7fa4c6a32700 1 -- 172.21.15.105:0/716510056 shutdown_connections 2024-04-23T14:40:54.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-23T14:40:54.989+0000 7fa4c6a32700 1 -- 172.21.15.105:0/716510056 wait complete. 2024-04-23T14:40:55.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:55 smithi105 ceph-mon[32555]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:40:55.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:55 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:55.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:55 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:40:55.588 DEBUG:teuthology.orchestra.run.smithi028:mds.a> sudo journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.a.service 2024-04-23T14:40:55.595 DEBUG:teuthology.orchestra.run.smithi028:mds.d> sudo journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.d.service 2024-04-23T14:40:55.642 DEBUG:teuthology.orchestra.run.smithi028:mds.g> sudo journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.g.service 2024-04-23T14:40:55.687 DEBUG:teuthology.orchestra.run.smithi028:mds.j> sudo journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.j.service 2024-04-23T14:40:55.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:55 smithi028 ceph-mon[27810]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:40:55.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:55 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:55.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:55 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:40:55.708 DEBUG:teuthology.orchestra.run.smithi038:mds.b> sudo journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.b.service 2024-04-23T14:40:55.711 DEBUG:teuthology.orchestra.run.smithi038:mds.e> sudo journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.e.service 2024-04-23T14:40:55.731 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:55 smithi038 ceph-mon[32348]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:40:55.731 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:55 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:55.731 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:55 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:40:55.732 DEBUG:teuthology.orchestra.run.smithi038:mds.h> sudo journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.h.service 2024-04-23T14:40:55.775 DEBUG:teuthology.orchestra.run.smithi038:mds.k> sudo journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.k.service 2024-04-23T14:40:55.818 DEBUG:teuthology.orchestra.run.smithi105:mds.c> sudo journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.c.service 2024-04-23T14:40:55.820 DEBUG:teuthology.orchestra.run.smithi105:mds.f> sudo journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.f.service 2024-04-23T14:40:55.863 DEBUG:teuthology.orchestra.run.smithi105:mds.i> sudo journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.i.service 2024-04-23T14:40:55.906 DEBUG:teuthology.orchestra.run.smithi105:mds.l> sudo journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.l.service 2024-04-23T14:40:55.949 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-23T14:40:55.950 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:40:56.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.163+0000 7f2c73400640 1 -- 172.21.15.28:0/3905103914 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2c6c105be0 msgr2=0x7f2c6c106060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:56.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.163+0000 7f2c73400640 1 --2- 172.21.15.28:0/3905103914 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2c6c105be0 0x7f2c6c106060 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2c64009ca0 tx=0x7f2c6402a8c0 comp rx=0 tx=0).stop 2024-04-23T14:40:56.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.164+0000 7f2c73400640 1 -- 172.21.15.28:0/3905103914 shutdown_connections 2024-04-23T14:40:56.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.164+0000 7f2c73400640 1 --2- 172.21.15.28:0/3905103914 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2c6c105be0 0x7f2c6c106060 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:56.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.164+0000 7f2c73400640 1 --2- 172.21.15.28:0/3905103914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c6c103230 0x7f2c6c1056a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:56.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.164+0000 7f2c73400640 1 --2- 172.21.15.28:0/3905103914 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2c6c1008c0 0x7f2c6c102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:56.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.164+0000 7f2c73400640 1 -- 172.21.15.28:0/3905103914 >> 172.21.15.28:0/3905103914 conn(0x7f2c6c0fc4e0 msgr2=0x7f2c6c0fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:40:56.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.164+0000 7f2c73400640 1 -- 172.21.15.28:0/3905103914 shutdown_connections 2024-04-23T14:40:56.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.164+0000 7f2c73400640 1 -- 172.21.15.28:0/3905103914 wait complete. 2024-04-23T14:40:56.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.165+0000 7f2c73400640 1 Processor -- start 2024-04-23T14:40:56.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.165+0000 7f2c73400640 1 -- start start 2024-04-23T14:40:56.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.165+0000 7f2c73400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2c6c1008c0 0x7f2c6c19e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:56.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.165+0000 7f2c73400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2c6c103230 0x7f2c6c19ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:56.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.165+0000 7f2c73400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c6c105be0 0x7f2c6c1a5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:56.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.165+0000 7f2c73400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6c114300 con 0x7f2c6c105be0 2024-04-23T14:40:56.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.165+0000 7f2c73400640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6c114180 con 0x7f2c6c103230 2024-04-23T14:40:56.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.165+0000 7f2c73400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6c114480 con 0x7f2c6c1008c0 2024-04-23T14:40:56.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.166+0000 7f2c71400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c6c105be0 0x7f2c6c1a5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:40:56.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.166+0000 7f2c70a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2c6c1008c0 0x7f2c6c19e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:40:56.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.166+0000 7f2c70a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2c6c1008c0 0x7f2c6c19e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:41144/0 (socket says 172.21.15.28:41144) 2024-04-23T14:40:56.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.166+0000 7f2c70a00640 1 -- 172.21.15.28:0/1451747067 learned_addr learned my addr 172.21.15.28:0/1451747067 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:40:56.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.166+0000 7f2c71400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c6c105be0 0x7f2c6c1a5ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43136/0 (socket says 172.21.15.28:43136) 2024-04-23T14:40:56.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.166+0000 7f2c70a00640 1 -- 172.21.15.28:0/1451747067 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2c6c103230 msgr2=0x7f2c6c19ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:56.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.166+0000 7f2c6be00640 1 --2- 172.21.15.28:0/1451747067 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2c6c103230 0x7f2c6c19ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:40:56.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.166+0000 7f2c70a00640 1 --2- 172.21.15.28:0/1451747067 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2c6c103230 0x7f2c6c19ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:56.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.166+0000 7f2c70a00640 1 -- 172.21.15.28:0/1451747067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c6c105be0 msgr2=0x7f2c6c1a5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:56.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.166+0000 7f2c70a00640 1 --2- 172.21.15.28:0/1451747067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c6c105be0 0x7f2c6c1a5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:56.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.166+0000 7f2c70a00640 1 -- 172.21.15.28:0/1451747067 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c6c1a6510 con 0x7f2c6c1008c0 2024-04-23T14:40:56.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.166+0000 7f2c71400640 1 --2- 172.21.15.28:0/1451747067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c6c105be0 0x7f2c6c1a5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:40:56.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.166+0000 7f2c6be00640 1 --2- 172.21.15.28:0/1451747067 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2c6c103230 0x7f2c6c19ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:40:56.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.166+0000 7f2c70a00640 1 --2- 172.21.15.28:0/1451747067 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2c6c1008c0 0x7f2c6c19e8f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2c6000e9f0 tx=0x7f2c6000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:40:56.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.167+0000 7f2c69600640 1 -- 172.21.15.28:0/1451747067 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c6000ccc0 con 0x7f2c6c1008c0 2024-04-23T14:40:56.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.167+0000 7f2c69600640 1 -- 172.21.15.28:0/1451747067 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c6000ce60 con 0x7f2c6c1008c0 2024-04-23T14:40:56.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.167+0000 7f2c69600640 1 -- 172.21.15.28:0/1451747067 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c60010690 con 0x7f2c6c1008c0 2024-04-23T14:40:56.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.167+0000 7f2c73400640 1 -- 172.21.15.28:0/1451747067 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c6c1a6800 con 0x7f2c6c1008c0 2024-04-23T14:40:56.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.167+0000 7f2c73400640 1 -- 172.21.15.28:0/1451747067 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c6c1a6d00 con 0x7f2c6c1008c0 2024-04-23T14:40:56.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.169+0000 7f2c73400640 1 -- 172.21.15.28:0/1451747067 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c6c0fdfd0 con 0x7f2c6c1008c0 2024-04-23T14:40:56.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.170+0000 7f2c69600640 1 -- 172.21.15.28:0/1451747067 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2c60010830 con 0x7f2c6c1008c0 2024-04-23T14:40:56.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.171+0000 7f2c69600640 1 --2- 172.21.15.28:0/1451747067 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2c540a3bb0 0x7f2c540a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:56.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.171+0000 7f2c69600640 1 -- 172.21.15.28:0/1451747067 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f2c600d0500 con 0x7f2c6c1008c0 2024-04-23T14:40:56.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.172+0000 7f2c6be00640 1 --2- 172.21.15.28:0/1451747067 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2c540a3bb0 0x7f2c540a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:40:56.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.173+0000 7f2c6be00640 1 --2- 172.21.15.28:0/1451747067 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2c540a3bb0 0x7f2c540a6070 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c19fe00 tx=0x7f2c58005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:40:56.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.175+0000 7f2c69600640 1 -- 172.21.15.28:0/1451747067 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2c6009d2e0 con 0x7f2c6c1008c0 2024-04-23T14:40:56.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.403+0000 7f2c73400640 1 -- 172.21.15.28:0/1451747067 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2c6c062c00 con 0x7f2c6c1008c0 2024-04-23T14:40:56.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.405+0000 7f2c69600640 1 -- 172.21.15.28:0/1451747067 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18407 (secure 0 0 0) 0x7f2c6009cbe0 con 0x7f2c6c1008c0 2024-04-23T14:40:56.407 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:40:56.407 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":81,"fsid":"1ee6e996-017e-11ef-bc93-c7b262605968","created":"2024-04-23T14:32:34.581082+0000","modified":"2024-04-23T14:40:44.389733+0000","last_up_change":"2024-04-23T14:40:43.375846+0000","last_in_change":"2024-04-23T14:40:23.576939+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:36:27.698112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e0b427bc-be65-4543-99cc-2f53ad98ddad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6803","nonce":2993445336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6805","nonce":2993445336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6809","nonce":2993445336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6807","nonce":2993445336}]},"public_addr":"172.21.15.28:6803/2993445336","cluster_addr":"172.21.15.28:6805/2993445336","heartbeat_back_addr":"172.21.15.28:6809/2993445336","heartbeat_front_addr":"172.21.15.28:6807/2993445336","state":["exists","up"]},{"osd":1,"uuid":"ef527ec9-1bf3-4bb0-8df3-f5a015af557c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6801","nonce":4024093383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6803","nonce":4024093383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6807","nonce":4024093383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6805","nonce":4024093383}]},"public_addr":"172.21.15.38:6801/4024093383","cluster_addr":"172.21.15.38:6803/4024093383","heartbeat_back_addr":"172.21.15.38:6807/4024093383","heartbeat_front_addr":"172.21.15.38:6805/4024093383","state":["exists","up"]},{"osd":2,"uuid":"b33b23d9-adce-4c11-b304-2e117bdf6bfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6801","nonce":570739029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6803","nonce":570739029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6807","nonce":570739029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6805","nonce":570739029}]},"public_addr":"172.21.15.105:6801/570739029","cluster_addr":"172.21.15.105:6803/570739029","heartbeat_back_addr":"172.21.15.105:6807/570739029","heartbeat_front_addr":"172.21.15.105:6805/570739029","state":["exists","up"]},{"osd":3,"uuid":"28bd5608-b426-434e-9db1-bdaa9828bdb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6811","nonce":2480400247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6813","nonce":2480400247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6817","nonce":2480400247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6815","nonce":2480400247}]},"public_addr":"172.21.15.28:6811/2480400247","cluster_addr":"172.21.15.28:6813/2480400247","heartbeat_back_addr":"172.21.15.28:6817/2480400247","heartbeat_front_addr":"172.21.15.28:6815/2480400247","state":["exists","up"]},{"osd":4,"uuid":"e5d1c89b-025f-4d57-8807-59d75c787b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6809","nonce":3772189246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6811","nonce":3772189246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6815","nonce":3772189246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6813","nonce":3772189246}]},"public_addr":"172.21.15.38:6809/3772189246","cluster_addr":"172.21.15.38:6811/3772189246","heartbeat_back_addr":"172.21.15.38:6815/3772189246","heartbeat_front_addr":"172.21.15.38:6813/3772189246","state":["exists","up"]},{"osd":5,"uuid":"9cf9cf4c-aafd-4ed3-bfb1-0527710a0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6809","nonce":928755894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6811","nonce":928755894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6815","nonce":928755894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6813","nonce":928755894}]},"public_addr":"172.21.15.105:6809/928755894","cluster_addr":"172.21.15.105:6811/928755894","heartbeat_back_addr":"172.21.15.105:6815/928755894","heartbeat_front_addr":"172.21.15.105:6813/928755894","state":["exists","up"]},{"osd":6,"uuid":"304db525-d24d-4687-bfa0-f92bdf66d1e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6819","nonce":897979441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6821","nonce":897979441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6825","nonce":897979441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6823","nonce":897979441}]},"public_addr":"172.21.15.28:6819/897979441","cluster_addr":"172.21.15.28:6821/897979441","heartbeat_back_addr":"172.21.15.28:6825/897979441","heartbeat_front_addr":"172.21.15.28:6823/897979441","state":["exists","up"]},{"osd":7,"uuid":"08e01d59-03f3-47b1-b45f-35ec5a50992c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6817","nonce":2400251786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6819","nonce":2400251786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6823","nonce":2400251786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6821","nonce":2400251786}]},"public_addr":"172.21.15.38:6817/2400251786","cluster_addr":"172.21.15.38:6819/2400251786","heartbeat_back_addr":"172.21.15.38:6823/2400251786","heartbeat_front_addr":"172.21.15.38:6821/2400251786","state":["exists","up"]},{"osd":8,"uuid":"64ee1e95-0756-48b1-9c00-472d0c3cbd29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6817","nonce":1682312269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6819","nonce":1682312269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6823","nonce":1682312269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6821","nonce":1682312269}]},"public_addr":"172.21.15.105:6817/1682312269","cluster_addr":"172.21.15.105:6819/1682312269","heartbeat_back_addr":"172.21.15.105:6823/1682312269","heartbeat_front_addr":"172.21.15.105:6821/1682312269","state":["exists","up"]},{"osd":9,"uuid":"18f60893-3ffd-48c2-9db5-a67d35611d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6827","nonce":1292942516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6829","nonce":1292942516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6833","nonce":1292942516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6831","nonce":1292942516}]},"public_addr":"172.21.15.28:6827/1292942516","cluster_addr":"172.21.15.28:6829/1292942516","heartbeat_back_addr":"172.21.15.28:6833/1292942516","heartbeat_front_addr":"172.21.15.28:6831/1292942516","state":["exists","up"]},{"osd":10,"uuid":"76c06725-7040-4118-8894-fe808a87f999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6825","nonce":2444616468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6827","nonce":2444616468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6831","nonce":2444616468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6829","nonce":2444616468}]},"public_addr":"172.21.15.38:6825/2444616468","cluster_addr":"172.21.15.38:6827/2444616468","heartbeat_back_addr":"172.21.15.38:6831/2444616468","heartbeat_front_addr":"172.21.15.38:6829/2444616468","state":["exists","up"]},{"osd":11,"uuid":"e8fc9454-5540-4e2e-91bf-3ba251dbd1a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6825","nonce":3400299562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6827","nonce":3400299562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6831","nonce":3400299562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6829","nonce":3400299562}]},"public_addr":"172.21.15.105:6825/3400299562","cluster_addr":"172.21.15.105:6827/3400299562","heartbeat_back_addr":"172.21.15.105:6831/3400299562","heartbeat_front_addr":"172.21.15.105:6829/3400299562","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:35:33.615395+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:00.545293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:24.940211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:57.684887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:24.519061+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:49.010804+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:22.638478+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:49.559104+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:14.011771+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:47.859950+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:16.298445+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:41.619069+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/590495740":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/1163172512":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/4290328460":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/4145020172":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/720626958":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/948450262":"2024-04-24T14:33:07.576363+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:40:56.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.408+0000 7f2c73400640 1 -- 172.21.15.28:0/1451747067 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2c540a3bb0 msgr2=0x7f2c540a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:56.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.408+0000 7f2c73400640 1 --2- 172.21.15.28:0/1451747067 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2c540a3bb0 0x7f2c540a6070 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c19fe00 tx=0x7f2c58005e50 comp rx=0 tx=0).stop 2024-04-23T14:40:56.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.408+0000 7f2c73400640 1 -- 172.21.15.28:0/1451747067 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2c6c1008c0 msgr2=0x7f2c6c19e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:56.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.408+0000 7f2c73400640 1 --2- 172.21.15.28:0/1451747067 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2c6c1008c0 0x7f2c6c19e8f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2c6000e9f0 tx=0x7f2c6000eeb0 comp rx=0 tx=0).stop 2024-04-23T14:40:56.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.408+0000 7f2c73400640 1 -- 172.21.15.28:0/1451747067 shutdown_connections 2024-04-23T14:40:56.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.408+0000 7f2c73400640 1 --2- 172.21.15.28:0/1451747067 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2c540a3bb0 0x7f2c540a6070 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:56.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.408+0000 7f2c73400640 1 --2- 172.21.15.28:0/1451747067 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2c6c103230 0x7f2c6c19ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:56.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.408+0000 7f2c73400640 1 --2- 172.21.15.28:0/1451747067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c6c105be0 0x7f2c6c1a5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:56.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.408+0000 7f2c73400640 1 --2- 172.21.15.28:0/1451747067 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2c6c1008c0 0x7f2c6c19e8f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:56.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.408+0000 7f2c73400640 1 -- 172.21.15.28:0/1451747067 >> 172.21.15.28:0/1451747067 conn(0x7f2c6c0fc4e0 msgr2=0x7f2c6c101240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:40:56.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.408+0000 7f2c73400640 1 -- 172.21.15.28:0/1451747067 shutdown_connections 2024-04-23T14:40:56.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.408+0000 7f2c73400640 1 -- 172.21.15.28:0/1451747067 wait complete. 2024-04-23T14:40:56.430 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T14:36:27.698112+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T14:40:56.431 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T14:40:56.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:56 smithi105 ceph-mon[32555]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi105=c;smithi105=f;smithi105=i;smithi105=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:40:56.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:56 smithi105 ceph-mon[32555]: Saving service mds.all spec with placement smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi105=c;smithi105=f;smithi105=i;smithi105=l;count:12 2024-04-23T14:40:56.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:56 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:40:56.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:56 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:40:56.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:56 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:56.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:56 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:40:56.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:56 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:40:56.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:56 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:40:56.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:56 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1451747067' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:40:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.646+0000 7faffb200640 1 -- 172.21.15.28:0/3882574964 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faffc1052c0 msgr2=0x7faffc1076b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.646+0000 7faffb200640 1 --2- 172.21.15.28:0/3882574964 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faffc1052c0 0x7faffc1076b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fafec009ca0 tx=0x7fafec02a8c0 comp rx=0 tx=0).stop 2024-04-23T14:40:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.646+0000 7faffb200640 1 -- 172.21.15.28:0/3882574964 shutdown_connections 2024-04-23T14:40:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.646+0000 7faffb200640 1 --2- 172.21.15.28:0/3882574964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faffc107bf0 0x7faffc10a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.646+0000 7faffb200640 1 --2- 172.21.15.28:0/3882574964 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faffc1052c0 0x7faffc1076b0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.646+0000 7faffb200640 1 --2- 172.21.15.28:0/3882574964 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faffc102990 0x7faffc104d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.646+0000 7faffb200640 1 -- 172.21.15.28:0/3882574964 >> 172.21.15.28:0/3882574964 conn(0x7faffc0fc3a0 msgr2=0x7faffc0fe7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:40:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.646+0000 7faffb200640 1 -- 172.21.15.28:0/3882574964 shutdown_connections 2024-04-23T14:40:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.646+0000 7faffb200640 1 -- 172.21.15.28:0/3882574964 wait complete. 2024-04-23T14:40:56.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.648+0000 7faffb200640 1 Processor -- start 2024-04-23T14:40:56.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.648+0000 7faffb200640 1 -- start start 2024-04-23T14:40:56.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.648+0000 7faffb200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faffc102990 0x7faffc102300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:56.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.648+0000 7faffb200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faffc1052c0 0x7faffc100940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:56.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.648+0000 7faffb200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faffc107bf0 0x7faffc100e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:56.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.648+0000 7faffb200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faffc10dbc0 con 0x7faffc107bf0 2024-04-23T14:40:56.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.648+0000 7faffb200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7faffc10da40 con 0x7faffc1052c0 2024-04-23T14:40:56.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.648+0000 7faffb200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7faffc10dd40 con 0x7faffc102990 2024-04-23T14:40:56.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.648+0000 7faff9e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faffc102990 0x7faffc102300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:40:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.649+0000 7faff9e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faffc102990 0x7faffc102300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:41166/0 (socket says 172.21.15.28:41166) 2024-04-23T14:40:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.649+0000 7faff9e00640 1 -- 172.21.15.28:0/661276768 learned_addr learned my addr 172.21.15.28:0/661276768 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:40:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.649+0000 7faffa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faffc107bf0 0x7faffc100e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:40:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.649+0000 7faff9400640 1 --2- 172.21.15.28:0/661276768 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faffc1052c0 0x7faffc100940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:40:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.649+0000 7faff9e00640 1 -- 172.21.15.28:0/661276768 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faffc1052c0 msgr2=0x7faffc100940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.649+0000 7faff9e00640 1 --2- 172.21.15.28:0/661276768 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faffc1052c0 0x7faffc100940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.649+0000 7faff9e00640 1 -- 172.21.15.28:0/661276768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faffc107bf0 msgr2=0x7faffc100e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.649+0000 7faff9e00640 1 --2- 172.21.15.28:0/661276768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faffc107bf0 0x7faffc100e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.649+0000 7faff9e00640 1 -- 172.21.15.28:0/661276768 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faffc101710 con 0x7faffc102990 2024-04-23T14:40:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.649+0000 7faff9400640 1 --2- 172.21.15.28:0/661276768 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faffc1052c0 0x7faffc100940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:40:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.649+0000 7faff9e00640 1 --2- 172.21.15.28:0/661276768 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faffc102990 0x7faffc102300 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fafe400e9f0 tx=0x7fafe400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:40:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.649+0000 7faff2a00640 1 -- 172.21.15.28:0/661276768 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe400cd80 con 0x7faffc102990 2024-04-23T14:40:56.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.650+0000 7faff2a00640 1 -- 172.21.15.28:0/661276768 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafe4010910 con 0x7faffc102990 2024-04-23T14:40:56.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.650+0000 7faff2a00640 1 -- 172.21.15.28:0/661276768 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe4018990 con 0x7faffc102990 2024-04-23T14:40:56.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.650+0000 7faffb200640 1 -- 172.21.15.28:0/661276768 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faffc1a2350 con 0x7faffc102990 2024-04-23T14:40:56.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.650+0000 7faffb200640 1 -- 172.21.15.28:0/661276768 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faffc1a28c0 con 0x7faffc102990 2024-04-23T14:40:56.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.651+0000 7faffb200640 1 -- 172.21.15.28:0/661276768 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faffc0fde90 con 0x7faffc102990 2024-04-23T14:40:56.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.653+0000 7faff2a00640 1 -- 172.21.15.28:0/661276768 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fafe4010ac0 con 0x7faffc102990 2024-04-23T14:40:56.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.654+0000 7faff2a00640 1 --2- 172.21.15.28:0/661276768 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fafe00a3c30 0x7fafe00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:56.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.654+0000 7faff2a00640 1 -- 172.21.15.28:0/661276768 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fafe40d0360 con 0x7faffc102990 2024-04-23T14:40:56.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.654+0000 7faff9400640 1 --2- 172.21.15.28:0/661276768 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fafe00a3c30 0x7fafe00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:40:56.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.655+0000 7faff9400640 1 --2- 172.21.15.28:0/661276768 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fafe00a3c30 0x7fafe00a60f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fafec006010 tx=0x7fafec009800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:40:56.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.658+0000 7faff2a00640 1 -- 172.21.15.28:0/661276768 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fafe409d140 con 0x7faffc102990 2024-04-23T14:40:56.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.867+0000 7faffb200640 1 -- 172.21.15.28:0/661276768 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7faffc062c00 con 0x7faffc102990 2024-04-23T14:40:56.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.868+0000 7faff2a00640 1 -- 172.21.15.28:0/661276768 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7fafe409ca40 con 0x7faffc102990 2024-04-23T14:40:56.869 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-23T14:40:56.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.871+0000 7faffb200640 1 -- 172.21.15.28:0/661276768 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fafe00a3c30 msgr2=0x7fafe00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:56.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.871+0000 7faffb200640 1 --2- 172.21.15.28:0/661276768 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fafe00a3c30 0x7fafe00a60f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fafec006010 tx=0x7fafec009800 comp rx=0 tx=0).stop 2024-04-23T14:40:56.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.871+0000 7faffb200640 1 -- 172.21.15.28:0/661276768 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faffc102990 msgr2=0x7faffc102300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:56.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.871+0000 7faffb200640 1 --2- 172.21.15.28:0/661276768 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faffc102990 0x7faffc102300 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fafe400e9f0 tx=0x7fafe400eeb0 comp rx=0 tx=0).stop 2024-04-23T14:40:56.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.871+0000 7faffb200640 1 -- 172.21.15.28:0/661276768 shutdown_connections 2024-04-23T14:40:56.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.871+0000 7faffb200640 1 --2- 172.21.15.28:0/661276768 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fafe00a3c30 0x7fafe00a60f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:56.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.871+0000 7faffb200640 1 --2- 172.21.15.28:0/661276768 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faffc1052c0 0x7faffc100940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:56.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.871+0000 7faffb200640 1 --2- 172.21.15.28:0/661276768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faffc107bf0 0x7faffc100e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:56.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.871+0000 7faffb200640 1 --2- 172.21.15.28:0/661276768 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faffc102990 0x7faffc102300 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:56.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.871+0000 7faffb200640 1 -- 172.21.15.28:0/661276768 >> 172.21.15.28:0/661276768 conn(0x7faffc0fc3a0 msgr2=0x7faffc1032d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:40:56.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.871+0000 7faffb200640 1 -- 172.21.15.28:0/661276768 shutdown_connections 2024-04-23T14:40:56.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:56.872+0000 7faffb200640 1 -- 172.21.15.28:0/661276768 wait complete. 2024-04-23T14:40:56.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:56 smithi038 ceph-mon[32348]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi105=c;smithi105=f;smithi105=i;smithi105=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:40:56.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:56 smithi038 ceph-mon[32348]: Saving service mds.all spec with placement smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi105=c;smithi105=f;smithi105=i;smithi105=l;count:12 2024-04-23T14:40:56.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:56 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:40:56.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:56 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:40:56.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:56 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:56.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:56 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:40:56.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:56 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:40:56.875 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:56 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:40:56.875 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:56 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1451747067' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:40:56.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:56 smithi028 ceph-mon[27810]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi105=c;smithi105=f;smithi105=i;smithi105=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:40:56.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:56 smithi028 ceph-mon[27810]: Saving service mds.all spec with placement smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi105=c;smithi105=f;smithi105=i;smithi105=l;count:12 2024-04-23T14:40:56.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:56 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:40:56.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:56 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:40:56.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:56 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:56.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:56 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:40:56.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:56 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:40:56.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:56 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:40:56.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:56 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1451747067' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:40:56.892 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:40:57.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.103+0000 7fd246e00640 1 -- 172.21.15.28:0/3586067847 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd240104080 msgr2=0x7fd2401044a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:57.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.103+0000 7fd246e00640 1 --2- 172.21.15.28:0/3586067847 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd240104080 0x7fd2401044a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd230009900 tx=0x7fd23002a410 comp rx=0 tx=0).stop 2024-04-23T14:40:57.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.103+0000 7fd246e00640 1 -- 172.21.15.28:0/3586067847 shutdown_connections 2024-04-23T14:40:57.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.103+0000 7fd246e00640 1 --2- 172.21.15.28:0/3586067847 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd240105c30 0x7fd24010c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:57.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.103+0000 7fd246e00640 1 --2- 172.21.15.28:0/3586067847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd240105270 0x7fd2401056f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:57.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.103+0000 7fd246e00640 1 --2- 172.21.15.28:0/3586067847 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd240104080 0x7fd2401044a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:57.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.103+0000 7fd246e00640 1 -- 172.21.15.28:0/3586067847 >> 172.21.15.28:0/3586067847 conn(0x7fd2400ff610 msgr2=0x7fd240101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:40:57.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.103+0000 7fd246e00640 1 -- 172.21.15.28:0/3586067847 shutdown_connections 2024-04-23T14:40:57.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.104+0000 7fd246e00640 1 -- 172.21.15.28:0/3586067847 wait complete. 2024-04-23T14:40:57.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.104+0000 7fd246e00640 1 Processor -- start 2024-04-23T14:40:57.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.105+0000 7fd246e00640 1 -- start start 2024-04-23T14:40:57.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.105+0000 7fd246e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd240104080 0x7fd24019a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:57.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.105+0000 7fd246e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd240105270 0x7fd24019a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:57.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.105+0000 7fd246e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd240105c30 0x7fd2401a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:57.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.105+0000 7fd246e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd24010fd60 con 0x7fd240104080 2024-04-23T14:40:57.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.105+0000 7fd246e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd24010fbe0 con 0x7fd240105270 2024-04-23T14:40:57.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.105+0000 7fd246e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd24010fee0 con 0x7fd240105c30 2024-04-23T14:40:57.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.105+0000 7fd245a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd240104080 0x7fd24019a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:40:57.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.105+0000 7fd245a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd240104080 0x7fd24019a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43178/0 (socket says 172.21.15.28:43178) 2024-04-23T14:40:57.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.105+0000 7fd245a00640 1 -- 172.21.15.28:0/1409708911 learned_addr learned my addr 172.21.15.28:0/1409708911 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:40:57.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.105+0000 7fd245000640 1 --2- 172.21.15.28:0/1409708911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd240105270 0x7fd24019a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:40:57.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.105+0000 7fd246400640 1 --2- 172.21.15.28:0/1409708911 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd240105c30 0x7fd2401a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:40:57.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.106+0000 7fd245a00640 1 -- 172.21.15.28:0/1409708911 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd240105c30 msgr2=0x7fd2401a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:57.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.106+0000 7fd245a00640 1 --2- 172.21.15.28:0/1409708911 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd240105c30 0x7fd2401a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:57.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.106+0000 7fd245a00640 1 -- 172.21.15.28:0/1409708911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd240105270 msgr2=0x7fd24019a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:57.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.106+0000 7fd245a00640 1 --2- 172.21.15.28:0/1409708911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd240105270 0x7fd24019a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:57.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.106+0000 7fd245a00640 1 -- 172.21.15.28:0/1409708911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2401a1fb0 con 0x7fd240104080 2024-04-23T14:40:57.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.106+0000 7fd245a00640 1 --2- 172.21.15.28:0/1409708911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd240104080 0x7fd24019a390 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd230031440 tx=0x7fd230032e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:40:57.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.106+0000 7fd23e000640 1 -- 172.21.15.28:0/1409708911 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd23003f030 con 0x7fd240104080 2024-04-23T14:40:57.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.107+0000 7fd246e00640 1 -- 172.21.15.28:0/1409708911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2401a2240 con 0x7fd240104080 2024-04-23T14:40:57.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.107+0000 7fd23e000640 1 -- 172.21.15.28:0/1409708911 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd230003ce0 con 0x7fd240104080 2024-04-23T14:40:57.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.107+0000 7fd23e000640 1 -- 172.21.15.28:0/1409708911 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd230035070 con 0x7fd240104080 2024-04-23T14:40:57.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.107+0000 7fd246e00640 1 -- 172.21.15.28:0/1409708911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2401a2730 con 0x7fd240104080 2024-04-23T14:40:57.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.108+0000 7fd246e00640 1 -- 172.21.15.28:0/1409708911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd24010b340 con 0x7fd240104080 2024-04-23T14:40:57.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.110+0000 7fd23e000640 1 -- 172.21.15.28:0/1409708911 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd230042030 con 0x7fd240104080 2024-04-23T14:40:57.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.111+0000 7fd23e000640 1 --2- 172.21.15.28:0/1409708911 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd2280a3bb0 0x7fd2280a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:57.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.111+0000 7fd23e000640 1 -- 172.21.15.28:0/1409708911 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd2300f0570 con 0x7fd240104080 2024-04-23T14:40:57.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.111+0000 7fd245000640 1 --2- 172.21.15.28:0/1409708911 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd2280a3bb0 0x7fd2280a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:40:57.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.112+0000 7fd245000640 1 --2- 172.21.15.28:0/1409708911 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd2280a3bb0 0x7fd2280a6070 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd24019b8a0 tx=0x7fd23400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:40:57.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.116+0000 7fd23e000640 1 -- 172.21.15.28:0/1409708911 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd2300bcd00 con 0x7fd240104080 2024-04-23T14:40:57.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.364+0000 7fd246e00640 1 -- 172.21.15.28:0/1409708911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd24002d620 con 0x7fd240104080 2024-04-23T14:40:57.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.365+0000 7fd23e000640 1 -- 172.21.15.28:0/1409708911 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7fd2300c1000 con 0x7fd240104080 2024-04-23T14:40:57.366 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:40:57.366 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-23T14:40:57.367 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 1 2024-04-23T14:40:57.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.368+0000 7fd246e00640 1 -- 172.21.15.28:0/1409708911 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd2280a3bb0 msgr2=0x7fd2280a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:57.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.368+0000 7fd246e00640 1 --2- 172.21.15.28:0/1409708911 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd2280a3bb0 0x7fd2280a6070 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd24019b8a0 tx=0x7fd23400b390 comp rx=0 tx=0).stop 2024-04-23T14:40:57.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.368+0000 7fd246e00640 1 -- 172.21.15.28:0/1409708911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd240104080 msgr2=0x7fd24019a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:57.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.368+0000 7fd246e00640 1 --2- 172.21.15.28:0/1409708911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd240104080 0x7fd24019a390 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd230031440 tx=0x7fd230032e20 comp rx=0 tx=0).stop 2024-04-23T14:40:57.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.368+0000 7fd246e00640 1 -- 172.21.15.28:0/1409708911 shutdown_connections 2024-04-23T14:40:57.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.368+0000 7fd246e00640 1 --2- 172.21.15.28:0/1409708911 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd2280a3bb0 0x7fd2280a6070 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:57.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.368+0000 7fd246e00640 1 --2- 172.21.15.28:0/1409708911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd240105270 0x7fd24019a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:57.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.368+0000 7fd246e00640 1 --2- 172.21.15.28:0/1409708911 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd240105c30 0x7fd2401a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:57.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.368+0000 7fd246e00640 1 --2- 172.21.15.28:0/1409708911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd240104080 0x7fd24019a390 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:57.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.368+0000 7fd246e00640 1 -- 172.21.15.28:0/1409708911 >> 172.21.15.28:0/1409708911 conn(0x7fd2400ff610 msgr2=0x7fd240108510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:40:57.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.368+0000 7fd246e00640 1 -- 172.21.15.28:0/1409708911 shutdown_connections 2024-04-23T14:40:57.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:57.368+0000 7fd246e00640 1 -- 172.21.15.28:0/1409708911 wait complete. 2024-04-23T14:40:57.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:57 smithi105 ceph-mon[32555]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:40:57.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:57 smithi105 ceph-mon[32555]: Deploying daemon mds.l on smithi105 2024-04-23T14:40:57.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:57 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/661276768' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:40:57.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:57 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1409708911' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:40:57.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:57 smithi038 ceph-mon[32348]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:40:57.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:57 smithi038 ceph-mon[32348]: Deploying daemon mds.l on smithi105 2024-04-23T14:40:57.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:57 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/661276768' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:40:57.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:57 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1409708911' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:40:57.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:57 smithi028 ceph-mon[27810]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:40:57.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:57 smithi028 ceph-mon[27810]: Deploying daemon mds.l on smithi105 2024-04-23T14:40:57.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:57 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/661276768' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:40:57.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:57 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1409708911' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:40:58.529 INFO:journalctl@ceph.mds.l.smithi105.stdout:Apr 23 14:40:58 smithi105 systemd[1]: Starting Ceph mds.l for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T14:40:58.783 INFO:journalctl@ceph.mds.l.smithi105.stdout:Apr 23 14:40:58 smithi105 podman[62727]: 2024-04-23 14:40:58.431547197 +0000 UTC m=+0.017670890 image pull 76e77f280c74312ca1add9b6019a4a921a516f91795d85bae00a0c8ca9b1c62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd 2024-04-23T14:40:58.783 INFO:journalctl@ceph.mds.l.smithi105.stdout:Apr 23 14:40:58 smithi105 podman[62727]: 2024-04-23 14:40:58.54699572 +0000 UTC m=+0.133119404 container create 9622fdb37276d39eec47032f74056b2dec4a0f554781c92a6b8a7678e2d66a82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-l, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-23T14:40:58.783 INFO:journalctl@ceph.mds.l.smithi105.stdout:Apr 23 14:40:58 smithi105 podman[62727]: 2024-04-23 14:40:58.782725597 +0000 UTC m=+0.368849281 container init 9622fdb37276d39eec47032f74056b2dec4a0f554781c92a6b8a7678e2d66a82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-l, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-23T14:40:59.036 INFO:journalctl@ceph.mds.l.smithi105.stdout:Apr 23 14:40:58 smithi105 podman[62727]: 2024-04-23 14:40:58.785317191 +0000 UTC m=+0.371440879 container start 9622fdb37276d39eec47032f74056b2dec4a0f554781c92a6b8a7678e2d66a82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-23T14:40:59.036 INFO:journalctl@ceph.mds.l.smithi105.stdout:Apr 23 14:40:58 smithi105 ceph-mds[62746]: -- 172.21.15.105:0/3633846610 <== mon.1 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a62eabfa40 con 0x55a62ea92c00 2024-04-23T14:40:59.036 INFO:journalctl@ceph.mds.l.smithi105.stdout:Apr 23 14:40:58 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-l[62742]: starting mds.l at 2024-04-23T14:40:59.037 INFO:journalctl@ceph.mds.l.smithi105.stdout:Apr 23 14:40:58 smithi105 bash[62727]: 9622fdb37276d39eec47032f74056b2dec4a0f554781c92a6b8a7678e2d66a82 2024-04-23T14:40:59.037 INFO:journalctl@ceph.mds.l.smithi105.stdout:Apr 23 14:40:58 smithi105 systemd[1]: Started Ceph mds.l for 1ee6e996-017e-11ef-bc93-c7b262605968. 2024-04-23T14:40:59.391 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:40:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.613+0000 7fcc1b200640 1 -- 172.21.15.28:0/501502690 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc1c105230 msgr2=0x7fcc1c107620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.613+0000 7fcc1b200640 1 --2- 172.21.15.28:0/501502690 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc1c105230 0x7fcc1c107620 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcc08009900 tx=0x7fcc0802a410 comp rx=0 tx=0).stop 2024-04-23T14:40:59.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.613+0000 7fcc1b200640 1 -- 172.21.15.28:0/501502690 shutdown_connections 2024-04-23T14:40:59.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.613+0000 7fcc1b200640 1 --2- 172.21.15.28:0/501502690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc1c107b60 0x7fcc1c109fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:59.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.613+0000 7fcc1b200640 1 --2- 172.21.15.28:0/501502690 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc1c105230 0x7fcc1c107620 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:59.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.613+0000 7fcc1b200640 1 --2- 172.21.15.28:0/501502690 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc1c102900 0x7fcc1c104cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:59.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.613+0000 7fcc1b200640 1 -- 172.21.15.28:0/501502690 >> 172.21.15.28:0/501502690 conn(0x7fcc1c0fc2d0 msgr2=0x7fcc1c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:40:59.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.613+0000 7fcc1b200640 1 -- 172.21.15.28:0/501502690 shutdown_connections 2024-04-23T14:40:59.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.613+0000 7fcc1b200640 1 -- 172.21.15.28:0/501502690 wait complete. 2024-04-23T14:40:59.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.614+0000 7fcc1b200640 1 Processor -- start 2024-04-23T14:40:59.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.614+0000 7fcc1b200640 1 -- start start 2024-04-23T14:40:59.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.614+0000 7fcc1b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc1c102900 0x7fcc1c19a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:59.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.614+0000 7fcc1b200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc1c105230 0x7fcc1c19a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:59.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.614+0000 7fcc1b200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc1c107b60 0x7fcc1c1a1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:59.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.614+0000 7fcc1b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcc1c10db60 con 0x7fcc1c102900 2024-04-23T14:40:59.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.614+0000 7fcc1b200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcc1c10d9e0 con 0x7fcc1c105230 2024-04-23T14:40:59.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.615+0000 7fcc1b200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcc1c10dce0 con 0x7fcc1c107b60 2024-04-23T14:40:59.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.615+0000 7fcc19e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc1c102900 0x7fcc1c19a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:40:59.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.615+0000 7fcc19e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc1c102900 0x7fcc1c19a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46356/0 (socket says 172.21.15.28:46356) 2024-04-23T14:40:59.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.615+0000 7fcc19e00640 1 -- 172.21.15.28:0/2845514698 learned_addr learned my addr 172.21.15.28:0/2845514698 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:40:59.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.615+0000 7fcc1a800640 1 --2- 172.21.15.28:0/2845514698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc1c107b60 0x7fcc1c1a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:40:59.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.615+0000 7fcc19400640 1 --2- 172.21.15.28:0/2845514698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc1c105230 0x7fcc1c19a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:40:59.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.615+0000 7fcc1a800640 1 -- 172.21.15.28:0/2845514698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc1c105230 msgr2=0x7fcc1c19a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:59.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.615+0000 7fcc1a800640 1 --2- 172.21.15.28:0/2845514698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc1c105230 0x7fcc1c19a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:59.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.615+0000 7fcc1a800640 1 -- 172.21.15.28:0/2845514698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc1c102900 msgr2=0x7fcc1c19a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:59.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.615+0000 7fcc1a800640 1 --2- 172.21.15.28:0/2845514698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc1c102900 0x7fcc1c19a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:59.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.615+0000 7fcc1a800640 1 -- 172.21.15.28:0/2845514698 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc1c1a1fe0 con 0x7fcc1c107b60 2024-04-23T14:40:59.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.615+0000 7fcc19e00640 1 --2- 172.21.15.28:0/2845514698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc1c102900 0x7fcc1c19a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:40:59.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.616+0000 7fcc1a800640 1 --2- 172.21.15.28:0/2845514698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc1c107b60 0x7fcc1c1a1970 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcc1400da10 tx=0x7fcc1400ded0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:40:59.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.616+0000 7fcc12a00640 1 -- 172.21.15.28:0/2845514698 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1400bd70 con 0x7fcc1c107b60 2024-04-23T14:40:59.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.616+0000 7fcc12a00640 1 -- 172.21.15.28:0/2845514698 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc14003e50 con 0x7fcc1c107b60 2024-04-23T14:40:59.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.616+0000 7fcc1b200640 1 -- 172.21.15.28:0/2845514698 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc1c1a22d0 con 0x7fcc1c107b60 2024-04-23T14:40:59.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.616+0000 7fcc12a00640 1 -- 172.21.15.28:0/2845514698 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc140106a0 con 0x7fcc1c107b60 2024-04-23T14:40:59.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.616+0000 7fcc1b200640 1 -- 172.21.15.28:0/2845514698 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc1c1a27a0 con 0x7fcc1c107b60 2024-04-23T14:40:59.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.617+0000 7fcc1b200640 1 -- 172.21.15.28:0/2845514698 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc1c1096b0 con 0x7fcc1c107b60 2024-04-23T14:40:59.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.619+0000 7fcc12a00640 1 -- 172.21.15.28:0/2845514698 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fcc14003970 con 0x7fcc1c107b60 2024-04-23T14:40:59.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.620+0000 7fcc12a00640 1 --2- 172.21.15.28:0/2845514698 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fcc000a3bb0 0x7fcc000a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:40:59.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.620+0000 7fcc12a00640 1 -- 172.21.15.28:0/2845514698 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fcc140d03c0 con 0x7fcc1c107b60 2024-04-23T14:40:59.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.621+0000 7fcc19e00640 1 --2- 172.21.15.28:0/2845514698 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fcc000a3bb0 0x7fcc000a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:40:59.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.621+0000 7fcc19e00640 1 --2- 172.21.15.28:0/2845514698 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fcc000a3bb0 0x7fcc000a6070 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c011500 tx=0x7fcc0c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:40:59.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.625+0000 7fcc12a00640 1 -- 172.21.15.28:0/2845514698 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcc1409d1a0 con 0x7fcc1c107b60 2024-04-23T14:40:59.723 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:59 smithi038 ceph-mon[32348]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:40:59.724 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:59 smithi038 ceph-mon[32348]: mds.? [v2:172.21.15.105:6832/573683071,v1:172.21.15.105:6833/573683071] up:boot 2024-04-23T14:40:59.724 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:59 smithi038 ceph-mon[32348]: fsmap 1 up:standby 2024-04-23T14:40:59.724 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:59 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-23T14:40:59.724 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:59 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:59.724 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:59 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:59.724 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:59 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:59.724 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:59 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:40:59.724 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:59 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:40:59.724 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:40:59 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:40:59.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:59 smithi105 ceph-mon[32555]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:40:59.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:59 smithi105 ceph-mon[32555]: mds.? [v2:172.21.15.105:6832/573683071,v1:172.21.15.105:6833/573683071] up:boot 2024-04-23T14:40:59.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:59 smithi105 ceph-mon[32555]: fsmap 1 up:standby 2024-04-23T14:40:59.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:59 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-23T14:40:59.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:59 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:59.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:59 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:59.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:59 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:59.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:59 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:40:59.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:59 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:40:59.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:40:59 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:40:59.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.872+0000 7fcc1b200640 1 -- 172.21.15.28:0/2845514698 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc1c02d620 con 0x7fcc1c107b60 2024-04-23T14:40:59.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.873+0000 7fcc12a00640 1 -- 172.21.15.28:0/2845514698 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1393 (secure 0 0 0) 0x7fcc1409caa0 con 0x7fcc1c107b60 2024-04-23T14:40:59.874 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:40:59.874 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-23T14:40:59.875 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 2 2024-04-23T14:40:59.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:59 smithi028 ceph-mon[27810]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:40:59.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:59 smithi028 ceph-mon[27810]: mds.? [v2:172.21.15.105:6832/573683071,v1:172.21.15.105:6833/573683071] up:boot 2024-04-23T14:40:59.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:59 smithi028 ceph-mon[27810]: fsmap 1 up:standby 2024-04-23T14:40:59.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:59 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-23T14:40:59.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:59 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:59.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:59 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:59.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:59 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:40:59.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:59 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:40:59.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:59 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:40:59.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:40:59 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:40:59.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.877+0000 7fcc1b200640 1 -- 172.21.15.28:0/2845514698 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fcc000a3bb0 msgr2=0x7fcc000a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:59.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.877+0000 7fcc1b200640 1 --2- 172.21.15.28:0/2845514698 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fcc000a3bb0 0x7fcc000a6070 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c011500 tx=0x7fcc0c005f90 comp rx=0 tx=0).stop 2024-04-23T14:40:59.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.877+0000 7fcc1b200640 1 -- 172.21.15.28:0/2845514698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc1c107b60 msgr2=0x7fcc1c1a1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:40:59.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.877+0000 7fcc1b200640 1 --2- 172.21.15.28:0/2845514698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc1c107b60 0x7fcc1c1a1970 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcc1400da10 tx=0x7fcc1400ded0 comp rx=0 tx=0).stop 2024-04-23T14:40:59.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.877+0000 7fcc1b200640 1 -- 172.21.15.28:0/2845514698 shutdown_connections 2024-04-23T14:40:59.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.877+0000 7fcc1b200640 1 --2- 172.21.15.28:0/2845514698 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fcc000a3bb0 0x7fcc000a6070 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:59.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.877+0000 7fcc1b200640 1 --2- 172.21.15.28:0/2845514698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc1c105230 0x7fcc1c19a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:59.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.877+0000 7fcc1b200640 1 --2- 172.21.15.28:0/2845514698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc1c107b60 0x7fcc1c1a1970 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:59.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.877+0000 7fcc1b200640 1 --2- 172.21.15.28:0/2845514698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc1c102900 0x7fcc1c19a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:40:59.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.877+0000 7fcc1b200640 1 -- 172.21.15.28:0/2845514698 >> 172.21.15.28:0/2845514698 conn(0x7fcc1c0fc2d0 msgr2=0x7fcc1c101fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:40:59.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.877+0000 7fcc1b200640 1 -- 172.21.15.28:0/2845514698 shutdown_connections 2024-04-23T14:40:59.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:40:59.877+0000 7fcc1b200640 1 -- 172.21.15.28:0/2845514698 wait complete. 2024-04-23T14:41:00.726 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:00 smithi038 ceph-mon[32348]: Deploying daemon mds.k on smithi038 2024-04-23T14:41:00.727 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:00 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2845514698' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:00.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:00 smithi105 ceph-mon[32555]: Deploying daemon mds.k on smithi038 2024-04-23T14:41:00.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:00 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2845514698' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:00.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:00 smithi028 ceph-mon[27810]: Deploying daemon mds.k on smithi038 2024-04-23T14:41:00.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:00 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2845514698' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:01.372 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 23 14:41:01 smithi038 systemd[1]: Starting Ceph mds.k for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T14:41:01.714 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:01 smithi038 ceph-mon[32348]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:01.715 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 23 14:41:01 smithi038 podman[62058]: 2024-04-23 14:41:01.317178135 +0000 UTC m=+0.018404300 image pull 76e77f280c74312ca1add9b6019a4a921a516f91795d85bae00a0c8ca9b1c62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd 2024-04-23T14:41:01.715 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 23 14:41:01 smithi038 podman[62058]: 2024-04-23 14:41:01.424236719 +0000 UTC m=+0.125462856 container create 2f2cbd0307e49a235c87f8e811dc6da2c3fb513619cf503739da08d58b198ed0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-k, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T14:41:01.715 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 23 14:41:01 smithi038 podman[62058]: 2024-04-23 14:41:01.635560392 +0000 UTC m=+0.336786529 container init 2f2cbd0307e49a235c87f8e811dc6da2c3fb513619cf503739da08d58b198ed0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-k, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T14:41:01.715 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 23 14:41:01 smithi038 podman[62058]: 2024-04-23 14:41:01.638567537 +0000 UTC m=+0.339793683 container start 2f2cbd0307e49a235c87f8e811dc6da2c3fb513619cf503739da08d58b198ed0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-k, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-23T14:41:01.715 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 23 14:41:01 smithi038 ceph-mds[62077]: -- 172.21.15.38:0/2027239363 <== mon.2 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5596fa019a40 con 0x5596f9fec800 2024-04-23T14:41:01.715 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 23 14:41:01 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-k[62073]: starting mds.k at 2024-04-23T14:41:01.715 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 23 14:41:01 smithi038 bash[62058]: 2f2cbd0307e49a235c87f8e811dc6da2c3fb513619cf503739da08d58b198ed0 2024-04-23T14:41:01.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:01 smithi105 ceph-mon[32555]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:01.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:01 smithi028 ceph-mon[27810]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:01.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:41:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.101+0000 7f2b95800640 1 -- 172.21.15.28:0/1222987470 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b90074a60 msgr2=0x7f2b90074ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.101+0000 7f2b95800640 1 --2- 172.21.15.28:0/1222987470 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b90074a60 0x7f2b90074ee0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c009890 tx=0x7f2b7c02a360 comp rx=0 tx=0).stop 2024-04-23T14:41:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.101+0000 7f2b95800640 1 -- 172.21.15.28:0/1222987470 shutdown_connections 2024-04-23T14:41:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.101+0000 7f2b95800640 1 --2- 172.21.15.28:0/1222987470 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b90074a60 0x7f2b90074ee0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.101+0000 7f2b95800640 1 --2- 172.21.15.28:0/1222987470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b9010e470 0x7f2b90110860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.101+0000 7f2b95800640 1 --2- 172.21.15.28:0/1222987470 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b90076040 0x7f2b90074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.101+0000 7f2b95800640 1 -- 172.21.15.28:0/1222987470 >> 172.21.15.28:0/1222987470 conn(0x7f2b900fd410 msgr2=0x7f2b900ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.102+0000 7f2b95800640 1 -- 172.21.15.28:0/1222987470 shutdown_connections 2024-04-23T14:41:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.102+0000 7f2b95800640 1 -- 172.21.15.28:0/1222987470 wait complete. 2024-04-23T14:41:02.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.103+0000 7f2b95800640 1 Processor -- start 2024-04-23T14:41:02.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.103+0000 7f2b95800640 1 -- start start 2024-04-23T14:41:02.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.103+0000 7f2b95800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b90074a60 0x7f2b9019e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:02.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.103+0000 7f2b95800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b90076040 0x7f2b9019ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:02.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.103+0000 7f2b95800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b9010e470 0x7f2b901a5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:02.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.103+0000 7f2b95800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2b90114130 con 0x7f2b90076040 2024-04-23T14:41:02.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.103+0000 7f2b95800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2b90113fb0 con 0x7f2b90074a60 2024-04-23T14:41:02.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.103+0000 7f2b95800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2b901142b0 con 0x7f2b9010e470 2024-04-23T14:41:02.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.103+0000 7f2b8fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b90074a60 0x7f2b9019e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:02.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.103+0000 7f2b8fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b90074a60 0x7f2b9019e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:37210/0 (socket says 172.21.15.28:37210) 2024-04-23T14:41:02.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.103+0000 7f2b8fe00640 1 -- 172.21.15.28:0/1190206760 learned_addr learned my addr 172.21.15.28:0/1190206760 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:02.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.103+0000 7f2b8f400640 1 --2- 172.21.15.28:0/1190206760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b90076040 0x7f2b9019ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.104+0000 7f2b94e00640 1 --2- 172.21.15.28:0/1190206760 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b9010e470 0x7f2b901a5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.104+0000 7f2b8fe00640 1 -- 172.21.15.28:0/1190206760 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b9010e470 msgr2=0x7f2b901a5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.104+0000 7f2b8fe00640 1 --2- 172.21.15.28:0/1190206760 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b9010e470 0x7f2b901a5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.104+0000 7f2b8fe00640 1 -- 172.21.15.28:0/1190206760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b90076040 msgr2=0x7f2b9019ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.104+0000 7f2b8fe00640 1 --2- 172.21.15.28:0/1190206760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b90076040 0x7f2b9019ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.104+0000 7f2b8fe00640 1 -- 172.21.15.28:0/1190206760 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b901a6410 con 0x7f2b90074a60 2024-04-23T14:41:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.104+0000 7f2b8f400640 1 --2- 172.21.15.28:0/1190206760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b90076040 0x7f2b9019ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:41:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.104+0000 7f2b8fe00640 1 --2- 172.21.15.28:0/1190206760 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b90074a60 0x7f2b9019e7f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2b8000e9f0 tx=0x7f2b8000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.104+0000 7f2b8cc00640 1 -- 172.21.15.28:0/1190206760 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8000cd80 con 0x7f2b90074a60 2024-04-23T14:41:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.104+0000 7f2b8cc00640 1 -- 172.21.15.28:0/1190206760 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b80005500 con 0x7f2b90074a60 2024-04-23T14:41:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.104+0000 7f2b8cc00640 1 -- 172.21.15.28:0/1190206760 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b80010690 con 0x7f2b90074a60 2024-04-23T14:41:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.104+0000 7f2b95800640 1 -- 172.21.15.28:0/1190206760 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b901a6700 con 0x7f2b90074a60 2024-04-23T14:41:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.104+0000 7f2b95800640 1 -- 172.21.15.28:0/1190206760 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b901a6bd0 con 0x7f2b90074a60 2024-04-23T14:41:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.106+0000 7f2b95800640 1 -- 172.21.15.28:0/1190206760 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b90104640 con 0x7f2b90074a60 2024-04-23T14:41:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.107+0000 7f2b8cc00640 1 -- 172.21.15.28:0/1190206760 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2b800046c0 con 0x7f2b90074a60 2024-04-23T14:41:02.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.108+0000 7f2b8cc00640 1 --2- 172.21.15.28:0/1190206760 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2b740a3c30 0x7f2b740a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:02.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.109+0000 7f2b8cc00640 1 -- 172.21.15.28:0/1190206760 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f2b800d04d0 con 0x7f2b90074a60 2024-04-23T14:41:02.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.109+0000 7f2b8f400640 1 --2- 172.21.15.28:0/1190206760 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2b740a3c30 0x7f2b740a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:02.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.110+0000 7f2b8f400640 1 --2- 172.21.15.28:0/1190206760 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2b740a3c30 0x7f2b740a60f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c02a950 tx=0x7f2b7c009540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:02.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.113+0000 7f2b8cc00640 1 -- 172.21.15.28:0/1190206760 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2b8009d2b0 con 0x7f2b90074a60 2024-04-23T14:41:02.122 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 23 14:41:01 smithi038 systemd[1]: Started Ceph mds.k for 1ee6e996-017e-11ef-bc93-c7b262605968. 2024-04-23T14:41:02.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.382+0000 7f2b95800640 1 -- 172.21.15.28:0/1190206760 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b9002d9e0 con 0x7f2b90074a60 2024-04-23T14:41:02.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.383+0000 7f2b8cc00640 1 -- 172.21.15.28:0/1190206760 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1393 (secure 0 0 0) 0x7f2b8009cbb0 con 0x7f2b90074a60 2024-04-23T14:41:02.384 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:41:02.385 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-23T14:41:02.385 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 2 2024-04-23T14:41:02.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.386+0000 7f2b95800640 1 -- 172.21.15.28:0/1190206760 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2b740a3c30 msgr2=0x7f2b740a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:02.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.386+0000 7f2b95800640 1 --2- 172.21.15.28:0/1190206760 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2b740a3c30 0x7f2b740a60f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c02a950 tx=0x7f2b7c009540 comp rx=0 tx=0).stop 2024-04-23T14:41:02.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.386+0000 7f2b95800640 1 -- 172.21.15.28:0/1190206760 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b90074a60 msgr2=0x7f2b9019e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:02.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.386+0000 7f2b95800640 1 --2- 172.21.15.28:0/1190206760 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b90074a60 0x7f2b9019e7f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2b8000e9f0 tx=0x7f2b8000edf0 comp rx=0 tx=0).stop 2024-04-23T14:41:02.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.386+0000 7f2b95800640 1 -- 172.21.15.28:0/1190206760 shutdown_connections 2024-04-23T14:41:02.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.386+0000 7f2b95800640 1 --2- 172.21.15.28:0/1190206760 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b9010e470 0x7f2b901a5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:02.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.386+0000 7f2b95800640 1 --2- 172.21.15.28:0/1190206760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b90076040 0x7f2b9019ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:02.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.386+0000 7f2b95800640 1 --2- 172.21.15.28:0/1190206760 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2b740a3c30 0x7f2b740a60f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:02.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.386+0000 7f2b95800640 1 --2- 172.21.15.28:0/1190206760 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b90074a60 0x7f2b9019e7f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:02.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.386+0000 7f2b95800640 1 -- 172.21.15.28:0/1190206760 >> 172.21.15.28:0/1190206760 conn(0x7f2b900fd410 msgr2=0x7f2b90101810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:02.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.387+0000 7f2b95800640 1 -- 172.21.15.28:0/1190206760 shutdown_connections 2024-04-23T14:41:02.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:02.387+0000 7f2b95800640 1 -- 172.21.15.28:0/1190206760 wait complete. 2024-04-23T14:41:03.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:02 smithi028 ceph-mon[27810]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:03.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:02 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:03.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:02 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:03.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:02 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:03.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:02 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:41:03.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:02 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:41:03.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:02 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:03.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:02 smithi028 ceph-mon[27810]: Deploying daemon mds.j on smithi028 2024-04-23T14:41:03.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:02 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1190206760' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:03.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:02 smithi028 ceph-mon[27810]: mds.? [v2:172.21.15.38:6832/1673432987,v1:172.21.15.38:6833/1673432987] up:boot 2024-04-23T14:41:03.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:02 smithi028 ceph-mon[27810]: fsmap 2 up:standby 2024-04-23T14:41:03.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:02 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-23T14:41:03.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:02 smithi105 ceph-mon[32555]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:03.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:02 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:03.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:02 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:03.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:02 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:03.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:02 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:41:03.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:02 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:41:03.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:02 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:03.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:02 smithi105 ceph-mon[32555]: Deploying daemon mds.j on smithi028 2024-04-23T14:41:03.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:02 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1190206760' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:03.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:02 smithi105 ceph-mon[32555]: mds.? [v2:172.21.15.38:6832/1673432987,v1:172.21.15.38:6833/1673432987] up:boot 2024-04-23T14:41:03.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:02 smithi105 ceph-mon[32555]: fsmap 2 up:standby 2024-04-23T14:41:03.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:02 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-23T14:41:03.044 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:02 smithi038 ceph-mon[32348]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:03.044 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:02 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:03.044 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:02 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:03.044 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:02 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:03.044 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:02 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:41:03.045 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:02 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:41:03.045 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:02 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:03.045 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:02 smithi038 ceph-mon[32348]: Deploying daemon mds.j on smithi028 2024-04-23T14:41:03.045 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:02 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1190206760' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:03.045 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:02 smithi038 ceph-mon[32348]: mds.? [v2:172.21.15.38:6832/1673432987,v1:172.21.15.38:6833/1673432987] up:boot 2024-04-23T14:41:03.045 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:02 smithi038 ceph-mon[32348]: fsmap 2 up:standby 2024-04-23T14:41:03.045 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:02 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-23T14:41:04.406 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:41:04.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.811+0000 7f00b7000640 1 -- 172.21.15.28:0/2249132294 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f00b010a210 msgr2=0x7f00b0110aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:04.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.811+0000 7f00b7000640 1 --2- 172.21.15.28:0/2249132294 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f00b010a210 0x7f00b0110aa0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f00a8009be0 tx=0x7f00a802a6c0 comp rx=0 tx=0).stop 2024-04-23T14:41:04.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.811+0000 7f00b7000640 1 -- 172.21.15.28:0/2249132294 shutdown_connections 2024-04-23T14:41:04.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.811+0000 7f00b7000640 1 --2- 172.21.15.28:0/2249132294 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00b0109850 0x7f00b0109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:04.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.811+0000 7f00b7000640 1 --2- 172.21.15.28:0/2249132294 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f00b010a210 0x7f00b0110aa0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:04.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.811+0000 7f00b7000640 1 --2- 172.21.15.28:0/2249132294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00b0108660 0x7f00b0108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:04.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.811+0000 7f00b7000640 1 -- 172.21.15.28:0/2249132294 >> 172.21.15.28:0/2249132294 conn(0x7f00b0077620 msgr2=0x7f00b0079a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:04.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.811+0000 7f00b7000640 1 -- 172.21.15.28:0/2249132294 shutdown_connections 2024-04-23T14:41:04.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.811+0000 7f00b7000640 1 -- 172.21.15.28:0/2249132294 wait complete. 2024-04-23T14:41:04.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.812+0000 7f00b7000640 1 Processor -- start 2024-04-23T14:41:04.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.813+0000 7f00b7000640 1 -- start start 2024-04-23T14:41:04.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.813+0000 7f00b7000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f00b0108660 0x7f00b019e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:04.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.813+0000 7f00b7000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00b0109850 0x7f00b019ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:04.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.813+0000 7f00b7000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00b010a210 0x7f00b01a5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:04.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.813+0000 7f00b7000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f00b01142f0 con 0x7f00b0109850 2024-04-23T14:41:04.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.813+0000 7f00b7000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f00b0114170 con 0x7f00b010a210 2024-04-23T14:41:04.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.813+0000 7f00b7000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f00b0114470 con 0x7f00b0108660 2024-04-23T14:41:04.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.813+0000 7f00af400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00b0109850 0x7f00b019ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:04.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.813+0000 7f00af400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00b0109850 0x7f00b019ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46396/0 (socket says 172.21.15.28:46396) 2024-04-23T14:41:04.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.813+0000 7f00af400640 1 -- 172.21.15.28:0/538101580 learned_addr learned my addr 172.21.15.28:0/538101580 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.813+0000 7f00afe00640 1 --2- 172.21.15.28:0/538101580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f00b0108660 0x7f00b019e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.813+0000 7f00b5000640 1 --2- 172.21.15.28:0/538101580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00b010a210 0x7f00b01a5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.814+0000 7f00af400640 1 -- 172.21.15.28:0/538101580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f00b0108660 msgr2=0x7f00b019e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.814+0000 7f00af400640 1 --2- 172.21.15.28:0/538101580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f00b0108660 0x7f00b019e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.814+0000 7f00af400640 1 -- 172.21.15.28:0/538101580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00b010a210 msgr2=0x7f00b01a5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.814+0000 7f00af400640 1 --2- 172.21.15.28:0/538101580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00b010a210 0x7f00b01a5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.814+0000 7f00af400640 1 -- 172.21.15.28:0/538101580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00b01a6560 con 0x7f00b0109850 2024-04-23T14:41:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.814+0000 7f00b5000640 1 --2- 172.21.15.28:0/538101580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00b010a210 0x7f00b01a5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:41:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.814+0000 7f00af400640 1 --2- 172.21.15.28:0/538101580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00b0109850 0x7f00b019ee80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f009c00e9f0 tx=0x7f009c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.814+0000 7f00acc00640 1 -- 172.21.15.28:0/538101580 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009c00cd80 con 0x7f00b0109850 2024-04-23T14:41:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.814+0000 7f00b7000640 1 -- 172.21.15.28:0/538101580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00b01a6850 con 0x7f00b0109850 2024-04-23T14:41:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.815+0000 7f00b7000640 1 -- 172.21.15.28:0/538101580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00b01a6da0 con 0x7f00b0109850 2024-04-23T14:41:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.815+0000 7f00acc00640 1 -- 172.21.15.28:0/538101580 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f009c005500 con 0x7f00b0109850 2024-04-23T14:41:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.815+0000 7f00acc00640 1 -- 172.21.15.28:0/538101580 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009c010690 con 0x7f00b0109850 2024-04-23T14:41:04.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.816+0000 7f00b7000640 1 -- 172.21.15.28:0/538101580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00900052c0 con 0x7f00b0109850 2024-04-23T14:41:04.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.819+0000 7f00acc00640 1 -- 172.21.15.28:0/538101580 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f009c0046c0 con 0x7f00b0109850 2024-04-23T14:41:04.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.820+0000 7f00acc00640 1 --2- 172.21.15.28:0/538101580 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f00940a3c30 0x7f00940a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:04.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.821+0000 7f00acc00640 1 -- 172.21.15.28:0/538101580 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f009c0d02a0 con 0x7f00b0109850 2024-04-23T14:41:04.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.821+0000 7f00afe00640 1 --2- 172.21.15.28:0/538101580 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f00940a3c30 0x7f00940a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:04.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.822+0000 7f00afe00640 1 --2- 172.21.15.28:0/538101580 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f00940a3c30 0x7f00940a60f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f00a0009530 tx=0x7f00a0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:04.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:04.826+0000 7f00acc00640 1 -- 172.21.15.28:0/538101580 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f009c09d080 con 0x7f00b0109850 2024-04-23T14:41:05.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:04 smithi105 ceph-mon[32555]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:05.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:04 smithi028 ceph-mon[27810]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:05.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:05.082+0000 7f00b7000640 1 -- 172.21.15.28:0/538101580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00900055b0 con 0x7f00b0109850 2024-04-23T14:41:05.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:05.083+0000 7f00acc00640 1 -- 172.21.15.28:0/538101580 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2218 (secure 0 0 0) 0x7f009c09c980 con 0x7f00b0109850 2024-04-23T14:41:05.083 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:41:05.083 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-23T14:41:05.084 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 3 2024-04-23T14:41:05.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:05.085+0000 7f00b7000640 1 -- 172.21.15.28:0/538101580 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f00940a3c30 msgr2=0x7f00940a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:05.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:05.085+0000 7f00b7000640 1 --2- 172.21.15.28:0/538101580 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f00940a3c30 0x7f00940a60f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f00a0009530 tx=0x7f00a0009260 comp rx=0 tx=0).stop 2024-04-23T14:41:05.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:05.085+0000 7f00b7000640 1 -- 172.21.15.28:0/538101580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00b0109850 msgr2=0x7f00b019ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:05.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:05.085+0000 7f00b7000640 1 --2- 172.21.15.28:0/538101580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00b0109850 0x7f00b019ee80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f009c00e9f0 tx=0x7f009c00edf0 comp rx=0 tx=0).stop 2024-04-23T14:41:05.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:05.086+0000 7f00b7000640 1 -- 172.21.15.28:0/538101580 shutdown_connections 2024-04-23T14:41:05.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:05.086+0000 7f00b7000640 1 --2- 172.21.15.28:0/538101580 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f00940a3c30 0x7f00940a60f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:05.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:05.086+0000 7f00b7000640 1 --2- 172.21.15.28:0/538101580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00b010a210 0x7f00b01a5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:05.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:05.086+0000 7f00b7000640 1 --2- 172.21.15.28:0/538101580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00b0109850 0x7f00b019ee80 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:05.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:05.086+0000 7f00b7000640 1 --2- 172.21.15.28:0/538101580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f00b0108660 0x7f00b019e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:05.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:05.086+0000 7f00b7000640 1 -- 172.21.15.28:0/538101580 >> 172.21.15.28:0/538101580 conn(0x7f00b0077620 msgr2=0x7f00b010cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:05.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:05.086+0000 7f00b7000640 1 -- 172.21.15.28:0/538101580 shutdown_connections 2024-04-23T14:41:05.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:05.086+0000 7f00b7000640 1 -- 172.21.15.28:0/538101580 wait complete. 2024-04-23T14:41:05.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:04 smithi038 ceph-mon[32348]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:05.624 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 23 14:41:05 smithi028 systemd[1]: Started Ceph mds.j for 1ee6e996-017e-11ef-bc93-c7b262605968. 2024-04-23T14:41:06.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:05 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/538101580' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:06.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:05 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:06.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:05 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:06.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:05 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:06.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:05 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:41:06.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:05 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:41:06.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:05 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:06.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:05 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/538101580' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:06.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:05 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:06.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:05 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:06.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:05 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:06.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:05 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:41:06.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:05 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:41:06.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:05 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:06.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:05 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/538101580' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:06.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:05 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:06.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:05 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:06.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:05 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:06.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:05 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:41:06.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:05 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:41:06.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:05 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:06.995 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:06 smithi028 ceph-mon[27810]: Deploying daemon mds.d on smithi028 2024-04-23T14:41:06.996 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:06 smithi028 ceph-mon[27810]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:06.996 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:06 smithi028 ceph-mon[27810]: mds.? [v2:172.21.15.28:6834/760880137,v1:172.21.15.28:6835/760880137] up:boot 2024-04-23T14:41:06.996 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:06 smithi028 ceph-mon[27810]: fsmap 3 up:standby 2024-04-23T14:41:06.996 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:06 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-23T14:41:07.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:06 smithi105 ceph-mon[32555]: Deploying daemon mds.d on smithi028 2024-04-23T14:41:07.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:06 smithi105 ceph-mon[32555]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:07.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:06 smithi105 ceph-mon[32555]: mds.? [v2:172.21.15.28:6834/760880137,v1:172.21.15.28:6835/760880137] up:boot 2024-04-23T14:41:07.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:06 smithi105 ceph-mon[32555]: fsmap 3 up:standby 2024-04-23T14:41:07.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:06 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-23T14:41:07.107 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:41:07.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:06 smithi038 ceph-mon[32348]: Deploying daemon mds.d on smithi028 2024-04-23T14:41:07.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:06 smithi038 ceph-mon[32348]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:07.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:06 smithi038 ceph-mon[32348]: mds.? [v2:172.21.15.28:6834/760880137,v1:172.21.15.28:6835/760880137] up:boot 2024-04-23T14:41:07.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:06 smithi038 ceph-mon[32348]: fsmap 3 up:standby 2024-04-23T14:41:07.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:06 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-23T14:41:07.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.318+0000 7fb4ea400640 1 -- 172.21.15.28:0/3338256390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4e4105270 msgr2=0x7fb4e41056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:07.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.318+0000 7fb4ea400640 1 --2- 172.21.15.28:0/3338256390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4e4105270 0x7fb4e41056f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0009900 tx=0x7fb4d002a3f0 comp rx=0 tx=0).stop 2024-04-23T14:41:07.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.318+0000 7fb4ea400640 1 -- 172.21.15.28:0/3338256390 shutdown_connections 2024-04-23T14:41:07.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.318+0000 7fb4ea400640 1 --2- 172.21.15.28:0/3338256390 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb4e4105c30 0x7fb4e410c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:07.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.318+0000 7fb4ea400640 1 --2- 172.21.15.28:0/3338256390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4e4105270 0x7fb4e41056f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:07.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.318+0000 7fb4ea400640 1 --2- 172.21.15.28:0/3338256390 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb4e4104080 0x7fb4e41044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:07.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.318+0000 7fb4ea400640 1 -- 172.21.15.28:0/3338256390 >> 172.21.15.28:0/3338256390 conn(0x7fb4e40ff610 msgr2=0x7fb4e4101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:07.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.318+0000 7fb4ea400640 1 -- 172.21.15.28:0/3338256390 shutdown_connections 2024-04-23T14:41:07.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.318+0000 7fb4ea400640 1 -- 172.21.15.28:0/3338256390 wait complete. 2024-04-23T14:41:07.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.319+0000 7fb4ea400640 1 Processor -- start 2024-04-23T14:41:07.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.319+0000 7fb4ea400640 1 -- start start 2024-04-23T14:41:07.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.320+0000 7fb4ea400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb4e4104080 0x7fb4e419a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:07.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.320+0000 7fb4ea400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4e4105270 0x7fb4e419a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:07.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.320+0000 7fb4ea400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb4e4105c30 0x7fb4e41a18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:07.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.320+0000 7fb4ea400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e410fd60 con 0x7fb4e4105270 2024-04-23T14:41:07.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.320+0000 7fb4ea400640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e410fbe0 con 0x7fb4e4105c30 2024-04-23T14:41:07.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.320+0000 7fb4ea400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e410fee0 con 0x7fb4e4104080 2024-04-23T14:41:07.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.320+0000 7fb4e3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4e4105270 0x7fb4e419a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:07.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.320+0000 7fb4e9000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb4e4104080 0x7fb4e419a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:07.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.320+0000 7fb4e9000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb4e4104080 0x7fb4e419a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:58858/0 (socket says 172.21.15.28:58858) 2024-04-23T14:41:07.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.320+0000 7fb4e9000640 1 -- 172.21.15.28:0/1769354632 learned_addr learned my addr 172.21.15.28:0/1769354632 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:07.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.320+0000 7fb4e9a00640 1 --2- 172.21.15.28:0/1769354632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb4e4105c30 0x7fb4e41a18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:07.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.320+0000 7fb4e3e00640 1 -- 172.21.15.28:0/1769354632 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb4e4104080 msgr2=0x7fb4e419a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:07.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.320+0000 7fb4e3e00640 1 --2- 172.21.15.28:0/1769354632 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb4e4104080 0x7fb4e419a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:07.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.320+0000 7fb4e3e00640 1 -- 172.21.15.28:0/1769354632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb4e4105c30 msgr2=0x7fb4e41a18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:07.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.320+0000 7fb4e3e00640 1 --2- 172.21.15.28:0/1769354632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb4e4105c30 0x7fb4e41a18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:07.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.320+0000 7fb4e3e00640 1 -- 172.21.15.28:0/1769354632 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4e41a1f40 con 0x7fb4e4105270 2024-04-23T14:41:07.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.321+0000 7fb4e3e00640 1 --2- 172.21.15.28:0/1769354632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4e4105270 0x7fb4e419a860 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb4d002a910 tx=0x7fb4d0032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:07.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.321+0000 7fb4e1600640 1 -- 172.21.15.28:0/1769354632 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d003f030 con 0x7fb4e4105270 2024-04-23T14:41:07.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.321+0000 7fb4e1600640 1 -- 172.21.15.28:0/1769354632 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4d0003ce0 con 0x7fb4e4105270 2024-04-23T14:41:07.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.321+0000 7fb4ea400640 1 -- 172.21.15.28:0/1769354632 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4e41a21d0 con 0x7fb4e4105270 2024-04-23T14:41:07.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.321+0000 7fb4e1600640 1 -- 172.21.15.28:0/1769354632 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d0035070 con 0x7fb4e4105270 2024-04-23T14:41:07.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.321+0000 7fb4ea400640 1 -- 172.21.15.28:0/1769354632 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4e41a2690 con 0x7fb4e4105270 2024-04-23T14:41:07.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.322+0000 7fb4ea400640 1 -- 172.21.15.28:0/1769354632 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4e410b3a0 con 0x7fb4e4105270 2024-04-23T14:41:07.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.324+0000 7fb4e1600640 1 -- 172.21.15.28:0/1769354632 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb4d0042030 con 0x7fb4e4105270 2024-04-23T14:41:07.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.326+0000 7fb4e1600640 1 --2- 172.21.15.28:0/1769354632 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb4c80a3bb0 0x7fb4c80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:07.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.326+0000 7fb4e1600640 1 -- 172.21.15.28:0/1769354632 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb4d00f0460 con 0x7fb4e4105270 2024-04-23T14:41:07.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.326+0000 7fb4e9000640 1 --2- 172.21.15.28:0/1769354632 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb4c80a3bb0 0x7fb4c80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:07.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.327+0000 7fb4e9000640 1 --2- 172.21.15.28:0/1769354632 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb4c80a3bb0 0x7fb4c80a6070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb4d4005fd0 tx=0x7fb4d400b4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:07.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.331+0000 7fb4e1600640 1 -- 172.21.15.28:0/1769354632 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb4d00bcbf0 con 0x7fb4e4105270 2024-04-23T14:41:07.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.584+0000 7fb4ea400640 1 -- 172.21.15.28:0/1769354632 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb4e402d620 con 0x7fb4e4105270 2024-04-23T14:41:07.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.586+0000 7fb4e1600640 1 -- 172.21.15.28:0/1769354632 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3040 (secure 0 0 0) 0x7fb4d00c0ef0 con 0x7fb4e4105270 2024-04-23T14:41:07.586 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:41:07.586 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-23T14:41:07.586 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 4 2024-04-23T14:41:07.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.588+0000 7fb4ea400640 1 -- 172.21.15.28:0/1769354632 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb4c80a3bb0 msgr2=0x7fb4c80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:07.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.588+0000 7fb4ea400640 1 --2- 172.21.15.28:0/1769354632 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb4c80a3bb0 0x7fb4c80a6070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb4d4005fd0 tx=0x7fb4d400b4a0 comp rx=0 tx=0).stop 2024-04-23T14:41:07.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.588+0000 7fb4ea400640 1 -- 172.21.15.28:0/1769354632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4e4105270 msgr2=0x7fb4e419a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:07.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.588+0000 7fb4ea400640 1 --2- 172.21.15.28:0/1769354632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4e4105270 0x7fb4e419a860 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb4d002a910 tx=0x7fb4d0032ce0 comp rx=0 tx=0).stop 2024-04-23T14:41:07.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.588+0000 7fb4ea400640 1 -- 172.21.15.28:0/1769354632 shutdown_connections 2024-04-23T14:41:07.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.588+0000 7fb4ea400640 1 --2- 172.21.15.28:0/1769354632 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb4c80a3bb0 0x7fb4c80a6070 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:07.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.588+0000 7fb4ea400640 1 --2- 172.21.15.28:0/1769354632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb4e4105c30 0x7fb4e41a18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:07.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.588+0000 7fb4ea400640 1 --2- 172.21.15.28:0/1769354632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4e4105270 0x7fb4e419a860 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:07.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.588+0000 7fb4ea400640 1 --2- 172.21.15.28:0/1769354632 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb4e4104080 0x7fb4e419a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:07.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.589+0000 7fb4ea400640 1 -- 172.21.15.28:0/1769354632 >> 172.21.15.28:0/1769354632 conn(0x7fb4e40ff610 msgr2=0x7fb4e4108570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:07.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.589+0000 7fb4ea400640 1 -- 172.21.15.28:0/1769354632 shutdown_connections 2024-04-23T14:41:07.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:07.589+0000 7fb4ea400640 1 -- 172.21.15.28:0/1769354632 wait complete. 2024-04-23T14:41:08.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:07 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1769354632' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:08.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:07 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1769354632' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:08.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:07 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1769354632' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:09.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:08 smithi038 ceph-mon[32348]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:09.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:08 smithi028 ceph-mon[27810]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:09.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:08 smithi105 ceph-mon[32555]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:09.462 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 23 14:41:09 smithi028 podman[66621]: 2024-04-23 14:41:09.156598402 +0000 UTC m=+0.150204970 container create 8bb57b364e12a222139c87969c10087984175a81d955a599be2665ac9b72296b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-d, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T14:41:09.463 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 23 14:41:09 smithi028 podman[66621]: 2024-04-23 14:41:09.460390486 +0000 UTC m=+0.453997079 container init 8bb57b364e12a222139c87969c10087984175a81d955a599be2665ac9b72296b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-d, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-23T14:41:09.609 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:41:09.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.833+0000 7f5d43200640 1 -- 172.21.15.28:0/1955283637 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d441040a0 msgr2=0x7f5d441044c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:09.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.833+0000 7f5d43200640 1 --2- 172.21.15.28:0/1955283637 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d441040a0 0x7f5d441044c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5d34009900 tx=0x7f5d3402a410 comp rx=0 tx=0).stop 2024-04-23T14:41:09.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.834+0000 7f5d43200640 1 -- 172.21.15.28:0/1955283637 shutdown_connections 2024-04-23T14:41:09.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.834+0000 7f5d43200640 1 --2- 172.21.15.28:0/1955283637 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5d44105c50 0x7f5d4410c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:09.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.834+0000 7f5d43200640 1 --2- 172.21.15.28:0/1955283637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d44105290 0x7f5d44105710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:09.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.834+0000 7f5d43200640 1 --2- 172.21.15.28:0/1955283637 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d441040a0 0x7f5d441044c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:09.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.834+0000 7f5d43200640 1 -- 172.21.15.28:0/1955283637 >> 172.21.15.28:0/1955283637 conn(0x7f5d440ff610 msgr2=0x7f5d44101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:09.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.834+0000 7f5d43200640 1 -- 172.21.15.28:0/1955283637 shutdown_connections 2024-04-23T14:41:09.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.834+0000 7f5d43200640 1 -- 172.21.15.28:0/1955283637 wait complete. 2024-04-23T14:41:09.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.835+0000 7f5d43200640 1 Processor -- start 2024-04-23T14:41:09.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.835+0000 7f5d43200640 1 -- start start 2024-04-23T14:41:09.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.835+0000 7f5d43200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d441040a0 0x7f5d4419a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:09.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.835+0000 7f5d43200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d44105290 0x7f5d4419a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:09.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.835+0000 7f5d43200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5d44105c50 0x7f5d441a1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:09.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.835+0000 7f5d43200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4410fd40 con 0x7f5d441040a0 2024-04-23T14:41:09.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.835+0000 7f5d43200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4410fbc0 con 0x7f5d44105290 2024-04-23T14:41:09.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.835+0000 7f5d43200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4410fec0 con 0x7f5d44105c50 2024-04-23T14:41:09.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.836+0000 7f5d41400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d44105290 0x7f5d4419a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:09.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.836+0000 7f5d41e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d441040a0 0x7f5d4419a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:09.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.836+0000 7f5d41400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d44105290 0x7f5d4419a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:51154/0 (socket says 172.21.15.28:51154) 2024-04-23T14:41:09.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.836+0000 7f5d41400640 1 -- 172.21.15.28:0/4213380471 learned_addr learned my addr 172.21.15.28:0/4213380471 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:09.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.836+0000 7f5d42800640 1 --2- 172.21.15.28:0/4213380471 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5d44105c50 0x7f5d441a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:09.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.836+0000 7f5d41400640 1 -- 172.21.15.28:0/4213380471 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5d44105c50 msgr2=0x7f5d441a1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:09.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.836+0000 7f5d41400640 1 --2- 172.21.15.28:0/4213380471 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5d44105c50 0x7f5d441a1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:09.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.836+0000 7f5d41400640 1 -- 172.21.15.28:0/4213380471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d441040a0 msgr2=0x7f5d4419a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:09.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.836+0000 7f5d41400640 1 --2- 172.21.15.28:0/4213380471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d441040a0 0x7f5d4419a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:09.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.836+0000 7f5d41400640 1 -- 172.21.15.28:0/4213380471 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d441a2000 con 0x7f5d44105290 2024-04-23T14:41:09.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.836+0000 7f5d42800640 1 --2- 172.21.15.28:0/4213380471 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5d44105c50 0x7f5d441a1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:41:09.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.836+0000 7f5d41e00640 1 --2- 172.21.15.28:0/4213380471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d441040a0 0x7f5d4419a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:41:09.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.836+0000 7f5d41400640 1 --2- 172.21.15.28:0/4213380471 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d44105290 0x7f5d4419a920 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5d3000e9f0 tx=0x7f5d3000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:09.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.837+0000 7f5d3aa00640 1 -- 172.21.15.28:0/4213380471 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d3000ccc0 con 0x7f5d44105290 2024-04-23T14:41:09.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.837+0000 7f5d3aa00640 1 -- 172.21.15.28:0/4213380471 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d30010910 con 0x7f5d44105290 2024-04-23T14:41:09.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.837+0000 7f5d43200640 1 -- 172.21.15.28:0/4213380471 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d441a22f0 con 0x7f5d44105290 2024-04-23T14:41:09.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.837+0000 7f5d3aa00640 1 -- 172.21.15.28:0/4213380471 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d300188a0 con 0x7f5d44105290 2024-04-23T14:41:09.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.837+0000 7f5d43200640 1 -- 172.21.15.28:0/4213380471 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d441a2890 con 0x7f5d44105290 2024-04-23T14:41:09.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.838+0000 7f5d43200640 1 -- 172.21.15.28:0/4213380471 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d4410b440 con 0x7f5d44105290 2024-04-23T14:41:09.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.841+0000 7f5d3aa00640 1 -- 172.21.15.28:0/4213380471 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5d30010ac0 con 0x7f5d44105290 2024-04-23T14:41:09.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.842+0000 7f5d3aa00640 1 --2- 172.21.15.28:0/4213380471 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f5d280a3c30 0x7f5d280a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:09.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.842+0000 7f5d3aa00640 1 -- 172.21.15.28:0/4213380471 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f5d300d03a0 con 0x7f5d44105290 2024-04-23T14:41:09.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.843+0000 7f5d41e00640 1 --2- 172.21.15.28:0/4213380471 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f5d280a3c30 0x7f5d280a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:09.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.843+0000 7f5d41e00640 1 --2- 172.21.15.28:0/4213380471 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f5d280a3c30 0x7f5d280a60f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5d3402aa00 tx=0x7f5d340315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:09.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:09.844+0000 7f5d3aa00640 1 -- 172.21.15.28:0/4213380471 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5d3009d180 con 0x7f5d44105290 2024-04-23T14:41:09.874 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 23 14:41:09 smithi028 podman[66621]: 2024-04-23 14:41:09.470697382 +0000 UTC m=+0.464303937 container start 8bb57b364e12a222139c87969c10087984175a81d955a599be2665ac9b72296b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-23T14:41:09.874 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 23 14:41:09 smithi028 ceph-mds[66640]: -- 172.21.15.28:0/3190889047 <== mon.2 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5599f1b45a40 con 0x5599f1b19000 2024-04-23T14:41:09.874 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 23 14:41:09 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-d[66636]: starting mds.d at 2024-04-23T14:41:09.874 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 23 14:41:09 smithi028 bash[66621]: 8bb57b364e12a222139c87969c10087984175a81d955a599be2665ac9b72296b 2024-04-23T14:41:09.874 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 23 14:41:09 smithi028 systemd[1]: Started Ceph mds.d for 1ee6e996-017e-11ef-bc93-c7b262605968. 2024-04-23T14:41:10.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:10.095+0000 7f5d43200640 1 -- 172.21.15.28:0/4213380471 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5d4402d620 con 0x7f5d44105290 2024-04-23T14:41:10.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:10.096+0000 7f5d3aa00640 1 -- 172.21.15.28:0/4213380471 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3865 (secure 0 0 0) 0x7f5d3009ca80 con 0x7f5d44105290 2024-04-23T14:41:10.099 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:41:10.100 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-23T14:41:10.101 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 5 2024-04-23T14:41:10.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:10.098+0000 7f5d43200640 1 -- 172.21.15.28:0/4213380471 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f5d280a3c30 msgr2=0x7f5d280a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:10.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:10.098+0000 7f5d43200640 1 --2- 172.21.15.28:0/4213380471 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f5d280a3c30 0x7f5d280a60f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5d3402aa00 tx=0x7f5d340315b0 comp rx=0 tx=0).stop 2024-04-23T14:41:10.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:10.098+0000 7f5d43200640 1 -- 172.21.15.28:0/4213380471 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d44105290 msgr2=0x7f5d4419a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:10.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:10.098+0000 7f5d43200640 1 --2- 172.21.15.28:0/4213380471 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d44105290 0x7f5d4419a920 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5d3000e9f0 tx=0x7f5d3000edf0 comp rx=0 tx=0).stop 2024-04-23T14:41:10.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:10.098+0000 7f5d43200640 1 -- 172.21.15.28:0/4213380471 shutdown_connections 2024-04-23T14:41:10.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:10.098+0000 7f5d43200640 1 --2- 172.21.15.28:0/4213380471 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f5d280a3c30 0x7f5d280a60f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:10.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:10.098+0000 7f5d43200640 1 --2- 172.21.15.28:0/4213380471 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d44105290 0x7f5d4419a920 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:10.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:10.098+0000 7f5d43200640 1 --2- 172.21.15.28:0/4213380471 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5d44105c50 0x7f5d441a1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:10.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:10.098+0000 7f5d43200640 1 --2- 172.21.15.28:0/4213380471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d441040a0 0x7f5d4419a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:10.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:10.098+0000 7f5d43200640 1 -- 172.21.15.28:0/4213380471 >> 172.21.15.28:0/4213380471 conn(0x7f5d440ff610 msgr2=0x7f5d44108610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:10.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:10.099+0000 7f5d43200640 1 -- 172.21.15.28:0/4213380471 shutdown_connections 2024-04-23T14:41:10.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:10.099+0000 7f5d43200640 1 -- 172.21.15.28:0/4213380471 wait complete. 2024-04-23T14:41:11.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:10 smithi105 ceph-mon[32555]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:11.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:10 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:11.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:10 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:11.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:10 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:11.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:10 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:41:11.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:10 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:41:11.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:10 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:11.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:10 smithi105 ceph-mon[32555]: Deploying daemon mds.e on smithi038 2024-04-23T14:41:11.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:10 smithi105 ceph-mon[32555]: mds.? [v2:172.21.15.28:6836/2963041370,v1:172.21.15.28:6837/2963041370] up:boot 2024-04-23T14:41:11.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:10 smithi105 ceph-mon[32555]: fsmap 4 up:standby 2024-04-23T14:41:11.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:10 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-23T14:41:11.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:10 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/4213380471' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:11.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:10 smithi038 ceph-mon[32348]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:11.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:10 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:11.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:10 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:11.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:10 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:11.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:10 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:41:11.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:10 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:41:11.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:10 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:11.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:10 smithi038 ceph-mon[32348]: Deploying daemon mds.e on smithi038 2024-04-23T14:41:11.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:10 smithi038 ceph-mon[32348]: mds.? [v2:172.21.15.28:6836/2963041370,v1:172.21.15.28:6837/2963041370] up:boot 2024-04-23T14:41:11.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:10 smithi038 ceph-mon[32348]: fsmap 4 up:standby 2024-04-23T14:41:11.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:10 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-23T14:41:11.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:10 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/4213380471' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:11.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:10 smithi028 ceph-mon[27810]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:11.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:10 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:11.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:10 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:11.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:10 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:11.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:10 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:41:11.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:10 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:41:11.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:10 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:11.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:10 smithi028 ceph-mon[27810]: Deploying daemon mds.e on smithi038 2024-04-23T14:41:11.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:10 smithi028 ceph-mon[27810]: mds.? [v2:172.21.15.28:6836/2963041370,v1:172.21.15.28:6837/2963041370] up:boot 2024-04-23T14:41:11.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:10 smithi028 ceph-mon[27810]: fsmap 4 up:standby 2024-04-23T14:41:11.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:10 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-23T14:41:11.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:10 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/4213380471' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:12.118 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:41:12.122 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 23 14:41:11 smithi038 systemd[1]: Starting Ceph mds.e for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T14:41:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.325+0000 7f28c4a00640 1 -- 172.21.15.28:0/1194487531 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28c00749e0 msgr2=0x7f28c0074e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.325+0000 7f28c4a00640 1 --2- 172.21.15.28:0/1194487531 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28c00749e0 0x7f28c0074e40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f28a8009900 tx=0x7f28a802a480 comp rx=0 tx=0).stop 2024-04-23T14:41:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.325+0000 7f28c4a00640 1 -- 172.21.15.28:0/1194487531 shutdown_connections 2024-04-23T14:41:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.325+0000 7f28c4a00640 1 --2- 172.21.15.28:0/1194487531 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28c00749e0 0x7f28c0074e40 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.325+0000 7f28c4a00640 1 --2- 172.21.15.28:0/1194487531 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f28c0109d50 0x7f28c010c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.325+0000 7f28c4a00640 1 --2- 172.21.15.28:0/1194487531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28c0075fc0 0x7f28c0074410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.325+0000 7f28c4a00640 1 -- 172.21.15.28:0/1194487531 >> 172.21.15.28:0/1194487531 conn(0x7f28c00fd410 msgr2=0x7f28c00ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.325+0000 7f28c4a00640 1 -- 172.21.15.28:0/1194487531 shutdown_connections 2024-04-23T14:41:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.326+0000 7f28c4a00640 1 -- 172.21.15.28:0/1194487531 wait complete. 2024-04-23T14:41:12.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.326+0000 7f28c4a00640 1 Processor -- start 2024-04-23T14:41:12.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.327+0000 7f28c4a00640 1 -- start start 2024-04-23T14:41:12.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.327+0000 7f28c4a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f28c00749e0 0x7f28c019a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:12.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.327+0000 7f28c4a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28c0075fc0 0x7f28c019a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:12.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.327+0000 7f28c4a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28c0109d50 0x7f28c01a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:12.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.327+0000 7f28c4a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f28c010fd60 con 0x7f28c0109d50 2024-04-23T14:41:12.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.327+0000 7f28c4a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f28c010fbe0 con 0x7f28c0075fc0 2024-04-23T14:41:12.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.327+0000 7f28c4a00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f28c010fee0 con 0x7f28c00749e0 2024-04-23T14:41:12.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.327+0000 7f28b7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28c0075fc0 0x7f28c019a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:12.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.327+0000 7f28bf200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28c0109d50 0x7f28c01a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:12.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.327+0000 7f28b7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28c0075fc0 0x7f28c019a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:51168/0 (socket says 172.21.15.28:51168) 2024-04-23T14:41:12.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.327+0000 7f28b7e00640 1 -- 172.21.15.28:0/3022682453 learned_addr learned my addr 172.21.15.28:0/3022682453 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:12.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.327+0000 7f28be800640 1 --2- 172.21.15.28:0/3022682453 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f28c00749e0 0x7f28c019a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:12.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.328+0000 7f28b7e00640 1 -- 172.21.15.28:0/3022682453 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f28c00749e0 msgr2=0x7f28c019a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:12.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.328+0000 7f28b7e00640 1 --2- 172.21.15.28:0/3022682453 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f28c00749e0 0x7f28c019a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:12.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.328+0000 7f28b7e00640 1 -- 172.21.15.28:0/3022682453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28c0109d50 msgr2=0x7f28c01a1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:12.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.328+0000 7f28b7e00640 1 --2- 172.21.15.28:0/3022682453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28c0109d50 0x7f28c01a1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:12.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.328+0000 7f28b7e00640 1 -- 172.21.15.28:0/3022682453 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28c01a1fa0 con 0x7f28c0075fc0 2024-04-23T14:41:12.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.328+0000 7f28bf200640 1 --2- 172.21.15.28:0/3022682453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28c0109d50 0x7f28c01a1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:41:12.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.328+0000 7f28b7e00640 1 --2- 172.21.15.28:0/3022682453 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28c0075fc0 0x7f28c019a910 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f28a8031440 tx=0x7f28a8032ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:12.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.328+0000 7f28b7400640 1 -- 172.21.15.28:0/3022682453 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a803f030 con 0x7f28c0075fc0 2024-04-23T14:41:12.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.328+0000 7f28c4a00640 1 -- 172.21.15.28:0/3022682453 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28c01a2230 con 0x7f28c0075fc0 2024-04-23T14:41:12.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.328+0000 7f28b7400640 1 -- 172.21.15.28:0/3022682453 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28a8003ce0 con 0x7f28c0075fc0 2024-04-23T14:41:12.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.328+0000 7f28b7400640 1 -- 172.21.15.28:0/3022682453 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a8035070 con 0x7f28c0075fc0 2024-04-23T14:41:12.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.329+0000 7f28c4a00640 1 -- 172.21.15.28:0/3022682453 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28c01a2720 con 0x7f28c0075fc0 2024-04-23T14:41:12.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.330+0000 7f28c4a00640 1 -- 172.21.15.28:0/3022682453 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28c0108b10 con 0x7f28c0075fc0 2024-04-23T14:41:12.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.332+0000 7f28b7400640 1 -- 172.21.15.28:0/3022682453 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f28a8042030 con 0x7f28c0075fc0 2024-04-23T14:41:12.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.333+0000 7f28b7400640 1 --2- 172.21.15.28:0/3022682453 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f28a40a3bb0 0x7f28a40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:12.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.333+0000 7f28b7400640 1 -- 172.21.15.28:0/3022682453 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f28a80f0500 con 0x7f28c0075fc0 2024-04-23T14:41:12.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.333+0000 7f28be800640 1 --2- 172.21.15.28:0/3022682453 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f28a40a3bb0 0x7f28a40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:12.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.334+0000 7f28be800640 1 --2- 172.21.15.28:0/3022682453 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f28a40a3bb0 0x7f28a40a6070 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f28b00095f0 tx=0x7f28b0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:12.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.337+0000 7f28b7400640 1 -- 172.21.15.28:0/3022682453 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f28a80bcc90 con 0x7f28c0075fc0 2024-04-23T14:41:12.473 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 23 14:41:12 smithi038 podman[62383]: 2024-04-23 14:41:12.119711005 +0000 UTC m=+0.019228426 image pull 76e77f280c74312ca1add9b6019a4a921a516f91795d85bae00a0c8ca9b1c62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd 2024-04-23T14:41:12.473 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 23 14:41:12 smithi038 podman[62383]: 2024-04-23 14:41:12.236949024 +0000 UTC m=+0.136466434 container create e2530c3f5fe27a6fa2349fdb565134628e1e28c85c6c6e7c3ef2cb767fad4032 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-e, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-23T14:41:12.473 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 23 14:41:12 smithi038 podman[62383]: 2024-04-23 14:41:12.472814732 +0000 UTC m=+0.372332140 container init e2530c3f5fe27a6fa2349fdb565134628e1e28c85c6c6e7c3ef2cb767fad4032 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-e, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416) 2024-04-23T14:41:12.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.585+0000 7f28c4a00640 1 -- 172.21.15.28:0/3022682453 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28c0062c00 con 0x7f28c0075fc0 2024-04-23T14:41:12.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.586+0000 7f28b7400640 1 -- 172.21.15.28:0/3022682453 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3865 (secure 0 0 0) 0x7f28a80c0f90 con 0x7f28c0075fc0 2024-04-23T14:41:12.587 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:41:12.587 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-23T14:41:12.587 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 5 2024-04-23T14:41:12.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.589+0000 7f28c4a00640 1 -- 172.21.15.28:0/3022682453 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f28a40a3bb0 msgr2=0x7f28a40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:12.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.589+0000 7f28c4a00640 1 --2- 172.21.15.28:0/3022682453 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f28a40a3bb0 0x7f28a40a6070 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f28b00095f0 tx=0x7f28b0005f90 comp rx=0 tx=0).stop 2024-04-23T14:41:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.589+0000 7f28c4a00640 1 -- 172.21.15.28:0/3022682453 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28c0075fc0 msgr2=0x7f28c019a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.589+0000 7f28c4a00640 1 --2- 172.21.15.28:0/3022682453 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28c0075fc0 0x7f28c019a910 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f28a8031440 tx=0x7f28a8032ce0 comp rx=0 tx=0).stop 2024-04-23T14:41:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.589+0000 7f28c4a00640 1 -- 172.21.15.28:0/3022682453 shutdown_connections 2024-04-23T14:41:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.589+0000 7f28c4a00640 1 --2- 172.21.15.28:0/3022682453 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f28a40a3bb0 0x7f28a40a6070 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.589+0000 7f28c4a00640 1 --2- 172.21.15.28:0/3022682453 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28c0075fc0 0x7f28c019a910 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.589+0000 7f28c4a00640 1 --2- 172.21.15.28:0/3022682453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28c0109d50 0x7f28c01a1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.589+0000 7f28c4a00640 1 --2- 172.21.15.28:0/3022682453 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f28c00749e0 0x7f28c019a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.589+0000 7f28c4a00640 1 -- 172.21.15.28:0/3022682453 >> 172.21.15.28:0/3022682453 conn(0x7f28c00fd410 msgr2=0x7f28c0105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.589+0000 7f28c4a00640 1 -- 172.21.15.28:0/3022682453 shutdown_connections 2024-04-23T14:41:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:12.589+0000 7f28c4a00640 1 -- 172.21.15.28:0/3022682453 wait complete. 2024-04-23T14:41:12.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:12 smithi038 ceph-mon[32348]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:12.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:12 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3022682453' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:12.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:12 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:12.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:12 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:12.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:12 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:12.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:12 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:41:12.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:12 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:41:12.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:12 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:12.873 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 23 14:41:12 smithi038 podman[62383]: 2024-04-23 14:41:12.475833531 +0000 UTC m=+0.375350939 container start e2530c3f5fe27a6fa2349fdb565134628e1e28c85c6c6e7c3ef2cb767fad4032 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-e, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T14:41:12.874 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 23 14:41:12 smithi038 ceph-mds[62402]: -- 172.21.15.38:0/3986686336 <== mon.2 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cc1233da40 con 0x55cc12310800 2024-04-23T14:41:12.874 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 23 14:41:12 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-e[62398]: starting mds.e at 2024-04-23T14:41:12.874 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 23 14:41:12 smithi038 bash[62383]: e2530c3f5fe27a6fa2349fdb565134628e1e28c85c6c6e7c3ef2cb767fad4032 2024-04-23T14:41:12.874 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 23 14:41:12 smithi038 systemd[1]: Started Ceph mds.e for 1ee6e996-017e-11ef-bc93-c7b262605968. 2024-04-23T14:41:12.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:12 smithi028 ceph-mon[27810]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:12.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:12 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3022682453' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:12.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:12 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:12.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:12 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:12.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:12 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:12.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:12 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:41:12.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:12 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:41:12.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:12 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:13.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:12 smithi105 ceph-mon[32555]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:13.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:12 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3022682453' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:13.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:12 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:13.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:12 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:13.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:12 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:13.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:12 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:41:13.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:12 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:41:13.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:12 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:13.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:13 smithi038 ceph-mon[32348]: Deploying daemon mds.f on smithi105 2024-04-23T14:41:13.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:13 smithi038 ceph-mon[32348]: mds.? [v2:172.21.15.38:6834/2393540699,v1:172.21.15.38:6835/2393540699] up:boot 2024-04-23T14:41:13.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:13 smithi038 ceph-mon[32348]: fsmap 5 up:standby 2024-04-23T14:41:13.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:13 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T14:41:14.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:13 smithi105 ceph-mon[32555]: Deploying daemon mds.f on smithi105 2024-04-23T14:41:14.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:13 smithi105 ceph-mon[32555]: mds.? [v2:172.21.15.38:6834/2393540699,v1:172.21.15.38:6835/2393540699] up:boot 2024-04-23T14:41:14.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:13 smithi105 ceph-mon[32555]: fsmap 5 up:standby 2024-04-23T14:41:14.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:13 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T14:41:14.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:13 smithi028 ceph-mon[27810]: Deploying daemon mds.f on smithi105 2024-04-23T14:41:14.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:13 smithi028 ceph-mon[27810]: mds.? [v2:172.21.15.38:6834/2393540699,v1:172.21.15.38:6835/2393540699] up:boot 2024-04-23T14:41:14.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:13 smithi028 ceph-mon[27810]: fsmap 5 up:standby 2024-04-23T14:41:14.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:13 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T14:41:14.617 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:41:14.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.857+0000 7f33fde00640 1 -- 172.21.15.28:0/3781813994 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33f8103c00 msgr2=0x7f33f810a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:14.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.857+0000 7f33fde00640 1 --2- 172.21.15.28:0/3781813994 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33f8103c00 0x7f33f810a490 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f33f0009ca0 tx=0x7f33f002a8c0 comp rx=0 tx=0).stop 2024-04-23T14:41:14.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.857+0000 7f33fde00640 1 -- 172.21.15.28:0/3781813994 shutdown_connections 2024-04-23T14:41:14.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.857+0000 7f33fde00640 1 --2- 172.21.15.28:0/3781813994 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33f8103c00 0x7f33f810a490 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:14.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.857+0000 7f33fde00640 1 --2- 172.21.15.28:0/3781813994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33f8103240 0x7f33f81036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:14.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.857+0000 7f33fde00640 1 --2- 172.21.15.28:0/3781813994 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f33f8102050 0x7f33f8102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:14.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.857+0000 7f33fde00640 1 -- 172.21.15.28:0/3781813994 >> 172.21.15.28:0/3781813994 conn(0x7f33f80fd5e0 msgr2=0x7f33f80ffa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:14.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.858+0000 7f33fde00640 1 -- 172.21.15.28:0/3781813994 shutdown_connections 2024-04-23T14:41:14.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.858+0000 7f33fde00640 1 -- 172.21.15.28:0/3781813994 wait complete. 2024-04-23T14:41:14.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.859+0000 7f33fde00640 1 Processor -- start 2024-04-23T14:41:14.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.859+0000 7f33fde00640 1 -- start start 2024-04-23T14:41:14.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.859+0000 7f33fde00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33f8102050 0x7f33f819a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:14.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.859+0000 7f33fde00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f33f8103240 0x7f33f819aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:14.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.859+0000 7f33fde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33f8103c00 0x7f33f81a1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:14.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.859+0000 7f33fde00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f33f810dd30 con 0x7f33f8103c00 2024-04-23T14:41:14.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.859+0000 7f33fde00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f33f810dbb0 con 0x7f33f8103240 2024-04-23T14:41:14.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.859+0000 7f33fde00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f33f810deb0 con 0x7f33f8102050 2024-04-23T14:41:14.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.859+0000 7f33f6a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33f8102050 0x7f33f819a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:14.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.859+0000 7f33f6a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33f8102050 0x7f33f819a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:42758/0 (socket says 172.21.15.28:42758) 2024-04-23T14:41:14.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.859+0000 7f33f7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33f8103c00 0x7f33f81a1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:14.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.859+0000 7f33f6a00640 1 -- 172.21.15.28:0/2988782255 learned_addr learned my addr 172.21.15.28:0/2988782255 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:14.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.859+0000 7f33f6000640 1 --2- 172.21.15.28:0/2988782255 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f33f8103240 0x7f33f819aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:14.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.860+0000 7f33f6a00640 1 -- 172.21.15.28:0/2988782255 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f33f8103240 msgr2=0x7f33f819aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:14.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.860+0000 7f33f6a00640 1 --2- 172.21.15.28:0/2988782255 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f33f8103240 0x7f33f819aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:14.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.860+0000 7f33f6a00640 1 -- 172.21.15.28:0/2988782255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33f8103c00 msgr2=0x7f33f81a1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:14.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.860+0000 7f33f6a00640 1 --2- 172.21.15.28:0/2988782255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33f8103c00 0x7f33f81a1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:14.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.860+0000 7f33f6a00640 1 -- 172.21.15.28:0/2988782255 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33f81a2170 con 0x7f33f8102050 2024-04-23T14:41:14.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.860+0000 7f33f6a00640 1 --2- 172.21.15.28:0/2988782255 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33f8102050 0x7f33f819a550 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f33e800e9c0 tx=0x7f33e800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:14.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.860+0000 7f33fca00640 1 -- 172.21.15.28:0/2988782255 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e800cd80 con 0x7f33f8102050 2024-04-23T14:41:14.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.860+0000 7f33fca00640 1 -- 172.21.15.28:0/2988782255 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33e80035f0 con 0x7f33f8102050 2024-04-23T14:41:14.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.860+0000 7f33fca00640 1 -- 172.21.15.28:0/2988782255 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e8010690 con 0x7f33f8102050 2024-04-23T14:41:14.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.860+0000 7f33fde00640 1 -- 172.21.15.28:0/2988782255 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33f81a2460 con 0x7f33f8102050 2024-04-23T14:41:14.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.861+0000 7f33fde00640 1 -- 172.21.15.28:0/2988782255 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33f81a2960 con 0x7f33f8102050 2024-04-23T14:41:14.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.862+0000 7f33fde00640 1 -- 172.21.15.28:0/2988782255 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33f81a2c20 con 0x7f33f8102050 2024-04-23T14:41:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.864+0000 7f33fca00640 1 -- 172.21.15.28:0/2988782255 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f33e80037a0 con 0x7f33f8102050 2024-04-23T14:41:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.865+0000 7f33fca00640 1 --2- 172.21.15.28:0/2988782255 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f33dc0a3bb0 0x7f33dc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.865+0000 7f33fca00640 1 -- 172.21.15.28:0/2988782255 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f33e80d0410 con 0x7f33f8102050 2024-04-23T14:41:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.865+0000 7f33f6000640 1 --2- 172.21.15.28:0/2988782255 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f33dc0a3bb0 0x7f33dc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:14.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.866+0000 7f33f6000640 1 --2- 172.21.15.28:0/2988782255 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f33dc0a3bb0 0x7f33dc0a6070 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f33f819ba60 tx=0x7f33e4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:14.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:14.870+0000 7f33fca00640 1 -- 172.21.15.28:0/2988782255 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f33e809d1f0 con 0x7f33f8102050 2024-04-23T14:41:15.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:14 smithi105 ceph-mon[32555]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:15.037 INFO:journalctl@ceph.mds.f.smithi105.stdout:Apr 23 14:41:14 smithi105 systemd[1]: Starting Ceph mds.f for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T14:41:15.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:15.116+0000 7f33fde00640 1 -- 172.21.15.28:0/2988782255 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f33f81a2df0 con 0x7f33f8102050 2024-04-23T14:41:15.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:15.117+0000 7f33fca00640 1 -- 172.21.15.28:0/2988782255 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4690 (secure 0 0 0) 0x7f33e809de20 con 0x7f33f8102050 2024-04-23T14:41:15.119 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:41:15.119 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-23T14:41:15.119 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 6 2024-04-23T14:41:15.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:15.120+0000 7f33fde00640 1 -- 172.21.15.28:0/2988782255 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f33dc0a3bb0 msgr2=0x7f33dc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:15.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:15.120+0000 7f33fde00640 1 --2- 172.21.15.28:0/2988782255 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f33dc0a3bb0 0x7f33dc0a6070 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f33f819ba60 tx=0x7f33e4005f90 comp rx=0 tx=0).stop 2024-04-23T14:41:15.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:15.120+0000 7f33fde00640 1 -- 172.21.15.28:0/2988782255 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33f8102050 msgr2=0x7f33f819a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:15.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:15.120+0000 7f33fde00640 1 --2- 172.21.15.28:0/2988782255 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33f8102050 0x7f33f819a550 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f33e800e9c0 tx=0x7f33e800ee80 comp rx=0 tx=0).stop 2024-04-23T14:41:15.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:15.120+0000 7f33fde00640 1 -- 172.21.15.28:0/2988782255 shutdown_connections 2024-04-23T14:41:15.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:15.120+0000 7f33fde00640 1 --2- 172.21.15.28:0/2988782255 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f33dc0a3bb0 0x7f33dc0a6070 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:15.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:15.120+0000 7f33fde00640 1 --2- 172.21.15.28:0/2988782255 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f33f8103240 0x7f33f819aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:15.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:15.120+0000 7f33fde00640 1 --2- 172.21.15.28:0/2988782255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33f8103c00 0x7f33f81a1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:15.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:15.120+0000 7f33fde00640 1 --2- 172.21.15.28:0/2988782255 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33f8102050 0x7f33f819a550 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:15.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:15.120+0000 7f33fde00640 1 -- 172.21.15.28:0/2988782255 >> 172.21.15.28:0/2988782255 conn(0x7f33f80fd5e0 msgr2=0x7f33f8074230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:15.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:15.121+0000 7f33fde00640 1 -- 172.21.15.28:0/2988782255 shutdown_connections 2024-04-23T14:41:15.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:15.121+0000 7f33fde00640 1 -- 172.21.15.28:0/2988782255 wait complete. 2024-04-23T14:41:15.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:14 smithi038 ceph-mon[32348]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:15.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:14 smithi028 ceph-mon[27810]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:15.369 INFO:journalctl@ceph.mds.f.smithi105.stdout:Apr 23 14:41:15 smithi105 podman[63034]: 2024-04-23 14:41:14.966435379 +0000 UTC m=+0.017588149 image pull 76e77f280c74312ca1add9b6019a4a921a516f91795d85bae00a0c8ca9b1c62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd 2024-04-23T14:41:15.369 INFO:journalctl@ceph.mds.f.smithi105.stdout:Apr 23 14:41:15 smithi105 podman[63034]: 2024-04-23 14:41:15.066523364 +0000 UTC m=+0.117676134 container create 75e2c6354570a72ab416acf11d511b0c2ee8b6c47d0d094a4fd75b82a137f82f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-f, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-23T14:41:15.369 INFO:journalctl@ceph.mds.f.smithi105.stdout:Apr 23 14:41:15 smithi105 podman[63034]: 2024-04-23 14:41:15.268708345 +0000 UTC m=+0.319861116 container init 75e2c6354570a72ab416acf11d511b0c2ee8b6c47d0d094a4fd75b82a137f82f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-f, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-23T14:41:15.369 INFO:journalctl@ceph.mds.f.smithi105.stdout:Apr 23 14:41:15 smithi105 podman[63034]: 2024-04-23 14:41:15.271184351 +0000 UTC m=+0.322337130 container start 75e2c6354570a72ab416acf11d511b0c2ee8b6c47d0d094a4fd75b82a137f82f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-f, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T14:41:15.369 INFO:journalctl@ceph.mds.f.smithi105.stdout:Apr 23 14:41:15 smithi105 ceph-mds[63053]: -- 172.21.15.105:0/1255600377 <== mon.1 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5590e1c15a40 con 0x5590e1be8c00 2024-04-23T14:41:15.369 INFO:journalctl@ceph.mds.f.smithi105.stdout:Apr 23 14:41:15 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-f[63049]: starting mds.f at 2024-04-23T14:41:15.369 INFO:journalctl@ceph.mds.f.smithi105.stdout:Apr 23 14:41:15 smithi105 bash[63034]: 75e2c6354570a72ab416acf11d511b0c2ee8b6c47d0d094a4fd75b82a137f82f 2024-04-23T14:41:15.724 INFO:journalctl@ceph.mds.f.smithi105.stdout:Apr 23 14:41:15 smithi105 systemd[1]: Started Ceph mds.f for 1ee6e996-017e-11ef-bc93-c7b262605968. 2024-04-23T14:41:16.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:15 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2988782255' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:16.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:15 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:16.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:15 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:16.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:15 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:16.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:15 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:41:16.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:15 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:41:16.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:15 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:16.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:15 smithi105 ceph-mon[32555]: Deploying daemon mds.c on smithi105 2024-04-23T14:41:16.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:15 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2988782255' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:16.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:15 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:16.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:15 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:16.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:15 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:16.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:15 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:41:16.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:15 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:41:16.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:15 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:16.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:15 smithi038 ceph-mon[32348]: Deploying daemon mds.c on smithi105 2024-04-23T14:41:16.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:15 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2988782255' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:16.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:15 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:16.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:15 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:16.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:15 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:16.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:15 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:41:16.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:15 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:41:16.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:15 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:16.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:15 smithi028 ceph-mon[27810]: Deploying daemon mds.c on smithi105 2024-04-23T14:41:17.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:16 smithi105 ceph-mon[32555]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:17.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:16 smithi105 ceph-mon[32555]: mds.? [v2:172.21.15.105:6834/1788458050,v1:172.21.15.105:6835/1788458050] up:boot 2024-04-23T14:41:17.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:16 smithi105 ceph-mon[32555]: fsmap 6 up:standby 2024-04-23T14:41:17.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:16 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T14:41:17.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:16 smithi038 ceph-mon[32348]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:17.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:16 smithi038 ceph-mon[32348]: mds.? [v2:172.21.15.105:6834/1788458050,v1:172.21.15.105:6835/1788458050] up:boot 2024-04-23T14:41:17.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:16 smithi038 ceph-mon[32348]: fsmap 6 up:standby 2024-04-23T14:41:17.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:16 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T14:41:17.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:16 smithi028 ceph-mon[27810]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:17.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:16 smithi028 ceph-mon[27810]: mds.? [v2:172.21.15.105:6834/1788458050,v1:172.21.15.105:6835/1788458050] up:boot 2024-04-23T14:41:17.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:16 smithi028 ceph-mon[27810]: fsmap 6 up:standby 2024-04-23T14:41:17.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:16 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T14:41:17.143 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:41:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.381+0000 7fef5c400640 1 -- 172.21.15.28:0/27758937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef540785a0 msgr2=0x7fef54078e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.381+0000 7fef5c400640 1 --2- 172.21.15.28:0/27758937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef540785a0 0x7fef54078e40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fef4c009860 tx=0x7fef4c02a340 comp rx=0 tx=0).stop 2024-04-23T14:41:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.381+0000 7fef5c400640 1 -- 172.21.15.28:0/27758937 shutdown_connections 2024-04-23T14:41:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.381+0000 7fef5c400640 1 --2- 172.21.15.28:0/27758937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef540785a0 0x7fef54078e40 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.381+0000 7fef5c400640 1 --2- 172.21.15.28:0/27758937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef54077c00 0x7fef54078060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.381+0000 7fef5c400640 1 --2- 172.21.15.28:0/27758937 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fef540769c0 0x7fef54076de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.381+0000 7fef5c400640 1 -- 172.21.15.28:0/27758937 >> 172.21.15.28:0/27758937 conn(0x7fef540ff7f0 msgr2=0x7fef54101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.381+0000 7fef5c400640 1 -- 172.21.15.28:0/27758937 shutdown_connections 2024-04-23T14:41:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.381+0000 7fef5c400640 1 -- 172.21.15.28:0/27758937 wait complete. 2024-04-23T14:41:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.382+0000 7fef5c400640 1 Processor -- start 2024-04-23T14:41:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.382+0000 7fef5c400640 1 -- start start 2024-04-23T14:41:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.382+0000 7fef5c400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef540769c0 0x7fef5419e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.382+0000 7fef5c400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef54077c00 0x7fef5419ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.382+0000 7fef5c400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fef540785a0 0x7fef541a5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:17.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.382+0000 7fef5c400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fef54114310 con 0x7fef54077c00 2024-04-23T14:41:17.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.382+0000 7fef5c400640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fef54114190 con 0x7fef540769c0 2024-04-23T14:41:17.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.382+0000 7fef5c400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fef54114490 con 0x7fef540785a0 2024-04-23T14:41:17.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.383+0000 7fef59000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef54077c00 0x7fef5419ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:17.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.383+0000 7fef59a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef540769c0 0x7fef5419e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:17.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.383+0000 7fef59a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef540769c0 0x7fef5419e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:51208/0 (socket says 172.21.15.28:51208) 2024-04-23T14:41:17.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.383+0000 7fef59000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef54077c00 0x7fef5419ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45470/0 (socket says 172.21.15.28:45470) 2024-04-23T14:41:17.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.383+0000 7fef59a00640 1 -- 172.21.15.28:0/4251208898 learned_addr learned my addr 172.21.15.28:0/4251208898 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:17.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.383+0000 7fef5a400640 1 --2- 172.21.15.28:0/4251208898 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fef540785a0 0x7fef541a5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:17.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.383+0000 7fef59a00640 1 -- 172.21.15.28:0/4251208898 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fef540785a0 msgr2=0x7fef541a5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:17.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.383+0000 7fef59a00640 1 --2- 172.21.15.28:0/4251208898 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fef540785a0 0x7fef541a5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:17.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.383+0000 7fef59a00640 1 -- 172.21.15.28:0/4251208898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef54077c00 msgr2=0x7fef5419ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:17.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.383+0000 7fef59a00640 1 --2- 172.21.15.28:0/4251208898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef54077c00 0x7fef5419ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:17.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.383+0000 7fef59a00640 1 -- 172.21.15.28:0/4251208898 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef541a6510 con 0x7fef540769c0 2024-04-23T14:41:17.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.383+0000 7fef59000640 1 --2- 172.21.15.28:0/4251208898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef54077c00 0x7fef5419ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:41:17.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.384+0000 7fef59a00640 1 --2- 172.21.15.28:0/4251208898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef540769c0 0x7fef5419e8f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fef4400e9c0 tx=0x7fef4400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:17.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.384+0000 7fef52000640 1 -- 172.21.15.28:0/4251208898 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4400ccc0 con 0x7fef540769c0 2024-04-23T14:41:17.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.384+0000 7fef5c400640 1 -- 172.21.15.28:0/4251208898 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef541a6800 con 0x7fef540769c0 2024-04-23T14:41:17.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.384+0000 7fef52000640 1 -- 172.21.15.28:0/4251208898 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef44005500 con 0x7fef540769c0 2024-04-23T14:41:17.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.384+0000 7fef52000640 1 -- 172.21.15.28:0/4251208898 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef44010430 con 0x7fef540769c0 2024-04-23T14:41:17.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.384+0000 7fef5c400640 1 -- 172.21.15.28:0/4251208898 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef541a6d50 con 0x7fef540769c0 2024-04-23T14:41:17.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.385+0000 7fef5c400640 1 -- 172.21.15.28:0/4251208898 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef54110d40 con 0x7fef540769c0 2024-04-23T14:41:17.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.387+0000 7fef52000640 1 -- 172.21.15.28:0/4251208898 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fef440046c0 con 0x7fef540769c0 2024-04-23T14:41:17.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.389+0000 7fef52000640 1 --2- 172.21.15.28:0/4251208898 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fef380a3bb0 0x7fef380a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:17.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.389+0000 7fef52000640 1 -- 172.21.15.28:0/4251208898 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fef440d0280 con 0x7fef540769c0 2024-04-23T14:41:17.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.389+0000 7fef59000640 1 --2- 172.21.15.28:0/4251208898 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fef380a3bb0 0x7fef380a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:17.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.390+0000 7fef59000640 1 --2- 172.21.15.28:0/4251208898 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fef380a3bb0 0x7fef380a6070 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fef5419fe00 tx=0x7fef40009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:17.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.391+0000 7fef52000640 1 -- 172.21.15.28:0/4251208898 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fef4409d060 con 0x7fef540769c0 2024-04-23T14:41:17.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.647+0000 7fef5c400640 1 -- 172.21.15.28:0/4251208898 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef5402d620 con 0x7fef540769c0 2024-04-23T14:41:17.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.648+0000 7fef52000640 1 -- 172.21.15.28:0/4251208898 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5518 (secure 0 0 0) 0x7fef4409c960 con 0x7fef540769c0 2024-04-23T14:41:17.649 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:41:17.650 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-23T14:41:17.650 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 7 2024-04-23T14:41:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.650+0000 7fef5c400640 1 -- 172.21.15.28:0/4251208898 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fef380a3bb0 msgr2=0x7fef380a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.650+0000 7fef5c400640 1 --2- 172.21.15.28:0/4251208898 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fef380a3bb0 0x7fef380a6070 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fef5419fe00 tx=0x7fef40009500 comp rx=0 tx=0).stop 2024-04-23T14:41:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.650+0000 7fef5c400640 1 -- 172.21.15.28:0/4251208898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef540769c0 msgr2=0x7fef5419e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.650+0000 7fef5c400640 1 --2- 172.21.15.28:0/4251208898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef540769c0 0x7fef5419e8f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fef4400e9c0 tx=0x7fef4400ee80 comp rx=0 tx=0).stop 2024-04-23T14:41:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.650+0000 7fef5c400640 1 -- 172.21.15.28:0/4251208898 shutdown_connections 2024-04-23T14:41:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.650+0000 7fef5c400640 1 --2- 172.21.15.28:0/4251208898 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fef540785a0 0x7fef541a5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.650+0000 7fef5c400640 1 --2- 172.21.15.28:0/4251208898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef54077c00 0x7fef5419ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.650+0000 7fef5c400640 1 --2- 172.21.15.28:0/4251208898 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fef380a3bb0 0x7fef380a6070 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.650+0000 7fef5c400640 1 --2- 172.21.15.28:0/4251208898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef540769c0 0x7fef5419e8f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.650+0000 7fef5c400640 1 -- 172.21.15.28:0/4251208898 >> 172.21.15.28:0/4251208898 conn(0x7fef540ff7f0 msgr2=0x7fef54100e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.651+0000 7fef5c400640 1 -- 172.21.15.28:0/4251208898 shutdown_connections 2024-04-23T14:41:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:17.651+0000 7fef5c400640 1 -- 172.21.15.28:0/4251208898 wait complete. 2024-04-23T14:41:17.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:17 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/4251208898' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:18.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:17 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/4251208898' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:18.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:17 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/4251208898' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:18.536 INFO:journalctl@ceph.mds.c.smithi105.stdout:Apr 23 14:41:18 smithi105 systemd[1]: Started Ceph mds.c for 1ee6e996-017e-11ef-bc93-c7b262605968. 2024-04-23T14:41:19.452 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:19 smithi038 ceph-mon[32348]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:19.453 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:19 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:19.453 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:19 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:19.453 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:19 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:19.453 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:19 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:41:19.453 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:19 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:41:19.453 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:19 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:19.453 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:19 smithi038 ceph-mon[32348]: Deploying daemon mds.h on smithi038 2024-04-23T14:41:19.453 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:19 smithi038 ceph-mon[32348]: mds.? [v2:172.21.15.105:6836/1691671537,v1:172.21.15.105:6837/1691671537] up:boot 2024-04-23T14:41:19.453 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:19 smithi038 ceph-mon[32348]: fsmap 7 up:standby 2024-04-23T14:41:19.453 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:19 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T14:41:19.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:19 smithi028 ceph-mon[27810]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:19.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:19 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:19.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:19 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:19.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:19 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:19.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:19 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:41:19.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:19 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:41:19.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:19 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:19.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:19 smithi028 ceph-mon[27810]: Deploying daemon mds.h on smithi038 2024-04-23T14:41:19.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:19 smithi028 ceph-mon[27810]: mds.? [v2:172.21.15.105:6836/1691671537,v1:172.21.15.105:6837/1691671537] up:boot 2024-04-23T14:41:19.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:19 smithi028 ceph-mon[27810]: fsmap 7 up:standby 2024-04-23T14:41:19.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:19 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T14:41:19.673 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:41:19.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:19 smithi105 ceph-mon[32555]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:19.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:19 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:19.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:19 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:19.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:19 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:19.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:19 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:41:19.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:19 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:41:19.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:19 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:19.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:19 smithi105 ceph-mon[32555]: Deploying daemon mds.h on smithi038 2024-04-23T14:41:19.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:19 smithi105 ceph-mon[32555]: mds.? [v2:172.21.15.105:6836/1691671537,v1:172.21.15.105:6837/1691671537] up:boot 2024-04-23T14:41:19.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:19 smithi105 ceph-mon[32555]: fsmap 7 up:standby 2024-04-23T14:41:19.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:19 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T14:41:19.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.894+0000 7fbae0c00640 1 -- 172.21.15.28:0/2768182645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbadc103ae0 msgr2=0x7fbadc10a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:19.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.894+0000 7fbae0c00640 1 --2- 172.21.15.28:0/2768182645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbadc103ae0 0x7fbadc10a370 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbad4009ca0 tx=0x7fbad402a8c0 comp rx=0 tx=0).stop 2024-04-23T14:41:19.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.894+0000 7fbae0c00640 1 -- 172.21.15.28:0/2768182645 shutdown_connections 2024-04-23T14:41:19.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.894+0000 7fbae0c00640 1 --2- 172.21.15.28:0/2768182645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbadc103ae0 0x7fbadc10a370 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:19.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.894+0000 7fbae0c00640 1 --2- 172.21.15.28:0/2768182645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbadc103120 0x7fbadc1035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:19.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.894+0000 7fbae0c00640 1 --2- 172.21.15.28:0/2768182645 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbadc100150 0x7fbadc100570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:19.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.894+0000 7fbae0c00640 1 -- 172.21.15.28:0/2768182645 >> 172.21.15.28:0/2768182645 conn(0x7fbadc0fba80 msgr2=0x7fbadc0fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:19.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.894+0000 7fbae0c00640 1 -- 172.21.15.28:0/2768182645 shutdown_connections 2024-04-23T14:41:19.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.894+0000 7fbae0c00640 1 -- 172.21.15.28:0/2768182645 wait complete. 2024-04-23T14:41:19.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.895+0000 7fbae0c00640 1 Processor -- start 2024-04-23T14:41:19.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.895+0000 7fbae0c00640 1 -- start start 2024-04-23T14:41:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.895+0000 7fbae0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbadc100150 0x7fbadc102a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.895+0000 7fbae0c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbadc103120 0x7fbadc1010d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.895+0000 7fbae0c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbadc103ae0 0x7fbadc101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.896+0000 7fbae0c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbadc10db70 con 0x7fbadc100150 2024-04-23T14:41:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.896+0000 7fbae0c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbadc10d9f0 con 0x7fbadc103ae0 2024-04-23T14:41:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.896+0000 7fbae0c00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fbadc10dcf0 con 0x7fbadc103120 2024-04-23T14:41:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.896+0000 7fbadbe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbadc103ae0 0x7fbadc101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.896+0000 7fbadb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbadc100150 0x7fbadc102a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.896+0000 7fbadbe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbadc103ae0 0x7fbadc101610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:44066/0 (socket says 172.21.15.28:44066) 2024-04-23T14:41:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.896+0000 7fbadbe00640 1 -- 172.21.15.28:0/4231015720 learned_addr learned my addr 172.21.15.28:0/4231015720 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.896+0000 7fbad2a00640 1 --2- 172.21.15.28:0/4231015720 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbadc103120 0x7fbadc1010d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.896+0000 7fbadbe00640 1 -- 172.21.15.28:0/4231015720 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbadc103120 msgr2=0x7fbadc1010d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.896+0000 7fbadbe00640 1 --2- 172.21.15.28:0/4231015720 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbadc103120 0x7fbadc1010d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.896+0000 7fbadbe00640 1 -- 172.21.15.28:0/4231015720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbadc100150 msgr2=0x7fbadc102a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.896+0000 7fbadbe00640 1 --2- 172.21.15.28:0/4231015720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbadc100150 0x7fbadc102a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.896+0000 7fbadbe00640 1 -- 172.21.15.28:0/4231015720 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbadc101ea0 con 0x7fbadc103ae0 2024-04-23T14:41:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.896+0000 7fbadb400640 1 --2- 172.21.15.28:0/4231015720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbadc100150 0x7fbadc102a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:41:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.897+0000 7fbadbe00640 1 --2- 172.21.15.28:0/4231015720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbadc103ae0 0x7fbadc101610 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbad402f540 tx=0x7fbad4031750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.897+0000 7fbad8c00640 1 -- 172.21.15.28:0/4231015720 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad4003b70 con 0x7fbadc103ae0 2024-04-23T14:41:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.897+0000 7fbad8c00640 1 -- 172.21.15.28:0/4231015720 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbad4003d10 con 0x7fbadc103ae0 2024-04-23T14:41:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.897+0000 7fbad8c00640 1 -- 172.21.15.28:0/4231015720 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad402fc00 con 0x7fbadc103ae0 2024-04-23T14:41:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.897+0000 7fbae0c00640 1 -- 172.21.15.28:0/4231015720 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbadc1a2270 con 0x7fbadc103ae0 2024-04-23T14:41:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.897+0000 7fbae0c00640 1 -- 172.21.15.28:0/4231015720 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbadc1a2810 con 0x7fbadc103ae0 2024-04-23T14:41:19.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.898+0000 7fbae0c00640 1 -- 172.21.15.28:0/4231015720 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbadc1091f0 con 0x7fbadc103ae0 2024-04-23T14:41:19.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.900+0000 7fbad8c00640 1 -- 172.21.15.28:0/4231015720 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fbad40046b0 con 0x7fbadc103ae0 2024-04-23T14:41:19.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.902+0000 7fbad8c00640 1 --2- 172.21.15.28:0/4231015720 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fbac00a3c30 0x7fbac00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:19.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.902+0000 7fbad8c00640 1 -- 172.21.15.28:0/4231015720 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fbad40f1500 con 0x7fbadc103ae0 2024-04-23T14:41:19.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.902+0000 7fbadb400640 1 --2- 172.21.15.28:0/4231015720 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fbac00a3c30 0x7fbac00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:19.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.903+0000 7fbadb400640 1 --2- 172.21.15.28:0/4231015720 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fbac00a3c30 0x7fbac00a60f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbadc06a400 tx=0x7fbacc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:19.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:19.906+0000 7fbad8c00640 1 -- 172.21.15.28:0/4231015720 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbad40be2e0 con 0x7fbadc103ae0 2024-04-23T14:41:20.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:20.152+0000 7fbae0c00640 1 -- 172.21.15.28:0/4231015720 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbadc062c00 con 0x7fbadc103ae0 2024-04-23T14:41:20.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:20.153+0000 7fbad8c00640 1 -- 172.21.15.28:0/4231015720 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6346 (secure 0 0 0) 0x7fbad40bdbe0 con 0x7fbadc103ae0 2024-04-23T14:41:20.154 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:41:20.154 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-23T14:41:20.155 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 8 2024-04-23T14:41:20.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:20.155+0000 7fbae0c00640 1 -- 172.21.15.28:0/4231015720 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fbac00a3c30 msgr2=0x7fbac00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:20.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:20.156+0000 7fbae0c00640 1 --2- 172.21.15.28:0/4231015720 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fbac00a3c30 0x7fbac00a60f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbadc06a400 tx=0x7fbacc005f90 comp rx=0 tx=0).stop 2024-04-23T14:41:20.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:20.156+0000 7fbae0c00640 1 -- 172.21.15.28:0/4231015720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbadc103ae0 msgr2=0x7fbadc101610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:20.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:20.156+0000 7fbae0c00640 1 --2- 172.21.15.28:0/4231015720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbadc103ae0 0x7fbadc101610 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbad402f540 tx=0x7fbad4031750 comp rx=0 tx=0).stop 2024-04-23T14:41:20.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:20.156+0000 7fbae0c00640 1 -- 172.21.15.28:0/4231015720 shutdown_connections 2024-04-23T14:41:20.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:20.156+0000 7fbae0c00640 1 --2- 172.21.15.28:0/4231015720 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fbac00a3c30 0x7fbac00a60f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:20.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:20.156+0000 7fbae0c00640 1 --2- 172.21.15.28:0/4231015720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbadc103ae0 0x7fbadc101610 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:20.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:20.156+0000 7fbae0c00640 1 --2- 172.21.15.28:0/4231015720 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbadc103120 0x7fbadc1010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:20.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:20.156+0000 7fbae0c00640 1 --2- 172.21.15.28:0/4231015720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbadc100150 0x7fbadc102a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:20.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:20.156+0000 7fbae0c00640 1 -- 172.21.15.28:0/4231015720 >> 172.21.15.28:0/4231015720 conn(0x7fbadc0fba80 msgr2=0x7fbadc1063c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:20.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:20.156+0000 7fbae0c00640 1 -- 172.21.15.28:0/4231015720 shutdown_connections 2024-04-23T14:41:20.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:20.156+0000 7fbae0c00640 1 -- 172.21.15.28:0/4231015720 wait complete. 2024-04-23T14:41:20.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:20 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/4231015720' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:20.623 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 23 14:41:20 smithi038 systemd[1]: Starting Ceph mds.h for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T14:41:20.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:20 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/4231015720' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:20.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:20 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/4231015720' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:20.953 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 23 14:41:20 smithi038 podman[62693]: 2024-04-23 14:41:20.601272738 +0000 UTC m=+0.018654494 image pull 76e77f280c74312ca1add9b6019a4a921a516f91795d85bae00a0c8ca9b1c62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd 2024-04-23T14:41:20.953 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 23 14:41:20 smithi038 podman[62693]: 2024-04-23 14:41:20.725426982 +0000 UTC m=+0.142808717 container create eea5e724a98ba815a6dbdb256e4546b4a03f65e8382b837e494d15a13aebec35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-h, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-23T14:41:20.953 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 23 14:41:20 smithi038 podman[62693]: 2024-04-23 14:41:20.953125558 +0000 UTC m=+0.370507297 container init eea5e724a98ba815a6dbdb256e4546b4a03f65e8382b837e494d15a13aebec35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0) 2024-04-23T14:41:21.372 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 23 14:41:20 smithi038 podman[62693]: 2024-04-23 14:41:20.955964518 +0000 UTC m=+0.373346263 container start eea5e724a98ba815a6dbdb256e4546b4a03f65e8382b837e494d15a13aebec35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-23T14:41:21.373 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 23 14:41:20 smithi038 ceph-mds[62712]: -- 172.21.15.38:0/4245082067 <== mon.2 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55effafdba40 con 0x55effafae800 2024-04-23T14:41:21.373 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 23 14:41:20 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-h[62708]: starting mds.h at 2024-04-23T14:41:21.373 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 23 14:41:21 smithi038 bash[62693]: eea5e724a98ba815a6dbdb256e4546b4a03f65e8382b837e494d15a13aebec35 2024-04-23T14:41:21.373 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 23 14:41:21 smithi038 systemd[1]: Started Ceph mds.h for 1ee6e996-017e-11ef-bc93-c7b262605968. 2024-04-23T14:41:21.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:21 smithi105 ceph-mon[32555]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:21.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:21 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:21.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:21 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:21.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:21 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:21.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:21 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:41:21.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:21 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:41:21.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:21 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:21.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:21 smithi038 ceph-mon[32348]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:21.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:21 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:21.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:21 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:21.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:21 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:21.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:21 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:41:21.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:21 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:41:21.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:21 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:21.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:21 smithi028 ceph-mon[27810]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:21.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:21 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:21.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:21 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:21.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:21 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:21.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:21 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:41:21.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:21 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:41:21.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:21 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:22.178 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:41:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.382+0000 7f922cc00640 1 -- 172.21.15.28:0/4112694923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9228105be0 msgr2=0x7f9228106060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.382+0000 7f922cc00640 1 --2- 172.21.15.28:0/4112694923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9228105be0 0x7f9228106060 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9220009c90 tx=0x7f922002a860 comp rx=0 tx=0).stop 2024-04-23T14:41:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.383+0000 7f922cc00640 1 -- 172.21.15.28:0/4112694923 shutdown_connections 2024-04-23T14:41:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.383+0000 7f922cc00640 1 --2- 172.21.15.28:0/4112694923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9228105be0 0x7f9228106060 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.383+0000 7f922cc00640 1 --2- 172.21.15.28:0/4112694923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9228103230 0x7f92281056a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.383+0000 7f922cc00640 1 --2- 172.21.15.28:0/4112694923 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92281008c0 0x7f9228102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:22.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.383+0000 7f922cc00640 1 -- 172.21.15.28:0/4112694923 >> 172.21.15.28:0/4112694923 conn(0x7f92280fc4e0 msgr2=0x7f92280fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:22.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.383+0000 7f922cc00640 1 -- 172.21.15.28:0/4112694923 shutdown_connections 2024-04-23T14:41:22.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.383+0000 7f922cc00640 1 -- 172.21.15.28:0/4112694923 wait complete. 2024-04-23T14:41:22.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.384+0000 7f922cc00640 1 Processor -- start 2024-04-23T14:41:22.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.384+0000 7f922cc00640 1 -- start start 2024-04-23T14:41:22.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.384+0000 7f922cc00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92281008c0 0x7f922819e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:22.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.384+0000 7f922cc00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9228103230 0x7f922819ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:22.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.384+0000 7f922cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9228105be0 0x7f92281a5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:22.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.384+0000 7f922cc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f92281142f0 con 0x7f9228105be0 2024-04-23T14:41:22.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.384+0000 7f922cc00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9228114170 con 0x7f92281008c0 2024-04-23T14:41:22.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.384+0000 7f922cc00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9228114470 con 0x7f9228103230 2024-04-23T14:41:22.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.384+0000 7f9225e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92281008c0 0x7f922819e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:22.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.384+0000 7f9225e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92281008c0 0x7f922819e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:44082/0 (socket says 172.21.15.28:44082) 2024-04-23T14:41:22.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.384+0000 7f9225e00640 1 -- 172.21.15.28:0/2253166771 learned_addr learned my addr 172.21.15.28:0/2253166771 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:22.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.384+0000 7f9226800640 1 --2- 172.21.15.28:0/2253166771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9228105be0 0x7f92281a5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:22.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.384+0000 7f921d400640 1 --2- 172.21.15.28:0/2253166771 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9228103230 0x7f922819ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:22.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.385+0000 7f9225e00640 1 -- 172.21.15.28:0/2253166771 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9228103230 msgr2=0x7f922819ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:22.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.385+0000 7f9225e00640 1 --2- 172.21.15.28:0/2253166771 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9228103230 0x7f922819ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:22.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.385+0000 7f9225e00640 1 -- 172.21.15.28:0/2253166771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9228105be0 msgr2=0x7f92281a5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:22.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.385+0000 7f9225e00640 1 --2- 172.21.15.28:0/2253166771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9228105be0 0x7f92281a5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:22.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.385+0000 7f9225e00640 1 -- 172.21.15.28:0/2253166771 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92281a6560 con 0x7f92281008c0 2024-04-23T14:41:22.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.385+0000 7f9225e00640 1 --2- 172.21.15.28:0/2253166771 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92281008c0 0x7f922819e940 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f921800d950 tx=0x7f921800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:22.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.386+0000 7f921f400640 1 -- 172.21.15.28:0/2253166771 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9218017070 con 0x7f92281008c0 2024-04-23T14:41:22.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.386+0000 7f921f400640 1 -- 172.21.15.28:0/2253166771 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9218009c30 con 0x7f92281008c0 2024-04-23T14:41:22.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.386+0000 7f921f400640 1 -- 172.21.15.28:0/2253166771 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f921800f6b0 con 0x7f92281008c0 2024-04-23T14:41:22.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.386+0000 7f922cc00640 1 -- 172.21.15.28:0/2253166771 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92281a6850 con 0x7f92281008c0 2024-04-23T14:41:22.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.386+0000 7f922cc00640 1 -- 172.21.15.28:0/2253166771 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92281a6df0 con 0x7f92281008c0 2024-04-23T14:41:22.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.387+0000 7f922cc00640 1 -- 172.21.15.28:0/2253166771 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92280fdfd0 con 0x7f92281008c0 2024-04-23T14:41:22.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.389+0000 7f921f400640 1 -- 172.21.15.28:0/2253166771 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f92180109d0 con 0x7f92281008c0 2024-04-23T14:41:22.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.390+0000 7f921f400640 1 --2- 172.21.15.28:0/2253166771 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f920c0a3c30 0x7f920c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:22.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.391+0000 7f921f400640 1 -- 172.21.15.28:0/2253166771 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f92180cfaf0 con 0x7f92281008c0 2024-04-23T14:41:22.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.391+0000 7f921d400640 1 --2- 172.21.15.28:0/2253166771 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f920c0a3c30 0x7f920c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:22.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.391+0000 7f921d400640 1 --2- 172.21.15.28:0/2253166771 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f920c0a3c30 0x7f920c0a60f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f922819fe50 tx=0x7f9210005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:22.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.395+0000 7f921f400640 1 -- 172.21.15.28:0/2253166771 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9218009de0 con 0x7f92281008c0 2024-04-23T14:41:22.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:22 smithi028 ceph-mon[27810]: Deploying daemon mds.i on smithi105 2024-04-23T14:41:22.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:22 smithi028 ceph-mon[27810]: mds.? [v2:172.21.15.38:6836/4078619599,v1:172.21.15.38:6837/4078619599] up:boot 2024-04-23T14:41:22.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:22 smithi028 ceph-mon[27810]: fsmap 8 up:standby 2024-04-23T14:41:22.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:22 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T14:41:22.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.645+0000 7f922cc00640 1 -- 172.21.15.28:0/2253166771 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f922802d620 con 0x7f92281008c0 2024-04-23T14:41:22.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.646+0000 7f921f400640 1 -- 172.21.15.28:0/2253166771 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7171 (secure 0 0 0) 0x7f921809c280 con 0x7f92281008c0 2024-04-23T14:41:22.648 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:41:22.648 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-23T14:41:22.648 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 9 2024-04-23T14:41:22.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.649+0000 7f922cc00640 1 -- 172.21.15.28:0/2253166771 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f920c0a3c30 msgr2=0x7f920c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:22.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.649+0000 7f922cc00640 1 --2- 172.21.15.28:0/2253166771 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f920c0a3c30 0x7f920c0a60f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f922819fe50 tx=0x7f9210005f90 comp rx=0 tx=0).stop 2024-04-23T14:41:22.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.649+0000 7f922cc00640 1 -- 172.21.15.28:0/2253166771 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92281008c0 msgr2=0x7f922819e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:22.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.649+0000 7f922cc00640 1 --2- 172.21.15.28:0/2253166771 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92281008c0 0x7f922819e940 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f921800d950 tx=0x7f921800de10 comp rx=0 tx=0).stop 2024-04-23T14:41:22.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.649+0000 7f922cc00640 1 -- 172.21.15.28:0/2253166771 shutdown_connections 2024-04-23T14:41:22.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.649+0000 7f922cc00640 1 --2- 172.21.15.28:0/2253166771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9228105be0 0x7f92281a5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:22.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.649+0000 7f922cc00640 1 --2- 172.21.15.28:0/2253166771 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9228103230 0x7f922819ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:22.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.649+0000 7f922cc00640 1 --2- 172.21.15.28:0/2253166771 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f920c0a3c30 0x7f920c0a60f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:22.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.649+0000 7f922cc00640 1 --2- 172.21.15.28:0/2253166771 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92281008c0 0x7f922819e940 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:22.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.649+0000 7f922cc00640 1 -- 172.21.15.28:0/2253166771 >> 172.21.15.28:0/2253166771 conn(0x7f92280fc4e0 msgr2=0x7f92281011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:22.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.649+0000 7f922cc00640 1 -- 172.21.15.28:0/2253166771 shutdown_connections 2024-04-23T14:41:22.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:22.650+0000 7f922cc00640 1 -- 172.21.15.28:0/2253166771 wait complete. 2024-04-23T14:41:22.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:22 smithi105 ceph-mon[32555]: Deploying daemon mds.i on smithi105 2024-04-23T14:41:22.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:22 smithi105 ceph-mon[32555]: mds.? [v2:172.21.15.38:6836/4078619599,v1:172.21.15.38:6837/4078619599] up:boot 2024-04-23T14:41:22.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:22 smithi105 ceph-mon[32555]: fsmap 8 up:standby 2024-04-23T14:41:22.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:22 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T14:41:22.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:22 smithi038 ceph-mon[32348]: Deploying daemon mds.i on smithi105 2024-04-23T14:41:22.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:22 smithi038 ceph-mon[32348]: mds.? [v2:172.21.15.38:6836/4078619599,v1:172.21.15.38:6837/4078619599] up:boot 2024-04-23T14:41:22.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:22 smithi038 ceph-mon[32348]: fsmap 8 up:standby 2024-04-23T14:41:22.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:22 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T14:41:23.406 INFO:journalctl@ceph.mds.i.smithi105.stdout:Apr 23 14:41:23 smithi105 systemd[1]: Starting Ceph mds.i for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T14:41:23.704 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:23 smithi105 ceph-mon[32555]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:23.704 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:23 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2253166771' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:23.704 INFO:journalctl@ceph.mds.i.smithi105.stdout:Apr 23 14:41:23 smithi105 podman[63656]: 2024-04-23 14:41:23.457727923 +0000 UTC m=+0.018168130 image pull 76e77f280c74312ca1add9b6019a4a921a516f91795d85bae00a0c8ca9b1c62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd 2024-04-23T14:41:23.704 INFO:journalctl@ceph.mds.i.smithi105.stdout:Apr 23 14:41:23 smithi105 podman[63656]: 2024-04-23 14:41:23.582390739 +0000 UTC m=+0.142830972 container create 56f412a338a7b1168f6e83a087604a0c7db5d59f6eec17879a9be93cf0801bcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-i, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-23T14:41:23.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:23 smithi038 ceph-mon[32348]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:23.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:23 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2253166771' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:23.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:23 smithi028 ceph-mon[27810]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:23.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:23 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2253166771' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:24.286 INFO:journalctl@ceph.mds.i.smithi105.stdout:Apr 23 14:41:23 smithi105 podman[63656]: 2024-04-23 14:41:23.826406309 +0000 UTC m=+0.386846506 container init 56f412a338a7b1168f6e83a087604a0c7db5d59f6eec17879a9be93cf0801bcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.schema-version=1.0) 2024-04-23T14:41:24.292 INFO:journalctl@ceph.mds.i.smithi105.stdout:Apr 23 14:41:23 smithi105 podman[63656]: 2024-04-23 14:41:23.828965815 +0000 UTC m=+0.389406007 container start 56f412a338a7b1168f6e83a087604a0c7db5d59f6eec17879a9be93cf0801bcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2) 2024-04-23T14:41:24.292 INFO:journalctl@ceph.mds.i.smithi105.stdout:Apr 23 14:41:23 smithi105 ceph-mds[63674]: -- 172.21.15.105:0/1299276733 <== mon.1 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b977005a40 con 0x55b976fd8800 2024-04-23T14:41:24.292 INFO:journalctl@ceph.mds.i.smithi105.stdout:Apr 23 14:41:23 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-i[63670]: starting mds.i at 2024-04-23T14:41:24.292 INFO:journalctl@ceph.mds.i.smithi105.stdout:Apr 23 14:41:23 smithi105 bash[63656]: 56f412a338a7b1168f6e83a087604a0c7db5d59f6eec17879a9be93cf0801bcf 2024-04-23T14:41:24.292 INFO:journalctl@ceph.mds.i.smithi105.stdout:Apr 23 14:41:23 smithi105 systemd[1]: Started Ceph mds.i for 1ee6e996-017e-11ef-bc93-c7b262605968. 2024-04-23T14:41:24.671 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:41:24.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.860+0000 7f8e29000640 1 -- 172.21.15.28:0/474093010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e24109f30 msgr2=0x7f8e2410c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:24.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.860+0000 7f8e29000640 1 --2- 172.21.15.28:0/474093010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e24109f30 0x7f8e2410c3c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c009ca0 tx=0x7f8e1c02a8c0 comp rx=0 tx=0).stop 2024-04-23T14:41:24.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.860+0000 7f8e29000640 1 -- 172.21.15.28:0/474093010 shutdown_connections 2024-04-23T14:41:24.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.860+0000 7f8e29000640 1 --2- 172.21.15.28:0/474093010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e24074a60 0x7f8e24074ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:24.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.860+0000 7f8e29000640 1 --2- 172.21.15.28:0/474093010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e24109f30 0x7f8e2410c3c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:24.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.860+0000 7f8e29000640 1 --2- 172.21.15.28:0/474093010 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8e24076040 0x7f8e24074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:24.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.860+0000 7f8e29000640 1 -- 172.21.15.28:0/474093010 >> 172.21.15.28:0/474093010 conn(0x7f8e240fd5e0 msgr2=0x7f8e240ffa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:24.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.860+0000 7f8e29000640 1 -- 172.21.15.28:0/474093010 shutdown_connections 2024-04-23T14:41:24.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.860+0000 7f8e29000640 1 -- 172.21.15.28:0/474093010 wait complete. 2024-04-23T14:41:24.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.861+0000 7f8e29000640 1 Processor -- start 2024-04-23T14:41:24.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.861+0000 7f8e29000640 1 -- start start 2024-04-23T14:41:24.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.862+0000 7f8e29000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e24074a60 0x7f8e2419a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:24.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.862+0000 7f8e29000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8e24076040 0x7f8e2419aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:24.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.862+0000 7f8e29000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e24109f30 0x7f8e241a1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:24.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.862+0000 7f8e29000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8e2410ff40 con 0x7f8e24109f30 2024-04-23T14:41:24.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.862+0000 7f8e29000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8e2410fdc0 con 0x7f8e24074a60 2024-04-23T14:41:24.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.862+0000 7f8e29000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8e241100c0 con 0x7f8e24076040 2024-04-23T14:41:24.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.862+0000 7f8e21e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e24074a60 0x7f8e2419a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:24.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.862+0000 7f8e22800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e24109f30 0x7f8e241a1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:24.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.862+0000 7f8e21e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e24074a60 0x7f8e2419a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:44114/0 (socket says 172.21.15.28:44114) 2024-04-23T14:41:24.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.862+0000 7f8e21e00640 1 -- 172.21.15.28:0/4279910433 learned_addr learned my addr 172.21.15.28:0/4279910433 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:24.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.862+0000 7f8e1be00640 1 --2- 172.21.15.28:0/4279910433 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8e24076040 0x7f8e2419aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:24.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.863+0000 7f8e21e00640 1 -- 172.21.15.28:0/4279910433 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8e24076040 msgr2=0x7f8e2419aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:24.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.863+0000 7f8e21e00640 1 --2- 172.21.15.28:0/4279910433 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8e24076040 0x7f8e2419aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:24.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.863+0000 7f8e21e00640 1 -- 172.21.15.28:0/4279910433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e24109f30 msgr2=0x7f8e241a1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:24.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.863+0000 7f8e21e00640 1 --2- 172.21.15.28:0/4279910433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e24109f30 0x7f8e241a1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:24.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.863+0000 7f8e21e00640 1 -- 172.21.15.28:0/4279910433 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e241a2140 con 0x7f8e24074a60 2024-04-23T14:41:24.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.863+0000 7f8e21e00640 1 --2- 172.21.15.28:0/4279910433 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e24074a60 0x7f8e2419a520 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8e1400e9f0 tx=0x7f8e1400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:24.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.863+0000 7f8e1aa00640 1 -- 172.21.15.28:0/4279910433 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e1400cd80 con 0x7f8e24074a60 2024-04-23T14:41:24.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.863+0000 7f8e1aa00640 1 -- 172.21.15.28:0/4279910433 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e14005500 con 0x7f8e24074a60 2024-04-23T14:41:24.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.863+0000 7f8e29000640 1 -- 172.21.15.28:0/4279910433 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e241a2430 con 0x7f8e24074a60 2024-04-23T14:41:24.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.863+0000 7f8e1aa00640 1 -- 172.21.15.28:0/4279910433 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e14010690 con 0x7f8e24074a60 2024-04-23T14:41:24.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.863+0000 7f8e29000640 1 -- 172.21.15.28:0/4279910433 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e241a2930 con 0x7f8e24074a60 2024-04-23T14:41:24.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.865+0000 7f8e29000640 1 -- 172.21.15.28:0/4279910433 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e24108c60 con 0x7f8e24074a60 2024-04-23T14:41:24.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.866+0000 7f8e1aa00640 1 -- 172.21.15.28:0/4279910433 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8e140046c0 con 0x7f8e24074a60 2024-04-23T14:41:24.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.868+0000 7f8e1aa00640 1 --2- 172.21.15.28:0/4279910433 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f8e080a3bb0 0x7f8e080a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:24.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.868+0000 7f8e1aa00640 1 -- 172.21.15.28:0/4279910433 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f8e140d0400 con 0x7f8e24074a60 2024-04-23T14:41:24.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.868+0000 7f8e1be00640 1 --2- 172.21.15.28:0/4279910433 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f8e080a3bb0 0x7f8e080a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:24.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.869+0000 7f8e1be00640 1 --2- 172.21.15.28:0/4279910433 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f8e080a3bb0 0x7f8e080a6070 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c0095f0 tx=0x7f8e0c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:24.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:24.872+0000 7f8e1aa00640 1 -- 172.21.15.28:0/4279910433 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8e1409d1e0 con 0x7f8e24074a60 2024-04-23T14:41:25.024 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:24 smithi028 ceph-mon[27810]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:25.024 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:24 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:25.024 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:24 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:25.024 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:24 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:25.024 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:24 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:41:25.024 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:24 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:41:25.025 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:24 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:25.025 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:24 smithi028 ceph-mon[27810]: Deploying daemon mds.g on smithi028 2024-04-23T14:41:25.025 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:24 smithi028 ceph-mon[27810]: mds.? [v2:172.21.15.105:6838/1987159450,v1:172.21.15.105:6839/1987159450] up:boot 2024-04-23T14:41:25.025 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:24 smithi028 ceph-mon[27810]: fsmap 9 up:standby 2024-04-23T14:41:25.025 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:24 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T14:41:25.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:25.129+0000 7f8e29000640 1 -- 172.21.15.28:0/4279910433 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8e2402d620 con 0x7f8e24074a60 2024-04-23T14:41:25.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:25.131+0000 7f8e1aa00640 1 -- 172.21.15.28:0/4279910433 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8001 (secure 0 0 0) 0x7f8e1409cae0 con 0x7f8e24074a60 2024-04-23T14:41:25.132 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:41:25.132 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-23T14:41:25.132 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 10 2024-04-23T14:41:25.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:25.133+0000 7f8e29000640 1 -- 172.21.15.28:0/4279910433 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f8e080a3bb0 msgr2=0x7f8e080a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:25.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:25.133+0000 7f8e29000640 1 --2- 172.21.15.28:0/4279910433 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f8e080a3bb0 0x7f8e080a6070 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c0095f0 tx=0x7f8e0c005f90 comp rx=0 tx=0).stop 2024-04-23T14:41:25.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:25.133+0000 7f8e29000640 1 -- 172.21.15.28:0/4279910433 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e24074a60 msgr2=0x7f8e2419a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:25.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:25.133+0000 7f8e29000640 1 --2- 172.21.15.28:0/4279910433 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e24074a60 0x7f8e2419a520 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8e1400e9f0 tx=0x7f8e1400edf0 comp rx=0 tx=0).stop 2024-04-23T14:41:25.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:25.133+0000 7f8e29000640 1 -- 172.21.15.28:0/4279910433 shutdown_connections 2024-04-23T14:41:25.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:25.133+0000 7f8e29000640 1 --2- 172.21.15.28:0/4279910433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e24109f30 0x7f8e241a1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:25.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:25.133+0000 7f8e29000640 1 --2- 172.21.15.28:0/4279910433 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8e24076040 0x7f8e2419aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:25.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:25.133+0000 7f8e29000640 1 --2- 172.21.15.28:0/4279910433 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f8e080a3bb0 0x7f8e080a6070 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:25.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:25.133+0000 7f8e29000640 1 --2- 172.21.15.28:0/4279910433 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e24074a60 0x7f8e2419a520 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:25.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:25.133+0000 7f8e29000640 1 -- 172.21.15.28:0/4279910433 >> 172.21.15.28:0/4279910433 conn(0x7f8e240fd5e0 msgr2=0x7f8e24105e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:25.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:25.134+0000 7f8e29000640 1 -- 172.21.15.28:0/4279910433 shutdown_connections 2024-04-23T14:41:25.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:25.134+0000 7f8e29000640 1 -- 172.21.15.28:0/4279910433 wait complete. 2024-04-23T14:41:25.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:24 smithi105 ceph-mon[32555]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:25.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:24 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:25.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:24 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:25.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:24 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:25.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:24 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:41:25.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:24 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:41:25.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:24 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:25.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:24 smithi105 ceph-mon[32555]: Deploying daemon mds.g on smithi028 2024-04-23T14:41:25.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:24 smithi105 ceph-mon[32555]: mds.? [v2:172.21.15.105:6838/1987159450,v1:172.21.15.105:6839/1987159450] up:boot 2024-04-23T14:41:25.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:24 smithi105 ceph-mon[32555]: fsmap 9 up:standby 2024-04-23T14:41:25.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:24 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T14:41:25.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:24 smithi038 ceph-mon[32348]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:25.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:24 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:25.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:24 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:25.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:24 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:25.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:24 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:41:25.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:24 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:41:25.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:24 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:25.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:24 smithi038 ceph-mon[32348]: Deploying daemon mds.g on smithi028 2024-04-23T14:41:25.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:24 smithi038 ceph-mon[32348]: mds.? [v2:172.21.15.105:6838/1987159450,v1:172.21.15.105:6839/1987159450] up:boot 2024-04-23T14:41:25.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:24 smithi038 ceph-mon[32348]: fsmap 9 up:standby 2024-04-23T14:41:25.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:24 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T14:41:26.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:25 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/4279910433' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:26.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:25 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/4279910433' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:26.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:25 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/4279910433' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:27.155 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:41:27.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:26 smithi028 ceph-mon[27810]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:27.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:26 smithi105 ceph-mon[32555]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:27.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:26 smithi038 ceph-mon[32348]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:27.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.435+0000 7f2e12c00640 1 -- 172.21.15.28:0/1144697548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e0c074a60 msgr2=0x7f2e0c074ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:27.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.435+0000 7f2e12c00640 1 --2- 172.21.15.28:0/1144697548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e0c074a60 0x7f2e0c074ee0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2df8009900 tx=0x7f2df802a3f0 comp rx=0 tx=0).stop 2024-04-23T14:41:27.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.435+0000 7f2e12c00640 1 -- 172.21.15.28:0/1144697548 shutdown_connections 2024-04-23T14:41:27.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.435+0000 7f2e12c00640 1 --2- 172.21.15.28:0/1144697548 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e0c109d60 0x7f2e0c10c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:27.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.435+0000 7f2e12c00640 1 --2- 172.21.15.28:0/1144697548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e0c074a60 0x7f2e0c074ee0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:27.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.435+0000 7f2e12c00640 1 --2- 172.21.15.28:0/1144697548 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e0c076040 0x7f2e0c074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:27.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.435+0000 7f2e12c00640 1 -- 172.21.15.28:0/1144697548 >> 172.21.15.28:0/1144697548 conn(0x7f2e0c0fd410 msgr2=0x7f2e0c0ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:27.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.435+0000 7f2e12c00640 1 -- 172.21.15.28:0/1144697548 shutdown_connections 2024-04-23T14:41:27.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.435+0000 7f2e12c00640 1 -- 172.21.15.28:0/1144697548 wait complete. 2024-04-23T14:41:27.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.436+0000 7f2e12c00640 1 Processor -- start 2024-04-23T14:41:27.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.436+0000 7f2e12c00640 1 -- start start 2024-04-23T14:41:27.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.437+0000 7f2e12c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e0c074a60 0x7f2e0c19a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:27.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.437+0000 7f2e12c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e0c076040 0x7f2e0c19a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:27.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.437+0000 7f2e12c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e0c109d60 0x7f2e0c1a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:27.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.437+0000 7f2e12c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2e0c10fce0 con 0x7f2e0c076040 2024-04-23T14:41:27.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.437+0000 7f2e12c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2e0c10fb60 con 0x7f2e0c074a60 2024-04-23T14:41:27.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.437+0000 7f2e12c00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2e0c10fe60 con 0x7f2e0c109d60 2024-04-23T14:41:27.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.437+0000 7f2e10e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e0c076040 0x7f2e0c19a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:27.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.437+0000 7f2e11800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e0c074a60 0x7f2e0c19a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:27.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.437+0000 7f2e10e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e0c076040 0x7f2e0c19a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58990/0 (socket says 172.21.15.28:58990) 2024-04-23T14:41:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.437+0000 7f2e12200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e0c109d60 0x7f2e0c1a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.437+0000 7f2e11800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e0c074a60 0x7f2e0c19a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:44144/0 (socket says 172.21.15.28:44144) 2024-04-23T14:41:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.437+0000 7f2e10e00640 1 -- 172.21.15.28:0/559947705 learned_addr learned my addr 172.21.15.28:0/559947705 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.438+0000 7f2e11800640 1 -- 172.21.15.28:0/559947705 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e0c109d60 msgr2=0x7f2e0c1a1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.438+0000 7f2e11800640 1 --2- 172.21.15.28:0/559947705 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e0c109d60 0x7f2e0c1a1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.438+0000 7f2e11800640 1 -- 172.21.15.28:0/559947705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e0c076040 msgr2=0x7f2e0c19a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.438+0000 7f2e11800640 1 --2- 172.21.15.28:0/559947705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e0c076040 0x7f2e0c19a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.438+0000 7f2e11800640 1 -- 172.21.15.28:0/559947705 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e0c1a1fc0 con 0x7f2e0c074a60 2024-04-23T14:41:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.438+0000 7f2e12200640 1 --2- 172.21.15.28:0/559947705 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e0c109d60 0x7f2e0c1a1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:41:27.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.438+0000 7f2e11800640 1 --2- 172.21.15.28:0/559947705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e0c074a60 0x7f2e0c19a3a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2e0000e9f0 tx=0x7f2e0000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:27.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.438+0000 7f2e0a000640 1 -- 172.21.15.28:0/559947705 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e0000cd80 con 0x7f2e0c074a60 2024-04-23T14:41:27.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.439+0000 7f2e0a000640 1 -- 172.21.15.28:0/559947705 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e000035f0 con 0x7f2e0c074a60 2024-04-23T14:41:27.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.439+0000 7f2e12c00640 1 -- 172.21.15.28:0/559947705 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e0c1a22b0 con 0x7f2e0c074a60 2024-04-23T14:41:27.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.439+0000 7f2e0a000640 1 -- 172.21.15.28:0/559947705 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e00010690 con 0x7f2e0c074a60 2024-04-23T14:41:27.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.439+0000 7f2e12c00640 1 -- 172.21.15.28:0/559947705 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e0c1a27d0 con 0x7f2e0c074a60 2024-04-23T14:41:27.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.440+0000 7f2e12c00640 1 -- 172.21.15.28:0/559947705 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e0c108a90 con 0x7f2e0c074a60 2024-04-23T14:41:27.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.442+0000 7f2e0a000640 1 -- 172.21.15.28:0/559947705 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2e00004020 con 0x7f2e0c074a60 2024-04-23T14:41:27.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.443+0000 7f2e0a000640 1 --2- 172.21.15.28:0/559947705 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2df40a3bb0 0x7f2df40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:27.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.444+0000 7f2e0a000640 1 -- 172.21.15.28:0/559947705 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f2e000d0280 con 0x7f2e0c074a60 2024-04-23T14:41:27.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.444+0000 7f2e10e00640 1 --2- 172.21.15.28:0/559947705 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2df40a3bb0 0x7f2df40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:27.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.445+0000 7f2e10e00640 1 --2- 172.21.15.28:0/559947705 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2df40a3bb0 0x7f2df40a6070 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2df8031440 tx=0x7f2df80315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:27.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.448+0000 7f2e0a000640 1 -- 172.21.15.28:0/559947705 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2e0009d060 con 0x7f2e0c074a60 2024-04-23T14:41:27.624 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 23 14:41:27 smithi028 systemd[1]: Starting Ceph mds.g for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T14:41:27.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.721+0000 7f2e12c00640 1 -- 172.21.15.28:0/559947705 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2e0c062c00 con 0x7f2e0c074a60 2024-04-23T14:41:27.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.722+0000 7f2e0a000640 1 -- 172.21.15.28:0/559947705 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8001 (secure 0 0 0) 0x7f2e0009c960 con 0x7f2e0c074a60 2024-04-23T14:41:27.724 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:41:27.724 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-23T14:41:27.725 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 10 2024-04-23T14:41:27.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.725+0000 7f2e12c00640 1 -- 172.21.15.28:0/559947705 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2df40a3bb0 msgr2=0x7f2df40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:27.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.725+0000 7f2e12c00640 1 --2- 172.21.15.28:0/559947705 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2df40a3bb0 0x7f2df40a6070 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2df8031440 tx=0x7f2df80315b0 comp rx=0 tx=0).stop 2024-04-23T14:41:27.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.725+0000 7f2e12c00640 1 -- 172.21.15.28:0/559947705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e0c074a60 msgr2=0x7f2e0c19a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:27.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.725+0000 7f2e12c00640 1 --2- 172.21.15.28:0/559947705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e0c074a60 0x7f2e0c19a3a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2e0000e9f0 tx=0x7f2e0000eeb0 comp rx=0 tx=0).stop 2024-04-23T14:41:27.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.725+0000 7f2e12c00640 1 -- 172.21.15.28:0/559947705 shutdown_connections 2024-04-23T14:41:27.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.725+0000 7f2e12c00640 1 --2- 172.21.15.28:0/559947705 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e0c109d60 0x7f2e0c1a1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:27.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.725+0000 7f2e12c00640 1 --2- 172.21.15.28:0/559947705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e0c076040 0x7f2e0c19a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:27.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.725+0000 7f2e12c00640 1 --2- 172.21.15.28:0/559947705 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2df40a3bb0 0x7f2df40a6070 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:27.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.725+0000 7f2e12c00640 1 --2- 172.21.15.28:0/559947705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e0c074a60 0x7f2e0c19a3a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:27.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.725+0000 7f2e12c00640 1 -- 172.21.15.28:0/559947705 >> 172.21.15.28:0/559947705 conn(0x7f2e0c0fd410 msgr2=0x7f2e0c105c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:27.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.725+0000 7f2e12c00640 1 -- 172.21.15.28:0/559947705 shutdown_connections 2024-04-23T14:41:27.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:27.725+0000 7f2e12c00640 1 -- 172.21.15.28:0/559947705 wait complete. 2024-04-23T14:41:27.969 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 23 14:41:27 smithi028 podman[67259]: 2024-04-23 14:41:27.673242798 +0000 UTC m=+0.038105344 image pull 76e77f280c74312ca1add9b6019a4a921a516f91795d85bae00a0c8ca9b1c62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd 2024-04-23T14:41:27.969 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 23 14:41:27 smithi028 podman[67259]: 2024-04-23 14:41:27.786443554 +0000 UTC m=+0.151306137 container create b54a6fd11f60c2887535c14e998fec3ddb5cd36fcfb2e165e57a8bf7651e0c49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-g, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd) 2024-04-23T14:41:28.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:27 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/559947705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:28.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:27 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/559947705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:28.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:27 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/559947705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:28.374 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 23 14:41:28 smithi028 podman[67259]: 2024-04-23 14:41:28.065422478 +0000 UTC m=+0.430285075 container init b54a6fd11f60c2887535c14e998fec3ddb5cd36fcfb2e165e57a8bf7651e0c49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-g, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-23T14:41:28.374 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 23 14:41:28 smithi028 podman[67259]: 2024-04-23 14:41:28.075226901 +0000 UTC m=+0.440089491 container start b54a6fd11f60c2887535c14e998fec3ddb5cd36fcfb2e165e57a8bf7651e0c49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-23T14:41:28.374 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 23 14:41:28 smithi028 ceph-mds[67287]: -- 172.21.15.28:0/1692433931 <== mon.2 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5609b5ceba40 con 0x5609b5cbf000 2024-04-23T14:41:28.374 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 23 14:41:28 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-g[67278]: starting mds.g at 2024-04-23T14:41:28.374 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 23 14:41:28 smithi028 bash[67259]: b54a6fd11f60c2887535c14e998fec3ddb5cd36fcfb2e165e57a8bf7651e0c49 2024-04-23T14:41:28.374 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 23 14:41:28 smithi028 systemd[1]: Started Ceph mds.g for 1ee6e996-017e-11ef-bc93-c7b262605968. 2024-04-23T14:41:29.277 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:29 smithi028 ceph-mon[27810]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:29.277 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:29 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:29.277 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:29 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:29.278 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:29 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:29.278 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:29 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:41:29.278 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:29 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:41:29.278 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:29 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:29.278 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:29 smithi028 ceph-mon[27810]: Deploying daemon mds.a on smithi028 2024-04-23T14:41:29.278 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:29 smithi028 ceph-mon[27810]: mds.? [v2:172.21.15.28:6838/940220693,v1:172.21.15.28:6839/940220693] up:boot 2024-04-23T14:41:29.278 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:29 smithi028 ceph-mon[27810]: fsmap 10 up:standby 2024-04-23T14:41:29.278 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:29 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-23T14:41:29.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:29 smithi105 ceph-mon[32555]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:29.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:29 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:29.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:29 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:29.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:29 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:29.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:29 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:41:29.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:29 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:41:29.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:29 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:29.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:29 smithi105 ceph-mon[32555]: Deploying daemon mds.a on smithi028 2024-04-23T14:41:29.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:29 smithi105 ceph-mon[32555]: mds.? [v2:172.21.15.28:6838/940220693,v1:172.21.15.28:6839/940220693] up:boot 2024-04-23T14:41:29.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:29 smithi105 ceph-mon[32555]: fsmap 10 up:standby 2024-04-23T14:41:29.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:29 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-23T14:41:29.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:29 smithi038 ceph-mon[32348]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:29.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:29 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:29.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:29 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:29.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:29 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:29.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:29 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:41:29.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:29 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:41:29.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:29 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:29.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:29 smithi038 ceph-mon[32348]: Deploying daemon mds.a on smithi028 2024-04-23T14:41:29.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:29 smithi038 ceph-mon[32348]: mds.? [v2:172.21.15.28:6838/940220693,v1:172.21.15.28:6839/940220693] up:boot 2024-04-23T14:41:29.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:29 smithi038 ceph-mon[32348]: fsmap 10 up:standby 2024-04-23T14:41:29.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:29 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-23T14:41:29.757 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:41:29.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.964+0000 7f037ee00640 1 -- 172.21.15.28:0/1731310412 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0378103c00 msgr2=0x7f037810a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:29.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.964+0000 7f037ee00640 1 --2- 172.21.15.28:0/1731310412 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0378103c00 0x7f037810a490 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0370009ca0 tx=0x7f037002a8c0 comp rx=0 tx=0).stop 2024-04-23T14:41:29.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.965+0000 7f037ee00640 1 -- 172.21.15.28:0/1731310412 shutdown_connections 2024-04-23T14:41:29.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.965+0000 7f037ee00640 1 --2- 172.21.15.28:0/1731310412 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0378103c00 0x7f037810a490 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:29.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.965+0000 7f037ee00640 1 --2- 172.21.15.28:0/1731310412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0378103240 0x7f03781036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:29.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.965+0000 7f037ee00640 1 --2- 172.21.15.28:0/1731310412 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0378102050 0x7f0378102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:29.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.965+0000 7f037ee00640 1 -- 172.21.15.28:0/1731310412 >> 172.21.15.28:0/1731310412 conn(0x7f03780fd5e0 msgr2=0x7f03780ffa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:29.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.965+0000 7f037ee00640 1 -- 172.21.15.28:0/1731310412 shutdown_connections 2024-04-23T14:41:29.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.965+0000 7f037ee00640 1 -- 172.21.15.28:0/1731310412 wait complete. 2024-04-23T14:41:29.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.967+0000 7f037ee00640 1 Processor -- start 2024-04-23T14:41:29.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.967+0000 7f037ee00640 1 -- start start 2024-04-23T14:41:29.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.967+0000 7f037ee00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0378102050 0x7f0378198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:29.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.967+0000 7f037ee00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0378103240 0x7f0378198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:29.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.967+0000 7f037ee00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0378103c00 0x7f037819f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:29.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.967+0000 7f037ee00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f037810dd30 con 0x7f0378103c00 2024-04-23T14:41:29.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.967+0000 7f037ee00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f037810dbb0 con 0x7f0378102050 2024-04-23T14:41:29.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.967+0000 7f037ee00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f037810deb0 con 0x7f0378103240 2024-04-23T14:41:29.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.968+0000 7f037ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0378103c00 0x7f037819f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:29.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.968+0000 7f0377e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0378102050 0x7f0378198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:29.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.968+0000 7f036fe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0378103240 0x7f0378198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:29.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.968+0000 7f0377e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0378102050 0x7f0378198300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:56176/0 (socket says 172.21.15.28:56176) 2024-04-23T14:41:29.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.968+0000 7f0377e00640 1 -- 172.21.15.28:0/2411242969 learned_addr learned my addr 172.21.15.28:0/2411242969 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:29.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.968+0000 7f036fe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0378103240 0x7f0378198840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:34278/0 (socket says 172.21.15.28:34278) 2024-04-23T14:41:29.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.968+0000 7f0377e00640 1 -- 172.21.15.28:0/2411242969 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0378103240 msgr2=0x7f0378198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:29.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.968+0000 7f0377e00640 1 --2- 172.21.15.28:0/2411242969 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0378103240 0x7f0378198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:29.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.968+0000 7f0377e00640 1 -- 172.21.15.28:0/2411242969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0378103c00 msgr2=0x7f037819f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:29.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.968+0000 7f0377e00640 1 --2- 172.21.15.28:0/2411242969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0378103c00 0x7f037819f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:29.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.968+0000 7f0377e00640 1 -- 172.21.15.28:0/2411242969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f037819ff20 con 0x7f0378102050 2024-04-23T14:41:29.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.968+0000 7f0377e00640 1 --2- 172.21.15.28:0/2411242969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0378102050 0x7f0378198300 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f036800e9c0 tx=0x7f036800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:29.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.969+0000 7f0375600640 1 -- 172.21.15.28:0/2411242969 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f036800cd80 con 0x7f0378102050 2024-04-23T14:41:29.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.969+0000 7f037ee00640 1 -- 172.21.15.28:0/2411242969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03781a0210 con 0x7f0378102050 2024-04-23T14:41:29.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.969+0000 7f0375600640 1 -- 172.21.15.28:0/2411242969 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03680035f0 con 0x7f0378102050 2024-04-23T14:41:29.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.969+0000 7f0375600640 1 -- 172.21.15.28:0/2411242969 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0368010690 con 0x7f0378102050 2024-04-23T14:41:29.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.969+0000 7f037ee00640 1 -- 172.21.15.28:0/2411242969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03781a0710 con 0x7f0378102050 2024-04-23T14:41:29.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.971+0000 7f037ee00640 1 -- 172.21.15.28:0/2411242969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0378109310 con 0x7f0378102050 2024-04-23T14:41:29.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.972+0000 7f0375600640 1 -- 172.21.15.28:0/2411242969 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0368010830 con 0x7f0378102050 2024-04-23T14:41:29.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.973+0000 7f0375600640 1 --2- 172.21.15.28:0/2411242969 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f035c0a3bb0 0x7f035c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:29.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.974+0000 7f0375600640 1 -- 172.21.15.28:0/2411242969 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f03680d03a0 con 0x7f0378102050 2024-04-23T14:41:29.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.974+0000 7f036fe00640 1 --2- 172.21.15.28:0/2411242969 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f035c0a3bb0 0x7f035c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:29.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.975+0000 7f036fe00640 1 --2- 172.21.15.28:0/2411242969 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f035c0a3bb0 0x7f035c0a6070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0378199810 tx=0x7f0360005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:29.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:29.978+0000 7f0375600640 1 -- 172.21.15.28:0/2411242969 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f036809d180 con 0x7f0378102050 2024-04-23T14:41:30.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:30.225+0000 7f037ee00640 1 -- 172.21.15.28:0/2411242969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f037802d620 con 0x7f0378102050 2024-04-23T14:41:30.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:30.226+0000 7f0375600640 1 -- 172.21.15.28:0/2411242969 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8824 (secure 0 0 0) 0x7f037802d620 con 0x7f0378102050 2024-04-23T14:41:30.228 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:41:30.228 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24583,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-04-23T14:41:30.228 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 11 2024-04-23T14:41:30.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:30.230+0000 7f037ee00640 1 -- 172.21.15.28:0/2411242969 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f035c0a3bb0 msgr2=0x7f035c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:30.230+0000 7f037ee00640 1 --2- 172.21.15.28:0/2411242969 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f035c0a3bb0 0x7f035c0a6070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0378199810 tx=0x7f0360005f90 comp rx=0 tx=0).stop 2024-04-23T14:41:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:30.230+0000 7f037ee00640 1 -- 172.21.15.28:0/2411242969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0378102050 msgr2=0x7f0378198300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:30.230+0000 7f037ee00640 1 --2- 172.21.15.28:0/2411242969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0378102050 0x7f0378198300 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f036800e9c0 tx=0x7f036800ee80 comp rx=0 tx=0).stop 2024-04-23T14:41:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:30.230+0000 7f037ee00640 1 -- 172.21.15.28:0/2411242969 shutdown_connections 2024-04-23T14:41:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:30.230+0000 7f037ee00640 1 --2- 172.21.15.28:0/2411242969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0378103c00 0x7f037819f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:30.230+0000 7f037ee00640 1 --2- 172.21.15.28:0/2411242969 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0378103240 0x7f0378198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:30.230+0000 7f037ee00640 1 --2- 172.21.15.28:0/2411242969 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f035c0a3bb0 0x7f035c0a6070 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:30.230+0000 7f037ee00640 1 --2- 172.21.15.28:0/2411242969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0378102050 0x7f0378198300 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:30.230+0000 7f037ee00640 1 -- 172.21.15.28:0/2411242969 >> 172.21.15.28:0/2411242969 conn(0x7f03780fd5e0 msgr2=0x7f03781064e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:30.230+0000 7f037ee00640 1 -- 172.21.15.28:0/2411242969 shutdown_connections 2024-04-23T14:41:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:30.231+0000 7f037ee00640 1 -- 172.21.15.28:0/2411242969 wait complete. 2024-04-23T14:41:30.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:30 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2411242969' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:30.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:30 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2411242969' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:30.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:30 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2411242969' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:31.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:31 smithi028 ceph-mon[27810]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:31.524 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:31 smithi038 ceph-mon[32348]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:31.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:31 smithi105 ceph-mon[32555]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:31.753 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 23 14:41:31 smithi028 systemd[1]: Starting Ceph mds.a for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T14:41:32.071 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 23 14:41:31 smithi028 podman[67610]: 2024-04-23 14:41:31.656095735 +0000 UTC m=+0.034209629 image pull 76e77f280c74312ca1add9b6019a4a921a516f91795d85bae00a0c8ca9b1c62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd 2024-04-23T14:41:32.071 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 23 14:41:31 smithi028 podman[67610]: 2024-04-23 14:41:31.799086394 +0000 UTC m=+0.177200326 container create cd19ff722a9bcea3344a05d273e392c6b10884e2858cf63ea37a6d32ff818e44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-a, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-23T14:41:32.071 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 23 14:41:32 smithi028 podman[67610]: 2024-04-23 14:41:32.069208798 +0000 UTC m=+0.447322753 container init cd19ff722a9bcea3344a05d273e392c6b10884e2858cf63ea37a6d32ff818e44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-a, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True) 2024-04-23T14:41:32.251 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:41:32.321 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 23 14:41:32 smithi028 podman[67610]: 2024-04-23 14:41:32.080488841 +0000 UTC m=+0.458602753 container start cd19ff722a9bcea3344a05d273e392c6b10884e2858cf63ea37a6d32ff818e44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.license=GPLv2) 2024-04-23T14:41:32.321 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 23 14:41:32 smithi028 ceph-mds[67629]: -- 172.21.15.28:0/2214595497 <== mon.2 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c87670fa40 con 0x55c8766e2c00 2024-04-23T14:41:32.321 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 23 14:41:32 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-a[67625]: starting mds.a at 2024-04-23T14:41:32.321 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 23 14:41:32 smithi028 bash[67610]: cd19ff722a9bcea3344a05d273e392c6b10884e2858cf63ea37a6d32ff818e44 2024-04-23T14:41:32.321 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 23 14:41:32 smithi028 systemd[1]: Started Ceph mds.a for 1ee6e996-017e-11ef-bc93-c7b262605968. 2024-04-23T14:41:32.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.469+0000 7f69fa800640 1 -- 172.21.15.28:0/1544476765 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69f4101e70 msgr2=0x7f69f4102290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:32.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.469+0000 7f69fa800640 1 --2- 172.21.15.28:0/1544476765 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69f4101e70 0x7f69f4102290 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f69e8009ca0 tx=0x7f69e802a8c0 comp rx=0 tx=0).stop 2024-04-23T14:41:32.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.469+0000 7f69fa800640 1 -- 172.21.15.28:0/1544476765 shutdown_connections 2024-04-23T14:41:32.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.469+0000 7f69fa800640 1 --2- 172.21.15.28:0/1544476765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69f4103980 0x7f69f410a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:32.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.469+0000 7f69fa800640 1 --2- 172.21.15.28:0/1544476765 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f69f4102fc0 0x7f69f4103440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:32.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.469+0000 7f69fa800640 1 --2- 172.21.15.28:0/1544476765 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69f4101e70 0x7f69f4102290 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:32.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.469+0000 7f69fa800640 1 -- 172.21.15.28:0/1544476765 >> 172.21.15.28:0/1544476765 conn(0x7f69f40fd3e0 msgr2=0x7f69f40ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:32.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.469+0000 7f69fa800640 1 -- 172.21.15.28:0/1544476765 shutdown_connections 2024-04-23T14:41:32.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.469+0000 7f69fa800640 1 -- 172.21.15.28:0/1544476765 wait complete. 2024-04-23T14:41:32.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.470+0000 7f69fa800640 1 Processor -- start 2024-04-23T14:41:32.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.471+0000 7f69fa800640 1 -- start start 2024-04-23T14:41:32.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.471+0000 7f69fa800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f69f4101e70 0x7f69f419a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:32.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.471+0000 7f69fa800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69f4102fc0 0x7f69f419a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:32.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.471+0000 7f69fa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69f4103980 0x7f69f41a1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:32.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.471+0000 7f69fa800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f69f410db60 con 0x7f69f4103980 2024-04-23T14:41:32.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.471+0000 7f69fa800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f69f410d9e0 con 0x7f69f4102fc0 2024-04-23T14:41:32.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.471+0000 7f69fa800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f69f410dce0 con 0x7f69f4101e70 2024-04-23T14:41:32.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.471+0000 7f69f8a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69f4102fc0 0x7f69f419a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:32.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.471+0000 7f69f9400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f69f4101e70 0x7f69f419a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:32.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.471+0000 7f69f8a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69f4102fc0 0x7f69f419a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:56214/0 (socket says 172.21.15.28:56214) 2024-04-23T14:41:32.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.471+0000 7f69f8a00640 1 -- 172.21.15.28:0/2853810506 learned_addr learned my addr 172.21.15.28:0/2853810506 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:32.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.471+0000 7f69f9e00640 1 --2- 172.21.15.28:0/2853810506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69f4103980 0x7f69f41a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:32.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.472+0000 7f69f8a00640 1 -- 172.21.15.28:0/2853810506 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f69f4101e70 msgr2=0x7f69f419a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:32.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.472+0000 7f69f8a00640 1 --2- 172.21.15.28:0/2853810506 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f69f4101e70 0x7f69f419a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:32.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.472+0000 7f69f8a00640 1 -- 172.21.15.28:0/2853810506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69f4103980 msgr2=0x7f69f41a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:32.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.472+0000 7f69f8a00640 1 --2- 172.21.15.28:0/2853810506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69f4103980 0x7f69f41a1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.472+0000 7f69f8a00640 1 -- 172.21.15.28:0/2853810506 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69f41a1fe0 con 0x7f69f4102fc0 2024-04-23T14:41:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.472+0000 7f69f8a00640 1 --2- 172.21.15.28:0/2853810506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69f4102fc0 0x7f69f419a900 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f69e000e9c0 tx=0x7f69e000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.472+0000 7f69f2000640 1 -- 172.21.15.28:0/2853810506 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e000cd80 con 0x7f69f4102fc0 2024-04-23T14:41:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.472+0000 7f69f2000640 1 -- 172.21.15.28:0/2853810506 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69e00035f0 con 0x7f69f4102fc0 2024-04-23T14:41:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.472+0000 7f69fa800640 1 -- 172.21.15.28:0/2853810506 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69f41a22d0 con 0x7f69f4102fc0 2024-04-23T14:41:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.472+0000 7f69f2000640 1 -- 172.21.15.28:0/2853810506 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e0010690 con 0x7f69f4102fc0 2024-04-23T14:41:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.472+0000 7f69fa800640 1 -- 172.21.15.28:0/2853810506 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69f41a27d0 con 0x7f69f4102fc0 2024-04-23T14:41:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.474+0000 7f69fa800640 1 -- 172.21.15.28:0/2853810506 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69f41a2a90 con 0x7f69f4102fc0 2024-04-23T14:41:32.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.476+0000 7f69f2000640 1 -- 172.21.15.28:0/2853810506 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f69e00037a0 con 0x7f69f4102fc0 2024-04-23T14:41:32.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.477+0000 7f69f2000640 1 --2- 172.21.15.28:0/2853810506 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f69dc0a3bb0 0x7f69dc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:32.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.478+0000 7f69f2000640 1 -- 172.21.15.28:0/2853810506 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f69e00d0360 con 0x7f69f4102fc0 2024-04-23T14:41:32.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.478+0000 7f69f9400640 1 --2- 172.21.15.28:0/2853810506 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f69dc0a3bb0 0x7f69dc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:32.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.478+0000 7f69f9400640 1 --2- 172.21.15.28:0/2853810506 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f69dc0a3bb0 0x7f69dc0a6070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f69e8005e30 tx=0x7f69e8005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:32.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.486+0000 7f69f2000640 1 -- 172.21.15.28:0/2853810506 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f69e009d140 con 0x7f69f4102fc0 2024-04-23T14:41:32.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.738+0000 7f69fa800640 1 -- 172.21.15.28:0/2853810506 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f69f41a2c60 con 0x7f69f4102fc0 2024-04-23T14:41:32.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.740+0000 7f69f2000640 1 -- 172.21.15.28:0/2853810506 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9650 (secure 0 0 0) 0x7f69e009ca40 con 0x7f69f4102fc0 2024-04-23T14:41:32.741 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:41:32.741 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24583,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-04-23T14:41:32.741 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 12 2024-04-23T14:41:32.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.743+0000 7f69fa800640 1 -- 172.21.15.28:0/2853810506 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f69dc0a3bb0 msgr2=0x7f69dc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:32.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.743+0000 7f69fa800640 1 --2- 172.21.15.28:0/2853810506 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f69dc0a3bb0 0x7f69dc0a6070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f69e8005e30 tx=0x7f69e8005f50 comp rx=0 tx=0).stop 2024-04-23T14:41:32.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.743+0000 7f69fa800640 1 -- 172.21.15.28:0/2853810506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69f4102fc0 msgr2=0x7f69f419a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:32.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.743+0000 7f69fa800640 1 --2- 172.21.15.28:0/2853810506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69f4102fc0 0x7f69f419a900 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f69e000e9c0 tx=0x7f69e000ee80 comp rx=0 tx=0).stop 2024-04-23T14:41:32.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.743+0000 7f69fa800640 1 -- 172.21.15.28:0/2853810506 shutdown_connections 2024-04-23T14:41:32.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.743+0000 7f69fa800640 1 --2- 172.21.15.28:0/2853810506 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f69dc0a3bb0 0x7f69dc0a6070 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:32.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.743+0000 7f69fa800640 1 --2- 172.21.15.28:0/2853810506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69f4102fc0 0x7f69f419a900 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:32.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.743+0000 7f69fa800640 1 --2- 172.21.15.28:0/2853810506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69f4103980 0x7f69f41a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:32.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.743+0000 7f69fa800640 1 --2- 172.21.15.28:0/2853810506 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f69f4101e70 0x7f69f419a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:32.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.743+0000 7f69fa800640 1 -- 172.21.15.28:0/2853810506 >> 172.21.15.28:0/2853810506 conn(0x7f69f40fd3e0 msgr2=0x7f69f4074230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:32.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.743+0000 7f69fa800640 1 -- 172.21.15.28:0/2853810506 shutdown_connections 2024-04-23T14:41:32.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:32.743+0000 7f69fa800640 1 -- 172.21.15.28:0/2853810506 wait complete. 2024-04-23T14:41:33.291 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:33 smithi038 ceph-mon[32348]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:33.291 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:33 smithi038 ceph-mon[32348]: mds.? [v2:172.21.15.28:6840/1569158142,v1:172.21.15.28:6841/1569158142] up:boot 2024-04-23T14:41:33.291 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:33 smithi038 ceph-mon[32348]: fsmap 11 up:standby 2024-04-23T14:41:33.291 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:33 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-23T14:41:33.291 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:33 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:33.291 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:33 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:33.291 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:33 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:33.291 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:33 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:41:33.291 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:33 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:41:33.291 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:33 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:33.291 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:33 smithi038 ceph-mon[32348]: Deploying daemon mds.b on smithi038 2024-04-23T14:41:33.291 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:33 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2853810506' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:33.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:33 smithi105 ceph-mon[32555]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:33.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:33 smithi105 ceph-mon[32555]: mds.? [v2:172.21.15.28:6840/1569158142,v1:172.21.15.28:6841/1569158142] up:boot 2024-04-23T14:41:33.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:33 smithi105 ceph-mon[32555]: fsmap 11 up:standby 2024-04-23T14:41:33.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:33 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-23T14:41:33.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:33 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:33.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:33 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:33.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:33 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:33.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:33 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:41:33.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:33 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:41:33.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:33 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:33.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:33 smithi105 ceph-mon[32555]: Deploying daemon mds.b on smithi038 2024-04-23T14:41:33.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:33 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2853810506' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:33.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:33 smithi028 ceph-mon[27810]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:33.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:33 smithi028 ceph-mon[27810]: mds.? [v2:172.21.15.28:6840/1569158142,v1:172.21.15.28:6841/1569158142] up:boot 2024-04-23T14:41:33.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:33 smithi028 ceph-mon[27810]: fsmap 11 up:standby 2024-04-23T14:41:33.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:33 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-23T14:41:33.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:33 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:33.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:33 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:33.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:33 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:33.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:33 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T14:41:33.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:33 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T14:41:33.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:33 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:33.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:33 smithi028 ceph-mon[27810]: Deploying daemon mds.b on smithi038 2024-04-23T14:41:33.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:33 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2853810506' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:34.623 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 23 14:41:34 smithi038 podman[63016]: 2024-04-23 14:41:34.579679322 +0000 UTC m=+0.116763547 container create bb8d937f28efb51feeb5f2a30227da9e0c491a9d9d221e6573f936989de13ced (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-b, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-23T14:41:34.764 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:41:34.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.984+0000 7f1558a00640 1 -- 172.21.15.28:0/612366146 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1554105c30 msgr2=0x7f155410c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:34.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.984+0000 7f1558a00640 1 --2- 172.21.15.28:0/612366146 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1554105c30 0x7f155410c4c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f154c009ca0 tx=0x7f154c02a8b0 comp rx=0 tx=0).stop 2024-04-23T14:41:34.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.984+0000 7f1558a00640 1 -- 172.21.15.28:0/612366146 shutdown_connections 2024-04-23T14:41:34.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.984+0000 7f1558a00640 1 --2- 172.21.15.28:0/612366146 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1554105c30 0x7f155410c4c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:34.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.984+0000 7f1558a00640 1 --2- 172.21.15.28:0/612366146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1554105270 0x7f15541056f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:34.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.984+0000 7f1558a00640 1 --2- 172.21.15.28:0/612366146 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1554104080 0x7f15541044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:34.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.984+0000 7f1558a00640 1 -- 172.21.15.28:0/612366146 >> 172.21.15.28:0/612366146 conn(0x7f15540ff610 msgr2=0x7f1554101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:34.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.984+0000 7f1558a00640 1 -- 172.21.15.28:0/612366146 shutdown_connections 2024-04-23T14:41:34.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.984+0000 7f1558a00640 1 -- 172.21.15.28:0/612366146 wait complete. 2024-04-23T14:41:34.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.985+0000 7f1558a00640 1 Processor -- start 2024-04-23T14:41:34.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.985+0000 7f1558a00640 1 -- start start 2024-04-23T14:41:34.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.985+0000 7f1558a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1554104080 0x7f155419a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:34.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.986+0000 7f1558a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1554105270 0x7f155419a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:34.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.986+0000 7f1558a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1554105c30 0x7f15541a1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:34.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.986+0000 7f1558a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f155410fd60 con 0x7f1554104080 2024-04-23T14:41:34.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.986+0000 7f1558a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f155410fbe0 con 0x7f1554105270 2024-04-23T14:41:34.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.986+0000 7f1558a00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f155410fee0 con 0x7f1554105c30 2024-04-23T14:41:34.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.986+0000 7f1553400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1554104080 0x7f155419a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:34.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.986+0000 7f1552a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1554105270 0x7f155419a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:34.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.986+0000 7f1552a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1554105270 0x7f155419a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:56224/0 (socket says 172.21.15.28:56224) 2024-04-23T14:41:34.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.986+0000 7f1553400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1554104080 0x7f155419a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56598/0 (socket says 172.21.15.28:56598) 2024-04-23T14:41:34.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.986+0000 7f1552a00640 1 -- 172.21.15.28:0/2631356588 learned_addr learned my addr 172.21.15.28:0/2631356588 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:34.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.986+0000 7f1553e00640 1 --2- 172.21.15.28:0/2631356588 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1554105c30 0x7f15541a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:34.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.986+0000 7f1552a00640 1 -- 172.21.15.28:0/2631356588 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1554105c30 msgr2=0x7f15541a1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:34.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.986+0000 7f1552a00640 1 --2- 172.21.15.28:0/2631356588 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1554105c30 0x7f15541a1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:34.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.986+0000 7f1552a00640 1 -- 172.21.15.28:0/2631356588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1554104080 msgr2=0x7f155419a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:34.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.986+0000 7f1552a00640 1 --2- 172.21.15.28:0/2631356588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1554104080 0x7f155419a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:34.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.986+0000 7f1552a00640 1 -- 172.21.15.28:0/2631356588 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15541a1ef0 con 0x7f1554105270 2024-04-23T14:41:34.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.986+0000 7f1552a00640 1 --2- 172.21.15.28:0/2631356588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1554105270 0x7f155419a810 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f154000e9f0 tx=0x7f154000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:34.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.987+0000 7f154be00640 1 -- 172.21.15.28:0/2631356588 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f154000cd80 con 0x7f1554105270 2024-04-23T14:41:34.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.987+0000 7f154be00640 1 -- 172.21.15.28:0/2631356588 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1540010910 con 0x7f1554105270 2024-04-23T14:41:34.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.987+0000 7f1558a00640 1 -- 172.21.15.28:0/2631356588 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15541a21e0 con 0x7f1554105270 2024-04-23T14:41:34.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.987+0000 7f154be00640 1 -- 172.21.15.28:0/2631356588 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1540018990 con 0x7f1554105270 2024-04-23T14:41:34.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.987+0000 7f1558a00640 1 -- 172.21.15.28:0/2631356588 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15541a2780 con 0x7f1554105270 2024-04-23T14:41:34.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.988+0000 7f1558a00640 1 -- 172.21.15.28:0/2631356588 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f155410b2d0 con 0x7f1554105270 2024-04-23T14:41:34.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.990+0000 7f154be00640 1 -- 172.21.15.28:0/2631356588 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1540018b30 con 0x7f1554105270 2024-04-23T14:41:34.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.992+0000 7f154be00640 1 --2- 172.21.15.28:0/2631356588 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f15380a3c30 0x7f15380a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:34.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.992+0000 7f154be00640 1 -- 172.21.15.28:0/2631356588 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f15400d02d0 con 0x7f1554105270 2024-04-23T14:41:34.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.992+0000 7f1553400640 1 --2- 172.21.15.28:0/2631356588 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f15380a3c30 0x7f15380a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:34.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.992+0000 7f1553400640 1 --2- 172.21.15.28:0/2631356588 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f15380a3c30 0x7f15380a60f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f15440095f0 tx=0x7f1544005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:34.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:34.996+0000 7f154be00640 1 -- 172.21.15.28:0/2631356588 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f154009ca60 con 0x7f1554105270 2024-04-23T14:41:35.123 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 23 14:41:34 smithi038 podman[63016]: 2024-04-23 14:41:34.824487572 +0000 UTC m=+0.361571800 container init bb8d937f28efb51feeb5f2a30227da9e0c491a9d9d221e6573f936989de13ced (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240416, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T14:41:35.123 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 23 14:41:34 smithi038 podman[63016]: 2024-04-23 14:41:34.827127325 +0000 UTC m=+0.364211553 container start bb8d937f28efb51feeb5f2a30227da9e0c491a9d9d221e6573f936989de13ced (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-b, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-23T14:41:35.124 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 23 14:41:34 smithi038 ceph-mds[63035]: -- 172.21.15.38:0/2468784525 <== mon.2 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55625606da40 con 0x556256041000 2024-04-23T14:41:35.124 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 23 14:41:34 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-b[63031]: starting mds.b at 2024-04-23T14:41:35.124 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 23 14:41:34 smithi038 bash[63016]: bb8d937f28efb51feeb5f2a30227da9e0c491a9d9d221e6573f936989de13ced 2024-04-23T14:41:35.124 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 23 14:41:34 smithi038 systemd[1]: Started Ceph mds.b for 1ee6e996-017e-11ef-bc93-c7b262605968. 2024-04-23T14:41:35.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.268+0000 7f1558a00640 1 -- 172.21.15.28:0/2631356588 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1554062c00 con 0x7f1554105270 2024-04-23T14:41:35.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.275+0000 7f154be00640 1 -- 172.21.15.28:0/2631356588 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10473 (secure 0 0 0) 0x7f15400a0d60 con 0x7f1554105270 2024-04-23T14:41:35.278 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:41:35.278 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24583,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24587,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-23T14:41:35.278 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 13 2024-04-23T14:41:35.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.279+0000 7f1558a00640 1 -- 172.21.15.28:0/2631356588 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f15380a3c30 msgr2=0x7f15380a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:35.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.279+0000 7f1558a00640 1 --2- 172.21.15.28:0/2631356588 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f15380a3c30 0x7f15380a60f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f15440095f0 tx=0x7f1544005f90 comp rx=0 tx=0).stop 2024-04-23T14:41:35.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.279+0000 7f1558a00640 1 -- 172.21.15.28:0/2631356588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1554105270 msgr2=0x7f155419a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:35.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.279+0000 7f1558a00640 1 --2- 172.21.15.28:0/2631356588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1554105270 0x7f155419a810 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f154000e9f0 tx=0x7f154000eeb0 comp rx=0 tx=0).stop 2024-04-23T14:41:35.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.279+0000 7f1558a00640 1 -- 172.21.15.28:0/2631356588 shutdown_connections 2024-04-23T14:41:35.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.279+0000 7f1558a00640 1 --2- 172.21.15.28:0/2631356588 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f15380a3c30 0x7f15380a60f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:35.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.279+0000 7f1558a00640 1 --2- 172.21.15.28:0/2631356588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1554105270 0x7f155419a810 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:35.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.279+0000 7f1558a00640 1 --2- 172.21.15.28:0/2631356588 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1554105c30 0x7f15541a1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:35.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.279+0000 7f1558a00640 1 --2- 172.21.15.28:0/2631356588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1554104080 0x7f155419a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:35.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.279+0000 7f1558a00640 1 -- 172.21.15.28:0/2631356588 >> 172.21.15.28:0/2631356588 conn(0x7f15540ff610 msgr2=0x7f15541084a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:35.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.279+0000 7f1558a00640 1 -- 172.21.15.28:0/2631356588 shutdown_connections 2024-04-23T14:41:35.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.279+0000 7f1558a00640 1 -- 172.21.15.28:0/2631356588 wait complete. 2024-04-23T14:41:35.301 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:41:35.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.475+0000 7fb678a00640 1 -- 172.21.15.28:0/2187479761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6741052d0 msgr2=0x7fb6741076c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:35.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.475+0000 7fb678a00640 1 --2- 172.21.15.28:0/2187479761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6741052d0 0x7fb6741076c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb660009900 tx=0x7fb66002a3f0 comp rx=0 tx=0).stop 2024-04-23T14:41:35.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.475+0000 7fb678a00640 1 -- 172.21.15.28:0/2187479761 shutdown_connections 2024-04-23T14:41:35.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.475+0000 7fb678a00640 1 --2- 172.21.15.28:0/2187479761 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb674107c00 0x7fb674109ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:35.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.475+0000 7fb678a00640 1 --2- 172.21.15.28:0/2187479761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6741052d0 0x7fb6741076c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:35.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.475+0000 7fb678a00640 1 --2- 172.21.15.28:0/2187479761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6741029a0 0x7fb674104d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:35.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.475+0000 7fb678a00640 1 -- 172.21.15.28:0/2187479761 >> 172.21.15.28:0/2187479761 conn(0x7fb6740fc2d0 msgr2=0x7fb6740fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:35.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.475+0000 7fb678a00640 1 -- 172.21.15.28:0/2187479761 shutdown_connections 2024-04-23T14:41:35.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.475+0000 7fb678a00640 1 -- 172.21.15.28:0/2187479761 wait complete. 2024-04-23T14:41:35.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.476+0000 7fb678a00640 1 Processor -- start 2024-04-23T14:41:35.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.476+0000 7fb678a00640 1 -- start start 2024-04-23T14:41:35.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.477+0000 7fb678a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6741029a0 0x7fb67419a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:35.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.477+0000 7fb678a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb6741052d0 0x7fb67419a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:35.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.477+0000 7fb678a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb674107c00 0x7fb6741a19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:35.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.477+0000 7fb678a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb67410db50 con 0x7fb674107c00 2024-04-23T14:41:35.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.477+0000 7fb678a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb67410d9d0 con 0x7fb6741052d0 2024-04-23T14:41:35.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.477+0000 7fb678a00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb67410dcd0 con 0x7fb6741029a0 2024-04-23T14:41:35.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.477+0000 7fb672800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6741029a0 0x7fb67419a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:35.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.477+0000 7fb671e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb6741052d0 0x7fb67419a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:35.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.477+0000 7fb672800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6741029a0 0x7fb67419a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:34348/0 (socket says 172.21.15.28:34348) 2024-04-23T14:41:35.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.477+0000 7fb672800640 1 -- 172.21.15.28:0/3659872341 learned_addr learned my addr 172.21.15.28:0/3659872341 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:35.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.477+0000 7fb673200640 1 --2- 172.21.15.28:0/3659872341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb674107c00 0x7fb6741a19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:35.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.477+0000 7fb672800640 1 -- 172.21.15.28:0/3659872341 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb6741052d0 msgr2=0x7fb67419a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:35.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.477+0000 7fb672800640 1 --2- 172.21.15.28:0/3659872341 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb6741052d0 0x7fb67419a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:35.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.477+0000 7fb672800640 1 -- 172.21.15.28:0/3659872341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb674107c00 msgr2=0x7fb6741a19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:35.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.477+0000 7fb672800640 1 --2- 172.21.15.28:0/3659872341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb674107c00 0x7fb6741a19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:35.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.477+0000 7fb672800640 1 -- 172.21.15.28:0/3659872341 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6741a2040 con 0x7fb6741029a0 2024-04-23T14:41:35.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.478+0000 7fb672800640 1 --2- 172.21.15.28:0/3659872341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6741029a0 0x7fb67419a420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb66400e9f0 tx=0x7fb66400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:35.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.478+0000 7fb66b400640 1 -- 172.21.15.28:0/3659872341 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb66400cd80 con 0x7fb6741029a0 2024-04-23T14:41:35.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.478+0000 7fb66b400640 1 -- 172.21.15.28:0/3659872341 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb664010910 con 0x7fb6741029a0 2024-04-23T14:41:35.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.478+0000 7fb66b400640 1 -- 172.21.15.28:0/3659872341 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb664018990 con 0x7fb6741029a0 2024-04-23T14:41:35.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.478+0000 7fb678a00640 1 -- 172.21.15.28:0/3659872341 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6741a2330 con 0x7fb6741029a0 2024-04-23T14:41:35.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.478+0000 7fb678a00640 1 -- 172.21.15.28:0/3659872341 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6741a2800 con 0x7fb6741029a0 2024-04-23T14:41:35.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.479+0000 7fb678a00640 1 -- 172.21.15.28:0/3659872341 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6741096f0 con 0x7fb6741029a0 2024-04-23T14:41:35.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.481+0000 7fb66b400640 1 -- 172.21.15.28:0/3659872341 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb664010ac0 con 0x7fb6741029a0 2024-04-23T14:41:35.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.482+0000 7fb66b400640 1 --2- 172.21.15.28:0/3659872341 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb6580a3c30 0x7fb6580a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:35.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.482+0000 7fb66b400640 1 -- 172.21.15.28:0/3659872341 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb6640d0270 con 0x7fb6741029a0 2024-04-23T14:41:35.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.483+0000 7fb671e00640 1 --2- 172.21.15.28:0/3659872341 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb6580a3c30 0x7fb6580a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:35.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.483+0000 7fb671e00640 1 --2- 172.21.15.28:0/3659872341 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb6580a3c30 0x7fb6580a60f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb66002a9e0 tx=0x7fb6600315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:35.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.486+0000 7fb66b400640 1 -- 172.21.15.28:0/3659872341 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb66409ca00 con 0x7fb6741029a0 2024-04-23T14:41:35.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:35 smithi105 ceph-mon[32555]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:35.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:35.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:35.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:35.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:35.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:35 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:35.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:35 smithi038 ceph-mon[32348]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:35.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:35 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:35.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:35 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:35.622 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:35 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:35.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:35 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:35.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:35 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:35.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:35 smithi028 ceph-mon[27810]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:35.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:35 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:35.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:35 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:35.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:35 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:35.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:35 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:35.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:35 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:35.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.693+0000 7fb678a00640 1 -- 172.21.15.28:0/3659872341 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb674062c00 con 0x7fb6741029a0 2024-04-23T14:41:35.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.694+0000 7fb66b400640 1 -- 172.21.15.28:0/3659872341 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18407 (secure 0 0 0) 0x7fb6640a0d00 con 0x7fb6741029a0 2024-04-23T14:41:35.696 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:41:35.696 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":81,"fsid":"1ee6e996-017e-11ef-bc93-c7b262605968","created":"2024-04-23T14:32:34.581082+0000","modified":"2024-04-23T14:40:44.389733+0000","last_up_change":"2024-04-23T14:40:43.375846+0000","last_in_change":"2024-04-23T14:40:23.576939+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:36:27.698112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e0b427bc-be65-4543-99cc-2f53ad98ddad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6803","nonce":2993445336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6805","nonce":2993445336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6809","nonce":2993445336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6807","nonce":2993445336}]},"public_addr":"172.21.15.28:6803/2993445336","cluster_addr":"172.21.15.28:6805/2993445336","heartbeat_back_addr":"172.21.15.28:6809/2993445336","heartbeat_front_addr":"172.21.15.28:6807/2993445336","state":["exists","up"]},{"osd":1,"uuid":"ef527ec9-1bf3-4bb0-8df3-f5a015af557c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6801","nonce":4024093383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6803","nonce":4024093383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6807","nonce":4024093383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6805","nonce":4024093383}]},"public_addr":"172.21.15.38:6801/4024093383","cluster_addr":"172.21.15.38:6803/4024093383","heartbeat_back_addr":"172.21.15.38:6807/4024093383","heartbeat_front_addr":"172.21.15.38:6805/4024093383","state":["exists","up"]},{"osd":2,"uuid":"b33b23d9-adce-4c11-b304-2e117bdf6bfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6801","nonce":570739029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6803","nonce":570739029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6807","nonce":570739029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6805","nonce":570739029}]},"public_addr":"172.21.15.105:6801/570739029","cluster_addr":"172.21.15.105:6803/570739029","heartbeat_back_addr":"172.21.15.105:6807/570739029","heartbeat_front_addr":"172.21.15.105:6805/570739029","state":["exists","up"]},{"osd":3,"uuid":"28bd5608-b426-434e-9db1-bdaa9828bdb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6811","nonce":2480400247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6813","nonce":2480400247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6817","nonce":2480400247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6815","nonce":2480400247}]},"public_addr":"172.21.15.28:6811/2480400247","cluster_addr":"172.21.15.28:6813/2480400247","heartbeat_back_addr":"172.21.15.28:6817/2480400247","heartbeat_front_addr":"172.21.15.28:6815/2480400247","state":["exists","up"]},{"osd":4,"uuid":"e5d1c89b-025f-4d57-8807-59d75c787b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6809","nonce":3772189246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6811","nonce":3772189246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6815","nonce":3772189246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6813","nonce":3772189246}]},"public_addr":"172.21.15.38:6809/3772189246","cluster_addr":"172.21.15.38:6811/3772189246","heartbeat_back_addr":"172.21.15.38:6815/3772189246","heartbeat_front_addr":"172.21.15.38:6813/3772189246","state":["exists","up"]},{"osd":5,"uuid":"9cf9cf4c-aafd-4ed3-bfb1-0527710a0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6809","nonce":928755894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6811","nonce":928755894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6815","nonce":928755894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6813","nonce":928755894}]},"public_addr":"172.21.15.105:6809/928755894","cluster_addr":"172.21.15.105:6811/928755894","heartbeat_back_addr":"172.21.15.105:6815/928755894","heartbeat_front_addr":"172.21.15.105:6813/928755894","state":["exists","up"]},{"osd":6,"uuid":"304db525-d24d-4687-bfa0-f92bdf66d1e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6819","nonce":897979441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6821","nonce":897979441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6825","nonce":897979441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6823","nonce":897979441}]},"public_addr":"172.21.15.28:6819/897979441","cluster_addr":"172.21.15.28:6821/897979441","heartbeat_back_addr":"172.21.15.28:6825/897979441","heartbeat_front_addr":"172.21.15.28:6823/897979441","state":["exists","up"]},{"osd":7,"uuid":"08e01d59-03f3-47b1-b45f-35ec5a50992c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6817","nonce":2400251786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6819","nonce":2400251786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6823","nonce":2400251786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6821","nonce":2400251786}]},"public_addr":"172.21.15.38:6817/2400251786","cluster_addr":"172.21.15.38:6819/2400251786","heartbeat_back_addr":"172.21.15.38:6823/2400251786","heartbeat_front_addr":"172.21.15.38:6821/2400251786","state":["exists","up"]},{"osd":8,"uuid":"64ee1e95-0756-48b1-9c00-472d0c3cbd29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6817","nonce":1682312269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6819","nonce":1682312269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6823","nonce":1682312269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6821","nonce":1682312269}]},"public_addr":"172.21.15.105:6817/1682312269","cluster_addr":"172.21.15.105:6819/1682312269","heartbeat_back_addr":"172.21.15.105:6823/1682312269","heartbeat_front_addr":"172.21.15.105:6821/1682312269","state":["exists","up"]},{"osd":9,"uuid":"18f60893-3ffd-48c2-9db5-a67d35611d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6827","nonce":1292942516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6829","nonce":1292942516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6833","nonce":1292942516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6831","nonce":1292942516}]},"public_addr":"172.21.15.28:6827/1292942516","cluster_addr":"172.21.15.28:6829/1292942516","heartbeat_back_addr":"172.21.15.28:6833/1292942516","heartbeat_front_addr":"172.21.15.28:6831/1292942516","state":["exists","up"]},{"osd":10,"uuid":"76c06725-7040-4118-8894-fe808a87f999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6825","nonce":2444616468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6827","nonce":2444616468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6831","nonce":2444616468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6829","nonce":2444616468}]},"public_addr":"172.21.15.38:6825/2444616468","cluster_addr":"172.21.15.38:6827/2444616468","heartbeat_back_addr":"172.21.15.38:6831/2444616468","heartbeat_front_addr":"172.21.15.38:6829/2444616468","state":["exists","up"]},{"osd":11,"uuid":"e8fc9454-5540-4e2e-91bf-3ba251dbd1a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6825","nonce":3400299562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6827","nonce":3400299562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6831","nonce":3400299562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6829","nonce":3400299562}]},"public_addr":"172.21.15.105:6825/3400299562","cluster_addr":"172.21.15.105:6827/3400299562","heartbeat_back_addr":"172.21.15.105:6831/3400299562","heartbeat_front_addr":"172.21.15.105:6829/3400299562","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:35:33.615395+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:00.545293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:24.940211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:57.684887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:24.519061+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:49.010804+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:22.638478+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:49.559104+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:14.011771+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:47.859950+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:16.298445+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:41.619069+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/590495740":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/1163172512":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/4290328460":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/4145020172":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/720626958":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/948450262":"2024-04-24T14:33:07.576363+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:41:35.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.697+0000 7fb678a00640 1 -- 172.21.15.28:0/3659872341 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb6580a3c30 msgr2=0x7fb6580a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:35.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.697+0000 7fb678a00640 1 --2- 172.21.15.28:0/3659872341 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb6580a3c30 0x7fb6580a60f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb66002a9e0 tx=0x7fb6600315b0 comp rx=0 tx=0).stop 2024-04-23T14:41:35.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.697+0000 7fb678a00640 1 -- 172.21.15.28:0/3659872341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6741029a0 msgr2=0x7fb67419a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:35.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.697+0000 7fb678a00640 1 --2- 172.21.15.28:0/3659872341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6741029a0 0x7fb67419a420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb66400e9f0 tx=0x7fb66400eeb0 comp rx=0 tx=0).stop 2024-04-23T14:41:35.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.697+0000 7fb678a00640 1 -- 172.21.15.28:0/3659872341 shutdown_connections 2024-04-23T14:41:35.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.697+0000 7fb678a00640 1 --2- 172.21.15.28:0/3659872341 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb6580a3c30 0x7fb6580a60f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:35.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.697+0000 7fb678a00640 1 --2- 172.21.15.28:0/3659872341 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb6741052d0 0x7fb67419a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:35.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.697+0000 7fb678a00640 1 --2- 172.21.15.28:0/3659872341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb674107c00 0x7fb6741a19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:35.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.697+0000 7fb678a00640 1 --2- 172.21.15.28:0/3659872341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6741029a0 0x7fb67419a420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:35.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.697+0000 7fb678a00640 1 -- 172.21.15.28:0/3659872341 >> 172.21.15.28:0/3659872341 conn(0x7fb6740fc2d0 msgr2=0x7fb674102120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:35.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.697+0000 7fb678a00640 1 -- 172.21.15.28:0/3659872341 shutdown_connections 2024-04-23T14:41:35.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.697+0000 7fb678a00640 1 -- 172.21.15.28:0/3659872341 wait complete. 2024-04-23T14:41:35.717 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T14:36:27.698112+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T14:41:35.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T14:41:35.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.918+0000 7f9edfc00640 1 -- 172.21.15.28:0/1607852308 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ed8102050 msgr2=0x7f9ed8102470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:35.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.918+0000 7f9edfc00640 1 --2- 172.21.15.28:0/1607852308 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ed8102050 0x7f9ed8102470 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc009900 tx=0x7f9ecc02a3f0 comp rx=0 tx=0).stop 2024-04-23T14:41:35.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.918+0000 7f9edfc00640 1 -- 172.21.15.28:0/1607852308 shutdown_connections 2024-04-23T14:41:35.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.918+0000 7f9edfc00640 1 --2- 172.21.15.28:0/1607852308 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9ed8103c00 0x7f9ed810a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:35.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.918+0000 7f9edfc00640 1 --2- 172.21.15.28:0/1607852308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ed8103240 0x7f9ed81036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:35.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.918+0000 7f9edfc00640 1 --2- 172.21.15.28:0/1607852308 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ed8102050 0x7f9ed8102470 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:35.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.918+0000 7f9edfc00640 1 -- 172.21.15.28:0/1607852308 >> 172.21.15.28:0/1607852308 conn(0x7f9ed80fd5e0 msgr2=0x7f9ed80ffa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:35.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.918+0000 7f9edfc00640 1 -- 172.21.15.28:0/1607852308 shutdown_connections 2024-04-23T14:41:35.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.918+0000 7f9edfc00640 1 -- 172.21.15.28:0/1607852308 wait complete. 2024-04-23T14:41:35.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.919+0000 7f9edfc00640 1 Processor -- start 2024-04-23T14:41:35.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.919+0000 7f9edfc00640 1 -- start start 2024-04-23T14:41:35.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.919+0000 7f9edfc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ed8102050 0x7f9ed819a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:35.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.919+0000 7f9edfc00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9ed8103240 0x7f9ed819ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:35.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.919+0000 7f9edfc00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ed8103c00 0x7f9ed81a1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:35.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.919+0000 7f9edfc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed810dd30 con 0x7f9ed8102050 2024-04-23T14:41:35.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.919+0000 7f9edfc00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed810dbb0 con 0x7f9ed8103c00 2024-04-23T14:41:35.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.919+0000 7f9edfc00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed810deb0 con 0x7f9ed8103240 2024-04-23T14:41:35.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.919+0000 7f9edd200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ed8102050 0x7f9ed819a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:35.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.919+0000 7f9eddc00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ed8103c00 0x7f9ed81a1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:35.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.919+0000 7f9edd200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ed8102050 0x7f9ed819a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56628/0 (socket says 172.21.15.28:56628) 2024-04-23T14:41:35.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.919+0000 7f9eddc00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ed8103c00 0x7f9ed81a1b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:56270/0 (socket says 172.21.15.28:56270) 2024-04-23T14:41:35.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.919+0000 7f9edd200640 1 -- 172.21.15.28:0/1920468793 learned_addr learned my addr 172.21.15.28:0/1920468793 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:35.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.919+0000 7f9ed7e00640 1 --2- 172.21.15.28:0/1920468793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9ed8103240 0x7f9ed819ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:35.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.920+0000 7f9eddc00640 1 -- 172.21.15.28:0/1920468793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9ed8103240 msgr2=0x7f9ed819ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:35.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.920+0000 7f9eddc00640 1 --2- 172.21.15.28:0/1920468793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9ed8103240 0x7f9ed819ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:35.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.920+0000 7f9eddc00640 1 -- 172.21.15.28:0/1920468793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ed8102050 msgr2=0x7f9ed819a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:35.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.920+0000 7f9eddc00640 1 --2- 172.21.15.28:0/1920468793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ed8102050 0x7f9ed819a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:35.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.920+0000 7f9eddc00640 1 -- 172.21.15.28:0/1920468793 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ed81a21f0 con 0x7f9ed8103c00 2024-04-23T14:41:35.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.920+0000 7f9eddc00640 1 --2- 172.21.15.28:0/1920468793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ed8103c00 0x7f9ed81a1b80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9ed0009fd0 tx=0x7f9ed000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:35.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.921+0000 7f9ed5600640 1 -- 172.21.15.28:0/1920468793 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed000ce20 con 0x7f9ed8103c00 2024-04-23T14:41:35.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.921+0000 7f9ed5600640 1 -- 172.21.15.28:0/1920468793 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ed0005020 con 0x7f9ed8103c00 2024-04-23T14:41:35.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.921+0000 7f9ed5600640 1 -- 172.21.15.28:0/1920468793 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed0010720 con 0x7f9ed8103c00 2024-04-23T14:41:35.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.921+0000 7f9edfc00640 1 -- 172.21.15.28:0/1920468793 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ed81a24e0 con 0x7f9ed8103c00 2024-04-23T14:41:35.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.921+0000 7f9edfc00640 1 -- 172.21.15.28:0/1920468793 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ed81a2a30 con 0x7f9ed8103c00 2024-04-23T14:41:35.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.922+0000 7f9edfc00640 1 -- 172.21.15.28:0/1920468793 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ed81a2cf0 con 0x7f9ed8103c00 2024-04-23T14:41:35.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.924+0000 7f9ed5600640 1 -- 172.21.15.28:0/1920468793 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9ed0005540 con 0x7f9ed8103c00 2024-04-23T14:41:35.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.925+0000 7f9ed5600640 1 --2- 172.21.15.28:0/1920468793 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f9ec00a3bb0 0x7f9ec00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:35.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.925+0000 7f9ed5600640 1 -- 172.21.15.28:0/1920468793 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f9ed00d05d0 con 0x7f9ed8103c00 2024-04-23T14:41:35.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.926+0000 7f9edd200640 1 --2- 172.21.15.28:0/1920468793 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f9ec00a3bb0 0x7f9ec00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:35.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.926+0000 7f9edd200640 1 --2- 172.21.15.28:0/1920468793 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f9ec00a3bb0 0x7f9ec00a6070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc000c00 tx=0x7f9ecc0315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:35.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:35.929+0000 7f9ed5600640 1 -- 172.21.15.28:0/1920468793 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9ed009d3b0 con 0x7f9ed8103c00 2024-04-23T14:41:36.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.146+0000 7f9edfc00640 1 -- 172.21.15.28:0/1920468793 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9ed81a2ec0 con 0x7f9ed8103c00 2024-04-23T14:41:36.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.146+0000 7f9ed5600640 1 -- 172.21.15.28:0/1920468793 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f9ed009ccb0 con 0x7f9ed8103c00 2024-04-23T14:41:36.148 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-23T14:41:36.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.148+0000 7f9edfc00640 1 -- 172.21.15.28:0/1920468793 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f9ec00a3bb0 msgr2=0x7f9ec00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:36.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.148+0000 7f9edfc00640 1 --2- 172.21.15.28:0/1920468793 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f9ec00a3bb0 0x7f9ec00a6070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc000c00 tx=0x7f9ecc0315b0 comp rx=0 tx=0).stop 2024-04-23T14:41:36.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.148+0000 7f9edfc00640 1 -- 172.21.15.28:0/1920468793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ed8103c00 msgr2=0x7f9ed81a1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:36.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.148+0000 7f9edfc00640 1 --2- 172.21.15.28:0/1920468793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ed8103c00 0x7f9ed81a1b80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9ed0009fd0 tx=0x7f9ed000ef10 comp rx=0 tx=0).stop 2024-04-23T14:41:36.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.149+0000 7f9edfc00640 1 -- 172.21.15.28:0/1920468793 shutdown_connections 2024-04-23T14:41:36.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.149+0000 7f9edfc00640 1 --2- 172.21.15.28:0/1920468793 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f9ec00a3bb0 0x7f9ec00a6070 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:36.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.149+0000 7f9edfc00640 1 --2- 172.21.15.28:0/1920468793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ed8103c00 0x7f9ed81a1b80 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:36.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.149+0000 7f9edfc00640 1 --2- 172.21.15.28:0/1920468793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9ed8103240 0x7f9ed819ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:36.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.149+0000 7f9edfc00640 1 --2- 172.21.15.28:0/1920468793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ed8102050 0x7f9ed819a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:36.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.149+0000 7f9edfc00640 1 -- 172.21.15.28:0/1920468793 >> 172.21.15.28:0/1920468793 conn(0x7f9ed80fd5e0 msgr2=0x7f9ed8074230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:36.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.149+0000 7f9edfc00640 1 -- 172.21.15.28:0/1920468793 shutdown_connections 2024-04-23T14:41:36.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.149+0000 7f9edfc00640 1 -- 172.21.15.28:0/1920468793 wait complete. 2024-04-23T14:41:36.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-23T14:41:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.352+0000 7ffa3fe00640 1 -- 172.21.15.28:0/65481732 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa40105270 msgr2=0x7ffa401056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.352+0000 7ffa3fe00640 1 --2- 172.21.15.28:0/65481732 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa40105270 0x7ffa401056f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c009860 tx=0x7ffa2c02a340 comp rx=0 tx=0).stop 2024-04-23T14:41:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.352+0000 7ffa3fe00640 1 -- 172.21.15.28:0/65481732 shutdown_connections 2024-04-23T14:41:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.352+0000 7ffa3fe00640 1 --2- 172.21.15.28:0/65481732 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa40105270 0x7ffa401056f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.352+0000 7ffa3fe00640 1 --2- 172.21.15.28:0/65481732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa40105c30 0x7ffa4010c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.352+0000 7ffa3fe00640 1 --2- 172.21.15.28:0/65481732 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa40104080 0x7ffa401044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.352+0000 7ffa3fe00640 1 -- 172.21.15.28:0/65481732 >> 172.21.15.28:0/65481732 conn(0x7ffa400ff610 msgr2=0x7ffa40101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.352+0000 7ffa3fe00640 1 -- 172.21.15.28:0/65481732 shutdown_connections 2024-04-23T14:41:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.352+0000 7ffa3fe00640 1 -- 172.21.15.28:0/65481732 wait complete. 2024-04-23T14:41:36.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.353+0000 7ffa3fe00640 1 Processor -- start 2024-04-23T14:41:36.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.353+0000 7ffa3fe00640 1 -- start start 2024-04-23T14:41:36.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.354+0000 7ffa3fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa40104080 0x7ffa4019a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:36.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.354+0000 7ffa3fe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa40105270 0x7ffa4019a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:36.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.354+0000 7ffa3fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa40105c30 0x7ffa401a1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:36.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.354+0000 7ffa3fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4010fd10 con 0x7ffa40105c30 2024-04-23T14:41:36.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.354+0000 7ffa3fe00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4010fb90 con 0x7ffa40104080 2024-04-23T14:41:36.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.354+0000 7ffa3fe00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4010fe90 con 0x7ffa40105270 2024-04-23T14:41:36.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.354+0000 7ffa3ea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa40104080 0x7ffa4019a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:36.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.354+0000 7ffa3ea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa40104080 0x7ffa4019a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:56290/0 (socket says 172.21.15.28:56290) 2024-04-23T14:41:36.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.354+0000 7ffa3ea00640 1 -- 172.21.15.28:0/918526449 learned_addr learned my addr 172.21.15.28:0/918526449 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:36.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.354+0000 7ffa3f400640 1 --2- 172.21.15.28:0/918526449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa40105c30 0x7ffa401a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:36.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.354+0000 7ffa3e000640 1 --2- 172.21.15.28:0/918526449 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa40105270 0x7ffa4019a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:36.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.354+0000 7ffa3ea00640 1 -- 172.21.15.28:0/918526449 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa40105270 msgr2=0x7ffa4019a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:36.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.354+0000 7ffa3ea00640 1 --2- 172.21.15.28:0/918526449 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa40105270 0x7ffa4019a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:36.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.354+0000 7ffa3ea00640 1 -- 172.21.15.28:0/918526449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa40105c30 msgr2=0x7ffa401a1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:36.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.354+0000 7ffa3ea00640 1 --2- 172.21.15.28:0/918526449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa40105c30 0x7ffa401a1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.354+0000 7ffa3ea00640 1 -- 172.21.15.28:0/918526449 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa401a2000 con 0x7ffa40104080 2024-04-23T14:41:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.355+0000 7ffa3ea00640 1 --2- 172.21.15.28:0/918526449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa40104080 0x7ffa4019a3e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ffa3000e9f0 tx=0x7ffa3000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.355+0000 7ffa37400640 1 -- 172.21.15.28:0/918526449 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3000ccc0 con 0x7ffa40104080 2024-04-23T14:41:36.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.355+0000 7ffa37400640 1 -- 172.21.15.28:0/918526449 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa300035f0 con 0x7ffa40104080 2024-04-23T14:41:36.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.355+0000 7ffa37400640 1 -- 172.21.15.28:0/918526449 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3000ce60 con 0x7ffa40104080 2024-04-23T14:41:36.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.355+0000 7ffa3fe00640 1 -- 172.21.15.28:0/918526449 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa401a22f0 con 0x7ffa40104080 2024-04-23T14:41:36.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.355+0000 7ffa3fe00640 1 -- 172.21.15.28:0/918526449 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa401a2810 con 0x7ffa40104080 2024-04-23T14:41:36.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.356+0000 7ffa3fe00640 1 -- 172.21.15.28:0/918526449 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa4010b3a0 con 0x7ffa40104080 2024-04-23T14:41:36.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.358+0000 7ffa37400640 1 -- 172.21.15.28:0/918526449 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ffa30003b10 con 0x7ffa40104080 2024-04-23T14:41:36.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.359+0000 7ffa37400640 1 --2- 172.21.15.28:0/918526449 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ffa240a3c30 0x7ffa240a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:36.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.359+0000 7ffa37400640 1 -- 172.21.15.28:0/918526449 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ffa300d0230 con 0x7ffa40104080 2024-04-23T14:41:36.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.360+0000 7ffa3e000640 1 --2- 172.21.15.28:0/918526449 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ffa240a3c30 0x7ffa240a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.360+0000 7ffa3e000640 1 --2- 172.21.15.28:0/918526449 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ffa240a3c30 0x7ffa240a60f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c009830 tx=0x7ffa2c030550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:36.362 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:36 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2631356588' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:36.363 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:36 smithi038 ceph-mon[32348]: mds.? [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] up:boot 2024-04-23T14:41:36.363 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:36 smithi038 ceph-mon[32348]: fsmap 12 up:standby 2024-04-23T14:41:36.363 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:36 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T14:41:36.363 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:36 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:41:36.363 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:36 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T14:41:36.363 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:36 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3659872341' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:41:36.363 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:36 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1920468793' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:41:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.364+0000 7ffa37400640 1 -- 172.21.15.28:0/918526449 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffa3009d010 con 0x7ffa40104080 2024-04-23T14:41:36.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:36 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2631356588' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:36.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:36 smithi028 ceph-mon[27810]: mds.? [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] up:boot 2024-04-23T14:41:36.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:36 smithi028 ceph-mon[27810]: fsmap 12 up:standby 2024-04-23T14:41:36.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:36 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T14:41:36.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:36 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:41:36.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:36 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T14:41:36.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:36 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3659872341' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:41:36.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:36 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1920468793' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:41:36.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:36 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2631356588' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:36.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:36 smithi105 ceph-mon[32555]: mds.? [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] up:boot 2024-04-23T14:41:36.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:36 smithi105 ceph-mon[32555]: fsmap 12 up:standby 2024-04-23T14:41:36.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:36 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T14:41:36.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:36 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:41:36.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:36 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T14:41:36.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:36 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3659872341' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:41:36.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:36 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1920468793' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:41:36.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.574+0000 7ffa3fe00640 1 -- 172.21.15.28:0/918526449 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7ffa40062c00 con 0x7ffa40104080 2024-04-23T14:41:36.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.575+0000 7ffa37400640 1 -- 172.21.15.28:0/918526449 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7ffa3009c910 con 0x7ffa40104080 2024-04-23T14:41:36.576 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:41:36.576 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-04-23T14:41:36.576 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:41:36.576 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-04-23T14:41:36.576 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-04-23T14:41:36.577 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:41:36.577 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-04-23T14:41:36.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.578+0000 7ffa3fe00640 1 -- 172.21.15.28:0/918526449 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ffa240a3c30 msgr2=0x7ffa240a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:36.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.578+0000 7ffa3fe00640 1 --2- 172.21.15.28:0/918526449 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ffa240a3c30 0x7ffa240a60f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c009830 tx=0x7ffa2c030550 comp rx=0 tx=0).stop 2024-04-23T14:41:36.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.578+0000 7ffa3fe00640 1 -- 172.21.15.28:0/918526449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa40104080 msgr2=0x7ffa4019a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:36.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.578+0000 7ffa3fe00640 1 --2- 172.21.15.28:0/918526449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa40104080 0x7ffa4019a3e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ffa3000e9f0 tx=0x7ffa3000eeb0 comp rx=0 tx=0).stop 2024-04-23T14:41:36.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.578+0000 7ffa3fe00640 1 -- 172.21.15.28:0/918526449 shutdown_connections 2024-04-23T14:41:36.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.578+0000 7ffa3fe00640 1 --2- 172.21.15.28:0/918526449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa40105c30 0x7ffa401a1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:36.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.578+0000 7ffa3fe00640 1 --2- 172.21.15.28:0/918526449 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa40105270 0x7ffa4019a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:36.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.578+0000 7ffa3fe00640 1 --2- 172.21.15.28:0/918526449 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ffa240a3c30 0x7ffa240a60f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:36.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.578+0000 7ffa3fe00640 1 --2- 172.21.15.28:0/918526449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa40104080 0x7ffa4019a3e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:36.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.578+0000 7ffa3fe00640 1 -- 172.21.15.28:0/918526449 >> 172.21.15.28:0/918526449 conn(0x7ffa400ff610 msgr2=0x7ffa40108570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:36.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.578+0000 7ffa3fe00640 1 -- 172.21.15.28:0/918526449 shutdown_connections 2024-04-23T14:41:36.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.578+0000 7ffa3fe00640 1 -- 172.21.15.28:0/918526449 wait complete. 2024-04-23T14:41:36.598 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-23T14:41:36.598 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-23T14:41:36.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.790+0000 7fdf6da00640 1 -- 172.21.15.28:0/1554205008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf681053f0 msgr2=0x7fdf681077e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:36.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.790+0000 7fdf6da00640 1 --2- 172.21.15.28:0/1554205008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf681053f0 0x7fdf681077e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fdf54009ca0 tx=0x7fdf5402a8c0 comp rx=0 tx=0).stop 2024-04-23T14:41:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.790+0000 7fdf6da00640 1 -- 172.21.15.28:0/1554205008 shutdown_connections 2024-04-23T14:41:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.790+0000 7fdf6da00640 1 --2- 172.21.15.28:0/1554205008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf68107d20 0x7fdf6810a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.790+0000 7fdf6da00640 1 --2- 172.21.15.28:0/1554205008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf681053f0 0x7fdf681077e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.790+0000 7fdf6da00640 1 --2- 172.21.15.28:0/1554205008 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf68102ac0 0x7fdf68104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.790+0000 7fdf6da00640 1 -- 172.21.15.28:0/1554205008 >> 172.21.15.28:0/1554205008 conn(0x7fdf680fc4b0 msgr2=0x7fdf680fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.790+0000 7fdf6da00640 1 -- 172.21.15.28:0/1554205008 shutdown_connections 2024-04-23T14:41:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.790+0000 7fdf6da00640 1 -- 172.21.15.28:0/1554205008 wait complete. 2024-04-23T14:41:36.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.791+0000 7fdf6da00640 1 Processor -- start 2024-04-23T14:41:36.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.791+0000 7fdf6da00640 1 -- start start 2024-04-23T14:41:36.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.792+0000 7fdf6da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf68102ac0 0x7fdf68198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:36.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.792+0000 7fdf6da00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf681053f0 0x7fdf681988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:36.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.792+0000 7fdf6da00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf68107d20 0x7fdf6819f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:36.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.792+0000 7fdf6da00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6810dd20 con 0x7fdf68102ac0 2024-04-23T14:41:36.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.792+0000 7fdf6da00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6810dba0 con 0x7fdf681053f0 2024-04-23T14:41:36.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.792+0000 7fdf6da00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6810dea0 con 0x7fdf68107d20 2024-04-23T14:41:36.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.792+0000 7fdf66a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf68102ac0 0x7fdf68198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:36.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.792+0000 7fdf67400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf68107d20 0x7fdf6819f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:36.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.792+0000 7fdf66000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf681053f0 0x7fdf681988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:36.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.792+0000 7fdf67400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf68107d20 0x7fdf6819f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:34424/0 (socket says 172.21.15.28:34424) 2024-04-23T14:41:36.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.792+0000 7fdf66000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf681053f0 0x7fdf681988c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:56316/0 (socket says 172.21.15.28:56316) 2024-04-23T14:41:36.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.792+0000 7fdf67400640 1 -- 172.21.15.28:0/2202409923 learned_addr learned my addr 172.21.15.28:0/2202409923 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:36.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.792+0000 7fdf66a00640 1 --2- 172.21.15.28:0/2202409923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf68102ac0 0x7fdf68198380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56654/0 (socket says 172.21.15.28:56654) 2024-04-23T14:41:36.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.792+0000 7fdf67400640 1 -- 172.21.15.28:0/2202409923 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf681053f0 msgr2=0x7fdf681988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:36.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.792+0000 7fdf67400640 1 --2- 172.21.15.28:0/2202409923 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf681053f0 0x7fdf681988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:36.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.792+0000 7fdf67400640 1 -- 172.21.15.28:0/2202409923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf68102ac0 msgr2=0x7fdf68198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:36.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.792+0000 7fdf67400640 1 --2- 172.21.15.28:0/2202409923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf68102ac0 0x7fdf68198380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:36.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.792+0000 7fdf67400640 1 -- 172.21.15.28:0/2202409923 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf6819ffa0 con 0x7fdf68107d20 2024-04-23T14:41:36.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.793+0000 7fdf67400640 1 --2- 172.21.15.28:0/2202409923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf68107d20 0x7fdf6819f930 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdf6000e970 tx=0x7fdf6000ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:36.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.793+0000 7fdf5f400640 1 -- 172.21.15.28:0/2202409923 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf6000ccc0 con 0x7fdf68107d20 2024-04-23T14:41:36.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.793+0000 7fdf5f400640 1 -- 172.21.15.28:0/2202409923 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf6000ce60 con 0x7fdf68107d20 2024-04-23T14:41:36.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.793+0000 7fdf6da00640 1 -- 172.21.15.28:0/2202409923 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf68100910 con 0x7fdf68107d20 2024-04-23T14:41:36.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.793+0000 7fdf5f400640 1 -- 172.21.15.28:0/2202409923 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf60010490 con 0x7fdf68107d20 2024-04-23T14:41:36.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.793+0000 7fdf6da00640 1 -- 172.21.15.28:0/2202409923 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf68100e80 con 0x7fdf68107d20 2024-04-23T14:41:36.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.794+0000 7fdf6da00640 1 -- 172.21.15.28:0/2202409923 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf680fdfa0 con 0x7fdf68107d20 2024-04-23T14:41:36.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.796+0000 7fdf5f400640 1 -- 172.21.15.28:0/2202409923 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fdf60005020 con 0x7fdf68107d20 2024-04-23T14:41:36.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.797+0000 7fdf5f400640 1 --2- 172.21.15.28:0/2202409923 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fdf4c0a3c30 0x7fdf4c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:36.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.798+0000 7fdf5f400640 1 -- 172.21.15.28:0/2202409923 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fdf600d0370 con 0x7fdf68107d20 2024-04-23T14:41:36.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.798+0000 7fdf66a00640 1 --2- 172.21.15.28:0/2202409923 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fdf4c0a3c30 0x7fdf4c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:36.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.798+0000 7fdf66a00640 1 --2- 172.21.15.28:0/2202409923 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fdf4c0a3c30 0x7fdf4c0a60f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdf580095f0 tx=0x7fdf58005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:36.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:36.802+0000 7fdf5f400640 1 -- 172.21.15.28:0/2202409923 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdf6009d150 con 0x7fdf68107d20 2024-04-23T14:41:37.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:37.016+0000 7fdf6da00640 1 -- 172.21.15.28:0/2202409923 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fdf681022c0 con 0x7fdf68107d20 2024-04-23T14:41:37.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:37 smithi038 ceph-mon[32348]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:37.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:37 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/918526449' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-23T14:41:37.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:37 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:37.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:37 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:37.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:37 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:37.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:37 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:37.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:37 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:37.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:37 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2202409923' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-23T14:41:37.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:37 smithi038 ceph-mon[32348]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-23T14:41:37.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:37 smithi028 ceph-mon[27810]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:37.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:37 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/918526449' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-23T14:41:37.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:37 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:37.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:37 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:37.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:37 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:37.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:37 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:37.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:37 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:37.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:37 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2202409923' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-23T14:41:37.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:37 smithi028 ceph-mon[27810]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-23T14:41:37.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:37 smithi105 ceph-mon[32555]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:37.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:37 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/918526449' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-23T14:41:37.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:37 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:37.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:37 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:37.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:37 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:37.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:37 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:37.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:37 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:37.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:37 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2202409923' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-23T14:41:37.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:37 smithi105 ceph-mon[32555]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-23T14:41:37.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:37.974+0000 7fdf5f400640 1 -- 172.21.15.28:0/2202409923 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7fdf6009ca50 con 0x7fdf68107d20 2024-04-23T14:41:37.976 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-04-23T14:41:37.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:37.979+0000 7fdf6da00640 1 -- 172.21.15.28:0/2202409923 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fdf4c0a3c30 msgr2=0x7fdf4c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:37.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:37.979+0000 7fdf6da00640 1 --2- 172.21.15.28:0/2202409923 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fdf4c0a3c30 0x7fdf4c0a60f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdf580095f0 tx=0x7fdf58005f90 comp rx=0 tx=0).stop 2024-04-23T14:41:37.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:37.979+0000 7fdf6da00640 1 -- 172.21.15.28:0/2202409923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf68107d20 msgr2=0x7fdf6819f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:37.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:37.979+0000 7fdf6da00640 1 --2- 172.21.15.28:0/2202409923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf68107d20 0x7fdf6819f930 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdf6000e970 tx=0x7fdf6000ed70 comp rx=0 tx=0).stop 2024-04-23T14:41:37.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:37.985+0000 7fdf6da00640 1 -- 172.21.15.28:0/2202409923 shutdown_connections 2024-04-23T14:41:37.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:37.985+0000 7fdf6da00640 1 --2- 172.21.15.28:0/2202409923 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fdf4c0a3c30 0x7fdf4c0a60f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:37.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:37.985+0000 7fdf6da00640 1 --2- 172.21.15.28:0/2202409923 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf681053f0 0x7fdf681988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:37.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:37.985+0000 7fdf6da00640 1 --2- 172.21.15.28:0/2202409923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf68107d20 0x7fdf6819f930 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:37.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:37.985+0000 7fdf6da00640 1 --2- 172.21.15.28:0/2202409923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf68102ac0 0x7fdf68198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:37.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:37.985+0000 7fdf6da00640 1 -- 172.21.15.28:0/2202409923 >> 172.21.15.28:0/2202409923 conn(0x7fdf680fc4b0 msgr2=0x7fdf68103520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:37.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:37.985+0000 7fdf6da00640 1 -- 172.21.15.28:0/2202409923 shutdown_connections 2024-04-23T14:41:37.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:37.985+0000 7fdf6da00640 1 -- 172.21.15.28:0/2202409923 wait complete. 2024-04-23T14:41:38.007 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-23T14:41:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.205+0000 7f0a0e000640 1 -- 172.21.15.28:0/4072207022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0a08104260 msgr2=0x7f0a08104680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.205+0000 7f0a0e000640 1 --2- 172.21.15.28:0/4072207022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0a08104260 0x7f0a08104680 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f09f8009900 tx=0x7f09f802a3f0 comp rx=0 tx=0).stop 2024-04-23T14:41:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.206+0000 7f0a0e000640 1 -- 172.21.15.28:0/4072207022 shutdown_connections 2024-04-23T14:41:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.206+0000 7f0a0e000640 1 --2- 172.21.15.28:0/4072207022 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a08105e10 0x7f0a0810c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.206+0000 7f0a0e000640 1 --2- 172.21.15.28:0/4072207022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a08105450 0x7f0a081058d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.206+0000 7f0a0e000640 1 --2- 172.21.15.28:0/4072207022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0a08104260 0x7f0a08104680 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.206+0000 7f0a0e000640 1 -- 172.21.15.28:0/4072207022 >> 172.21.15.28:0/4072207022 conn(0x7f0a080ff7b0 msgr2=0x7f0a08101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.206+0000 7f0a0e000640 1 -- 172.21.15.28:0/4072207022 shutdown_connections 2024-04-23T14:41:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.206+0000 7f0a0e000640 1 -- 172.21.15.28:0/4072207022 wait complete. 2024-04-23T14:41:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.207+0000 7f0a0e000640 1 Processor -- start 2024-04-23T14:41:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.207+0000 7f0a0e000640 1 -- start start 2024-04-23T14:41:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.207+0000 7f0a0e000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0a08104260 0x7f0a0819a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.207+0000 7f0a0e000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a08105450 0x7f0a0819aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.207+0000 7f0a0e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a08105e10 0x7f0a081a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.207+0000 7f0a0e000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0a0810ff40 con 0x7f0a08105e10 2024-04-23T14:41:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.207+0000 7f0a0e000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0a0810fdc0 con 0x7f0a08105450 2024-04-23T14:41:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.207+0000 7f0a0e000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0a081100c0 con 0x7f0a08104260 2024-04-23T14:41:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.208+0000 7f0a06a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a08105450 0x7f0a0819aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.208+0000 7f0a06a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a08105450 0x7f0a0819aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48264/0 (socket says 172.21.15.28:48264) 2024-04-23T14:41:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.208+0000 7f0a06a00640 1 -- 172.21.15.28:0/3990763761 learned_addr learned my addr 172.21.15.28:0/3990763761 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:38.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.208+0000 7f0a07400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0a08104260 0x7f0a0819a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:38.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.208+0000 7f0a07e00640 1 --2- 172.21.15.28:0/3990763761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a08105e10 0x7f0a081a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:38.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.208+0000 7f0a06a00640 1 -- 172.21.15.28:0/3990763761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0a08104260 msgr2=0x7f0a0819a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:38.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.208+0000 7f0a06a00640 1 --2- 172.21.15.28:0/3990763761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0a08104260 0x7f0a0819a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:38.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.208+0000 7f0a06a00640 1 -- 172.21.15.28:0/3990763761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a08105e10 msgr2=0x7f0a081a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:38.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.208+0000 7f0a06a00640 1 --2- 172.21.15.28:0/3990763761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a08105e10 0x7f0a081a1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:38.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.208+0000 7f0a06a00640 1 -- 172.21.15.28:0/3990763761 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a081a21b0 con 0x7f0a08105450 2024-04-23T14:41:38.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.208+0000 7f0a07e00640 1 --2- 172.21.15.28:0/3990763761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a08105e10 0x7f0a081a1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:41:38.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.208+0000 7f0a06a00640 1 --2- 172.21.15.28:0/3990763761 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a08105450 0x7f0a0819aad0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f09f400e9f0 tx=0x7f09f400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:38.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.208+0000 7f09ffe00640 1 -- 172.21.15.28:0/3990763761 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09f400cd80 con 0x7f0a08105450 2024-04-23T14:41:38.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.209+0000 7f0a0e000640 1 -- 172.21.15.28:0/3990763761 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a081a24a0 con 0x7f0a08105450 2024-04-23T14:41:38.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.209+0000 7f09ffe00640 1 -- 172.21.15.28:0/3990763761 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09f4010910 con 0x7f0a08105450 2024-04-23T14:41:38.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.209+0000 7f09ffe00640 1 -- 172.21.15.28:0/3990763761 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09f40188f0 con 0x7f0a08105450 2024-04-23T14:41:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.209+0000 7f0a0e000640 1 -- 172.21.15.28:0/3990763761 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a081a29f0 con 0x7f0a08105450 2024-04-23T14:41:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.210+0000 7f0a0e000640 1 -- 172.21.15.28:0/3990763761 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a0810b580 con 0x7f0a08105450 2024-04-23T14:41:38.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.213+0000 7f09ffe00640 1 -- 172.21.15.28:0/3990763761 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f09f4010ac0 con 0x7f0a08105450 2024-04-23T14:41:38.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.214+0000 7f09ffe00640 1 --2- 172.21.15.28:0/3990763761 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f09ec0a3bb0 0x7f09ec0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:38.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.214+0000 7f09ffe00640 1 -- 172.21.15.28:0/3990763761 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7f09f40d05b0 con 0x7f0a08105450 2024-04-23T14:41:38.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.214+0000 7f0a07400640 1 --2- 172.21.15.28:0/3990763761 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f09ec0a3bb0 0x7f09ec0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:38.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.215+0000 7f0a07400640 1 --2- 172.21.15.28:0/3990763761 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f09ec0a3bb0 0x7f09ec0a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f09f802a9e0 tx=0x7f09f80315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:38.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.219+0000 7f09ffe00640 1 -- 172.21.15.28:0/3990763761 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f09f409d220 con 0x7f0a08105450 2024-04-23T14:41:38.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.433+0000 7f0a0e000640 1 -- 172.21.15.28:0/3990763761 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f0a08062c00 con 0x7f0a08105450 2024-04-23T14:41:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:38.995+0000 7f09ffe00640 1 -- 172.21.15.28:0/3990763761 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v83) ==== 170+0+0 (secure 0 0 0) 0x7f09f409cb20 con 0x7f0a08105450 2024-04-23T14:41:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-04-23T14:41:39.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.007+0000 7f0a0e000640 1 -- 172.21.15.28:0/3990763761 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f09ec0a3bb0 msgr2=0x7f09ec0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:39.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.007+0000 7f0a0e000640 1 --2- 172.21.15.28:0/3990763761 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f09ec0a3bb0 0x7f09ec0a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f09f802a9e0 tx=0x7f09f80315b0 comp rx=0 tx=0).stop 2024-04-23T14:41:39.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.007+0000 7f0a0e000640 1 -- 172.21.15.28:0/3990763761 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a08105450 msgr2=0x7f0a0819aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:39.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.007+0000 7f0a0e000640 1 --2- 172.21.15.28:0/3990763761 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a08105450 0x7f0a0819aad0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f09f400e9f0 tx=0x7f09f400eeb0 comp rx=0 tx=0).stop 2024-04-23T14:41:39.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.008+0000 7f0a0e000640 1 -- 172.21.15.28:0/3990763761 shutdown_connections 2024-04-23T14:41:39.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.008+0000 7f0a0e000640 1 --2- 172.21.15.28:0/3990763761 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f09ec0a3bb0 0x7f09ec0a6070 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:39.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.008+0000 7f0a0e000640 1 --2- 172.21.15.28:0/3990763761 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a08105450 0x7f0a0819aad0 secure :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f09f400e9f0 tx=0x7f09f400eeb0 comp rx=0 tx=0).stop 2024-04-23T14:41:39.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.008+0000 7f0a0e000640 1 --2- 172.21.15.28:0/3990763761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a08105e10 0x7f0a081a1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:39.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.008+0000 7f0a0e000640 1 --2- 172.21.15.28:0/3990763761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0a08104260 0x7f0a0819a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:39.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.008+0000 7f0a0e000640 1 -- 172.21.15.28:0/3990763761 >> 172.21.15.28:0/3990763761 conn(0x7f0a080ff7b0 msgr2=0x7f0a08108750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:39.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.008+0000 7f0a0e000640 1 -- 172.21.15.28:0/3990763761 shutdown_connections 2024-04-23T14:41:39.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.009+0000 7f0a0e000640 1 -- 172.21.15.28:0/3990763761 wait complete. 2024-04-23T14:41:39.041 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-23T14:41:39.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.225+0000 7f4519c00640 1 -- 172.21.15.28:0/2118535161 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4514104280 msgr2=0x7f45141046a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:39.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.225+0000 7f4519c00640 1 --2- 172.21.15.28:0/2118535161 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4514104280 0x7f45141046a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4504009900 tx=0x7f450402a3f0 comp rx=0 tx=0).stop 2024-04-23T14:41:39.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.225+0000 7f4519c00640 1 -- 172.21.15.28:0/2118535161 shutdown_connections 2024-04-23T14:41:39.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.225+0000 7f4519c00640 1 --2- 172.21.15.28:0/2118535161 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4514105e30 0x7f451410c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:39.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.225+0000 7f4519c00640 1 --2- 172.21.15.28:0/2118535161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4514105470 0x7f45141058f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:39.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.225+0000 7f4519c00640 1 --2- 172.21.15.28:0/2118535161 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4514104280 0x7f45141046a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:39.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.225+0000 7f4519c00640 1 -- 172.21.15.28:0/2118535161 >> 172.21.15.28:0/2118535161 conn(0x7f45140ff7f0 msgr2=0x7f4514101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:39.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.225+0000 7f4519c00640 1 -- 172.21.15.28:0/2118535161 shutdown_connections 2024-04-23T14:41:39.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.225+0000 7f4519c00640 1 -- 172.21.15.28:0/2118535161 wait complete. 2024-04-23T14:41:39.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.226+0000 7f4519c00640 1 Processor -- start 2024-04-23T14:41:39.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.226+0000 7f4519c00640 1 -- start start 2024-04-23T14:41:39.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.226+0000 7f4519c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4514104280 0x7f451419a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:39.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.226+0000 7f4519c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4514105470 0x7f451419aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:39.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.226+0000 7f4519c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4514105e30 0x7f45141a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:39.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.226+0000 7f4519c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f451410ff10 con 0x7f4514105470 2024-04-23T14:41:39.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.226+0000 7f4519c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f451410fd90 con 0x7f4514105e30 2024-04-23T14:41:39.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.226+0000 7f4519c00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4514110090 con 0x7f4514104280 2024-04-23T14:41:39.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.227+0000 7f4512a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4514104280 0x7f451419a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:39.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.227+0000 7f4513400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4514105e30 0x7f45141a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:39.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.227+0000 7f4512a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4514104280 0x7f451419a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:59928/0 (socket says 172.21.15.28:59928) 2024-04-23T14:41:39.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.227+0000 7f4512a00640 1 -- 172.21.15.28:0/2914445592 learned_addr learned my addr 172.21.15.28:0/2914445592 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:39.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.227+0000 7f4512000640 1 --2- 172.21.15.28:0/2914445592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4514105470 0x7f451419aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:39.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.227+0000 7f4512a00640 1 -- 172.21.15.28:0/2914445592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4514105e30 msgr2=0x7f45141a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:39.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.227+0000 7f4512a00640 1 --2- 172.21.15.28:0/2914445592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4514105e30 0x7f45141a1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:39.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.227+0000 7f4512a00640 1 -- 172.21.15.28:0/2914445592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4514105470 msgr2=0x7f451419aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:39.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.227+0000 7f4512a00640 1 --2- 172.21.15.28:0/2914445592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4514105470 0x7f451419aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:39.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.227+0000 7f4512a00640 1 -- 172.21.15.28:0/2914445592 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45141a21b0 con 0x7f4514104280 2024-04-23T14:41:39.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.227+0000 7f4513400640 1 --2- 172.21.15.28:0/2914445592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4514105e30 0x7f45141a1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:41:39.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.227+0000 7f4512a00640 1 --2- 172.21.15.28:0/2914445592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4514104280 0x7f451419a590 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4504031610 tx=0x7f4504004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:39.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.228+0000 7f450b400640 1 -- 172.21.15.28:0/2914445592 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f450403f030 con 0x7f4514104280 2024-04-23T14:41:39.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.228+0000 7f4519c00640 1 -- 172.21.15.28:0/2914445592 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45141a2440 con 0x7f4514104280 2024-04-23T14:41:39.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.228+0000 7f450b400640 1 -- 172.21.15.28:0/2914445592 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4504003ad0 con 0x7f4514104280 2024-04-23T14:41:39.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.228+0000 7f450b400640 1 -- 172.21.15.28:0/2914445592 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4504035070 con 0x7f4514104280 2024-04-23T14:41:39.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.228+0000 7f4519c00640 1 -- 172.21.15.28:0/2914445592 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45141a2930 con 0x7f4514104280 2024-04-23T14:41:39.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.229+0000 7f4519c00640 1 -- 172.21.15.28:0/2914445592 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f451410b670 con 0x7f4514104280 2024-04-23T14:41:39.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.232+0000 7f450b400640 1 -- 172.21.15.28:0/2914445592 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4504042030 con 0x7f4514104280 2024-04-23T14:41:39.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.234+0000 7f450b400640 1 --2- 172.21.15.28:0/2914445592 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f44f80a3bb0 0x7f44f80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:39.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.234+0000 7f450b400640 1 -- 172.21.15.28:0/2914445592 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7f45040f0860 con 0x7f4514104280 2024-04-23T14:41:39.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.234+0000 7f4512000640 1 --2- 172.21.15.28:0/2914445592 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f44f80a3bb0 0x7f44f80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:39.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.235+0000 7f4512000640 1 --2- 172.21.15.28:0/2914445592 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f44f80a3bb0 0x7f44f80a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f451419baa0 tx=0x7f4500005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:39.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.238+0000 7f450b400640 1 -- 172.21.15.28:0/2914445592 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f45040bcd00 con 0x7f4514104280 2024-04-23T14:41:39.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:38 smithi105 ceph-mon[32555]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:39.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:38 smithi105 ceph-mon[32555]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-23T14:41:39.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:38 smithi105 ceph-mon[32555]: osdmap e82: 12 total, 12 up, 12 in 2024-04-23T14:41:39.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:38 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:39.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:38 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:39.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:38 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:39.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:38 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:41:39.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:38 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:39.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:38 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:41:39.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:38 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3990763761' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-23T14:41:39.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:38 smithi105 ceph-mon[32555]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-23T14:41:39.353 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:38 smithi038 ceph-mon[32348]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:39.353 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:38 smithi038 ceph-mon[32348]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-23T14:41:39.353 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:38 smithi038 ceph-mon[32348]: osdmap e82: 12 total, 12 up, 12 in 2024-04-23T14:41:39.353 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:38 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:39.353 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:38 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:39.354 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:38 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:39.354 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:38 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:41:39.354 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:38 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:39.354 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:38 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:41:39.354 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:38 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3990763761' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-23T14:41:39.354 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:38 smithi038 ceph-mon[32348]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-23T14:41:39.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:38 smithi028 ceph-mon[27810]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:39.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:38 smithi028 ceph-mon[27810]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-23T14:41:39.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:38 smithi028 ceph-mon[27810]: osdmap e82: 12 total, 12 up, 12 in 2024-04-23T14:41:39.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:38 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:39.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:38 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:39.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:38 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:39.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:38 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:41:39.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:38 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:39.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:38 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:41:39.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:38 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3990763761' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-23T14:41:39.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:38 smithi028 ceph-mon[27810]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-23T14:41:39.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.492+0000 7f4519c00640 1 -- 172.21.15.28:0/2914445592 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f451402d620 con 0x7f4514104280 2024-04-23T14:41:39.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.993+0000 7f450b400640 1 -- 172.21.15.28:0/2914445592 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f45040c1000 con 0x7f4514104280 2024-04-23T14:41:39.994 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 2 and data pool 3 2024-04-23T14:41:39.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.997+0000 7f4519c00640 1 -- 172.21.15.28:0/2914445592 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f44f80a3bb0 msgr2=0x7f44f80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:39.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.997+0000 7f4519c00640 1 --2- 172.21.15.28:0/2914445592 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f44f80a3bb0 0x7f44f80a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f451419baa0 tx=0x7f4500005f90 comp rx=0 tx=0).stop 2024-04-23T14:41:39.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.997+0000 7f4519c00640 1 -- 172.21.15.28:0/2914445592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4514104280 msgr2=0x7f451419a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:39.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.997+0000 7f4519c00640 1 --2- 172.21.15.28:0/2914445592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4514104280 0x7f451419a590 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4504031610 tx=0x7f4504004270 comp rx=0 tx=0).stop 2024-04-23T14:41:39.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.998+0000 7f4519c00640 1 -- 172.21.15.28:0/2914445592 shutdown_connections 2024-04-23T14:41:39.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.999+0000 7f4519c00640 1 --2- 172.21.15.28:0/2914445592 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f44f80a3bb0 0x7f44f80a6070 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:39.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.999+0000 7f4519c00640 1 --2- 172.21.15.28:0/2914445592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4514105e30 0x7f45141a1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:39.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.999+0000 7f4519c00640 1 --2- 172.21.15.28:0/2914445592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4514105470 0x7f451419aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:39.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.999+0000 7f4519c00640 1 --2- 172.21.15.28:0/2914445592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4514104280 0x7f451419a590 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:39.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:39.999+0000 7f4519c00640 1 -- 172.21.15.28:0/2914445592 >> 172.21.15.28:0/2914445592 conn(0x7f45140ff7f0 msgr2=0x7f4514108840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:40.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.000+0000 7f4519c00640 1 -- 172.21.15.28:0/2914445592 shutdown_connections 2024-04-23T14:41:40.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.001+0000 7f4519c00640 1 -- 172.21.15.28:0/2914445592 wait complete. 2024-04-23T14:41:40.011 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:40 smithi028 ceph-mon[27810]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-23T14:41:40.011 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:40 smithi028 ceph-mon[27810]: osdmap e83: 12 total, 12 up, 12 in 2024-04-23T14:41:40.011 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:40 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2914445592' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-23T14:41:40.011 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:40 smithi028 ceph-mon[27810]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-23T14:41:40.011 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:40 smithi028 ceph-mon[27810]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-23T14:41:40.042 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:41:40.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.221+0000 7fba09800640 1 -- 172.21.15.28:0/3322580842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba04101290 msgr2=0x7fba041016f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:40.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.221+0000 7fba09800640 1 --2- 172.21.15.28:0/3322580842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba04101290 0x7fba041016f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec009900 tx=0x7fb9ec02a480 comp rx=0 tx=0).stop 2024-04-23T14:41:40.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.221+0000 7fba09800640 1 -- 172.21.15.28:0/3322580842 shutdown_connections 2024-04-23T14:41:40.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.221+0000 7fba09800640 1 --2- 172.21.15.28:0/3322580842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fba04101c30 0x7fba041020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:40.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.221+0000 7fba09800640 1 --2- 172.21.15.28:0/3322580842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba04101290 0x7fba041016f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:40.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.221+0000 7fba09800640 1 --2- 172.21.15.28:0/3322580842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba04100930 0x7fba04100d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:40.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.221+0000 7fba09800640 1 -- 172.21.15.28:0/3322580842 >> 172.21.15.28:0/3322580842 conn(0x7fba040fc4b0 msgr2=0x7fba040fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:40.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.221+0000 7fba09800640 1 -- 172.21.15.28:0/3322580842 shutdown_connections 2024-04-23T14:41:40.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.221+0000 7fba09800640 1 -- 172.21.15.28:0/3322580842 wait complete. 2024-04-23T14:41:40.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.222+0000 7fba09800640 1 Processor -- start 2024-04-23T14:41:40.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.222+0000 7fba09800640 1 -- start start 2024-04-23T14:41:40.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.222+0000 7fba09800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba04100930 0x7fba0419a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:40.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.222+0000 7fba09800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fba04101290 0x7fba0419aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:40.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.223+0000 7fba09800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba04101c30 0x7fba041a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:40.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.223+0000 7fba09800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fba0410fef0 con 0x7fba04100930 2024-04-23T14:41:40.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.223+0000 7fba09800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fba0410fd70 con 0x7fba04101c30 2024-04-23T14:41:40.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.223+0000 7fba09800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fba04110070 con 0x7fba04101290 2024-04-23T14:41:40.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.223+0000 7fba02a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba04100930 0x7fba0419a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:40.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.223+0000 7fba02a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba04100930 0x7fba0419a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35354/0 (socket says 172.21.15.28:35354) 2024-04-23T14:41:40.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.223+0000 7fba02a00640 1 -- 172.21.15.28:0/701789592 learned_addr learned my addr 172.21.15.28:0/701789592 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:40.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.223+0000 7fb9fbe00640 1 --2- 172.21.15.28:0/701789592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fba04101290 0x7fba0419aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:40.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.223+0000 7fba03400640 1 --2- 172.21.15.28:0/701789592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba04101c30 0x7fba041a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:40.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.223+0000 7fba02a00640 1 -- 172.21.15.28:0/701789592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fba04101290 msgr2=0x7fba0419aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:40.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.223+0000 7fba02a00640 1 --2- 172.21.15.28:0/701789592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fba04101290 0x7fba0419aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:40.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.223+0000 7fba02a00640 1 -- 172.21.15.28:0/701789592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba04101c30 msgr2=0x7fba041a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:40.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.223+0000 7fba02a00640 1 --2- 172.21.15.28:0/701789592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba04101c30 0x7fba041a1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:40.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.223+0000 7fba02a00640 1 -- 172.21.15.28:0/701789592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba041a2180 con 0x7fba04100930 2024-04-23T14:41:40.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.223+0000 7fba03400640 1 --2- 172.21.15.28:0/701789592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba04101c30 0x7fba041a1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:41:40.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.224+0000 7fba02a00640 1 --2- 172.21.15.28:0/701789592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba04100930 0x7fba0419a560 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb9f400d950 tx=0x7fb9f400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:40.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.224+0000 7fb9fb400640 1 -- 172.21.15.28:0/701789592 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f4017070 con 0x7fba04100930 2024-04-23T14:41:40.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.224+0000 7fb9fb400640 1 -- 172.21.15.28:0/701789592 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9f4009c30 con 0x7fba04100930 2024-04-23T14:41:40.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.224+0000 7fba09800640 1 -- 172.21.15.28:0/701789592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba041a2470 con 0x7fba04100930 2024-04-23T14:41:40.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.224+0000 7fb9fb400640 1 -- 172.21.15.28:0/701789592 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f400f6b0 con 0x7fba04100930 2024-04-23T14:41:40.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.224+0000 7fba09800640 1 -- 172.21.15.28:0/701789592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba041a2990 con 0x7fba04100930 2024-04-23T14:41:40.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.225+0000 7fba09800640 1 -- 172.21.15.28:0/701789592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba0410b7c0 con 0x7fba04100930 2024-04-23T14:41:40.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.228+0000 7fb9fb400640 1 -- 172.21.15.28:0/701789592 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb9f40109d0 con 0x7fba04100930 2024-04-23T14:41:40.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.230+0000 7fb9fb400640 1 --2- 172.21.15.28:0/701789592 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb9e80a3bb0 0x7fb9e80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:40.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.230+0000 7fb9fb400640 1 -- 172.21.15.28:0/701789592 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7fb9f40cff10 con 0x7fba04100930 2024-04-23T14:41:40.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.230+0000 7fb9fbe00640 1 --2- 172.21.15.28:0/701789592 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb9e80a3bb0 0x7fb9e80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:40.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.231+0000 7fb9fbe00640 1 --2- 172.21.15.28:0/701789592 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb9e80a3bb0 0x7fb9e80a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec031440 tx=0x7fb9ec0315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:40.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.234+0000 7fb9fb400640 1 -- 172.21.15.28:0/701789592 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb9f4009de0 con 0x7fba04100930 2024-04-23T14:41:40.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:40 smithi105 ceph-mon[32555]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-23T14:41:40.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:40 smithi105 ceph-mon[32555]: osdmap e83: 12 total, 12 up, 12 in 2024-04-23T14:41:40.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:40 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2914445592' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-23T14:41:40.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:40 smithi105 ceph-mon[32555]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-23T14:41:40.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:40 smithi105 ceph-mon[32555]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-23T14:41:40.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:40 smithi038 ceph-mon[32348]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-23T14:41:40.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:40 smithi038 ceph-mon[32348]: osdmap e83: 12 total, 12 up, 12 in 2024-04-23T14:41:40.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:40 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2914445592' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-23T14:41:40.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:40 smithi038 ceph-mon[32348]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-23T14:41:40.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:40 smithi038 ceph-mon[32348]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-23T14:41:40.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.453+0000 7fba09800640 1 -- 172.21.15.28:0/701789592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fba0402d810 con 0x7fba04100930 2024-04-23T14:41:40.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.455+0000 7fb9fb400640 1 -- 172.21.15.28:0/701789592 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+22183 (secure 0 0 0) 0x7fb9f409c350 con 0x7fba04100930 2024-04-23T14:41:40.456 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:41:40.456 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":84,"fsid":"1ee6e996-017e-11ef-bc93-c7b262605968","created":"2024-04-23T14:32:34.581082+0000","modified":"2024-04-23T14:41:39.495112+0000","last_up_change":"2024-04-23T14:40:43.375846+0000","last_in_change":"2024-04-23T14:40:23.576939+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:36:27.698112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:41:37.030341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:41:38.444742+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e0b427bc-be65-4543-99cc-2f53ad98ddad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6803","nonce":2993445336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6805","nonce":2993445336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6809","nonce":2993445336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6807","nonce":2993445336}]},"public_addr":"172.21.15.28:6803/2993445336","cluster_addr":"172.21.15.28:6805/2993445336","heartbeat_back_addr":"172.21.15.28:6809/2993445336","heartbeat_front_addr":"172.21.15.28:6807/2993445336","state":["exists","up"]},{"osd":1,"uuid":"ef527ec9-1bf3-4bb0-8df3-f5a015af557c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6801","nonce":4024093383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6803","nonce":4024093383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6807","nonce":4024093383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6805","nonce":4024093383}]},"public_addr":"172.21.15.38:6801/4024093383","cluster_addr":"172.21.15.38:6803/4024093383","heartbeat_back_addr":"172.21.15.38:6807/4024093383","heartbeat_front_addr":"172.21.15.38:6805/4024093383","state":["exists","up"]},{"osd":2,"uuid":"b33b23d9-adce-4c11-b304-2e117bdf6bfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6801","nonce":570739029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6803","nonce":570739029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6807","nonce":570739029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6805","nonce":570739029}]},"public_addr":"172.21.15.105:6801/570739029","cluster_addr":"172.21.15.105:6803/570739029","heartbeat_back_addr":"172.21.15.105:6807/570739029","heartbeat_front_addr":"172.21.15.105:6805/570739029","state":["exists","up"]},{"osd":3,"uuid":"28bd5608-b426-434e-9db1-bdaa9828bdb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6811","nonce":2480400247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6813","nonce":2480400247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6817","nonce":2480400247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6815","nonce":2480400247}]},"public_addr":"172.21.15.28:6811/2480400247","cluster_addr":"172.21.15.28:6813/2480400247","heartbeat_back_addr":"172.21.15.28:6817/2480400247","heartbeat_front_addr":"172.21.15.28:6815/2480400247","state":["exists","up"]},{"osd":4,"uuid":"e5d1c89b-025f-4d57-8807-59d75c787b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6809","nonce":3772189246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6811","nonce":3772189246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6815","nonce":3772189246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6813","nonce":3772189246}]},"public_addr":"172.21.15.38:6809/3772189246","cluster_addr":"172.21.15.38:6811/3772189246","heartbeat_back_addr":"172.21.15.38:6815/3772189246","heartbeat_front_addr":"172.21.15.38:6813/3772189246","state":["exists","up"]},{"osd":5,"uuid":"9cf9cf4c-aafd-4ed3-bfb1-0527710a0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6809","nonce":928755894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6811","nonce":928755894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6815","nonce":928755894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6813","nonce":928755894}]},"public_addr":"172.21.15.105:6809/928755894","cluster_addr":"172.21.15.105:6811/928755894","heartbeat_back_addr":"172.21.15.105:6815/928755894","heartbeat_front_addr":"172.21.15.105:6813/928755894","state":["exists","up"]},{"osd":6,"uuid":"304db525-d24d-4687-bfa0-f92bdf66d1e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6819","nonce":897979441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6821","nonce":897979441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6825","nonce":897979441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6823","nonce":897979441}]},"public_addr":"172.21.15.28:6819/897979441","cluster_addr":"172.21.15.28:6821/897979441","heartbeat_back_addr":"172.21.15.28:6825/897979441","heartbeat_front_addr":"172.21.15.28:6823/897979441","state":["exists","up"]},{"osd":7,"uuid":"08e01d59-03f3-47b1-b45f-35ec5a50992c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6817","nonce":2400251786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6819","nonce":2400251786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6823","nonce":2400251786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6821","nonce":2400251786}]},"public_addr":"172.21.15.38:6817/2400251786","cluster_addr":"172.21.15.38:6819/2400251786","heartbeat_back_addr":"172.21.15.38:6823/2400251786","heartbeat_front_addr":"172.21.15.38:6821/2400251786","state":["exists","up"]},{"osd":8,"uuid":"64ee1e95-0756-48b1-9c00-472d0c3cbd29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6817","nonce":1682312269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6819","nonce":1682312269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6823","nonce":1682312269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6821","nonce":1682312269}]},"public_addr":"172.21.15.105:6817/1682312269","cluster_addr":"172.21.15.105:6819/1682312269","heartbeat_back_addr":"172.21.15.105:6823/1682312269","heartbeat_front_addr":"172.21.15.105:6821/1682312269","state":["exists","up"]},{"osd":9,"uuid":"18f60893-3ffd-48c2-9db5-a67d35611d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6827","nonce":1292942516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6829","nonce":1292942516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6833","nonce":1292942516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6831","nonce":1292942516}]},"public_addr":"172.21.15.28:6827/1292942516","cluster_addr":"172.21.15.28:6829/1292942516","heartbeat_back_addr":"172.21.15.28:6833/1292942516","heartbeat_front_addr":"172.21.15.28:6831/1292942516","state":["exists","up"]},{"osd":10,"uuid":"76c06725-7040-4118-8894-fe808a87f999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6825","nonce":2444616468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6827","nonce":2444616468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6831","nonce":2444616468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6829","nonce":2444616468}]},"public_addr":"172.21.15.38:6825/2444616468","cluster_addr":"172.21.15.38:6827/2444616468","heartbeat_back_addr":"172.21.15.38:6831/2444616468","heartbeat_front_addr":"172.21.15.38:6829/2444616468","state":["exists","up"]},{"osd":11,"uuid":"e8fc9454-5540-4e2e-91bf-3ba251dbd1a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6825","nonce":3400299562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6827","nonce":3400299562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6831","nonce":3400299562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6829","nonce":3400299562}]},"public_addr":"172.21.15.105:6825/3400299562","cluster_addr":"172.21.15.105:6827/3400299562","heartbeat_back_addr":"172.21.15.105:6831/3400299562","heartbeat_front_addr":"172.21.15.105:6829/3400299562","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:35:33.615395+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:00.545293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:24.940211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:57.684887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:24.519061+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:49.010804+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:22.638478+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:49.559104+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:14.011771+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:47.859950+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:16.298445+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:41.619069+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/590495740":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/1163172512":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/4290328460":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/4145020172":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/720626958":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/948450262":"2024-04-24T14:33:07.576363+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:41:40.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.458+0000 7fba09800640 1 -- 172.21.15.28:0/701789592 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb9e80a3bb0 msgr2=0x7fb9e80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:40.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.458+0000 7fba09800640 1 --2- 172.21.15.28:0/701789592 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb9e80a3bb0 0x7fb9e80a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec031440 tx=0x7fb9ec0315b0 comp rx=0 tx=0).stop 2024-04-23T14:41:40.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.458+0000 7fba09800640 1 -- 172.21.15.28:0/701789592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba04100930 msgr2=0x7fba0419a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:40.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.458+0000 7fba09800640 1 --2- 172.21.15.28:0/701789592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba04100930 0x7fba0419a560 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb9f400d950 tx=0x7fb9f400de10 comp rx=0 tx=0).stop 2024-04-23T14:41:40.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.458+0000 7fba09800640 1 -- 172.21.15.28:0/701789592 shutdown_connections 2024-04-23T14:41:40.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.458+0000 7fba09800640 1 --2- 172.21.15.28:0/701789592 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb9e80a3bb0 0x7fb9e80a6070 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:40.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.458+0000 7fba09800640 1 --2- 172.21.15.28:0/701789592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba04101c30 0x7fba041a1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:40.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.458+0000 7fba09800640 1 --2- 172.21.15.28:0/701789592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fba04101290 0x7fba0419aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:40.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.458+0000 7fba09800640 1 --2- 172.21.15.28:0/701789592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba04100930 0x7fba0419a560 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:40.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.458+0000 7fba09800640 1 -- 172.21.15.28:0/701789592 >> 172.21.15.28:0/701789592 conn(0x7fba040fc4b0 msgr2=0x7fba040fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:40.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.458+0000 7fba09800640 1 -- 172.21.15.28:0/701789592 shutdown_connections 2024-04-23T14:41:40.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.458+0000 7fba09800640 1 -- 172.21.15.28:0/701789592 wait complete. 2024-04-23T14:41:40.477 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:41:40.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.667+0000 7f40cb200640 1 -- 172.21.15.28:0/2066977696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40c410e470 msgr2=0x7f40c4110860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:40.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.667+0000 7f40cb200640 1 --2- 172.21.15.28:0/2066977696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40c410e470 0x7f40c4110860 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f40c0009cd0 tx=0x7f40c002a8f0 comp rx=0 tx=0).stop 2024-04-23T14:41:40.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.667+0000 7f40cb200640 1 -- 172.21.15.28:0/2066977696 shutdown_connections 2024-04-23T14:41:40.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.667+0000 7f40cb200640 1 --2- 172.21.15.28:0/2066977696 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40c4074a60 0x7f40c4074ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:40.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.667+0000 7f40cb200640 1 --2- 172.21.15.28:0/2066977696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40c410e470 0x7f40c4110860 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:40.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.667+0000 7f40cb200640 1 --2- 172.21.15.28:0/2066977696 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f40c4076040 0x7f40c4074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:40.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.667+0000 7f40cb200640 1 -- 172.21.15.28:0/2066977696 >> 172.21.15.28:0/2066977696 conn(0x7f40c40fd410 msgr2=0x7f40c40ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:40.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.667+0000 7f40cb200640 1 -- 172.21.15.28:0/2066977696 shutdown_connections 2024-04-23T14:41:40.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.667+0000 7f40cb200640 1 -- 172.21.15.28:0/2066977696 wait complete. 2024-04-23T14:41:40.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.668+0000 7f40cb200640 1 Processor -- start 2024-04-23T14:41:40.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.668+0000 7f40cb200640 1 -- start start 2024-04-23T14:41:40.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.668+0000 7f40cb200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f40c4074a60 0x7f40c419e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:40.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.668+0000 7f40cb200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40c4076040 0x7f40c419ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:40.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.668+0000 7f40cb200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40c410e470 0x7f40c41a5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:40.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.668+0000 7f40cb200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f40c4114180 con 0x7f40c4076040 2024-04-23T14:41:40.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.668+0000 7f40cb200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f40c4114000 con 0x7f40c410e470 2024-04-23T14:41:40.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.668+0000 7f40cb200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f40c4114300 con 0x7f40c4074a60 2024-04-23T14:41:40.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.669+0000 7f40c9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40c4076040 0x7f40c419ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:40.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.669+0000 7f40c9e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f40c4074a60 0x7f40c419e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:40.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.669+0000 7f40c9e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f40c4074a60 0x7f40c419e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:59964/0 (socket says 172.21.15.28:59964) 2024-04-23T14:41:40.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.669+0000 7f40c9e00640 1 -- 172.21.15.28:0/1431921028 learned_addr learned my addr 172.21.15.28:0/1431921028 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:40.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.669+0000 7f40ca800640 1 --2- 172.21.15.28:0/1431921028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40c410e470 0x7f40c41a5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:40.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.669+0000 7f40c9400640 1 -- 172.21.15.28:0/1431921028 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f40c4074a60 msgr2=0x7f40c419e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:40.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.669+0000 7f40c9400640 1 --2- 172.21.15.28:0/1431921028 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f40c4074a60 0x7f40c419e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:40.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.669+0000 7f40c9400640 1 -- 172.21.15.28:0/1431921028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40c410e470 msgr2=0x7f40c41a5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:40.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.669+0000 7f40c9400640 1 --2- 172.21.15.28:0/1431921028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40c410e470 0x7f40c41a5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:40.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.669+0000 7f40c9400640 1 -- 172.21.15.28:0/1431921028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40c41a6390 con 0x7f40c4076040 2024-04-23T14:41:40.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.669+0000 7f40ca800640 1 --2- 172.21.15.28:0/1431921028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40c410e470 0x7f40c41a5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:41:40.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.669+0000 7f40c9400640 1 --2- 172.21.15.28:0/1431921028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40c4076040 0x7f40c419ecb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f40b800e9f0 tx=0x7f40b800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:40.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.670+0000 7f40bea00640 1 -- 172.21.15.28:0/1431921028 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b800ccc0 con 0x7f40c4076040 2024-04-23T14:41:40.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.670+0000 7f40bea00640 1 -- 172.21.15.28:0/1431921028 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40b8010910 con 0x7f40c4076040 2024-04-23T14:41:40.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.670+0000 7f40cb200640 1 -- 172.21.15.28:0/1431921028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40c41a6680 con 0x7f40c4076040 2024-04-23T14:41:40.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.670+0000 7f40bea00640 1 -- 172.21.15.28:0/1431921028 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b8018960 con 0x7f40c4076040 2024-04-23T14:41:40.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.670+0000 7f40cb200640 1 -- 172.21.15.28:0/1431921028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40c41a6c20 con 0x7f40c4076040 2024-04-23T14:41:40.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.671+0000 7f40cb200640 1 -- 172.21.15.28:0/1431921028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40c4104640 con 0x7f40c4076040 2024-04-23T14:41:40.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.673+0000 7f40bea00640 1 -- 172.21.15.28:0/1431921028 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f40b8010ac0 con 0x7f40c4076040 2024-04-23T14:41:40.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.674+0000 7f40bea00640 1 --2- 172.21.15.28:0/1431921028 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f40ac0a3c30 0x7f40ac0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:40.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.675+0000 7f40bea00640 1 -- 172.21.15.28:0/1431921028 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f40b80d0650 con 0x7f40c4076040 2024-04-23T14:41:40.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.675+0000 7f40c9e00640 1 --2- 172.21.15.28:0/1431921028 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f40ac0a3c30 0x7f40ac0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:40.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.676+0000 7f40c9e00640 1 --2- 172.21.15.28:0/1431921028 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f40ac0a3c30 0x7f40ac0a60f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f40b4009cd0 tx=0x7f40b4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:40.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.681+0000 7f40bea00640 1 -- 172.21.15.28:0/1431921028 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f40b809ca90 con 0x7f40c4076040 2024-04-23T14:41:40.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.899+0000 7f40cb200640 1 -- 172.21.15.28:0/1431921028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f40c4062c00 con 0x7f40c4076040 2024-04-23T14:41:40.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.901+0000 7f40bea00640 1 -- 172.21.15.28:0/1431921028 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+22183 (secure 0 0 0) 0x7f40b80a0d90 con 0x7f40c4076040 2024-04-23T14:41:40.901 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:41:40.902 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":84,"fsid":"1ee6e996-017e-11ef-bc93-c7b262605968","created":"2024-04-23T14:32:34.581082+0000","modified":"2024-04-23T14:41:39.495112+0000","last_up_change":"2024-04-23T14:40:43.375846+0000","last_in_change":"2024-04-23T14:40:23.576939+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:36:27.698112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:41:37.030341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:41:38.444742+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e0b427bc-be65-4543-99cc-2f53ad98ddad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6803","nonce":2993445336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6805","nonce":2993445336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6809","nonce":2993445336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6807","nonce":2993445336}]},"public_addr":"172.21.15.28:6803/2993445336","cluster_addr":"172.21.15.28:6805/2993445336","heartbeat_back_addr":"172.21.15.28:6809/2993445336","heartbeat_front_addr":"172.21.15.28:6807/2993445336","state":["exists","up"]},{"osd":1,"uuid":"ef527ec9-1bf3-4bb0-8df3-f5a015af557c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6801","nonce":4024093383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6803","nonce":4024093383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6807","nonce":4024093383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6805","nonce":4024093383}]},"public_addr":"172.21.15.38:6801/4024093383","cluster_addr":"172.21.15.38:6803/4024093383","heartbeat_back_addr":"172.21.15.38:6807/4024093383","heartbeat_front_addr":"172.21.15.38:6805/4024093383","state":["exists","up"]},{"osd":2,"uuid":"b33b23d9-adce-4c11-b304-2e117bdf6bfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6801","nonce":570739029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6803","nonce":570739029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6807","nonce":570739029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6805","nonce":570739029}]},"public_addr":"172.21.15.105:6801/570739029","cluster_addr":"172.21.15.105:6803/570739029","heartbeat_back_addr":"172.21.15.105:6807/570739029","heartbeat_front_addr":"172.21.15.105:6805/570739029","state":["exists","up"]},{"osd":3,"uuid":"28bd5608-b426-434e-9db1-bdaa9828bdb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6811","nonce":2480400247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6813","nonce":2480400247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6817","nonce":2480400247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6815","nonce":2480400247}]},"public_addr":"172.21.15.28:6811/2480400247","cluster_addr":"172.21.15.28:6813/2480400247","heartbeat_back_addr":"172.21.15.28:6817/2480400247","heartbeat_front_addr":"172.21.15.28:6815/2480400247","state":["exists","up"]},{"osd":4,"uuid":"e5d1c89b-025f-4d57-8807-59d75c787b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6809","nonce":3772189246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6811","nonce":3772189246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6815","nonce":3772189246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6813","nonce":3772189246}]},"public_addr":"172.21.15.38:6809/3772189246","cluster_addr":"172.21.15.38:6811/3772189246","heartbeat_back_addr":"172.21.15.38:6815/3772189246","heartbeat_front_addr":"172.21.15.38:6813/3772189246","state":["exists","up"]},{"osd":5,"uuid":"9cf9cf4c-aafd-4ed3-bfb1-0527710a0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6809","nonce":928755894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6811","nonce":928755894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6815","nonce":928755894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6813","nonce":928755894}]},"public_addr":"172.21.15.105:6809/928755894","cluster_addr":"172.21.15.105:6811/928755894","heartbeat_back_addr":"172.21.15.105:6815/928755894","heartbeat_front_addr":"172.21.15.105:6813/928755894","state":["exists","up"]},{"osd":6,"uuid":"304db525-d24d-4687-bfa0-f92bdf66d1e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6819","nonce":897979441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6821","nonce":897979441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6825","nonce":897979441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6823","nonce":897979441}]},"public_addr":"172.21.15.28:6819/897979441","cluster_addr":"172.21.15.28:6821/897979441","heartbeat_back_addr":"172.21.15.28:6825/897979441","heartbeat_front_addr":"172.21.15.28:6823/897979441","state":["exists","up"]},{"osd":7,"uuid":"08e01d59-03f3-47b1-b45f-35ec5a50992c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6817","nonce":2400251786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6819","nonce":2400251786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6823","nonce":2400251786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6821","nonce":2400251786}]},"public_addr":"172.21.15.38:6817/2400251786","cluster_addr":"172.21.15.38:6819/2400251786","heartbeat_back_addr":"172.21.15.38:6823/2400251786","heartbeat_front_addr":"172.21.15.38:6821/2400251786","state":["exists","up"]},{"osd":8,"uuid":"64ee1e95-0756-48b1-9c00-472d0c3cbd29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6817","nonce":1682312269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6819","nonce":1682312269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6823","nonce":1682312269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6821","nonce":1682312269}]},"public_addr":"172.21.15.105:6817/1682312269","cluster_addr":"172.21.15.105:6819/1682312269","heartbeat_back_addr":"172.21.15.105:6823/1682312269","heartbeat_front_addr":"172.21.15.105:6821/1682312269","state":["exists","up"]},{"osd":9,"uuid":"18f60893-3ffd-48c2-9db5-a67d35611d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6827","nonce":1292942516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6829","nonce":1292942516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6833","nonce":1292942516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6831","nonce":1292942516}]},"public_addr":"172.21.15.28:6827/1292942516","cluster_addr":"172.21.15.28:6829/1292942516","heartbeat_back_addr":"172.21.15.28:6833/1292942516","heartbeat_front_addr":"172.21.15.28:6831/1292942516","state":["exists","up"]},{"osd":10,"uuid":"76c06725-7040-4118-8894-fe808a87f999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6825","nonce":2444616468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6827","nonce":2444616468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6831","nonce":2444616468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6829","nonce":2444616468}]},"public_addr":"172.21.15.38:6825/2444616468","cluster_addr":"172.21.15.38:6827/2444616468","heartbeat_back_addr":"172.21.15.38:6831/2444616468","heartbeat_front_addr":"172.21.15.38:6829/2444616468","state":["exists","up"]},{"osd":11,"uuid":"e8fc9454-5540-4e2e-91bf-3ba251dbd1a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6825","nonce":3400299562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6827","nonce":3400299562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6831","nonce":3400299562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6829","nonce":3400299562}]},"public_addr":"172.21.15.105:6825/3400299562","cluster_addr":"172.21.15.105:6827/3400299562","heartbeat_back_addr":"172.21.15.105:6831/3400299562","heartbeat_front_addr":"172.21.15.105:6829/3400299562","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:35:33.615395+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:00.545293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:24.940211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:57.684887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:24.519061+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:49.010804+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:22.638478+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:49.559104+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:14.011771+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:47.859950+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:16.298445+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:41.619069+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/590495740":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/1163172512":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/4290328460":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/4145020172":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/720626958":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/948450262":"2024-04-24T14:33:07.576363+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:41:40.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.904+0000 7f40cb200640 1 -- 172.21.15.28:0/1431921028 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f40ac0a3c30 msgr2=0x7f40ac0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:40.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.904+0000 7f40cb200640 1 --2- 172.21.15.28:0/1431921028 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f40ac0a3c30 0x7f40ac0a60f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f40b4009cd0 tx=0x7f40b4005f90 comp rx=0 tx=0).stop 2024-04-23T14:41:40.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.904+0000 7f40cb200640 1 -- 172.21.15.28:0/1431921028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40c4076040 msgr2=0x7f40c419ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:40.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.904+0000 7f40cb200640 1 --2- 172.21.15.28:0/1431921028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40c4076040 0x7f40c419ecb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f40b800e9f0 tx=0x7f40b800eeb0 comp rx=0 tx=0).stop 2024-04-23T14:41:40.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.904+0000 7f40cb200640 1 -- 172.21.15.28:0/1431921028 shutdown_connections 2024-04-23T14:41:40.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.904+0000 7f40cb200640 1 --2- 172.21.15.28:0/1431921028 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f40ac0a3c30 0x7f40ac0a60f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:40.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.904+0000 7f40cb200640 1 --2- 172.21.15.28:0/1431921028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40c410e470 0x7f40c41a5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:40.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.904+0000 7f40cb200640 1 --2- 172.21.15.28:0/1431921028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40c4076040 0x7f40c419ecb0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:40.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.904+0000 7f40cb200640 1 --2- 172.21.15.28:0/1431921028 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f40c4074a60 0x7f40c419e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:40.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.904+0000 7f40cb200640 1 -- 172.21.15.28:0/1431921028 >> 172.21.15.28:0/1431921028 conn(0x7f40c40fd410 msgr2=0x7f40c4101810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:40.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.904+0000 7f40cb200640 1 -- 172.21.15.28:0/1431921028 shutdown_connections 2024-04-23T14:41:40.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:40.904+0000 7f40cb200640 1 -- 172.21.15.28:0/1431921028 wait complete. 2024-04-23T14:41:40.922 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-23T14:41:41.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.126+0000 7efd19800640 1 -- 172.21.15.28:0/2183404939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd14105470 msgr2=0x7efd141058f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:41.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.126+0000 7efd19800640 1 --2- 172.21.15.28:0/2183404939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd14105470 0x7efd141058f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7efd00009900 tx=0x7efd0002a3f0 comp rx=0 tx=0).stop 2024-04-23T14:41:41.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.126+0000 7efd19800640 1 -- 172.21.15.28:0/2183404939 shutdown_connections 2024-04-23T14:41:41.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.126+0000 7efd19800640 1 --2- 172.21.15.28:0/2183404939 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efd14105e30 0x7efd1410c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:41.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.126+0000 7efd19800640 1 --2- 172.21.15.28:0/2183404939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd14105470 0x7efd141058f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:41.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.126+0000 7efd19800640 1 --2- 172.21.15.28:0/2183404939 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efd14104280 0x7efd141046a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:41.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.126+0000 7efd19800640 1 -- 172.21.15.28:0/2183404939 >> 172.21.15.28:0/2183404939 conn(0x7efd140ff7f0 msgr2=0x7efd14101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:41.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.126+0000 7efd19800640 1 -- 172.21.15.28:0/2183404939 shutdown_connections 2024-04-23T14:41:41.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.126+0000 7efd19800640 1 -- 172.21.15.28:0/2183404939 wait complete. 2024-04-23T14:41:41.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.127+0000 7efd19800640 1 Processor -- start 2024-04-23T14:41:41.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.128+0000 7efd19800640 1 -- start start 2024-04-23T14:41:41.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.128+0000 7efd19800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd14104280 0x7efd1419a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:41.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.128+0000 7efd19800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efd14105470 0x7efd1419aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:41.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.128+0000 7efd19800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efd14105e30 0x7efd141a1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:41.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.128+0000 7efd19800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efd1410ff60 con 0x7efd14104280 2024-04-23T14:41:41.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.128+0000 7efd19800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efd1410fde0 con 0x7efd14105470 2024-04-23T14:41:41.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.128+0000 7efd19800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7efd141100e0 con 0x7efd14105e30 2024-04-23T14:41:41.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.128+0000 7efd12000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efd14105470 0x7efd1419aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:41.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.128+0000 7efd12000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efd14105470 0x7efd1419aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48348/0 (socket says 172.21.15.28:48348) 2024-04-23T14:41:41.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.128+0000 7efd12000640 1 -- 172.21.15.28:0/4040898371 learned_addr learned my addr 172.21.15.28:0/4040898371 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:41.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.128+0000 7efd12a00640 1 --2- 172.21.15.28:0/4040898371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd14104280 0x7efd1419a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:41.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.129+0000 7efd13400640 1 --2- 172.21.15.28:0/4040898371 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efd14105e30 0x7efd141a1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:41.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.129+0000 7efd12000640 1 -- 172.21.15.28:0/4040898371 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efd14105e30 msgr2=0x7efd141a1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:41.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.129+0000 7efd12000640 1 --2- 172.21.15.28:0/4040898371 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efd14105e30 0x7efd141a1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:41.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.129+0000 7efd12000640 1 -- 172.21.15.28:0/4040898371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd14104280 msgr2=0x7efd1419a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:41.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.129+0000 7efd12000640 1 --2- 172.21.15.28:0/4040898371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd14104280 0x7efd1419a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:41.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.129+0000 7efd12000640 1 -- 172.21.15.28:0/4040898371 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd141a21a0 con 0x7efd14105470 2024-04-23T14:41:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.129+0000 7efd12000640 1 --2- 172.21.15.28:0/4040898371 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efd14105470 0x7efd1419aac0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7efd00031440 tx=0x7efd00032ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.130+0000 7efd0b400640 1 -- 172.21.15.28:0/4040898371 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd0003f030 con 0x7efd14105470 2024-04-23T14:41:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.130+0000 7efd0b400640 1 -- 172.21.15.28:0/4040898371 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd00003ce0 con 0x7efd14105470 2024-04-23T14:41:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.130+0000 7efd19800640 1 -- 172.21.15.28:0/4040898371 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd141a2430 con 0x7efd14105470 2024-04-23T14:41:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.130+0000 7efd0b400640 1 -- 172.21.15.28:0/4040898371 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd00035070 con 0x7efd14105470 2024-04-23T14:41:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.130+0000 7efd19800640 1 -- 172.21.15.28:0/4040898371 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd141a2920 con 0x7efd14105470 2024-04-23T14:41:41.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.131+0000 7efd19800640 1 -- 172.21.15.28:0/4040898371 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd1410b5a0 con 0x7efd14105470 2024-04-23T14:41:41.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.133+0000 7efd0b400640 1 -- 172.21.15.28:0/4040898371 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7efd00042030 con 0x7efd14105470 2024-04-23T14:41:41.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.135+0000 7efd0b400640 1 --2- 172.21.15.28:0/4040898371 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7efcf80a3bb0 0x7efcf80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:41.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.135+0000 7efd0b400640 1 -- 172.21.15.28:0/4040898371 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7efd000f07c0 con 0x7efd14105470 2024-04-23T14:41:41.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.135+0000 7efd12a00640 1 --2- 172.21.15.28:0/4040898371 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7efcf80a3bb0 0x7efcf80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.136+0000 7efd12a00640 1 --2- 172.21.15.28:0/4040898371 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7efcf80a3bb0 0x7efcf80a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7efd141052d0 tx=0x7efd04005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:41.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.139+0000 7efd0b400640 1 -- 172.21.15.28:0/4040898371 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efd000bcc00 con 0x7efd14105470 2024-04-23T14:41:41.278 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:41 smithi038 ceph-mon[32348]: pgmap v312: 97 pgs: 28 creating+peering, 7 active+clean, 62 unknown; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:41.278 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:41 smithi038 ceph-mon[32348]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T14:41:41.278 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:41 smithi038 ceph-mon[32348]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-23T14:41:41.279 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:41 smithi038 ceph-mon[32348]: osdmap e84: 12 total, 12 up, 12 in 2024-04-23T14:41:41.279 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:41 smithi038 ceph-mon[32348]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-23T14:41:41.279 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:41 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:41.279 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:41 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:41.279 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:41 smithi038 ceph-mon[32348]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-23T14:41:41.279 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:41 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:41.279 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:41 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:41.279 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:41 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/701789592' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:41:41.279 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:41 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1431921028' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:41:41.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:41 smithi105 ceph-mon[32555]: pgmap v312: 97 pgs: 28 creating+peering, 7 active+clean, 62 unknown; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:41.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:41 smithi105 ceph-mon[32555]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T14:41:41.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:41 smithi105 ceph-mon[32555]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-23T14:41:41.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:41 smithi105 ceph-mon[32555]: osdmap e84: 12 total, 12 up, 12 in 2024-04-23T14:41:41.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:41 smithi105 ceph-mon[32555]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-23T14:41:41.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:41 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:41.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:41 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:41.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:41 smithi105 ceph-mon[32555]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-23T14:41:41.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:41 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:41.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:41 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:41.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:41 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/701789592' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:41:41.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:41 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1431921028' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:41:41.376 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:41 smithi028 ceph-mon[27810]: pgmap v312: 97 pgs: 28 creating+peering, 7 active+clean, 62 unknown; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:41:41.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:41 smithi028 ceph-mon[27810]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T14:41:41.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:41 smithi028 ceph-mon[27810]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-23T14:41:41.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:41 smithi028 ceph-mon[27810]: osdmap e84: 12 total, 12 up, 12 in 2024-04-23T14:41:41.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:41 smithi028 ceph-mon[27810]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-23T14:41:41.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:41 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:41.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:41 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:41.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:41 smithi028 ceph-mon[27810]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-23T14:41:41.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:41 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:41.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:41 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:41.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:41 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/701789592' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:41:41.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:41 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1431921028' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:41:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:41.412+0000 7efd19800640 1 -- 172.21.15.28:0/4040898371 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7efd14062c00 con 0x7efd14105470 2024-04-23T14:41:41.622 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 23 14:41:41 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-b[63031]: 2024-04-23T14:41:41.275+0000 7f9a2586c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-23T14:41:42.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:42 smithi105 ceph-mon[32555]: osdmap e85: 12 total, 12 up, 12 in 2024-04-23T14:41:42.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:42 smithi105 ceph-mon[32555]: mds.? [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] up:active 2024-04-23T14:41:42.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:42 smithi105 ceph-mon[32555]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T14:41:42.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:42 smithi105 ceph-mon[32555]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T14:41:42.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:42 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:42.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:42 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:42.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:42 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:42.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:42 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:41:42.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:42 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:42.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:42 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/4040898371' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-23T14:41:42.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:42 smithi105 ceph-mon[32555]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-23T14:41:42.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:42 smithi038 ceph-mon[32348]: osdmap e85: 12 total, 12 up, 12 in 2024-04-23T14:41:42.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:42 smithi038 ceph-mon[32348]: mds.? [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] up:active 2024-04-23T14:41:42.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:42 smithi038 ceph-mon[32348]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T14:41:42.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:42 smithi038 ceph-mon[32348]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T14:41:42.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:42 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:42.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:42 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:42.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:42 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:42.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:42 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:41:42.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:42 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:42.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:42 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/4040898371' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-23T14:41:42.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:42 smithi038 ceph-mon[32348]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-23T14:41:42.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:42 smithi028 ceph-mon[27810]: osdmap e85: 12 total, 12 up, 12 in 2024-04-23T14:41:42.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:42 smithi028 ceph-mon[27810]: mds.? [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] up:active 2024-04-23T14:41:42.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:42 smithi028 ceph-mon[27810]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T14:41:42.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:42 smithi028 ceph-mon[27810]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T14:41:42.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:42 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:42.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:42 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:42.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:42 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:41:42.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:42 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:41:42.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:42 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:41:42.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:42 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/4040898371' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-23T14:41:42.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:42 smithi028 ceph-mon[27810]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-23T14:41:42.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.378+0000 7efd0b400640 1 -- 172.21.15.28:0/4040898371 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7efd000c0f00 con 0x7efd14105470 2024-04-23T14:41:42.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.381+0000 7efd19800640 1 -- 172.21.15.28:0/4040898371 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7efcf80a3bb0 msgr2=0x7efcf80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:42.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.381+0000 7efd19800640 1 --2- 172.21.15.28:0/4040898371 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7efcf80a3bb0 0x7efcf80a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7efd141052d0 tx=0x7efd04005f90 comp rx=0 tx=0).stop 2024-04-23T14:41:42.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.381+0000 7efd19800640 1 -- 172.21.15.28:0/4040898371 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efd14105470 msgr2=0x7efd1419aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:42.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.381+0000 7efd19800640 1 --2- 172.21.15.28:0/4040898371 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efd14105470 0x7efd1419aac0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7efd00031440 tx=0x7efd00032ce0 comp rx=0 tx=0).stop 2024-04-23T14:41:42.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.381+0000 7efd19800640 1 -- 172.21.15.28:0/4040898371 shutdown_connections 2024-04-23T14:41:42.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.381+0000 7efd19800640 1 --2- 172.21.15.28:0/4040898371 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7efcf80a3bb0 0x7efcf80a6070 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:42.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.381+0000 7efd19800640 1 --2- 172.21.15.28:0/4040898371 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efd14105470 0x7efd1419aac0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:42.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.381+0000 7efd19800640 1 --2- 172.21.15.28:0/4040898371 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efd14105e30 0x7efd141a1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:42.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.381+0000 7efd19800640 1 --2- 172.21.15.28:0/4040898371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd14104280 0x7efd1419a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:42.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.381+0000 7efd19800640 1 -- 172.21.15.28:0/4040898371 >> 172.21.15.28:0/4040898371 conn(0x7efd140ff7f0 msgr2=0x7efd14108770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:42.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.382+0000 7efd19800640 1 -- 172.21.15.28:0/4040898371 shutdown_connections 2024-04-23T14:41:42.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.382+0000 7efd19800640 1 -- 172.21.15.28:0/4040898371 wait complete. 2024-04-23T14:41:42.404 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 1, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-04-23T14:41:42.404 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-23T14:41:42.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.605+0000 7f9155400640 1 -- 172.21.15.28:0/993936644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91501052c0 msgr2=0x7f91501076b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:42.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.605+0000 7f9155400640 1 --2- 172.21.15.28:0/993936644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91501052c0 0x7f91501076b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f913c009900 tx=0x7f913c02a3f0 comp rx=0 tx=0).stop 2024-04-23T14:41:42.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.605+0000 7f9155400640 1 -- 172.21.15.28:0/993936644 shutdown_connections 2024-04-23T14:41:42.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.605+0000 7f9155400640 1 --2- 172.21.15.28:0/993936644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91501052c0 0x7f91501076b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:42.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.605+0000 7f9155400640 1 --2- 172.21.15.28:0/993936644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9150107bf0 0x7f915010a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:42.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.605+0000 7f9155400640 1 --2- 172.21.15.28:0/993936644 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9150102990 0x7f9150104d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:42.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.605+0000 7f9155400640 1 -- 172.21.15.28:0/993936644 >> 172.21.15.28:0/993936644 conn(0x7f91500fc3a0 msgr2=0x7f91500fe7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:42.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.606+0000 7f9155400640 1 -- 172.21.15.28:0/993936644 shutdown_connections 2024-04-23T14:41:42.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.606+0000 7f9155400640 1 -- 172.21.15.28:0/993936644 wait complete. 2024-04-23T14:41:42.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.606+0000 7f9155400640 1 Processor -- start 2024-04-23T14:41:42.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.607+0000 7f9155400640 1 -- start start 2024-04-23T14:41:42.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.607+0000 7f9155400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9150102990 0x7f915019a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:42.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.607+0000 7f9155400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91501052c0 0x7f915019a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:42.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.607+0000 7f9155400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9150107bf0 0x7f91501a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:42.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.607+0000 7f9155400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f915010dbc0 con 0x7f9150102990 2024-04-23T14:41:42.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.607+0000 7f9155400640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f915010da40 con 0x7f9150107bf0 2024-04-23T14:41:42.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.607+0000 7f9155400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f915010dd40 con 0x7f91501052c0 2024-04-23T14:41:42.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.607+0000 7f914fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9150102990 0x7f915019a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:42.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.607+0000 7f914fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9150102990 0x7f915019a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35414/0 (socket says 172.21.15.28:35414) 2024-04-23T14:41:42.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.607+0000 7f914fe00640 1 -- 172.21.15.28:0/1133573718 learned_addr learned my addr 172.21.15.28:0/1133573718 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:42.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.607+0000 7f9154a00640 1 --2- 172.21.15.28:0/1133573718 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9150107bf0 0x7f91501a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:42.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.607+0000 7f914f400640 1 --2- 172.21.15.28:0/1133573718 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91501052c0 0x7f915019a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:42.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.608+0000 7f914fe00640 1 -- 172.21.15.28:0/1133573718 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91501052c0 msgr2=0x7f915019a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:42.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.608+0000 7f914fe00640 1 --2- 172.21.15.28:0/1133573718 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91501052c0 0x7f915019a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:42.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.608+0000 7f914fe00640 1 -- 172.21.15.28:0/1133573718 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9150107bf0 msgr2=0x7f91501a19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:42.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.608+0000 7f914fe00640 1 --2- 172.21.15.28:0/1133573718 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9150107bf0 0x7f91501a19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:42.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.608+0000 7f914fe00640 1 -- 172.21.15.28:0/1133573718 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91501a2030 con 0x7f9150102990 2024-04-23T14:41:42.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.608+0000 7f9154a00640 1 --2- 172.21.15.28:0/1133573718 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9150107bf0 0x7f91501a19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:41:42.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.608+0000 7f914fe00640 1 --2- 172.21.15.28:0/1133573718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9150102990 0x7f915019a410 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f914000d950 tx=0x7f914000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:42.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.608+0000 7f914cc00640 1 -- 172.21.15.28:0/1133573718 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9140017070 con 0x7f9150102990 2024-04-23T14:41:42.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.608+0000 7f914cc00640 1 -- 172.21.15.28:0/1133573718 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9140009c30 con 0x7f9150102990 2024-04-23T14:41:42.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.608+0000 7f9155400640 1 -- 172.21.15.28:0/1133573718 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91501a2320 con 0x7f9150102990 2024-04-23T14:41:42.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.608+0000 7f914cc00640 1 -- 172.21.15.28:0/1133573718 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914000f6b0 con 0x7f9150102990 2024-04-23T14:41:42.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.609+0000 7f9155400640 1 -- 172.21.15.28:0/1133573718 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91501a2890 con 0x7f9150102990 2024-04-23T14:41:42.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.610+0000 7f9155400640 1 -- 172.21.15.28:0/1133573718 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9150109740 con 0x7f9150102990 2024-04-23T14:41:42.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.613+0000 7f914cc00640 1 -- 172.21.15.28:0/1133573718 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f91400109d0 con 0x7f9150102990 2024-04-23T14:41:42.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.615+0000 7f914cc00640 1 --2- 172.21.15.28:0/1133573718 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f91340a3c30 0x7f91340a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:42.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.616+0000 7f914cc00640 1 -- 172.21.15.28:0/1133573718 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f91400cfe40 con 0x7f9150102990 2024-04-23T14:41:42.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.616+0000 7f914f400640 1 --2- 172.21.15.28:0/1133573718 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f91340a3c30 0x7f91340a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:42.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.617+0000 7f914f400640 1 --2- 172.21.15.28:0/1133573718 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f91340a3c30 0x7f91340a60f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f913c02a9e0 tx=0x7f913c0315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:42.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.620+0000 7f914cc00640 1 -- 172.21.15.28:0/1133573718 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9140009de0 con 0x7f9150102990 2024-04-23T14:41:42.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:42.865+0000 7f9155400640 1 -- 172.21.15.28:0/1133573718 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f915002d620 con 0x7f9150102990 2024-04-23T14:41:43.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:43 smithi105 ceph-mon[32555]: pgmap v315: 97 pgs: 28 creating+peering, 36 active+clean, 33 unknown; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 6 op/s 2024-04-23T14:41:43.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:43 smithi105 ceph-mon[32555]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-23T14:41:43.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:43 smithi105 ceph-mon[32555]: Cluster is now healthy 2024-04-23T14:41:43.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:43 smithi105 ceph-mon[32555]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-23T14:41:43.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:43 smithi105 ceph-mon[32555]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T14:41:43.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:43 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1133573718' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-23T14:41:43.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:43 smithi038 ceph-mon[32348]: pgmap v315: 97 pgs: 28 creating+peering, 36 active+clean, 33 unknown; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 6 op/s 2024-04-23T14:41:43.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:43 smithi038 ceph-mon[32348]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-23T14:41:43.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:43 smithi038 ceph-mon[32348]: Cluster is now healthy 2024-04-23T14:41:43.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:43 smithi038 ceph-mon[32348]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-23T14:41:43.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:43 smithi038 ceph-mon[32348]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T14:41:43.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:43 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1133573718' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-23T14:41:43.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:43 smithi028 ceph-mon[27810]: pgmap v315: 97 pgs: 28 creating+peering, 36 active+clean, 33 unknown; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 6 op/s 2024-04-23T14:41:43.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:43 smithi028 ceph-mon[27810]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-23T14:41:43.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:43 smithi028 ceph-mon[27810]: Cluster is now healthy 2024-04-23T14:41:43.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:43 smithi028 ceph-mon[27810]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-23T14:41:43.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:43 smithi028 ceph-mon[27810]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T14:41:43.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:43 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1133573718' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-23T14:41:43.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.377+0000 7f914cc00640 1 -- 172.21.15.28:0/1133573718 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v18) ==== 121+0+0 (secure 0 0 0) 0x7f914009c280 con 0x7f9150102990 2024-04-23T14:41:43.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.381+0000 7f9155400640 1 -- 172.21.15.28:0/1133573718 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f91340a3c30 msgr2=0x7f91340a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:43.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.381+0000 7f9155400640 1 --2- 172.21.15.28:0/1133573718 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f91340a3c30 0x7f91340a60f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f913c02a9e0 tx=0x7f913c0315b0 comp rx=0 tx=0).stop 2024-04-23T14:41:43.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.381+0000 7f9155400640 1 -- 172.21.15.28:0/1133573718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9150102990 msgr2=0x7f915019a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:43.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.381+0000 7f9155400640 1 --2- 172.21.15.28:0/1133573718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9150102990 0x7f915019a410 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f914000d950 tx=0x7f914000de10 comp rx=0 tx=0).stop 2024-04-23T14:41:43.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.381+0000 7f9155400640 1 -- 172.21.15.28:0/1133573718 shutdown_connections 2024-04-23T14:41:43.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.381+0000 7f9155400640 1 --2- 172.21.15.28:0/1133573718 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f91340a3c30 0x7f91340a60f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:43.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.381+0000 7f9155400640 1 --2- 172.21.15.28:0/1133573718 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9150107bf0 0x7f91501a19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:43.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.381+0000 7f9155400640 1 --2- 172.21.15.28:0/1133573718 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91501052c0 0x7f915019a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:43.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.381+0000 7f9155400640 1 --2- 172.21.15.28:0/1133573718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9150102990 0x7f915019a410 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:43.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.381+0000 7f9155400640 1 -- 172.21.15.28:0/1133573718 >> 172.21.15.28:0/1133573718 conn(0x7f91500fc3a0 msgr2=0x7f9150102110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:43.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.382+0000 7f9155400640 1 -- 172.21.15.28:0/1133573718 shutdown_connections 2024-04-23T14:41:43.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.382+0000 7f9155400640 1 -- 172.21.15.28:0/1133573718 wait complete. 2024-04-23T14:41:43.404 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-23T14:41:43.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.609+0000 7f2b27e00640 1 -- 172.21.15.28:0/410066632 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b28105330 msgr2=0x7f2b281057b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:43.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.609+0000 7f2b27e00640 1 --2- 172.21.15.28:0/410066632 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b28105330 0x7f2b281057b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2b14009900 tx=0x7f2b1402a410 comp rx=0 tx=0).stop 2024-04-23T14:41:43.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.609+0000 7f2b27e00640 1 -- 172.21.15.28:0/410066632 shutdown_connections 2024-04-23T14:41:43.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.609+0000 7f2b27e00640 1 --2- 172.21.15.28:0/410066632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b28105cf0 0x7f2b2810c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:43.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.609+0000 7f2b27e00640 1 --2- 172.21.15.28:0/410066632 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b28105330 0x7f2b281057b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:43.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.609+0000 7f2b27e00640 1 --2- 172.21.15.28:0/410066632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b28104140 0x7f2b28104560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:43.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.609+0000 7f2b27e00640 1 -- 172.21.15.28:0/410066632 >> 172.21.15.28:0/410066632 conn(0x7f2b280ff6b0 msgr2=0x7f2b28101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:43.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.610+0000 7f2b27e00640 1 -- 172.21.15.28:0/410066632 shutdown_connections 2024-04-23T14:41:43.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.610+0000 7f2b27e00640 1 -- 172.21.15.28:0/410066632 wait complete. 2024-04-23T14:41:43.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.611+0000 7f2b27e00640 1 Processor -- start 2024-04-23T14:41:43.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.611+0000 7f2b27e00640 1 -- start start 2024-04-23T14:41:43.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.612+0000 7f2b27e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b28104140 0x7f2b28079db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:43.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.612+0000 7f2b27e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b28105330 0x7f2b2807a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:43.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.612+0000 7f2b27e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b28105cf0 0x7f2b28076920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:43.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.612+0000 7f2b27e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2b2810fe20 con 0x7f2b28104140 2024-04-23T14:41:43.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.612+0000 7f2b27e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2b2810fca0 con 0x7f2b28105330 2024-04-23T14:41:43.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.612+0000 7f2b27e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2b2810ffa0 con 0x7f2b28105cf0 2024-04-23T14:41:43.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.612+0000 7f2b26a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b28104140 0x7f2b28079db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:43.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.612+0000 7f2b26a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b28104140 0x7f2b28079db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35428/0 (socket says 172.21.15.28:35428) 2024-04-23T14:41:43.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.612+0000 7f2b26a00640 1 -- 172.21.15.28:0/2849659610 learned_addr learned my addr 172.21.15.28:0/2849659610 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:43.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.612+0000 7f2b27400640 1 --2- 172.21.15.28:0/2849659610 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b28105cf0 0x7f2b28076920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:43.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.612+0000 7f2b26000640 1 --2- 172.21.15.28:0/2849659610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b28105330 0x7f2b2807a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:43.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.612+0000 7f2b26a00640 1 -- 172.21.15.28:0/2849659610 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b28105cf0 msgr2=0x7f2b28076920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:43.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.612+0000 7f2b26a00640 1 --2- 172.21.15.28:0/2849659610 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b28105cf0 0x7f2b28076920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:43.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.612+0000 7f2b26a00640 1 -- 172.21.15.28:0/2849659610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b28105330 msgr2=0x7f2b2807a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:43.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.612+0000 7f2b26a00640 1 --2- 172.21.15.28:0/2849659610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b28105330 0x7f2b2807a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:43.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.612+0000 7f2b26a00640 1 -- 172.21.15.28:0/2849659610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b280771e0 con 0x7f2b28104140 2024-04-23T14:41:43.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.612+0000 7f2b27400640 1 --2- 172.21.15.28:0/2849659610 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b28105cf0 0x7f2b28076920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:41:43.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.613+0000 7f2b26a00640 1 --2- 172.21.15.28:0/2849659610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b28104140 0x7f2b28079db0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2b1800e9c0 tx=0x7f2b1800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:43.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.613+0000 7f2b1f400640 1 -- 172.21.15.28:0/2849659610 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b1800cd80 con 0x7f2b28104140 2024-04-23T14:41:43.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.613+0000 7f2b1f400640 1 -- 172.21.15.28:0/2849659610 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b180035f0 con 0x7f2b28104140 2024-04-23T14:41:43.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.613+0000 7f2b27e00640 1 -- 172.21.15.28:0/2849659610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b281a68b0 con 0x7f2b28104140 2024-04-23T14:41:43.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.613+0000 7f2b1f400640 1 -- 172.21.15.28:0/2849659610 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b18010690 con 0x7f2b28104140 2024-04-23T14:41:43.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.613+0000 7f2b27e00640 1 -- 172.21.15.28:0/2849659610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b281a6cb0 con 0x7f2b28104140 2024-04-23T14:41:43.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.614+0000 7f2b27e00640 1 -- 172.21.15.28:0/2849659610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b28069190 con 0x7f2b28104140 2024-04-23T14:41:43.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.617+0000 7f2b1f400640 1 -- 172.21.15.28:0/2849659610 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2b180037a0 con 0x7f2b28104140 2024-04-23T14:41:43.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.619+0000 7f2b1f400640 1 --2- 172.21.15.28:0/2849659610 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2b0c0a3bb0 0x7f2b0c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:43.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.619+0000 7f2b1f400640 1 -- 172.21.15.28:0/2849659610 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f2b180d07e0 con 0x7f2b28104140 2024-04-23T14:41:43.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.619+0000 7f2b26000640 1 --2- 172.21.15.28:0/2849659610 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2b0c0a3bb0 0x7f2b0c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:43.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.620+0000 7f2b26000640 1 --2- 172.21.15.28:0/2849659610 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2b0c0a3bb0 0x7f2b0c0a6070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2b14005ab0 tx=0x7f2b140315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:43.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.621+0000 7f2b1f400640 1 -- 172.21.15.28:0/2849659610 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2b1809d300 con 0x7f2b28104140 2024-04-23T14:41:43.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:43.868+0000 7f2b27e00640 1 -- 172.21.15.28:0/2849659610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f2b2810b400 con 0x7f2b28104140 2024-04-23T14:41:44.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.398+0000 7f2b1f400640 1 -- 172.21.15.28:0/2849659610 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v20) ==== 115+0+0 (secure 0 0 0) 0x7f2b1809cc00 con 0x7f2b28104140 2024-04-23T14:41:44.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.401+0000 7f2b27e00640 1 -- 172.21.15.28:0/2849659610 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2b0c0a3bb0 msgr2=0x7f2b0c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:44.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.401+0000 7f2b27e00640 1 --2- 172.21.15.28:0/2849659610 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2b0c0a3bb0 0x7f2b0c0a6070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2b14005ab0 tx=0x7f2b140315b0 comp rx=0 tx=0).stop 2024-04-23T14:41:44.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.401+0000 7f2b27e00640 1 -- 172.21.15.28:0/2849659610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b28104140 msgr2=0x7f2b28079db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:44.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.401+0000 7f2b27e00640 1 --2- 172.21.15.28:0/2849659610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b28104140 0x7f2b28079db0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2b1800e9c0 tx=0x7f2b1800ee80 comp rx=0 tx=0).stop 2024-04-23T14:41:44.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.401+0000 7f2b27e00640 1 -- 172.21.15.28:0/2849659610 shutdown_connections 2024-04-23T14:41:44.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.401+0000 7f2b27e00640 1 --2- 172.21.15.28:0/2849659610 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2b0c0a3bb0 0x7f2b0c0a6070 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:44.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.401+0000 7f2b27e00640 1 --2- 172.21.15.28:0/2849659610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b28105330 0x7f2b2807a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:44.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.401+0000 7f2b27e00640 1 --2- 172.21.15.28:0/2849659610 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b28105cf0 0x7f2b28076920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:44.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.401+0000 7f2b27e00640 1 --2- 172.21.15.28:0/2849659610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b28104140 0x7f2b28079db0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:44.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.401+0000 7f2b27e00640 1 -- 172.21.15.28:0/2849659610 >> 172.21.15.28:0/2849659610 conn(0x7f2b280ff6b0 msgr2=0x7f2b281085d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:44.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.402+0000 7f2b27e00640 1 -- 172.21.15.28:0/2849659610 shutdown_connections 2024-04-23T14:41:44.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.402+0000 7f2b27e00640 1 -- 172.21.15.28:0/2849659610 wait complete. 2024-04-23T14:41:44.422 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-04-23T14:41:44.422 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-23T14:41:44.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.641+0000 7f95b5600640 1 -- 172.21.15.28:0/855318143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95b0109e50 msgr2=0x7f95b010c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:44.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.641+0000 7f95b5600640 1 --2- 172.21.15.28:0/855318143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95b0109e50 0x7f95b010c2e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f95a8009ca0 tx=0x7f95a802a8c0 comp rx=0 tx=0).stop 2024-04-23T14:41:44.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.641+0000 7f95b5600640 1 -- 172.21.15.28:0/855318143 shutdown_connections 2024-04-23T14:41:44.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.641+0000 7f95b5600640 1 --2- 172.21.15.28:0/855318143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95b0109e50 0x7f95b010c2e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:44.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.641+0000 7f95b5600640 1 --2- 172.21.15.28:0/855318143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95b0074a60 0x7f95b0074ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:44.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.641+0000 7f95b5600640 1 --2- 172.21.15.28:0/855318143 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95b0076040 0x7f95b0074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:44.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.641+0000 7f95b5600640 1 -- 172.21.15.28:0/855318143 >> 172.21.15.28:0/855318143 conn(0x7f95b00fd500 msgr2=0x7f95b00ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:44.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.641+0000 7f95b5600640 1 -- 172.21.15.28:0/855318143 shutdown_connections 2024-04-23T14:41:44.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.641+0000 7f95b5600640 1 -- 172.21.15.28:0/855318143 wait complete. 2024-04-23T14:41:44.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.642+0000 7f95b5600640 1 Processor -- start 2024-04-23T14:41:44.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.642+0000 7f95b5600640 1 -- start start 2024-04-23T14:41:44.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.643+0000 7f95b5600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95b0074a60 0x7f95b019a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:44.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.643+0000 7f95b5600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95b0076040 0x7f95b019a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:44.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.643+0000 7f95b5600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95b0109e50 0x7f95b01a1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:44.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.643+0000 7f95b5600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f95b010fdc0 con 0x7f95b0109e50 2024-04-23T14:41:44.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.643+0000 7f95b5600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f95b010fc40 con 0x7f95b0074a60 2024-04-23T14:41:44.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.643+0000 7f95b5600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f95b010ff40 con 0x7f95b0076040 2024-04-23T14:41:44.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.643+0000 7f95af400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95b0076040 0x7f95b019a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:44.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.643+0000 7f95b4c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95b0109e50 0x7f95b01a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:44.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.643+0000 7f95af400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95b0076040 0x7f95b019a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:60042/0 (socket says 172.21.15.28:60042) 2024-04-23T14:41:44.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.643+0000 7f95af400640 1 -- 172.21.15.28:0/3718443632 learned_addr learned my addr 172.21.15.28:0/3718443632 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:44.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.643+0000 7f95afe00640 1 --2- 172.21.15.28:0/3718443632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95b0074a60 0x7f95b019a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:44.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.644+0000 7f95afe00640 1 -- 172.21.15.28:0/3718443632 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95b0076040 msgr2=0x7f95b019a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:44.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.644+0000 7f95afe00640 1 --2- 172.21.15.28:0/3718443632 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95b0076040 0x7f95b019a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:44.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.644+0000 7f95afe00640 1 -- 172.21.15.28:0/3718443632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95b0109e50 msgr2=0x7f95b01a1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:44.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.644+0000 7f95afe00640 1 --2- 172.21.15.28:0/3718443632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95b0109e50 0x7f95b01a1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:44.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.644+0000 7f95afe00640 1 -- 172.21.15.28:0/3718443632 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95b01a1fd0 con 0x7f95b0074a60 2024-04-23T14:41:44.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.644+0000 7f95af400640 1 --2- 172.21.15.28:0/3718443632 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95b0076040 0x7f95b019a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:41:44.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.644+0000 7f95b4c00640 1 --2- 172.21.15.28:0/3718443632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95b0109e50 0x7f95b01a1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:41:44.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.644+0000 7f95afe00640 1 --2- 172.21.15.28:0/3718443632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95b0074a60 0x7f95b019a3b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f95a000e9f0 tx=0x7f95a000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:44.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.644+0000 7f95acc00640 1 -- 172.21.15.28:0/3718443632 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a000ccc0 con 0x7f95b0074a60 2024-04-23T14:41:44.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.644+0000 7f95acc00640 1 -- 172.21.15.28:0/3718443632 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95a0010910 con 0x7f95b0074a60 2024-04-23T14:41:44.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.644+0000 7f95acc00640 1 -- 172.21.15.28:0/3718443632 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a0018960 con 0x7f95b0074a60 2024-04-23T14:41:44.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.644+0000 7f95b5600640 1 -- 172.21.15.28:0/3718443632 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95b01a22c0 con 0x7f95b0074a60 2024-04-23T14:41:44.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.644+0000 7f95b5600640 1 -- 172.21.15.28:0/3718443632 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95b01a2860 con 0x7f95b0074a60 2024-04-23T14:41:44.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.646+0000 7f95b5600640 1 -- 172.21.15.28:0/3718443632 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95b0108b80 con 0x7f95b0074a60 2024-04-23T14:41:44.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.648+0000 7f95acc00640 1 -- 172.21.15.28:0/3718443632 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f95a0010ac0 con 0x7f95b0074a60 2024-04-23T14:41:44.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.649+0000 7f95acc00640 1 --2- 172.21.15.28:0/3718443632 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f95940a3c30 0x7f95940a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:44.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.650+0000 7f95acc00640 1 -- 172.21.15.28:0/3718443632 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f95a00d0650 con 0x7f95b0074a60 2024-04-23T14:41:44.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.650+0000 7f95af400640 1 --2- 172.21.15.28:0/3718443632 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f95940a3c30 0x7f95940a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:44.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.650+0000 7f95af400640 1 --2- 172.21.15.28:0/3718443632 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f95940a3c30 0x7f95940a60f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f95b019b8c0 tx=0x7f9598005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:44.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.654+0000 7f95acc00640 1 -- 172.21.15.28:0/3718443632 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f95a009ca90 con 0x7f95b0074a60 2024-04-23T14:41:44.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:44 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1133573718' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-23T14:41:44.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:44 smithi105 ceph-mon[32555]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T14:41:44.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:44 smithi105 ceph-mon[32555]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-23T14:41:44.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:44 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2849659610' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-23T14:41:44.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:44 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1133573718' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-23T14:41:44.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:44 smithi038 ceph-mon[32348]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T14:41:44.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:44 smithi038 ceph-mon[32348]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-23T14:41:44.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:44 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2849659610' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-23T14:41:44.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:44 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1133573718' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-23T14:41:44.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:44 smithi028 ceph-mon[27810]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T14:41:44.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:44 smithi028 ceph-mon[27810]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-23T14:41:44.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:44 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2849659610' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-23T14:41:44.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.890+0000 7f95b5600640 1 -- 172.21.15.28:0/3718443632 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f95b0062440 con 0x7f95940a3c30 2024-04-23T14:41:44.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.921+0000 7f95acc00640 1 -- 172.21.15.28:0/3718443632 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f95b0062440 con 0x7f95940a3c30 2024-04-23T14:41:44.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.923+0000 7f95b5600640 1 -- 172.21.15.28:0/3718443632 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f95940a3c30 msgr2=0x7f95940a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:44.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.923+0000 7f95b5600640 1 --2- 172.21.15.28:0/3718443632 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f95940a3c30 0x7f95940a60f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f95b019b8c0 tx=0x7f9598005f90 comp rx=0 tx=0).stop 2024-04-23T14:41:44.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.923+0000 7f95b5600640 1 -- 172.21.15.28:0/3718443632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95b0074a60 msgr2=0x7f95b019a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:44.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.923+0000 7f95b5600640 1 --2- 172.21.15.28:0/3718443632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95b0074a60 0x7f95b019a3b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f95a000e9f0 tx=0x7f95a000eeb0 comp rx=0 tx=0).stop 2024-04-23T14:41:44.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.924+0000 7f95b5600640 1 -- 172.21.15.28:0/3718443632 shutdown_connections 2024-04-23T14:41:44.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.924+0000 7f95b5600640 1 --2- 172.21.15.28:0/3718443632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95b0109e50 0x7f95b01a1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:44.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.924+0000 7f95b5600640 1 --2- 172.21.15.28:0/3718443632 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95b0076040 0x7f95b019a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:44.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.924+0000 7f95b5600640 1 --2- 172.21.15.28:0/3718443632 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f95940a3c30 0x7f95940a60f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:44.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.924+0000 7f95b5600640 1 --2- 172.21.15.28:0/3718443632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95b0074a60 0x7f95b019a3b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:44.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.924+0000 7f95b5600640 1 -- 172.21.15.28:0/3718443632 >> 172.21.15.28:0/3718443632 conn(0x7f95b00fd500 msgr2=0x7f95b0105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:44.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.924+0000 7f95b5600640 1 -- 172.21.15.28:0/3718443632 shutdown_connections 2024-04-23T14:41:44.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:44.924+0000 7f95b5600640 1 -- 172.21.15.28:0/3718443632 wait complete. 2024-04-23T14:41:44.944 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-23T14:41:45.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.160+0000 7ff68ac00640 1 -- 172.21.15.28:0/2476907105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff684102ac0 msgr2=0x7ff684104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:45.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.160+0000 7ff68ac00640 1 --2- 172.21.15.28:0/2476907105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff684102ac0 0x7ff684104eb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff674009900 tx=0x7ff67402a410 comp rx=0 tx=0).stop 2024-04-23T14:41:45.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.160+0000 7ff68ac00640 1 -- 172.21.15.28:0/2476907105 shutdown_connections 2024-04-23T14:41:45.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.160+0000 7ff68ac00640 1 --2- 172.21.15.28:0/2476907105 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6841053f0 0x7ff6841077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:45.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.160+0000 7ff68ac00640 1 --2- 172.21.15.28:0/2476907105 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff684107d20 0x7ff68410a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:45.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.160+0000 7ff68ac00640 1 --2- 172.21.15.28:0/2476907105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff684102ac0 0x7ff684104eb0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:45.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.160+0000 7ff68ac00640 1 -- 172.21.15.28:0/2476907105 >> 172.21.15.28:0/2476907105 conn(0x7ff6840fc4b0 msgr2=0x7ff6840fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:45.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.160+0000 7ff68ac00640 1 -- 172.21.15.28:0/2476907105 shutdown_connections 2024-04-23T14:41:45.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.160+0000 7ff68ac00640 1 -- 172.21.15.28:0/2476907105 wait complete. 2024-04-23T14:41:45.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.161+0000 7ff68ac00640 1 Processor -- start 2024-04-23T14:41:45.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.161+0000 7ff68ac00640 1 -- start start 2024-04-23T14:41:45.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.162+0000 7ff68ac00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff684102ac0 0x7ff68419a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:45.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.162+0000 7ff68ac00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6841053f0 0x7ff68419ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:45.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.162+0000 7ff68ac00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff684107d20 0x7ff6841a1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:45.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.162+0000 7ff68ac00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff68410dd20 con 0x7ff6841053f0 2024-04-23T14:41:45.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.162+0000 7ff68ac00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff68410dba0 con 0x7ff684102ac0 2024-04-23T14:41:45.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.162+0000 7ff68ac00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff68410dea0 con 0x7ff684107d20 2024-04-23T14:41:45.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.162+0000 7ff683400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6841053f0 0x7ff68419ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:45.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.162+0000 7ff683400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6841053f0 0x7ff68419ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35460/0 (socket says 172.21.15.28:35460) 2024-04-23T14:41:45.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.162+0000 7ff683400640 1 -- 172.21.15.28:0/1689236759 learned_addr learned my addr 172.21.15.28:0/1689236759 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:45.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.162+0000 7ff683e00640 1 --2- 172.21.15.28:0/1689236759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff684102ac0 0x7ff68419a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:45.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.162+0000 7ff688c00640 1 --2- 172.21.15.28:0/1689236759 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff684107d20 0x7ff6841a1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:45.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.162+0000 7ff683400640 1 -- 172.21.15.28:0/1689236759 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff684107d20 msgr2=0x7ff6841a1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:45.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.162+0000 7ff683400640 1 --2- 172.21.15.28:0/1689236759 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff684107d20 0x7ff6841a1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:45.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.162+0000 7ff683400640 1 -- 172.21.15.28:0/1689236759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff684102ac0 msgr2=0x7ff68419a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:45.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.162+0000 7ff683400640 1 --2- 172.21.15.28:0/1689236759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff684102ac0 0x7ff68419a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:45.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.162+0000 7ff683400640 1 -- 172.21.15.28:0/1689236759 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6841a21f0 con 0x7ff6841053f0 2024-04-23T14:41:45.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.162+0000 7ff688c00640 1 --2- 172.21.15.28:0/1689236759 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff684107d20 0x7ff6841a1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:41:45.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.163+0000 7ff683400640 1 --2- 172.21.15.28:0/1689236759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6841053f0 0x7ff68419ab10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff66c00e9c0 tx=0x7ff66c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:45.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.163+0000 7ff680c00640 1 -- 172.21.15.28:0/1689236759 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff66c00cd80 con 0x7ff6841053f0 2024-04-23T14:41:45.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.163+0000 7ff680c00640 1 -- 172.21.15.28:0/1689236759 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff66c005500 con 0x7ff6841053f0 2024-04-23T14:41:45.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.163+0000 7ff68ac00640 1 -- 172.21.15.28:0/1689236759 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6841a24e0 con 0x7ff6841053f0 2024-04-23T14:41:45.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.163+0000 7ff680c00640 1 -- 172.21.15.28:0/1689236759 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff66c010690 con 0x7ff6841053f0 2024-04-23T14:41:45.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.163+0000 7ff68ac00640 1 -- 172.21.15.28:0/1689236759 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6841a2a30 con 0x7ff6841053f0 2024-04-23T14:41:45.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.165+0000 7ff68ac00640 1 -- 172.21.15.28:0/1689236759 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff684109870 con 0x7ff6841053f0 2024-04-23T14:41:45.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.170+0000 7ff680c00640 1 -- 172.21.15.28:0/1689236759 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff66c0046c0 con 0x7ff6841053f0 2024-04-23T14:41:45.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.171+0000 7ff680c00640 1 --2- 172.21.15.28:0/1689236759 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ff6640a3bb0 0x7ff6640a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:45.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.171+0000 7ff680c00640 1 -- 172.21.15.28:0/1689236759 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7ff66c0d0960 con 0x7ff6841053f0 2024-04-23T14:41:45.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.172+0000 7ff683e00640 1 --2- 172.21.15.28:0/1689236759 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ff6640a3bb0 0x7ff6640a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:45.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.172+0000 7ff683e00640 1 --2- 172.21.15.28:0/1689236759 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ff6640a3bb0 0x7ff6640a6070 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff674031440 tx=0x7ff6740315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:45.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.173+0000 7ff680c00640 1 -- 172.21.15.28:0/1689236759 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff66c053ff0 con 0x7ff6841053f0 2024-04-23T14:41:45.374 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:41:44 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:41:44.901+0000 7f21ed200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T14:41:45.374 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:41:44 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:41:44.901+0000 7f21ed200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T14:41:45.374 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:41:44 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:41:44.901+0000 7f21ed200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T14:41:45.375 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:41:44 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:41:44.901+0000 7f21ed200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T14:41:45.375 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:41:44 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:41:44.901+0000 7f21ed200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T14:41:45.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.413+0000 7ff68ac00640 1 -- 172.21.15.28:0/1689236759 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7ff684062440 con 0x7ff6640a3bb0 2024-04-23T14:41:45.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.452+0000 7ff680c00640 1 -- 172.21.15.28:0/1689236759 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff684062440 con 0x7ff6640a3bb0 2024-04-23T14:41:45.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.455+0000 7ff68ac00640 1 -- 172.21.15.28:0/1689236759 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ff6640a3bb0 msgr2=0x7ff6640a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:45.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.455+0000 7ff68ac00640 1 --2- 172.21.15.28:0/1689236759 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ff6640a3bb0 0x7ff6640a6070 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff674031440 tx=0x7ff6740315b0 comp rx=0 tx=0).stop 2024-04-23T14:41:45.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.455+0000 7ff68ac00640 1 -- 172.21.15.28:0/1689236759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6841053f0 msgr2=0x7ff68419ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:45.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.455+0000 7ff68ac00640 1 --2- 172.21.15.28:0/1689236759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6841053f0 0x7ff68419ab10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff66c00e9c0 tx=0x7ff66c00ee80 comp rx=0 tx=0).stop 2024-04-23T14:41:45.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.455+0000 7ff68ac00640 1 -- 172.21.15.28:0/1689236759 shutdown_connections 2024-04-23T14:41:45.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.455+0000 7ff68ac00640 1 --2- 172.21.15.28:0/1689236759 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff684107d20 0x7ff6841a1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:45.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.455+0000 7ff68ac00640 1 --2- 172.21.15.28:0/1689236759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6841053f0 0x7ff68419ab10 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:45.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.455+0000 7ff68ac00640 1 --2- 172.21.15.28:0/1689236759 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ff6640a3bb0 0x7ff6640a6070 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:45.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.455+0000 7ff68ac00640 1 --2- 172.21.15.28:0/1689236759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff684102ac0 0x7ff68419a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:45.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.455+0000 7ff68ac00640 1 -- 172.21.15.28:0/1689236759 >> 172.21.15.28:0/1689236759 conn(0x7ff6840fc4b0 msgr2=0x7ff684102230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:45.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.455+0000 7ff68ac00640 1 -- 172.21.15.28:0/1689236759 shutdown_connections 2024-04-23T14:41:45.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.455+0000 7ff68ac00640 1 -- 172.21.15.28:0/1689236759 wait complete. 2024-04-23T14:41:45.477 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-23T14:41:45.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.704+0000 7f3e3d800640 1 -- 172.21.15.28:0/4082905154 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e38074a60 msgr2=0x7f3e38074ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:45.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.704+0000 7f3e3d800640 1 --2- 172.21.15.28:0/4082905154 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e38074a60 0x7f3e38074ee0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3e24009860 tx=0x7f3e2402a340 comp rx=0 tx=0).stop 2024-04-23T14:41:45.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.704+0000 7f3e3d800640 1 -- 172.21.15.28:0/4082905154 shutdown_connections 2024-04-23T14:41:45.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.704+0000 7f3e3d800640 1 --2- 172.21.15.28:0/4082905154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e38109e50 0x7f3e3810c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:45.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.704+0000 7f3e3d800640 1 --2- 172.21.15.28:0/4082905154 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e38074a60 0x7f3e38074ee0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:45.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.704+0000 7f3e3d800640 1 --2- 172.21.15.28:0/4082905154 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e38076040 0x7f3e38074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:45.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.704+0000 7f3e3d800640 1 -- 172.21.15.28:0/4082905154 >> 172.21.15.28:0/4082905154 conn(0x7f3e380fd500 msgr2=0x7f3e380ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:45.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.704+0000 7f3e3d800640 1 -- 172.21.15.28:0/4082905154 shutdown_connections 2024-04-23T14:41:45.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.705+0000 7f3e3d800640 1 -- 172.21.15.28:0/4082905154 wait complete. 2024-04-23T14:41:45.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.706+0000 7f3e3d800640 1 Processor -- start 2024-04-23T14:41:45.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.706+0000 7f3e3d800640 1 -- start start 2024-04-23T14:41:45.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.706+0000 7f3e3d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e38074a60 0x7f3e3819a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:45.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.706+0000 7f3e3d800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e38076040 0x7f3e3819a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:45.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.706+0000 7f3e3d800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e38109e50 0x7f3e381a19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:45.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.706+0000 7f3e3d800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3e3810fda0 con 0x7f3e38074a60 2024-04-23T14:41:45.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.706+0000 7f3e3d800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3e3810fc20 con 0x7f3e38109e50 2024-04-23T14:41:45.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.706+0000 7f3e3d800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3e3810ff20 con 0x7f3e38076040 2024-04-23T14:41:45.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.707+0000 7f3e37400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e38076040 0x7f3e3819a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:45.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.707+0000 7f3e37e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e38074a60 0x7f3e3819a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:45.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.707+0000 7f3e37400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e38076040 0x7f3e3819a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:60106/0 (socket says 172.21.15.28:60106) 2024-04-23T14:41:45.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.707+0000 7f3e37400640 1 -- 172.21.15.28:0/623299831 learned_addr learned my addr 172.21.15.28:0/623299831 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:45.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.707+0000 7f3e3ce00640 1 --2- 172.21.15.28:0/623299831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e38109e50 0x7f3e381a19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:45.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.707+0000 7f3e37400640 1 -- 172.21.15.28:0/623299831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e38109e50 msgr2=0x7f3e381a19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:45.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.707+0000 7f3e37400640 1 --2- 172.21.15.28:0/623299831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e38109e50 0x7f3e381a19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:45.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.707+0000 7f3e37400640 1 -- 172.21.15.28:0/623299831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e38074a60 msgr2=0x7f3e3819a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:45.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.707+0000 7f3e37400640 1 --2- 172.21.15.28:0/623299831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e38074a60 0x7f3e3819a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:45.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.707+0000 7f3e37400640 1 -- 172.21.15.28:0/623299831 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e381a2040 con 0x7f3e38076040 2024-04-23T14:41:45.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.707+0000 7f3e3ce00640 1 --2- 172.21.15.28:0/623299831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e38109e50 0x7f3e381a19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:41:45.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.707+0000 7f3e37400640 1 --2- 172.21.15.28:0/623299831 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e38076040 0x7f3e3819a960 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3e24005940 tx=0x7f3e24033d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:45.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.708+0000 7f3e34c00640 1 -- 172.21.15.28:0/623299831 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e2403f020 con 0x7f3e38076040 2024-04-23T14:41:45.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.708+0000 7f3e34c00640 1 -- 172.21.15.28:0/623299831 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e24030850 con 0x7f3e38076040 2024-04-23T14:41:45.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.708+0000 7f3e34c00640 1 -- 172.21.15.28:0/623299831 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e24037040 con 0x7f3e38076040 2024-04-23T14:41:45.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.708+0000 7f3e3d800640 1 -- 172.21.15.28:0/623299831 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e381a22d0 con 0x7f3e38076040 2024-04-23T14:41:45.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.708+0000 7f3e3d800640 1 -- 172.21.15.28:0/623299831 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e381a2770 con 0x7f3e38076040 2024-04-23T14:41:45.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.709+0000 7f3e3d800640 1 -- 172.21.15.28:0/623299831 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e38108b80 con 0x7f3e38076040 2024-04-23T14:41:45.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.712+0000 7f3e34c00640 1 -- 172.21.15.28:0/623299831 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3e24042020 con 0x7f3e38076040 2024-04-23T14:41:45.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.713+0000 7f3e34c00640 1 --2- 172.21.15.28:0/623299831 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f3e1c0a3c30 0x7f3e1c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:45.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.713+0000 7f3e34c00640 1 -- 172.21.15.28:0/623299831 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f3e240f07d0 con 0x7f3e38076040 2024-04-23T14:41:45.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.714+0000 7f3e37e00640 1 --2- 172.21.15.28:0/623299831 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f3e1c0a3c30 0x7f3e1c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:45.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.714+0000 7f3e37e00640 1 --2- 172.21.15.28:0/623299831 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f3e1c0a3c30 0x7f3e1c0a60f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3e28009bd0 tx=0x7f3e28005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:45.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.716+0000 7f3e34c00640 1 -- 172.21.15.28:0/623299831 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3e24049020 con 0x7f3e38076040 2024-04-23T14:41:45.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:45 smithi105 ceph-mon[32555]: pgmap v316: 97 pgs: 25 creating+peering, 72 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 6 op/s 2024-04-23T14:41:45.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:45 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2849659610' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-23T14:41:45.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:45 smithi105 ceph-mon[32555]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-23T14:41:45.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:45 smithi038 ceph-mon[32348]: pgmap v316: 97 pgs: 25 creating+peering, 72 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 6 op/s 2024-04-23T14:41:45.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:45 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2849659610' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-23T14:41:45.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:45 smithi038 ceph-mon[32348]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-23T14:41:45.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:45 smithi028 ceph-mon[27810]: pgmap v316: 97 pgs: 25 creating+peering, 72 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 6 op/s 2024-04-23T14:41:45.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:45 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2849659610' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-23T14:41:45.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:45 smithi028 ceph-mon[27810]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-23T14:41:45.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.955+0000 7f3e3d800640 1 -- 172.21.15.28:0/623299831 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f3e380624d0 con 0x7f3e1c0a3c30 2024-04-23T14:41:45.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.971+0000 7f3e34c00640 1 -- 172.21.15.28:0/623299831 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f3e380624d0 con 0x7f3e1c0a3c30 2024-04-23T14:41:45.972 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/qa/sv_0/bf39579a-6785-4155-bdb7-db3f51f57b78 2024-04-23T14:41:45.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.974+0000 7f3e3d800640 1 -- 172.21.15.28:0/623299831 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f3e1c0a3c30 msgr2=0x7f3e1c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:45.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.974+0000 7f3e3d800640 1 --2- 172.21.15.28:0/623299831 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f3e1c0a3c30 0x7f3e1c0a60f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3e28009bd0 tx=0x7f3e28005f90 comp rx=0 tx=0).stop 2024-04-23T14:41:45.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.974+0000 7f3e3d800640 1 -- 172.21.15.28:0/623299831 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e38076040 msgr2=0x7f3e3819a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:45.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.974+0000 7f3e3d800640 1 --2- 172.21.15.28:0/623299831 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e38076040 0x7f3e3819a960 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3e24005940 tx=0x7f3e24033d90 comp rx=0 tx=0).stop 2024-04-23T14:41:45.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.974+0000 7f3e3d800640 1 -- 172.21.15.28:0/623299831 shutdown_connections 2024-04-23T14:41:45.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.974+0000 7f3e3d800640 1 --2- 172.21.15.28:0/623299831 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f3e1c0a3c30 0x7f3e1c0a60f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:45.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.974+0000 7f3e3d800640 1 --2- 172.21.15.28:0/623299831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e38109e50 0x7f3e381a19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:45.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.974+0000 7f3e3d800640 1 --2- 172.21.15.28:0/623299831 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e38076040 0x7f3e3819a960 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:45.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.974+0000 7f3e3d800640 1 --2- 172.21.15.28:0/623299831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e38074a60 0x7f3e3819a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:45.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.974+0000 7f3e3d800640 1 -- 172.21.15.28:0/623299831 >> 172.21.15.28:0/623299831 conn(0x7f3e380fd500 msgr2=0x7f3e38105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:45.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.974+0000 7f3e3d800640 1 -- 172.21.15.28:0/623299831 shutdown_connections 2024-04-23T14:41:45.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:45.974+0000 7f3e3d800640 1 -- 172.21.15.28:0/623299831 wait complete. 2024-04-23T14:41:45.996 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-23T14:41:46.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.210+0000 7f9f7ee00640 1 -- 172.21.15.28:0/2539016266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f78108540 msgr2=0x7f9f78108960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:46.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.210+0000 7f9f7ee00640 1 --2- 172.21.15.28:0/2539016266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f78108540 0x7f9f78108960 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9f68009ca0 tx=0x7f9f6802a8c0 comp rx=0 tx=0).stop 2024-04-23T14:41:46.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.210+0000 7f9f7ee00640 1 -- 172.21.15.28:0/2539016266 shutdown_connections 2024-04-23T14:41:46.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.210+0000 7f9f7ee00640 1 --2- 172.21.15.28:0/2539016266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f7810a0f0 0x7f9f78110980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:46.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.210+0000 7f9f7ee00640 1 --2- 172.21.15.28:0/2539016266 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f78109730 0x7f9f78109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:46.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.210+0000 7f9f7ee00640 1 --2- 172.21.15.28:0/2539016266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f78108540 0x7f9f78108960 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:46.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.210+0000 7f9f7ee00640 1 -- 172.21.15.28:0/2539016266 >> 172.21.15.28:0/2539016266 conn(0x7f9f780774d0 msgr2=0x7f9f78079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:46.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.210+0000 7f9f7ee00640 1 -- 172.21.15.28:0/2539016266 shutdown_connections 2024-04-23T14:41:46.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.210+0000 7f9f7ee00640 1 -- 172.21.15.28:0/2539016266 wait complete. 2024-04-23T14:41:46.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.211+0000 7f9f7ee00640 1 Processor -- start 2024-04-23T14:41:46.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.212+0000 7f9f7ee00640 1 -- start start 2024-04-23T14:41:46.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.212+0000 7f9f7ee00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f78108540 0x7f9f7819e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:46.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.212+0000 7f9f7ee00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f78109730 0x7f9f7819edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:46.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.212+0000 7f9f7ee00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f7810a0f0 0x7f9f781a5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:46.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.212+0000 7f9f7ee00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9f781141d0 con 0x7f9f78109730 2024-04-23T14:41:46.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.212+0000 7f9f7ee00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9f78114050 con 0x7f9f78108540 2024-04-23T14:41:46.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.212+0000 7f9f7ee00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9f78114350 con 0x7f9f7810a0f0 2024-04-23T14:41:46.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.212+0000 7f9f7d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f78109730 0x7f9f7819edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:46.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.212+0000 7f9f7d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f78109730 0x7f9f7819edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35502/0 (socket says 172.21.15.28:35502) 2024-04-23T14:41:46.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.212+0000 7f9f7d000640 1 -- 172.21.15.28:0/945156811 learned_addr learned my addr 172.21.15.28:0/945156811 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:46.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.212+0000 7f9f7da00640 1 --2- 172.21.15.28:0/945156811 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f78108540 0x7f9f7819e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:46.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.212+0000 7f9f7e400640 1 --2- 172.21.15.28:0/945156811 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f7810a0f0 0x7f9f781a5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:46.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.213+0000 7f9f7da00640 1 -- 172.21.15.28:0/945156811 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f7810a0f0 msgr2=0x7f9f781a5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:46.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.213+0000 7f9f7da00640 1 --2- 172.21.15.28:0/945156811 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f7810a0f0 0x7f9f781a5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:46.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.213+0000 7f9f7da00640 1 -- 172.21.15.28:0/945156811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f78109730 msgr2=0x7f9f7819edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:46.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.213+0000 7f9f7da00640 1 --2- 172.21.15.28:0/945156811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f78109730 0x7f9f7819edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:46.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.213+0000 7f9f7da00640 1 -- 172.21.15.28:0/945156811 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f781a64a0 con 0x7f9f78108540 2024-04-23T14:41:46.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.213+0000 7f9f7da00640 1 --2- 172.21.15.28:0/945156811 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f78108540 0x7f9f7819e880 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9f68009bf0 tx=0x7f9f68031ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:46.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.213+0000 7f9f76000640 1 -- 172.21.15.28:0/945156811 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f6802fae0 con 0x7f9f78108540 2024-04-23T14:41:46.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.214+0000 7f9f76000640 1 -- 172.21.15.28:0/945156811 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f6802fc80 con 0x7f9f78108540 2024-04-23T14:41:46.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.214+0000 7f9f7ee00640 1 -- 172.21.15.28:0/945156811 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f781a6730 con 0x7f9f78108540 2024-04-23T14:41:46.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.214+0000 7f9f76000640 1 -- 172.21.15.28:0/945156811 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f68046800 con 0x7f9f78108540 2024-04-23T14:41:46.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.214+0000 7f9f7ee00640 1 -- 172.21.15.28:0/945156811 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f781a6cd0 con 0x7f9f78108540 2024-04-23T14:41:46.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.215+0000 7f9f7ee00640 1 -- 172.21.15.28:0/945156811 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f7810f8b0 con 0x7f9f78108540 2024-04-23T14:41:46.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.217+0000 7f9f76000640 1 -- 172.21.15.28:0/945156811 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9f68030b30 con 0x7f9f78108540 2024-04-23T14:41:46.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.218+0000 7f9f76000640 1 --2- 172.21.15.28:0/945156811 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f9f5c0a3c30 0x7f9f5c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:46.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.218+0000 7f9f76000640 1 -- 172.21.15.28:0/945156811 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f9f680f1620 con 0x7f9f78108540 2024-04-23T14:41:46.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.219+0000 7f9f7d000640 1 --2- 172.21.15.28:0/945156811 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f9f5c0a3c30 0x7f9f5c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:46.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.219+0000 7f9f7d000640 1 --2- 172.21.15.28:0/945156811 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f9f5c0a3c30 0x7f9f5c0a60f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9f7819fd90 tx=0x7f9f64005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:46.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.223+0000 7f9f76000640 1 -- 172.21.15.28:0/945156811 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9f680be0b0 con 0x7f9f78108540 2024-04-23T14:41:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.432+0000 7f9f76000640 1 -- 172.21.15.28:0/945156811 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9f680ba020 con 0x7f9f78108540 2024-04-23T14:41:46.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.470+0000 7f9f7ee00640 1 -- 172.21.15.28:0/945156811 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f9f78062440 con 0x7f9f5c0a3c30 2024-04-23T14:41:46.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.508+0000 7f9f76000640 1 -- 172.21.15.28:0/945156811 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f9f78062440 con 0x7f9f5c0a3c30 2024-04-23T14:41:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.512+0000 7f9f7ee00640 1 -- 172.21.15.28:0/945156811 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f9f5c0a3c30 msgr2=0x7f9f5c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.512+0000 7f9f7ee00640 1 --2- 172.21.15.28:0/945156811 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f9f5c0a3c30 0x7f9f5c0a60f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9f7819fd90 tx=0x7f9f64005f90 comp rx=0 tx=0).stop 2024-04-23T14:41:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.512+0000 7f9f7ee00640 1 -- 172.21.15.28:0/945156811 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f78108540 msgr2=0x7f9f7819e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.512+0000 7f9f7ee00640 1 --2- 172.21.15.28:0/945156811 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f78108540 0x7f9f7819e880 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9f68009bf0 tx=0x7f9f68031ee0 comp rx=0 tx=0).stop 2024-04-23T14:41:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.512+0000 7f9f7ee00640 1 -- 172.21.15.28:0/945156811 shutdown_connections 2024-04-23T14:41:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.512+0000 7f9f7ee00640 1 --2- 172.21.15.28:0/945156811 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f7810a0f0 0x7f9f781a5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.512+0000 7f9f7ee00640 1 --2- 172.21.15.28:0/945156811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f78109730 0x7f9f7819edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.512+0000 7f9f7ee00640 1 --2- 172.21.15.28:0/945156811 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f9f5c0a3c30 0x7f9f5c0a60f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.512+0000 7f9f7ee00640 1 --2- 172.21.15.28:0/945156811 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f78108540 0x7f9f7819e880 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:46.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.512+0000 7f9f7ee00640 1 -- 172.21.15.28:0/945156811 >> 172.21.15.28:0/945156811 conn(0x7f9f780774d0 msgr2=0x7f9f7810ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:46.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.513+0000 7f9f7ee00640 1 -- 172.21.15.28:0/945156811 shutdown_connections 2024-04-23T14:41:46.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.513+0000 7f9f7ee00640 1 -- 172.21.15.28:0/945156811 wait complete. 2024-04-23T14:41:46.534 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-23T14:41:46.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.748+0000 7f920ea00640 1 -- 172.21.15.28:0/429009328 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92081029a0 msgr2=0x7f9208104d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:46.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.748+0000 7f920ea00640 1 --2- 172.21.15.28:0/429009328 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92081029a0 0x7f9208104d90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f91fc009900 tx=0x7f91fc02a410 comp rx=0 tx=0).stop 2024-04-23T14:41:46.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.748+0000 7f920ea00640 1 -- 172.21.15.28:0/429009328 shutdown_connections 2024-04-23T14:41:46.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.748+0000 7f920ea00640 1 --2- 172.21.15.28:0/429009328 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92081052d0 0x7f92081076c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:46.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.748+0000 7f920ea00640 1 --2- 172.21.15.28:0/429009328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9208107c00 0x7f920810a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:46.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.748+0000 7f920ea00640 1 --2- 172.21.15.28:0/429009328 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92081029a0 0x7f9208104d90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:46.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.748+0000 7f920ea00640 1 -- 172.21.15.28:0/429009328 >> 172.21.15.28:0/429009328 conn(0x7f92080fc3b0 msgr2=0x7f92080fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:46.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.748+0000 7f920ea00640 1 -- 172.21.15.28:0/429009328 shutdown_connections 2024-04-23T14:41:46.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.748+0000 7f920ea00640 1 -- 172.21.15.28:0/429009328 wait complete. 2024-04-23T14:41:46.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.750+0000 7f920ea00640 1 Processor -- start 2024-04-23T14:41:46.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.750+0000 7f920ea00640 1 -- start start 2024-04-23T14:41:46.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.750+0000 7f920ea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92081029a0 0x7f920819a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:46.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.750+0000 7f920ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92081052d0 0x7f920819a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:46.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.750+0000 7f920ea00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9208107c00 0x7f92081a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:46.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.750+0000 7f920ea00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f920810db90 con 0x7f92081052d0 2024-04-23T14:41:46.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.751+0000 7f920ea00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f920810da10 con 0x7f92081029a0 2024-04-23T14:41:46.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.751+0000 7f920d600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92081029a0 0x7f920819a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:46.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.751+0000 7f920ea00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f920810dd10 con 0x7f9208107c00 2024-04-23T14:41:46.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.751+0000 7f920d600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92081029a0 0x7f920819a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48500/0 (socket says 172.21.15.28:48500) 2024-04-23T14:41:46.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.751+0000 7f920d600640 1 -- 172.21.15.28:0/3715335868 learned_addr learned my addr 172.21.15.28:0/3715335868 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:46.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.751+0000 7f920cc00640 1 --2- 172.21.15.28:0/3715335868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92081052d0 0x7f920819a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:46.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.751+0000 7f920e000640 1 --2- 172.21.15.28:0/3715335868 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9208107c00 0x7f92081a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:46.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.751+0000 7f920d600640 1 -- 172.21.15.28:0/3715335868 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9208107c00 msgr2=0x7f92081a1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:46.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.751+0000 7f920d600640 1 --2- 172.21.15.28:0/3715335868 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9208107c00 0x7f92081a1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:46.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.751+0000 7f920d600640 1 -- 172.21.15.28:0/3715335868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92081052d0 msgr2=0x7f920819a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:46.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.751+0000 7f920d600640 1 --2- 172.21.15.28:0/3715335868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92081052d0 0x7f920819a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:46.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.751+0000 7f920d600640 1 -- 172.21.15.28:0/3715335868 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92081a2080 con 0x7f92081029a0 2024-04-23T14:41:46.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.751+0000 7f920d600640 1 --2- 172.21.15.28:0/3715335868 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92081029a0 0x7f920819a460 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f91fc009230 tx=0x7f91fc0042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:46.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.752+0000 7f9206000640 1 -- 172.21.15.28:0/3715335868 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91fc03f030 con 0x7f92081029a0 2024-04-23T14:41:46.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.752+0000 7f9206000640 1 -- 172.21.15.28:0/3715335868 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91fc003ce0 con 0x7f92081029a0 2024-04-23T14:41:46.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.752+0000 7f9206000640 1 -- 172.21.15.28:0/3715335868 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91fc035070 con 0x7f92081029a0 2024-04-23T14:41:46.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.752+0000 7f920ea00640 1 -- 172.21.15.28:0/3715335868 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92081a2310 con 0x7f92081029a0 2024-04-23T14:41:46.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.752+0000 7f920ea00640 1 -- 172.21.15.28:0/3715335868 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92081a2850 con 0x7f92081029a0 2024-04-23T14:41:46.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.753+0000 7f920ea00640 1 -- 172.21.15.28:0/3715335868 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9208109750 con 0x7f92081029a0 2024-04-23T14:41:46.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.755+0000 7f9206000640 1 -- 172.21.15.28:0/3715335868 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f91fc042030 con 0x7f92081029a0 2024-04-23T14:41:46.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.756+0000 7f9206000640 1 --2- 172.21.15.28:0/3715335868 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f91f00a3cc0 0x7f91f00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:46.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.757+0000 7f9206000640 1 -- 172.21.15.28:0/3715335868 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f91fc0f0970 con 0x7f92081029a0 2024-04-23T14:41:46.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.757+0000 7f920cc00640 1 --2- 172.21.15.28:0/3715335868 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f91f00a3cc0 0x7f91f00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:46.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.757+0000 7f920cc00640 1 --2- 172.21.15.28:0/3715335868 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f91f00a3cc0 0x7f91f00a6180 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f91f40095f0 tx=0x7f91f4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:46.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:46.761+0000 7f9206000640 1 -- 172.21.15.28:0/3715335868 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f91fc0bd2a0 con 0x7f92081029a0 2024-04-23T14:41:46.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:46 smithi105 ceph-mon[32555]: from='client.24665 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:41:46.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:46 smithi105 ceph-mon[32555]: from='client.14553 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:41:46.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:46 smithi038 ceph-mon[32348]: from='client.24665 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:41:46.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:46 smithi038 ceph-mon[32348]: from='client.14553 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:41:46.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:46 smithi028 ceph-mon[27810]: from='client.24665 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:41:46.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:46 smithi028 ceph-mon[27810]: from='client.14553 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:41:47.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.002+0000 7f920ea00640 1 -- 172.21.15.28:0/3715335868 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f92080624d0 con 0x7f91f00a3cc0 2024-04-23T14:41:47.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.011+0000 7f9206000640 1 -- 172.21.15.28:0/3715335868 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f92080624d0 con 0x7f91f00a3cc0 2024-04-23T14:41:47.012 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/qa/sv_1/fe56c074-80ef-44de-886c-d211b0a30b1b 2024-04-23T14:41:47.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.013+0000 7f920ea00640 1 -- 172.21.15.28:0/3715335868 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f91f00a3cc0 msgr2=0x7f91f00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:47.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.013+0000 7f920ea00640 1 --2- 172.21.15.28:0/3715335868 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f91f00a3cc0 0x7f91f00a6180 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f91f40095f0 tx=0x7f91f4005f90 comp rx=0 tx=0).stop 2024-04-23T14:41:47.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.013+0000 7f920ea00640 1 -- 172.21.15.28:0/3715335868 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92081029a0 msgr2=0x7f920819a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:47.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.013+0000 7f920ea00640 1 --2- 172.21.15.28:0/3715335868 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92081029a0 0x7f920819a460 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f91fc009230 tx=0x7f91fc0042f0 comp rx=0 tx=0).stop 2024-04-23T14:41:47.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.014+0000 7f920ea00640 1 -- 172.21.15.28:0/3715335868 shutdown_connections 2024-04-23T14:41:47.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.014+0000 7f920ea00640 1 --2- 172.21.15.28:0/3715335868 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9208107c00 0x7f92081a1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:47.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.014+0000 7f920ea00640 1 --2- 172.21.15.28:0/3715335868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92081052d0 0x7f920819a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:47.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.014+0000 7f920ea00640 1 --2- 172.21.15.28:0/3715335868 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f91f00a3cc0 0x7f91f00a6180 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:47.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.014+0000 7f920ea00640 1 --2- 172.21.15.28:0/3715335868 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92081029a0 0x7f920819a460 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:47.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.014+0000 7f920ea00640 1 -- 172.21.15.28:0/3715335868 >> 172.21.15.28:0/3715335868 conn(0x7f92080fc3b0 msgr2=0x7f9208102190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:47.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.014+0000 7f920ea00640 1 -- 172.21.15.28:0/3715335868 shutdown_connections 2024-04-23T14:41:47.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.014+0000 7f920ea00640 1 -- 172.21.15.28:0/3715335868 wait complete. 2024-04-23T14:41:47.035 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:41:47.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.245+0000 7ff121a00640 1 -- 172.21.15.28:0/2268007760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff11c107b50 msgr2=0x7ff11c109fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:47.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.245+0000 7ff121a00640 1 --2- 172.21.15.28:0/2268007760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff11c107b50 0x7ff11c109fa0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff114009ca0 tx=0x7ff11402a8c0 comp rx=0 tx=0).stop 2024-04-23T14:41:47.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.245+0000 7ff121a00640 1 -- 172.21.15.28:0/2268007760 shutdown_connections 2024-04-23T14:41:47.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.245+0000 7ff121a00640 1 --2- 172.21.15.28:0/2268007760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff11c107b50 0x7ff11c109fa0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:47.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.245+0000 7ff121a00640 1 --2- 172.21.15.28:0/2268007760 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff11c105220 0x7ff11c107610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:47.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.245+0000 7ff121a00640 1 --2- 172.21.15.28:0/2268007760 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff11c1028f0 0x7ff11c104ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:47.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.245+0000 7ff121a00640 1 -- 172.21.15.28:0/2268007760 >> 172.21.15.28:0/2268007760 conn(0x7ff11c0fc2e0 msgr2=0x7ff11c0fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:47.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.245+0000 7ff121a00640 1 -- 172.21.15.28:0/2268007760 shutdown_connections 2024-04-23T14:41:47.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.245+0000 7ff121a00640 1 -- 172.21.15.28:0/2268007760 wait complete. 2024-04-23T14:41:47.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.246+0000 7ff121a00640 1 Processor -- start 2024-04-23T14:41:47.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.246+0000 7ff121a00640 1 -- start start 2024-04-23T14:41:47.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.247+0000 7ff121a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff11c1028f0 0x7ff11c102260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:47.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.247+0000 7ff121a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff11c105220 0x7ff11c1008a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:47.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.247+0000 7ff121a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff11c107b50 0x7ff11c100de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:47.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.247+0000 7ff121a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff11c10db10 con 0x7ff11c1028f0 2024-04-23T14:41:47.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.247+0000 7ff121a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff11c10d990 con 0x7ff11c105220 2024-04-23T14:41:47.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.247+0000 7ff121a00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff11c10dc90 con 0x7ff11c107b50 2024-04-23T14:41:47.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.247+0000 7ff11be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff11c1028f0 0x7ff11c102260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:47.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.247+0000 7ff11be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff11c1028f0 0x7ff11c102260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35520/0 (socket says 172.21.15.28:35520) 2024-04-23T14:41:47.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.247+0000 7ff11be00640 1 -- 172.21.15.28:0/1936076258 learned_addr learned my addr 172.21.15.28:0/1936076258 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:47.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.247+0000 7ff11b400640 1 --2- 172.21.15.28:0/1936076258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff11c105220 0x7ff11c1008a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:47.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.247+0000 7ff121000640 1 --2- 172.21.15.28:0/1936076258 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff11c107b50 0x7ff11c100de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:47.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.248+0000 7ff11be00640 1 -- 172.21.15.28:0/1936076258 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff11c107b50 msgr2=0x7ff11c100de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:47.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.248+0000 7ff11be00640 1 --2- 172.21.15.28:0/1936076258 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff11c107b50 0x7ff11c100de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:47.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.248+0000 7ff11be00640 1 -- 172.21.15.28:0/1936076258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff11c105220 msgr2=0x7ff11c1008a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:47.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.248+0000 7ff11be00640 1 --2- 172.21.15.28:0/1936076258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff11c105220 0x7ff11c1008a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:47.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.248+0000 7ff11be00640 1 -- 172.21.15.28:0/1936076258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff11c101670 con 0x7ff11c1028f0 2024-04-23T14:41:47.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.248+0000 7ff11b400640 1 --2- 172.21.15.28:0/1936076258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff11c105220 0x7ff11c1008a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:41:47.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.248+0000 7ff11be00640 1 --2- 172.21.15.28:0/1936076258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff11c1028f0 0x7ff11c102260 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff10c00e9f0 tx=0x7ff10c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:47.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.248+0000 7ff118c00640 1 -- 172.21.15.28:0/1936076258 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff10c00cd80 con 0x7ff11c1028f0 2024-04-23T14:41:47.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.248+0000 7ff121a00640 1 -- 172.21.15.28:0/1936076258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff11c1a22a0 con 0x7ff11c1028f0 2024-04-23T14:41:47.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.248+0000 7ff118c00640 1 -- 172.21.15.28:0/1936076258 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff10c010910 con 0x7ff11c1028f0 2024-04-23T14:41:47.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.248+0000 7ff118c00640 1 -- 172.21.15.28:0/1936076258 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff10c018990 con 0x7ff11c1028f0 2024-04-23T14:41:47.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.248+0000 7ff121a00640 1 -- 172.21.15.28:0/1936076258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff11c1a27f0 con 0x7ff11c1028f0 2024-04-23T14:41:47.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.250+0000 7ff121a00640 1 -- 172.21.15.28:0/1936076258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff11c0fddd0 con 0x7ff11c1028f0 2024-04-23T14:41:47.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.252+0000 7ff118c00640 1 -- 172.21.15.28:0/1936076258 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff10c010ac0 con 0x7ff11c1028f0 2024-04-23T14:41:47.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.253+0000 7ff118c00640 1 --2- 172.21.15.28:0/1936076258 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ff1000a3c40 0x7ff1000a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:47.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.253+0000 7ff118c00640 1 -- 172.21.15.28:0/1936076258 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7ff10c0d0800 con 0x7ff11c1028f0 2024-04-23T14:41:47.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.254+0000 7ff11b400640 1 --2- 172.21.15.28:0/1936076258 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ff1000a3c40 0x7ff1000a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:47.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.254+0000 7ff11b400640 1 --2- 172.21.15.28:0/1936076258 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ff1000a3c40 0x7ff1000a6100 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff11c101fd0 tx=0x7ff108005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:47.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.258+0000 7ff118c00640 1 -- 172.21.15.28:0/1936076258 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff10c09cae0 con 0x7ff11c1028f0 2024-04-23T14:41:47.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.516+0000 7ff121a00640 1 -- 172.21.15.28:0/1936076258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff11c02d620 con 0x7ff11c1028f0 2024-04-23T14:41:47.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.518+0000 7ff118c00640 1 -- 172.21.15.28:0/1936076258 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7ff10c0a0de0 con 0x7ff11c1028f0 2024-04-23T14:41:47.519 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:41:47.519 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:41:47.519 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:41:47.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.521+0000 7ff121a00640 1 -- 172.21.15.28:0/1936076258 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ff1000a3c40 msgr2=0x7ff1000a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:47.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.521+0000 7ff121a00640 1 --2- 172.21.15.28:0/1936076258 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ff1000a3c40 0x7ff1000a6100 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff11c101fd0 tx=0x7ff108005f90 comp rx=0 tx=0).stop 2024-04-23T14:41:47.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.521+0000 7ff121a00640 1 -- 172.21.15.28:0/1936076258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff11c1028f0 msgr2=0x7ff11c102260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:47.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.521+0000 7ff121a00640 1 --2- 172.21.15.28:0/1936076258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff11c1028f0 0x7ff11c102260 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff10c00e9f0 tx=0x7ff10c00eeb0 comp rx=0 tx=0).stop 2024-04-23T14:41:47.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.521+0000 7ff121a00640 1 -- 172.21.15.28:0/1936076258 shutdown_connections 2024-04-23T14:41:47.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.521+0000 7ff121a00640 1 --2- 172.21.15.28:0/1936076258 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ff1000a3c40 0x7ff1000a6100 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:47.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.521+0000 7ff121a00640 1 --2- 172.21.15.28:0/1936076258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff11c105220 0x7ff11c1008a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:47.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.521+0000 7ff121a00640 1 --2- 172.21.15.28:0/1936076258 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff11c107b50 0x7ff11c100de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:47.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.521+0000 7ff121a00640 1 --2- 172.21.15.28:0/1936076258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff11c1028f0 0x7ff11c102260 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:47.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.521+0000 7ff121a00640 1 -- 172.21.15.28:0/1936076258 >> 172.21.15.28:0/1936076258 conn(0x7ff11c0fc2e0 msgr2=0x7ff11c103230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:47.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.521+0000 7ff121a00640 1 -- 172.21.15.28:0/1936076258 shutdown_connections 2024-04-23T14:41:47.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.521+0000 7ff121a00640 1 -- 172.21.15.28:0/1936076258 wait complete. 2024-04-23T14:41:47.544 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:41:47.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.761+0000 7f44f8c00640 1 -- 172.21.15.28:0/2893906241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44f4104280 msgr2=0x7f44f41046a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:47.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.761+0000 7f44f8c00640 1 --2- 172.21.15.28:0/2893906241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44f4104280 0x7f44f41046a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f44e4009ca0 tx=0x7f44e402a8c0 comp rx=0 tx=0).stop 2024-04-23T14:41:47.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.761+0000 7f44f8c00640 1 -- 172.21.15.28:0/2893906241 shutdown_connections 2024-04-23T14:41:47.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.761+0000 7f44f8c00640 1 --2- 172.21.15.28:0/2893906241 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44f4105470 0x7f44f41058f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:47.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.761+0000 7f44f8c00640 1 --2- 172.21.15.28:0/2893906241 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f44f4105e30 0x7f44f410c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:47.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.761+0000 7f44f8c00640 1 --2- 172.21.15.28:0/2893906241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44f4104280 0x7f44f41046a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:47.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.761+0000 7f44f8c00640 1 -- 172.21.15.28:0/2893906241 >> 172.21.15.28:0/2893906241 conn(0x7f44f40ff7f0 msgr2=0x7f44f4101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:47.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.761+0000 7f44f8c00640 1 -- 172.21.15.28:0/2893906241 shutdown_connections 2024-04-23T14:41:47.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.761+0000 7f44f8c00640 1 -- 172.21.15.28:0/2893906241 wait complete. 2024-04-23T14:41:47.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.762+0000 7f44f8c00640 1 Processor -- start 2024-04-23T14:41:47.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.762+0000 7f44f8c00640 1 -- start start 2024-04-23T14:41:47.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.763+0000 7f44f8c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f44f4104280 0x7f44f419a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:47.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.763+0000 7f44f8c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44f4105470 0x7f44f419aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:47.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.763+0000 7f44f8c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44f4105e30 0x7f44f41a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:47.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.763+0000 7f44f8c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f44f410ff10 con 0x7f44f4105470 2024-04-23T14:41:47.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.763+0000 7f44f8c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f44f410fd90 con 0x7f44f4105e30 2024-04-23T14:41:47.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.763+0000 7f44f8c00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f44f4110090 con 0x7f44f4104280 2024-04-23T14:41:47.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.763+0000 7f44f1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44f4105470 0x7f44f419aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:47.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.763+0000 7f44f1e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f44f4104280 0x7f44f419a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:47.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.763+0000 7f44f1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44f4105470 0x7f44f419aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35528/0 (socket says 172.21.15.28:35528) 2024-04-23T14:41:47.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.763+0000 7f44f1400640 1 -- 172.21.15.28:0/2409072573 learned_addr learned my addr 172.21.15.28:0/2409072573 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:47.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.763+0000 7f44f2800640 1 --2- 172.21.15.28:0/2409072573 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44f4105e30 0x7f44f41a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:47.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.763+0000 7f44f1e00640 1 -- 172.21.15.28:0/2409072573 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44f4105e30 msgr2=0x7f44f41a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:47.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.763+0000 7f44f1e00640 1 --2- 172.21.15.28:0/2409072573 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44f4105e30 0x7f44f41a1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:47.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.763+0000 7f44f1e00640 1 -- 172.21.15.28:0/2409072573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44f4105470 msgr2=0x7f44f419aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:47.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.763+0000 7f44f1e00640 1 --2- 172.21.15.28:0/2409072573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44f4105470 0x7f44f419aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:47.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.763+0000 7f44f1e00640 1 -- 172.21.15.28:0/2409072573 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44f41a21b0 con 0x7f44f4104280 2024-04-23T14:41:47.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.764+0000 7f44f2800640 1 --2- 172.21.15.28:0/2409072573 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44f4105e30 0x7f44f41a1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:41:47.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.764+0000 7f44f1400640 1 --2- 172.21.15.28:0/2409072573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44f4105470 0x7f44f419aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:41:47.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.764+0000 7f44f1e00640 1 --2- 172.21.15.28:0/2409072573 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f44f4104280 0x7f44f419a590 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f44e4009bf0 tx=0x7f44e4031ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:47.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.764+0000 7f44eaa00640 1 -- 172.21.15.28:0/2409072573 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e402fa20 con 0x7f44f4104280 2024-04-23T14:41:47.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.764+0000 7f44f8c00640 1 -- 172.21.15.28:0/2409072573 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44f41a2440 con 0x7f44f4104280 2024-04-23T14:41:47.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.764+0000 7f44eaa00640 1 -- 172.21.15.28:0/2409072573 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44e402fbc0 con 0x7f44f4104280 2024-04-23T14:41:47.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.764+0000 7f44eaa00640 1 -- 172.21.15.28:0/2409072573 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e4046750 con 0x7f44f4104280 2024-04-23T14:41:47.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.764+0000 7f44f8c00640 1 -- 172.21.15.28:0/2409072573 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44f41a2990 con 0x7f44f4104280 2024-04-23T14:41:47.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.766+0000 7f44f8c00640 1 -- 172.21.15.28:0/2409072573 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44f410b620 con 0x7f44f4104280 2024-04-23T14:41:47.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.767+0000 7f44eaa00640 1 -- 172.21.15.28:0/2409072573 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f44e402fd70 con 0x7f44f4104280 2024-04-23T14:41:47.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.769+0000 7f44eaa00640 1 --2- 172.21.15.28:0/2409072573 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f44d80a3c40 0x7f44d80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:47.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.769+0000 7f44eaa00640 1 -- 172.21.15.28:0/2409072573 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f44e40f16d0 con 0x7f44f4104280 2024-04-23T14:41:47.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.769+0000 7f44f1400640 1 --2- 172.21.15.28:0/2409072573 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f44d80a3c40 0x7f44d80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:47.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.770+0000 7f44f1400640 1 --2- 172.21.15.28:0/2409072573 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f44d80a3c40 0x7f44d80a6100 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f44e00095f0 tx=0x7f44e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:47.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.773+0000 7f44eaa00640 1 -- 172.21.15.28:0/2409072573 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f44e40be110 con 0x7f44f4104280 2024-04-23T14:41:47.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:47 smithi105 ceph-mon[32555]: pgmap v317: 97 pgs: 97 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 459 B/s rd, 2.1 KiB/s wr, 7 op/s 2024-04-23T14:41:47.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:47 smithi105 ceph-mon[32555]: from='client.24700 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:41:47.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:47 smithi105 ceph-mon[32555]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-23T14:41:47.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:47 smithi105 ceph-mon[32555]: from='client.24689 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:41:47.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:47 smithi038 ceph-mon[32348]: pgmap v317: 97 pgs: 97 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 459 B/s rd, 2.1 KiB/s wr, 7 op/s 2024-04-23T14:41:47.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:47 smithi038 ceph-mon[32348]: from='client.24700 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:41:47.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:47 smithi038 ceph-mon[32348]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-23T14:41:47.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:47 smithi038 ceph-mon[32348]: from='client.24689 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:41:47.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:47 smithi028 ceph-mon[27810]: pgmap v317: 97 pgs: 97 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 459 B/s rd, 2.1 KiB/s wr, 7 op/s 2024-04-23T14:41:47.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:47 smithi028 ceph-mon[27810]: from='client.24700 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:41:47.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:47 smithi028 ceph-mon[27810]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-23T14:41:47.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:47 smithi028 ceph-mon[27810]: from='client.24689 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:41:47.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.982+0000 7f44f8c00640 1 -- 172.21.15.28:0/2409072573 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f44f4062c00 con 0x7f44f4104280 2024-04-23T14:41:47.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.984+0000 7f44eaa00640 1 -- 172.21.15.28:0/2409072573 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22170 (secure 0 0 0) 0x7f44e40bda10 con 0x7f44f4104280 2024-04-23T14:41:47.985 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:41:47.985 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":85,"fsid":"1ee6e996-017e-11ef-bc93-c7b262605968","created":"2024-04-23T14:32:34.581082+0000","modified":"2024-04-23T14:41:41.010290+0000","last_up_change":"2024-04-23T14:40:43.375846+0000","last_in_change":"2024-04-23T14:40:23.576939+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:36:27.698112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:41:37.030341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:41:38.444742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e0b427bc-be65-4543-99cc-2f53ad98ddad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6803","nonce":2993445336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6805","nonce":2993445336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6809","nonce":2993445336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6807","nonce":2993445336}]},"public_addr":"172.21.15.28:6803/2993445336","cluster_addr":"172.21.15.28:6805/2993445336","heartbeat_back_addr":"172.21.15.28:6809/2993445336","heartbeat_front_addr":"172.21.15.28:6807/2993445336","state":["exists","up"]},{"osd":1,"uuid":"ef527ec9-1bf3-4bb0-8df3-f5a015af557c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6801","nonce":4024093383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6803","nonce":4024093383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6807","nonce":4024093383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6805","nonce":4024093383}]},"public_addr":"172.21.15.38:6801/4024093383","cluster_addr":"172.21.15.38:6803/4024093383","heartbeat_back_addr":"172.21.15.38:6807/4024093383","heartbeat_front_addr":"172.21.15.38:6805/4024093383","state":["exists","up"]},{"osd":2,"uuid":"b33b23d9-adce-4c11-b304-2e117bdf6bfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6801","nonce":570739029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6803","nonce":570739029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6807","nonce":570739029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6805","nonce":570739029}]},"public_addr":"172.21.15.105:6801/570739029","cluster_addr":"172.21.15.105:6803/570739029","heartbeat_back_addr":"172.21.15.105:6807/570739029","heartbeat_front_addr":"172.21.15.105:6805/570739029","state":["exists","up"]},{"osd":3,"uuid":"28bd5608-b426-434e-9db1-bdaa9828bdb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6811","nonce":2480400247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6813","nonce":2480400247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6817","nonce":2480400247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6815","nonce":2480400247}]},"public_addr":"172.21.15.28:6811/2480400247","cluster_addr":"172.21.15.28:6813/2480400247","heartbeat_back_addr":"172.21.15.28:6817/2480400247","heartbeat_front_addr":"172.21.15.28:6815/2480400247","state":["exists","up"]},{"osd":4,"uuid":"e5d1c89b-025f-4d57-8807-59d75c787b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6809","nonce":3772189246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6811","nonce":3772189246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6815","nonce":3772189246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6813","nonce":3772189246}]},"public_addr":"172.21.15.38:6809/3772189246","cluster_addr":"172.21.15.38:6811/3772189246","heartbeat_back_addr":"172.21.15.38:6815/3772189246","heartbeat_front_addr":"172.21.15.38:6813/3772189246","state":["exists","up"]},{"osd":5,"uuid":"9cf9cf4c-aafd-4ed3-bfb1-0527710a0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6809","nonce":928755894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6811","nonce":928755894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6815","nonce":928755894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6813","nonce":928755894}]},"public_addr":"172.21.15.105:6809/928755894","cluster_addr":"172.21.15.105:6811/928755894","heartbeat_back_addr":"172.21.15.105:6815/928755894","heartbeat_front_addr":"172.21.15.105:6813/928755894","state":["exists","up"]},{"osd":6,"uuid":"304db525-d24d-4687-bfa0-f92bdf66d1e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6819","nonce":897979441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6821","nonce":897979441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6825","nonce":897979441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6823","nonce":897979441}]},"public_addr":"172.21.15.28:6819/897979441","cluster_addr":"172.21.15.28:6821/897979441","heartbeat_back_addr":"172.21.15.28:6825/897979441","heartbeat_front_addr":"172.21.15.28:6823/897979441","state":["exists","up"]},{"osd":7,"uuid":"08e01d59-03f3-47b1-b45f-35ec5a50992c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6817","nonce":2400251786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6819","nonce":2400251786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6823","nonce":2400251786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6821","nonce":2400251786}]},"public_addr":"172.21.15.38:6817/2400251786","cluster_addr":"172.21.15.38:6819/2400251786","heartbeat_back_addr":"172.21.15.38:6823/2400251786","heartbeat_front_addr":"172.21.15.38:6821/2400251786","state":["exists","up"]},{"osd":8,"uuid":"64ee1e95-0756-48b1-9c00-472d0c3cbd29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6817","nonce":1682312269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6819","nonce":1682312269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6823","nonce":1682312269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6821","nonce":1682312269}]},"public_addr":"172.21.15.105:6817/1682312269","cluster_addr":"172.21.15.105:6819/1682312269","heartbeat_back_addr":"172.21.15.105:6823/1682312269","heartbeat_front_addr":"172.21.15.105:6821/1682312269","state":["exists","up"]},{"osd":9,"uuid":"18f60893-3ffd-48c2-9db5-a67d35611d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6827","nonce":1292942516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6829","nonce":1292942516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6833","nonce":1292942516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6831","nonce":1292942516}]},"public_addr":"172.21.15.28:6827/1292942516","cluster_addr":"172.21.15.28:6829/1292942516","heartbeat_back_addr":"172.21.15.28:6833/1292942516","heartbeat_front_addr":"172.21.15.28:6831/1292942516","state":["exists","up"]},{"osd":10,"uuid":"76c06725-7040-4118-8894-fe808a87f999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6825","nonce":2444616468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6827","nonce":2444616468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6831","nonce":2444616468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6829","nonce":2444616468}]},"public_addr":"172.21.15.38:6825/2444616468","cluster_addr":"172.21.15.38:6827/2444616468","heartbeat_back_addr":"172.21.15.38:6831/2444616468","heartbeat_front_addr":"172.21.15.38:6829/2444616468","state":["exists","up"]},{"osd":11,"uuid":"e8fc9454-5540-4e2e-91bf-3ba251dbd1a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6825","nonce":3400299562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6827","nonce":3400299562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6831","nonce":3400299562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6829","nonce":3400299562}]},"public_addr":"172.21.15.105:6825/3400299562","cluster_addr":"172.21.15.105:6827/3400299562","heartbeat_back_addr":"172.21.15.105:6831/3400299562","heartbeat_front_addr":"172.21.15.105:6829/3400299562","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:35:33.615395+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:00.545293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:24.940211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:57.684887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:24.519061+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:49.010804+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:22.638478+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:49.559104+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:14.011771+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:47.859950+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:16.298445+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:41.619069+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/590495740":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/1163172512":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/4290328460":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/4145020172":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/720626958":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/948450262":"2024-04-24T14:33:07.576363+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:41:47.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.986+0000 7f44f8c00640 1 -- 172.21.15.28:0/2409072573 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f44d80a3c40 msgr2=0x7f44d80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:47.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.986+0000 7f44f8c00640 1 --2- 172.21.15.28:0/2409072573 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f44d80a3c40 0x7f44d80a6100 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f44e00095f0 tx=0x7f44e0005f90 comp rx=0 tx=0).stop 2024-04-23T14:41:47.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.986+0000 7f44f8c00640 1 -- 172.21.15.28:0/2409072573 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f44f4104280 msgr2=0x7f44f419a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:47.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.986+0000 7f44f8c00640 1 --2- 172.21.15.28:0/2409072573 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f44f4104280 0x7f44f419a590 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f44e4009bf0 tx=0x7f44e4031ee0 comp rx=0 tx=0).stop 2024-04-23T14:41:47.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.987+0000 7f44f8c00640 1 -- 172.21.15.28:0/2409072573 shutdown_connections 2024-04-23T14:41:47.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.987+0000 7f44f8c00640 1 --2- 172.21.15.28:0/2409072573 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f44d80a3c40 0x7f44d80a6100 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:47.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.987+0000 7f44f8c00640 1 --2- 172.21.15.28:0/2409072573 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44f4105e30 0x7f44f41a1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:47.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.987+0000 7f44f8c00640 1 --2- 172.21.15.28:0/2409072573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44f4105470 0x7f44f419aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:47.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.987+0000 7f44f8c00640 1 --2- 172.21.15.28:0/2409072573 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f44f4104280 0x7f44f419a590 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:47.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.987+0000 7f44f8c00640 1 -- 172.21.15.28:0/2409072573 >> 172.21.15.28:0/2409072573 conn(0x7f44f40ff7f0 msgr2=0x7f44f41087f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:47.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.987+0000 7f44f8c00640 1 -- 172.21.15.28:0/2409072573 shutdown_connections 2024-04-23T14:41:47.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:47.987+0000 7f44f8c00640 1 -- 172.21.15.28:0/2409072573 wait complete. 2024-04-23T14:41:48.007 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-23T14:41:48.007 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T14:41:48.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.220+0000 7f63b3800640 1 -- 172.21.15.28:0/3670523506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63ac104280 msgr2=0x7f63ac1046a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:48.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.220+0000 7f63b3800640 1 --2- 172.21.15.28:0/3670523506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63ac104280 0x7f63ac1046a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f63a0009900 tx=0x7f63a002a3f0 comp rx=0 tx=0).stop 2024-04-23T14:41:48.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.220+0000 7f63b3800640 1 -- 172.21.15.28:0/3670523506 shutdown_connections 2024-04-23T14:41:48.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.220+0000 7f63b3800640 1 --2- 172.21.15.28:0/3670523506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f63ac105470 0x7f63ac1058f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:48.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.220+0000 7f63b3800640 1 --2- 172.21.15.28:0/3670523506 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63ac105e30 0x7f63ac10c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:48.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.220+0000 7f63b3800640 1 --2- 172.21.15.28:0/3670523506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63ac104280 0x7f63ac1046a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:48.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.220+0000 7f63b3800640 1 -- 172.21.15.28:0/3670523506 >> 172.21.15.28:0/3670523506 conn(0x7f63ac0ff7f0 msgr2=0x7f63ac101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:48.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.220+0000 7f63b3800640 1 -- 172.21.15.28:0/3670523506 shutdown_connections 2024-04-23T14:41:48.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.220+0000 7f63b3800640 1 -- 172.21.15.28:0/3670523506 wait complete. 2024-04-23T14:41:48.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.221+0000 7f63b3800640 1 Processor -- start 2024-04-23T14:41:48.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.221+0000 7f63b3800640 1 -- start start 2024-04-23T14:41:48.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.222+0000 7f63b3800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f63ac104280 0x7f63ac19a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:48.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.222+0000 7f63b3800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63ac105470 0x7f63ac19ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:48.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.222+0000 7f63b3800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63ac105e30 0x7f63ac1a1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:48.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.222+0000 7f63b3800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac10ff30 con 0x7f63ac105e30 2024-04-23T14:41:48.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.222+0000 7f63b3800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac10fdb0 con 0x7f63ac104280 2024-04-23T14:41:48.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.222+0000 7f63b3800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac1100b0 con 0x7f63ac105470 2024-04-23T14:41:48.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.222+0000 7f63b1800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63ac105e30 0x7f63ac1a1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:48.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.222+0000 7f63b0e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f63ac104280 0x7f63ac19a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:48.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.222+0000 7f63b0e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f63ac104280 0x7f63ac19a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:46960/0 (socket says 172.21.15.28:46960) 2024-04-23T14:41:48.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.222+0000 7f63b0e00640 1 -- 172.21.15.28:0/1339544341 learned_addr learned my addr 172.21.15.28:0/1339544341 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:48.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.222+0000 7f63abe00640 1 --2- 172.21.15.28:0/1339544341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63ac105470 0x7f63ac19ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:48.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.223+0000 7f63b1800640 1 -- 172.21.15.28:0/1339544341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63ac105470 msgr2=0x7f63ac19ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:48.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.223+0000 7f63b1800640 1 --2- 172.21.15.28:0/1339544341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63ac105470 0x7f63ac19ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:48.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.223+0000 7f63b1800640 1 -- 172.21.15.28:0/1339544341 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f63ac104280 msgr2=0x7f63ac19a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:48.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.223+0000 7f63b1800640 1 --2- 172.21.15.28:0/1339544341 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f63ac104280 0x7f63ac19a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:48.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.223+0000 7f63b1800640 1 -- 172.21.15.28:0/1339544341 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63ac1a21f0 con 0x7f63ac105e30 2024-04-23T14:41:48.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.223+0000 7f63b1800640 1 --2- 172.21.15.28:0/1339544341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63ac105e30 0x7f63ac1a1b80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f63a400da10 tx=0x7f63a400ded0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:48.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.223+0000 7f63a9600640 1 -- 172.21.15.28:0/1339544341 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a400bd70 con 0x7f63ac105e30 2024-04-23T14:41:48.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.223+0000 7f63a9600640 1 -- 172.21.15.28:0/1339544341 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63a4003e50 con 0x7f63ac105e30 2024-04-23T14:41:48.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.223+0000 7f63b3800640 1 -- 172.21.15.28:0/1339544341 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63ac1a24e0 con 0x7f63ac105e30 2024-04-23T14:41:48.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.223+0000 7f63a9600640 1 -- 172.21.15.28:0/1339544341 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a40106a0 con 0x7f63ac105e30 2024-04-23T14:41:48.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.223+0000 7f63b3800640 1 -- 172.21.15.28:0/1339544341 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63ac1a29e0 con 0x7f63ac105e30 2024-04-23T14:41:48.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.225+0000 7f63b3800640 1 -- 172.21.15.28:0/1339544341 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63ac10b5a0 con 0x7f63ac105e30 2024-04-23T14:41:48.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.227+0000 7f63a9600640 1 -- 172.21.15.28:0/1339544341 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f63a4003970 con 0x7f63ac105e30 2024-04-23T14:41:48.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.228+0000 7f63a9600640 1 --2- 172.21.15.28:0/1339544341 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f63940a3c40 0x7f63940a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:48.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.228+0000 7f63a9600640 1 -- 172.21.15.28:0/1339544341 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f63a40d08a0 con 0x7f63ac105e30 2024-04-23T14:41:48.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.228+0000 7f63b0e00640 1 --2- 172.21.15.28:0/1339544341 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f63940a3c40 0x7f63940a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:48.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.229+0000 7f63b0e00640 1 --2- 172.21.15.28:0/1339544341 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f63940a3c40 0x7f63940a6100 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f63a0000c00 tx=0x7f63a00315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:48.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.233+0000 7f63a9600640 1 -- 172.21.15.28:0/1339544341 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f63a409d1d0 con 0x7f63ac105e30 2024-04-23T14:41:48.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.439+0000 7f63b3800640 1 -- 172.21.15.28:0/1339544341 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f63ac062440 con 0x7f63940a3c40 2024-04-23T14:41:48.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.453+0000 7f63a9600640 1 -- 172.21.15.28:0/1339544341 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258691 (secure 0 0 0) 0x7f63ac062440 con 0x7f63940a3c40 2024-04-23T14:41:48.458 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:41:48.460 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-23T14:41:48.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.459+0000 7f63b3800640 1 -- 172.21.15.28:0/1339544341 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f63940a3c40 msgr2=0x7f63940a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:48.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.459+0000 7f63b3800640 1 --2- 172.21.15.28:0/1339544341 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f63940a3c40 0x7f63940a6100 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f63a0000c00 tx=0x7f63a00315b0 comp rx=0 tx=0).stop 2024-04-23T14:41:48.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.459+0000 7f63b3800640 1 -- 172.21.15.28:0/1339544341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63ac105e30 msgr2=0x7f63ac1a1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:48.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.459+0000 7f63b3800640 1 --2- 172.21.15.28:0/1339544341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63ac105e30 0x7f63ac1a1b80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f63a400da10 tx=0x7f63a400ded0 comp rx=0 tx=0).stop 2024-04-23T14:41:48.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.459+0000 7f63b3800640 1 -- 172.21.15.28:0/1339544341 shutdown_connections 2024-04-23T14:41:48.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.459+0000 7f63b3800640 1 --2- 172.21.15.28:0/1339544341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63ac105e30 0x7f63ac1a1b80 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:48.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.459+0000 7f63b3800640 1 --2- 172.21.15.28:0/1339544341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63ac105470 0x7f63ac19ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:48.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.459+0000 7f63b3800640 1 --2- 172.21.15.28:0/1339544341 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f63940a3c40 0x7f63940a6100 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:48.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.459+0000 7f63b3800640 1 --2- 172.21.15.28:0/1339544341 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f63ac104280 0x7f63ac19a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:48.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.459+0000 7f63b3800640 1 -- 172.21.15.28:0/1339544341 >> 172.21.15.28:0/1339544341 conn(0x7f63ac0ff7f0 msgr2=0x7f63ac108770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:48.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.460+0000 7f63b3800640 1 -- 172.21.15.28:0/1339544341 shutdown_connections 2024-04-23T14:41:48.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.460+0000 7f63b3800640 1 -- 172.21.15.28:0/1339544341 wait complete. 2024-04-23T14:41:48.482 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":318,"stamp":"2024-04-23T14:41:47.669539+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592895,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":145,"num_read_kb":127,"num_write":231,"num_write_kb":2955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":179,"ondisk_log_size":179,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378392,"kb_used_data":12544,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563432,"statfs":{"total":1151940427776,"available":1151552954368,"internally_reserved":0,"allocated":12845056,"data_stored":7040703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2527,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9,"num_read_kb":8,"num_write":50,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003443"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.375045+0000","last_change":"2024-04-23T14:41:40.033061+0000","last_active":"2024-04-23T14:41:41.375045+0000","last_peered":"2024-04-23T14:41:41.375045+0000","last_clean":"2024-04-23T14:41:41.375045+0000","last_became_active":"2024-04-23T14:41:40.032750+0000","last_became_peered":"2024-04-23T14:41:40.032750+0000","last_unstale":"2024-04-23T14:41:41.375045+0000","last_undegraded":"2024-04-23T14:41:41.375045+0000","last_fullsized":"2024-04-23T14:41:41.375045+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:58:03.451539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.382632+0000","last_change":"2024-04-23T14:41:40.037435+0000","last_active":"2024-04-23T14:41:41.382632+0000","last_peered":"2024-04-23T14:41:41.382632+0000","last_clean":"2024-04-23T14:41:41.382632+0000","last_became_active":"2024-04-23T14:41:40.036655+0000","last_became_peered":"2024-04-23T14:41:40.036655+0000","last_unstale":"2024-04-23T14:41:41.382632+0000","last_undegraded":"2024-04-23T14:41:41.382632+0000","last_fullsized":"2024-04-23T14:41:41.382632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:51:58.847850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.383061+0000","last_change":"2024-04-23T14:41:40.024977+0000","last_active":"2024-04-23T14:41:41.383061+0000","last_peered":"2024-04-23T14:41:41.383061+0000","last_clean":"2024-04-23T14:41:41.383061+0000","last_became_active":"2024-04-23T14:41:40.023879+0000","last_became_peered":"2024-04-23T14:41:40.023879+0000","last_unstale":"2024-04-23T14:41:41.383061+0000","last_undegraded":"2024-04-23T14:41:41.383061+0000","last_fullsized":"2024-04-23T14:41:41.383061+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:03:40.672661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373767+0000","last_change":"2024-04-23T14:41:40.017757+0000","last_active":"2024-04-23T14:41:41.373767+0000","last_peered":"2024-04-23T14:41:41.373767+0000","last_clean":"2024-04-23T14:41:41.373767+0000","last_became_active":"2024-04-23T14:41:40.013996+0000","last_became_peered":"2024-04-23T14:41:40.013996+0000","last_unstale":"2024-04-23T14:41:41.373767+0000","last_undegraded":"2024-04-23T14:41:41.373767+0000","last_fullsized":"2024-04-23T14:41:41.373767+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:20:03.376733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.375077+0000","last_change":"2024-04-23T14:41:40.038247+0000","last_active":"2024-04-23T14:41:41.375077+0000","last_peered":"2024-04-23T14:41:41.375077+0000","last_clean":"2024-04-23T14:41:41.375077+0000","last_became_active":"2024-04-23T14:41:40.037934+0000","last_became_peered":"2024-04-23T14:41:40.037934+0000","last_unstale":"2024-04-23T14:41:41.375077+0000","last_undegraded":"2024-04-23T14:41:41.375077+0000","last_fullsized":"2024-04-23T14:41:41.375077+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:15.637733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374016+0000","last_change":"2024-04-23T14:41:40.015341+0000","last_active":"2024-04-23T14:41:41.374016+0000","last_peered":"2024-04-23T14:41:41.374016+0000","last_clean":"2024-04-23T14:41:41.374016+0000","last_became_active":"2024-04-23T14:41:40.013627+0000","last_became_peered":"2024-04-23T14:41:40.013627+0000","last_unstale":"2024-04-23T14:41:41.374016+0000","last_undegraded":"2024-04-23T14:41:41.374016+0000","last_fullsized":"2024-04-23T14:41:41.374016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:05:36.248107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373215+0000","last_change":"2024-04-23T14:41:40.010950+0000","last_active":"2024-04-23T14:41:41.373215+0000","last_peered":"2024-04-23T14:41:41.373215+0000","last_clean":"2024-04-23T14:41:41.373215+0000","last_became_active":"2024-04-23T14:41:40.010584+0000","last_became_peered":"2024-04-23T14:41:40.010584+0000","last_unstale":"2024-04-23T14:41:41.373215+0000","last_undegraded":"2024-04-23T14:41:41.373215+0000","last_fullsized":"2024-04-23T14:41:41.373215+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:46:25.379243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.027387+0000","last_change":"2024-04-23T14:41:40.031198+0000","last_active":"2024-04-23T14:41:41.027387+0000","last_peered":"2024-04-23T14:41:41.027387+0000","last_clean":"2024-04-23T14:41:41.027387+0000","last_became_active":"2024-04-23T14:41:40.030652+0000","last_became_peered":"2024-04-23T14:41:40.030652+0000","last_unstale":"2024-04-23T14:41:41.027387+0000","last_undegraded":"2024-04-23T14:41:41.027387+0000","last_fullsized":"2024-04-23T14:41:41.027387+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:26:00.520182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.031569+0000","last_change":"2024-04-23T14:41:40.009820+0000","last_active":"2024-04-23T14:41:41.031569+0000","last_peered":"2024-04-23T14:41:41.031569+0000","last_clean":"2024-04-23T14:41:41.031569+0000","last_became_active":"2024-04-23T14:41:40.009468+0000","last_became_peered":"2024-04-23T14:41:40.009468+0000","last_unstale":"2024-04-23T14:41:41.031569+0000","last_undegraded":"2024-04-23T14:41:41.031569+0000","last_fullsized":"2024-04-23T14:41:41.031569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:03:50.178050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374485+0000","last_change":"2024-04-23T14:41:40.008363+0000","last_active":"2024-04-23T14:41:41.374485+0000","last_peered":"2024-04-23T14:41:41.374485+0000","last_clean":"2024-04-23T14:41:41.374485+0000","last_became_active":"2024-04-23T14:41:40.006823+0000","last_became_peered":"2024-04-23T14:41:40.006823+0000","last_unstale":"2024-04-23T14:41:41.374485+0000","last_undegraded":"2024-04-23T14:41:41.374485+0000","last_fullsized":"2024-04-23T14:41:41.374485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:12:33.718815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373461+0000","last_change":"2024-04-23T14:41:40.021069+0000","last_active":"2024-04-23T14:41:41.373461+0000","last_peered":"2024-04-23T14:41:41.373461+0000","last_clean":"2024-04-23T14:41:41.373461+0000","last_became_active":"2024-04-23T14:41:40.020334+0000","last_became_peered":"2024-04-23T14:41:40.020334+0000","last_unstale":"2024-04-23T14:41:41.373461+0000","last_undegraded":"2024-04-23T14:41:41.373461+0000","last_fullsized":"2024-04-23T14:41:41.373461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:50:56.560928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373819+0000","last_change":"2024-04-23T14:41:40.017746+0000","last_active":"2024-04-23T14:41:41.373819+0000","last_peered":"2024-04-23T14:41:41.373819+0000","last_clean":"2024-04-23T14:41:41.373819+0000","last_became_active":"2024-04-23T14:41:40.014495+0000","last_became_peered":"2024-04-23T14:41:40.014495+0000","last_unstale":"2024-04-23T14:41:41.373819+0000","last_undegraded":"2024-04-23T14:41:41.373819+0000","last_fullsized":"2024-04-23T14:41:41.373819+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:23:22.188110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.375114+0000","last_change":"2024-04-23T14:41:40.021429+0000","last_active":"2024-04-23T14:41:41.375114+0000","last_peered":"2024-04-23T14:41:41.375114+0000","last_clean":"2024-04-23T14:41:41.375114+0000","last_became_active":"2024-04-23T14:41:40.021087+0000","last_became_peered":"2024-04-23T14:41:40.021087+0000","last_unstale":"2024-04-23T14:41:41.375114+0000","last_undegraded":"2024-04-23T14:41:41.375114+0000","last_fullsized":"2024-04-23T14:41:41.375114+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:51:16.633253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.381552+0000","last_change":"2024-04-23T14:41:40.031839+0000","last_active":"2024-04-23T14:41:41.381552+0000","last_peered":"2024-04-23T14:41:41.381552+0000","last_clean":"2024-04-23T14:41:41.381552+0000","last_became_active":"2024-04-23T14:41:40.031463+0000","last_became_peered":"2024-04-23T14:41:40.031463+0000","last_unstale":"2024-04-23T14:41:41.381552+0000","last_undegraded":"2024-04-23T14:41:41.381552+0000","last_fullsized":"2024-04-23T14:41:41.381552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:51:51.200463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.381081+0000","last_change":"2024-04-23T14:41:40.042575+0000","last_active":"2024-04-23T14:41:41.381081+0000","last_peered":"2024-04-23T14:41:41.381081+0000","last_clean":"2024-04-23T14:41:41.381081+0000","last_became_active":"2024-04-23T14:41:40.033758+0000","last_became_peered":"2024-04-23T14:41:40.033758+0000","last_unstale":"2024-04-23T14:41:41.381081+0000","last_undegraded":"2024-04-23T14:41:41.381081+0000","last_fullsized":"2024-04-23T14:41:41.381081+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:55:17.338169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.030972+0000","last_change":"2024-04-23T14:41:40.036214+0000","last_active":"2024-04-23T14:41:41.030972+0000","last_peered":"2024-04-23T14:41:41.030972+0000","last_clean":"2024-04-23T14:41:41.030972+0000","last_became_active":"2024-04-23T14:41:40.035892+0000","last_became_peered":"2024-04-23T14:41:40.035892+0000","last_unstale":"2024-04-23T14:41:41.030972+0000","last_undegraded":"2024-04-23T14:41:41.030972+0000","last_fullsized":"2024-04-23T14:41:41.030972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:29:18.668690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.375254+0000","last_change":"2024-04-23T14:41:40.029837+0000","last_active":"2024-04-23T14:41:41.375254+0000","last_peered":"2024-04-23T14:41:41.375254+0000","last_clean":"2024-04-23T14:41:41.375254+0000","last_became_active":"2024-04-23T14:41:40.029472+0000","last_became_peered":"2024-04-23T14:41:40.029472+0000","last_unstale":"2024-04-23T14:41:41.375254+0000","last_undegraded":"2024-04-23T14:41:41.375254+0000","last_fullsized":"2024-04-23T14:41:41.375254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:28:52.194898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.378933+0000","last_change":"2024-04-23T14:41:40.015631+0000","last_active":"2024-04-23T14:41:41.378933+0000","last_peered":"2024-04-23T14:41:41.378933+0000","last_clean":"2024-04-23T14:41:41.378933+0000","last_became_active":"2024-04-23T14:41:40.015211+0000","last_became_peered":"2024-04-23T14:41:40.015211+0000","last_unstale":"2024-04-23T14:41:41.378933+0000","last_undegraded":"2024-04-23T14:41:41.378933+0000","last_fullsized":"2024-04-23T14:41:41.378933+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:03:33.485364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373975+0000","last_change":"2024-04-23T14:41:40.041143+0000","last_active":"2024-04-23T14:41:41.373975+0000","last_peered":"2024-04-23T14:41:41.373975+0000","last_clean":"2024-04-23T14:41:41.373975+0000","last_became_active":"2024-04-23T14:41:40.040793+0000","last_became_peered":"2024-04-23T14:41:40.040793+0000","last_unstale":"2024-04-23T14:41:41.373975+0000","last_undegraded":"2024-04-23T14:41:41.373975+0000","last_fullsized":"2024-04-23T14:41:41.373975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:39:14.660719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.818393+0000","last_change":"2024-04-23T14:41:39.017031+0000","last_active":"2024-04-23T14:41:41.818393+0000","last_peered":"2024-04-23T14:41:41.818393+0000","last_clean":"2024-04-23T14:41:41.818393+0000","last_became_active":"2024-04-23T14:41:39.016699+0000","last_became_peered":"2024-04-23T14:41:39.016699+0000","last_unstale":"2024-04-23T14:41:41.818393+0000","last_undegraded":"2024-04-23T14:41:41.818393+0000","last_fullsized":"2024-04-23T14:41:41.818393+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:17:49.812601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.028446+0000","last_change":"2024-04-23T14:41:40.031957+0000","last_active":"2024-04-23T14:41:41.028446+0000","last_peered":"2024-04-23T14:41:41.028446+0000","last_clean":"2024-04-23T14:41:41.028446+0000","last_became_active":"2024-04-23T14:41:40.028151+0000","last_became_peered":"2024-04-23T14:41:40.028151+0000","last_unstale":"2024-04-23T14:41:41.028446+0000","last_undegraded":"2024-04-23T14:41:41.028446+0000","last_fullsized":"2024-04-23T14:41:41.028446+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:46:48.455032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.028416+0000","last_change":"2024-04-23T14:41:39.022503+0000","last_active":"2024-04-23T14:41:41.028416+0000","last_peered":"2024-04-23T14:41:41.028416+0000","last_clean":"2024-04-23T14:41:41.028416+0000","last_became_active":"2024-04-23T14:41:39.022093+0000","last_became_peered":"2024-04-23T14:41:39.022093+0000","last_unstale":"2024-04-23T14:41:41.028416+0000","last_undegraded":"2024-04-23T14:41:41.028416+0000","last_fullsized":"2024-04-23T14:41:41.028416+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:48:27.861140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.383795+0000","last_change":"2024-04-23T14:41:40.032878+0000","last_active":"2024-04-23T14:41:41.383795+0000","last_peered":"2024-04-23T14:41:41.383795+0000","last_clean":"2024-04-23T14:41:41.383795+0000","last_became_active":"2024-04-23T14:41:40.027473+0000","last_became_peered":"2024-04-23T14:41:40.027473+0000","last_unstale":"2024-04-23T14:41:41.383795+0000","last_undegraded":"2024-04-23T14:41:41.383795+0000","last_fullsized":"2024-04-23T14:41:41.383795+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:00:17.677306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.375755+0000","last_change":"2024-04-23T14:41:39.005919+0000","last_active":"2024-04-23T14:41:41.375755+0000","last_peered":"2024-04-23T14:41:41.375755+0000","last_clean":"2024-04-23T14:41:41.375755+0000","last_became_active":"2024-04-23T14:41:39.005550+0000","last_became_peered":"2024-04-23T14:41:39.005550+0000","last_unstale":"2024-04-23T14:41:41.375755+0000","last_undegraded":"2024-04-23T14:41:41.375755+0000","last_fullsized":"2024-04-23T14:41:41.375755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:49:10.362939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.031022+0000","last_change":"2024-04-23T14:41:40.019955+0000","last_active":"2024-04-23T14:41:41.031022+0000","last_peered":"2024-04-23T14:41:41.031022+0000","last_clean":"2024-04-23T14:41:41.031022+0000","last_became_active":"2024-04-23T14:41:40.019579+0000","last_became_peered":"2024-04-23T14:41:40.019579+0000","last_unstale":"2024-04-23T14:41:41.031022+0000","last_undegraded":"2024-04-23T14:41:41.031022+0000","last_fullsized":"2024-04-23T14:41:41.031022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:13:14.330051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.375509+0000","last_change":"2024-04-23T14:41:38.997759+0000","last_active":"2024-04-23T14:41:41.375509+0000","last_peered":"2024-04-23T14:41:41.375509+0000","last_clean":"2024-04-23T14:41:41.375509+0000","last_became_active":"2024-04-23T14:41:38.997445+0000","last_became_peered":"2024-04-23T14:41:38.997445+0000","last_unstale":"2024-04-23T14:41:41.375509+0000","last_undegraded":"2024-04-23T14:41:41.375509+0000","last_fullsized":"2024-04-23T14:41:41.375509+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:35:02.583929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373248+0000","last_change":"2024-04-23T14:41:40.015592+0000","last_active":"2024-04-23T14:41:41.373248+0000","last_peered":"2024-04-23T14:41:41.373248+0000","last_clean":"2024-04-23T14:41:41.373248+0000","last_became_active":"2024-04-23T14:41:40.015292+0000","last_became_peered":"2024-04-23T14:41:40.015292+0000","last_unstale":"2024-04-23T14:41:41.373248+0000","last_undegraded":"2024-04-23T14:41:41.373248+0000","last_fullsized":"2024-04-23T14:41:41.373248+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:52:01.641962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.372978+0000","last_change":"2024-04-23T14:41:39.008526+0000","last_active":"2024-04-23T14:41:41.372978+0000","last_peered":"2024-04-23T14:41:41.372978+0000","last_clean":"2024-04-23T14:41:41.372978+0000","last_became_active":"2024-04-23T14:41:39.008222+0000","last_became_peered":"2024-04-23T14:41:39.008222+0000","last_unstale":"2024-04-23T14:41:41.372978+0000","last_undegraded":"2024-04-23T14:41:41.372978+0000","last_fullsized":"2024-04-23T14:41:41.372978+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:23:09.191426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374601+0000","last_change":"2024-04-23T14:41:40.011486+0000","last_active":"2024-04-23T14:41:41.374601+0000","last_peered":"2024-04-23T14:41:41.374601+0000","last_clean":"2024-04-23T14:41:41.374601+0000","last_became_active":"2024-04-23T14:41:40.011167+0000","last_became_peered":"2024-04-23T14:41:40.011167+0000","last_unstale":"2024-04-23T14:41:41.374601+0000","last_undegraded":"2024-04-23T14:41:41.374601+0000","last_fullsized":"2024-04-23T14:41:41.374601+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:09:12.418428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373832+0000","last_change":"2024-04-23T14:41:39.028566+0000","last_active":"2024-04-23T14:41:41.373832+0000","last_peered":"2024-04-23T14:41:41.373832+0000","last_clean":"2024-04-23T14:41:41.373832+0000","last_became_active":"2024-04-23T14:41:39.028174+0000","last_became_peered":"2024-04-23T14:41:39.028174+0000","last_unstale":"2024-04-23T14:41:41.373832+0000","last_undegraded":"2024-04-23T14:41:41.373832+0000","last_fullsized":"2024-04-23T14:41:41.373832+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:50:55.424741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373195+0000","last_change":"2024-04-23T14:41:40.036171+0000","last_active":"2024-04-23T14:41:41.373195+0000","last_peered":"2024-04-23T14:41:41.373195+0000","last_clean":"2024-04-23T14:41:41.373195+0000","last_became_active":"2024-04-23T14:41:40.035815+0000","last_became_peered":"2024-04-23T14:41:40.035815+0000","last_unstale":"2024-04-23T14:41:41.373195+0000","last_undegraded":"2024-04-23T14:41:41.373195+0000","last_fullsized":"2024-04-23T14:41:41.373195+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:16:38.036654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.375487+0000","last_change":"2024-04-23T14:41:39.028618+0000","last_active":"2024-04-23T14:41:41.375487+0000","last_peered":"2024-04-23T14:41:41.375487+0000","last_clean":"2024-04-23T14:41:41.375487+0000","last_became_active":"2024-04-23T14:41:39.028274+0000","last_became_peered":"2024-04-23T14:41:39.028274+0000","last_unstale":"2024-04-23T14:41:41.375487+0000","last_undegraded":"2024-04-23T14:41:41.375487+0000","last_fullsized":"2024-04-23T14:41:41.375487+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:14:31.571760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.027286+0000","last_change":"2024-04-23T14:41:40.020687+0000","last_active":"2024-04-23T14:41:41.027286+0000","last_peered":"2024-04-23T14:41:41.027286+0000","last_clean":"2024-04-23T14:41:41.027286+0000","last_became_active":"2024-04-23T14:41:40.020352+0000","last_became_peered":"2024-04-23T14:41:40.020352+0000","last_unstale":"2024-04-23T14:41:41.027286+0000","last_undegraded":"2024-04-23T14:41:41.027286+0000","last_fullsized":"2024-04-23T14:41:41.027286+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:29:37.013486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.372658+0000","last_change":"2024-04-23T14:41:39.006002+0000","last_active":"2024-04-23T14:41:41.372658+0000","last_peered":"2024-04-23T14:41:41.372658+0000","last_clean":"2024-04-23T14:41:41.372658+0000","last_became_active":"2024-04-23T14:41:39.005415+0000","last_became_peered":"2024-04-23T14:41:39.005415+0000","last_unstale":"2024-04-23T14:41:41.372658+0000","last_undegraded":"2024-04-23T14:41:41.372658+0000","last_fullsized":"2024-04-23T14:41:41.372658+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:13:55.491931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.378568+0000","last_change":"2024-04-23T14:41:40.029393+0000","last_active":"2024-04-23T14:41:41.378568+0000","last_peered":"2024-04-23T14:41:41.378568+0000","last_clean":"2024-04-23T14:41:41.378568+0000","last_became_active":"2024-04-23T14:41:40.029039+0000","last_became_peered":"2024-04-23T14:41:40.029039+0000","last_unstale":"2024-04-23T14:41:41.378568+0000","last_undegraded":"2024-04-23T14:41:41.378568+0000","last_fullsized":"2024-04-23T14:41:41.378568+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:39:24.461260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374477+0000","last_change":"2024-04-23T14:41:39.014638+0000","last_active":"2024-04-23T14:41:41.374477+0000","last_peered":"2024-04-23T14:41:41.374477+0000","last_clean":"2024-04-23T14:41:41.374477+0000","last_became_active":"2024-04-23T14:41:39.014289+0000","last_became_peered":"2024-04-23T14:41:39.014289+0000","last_unstale":"2024-04-23T14:41:41.374477+0000","last_undegraded":"2024-04-23T14:41:41.374477+0000","last_fullsized":"2024-04-23T14:41:41.374477+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:15:15.855399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.031483+0000","last_change":"2024-04-23T14:41:40.015781+0000","last_active":"2024-04-23T14:41:41.031483+0000","last_peered":"2024-04-23T14:41:41.031483+0000","last_clean":"2024-04-23T14:41:41.031483+0000","last_became_active":"2024-04-23T14:41:40.015423+0000","last_became_peered":"2024-04-23T14:41:40.015423+0000","last_unstale":"2024-04-23T14:41:41.031483+0000","last_undegraded":"2024-04-23T14:41:41.031483+0000","last_fullsized":"2024-04-23T14:41:41.031483+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:56:55.947585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.372817+0000","last_change":"2024-04-23T14:41:39.011021+0000","last_active":"2024-04-23T14:41:41.372817+0000","last_peered":"2024-04-23T14:41:41.372817+0000","last_clean":"2024-04-23T14:41:41.372817+0000","last_became_active":"2024-04-23T14:41:39.010637+0000","last_became_peered":"2024-04-23T14:41:39.010637+0000","last_unstale":"2024-04-23T14:41:41.372817+0000","last_undegraded":"2024-04-23T14:41:41.372817+0000","last_fullsized":"2024-04-23T14:41:41.372817+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:22:22.647170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.383250+0000","last_change":"2024-04-23T14:41:40.042797+0000","last_active":"2024-04-23T14:41:41.383250+0000","last_peered":"2024-04-23T14:41:41.383250+0000","last_clean":"2024-04-23T14:41:41.383250+0000","last_became_active":"2024-04-23T14:41:40.037797+0000","last_became_peered":"2024-04-23T14:41:40.037797+0000","last_unstale":"2024-04-23T14:41:41.383250+0000","last_undegraded":"2024-04-23T14:41:41.383250+0000","last_fullsized":"2024-04-23T14:41:41.383250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:37:32.604276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373338+0000","last_change":"2024-04-23T14:41:39.010585+0000","last_active":"2024-04-23T14:41:41.373338+0000","last_peered":"2024-04-23T14:41:41.373338+0000","last_clean":"2024-04-23T14:41:41.373338+0000","last_became_active":"2024-04-23T14:41:39.010294+0000","last_became_peered":"2024-04-23T14:41:39.010294+0000","last_unstale":"2024-04-23T14:41:41.373338+0000","last_undegraded":"2024-04-23T14:41:41.373338+0000","last_fullsized":"2024-04-23T14:41:41.373338+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:38:03.146904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.031264+0000","last_change":"2024-04-23T14:41:40.013150+0000","last_active":"2024-04-23T14:41:41.031264+0000","last_peered":"2024-04-23T14:41:41.031264+0000","last_clean":"2024-04-23T14:41:41.031264+0000","last_became_active":"2024-04-23T14:41:40.012310+0000","last_became_peered":"2024-04-23T14:41:40.012310+0000","last_unstale":"2024-04-23T14:41:41.031264+0000","last_undegraded":"2024-04-23T14:41:41.031264+0000","last_fullsized":"2024-04-23T14:41:41.031264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:45:25.777200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"84'6","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:43.398339+0000","last_change":"2024-04-23T14:41:39.004256+0000","last_active":"2024-04-23T14:41:43.398339+0000","last_peered":"2024-04-23T14:41:43.398339+0000","last_clean":"2024-04-23T14:41:43.398339+0000","last_became_active":"2024-04-23T14:41:39.003874+0000","last_became_peered":"2024-04-23T14:41:39.003874+0000","last_unstale":"2024-04-23T14:41:43.398339+0000","last_undegraded":"2024-04-23T14:41:43.398339+0000","last_fullsized":"2024-04-23T14:41:43.398339+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:38:30.727149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.375615+0000","last_change":"2024-04-23T14:41:40.012131+0000","last_active":"2024-04-23T14:41:41.375615+0000","last_peered":"2024-04-23T14:41:41.375615+0000","last_clean":"2024-04-23T14:41:41.375615+0000","last_became_active":"2024-04-23T14:41:40.011225+0000","last_became_peered":"2024-04-23T14:41:40.011225+0000","last_unstale":"2024-04-23T14:41:41.375615+0000","last_undegraded":"2024-04-23T14:41:41.375615+0000","last_fullsized":"2024-04-23T14:41:41.375615+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:36:12.085228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373419+0000","last_change":"2024-04-23T14:41:39.001935+0000","last_active":"2024-04-23T14:41:41.373419+0000","last_peered":"2024-04-23T14:41:41.373419+0000","last_clean":"2024-04-23T14:41:41.373419+0000","last_became_active":"2024-04-23T14:41:39.001583+0000","last_became_peered":"2024-04-23T14:41:39.001583+0000","last_unstale":"2024-04-23T14:41:41.373419+0000","last_undegraded":"2024-04-23T14:41:41.373419+0000","last_fullsized":"2024-04-23T14:41:41.373419+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:47:40.986280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374425+0000","last_change":"2024-04-23T14:41:40.036515+0000","last_active":"2024-04-23T14:41:41.374425+0000","last_peered":"2024-04-23T14:41:41.374425+0000","last_clean":"2024-04-23T14:41:41.374425+0000","last_became_active":"2024-04-23T14:41:40.036125+0000","last_became_peered":"2024-04-23T14:41:40.036125+0000","last_unstale":"2024-04-23T14:41:41.374425+0000","last_undegraded":"2024-04-23T14:41:41.374425+0000","last_fullsized":"2024-04-23T14:41:41.374425+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:55:28.465378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.372826+0000","last_change":"2024-04-23T14:41:39.006127+0000","last_active":"2024-04-23T14:41:41.372826+0000","last_peered":"2024-04-23T14:41:41.372826+0000","last_clean":"2024-04-23T14:41:41.372826+0000","last_became_active":"2024-04-23T14:41:39.005761+0000","last_became_peered":"2024-04-23T14:41:39.005761+0000","last_unstale":"2024-04-23T14:41:41.372826+0000","last_undegraded":"2024-04-23T14:41:41.372826+0000","last_fullsized":"2024-04-23T14:41:41.372826+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:10:24.050016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373287+0000","last_change":"2024-04-23T14:41:40.028762+0000","last_active":"2024-04-23T14:41:41.373287+0000","last_peered":"2024-04-23T14:41:41.373287+0000","last_clean":"2024-04-23T14:41:41.373287+0000","last_became_active":"2024-04-23T14:41:40.028440+0000","last_became_peered":"2024-04-23T14:41:40.028440+0000","last_unstale":"2024-04-23T14:41:41.373287+0000","last_undegraded":"2024-04-23T14:41:41.373287+0000","last_fullsized":"2024-04-23T14:41:41.373287+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:22:15.565087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373364+0000","last_change":"2024-04-23T14:41:39.005873+0000","last_active":"2024-04-23T14:41:41.373364+0000","last_peered":"2024-04-23T14:41:41.373364+0000","last_clean":"2024-04-23T14:41:41.373364+0000","last_became_active":"2024-04-23T14:41:39.005481+0000","last_became_peered":"2024-04-23T14:41:39.005481+0000","last_unstale":"2024-04-23T14:41:41.373364+0000","last_undegraded":"2024-04-23T14:41:41.373364+0000","last_fullsized":"2024-04-23T14:41:41.373364+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:49:59.117451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.372278+0000","last_change":"2024-04-23T14:41:40.009346+0000","last_active":"2024-04-23T14:41:41.372278+0000","last_peered":"2024-04-23T14:41:41.372278+0000","last_clean":"2024-04-23T14:41:41.372278+0000","last_became_active":"2024-04-23T14:41:40.008927+0000","last_became_peered":"2024-04-23T14:41:40.008927+0000","last_unstale":"2024-04-23T14:41:41.372278+0000","last_undegraded":"2024-04-23T14:41:41.372278+0000","last_fullsized":"2024-04-23T14:41:41.372278+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:33:36.513349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374236+0000","last_change":"2024-04-23T14:41:39.012298+0000","last_active":"2024-04-23T14:41:41.374236+0000","last_peered":"2024-04-23T14:41:41.374236+0000","last_clean":"2024-04-23T14:41:41.374236+0000","last_became_active":"2024-04-23T14:41:39.012002+0000","last_became_peered":"2024-04-23T14:41:39.012002+0000","last_unstale":"2024-04-23T14:41:41.374236+0000","last_undegraded":"2024-04-23T14:41:41.374236+0000","last_fullsized":"2024-04-23T14:41:41.374236+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:30:06.150083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373386+0000","last_change":"2024-04-23T14:41:40.016845+0000","last_active":"2024-04-23T14:41:41.373386+0000","last_peered":"2024-04-23T14:41:41.373386+0000","last_clean":"2024-04-23T14:41:41.373386+0000","last_became_active":"2024-04-23T14:41:40.016543+0000","last_became_peered":"2024-04-23T14:41:40.016543+0000","last_unstale":"2024-04-23T14:41:41.373386+0000","last_undegraded":"2024-04-23T14:41:41.373386+0000","last_fullsized":"2024-04-23T14:41:41.373386+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:52:41.054454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.375704+0000","last_change":"2024-04-23T14:41:38.999968+0000","last_active":"2024-04-23T14:41:41.375704+0000","last_peered":"2024-04-23T14:41:41.375704+0000","last_clean":"2024-04-23T14:41:41.375704+0000","last_became_active":"2024-04-23T14:41:38.999658+0000","last_became_peered":"2024-04-23T14:41:38.999658+0000","last_unstale":"2024-04-23T14:41:41.375704+0000","last_undegraded":"2024-04-23T14:41:41.375704+0000","last_fullsized":"2024-04-23T14:41:41.375704+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:14:20.990331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374929+0000","last_change":"2024-04-23T14:41:40.011753+0000","last_active":"2024-04-23T14:41:41.374929+0000","last_peered":"2024-04-23T14:41:41.374929+0000","last_clean":"2024-04-23T14:41:41.374929+0000","last_became_active":"2024-04-23T14:41:40.011264+0000","last_became_peered":"2024-04-23T14:41:40.011264+0000","last_unstale":"2024-04-23T14:41:41.374929+0000","last_undegraded":"2024-04-23T14:41:41.374929+0000","last_fullsized":"2024-04-23T14:41:41.374929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:52:22.812046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.378675+0000","last_change":"2024-04-23T14:41:39.027384+0000","last_active":"2024-04-23T14:41:41.378675+0000","last_peered":"2024-04-23T14:41:41.378675+0000","last_clean":"2024-04-23T14:41:41.378675+0000","last_became_active":"2024-04-23T14:41:39.027018+0000","last_became_peered":"2024-04-23T14:41:39.027018+0000","last_unstale":"2024-04-23T14:41:41.378675+0000","last_undegraded":"2024-04-23T14:41:41.378675+0000","last_fullsized":"2024-04-23T14:41:41.378675+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:13:54.685704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374309+0000","last_change":"2024-04-23T14:41:40.041216+0000","last_active":"2024-04-23T14:41:41.374309+0000","last_peered":"2024-04-23T14:41:41.374309+0000","last_clean":"2024-04-23T14:41:41.374309+0000","last_became_active":"2024-04-23T14:41:40.040789+0000","last_became_peered":"2024-04-23T14:41:40.040789+0000","last_unstale":"2024-04-23T14:41:41.374309+0000","last_undegraded":"2024-04-23T14:41:41.374309+0000","last_fullsized":"2024-04-23T14:41:41.374309+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:01:13.156704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.372784+0000","last_change":"2024-04-23T14:41:39.024399+0000","last_active":"2024-04-23T14:41:41.372784+0000","last_peered":"2024-04-23T14:41:41.372784+0000","last_clean":"2024-04-23T14:41:41.372784+0000","last_became_active":"2024-04-23T14:41:39.024141+0000","last_became_peered":"2024-04-23T14:41:39.024141+0000","last_unstale":"2024-04-23T14:41:41.372784+0000","last_undegraded":"2024-04-23T14:41:41.372784+0000","last_fullsized":"2024-04-23T14:41:41.372784+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:34:27.516614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374016+0000","last_change":"2024-04-23T14:41:40.032822+0000","last_active":"2024-04-23T14:41:41.374016+0000","last_peered":"2024-04-23T14:41:41.374016+0000","last_clean":"2024-04-23T14:41:41.374016+0000","last_became_active":"2024-04-23T14:41:40.032473+0000","last_became_peered":"2024-04-23T14:41:40.032473+0000","last_unstale":"2024-04-23T14:41:41.374016+0000","last_undegraded":"2024-04-23T14:41:41.374016+0000","last_fullsized":"2024-04-23T14:41:41.374016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:46:54.316960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.372873+0000","last_change":"2024-04-23T14:41:40.014689+0000","last_active":"2024-04-23T14:41:41.372873+0000","last_peered":"2024-04-23T14:41:41.372873+0000","last_clean":"2024-04-23T14:41:41.372873+0000","last_became_active":"2024-04-23T14:41:40.014371+0000","last_became_peered":"2024-04-23T14:41:40.014371+0000","last_unstale":"2024-04-23T14:41:41.372873+0000","last_undegraded":"2024-04-23T14:41:41.372873+0000","last_fullsized":"2024-04-23T14:41:41.372873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:56:41.284144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.375285+0000","last_change":"2024-04-23T14:41:39.031327+0000","last_active":"2024-04-23T14:41:41.375285+0000","last_peered":"2024-04-23T14:41:41.375285+0000","last_clean":"2024-04-23T14:41:41.375285+0000","last_became_active":"2024-04-23T14:41:39.030864+0000","last_became_peered":"2024-04-23T14:41:39.030864+0000","last_unstale":"2024-04-23T14:41:41.375285+0000","last_undegraded":"2024-04-23T14:41:41.375285+0000","last_fullsized":"2024-04-23T14:41:41.375285+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:58:24.048207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.028216+0000","last_change":"2024-04-23T14:41:40.035104+0000","last_active":"2024-04-23T14:41:41.028216+0000","last_peered":"2024-04-23T14:41:41.028216+0000","last_clean":"2024-04-23T14:41:41.028216+0000","last_became_active":"2024-04-23T14:41:40.034678+0000","last_became_peered":"2024-04-23T14:41:40.034678+0000","last_unstale":"2024-04-23T14:41:41.028216+0000","last_undegraded":"2024-04-23T14:41:41.028216+0000","last_fullsized":"2024-04-23T14:41:41.028216+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:29:53.040424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373833+0000","last_change":"2024-04-23T14:41:39.016684+0000","last_active":"2024-04-23T14:41:41.373833+0000","last_peered":"2024-04-23T14:41:41.373833+0000","last_clean":"2024-04-23T14:41:41.373833+0000","last_became_active":"2024-04-23T14:41:39.016375+0000","last_became_peered":"2024-04-23T14:41:39.016375+0000","last_unstale":"2024-04-23T14:41:41.373833+0000","last_undegraded":"2024-04-23T14:41:41.373833+0000","last_fullsized":"2024-04-23T14:41:41.373833+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:09.449590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":350,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.372242+0000","last_change":"2024-04-23T14:40:22.305413+0000","last_active":"2024-04-23T14:41:41.372242+0000","last_peered":"2024-04-23T14:41:41.372242+0000","last_clean":"2024-04-23T14:41:41.372242+0000","last_became_active":"2024-04-23T14:40:22.304977+0000","last_became_peered":"2024-04-23T14:40:22.304977+0000","last_unstale":"2024-04-23T14:41:41.372242+0000","last_undegraded":"2024-04-23T14:41:41.372242+0000","last_fullsized":"2024-04-23T14:41:41.372242+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:36:28.336753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:36:28.336753+0000","last_clean_scrub_stamp":"2024-04-23T14:36:28.336753+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:42:27.611770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.378793+0000","last_change":"2024-04-23T14:41:40.023300+0000","last_active":"2024-04-23T14:41:41.378793+0000","last_peered":"2024-04-23T14:41:41.378793+0000","last_clean":"2024-04-23T14:41:41.378793+0000","last_became_active":"2024-04-23T14:41:40.022888+0000","last_became_peered":"2024-04-23T14:41:40.022888+0000","last_unstale":"2024-04-23T14:41:41.378793+0000","last_undegraded":"2024-04-23T14:41:41.378793+0000","last_fullsized":"2024-04-23T14:41:41.378793+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:31:48.705005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373248+0000","last_change":"2024-04-23T14:41:39.005482+0000","last_active":"2024-04-23T14:41:41.373248+0000","last_peered":"2024-04-23T14:41:41.373248+0000","last_clean":"2024-04-23T14:41:41.373248+0000","last_became_active":"2024-04-23T14:41:39.004552+0000","last_became_peered":"2024-04-23T14:41:39.004552+0000","last_unstale":"2024-04-23T14:41:41.373248+0000","last_undegraded":"2024-04-23T14:41:41.373248+0000","last_fullsized":"2024-04-23T14:41:41.373248+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:05:51.411210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.375552+0000","last_change":"2024-04-23T14:41:39.015605+0000","last_active":"2024-04-23T14:41:41.375552+0000","last_peered":"2024-04-23T14:41:41.375552+0000","last_clean":"2024-04-23T14:41:41.375552+0000","last_became_active":"2024-04-23T14:41:39.015260+0000","last_became_peered":"2024-04-23T14:41:39.015260+0000","last_unstale":"2024-04-23T14:41:41.375552+0000","last_undegraded":"2024-04-23T14:41:41.375552+0000","last_fullsized":"2024-04-23T14:41:41.375552+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:23:21.251483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.378260+0000","last_change":"2024-04-23T14:41:40.020619+0000","last_active":"2024-04-23T14:41:41.378260+0000","last_peered":"2024-04-23T14:41:41.378260+0000","last_clean":"2024-04-23T14:41:41.378260+0000","last_became_active":"2024-04-23T14:41:40.020191+0000","last_became_peered":"2024-04-23T14:41:40.020191+0000","last_unstale":"2024-04-23T14:41:41.378260+0000","last_undegraded":"2024-04-23T14:41:41.378260+0000","last_fullsized":"2024-04-23T14:41:41.378260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:54:34.036464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:44.998205+0000","last_change":"2024-04-23T14:41:39.019507+0000","last_active":"2024-04-23T14:41:44.998205+0000","last_peered":"2024-04-23T14:41:44.998205+0000","last_clean":"2024-04-23T14:41:44.998205+0000","last_became_active":"2024-04-23T14:41:39.019116+0000","last_became_peered":"2024-04-23T14:41:39.019116+0000","last_unstale":"2024-04-23T14:41:44.998205+0000","last_undegraded":"2024-04-23T14:41:44.998205+0000","last_fullsized":"2024-04-23T14:41:44.998205+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:06:05.777908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.028579+0000","last_change":"2024-04-23T14:41:40.032765+0000","last_active":"2024-04-23T14:41:41.028579+0000","last_peered":"2024-04-23T14:41:41.028579+0000","last_clean":"2024-04-23T14:41:41.028579+0000","last_became_active":"2024-04-23T14:41:40.027896+0000","last_became_peered":"2024-04-23T14:41:40.027896+0000","last_unstale":"2024-04-23T14:41:41.028579+0000","last_undegraded":"2024-04-23T14:41:41.028579+0000","last_fullsized":"2024-04-23T14:41:41.028579+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:36:16.225444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373926+0000","last_change":"2024-04-23T14:41:39.021681+0000","last_active":"2024-04-23T14:41:41.373926+0000","last_peered":"2024-04-23T14:41:41.373926+0000","last_clean":"2024-04-23T14:41:41.373926+0000","last_became_active":"2024-04-23T14:41:39.021423+0000","last_became_peered":"2024-04-23T14:41:39.021423+0000","last_unstale":"2024-04-23T14:41:41.373926+0000","last_undegraded":"2024-04-23T14:41:41.373926+0000","last_fullsized":"2024-04-23T14:41:41.373926+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:42:55.351296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.383518+0000","last_change":"2024-04-23T14:41:40.037250+0000","last_active":"2024-04-23T14:41:41.383518+0000","last_peered":"2024-04-23T14:41:41.383518+0000","last_clean":"2024-04-23T14:41:41.383518+0000","last_became_active":"2024-04-23T14:41:40.036645+0000","last_became_peered":"2024-04-23T14:41:40.036645+0000","last_unstale":"2024-04-23T14:41:41.383518+0000","last_undegraded":"2024-04-23T14:41:41.383518+0000","last_fullsized":"2024-04-23T14:41:41.383518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:50:35.556841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.030781+0000","last_change":"2024-04-23T14:41:39.007764+0000","last_active":"2024-04-23T14:41:41.030781+0000","last_peered":"2024-04-23T14:41:41.030781+0000","last_clean":"2024-04-23T14:41:41.030781+0000","last_became_active":"2024-04-23T14:41:39.007312+0000","last_became_peered":"2024-04-23T14:41:39.007312+0000","last_unstale":"2024-04-23T14:41:41.030781+0000","last_undegraded":"2024-04-23T14:41:41.030781+0000","last_fullsized":"2024-04-23T14:41:41.030781+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:38:56.021238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374265+0000","last_change":"2024-04-23T14:41:40.004343+0000","last_active":"2024-04-23T14:41:41.374265+0000","last_peered":"2024-04-23T14:41:41.374265+0000","last_clean":"2024-04-23T14:41:41.374265+0000","last_became_active":"2024-04-23T14:41:40.004015+0000","last_became_peered":"2024-04-23T14:41:40.004015+0000","last_unstale":"2024-04-23T14:41:41.374265+0000","last_undegraded":"2024-04-23T14:41:41.374265+0000","last_fullsized":"2024-04-23T14:41:41.374265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:05:33.863692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.027705+0000","last_change":"2024-04-23T14:41:39.016271+0000","last_active":"2024-04-23T14:41:41.027705+0000","last_peered":"2024-04-23T14:41:41.027705+0000","last_clean":"2024-04-23T14:41:41.027705+0000","last_became_active":"2024-04-23T14:41:39.015851+0000","last_became_peered":"2024-04-23T14:41:39.015851+0000","last_unstale":"2024-04-23T14:41:41.027705+0000","last_undegraded":"2024-04-23T14:41:41.027705+0000","last_fullsized":"2024-04-23T14:41:41.027705+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:50:52.943534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374350+0000","last_change":"2024-04-23T14:41:40.029815+0000","last_active":"2024-04-23T14:41:41.374350+0000","last_peered":"2024-04-23T14:41:41.374350+0000","last_clean":"2024-04-23T14:41:41.374350+0000","last_became_active":"2024-04-23T14:41:40.029496+0000","last_became_peered":"2024-04-23T14:41:40.029496+0000","last_unstale":"2024-04-23T14:41:41.374350+0000","last_undegraded":"2024-04-23T14:41:41.374350+0000","last_fullsized":"2024-04-23T14:41:41.374350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:36:09.228993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374080+0000","last_change":"2024-04-23T14:41:39.001946+0000","last_active":"2024-04-23T14:41:41.374080+0000","last_peered":"2024-04-23T14:41:41.374080+0000","last_clean":"2024-04-23T14:41:41.374080+0000","last_became_active":"2024-04-23T14:41:39.001619+0000","last_became_peered":"2024-04-23T14:41:39.001619+0000","last_unstale":"2024-04-23T14:41:41.374080+0000","last_undegraded":"2024-04-23T14:41:41.374080+0000","last_fullsized":"2024-04-23T14:41:41.374080+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:43:57.680195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.028068+0000","last_change":"2024-04-23T14:41:40.045728+0000","last_active":"2024-04-23T14:41:41.028068+0000","last_peered":"2024-04-23T14:41:41.028068+0000","last_clean":"2024-04-23T14:41:41.028068+0000","last_became_active":"2024-04-23T14:41:40.045293+0000","last_became_peered":"2024-04-23T14:41:40.045293+0000","last_unstale":"2024-04-23T14:41:41.028068+0000","last_undegraded":"2024-04-23T14:41:41.028068+0000","last_fullsized":"2024-04-23T14:41:41.028068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:35:54.340644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.375279+0000","last_change":"2024-04-23T14:41:39.005572+0000","last_active":"2024-04-23T14:41:41.375279+0000","last_peered":"2024-04-23T14:41:41.375279+0000","last_clean":"2024-04-23T14:41:41.375279+0000","last_became_active":"2024-04-23T14:41:39.005070+0000","last_became_peered":"2024-04-23T14:41:39.005070+0000","last_unstale":"2024-04-23T14:41:41.375279+0000","last_undegraded":"2024-04-23T14:41:41.375279+0000","last_fullsized":"2024-04-23T14:41:41.375279+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:45:58.096174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373167+0000","last_change":"2024-04-23T14:41:40.028897+0000","last_active":"2024-04-23T14:41:41.373167+0000","last_peered":"2024-04-23T14:41:41.373167+0000","last_clean":"2024-04-23T14:41:41.373167+0000","last_became_active":"2024-04-23T14:41:40.028557+0000","last_became_peered":"2024-04-23T14:41:40.028557+0000","last_unstale":"2024-04-23T14:41:41.373167+0000","last_undegraded":"2024-04-23T14:41:41.373167+0000","last_fullsized":"2024-04-23T14:41:41.373167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:26:57.871013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373843+0000","last_change":"2024-04-23T14:41:39.012861+0000","last_active":"2024-04-23T14:41:41.373843+0000","last_peered":"2024-04-23T14:41:41.373843+0000","last_clean":"2024-04-23T14:41:41.373843+0000","last_became_active":"2024-04-23T14:41:39.012503+0000","last_became_peered":"2024-04-23T14:41:39.012503+0000","last_unstale":"2024-04-23T14:41:41.373843+0000","last_undegraded":"2024-04-23T14:41:41.373843+0000","last_fullsized":"2024-04-23T14:41:41.373843+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:41:57.347221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373418+0000","last_change":"2024-04-23T14:41:40.021580+0000","last_active":"2024-04-23T14:41:41.373418+0000","last_peered":"2024-04-23T14:41:41.373418+0000","last_clean":"2024-04-23T14:41:41.373418+0000","last_became_active":"2024-04-23T14:41:40.020706+0000","last_became_peered":"2024-04-23T14:41:40.020706+0000","last_unstale":"2024-04-23T14:41:41.373418+0000","last_undegraded":"2024-04-23T14:41:41.373418+0000","last_fullsized":"2024-04-23T14:41:41.373418+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:59:42.576276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373112+0000","last_change":"2024-04-23T14:41:39.006425+0000","last_active":"2024-04-23T14:41:41.373112+0000","last_peered":"2024-04-23T14:41:41.373112+0000","last_clean":"2024-04-23T14:41:41.373112+0000","last_became_active":"2024-04-23T14:41:39.006123+0000","last_became_peered":"2024-04-23T14:41:39.006123+0000","last_unstale":"2024-04-23T14:41:41.373112+0000","last_undegraded":"2024-04-23T14:41:41.373112+0000","last_fullsized":"2024-04-23T14:41:41.373112+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:08:37.261092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"85'1","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:42.263456+0000","last_change":"2024-04-23T14:41:40.034063+0000","last_active":"2024-04-23T14:41:42.263456+0000","last_peered":"2024-04-23T14:41:42.263456+0000","last_clean":"2024-04-23T14:41:42.263456+0000","last_became_active":"2024-04-23T14:41:40.033669+0000","last_became_peered":"2024-04-23T14:41:40.033669+0000","last_unstale":"2024-04-23T14:41:42.263456+0000","last_undegraded":"2024-04-23T14:41:42.263456+0000","last_fullsized":"2024-04-23T14:41:42.263456+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:23:50.608041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374805+0000","last_change":"2024-04-23T14:41:39.025975+0000","last_active":"2024-04-23T14:41:41.374805+0000","last_peered":"2024-04-23T14:41:41.374805+0000","last_clean":"2024-04-23T14:41:41.374805+0000","last_became_active":"2024-04-23T14:41:39.025629+0000","last_became_peered":"2024-04-23T14:41:39.025629+0000","last_unstale":"2024-04-23T14:41:41.374805+0000","last_undegraded":"2024-04-23T14:41:41.374805+0000","last_fullsized":"2024-04-23T14:41:41.374805+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:00:03.753547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373150+0000","last_change":"2024-04-23T14:41:40.013030+0000","last_active":"2024-04-23T14:41:41.373150+0000","last_peered":"2024-04-23T14:41:41.373150+0000","last_clean":"2024-04-23T14:41:41.373150+0000","last_became_active":"2024-04-23T14:41:40.012708+0000","last_became_peered":"2024-04-23T14:41:40.012708+0000","last_unstale":"2024-04-23T14:41:41.373150+0000","last_undegraded":"2024-04-23T14:41:41.373150+0000","last_fullsized":"2024-04-23T14:41:41.373150+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:05:29.217273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374604+0000","last_change":"2024-04-23T14:41:40.010551+0000","last_active":"2024-04-23T14:41:41.374604+0000","last_peered":"2024-04-23T14:41:41.374604+0000","last_clean":"2024-04-23T14:41:41.374604+0000","last_became_active":"2024-04-23T14:41:40.010244+0000","last_became_peered":"2024-04-23T14:41:40.010244+0000","last_unstale":"2024-04-23T14:41:41.374604+0000","last_undegraded":"2024-04-23T14:41:41.374604+0000","last_fullsized":"2024-04-23T14:41:41.374604+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:03:57.223815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373326+0000","last_change":"2024-04-23T14:41:40.009132+0000","last_active":"2024-04-23T14:41:41.373326+0000","last_peered":"2024-04-23T14:41:41.373326+0000","last_clean":"2024-04-23T14:41:41.373326+0000","last_became_active":"2024-04-23T14:41:40.008822+0000","last_became_peered":"2024-04-23T14:41:40.008822+0000","last_unstale":"2024-04-23T14:41:41.373326+0000","last_undegraded":"2024-04-23T14:41:41.373326+0000","last_fullsized":"2024-04-23T14:41:41.373326+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:10:28.513846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374478+0000","last_change":"2024-04-23T14:41:40.041193+0000","last_active":"2024-04-23T14:41:41.374478+0000","last_peered":"2024-04-23T14:41:41.374478+0000","last_clean":"2024-04-23T14:41:41.374478+0000","last_became_active":"2024-04-23T14:41:40.040851+0000","last_became_peered":"2024-04-23T14:41:40.040851+0000","last_unstale":"2024-04-23T14:41:41.374478+0000","last_undegraded":"2024-04-23T14:41:41.374478+0000","last_fullsized":"2024-04-23T14:41:41.374478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:13:30.090015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.382091+0000","last_change":"2024-04-23T14:41:40.042703+0000","last_active":"2024-04-23T14:41:41.382091+0000","last_peered":"2024-04-23T14:41:41.382091+0000","last_clean":"2024-04-23T14:41:41.382091+0000","last_became_active":"2024-04-23T14:41:40.037666+0000","last_became_peered":"2024-04-23T14:41:40.037666+0000","last_unstale":"2024-04-23T14:41:41.382091+0000","last_undegraded":"2024-04-23T14:41:41.382091+0000","last_fullsized":"2024-04-23T14:41:41.382091+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:04:04.240689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.378533+0000","last_change":"2024-04-23T14:41:40.033487+0000","last_active":"2024-04-23T14:41:41.378533+0000","last_peered":"2024-04-23T14:41:41.378533+0000","last_clean":"2024-04-23T14:41:41.378533+0000","last_became_active":"2024-04-23T14:41:40.031799+0000","last_became_peered":"2024-04-23T14:41:40.031799+0000","last_unstale":"2024-04-23T14:41:41.378533+0000","last_undegraded":"2024-04-23T14:41:41.378533+0000","last_fullsized":"2024-04-23T14:41:41.378533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:36:03.930264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374426+0000","last_change":"2024-04-23T14:41:40.012833+0000","last_active":"2024-04-23T14:41:41.374426+0000","last_peered":"2024-04-23T14:41:41.374426+0000","last_clean":"2024-04-23T14:41:41.374426+0000","last_became_active":"2024-04-23T14:41:40.012251+0000","last_became_peered":"2024-04-23T14:41:40.012251+0000","last_unstale":"2024-04-23T14:41:41.374426+0000","last_undegraded":"2024-04-23T14:41:41.374426+0000","last_fullsized":"2024-04-23T14:41:41.374426+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:17:09.415998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374586+0000","last_change":"2024-04-23T14:41:40.013336+0000","last_active":"2024-04-23T14:41:41.374586+0000","last_peered":"2024-04-23T14:41:41.374586+0000","last_clean":"2024-04-23T14:41:41.374586+0000","last_became_active":"2024-04-23T14:41:40.013043+0000","last_became_peered":"2024-04-23T14:41:40.013043+0000","last_unstale":"2024-04-23T14:41:41.374586+0000","last_undegraded":"2024-04-23T14:41:41.374586+0000","last_fullsized":"2024-04-23T14:41:41.374586+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:27:59.118986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.027825+0000","last_change":"2024-04-23T14:41:40.028236+0000","last_active":"2024-04-23T14:41:41.027825+0000","last_peered":"2024-04-23T14:41:41.027825+0000","last_clean":"2024-04-23T14:41:41.027825+0000","last_became_active":"2024-04-23T14:41:40.027504+0000","last_became_peered":"2024-04-23T14:41:40.027504+0000","last_unstale":"2024-04-23T14:41:41.027825+0000","last_undegraded":"2024-04-23T14:41:41.027825+0000","last_fullsized":"2024-04-23T14:41:41.027825+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:04:31.540482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373490+0000","last_change":"2024-04-23T14:41:40.029252+0000","last_active":"2024-04-23T14:41:41.373490+0000","last_peered":"2024-04-23T14:41:41.373490+0000","last_clean":"2024-04-23T14:41:41.373490+0000","last_became_active":"2024-04-23T14:41:40.028937+0000","last_became_peered":"2024-04-23T14:41:40.028937+0000","last_unstale":"2024-04-23T14:41:41.373490+0000","last_undegraded":"2024-04-23T14:41:41.373490+0000","last_fullsized":"2024-04-23T14:41:41.373490+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:46:44.441319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.027635+0000","last_change":"2024-04-23T14:41:40.027705+0000","last_active":"2024-04-23T14:41:41.027635+0000","last_peered":"2024-04-23T14:41:41.027635+0000","last_clean":"2024-04-23T14:41:41.027635+0000","last_became_active":"2024-04-23T14:41:40.027260+0000","last_became_peered":"2024-04-23T14:41:40.027260+0000","last_unstale":"2024-04-23T14:41:41.027635+0000","last_undegraded":"2024-04-23T14:41:41.027635+0000","last_fullsized":"2024-04-23T14:41:41.027635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:33:42.461219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.031062+0000","last_change":"2024-04-23T14:41:40.026490+0000","last_active":"2024-04-23T14:41:41.031062+0000","last_peered":"2024-04-23T14:41:41.031062+0000","last_clean":"2024-04-23T14:41:41.031062+0000","last_became_active":"2024-04-23T14:41:40.026117+0000","last_became_peered":"2024-04-23T14:41:40.026117+0000","last_unstale":"2024-04-23T14:41:41.031062+0000","last_undegraded":"2024-04-23T14:41:41.031062+0000","last_fullsized":"2024-04-23T14:41:41.031062+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:06:17.626663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374770+0000","last_change":"2024-04-23T14:41:40.038489+0000","last_active":"2024-04-23T14:41:41.374770+0000","last_peered":"2024-04-23T14:41:41.374770+0000","last_clean":"2024-04-23T14:41:41.374770+0000","last_became_active":"2024-04-23T14:41:40.038078+0000","last_became_peered":"2024-04-23T14:41:40.038078+0000","last_unstale":"2024-04-23T14:41:41.374770+0000","last_undegraded":"2024-04-23T14:41:41.374770+0000","last_fullsized":"2024-04-23T14:41:41.374770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:58:59.440468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374555+0000","last_change":"2024-04-23T14:41:40.009463+0000","last_active":"2024-04-23T14:41:41.374555+0000","last_peered":"2024-04-23T14:41:41.374555+0000","last_clean":"2024-04-23T14:41:41.374555+0000","last_became_active":"2024-04-23T14:41:40.009154+0000","last_became_peered":"2024-04-23T14:41:40.009154+0000","last_unstale":"2024-04-23T14:41:41.374555+0000","last_undegraded":"2024-04-23T14:41:41.374555+0000","last_fullsized":"2024-04-23T14:41:41.374555+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:13:47.641863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":9,"num_read_kb":8,"num_write":49,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":7230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383694,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":913408,"data_stored":438521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645331,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28520,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716632,"statfs":{"total":95995035648,"available":95965831168,"internally_reserved":0,"allocated":1531904,"data_stored":1030695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808857,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36236,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708916,"statfs":{"total":95995035648,"available":95957929984,"internally_reserved":0,"allocated":917504,"data_stored":439103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070495,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27920,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717232,"statfs":{"total":95995035648,"available":95966445568,"internally_reserved":0,"allocated":921600,"data_stored":439125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332134,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32656,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712496,"statfs":{"total":95995035648,"available":95961595904,"internally_reserved":0,"allocated":1515520,"data_stored":1029588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495658,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":913408,"data_stored":438521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574919,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36244,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708908,"statfs":{"total":95995035648,"available":95957921792,"internally_reserved":0,"allocated":925696,"data_stored":440133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738445,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36236,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708916,"statfs":{"total":95995035648,"available":95957929984,"internally_reserved":0,"allocated":917504,"data_stored":438611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411394,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36248,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708904,"statfs":{"total":95995035648,"available":95957917696,"internally_reserved":0,"allocated":929792,"data_stored":440167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182459,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":921600,"data_stored":438625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920823,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":917504,"data_stored":438543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757296,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28508,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716644,"statfs":{"total":95995035648,"available":95965843456,"internally_reserved":0,"allocated":1519616,"data_stored":1029071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T14:41:48.487 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T14:41:48.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.740+0000 7f294be00640 1 -- 172.21.15.28:0/353299293 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f294c102930 msgr2=0x7f294c1091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:48.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.740+0000 7f294be00640 1 --2- 172.21.15.28:0/353299293 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f294c102930 0x7f294c1091c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2944009ca0 tx=0x7f294402a8c0 comp rx=0 tx=0).stop 2024-04-23T14:41:48.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.740+0000 7f294be00640 1 -- 172.21.15.28:0/353299293 shutdown_connections 2024-04-23T14:41:48.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.740+0000 7f294be00640 1 --2- 172.21.15.28:0/353299293 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f294c101f70 0x7f294c1023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:48.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.740+0000 7f294be00640 1 --2- 172.21.15.28:0/353299293 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f294c102930 0x7f294c1091c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:48.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.740+0000 7f294be00640 1 --2- 172.21.15.28:0/353299293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f294c100d80 0x7f294c1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:48.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.740+0000 7f294be00640 1 -- 172.21.15.28:0/353299293 >> 172.21.15.28:0/353299293 conn(0x7f294c0fc310 msgr2=0x7f294c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:48.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.740+0000 7f294be00640 1 -- 172.21.15.28:0/353299293 shutdown_connections 2024-04-23T14:41:48.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.740+0000 7f294be00640 1 -- 172.21.15.28:0/353299293 wait complete. 2024-04-23T14:41:48.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.741+0000 7f294be00640 1 Processor -- start 2024-04-23T14:41:48.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.741+0000 7f294be00640 1 -- start start 2024-04-23T14:41:48.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.741+0000 7f294be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f294c100d80 0x7f294c19a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:48.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.741+0000 7f294be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f294c101f70 0x7f294c19a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:48.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.741+0000 7f294be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f294c102930 0x7f294c1a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:48.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.741+0000 7f294be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f294c10ca00 con 0x7f294c100d80 2024-04-23T14:41:48.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.741+0000 7f294be00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f294c10c880 con 0x7f294c101f70 2024-04-23T14:41:48.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.741+0000 7f294be00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f294c10cb80 con 0x7f294c102930 2024-04-23T14:41:48.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.742+0000 7f2943e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f294c101f70 0x7f294c19a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:48.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.742+0000 7f294aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f294c100d80 0x7f294c19a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:48.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.742+0000 7f2943e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f294c101f70 0x7f294c19a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:46978/0 (socket says 172.21.15.28:46978) 2024-04-23T14:41:48.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.742+0000 7f2943e00640 1 -- 172.21.15.28:0/1256085777 learned_addr learned my addr 172.21.15.28:0/1256085777 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.742+0000 7f294b400640 1 --2- 172.21.15.28:0/1256085777 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f294c102930 0x7f294c1a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.742+0000 7f294aa00640 1 -- 172.21.15.28:0/1256085777 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f294c102930 msgr2=0x7f294c1a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.742+0000 7f294aa00640 1 --2- 172.21.15.28:0/1256085777 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f294c102930 0x7f294c1a1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.742+0000 7f294aa00640 1 -- 172.21.15.28:0/1256085777 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f294c101f70 msgr2=0x7f294c19a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.742+0000 7f294aa00640 1 --2- 172.21.15.28:0/1256085777 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f294c101f70 0x7f294c19a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.742+0000 7f294aa00640 1 -- 172.21.15.28:0/1256085777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f294c1a1f90 con 0x7f294c100d80 2024-04-23T14:41:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.742+0000 7f2943e00640 1 --2- 172.21.15.28:0/1256085777 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f294c101f70 0x7f294c19a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:41:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.742+0000 7f294b400640 1 --2- 172.21.15.28:0/1256085777 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f294c102930 0x7f294c1a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:41:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.743+0000 7f294aa00640 1 --2- 172.21.15.28:0/1256085777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f294c100d80 0x7f294c19a370 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f294c101dd0 tx=0x7f293c00dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.743+0000 7f2950a00640 1 -- 172.21.15.28:0/1256085777 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293c017070 con 0x7f294c100d80 2024-04-23T14:41:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.743+0000 7f294be00640 1 -- 172.21.15.28:0/1256085777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f294c1a2280 con 0x7f294c100d80 2024-04-23T14:41:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.743+0000 7f2950a00640 1 -- 172.21.15.28:0/1256085777 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f293c009c30 con 0x7f294c100d80 2024-04-23T14:41:48.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.743+0000 7f2950a00640 1 -- 172.21.15.28:0/1256085777 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293c00f620 con 0x7f294c100d80 2024-04-23T14:41:48.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.743+0000 7f294be00640 1 -- 172.21.15.28:0/1256085777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f294c1a27d0 con 0x7f294c100d80 2024-04-23T14:41:48.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.744+0000 7f294be00640 1 -- 172.21.15.28:0/1256085777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f294c1a2a30 con 0x7f294c100d80 2024-04-23T14:41:48.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.746+0000 7f2950a00640 1 -- 172.21.15.28:0/1256085777 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f293c01a070 con 0x7f294c100d80 2024-04-23T14:41:48.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.748+0000 7f2950a00640 1 --2- 172.21.15.28:0/1256085777 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f29300a3c40 0x7f29300a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:48.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.749+0000 7f2950a00640 1 -- 172.21.15.28:0/1256085777 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f293c0cfeb0 con 0x7f294c100d80 2024-04-23T14:41:48.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.749+0000 7f2943e00640 1 --2- 172.21.15.28:0/1256085777 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f29300a3c40 0x7f29300a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:48.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.749+0000 7f2943e00640 1 --2- 172.21.15.28:0/1256085777 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f29300a3c40 0x7f29300a6100 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f294c19b880 tx=0x7f2934005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:48.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.754+0000 7f2950a00640 1 -- 172.21.15.28:0/1256085777 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f293c09c190 con 0x7f294c100d80 2024-04-23T14:41:48.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:48 smithi105 ceph-mon[32555]: from='client.24695 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:41:48.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:48 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1936076258' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:48.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:48 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2409072573' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:41:48.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:48 smithi038 ceph-mon[32348]: from='client.24695 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:41:48.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:48 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1936076258' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:48.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:48 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2409072573' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:41:48.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:48 smithi028 ceph-mon[27810]: from='client.24695 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:41:48.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:48 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1936076258' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:41:48.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:48 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2409072573' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:41:48.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.959+0000 7f294be00640 1 -- 172.21.15.28:0/1256085777 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f294c0fe460 con 0x7f29300a3c40 2024-04-23T14:41:48.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.972+0000 7f2950a00640 1 -- 172.21.15.28:0/1256085777 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258691 (secure 0 0 0) 0x7f294c0fe460 con 0x7f29300a3c40 2024-04-23T14:41:48.977 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:41:48.978 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-23T14:41:48.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.977+0000 7f294be00640 1 -- 172.21.15.28:0/1256085777 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f29300a3c40 msgr2=0x7f29300a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:48.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.977+0000 7f294be00640 1 --2- 172.21.15.28:0/1256085777 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f29300a3c40 0x7f29300a6100 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f294c19b880 tx=0x7f2934005f90 comp rx=0 tx=0).stop 2024-04-23T14:41:48.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.977+0000 7f294be00640 1 -- 172.21.15.28:0/1256085777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f294c100d80 msgr2=0x7f294c19a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:48.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.977+0000 7f294be00640 1 --2- 172.21.15.28:0/1256085777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f294c100d80 0x7f294c19a370 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f294c101dd0 tx=0x7f293c00dde0 comp rx=0 tx=0).stop 2024-04-23T14:41:48.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.977+0000 7f294be00640 1 -- 172.21.15.28:0/1256085777 shutdown_connections 2024-04-23T14:41:48.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.977+0000 7f294be00640 1 --2- 172.21.15.28:0/1256085777 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f29300a3c40 0x7f29300a6100 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:48.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.977+0000 7f294be00640 1 --2- 172.21.15.28:0/1256085777 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f294c101f70 0x7f294c19a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:48.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.977+0000 7f294be00640 1 --2- 172.21.15.28:0/1256085777 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f294c102930 0x7f294c1a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:48.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.977+0000 7f294be00640 1 --2- 172.21.15.28:0/1256085777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f294c100d80 0x7f294c19a370 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:48.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.977+0000 7f294be00640 1 -- 172.21.15.28:0/1256085777 >> 172.21.15.28:0/1256085777 conn(0x7f294c0fc310 msgr2=0x7f294c0fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:48.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.977+0000 7f294be00640 1 -- 172.21.15.28:0/1256085777 shutdown_connections 2024-04-23T14:41:48.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:48.977+0000 7f294be00640 1 -- 172.21.15.28:0/1256085777 wait complete. 2024-04-23T14:41:48.999 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":318,"stamp":"2024-04-23T14:41:47.669539+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592895,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":145,"num_read_kb":127,"num_write":231,"num_write_kb":2955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":179,"ondisk_log_size":179,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378392,"kb_used_data":12544,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563432,"statfs":{"total":1151940427776,"available":1151552954368,"internally_reserved":0,"allocated":12845056,"data_stored":7040703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2527,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":9,"num_read_kb":8,"num_write":50,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003443"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.375045+0000","last_change":"2024-04-23T14:41:40.033061+0000","last_active":"2024-04-23T14:41:41.375045+0000","last_peered":"2024-04-23T14:41:41.375045+0000","last_clean":"2024-04-23T14:41:41.375045+0000","last_became_active":"2024-04-23T14:41:40.032750+0000","last_became_peered":"2024-04-23T14:41:40.032750+0000","last_unstale":"2024-04-23T14:41:41.375045+0000","last_undegraded":"2024-04-23T14:41:41.375045+0000","last_fullsized":"2024-04-23T14:41:41.375045+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:58:03.451539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.382632+0000","last_change":"2024-04-23T14:41:40.037435+0000","last_active":"2024-04-23T14:41:41.382632+0000","last_peered":"2024-04-23T14:41:41.382632+0000","last_clean":"2024-04-23T14:41:41.382632+0000","last_became_active":"2024-04-23T14:41:40.036655+0000","last_became_peered":"2024-04-23T14:41:40.036655+0000","last_unstale":"2024-04-23T14:41:41.382632+0000","last_undegraded":"2024-04-23T14:41:41.382632+0000","last_fullsized":"2024-04-23T14:41:41.382632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:51:58.847850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.383061+0000","last_change":"2024-04-23T14:41:40.024977+0000","last_active":"2024-04-23T14:41:41.383061+0000","last_peered":"2024-04-23T14:41:41.383061+0000","last_clean":"2024-04-23T14:41:41.383061+0000","last_became_active":"2024-04-23T14:41:40.023879+0000","last_became_peered":"2024-04-23T14:41:40.023879+0000","last_unstale":"2024-04-23T14:41:41.383061+0000","last_undegraded":"2024-04-23T14:41:41.383061+0000","last_fullsized":"2024-04-23T14:41:41.383061+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:03:40.672661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373767+0000","last_change":"2024-04-23T14:41:40.017757+0000","last_active":"2024-04-23T14:41:41.373767+0000","last_peered":"2024-04-23T14:41:41.373767+0000","last_clean":"2024-04-23T14:41:41.373767+0000","last_became_active":"2024-04-23T14:41:40.013996+0000","last_became_peered":"2024-04-23T14:41:40.013996+0000","last_unstale":"2024-04-23T14:41:41.373767+0000","last_undegraded":"2024-04-23T14:41:41.373767+0000","last_fullsized":"2024-04-23T14:41:41.373767+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:20:03.376733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.375077+0000","last_change":"2024-04-23T14:41:40.038247+0000","last_active":"2024-04-23T14:41:41.375077+0000","last_peered":"2024-04-23T14:41:41.375077+0000","last_clean":"2024-04-23T14:41:41.375077+0000","last_became_active":"2024-04-23T14:41:40.037934+0000","last_became_peered":"2024-04-23T14:41:40.037934+0000","last_unstale":"2024-04-23T14:41:41.375077+0000","last_undegraded":"2024-04-23T14:41:41.375077+0000","last_fullsized":"2024-04-23T14:41:41.375077+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:15.637733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374016+0000","last_change":"2024-04-23T14:41:40.015341+0000","last_active":"2024-04-23T14:41:41.374016+0000","last_peered":"2024-04-23T14:41:41.374016+0000","last_clean":"2024-04-23T14:41:41.374016+0000","last_became_active":"2024-04-23T14:41:40.013627+0000","last_became_peered":"2024-04-23T14:41:40.013627+0000","last_unstale":"2024-04-23T14:41:41.374016+0000","last_undegraded":"2024-04-23T14:41:41.374016+0000","last_fullsized":"2024-04-23T14:41:41.374016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:05:36.248107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373215+0000","last_change":"2024-04-23T14:41:40.010950+0000","last_active":"2024-04-23T14:41:41.373215+0000","last_peered":"2024-04-23T14:41:41.373215+0000","last_clean":"2024-04-23T14:41:41.373215+0000","last_became_active":"2024-04-23T14:41:40.010584+0000","last_became_peered":"2024-04-23T14:41:40.010584+0000","last_unstale":"2024-04-23T14:41:41.373215+0000","last_undegraded":"2024-04-23T14:41:41.373215+0000","last_fullsized":"2024-04-23T14:41:41.373215+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:46:25.379243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.027387+0000","last_change":"2024-04-23T14:41:40.031198+0000","last_active":"2024-04-23T14:41:41.027387+0000","last_peered":"2024-04-23T14:41:41.027387+0000","last_clean":"2024-04-23T14:41:41.027387+0000","last_became_active":"2024-04-23T14:41:40.030652+0000","last_became_peered":"2024-04-23T14:41:40.030652+0000","last_unstale":"2024-04-23T14:41:41.027387+0000","last_undegraded":"2024-04-23T14:41:41.027387+0000","last_fullsized":"2024-04-23T14:41:41.027387+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:26:00.520182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.031569+0000","last_change":"2024-04-23T14:41:40.009820+0000","last_active":"2024-04-23T14:41:41.031569+0000","last_peered":"2024-04-23T14:41:41.031569+0000","last_clean":"2024-04-23T14:41:41.031569+0000","last_became_active":"2024-04-23T14:41:40.009468+0000","last_became_peered":"2024-04-23T14:41:40.009468+0000","last_unstale":"2024-04-23T14:41:41.031569+0000","last_undegraded":"2024-04-23T14:41:41.031569+0000","last_fullsized":"2024-04-23T14:41:41.031569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:03:50.178050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374485+0000","last_change":"2024-04-23T14:41:40.008363+0000","last_active":"2024-04-23T14:41:41.374485+0000","last_peered":"2024-04-23T14:41:41.374485+0000","last_clean":"2024-04-23T14:41:41.374485+0000","last_became_active":"2024-04-23T14:41:40.006823+0000","last_became_peered":"2024-04-23T14:41:40.006823+0000","last_unstale":"2024-04-23T14:41:41.374485+0000","last_undegraded":"2024-04-23T14:41:41.374485+0000","last_fullsized":"2024-04-23T14:41:41.374485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:12:33.718815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373461+0000","last_change":"2024-04-23T14:41:40.021069+0000","last_active":"2024-04-23T14:41:41.373461+0000","last_peered":"2024-04-23T14:41:41.373461+0000","last_clean":"2024-04-23T14:41:41.373461+0000","last_became_active":"2024-04-23T14:41:40.020334+0000","last_became_peered":"2024-04-23T14:41:40.020334+0000","last_unstale":"2024-04-23T14:41:41.373461+0000","last_undegraded":"2024-04-23T14:41:41.373461+0000","last_fullsized":"2024-04-23T14:41:41.373461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:50:56.560928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373819+0000","last_change":"2024-04-23T14:41:40.017746+0000","last_active":"2024-04-23T14:41:41.373819+0000","last_peered":"2024-04-23T14:41:41.373819+0000","last_clean":"2024-04-23T14:41:41.373819+0000","last_became_active":"2024-04-23T14:41:40.014495+0000","last_became_peered":"2024-04-23T14:41:40.014495+0000","last_unstale":"2024-04-23T14:41:41.373819+0000","last_undegraded":"2024-04-23T14:41:41.373819+0000","last_fullsized":"2024-04-23T14:41:41.373819+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:23:22.188110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.375114+0000","last_change":"2024-04-23T14:41:40.021429+0000","last_active":"2024-04-23T14:41:41.375114+0000","last_peered":"2024-04-23T14:41:41.375114+0000","last_clean":"2024-04-23T14:41:41.375114+0000","last_became_active":"2024-04-23T14:41:40.021087+0000","last_became_peered":"2024-04-23T14:41:40.021087+0000","last_unstale":"2024-04-23T14:41:41.375114+0000","last_undegraded":"2024-04-23T14:41:41.375114+0000","last_fullsized":"2024-04-23T14:41:41.375114+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:51:16.633253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.381552+0000","last_change":"2024-04-23T14:41:40.031839+0000","last_active":"2024-04-23T14:41:41.381552+0000","last_peered":"2024-04-23T14:41:41.381552+0000","last_clean":"2024-04-23T14:41:41.381552+0000","last_became_active":"2024-04-23T14:41:40.031463+0000","last_became_peered":"2024-04-23T14:41:40.031463+0000","last_unstale":"2024-04-23T14:41:41.381552+0000","last_undegraded":"2024-04-23T14:41:41.381552+0000","last_fullsized":"2024-04-23T14:41:41.381552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:51:51.200463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.381081+0000","last_change":"2024-04-23T14:41:40.042575+0000","last_active":"2024-04-23T14:41:41.381081+0000","last_peered":"2024-04-23T14:41:41.381081+0000","last_clean":"2024-04-23T14:41:41.381081+0000","last_became_active":"2024-04-23T14:41:40.033758+0000","last_became_peered":"2024-04-23T14:41:40.033758+0000","last_unstale":"2024-04-23T14:41:41.381081+0000","last_undegraded":"2024-04-23T14:41:41.381081+0000","last_fullsized":"2024-04-23T14:41:41.381081+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:55:17.338169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.030972+0000","last_change":"2024-04-23T14:41:40.036214+0000","last_active":"2024-04-23T14:41:41.030972+0000","last_peered":"2024-04-23T14:41:41.030972+0000","last_clean":"2024-04-23T14:41:41.030972+0000","last_became_active":"2024-04-23T14:41:40.035892+0000","last_became_peered":"2024-04-23T14:41:40.035892+0000","last_unstale":"2024-04-23T14:41:41.030972+0000","last_undegraded":"2024-04-23T14:41:41.030972+0000","last_fullsized":"2024-04-23T14:41:41.030972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:29:18.668690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.375254+0000","last_change":"2024-04-23T14:41:40.029837+0000","last_active":"2024-04-23T14:41:41.375254+0000","last_peered":"2024-04-23T14:41:41.375254+0000","last_clean":"2024-04-23T14:41:41.375254+0000","last_became_active":"2024-04-23T14:41:40.029472+0000","last_became_peered":"2024-04-23T14:41:40.029472+0000","last_unstale":"2024-04-23T14:41:41.375254+0000","last_undegraded":"2024-04-23T14:41:41.375254+0000","last_fullsized":"2024-04-23T14:41:41.375254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:28:52.194898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.378933+0000","last_change":"2024-04-23T14:41:40.015631+0000","last_active":"2024-04-23T14:41:41.378933+0000","last_peered":"2024-04-23T14:41:41.378933+0000","last_clean":"2024-04-23T14:41:41.378933+0000","last_became_active":"2024-04-23T14:41:40.015211+0000","last_became_peered":"2024-04-23T14:41:40.015211+0000","last_unstale":"2024-04-23T14:41:41.378933+0000","last_undegraded":"2024-04-23T14:41:41.378933+0000","last_fullsized":"2024-04-23T14:41:41.378933+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:03:33.485364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373975+0000","last_change":"2024-04-23T14:41:40.041143+0000","last_active":"2024-04-23T14:41:41.373975+0000","last_peered":"2024-04-23T14:41:41.373975+0000","last_clean":"2024-04-23T14:41:41.373975+0000","last_became_active":"2024-04-23T14:41:40.040793+0000","last_became_peered":"2024-04-23T14:41:40.040793+0000","last_unstale":"2024-04-23T14:41:41.373975+0000","last_undegraded":"2024-04-23T14:41:41.373975+0000","last_fullsized":"2024-04-23T14:41:41.373975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:39:14.660719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.818393+0000","last_change":"2024-04-23T14:41:39.017031+0000","last_active":"2024-04-23T14:41:41.818393+0000","last_peered":"2024-04-23T14:41:41.818393+0000","last_clean":"2024-04-23T14:41:41.818393+0000","last_became_active":"2024-04-23T14:41:39.016699+0000","last_became_peered":"2024-04-23T14:41:39.016699+0000","last_unstale":"2024-04-23T14:41:41.818393+0000","last_undegraded":"2024-04-23T14:41:41.818393+0000","last_fullsized":"2024-04-23T14:41:41.818393+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:17:49.812601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.028446+0000","last_change":"2024-04-23T14:41:40.031957+0000","last_active":"2024-04-23T14:41:41.028446+0000","last_peered":"2024-04-23T14:41:41.028446+0000","last_clean":"2024-04-23T14:41:41.028446+0000","last_became_active":"2024-04-23T14:41:40.028151+0000","last_became_peered":"2024-04-23T14:41:40.028151+0000","last_unstale":"2024-04-23T14:41:41.028446+0000","last_undegraded":"2024-04-23T14:41:41.028446+0000","last_fullsized":"2024-04-23T14:41:41.028446+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:46:48.455032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.028416+0000","last_change":"2024-04-23T14:41:39.022503+0000","last_active":"2024-04-23T14:41:41.028416+0000","last_peered":"2024-04-23T14:41:41.028416+0000","last_clean":"2024-04-23T14:41:41.028416+0000","last_became_active":"2024-04-23T14:41:39.022093+0000","last_became_peered":"2024-04-23T14:41:39.022093+0000","last_unstale":"2024-04-23T14:41:41.028416+0000","last_undegraded":"2024-04-23T14:41:41.028416+0000","last_fullsized":"2024-04-23T14:41:41.028416+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:48:27.861140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.383795+0000","last_change":"2024-04-23T14:41:40.032878+0000","last_active":"2024-04-23T14:41:41.383795+0000","last_peered":"2024-04-23T14:41:41.383795+0000","last_clean":"2024-04-23T14:41:41.383795+0000","last_became_active":"2024-04-23T14:41:40.027473+0000","last_became_peered":"2024-04-23T14:41:40.027473+0000","last_unstale":"2024-04-23T14:41:41.383795+0000","last_undegraded":"2024-04-23T14:41:41.383795+0000","last_fullsized":"2024-04-23T14:41:41.383795+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:00:17.677306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.375755+0000","last_change":"2024-04-23T14:41:39.005919+0000","last_active":"2024-04-23T14:41:41.375755+0000","last_peered":"2024-04-23T14:41:41.375755+0000","last_clean":"2024-04-23T14:41:41.375755+0000","last_became_active":"2024-04-23T14:41:39.005550+0000","last_became_peered":"2024-04-23T14:41:39.005550+0000","last_unstale":"2024-04-23T14:41:41.375755+0000","last_undegraded":"2024-04-23T14:41:41.375755+0000","last_fullsized":"2024-04-23T14:41:41.375755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:49:10.362939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.031022+0000","last_change":"2024-04-23T14:41:40.019955+0000","last_active":"2024-04-23T14:41:41.031022+0000","last_peered":"2024-04-23T14:41:41.031022+0000","last_clean":"2024-04-23T14:41:41.031022+0000","last_became_active":"2024-04-23T14:41:40.019579+0000","last_became_peered":"2024-04-23T14:41:40.019579+0000","last_unstale":"2024-04-23T14:41:41.031022+0000","last_undegraded":"2024-04-23T14:41:41.031022+0000","last_fullsized":"2024-04-23T14:41:41.031022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:13:14.330051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.375509+0000","last_change":"2024-04-23T14:41:38.997759+0000","last_active":"2024-04-23T14:41:41.375509+0000","last_peered":"2024-04-23T14:41:41.375509+0000","last_clean":"2024-04-23T14:41:41.375509+0000","last_became_active":"2024-04-23T14:41:38.997445+0000","last_became_peered":"2024-04-23T14:41:38.997445+0000","last_unstale":"2024-04-23T14:41:41.375509+0000","last_undegraded":"2024-04-23T14:41:41.375509+0000","last_fullsized":"2024-04-23T14:41:41.375509+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:35:02.583929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373248+0000","last_change":"2024-04-23T14:41:40.015592+0000","last_active":"2024-04-23T14:41:41.373248+0000","last_peered":"2024-04-23T14:41:41.373248+0000","last_clean":"2024-04-23T14:41:41.373248+0000","last_became_active":"2024-04-23T14:41:40.015292+0000","last_became_peered":"2024-04-23T14:41:40.015292+0000","last_unstale":"2024-04-23T14:41:41.373248+0000","last_undegraded":"2024-04-23T14:41:41.373248+0000","last_fullsized":"2024-04-23T14:41:41.373248+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:52:01.641962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.372978+0000","last_change":"2024-04-23T14:41:39.008526+0000","last_active":"2024-04-23T14:41:41.372978+0000","last_peered":"2024-04-23T14:41:41.372978+0000","last_clean":"2024-04-23T14:41:41.372978+0000","last_became_active":"2024-04-23T14:41:39.008222+0000","last_became_peered":"2024-04-23T14:41:39.008222+0000","last_unstale":"2024-04-23T14:41:41.372978+0000","last_undegraded":"2024-04-23T14:41:41.372978+0000","last_fullsized":"2024-04-23T14:41:41.372978+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:23:09.191426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374601+0000","last_change":"2024-04-23T14:41:40.011486+0000","last_active":"2024-04-23T14:41:41.374601+0000","last_peered":"2024-04-23T14:41:41.374601+0000","last_clean":"2024-04-23T14:41:41.374601+0000","last_became_active":"2024-04-23T14:41:40.011167+0000","last_became_peered":"2024-04-23T14:41:40.011167+0000","last_unstale":"2024-04-23T14:41:41.374601+0000","last_undegraded":"2024-04-23T14:41:41.374601+0000","last_fullsized":"2024-04-23T14:41:41.374601+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:09:12.418428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373832+0000","last_change":"2024-04-23T14:41:39.028566+0000","last_active":"2024-04-23T14:41:41.373832+0000","last_peered":"2024-04-23T14:41:41.373832+0000","last_clean":"2024-04-23T14:41:41.373832+0000","last_became_active":"2024-04-23T14:41:39.028174+0000","last_became_peered":"2024-04-23T14:41:39.028174+0000","last_unstale":"2024-04-23T14:41:41.373832+0000","last_undegraded":"2024-04-23T14:41:41.373832+0000","last_fullsized":"2024-04-23T14:41:41.373832+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:50:55.424741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373195+0000","last_change":"2024-04-23T14:41:40.036171+0000","last_active":"2024-04-23T14:41:41.373195+0000","last_peered":"2024-04-23T14:41:41.373195+0000","last_clean":"2024-04-23T14:41:41.373195+0000","last_became_active":"2024-04-23T14:41:40.035815+0000","last_became_peered":"2024-04-23T14:41:40.035815+0000","last_unstale":"2024-04-23T14:41:41.373195+0000","last_undegraded":"2024-04-23T14:41:41.373195+0000","last_fullsized":"2024-04-23T14:41:41.373195+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:16:38.036654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.375487+0000","last_change":"2024-04-23T14:41:39.028618+0000","last_active":"2024-04-23T14:41:41.375487+0000","last_peered":"2024-04-23T14:41:41.375487+0000","last_clean":"2024-04-23T14:41:41.375487+0000","last_became_active":"2024-04-23T14:41:39.028274+0000","last_became_peered":"2024-04-23T14:41:39.028274+0000","last_unstale":"2024-04-23T14:41:41.375487+0000","last_undegraded":"2024-04-23T14:41:41.375487+0000","last_fullsized":"2024-04-23T14:41:41.375487+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:14:31.571760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.027286+0000","last_change":"2024-04-23T14:41:40.020687+0000","last_active":"2024-04-23T14:41:41.027286+0000","last_peered":"2024-04-23T14:41:41.027286+0000","last_clean":"2024-04-23T14:41:41.027286+0000","last_became_active":"2024-04-23T14:41:40.020352+0000","last_became_peered":"2024-04-23T14:41:40.020352+0000","last_unstale":"2024-04-23T14:41:41.027286+0000","last_undegraded":"2024-04-23T14:41:41.027286+0000","last_fullsized":"2024-04-23T14:41:41.027286+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:29:37.013486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.372658+0000","last_change":"2024-04-23T14:41:39.006002+0000","last_active":"2024-04-23T14:41:41.372658+0000","last_peered":"2024-04-23T14:41:41.372658+0000","last_clean":"2024-04-23T14:41:41.372658+0000","last_became_active":"2024-04-23T14:41:39.005415+0000","last_became_peered":"2024-04-23T14:41:39.005415+0000","last_unstale":"2024-04-23T14:41:41.372658+0000","last_undegraded":"2024-04-23T14:41:41.372658+0000","last_fullsized":"2024-04-23T14:41:41.372658+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:13:55.491931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.378568+0000","last_change":"2024-04-23T14:41:40.029393+0000","last_active":"2024-04-23T14:41:41.378568+0000","last_peered":"2024-04-23T14:41:41.378568+0000","last_clean":"2024-04-23T14:41:41.378568+0000","last_became_active":"2024-04-23T14:41:40.029039+0000","last_became_peered":"2024-04-23T14:41:40.029039+0000","last_unstale":"2024-04-23T14:41:41.378568+0000","last_undegraded":"2024-04-23T14:41:41.378568+0000","last_fullsized":"2024-04-23T14:41:41.378568+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:39:24.461260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374477+0000","last_change":"2024-04-23T14:41:39.014638+0000","last_active":"2024-04-23T14:41:41.374477+0000","last_peered":"2024-04-23T14:41:41.374477+0000","last_clean":"2024-04-23T14:41:41.374477+0000","last_became_active":"2024-04-23T14:41:39.014289+0000","last_became_peered":"2024-04-23T14:41:39.014289+0000","last_unstale":"2024-04-23T14:41:41.374477+0000","last_undegraded":"2024-04-23T14:41:41.374477+0000","last_fullsized":"2024-04-23T14:41:41.374477+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:15:15.855399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.031483+0000","last_change":"2024-04-23T14:41:40.015781+0000","last_active":"2024-04-23T14:41:41.031483+0000","last_peered":"2024-04-23T14:41:41.031483+0000","last_clean":"2024-04-23T14:41:41.031483+0000","last_became_active":"2024-04-23T14:41:40.015423+0000","last_became_peered":"2024-04-23T14:41:40.015423+0000","last_unstale":"2024-04-23T14:41:41.031483+0000","last_undegraded":"2024-04-23T14:41:41.031483+0000","last_fullsized":"2024-04-23T14:41:41.031483+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:56:55.947585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.372817+0000","last_change":"2024-04-23T14:41:39.011021+0000","last_active":"2024-04-23T14:41:41.372817+0000","last_peered":"2024-04-23T14:41:41.372817+0000","last_clean":"2024-04-23T14:41:41.372817+0000","last_became_active":"2024-04-23T14:41:39.010637+0000","last_became_peered":"2024-04-23T14:41:39.010637+0000","last_unstale":"2024-04-23T14:41:41.372817+0000","last_undegraded":"2024-04-23T14:41:41.372817+0000","last_fullsized":"2024-04-23T14:41:41.372817+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:22:22.647170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.383250+0000","last_change":"2024-04-23T14:41:40.042797+0000","last_active":"2024-04-23T14:41:41.383250+0000","last_peered":"2024-04-23T14:41:41.383250+0000","last_clean":"2024-04-23T14:41:41.383250+0000","last_became_active":"2024-04-23T14:41:40.037797+0000","last_became_peered":"2024-04-23T14:41:40.037797+0000","last_unstale":"2024-04-23T14:41:41.383250+0000","last_undegraded":"2024-04-23T14:41:41.383250+0000","last_fullsized":"2024-04-23T14:41:41.383250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:37:32.604276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373338+0000","last_change":"2024-04-23T14:41:39.010585+0000","last_active":"2024-04-23T14:41:41.373338+0000","last_peered":"2024-04-23T14:41:41.373338+0000","last_clean":"2024-04-23T14:41:41.373338+0000","last_became_active":"2024-04-23T14:41:39.010294+0000","last_became_peered":"2024-04-23T14:41:39.010294+0000","last_unstale":"2024-04-23T14:41:41.373338+0000","last_undegraded":"2024-04-23T14:41:41.373338+0000","last_fullsized":"2024-04-23T14:41:41.373338+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:38:03.146904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.031264+0000","last_change":"2024-04-23T14:41:40.013150+0000","last_active":"2024-04-23T14:41:41.031264+0000","last_peered":"2024-04-23T14:41:41.031264+0000","last_clean":"2024-04-23T14:41:41.031264+0000","last_became_active":"2024-04-23T14:41:40.012310+0000","last_became_peered":"2024-04-23T14:41:40.012310+0000","last_unstale":"2024-04-23T14:41:41.031264+0000","last_undegraded":"2024-04-23T14:41:41.031264+0000","last_fullsized":"2024-04-23T14:41:41.031264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:45:25.777200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"84'6","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:43.398339+0000","last_change":"2024-04-23T14:41:39.004256+0000","last_active":"2024-04-23T14:41:43.398339+0000","last_peered":"2024-04-23T14:41:43.398339+0000","last_clean":"2024-04-23T14:41:43.398339+0000","last_became_active":"2024-04-23T14:41:39.003874+0000","last_became_peered":"2024-04-23T14:41:39.003874+0000","last_unstale":"2024-04-23T14:41:43.398339+0000","last_undegraded":"2024-04-23T14:41:43.398339+0000","last_fullsized":"2024-04-23T14:41:43.398339+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:38:30.727149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.375615+0000","last_change":"2024-04-23T14:41:40.012131+0000","last_active":"2024-04-23T14:41:41.375615+0000","last_peered":"2024-04-23T14:41:41.375615+0000","last_clean":"2024-04-23T14:41:41.375615+0000","last_became_active":"2024-04-23T14:41:40.011225+0000","last_became_peered":"2024-04-23T14:41:40.011225+0000","last_unstale":"2024-04-23T14:41:41.375615+0000","last_undegraded":"2024-04-23T14:41:41.375615+0000","last_fullsized":"2024-04-23T14:41:41.375615+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:36:12.085228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373419+0000","last_change":"2024-04-23T14:41:39.001935+0000","last_active":"2024-04-23T14:41:41.373419+0000","last_peered":"2024-04-23T14:41:41.373419+0000","last_clean":"2024-04-23T14:41:41.373419+0000","last_became_active":"2024-04-23T14:41:39.001583+0000","last_became_peered":"2024-04-23T14:41:39.001583+0000","last_unstale":"2024-04-23T14:41:41.373419+0000","last_undegraded":"2024-04-23T14:41:41.373419+0000","last_fullsized":"2024-04-23T14:41:41.373419+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:47:40.986280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374425+0000","last_change":"2024-04-23T14:41:40.036515+0000","last_active":"2024-04-23T14:41:41.374425+0000","last_peered":"2024-04-23T14:41:41.374425+0000","last_clean":"2024-04-23T14:41:41.374425+0000","last_became_active":"2024-04-23T14:41:40.036125+0000","last_became_peered":"2024-04-23T14:41:40.036125+0000","last_unstale":"2024-04-23T14:41:41.374425+0000","last_undegraded":"2024-04-23T14:41:41.374425+0000","last_fullsized":"2024-04-23T14:41:41.374425+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:55:28.465378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.372826+0000","last_change":"2024-04-23T14:41:39.006127+0000","last_active":"2024-04-23T14:41:41.372826+0000","last_peered":"2024-04-23T14:41:41.372826+0000","last_clean":"2024-04-23T14:41:41.372826+0000","last_became_active":"2024-04-23T14:41:39.005761+0000","last_became_peered":"2024-04-23T14:41:39.005761+0000","last_unstale":"2024-04-23T14:41:41.372826+0000","last_undegraded":"2024-04-23T14:41:41.372826+0000","last_fullsized":"2024-04-23T14:41:41.372826+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:10:24.050016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373287+0000","last_change":"2024-04-23T14:41:40.028762+0000","last_active":"2024-04-23T14:41:41.373287+0000","last_peered":"2024-04-23T14:41:41.373287+0000","last_clean":"2024-04-23T14:41:41.373287+0000","last_became_active":"2024-04-23T14:41:40.028440+0000","last_became_peered":"2024-04-23T14:41:40.028440+0000","last_unstale":"2024-04-23T14:41:41.373287+0000","last_undegraded":"2024-04-23T14:41:41.373287+0000","last_fullsized":"2024-04-23T14:41:41.373287+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:22:15.565087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373364+0000","last_change":"2024-04-23T14:41:39.005873+0000","last_active":"2024-04-23T14:41:41.373364+0000","last_peered":"2024-04-23T14:41:41.373364+0000","last_clean":"2024-04-23T14:41:41.373364+0000","last_became_active":"2024-04-23T14:41:39.005481+0000","last_became_peered":"2024-04-23T14:41:39.005481+0000","last_unstale":"2024-04-23T14:41:41.373364+0000","last_undegraded":"2024-04-23T14:41:41.373364+0000","last_fullsized":"2024-04-23T14:41:41.373364+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:49:59.117451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.372278+0000","last_change":"2024-04-23T14:41:40.009346+0000","last_active":"2024-04-23T14:41:41.372278+0000","last_peered":"2024-04-23T14:41:41.372278+0000","last_clean":"2024-04-23T14:41:41.372278+0000","last_became_active":"2024-04-23T14:41:40.008927+0000","last_became_peered":"2024-04-23T14:41:40.008927+0000","last_unstale":"2024-04-23T14:41:41.372278+0000","last_undegraded":"2024-04-23T14:41:41.372278+0000","last_fullsized":"2024-04-23T14:41:41.372278+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:33:36.513349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374236+0000","last_change":"2024-04-23T14:41:39.012298+0000","last_active":"2024-04-23T14:41:41.374236+0000","last_peered":"2024-04-23T14:41:41.374236+0000","last_clean":"2024-04-23T14:41:41.374236+0000","last_became_active":"2024-04-23T14:41:39.012002+0000","last_became_peered":"2024-04-23T14:41:39.012002+0000","last_unstale":"2024-04-23T14:41:41.374236+0000","last_undegraded":"2024-04-23T14:41:41.374236+0000","last_fullsized":"2024-04-23T14:41:41.374236+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:30:06.150083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373386+0000","last_change":"2024-04-23T14:41:40.016845+0000","last_active":"2024-04-23T14:41:41.373386+0000","last_peered":"2024-04-23T14:41:41.373386+0000","last_clean":"2024-04-23T14:41:41.373386+0000","last_became_active":"2024-04-23T14:41:40.016543+0000","last_became_peered":"2024-04-23T14:41:40.016543+0000","last_unstale":"2024-04-23T14:41:41.373386+0000","last_undegraded":"2024-04-23T14:41:41.373386+0000","last_fullsized":"2024-04-23T14:41:41.373386+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:52:41.054454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.375704+0000","last_change":"2024-04-23T14:41:38.999968+0000","last_active":"2024-04-23T14:41:41.375704+0000","last_peered":"2024-04-23T14:41:41.375704+0000","last_clean":"2024-04-23T14:41:41.375704+0000","last_became_active":"2024-04-23T14:41:38.999658+0000","last_became_peered":"2024-04-23T14:41:38.999658+0000","last_unstale":"2024-04-23T14:41:41.375704+0000","last_undegraded":"2024-04-23T14:41:41.375704+0000","last_fullsized":"2024-04-23T14:41:41.375704+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:14:20.990331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374929+0000","last_change":"2024-04-23T14:41:40.011753+0000","last_active":"2024-04-23T14:41:41.374929+0000","last_peered":"2024-04-23T14:41:41.374929+0000","last_clean":"2024-04-23T14:41:41.374929+0000","last_became_active":"2024-04-23T14:41:40.011264+0000","last_became_peered":"2024-04-23T14:41:40.011264+0000","last_unstale":"2024-04-23T14:41:41.374929+0000","last_undegraded":"2024-04-23T14:41:41.374929+0000","last_fullsized":"2024-04-23T14:41:41.374929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:52:22.812046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.378675+0000","last_change":"2024-04-23T14:41:39.027384+0000","last_active":"2024-04-23T14:41:41.378675+0000","last_peered":"2024-04-23T14:41:41.378675+0000","last_clean":"2024-04-23T14:41:41.378675+0000","last_became_active":"2024-04-23T14:41:39.027018+0000","last_became_peered":"2024-04-23T14:41:39.027018+0000","last_unstale":"2024-04-23T14:41:41.378675+0000","last_undegraded":"2024-04-23T14:41:41.378675+0000","last_fullsized":"2024-04-23T14:41:41.378675+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:13:54.685704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374309+0000","last_change":"2024-04-23T14:41:40.041216+0000","last_active":"2024-04-23T14:41:41.374309+0000","last_peered":"2024-04-23T14:41:41.374309+0000","last_clean":"2024-04-23T14:41:41.374309+0000","last_became_active":"2024-04-23T14:41:40.040789+0000","last_became_peered":"2024-04-23T14:41:40.040789+0000","last_unstale":"2024-04-23T14:41:41.374309+0000","last_undegraded":"2024-04-23T14:41:41.374309+0000","last_fullsized":"2024-04-23T14:41:41.374309+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:01:13.156704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.372784+0000","last_change":"2024-04-23T14:41:39.024399+0000","last_active":"2024-04-23T14:41:41.372784+0000","last_peered":"2024-04-23T14:41:41.372784+0000","last_clean":"2024-04-23T14:41:41.372784+0000","last_became_active":"2024-04-23T14:41:39.024141+0000","last_became_peered":"2024-04-23T14:41:39.024141+0000","last_unstale":"2024-04-23T14:41:41.372784+0000","last_undegraded":"2024-04-23T14:41:41.372784+0000","last_fullsized":"2024-04-23T14:41:41.372784+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:34:27.516614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374016+0000","last_change":"2024-04-23T14:41:40.032822+0000","last_active":"2024-04-23T14:41:41.374016+0000","last_peered":"2024-04-23T14:41:41.374016+0000","last_clean":"2024-04-23T14:41:41.374016+0000","last_became_active":"2024-04-23T14:41:40.032473+0000","last_became_peered":"2024-04-23T14:41:40.032473+0000","last_unstale":"2024-04-23T14:41:41.374016+0000","last_undegraded":"2024-04-23T14:41:41.374016+0000","last_fullsized":"2024-04-23T14:41:41.374016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:46:54.316960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.372873+0000","last_change":"2024-04-23T14:41:40.014689+0000","last_active":"2024-04-23T14:41:41.372873+0000","last_peered":"2024-04-23T14:41:41.372873+0000","last_clean":"2024-04-23T14:41:41.372873+0000","last_became_active":"2024-04-23T14:41:40.014371+0000","last_became_peered":"2024-04-23T14:41:40.014371+0000","last_unstale":"2024-04-23T14:41:41.372873+0000","last_undegraded":"2024-04-23T14:41:41.372873+0000","last_fullsized":"2024-04-23T14:41:41.372873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:56:41.284144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.375285+0000","last_change":"2024-04-23T14:41:39.031327+0000","last_active":"2024-04-23T14:41:41.375285+0000","last_peered":"2024-04-23T14:41:41.375285+0000","last_clean":"2024-04-23T14:41:41.375285+0000","last_became_active":"2024-04-23T14:41:39.030864+0000","last_became_peered":"2024-04-23T14:41:39.030864+0000","last_unstale":"2024-04-23T14:41:41.375285+0000","last_undegraded":"2024-04-23T14:41:41.375285+0000","last_fullsized":"2024-04-23T14:41:41.375285+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:58:24.048207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.028216+0000","last_change":"2024-04-23T14:41:40.035104+0000","last_active":"2024-04-23T14:41:41.028216+0000","last_peered":"2024-04-23T14:41:41.028216+0000","last_clean":"2024-04-23T14:41:41.028216+0000","last_became_active":"2024-04-23T14:41:40.034678+0000","last_became_peered":"2024-04-23T14:41:40.034678+0000","last_unstale":"2024-04-23T14:41:41.028216+0000","last_undegraded":"2024-04-23T14:41:41.028216+0000","last_fullsized":"2024-04-23T14:41:41.028216+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:29:53.040424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373833+0000","last_change":"2024-04-23T14:41:39.016684+0000","last_active":"2024-04-23T14:41:41.373833+0000","last_peered":"2024-04-23T14:41:41.373833+0000","last_clean":"2024-04-23T14:41:41.373833+0000","last_became_active":"2024-04-23T14:41:39.016375+0000","last_became_peered":"2024-04-23T14:41:39.016375+0000","last_unstale":"2024-04-23T14:41:41.373833+0000","last_undegraded":"2024-04-23T14:41:41.373833+0000","last_fullsized":"2024-04-23T14:41:41.373833+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:09.449590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":350,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.372242+0000","last_change":"2024-04-23T14:40:22.305413+0000","last_active":"2024-04-23T14:41:41.372242+0000","last_peered":"2024-04-23T14:41:41.372242+0000","last_clean":"2024-04-23T14:41:41.372242+0000","last_became_active":"2024-04-23T14:40:22.304977+0000","last_became_peered":"2024-04-23T14:40:22.304977+0000","last_unstale":"2024-04-23T14:41:41.372242+0000","last_undegraded":"2024-04-23T14:41:41.372242+0000","last_fullsized":"2024-04-23T14:41:41.372242+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:36:28.336753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:36:28.336753+0000","last_clean_scrub_stamp":"2024-04-23T14:36:28.336753+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:42:27.611770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.378793+0000","last_change":"2024-04-23T14:41:40.023300+0000","last_active":"2024-04-23T14:41:41.378793+0000","last_peered":"2024-04-23T14:41:41.378793+0000","last_clean":"2024-04-23T14:41:41.378793+0000","last_became_active":"2024-04-23T14:41:40.022888+0000","last_became_peered":"2024-04-23T14:41:40.022888+0000","last_unstale":"2024-04-23T14:41:41.378793+0000","last_undegraded":"2024-04-23T14:41:41.378793+0000","last_fullsized":"2024-04-23T14:41:41.378793+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:31:48.705005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373248+0000","last_change":"2024-04-23T14:41:39.005482+0000","last_active":"2024-04-23T14:41:41.373248+0000","last_peered":"2024-04-23T14:41:41.373248+0000","last_clean":"2024-04-23T14:41:41.373248+0000","last_became_active":"2024-04-23T14:41:39.004552+0000","last_became_peered":"2024-04-23T14:41:39.004552+0000","last_unstale":"2024-04-23T14:41:41.373248+0000","last_undegraded":"2024-04-23T14:41:41.373248+0000","last_fullsized":"2024-04-23T14:41:41.373248+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:05:51.411210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.375552+0000","last_change":"2024-04-23T14:41:39.015605+0000","last_active":"2024-04-23T14:41:41.375552+0000","last_peered":"2024-04-23T14:41:41.375552+0000","last_clean":"2024-04-23T14:41:41.375552+0000","last_became_active":"2024-04-23T14:41:39.015260+0000","last_became_peered":"2024-04-23T14:41:39.015260+0000","last_unstale":"2024-04-23T14:41:41.375552+0000","last_undegraded":"2024-04-23T14:41:41.375552+0000","last_fullsized":"2024-04-23T14:41:41.375552+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:23:21.251483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.378260+0000","last_change":"2024-04-23T14:41:40.020619+0000","last_active":"2024-04-23T14:41:41.378260+0000","last_peered":"2024-04-23T14:41:41.378260+0000","last_clean":"2024-04-23T14:41:41.378260+0000","last_became_active":"2024-04-23T14:41:40.020191+0000","last_became_peered":"2024-04-23T14:41:40.020191+0000","last_unstale":"2024-04-23T14:41:41.378260+0000","last_undegraded":"2024-04-23T14:41:41.378260+0000","last_fullsized":"2024-04-23T14:41:41.378260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:54:34.036464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:44.998205+0000","last_change":"2024-04-23T14:41:39.019507+0000","last_active":"2024-04-23T14:41:44.998205+0000","last_peered":"2024-04-23T14:41:44.998205+0000","last_clean":"2024-04-23T14:41:44.998205+0000","last_became_active":"2024-04-23T14:41:39.019116+0000","last_became_peered":"2024-04-23T14:41:39.019116+0000","last_unstale":"2024-04-23T14:41:44.998205+0000","last_undegraded":"2024-04-23T14:41:44.998205+0000","last_fullsized":"2024-04-23T14:41:44.998205+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:06:05.777908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.028579+0000","last_change":"2024-04-23T14:41:40.032765+0000","last_active":"2024-04-23T14:41:41.028579+0000","last_peered":"2024-04-23T14:41:41.028579+0000","last_clean":"2024-04-23T14:41:41.028579+0000","last_became_active":"2024-04-23T14:41:40.027896+0000","last_became_peered":"2024-04-23T14:41:40.027896+0000","last_unstale":"2024-04-23T14:41:41.028579+0000","last_undegraded":"2024-04-23T14:41:41.028579+0000","last_fullsized":"2024-04-23T14:41:41.028579+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:36:16.225444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373926+0000","last_change":"2024-04-23T14:41:39.021681+0000","last_active":"2024-04-23T14:41:41.373926+0000","last_peered":"2024-04-23T14:41:41.373926+0000","last_clean":"2024-04-23T14:41:41.373926+0000","last_became_active":"2024-04-23T14:41:39.021423+0000","last_became_peered":"2024-04-23T14:41:39.021423+0000","last_unstale":"2024-04-23T14:41:41.373926+0000","last_undegraded":"2024-04-23T14:41:41.373926+0000","last_fullsized":"2024-04-23T14:41:41.373926+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:42:55.351296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.383518+0000","last_change":"2024-04-23T14:41:40.037250+0000","last_active":"2024-04-23T14:41:41.383518+0000","last_peered":"2024-04-23T14:41:41.383518+0000","last_clean":"2024-04-23T14:41:41.383518+0000","last_became_active":"2024-04-23T14:41:40.036645+0000","last_became_peered":"2024-04-23T14:41:40.036645+0000","last_unstale":"2024-04-23T14:41:41.383518+0000","last_undegraded":"2024-04-23T14:41:41.383518+0000","last_fullsized":"2024-04-23T14:41:41.383518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:50:35.556841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.030781+0000","last_change":"2024-04-23T14:41:39.007764+0000","last_active":"2024-04-23T14:41:41.030781+0000","last_peered":"2024-04-23T14:41:41.030781+0000","last_clean":"2024-04-23T14:41:41.030781+0000","last_became_active":"2024-04-23T14:41:39.007312+0000","last_became_peered":"2024-04-23T14:41:39.007312+0000","last_unstale":"2024-04-23T14:41:41.030781+0000","last_undegraded":"2024-04-23T14:41:41.030781+0000","last_fullsized":"2024-04-23T14:41:41.030781+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:38:56.021238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374265+0000","last_change":"2024-04-23T14:41:40.004343+0000","last_active":"2024-04-23T14:41:41.374265+0000","last_peered":"2024-04-23T14:41:41.374265+0000","last_clean":"2024-04-23T14:41:41.374265+0000","last_became_active":"2024-04-23T14:41:40.004015+0000","last_became_peered":"2024-04-23T14:41:40.004015+0000","last_unstale":"2024-04-23T14:41:41.374265+0000","last_undegraded":"2024-04-23T14:41:41.374265+0000","last_fullsized":"2024-04-23T14:41:41.374265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:05:33.863692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.027705+0000","last_change":"2024-04-23T14:41:39.016271+0000","last_active":"2024-04-23T14:41:41.027705+0000","last_peered":"2024-04-23T14:41:41.027705+0000","last_clean":"2024-04-23T14:41:41.027705+0000","last_became_active":"2024-04-23T14:41:39.015851+0000","last_became_peered":"2024-04-23T14:41:39.015851+0000","last_unstale":"2024-04-23T14:41:41.027705+0000","last_undegraded":"2024-04-23T14:41:41.027705+0000","last_fullsized":"2024-04-23T14:41:41.027705+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:50:52.943534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374350+0000","last_change":"2024-04-23T14:41:40.029815+0000","last_active":"2024-04-23T14:41:41.374350+0000","last_peered":"2024-04-23T14:41:41.374350+0000","last_clean":"2024-04-23T14:41:41.374350+0000","last_became_active":"2024-04-23T14:41:40.029496+0000","last_became_peered":"2024-04-23T14:41:40.029496+0000","last_unstale":"2024-04-23T14:41:41.374350+0000","last_undegraded":"2024-04-23T14:41:41.374350+0000","last_fullsized":"2024-04-23T14:41:41.374350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:36:09.228993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374080+0000","last_change":"2024-04-23T14:41:39.001946+0000","last_active":"2024-04-23T14:41:41.374080+0000","last_peered":"2024-04-23T14:41:41.374080+0000","last_clean":"2024-04-23T14:41:41.374080+0000","last_became_active":"2024-04-23T14:41:39.001619+0000","last_became_peered":"2024-04-23T14:41:39.001619+0000","last_unstale":"2024-04-23T14:41:41.374080+0000","last_undegraded":"2024-04-23T14:41:41.374080+0000","last_fullsized":"2024-04-23T14:41:41.374080+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:43:57.680195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.028068+0000","last_change":"2024-04-23T14:41:40.045728+0000","last_active":"2024-04-23T14:41:41.028068+0000","last_peered":"2024-04-23T14:41:41.028068+0000","last_clean":"2024-04-23T14:41:41.028068+0000","last_became_active":"2024-04-23T14:41:40.045293+0000","last_became_peered":"2024-04-23T14:41:40.045293+0000","last_unstale":"2024-04-23T14:41:41.028068+0000","last_undegraded":"2024-04-23T14:41:41.028068+0000","last_fullsized":"2024-04-23T14:41:41.028068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:35:54.340644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.375279+0000","last_change":"2024-04-23T14:41:39.005572+0000","last_active":"2024-04-23T14:41:41.375279+0000","last_peered":"2024-04-23T14:41:41.375279+0000","last_clean":"2024-04-23T14:41:41.375279+0000","last_became_active":"2024-04-23T14:41:39.005070+0000","last_became_peered":"2024-04-23T14:41:39.005070+0000","last_unstale":"2024-04-23T14:41:41.375279+0000","last_undegraded":"2024-04-23T14:41:41.375279+0000","last_fullsized":"2024-04-23T14:41:41.375279+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:45:58.096174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373167+0000","last_change":"2024-04-23T14:41:40.028897+0000","last_active":"2024-04-23T14:41:41.373167+0000","last_peered":"2024-04-23T14:41:41.373167+0000","last_clean":"2024-04-23T14:41:41.373167+0000","last_became_active":"2024-04-23T14:41:40.028557+0000","last_became_peered":"2024-04-23T14:41:40.028557+0000","last_unstale":"2024-04-23T14:41:41.373167+0000","last_undegraded":"2024-04-23T14:41:41.373167+0000","last_fullsized":"2024-04-23T14:41:41.373167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:26:57.871013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373843+0000","last_change":"2024-04-23T14:41:39.012861+0000","last_active":"2024-04-23T14:41:41.373843+0000","last_peered":"2024-04-23T14:41:41.373843+0000","last_clean":"2024-04-23T14:41:41.373843+0000","last_became_active":"2024-04-23T14:41:39.012503+0000","last_became_peered":"2024-04-23T14:41:39.012503+0000","last_unstale":"2024-04-23T14:41:41.373843+0000","last_undegraded":"2024-04-23T14:41:41.373843+0000","last_fullsized":"2024-04-23T14:41:41.373843+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:41:57.347221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373418+0000","last_change":"2024-04-23T14:41:40.021580+0000","last_active":"2024-04-23T14:41:41.373418+0000","last_peered":"2024-04-23T14:41:41.373418+0000","last_clean":"2024-04-23T14:41:41.373418+0000","last_became_active":"2024-04-23T14:41:40.020706+0000","last_became_peered":"2024-04-23T14:41:40.020706+0000","last_unstale":"2024-04-23T14:41:41.373418+0000","last_undegraded":"2024-04-23T14:41:41.373418+0000","last_fullsized":"2024-04-23T14:41:41.373418+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:59:42.576276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373112+0000","last_change":"2024-04-23T14:41:39.006425+0000","last_active":"2024-04-23T14:41:41.373112+0000","last_peered":"2024-04-23T14:41:41.373112+0000","last_clean":"2024-04-23T14:41:41.373112+0000","last_became_active":"2024-04-23T14:41:39.006123+0000","last_became_peered":"2024-04-23T14:41:39.006123+0000","last_unstale":"2024-04-23T14:41:41.373112+0000","last_undegraded":"2024-04-23T14:41:41.373112+0000","last_fullsized":"2024-04-23T14:41:41.373112+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:08:37.261092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"85'1","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:42.263456+0000","last_change":"2024-04-23T14:41:40.034063+0000","last_active":"2024-04-23T14:41:42.263456+0000","last_peered":"2024-04-23T14:41:42.263456+0000","last_clean":"2024-04-23T14:41:42.263456+0000","last_became_active":"2024-04-23T14:41:40.033669+0000","last_became_peered":"2024-04-23T14:41:40.033669+0000","last_unstale":"2024-04-23T14:41:42.263456+0000","last_undegraded":"2024-04-23T14:41:42.263456+0000","last_fullsized":"2024-04-23T14:41:42.263456+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:23:50.608041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374805+0000","last_change":"2024-04-23T14:41:39.025975+0000","last_active":"2024-04-23T14:41:41.374805+0000","last_peered":"2024-04-23T14:41:41.374805+0000","last_clean":"2024-04-23T14:41:41.374805+0000","last_became_active":"2024-04-23T14:41:39.025629+0000","last_became_peered":"2024-04-23T14:41:39.025629+0000","last_unstale":"2024-04-23T14:41:41.374805+0000","last_undegraded":"2024-04-23T14:41:41.374805+0000","last_fullsized":"2024-04-23T14:41:41.374805+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:00:03.753547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373150+0000","last_change":"2024-04-23T14:41:40.013030+0000","last_active":"2024-04-23T14:41:41.373150+0000","last_peered":"2024-04-23T14:41:41.373150+0000","last_clean":"2024-04-23T14:41:41.373150+0000","last_became_active":"2024-04-23T14:41:40.012708+0000","last_became_peered":"2024-04-23T14:41:40.012708+0000","last_unstale":"2024-04-23T14:41:41.373150+0000","last_undegraded":"2024-04-23T14:41:41.373150+0000","last_fullsized":"2024-04-23T14:41:41.373150+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:05:29.217273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374604+0000","last_change":"2024-04-23T14:41:40.010551+0000","last_active":"2024-04-23T14:41:41.374604+0000","last_peered":"2024-04-23T14:41:41.374604+0000","last_clean":"2024-04-23T14:41:41.374604+0000","last_became_active":"2024-04-23T14:41:40.010244+0000","last_became_peered":"2024-04-23T14:41:40.010244+0000","last_unstale":"2024-04-23T14:41:41.374604+0000","last_undegraded":"2024-04-23T14:41:41.374604+0000","last_fullsized":"2024-04-23T14:41:41.374604+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:03:57.223815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373326+0000","last_change":"2024-04-23T14:41:40.009132+0000","last_active":"2024-04-23T14:41:41.373326+0000","last_peered":"2024-04-23T14:41:41.373326+0000","last_clean":"2024-04-23T14:41:41.373326+0000","last_became_active":"2024-04-23T14:41:40.008822+0000","last_became_peered":"2024-04-23T14:41:40.008822+0000","last_unstale":"2024-04-23T14:41:41.373326+0000","last_undegraded":"2024-04-23T14:41:41.373326+0000","last_fullsized":"2024-04-23T14:41:41.373326+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:10:28.513846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374478+0000","last_change":"2024-04-23T14:41:40.041193+0000","last_active":"2024-04-23T14:41:41.374478+0000","last_peered":"2024-04-23T14:41:41.374478+0000","last_clean":"2024-04-23T14:41:41.374478+0000","last_became_active":"2024-04-23T14:41:40.040851+0000","last_became_peered":"2024-04-23T14:41:40.040851+0000","last_unstale":"2024-04-23T14:41:41.374478+0000","last_undegraded":"2024-04-23T14:41:41.374478+0000","last_fullsized":"2024-04-23T14:41:41.374478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:13:30.090015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.382091+0000","last_change":"2024-04-23T14:41:40.042703+0000","last_active":"2024-04-23T14:41:41.382091+0000","last_peered":"2024-04-23T14:41:41.382091+0000","last_clean":"2024-04-23T14:41:41.382091+0000","last_became_active":"2024-04-23T14:41:40.037666+0000","last_became_peered":"2024-04-23T14:41:40.037666+0000","last_unstale":"2024-04-23T14:41:41.382091+0000","last_undegraded":"2024-04-23T14:41:41.382091+0000","last_fullsized":"2024-04-23T14:41:41.382091+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:04:04.240689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.378533+0000","last_change":"2024-04-23T14:41:40.033487+0000","last_active":"2024-04-23T14:41:41.378533+0000","last_peered":"2024-04-23T14:41:41.378533+0000","last_clean":"2024-04-23T14:41:41.378533+0000","last_became_active":"2024-04-23T14:41:40.031799+0000","last_became_peered":"2024-04-23T14:41:40.031799+0000","last_unstale":"2024-04-23T14:41:41.378533+0000","last_undegraded":"2024-04-23T14:41:41.378533+0000","last_fullsized":"2024-04-23T14:41:41.378533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:36:03.930264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374426+0000","last_change":"2024-04-23T14:41:40.012833+0000","last_active":"2024-04-23T14:41:41.374426+0000","last_peered":"2024-04-23T14:41:41.374426+0000","last_clean":"2024-04-23T14:41:41.374426+0000","last_became_active":"2024-04-23T14:41:40.012251+0000","last_became_peered":"2024-04-23T14:41:40.012251+0000","last_unstale":"2024-04-23T14:41:41.374426+0000","last_undegraded":"2024-04-23T14:41:41.374426+0000","last_fullsized":"2024-04-23T14:41:41.374426+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:17:09.415998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374586+0000","last_change":"2024-04-23T14:41:40.013336+0000","last_active":"2024-04-23T14:41:41.374586+0000","last_peered":"2024-04-23T14:41:41.374586+0000","last_clean":"2024-04-23T14:41:41.374586+0000","last_became_active":"2024-04-23T14:41:40.013043+0000","last_became_peered":"2024-04-23T14:41:40.013043+0000","last_unstale":"2024-04-23T14:41:41.374586+0000","last_undegraded":"2024-04-23T14:41:41.374586+0000","last_fullsized":"2024-04-23T14:41:41.374586+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:27:59.118986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.027825+0000","last_change":"2024-04-23T14:41:40.028236+0000","last_active":"2024-04-23T14:41:41.027825+0000","last_peered":"2024-04-23T14:41:41.027825+0000","last_clean":"2024-04-23T14:41:41.027825+0000","last_became_active":"2024-04-23T14:41:40.027504+0000","last_became_peered":"2024-04-23T14:41:40.027504+0000","last_unstale":"2024-04-23T14:41:41.027825+0000","last_undegraded":"2024-04-23T14:41:41.027825+0000","last_fullsized":"2024-04-23T14:41:41.027825+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:04:31.540482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.373490+0000","last_change":"2024-04-23T14:41:40.029252+0000","last_active":"2024-04-23T14:41:41.373490+0000","last_peered":"2024-04-23T14:41:41.373490+0000","last_clean":"2024-04-23T14:41:41.373490+0000","last_became_active":"2024-04-23T14:41:40.028937+0000","last_became_peered":"2024-04-23T14:41:40.028937+0000","last_unstale":"2024-04-23T14:41:41.373490+0000","last_undegraded":"2024-04-23T14:41:41.373490+0000","last_fullsized":"2024-04-23T14:41:41.373490+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:46:44.441319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.027635+0000","last_change":"2024-04-23T14:41:40.027705+0000","last_active":"2024-04-23T14:41:41.027635+0000","last_peered":"2024-04-23T14:41:41.027635+0000","last_clean":"2024-04-23T14:41:41.027635+0000","last_became_active":"2024-04-23T14:41:40.027260+0000","last_became_peered":"2024-04-23T14:41:40.027260+0000","last_unstale":"2024-04-23T14:41:41.027635+0000","last_undegraded":"2024-04-23T14:41:41.027635+0000","last_fullsized":"2024-04-23T14:41:41.027635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:33:42.461219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.031062+0000","last_change":"2024-04-23T14:41:40.026490+0000","last_active":"2024-04-23T14:41:41.031062+0000","last_peered":"2024-04-23T14:41:41.031062+0000","last_clean":"2024-04-23T14:41:41.031062+0000","last_became_active":"2024-04-23T14:41:40.026117+0000","last_became_peered":"2024-04-23T14:41:40.026117+0000","last_unstale":"2024-04-23T14:41:41.031062+0000","last_undegraded":"2024-04-23T14:41:41.031062+0000","last_fullsized":"2024-04-23T14:41:41.031062+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:06:17.626663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374770+0000","last_change":"2024-04-23T14:41:40.038489+0000","last_active":"2024-04-23T14:41:41.374770+0000","last_peered":"2024-04-23T14:41:41.374770+0000","last_clean":"2024-04-23T14:41:41.374770+0000","last_became_active":"2024-04-23T14:41:40.038078+0000","last_became_peered":"2024-04-23T14:41:40.038078+0000","last_unstale":"2024-04-23T14:41:41.374770+0000","last_undegraded":"2024-04-23T14:41:41.374770+0000","last_fullsized":"2024-04-23T14:41:41.374770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:58:59.440468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-23T14:41:41.374555+0000","last_change":"2024-04-23T14:41:40.009463+0000","last_active":"2024-04-23T14:41:41.374555+0000","last_peered":"2024-04-23T14:41:41.374555+0000","last_clean":"2024-04-23T14:41:41.374555+0000","last_became_active":"2024-04-23T14:41:40.009154+0000","last_became_peered":"2024-04-23T14:41:40.009154+0000","last_unstale":"2024-04-23T14:41:41.374555+0000","last_undegraded":"2024-04-23T14:41:41.374555+0000","last_fullsized":"2024-04-23T14:41:41.374555+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:13:47.641863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":9,"num_read_kb":8,"num_write":49,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":7230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383694,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":913408,"data_stored":438521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645331,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28520,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716632,"statfs":{"total":95995035648,"available":95965831168,"internally_reserved":0,"allocated":1531904,"data_stored":1030695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808857,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36236,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708916,"statfs":{"total":95995035648,"available":95957929984,"internally_reserved":0,"allocated":917504,"data_stored":439103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070495,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27920,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717232,"statfs":{"total":95995035648,"available":95966445568,"internally_reserved":0,"allocated":921600,"data_stored":439125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332134,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32656,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712496,"statfs":{"total":95995035648,"available":95961595904,"internally_reserved":0,"allocated":1515520,"data_stored":1029588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495658,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":913408,"data_stored":438521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574919,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36244,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708908,"statfs":{"total":95995035648,"available":95957921792,"internally_reserved":0,"allocated":925696,"data_stored":440133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738445,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36236,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708916,"statfs":{"total":95995035648,"available":95957929984,"internally_reserved":0,"allocated":917504,"data_stored":438611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411394,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36248,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708904,"statfs":{"total":95995035648,"available":95957917696,"internally_reserved":0,"allocated":929792,"data_stored":440167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182459,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":921600,"data_stored":438625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920823,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":917504,"data_stored":438543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757296,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28508,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716644,"statfs":{"total":95995035648,"available":95965843456,"internally_reserved":0,"allocated":1519616,"data_stored":1029071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T14:41:49.005 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-23T14:41:49.006 INFO:tasks.cephadm:Setting up client nodes... 2024-04-23T14:41:49.006 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-23T14:41:49.513 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:41:49.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:49 smithi105 ceph-mon[32555]: pgmap v318: 97 pgs: 97 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 2.1 KiB/s wr, 7 op/s 2024-04-23T14:41:49.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:49 smithi105 ceph-mon[32555]: from='client.14589 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T14:41:49.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:49 smithi028 ceph-mon[27810]: pgmap v318: 97 pgs: 97 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 2.1 KiB/s wr, 7 op/s 2024-04-23T14:41:49.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:49 smithi028 ceph-mon[27810]: from='client.14589 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T14:41:49.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:49 smithi038 ceph-mon[32348]: pgmap v318: 97 pgs: 97 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 2.1 KiB/s wr, 7 op/s 2024-04-23T14:41:49.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:49 smithi038 ceph-mon[32348]: from='client.14589 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T14:41:50.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:50 smithi028 ceph-mon[27810]: from='client.14595 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T14:41:50.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:50 smithi105 ceph-mon[32555]: from='client.14595 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T14:41:50.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:50 smithi038 ceph-mon[32348]: from='client.14595 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T14:41:51.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.657+0000 7f7382a00700 1 -- 172.21.15.28:0/1502864069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f737c106940 msgr2=0x7f737c10d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:51.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.657+0000 7f7382a00700 1 --2- 172.21.15.28:0/1502864069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f737c106940 0x7f737c10d1f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f737400b0d0 tx=0x7f737400b3e0 comp rx=0 tx=0).stop 2024-04-23T14:41:51.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.658+0000 7f7382a00700 1 -- 172.21.15.28:0/1502864069 shutdown_connections 2024-04-23T14:41:51.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.658+0000 7f7382a00700 1 --2- 172.21.15.28:0/1502864069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f737c106940 0x7f737c10d1f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:51.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.658+0000 7f7382a00700 1 --2- 172.21.15.28:0/1502864069 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f737c104da0 0x7f737c1051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:51.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.658+0000 7f7382a00700 1 --2- 172.21.15.28:0/1502864069 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f737c105fa0 0x7f737c106400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:51.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.658+0000 7f7382a00700 1 -- 172.21.15.28:0/1502864069 >> 172.21.15.28:0/1502864069 conn(0x7f737c100340 msgr2=0x7f737c102780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:51.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.658+0000 7f7382a00700 1 -- 172.21.15.28:0/1502864069 shutdown_connections 2024-04-23T14:41:51.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.658+0000 7f7382a00700 1 -- 172.21.15.28:0/1502864069 wait complete. 2024-04-23T14:41:51.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.659+0000 7f7382a00700 1 Processor -- start 2024-04-23T14:41:51.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.659+0000 7f7382a00700 1 -- start start 2024-04-23T14:41:51.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.659+0000 7f7382a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f737c104da0 0x7f737c19ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:51.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.659+0000 7f7382a00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f737c105fa0 0x7f737c19b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:51.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.659+0000 7f7382a00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f737c106940 0x7f737c1a2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:51.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.659+0000 7f7382a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f737c1109e0 con 0x7f737c104da0 2024-04-23T14:41:51.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.659+0000 7f7382a00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f737c110860 con 0x7f737c105fa0 2024-04-23T14:41:51.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.659+0000 7f7382a00700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f737c110b60 con 0x7f737c106940 2024-04-23T14:41:51.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.660+0000 7f737be00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f737c104da0 0x7f737c19ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:51.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.660+0000 7f7373e00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f737c105fa0 0x7f737c19b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:51.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.660+0000 7f737be00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f737c104da0 0x7f737c19ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59730/0 (socket says 172.21.15.28:59730) 2024-04-23T14:41:51.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.660+0000 7f7373e00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f737c105fa0 0x7f737c19b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:46994/0 (socket says 172.21.15.28:46994) 2024-04-23T14:41:51.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.660+0000 7f737be00700 1 -- 172.21.15.28:0/2778966335 learned_addr learned my addr 172.21.15.28:0/2778966335 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:51.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.660+0000 7f7380a00700 1 --2- 172.21.15.28:0/2778966335 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f737c106940 0x7f737c1a2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:51.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.660+0000 7f7373e00700 1 -- 172.21.15.28:0/2778966335 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f737c106940 msgr2=0x7f737c1a2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:51.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.660+0000 7f7373e00700 1 --2- 172.21.15.28:0/2778966335 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f737c106940 0x7f737c1a2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:51.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.660+0000 7f7373e00700 1 -- 172.21.15.28:0/2778966335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f737c104da0 msgr2=0x7f737c19ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:51.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.660+0000 7f7373e00700 1 --2- 172.21.15.28:0/2778966335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f737c104da0 0x7f737c19ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:51.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.660+0000 7f7373e00700 1 -- 172.21.15.28:0/2778966335 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f737c1a2ac0 con 0x7f737c105fa0 2024-04-23T14:41:51.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.661+0000 7f7373e00700 1 --2- 172.21.15.28:0/2778966335 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f737c105fa0 0x7f737c19b3a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f736400e9f0 tx=0x7f736400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:51.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.661+0000 7f7379600700 1 -- 172.21.15.28:0/2778966335 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7364004220 con 0x7f737c105fa0 2024-04-23T14:41:51.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.661+0000 7f7379600700 1 -- 172.21.15.28:0/2778966335 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7364004d10 con 0x7f737c105fa0 2024-04-23T14:41:51.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.661+0000 7f7379600700 1 -- 172.21.15.28:0/2778966335 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7364005020 con 0x7f737c105fa0 2024-04-23T14:41:51.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.661+0000 7f7382a00700 1 -- 172.21.15.28:0/2778966335 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f737c1a2db0 con 0x7f737c105fa0 2024-04-23T14:41:51.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.661+0000 7f7382a00700 1 -- 172.21.15.28:0/2778966335 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f737c1a3210 con 0x7f737c105fa0 2024-04-23T14:41:51.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.662+0000 7f7382a00700 1 -- 172.21.15.28:0/2778966335 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f737c02dfe0 con 0x7f737c105fa0 2024-04-23T14:41:51.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.664+0000 7f7379600700 1 -- 172.21.15.28:0/2778966335 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7364004750 con 0x7f737c105fa0 2024-04-23T14:41:51.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.665+0000 7f7379600700 1 --2- 172.21.15.28:0/2778966335 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f73600a3e00 0x7f73600a62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:51.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.666+0000 7f7379600700 1 -- 172.21.15.28:0/2778966335 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f73640d02d0 con 0x7f737c105fa0 2024-04-23T14:41:51.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.666+0000 7f737be00700 1 --2- 172.21.15.28:0/2778966335 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f73600a3e00 0x7f73600a62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:51.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.666+0000 7f737be00700 1 --2- 172.21.15.28:0/2778966335 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f73600a3e00 0x7f73600a62c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f736c005950 tx=0x7f736c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:51.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.670+0000 7f7379600700 1 -- 172.21.15.28:0/2778966335 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f736409cc00 con 0x7f737c105fa0 2024-04-23T14:41:51.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:51 smithi105 ceph-mon[32555]: pgmap v319: 97 pgs: 97 active+clean; 616 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 6.0 KiB/s wr, 8 op/s 2024-04-23T14:41:51.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:51 smithi038 ceph-mon[32348]: pgmap v319: 97 pgs: 97 active+clean; 616 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 6.0 KiB/s wr, 8 op/s 2024-04-23T14:41:51.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:51 smithi028 ceph-mon[27810]: pgmap v319: 97 pgs: 97 active+clean; 616 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 6.0 KiB/s wr, 8 op/s 2024-04-23T14:41:51.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:51.990+0000 7f7382a00700 1 -- 172.21.15.28:0/2778966335 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f737c02e260 con 0x7f737c105fa0 2024-04-23T14:41:52.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:52.000+0000 7f7379600700 1 -- 172.21.15.28:0/2778966335 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f736409c500 con 0x7f737c105fa0 2024-04-23T14:41:52.001 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-04-23T14:41:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQCvyCdmCaI8OxAAEP/HHpaGH6iAdcY6sSgCKg== 2024-04-23T14:41:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:52.003+0000 7f7382a00700 1 -- 172.21.15.28:0/2778966335 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f73600a3e00 msgr2=0x7f73600a62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:52.003+0000 7f7382a00700 1 --2- 172.21.15.28:0/2778966335 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f73600a3e00 0x7f73600a62c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f736c005950 tx=0x7f736c009260 comp rx=0 tx=0).stop 2024-04-23T14:41:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:52.003+0000 7f7382a00700 1 -- 172.21.15.28:0/2778966335 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f737c105fa0 msgr2=0x7f737c19b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:52.003+0000 7f7382a00700 1 --2- 172.21.15.28:0/2778966335 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f737c105fa0 0x7f737c19b3a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f736400e9f0 tx=0x7f736400ed00 comp rx=0 tx=0).stop 2024-04-23T14:41:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:52.003+0000 7f7382a00700 1 -- 172.21.15.28:0/2778966335 shutdown_connections 2024-04-23T14:41:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:52.003+0000 7f7382a00700 1 --2- 172.21.15.28:0/2778966335 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f73600a3e00 0x7f73600a62c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:52.003+0000 7f7382a00700 1 --2- 172.21.15.28:0/2778966335 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f737c106940 0x7f737c1a2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:52.003+0000 7f7382a00700 1 --2- 172.21.15.28:0/2778966335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f737c104da0 0x7f737c19ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:52.003+0000 7f7382a00700 1 --2- 172.21.15.28:0/2778966335 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f737c105fa0 0x7f737c19b3a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:52.003+0000 7f7382a00700 1 -- 172.21.15.28:0/2778966335 >> 172.21.15.28:0/2778966335 conn(0x7f737c100340 msgr2=0x7f737c1091d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:52.004+0000 7f7382a00700 1 -- 172.21.15.28:0/2778966335 shutdown_connections 2024-04-23T14:41:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:52.004+0000 7f7382a00700 1 -- 172.21.15.28:0/2778966335 wait complete. 2024-04-23T14:41:52.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:52 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2778966335' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T14:41:52.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:52 smithi028 ceph-mon[27810]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T14:41:52.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:52 smithi028 ceph-mon[27810]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-23T14:41:52.782 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-23T14:41:52.783 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-23T14:41:52.783 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-23T14:41:52.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:52 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2778966335' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T14:41:52.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:52 smithi105 ceph-mon[32555]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T14:41:52.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:52 smithi105 ceph-mon[32555]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-23T14:41:52.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:52 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2778966335' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T14:41:52.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:52 smithi038 ceph-mon[32348]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T14:41:52.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:52 smithi038 ceph-mon[32348]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-23T14:41:52.929 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-23T14:41:52.929 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-23T14:41:52.930 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-23T14:41:53.458 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:41:53.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:53 smithi105 ceph-mon[32555]: pgmap v320: 97 pgs: 97 active+clean; 616 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 5.2 KiB/s wr, 5 op/s 2024-04-23T14:41:53.830 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:53 smithi028 ceph-mon[27810]: pgmap v320: 97 pgs: 97 active+clean; 616 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 5.2 KiB/s wr, 5 op/s 2024-04-23T14:41:53.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:53 smithi038 ceph-mon[32348]: pgmap v320: 97 pgs: 97 active+clean; 616 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 5.2 KiB/s wr, 5 op/s 2024-04-23T14:41:55.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.697+0000 7fbb27200700 1 -- 172.21.15.28:0/646075136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb20107ab0 msgr2=0x7fbb20107e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:55.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.697+0000 7fbb27200700 1 --2- 172.21.15.28:0/646075136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb20107ab0 0x7fbb20107e90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbb10009990 tx=0x7fbb10009ca0 comp rx=0 tx=0).stop 2024-04-23T14:41:55.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.697+0000 7fbb27200700 1 -- 172.21.15.28:0/646075136 shutdown_connections 2024-04-23T14:41:55.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.697+0000 7fbb27200700 1 --2- 172.21.15.28:0/646075136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb20107ab0 0x7fbb20107e90 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:55.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.697+0000 7fbb27200700 1 --2- 172.21.15.28:0/646075136 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb20103540 0x7fbb2010fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:55.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.697+0000 7fbb27200700 1 --2- 172.21.15.28:0/646075136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb20102ba0 0x7fbb20103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:55.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.697+0000 7fbb27200700 1 -- 172.21.15.28:0/646075136 >> 172.21.15.28:0/646075136 conn(0x7fbb20079c50 msgr2=0x7fbb2007a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:55.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.697+0000 7fbb27200700 1 -- 172.21.15.28:0/646075136 shutdown_connections 2024-04-23T14:41:55.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.697+0000 7fbb27200700 1 -- 172.21.15.28:0/646075136 wait complete. 2024-04-23T14:41:55.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.699+0000 7fbb27200700 1 Processor -- start 2024-04-23T14:41:55.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.699+0000 7fbb27200700 1 -- start start 2024-04-23T14:41:55.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.700+0000 7fbb27200700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb20102ba0 0x7fbb2019b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:55.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.700+0000 7fbb27200700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb20103540 0x7fbb2019bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:55.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.700+0000 7fbb27200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb20107ab0 0x7fbb2019fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:55.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.700+0000 7fbb27200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbb20112e90 con 0x7fbb20107ab0 2024-04-23T14:41:55.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.700+0000 7fbb27200700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbb20112d10 con 0x7fbb20102ba0 2024-04-23T14:41:55.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.700+0000 7fbb27200700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fbb20113010 con 0x7fbb20103540 2024-04-23T14:41:55.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.700+0000 7fbb17400700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb20103540 0x7fbb2019bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:55.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.700+0000 7fbb25200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb20107ab0 0x7fbb2019fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:55.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.700+0000 7fbb17400700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb20103540 0x7fbb2019bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:42844/0 (socket says 172.21.15.28:42844) 2024-04-23T14:41:55.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.700+0000 7fbb1fe00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb20102ba0 0x7fbb2019b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:55.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.700+0000 7fbb17400700 1 -- 172.21.15.28:0/2169479597 learned_addr learned my addr 172.21.15.28:0/2169479597 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:55.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.700+0000 7fbb17400700 1 -- 172.21.15.28:0/2169479597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb20102ba0 msgr2=0x7fbb2019b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:55.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.700+0000 7fbb17400700 1 --2- 172.21.15.28:0/2169479597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb20102ba0 0x7fbb2019b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:55.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.700+0000 7fbb17400700 1 -- 172.21.15.28:0/2169479597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb20107ab0 msgr2=0x7fbb2019fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:55.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.700+0000 7fbb17400700 1 --2- 172.21.15.28:0/2169479597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb20107ab0 0x7fbb2019fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:55.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.700+0000 7fbb17400700 1 -- 172.21.15.28:0/2169479597 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb201a0590 con 0x7fbb20103540 2024-04-23T14:41:55.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.701+0000 7fbb17400700 1 --2- 172.21.15.28:0/2169479597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb20103540 0x7fbb2019bac0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbb0800eb30 tx=0x7fbb0800ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:55.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.701+0000 7fbb1d600700 1 -- 172.21.15.28:0/2169479597 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb0800cbd0 con 0x7fbb20103540 2024-04-23T14:41:55.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.701+0000 7fbb1d600700 1 -- 172.21.15.28:0/2169479597 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb08004500 con 0x7fbb20103540 2024-04-23T14:41:55.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.701+0000 7fbb1d600700 1 -- 172.21.15.28:0/2169479597 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb08010430 con 0x7fbb20103540 2024-04-23T14:41:55.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.701+0000 7fbb27200700 1 -- 172.21.15.28:0/2169479597 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb201a0880 con 0x7fbb20103540 2024-04-23T14:41:55.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.701+0000 7fbb27200700 1 -- 172.21.15.28:0/2169479597 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb201a8100 con 0x7fbb20103540 2024-04-23T14:41:55.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.702+0000 7fbb27200700 1 -- 172.21.15.28:0/2169479597 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb200697e0 con 0x7fbb20103540 2024-04-23T14:41:55.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.705+0000 7fbb1d600700 1 -- 172.21.15.28:0/2169479597 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fbb08003f00 con 0x7fbb20103540 2024-04-23T14:41:55.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.706+0000 7fbb1d600700 1 --2- 172.21.15.28:0/2169479597 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fbb000a3d50 0x7fbb000a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:55.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.706+0000 7fbb1d600700 1 -- 172.21.15.28:0/2169479597 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fbb080d0950 con 0x7fbb20103540 2024-04-23T14:41:55.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.706+0000 7fbb1fe00700 1 --2- 172.21.15.28:0/2169479597 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fbb000a3d50 0x7fbb000a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:55.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.707+0000 7fbb1fe00700 1 --2- 172.21.15.28:0/2169479597 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fbb000a3d50 0x7fbb000a6210 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fbb10006010 tx=0x7fbb1000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:55.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.708+0000 7fbb1d600700 1 -- 172.21.15.28:0/2169479597 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbb0801f340 con 0x7fbb20103540 2024-04-23T14:41:55.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:55 smithi105 ceph-mon[32555]: pgmap v321: 97 pgs: 97 active+clean; 616 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 4.7 KiB/s wr, 5 op/s 2024-04-23T14:41:55.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:55 smithi038 ceph-mon[32348]: pgmap v321: 97 pgs: 97 active+clean; 616 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 4.7 KiB/s wr, 5 op/s 2024-04-23T14:41:55.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:55 smithi028 ceph-mon[27810]: pgmap v321: 97 pgs: 97 active+clean; 616 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 4.7 KiB/s wr, 5 op/s 2024-04-23T14:41:56.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:55.999+0000 7fbb27200700 1 -- 172.21.15.28:0/2169479597 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fbb201a82d0 con 0x7fbb20103540 2024-04-23T14:41:56.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:56.005+0000 7fbb1d600700 1 -- 172.21.15.28:0/2169479597 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264995 (secure 0 0 0) 0x7fbb0809d280 con 0x7fbb20103540 2024-04-23T14:41:56.008 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:41:56.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:56.010+0000 7fbb27200700 1 -- 172.21.15.28:0/2169479597 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fbb000a3d50 msgr2=0x7fbb000a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:56.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:56.011+0000 7fbb27200700 1 --2- 172.21.15.28:0/2169479597 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fbb000a3d50 0x7fbb000a6210 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fbb10006010 tx=0x7fbb1000b260 comp rx=0 tx=0).stop 2024-04-23T14:41:56.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:56.011+0000 7fbb27200700 1 -- 172.21.15.28:0/2169479597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb20103540 msgr2=0x7fbb2019bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:56.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:56.011+0000 7fbb27200700 1 --2- 172.21.15.28:0/2169479597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb20103540 0x7fbb2019bac0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbb0800eb30 tx=0x7fbb0800ee40 comp rx=0 tx=0).stop 2024-04-23T14:41:56.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:56.011+0000 7fbb27200700 1 -- 172.21.15.28:0/2169479597 shutdown_connections 2024-04-23T14:41:56.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:56.011+0000 7fbb27200700 1 --2- 172.21.15.28:0/2169479597 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fbb000a3d50 0x7fbb000a6210 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:56.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:56.011+0000 7fbb27200700 1 --2- 172.21.15.28:0/2169479597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb20107ab0 0x7fbb2019fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:56.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:56.011+0000 7fbb27200700 1 --2- 172.21.15.28:0/2169479597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb20102ba0 0x7fbb2019b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:56.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:56.011+0000 7fbb27200700 1 --2- 172.21.15.28:0/2169479597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb20103540 0x7fbb2019bac0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:56.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:56.011+0000 7fbb27200700 1 -- 172.21.15.28:0/2169479597 >> 172.21.15.28:0/2169479597 conn(0x7fbb20079c50 msgr2=0x7fbb20100a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:56.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:56.011+0000 7fbb27200700 1 -- 172.21.15.28:0/2169479597 shutdown_connections 2024-04-23T14:41:56.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:56.011+0000 7fbb27200700 1 -- 172.21.15.28:0/2169479597 wait complete. 2024-04-23T14:41:56.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:56 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2169479597' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-23T14:41:56.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:56 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2169479597' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-23T14:41:56.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:56 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2169479597' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-23T14:41:56.881 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1997136173},{"type":"v1","addr":"172.21.15.28:6801","nonce":1997136173}]},"active_addr":"172.21.15.28:6801/1997136173","active_change":"2024-04-23T14:33:41.530623+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.28:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":905362079}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":38494154}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":3847772687}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":1997097543}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":3267144610}]}]} 2024-04-23T14:41:56.887 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-23T14:41:56.887 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-23T14:41:56.887 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-23T14:41:57.341 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:41:57.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:57 smithi105 ceph-mon[32555]: pgmap v322: 97 pgs: 97 active+clean; 621 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.5 KiB/s wr, 5 op/s 2024-04-23T14:41:57.866 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:57 smithi028 ceph-mon[27810]: pgmap v322: 97 pgs: 97 active+clean; 621 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.5 KiB/s wr, 5 op/s 2024-04-23T14:41:57.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:57 smithi038 ceph-mon[32348]: pgmap v322: 97 pgs: 97 active+clean; 621 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.5 KiB/s wr, 5 op/s 2024-04-23T14:41:59.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:41:59 smithi028 ceph-mon[27810]: pgmap v323: 97 pgs: 97 active+clean; 621 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 4.2 KiB/s wr, 4 op/s 2024-04-23T14:41:59.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.753+0000 7f1e2cc00700 1 -- 172.21.15.28:0/1783880417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e2806b120 msgr2=0x7f1e2810f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:59.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.753+0000 7f1e2cc00700 1 --2- 172.21.15.28:0/1783880417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e2806b120 0x7f1e2810f060 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1e10009990 tx=0x7f1e10009ca0 comp rx=0 tx=0).stop 2024-04-23T14:41:59.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.753+0000 7f1e2cc00700 1 -- 172.21.15.28:0/1783880417 shutdown_connections 2024-04-23T14:41:59.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.753+0000 7f1e2cc00700 1 --2- 172.21.15.28:0/1783880417 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1e2810f5a0 0x7f1e28111990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:59.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.753+0000 7f1e2cc00700 1 --2- 172.21.15.28:0/1783880417 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e2806a800 0x7f1e2806abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:59.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.753+0000 7f1e2cc00700 1 --2- 172.21.15.28:0/1783880417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e2806b120 0x7f1e2810f060 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:59.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.753+0000 7f1e2cc00700 1 -- 172.21.15.28:0/1783880417 >> 172.21.15.28:0/1783880417 conn(0x7f1e280789a0 msgr2=0x7f1e28078db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:41:59.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.753+0000 7f1e2cc00700 1 -- 172.21.15.28:0/1783880417 shutdown_connections 2024-04-23T14:41:59.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.754+0000 7f1e2cc00700 1 -- 172.21.15.28:0/1783880417 wait complete. 2024-04-23T14:41:59.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.755+0000 7f1e2cc00700 1 Processor -- start 2024-04-23T14:41:59.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.755+0000 7f1e2cc00700 1 -- start start 2024-04-23T14:41:59.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.755+0000 7f1e2cc00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1e2806a800 0x7f1e2819b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:59.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.755+0000 7f1e2cc00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e2806b120 0x7f1e2819bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:59.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.755+0000 7f1e2cc00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e2810f5a0 0x7f1e2819fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:59.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.755+0000 7f1e2cc00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1e28115030 con 0x7f1e2806b120 2024-04-23T14:41:59.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.755+0000 7f1e2cc00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1e28114eb0 con 0x7f1e2810f5a0 2024-04-23T14:41:59.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.755+0000 7f1e2cc00700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1e281151b0 con 0x7f1e2806a800 2024-04-23T14:41:59.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.756+0000 7f1e1fe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e2806b120 0x7f1e2819bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:59.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.756+0000 7f1e25e00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1e2806a800 0x7f1e2819b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:59.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.756+0000 7f1e1fe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e2806b120 0x7f1e2819bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52048/0 (socket says 172.21.15.28:52048) 2024-04-23T14:41:59.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.756+0000 7f1e1fe00700 1 -- 172.21.15.28:0/3101497528 learned_addr learned my addr 172.21.15.28:0/3101497528 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:41:59.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.756+0000 7f1e26800700 1 --2- 172.21.15.28:0/3101497528 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e2810f5a0 0x7f1e2819fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:59.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.756+0000 7f1e25e00700 1 -- 172.21.15.28:0/3101497528 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e2810f5a0 msgr2=0x7f1e2819fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:59.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.756+0000 7f1e25e00700 1 --2- 172.21.15.28:0/3101497528 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e2810f5a0 0x7f1e2819fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:59.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.756+0000 7f1e25e00700 1 -- 172.21.15.28:0/3101497528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e2806b120 msgr2=0x7f1e2819bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:41:59.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.756+0000 7f1e25e00700 1 --2- 172.21.15.28:0/3101497528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e2806b120 0x7f1e2819bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:41:59.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.756+0000 7f1e25e00700 1 -- 172.21.15.28:0/3101497528 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e281a05b0 con 0x7f1e2806a800 2024-04-23T14:41:59.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.756+0000 7f1e1fe00700 1 --2- 172.21.15.28:0/3101497528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e2806b120 0x7f1e2819bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:41:59.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.757+0000 7f1e25e00700 1 --2- 172.21.15.28:0/3101497528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1e2806a800 0x7f1e2819b5a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1e1800e9f0 tx=0x7f1e1800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:59.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.757+0000 7f1e1ea00700 1 -- 172.21.15.28:0/3101497528 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e1800cbd0 con 0x7f1e2806a800 2024-04-23T14:41:59.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.757+0000 7f1e2cc00700 1 -- 172.21.15.28:0/3101497528 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e281a08a0 con 0x7f1e2806a800 2024-04-23T14:41:59.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.757+0000 7f1e1ea00700 1 -- 172.21.15.28:0/3101497528 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e1800cd70 con 0x7f1e2806a800 2024-04-23T14:41:59.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.757+0000 7f1e1ea00700 1 -- 172.21.15.28:0/3101497528 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e18018710 con 0x7f1e2806a800 2024-04-23T14:41:59.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.757+0000 7f1e2cc00700 1 -- 172.21.15.28:0/3101497528 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e281a8120 con 0x7f1e2806a800 2024-04-23T14:41:59.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.758+0000 7f1e2cc00700 1 -- 172.21.15.28:0/3101497528 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e280697e0 con 0x7f1e2806a800 2024-04-23T14:41:59.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.760+0000 7f1e1ea00700 1 -- 172.21.15.28:0/3101497528 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1e18010430 con 0x7f1e2806a800 2024-04-23T14:41:59.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.761+0000 7f1e1ea00700 1 --2- 172.21.15.28:0/3101497528 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f1e0c0a3cd0 0x7f1e0c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:41:59.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.762+0000 7f1e1ea00700 1 -- 172.21.15.28:0/3101497528 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f1e180d06d0 con 0x7f1e2806a800 2024-04-23T14:41:59.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.762+0000 7f1e1fe00700 1 --2- 172.21.15.28:0/3101497528 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f1e0c0a3cd0 0x7f1e0c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:41:59.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.762+0000 7f1e1fe00700 1 --2- 172.21.15.28:0/3101497528 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f1e0c0a3cd0 0x7f1e0c0a6190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1e28102fb0 tx=0x7f1e10009590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:41:59.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:41:59.766+0000 7f1e1ea00700 1 -- 172.21.15.28:0/3101497528 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1e1809d000 con 0x7f1e2806a800 2024-04-23T14:41:59.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:41:59 smithi105 ceph-mon[32555]: pgmap v323: 97 pgs: 97 active+clean; 621 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 4.2 KiB/s wr, 4 op/s 2024-04-23T14:41:59.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:41:59 smithi038 ceph-mon[32348]: pgmap v323: 97 pgs: 97 active+clean; 621 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 4.2 KiB/s wr, 4 op/s 2024-04-23T14:42:00.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:00.002+0000 7f1e2cc00700 1 -- 172.21.15.28:0/3101497528 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1e2802dfe0 con 0x7f1e2806a800 2024-04-23T14:42:00.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:00.004+0000 7f1e1ea00700 1 -- 172.21.15.28:0/3101497528 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22170 (secure 0 0 0) 0x7f1e1809c900 con 0x7f1e2806a800 2024-04-23T14:42:00.006 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:42:00.006 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":85,"fsid":"1ee6e996-017e-11ef-bc93-c7b262605968","created":"2024-04-23T14:32:34.581082+0000","modified":"2024-04-23T14:41:41.010290+0000","last_up_change":"2024-04-23T14:40:43.375846+0000","last_in_change":"2024-04-23T14:40:23.576939+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:36:27.698112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:41:37.030341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:41:38.444742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e0b427bc-be65-4543-99cc-2f53ad98ddad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6803","nonce":2993445336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6805","nonce":2993445336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6809","nonce":2993445336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6807","nonce":2993445336}]},"public_addr":"172.21.15.28:6803/2993445336","cluster_addr":"172.21.15.28:6805/2993445336","heartbeat_back_addr":"172.21.15.28:6809/2993445336","heartbeat_front_addr":"172.21.15.28:6807/2993445336","state":["exists","up"]},{"osd":1,"uuid":"ef527ec9-1bf3-4bb0-8df3-f5a015af557c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6801","nonce":4024093383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6803","nonce":4024093383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6807","nonce":4024093383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6805","nonce":4024093383}]},"public_addr":"172.21.15.38:6801/4024093383","cluster_addr":"172.21.15.38:6803/4024093383","heartbeat_back_addr":"172.21.15.38:6807/4024093383","heartbeat_front_addr":"172.21.15.38:6805/4024093383","state":["exists","up"]},{"osd":2,"uuid":"b33b23d9-adce-4c11-b304-2e117bdf6bfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6801","nonce":570739029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6803","nonce":570739029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6807","nonce":570739029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6805","nonce":570739029}]},"public_addr":"172.21.15.105:6801/570739029","cluster_addr":"172.21.15.105:6803/570739029","heartbeat_back_addr":"172.21.15.105:6807/570739029","heartbeat_front_addr":"172.21.15.105:6805/570739029","state":["exists","up"]},{"osd":3,"uuid":"28bd5608-b426-434e-9db1-bdaa9828bdb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6811","nonce":2480400247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6813","nonce":2480400247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6817","nonce":2480400247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6815","nonce":2480400247}]},"public_addr":"172.21.15.28:6811/2480400247","cluster_addr":"172.21.15.28:6813/2480400247","heartbeat_back_addr":"172.21.15.28:6817/2480400247","heartbeat_front_addr":"172.21.15.28:6815/2480400247","state":["exists","up"]},{"osd":4,"uuid":"e5d1c89b-025f-4d57-8807-59d75c787b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6809","nonce":3772189246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6811","nonce":3772189246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6815","nonce":3772189246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6813","nonce":3772189246}]},"public_addr":"172.21.15.38:6809/3772189246","cluster_addr":"172.21.15.38:6811/3772189246","heartbeat_back_addr":"172.21.15.38:6815/3772189246","heartbeat_front_addr":"172.21.15.38:6813/3772189246","state":["exists","up"]},{"osd":5,"uuid":"9cf9cf4c-aafd-4ed3-bfb1-0527710a0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6809","nonce":928755894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6811","nonce":928755894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6815","nonce":928755894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6813","nonce":928755894}]},"public_addr":"172.21.15.105:6809/928755894","cluster_addr":"172.21.15.105:6811/928755894","heartbeat_back_addr":"172.21.15.105:6815/928755894","heartbeat_front_addr":"172.21.15.105:6813/928755894","state":["exists","up"]},{"osd":6,"uuid":"304db525-d24d-4687-bfa0-f92bdf66d1e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6819","nonce":897979441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6821","nonce":897979441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6825","nonce":897979441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6823","nonce":897979441}]},"public_addr":"172.21.15.28:6819/897979441","cluster_addr":"172.21.15.28:6821/897979441","heartbeat_back_addr":"172.21.15.28:6825/897979441","heartbeat_front_addr":"172.21.15.28:6823/897979441","state":["exists","up"]},{"osd":7,"uuid":"08e01d59-03f3-47b1-b45f-35ec5a50992c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6817","nonce":2400251786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6819","nonce":2400251786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6823","nonce":2400251786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6821","nonce":2400251786}]},"public_addr":"172.21.15.38:6817/2400251786","cluster_addr":"172.21.15.38:6819/2400251786","heartbeat_back_addr":"172.21.15.38:6823/2400251786","heartbeat_front_addr":"172.21.15.38:6821/2400251786","state":["exists","up"]},{"osd":8,"uuid":"64ee1e95-0756-48b1-9c00-472d0c3cbd29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6817","nonce":1682312269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6819","nonce":1682312269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6823","nonce":1682312269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6821","nonce":1682312269}]},"public_addr":"172.21.15.105:6817/1682312269","cluster_addr":"172.21.15.105:6819/1682312269","heartbeat_back_addr":"172.21.15.105:6823/1682312269","heartbeat_front_addr":"172.21.15.105:6821/1682312269","state":["exists","up"]},{"osd":9,"uuid":"18f60893-3ffd-48c2-9db5-a67d35611d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6827","nonce":1292942516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6829","nonce":1292942516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6833","nonce":1292942516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6831","nonce":1292942516}]},"public_addr":"172.21.15.28:6827/1292942516","cluster_addr":"172.21.15.28:6829/1292942516","heartbeat_back_addr":"172.21.15.28:6833/1292942516","heartbeat_front_addr":"172.21.15.28:6831/1292942516","state":["exists","up"]},{"osd":10,"uuid":"76c06725-7040-4118-8894-fe808a87f999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6825","nonce":2444616468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6827","nonce":2444616468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6831","nonce":2444616468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6829","nonce":2444616468}]},"public_addr":"172.21.15.38:6825/2444616468","cluster_addr":"172.21.15.38:6827/2444616468","heartbeat_back_addr":"172.21.15.38:6831/2444616468","heartbeat_front_addr":"172.21.15.38:6829/2444616468","state":["exists","up"]},{"osd":11,"uuid":"e8fc9454-5540-4e2e-91bf-3ba251dbd1a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6825","nonce":3400299562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6827","nonce":3400299562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6831","nonce":3400299562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6829","nonce":3400299562}]},"public_addr":"172.21.15.105:6825/3400299562","cluster_addr":"172.21.15.105:6827/3400299562","heartbeat_back_addr":"172.21.15.105:6831/3400299562","heartbeat_front_addr":"172.21.15.105:6829/3400299562","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:35:33.615395+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:00.545293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:24.940211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:57.684887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:24.519061+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:49.010804+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:22.638478+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:49.559104+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:14.011771+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:47.859950+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:16.298445+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:41.619069+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/590495740":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/1163172512":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/4290328460":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/4145020172":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/720626958":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/948450262":"2024-04-24T14:33:07.576363+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:42:00.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:00.007+0000 7f1e2cc00700 1 -- 172.21.15.28:0/3101497528 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f1e0c0a3cd0 msgr2=0x7f1e0c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:00.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:00.007+0000 7f1e2cc00700 1 --2- 172.21.15.28:0/3101497528 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f1e0c0a3cd0 0x7f1e0c0a6190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1e28102fb0 tx=0x7f1e10009590 comp rx=0 tx=0).stop 2024-04-23T14:42:00.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:00.007+0000 7f1e2cc00700 1 -- 172.21.15.28:0/3101497528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1e2806a800 msgr2=0x7f1e2819b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:00.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:00.007+0000 7f1e2cc00700 1 --2- 172.21.15.28:0/3101497528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1e2806a800 0x7f1e2819b5a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1e1800e9f0 tx=0x7f1e1800ed00 comp rx=0 tx=0).stop 2024-04-23T14:42:00.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:00.007+0000 7f1e2cc00700 1 -- 172.21.15.28:0/3101497528 shutdown_connections 2024-04-23T14:42:00.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:00.007+0000 7f1e2cc00700 1 --2- 172.21.15.28:0/3101497528 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f1e0c0a3cd0 0x7f1e0c0a6190 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:00.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:00.007+0000 7f1e2cc00700 1 --2- 172.21.15.28:0/3101497528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1e2806a800 0x7f1e2819b5a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:00.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:00.007+0000 7f1e2cc00700 1 --2- 172.21.15.28:0/3101497528 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e2810f5a0 0x7f1e2819fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:00.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:00.007+0000 7f1e2cc00700 1 --2- 172.21.15.28:0/3101497528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e2806b120 0x7f1e2819bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:00.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:00.007+0000 7f1e2cc00700 1 -- 172.21.15.28:0/3101497528 >> 172.21.15.28:0/3101497528 conn(0x7f1e280789a0 msgr2=0x7f1e28110af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:00.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:00.007+0000 7f1e2cc00700 1 -- 172.21.15.28:0/3101497528 shutdown_connections 2024-04-23T14:42:00.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:00.008+0000 7f1e2cc00700 1 -- 172.21.15.28:0/3101497528 wait complete. 2024-04-23T14:42:00.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:00 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3101497528' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:42:00.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:00 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3101497528' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:42:00.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:00 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3101497528' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:42:00.880 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-23T14:42:00.880 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-23T14:42:01.322 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:42:01.729 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:01 smithi028 ceph-mon[27810]: pgmap v324: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 5.6 KiB/s wr, 5 op/s 2024-04-23T14:42:01.774 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:01 smithi038 ceph-mon[32348]: pgmap v324: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 5.6 KiB/s wr, 5 op/s 2024-04-23T14:42:01.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:01 smithi105 ceph-mon[32555]: pgmap v324: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 5.6 KiB/s wr, 5 op/s 2024-04-23T14:42:03.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.376+0000 7f4ffbe00700 1 -- 172.21.15.28:0/1302485054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ffc104e60 msgr2=0x7f4ffc105240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:03.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.376+0000 7f4ffbe00700 1 --2- 172.21.15.28:0/1302485054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ffc104e60 0x7f4ffc105240 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4fec0099c0 tx=0x7f4fec009cd0 comp rx=0 tx=0).stop 2024-04-23T14:42:03.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.376+0000 7f4ffbe00700 1 -- 172.21.15.28:0/1302485054 shutdown_connections 2024-04-23T14:42:03.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.376+0000 7f4ffbe00700 1 --2- 172.21.15.28:0/1302485054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ffc109f40 0x7f4ffc111b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:03.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.376+0000 7f4ffbe00700 1 --2- 172.21.15.28:0/1302485054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ffc104e60 0x7f4ffc105240 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:03.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.376+0000 7f4ffbe00700 1 --2- 172.21.15.28:0/1302485054 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ffc105810 0x7f4ffc109860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:03.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.376+0000 7f4ffbe00700 1 -- 172.21.15.28:0/1302485054 >> 172.21.15.28:0/1302485054 conn(0x7f4ffc100730 msgr2=0x7f4ffc102b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:03.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.376+0000 7f4ffbe00700 1 -- 172.21.15.28:0/1302485054 shutdown_connections 2024-04-23T14:42:03.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.376+0000 7f4ffbe00700 1 -- 172.21.15.28:0/1302485054 wait complete. 2024-04-23T14:42:03.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.377+0000 7f4ffbe00700 1 Processor -- start 2024-04-23T14:42:03.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.378+0000 7f4ffbe00700 1 -- start start 2024-04-23T14:42:03.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.378+0000 7f4ffbe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ffc104e60 0x7f4ffc19b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:03.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.378+0000 7f4ffbe00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ffc105810 0x7f4ffc19b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:03.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.378+0000 7f4ffbe00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ffc109f40 0x7f4ffc19fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:03.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.378+0000 7f4ffbe00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4ffc114f40 con 0x7f4ffc104e60 2024-04-23T14:42:03.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.378+0000 7f4ffbe00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4ffc114dc0 con 0x7f4ffc105810 2024-04-23T14:42:03.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.378+0000 7f4ffbe00700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4ffc1150c0 con 0x7f4ffc109f40 2024-04-23T14:42:03.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.378+0000 7f4ffaa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ffc104e60 0x7f4ffc19b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:03.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.379+0000 7f4ff2000700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ffc105810 0x7f4ffc19b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:03.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.379+0000 7f4ff2000700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ffc105810 0x7f4ffc19b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:42878/0 (socket says 172.21.15.28:42878) 2024-04-23T14:42:03.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.379+0000 7f4ff2000700 1 -- 172.21.15.28:0/651569618 learned_addr learned my addr 172.21.15.28:0/651569618 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:42:03.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.379+0000 7f4ffb400700 1 --2- 172.21.15.28:0/651569618 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ffc109f40 0x7f4ffc19fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:03.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.379+0000 7f4ff2000700 1 -- 172.21.15.28:0/651569618 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ffc109f40 msgr2=0x7f4ffc19fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:03.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.379+0000 7f4ff2000700 1 --2- 172.21.15.28:0/651569618 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ffc109f40 0x7f4ffc19fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:03.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.379+0000 7f4ff2000700 1 -- 172.21.15.28:0/651569618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ffc104e60 msgr2=0x7f4ffc19b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:03.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.379+0000 7f4ff2000700 1 --2- 172.21.15.28:0/651569618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ffc104e60 0x7f4ffc19b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:03.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.379+0000 7f4ff2000700 1 -- 172.21.15.28:0/651569618 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ffc1a0400 con 0x7f4ffc105810 2024-04-23T14:42:03.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.379+0000 7f4ffaa00700 1 --2- 172.21.15.28:0/651569618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ffc104e60 0x7f4ffc19b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:42:03.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.380+0000 7f4ff2000700 1 --2- 172.21.15.28:0/651569618 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ffc105810 0x7f4ffc19b930 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4fe400ef90 tx=0x7f4fe400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:03.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.380+0000 7f5000a00700 1 -- 172.21.15.28:0/651569618 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe40092e0 con 0x7f4ffc105810 2024-04-23T14:42:03.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.380+0000 7f5000a00700 1 -- 172.21.15.28:0/651569618 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4fe4004bb0 con 0x7f4ffc105810 2024-04-23T14:42:03.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.380+0000 7f4ffbe00700 1 -- 172.21.15.28:0/651569618 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ffc1a0690 con 0x7f4ffc105810 2024-04-23T14:42:03.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.380+0000 7f5000a00700 1 -- 172.21.15.28:0/651569618 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe4004280 con 0x7f4ffc105810 2024-04-23T14:42:03.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.380+0000 7f4ffbe00700 1 -- 172.21.15.28:0/651569618 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ffc1a7f70 con 0x7f4ffc105810 2024-04-23T14:42:03.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.381+0000 7f4ffbe00700 1 -- 172.21.15.28:0/651569618 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ffc0697e0 con 0x7f4ffc105810 2024-04-23T14:42:03.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.383+0000 7f5000a00700 1 -- 172.21.15.28:0/651569618 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4fe401b030 con 0x7f4ffc105810 2024-04-23T14:42:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.384+0000 7f5000a00700 1 --2- 172.21.15.28:0/651569618 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f4fe00a3d50 0x7f4fe00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.385+0000 7f5000a00700 1 -- 172.21.15.28:0/651569618 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f4fe40d03c0 con 0x7f4ffc105810 2024-04-23T14:42:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.385+0000 7f4ffaa00700 1 --2- 172.21.15.28:0/651569618 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f4fe00a3d50 0x7f4fe00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:03.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.386+0000 7f4ffaa00700 1 --2- 172.21.15.28:0/651569618 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f4fe00a3d50 0x7f4fe00a6210 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4fec000c00 tx=0x7f4fec005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:03.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.387+0000 7f5000a00700 1 -- 172.21.15.28:0/651569618 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4fe4022020 con 0x7f4ffc105810 2024-04-23T14:42:03.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.622+0000 7f4ffbe00700 1 -- 172.21.15.28:0/651569618 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4ffc02dfe0 con 0x7f4ffc105810 2024-04-23T14:42:03.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.624+0000 7f5000a00700 1 -- 172.21.15.28:0/651569618 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22170 (secure 0 0 0) 0x7f4fe4010040 con 0x7f4ffc105810 2024-04-23T14:42:03.626 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:42:03.626 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":85,"fsid":"1ee6e996-017e-11ef-bc93-c7b262605968","created":"2024-04-23T14:32:34.581082+0000","modified":"2024-04-23T14:41:41.010290+0000","last_up_change":"2024-04-23T14:40:43.375846+0000","last_in_change":"2024-04-23T14:40:23.576939+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:36:27.698112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:41:37.030341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:41:38.444742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e0b427bc-be65-4543-99cc-2f53ad98ddad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6803","nonce":2993445336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6805","nonce":2993445336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6809","nonce":2993445336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6807","nonce":2993445336}]},"public_addr":"172.21.15.28:6803/2993445336","cluster_addr":"172.21.15.28:6805/2993445336","heartbeat_back_addr":"172.21.15.28:6809/2993445336","heartbeat_front_addr":"172.21.15.28:6807/2993445336","state":["exists","up"]},{"osd":1,"uuid":"ef527ec9-1bf3-4bb0-8df3-f5a015af557c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6801","nonce":4024093383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6803","nonce":4024093383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6807","nonce":4024093383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6805","nonce":4024093383}]},"public_addr":"172.21.15.38:6801/4024093383","cluster_addr":"172.21.15.38:6803/4024093383","heartbeat_back_addr":"172.21.15.38:6807/4024093383","heartbeat_front_addr":"172.21.15.38:6805/4024093383","state":["exists","up"]},{"osd":2,"uuid":"b33b23d9-adce-4c11-b304-2e117bdf6bfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6801","nonce":570739029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6803","nonce":570739029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6807","nonce":570739029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6805","nonce":570739029}]},"public_addr":"172.21.15.105:6801/570739029","cluster_addr":"172.21.15.105:6803/570739029","heartbeat_back_addr":"172.21.15.105:6807/570739029","heartbeat_front_addr":"172.21.15.105:6805/570739029","state":["exists","up"]},{"osd":3,"uuid":"28bd5608-b426-434e-9db1-bdaa9828bdb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6811","nonce":2480400247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6813","nonce":2480400247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6817","nonce":2480400247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6815","nonce":2480400247}]},"public_addr":"172.21.15.28:6811/2480400247","cluster_addr":"172.21.15.28:6813/2480400247","heartbeat_back_addr":"172.21.15.28:6817/2480400247","heartbeat_front_addr":"172.21.15.28:6815/2480400247","state":["exists","up"]},{"osd":4,"uuid":"e5d1c89b-025f-4d57-8807-59d75c787b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6809","nonce":3772189246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6811","nonce":3772189246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6815","nonce":3772189246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6813","nonce":3772189246}]},"public_addr":"172.21.15.38:6809/3772189246","cluster_addr":"172.21.15.38:6811/3772189246","heartbeat_back_addr":"172.21.15.38:6815/3772189246","heartbeat_front_addr":"172.21.15.38:6813/3772189246","state":["exists","up"]},{"osd":5,"uuid":"9cf9cf4c-aafd-4ed3-bfb1-0527710a0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6809","nonce":928755894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6811","nonce":928755894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6815","nonce":928755894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6813","nonce":928755894}]},"public_addr":"172.21.15.105:6809/928755894","cluster_addr":"172.21.15.105:6811/928755894","heartbeat_back_addr":"172.21.15.105:6815/928755894","heartbeat_front_addr":"172.21.15.105:6813/928755894","state":["exists","up"]},{"osd":6,"uuid":"304db525-d24d-4687-bfa0-f92bdf66d1e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6819","nonce":897979441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6821","nonce":897979441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6825","nonce":897979441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6823","nonce":897979441}]},"public_addr":"172.21.15.28:6819/897979441","cluster_addr":"172.21.15.28:6821/897979441","heartbeat_back_addr":"172.21.15.28:6825/897979441","heartbeat_front_addr":"172.21.15.28:6823/897979441","state":["exists","up"]},{"osd":7,"uuid":"08e01d59-03f3-47b1-b45f-35ec5a50992c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6817","nonce":2400251786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6819","nonce":2400251786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6823","nonce":2400251786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6821","nonce":2400251786}]},"public_addr":"172.21.15.38:6817/2400251786","cluster_addr":"172.21.15.38:6819/2400251786","heartbeat_back_addr":"172.21.15.38:6823/2400251786","heartbeat_front_addr":"172.21.15.38:6821/2400251786","state":["exists","up"]},{"osd":8,"uuid":"64ee1e95-0756-48b1-9c00-472d0c3cbd29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6817","nonce":1682312269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6819","nonce":1682312269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6823","nonce":1682312269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6821","nonce":1682312269}]},"public_addr":"172.21.15.105:6817/1682312269","cluster_addr":"172.21.15.105:6819/1682312269","heartbeat_back_addr":"172.21.15.105:6823/1682312269","heartbeat_front_addr":"172.21.15.105:6821/1682312269","state":["exists","up"]},{"osd":9,"uuid":"18f60893-3ffd-48c2-9db5-a67d35611d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6827","nonce":1292942516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6829","nonce":1292942516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6833","nonce":1292942516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6831","nonce":1292942516}]},"public_addr":"172.21.15.28:6827/1292942516","cluster_addr":"172.21.15.28:6829/1292942516","heartbeat_back_addr":"172.21.15.28:6833/1292942516","heartbeat_front_addr":"172.21.15.28:6831/1292942516","state":["exists","up"]},{"osd":10,"uuid":"76c06725-7040-4118-8894-fe808a87f999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6825","nonce":2444616468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6827","nonce":2444616468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6831","nonce":2444616468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6829","nonce":2444616468}]},"public_addr":"172.21.15.38:6825/2444616468","cluster_addr":"172.21.15.38:6827/2444616468","heartbeat_back_addr":"172.21.15.38:6831/2444616468","heartbeat_front_addr":"172.21.15.38:6829/2444616468","state":["exists","up"]},{"osd":11,"uuid":"e8fc9454-5540-4e2e-91bf-3ba251dbd1a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6825","nonce":3400299562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6827","nonce":3400299562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6831","nonce":3400299562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6829","nonce":3400299562}]},"public_addr":"172.21.15.105:6825/3400299562","cluster_addr":"172.21.15.105:6827/3400299562","heartbeat_back_addr":"172.21.15.105:6831/3400299562","heartbeat_front_addr":"172.21.15.105:6829/3400299562","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:35:33.615395+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:00.545293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:24.940211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:57.684887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:24.519061+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:49.010804+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:22.638478+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:49.559104+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:14.011771+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:47.859950+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:16.298445+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:41.619069+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/590495740":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/1163172512":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/4290328460":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/4145020172":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/720626958":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/948450262":"2024-04-24T14:33:07.576363+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:42:03.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.627+0000 7f4ffbe00700 1 -- 172.21.15.28:0/651569618 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f4fe00a3d50 msgr2=0x7f4fe00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:03.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.627+0000 7f4ffbe00700 1 --2- 172.21.15.28:0/651569618 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f4fe00a3d50 0x7f4fe00a6210 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4fec000c00 tx=0x7f4fec005fb0 comp rx=0 tx=0).stop 2024-04-23T14:42:03.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.627+0000 7f4ffbe00700 1 -- 172.21.15.28:0/651569618 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ffc105810 msgr2=0x7f4ffc19b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:03.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.627+0000 7f4ffbe00700 1 --2- 172.21.15.28:0/651569618 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ffc105810 0x7f4ffc19b930 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4fe400ef90 tx=0x7f4fe400c3a0 comp rx=0 tx=0).stop 2024-04-23T14:42:03.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.627+0000 7f4ffbe00700 1 -- 172.21.15.28:0/651569618 shutdown_connections 2024-04-23T14:42:03.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.627+0000 7f4ffbe00700 1 --2- 172.21.15.28:0/651569618 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f4fe00a3d50 0x7f4fe00a6210 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:03.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.627+0000 7f4ffbe00700 1 --2- 172.21.15.28:0/651569618 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ffc109f40 0x7f4ffc19fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:03.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.627+0000 7f4ffbe00700 1 --2- 172.21.15.28:0/651569618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ffc104e60 0x7f4ffc19b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:03.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.627+0000 7f4ffbe00700 1 --2- 172.21.15.28:0/651569618 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ffc105810 0x7f4ffc19b930 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:03.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.627+0000 7f4ffbe00700 1 -- 172.21.15.28:0/651569618 >> 172.21.15.28:0/651569618 conn(0x7f4ffc100730 msgr2=0x7f4ffc102a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:03.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.627+0000 7f4ffbe00700 1 -- 172.21.15.28:0/651569618 shutdown_connections 2024-04-23T14:42:03.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:03.627+0000 7f4ffbe00700 1 -- 172.21.15.28:0/651569618 wait complete. 2024-04-23T14:42:03.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:03 smithi105 ceph-mon[32555]: pgmap v325: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-04-23T14:42:03.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:03 smithi038 ceph-mon[32348]: pgmap v325: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-04-23T14:42:03.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:03 smithi028 ceph-mon[27810]: pgmap v325: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-04-23T14:42:04.529 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-23T14:42:04.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-23T14:42:04.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-23T14:42:04.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-23T14:42:04.531 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-23T14:42:04.531 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-23T14:42:04.532 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-23T14:42:04.532 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-23T14:42:04.532 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-23T14:42:04.533 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-23T14:42:04.533 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-23T14:42:04.534 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-23T14:42:04.708 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:04 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/651569618' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:42:04.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:04 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/651569618' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:42:04.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:04 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/651569618' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:42:05.542 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:42:05.562 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:42:05.564 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:42:05.570 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:42:05.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:05 smithi028 ceph-mon[27810]: pgmap v326: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-04-23T14:42:05.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:05 smithi105 ceph-mon[32555]: pgmap v326: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-04-23T14:42:05.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:05 smithi038 ceph-mon[32348]: pgmap v326: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-04-23T14:42:06.780 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:42:06.786 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:42:06.787 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:42:06.790 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:42:07.115 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:42:07.116 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:42:07.734 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:07 smithi028 ceph-mon[27810]: pgmap v327: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-04-23T14:42:07.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:07 smithi105 ceph-mon[32555]: pgmap v327: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-04-23T14:42:07.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:07 smithi038 ceph-mon[32348]: pgmap v327: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-04-23T14:42:09.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:09 smithi028 ceph-mon[27810]: pgmap v328: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-23T14:42:09.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.454+0000 7f93a4600700 1 -- 172.21.15.28:0/3072046166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f939c105920 msgr2=0x7f939c109970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:09.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.454+0000 7f93a4600700 1 --2- 172.21.15.28:0/3072046166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f939c105920 0x7f939c109970 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f93880099c0 tx=0x7f9388009cd0 comp rx=0 tx=0).stop 2024-04-23T14:42:09.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.454+0000 7f93a4600700 1 -- 172.21.15.28:0/3072046166 shutdown_connections 2024-04-23T14:42:09.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.454+0000 7f93a4600700 1 --2- 172.21.15.28:0/3072046166 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f939c10a050 0x7f939c111c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:09.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.454+0000 7f93a4600700 1 --2- 172.21.15.28:0/3072046166 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f939c104f70 0x7f939c105350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:09.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.454+0000 7f93a4600700 1 --2- 172.21.15.28:0/3072046166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f939c105920 0x7f939c109970 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:09.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.454+0000 7f93a4600700 1 -- 172.21.15.28:0/3072046166 >> 172.21.15.28:0/3072046166 conn(0x7f939c100800 msgr2=0x7f939c102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:09.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.454+0000 7f93a4600700 1 -- 172.21.15.28:0/3072046166 shutdown_connections 2024-04-23T14:42:09.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.454+0000 7f93a4600700 1 -- 172.21.15.28:0/3072046166 wait complete. 2024-04-23T14:42:09.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.455+0000 7f93a4600700 1 Processor -- start 2024-04-23T14:42:09.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.455+0000 7f93a4600700 1 -- start start 2024-04-23T14:42:09.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.456+0000 7f93a4600700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f939c104f70 0x7f939c19b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:09.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.456+0000 7f93a4600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f939c105920 0x7f939c19b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:09.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.456+0000 7f93a4600700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f939c10a050 0x7f939c19fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:09.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.456+0000 7f93a4600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f939c115060 con 0x7f939c105920 2024-04-23T14:42:09.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.456+0000 7f93a4600700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f939c114ee0 con 0x7f939c10a050 2024-04-23T14:42:09.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.456+0000 7f93a4600700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f939c1151e0 con 0x7f939c104f70 2024-04-23T14:42:09.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.456+0000 7f93a1200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f939c105920 0x7f939c19b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:09.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.456+0000 7f93a1200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f939c105920 0x7f939c19b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38560/0 (socket says 172.21.15.28:38560) 2024-04-23T14:42:09.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.456+0000 7f93a1200700 1 -- 172.21.15.28:0/3703049294 learned_addr learned my addr 172.21.15.28:0/3703049294 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:42:09.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.456+0000 7f93a1c00700 1 --2- 172.21.15.28:0/3703049294 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f939c104f70 0x7f939c19b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:09.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.456+0000 7f93a2600700 1 --2- 172.21.15.28:0/3703049294 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f939c10a050 0x7f939c19fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:09.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.456+0000 7f93a1200700 1 -- 172.21.15.28:0/3703049294 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f939c104f70 msgr2=0x7f939c19b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:09.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.456+0000 7f93a1200700 1 --2- 172.21.15.28:0/3703049294 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f939c104f70 0x7f939c19b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:09.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.457+0000 7f93a1200700 1 -- 172.21.15.28:0/3703049294 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f939c10a050 msgr2=0x7f939c19fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:09.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.457+0000 7f93a1200700 1 --2- 172.21.15.28:0/3703049294 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f939c10a050 0x7f939c19fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:09.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.457+0000 7f93a1200700 1 -- 172.21.15.28:0/3703049294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f939c1a04a0 con 0x7f939c105920 2024-04-23T14:42:09.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.457+0000 7f93a1200700 1 --2- 172.21.15.28:0/3703049294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f939c105920 0x7f939c19b9d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f938800f690 tx=0x7f9388005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:09.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.457+0000 7f939a000700 1 -- 172.21.15.28:0/3703049294 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9388027070 con 0x7f939c105920 2024-04-23T14:42:09.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.457+0000 7f939a000700 1 -- 172.21.15.28:0/3703049294 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9388003680 con 0x7f939c105920 2024-04-23T14:42:09.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.457+0000 7f93a4600700 1 -- 172.21.15.28:0/3703049294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f939c1a0730 con 0x7f939c105920 2024-04-23T14:42:09.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.457+0000 7f939a000700 1 -- 172.21.15.28:0/3703049294 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93880173f0 con 0x7f939c105920 2024-04-23T14:42:09.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.457+0000 7f93a4600700 1 -- 172.21.15.28:0/3703049294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f939c1a8010 con 0x7f939c105920 2024-04-23T14:42:09.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.458+0000 7f93a4600700 1 -- 172.21.15.28:0/3703049294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f939c063280 con 0x7f939c105920 2024-04-23T14:42:09.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.461+0000 7f939a000700 1 -- 172.21.15.28:0/3703049294 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f938802a070 con 0x7f939c105920 2024-04-23T14:42:09.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.462+0000 7f939a000700 1 --2- 172.21.15.28:0/3703049294 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f93840a3d50 0x7f93840a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:09.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.462+0000 7f939a000700 1 -- 172.21.15.28:0/3703049294 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f93880d1920 con 0x7f939c105920 2024-04-23T14:42:09.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.462+0000 7f93a1c00700 1 --2- 172.21.15.28:0/3703049294 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f93840a3d50 0x7f93840a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:09.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.463+0000 7f939a000700 1 --2- 172.21.15.28:0/3703049294 >> [v2:172.21.15.105:6816/1682312269,v1:172.21.15.105:6817/1682312269] conn(0x7f93840ad530 0x7f93840af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:09.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.463+0000 7f939a000700 1 -- 172.21.15.28:0/3703049294 --> [v2:172.21.15.105:6816/1682312269,v1:172.21.15.105:6817/1682312269] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f93840b0000 con 0x7f93840ad530 2024-04-23T14:42:09.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.463+0000 7f939a000700 1 -- 172.21.15.28:0/3703049294 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f93880d1cd0 con 0x7f939c105920 2024-04-23T14:42:09.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.463+0000 7f93a2600700 1 --2- 172.21.15.28:0/3703049294 >> [v2:172.21.15.105:6816/1682312269,v1:172.21.15.105:6817/1682312269] conn(0x7f93840ad530 0x7f93840af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:09.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.463+0000 7f93a1c00700 1 --2- 172.21.15.28:0/3703049294 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f93840a3d50 0x7f93840a6210 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9390009c60 tx=0x7f9390009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:09.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.463+0000 7f93a2600700 1 --2- 172.21.15.28:0/3703049294 >> [v2:172.21.15.105:6816/1682312269,v1:172.21.15.105:6817/1682312269] conn(0x7f93840ad530 0x7f93840af950 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:09.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.465+0000 7f939a000700 1 -- 172.21.15.28:0/3703049294 <== osd.8 v2:172.21.15.105:6816/1682312269 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f93840b0000 con 0x7f93840ad530 2024-04-23T14:42:09.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.489+0000 7f93a4600700 1 -- 172.21.15.28:0/3703049294 --> [v2:172.21.15.105:6816/1682312269,v1:172.21.15.105:6817/1682312269] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f939c063680 con 0x7f93840ad530 2024-04-23T14:42:09.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.490+0000 7f939a000700 1 -- 172.21.15.28:0/3703049294 <== osd.8 v2:172.21.15.105:6816/1682312269 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f939c063680 con 0x7f93840ad530 2024-04-23T14:42:09.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.490+0000 7f93a4600700 1 -- 172.21.15.28:0/3703049294 >> [v2:172.21.15.105:6816/1682312269,v1:172.21.15.105:6817/1682312269] conn(0x7f93840ad530 msgr2=0x7f93840af950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:09.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.490+0000 7f93a4600700 1 --2- 172.21.15.28:0/3703049294 >> [v2:172.21.15.105:6816/1682312269,v1:172.21.15.105:6817/1682312269] conn(0x7f93840ad530 0x7f93840af950 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:09.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.490+0000 7f93a4600700 1 -- 172.21.15.28:0/3703049294 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f93840a3d50 msgr2=0x7f93840a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:09.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.490+0000 7f93a4600700 1 --2- 172.21.15.28:0/3703049294 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f93840a3d50 0x7f93840a6210 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9390009c60 tx=0x7f9390009260 comp rx=0 tx=0).stop 2024-04-23T14:42:09.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.490+0000 7f93a4600700 1 -- 172.21.15.28:0/3703049294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f939c105920 msgr2=0x7f939c19b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:09.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.490+0000 7f93a4600700 1 --2- 172.21.15.28:0/3703049294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f939c105920 0x7f939c19b9d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f938800f690 tx=0x7f9388005f10 comp rx=0 tx=0).stop 2024-04-23T14:42:09.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.490+0000 7f93a2600700 1 -- 172.21.15.28:0/3703049294 reap_dead start 2024-04-23T14:42:09.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.491+0000 7f93a4600700 1 -- 172.21.15.28:0/3703049294 shutdown_connections 2024-04-23T14:42:09.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.491+0000 7f93a4600700 1 -- 172.21.15.28:0/3703049294 >> 172.21.15.28:0/3703049294 conn(0x7f939c100800 msgr2=0x7f939c101320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:09.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.491+0000 7f93a4600700 1 -- 172.21.15.28:0/3703049294 shutdown_connections 2024-04-23T14:42:09.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:09.491+0000 7f93a4600700 1 -- 172.21.15.28:0/3703049294 wait complete. 2024-04-23T14:42:09.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:09 smithi105 ceph-mon[32555]: pgmap v328: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-23T14:42:09.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:09 smithi038 ceph-mon[32348]: pgmap v328: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-23T14:42:11.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:11 smithi028 ceph-mon[27810]: pgmap v329: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-04-23T14:42:11.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:11 smithi105 ceph-mon[32555]: pgmap v329: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-04-23T14:42:11.870 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:42:11.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:11 smithi038 ceph-mon[32348]: pgmap v329: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-04-23T14:42:13.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:13 smithi028 ceph-mon[27810]: pgmap v330: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-04-23T14:42:13.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:13 smithi105 ceph-mon[32555]: pgmap v330: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-04-23T14:42:13.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:13 smithi038 ceph-mon[32348]: pgmap v330: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-04-23T14:42:14.706 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:42:14.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.860+0000 7f021ea00700 1 -- 172.21.15.28:0/96967881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f021806ba20 msgr2=0x7f021810f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:14.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.860+0000 7f021ea00700 1 --2- 172.21.15.28:0/96967881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f021806ba20 0x7f021810f8f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f021000b0a0 tx=0x7f021000b3b0 comp rx=0 tx=0).stop 2024-04-23T14:42:14.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.860+0000 7f021ea00700 1 -- 172.21.15.28:0/96967881 shutdown_connections 2024-04-23T14:42:14.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.860+0000 7f021ea00700 1 --2- 172.21.15.28:0/96967881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f021806ba20 0x7f021810f8f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:14.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.860+0000 7f021ea00700 1 --2- 172.21.15.28:0/96967881 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0218108ad0 0x7f0218108eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:14.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.860+0000 7f021ea00700 1 --2- 172.21.15.28:0/96967881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f021806b060 0x7f021806b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:14.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.860+0000 7f021ea00700 1 -- 172.21.15.28:0/96967881 >> 172.21.15.28:0/96967881 conn(0x7f0218078830 msgr2=0x7f0218078c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:14.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.860+0000 7f021ea00700 1 -- 172.21.15.28:0/96967881 shutdown_connections 2024-04-23T14:42:14.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.860+0000 7f021ea00700 1 -- 172.21.15.28:0/96967881 wait complete. 2024-04-23T14:42:14.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.861+0000 7f021ea00700 1 Processor -- start 2024-04-23T14:42:14.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.862+0000 7f021ea00700 1 -- start start 2024-04-23T14:42:14.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.862+0000 7f021ea00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f021806b060 0x7f02181990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:14.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.862+0000 7f021ea00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f021806ba20 0x7f0218199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:14.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.862+0000 7f021ea00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0218108ad0 0x7f021819d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:14.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.862+0000 7f021ea00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0218112da0 con 0x7f021806b060 2024-04-23T14:42:14.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.862+0000 7f021ea00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0218112c20 con 0x7f0218108ad0 2024-04-23T14:42:14.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.862+0000 7f021ea00700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0218112f20 con 0x7f021806ba20 2024-04-23T14:42:14.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.862+0000 7f021d600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f021806b060 0x7f02181990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:14.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.862+0000 7f021d600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f021806b060 0x7f02181990f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38592/0 (socket says 172.21.15.28:38592) 2024-04-23T14:42:14.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.862+0000 7f021d600700 1 -- 172.21.15.28:0/3026924212 learned_addr learned my addr 172.21.15.28:0/3026924212 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:42:14.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.862+0000 7f021cc00700 1 --2- 172.21.15.28:0/3026924212 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f021806ba20 0x7f0218199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:14.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.863+0000 7f021e000700 1 --2- 172.21.15.28:0/3026924212 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0218108ad0 0x7f021819d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.863+0000 7f021d600700 1 -- 172.21.15.28:0/3026924212 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f021806ba20 msgr2=0x7f0218199630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.863+0000 7f021d600700 1 --2- 172.21.15.28:0/3026924212 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f021806ba20 0x7f0218199630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.863+0000 7f021d600700 1 -- 172.21.15.28:0/3026924212 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0218108ad0 msgr2=0x7f021819d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.863+0000 7f021d600700 1 --2- 172.21.15.28:0/3026924212 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0218108ad0 0x7f021819d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.863+0000 7f021d600700 1 -- 172.21.15.28:0/3026924212 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f021819e100 con 0x7f021806b060 2024-04-23T14:42:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.863+0000 7f021cc00700 1 --2- 172.21.15.28:0/3026924212 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f021806ba20 0x7f0218199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:42:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.863+0000 7f021d600700 1 --2- 172.21.15.28:0/3026924212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f021806b060 0x7f02181990f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f020c00eb30 tx=0x7f020c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.863+0000 7f0216000700 1 -- 172.21.15.28:0/3026924212 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f020c00cbd0 con 0x7f021806b060 2024-04-23T14:42:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.864+0000 7f021ea00700 1 -- 172.21.15.28:0/3026924212 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f021819e3f0 con 0x7f021806b060 2024-04-23T14:42:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.864+0000 7f0216000700 1 -- 172.21.15.28:0/3026924212 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f020c004500 con 0x7f021806b060 2024-04-23T14:42:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.864+0000 7f0216000700 1 -- 172.21.15.28:0/3026924212 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f020c010430 con 0x7f021806b060 2024-04-23T14:42:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.864+0000 7f021ea00700 1 -- 172.21.15.28:0/3026924212 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02181a5c70 con 0x7f021806b060 2024-04-23T14:42:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.865+0000 7f021ea00700 1 -- 172.21.15.28:0/3026924212 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0218100540 con 0x7f021806b060 2024-04-23T14:42:14.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.868+0000 7f0216000700 1 -- 172.21.15.28:0/3026924212 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f020c003f00 con 0x7f021806b060 2024-04-23T14:42:14.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.869+0000 7f0216000700 1 --2- 172.21.15.28:0/3026924212 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f02000a3cd0 0x7f02000a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:14.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.869+0000 7f0216000700 1 -- 172.21.15.28:0/3026924212 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f020c0d0a20 con 0x7f021806b060 2024-04-23T14:42:14.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.869+0000 7f021cc00700 1 --2- 172.21.15.28:0/3026924212 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f02000a3cd0 0x7f02000a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:14.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.869+0000 7f0216000700 1 --2- 172.21.15.28:0/3026924212 >> [v2:172.21.15.38:6800/4024093383,v1:172.21.15.38:6801/4024093383] conn(0x7f02000ad4b0 0x7f02000af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:14.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.869+0000 7f0216000700 1 -- 172.21.15.28:0/3026924212 --> [v2:172.21.15.38:6800/4024093383,v1:172.21.15.38:6801/4024093383] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f020c023e50 con 0x7f02000ad4b0 2024-04-23T14:42:14.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.869+0000 7f0216000700 1 -- 172.21.15.28:0/3026924212 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f020c0d0dd0 con 0x7f021806b060 2024-04-23T14:42:14.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.869+0000 7f021e000700 1 --2- 172.21.15.28:0/3026924212 >> [v2:172.21.15.38:6800/4024093383,v1:172.21.15.38:6801/4024093383] conn(0x7f02000ad4b0 0x7f02000af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:14.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.869+0000 7f021cc00700 1 --2- 172.21.15.28:0/3026924212 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f02000a3cd0 0x7f02000a6190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0218078ce0 tx=0x7f02040073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:14.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.870+0000 7f021e000700 1 --2- 172.21.15.28:0/3026924212 >> [v2:172.21.15.38:6800/4024093383,v1:172.21.15.38:6801/4024093383] conn(0x7f02000ad4b0 0x7f02000af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:14.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.871+0000 7f0216000700 1 -- 172.21.15.28:0/3026924212 <== osd.1 v2:172.21.15.38:6800/4024093383 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f020c023e50 con 0x7f02000ad4b0 2024-04-23T14:42:14.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.889+0000 7f021ea00700 1 -- 172.21.15.28:0/3026924212 --> [v2:172.21.15.38:6800/4024093383,v1:172.21.15.38:6801/4024093383] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0218104160 con 0x7f02000ad4b0 2024-04-23T14:42:14.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.890+0000 7f0216000700 1 -- 172.21.15.28:0/3026924212 <== osd.1 v2:172.21.15.38:6800/4024093383 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0218104160 con 0x7f02000ad4b0 2024-04-23T14:42:14.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.891+0000 7f021ea00700 1 -- 172.21.15.28:0/3026924212 >> [v2:172.21.15.38:6800/4024093383,v1:172.21.15.38:6801/4024093383] conn(0x7f02000ad4b0 msgr2=0x7f02000af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:14.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.891+0000 7f021ea00700 1 --2- 172.21.15.28:0/3026924212 >> [v2:172.21.15.38:6800/4024093383,v1:172.21.15.38:6801/4024093383] conn(0x7f02000ad4b0 0x7f02000af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.891+0000 7f021ea00700 1 -- 172.21.15.28:0/3026924212 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f02000a3cd0 msgr2=0x7f02000a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.891+0000 7f021ea00700 1 --2- 172.21.15.28:0/3026924212 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f02000a3cd0 0x7f02000a6190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0218078ce0 tx=0x7f02040073d0 comp rx=0 tx=0).stop 2024-04-23T14:42:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.891+0000 7f021ea00700 1 -- 172.21.15.28:0/3026924212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f021806b060 msgr2=0x7f02181990f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.891+0000 7f021ea00700 1 --2- 172.21.15.28:0/3026924212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f021806b060 0x7f02181990f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f020c00eb30 tx=0x7f020c00ee40 comp rx=0 tx=0).stop 2024-04-23T14:42:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.891+0000 7f021e000700 1 -- 172.21.15.28:0/3026924212 reap_dead start 2024-04-23T14:42:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.891+0000 7f021ea00700 1 -- 172.21.15.28:0/3026924212 shutdown_connections 2024-04-23T14:42:14.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.891+0000 7f021ea00700 1 -- 172.21.15.28:0/3026924212 >> 172.21.15.28:0/3026924212 conn(0x7f0218078830 msgr2=0x7f02180ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:14.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.891+0000 7f021ea00700 1 -- 172.21.15.28:0/3026924212 shutdown_connections 2024-04-23T14:42:14.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:14.891+0000 7f021ea00700 1 -- 172.21.15.28:0/3026924212 wait complete. 2024-04-23T14:42:15.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.656+0000 7fa3cf800700 1 -- 172.21.15.28:0/2342590592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3c8108c50 msgr2=0x7fa3c8109030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:15.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.656+0000 7fa3cf800700 1 --2- 172.21.15.28:0/2342590592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3c8108c50 0x7fa3c8109030 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc009990 tx=0x7fa3bc009ca0 comp rx=0 tx=0).stop 2024-04-23T14:42:15.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.656+0000 7fa3cf800700 1 -- 172.21.15.28:0/2342590592 shutdown_connections 2024-04-23T14:42:15.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.656+0000 7fa3cf800700 1 --2- 172.21.15.28:0/2342590592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa3c8103500 0x7fa3c810f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:15.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.656+0000 7fa3cf800700 1 --2- 172.21.15.28:0/2342590592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3c8108c50 0x7fa3c8109030 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:15.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.656+0000 7fa3cf800700 1 --2- 172.21.15.28:0/2342590592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3c8102b40 0x7fa3c8102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:15.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.656+0000 7fa3cf800700 1 -- 172.21.15.28:0/2342590592 >> 172.21.15.28:0/2342590592 conn(0x7fa3c8079ca0 msgr2=0x7fa3c807a0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:15.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.657+0000 7fa3cf800700 1 -- 172.21.15.28:0/2342590592 shutdown_connections 2024-04-23T14:42:15.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.657+0000 7fa3cf800700 1 -- 172.21.15.28:0/2342590592 wait complete. 2024-04-23T14:42:15.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.658+0000 7fa3cf800700 1 Processor -- start 2024-04-23T14:42:15.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.658+0000 7fa3cf800700 1 -- start start 2024-04-23T14:42:15.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.658+0000 7fa3cf800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3c8102b40 0x7fa3c819b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:15.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.658+0000 7fa3cf800700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa3c8103500 0x7fa3c819ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:15.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.658+0000 7fa3cf800700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3c8108c50 0x7fa3c819fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:15.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.658+0000 7fa3cf800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c8112e10 con 0x7fa3c8102b40 2024-04-23T14:42:15.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.658+0000 7fa3cf800700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c8112c90 con 0x7fa3c8108c50 2024-04-23T14:42:15.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.658+0000 7fa3cf800700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c8112f90 con 0x7fa3c8103500 2024-04-23T14:42:15.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.659+0000 7fa3cce00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3c8102b40 0x7fa3c819b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:15.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.659+0000 7fa3cce00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3c8102b40 0x7fa3c819b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38612/0 (socket says 172.21.15.28:38612) 2024-04-23T14:42:15.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.659+0000 7fa3cce00700 1 -- 172.21.15.28:0/1350416583 learned_addr learned my addr 172.21.15.28:0/1350416583 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:42:15.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.659+0000 7fa3c7e00700 1 --2- 172.21.15.28:0/1350416583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa3c8103500 0x7fa3c819ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:15.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.659+0000 7fa3cd800700 1 --2- 172.21.15.28:0/1350416583 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3c8108c50 0x7fa3c819fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:15.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.659+0000 7fa3cce00700 1 -- 172.21.15.28:0/1350416583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa3c8103500 msgr2=0x7fa3c819ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:15.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.659+0000 7fa3cce00700 1 --2- 172.21.15.28:0/1350416583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa3c8103500 0x7fa3c819ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:15.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.659+0000 7fa3cce00700 1 -- 172.21.15.28:0/1350416583 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3c8108c50 msgr2=0x7fa3c819fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:15.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.659+0000 7fa3cce00700 1 --2- 172.21.15.28:0/1350416583 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3c8108c50 0x7fa3c819fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:15.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.659+0000 7fa3cce00700 1 -- 172.21.15.28:0/1350416583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3c81a04f0 con 0x7fa3c8102b40 2024-04-23T14:42:15.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.660+0000 7fa3cce00700 1 --2- 172.21.15.28:0/1350416583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3c8102b40 0x7fa3c819b4e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc00bae0 tx=0x7fa3bc00bb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:15.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.660+0000 7fa3c5600700 1 -- 172.21.15.28:0/1350416583 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc026070 con 0x7fa3c8102b40 2024-04-23T14:42:15.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.660+0000 7fa3c5600700 1 -- 172.21.15.28:0/1350416583 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3bc003d70 con 0x7fa3c8102b40 2024-04-23T14:42:15.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.660+0000 7fa3c5600700 1 -- 172.21.15.28:0/1350416583 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc021410 con 0x7fa3c8102b40 2024-04-23T14:42:15.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.660+0000 7fa3cf800700 1 -- 172.21.15.28:0/1350416583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3c81a0780 con 0x7fa3c8102b40 2024-04-23T14:42:15.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.660+0000 7fa3cf800700 1 -- 172.21.15.28:0/1350416583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3c81a8060 con 0x7fa3c8102b40 2024-04-23T14:42:15.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.661+0000 7fa3cf800700 1 -- 172.21.15.28:0/1350416583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa3c810f9b0 con 0x7fa3c8102b40 2024-04-23T14:42:15.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.663+0000 7fa3c5600700 1 -- 172.21.15.28:0/1350416583 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa3bc029070 con 0x7fa3c8102b40 2024-04-23T14:42:15.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.665+0000 7fa3c5600700 1 --2- 172.21.15.28:0/1350416583 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fa3b00a3cd0 0x7fa3b00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:15.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.665+0000 7fa3c5600700 1 -- 172.21.15.28:0/1350416583 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fa3bc0d1950 con 0x7fa3c8102b40 2024-04-23T14:42:15.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.665+0000 7fa3c7e00700 1 --2- 172.21.15.28:0/1350416583 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fa3b00a3cd0 0x7fa3b00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:15.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.665+0000 7fa3c5600700 1 --2- 172.21.15.28:0/1350416583 >> [v2:172.21.15.105:6800/570739029,v1:172.21.15.105:6801/570739029] conn(0x7fa3b00ad4b0 0x7fa3b00af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:15.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.665+0000 7fa3c5600700 1 -- 172.21.15.28:0/1350416583 --> [v2:172.21.15.105:6800/570739029,v1:172.21.15.105:6801/570739029] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa3bc023ec0 con 0x7fa3b00ad4b0 2024-04-23T14:42:15.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.665+0000 7fa3c5600700 1 -- 172.21.15.28:0/1350416583 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fa3bc0d1d00 con 0x7fa3c8102b40 2024-04-23T14:42:15.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.666+0000 7fa3cd800700 1 --2- 172.21.15.28:0/1350416583 >> [v2:172.21.15.105:6800/570739029,v1:172.21.15.105:6801/570739029] conn(0x7fa3b00ad4b0 0x7fa3b00af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:15.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.666+0000 7fa3c7e00700 1 --2- 172.21.15.28:0/1350416583 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fa3b00a3cd0 0x7fa3b00a6190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa3c807a620 tx=0x7fa3b4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:15.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.666+0000 7fa3cd800700 1 --2- 172.21.15.28:0/1350416583 >> [v2:172.21.15.105:6800/570739029,v1:172.21.15.105:6801/570739029] conn(0x7fa3b00ad4b0 0x7fa3b00af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:15.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.667+0000 7fa3c5600700 1 -- 172.21.15.28:0/1350416583 <== osd.2 v2:172.21.15.105:6800/570739029 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa3bc023ec0 con 0x7fa3b00ad4b0 2024-04-23T14:42:15.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.686+0000 7fa3cf800700 1 -- 172.21.15.28:0/1350416583 --> [v2:172.21.15.105:6800/570739029,v1:172.21.15.105:6801/570739029] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa3c81041f0 con 0x7fa3b00ad4b0 2024-04-23T14:42:15.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.686+0000 7fa3c5600700 1 -- 172.21.15.28:0/1350416583 <== osd.2 v2:172.21.15.105:6800/570739029 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa3c81041f0 con 0x7fa3b00ad4b0 2024-04-23T14:42:15.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.687+0000 7fa3cf800700 1 -- 172.21.15.28:0/1350416583 >> [v2:172.21.15.105:6800/570739029,v1:172.21.15.105:6801/570739029] conn(0x7fa3b00ad4b0 msgr2=0x7fa3b00af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:15.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.687+0000 7fa3cf800700 1 --2- 172.21.15.28:0/1350416583 >> [v2:172.21.15.105:6800/570739029,v1:172.21.15.105:6801/570739029] conn(0x7fa3b00ad4b0 0x7fa3b00af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:15.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.687+0000 7fa3cf800700 1 -- 172.21.15.28:0/1350416583 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fa3b00a3cd0 msgr2=0x7fa3b00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:15.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.687+0000 7fa3cf800700 1 --2- 172.21.15.28:0/1350416583 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fa3b00a3cd0 0x7fa3b00a6190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa3c807a620 tx=0x7fa3b4009260 comp rx=0 tx=0).stop 2024-04-23T14:42:15.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.687+0000 7fa3cf800700 1 -- 172.21.15.28:0/1350416583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3c8102b40 msgr2=0x7fa3c819b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:15.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.687+0000 7fa3cf800700 1 --2- 172.21.15.28:0/1350416583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3c8102b40 0x7fa3c819b4e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc00bae0 tx=0x7fa3bc00bb10 comp rx=0 tx=0).stop 2024-04-23T14:42:15.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.687+0000 7fa3cd800700 1 -- 172.21.15.28:0/1350416583 reap_dead start 2024-04-23T14:42:15.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.687+0000 7fa3cf800700 1 -- 172.21.15.28:0/1350416583 shutdown_connections 2024-04-23T14:42:15.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.687+0000 7fa3cf800700 1 -- 172.21.15.28:0/1350416583 >> 172.21.15.28:0/1350416583 conn(0x7fa3c8079ca0 msgr2=0x7fa3c810df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:15.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.687+0000 7fa3cf800700 1 -- 172.21.15.28:0/1350416583 shutdown_connections 2024-04-23T14:42:15.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:15.687+0000 7fa3cf800700 1 -- 172.21.15.28:0/1350416583 wait complete. 2024-04-23T14:42:15.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:15 smithi105 ceph-mon[32555]: pgmap v331: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-04-23T14:42:15.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:15 smithi038 ceph-mon[32348]: pgmap v331: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-04-23T14:42:15.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:15 smithi028 ceph-mon[27810]: pgmap v331: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-04-23T14:42:16.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.867+0000 7f9c1f600700 1 -- 172.21.15.28:0/3025971715 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c18109380 msgr2=0x7f9c18109760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:16.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.867+0000 7f9c1f600700 1 --2- 172.21.15.28:0/3025971715 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c18109380 0x7f9c18109760 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c0098f0 tx=0x7f9c0c009c00 comp rx=0 tx=0).stop 2024-04-23T14:42:16.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.867+0000 7f9c1f600700 1 -- 172.21.15.28:0/3025971715 shutdown_connections 2024-04-23T14:42:16.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.867+0000 7f9c1f600700 1 --2- 172.21.15.28:0/3025971715 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c1810e460 0x7f9c18116030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:16.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.867+0000 7f9c1f600700 1 --2- 172.21.15.28:0/3025971715 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c18109380 0x7f9c18109760 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:16.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.867+0000 7f9c1f600700 1 --2- 172.21.15.28:0/3025971715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c18109d30 0x7f9c1810dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:16.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.867+0000 7f9c1f600700 1 -- 172.21.15.28:0/3025971715 >> 172.21.15.28:0/3025971715 conn(0x7f9c180762c0 msgr2=0x7f9c18107910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:16.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.867+0000 7f9c1f600700 1 -- 172.21.15.28:0/3025971715 shutdown_connections 2024-04-23T14:42:16.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.868+0000 7f9c1f600700 1 -- 172.21.15.28:0/3025971715 wait complete. 2024-04-23T14:42:16.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.869+0000 7f9c1f600700 1 Processor -- start 2024-04-23T14:42:16.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.869+0000 7f9c1f600700 1 -- start start 2024-04-23T14:42:16.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.869+0000 7f9c1f600700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c18109380 0x7f9c181a5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:16.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.869+0000 7f9c1f600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c18109d30 0x7f9c181a5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:16.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.869+0000 7f9c1f600700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c1810e460 0x7f9c1819f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:16.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.869+0000 7f9c1f600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9c18119490 con 0x7f9c18109d30 2024-04-23T14:42:16.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.869+0000 7f9c1f600700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9c18119310 con 0x7f9c18109380 2024-04-23T14:42:16.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.869+0000 7f9c1f600700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9c18119610 con 0x7f9c1810e460 2024-04-23T14:42:16.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.870+0000 7f9c1cc00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c18109380 0x7f9c181a5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:16.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.870+0000 7f9c1cc00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c18109380 0x7f9c181a5630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:44742/0 (socket says 172.21.15.28:44742) 2024-04-23T14:42:16.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.870+0000 7f9c1cc00700 1 -- 172.21.15.28:0/2907636425 learned_addr learned my addr 172.21.15.28:0/2907636425 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:42:16.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.870+0000 7f9c17e00700 1 --2- 172.21.15.28:0/2907636425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c18109d30 0x7f9c181a5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:16.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.870+0000 7f9c1d600700 1 --2- 172.21.15.28:0/2907636425 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c1810e460 0x7f9c1819f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:16.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.870+0000 7f9c1cc00700 1 -- 172.21.15.28:0/2907636425 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c1810e460 msgr2=0x7f9c1819f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:16.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.870+0000 7f9c1cc00700 1 --2- 172.21.15.28:0/2907636425 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c1810e460 0x7f9c1819f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:16.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.870+0000 7f9c1cc00700 1 -- 172.21.15.28:0/2907636425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c18109d30 msgr2=0x7f9c181a5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:16.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.870+0000 7f9c1cc00700 1 --2- 172.21.15.28:0/2907636425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c18109d30 0x7f9c181a5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:16.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.870+0000 7f9c1cc00700 1 -- 172.21.15.28:0/2907636425 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c181a0030 con 0x7f9c18109380 2024-04-23T14:42:16.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.870+0000 7f9c1cc00700 1 --2- 172.21.15.28:0/2907636425 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c18109380 0x7f9c181a5630 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c009fd0 tx=0x7f9c0c0094b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:16.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.871+0000 7f9c15600700 1 -- 172.21.15.28:0/2907636425 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c025070 con 0x7f9c18109380 2024-04-23T14:42:16.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.871+0000 7f9c15600700 1 -- 172.21.15.28:0/2907636425 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c0c00b560 con 0x7f9c18109380 2024-04-23T14:42:16.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.871+0000 7f9c1f600700 1 -- 172.21.15.28:0/2907636425 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c181a02c0 con 0x7f9c18109380 2024-04-23T14:42:16.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.871+0000 7f9c15600700 1 -- 172.21.15.28:0/2907636425 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c004c30 con 0x7f9c18109380 2024-04-23T14:42:16.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.871+0000 7f9c1f600700 1 -- 172.21.15.28:0/2907636425 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c181ac5b0 con 0x7f9c18109380 2024-04-23T14:42:16.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.872+0000 7f9c1f600700 1 -- 172.21.15.28:0/2907636425 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9c18063280 con 0x7f9c18109380 2024-04-23T14:42:16.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.874+0000 7f9c15600700 1 -- 172.21.15.28:0/2907636425 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9c0c028070 con 0x7f9c18109380 2024-04-23T14:42:16.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.876+0000 7f9c15600700 1 --2- 172.21.15.28:0/2907636425 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f9c000a3d50 0x7f9c000a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:16.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.876+0000 7f9c15600700 1 -- 172.21.15.28:0/2907636425 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f9c0c0d0a30 con 0x7f9c18109380 2024-04-23T14:42:16.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.876+0000 7f9c17e00700 1 --2- 172.21.15.28:0/2907636425 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f9c000a3d50 0x7f9c000a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:16.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.876+0000 7f9c15600700 1 --2- 172.21.15.28:0/2907636425 >> [v2:172.21.15.38:6824/2444616468,v1:172.21.15.38:6825/2444616468] conn(0x7f9c000ad530 0x7f9c000af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:16.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.876+0000 7f9c15600700 1 -- 172.21.15.28:0/2907636425 --> [v2:172.21.15.38:6824/2444616468,v1:172.21.15.38:6825/2444616468] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9c000b0000 con 0x7f9c000ad530 2024-04-23T14:42:16.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.876+0000 7f9c15600700 1 -- 172.21.15.28:0/2907636425 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f9c0c0d0de0 con 0x7f9c18109380 2024-04-23T14:42:16.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.877+0000 7f9c1d600700 1 --2- 172.21.15.28:0/2907636425 >> [v2:172.21.15.38:6824/2444616468,v1:172.21.15.38:6825/2444616468] conn(0x7f9c000ad530 0x7f9c000af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:16.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.877+0000 7f9c17e00700 1 --2- 172.21.15.28:0/2907636425 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f9c000a3d50 0x7f9c000a6210 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9c04009750 tx=0x7f9c04009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:16.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.877+0000 7f9c1d600700 1 --2- 172.21.15.28:0/2907636425 >> [v2:172.21.15.38:6824/2444616468,v1:172.21.15.38:6825/2444616468] conn(0x7f9c000ad530 0x7f9c000af950 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:16.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.878+0000 7f9c15600700 1 -- 172.21.15.28:0/2907636425 <== osd.10 v2:172.21.15.38:6824/2444616468 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9c000b0000 con 0x7f9c000ad530 2024-04-23T14:42:16.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.897+0000 7f9c1f600700 1 -- 172.21.15.28:0/2907636425 --> [v2:172.21.15.38:6824/2444616468,v1:172.21.15.38:6825/2444616468] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9c180636b0 con 0x7f9c000ad530 2024-04-23T14:42:16.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.898+0000 7f9c15600700 1 -- 172.21.15.28:0/2907636425 <== osd.10 v2:172.21.15.38:6824/2444616468 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9c180636b0 con 0x7f9c000ad530 2024-04-23T14:42:16.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.898+0000 7f9c1f600700 1 -- 172.21.15.28:0/2907636425 >> [v2:172.21.15.38:6824/2444616468,v1:172.21.15.38:6825/2444616468] conn(0x7f9c000ad530 msgr2=0x7f9c000af950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:16.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.898+0000 7f9c1f600700 1 --2- 172.21.15.28:0/2907636425 >> [v2:172.21.15.38:6824/2444616468,v1:172.21.15.38:6825/2444616468] conn(0x7f9c000ad530 0x7f9c000af950 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:16.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.899+0000 7f9c1f600700 1 -- 172.21.15.28:0/2907636425 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f9c000a3d50 msgr2=0x7f9c000a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:16.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.899+0000 7f9c1f600700 1 --2- 172.21.15.28:0/2907636425 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f9c000a3d50 0x7f9c000a6210 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9c04009750 tx=0x7f9c04009310 comp rx=0 tx=0).stop 2024-04-23T14:42:16.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.899+0000 7f9c1f600700 1 -- 172.21.15.28:0/2907636425 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c18109380 msgr2=0x7f9c181a5630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:16.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.899+0000 7f9c1f600700 1 --2- 172.21.15.28:0/2907636425 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c18109380 0x7f9c181a5630 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c009fd0 tx=0x7f9c0c0094b0 comp rx=0 tx=0).stop 2024-04-23T14:42:16.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.899+0000 7f9c1d600700 1 -- 172.21.15.28:0/2907636425 reap_dead start 2024-04-23T14:42:16.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.899+0000 7f9c1f600700 1 -- 172.21.15.28:0/2907636425 shutdown_connections 2024-04-23T14:42:16.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.899+0000 7f9c1f600700 1 -- 172.21.15.28:0/2907636425 >> 172.21.15.28:0/2907636425 conn(0x7f9c180762c0 msgr2=0x7f9c18077950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:16.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.899+0000 7f9c1f600700 1 -- 172.21.15.28:0/2907636425 shutdown_connections 2024-04-23T14:42:16.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:16.899+0000 7f9c1f600700 1 -- 172.21.15.28:0/2907636425 wait complete. 2024-04-23T14:42:17.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:17 smithi038 ceph-mon[32348]: pgmap v332: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-23T14:42:17.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:17 smithi028 ceph-mon[27810]: pgmap v332: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-23T14:42:18.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:17 smithi105 ceph-mon[32555]: pgmap v332: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-23T14:42:19.413 INFO:teuthology.orchestra.run.smithi028.stdout:253403070500 2024-04-23T14:42:19.413 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-23T14:42:19.764 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:19 smithi028 ceph-mon[27810]: pgmap v333: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-23T14:42:19.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:19 smithi038 ceph-mon[32348]: pgmap v333: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-23T14:42:20.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:19 smithi105 ceph-mon[32555]: pgmap v333: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-23T14:42:21.342 INFO:teuthology.orchestra.run.smithi028.stdout:55834574926 2024-04-23T14:42:21.343 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-23T14:42:21.819 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:21 smithi028 ceph-mon[27810]: pgmap v334: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-23T14:42:21.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:21 smithi038 ceph-mon[32348]: pgmap v334: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-23T14:42:21.885 INFO:teuthology.orchestra.run.smithi028.stdout:77309411401 2024-04-23T14:42:21.885 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-23T14:42:22.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:21 smithi105 ceph-mon[32555]: pgmap v334: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-23T14:42:22.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.510+0000 7fb811200700 1 -- 172.21.15.28:0/383684741 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb80c105910 msgr2=0x7fb80c109960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:22.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.510+0000 7fb811200700 1 --2- 172.21.15.28:0/383684741 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb80c105910 0x7fb80c109960 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8009920 tx=0x7fb7f8009c30 comp rx=0 tx=0).stop 2024-04-23T14:42:22.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.511+0000 7fb811200700 1 -- 172.21.15.28:0/383684741 shutdown_connections 2024-04-23T14:42:22.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.511+0000 7fb811200700 1 --2- 172.21.15.28:0/383684741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb80c10a040 0x7fb80c111c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:22.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.511+0000 7fb811200700 1 --2- 172.21.15.28:0/383684741 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb80c104f60 0x7fb80c105340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.511+0000 7fb811200700 1 --2- 172.21.15.28:0/383684741 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb80c105910 0x7fb80c109960 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.511+0000 7fb811200700 1 -- 172.21.15.28:0/383684741 >> 172.21.15.28:0/383684741 conn(0x7fb80c1007d0 msgr2=0x7fb80c102bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.511+0000 7fb811200700 1 -- 172.21.15.28:0/383684741 shutdown_connections 2024-04-23T14:42:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.511+0000 7fb811200700 1 -- 172.21.15.28:0/383684741 wait complete. 2024-04-23T14:42:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.512+0000 7fb811200700 1 Processor -- start 2024-04-23T14:42:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.512+0000 7fb811200700 1 -- start start 2024-04-23T14:42:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.513+0000 7fb811200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb80c104f60 0x7fb80c19b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.513+0000 7fb811200700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb80c105910 0x7fb80c19bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.513+0000 7fb811200700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb80c10a040 0x7fb80c19fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.513+0000 7fb811200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb80c115070 con 0x7fb80c104f60 2024-04-23T14:42:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.513+0000 7fb811200700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb80c114ef0 con 0x7fb80c10a040 2024-04-23T14:42:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.513+0000 7fb811200700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb80c1151f0 con 0x7fb80c105910 2024-04-23T14:42:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.513+0000 7fb809e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb80c104f60 0x7fb80c19b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.513+0000 7fb809e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb80c104f60 0x7fb80c19b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54276/0 (socket says 172.21.15.28:54276) 2024-04-23T14:42:22.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.513+0000 7fb809e00700 1 -- 172.21.15.28:0/3267941168 learned_addr learned my addr 172.21.15.28:0/3267941168 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:42:22.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.513+0000 7fb809400700 1 --2- 172.21.15.28:0/3267941168 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb80c105910 0x7fb80c19bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:22.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.513+0000 7fb80a800700 1 --2- 172.21.15.28:0/3267941168 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb80c10a040 0x7fb80c19fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:22.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.513+0000 7fb809400700 1 -- 172.21.15.28:0/3267941168 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb80c10a040 msgr2=0x7fb80c19fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:22.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.513+0000 7fb809400700 1 --2- 172.21.15.28:0/3267941168 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb80c10a040 0x7fb80c19fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:22.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.513+0000 7fb809400700 1 -- 172.21.15.28:0/3267941168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb80c104f60 msgr2=0x7fb80c19b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:22.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.513+0000 7fb809400700 1 --2- 172.21.15.28:0/3267941168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb80c104f60 0x7fb80c19b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:22.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.513+0000 7fb809400700 1 -- 172.21.15.28:0/3267941168 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb80c1a05a0 con 0x7fb80c105910 2024-04-23T14:42:22.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.513+0000 7fb80a800700 1 --2- 172.21.15.28:0/3267941168 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb80c10a040 0x7fb80c19fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:42:22.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.514+0000 7fb809400700 1 --2- 172.21.15.28:0/3267941168 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb80c105910 0x7fb80c19bb20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8000c00 tx=0x7fb7f800fa70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:22.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.514+0000 7fb802a00700 1 -- 172.21.15.28:0/3267941168 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f8005140 con 0x7fb80c105910 2024-04-23T14:42:22.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.514+0000 7fb802a00700 1 -- 172.21.15.28:0/3267941168 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7f800b560 con 0x7fb80c105910 2024-04-23T14:42:22.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.514+0000 7fb802a00700 1 -- 172.21.15.28:0/3267941168 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f8003aa0 con 0x7fb80c105910 2024-04-23T14:42:22.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.514+0000 7fb811200700 1 -- 172.21.15.28:0/3267941168 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb80c1a0830 con 0x7fb80c105910 2024-04-23T14:42:22.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.515+0000 7fb811200700 1 -- 172.21.15.28:0/3267941168 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb80c1a0c30 con 0x7fb80c105910 2024-04-23T14:42:22.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.515+0000 7fb811200700 1 -- 172.21.15.28:0/3267941168 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb80c063240 con 0x7fb80c105910 2024-04-23T14:42:22.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.518+0000 7fb802a00700 1 -- 172.21.15.28:0/3267941168 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb7f8029060 con 0x7fb80c105910 2024-04-23T14:42:22.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.519+0000 7fb802a00700 1 --2- 172.21.15.28:0/3267941168 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb7f00a3cd0 0x7fb7f00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:22.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.519+0000 7fb802a00700 1 -- 172.21.15.28:0/3267941168 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fb7f80d1520 con 0x7fb80c105910 2024-04-23T14:42:22.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.519+0000 7fb809e00700 1 --2- 172.21.15.28:0/3267941168 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb7f00a3cd0 0x7fb7f00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:22.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.519+0000 7fb802a00700 1 --2- 172.21.15.28:0/3267941168 >> [v2:172.21.15.38:6816/2400251786,v1:172.21.15.38:6817/2400251786] conn(0x7fb7f00ad380 0x7fb7f00af7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:22.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.520+0000 7fb802a00700 1 -- 172.21.15.28:0/3267941168 --> [v2:172.21.15.38:6816/2400251786,v1:172.21.15.38:6817/2400251786] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb7f800b0b0 con 0x7fb7f00ad380 2024-04-23T14:42:22.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.520+0000 7fb802a00700 1 -- 172.21.15.28:0/3267941168 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fb7f80d18d0 con 0x7fb80c105910 2024-04-23T14:42:22.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.520+0000 7fb80a800700 1 --2- 172.21.15.28:0/3267941168 >> [v2:172.21.15.38:6816/2400251786,v1:172.21.15.38:6817/2400251786] conn(0x7fb7f00ad380 0x7fb7f00af7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:22.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.520+0000 7fb809e00700 1 --2- 172.21.15.28:0/3267941168 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb7f00a3cd0 0x7fb7f00a6190 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc009d60 tx=0x7fb7fc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:22.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.521+0000 7fb80a800700 1 --2- 172.21.15.28:0/3267941168 >> [v2:172.21.15.38:6816/2400251786,v1:172.21.15.38:6817/2400251786] conn(0x7fb7f00ad380 0x7fb7f00af7a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:22.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.522+0000 7fb802a00700 1 -- 172.21.15.28:0/3267941168 <== osd.7 v2:172.21.15.38:6816/2400251786 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb7f800b0b0 con 0x7fb7f00ad380 2024-04-23T14:42:22.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.541+0000 7fb811200700 1 -- 172.21.15.28:0/3267941168 --> [v2:172.21.15.38:6816/2400251786,v1:172.21.15.38:6817/2400251786] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb80c0636c0 con 0x7fb7f00ad380 2024-04-23T14:42:22.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.541+0000 7fb802a00700 1 -- 172.21.15.28:0/3267941168 <== osd.7 v2:172.21.15.38:6816/2400251786 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb80c0636c0 con 0x7fb7f00ad380 2024-04-23T14:42:22.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.542+0000 7fb811200700 1 -- 172.21.15.28:0/3267941168 >> [v2:172.21.15.38:6816/2400251786,v1:172.21.15.38:6817/2400251786] conn(0x7fb7f00ad380 msgr2=0x7fb7f00af7a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:22.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.542+0000 7fb811200700 1 --2- 172.21.15.28:0/3267941168 >> [v2:172.21.15.38:6816/2400251786,v1:172.21.15.38:6817/2400251786] conn(0x7fb7f00ad380 0x7fb7f00af7a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:22.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.542+0000 7fb811200700 1 -- 172.21.15.28:0/3267941168 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb7f00a3cd0 msgr2=0x7fb7f00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:22.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.542+0000 7fb811200700 1 --2- 172.21.15.28:0/3267941168 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb7f00a3cd0 0x7fb7f00a6190 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc009d60 tx=0x7fb7fc009260 comp rx=0 tx=0).stop 2024-04-23T14:42:22.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.542+0000 7fb811200700 1 -- 172.21.15.28:0/3267941168 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb80c105910 msgr2=0x7fb80c19bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:22.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.542+0000 7fb811200700 1 --2- 172.21.15.28:0/3267941168 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb80c105910 0x7fb80c19bb20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8000c00 tx=0x7fb7f800fa70 comp rx=0 tx=0).stop 2024-04-23T14:42:22.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.542+0000 7fb80a800700 1 -- 172.21.15.28:0/3267941168 reap_dead start 2024-04-23T14:42:22.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.543+0000 7fb811200700 1 -- 172.21.15.28:0/3267941168 shutdown_connections 2024-04-23T14:42:22.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.543+0000 7fb811200700 1 -- 172.21.15.28:0/3267941168 >> 172.21.15.28:0/3267941168 conn(0x7fb80c1007d0 msgr2=0x7fb80c102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:22.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.543+0000 7fb811200700 1 -- 172.21.15.28:0/3267941168 shutdown_connections 2024-04-23T14:42:22.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.543+0000 7fb811200700 1 -- 172.21.15.28:0/3267941168 wait complete. 2024-04-23T14:42:23.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.659+0000 7f7f6b200700 1 -- 172.21.15.28:0/3073026527 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f6410a070 msgr2=0x7f7f64111c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:23.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.659+0000 7f7f6b200700 1 --2- 172.21.15.28:0/3073026527 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f6410a070 0x7f7f64111c40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c00b0a0 tx=0x7f7f5c00b3b0 comp rx=0 tx=0).stop 2024-04-23T14:42:23.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.659+0000 7f7f6b200700 1 -- 172.21.15.28:0/3073026527 shutdown_connections 2024-04-23T14:42:23.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.659+0000 7f7f6b200700 1 --2- 172.21.15.28:0/3073026527 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f64104f90 0x7f7f64105370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:23.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.659+0000 7f7f6b200700 1 --2- 172.21.15.28:0/3073026527 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f6410a070 0x7f7f64111c40 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:23.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.659+0000 7f7f6b200700 1 --2- 172.21.15.28:0/3073026527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f64105940 0x7f7f64109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:23.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.659+0000 7f7f6b200700 1 -- 172.21.15.28:0/3073026527 >> 172.21.15.28:0/3073026527 conn(0x7f7f64100820 msgr2=0x7f7f64102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:23.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.659+0000 7f7f6b200700 1 -- 172.21.15.28:0/3073026527 shutdown_connections 2024-04-23T14:42:23.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.659+0000 7f7f6b200700 1 -- 172.21.15.28:0/3073026527 wait complete. 2024-04-23T14:42:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.661+0000 7f7f6b200700 1 Processor -- start 2024-04-23T14:42:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.661+0000 7f7f6b200700 1 -- start start 2024-04-23T14:42:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.661+0000 7f7f6b200700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f64104f90 0x7f7f6419b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.661+0000 7f7f6b200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f64105940 0x7f7f6419b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.661+0000 7f7f6b200700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f6410a070 0x7f7f6419fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.661+0000 7f7f6b200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7f64115060 con 0x7f7f64105940 2024-04-23T14:42:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.661+0000 7f7f6b200700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7f64114ee0 con 0x7f7f6410a070 2024-04-23T14:42:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.661+0000 7f7f6b200700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7f641151e0 con 0x7f7f64104f90 2024-04-23T14:42:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.662+0000 7f7f63400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f64105940 0x7f7f6419b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.662+0000 7f7f63e00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f64104f90 0x7f7f6419b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.662+0000 7f7f69200700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f6410a070 0x7f7f6419fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:23.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.662+0000 7f7f63e00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f64104f90 0x7f7f6419b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:52500/0 (socket says 172.21.15.28:52500) 2024-04-23T14:42:23.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.662+0000 7f7f63e00700 1 -- 172.21.15.28:0/2201810028 learned_addr learned my addr 172.21.15.28:0/2201810028 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:42:23.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.662+0000 7f7f63e00700 1 -- 172.21.15.28:0/2201810028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f6410a070 msgr2=0x7f7f6419fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:23.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.662+0000 7f7f63e00700 1 --2- 172.21.15.28:0/2201810028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f6410a070 0x7f7f6419fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:23.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.662+0000 7f7f63e00700 1 -- 172.21.15.28:0/2201810028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f64105940 msgr2=0x7f7f6419b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:23.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.662+0000 7f7f63e00700 1 --2- 172.21.15.28:0/2201810028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f64105940 0x7f7f6419b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:23.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.662+0000 7f7f63e00700 1 -- 172.21.15.28:0/2201810028 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f641a0460 con 0x7f7f64104f90 2024-04-23T14:42:23.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.662+0000 7f7f63e00700 1 --2- 172.21.15.28:0/2201810028 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f64104f90 0x7f7f6419b450 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7f5400e9a0 tx=0x7f7f5400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:23.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.663+0000 7f7f60c00700 1 -- 172.21.15.28:0/2201810028 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5400ca10 con 0x7f7f64104f90 2024-04-23T14:42:23.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.663+0000 7f7f60c00700 1 -- 172.21.15.28:0/2201810028 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f5400cbb0 con 0x7f7f64104f90 2024-04-23T14:42:23.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.663+0000 7f7f60c00700 1 -- 172.21.15.28:0/2201810028 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f54018680 con 0x7f7f64104f90 2024-04-23T14:42:23.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.663+0000 7f7f6b200700 1 -- 172.21.15.28:0/2201810028 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f641a0750 con 0x7f7f64104f90 2024-04-23T14:42:23.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.663+0000 7f7f6b200700 1 -- 172.21.15.28:0/2201810028 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f641a7fd0 con 0x7f7f64104f90 2024-04-23T14:42:23.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.664+0000 7f7f6b200700 1 -- 172.21.15.28:0/2201810028 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7f64063200 con 0x7f7f64104f90 2024-04-23T14:42:23.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.666+0000 7f7f60c00700 1 -- 172.21.15.28:0/2201810028 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7f54010430 con 0x7f7f64104f90 2024-04-23T14:42:23.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.667+0000 7f7f60c00700 1 --2- 172.21.15.28:0/2201810028 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f7f480a3d50 0x7f7f480a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:23.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.668+0000 7f7f60c00700 1 -- 172.21.15.28:0/2201810028 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f7f540d0e90 con 0x7f7f64104f90 2024-04-23T14:42:23.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.668+0000 7f7f63400700 1 --2- 172.21.15.28:0/2201810028 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f7f480a3d50 0x7f7f480a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:23.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.668+0000 7f7f60c00700 1 --2- 172.21.15.28:0/2201810028 >> [v2:172.21.15.105:6824/3400299562,v1:172.21.15.105:6825/3400299562] conn(0x7f7f480ad530 0x7f7f480af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:23.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.668+0000 7f7f60c00700 1 -- 172.21.15.28:0/2201810028 --> [v2:172.21.15.105:6824/3400299562,v1:172.21.15.105:6825/3400299562] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7f54027e50 con 0x7f7f480ad530 2024-04-23T14:42:23.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.668+0000 7f7f60c00700 1 -- 172.21.15.28:0/2201810028 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f7f540d1240 con 0x7f7f64104f90 2024-04-23T14:42:23.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.668+0000 7f7f69200700 1 --2- 172.21.15.28:0/2201810028 >> [v2:172.21.15.105:6824/3400299562,v1:172.21.15.105:6825/3400299562] conn(0x7f7f480ad530 0x7f7f480af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:23.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.668+0000 7f7f63400700 1 --2- 172.21.15.28:0/2201810028 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f7f480a3d50 0x7f7f480a6210 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7f64103f20 tx=0x7f7f50009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:23.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.669+0000 7f7f69200700 1 --2- 172.21.15.28:0/2201810028 >> [v2:172.21.15.105:6824/3400299562,v1:172.21.15.105:6825/3400299562] conn(0x7f7f480ad530 0x7f7f480af950 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:23.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.670+0000 7f7f60c00700 1 -- 172.21.15.28:0/2201810028 <== osd.11 v2:172.21.15.105:6824/3400299562 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7f54027e50 con 0x7f7f480ad530 2024-04-23T14:42:23.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.690+0000 7f7f6b200700 1 -- 172.21.15.28:0/2201810028 --> [v2:172.21.15.105:6824/3400299562,v1:172.21.15.105:6825/3400299562] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7f640636d0 con 0x7f7f480ad530 2024-04-23T14:42:23.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.691+0000 7f7f60c00700 1 -- 172.21.15.28:0/2201810028 <== osd.11 v2:172.21.15.105:6824/3400299562 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7f640636d0 con 0x7f7f480ad530 2024-04-23T14:42:23.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.691+0000 7f7f6b200700 1 -- 172.21.15.28:0/2201810028 >> [v2:172.21.15.105:6824/3400299562,v1:172.21.15.105:6825/3400299562] conn(0x7f7f480ad530 msgr2=0x7f7f480af950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:23.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.691+0000 7f7f6b200700 1 --2- 172.21.15.28:0/2201810028 >> [v2:172.21.15.105:6824/3400299562,v1:172.21.15.105:6825/3400299562] conn(0x7f7f480ad530 0x7f7f480af950 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:23.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.691+0000 7f7f6b200700 1 -- 172.21.15.28:0/2201810028 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f7f480a3d50 msgr2=0x7f7f480a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:23.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.691+0000 7f7f6b200700 1 --2- 172.21.15.28:0/2201810028 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f7f480a3d50 0x7f7f480a6210 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7f64103f20 tx=0x7f7f50009260 comp rx=0 tx=0).stop 2024-04-23T14:42:23.820 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:23 smithi028 ceph-mon[27810]: pgmap v335: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:42:23.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.691+0000 7f7f6b200700 1 -- 172.21.15.28:0/2201810028 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f64104f90 msgr2=0x7f7f6419b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:23.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.691+0000 7f7f6b200700 1 --2- 172.21.15.28:0/2201810028 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f64104f90 0x7f7f6419b450 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7f5400e9a0 tx=0x7f7f5400ecb0 comp rx=0 tx=0).stop 2024-04-23T14:42:23.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.691+0000 7f7f69200700 1 -- 172.21.15.28:0/2201810028 reap_dead start 2024-04-23T14:42:23.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.691+0000 7f7f6b200700 1 -- 172.21.15.28:0/2201810028 shutdown_connections 2024-04-23T14:42:23.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.691+0000 7f7f6b200700 1 -- 172.21.15.28:0/2201810028 >> 172.21.15.28:0/2201810028 conn(0x7f7f64100820 msgr2=0x7f7f64101380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:23.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.691+0000 7f7f6b200700 1 -- 172.21.15.28:0/2201810028 shutdown_connections 2024-04-23T14:42:23.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:22.692+0000 7f7f6b200700 1 -- 172.21.15.28:0/2201810028 wait complete. 2024-04-23T14:42:23.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:23 smithi038 ceph-mon[32348]: pgmap v335: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:42:24.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:23 smithi105 ceph-mon[32555]: pgmap v335: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:42:25.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.130+0000 7f497aa00700 1 -- 172.21.15.28:0/3274099409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4974074d20 msgr2=0x7f4974075100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:25.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.130+0000 7f497aa00700 1 --2- 172.21.15.28:0/3274099409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4974074d20 0x7f4974075100 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4964009990 tx=0x7f4964009ca0 comp rx=0 tx=0).stop 2024-04-23T14:42:25.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.130+0000 7f497aa00700 1 -- 172.21.15.28:0/3274099409 shutdown_connections 2024-04-23T14:42:25.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.130+0000 7f497aa00700 1 --2- 172.21.15.28:0/3274099409 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f497410f4f0 0x7f49741118e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:25.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.130+0000 7f497aa00700 1 --2- 172.21.15.28:0/3274099409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4974074d20 0x7f4974075100 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:25.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.130+0000 7f497aa00700 1 --2- 172.21.15.28:0/3274099409 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4974075640 0x7f497410efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:25.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.130+0000 7f497aa00700 1 -- 172.21.15.28:0/3274099409 >> 172.21.15.28:0/3274099409 conn(0x7f4974079ca0 msgr2=0x7f497407a0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:25.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.130+0000 7f497aa00700 1 -- 172.21.15.28:0/3274099409 shutdown_connections 2024-04-23T14:42:25.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.130+0000 7f497aa00700 1 -- 172.21.15.28:0/3274099409 wait complete. 2024-04-23T14:42:25.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.131+0000 7f497aa00700 1 Processor -- start 2024-04-23T14:42:25.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.132+0000 7f497aa00700 1 -- start start 2024-04-23T14:42:25.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.132+0000 7f497aa00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4974074d20 0x7f497419b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:25.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.132+0000 7f497aa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4974075640 0x7f497419ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:25.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.132+0000 7f497aa00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f497410f4f0 0x7f497419fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:25.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.132+0000 7f497aa00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f49741150a0 con 0x7f4974075640 2024-04-23T14:42:25.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.132+0000 7f497aa00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4974114f20 con 0x7f497410f4f0 2024-04-23T14:42:25.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.132+0000 7f497aa00700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4974115220 con 0x7f4974074d20 2024-04-23T14:42:25.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.132+0000 7f496be00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4974075640 0x7f497419ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:25.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.132+0000 7f4973e00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4974074d20 0x7f497419b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:25.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.133+0000 7f4978a00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f497410f4f0 0x7f497419fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:25.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.133+0000 7f496be00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4974075640 0x7f497419ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54338/0 (socket says 172.21.15.28:54338) 2024-04-23T14:42:25.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.133+0000 7f4973e00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4974074d20 0x7f497419b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:52544/0 (socket says 172.21.15.28:52544) 2024-04-23T14:42:25.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.133+0000 7f496be00700 1 -- 172.21.15.28:0/1222341055 learned_addr learned my addr 172.21.15.28:0/1222341055 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:42:25.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.133+0000 7f4978a00700 1 -- 172.21.15.28:0/1222341055 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4974074d20 msgr2=0x7f497419b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:25.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.133+0000 7f4978a00700 1 --2- 172.21.15.28:0/1222341055 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4974074d20 0x7f497419b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:25.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.133+0000 7f4978a00700 1 -- 172.21.15.28:0/1222341055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4974075640 msgr2=0x7f497419ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:25.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.133+0000 7f4978a00700 1 --2- 172.21.15.28:0/1222341055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4974075640 0x7f497419ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:25.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.133+0000 7f4978a00700 1 -- 172.21.15.28:0/1222341055 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49741a0560 con 0x7f497410f4f0 2024-04-23T14:42:25.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.133+0000 7f4978a00700 1 --2- 172.21.15.28:0/1222341055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f497410f4f0 0x7f497419fe20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f497406b320 tx=0x7f496c00d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:25.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.134+0000 7f4971600700 1 -- 172.21.15.28:0/1222341055 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496c01a070 con 0x7f497410f4f0 2024-04-23T14:42:25.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.134+0000 7f4971600700 1 -- 172.21.15.28:0/1222341055 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f496c015400 con 0x7f497410f4f0 2024-04-23T14:42:25.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.134+0000 7f4971600700 1 -- 172.21.15.28:0/1222341055 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496c0145e0 con 0x7f497410f4f0 2024-04-23T14:42:25.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.134+0000 7f497aa00700 1 -- 172.21.15.28:0/1222341055 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49741a0850 con 0x7f497410f4f0 2024-04-23T14:42:25.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.134+0000 7f497aa00700 1 -- 172.21.15.28:0/1222341055 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49741a80d0 con 0x7f497410f4f0 2024-04-23T14:42:25.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.135+0000 7f497aa00700 1 -- 172.21.15.28:0/1222341055 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4974063280 con 0x7f497410f4f0 2024-04-23T14:42:25.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.137+0000 7f4971600700 1 -- 172.21.15.28:0/1222341055 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f496c01d070 con 0x7f497410f4f0 2024-04-23T14:42:25.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.138+0000 7f4971600700 1 --2- 172.21.15.28:0/1222341055 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f49580a3cd0 0x7f49580a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:25.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.138+0000 7f4971600700 1 -- 172.21.15.28:0/1222341055 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f496c0d01b0 con 0x7f497410f4f0 2024-04-23T14:42:25.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.139+0000 7f4973e00700 1 --2- 172.21.15.28:0/1222341055 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f49580a3cd0 0x7f49580a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:25.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.139+0000 7f4971600700 1 --2- 172.21.15.28:0/1222341055 >> [v2:172.21.15.28:6810/2480400247,v1:172.21.15.28:6811/2480400247] conn(0x7f49580ad4b0 0x7f49580af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:25.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.139+0000 7f4971600700 1 -- 172.21.15.28:0/1222341055 --> [v2:172.21.15.28:6810/2480400247,v1:172.21.15.28:6811/2480400247] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f496c013ec0 con 0x7f49580ad4b0 2024-04-23T14:42:25.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.139+0000 7f4971600700 1 -- 172.21.15.28:0/1222341055 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f496c0d0560 con 0x7f497410f4f0 2024-04-23T14:42:25.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.139+0000 7f496be00700 1 --2- 172.21.15.28:0/1222341055 >> [v2:172.21.15.28:6810/2480400247,v1:172.21.15.28:6811/2480400247] conn(0x7f49580ad4b0 0x7f49580af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:25.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.139+0000 7f4973e00700 1 --2- 172.21.15.28:0/1222341055 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f49580a3cd0 0x7f49580a6190 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f496400b2e0 tx=0x7f4964005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:25.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.140+0000 7f496be00700 1 --2- 172.21.15.28:0/1222341055 >> [v2:172.21.15.28:6810/2480400247,v1:172.21.15.28:6811/2480400247] conn(0x7f49580ad4b0 0x7f49580af8d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:25.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.141+0000 7f4971600700 1 -- 172.21.15.28:0/1222341055 <== osd.3 v2:172.21.15.28:6810/2480400247 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f496c013ec0 con 0x7f49580ad4b0 2024-04-23T14:42:25.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.159+0000 7f497aa00700 1 -- 172.21.15.28:0/1222341055 --> [v2:172.21.15.28:6810/2480400247,v1:172.21.15.28:6811/2480400247] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4974063700 con 0x7f49580ad4b0 2024-04-23T14:42:25.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.160+0000 7f4971600700 1 -- 172.21.15.28:0/1222341055 <== osd.3 v2:172.21.15.28:6810/2480400247 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4974063700 con 0x7f49580ad4b0 2024-04-23T14:42:25.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.160+0000 7f497aa00700 1 -- 172.21.15.28:0/1222341055 >> [v2:172.21.15.28:6810/2480400247,v1:172.21.15.28:6811/2480400247] conn(0x7f49580ad4b0 msgr2=0x7f49580af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:25.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.160+0000 7f497aa00700 1 --2- 172.21.15.28:0/1222341055 >> [v2:172.21.15.28:6810/2480400247,v1:172.21.15.28:6811/2480400247] conn(0x7f49580ad4b0 0x7f49580af8d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:25.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.161+0000 7f497aa00700 1 -- 172.21.15.28:0/1222341055 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f49580a3cd0 msgr2=0x7f49580a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:25.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.161+0000 7f497aa00700 1 --2- 172.21.15.28:0/1222341055 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f49580a3cd0 0x7f49580a6190 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f496400b2e0 tx=0x7f4964005fd0 comp rx=0 tx=0).stop 2024-04-23T14:42:25.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.161+0000 7f497aa00700 1 -- 172.21.15.28:0/1222341055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f497410f4f0 msgr2=0x7f497419fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:25.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.161+0000 7f497aa00700 1 --2- 172.21.15.28:0/1222341055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f497410f4f0 0x7f497419fe20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f497406b320 tx=0x7f496c00d3b0 comp rx=0 tx=0).stop 2024-04-23T14:42:25.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.161+0000 7f4978a00700 1 -- 172.21.15.28:0/1222341055 reap_dead start 2024-04-23T14:42:25.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.161+0000 7f497aa00700 1 -- 172.21.15.28:0/1222341055 shutdown_connections 2024-04-23T14:42:25.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.161+0000 7f497aa00700 1 -- 172.21.15.28:0/1222341055 >> 172.21.15.28:0/1222341055 conn(0x7f4974079ca0 msgr2=0x7f49740ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:25.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.161+0000 7f497aa00700 1 -- 172.21.15.28:0/1222341055 shutdown_connections 2024-04-23T14:42:25.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:25.161+0000 7f497aa00700 1 -- 172.21.15.28:0/1222341055 wait complete. 2024-04-23T14:42:25.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:25 smithi028 ceph-mon[27810]: pgmap v336: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:42:25.791 INFO:teuthology.orchestra.run.smithi028.stdout:309237645338 2024-04-23T14:42:25.792 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-23T14:42:25.838 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:25 smithi105 ceph-mon[32555]: pgmap v336: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:42:25.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:25 smithi038 ceph-mon[32348]: pgmap v336: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:42:26.293 INFO:teuthology.orchestra.run.smithi028.stdout:219043332142 2024-04-23T14:42:26.294 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-23T14:42:27.173 INFO:teuthology.orchestra.run.smithi028.stdout:107374182468 2024-04-23T14:42:27.173 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-23T14:42:27.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.102+0000 7f1d08200700 1 -- 172.21.15.28:0/2244078024 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d00103a10 msgr2=0x7f1d0006b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:27.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.102+0000 7f1d08200700 1 --2- 172.21.15.28:0/2244078024 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d00103a10 0x7f1d0006b1a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1cf40098f0 tx=0x7f1cf4009c00 comp rx=0 tx=0).stop 2024-04-23T14:42:27.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.102+0000 7f1d08200700 1 -- 172.21.15.28:0/2244078024 shutdown_connections 2024-04-23T14:42:27.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.102+0000 7f1d08200700 1 --2- 172.21.15.28:0/2244078024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d0010f380 0x7f1d001117b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:27.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.102+0000 7f1d08200700 1 --2- 172.21.15.28:0/2244078024 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d00103a10 0x7f1d0006b1a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:27.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.102+0000 7f1d08200700 1 --2- 172.21.15.28:0/2244078024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1d0006b6e0 0x7f1d0006bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:27.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.102+0000 7f1d08200700 1 -- 172.21.15.28:0/2244078024 >> 172.21.15.28:0/2244078024 conn(0x7f1d00078750 msgr2=0x7f1d00078b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:27.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.102+0000 7f1d08200700 1 -- 172.21.15.28:0/2244078024 shutdown_connections 2024-04-23T14:42:27.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.102+0000 7f1d08200700 1 -- 172.21.15.28:0/2244078024 wait complete. 2024-04-23T14:42:27.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.103+0000 7f1d08200700 1 Processor -- start 2024-04-23T14:42:27.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.104+0000 7f1d08200700 1 -- start start 2024-04-23T14:42:27.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.104+0000 7f1d08200700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1d0006b6e0 0x7f1d0019b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:27.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.104+0000 7f1d08200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d00103a10 0x7f1d0019ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:27.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.104+0000 7f1d08200700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d0010f380 0x7f1d0019fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:27.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.104+0000 7f1d08200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1d00115060 con 0x7f1d00103a10 2024-04-23T14:42:27.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.104+0000 7f1d08200700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1d00114ee0 con 0x7f1d0010f380 2024-04-23T14:42:27.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.104+0000 7f1d08200700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1d001151e0 con 0x7f1d0006b6e0 2024-04-23T14:42:27.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.104+0000 7f1d04e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d00103a10 0x7f1d0019ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:27.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.104+0000 7f1d04e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d00103a10 0x7f1d0019ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54350/0 (socket says 172.21.15.28:54350) 2024-04-23T14:42:27.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.104+0000 7f1d04e00700 1 -- 172.21.15.28:0/2241231973 learned_addr learned my addr 172.21.15.28:0/2241231973 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:42:27.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.104+0000 7f1d05800700 1 --2- 172.21.15.28:0/2241231973 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1d0006b6e0 0x7f1d0019b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:27.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.104+0000 7f1d06200700 1 --2- 172.21.15.28:0/2241231973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d0010f380 0x7f1d0019fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:27.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.105+0000 7f1d05800700 1 -- 172.21.15.28:0/2241231973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d0010f380 msgr2=0x7f1d0019fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:27.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.105+0000 7f1d05800700 1 --2- 172.21.15.28:0/2241231973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d0010f380 0x7f1d0019fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:27.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.105+0000 7f1d05800700 1 -- 172.21.15.28:0/2241231973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d00103a10 msgr2=0x7f1d0019ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:27.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.105+0000 7f1d05800700 1 --2- 172.21.15.28:0/2241231973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d00103a10 0x7f1d0019ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:27.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.105+0000 7f1d05800700 1 -- 172.21.15.28:0/2241231973 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d001a0500 con 0x7f1d0006b6e0 2024-04-23T14:42:27.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.105+0000 7f1d04e00700 1 --2- 172.21.15.28:0/2241231973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d00103a10 0x7f1d0019ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:42:27.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.105+0000 7f1d05800700 1 --2- 172.21.15.28:0/2241231973 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1d0006b6e0 0x7f1d0019b4f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1cf400c010 tx=0x7f1cf400fa70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:27.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.105+0000 7f1cfe000700 1 -- 172.21.15.28:0/2241231973 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cf4005180 con 0x7f1d0006b6e0 2024-04-23T14:42:27.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.105+0000 7f1d08200700 1 -- 172.21.15.28:0/2241231973 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d001a0790 con 0x7f1d0006b6e0 2024-04-23T14:42:27.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.105+0000 7f1cfe000700 1 -- 172.21.15.28:0/2241231973 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cf400b560 con 0x7f1d0006b6e0 2024-04-23T14:42:27.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.105+0000 7f1cfe000700 1 -- 172.21.15.28:0/2241231973 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cf4003a80 con 0x7f1d0006b6e0 2024-04-23T14:42:27.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.106+0000 7f1d08200700 1 -- 172.21.15.28:0/2241231973 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d001a8070 con 0x7f1d0006b6e0 2024-04-23T14:42:27.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.106+0000 7f1d08200700 1 -- 172.21.15.28:0/2241231973 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1d00101760 con 0x7f1d0006b6e0 2024-04-23T14:42:27.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.109+0000 7f1cfe000700 1 -- 172.21.15.28:0/2241231973 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1cf400b710 con 0x7f1d0006b6e0 2024-04-23T14:42:27.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.110+0000 7f1cfe000700 1 --2- 172.21.15.28:0/2241231973 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f1ce80a3cd0 0x7f1ce80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:27.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.110+0000 7f1cfe000700 1 -- 172.21.15.28:0/2241231973 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f1cf40d1830 con 0x7f1d0006b6e0 2024-04-23T14:42:27.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.110+0000 7f1d04e00700 1 --2- 172.21.15.28:0/2241231973 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f1ce80a3cd0 0x7f1ce80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:27.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.110+0000 7f1cfe000700 1 --2- 172.21.15.28:0/2241231973 >> [v2:172.21.15.105:6808/928755894,v1:172.21.15.105:6809/928755894] conn(0x7f1ce80ad4b0 0x7f1ce80af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:27.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.110+0000 7f1cfe000700 1 -- 172.21.15.28:0/2241231973 --> [v2:172.21.15.105:6808/928755894,v1:172.21.15.105:6809/928755894] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1cf4010aa0 con 0x7f1ce80ad4b0 2024-04-23T14:42:27.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.110+0000 7f1cfe000700 1 -- 172.21.15.28:0/2241231973 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f1cf40d1be0 con 0x7f1d0006b6e0 2024-04-23T14:42:27.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.111+0000 7f1d06200700 1 --2- 172.21.15.28:0/2241231973 >> [v2:172.21.15.105:6808/928755894,v1:172.21.15.105:6809/928755894] conn(0x7f1ce80ad4b0 0x7f1ce80af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:27.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.111+0000 7f1d04e00700 1 --2- 172.21.15.28:0/2241231973 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f1ce80a3cd0 0x7f1ce80a6190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1d00078fe0 tx=0x7f1cec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:27.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.111+0000 7f1d06200700 1 --2- 172.21.15.28:0/2241231973 >> [v2:172.21.15.105:6808/928755894,v1:172.21.15.105:6809/928755894] conn(0x7f1ce80ad4b0 0x7f1ce80af8d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:27.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.112+0000 7f1cfe000700 1 -- 172.21.15.28:0/2241231973 <== osd.5 v2:172.21.15.105:6808/928755894 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1cf4010aa0 con 0x7f1ce80ad4b0 2024-04-23T14:42:27.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.134+0000 7f1d08200700 1 -- 172.21.15.28:0/2241231973 --> [v2:172.21.15.105:6808/928755894,v1:172.21.15.105:6809/928755894] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1d00102090 con 0x7f1ce80ad4b0 2024-04-23T14:42:27.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.134+0000 7f1cfe000700 1 -- 172.21.15.28:0/2241231973 <== osd.5 v2:172.21.15.105:6808/928755894 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1d00102090 con 0x7f1ce80ad4b0 2024-04-23T14:42:27.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.135+0000 7f1d08200700 1 -- 172.21.15.28:0/2241231973 >> [v2:172.21.15.105:6808/928755894,v1:172.21.15.105:6809/928755894] conn(0x7f1ce80ad4b0 msgr2=0x7f1ce80af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:27.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.135+0000 7f1d08200700 1 --2- 172.21.15.28:0/2241231973 >> [v2:172.21.15.105:6808/928755894,v1:172.21.15.105:6809/928755894] conn(0x7f1ce80ad4b0 0x7f1ce80af8d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:27.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.135+0000 7f1d08200700 1 -- 172.21.15.28:0/2241231973 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f1ce80a3cd0 msgr2=0x7f1ce80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:27.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.135+0000 7f1d08200700 1 --2- 172.21.15.28:0/2241231973 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f1ce80a3cd0 0x7f1ce80a6190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1d00078fe0 tx=0x7f1cec009260 comp rx=0 tx=0).stop 2024-04-23T14:42:27.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.135+0000 7f1d08200700 1 -- 172.21.15.28:0/2241231973 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1d0006b6e0 msgr2=0x7f1d0019b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:27.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.135+0000 7f1d08200700 1 --2- 172.21.15.28:0/2241231973 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1d0006b6e0 0x7f1d0019b4f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1cf400c010 tx=0x7f1cf400fa70 comp rx=0 tx=0).stop 2024-04-23T14:42:27.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.135+0000 7f1d06200700 1 -- 172.21.15.28:0/2241231973 reap_dead start 2024-04-23T14:42:27.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.135+0000 7f1d08200700 1 -- 172.21.15.28:0/2241231973 shutdown_connections 2024-04-23T14:42:27.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.135+0000 7f1d08200700 1 -- 172.21.15.28:0/2241231973 >> 172.21.15.28:0/2241231973 conn(0x7f1d00078750 msgr2=0x7f1d00100d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:27.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.136+0000 7f1d08200700 1 -- 172.21.15.28:0/2241231973 shutdown_connections 2024-04-23T14:42:27.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:27.136+0000 7f1d08200700 1 -- 172.21.15.28:0/2241231973 wait complete. 2024-04-23T14:42:27.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:27 smithi038 ceph-mon[32348]: pgmap v337: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:42:27.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:27 smithi028 ceph-mon[27810]: pgmap v337: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:42:28.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:27 smithi105 ceph-mon[32555]: pgmap v337: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:42:28.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.134+0000 7fa82ea00700 1 -- 172.21.15.28:0/1117352881 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa82806b1a0 msgr2=0x7fa82810f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:28.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.134+0000 7fa82ea00700 1 --2- 172.21.15.28:0/1117352881 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa82806b1a0 0x7fa82810f9b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa82000b0a0 tx=0x7fa82000b3b0 comp rx=0 tx=0).stop 2024-04-23T14:42:28.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.134+0000 7fa82ea00700 1 -- 172.21.15.28:0/1117352881 shutdown_connections 2024-04-23T14:42:28.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.134+0000 7fa82ea00700 1 --2- 172.21.15.28:0/1117352881 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa82806b1a0 0x7fa82810f9b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:28.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.134+0000 7fa82ea00700 1 --2- 172.21.15.28:0/1117352881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa828108b70 0x7fa828108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:28.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.134+0000 7fa82ea00700 1 --2- 172.21.15.28:0/1117352881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa82806a800 0x7fa82806ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:28.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.134+0000 7fa82ea00700 1 -- 172.21.15.28:0/1117352881 >> 172.21.15.28:0/1117352881 conn(0x7fa8280789a0 msgr2=0x7fa828078db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:28.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.134+0000 7fa82ea00700 1 -- 172.21.15.28:0/1117352881 shutdown_connections 2024-04-23T14:42:28.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.134+0000 7fa82ea00700 1 -- 172.21.15.28:0/1117352881 wait complete. 2024-04-23T14:42:28.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.135+0000 7fa82ea00700 1 Processor -- start 2024-04-23T14:42:28.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.135+0000 7fa82ea00700 1 -- start start 2024-04-23T14:42:28.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.136+0000 7fa82ea00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa82806a800 0x7fa82819b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:28.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.136+0000 7fa82ea00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa82806b1a0 0x7fa82819ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:28.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.136+0000 7fa82ea00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa828108b70 0x7fa82819fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:28.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.136+0000 7fa82ea00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa828112e60 con 0x7fa82806b1a0 2024-04-23T14:42:28.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.136+0000 7fa82ea00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa828112ce0 con 0x7fa828108b70 2024-04-23T14:42:28.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.136+0000 7fa82ea00700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa828112fe0 con 0x7fa82806a800 2024-04-23T14:42:28.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.136+0000 7fa827400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa82806b1a0 0x7fa82819ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:28.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.136+0000 7fa827400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa82806b1a0 0x7fa82819ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57554/0 (socket says 172.21.15.28:57554) 2024-04-23T14:42:28.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.136+0000 7fa827400700 1 -- 172.21.15.28:0/2308714813 learned_addr learned my addr 172.21.15.28:0/2308714813 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:42:28.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.136+0000 7fa82ca00700 1 --2- 172.21.15.28:0/2308714813 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa828108b70 0x7fa82819fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:28.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.137+0000 7fa827e00700 1 --2- 172.21.15.28:0/2308714813 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa82806a800 0x7fa82819b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:28.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.137+0000 7fa82ca00700 1 -- 172.21.15.28:0/2308714813 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa82806a800 msgr2=0x7fa82819b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:28.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.137+0000 7fa82ca00700 1 --2- 172.21.15.28:0/2308714813 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa82806a800 0x7fa82819b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:28.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.137+0000 7fa82ca00700 1 -- 172.21.15.28:0/2308714813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa82806b1a0 msgr2=0x7fa82819ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:28.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.137+0000 7fa82ca00700 1 --2- 172.21.15.28:0/2308714813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa82806b1a0 0x7fa82819ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:28.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.137+0000 7fa82ca00700 1 -- 172.21.15.28:0/2308714813 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8281a04b0 con 0x7fa828108b70 2024-04-23T14:42:28.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.137+0000 7fa827400700 1 --2- 172.21.15.28:0/2308714813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa82806b1a0 0x7fa82819ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:42:28.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.137+0000 7fa82ca00700 1 --2- 172.21.15.28:0/2308714813 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa828108b70 0x7fa82819fd70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa82000b070 tx=0x7fa820004c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:28.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.138+0000 7fa824c00700 1 -- 172.21.15.28:0/2308714813 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa820021070 con 0x7fa828108b70 2024-04-23T14:42:28.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.138+0000 7fa824c00700 1 -- 172.21.15.28:0/2308714813 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa820007b80 con 0x7fa828108b70 2024-04-23T14:42:28.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.138+0000 7fa824c00700 1 -- 172.21.15.28:0/2308714813 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa820005040 con 0x7fa828108b70 2024-04-23T14:42:28.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.138+0000 7fa82ea00700 1 -- 172.21.15.28:0/2308714813 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8281a0740 con 0x7fa828108b70 2024-04-23T14:42:28.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.138+0000 7fa82ea00700 1 -- 172.21.15.28:0/2308714813 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8281a8020 con 0x7fa828108b70 2024-04-23T14:42:28.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.139+0000 7fa82ea00700 1 -- 172.21.15.28:0/2308714813 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa828063280 con 0x7fa828108b70 2024-04-23T14:42:28.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.141+0000 7fa824c00700 1 -- 172.21.15.28:0/2308714813 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa820024070 con 0x7fa828108b70 2024-04-23T14:42:28.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.143+0000 7fa824c00700 1 --2- 172.21.15.28:0/2308714813 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fa80c0a3cd0 0x7fa80c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:28.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.143+0000 7fa824c00700 1 -- 172.21.15.28:0/2308714813 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fa82000f020 con 0x7fa828108b70 2024-04-23T14:42:28.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.143+0000 7fa827e00700 1 --2- 172.21.15.28:0/2308714813 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fa80c0a3cd0 0x7fa80c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:28.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.143+0000 7fa824c00700 1 --2- 172.21.15.28:0/2308714813 >> [v2:172.21.15.28:6802/2993445336,v1:172.21.15.28:6803/2993445336] conn(0x7fa80c0ad380 0x7fa80c0af7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:28.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.143+0000 7fa824c00700 1 -- 172.21.15.28:0/2308714813 --> [v2:172.21.15.28:6802/2993445336,v1:172.21.15.28:6803/2993445336] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa820027e50 con 0x7fa80c0ad380 2024-04-23T14:42:28.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.143+0000 7fa824c00700 1 -- 172.21.15.28:0/2308714813 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fa8200d1e00 con 0x7fa828108b70 2024-04-23T14:42:28.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.144+0000 7fa827400700 1 --2- 172.21.15.28:0/2308714813 >> [v2:172.21.15.28:6802/2993445336,v1:172.21.15.28:6803/2993445336] conn(0x7fa80c0ad380 0x7fa80c0af7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:28.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.144+0000 7fa827e00700 1 --2- 172.21.15.28:0/2308714813 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fa80c0a3cd0 0x7fa80c0a6190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa818009be0 tx=0x7fa818009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:28.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.144+0000 7fa827400700 1 --2- 172.21.15.28:0/2308714813 >> [v2:172.21.15.28:6802/2993445336,v1:172.21.15.28:6803/2993445336] conn(0x7fa80c0ad380 0x7fa80c0af7a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:28.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.145+0000 7fa824c00700 1 -- 172.21.15.28:0/2308714813 <== osd.0 v2:172.21.15.28:6802/2993445336 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa820027e50 con 0x7fa80c0ad380 2024-04-23T14:42:28.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.163+0000 7fa82ea00700 1 -- 172.21.15.28:0/2308714813 --> [v2:172.21.15.28:6802/2993445336,v1:172.21.15.28:6803/2993445336] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa828063700 con 0x7fa80c0ad380 2024-04-23T14:42:28.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.164+0000 7fa824c00700 1 -- 172.21.15.28:0/2308714813 <== osd.0 v2:172.21.15.28:6802/2993445336 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa828063700 con 0x7fa80c0ad380 2024-04-23T14:42:28.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.164+0000 7fa82ea00700 1 -- 172.21.15.28:0/2308714813 >> [v2:172.21.15.28:6802/2993445336,v1:172.21.15.28:6803/2993445336] conn(0x7fa80c0ad380 msgr2=0x7fa80c0af7a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:28.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.164+0000 7fa82ea00700 1 --2- 172.21.15.28:0/2308714813 >> [v2:172.21.15.28:6802/2993445336,v1:172.21.15.28:6803/2993445336] conn(0x7fa80c0ad380 0x7fa80c0af7a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:28.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.164+0000 7fa82ea00700 1 -- 172.21.15.28:0/2308714813 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fa80c0a3cd0 msgr2=0x7fa80c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:28.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.165+0000 7fa82ea00700 1 --2- 172.21.15.28:0/2308714813 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fa80c0a3cd0 0x7fa80c0a6190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa818009be0 tx=0x7fa818009260 comp rx=0 tx=0).stop 2024-04-23T14:42:28.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.165+0000 7fa82ea00700 1 -- 172.21.15.28:0/2308714813 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa828108b70 msgr2=0x7fa82819fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:28.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.165+0000 7fa82ea00700 1 --2- 172.21.15.28:0/2308714813 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa828108b70 0x7fa82819fd70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa82000b070 tx=0x7fa820004c80 comp rx=0 tx=0).stop 2024-04-23T14:42:28.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.165+0000 7fa82ca00700 1 -- 172.21.15.28:0/2308714813 reap_dead start 2024-04-23T14:42:28.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.165+0000 7fa82ea00700 1 -- 172.21.15.28:0/2308714813 shutdown_connections 2024-04-23T14:42:28.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.165+0000 7fa82ea00700 1 -- 172.21.15.28:0/2308714813 >> 172.21.15.28:0/2308714813 conn(0x7fa8280789a0 msgr2=0x7fa82810df50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:28.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.165+0000 7fa82ea00700 1 -- 172.21.15.28:0/2308714813 shutdown_connections 2024-04-23T14:42:28.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:28.165+0000 7fa82ea00700 1 -- 172.21.15.28:0/2308714813 wait complete. 2024-04-23T14:42:28.429 INFO:teuthology.orchestra.run.smithi028.stdout:163208757305 2024-04-23T14:42:28.430 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-23T14:42:28.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.504+0000 7f5787200700 1 -- 172.21.15.28:0/967143173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f578010f0e0 msgr2=0x7f578010f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:28.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.504+0000 7f5787200700 1 --2- 172.21.15.28:0/967143173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f578010f0e0 0x7f578010f4c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5774009990 tx=0x7f5774009ca0 comp rx=0 tx=0).stop 2024-04-23T14:42:28.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.504+0000 7f5787200700 1 -- 172.21.15.28:0/967143173 shutdown_connections 2024-04-23T14:42:28.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.504+0000 7f5787200700 1 --2- 172.21.15.28:0/967143173 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57801099f0 0x7f5780115e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:28.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.504+0000 7f5787200700 1 --2- 172.21.15.28:0/967143173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f578010f0e0 0x7f578010f4c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:28.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.504+0000 7f5787200700 1 --2- 172.21.15.28:0/967143173 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5780109030 0x7f57801094b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:28.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.504+0000 7f5787200700 1 -- 172.21.15.28:0/967143173 >> 172.21.15.28:0/967143173 conn(0x7f5780078880 msgr2=0x7f5780107910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:28.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.504+0000 7f5787200700 1 -- 172.21.15.28:0/967143173 shutdown_connections 2024-04-23T14:42:28.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.506+0000 7f5787200700 1 -- 172.21.15.28:0/967143173 wait complete. 2024-04-23T14:42:28.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.507+0000 7f5787200700 1 Processor -- start 2024-04-23T14:42:28.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.507+0000 7f5787200700 1 -- start start 2024-04-23T14:42:28.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.507+0000 7f5787200700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5780109030 0x7f57801a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:28.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.507+0000 7f5787200700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57801099f0 0x7f57801a5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:28.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.507+0000 7f5787200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f578010f0e0 0x7f578019f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:28.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.507+0000 7f5787200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5780119340 con 0x7f578010f0e0 2024-04-23T14:42:28.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.507+0000 7f5787200700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f57801191c0 con 0x7f5780109030 2024-04-23T14:42:28.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.507+0000 7f5787200700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f57801194c0 con 0x7f57801099f0 2024-04-23T14:42:28.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.508+0000 7f5785400700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57801099f0 0x7f57801a5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:28.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.508+0000 7f5786800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f578010f0e0 0x7f578019f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:28.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.508+0000 7f5785400700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57801099f0 0x7f57801a5980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:52512/0 (socket says 172.21.15.28:52512) 2024-04-23T14:42:28.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.508+0000 7f5785400700 1 -- 172.21.15.28:0/665204176 learned_addr learned my addr 172.21.15.28:0/665204176 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:42:28.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.508+0000 7f5785e00700 1 --2- 172.21.15.28:0/665204176 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5780109030 0x7f57801a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:28.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.508+0000 7f5786800700 1 -- 172.21.15.28:0/665204176 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57801099f0 msgr2=0x7f57801a5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:28.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.508+0000 7f5786800700 1 --2- 172.21.15.28:0/665204176 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57801099f0 0x7f57801a5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:28.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.508+0000 7f5786800700 1 -- 172.21.15.28:0/665204176 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5780109030 msgr2=0x7f57801a5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:28.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.508+0000 7f5786800700 1 --2- 172.21.15.28:0/665204176 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5780109030 0x7f57801a5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:28.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.508+0000 7f5786800700 1 -- 172.21.15.28:0/665204176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f578019fd60 con 0x7f578010f0e0 2024-04-23T14:42:28.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.508+0000 7f5786800700 1 --2- 172.21.15.28:0/665204176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f578010f0e0 0x7f578019f600 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f577c00efb0 tx=0x7f577c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:28.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.509+0000 7f577aa00700 1 -- 172.21.15.28:0/665204176 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f577c003a00 con 0x7f578010f0e0 2024-04-23T14:42:28.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.509+0000 7f577aa00700 1 -- 172.21.15.28:0/665204176 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f577c003ba0 con 0x7f578010f0e0 2024-04-23T14:42:28.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.509+0000 7f577aa00700 1 -- 172.21.15.28:0/665204176 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f577c004850 con 0x7f578010f0e0 2024-04-23T14:42:28.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.509+0000 7f5787200700 1 -- 172.21.15.28:0/665204176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57801a0050 con 0x7f578010f0e0 2024-04-23T14:42:28.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.509+0000 7f5787200700 1 -- 172.21.15.28:0/665204176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57801ac310 con 0x7f578010f0e0 2024-04-23T14:42:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.510+0000 7f5787200700 1 -- 172.21.15.28:0/665204176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5780063240 con 0x7f578010f0e0 2024-04-23T14:42:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.512+0000 7f577aa00700 1 -- 172.21.15.28:0/665204176 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f577c012050 con 0x7f578010f0e0 2024-04-23T14:42:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.514+0000 7f577aa00700 1 --2- 172.21.15.28:0/665204176 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f57680a3cd0 0x7f57680a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.515+0000 7f577aa00700 1 -- 172.21.15.28:0/665204176 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f577c0d0250 con 0x7f578010f0e0 2024-04-23T14:42:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.515+0000 7f5785e00700 1 --2- 172.21.15.28:0/665204176 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f57680a3cd0 0x7f57680a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.515+0000 7f577aa00700 1 --2- 172.21.15.28:0/665204176 >> [v2:172.21.15.28:6818/897979441,v1:172.21.15.28:6819/897979441] conn(0x7f57680ad4b0 0x7f57680af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.515+0000 7f577aa00700 1 -- 172.21.15.28:0/665204176 --> [v2:172.21.15.28:6818/897979441,v1:172.21.15.28:6819/897979441] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f577c01ee50 con 0x7f57680ad4b0 2024-04-23T14:42:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.515+0000 7f577aa00700 1 -- 172.21.15.28:0/665204176 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f577c0d0600 con 0x7f578010f0e0 2024-04-23T14:42:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.515+0000 7f5785400700 1 --2- 172.21.15.28:0/665204176 >> [v2:172.21.15.28:6818/897979441,v1:172.21.15.28:6819/897979441] conn(0x7f57680ad4b0 0x7f57680af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.516+0000 7f5785e00700 1 --2- 172.21.15.28:0/665204176 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f57680a3cd0 0x7f57680a6190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f577400b2e0 tx=0x7f5774005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.516+0000 7f5785400700 1 --2- 172.21.15.28:0/665204176 >> [v2:172.21.15.28:6818/897979441,v1:172.21.15.28:6819/897979441] conn(0x7f57680ad4b0 0x7f57680af8d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.517+0000 7f577aa00700 1 -- 172.21.15.28:0/665204176 <== osd.6 v2:172.21.15.28:6818/897979441 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f577c01ee50 con 0x7f57680ad4b0 2024-04-23T14:42:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.536+0000 7f5787200700 1 -- 172.21.15.28:0/665204176 --> [v2:172.21.15.28:6818/897979441,v1:172.21.15.28:6819/897979441] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f57800636c0 con 0x7f57680ad4b0 2024-04-23T14:42:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.536+0000 7f577aa00700 1 -- 172.21.15.28:0/665204176 <== osd.6 v2:172.21.15.28:6818/897979441 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f57800636c0 con 0x7f57680ad4b0 2024-04-23T14:42:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.536+0000 7f5787200700 1 -- 172.21.15.28:0/665204176 >> [v2:172.21.15.28:6818/897979441,v1:172.21.15.28:6819/897979441] conn(0x7f57680ad4b0 msgr2=0x7f57680af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:28.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.536+0000 7f5787200700 1 --2- 172.21.15.28:0/665204176 >> [v2:172.21.15.28:6818/897979441,v1:172.21.15.28:6819/897979441] conn(0x7f57680ad4b0 0x7f57680af8d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:28.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.537+0000 7f5787200700 1 -- 172.21.15.28:0/665204176 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f57680a3cd0 msgr2=0x7f57680a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:28.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.537+0000 7f5787200700 1 --2- 172.21.15.28:0/665204176 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f57680a3cd0 0x7f57680a6190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f577400b2e0 tx=0x7f5774005fd0 comp rx=0 tx=0).stop 2024-04-23T14:42:28.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.537+0000 7f5787200700 1 -- 172.21.15.28:0/665204176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f578010f0e0 msgr2=0x7f578019f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:28.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.537+0000 7f5787200700 1 --2- 172.21.15.28:0/665204176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f578010f0e0 0x7f578019f600 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f577c00efb0 tx=0x7f577c00c3a0 comp rx=0 tx=0).stop 2024-04-23T14:42:28.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.537+0000 7f5786800700 1 -- 172.21.15.28:0/665204176 reap_dead start 2024-04-23T14:42:28.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.537+0000 7f5787200700 1 -- 172.21.15.28:0/665204176 shutdown_connections 2024-04-23T14:42:28.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.537+0000 7f5787200700 1 -- 172.21.15.28:0/665204176 >> 172.21.15.28:0/665204176 conn(0x7f5780078880 msgr2=0x7f5780079930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:28.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.537+0000 7f5787200700 1 -- 172.21.15.28:0/665204176 shutdown_connections 2024-04-23T14:42:28.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:23.537+0000 7f5787200700 1 -- 172.21.15.28:0/665204176 wait complete. 2024-04-23T14:42:29.267 INFO:teuthology.orchestra.run.smithi028.stdout:197568495666 2024-04-23T14:42:29.268 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-23T14:42:29.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:29 smithi038 ceph-mon[32348]: pgmap v338: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:42:29.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:29 smithi028 ceph-mon[27810]: pgmap v338: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:42:30.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:29 smithi105 ceph-mon[32555]: pgmap v338: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:42:30.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.342+0000 7fac1a000700 1 -- 172.21.15.28:0/1211116017 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fac14109d30 msgr2=0x7fac1410dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:30.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.342+0000 7fac1a000700 1 --2- 172.21.15.28:0/1211116017 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fac14109d30 0x7fac1410dd80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fac000099c0 tx=0x7fac00009cd0 comp rx=0 tx=0).stop 2024-04-23T14:42:30.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.342+0000 7fac1a000700 1 -- 172.21.15.28:0/1211116017 shutdown_connections 2024-04-23T14:42:30.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.342+0000 7fac1a000700 1 --2- 172.21.15.28:0/1211116017 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac1410e460 0x7fac14116030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:30.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.342+0000 7fac1a000700 1 --2- 172.21.15.28:0/1211116017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac14109380 0x7fac14109760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.342+0000 7fac1a000700 1 --2- 172.21.15.28:0/1211116017 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fac14109d30 0x7fac1410dd80 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.342+0000 7fac1a000700 1 -- 172.21.15.28:0/1211116017 >> 172.21.15.28:0/1211116017 conn(0x7fac140762c0 msgr2=0x7fac14107910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.343+0000 7fac1a000700 1 -- 172.21.15.28:0/1211116017 shutdown_connections 2024-04-23T14:42:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.343+0000 7fac1a000700 1 -- 172.21.15.28:0/1211116017 wait complete. 2024-04-23T14:42:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.344+0000 7fac1a000700 1 Processor -- start 2024-04-23T14:42:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.344+0000 7fac1a000700 1 -- start start 2024-04-23T14:42:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.345+0000 7fac1a000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac14109380 0x7fac141a5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.345+0000 7fac1a000700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac14109d30 0x7fac141a5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.345+0000 7fac1a000700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fac1410e460 0x7fac1419f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.345+0000 7fac1a000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fac14119420 con 0x7fac14109380 2024-04-23T14:42:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.345+0000 7fac1a000700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fac141192a0 con 0x7fac14109d30 2024-04-23T14:42:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.345+0000 7fac1a000700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fac141195a0 con 0x7fac1410e460 2024-04-23T14:42:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.345+0000 7fac13400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac14109380 0x7fac141a5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.345+0000 7fac12a00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac14109d30 0x7fac141a5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.345+0000 7fac13400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac14109380 0x7fac141a5680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57576/0 (socket says 172.21.15.28:57576) 2024-04-23T14:42:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.345+0000 7fac13400700 1 -- 172.21.15.28:0/4191057199 learned_addr learned my addr 172.21.15.28:0/4191057199 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:42:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.345+0000 7fac13e00700 1 --2- 172.21.15.28:0/4191057199 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fac1410e460 0x7fac1419f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.346+0000 7fac12a00700 1 -- 172.21.15.28:0/4191057199 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fac1410e460 msgr2=0x7fac1419f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.346+0000 7fac12a00700 1 --2- 172.21.15.28:0/4191057199 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fac1410e460 0x7fac1419f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.346+0000 7fac12a00700 1 -- 172.21.15.28:0/4191057199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac14109380 msgr2=0x7fac141a5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.346+0000 7fac12a00700 1 --2- 172.21.15.28:0/4191057199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac14109380 0x7fac141a5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.346+0000 7fac12a00700 1 -- 172.21.15.28:0/4191057199 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac141a0080 con 0x7fac14109d30 2024-04-23T14:42:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.346+0000 7fac13400700 1 --2- 172.21.15.28:0/4191057199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac14109380 0x7fac141a5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:42:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.346+0000 7fac12a00700 1 --2- 172.21.15.28:0/4191057199 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac14109d30 0x7fac141a5bc0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fac0000b580 tx=0x7fac00003a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.346+0000 7fac0be00700 1 -- 172.21.15.28:0/4191057199 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac00003680 con 0x7fac14109d30 2024-04-23T14:42:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.346+0000 7fac0be00700 1 -- 172.21.15.28:0/4191057199 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac000055a0 con 0x7fac14109d30 2024-04-23T14:42:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.346+0000 7fac1a000700 1 -- 172.21.15.28:0/4191057199 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac141a0310 con 0x7fac14109d30 2024-04-23T14:42:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.346+0000 7fac0be00700 1 -- 172.21.15.28:0/4191057199 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac0000fc60 con 0x7fac14109d30 2024-04-23T14:42:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.347+0000 7fac1a000700 1 -- 172.21.15.28:0/4191057199 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac141ac600 con 0x7fac14109d30 2024-04-23T14:42:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.347+0000 7fac1a000700 1 -- 172.21.15.28:0/4191057199 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fac140631c0 con 0x7fac14109d30 2024-04-23T14:42:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.350+0000 7fac0be00700 1 -- 172.21.15.28:0/4191057199 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fac0000f460 con 0x7fac14109d30 2024-04-23T14:42:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.351+0000 7fac0be00700 1 --2- 172.21.15.28:0/4191057199 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fabf80a3ef0 0x7fabf80a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.351+0000 7fac0be00700 1 -- 172.21.15.28:0/4191057199 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fac000d14e0 con 0x7fac14109d30 2024-04-23T14:42:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.351+0000 7fac13400700 1 --2- 172.21.15.28:0/4191057199 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fabf80a3ef0 0x7fabf80a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.352+0000 7fac0be00700 1 --2- 172.21.15.28:0/4191057199 >> [v2:172.21.15.38:6808/3772189246,v1:172.21.15.38:6809/3772189246] conn(0x7fabf80ad6d0 0x7fabf80afaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.352+0000 7fac0be00700 1 -- 172.21.15.28:0/4191057199 --> [v2:172.21.15.38:6808/3772189246,v1:172.21.15.38:6809/3772189246] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fac00003680 con 0x7fabf80ad6d0 2024-04-23T14:42:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.352+0000 7fac0be00700 1 -- 172.21.15.28:0/4191057199 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fac000d1890 con 0x7fac14109d30 2024-04-23T14:42:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.352+0000 7fac13e00700 1 --2- 172.21.15.28:0/4191057199 >> [v2:172.21.15.38:6808/3772189246,v1:172.21.15.38:6809/3772189246] conn(0x7fabf80ad6d0 0x7fabf80afaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.352+0000 7fac13400700 1 --2- 172.21.15.28:0/4191057199 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fabf80a3ef0 0x7fabf80a63b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fac04011550 tx=0x7fac04009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.353+0000 7fac13e00700 1 --2- 172.21.15.28:0/4191057199 >> [v2:172.21.15.38:6808/3772189246,v1:172.21.15.38:6809/3772189246] conn(0x7fabf80ad6d0 0x7fabf80afaf0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.354+0000 7fac0be00700 1 -- 172.21.15.28:0/4191057199 <== osd.4 v2:172.21.15.38:6808/3772189246 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fac00003680 con 0x7fabf80ad6d0 2024-04-23T14:42:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.372+0000 7fac1a000700 1 -- 172.21.15.28:0/4191057199 --> [v2:172.21.15.38:6808/3772189246,v1:172.21.15.38:6809/3772189246] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fac140635c0 con 0x7fabf80ad6d0 2024-04-23T14:42:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.373+0000 7fac0be00700 1 -- 172.21.15.28:0/4191057199 <== osd.4 v2:172.21.15.38:6808/3772189246 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fac140635c0 con 0x7fabf80ad6d0 2024-04-23T14:42:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.373+0000 7fac1a000700 1 -- 172.21.15.28:0/4191057199 >> [v2:172.21.15.38:6808/3772189246,v1:172.21.15.38:6809/3772189246] conn(0x7fabf80ad6d0 msgr2=0x7fabf80afaf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.373+0000 7fac1a000700 1 --2- 172.21.15.28:0/4191057199 >> [v2:172.21.15.38:6808/3772189246,v1:172.21.15.38:6809/3772189246] conn(0x7fabf80ad6d0 0x7fabf80afaf0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.374+0000 7fac1a000700 1 -- 172.21.15.28:0/4191057199 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fabf80a3ef0 msgr2=0x7fabf80a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.374+0000 7fac1a000700 1 --2- 172.21.15.28:0/4191057199 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fabf80a3ef0 0x7fabf80a63b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fac04011550 tx=0x7fac04009260 comp rx=0 tx=0).stop 2024-04-23T14:42:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.374+0000 7fac1a000700 1 -- 172.21.15.28:0/4191057199 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac14109d30 msgr2=0x7fac141a5bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:30.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.374+0000 7fac1a000700 1 --2- 172.21.15.28:0/4191057199 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac14109d30 0x7fac141a5bc0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fac0000b580 tx=0x7fac00003a90 comp rx=0 tx=0).stop 2024-04-23T14:42:30.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.374+0000 7fac13e00700 1 -- 172.21.15.28:0/4191057199 reap_dead start 2024-04-23T14:42:30.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.374+0000 7fac1a000700 1 -- 172.21.15.28:0/4191057199 shutdown_connections 2024-04-23T14:42:30.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.374+0000 7fac1a000700 1 -- 172.21.15.28:0/4191057199 >> 172.21.15.28:0/4191057199 conn(0x7fac140762c0 msgr2=0x7fac14076920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:30.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.374+0000 7fac1a000700 1 -- 172.21.15.28:0/4191057199 shutdown_connections 2024-04-23T14:42:30.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:30.374+0000 7fac1a000700 1 -- 172.21.15.28:0/4191057199 wait complete. 2024-04-23T14:42:30.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:30 smithi028 ceph-mon[27810]: pgmap v339: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-23T14:42:30.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:30 smithi038 ceph-mon[32348]: pgmap v339: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-23T14:42:31.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:30 smithi105 ceph-mon[32555]: pgmap v339: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-23T14:42:31.239 INFO:teuthology.orchestra.run.smithi028.stdout:343597383702 2024-04-23T14:42:31.239 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-23T14:42:31.734 INFO:teuthology.orchestra.run.smithi028.stdout:34359738454 2024-04-23T14:42:31.735 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-23T14:42:32.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.188+0000 7fbac3000700 1 -- 172.21.15.28:0/3883867775 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbabc1019f0 msgr2=0x7fbabc101e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:32.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.188+0000 7fbac3000700 1 --2- 172.21.15.28:0/3883867775 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbabc1019f0 0x7fbabc101e70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbaa80098f0 tx=0x7fbaa8009c00 comp rx=0 tx=0).stop 2024-04-23T14:42:32.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.188+0000 7fbac3000700 1 -- 172.21.15.28:0/3883867775 shutdown_connections 2024-04-23T14:42:32.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.188+0000 7fbac3000700 1 --2- 172.21.15.28:0/3883867775 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbabc1023b0 0x7fbabc10e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:32.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.188+0000 7fbac3000700 1 --2- 172.21.15.28:0/3883867775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbabc107a40 0x7fbabc107e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:32.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.188+0000 7fbac3000700 1 --2- 172.21.15.28:0/3883867775 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbabc1019f0 0x7fbabc101e70 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:32.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.188+0000 7fbac3000700 1 -- 172.21.15.28:0/3883867775 >> 172.21.15.28:0/3883867775 conn(0x7fbabc0789a0 msgr2=0x7fbabc078db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:32.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.189+0000 7fbac3000700 1 -- 172.21.15.28:0/3883867775 shutdown_connections 2024-04-23T14:42:32.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.189+0000 7fbac3000700 1 -- 172.21.15.28:0/3883867775 wait complete. 2024-04-23T14:42:32.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.190+0000 7fbac3000700 1 Processor -- start 2024-04-23T14:42:32.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.190+0000 7fbac3000700 1 -- start start 2024-04-23T14:42:32.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.190+0000 7fbac3000700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbabc1019f0 0x7fbabc199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:32.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.190+0000 7fbac3000700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbabc1023b0 0x7fbabc199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:32.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.190+0000 7fbac3000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbabc107a40 0x7fbabc19dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:32.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.190+0000 7fbac3000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbabc111d30 con 0x7fbabc107a40 2024-04-23T14:42:32.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.190+0000 7fbac3000700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbabc111bb0 con 0x7fbabc1019f0 2024-04-23T14:42:32.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.190+0000 7fbac3000700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fbabc111eb0 con 0x7fbabc1023b0 2024-04-23T14:42:32.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.191+0000 7fbabbe00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbabc1019f0 0x7fbabc199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:32.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.191+0000 7fbabbe00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbabc1019f0 0x7fbabc199310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:50212/0 (socket says 172.21.15.28:50212) 2024-04-23T14:42:32.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.191+0000 7fbabbe00700 1 -- 172.21.15.28:0/4108796162 learned_addr learned my addr 172.21.15.28:0/4108796162 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:42:32.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.191+0000 7fbac1000700 1 --2- 172.21.15.28:0/4108796162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbabc107a40 0x7fbabc19dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:32.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.191+0000 7fbabb400700 1 --2- 172.21.15.28:0/4108796162 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbabc1023b0 0x7fbabc199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:32.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.191+0000 7fbabb400700 1 -- 172.21.15.28:0/4108796162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbabc1019f0 msgr2=0x7fbabc199310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:32.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.191+0000 7fbabb400700 1 --2- 172.21.15.28:0/4108796162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbabc1019f0 0x7fbabc199310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:32.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.191+0000 7fbabb400700 1 -- 172.21.15.28:0/4108796162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbabc107a40 msgr2=0x7fbabc19dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:32.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.191+0000 7fbabb400700 1 --2- 172.21.15.28:0/4108796162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbabc107a40 0x7fbabc19dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:32.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.191+0000 7fbabb400700 1 -- 172.21.15.28:0/4108796162 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbabc19e320 con 0x7fbabc1023b0 2024-04-23T14:42:32.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.191+0000 7fbac1000700 1 --2- 172.21.15.28:0/4108796162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbabc107a40 0x7fbabc19dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:42:32.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.192+0000 7fbabb400700 1 --2- 172.21.15.28:0/4108796162 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbabc1023b0 0x7fbabc199850 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8000c00 tx=0x7fbaa800fb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:32.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.192+0000 7fbab8c00700 1 -- 172.21.15.28:0/4108796162 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaa8005140 con 0x7fbabc1023b0 2024-04-23T14:42:32.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.192+0000 7fbab8c00700 1 -- 172.21.15.28:0/4108796162 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbaa800b560 con 0x7fbabc1023b0 2024-04-23T14:42:32.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.192+0000 7fbab8c00700 1 -- 172.21.15.28:0/4108796162 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaa8003b40 con 0x7fbabc1023b0 2024-04-23T14:42:32.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.192+0000 7fbac3000700 1 -- 172.21.15.28:0/4108796162 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbabc19e5b0 con 0x7fbabc1023b0 2024-04-23T14:42:32.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.192+0000 7fbac3000700 1 -- 172.21.15.28:0/4108796162 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbabc1034d0 con 0x7fbabc1023b0 2024-04-23T14:42:32.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.193+0000 7fbac3000700 1 -- 172.21.15.28:0/4108796162 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbabc19ea40 con 0x7fbabc1023b0 2024-04-23T14:42:32.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.197+0000 7fbab8c00700 1 -- 172.21.15.28:0/4108796162 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fbaa800baa0 con 0x7fbabc1023b0 2024-04-23T14:42:32.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.199+0000 7fbab8c00700 1 --2- 172.21.15.28:0/4108796162 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fbaa00a3d50 0x7fbaa00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:32.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.199+0000 7fbab8c00700 1 -- 172.21.15.28:0/4108796162 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fbaa80d17d0 con 0x7fbabc1023b0 2024-04-23T14:42:32.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.200+0000 7fbabbe00700 1 --2- 172.21.15.28:0/4108796162 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fbaa00a3d50 0x7fbaa00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:32.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.200+0000 7fbab8c00700 1 --2- 172.21.15.28:0/4108796162 >> [v2:172.21.15.28:6826/1292942516,v1:172.21.15.28:6827/1292942516] conn(0x7fbaa00ad400 0x7fbaa00af820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:32.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.200+0000 7fbab8c00700 1 -- 172.21.15.28:0/4108796162 --> [v2:172.21.15.28:6826/1292942516,v1:172.21.15.28:6827/1292942516] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbaa800b0b0 con 0x7fbaa00ad400 2024-04-23T14:42:32.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.200+0000 7fbab8c00700 1 -- 172.21.15.28:0/4108796162 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fbaa80d1b80 con 0x7fbabc1023b0 2024-04-23T14:42:32.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.200+0000 7fbac1000700 1 --2- 172.21.15.28:0/4108796162 >> [v2:172.21.15.28:6826/1292942516,v1:172.21.15.28:6827/1292942516] conn(0x7fbaa00ad400 0x7fbaa00af820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:32.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.200+0000 7fbabbe00700 1 --2- 172.21.15.28:0/4108796162 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fbaa00a3d50 0x7fbaa00a6210 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fbaac009750 tx=0x7fbaac009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:32.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.201+0000 7fbac1000700 1 --2- 172.21.15.28:0/4108796162 >> [v2:172.21.15.28:6826/1292942516,v1:172.21.15.28:6827/1292942516] conn(0x7fbaa00ad400 0x7fbaa00af820 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:32.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.202+0000 7fbab8c00700 1 -- 172.21.15.28:0/4108796162 <== osd.9 v2:172.21.15.28:6826/1292942516 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbaa800b0b0 con 0x7fbaa00ad400 2024-04-23T14:42:32.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.220+0000 7fbac3000700 1 -- 172.21.15.28:0/4108796162 --> [v2:172.21.15.28:6826/1292942516,v1:172.21.15.28:6827/1292942516] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbabc06ad70 con 0x7fbaa00ad400 2024-04-23T14:42:32.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.220+0000 7fbab8c00700 1 -- 172.21.15.28:0/4108796162 <== osd.9 v2:172.21.15.28:6826/1292942516 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fbabc06ad70 con 0x7fbaa00ad400 2024-04-23T14:42:32.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.221+0000 7fbac3000700 1 -- 172.21.15.28:0/4108796162 >> [v2:172.21.15.28:6826/1292942516,v1:172.21.15.28:6827/1292942516] conn(0x7fbaa00ad400 msgr2=0x7fbaa00af820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:32.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.221+0000 7fbac3000700 1 --2- 172.21.15.28:0/4108796162 >> [v2:172.21.15.28:6826/1292942516,v1:172.21.15.28:6827/1292942516] conn(0x7fbaa00ad400 0x7fbaa00af820 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:32.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.221+0000 7fbac3000700 1 -- 172.21.15.28:0/4108796162 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fbaa00a3d50 msgr2=0x7fbaa00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:32.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.221+0000 7fbac3000700 1 --2- 172.21.15.28:0/4108796162 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fbaa00a3d50 0x7fbaa00a6210 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fbaac009750 tx=0x7fbaac009310 comp rx=0 tx=0).stop 2024-04-23T14:42:32.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.221+0000 7fbac3000700 1 -- 172.21.15.28:0/4108796162 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbabc1023b0 msgr2=0x7fbabc199850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:32.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.221+0000 7fbac3000700 1 --2- 172.21.15.28:0/4108796162 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbabc1023b0 0x7fbabc199850 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8000c00 tx=0x7fbaa800fb40 comp rx=0 tx=0).stop 2024-04-23T14:42:32.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.221+0000 7fbac1000700 1 -- 172.21.15.28:0/4108796162 reap_dead start 2024-04-23T14:42:32.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.221+0000 7fbac3000700 1 -- 172.21.15.28:0/4108796162 shutdown_connections 2024-04-23T14:42:32.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.221+0000 7fbac3000700 1 -- 172.21.15.28:0/4108796162 >> 172.21.15.28:0/4108796162 conn(0x7fbabc0789a0 msgr2=0x7fbabc105820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:32.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.221+0000 7fbac3000700 1 -- 172.21.15.28:0/4108796162 shutdown_connections 2024-04-23T14:42:32.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:32.221+0000 7fbac3000700 1 -- 172.21.15.28:0/4108796162 wait complete. 2024-04-23T14:42:33.110 INFO:teuthology.orchestra.run.smithi028.stdout:141733920832 2024-04-23T14:42:33.111 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-23T14:42:33.416 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:42:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:42:33.445 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:42:33.506 INFO:teuthology.orchestra.run.smithi028.stdout:287762808867 2024-04-23T14:42:33.506 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-23T14:42:33.569 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:42:33.673 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:42:33.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:33 smithi028 ceph-mon[27810]: pgmap v340: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:42:33.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:33 smithi105 ceph-mon[32555]: pgmap v340: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:42:33.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:33 smithi038 ceph-mon[32348]: pgmap v340: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:42:35.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:35 smithi028 ceph-mon[27810]: pgmap v341: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:42:35.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:35 smithi105 ceph-mon[32555]: pgmap v341: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:42:35.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:35 smithi038 ceph-mon[32348]: pgmap v341: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:42:37.744 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:37 smithi028 ceph-mon[27810]: pgmap v342: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:42:37.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:37 smithi105 ceph-mon[32555]: pgmap v342: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:42:37.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:37 smithi038 ceph-mon[32348]: pgmap v342: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:42:38.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.078+0000 7f99d0a00700 1 -- 172.21.15.28:0/1518214305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99cc105830 msgr2=0x7f99cc109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:38.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.078+0000 7f99d0a00700 1 --2- 172.21.15.28:0/1518214305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99cc105830 0x7f99cc109880 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f99b400b0d0 tx=0x7f99b400b3e0 comp rx=0 tx=0).stop 2024-04-23T14:42:38.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.078+0000 7f99d0a00700 1 -- 172.21.15.28:0/1518214305 shutdown_connections 2024-04-23T14:42:38.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.078+0000 7f99d0a00700 1 --2- 172.21.15.28:0/1518214305 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f99cc104e80 0x7f99cc105260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:38.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.078+0000 7f99d0a00700 1 --2- 172.21.15.28:0/1518214305 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f99cc109f60 0x7f99cc111b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:38.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.078+0000 7f99d0a00700 1 --2- 172.21.15.28:0/1518214305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99cc105830 0x7f99cc109880 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:38.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.078+0000 7f99d0a00700 1 -- 172.21.15.28:0/1518214305 >> 172.21.15.28:0/1518214305 conn(0x7f99cc100710 msgr2=0x7f99cc102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:38.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.078+0000 7f99d0a00700 1 -- 172.21.15.28:0/1518214305 shutdown_connections 2024-04-23T14:42:38.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.078+0000 7f99d0a00700 1 -- 172.21.15.28:0/1518214305 wait complete. 2024-04-23T14:42:38.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.079+0000 7f99d0a00700 1 Processor -- start 2024-04-23T14:42:38.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.079+0000 7f99d0a00700 1 -- start start 2024-04-23T14:42:38.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.080+0000 7f99d0a00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f99cc104e80 0x7f99cc19b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:38.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.080+0000 7f99d0a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99cc105830 0x7f99cc19b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:38.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.080+0000 7f99d0a00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f99cc109f60 0x7f99cc19fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:38.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.080+0000 7f99d0a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f99cc114f90 con 0x7f99cc105830 2024-04-23T14:42:38.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.080+0000 7f99d0a00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f99cc114e10 con 0x7f99cc109f60 2024-04-23T14:42:38.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.080+0000 7f99d0a00700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f99cc115110 con 0x7f99cc104e80 2024-04-23T14:42:38.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.080+0000 7f99c2a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99cc105830 0x7f99cc19b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:38.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.080+0000 7f99c2a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99cc105830 0x7f99cc19b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43782/0 (socket says 172.21.15.28:43782) 2024-04-23T14:42:38.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.080+0000 7f99c2a00700 1 -- 172.21.15.28:0/1196151645 learned_addr learned my addr 172.21.15.28:0/1196151645 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:42:38.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.080+0000 7f99cb400700 1 --2- 172.21.15.28:0/1196151645 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f99cc104e80 0x7f99cc19b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:38.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.080+0000 7f99cbe00700 1 --2- 172.21.15.28:0/1196151645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f99cc109f60 0x7f99cc19fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:38.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.081+0000 7f99c2a00700 1 -- 172.21.15.28:0/1196151645 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f99cc104e80 msgr2=0x7f99cc19b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:38.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.081+0000 7f99c2a00700 1 --2- 172.21.15.28:0/1196151645 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f99cc104e80 0x7f99cc19b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:38.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.081+0000 7f99c2a00700 1 -- 172.21.15.28:0/1196151645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f99cc109f60 msgr2=0x7f99cc19fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:38.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.081+0000 7f99c2a00700 1 --2- 172.21.15.28:0/1196151645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f99cc109f60 0x7f99cc19fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:38.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.081+0000 7f99c2a00700 1 -- 172.21.15.28:0/1196151645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99cc1a0410 con 0x7f99cc105830 2024-04-23T14:42:38.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.081+0000 7f99c2a00700 1 --2- 172.21.15.28:0/1196151645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99cc105830 0x7f99cc19b940 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f99b4000f80 tx=0x7f99b4007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:38.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.081+0000 7f99c8c00700 1 -- 172.21.15.28:0/1196151645 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b4003c00 con 0x7f99cc105830 2024-04-23T14:42:38.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.081+0000 7f99c8c00700 1 -- 172.21.15.28:0/1196151645 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99b4022070 con 0x7f99cc105830 2024-04-23T14:42:38.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.081+0000 7f99d0a00700 1 -- 172.21.15.28:0/1196151645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99cc1a06a0 con 0x7f99cc105830 2024-04-23T14:42:38.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.081+0000 7f99c8c00700 1 -- 172.21.15.28:0/1196151645 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b401b6c0 con 0x7f99cc105830 2024-04-23T14:42:38.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.082+0000 7f99d0a00700 1 -- 172.21.15.28:0/1196151645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99cc1a7f80 con 0x7f99cc105830 2024-04-23T14:42:38.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.083+0000 7f99d0a00700 1 -- 172.21.15.28:0/1196151645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99cc106360 con 0x7f99cc105830 2024-04-23T14:42:38.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.085+0000 7f99c8c00700 1 -- 172.21.15.28:0/1196151645 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f99b401d470 con 0x7f99cc105830 2024-04-23T14:42:38.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.087+0000 7f99c8c00700 1 --2- 172.21.15.28:0/1196151645 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f99b00a3cd0 0x7f99b00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:38.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.087+0000 7f99c8c00700 1 -- 172.21.15.28:0/1196151645 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f99b400f020 con 0x7f99cc105830 2024-04-23T14:42:38.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.087+0000 7f99cb400700 1 --2- 172.21.15.28:0/1196151645 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f99b00a3cd0 0x7f99b00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:38.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.088+0000 7f99cb400700 1 --2- 172.21.15.28:0/1196151645 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f99b00a3cd0 0x7f99b00a6190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f99bc009c60 tx=0x7f99bc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:38.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.092+0000 7f99c8c00700 1 -- 172.21.15.28:0/1196151645 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f99b409db90 con 0x7f99cc105830 2024-04-23T14:42:38.250 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:42:38.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.366+0000 7f99d0a00700 1 -- 172.21.15.28:0/1196151645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f99cc19c760 con 0x7f99cc105830 2024-04-23T14:42:38.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.366+0000 7f99c8c00700 1 -- 172.21.15.28:0/1196151645 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f99b409d490 con 0x7f99cc105830 2024-04-23T14:42:38.368 INFO:teuthology.orchestra.run.smithi028.stdout:163208757306 2024-04-23T14:42:38.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.369+0000 7f99d0a00700 1 -- 172.21.15.28:0/1196151645 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f99b00a3cd0 msgr2=0x7f99b00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.369+0000 7f99d0a00700 1 --2- 172.21.15.28:0/1196151645 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f99b00a3cd0 0x7f99b00a6190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f99bc009c60 tx=0x7f99bc009260 comp rx=0 tx=0).stop 2024-04-23T14:42:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.369+0000 7f99d0a00700 1 -- 172.21.15.28:0/1196151645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99cc105830 msgr2=0x7f99cc19b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.369+0000 7f99d0a00700 1 --2- 172.21.15.28:0/1196151645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99cc105830 0x7f99cc19b940 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f99b4000f80 tx=0x7f99b4007a20 comp rx=0 tx=0).stop 2024-04-23T14:42:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.369+0000 7f99d0a00700 1 -- 172.21.15.28:0/1196151645 shutdown_connections 2024-04-23T14:42:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.369+0000 7f99d0a00700 1 --2- 172.21.15.28:0/1196151645 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f99b00a3cd0 0x7f99b00a6190 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.369+0000 7f99d0a00700 1 --2- 172.21.15.28:0/1196151645 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f99cc104e80 0x7f99cc19b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.369+0000 7f99d0a00700 1 --2- 172.21.15.28:0/1196151645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f99cc109f60 0x7f99cc19fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.369+0000 7f99d0a00700 1 --2- 172.21.15.28:0/1196151645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99cc105830 0x7f99cc19b940 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.369+0000 7f99d0a00700 1 -- 172.21.15.28:0/1196151645 >> 172.21.15.28:0/1196151645 conn(0x7f99cc100710 msgr2=0x7f99cc1012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.370+0000 7f99d0a00700 1 -- 172.21.15.28:0/1196151645 shutdown_connections 2024-04-23T14:42:38.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:38.370+0000 7f99d0a00700 1 -- 172.21.15.28:0/1196151645 wait complete. 2024-04-23T14:42:38.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:38 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1196151645' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-23T14:42:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:42:38.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:38 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1196151645' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-23T14:42:38.789 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:42:38.808 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:42:38.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:38 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1196151645' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-23T14:42:39.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:39 smithi028 ceph-mon[27810]: pgmap v343: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:42:39.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:39 smithi105 ceph-mon[32555]: pgmap v343: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:42:39.797 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757305 got 163208757306 for osd.5 2024-04-23T14:42:39.798 DEBUG:teuthology.parallel:result is None 2024-04-23T14:42:39.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:39 smithi038 ceph-mon[32348]: pgmap v343: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:42:40.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.185+0000 7f52de800700 1 -- 172.21.15.28:0/3502207495 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52d8105910 msgr2=0x7f52d8109960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:40.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.185+0000 7f52de800700 1 --2- 172.21.15.28:0/3502207495 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52d8105910 0x7f52d8109960 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f52c0009990 tx=0x7f52c0009ca0 comp rx=0 tx=0).stop 2024-04-23T14:42:40.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.185+0000 7f52de800700 1 -- 172.21.15.28:0/3502207495 shutdown_connections 2024-04-23T14:42:40.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.185+0000 7f52de800700 1 --2- 172.21.15.28:0/3502207495 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52d810a040 0x7f52d8111c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:40.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.185+0000 7f52de800700 1 --2- 172.21.15.28:0/3502207495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52d8104f60 0x7f52d8105340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:40.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.185+0000 7f52de800700 1 --2- 172.21.15.28:0/3502207495 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52d8105910 0x7f52d8109960 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:40.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.185+0000 7f52de800700 1 -- 172.21.15.28:0/3502207495 >> 172.21.15.28:0/3502207495 conn(0x7f52d81007f0 msgr2=0x7f52d8102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:40.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.185+0000 7f52de800700 1 -- 172.21.15.28:0/3502207495 shutdown_connections 2024-04-23T14:42:40.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.185+0000 7f52de800700 1 -- 172.21.15.28:0/3502207495 wait complete. 2024-04-23T14:42:40.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.186+0000 7f52de800700 1 Processor -- start 2024-04-23T14:42:40.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.187+0000 7f52de800700 1 -- start start 2024-04-23T14:42:40.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.187+0000 7f52de800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52d8104f60 0x7f52d819b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:40.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.187+0000 7f52de800700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52d810a040 0x7f52d819bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:40.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.187+0000 7f52de800700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52d81a0010 0x7f52d81a0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:40.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.187+0000 7f52de800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f52d8115070 con 0x7f52d8104f60 2024-04-23T14:42:40.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.187+0000 7f52de800700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f52d8114ef0 con 0x7f52d81a0010 2024-04-23T14:42:40.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.187+0000 7f52de800700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f52d81151f0 con 0x7f52d810a040 2024-04-23T14:42:40.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.187+0000 7f52d7400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52d8104f60 0x7f52d819b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:40.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.187+0000 7f52d7400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52d8104f60 0x7f52d819b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43808/0 (socket says 172.21.15.28:43808) 2024-04-23T14:42:40.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.187+0000 7f52d7400700 1 -- 172.21.15.28:0/3502360999 learned_addr learned my addr 172.21.15.28:0/3502360999 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:42:40.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.187+0000 7f52cfe00700 1 --2- 172.21.15.28:0/3502360999 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52d810a040 0x7f52d819bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:40.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.187+0000 7f52d7e00700 1 --2- 172.21.15.28:0/3502360999 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52d81a0010 0x7f52d81a0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:40.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.188+0000 7f52d7400700 1 -- 172.21.15.28:0/3502360999 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52d810a040 msgr2=0x7f52d819bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:40.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.188+0000 7f52d7400700 1 --2- 172.21.15.28:0/3502360999 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52d810a040 0x7f52d819bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:40.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.188+0000 7f52d7400700 1 -- 172.21.15.28:0/3502360999 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52d81a0010 msgr2=0x7f52d81a0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:40.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.188+0000 7f52d7400700 1 --2- 172.21.15.28:0/3502360999 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52d81a0010 0x7f52d81a0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:40.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.188+0000 7f52d7400700 1 -- 172.21.15.28:0/3502360999 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52d81a0b30 con 0x7f52d8104f60 2024-04-23T14:42:40.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.188+0000 7f52d7400700 1 --2- 172.21.15.28:0/3502360999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52d8104f60 0x7f52d819b740 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f52c800eb30 tx=0x7f52c800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:40.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.189+0000 7f52d4c00700 1 -- 172.21.15.28:0/3502360999 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c800cbd0 con 0x7f52d8104f60 2024-04-23T14:42:40.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.189+0000 7f52de800700 1 -- 172.21.15.28:0/3502360999 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52d81a0e20 con 0x7f52d8104f60 2024-04-23T14:42:40.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.189+0000 7f52d4c00700 1 -- 172.21.15.28:0/3502360999 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52c8004500 con 0x7f52d8104f60 2024-04-23T14:42:40.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.189+0000 7f52d4c00700 1 -- 172.21.15.28:0/3502360999 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c8010430 con 0x7f52d8104f60 2024-04-23T14:42:40.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.189+0000 7f52de800700 1 -- 172.21.15.28:0/3502360999 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52d81a8650 con 0x7f52d8104f60 2024-04-23T14:42:40.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.191+0000 7f52de800700 1 -- 172.21.15.28:0/3502360999 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52d80697e0 con 0x7f52d8104f60 2024-04-23T14:42:40.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.192+0000 7f52d4c00700 1 -- 172.21.15.28:0/3502360999 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f52c8004020 con 0x7f52d8104f60 2024-04-23T14:42:40.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.194+0000 7f52d4c00700 1 --2- 172.21.15.28:0/3502360999 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f52bc0a3d50 0x7f52bc0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:40.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.194+0000 7f52d4c00700 1 -- 172.21.15.28:0/3502360999 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f52c80d07c0 con 0x7f52d8104f60 2024-04-23T14:42:40.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.197+0000 7f52cfe00700 1 --2- 172.21.15.28:0/3502360999 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f52bc0a3d50 0x7f52bc0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:40.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.198+0000 7f52cfe00700 1 --2- 172.21.15.28:0/3502360999 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f52bc0a3d50 0x7f52bc0a6210 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f52c00098e0 tx=0x7f52c0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:40.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.199+0000 7f52d4c00700 1 -- 172.21.15.28:0/3502360999 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f52c809d0f0 con 0x7f52d8104f60 2024-04-23T14:42:40.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.449+0000 7f52de800700 1 -- 172.21.15.28:0/3502360999 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f52d819caa0 con 0x7f52d8104f60 2024-04-23T14:42:40.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.450+0000 7f52d4c00700 1 -- 172.21.15.28:0/3502360999 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f52c809c9f0 con 0x7f52d8104f60 2024-04-23T14:42:40.059 INFO:teuthology.orchestra.run.smithi028.stdout:77309411405 2024-04-23T14:42:40.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.452+0000 7f52de800700 1 -- 172.21.15.28:0/3502360999 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f52bc0a3d50 msgr2=0x7f52bc0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:40.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.452+0000 7f52de800700 1 --2- 172.21.15.28:0/3502360999 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f52bc0a3d50 0x7f52bc0a6210 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f52c00098e0 tx=0x7f52c0005fd0 comp rx=0 tx=0).stop 2024-04-23T14:42:40.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.452+0000 7f52de800700 1 -- 172.21.15.28:0/3502360999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52d8104f60 msgr2=0x7f52d819b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:40.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.452+0000 7f52de800700 1 --2- 172.21.15.28:0/3502360999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52d8104f60 0x7f52d819b740 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f52c800eb30 tx=0x7f52c800ee40 comp rx=0 tx=0).stop 2024-04-23T14:42:40.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.453+0000 7f52de800700 1 -- 172.21.15.28:0/3502360999 shutdown_connections 2024-04-23T14:42:40.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.453+0000 7f52de800700 1 --2- 172.21.15.28:0/3502360999 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f52bc0a3d50 0x7f52bc0a6210 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:40.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.453+0000 7f52de800700 1 --2- 172.21.15.28:0/3502360999 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52d81a0010 0x7f52d81a0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:40.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.453+0000 7f52de800700 1 --2- 172.21.15.28:0/3502360999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52d8104f60 0x7f52d819b740 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:40.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.453+0000 7f52de800700 1 --2- 172.21.15.28:0/3502360999 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52d810a040 0x7f52d819bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:40.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.453+0000 7f52de800700 1 -- 172.21.15.28:0/3502360999 >> 172.21.15.28:0/3502360999 conn(0x7f52d81007f0 msgr2=0x7f52d81010c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:40.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.453+0000 7f52de800700 1 -- 172.21.15.28:0/3502360999 shutdown_connections 2024-04-23T14:42:40.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:39.453+0000 7f52de800700 1 -- 172.21.15.28:0/3502360999 wait complete. 2024-04-23T14:42:40.760 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:40 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3502360999' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-23T14:42:40.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:40 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3502360999' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-23T14:42:40.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:40 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3502360999' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-23T14:42:41.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:41 smithi028 ceph-mon[27810]: pgmap v344: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:42:41.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:41 smithi105 ceph-mon[32555]: pgmap v344: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:42:41.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:41 smithi038 ceph-mon[32348]: pgmap v344: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:42:42.388 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411401 got 77309411405 for osd.2 2024-04-23T14:42:42.388 DEBUG:teuthology.parallel:result is None 2024-04-23T14:42:42.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:42 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:42:42.536 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:42 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]: dispatch 2024-04-23T14:42:42.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:42 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]: dispatch 2024-04-23T14:42:42.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:42 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-04-23T14:42:42.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:42 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-04-23T14:42:42.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:42 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-04-23T14:42:42.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:42 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-04-23T14:42:42.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:42 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]: dispatch 2024-04-23T14:42:42.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:42 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-04-23T14:42:42.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:42 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2d", "id": [3, 9]}]: dispatch 2024-04-23T14:42:42.537 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:42 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2829419591' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-23T14:42:42.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:42 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:42:42.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:42 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]: dispatch 2024-04-23T14:42:42.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:42 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]: dispatch 2024-04-23T14:42:42.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:42 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-04-23T14:42:42.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:42 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-04-23T14:42:42.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:42 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-04-23T14:42:42.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:42 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-04-23T14:42:42.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:42 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]: dispatch 2024-04-23T14:42:42.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:42 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-04-23T14:42:42.623 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:42 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2d", "id": [3, 9]}]: dispatch 2024-04-23T14:42:42.624 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:42 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2829419591' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-23T14:42:42.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:42 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:42:42.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:42 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]: dispatch 2024-04-23T14:42:42.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:42 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]: dispatch 2024-04-23T14:42:42.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:42 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-04-23T14:42:42.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:42 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-04-23T14:42:42.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:42 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-04-23T14:42:42.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:42 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-04-23T14:42:42.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:42 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]: dispatch 2024-04-23T14:42:42.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:42 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-04-23T14:42:42.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:42 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2d", "id": [3, 9]}]: dispatch 2024-04-23T14:42:42.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:42 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2829419591' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-23T14:42:43.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.354+0000 7fcc7de00700 1 -- 172.21.15.28:0/3700793683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc78109030 msgr2=0x7fcc781094b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:43.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.354+0000 7fcc7de00700 1 --2- 172.21.15.28:0/3700793683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc78109030 0x7fcc781094b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fcc640098f0 tx=0x7fcc64009c00 comp rx=0 tx=0).stop 2024-04-23T14:42:43.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.354+0000 7fcc7de00700 1 -- 172.21.15.28:0/3700793683 shutdown_connections 2024-04-23T14:42:43.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.354+0000 7fcc7de00700 1 --2- 172.21.15.28:0/3700793683 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc7810f0e0 0x7fcc7810f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:43.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.354+0000 7fcc7de00700 1 --2- 172.21.15.28:0/3700793683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc781099f0 0x7fcc78115e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:43.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.354+0000 7fcc7de00700 1 --2- 172.21.15.28:0/3700793683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc78109030 0x7fcc781094b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:43.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.354+0000 7fcc7de00700 1 -- 172.21.15.28:0/3700793683 >> 172.21.15.28:0/3700793683 conn(0x7fcc78078880 msgr2=0x7fcc78107910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:43.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.354+0000 7fcc7de00700 1 -- 172.21.15.28:0/3700793683 shutdown_connections 2024-04-23T14:42:43.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.355+0000 7fcc7de00700 1 -- 172.21.15.28:0/3700793683 wait complete. 2024-04-23T14:42:43.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.356+0000 7fcc7de00700 1 Processor -- start 2024-04-23T14:42:43.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.357+0000 7fcc7de00700 1 -- start start 2024-04-23T14:42:43.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.357+0000 7fcc7de00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc78109030 0x7fcc781a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:43.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.357+0000 7fcc7de00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc781099f0 0x7fcc781a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:43.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.357+0000 7fcc7de00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc7810f0e0 0x7fcc7819f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:43.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.357+0000 7fcc7de00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcc78119340 con 0x7fcc78109030 2024-04-23T14:42:43.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.357+0000 7fcc7de00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcc781191c0 con 0x7fcc781099f0 2024-04-23T14:42:43.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.357+0000 7fcc7de00700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcc781194c0 con 0x7fcc7810f0e0 2024-04-23T14:42:43.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.358+0000 7fcc77e00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc781099f0 0x7fcc781a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:43.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.358+0000 7fcc7ca00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc78109030 0x7fcc781a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:43.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.358+0000 7fcc7d400700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc7810f0e0 0x7fcc7819f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:43.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.358+0000 7fcc77e00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc781099f0 0x7fcc781a5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:52018/0 (socket says 172.21.15.28:52018) 2024-04-23T14:42:43.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.358+0000 7fcc7ca00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc78109030 0x7fcc781a53e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43822/0 (socket says 172.21.15.28:43822) 2024-04-23T14:42:43.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.358+0000 7fcc77e00700 1 -- 172.21.15.28:0/2829419591 learned_addr learned my addr 172.21.15.28:0/2829419591 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:42:43.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.358+0000 7fcc7d400700 1 -- 172.21.15.28:0/2829419591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc781099f0 msgr2=0x7fcc781a5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:43.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.358+0000 7fcc7d400700 1 --2- 172.21.15.28:0/2829419591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc781099f0 0x7fcc781a5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:43.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.358+0000 7fcc7d400700 1 -- 172.21.15.28:0/2829419591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc78109030 msgr2=0x7fcc781a53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:43.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.358+0000 7fcc7d400700 1 --2- 172.21.15.28:0/2829419591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc78109030 0x7fcc781a53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:43.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.358+0000 7fcc7d400700 1 -- 172.21.15.28:0/2829419591 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc7819fd00 con 0x7fcc7810f0e0 2024-04-23T14:42:43.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.358+0000 7fcc7ca00700 1 --2- 172.21.15.28:0/2829419591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc78109030 0x7fcc781a53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:42:43.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.359+0000 7fcc77e00700 1 --2- 172.21.15.28:0/2829419591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc781099f0 0x7fcc781a5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:42:43.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.359+0000 7fcc7d400700 1 --2- 172.21.15.28:0/2829419591 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc7810f0e0 0x7fcc7819f5a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcc70009cd0 tx=0x7fcc7000c620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:43.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.359+0000 7fcc75600700 1 -- 172.21.15.28:0/2829419591 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc7000cd10 con 0x7fcc7810f0e0 2024-04-23T14:42:43.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.359+0000 7fcc75600700 1 -- 172.21.15.28:0/2829419591 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc70003a00 con 0x7fcc7810f0e0 2024-04-23T14:42:43.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.359+0000 7fcc7de00700 1 -- 172.21.15.28:0/2829419591 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc7819ff90 con 0x7fcc7810f0e0 2024-04-23T14:42:43.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.359+0000 7fcc75600700 1 -- 172.21.15.28:0/2829419591 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc700045a0 con 0x7fcc7810f0e0 2024-04-23T14:42:43.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.359+0000 7fcc7de00700 1 -- 172.21.15.28:0/2829419591 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc781ac2b0 con 0x7fcc7810f0e0 2024-04-23T14:42:43.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.360+0000 7fcc7de00700 1 -- 172.21.15.28:0/2829419591 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc78111430 con 0x7fcc7810f0e0 2024-04-23T14:42:43.628 INFO:teuthology.orchestra.run.smithi028.stdout:343597383706 2024-04-23T14:42:43.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.363+0000 7fcc75600700 1 -- 172.21.15.28:0/2829419591 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fcc7001a020 con 0x7fcc7810f0e0 2024-04-23T14:42:43.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.365+0000 7fcc75600700 1 --2- 172.21.15.28:0/2829419591 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fcc600a3cd0 0x7fcc600a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:43.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.365+0000 7fcc75600700 1 -- 172.21.15.28:0/2829419591 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fcc700cfd70 con 0x7fcc7810f0e0 2024-04-23T14:42:43.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.365+0000 7fcc7ca00700 1 --2- 172.21.15.28:0/2829419591 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fcc600a3cd0 0x7fcc600a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:43.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.366+0000 7fcc7ca00700 1 --2- 172.21.15.28:0/2829419591 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fcc600a3cd0 0x7fcc600a6190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c005950 tx=0x7fcc6c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:43.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.369+0000 7fcc75600700 1 -- 172.21.15.28:0/2829419591 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcc70010040 con 0x7fcc7810f0e0 2024-04-23T14:42:43.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.633+0000 7fcc7de00700 1 -- 172.21.15.28:0/2829419591 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fcc781a0ad0 con 0x7fcc7810f0e0 2024-04-23T14:42:43.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.633+0000 7fcc75600700 1 -- 172.21.15.28:0/2829419591 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fcc7009c6a0 con 0x7fcc7810f0e0 2024-04-23T14:42:43.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.636+0000 7fcc7de00700 1 -- 172.21.15.28:0/2829419591 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fcc600a3cd0 msgr2=0x7fcc600a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:43.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.636+0000 7fcc7de00700 1 --2- 172.21.15.28:0/2829419591 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fcc600a3cd0 0x7fcc600a6190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c005950 tx=0x7fcc6c00b410 comp rx=0 tx=0).stop 2024-04-23T14:42:43.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.636+0000 7fcc7de00700 1 -- 172.21.15.28:0/2829419591 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc7810f0e0 msgr2=0x7fcc7819f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:43.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.636+0000 7fcc7de00700 1 --2- 172.21.15.28:0/2829419591 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc7810f0e0 0x7fcc7819f5a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcc70009cd0 tx=0x7fcc7000c620 comp rx=0 tx=0).stop 2024-04-23T14:42:43.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.637+0000 7fcc7de00700 1 -- 172.21.15.28:0/2829419591 shutdown_connections 2024-04-23T14:42:43.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.637+0000 7fcc7de00700 1 --2- 172.21.15.28:0/2829419591 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fcc600a3cd0 0x7fcc600a6190 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:43.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.637+0000 7fcc7de00700 1 --2- 172.21.15.28:0/2829419591 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc7810f0e0 0x7fcc7819f5a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:43.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.637+0000 7fcc7de00700 1 --2- 172.21.15.28:0/2829419591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc78109030 0x7fcc781a53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:43.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.637+0000 7fcc7de00700 1 --2- 172.21.15.28:0/2829419591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc781099f0 0x7fcc781a5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:43.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.637+0000 7fcc7de00700 1 -- 172.21.15.28:0/2829419591 >> 172.21.15.28:0/2829419591 conn(0x7fcc78078880 msgr2=0x7fcc78079ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:43.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.637+0000 7fcc7de00700 1 -- 172.21.15.28:0/2829419591 shutdown_connections 2024-04-23T14:42:43.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:41.637+0000 7fcc7de00700 1 -- 172.21.15.28:0/2829419591 wait complete. 2024-04-23T14:42:43.749 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:43 smithi028 ceph-mon[27810]: pgmap v345: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:42:43.750 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:43 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-04-23T14:42:43.750 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:43 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]': finished 2024-04-23T14:42:43.750 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:43 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-04-23T14:42:43.750 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:43 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-04-23T14:42:43.750 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:43 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-04-23T14:42:43.750 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:43 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-04-23T14:42:43.750 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:43 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]': finished 2024-04-23T14:42:43.750 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:43 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-04-23T14:42:43.750 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:43 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2d", "id": [3, 9]}]': finished 2024-04-23T14:42:43.750 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:43 smithi028 ceph-mon[27810]: osdmap e86: 12 total, 12 up, 12 in 2024-04-23T14:42:43.750 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:43 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:42:43.750 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:43 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:42:43.750 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:43 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:42:43.751 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:43 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:42:43.751 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:43 smithi028 ceph-mon[27810]: osdmap e87: 12 total, 12 up, 12 in 2024-04-23T14:42:43.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:43 smithi105 ceph-mon[32555]: pgmap v345: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:42:43.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:43 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-04-23T14:42:43.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:43 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]': finished 2024-04-23T14:42:43.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:43 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-04-23T14:42:43.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:43 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-04-23T14:42:43.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:43 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-04-23T14:42:43.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:43 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-04-23T14:42:43.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:43 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]': finished 2024-04-23T14:42:43.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:43 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-04-23T14:42:43.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:43 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2d", "id": [3, 9]}]': finished 2024-04-23T14:42:43.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:43 smithi105 ceph-mon[32555]: osdmap e86: 12 total, 12 up, 12 in 2024-04-23T14:42:43.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:43 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:42:43.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:43 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:42:43.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:43 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:42:43.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:43 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:42:43.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:43 smithi105 ceph-mon[32555]: osdmap e87: 12 total, 12 up, 12 in 2024-04-23T14:42:43.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:43 smithi038 ceph-mon[32348]: pgmap v345: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:42:43.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:43 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-04-23T14:42:43.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:43 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]': finished 2024-04-23T14:42:43.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:43 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-04-23T14:42:43.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:43 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-04-23T14:42:43.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:43 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-04-23T14:42:43.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:43 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-04-23T14:42:43.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:43 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]': finished 2024-04-23T14:42:43.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:43 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-04-23T14:42:43.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:43 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2d", "id": [3, 9]}]': finished 2024-04-23T14:42:43.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:43 smithi038 ceph-mon[32348]: osdmap e86: 12 total, 12 up, 12 in 2024-04-23T14:42:43.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:43 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:42:43.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:43 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:42:43.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:43 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:42:43.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:43 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:42:43.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:43 smithi038 ceph-mon[32348]: osdmap e87: 12 total, 12 up, 12 in 2024-04-23T14:42:43.976 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:42:44.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:44 smithi028 ceph-mon[27810]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-23T14:42:44.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:44 smithi105 ceph-mon[32555]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-23T14:42:44.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:44 smithi038 ceph-mon[32348]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-23T14:42:45.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:45 smithi028 ceph-mon[27810]: pgmap v348: 97 pgs: 1 peering, 96 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s 2024-04-23T14:42:45.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:45 smithi105 ceph-mon[32555]: pgmap v348: 97 pgs: 1 peering, 96 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s 2024-04-23T14:42:45.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:45 smithi038 ceph-mon[32348]: pgmap v348: 97 pgs: 1 peering, 96 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s 2024-04-23T14:42:46.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:46 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3444331738' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-23T14:42:46.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:46 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2454181559' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-23T14:42:46.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:46 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3444331738' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-23T14:42:46.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:46 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2454181559' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-23T14:42:46.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:46 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3444331738' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-23T14:42:46.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:46 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2454181559' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-23T14:42:46.878 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383702 got 343597383706 for osd.11 2024-04-23T14:42:46.878 DEBUG:teuthology.parallel:result is None 2024-04-23T14:42:47.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:47 smithi105 ceph-mon[32555]: pgmap v349: 97 pgs: 1 peering, 96 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-04-23T14:42:47.832 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:47 smithi028 ceph-mon[27810]: pgmap v349: 97 pgs: 1 peering, 96 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-04-23T14:42:47.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:47 smithi038 ceph-mon[32348]: pgmap v349: 97 pgs: 1 peering, 96 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-04-23T14:42:47.873 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:42:48.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.024+0000 7fb48b600700 1 -- 172.21.15.28:0/396840337 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb4841019f0 msgr2=0x7fb484101e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:48.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.024+0000 7fb48b600700 1 --2- 172.21.15.28:0/396840337 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb4841019f0 0x7fb484101e70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb4700098f0 tx=0x7fb470009c00 comp rx=0 tx=0).stop 2024-04-23T14:42:48.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.024+0000 7fb48b600700 1 -- 172.21.15.28:0/396840337 shutdown_connections 2024-04-23T14:42:48.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.024+0000 7fb48b600700 1 --2- 172.21.15.28:0/396840337 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb4841023b0 0x7fb48410e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:48.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.024+0000 7fb48b600700 1 --2- 172.21.15.28:0/396840337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb484107a40 0x7fb484107e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:48.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.024+0000 7fb48b600700 1 --2- 172.21.15.28:0/396840337 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb4841019f0 0x7fb484101e70 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:48.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.024+0000 7fb48b600700 1 -- 172.21.15.28:0/396840337 >> 172.21.15.28:0/396840337 conn(0x7fb4840789a0 msgr2=0x7fb484078db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.024+0000 7fb48b600700 1 -- 172.21.15.28:0/396840337 shutdown_connections 2024-04-23T14:42:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.024+0000 7fb48b600700 1 -- 172.21.15.28:0/396840337 wait complete. 2024-04-23T14:42:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.025+0000 7fb48b600700 1 Processor -- start 2024-04-23T14:42:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.026+0000 7fb48b600700 1 -- start start 2024-04-23T14:42:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.026+0000 7fb48b600700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb4841019f0 0x7fb48419b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.026+0000 7fb48b600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4841023b0 0x7fb48419bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.026+0000 7fb48b600700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb484107a40 0x7fb48419fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.026+0000 7fb48b600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb484111d30 con 0x7fb4841023b0 2024-04-23T14:42:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.026+0000 7fb48b600700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb484111bb0 con 0x7fb484107a40 2024-04-23T14:42:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.026+0000 7fb48b600700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb484111eb0 con 0x7fb4841019f0 2024-04-23T14:42:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.026+0000 7fb483e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4841023b0 0x7fb48419bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.026+0000 7fb483e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4841023b0 0x7fb48419bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43858/0 (socket says 172.21.15.28:43858) 2024-04-23T14:42:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.026+0000 7fb483e00700 1 -- 172.21.15.28:0/2454181559 learned_addr learned my addr 172.21.15.28:0/2454181559 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:42:48.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.026+0000 7fb488c00700 1 --2- 172.21.15.28:0/2454181559 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb4841019f0 0x7fb48419b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:48.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.026+0000 7fb489600700 1 --2- 172.21.15.28:0/2454181559 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb484107a40 0x7fb48419fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:48.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.027+0000 7fb483e00700 1 -- 172.21.15.28:0/2454181559 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb4841019f0 msgr2=0x7fb48419b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:48.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.027+0000 7fb483e00700 1 --2- 172.21.15.28:0/2454181559 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb4841019f0 0x7fb48419b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:48.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.027+0000 7fb483e00700 1 -- 172.21.15.28:0/2454181559 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb484107a40 msgr2=0x7fb48419fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:48.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.027+0000 7fb483e00700 1 --2- 172.21.15.28:0/2454181559 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb484107a40 0x7fb48419fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:48.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.027+0000 7fb483e00700 1 -- 172.21.15.28:0/2454181559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4841a0610 con 0x7fb4841023b0 2024-04-23T14:42:48.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.027+0000 7fb489600700 1 --2- 172.21.15.28:0/2454181559 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb484107a40 0x7fb48419fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:42:48.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.027+0000 7fb483e00700 1 --2- 172.21.15.28:0/2454181559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4841023b0 0x7fb48419bb90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb470005950 tx=0x7fb470005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:48.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.027+0000 7fb481600700 1 -- 172.21.15.28:0/2454181559 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4700052e0 con 0x7fb4841023b0 2024-04-23T14:42:48.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.028+0000 7fb481600700 1 -- 172.21.15.28:0/2454181559 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb47000ba40 con 0x7fb4841023b0 2024-04-23T14:42:48.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.028+0000 7fb48b600700 1 -- 172.21.15.28:0/2454181559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4841a08a0 con 0x7fb4841023b0 2024-04-23T14:42:48.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.028+0000 7fb481600700 1 -- 172.21.15.28:0/2454181559 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb470003b80 con 0x7fb4841023b0 2024-04-23T14:42:48.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.028+0000 7fb48b600700 1 -- 172.21.15.28:0/2454181559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4841a8180 con 0x7fb4841023b0 2024-04-23T14:42:48.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.029+0000 7fb48b600700 1 -- 172.21.15.28:0/2454181559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4840697e0 con 0x7fb4841023b0 2024-04-23T14:42:48.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.031+0000 7fb481600700 1 -- 172.21.15.28:0/2454181559 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb470005480 con 0x7fb4841023b0 2024-04-23T14:42:48.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.032+0000 7fb481600700 1 --2- 172.21.15.28:0/2454181559 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb46c0a3d50 0x7fb46c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:48.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.033+0000 7fb481600700 1 -- 172.21.15.28:0/2454181559 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7fb4700d1c80 con 0x7fb4841023b0 2024-04-23T14:42:48.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.033+0000 7fb488c00700 1 --2- 172.21.15.28:0/2454181559 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb46c0a3d50 0x7fb46c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:48.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.034+0000 7fb488c00700 1 --2- 172.21.15.28:0/2454181559 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb46c0a3d50 0x7fb46c0a6210 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb478005fd0 tx=0x7fb478009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:48.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.035+0000 7fb481600700 1 -- 172.21.15.28:0/2454181559 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb470013030 con 0x7fb4841023b0 2024-04-23T14:42:48.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.285+0000 7fb48b600700 1 -- 172.21.15.28:0/2454181559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fb4841a8350 con 0x7fb4841023b0 2024-04-23T14:42:48.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.286+0000 7fb481600700 1 -- 172.21.15.28:0/2454181559 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb47009e550 con 0x7fb4841023b0 2024-04-23T14:42:48.384 INFO:teuthology.orchestra.run.smithi028.stdout:55834574931 2024-04-23T14:42:48.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.291+0000 7fb48b600700 1 -- 172.21.15.28:0/2454181559 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb46c0a3d50 msgr2=0x7fb46c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:48.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.291+0000 7fb48b600700 1 --2- 172.21.15.28:0/2454181559 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb46c0a3d50 0x7fb46c0a6210 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb478005fd0 tx=0x7fb478009310 comp rx=0 tx=0).stop 2024-04-23T14:42:48.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.291+0000 7fb48b600700 1 -- 172.21.15.28:0/2454181559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4841023b0 msgr2=0x7fb48419bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:48.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.291+0000 7fb48b600700 1 --2- 172.21.15.28:0/2454181559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4841023b0 0x7fb48419bb90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb470005950 tx=0x7fb470005040 comp rx=0 tx=0).stop 2024-04-23T14:42:48.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.291+0000 7fb48b600700 1 -- 172.21.15.28:0/2454181559 shutdown_connections 2024-04-23T14:42:48.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.291+0000 7fb48b600700 1 --2- 172.21.15.28:0/2454181559 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb46c0a3d50 0x7fb46c0a6210 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:48.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.291+0000 7fb48b600700 1 --2- 172.21.15.28:0/2454181559 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb4841019f0 0x7fb48419b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:48.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.291+0000 7fb48b600700 1 --2- 172.21.15.28:0/2454181559 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb484107a40 0x7fb48419fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:48.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.291+0000 7fb48b600700 1 --2- 172.21.15.28:0/2454181559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4841023b0 0x7fb48419bb90 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:48.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.291+0000 7fb48b600700 1 -- 172.21.15.28:0/2454181559 >> 172.21.15.28:0/2454181559 conn(0x7fb4840789a0 msgr2=0x7fb48406a870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:48.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.291+0000 7fb48b600700 1 -- 172.21.15.28:0/2454181559 shutdown_connections 2024-04-23T14:42:48.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:46.291+0000 7fb48b600700 1 -- 172.21.15.28:0/2454181559 wait complete. 2024-04-23T14:42:48.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.726+0000 7f06a4200700 1 -- 172.21.15.28:0/931502319 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f069c075640 msgr2=0x7f069c10efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:48.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.726+0000 7f06a4200700 1 --2- 172.21.15.28:0/931502319 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f069c075640 0x7f069c10efe0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0688009990 tx=0x7f0688009ca0 comp rx=0 tx=0).stop 2024-04-23T14:42:48.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.726+0000 7f06a4200700 1 -- 172.21.15.28:0/931502319 shutdown_connections 2024-04-23T14:42:48.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.726+0000 7f06a4200700 1 --2- 172.21.15.28:0/931502319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f069c10f520 0x7f069c111910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:48.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.726+0000 7f06a4200700 1 --2- 172.21.15.28:0/931502319 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f069c074d20 0x7f069c075100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:48.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.726+0000 7f06a4200700 1 --2- 172.21.15.28:0/931502319 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f069c075640 0x7f069c10efe0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.726+0000 7f06a4200700 1 -- 172.21.15.28:0/931502319 >> 172.21.15.28:0/931502319 conn(0x7f069c079bc0 msgr2=0x7f069c079fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.726+0000 7f06a4200700 1 -- 172.21.15.28:0/931502319 shutdown_connections 2024-04-23T14:42:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.726+0000 7f06a4200700 1 -- 172.21.15.28:0/931502319 wait complete. 2024-04-23T14:42:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.727+0000 7f06a4200700 1 Processor -- start 2024-04-23T14:42:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.728+0000 7f06a4200700 1 -- start start 2024-04-23T14:42:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.728+0000 7f06a4200700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f069c074d20 0x7f069c19b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.728+0000 7f06a4200700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f069c075640 0x7f069c19ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.728+0000 7f06a4200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f069c10f520 0x7f069c19fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.728+0000 7f06a4200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f069c115080 con 0x7f069c10f520 2024-04-23T14:42:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.728+0000 7f06a4200700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f069c114f00 con 0x7f069c074d20 2024-04-23T14:42:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.728+0000 7f06a4200700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f069c115200 con 0x7f069c075640 2024-04-23T14:42:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.728+0000 7f06a1800700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f069c074d20 0x7f069c19b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.728+0000 7f06a2200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f069c10f520 0x7f069c19fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:48.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.728+0000 7f06a1800700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f069c074d20 0x7f069c19b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:52042/0 (socket says 172.21.15.28:52042) 2024-04-23T14:42:48.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.728+0000 7f06a1800700 1 -- 172.21.15.28:0/3444331738 learned_addr learned my addr 172.21.15.28:0/3444331738 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:42:48.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.729+0000 7f06a2200700 1 -- 172.21.15.28:0/3444331738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f069c075640 msgr2=0x7f069c19ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:48.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.729+0000 7f06a2200700 1 --2- 172.21.15.28:0/3444331738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f069c075640 0x7f069c19ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:48.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.729+0000 7f06a2200700 1 -- 172.21.15.28:0/3444331738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f069c074d20 msgr2=0x7f069c19b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:48.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.729+0000 7f06a2200700 1 --2- 172.21.15.28:0/3444331738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f069c074d20 0x7f069c19b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:48.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.729+0000 7f06a2200700 1 -- 172.21.15.28:0/3444331738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f069c1a04f0 con 0x7f069c10f520 2024-04-23T14:42:48.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.729+0000 7f06a1800700 1 --2- 172.21.15.28:0/3444331738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f069c074d20 0x7f069c19b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:42:48.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.729+0000 7f06a2200700 1 --2- 172.21.15.28:0/3444331738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f069c10f520 0x7f069c19fdb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f069400ef90 tx=0x7f069400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:48.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.729+0000 7f069a000700 1 -- 172.21.15.28:0/3444331738 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06940092e0 con 0x7f069c10f520 2024-04-23T14:42:48.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.729+0000 7f069a000700 1 -- 172.21.15.28:0/3444331738 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0694004bb0 con 0x7f069c10f520 2024-04-23T14:42:48.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.729+0000 7f06a4200700 1 -- 172.21.15.28:0/3444331738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f069c1a0780 con 0x7f069c10f520 2024-04-23T14:42:48.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.729+0000 7f069a000700 1 -- 172.21.15.28:0/3444331738 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0694004280 con 0x7f069c10f520 2024-04-23T14:42:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.730+0000 7f06a4200700 1 -- 172.21.15.28:0/3444331738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f069c1a8060 con 0x7f069c10f520 2024-04-23T14:42:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.731+0000 7f06a4200700 1 -- 172.21.15.28:0/3444331738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f069c0697e0 con 0x7f069c10f520 2024-04-23T14:42:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.733+0000 7f069a000700 1 -- 172.21.15.28:0/3444331738 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f069401b030 con 0x7f069c10f520 2024-04-23T14:42:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.734+0000 7f069a000700 1 --2- 172.21.15.28:0/3444331738 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f06840a3cd0 0x7f06840a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.734+0000 7f069a000700 1 -- 172.21.15.28:0/3444331738 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7f06940cfed0 con 0x7f069c10f520 2024-04-23T14:42:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.734+0000 7f06a1800700 1 --2- 172.21.15.28:0/3444331738 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f06840a3cd0 0x7f06840a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.735+0000 7f06a1800700 1 --2- 172.21.15.28:0/3444331738 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f06840a3cd0 0x7f06840a6190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0690011630 tx=0x7f0690009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.739+0000 7f069a000700 1 -- 172.21.15.28:0/3444331738 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0694022020 con 0x7f069c10f520 2024-04-23T14:42:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.983+0000 7f06a4200700 1 -- 172.21.15.28:0/3444331738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f069c1a8230 con 0x7f069c10f520 2024-04-23T14:42:48.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.984+0000 7f069a000700 1 -- 172.21.15.28:0/3444331738 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f0694010040 con 0x7f069c10f520 2024-04-23T14:42:48.818 INFO:teuthology.orchestra.run.smithi028.stdout:253403070507 2024-04-23T14:42:48.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.986+0000 7f06a4200700 1 -- 172.21.15.28:0/3444331738 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f06840a3cd0 msgr2=0x7f06840a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:48.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.986+0000 7f06a4200700 1 --2- 172.21.15.28:0/3444331738 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f06840a3cd0 0x7f06840a6190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0690011630 tx=0x7f0690009260 comp rx=0 tx=0).stop 2024-04-23T14:42:48.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.986+0000 7f06a4200700 1 -- 172.21.15.28:0/3444331738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f069c10f520 msgr2=0x7f069c19fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:48.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.986+0000 7f06a4200700 1 --2- 172.21.15.28:0/3444331738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f069c10f520 0x7f069c19fdb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f069400ef90 tx=0x7f069400c3a0 comp rx=0 tx=0).stop 2024-04-23T14:42:48.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.987+0000 7f06a4200700 1 -- 172.21.15.28:0/3444331738 shutdown_connections 2024-04-23T14:42:48.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.987+0000 7f06a4200700 1 --2- 172.21.15.28:0/3444331738 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f06840a3cd0 0x7f06840a6190 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:48.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.987+0000 7f06a4200700 1 --2- 172.21.15.28:0/3444331738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f069c10f520 0x7f069c19fdb0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:48.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.987+0000 7f06a4200700 1 --2- 172.21.15.28:0/3444331738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f069c074d20 0x7f069c19b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:48.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.987+0000 7f06a4200700 1 --2- 172.21.15.28:0/3444331738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f069c075640 0x7f069c19ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:48.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.987+0000 7f06a4200700 1 -- 172.21.15.28:0/3444331738 >> 172.21.15.28:0/3444331738 conn(0x7f069c079bc0 msgr2=0x7f069c100f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:48.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.987+0000 7f06a4200700 1 -- 172.21.15.28:0/3444331738 shutdown_connections 2024-04-23T14:42:48.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:45.987+0000 7f06a4200700 1 -- 172.21.15.28:0/3444331738 wait complete. 2024-04-23T14:42:49.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:49 smithi105 ceph-mon[32555]: pgmap v350: 97 pgs: 1 peering, 96 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-23T14:42:49.796 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:49 smithi028 ceph-mon[27810]: pgmap v350: 97 pgs: 1 peering, 96 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-23T14:42:49.863 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070500 got 253403070507 for osd.8 2024-04-23T14:42:49.863 DEBUG:teuthology.parallel:result is None 2024-04-23T14:42:49.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:49 smithi038 ceph-mon[32348]: pgmap v350: 97 pgs: 1 peering, 96 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-23T14:42:50.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.692+0000 7f062fe00700 1 -- 172.21.15.28:0/924861875 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f062806a800 msgr2=0x7f062806ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:50.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.692+0000 7f062fe00700 1 --2- 172.21.15.28:0/924861875 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f062806a800 0x7f062806ac60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0614009990 tx=0x7f0614009ca0 comp rx=0 tx=0).stop 2024-04-23T14:42:50.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.692+0000 7f062fe00700 1 -- 172.21.15.28:0/924861875 shutdown_connections 2024-04-23T14:42:50.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.692+0000 7f062fe00700 1 --2- 172.21.15.28:0/924861875 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f062806b1a0 0x7f062810f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:50.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.692+0000 7f062fe00700 1 --2- 172.21.15.28:0/924861875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0628108b70 0x7f0628108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:50.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.692+0000 7f062fe00700 1 --2- 172.21.15.28:0/924861875 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f062806a800 0x7f062806ac60 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:50.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.692+0000 7f062fe00700 1 -- 172.21.15.28:0/924861875 >> 172.21.15.28:0/924861875 conn(0x7f06280789a0 msgr2=0x7f0628078db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:50.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.692+0000 7f062fe00700 1 -- 172.21.15.28:0/924861875 shutdown_connections 2024-04-23T14:42:50.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.692+0000 7f062fe00700 1 -- 172.21.15.28:0/924861875 wait complete. 2024-04-23T14:42:50.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.693+0000 7f062fe00700 1 Processor -- start 2024-04-23T14:42:50.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.693+0000 7f062fe00700 1 -- start start 2024-04-23T14:42:50.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.694+0000 7f062fe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f062806a800 0x7f0628199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:50.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.694+0000 7f062fe00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f062806b1a0 0x7f0628199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:50.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.694+0000 7f062fe00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0628108b70 0x7f062819dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:50.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.694+0000 7f062fe00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0628112e60 con 0x7f062806a800 2024-04-23T14:42:50.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.694+0000 7f062fe00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0628112ce0 con 0x7f0628108b70 2024-04-23T14:42:50.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.694+0000 7f062fe00700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0628112fe0 con 0x7f062806b1a0 2024-04-23T14:42:50.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.694+0000 7f062d400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f062806a800 0x7f0628199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:50.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.694+0000 7f062ca00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f062806b1a0 0x7f0628199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:50.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.694+0000 7f062ca00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f062806b1a0 0x7f0628199870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:59516/0 (socket says 172.21.15.28:59516) 2024-04-23T14:42:50.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.694+0000 7f062d400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f062806a800 0x7f0628199330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48384/0 (socket says 172.21.15.28:48384) 2024-04-23T14:42:50.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.694+0000 7f062ca00700 1 -- 172.21.15.28:0/3294383600 learned_addr learned my addr 172.21.15.28:0/3294383600 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:42:50.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.694+0000 7f062de00700 1 --2- 172.21.15.28:0/3294383600 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0628108b70 0x7f062819dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:50.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.695+0000 7f062ca00700 1 -- 172.21.15.28:0/3294383600 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0628108b70 msgr2=0x7f062819dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:50.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.695+0000 7f062ca00700 1 --2- 172.21.15.28:0/3294383600 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0628108b70 0x7f062819dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:50.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.695+0000 7f062ca00700 1 -- 172.21.15.28:0/3294383600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f062806a800 msgr2=0x7f0628199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:50.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.695+0000 7f062ca00700 1 --2- 172.21.15.28:0/3294383600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f062806a800 0x7f0628199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:50.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.695+0000 7f062ca00700 1 -- 172.21.15.28:0/3294383600 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f062819e340 con 0x7f062806b1a0 2024-04-23T14:42:50.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.695+0000 7f062ca00700 1 --2- 172.21.15.28:0/3294383600 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f062806b1a0 0x7f0628199870 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0614000c00 tx=0x7f061400bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:50.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.695+0000 7f0626000700 1 -- 172.21.15.28:0/3294383600 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0614027070 con 0x7f062806b1a0 2024-04-23T14:42:50.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.695+0000 7f0626000700 1 -- 172.21.15.28:0/3294383600 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06140056d0 con 0x7f062806b1a0 2024-04-23T14:42:50.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.695+0000 7f062fe00700 1 -- 172.21.15.28:0/3294383600 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f062819e5d0 con 0x7f062806b1a0 2024-04-23T14:42:50.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.695+0000 7f0626000700 1 -- 172.21.15.28:0/3294383600 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0614022e10 con 0x7f062806b1a0 2024-04-23T14:42:50.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.695+0000 7f062fe00700 1 -- 172.21.15.28:0/3294383600 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06281a5eb0 con 0x7f062806b1a0 2024-04-23T14:42:50.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.696+0000 7f062fe00700 1 -- 172.21.15.28:0/3294383600 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06280697e0 con 0x7f062806b1a0 2024-04-23T14:42:50.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.699+0000 7f0626000700 1 -- 172.21.15.28:0/3294383600 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f061402a070 con 0x7f062806b1a0 2024-04-23T14:42:50.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.700+0000 7f0626000700 1 --2- 172.21.15.28:0/3294383600 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f06100a3d50 0x7f06100a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:50.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.700+0000 7f0626000700 1 -- 172.21.15.28:0/3294383600 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7f06140d11f0 con 0x7f062806b1a0 2024-04-23T14:42:50.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.700+0000 7f062d400700 1 --2- 172.21.15.28:0/3294383600 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f06100a3d50 0x7f06100a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:50.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.701+0000 7f062d400700 1 --2- 172.21.15.28:0/3294383600 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f06100a3d50 0x7f06100a6210 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f061c00f540 tx=0x7f061c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:50.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.704+0000 7f0626000700 1 -- 172.21.15.28:0/3294383600 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0614013070 con 0x7f062806b1a0 2024-04-23T14:42:50.841 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:50 smithi028 ceph-mon[27810]: pgmap v351: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-04-23T14:42:50.841 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:50 smithi028 ceph-mon[27810]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-23T14:42:50.842 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:50 smithi028 ceph-mon[27810]: Cluster is now healthy 2024-04-23T14:42:50.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:50 smithi038 ceph-mon[32348]: pgmap v351: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-04-23T14:42:50.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:50 smithi038 ceph-mon[32348]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-23T14:42:50.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:50 smithi038 ceph-mon[32348]: Cluster is now healthy 2024-04-23T14:42:50.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.943+0000 7f062fe00700 1 -- 172.21.15.28:0/3294383600 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f062802dfe0 con 0x7f062806b1a0 2024-04-23T14:42:50.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.943+0000 7f0626000700 1 -- 172.21.15.28:0/3294383600 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f061409dac0 con 0x7f062806b1a0 2024-04-23T14:42:50.944 INFO:teuthology.orchestra.run.smithi028.stdout:34359738458 2024-04-23T14:42:50.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.947+0000 7f062fe00700 1 -- 172.21.15.28:0/3294383600 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f06100a3d50 msgr2=0x7f06100a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:50.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.947+0000 7f062fe00700 1 --2- 172.21.15.28:0/3294383600 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f06100a3d50 0x7f06100a6210 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f061c00f540 tx=0x7f061c00b040 comp rx=0 tx=0).stop 2024-04-23T14:42:50.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.947+0000 7f062fe00700 1 -- 172.21.15.28:0/3294383600 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f062806b1a0 msgr2=0x7f0628199870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:50.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.947+0000 7f062fe00700 1 --2- 172.21.15.28:0/3294383600 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f062806b1a0 0x7f0628199870 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0614000c00 tx=0x7f061400bab0 comp rx=0 tx=0).stop 2024-04-23T14:42:50.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.947+0000 7f062fe00700 1 -- 172.21.15.28:0/3294383600 shutdown_connections 2024-04-23T14:42:50.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.947+0000 7f062fe00700 1 --2- 172.21.15.28:0/3294383600 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f06100a3d50 0x7f06100a6210 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:50.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.947+0000 7f062fe00700 1 --2- 172.21.15.28:0/3294383600 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0628108b70 0x7f062819dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:50.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.947+0000 7f062fe00700 1 --2- 172.21.15.28:0/3294383600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f062806a800 0x7f0628199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:50.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.947+0000 7f062fe00700 1 --2- 172.21.15.28:0/3294383600 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f062806b1a0 0x7f0628199870 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:50.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.947+0000 7f062fe00700 1 -- 172.21.15.28:0/3294383600 >> 172.21.15.28:0/3294383600 conn(0x7f06280789a0 msgr2=0x7f06280ffa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:50.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.947+0000 7f062fe00700 1 -- 172.21.15.28:0/3294383600 shutdown_connections 2024-04-23T14:42:50.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:50.947+0000 7f062fe00700 1 -- 172.21.15.28:0/3294383600 wait complete. 2024-04-23T14:42:51.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.001+0000 7f5b31400700 1 -- 172.21.15.28:0/367278542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b2c10a070 msgr2=0x7f5b2c111c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:51.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.001+0000 7f5b31400700 1 --2- 172.21.15.28:0/367278542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b2c10a070 0x7f5b2c111c40 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5b2400b0a0 tx=0x7f5b2400b3b0 comp rx=0 tx=0).stop 2024-04-23T14:42:51.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.001+0000 7f5b31400700 1 -- 172.21.15.28:0/367278542 shutdown_connections 2024-04-23T14:42:51.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.001+0000 7f5b31400700 1 --2- 172.21.15.28:0/367278542 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b2c104f90 0x7f5b2c105370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:51.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.001+0000 7f5b31400700 1 --2- 172.21.15.28:0/367278542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b2c10a070 0x7f5b2c111c40 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:51.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.001+0000 7f5b31400700 1 --2- 172.21.15.28:0/367278542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b2c105940 0x7f5b2c109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:51.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.001+0000 7f5b31400700 1 -- 172.21.15.28:0/367278542 >> 172.21.15.28:0/367278542 conn(0x7f5b2c100820 msgr2=0x7f5b2c102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:51.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.001+0000 7f5b31400700 1 -- 172.21.15.28:0/367278542 shutdown_connections 2024-04-23T14:42:51.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.001+0000 7f5b31400700 1 -- 172.21.15.28:0/367278542 wait complete. 2024-04-23T14:42:51.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.002+0000 7f5b31400700 1 Processor -- start 2024-04-23T14:42:51.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.002+0000 7f5b31400700 1 -- start start 2024-04-23T14:42:51.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.003+0000 7f5b31400700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b2c104f90 0x7f5b2c19b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:51.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.003+0000 7f5b31400700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b2c105940 0x7f5b2c19b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:51.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.003+0000 7f5b31400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b2c10a070 0x7f5b2c19fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:51.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.003+0000 7f5b31400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5b2c115060 con 0x7f5b2c10a070 2024-04-23T14:42:51.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.003+0000 7f5b31400700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5b2c114ee0 con 0x7f5b2c105940 2024-04-23T14:42:51.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.003+0000 7f5b31400700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5b2c1151e0 con 0x7f5b2c104f90 2024-04-23T14:42:51.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.003+0000 7f5b29e00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b2c105940 0x7f5b2c19b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:51.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.003+0000 7f5b2a800700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b2c104f90 0x7f5b2c19b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:51.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.003+0000 7f5b29e00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b2c105940 0x7f5b2c19b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:50218/0 (socket says 172.21.15.28:50218) 2024-04-23T14:42:51.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.003+0000 7f5b2a800700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b2c104f90 0x7f5b2c19b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:59538/0 (socket says 172.21.15.28:59538) 2024-04-23T14:42:51.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.003+0000 7f5b29e00700 1 -- 172.21.15.28:0/1710290511 learned_addr learned my addr 172.21.15.28:0/1710290511 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:42:51.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.004+0000 7f5b29e00700 1 -- 172.21.15.28:0/1710290511 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b2c104f90 msgr2=0x7f5b2c19b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:51.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.004+0000 7f5b29e00700 1 --2- 172.21.15.28:0/1710290511 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b2c104f90 0x7f5b2c19b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:51.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.004+0000 7f5b29e00700 1 -- 172.21.15.28:0/1710290511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b2c10a070 msgr2=0x7f5b2c19fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:51.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.004+0000 7f5b2b200700 1 --2- 172.21.15.28:0/1710290511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b2c10a070 0x7f5b2c19fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:51.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.004+0000 7f5b29e00700 1 --2- 172.21.15.28:0/1710290511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b2c10a070 0x7f5b2c19fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:51.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.004+0000 7f5b29e00700 1 -- 172.21.15.28:0/1710290511 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b2c1a0460 con 0x7f5b2c105940 2024-04-23T14:42:51.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.004+0000 7f5b29e00700 1 --2- 172.21.15.28:0/1710290511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b2c105940 0x7f5b2c19b990 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5b1800e970 tx=0x7f5b1800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:51.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.005+0000 7f5b23400700 1 -- 172.21.15.28:0/1710290511 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b1800cbd0 con 0x7f5b2c105940 2024-04-23T14:42:51.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.005+0000 7f5b23400700 1 -- 172.21.15.28:0/1710290511 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b1800cd70 con 0x7f5b2c105940 2024-04-23T14:42:51.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.005+0000 7f5b23400700 1 -- 172.21.15.28:0/1710290511 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b180056b0 con 0x7f5b2c105940 2024-04-23T14:42:51.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.005+0000 7f5b31400700 1 -- 172.21.15.28:0/1710290511 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b2c1a0750 con 0x7f5b2c105940 2024-04-23T14:42:51.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.005+0000 7f5b31400700 1 -- 172.21.15.28:0/1710290511 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b2c1a7fd0 con 0x7f5b2c105940 2024-04-23T14:42:51.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.007+0000 7f5b31400700 1 -- 172.21.15.28:0/1710290511 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b2c0697e0 con 0x7f5b2c105940 2024-04-23T14:42:51.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.011+0000 7f5b23400700 1 -- 172.21.15.28:0/1710290511 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5b18004750 con 0x7f5b2c105940 2024-04-23T14:42:51.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.012+0000 7f5b23400700 1 --2- 172.21.15.28:0/1710290511 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f5b100a3d50 0x7f5b100a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:51.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.012+0000 7f5b23400700 1 -- 172.21.15.28:0/1710290511 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7f5b180d0970 con 0x7f5b2c105940 2024-04-23T14:42:51.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.013+0000 7f5b2a800700 1 --2- 172.21.15.28:0/1710290511 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f5b100a3d50 0x7f5b100a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:51.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.014+0000 7f5b23400700 1 -- 172.21.15.28:0/1710290511 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5b1809d1f0 con 0x7f5b2c105940 2024-04-23T14:42:51.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.014+0000 7f5b2a800700 1 --2- 172.21.15.28:0/1710290511 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f5b100a3d50 0x7f5b100a6210 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c007be0 tx=0x7f5b1c0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:51.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:50 smithi105 ceph-mon[32555]: pgmap v351: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-04-23T14:42:51.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:50 smithi105 ceph-mon[32555]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-23T14:42:51.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:50 smithi105 ceph-mon[32555]: Cluster is now healthy 2024-04-23T14:42:51.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.259+0000 7f5b31400700 1 -- 172.21.15.28:0/1710290511 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f5b2c02dfe0 con 0x7f5b2c105940 2024-04-23T14:42:51.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.260+0000 7f5b23400700 1 -- 172.21.15.28:0/1710290511 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f5b1809caf0 con 0x7f5b2c105940 2024-04-23T14:42:51.260 INFO:teuthology.orchestra.run.smithi028.stdout:309237645345 2024-04-23T14:42:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.262+0000 7f5b31400700 1 -- 172.21.15.28:0/1710290511 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f5b100a3d50 msgr2=0x7f5b100a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.262+0000 7f5b31400700 1 --2- 172.21.15.28:0/1710290511 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f5b100a3d50 0x7f5b100a6210 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c007be0 tx=0x7f5b1c0073d0 comp rx=0 tx=0).stop 2024-04-23T14:42:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.263+0000 7f5b31400700 1 -- 172.21.15.28:0/1710290511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b2c105940 msgr2=0x7f5b2c19b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.263+0000 7f5b31400700 1 --2- 172.21.15.28:0/1710290511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b2c105940 0x7f5b2c19b990 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5b1800e970 tx=0x7f5b1800ec80 comp rx=0 tx=0).stop 2024-04-23T14:42:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.263+0000 7f5b31400700 1 -- 172.21.15.28:0/1710290511 shutdown_connections 2024-04-23T14:42:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.263+0000 7f5b31400700 1 --2- 172.21.15.28:0/1710290511 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f5b100a3d50 0x7f5b100a6210 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:51.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.263+0000 7f5b31400700 1 --2- 172.21.15.28:0/1710290511 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b2c104f90 0x7f5b2c19b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:51.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.263+0000 7f5b31400700 1 --2- 172.21.15.28:0/1710290511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b2c10a070 0x7f5b2c19fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:51.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.263+0000 7f5b31400700 1 --2- 172.21.15.28:0/1710290511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b2c105940 0x7f5b2c19b990 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:51.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.263+0000 7f5b31400700 1 -- 172.21.15.28:0/1710290511 >> 172.21.15.28:0/1710290511 conn(0x7f5b2c100820 msgr2=0x7f5b2c101380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:51.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.263+0000 7f5b31400700 1 -- 172.21.15.28:0/1710290511 shutdown_connections 2024-04-23T14:42:51.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.263+0000 7f5b31400700 1 -- 172.21.15.28:0/1710290511 wait complete. 2024-04-23T14:42:51.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.486+0000 7f2e6e400700 1 -- 172.21.15.28:0/2051657333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e68104f90 msgr2=0x7f2e68105370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:51.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.486+0000 7f2e6e400700 1 --2- 172.21.15.28:0/2051657333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e68104f90 0x7f2e68105370 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2e580099c0 tx=0x7f2e58009cd0 comp rx=0 tx=0).stop 2024-04-23T14:42:51.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.486+0000 7f2e6e400700 1 -- 172.21.15.28:0/2051657333 shutdown_connections 2024-04-23T14:42:51.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.486+0000 7f2e6e400700 1 --2- 172.21.15.28:0/2051657333 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e6810a070 0x7f2e68111c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:51.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.486+0000 7f2e6e400700 1 --2- 172.21.15.28:0/2051657333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e68104f90 0x7f2e68105370 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:51.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.486+0000 7f2e6e400700 1 --2- 172.21.15.28:0/2051657333 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e68105940 0x7f2e68109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:51.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.486+0000 7f2e6e400700 1 -- 172.21.15.28:0/2051657333 >> 172.21.15.28:0/2051657333 conn(0x7f2e68100820 msgr2=0x7f2e68102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:51.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.486+0000 7f2e6e400700 1 -- 172.21.15.28:0/2051657333 shutdown_connections 2024-04-23T14:42:51.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.486+0000 7f2e6e400700 1 -- 172.21.15.28:0/2051657333 wait complete. 2024-04-23T14:42:51.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.487+0000 7f2e6e400700 1 Processor -- start 2024-04-23T14:42:51.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.487+0000 7f2e6e400700 1 -- start start 2024-04-23T14:42:51.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.488+0000 7f2e6e400700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e68104f90 0x7f2e6819b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:51.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.488+0000 7f2e6e400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e68105940 0x7f2e6819ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:51.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.488+0000 7f2e6e400700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e6810a070 0x7f2e6819fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:51.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.488+0000 7f2e6e400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2e68115050 con 0x7f2e68105940 2024-04-23T14:42:51.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.488+0000 7f2e6e400700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2e68114ed0 con 0x7f2e68104f90 2024-04-23T14:42:51.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.488+0000 7f2e6e400700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2e681151d0 con 0x7f2e6810a070 2024-04-23T14:42:51.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.488+0000 7f2e66a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e68105940 0x7f2e6819ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:51.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.488+0000 7f2e66a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e68105940 0x7f2e6819ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48422/0 (socket says 172.21.15.28:48422) 2024-04-23T14:42:51.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.488+0000 7f2e66a00700 1 -- 172.21.15.28:0/1768686931 learned_addr learned my addr 172.21.15.28:0/1768686931 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:42:51.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.488+0000 7f2e67e00700 1 --2- 172.21.15.28:0/1768686931 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e6810a070 0x7f2e6819fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:51.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.488+0000 7f2e67400700 1 --2- 172.21.15.28:0/1768686931 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e68104f90 0x7f2e6819b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:51.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.489+0000 7f2e67400700 1 -- 172.21.15.28:0/1768686931 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e6810a070 msgr2=0x7f2e6819fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:51.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.489+0000 7f2e67400700 1 --2- 172.21.15.28:0/1768686931 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e6810a070 0x7f2e6819fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:51.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.489+0000 7f2e67400700 1 -- 172.21.15.28:0/1768686931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e68105940 msgr2=0x7f2e6819ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:51.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.489+0000 7f2e67400700 1 --2- 172.21.15.28:0/1768686931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e68105940 0x7f2e6819ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:51.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.489+0000 7f2e67400700 1 -- 172.21.15.28:0/1768686931 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e681a04e0 con 0x7f2e68104f90 2024-04-23T14:42:51.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.489+0000 7f2e67e00700 1 --2- 172.21.15.28:0/1768686931 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e6810a070 0x7f2e6819fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:42:51.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.489+0000 7f2e66a00700 1 --2- 172.21.15.28:0/1768686931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e68105940 0x7f2e6819ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:42:51.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.489+0000 7f2e67400700 1 --2- 172.21.15.28:0/1768686931 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e68104f90 0x7f2e6819b520 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2e5800b580 tx=0x7f2e58003a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:51.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.489+0000 7f2e5fe00700 1 -- 172.21.15.28:0/1768686931 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e58003680 con 0x7f2e68104f90 2024-04-23T14:42:51.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.489+0000 7f2e5fe00700 1 -- 172.21.15.28:0/1768686931 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e580055a0 con 0x7f2e68104f90 2024-04-23T14:42:51.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.489+0000 7f2e6e400700 1 -- 172.21.15.28:0/1768686931 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e681a0770 con 0x7f2e68104f90 2024-04-23T14:42:51.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.489+0000 7f2e5fe00700 1 -- 172.21.15.28:0/1768686931 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e5800fd00 con 0x7f2e68104f90 2024-04-23T14:42:51.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.490+0000 7f2e6e400700 1 -- 172.21.15.28:0/1768686931 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e681a8050 con 0x7f2e68104f90 2024-04-23T14:42:51.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.491+0000 7f2e6e400700 1 -- 172.21.15.28:0/1768686931 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e680697e0 con 0x7f2e68104f90 2024-04-23T14:42:51.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.493+0000 7f2e5fe00700 1 -- 172.21.15.28:0/1768686931 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2e5800f460 con 0x7f2e68104f90 2024-04-23T14:42:51.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.494+0000 7f2e5fe00700 1 --2- 172.21.15.28:0/1768686931 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2e4c0a3ef0 0x7f2e4c0a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:51.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.494+0000 7f2e5fe00700 1 -- 172.21.15.28:0/1768686931 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7f2e580d1370 con 0x7f2e68104f90 2024-04-23T14:42:51.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.495+0000 7f2e66a00700 1 --2- 172.21.15.28:0/1768686931 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2e4c0a3ef0 0x7f2e4c0a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:51.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.495+0000 7f2e66a00700 1 --2- 172.21.15.28:0/1768686931 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2e4c0a3ef0 0x7f2e4c0a63b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2e6810b5c0 tx=0x7f2e54009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:51.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.497+0000 7f2e5fe00700 1 -- 172.21.15.28:0/1768686931 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2e58013070 con 0x7f2e68104f90 2024-04-23T14:42:51.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:51 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3294383600' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-23T14:42:51.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:51 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1710290511' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-23T14:42:51.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.744+0000 7f2e6e400700 1 -- 172.21.15.28:0/1768686931 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f2e6802dfe0 con 0x7f2e68104f90 2024-04-23T14:42:51.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.745+0000 7f2e5fe00700 1 -- 172.21.15.28:0/1768686931 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f2e5809dc40 con 0x7f2e68104f90 2024-04-23T14:42:51.745 INFO:teuthology.orchestra.run.smithi028.stdout:219043332147 2024-04-23T14:42:51.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.747+0000 7f2e6e400700 1 -- 172.21.15.28:0/1768686931 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2e4c0a3ef0 msgr2=0x7f2e4c0a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:51.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.747+0000 7f2e6e400700 1 --2- 172.21.15.28:0/1768686931 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2e4c0a3ef0 0x7f2e4c0a63b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2e6810b5c0 tx=0x7f2e54009260 comp rx=0 tx=0).stop 2024-04-23T14:42:51.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.747+0000 7f2e6e400700 1 -- 172.21.15.28:0/1768686931 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e68104f90 msgr2=0x7f2e6819b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:51.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.747+0000 7f2e6e400700 1 --2- 172.21.15.28:0/1768686931 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e68104f90 0x7f2e6819b520 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2e5800b580 tx=0x7f2e58003a90 comp rx=0 tx=0).stop 2024-04-23T14:42:51.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.748+0000 7f2e6e400700 1 -- 172.21.15.28:0/1768686931 shutdown_connections 2024-04-23T14:42:51.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.748+0000 7f2e6e400700 1 --2- 172.21.15.28:0/1768686931 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2e4c0a3ef0 0x7f2e4c0a63b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:51.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.748+0000 7f2e6e400700 1 --2- 172.21.15.28:0/1768686931 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e6810a070 0x7f2e6819fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:51.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.748+0000 7f2e6e400700 1 --2- 172.21.15.28:0/1768686931 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e68104f90 0x7f2e6819b520 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:51.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.748+0000 7f2e6e400700 1 --2- 172.21.15.28:0/1768686931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e68105940 0x7f2e6819ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:51.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.748+0000 7f2e6e400700 1 -- 172.21.15.28:0/1768686931 >> 172.21.15.28:0/1768686931 conn(0x7f2e68100820 msgr2=0x7f2e68102b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:51.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.748+0000 7f2e6e400700 1 -- 172.21.15.28:0/1768686931 shutdown_connections 2024-04-23T14:42:51.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:51.748+0000 7f2e6e400700 1 -- 172.21.15.28:0/1768686931 wait complete. 2024-04-23T14:42:51.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:51 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3294383600' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-23T14:42:51.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:51 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1710290511' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-23T14:42:52.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:51 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3294383600' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-23T14:42:52.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:51 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1710290511' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-23T14:42:52.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:52 smithi038 ceph-mon[32348]: pgmap v352: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-04-23T14:42:52.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:52 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1768686931' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-23T14:42:52.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:52 smithi028 ceph-mon[27810]: pgmap v352: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-04-23T14:42:52.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:52 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1768686931' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-23T14:42:53.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:52 smithi105 ceph-mon[32555]: pgmap v352: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-04-23T14:42:53.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:52 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1768686931' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-23T14:42:53.864 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:53 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1470361281' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-23T14:42:53.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:53 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1470361281' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-23T14:42:54.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:53 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1470361281' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-23T14:42:54.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.630+0000 7ff49b200700 1 -- 172.21.15.28:0/3267084101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff49c10f0e0 msgr2=0x7ff49c10f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:54.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.630+0000 7ff49b200700 1 --2- 172.21.15.28:0/3267084101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff49c10f0e0 0x7ff49c10f4c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff48c009990 tx=0x7ff48c009ca0 comp rx=0 tx=0).stop 2024-04-23T14:42:54.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.630+0000 7ff49b200700 1 -- 172.21.15.28:0/3267084101 shutdown_connections 2024-04-23T14:42:54.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.630+0000 7ff49b200700 1 --2- 172.21.15.28:0/3267084101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff49c1099f0 0x7ff49c115e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:54.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.630+0000 7ff49b200700 1 --2- 172.21.15.28:0/3267084101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff49c10f0e0 0x7ff49c10f4c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:54.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.630+0000 7ff49b200700 1 --2- 172.21.15.28:0/3267084101 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff49c109030 0x7ff49c1094b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:54.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.630+0000 7ff49b200700 1 -- 172.21.15.28:0/3267084101 >> 172.21.15.28:0/3267084101 conn(0x7ff49c078880 msgr2=0x7ff49c107910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:54.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.631+0000 7ff49b200700 1 -- 172.21.15.28:0/3267084101 shutdown_connections 2024-04-23T14:42:54.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.631+0000 7ff49b200700 1 -- 172.21.15.28:0/3267084101 wait complete. 2024-04-23T14:42:54.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.632+0000 7ff49b200700 1 Processor -- start 2024-04-23T14:42:54.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.632+0000 7ff49b200700 1 -- start start 2024-04-23T14:42:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.632+0000 7ff49b200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff49c109030 0x7ff49c1a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.632+0000 7ff49b200700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff49c1099f0 0x7ff49c1a5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.632+0000 7ff49b200700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff49c10f0e0 0x7ff49c19f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.632+0000 7ff49b200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff49c119320 con 0x7ff49c109030 2024-04-23T14:42:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.632+0000 7ff49b200700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff49c1191a0 con 0x7ff49c1099f0 2024-04-23T14:42:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.632+0000 7ff49b200700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff49c1194a0 con 0x7ff49c10f0e0 2024-04-23T14:42:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.633+0000 7ff499e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff49c109030 0x7ff49c1a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.633+0000 7ff499e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff49c109030 0x7ff49c1a5420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48436/0 (socket says 172.21.15.28:48436) 2024-04-23T14:42:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.633+0000 7ff499e00700 1 -- 172.21.15.28:0/1470361281 learned_addr learned my addr 172.21.15.28:0/1470361281 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:42:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.633+0000 7ff499400700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff49c1099f0 0x7ff49c1a5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.633+0000 7ff49a800700 1 --2- 172.21.15.28:0/1470361281 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff49c10f0e0 0x7ff49c19f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.633+0000 7ff499e00700 1 -- 172.21.15.28:0/1470361281 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff49c10f0e0 msgr2=0x7ff49c19f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.633+0000 7ff499e00700 1 --2- 172.21.15.28:0/1470361281 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff49c10f0e0 0x7ff49c19f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:54.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.633+0000 7ff499e00700 1 -- 172.21.15.28:0/1470361281 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff49c1099f0 msgr2=0x7ff49c1a5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:54.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.633+0000 7ff499e00700 1 --2- 172.21.15.28:0/1470361281 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff49c1099f0 0x7ff49c1a5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:54.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.633+0000 7ff499e00700 1 -- 172.21.15.28:0/1470361281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff49c19fd40 con 0x7ff49c109030 2024-04-23T14:42:54.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.633+0000 7ff499400700 1 --2- 172.21.15.28:0/1470361281 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff49c1099f0 0x7ff49c1a5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:42:54.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.633+0000 7ff499e00700 1 --2- 172.21.15.28:0/1470361281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff49c109030 0x7ff49c1a5420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff48c00bae0 tx=0x7ff48c00bb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:54.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.634+0000 7ff492a00700 1 -- 172.21.15.28:0/1470361281 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff48c026070 con 0x7ff49c109030 2024-04-23T14:42:54.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.634+0000 7ff492a00700 1 -- 172.21.15.28:0/1470361281 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff48c003d70 con 0x7ff49c109030 2024-04-23T14:42:54.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.634+0000 7ff492a00700 1 -- 172.21.15.28:0/1470361281 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff48c021410 con 0x7ff49c109030 2024-04-23T14:42:54.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.634+0000 7ff49b200700 1 -- 172.21.15.28:0/1470361281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff49c19ffd0 con 0x7ff49c109030 2024-04-23T14:42:54.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.634+0000 7ff49b200700 1 -- 172.21.15.28:0/1470361281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff49c1ac2f0 con 0x7ff49c109030 2024-04-23T14:42:54.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.635+0000 7ff49b200700 1 -- 172.21.15.28:0/1470361281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff49c0697e0 con 0x7ff49c109030 2024-04-23T14:42:54.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.637+0000 7ff492a00700 1 -- 172.21.15.28:0/1470361281 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff48c029070 con 0x7ff49c109030 2024-04-23T14:42:54.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.638+0000 7ff492a00700 1 --2- 172.21.15.28:0/1470361281 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ff4800a3cd0 0x7ff4800a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.639+0000 7ff492a00700 1 -- 172.21.15.28:0/1470361281 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7ff48c0d18a0 con 0x7ff49c109030 2024-04-23T14:42:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.639+0000 7ff499400700 1 --2- 172.21.15.28:0/1470361281 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ff4800a3cd0 0x7ff4800a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.640+0000 7ff499400700 1 --2- 172.21.15.28:0/1470361281 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ff4800a3cd0 0x7ff4800a6190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff49c10e800 tx=0x7ff488009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.642+0000 7ff492a00700 1 -- 172.21.15.28:0/1470361281 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff48c030080 con 0x7ff49c109030 2024-04-23T14:42:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.882+0000 7ff49b200700 1 -- 172.21.15.28:0/1470361281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7ff49c02dfe0 con 0x7ff49c109030 2024-04-23T14:42:54.121 INFO:teuthology.orchestra.run.smithi028.stdout:197568495672 2024-04-23T14:42:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.883+0000 7ff492a00700 1 -- 172.21.15.28:0/1470361281 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff48c013070 con 0x7ff49c109030 2024-04-23T14:42:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.886+0000 7ff49b200700 1 -- 172.21.15.28:0/1470361281 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ff4800a3cd0 msgr2=0x7ff4800a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.886+0000 7ff49b200700 1 --2- 172.21.15.28:0/1470361281 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ff4800a3cd0 0x7ff4800a6190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff49c10e800 tx=0x7ff488009260 comp rx=0 tx=0).stop 2024-04-23T14:42:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.886+0000 7ff49b200700 1 -- 172.21.15.28:0/1470361281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff49c109030 msgr2=0x7ff49c1a5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.886+0000 7ff49b200700 1 --2- 172.21.15.28:0/1470361281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff49c109030 0x7ff49c1a5420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff48c00bae0 tx=0x7ff48c00bb10 comp rx=0 tx=0).stop 2024-04-23T14:42:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.887+0000 7ff49b200700 1 -- 172.21.15.28:0/1470361281 shutdown_connections 2024-04-23T14:42:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.887+0000 7ff49b200700 1 --2- 172.21.15.28:0/1470361281 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7ff4800a3cd0 0x7ff4800a6190 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.887+0000 7ff49b200700 1 --2- 172.21.15.28:0/1470361281 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff49c10f0e0 0x7ff49c19f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.887+0000 7ff49b200700 1 --2- 172.21.15.28:0/1470361281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff49c109030 0x7ff49c1a5420 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.887+0000 7ff49b200700 1 --2- 172.21.15.28:0/1470361281 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff49c1099f0 0x7ff49c1a5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.887+0000 7ff49b200700 1 -- 172.21.15.28:0/1470361281 >> 172.21.15.28:0/1470361281 conn(0x7ff49c078880 msgr2=0x7ff49c079930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.887+0000 7ff49b200700 1 -- 172.21.15.28:0/1470361281 shutdown_connections 2024-04-23T14:42:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:52.887+0000 7ff49b200700 1 -- 172.21.15.28:0/1470361281 wait complete. 2024-04-23T14:42:54.411 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738454 got 34359738458 for osd.0 2024-04-23T14:42:54.412 DEBUG:teuthology.parallel:result is None 2024-04-23T14:42:54.680 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:54 smithi028 ceph-mon[27810]: pgmap v353: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 997 B/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-23T14:42:54.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:54 smithi038 ceph-mon[32348]: pgmap v353: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 997 B/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-23T14:42:55.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:54 smithi105 ceph-mon[32555]: pgmap v353: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 997 B/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-23T14:42:55.838 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574926 got 55834574931 for osd.1 2024-04-23T14:42:55.838 DEBUG:teuthology.parallel:result is None 2024-04-23T14:42:56.123 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645338 got 309237645345 for osd.10 2024-04-23T14:42:56.124 DEBUG:teuthology.parallel:result is None 2024-04-23T14:42:56.476 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332142 got 219043332147 for osd.7 2024-04-23T14:42:56.476 DEBUG:teuthology.parallel:result is None 2024-04-23T14:42:56.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.849+0000 7fd95f200700 1 -- 172.21.15.28:0/2357117189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd958075640 msgr2=0x7fd95810efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:56.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.849+0000 7fd95f200700 1 --2- 172.21.15.28:0/2357117189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd958075640 0x7fd95810efb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd944009990 tx=0x7fd944009ca0 comp rx=0 tx=0).stop 2024-04-23T14:42:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.849+0000 7fd95f200700 1 -- 172.21.15.28:0/2357117189 shutdown_connections 2024-04-23T14:42:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.849+0000 7fd95f200700 1 --2- 172.21.15.28:0/2357117189 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd95810f4f0 0x7fd9581118e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.849+0000 7fd95f200700 1 --2- 172.21.15.28:0/2357117189 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd958074d20 0x7fd958075100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.849+0000 7fd95f200700 1 --2- 172.21.15.28:0/2357117189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd958075640 0x7fd95810efb0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.849+0000 7fd95f200700 1 -- 172.21.15.28:0/2357117189 >> 172.21.15.28:0/2357117189 conn(0x7fd958079c50 msgr2=0x7fd95807a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.849+0000 7fd95f200700 1 -- 172.21.15.28:0/2357117189 shutdown_connections 2024-04-23T14:42:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.849+0000 7fd95f200700 1 -- 172.21.15.28:0/2357117189 wait complete. 2024-04-23T14:42:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.850+0000 7fd95f200700 1 Processor -- start 2024-04-23T14:42:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.851+0000 7fd95f200700 1 -- start start 2024-04-23T14:42:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.851+0000 7fd95f200700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd958074d20 0x7fd95819b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.851+0000 7fd95f200700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd958075640 0x7fd95819ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.851+0000 7fd95f200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd95810f4f0 0x7fd95819fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.851+0000 7fd95f200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd958114fd0 con 0x7fd95810f4f0 2024-04-23T14:42:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.851+0000 7fd95f200700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd958114e50 con 0x7fd958075640 2024-04-23T14:42:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.851+0000 7fd95f200700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd958115150 con 0x7fd958074d20 2024-04-23T14:42:56.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.851+0000 7fd95d200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd95810f4f0 0x7fd95819fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:56.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.851+0000 7fd957e00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd958074d20 0x7fd95819b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:56.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.851+0000 7fd95d200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd95810f4f0 0x7fd95819fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48446/0 (socket says 172.21.15.28:48446) 2024-04-23T14:42:56.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.851+0000 7fd95d200700 1 -- 172.21.15.28:0/1576548252 learned_addr learned my addr 172.21.15.28:0/1576548252 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:42:56.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.851+0000 7fd957400700 1 --2- 172.21.15.28:0/1576548252 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd958075640 0x7fd95819ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:56.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.852+0000 7fd957e00700 1 -- 172.21.15.28:0/1576548252 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd958075640 msgr2=0x7fd95819ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:56.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.852+0000 7fd957e00700 1 --2- 172.21.15.28:0/1576548252 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd958075640 0x7fd95819ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:56.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.852+0000 7fd957e00700 1 -- 172.21.15.28:0/1576548252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd95810f4f0 msgr2=0x7fd95819fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:56.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.852+0000 7fd957e00700 1 --2- 172.21.15.28:0/1576548252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd95810f4f0 0x7fd95819fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:56.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.852+0000 7fd957e00700 1 -- 172.21.15.28:0/1576548252 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9581a04a0 con 0x7fd958074d20 2024-04-23T14:42:56.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.852+0000 7fd95d200700 1 --2- 172.21.15.28:0/1576548252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd95810f4f0 0x7fd95819fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:42:56.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.852+0000 7fd957400700 1 --2- 172.21.15.28:0/1576548252 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd958075640 0x7fd95819ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:42:56.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.852+0000 7fd957e00700 1 --2- 172.21.15.28:0/1576548252 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd958074d20 0x7fd95819b4e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd94800e9f0 tx=0x7fd94800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:56.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.852+0000 7fd954c00700 1 -- 172.21.15.28:0/1576548252 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd94800c9c0 con 0x7fd958074d20 2024-04-23T14:42:56.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.853+0000 7fd95f200700 1 -- 172.21.15.28:0/1576548252 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9581a0790 con 0x7fd958074d20 2024-04-23T14:42:56.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.853+0000 7fd954c00700 1 -- 172.21.15.28:0/1576548252 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd94800cb60 con 0x7fd958074d20 2024-04-23T14:42:56.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.853+0000 7fd954c00700 1 -- 172.21.15.28:0/1576548252 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd948018680 con 0x7fd958074d20 2024-04-23T14:42:56.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.853+0000 7fd95f200700 1 -- 172.21.15.28:0/1576548252 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9581a8010 con 0x7fd958074d20 2024-04-23T14:42:56.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.854+0000 7fd95f200700 1 -- 172.21.15.28:0/1576548252 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9580697e0 con 0x7fd958074d20 2024-04-23T14:42:56.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.856+0000 7fd954c00700 1 -- 172.21.15.28:0/1576548252 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd948010430 con 0x7fd958074d20 2024-04-23T14:42:56.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.857+0000 7fd954c00700 1 --2- 172.21.15.28:0/1576548252 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd93c0a3cd0 0x7fd93c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:42:56.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.858+0000 7fd954c00700 1 -- 172.21.15.28:0/1576548252 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7fd9480d06e0 con 0x7fd958074d20 2024-04-23T14:42:56.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.858+0000 7fd957400700 1 --2- 172.21.15.28:0/1576548252 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd93c0a3cd0 0x7fd93c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:42:56.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.858+0000 7fd957400700 1 --2- 172.21.15.28:0/1576548252 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd93c0a3cd0 0x7fd93c0a6190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd9440098e0 tx=0x7fd944005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:42:56.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:56.862+0000 7fd954c00700 1 -- 172.21.15.28:0/1576548252 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd94809cf20 con 0x7fd958074d20 2024-04-23T14:42:57.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:56 smithi105 ceph-mon[32555]: pgmap v354: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-04-23T14:42:57.057 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:56 smithi028 ceph-mon[27810]: pgmap v354: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-04-23T14:42:57.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:57.116+0000 7fd95f200700 1 -- 172.21.15.28:0/1576548252 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fd95802dfe0 con 0x7fd958074d20 2024-04-23T14:42:57.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:57.116+0000 7fd954c00700 1 -- 172.21.15.28:0/1576548252 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd94809c820 con 0x7fd958074d20 2024-04-23T14:42:57.118 INFO:teuthology.orchestra.run.smithi028.stdout:107374182474 2024-04-23T14:42:57.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:57.119+0000 7fd95f200700 1 -- 172.21.15.28:0/1576548252 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd93c0a3cd0 msgr2=0x7fd93c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:57.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:57.119+0000 7fd95f200700 1 --2- 172.21.15.28:0/1576548252 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd93c0a3cd0 0x7fd93c0a6190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd9440098e0 tx=0x7fd944005960 comp rx=0 tx=0).stop 2024-04-23T14:42:57.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:57.119+0000 7fd95f200700 1 -- 172.21.15.28:0/1576548252 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd958074d20 msgr2=0x7fd95819b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:42:57.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:57.119+0000 7fd95f200700 1 --2- 172.21.15.28:0/1576548252 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd958074d20 0x7fd95819b4e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd94800e9f0 tx=0x7fd94800ed00 comp rx=0 tx=0).stop 2024-04-23T14:42:57.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:57.119+0000 7fd95f200700 1 -- 172.21.15.28:0/1576548252 shutdown_connections 2024-04-23T14:42:57.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:57.119+0000 7fd95f200700 1 --2- 172.21.15.28:0/1576548252 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd93c0a3cd0 0x7fd93c0a6190 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:57.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:57.119+0000 7fd95f200700 1 --2- 172.21.15.28:0/1576548252 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd958074d20 0x7fd95819b4e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:57.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:57.119+0000 7fd95f200700 1 --2- 172.21.15.28:0/1576548252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd95810f4f0 0x7fd95819fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:57.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:57.120+0000 7fd95f200700 1 --2- 172.21.15.28:0/1576548252 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd958075640 0x7fd95819ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:42:57.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:57.120+0000 7fd95f200700 1 -- 172.21.15.28:0/1576548252 >> 172.21.15.28:0/1576548252 conn(0x7fd958079c50 msgr2=0x7fd9580ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:42:57.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:57.120+0000 7fd95f200700 1 -- 172.21.15.28:0/1576548252 shutdown_connections 2024-04-23T14:42:57.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:42:57.120+0000 7fd95f200700 1 -- 172.21.15.28:0/1576548252 wait complete. 2024-04-23T14:42:57.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:56 smithi038 ceph-mon[32348]: pgmap v354: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-04-23T14:42:57.584 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495666 got 197568495672 for osd.6 2024-04-23T14:42:57.584 DEBUG:teuthology.parallel:result is None 2024-04-23T14:42:58.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:57 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1576548252' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-23T14:42:58.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:57 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1576548252' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-23T14:42:58.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:57 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1576548252' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-23T14:42:58.300 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182468 got 107374182474 for osd.3 2024-04-23T14:42:58.300 DEBUG:teuthology.parallel:result is None 2024-04-23T14:42:58.548 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:42:59.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:42:58 smithi105 ceph-mon[32555]: pgmap v355: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-23T14:42:59.120 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:42:58 smithi028 ceph-mon[27810]: pgmap v355: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-23T14:42:59.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:42:58 smithi038 ceph-mon[32348]: pgmap v355: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-23T14:43:00.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.058+0000 7feecba00700 1 -- 172.21.15.28:0/258619708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feec410f5a0 msgr2=0x7feec4111990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:00.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.058+0000 7feecba00700 1 --2- 172.21.15.28:0/258619708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feec410f5a0 0x7feec4111990 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7feebc00b0a0 tx=0x7feebc00b3b0 comp rx=0 tx=0).stop 2024-04-23T14:43:00.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.059+0000 7feecba00700 1 -- 172.21.15.28:0/258619708 shutdown_connections 2024-04-23T14:43:00.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.059+0000 7feecba00700 1 --2- 172.21.15.28:0/258619708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feec410f5a0 0x7feec4111990 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:00.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.059+0000 7feecba00700 1 --2- 172.21.15.28:0/258619708 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feec406a800 0x7feec406abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:00.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.059+0000 7feecba00700 1 --2- 172.21.15.28:0/258619708 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feec406b120 0x7feec410f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:00.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.059+0000 7feecba00700 1 -- 172.21.15.28:0/258619708 >> 172.21.15.28:0/258619708 conn(0x7feec40789a0 msgr2=0x7feec4078db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:00.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.059+0000 7feecba00700 1 -- 172.21.15.28:0/258619708 shutdown_connections 2024-04-23T14:43:00.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.059+0000 7feecba00700 1 -- 172.21.15.28:0/258619708 wait complete. 2024-04-23T14:43:00.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.061+0000 7feecba00700 1 Processor -- start 2024-04-23T14:43:00.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.061+0000 7feecba00700 1 -- start start 2024-04-23T14:43:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.062+0000 7feecba00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feec406a800 0x7feec419b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.062+0000 7feecba00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feec406b120 0x7feec419bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.062+0000 7feecba00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feec410f5a0 0x7feec419fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.062+0000 7feecba00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7feec4115040 con 0x7feec410f5a0 2024-04-23T14:43:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.062+0000 7feecba00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7feec4114ec0 con 0x7feec406b120 2024-04-23T14:43:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.062+0000 7feecba00700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7feec41151c0 con 0x7feec406a800 2024-04-23T14:43:00.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.062+0000 7feec3e00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feec406b120 0x7feec419bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:00.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.062+0000 7feec9a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feec410f5a0 0x7feec419fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:00.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.062+0000 7feec3e00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feec406b120 0x7feec419bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:50470/0 (socket says 172.21.15.28:50470) 2024-04-23T14:43:00.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.062+0000 7feec3e00700 1 -- 172.21.15.28:0/2744730565 learned_addr learned my addr 172.21.15.28:0/2744730565 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:00.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.062+0000 7feec9000700 1 --2- 172.21.15.28:0/2744730565 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feec406a800 0x7feec419b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:00.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.062+0000 7feec3e00700 1 -- 172.21.15.28:0/2744730565 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feec406a800 msgr2=0x7feec419b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:00.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.063+0000 7feec3e00700 1 --2- 172.21.15.28:0/2744730565 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feec406a800 0x7feec419b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:00.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.063+0000 7feec3e00700 1 -- 172.21.15.28:0/2744730565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feec410f5a0 msgr2=0x7feec419fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:00.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.063+0000 7feec3e00700 1 --2- 172.21.15.28:0/2744730565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feec410f5a0 0x7feec419fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:00.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.063+0000 7feec3e00700 1 -- 172.21.15.28:0/2744730565 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feec41a05e0 con 0x7feec406b120 2024-04-23T14:43:00.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.063+0000 7feec9a00700 1 --2- 172.21.15.28:0/2744730565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feec410f5a0 0x7feec419fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:43:00.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.063+0000 7feec3e00700 1 --2- 172.21.15.28:0/2744730565 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feec406b120 0x7feec419bb10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7feeb400e9c0 tx=0x7feeb400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:00.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.063+0000 7feec1600700 1 -- 172.21.15.28:0/2744730565 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb400cbd0 con 0x7feec406b120 2024-04-23T14:43:00.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.063+0000 7feec1600700 1 -- 172.21.15.28:0/2744730565 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feeb400ee60 con 0x7feec406b120 2024-04-23T14:43:00.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.063+0000 7feecba00700 1 -- 172.21.15.28:0/2744730565 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feec41a08d0 con 0x7feec406b120 2024-04-23T14:43:00.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.063+0000 7feec1600700 1 -- 172.21.15.28:0/2744730565 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb4005630 con 0x7feec406b120 2024-04-23T14:43:00.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.064+0000 7feecba00700 1 -- 172.21.15.28:0/2744730565 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feec41a8150 con 0x7feec406b120 2024-04-23T14:43:00.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.065+0000 7feecba00700 1 -- 172.21.15.28:0/2744730565 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feec40697e0 con 0x7feec406b120 2024-04-23T14:43:00.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.067+0000 7feec1600700 1 -- 172.21.15.28:0/2744730565 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7feeb4004750 con 0x7feec406b120 2024-04-23T14:43:00.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.068+0000 7feec1600700 1 --2- 172.21.15.28:0/2744730565 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7feeac0a3d50 0x7feeac0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:00.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.068+0000 7feec1600700 1 -- 172.21.15.28:0/2744730565 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7feeb40d07b0 con 0x7feec406b120 2024-04-23T14:43:00.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.068+0000 7feec9000700 1 --2- 172.21.15.28:0/2744730565 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7feeac0a3d50 0x7feeac0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:00.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.069+0000 7feec9000700 1 --2- 172.21.15.28:0/2744730565 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7feeac0a3d50 0x7feeac0a6210 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7feeb8009c60 tx=0x7feeb8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:00.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.072+0000 7feec1600700 1 -- 172.21.15.28:0/2744730565 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feeb409cff0 con 0x7feec406b120 2024-04-23T14:43:00.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.308+0000 7feecba00700 1 -- 172.21.15.28:0/2744730565 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7feec419c930 con 0x7feec406b120 2024-04-23T14:43:00.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.309+0000 7feec1600700 1 -- 172.21.15.28:0/2744730565 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7feeb409c8f0 con 0x7feec406b120 2024-04-23T14:43:00.310 INFO:teuthology.orchestra.run.smithi028.stdout:141733920838 2024-04-23T14:43:00.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.312+0000 7feecba00700 1 -- 172.21.15.28:0/2744730565 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7feeac0a3d50 msgr2=0x7feeac0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:00.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.312+0000 7feecba00700 1 --2- 172.21.15.28:0/2744730565 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7feeac0a3d50 0x7feeac0a6210 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7feeb8009c60 tx=0x7feeb8009260 comp rx=0 tx=0).stop 2024-04-23T14:43:00.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.312+0000 7feecba00700 1 -- 172.21.15.28:0/2744730565 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feec406b120 msgr2=0x7feec419bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:00.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.312+0000 7feecba00700 1 --2- 172.21.15.28:0/2744730565 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feec406b120 0x7feec419bb10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7feeb400e9c0 tx=0x7feeb400ecd0 comp rx=0 tx=0).stop 2024-04-23T14:43:00.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.312+0000 7feecba00700 1 -- 172.21.15.28:0/2744730565 shutdown_connections 2024-04-23T14:43:00.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.312+0000 7feecba00700 1 --2- 172.21.15.28:0/2744730565 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7feeac0a3d50 0x7feeac0a6210 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:00.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.312+0000 7feecba00700 1 --2- 172.21.15.28:0/2744730565 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feec406a800 0x7feec419b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:00.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.312+0000 7feecba00700 1 --2- 172.21.15.28:0/2744730565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feec410f5a0 0x7feec419fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:00.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.312+0000 7feecba00700 1 --2- 172.21.15.28:0/2744730565 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feec406b120 0x7feec419bb10 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:00.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.312+0000 7feecba00700 1 -- 172.21.15.28:0/2744730565 >> 172.21.15.28:0/2744730565 conn(0x7feec40789a0 msgr2=0x7feec4110af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:00.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.313+0000 7feecba00700 1 -- 172.21.15.28:0/2744730565 shutdown_connections 2024-04-23T14:43:00.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:00.313+0000 7feecba00700 1 -- 172.21.15.28:0/2744730565 wait complete. 2024-04-23T14:43:01.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:00 smithi028 ceph-mon[27810]: pgmap v356: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:43:01.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:00 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2744730565' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-23T14:43:01.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:00 smithi105 ceph-mon[32555]: pgmap v356: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:43:01.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:00 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2744730565' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-23T14:43:01.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:00 smithi038 ceph-mon[32348]: pgmap v356: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:43:01.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:00 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2744730565' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-23T14:43:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.333+0000 7fa765000700 1 -- 172.21.15.28:0/3235703884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa760104f90 msgr2=0x7fa760105370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.333+0000 7fa765000700 1 --2- 172.21.15.28:0/3235703884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa760104f90 0x7fa760105370 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa7500099c0 tx=0x7fa750009cd0 comp rx=0 tx=0).stop 2024-04-23T14:43:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.334+0000 7fa765000700 1 -- 172.21.15.28:0/3235703884 shutdown_connections 2024-04-23T14:43:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.334+0000 7fa765000700 1 --2- 172.21.15.28:0/3235703884 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa76010a070 0x7fa760111c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.334+0000 7fa765000700 1 --2- 172.21.15.28:0/3235703884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa760104f90 0x7fa760105370 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.334+0000 7fa765000700 1 --2- 172.21.15.28:0/3235703884 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa760105940 0x7fa760109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.334+0000 7fa765000700 1 -- 172.21.15.28:0/3235703884 >> 172.21.15.28:0/3235703884 conn(0x7fa760100820 msgr2=0x7fa760102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.334+0000 7fa765000700 1 -- 172.21.15.28:0/3235703884 shutdown_connections 2024-04-23T14:43:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.334+0000 7fa765000700 1 -- 172.21.15.28:0/3235703884 wait complete. 2024-04-23T14:43:01.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.335+0000 7fa765000700 1 Processor -- start 2024-04-23T14:43:01.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.335+0000 7fa765000700 1 -- start start 2024-04-23T14:43:01.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.336+0000 7fa765000700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa760104f90 0x7fa76019b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:01.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.336+0000 7fa765000700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa760105940 0x7fa76019ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:01.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.336+0000 7fa765000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa76010a070 0x7fa76019fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:01.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.336+0000 7fa765000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa7601150a0 con 0x7fa76010a070 2024-04-23T14:43:01.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.336+0000 7fa765000700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa760114f20 con 0x7fa760104f90 2024-04-23T14:43:01.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.336+0000 7fa765000700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa760115220 con 0x7fa760105940 2024-04-23T14:43:01.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.336+0000 7fa75e800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa76010a070 0x7fa76019fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:01.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.336+0000 7fa75de00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa760104f90 0x7fa76019b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:01.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.336+0000 7fa75e800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa76010a070 0x7fa76019fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39508/0 (socket says 172.21.15.28:39508) 2024-04-23T14:43:01.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.336+0000 7fa75e800700 1 -- 172.21.15.28:0/3190907030 learned_addr learned my addr 172.21.15.28:0/3190907030 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:01.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.336+0000 7fa75d400700 1 --2- 172.21.15.28:0/3190907030 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa760105940 0x7fa76019ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:01.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.337+0000 7fa75de00700 1 -- 172.21.15.28:0/3190907030 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa760105940 msgr2=0x7fa76019ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:01.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.337+0000 7fa75de00700 1 --2- 172.21.15.28:0/3190907030 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa760105940 0x7fa76019ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:01.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.337+0000 7fa75de00700 1 -- 172.21.15.28:0/3190907030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa76010a070 msgr2=0x7fa76019fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:01.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.337+0000 7fa75de00700 1 --2- 172.21.15.28:0/3190907030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa76010a070 0x7fa76019fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:01.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.337+0000 7fa75de00700 1 -- 172.21.15.28:0/3190907030 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7601a0520 con 0x7fa760104f90 2024-04-23T14:43:01.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.337+0000 7fa75e800700 1 --2- 172.21.15.28:0/3190907030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa76010a070 0x7fa76019fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:43:01.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.337+0000 7fa75de00700 1 --2- 172.21.15.28:0/3190907030 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa760104f90 0x7fa76019b510 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa7500098e0 tx=0x7fa750003e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:01.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.338+0000 7fa756a00700 1 -- 172.21.15.28:0/3190907030 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa750009e60 con 0x7fa760104f90 2024-04-23T14:43:01.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.338+0000 7fa756a00700 1 -- 172.21.15.28:0/3190907030 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa750003680 con 0x7fa760104f90 2024-04-23T14:43:01.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.338+0000 7fa756a00700 1 -- 172.21.15.28:0/3190907030 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75000fd40 con 0x7fa760104f90 2024-04-23T14:43:01.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.338+0000 7fa765000700 1 -- 172.21.15.28:0/3190907030 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7601a07b0 con 0x7fa760104f90 2024-04-23T14:43:01.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.338+0000 7fa765000700 1 -- 172.21.15.28:0/3190907030 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7601a8090 con 0x7fa760104f90 2024-04-23T14:43:01.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.339+0000 7fa765000700 1 -- 172.21.15.28:0/3190907030 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7600697e0 con 0x7fa760104f90 2024-04-23T14:43:01.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.341+0000 7fa756a00700 1 -- 172.21.15.28:0/3190907030 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa75000f460 con 0x7fa760104f90 2024-04-23T14:43:01.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.342+0000 7fa756a00700 1 --2- 172.21.15.28:0/3190907030 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fa7440a3d50 0x7fa7440a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:01.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.342+0000 7fa756a00700 1 -- 172.21.15.28:0/3190907030 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7fa7500d1770 con 0x7fa760104f90 2024-04-23T14:43:01.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.343+0000 7fa75d400700 1 --2- 172.21.15.28:0/3190907030 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fa7440a3d50 0x7fa7440a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:01.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.343+0000 7fa75d400700 1 --2- 172.21.15.28:0/3190907030 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fa7440a3d50 0x7fa7440a6210 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa76010b680 tx=0x7fa74c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:01.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.345+0000 7fa756a00700 1 -- 172.21.15.28:0/3190907030 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa750013070 con 0x7fa760104f90 2024-04-23T14:43:01.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.583+0000 7fa765000700 1 -- 172.21.15.28:0/3190907030 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fa76002dfe0 con 0x7fa760104f90 2024-04-23T14:43:01.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.583+0000 7fa756a00700 1 -- 172.21.15.28:0/3190907030 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fa75009e040 con 0x7fa760104f90 2024-04-23T14:43:01.585 INFO:teuthology.orchestra.run.smithi028.stdout:287762808872 2024-04-23T14:43:01.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.586+0000 7fa765000700 1 -- 172.21.15.28:0/3190907030 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fa7440a3d50 msgr2=0x7fa7440a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:01.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.587+0000 7fa765000700 1 --2- 172.21.15.28:0/3190907030 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fa7440a3d50 0x7fa7440a6210 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa76010b680 tx=0x7fa74c00b410 comp rx=0 tx=0).stop 2024-04-23T14:43:01.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.587+0000 7fa765000700 1 -- 172.21.15.28:0/3190907030 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa760104f90 msgr2=0x7fa76019b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:01.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.587+0000 7fa765000700 1 --2- 172.21.15.28:0/3190907030 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa760104f90 0x7fa76019b510 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa7500098e0 tx=0x7fa750003e00 comp rx=0 tx=0).stop 2024-04-23T14:43:01.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.587+0000 7fa765000700 1 -- 172.21.15.28:0/3190907030 shutdown_connections 2024-04-23T14:43:01.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.587+0000 7fa765000700 1 --2- 172.21.15.28:0/3190907030 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fa7440a3d50 0x7fa7440a6210 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:01.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.587+0000 7fa765000700 1 --2- 172.21.15.28:0/3190907030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa76010a070 0x7fa76019fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:01.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.587+0000 7fa765000700 1 --2- 172.21.15.28:0/3190907030 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa760104f90 0x7fa76019b510 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:01.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.587+0000 7fa765000700 1 --2- 172.21.15.28:0/3190907030 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa760105940 0x7fa76019ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:01.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.587+0000 7fa765000700 1 -- 172.21.15.28:0/3190907030 >> 172.21.15.28:0/3190907030 conn(0x7fa760100820 msgr2=0x7fa760102b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:01.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.587+0000 7fa765000700 1 -- 172.21.15.28:0/3190907030 shutdown_connections 2024-04-23T14:43:01.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:01.587+0000 7fa765000700 1 -- 172.21.15.28:0/3190907030 wait complete. 2024-04-23T14:43:01.604 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920832 got 141733920838 for osd.4 2024-04-23T14:43:01.604 DEBUG:teuthology.parallel:result is None 2024-04-23T14:43:01.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:01 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3190907030' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-23T14:43:02.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:01 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3190907030' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-23T14:43:02.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:01 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3190907030' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-23T14:43:02.313 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808867 got 287762808872 for osd.9 2024-04-23T14:43:02.314 DEBUG:teuthology.parallel:result is None 2024-04-23T14:43:02.314 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-23T14:43:02.314 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T14:43:02.785 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:43:03.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:03 smithi028 ceph-mon[27810]: pgmap v357: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:43:03.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:03 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:43:03.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:03 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:43:03.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:03 smithi105 ceph-mon[32555]: pgmap v357: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:43:03.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:03 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:43:03.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:03 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:43:03.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:03 smithi038 ceph-mon[32348]: pgmap v357: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:43:03.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:03 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:43:03.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:03 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:43:04.748 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:04 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:43:04.749 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:04 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:43:04.749 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:04 smithi028 ceph-mon[27810]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:43:04.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:04 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:43:04.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:04 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:43:04.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:04 smithi105 ceph-mon[32555]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:43:04.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:04 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:43:04.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:04 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:43:04.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:04 smithi038 ceph-mon[32348]: from='mgr.14152 172.21.15.28:0/3173282737' entity='mgr.x' 2024-04-23T14:43:04.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.919+0000 7f6061000700 1 -- 172.21.15.28:0/2450451959 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f605c10f0e0 msgr2=0x7f605c10f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:04.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.919+0000 7f6061000700 1 --2- 172.21.15.28:0/2450451959 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f605c10f0e0 0x7f605c10f4c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f604c0098f0 tx=0x7f604c009c00 comp rx=0 tx=0).stop 2024-04-23T14:43:04.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.919+0000 7f6061000700 1 -- 172.21.15.28:0/2450451959 shutdown_connections 2024-04-23T14:43:04.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.919+0000 7f6061000700 1 --2- 172.21.15.28:0/2450451959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f605c1099f0 0x7f605c115e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:04.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.919+0000 7f6061000700 1 --2- 172.21.15.28:0/2450451959 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f605c10f0e0 0x7f605c10f4c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:04.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.919+0000 7f6061000700 1 --2- 172.21.15.28:0/2450451959 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f605c109030 0x7f605c1094b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:04.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.919+0000 7f6061000700 1 -- 172.21.15.28:0/2450451959 >> 172.21.15.28:0/2450451959 conn(0x7f605c078880 msgr2=0x7f605c107910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:04.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.919+0000 7f6061000700 1 -- 172.21.15.28:0/2450451959 shutdown_connections 2024-04-23T14:43:04.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.919+0000 7f6061000700 1 -- 172.21.15.28:0/2450451959 wait complete. 2024-04-23T14:43:04.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.920+0000 7f6061000700 1 Processor -- start 2024-04-23T14:43:04.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.921+0000 7f6061000700 1 -- start start 2024-04-23T14:43:04.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.921+0000 7f6061000700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f605c109030 0x7f605c1a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:04.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.921+0000 7f6061000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f605c1099f0 0x7f605c1a5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:04.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.921+0000 7f6061000700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f605c10f0e0 0x7f605c19f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:04.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.921+0000 7f6061000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f605c119370 con 0x7f605c1099f0 2024-04-23T14:43:04.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.921+0000 7f6061000700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f605c1191f0 con 0x7f605c10f0e0 2024-04-23T14:43:04.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.921+0000 7f6061000700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f605c1194f0 con 0x7f605c109030 2024-04-23T14:43:04.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.921+0000 7f6053e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f605c1099f0 0x7f605c1a5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:04.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.921+0000 7f6053e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f605c1099f0 0x7f605c1a5970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39530/0 (socket says 172.21.15.28:39530) 2024-04-23T14:43:04.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.921+0000 7f6053e00700 1 -- 172.21.15.28:0/3752144732 learned_addr learned my addr 172.21.15.28:0/3752144732 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:04.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.921+0000 7f605b400700 1 --2- 172.21.15.28:0/3752144732 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f605c109030 0x7f605c1a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:04.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.922+0000 7f605be00700 1 --2- 172.21.15.28:0/3752144732 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f605c10f0e0 0x7f605c19f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:04.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.922+0000 7f6053e00700 1 -- 172.21.15.28:0/3752144732 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f605c109030 msgr2=0x7f605c1a5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:04.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.922+0000 7f6053e00700 1 --2- 172.21.15.28:0/3752144732 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f605c109030 0x7f605c1a5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:04.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.922+0000 7f6053e00700 1 -- 172.21.15.28:0/3752144732 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f605c10f0e0 msgr2=0x7f605c19f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:04.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.922+0000 7f6053e00700 1 --2- 172.21.15.28:0/3752144732 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f605c10f0e0 0x7f605c19f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:04.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.922+0000 7f6053e00700 1 -- 172.21.15.28:0/3752144732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f605c19fd50 con 0x7f605c1099f0 2024-04-23T14:43:04.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.922+0000 7f6053e00700 1 --2- 172.21.15.28:0/3752144732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f605c1099f0 0x7f605c1a5970 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f604400e9c0 tx=0x7f604400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:04.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.923+0000 7f6058c00700 1 -- 172.21.15.28:0/3752144732 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f604400cbd0 con 0x7f605c1099f0 2024-04-23T14:43:04.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.923+0000 7f6058c00700 1 -- 172.21.15.28:0/3752144732 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f604400ee60 con 0x7f605c1099f0 2024-04-23T14:43:04.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.923+0000 7f6061000700 1 -- 172.21.15.28:0/3752144732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f605c1a0040 con 0x7f605c1099f0 2024-04-23T14:43:04.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.923+0000 7f6058c00700 1 -- 172.21.15.28:0/3752144732 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6044005630 con 0x7f605c1099f0 2024-04-23T14:43:04.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.923+0000 7f6061000700 1 -- 172.21.15.28:0/3752144732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f605c1ac300 con 0x7f605c1099f0 2024-04-23T14:43:04.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.924+0000 7f6061000700 1 -- 172.21.15.28:0/3752144732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f605c111430 con 0x7f605c1099f0 2024-04-23T14:43:04.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.926+0000 7f6058c00700 1 -- 172.21.15.28:0/3752144732 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6044004750 con 0x7f605c1099f0 2024-04-23T14:43:04.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.927+0000 7f6058c00700 1 --2- 172.21.15.28:0/3752144732 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f60400a3cd0 0x7f60400a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:04.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.928+0000 7f6058c00700 1 -- 172.21.15.28:0/3752144732 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7f60440d0a30 con 0x7f605c1099f0 2024-04-23T14:43:04.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.928+0000 7f605b400700 1 --2- 172.21.15.28:0/3752144732 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f60400a3cd0 0x7f60400a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:04.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.929+0000 7f605b400700 1 --2- 172.21.15.28:0/3752144732 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f60400a3cd0 0x7f60400a6190 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f604c009810 tx=0x7f604c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:04.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:04.930+0000 7f6058c00700 1 -- 172.21.15.28:0/3752144732 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6044053ff0 con 0x7f605c1099f0 2024-04-23T14:43:05.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:05.158+0000 7f6061000700 1 -- 172.21.15.28:0/3752144732 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f605c1a0b20 con 0x7f60400a3cd0 2024-04-23T14:43:05.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:05.172+0000 7f6058c00700 1 -- 172.21.15.28:0/3752144732 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258631 (secure 0 0 0) 0x7f604c066260 con 0x7f60400a3cd0 2024-04-23T14:43:05.175 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:43:05.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:05.178+0000 7f6061000700 1 -- 172.21.15.28:0/3752144732 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f60400a3cd0 msgr2=0x7f60400a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:05.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:05.178+0000 7f6061000700 1 --2- 172.21.15.28:0/3752144732 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f60400a3cd0 0x7f60400a6190 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f604c009810 tx=0x7f604c019040 comp rx=0 tx=0).stop 2024-04-23T14:43:05.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:05.178+0000 7f6061000700 1 -- 172.21.15.28:0/3752144732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f605c1099f0 msgr2=0x7f605c1a5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:05.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:05.178+0000 7f6061000700 1 --2- 172.21.15.28:0/3752144732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f605c1099f0 0x7f605c1a5970 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f604400e9c0 tx=0x7f604400ecd0 comp rx=0 tx=0).stop 2024-04-23T14:43:05.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:05.178+0000 7f6061000700 1 -- 172.21.15.28:0/3752144732 shutdown_connections 2024-04-23T14:43:05.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:05.178+0000 7f6061000700 1 --2- 172.21.15.28:0/3752144732 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f60400a3cd0 0x7f60400a6190 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:05.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:05.178+0000 7f6061000700 1 --2- 172.21.15.28:0/3752144732 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f605c109030 0x7f605c1a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:05.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:05.178+0000 7f6061000700 1 --2- 172.21.15.28:0/3752144732 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f605c10f0e0 0x7f605c19f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:05.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:05.178+0000 7f6061000700 1 --2- 172.21.15.28:0/3752144732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f605c1099f0 0x7f605c1a5970 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:05.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:05.178+0000 7f6061000700 1 -- 172.21.15.28:0/3752144732 >> 172.21.15.28:0/3752144732 conn(0x7f605c078880 msgr2=0x7f605c079ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:05.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:05.179+0000 7f6061000700 1 -- 172.21.15.28:0/3752144732 shutdown_connections 2024-04-23T14:43:05.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:05.179+0000 7f6061000700 1 -- 172.21.15.28:0/3752144732 wait complete. 2024-04-23T14:43:05.181 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-23T14:43:05.466 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:05 smithi028 ceph-mon[27810]: pgmap v358: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:43:05.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:05 smithi105 ceph-mon[32555]: pgmap v358: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:43:05.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:05 smithi038 ceph-mon[32348]: pgmap v358: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:43:05.882 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":358,"stamp":"2024-04-23T14:43:03.702755+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653296,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":302,"num_read_kb":265,"num_write":262,"num_write_kb":3025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378980,"kb_used_data":13116,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124562844,"statfs":{"total":1151940427776,"available":1151552352256,"internally_reserved":0,"allocated":13430784,"data_stored":7428408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004987"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.439594+0000","last_change":"2024-04-23T14:41:40.033061+0000","last_active":"2024-04-23T14:42:43.439594+0000","last_peered":"2024-04-23T14:42:43.439594+0000","last_clean":"2024-04-23T14:42:43.439594+0000","last_became_active":"2024-04-23T14:41:40.032750+0000","last_became_peered":"2024-04-23T14:41:40.032750+0000","last_unstale":"2024-04-23T14:42:43.439594+0000","last_undegraded":"2024-04-23T14:42:43.439594+0000","last_fullsized":"2024-04-23T14:42:43.439594+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:58:03.451539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.443362+0000","last_change":"2024-04-23T14:41:40.037435+0000","last_active":"2024-04-23T14:42:43.443362+0000","last_peered":"2024-04-23T14:42:43.443362+0000","last_clean":"2024-04-23T14:42:43.443362+0000","last_became_active":"2024-04-23T14:41:40.036655+0000","last_became_peered":"2024-04-23T14:41:40.036655+0000","last_unstale":"2024-04-23T14:42:43.443362+0000","last_undegraded":"2024-04-23T14:42:43.443362+0000","last_fullsized":"2024-04-23T14:42:43.443362+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:51:58.847850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.443240+0000","last_change":"2024-04-23T14:41:40.024977+0000","last_active":"2024-04-23T14:42:43.443240+0000","last_peered":"2024-04-23T14:42:43.443240+0000","last_clean":"2024-04-23T14:42:43.443240+0000","last_became_active":"2024-04-23T14:41:40.023879+0000","last_became_peered":"2024-04-23T14:41:40.023879+0000","last_unstale":"2024-04-23T14:42:43.443240+0000","last_undegraded":"2024-04-23T14:42:43.443240+0000","last_fullsized":"2024-04-23T14:42:43.443240+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:03:40.672661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.820477+0000","last_change":"2024-04-23T14:41:40.017757+0000","last_active":"2024-04-23T14:42:43.820477+0000","last_peered":"2024-04-23T14:42:43.820477+0000","last_clean":"2024-04-23T14:42:43.820477+0000","last_became_active":"2024-04-23T14:41:40.013996+0000","last_became_peered":"2024-04-23T14:41:40.013996+0000","last_unstale":"2024-04-23T14:42:43.820477+0000","last_undegraded":"2024-04-23T14:42:43.820477+0000","last_fullsized":"2024-04-23T14:42:43.820477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:20:03.376733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.438683+0000","last_change":"2024-04-23T14:41:40.038247+0000","last_active":"2024-04-23T14:42:43.438683+0000","last_peered":"2024-04-23T14:42:43.438683+0000","last_clean":"2024-04-23T14:42:43.438683+0000","last_became_active":"2024-04-23T14:41:40.037934+0000","last_became_peered":"2024-04-23T14:41:40.037934+0000","last_unstale":"2024-04-23T14:42:43.438683+0000","last_undegraded":"2024-04-23T14:42:43.438683+0000","last_fullsized":"2024-04-23T14:42:43.438683+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:15.637733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.440257+0000","last_change":"2024-04-23T14:41:40.015341+0000","last_active":"2024-04-23T14:42:43.440257+0000","last_peered":"2024-04-23T14:42:43.440257+0000","last_clean":"2024-04-23T14:42:43.440257+0000","last_became_active":"2024-04-23T14:41:40.013627+0000","last_became_peered":"2024-04-23T14:41:40.013627+0000","last_unstale":"2024-04-23T14:42:43.440257+0000","last_undegraded":"2024-04-23T14:42:43.440257+0000","last_fullsized":"2024-04-23T14:42:43.440257+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:05:36.248107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.437295+0000","last_change":"2024-04-23T14:41:40.010950+0000","last_active":"2024-04-23T14:42:43.437295+0000","last_peered":"2024-04-23T14:42:43.437295+0000","last_clean":"2024-04-23T14:42:43.437295+0000","last_became_active":"2024-04-23T14:41:40.010584+0000","last_became_peered":"2024-04-23T14:41:40.010584+0000","last_unstale":"2024-04-23T14:42:43.437295+0000","last_undegraded":"2024-04-23T14:42:43.437295+0000","last_fullsized":"2024-04-23T14:42:43.437295+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:46:25.379243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.528280+0000","last_change":"2024-04-23T14:41:40.031198+0000","last_active":"2024-04-23T14:42:43.528280+0000","last_peered":"2024-04-23T14:42:43.528280+0000","last_clean":"2024-04-23T14:42:43.528280+0000","last_became_active":"2024-04-23T14:41:40.030652+0000","last_became_peered":"2024-04-23T14:41:40.030652+0000","last_unstale":"2024-04-23T14:42:43.528280+0000","last_undegraded":"2024-04-23T14:42:43.528280+0000","last_fullsized":"2024-04-23T14:42:43.528280+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:26:00.520182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442311+0000","last_change":"2024-04-23T14:41:40.009820+0000","last_active":"2024-04-23T14:42:43.442311+0000","last_peered":"2024-04-23T14:42:43.442311+0000","last_clean":"2024-04-23T14:42:43.442311+0000","last_became_active":"2024-04-23T14:41:40.009468+0000","last_became_peered":"2024-04-23T14:41:40.009468+0000","last_unstale":"2024-04-23T14:42:43.442311+0000","last_undegraded":"2024-04-23T14:42:43.442311+0000","last_fullsized":"2024-04-23T14:42:43.442311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:03:50.178050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.441311+0000","last_change":"2024-04-23T14:41:40.008363+0000","last_active":"2024-04-23T14:42:43.441311+0000","last_peered":"2024-04-23T14:42:43.441311+0000","last_clean":"2024-04-23T14:42:43.441311+0000","last_became_active":"2024-04-23T14:41:40.006823+0000","last_became_peered":"2024-04-23T14:41:40.006823+0000","last_unstale":"2024-04-23T14:42:43.441311+0000","last_undegraded":"2024-04-23T14:42:43.441311+0000","last_fullsized":"2024-04-23T14:42:43.441311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:12:33.718815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.437518+0000","last_change":"2024-04-23T14:41:40.021069+0000","last_active":"2024-04-23T14:42:43.437518+0000","last_peered":"2024-04-23T14:42:43.437518+0000","last_clean":"2024-04-23T14:42:43.437518+0000","last_became_active":"2024-04-23T14:41:40.020334+0000","last_became_peered":"2024-04-23T14:41:40.020334+0000","last_unstale":"2024-04-23T14:42:43.437518+0000","last_undegraded":"2024-04-23T14:42:43.437518+0000","last_fullsized":"2024-04-23T14:42:43.437518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:50:56.560928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.820435+0000","last_change":"2024-04-23T14:41:40.017746+0000","last_active":"2024-04-23T14:42:43.820435+0000","last_peered":"2024-04-23T14:42:43.820435+0000","last_clean":"2024-04-23T14:42:43.820435+0000","last_became_active":"2024-04-23T14:41:40.014495+0000","last_became_peered":"2024-04-23T14:41:40.014495+0000","last_unstale":"2024-04-23T14:42:43.820435+0000","last_undegraded":"2024-04-23T14:42:43.820435+0000","last_fullsized":"2024-04-23T14:42:43.820435+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:23:22.188110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.593556+0000","last_change":"2024-04-23T14:41:40.021429+0000","last_active":"2024-04-23T14:42:43.593556+0000","last_peered":"2024-04-23T14:42:43.593556+0000","last_clean":"2024-04-23T14:42:43.593556+0000","last_became_active":"2024-04-23T14:41:40.021087+0000","last_became_peered":"2024-04-23T14:41:40.021087+0000","last_unstale":"2024-04-23T14:42:43.593556+0000","last_undegraded":"2024-04-23T14:42:43.593556+0000","last_fullsized":"2024-04-23T14:42:43.593556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:51:16.633253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442705+0000","last_change":"2024-04-23T14:41:40.031839+0000","last_active":"2024-04-23T14:42:43.442705+0000","last_peered":"2024-04-23T14:42:43.442705+0000","last_clean":"2024-04-23T14:42:43.442705+0000","last_became_active":"2024-04-23T14:41:40.031463+0000","last_became_peered":"2024-04-23T14:41:40.031463+0000","last_unstale":"2024-04-23T14:42:43.442705+0000","last_undegraded":"2024-04-23T14:42:43.442705+0000","last_fullsized":"2024-04-23T14:42:43.442705+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:51:51.200463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442441+0000","last_change":"2024-04-23T14:41:40.042575+0000","last_active":"2024-04-23T14:42:43.442441+0000","last_peered":"2024-04-23T14:42:43.442441+0000","last_clean":"2024-04-23T14:42:43.442441+0000","last_became_active":"2024-04-23T14:41:40.033758+0000","last_became_peered":"2024-04-23T14:41:40.033758+0000","last_unstale":"2024-04-23T14:42:43.442441+0000","last_undegraded":"2024-04-23T14:42:43.442441+0000","last_fullsized":"2024-04-23T14:42:43.442441+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:55:17.338169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442663+0000","last_change":"2024-04-23T14:41:40.036214+0000","last_active":"2024-04-23T14:42:43.442663+0000","last_peered":"2024-04-23T14:42:43.442663+0000","last_clean":"2024-04-23T14:42:43.442663+0000","last_became_active":"2024-04-23T14:41:40.035892+0000","last_became_peered":"2024-04-23T14:41:40.035892+0000","last_unstale":"2024-04-23T14:42:43.442663+0000","last_undegraded":"2024-04-23T14:42:43.442663+0000","last_fullsized":"2024-04-23T14:42:43.442663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:29:18.668690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.439826+0000","last_change":"2024-04-23T14:41:40.029837+0000","last_active":"2024-04-23T14:42:43.439826+0000","last_peered":"2024-04-23T14:42:43.439826+0000","last_clean":"2024-04-23T14:42:43.439826+0000","last_became_active":"2024-04-23T14:41:40.029472+0000","last_became_peered":"2024-04-23T14:41:40.029472+0000","last_unstale":"2024-04-23T14:42:43.439826+0000","last_undegraded":"2024-04-23T14:42:43.439826+0000","last_fullsized":"2024-04-23T14:42:43.439826+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:28:52.194898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.677920+0000","last_change":"2024-04-23T14:41:40.015631+0000","last_active":"2024-04-23T14:42:43.677920+0000","last_peered":"2024-04-23T14:42:43.677920+0000","last_clean":"2024-04-23T14:42:43.677920+0000","last_became_active":"2024-04-23T14:41:40.015211+0000","last_became_peered":"2024-04-23T14:41:40.015211+0000","last_unstale":"2024-04-23T14:42:43.677920+0000","last_undegraded":"2024-04-23T14:42:43.677920+0000","last_fullsized":"2024-04-23T14:42:43.677920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:03:33.485364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.680530+0000","last_change":"2024-04-23T14:42:43.680530+0000","last_active":"2024-04-23T14:42:43.680530+0000","last_peered":"2024-04-23T14:42:43.680530+0000","last_clean":"2024-04-23T14:42:43.680530+0000","last_became_active":"2024-04-23T14:42:43.680249+0000","last_became_peered":"2024-04-23T14:42:43.680249+0000","last_unstale":"2024-04-23T14:42:43.680530+0000","last_undegraded":"2024-04-23T14:42:43.680530+0000","last_fullsized":"2024-04-23T14:42:43.680530+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:55:36.334572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,6],"acting":[2,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.677891+0000","last_change":"2024-04-23T14:41:39.017031+0000","last_active":"2024-04-23T14:42:43.677891+0000","last_peered":"2024-04-23T14:42:43.677891+0000","last_clean":"2024-04-23T14:42:43.677891+0000","last_became_active":"2024-04-23T14:41:39.016699+0000","last_became_peered":"2024-04-23T14:41:39.016699+0000","last_unstale":"2024-04-23T14:42:43.677891+0000","last_undegraded":"2024-04-23T14:42:43.677891+0000","last_fullsized":"2024-04-23T14:42:43.677891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:17:49.812601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.530569+0000","last_change":"2024-04-23T14:41:40.031957+0000","last_active":"2024-04-23T14:42:43.530569+0000","last_peered":"2024-04-23T14:42:43.530569+0000","last_clean":"2024-04-23T14:42:43.530569+0000","last_became_active":"2024-04-23T14:41:40.028151+0000","last_became_peered":"2024-04-23T14:41:40.028151+0000","last_unstale":"2024-04-23T14:42:43.530569+0000","last_undegraded":"2024-04-23T14:42:43.530569+0000","last_fullsized":"2024-04-23T14:42:43.530569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:46:48.455032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.530852+0000","last_change":"2024-04-23T14:41:39.022503+0000","last_active":"2024-04-23T14:42:43.530852+0000","last_peered":"2024-04-23T14:42:43.530852+0000","last_clean":"2024-04-23T14:42:43.530852+0000","last_became_active":"2024-04-23T14:41:39.022093+0000","last_became_peered":"2024-04-23T14:41:39.022093+0000","last_unstale":"2024-04-23T14:42:43.530852+0000","last_undegraded":"2024-04-23T14:42:43.530852+0000","last_fullsized":"2024-04-23T14:42:43.530852+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:48:27.861140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442653+0000","last_change":"2024-04-23T14:41:40.032878+0000","last_active":"2024-04-23T14:42:43.442653+0000","last_peered":"2024-04-23T14:42:43.442653+0000","last_clean":"2024-04-23T14:42:43.442653+0000","last_became_active":"2024-04-23T14:41:40.027473+0000","last_became_peered":"2024-04-23T14:41:40.027473+0000","last_unstale":"2024-04-23T14:42:43.442653+0000","last_undegraded":"2024-04-23T14:42:43.442653+0000","last_fullsized":"2024-04-23T14:42:43.442653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:00:17.677306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.449136+0000","last_change":"2024-04-23T14:42:43.448728+0000","last_active":"2024-04-23T14:42:43.449136+0000","last_peered":"2024-04-23T14:42:43.449136+0000","last_clean":"2024-04-23T14:42:43.449136+0000","last_became_active":"2024-04-23T14:42:43.443993+0000","last_became_peered":"2024-04-23T14:42:43.443993+0000","last_unstale":"2024-04-23T14:42:43.449136+0000","last_undegraded":"2024-04-23T14:42:43.449136+0000","last_fullsized":"2024-04-23T14:42:43.449136+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:04:16.430822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442116+0000","last_change":"2024-04-23T14:41:40.019955+0000","last_active":"2024-04-23T14:42:43.442116+0000","last_peered":"2024-04-23T14:42:43.442116+0000","last_clean":"2024-04-23T14:42:43.442116+0000","last_became_active":"2024-04-23T14:41:40.019579+0000","last_became_peered":"2024-04-23T14:41:40.019579+0000","last_unstale":"2024-04-23T14:42:43.442116+0000","last_undegraded":"2024-04-23T14:42:43.442116+0000","last_fullsized":"2024-04-23T14:42:43.442116+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:13:14.330051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.447172+0000","last_change":"2024-04-23T14:42:43.446652+0000","last_active":"2024-04-23T14:42:43.447172+0000","last_peered":"2024-04-23T14:42:43.447172+0000","last_clean":"2024-04-23T14:42:43.447172+0000","last_became_active":"2024-04-23T14:42:43.443549+0000","last_became_peered":"2024-04-23T14:42:43.443549+0000","last_unstale":"2024-04-23T14:42:43.447172+0000","last_undegraded":"2024-04-23T14:42:43.447172+0000","last_fullsized":"2024-04-23T14:42:43.447172+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:31:12.927663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442183+0000","last_change":"2024-04-23T14:41:40.015592+0000","last_active":"2024-04-23T14:42:43.442183+0000","last_peered":"2024-04-23T14:42:43.442183+0000","last_clean":"2024-04-23T14:42:43.442183+0000","last_became_active":"2024-04-23T14:41:40.015292+0000","last_became_peered":"2024-04-23T14:41:40.015292+0000","last_unstale":"2024-04-23T14:42:43.442183+0000","last_undegraded":"2024-04-23T14:42:43.442183+0000","last_fullsized":"2024-04-23T14:42:43.442183+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:52:01.641962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.443168+0000","last_change":"2024-04-23T14:41:39.008526+0000","last_active":"2024-04-23T14:42:43.443168+0000","last_peered":"2024-04-23T14:42:43.443168+0000","last_clean":"2024-04-23T14:42:43.443168+0000","last_became_active":"2024-04-23T14:41:39.008222+0000","last_became_peered":"2024-04-23T14:41:39.008222+0000","last_unstale":"2024-04-23T14:42:43.443168+0000","last_undegraded":"2024-04-23T14:42:43.443168+0000","last_fullsized":"2024-04-23T14:42:43.443168+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:23:09.191426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.677846+0000","last_change":"2024-04-23T14:42:43.677846+0000","last_active":"2024-04-23T14:42:43.677846+0000","last_peered":"2024-04-23T14:42:43.677846+0000","last_clean":"2024-04-23T14:42:43.677846+0000","last_became_active":"2024-04-23T14:42:43.677413+0000","last_became_peered":"2024-04-23T14:42:43.677413+0000","last_unstale":"2024-04-23T14:42:43.677846+0000","last_undegraded":"2024-04-23T14:42:43.677846+0000","last_fullsized":"2024-04-23T14:42:43.677846+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:35:55.039607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.440227+0000","last_change":"2024-04-23T14:41:39.028566+0000","last_active":"2024-04-23T14:42:43.440227+0000","last_peered":"2024-04-23T14:42:43.440227+0000","last_clean":"2024-04-23T14:42:43.440227+0000","last_became_active":"2024-04-23T14:41:39.028174+0000","last_became_peered":"2024-04-23T14:41:39.028174+0000","last_unstale":"2024-04-23T14:42:43.440227+0000","last_undegraded":"2024-04-23T14:42:43.440227+0000","last_fullsized":"2024-04-23T14:42:43.440227+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:50:55.424741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.530152+0000","last_change":"2024-04-23T14:42:43.530152+0000","last_active":"2024-04-23T14:42:43.530152+0000","last_peered":"2024-04-23T14:42:43.530152+0000","last_clean":"2024-04-23T14:42:43.530152+0000","last_became_active":"2024-04-23T14:42:43.529776+0000","last_became_peered":"2024-04-23T14:42:43.529776+0000","last_unstale":"2024-04-23T14:42:43.530152+0000","last_undegraded":"2024-04-23T14:42:43.530152+0000","last_fullsized":"2024-04-23T14:42:43.530152+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:53:29.597081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.591429+0000","last_change":"2024-04-23T14:41:39.028618+0000","last_active":"2024-04-23T14:42:43.591429+0000","last_peered":"2024-04-23T14:42:43.591429+0000","last_clean":"2024-04-23T14:42:43.591429+0000","last_became_active":"2024-04-23T14:41:39.028274+0000","last_became_peered":"2024-04-23T14:41:39.028274+0000","last_unstale":"2024-04-23T14:42:43.591429+0000","last_undegraded":"2024-04-23T14:42:43.591429+0000","last_fullsized":"2024-04-23T14:42:43.591429+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:14:31.571760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.528085+0000","last_change":"2024-04-23T14:41:40.020687+0000","last_active":"2024-04-23T14:42:43.528085+0000","last_peered":"2024-04-23T14:42:43.528085+0000","last_clean":"2024-04-23T14:42:43.528085+0000","last_became_active":"2024-04-23T14:41:40.020352+0000","last_became_peered":"2024-04-23T14:41:40.020352+0000","last_unstale":"2024-04-23T14:42:43.528085+0000","last_undegraded":"2024-04-23T14:42:43.528085+0000","last_fullsized":"2024-04-23T14:42:43.528085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:29:37.013486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.441360+0000","last_change":"2024-04-23T14:41:39.006002+0000","last_active":"2024-04-23T14:42:43.441360+0000","last_peered":"2024-04-23T14:42:43.441360+0000","last_clean":"2024-04-23T14:42:43.441360+0000","last_became_active":"2024-04-23T14:41:39.005415+0000","last_became_peered":"2024-04-23T14:41:39.005415+0000","last_unstale":"2024-04-23T14:42:43.441360+0000","last_undegraded":"2024-04-23T14:42:43.441360+0000","last_fullsized":"2024-04-23T14:42:43.441360+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:13:55.491931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.677560+0000","last_change":"2024-04-23T14:41:40.029393+0000","last_active":"2024-04-23T14:42:43.677560+0000","last_peered":"2024-04-23T14:42:43.677560+0000","last_clean":"2024-04-23T14:42:43.677560+0000","last_became_active":"2024-04-23T14:41:40.029039+0000","last_became_peered":"2024-04-23T14:41:40.029039+0000","last_unstale":"2024-04-23T14:42:43.677560+0000","last_undegraded":"2024-04-23T14:42:43.677560+0000","last_fullsized":"2024-04-23T14:42:43.677560+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:39:24.461260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.441220+0000","last_change":"2024-04-23T14:41:39.014638+0000","last_active":"2024-04-23T14:42:43.441220+0000","last_peered":"2024-04-23T14:42:43.441220+0000","last_clean":"2024-04-23T14:42:43.441220+0000","last_became_active":"2024-04-23T14:41:39.014289+0000","last_became_peered":"2024-04-23T14:41:39.014289+0000","last_unstale":"2024-04-23T14:42:43.441220+0000","last_undegraded":"2024-04-23T14:42:43.441220+0000","last_fullsized":"2024-04-23T14:42:43.441220+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:15:15.855399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442333+0000","last_change":"2024-04-23T14:41:40.015781+0000","last_active":"2024-04-23T14:42:43.442333+0000","last_peered":"2024-04-23T14:42:43.442333+0000","last_clean":"2024-04-23T14:42:43.442333+0000","last_became_active":"2024-04-23T14:41:40.015423+0000","last_became_peered":"2024-04-23T14:41:40.015423+0000","last_unstale":"2024-04-23T14:42:43.442333+0000","last_undegraded":"2024-04-23T14:42:43.442333+0000","last_fullsized":"2024-04-23T14:42:43.442333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:56:55.947585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.437580+0000","last_change":"2024-04-23T14:41:39.011021+0000","last_active":"2024-04-23T14:42:43.437580+0000","last_peered":"2024-04-23T14:42:43.437580+0000","last_clean":"2024-04-23T14:42:43.437580+0000","last_became_active":"2024-04-23T14:41:39.010637+0000","last_became_peered":"2024-04-23T14:41:39.010637+0000","last_unstale":"2024-04-23T14:42:43.437580+0000","last_undegraded":"2024-04-23T14:42:43.437580+0000","last_fullsized":"2024-04-23T14:42:43.437580+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:22:22.647170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442545+0000","last_change":"2024-04-23T14:41:40.042797+0000","last_active":"2024-04-23T14:42:43.442545+0000","last_peered":"2024-04-23T14:42:43.442545+0000","last_clean":"2024-04-23T14:42:43.442545+0000","last_became_active":"2024-04-23T14:41:40.037797+0000","last_became_peered":"2024-04-23T14:41:40.037797+0000","last_unstale":"2024-04-23T14:42:43.442545+0000","last_undegraded":"2024-04-23T14:42:43.442545+0000","last_fullsized":"2024-04-23T14:42:43.442545+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:37:32.604276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442286+0000","last_change":"2024-04-23T14:41:39.010585+0000","last_active":"2024-04-23T14:42:43.442286+0000","last_peered":"2024-04-23T14:42:43.442286+0000","last_clean":"2024-04-23T14:42:43.442286+0000","last_became_active":"2024-04-23T14:41:39.010294+0000","last_became_peered":"2024-04-23T14:41:39.010294+0000","last_unstale":"2024-04-23T14:42:43.442286+0000","last_undegraded":"2024-04-23T14:42:43.442286+0000","last_fullsized":"2024-04-23T14:42:43.442286+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:38:03.146904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442204+0000","last_change":"2024-04-23T14:41:40.013150+0000","last_active":"2024-04-23T14:42:43.442204+0000","last_peered":"2024-04-23T14:42:43.442204+0000","last_clean":"2024-04-23T14:42:43.442204+0000","last_became_active":"2024-04-23T14:41:40.012310+0000","last_became_peered":"2024-04-23T14:41:40.012310+0000","last_unstale":"2024-04-23T14:42:43.442204+0000","last_undegraded":"2024-04-23T14:42:43.442204+0000","last_fullsized":"2024-04-23T14:42:43.442204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:45:25.777200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'26","reported_seq":205,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:58.572166+0000","last_change":"2024-04-23T14:41:39.004256+0000","last_active":"2024-04-23T14:42:58.572166+0000","last_peered":"2024-04-23T14:42:58.572166+0000","last_clean":"2024-04-23T14:42:58.572166+0000","last_became_active":"2024-04-23T14:41:39.003874+0000","last_became_peered":"2024-04-23T14:41:39.003874+0000","last_unstale":"2024-04-23T14:42:58.572166+0000","last_undegraded":"2024-04-23T14:42:58.572166+0000","last_fullsized":"2024-04-23T14:42:58.572166+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:38:30.727149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61896,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":160,"num_read_kb":140,"num_write":26,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.438243+0000","last_change":"2024-04-23T14:41:40.012131+0000","last_active":"2024-04-23T14:42:43.438243+0000","last_peered":"2024-04-23T14:42:43.438243+0000","last_clean":"2024-04-23T14:42:43.438243+0000","last_became_active":"2024-04-23T14:41:40.011225+0000","last_became_peered":"2024-04-23T14:41:40.011225+0000","last_unstale":"2024-04-23T14:42:43.438243+0000","last_undegraded":"2024-04-23T14:42:43.438243+0000","last_fullsized":"2024-04-23T14:42:43.438243+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:36:12.085228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.441735+0000","last_change":"2024-04-23T14:41:39.001935+0000","last_active":"2024-04-23T14:42:43.441735+0000","last_peered":"2024-04-23T14:42:43.441735+0000","last_clean":"2024-04-23T14:42:43.441735+0000","last_became_active":"2024-04-23T14:41:39.001583+0000","last_became_peered":"2024-04-23T14:41:39.001583+0000","last_unstale":"2024-04-23T14:42:43.441735+0000","last_undegraded":"2024-04-23T14:42:43.441735+0000","last_fullsized":"2024-04-23T14:42:43.441735+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:47:40.986280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.592986+0000","last_change":"2024-04-23T14:41:40.036515+0000","last_active":"2024-04-23T14:42:43.592986+0000","last_peered":"2024-04-23T14:42:43.592986+0000","last_clean":"2024-04-23T14:42:43.592986+0000","last_became_active":"2024-04-23T14:41:40.036125+0000","last_became_peered":"2024-04-23T14:41:40.036125+0000","last_unstale":"2024-04-23T14:42:43.592986+0000","last_undegraded":"2024-04-23T14:42:43.592986+0000","last_fullsized":"2024-04-23T14:42:43.592986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:55:28.465378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.441733+0000","last_change":"2024-04-23T14:41:39.006127+0000","last_active":"2024-04-23T14:42:43.441733+0000","last_peered":"2024-04-23T14:42:43.441733+0000","last_clean":"2024-04-23T14:42:43.441733+0000","last_became_active":"2024-04-23T14:41:39.005761+0000","last_became_peered":"2024-04-23T14:41:39.005761+0000","last_unstale":"2024-04-23T14:42:43.441733+0000","last_undegraded":"2024-04-23T14:42:43.441733+0000","last_fullsized":"2024-04-23T14:42:43.441733+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:10:24.050016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.678799+0000","last_change":"2024-04-23T14:42:43.678799+0000","last_active":"2024-04-23T14:42:43.678799+0000","last_peered":"2024-04-23T14:42:43.678799+0000","last_clean":"2024-04-23T14:42:43.678799+0000","last_became_active":"2024-04-23T14:42:43.678318+0000","last_became_peered":"2024-04-23T14:42:43.678318+0000","last_unstale":"2024-04-23T14:42:43.678799+0000","last_undegraded":"2024-04-23T14:42:43.678799+0000","last_fullsized":"2024-04-23T14:42:43.678799+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:26:21.827247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.820290+0000","last_change":"2024-04-23T14:41:39.005873+0000","last_active":"2024-04-23T14:42:43.820290+0000","last_peered":"2024-04-23T14:42:43.820290+0000","last_clean":"2024-04-23T14:42:43.820290+0000","last_became_active":"2024-04-23T14:41:39.005481+0000","last_became_peered":"2024-04-23T14:41:39.005481+0000","last_unstale":"2024-04-23T14:42:43.820290+0000","last_undegraded":"2024-04-23T14:42:43.820290+0000","last_fullsized":"2024-04-23T14:42:43.820290+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:49:59.117451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.437246+0000","last_change":"2024-04-23T14:41:40.009346+0000","last_active":"2024-04-23T14:42:43.437246+0000","last_peered":"2024-04-23T14:42:43.437246+0000","last_clean":"2024-04-23T14:42:43.437246+0000","last_became_active":"2024-04-23T14:41:40.008927+0000","last_became_peered":"2024-04-23T14:41:40.008927+0000","last_unstale":"2024-04-23T14:42:43.437246+0000","last_undegraded":"2024-04-23T14:42:43.437246+0000","last_fullsized":"2024-04-23T14:42:43.437246+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:33:36.513349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.452178+0000","last_change":"2024-04-23T14:42:43.452178+0000","last_active":"2024-04-23T14:42:43.452178+0000","last_peered":"2024-04-23T14:42:43.452178+0000","last_clean":"2024-04-23T14:42:43.452178+0000","last_became_active":"2024-04-23T14:42:43.444786+0000","last_became_peered":"2024-04-23T14:42:43.444786+0000","last_unstale":"2024-04-23T14:42:43.452178+0000","last_undegraded":"2024-04-23T14:42:43.452178+0000","last_fullsized":"2024-04-23T14:42:43.452178+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:00:21.945663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,1],"acting":[8,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.441367+0000","last_change":"2024-04-23T14:41:40.016845+0000","last_active":"2024-04-23T14:42:43.441367+0000","last_peered":"2024-04-23T14:42:43.441367+0000","last_clean":"2024-04-23T14:42:43.441367+0000","last_became_active":"2024-04-23T14:41:40.016543+0000","last_became_peered":"2024-04-23T14:41:40.016543+0000","last_unstale":"2024-04-23T14:42:43.441367+0000","last_undegraded":"2024-04-23T14:42:43.441367+0000","last_fullsized":"2024-04-23T14:42:43.441367+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:52:41.054454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.438163+0000","last_change":"2024-04-23T14:41:38.999968+0000","last_active":"2024-04-23T14:42:43.438163+0000","last_peered":"2024-04-23T14:42:43.438163+0000","last_clean":"2024-04-23T14:42:43.438163+0000","last_became_active":"2024-04-23T14:41:38.999658+0000","last_became_peered":"2024-04-23T14:41:38.999658+0000","last_unstale":"2024-04-23T14:42:43.438163+0000","last_undegraded":"2024-04-23T14:42:43.438163+0000","last_fullsized":"2024-04-23T14:42:43.438163+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:14:20.990331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.439374+0000","last_change":"2024-04-23T14:41:40.011753+0000","last_active":"2024-04-23T14:42:43.439374+0000","last_peered":"2024-04-23T14:42:43.439374+0000","last_clean":"2024-04-23T14:42:43.439374+0000","last_became_active":"2024-04-23T14:41:40.011264+0000","last_became_peered":"2024-04-23T14:41:40.011264+0000","last_unstale":"2024-04-23T14:42:43.439374+0000","last_undegraded":"2024-04-23T14:42:43.439374+0000","last_fullsized":"2024-04-23T14:42:43.439374+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:52:22.812046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.677877+0000","last_change":"2024-04-23T14:41:39.027384+0000","last_active":"2024-04-23T14:42:43.677877+0000","last_peered":"2024-04-23T14:42:43.677877+0000","last_clean":"2024-04-23T14:42:43.677877+0000","last_became_active":"2024-04-23T14:41:39.027018+0000","last_became_peered":"2024-04-23T14:41:39.027018+0000","last_unstale":"2024-04-23T14:42:43.677877+0000","last_undegraded":"2024-04-23T14:42:43.677877+0000","last_fullsized":"2024-04-23T14:42:43.677877+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:13:54.685704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.441143+0000","last_change":"2024-04-23T14:41:40.041216+0000","last_active":"2024-04-23T14:42:43.441143+0000","last_peered":"2024-04-23T14:42:43.441143+0000","last_clean":"2024-04-23T14:42:43.441143+0000","last_became_active":"2024-04-23T14:41:40.040789+0000","last_became_peered":"2024-04-23T14:41:40.040789+0000","last_unstale":"2024-04-23T14:42:43.441143+0000","last_undegraded":"2024-04-23T14:42:43.441143+0000","last_fullsized":"2024-04-23T14:42:43.441143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:01:13.156704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.437436+0000","last_change":"2024-04-23T14:41:39.024399+0000","last_active":"2024-04-23T14:42:43.437436+0000","last_peered":"2024-04-23T14:42:43.437436+0000","last_clean":"2024-04-23T14:42:43.437436+0000","last_became_active":"2024-04-23T14:41:39.024141+0000","last_became_peered":"2024-04-23T14:41:39.024141+0000","last_unstale":"2024-04-23T14:42:43.437436+0000","last_undegraded":"2024-04-23T14:42:43.437436+0000","last_fullsized":"2024-04-23T14:42:43.437436+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:34:27.516614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.680818+0000","last_change":"2024-04-23T14:42:43.680818+0000","last_active":"2024-04-23T14:42:43.680818+0000","last_peered":"2024-04-23T14:42:43.680818+0000","last_clean":"2024-04-23T14:42:43.680818+0000","last_became_active":"2024-04-23T14:42:43.680463+0000","last_became_peered":"2024-04-23T14:42:43.680463+0000","last_unstale":"2024-04-23T14:42:43.680818+0000","last_undegraded":"2024-04-23T14:42:43.680818+0000","last_fullsized":"2024-04-23T14:42:43.680818+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:34:36.964359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442386+0000","last_change":"2024-04-23T14:41:40.014689+0000","last_active":"2024-04-23T14:42:43.442386+0000","last_peered":"2024-04-23T14:42:43.442386+0000","last_clean":"2024-04-23T14:42:43.442386+0000","last_became_active":"2024-04-23T14:41:40.014371+0000","last_became_peered":"2024-04-23T14:41:40.014371+0000","last_unstale":"2024-04-23T14:42:43.442386+0000","last_undegraded":"2024-04-23T14:42:43.442386+0000","last_fullsized":"2024-04-23T14:42:43.442386+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:56:41.284144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.591269+0000","last_change":"2024-04-23T14:41:39.031327+0000","last_active":"2024-04-23T14:42:43.591269+0000","last_peered":"2024-04-23T14:42:43.591269+0000","last_clean":"2024-04-23T14:42:43.591269+0000","last_became_active":"2024-04-23T14:41:39.030864+0000","last_became_peered":"2024-04-23T14:41:39.030864+0000","last_unstale":"2024-04-23T14:42:43.591269+0000","last_undegraded":"2024-04-23T14:42:43.591269+0000","last_fullsized":"2024-04-23T14:42:43.591269+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:58:24.048207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.529243+0000","last_change":"2024-04-23T14:41:40.035104+0000","last_active":"2024-04-23T14:42:43.529243+0000","last_peered":"2024-04-23T14:42:43.529243+0000","last_clean":"2024-04-23T14:42:43.529243+0000","last_became_active":"2024-04-23T14:41:40.034678+0000","last_became_peered":"2024-04-23T14:41:40.034678+0000","last_unstale":"2024-04-23T14:42:43.529243+0000","last_undegraded":"2024-04-23T14:42:43.529243+0000","last_fullsized":"2024-04-23T14:42:43.529243+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:29:53.040424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.821031+0000","last_change":"2024-04-23T14:41:39.016684+0000","last_active":"2024-04-23T14:42:43.821031+0000","last_peered":"2024-04-23T14:42:43.821031+0000","last_clean":"2024-04-23T14:42:43.821031+0000","last_became_active":"2024-04-23T14:41:39.016375+0000","last_became_peered":"2024-04-23T14:41:39.016375+0000","last_unstale":"2024-04-23T14:42:43.821031+0000","last_undegraded":"2024-04-23T14:42:43.821031+0000","last_fullsized":"2024-04-23T14:42:43.821031+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:09.449590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":355,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.437363+0000","last_change":"2024-04-23T14:40:22.305413+0000","last_active":"2024-04-23T14:42:43.437363+0000","last_peered":"2024-04-23T14:42:43.437363+0000","last_clean":"2024-04-23T14:42:43.437363+0000","last_became_active":"2024-04-23T14:40:22.304977+0000","last_became_peered":"2024-04-23T14:40:22.304977+0000","last_unstale":"2024-04-23T14:42:43.437363+0000","last_undegraded":"2024-04-23T14:42:43.437363+0000","last_fullsized":"2024-04-23T14:42:43.437363+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:36:28.336753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:36:28.336753+0000","last_clean_scrub_stamp":"2024-04-23T14:36:28.336753+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:42:27.611770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.591222+0000","last_change":"2024-04-23T14:41:40.023300+0000","last_active":"2024-04-23T14:42:43.591222+0000","last_peered":"2024-04-23T14:42:43.591222+0000","last_clean":"2024-04-23T14:42:43.591222+0000","last_became_active":"2024-04-23T14:41:40.022888+0000","last_became_peered":"2024-04-23T14:41:40.022888+0000","last_unstale":"2024-04-23T14:42:43.591222+0000","last_undegraded":"2024-04-23T14:42:43.591222+0000","last_fullsized":"2024-04-23T14:42:43.591222+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:31:48.705005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.437426+0000","last_change":"2024-04-23T14:41:39.005482+0000","last_active":"2024-04-23T14:42:43.437426+0000","last_peered":"2024-04-23T14:42:43.437426+0000","last_clean":"2024-04-23T14:42:43.437426+0000","last_became_active":"2024-04-23T14:41:39.004552+0000","last_became_peered":"2024-04-23T14:41:39.004552+0000","last_unstale":"2024-04-23T14:42:43.437426+0000","last_undegraded":"2024-04-23T14:42:43.437426+0000","last_fullsized":"2024-04-23T14:42:43.437426+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:05:51.411210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.438626+0000","last_change":"2024-04-23T14:41:39.015605+0000","last_active":"2024-04-23T14:42:43.438626+0000","last_peered":"2024-04-23T14:42:43.438626+0000","last_clean":"2024-04-23T14:42:43.438626+0000","last_became_active":"2024-04-23T14:41:39.015260+0000","last_became_peered":"2024-04-23T14:41:39.015260+0000","last_unstale":"2024-04-23T14:42:43.438626+0000","last_undegraded":"2024-04-23T14:42:43.438626+0000","last_fullsized":"2024-04-23T14:42:43.438626+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:23:21.251483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.593376+0000","last_change":"2024-04-23T14:41:40.020619+0000","last_active":"2024-04-23T14:42:43.593376+0000","last_peered":"2024-04-23T14:42:43.593376+0000","last_clean":"2024-04-23T14:42:43.593376+0000","last_became_active":"2024-04-23T14:41:40.020191+0000","last_became_peered":"2024-04-23T14:41:40.020191+0000","last_unstale":"2024-04-23T14:42:43.593376+0000","last_undegraded":"2024-04-23T14:42:43.593376+0000","last_fullsized":"2024-04-23T14:42:43.593376+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:54:34.036464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.677562+0000","last_change":"2024-04-23T14:41:39.019507+0000","last_active":"2024-04-23T14:42:43.677562+0000","last_peered":"2024-04-23T14:42:43.677562+0000","last_clean":"2024-04-23T14:42:43.677562+0000","last_became_active":"2024-04-23T14:41:39.019116+0000","last_became_peered":"2024-04-23T14:41:39.019116+0000","last_unstale":"2024-04-23T14:42:43.677562+0000","last_undegraded":"2024-04-23T14:42:43.677562+0000","last_fullsized":"2024-04-23T14:42:43.677562+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:06:05.777908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.531207+0000","last_change":"2024-04-23T14:41:40.032765+0000","last_active":"2024-04-23T14:42:43.531207+0000","last_peered":"2024-04-23T14:42:43.531207+0000","last_clean":"2024-04-23T14:42:43.531207+0000","last_became_active":"2024-04-23T14:41:40.027896+0000","last_became_peered":"2024-04-23T14:41:40.027896+0000","last_unstale":"2024-04-23T14:42:43.531207+0000","last_undegraded":"2024-04-23T14:42:43.531207+0000","last_fullsized":"2024-04-23T14:42:43.531207+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:36:16.225444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.820509+0000","last_change":"2024-04-23T14:41:39.021681+0000","last_active":"2024-04-23T14:42:43.820509+0000","last_peered":"2024-04-23T14:42:43.820509+0000","last_clean":"2024-04-23T14:42:43.820509+0000","last_became_active":"2024-04-23T14:41:39.021423+0000","last_became_peered":"2024-04-23T14:41:39.021423+0000","last_unstale":"2024-04-23T14:42:43.820509+0000","last_undegraded":"2024-04-23T14:42:43.820509+0000","last_fullsized":"2024-04-23T14:42:43.820509+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:42:55.351296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.443567+0000","last_change":"2024-04-23T14:41:40.037250+0000","last_active":"2024-04-23T14:42:43.443567+0000","last_peered":"2024-04-23T14:42:43.443567+0000","last_clean":"2024-04-23T14:42:43.443567+0000","last_became_active":"2024-04-23T14:41:40.036645+0000","last_became_peered":"2024-04-23T14:41:40.036645+0000","last_unstale":"2024-04-23T14:42:43.443567+0000","last_undegraded":"2024-04-23T14:42:43.443567+0000","last_fullsized":"2024-04-23T14:42:43.443567+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:50:35.556841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442618+0000","last_change":"2024-04-23T14:41:39.007764+0000","last_active":"2024-04-23T14:42:43.442618+0000","last_peered":"2024-04-23T14:42:43.442618+0000","last_clean":"2024-04-23T14:42:43.442618+0000","last_became_active":"2024-04-23T14:41:39.007312+0000","last_became_peered":"2024-04-23T14:41:39.007312+0000","last_unstale":"2024-04-23T14:42:43.442618+0000","last_undegraded":"2024-04-23T14:42:43.442618+0000","last_fullsized":"2024-04-23T14:42:43.442618+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:38:56.021238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.437731+0000","last_change":"2024-04-23T14:41:40.004343+0000","last_active":"2024-04-23T14:42:43.437731+0000","last_peered":"2024-04-23T14:42:43.437731+0000","last_clean":"2024-04-23T14:42:43.437731+0000","last_became_active":"2024-04-23T14:41:40.004015+0000","last_became_peered":"2024-04-23T14:41:40.004015+0000","last_unstale":"2024-04-23T14:42:43.437731+0000","last_undegraded":"2024-04-23T14:42:43.437731+0000","last_fullsized":"2024-04-23T14:42:43.437731+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:05:33.863692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.528699+0000","last_change":"2024-04-23T14:41:39.016271+0000","last_active":"2024-04-23T14:42:43.528699+0000","last_peered":"2024-04-23T14:42:43.528699+0000","last_clean":"2024-04-23T14:42:43.528699+0000","last_became_active":"2024-04-23T14:41:39.015851+0000","last_became_peered":"2024-04-23T14:41:39.015851+0000","last_unstale":"2024-04-23T14:42:43.528699+0000","last_undegraded":"2024-04-23T14:42:43.528699+0000","last_fullsized":"2024-04-23T14:42:43.528699+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:50:52.943534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.439085+0000","last_change":"2024-04-23T14:41:40.029815+0000","last_active":"2024-04-23T14:42:43.439085+0000","last_peered":"2024-04-23T14:42:43.439085+0000","last_clean":"2024-04-23T14:42:43.439085+0000","last_became_active":"2024-04-23T14:41:40.029496+0000","last_became_peered":"2024-04-23T14:41:40.029496+0000","last_unstale":"2024-04-23T14:42:43.439085+0000","last_undegraded":"2024-04-23T14:42:43.439085+0000","last_fullsized":"2024-04-23T14:42:43.439085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:36:09.228993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442420+0000","last_change":"2024-04-23T14:41:39.001946+0000","last_active":"2024-04-23T14:42:43.442420+0000","last_peered":"2024-04-23T14:42:43.442420+0000","last_clean":"2024-04-23T14:42:43.442420+0000","last_became_active":"2024-04-23T14:41:39.001619+0000","last_became_peered":"2024-04-23T14:41:39.001619+0000","last_unstale":"2024-04-23T14:42:43.442420+0000","last_undegraded":"2024-04-23T14:42:43.442420+0000","last_fullsized":"2024-04-23T14:42:43.442420+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:43:57.680195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.528966+0000","last_change":"2024-04-23T14:41:40.045728+0000","last_active":"2024-04-23T14:42:43.528966+0000","last_peered":"2024-04-23T14:42:43.528966+0000","last_clean":"2024-04-23T14:42:43.528966+0000","last_became_active":"2024-04-23T14:41:40.045293+0000","last_became_peered":"2024-04-23T14:41:40.045293+0000","last_unstale":"2024-04-23T14:42:43.528966+0000","last_undegraded":"2024-04-23T14:42:43.528966+0000","last_fullsized":"2024-04-23T14:42:43.528966+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:35:54.340644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.823943+0000","last_change":"2024-04-23T14:42:43.823943+0000","last_active":"2024-04-23T14:42:43.823943+0000","last_peered":"2024-04-23T14:42:43.823943+0000","last_clean":"2024-04-23T14:42:43.823943+0000","last_became_active":"2024-04-23T14:42:43.821663+0000","last_became_peered":"2024-04-23T14:42:43.821663+0000","last_unstale":"2024-04-23T14:42:43.823943+0000","last_undegraded":"2024-04-23T14:42:43.823943+0000","last_fullsized":"2024-04-23T14:42:43.823943+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:18:54.895943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.437538+0000","last_change":"2024-04-23T14:41:40.028897+0000","last_active":"2024-04-23T14:42:43.437538+0000","last_peered":"2024-04-23T14:42:43.437538+0000","last_clean":"2024-04-23T14:42:43.437538+0000","last_became_active":"2024-04-23T14:41:40.028557+0000","last_became_peered":"2024-04-23T14:41:40.028557+0000","last_unstale":"2024-04-23T14:42:43.437538+0000","last_undegraded":"2024-04-23T14:42:43.437538+0000","last_fullsized":"2024-04-23T14:42:43.437538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:26:57.871013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442160+0000","last_change":"2024-04-23T14:41:39.012861+0000","last_active":"2024-04-23T14:42:43.442160+0000","last_peered":"2024-04-23T14:42:43.442160+0000","last_clean":"2024-04-23T14:42:43.442160+0000","last_became_active":"2024-04-23T14:41:39.012503+0000","last_became_peered":"2024-04-23T14:41:39.012503+0000","last_unstale":"2024-04-23T14:42:43.442160+0000","last_undegraded":"2024-04-23T14:42:43.442160+0000","last_fullsized":"2024-04-23T14:42:43.442160+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:41:57.347221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.437801+0000","last_change":"2024-04-23T14:41:40.021580+0000","last_active":"2024-04-23T14:42:43.437801+0000","last_peered":"2024-04-23T14:42:43.437801+0000","last_clean":"2024-04-23T14:42:43.437801+0000","last_became_active":"2024-04-23T14:41:40.020706+0000","last_became_peered":"2024-04-23T14:41:40.020706+0000","last_unstale":"2024-04-23T14:42:43.437801+0000","last_undegraded":"2024-04-23T14:42:43.437801+0000","last_fullsized":"2024-04-23T14:42:43.437801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:59:42.576276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.443072+0000","last_change":"2024-04-23T14:41:39.006425+0000","last_active":"2024-04-23T14:42:43.443072+0000","last_peered":"2024-04-23T14:42:43.443072+0000","last_clean":"2024-04-23T14:42:43.443072+0000","last_became_active":"2024-04-23T14:41:39.006123+0000","last_became_peered":"2024-04-23T14:41:39.006123+0000","last_unstale":"2024-04-23T14:42:43.443072+0000","last_undegraded":"2024-04-23T14:42:43.443072+0000","last_fullsized":"2024-04-23T14:42:43.443072+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:08:37.261092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.437457+0000","last_change":"2024-04-23T14:41:40.034063+0000","last_active":"2024-04-23T14:42:43.437457+0000","last_peered":"2024-04-23T14:42:43.437457+0000","last_clean":"2024-04-23T14:42:43.437457+0000","last_became_active":"2024-04-23T14:41:40.033669+0000","last_became_peered":"2024-04-23T14:41:40.033669+0000","last_unstale":"2024-04-23T14:42:43.437457+0000","last_undegraded":"2024-04-23T14:42:43.437457+0000","last_fullsized":"2024-04-23T14:42:43.437457+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:23:50.608041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.593343+0000","last_change":"2024-04-23T14:41:39.025975+0000","last_active":"2024-04-23T14:42:43.593343+0000","last_peered":"2024-04-23T14:42:43.593343+0000","last_clean":"2024-04-23T14:42:43.593343+0000","last_became_active":"2024-04-23T14:41:39.025629+0000","last_became_peered":"2024-04-23T14:41:39.025629+0000","last_unstale":"2024-04-23T14:42:43.593343+0000","last_undegraded":"2024-04-23T14:42:43.593343+0000","last_fullsized":"2024-04-23T14:42:43.593343+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:00:03.753547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.437798+0000","last_change":"2024-04-23T14:41:40.013030+0000","last_active":"2024-04-23T14:42:43.437798+0000","last_peered":"2024-04-23T14:42:43.437798+0000","last_clean":"2024-04-23T14:42:43.437798+0000","last_became_active":"2024-04-23T14:41:40.012708+0000","last_became_peered":"2024-04-23T14:41:40.012708+0000","last_unstale":"2024-04-23T14:42:43.437798+0000","last_undegraded":"2024-04-23T14:42:43.437798+0000","last_fullsized":"2024-04-23T14:42:43.437798+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:05:29.217273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.439174+0000","last_change":"2024-04-23T14:41:40.010551+0000","last_active":"2024-04-23T14:42:43.439174+0000","last_peered":"2024-04-23T14:42:43.439174+0000","last_clean":"2024-04-23T14:42:43.439174+0000","last_became_active":"2024-04-23T14:41:40.010244+0000","last_became_peered":"2024-04-23T14:41:40.010244+0000","last_unstale":"2024-04-23T14:42:43.439174+0000","last_undegraded":"2024-04-23T14:42:43.439174+0000","last_fullsized":"2024-04-23T14:42:43.439174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:03:57.223815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.441829+0000","last_change":"2024-04-23T14:41:40.009132+0000","last_active":"2024-04-23T14:42:43.441829+0000","last_peered":"2024-04-23T14:42:43.441829+0000","last_clean":"2024-04-23T14:42:43.441829+0000","last_became_active":"2024-04-23T14:41:40.008822+0000","last_became_peered":"2024-04-23T14:41:40.008822+0000","last_unstale":"2024-04-23T14:42:43.441829+0000","last_undegraded":"2024-04-23T14:42:43.441829+0000","last_fullsized":"2024-04-23T14:42:43.441829+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:10:28.513846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442333+0000","last_change":"2024-04-23T14:41:40.041193+0000","last_active":"2024-04-23T14:42:43.442333+0000","last_peered":"2024-04-23T14:42:43.442333+0000","last_clean":"2024-04-23T14:42:43.442333+0000","last_became_active":"2024-04-23T14:41:40.040851+0000","last_became_peered":"2024-04-23T14:41:40.040851+0000","last_unstale":"2024-04-23T14:42:43.442333+0000","last_undegraded":"2024-04-23T14:42:43.442333+0000","last_fullsized":"2024-04-23T14:42:43.442333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:13:30.090015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442731+0000","last_change":"2024-04-23T14:41:40.042703+0000","last_active":"2024-04-23T14:42:43.442731+0000","last_peered":"2024-04-23T14:42:43.442731+0000","last_clean":"2024-04-23T14:42:43.442731+0000","last_became_active":"2024-04-23T14:41:40.037666+0000","last_became_peered":"2024-04-23T14:41:40.037666+0000","last_unstale":"2024-04-23T14:42:43.442731+0000","last_undegraded":"2024-04-23T14:42:43.442731+0000","last_fullsized":"2024-04-23T14:42:43.442731+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:04:04.240689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.593484+0000","last_change":"2024-04-23T14:41:40.033487+0000","last_active":"2024-04-23T14:42:43.593484+0000","last_peered":"2024-04-23T14:42:43.593484+0000","last_clean":"2024-04-23T14:42:43.593484+0000","last_became_active":"2024-04-23T14:41:40.031799+0000","last_became_peered":"2024-04-23T14:41:40.031799+0000","last_unstale":"2024-04-23T14:42:43.593484+0000","last_undegraded":"2024-04-23T14:42:43.593484+0000","last_fullsized":"2024-04-23T14:42:43.593484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:36:03.930264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.439616+0000","last_change":"2024-04-23T14:41:40.012833+0000","last_active":"2024-04-23T14:42:43.439616+0000","last_peered":"2024-04-23T14:42:43.439616+0000","last_clean":"2024-04-23T14:42:43.439616+0000","last_became_active":"2024-04-23T14:41:40.012251+0000","last_became_peered":"2024-04-23T14:41:40.012251+0000","last_unstale":"2024-04-23T14:42:43.439616+0000","last_undegraded":"2024-04-23T14:42:43.439616+0000","last_fullsized":"2024-04-23T14:42:43.439616+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:17:09.415998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442491+0000","last_change":"2024-04-23T14:41:40.013336+0000","last_active":"2024-04-23T14:42:43.442491+0000","last_peered":"2024-04-23T14:42:43.442491+0000","last_clean":"2024-04-23T14:42:43.442491+0000","last_became_active":"2024-04-23T14:41:40.013043+0000","last_became_peered":"2024-04-23T14:41:40.013043+0000","last_unstale":"2024-04-23T14:42:43.442491+0000","last_undegraded":"2024-04-23T14:42:43.442491+0000","last_fullsized":"2024-04-23T14:42:43.442491+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:27:59.118986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.528540+0000","last_change":"2024-04-23T14:41:40.028236+0000","last_active":"2024-04-23T14:42:43.528540+0000","last_peered":"2024-04-23T14:42:43.528540+0000","last_clean":"2024-04-23T14:42:43.528540+0000","last_became_active":"2024-04-23T14:41:40.027504+0000","last_became_peered":"2024-04-23T14:41:40.027504+0000","last_unstale":"2024-04-23T14:42:43.528540+0000","last_undegraded":"2024-04-23T14:42:43.528540+0000","last_fullsized":"2024-04-23T14:42:43.528540+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:04:31.540482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442292+0000","last_change":"2024-04-23T14:41:40.029252+0000","last_active":"2024-04-23T14:42:43.442292+0000","last_peered":"2024-04-23T14:42:43.442292+0000","last_clean":"2024-04-23T14:42:43.442292+0000","last_became_active":"2024-04-23T14:41:40.028937+0000","last_became_peered":"2024-04-23T14:41:40.028937+0000","last_unstale":"2024-04-23T14:42:43.442292+0000","last_undegraded":"2024-04-23T14:42:43.442292+0000","last_fullsized":"2024-04-23T14:42:43.442292+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:46:44.441319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.528988+0000","last_change":"2024-04-23T14:41:40.027705+0000","last_active":"2024-04-23T14:42:43.528988+0000","last_peered":"2024-04-23T14:42:43.528988+0000","last_clean":"2024-04-23T14:42:43.528988+0000","last_became_active":"2024-04-23T14:41:40.027260+0000","last_became_peered":"2024-04-23T14:41:40.027260+0000","last_unstale":"2024-04-23T14:42:43.528988+0000","last_undegraded":"2024-04-23T14:42:43.528988+0000","last_fullsized":"2024-04-23T14:42:43.528988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:33:42.461219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442119+0000","last_change":"2024-04-23T14:41:40.026490+0000","last_active":"2024-04-23T14:42:43.442119+0000","last_peered":"2024-04-23T14:42:43.442119+0000","last_clean":"2024-04-23T14:42:43.442119+0000","last_became_active":"2024-04-23T14:41:40.026117+0000","last_became_peered":"2024-04-23T14:41:40.026117+0000","last_unstale":"2024-04-23T14:42:43.442119+0000","last_undegraded":"2024-04-23T14:42:43.442119+0000","last_fullsized":"2024-04-23T14:42:43.442119+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:06:17.626663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.593324+0000","last_change":"2024-04-23T14:41:40.038489+0000","last_active":"2024-04-23T14:42:43.593324+0000","last_peered":"2024-04-23T14:42:43.593324+0000","last_clean":"2024-04-23T14:42:43.593324+0000","last_became_active":"2024-04-23T14:41:40.038078+0000","last_became_peered":"2024-04-23T14:41:40.038078+0000","last_unstale":"2024-04-23T14:42:43.593324+0000","last_undegraded":"2024-04-23T14:42:43.593324+0000","last_fullsized":"2024-04-23T14:42:43.593324+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:58:59.440468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.439967+0000","last_change":"2024-04-23T14:41:40.009463+0000","last_active":"2024-04-23T14:42:43.439967+0000","last_peered":"2024-04-23T14:42:43.439967+0000","last_clean":"2024-04-23T14:42:43.439967+0000","last_became_active":"2024-04-23T14:41:40.009154+0000","last_became_peered":"2024-04-23T14:41:40.009154+0000","last_unstale":"2024-04-23T14:42:43.439967+0000","last_undegraded":"2024-04-23T14:42:43.439967+0000","last_fullsized":"2024-04-23T14:42:43.439967+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:13:47.641863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":62694,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":166,"num_read_kb":146,"num_write":79,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":188082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383710,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":946176,"data_stored":455710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645347,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28608,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716544,"statfs":{"total":95995035648,"available":95965741056,"internally_reserved":0,"allocated":1626112,"data_stored":1108285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808873,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36272,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708880,"statfs":{"total":95995035648,"available":95957893120,"internally_reserved":0,"allocated":950272,"data_stored":456292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070511,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":456314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332150,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32696,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712456,"statfs":{"total":95995035648,"available":95961554944,"internally_reserved":0,"allocated":1548288,"data_stored":1046777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495674,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":950272,"data_stored":455827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574935,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1015808,"data_stored":517606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738461,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36272,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708880,"statfs":{"total":95995035648,"available":95957893120,"internally_reserved":0,"allocated":950272,"data_stored":455800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411411,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36340,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708812,"statfs":{"total":95995035648,"available":95957823488,"internally_reserved":0,"allocated":1019904,"data_stored":517640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182475,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":962560,"data_stored":456048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920839,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":950272,"data_stored":455732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757312,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28540,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716612,"statfs":{"total":95995035648,"available":95965810688,"internally_reserved":0,"allocated":1556480,"data_stored":1046377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":61930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":62090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T14:43:05.887 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T14:43:06.427 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:43:06.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:06 smithi028 ceph-mon[27810]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T14:43:06.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:06 smithi105 ceph-mon[32555]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T14:43:06.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:06 smithi038 ceph-mon[32348]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T14:43:07.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:07 smithi105 ceph-mon[32555]: pgmap v359: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:43:07.849 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:07 smithi028 ceph-mon[27810]: pgmap v359: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:43:07.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:07 smithi038 ceph-mon[32348]: pgmap v359: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:43:08.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.684+0000 7febb5400700 1 -- 172.21.15.28:0/814113841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febb006a800 msgr2=0x7febb006ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:08.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.685+0000 7febb5400700 1 --2- 172.21.15.28:0/814113841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febb006a800 0x7febb006ac60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7feb98009990 tx=0x7feb98009ca0 comp rx=0 tx=0).stop 2024-04-23T14:43:08.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.685+0000 7febb5400700 1 -- 172.21.15.28:0/814113841 shutdown_connections 2024-04-23T14:43:08.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.685+0000 7febb5400700 1 --2- 172.21.15.28:0/814113841 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7febb0108b70 0x7febb0108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:08.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.685+0000 7febb5400700 1 --2- 172.21.15.28:0/814113841 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febb006b1a0 0x7febb010f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:08.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.685+0000 7febb5400700 1 --2- 172.21.15.28:0/814113841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febb006a800 0x7febb006ac60 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:08.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.685+0000 7febb5400700 1 -- 172.21.15.28:0/814113841 >> 172.21.15.28:0/814113841 conn(0x7febb00789a0 msgr2=0x7febb0078db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:08.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.685+0000 7febb5400700 1 -- 172.21.15.28:0/814113841 shutdown_connections 2024-04-23T14:43:08.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.685+0000 7febb5400700 1 -- 172.21.15.28:0/814113841 wait complete. 2024-04-23T14:43:08.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.686+0000 7febb5400700 1 Processor -- start 2024-04-23T14:43:08.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.686+0000 7febb5400700 1 -- start start 2024-04-23T14:43:08.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.687+0000 7febb5400700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febb006a800 0x7febb019b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:08.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.687+0000 7febb5400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febb006b1a0 0x7febb019ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:08.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.687+0000 7febb5400700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7febb0108b70 0x7febb019fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:08.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.687+0000 7febb5400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7febb0112e20 con 0x7febb006b1a0 2024-04-23T14:43:08.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.687+0000 7febb5400700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7febb0112ca0 con 0x7febb006a800 2024-04-23T14:43:08.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.687+0000 7febb5400700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7febb0112fa0 con 0x7febb0108b70 2024-04-23T14:43:08.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.687+0000 7feba7e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febb006b1a0 0x7febb019ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:08.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.687+0000 7febae800700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febb006a800 0x7febb019b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:08.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.687+0000 7febae800700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febb006a800 0x7febb019b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:42388/0 (socket says 172.21.15.28:42388) 2024-04-23T14:43:08.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.687+0000 7febae800700 1 -- 172.21.15.28:0/1285503308 learned_addr learned my addr 172.21.15.28:0/1285503308 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:08.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.687+0000 7febaf200700 1 --2- 172.21.15.28:0/1285503308 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7febb0108b70 0x7febb019fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:08.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.687+0000 7febae800700 1 -- 172.21.15.28:0/1285503308 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7febb0108b70 msgr2=0x7febb019fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:08.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.687+0000 7febae800700 1 --2- 172.21.15.28:0/1285503308 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7febb0108b70 0x7febb019fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:08.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.687+0000 7febae800700 1 -- 172.21.15.28:0/1285503308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febb006b1a0 msgr2=0x7febb019ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:08.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.687+0000 7febae800700 1 --2- 172.21.15.28:0/1285503308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febb006b1a0 0x7febb019ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:08.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.687+0000 7febae800700 1 -- 172.21.15.28:0/1285503308 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febb01a04f0 con 0x7febb006a800 2024-04-23T14:43:08.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.687+0000 7feba7e00700 1 --2- 172.21.15.28:0/1285503308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febb006b1a0 0x7febb019ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:43:08.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.688+0000 7febae800700 1 --2- 172.21.15.28:0/1285503308 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febb006a800 0x7febb019b4e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7feba000e9c0 tx=0x7feba000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:08.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.688+0000 7feba7400700 1 -- 172.21.15.28:0/1285503308 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba000cbd0 con 0x7febb006a800 2024-04-23T14:43:08.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.688+0000 7feba7400700 1 -- 172.21.15.28:0/1285503308 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feba000ee60 con 0x7febb006a800 2024-04-23T14:43:08.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.688+0000 7febb5400700 1 -- 172.21.15.28:0/1285503308 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febb01a07e0 con 0x7febb006a800 2024-04-23T14:43:08.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.688+0000 7feba7400700 1 -- 172.21.15.28:0/1285503308 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba0005630 con 0x7febb006a800 2024-04-23T14:43:08.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.688+0000 7febb5400700 1 -- 172.21.15.28:0/1285503308 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febb01a8060 con 0x7febb006a800 2024-04-23T14:43:08.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.690+0000 7febb5400700 1 -- 172.21.15.28:0/1285503308 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febb00697e0 con 0x7febb006a800 2024-04-23T14:43:08.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.691+0000 7feba7400700 1 -- 172.21.15.28:0/1285503308 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7feba0004750 con 0x7febb006a800 2024-04-23T14:43:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.693+0000 7feba7400700 1 --2- 172.21.15.28:0/1285503308 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7feb940a3cd0 0x7feb940a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.693+0000 7feba7400700 1 -- 172.21.15.28:0/1285503308 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7feba00d07c0 con 0x7febb006a800 2024-04-23T14:43:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.693+0000 7feba7e00700 1 --2- 172.21.15.28:0/1285503308 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7feb940a3cd0 0x7feb940a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:08.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.694+0000 7feba7e00700 1 --2- 172.21.15.28:0/1285503308 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7feb940a3cd0 0x7feb940a6190 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7febb0078fa0 tx=0x7feb9800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:08.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.697+0000 7feba7400700 1 -- 172.21.15.28:0/1285503308 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feba009d000 con 0x7febb006a800 2024-04-23T14:43:08.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.930+0000 7febb5400700 1 -- 172.21.15.28:0/1285503308 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7febb0062f30 con 0x7feb940a3cd0 2024-04-23T14:43:08.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.939+0000 7feba7400700 1 -- 172.21.15.28:0/1285503308 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258631 (secure 0 0 0) 0x7febb0062f30 con 0x7feb940a3cd0 2024-04-23T14:43:08.942 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:43:08.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.947+0000 7febb5400700 1 -- 172.21.15.28:0/1285503308 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7feb940a3cd0 msgr2=0x7feb940a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:08.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.947+0000 7febb5400700 1 --2- 172.21.15.28:0/1285503308 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7feb940a3cd0 0x7feb940a6190 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7febb0078fa0 tx=0x7feb9800b260 comp rx=0 tx=0).stop 2024-04-23T14:43:08.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.947+0000 7febb5400700 1 -- 172.21.15.28:0/1285503308 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febb006a800 msgr2=0x7febb019b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:08.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.947+0000 7febb5400700 1 --2- 172.21.15.28:0/1285503308 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febb006a800 0x7febb019b4e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7feba000e9c0 tx=0x7feba000ecd0 comp rx=0 tx=0).stop 2024-04-23T14:43:08.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.948+0000 7febb5400700 1 -- 172.21.15.28:0/1285503308 shutdown_connections 2024-04-23T14:43:08.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.948+0000 7febb5400700 1 --2- 172.21.15.28:0/1285503308 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7feb940a3cd0 0x7feb940a6190 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:08.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.948+0000 7febb5400700 1 --2- 172.21.15.28:0/1285503308 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7febb0108b70 0x7febb019fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:08.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.948+0000 7febb5400700 1 --2- 172.21.15.28:0/1285503308 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febb006a800 0x7febb019b4e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:08.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.948+0000 7febb5400700 1 --2- 172.21.15.28:0/1285503308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febb006b1a0 0x7febb019ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:08.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.948+0000 7febb5400700 1 -- 172.21.15.28:0/1285503308 >> 172.21.15.28:0/1285503308 conn(0x7febb00789a0 msgr2=0x7febb010df50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:08.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.948+0000 7febb5400700 1 -- 172.21.15.28:0/1285503308 shutdown_connections 2024-04-23T14:43:08.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:08.948+0000 7febb5400700 1 -- 172.21.15.28:0/1285503308 wait complete. 2024-04-23T14:43:08.951 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-23T14:43:09.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:09 smithi028 ceph-mon[27810]: pgmap v360: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:43:09.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:09 smithi105 ceph-mon[32555]: pgmap v360: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:43:09.824 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":360,"stamp":"2024-04-23T14:43:07.704459+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653296,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":312,"num_read_kb":270,"num_write":262,"num_write_kb":3025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378980,"kb_used_data":13116,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124562844,"statfs":{"total":1151940427776,"available":1151552352256,"internally_reserved":0,"allocated":13430784,"data_stored":7428408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005072"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.439594+0000","last_change":"2024-04-23T14:41:40.033061+0000","last_active":"2024-04-23T14:42:43.439594+0000","last_peered":"2024-04-23T14:42:43.439594+0000","last_clean":"2024-04-23T14:42:43.439594+0000","last_became_active":"2024-04-23T14:41:40.032750+0000","last_became_peered":"2024-04-23T14:41:40.032750+0000","last_unstale":"2024-04-23T14:42:43.439594+0000","last_undegraded":"2024-04-23T14:42:43.439594+0000","last_fullsized":"2024-04-23T14:42:43.439594+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:58:03.451539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.443362+0000","last_change":"2024-04-23T14:41:40.037435+0000","last_active":"2024-04-23T14:42:43.443362+0000","last_peered":"2024-04-23T14:42:43.443362+0000","last_clean":"2024-04-23T14:42:43.443362+0000","last_became_active":"2024-04-23T14:41:40.036655+0000","last_became_peered":"2024-04-23T14:41:40.036655+0000","last_unstale":"2024-04-23T14:42:43.443362+0000","last_undegraded":"2024-04-23T14:42:43.443362+0000","last_fullsized":"2024-04-23T14:42:43.443362+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:51:58.847850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.443240+0000","last_change":"2024-04-23T14:41:40.024977+0000","last_active":"2024-04-23T14:42:43.443240+0000","last_peered":"2024-04-23T14:42:43.443240+0000","last_clean":"2024-04-23T14:42:43.443240+0000","last_became_active":"2024-04-23T14:41:40.023879+0000","last_became_peered":"2024-04-23T14:41:40.023879+0000","last_unstale":"2024-04-23T14:42:43.443240+0000","last_undegraded":"2024-04-23T14:42:43.443240+0000","last_fullsized":"2024-04-23T14:42:43.443240+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:03:40.672661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.820477+0000","last_change":"2024-04-23T14:41:40.017757+0000","last_active":"2024-04-23T14:42:43.820477+0000","last_peered":"2024-04-23T14:42:43.820477+0000","last_clean":"2024-04-23T14:42:43.820477+0000","last_became_active":"2024-04-23T14:41:40.013996+0000","last_became_peered":"2024-04-23T14:41:40.013996+0000","last_unstale":"2024-04-23T14:42:43.820477+0000","last_undegraded":"2024-04-23T14:42:43.820477+0000","last_fullsized":"2024-04-23T14:42:43.820477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:20:03.376733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.438683+0000","last_change":"2024-04-23T14:41:40.038247+0000","last_active":"2024-04-23T14:42:43.438683+0000","last_peered":"2024-04-23T14:42:43.438683+0000","last_clean":"2024-04-23T14:42:43.438683+0000","last_became_active":"2024-04-23T14:41:40.037934+0000","last_became_peered":"2024-04-23T14:41:40.037934+0000","last_unstale":"2024-04-23T14:42:43.438683+0000","last_undegraded":"2024-04-23T14:42:43.438683+0000","last_fullsized":"2024-04-23T14:42:43.438683+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:15.637733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.440257+0000","last_change":"2024-04-23T14:41:40.015341+0000","last_active":"2024-04-23T14:42:43.440257+0000","last_peered":"2024-04-23T14:42:43.440257+0000","last_clean":"2024-04-23T14:42:43.440257+0000","last_became_active":"2024-04-23T14:41:40.013627+0000","last_became_peered":"2024-04-23T14:41:40.013627+0000","last_unstale":"2024-04-23T14:42:43.440257+0000","last_undegraded":"2024-04-23T14:42:43.440257+0000","last_fullsized":"2024-04-23T14:42:43.440257+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:05:36.248107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.437295+0000","last_change":"2024-04-23T14:41:40.010950+0000","last_active":"2024-04-23T14:42:43.437295+0000","last_peered":"2024-04-23T14:42:43.437295+0000","last_clean":"2024-04-23T14:42:43.437295+0000","last_became_active":"2024-04-23T14:41:40.010584+0000","last_became_peered":"2024-04-23T14:41:40.010584+0000","last_unstale":"2024-04-23T14:42:43.437295+0000","last_undegraded":"2024-04-23T14:42:43.437295+0000","last_fullsized":"2024-04-23T14:42:43.437295+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:46:25.379243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.528280+0000","last_change":"2024-04-23T14:41:40.031198+0000","last_active":"2024-04-23T14:42:43.528280+0000","last_peered":"2024-04-23T14:42:43.528280+0000","last_clean":"2024-04-23T14:42:43.528280+0000","last_became_active":"2024-04-23T14:41:40.030652+0000","last_became_peered":"2024-04-23T14:41:40.030652+0000","last_unstale":"2024-04-23T14:42:43.528280+0000","last_undegraded":"2024-04-23T14:42:43.528280+0000","last_fullsized":"2024-04-23T14:42:43.528280+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:26:00.520182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442311+0000","last_change":"2024-04-23T14:41:40.009820+0000","last_active":"2024-04-23T14:42:43.442311+0000","last_peered":"2024-04-23T14:42:43.442311+0000","last_clean":"2024-04-23T14:42:43.442311+0000","last_became_active":"2024-04-23T14:41:40.009468+0000","last_became_peered":"2024-04-23T14:41:40.009468+0000","last_unstale":"2024-04-23T14:42:43.442311+0000","last_undegraded":"2024-04-23T14:42:43.442311+0000","last_fullsized":"2024-04-23T14:42:43.442311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:03:50.178050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.441311+0000","last_change":"2024-04-23T14:41:40.008363+0000","last_active":"2024-04-23T14:42:43.441311+0000","last_peered":"2024-04-23T14:42:43.441311+0000","last_clean":"2024-04-23T14:42:43.441311+0000","last_became_active":"2024-04-23T14:41:40.006823+0000","last_became_peered":"2024-04-23T14:41:40.006823+0000","last_unstale":"2024-04-23T14:42:43.441311+0000","last_undegraded":"2024-04-23T14:42:43.441311+0000","last_fullsized":"2024-04-23T14:42:43.441311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:12:33.718815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.437518+0000","last_change":"2024-04-23T14:41:40.021069+0000","last_active":"2024-04-23T14:42:43.437518+0000","last_peered":"2024-04-23T14:42:43.437518+0000","last_clean":"2024-04-23T14:42:43.437518+0000","last_became_active":"2024-04-23T14:41:40.020334+0000","last_became_peered":"2024-04-23T14:41:40.020334+0000","last_unstale":"2024-04-23T14:42:43.437518+0000","last_undegraded":"2024-04-23T14:42:43.437518+0000","last_fullsized":"2024-04-23T14:42:43.437518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:50:56.560928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.820435+0000","last_change":"2024-04-23T14:41:40.017746+0000","last_active":"2024-04-23T14:42:43.820435+0000","last_peered":"2024-04-23T14:42:43.820435+0000","last_clean":"2024-04-23T14:42:43.820435+0000","last_became_active":"2024-04-23T14:41:40.014495+0000","last_became_peered":"2024-04-23T14:41:40.014495+0000","last_unstale":"2024-04-23T14:42:43.820435+0000","last_undegraded":"2024-04-23T14:42:43.820435+0000","last_fullsized":"2024-04-23T14:42:43.820435+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:23:22.188110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.593556+0000","last_change":"2024-04-23T14:41:40.021429+0000","last_active":"2024-04-23T14:42:43.593556+0000","last_peered":"2024-04-23T14:42:43.593556+0000","last_clean":"2024-04-23T14:42:43.593556+0000","last_became_active":"2024-04-23T14:41:40.021087+0000","last_became_peered":"2024-04-23T14:41:40.021087+0000","last_unstale":"2024-04-23T14:42:43.593556+0000","last_undegraded":"2024-04-23T14:42:43.593556+0000","last_fullsized":"2024-04-23T14:42:43.593556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:51:16.633253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442705+0000","last_change":"2024-04-23T14:41:40.031839+0000","last_active":"2024-04-23T14:42:43.442705+0000","last_peered":"2024-04-23T14:42:43.442705+0000","last_clean":"2024-04-23T14:42:43.442705+0000","last_became_active":"2024-04-23T14:41:40.031463+0000","last_became_peered":"2024-04-23T14:41:40.031463+0000","last_unstale":"2024-04-23T14:42:43.442705+0000","last_undegraded":"2024-04-23T14:42:43.442705+0000","last_fullsized":"2024-04-23T14:42:43.442705+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:51:51.200463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442441+0000","last_change":"2024-04-23T14:41:40.042575+0000","last_active":"2024-04-23T14:42:43.442441+0000","last_peered":"2024-04-23T14:42:43.442441+0000","last_clean":"2024-04-23T14:42:43.442441+0000","last_became_active":"2024-04-23T14:41:40.033758+0000","last_became_peered":"2024-04-23T14:41:40.033758+0000","last_unstale":"2024-04-23T14:42:43.442441+0000","last_undegraded":"2024-04-23T14:42:43.442441+0000","last_fullsized":"2024-04-23T14:42:43.442441+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:55:17.338169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442663+0000","last_change":"2024-04-23T14:41:40.036214+0000","last_active":"2024-04-23T14:42:43.442663+0000","last_peered":"2024-04-23T14:42:43.442663+0000","last_clean":"2024-04-23T14:42:43.442663+0000","last_became_active":"2024-04-23T14:41:40.035892+0000","last_became_peered":"2024-04-23T14:41:40.035892+0000","last_unstale":"2024-04-23T14:42:43.442663+0000","last_undegraded":"2024-04-23T14:42:43.442663+0000","last_fullsized":"2024-04-23T14:42:43.442663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:29:18.668690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.439826+0000","last_change":"2024-04-23T14:41:40.029837+0000","last_active":"2024-04-23T14:42:43.439826+0000","last_peered":"2024-04-23T14:42:43.439826+0000","last_clean":"2024-04-23T14:42:43.439826+0000","last_became_active":"2024-04-23T14:41:40.029472+0000","last_became_peered":"2024-04-23T14:41:40.029472+0000","last_unstale":"2024-04-23T14:42:43.439826+0000","last_undegraded":"2024-04-23T14:42:43.439826+0000","last_fullsized":"2024-04-23T14:42:43.439826+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:28:52.194898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.677920+0000","last_change":"2024-04-23T14:41:40.015631+0000","last_active":"2024-04-23T14:42:43.677920+0000","last_peered":"2024-04-23T14:42:43.677920+0000","last_clean":"2024-04-23T14:42:43.677920+0000","last_became_active":"2024-04-23T14:41:40.015211+0000","last_became_peered":"2024-04-23T14:41:40.015211+0000","last_unstale":"2024-04-23T14:42:43.677920+0000","last_undegraded":"2024-04-23T14:42:43.677920+0000","last_fullsized":"2024-04-23T14:42:43.677920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:03:33.485364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.680530+0000","last_change":"2024-04-23T14:42:43.680530+0000","last_active":"2024-04-23T14:42:43.680530+0000","last_peered":"2024-04-23T14:42:43.680530+0000","last_clean":"2024-04-23T14:42:43.680530+0000","last_became_active":"2024-04-23T14:42:43.680249+0000","last_became_peered":"2024-04-23T14:42:43.680249+0000","last_unstale":"2024-04-23T14:42:43.680530+0000","last_undegraded":"2024-04-23T14:42:43.680530+0000","last_fullsized":"2024-04-23T14:42:43.680530+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:55:36.334572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,6],"acting":[2,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.677891+0000","last_change":"2024-04-23T14:41:39.017031+0000","last_active":"2024-04-23T14:42:43.677891+0000","last_peered":"2024-04-23T14:42:43.677891+0000","last_clean":"2024-04-23T14:42:43.677891+0000","last_became_active":"2024-04-23T14:41:39.016699+0000","last_became_peered":"2024-04-23T14:41:39.016699+0000","last_unstale":"2024-04-23T14:42:43.677891+0000","last_undegraded":"2024-04-23T14:42:43.677891+0000","last_fullsized":"2024-04-23T14:42:43.677891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:17:49.812601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.530569+0000","last_change":"2024-04-23T14:41:40.031957+0000","last_active":"2024-04-23T14:42:43.530569+0000","last_peered":"2024-04-23T14:42:43.530569+0000","last_clean":"2024-04-23T14:42:43.530569+0000","last_became_active":"2024-04-23T14:41:40.028151+0000","last_became_peered":"2024-04-23T14:41:40.028151+0000","last_unstale":"2024-04-23T14:42:43.530569+0000","last_undegraded":"2024-04-23T14:42:43.530569+0000","last_fullsized":"2024-04-23T14:42:43.530569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:46:48.455032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.530852+0000","last_change":"2024-04-23T14:41:39.022503+0000","last_active":"2024-04-23T14:42:43.530852+0000","last_peered":"2024-04-23T14:42:43.530852+0000","last_clean":"2024-04-23T14:42:43.530852+0000","last_became_active":"2024-04-23T14:41:39.022093+0000","last_became_peered":"2024-04-23T14:41:39.022093+0000","last_unstale":"2024-04-23T14:42:43.530852+0000","last_undegraded":"2024-04-23T14:42:43.530852+0000","last_fullsized":"2024-04-23T14:42:43.530852+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:48:27.861140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442653+0000","last_change":"2024-04-23T14:41:40.032878+0000","last_active":"2024-04-23T14:42:43.442653+0000","last_peered":"2024-04-23T14:42:43.442653+0000","last_clean":"2024-04-23T14:42:43.442653+0000","last_became_active":"2024-04-23T14:41:40.027473+0000","last_became_peered":"2024-04-23T14:41:40.027473+0000","last_unstale":"2024-04-23T14:42:43.442653+0000","last_undegraded":"2024-04-23T14:42:43.442653+0000","last_fullsized":"2024-04-23T14:42:43.442653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:00:17.677306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.449136+0000","last_change":"2024-04-23T14:42:43.448728+0000","last_active":"2024-04-23T14:42:43.449136+0000","last_peered":"2024-04-23T14:42:43.449136+0000","last_clean":"2024-04-23T14:42:43.449136+0000","last_became_active":"2024-04-23T14:42:43.443993+0000","last_became_peered":"2024-04-23T14:42:43.443993+0000","last_unstale":"2024-04-23T14:42:43.449136+0000","last_undegraded":"2024-04-23T14:42:43.449136+0000","last_fullsized":"2024-04-23T14:42:43.449136+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:04:16.430822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442116+0000","last_change":"2024-04-23T14:41:40.019955+0000","last_active":"2024-04-23T14:42:43.442116+0000","last_peered":"2024-04-23T14:42:43.442116+0000","last_clean":"2024-04-23T14:42:43.442116+0000","last_became_active":"2024-04-23T14:41:40.019579+0000","last_became_peered":"2024-04-23T14:41:40.019579+0000","last_unstale":"2024-04-23T14:42:43.442116+0000","last_undegraded":"2024-04-23T14:42:43.442116+0000","last_fullsized":"2024-04-23T14:42:43.442116+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:13:14.330051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.447172+0000","last_change":"2024-04-23T14:42:43.446652+0000","last_active":"2024-04-23T14:42:43.447172+0000","last_peered":"2024-04-23T14:42:43.447172+0000","last_clean":"2024-04-23T14:42:43.447172+0000","last_became_active":"2024-04-23T14:42:43.443549+0000","last_became_peered":"2024-04-23T14:42:43.443549+0000","last_unstale":"2024-04-23T14:42:43.447172+0000","last_undegraded":"2024-04-23T14:42:43.447172+0000","last_fullsized":"2024-04-23T14:42:43.447172+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:31:12.927663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442183+0000","last_change":"2024-04-23T14:41:40.015592+0000","last_active":"2024-04-23T14:42:43.442183+0000","last_peered":"2024-04-23T14:42:43.442183+0000","last_clean":"2024-04-23T14:42:43.442183+0000","last_became_active":"2024-04-23T14:41:40.015292+0000","last_became_peered":"2024-04-23T14:41:40.015292+0000","last_unstale":"2024-04-23T14:42:43.442183+0000","last_undegraded":"2024-04-23T14:42:43.442183+0000","last_fullsized":"2024-04-23T14:42:43.442183+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:52:01.641962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.443168+0000","last_change":"2024-04-23T14:41:39.008526+0000","last_active":"2024-04-23T14:42:43.443168+0000","last_peered":"2024-04-23T14:42:43.443168+0000","last_clean":"2024-04-23T14:42:43.443168+0000","last_became_active":"2024-04-23T14:41:39.008222+0000","last_became_peered":"2024-04-23T14:41:39.008222+0000","last_unstale":"2024-04-23T14:42:43.443168+0000","last_undegraded":"2024-04-23T14:42:43.443168+0000","last_fullsized":"2024-04-23T14:42:43.443168+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:23:09.191426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.677846+0000","last_change":"2024-04-23T14:42:43.677846+0000","last_active":"2024-04-23T14:42:43.677846+0000","last_peered":"2024-04-23T14:42:43.677846+0000","last_clean":"2024-04-23T14:42:43.677846+0000","last_became_active":"2024-04-23T14:42:43.677413+0000","last_became_peered":"2024-04-23T14:42:43.677413+0000","last_unstale":"2024-04-23T14:42:43.677846+0000","last_undegraded":"2024-04-23T14:42:43.677846+0000","last_fullsized":"2024-04-23T14:42:43.677846+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:35:55.039607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.440227+0000","last_change":"2024-04-23T14:41:39.028566+0000","last_active":"2024-04-23T14:42:43.440227+0000","last_peered":"2024-04-23T14:42:43.440227+0000","last_clean":"2024-04-23T14:42:43.440227+0000","last_became_active":"2024-04-23T14:41:39.028174+0000","last_became_peered":"2024-04-23T14:41:39.028174+0000","last_unstale":"2024-04-23T14:42:43.440227+0000","last_undegraded":"2024-04-23T14:42:43.440227+0000","last_fullsized":"2024-04-23T14:42:43.440227+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:50:55.424741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.530152+0000","last_change":"2024-04-23T14:42:43.530152+0000","last_active":"2024-04-23T14:42:43.530152+0000","last_peered":"2024-04-23T14:42:43.530152+0000","last_clean":"2024-04-23T14:42:43.530152+0000","last_became_active":"2024-04-23T14:42:43.529776+0000","last_became_peered":"2024-04-23T14:42:43.529776+0000","last_unstale":"2024-04-23T14:42:43.530152+0000","last_undegraded":"2024-04-23T14:42:43.530152+0000","last_fullsized":"2024-04-23T14:42:43.530152+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:53:29.597081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.591429+0000","last_change":"2024-04-23T14:41:39.028618+0000","last_active":"2024-04-23T14:42:43.591429+0000","last_peered":"2024-04-23T14:42:43.591429+0000","last_clean":"2024-04-23T14:42:43.591429+0000","last_became_active":"2024-04-23T14:41:39.028274+0000","last_became_peered":"2024-04-23T14:41:39.028274+0000","last_unstale":"2024-04-23T14:42:43.591429+0000","last_undegraded":"2024-04-23T14:42:43.591429+0000","last_fullsized":"2024-04-23T14:42:43.591429+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:14:31.571760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.528085+0000","last_change":"2024-04-23T14:41:40.020687+0000","last_active":"2024-04-23T14:42:43.528085+0000","last_peered":"2024-04-23T14:42:43.528085+0000","last_clean":"2024-04-23T14:42:43.528085+0000","last_became_active":"2024-04-23T14:41:40.020352+0000","last_became_peered":"2024-04-23T14:41:40.020352+0000","last_unstale":"2024-04-23T14:42:43.528085+0000","last_undegraded":"2024-04-23T14:42:43.528085+0000","last_fullsized":"2024-04-23T14:42:43.528085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:29:37.013486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.441360+0000","last_change":"2024-04-23T14:41:39.006002+0000","last_active":"2024-04-23T14:42:43.441360+0000","last_peered":"2024-04-23T14:42:43.441360+0000","last_clean":"2024-04-23T14:42:43.441360+0000","last_became_active":"2024-04-23T14:41:39.005415+0000","last_became_peered":"2024-04-23T14:41:39.005415+0000","last_unstale":"2024-04-23T14:42:43.441360+0000","last_undegraded":"2024-04-23T14:42:43.441360+0000","last_fullsized":"2024-04-23T14:42:43.441360+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:13:55.491931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.677560+0000","last_change":"2024-04-23T14:41:40.029393+0000","last_active":"2024-04-23T14:42:43.677560+0000","last_peered":"2024-04-23T14:42:43.677560+0000","last_clean":"2024-04-23T14:42:43.677560+0000","last_became_active":"2024-04-23T14:41:40.029039+0000","last_became_peered":"2024-04-23T14:41:40.029039+0000","last_unstale":"2024-04-23T14:42:43.677560+0000","last_undegraded":"2024-04-23T14:42:43.677560+0000","last_fullsized":"2024-04-23T14:42:43.677560+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:39:24.461260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.441220+0000","last_change":"2024-04-23T14:41:39.014638+0000","last_active":"2024-04-23T14:42:43.441220+0000","last_peered":"2024-04-23T14:42:43.441220+0000","last_clean":"2024-04-23T14:42:43.441220+0000","last_became_active":"2024-04-23T14:41:39.014289+0000","last_became_peered":"2024-04-23T14:41:39.014289+0000","last_unstale":"2024-04-23T14:42:43.441220+0000","last_undegraded":"2024-04-23T14:42:43.441220+0000","last_fullsized":"2024-04-23T14:42:43.441220+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:15:15.855399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442333+0000","last_change":"2024-04-23T14:41:40.015781+0000","last_active":"2024-04-23T14:42:43.442333+0000","last_peered":"2024-04-23T14:42:43.442333+0000","last_clean":"2024-04-23T14:42:43.442333+0000","last_became_active":"2024-04-23T14:41:40.015423+0000","last_became_peered":"2024-04-23T14:41:40.015423+0000","last_unstale":"2024-04-23T14:42:43.442333+0000","last_undegraded":"2024-04-23T14:42:43.442333+0000","last_fullsized":"2024-04-23T14:42:43.442333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:56:55.947585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.437580+0000","last_change":"2024-04-23T14:41:39.011021+0000","last_active":"2024-04-23T14:42:43.437580+0000","last_peered":"2024-04-23T14:42:43.437580+0000","last_clean":"2024-04-23T14:42:43.437580+0000","last_became_active":"2024-04-23T14:41:39.010637+0000","last_became_peered":"2024-04-23T14:41:39.010637+0000","last_unstale":"2024-04-23T14:42:43.437580+0000","last_undegraded":"2024-04-23T14:42:43.437580+0000","last_fullsized":"2024-04-23T14:42:43.437580+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:22:22.647170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442545+0000","last_change":"2024-04-23T14:41:40.042797+0000","last_active":"2024-04-23T14:42:43.442545+0000","last_peered":"2024-04-23T14:42:43.442545+0000","last_clean":"2024-04-23T14:42:43.442545+0000","last_became_active":"2024-04-23T14:41:40.037797+0000","last_became_peered":"2024-04-23T14:41:40.037797+0000","last_unstale":"2024-04-23T14:42:43.442545+0000","last_undegraded":"2024-04-23T14:42:43.442545+0000","last_fullsized":"2024-04-23T14:42:43.442545+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:37:32.604276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442286+0000","last_change":"2024-04-23T14:41:39.010585+0000","last_active":"2024-04-23T14:42:43.442286+0000","last_peered":"2024-04-23T14:42:43.442286+0000","last_clean":"2024-04-23T14:42:43.442286+0000","last_became_active":"2024-04-23T14:41:39.010294+0000","last_became_peered":"2024-04-23T14:41:39.010294+0000","last_unstale":"2024-04-23T14:42:43.442286+0000","last_undegraded":"2024-04-23T14:42:43.442286+0000","last_fullsized":"2024-04-23T14:42:43.442286+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:38:03.146904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442204+0000","last_change":"2024-04-23T14:41:40.013150+0000","last_active":"2024-04-23T14:42:43.442204+0000","last_peered":"2024-04-23T14:42:43.442204+0000","last_clean":"2024-04-23T14:42:43.442204+0000","last_became_active":"2024-04-23T14:41:40.012310+0000","last_became_peered":"2024-04-23T14:41:40.012310+0000","last_unstale":"2024-04-23T14:42:43.442204+0000","last_undegraded":"2024-04-23T14:42:43.442204+0000","last_fullsized":"2024-04-23T14:42:43.442204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:45:25.777200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'26","reported_seq":215,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:43:03.582686+0000","last_change":"2024-04-23T14:41:39.004256+0000","last_active":"2024-04-23T14:43:03.582686+0000","last_peered":"2024-04-23T14:43:03.582686+0000","last_clean":"2024-04-23T14:43:03.582686+0000","last_became_active":"2024-04-23T14:41:39.003874+0000","last_became_peered":"2024-04-23T14:41:39.003874+0000","last_unstale":"2024-04-23T14:43:03.582686+0000","last_undegraded":"2024-04-23T14:43:03.582686+0000","last_fullsized":"2024-04-23T14:43:03.582686+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:38:30.727149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61896,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":170,"num_read_kb":145,"num_write":26,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.438243+0000","last_change":"2024-04-23T14:41:40.012131+0000","last_active":"2024-04-23T14:42:43.438243+0000","last_peered":"2024-04-23T14:42:43.438243+0000","last_clean":"2024-04-23T14:42:43.438243+0000","last_became_active":"2024-04-23T14:41:40.011225+0000","last_became_peered":"2024-04-23T14:41:40.011225+0000","last_unstale":"2024-04-23T14:42:43.438243+0000","last_undegraded":"2024-04-23T14:42:43.438243+0000","last_fullsized":"2024-04-23T14:42:43.438243+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:36:12.085228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.441735+0000","last_change":"2024-04-23T14:41:39.001935+0000","last_active":"2024-04-23T14:42:43.441735+0000","last_peered":"2024-04-23T14:42:43.441735+0000","last_clean":"2024-04-23T14:42:43.441735+0000","last_became_active":"2024-04-23T14:41:39.001583+0000","last_became_peered":"2024-04-23T14:41:39.001583+0000","last_unstale":"2024-04-23T14:42:43.441735+0000","last_undegraded":"2024-04-23T14:42:43.441735+0000","last_fullsized":"2024-04-23T14:42:43.441735+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:47:40.986280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.592986+0000","last_change":"2024-04-23T14:41:40.036515+0000","last_active":"2024-04-23T14:42:43.592986+0000","last_peered":"2024-04-23T14:42:43.592986+0000","last_clean":"2024-04-23T14:42:43.592986+0000","last_became_active":"2024-04-23T14:41:40.036125+0000","last_became_peered":"2024-04-23T14:41:40.036125+0000","last_unstale":"2024-04-23T14:42:43.592986+0000","last_undegraded":"2024-04-23T14:42:43.592986+0000","last_fullsized":"2024-04-23T14:42:43.592986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:55:28.465378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.441733+0000","last_change":"2024-04-23T14:41:39.006127+0000","last_active":"2024-04-23T14:42:43.441733+0000","last_peered":"2024-04-23T14:42:43.441733+0000","last_clean":"2024-04-23T14:42:43.441733+0000","last_became_active":"2024-04-23T14:41:39.005761+0000","last_became_peered":"2024-04-23T14:41:39.005761+0000","last_unstale":"2024-04-23T14:42:43.441733+0000","last_undegraded":"2024-04-23T14:42:43.441733+0000","last_fullsized":"2024-04-23T14:42:43.441733+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:10:24.050016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.678799+0000","last_change":"2024-04-23T14:42:43.678799+0000","last_active":"2024-04-23T14:42:43.678799+0000","last_peered":"2024-04-23T14:42:43.678799+0000","last_clean":"2024-04-23T14:42:43.678799+0000","last_became_active":"2024-04-23T14:42:43.678318+0000","last_became_peered":"2024-04-23T14:42:43.678318+0000","last_unstale":"2024-04-23T14:42:43.678799+0000","last_undegraded":"2024-04-23T14:42:43.678799+0000","last_fullsized":"2024-04-23T14:42:43.678799+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:26:21.827247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.820290+0000","last_change":"2024-04-23T14:41:39.005873+0000","last_active":"2024-04-23T14:42:43.820290+0000","last_peered":"2024-04-23T14:42:43.820290+0000","last_clean":"2024-04-23T14:42:43.820290+0000","last_became_active":"2024-04-23T14:41:39.005481+0000","last_became_peered":"2024-04-23T14:41:39.005481+0000","last_unstale":"2024-04-23T14:42:43.820290+0000","last_undegraded":"2024-04-23T14:42:43.820290+0000","last_fullsized":"2024-04-23T14:42:43.820290+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:49:59.117451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.437246+0000","last_change":"2024-04-23T14:41:40.009346+0000","last_active":"2024-04-23T14:42:43.437246+0000","last_peered":"2024-04-23T14:42:43.437246+0000","last_clean":"2024-04-23T14:42:43.437246+0000","last_became_active":"2024-04-23T14:41:40.008927+0000","last_became_peered":"2024-04-23T14:41:40.008927+0000","last_unstale":"2024-04-23T14:42:43.437246+0000","last_undegraded":"2024-04-23T14:42:43.437246+0000","last_fullsized":"2024-04-23T14:42:43.437246+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:33:36.513349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":43,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.452178+0000","last_change":"2024-04-23T14:42:43.452178+0000","last_active":"2024-04-23T14:42:43.452178+0000","last_peered":"2024-04-23T14:42:43.452178+0000","last_clean":"2024-04-23T14:42:43.452178+0000","last_became_active":"2024-04-23T14:42:43.444786+0000","last_became_peered":"2024-04-23T14:42:43.444786+0000","last_unstale":"2024-04-23T14:42:43.452178+0000","last_undegraded":"2024-04-23T14:42:43.452178+0000","last_fullsized":"2024-04-23T14:42:43.452178+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:00:21.945663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,1],"acting":[8,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.441367+0000","last_change":"2024-04-23T14:41:40.016845+0000","last_active":"2024-04-23T14:42:43.441367+0000","last_peered":"2024-04-23T14:42:43.441367+0000","last_clean":"2024-04-23T14:42:43.441367+0000","last_became_active":"2024-04-23T14:41:40.016543+0000","last_became_peered":"2024-04-23T14:41:40.016543+0000","last_unstale":"2024-04-23T14:42:43.441367+0000","last_undegraded":"2024-04-23T14:42:43.441367+0000","last_fullsized":"2024-04-23T14:42:43.441367+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:52:41.054454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.438163+0000","last_change":"2024-04-23T14:41:38.999968+0000","last_active":"2024-04-23T14:42:43.438163+0000","last_peered":"2024-04-23T14:42:43.438163+0000","last_clean":"2024-04-23T14:42:43.438163+0000","last_became_active":"2024-04-23T14:41:38.999658+0000","last_became_peered":"2024-04-23T14:41:38.999658+0000","last_unstale":"2024-04-23T14:42:43.438163+0000","last_undegraded":"2024-04-23T14:42:43.438163+0000","last_fullsized":"2024-04-23T14:42:43.438163+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:14:20.990331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.439374+0000","last_change":"2024-04-23T14:41:40.011753+0000","last_active":"2024-04-23T14:42:43.439374+0000","last_peered":"2024-04-23T14:42:43.439374+0000","last_clean":"2024-04-23T14:42:43.439374+0000","last_became_active":"2024-04-23T14:41:40.011264+0000","last_became_peered":"2024-04-23T14:41:40.011264+0000","last_unstale":"2024-04-23T14:42:43.439374+0000","last_undegraded":"2024-04-23T14:42:43.439374+0000","last_fullsized":"2024-04-23T14:42:43.439374+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:52:22.812046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.677877+0000","last_change":"2024-04-23T14:41:39.027384+0000","last_active":"2024-04-23T14:42:43.677877+0000","last_peered":"2024-04-23T14:42:43.677877+0000","last_clean":"2024-04-23T14:42:43.677877+0000","last_became_active":"2024-04-23T14:41:39.027018+0000","last_became_peered":"2024-04-23T14:41:39.027018+0000","last_unstale":"2024-04-23T14:42:43.677877+0000","last_undegraded":"2024-04-23T14:42:43.677877+0000","last_fullsized":"2024-04-23T14:42:43.677877+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:13:54.685704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.441143+0000","last_change":"2024-04-23T14:41:40.041216+0000","last_active":"2024-04-23T14:42:43.441143+0000","last_peered":"2024-04-23T14:42:43.441143+0000","last_clean":"2024-04-23T14:42:43.441143+0000","last_became_active":"2024-04-23T14:41:40.040789+0000","last_became_peered":"2024-04-23T14:41:40.040789+0000","last_unstale":"2024-04-23T14:42:43.441143+0000","last_undegraded":"2024-04-23T14:42:43.441143+0000","last_fullsized":"2024-04-23T14:42:43.441143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:01:13.156704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.437436+0000","last_change":"2024-04-23T14:41:39.024399+0000","last_active":"2024-04-23T14:42:43.437436+0000","last_peered":"2024-04-23T14:42:43.437436+0000","last_clean":"2024-04-23T14:42:43.437436+0000","last_became_active":"2024-04-23T14:41:39.024141+0000","last_became_peered":"2024-04-23T14:41:39.024141+0000","last_unstale":"2024-04-23T14:42:43.437436+0000","last_undegraded":"2024-04-23T14:42:43.437436+0000","last_fullsized":"2024-04-23T14:42:43.437436+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:34:27.516614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.680818+0000","last_change":"2024-04-23T14:42:43.680818+0000","last_active":"2024-04-23T14:42:43.680818+0000","last_peered":"2024-04-23T14:42:43.680818+0000","last_clean":"2024-04-23T14:42:43.680818+0000","last_became_active":"2024-04-23T14:42:43.680463+0000","last_became_peered":"2024-04-23T14:42:43.680463+0000","last_unstale":"2024-04-23T14:42:43.680818+0000","last_undegraded":"2024-04-23T14:42:43.680818+0000","last_fullsized":"2024-04-23T14:42:43.680818+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:34:36.964359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442386+0000","last_change":"2024-04-23T14:41:40.014689+0000","last_active":"2024-04-23T14:42:43.442386+0000","last_peered":"2024-04-23T14:42:43.442386+0000","last_clean":"2024-04-23T14:42:43.442386+0000","last_became_active":"2024-04-23T14:41:40.014371+0000","last_became_peered":"2024-04-23T14:41:40.014371+0000","last_unstale":"2024-04-23T14:42:43.442386+0000","last_undegraded":"2024-04-23T14:42:43.442386+0000","last_fullsized":"2024-04-23T14:42:43.442386+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:56:41.284144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.591269+0000","last_change":"2024-04-23T14:41:39.031327+0000","last_active":"2024-04-23T14:42:43.591269+0000","last_peered":"2024-04-23T14:42:43.591269+0000","last_clean":"2024-04-23T14:42:43.591269+0000","last_became_active":"2024-04-23T14:41:39.030864+0000","last_became_peered":"2024-04-23T14:41:39.030864+0000","last_unstale":"2024-04-23T14:42:43.591269+0000","last_undegraded":"2024-04-23T14:42:43.591269+0000","last_fullsized":"2024-04-23T14:42:43.591269+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:58:24.048207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.529243+0000","last_change":"2024-04-23T14:41:40.035104+0000","last_active":"2024-04-23T14:42:43.529243+0000","last_peered":"2024-04-23T14:42:43.529243+0000","last_clean":"2024-04-23T14:42:43.529243+0000","last_became_active":"2024-04-23T14:41:40.034678+0000","last_became_peered":"2024-04-23T14:41:40.034678+0000","last_unstale":"2024-04-23T14:42:43.529243+0000","last_undegraded":"2024-04-23T14:42:43.529243+0000","last_fullsized":"2024-04-23T14:42:43.529243+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:29:53.040424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.821031+0000","last_change":"2024-04-23T14:41:39.016684+0000","last_active":"2024-04-23T14:42:43.821031+0000","last_peered":"2024-04-23T14:42:43.821031+0000","last_clean":"2024-04-23T14:42:43.821031+0000","last_became_active":"2024-04-23T14:41:39.016375+0000","last_became_peered":"2024-04-23T14:41:39.016375+0000","last_unstale":"2024-04-23T14:42:43.821031+0000","last_undegraded":"2024-04-23T14:42:43.821031+0000","last_fullsized":"2024-04-23T14:42:43.821031+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:09.449590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":355,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.437363+0000","last_change":"2024-04-23T14:40:22.305413+0000","last_active":"2024-04-23T14:42:43.437363+0000","last_peered":"2024-04-23T14:42:43.437363+0000","last_clean":"2024-04-23T14:42:43.437363+0000","last_became_active":"2024-04-23T14:40:22.304977+0000","last_became_peered":"2024-04-23T14:40:22.304977+0000","last_unstale":"2024-04-23T14:42:43.437363+0000","last_undegraded":"2024-04-23T14:42:43.437363+0000","last_fullsized":"2024-04-23T14:42:43.437363+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:36:28.336753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:36:28.336753+0000","last_clean_scrub_stamp":"2024-04-23T14:36:28.336753+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:42:27.611770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.591222+0000","last_change":"2024-04-23T14:41:40.023300+0000","last_active":"2024-04-23T14:42:43.591222+0000","last_peered":"2024-04-23T14:42:43.591222+0000","last_clean":"2024-04-23T14:42:43.591222+0000","last_became_active":"2024-04-23T14:41:40.022888+0000","last_became_peered":"2024-04-23T14:41:40.022888+0000","last_unstale":"2024-04-23T14:42:43.591222+0000","last_undegraded":"2024-04-23T14:42:43.591222+0000","last_fullsized":"2024-04-23T14:42:43.591222+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:31:48.705005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.437426+0000","last_change":"2024-04-23T14:41:39.005482+0000","last_active":"2024-04-23T14:42:43.437426+0000","last_peered":"2024-04-23T14:42:43.437426+0000","last_clean":"2024-04-23T14:42:43.437426+0000","last_became_active":"2024-04-23T14:41:39.004552+0000","last_became_peered":"2024-04-23T14:41:39.004552+0000","last_unstale":"2024-04-23T14:42:43.437426+0000","last_undegraded":"2024-04-23T14:42:43.437426+0000","last_fullsized":"2024-04-23T14:42:43.437426+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:05:51.411210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.438626+0000","last_change":"2024-04-23T14:41:39.015605+0000","last_active":"2024-04-23T14:42:43.438626+0000","last_peered":"2024-04-23T14:42:43.438626+0000","last_clean":"2024-04-23T14:42:43.438626+0000","last_became_active":"2024-04-23T14:41:39.015260+0000","last_became_peered":"2024-04-23T14:41:39.015260+0000","last_unstale":"2024-04-23T14:42:43.438626+0000","last_undegraded":"2024-04-23T14:42:43.438626+0000","last_fullsized":"2024-04-23T14:42:43.438626+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:23:21.251483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.593376+0000","last_change":"2024-04-23T14:41:40.020619+0000","last_active":"2024-04-23T14:42:43.593376+0000","last_peered":"2024-04-23T14:42:43.593376+0000","last_clean":"2024-04-23T14:42:43.593376+0000","last_became_active":"2024-04-23T14:41:40.020191+0000","last_became_peered":"2024-04-23T14:41:40.020191+0000","last_unstale":"2024-04-23T14:42:43.593376+0000","last_undegraded":"2024-04-23T14:42:43.593376+0000","last_fullsized":"2024-04-23T14:42:43.593376+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:54:34.036464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.677562+0000","last_change":"2024-04-23T14:41:39.019507+0000","last_active":"2024-04-23T14:42:43.677562+0000","last_peered":"2024-04-23T14:42:43.677562+0000","last_clean":"2024-04-23T14:42:43.677562+0000","last_became_active":"2024-04-23T14:41:39.019116+0000","last_became_peered":"2024-04-23T14:41:39.019116+0000","last_unstale":"2024-04-23T14:42:43.677562+0000","last_undegraded":"2024-04-23T14:42:43.677562+0000","last_fullsized":"2024-04-23T14:42:43.677562+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:06:05.777908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.531207+0000","last_change":"2024-04-23T14:41:40.032765+0000","last_active":"2024-04-23T14:42:43.531207+0000","last_peered":"2024-04-23T14:42:43.531207+0000","last_clean":"2024-04-23T14:42:43.531207+0000","last_became_active":"2024-04-23T14:41:40.027896+0000","last_became_peered":"2024-04-23T14:41:40.027896+0000","last_unstale":"2024-04-23T14:42:43.531207+0000","last_undegraded":"2024-04-23T14:42:43.531207+0000","last_fullsized":"2024-04-23T14:42:43.531207+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:36:16.225444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.820509+0000","last_change":"2024-04-23T14:41:39.021681+0000","last_active":"2024-04-23T14:42:43.820509+0000","last_peered":"2024-04-23T14:42:43.820509+0000","last_clean":"2024-04-23T14:42:43.820509+0000","last_became_active":"2024-04-23T14:41:39.021423+0000","last_became_peered":"2024-04-23T14:41:39.021423+0000","last_unstale":"2024-04-23T14:42:43.820509+0000","last_undegraded":"2024-04-23T14:42:43.820509+0000","last_fullsized":"2024-04-23T14:42:43.820509+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:42:55.351296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.443567+0000","last_change":"2024-04-23T14:41:40.037250+0000","last_active":"2024-04-23T14:42:43.443567+0000","last_peered":"2024-04-23T14:42:43.443567+0000","last_clean":"2024-04-23T14:42:43.443567+0000","last_became_active":"2024-04-23T14:41:40.036645+0000","last_became_peered":"2024-04-23T14:41:40.036645+0000","last_unstale":"2024-04-23T14:42:43.443567+0000","last_undegraded":"2024-04-23T14:42:43.443567+0000","last_fullsized":"2024-04-23T14:42:43.443567+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:50:35.556841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442618+0000","last_change":"2024-04-23T14:41:39.007764+0000","last_active":"2024-04-23T14:42:43.442618+0000","last_peered":"2024-04-23T14:42:43.442618+0000","last_clean":"2024-04-23T14:42:43.442618+0000","last_became_active":"2024-04-23T14:41:39.007312+0000","last_became_peered":"2024-04-23T14:41:39.007312+0000","last_unstale":"2024-04-23T14:42:43.442618+0000","last_undegraded":"2024-04-23T14:42:43.442618+0000","last_fullsized":"2024-04-23T14:42:43.442618+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:38:56.021238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.437731+0000","last_change":"2024-04-23T14:41:40.004343+0000","last_active":"2024-04-23T14:42:43.437731+0000","last_peered":"2024-04-23T14:42:43.437731+0000","last_clean":"2024-04-23T14:42:43.437731+0000","last_became_active":"2024-04-23T14:41:40.004015+0000","last_became_peered":"2024-04-23T14:41:40.004015+0000","last_unstale":"2024-04-23T14:42:43.437731+0000","last_undegraded":"2024-04-23T14:42:43.437731+0000","last_fullsized":"2024-04-23T14:42:43.437731+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:05:33.863692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.528699+0000","last_change":"2024-04-23T14:41:39.016271+0000","last_active":"2024-04-23T14:42:43.528699+0000","last_peered":"2024-04-23T14:42:43.528699+0000","last_clean":"2024-04-23T14:42:43.528699+0000","last_became_active":"2024-04-23T14:41:39.015851+0000","last_became_peered":"2024-04-23T14:41:39.015851+0000","last_unstale":"2024-04-23T14:42:43.528699+0000","last_undegraded":"2024-04-23T14:42:43.528699+0000","last_fullsized":"2024-04-23T14:42:43.528699+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:50:52.943534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.439085+0000","last_change":"2024-04-23T14:41:40.029815+0000","last_active":"2024-04-23T14:42:43.439085+0000","last_peered":"2024-04-23T14:42:43.439085+0000","last_clean":"2024-04-23T14:42:43.439085+0000","last_became_active":"2024-04-23T14:41:40.029496+0000","last_became_peered":"2024-04-23T14:41:40.029496+0000","last_unstale":"2024-04-23T14:42:43.439085+0000","last_undegraded":"2024-04-23T14:42:43.439085+0000","last_fullsized":"2024-04-23T14:42:43.439085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:36:09.228993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442420+0000","last_change":"2024-04-23T14:41:39.001946+0000","last_active":"2024-04-23T14:42:43.442420+0000","last_peered":"2024-04-23T14:42:43.442420+0000","last_clean":"2024-04-23T14:42:43.442420+0000","last_became_active":"2024-04-23T14:41:39.001619+0000","last_became_peered":"2024-04-23T14:41:39.001619+0000","last_unstale":"2024-04-23T14:42:43.442420+0000","last_undegraded":"2024-04-23T14:42:43.442420+0000","last_fullsized":"2024-04-23T14:42:43.442420+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:43:57.680195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.528966+0000","last_change":"2024-04-23T14:41:40.045728+0000","last_active":"2024-04-23T14:42:43.528966+0000","last_peered":"2024-04-23T14:42:43.528966+0000","last_clean":"2024-04-23T14:42:43.528966+0000","last_became_active":"2024-04-23T14:41:40.045293+0000","last_became_peered":"2024-04-23T14:41:40.045293+0000","last_unstale":"2024-04-23T14:42:43.528966+0000","last_undegraded":"2024-04-23T14:42:43.528966+0000","last_fullsized":"2024-04-23T14:42:43.528966+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:35:54.340644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.823943+0000","last_change":"2024-04-23T14:42:43.823943+0000","last_active":"2024-04-23T14:42:43.823943+0000","last_peered":"2024-04-23T14:42:43.823943+0000","last_clean":"2024-04-23T14:42:43.823943+0000","last_became_active":"2024-04-23T14:42:43.821663+0000","last_became_peered":"2024-04-23T14:42:43.821663+0000","last_unstale":"2024-04-23T14:42:43.823943+0000","last_undegraded":"2024-04-23T14:42:43.823943+0000","last_fullsized":"2024-04-23T14:42:43.823943+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:18:54.895943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.437538+0000","last_change":"2024-04-23T14:41:40.028897+0000","last_active":"2024-04-23T14:42:43.437538+0000","last_peered":"2024-04-23T14:42:43.437538+0000","last_clean":"2024-04-23T14:42:43.437538+0000","last_became_active":"2024-04-23T14:41:40.028557+0000","last_became_peered":"2024-04-23T14:41:40.028557+0000","last_unstale":"2024-04-23T14:42:43.437538+0000","last_undegraded":"2024-04-23T14:42:43.437538+0000","last_fullsized":"2024-04-23T14:42:43.437538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:26:57.871013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442160+0000","last_change":"2024-04-23T14:41:39.012861+0000","last_active":"2024-04-23T14:42:43.442160+0000","last_peered":"2024-04-23T14:42:43.442160+0000","last_clean":"2024-04-23T14:42:43.442160+0000","last_became_active":"2024-04-23T14:41:39.012503+0000","last_became_peered":"2024-04-23T14:41:39.012503+0000","last_unstale":"2024-04-23T14:42:43.442160+0000","last_undegraded":"2024-04-23T14:42:43.442160+0000","last_fullsized":"2024-04-23T14:42:43.442160+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:41:57.347221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.437801+0000","last_change":"2024-04-23T14:41:40.021580+0000","last_active":"2024-04-23T14:42:43.437801+0000","last_peered":"2024-04-23T14:42:43.437801+0000","last_clean":"2024-04-23T14:42:43.437801+0000","last_became_active":"2024-04-23T14:41:40.020706+0000","last_became_peered":"2024-04-23T14:41:40.020706+0000","last_unstale":"2024-04-23T14:42:43.437801+0000","last_undegraded":"2024-04-23T14:42:43.437801+0000","last_fullsized":"2024-04-23T14:42:43.437801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:59:42.576276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.443072+0000","last_change":"2024-04-23T14:41:39.006425+0000","last_active":"2024-04-23T14:42:43.443072+0000","last_peered":"2024-04-23T14:42:43.443072+0000","last_clean":"2024-04-23T14:42:43.443072+0000","last_became_active":"2024-04-23T14:41:39.006123+0000","last_became_peered":"2024-04-23T14:41:39.006123+0000","last_unstale":"2024-04-23T14:42:43.443072+0000","last_undegraded":"2024-04-23T14:42:43.443072+0000","last_fullsized":"2024-04-23T14:42:43.443072+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:08:37.261092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.437457+0000","last_change":"2024-04-23T14:41:40.034063+0000","last_active":"2024-04-23T14:42:43.437457+0000","last_peered":"2024-04-23T14:42:43.437457+0000","last_clean":"2024-04-23T14:42:43.437457+0000","last_became_active":"2024-04-23T14:41:40.033669+0000","last_became_peered":"2024-04-23T14:41:40.033669+0000","last_unstale":"2024-04-23T14:42:43.437457+0000","last_undegraded":"2024-04-23T14:42:43.437457+0000","last_fullsized":"2024-04-23T14:42:43.437457+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:23:50.608041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.593343+0000","last_change":"2024-04-23T14:41:39.025975+0000","last_active":"2024-04-23T14:42:43.593343+0000","last_peered":"2024-04-23T14:42:43.593343+0000","last_clean":"2024-04-23T14:42:43.593343+0000","last_became_active":"2024-04-23T14:41:39.025629+0000","last_became_peered":"2024-04-23T14:41:39.025629+0000","last_unstale":"2024-04-23T14:42:43.593343+0000","last_undegraded":"2024-04-23T14:42:43.593343+0000","last_fullsized":"2024-04-23T14:42:43.593343+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:00:03.753547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.437798+0000","last_change":"2024-04-23T14:41:40.013030+0000","last_active":"2024-04-23T14:42:43.437798+0000","last_peered":"2024-04-23T14:42:43.437798+0000","last_clean":"2024-04-23T14:42:43.437798+0000","last_became_active":"2024-04-23T14:41:40.012708+0000","last_became_peered":"2024-04-23T14:41:40.012708+0000","last_unstale":"2024-04-23T14:42:43.437798+0000","last_undegraded":"2024-04-23T14:42:43.437798+0000","last_fullsized":"2024-04-23T14:42:43.437798+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:05:29.217273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.439174+0000","last_change":"2024-04-23T14:41:40.010551+0000","last_active":"2024-04-23T14:42:43.439174+0000","last_peered":"2024-04-23T14:42:43.439174+0000","last_clean":"2024-04-23T14:42:43.439174+0000","last_became_active":"2024-04-23T14:41:40.010244+0000","last_became_peered":"2024-04-23T14:41:40.010244+0000","last_unstale":"2024-04-23T14:42:43.439174+0000","last_undegraded":"2024-04-23T14:42:43.439174+0000","last_fullsized":"2024-04-23T14:42:43.439174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:03:57.223815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.441829+0000","last_change":"2024-04-23T14:41:40.009132+0000","last_active":"2024-04-23T14:42:43.441829+0000","last_peered":"2024-04-23T14:42:43.441829+0000","last_clean":"2024-04-23T14:42:43.441829+0000","last_became_active":"2024-04-23T14:41:40.008822+0000","last_became_peered":"2024-04-23T14:41:40.008822+0000","last_unstale":"2024-04-23T14:42:43.441829+0000","last_undegraded":"2024-04-23T14:42:43.441829+0000","last_fullsized":"2024-04-23T14:42:43.441829+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:10:28.513846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442333+0000","last_change":"2024-04-23T14:41:40.041193+0000","last_active":"2024-04-23T14:42:43.442333+0000","last_peered":"2024-04-23T14:42:43.442333+0000","last_clean":"2024-04-23T14:42:43.442333+0000","last_became_active":"2024-04-23T14:41:40.040851+0000","last_became_peered":"2024-04-23T14:41:40.040851+0000","last_unstale":"2024-04-23T14:42:43.442333+0000","last_undegraded":"2024-04-23T14:42:43.442333+0000","last_fullsized":"2024-04-23T14:42:43.442333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:13:30.090015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442731+0000","last_change":"2024-04-23T14:41:40.042703+0000","last_active":"2024-04-23T14:42:43.442731+0000","last_peered":"2024-04-23T14:42:43.442731+0000","last_clean":"2024-04-23T14:42:43.442731+0000","last_became_active":"2024-04-23T14:41:40.037666+0000","last_became_peered":"2024-04-23T14:41:40.037666+0000","last_unstale":"2024-04-23T14:42:43.442731+0000","last_undegraded":"2024-04-23T14:42:43.442731+0000","last_fullsized":"2024-04-23T14:42:43.442731+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:04:04.240689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.593484+0000","last_change":"2024-04-23T14:41:40.033487+0000","last_active":"2024-04-23T14:42:43.593484+0000","last_peered":"2024-04-23T14:42:43.593484+0000","last_clean":"2024-04-23T14:42:43.593484+0000","last_became_active":"2024-04-23T14:41:40.031799+0000","last_became_peered":"2024-04-23T14:41:40.031799+0000","last_unstale":"2024-04-23T14:42:43.593484+0000","last_undegraded":"2024-04-23T14:42:43.593484+0000","last_fullsized":"2024-04-23T14:42:43.593484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:36:03.930264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.439616+0000","last_change":"2024-04-23T14:41:40.012833+0000","last_active":"2024-04-23T14:42:43.439616+0000","last_peered":"2024-04-23T14:42:43.439616+0000","last_clean":"2024-04-23T14:42:43.439616+0000","last_became_active":"2024-04-23T14:41:40.012251+0000","last_became_peered":"2024-04-23T14:41:40.012251+0000","last_unstale":"2024-04-23T14:42:43.439616+0000","last_undegraded":"2024-04-23T14:42:43.439616+0000","last_fullsized":"2024-04-23T14:42:43.439616+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:17:09.415998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442491+0000","last_change":"2024-04-23T14:41:40.013336+0000","last_active":"2024-04-23T14:42:43.442491+0000","last_peered":"2024-04-23T14:42:43.442491+0000","last_clean":"2024-04-23T14:42:43.442491+0000","last_became_active":"2024-04-23T14:41:40.013043+0000","last_became_peered":"2024-04-23T14:41:40.013043+0000","last_unstale":"2024-04-23T14:42:43.442491+0000","last_undegraded":"2024-04-23T14:42:43.442491+0000","last_fullsized":"2024-04-23T14:42:43.442491+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:27:59.118986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.528540+0000","last_change":"2024-04-23T14:41:40.028236+0000","last_active":"2024-04-23T14:42:43.528540+0000","last_peered":"2024-04-23T14:42:43.528540+0000","last_clean":"2024-04-23T14:42:43.528540+0000","last_became_active":"2024-04-23T14:41:40.027504+0000","last_became_peered":"2024-04-23T14:41:40.027504+0000","last_unstale":"2024-04-23T14:42:43.528540+0000","last_undegraded":"2024-04-23T14:42:43.528540+0000","last_fullsized":"2024-04-23T14:42:43.528540+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:04:31.540482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442292+0000","last_change":"2024-04-23T14:41:40.029252+0000","last_active":"2024-04-23T14:42:43.442292+0000","last_peered":"2024-04-23T14:42:43.442292+0000","last_clean":"2024-04-23T14:42:43.442292+0000","last_became_active":"2024-04-23T14:41:40.028937+0000","last_became_peered":"2024-04-23T14:41:40.028937+0000","last_unstale":"2024-04-23T14:42:43.442292+0000","last_undegraded":"2024-04-23T14:42:43.442292+0000","last_fullsized":"2024-04-23T14:42:43.442292+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:46:44.441319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.528988+0000","last_change":"2024-04-23T14:41:40.027705+0000","last_active":"2024-04-23T14:42:43.528988+0000","last_peered":"2024-04-23T14:42:43.528988+0000","last_clean":"2024-04-23T14:42:43.528988+0000","last_became_active":"2024-04-23T14:41:40.027260+0000","last_became_peered":"2024-04-23T14:41:40.027260+0000","last_unstale":"2024-04-23T14:42:43.528988+0000","last_undegraded":"2024-04-23T14:42:43.528988+0000","last_fullsized":"2024-04-23T14:42:43.528988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:33:42.461219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.442119+0000","last_change":"2024-04-23T14:41:40.026490+0000","last_active":"2024-04-23T14:42:43.442119+0000","last_peered":"2024-04-23T14:42:43.442119+0000","last_clean":"2024-04-23T14:42:43.442119+0000","last_became_active":"2024-04-23T14:41:40.026117+0000","last_became_peered":"2024-04-23T14:41:40.026117+0000","last_unstale":"2024-04-23T14:42:43.442119+0000","last_undegraded":"2024-04-23T14:42:43.442119+0000","last_fullsized":"2024-04-23T14:42:43.442119+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:06:17.626663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.593324+0000","last_change":"2024-04-23T14:41:40.038489+0000","last_active":"2024-04-23T14:42:43.593324+0000","last_peered":"2024-04-23T14:42:43.593324+0000","last_clean":"2024-04-23T14:42:43.593324+0000","last_became_active":"2024-04-23T14:41:40.038078+0000","last_became_peered":"2024-04-23T14:41:40.038078+0000","last_unstale":"2024-04-23T14:42:43.593324+0000","last_undegraded":"2024-04-23T14:42:43.593324+0000","last_fullsized":"2024-04-23T14:42:43.593324+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:58:59.440468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T14:42:43.439967+0000","last_change":"2024-04-23T14:41:40.009463+0000","last_active":"2024-04-23T14:42:43.439967+0000","last_peered":"2024-04-23T14:42:43.439967+0000","last_clean":"2024-04-23T14:42:43.439967+0000","last_became_active":"2024-04-23T14:41:40.009154+0000","last_became_peered":"2024-04-23T14:41:40.009154+0000","last_unstale":"2024-04-23T14:42:43.439967+0000","last_undegraded":"2024-04-23T14:42:43.439967+0000","last_fullsized":"2024-04-23T14:42:43.439967+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:13:47.641863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":62694,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":176,"num_read_kb":151,"num_write":79,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":188082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383711,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":946176,"data_stored":455710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645348,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28608,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716544,"statfs":{"total":95995035648,"available":95965741056,"internally_reserved":0,"allocated":1626112,"data_stored":1108285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808874,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36272,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708880,"statfs":{"total":95995035648,"available":95957893120,"internally_reserved":0,"allocated":950272,"data_stored":456292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070512,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":456314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332151,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32696,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712456,"statfs":{"total":95995035648,"available":95961554944,"internally_reserved":0,"allocated":1548288,"data_stored":1046777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495675,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":950272,"data_stored":455827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1015808,"data_stored":517606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738462,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36272,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708880,"statfs":{"total":95995035648,"available":95957893120,"internally_reserved":0,"allocated":950272,"data_stored":455800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411411,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36340,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708812,"statfs":{"total":95995035648,"available":95957823488,"internally_reserved":0,"allocated":1019904,"data_stored":517640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182476,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":962560,"data_stored":456048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920840,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":950272,"data_stored":455732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757313,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28540,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716612,"statfs":{"total":95995035648,"available":95965810688,"internally_reserved":0,"allocated":1556480,"data_stored":1046377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":61930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":62090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T14:43:09.828 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-23T14:43:09.828 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-23T14:43:09.828 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-23T14:43:09.828 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-23T14:43:09.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:09 smithi038 ceph-mon[32348]: pgmap v360: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:43:10.325 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:43:10.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:10 smithi028 ceph-mon[27810]: from='client.24899 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T14:43:10.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:10 smithi105 ceph-mon[32555]: from='client.24899 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T14:43:10.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:10 smithi038 ceph-mon[32348]: from='client.24899 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T14:43:11.765 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:11 smithi028 ceph-mon[27810]: pgmap v361: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:43:11.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:11 smithi105 ceph-mon[32555]: pgmap v361: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:43:11.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:11 smithi038 ceph-mon[32348]: pgmap v361: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:43:12.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.600+0000 7fdb9ba00700 1 -- 172.21.15.28:0/4260104575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb94077020 msgr2=0x7fdb94077400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:12.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.600+0000 7fdb9ba00700 1 --2- 172.21.15.28:0/4260104575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb94077020 0x7fdb94077400 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdb84009f90 tx=0x7fdb8400b330 comp rx=0 tx=0).stop 2024-04-23T14:43:12.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.600+0000 7fdb9ba00700 1 -- 172.21.15.28:0/4260104575 shutdown_connections 2024-04-23T14:43:12.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.600+0000 7fdb9ba00700 1 --2- 172.21.15.28:0/4260104575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb941139a0 0x7fdb94115d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:12.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.600+0000 7fdb9ba00700 1 --2- 172.21.15.28:0/4260104575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb94077020 0x7fdb94077400 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:12.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.600+0000 7fdb9ba00700 1 --2- 172.21.15.28:0/4260104575 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb94077940 0x7fdb94113460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:12.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.600+0000 7fdb9ba00700 1 -- 172.21.15.28:0/4260104575 >> 172.21.15.28:0/4260104575 conn(0x7fdb941007f0 msgr2=0x7fdb94102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:12.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.601+0000 7fdb9ba00700 1 -- 172.21.15.28:0/4260104575 shutdown_connections 2024-04-23T14:43:12.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.601+0000 7fdb9ba00700 1 -- 172.21.15.28:0/4260104575 wait complete. 2024-04-23T14:43:12.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.602+0000 7fdb9ba00700 1 Processor -- start 2024-04-23T14:43:12.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.602+0000 7fdb9ba00700 1 -- start start 2024-04-23T14:43:12.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.603+0000 7fdb9ba00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb94077020 0x7fdb941a5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:12.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.603+0000 7fdb9ba00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb94077940 0x7fdb941a5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:12.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.603+0000 7fdb9ba00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb941139a0 0x7fdb9419f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:12.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.603+0000 7fdb9ba00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdb94119490 con 0x7fdb94077940 2024-04-23T14:43:12.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.603+0000 7fdb9ba00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdb94119310 con 0x7fdb941139a0 2024-04-23T14:43:12.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.603+0000 7fdb9ba00700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdb94119610 con 0x7fdb94077020 2024-04-23T14:43:12.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.603+0000 7fdb99000700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb94077020 0x7fdb941a5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:12.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.603+0000 7fdb8be00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb94077940 0x7fdb941a5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:12.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.603+0000 7fdb99000700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb94077020 0x7fdb941a5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:58734/0 (socket says 172.21.15.28:58734) 2024-04-23T14:43:12.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.603+0000 7fdb99000700 1 -- 172.21.15.28:0/1381541225 learned_addr learned my addr 172.21.15.28:0/1381541225 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:12.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.603+0000 7fdb99a00700 1 --2- 172.21.15.28:0/1381541225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb941139a0 0x7fdb9419f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:12.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.604+0000 7fdb99000700 1 -- 172.21.15.28:0/1381541225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb941139a0 msgr2=0x7fdb9419f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:12.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.604+0000 7fdb99000700 1 --2- 172.21.15.28:0/1381541225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb941139a0 0x7fdb9419f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:12.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.604+0000 7fdb99000700 1 -- 172.21.15.28:0/1381541225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb94077940 msgr2=0x7fdb941a5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:12.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.604+0000 7fdb99000700 1 --2- 172.21.15.28:0/1381541225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb94077940 0x7fdb941a5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:12.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.604+0000 7fdb99000700 1 -- 172.21.15.28:0/1381541225 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb941a0050 con 0x7fdb94077020 2024-04-23T14:43:12.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.604+0000 7fdb99a00700 1 --2- 172.21.15.28:0/1381541225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb941139a0 0x7fdb9419f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:43:12.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.604+0000 7fdb99000700 1 --2- 172.21.15.28:0/1381541225 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb94077020 0x7fdb941a5650 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdb84004f20 tx=0x7fdb84004f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:12.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.604+0000 7fdb92000700 1 -- 172.21.15.28:0/1381541225 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb84021070 con 0x7fdb94077020 2024-04-23T14:43:12.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.604+0000 7fdb9ba00700 1 -- 172.21.15.28:0/1381541225 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb941a02e0 con 0x7fdb94077020 2024-04-23T14:43:12.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.605+0000 7fdb9ba00700 1 -- 172.21.15.28:0/1381541225 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb941ac5d0 con 0x7fdb94077020 2024-04-23T14:43:12.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.605+0000 7fdb92000700 1 -- 172.21.15.28:0/1381541225 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb84007b80 con 0x7fdb94077020 2024-04-23T14:43:12.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.605+0000 7fdb92000700 1 -- 172.21.15.28:0/1381541225 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb84005040 con 0x7fdb94077020 2024-04-23T14:43:12.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.606+0000 7fdb9ba00700 1 -- 172.21.15.28:0/1381541225 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb94078760 con 0x7fdb94077020 2024-04-23T14:43:12.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.608+0000 7fdb92000700 1 -- 172.21.15.28:0/1381541225 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fdb84024070 con 0x7fdb94077020 2024-04-23T14:43:12.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.609+0000 7fdb92000700 1 --2- 172.21.15.28:0/1381541225 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fdb780a3cc0 0x7fdb780a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:12.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.609+0000 7fdb92000700 1 -- 172.21.15.28:0/1381541225 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7fdb840d1060 con 0x7fdb94077020 2024-04-23T14:43:12.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.609+0000 7fdb8be00700 1 --2- 172.21.15.28:0/1381541225 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fdb780a3cc0 0x7fdb780a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:12.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.610+0000 7fdb8be00700 1 --2- 172.21.15.28:0/1381541225 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fdb780a3cc0 0x7fdb780a6180 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fdb941a0ee0 tx=0x7fdb7c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:12.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.613+0000 7fdb92000700 1 -- 172.21.15.28:0/1381541225 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdb8402b030 con 0x7fdb94077020 2024-04-23T14:43:12.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.898+0000 7fdb9ba00700 1 -- 172.21.15.28:0/1381541225 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fdb941a0c50 con 0x7fdb94077020 2024-04-23T14:43:12.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.899+0000 7fdb92000700 1 -- 172.21.15.28:0/1381541225 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fdb8409da90 con 0x7fdb94077020 2024-04-23T14:43:12.901 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:43:12.901 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-23T14:43:12.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.902+0000 7fdb9ba00700 1 -- 172.21.15.28:0/1381541225 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fdb780a3cc0 msgr2=0x7fdb780a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:12.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.902+0000 7fdb9ba00700 1 --2- 172.21.15.28:0/1381541225 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fdb780a3cc0 0x7fdb780a6180 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fdb941a0ee0 tx=0x7fdb7c009260 comp rx=0 tx=0).stop 2024-04-23T14:43:12.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.902+0000 7fdb9ba00700 1 -- 172.21.15.28:0/1381541225 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb94077020 msgr2=0x7fdb941a5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:12.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.902+0000 7fdb9ba00700 1 --2- 172.21.15.28:0/1381541225 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb94077020 0x7fdb941a5650 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdb84004f20 tx=0x7fdb84004f50 comp rx=0 tx=0).stop 2024-04-23T14:43:12.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.903+0000 7fdb9ba00700 1 -- 172.21.15.28:0/1381541225 shutdown_connections 2024-04-23T14:43:12.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.903+0000 7fdb9ba00700 1 --2- 172.21.15.28:0/1381541225 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fdb780a3cc0 0x7fdb780a6180 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:12.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.903+0000 7fdb9ba00700 1 --2- 172.21.15.28:0/1381541225 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb94077020 0x7fdb941a5650 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:12.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.903+0000 7fdb9ba00700 1 --2- 172.21.15.28:0/1381541225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb941139a0 0x7fdb9419f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:12.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.903+0000 7fdb9ba00700 1 --2- 172.21.15.28:0/1381541225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb94077940 0x7fdb941a5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:12.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.903+0000 7fdb9ba00700 1 -- 172.21.15.28:0/1381541225 >> 172.21.15.28:0/1381541225 conn(0x7fdb941007f0 msgr2=0x7fdb94102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:12.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.903+0000 7fdb9ba00700 1 -- 172.21.15.28:0/1381541225 shutdown_connections 2024-04-23T14:43:12.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:12.903+0000 7fdb9ba00700 1 -- 172.21.15.28:0/1381541225 wait complete. 2024-04-23T14:43:13.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:13 smithi028 ceph-mon[27810]: pgmap v362: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:43:13.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:13 smithi028 ceph-mon[27810]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-23T14:43:13.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:13 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1381541225' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-23T14:43:13.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:13 smithi105 ceph-mon[32555]: pgmap v362: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:43:13.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:13 smithi105 ceph-mon[32555]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-23T14:43:13.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:13 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1381541225' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-23T14:43:13.804 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-23T14:43:13.804 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:43:13.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:13 smithi038 ceph-mon[32348]: pgmap v362: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:43:13.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:13 smithi038 ceph-mon[32348]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-23T14:43:13.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:13 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1381541225' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-23T14:43:13.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.980+0000 7f4c97200640 1 -- 172.21.15.28:0/2188702459 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c9010a030 msgr2=0x7f4c901108c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:13.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.980+0000 7f4c97200640 1 --2- 172.21.15.28:0/2188702459 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c9010a030 0x7f4c901108c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c009be0 tx=0x7f4c8c02a6c0 comp rx=0 tx=0).stop 2024-04-23T14:43:13.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.980+0000 7f4c97200640 1 -- 172.21.15.28:0/2188702459 shutdown_connections 2024-04-23T14:43:13.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.980+0000 7f4c97200640 1 --2- 172.21.15.28:0/2188702459 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c9010a030 0x7f4c901108c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:13.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.980+0000 7f4c97200640 1 --2- 172.21.15.28:0/2188702459 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c90109670 0x7f4c90109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:13.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.980+0000 7f4c97200640 1 --2- 172.21.15.28:0/2188702459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c90108480 0x7f4c901088a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:13.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.980+0000 7f4c97200640 1 -- 172.21.15.28:0/2188702459 >> 172.21.15.28:0/2188702459 conn(0x7f4c90077440 msgr2=0x7f4c90079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:13.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.980+0000 7f4c97200640 1 -- 172.21.15.28:0/2188702459 shutdown_connections 2024-04-23T14:43:13.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.980+0000 7f4c97200640 1 -- 172.21.15.28:0/2188702459 wait complete. 2024-04-23T14:43:13.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.981+0000 7f4c97200640 1 Processor -- start 2024-04-23T14:43:13.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.982+0000 7f4c97200640 1 -- start start 2024-04-23T14:43:13.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.982+0000 7f4c97200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c90108480 0x7f4c9019e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:13.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.982+0000 7f4c97200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c90109670 0x7f4c9019ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:13.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.982+0000 7f4c97200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c9010a030 0x7f4c901a5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:13.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.982+0000 7f4c97200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4c901140e0 con 0x7f4c90109670 2024-04-23T14:43:13.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.982+0000 7f4c97200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4c90113f60 con 0x7f4c90108480 2024-04-23T14:43:13.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.982+0000 7f4c97200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4c90114260 con 0x7f4c9010a030 2024-04-23T14:43:13.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.982+0000 7f4c96800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c9010a030 0x7f4c901a5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:13.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.982+0000 7f4c95400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c90109670 0x7f4c9019ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:13.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.982+0000 7f4c96800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c9010a030 0x7f4c901a5cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:58756/0 (socket says 172.21.15.28:58756) 2024-04-23T14:43:13.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.982+0000 7f4c96800640 1 -- 172.21.15.28:0/2551478093 learned_addr learned my addr 172.21.15.28:0/2551478093 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:13.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.982+0000 7f4c95e00640 1 --2- 172.21.15.28:0/2551478093 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c90108480 0x7f4c9019e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:13.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.983+0000 7f4c96800640 1 -- 172.21.15.28:0/2551478093 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c90108480 msgr2=0x7f4c9019e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:13.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.983+0000 7f4c96800640 1 --2- 172.21.15.28:0/2551478093 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c90108480 0x7f4c9019e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:13.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.983+0000 7f4c96800640 1 -- 172.21.15.28:0/2551478093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c90109670 msgr2=0x7f4c9019ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:13.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.983+0000 7f4c96800640 1 --2- 172.21.15.28:0/2551478093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c90109670 0x7f4c9019ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:13.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.983+0000 7f4c96800640 1 -- 172.21.15.28:0/2551478093 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c901a6340 con 0x7f4c9010a030 2024-04-23T14:43:13.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.983+0000 7f4c96800640 1 --2- 172.21.15.28:0/2551478093 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c9010a030 0x7f4c901a5cd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c009b30 tx=0x7f4c8c0305d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:13.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.983+0000 7f4c8aa00640 1 -- 172.21.15.28:0/2551478093 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c041020 con 0x7f4c9010a030 2024-04-23T14:43:13.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.983+0000 7f4c8aa00640 1 -- 172.21.15.28:0/2551478093 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c8c030740 con 0x7f4c9010a030 2024-04-23T14:43:13.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.983+0000 7f4c97200640 1 -- 172.21.15.28:0/2551478093 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c901a65d0 con 0x7f4c9010a030 2024-04-23T14:43:13.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.983+0000 7f4c8aa00640 1 -- 172.21.15.28:0/2551478093 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c036040 con 0x7f4c9010a030 2024-04-23T14:43:13.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.984+0000 7f4c97200640 1 -- 172.21.15.28:0/2551478093 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c901a6ac0 con 0x7f4c9010a030 2024-04-23T14:43:13.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.985+0000 7f4c97200640 1 -- 172.21.15.28:0/2551478093 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c9010f7a0 con 0x7f4c9010a030 2024-04-23T14:43:13.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.987+0000 7f4c8aa00640 1 -- 172.21.15.28:0/2551478093 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f4c8c044020 con 0x7f4c9010a030 2024-04-23T14:43:13.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.988+0000 7f4c8aa00640 1 --2- 172.21.15.28:0/2551478093 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f4c780a3bb0 0x7f4c780a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:13.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.988+0000 7f4c8aa00640 1 -- 172.21.15.28:0/2551478093 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7f4c8c0f1510 con 0x7f4c9010a030 2024-04-23T14:43:13.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.988+0000 7f4c95e00640 1 --2- 172.21.15.28:0/2551478093 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f4c780a3bb0 0x7f4c780a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:13.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.989+0000 7f4c95e00640 1 --2- 172.21.15.28:0/2551478093 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f4c780a3bb0 0x7f4c780a6070 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4c901094d0 tx=0x7f4c84009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:13.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:13.990+0000 7f4c8aa00640 1 -- 172.21.15.28:0/2551478093 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4c8c04b020 con 0x7f4c9010a030 2024-04-23T14:43:14.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.201+0000 7f4c97200640 1 -- 172.21.15.28:0/2551478093 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4c9002d7f0 con 0x7f4c9010a030 2024-04-23T14:43:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.204+0000 7f4c8aa00640 1 -- 172.21.15.28:0/2551478093 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22651 (secure 0 0 0) 0x7f4c8c0bdeb0 con 0x7f4c9010a030 2024-04-23T14:43:14.205 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:43:14.205 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":87,"fsid":"1ee6e996-017e-11ef-bc93-c7b262605968","created":"2024-04-23T14:32:34.581082+0000","modified":"2024-04-23T14:42:43.429141+0000","last_up_change":"2024-04-23T14:40:43.375846+0000","last_in_change":"2024-04-23T14:40:23.576939+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:36:27.698112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:41:37.030341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:41:38.444742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e0b427bc-be65-4543-99cc-2f53ad98ddad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6803","nonce":2993445336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6805","nonce":2993445336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6809","nonce":2993445336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6807","nonce":2993445336}]},"public_addr":"172.21.15.28:6803/2993445336","cluster_addr":"172.21.15.28:6805/2993445336","heartbeat_back_addr":"172.21.15.28:6809/2993445336","heartbeat_front_addr":"172.21.15.28:6807/2993445336","state":["exists","up"]},{"osd":1,"uuid":"ef527ec9-1bf3-4bb0-8df3-f5a015af557c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6801","nonce":4024093383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6803","nonce":4024093383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6807","nonce":4024093383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6805","nonce":4024093383}]},"public_addr":"172.21.15.38:6801/4024093383","cluster_addr":"172.21.15.38:6803/4024093383","heartbeat_back_addr":"172.21.15.38:6807/4024093383","heartbeat_front_addr":"172.21.15.38:6805/4024093383","state":["exists","up"]},{"osd":2,"uuid":"b33b23d9-adce-4c11-b304-2e117bdf6bfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6801","nonce":570739029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6803","nonce":570739029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6807","nonce":570739029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6805","nonce":570739029}]},"public_addr":"172.21.15.105:6801/570739029","cluster_addr":"172.21.15.105:6803/570739029","heartbeat_back_addr":"172.21.15.105:6807/570739029","heartbeat_front_addr":"172.21.15.105:6805/570739029","state":["exists","up"]},{"osd":3,"uuid":"28bd5608-b426-434e-9db1-bdaa9828bdb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6811","nonce":2480400247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6813","nonce":2480400247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6817","nonce":2480400247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6815","nonce":2480400247}]},"public_addr":"172.21.15.28:6811/2480400247","cluster_addr":"172.21.15.28:6813/2480400247","heartbeat_back_addr":"172.21.15.28:6817/2480400247","heartbeat_front_addr":"172.21.15.28:6815/2480400247","state":["exists","up"]},{"osd":4,"uuid":"e5d1c89b-025f-4d57-8807-59d75c787b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6809","nonce":3772189246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6811","nonce":3772189246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6815","nonce":3772189246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6813","nonce":3772189246}]},"public_addr":"172.21.15.38:6809/3772189246","cluster_addr":"172.21.15.38:6811/3772189246","heartbeat_back_addr":"172.21.15.38:6815/3772189246","heartbeat_front_addr":"172.21.15.38:6813/3772189246","state":["exists","up"]},{"osd":5,"uuid":"9cf9cf4c-aafd-4ed3-bfb1-0527710a0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6809","nonce":928755894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6811","nonce":928755894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6815","nonce":928755894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6813","nonce":928755894}]},"public_addr":"172.21.15.105:6809/928755894","cluster_addr":"172.21.15.105:6811/928755894","heartbeat_back_addr":"172.21.15.105:6815/928755894","heartbeat_front_addr":"172.21.15.105:6813/928755894","state":["exists","up"]},{"osd":6,"uuid":"304db525-d24d-4687-bfa0-f92bdf66d1e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6819","nonce":897979441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6821","nonce":897979441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6825","nonce":897979441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6823","nonce":897979441}]},"public_addr":"172.21.15.28:6819/897979441","cluster_addr":"172.21.15.28:6821/897979441","heartbeat_back_addr":"172.21.15.28:6825/897979441","heartbeat_front_addr":"172.21.15.28:6823/897979441","state":["exists","up"]},{"osd":7,"uuid":"08e01d59-03f3-47b1-b45f-35ec5a50992c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6817","nonce":2400251786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6819","nonce":2400251786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6823","nonce":2400251786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6821","nonce":2400251786}]},"public_addr":"172.21.15.38:6817/2400251786","cluster_addr":"172.21.15.38:6819/2400251786","heartbeat_back_addr":"172.21.15.38:6823/2400251786","heartbeat_front_addr":"172.21.15.38:6821/2400251786","state":["exists","up"]},{"osd":8,"uuid":"64ee1e95-0756-48b1-9c00-472d0c3cbd29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6817","nonce":1682312269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6819","nonce":1682312269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6823","nonce":1682312269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6821","nonce":1682312269}]},"public_addr":"172.21.15.105:6817/1682312269","cluster_addr":"172.21.15.105:6819/1682312269","heartbeat_back_addr":"172.21.15.105:6823/1682312269","heartbeat_front_addr":"172.21.15.105:6821/1682312269","state":["exists","up"]},{"osd":9,"uuid":"18f60893-3ffd-48c2-9db5-a67d35611d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6827","nonce":1292942516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6829","nonce":1292942516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6833","nonce":1292942516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6831","nonce":1292942516}]},"public_addr":"172.21.15.28:6827/1292942516","cluster_addr":"172.21.15.28:6829/1292942516","heartbeat_back_addr":"172.21.15.28:6833/1292942516","heartbeat_front_addr":"172.21.15.28:6831/1292942516","state":["exists","up"]},{"osd":10,"uuid":"76c06725-7040-4118-8894-fe808a87f999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6825","nonce":2444616468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6827","nonce":2444616468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6831","nonce":2444616468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6829","nonce":2444616468}]},"public_addr":"172.21.15.38:6825/2444616468","cluster_addr":"172.21.15.38:6827/2444616468","heartbeat_back_addr":"172.21.15.38:6831/2444616468","heartbeat_front_addr":"172.21.15.38:6829/2444616468","state":["exists","up"]},{"osd":11,"uuid":"e8fc9454-5540-4e2e-91bf-3ba251dbd1a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6825","nonce":3400299562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6827","nonce":3400299562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6831","nonce":3400299562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6829","nonce":3400299562}]},"public_addr":"172.21.15.105:6825/3400299562","cluster_addr":"172.21.15.105:6827/3400299562","heartbeat_back_addr":"172.21.15.105:6831/3400299562","heartbeat_front_addr":"172.21.15.105:6829/3400299562","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:35:33.615395+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:00.545293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:24.940211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:57.684887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:24.519061+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:49.010804+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:22.638478+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:49.559104+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:14.011771+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:47.859950+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:16.298445+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:41.619069+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.2d","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/590495740":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/1163172512":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/4290328460":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/4145020172":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/720626958":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/948450262":"2024-04-24T14:33:07.576363+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:43:14.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.207+0000 7f4c97200640 1 -- 172.21.15.28:0/2551478093 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f4c780a3bb0 msgr2=0x7f4c780a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:14.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.207+0000 7f4c97200640 1 --2- 172.21.15.28:0/2551478093 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f4c780a3bb0 0x7f4c780a6070 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4c901094d0 tx=0x7f4c84009260 comp rx=0 tx=0).stop 2024-04-23T14:43:14.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.207+0000 7f4c97200640 1 -- 172.21.15.28:0/2551478093 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c9010a030 msgr2=0x7f4c901a5cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:14.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.207+0000 7f4c97200640 1 --2- 172.21.15.28:0/2551478093 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c9010a030 0x7f4c901a5cd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c009b30 tx=0x7f4c8c0305d0 comp rx=0 tx=0).stop 2024-04-23T14:43:14.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.207+0000 7f4c97200640 1 -- 172.21.15.28:0/2551478093 shutdown_connections 2024-04-23T14:43:14.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.207+0000 7f4c97200640 1 --2- 172.21.15.28:0/2551478093 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c9010a030 0x7f4c901a5cd0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:14.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.207+0000 7f4c97200640 1 --2- 172.21.15.28:0/2551478093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c90109670 0x7f4c9019ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:14.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.207+0000 7f4c97200640 1 --2- 172.21.15.28:0/2551478093 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f4c780a3bb0 0x7f4c780a6070 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:14.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.207+0000 7f4c97200640 1 --2- 172.21.15.28:0/2551478093 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c90108480 0x7f4c9019e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:14.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.207+0000 7f4c97200640 1 -- 172.21.15.28:0/2551478093 >> 172.21.15.28:0/2551478093 conn(0x7f4c90077440 msgr2=0x7f4c9010c970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:14.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.207+0000 7f4c97200640 1 -- 172.21.15.28:0/2551478093 shutdown_connections 2024-04-23T14:43:14.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.207+0000 7f4c97200640 1 -- 172.21.15.28:0/2551478093 wait complete. 2024-04-23T14:43:14.228 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T14:36:27.698112+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T14:41:37.030341+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T14:41:38.444742+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T14:43:14.228 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T14:43:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.441+0000 7f7948600640 1 -- 172.21.15.28:0/1498868839 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7940103240 msgr2=0x7f79401036c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.441+0000 7f7948600640 1 --2- 172.21.15.28:0/1498868839 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7940103240 0x7f79401036c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f792c009900 tx=0x7f792c02a3f0 comp rx=0 tx=0).stop 2024-04-23T14:43:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.441+0000 7f7948600640 1 -- 172.21.15.28:0/1498868839 shutdown_connections 2024-04-23T14:43:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.441+0000 7f7948600640 1 --2- 172.21.15.28:0/1498868839 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7940103240 0x7f79401036c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.441+0000 7f7948600640 1 --2- 172.21.15.28:0/1498868839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7940103c00 0x7f794010a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.441+0000 7f7948600640 1 --2- 172.21.15.28:0/1498868839 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7940102050 0x7f7940102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.441+0000 7f7948600640 1 -- 172.21.15.28:0/1498868839 >> 172.21.15.28:0/1498868839 conn(0x7f79400fd5e0 msgr2=0x7f79400ffa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.442+0000 7f7948600640 1 -- 172.21.15.28:0/1498868839 shutdown_connections 2024-04-23T14:43:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.442+0000 7f7948600640 1 -- 172.21.15.28:0/1498868839 wait complete. 2024-04-23T14:43:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.443+0000 7f7948600640 1 Processor -- start 2024-04-23T14:43:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.443+0000 7f7948600640 1 -- start start 2024-04-23T14:43:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.443+0000 7f7948600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7940102050 0x7f7940075db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.443+0000 7f7948600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7940103240 0x7f79400743f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.443+0000 7f7948600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7940103c00 0x7f7940074930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.443+0000 7f7948600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f794010dd30 con 0x7f7940102050 2024-04-23T14:43:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.443+0000 7f7948600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f794010dbb0 con 0x7f7940103c00 2024-04-23T14:43:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.443+0000 7f7948600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f794010deb0 con 0x7f7940103240 2024-04-23T14:43:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.444+0000 7f7945200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7940103240 0x7f79400743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:14.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.444+0000 7f7945200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7940103240 0x7f79400743f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:58770/0 (socket says 172.21.15.28:58770) 2024-04-23T14:43:14.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.444+0000 7f7945200640 1 -- 172.21.15.28:0/312402066 learned_addr learned my addr 172.21.15.28:0/312402066 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:14.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.444+0000 7f7945c00640 1 --2- 172.21.15.28:0/312402066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7940102050 0x7f7940075db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:14.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.444+0000 7f7946600640 1 --2- 172.21.15.28:0/312402066 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7940103c00 0x7f7940074930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:14.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.444+0000 7f7945200640 1 -- 172.21.15.28:0/312402066 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7940103c00 msgr2=0x7f7940074930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:14.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.444+0000 7f7945200640 1 --2- 172.21.15.28:0/312402066 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7940103c00 0x7f7940074930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:14.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.444+0000 7f7945200640 1 -- 172.21.15.28:0/312402066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7940102050 msgr2=0x7f7940075db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:14.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.444+0000 7f7945200640 1 --2- 172.21.15.28:0/312402066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7940102050 0x7f7940075db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:14.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.444+0000 7f7945200640 1 -- 172.21.15.28:0/312402066 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79400751c0 con 0x7f7940103240 2024-04-23T14:43:14.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.444+0000 7f7945200640 1 --2- 172.21.15.28:0/312402066 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7940103240 0x7f79400743f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f792c031440 tx=0x7f792c032e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:14.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.446+0000 7f793e000640 1 -- 172.21.15.28:0/312402066 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f792c03f030 con 0x7f7940103240 2024-04-23T14:43:14.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.446+0000 7f793e000640 1 -- 172.21.15.28:0/312402066 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f792c003ce0 con 0x7f7940103240 2024-04-23T14:43:14.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.446+0000 7f793e000640 1 -- 172.21.15.28:0/312402066 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f792c035070 con 0x7f7940103240 2024-04-23T14:43:14.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.446+0000 7f7948600640 1 -- 172.21.15.28:0/312402066 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79401a23f0 con 0x7f7940103240 2024-04-23T14:43:14.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.447+0000 7f7948600640 1 -- 172.21.15.28:0/312402066 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79401a2990 con 0x7f7940103240 2024-04-23T14:43:14.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.448+0000 7f7948600640 1 -- 172.21.15.28:0/312402066 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7940109440 con 0x7f7940103240 2024-04-23T14:43:14.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.450+0000 7f793e000640 1 -- 172.21.15.28:0/312402066 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f792c042030 con 0x7f7940103240 2024-04-23T14:43:14.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.451+0000 7f793e000640 1 --2- 172.21.15.28:0/312402066 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f79240a3c30 0x7f79240a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:14.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.452+0000 7f793e000640 1 -- 172.21.15.28:0/312402066 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7f792c0f09c0 con 0x7f7940103240 2024-04-23T14:43:14.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.452+0000 7f7945c00640 1 --2- 172.21.15.28:0/312402066 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f79240a3c30 0x7f79240a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:14.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.452+0000 7f7945c00640 1 --2- 172.21.15.28:0/312402066 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f79240a3c30 0x7f79240a60f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7930005fd0 tx=0x7f793000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:14.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.456+0000 7f793e000640 1 -- 172.21.15.28:0/312402066 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f792c0bcd10 con 0x7f7940103240 2024-04-23T14:43:14.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:14 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2551478093' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:43:14.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.663+0000 7f7948600640 1 -- 172.21.15.28:0/312402066 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7940062c00 con 0x7f7940103240 2024-04-23T14:43:14.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.663+0000 7f793e000640 1 -- 172.21.15.28:0/312402066 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7f792c0dbe20 con 0x7f7940103240 2024-04-23T14:43:14.665 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-23T14:43:14.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.666+0000 7f7948600640 1 -- 172.21.15.28:0/312402066 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f79240a3c30 msgr2=0x7f79240a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:14.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.666+0000 7f7948600640 1 --2- 172.21.15.28:0/312402066 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f79240a3c30 0x7f79240a60f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7930005fd0 tx=0x7f793000b410 comp rx=0 tx=0).stop 2024-04-23T14:43:14.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.666+0000 7f7948600640 1 -- 172.21.15.28:0/312402066 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7940103240 msgr2=0x7f79400743f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:14.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.666+0000 7f7948600640 1 --2- 172.21.15.28:0/312402066 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7940103240 0x7f79400743f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f792c031440 tx=0x7f792c032e20 comp rx=0 tx=0).stop 2024-04-23T14:43:14.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.666+0000 7f7948600640 1 -- 172.21.15.28:0/312402066 shutdown_connections 2024-04-23T14:43:14.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.666+0000 7f7948600640 1 --2- 172.21.15.28:0/312402066 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f79240a3c30 0x7f79240a60f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:14.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.666+0000 7f7948600640 1 --2- 172.21.15.28:0/312402066 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7940103c00 0x7f7940074930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:14.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.666+0000 7f7948600640 1 --2- 172.21.15.28:0/312402066 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7940103240 0x7f79400743f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:14.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.666+0000 7f7948600640 1 --2- 172.21.15.28:0/312402066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7940102050 0x7f7940075db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:14.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.666+0000 7f7948600640 1 -- 172.21.15.28:0/312402066 >> 172.21.15.28:0/312402066 conn(0x7f79400fd5e0 msgr2=0x7f7940106610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:14.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.666+0000 7f7948600640 1 -- 172.21.15.28:0/312402066 shutdown_connections 2024-04-23T14:43:14.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.667+0000 7f7948600640 1 -- 172.21.15.28:0/312402066 wait complete. 2024-04-23T14:43:14.687 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T14:43:14.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:14 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2551478093' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:43:14.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:14 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2551478093' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:43:14.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.905+0000 7f8241a00640 1 -- 172.21.15.28:0/2173474488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f823c103230 msgr2=0x7f823c1036b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:14.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.905+0000 7f8241a00640 1 --2- 172.21.15.28:0/2173474488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f823c103230 0x7f823c1036b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8228009860 tx=0x7f822802a340 comp rx=0 tx=0).stop 2024-04-23T14:43:14.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.905+0000 7f8241a00640 1 -- 172.21.15.28:0/2173474488 shutdown_connections 2024-04-23T14:43:14.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.905+0000 7f8241a00640 1 --2- 172.21.15.28:0/2173474488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f823c103230 0x7f823c1036b0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:14.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.905+0000 7f8241a00640 1 --2- 172.21.15.28:0/2173474488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f823c103bf0 0x7f823c10a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:14.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.905+0000 7f8241a00640 1 --2- 172.21.15.28:0/2173474488 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f823c102040 0x7f823c102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:14.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.905+0000 7f8241a00640 1 -- 172.21.15.28:0/2173474488 >> 172.21.15.28:0/2173474488 conn(0x7f823c0fd5b0 msgr2=0x7f823c0ff9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:14.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.906+0000 7f8241a00640 1 -- 172.21.15.28:0/2173474488 shutdown_connections 2024-04-23T14:43:14.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.906+0000 7f8241a00640 1 -- 172.21.15.28:0/2173474488 wait complete. 2024-04-23T14:43:14.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.907+0000 7f8241a00640 1 Processor -- start 2024-04-23T14:43:14.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.907+0000 7f8241a00640 1 -- start start 2024-04-23T14:43:14.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.907+0000 7f8241a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f823c102040 0x7f823c19a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:14.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.907+0000 7f8241a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f823c103230 0x7f823c19aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:14.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.907+0000 7f8241a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f823c103bf0 0x7f823c1a1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:14.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.907+0000 7f8241a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f823c10dcd0 con 0x7f823c103230 2024-04-23T14:43:14.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.907+0000 7f8241a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f823c10db50 con 0x7f823c102040 2024-04-23T14:43:14.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.907+0000 7f8241a00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f823c10de50 con 0x7f823c103bf0 2024-04-23T14:43:14.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.907+0000 7f823a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f823c103230 0x7f823c19aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:14.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.907+0000 7f823aa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f823c102040 0x7f823c19a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:14.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.907+0000 7f823a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f823c103230 0x7f823c19aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38896/0 (socket says 172.21.15.28:38896) 2024-04-23T14:43:14.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.907+0000 7f823a000640 1 -- 172.21.15.28:0/2645424246 learned_addr learned my addr 172.21.15.28:0/2645424246 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:14.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.908+0000 7f823b400640 1 --2- 172.21.15.28:0/2645424246 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f823c103bf0 0x7f823c1a1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:14.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.908+0000 7f823aa00640 1 -- 172.21.15.28:0/2645424246 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f823c103bf0 msgr2=0x7f823c1a1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:14.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.908+0000 7f823aa00640 1 --2- 172.21.15.28:0/2645424246 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f823c103bf0 0x7f823c1a1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:14.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.908+0000 7f823aa00640 1 -- 172.21.15.28:0/2645424246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f823c103230 msgr2=0x7f823c19aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:14.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.908+0000 7f823aa00640 1 --2- 172.21.15.28:0/2645424246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f823c103230 0x7f823c19aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:14.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.908+0000 7f823aa00640 1 -- 172.21.15.28:0/2645424246 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f823c1a2190 con 0x7f823c102040 2024-04-23T14:43:14.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.908+0000 7f823a000640 1 --2- 172.21.15.28:0/2645424246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f823c103230 0x7f823c19aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:43:14.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.908+0000 7f823aa00640 1 --2- 172.21.15.28:0/2645424246 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f823c102040 0x7f823c19a570 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f822c00e9c0 tx=0x7f822c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:14.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.908+0000 7f8233400640 1 -- 172.21.15.28:0/2645424246 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f822c00ccc0 con 0x7f823c102040 2024-04-23T14:43:14.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.908+0000 7f8233400640 1 -- 172.21.15.28:0/2645424246 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f822c0035f0 con 0x7f823c102040 2024-04-23T14:43:14.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.908+0000 7f8241a00640 1 -- 172.21.15.28:0/2645424246 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f823c1a2480 con 0x7f823c102040 2024-04-23T14:43:14.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.909+0000 7f8233400640 1 -- 172.21.15.28:0/2645424246 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f822c00ce60 con 0x7f823c102040 2024-04-23T14:43:14.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.909+0000 7f8241a00640 1 -- 172.21.15.28:0/2645424246 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f823c1a29d0 con 0x7f823c102040 2024-04-23T14:43:14.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.910+0000 7f8241a00640 1 -- 172.21.15.28:0/2645424246 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f823c1a2c90 con 0x7f823c102040 2024-04-23T14:43:14.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.912+0000 7f8233400640 1 -- 172.21.15.28:0/2645424246 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f822c0037a0 con 0x7f823c102040 2024-04-23T14:43:14.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.913+0000 7f8233400640 1 --2- 172.21.15.28:0/2645424246 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f82200a3bb0 0x7f82200a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:14.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.913+0000 7f8233400640 1 -- 172.21.15.28:0/2645424246 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7f822c0d0780 con 0x7f823c102040 2024-04-23T14:43:14.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.914+0000 7f823a000640 1 --2- 172.21.15.28:0/2645424246 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f82200a3bb0 0x7f82200a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:14.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.915+0000 7f823a000640 1 --2- 172.21.15.28:0/2645424246 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f82200a3bb0 0x7f82200a6070 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f823c19ba80 tx=0x7f8228030550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:14.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:14.918+0000 7f8233400640 1 -- 172.21.15.28:0/2645424246 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f822c09d120 con 0x7f823c102040 2024-04-23T14:43:15.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.132+0000 7f8241a00640 1 -- 172.21.15.28:0/2645424246 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f823c1a2e60 con 0x7f823c102040 2024-04-23T14:43:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.133+0000 7f8233400640 1 -- 172.21.15.28:0/2645424246 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f822c09ca20 con 0x7f823c102040 2024-04-23T14:43:15.134 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-23T14:43:15.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.136+0000 7f8241a00640 1 -- 172.21.15.28:0/2645424246 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f82200a3bb0 msgr2=0x7f82200a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:15.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.136+0000 7f8241a00640 1 --2- 172.21.15.28:0/2645424246 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f82200a3bb0 0x7f82200a6070 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f823c19ba80 tx=0x7f8228030550 comp rx=0 tx=0).stop 2024-04-23T14:43:15.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.136+0000 7f8241a00640 1 -- 172.21.15.28:0/2645424246 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f823c102040 msgr2=0x7f823c19a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:15.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.136+0000 7f8241a00640 1 --2- 172.21.15.28:0/2645424246 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f823c102040 0x7f823c19a570 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f822c00e9c0 tx=0x7f822c00ee80 comp rx=0 tx=0).stop 2024-04-23T14:43:15.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.136+0000 7f8241a00640 1 -- 172.21.15.28:0/2645424246 shutdown_connections 2024-04-23T14:43:15.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.136+0000 7f8241a00640 1 --2- 172.21.15.28:0/2645424246 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f823c103bf0 0x7f823c1a1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:15.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.136+0000 7f8241a00640 1 --2- 172.21.15.28:0/2645424246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f823c103230 0x7f823c19aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:15.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.136+0000 7f8241a00640 1 --2- 172.21.15.28:0/2645424246 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f82200a3bb0 0x7f82200a6070 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:15.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.136+0000 7f8241a00640 1 --2- 172.21.15.28:0/2645424246 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f823c102040 0x7f823c19a570 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:15.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.136+0000 7f8241a00640 1 -- 172.21.15.28:0/2645424246 >> 172.21.15.28:0/2645424246 conn(0x7f823c0fd5b0 msgr2=0x7f823c074230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:15.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.136+0000 7f8241a00640 1 -- 172.21.15.28:0/2645424246 shutdown_connections 2024-04-23T14:43:15.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.136+0000 7f8241a00640 1 -- 172.21.15.28:0/2645424246 wait complete. 2024-04-23T14:43:15.155 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T14:43:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.371+0000 7f26b3e00640 1 -- 172.21.15.28:0/808472834 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26b41052d0 msgr2=0x7f26b41076c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.371+0000 7f26b3e00640 1 --2- 172.21.15.28:0/808472834 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26b41052d0 0x7f26b41076c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f26a0009900 tx=0x7f26a002a410 comp rx=0 tx=0).stop 2024-04-23T14:43:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.371+0000 7f26b3e00640 1 -- 172.21.15.28:0/808472834 shutdown_connections 2024-04-23T14:43:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.371+0000 7f26b3e00640 1 --2- 172.21.15.28:0/808472834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26b4107c00 0x7f26b410a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.371+0000 7f26b3e00640 1 --2- 172.21.15.28:0/808472834 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26b41052d0 0x7f26b41076c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.371+0000 7f26b3e00640 1 --2- 172.21.15.28:0/808472834 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26b41029a0 0x7f26b4104d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.371+0000 7f26b3e00640 1 -- 172.21.15.28:0/808472834 >> 172.21.15.28:0/808472834 conn(0x7f26b40fc370 msgr2=0x7f26b40fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:15.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.371+0000 7f26b3e00640 1 -- 172.21.15.28:0/808472834 shutdown_connections 2024-04-23T14:43:15.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.371+0000 7f26b3e00640 1 -- 172.21.15.28:0/808472834 wait complete. 2024-04-23T14:43:15.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.372+0000 7f26b3e00640 1 Processor -- start 2024-04-23T14:43:15.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.373+0000 7f26b3e00640 1 -- start start 2024-04-23T14:43:15.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.373+0000 7f26b3e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26b41029a0 0x7f26b4198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:15.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.373+0000 7f26b3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26b41052d0 0x7f26b4198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:15.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.373+0000 7f26b3e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26b4107c00 0x7f26b419f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:15.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.373+0000 7f26b3e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f26b410dbd0 con 0x7f26b41052d0 2024-04-23T14:43:15.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.373+0000 7f26b3e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f26b410da50 con 0x7f26b41029a0 2024-04-23T14:43:15.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.373+0000 7f26b3e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f26b410dd50 con 0x7f26b4107c00 2024-04-23T14:43:15.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.373+0000 7f26b2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26b41052d0 0x7f26b4198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:15.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.373+0000 7f26b2a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26b41029a0 0x7f26b4198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:15.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.373+0000 7f26b2a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26b41029a0 0x7f26b4198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:42502/0 (socket says 172.21.15.28:42502) 2024-04-23T14:43:15.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.373+0000 7f26b2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26b41052d0 0x7f26b4198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38904/0 (socket says 172.21.15.28:38904) 2024-04-23T14:43:15.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.373+0000 7f26b2a00640 1 -- 172.21.15.28:0/3622525060 learned_addr learned my addr 172.21.15.28:0/3622525060 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:15.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.374+0000 7f26b3400640 1 --2- 172.21.15.28:0/3622525060 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26b4107c00 0x7f26b419f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:15.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.374+0000 7f26b2a00640 1 -- 172.21.15.28:0/3622525060 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26b4107c00 msgr2=0x7f26b419f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:15.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.374+0000 7f26b2a00640 1 --2- 172.21.15.28:0/3622525060 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26b4107c00 0x7f26b419f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:15.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.374+0000 7f26b2a00640 1 -- 172.21.15.28:0/3622525060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26b41052d0 msgr2=0x7f26b4198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:15.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.374+0000 7f26b2a00640 1 --2- 172.21.15.28:0/3622525060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26b41052d0 0x7f26b4198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:15.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.374+0000 7f26b2a00640 1 -- 172.21.15.28:0/3622525060 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26b419fe40 con 0x7f26b41029a0 2024-04-23T14:43:15.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.374+0000 7f26b2a00640 1 --2- 172.21.15.28:0/3622525060 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26b41029a0 0x7f26b4198220 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f26a400e9f0 tx=0x7f26a400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:15.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.374+0000 7f26ab400640 1 -- 172.21.15.28:0/3622525060 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a400cd80 con 0x7f26b41029a0 2024-04-23T14:43:15.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.374+0000 7f26ab400640 1 -- 172.21.15.28:0/3622525060 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26a4010910 con 0x7f26b41029a0 2024-04-23T14:43:15.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.375+0000 7f26b3e00640 1 -- 172.21.15.28:0/3622525060 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26b41a0130 con 0x7f26b41029a0 2024-04-23T14:43:15.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.375+0000 7f26ab400640 1 -- 172.21.15.28:0/3622525060 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a40188f0 con 0x7f26b41029a0 2024-04-23T14:43:15.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.375+0000 7f26b3e00640 1 -- 172.21.15.28:0/3622525060 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26b41a0650 con 0x7f26b41029a0 2024-04-23T14:43:15.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.376+0000 7f26b3e00640 1 -- 172.21.15.28:0/3622525060 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26b40fde60 con 0x7f26b41029a0 2024-04-23T14:43:15.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.378+0000 7f26ab400640 1 -- 172.21.15.28:0/3622525060 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f26a4010ac0 con 0x7f26b41029a0 2024-04-23T14:43:15.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.379+0000 7f26ab400640 1 --2- 172.21.15.28:0/3622525060 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f26980a3bb0 0x7f26980a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:15.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.379+0000 7f26ab400640 1 -- 172.21.15.28:0/3622525060 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7f26a40d0960 con 0x7f26b41029a0 2024-04-23T14:43:15.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.380+0000 7f26b2000640 1 --2- 172.21.15.28:0/3622525060 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f26980a3bb0 0x7f26980a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:15.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.380+0000 7f26b2000640 1 --2- 172.21.15.28:0/3622525060 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f26980a3bb0 0x7f26980a6070 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f26a002aa00 tx=0x7f26a00315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:15.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.382+0000 7f26ab400640 1 -- 172.21.15.28:0/3622525060 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f26a409ccb0 con 0x7f26b41029a0 2024-04-23T14:43:15.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.588+0000 7f26b3e00640 1 -- 172.21.15.28:0/3622525060 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f26b4062c00 con 0x7f26b41029a0 2024-04-23T14:43:15.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.589+0000 7f26ab400640 1 -- 172.21.15.28:0/3622525060 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7f26a40a0fb0 con 0x7f26b41029a0 2024-04-23T14:43:15.591 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-23T14:43:15.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.592+0000 7f26b3e00640 1 -- 172.21.15.28:0/3622525060 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f26980a3bb0 msgr2=0x7f26980a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:15.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.592+0000 7f26b3e00640 1 --2- 172.21.15.28:0/3622525060 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f26980a3bb0 0x7f26980a6070 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f26a002aa00 tx=0x7f26a00315b0 comp rx=0 tx=0).stop 2024-04-23T14:43:15.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.592+0000 7f26b3e00640 1 -- 172.21.15.28:0/3622525060 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26b41029a0 msgr2=0x7f26b4198220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:15.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.592+0000 7f26b3e00640 1 --2- 172.21.15.28:0/3622525060 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26b41029a0 0x7f26b4198220 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f26a400e9f0 tx=0x7f26a400eeb0 comp rx=0 tx=0).stop 2024-04-23T14:43:15.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.592+0000 7f26b3e00640 1 -- 172.21.15.28:0/3622525060 shutdown_connections 2024-04-23T14:43:15.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.592+0000 7f26b3e00640 1 --2- 172.21.15.28:0/3622525060 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26b4107c00 0x7f26b419f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:15.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.592+0000 7f26b3e00640 1 --2- 172.21.15.28:0/3622525060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26b41052d0 0x7f26b4198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:15.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.592+0000 7f26b3e00640 1 --2- 172.21.15.28:0/3622525060 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f26980a3bb0 0x7f26980a6070 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:15.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.592+0000 7f26b3e00640 1 --2- 172.21.15.28:0/3622525060 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26b41029a0 0x7f26b4198220 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:15.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.592+0000 7f26b3e00640 1 -- 172.21.15.28:0/3622525060 >> 172.21.15.28:0/3622525060 conn(0x7f26b40fc370 msgr2=0x7f26b41033c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:15.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.592+0000 7f26b3e00640 1 -- 172.21.15.28:0/3622525060 shutdown_connections 2024-04-23T14:43:15.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.592+0000 7f26b3e00640 1 -- 172.21.15.28:0/3622525060 wait complete. 2024-04-23T14:43:15.614 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:43:15.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:15 smithi105 ceph-mon[32555]: pgmap v363: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:43:15.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:15 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/312402066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:43:15.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:15 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2645424246' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T14:43:15.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.831+0000 7ff017e00640 1 -- 172.21.15.28:0/1045547170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff018108480 msgr2=0x7ff0181088a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:15.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.831+0000 7ff017e00640 1 --2- 172.21.15.28:0/1045547170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff018108480 0x7ff0181088a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff008009900 tx=0x7ff00802a410 comp rx=0 tx=0).stop 2024-04-23T14:43:15.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.832+0000 7ff017e00640 1 -- 172.21.15.28:0/1045547170 shutdown_connections 2024-04-23T14:43:15.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.832+0000 7ff017e00640 1 --2- 172.21.15.28:0/1045547170 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff018109670 0x7ff018109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:15.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.832+0000 7ff017e00640 1 --2- 172.21.15.28:0/1045547170 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff01810a030 0x7ff0181108c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:15.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.832+0000 7ff017e00640 1 --2- 172.21.15.28:0/1045547170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff018108480 0x7ff0181088a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:15.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.832+0000 7ff017e00640 1 -- 172.21.15.28:0/1045547170 >> 172.21.15.28:0/1045547170 conn(0x7ff018077440 msgr2=0x7ff018079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:15.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.832+0000 7ff017e00640 1 -- 172.21.15.28:0/1045547170 shutdown_connections 2024-04-23T14:43:15.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.832+0000 7ff017e00640 1 -- 172.21.15.28:0/1045547170 wait complete. 2024-04-23T14:43:15.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.833+0000 7ff017e00640 1 Processor -- start 2024-04-23T14:43:15.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.833+0000 7ff017e00640 1 -- start start 2024-04-23T14:43:15.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.833+0000 7ff017e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff018108480 0x7ff01819e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:15.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.833+0000 7ff017e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff018109670 0x7ff01819eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:15.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.833+0000 7ff017e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff01810a030 0x7ff0181a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:15.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.833+0000 7ff017e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff018114110 con 0x7ff018108480 2024-04-23T14:43:15.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.833+0000 7ff017e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff018113f90 con 0x7ff018109670 2024-04-23T14:43:15.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.833+0000 7ff017e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff018114290 con 0x7ff01810a030 2024-04-23T14:43:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.833+0000 7ff016a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff018108480 0x7ff01819e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.833+0000 7ff016a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff018108480 0x7ff01819e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38928/0 (socket says 172.21.15.28:38928) 2024-04-23T14:43:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.833+0000 7ff016a00640 1 -- 172.21.15.28:0/1603621679 learned_addr learned my addr 172.21.15.28:0/1603621679 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.833+0000 7ff016000640 1 --2- 172.21.15.28:0/1603621679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff018109670 0x7ff01819eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.834+0000 7ff017400640 1 --2- 172.21.15.28:0/1603621679 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff01810a030 0x7ff0181a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.834+0000 7ff016a00640 1 -- 172.21.15.28:0/1603621679 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff01810a030 msgr2=0x7ff0181a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.834+0000 7ff016a00640 1 --2- 172.21.15.28:0/1603621679 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff01810a030 0x7ff0181a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.834+0000 7ff016a00640 1 -- 172.21.15.28:0/1603621679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff018109670 msgr2=0x7ff01819eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.834+0000 7ff016a00640 1 --2- 172.21.15.28:0/1603621679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff018109670 0x7ff01819eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.834+0000 7ff016a00640 1 -- 172.21.15.28:0/1603621679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0181a6380 con 0x7ff018108480 2024-04-23T14:43:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.834+0000 7ff016000640 1 --2- 172.21.15.28:0/1603621679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff018109670 0x7ff01819eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:43:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.834+0000 7ff016a00640 1 --2- 172.21.15.28:0/1603621679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff018108480 0x7ff01819e760 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff008031610 tx=0x7ff008004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.835+0000 7ff01ca00640 1 -- 172.21.15.28:0/1603621679 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff00803f030 con 0x7ff018108480 2024-04-23T14:43:15.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.835+0000 7ff01ca00640 1 -- 172.21.15.28:0/1603621679 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff008003ad0 con 0x7ff018108480 2024-04-23T14:43:15.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.835+0000 7ff01ca00640 1 -- 172.21.15.28:0/1603621679 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff008035070 con 0x7ff018108480 2024-04-23T14:43:15.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.835+0000 7ff017e00640 1 -- 172.21.15.28:0/1603621679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0181a6610 con 0x7ff018108480 2024-04-23T14:43:15.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.835+0000 7ff017e00640 1 -- 172.21.15.28:0/1603621679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0181a6ab0 con 0x7ff018108480 2024-04-23T14:43:15.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.836+0000 7ff017e00640 1 -- 172.21.15.28:0/1603621679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff01810f820 con 0x7ff018108480 2024-04-23T14:43:15.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.838+0000 7ff01ca00640 1 -- 172.21.15.28:0/1603621679 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7ff008042030 con 0x7ff018108480 2024-04-23T14:43:15.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.839+0000 7ff01ca00640 1 --2- 172.21.15.28:0/1603621679 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7feffc0a3c30 0x7feffc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:15.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.840+0000 7ff01ca00640 1 -- 172.21.15.28:0/1603621679 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7ff0080f0910 con 0x7ff018108480 2024-04-23T14:43:15.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.840+0000 7ff016000640 1 --2- 172.21.15.28:0/1603621679 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7feffc0a3c30 0x7feffc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:15.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.840+0000 7ff016000640 1 --2- 172.21.15.28:0/1603621679 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7feffc0a3c30 0x7feffc0a60f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff0000095f0 tx=0x7ff000005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:15.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:15.844+0000 7ff01ca00640 1 -- 172.21.15.28:0/1603621679 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff0080bcc60 con 0x7ff018108480 2024-04-23T14:43:15.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:15 smithi038 ceph-mon[32348]: pgmap v363: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:43:15.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:15 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/312402066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:43:15.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:15 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2645424246' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T14:43:15.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:15 smithi028 ceph-mon[27810]: pgmap v363: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:43:15.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:15 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/312402066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:43:15.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:15 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2645424246' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T14:43:16.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.100+0000 7ff017e00640 1 -- 172.21.15.28:0/1603621679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff01802d620 con 0x7ff018108480 2024-04-23T14:43:16.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.102+0000 7ff01ca00640 1 -- 172.21.15.28:0/1603621679 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7ff0080c0f60 con 0x7ff018108480 2024-04-23T14:43:16.103 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:43:16.104 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:43:16.104 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:43:16.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.105+0000 7ff017e00640 1 -- 172.21.15.28:0/1603621679 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7feffc0a3c30 msgr2=0x7feffc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:16.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.105+0000 7ff017e00640 1 --2- 172.21.15.28:0/1603621679 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7feffc0a3c30 0x7feffc0a60f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff0000095f0 tx=0x7ff000005f90 comp rx=0 tx=0).stop 2024-04-23T14:43:16.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.105+0000 7ff017e00640 1 -- 172.21.15.28:0/1603621679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff018108480 msgr2=0x7ff01819e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:16.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.105+0000 7ff017e00640 1 --2- 172.21.15.28:0/1603621679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff018108480 0x7ff01819e760 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff008031610 tx=0x7ff008004270 comp rx=0 tx=0).stop 2024-04-23T14:43:16.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.105+0000 7ff017e00640 1 -- 172.21.15.28:0/1603621679 shutdown_connections 2024-04-23T14:43:16.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.105+0000 7ff017e00640 1 --2- 172.21.15.28:0/1603621679 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7feffc0a3c30 0x7feffc0a60f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:16.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.105+0000 7ff017e00640 1 --2- 172.21.15.28:0/1603621679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff018109670 0x7ff01819eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:16.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.105+0000 7ff017e00640 1 --2- 172.21.15.28:0/1603621679 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff01810a030 0x7ff0181a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:16.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.105+0000 7ff017e00640 1 --2- 172.21.15.28:0/1603621679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff018108480 0x7ff01819e760 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:16.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.105+0000 7ff017e00640 1 -- 172.21.15.28:0/1603621679 >> 172.21.15.28:0/1603621679 conn(0x7ff018077440 msgr2=0x7ff01810c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:16.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.105+0000 7ff017e00640 1 -- 172.21.15.28:0/1603621679 shutdown_connections 2024-04-23T14:43:16.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.105+0000 7ff017e00640 1 -- 172.21.15.28:0/1603621679 wait complete. 2024-04-23T14:43:16.128 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:43:16.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.331+0000 7f98cde00640 1 -- 172.21.15.28:0/930200539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98c8108480 msgr2=0x7f98c81088a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:16.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.331+0000 7f98cde00640 1 --2- 172.21.15.28:0/930200539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98c8108480 0x7f98c81088a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f98b8009900 tx=0x7f98b802a410 comp rx=0 tx=0).stop 2024-04-23T14:43:16.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.331+0000 7f98cde00640 1 -- 172.21.15.28:0/930200539 shutdown_connections 2024-04-23T14:43:16.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.331+0000 7f98cde00640 1 --2- 172.21.15.28:0/930200539 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98c810a030 0x7f98c81108c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:16.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.331+0000 7f98cde00640 1 --2- 172.21.15.28:0/930200539 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f98c8109670 0x7f98c8109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:16.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.331+0000 7f98cde00640 1 --2- 172.21.15.28:0/930200539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98c8108480 0x7f98c81088a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:16.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.331+0000 7f98cde00640 1 -- 172.21.15.28:0/930200539 >> 172.21.15.28:0/930200539 conn(0x7f98c8077440 msgr2=0x7f98c8079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:16.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.331+0000 7f98cde00640 1 -- 172.21.15.28:0/930200539 shutdown_connections 2024-04-23T14:43:16.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.331+0000 7f98cde00640 1 -- 172.21.15.28:0/930200539 wait complete. 2024-04-23T14:43:16.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.332+0000 7f98cde00640 1 Processor -- start 2024-04-23T14:43:16.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.333+0000 7f98cde00640 1 -- start start 2024-04-23T14:43:16.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.333+0000 7f98cde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98c8108480 0x7f98c819e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.333+0000 7f98cde00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f98c8109670 0x7f98c819ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.333+0000 7f98cde00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98c810a030 0x7f98c81a5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.333+0000 7f98cde00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f98c8114130 con 0x7f98c8108480 2024-04-23T14:43:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.333+0000 7f98cde00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f98c8113fb0 con 0x7f98c810a030 2024-04-23T14:43:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.333+0000 7f98cde00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f98c81142b0 con 0x7f98c8109670 2024-04-23T14:43:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.333+0000 7f98cca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98c8108480 0x7f98c819e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.333+0000 7f98cca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98c8108480 0x7f98c819e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38940/0 (socket says 172.21.15.28:38940) 2024-04-23T14:43:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.333+0000 7f98cca00640 1 -- 172.21.15.28:0/3733410183 learned_addr learned my addr 172.21.15.28:0/3733410183 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.333+0000 7f98c7e00640 1 --2- 172.21.15.28:0/3733410183 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f98c8109670 0x7f98c819ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.333+0000 7f98cd400640 1 --2- 172.21.15.28:0/3733410183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98c810a030 0x7f98c81a5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.334+0000 7f98cca00640 1 -- 172.21.15.28:0/3733410183 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f98c8109670 msgr2=0x7f98c819ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.334+0000 7f98cca00640 1 --2- 172.21.15.28:0/3733410183 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f98c8109670 0x7f98c819ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.334+0000 7f98cca00640 1 -- 172.21.15.28:0/3733410183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98c810a030 msgr2=0x7f98c81a5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.334+0000 7f98cca00640 1 --2- 172.21.15.28:0/3733410183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98c810a030 0x7f98c81a5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:16.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.334+0000 7f98cca00640 1 -- 172.21.15.28:0/3733410183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98c81a6390 con 0x7f98c8108480 2024-04-23T14:43:16.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.334+0000 7f98cca00640 1 --2- 172.21.15.28:0/3733410183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98c8108480 0x7f98c819e770 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f98b8031440 tx=0x7f98b8032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:16.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.334+0000 7f98c5600640 1 -- 172.21.15.28:0/3733410183 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b803f030 con 0x7f98c8108480 2024-04-23T14:43:16.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.334+0000 7f98c5600640 1 -- 172.21.15.28:0/3733410183 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98b8003ce0 con 0x7f98c8108480 2024-04-23T14:43:16.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.334+0000 7f98cde00640 1 -- 172.21.15.28:0/3733410183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98c81a6620 con 0x7f98c8108480 2024-04-23T14:43:16.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.334+0000 7f98c5600640 1 -- 172.21.15.28:0/3733410183 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b8035070 con 0x7f98c8108480 2024-04-23T14:43:16.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.335+0000 7f98cde00640 1 -- 172.21.15.28:0/3733410183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98c81a6b60 con 0x7f98c8108480 2024-04-23T14:43:16.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.336+0000 7f98cde00640 1 -- 172.21.15.28:0/3733410183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98c810f7a0 con 0x7f98c8108480 2024-04-23T14:43:16.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.338+0000 7f98c5600640 1 -- 172.21.15.28:0/3733410183 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f98b8042030 con 0x7f98c8108480 2024-04-23T14:43:16.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.339+0000 7f98c5600640 1 --2- 172.21.15.28:0/3733410183 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f98ac0a3c30 0x7f98ac0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:16.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.340+0000 7f98c5600640 1 -- 172.21.15.28:0/3733410183 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7f98b80f0c30 con 0x7f98c8108480 2024-04-23T14:43:16.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.340+0000 7f98c7e00640 1 --2- 172.21.15.28:0/3733410183 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f98ac0a3c30 0x7f98ac0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:16.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.340+0000 7f98c7e00640 1 --2- 172.21.15.28:0/3733410183 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f98ac0a3c30 0x7f98ac0a60f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f98c819fc80 tx=0x7f98b4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:16.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.342+0000 7f98c5600640 1 -- 172.21.15.28:0/3733410183 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f98b80bcf80 con 0x7f98c8108480 2024-04-23T14:43:16.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.555+0000 7f98cde00640 1 -- 172.21.15.28:0/3733410183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f98c802d7f0 con 0x7f98c8108480 2024-04-23T14:43:16.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.558+0000 7f98c5600640 1 -- 172.21.15.28:0/3733410183 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22651 (secure 0 0 0) 0x7f98b80c1280 con 0x7f98c8108480 2024-04-23T14:43:16.560 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:43:16.560 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":87,"fsid":"1ee6e996-017e-11ef-bc93-c7b262605968","created":"2024-04-23T14:32:34.581082+0000","modified":"2024-04-23T14:42:43.429141+0000","last_up_change":"2024-04-23T14:40:43.375846+0000","last_in_change":"2024-04-23T14:40:23.576939+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:36:27.698112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:41:37.030341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:41:38.444742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e0b427bc-be65-4543-99cc-2f53ad98ddad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6803","nonce":2993445336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6805","nonce":2993445336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6809","nonce":2993445336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6807","nonce":2993445336}]},"public_addr":"172.21.15.28:6803/2993445336","cluster_addr":"172.21.15.28:6805/2993445336","heartbeat_back_addr":"172.21.15.28:6809/2993445336","heartbeat_front_addr":"172.21.15.28:6807/2993445336","state":["exists","up"]},{"osd":1,"uuid":"ef527ec9-1bf3-4bb0-8df3-f5a015af557c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6801","nonce":4024093383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6803","nonce":4024093383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6807","nonce":4024093383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6805","nonce":4024093383}]},"public_addr":"172.21.15.38:6801/4024093383","cluster_addr":"172.21.15.38:6803/4024093383","heartbeat_back_addr":"172.21.15.38:6807/4024093383","heartbeat_front_addr":"172.21.15.38:6805/4024093383","state":["exists","up"]},{"osd":2,"uuid":"b33b23d9-adce-4c11-b304-2e117bdf6bfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6801","nonce":570739029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6803","nonce":570739029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6807","nonce":570739029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6805","nonce":570739029}]},"public_addr":"172.21.15.105:6801/570739029","cluster_addr":"172.21.15.105:6803/570739029","heartbeat_back_addr":"172.21.15.105:6807/570739029","heartbeat_front_addr":"172.21.15.105:6805/570739029","state":["exists","up"]},{"osd":3,"uuid":"28bd5608-b426-434e-9db1-bdaa9828bdb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6811","nonce":2480400247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6813","nonce":2480400247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6817","nonce":2480400247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6815","nonce":2480400247}]},"public_addr":"172.21.15.28:6811/2480400247","cluster_addr":"172.21.15.28:6813/2480400247","heartbeat_back_addr":"172.21.15.28:6817/2480400247","heartbeat_front_addr":"172.21.15.28:6815/2480400247","state":["exists","up"]},{"osd":4,"uuid":"e5d1c89b-025f-4d57-8807-59d75c787b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6809","nonce":3772189246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6811","nonce":3772189246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6815","nonce":3772189246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6813","nonce":3772189246}]},"public_addr":"172.21.15.38:6809/3772189246","cluster_addr":"172.21.15.38:6811/3772189246","heartbeat_back_addr":"172.21.15.38:6815/3772189246","heartbeat_front_addr":"172.21.15.38:6813/3772189246","state":["exists","up"]},{"osd":5,"uuid":"9cf9cf4c-aafd-4ed3-bfb1-0527710a0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6809","nonce":928755894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6811","nonce":928755894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6815","nonce":928755894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6813","nonce":928755894}]},"public_addr":"172.21.15.105:6809/928755894","cluster_addr":"172.21.15.105:6811/928755894","heartbeat_back_addr":"172.21.15.105:6815/928755894","heartbeat_front_addr":"172.21.15.105:6813/928755894","state":["exists","up"]},{"osd":6,"uuid":"304db525-d24d-4687-bfa0-f92bdf66d1e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6819","nonce":897979441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6821","nonce":897979441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6825","nonce":897979441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6823","nonce":897979441}]},"public_addr":"172.21.15.28:6819/897979441","cluster_addr":"172.21.15.28:6821/897979441","heartbeat_back_addr":"172.21.15.28:6825/897979441","heartbeat_front_addr":"172.21.15.28:6823/897979441","state":["exists","up"]},{"osd":7,"uuid":"08e01d59-03f3-47b1-b45f-35ec5a50992c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6817","nonce":2400251786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6819","nonce":2400251786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6823","nonce":2400251786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6821","nonce":2400251786}]},"public_addr":"172.21.15.38:6817/2400251786","cluster_addr":"172.21.15.38:6819/2400251786","heartbeat_back_addr":"172.21.15.38:6823/2400251786","heartbeat_front_addr":"172.21.15.38:6821/2400251786","state":["exists","up"]},{"osd":8,"uuid":"64ee1e95-0756-48b1-9c00-472d0c3cbd29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6817","nonce":1682312269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6819","nonce":1682312269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6823","nonce":1682312269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6821","nonce":1682312269}]},"public_addr":"172.21.15.105:6817/1682312269","cluster_addr":"172.21.15.105:6819/1682312269","heartbeat_back_addr":"172.21.15.105:6823/1682312269","heartbeat_front_addr":"172.21.15.105:6821/1682312269","state":["exists","up"]},{"osd":9,"uuid":"18f60893-3ffd-48c2-9db5-a67d35611d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6827","nonce":1292942516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6829","nonce":1292942516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6833","nonce":1292942516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6831","nonce":1292942516}]},"public_addr":"172.21.15.28:6827/1292942516","cluster_addr":"172.21.15.28:6829/1292942516","heartbeat_back_addr":"172.21.15.28:6833/1292942516","heartbeat_front_addr":"172.21.15.28:6831/1292942516","state":["exists","up"]},{"osd":10,"uuid":"76c06725-7040-4118-8894-fe808a87f999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6825","nonce":2444616468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6827","nonce":2444616468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6831","nonce":2444616468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6829","nonce":2444616468}]},"public_addr":"172.21.15.38:6825/2444616468","cluster_addr":"172.21.15.38:6827/2444616468","heartbeat_back_addr":"172.21.15.38:6831/2444616468","heartbeat_front_addr":"172.21.15.38:6829/2444616468","state":["exists","up"]},{"osd":11,"uuid":"e8fc9454-5540-4e2e-91bf-3ba251dbd1a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6825","nonce":3400299562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6827","nonce":3400299562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6831","nonce":3400299562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6829","nonce":3400299562}]},"public_addr":"172.21.15.105:6825/3400299562","cluster_addr":"172.21.15.105:6827/3400299562","heartbeat_back_addr":"172.21.15.105:6831/3400299562","heartbeat_front_addr":"172.21.15.105:6829/3400299562","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:35:33.615395+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:00.545293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:24.940211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:57.684887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:24.519061+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:49.010804+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:22.638478+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:49.559104+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:14.011771+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:47.859950+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:16.298445+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:41.619069+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.2d","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/590495740":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/1163172512":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/4290328460":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/4145020172":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/720626958":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/948450262":"2024-04-24T14:33:07.576363+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:43:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.562+0000 7f98cde00640 1 -- 172.21.15.28:0/3733410183 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f98ac0a3c30 msgr2=0x7f98ac0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.562+0000 7f98cde00640 1 --2- 172.21.15.28:0/3733410183 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f98ac0a3c30 0x7f98ac0a60f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f98c819fc80 tx=0x7f98b4005f90 comp rx=0 tx=0).stop 2024-04-23T14:43:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.562+0000 7f98cde00640 1 -- 172.21.15.28:0/3733410183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98c8108480 msgr2=0x7f98c819e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.562+0000 7f98cde00640 1 --2- 172.21.15.28:0/3733410183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98c8108480 0x7f98c819e770 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f98b8031440 tx=0x7f98b8032ce0 comp rx=0 tx=0).stop 2024-04-23T14:43:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.562+0000 7f98cde00640 1 -- 172.21.15.28:0/3733410183 shutdown_connections 2024-04-23T14:43:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.562+0000 7f98cde00640 1 --2- 172.21.15.28:0/3733410183 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f98ac0a3c30 0x7f98ac0a60f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.562+0000 7f98cde00640 1 --2- 172.21.15.28:0/3733410183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98c810a030 0x7f98c81a5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:16.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.562+0000 7f98cde00640 1 --2- 172.21.15.28:0/3733410183 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f98c8109670 0x7f98c819ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:16.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.562+0000 7f98cde00640 1 --2- 172.21.15.28:0/3733410183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98c8108480 0x7f98c819e770 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:16.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.562+0000 7f98cde00640 1 -- 172.21.15.28:0/3733410183 >> 172.21.15.28:0/3733410183 conn(0x7f98c8077440 msgr2=0x7f98c810c970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:16.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.562+0000 7f98cde00640 1 -- 172.21.15.28:0/3733410183 shutdown_connections 2024-04-23T14:43:16.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.562+0000 7f98cde00640 1 -- 172.21.15.28:0/3733410183 wait complete. 2024-04-23T14:43:16.584 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T14:36:27.698112+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T14:41:37.030341+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T14:41:38.444742+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T14:43:16.584 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T14:43:16.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:16 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3622525060' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T14:43:16.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:16 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1603621679' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:43:16.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.804+0000 7f988fc00640 1 -- 172.21.15.28:0/3805981396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9888105470 msgr2=0x7f98881058f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:16.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.804+0000 7f988fc00640 1 --2- 172.21.15.28:0/3805981396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9888105470 0x7f98881058f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9870009900 tx=0x7f987002a3f0 comp rx=0 tx=0).stop 2024-04-23T14:43:16.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.804+0000 7f988fc00640 1 -- 172.21.15.28:0/3805981396 shutdown_connections 2024-04-23T14:43:16.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.804+0000 7f988fc00640 1 --2- 172.21.15.28:0/3805981396 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9888105e30 0x7f988810c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:16.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.804+0000 7f988fc00640 1 --2- 172.21.15.28:0/3805981396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9888105470 0x7f98881058f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:16.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.804+0000 7f988fc00640 1 --2- 172.21.15.28:0/3805981396 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9888104280 0x7f98881046a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:16.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.804+0000 7f988fc00640 1 -- 172.21.15.28:0/3805981396 >> 172.21.15.28:0/3805981396 conn(0x7f98880ff7f0 msgr2=0x7f9888101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:16.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.804+0000 7f988fc00640 1 -- 172.21.15.28:0/3805981396 shutdown_connections 2024-04-23T14:43:16.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.804+0000 7f988fc00640 1 -- 172.21.15.28:0/3805981396 wait complete. 2024-04-23T14:43:16.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.805+0000 7f988fc00640 1 Processor -- start 2024-04-23T14:43:16.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.805+0000 7f988fc00640 1 -- start start 2024-04-23T14:43:16.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.805+0000 7f988fc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9888104280 0x7f9888079db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:16.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.805+0000 7f988fc00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9888105470 0x7f988807a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:16.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.806+0000 7f988fc00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9888105e30 0x7f9888076920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:16.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.806+0000 7f988fc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f988810ff60 con 0x7f9888104280 2024-04-23T14:43:16.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.806+0000 7f988fc00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f988810fde0 con 0x7f9888105e30 2024-04-23T14:43:16.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.806+0000 7f988fc00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f98881100e0 con 0x7f9888105470 2024-04-23T14:43:16.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.806+0000 7f988d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9888104280 0x7f9888079db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:16.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.806+0000 7f988d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9888104280 0x7f9888079db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38954/0 (socket says 172.21.15.28:38954) 2024-04-23T14:43:16.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.806+0000 7f988d200640 1 -- 172.21.15.28:0/4183383562 learned_addr learned my addr 172.21.15.28:0/4183383562 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:16.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.806+0000 7f9887e00640 1 --2- 172.21.15.28:0/4183383562 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9888105470 0x7f988807a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:16.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.806+0000 7f988dc00640 1 --2- 172.21.15.28:0/4183383562 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9888105e30 0x7f9888076920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:16.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.806+0000 7f988d200640 1 -- 172.21.15.28:0/4183383562 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9888105470 msgr2=0x7f988807a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:16.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.806+0000 7f988d200640 1 --2- 172.21.15.28:0/4183383562 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9888105470 0x7f988807a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:16.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.806+0000 7f988d200640 1 -- 172.21.15.28:0/4183383562 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9888105e30 msgr2=0x7f9888076920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:16.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.806+0000 7f988d200640 1 --2- 172.21.15.28:0/4183383562 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9888105e30 0x7f9888076920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:16.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.806+0000 7f988d200640 1 -- 172.21.15.28:0/4183383562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98880771e0 con 0x7f9888104280 2024-04-23T14:43:16.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.806+0000 7f9887e00640 1 --2- 172.21.15.28:0/4183383562 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9888105470 0x7f988807a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:43:16.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.807+0000 7f988d200640 1 --2- 172.21.15.28:0/4183383562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9888104280 0x7f9888079db0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f987800ebc0 tx=0x7f987800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:16.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.807+0000 7f9885600640 1 -- 172.21.15.28:0/4183383562 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9878003b70 con 0x7f9888104280 2024-04-23T14:43:16.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.807+0000 7f988fc00640 1 -- 172.21.15.28:0/4183383562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98881a69b0 con 0x7f9888104280 2024-04-23T14:43:16.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.807+0000 7f9885600640 1 -- 172.21.15.28:0/4183383562 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9878003d10 con 0x7f9888104280 2024-04-23T14:43:16.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.807+0000 7f9885600640 1 -- 172.21.15.28:0/4183383562 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9878019980 con 0x7f9888104280 2024-04-23T14:43:16.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.807+0000 7f988fc00640 1 -- 172.21.15.28:0/4183383562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98881a6db0 con 0x7f9888104280 2024-04-23T14:43:16.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.808+0000 7f988fc00640 1 -- 172.21.15.28:0/4183383562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9888069270 con 0x7f9888104280 2024-04-23T14:43:16.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.810+0000 7f9885600640 1 -- 172.21.15.28:0/4183383562 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f9878019b20 con 0x7f9888104280 2024-04-23T14:43:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.812+0000 7f9885600640 1 --2- 172.21.15.28:0/4183383562 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f986c0a3bb0 0x7f986c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.812+0000 7f9885600640 1 -- 172.21.15.28:0/4183383562 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7f98780d0c00 con 0x7f9888104280 2024-04-23T14:43:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.812+0000 7f9887e00640 1 --2- 172.21.15.28:0/4183383562 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f986c0a3bb0 0x7f986c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.813+0000 7f9887e00640 1 --2- 172.21.15.28:0/4183383562 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f986c0a3bb0 0x7f986c0a6070 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9870031440 tx=0x7f98700315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:16.816+0000 7f9885600640 1 -- 172.21.15.28:0/4183383562 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f987809d5e0 con 0x7f9888104280 2024-04-23T14:43:16.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:16 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3622525060' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T14:43:16.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:16 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1603621679' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:43:16.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:16 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3622525060' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T14:43:16.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:16 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1603621679' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:43:17.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.022+0000 7f988fc00640 1 -- 172.21.15.28:0/4183383562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f988802d620 con 0x7f9888104280 2024-04-23T14:43:17.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.023+0000 7f9885600640 1 -- 172.21.15.28:0/4183383562 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7f987809cee0 con 0x7f9888104280 2024-04-23T14:43:17.025 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-23T14:43:17.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.026+0000 7f988fc00640 1 -- 172.21.15.28:0/4183383562 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f986c0a3bb0 msgr2=0x7f986c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:17.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.026+0000 7f988fc00640 1 --2- 172.21.15.28:0/4183383562 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f986c0a3bb0 0x7f986c0a6070 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9870031440 tx=0x7f98700315b0 comp rx=0 tx=0).stop 2024-04-23T14:43:17.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.026+0000 7f988fc00640 1 -- 172.21.15.28:0/4183383562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9888104280 msgr2=0x7f9888079db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:17.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.026+0000 7f988fc00640 1 --2- 172.21.15.28:0/4183383562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9888104280 0x7f9888079db0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f987800ebc0 tx=0x7f987800c6a0 comp rx=0 tx=0).stop 2024-04-23T14:43:17.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.026+0000 7f988fc00640 1 -- 172.21.15.28:0/4183383562 shutdown_connections 2024-04-23T14:43:17.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.026+0000 7f988fc00640 1 --2- 172.21.15.28:0/4183383562 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f986c0a3bb0 0x7f986c0a6070 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:17.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.026+0000 7f988fc00640 1 --2- 172.21.15.28:0/4183383562 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9888105e30 0x7f9888076920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:17.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.026+0000 7f988fc00640 1 --2- 172.21.15.28:0/4183383562 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9888105470 0x7f988807a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:17.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.026+0000 7f988fc00640 1 --2- 172.21.15.28:0/4183383562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9888104280 0x7f9888079db0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:17.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.026+0000 7f988fc00640 1 -- 172.21.15.28:0/4183383562 >> 172.21.15.28:0/4183383562 conn(0x7f98880ff7f0 msgr2=0x7f9888108770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:17.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.026+0000 7f988fc00640 1 -- 172.21.15.28:0/4183383562 shutdown_connections 2024-04-23T14:43:17.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.027+0000 7f988fc00640 1 -- 172.21.15.28:0/4183383562 wait complete. 2024-04-23T14:43:17.046 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T14:43:17.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.245+0000 7fdf8b200640 1 -- 172.21.15.28:0/230081094 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf8c1052d0 msgr2=0x7fdf8c1076c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:17.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.245+0000 7fdf8b200640 1 --2- 172.21.15.28:0/230081094 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf8c1052d0 0x7fdf8c1076c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdf78009900 tx=0x7fdf7802a3f0 comp rx=0 tx=0).stop 2024-04-23T14:43:17.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.245+0000 7fdf8b200640 1 -- 172.21.15.28:0/230081094 shutdown_connections 2024-04-23T14:43:17.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.245+0000 7fdf8b200640 1 --2- 172.21.15.28:0/230081094 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf8c1052d0 0x7fdf8c1076c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:17.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.245+0000 7fdf8b200640 1 --2- 172.21.15.28:0/230081094 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf8c107c00 0x7fdf8c10a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:17.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.245+0000 7fdf8b200640 1 --2- 172.21.15.28:0/230081094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf8c1029a0 0x7fdf8c104d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:17.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.245+0000 7fdf8b200640 1 -- 172.21.15.28:0/230081094 >> 172.21.15.28:0/230081094 conn(0x7fdf8c0fc370 msgr2=0x7fdf8c0fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:17.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.245+0000 7fdf8b200640 1 -- 172.21.15.28:0/230081094 shutdown_connections 2024-04-23T14:43:17.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.245+0000 7fdf8b200640 1 -- 172.21.15.28:0/230081094 wait complete. 2024-04-23T14:43:17.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.246+0000 7fdf8b200640 1 Processor -- start 2024-04-23T14:43:17.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.247+0000 7fdf8b200640 1 -- start start 2024-04-23T14:43:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.247+0000 7fdf8b200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf8c1029a0 0x7fdf8c19a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.247+0000 7fdf8b200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf8c1052d0 0x7fdf8c19a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.247+0000 7fdf8b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf8c107c00 0x7fdf8c1a19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.247+0000 7fdf8b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdf8c10dbd0 con 0x7fdf8c107c00 2024-04-23T14:43:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.247+0000 7fdf8b200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdf8c10da50 con 0x7fdf8c1029a0 2024-04-23T14:43:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.247+0000 7fdf8b200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdf8c10dd50 con 0x7fdf8c1052d0 2024-04-23T14:43:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.247+0000 7fdf89e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf8c1029a0 0x7fdf8c19a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.247+0000 7fdf8a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf8c107c00 0x7fdf8c1a19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.247+0000 7fdf89e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf8c1029a0 0x7fdf8c19a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:42572/0 (socket says 172.21.15.28:42572) 2024-04-23T14:43:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.247+0000 7fdf89e00640 1 -- 172.21.15.28:0/3402432766 learned_addr learned my addr 172.21.15.28:0/3402432766 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.247+0000 7fdf89400640 1 --2- 172.21.15.28:0/3402432766 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf8c1052d0 0x7fdf8c19a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.248+0000 7fdf89e00640 1 -- 172.21.15.28:0/3402432766 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf8c1052d0 msgr2=0x7fdf8c19a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.248+0000 7fdf89e00640 1 --2- 172.21.15.28:0/3402432766 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf8c1052d0 0x7fdf8c19a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:17.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.248+0000 7fdf89e00640 1 -- 172.21.15.28:0/3402432766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf8c107c00 msgr2=0x7fdf8c1a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:17.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.248+0000 7fdf89e00640 1 --2- 172.21.15.28:0/3402432766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf8c107c00 0x7fdf8c1a19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:17.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.248+0000 7fdf89e00640 1 -- 172.21.15.28:0/3402432766 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf8c1a2010 con 0x7fdf8c1029a0 2024-04-23T14:43:17.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.248+0000 7fdf89e00640 1 --2- 172.21.15.28:0/3402432766 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf8c1029a0 0x7fdf8c19a460 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c00ebc0 tx=0x7fdf7c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:17.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.248+0000 7fdf82a00640 1 -- 172.21.15.28:0/3402432766 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c010630 con 0x7fdf8c1029a0 2024-04-23T14:43:17.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.248+0000 7fdf82a00640 1 -- 172.21.15.28:0/3402432766 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf7c010cb0 con 0x7fdf8c1029a0 2024-04-23T14:43:17.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.248+0000 7fdf8b200640 1 -- 172.21.15.28:0/3402432766 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf8c1a2300 con 0x7fdf8c1029a0 2024-04-23T14:43:17.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.248+0000 7fdf82a00640 1 -- 172.21.15.28:0/3402432766 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c018bd0 con 0x7fdf8c1029a0 2024-04-23T14:43:17.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.248+0000 7fdf8b200640 1 -- 172.21.15.28:0/3402432766 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf8c1a2870 con 0x7fdf8c1029a0 2024-04-23T14:43:17.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.249+0000 7fdf8b200640 1 -- 172.21.15.28:0/3402432766 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf8c109750 con 0x7fdf8c1029a0 2024-04-23T14:43:17.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.251+0000 7fdf82a00640 1 -- 172.21.15.28:0/3402432766 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fdf7c0107d0 con 0x7fdf8c1029a0 2024-04-23T14:43:17.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.253+0000 7fdf82a00640 1 --2- 172.21.15.28:0/3402432766 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fdf700a3c30 0x7fdf700a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:17.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.253+0000 7fdf82a00640 1 -- 172.21.15.28:0/3402432766 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7fdf7c0d09e0 con 0x7fdf8c1029a0 2024-04-23T14:43:17.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.253+0000 7fdf89400640 1 --2- 172.21.15.28:0/3402432766 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fdf700a3c30 0x7fdf700a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:17.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.254+0000 7fdf89400640 1 --2- 172.21.15.28:0/3402432766 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fdf700a3c30 0x7fdf700a60f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fdf7802a9e0 tx=0x7fdf780315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:17.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.257+0000 7fdf82a00640 1 -- 172.21.15.28:0/3402432766 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdf7c09cd30 con 0x7fdf8c1029a0 2024-04-23T14:43:17.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.462+0000 7fdf8b200640 1 -- 172.21.15.28:0/3402432766 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdf8c062c00 con 0x7fdf8c1029a0 2024-04-23T14:43:17.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.463+0000 7fdf82a00640 1 -- 172.21.15.28:0/3402432766 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7fdf7c0a1030 con 0x7fdf8c1029a0 2024-04-23T14:43:17.464 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-23T14:43:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.465+0000 7fdf8b200640 1 -- 172.21.15.28:0/3402432766 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fdf700a3c30 msgr2=0x7fdf700a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.465+0000 7fdf8b200640 1 --2- 172.21.15.28:0/3402432766 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fdf700a3c30 0x7fdf700a60f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fdf7802a9e0 tx=0x7fdf780315b0 comp rx=0 tx=0).stop 2024-04-23T14:43:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.465+0000 7fdf8b200640 1 -- 172.21.15.28:0/3402432766 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf8c1029a0 msgr2=0x7fdf8c19a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.465+0000 7fdf8b200640 1 --2- 172.21.15.28:0/3402432766 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf8c1029a0 0x7fdf8c19a460 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c00ebc0 tx=0x7fdf7c00c6a0 comp rx=0 tx=0).stop 2024-04-23T14:43:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.465+0000 7fdf8b200640 1 -- 172.21.15.28:0/3402432766 shutdown_connections 2024-04-23T14:43:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.465+0000 7fdf8b200640 1 --2- 172.21.15.28:0/3402432766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf8c107c00 0x7fdf8c1a19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.465+0000 7fdf8b200640 1 --2- 172.21.15.28:0/3402432766 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf8c1052d0 0x7fdf8c19a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.465+0000 7fdf8b200640 1 --2- 172.21.15.28:0/3402432766 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fdf700a3c30 0x7fdf700a60f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.465+0000 7fdf8b200640 1 --2- 172.21.15.28:0/3402432766 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf8c1029a0 0x7fdf8c19a460 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.465+0000 7fdf8b200640 1 -- 172.21.15.28:0/3402432766 >> 172.21.15.28:0/3402432766 conn(0x7fdf8c0fc370 msgr2=0x7fdf8c1020f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.466+0000 7fdf8b200640 1 -- 172.21.15.28:0/3402432766 shutdown_connections 2024-04-23T14:43:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.466+0000 7fdf8b200640 1 -- 172.21.15.28:0/3402432766 wait complete. 2024-04-23T14:43:17.486 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T14:43:17.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.689+0000 7fa0eea00640 1 -- 172.21.15.28:0/884336654 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa0e8101900 msgr2=0x7fa0e81061a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:17.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.689+0000 7fa0eea00640 1 --2- 172.21.15.28:0/884336654 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa0e8101900 0x7fa0e81061a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0009be0 tx=0x7fa0e002a6c0 comp rx=0 tx=0).stop 2024-04-23T14:43:17.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.689+0000 7fa0eea00640 1 -- 172.21.15.28:0/884336654 shutdown_connections 2024-04-23T14:43:17.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.689+0000 7fa0eea00640 1 --2- 172.21.15.28:0/884336654 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0e8100f40 0x7fa0e81013c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:17.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.689+0000 7fa0eea00640 1 --2- 172.21.15.28:0/884336654 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa0e8101900 0x7fa0e81061a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:17.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.689+0000 7fa0eea00640 1 --2- 172.21.15.28:0/884336654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0e81001e0 0x7fa0e8100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:17.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.689+0000 7fa0eea00640 1 -- 172.21.15.28:0/884336654 >> 172.21.15.28:0/884336654 conn(0x7fa0e80fbb30 msgr2=0x7fa0e80fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:17.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.689+0000 7fa0eea00640 1 -- 172.21.15.28:0/884336654 shutdown_connections 2024-04-23T14:43:17.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.689+0000 7fa0eea00640 1 -- 172.21.15.28:0/884336654 wait complete. 2024-04-23T14:43:17.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.690+0000 7fa0eea00640 1 Processor -- start 2024-04-23T14:43:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.691+0000 7fa0eea00640 1 -- start start 2024-04-23T14:43:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.691+0000 7fa0eea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0e81001e0 0x7fa0e819e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.691+0000 7fa0eea00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa0e8100f40 0x7fa0e819ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.691+0000 7fa0eea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0e8101900 0x7fa0e81a5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.691+0000 7fa0eea00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e8114210 con 0x7fa0e81001e0 2024-04-23T14:43:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.691+0000 7fa0eea00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e8114090 con 0x7fa0e8101900 2024-04-23T14:43:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.691+0000 7fa0eea00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e8114390 con 0x7fa0e8100f40 2024-04-23T14:43:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.691+0000 7fa0ed600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0e81001e0 0x7fa0e819e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.691+0000 7fa0ed600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0e81001e0 0x7fa0e819e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38978/0 (socket says 172.21.15.28:38978) 2024-04-23T14:43:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.691+0000 7fa0ed600640 1 -- 172.21.15.28:0/3476652124 learned_addr learned my addr 172.21.15.28:0/3476652124 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.691+0000 7fa0ecc00640 1 --2- 172.21.15.28:0/3476652124 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa0e8100f40 0x7fa0e819ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.691+0000 7fa0ee000640 1 --2- 172.21.15.28:0/3476652124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0e8101900 0x7fa0e81a5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.692+0000 7fa0ed600640 1 -- 172.21.15.28:0/3476652124 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa0e8100f40 msgr2=0x7fa0e819ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:17.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.692+0000 7fa0ed600640 1 --2- 172.21.15.28:0/3476652124 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa0e8100f40 0x7fa0e819ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:17.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.692+0000 7fa0ed600640 1 -- 172.21.15.28:0/3476652124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0e8101900 msgr2=0x7fa0e81a5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:17.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.692+0000 7fa0ed600640 1 --2- 172.21.15.28:0/3476652124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0e8101900 0x7fa0e81a5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:17.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.692+0000 7fa0ed600640 1 -- 172.21.15.28:0/3476652124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0e81a63f0 con 0x7fa0e81001e0 2024-04-23T14:43:17.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.692+0000 7fa0ed600640 1 --2- 172.21.15.28:0/3476652124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0e81001e0 0x7fa0e819e820 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc00e9c0 tx=0x7fa0dc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:17.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.692+0000 7fa0e6000640 1 -- 172.21.15.28:0/3476652124 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc00cde0 con 0x7fa0e81001e0 2024-04-23T14:43:17.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.692+0000 7fa0eea00640 1 -- 172.21.15.28:0/3476652124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0e81a66e0 con 0x7fa0e81001e0 2024-04-23T14:43:17.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.692+0000 7fa0e6000640 1 -- 172.21.15.28:0/3476652124 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0dc0035f0 con 0x7fa0e81001e0 2024-04-23T14:43:17.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.692+0000 7fa0e6000640 1 -- 172.21.15.28:0/3476652124 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc010650 con 0x7fa0e81001e0 2024-04-23T14:43:17.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.693+0000 7fa0eea00640 1 -- 172.21.15.28:0/3476652124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0e81a6c80 con 0x7fa0e81001e0 2024-04-23T14:43:17.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.694+0000 7fa0eea00640 1 -- 172.21.15.28:0/3476652124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0e81058a0 con 0x7fa0e81001e0 2024-04-23T14:43:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.696+0000 7fa0e6000640 1 -- 172.21.15.28:0/3476652124 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fa0dc0037a0 con 0x7fa0e81001e0 2024-04-23T14:43:17.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.697+0000 7fa0e6000640 1 --2- 172.21.15.28:0/3476652124 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fa0d00a3c30 0x7fa0d00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:17.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.697+0000 7fa0e6000640 1 -- 172.21.15.28:0/3476652124 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7fa0dc0d06d0 con 0x7fa0e81001e0 2024-04-23T14:43:17.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.697+0000 7fa0ecc00640 1 --2- 172.21.15.28:0/3476652124 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fa0d00a3c30 0x7fa0d00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:17.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.698+0000 7fa0ecc00640 1 --2- 172.21.15.28:0/3476652124 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fa0d00a3c30 0x7fa0d00a60f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa0e819fce0 tx=0x7fa0d4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:17.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.701+0000 7fa0e6000640 1 -- 172.21.15.28:0/3476652124 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa0dc09d070 con 0x7fa0e81001e0 2024-04-23T14:43:17.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:17 smithi105 ceph-mon[32555]: pgmap v364: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-23T14:43:17.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:17 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3733410183' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:43:17.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:17 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/4183383562' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:43:17.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:17 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3402432766' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T14:43:17.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:17 smithi038 ceph-mon[32348]: pgmap v364: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-23T14:43:17.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:17 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3733410183' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:43:17.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:17 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/4183383562' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:43:17.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:17 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3402432766' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T14:43:17.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:17 smithi028 ceph-mon[27810]: pgmap v364: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-23T14:43:17.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:17 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3733410183' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:43:17.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:17 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/4183383562' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:43:17.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:17 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3402432766' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T14:43:17.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.903+0000 7fa0eea00640 1 -- 172.21.15.28:0/3476652124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa0e8062c00 con 0x7fa0e81001e0 2024-04-23T14:43:17.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.904+0000 7fa0e6000640 1 -- 172.21.15.28:0/3476652124 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7fa0dc09c970 con 0x7fa0e81001e0 2024-04-23T14:43:17.906 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-23T14:43:17.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.907+0000 7fa0eea00640 1 -- 172.21.15.28:0/3476652124 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fa0d00a3c30 msgr2=0x7fa0d00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:17.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.907+0000 7fa0eea00640 1 --2- 172.21.15.28:0/3476652124 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fa0d00a3c30 0x7fa0d00a60f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa0e819fce0 tx=0x7fa0d4005f90 comp rx=0 tx=0).stop 2024-04-23T14:43:17.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.907+0000 7fa0eea00640 1 -- 172.21.15.28:0/3476652124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0e81001e0 msgr2=0x7fa0e819e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:17.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.907+0000 7fa0eea00640 1 --2- 172.21.15.28:0/3476652124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0e81001e0 0x7fa0e819e820 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc00e9c0 tx=0x7fa0dc00ee80 comp rx=0 tx=0).stop 2024-04-23T14:43:17.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.907+0000 7fa0eea00640 1 -- 172.21.15.28:0/3476652124 shutdown_connections 2024-04-23T14:43:17.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.907+0000 7fa0eea00640 1 --2- 172.21.15.28:0/3476652124 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fa0d00a3c30 0x7fa0d00a60f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:17.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.907+0000 7fa0eea00640 1 --2- 172.21.15.28:0/3476652124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0e8101900 0x7fa0e81a5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:17.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.907+0000 7fa0eea00640 1 --2- 172.21.15.28:0/3476652124 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa0e8100f40 0x7fa0e819ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:17.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.907+0000 7fa0eea00640 1 --2- 172.21.15.28:0/3476652124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0e81001e0 0x7fa0e819e820 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:17.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.907+0000 7fa0eea00640 1 -- 172.21.15.28:0/3476652124 >> 172.21.15.28:0/3476652124 conn(0x7fa0e80fbb30 msgr2=0x7fa0e80fc7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:17.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.907+0000 7fa0eea00640 1 -- 172.21.15.28:0/3476652124 shutdown_connections 2024-04-23T14:43:17.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:17.907+0000 7fa0eea00640 1 -- 172.21.15.28:0/3476652124 wait complete. 2024-04-23T14:43:17.927 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:43:18.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.135+0000 7f6a6b200640 1 -- 172.21.15.28:0/990689714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a64109670 msgr2=0x7f6a64109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:18.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.135+0000 7f6a6b200640 1 --2- 172.21.15.28:0/990689714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a64109670 0x7f6a64109af0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6a54009900 tx=0x7f6a5402a3f0 comp rx=0 tx=0).stop 2024-04-23T14:43:18.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.135+0000 7f6a6b200640 1 -- 172.21.15.28:0/990689714 shutdown_connections 2024-04-23T14:43:18.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.135+0000 7f6a6b200640 1 --2- 172.21.15.28:0/990689714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a64109670 0x7f6a64109af0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:18.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.135+0000 7f6a6b200640 1 --2- 172.21.15.28:0/990689714 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6a6410a030 0x7f6a641108c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:18.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.135+0000 7f6a6b200640 1 --2- 172.21.15.28:0/990689714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a64108480 0x7f6a641088a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:18.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.136+0000 7f6a6b200640 1 -- 172.21.15.28:0/990689714 >> 172.21.15.28:0/990689714 conn(0x7f6a64077440 msgr2=0x7f6a64079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:18.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.136+0000 7f6a6b200640 1 -- 172.21.15.28:0/990689714 shutdown_connections 2024-04-23T14:43:18.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.136+0000 7f6a6b200640 1 -- 172.21.15.28:0/990689714 wait complete. 2024-04-23T14:43:18.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.137+0000 7f6a6b200640 1 Processor -- start 2024-04-23T14:43:18.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.137+0000 7f6a6b200640 1 -- start start 2024-04-23T14:43:18.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.137+0000 7f6a6b200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6a64108480 0x7f6a6419e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:18.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.137+0000 7f6a6b200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a64109670 0x7f6a6419eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:18.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.137+0000 7f6a6b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a6410a030 0x7f6a641a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:18.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.137+0000 7f6a6b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6a64114110 con 0x7f6a6410a030 2024-04-23T14:43:18.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.137+0000 7f6a6b200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6a64113f90 con 0x7f6a64109670 2024-04-23T14:43:18.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.137+0000 7f6a6b200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6a64114290 con 0x7f6a64108480 2024-04-23T14:43:18.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.138+0000 7f6a69400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a64109670 0x7f6a6419eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:18.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.138+0000 7f6a6a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a6410a030 0x7f6a641a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:18.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.138+0000 7f6a69e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6a64108480 0x7f6a6419e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:18.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.138+0000 7f6a69400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a64109670 0x7f6a6419eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:33930/0 (socket says 172.21.15.28:33930) 2024-04-23T14:43:18.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.138+0000 7f6a69400640 1 -- 172.21.15.28:0/2214194533 learned_addr learned my addr 172.21.15.28:0/2214194533 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:18.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.138+0000 7f6a69400640 1 -- 172.21.15.28:0/2214194533 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6a64108480 msgr2=0x7f6a6419e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:18.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.138+0000 7f6a69400640 1 --2- 172.21.15.28:0/2214194533 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6a64108480 0x7f6a6419e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:18.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.138+0000 7f6a69400640 1 -- 172.21.15.28:0/2214194533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a6410a030 msgr2=0x7f6a641a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:18.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.138+0000 7f6a69400640 1 --2- 172.21.15.28:0/2214194533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a6410a030 0x7f6a641a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:18.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.138+0000 7f6a69400640 1 -- 172.21.15.28:0/2214194533 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a641a6380 con 0x7f6a64109670 2024-04-23T14:43:18.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.138+0000 7f6a69e00640 1 --2- 172.21.15.28:0/2214194533 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6a64108480 0x7f6a6419e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:43:18.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.138+0000 7f6a6a800640 1 --2- 172.21.15.28:0/2214194533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a6410a030 0x7f6a641a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:43:18.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.138+0000 7f6a69400640 1 --2- 172.21.15.28:0/2214194533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a64109670 0x7f6a6419eca0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6a54009230 tx=0x7f6a540042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:18.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.139+0000 7f6a5ea00640 1 -- 172.21.15.28:0/2214194533 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a5403f030 con 0x7f6a64109670 2024-04-23T14:43:18.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.139+0000 7f6a5ea00640 1 -- 172.21.15.28:0/2214194533 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a54003ce0 con 0x7f6a64109670 2024-04-23T14:43:18.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.139+0000 7f6a6b200640 1 -- 172.21.15.28:0/2214194533 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a641a6610 con 0x7f6a64109670 2024-04-23T14:43:18.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.139+0000 7f6a5ea00640 1 -- 172.21.15.28:0/2214194533 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a54035070 con 0x7f6a64109670 2024-04-23T14:43:18.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.139+0000 7f6a6b200640 1 -- 172.21.15.28:0/2214194533 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a641a6b00 con 0x7f6a64109670 2024-04-23T14:43:18.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.140+0000 7f6a6b200640 1 -- 172.21.15.28:0/2214194533 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a6410f7a0 con 0x7f6a64109670 2024-04-23T14:43:18.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.142+0000 7f6a5ea00640 1 -- 172.21.15.28:0/2214194533 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f6a54042030 con 0x7f6a64109670 2024-04-23T14:43:18.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.144+0000 7f6a5ea00640 1 --2- 172.21.15.28:0/2214194533 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f6a4c0a3bb0 0x7f6a4c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:18.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.144+0000 7f6a5ea00640 1 -- 172.21.15.28:0/2214194533 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7f6a540f0a40 con 0x7f6a64109670 2024-04-23T14:43:18.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.144+0000 7f6a69e00640 1 --2- 172.21.15.28:0/2214194533 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f6a4c0a3bb0 0x7f6a4c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:18.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.145+0000 7f6a69e00640 1 --2- 172.21.15.28:0/2214194533 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f6a4c0a3bb0 0x7f6a4c0a6070 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6a641094d0 tx=0x7f6a58005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:18.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.147+0000 7f6a5ea00640 1 -- 172.21.15.28:0/2214194533 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6a540bcd90 con 0x7f6a64109670 2024-04-23T14:43:18.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.381+0000 7f6a6b200640 1 -- 172.21.15.28:0/2214194533 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6a6402d620 con 0x7f6a64109670 2024-04-23T14:43:18.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.383+0000 7f6a5ea00640 1 -- 172.21.15.28:0/2214194533 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7f6a540c1090 con 0x7f6a64109670 2024-04-23T14:43:18.384 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:43:18.385 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:43:18.385 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:43:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.385+0000 7f6a6b200640 1 -- 172.21.15.28:0/2214194533 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f6a4c0a3bb0 msgr2=0x7f6a4c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.385+0000 7f6a6b200640 1 --2- 172.21.15.28:0/2214194533 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f6a4c0a3bb0 0x7f6a4c0a6070 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6a641094d0 tx=0x7f6a58005f90 comp rx=0 tx=0).stop 2024-04-23T14:43:18.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.385+0000 7f6a6b200640 1 -- 172.21.15.28:0/2214194533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a64109670 msgr2=0x7f6a6419eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:18.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.385+0000 7f6a6b200640 1 --2- 172.21.15.28:0/2214194533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a64109670 0x7f6a6419eca0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6a54009230 tx=0x7f6a540042f0 comp rx=0 tx=0).stop 2024-04-23T14:43:18.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.385+0000 7f6a6b200640 1 -- 172.21.15.28:0/2214194533 shutdown_connections 2024-04-23T14:43:18.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.385+0000 7f6a6b200640 1 --2- 172.21.15.28:0/2214194533 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f6a4c0a3bb0 0x7f6a4c0a6070 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:18.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.385+0000 7f6a6b200640 1 --2- 172.21.15.28:0/2214194533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a64109670 0x7f6a6419eca0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:18.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.385+0000 7f6a6b200640 1 --2- 172.21.15.28:0/2214194533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a6410a030 0x7f6a641a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:18.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.385+0000 7f6a6b200640 1 --2- 172.21.15.28:0/2214194533 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6a64108480 0x7f6a6419e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:18.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.385+0000 7f6a6b200640 1 -- 172.21.15.28:0/2214194533 >> 172.21.15.28:0/2214194533 conn(0x7f6a64077440 msgr2=0x7f6a6410c970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:18.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.386+0000 7f6a6b200640 1 -- 172.21.15.28:0/2214194533 shutdown_connections 2024-04-23T14:43:18.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.386+0000 7f6a6b200640 1 -- 172.21.15.28:0/2214194533 wait complete. 2024-04-23T14:43:18.406 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:43:18.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.611+0000 7f26d5600640 1 -- 172.21.15.28:0/3482843390 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26d0101050 msgr2=0x7f26d01014b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:18.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.611+0000 7f26d5600640 1 --2- 172.21.15.28:0/3482843390 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26d0101050 0x7f26d01014b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f26b8009900 tx=0x7f26b802a3f0 comp rx=0 tx=0).stop 2024-04-23T14:43:18.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.611+0000 7f26d5600640 1 -- 172.21.15.28:0/3482843390 shutdown_connections 2024-04-23T14:43:18.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.611+0000 7f26d5600640 1 --2- 172.21.15.28:0/3482843390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26d01019f0 0x7f26d0101e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:18.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.611+0000 7f26d5600640 1 --2- 172.21.15.28:0/3482843390 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26d0101050 0x7f26d01014b0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:18.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.611+0000 7f26d5600640 1 --2- 172.21.15.28:0/3482843390 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26d01006f0 0x7f26d0100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:18.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.611+0000 7f26d5600640 1 -- 172.21.15.28:0/3482843390 >> 172.21.15.28:0/3482843390 conn(0x7f26d00fc2d0 msgr2=0x7f26d00fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:18.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.611+0000 7f26d5600640 1 -- 172.21.15.28:0/3482843390 shutdown_connections 2024-04-23T14:43:18.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.611+0000 7f26d5600640 1 -- 172.21.15.28:0/3482843390 wait complete. 2024-04-23T14:43:18.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.612+0000 7f26d5600640 1 Processor -- start 2024-04-23T14:43:18.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.612+0000 7f26d5600640 1 -- start start 2024-04-23T14:43:18.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.612+0000 7f26d5600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26d01006f0 0x7f26d019a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:18.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.612+0000 7f26d5600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26d0101050 0x7f26d019a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:18.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.612+0000 7f26d5600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26d01019f0 0x7f26d01a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:18.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.612+0000 7f26d5600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f26d010fd70 con 0x7f26d0101050 2024-04-23T14:43:18.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.612+0000 7f26d5600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f26d010fbf0 con 0x7f26d01019f0 2024-04-23T14:43:18.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.612+0000 7f26d5600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f26d010fef0 con 0x7f26d01006f0 2024-04-23T14:43:18.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.612+0000 7f26c7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26d0101050 0x7f26d019a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:18.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.612+0000 7f26c7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26d0101050 0x7f26d019a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53600/0 (socket says 172.21.15.28:53600) 2024-04-23T14:43:18.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.612+0000 7f26c7400640 1 -- 172.21.15.28:0/1264815913 learned_addr learned my addr 172.21.15.28:0/1264815913 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:18.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.613+0000 7f26cfe00640 1 --2- 172.21.15.28:0/1264815913 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26d01006f0 0x7f26d019a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:18.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.613+0000 7f26d4c00640 1 --2- 172.21.15.28:0/1264815913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26d01019f0 0x7f26d01a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:18.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.613+0000 7f26c7400640 1 -- 172.21.15.28:0/1264815913 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26d01006f0 msgr2=0x7f26d019a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:18.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.613+0000 7f26c7400640 1 --2- 172.21.15.28:0/1264815913 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26d01006f0 0x7f26d019a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:18.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.613+0000 7f26c7400640 1 -- 172.21.15.28:0/1264815913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26d01019f0 msgr2=0x7f26d01a1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:18.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.613+0000 7f26c7400640 1 --2- 172.21.15.28:0/1264815913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26d01019f0 0x7f26d01a1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:18.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.613+0000 7f26c7400640 1 -- 172.21.15.28:0/1264815913 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26d01a1fc0 con 0x7f26d0101050 2024-04-23T14:43:18.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.613+0000 7f26c7400640 1 --2- 172.21.15.28:0/1264815913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26d0101050 0x7f26d019a8e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f26b8031440 tx=0x7f26b8032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:18.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.614+0000 7f26cd600640 1 -- 172.21.15.28:0/1264815913 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b803f030 con 0x7f26d0101050 2024-04-23T14:43:18.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.614+0000 7f26cd600640 1 -- 172.21.15.28:0/1264815913 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26b8003ce0 con 0x7f26d0101050 2024-04-23T14:43:18.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.614+0000 7f26d5600640 1 -- 172.21.15.28:0/1264815913 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26d01a2250 con 0x7f26d0101050 2024-04-23T14:43:18.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.614+0000 7f26cd600640 1 -- 172.21.15.28:0/1264815913 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b8035070 con 0x7f26d0101050 2024-04-23T14:43:18.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.614+0000 7f26d5600640 1 -- 172.21.15.28:0/1264815913 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26d01a2790 con 0x7f26d0101050 2024-04-23T14:43:18.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.615+0000 7f26d5600640 1 -- 172.21.15.28:0/1264815913 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26d010b440 con 0x7f26d0101050 2024-04-23T14:43:18.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.617+0000 7f26cd600640 1 -- 172.21.15.28:0/1264815913 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f26b8042030 con 0x7f26d0101050 2024-04-23T14:43:18.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.618+0000 7f26cd600640 1 --2- 172.21.15.28:0/1264815913 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f26b40a3c30 0x7f26b40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:18.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.618+0000 7f26cd600640 1 -- 172.21.15.28:0/1264815913 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7f26b80f0910 con 0x7f26d0101050 2024-04-23T14:43:18.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.619+0000 7f26cfe00640 1 --2- 172.21.15.28:0/1264815913 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f26b40a3c30 0x7f26b40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:18.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.619+0000 7f26cfe00640 1 --2- 172.21.15.28:0/1264815913 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f26b40a3c30 0x7f26b40a60f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f26c0009bd0 tx=0x7f26c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:18.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.623+0000 7f26cd600640 1 -- 172.21.15.28:0/1264815913 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f26b80bcc60 con 0x7f26d0101050 2024-04-23T14:43:18.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:18 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3476652124' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T14:43:18.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:18 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2214194533' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:43:18.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.838+0000 7f26d5600640 1 -- 172.21.15.28:0/1264815913 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f26d0062c00 con 0x7f26d0101050 2024-04-23T14:43:18.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.840+0000 7f26cd600640 1 -- 172.21.15.28:0/1264815913 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22651 (secure 0 0 0) 0x7f26b80c0f60 con 0x7f26d0101050 2024-04-23T14:43:18.842 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:43:18.842 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":87,"fsid":"1ee6e996-017e-11ef-bc93-c7b262605968","created":"2024-04-23T14:32:34.581082+0000","modified":"2024-04-23T14:42:43.429141+0000","last_up_change":"2024-04-23T14:40:43.375846+0000","last_in_change":"2024-04-23T14:40:23.576939+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:36:27.698112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:41:37.030341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:41:38.444742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e0b427bc-be65-4543-99cc-2f53ad98ddad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6803","nonce":2993445336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6805","nonce":2993445336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6809","nonce":2993445336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6807","nonce":2993445336}]},"public_addr":"172.21.15.28:6803/2993445336","cluster_addr":"172.21.15.28:6805/2993445336","heartbeat_back_addr":"172.21.15.28:6809/2993445336","heartbeat_front_addr":"172.21.15.28:6807/2993445336","state":["exists","up"]},{"osd":1,"uuid":"ef527ec9-1bf3-4bb0-8df3-f5a015af557c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6801","nonce":4024093383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6803","nonce":4024093383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6807","nonce":4024093383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6805","nonce":4024093383}]},"public_addr":"172.21.15.38:6801/4024093383","cluster_addr":"172.21.15.38:6803/4024093383","heartbeat_back_addr":"172.21.15.38:6807/4024093383","heartbeat_front_addr":"172.21.15.38:6805/4024093383","state":["exists","up"]},{"osd":2,"uuid":"b33b23d9-adce-4c11-b304-2e117bdf6bfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6801","nonce":570739029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6803","nonce":570739029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6807","nonce":570739029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6805","nonce":570739029}]},"public_addr":"172.21.15.105:6801/570739029","cluster_addr":"172.21.15.105:6803/570739029","heartbeat_back_addr":"172.21.15.105:6807/570739029","heartbeat_front_addr":"172.21.15.105:6805/570739029","state":["exists","up"]},{"osd":3,"uuid":"28bd5608-b426-434e-9db1-bdaa9828bdb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6811","nonce":2480400247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6813","nonce":2480400247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6817","nonce":2480400247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6815","nonce":2480400247}]},"public_addr":"172.21.15.28:6811/2480400247","cluster_addr":"172.21.15.28:6813/2480400247","heartbeat_back_addr":"172.21.15.28:6817/2480400247","heartbeat_front_addr":"172.21.15.28:6815/2480400247","state":["exists","up"]},{"osd":4,"uuid":"e5d1c89b-025f-4d57-8807-59d75c787b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6809","nonce":3772189246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6811","nonce":3772189246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6815","nonce":3772189246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6813","nonce":3772189246}]},"public_addr":"172.21.15.38:6809/3772189246","cluster_addr":"172.21.15.38:6811/3772189246","heartbeat_back_addr":"172.21.15.38:6815/3772189246","heartbeat_front_addr":"172.21.15.38:6813/3772189246","state":["exists","up"]},{"osd":5,"uuid":"9cf9cf4c-aafd-4ed3-bfb1-0527710a0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6809","nonce":928755894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6811","nonce":928755894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6815","nonce":928755894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6813","nonce":928755894}]},"public_addr":"172.21.15.105:6809/928755894","cluster_addr":"172.21.15.105:6811/928755894","heartbeat_back_addr":"172.21.15.105:6815/928755894","heartbeat_front_addr":"172.21.15.105:6813/928755894","state":["exists","up"]},{"osd":6,"uuid":"304db525-d24d-4687-bfa0-f92bdf66d1e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6819","nonce":897979441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6821","nonce":897979441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6825","nonce":897979441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6823","nonce":897979441}]},"public_addr":"172.21.15.28:6819/897979441","cluster_addr":"172.21.15.28:6821/897979441","heartbeat_back_addr":"172.21.15.28:6825/897979441","heartbeat_front_addr":"172.21.15.28:6823/897979441","state":["exists","up"]},{"osd":7,"uuid":"08e01d59-03f3-47b1-b45f-35ec5a50992c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6817","nonce":2400251786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6819","nonce":2400251786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6823","nonce":2400251786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6821","nonce":2400251786}]},"public_addr":"172.21.15.38:6817/2400251786","cluster_addr":"172.21.15.38:6819/2400251786","heartbeat_back_addr":"172.21.15.38:6823/2400251786","heartbeat_front_addr":"172.21.15.38:6821/2400251786","state":["exists","up"]},{"osd":8,"uuid":"64ee1e95-0756-48b1-9c00-472d0c3cbd29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6817","nonce":1682312269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6819","nonce":1682312269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6823","nonce":1682312269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6821","nonce":1682312269}]},"public_addr":"172.21.15.105:6817/1682312269","cluster_addr":"172.21.15.105:6819/1682312269","heartbeat_back_addr":"172.21.15.105:6823/1682312269","heartbeat_front_addr":"172.21.15.105:6821/1682312269","state":["exists","up"]},{"osd":9,"uuid":"18f60893-3ffd-48c2-9db5-a67d35611d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6827","nonce":1292942516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6829","nonce":1292942516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6833","nonce":1292942516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6831","nonce":1292942516}]},"public_addr":"172.21.15.28:6827/1292942516","cluster_addr":"172.21.15.28:6829/1292942516","heartbeat_back_addr":"172.21.15.28:6833/1292942516","heartbeat_front_addr":"172.21.15.28:6831/1292942516","state":["exists","up"]},{"osd":10,"uuid":"76c06725-7040-4118-8894-fe808a87f999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6825","nonce":2444616468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6827","nonce":2444616468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6831","nonce":2444616468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6829","nonce":2444616468}]},"public_addr":"172.21.15.38:6825/2444616468","cluster_addr":"172.21.15.38:6827/2444616468","heartbeat_back_addr":"172.21.15.38:6831/2444616468","heartbeat_front_addr":"172.21.15.38:6829/2444616468","state":["exists","up"]},{"osd":11,"uuid":"e8fc9454-5540-4e2e-91bf-3ba251dbd1a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6825","nonce":3400299562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6827","nonce":3400299562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6831","nonce":3400299562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6829","nonce":3400299562}]},"public_addr":"172.21.15.105:6825/3400299562","cluster_addr":"172.21.15.105:6827/3400299562","heartbeat_back_addr":"172.21.15.105:6831/3400299562","heartbeat_front_addr":"172.21.15.105:6829/3400299562","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:35:33.615395+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:00.545293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:24.940211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:57.684887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:24.519061+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:49.010804+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:22.638478+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:49.559104+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:14.011771+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:47.859950+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:16.298445+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:41.619069+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.2d","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/590495740":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/1163172512":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/4290328460":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/4145020172":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/720626958":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/948450262":"2024-04-24T14:33:07.576363+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:43:18.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.843+0000 7f26d5600640 1 -- 172.21.15.28:0/1264815913 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f26b40a3c30 msgr2=0x7f26b40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:18.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.843+0000 7f26d5600640 1 --2- 172.21.15.28:0/1264815913 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f26b40a3c30 0x7f26b40a60f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f26c0009bd0 tx=0x7f26c0005f90 comp rx=0 tx=0).stop 2024-04-23T14:43:18.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.843+0000 7f26d5600640 1 -- 172.21.15.28:0/1264815913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26d0101050 msgr2=0x7f26d019a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:18.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.843+0000 7f26d5600640 1 --2- 172.21.15.28:0/1264815913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26d0101050 0x7f26d019a8e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f26b8031440 tx=0x7f26b8032ce0 comp rx=0 tx=0).stop 2024-04-23T14:43:18.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.843+0000 7f26d5600640 1 -- 172.21.15.28:0/1264815913 shutdown_connections 2024-04-23T14:43:18.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.843+0000 7f26d5600640 1 --2- 172.21.15.28:0/1264815913 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f26b40a3c30 0x7f26b40a60f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:18.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.843+0000 7f26d5600640 1 --2- 172.21.15.28:0/1264815913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26d01019f0 0x7f26d01a1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:18.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.843+0000 7f26d5600640 1 --2- 172.21.15.28:0/1264815913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26d0101050 0x7f26d019a8e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:18.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.843+0000 7f26d5600640 1 --2- 172.21.15.28:0/1264815913 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26d01006f0 0x7f26d019a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:18.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.843+0000 7f26d5600640 1 -- 172.21.15.28:0/1264815913 >> 172.21.15.28:0/1264815913 conn(0x7f26d00fc2d0 msgr2=0x7f26d00fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:18.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.843+0000 7f26d5600640 1 -- 172.21.15.28:0/1264815913 shutdown_connections 2024-04-23T14:43:18.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:18.844+0000 7f26d5600640 1 -- 172.21.15.28:0/1264815913 wait complete. 2024-04-23T14:43:18.861 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:18 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3476652124' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T14:43:18.861 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:18 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2214194533' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:43:18.865 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:43:18.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:18 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3476652124' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T14:43:18.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:18 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2214194533' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:43:19.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.076+0000 7f6e19e00640 1 -- 172.21.15.28:0/431591223 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e14074a60 msgr2=0x7f6e14074ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:19.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.076+0000 7f6e19e00640 1 --2- 172.21.15.28:0/431591223 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e14074a60 0x7f6e14074ee0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc009930 tx=0x7f6dfc02a420 comp rx=0 tx=0).stop 2024-04-23T14:43:19.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.076+0000 7f6e19e00640 1 -- 172.21.15.28:0/431591223 shutdown_connections 2024-04-23T14:43:19.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.076+0000 7f6e19e00640 1 --2- 172.21.15.28:0/431591223 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e14074a60 0x7f6e14074ee0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.076+0000 7f6e19e00640 1 --2- 172.21.15.28:0/431591223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e1410e440 0x7f6e14110830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.076+0000 7f6e19e00640 1 --2- 172.21.15.28:0/431591223 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6e14076040 0x7f6e14074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.076+0000 7f6e19e00640 1 -- 172.21.15.28:0/431591223 >> 172.21.15.28:0/431591223 conn(0x7f6e140fd3e0 msgr2=0x7f6e140ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:19.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.076+0000 7f6e19e00640 1 -- 172.21.15.28:0/431591223 shutdown_connections 2024-04-23T14:43:19.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.077+0000 7f6e19e00640 1 -- 172.21.15.28:0/431591223 wait complete. 2024-04-23T14:43:19.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.078+0000 7f6e19e00640 1 Processor -- start 2024-04-23T14:43:19.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.078+0000 7f6e19e00640 1 -- start start 2024-04-23T14:43:19.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.078+0000 7f6e19e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e14074a60 0x7f6e1419e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:19.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.078+0000 7f6e19e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e14076040 0x7f6e1419ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:19.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.078+0000 7f6e19e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6e1410e440 0x7f6e141a5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:19.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.078+0000 7f6e19e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6e14114150 con 0x7f6e14074a60 2024-04-23T14:43:19.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.078+0000 7f6e19e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6e14113fd0 con 0x7f6e14076040 2024-04-23T14:43:19.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.078+0000 7f6e19e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6e141142d0 con 0x7f6e1410e440 2024-04-23T14:43:19.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.079+0000 7f6e18a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e14074a60 0x7f6e1419e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:19.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.079+0000 7f6e18a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e14074a60 0x7f6e1419e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53614/0 (socket says 172.21.15.28:53614) 2024-04-23T14:43:19.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.079+0000 7f6e18a00640 1 -- 172.21.15.28:0/2815384429 learned_addr learned my addr 172.21.15.28:0/2815384429 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:19.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.079+0000 7f6e0be00640 1 --2- 172.21.15.28:0/2815384429 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e14076040 0x7f6e1419ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:19.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.079+0000 7f6e19400640 1 --2- 172.21.15.28:0/2815384429 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6e1410e440 0x7f6e141a5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:19.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.079+0000 7f6e18a00640 1 -- 172.21.15.28:0/2815384429 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6e1410e440 msgr2=0x7f6e141a5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:19.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.079+0000 7f6e18a00640 1 --2- 172.21.15.28:0/2815384429 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6e1410e440 0x7f6e141a5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.079+0000 7f6e18a00640 1 -- 172.21.15.28:0/2815384429 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e14076040 msgr2=0x7f6e1419ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:19.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.079+0000 7f6e18a00640 1 --2- 172.21.15.28:0/2815384429 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e14076040 0x7f6e1419ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.079+0000 7f6e18a00640 1 -- 172.21.15.28:0/2815384429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e141a63e0 con 0x7f6e14074a60 2024-04-23T14:43:19.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.079+0000 7f6e19400640 1 --2- 172.21.15.28:0/2815384429 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6e1410e440 0x7f6e141a5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:43:19.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.079+0000 7f6e18a00640 1 --2- 172.21.15.28:0/2815384429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e14074a60 0x7f6e1419e7c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6e0400eab0 tx=0x7f6e0400ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:19.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.080+0000 7f6e12000640 1 -- 172.21.15.28:0/2815384429 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e0400ce40 con 0x7f6e14074a60 2024-04-23T14:43:19.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.080+0000 7f6e12000640 1 -- 172.21.15.28:0/2815384429 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e04011950 con 0x7f6e14074a60 2024-04-23T14:43:19.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.080+0000 7f6e12000640 1 -- 172.21.15.28:0/2815384429 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e04010ae0 con 0x7f6e14074a60 2024-04-23T14:43:19.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.080+0000 7f6e19e00640 1 -- 172.21.15.28:0/2815384429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e141a66d0 con 0x7f6e14074a60 2024-04-23T14:43:19.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.080+0000 7f6e19e00640 1 -- 172.21.15.28:0/2815384429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e141a6bf0 con 0x7f6e14074a60 2024-04-23T14:43:19.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.081+0000 7f6e19e00640 1 -- 172.21.15.28:0/2815384429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e14104500 con 0x7f6e14074a60 2024-04-23T14:43:19.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.084+0000 7f6e12000640 1 -- 172.21.15.28:0/2815384429 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f6e04011b00 con 0x7f6e14074a60 2024-04-23T14:43:19.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.086+0000 7f6e12000640 1 --2- 172.21.15.28:0/2815384429 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f6df80a3bb0 0x7f6df80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:19.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.086+0000 7f6e12000640 1 -- 172.21.15.28:0/2815384429 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7f6e040d0ac0 con 0x7f6e14074a60 2024-04-23T14:43:19.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.086+0000 7f6e0be00640 1 --2- 172.21.15.28:0/2815384429 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f6df80a3bb0 0x7f6df80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:19.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.087+0000 7f6e0be00640 1 --2- 172.21.15.28:0/2815384429 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f6df80a3bb0 0x7f6df80a6070 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc009900 tx=0x7f6dfc0315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:19.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.091+0000 7f6e12000640 1 -- 172.21.15.28:0/2815384429 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6e04016030 con 0x7f6e14074a60 2024-04-23T14:43:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.337+0000 7f6e19e00640 1 -- 172.21.15.28:0/2815384429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6e1402d620 con 0x7f6e14074a60 2024-04-23T14:43:19.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.339+0000 7f6e12000640 1 -- 172.21.15.28:0/2815384429 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7f6e0409d4a0 con 0x7f6e14074a60 2024-04-23T14:43:19.341 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:43:19.341 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:43:19.341 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:43:19.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.342+0000 7f6e19e00640 1 -- 172.21.15.28:0/2815384429 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f6df80a3bb0 msgr2=0x7f6df80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.342+0000 7f6e19e00640 1 --2- 172.21.15.28:0/2815384429 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f6df80a3bb0 0x7f6df80a6070 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc009900 tx=0x7f6dfc0315b0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.342+0000 7f6e19e00640 1 -- 172.21.15.28:0/2815384429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e14074a60 msgr2=0x7f6e1419e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.342+0000 7f6e19e00640 1 --2- 172.21.15.28:0/2815384429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e14074a60 0x7f6e1419e7c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6e0400eab0 tx=0x7f6e0400ef70 comp rx=0 tx=0).stop 2024-04-23T14:43:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.342+0000 7f6e19e00640 1 -- 172.21.15.28:0/2815384429 shutdown_connections 2024-04-23T14:43:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.342+0000 7f6e19e00640 1 --2- 172.21.15.28:0/2815384429 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f6df80a3bb0 0x7f6df80a6070 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.342+0000 7f6e19e00640 1 --2- 172.21.15.28:0/2815384429 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e14076040 0x7f6e1419ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.342+0000 7f6e19e00640 1 --2- 172.21.15.28:0/2815384429 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6e1410e440 0x7f6e141a5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.342+0000 7f6e19e00640 1 --2- 172.21.15.28:0/2815384429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e14074a60 0x7f6e1419e7c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.342+0000 7f6e19e00640 1 -- 172.21.15.28:0/2815384429 >> 172.21.15.28:0/2815384429 conn(0x7f6e140fd3e0 msgr2=0x7f6e141016d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.343+0000 7f6e19e00640 1 -- 172.21.15.28:0/2815384429 shutdown_connections 2024-04-23T14:43:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.343+0000 7f6e19e00640 1 -- 172.21.15.28:0/2815384429 wait complete. 2024-04-23T14:43:19.364 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-23T14:41:39.495067+0000', 'modified': '2024-04-23T14:41:43.869888+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24587}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24583': {'gid': 24583, 'name': 'g', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.28:6839/940220693', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6838', 'nonce': 940220693}, {'type': 'v1', 'addr': '172.21.15.28:6839', 'nonce': 940220693}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24587': {'gid': 24587, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.38:6839/377787688', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6838', 'nonce': 377787688}, {'type': 'v1', 'addr': '172.21.15.38:6839', 'nonce': 377787688}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24587, 'qdb_cluster': [24587]} 2024-04-23T14:43:19.365 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-23T14:43:19.365 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-23T14:43:19.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.572+0000 7f9036200640 1 -- 172.21.15.28:0/1707718830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9030104260 msgr2=0x7f9030104680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:19.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.572+0000 7f9036200640 1 --2- 172.21.15.28:0/1707718830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9030104260 0x7f9030104680 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9020009900 tx=0x7f902002a410 comp rx=0 tx=0).stop 2024-04-23T14:43:19.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.572+0000 7f9036200640 1 -- 172.21.15.28:0/1707718830 shutdown_connections 2024-04-23T14:43:19.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.572+0000 7f9036200640 1 --2- 172.21.15.28:0/1707718830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9030105450 0x7f90301058d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.572+0000 7f9036200640 1 --2- 172.21.15.28:0/1707718830 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9030105e10 0x7f903010c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.572+0000 7f9036200640 1 --2- 172.21.15.28:0/1707718830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9030104260 0x7f9030104680 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.572+0000 7f9036200640 1 -- 172.21.15.28:0/1707718830 >> 172.21.15.28:0/1707718830 conn(0x7f90300ff7f0 msgr2=0x7f9030101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:19.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.572+0000 7f9036200640 1 -- 172.21.15.28:0/1707718830 shutdown_connections 2024-04-23T14:43:19.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.572+0000 7f9036200640 1 -- 172.21.15.28:0/1707718830 wait complete. 2024-04-23T14:43:19.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.573+0000 7f9036200640 1 Processor -- start 2024-04-23T14:43:19.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.573+0000 7f9036200640 1 -- start start 2024-04-23T14:43:19.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.573+0000 7f9036200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9030104260 0x7f903019a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:19.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.573+0000 7f9036200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9030105450 0x7f903019aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:19.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.573+0000 7f9036200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9030105e10 0x7f90301a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:19.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.573+0000 7f9036200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f903010ff40 con 0x7f9030105450 2024-04-23T14:43:19.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.573+0000 7f9036200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f903010fdc0 con 0x7f9030104260 2024-04-23T14:43:19.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.573+0000 7f9036200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f90301100c0 con 0x7f9030105e10 2024-04-23T14:43:19.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.574+0000 7f902ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9030105450 0x7f903019aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:19.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.574+0000 7f902fe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9030105e10 0x7f90301a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:19.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.574+0000 7f902ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9030105450 0x7f903019aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53620/0 (socket says 172.21.15.28:53620) 2024-04-23T14:43:19.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.574+0000 7f902f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9030104260 0x7f903019a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:19.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.574+0000 7f902fe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9030105e10 0x7f90301a1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:52344/0 (socket says 172.21.15.28:52344) 2024-04-23T14:43:19.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.574+0000 7f902ea00640 1 -- 172.21.15.28:0/487571653 learned_addr learned my addr 172.21.15.28:0/487571653 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:19.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.574+0000 7f902fe00640 1 -- 172.21.15.28:0/487571653 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9030104260 msgr2=0x7f903019a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:19.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.574+0000 7f902fe00640 1 --2- 172.21.15.28:0/487571653 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9030104260 0x7f903019a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.574+0000 7f902fe00640 1 -- 172.21.15.28:0/487571653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9030105450 msgr2=0x7f903019aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:19.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.574+0000 7f902fe00640 1 --2- 172.21.15.28:0/487571653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9030105450 0x7f903019aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.574+0000 7f902fe00640 1 -- 172.21.15.28:0/487571653 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90301a21b0 con 0x7f9030105e10 2024-04-23T14:43:19.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.574+0000 7f902ea00640 1 --2- 172.21.15.28:0/487571653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9030105450 0x7f903019aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:43:19.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.574+0000 7f902fe00640 1 --2- 172.21.15.28:0/487571653 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9030105e10 0x7f90301a1b40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f902800e940 tx=0x7f902800ee00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:19.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.575+0000 7f9027e00640 1 -- 172.21.15.28:0/487571653 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9028004d20 con 0x7f9030105e10 2024-04-23T14:43:19.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.575+0000 7f9027e00640 1 -- 172.21.15.28:0/487571653 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9028005020 con 0x7f9030105e10 2024-04-23T14:43:19.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.575+0000 7f9027e00640 1 -- 172.21.15.28:0/487571653 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90280116b0 con 0x7f9030105e10 2024-04-23T14:43:19.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.575+0000 7f9036200640 1 -- 172.21.15.28:0/487571653 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90301a24a0 con 0x7f9030105e10 2024-04-23T14:43:19.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.575+0000 7f9036200640 1 -- 172.21.15.28:0/487571653 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90301a29a0 con 0x7f9030105e10 2024-04-23T14:43:19.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.578+0000 7f9039173740 1 Processor -- start 2024-04-23T14:43:19.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.578+0000 7f9027e00640 1 -- 172.21.15.28:0/487571653 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f90280054d0 con 0x7f9030105e10 2024-04-23T14:43:19.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.578+0000 7f9039173740 1 -- start start 2024-04-23T14:43:19.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.578+0000 7f9039173740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d5847e8a0 0x558d5847ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:19.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.578+0000 7f9039173740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d5847f290 0x558d58483d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:19.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.579+0000 7f9039173740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d58484400 0x558d58486890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:19.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.579+0000 7f9039173740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558d583dbb10 con 0x558d58484400 2024-04-23T14:43:19.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.579+0000 7f9039173740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558d583822b0 con 0x558d5847e8a0 2024-04-23T14:43:19.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.579+0000 7f9039173740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558d583d7ad0 con 0x558d5847f290 2024-04-23T14:43:19.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.579+0000 7f902ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d58484400 0x558d58486890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:19.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.579+0000 7f902ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d58484400 0x558d58486890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53622/0 (socket says 172.21.15.28:53622) 2024-04-23T14:43:19.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.579+0000 7f902ea00640 1 -- 172.21.15.28:0/3194006271 learned_addr learned my addr 172.21.15.28:0/3194006271 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:19.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.579+0000 7f902f400640 1 --2- 172.21.15.28:0/3194006271 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d5847f290 0x558d58483d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:19.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.579+0000 7f902ea00640 1 --2- 172.21.15.28:0/3194006271 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d5847e8a0 0x558d5847ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:19.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.579+0000 7f902f400640 1 -- 172.21.15.28:0/3194006271 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d5847e8a0 msgr2=0x558d5847ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:43:19.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.579+0000 7f902f400640 1 --2- 172.21.15.28:0/3194006271 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d5847e8a0 0x558d5847ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.579+0000 7f902f400640 1 -- 172.21.15.28:0/3194006271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d58484400 msgr2=0x558d58486890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:43:19.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.579+0000 7f902f400640 1 --2- 172.21.15.28:0/3194006271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d58484400 0x558d58486890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.579+0000 7f902f400640 1 -- 172.21.15.28:0/3194006271 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d580ee2d0 con 0x558d5847f290 2024-04-23T14:43:19.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.579+0000 7f9027e00640 1 --2- 172.21.15.28:0/487571653 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f90140a3c30 0x7f90140a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:19.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.579+0000 7f902ea00640 1 --2- 172.21.15.28:0/3194006271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d58484400 0x558d58486890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:43:19.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.580+0000 7f9027e00640 1 -- 172.21.15.28:0/487571653 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7f90280d0730 con 0x7f9030105e10 2024-04-23T14:43:19.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.580+0000 7f902f400640 1 --2- 172.21.15.28:0/3194006271 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d5847f290 0x558d58483d90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9020009230 tx=0x7f9020005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca7e710a3c682b16 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:19.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.580+0000 7f902ea00640 1 --2- 172.21.15.28:0/487571653 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f90140a3c30 0x7f90140a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:19.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.580+0000 7f901be00640 1 -- 172.21.15.28:0/3194006271 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9020035040 con 0x558d5847f290 2024-04-23T14:43:19.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.580+0000 7f901be00640 1 -- 172.21.15.28:0/3194006271 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9020004490 con 0x558d5847f290 2024-04-23T14:43:19.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.580+0000 7f901be00640 1 -- 172.21.15.28:0/3194006271 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90200316b0 con 0x558d5847f290 2024-04-23T14:43:19.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.581+0000 7f9039173740 1 -- 172.21.15.28:0/3194006271 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d5847f290 msgr2=0x558d58483d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:19.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.581+0000 7f9039173740 1 --2- 172.21.15.28:0/3194006271 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d5847f290 0x558d58483d90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9020009230 tx=0x7f9020005c00 comp rx=0 tx=0).stop 2024-04-23T14:43:19.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.581+0000 7f902ea00640 1 --2- 172.21.15.28:0/487571653 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f90140a3c30 0x7f90140a60f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x558d5847faf0 tx=0x7f901c009c60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:19.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.581+0000 7f9039173740 1 -- 172.21.15.28:0/3194006271 shutdown_connections 2024-04-23T14:43:19.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.581+0000 7f9039173740 1 --2- 172.21.15.28:0/3194006271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d58484400 0x558d58486890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.581+0000 7f9039173740 1 --2- 172.21.15.28:0/3194006271 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d5847f290 0x558d58483d90 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.581+0000 7f9039173740 1 --2- 172.21.15.28:0/3194006271 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d5847e8a0 0x558d5847ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.581+0000 7f9039173740 1 -- 172.21.15.28:0/3194006271 >> 172.21.15.28:0/3194006271 conn(0x558d58471700 msgr2=0x558d58471b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:19.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.581+0000 7f9039173740 1 -- 172.21.15.28:0/3194006271 shutdown_connections 2024-04-23T14:43:19.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.581+0000 7f9039173740 1 -- 172.21.15.28:0/3194006271 wait complete. 2024-04-23T14:43:19.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.581+0000 7f9039173740 1 -- start start 2024-04-23T14:43:19.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.582+0000 7f9039173740 1 Processor -- start 2024-04-23T14:43:19.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.582+0000 7f9039173740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d5847e8a0 0x558d58488c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:19.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.582+0000 7f9039173740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d5847f290 0x558d584891d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:19.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.582+0000 7f9039173740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d58484400 0x558d58490210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:19.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.582+0000 7f9039173740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558d58487350 con 0x558d58484400 2024-04-23T14:43:19.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.582+0000 7f9039173740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558d58395590 con 0x558d5847f290 2024-04-23T14:43:19.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.582+0000 7f9039173740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558d584874d0 con 0x558d5847e8a0 2024-04-23T14:43:19.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.582+0000 7f9039173740 10 client.0 _ll_register_callbacks cb 0x558d583783a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:43:19.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.582+0000 7f902f400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d5847e8a0 0x558d58488c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:19.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.582+0000 7f902fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d58484400 0x558d58490210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:19.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.582+0000 7f902f400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d5847e8a0 0x558d58488c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:52368/0 (socket says 172.21.15.28:52368) 2024-04-23T14:43:19.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.582+0000 7f902f400640 1 -- 172.21.15.28:0/4109074889 learned_addr learned my addr 172.21.15.28:0/4109074889 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:19.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.583+0000 7f902ea00640 1 --2- 172.21.15.28:0/4109074889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d5847f290 0x558d584891d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:19.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.583+0000 7f902f400640 1 -- 172.21.15.28:0/4109074889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d5847f290 msgr2=0x558d584891d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:43:19.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.583+0000 7f902f400640 1 --2- 172.21.15.28:0/4109074889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d5847f290 0x558d584891d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.583+0000 7f902f400640 1 -- 172.21.15.28:0/4109074889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d58484400 msgr2=0x558d58490210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:43:19.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.583+0000 7f902f400640 1 --2- 172.21.15.28:0/4109074889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d58484400 0x558d58490210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.583+0000 7f902f400640 1 -- 172.21.15.28:0/4109074889 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d581b86b0 con 0x558d5847e8a0 2024-04-23T14:43:19.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.583+0000 7f902fe00640 1 --2- 172.21.15.28:0/4109074889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d58484400 0x558d58490210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:43:19.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.583+0000 7f902f400640 1 --2- 172.21.15.28:0/4109074889 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d5847e8a0 0x558d58488c90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f902002aa00 tx=0x7f9020009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15bf9a7bd57073ce server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:19.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.583+0000 7f9018c00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:43:19.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.583+0000 7f9018c00640 1 -- 172.21.15.28:0/4109074889 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9020053060 con 0x558d5847e8a0 2024-04-23T14:43:19.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.583+0000 7f9039173740 1 -- 172.21.15.28:0/4109074889 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9020004490 con 0x558d5847e8a0 2024-04-23T14:43:19.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.583+0000 7f9018c00640 1 -- 172.21.15.28:0/4109074889 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7f9020004290 con 0x558d5847e8a0 2024-04-23T14:43:19.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.584+0000 7f9018c00640 1 -- 172.21.15.28:0/4109074889 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f9020035050 con 0x558d5847e8a0 2024-04-23T14:43:19.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.584+0000 7f9039173740 10 client.25000 fetch_fsmap learned FSMap version 20 2024-04-23T14:43:19.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.584+0000 7f9039173740 1 -- 172.21.15.28:0/4109074889 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x558d584873a0 con 0x558d5847e8a0 2024-04-23T14:43:19.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.584+0000 7f9018c00640 1 -- 172.21.15.28:0/4109074889 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9020004490 con 0x558d5847e8a0 2024-04-23T14:43:19.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.584+0000 7f9018c00640 1 -- 172.21.15.28:0/4109074889 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f9020003ba0 con 0x558d5847e8a0 2024-04-23T14:43:19.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.584+0000 7f9039173740 10 client.25000 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:43:19.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.584+0000 7f9039173740 10 client.25000 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-23T14:43:19.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.584+0000 7f9039173740 20 client.25000 populate_metadata read hostname 'smithi028' 2024-04-23T14:43:19.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.584+0000 7f9039173740 1 --2- 172.21.15.28:0/4109074889 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558d58495740 0x558d58496bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:19.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.584+0000 7f9039173740 4 client.25000 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:43:19.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.584+0000 7f9039173740 1 -- 172.21.15.28:0/4109074889 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558d583e9530 con 0x558d58495740 2024-04-23T14:43:19.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.585+0000 7f902ea00640 1 --2- 172.21.15.28:0/4109074889 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558d58495740 0x558d58496bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.585+0000 7f902ea00640 1 --2- 172.21.15.28:0/4109074889 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558d58495740 0x558d58496bb0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eee6b4851d5049 server_cookie=1c80706876a65698 in_seq=0 out_seq=0 2024-04-23T14:43:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.585+0000 7f9018c00640 10 client.25000 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:43:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.586+0000 7f9018c00640 1 -- 172.21.15.28:0/4109074889 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558d583e9530 con 0x558d58495740 2024-04-23T14:43:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.586+0000 7f9018c00640 10 client.25000 handle_command_reply: tid=0 2024-04-23T14:43:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.586+0000 7f9039173740 1 client.25000 shutdown 2024-04-23T14:43:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.586+0000 7f9039173740 1 -- 172.21.15.28:0/4109074889 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d5847e8a0 msgr2=0x558d58488c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.586+0000 7f9039173740 1 --2- 172.21.15.28:0/4109074889 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d5847e8a0 0x558d58488c90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f902002aa00 tx=0x7f9020009260 comp rx=0 tx=0).stop 2024-04-23T14:43:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.586+0000 7f9039173740 1 -- 172.21.15.28:0/4109074889 shutdown_connections 2024-04-23T14:43:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.586+0000 7f9039173740 1 --2- 172.21.15.28:0/4109074889 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558d58495740 0x558d58496bb0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.586+0000 7f9039173740 1 --2- 172.21.15.28:0/4109074889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d5847f290 0x558d584891d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.586+0000 7f9039173740 1 --2- 172.21.15.28:0/4109074889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d58484400 0x558d58490210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.586+0000 7f9018c00640 0 client.25000 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:43:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.586+0000 7f9039173740 1 --2- 172.21.15.28:0/4109074889 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d5847e8a0 0x558d58488c90 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.586+0000 7f9039173740 1 -- 172.21.15.28:0/4109074889 >> 172.21.15.28:0/4109074889 conn(0x558d58471700 msgr2=0x558d584708c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.587+0000 7f9039173740 1 -- 172.21.15.28:0/4109074889 shutdown_connections 2024-04-23T14:43:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.587+0000 7f9039173740 1 -- 172.21.15.28:0/4109074889 wait complete. 2024-04-23T14:43:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.587+0000 7f9039173740 20 client.25000 trim_cache size 0 max 16384 2024-04-23T14:43:19.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.606+0000 7f9039173740 1 Processor -- start 2024-04-23T14:43:19.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.606+0000 7f9039173740 1 -- start start 2024-04-23T14:43:19.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.607+0000 7f9039173740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d5847e8a0 0x558d584745b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:19.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.607+0000 7f9039173740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d5847f290 0x558d58490130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:19.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.607+0000 7f9039173740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d58484400 0x558d58474af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:19.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.607+0000 7f9039173740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558d58494ef0 con 0x558d5847f290 2024-04-23T14:43:19.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.607+0000 7f9039173740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558d584955c0 con 0x558d58484400 2024-04-23T14:43:19.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.607+0000 7f9039173740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558d58495c90 con 0x558d5847e8a0 2024-04-23T14:43:19.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.607+0000 7f902ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d5847f290 0x558d58490130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:19.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.607+0000 7f902f400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d5847e8a0 0x558d584745b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:19.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.607+0000 7f902ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d5847f290 0x558d58490130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53650/0 (socket says 172.21.15.28:53650) 2024-04-23T14:43:19.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.607+0000 7f902ea00640 1 -- 172.21.15.28:0/717139657 learned_addr learned my addr 172.21.15.28:0/717139657 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:19.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.607+0000 7f902fe00640 1 --2- 172.21.15.28:0/717139657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d58484400 0x558d58474af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:19.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.607+0000 7f902ea00640 1 -- 172.21.15.28:0/717139657 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d5847e8a0 msgr2=0x558d584745b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:43:19.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.607+0000 7f902ea00640 1 --2- 172.21.15.28:0/717139657 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d5847e8a0 0x558d584745b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.607+0000 7f902ea00640 1 -- 172.21.15.28:0/717139657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d58484400 msgr2=0x558d58474af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:43:19.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.607+0000 7f902ea00640 1 --2- 172.21.15.28:0/717139657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d58484400 0x558d58474af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.607+0000 7f902ea00640 1 -- 172.21.15.28:0/717139657 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d58471530 con 0x558d5847f290 2024-04-23T14:43:19.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.607+0000 7f902f400640 1 --2- 172.21.15.28:0/717139657 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d5847e8a0 0x558d584745b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:43:19.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.608+0000 7f902ea00640 1 --2- 172.21.15.28:0/717139657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d5847f290 0x558d58490130 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x558d58488280 tx=0x7f901c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41b07310c7e3fe3f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:19.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.608+0000 7f901b400640 1 -- 172.21.15.28:0/717139657 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901c030070 con 0x558d5847f290 2024-04-23T14:43:19.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.608+0000 7f901b400640 1 -- 172.21.15.28:0/717139657 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f901c02b910 con 0x558d5847f290 2024-04-23T14:43:19.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.608+0000 7f901b400640 1 -- 172.21.15.28:0/717139657 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901c01e430 con 0x558d5847f290 2024-04-23T14:43:19.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.609+0000 7f9039173740 1 -- 172.21.15.28:0/717139657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d5847f290 msgr2=0x558d58490130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:19.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.609+0000 7f9039173740 1 --2- 172.21.15.28:0/717139657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d5847f290 0x558d58490130 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x558d58488280 tx=0x7f901c0046c0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.609+0000 7f9039173740 1 -- 172.21.15.28:0/717139657 shutdown_connections 2024-04-23T14:43:19.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.609+0000 7f9039173740 1 --2- 172.21.15.28:0/717139657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d58484400 0x558d58474af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.609+0000 7f9039173740 1 --2- 172.21.15.28:0/717139657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d5847f290 0x558d58490130 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.609+0000 7f9039173740 1 --2- 172.21.15.28:0/717139657 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d5847e8a0 0x558d584745b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.609+0000 7f9039173740 1 -- 172.21.15.28:0/717139657 >> 172.21.15.28:0/717139657 conn(0x558d58471700 msgr2=0x558d5846c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:19.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.609+0000 7f9039173740 1 -- 172.21.15.28:0/717139657 shutdown_connections 2024-04-23T14:43:19.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.609+0000 7f9039173740 1 -- 172.21.15.28:0/717139657 wait complete. 2024-04-23T14:43:19.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.609+0000 7f9039173740 1 -- start start 2024-04-23T14:43:19.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.609+0000 7f9039173740 1 Processor -- start 2024-04-23T14:43:19.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.610+0000 7f9039173740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d5847e8a0 0x558d5848c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:19.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.610+0000 7f9039173740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d5847f290 0x558d5848c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:19.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.610+0000 7f9039173740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d58484400 0x558d5848f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:19.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.610+0000 7f9039173740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558d58496a30 con 0x558d58484400 2024-04-23T14:43:19.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.610+0000 7f9039173740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558d58496360 con 0x558d5847f290 2024-04-23T14:43:19.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.610+0000 7f9039173740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558d58497100 con 0x558d5847e8a0 2024-04-23T14:43:19.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.610+0000 7f9039173740 10 client.0 _ll_register_callbacks cb 0x7f9020004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:43:19.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.610+0000 7f902fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d58484400 0x558d5848f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:19.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.610+0000 7f902ea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d5847f290 0x558d5848c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:19.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.610+0000 7f902fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d58484400 0x558d5848f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53656/0 (socket says 172.21.15.28:53656) 2024-04-23T14:43:19.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.610+0000 7f902fe00640 1 -- 172.21.15.28:0/2308781092 learned_addr learned my addr 172.21.15.28:0/2308781092 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:19.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.610+0000 7f902f400640 1 --2- 172.21.15.28:0/2308781092 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d5847e8a0 0x558d5848c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:19.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.611+0000 7f902fe00640 1 -- 172.21.15.28:0/2308781092 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d5847e8a0 msgr2=0x558d5848c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:43:19.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.611+0000 7f902fe00640 1 --2- 172.21.15.28:0/2308781092 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d5847e8a0 0x558d5848c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.611+0000 7f902fe00640 1 -- 172.21.15.28:0/2308781092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d5847f290 msgr2=0x558d5848c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:43:19.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.611+0000 7f902fe00640 1 --2- 172.21.15.28:0/2308781092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d5847f290 0x558d5848c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.611+0000 7f902fe00640 1 -- 172.21.15.28:0/2308781092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d5848fc50 con 0x558d58484400 2024-04-23T14:43:19.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.611+0000 7f902fe00640 1 --2- 172.21.15.28:0/2308781092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d58484400 0x558d5848f660 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f902800c640 tx=0x7f902809c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc5ea13d9fe0c875 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:19.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.611+0000 7f9018c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-23T14:43:19.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.611+0000 7f9018c00640 1 -- 172.21.15.28:0/2308781092 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f902801f040 con 0x558d58484400 2024-04-23T14:43:19.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.611+0000 7f9018c00640 1 -- 172.21.15.28:0/2308781092 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7f90280bbb90 con 0x558d58484400 2024-04-23T14:43:19.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.611+0000 7f9039173740 1 -- 172.21.15.28:0/2308781092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f901c02b910 con 0x558d58484400 2024-04-23T14:43:19.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.611+0000 7f9018c00640 1 -- 172.21.15.28:0/2308781092 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f90280a5640 con 0x558d58484400 2024-04-23T14:43:19.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.611+0000 7f9039173740 10 client.14868 fetch_fsmap learned FSMap version 20 2024-04-23T14:43:19.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.611+0000 7f9039173740 1 -- 172.21.15.28:0/2308781092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x558d584fbf20 con 0x558d58484400 2024-04-23T14:43:19.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.612+0000 7f9018c00640 1 -- 172.21.15.28:0/2308781092 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f901c02b910 con 0x558d58484400 2024-04-23T14:43:19.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.612+0000 7f9018c00640 1 -- 172.21.15.28:0/2308781092 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f902800f040 con 0x558d58484400 2024-04-23T14:43:19.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.612+0000 7f9039173740 10 client.14868 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:43:19.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.612+0000 7f9039173740 10 client.14868 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-23T14:43:19.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.612+0000 7f9039173740 20 client.14868 populate_metadata read hostname 'smithi028' 2024-04-23T14:43:19.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.612+0000 7f9039173740 1 --2- 172.21.15.28:0/2308781092 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558d58500b60 0x558d584fcfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:19.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.612+0000 7f9039173740 4 client.14868 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T14:43:19.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.612+0000 7f9039173740 1 -- 172.21.15.28:0/2308781092 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "status"}) -- 0x558d583e33a0 con 0x558d58500b60 2024-04-23T14:43:19.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.612+0000 7f902f400640 1 --2- 172.21.15.28:0/2308781092 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558d58500b60 0x558d584fcfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:19.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.613+0000 7f9018c00640 10 client.14868 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:43:19.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.613+0000 7f902f400640 1 --2- 172.21.15.28:0/2308781092 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558d58500b60 0x558d584fcfa0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a475d7a3576b7400 server_cookie=dc72110e17ee73 in_seq=0 out_seq=0 2024-04-23T14:43:19.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.613+0000 7f9018c00640 1 -- 172.21.15.28:0/2308781092 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x558d583e33a0 con 0x558d58500b60 2024-04-23T14:43:19.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.613+0000 7f9018c00640 10 client.14868 handle_command_reply: tid=0 2024-04-23T14:43:19.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.613+0000 7f9039173740 1 client.14868 shutdown 2024-04-23T14:43:19.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.613+0000 7f9039173740 1 -- 172.21.15.28:0/2308781092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d58484400 msgr2=0x558d5848f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:19.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.613+0000 7f9039173740 1 --2- 172.21.15.28:0/2308781092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d58484400 0x558d5848f660 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f902800c640 tx=0x7f902809c3e0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.613+0000 7f9039173740 1 -- 172.21.15.28:0/2308781092 shutdown_connections 2024-04-23T14:43:19.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.613+0000 7f9039173740 1 --2- 172.21.15.28:0/2308781092 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558d58500b60 0x558d584fcfa0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.613+0000 7f9039173740 1 --2- 172.21.15.28:0/2308781092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d5847f290 0x558d5848c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.613+0000 7f9039173740 1 --2- 172.21.15.28:0/2308781092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d58484400 0x558d5848f660 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.613+0000 7f9039173740 1 --2- 172.21.15.28:0/2308781092 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d5847e8a0 0x558d5848c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.613+0000 7f9039173740 1 -- 172.21.15.28:0/2308781092 >> 172.21.15.28:0/2308781092 conn(0x558d58471700 msgr2=0x558d58470030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:19.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.613+0000 7f9018c00640 0 client.14868 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:43:19.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.613+0000 7f9039173740 1 -- 172.21.15.28:0/2308781092 shutdown_connections 2024-04-23T14:43:19.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.613+0000 7f9039173740 1 -- 172.21.15.28:0/2308781092 wait complete. 2024-04-23T14:43:19.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.614+0000 7f9039173740 20 client.14868 trim_cache size 0 max 16384 2024-04-23T14:43:19.620 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:43:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "1ee6e996-017e-11ef-bc93-c7b262605968", 2024-04-23T14:43:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-04-23T14:43:19.620 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24587, 2024-04-23T14:43:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-23T14:43:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-23T14:43:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-23T14:43:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 99.623676086000003, 2024-04-23T14:43:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 20, 2024-04-23T14:43:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 87, 2024-04-23T14:43:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 85, 2024-04-23T14:43:19.621 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 104.76086700499999 2024-04-23T14:43:19.621 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:43:19.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.614+0000 7f9036200640 1 -- 172.21.15.28:0/487571653 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f90140a3c30 msgr2=0x7f90140a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:19.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.614+0000 7f9036200640 1 --2- 172.21.15.28:0/487571653 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f90140a3c30 0x7f90140a60f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x558d5847faf0 tx=0x7f901c009c60 comp rx=0 tx=0).stop 2024-04-23T14:43:19.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.614+0000 7f9036200640 1 -- 172.21.15.28:0/487571653 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9030105e10 msgr2=0x7f90301a1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:19.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.614+0000 7f9036200640 1 --2- 172.21.15.28:0/487571653 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9030105e10 0x7f90301a1b40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f902800e940 tx=0x7f902800ee00 comp rx=0 tx=0).stop 2024-04-23T14:43:19.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.614+0000 7f9036200640 1 -- 172.21.15.28:0/487571653 shutdown_connections 2024-04-23T14:43:19.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.614+0000 7f9036200640 1 --2- 172.21.15.28:0/487571653 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9030105e10 0x7f90301a1b40 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.614+0000 7f9036200640 1 --2- 172.21.15.28:0/487571653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9030105450 0x7f903019aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.614+0000 7f9036200640 1 --2- 172.21.15.28:0/487571653 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f90140a3c30 0x7f90140a60f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.614+0000 7f9036200640 1 --2- 172.21.15.28:0/487571653 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9030104260 0x7f903019a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:19.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.614+0000 7f9036200640 1 -- 172.21.15.28:0/487571653 >> 172.21.15.28:0/487571653 conn(0x7f90300ff7f0 msgr2=0x7f9030108820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:19.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.614+0000 7f9036200640 1 -- 172.21.15.28:0/487571653 shutdown_connections 2024-04-23T14:43:19.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:19.614+0000 7f9036200640 1 -- 172.21.15.28:0/487571653 wait complete. 2024-04-23T14:43:19.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:19 smithi028 ceph-mon[27810]: pgmap v365: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-23T14:43:19.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:19 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1264815913' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:43:19.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:19 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2815384429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:43:19.632 INFO:tasks.cephadm:Setup complete, yielding 2024-04-23T14:43:19.632 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-23T14:43:19.642 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-04-23T14:43:19.643 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-23T14:43:19.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:19 smithi105 ceph-mon[32555]: pgmap v365: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-23T14:43:19.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:19 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1264815913' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:43:19.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:19 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2815384429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:43:19.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:19 smithi038 ceph-mon[32348]: pgmap v365: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-23T14:43:19.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:19 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1264815913' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:43:19.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:19 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2815384429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:43:20.080 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:43:21.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:21 smithi105 ceph-mon[32555]: pgmap v366: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-23T14:43:21.818 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:21 smithi028 ceph-mon[27810]: pgmap v366: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-23T14:43:21.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:21 smithi038 ceph-mon[32348]: pgmap v366: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-23T14:43:22.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.386+0000 7f4081600700 1 -- 172.21.15.28:0/2181048225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f407c0786f0 msgr2=0x7f407c078b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:22.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.386+0000 7f4081600700 1 --2- 172.21.15.28:0/2181048225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f407c0786f0 0x7f407c078b70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4068009990 tx=0x7f4068009ca0 comp rx=0 tx=0).stop 2024-04-23T14:43:22.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.387+0000 7f4081600700 1 -- 172.21.15.28:0/2181048225 shutdown_connections 2024-04-23T14:43:22.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.387+0000 7f4081600700 1 --2- 172.21.15.28:0/2181048225 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f407c0790b0 0x7f407c079930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:22.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.387+0000 7f4081600700 1 --2- 172.21.15.28:0/2181048225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f407c0774a0 0x7f407c0778c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:22.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.387+0000 7f4081600700 1 --2- 172.21.15.28:0/2181048225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f407c0786f0 0x7f407c078b70 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:22.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.387+0000 7f4081600700 1 -- 172.21.15.28:0/2181048225 >> 172.21.15.28:0/2181048225 conn(0x7f407c100300 msgr2=0x7f407c102740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:22.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.387+0000 7f4081600700 1 -- 172.21.15.28:0/2181048225 shutdown_connections 2024-04-23T14:43:22.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.387+0000 7f4081600700 1 -- 172.21.15.28:0/2181048225 wait complete. 2024-04-23T14:43:22.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.388+0000 7f4081600700 1 Processor -- start 2024-04-23T14:43:22.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.388+0000 7f4081600700 1 -- start start 2024-04-23T14:43:22.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.388+0000 7f4081600700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f407c0774a0 0x7f407c19f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:22.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.388+0000 7f4081600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f407c0786f0 0x7f407c19f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:22.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.388+0000 7f4081600700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f407c0790b0 0x7f407c1a68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:22.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.388+0000 7f4081600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f407c114d60 con 0x7f407c0786f0 2024-04-23T14:43:22.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.388+0000 7f4081600700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f407c114be0 con 0x7f407c0790b0 2024-04-23T14:43:22.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.388+0000 7f4081600700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f407c114ee0 con 0x7f407c0774a0 2024-04-23T14:43:22.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.389+0000 7f407aa00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f407c0774a0 0x7f407c19f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:22.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.389+0000 7f407aa00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f407c0774a0 0x7f407c19f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:52412/0 (socket says 172.21.15.28:52412) 2024-04-23T14:43:22.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.389+0000 7f407aa00700 1 -- 172.21.15.28:0/3188568845 learned_addr learned my addr 172.21.15.28:0/3188568845 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:22.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.389+0000 7f407a000700 1 --2- 172.21.15.28:0/3188568845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f407c0786f0 0x7f407c19f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:22.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.389+0000 7f407b400700 1 --2- 172.21.15.28:0/3188568845 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f407c0790b0 0x7f407c1a68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:22.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.389+0000 7f407aa00700 1 -- 172.21.15.28:0/3188568845 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f407c0790b0 msgr2=0x7f407c1a68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:22.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.389+0000 7f407aa00700 1 --2- 172.21.15.28:0/3188568845 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f407c0790b0 0x7f407c1a68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:22.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.389+0000 7f407aa00700 1 -- 172.21.15.28:0/3188568845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f407c0786f0 msgr2=0x7f407c19f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:22.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.389+0000 7f407aa00700 1 --2- 172.21.15.28:0/3188568845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f407c0786f0 0x7f407c19f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:22.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.389+0000 7f407aa00700 1 -- 172.21.15.28:0/3188568845 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f407c1a6f80 con 0x7f407c0774a0 2024-04-23T14:43:22.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.390+0000 7f407aa00700 1 --2- 172.21.15.28:0/3188568845 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f407c0774a0 0x7f407c19f320 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f406c00e9d0 tx=0x7f406c00ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:22.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.390+0000 7f4073400700 1 -- 172.21.15.28:0/3188568845 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f406c00cbd0 con 0x7f407c0774a0 2024-04-23T14:43:22.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.390+0000 7f4081600700 1 -- 172.21.15.28:0/3188568845 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f407c1a7270 con 0x7f407c0774a0 2024-04-23T14:43:22.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.390+0000 7f4073400700 1 -- 172.21.15.28:0/3188568845 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f406c00cd70 con 0x7f407c0774a0 2024-04-23T14:43:22.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.390+0000 7f4073400700 1 -- 172.21.15.28:0/3188568845 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f406c018660 con 0x7f407c0774a0 2024-04-23T14:43:22.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.390+0000 7f4081600700 1 -- 172.21.15.28:0/3188568845 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f407c1a77b0 con 0x7f407c0774a0 2024-04-23T14:43:22.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.391+0000 7f4081600700 1 -- 172.21.15.28:0/3188568845 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f407c0697e0 con 0x7f407c0774a0 2024-04-23T14:43:22.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.393+0000 7f4073400700 1 -- 172.21.15.28:0/3188568845 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f406c018840 con 0x7f407c0774a0 2024-04-23T14:43:22.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.395+0000 7f4073400700 1 --2- 172.21.15.28:0/3188568845 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f40600a3c40 0x7f40600a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:22.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.395+0000 7f4073400700 1 -- 172.21.15.28:0/3188568845 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7f406c0d06b0 con 0x7f407c0774a0 2024-04-23T14:43:22.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.395+0000 7f407a000700 1 --2- 172.21.15.28:0/3188568845 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f40600a3c40 0x7f40600a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:22.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.396+0000 7f407a000700 1 --2- 172.21.15.28:0/3188568845 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f40600a3c40 0x7f40600a6100 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f406800b8e0 tx=0x7f406801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:22.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.397+0000 7f4073400700 1 -- 172.21.15.28:0/3188568845 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f406c09d050 con 0x7f407c0774a0 2024-04-23T14:43:22.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.654+0000 7f4081600700 1 -- 172.21.15.28:0/3188568845 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f407c062f30 con 0x7f40600a3c40 2024-04-23T14:43:22.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.656+0000 7f4073400700 1 -- 172.21.15.28:0/3188568845 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f407c062f30 con 0x7f40600a3c40 2024-04-23T14:43:22.658 INFO:teuthology.orchestra.run.smithi028.stdout:Backend: cephadm 2024-04-23T14:43:22.658 INFO:teuthology.orchestra.run.smithi028.stdout:Available: Yes 2024-04-23T14:43:22.658 INFO:teuthology.orchestra.run.smithi028.stdout:Paused: No 2024-04-23T14:43:22.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.659+0000 7f4081600700 1 -- 172.21.15.28:0/3188568845 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f40600a3c40 msgr2=0x7f40600a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:22.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.659+0000 7f4081600700 1 --2- 172.21.15.28:0/3188568845 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f40600a3c40 0x7f40600a6100 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f406800b8e0 tx=0x7f406801a040 comp rx=0 tx=0).stop 2024-04-23T14:43:22.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.659+0000 7f4081600700 1 -- 172.21.15.28:0/3188568845 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f407c0774a0 msgr2=0x7f407c19f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:22.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.659+0000 7f4081600700 1 --2- 172.21.15.28:0/3188568845 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f407c0774a0 0x7f407c19f320 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f406c00e9d0 tx=0x7f406c00ece0 comp rx=0 tx=0).stop 2024-04-23T14:43:22.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.659+0000 7f4081600700 1 -- 172.21.15.28:0/3188568845 shutdown_connections 2024-04-23T14:43:22.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.659+0000 7f4081600700 1 --2- 172.21.15.28:0/3188568845 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f40600a3c40 0x7f40600a6100 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:22.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.659+0000 7f4081600700 1 --2- 172.21.15.28:0/3188568845 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f407c0774a0 0x7f407c19f320 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:22.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.659+0000 7f4081600700 1 --2- 172.21.15.28:0/3188568845 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f407c0790b0 0x7f407c1a68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:22.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.659+0000 7f4081600700 1 --2- 172.21.15.28:0/3188568845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f407c0786f0 0x7f407c19f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:22.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.659+0000 7f4081600700 1 -- 172.21.15.28:0/3188568845 >> 172.21.15.28:0/3188568845 conn(0x7f407c100300 msgr2=0x7f407c101e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:22.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.660+0000 7f4081600700 1 -- 172.21.15.28:0/3188568845 shutdown_connections 2024-04-23T14:43:22.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:22.660+0000 7f4081600700 1 -- 172.21.15.28:0/3188568845 wait complete. 2024-04-23T14:43:23.526 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-23T14:43:23.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:23 smithi105 ceph-mon[32555]: pgmap v367: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-23T14:43:23.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:23 smithi038 ceph-mon[32348]: pgmap v367: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-23T14:43:23.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:23 smithi028 ceph-mon[27810]: pgmap v367: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-23T14:43:23.989 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:43:24.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:24 smithi028 ceph-mon[27810]: from='client.25012 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:43:24.789 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:24 smithi105 ceph-mon[32555]: from='client.25012 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:43:24.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:24 smithi038 ceph-mon[32348]: from='client.25012 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:43:25.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:25 smithi105 ceph-mon[32555]: pgmap v368: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-23T14:43:25.850 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:25 smithi028 ceph-mon[27810]: pgmap v368: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-23T14:43:25.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:25 smithi038 ceph-mon[32348]: pgmap v368: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-23T14:43:26.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.026+0000 7efeed000700 1 -- 172.21.15.28:0/2534881448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efee8105f00 msgr2=0x7efee81082f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:26.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.026+0000 7efeed000700 1 --2- 172.21.15.28:0/2534881448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efee8105f00 0x7efee81082f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7efed0009990 tx=0x7efed0009ca0 comp rx=0 tx=0).stop 2024-04-23T14:43:26.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.026+0000 7efeed000700 1 -- 172.21.15.28:0/2534881448 shutdown_connections 2024-04-23T14:43:26.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.026+0000 7efeed000700 1 --2- 172.21.15.28:0/2534881448 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efee81035d0 0x7efee81059c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:26.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.026+0000 7efeed000700 1 --2- 172.21.15.28:0/2534881448 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efee8108830 0x7efee810aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:26.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.026+0000 7efeed000700 1 --2- 172.21.15.28:0/2534881448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efee8105f00 0x7efee81082f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:26.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.026+0000 7efeed000700 1 -- 172.21.15.28:0/2534881448 >> 172.21.15.28:0/2534881448 conn(0x7efee80fcfe0 msgr2=0x7efee80ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:26.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.026+0000 7efeed000700 1 -- 172.21.15.28:0/2534881448 shutdown_connections 2024-04-23T14:43:26.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.026+0000 7efeed000700 1 -- 172.21.15.28:0/2534881448 wait complete. 2024-04-23T14:43:26.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.027+0000 7efeed000700 1 Processor -- start 2024-04-23T14:43:26.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.027+0000 7efeed000700 1 -- start start 2024-04-23T14:43:26.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.027+0000 7efeed000700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efee81035d0 0x7efee819afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:26.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.027+0000 7efeed000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efee8105f00 0x7efee819b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:26.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.027+0000 7efeed000700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efee8108830 0x7efee81a2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:26.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.027+0000 7efeed000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efee810e770 con 0x7efee8105f00 2024-04-23T14:43:26.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.027+0000 7efeed000700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efee810e5f0 con 0x7efee81035d0 2024-04-23T14:43:26.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.027+0000 7efeed000700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7efee810e8f0 con 0x7efee8108830 2024-04-23T14:43:26.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.028+0000 7efee5e00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efee81035d0 0x7efee819afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:26.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.028+0000 7efee5e00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efee81035d0 0x7efee819afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:34054/0 (socket says 172.21.15.28:34054) 2024-04-23T14:43:26.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.028+0000 7efedd400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efee8105f00 0x7efee819b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:26.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.028+0000 7efee5e00700 1 -- 172.21.15.28:0/1521750792 learned_addr learned my addr 172.21.15.28:0/1521750792 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:26.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.028+0000 7efee6800700 1 --2- 172.21.15.28:0/1521750792 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efee8108830 0x7efee81a2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:26.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.028+0000 7efee5e00700 1 -- 172.21.15.28:0/1521750792 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efee8108830 msgr2=0x7efee81a2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:26.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.028+0000 7efee5e00700 1 --2- 172.21.15.28:0/1521750792 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efee8108830 0x7efee81a2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:26.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.028+0000 7efee5e00700 1 -- 172.21.15.28:0/1521750792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efee8105f00 msgr2=0x7efee819b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:26.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.028+0000 7efee5e00700 1 --2- 172.21.15.28:0/1521750792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efee8105f00 0x7efee819b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:26.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.028+0000 7efee5e00700 1 -- 172.21.15.28:0/1521750792 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efee81a2c30 con 0x7efee81035d0 2024-04-23T14:43:26.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.028+0000 7efee5e00700 1 --2- 172.21.15.28:0/1521750792 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efee81035d0 0x7efee819afd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7efed800e9a0 tx=0x7efed800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:26.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.029+0000 7efedf400700 1 -- 172.21.15.28:0/1521750792 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed800ca10 con 0x7efee81035d0 2024-04-23T14:43:26.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.029+0000 7efedf400700 1 -- 172.21.15.28:0/1521750792 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efed800cbb0 con 0x7efee81035d0 2024-04-23T14:43:26.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.029+0000 7efedf400700 1 -- 172.21.15.28:0/1521750792 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed8018710 con 0x7efee81035d0 2024-04-23T14:43:26.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.029+0000 7efeed000700 1 -- 172.21.15.28:0/1521750792 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efee81a2f20 con 0x7efee81035d0 2024-04-23T14:43:26.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.029+0000 7efeed000700 1 -- 172.21.15.28:0/1521750792 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efee81a32c0 con 0x7efee81035d0 2024-04-23T14:43:26.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.030+0000 7efeed000700 1 -- 172.21.15.28:0/1521750792 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efee80697e0 con 0x7efee81035d0 2024-04-23T14:43:26.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.032+0000 7efedf400700 1 -- 172.21.15.28:0/1521750792 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7efed80188b0 con 0x7efee81035d0 2024-04-23T14:43:26.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.033+0000 7efedf400700 1 --2- 172.21.15.28:0/1521750792 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7efecc0a3c40 0x7efecc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:26.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.034+0000 7efedf400700 1 -- 172.21.15.28:0/1521750792 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7efed80d0720 con 0x7efee81035d0 2024-04-23T14:43:26.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.034+0000 7efedd400700 1 --2- 172.21.15.28:0/1521750792 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7efecc0a3c40 0x7efecc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:26.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.035+0000 7efedd400700 1 --2- 172.21.15.28:0/1521750792 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7efecc0a3c40 0x7efecc0a6100 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7efed000b2e0 tx=0x7efed0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:26.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.036+0000 7efedf400700 1 -- 172.21.15.28:0/1521750792 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efed809d0c0 con 0x7efee81035d0 2024-04-23T14:43:26.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.278+0000 7efeed000700 1 -- 172.21.15.28:0/1521750792 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7efee8062f30 con 0x7efecc0a3c40 2024-04-23T14:43:26.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.302+0000 7efedf400700 1 -- 172.21.15.28:0/1521750792 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7efee8062f30 con 0x7efecc0a3c40 2024-04-23T14:43:26.302 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-23T14:43:26.302 INFO:teuthology.orchestra.run.smithi028.stdout:mds.a smithi028 running (114s) 23s ago 114s 12.2M - 19.0.0-3126-gd44fddd9 76e77f280c74 cd19ff722a9b 2024-04-23T14:43:26.302 INFO:teuthology.orchestra.run.smithi028.stdout:mds.b smithi038 running (111s) 43s ago 111s 25.5M - 19.0.0-3126-gd44fddd9 76e77f280c74 bb8d937f28ef 2024-04-23T14:43:26.303 INFO:teuthology.orchestra.run.smithi028.stdout:mds.c smithi105 running (2m) 43s ago 2m 13.0M - 19.0.0-3126-gd44fddd9 76e77f280c74 bca81fe91c8f 2024-04-23T14:43:26.303 INFO:teuthology.orchestra.run.smithi028.stdout:mds.d smithi028 running (2m) 23s ago 2m 12.5M - 19.0.0-3126-gd44fddd9 76e77f280c74 8bb57b364e12 2024-04-23T14:43:26.303 INFO:teuthology.orchestra.run.smithi028.stdout:mds.e smithi038 running (2m) 43s ago 2m 14.1M - 19.0.0-3126-gd44fddd9 76e77f280c74 e2530c3f5fe2 2024-04-23T14:43:26.303 INFO:teuthology.orchestra.run.smithi028.stdout:mds.f smithi105 running (2m) 43s ago 2m 16.4M - 19.0.0-3126-gd44fddd9 76e77f280c74 75e2c6354570 2024-04-23T14:43:26.303 INFO:teuthology.orchestra.run.smithi028.stdout:mds.g smithi028 running (118s) 23s ago 118s 16.8M - 19.0.0-3126-gd44fddd9 76e77f280c74 b54a6fd11f60 2024-04-23T14:43:26.303 INFO:teuthology.orchestra.run.smithi028.stdout:mds.h smithi038 running (2m) 43s ago 2m 16.2M - 19.0.0-3126-gd44fddd9 76e77f280c74 eea5e724a98b 2024-04-23T14:43:26.303 INFO:teuthology.orchestra.run.smithi028.stdout:mds.i smithi105 running (2m) 43s ago 2m 16.3M - 19.0.0-3126-gd44fddd9 76e77f280c74 56f412a338a7 2024-04-23T14:43:26.303 INFO:teuthology.orchestra.run.smithi028.stdout:mds.j smithi028 running (2m) 23s ago 2m 12.5M - 19.0.0-3126-gd44fddd9 76e77f280c74 77263f305569 2024-04-23T14:43:26.303 INFO:teuthology.orchestra.run.smithi028.stdout:mds.k smithi038 running (2m) 43s ago 2m 16.5M - 19.0.0-3126-gd44fddd9 76e77f280c74 2f2cbd0307e4 2024-04-23T14:43:26.303 INFO:teuthology.orchestra.run.smithi028.stdout:mds.l smithi105 running (2m) 43s ago 2m 15.4M - 19.0.0-3126-gd44fddd9 76e77f280c74 9622fdb37276 2024-04-23T14:43:26.303 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi028 *:9283,8765 running (10m) 23s ago 10m 515M - 19.0.0-3126-gd44fddd9 76e77f280c74 7f8a92032a8c 2024-04-23T14:43:26.303 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi038 *:8443,8765 running (8m) 43s ago 8m 454M - 19.0.0-3126-gd44fddd9 76e77f280c74 bf6cd51d0f02 2024-04-23T14:43:26.303 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.z smithi105 *:8443,8765 running (8m) 43s ago 8m 454M - 19.0.0-3126-gd44fddd9 76e77f280c74 afe82dbc72f2 2024-04-23T14:43:26.303 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (10m) 23s ago 10m 48.2M 2048M 19.0.0-3126-gd44fddd9 76e77f280c74 4e1b4802bb2e 2024-04-23T14:43:26.303 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi038 running (8m) 43s ago 8m 41.8M 2048M 19.0.0-3126-gd44fddd9 76e77f280c74 8a2678fe19a1 2024-04-23T14:43:26.304 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi105 running (8m) 43s ago 8m 42.2M 2048M 19.0.0-3126-gd44fddd9 76e77f280c74 4684ef5a77bf 2024-04-23T14:43:26.304 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (7m) 23s ago 7m 54.4M 4273M 19.0.0-3126-gd44fddd9 76e77f280c74 8a704935cdbc 2024-04-23T14:43:26.304 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi038 running (7m) 43s ago 7m 59.5M 4273M 19.0.0-3126-gd44fddd9 76e77f280c74 eaaea8b5b820 2024-04-23T14:43:26.304 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi105 running (7m) 43s ago 7m 57.4M 4272M 19.0.0-3126-gd44fddd9 76e77f280c74 90577d471727 2024-04-23T14:43:26.304 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (6m) 23s ago 6m 45.2M 4273M 19.0.0-3126-gd44fddd9 76e77f280c74 3b027f125e87 2024-04-23T14:43:26.304 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi038 running (6m) 43s ago 6m 52.5M 4273M 19.0.0-3126-gd44fddd9 76e77f280c74 2b2f2acf8612 2024-04-23T14:43:26.304 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi105 running (5m) 43s ago 5m 46.9M 4272M 19.0.0-3126-gd44fddd9 76e77f280c74 703b42a945a9 2024-04-23T14:43:26.304 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi028 running (5m) 23s ago 5m 43.7M 4273M 19.0.0-3126-gd44fddd9 76e77f280c74 4027f1fa6f09 2024-04-23T14:43:26.304 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi038 running (4m) 43s ago 4m 54.8M 4273M 19.0.0-3126-gd44fddd9 76e77f280c74 aa1a593aa447 2024-04-23T14:43:26.304 INFO:teuthology.orchestra.run.smithi028.stdout:osd.8 smithi105 running (4m) 43s ago 4m 46.0M 4272M 19.0.0-3126-gd44fddd9 76e77f280c74 19c61452163f 2024-04-23T14:43:26.304 INFO:teuthology.orchestra.run.smithi028.stdout:osd.9 smithi028 running (3m) 23s ago 3m 49.4M 4273M 19.0.0-3126-gd44fddd9 76e77f280c74 60a654a97fd8 2024-04-23T14:43:26.304 INFO:teuthology.orchestra.run.smithi028.stdout:osd.10 smithi038 running (3m) 43s ago 3m 48.8M 4273M 19.0.0-3126-gd44fddd9 76e77f280c74 1f16cac69cb9 2024-04-23T14:43:26.304 INFO:teuthology.orchestra.run.smithi028.stdout:osd.11 smithi105 running (2m) 43s ago 2m 45.6M 4272M 19.0.0-3126-gd44fddd9 76e77f280c74 4859788ddf79 2024-04-23T14:43:26.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.305+0000 7efeed000700 1 -- 172.21.15.28:0/1521750792 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7efecc0a3c40 msgr2=0x7efecc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:26.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.305+0000 7efeed000700 1 --2- 172.21.15.28:0/1521750792 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7efecc0a3c40 0x7efecc0a6100 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7efed000b2e0 tx=0x7efed0005fb0 comp rx=0 tx=0).stop 2024-04-23T14:43:26.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.305+0000 7efeed000700 1 -- 172.21.15.28:0/1521750792 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efee81035d0 msgr2=0x7efee819afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:26.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.305+0000 7efeed000700 1 --2- 172.21.15.28:0/1521750792 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efee81035d0 0x7efee819afd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7efed800e9a0 tx=0x7efed800ecb0 comp rx=0 tx=0).stop 2024-04-23T14:43:26.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.305+0000 7efeed000700 1 -- 172.21.15.28:0/1521750792 shutdown_connections 2024-04-23T14:43:26.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.305+0000 7efeed000700 1 --2- 172.21.15.28:0/1521750792 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7efecc0a3c40 0x7efecc0a6100 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:26.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.305+0000 7efeed000700 1 --2- 172.21.15.28:0/1521750792 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efee8108830 0x7efee81a2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:26.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.305+0000 7efeed000700 1 --2- 172.21.15.28:0/1521750792 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efee81035d0 0x7efee819afd0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:26.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.305+0000 7efeed000700 1 --2- 172.21.15.28:0/1521750792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efee8105f00 0x7efee819b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:26.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.305+0000 7efeed000700 1 -- 172.21.15.28:0/1521750792 >> 172.21.15.28:0/1521750792 conn(0x7efee80fcfe0 msgr2=0x7efee8102d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:26.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.305+0000 7efeed000700 1 -- 172.21.15.28:0/1521750792 shutdown_connections 2024-04-23T14:43:26.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:26.305+0000 7efeed000700 1 -- 172.21.15.28:0/1521750792 wait complete. 2024-04-23T14:43:27.008 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-23T14:43:27.474 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:43:27.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:27 smithi105 ceph-mon[32555]: pgmap v369: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-23T14:43:27.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:27 smithi105 ceph-mon[32555]: from='client.25007 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:43:27.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:27 smithi028 ceph-mon[27810]: pgmap v369: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-23T14:43:27.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:27 smithi028 ceph-mon[27810]: from='client.25007 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:43:27.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:27 smithi038 ceph-mon[32348]: pgmap v369: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-23T14:43:27.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:27 smithi038 ceph-mon[32348]: from='client.25007 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:43:29.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.771+0000 7fd80f000700 1 -- 172.21.15.28:0/1483853148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd80810ad30 msgr2=0x7fd8081115e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:29.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.771+0000 7fd80f000700 1 --2- 172.21.15.28:0/1483853148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd80810ad30 0x7fd8081115e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd8000099c0 tx=0x7fd800009cd0 comp rx=0 tx=0).stop 2024-04-23T14:43:29.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.771+0000 7fd80f000700 1 -- 172.21.15.28:0/1483853148 shutdown_connections 2024-04-23T14:43:29.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.771+0000 7fd80f000700 1 --2- 172.21.15.28:0/1483853148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd80810ad30 0x7fd8081115e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:29.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.771+0000 7fd80f000700 1 --2- 172.21.15.28:0/1483853148 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd808109190 0x7fd8081095b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:29.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.771+0000 7fd80f000700 1 --2- 172.21.15.28:0/1483853148 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd80810a390 0x7fd80810a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:29.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.771+0000 7fd80f000700 1 -- 172.21.15.28:0/1483853148 >> 172.21.15.28:0/1483853148 conn(0x7fd808078160 msgr2=0x7fd80807a5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:29.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.771+0000 7fd80f000700 1 -- 172.21.15.28:0/1483853148 shutdown_connections 2024-04-23T14:43:29.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.771+0000 7fd80f000700 1 -- 172.21.15.28:0/1483853148 wait complete. 2024-04-23T14:43:29.773 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:29 smithi028 ceph-mon[27810]: pgmap v370: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-04-23T14:43:29.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.772+0000 7fd80f000700 1 Processor -- start 2024-04-23T14:43:29.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.772+0000 7fd80f000700 1 -- start start 2024-04-23T14:43:29.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.773+0000 7fd80f000700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd808109190 0x7fd80819f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:29.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.773+0000 7fd80f000700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd80810a390 0x7fd80819f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:29.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.773+0000 7fd80f000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd80810ad30 0x7fd8081a68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:29.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.773+0000 7fd80f000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd808114de0 con 0x7fd80810ad30 2024-04-23T14:43:29.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.773+0000 7fd80f000700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd808114c60 con 0x7fd808109190 2024-04-23T14:43:29.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.773+0000 7fd80f000700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd808114f60 con 0x7fd80810a390 2024-04-23T14:43:29.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.773+0000 7fd80d000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd80810ad30 0x7fd8081a68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:29.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.773+0000 7fd807e00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd808109190 0x7fd80819f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:29.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.773+0000 7fd7ff400700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd80810a390 0x7fd80819f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:29.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.773+0000 7fd80d000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd80810ad30 0x7fd8081a68c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46258/0 (socket says 172.21.15.28:46258) 2024-04-23T14:43:29.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.773+0000 7fd80d000700 1 -- 172.21.15.28:0/1541546137 learned_addr learned my addr 172.21.15.28:0/1541546137 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:29.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.774+0000 7fd807e00700 1 -- 172.21.15.28:0/1541546137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd80810a390 msgr2=0x7fd80819f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:29.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.774+0000 7fd807e00700 1 --2- 172.21.15.28:0/1541546137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd80810a390 0x7fd80819f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:29.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.774+0000 7fd807e00700 1 -- 172.21.15.28:0/1541546137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd80810ad30 msgr2=0x7fd8081a68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:29.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.774+0000 7fd807e00700 1 --2- 172.21.15.28:0/1541546137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd80810ad30 0x7fd8081a68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:29.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.774+0000 7fd807e00700 1 -- 172.21.15.28:0/1541546137 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8081a6f60 con 0x7fd808109190 2024-04-23T14:43:29.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.774+0000 7fd807e00700 1 --2- 172.21.15.28:0/1541546137 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd808109190 0x7fd80819f300 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd7f800e9c0 tx=0x7fd7f800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:29.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.774+0000 7fd805600700 1 -- 172.21.15.28:0/1541546137 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f8004220 con 0x7fd808109190 2024-04-23T14:43:29.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.774+0000 7fd80f000700 1 -- 172.21.15.28:0/1541546137 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8081a7250 con 0x7fd808109190 2024-04-23T14:43:29.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.775+0000 7fd805600700 1 -- 172.21.15.28:0/1541546137 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7f80043c0 con 0x7fd808109190 2024-04-23T14:43:29.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.775+0000 7fd805600700 1 -- 172.21.15.28:0/1541546137 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f8003ce0 con 0x7fd808109190 2024-04-23T14:43:29.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.775+0000 7fd80f000700 1 -- 172.21.15.28:0/1541546137 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8081a77e0 con 0x7fd808109190 2024-04-23T14:43:29.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.776+0000 7fd80f000700 1 -- 172.21.15.28:0/1541546137 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8080697e0 con 0x7fd808109190 2024-04-23T14:43:29.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.778+0000 7fd805600700 1 -- 172.21.15.28:0/1541546137 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fd7f800c7c0 con 0x7fd808109190 2024-04-23T14:43:29.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.779+0000 7fd805600700 1 --2- 172.21.15.28:0/1541546137 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd7ec0a3cc0 0x7fd7ec0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:29.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.779+0000 7fd805600700 1 -- 172.21.15.28:0/1541546137 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7fd7f80d05d0 con 0x7fd808109190 2024-04-23T14:43:29.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.779+0000 7fd7ff400700 1 --2- 172.21.15.28:0/1541546137 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd7ec0a3cc0 0x7fd7ec0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:29.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.780+0000 7fd7ff400700 1 --2- 172.21.15.28:0/1541546137 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd7ec0a3cc0 0x7fd7ec0a6180 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd7f000f540 tx=0x7fd7f000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:29.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:29.784+0000 7fd805600700 1 -- 172.21.15.28:0/1541546137 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd7f8016330 con 0x7fd808109190 2024-04-23T14:43:29.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:29 smithi105 ceph-mon[32555]: pgmap v370: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-04-23T14:43:29.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:29 smithi038 ceph-mon[32348]: pgmap v370: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-04-23T14:43:30.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:30.025+0000 7fd80f000700 1 -- 172.21.15.28:0/1541546137 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fd808062f30 con 0x7fd7ec0a3cc0 2024-04-23T14:43:30.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:30.042+0000 7fd805600700 1 -- 172.21.15.28:0/1541546137 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fd808062f30 con 0x7fd7ec0a3cc0 2024-04-23T14:43:30.043 INFO:teuthology.orchestra.run.smithi028.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-23T14:43:30.043 INFO:teuthology.orchestra.run.smithi028.stdout:mds.all 12/12 46s ago 2m smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi105=c;smithi105=f;smithi105=i;smithi105=l;count:12 2024-04-23T14:43:30.043 INFO:teuthology.orchestra.run.smithi028.stdout:mgr 3/3 46s ago 8m smithi028=x;smithi038=y;smithi105=z;count:3 2024-04-23T14:43:30.043 INFO:teuthology.orchestra.run.smithi028.stdout:mon 3/3 46s ago 8m smithi028:172.21.15.28=a;smithi038:172.21.15.38=b;smithi105:172.21.15.105=c;count:3 2024-04-23T14:43:30.043 INFO:teuthology.orchestra.run.smithi028.stdout:osd 12 46s ago - 2024-04-23T14:43:30.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:30.045+0000 7fd80f000700 1 -- 172.21.15.28:0/1541546137 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd7ec0a3cc0 msgr2=0x7fd7ec0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:30.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:30.045+0000 7fd80f000700 1 --2- 172.21.15.28:0/1541546137 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd7ec0a3cc0 0x7fd7ec0a6180 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd7f000f540 tx=0x7fd7f000b040 comp rx=0 tx=0).stop 2024-04-23T14:43:30.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:30.045+0000 7fd80f000700 1 -- 172.21.15.28:0/1541546137 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd808109190 msgr2=0x7fd80819f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:30.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:30.045+0000 7fd80f000700 1 --2- 172.21.15.28:0/1541546137 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd808109190 0x7fd80819f300 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd7f800e9c0 tx=0x7fd7f800ecd0 comp rx=0 tx=0).stop 2024-04-23T14:43:30.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:30.046+0000 7fd80f000700 1 -- 172.21.15.28:0/1541546137 shutdown_connections 2024-04-23T14:43:30.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:30.046+0000 7fd80f000700 1 --2- 172.21.15.28:0/1541546137 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd7ec0a3cc0 0x7fd7ec0a6180 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:30.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:30.046+0000 7fd80f000700 1 --2- 172.21.15.28:0/1541546137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd80810ad30 0x7fd8081a68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:30.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:30.046+0000 7fd80f000700 1 --2- 172.21.15.28:0/1541546137 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd808109190 0x7fd80819f300 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:30.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:30.046+0000 7fd80f000700 1 --2- 172.21.15.28:0/1541546137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd80810a390 0x7fd80819f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:30.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:30.046+0000 7fd80f000700 1 -- 172.21.15.28:0/1541546137 >> 172.21.15.28:0/1541546137 conn(0x7fd808078160 msgr2=0x7fd80810d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:30.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:30.046+0000 7fd80f000700 1 -- 172.21.15.28:0/1541546137 shutdown_connections 2024-04-23T14:43:30.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:30.046+0000 7fd80f000700 1 -- 172.21.15.28:0/1541546137 wait complete. 2024-04-23T14:43:30.795 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-23T14:43:31.233 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:43:31.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:31 smithi028 ceph-mon[27810]: pgmap v371: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:43:31.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:31 smithi028 ceph-mon[27810]: from='client.25013 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:43:31.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:31 smithi105 ceph-mon[32555]: pgmap v371: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:43:31.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:31 smithi105 ceph-mon[32555]: from='client.25013 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:43:31.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:31 smithi038 ceph-mon[32348]: pgmap v371: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:43:31.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:31 smithi038 ceph-mon[32348]: from='client.25013 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:43:33.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.252+0000 7feb44400700 1 -- 172.21.15.28:0/330551805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb3c106710 msgr2=0x7feb3c106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:33.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.252+0000 7feb44400700 1 --2- 172.21.15.28:0/330551805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb3c106710 0x7feb3c106b90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7feb3400b0a0 tx=0x7feb3400b3b0 comp rx=0 tx=0).stop 2024-04-23T14:43:33.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.252+0000 7feb44400700 1 -- 172.21.15.28:0/330551805 shutdown_connections 2024-04-23T14:43:33.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.252+0000 7feb44400700 1 --2- 172.21.15.28:0/330551805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb3c106710 0x7feb3c106b90 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:33.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.252+0000 7feb44400700 1 --2- 172.21.15.28:0/330551805 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb3c1013d0 0x7feb3c103800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:33.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.252+0000 7feb44400700 1 --2- 172.21.15.28:0/330551805 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feb3c103d40 0x7feb3c1061d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:33.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.252+0000 7feb44400700 1 -- 172.21.15.28:0/330551805 >> 172.21.15.28:0/330551805 conn(0x7feb3c0fcff0 msgr2=0x7feb3c0ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:33.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.252+0000 7feb44400700 1 -- 172.21.15.28:0/330551805 shutdown_connections 2024-04-23T14:43:33.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.252+0000 7feb44400700 1 -- 172.21.15.28:0/330551805 wait complete. 2024-04-23T14:43:33.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.253+0000 7feb44400700 1 Processor -- start 2024-04-23T14:43:33.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.253+0000 7feb44400700 1 -- start start 2024-04-23T14:43:33.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.254+0000 7feb44400700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feb3c1013d0 0x7feb3c19f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:33.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.254+0000 7feb44400700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb3c103d40 0x7feb3c19f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:33.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.254+0000 7feb44400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb3c106710 0x7feb3c1a6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:33.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.254+0000 7feb44400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7feb3c114ca0 con 0x7feb3c106710 2024-04-23T14:43:33.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.254+0000 7feb44400700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7feb3c114b20 con 0x7feb3c103d40 2024-04-23T14:43:33.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.254+0000 7feb44400700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7feb3c114e20 con 0x7feb3c1013d0 2024-04-23T14:43:33.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.254+0000 7feb42400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb3c106710 0x7feb3c1a6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:33.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.254+0000 7feb41a00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feb3c1013d0 0x7feb3c19f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:33.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.254+0000 7feb42400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb3c106710 0x7feb3c1a6880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46264/0 (socket says 172.21.15.28:46264) 2024-04-23T14:43:33.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.254+0000 7feb42400700 1 -- 172.21.15.28:0/684460805 learned_addr learned my addr 172.21.15.28:0/684460805 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:33.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.254+0000 7feb41000700 1 --2- 172.21.15.28:0/684460805 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb3c103d40 0x7feb3c19f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:33.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.255+0000 7feb42400700 1 -- 172.21.15.28:0/684460805 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feb3c1013d0 msgr2=0x7feb3c19f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:33.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.255+0000 7feb42400700 1 --2- 172.21.15.28:0/684460805 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feb3c1013d0 0x7feb3c19f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:33.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.255+0000 7feb42400700 1 -- 172.21.15.28:0/684460805 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb3c103d40 msgr2=0x7feb3c19f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:33.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.255+0000 7feb42400700 1 --2- 172.21.15.28:0/684460805 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb3c103d40 0x7feb3c19f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:33.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.255+0000 7feb42400700 1 -- 172.21.15.28:0/684460805 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb3c1a6f20 con 0x7feb3c106710 2024-04-23T14:43:33.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.255+0000 7feb41000700 1 --2- 172.21.15.28:0/684460805 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb3c103d40 0x7feb3c19f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:43:33.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.255+0000 7feb42400700 1 --2- 172.21.15.28:0/684460805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb3c106710 0x7feb3c1a6880 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7feb34004f40 tx=0x7feb34004f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:33.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.255+0000 7feb3a000700 1 -- 172.21.15.28:0/684460805 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb34021070 con 0x7feb3c106710 2024-04-23T14:43:33.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.255+0000 7feb3a000700 1 -- 172.21.15.28:0/684460805 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb34007b80 con 0x7feb3c106710 2024-04-23T14:43:33.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.255+0000 7feb44400700 1 -- 172.21.15.28:0/684460805 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb3c1a71b0 con 0x7feb3c106710 2024-04-23T14:43:33.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.255+0000 7feb3a000700 1 -- 172.21.15.28:0/684460805 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb340052a0 con 0x7feb3c106710 2024-04-23T14:43:33.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.256+0000 7feb44400700 1 -- 172.21.15.28:0/684460805 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb3c1a7660 con 0x7feb3c106710 2024-04-23T14:43:33.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.257+0000 7feb44400700 1 -- 172.21.15.28:0/684460805 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb3c0697e0 con 0x7feb3c106710 2024-04-23T14:43:33.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.259+0000 7feb3a000700 1 -- 172.21.15.28:0/684460805 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7feb34024070 con 0x7feb3c106710 2024-04-23T14:43:33.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.260+0000 7feb3a000700 1 --2- 172.21.15.28:0/684460805 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7feb240a3c40 0x7feb240a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:33.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.260+0000 7feb3a000700 1 -- 172.21.15.28:0/684460805 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7feb3400f020 con 0x7feb3c106710 2024-04-23T14:43:33.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.260+0000 7feb41a00700 1 --2- 172.21.15.28:0/684460805 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7feb240a3c40 0x7feb240a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:33.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.261+0000 7feb41a00700 1 --2- 172.21.15.28:0/684460805 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7feb240a3c40 0x7feb240a6100 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7feb30009c60 tx=0x7feb30009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:33.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.265+0000 7feb3a000700 1 -- 172.21.15.28:0/684460805 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feb3402b020 con 0x7feb3c106710 2024-04-23T14:43:33.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.507+0000 7feb44400700 1 -- 172.21.15.28:0/684460805 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7feb3c062f30 con 0x7feb240a3c40 2024-04-23T14:43:33.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.510+0000 7feb3a000700 1 -- 172.21.15.28:0/684460805 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7feb3c062f30 con 0x7feb240a3c40 2024-04-23T14:43:33.511 INFO:teuthology.orchestra.run.smithi028.stdout:HOST ADDR LABELS STATUS 2024-04-23T14:43:33.511 INFO:teuthology.orchestra.run.smithi028.stdout:smithi028 172.21.15.28 2024-04-23T14:43:33.511 INFO:teuthology.orchestra.run.smithi028.stdout:smithi038 172.21.15.38 2024-04-23T14:43:33.512 INFO:teuthology.orchestra.run.smithi028.stdout:smithi105 172.21.15.105 2024-04-23T14:43:33.512 INFO:teuthology.orchestra.run.smithi028.stdout:3 hosts in cluster 2024-04-23T14:43:33.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.513+0000 7feb44400700 1 -- 172.21.15.28:0/684460805 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7feb240a3c40 msgr2=0x7feb240a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:33.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.513+0000 7feb44400700 1 --2- 172.21.15.28:0/684460805 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7feb240a3c40 0x7feb240a6100 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7feb30009c60 tx=0x7feb30009260 comp rx=0 tx=0).stop 2024-04-23T14:43:33.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.513+0000 7feb44400700 1 -- 172.21.15.28:0/684460805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb3c106710 msgr2=0x7feb3c1a6880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:33.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.513+0000 7feb44400700 1 --2- 172.21.15.28:0/684460805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb3c106710 0x7feb3c1a6880 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7feb34004f40 tx=0x7feb34004f70 comp rx=0 tx=0).stop 2024-04-23T14:43:33.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.513+0000 7feb44400700 1 -- 172.21.15.28:0/684460805 shutdown_connections 2024-04-23T14:43:33.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.513+0000 7feb44400700 1 --2- 172.21.15.28:0/684460805 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7feb240a3c40 0x7feb240a6100 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:33.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.513+0000 7feb44400700 1 --2- 172.21.15.28:0/684460805 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feb3c1013d0 0x7feb3c19f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:33.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.513+0000 7feb44400700 1 --2- 172.21.15.28:0/684460805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb3c106710 0x7feb3c1a6880 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:33.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.513+0000 7feb44400700 1 --2- 172.21.15.28:0/684460805 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb3c103d40 0x7feb3c19f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:33.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.513+0000 7feb44400700 1 -- 172.21.15.28:0/684460805 >> 172.21.15.28:0/684460805 conn(0x7feb3c0fcff0 msgr2=0x7feb3c102100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:33.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.513+0000 7feb44400700 1 -- 172.21.15.28:0/684460805 shutdown_connections 2024-04-23T14:43:33.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:33.513+0000 7feb44400700 1 -- 172.21.15.28:0/684460805 wait complete. 2024-04-23T14:43:33.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:33 smithi105 ceph-mon[32555]: pgmap v372: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-23T14:43:33.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:33 smithi038 ceph-mon[32348]: pgmap v372: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-23T14:43:33.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:33 smithi028 ceph-mon[27810]: pgmap v372: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-23T14:43:34.288 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-23T14:43:34.772 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:43:34.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:34 smithi105 ceph-mon[32555]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:43:34.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:34 smithi038 ceph-mon[32348]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:43:34.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:34 smithi028 ceph-mon[27810]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:43:35.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:35 smithi105 ceph-mon[32555]: pgmap v373: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-23T14:43:35.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:35 smithi038 ceph-mon[32348]: pgmap v373: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-23T14:43:35.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:35 smithi028 ceph-mon[27810]: pgmap v373: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-23T14:43:36.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.786+0000 7fbc96e00700 1 -- 172.21.15.28:0/4000997732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc90104710 msgr2=0x7fbc9010afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:36.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.786+0000 7fbc96e00700 1 --2- 172.21.15.28:0/4000997732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc90104710 0x7fbc9010afc0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbc88009fc0 tx=0x7fbc8800b330 comp rx=0 tx=0).stop 2024-04-23T14:43:36.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.786+0000 7fbc96e00700 1 -- 172.21.15.28:0/4000997732 shutdown_connections 2024-04-23T14:43:36.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.786+0000 7fbc96e00700 1 --2- 172.21.15.28:0/4000997732 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc90102b70 0x7fbc90102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:36.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.786+0000 7fbc96e00700 1 --2- 172.21.15.28:0/4000997732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc90104710 0x7fbc9010afc0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:36.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.786+0000 7fbc96e00700 1 --2- 172.21.15.28:0/4000997732 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc90103d70 0x7fbc901041d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:36.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.786+0000 7fbc96e00700 1 -- 172.21.15.28:0/4000997732 >> 172.21.15.28:0/4000997732 conn(0x7fbc900fe110 msgr2=0x7fbc90100550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:36.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.786+0000 7fbc96e00700 1 -- 172.21.15.28:0/4000997732 shutdown_connections 2024-04-23T14:43:36.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.786+0000 7fbc96e00700 1 -- 172.21.15.28:0/4000997732 wait complete. 2024-04-23T14:43:36.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.787+0000 7fbc96e00700 1 Processor -- start 2024-04-23T14:43:36.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.788+0000 7fbc96e00700 1 -- start start 2024-04-23T14:43:36.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.788+0000 7fbc96e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc90102b70 0x7fbc9019af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:36.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.788+0000 7fbc96e00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc90103d70 0x7fbc9019b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:36.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.788+0000 7fbc96e00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc90104710 0x7fbc901a2500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:36.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.788+0000 7fbc96e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9010e7b0 con 0x7fbc90102b70 2024-04-23T14:43:36.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.788+0000 7fbc96e00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9010e630 con 0x7fbc90103d70 2024-04-23T14:43:36.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.788+0000 7fbc96e00700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9010e930 con 0x7fbc90104710 2024-04-23T14:43:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.788+0000 7fbc8fe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc90102b70 0x7fbc9019af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.788+0000 7fbc8fe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc90102b70 0x7fbc9019af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46274/0 (socket says 172.21.15.28:46274) 2024-04-23T14:43:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.788+0000 7fbc8fe00700 1 -- 172.21.15.28:0/942585661 learned_addr learned my addr 172.21.15.28:0/942585661 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.788+0000 7fbc8f400700 1 --2- 172.21.15.28:0/942585661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc90103d70 0x7fbc9019b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.788+0000 7fbc94e00700 1 --2- 172.21.15.28:0/942585661 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc90104710 0x7fbc901a2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.789+0000 7fbc8fe00700 1 -- 172.21.15.28:0/942585661 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc90104710 msgr2=0x7fbc901a2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.789+0000 7fbc8fe00700 1 --2- 172.21.15.28:0/942585661 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc90104710 0x7fbc901a2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.789+0000 7fbc8fe00700 1 -- 172.21.15.28:0/942585661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc90103d70 msgr2=0x7fbc9019b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.789+0000 7fbc8fe00700 1 --2- 172.21.15.28:0/942585661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc90103d70 0x7fbc9019b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.789+0000 7fbc8fe00700 1 -- 172.21.15.28:0/942585661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc901a2ba0 con 0x7fbc90102b70 2024-04-23T14:43:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.789+0000 7fbc8fe00700 1 --2- 172.21.15.28:0/942585661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc90102b70 0x7fbc9019af40 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fbc8000e980 tx=0x7fbc8000ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.789+0000 7fbc8cc00700 1 -- 172.21.15.28:0/942585661 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc80004230 con 0x7fbc90102b70 2024-04-23T14:43:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.789+0000 7fbc8cc00700 1 -- 172.21.15.28:0/942585661 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc800043d0 con 0x7fbc90102b70 2024-04-23T14:43:36.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.789+0000 7fbc8cc00700 1 -- 172.21.15.28:0/942585661 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc80003dc0 con 0x7fbc90102b70 2024-04-23T14:43:36.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.789+0000 7fbc96e00700 1 -- 172.21.15.28:0/942585661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc901a2e90 con 0x7fbc90102b70 2024-04-23T14:43:36.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.790+0000 7fbc96e00700 1 -- 172.21.15.28:0/942585661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc901a3290 con 0x7fbc90102b70 2024-04-23T14:43:36.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.791+0000 7fbc96e00700 1 -- 172.21.15.28:0/942585661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc901a3580 con 0x7fbc90102b70 2024-04-23T14:43:36.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.795+0000 7fbc8cc00700 1 -- 172.21.15.28:0/942585661 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fbc8000c810 con 0x7fbc90102b70 2024-04-23T14:43:36.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.796+0000 7fbc8cc00700 1 --2- 172.21.15.28:0/942585661 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fbc740a3cc0 0x7fbc740a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:36.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.796+0000 7fbc8cc00700 1 -- 172.21.15.28:0/942585661 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7fbc800d0640 con 0x7fbc90102b70 2024-04-23T14:43:36.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.796+0000 7fbc8f400700 1 --2- 172.21.15.28:0/942585661 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fbc740a3cc0 0x7fbc740a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:36.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.797+0000 7fbc8f400700 1 --2- 172.21.15.28:0/942585661 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fbc740a3cc0 0x7fbc740a6180 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c009750 tx=0x7fbc7c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:36.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:36.799+0000 7fbc8cc00700 1 -- 172.21.15.28:0/942585661 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbc8009cfe0 con 0x7fbc90102b70 2024-04-23T14:43:37.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:37.035+0000 7fbc96e00700 1 -- 172.21.15.28:0/942585661 --> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fbc901a3750 con 0x7fbc740a3cc0 2024-04-23T14:43:37.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:37.038+0000 7fbc8cc00700 1 -- 172.21.15.28:0/942585661 <== mgr.14152 v2:172.21.15.28:6800/1997136173 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fbc901a3750 con 0x7fbc740a3cc0 2024-04-23T14:43:37.039 INFO:teuthology.orchestra.run.smithi028.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-23T14:43:37.039 INFO:teuthology.orchestra.run.smithi028.stdout:smithi028 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB747401F4480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-04-23T14:43:37.039 INFO:teuthology.orchestra.run.smithi028.stdout:smithi038 /dev/nvme0n1 ssd INTEL_SSDPEDKE040T7_BTLE736103BB4P0KGN 3726G No 3m ago Has a FileSystem, LVM detected 2024-04-23T14:43:37.040 INFO:teuthology.orchestra.run.smithi028.stdout:smithi105 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400TR400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-23T14:43:37.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:37.041+0000 7fbc96e00700 1 -- 172.21.15.28:0/942585661 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fbc740a3cc0 msgr2=0x7fbc740a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:37.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:37.041+0000 7fbc96e00700 1 --2- 172.21.15.28:0/942585661 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fbc740a3cc0 0x7fbc740a6180 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c009750 tx=0x7fbc7c009310 comp rx=0 tx=0).stop 2024-04-23T14:43:37.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:37.041+0000 7fbc96e00700 1 -- 172.21.15.28:0/942585661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc90102b70 msgr2=0x7fbc9019af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:37.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:37.041+0000 7fbc96e00700 1 --2- 172.21.15.28:0/942585661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc90102b70 0x7fbc9019af40 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fbc8000e980 tx=0x7fbc8000ec90 comp rx=0 tx=0).stop 2024-04-23T14:43:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:37.041+0000 7fbc96e00700 1 -- 172.21.15.28:0/942585661 shutdown_connections 2024-04-23T14:43:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:37.041+0000 7fbc96e00700 1 --2- 172.21.15.28:0/942585661 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fbc740a3cc0 0x7fbc740a6180 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:37.041+0000 7fbc96e00700 1 --2- 172.21.15.28:0/942585661 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc90104710 0x7fbc901a2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:37.041+0000 7fbc96e00700 1 --2- 172.21.15.28:0/942585661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc90102b70 0x7fbc9019af40 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:37.041+0000 7fbc96e00700 1 --2- 172.21.15.28:0/942585661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc90103d70 0x7fbc9019b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:37.041+0000 7fbc96e00700 1 -- 172.21.15.28:0/942585661 >> 172.21.15.28:0/942585661 conn(0x7fbc900fe110 msgr2=0x7fbc90074d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:37.041+0000 7fbc96e00700 1 -- 172.21.15.28:0/942585661 shutdown_connections 2024-04-23T14:43:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:37.041+0000 7fbc96e00700 1 -- 172.21.15.28:0/942585661 wait complete. 2024-04-23T14:43:37.758 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:37 smithi028 ceph-mon[27810]: pgmap v374: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:43:37.837 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-23T14:43:37.847 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-04-23T14:43:37.848 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-23T14:43:37.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:37 smithi038 ceph-mon[32348]: pgmap v374: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:43:38.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:37 smithi105 ceph-mon[32555]: pgmap v374: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:43:38.325 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:43:38.772 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:38 smithi028 ceph-mon[27810]: from='client.14898 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:43:38.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:38 smithi038 ceph-mon[32348]: from='client.14898 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:43:39.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:38 smithi105 ceph-mon[32555]: from='client.14898 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:43:39.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:39 smithi028 ceph-mon[27810]: pgmap v375: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:43:39.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:39 smithi038 ceph-mon[32348]: pgmap v375: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:43:40.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:39 smithi105 ceph-mon[32555]: pgmap v375: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:43:40.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.517+0000 7f1e5d400700 1 -- 172.21.15.28:0/3128509008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e58105f10 msgr2=0x7f1e58108300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:40.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.517+0000 7f1e5d400700 1 --2- 172.21.15.28:0/3128509008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e58105f10 0x7f1e58108300 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1e44009990 tx=0x7f1e44009ca0 comp rx=0 tx=0).stop 2024-04-23T14:43:40.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.517+0000 7f1e5d400700 1 -- 172.21.15.28:0/3128509008 shutdown_connections 2024-04-23T14:43:40.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.518+0000 7f1e5d400700 1 --2- 172.21.15.28:0/3128509008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1e58108840 0x7f1e5810acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:40.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.518+0000 7f1e5d400700 1 --2- 172.21.15.28:0/3128509008 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e581035e0 0x7f1e581059d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:40.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.518+0000 7f1e5d400700 1 --2- 172.21.15.28:0/3128509008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e58105f10 0x7f1e58108300 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:40.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.518+0000 7f1e5d400700 1 -- 172.21.15.28:0/3128509008 >> 172.21.15.28:0/3128509008 conn(0x7f1e580fcff0 msgr2=0x7f1e580ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:40.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.518+0000 7f1e5d400700 1 -- 172.21.15.28:0/3128509008 shutdown_connections 2024-04-23T14:43:40.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.518+0000 7f1e5d400700 1 -- 172.21.15.28:0/3128509008 wait complete. 2024-04-23T14:43:40.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.519+0000 7f1e5d400700 1 Processor -- start 2024-04-23T14:43:40.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.519+0000 7f1e5d400700 1 -- start start 2024-04-23T14:43:40.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.519+0000 7f1e5d400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e581035e0 0x7f1e5819b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:40.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.519+0000 7f1e5d400700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1e58108840 0x7f1e5819b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:40.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.519+0000 7f1e5d400700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e581a0830 0x7f1e581a2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:40.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.520+0000 7f1e5d400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5810e780 con 0x7f1e581035e0 2024-04-23T14:43:40.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.520+0000 7f1e5d400700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5810e600 con 0x7f1e581a0830 2024-04-23T14:43:40.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.520+0000 7f1e5d400700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5810e900 con 0x7f1e58108840 2024-04-23T14:43:40.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.520+0000 7f1e56800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e581035e0 0x7f1e5819b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:40.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.520+0000 7f1e55e00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1e58108840 0x7f1e5819b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:40.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.520+0000 7f1e56800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e581035e0 0x7f1e5819b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36692/0 (socket says 172.21.15.28:36692) 2024-04-23T14:43:40.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.520+0000 7f1e56800700 1 -- 172.21.15.28:0/3611799826 learned_addr learned my addr 172.21.15.28:0/3611799826 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:40.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.520+0000 7f1e57200700 1 --2- 172.21.15.28:0/3611799826 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e581a0830 0x7f1e581a2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:40.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.520+0000 7f1e55e00700 1 -- 172.21.15.28:0/3611799826 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e581a0830 msgr2=0x7f1e581a2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:40.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.520+0000 7f1e55e00700 1 --2- 172.21.15.28:0/3611799826 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e581a0830 0x7f1e581a2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:40.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.520+0000 7f1e55e00700 1 -- 172.21.15.28:0/3611799826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e581035e0 msgr2=0x7f1e5819b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:40.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.520+0000 7f1e55e00700 1 --2- 172.21.15.28:0/3611799826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e581035e0 0x7f1e5819b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:40.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.520+0000 7f1e55e00700 1 -- 172.21.15.28:0/3611799826 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e581a32c0 con 0x7f1e58108840 2024-04-23T14:43:40.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.521+0000 7f1e55e00700 1 --2- 172.21.15.28:0/3611799826 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1e58108840 0x7f1e5819b7c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1e44008000 tx=0x7f1e4400bb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:40.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.521+0000 7f1e4f400700 1 -- 172.21.15.28:0/3611799826 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e44026070 con 0x7f1e58108840 2024-04-23T14:43:40.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.521+0000 7f1e5d400700 1 -- 172.21.15.28:0/3611799826 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e581a3550 con 0x7f1e58108840 2024-04-23T14:43:40.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.521+0000 7f1e4f400700 1 -- 172.21.15.28:0/3611799826 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e44003d70 con 0x7f1e58108840 2024-04-23T14:43:40.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.521+0000 7f1e4f400700 1 -- 172.21.15.28:0/3611799826 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e4400fe00 con 0x7f1e58108840 2024-04-23T14:43:40.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.521+0000 7f1e5d400700 1 -- 172.21.15.28:0/3611799826 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e581a39a0 con 0x7f1e58108840 2024-04-23T14:43:40.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.522+0000 7f1e5d400700 1 -- 172.21.15.28:0/3611799826 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e580697e0 con 0x7f1e58108840 2024-04-23T14:43:40.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.524+0000 7f1e4f400700 1 -- 172.21.15.28:0/3611799826 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f1e44029070 con 0x7f1e58108840 2024-04-23T14:43:40.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.526+0000 7f1e4f400700 1 --2- 172.21.15.28:0/3611799826 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f1e3c0a3cc0 0x7f1e3c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:40.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.526+0000 7f1e4f400700 1 -- 172.21.15.28:0/3611799826 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7f1e440d1580 con 0x7f1e58108840 2024-04-23T14:43:40.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.526+0000 7f1e56800700 1 --2- 172.21.15.28:0/3611799826 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f1e3c0a3cc0 0x7f1e3c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:40.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.527+0000 7f1e56800700 1 --2- 172.21.15.28:0/3611799826 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f1e3c0a3cc0 0x7f1e3c0a6180 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1e48009c10 tx=0x7f1e4800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:40.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.528+0000 7f1e4f400700 1 -- 172.21.15.28:0/3611799826 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1e44030080 con 0x7f1e58108840 2024-04-23T14:43:40.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.813+0000 7f1e5d400700 1 -- 172.21.15.28:0/3611799826 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f1e581a3cc0 con 0x7f1e58108840 2024-04-23T14:43:40.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.814+0000 7f1e4f400700 1 -- 172.21.15.28:0/3611799826 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 20 v20) ==== 76+0+3002 (secure 0 0 0) 0x7f1e44013070 con 0x7f1e58108840 2024-04-23T14:43:40.816 INFO:teuthology.orchestra.run.smithi028.stdout:e20 2024-04-23T14:43:40.816 INFO:teuthology.orchestra.run.smithi028.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-23T14:43:40.816 INFO:teuthology.orchestra.run.smithi028.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-23T14:43:40.816 INFO:teuthology.orchestra.run.smithi028.stdout:legacy client fscid: 1 2024-04-23T14:43:40.816 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:43:40.816 INFO:teuthology.orchestra.run.smithi028.stdout:Filesystem 'cephfs' (1) 2024-04-23T14:43:40.816 INFO:teuthology.orchestra.run.smithi028.stdout:fs_name cephfs 2024-04-23T14:43:40.816 INFO:teuthology.orchestra.run.smithi028.stdout:epoch 20 2024-04-23T14:43:40.816 INFO:teuthology.orchestra.run.smithi028.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-23T14:43:40.816 INFO:teuthology.orchestra.run.smithi028.stdout:created 2024-04-23T14:41:39.495067+0000 2024-04-23T14:43:40.816 INFO:teuthology.orchestra.run.smithi028.stdout:modified 2024-04-23T14:41:43.869888+0000 2024-04-23T14:43:40.816 INFO:teuthology.orchestra.run.smithi028.stdout:tableserver 0 2024-04-23T14:43:40.816 INFO:teuthology.orchestra.run.smithi028.stdout:root 0 2024-04-23T14:43:40.817 INFO:teuthology.orchestra.run.smithi028.stdout:session_timeout 300 2024-04-23T14:43:40.817 INFO:teuthology.orchestra.run.smithi028.stdout:session_autoclose 300 2024-04-23T14:43:40.817 INFO:teuthology.orchestra.run.smithi028.stdout:max_file_size 1099511627776 2024-04-23T14:43:40.817 INFO:teuthology.orchestra.run.smithi028.stdout:max_xattr_size 65536 2024-04-23T14:43:40.817 INFO:teuthology.orchestra.run.smithi028.stdout:required_client_features {} 2024-04-23T14:43:40.817 INFO:teuthology.orchestra.run.smithi028.stdout:last_failure 0 2024-04-23T14:43:40.817 INFO:teuthology.orchestra.run.smithi028.stdout:last_failure_osd_epoch 0 2024-04-23T14:43:40.817 INFO:teuthology.orchestra.run.smithi028.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-23T14:43:40.817 INFO:teuthology.orchestra.run.smithi028.stdout:max_mds 1 2024-04-23T14:43:40.817 INFO:teuthology.orchestra.run.smithi028.stdout:in 0 2024-04-23T14:43:40.817 INFO:teuthology.orchestra.run.smithi028.stdout:up {0=24587} 2024-04-23T14:43:40.817 INFO:teuthology.orchestra.run.smithi028.stdout:failed 2024-04-23T14:43:40.817 INFO:teuthology.orchestra.run.smithi028.stdout:damaged 2024-04-23T14:43:40.817 INFO:teuthology.orchestra.run.smithi028.stdout:stopped 2024-04-23T14:43:40.818 INFO:teuthology.orchestra.run.smithi028.stdout:data_pools [3] 2024-04-23T14:43:40.818 INFO:teuthology.orchestra.run.smithi028.stdout:metadata_pool 2 2024-04-23T14:43:40.818 INFO:teuthology.orchestra.run.smithi028.stdout:inline_data disabled 2024-04-23T14:43:40.818 INFO:teuthology.orchestra.run.smithi028.stdout:balancer 2024-04-23T14:43:40.818 INFO:teuthology.orchestra.run.smithi028.stdout:bal_rank_mask -1 2024-04-23T14:43:40.818 INFO:teuthology.orchestra.run.smithi028.stdout:standby_count_wanted 0 2024-04-23T14:43:40.818 INFO:teuthology.orchestra.run.smithi028.stdout:qdb_cluster leader: 24587 members: 24587 2024-04-23T14:43:40.818 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.b{0:24587} state up:active seq 3 addr [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T14:43:40.818 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.g{0:24583} state up:standby-replay seq 1 addr [v2:172.21.15.28:6838/940220693,v1:172.21.15.28:6839/940220693] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T14:43:40.818 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:43:40.818 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:43:40.818 INFO:teuthology.orchestra.run.smithi028.stdout:Standby daemons: 2024-04-23T14:43:40.818 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:43:40.819 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.j{-1:14379} state up:standby seq 1 addr [v2:172.21.15.28:6834/760880137,v1:172.21.15.28:6835/760880137] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T14:43:40.819 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.d{-1:14391} state up:standby seq 1 addr [v2:172.21.15.28:6836/2963041370,v1:172.21.15.28:6837/2963041370] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T14:43:40.819 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.k{-1:24458} state up:standby seq 1 addr [v2:172.21.15.38:6832/1673432987,v1:172.21.15.38:6833/1673432987] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T14:43:40.819 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.l{-1:24460} state up:standby seq 1 addr [v2:172.21.15.105:6832/573683071,v1:172.21.15.105:6833/573683071] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T14:43:40.819 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.e{-1:24506} state up:standby seq 1 addr [v2:172.21.15.38:6834/2393540699,v1:172.21.15.38:6835/2393540699] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T14:43:40.819 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.f{-1:24529} state up:standby seq 1 addr [v2:172.21.15.105:6834/1788458050,v1:172.21.15.105:6835/1788458050] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T14:43:40.819 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.h{-1:24536} state up:standby seq 1 addr [v2:172.21.15.38:6836/4078619599,v1:172.21.15.38:6837/4078619599] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T14:43:40.819 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.c{-1:24541} state up:standby seq 1 addr [v2:172.21.15.105:6836/1691671537,v1:172.21.15.105:6837/1691671537] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T14:43:40.819 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.i{-1:24565} state up:standby seq 1 addr [v2:172.21.15.105:6838/1987159450,v1:172.21.15.105:6839/1987159450] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T14:43:40.819 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.a{-1:24575} state up:standby seq 1 addr [v2:172.21.15.28:6840/1569158142,v1:172.21.15.28:6841/1569158142] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T14:43:40.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.816+0000 7f1e5d400700 1 -- 172.21.15.28:0/3611799826 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f1e3c0a3cc0 msgr2=0x7f1e3c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:40.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.817+0000 7f1e5d400700 1 --2- 172.21.15.28:0/3611799826 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f1e3c0a3cc0 0x7f1e3c0a6180 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1e48009c10 tx=0x7f1e4800b040 comp rx=0 tx=0).stop 2024-04-23T14:43:40.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.817+0000 7f1e5d400700 1 -- 172.21.15.28:0/3611799826 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1e58108840 msgr2=0x7f1e5819b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:40.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.817+0000 7f1e5d400700 1 --2- 172.21.15.28:0/3611799826 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1e58108840 0x7f1e5819b7c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1e44008000 tx=0x7f1e4400bb90 comp rx=0 tx=0).stop 2024-04-23T14:43:40.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.817+0000 7f1e5d400700 1 -- 172.21.15.28:0/3611799826 shutdown_connections 2024-04-23T14:43:40.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.817+0000 7f1e5d400700 1 --2- 172.21.15.28:0/3611799826 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f1e3c0a3cc0 0x7f1e3c0a6180 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:40.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.817+0000 7f1e5d400700 1 --2- 172.21.15.28:0/3611799826 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e581a0830 0x7f1e581a2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:40.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.817+0000 7f1e5d400700 1 --2- 172.21.15.28:0/3611799826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e581035e0 0x7f1e5819b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:40.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.817+0000 7f1e5d400700 1 --2- 172.21.15.28:0/3611799826 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1e58108840 0x7f1e5819b7c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:40.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.817+0000 7f1e5d400700 1 -- 172.21.15.28:0/3611799826 >> 172.21.15.28:0/3611799826 conn(0x7f1e580fcff0 msgr2=0x7f1e58101610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:40.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.817+0000 7f1e5d400700 1 -- 172.21.15.28:0/3611799826 shutdown_connections 2024-04-23T14:43:40.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:40.817+0000 7f1e5d400700 1 -- 172.21.15.28:0/3611799826 wait complete. 2024-04-23T14:43:40.821 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:43:41.597 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-23T14:43:41.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:41 smithi028 ceph-mon[27810]: pgmap v376: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:43:41.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:41 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3611799826' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-23T14:43:41.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:41 smithi038 ceph-mon[32348]: pgmap v376: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:43:41.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:41 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3611799826' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-23T14:43:42.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:41 smithi105 ceph-mon[32555]: pgmap v376: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:43:42.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:41 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3611799826' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-23T14:43:42.053 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:43:43.829 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:43 smithi028 ceph-mon[27810]: pgmap v377: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:43:43.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:43 smithi038 ceph-mon[32348]: pgmap v377: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:43:44.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:43 smithi105 ceph-mon[32555]: pgmap v377: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:43:44.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.320+0000 7f5bb9e00700 1 -- 172.21.15.28:0/493637618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bb40790b0 msgr2=0x7f5bb4079930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:44.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.320+0000 7f5bb9e00700 1 --2- 172.21.15.28:0/493637618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bb40790b0 0x7f5bb4079930 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5bac00b0a0 tx=0x7f5bac00b3b0 comp rx=0 tx=0).stop 2024-04-23T14:43:44.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.321+0000 7f5bb9e00700 1 -- 172.21.15.28:0/493637618 shutdown_connections 2024-04-23T14:43:44.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.321+0000 7f5bb9e00700 1 --2- 172.21.15.28:0/493637618 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5bb40774a0 0x7f5bb40778c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:44.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.321+0000 7f5bb9e00700 1 --2- 172.21.15.28:0/493637618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bb40790b0 0x7f5bb4079930 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:44.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.321+0000 7f5bb9e00700 1 --2- 172.21.15.28:0/493637618 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bb40786f0 0x7f5bb4078b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:44.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.321+0000 7f5bb9e00700 1 -- 172.21.15.28:0/493637618 >> 172.21.15.28:0/493637618 conn(0x7f5bb4100340 msgr2=0x7f5bb4102780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:44.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.321+0000 7f5bb9e00700 1 -- 172.21.15.28:0/493637618 shutdown_connections 2024-04-23T14:43:44.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.321+0000 7f5bb9e00700 1 -- 172.21.15.28:0/493637618 wait complete. 2024-04-23T14:43:44.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.322+0000 7f5bb9e00700 1 Processor -- start 2024-04-23T14:43:44.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.322+0000 7f5bb9e00700 1 -- start start 2024-04-23T14:43:44.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.323+0000 7f5bb9e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bb40774a0 0x7f5bb419f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:44.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.323+0000 7f5bb9e00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bb40786f0 0x7f5bb419f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:44.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.323+0000 7f5bb9e00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5bb40790b0 0x7f5bb41a68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:44.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.323+0000 7f5bb9e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb4114de0 con 0x7f5bb40774a0 2024-04-23T14:43:44.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.323+0000 7f5bb9e00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb4114c60 con 0x7f5bb40786f0 2024-04-23T14:43:44.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.323+0000 7f5bb9e00700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb4114f60 con 0x7f5bb40790b0 2024-04-23T14:43:44.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.323+0000 7f5bb2a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bb40774a0 0x7f5bb419f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:44.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.323+0000 7f5bb2a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bb40774a0 0x7f5bb419f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36716/0 (socket says 172.21.15.28:36716) 2024-04-23T14:43:44.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.323+0000 7f5bb2a00700 1 -- 172.21.15.28:0/2329633632 learned_addr learned my addr 172.21.15.28:0/2329633632 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:44.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.323+0000 7f5bb2000700 1 --2- 172.21.15.28:0/2329633632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bb40786f0 0x7f5bb419f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:44.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.323+0000 7f5bb3400700 1 --2- 172.21.15.28:0/2329633632 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5bb40790b0 0x7f5bb41a68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:44.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.323+0000 7f5bb2a00700 1 -- 172.21.15.28:0/2329633632 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5bb40790b0 msgr2=0x7f5bb41a68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:44.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.323+0000 7f5bb2a00700 1 --2- 172.21.15.28:0/2329633632 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5bb40790b0 0x7f5bb41a68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:44.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.323+0000 7f5bb2a00700 1 -- 172.21.15.28:0/2329633632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bb40786f0 msgr2=0x7f5bb419f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:44.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.323+0000 7f5bb2a00700 1 --2- 172.21.15.28:0/2329633632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bb40786f0 0x7f5bb419f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:44.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.323+0000 7f5bb2a00700 1 -- 172.21.15.28:0/2329633632 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bb41a6f60 con 0x7f5bb40774a0 2024-04-23T14:43:44.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.324+0000 7f5bb2a00700 1 --2- 172.21.15.28:0/2329633632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bb40774a0 0x7f5bb419f350 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5ba400e9f0 tx=0x7f5ba400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:44.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.324+0000 7f5bb8a00700 1 -- 172.21.15.28:0/2329633632 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba400cbd0 con 0x7f5bb40774a0 2024-04-23T14:43:44.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.324+0000 7f5bb8a00700 1 -- 172.21.15.28:0/2329633632 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ba400ee90 con 0x7f5bb40774a0 2024-04-23T14:43:44.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.324+0000 7f5bb9e00700 1 -- 172.21.15.28:0/2329633632 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bb41a7250 con 0x7f5bb40774a0 2024-04-23T14:43:44.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.324+0000 7f5bb8a00700 1 -- 172.21.15.28:0/2329633632 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba40056c0 con 0x7f5bb40774a0 2024-04-23T14:43:44.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.324+0000 7f5bb9e00700 1 -- 172.21.15.28:0/2329633632 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bb41a7740 con 0x7f5bb40774a0 2024-04-23T14:43:44.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.326+0000 7f5bb9e00700 1 -- 172.21.15.28:0/2329633632 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bb40697e0 con 0x7f5bb40774a0 2024-04-23T14:43:44.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.327+0000 7f5bb8a00700 1 -- 172.21.15.28:0/2329633632 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f5ba4004750 con 0x7f5bb40774a0 2024-04-23T14:43:44.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.329+0000 7f5bb8a00700 1 --2- 172.21.15.28:0/2329633632 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f5b980a3c40 0x7f5b980a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:44.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.329+0000 7f5bb8a00700 1 -- 172.21.15.28:0/2329633632 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7f5ba40d0680 con 0x7f5bb40774a0 2024-04-23T14:43:44.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.329+0000 7f5bb2000700 1 --2- 172.21.15.28:0/2329633632 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f5b980a3c40 0x7f5b980a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:44.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.330+0000 7f5bb2000700 1 --2- 172.21.15.28:0/2329633632 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f5b980a3c40 0x7f5b980a6100 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5ba0009c60 tx=0x7f5ba0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:44.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.333+0000 7f5bb8a00700 1 -- 172.21.15.28:0/2329633632 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5ba409d060 con 0x7f5bb40774a0 2024-04-23T14:43:44.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.555+0000 7f5bb9e00700 1 -- 172.21.15.28:0/2329633632 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f5bb4063700 con 0x7f5bb40774a0 2024-04-23T14:43:44.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.556+0000 7f5bb8a00700 1 -- 172.21.15.28:0/2329633632 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v87) ==== 56+0+5293 (secure 0 0 0) 0x7f5ba409c960 con 0x7f5bb40774a0 2024-04-23T14:43:44.558 INFO:teuthology.orchestra.run.smithi028.stdout:epoch 87 2024-04-23T14:43:44.558 INFO:teuthology.orchestra.run.smithi028.stdout:fsid 1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T14:43:44.558 INFO:teuthology.orchestra.run.smithi028.stdout:created 2024-04-23T14:32:34.581082+0000 2024-04-23T14:43:44.559 INFO:teuthology.orchestra.run.smithi028.stdout:modified 2024-04-23T14:42:43.429141+0000 2024-04-23T14:43:44.559 INFO:teuthology.orchestra.run.smithi028.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-23T14:43:44.559 INFO:teuthology.orchestra.run.smithi028.stdout:crush_version 26 2024-04-23T14:43:44.559 INFO:teuthology.orchestra.run.smithi028.stdout:full_ratio 0.95 2024-04-23T14:43:44.559 INFO:teuthology.orchestra.run.smithi028.stdout:backfillfull_ratio 0.9 2024-04-23T14:43:44.559 INFO:teuthology.orchestra.run.smithi028.stdout:nearfull_ratio 0.85 2024-04-23T14:43:44.559 INFO:teuthology.orchestra.run.smithi028.stdout:require_min_compat_client luminous 2024-04-23T14:43:44.559 INFO:teuthology.orchestra.run.smithi028.stdout:min_compat_client luminous 2024-04-23T14:43:44.559 INFO:teuthology.orchestra.run.smithi028.stdout:require_osd_release squid 2024-04-23T14:43:44.559 INFO:teuthology.orchestra.run.smithi028.stdout:stretch_mode_enabled false 2024-04-23T14:43:44.559 INFO:teuthology.orchestra.run.smithi028.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-23T14:43:44.559 INFO:teuthology.orchestra.run.smithi028.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-23T14:43:44.559 INFO:teuthology.orchestra.run.smithi028.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-04-23T14:43:44.559 INFO:teuthology.orchestra.run.smithi028.stdout:max_osd 12 2024-04-23T14:43:44.559 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 up in weight 1 up_from 8 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6802/2993445336,v1:172.21.15.28:6803/2993445336] [v2:172.21.15.28:6804/2993445336,v1:172.21.15.28:6805/2993445336] exists,up e0b427bc-be65-4543-99cc-2f53ad98ddad 2024-04-23T14:43:44.560 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 up in weight 1 up_from 13 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6800/4024093383,v1:172.21.15.38:6801/4024093383] [v2:172.21.15.38:6802/4024093383,v1:172.21.15.38:6803/4024093383] exists,up ef527ec9-1bf3-4bb0-8df3-f5a015af557c 2024-04-23T14:43:44.560 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 up in weight 1 up_from 18 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.105:6800/570739029,v1:172.21.15.105:6801/570739029] [v2:172.21.15.105:6802/570739029,v1:172.21.15.105:6803/570739029] exists,up b33b23d9-adce-4c11-b304-2e117bdf6bfe 2024-04-23T14:43:44.560 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 up in weight 1 up_from 25 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6810/2480400247,v1:172.21.15.28:6811/2480400247] [v2:172.21.15.28:6812/2480400247,v1:172.21.15.28:6813/2480400247] exists,up 28bd5608-b426-434e-9db1-bdaa9828bdb9 2024-04-23T14:43:44.560 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 up in weight 1 up_from 33 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6808/3772189246,v1:172.21.15.38:6809/3772189246] [v2:172.21.15.38:6810/3772189246,v1:172.21.15.38:6811/3772189246] exists,up e5d1c89b-025f-4d57-8807-59d75c787b98 2024-04-23T14:43:44.560 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 up in weight 1 up_from 38 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.105:6808/928755894,v1:172.21.15.105:6809/928755894] [v2:172.21.15.105:6810/928755894,v1:172.21.15.105:6811/928755894] exists,up 9cf9cf4c-aafd-4ed3-bfb1-0527710a0292 2024-04-23T14:43:44.560 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 up in weight 1 up_from 46 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6818/897979441,v1:172.21.15.28:6819/897979441] [v2:172.21.15.28:6820/897979441,v1:172.21.15.28:6821/897979441] exists,up 304db525-d24d-4687-bfa0-f92bdf66d1e1 2024-04-23T14:43:44.560 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 up in weight 1 up_from 51 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6816/2400251786,v1:172.21.15.38:6817/2400251786] [v2:172.21.15.38:6818/2400251786,v1:172.21.15.38:6819/2400251786] exists,up 08e01d59-03f3-47b1-b45f-35ec5a50992c 2024-04-23T14:43:44.560 INFO:teuthology.orchestra.run.smithi028.stdout:osd.8 up in weight 1 up_from 59 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.105:6816/1682312269,v1:172.21.15.105:6817/1682312269] [v2:172.21.15.105:6818/1682312269,v1:172.21.15.105:6819/1682312269] exists,up 64ee1e95-0756-48b1-9c00-472d0c3cbd29 2024-04-23T14:43:44.560 INFO:teuthology.orchestra.run.smithi028.stdout:osd.9 up in weight 1 up_from 67 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6826/1292942516,v1:172.21.15.28:6827/1292942516] [v2:172.21.15.28:6828/1292942516,v1:172.21.15.28:6829/1292942516] exists,up 18f60893-3ffd-48c2-9db5-a67d35611d70 2024-04-23T14:43:44.560 INFO:teuthology.orchestra.run.smithi028.stdout:osd.10 up in weight 1 up_from 72 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6824/2444616468,v1:172.21.15.38:6825/2444616468] [v2:172.21.15.38:6826/2444616468,v1:172.21.15.38:6827/2444616468] exists,up 76c06725-7040-4118-8894-fe808a87f999 2024-04-23T14:43:44.560 INFO:teuthology.orchestra.run.smithi028.stdout:osd.11 up in weight 1 up_from 80 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.105:6824/3400299562,v1:172.21.15.105:6825/3400299562] [v2:172.21.15.105:6826/3400299562,v1:172.21.15.105:6827/3400299562] exists,up e8fc9454-5540-4e2e-91bf-3ba251dbd1a8 2024-04-23T14:43:44.560 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.b [2,4] 2024-04-23T14:43:44.560 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.10 [2,1] 2024-04-23T14:43:44.561 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.1c [8,11] 2024-04-23T14:43:44.561 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.1d [8,11] 2024-04-23T14:43:44.561 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.3 [4,9] 2024-04-23T14:43:44.561 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.13 [4,9] 2024-04-23T14:43:44.561 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.1a [5,6] 2024-04-23T14:43:44.561 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.1b [4,8] 2024-04-23T14:43:44.561 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.2d [3,9] 2024-04-23T14:43:44.561 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/590495740 expires 2024-04-24T14:33:41.530295+0000 2024-04-23T14:43:44.561 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/1163172512 expires 2024-04-24T14:33:41.530295+0000 2024-04-23T14:43:44.561 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6801/2222038555 expires 2024-04-24T14:33:41.530295+0000 2024-04-23T14:43:44.561 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/4290328460 expires 2024-04-24T14:33:07.576363+0000 2024-04-23T14:43:44.561 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/4145020172 expires 2024-04-24T14:33:07.576363+0000 2024-04-23T14:43:44.561 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6800/1307025091 expires 2024-04-24T14:33:07.576363+0000 2024-04-23T14:43:44.561 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6800/2222038555 expires 2024-04-24T14:33:41.530295+0000 2024-04-23T14:43:44.561 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/720626958 expires 2024-04-24T14:33:41.530295+0000 2024-04-23T14:43:44.562 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6801/1307025091 expires 2024-04-24T14:33:07.576363+0000 2024-04-23T14:43:44.562 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/948450262 expires 2024-04-24T14:33:07.576363+0000 2024-04-23T14:43:44.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.560+0000 7f5bb9e00700 1 -- 172.21.15.28:0/2329633632 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f5b980a3c40 msgr2=0x7f5b980a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:44.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.560+0000 7f5bb9e00700 1 --2- 172.21.15.28:0/2329633632 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f5b980a3c40 0x7f5b980a6100 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5ba0009c60 tx=0x7f5ba0009260 comp rx=0 tx=0).stop 2024-04-23T14:43:44.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.560+0000 7f5bb9e00700 1 -- 172.21.15.28:0/2329633632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bb40774a0 msgr2=0x7f5bb419f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:44.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.560+0000 7f5bb9e00700 1 --2- 172.21.15.28:0/2329633632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bb40774a0 0x7f5bb419f350 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5ba400e9f0 tx=0x7f5ba400ed00 comp rx=0 tx=0).stop 2024-04-23T14:43:44.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.560+0000 7f5bb9e00700 1 -- 172.21.15.28:0/2329633632 shutdown_connections 2024-04-23T14:43:44.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.560+0000 7f5bb9e00700 1 --2- 172.21.15.28:0/2329633632 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f5b980a3c40 0x7f5b980a6100 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:44.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.560+0000 7f5bb9e00700 1 --2- 172.21.15.28:0/2329633632 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5bb40790b0 0x7f5bb41a68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:44.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.560+0000 7f5bb9e00700 1 --2- 172.21.15.28:0/2329633632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bb40774a0 0x7f5bb419f350 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:44.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.560+0000 7f5bb9e00700 1 --2- 172.21.15.28:0/2329633632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bb40786f0 0x7f5bb419f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:44.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.560+0000 7f5bb9e00700 1 -- 172.21.15.28:0/2329633632 >> 172.21.15.28:0/2329633632 conn(0x7f5bb4100340 msgr2=0x7f5bb4101e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:44.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.560+0000 7f5bb9e00700 1 -- 172.21.15.28:0/2329633632 shutdown_connections 2024-04-23T14:43:44.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:44.560+0000 7f5bb9e00700 1 -- 172.21.15.28:0/2329633632 wait complete. 2024-04-23T14:43:44.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:44 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2329633632' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-23T14:43:44.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:44 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2329633632' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-23T14:43:45.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:44 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2329633632' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-23T14:43:45.348 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-23T14:43:45.359 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:43:45.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.587+0000 7f6c17000640 1 -- 172.21.15.28:0/2278157957 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c10075fe0 msgr2=0x7f6c10074430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:45.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.587+0000 7f6c17000640 1 --2- 172.21.15.28:0/2278157957 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c10075fe0 0x7f6c10074430 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6c00009900 tx=0x7f6c0002a480 comp rx=0 tx=0).stop 2024-04-23T14:43:45.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.587+0000 7f6c17000640 1 -- 172.21.15.28:0/2278157957 shutdown_connections 2024-04-23T14:43:45.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.587+0000 7f6c17000640 1 --2- 172.21.15.28:0/2278157957 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c10074a00 0x7f6c10074e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:45.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.587+0000 7f6c17000640 1 --2- 172.21.15.28:0/2278157957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c10109d30 0x7f6c1010c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:45.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.587+0000 7f6c17000640 1 --2- 172.21.15.28:0/2278157957 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c10075fe0 0x7f6c10074430 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:45.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.587+0000 7f6c17000640 1 -- 172.21.15.28:0/2278157957 >> 172.21.15.28:0/2278157957 conn(0x7f6c100fd410 msgr2=0x7f6c100ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:45.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.588+0000 7f6c17000640 1 -- 172.21.15.28:0/2278157957 shutdown_connections 2024-04-23T14:43:45.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.588+0000 7f6c17000640 1 -- 172.21.15.28:0/2278157957 wait complete. 2024-04-23T14:43:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.589+0000 7f6c17000640 1 Processor -- start 2024-04-23T14:43:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.589+0000 7f6c17000640 1 -- start start 2024-04-23T14:43:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.589+0000 7f6c17000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c10074a00 0x7f6c1019a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.589+0000 7f6c17000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c10075fe0 0x7f6c1019a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.589+0000 7f6c17000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c10109d30 0x7f6c101a1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.589+0000 7f6c17000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1010fd80 con 0x7f6c10074a00 2024-04-23T14:43:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.589+0000 7f6c17000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1010fc00 con 0x7f6c10075fe0 2024-04-23T14:43:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.589+0000 7f6c17000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1010ff00 con 0x7f6c10109d30 2024-04-23T14:43:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.589+0000 7f6c15c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c10074a00 0x7f6c1019a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.589+0000 7f6c15200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c10075fe0 0x7f6c1019a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.589+0000 7f6c15c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c10074a00 0x7f6c1019a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36720/0 (socket says 172.21.15.28:36720) 2024-04-23T14:43:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.589+0000 7f6c15c00640 1 -- 172.21.15.28:0/1286241797 learned_addr learned my addr 172.21.15.28:0/1286241797 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.589+0000 7f6c16600640 1 --2- 172.21.15.28:0/1286241797 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c10109d30 0x7f6c101a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.590+0000 7f6c15200640 1 -- 172.21.15.28:0/1286241797 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c10109d30 msgr2=0x7f6c101a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:45.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.590+0000 7f6c15200640 1 --2- 172.21.15.28:0/1286241797 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c10109d30 0x7f6c101a1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:45.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.590+0000 7f6c15200640 1 -- 172.21.15.28:0/1286241797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c10074a00 msgr2=0x7f6c1019a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:45.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.590+0000 7f6c15200640 1 --2- 172.21.15.28:0/1286241797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c10074a00 0x7f6c1019a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:45.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.590+0000 7f6c15200640 1 -- 172.21.15.28:0/1286241797 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c101a1f80 con 0x7f6c10075fe0 2024-04-23T14:43:45.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.590+0000 7f6c15200640 1 --2- 172.21.15.28:0/1286241797 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c10075fe0 0x7f6c1019a8a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6c0400e9f0 tx=0x7f6c0400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:45.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.590+0000 7f6c0e000640 1 -- 172.21.15.28:0/1286241797 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c0400ccc0 con 0x7f6c10075fe0 2024-04-23T14:43:45.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.590+0000 7f6c0e000640 1 -- 172.21.15.28:0/1286241797 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c04010910 con 0x7f6c10075fe0 2024-04-23T14:43:45.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.590+0000 7f6c0e000640 1 -- 172.21.15.28:0/1286241797 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c04018960 con 0x7f6c10075fe0 2024-04-23T14:43:45.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.590+0000 7f6c17000640 1 -- 172.21.15.28:0/1286241797 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c101a2270 con 0x7f6c10075fe0 2024-04-23T14:43:45.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.590+0000 7f6c17000640 1 -- 172.21.15.28:0/1286241797 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c101a27c0 con 0x7f6c10075fe0 2024-04-23T14:43:45.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.592+0000 7f6c17000640 1 -- 172.21.15.28:0/1286241797 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c10108af0 con 0x7f6c10075fe0 2024-04-23T14:43:45.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.593+0000 7f6c0e000640 1 -- 172.21.15.28:0/1286241797 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f6c04010ac0 con 0x7f6c10075fe0 2024-04-23T14:43:45.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.595+0000 7f6c0e000640 1 --2- 172.21.15.28:0/1286241797 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f6bf80a3bb0 0x7f6bf80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:45.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.595+0000 7f6c0e000640 1 -- 172.21.15.28:0/1286241797 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7f6c040d0780 con 0x7f6c10075fe0 2024-04-23T14:43:45.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.595+0000 7f6c15c00640 1 --2- 172.21.15.28:0/1286241797 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f6bf80a3bb0 0x7f6bf80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:45.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.596+0000 7f6c15c00640 1 --2- 172.21.15.28:0/1286241797 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f6bf80a3bb0 0x7f6bf80a6070 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6c000096c0 tx=0x7f6c000315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:45.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.599+0000 7f6c0e000640 1 -- 172.21.15.28:0/1286241797 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6c0409cad0 con 0x7f6c10075fe0 2024-04-23T14:43:45.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.802+0000 7f6c17000640 1 -- 172.21.15.28:0/1286241797 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6c10062c00 con 0x7f6c10075fe0 2024-04-23T14:43:45.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.804+0000 7f6c0e000640 1 -- 172.21.15.28:0/1286241797 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22651 (secure 0 0 0) 0x7f6c040a0dd0 con 0x7f6c10075fe0 2024-04-23T14:43:45.806 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:43:45.806 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":87,"fsid":"1ee6e996-017e-11ef-bc93-c7b262605968","created":"2024-04-23T14:32:34.581082+0000","modified":"2024-04-23T14:42:43.429141+0000","last_up_change":"2024-04-23T14:40:43.375846+0000","last_in_change":"2024-04-23T14:40:23.576939+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:36:27.698112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:41:37.030341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:41:38.444742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e0b427bc-be65-4543-99cc-2f53ad98ddad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6803","nonce":2993445336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6805","nonce":2993445336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6809","nonce":2993445336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6807","nonce":2993445336}]},"public_addr":"172.21.15.28:6803/2993445336","cluster_addr":"172.21.15.28:6805/2993445336","heartbeat_back_addr":"172.21.15.28:6809/2993445336","heartbeat_front_addr":"172.21.15.28:6807/2993445336","state":["exists","up"]},{"osd":1,"uuid":"ef527ec9-1bf3-4bb0-8df3-f5a015af557c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6801","nonce":4024093383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6803","nonce":4024093383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6807","nonce":4024093383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6805","nonce":4024093383}]},"public_addr":"172.21.15.38:6801/4024093383","cluster_addr":"172.21.15.38:6803/4024093383","heartbeat_back_addr":"172.21.15.38:6807/4024093383","heartbeat_front_addr":"172.21.15.38:6805/4024093383","state":["exists","up"]},{"osd":2,"uuid":"b33b23d9-adce-4c11-b304-2e117bdf6bfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6801","nonce":570739029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6803","nonce":570739029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6807","nonce":570739029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6805","nonce":570739029}]},"public_addr":"172.21.15.105:6801/570739029","cluster_addr":"172.21.15.105:6803/570739029","heartbeat_back_addr":"172.21.15.105:6807/570739029","heartbeat_front_addr":"172.21.15.105:6805/570739029","state":["exists","up"]},{"osd":3,"uuid":"28bd5608-b426-434e-9db1-bdaa9828bdb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6811","nonce":2480400247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6813","nonce":2480400247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6817","nonce":2480400247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6815","nonce":2480400247}]},"public_addr":"172.21.15.28:6811/2480400247","cluster_addr":"172.21.15.28:6813/2480400247","heartbeat_back_addr":"172.21.15.28:6817/2480400247","heartbeat_front_addr":"172.21.15.28:6815/2480400247","state":["exists","up"]},{"osd":4,"uuid":"e5d1c89b-025f-4d57-8807-59d75c787b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6809","nonce":3772189246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6811","nonce":3772189246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6815","nonce":3772189246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6813","nonce":3772189246}]},"public_addr":"172.21.15.38:6809/3772189246","cluster_addr":"172.21.15.38:6811/3772189246","heartbeat_back_addr":"172.21.15.38:6815/3772189246","heartbeat_front_addr":"172.21.15.38:6813/3772189246","state":["exists","up"]},{"osd":5,"uuid":"9cf9cf4c-aafd-4ed3-bfb1-0527710a0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6809","nonce":928755894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6811","nonce":928755894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6815","nonce":928755894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6813","nonce":928755894}]},"public_addr":"172.21.15.105:6809/928755894","cluster_addr":"172.21.15.105:6811/928755894","heartbeat_back_addr":"172.21.15.105:6815/928755894","heartbeat_front_addr":"172.21.15.105:6813/928755894","state":["exists","up"]},{"osd":6,"uuid":"304db525-d24d-4687-bfa0-f92bdf66d1e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6819","nonce":897979441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6821","nonce":897979441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6825","nonce":897979441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6823","nonce":897979441}]},"public_addr":"172.21.15.28:6819/897979441","cluster_addr":"172.21.15.28:6821/897979441","heartbeat_back_addr":"172.21.15.28:6825/897979441","heartbeat_front_addr":"172.21.15.28:6823/897979441","state":["exists","up"]},{"osd":7,"uuid":"08e01d59-03f3-47b1-b45f-35ec5a50992c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6817","nonce":2400251786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6819","nonce":2400251786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6823","nonce":2400251786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6821","nonce":2400251786}]},"public_addr":"172.21.15.38:6817/2400251786","cluster_addr":"172.21.15.38:6819/2400251786","heartbeat_back_addr":"172.21.15.38:6823/2400251786","heartbeat_front_addr":"172.21.15.38:6821/2400251786","state":["exists","up"]},{"osd":8,"uuid":"64ee1e95-0756-48b1-9c00-472d0c3cbd29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6817","nonce":1682312269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6819","nonce":1682312269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6823","nonce":1682312269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6821","nonce":1682312269}]},"public_addr":"172.21.15.105:6817/1682312269","cluster_addr":"172.21.15.105:6819/1682312269","heartbeat_back_addr":"172.21.15.105:6823/1682312269","heartbeat_front_addr":"172.21.15.105:6821/1682312269","state":["exists","up"]},{"osd":9,"uuid":"18f60893-3ffd-48c2-9db5-a67d35611d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6827","nonce":1292942516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6829","nonce":1292942516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6833","nonce":1292942516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6831","nonce":1292942516}]},"public_addr":"172.21.15.28:6827/1292942516","cluster_addr":"172.21.15.28:6829/1292942516","heartbeat_back_addr":"172.21.15.28:6833/1292942516","heartbeat_front_addr":"172.21.15.28:6831/1292942516","state":["exists","up"]},{"osd":10,"uuid":"76c06725-7040-4118-8894-fe808a87f999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6825","nonce":2444616468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6827","nonce":2444616468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6831","nonce":2444616468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6829","nonce":2444616468}]},"public_addr":"172.21.15.38:6825/2444616468","cluster_addr":"172.21.15.38:6827/2444616468","heartbeat_back_addr":"172.21.15.38:6831/2444616468","heartbeat_front_addr":"172.21.15.38:6829/2444616468","state":["exists","up"]},{"osd":11,"uuid":"e8fc9454-5540-4e2e-91bf-3ba251dbd1a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6825","nonce":3400299562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6827","nonce":3400299562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6831","nonce":3400299562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6829","nonce":3400299562}]},"public_addr":"172.21.15.105:6825/3400299562","cluster_addr":"172.21.15.105:6827/3400299562","heartbeat_back_addr":"172.21.15.105:6831/3400299562","heartbeat_front_addr":"172.21.15.105:6829/3400299562","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:35:33.615395+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:00.545293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:24.940211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:57.684887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:24.519061+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:49.010804+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:22.638478+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:49.559104+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:14.011771+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:47.859950+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:16.298445+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:41.619069+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.2d","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/590495740":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/1163172512":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/4290328460":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/4145020172":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/720626958":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/948450262":"2024-04-24T14:33:07.576363+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:43:45.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.807+0000 7f6c17000640 1 -- 172.21.15.28:0/1286241797 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f6bf80a3bb0 msgr2=0x7f6bf80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:45.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.807+0000 7f6c17000640 1 --2- 172.21.15.28:0/1286241797 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f6bf80a3bb0 0x7f6bf80a6070 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6c000096c0 tx=0x7f6c000315b0 comp rx=0 tx=0).stop 2024-04-23T14:43:45.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.807+0000 7f6c17000640 1 -- 172.21.15.28:0/1286241797 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c10075fe0 msgr2=0x7f6c1019a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:45.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.807+0000 7f6c17000640 1 --2- 172.21.15.28:0/1286241797 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c10075fe0 0x7f6c1019a8a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6c0400e9f0 tx=0x7f6c0400eeb0 comp rx=0 tx=0).stop 2024-04-23T14:43:45.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.807+0000 7f6c17000640 1 -- 172.21.15.28:0/1286241797 shutdown_connections 2024-04-23T14:43:45.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.807+0000 7f6c17000640 1 --2- 172.21.15.28:0/1286241797 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f6bf80a3bb0 0x7f6bf80a6070 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:45.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.807+0000 7f6c17000640 1 --2- 172.21.15.28:0/1286241797 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c10075fe0 0x7f6c1019a8a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:45.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.807+0000 7f6c17000640 1 --2- 172.21.15.28:0/1286241797 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c10109d30 0x7f6c101a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:45.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.807+0000 7f6c17000640 1 --2- 172.21.15.28:0/1286241797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c10074a00 0x7f6c1019a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:45.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.807+0000 7f6c17000640 1 -- 172.21.15.28:0/1286241797 >> 172.21.15.28:0/1286241797 conn(0x7f6c100fd410 msgr2=0x7f6c10105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:45.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.807+0000 7f6c17000640 1 -- 172.21.15.28:0/1286241797 shutdown_connections 2024-04-23T14:43:45.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:45.807+0000 7f6c17000640 1 -- 172.21.15.28:0/1286241797 wait complete. 2024-04-23T14:43:45.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:45 smithi028 ceph-mon[27810]: pgmap v378: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:43:45.825 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T14:36:27.698112+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T14:41:37.030341+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T14:41:38.444742+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T14:43:45.826 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T14:43:45.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:45 smithi038 ceph-mon[32348]: pgmap v378: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:43:46.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:45 smithi105 ceph-mon[32555]: pgmap v378: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:43:46.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.038+0000 7fd0cf200640 1 -- 172.21.15.28:0/2624571749 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0d00769c0 msgr2=0x7fd0d0076de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:46.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.038+0000 7fd0cf200640 1 --2- 172.21.15.28:0/2624571749 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0d00769c0 0x7fd0d0076de0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd0c0009900 tx=0x7fd0c002a410 comp rx=0 tx=0).stop 2024-04-23T14:43:46.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.038+0000 7fd0cf200640 1 -- 172.21.15.28:0/2624571749 shutdown_connections 2024-04-23T14:43:46.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.038+0000 7fd0cf200640 1 --2- 172.21.15.28:0/2624571749 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0d00785a0 0x7fd0d0078e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:46.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.038+0000 7fd0cf200640 1 --2- 172.21.15.28:0/2624571749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0d0077c00 0x7fd0d0078060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:46.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.038+0000 7fd0cf200640 1 --2- 172.21.15.28:0/2624571749 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0d00769c0 0x7fd0d0076de0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:46.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.038+0000 7fd0cf200640 1 -- 172.21.15.28:0/2624571749 >> 172.21.15.28:0/2624571749 conn(0x7fd0d00ff6b0 msgr2=0x7fd0d0101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:46.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.038+0000 7fd0cf200640 1 -- 172.21.15.28:0/2624571749 shutdown_connections 2024-04-23T14:43:46.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.038+0000 7fd0cf200640 1 -- 172.21.15.28:0/2624571749 wait complete. 2024-04-23T14:43:46.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.039+0000 7fd0cf200640 1 Processor -- start 2024-04-23T14:43:46.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.039+0000 7fd0cf200640 1 -- start start 2024-04-23T14:43:46.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.040+0000 7fd0cf200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0d00769c0 0x7fd0d019e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:46.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.040+0000 7fd0cf200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0d0077c00 0x7fd0d019ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:46.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.040+0000 7fd0cf200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0d00785a0 0x7fd0d01a5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:46.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.040+0000 7fd0cf200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d01141d0 con 0x7fd0d0077c00 2024-04-23T14:43:46.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.040+0000 7fd0cf200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d0114050 con 0x7fd0d00769c0 2024-04-23T14:43:46.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.040+0000 7fd0cf200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d0114350 con 0x7fd0d00785a0 2024-04-23T14:43:46.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.040+0000 7fd0c7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0d0077c00 0x7fd0d019ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:46.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.040+0000 7fd0c7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0d0077c00 0x7fd0d019ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36724/0 (socket says 172.21.15.28:36724) 2024-04-23T14:43:46.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.040+0000 7fd0c7e00640 1 -- 172.21.15.28:0/1059772334 learned_addr learned my addr 172.21.15.28:0/1059772334 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:46.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.040+0000 7fd0ce800640 1 --2- 172.21.15.28:0/1059772334 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0d00785a0 0x7fd0d01a5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:46.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.040+0000 7fd0cde00640 1 --2- 172.21.15.28:0/1059772334 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0d00769c0 0x7fd0d019e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:46.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.041+0000 7fd0ce800640 1 -- 172.21.15.28:0/1059772334 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0d00769c0 msgr2=0x7fd0d019e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:46.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.041+0000 7fd0ce800640 1 --2- 172.21.15.28:0/1059772334 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0d00769c0 0x7fd0d019e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:46.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.041+0000 7fd0ce800640 1 -- 172.21.15.28:0/1059772334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0d0077c00 msgr2=0x7fd0d019ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:46.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.041+0000 7fd0ce800640 1 --2- 172.21.15.28:0/1059772334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0d0077c00 0x7fd0d019ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:46.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.041+0000 7fd0ce800640 1 -- 172.21.15.28:0/1059772334 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0d01a6460 con 0x7fd0d00785a0 2024-04-23T14:43:46.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.041+0000 7fd0c7e00640 1 --2- 172.21.15.28:0/1059772334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0d0077c00 0x7fd0d019ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:43:46.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.041+0000 7fd0ce800640 1 --2- 172.21.15.28:0/1059772334 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0d00785a0 0x7fd0d01a5df0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd0c800da10 tx=0x7fd0c800ded0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:46.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.041+0000 7fd0c6a00640 1 -- 172.21.15.28:0/1059772334 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c800bd70 con 0x7fd0d00785a0 2024-04-23T14:43:46.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.041+0000 7fd0c6a00640 1 -- 172.21.15.28:0/1059772334 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0c8003e50 con 0x7fd0d00785a0 2024-04-23T14:43:46.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.041+0000 7fd0cf200640 1 -- 172.21.15.28:0/1059772334 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0d01a6750 con 0x7fd0d00785a0 2024-04-23T14:43:46.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.041+0000 7fd0c6a00640 1 -- 172.21.15.28:0/1059772334 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c80106a0 con 0x7fd0d00785a0 2024-04-23T14:43:46.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.042+0000 7fd0cf200640 1 -- 172.21.15.28:0/1059772334 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0d01a6ca0 con 0x7fd0d00785a0 2024-04-23T14:43:46.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.043+0000 7fd0cf200640 1 -- 172.21.15.28:0/1059772334 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0d0110ab0 con 0x7fd0d00785a0 2024-04-23T14:43:46.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.045+0000 7fd0c6a00640 1 -- 172.21.15.28:0/1059772334 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fd0c8003970 con 0x7fd0d00785a0 2024-04-23T14:43:46.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.046+0000 7fd0c6a00640 1 --2- 172.21.15.28:0/1059772334 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd0b40a3c30 0x7fd0b40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:46.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.046+0000 7fd0c6a00640 1 -- 172.21.15.28:0/1059772334 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7fd0c80d0780 con 0x7fd0d00785a0 2024-04-23T14:43:46.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.046+0000 7fd0cde00640 1 --2- 172.21.15.28:0/1059772334 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd0b40a3c30 0x7fd0b40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:46.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.047+0000 7fd0cde00640 1 --2- 172.21.15.28:0/1059772334 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd0b40a3c30 0x7fd0b40a60f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd0c0031440 tx=0x7fd0c00315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:46.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.051+0000 7fd0c6a00640 1 -- 172.21.15.28:0/1059772334 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd0c809d120 con 0x7fd0d00785a0 2024-04-23T14:43:46.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.260+0000 7fd0cf200640 1 -- 172.21.15.28:0/1059772334 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd0d0062c00 con 0x7fd0d00785a0 2024-04-23T14:43:46.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.261+0000 7fd0c6a00640 1 -- 172.21.15.28:0/1059772334 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7fd0c809ca20 con 0x7fd0d00785a0 2024-04-23T14:43:46.262 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-23T14:43:46.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.263+0000 7fd0cf200640 1 -- 172.21.15.28:0/1059772334 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd0b40a3c30 msgr2=0x7fd0b40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:46.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.263+0000 7fd0cf200640 1 --2- 172.21.15.28:0/1059772334 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd0b40a3c30 0x7fd0b40a60f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd0c0031440 tx=0x7fd0c00315b0 comp rx=0 tx=0).stop 2024-04-23T14:43:46.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.263+0000 7fd0cf200640 1 -- 172.21.15.28:0/1059772334 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0d00785a0 msgr2=0x7fd0d01a5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:46.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.263+0000 7fd0cf200640 1 --2- 172.21.15.28:0/1059772334 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0d00785a0 0x7fd0d01a5df0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd0c800da10 tx=0x7fd0c800ded0 comp rx=0 tx=0).stop 2024-04-23T14:43:46.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.264+0000 7fd0cf200640 1 -- 172.21.15.28:0/1059772334 shutdown_connections 2024-04-23T14:43:46.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.264+0000 7fd0cf200640 1 --2- 172.21.15.28:0/1059772334 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0d00785a0 0x7fd0d01a5df0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:46.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.264+0000 7fd0cf200640 1 --2- 172.21.15.28:0/1059772334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0d0077c00 0x7fd0d019ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:46.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.264+0000 7fd0cf200640 1 --2- 172.21.15.28:0/1059772334 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd0b40a3c30 0x7fd0b40a60f0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:46.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.264+0000 7fd0cf200640 1 --2- 172.21.15.28:0/1059772334 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0d00769c0 0x7fd0d019e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:46.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.264+0000 7fd0cf200640 1 -- 172.21.15.28:0/1059772334 >> 172.21.15.28:0/1059772334 conn(0x7fd0d00ff6b0 msgr2=0x7fd0d01011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:46.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.264+0000 7fd0cf200640 1 -- 172.21.15.28:0/1059772334 shutdown_connections 2024-04-23T14:43:46.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.264+0000 7fd0cf200640 1 -- 172.21.15.28:0/1059772334 wait complete. 2024-04-23T14:43:46.285 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T14:43:46.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.503+0000 7fb661e00640 1 -- 172.21.15.28:0/3582154234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb65c10a0f0 msgr2=0x7fb65c110980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:46.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.503+0000 7fb661e00640 1 --2- 172.21.15.28:0/3582154234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb65c10a0f0 0x7fb65c110980 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb654009be0 tx=0x7fb65402a6c0 comp rx=0 tx=0).stop 2024-04-23T14:43:46.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.504+0000 7fb661e00640 1 -- 172.21.15.28:0/3582154234 shutdown_connections 2024-04-23T14:43:46.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.504+0000 7fb661e00640 1 --2- 172.21.15.28:0/3582154234 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb65c109730 0x7fb65c109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:46.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.504+0000 7fb661e00640 1 --2- 172.21.15.28:0/3582154234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb65c10a0f0 0x7fb65c110980 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:46.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.504+0000 7fb661e00640 1 --2- 172.21.15.28:0/3582154234 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb65c108540 0x7fb65c108960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:46.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.504+0000 7fb661e00640 1 -- 172.21.15.28:0/3582154234 >> 172.21.15.28:0/3582154234 conn(0x7fb65c0774d0 msgr2=0x7fb65c079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:46.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.504+0000 7fb661e00640 1 -- 172.21.15.28:0/3582154234 shutdown_connections 2024-04-23T14:43:46.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.504+0000 7fb661e00640 1 -- 172.21.15.28:0/3582154234 wait complete. 2024-04-23T14:43:46.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.505+0000 7fb661e00640 1 Processor -- start 2024-04-23T14:43:46.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.505+0000 7fb661e00640 1 -- start start 2024-04-23T14:43:46.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.505+0000 7fb661e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb65c108540 0x7fb65c19e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:46.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.505+0000 7fb661e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb65c109730 0x7fb65c19ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:46.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.505+0000 7fb661e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb65c10a0f0 0x7fb65c1a5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:46.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.505+0000 7fb661e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb65c114180 con 0x7fb65c108540 2024-04-23T14:43:46.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.505+0000 7fb661e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb65c114000 con 0x7fb65c109730 2024-04-23T14:43:46.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.505+0000 7fb661e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb65c114300 con 0x7fb65c10a0f0 2024-04-23T14:43:46.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.506+0000 7fb65be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb65c109730 0x7fb65c19ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:46.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.506+0000 7fb661400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb65c10a0f0 0x7fb65c1a5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:46.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.506+0000 7fb660a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb65c108540 0x7fb65c19e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:46.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.506+0000 7fb65be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb65c109730 0x7fb65c19ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:56398/0 (socket says 172.21.15.28:56398) 2024-04-23T14:43:46.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.506+0000 7fb661400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb65c10a0f0 0x7fb65c1a5d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:50132/0 (socket says 172.21.15.28:50132) 2024-04-23T14:43:46.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.506+0000 7fb65be00640 1 -- 172.21.15.28:0/1813840279 learned_addr learned my addr 172.21.15.28:0/1813840279 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:46.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.506+0000 7fb65be00640 1 -- 172.21.15.28:0/1813840279 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb65c10a0f0 msgr2=0x7fb65c1a5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:46.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.506+0000 7fb65be00640 1 --2- 172.21.15.28:0/1813840279 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb65c10a0f0 0x7fb65c1a5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:46.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.506+0000 7fb65be00640 1 -- 172.21.15.28:0/1813840279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb65c108540 msgr2=0x7fb65c19e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:46.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.506+0000 7fb65be00640 1 --2- 172.21.15.28:0/1813840279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb65c108540 0x7fb65c19e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:46.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.506+0000 7fb65be00640 1 -- 172.21.15.28:0/1813840279 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb65c1a63a0 con 0x7fb65c109730 2024-04-23T14:43:46.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.506+0000 7fb661400640 1 --2- 172.21.15.28:0/1813840279 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb65c10a0f0 0x7fb65c1a5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:43:46.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.506+0000 7fb660a00640 1 --2- 172.21.15.28:0/1813840279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb65c108540 0x7fb65c19e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:43:46.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.507+0000 7fb65be00640 1 --2- 172.21.15.28:0/1813840279 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb65c109730 0x7fb65c19ecc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb64800ea60 tx=0x7fb64800ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:46.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.507+0000 7fb659600640 1 -- 172.21.15.28:0/1813840279 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb64800cd20 con 0x7fb65c109730 2024-04-23T14:43:46.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.507+0000 7fb659600640 1 -- 172.21.15.28:0/1813840279 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6480035f0 con 0x7fb65c109730 2024-04-23T14:43:46.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.507+0000 7fb661e00640 1 -- 172.21.15.28:0/1813840279 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb65c1a6690 con 0x7fb65c109730 2024-04-23T14:43:46.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.507+0000 7fb659600640 1 -- 172.21.15.28:0/1813840279 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb648010690 con 0x7fb65c109730 2024-04-23T14:43:46.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.507+0000 7fb661e00640 1 -- 172.21.15.28:0/1813840279 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb65c1a6be0 con 0x7fb65c109730 2024-04-23T14:43:46.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.508+0000 7fb661e00640 1 -- 172.21.15.28:0/1813840279 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb65c10f860 con 0x7fb65c109730 2024-04-23T14:43:46.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.510+0000 7fb659600640 1 -- 172.21.15.28:0/1813840279 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb6480037a0 con 0x7fb65c109730 2024-04-23T14:43:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.512+0000 7fb659600640 1 --2- 172.21.15.28:0/1813840279 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb6440a3bb0 0x7fb6440a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.512+0000 7fb659600640 1 -- 172.21.15.28:0/1813840279 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7fb6480d0750 con 0x7fb65c109730 2024-04-23T14:43:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.512+0000 7fb660a00640 1 --2- 172.21.15.28:0/1813840279 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb6440a3bb0 0x7fb6440a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.513+0000 7fb660a00640 1 --2- 172.21.15.28:0/1813840279 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb6440a3bb0 0x7fb6440a6070 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb65c109590 tx=0x7fb650009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:46.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.516+0000 7fb659600640 1 -- 172.21.15.28:0/1813840279 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb64809d0f0 con 0x7fb65c109730 2024-04-23T14:43:46.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.728+0000 7fb661e00640 1 -- 172.21.15.28:0/1813840279 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb65c062c00 con 0x7fb65c109730 2024-04-23T14:43:46.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.729+0000 7fb659600640 1 -- 172.21.15.28:0/1813840279 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7fb64809c9f0 con 0x7fb65c109730 2024-04-23T14:43:46.730 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-23T14:43:46.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.731+0000 7fb661e00640 1 -- 172.21.15.28:0/1813840279 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb6440a3bb0 msgr2=0x7fb6440a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:46.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.731+0000 7fb661e00640 1 --2- 172.21.15.28:0/1813840279 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb6440a3bb0 0x7fb6440a6070 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb65c109590 tx=0x7fb650009260 comp rx=0 tx=0).stop 2024-04-23T14:43:46.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.731+0000 7fb661e00640 1 -- 172.21.15.28:0/1813840279 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb65c109730 msgr2=0x7fb65c19ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:46.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.731+0000 7fb661e00640 1 --2- 172.21.15.28:0/1813840279 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb65c109730 0x7fb65c19ecc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb64800ea60 tx=0x7fb64800ee60 comp rx=0 tx=0).stop 2024-04-23T14:43:46.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.732+0000 7fb661e00640 1 -- 172.21.15.28:0/1813840279 shutdown_connections 2024-04-23T14:43:46.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.732+0000 7fb661e00640 1 --2- 172.21.15.28:0/1813840279 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb6440a3bb0 0x7fb6440a6070 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:46.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.732+0000 7fb661e00640 1 --2- 172.21.15.28:0/1813840279 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb65c109730 0x7fb65c19ecc0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:46.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.732+0000 7fb661e00640 1 --2- 172.21.15.28:0/1813840279 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb65c10a0f0 0x7fb65c1a5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:46.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.732+0000 7fb661e00640 1 --2- 172.21.15.28:0/1813840279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb65c108540 0x7fb65c19e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:46.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.732+0000 7fb661e00640 1 -- 172.21.15.28:0/1813840279 >> 172.21.15.28:0/1813840279 conn(0x7fb65c0774d0 msgr2=0x7fb65c10ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:46.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.732+0000 7fb661e00640 1 -- 172.21.15.28:0/1813840279 shutdown_connections 2024-04-23T14:43:46.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.732+0000 7fb661e00640 1 -- 172.21.15.28:0/1813840279 wait complete. 2024-04-23T14:43:46.751 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T14:43:46.827 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:46 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1286241797' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:43:46.827 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:46 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1059772334' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:43:46.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:46 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1286241797' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:43:46.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:46 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1059772334' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:43:46.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.970+0000 7f4acca00640 1 -- 172.21.15.28:0/4006019994 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ac81052c0 msgr2=0x7f4ac81076b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:46.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.970+0000 7f4acca00640 1 --2- 172.21.15.28:0/4006019994 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ac81052c0 0x7f4ac81076b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4ab4009860 tx=0x7f4ab402a340 comp rx=0 tx=0).stop 2024-04-23T14:43:46.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.970+0000 7f4acca00640 1 -- 172.21.15.28:0/4006019994 shutdown_connections 2024-04-23T14:43:46.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.970+0000 7f4acca00640 1 --2- 172.21.15.28:0/4006019994 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ac81052c0 0x7f4ac81076b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:46.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.970+0000 7f4acca00640 1 --2- 172.21.15.28:0/4006019994 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ac8107bf0 0x7f4ac810a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:46.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.970+0000 7f4acca00640 1 --2- 172.21.15.28:0/4006019994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ac8102990 0x7f4ac8104d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:46.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.970+0000 7f4acca00640 1 -- 172.21.15.28:0/4006019994 >> 172.21.15.28:0/4006019994 conn(0x7f4ac80fc3a0 msgr2=0x7f4ac80fe7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:46.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.970+0000 7f4acca00640 1 -- 172.21.15.28:0/4006019994 shutdown_connections 2024-04-23T14:43:46.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.970+0000 7f4acca00640 1 -- 172.21.15.28:0/4006019994 wait complete. 2024-04-23T14:43:46.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.971+0000 7f4acca00640 1 Processor -- start 2024-04-23T14:43:46.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.971+0000 7f4acca00640 1 -- start start 2024-04-23T14:43:46.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.972+0000 7f4acca00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ac8102990 0x7f4ac8196060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:46.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.972+0000 7f4acca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ac81052c0 0x7f4ac81965a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:46.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.972+0000 7f4acca00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ac8107bf0 0x7f4ac81a5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:46.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.972+0000 7f4acca00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac810db70 con 0x7f4ac81052c0 2024-04-23T14:43:46.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.972+0000 7f4acca00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac810d9f0 con 0x7f4ac8107bf0 2024-04-23T14:43:46.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.972+0000 7f4acca00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac810dcf0 con 0x7f4ac8102990 2024-04-23T14:43:46.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.972+0000 7f4ac6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ac81052c0 0x7f4ac81965a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:46.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.972+0000 7f4ac6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ac81052c0 0x7f4ac81965a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36756/0 (socket says 172.21.15.28:36756) 2024-04-23T14:43:46.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.972+0000 7f4ac6a00640 1 -- 172.21.15.28:0/2188928170 learned_addr learned my addr 172.21.15.28:0/2188928170 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:46.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.972+0000 7f4ac7400640 1 --2- 172.21.15.28:0/2188928170 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ac8102990 0x7f4ac8196060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:46.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.972+0000 7f4ac7e00640 1 --2- 172.21.15.28:0/2188928170 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ac8107bf0 0x7f4ac81a5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:46.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.972+0000 7f4ac6a00640 1 -- 172.21.15.28:0/2188928170 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ac8102990 msgr2=0x7f4ac8196060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:46.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.972+0000 7f4ac6a00640 1 --2- 172.21.15.28:0/2188928170 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ac8102990 0x7f4ac8196060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:46.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.972+0000 7f4ac6a00640 1 -- 172.21.15.28:0/2188928170 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ac8107bf0 msgr2=0x7f4ac81a5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:46.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.972+0000 7f4ac6a00640 1 --2- 172.21.15.28:0/2188928170 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ac8107bf0 0x7f4ac81a5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:46.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.972+0000 7f4ac6a00640 1 -- 172.21.15.28:0/2188928170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ac81a6490 con 0x7f4ac81052c0 2024-04-23T14:43:46.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.973+0000 7f4ac6a00640 1 --2- 172.21.15.28:0/2188928170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ac81052c0 0x7f4ac81965a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4ab4035040 tx=0x7f4ab4033d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:46.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.973+0000 7f4abfe00640 1 -- 172.21.15.28:0/2188928170 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab4037040 con 0x7f4ac81052c0 2024-04-23T14:43:46.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.973+0000 7f4acca00640 1 -- 172.21.15.28:0/2188928170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ac81a6720 con 0x7f4ac81052c0 2024-04-23T14:43:46.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.973+0000 7f4abfe00640 1 -- 172.21.15.28:0/2188928170 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ab4030850 con 0x7f4ac81052c0 2024-04-23T14:43:46.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.973+0000 7f4abfe00640 1 -- 172.21.15.28:0/2188928170 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab4004920 con 0x7f4ac81052c0 2024-04-23T14:43:46.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.973+0000 7f4acca00640 1 -- 172.21.15.28:0/2188928170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ac81a6c60 con 0x7f4ac81052c0 2024-04-23T14:43:46.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.974+0000 7f4acca00640 1 -- 172.21.15.28:0/2188928170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ac8109740 con 0x7f4ac81052c0 2024-04-23T14:43:46.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.976+0000 7f4abfe00640 1 -- 172.21.15.28:0/2188928170 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f4ab4004050 con 0x7f4ac81052c0 2024-04-23T14:43:46.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.978+0000 7f4abfe00640 1 --2- 172.21.15.28:0/2188928170 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f4aac0a3bb0 0x7f4aac0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:46.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.978+0000 7f4abfe00640 1 -- 172.21.15.28:0/2188928170 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7f4ab40f0640 con 0x7f4ac81052c0 2024-04-23T14:43:46.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.978+0000 7f4ac7400640 1 --2- 172.21.15.28:0/2188928170 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f4aac0a3bb0 0x7f4aac0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:46.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.979+0000 7f4ac7400640 1 --2- 172.21.15.28:0/2188928170 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f4aac0a3bb0 0x7f4aac0a6070 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8009bd0 tx=0x7f4ab8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:46.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:46.982+0000 7f4abfe00640 1 -- 172.21.15.28:0/2188928170 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4ab40bcfe0 con 0x7f4ac81052c0 2024-04-23T14:43:47.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:46 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1286241797' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:43:47.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:46 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1059772334' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:43:47.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.191+0000 7f4acca00640 1 -- 172.21.15.28:0/2188928170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4ac802d870 con 0x7f4ac81052c0 2024-04-23T14:43:47.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.192+0000 7f4abfe00640 1 -- 172.21.15.28:0/2188928170 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7f4ab40bc8e0 con 0x7f4ac81052c0 2024-04-23T14:43:47.193 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-23T14:43:47.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.194+0000 7f4acca00640 1 -- 172.21.15.28:0/2188928170 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f4aac0a3bb0 msgr2=0x7f4aac0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:47.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.194+0000 7f4acca00640 1 --2- 172.21.15.28:0/2188928170 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f4aac0a3bb0 0x7f4aac0a6070 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8009bd0 tx=0x7f4ab8005f90 comp rx=0 tx=0).stop 2024-04-23T14:43:47.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.194+0000 7f4acca00640 1 -- 172.21.15.28:0/2188928170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ac81052c0 msgr2=0x7f4ac81965a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:47.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.194+0000 7f4acca00640 1 --2- 172.21.15.28:0/2188928170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ac81052c0 0x7f4ac81965a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4ab4035040 tx=0x7f4ab4033d90 comp rx=0 tx=0).stop 2024-04-23T14:43:47.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.194+0000 7f4acca00640 1 -- 172.21.15.28:0/2188928170 shutdown_connections 2024-04-23T14:43:47.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.195+0000 7f4acca00640 1 --2- 172.21.15.28:0/2188928170 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f4aac0a3bb0 0x7f4aac0a6070 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:47.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.195+0000 7f4acca00640 1 --2- 172.21.15.28:0/2188928170 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ac8107bf0 0x7f4ac81a5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:47.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.195+0000 7f4acca00640 1 --2- 172.21.15.28:0/2188928170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ac81052c0 0x7f4ac81965a0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:47.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.195+0000 7f4acca00640 1 --2- 172.21.15.28:0/2188928170 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ac8102990 0x7f4ac8196060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:47.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.195+0000 7f4acca00640 1 -- 172.21.15.28:0/2188928170 >> 172.21.15.28:0/2188928170 conn(0x7f4ac80fc3a0 msgr2=0x7f4ac81020c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:47.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.195+0000 7f4acca00640 1 -- 172.21.15.28:0/2188928170 shutdown_connections 2024-04-23T14:43:47.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.195+0000 7f4acca00640 1 -- 172.21.15.28:0/2188928170 wait complete. 2024-04-23T14:43:47.215 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:43:47.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.428+0000 7f4970c00640 1 -- 172.21.15.28:0/1759437779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f496c1029c0 msgr2=0x7f496c104db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:47.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.428+0000 7f4970c00640 1 --2- 172.21.15.28:0/1759437779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f496c1029c0 0x7f496c104db0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f495c009900 tx=0x7f495c02a410 comp rx=0 tx=0).stop 2024-04-23T14:43:47.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.429+0000 7f4970c00640 1 -- 172.21.15.28:0/1759437779 shutdown_connections 2024-04-23T14:43:47.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.429+0000 7f4970c00640 1 --2- 172.21.15.28:0/1759437779 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f496c1052f0 0x7f496c1076e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:47.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.429+0000 7f4970c00640 1 --2- 172.21.15.28:0/1759437779 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f496c107c20 0x7f496c10a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:47.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.429+0000 7f4970c00640 1 --2- 172.21.15.28:0/1759437779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f496c1029c0 0x7f496c104db0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:47.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.429+0000 7f4970c00640 1 -- 172.21.15.28:0/1759437779 >> 172.21.15.28:0/1759437779 conn(0x7f496c0fc2d0 msgr2=0x7f496c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:47.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.429+0000 7f4970c00640 1 -- 172.21.15.28:0/1759437779 shutdown_connections 2024-04-23T14:43:47.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.429+0000 7f4970c00640 1 -- 172.21.15.28:0/1759437779 wait complete. 2024-04-23T14:43:47.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.430+0000 7f4970c00640 1 Processor -- start 2024-04-23T14:43:47.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.430+0000 7f4970c00640 1 -- start start 2024-04-23T14:43:47.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.430+0000 7f4970c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f496c1029c0 0x7f496c19a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:47.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.430+0000 7f4970c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f496c1052f0 0x7f496c19a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:47.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.430+0000 7f4970c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f496c107c20 0x7f496c1a1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:47.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.430+0000 7f4970c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f496c10db80 con 0x7f496c1029c0 2024-04-23T14:43:47.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.430+0000 7f4970c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f496c10da00 con 0x7f496c1052f0 2024-04-23T14:43:47.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.430+0000 7f4970c00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f496c10dd00 con 0x7f496c107c20 2024-04-23T14:43:47.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.431+0000 7f496b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f496c1029c0 0x7f496c19a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:47.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.431+0000 7f496b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f496c1029c0 0x7f496c19a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36772/0 (socket says 172.21.15.28:36772) 2024-04-23T14:43:47.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.431+0000 7f496b400640 1 -- 172.21.15.28:0/3427119120 learned_addr learned my addr 172.21.15.28:0/3427119120 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:47.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.431+0000 7f4962a00640 1 --2- 172.21.15.28:0/3427119120 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f496c1052f0 0x7f496c19a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:47.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.431+0000 7f496be00640 1 --2- 172.21.15.28:0/3427119120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f496c107c20 0x7f496c1a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:47.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.431+0000 7f496b400640 1 -- 172.21.15.28:0/3427119120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f496c107c20 msgr2=0x7f496c1a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:47.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.431+0000 7f496b400640 1 --2- 172.21.15.28:0/3427119120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f496c107c20 0x7f496c1a1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:47.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.431+0000 7f496b400640 1 -- 172.21.15.28:0/3427119120 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f496c1052f0 msgr2=0x7f496c19a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:47.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.431+0000 7f496b400640 1 --2- 172.21.15.28:0/3427119120 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f496c1052f0 0x7f496c19a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:47.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.431+0000 7f496b400640 1 -- 172.21.15.28:0/3427119120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f496c1a1ff0 con 0x7f496c1029c0 2024-04-23T14:43:47.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.431+0000 7f496be00640 1 --2- 172.21.15.28:0/3427119120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f496c107c20 0x7f496c1a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:43:47.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.431+0000 7f4962a00640 1 --2- 172.21.15.28:0/3427119120 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f496c1052f0 0x7f496c19a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:43:47.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.432+0000 7f496b400640 1 --2- 172.21.15.28:0/3427119120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f496c1029c0 0x7f496c19a3d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f495c031440 tx=0x7f495c032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:47.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.432+0000 7f4968c00640 1 -- 172.21.15.28:0/3427119120 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f495c03f030 con 0x7f496c1029c0 2024-04-23T14:43:47.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.432+0000 7f4968c00640 1 -- 172.21.15.28:0/3427119120 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f495c003ce0 con 0x7f496c1029c0 2024-04-23T14:43:47.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.432+0000 7f4970c00640 1 -- 172.21.15.28:0/3427119120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f496c1a2280 con 0x7f496c1029c0 2024-04-23T14:43:47.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.432+0000 7f4968c00640 1 -- 172.21.15.28:0/3427119120 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f495c035070 con 0x7f496c1029c0 2024-04-23T14:43:47.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.432+0000 7f4970c00640 1 -- 172.21.15.28:0/3427119120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f496c1a27c0 con 0x7f496c1029c0 2024-04-23T14:43:47.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.433+0000 7f4970c00640 1 -- 172.21.15.28:0/3427119120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f496c109710 con 0x7f496c1029c0 2024-04-23T14:43:47.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.435+0000 7f4968c00640 1 -- 172.21.15.28:0/3427119120 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f495c042030 con 0x7f496c1029c0 2024-04-23T14:43:47.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.437+0000 7f4968c00640 1 --2- 172.21.15.28:0/3427119120 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f49500a3c30 0x7f49500a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:47.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.437+0000 7f4968c00640 1 -- 172.21.15.28:0/3427119120 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7f495c0f0890 con 0x7f496c1029c0 2024-04-23T14:43:47.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.437+0000 7f4962a00640 1 --2- 172.21.15.28:0/3427119120 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f49500a3c30 0x7f49500a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:47.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.438+0000 7f4962a00640 1 --2- 172.21.15.28:0/3427119120 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f49500a3c30 0x7f49500a60f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f496c19b8e0 tx=0x7f4954005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:47.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.441+0000 7f4968c00640 1 -- 172.21.15.28:0/3427119120 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f495c0bcbe0 con 0x7f496c1029c0 2024-04-23T14:43:47.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.695+0000 7f4970c00640 1 -- 172.21.15.28:0/3427119120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f496c02d620 con 0x7f496c1029c0 2024-04-23T14:43:47.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.696+0000 7f4968c00640 1 -- 172.21.15.28:0/3427119120 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7f495c0c0ee0 con 0x7f496c1029c0 2024-04-23T14:43:47.698 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:43:47.698 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:43:47.698 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:43:47.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.699+0000 7f4970c00640 1 -- 172.21.15.28:0/3427119120 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f49500a3c30 msgr2=0x7f49500a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:47.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.699+0000 7f4970c00640 1 --2- 172.21.15.28:0/3427119120 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f49500a3c30 0x7f49500a60f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f496c19b8e0 tx=0x7f4954005f90 comp rx=0 tx=0).stop 2024-04-23T14:43:47.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.699+0000 7f4970c00640 1 -- 172.21.15.28:0/3427119120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f496c1029c0 msgr2=0x7f496c19a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:47.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.699+0000 7f4970c00640 1 --2- 172.21.15.28:0/3427119120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f496c1029c0 0x7f496c19a3d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f495c031440 tx=0x7f495c032ce0 comp rx=0 tx=0).stop 2024-04-23T14:43:47.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.699+0000 7f4970c00640 1 -- 172.21.15.28:0/3427119120 shutdown_connections 2024-04-23T14:43:47.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.699+0000 7f4970c00640 1 --2- 172.21.15.28:0/3427119120 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f49500a3c30 0x7f49500a60f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:47.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.699+0000 7f4970c00640 1 --2- 172.21.15.28:0/3427119120 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f496c1052f0 0x7f496c19a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:47.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.699+0000 7f4970c00640 1 --2- 172.21.15.28:0/3427119120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f496c107c20 0x7f496c1a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:47.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.699+0000 7f4970c00640 1 --2- 172.21.15.28:0/3427119120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f496c1029c0 0x7f496c19a3d0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:47.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.699+0000 7f4970c00640 1 -- 172.21.15.28:0/3427119120 >> 172.21.15.28:0/3427119120 conn(0x7f496c0fc2d0 msgr2=0x7f496c102130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:47.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.699+0000 7f4970c00640 1 -- 172.21.15.28:0/3427119120 shutdown_connections 2024-04-23T14:43:47.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.699+0000 7f4970c00640 1 -- 172.21.15.28:0/3427119120 wait complete. 2024-04-23T14:43:47.721 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:43:47.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:47 smithi038 ceph-mon[32348]: pgmap v379: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:43:47.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:47 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1813840279' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T14:43:47.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:47 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2188928170' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T14:43:47.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:47 smithi028 ceph-mon[27810]: pgmap v379: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:43:47.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:47 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1813840279' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T14:43:47.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:47 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2188928170' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T14:43:47.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.937+0000 7fd576800640 1 -- 172.21.15.28:0/949701013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5701054b0 msgr2=0x7fd5701078a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:47.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.937+0000 7fd576800640 1 --2- 172.21.15.28:0/949701013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5701054b0 0x7fd5701078a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd558009900 tx=0x7fd55802a3f0 comp rx=0 tx=0).stop 2024-04-23T14:43:47.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.937+0000 7fd576800640 1 -- 172.21.15.28:0/949701013 shutdown_connections 2024-04-23T14:43:47.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.937+0000 7fd576800640 1 --2- 172.21.15.28:0/949701013 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd570107de0 0x7fd57010a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:47.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.937+0000 7fd576800640 1 --2- 172.21.15.28:0/949701013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5701054b0 0x7fd5701078a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:47.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.937+0000 7fd576800640 1 --2- 172.21.15.28:0/949701013 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd570102b80 0x7fd570104f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:47.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.937+0000 7fd576800640 1 -- 172.21.15.28:0/949701013 >> 172.21.15.28:0/949701013 conn(0x7fd5700fc4f0 msgr2=0x7fd5700fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:47.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.938+0000 7fd576800640 1 -- 172.21.15.28:0/949701013 shutdown_connections 2024-04-23T14:43:47.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.938+0000 7fd576800640 1 -- 172.21.15.28:0/949701013 wait complete. 2024-04-23T14:43:47.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.939+0000 7fd576800640 1 Processor -- start 2024-04-23T14:43:47.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.939+0000 7fd576800640 1 -- start start 2024-04-23T14:43:47.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.939+0000 7fd576800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd570102b80 0x7fd57019a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:47.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.939+0000 7fd576800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5701054b0 0x7fd57019aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:47.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.939+0000 7fd576800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd570107de0 0x7fd5701a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:47.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.939+0000 7fd576800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd57010dd20 con 0x7fd570107de0 2024-04-23T14:43:47.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.939+0000 7fd576800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd57010dba0 con 0x7fd5701054b0 2024-04-23T14:43:47.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.939+0000 7fd576800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd57010dea0 con 0x7fd570102b80 2024-04-23T14:43:47.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.940+0000 7fd56fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd570107de0 0x7fd5701a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:47.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.940+0000 7fd56f400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd570102b80 0x7fd57019a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:47.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.940+0000 7fd567e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5701054b0 0x7fd57019aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:47.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.940+0000 7fd56fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd570107de0 0x7fd5701a1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45130/0 (socket says 172.21.15.28:45130) 2024-04-23T14:43:47.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.940+0000 7fd56f400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd570102b80 0x7fd57019a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:40700/0 (socket says 172.21.15.28:40700) 2024-04-23T14:43:47.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.940+0000 7fd56fe00640 1 -- 172.21.15.28:0/3727765835 learned_addr learned my addr 172.21.15.28:0/3727765835 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:47.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.940+0000 7fd56f400640 1 -- 172.21.15.28:0/3727765835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5701054b0 msgr2=0x7fd57019aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:47.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.940+0000 7fd56f400640 1 --2- 172.21.15.28:0/3727765835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5701054b0 0x7fd57019aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:47.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.940+0000 7fd56f400640 1 -- 172.21.15.28:0/3727765835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd570107de0 msgr2=0x7fd5701a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:47.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.940+0000 7fd56f400640 1 --2- 172.21.15.28:0/3727765835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd570107de0 0x7fd5701a1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:47.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.940+0000 7fd56f400640 1 -- 172.21.15.28:0/3727765835 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5701a21b0 con 0x7fd570102b80 2024-04-23T14:43:47.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.940+0000 7fd56f400640 1 --2- 172.21.15.28:0/3727765835 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd570102b80 0x7fd57019a590 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd56000e9f0 tx=0x7fd56000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:47.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.941+0000 7fd56cc00640 1 -- 172.21.15.28:0/3727765835 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd56000cd80 con 0x7fd570102b80 2024-04-23T14:43:47.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.941+0000 7fd56cc00640 1 -- 172.21.15.28:0/3727765835 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5600035f0 con 0x7fd570102b80 2024-04-23T14:43:47.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.941+0000 7fd576800640 1 -- 172.21.15.28:0/3727765835 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5701a24a0 con 0x7fd570102b80 2024-04-23T14:43:47.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.941+0000 7fd56cc00640 1 -- 172.21.15.28:0/3727765835 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd560010690 con 0x7fd570102b80 2024-04-23T14:43:47.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.941+0000 7fd576800640 1 -- 172.21.15.28:0/3727765835 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5701a29c0 con 0x7fd570102b80 2024-04-23T14:43:47.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.942+0000 7fd576800640 1 -- 172.21.15.28:0/3727765835 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5701098d0 con 0x7fd570102b80 2024-04-23T14:43:47.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.944+0000 7fd56cc00640 1 -- 172.21.15.28:0/3727765835 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fd560003b30 con 0x7fd570102b80 2024-04-23T14:43:47.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.945+0000 7fd56cc00640 1 --2- 172.21.15.28:0/3727765835 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd5540a3bb0 0x7fd5540a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:47.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.946+0000 7fd56cc00640 1 -- 172.21.15.28:0/3727765835 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7fd5600d06f0 con 0x7fd570102b80 2024-04-23T14:43:47.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.946+0000 7fd567e00640 1 --2- 172.21.15.28:0/3727765835 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd5540a3bb0 0x7fd5540a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:47.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.946+0000 7fd567e00640 1 --2- 172.21.15.28:0/3727765835 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd5540a3bb0 0x7fd5540a6070 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd55802a9e0 tx=0x7fd5580315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:47.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:47.949+0000 7fd56cc00640 1 -- 172.21.15.28:0/3727765835 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd56009d090 con 0x7fd570102b80 2024-04-23T14:43:48.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:47 smithi105 ceph-mon[32555]: pgmap v379: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:43:48.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:47 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1813840279' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T14:43:48.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:47 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2188928170' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T14:43:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.170+0000 7fd576800640 1 -- 172.21.15.28:0/3727765835 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd57002d620 con 0x7fd570102b80 2024-04-23T14:43:48.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.173+0000 7fd56cc00640 1 -- 172.21.15.28:0/3727765835 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22651 (secure 0 0 0) 0x7fd56009c990 con 0x7fd570102b80 2024-04-23T14:43:48.174 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:43:48.174 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":87,"fsid":"1ee6e996-017e-11ef-bc93-c7b262605968","created":"2024-04-23T14:32:34.581082+0000","modified":"2024-04-23T14:42:43.429141+0000","last_up_change":"2024-04-23T14:40:43.375846+0000","last_in_change":"2024-04-23T14:40:23.576939+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:36:27.698112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:41:37.030341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:41:38.444742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e0b427bc-be65-4543-99cc-2f53ad98ddad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6803","nonce":2993445336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6805","nonce":2993445336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6809","nonce":2993445336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6807","nonce":2993445336}]},"public_addr":"172.21.15.28:6803/2993445336","cluster_addr":"172.21.15.28:6805/2993445336","heartbeat_back_addr":"172.21.15.28:6809/2993445336","heartbeat_front_addr":"172.21.15.28:6807/2993445336","state":["exists","up"]},{"osd":1,"uuid":"ef527ec9-1bf3-4bb0-8df3-f5a015af557c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6801","nonce":4024093383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6803","nonce":4024093383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6807","nonce":4024093383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6805","nonce":4024093383}]},"public_addr":"172.21.15.38:6801/4024093383","cluster_addr":"172.21.15.38:6803/4024093383","heartbeat_back_addr":"172.21.15.38:6807/4024093383","heartbeat_front_addr":"172.21.15.38:6805/4024093383","state":["exists","up"]},{"osd":2,"uuid":"b33b23d9-adce-4c11-b304-2e117bdf6bfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6801","nonce":570739029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6803","nonce":570739029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6807","nonce":570739029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6805","nonce":570739029}]},"public_addr":"172.21.15.105:6801/570739029","cluster_addr":"172.21.15.105:6803/570739029","heartbeat_back_addr":"172.21.15.105:6807/570739029","heartbeat_front_addr":"172.21.15.105:6805/570739029","state":["exists","up"]},{"osd":3,"uuid":"28bd5608-b426-434e-9db1-bdaa9828bdb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6811","nonce":2480400247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6813","nonce":2480400247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6817","nonce":2480400247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6815","nonce":2480400247}]},"public_addr":"172.21.15.28:6811/2480400247","cluster_addr":"172.21.15.28:6813/2480400247","heartbeat_back_addr":"172.21.15.28:6817/2480400247","heartbeat_front_addr":"172.21.15.28:6815/2480400247","state":["exists","up"]},{"osd":4,"uuid":"e5d1c89b-025f-4d57-8807-59d75c787b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6809","nonce":3772189246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6811","nonce":3772189246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6815","nonce":3772189246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6813","nonce":3772189246}]},"public_addr":"172.21.15.38:6809/3772189246","cluster_addr":"172.21.15.38:6811/3772189246","heartbeat_back_addr":"172.21.15.38:6815/3772189246","heartbeat_front_addr":"172.21.15.38:6813/3772189246","state":["exists","up"]},{"osd":5,"uuid":"9cf9cf4c-aafd-4ed3-bfb1-0527710a0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6809","nonce":928755894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6811","nonce":928755894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6815","nonce":928755894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6813","nonce":928755894}]},"public_addr":"172.21.15.105:6809/928755894","cluster_addr":"172.21.15.105:6811/928755894","heartbeat_back_addr":"172.21.15.105:6815/928755894","heartbeat_front_addr":"172.21.15.105:6813/928755894","state":["exists","up"]},{"osd":6,"uuid":"304db525-d24d-4687-bfa0-f92bdf66d1e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6819","nonce":897979441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6821","nonce":897979441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6825","nonce":897979441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6823","nonce":897979441}]},"public_addr":"172.21.15.28:6819/897979441","cluster_addr":"172.21.15.28:6821/897979441","heartbeat_back_addr":"172.21.15.28:6825/897979441","heartbeat_front_addr":"172.21.15.28:6823/897979441","state":["exists","up"]},{"osd":7,"uuid":"08e01d59-03f3-47b1-b45f-35ec5a50992c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6817","nonce":2400251786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6819","nonce":2400251786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6823","nonce":2400251786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6821","nonce":2400251786}]},"public_addr":"172.21.15.38:6817/2400251786","cluster_addr":"172.21.15.38:6819/2400251786","heartbeat_back_addr":"172.21.15.38:6823/2400251786","heartbeat_front_addr":"172.21.15.38:6821/2400251786","state":["exists","up"]},{"osd":8,"uuid":"64ee1e95-0756-48b1-9c00-472d0c3cbd29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6817","nonce":1682312269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6819","nonce":1682312269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6823","nonce":1682312269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6821","nonce":1682312269}]},"public_addr":"172.21.15.105:6817/1682312269","cluster_addr":"172.21.15.105:6819/1682312269","heartbeat_back_addr":"172.21.15.105:6823/1682312269","heartbeat_front_addr":"172.21.15.105:6821/1682312269","state":["exists","up"]},{"osd":9,"uuid":"18f60893-3ffd-48c2-9db5-a67d35611d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6827","nonce":1292942516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6829","nonce":1292942516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6833","nonce":1292942516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6831","nonce":1292942516}]},"public_addr":"172.21.15.28:6827/1292942516","cluster_addr":"172.21.15.28:6829/1292942516","heartbeat_back_addr":"172.21.15.28:6833/1292942516","heartbeat_front_addr":"172.21.15.28:6831/1292942516","state":["exists","up"]},{"osd":10,"uuid":"76c06725-7040-4118-8894-fe808a87f999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6825","nonce":2444616468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6827","nonce":2444616468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6831","nonce":2444616468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6829","nonce":2444616468}]},"public_addr":"172.21.15.38:6825/2444616468","cluster_addr":"172.21.15.38:6827/2444616468","heartbeat_back_addr":"172.21.15.38:6831/2444616468","heartbeat_front_addr":"172.21.15.38:6829/2444616468","state":["exists","up"]},{"osd":11,"uuid":"e8fc9454-5540-4e2e-91bf-3ba251dbd1a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6825","nonce":3400299562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6827","nonce":3400299562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6831","nonce":3400299562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6829","nonce":3400299562}]},"public_addr":"172.21.15.105:6825/3400299562","cluster_addr":"172.21.15.105:6827/3400299562","heartbeat_back_addr":"172.21.15.105:6831/3400299562","heartbeat_front_addr":"172.21.15.105:6829/3400299562","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:35:33.615395+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:00.545293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:24.940211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:57.684887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:24.519061+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:49.010804+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:22.638478+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:49.559104+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:14.011771+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:47.859950+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:16.298445+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:41.619069+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.2d","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/590495740":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/1163172512":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/4290328460":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/4145020172":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/720626958":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/948450262":"2024-04-24T14:33:07.576363+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:43:48.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.175+0000 7fd576800640 1 -- 172.21.15.28:0/3727765835 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd5540a3bb0 msgr2=0x7fd5540a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:48.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.175+0000 7fd576800640 1 --2- 172.21.15.28:0/3727765835 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd5540a3bb0 0x7fd5540a6070 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd55802a9e0 tx=0x7fd5580315b0 comp rx=0 tx=0).stop 2024-04-23T14:43:48.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.175+0000 7fd576800640 1 -- 172.21.15.28:0/3727765835 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd570102b80 msgr2=0x7fd57019a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:48.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.175+0000 7fd576800640 1 --2- 172.21.15.28:0/3727765835 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd570102b80 0x7fd57019a590 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd56000e9f0 tx=0x7fd56000eeb0 comp rx=0 tx=0).stop 2024-04-23T14:43:48.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.176+0000 7fd576800640 1 -- 172.21.15.28:0/3727765835 shutdown_connections 2024-04-23T14:43:48.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.176+0000 7fd576800640 1 --2- 172.21.15.28:0/3727765835 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd5540a3bb0 0x7fd5540a6070 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:48.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.176+0000 7fd576800640 1 --2- 172.21.15.28:0/3727765835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5701054b0 0x7fd57019aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:48.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.176+0000 7fd576800640 1 --2- 172.21.15.28:0/3727765835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd570107de0 0x7fd5701a1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:48.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.176+0000 7fd576800640 1 --2- 172.21.15.28:0/3727765835 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd570102b80 0x7fd57019a590 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:48.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.176+0000 7fd576800640 1 -- 172.21.15.28:0/3727765835 >> 172.21.15.28:0/3727765835 conn(0x7fd5700fc4f0 msgr2=0x7fd570102290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:48.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.176+0000 7fd576800640 1 -- 172.21.15.28:0/3727765835 shutdown_connections 2024-04-23T14:43:48.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.176+0000 7fd576800640 1 -- 172.21.15.28:0/3727765835 wait complete. 2024-04-23T14:43:48.198 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T14:36:27.698112+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T14:41:37.030341+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T14:41:38.444742+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T14:43:48.198 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T14:43:48.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.419+0000 7f7acca00640 1 -- 172.21.15.28:0/3742751910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac8103a20 msgr2=0x7f7ac810a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:48.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.419+0000 7f7acca00640 1 --2- 172.21.15.28:0/3742751910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac8103a20 0x7f7ac810a340 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7ac0009ca0 tx=0x7f7ac002a8c0 comp rx=0 tx=0).stop 2024-04-23T14:43:48.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.419+0000 7f7acca00640 1 -- 172.21.15.28:0/3742751910 shutdown_connections 2024-04-23T14:43:48.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.419+0000 7f7acca00640 1 --2- 172.21.15.28:0/3742751910 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ac8103060 0x7f7ac81034e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:48.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.419+0000 7f7acca00640 1 --2- 172.21.15.28:0/3742751910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac8103a20 0x7f7ac810a340 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:48.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.419+0000 7f7acca00640 1 --2- 172.21.15.28:0/3742751910 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7ac8101f10 0x7f7ac8102330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:48.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.419+0000 7f7acca00640 1 -- 172.21.15.28:0/3742751910 >> 172.21.15.28:0/3742751910 conn(0x7f7ac80fd4a0 msgr2=0x7f7ac80ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:48.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.419+0000 7f7acca00640 1 -- 172.21.15.28:0/3742751910 shutdown_connections 2024-04-23T14:43:48.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.419+0000 7f7acca00640 1 -- 172.21.15.28:0/3742751910 wait complete. 2024-04-23T14:43:48.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.420+0000 7f7acca00640 1 Processor -- start 2024-04-23T14:43:48.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.420+0000 7f7acca00640 1 -- start start 2024-04-23T14:43:48.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.420+0000 7f7acca00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ac8101f10 0x7f7ac819a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:48.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.420+0000 7f7acca00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7ac8103060 0x7f7ac819a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:48.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.420+0000 7f7acca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac8103a20 0x7f7ac81a1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.421+0000 7f7acca00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7ac810dc10 con 0x7f7ac8103a20 2024-04-23T14:43:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.421+0000 7f7acca00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7ac810da90 con 0x7f7ac8101f10 2024-04-23T14:43:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.421+0000 7f7acca00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7ac810dd90 con 0x7f7ac8103060 2024-04-23T14:43:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.421+0000 7f7ac5e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7ac8103060 0x7f7ac819a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.421+0000 7f7ac7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac8103a20 0x7f7ac81a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.421+0000 7f7ac5e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7ac8103060 0x7f7ac819a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:40706/0 (socket says 172.21.15.28:40706) 2024-04-23T14:43:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.421+0000 7f7ac7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac8103a20 0x7f7ac81a1990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45150/0 (socket says 172.21.15.28:45150) 2024-04-23T14:43:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.421+0000 7f7ac5e00640 1 -- 172.21.15.28:0/2602039968 learned_addr learned my addr 172.21.15.28:0/2602039968 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.421+0000 7f7ac6800640 1 --2- 172.21.15.28:0/2602039968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ac8101f10 0x7f7ac819a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.421+0000 7f7ac7200640 1 -- 172.21.15.28:0/2602039968 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7ac8103060 msgr2=0x7f7ac819a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.421+0000 7f7ac7200640 1 --2- 172.21.15.28:0/2602039968 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7ac8103060 0x7f7ac819a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.421+0000 7f7ac7200640 1 -- 172.21.15.28:0/2602039968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ac8101f10 msgr2=0x7f7ac819a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.421+0000 7f7ac7200640 1 --2- 172.21.15.28:0/2602039968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ac8101f10 0x7f7ac819a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.421+0000 7f7ac7200640 1 -- 172.21.15.28:0/2602039968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ac81a2000 con 0x7f7ac8103a20 2024-04-23T14:43:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.422+0000 7f7ac7200640 1 --2- 172.21.15.28:0/2602039968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac8103a20 0x7f7ac81a1990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7ac0031ae0 tx=0x7f7ac00319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.422+0000 7f7abf400640 1 -- 172.21.15.28:0/2602039968 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac0003b70 con 0x7f7ac8103a20 2024-04-23T14:43:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.422+0000 7f7abf400640 1 -- 172.21.15.28:0/2602039968 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ac0003d10 con 0x7f7ac8103a20 2024-04-23T14:43:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.422+0000 7f7abf400640 1 -- 172.21.15.28:0/2602039968 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac002fbd0 con 0x7f7ac8103a20 2024-04-23T14:43:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.422+0000 7f7acca00640 1 -- 172.21.15.28:0/2602039968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ac81a2290 con 0x7f7ac8103a20 2024-04-23T14:43:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.422+0000 7f7acca00640 1 -- 172.21.15.28:0/2602039968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ac81a2790 con 0x7f7ac8103a20 2024-04-23T14:43:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.423+0000 7f7acca00640 1 -- 172.21.15.28:0/2602039968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ac81a29f0 con 0x7f7ac8103a20 2024-04-23T14:43:48.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.425+0000 7f7abf400640 1 -- 172.21.15.28:0/2602039968 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f7ac002fd70 con 0x7f7ac8103a20 2024-04-23T14:43:48.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.428+0000 7f7abf400640 1 --2- 172.21.15.28:0/2602039968 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f7aac0a3c30 0x7f7aac0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:48.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.428+0000 7f7abf400640 1 -- 172.21.15.28:0/2602039968 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7f7ac00f17d0 con 0x7f7ac8103a20 2024-04-23T14:43:48.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.428+0000 7f7ac6800640 1 --2- 172.21.15.28:0/2602039968 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f7aac0a3c30 0x7f7aac0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:48.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.429+0000 7f7ac6800640 1 --2- 172.21.15.28:0/2602039968 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f7aac0a3c30 0x7f7aac0a60f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7ab8009bd0 tx=0x7f7ab8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:48.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.432+0000 7f7abf400640 1 -- 172.21.15.28:0/2602039968 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7ac00be170 con 0x7f7ac8103a20 2024-04-23T14:43:48.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.641+0000 7f7acca00640 1 -- 172.21.15.28:0/2602039968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7ac81a2d80 con 0x7f7ac8103a20 2024-04-23T14:43:48.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.642+0000 7f7abf400640 1 -- 172.21.15.28:0/2602039968 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7f7ac00bda70 con 0x7f7ac8103a20 2024-04-23T14:43:48.643 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-23T14:43:48.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.645+0000 7f7acca00640 1 -- 172.21.15.28:0/2602039968 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f7aac0a3c30 msgr2=0x7f7aac0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:48.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.645+0000 7f7acca00640 1 --2- 172.21.15.28:0/2602039968 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f7aac0a3c30 0x7f7aac0a60f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7ab8009bd0 tx=0x7f7ab8005f90 comp rx=0 tx=0).stop 2024-04-23T14:43:48.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.645+0000 7f7acca00640 1 -- 172.21.15.28:0/2602039968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac8103a20 msgr2=0x7f7ac81a1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:48.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.645+0000 7f7acca00640 1 --2- 172.21.15.28:0/2602039968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac8103a20 0x7f7ac81a1990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7ac0031ae0 tx=0x7f7ac00319f0 comp rx=0 tx=0).stop 2024-04-23T14:43:48.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.645+0000 7f7acca00640 1 -- 172.21.15.28:0/2602039968 shutdown_connections 2024-04-23T14:43:48.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.645+0000 7f7acca00640 1 --2- 172.21.15.28:0/2602039968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac8103a20 0x7f7ac81a1990 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:48.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.645+0000 7f7acca00640 1 --2- 172.21.15.28:0/2602039968 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7ac8103060 0x7f7ac819a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:48.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.645+0000 7f7acca00640 1 --2- 172.21.15.28:0/2602039968 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f7aac0a3c30 0x7f7aac0a60f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:48.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.645+0000 7f7acca00640 1 --2- 172.21.15.28:0/2602039968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ac8101f10 0x7f7ac819a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:48.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.645+0000 7f7acca00640 1 -- 172.21.15.28:0/2602039968 >> 172.21.15.28:0/2602039968 conn(0x7f7ac80fd4a0 msgr2=0x7f7ac8074230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:48.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.645+0000 7f7acca00640 1 -- 172.21.15.28:0/2602039968 shutdown_connections 2024-04-23T14:43:48.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.645+0000 7f7acca00640 1 -- 172.21.15.28:0/2602039968 wait complete. 2024-04-23T14:43:48.665 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T14:43:48.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:48 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3427119120' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:43:48.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:48 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3727765835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:43:48.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:48 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3427119120' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:43:48.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:48 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3727765835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:43:48.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.892+0000 7f4135200640 1 -- 172.21.15.28:0/233206061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4130109730 msgr2=0x7f4130109bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:48.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.892+0000 7f4135200640 1 --2- 172.21.15.28:0/233206061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4130109730 0x7f4130109bb0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4118009900 tx=0x7f411802a410 comp rx=0 tx=0).stop 2024-04-23T14:43:48.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.892+0000 7f4135200640 1 -- 172.21.15.28:0/233206061 shutdown_connections 2024-04-23T14:43:48.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.892+0000 7f4135200640 1 --2- 172.21.15.28:0/233206061 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f413010a0f0 0x7f4130110980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:48.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.892+0000 7f4135200640 1 --2- 172.21.15.28:0/233206061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4130109730 0x7f4130109bb0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:48.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.892+0000 7f4135200640 1 --2- 172.21.15.28:0/233206061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4130108540 0x7f4130108960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:48.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.892+0000 7f4135200640 1 -- 172.21.15.28:0/233206061 >> 172.21.15.28:0/233206061 conn(0x7f41300774d0 msgr2=0x7f4130079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:48.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.892+0000 7f4135200640 1 -- 172.21.15.28:0/233206061 shutdown_connections 2024-04-23T14:43:48.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.892+0000 7f4135200640 1 -- 172.21.15.28:0/233206061 wait complete. 2024-04-23T14:43:48.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.893+0000 7f4135200640 1 Processor -- start 2024-04-23T14:43:48.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.894+0000 7f4135200640 1 -- start start 2024-04-23T14:43:48.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.894+0000 7f4135200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4130108540 0x7f413019e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:48.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.894+0000 7f4135200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4130109730 0x7f413019eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:48.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.894+0000 7f4135200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f413010a0f0 0x7f41301a5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:48.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.894+0000 7f4135200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f41301141d0 con 0x7f4130109730 2024-04-23T14:43:48.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.894+0000 7f4135200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4130114050 con 0x7f413010a0f0 2024-04-23T14:43:48.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.894+0000 7f4135200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4130114350 con 0x7f4130108540 2024-04-23T14:43:48.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.894+0000 7f4127e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4130109730 0x7f413019eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:48.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.894+0000 7f412f400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4130108540 0x7f413019e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:48.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.894+0000 7f4127e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4130109730 0x7f413019eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45180/0 (socket says 172.21.15.28:45180) 2024-04-23T14:43:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.894+0000 7f4127e00640 1 -- 172.21.15.28:0/2226662200 learned_addr learned my addr 172.21.15.28:0/2226662200 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.894+0000 7f412fe00640 1 --2- 172.21.15.28:0/2226662200 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f413010a0f0 0x7f41301a5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.895+0000 7f4127e00640 1 -- 172.21.15.28:0/2226662200 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4130108540 msgr2=0x7f413019e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.895+0000 7f4127e00640 1 --2- 172.21.15.28:0/2226662200 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4130108540 0x7f413019e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.895+0000 7f4127e00640 1 -- 172.21.15.28:0/2226662200 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f413010a0f0 msgr2=0x7f41301a5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.895+0000 7f4127e00640 1 --2- 172.21.15.28:0/2226662200 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f413010a0f0 0x7f41301a5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.895+0000 7f4127e00640 1 -- 172.21.15.28:0/2226662200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41301a63e0 con 0x7f4130109730 2024-04-23T14:43:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.895+0000 7f412f400640 1 --2- 172.21.15.28:0/2226662200 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4130108540 0x7f413019e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:43:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.895+0000 7f4127e00640 1 --2- 172.21.15.28:0/2226662200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4130109730 0x7f413019eda0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4118009230 tx=0x7f41180042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:48.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.895+0000 7f412cc00640 1 -- 172.21.15.28:0/2226662200 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411803f030 con 0x7f4130109730 2024-04-23T14:43:48.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.895+0000 7f412cc00640 1 -- 172.21.15.28:0/2226662200 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4118003ce0 con 0x7f4130109730 2024-04-23T14:43:48.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.895+0000 7f412cc00640 1 -- 172.21.15.28:0/2226662200 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4118035070 con 0x7f4130109730 2024-04-23T14:43:48.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.895+0000 7f4135200640 1 -- 172.21.15.28:0/2226662200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41301a6670 con 0x7f4130109730 2024-04-23T14:43:48.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.895+0000 7f4135200640 1 -- 172.21.15.28:0/2226662200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41301a6b10 con 0x7f4130109730 2024-04-23T14:43:48.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.896+0000 7f4135200640 1 -- 172.21.15.28:0/2226662200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f413010f860 con 0x7f4130109730 2024-04-23T14:43:48.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.898+0000 7f412cc00640 1 -- 172.21.15.28:0/2226662200 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f4118042030 con 0x7f4130109730 2024-04-23T14:43:48.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.900+0000 7f412cc00640 1 --2- 172.21.15.28:0/2226662200 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f41140a3bb0 0x7f41140a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:48.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.900+0000 7f412cc00640 1 -- 172.21.15.28:0/2226662200 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7f41180f08e0 con 0x7f4130109730 2024-04-23T14:43:48.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.900+0000 7f412f400640 1 --2- 172.21.15.28:0/2226662200 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f41140a3bb0 0x7f41140a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:48.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.901+0000 7f412f400640 1 --2- 172.21.15.28:0/2226662200 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f41140a3bb0 0x7f41140a6070 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4130109590 tx=0x7f4120005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:48.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:48.904+0000 7f412cc00640 1 -- 172.21.15.28:0/2226662200 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f41180bcc30 con 0x7f4130109730 2024-04-23T14:43:49.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:48 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3427119120' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:43:49.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:48 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3727765835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:43:49.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.112+0000 7f4135200640 1 -- 172.21.15.28:0/2226662200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4130062c00 con 0x7f4130109730 2024-04-23T14:43:49.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.113+0000 7f412cc00640 1 -- 172.21.15.28:0/2226662200 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f41180c0f30 con 0x7f4130109730 2024-04-23T14:43:49.114 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-23T14:43:49.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.116+0000 7f4135200640 1 -- 172.21.15.28:0/2226662200 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f41140a3bb0 msgr2=0x7f41140a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:49.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.116+0000 7f4135200640 1 --2- 172.21.15.28:0/2226662200 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f41140a3bb0 0x7f41140a6070 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4130109590 tx=0x7f4120005f90 comp rx=0 tx=0).stop 2024-04-23T14:43:49.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.116+0000 7f4135200640 1 -- 172.21.15.28:0/2226662200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4130109730 msgr2=0x7f413019eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:49.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.116+0000 7f4135200640 1 --2- 172.21.15.28:0/2226662200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4130109730 0x7f413019eda0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4118009230 tx=0x7f41180042f0 comp rx=0 tx=0).stop 2024-04-23T14:43:49.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.116+0000 7f4135200640 1 -- 172.21.15.28:0/2226662200 shutdown_connections 2024-04-23T14:43:49.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.116+0000 7f4135200640 1 --2- 172.21.15.28:0/2226662200 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f41140a3bb0 0x7f41140a6070 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:49.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.116+0000 7f4135200640 1 --2- 172.21.15.28:0/2226662200 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f413010a0f0 0x7f41301a5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:49.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.116+0000 7f4135200640 1 --2- 172.21.15.28:0/2226662200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4130109730 0x7f413019eda0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:49.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.116+0000 7f4135200640 1 --2- 172.21.15.28:0/2226662200 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4130108540 0x7f413019e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:49.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.116+0000 7f4135200640 1 -- 172.21.15.28:0/2226662200 >> 172.21.15.28:0/2226662200 conn(0x7f41300774d0 msgr2=0x7f413010ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:49.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.116+0000 7f4135200640 1 -- 172.21.15.28:0/2226662200 shutdown_connections 2024-04-23T14:43:49.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.116+0000 7f4135200640 1 -- 172.21.15.28:0/2226662200 wait complete. 2024-04-23T14:43:49.137 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T14:43:49.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.353+0000 7feb37e00640 1 -- 172.21.15.28:0/543874114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb38105330 msgr2=0x7feb381057b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:49.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.353+0000 7feb37e00640 1 --2- 172.21.15.28:0/543874114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb38105330 0x7feb381057b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7feb20009860 tx=0x7feb2002a340 comp rx=0 tx=0).stop 2024-04-23T14:43:49.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.353+0000 7feb37e00640 1 -- 172.21.15.28:0/543874114 shutdown_connections 2024-04-23T14:43:49.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.353+0000 7feb37e00640 1 --2- 172.21.15.28:0/543874114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb38105330 0x7feb381057b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:49.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.353+0000 7feb37e00640 1 --2- 172.21.15.28:0/543874114 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feb38105cf0 0x7feb3810c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:49.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.353+0000 7feb37e00640 1 --2- 172.21.15.28:0/543874114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb38104140 0x7feb38104560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:49.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.353+0000 7feb37e00640 1 -- 172.21.15.28:0/543874114 >> 172.21.15.28:0/543874114 conn(0x7feb380ff6b0 msgr2=0x7feb38101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:49.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.354+0000 7feb37e00640 1 -- 172.21.15.28:0/543874114 shutdown_connections 2024-04-23T14:43:49.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.354+0000 7feb37e00640 1 -- 172.21.15.28:0/543874114 wait complete. 2024-04-23T14:43:49.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.355+0000 7feb37e00640 1 Processor -- start 2024-04-23T14:43:49.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.355+0000 7feb37e00640 1 -- start start 2024-04-23T14:43:49.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.355+0000 7feb37e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb38104140 0x7feb3819a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:49.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.355+0000 7feb37e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb38105330 0x7feb3819a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:49.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.355+0000 7feb37e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feb38105cf0 0x7feb381a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:49.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.355+0000 7feb37e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7feb3810fdd0 con 0x7feb38105330 2024-04-23T14:43:49.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.355+0000 7feb37e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7feb3810fc50 con 0x7feb38104140 2024-04-23T14:43:49.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.355+0000 7feb37e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7feb3810ff50 con 0x7feb38105cf0 2024-04-23T14:43:49.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.356+0000 7feb36a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb38104140 0x7feb3819a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:49.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.356+0000 7feb36a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb38104140 0x7feb3819a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:53582/0 (socket says 172.21.15.28:53582) 2024-04-23T14:43:49.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.356+0000 7feb36a00640 1 -- 172.21.15.28:0/830867672 learned_addr learned my addr 172.21.15.28:0/830867672 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:49.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.356+0000 7feb37400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feb38105cf0 0x7feb381a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:49.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.356+0000 7feb2fe00640 1 --2- 172.21.15.28:0/830867672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb38105330 0x7feb3819a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:49.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.356+0000 7feb36a00640 1 -- 172.21.15.28:0/830867672 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feb38105cf0 msgr2=0x7feb381a19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:49.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.356+0000 7feb36a00640 1 --2- 172.21.15.28:0/830867672 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feb38105cf0 0x7feb381a19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:49.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.356+0000 7feb36a00640 1 -- 172.21.15.28:0/830867672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb38105330 msgr2=0x7feb3819a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:49.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.356+0000 7feb36a00640 1 --2- 172.21.15.28:0/830867672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb38105330 0x7feb3819a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:49.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.356+0000 7feb36a00640 1 -- 172.21.15.28:0/830867672 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb381a2020 con 0x7feb38104140 2024-04-23T14:43:49.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.356+0000 7feb2fe00640 1 --2- 172.21.15.28:0/830867672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb38105330 0x7feb3819a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:43:49.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.357+0000 7feb36a00640 1 --2- 172.21.15.28:0/830867672 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb38104140 0x7feb3819a400 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7feb2800e9c0 tx=0x7feb2800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:49.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.357+0000 7feb2f400640 1 -- 172.21.15.28:0/830867672 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2800cd80 con 0x7feb38104140 2024-04-23T14:43:49.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.357+0000 7feb2f400640 1 -- 172.21.15.28:0/830867672 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb280035f0 con 0x7feb38104140 2024-04-23T14:43:49.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.357+0000 7feb2f400640 1 -- 172.21.15.28:0/830867672 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb28010690 con 0x7feb38104140 2024-04-23T14:43:49.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.357+0000 7feb37e00640 1 -- 172.21.15.28:0/830867672 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb381a2310 con 0x7feb38104140 2024-04-23T14:43:49.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.357+0000 7feb37e00640 1 -- 172.21.15.28:0/830867672 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb381a2830 con 0x7feb38104140 2024-04-23T14:43:49.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.358+0000 7feb37e00640 1 -- 172.21.15.28:0/830867672 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb3810b460 con 0x7feb38104140 2024-04-23T14:43:49.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.360+0000 7feb2f400640 1 -- 172.21.15.28:0/830867672 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7feb280037a0 con 0x7feb38104140 2024-04-23T14:43:49.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.362+0000 7feb2f400640 1 --2- 172.21.15.28:0/830867672 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7feb1c0a3bb0 0x7feb1c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:49.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.362+0000 7feb2f400640 1 -- 172.21.15.28:0/830867672 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7feb280d08f0 con 0x7feb38104140 2024-04-23T14:43:49.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.362+0000 7feb2fe00640 1 --2- 172.21.15.28:0/830867672 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7feb1c0a3bb0 0x7feb1c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:49.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.363+0000 7feb2fe00640 1 --2- 172.21.15.28:0/830867672 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7feb1c0a3bb0 0x7feb1c0a6070 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7feb2002a930 tx=0x7feb20030550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:49.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.366+0000 7feb2f400640 1 -- 172.21.15.28:0/830867672 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feb2809d290 con 0x7feb38104140 2024-04-23T14:43:49.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.572+0000 7feb37e00640 1 -- 172.21.15.28:0/830867672 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7feb38062c00 con 0x7feb38104140 2024-04-23T14:43:49.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.573+0000 7feb2f400640 1 -- 172.21.15.28:0/830867672 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7feb2809cb90 con 0x7feb38104140 2024-04-23T14:43:49.574 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-23T14:43:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.576+0000 7feb37e00640 1 -- 172.21.15.28:0/830867672 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7feb1c0a3bb0 msgr2=0x7feb1c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.576+0000 7feb37e00640 1 --2- 172.21.15.28:0/830867672 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7feb1c0a3bb0 0x7feb1c0a6070 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7feb2002a930 tx=0x7feb20030550 comp rx=0 tx=0).stop 2024-04-23T14:43:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.576+0000 7feb37e00640 1 -- 172.21.15.28:0/830867672 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb38104140 msgr2=0x7feb3819a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.576+0000 7feb37e00640 1 --2- 172.21.15.28:0/830867672 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb38104140 0x7feb3819a400 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7feb2800e9c0 tx=0x7feb2800ee80 comp rx=0 tx=0).stop 2024-04-23T14:43:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.576+0000 7feb37e00640 1 -- 172.21.15.28:0/830867672 shutdown_connections 2024-04-23T14:43:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.576+0000 7feb37e00640 1 --2- 172.21.15.28:0/830867672 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feb38105cf0 0x7feb381a19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:49.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.576+0000 7feb37e00640 1 --2- 172.21.15.28:0/830867672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb38105330 0x7feb3819a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:49.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.576+0000 7feb37e00640 1 --2- 172.21.15.28:0/830867672 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7feb1c0a3bb0 0x7feb1c0a6070 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:49.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.576+0000 7feb37e00640 1 --2- 172.21.15.28:0/830867672 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb38104140 0x7feb3819a400 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:49.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.576+0000 7feb37e00640 1 -- 172.21.15.28:0/830867672 >> 172.21.15.28:0/830867672 conn(0x7feb380ff6b0 msgr2=0x7feb38108630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:49.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.576+0000 7feb37e00640 1 -- 172.21.15.28:0/830867672 shutdown_connections 2024-04-23T14:43:49.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.576+0000 7feb37e00640 1 -- 172.21.15.28:0/830867672 wait complete. 2024-04-23T14:43:49.598 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:43:49.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.802+0000 7f1365e00640 1 -- 172.21.15.28:0/3043671321 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1360101e70 msgr2=0x7f1360102290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:49.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.802+0000 7f1365e00640 1 --2- 172.21.15.28:0/3043671321 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1360101e70 0x7f1360102290 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1354009900 tx=0x7f135402a410 comp rx=0 tx=0).stop 2024-04-23T14:43:49.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.802+0000 7f1365e00640 1 -- 172.21.15.28:0/3043671321 shutdown_connections 2024-04-23T14:43:49.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.802+0000 7f1365e00640 1 --2- 172.21.15.28:0/3043671321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1360102fc0 0x7f1360103440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:49.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.802+0000 7f1365e00640 1 --2- 172.21.15.28:0/3043671321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1360103980 0x7f136010a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:49.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.802+0000 7f1365e00640 1 --2- 172.21.15.28:0/3043671321 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1360101e70 0x7f1360102290 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:49.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.802+0000 7f1365e00640 1 -- 172.21.15.28:0/3043671321 >> 172.21.15.28:0/3043671321 conn(0x7f13600fd3e0 msgr2=0x7f13600ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:49.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.802+0000 7f1365e00640 1 -- 172.21.15.28:0/3043671321 shutdown_connections 2024-04-23T14:43:49.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.802+0000 7f1365e00640 1 -- 172.21.15.28:0/3043671321 wait complete. 2024-04-23T14:43:49.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.803+0000 7f1365e00640 1 Processor -- start 2024-04-23T14:43:49.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.804+0000 7f1365e00640 1 -- start start 2024-04-23T14:43:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.804+0000 7f1365e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1360101e70 0x7f136019a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.804+0000 7f1365e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1360102fc0 0x7f136019a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.804+0000 7f1365e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1360103980 0x7f13601a1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.804+0000 7f1365e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f136010db10 con 0x7f1360103980 2024-04-23T14:43:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.804+0000 7f1365e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f136010d990 con 0x7f1360101e70 2024-04-23T14:43:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.804+0000 7f1365e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f136010dc90 con 0x7f1360102fc0 2024-04-23T14:43:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.804+0000 7f1365400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1360103980 0x7f13601a1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.804+0000 7f135fe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1360102fc0 0x7f136019a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.804+0000 7f135fe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1360102fc0 0x7f136019a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:40748/0 (socket says 172.21.15.28:40748) 2024-04-23T14:43:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.804+0000 7f135fe00640 1 -- 172.21.15.28:0/4017405352 learned_addr learned my addr 172.21.15.28:0/4017405352 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.804+0000 7f1364a00640 1 --2- 172.21.15.28:0/4017405352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1360101e70 0x7f136019a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:49.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.805+0000 7f135fe00640 1 -- 172.21.15.28:0/4017405352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1360101e70 msgr2=0x7f136019a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:49.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.805+0000 7f135fe00640 1 --2- 172.21.15.28:0/4017405352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1360101e70 0x7f136019a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:49.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.805+0000 7f135fe00640 1 -- 172.21.15.28:0/4017405352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1360103980 msgr2=0x7f13601a1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:49.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.805+0000 7f135fe00640 1 --2- 172.21.15.28:0/4017405352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1360103980 0x7f13601a1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:49.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.805+0000 7f135fe00640 1 -- 172.21.15.28:0/4017405352 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13601a2070 con 0x7f1360102fc0 2024-04-23T14:43:49.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.805+0000 7f135fe00640 1 --2- 172.21.15.28:0/4017405352 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1360102fc0 0x7f136019a9e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f134c00e9c0 tx=0x7f134c00edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:49.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.805+0000 7f135d600640 1 -- 172.21.15.28:0/4017405352 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f134c00ccc0 con 0x7f1360102fc0 2024-04-23T14:43:49.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.805+0000 7f135d600640 1 -- 172.21.15.28:0/4017405352 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f134c0035f0 con 0x7f1360102fc0 2024-04-23T14:43:49.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.805+0000 7f1365e00640 1 -- 172.21.15.28:0/4017405352 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13601a2360 con 0x7f1360102fc0 2024-04-23T14:43:49.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.805+0000 7f135d600640 1 -- 172.21.15.28:0/4017405352 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f134c00ce60 con 0x7f1360102fc0 2024-04-23T14:43:49.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.806+0000 7f1365e00640 1 -- 172.21.15.28:0/4017405352 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13601a2860 con 0x7f1360102fc0 2024-04-23T14:43:49.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.807+0000 7f1365e00640 1 -- 172.21.15.28:0/4017405352 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13601a2b20 con 0x7f1360102fc0 2024-04-23T14:43:49.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.809+0000 7f135d600640 1 -- 172.21.15.28:0/4017405352 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f134c0037a0 con 0x7f1360102fc0 2024-04-23T14:43:49.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.811+0000 7f135d600640 1 --2- 172.21.15.28:0/4017405352 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f13480a3bb0 0x7f13480a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:49.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.811+0000 7f135d600640 1 -- 172.21.15.28:0/4017405352 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7f134c0d0790 con 0x7f1360102fc0 2024-04-23T14:43:49.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.811+0000 7f1364a00640 1 --2- 172.21.15.28:0/4017405352 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f13480a3bb0 0x7f13480a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:49.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.812+0000 7f1364a00640 1 --2- 172.21.15.28:0/4017405352 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f13480a3bb0 0x7f13480a6070 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f13540314e0 tx=0x7f1354000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:49.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:49.815+0000 7f135d600640 1 -- 172.21.15.28:0/4017405352 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f134c09d130 con 0x7f1360102fc0 2024-04-23T14:43:49.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:49 smithi038 ceph-mon[32348]: pgmap v380: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:43:49.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:49 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2602039968' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:43:49.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:49 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2226662200' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T14:43:49.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:49 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/830867672' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T14:43:49.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:49 smithi028 ceph-mon[27810]: pgmap v380: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:43:49.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:49 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2602039968' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:43:49.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:49 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2226662200' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T14:43:49.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:49 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/830867672' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T14:43:50.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:49 smithi105 ceph-mon[32555]: pgmap v380: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:43:50.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:49 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2602039968' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:43:50.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:49 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2226662200' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T14:43:50.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:49 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/830867672' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T14:43:50.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.064+0000 7f1365e00640 1 -- 172.21.15.28:0/4017405352 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f13601a2cf0 con 0x7f1360102fc0 2024-04-23T14:43:50.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.065+0000 7f135d600640 1 -- 172.21.15.28:0/4017405352 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7f134c09ca30 con 0x7f1360102fc0 2024-04-23T14:43:50.067 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:43:50.067 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:43:50.067 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:43:50.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.068+0000 7f1365e00640 1 -- 172.21.15.28:0/4017405352 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f13480a3bb0 msgr2=0x7f13480a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:50.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.068+0000 7f1365e00640 1 --2- 172.21.15.28:0/4017405352 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f13480a3bb0 0x7f13480a6070 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f13540314e0 tx=0x7f1354000bc0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.068+0000 7f1365e00640 1 -- 172.21.15.28:0/4017405352 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1360102fc0 msgr2=0x7f136019a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:50.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.068+0000 7f1365e00640 1 --2- 172.21.15.28:0/4017405352 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1360102fc0 0x7f136019a9e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f134c00e9c0 tx=0x7f134c00edc0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.069+0000 7f1365e00640 1 -- 172.21.15.28:0/4017405352 shutdown_connections 2024-04-23T14:43:50.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.069+0000 7f1365e00640 1 --2- 172.21.15.28:0/4017405352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1360103980 0x7f13601a1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.069+0000 7f1365e00640 1 --2- 172.21.15.28:0/4017405352 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1360102fc0 0x7f136019a9e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.069+0000 7f1365e00640 1 --2- 172.21.15.28:0/4017405352 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f13480a3bb0 0x7f13480a6070 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.069+0000 7f1365e00640 1 --2- 172.21.15.28:0/4017405352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1360101e70 0x7f136019a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.069+0000 7f1365e00640 1 -- 172.21.15.28:0/4017405352 >> 172.21.15.28:0/4017405352 conn(0x7f13600fd3e0 msgr2=0x7f1360074230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:50.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.069+0000 7f1365e00640 1 -- 172.21.15.28:0/4017405352 shutdown_connections 2024-04-23T14:43:50.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.069+0000 7f1365e00640 1 -- 172.21.15.28:0/4017405352 wait complete. 2024-04-23T14:43:50.090 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:43:50.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.290+0000 7fd4b0c00640 1 -- 172.21.15.28:0/2746162771 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4ac10a030 msgr2=0x7fd4ac1108c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:50.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.290+0000 7fd4b0c00640 1 --2- 172.21.15.28:0/2746162771 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4ac10a030 0x7fd4ac1108c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4009ca0 tx=0x7fd4a402a8c0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.291+0000 7fd4b0c00640 1 -- 172.21.15.28:0/2746162771 shutdown_connections 2024-04-23T14:43:50.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.291+0000 7fd4b0c00640 1 --2- 172.21.15.28:0/2746162771 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd4ac109670 0x7fd4ac109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.291+0000 7fd4b0c00640 1 --2- 172.21.15.28:0/2746162771 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4ac10a030 0x7fd4ac1108c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.291+0000 7fd4b0c00640 1 --2- 172.21.15.28:0/2746162771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4ac108480 0x7fd4ac1088a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.291+0000 7fd4b0c00640 1 -- 172.21.15.28:0/2746162771 >> 172.21.15.28:0/2746162771 conn(0x7fd4ac077440 msgr2=0x7fd4ac079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:50.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.291+0000 7fd4b0c00640 1 -- 172.21.15.28:0/2746162771 shutdown_connections 2024-04-23T14:43:50.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.291+0000 7fd4b0c00640 1 -- 172.21.15.28:0/2746162771 wait complete. 2024-04-23T14:43:50.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.292+0000 7fd4b0c00640 1 Processor -- start 2024-04-23T14:43:50.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.292+0000 7fd4b0c00640 1 -- start start 2024-04-23T14:43:50.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.292+0000 7fd4b0c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd4ac108480 0x7fd4ac19e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:50.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.292+0000 7fd4b0c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4ac109670 0x7fd4ac19ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:50.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.292+0000 7fd4b0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4ac10a030 0x7fd4ac1a5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:50.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.292+0000 7fd4b0c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd4ac1140e0 con 0x7fd4ac10a030 2024-04-23T14:43:50.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.292+0000 7fd4b0c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd4ac113f60 con 0x7fd4ac108480 2024-04-23T14:43:50.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.292+0000 7fd4b0c00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd4ac114260 con 0x7fd4ac109670 2024-04-23T14:43:50.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.292+0000 7fd4abe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4ac10a030 0x7fd4ac1a5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:50.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.292+0000 7fd4ab400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd4ac108480 0x7fd4ac19e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:50.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.292+0000 7fd4a3e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4ac109670 0x7fd4ac19ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:50.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.292+0000 7fd4abe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4ac10a030 0x7fd4ac1a5cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45254/0 (socket says 172.21.15.28:45254) 2024-04-23T14:43:50.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.292+0000 7fd4abe00640 1 -- 172.21.15.28:0/2870364607 learned_addr learned my addr 172.21.15.28:0/2870364607 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:50.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.293+0000 7fd4abe00640 1 -- 172.21.15.28:0/2870364607 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4ac109670 msgr2=0x7fd4ac19ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:50.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.293+0000 7fd4abe00640 1 --2- 172.21.15.28:0/2870364607 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4ac109670 0x7fd4ac19ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.293+0000 7fd4abe00640 1 -- 172.21.15.28:0/2870364607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd4ac108480 msgr2=0x7fd4ac19e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:50.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.293+0000 7fd4abe00640 1 --2- 172.21.15.28:0/2870364607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd4ac108480 0x7fd4ac19e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.293+0000 7fd4abe00640 1 -- 172.21.15.28:0/2870364607 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4ac1a6340 con 0x7fd4ac10a030 2024-04-23T14:43:50.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.293+0000 7fd4a3e00640 1 --2- 172.21.15.28:0/2870364607 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4ac109670 0x7fd4ac19ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:43:50.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.293+0000 7fd4ab400640 1 --2- 172.21.15.28:0/2870364607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd4ac108480 0x7fd4ac19e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:43:50.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.293+0000 7fd4abe00640 1 --2- 172.21.15.28:0/2870364607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4ac10a030 0x7fd4ac1a5cd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4031ae0 tx=0x7fd4a4031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:50.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.293+0000 7fd4a8c00640 1 -- 172.21.15.28:0/2870364607 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a4003b70 con 0x7fd4ac10a030 2024-04-23T14:43:50.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.294+0000 7fd4a8c00640 1 -- 172.21.15.28:0/2870364607 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4a4003d10 con 0x7fd4ac10a030 2024-04-23T14:43:50.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.294+0000 7fd4b0c00640 1 -- 172.21.15.28:0/2870364607 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4ac1a65d0 con 0x7fd4ac10a030 2024-04-23T14:43:50.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.294+0000 7fd4a8c00640 1 -- 172.21.15.28:0/2870364607 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a402fc40 con 0x7fd4ac10a030 2024-04-23T14:43:50.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.294+0000 7fd4b0c00640 1 -- 172.21.15.28:0/2870364607 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4ac1a6b70 con 0x7fd4ac10a030 2024-04-23T14:43:50.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.295+0000 7fd4b0c00640 1 -- 172.21.15.28:0/2870364607 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4ac10f820 con 0x7fd4ac10a030 2024-04-23T14:43:50.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.297+0000 7fd4a8c00640 1 -- 172.21.15.28:0/2870364607 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fd4a4004050 con 0x7fd4ac10a030 2024-04-23T14:43:50.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.298+0000 7fd4a8c00640 1 --2- 172.21.15.28:0/2870364607 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd4900a3c30 0x7fd4900a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:50.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.299+0000 7fd4a8c00640 1 -- 172.21.15.28:0/2870364607 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7fd4a40f1420 con 0x7fd4ac10a030 2024-04-23T14:43:50.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.299+0000 7fd4ab400640 1 --2- 172.21.15.28:0/2870364607 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd4900a3c30 0x7fd4900a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:50.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.299+0000 7fd4ab400640 1 --2- 172.21.15.28:0/2870364607 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd4900a3c30 0x7fd4900a60f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac1094d0 tx=0x7fd49c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:50.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.303+0000 7fd4a8c00640 1 -- 172.21.15.28:0/2870364607 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd4a40bddc0 con 0x7fd4ac10a030 2024-04-23T14:43:50.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.511+0000 7fd4b0c00640 1 -- 172.21.15.28:0/2870364607 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd4ac062c00 con 0x7fd4ac10a030 2024-04-23T14:43:50.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.514+0000 7fd4a8c00640 1 -- 172.21.15.28:0/2870364607 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22651 (secure 0 0 0) 0x7fd4a40bd6c0 con 0x7fd4ac10a030 2024-04-23T14:43:50.516 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:43:50.516 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":87,"fsid":"1ee6e996-017e-11ef-bc93-c7b262605968","created":"2024-04-23T14:32:34.581082+0000","modified":"2024-04-23T14:42:43.429141+0000","last_up_change":"2024-04-23T14:40:43.375846+0000","last_in_change":"2024-04-23T14:40:23.576939+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:36:27.698112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:41:37.030341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:41:38.444742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e0b427bc-be65-4543-99cc-2f53ad98ddad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6803","nonce":2993445336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6805","nonce":2993445336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6809","nonce":2993445336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6807","nonce":2993445336}]},"public_addr":"172.21.15.28:6803/2993445336","cluster_addr":"172.21.15.28:6805/2993445336","heartbeat_back_addr":"172.21.15.28:6809/2993445336","heartbeat_front_addr":"172.21.15.28:6807/2993445336","state":["exists","up"]},{"osd":1,"uuid":"ef527ec9-1bf3-4bb0-8df3-f5a015af557c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6801","nonce":4024093383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6803","nonce":4024093383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6807","nonce":4024093383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6805","nonce":4024093383}]},"public_addr":"172.21.15.38:6801/4024093383","cluster_addr":"172.21.15.38:6803/4024093383","heartbeat_back_addr":"172.21.15.38:6807/4024093383","heartbeat_front_addr":"172.21.15.38:6805/4024093383","state":["exists","up"]},{"osd":2,"uuid":"b33b23d9-adce-4c11-b304-2e117bdf6bfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6801","nonce":570739029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6803","nonce":570739029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6807","nonce":570739029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6805","nonce":570739029}]},"public_addr":"172.21.15.105:6801/570739029","cluster_addr":"172.21.15.105:6803/570739029","heartbeat_back_addr":"172.21.15.105:6807/570739029","heartbeat_front_addr":"172.21.15.105:6805/570739029","state":["exists","up"]},{"osd":3,"uuid":"28bd5608-b426-434e-9db1-bdaa9828bdb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6811","nonce":2480400247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6813","nonce":2480400247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6817","nonce":2480400247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6815","nonce":2480400247}]},"public_addr":"172.21.15.28:6811/2480400247","cluster_addr":"172.21.15.28:6813/2480400247","heartbeat_back_addr":"172.21.15.28:6817/2480400247","heartbeat_front_addr":"172.21.15.28:6815/2480400247","state":["exists","up"]},{"osd":4,"uuid":"e5d1c89b-025f-4d57-8807-59d75c787b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6809","nonce":3772189246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6811","nonce":3772189246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6815","nonce":3772189246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6813","nonce":3772189246}]},"public_addr":"172.21.15.38:6809/3772189246","cluster_addr":"172.21.15.38:6811/3772189246","heartbeat_back_addr":"172.21.15.38:6815/3772189246","heartbeat_front_addr":"172.21.15.38:6813/3772189246","state":["exists","up"]},{"osd":5,"uuid":"9cf9cf4c-aafd-4ed3-bfb1-0527710a0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6809","nonce":928755894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6811","nonce":928755894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6815","nonce":928755894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6813","nonce":928755894}]},"public_addr":"172.21.15.105:6809/928755894","cluster_addr":"172.21.15.105:6811/928755894","heartbeat_back_addr":"172.21.15.105:6815/928755894","heartbeat_front_addr":"172.21.15.105:6813/928755894","state":["exists","up"]},{"osd":6,"uuid":"304db525-d24d-4687-bfa0-f92bdf66d1e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6819","nonce":897979441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6821","nonce":897979441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6825","nonce":897979441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6823","nonce":897979441}]},"public_addr":"172.21.15.28:6819/897979441","cluster_addr":"172.21.15.28:6821/897979441","heartbeat_back_addr":"172.21.15.28:6825/897979441","heartbeat_front_addr":"172.21.15.28:6823/897979441","state":["exists","up"]},{"osd":7,"uuid":"08e01d59-03f3-47b1-b45f-35ec5a50992c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6817","nonce":2400251786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6819","nonce":2400251786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6823","nonce":2400251786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6821","nonce":2400251786}]},"public_addr":"172.21.15.38:6817/2400251786","cluster_addr":"172.21.15.38:6819/2400251786","heartbeat_back_addr":"172.21.15.38:6823/2400251786","heartbeat_front_addr":"172.21.15.38:6821/2400251786","state":["exists","up"]},{"osd":8,"uuid":"64ee1e95-0756-48b1-9c00-472d0c3cbd29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6817","nonce":1682312269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6819","nonce":1682312269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6823","nonce":1682312269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6821","nonce":1682312269}]},"public_addr":"172.21.15.105:6817/1682312269","cluster_addr":"172.21.15.105:6819/1682312269","heartbeat_back_addr":"172.21.15.105:6823/1682312269","heartbeat_front_addr":"172.21.15.105:6821/1682312269","state":["exists","up"]},{"osd":9,"uuid":"18f60893-3ffd-48c2-9db5-a67d35611d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6827","nonce":1292942516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6829","nonce":1292942516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6833","nonce":1292942516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6831","nonce":1292942516}]},"public_addr":"172.21.15.28:6827/1292942516","cluster_addr":"172.21.15.28:6829/1292942516","heartbeat_back_addr":"172.21.15.28:6833/1292942516","heartbeat_front_addr":"172.21.15.28:6831/1292942516","state":["exists","up"]},{"osd":10,"uuid":"76c06725-7040-4118-8894-fe808a87f999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6825","nonce":2444616468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6827","nonce":2444616468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6831","nonce":2444616468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6829","nonce":2444616468}]},"public_addr":"172.21.15.38:6825/2444616468","cluster_addr":"172.21.15.38:6827/2444616468","heartbeat_back_addr":"172.21.15.38:6831/2444616468","heartbeat_front_addr":"172.21.15.38:6829/2444616468","state":["exists","up"]},{"osd":11,"uuid":"e8fc9454-5540-4e2e-91bf-3ba251dbd1a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6825","nonce":3400299562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6827","nonce":3400299562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6831","nonce":3400299562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6829","nonce":3400299562}]},"public_addr":"172.21.15.105:6825/3400299562","cluster_addr":"172.21.15.105:6827/3400299562","heartbeat_back_addr":"172.21.15.105:6831/3400299562","heartbeat_front_addr":"172.21.15.105:6829/3400299562","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:35:33.615395+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:00.545293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:24.940211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:57.684887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:24.519061+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:49.010804+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:22.638478+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:49.559104+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:14.011771+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:47.859950+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:16.298445+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:41.619069+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.2d","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/590495740":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/1163172512":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/4290328460":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/4145020172":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/720626958":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/948450262":"2024-04-24T14:33:07.576363+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:43:50.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.517+0000 7fd4b0c00640 1 -- 172.21.15.28:0/2870364607 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd4900a3c30 msgr2=0x7fd4900a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:50.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.517+0000 7fd4b0c00640 1 --2- 172.21.15.28:0/2870364607 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd4900a3c30 0x7fd4900a60f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac1094d0 tx=0x7fd49c005f90 comp rx=0 tx=0).stop 2024-04-23T14:43:50.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.517+0000 7fd4b0c00640 1 -- 172.21.15.28:0/2870364607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4ac10a030 msgr2=0x7fd4ac1a5cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:50.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.517+0000 7fd4b0c00640 1 --2- 172.21.15.28:0/2870364607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4ac10a030 0x7fd4ac1a5cd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4031ae0 tx=0x7fd4a4031ee0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.517+0000 7fd4b0c00640 1 -- 172.21.15.28:0/2870364607 shutdown_connections 2024-04-23T14:43:50.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.517+0000 7fd4b0c00640 1 --2- 172.21.15.28:0/2870364607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4ac10a030 0x7fd4ac1a5cd0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.517+0000 7fd4b0c00640 1 --2- 172.21.15.28:0/2870364607 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4ac109670 0x7fd4ac19ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.517+0000 7fd4b0c00640 1 --2- 172.21.15.28:0/2870364607 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd4900a3c30 0x7fd4900a60f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.517+0000 7fd4b0c00640 1 --2- 172.21.15.28:0/2870364607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd4ac108480 0x7fd4ac19e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.517+0000 7fd4b0c00640 1 -- 172.21.15.28:0/2870364607 >> 172.21.15.28:0/2870364607 conn(0x7fd4ac077440 msgr2=0x7fd4ac10c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:50.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.517+0000 7fd4b0c00640 1 -- 172.21.15.28:0/2870364607 shutdown_connections 2024-04-23T14:43:50.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.517+0000 7fd4b0c00640 1 -- 172.21.15.28:0/2870364607 wait complete. 2024-04-23T14:43:50.538 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-23T14:41:39.495067+0000', 'modified': '2024-04-23T14:41:43.869888+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24587}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24583': {'gid': 24583, 'name': 'g', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.28:6839/940220693', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6838', 'nonce': 940220693}, {'type': 'v1', 'addr': '172.21.15.28:6839', 'nonce': 940220693}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24587': {'gid': 24587, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.38:6839/377787688', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6838', 'nonce': 377787688}, {'type': 'v1', 'addr': '172.21.15.38:6839', 'nonce': 377787688}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24587, 'qdb_cluster': [24587]} 2024-04-23T14:43:50.538 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-23T14:43:50.538 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-23T14:43:50.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.740+0000 7f89f8200640 1 -- 172.21.15.28:0/3108605517 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89f0105470 msgr2=0x7f89f01058f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:50.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.740+0000 7f89f8200640 1 --2- 172.21.15.28:0/3108605517 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89f0105470 0x7f89f01058f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f89dc009900 tx=0x7f89dc02a3f0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.740+0000 7f89f8200640 1 -- 172.21.15.28:0/3108605517 shutdown_connections 2024-04-23T14:43:50.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.740+0000 7f89f8200640 1 --2- 172.21.15.28:0/3108605517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89f0105e30 0x7f89f010c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.740+0000 7f89f8200640 1 --2- 172.21.15.28:0/3108605517 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89f0105470 0x7f89f01058f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.740+0000 7f89f8200640 1 --2- 172.21.15.28:0/3108605517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89f0104280 0x7f89f01046a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.740+0000 7f89f8200640 1 -- 172.21.15.28:0/3108605517 >> 172.21.15.28:0/3108605517 conn(0x7f89f00ff7f0 msgr2=0x7f89f0101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:50.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.740+0000 7f89f8200640 1 -- 172.21.15.28:0/3108605517 shutdown_connections 2024-04-23T14:43:50.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.740+0000 7f89f8200640 1 -- 172.21.15.28:0/3108605517 wait complete. 2024-04-23T14:43:50.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.742+0000 7f89f8200640 1 Processor -- start 2024-04-23T14:43:50.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.742+0000 7f89f8200640 1 -- start start 2024-04-23T14:43:50.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.742+0000 7f89f8200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89f0104280 0x7f89f019a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:50.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.742+0000 7f89f8200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89f0105470 0x7f89f019aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:50.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.743+0000 7f89f8200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89f0105e30 0x7f89f01a1b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:50.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.743+0000 7f89f8200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f89f010ff10 con 0x7f89f0104280 2024-04-23T14:43:50.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.743+0000 7f89f8200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f89f010fd90 con 0x7f89f0105470 2024-04-23T14:43:50.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.743+0000 7f89f8200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f89f0110090 con 0x7f89f0105e30 2024-04-23T14:43:50.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.743+0000 7f89f5800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89f0104280 0x7f89f019a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:50.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.743+0000 7f89f4e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89f0105470 0x7f89f019aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:50.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.743+0000 7f89f4e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89f0105470 0x7f89f019aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:53608/0 (socket says 172.21.15.28:53608) 2024-04-23T14:43:50.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.743+0000 7f89f4e00640 1 -- 172.21.15.28:0/3685964369 learned_addr learned my addr 172.21.15.28:0/3685964369 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:50.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.743+0000 7f89f6200640 1 --2- 172.21.15.28:0/3685964369 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89f0105e30 0x7f89f01a1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:50.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.743+0000 7f89f5800640 1 -- 172.21.15.28:0/3685964369 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89f0105e30 msgr2=0x7f89f01a1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:50.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.743+0000 7f89f5800640 1 --2- 172.21.15.28:0/3685964369 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89f0105e30 0x7f89f01a1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.743+0000 7f89f5800640 1 -- 172.21.15.28:0/3685964369 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89f0105470 msgr2=0x7f89f019aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:50.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.743+0000 7f89f5800640 1 --2- 172.21.15.28:0/3685964369 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89f0105470 0x7f89f019aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.743+0000 7f89f5800640 1 -- 172.21.15.28:0/3685964369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89f01a21c0 con 0x7f89f0104280 2024-04-23T14:43:50.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.744+0000 7f89f5800640 1 --2- 172.21.15.28:0/3685964369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89f0104280 0x7f89f019a5a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f89e000e9c0 tx=0x7f89e000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:50.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.744+0000 7f89ee000640 1 -- 172.21.15.28:0/3685964369 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e000ccc0 con 0x7f89f0104280 2024-04-23T14:43:50.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.744+0000 7f89f8200640 1 -- 172.21.15.28:0/3685964369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89f01a24b0 con 0x7f89f0104280 2024-04-23T14:43:50.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.744+0000 7f89ee000640 1 -- 172.21.15.28:0/3685964369 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89e00035f0 con 0x7f89f0104280 2024-04-23T14:43:50.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.744+0000 7f89ee000640 1 -- 172.21.15.28:0/3685964369 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e000ce60 con 0x7f89f0104280 2024-04-23T14:43:50.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.744+0000 7f89f8200640 1 -- 172.21.15.28:0/3685964369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89f01a2980 con 0x7f89f0104280 2024-04-23T14:43:50.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.748+0000 7f89fb035740 1 Processor -- start 2024-04-23T14:43:50.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.748+0000 7f89fb035740 1 -- start start 2024-04-23T14:43:50.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.748+0000 7f89ee000640 1 -- 172.21.15.28:0/3685964369 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f89e00037a0 con 0x7f89f0104280 2024-04-23T14:43:50.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.748+0000 7f89fb035740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1bc01f090 0x55d1bc01f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:50.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.748+0000 7f89fb035740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d1bc01fa80 0x55d1bc024580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:50.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.748+0000 7f89fb035740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1bc024bf0 0x55d1bc027080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:50.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.748+0000 7f89fb035740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d1bbf84b10 con 0x55d1bc01f090 2024-04-23T14:43:50.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.748+0000 7f89fb035740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d1bbf2b2b0 con 0x55d1bc024bf0 2024-04-23T14:43:50.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.748+0000 7f89fb035740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d1bbf80ad0 con 0x55d1bc01fa80 2024-04-23T14:43:50.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.749+0000 7f89f5800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d1bc01fa80 0x55d1bc024580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:50.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.749+0000 7f89f6200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1bc01f090 0x55d1bc01f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:50.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.749+0000 7f89f4e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1bc024bf0 0x55d1bc027080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:50.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.749+0000 7f89f5800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d1bc01fa80 0x55d1bc024580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:40786/0 (socket says 172.21.15.28:40786) 2024-04-23T14:43:50.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.749+0000 7f89f5800640 1 -- 172.21.15.28:0/3123263958 learned_addr learned my addr 172.21.15.28:0/3123263958 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:50.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.749+0000 7f89f4e00640 1 -- 172.21.15.28:0/3123263958 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d1bc01fa80 msgr2=0x55d1bc024580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:43:50.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.749+0000 7f89f4e00640 1 --2- 172.21.15.28:0/3123263958 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d1bc01fa80 0x55d1bc024580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.749+0000 7f89f4e00640 1 -- 172.21.15.28:0/3123263958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1bc01f090 msgr2=0x55d1bc01f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:43:50.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.749+0000 7f89f4e00640 1 --2- 172.21.15.28:0/3123263958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1bc01f090 0x55d1bc01f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.749+0000 7f89f4e00640 1 -- 172.21.15.28:0/3123263958 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1bbc972d0 con 0x55d1bc024bf0 2024-04-23T14:43:50.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.749+0000 7f89f5800640 1 --2- 172.21.15.28:0/3123263958 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d1bc01fa80 0x55d1bc024580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:43:50.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.750+0000 7f89f4e00640 1 --2- 172.21.15.28:0/3123263958 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1bc024bf0 0x55d1bc027080 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f89dc0314e0 tx=0x7f89dc005dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=492602de98760172 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:50.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.750+0000 7f89ee000640 1 --2- 172.21.15.28:0/3685964369 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f89d40a3bb0 0x7f89d40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:50.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.750+0000 7f89e6a00640 1 -- 172.21.15.28:0/3123263958 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89dc040070 con 0x55d1bc024bf0 2024-04-23T14:43:50.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.750+0000 7f89e6a00640 1 -- 172.21.15.28:0/3123263958 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89dc005030 con 0x55d1bc024bf0 2024-04-23T14:43:50.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.750+0000 7f89e6a00640 1 -- 172.21.15.28:0/3123263958 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89dc035040 con 0x55d1bc024bf0 2024-04-23T14:43:50.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.750+0000 7f89ee000640 1 -- 172.21.15.28:0/3685964369 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7f89e00d06f0 con 0x7f89f0104280 2024-04-23T14:43:50.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.750+0000 7f89f6200640 1 --2- 172.21.15.28:0/3685964369 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f89d40a3bb0 0x7f89d40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:50.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.751+0000 7f89fb035740 1 -- 172.21.15.28:0/3123263958 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1bc024bf0 msgr2=0x55d1bc027080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:50.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.751+0000 7f89fb035740 1 --2- 172.21.15.28:0/3123263958 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1bc024bf0 0x55d1bc027080 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f89dc0314e0 tx=0x7f89dc005dd0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.751+0000 7f89fb035740 1 -- 172.21.15.28:0/3123263958 shutdown_connections 2024-04-23T14:43:50.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.751+0000 7f89fb035740 1 --2- 172.21.15.28:0/3123263958 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1bc024bf0 0x55d1bc027080 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.751+0000 7f89fb035740 1 --2- 172.21.15.28:0/3123263958 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d1bc01fa80 0x55d1bc024580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.751+0000 7f89fb035740 1 --2- 172.21.15.28:0/3123263958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1bc01f090 0x55d1bc01f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.751+0000 7f89fb035740 1 -- 172.21.15.28:0/3123263958 >> 172.21.15.28:0/3123263958 conn(0x55d1bc01a700 msgr2=0x55d1bc01ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:50.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.751+0000 7f89f6200640 1 --2- 172.21.15.28:0/3685964369 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f89d40a3bb0 0x7f89d40a6070 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x55d1bc0202e0 tx=0x7f89e8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:50.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.751+0000 7f89fb035740 1 -- 172.21.15.28:0/3123263958 shutdown_connections 2024-04-23T14:43:50.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.751+0000 7f89fb035740 1 -- 172.21.15.28:0/3123263958 wait complete. 2024-04-23T14:43:50.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.751+0000 7f89fb035740 1 -- start start 2024-04-23T14:43:50.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.752+0000 7f89fb035740 1 Processor -- start 2024-04-23T14:43:50.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.752+0000 7f89fb035740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1bc01f090 0x55d1bc029460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:50.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.752+0000 7f89fb035740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1bc01fa80 0x55d1bc0299a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:50.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.752+0000 7f89fb035740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d1bc024bf0 0x55d1bc0309e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:50.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.752+0000 7f89fb035740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d1bc027b40 con 0x55d1bc01fa80 2024-04-23T14:43:50.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.752+0000 7f89fb035740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d1bbf3e590 con 0x55d1bc01f090 2024-04-23T14:43:50.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.752+0000 7f89fb035740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d1bc027cc0 con 0x55d1bc024bf0 2024-04-23T14:43:50.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.752+0000 7f89fb035740 10 client.0 _ll_register_callbacks cb 0x55d1bbf213a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:43:50.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.752+0000 7f89f4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1bc01fa80 0x55d1bc0299a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:50.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.752+0000 7f89f4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1bc01fa80 0x55d1bc0299a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45290/0 (socket says 172.21.15.28:45290) 2024-04-23T14:43:50.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.752+0000 7f89f4e00640 1 -- 172.21.15.28:0/2710776002 learned_addr learned my addr 172.21.15.28:0/2710776002 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:50.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.752+0000 7f89f6200640 1 --2- 172.21.15.28:0/2710776002 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d1bc024bf0 0x55d1bc0309e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:50.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.752+0000 7f89f5800640 1 --2- 172.21.15.28:0/2710776002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1bc01f090 0x55d1bc029460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:50.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.753+0000 7f89f4e00640 1 -- 172.21.15.28:0/2710776002 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d1bc024bf0 msgr2=0x55d1bc0309e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:43:50.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.753+0000 7f89f4e00640 1 --2- 172.21.15.28:0/2710776002 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d1bc024bf0 0x55d1bc0309e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.753+0000 7f89f4e00640 1 -- 172.21.15.28:0/2710776002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1bc01f090 msgr2=0x55d1bc029460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:43:50.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.753+0000 7f89f4e00640 1 --2- 172.21.15.28:0/2710776002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1bc01f090 0x55d1bc029460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.753+0000 7f89f4e00640 1 -- 172.21.15.28:0/2710776002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1bbd616b0 con 0x55d1bc01fa80 2024-04-23T14:43:50.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.753+0000 7f89f4e00640 1 --2- 172.21.15.28:0/2710776002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1bc01fa80 0x55d1bc0299a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f89dc005ab0 tx=0x7f89dc009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f36ca7ccf74a3416 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:50.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.753+0000 7f89db400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-23T14:43:50.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.753+0000 7f89db400640 1 -- 172.21.15.28:0/2710776002 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89dc040030 con 0x55d1bc01fa80 2024-04-23T14:43:50.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.754+0000 7f89fb035740 1 -- 172.21.15.28:0/2710776002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89dc005030 con 0x55d1bc01fa80 2024-04-23T14:43:50.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.754+0000 7f89db400640 1 -- 172.21.15.28:0/2710776002 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7f89dc004230 con 0x55d1bc01fa80 2024-04-23T14:43:50.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.754+0000 7f89db400640 1 -- 172.21.15.28:0/2710776002 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f89dc035070 con 0x55d1bc01fa80 2024-04-23T14:43:50.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.754+0000 7f89fb035740 10 client.14988 fetch_fsmap learned FSMap version 20 2024-04-23T14:43:50.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.754+0000 7f89fb035740 1 -- 172.21.15.28:0/2710776002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55d1bc033d50 con 0x55d1bc01fa80 2024-04-23T14:43:50.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.754+0000 7f89db400640 1 -- 172.21.15.28:0/2710776002 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89dc005030 con 0x55d1bc01fa80 2024-04-23T14:43:50.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.754+0000 7f89db400640 1 -- 172.21.15.28:0/2710776002 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f89dc0035f0 con 0x55d1bc01fa80 2024-04-23T14:43:50.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.755+0000 7f89fb035740 10 client.14988 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:43:50.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.755+0000 7f89fb035740 10 client.14988 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-23T14:43:50.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.755+0000 7f89fb035740 20 client.14988 populate_metadata read hostname 'smithi028' 2024-04-23T14:43:50.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.755+0000 7f89fb035740 1 --2- 172.21.15.28:0/2710776002 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d1bc035f80 0x55d1bc0373f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:50.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.755+0000 7f89fb035740 4 client.14988 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:43:50.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.755+0000 7f89fb035740 1 -- 172.21.15.28:0/2710776002 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d1bbf92530 con 0x55d1bc035f80 2024-04-23T14:43:50.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.755+0000 7f89f5800640 1 --2- 172.21.15.28:0/2710776002 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d1bc035f80 0x55d1bc0373f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:50.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.756+0000 7f89f5800640 1 --2- 172.21.15.28:0/2710776002 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d1bc035f80 0x55d1bc0373f0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a50a3a1ff0ce820 server_cookie=414b6199f2bd0c71 in_seq=0 out_seq=0 2024-04-23T14:43:50.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.756+0000 7f89db400640 10 client.14988 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:43:50.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.756+0000 7f89db400640 1 -- 172.21.15.28:0/2710776002 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d1bbf92530 con 0x55d1bc035f80 2024-04-23T14:43:50.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.756+0000 7f89db400640 10 client.14988 handle_command_reply: tid=0 2024-04-23T14:43:50.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.756+0000 7f89fb035740 1 client.14988 shutdown 2024-04-23T14:43:50.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.757+0000 7f89fb035740 1 -- 172.21.15.28:0/2710776002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1bc01fa80 msgr2=0x55d1bc0299a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:50.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.757+0000 7f89fb035740 1 --2- 172.21.15.28:0/2710776002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1bc01fa80 0x55d1bc0299a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f89dc005ab0 tx=0x7f89dc009520 comp rx=0 tx=0).stop 2024-04-23T14:43:50.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.757+0000 7f89fb035740 1 -- 172.21.15.28:0/2710776002 shutdown_connections 2024-04-23T14:43:50.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.757+0000 7f89fb035740 1 --2- 172.21.15.28:0/2710776002 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d1bc035f80 0x55d1bc0373f0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.757+0000 7f89fb035740 1 --2- 172.21.15.28:0/2710776002 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d1bc024bf0 0x55d1bc0309e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.757+0000 7f89fb035740 1 --2- 172.21.15.28:0/2710776002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1bc01fa80 0x55d1bc0299a0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.757+0000 7f89fb035740 1 --2- 172.21.15.28:0/2710776002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1bc01f090 0x55d1bc029460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.757+0000 7f89db400640 0 client.14988 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:43:50.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.757+0000 7f89fb035740 1 -- 172.21.15.28:0/2710776002 >> 172.21.15.28:0/2710776002 conn(0x55d1bc01a700 msgr2=0x55d1bc0198c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:50.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.757+0000 7f89fb035740 1 -- 172.21.15.28:0/2710776002 shutdown_connections 2024-04-23T14:43:50.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.757+0000 7f89fb035740 1 -- 172.21.15.28:0/2710776002 wait complete. 2024-04-23T14:43:50.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.757+0000 7f89fb035740 20 client.14988 trim_cache size 0 max 16384 2024-04-23T14:43:50.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.777+0000 7f89fb035740 1 Processor -- start 2024-04-23T14:43:50.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.777+0000 7f89fb035740 1 -- start start 2024-04-23T14:43:50.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.777+0000 7f89fb035740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1bc01f090 0x55d1bc0244b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:50.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.777+0000 7f89fb035740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d1bc01fa80 0x55d1bc0309a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:50.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.777+0000 7f89fb035740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1bc024bf0 0x55d1bc026fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:50.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.777+0000 7f89fb035740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d1bc035730 con 0x55d1bc024bf0 2024-04-23T14:43:50.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.777+0000 7f89fb035740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d1bc035e00 con 0x55d1bc01f090 2024-04-23T14:43:50.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.777+0000 7f89fb035740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d1bc0364d0 con 0x55d1bc01fa80 2024-04-23T14:43:50.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.778+0000 7f89f6200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1bc024bf0 0x55d1bc026fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:50.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.778+0000 7f89f5800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1bc01f090 0x55d1bc0244b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:50.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.778+0000 7f89f5800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1bc01f090 0x55d1bc0244b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:53636/0 (socket says 172.21.15.28:53636) 2024-04-23T14:43:50.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.778+0000 7f89f5800640 1 -- 172.21.15.28:0/119602396 learned_addr learned my addr 172.21.15.28:0/119602396 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:50.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.778+0000 7f89f4e00640 1 --2- 172.21.15.28:0/119602396 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d1bc01fa80 0x55d1bc0309a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:50.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.778+0000 7f89f5800640 1 -- 172.21.15.28:0/119602396 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d1bc01fa80 msgr2=0x55d1bc0309a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:43:50.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.778+0000 7f89f5800640 1 --2- 172.21.15.28:0/119602396 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d1bc01fa80 0x55d1bc0309a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.778+0000 7f89f5800640 1 -- 172.21.15.28:0/119602396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1bc024bf0 msgr2=0x55d1bc026fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:43:50.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.778+0000 7f89f5800640 1 --2- 172.21.15.28:0/119602396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1bc024bf0 0x55d1bc026fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.778+0000 7f89f5800640 1 -- 172.21.15.28:0/119602396 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1bc016060 con 0x55d1bc01f090 2024-04-23T14:43:50.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.778+0000 7f89f4e00640 1 --2- 172.21.15.28:0/119602396 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d1bc01fa80 0x55d1bc0309a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:43:50.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.778+0000 7f89f5800640 1 --2- 172.21.15.28:0/119602396 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1bc01f090 0x55d1bc0244b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f89e0098e30 tx=0x7f89e00ce0c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f16103986af22c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:50.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.779+0000 7f89e6000640 1 -- 172.21.15.28:0/119602396 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e009c330 con 0x55d1bc01f090 2024-04-23T14:43:50.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.779+0000 7f89e6000640 1 -- 172.21.15.28:0/119602396 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89e009d0d0 con 0x55d1bc01f090 2024-04-23T14:43:50.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.779+0000 7f89e6000640 1 -- 172.21.15.28:0/119602396 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e00ce300 con 0x55d1bc01f090 2024-04-23T14:43:50.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.779+0000 7f89fb035740 1 -- 172.21.15.28:0/119602396 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1bc01f090 msgr2=0x55d1bc0244b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:50.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.779+0000 7f89fb035740 1 --2- 172.21.15.28:0/119602396 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1bc01f090 0x55d1bc0244b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f89e0098e30 tx=0x7f89e00ce0c0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.779+0000 7f89fb035740 1 -- 172.21.15.28:0/119602396 shutdown_connections 2024-04-23T14:43:50.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.779+0000 7f89fb035740 1 --2- 172.21.15.28:0/119602396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1bc024bf0 0x55d1bc026fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.779+0000 7f89fb035740 1 --2- 172.21.15.28:0/119602396 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d1bc01fa80 0x55d1bc0309a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.779+0000 7f89fb035740 1 --2- 172.21.15.28:0/119602396 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1bc01f090 0x55d1bc0244b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.779+0000 7f89fb035740 1 -- 172.21.15.28:0/119602396 >> 172.21.15.28:0/119602396 conn(0x55d1bc01a700 msgr2=0x55d1bc01aae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:50.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.780+0000 7f89fb035740 1 -- 172.21.15.28:0/119602396 shutdown_connections 2024-04-23T14:43:50.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.780+0000 7f89fb035740 1 -- 172.21.15.28:0/119602396 wait complete. 2024-04-23T14:43:50.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.780+0000 7f89fb035740 1 -- start start 2024-04-23T14:43:50.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.780+0000 7f89fb035740 1 Processor -- start 2024-04-23T14:43:50.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.781+0000 7f89fb035740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d1bc01f090 0x55d1bc02cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:50.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.781+0000 7f89fb035740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1bc01fa80 0x55d1bc02d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:50.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.781+0000 7f89fb035740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1bc024bf0 0x55d1bc02fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:50.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.781+0000 7f89fb035740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d1bc037270 con 0x55d1bc01fa80 2024-04-23T14:43:50.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.781+0000 7f89fb035740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d1bc036ba0 con 0x55d1bc024bf0 2024-04-23T14:43:50.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.781+0000 7f89fb035740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d1bc037940 con 0x55d1bc01f090 2024-04-23T14:43:50.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.781+0000 7f89fb035740 10 client.0 _ll_register_callbacks cb 0x7f89dc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:43:50.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.781+0000 7f89f4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1bc01fa80 0x55d1bc02d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:50.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.781+0000 7f89f4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1bc01fa80 0x55d1bc02d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45304/0 (socket says 172.21.15.28:45304) 2024-04-23T14:43:50.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.781+0000 7f89f4e00640 1 -- 172.21.15.28:0/926093779 learned_addr learned my addr 172.21.15.28:0/926093779 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:50.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.781+0000 7f89f5800640 1 --2- 172.21.15.28:0/926093779 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d1bc01f090 0x55d1bc02cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:50.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.781+0000 7f89f6200640 1 --2- 172.21.15.28:0/926093779 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1bc024bf0 0x55d1bc02fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:50.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.781+0000 7f89f4e00640 1 -- 172.21.15.28:0/926093779 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d1bc01f090 msgr2=0x55d1bc02cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:43:50.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.781+0000 7f89f4e00640 1 --2- 172.21.15.28:0/926093779 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d1bc01f090 0x55d1bc02cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.781+0000 7f89f4e00640 1 -- 172.21.15.28:0/926093779 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1bc024bf0 msgr2=0x55d1bc02fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:43:50.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.781+0000 7f89f4e00640 1 --2- 172.21.15.28:0/926093779 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1bc024bf0 0x55d1bc02fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.781+0000 7f89f4e00640 1 -- 172.21.15.28:0/926093779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1bc030020 con 0x55d1bc01fa80 2024-04-23T14:43:50.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.782+0000 7f89f4e00640 1 --2- 172.21.15.28:0/926093779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1bc01fa80 0x55d1bc02d110 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f89dc004650 tx=0x7f89dc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=516a8c7f8eb33577 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:50.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.782+0000 7f89db400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-23T14:43:50.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.782+0000 7f89db400640 1 -- 172.21.15.28:0/926093779 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89dc040040 con 0x55d1bc01fa80 2024-04-23T14:43:50.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.782+0000 7f89db400640 1 -- 172.21.15.28:0/926093779 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7f89dc04a070 con 0x55d1bc01fa80 2024-04-23T14:43:50.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.782+0000 7f89fb035740 1 -- 172.21.15.28:0/926093779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89e009d0d0 con 0x55d1bc01fa80 2024-04-23T14:43:50.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.782+0000 7f89db400640 1 -- 172.21.15.28:0/926093779 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f89dc068050 con 0x55d1bc01fa80 2024-04-23T14:43:50.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.782+0000 7f89fb035740 10 client.14994 fetch_fsmap learned FSMap version 20 2024-04-23T14:43:50.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.782+0000 7f89fb035740 1 -- 172.21.15.28:0/926093779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55d1bc0a4b80 con 0x55d1bc01fa80 2024-04-23T14:43:50.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.782+0000 7f89db400640 1 -- 172.21.15.28:0/926093779 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89e009d0d0 con 0x55d1bc01fa80 2024-04-23T14:43:50.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.783+0000 7f89db400640 1 -- 172.21.15.28:0/926093779 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f89dc03ca40 con 0x55d1bc01fa80 2024-04-23T14:43:50.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.783+0000 7f89fb035740 10 client.14994 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:43:50.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.783+0000 7f89fb035740 10 client.14994 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-23T14:43:50.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.783+0000 7f89fb035740 20 client.14994 populate_metadata read hostname 'smithi028' 2024-04-23T14:43:50.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.783+0000 7f89fb035740 1 --2- 172.21.15.28:0/926093779 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d1bc0a9750 0x55d1bc0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:50.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.783+0000 7f89fb035740 4 client.14994 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T14:43:50.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.783+0000 7f89fb035740 1 -- 172.21.15.28:0/926093779 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "status"}) -- 0x55d1bbf8c3a0 con 0x55d1bc0a9750 2024-04-23T14:43:50.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.783+0000 7f89f5800640 1 --2- 172.21.15.28:0/926093779 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d1bc0a9750 0x55d1bc0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:50.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.784+0000 7f89f5800640 1 --2- 172.21.15.28:0/926093779 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d1bc0a9750 0x55d1bc0a6260 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e010e2d0e8f5f9bd server_cookie=85f8f1f19dc6d946 in_seq=0 out_seq=0 2024-04-23T14:43:50.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.784+0000 7f89db400640 10 client.14994 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:43:50.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.784+0000 7f89db400640 1 -- 172.21.15.28:0/926093779 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55d1bbf8c3a0 con 0x55d1bc0a9750 2024-04-23T14:43:50.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.784+0000 7f89db400640 10 client.14994 handle_command_reply: tid=0 2024-04-23T14:43:50.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.784+0000 7f89fb035740 1 client.14994 shutdown 2024-04-23T14:43:50.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.784+0000 7f89fb035740 1 -- 172.21.15.28:0/926093779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1bc01fa80 msgr2=0x55d1bc02d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:50.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.784+0000 7f89fb035740 1 --2- 172.21.15.28:0/926093779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1bc01fa80 0x55d1bc02d110 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f89dc004650 tx=0x7f89dc000bc0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.784+0000 7f89fb035740 1 -- 172.21.15.28:0/926093779 shutdown_connections 2024-04-23T14:43:50.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.784+0000 7f89fb035740 1 --2- 172.21.15.28:0/926093779 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d1bc0a9750 0x55d1bc0a6260 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.784+0000 7f89fb035740 1 --2- 172.21.15.28:0/926093779 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1bc024bf0 0x55d1bc02fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.784+0000 7f89fb035740 1 --2- 172.21.15.28:0/926093779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1bc01fa80 0x55d1bc02d110 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.784+0000 7f89fb035740 1 --2- 172.21.15.28:0/926093779 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d1bc01f090 0x55d1bc02cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.784+0000 7f89fb035740 1 -- 172.21.15.28:0/926093779 >> 172.21.15.28:0/926093779 conn(0x55d1bc01a700 msgr2=0x55d1bc01df20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:50.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.784+0000 7f89db400640 0 client.14994 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:43:50.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.785+0000 7f89fb035740 1 -- 172.21.15.28:0/926093779 shutdown_connections 2024-04-23T14:43:50.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.785+0000 7f89fb035740 1 -- 172.21.15.28:0/926093779 wait complete. 2024-04-23T14:43:50.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.785+0000 7f89fb035740 20 client.14994 trim_cache size 0 max 16384 2024-04-23T14:43:50.791 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:43:50.791 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "1ee6e996-017e-11ef-bc93-c7b262605968", 2024-04-23T14:43:50.791 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-04-23T14:43:50.791 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24587, 2024-04-23T14:43:50.791 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-23T14:43:50.791 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-23T14:43:50.791 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-23T14:43:50.791 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 130.79472678499999, 2024-04-23T14:43:50.791 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 20, 2024-04-23T14:43:50.791 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 87, 2024-04-23T14:43:50.791 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 85, 2024-04-23T14:43:50.791 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 135.93191777800001 2024-04-23T14:43:50.791 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:43:50.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.785+0000 7f89f8200640 1 -- 172.21.15.28:0/3685964369 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f89d40a3bb0 msgr2=0x7f89d40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:50.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.785+0000 7f89f8200640 1 --2- 172.21.15.28:0/3685964369 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f89d40a3bb0 0x7f89d40a6070 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x55d1bc0202e0 tx=0x7f89e8009500 comp rx=0 tx=0).stop 2024-04-23T14:43:50.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.785+0000 7f89f8200640 1 -- 172.21.15.28:0/3685964369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89f0104280 msgr2=0x7f89f019a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:50.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.785+0000 7f89f8200640 1 --2- 172.21.15.28:0/3685964369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89f0104280 0x7f89f019a5a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f89e000e9c0 tx=0x7f89e000edc0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.786+0000 7f89f8200640 1 -- 172.21.15.28:0/3685964369 shutdown_connections 2024-04-23T14:43:50.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.786+0000 7f89f8200640 1 --2- 172.21.15.28:0/3685964369 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f89d40a3bb0 0x7f89d40a6070 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.786+0000 7f89f8200640 1 --2- 172.21.15.28:0/3685964369 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89f0105470 0x7f89f019aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.786+0000 7f89f8200640 1 --2- 172.21.15.28:0/3685964369 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89f0105e30 0x7f89f01a1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.786+0000 7f89f8200640 1 --2- 172.21.15.28:0/3685964369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89f0104280 0x7f89f019a5a0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:50.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.786+0000 7f89f8200640 1 -- 172.21.15.28:0/3685964369 >> 172.21.15.28:0/3685964369 conn(0x7f89f00ff7f0 msgr2=0x7f89f01087f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:50.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.786+0000 7f89f8200640 1 -- 172.21.15.28:0/3685964369 shutdown_connections 2024-04-23T14:43:50.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:50.786+0000 7f89f8200640 1 -- 172.21.15.28:0/3685964369 wait complete. 2024-04-23T14:43:50.802 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:50 smithi028 ceph-mon[27810]: pgmap v381: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:43:50.803 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:50 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/4017405352' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:43:50.803 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:50 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2870364607' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:43:50.806 INFO:teuthology.run_tasks:Running task sequential... 2024-04-23T14:43:50.816 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-23T14:43:50.816 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-23T14:43:50.816 INFO:teuthology.task.print:Enabling mgr modules 2024-04-23T14:43:50.816 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-23T14:43:50.817 INFO:teuthology.task.exec:Executing custom commands... 2024-04-23T14:43:50.817 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-04-23T14:43:50.817 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-23T14:43:50.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:50 smithi038 ceph-mon[32348]: pgmap v381: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:43:50.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:50 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/4017405352' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:43:50.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:50 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2870364607' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:43:51.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.016+0000 7f5703000640 1 -- 172.21.15.28:0/1960559006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56fc076040 msgr2=0x7f56fc074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:51.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.016+0000 7f5703000640 1 --2- 172.21.15.28:0/1960559006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56fc076040 0x7f56fc074490 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f56f0009cd0 tx=0x7f56f002a8e0 comp rx=0 tx=0).stop 2024-04-23T14:43:51.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.016+0000 7f5703000640 1 -- 172.21.15.28:0/1960559006 shutdown_connections 2024-04-23T14:43:51.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.016+0000 7f5703000640 1 --2- 172.21.15.28:0/1960559006 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f56fc10e460 0x7f56fc110850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:51.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.016+0000 7f5703000640 1 --2- 172.21.15.28:0/1960559006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56fc074a60 0x7f56fc074ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:51.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.016+0000 7f5703000640 1 --2- 172.21.15.28:0/1960559006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56fc076040 0x7f56fc074490 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:51.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.016+0000 7f5703000640 1 -- 172.21.15.28:0/1960559006 >> 172.21.15.28:0/1960559006 conn(0x7f56fc0fd400 msgr2=0x7f56fc0ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:51.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.016+0000 7f5703000640 1 -- 172.21.15.28:0/1960559006 shutdown_connections 2024-04-23T14:43:51.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.016+0000 7f5703000640 1 -- 172.21.15.28:0/1960559006 wait complete. 2024-04-23T14:43:51.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.017+0000 7f5703000640 1 Processor -- start 2024-04-23T14:43:51.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.017+0000 7f5703000640 1 -- start start 2024-04-23T14:43:51.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.018+0000 7f5703000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f56fc074a60 0x7f56fc19e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:51.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.018+0000 7f5703000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56fc076040 0x7f56fc19ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:51.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.018+0000 7f5703000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56fc10e460 0x7f56fc1a5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:51.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.018+0000 7f5703000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f56fc114130 con 0x7f56fc076040 2024-04-23T14:43:51.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.018+0000 7f5703000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f56fc113fb0 con 0x7f56fc10e460 2024-04-23T14:43:51.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.018+0000 7f5703000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f56fc1142b0 con 0x7f56fc074a60 2024-04-23T14:43:51.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.018+0000 7f5701200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56fc076040 0x7f56fc19ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:51.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.018+0000 7f5701200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56fc076040 0x7f56fc19ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45314/0 (socket says 172.21.15.28:45314) 2024-04-23T14:43:51.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.018+0000 7f5701200640 1 -- 172.21.15.28:0/3935327290 learned_addr learned my addr 172.21.15.28:0/3935327290 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:51.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.018+0000 7f5701c00640 1 --2- 172.21.15.28:0/3935327290 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f56fc074a60 0x7f56fc19e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:51.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.018+0000 7f5702600640 1 --2- 172.21.15.28:0/3935327290 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56fc10e460 0x7f56fc1a5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:51.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.018+0000 7f5701200640 1 -- 172.21.15.28:0/3935327290 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f56fc074a60 msgr2=0x7f56fc19e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:51.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.019+0000 7f5701200640 1 --2- 172.21.15.28:0/3935327290 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f56fc074a60 0x7f56fc19e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:51.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.019+0000 7f5701200640 1 -- 172.21.15.28:0/3935327290 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56fc10e460 msgr2=0x7f56fc1a5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:51.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.019+0000 7f5701200640 1 --2- 172.21.15.28:0/3935327290 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56fc10e460 0x7f56fc1a5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:51.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.019+0000 7f5701200640 1 -- 172.21.15.28:0/3935327290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56fc1a63e0 con 0x7f56fc076040 2024-04-23T14:43:51.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.019+0000 7f5701200640 1 --2- 172.21.15.28:0/3935327290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56fc076040 0x7f56fc19ed50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f56e800e9f0 tx=0x7f56e800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:51.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.019+0000 7f56fa000640 1 -- 172.21.15.28:0/3935327290 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e800cd80 con 0x7f56fc076040 2024-04-23T14:43:51.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.019+0000 7f56fa000640 1 -- 172.21.15.28:0/3935327290 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56e8010910 con 0x7f56fc076040 2024-04-23T14:43:51.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.019+0000 7f5703000640 1 -- 172.21.15.28:0/3935327290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56fc1a66d0 con 0x7f56fc076040 2024-04-23T14:43:51.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.019+0000 7f56fa000640 1 -- 172.21.15.28:0/3935327290 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e8018990 con 0x7f56fc076040 2024-04-23T14:43:51.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.019+0000 7f5703000640 1 -- 172.21.15.28:0/3935327290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56fc1a6c20 con 0x7f56fc076040 2024-04-23T14:43:51.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.020+0000 7f5703000640 1 -- 172.21.15.28:0/3935327290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56fc0505e0 con 0x7f56fc076040 2024-04-23T14:43:51.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.022+0000 7f56fa000640 1 -- 172.21.15.28:0/3935327290 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f56e8010ac0 con 0x7f56fc076040 2024-04-23T14:43:51.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.024+0000 7f56fa000640 1 --2- 172.21.15.28:0/3935327290 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f56e40a3ba0 0x7f56e40a6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:51.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.025+0000 7f56fa000640 1 -- 172.21.15.28:0/3935327290 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7f56e80d0810 con 0x7f56fc076040 2024-04-23T14:43:51.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.025+0000 7f5701c00640 1 --2- 172.21.15.28:0/3935327290 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f56e40a3ba0 0x7f56e40a6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:51.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.026+0000 7f5701c00640 1 --2- 172.21.15.28:0/3935327290 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f56e40a3ba0 0x7f56e40a6060 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f56f0005e30 tx=0x7f56f0009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:51.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.028+0000 7f56fa000640 1 -- 172.21.15.28:0/3935327290 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f56e809d1b0 con 0x7f56fc076040 2024-04-23T14:43:51.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:50 smithi105 ceph-mon[32555]: pgmap v381: 97 pgs: 97 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:43:51.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:50 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/4017405352' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:43:51.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:50 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2870364607' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:43:51.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.285+0000 7f5703000640 1 -- 172.21.15.28:0/3935327290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f56fc062c00 con 0x7f56fc076040 2024-04-23T14:43:51.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.613+0000 7f56fa000640 1 -- 172.21.15.28:0/3935327290 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f56e809cab0 con 0x7f56fc076040 2024-04-23T14:43:51.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.618+0000 7f5703000640 1 -- 172.21.15.28:0/3935327290 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f56e40a3ba0 msgr2=0x7f56e40a6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:51.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.618+0000 7f5703000640 1 --2- 172.21.15.28:0/3935327290 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f56e40a3ba0 0x7f56e40a6060 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f56f0005e30 tx=0x7f56f0009360 comp rx=0 tx=0).stop 2024-04-23T14:43:51.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.618+0000 7f5703000640 1 -- 172.21.15.28:0/3935327290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56fc076040 msgr2=0x7f56fc19ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:51.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.618+0000 7f5703000640 1 --2- 172.21.15.28:0/3935327290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56fc076040 0x7f56fc19ed50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f56e800e9f0 tx=0x7f56e800eeb0 comp rx=0 tx=0).stop 2024-04-23T14:43:51.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.618+0000 7f5703000640 1 -- 172.21.15.28:0/3935327290 shutdown_connections 2024-04-23T14:43:51.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.618+0000 7f5703000640 1 --2- 172.21.15.28:0/3935327290 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f56e40a3ba0 0x7f56e40a6060 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:51.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.618+0000 7f5703000640 1 --2- 172.21.15.28:0/3935327290 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56fc10e460 0x7f56fc1a5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:51.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.618+0000 7f5703000640 1 --2- 172.21.15.28:0/3935327290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56fc076040 0x7f56fc19ed50 secure :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f56e800e9f0 tx=0x7f56e800eeb0 comp rx=0 tx=0).stop 2024-04-23T14:43:51.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.618+0000 7f5703000640 1 --2- 172.21.15.28:0/3935327290 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f56fc074a60 0x7f56fc19e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:51.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.618+0000 7f5703000640 1 -- 172.21.15.28:0/3935327290 >> 172.21.15.28:0/3935327290 conn(0x7f56fc0fd400 msgr2=0x7f56fc101800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:51.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.620+0000 7f5703000640 1 -- 172.21.15.28:0/3935327290 shutdown_connections 2024-04-23T14:43:51.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.620+0000 7f5703000640 1 -- 172.21.15.28:0/3935327290 wait complete. 2024-04-23T14:43:51.651 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-23T14:43:51.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.832+0000 7f1980000640 1 -- 172.21.15.28:0/805547812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1978104280 msgr2=0x7f19781046a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:51.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.832+0000 7f1980000640 1 --2- 172.21.15.28:0/805547812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1978104280 0x7f19781046a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1968009ca0 tx=0x7f196802a8c0 comp rx=0 tx=0).stop 2024-04-23T14:43:51.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.832+0000 7f1980000640 1 -- 172.21.15.28:0/805547812 shutdown_connections 2024-04-23T14:43:51.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.832+0000 7f1980000640 1 --2- 172.21.15.28:0/805547812 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1978105470 0x7f19781058f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:51.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.832+0000 7f1980000640 1 --2- 172.21.15.28:0/805547812 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1978105e30 0x7f197810c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:51.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.832+0000 7f1980000640 1 --2- 172.21.15.28:0/805547812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1978104280 0x7f19781046a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:51.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.832+0000 7f1980000640 1 -- 172.21.15.28:0/805547812 >> 172.21.15.28:0/805547812 conn(0x7f19780ff7f0 msgr2=0x7f1978101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:51.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.832+0000 7f1980000640 1 -- 172.21.15.28:0/805547812 shutdown_connections 2024-04-23T14:43:51.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.832+0000 7f1980000640 1 -- 172.21.15.28:0/805547812 wait complete. 2024-04-23T14:43:51.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.833+0000 7f1980000640 1 Processor -- start 2024-04-23T14:43:51.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.833+0000 7f1980000640 1 -- start start 2024-04-23T14:43:51.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.834+0000 7f1980000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1978104280 0x7f1978079db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:51.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.834+0000 7f1980000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1978105470 0x7f197807a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:51.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.834+0000 7f1980000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1978105e30 0x7f1978076920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:51.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.834+0000 7f1980000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f197810ff10 con 0x7f1978105470 2024-04-23T14:43:51.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.834+0000 7f1980000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f197810fd90 con 0x7f1978104280 2024-04-23T14:43:51.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.834+0000 7f1980000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1978110090 con 0x7f1978105e30 2024-04-23T14:43:51.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.834+0000 7f197d600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1978104280 0x7f1978079db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:51.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.834+0000 7f197cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1978105470 0x7f197807a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:51.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.834+0000 7f197d600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1978104280 0x7f1978079db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:53674/0 (socket says 172.21.15.28:53674) 2024-04-23T14:43:51.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.834+0000 7f197d600640 1 -- 172.21.15.28:0/1065435620 learned_addr learned my addr 172.21.15.28:0/1065435620 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:51.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.834+0000 7f197e000640 1 --2- 172.21.15.28:0/1065435620 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1978105e30 0x7f1978076920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:51.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.834+0000 7f197d600640 1 -- 172.21.15.28:0/1065435620 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1978105e30 msgr2=0x7f1978076920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:51.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.834+0000 7f197d600640 1 --2- 172.21.15.28:0/1065435620 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1978105e30 0x7f1978076920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:51.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.834+0000 7f197d600640 1 -- 172.21.15.28:0/1065435620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1978105470 msgr2=0x7f197807a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:51.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.834+0000 7f197d600640 1 --2- 172.21.15.28:0/1065435620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1978105470 0x7f197807a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:51.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.834+0000 7f197d600640 1 -- 172.21.15.28:0/1065435620 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19780771e0 con 0x7f1978104280 2024-04-23T14:43:51.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.835+0000 7f197d600640 1 --2- 172.21.15.28:0/1065435620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1978104280 0x7f1978079db0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1968009bf0 tx=0x7f1968031ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:51.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.835+0000 7f1976000640 1 -- 172.21.15.28:0/1065435620 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196802fa20 con 0x7f1978104280 2024-04-23T14:43:51.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.835+0000 7f1980000640 1 -- 172.21.15.28:0/1065435620 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19781a6930 con 0x7f1978104280 2024-04-23T14:43:51.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.835+0000 7f1976000640 1 -- 172.21.15.28:0/1065435620 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f196802fbc0 con 0x7f1978104280 2024-04-23T14:43:51.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.835+0000 7f1976000640 1 -- 172.21.15.28:0/1065435620 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19680466e0 con 0x7f1978104280 2024-04-23T14:43:51.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.835+0000 7f1980000640 1 -- 172.21.15.28:0/1065435620 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19781a6d30 con 0x7f1978104280 2024-04-23T14:43:51.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.837+0000 7f1980000640 1 -- 172.21.15.28:0/1065435620 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f197804fd90 con 0x7f1978104280 2024-04-23T14:43:51.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.838+0000 7f1976000640 1 -- 172.21.15.28:0/1065435620 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f196802fd70 con 0x7f1978104280 2024-04-23T14:43:51.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.839+0000 7f1976000640 1 --2- 172.21.15.28:0/1065435620 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f195c0a3ca0 0x7f195c0a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:51.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.840+0000 7f197cc00640 1 -- 172.21.15.28:0/1065435620 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f195c0a3ca0 msgr2=0x7f195c0a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1997136173 2024-04-23T14:43:51.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.840+0000 7f1976000640 1 -- 172.21.15.28:0/1065435620 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7f19680f17c0 con 0x7f1978104280 2024-04-23T14:43:51.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.840+0000 7f197cc00640 1 --2- 172.21.15.28:0/1065435620 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f195c0a3ca0 0x7f195c0a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T14:43:51.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:51.844+0000 7f1976000640 1 -- 172.21.15.28:0/1065435620 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f19680be160 con 0x7f1978104280 2024-04-23T14:43:51.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:51 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3935327290' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-23T14:43:51.873 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:43:51 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: ignoring --setuser ceph since I am not root 2024-04-23T14:43:51.873 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:43:51 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: ignoring --setgroup ceph since I am not root 2024-04-23T14:43:51.873 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:43:51 smithi038 ceph-mgr[33746]: -- 172.21.15.38:0/632404848 <== mon.2 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55596992c5a0 con 0x555969900c00 2024-04-23T14:43:51.873 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:43:51 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:43:51.805+0000 7fdc4ee61200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T14:43:51.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:51 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3935327290' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-23T14:43:51.874 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:43:51 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: ignoring --setuser ceph since I am not root 2024-04-23T14:43:51.874 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:43:51 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: ignoring --setgroup ceph since I am not root 2024-04-23T14:43:51.979 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:51 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3935327290' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-23T14:43:51.979 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:43:51 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: ignoring --setuser ceph since I am not root 2024-04-23T14:43:51.979 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:43:51 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: ignoring --setgroup ceph since I am not root 2024-04-23T14:43:51.979 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:43:51 smithi105 ceph-mgr[34429]: -- 172.21.15.105:0/2139795033 <== mon.1 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56073c9525a0 con 0x56073c926c00 2024-04-23T14:43:51.979 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:43:51 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:43:51.793+0000 7f0f56c96200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T14:43:51.979 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:43:51 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:43:51.977+0000 7f0f56c96200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T14:43:52.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.041+0000 7f197cc00640 1 -- 172.21.15.28:0/1065435620 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f195c0a3ca0 msgr2=0x7f195c0a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1997136173 2024-04-23T14:43:52.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.041+0000 7f197cc00640 1 --2- 172.21.15.28:0/1065435620 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f195c0a3ca0 0x7f195c0a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T14:43:52.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.050+0000 7f1980000640 1 -- 172.21.15.28:0/1065435620 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f1978062c00 con 0x7f1978104280 2024-04-23T14:43:52.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.056+0000 7f1976000640 1 -- 172.21.15.28:0/1065435620 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) ==== 138+0+0 (secure 0 0 0) 0x7f19680bda60 con 0x7f1978104280 2024-04-23T14:43:52.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.059+0000 7f1980000640 1 -- 172.21.15.28:0/1065435620 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f195c0a3ca0 msgr2=0x7f195c0a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T14:43:52.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.059+0000 7f1980000640 1 --2- 172.21.15.28:0/1065435620 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f195c0a3ca0 0x7f195c0a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:52.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.059+0000 7f1980000640 1 -- 172.21.15.28:0/1065435620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1978104280 msgr2=0x7f1978079db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:52.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.059+0000 7f1980000640 1 --2- 172.21.15.28:0/1065435620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1978104280 0x7f1978079db0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1968009bf0 tx=0x7f1968031ee0 comp rx=0 tx=0).stop 2024-04-23T14:43:52.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.059+0000 7f1980000640 1 -- 172.21.15.28:0/1065435620 shutdown_connections 2024-04-23T14:43:52.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.059+0000 7f1980000640 1 --2- 172.21.15.28:0/1065435620 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1978105e30 0x7f1978076920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:52.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.059+0000 7f1980000640 1 --2- 172.21.15.28:0/1065435620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1978105470 0x7f197807a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:52.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.059+0000 7f1980000640 1 --2- 172.21.15.28:0/1065435620 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f195c0a3ca0 0x7f195c0a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:52.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.059+0000 7f1980000640 1 --2- 172.21.15.28:0/1065435620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1978104280 0x7f1978079db0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:52.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.059+0000 7f1980000640 1 -- 172.21.15.28:0/1065435620 >> 172.21.15.28:0/1065435620 conn(0x7f19780ff7f0 msgr2=0x7f1978108770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:52.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.059+0000 7f1980000640 1 -- 172.21.15.28:0/1065435620 shutdown_connections 2024-04-23T14:43:52.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.059+0000 7f1980000640 1 -- 172.21.15.28:0/1065435620 wait complete. 2024-04-23T14:43:52.079 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-04-23T14:43:52.186 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:43:51 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:43:51.934+0000 7f0b11938200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T14:43:52.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.264+0000 7fb067200640 1 -- 172.21.15.28:0/4052499266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb068105230 msgr2=0x7fb068107620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.264+0000 7fb067200640 1 --2- 172.21.15.28:0/4052499266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb068105230 0x7fb068107620 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb054009900 tx=0x7fb05402a3f0 comp rx=0 tx=0).stop 2024-04-23T14:43:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.264+0000 7fb067200640 1 -- 172.21.15.28:0/4052499266 shutdown_connections 2024-04-23T14:43:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.264+0000 7fb067200640 1 --2- 172.21.15.28:0/4052499266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb068105230 0x7fb068107620 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.264+0000 7fb067200640 1 --2- 172.21.15.28:0/4052499266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb068107b60 0x7fb068109fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.264+0000 7fb067200640 1 --2- 172.21.15.28:0/4052499266 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb068102900 0x7fb068104cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.264+0000 7fb067200640 1 -- 172.21.15.28:0/4052499266 >> 172.21.15.28:0/4052499266 conn(0x7fb0680fc2f0 msgr2=0x7fb0680fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.264+0000 7fb067200640 1 -- 172.21.15.28:0/4052499266 shutdown_connections 2024-04-23T14:43:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.264+0000 7fb067200640 1 -- 172.21.15.28:0/4052499266 wait complete. 2024-04-23T14:43:52.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.265+0000 7fb067200640 1 Processor -- start 2024-04-23T14:43:52.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.265+0000 7fb067200640 1 -- start start 2024-04-23T14:43:52.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.265+0000 7fb067200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb068102900 0x7fb0681981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:52.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.265+0000 7fb067200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb068105230 0x7fb0681986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:52.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.265+0000 7fb067200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb068107b60 0x7fb06819f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:52.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.265+0000 7fb067200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb06810db40 con 0x7fb068105230 2024-04-23T14:43:52.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.265+0000 7fb067200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb06810d9c0 con 0x7fb068102900 2024-04-23T14:43:52.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.265+0000 7fb067200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb06810dcc0 con 0x7fb068107b60 2024-04-23T14:43:52.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.266+0000 7fb065e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb068102900 0x7fb0681981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:52.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.266+0000 7fb065e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb068102900 0x7fb0681981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:53692/0 (socket says 172.21.15.28:53692) 2024-04-23T14:43:52.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.266+0000 7fb065e00640 1 -- 172.21.15.28:0/648754370 learned_addr learned my addr 172.21.15.28:0/648754370 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:52.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.266+0000 7fb066800640 1 --2- 172.21.15.28:0/648754370 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb068107b60 0x7fb06819f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:52.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.266+0000 7fb065400640 1 --2- 172.21.15.28:0/648754370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb068105230 0x7fb0681986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:52.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.266+0000 7fb065e00640 1 -- 172.21.15.28:0/648754370 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb068107b60 msgr2=0x7fb06819f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:52.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.266+0000 7fb065e00640 1 --2- 172.21.15.28:0/648754370 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb068107b60 0x7fb06819f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:52.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.266+0000 7fb065e00640 1 -- 172.21.15.28:0/648754370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb068105230 msgr2=0x7fb0681986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:52.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.266+0000 7fb065e00640 1 --2- 172.21.15.28:0/648754370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb068105230 0x7fb0681986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:52.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.266+0000 7fb065e00640 1 -- 172.21.15.28:0/648754370 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb06819fdd0 con 0x7fb068102900 2024-04-23T14:43:52.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.266+0000 7fb065e00640 1 --2- 172.21.15.28:0/648754370 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb068102900 0x7fb0681981b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb05800e9f0 tx=0x7fb05800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:52.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.266+0000 7fb05ea00640 1 -- 172.21.15.28:0/648754370 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05800cd80 con 0x7fb068102900 2024-04-23T14:43:52.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.267+0000 7fb05ea00640 1 -- 172.21.15.28:0/648754370 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb058010910 con 0x7fb068102900 2024-04-23T14:43:52.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.267+0000 7fb05ea00640 1 -- 172.21.15.28:0/648754370 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb058018990 con 0x7fb068102900 2024-04-23T14:43:52.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.267+0000 7fb067200640 1 -- 172.21.15.28:0/648754370 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0681a00c0 con 0x7fb068102900 2024-04-23T14:43:52.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.267+0000 7fb067200640 1 -- 172.21.15.28:0/648754370 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0681a0630 con 0x7fb068102900 2024-04-23T14:43:52.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.268+0000 7fb067200640 1 -- 172.21.15.28:0/648754370 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0680fdde0 con 0x7fb068102900 2024-04-23T14:43:52.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.270+0000 7fb05ea00640 1 -- 172.21.15.28:0/648754370 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7fb058010ac0 con 0x7fb068102900 2024-04-23T14:43:52.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.271+0000 7fb05ea00640 1 --2- 172.21.15.28:0/648754370 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb04c0a3c70 0x7fb04c0a6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:52.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.271+0000 7fb065400640 1 -- 172.21.15.28:0/648754370 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb04c0a3c70 msgr2=0x7fb04c0a6130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1997136173 2024-04-23T14:43:52.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.271+0000 7fb05ea00640 1 -- 172.21.15.28:0/648754370 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7fb0580d0710 con 0x7fb068102900 2024-04-23T14:43:52.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.271+0000 7fb065400640 1 --2- 172.21.15.28:0/648754370 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb04c0a3c70 0x7fb04c0a6130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T14:43:52.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.275+0000 7fb05ea00640 1 -- 172.21.15.28:0/648754370 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb05809ca60 con 0x7fb068102900 2024-04-23T14:43:52.286 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:43:52 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:43:52.081+0000 7f0f56c96200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T14:43:52.372 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:43:51 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:43:51.996+0000 7fdc4ee61200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T14:43:52.373 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:43:52 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:43:52.103+0000 7fdc4ee61200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T14:43:52.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.472+0000 7fb065400640 1 -- 172.21.15.28:0/648754370 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb04c0a3c70 msgr2=0x7fb04c0a6130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1997136173 2024-04-23T14:43:52.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.472+0000 7fb065400640 1 --2- 172.21.15.28:0/648754370 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb04c0a3c70 0x7fb04c0a6130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T14:43:52.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.482+0000 7fb067200640 1 -- 172.21.15.28:0/648754370 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7fb06802d620 con 0x7fb068102900 2024-04-23T14:43:52.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.492+0000 7fb05ea00640 1 -- 172.21.15.28:0/648754370 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) ==== 133+0+0 (secure 0 0 0) 0x7fb0580a0d60 con 0x7fb068102900 2024-04-23T14:43:52.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.495+0000 7fb067200640 1 -- 172.21.15.28:0/648754370 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb04c0a3c70 msgr2=0x7fb04c0a6130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T14:43:52.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.495+0000 7fb067200640 1 --2- 172.21.15.28:0/648754370 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb04c0a3c70 0x7fb04c0a6130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:52.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.495+0000 7fb067200640 1 -- 172.21.15.28:0/648754370 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb068102900 msgr2=0x7fb0681981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:52.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.495+0000 7fb067200640 1 --2- 172.21.15.28:0/648754370 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb068102900 0x7fb0681981b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb05800e9f0 tx=0x7fb05800eeb0 comp rx=0 tx=0).stop 2024-04-23T14:43:52.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.495+0000 7fb067200640 1 -- 172.21.15.28:0/648754370 shutdown_connections 2024-04-23T14:43:52.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.495+0000 7fb067200640 1 --2- 172.21.15.28:0/648754370 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb068107b60 0x7fb06819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:52.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.495+0000 7fb067200640 1 --2- 172.21.15.28:0/648754370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb068105230 0x7fb0681986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:52.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.495+0000 7fb067200640 1 --2- 172.21.15.28:0/648754370 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fb04c0a3c70 0x7fb04c0a6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:52.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.495+0000 7fb067200640 1 --2- 172.21.15.28:0/648754370 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb068102900 0x7fb0681981b0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:52.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.495+0000 7fb067200640 1 -- 172.21.15.28:0/648754370 >> 172.21.15.28:0/648754370 conn(0x7fb0680fc2f0 msgr2=0x7fb068103320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:52.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.495+0000 7fb067200640 1 -- 172.21.15.28:0/648754370 shutdown_connections 2024-04-23T14:43:52.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:52.495+0000 7fb067200640 1 -- 172.21.15.28:0/648754370 wait complete. 2024-04-23T14:43:52.512 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:43:52 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:43:52.184+0000 7f0b11938200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T14:43:52.512 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:43:52 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:43:52.323+0000 7f0b11938200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T14:43:52.514 INFO:teuthology.run_tasks:Running task kclient... 2024-04-23T14:43:52.539 INFO:tasks.kclient:Mounting kernel clients... 2024-04-23T14:43:52.539 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-23T14:43:52.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:43:52.540 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-04-23T14:43:52.592 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:43:52.592 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-04-23T14:43:52.668 INFO:teuthology.orchestra.run.smithi028.stderr:Cannot find device "ceph-brx" 2024-04-23T14:43:52.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T14:43:52.671 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'nowsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-04-23T14:43:52.671 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-23T14:43:52.671 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-23T14:43:52.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-23T14:43:52.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:52 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3935327290' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-23T14:43:52.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:52 smithi105 ceph-mon[32555]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-04-23T14:43:52.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:52 smithi105 ceph-mon[32555]: from='client.? ' entity='client.admin' 2024-04-23T14:43:52.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:52 smithi105 ceph-mon[32555]: from='client.? ' entity='client.admin' 2024-04-23T14:43:52.787 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:43:52 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:43:52.391+0000 7f0f56c96200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T14:43:52.787 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:43:52 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:43:52.475+0000 7f0f56c96200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T14:43:52.869 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:43:52 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:43:52.747+0000 7f0b11938200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T14:43:52.870 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:52 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3935327290' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-23T14:43:52.870 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:52 smithi028 ceph-mon[27810]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-04-23T14:43:52.870 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:52 smithi028 ceph-mon[27810]: from='client.? ' entity='client.admin' 2024-04-23T14:43:52.870 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:52 smithi028 ceph-mon[27810]: from='client.? ' entity='client.admin' 2024-04-23T14:43:52.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:52 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3935327290' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-23T14:43:52.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:52 smithi038 ceph-mon[32348]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-04-23T14:43:52.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:52 smithi038 ceph-mon[32348]: from='client.? ' entity='client.admin' 2024-04-23T14:43:52.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:52 smithi038 ceph-mon[32348]: from='client.? ' entity='client.admin' 2024-04-23T14:43:52.873 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:43:52 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:43:52.423+0000 7fdc4ee61200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T14:43:52.873 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:43:52 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:43:52.512+0000 7fdc4ee61200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T14:43:53.113 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:43:53.124 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:43:52 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:43:52.866+0000 7f0b11938200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T14:43:53.996 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:43:53 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:43:53.657+0000 7f0f56c96200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T14:43:53.996 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:43:53 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:43:53.828+0000 7f0f56c96200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T14:43:53.997 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:43:53 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:43:53.995+0000 7f0f56c96200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T14:43:54.108 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:43:53 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:43:53.754+0000 7fdc4ee61200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T14:43:54.109 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:43:53 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:43:53.931+0000 7fdc4ee61200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T14:43:54.109 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:43:54 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:43:54.107+0000 7fdc4ee61200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T14:43:54.536 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:43:54 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:43:54.284+0000 7f0f56c96200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T14:43:54.536 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:43:54 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:43:54.377+0000 7f0f56c96200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T14:43:54.717 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:43:54 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:43:54.407+0000 7fdc4ee61200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T14:43:54.717 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:43:54 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:43:54.503+0000 7fdc4ee61200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T14:43:54.717 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:43:54 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:43:54.714+0000 7fdc4ee61200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T14:43:54.750 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:43:54 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:43:54.504+0000 7f0b11938200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T14:43:55.018 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:43:54 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:43:54.748+0000 7f0b11938200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T14:43:55.018 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:43:54 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:43:54.994+0000 7f0b11938200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T14:43:55.036 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:43:54 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:43:54.578+0000 7f0f56c96200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T14:43:55.036 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:43:54 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:43:54.771+0000 7f0f56c96200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T14:43:55.122 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:43:54 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:43:54.915+0000 7fdc4ee61200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T14:43:55.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.291+0000 7f2fa4e00700 1 -- 172.21.15.28:0/2974226615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fa0108b70 msgr2=0x7f2fa0108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:55.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.291+0000 7f2fa4e00700 1 --2- 172.21.15.28:0/2974226615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fa0108b70 0x7f2fa0108f50 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2f90009990 tx=0x7f2f90009ca0 comp rx=0 tx=0).stop 2024-04-23T14:43:55.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.291+0000 7f2fa4e00700 1 -- 172.21.15.28:0/2974226615 shutdown_connections 2024-04-23T14:43:55.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.291+0000 7f2fa4e00700 1 --2- 172.21.15.28:0/2974226615 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2fa006b1a0 0x7f2fa010f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:55.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.291+0000 7f2fa4e00700 1 --2- 172.21.15.28:0/2974226615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fa0108b70 0x7f2fa0108f50 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:55.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.291+0000 7f2fa4e00700 1 --2- 172.21.15.28:0/2974226615 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2fa006a800 0x7f2fa006ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:55.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.291+0000 7f2fa4e00700 1 -- 172.21.15.28:0/2974226615 >> 172.21.15.28:0/2974226615 conn(0x7f2fa00789a0 msgr2=0x7f2fa0078db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:55.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.292+0000 7f2fa4e00700 1 -- 172.21.15.28:0/2974226615 shutdown_connections 2024-04-23T14:43:55.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.292+0000 7f2fa4e00700 1 -- 172.21.15.28:0/2974226615 wait complete. 2024-04-23T14:43:55.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.293+0000 7f2fa4e00700 1 Processor -- start 2024-04-23T14:43:55.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.293+0000 7f2fa4e00700 1 -- start start 2024-04-23T14:43:55.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.293+0000 7f2fa4e00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2fa006a800 0x7f2fa019b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:55.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.293+0000 7f2fa4e00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2fa006b1a0 0x7f2fa019ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:55.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.293+0000 7f2fa4e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fa0108b70 0x7f2fa019fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:55.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.293+0000 7f2fa4e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa0112e00 con 0x7f2fa0108b70 2024-04-23T14:43:55.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.293+0000 7f2fa4e00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa0112c80 con 0x7f2fa006a800 2024-04-23T14:43:55.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.293+0000 7f2fa4e00700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa0112f80 con 0x7f2fa006b1a0 2024-04-23T14:43:55.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.293+0000 7f2f9de00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2fa006a800 0x7f2fa019b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:55.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.294+0000 7f2f9e800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fa0108b70 0x7f2fa019fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:55.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.294+0000 7f2f9de00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2fa006a800 0x7f2fa019b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:53704/0 (socket says 172.21.15.28:53704) 2024-04-23T14:43:55.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.294+0000 7f2f9de00700 1 -- 172.21.15.28:0/3047023309 learned_addr learned my addr 172.21.15.28:0/3047023309 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:43:55.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.294+0000 7f2f95400700 1 --2- 172.21.15.28:0/3047023309 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2fa006b1a0 0x7f2fa019ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:43:55.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.294+0000 7f2f9de00700 1 -- 172.21.15.28:0/3047023309 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2fa006b1a0 msgr2=0x7f2fa019ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:55.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.294+0000 7f2f9de00700 1 --2- 172.21.15.28:0/3047023309 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2fa006b1a0 0x7f2fa019ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:55.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.294+0000 7f2f9de00700 1 -- 172.21.15.28:0/3047023309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fa0108b70 msgr2=0x7f2fa019fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:55.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.294+0000 7f2f9de00700 1 --2- 172.21.15.28:0/3047023309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fa0108b70 0x7f2fa019fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:55.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.294+0000 7f2f9de00700 1 -- 172.21.15.28:0/3047023309 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fa01a04d0 con 0x7f2fa006a800 2024-04-23T14:43:55.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.294+0000 7f2f9de00700 1 --2- 172.21.15.28:0/3047023309 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2fa006a800 0x7f2fa019b4c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2f90005910 tx=0x7f2f9000bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:43:55.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.295+0000 7f2f97400700 1 -- 172.21.15.28:0/3047023309 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f90027070 con 0x7f2fa006a800 2024-04-23T14:43:55.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.295+0000 7f2f97400700 1 -- 172.21.15.28:0/3047023309 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f90004650 con 0x7f2fa006a800 2024-04-23T14:43:55.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.295+0000 7f2f97400700 1 -- 172.21.15.28:0/3047023309 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f9000f460 con 0x7f2fa006a800 2024-04-23T14:43:55.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.295+0000 7f2fa4e00700 1 -- 172.21.15.28:0/3047023309 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fa01a0760 con 0x7f2fa006a800 2024-04-23T14:43:55.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.295+0000 7f2fa4e00700 1 -- 172.21.15.28:0/3047023309 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fa01a8040 con 0x7f2fa006a800 2024-04-23T14:43:55.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.296+0000 7f2fa4e00700 1 -- 172.21.15.28:0/3047023309 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fa01041e0 con 0x7f2fa006a800 2024-04-23T14:43:55.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.298+0000 7f2f97400700 1 -- 172.21.15.28:0/3047023309 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f2f9002a070 con 0x7f2fa006a800 2024-04-23T14:43:55.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.299+0000 7f2f97400700 1 --2- 172.21.15.28:0/3047023309 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2f840a3d10 0x7f2f840a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:43:55.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.299+0000 7f2f95400700 1 -- 172.21.15.28:0/3047023309 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2f840a3d10 msgr2=0x7f2f840a61d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1997136173 2024-04-23T14:43:55.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.299+0000 7f2f97400700 1 -- 172.21.15.28:0/3047023309 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7f2f900d12c0 con 0x7f2fa006a800 2024-04-23T14:43:55.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.299+0000 7f2f95400700 1 --2- 172.21.15.28:0/3047023309 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2f840a3d10 0x7f2f840a61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T14:43:55.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.303+0000 7f2f97400700 1 -- 172.21.15.28:0/3047023309 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2f9005eff0 con 0x7f2fa006a800 2024-04-23T14:43:55.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.500+0000 7f2f95400700 1 -- 172.21.15.28:0/3047023309 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2f840a3d10 msgr2=0x7f2f840a61d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1997136173 2024-04-23T14:43:55.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.500+0000 7f2f95400700 1 --2- 172.21.15.28:0/3047023309 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2f840a3d10 0x7f2f840a61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T14:43:55.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.582+0000 7f2fa4e00700 1 -- 172.21.15.28:0/3047023309 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f2fa002dfe0 con 0x7f2fa006a800 2024-04-23T14:43:55.584 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:43:55 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:43:55.410+0000 7f0b11938200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T14:43:55.585 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:43:55 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:43:55.538+0000 7f0b11938200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T14:43:55.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.583+0000 7f2f97400700 1 -- 172.21.15.28:0/3047023309 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v20) ==== 53+0+73 (secure 0 0 0) 0x7f2f90013070 con 0x7f2fa006a800 2024-04-23T14:43:55.585 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-23T14:43:55.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.586+0000 7f2fa4e00700 1 -- 172.21.15.28:0/3047023309 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2f840a3d10 msgr2=0x7f2f840a61d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T14:43:55.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.586+0000 7f2fa4e00700 1 --2- 172.21.15.28:0/3047023309 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2f840a3d10 0x7f2f840a61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:55.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.586+0000 7f2fa4e00700 1 -- 172.21.15.28:0/3047023309 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2fa006a800 msgr2=0x7f2fa019b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:43:55.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.586+0000 7f2fa4e00700 1 --2- 172.21.15.28:0/3047023309 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2fa006a800 0x7f2fa019b4c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2f90005910 tx=0x7f2f9000bab0 comp rx=0 tx=0).stop 2024-04-23T14:43:55.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.586+0000 7f2fa4e00700 1 -- 172.21.15.28:0/3047023309 shutdown_connections 2024-04-23T14:43:55.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.586+0000 7f2fa4e00700 1 --2- 172.21.15.28:0/3047023309 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7f2f840a3d10 0x7f2f840a61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:55.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.586+0000 7f2fa4e00700 1 --2- 172.21.15.28:0/3047023309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fa0108b70 0x7f2fa019fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:55.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.586+0000 7f2fa4e00700 1 --2- 172.21.15.28:0/3047023309 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2fa006a800 0x7f2fa019b4c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:55.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.586+0000 7f2fa4e00700 1 --2- 172.21.15.28:0/3047023309 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2fa006b1a0 0x7f2fa019ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:43:55.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.586+0000 7f2fa4e00700 1 -- 172.21.15.28:0/3047023309 >> 172.21.15.28:0/3047023309 conn(0x7f2fa00789a0 msgr2=0x7f2fa010df00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:43:55.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.586+0000 7f2fa4e00700 1 -- 172.21.15.28:0/3047023309 shutdown_connections 2024-04-23T14:43:55.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:55.587+0000 7f2fa4e00700 1 -- 172.21.15.28:0/3047023309 wait complete. 2024-04-23T14:43:55.636 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:43:55 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:43:55.367+0000 7f0f56c96200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T14:43:55.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:55 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3047023309' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-23T14:43:55.873 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:43:55 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:43:55.543+0000 7fdc4ee61200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T14:43:55.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:55 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3047023309' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-23T14:43:55.874 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:43:55 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:43:55.803+0000 7f0b11938200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T14:43:55.887 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:55 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3047023309' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-23T14:43:55.888 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:43:55 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:43:55.789+0000 7f0f56c96200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T14:43:56.286 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:43:55 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:43:55.886+0000 7f0f56c96200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T14:43:56.286 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:43:56 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:43:56.091+0000 7f0f56c96200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T14:43:56.294 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:43:55 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:43:55.982+0000 7fdc4ee61200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T14:43:56.295 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:43:56 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:43:56.084+0000 7fdc4ee61200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T14:43:56.295 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:43:56 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:43:56.293+0000 7fdc4ee61200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T14:43:56.331 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:43:56 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:43:56.075+0000 7f0b11938200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T14:43:56.385 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-23T14:43:56.386 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-23T14:43:56.386 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-04-23T14:43:56.386 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-23T14:43:56.386 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-23T14:43:56.386 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-23T14:43:56.386 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-23T14:43:56.386 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-23T14:43:56.386 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-23T14:43:56.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:43:56.386 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-04-23T14:43:56.439 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-23T14:43:56.440 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-23T14:43:56.440 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-04-23T14:43:56.440 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-23T14:43:56.440 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-04-23T14:43:56.440 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-23T14:43:56.440 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-23T14:43:56.440 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-04-23T14:43:56.440 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-04-23T14:43:56.440 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-23T14:43:56.440 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-04-23T14:43:56.440 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-04-23T14:43:56.440 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-23T14:43:56.440 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-04-23T14:43:56.441 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-04-23T14:43:56.441 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-23T14:43:56.441 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 41959sec preferred_lft 41959sec 2024-04-23T14:43:56.441 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link noprefixroute 2024-04-23T14:43:56.441 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-23T14:43:56.441 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-23T14:43:56.441 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-04-23T14:43:56.441 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-04-23T14:43:56.441 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-23T14:43:56.442 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-23T14:43:56.442 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-23T14:43:56.442 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add name ceph-brx type bridge 2024-04-23T14:43:56.442 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr flush dev ceph-brx 2024-04-23T14:43:56.442 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set ceph-brx up 2024-04-23T14:43:56.442 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-23T14:43:56.442 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-23T14:43:56.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T14:43:56.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T14:43:56.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:43:56.910 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-23T14:43:56.937 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:43:56 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:43:56.605+0000 7f0f56c96200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T14:43:56.937 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:43:56 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:43:56.697+0000 7f0f56c96200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T14:43:56.976 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-04-23T14:43:56.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:43:56.980 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-04-23T14:43:57.030 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-04-23T14:43:57.030 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-23T14:43:57.030 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-23T14:43:57.030 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-23T14:43:57.030 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-23T14:43:57.031 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-23T14:43:57.031 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-23T14:43:57.031 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-23T14:43:57.031 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-23T14:43:57.031 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-23T14:43:57.031 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-23T14:43:57.122 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:43:56 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:43:56.840+0000 7fdc4ee61200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T14:43:57.122 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:43:56 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:43:56.936+0000 7fdc4ee61200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T14:43:57.124 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:43:56 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:43:56.930+0000 7f0b11938200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T14:43:57.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T14:43:57.272 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:43:56 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:43:56.936+0000 7f0f56c96200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T14:43:57.273 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:43:57 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:43:57.066+0000 7f0f56c96200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T14:43:57.273 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:43:57 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:43:57.159+0000 7f0f56c96200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T14:43:57.273 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:43:57 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:43:57.271+0000 7f0f56c96200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T14:43:57.551 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:43:57 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:43:57.191+0000 7fdc4ee61200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T14:43:57.551 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:43:57 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:43:57.330+0000 7fdc4ee61200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T14:43:57.551 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:43:57 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:43:57.430+0000 7fdc4ee61200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T14:43:57.551 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:43:57 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:43:57.550+0000 7fdc4ee61200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T14:43:57.763 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:43:57 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:43:57.525+0000 7f0b11938200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T14:43:57.764 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:43:57 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:43:57.660+0000 7f0b11938200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T14:43:57.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T14:43:57.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:43:57.776 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-04-23T14:43:57.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:43:57.823 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list-id 2024-04-23T14:43:57.904 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-23T14:43:57.976 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-23T14:43:57.977 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-23T14:43:57.977 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-23T14:43:57.977 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-23T14:43:58.036 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:43:57 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:43:57.570+0000 7f0f56c96200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T14:43:58.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:58 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T14:43:58.108 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:43:57 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:43:57.947+0000 7f0b11938200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T14:43:58.122 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:43:57 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:43:57.863+0000 7fdc4ee61200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T14:43:58.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T14:43:58.163 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-23T14:43:58.163 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-23T14:43:58.163 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-23T14:43:58.163 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-23T14:43:58.163 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-23T14:43:58.163 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-23T14:43:58.163 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-23T14:43:58.163 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-23T14:43:58.164 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-23T14:43:58.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:58 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T14:43:58.445 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:43:58 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:43:58.162+0000 7f0f56c96200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T14:43:58.446 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:43:58 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z[34425]: 2024-04-23T14:43:58.374+0000 7f0f56c96200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T14:43:58.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T14:43:58.482 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-23T14:43:58.482 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-23T14:43:58.482 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-04-23T14:43:58.482 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set dev brx.0 master ceph-brx 2024-04-23T14:43:58.482 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-23T14:43:58.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:58 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T14:43:58.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:58 smithi028 ceph-mon[27810]: Standby manager daemon z restarted 2024-04-23T14:43:58.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:58 smithi028 ceph-mon[27810]: Standby manager daemon z started 2024-04-23T14:43:58.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:58 smithi028 ceph-mon[27810]: from='mgr.? 172.21.15.105:0/1638148313' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-23T14:43:58.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:58 smithi028 ceph-mon[27810]: from='mgr.? 172.21.15.105:0/1638148313' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T14:43:58.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:58 smithi028 ceph-mon[27810]: from='mgr.? 172.21.15.105:0/1638148313' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-23T14:43:58.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:58 smithi028 ceph-mon[27810]: from='mgr.? 172.21.15.105:0/1638148313' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T14:43:58.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T14:43:58.679 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-23T14:43:58.679 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-23T14:43:58.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:58 smithi038 ceph-mon[32348]: Standby manager daemon z restarted 2024-04-23T14:43:58.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:58 smithi038 ceph-mon[32348]: Standby manager daemon z started 2024-04-23T14:43:58.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:58 smithi038 ceph-mon[32348]: from='mgr.? 172.21.15.105:0/1638148313' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-23T14:43:58.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:58 smithi038 ceph-mon[32348]: from='mgr.? 172.21.15.105:0/1638148313' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T14:43:58.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:58 smithi038 ceph-mon[32348]: from='mgr.? 172.21.15.105:0/1638148313' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-23T14:43:58.719 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:58 smithi038 ceph-mon[32348]: from='mgr.? 172.21.15.105:0/1638148313' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T14:43:58.720 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:43:58 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:43:58.494+0000 7fdc4ee61200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T14:43:58.720 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:43:58 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y[33742]: 2024-04-23T14:43:58.718+0000 7fdc4ee61200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T14:43:58.720 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-23T14:43:58.720 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-23T14:43:58.720 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-23T14:43:58.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:58 smithi105 ceph-mon[32555]: Standby manager daemon z restarted 2024-04-23T14:43:58.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:58 smithi105 ceph-mon[32555]: Standby manager daemon z started 2024-04-23T14:43:58.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:58 smithi105 ceph-mon[32555]: from='mgr.? 172.21.15.105:0/1638148313' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-23T14:43:58.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:58 smithi105 ceph-mon[32555]: from='mgr.? 172.21.15.105:0/1638148313' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T14:43:58.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:58 smithi105 ceph-mon[32555]: from='mgr.? 172.21.15.105:0/1638148313' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-23T14:43:58.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:58 smithi105 ceph-mon[32555]: from='mgr.? 172.21.15.105:0/1638148313' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T14:43:58.811 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-23T14:43:58.811 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-23T14:43:58.811 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/fe56c074-80ef-44de-886c-d211b0a30b1b 2024-04-23T14:43:58.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:43:58.811 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/fe56c074-80ef-44de-886c-d211b0a30b1b /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2024-04-23T14:43:58.916 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:43:58 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:43:58.676+0000 7f0b11938200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T14:43:58.917 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:43:58 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:43:58.818+0000 7f0b11938200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T14:43:59.017 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2024-04-23T14:43:59.017 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync". 2024-04-23T14:43:59.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:59 smithi028 ceph-mon[27810]: mgrmap e20: x(active, since 10m), standbys: y, z 2024-04-23T14:43:59.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:59 smithi028 ceph-mon[27810]: Standby manager daemon y restarted 2024-04-23T14:43:59.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:59 smithi028 ceph-mon[27810]: Standby manager daemon y started 2024-04-23T14:43:59.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:59 smithi028 ceph-mon[27810]: from='mgr.? 172.21.15.38:0/3873017320' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T14:43:59.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:59 smithi028 ceph-mon[27810]: from='mgr.? 172.21.15.38:0/3873017320' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T14:43:59.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:59 smithi028 ceph-mon[27810]: from='mgr.? 172.21.15.38:0/3873017320' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T14:43:59.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:43:59 smithi028 ceph-mon[27810]: from='mgr.? 172.21.15.38:0/3873017320' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T14:43:59.465 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:43:59 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:43:59.196+0000 7f0b11938200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T14:43:59.465 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:43:59 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:43:59.405+0000 7f0b11938200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T14:43:59.733 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:43:59 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:43:59.554+0000 7f0b11938200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T14:43:59.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:59 smithi105 ceph-mon[32555]: mgrmap e20: x(active, since 10m), standbys: y, z 2024-04-23T14:43:59.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:59 smithi105 ceph-mon[32555]: Standby manager daemon y restarted 2024-04-23T14:43:59.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:59 smithi105 ceph-mon[32555]: Standby manager daemon y started 2024-04-23T14:43:59.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:59 smithi105 ceph-mon[32555]: from='mgr.? 172.21.15.38:0/3873017320' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T14:43:59.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:59 smithi105 ceph-mon[32555]: from='mgr.? 172.21.15.38:0/3873017320' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T14:43:59.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:59 smithi105 ceph-mon[32555]: from='mgr.? 172.21.15.38:0/3873017320' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T14:43:59.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:43:59 smithi105 ceph-mon[32555]: from='mgr.? 172.21.15.38:0/3873017320' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T14:43:59.790 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2024-04-23T14:43:59.790 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync". 2024-04-23T14:43:59.790 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.38:3300,172.21.15.105:3300" 2024-04-23T14:43:59.790 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with new device syntax: 0@1ee6e996-017e-11ef-bc93-c7b262605968.cephfs=/volumes/qa/sv_1/fe56c074-80ef-44de-886c-d211b0a30b1b 2024-04-23T14:43:59.790 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync,name=0,key=0,mon_addr=172.21.15.28:3300/172.21.15.38:3300/172.21.15.105:3300" will pass to kernel 2024-04-23T14:43:59.794 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-23T14:43:59.794 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-04-23T14:43:59.795 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-23T14:43:59.795 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-04-23T14:43:59.795 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-23T14:43:59.801 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-23T14:43:59.801 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:43:59.801 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-23T14:43:59.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:59 smithi038 ceph-mon[32348]: mgrmap e20: x(active, since 10m), standbys: y, z 2024-04-23T14:43:59.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:59 smithi038 ceph-mon[32348]: Standby manager daemon y restarted 2024-04-23T14:43:59.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:59 smithi038 ceph-mon[32348]: Standby manager daemon y started 2024-04-23T14:43:59.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:59 smithi038 ceph-mon[32348]: from='mgr.? 172.21.15.38:0/3873017320' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T14:43:59.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:59 smithi038 ceph-mon[32348]: from='mgr.? 172.21.15.38:0/3873017320' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T14:43:59.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:59 smithi038 ceph-mon[32348]: from='mgr.? 172.21.15.38:0/3873017320' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T14:43:59.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:43:59 smithi038 ceph-mon[32348]: from='mgr.? 172.21.15.38:0/3873017320' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T14:43:59.875 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-23T14:43:59.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:59 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-23T14:43:59.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:43:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T14:43:59.979 INFO:teuthology.orchestra.run.smithi028.stdout:client25163 2024-04-23T14:43:59.979 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-23T14:43:59.979 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-23T14:44:00.024 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-23T14:44:00.124 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:43:59 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:43:59.731+0000 7f0b11938200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T14:44:00.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-23T14:44:00.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T14:44:00.184 INFO:teuthology.orchestra.run.smithi028.stdout:client25163 2024-04-23T14:44:00.185 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-23T14:44:00.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-23T14:44:00.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T14:44:00.290 INFO:teuthology.orchestra.run.smithi028.stdout:client25163 2024-04-23T14:44:00.290 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/1ee6e996-017e-11ef-bc93-c7b262605968.client25163/status') 2024-04-23T14:44:00.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T14:44:00.414 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:44:00 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:44:00.140+0000 7f0b11938200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T14:44:00.421 INFO:teuthology.orchestra.run.smithi028.stdout:instance: client.25163 (3)172.21.15.28:0/252524176 2024-04-23T14:44:00.421 INFO:teuthology.orchestra.run.smithi028.stdout:blocklisted: false 2024-04-23T14:44:00.422 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records in 2024-04-23T14:44:00.422 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records out 2024-04-23T14:44:00.422 INFO:teuthology.orchestra.run.smithi028.stderr:70 bytes copied, 7.2519e-05 s, 965 kB/s 2024-04-23T14:44:00.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T14:44:00.435 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-23T14:44:00.447 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-04-23T14:44:00.457 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:44:00.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.639+0000 7fd255a00640 1 -- 172.21.15.28:0/201583396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd250102ad0 msgr2=0x7fd250104ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:00.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.639+0000 7fd255a00640 1 --2- 172.21.15.28:0/201583396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd250102ad0 0x7fd250104ec0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd240009ca0 tx=0x7fd24002a8c0 comp rx=0 tx=0).stop 2024-04-23T14:44:00.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.639+0000 7fd255a00640 1 -- 172.21.15.28:0/201583396 shutdown_connections 2024-04-23T14:44:00.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.639+0000 7fd255a00640 1 --2- 172.21.15.28:0/201583396 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd250107d30 0x7fd25010a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:00.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.639+0000 7fd255a00640 1 --2- 172.21.15.28:0/201583396 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd250105400 0x7fd2501077f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:00.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.639+0000 7fd255a00640 1 --2- 172.21.15.28:0/201583396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd250102ad0 0x7fd250104ec0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:00.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.639+0000 7fd255a00640 1 -- 172.21.15.28:0/201583396 >> 172.21.15.28:0/201583396 conn(0x7fd2500fc4e0 msgr2=0x7fd2500fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:00.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.639+0000 7fd255a00640 1 -- 172.21.15.28:0/201583396 shutdown_connections 2024-04-23T14:44:00.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.639+0000 7fd255a00640 1 -- 172.21.15.28:0/201583396 wait complete. 2024-04-23T14:44:00.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.641+0000 7fd255a00640 1 Processor -- start 2024-04-23T14:44:00.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.641+0000 7fd255a00640 1 -- start start 2024-04-23T14:44:00.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.641+0000 7fd255a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd250102ad0 0x7fd25019a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:00.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.641+0000 7fd255a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd250105400 0x7fd25019ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:00.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.641+0000 7fd255a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd250107d30 0x7fd2501a1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:00.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.641+0000 7fd255a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd25010dd30 con 0x7fd250105400 2024-04-23T14:44:00.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.641+0000 7fd255a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd25010dbb0 con 0x7fd250107d30 2024-04-23T14:44:00.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.641+0000 7fd255a00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd25010deb0 con 0x7fd250102ad0 2024-04-23T14:44:00.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.642+0000 7fd24ea00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd250102ad0 0x7fd25019a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:00.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.642+0000 7fd246000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd250105400 0x7fd25019ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:00.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.642+0000 7fd24ea00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd250102ad0 0x7fd25019a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:43962/0 (socket says 172.21.15.28:43962) 2024-04-23T14:44:00.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.642+0000 7fd24ea00640 1 -- 172.21.15.28:0/800019970 learned_addr learned my addr 172.21.15.28:0/800019970 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:00.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.642+0000 7fd24f400640 1 --2- 172.21.15.28:0/800019970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd250107d30 0x7fd2501a1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:00.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.642+0000 7fd24ea00640 1 -- 172.21.15.28:0/800019970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd250107d30 msgr2=0x7fd2501a1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:00.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.642+0000 7fd24ea00640 1 --2- 172.21.15.28:0/800019970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd250107d30 0x7fd2501a1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:00.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.642+0000 7fd24ea00640 1 -- 172.21.15.28:0/800019970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd250105400 msgr2=0x7fd25019ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:00.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.642+0000 7fd24ea00640 1 --2- 172.21.15.28:0/800019970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd250105400 0x7fd25019ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:00.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.642+0000 7fd24ea00640 1 -- 172.21.15.28:0/800019970 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2501a2220 con 0x7fd250102ad0 2024-04-23T14:44:00.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.642+0000 7fd24ea00640 1 --2- 172.21.15.28:0/800019970 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd250102ad0 0x7fd25019a600 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd240000f80 tx=0x7fd2400319f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:00.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.643+0000 7fd247e00640 1 -- 172.21.15.28:0/800019970 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd240003b70 con 0x7fd250102ad0 2024-04-23T14:44:00.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.643+0000 7fd247e00640 1 -- 172.21.15.28:0/800019970 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd240003d10 con 0x7fd250102ad0 2024-04-23T14:44:00.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.643+0000 7fd255a00640 1 -- 172.21.15.28:0/800019970 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2501a24b0 con 0x7fd250102ad0 2024-04-23T14:44:00.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.643+0000 7fd247e00640 1 -- 172.21.15.28:0/800019970 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd24002fb50 con 0x7fd250102ad0 2024-04-23T14:44:00.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.643+0000 7fd255a00640 1 -- 172.21.15.28:0/800019970 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2501a2a00 con 0x7fd250102ad0 2024-04-23T14:44:00.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.644+0000 7fd255a00640 1 -- 172.21.15.28:0/800019970 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd250109880 con 0x7fd250102ad0 2024-04-23T14:44:00.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.646+0000 7fd247e00640 1 -- 172.21.15.28:0/800019970 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 21) ==== 138663+0+0 (secure 0 0 0) 0x7fd24002fcf0 con 0x7fd250102ad0 2024-04-23T14:44:00.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.647+0000 7fd247e00640 1 --2- 172.21.15.28:0/800019970 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd2340a3c00 0x7fd2340a60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:00.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.647+0000 7fd246000640 1 -- 172.21.15.28:0/800019970 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd2340a3c00 msgr2=0x7fd2340a60c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1997136173 2024-04-23T14:44:00.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.647+0000 7fd247e00640 1 -- 172.21.15.28:0/800019970 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7fd2400f17e0 con 0x7fd250102ad0 2024-04-23T14:44:00.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.647+0000 7fd246000640 1 --2- 172.21.15.28:0/800019970 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd2340a3c00 0x7fd2340a60c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T14:44:00.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.651+0000 7fd247e00640 1 -- 172.21.15.28:0/800019970 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd2400be180 con 0x7fd250102ad0 2024-04-23T14:44:00.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:00 smithi105 ceph-mon[32555]: mgrmap e21: x(active, since 10m), standbys: z, y 2024-04-23T14:44:00.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.848+0000 7fd246000640 1 -- 172.21.15.28:0/800019970 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd2340a3c00 msgr2=0x7fd2340a60c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1997136173 2024-04-23T14:44:00.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.848+0000 7fd246000640 1 --2- 172.21.15.28:0/800019970 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd2340a3c00 0x7fd2340a60c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T14:44:00.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.853+0000 7fd255a00640 1 -- 172.21.15.28:0/800019970 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd250062c00 con 0x7fd250102ad0 2024-04-23T14:44:00.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.855+0000 7fd247e00640 1 -- 172.21.15.28:0/800019970 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22651 (secure 0 0 0) 0x7fd2400de0c0 con 0x7fd250102ad0 2024-04-23T14:44:00.856 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:44:00.856 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":87,"fsid":"1ee6e996-017e-11ef-bc93-c7b262605968","created":"2024-04-23T14:32:34.581082+0000","modified":"2024-04-23T14:42:43.429141+0000","last_up_change":"2024-04-23T14:40:43.375846+0000","last_in_change":"2024-04-23T14:40:23.576939+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:36:27.698112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:41:37.030341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:41:38.444742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e0b427bc-be65-4543-99cc-2f53ad98ddad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6803","nonce":2993445336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6805","nonce":2993445336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6809","nonce":2993445336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6807","nonce":2993445336}]},"public_addr":"172.21.15.28:6803/2993445336","cluster_addr":"172.21.15.28:6805/2993445336","heartbeat_back_addr":"172.21.15.28:6809/2993445336","heartbeat_front_addr":"172.21.15.28:6807/2993445336","state":["exists","up"]},{"osd":1,"uuid":"ef527ec9-1bf3-4bb0-8df3-f5a015af557c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6801","nonce":4024093383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6803","nonce":4024093383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6807","nonce":4024093383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6805","nonce":4024093383}]},"public_addr":"172.21.15.38:6801/4024093383","cluster_addr":"172.21.15.38:6803/4024093383","heartbeat_back_addr":"172.21.15.38:6807/4024093383","heartbeat_front_addr":"172.21.15.38:6805/4024093383","state":["exists","up"]},{"osd":2,"uuid":"b33b23d9-adce-4c11-b304-2e117bdf6bfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6801","nonce":570739029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6803","nonce":570739029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6807","nonce":570739029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6805","nonce":570739029}]},"public_addr":"172.21.15.105:6801/570739029","cluster_addr":"172.21.15.105:6803/570739029","heartbeat_back_addr":"172.21.15.105:6807/570739029","heartbeat_front_addr":"172.21.15.105:6805/570739029","state":["exists","up"]},{"osd":3,"uuid":"28bd5608-b426-434e-9db1-bdaa9828bdb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6811","nonce":2480400247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6813","nonce":2480400247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6817","nonce":2480400247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6815","nonce":2480400247}]},"public_addr":"172.21.15.28:6811/2480400247","cluster_addr":"172.21.15.28:6813/2480400247","heartbeat_back_addr":"172.21.15.28:6817/2480400247","heartbeat_front_addr":"172.21.15.28:6815/2480400247","state":["exists","up"]},{"osd":4,"uuid":"e5d1c89b-025f-4d57-8807-59d75c787b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6809","nonce":3772189246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6811","nonce":3772189246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6815","nonce":3772189246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6813","nonce":3772189246}]},"public_addr":"172.21.15.38:6809/3772189246","cluster_addr":"172.21.15.38:6811/3772189246","heartbeat_back_addr":"172.21.15.38:6815/3772189246","heartbeat_front_addr":"172.21.15.38:6813/3772189246","state":["exists","up"]},{"osd":5,"uuid":"9cf9cf4c-aafd-4ed3-bfb1-0527710a0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6809","nonce":928755894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6811","nonce":928755894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6815","nonce":928755894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6813","nonce":928755894}]},"public_addr":"172.21.15.105:6809/928755894","cluster_addr":"172.21.15.105:6811/928755894","heartbeat_back_addr":"172.21.15.105:6815/928755894","heartbeat_front_addr":"172.21.15.105:6813/928755894","state":["exists","up"]},{"osd":6,"uuid":"304db525-d24d-4687-bfa0-f92bdf66d1e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6819","nonce":897979441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6821","nonce":897979441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6825","nonce":897979441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6823","nonce":897979441}]},"public_addr":"172.21.15.28:6819/897979441","cluster_addr":"172.21.15.28:6821/897979441","heartbeat_back_addr":"172.21.15.28:6825/897979441","heartbeat_front_addr":"172.21.15.28:6823/897979441","state":["exists","up"]},{"osd":7,"uuid":"08e01d59-03f3-47b1-b45f-35ec5a50992c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6817","nonce":2400251786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6819","nonce":2400251786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6823","nonce":2400251786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6821","nonce":2400251786}]},"public_addr":"172.21.15.38:6817/2400251786","cluster_addr":"172.21.15.38:6819/2400251786","heartbeat_back_addr":"172.21.15.38:6823/2400251786","heartbeat_front_addr":"172.21.15.38:6821/2400251786","state":["exists","up"]},{"osd":8,"uuid":"64ee1e95-0756-48b1-9c00-472d0c3cbd29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6817","nonce":1682312269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6819","nonce":1682312269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6823","nonce":1682312269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6821","nonce":1682312269}]},"public_addr":"172.21.15.105:6817/1682312269","cluster_addr":"172.21.15.105:6819/1682312269","heartbeat_back_addr":"172.21.15.105:6823/1682312269","heartbeat_front_addr":"172.21.15.105:6821/1682312269","state":["exists","up"]},{"osd":9,"uuid":"18f60893-3ffd-48c2-9db5-a67d35611d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6827","nonce":1292942516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6829","nonce":1292942516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6833","nonce":1292942516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6831","nonce":1292942516}]},"public_addr":"172.21.15.28:6827/1292942516","cluster_addr":"172.21.15.28:6829/1292942516","heartbeat_back_addr":"172.21.15.28:6833/1292942516","heartbeat_front_addr":"172.21.15.28:6831/1292942516","state":["exists","up"]},{"osd":10,"uuid":"76c06725-7040-4118-8894-fe808a87f999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6825","nonce":2444616468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6827","nonce":2444616468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6831","nonce":2444616468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6829","nonce":2444616468}]},"public_addr":"172.21.15.38:6825/2444616468","cluster_addr":"172.21.15.38:6827/2444616468","heartbeat_back_addr":"172.21.15.38:6831/2444616468","heartbeat_front_addr":"172.21.15.38:6829/2444616468","state":["exists","up"]},{"osd":11,"uuid":"e8fc9454-5540-4e2e-91bf-3ba251dbd1a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6825","nonce":3400299562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6827","nonce":3400299562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6831","nonce":3400299562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6829","nonce":3400299562}]},"public_addr":"172.21.15.105:6825/3400299562","cluster_addr":"172.21.15.105:6827/3400299562","heartbeat_back_addr":"172.21.15.105:6831/3400299562","heartbeat_front_addr":"172.21.15.105:6829/3400299562","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:35:33.615395+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:00.545293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:24.940211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:57.684887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:24.519061+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:49.010804+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:22.638478+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:49.559104+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:14.011771+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:47.859950+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:16.298445+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:41.619069+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.2d","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/590495740":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/1163172512":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/4290328460":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/4145020172":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/720626958":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/948450262":"2024-04-24T14:33:07.576363+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:44:00.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.858+0000 7fd255a00640 1 -- 172.21.15.28:0/800019970 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd2340a3c00 msgr2=0x7fd2340a60c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T14:44:00.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.858+0000 7fd255a00640 1 --2- 172.21.15.28:0/800019970 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd2340a3c00 0x7fd2340a60c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:00.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.858+0000 7fd255a00640 1 -- 172.21.15.28:0/800019970 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd250102ad0 msgr2=0x7fd25019a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:00.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.858+0000 7fd255a00640 1 --2- 172.21.15.28:0/800019970 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd250102ad0 0x7fd25019a600 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd240000f80 tx=0x7fd2400319f0 comp rx=0 tx=0).stop 2024-04-23T14:44:00.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.858+0000 7fd255a00640 1 -- 172.21.15.28:0/800019970 shutdown_connections 2024-04-23T14:44:00.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.858+0000 7fd255a00640 1 --2- 172.21.15.28:0/800019970 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fd2340a3c00 0x7fd2340a60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:00.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.858+0000 7fd255a00640 1 --2- 172.21.15.28:0/800019970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd250107d30 0x7fd2501a1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:00.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.858+0000 7fd255a00640 1 --2- 172.21.15.28:0/800019970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd250105400 0x7fd25019ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:00.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.858+0000 7fd255a00640 1 --2- 172.21.15.28:0/800019970 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd250102ad0 0x7fd25019a600 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:00.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.858+0000 7fd255a00640 1 -- 172.21.15.28:0/800019970 >> 172.21.15.28:0/800019970 conn(0x7fd2500fc4e0 msgr2=0x7fd2501021e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:00.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.858+0000 7fd255a00640 1 -- 172.21.15.28:0/800019970 shutdown_connections 2024-04-23T14:44:00.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:00.858+0000 7fd255a00640 1 -- 172.21.15.28:0/800019970 wait complete. 2024-04-23T14:44:00.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:00 smithi038 ceph-mon[32348]: mgrmap e21: x(active, since 10m), standbys: z, y 2024-04-23T14:44:00.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:00 smithi028 ceph-mon[27810]: mgrmap e21: x(active, since 10m), standbys: z, y 2024-04-23T14:44:00.877 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T14:36:27.698112+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T14:41:37.030341+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T14:41:38.444742+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T14:44:00.877 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T14:44:01.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.084+0000 7fcd2ac00640 1 -- 172.21.15.28:0/251202902 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd24103c00 msgr2=0x7fcd2410a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:01.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.084+0000 7fcd2ac00640 1 --2- 172.21.15.28:0/251202902 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd24103c00 0x7fcd2410a490 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c009860 tx=0x7fcd1c02a340 comp rx=0 tx=0).stop 2024-04-23T14:44:01.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.084+0000 7fcd2ac00640 1 -- 172.21.15.28:0/251202902 shutdown_connections 2024-04-23T14:44:01.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.084+0000 7fcd2ac00640 1 --2- 172.21.15.28:0/251202902 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd24103c00 0x7fcd2410a490 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:01.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.084+0000 7fcd2ac00640 1 --2- 172.21.15.28:0/251202902 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd24103240 0x7fcd241036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:01.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.084+0000 7fcd2ac00640 1 --2- 172.21.15.28:0/251202902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd24102050 0x7fcd24102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:01.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.084+0000 7fcd2ac00640 1 -- 172.21.15.28:0/251202902 >> 172.21.15.28:0/251202902 conn(0x7fcd240fd5e0 msgr2=0x7fcd240ffa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:01.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.084+0000 7fcd2ac00640 1 -- 172.21.15.28:0/251202902 shutdown_connections 2024-04-23T14:44:01.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.084+0000 7fcd2ac00640 1 -- 172.21.15.28:0/251202902 wait complete. 2024-04-23T14:44:01.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.085+0000 7fcd2ac00640 1 Processor -- start 2024-04-23T14:44:01.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.085+0000 7fcd2ac00640 1 -- start start 2024-04-23T14:44:01.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.086+0000 7fcd2ac00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd24102050 0x7fcd24075db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:01.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.086+0000 7fcd2ac00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd24103240 0x7fcd240743f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:01.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.086+0000 7fcd2ac00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd24103c00 0x7fcd24074930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:01.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.086+0000 7fcd2ac00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcd2410dce0 con 0x7fcd24103240 2024-04-23T14:44:01.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.086+0000 7fcd2ac00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcd2410db60 con 0x7fcd24102050 2024-04-23T14:44:01.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.086+0000 7fcd2ac00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcd2410de60 con 0x7fcd24103c00 2024-04-23T14:44:01.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.086+0000 7fcd1b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd24103240 0x7fcd240743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:01.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.086+0000 7fcd1b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd24103240 0x7fcd240743f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35992/0 (socket says 172.21.15.28:35992) 2024-04-23T14:44:01.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.086+0000 7fcd1b400640 1 -- 172.21.15.28:0/4035482927 learned_addr learned my addr 172.21.15.28:0/4035482927 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:01.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.086+0000 7fcd23e00640 1 --2- 172.21.15.28:0/4035482927 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd24102050 0x7fcd24075db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:01.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.086+0000 7fcd28c00640 1 --2- 172.21.15.28:0/4035482927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd24103c00 0x7fcd24074930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:01.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.086+0000 7fcd1b400640 1 -- 172.21.15.28:0/4035482927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd24103c00 msgr2=0x7fcd24074930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:01.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.086+0000 7fcd1b400640 1 --2- 172.21.15.28:0/4035482927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd24103c00 0x7fcd24074930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:01.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.086+0000 7fcd1b400640 1 -- 172.21.15.28:0/4035482927 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd24102050 msgr2=0x7fcd24075db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:01.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.086+0000 7fcd1b400640 1 --2- 172.21.15.28:0/4035482927 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd24102050 0x7fcd24075db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:01.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.087+0000 7fcd1b400640 1 -- 172.21.15.28:0/4035482927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd240751c0 con 0x7fcd24103240 2024-04-23T14:44:01.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.087+0000 7fcd23e00640 1 --2- 172.21.15.28:0/4035482927 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd24102050 0x7fcd24075db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:44:01.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.087+0000 7fcd1b400640 1 --2- 172.21.15.28:0/4035482927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd24103240 0x7fcd240743f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c00e9f0 tx=0x7fcd0c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:01.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.087+0000 7fcd21600640 1 -- 172.21.15.28:0/4035482927 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c00cd80 con 0x7fcd24103240 2024-04-23T14:44:01.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.087+0000 7fcd2ac00640 1 -- 172.21.15.28:0/4035482927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd241a23f0 con 0x7fcd24103240 2024-04-23T14:44:01.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.087+0000 7fcd21600640 1 -- 172.21.15.28:0/4035482927 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd0c005500 con 0x7fcd24103240 2024-04-23T14:44:01.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.087+0000 7fcd21600640 1 -- 172.21.15.28:0/4035482927 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c010690 con 0x7fcd24103240 2024-04-23T14:44:01.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.087+0000 7fcd2ac00640 1 -- 172.21.15.28:0/4035482927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd241a2940 con 0x7fcd24103240 2024-04-23T14:44:01.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.088+0000 7fcd2ac00640 1 -- 172.21.15.28:0/4035482927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd24109370 con 0x7fcd24103240 2024-04-23T14:44:01.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.090+0000 7fcd21600640 1 -- 172.21.15.28:0/4035482927 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 138663+0+0 (secure 0 0 0) 0x7fcd0c0046c0 con 0x7fcd24103240 2024-04-23T14:44:01.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.092+0000 7fcd21600640 1 --2- 172.21.15.28:0/4035482927 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fcd080a3c00 0x7fcd080a60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:01.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.092+0000 7fcd23e00640 1 -- 172.21.15.28:0/4035482927 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fcd080a3c00 msgr2=0x7fcd080a60c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1997136173 2024-04-23T14:44:01.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.092+0000 7fcd21600640 1 -- 172.21.15.28:0/4035482927 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8240+0+0 (secure 0 0 0) 0x7fcd0c0d0920 con 0x7fcd24103240 2024-04-23T14:44:01.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.092+0000 7fcd23e00640 1 --2- 172.21.15.28:0/4035482927 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fcd080a3c00 0x7fcd080a60c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T14:44:01.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.101+0000 7fcd21600640 1 -- 172.21.15.28:0/4035482927 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcd0c09d2c0 con 0x7fcd24103240 2024-04-23T14:44:01.283 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:44:00 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:44:00.975+0000 7f0b11938200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T14:44:01.283 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:44:01.282+0000 7f0b11938200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T14:44:01.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.293+0000 7fcd23e00640 1 -- 172.21.15.28:0/4035482927 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fcd080a3c00 msgr2=0x7fcd080a60c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1997136173 2024-04-23T14:44:01.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.293+0000 7fcd23e00640 1 --2- 172.21.15.28:0/4035482927 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fcd080a3c00 0x7fcd080a60c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T14:44:01.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.302+0000 7fcd21600640 1 -- 172.21.15.28:0/4035482927 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 22) ==== 138331+0+0 (secure 0 0 0) 0x7fcd0c099230 con 0x7fcd24103240 2024-04-23T14:44:01.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.303+0000 7fcd21600640 1 -- 172.21.15.28:0/4035482927 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fcd080a3c00 msgr2=0x7fcd080a60c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T14:44:01.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.303+0000 7fcd21600640 1 --2- 172.21.15.28:0/4035482927 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fcd080a3c00 0x7fcd080a60c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:01.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.318+0000 7fcd2ac00640 1 -- 172.21.15.28:0/4035482927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcd24062c00 con 0x7fcd24103240 2024-04-23T14:44:01.320 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-23T14:44:01.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.320+0000 7fcd21600640 1 -- 172.21.15.28:0/4035482927 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7fcd0c0c9b90 con 0x7fcd24103240 2024-04-23T14:44:01.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.323+0000 7fcd2ac00640 1 -- 172.21.15.28:0/4035482927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd24103240 msgr2=0x7fcd240743f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:01.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.323+0000 7fcd2ac00640 1 --2- 172.21.15.28:0/4035482927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd24103240 0x7fcd240743f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c00e9f0 tx=0x7fcd0c00edf0 comp rx=0 tx=0).stop 2024-04-23T14:44:01.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.323+0000 7fcd2ac00640 1 -- 172.21.15.28:0/4035482927 shutdown_connections 2024-04-23T14:44:01.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.323+0000 7fcd2ac00640 1 --2- 172.21.15.28:0/4035482927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd24103c00 0x7fcd24074930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:01.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.323+0000 7fcd2ac00640 1 --2- 172.21.15.28:0/4035482927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd24103240 0x7fcd240743f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:01.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.323+0000 7fcd2ac00640 1 --2- 172.21.15.28:0/4035482927 >> [v2:172.21.15.28:6800/1997136173,v1:172.21.15.28:6801/1997136173] conn(0x7fcd080a3c00 0x7fcd080a60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:01.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.323+0000 7fcd2ac00640 1 --2- 172.21.15.28:0/4035482927 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd24102050 0x7fcd24075db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:01.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.323+0000 7fcd2ac00640 1 -- 172.21.15.28:0/4035482927 >> 172.21.15.28:0/4035482927 conn(0x7fcd240fd5e0 msgr2=0x7fcd24106540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:01.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.323+0000 7fcd2ac00640 1 -- 172.21.15.28:0/4035482927 shutdown_connections 2024-04-23T14:44:01.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.323+0000 7fcd2ac00640 1 -- 172.21.15.28:0/4035482927 wait complete. 2024-04-23T14:44:01.342 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T14:44:01.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.564+0000 7ff73ca00640 1 -- 172.21.15.28:0/229401232 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff738103c80 msgr2=0x7ff73810a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:01.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.564+0000 7ff73ca00640 1 --2- 172.21.15.28:0/229401232 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff738103c80 0x7ff73810a510 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff730009be0 tx=0x7ff73002a6c0 comp rx=0 tx=0).stop 2024-04-23T14:44:01.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.565+0000 7ff73ca00640 1 -- 172.21.15.28:0/229401232 shutdown_connections 2024-04-23T14:44:01.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.565+0000 7ff73ca00640 1 --2- 172.21.15.28:0/229401232 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff738103c80 0x7ff73810a510 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:01.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.565+0000 7ff73ca00640 1 --2- 172.21.15.28:0/229401232 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff7381032c0 0x7ff738103740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:01.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.565+0000 7ff73ca00640 1 --2- 172.21.15.28:0/229401232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7381020d0 0x7ff7381024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:01.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.565+0000 7ff73ca00640 1 -- 172.21.15.28:0/229401232 >> 172.21.15.28:0/229401232 conn(0x7ff7380fd640 msgr2=0x7ff7380ffa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:01.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.565+0000 7ff73ca00640 1 -- 172.21.15.28:0/229401232 shutdown_connections 2024-04-23T14:44:01.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.565+0000 7ff73ca00640 1 -- 172.21.15.28:0/229401232 wait complete. 2024-04-23T14:44:01.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.566+0000 7ff73ca00640 1 Processor -- start 2024-04-23T14:44:01.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.566+0000 7ff73ca00640 1 -- start start 2024-04-23T14:44:01.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.567+0000 7ff73ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7381020d0 0x7ff7381960d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:01.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.567+0000 7ff73ca00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7381032c0 0x7ff738196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:01.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.567+0000 7ff73ca00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff738103c80 0x7ff73819d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:01.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.567+0000 7ff73ca00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff73810baa0 con 0x7ff7381020d0 2024-04-23T14:44:01.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.567+0000 7ff73ca00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff73810b920 con 0x7ff7381032c0 2024-04-23T14:44:01.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.567+0000 7ff73ca00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff73810bc20 con 0x7ff738103c80 2024-04-23T14:44:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.567+0000 7ff735400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7381032c0 0x7ff738196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.567+0000 7ff735400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7381032c0 0x7ff738196610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:36906/0 (socket says 172.21.15.28:36906) 2024-04-23T14:44:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.567+0000 7ff735400640 1 -- 172.21.15.28:0/371185660 learned_addr learned my addr 172.21.15.28:0/371185660 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.567+0000 7ff735e00640 1 --2- 172.21.15.28:0/371185660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7381020d0 0x7ff7381960d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.567+0000 7ff736800640 1 --2- 172.21.15.28:0/371185660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff738103c80 0x7ff73819d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.568+0000 7ff735400640 1 -- 172.21.15.28:0/371185660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff738103c80 msgr2=0x7ff73819d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.568+0000 7ff735400640 1 --2- 172.21.15.28:0/371185660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff738103c80 0x7ff73819d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.568+0000 7ff735400640 1 -- 172.21.15.28:0/371185660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7381020d0 msgr2=0x7ff7381960d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.568+0000 7ff735400640 1 --2- 172.21.15.28:0/371185660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7381020d0 0x7ff7381960d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:01.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.568+0000 7ff735400640 1 -- 172.21.15.28:0/371185660 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff73819dcf0 con 0x7ff7381032c0 2024-04-23T14:44:01.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.569+0000 7ff735400640 1 --2- 172.21.15.28:0/371185660 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7381032c0 0x7ff738196610 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff72400e9c0 tx=0x7ff72400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:01.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.569+0000 7ff72ea00640 1 -- 172.21.15.28:0/371185660 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff72400ccc0 con 0x7ff7381032c0 2024-04-23T14:44:01.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.570+0000 7ff72ea00640 1 -- 172.21.15.28:0/371185660 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff724005500 con 0x7ff7381032c0 2024-04-23T14:44:01.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.570+0000 7ff72ea00640 1 -- 172.21.15.28:0/371185660 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff724010430 con 0x7ff7381032c0 2024-04-23T14:44:01.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.570+0000 7ff73ca00640 1 -- 172.21.15.28:0/371185660 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff73819dfe0 con 0x7ff7381032c0 2024-04-23T14:44:01.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.570+0000 7ff73ca00640 1 -- 172.21.15.28:0/371185660 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff73819e450 con 0x7ff7381032c0 2024-04-23T14:44:01.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.571+0000 7ff73ca00640 1 -- 172.21.15.28:0/371185660 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7381093f0 con 0x7ff7381032c0 2024-04-23T14:44:01.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.575+0000 7ff72ea00640 1 -- 172.21.15.28:0/371185660 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 22) ==== 138331+0+0 (secure 0 0 0) 0x7ff7240046c0 con 0x7ff7381032c0 2024-04-23T14:44:01.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.576+0000 7ff72ea00640 1 -- 172.21.15.28:0/371185660 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7ff7240d0210 con 0x7ff7381032c0 2024-04-23T14:44:01.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.581+0000 7ff72ea00640 1 -- 172.21.15.28:0/371185660 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff72409cee0 con 0x7ff7381032c0 2024-04-23T14:44:01.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/800019970' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:44:01.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: Active manager daemon x restarted 2024-04-23T14:44:01.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: Activating manager daemon x 2024-04-23T14:44:01.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: osdmap e88: 12 total, 12 up, 12 in 2024-04-23T14:44:01.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: mgrmap e22: x(active, starting, since 0.0126133s), standbys: z, y 2024-04-23T14:44:01.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T14:44:01.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:44:01.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:44:01.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-23T14:44:01.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-23T14:44:01.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-23T14:44:01.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-23T14:44:01.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T14:44:01.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T14:44:01.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T14:44:01.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T14:44:01.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T14:44:01.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-23T14:44:01.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-23T14:44:01.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/4035482927' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:44:01.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T14:44:01.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T14:44:01.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-23T14:44:01.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T14:44:01.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T14:44:01.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T14:44:01.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T14:44:01.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T14:44:01.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T14:44:01.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T14:44:01.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T14:44:01.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T14:44:01.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T14:44:01.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T14:44:01.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T14:44:01.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T14:44:01.627 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T14:44:01.627 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T14:44:01.627 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T14:44:01.627 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: Manager daemon x is now available 2024-04-23T14:44:01.627 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:44:01.627 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T14:44:01.627 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T14:44:01.627 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T14:44:01.627 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T14:44:01.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/800019970' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:44:01.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: Active manager daemon x restarted 2024-04-23T14:44:01.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: Activating manager daemon x 2024-04-23T14:44:01.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: osdmap e88: 12 total, 12 up, 12 in 2024-04-23T14:44:01.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: mgrmap e22: x(active, starting, since 0.0126133s), standbys: z, y 2024-04-23T14:44:01.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T14:44:01.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:44:01.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:44:01.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-23T14:44:01.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-23T14:44:01.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-23T14:44:01.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-23T14:44:01.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T14:44:01.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T14:44:01.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T14:44:01.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T14:44:01.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T14:44:01.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-23T14:44:01.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-23T14:44:01.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/4035482927' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:44:01.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T14:44:01.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T14:44:01.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-23T14:44:01.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T14:44:01.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T14:44:01.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T14:44:01.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T14:44:01.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T14:44:01.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T14:44:01.788 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T14:44:01.789 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T14:44:01.789 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T14:44:01.789 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T14:44:01.789 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T14:44:01.789 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T14:44:01.789 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T14:44:01.789 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T14:44:01.789 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T14:44:01.789 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T14:44:01.789 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: Manager daemon x is now available 2024-04-23T14:44:01.789 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:44:01.789 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T14:44:01.789 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T14:44:01.789 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T14:44:01.789 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:01 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T14:44:01.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.844+0000 7ff73ca00640 1 -- 172.21.15.28:0/371185660 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff738062d00 con 0x7ff7381032c0 2024-04-23T14:44:01.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.846+0000 7ff72ea00640 1 -- 172.21.15.28:0/371185660 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7ff72409c7e0 con 0x7ff7381032c0 2024-04-23T14:44:01.846 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-23T14:44:01.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.848+0000 7ff73ca00640 1 -- 172.21.15.28:0/371185660 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7381032c0 msgr2=0x7ff738196610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:01.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.848+0000 7ff73ca00640 1 --2- 172.21.15.28:0/371185660 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7381032c0 0x7ff738196610 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff72400e9c0 tx=0x7ff72400ee80 comp rx=0 tx=0).stop 2024-04-23T14:44:01.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.848+0000 7ff73ca00640 1 -- 172.21.15.28:0/371185660 shutdown_connections 2024-04-23T14:44:01.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.848+0000 7ff73ca00640 1 --2- 172.21.15.28:0/371185660 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7381032c0 0x7ff738196610 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:01.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.848+0000 7ff73ca00640 1 --2- 172.21.15.28:0/371185660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff738103c80 0x7ff73819d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:01.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.848+0000 7ff73ca00640 1 --2- 172.21.15.28:0/371185660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7381020d0 0x7ff7381960d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:01.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.848+0000 7ff73ca00640 1 -- 172.21.15.28:0/371185660 >> 172.21.15.28:0/371185660 conn(0x7ff7380fd640 msgr2=0x7ff7381065c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:01.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.848+0000 7ff73ca00640 1 -- 172.21.15.28:0/371185660 shutdown_connections 2024-04-23T14:44:01.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:01.849+0000 7ff73ca00640 1 -- 172.21.15.28:0/371185660 wait complete. 2024-04-23T14:44:01.865 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T14:44:01.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/800019970' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:44:01.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: Active manager daemon x restarted 2024-04-23T14:44:01.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: Activating manager daemon x 2024-04-23T14:44:01.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: osdmap e88: 12 total, 12 up, 12 in 2024-04-23T14:44:01.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: mgrmap e22: x(active, starting, since 0.0126133s), standbys: z, y 2024-04-23T14:44:01.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T14:44:01.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T14:44:01.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T14:44:01.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-23T14:44:01.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-23T14:44:01.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-23T14:44:01.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-23T14:44:01.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T14:44:01.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T14:44:01.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T14:44:01.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T14:44:01.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T14:44:01.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-23T14:44:01.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-23T14:44:01.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/4035482927' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:44:01.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T14:44:01.875 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T14:44:01.875 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-23T14:44:01.875 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T14:44:01.875 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T14:44:01.875 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T14:44:01.875 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T14:44:01.875 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T14:44:01.875 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T14:44:01.875 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T14:44:01.875 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T14:44:01.875 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T14:44:01.875 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T14:44:01.875 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T14:44:01.875 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T14:44:01.875 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T14:44:01.876 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T14:44:01.876 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T14:44:01.876 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T14:44:01.876 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: Manager daemon x is now available 2024-04-23T14:44:01.876 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:44:01.876 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T14:44:01.876 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T14:44:01.876 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T14:44:01.876 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:01 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T14:44:02.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.044+0000 7f6d58e00640 1 -- 172.21.15.28:0/1358612812 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d54104280 msgr2=0x7f6d541046a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:02.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.044+0000 7f6d58e00640 1 --2- 172.21.15.28:0/1358612812 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d54104280 0x7f6d541046a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6d44009900 tx=0x7f6d4402a410 comp rx=0 tx=0).stop 2024-04-23T14:44:02.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.044+0000 7f6d58e00640 1 -- 172.21.15.28:0/1358612812 shutdown_connections 2024-04-23T14:44:02.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.044+0000 7f6d58e00640 1 --2- 172.21.15.28:0/1358612812 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6d54105e30 0x7f6d5410c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:02.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.044+0000 7f6d58e00640 1 --2- 172.21.15.28:0/1358612812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d54105470 0x7f6d541058f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:02.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.044+0000 7f6d58e00640 1 --2- 172.21.15.28:0/1358612812 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d54104280 0x7f6d541046a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:02.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.044+0000 7f6d58e00640 1 -- 172.21.15.28:0/1358612812 >> 172.21.15.28:0/1358612812 conn(0x7f6d540ff7f0 msgr2=0x7f6d54101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:02.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.044+0000 7f6d58e00640 1 -- 172.21.15.28:0/1358612812 shutdown_connections 2024-04-23T14:44:02.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.045+0000 7f6d58e00640 1 -- 172.21.15.28:0/1358612812 wait complete. 2024-04-23T14:44:02.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.045+0000 7f6d58e00640 1 Processor -- start 2024-04-23T14:44:02.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.046+0000 7f6d58e00640 1 -- start start 2024-04-23T14:44:02.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.046+0000 7f6d58e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d54104280 0x7f6d5419a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:02.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.046+0000 7f6d58e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d54105470 0x7f6d5419aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:02.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.046+0000 7f6d58e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6d54105e30 0x7f6d541a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:02.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.046+0000 7f6d58e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6d5410ff60 con 0x7f6d54105470 2024-04-23T14:44:02.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.046+0000 7f6d58e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6d5410fde0 con 0x7f6d54104280 2024-04-23T14:44:02.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.046+0000 7f6d58e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6d541100e0 con 0x7f6d54105e30 2024-04-23T14:44:02.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.046+0000 7f6d51e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d54104280 0x7f6d5419a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:02.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.046+0000 7f6d51e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d54104280 0x7f6d5419a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:36958/0 (socket says 172.21.15.28:36958) 2024-04-23T14:44:02.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.046+0000 7f6d51e00640 1 -- 172.21.15.28:0/4284898779 learned_addr learned my addr 172.21.15.28:0/4284898779 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:02.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.046+0000 7f6d51400640 1 --2- 172.21.15.28:0/4284898779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d54105470 0x7f6d5419aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:02.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.046+0000 7f6d52800640 1 --2- 172.21.15.28:0/4284898779 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6d54105e30 0x7f6d541a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:02.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.046+0000 7f6d51e00640 1 -- 172.21.15.28:0/4284898779 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6d54105e30 msgr2=0x7f6d541a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:02.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.046+0000 7f6d51e00640 1 --2- 172.21.15.28:0/4284898779 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6d54105e30 0x7f6d541a1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:02.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.046+0000 7f6d51e00640 1 -- 172.21.15.28:0/4284898779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d54105470 msgr2=0x7f6d5419aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:02.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.046+0000 7f6d51e00640 1 --2- 172.21.15.28:0/4284898779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d54105470 0x7f6d5419aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:02.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.046+0000 7f6d51e00640 1 -- 172.21.15.28:0/4284898779 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d541a21b0 con 0x7f6d54104280 2024-04-23T14:44:02.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.046+0000 7f6d51400640 1 --2- 172.21.15.28:0/4284898779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d54105470 0x7f6d5419aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:44:02.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.046+0000 7f6d52800640 1 --2- 172.21.15.28:0/4284898779 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6d54105e30 0x7f6d541a1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:44:02.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.047+0000 7f6d51e00640 1 --2- 172.21.15.28:0/4284898779 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d54104280 0x7f6d5419a590 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6d44031440 tx=0x7f6d44032ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:02.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.047+0000 7f6d4aa00640 1 -- 172.21.15.28:0/4284898779 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d4403f030 con 0x7f6d54104280 2024-04-23T14:44:02.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.047+0000 7f6d4aa00640 1 -- 172.21.15.28:0/4284898779 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d44003ce0 con 0x7f6d54104280 2024-04-23T14:44:02.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.047+0000 7f6d4aa00640 1 -- 172.21.15.28:0/4284898779 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d44035070 con 0x7f6d54104280 2024-04-23T14:44:02.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.047+0000 7f6d58e00640 1 -- 172.21.15.28:0/4284898779 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d541a2440 con 0x7f6d54104280 2024-04-23T14:44:02.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.047+0000 7f6d58e00640 1 -- 172.21.15.28:0/4284898779 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d541a28e0 con 0x7f6d54104280 2024-04-23T14:44:02.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.049+0000 7f6d58e00640 1 -- 172.21.15.28:0/4284898779 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d5410b5a0 con 0x7f6d54104280 2024-04-23T14:44:02.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.050+0000 7f6d4aa00640 1 -- 172.21.15.28:0/4284898779 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 22) ==== 138331+0+0 (secure 0 0 0) 0x7f6d44042030 con 0x7f6d54104280 2024-04-23T14:44:02.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.051+0000 7f6d4aa00640 1 -- 172.21.15.28:0/4284898779 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f6d440f0900 con 0x7f6d54104280 2024-04-23T14:44:02.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.055+0000 7f6d4aa00640 1 -- 172.21.15.28:0/4284898779 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6d440bd570 con 0x7f6d54104280 2024-04-23T14:44:02.125 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:44:01.696+0000 7f0a67400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T14:44:02.125 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:44:01.696+0000 7f0a67400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T14:44:02.125 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:44:01.696+0000 7f0a67400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T14:44:02.125 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:44:01.696+0000 7f0a67400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T14:44:02.125 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:44:01.696+0000 7f0a67400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T14:44:02.125 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:44:01.697+0000 7f0a64200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T14:44:02.125 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:44:01.697+0000 7f0a64200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T14:44:02.125 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:44:01.697+0000 7f0a64200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T14:44:02.125 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:44:01.697+0000 7f0a64200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T14:44:02.125 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:44:01 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:44:01.697+0000 7f0a64200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T14:44:02.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.260+0000 7f6d58e00640 1 -- 172.21.15.28:0/4284898779 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6d54062c00 con 0x7f6d54104280 2024-04-23T14:44:02.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.261+0000 7f6d4aa00640 1 -- 172.21.15.28:0/4284898779 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f6d440bce70 con 0x7f6d54104280 2024-04-23T14:44:02.261 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-23T14:44:02.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.263+0000 7f6d58e00640 1 -- 172.21.15.28:0/4284898779 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d54104280 msgr2=0x7f6d5419a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:02.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.263+0000 7f6d58e00640 1 --2- 172.21.15.28:0/4284898779 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d54104280 0x7f6d5419a590 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6d44031440 tx=0x7f6d44032ce0 comp rx=0 tx=0).stop 2024-04-23T14:44:02.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.263+0000 7f6d58e00640 1 -- 172.21.15.28:0/4284898779 shutdown_connections 2024-04-23T14:44:02.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.263+0000 7f6d58e00640 1 --2- 172.21.15.28:0/4284898779 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6d54105e30 0x7f6d541a1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:02.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.263+0000 7f6d58e00640 1 --2- 172.21.15.28:0/4284898779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d54105470 0x7f6d5419aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:02.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.263+0000 7f6d58e00640 1 --2- 172.21.15.28:0/4284898779 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d54104280 0x7f6d5419a590 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:02.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.263+0000 7f6d58e00640 1 -- 172.21.15.28:0/4284898779 >> 172.21.15.28:0/4284898779 conn(0x7f6d540ff7f0 msgr2=0x7f6d54108770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:02.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.263+0000 7f6d58e00640 1 -- 172.21.15.28:0/4284898779 shutdown_connections 2024-04-23T14:44:02.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.264+0000 7f6d58e00640 1 -- 172.21.15.28:0/4284898779 wait complete. 2024-04-23T14:44:02.281 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:44:02.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.468+0000 7f0487e00640 1 -- 172.21.15.28:0/1926889283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f048010ca40 msgr2=0x7f048010ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:02.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.468+0000 7f0487e00640 1 --2- 172.21.15.28:0/1926889283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f048010ca40 0x7f048010ee30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f046c009900 tx=0x7f046c02a3f0 comp rx=0 tx=0).stop 2024-04-23T14:44:02.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.469+0000 7f0487e00640 1 -- 172.21.15.28:0/1926889283 shutdown_connections 2024-04-23T14:44:02.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.469+0000 7f0487e00640 1 --2- 172.21.15.28:0/1926889283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f048010ca40 0x7f048010ee30 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:02.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.469+0000 7f0487e00640 1 --2- 172.21.15.28:0/1926889283 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f048010f3c0 0x7f0480111880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:02.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.469+0000 7f0487e00640 1 --2- 172.21.15.28:0/1926889283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f048010a110 0x7f048010c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:02.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.469+0000 7f0487e00640 1 -- 172.21.15.28:0/1926889283 >> 172.21.15.28:0/1926889283 conn(0x7f048006f240 msgr2=0x7f0480071660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:02.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.469+0000 7f0487e00640 1 -- 172.21.15.28:0/1926889283 shutdown_connections 2024-04-23T14:44:02.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.469+0000 7f0487e00640 1 -- 172.21.15.28:0/1926889283 wait complete. 2024-04-23T14:44:02.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.470+0000 7f0487e00640 1 Processor -- start 2024-04-23T14:44:02.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.471+0000 7f0487e00640 1 -- start start 2024-04-23T14:44:02.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.471+0000 7f0487e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f048010a110 0x7f04801a2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:02.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.471+0000 7f0487e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f048010ca40 0x7f04801a32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:02.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.471+0000 7f0487e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f048010f3c0 0x7f04801aa310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:02.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.471+0000 7f0487e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0480115430 con 0x7f048010f3c0 2024-04-23T14:44:02.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.471+0000 7f0487e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f04801152b0 con 0x7f048010ca40 2024-04-23T14:44:02.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.471+0000 7f0487e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f04801155b0 con 0x7f048010a110 2024-04-23T14:44:02.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.471+0000 7f0485400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f048010a110 0x7f04801a2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:02.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.471+0000 7f0484a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f048010ca40 0x7f04801a32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:02.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.471+0000 7f0485400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f048010a110 0x7f04801a2da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:44152/0 (socket says 172.21.15.28:44152) 2024-04-23T14:44:02.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.471+0000 7f0484a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f048010ca40 0x7f04801a32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:36974/0 (socket says 172.21.15.28:36974) 2024-04-23T14:44:02.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.471+0000 7f0485400640 1 -- 172.21.15.28:0/2156419162 learned_addr learned my addr 172.21.15.28:0/2156419162 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:02.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.471+0000 7f0485e00640 1 --2- 172.21.15.28:0/2156419162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f048010f3c0 0x7f04801aa310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:02.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.472+0000 7f0484a00640 1 -- 172.21.15.28:0/2156419162 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f048010a110 msgr2=0x7f04801a2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:02.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.472+0000 7f0484a00640 1 --2- 172.21.15.28:0/2156419162 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f048010a110 0x7f04801a2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:02.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.472+0000 7f0484a00640 1 -- 172.21.15.28:0/2156419162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f048010f3c0 msgr2=0x7f04801aa310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:02.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.472+0000 7f0484a00640 1 --2- 172.21.15.28:0/2156419162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f048010f3c0 0x7f04801aa310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:02.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.472+0000 7f0484a00640 1 -- 172.21.15.28:0/2156419162 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04801aa9b0 con 0x7f048010ca40 2024-04-23T14:44:02.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.472+0000 7f0485400640 1 --2- 172.21.15.28:0/2156419162 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f048010a110 0x7f04801a2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:44:02.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.472+0000 7f0485e00640 1 --2- 172.21.15.28:0/2156419162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f048010f3c0 0x7f04801aa310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:44:02.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.472+0000 7f0484a00640 1 --2- 172.21.15.28:0/2156419162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f048010ca40 0x7f04801a32e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f046c02a910 tx=0x7f046c032ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:02.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.475+0000 7f047e000640 1 -- 172.21.15.28:0/2156419162 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f046c03f030 con 0x7f048010ca40 2024-04-23T14:44:02.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.476+0000 7f047e000640 1 -- 172.21.15.28:0/2156419162 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f046c003ce0 con 0x7f048010ca40 2024-04-23T14:44:02.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.476+0000 7f047e000640 1 -- 172.21.15.28:0/2156419162 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f046c035070 con 0x7f048010ca40 2024-04-23T14:44:02.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.476+0000 7f0487e00640 1 -- 172.21.15.28:0/2156419162 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04801aac40 con 0x7f048010ca40 2024-04-23T14:44:02.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.476+0000 7f0487e00640 1 -- 172.21.15.28:0/2156419162 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04801ab150 con 0x7f048010ca40 2024-04-23T14:44:02.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.477+0000 7f0487e00640 1 -- 172.21.15.28:0/2156419162 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0480068e20 con 0x7f048010ca40 2024-04-23T14:44:02.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.478+0000 7f047e000640 1 -- 172.21.15.28:0/2156419162 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 23) ==== 138458+0+0 (secure 0 0 0) 0x7f046c042030 con 0x7f048010ca40 2024-04-23T14:44:02.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.480+0000 7f047e000640 1 --2- 172.21.15.28:0/2156419162 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f04640a3a30 0x7f04640a5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:02.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.481+0000 7f047e000640 1 -- 172.21.15.28:0/2156419162 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f046c0f05e0 con 0x7f048010ca40 2024-04-23T14:44:02.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.481+0000 7f0485400640 1 --2- 172.21.15.28:0/2156419162 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f04640a3a30 0x7f04640a5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:02.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.481+0000 7f0485400640 1 --2- 172.21.15.28:0/2156419162 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f04640a3a30 0x7f04640a5ef0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f04801a4270 tx=0x7f0470009500 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:02.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.484+0000 7f047e000640 1 -- 172.21.15.28:0/2156419162 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f046c0bd160 con 0x7f048010ca40 2024-04-23T14:44:02.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:02 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/371185660' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T14:44:02.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:02 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/4284898779' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T14:44:02.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:02 smithi028 ceph-mon[27810]: mgrmap e23: x(active, since 1.02389s), standbys: z, y 2024-04-23T14:44:02.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.696+0000 7f0487e00640 1 -- 172.21.15.28:0/2156419162 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f04801ab9f0 con 0x7f048010ca40 2024-04-23T14:44:02.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.698+0000 7f047e000640 1 -- 172.21.15.28:0/2156419162 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+23026 (secure 0 0 0) 0x7f046c0bca60 con 0x7f048010ca40 2024-04-23T14:44:02.698 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:44:02.698 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":88,"fsid":"1ee6e996-017e-11ef-bc93-c7b262605968","created":"2024-04-23T14:32:34.581082+0000","modified":"2024-04-23T14:44:01.286339+0000","last_up_change":"2024-04-23T14:40:43.375846+0000","last_in_change":"2024-04-23T14:40:23.576939+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:36:27.698112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:41:37.030341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:41:38.444742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e0b427bc-be65-4543-99cc-2f53ad98ddad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6803","nonce":2993445336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6805","nonce":2993445336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6809","nonce":2993445336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6807","nonce":2993445336}]},"public_addr":"172.21.15.28:6803/2993445336","cluster_addr":"172.21.15.28:6805/2993445336","heartbeat_back_addr":"172.21.15.28:6809/2993445336","heartbeat_front_addr":"172.21.15.28:6807/2993445336","state":["exists","up"]},{"osd":1,"uuid":"ef527ec9-1bf3-4bb0-8df3-f5a015af557c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6801","nonce":4024093383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6803","nonce":4024093383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6807","nonce":4024093383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6805","nonce":4024093383}]},"public_addr":"172.21.15.38:6801/4024093383","cluster_addr":"172.21.15.38:6803/4024093383","heartbeat_back_addr":"172.21.15.38:6807/4024093383","heartbeat_front_addr":"172.21.15.38:6805/4024093383","state":["exists","up"]},{"osd":2,"uuid":"b33b23d9-adce-4c11-b304-2e117bdf6bfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6801","nonce":570739029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6803","nonce":570739029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6807","nonce":570739029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6805","nonce":570739029}]},"public_addr":"172.21.15.105:6801/570739029","cluster_addr":"172.21.15.105:6803/570739029","heartbeat_back_addr":"172.21.15.105:6807/570739029","heartbeat_front_addr":"172.21.15.105:6805/570739029","state":["exists","up"]},{"osd":3,"uuid":"28bd5608-b426-434e-9db1-bdaa9828bdb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6811","nonce":2480400247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6813","nonce":2480400247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6817","nonce":2480400247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6815","nonce":2480400247}]},"public_addr":"172.21.15.28:6811/2480400247","cluster_addr":"172.21.15.28:6813/2480400247","heartbeat_back_addr":"172.21.15.28:6817/2480400247","heartbeat_front_addr":"172.21.15.28:6815/2480400247","state":["exists","up"]},{"osd":4,"uuid":"e5d1c89b-025f-4d57-8807-59d75c787b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6809","nonce":3772189246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6811","nonce":3772189246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6815","nonce":3772189246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6813","nonce":3772189246}]},"public_addr":"172.21.15.38:6809/3772189246","cluster_addr":"172.21.15.38:6811/3772189246","heartbeat_back_addr":"172.21.15.38:6815/3772189246","heartbeat_front_addr":"172.21.15.38:6813/3772189246","state":["exists","up"]},{"osd":5,"uuid":"9cf9cf4c-aafd-4ed3-bfb1-0527710a0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6809","nonce":928755894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6811","nonce":928755894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6815","nonce":928755894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6813","nonce":928755894}]},"public_addr":"172.21.15.105:6809/928755894","cluster_addr":"172.21.15.105:6811/928755894","heartbeat_back_addr":"172.21.15.105:6815/928755894","heartbeat_front_addr":"172.21.15.105:6813/928755894","state":["exists","up"]},{"osd":6,"uuid":"304db525-d24d-4687-bfa0-f92bdf66d1e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6819","nonce":897979441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6821","nonce":897979441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6825","nonce":897979441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6823","nonce":897979441}]},"public_addr":"172.21.15.28:6819/897979441","cluster_addr":"172.21.15.28:6821/897979441","heartbeat_back_addr":"172.21.15.28:6825/897979441","heartbeat_front_addr":"172.21.15.28:6823/897979441","state":["exists","up"]},{"osd":7,"uuid":"08e01d59-03f3-47b1-b45f-35ec5a50992c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6817","nonce":2400251786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6819","nonce":2400251786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6823","nonce":2400251786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6821","nonce":2400251786}]},"public_addr":"172.21.15.38:6817/2400251786","cluster_addr":"172.21.15.38:6819/2400251786","heartbeat_back_addr":"172.21.15.38:6823/2400251786","heartbeat_front_addr":"172.21.15.38:6821/2400251786","state":["exists","up"]},{"osd":8,"uuid":"64ee1e95-0756-48b1-9c00-472d0c3cbd29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6817","nonce":1682312269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6819","nonce":1682312269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6823","nonce":1682312269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6821","nonce":1682312269}]},"public_addr":"172.21.15.105:6817/1682312269","cluster_addr":"172.21.15.105:6819/1682312269","heartbeat_back_addr":"172.21.15.105:6823/1682312269","heartbeat_front_addr":"172.21.15.105:6821/1682312269","state":["exists","up"]},{"osd":9,"uuid":"18f60893-3ffd-48c2-9db5-a67d35611d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6827","nonce":1292942516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6829","nonce":1292942516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6833","nonce":1292942516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6831","nonce":1292942516}]},"public_addr":"172.21.15.28:6827/1292942516","cluster_addr":"172.21.15.28:6829/1292942516","heartbeat_back_addr":"172.21.15.28:6833/1292942516","heartbeat_front_addr":"172.21.15.28:6831/1292942516","state":["exists","up"]},{"osd":10,"uuid":"76c06725-7040-4118-8894-fe808a87f999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6825","nonce":2444616468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6827","nonce":2444616468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6831","nonce":2444616468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6829","nonce":2444616468}]},"public_addr":"172.21.15.38:6825/2444616468","cluster_addr":"172.21.15.38:6827/2444616468","heartbeat_back_addr":"172.21.15.38:6831/2444616468","heartbeat_front_addr":"172.21.15.38:6829/2444616468","state":["exists","up"]},{"osd":11,"uuid":"e8fc9454-5540-4e2e-91bf-3ba251dbd1a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6825","nonce":3400299562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6827","nonce":3400299562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6831","nonce":3400299562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6829","nonce":3400299562}]},"public_addr":"172.21.15.105:6825/3400299562","cluster_addr":"172.21.15.105:6827/3400299562","heartbeat_back_addr":"172.21.15.105:6831/3400299562","heartbeat_front_addr":"172.21.15.105:6829/3400299562","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:35:33.615395+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:00.545293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:24.940211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:57.684887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:24.519061+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:49.010804+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:22.638478+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:49.559104+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:14.011771+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:47.859950+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:16.298445+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:41.619069+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.2d","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/38494154":"2024-04-24T14:44:01.286264+0000","172.21.15.28:0/905362079":"2024-04-24T14:44:01.286264+0000","172.21.15.28:6801/1997136173":"2024-04-24T14:44:01.286264+0000","172.21.15.28:6800/1997136173":"2024-04-24T14:44:01.286264+0000","172.21.15.28:0/590495740":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/1163172512":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/4290328460":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/4145020172":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/1997097543":"2024-04-24T14:44:01.286264+0000","172.21.15.28:0/3847772687":"2024-04-24T14:44:01.286264+0000","172.21.15.28:6800/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/720626958":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/948450262":"2024-04-24T14:33:07.576363+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:44:02.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.700+0000 7f0487e00640 1 -- 172.21.15.28:0/2156419162 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f04640a3a30 msgr2=0x7f04640a5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:02.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.700+0000 7f0487e00640 1 --2- 172.21.15.28:0/2156419162 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f04640a3a30 0x7f04640a5ef0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f04801a4270 tx=0x7f0470009500 comp rx=0 tx=0).stop 2024-04-23T14:44:02.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.700+0000 7f0487e00640 1 -- 172.21.15.28:0/2156419162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f048010ca40 msgr2=0x7f04801a32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:02.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.700+0000 7f0487e00640 1 --2- 172.21.15.28:0/2156419162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f048010ca40 0x7f04801a32e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f046c02a910 tx=0x7f046c032ce0 comp rx=0 tx=0).stop 2024-04-23T14:44:02.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.701+0000 7f0487e00640 1 -- 172.21.15.28:0/2156419162 shutdown_connections 2024-04-23T14:44:02.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.701+0000 7f0487e00640 1 --2- 172.21.15.28:0/2156419162 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f04640a3a30 0x7f04640a5ef0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:02.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.701+0000 7f0487e00640 1 --2- 172.21.15.28:0/2156419162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f048010ca40 0x7f04801a32e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:02.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.701+0000 7f0487e00640 1 --2- 172.21.15.28:0/2156419162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f048010f3c0 0x7f04801aa310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:02.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.701+0000 7f0487e00640 1 --2- 172.21.15.28:0/2156419162 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f048010a110 0x7f04801a2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:02.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.701+0000 7f0487e00640 1 -- 172.21.15.28:0/2156419162 >> 172.21.15.28:0/2156419162 conn(0x7f048006f240 msgr2=0x7f0480109460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:02.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.701+0000 7f0487e00640 1 -- 172.21.15.28:0/2156419162 shutdown_connections 2024-04-23T14:44:02.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.701+0000 7f0487e00640 1 -- 172.21.15.28:0/2156419162 wait complete. 2024-04-23T14:44:02.719 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T14:36:27.698112+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T14:41:37.030341+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T14:41:38.444742+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T14:44:02.719 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T14:44:02.763 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:02 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/371185660' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T14:44:02.763 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:02 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/4284898779' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T14:44:02.764 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:02 smithi038 ceph-mon[32348]: mgrmap e23: x(active, since 1.02389s), standbys: z, y 2024-04-23T14:44:02.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:02 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/371185660' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T14:44:02.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:02 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/4284898779' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T14:44:02.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:02 smithi105 ceph-mon[32555]: mgrmap e23: x(active, since 1.02389s), standbys: z, y 2024-04-23T14:44:02.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.897+0000 7f8d94400640 1 -- 172.21.15.28:0/987704333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d8c105470 msgr2=0x7f8d8c1058f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:02.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.897+0000 7f8d94400640 1 --2- 172.21.15.28:0/987704333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d8c105470 0x7f8d8c1058f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8d78009900 tx=0x7f8d7802a3f0 comp rx=0 tx=0).stop 2024-04-23T14:44:02.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.897+0000 7f8d94400640 1 -- 172.21.15.28:0/987704333 shutdown_connections 2024-04-23T14:44:02.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.897+0000 7f8d94400640 1 --2- 172.21.15.28:0/987704333 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d8c105e30 0x7f8d8c10c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:02.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.897+0000 7f8d94400640 1 --2- 172.21.15.28:0/987704333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d8c105470 0x7f8d8c1058f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:02.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.897+0000 7f8d94400640 1 --2- 172.21.15.28:0/987704333 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d8c104280 0x7f8d8c1046a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:02.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.897+0000 7f8d94400640 1 -- 172.21.15.28:0/987704333 >> 172.21.15.28:0/987704333 conn(0x7f8d8c0ff7f0 msgr2=0x7f8d8c101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:02.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.897+0000 7f8d94400640 1 -- 172.21.15.28:0/987704333 shutdown_connections 2024-04-23T14:44:02.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.897+0000 7f8d94400640 1 -- 172.21.15.28:0/987704333 wait complete. 2024-04-23T14:44:02.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.898+0000 7f8d94400640 1 Processor -- start 2024-04-23T14:44:02.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.898+0000 7f8d94400640 1 -- start start 2024-04-23T14:44:02.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.898+0000 7f8d94400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d8c104280 0x7f8d8c19a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:02.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.898+0000 7f8d94400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d8c105470 0x7f8d8c19aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:02.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.898+0000 7f8d94400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d8c105e30 0x7f8d8c1a1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:02.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.898+0000 7f8d94400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c10ff10 con 0x7f8d8c105e30 2024-04-23T14:44:02.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.898+0000 7f8d94400640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c10fd90 con 0x7f8d8c105470 2024-04-23T14:44:02.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.898+0000 7f8d94400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c110090 con 0x7f8d8c104280 2024-04-23T14:44:02.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.898+0000 7f8d92400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d8c105e30 0x7f8d8c1a1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:02.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.898+0000 7f8d92400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d8c105e30 0x7f8d8c1a1ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36208/0 (socket says 172.21.15.28:36208) 2024-04-23T14:44:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.898+0000 7f8d92400640 1 -- 172.21.15.28:0/2558556868 learned_addr learned my addr 172.21.15.28:0/2558556868 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.898+0000 7f8d91000640 1 --2- 172.21.15.28:0/2558556868 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d8c105470 0x7f8d8c19aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.898+0000 7f8d91a00640 1 --2- 172.21.15.28:0/2558556868 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d8c104280 0x7f8d8c19a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.899+0000 7f8d92400640 1 -- 172.21.15.28:0/2558556868 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d8c104280 msgr2=0x7f8d8c19a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.899+0000 7f8d92400640 1 --2- 172.21.15.28:0/2558556868 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d8c104280 0x7f8d8c19a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.899+0000 7f8d92400640 1 -- 172.21.15.28:0/2558556868 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d8c105470 msgr2=0x7f8d8c19aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.899+0000 7f8d92400640 1 --2- 172.21.15.28:0/2558556868 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d8c105470 0x7f8d8c19aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.899+0000 7f8d92400640 1 -- 172.21.15.28:0/2558556868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d8c1a2150 con 0x7f8d8c105e30 2024-04-23T14:44:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.899+0000 7f8d92400640 1 --2- 172.21.15.28:0/2558556868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d8c105e30 0x7f8d8c1a1ae0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8d8400eb10 tx=0x7f8d8400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.899+0000 7f8d8a000640 1 -- 172.21.15.28:0/2558556868 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d84004b20 con 0x7f8d8c105e30 2024-04-23T14:44:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.899+0000 7f8d8a000640 1 -- 172.21.15.28:0/2558556868 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d84004cc0 con 0x7f8d8c105e30 2024-04-23T14:44:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.899+0000 7f8d94400640 1 -- 172.21.15.28:0/2558556868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d8c1a2440 con 0x7f8d8c105e30 2024-04-23T14:44:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.899+0000 7f8d8a000640 1 -- 172.21.15.28:0/2558556868 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d84010760 con 0x7f8d8c105e30 2024-04-23T14:44:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.900+0000 7f8d94400640 1 -- 172.21.15.28:0/2558556868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d8c1a2960 con 0x7f8d8c105e30 2024-04-23T14:44:02.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.901+0000 7f8d94400640 1 -- 172.21.15.28:0/2558556868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d8c10b620 con 0x7f8d8c105e30 2024-04-23T14:44:02.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.903+0000 7f8d8a000640 1 -- 172.21.15.28:0/2558556868 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138458+0+0 (secure 0 0 0) 0x7f8d84005020 con 0x7f8d8c105e30 2024-04-23T14:44:02.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.904+0000 7f8d8a000640 1 --2- 172.21.15.28:0/2558556868 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f8d740a39b0 0x7f8d740a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:02.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.904+0000 7f8d8a000640 1 -- 172.21.15.28:0/2558556868 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f8d840d07e0 con 0x7f8d8c105e30 2024-04-23T14:44:02.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.904+0000 7f8d91a00640 1 --2- 172.21.15.28:0/2558556868 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f8d740a39b0 0x7f8d740a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:02.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.905+0000 7f8d91a00640 1 --2- 172.21.15.28:0/2558556868 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f8d740a39b0 0x7f8d740a5e70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8d800095f0 tx=0x7f8d80005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:02.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:02.913+0000 7f8d8a000640 1 -- 172.21.15.28:0/2558556868 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8d8409d360 con 0x7f8d8c105e30 2024-04-23T14:44:03.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.118+0000 7f8d94400640 1 -- 172.21.15.28:0/2558556868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8d8c062c00 con 0x7f8d8c105e30 2024-04-23T14:44:03.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.119+0000 7f8d8a000640 1 -- 172.21.15.28:0/2558556868 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f8d840bbe20 con 0x7f8d8c105e30 2024-04-23T14:44:03.119 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-23T14:44:03.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.122+0000 7f8d94400640 1 -- 172.21.15.28:0/2558556868 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f8d740a39b0 msgr2=0x7f8d740a5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:03.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.122+0000 7f8d94400640 1 --2- 172.21.15.28:0/2558556868 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f8d740a39b0 0x7f8d740a5e70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8d800095f0 tx=0x7f8d80005f90 comp rx=0 tx=0).stop 2024-04-23T14:44:03.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.122+0000 7f8d94400640 1 -- 172.21.15.28:0/2558556868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d8c105e30 msgr2=0x7f8d8c1a1ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:03.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.122+0000 7f8d94400640 1 --2- 172.21.15.28:0/2558556868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d8c105e30 0x7f8d8c1a1ae0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8d8400eb10 tx=0x7f8d8400c2d0 comp rx=0 tx=0).stop 2024-04-23T14:44:03.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.122+0000 7f8d94400640 1 -- 172.21.15.28:0/2558556868 shutdown_connections 2024-04-23T14:44:03.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.122+0000 7f8d94400640 1 --2- 172.21.15.28:0/2558556868 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f8d740a39b0 0x7f8d740a5e70 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:03.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.122+0000 7f8d94400640 1 --2- 172.21.15.28:0/2558556868 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d8c105470 0x7f8d8c19aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:03.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.122+0000 7f8d94400640 1 --2- 172.21.15.28:0/2558556868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d8c105e30 0x7f8d8c1a1ae0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:03.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.122+0000 7f8d94400640 1 --2- 172.21.15.28:0/2558556868 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d8c104280 0x7f8d8c19a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:03.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.122+0000 7f8d94400640 1 -- 172.21.15.28:0/2558556868 >> 172.21.15.28:0/2558556868 conn(0x7f8d8c0ff7f0 msgr2=0x7f8d8c1087f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:03.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.122+0000 7f8d94400640 1 -- 172.21.15.28:0/2558556868 shutdown_connections 2024-04-23T14:44:03.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.122+0000 7f8d94400640 1 -- 172.21.15.28:0/2558556868 wait complete. 2024-04-23T14:44:03.140 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T14:44:03.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.332+0000 7fcd44200640 1 -- 172.21.15.28:0/2689559136 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd3c077c00 msgr2=0x7fcd3c078060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:03.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.332+0000 7fcd44200640 1 --2- 172.21.15.28:0/2689559136 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd3c077c00 0x7fcd3c078060 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fcd28009900 tx=0x7fcd2802a410 comp rx=0 tx=0).stop 2024-04-23T14:44:03.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.332+0000 7fcd44200640 1 -- 172.21.15.28:0/2689559136 shutdown_connections 2024-04-23T14:44:03.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.332+0000 7fcd44200640 1 --2- 172.21.15.28:0/2689559136 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd3c077c00 0x7fcd3c078060 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:03.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.332+0000 7fcd44200640 1 --2- 172.21.15.28:0/2689559136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd3c0785a0 0x7fcd3c078e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:03.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.332+0000 7fcd44200640 1 --2- 172.21.15.28:0/2689559136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd3c0769c0 0x7fcd3c076de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:03.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.332+0000 7fcd44200640 1 -- 172.21.15.28:0/2689559136 >> 172.21.15.28:0/2689559136 conn(0x7fcd3c0ff7f0 msgr2=0x7fcd3c101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:03.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.332+0000 7fcd44200640 1 -- 172.21.15.28:0/2689559136 shutdown_connections 2024-04-23T14:44:03.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.332+0000 7fcd44200640 1 -- 172.21.15.28:0/2689559136 wait complete. 2024-04-23T14:44:03.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.333+0000 7fcd44200640 1 Processor -- start 2024-04-23T14:44:03.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.333+0000 7fcd44200640 1 -- start start 2024-04-23T14:44:03.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.333+0000 7fcd44200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd3c0769c0 0x7fcd3c19e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:03.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.333+0000 7fcd44200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd3c077c00 0x7fcd3c19ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:03.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.333+0000 7fcd44200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd3c0785a0 0x7fcd3c1a5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:03.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.333+0000 7fcd44200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcd3c114350 con 0x7fcd3c0769c0 2024-04-23T14:44:03.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.333+0000 7fcd44200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcd3c1141d0 con 0x7fcd3c077c00 2024-04-23T14:44:03.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.333+0000 7fcd44200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcd3c1144d0 con 0x7fcd3c0785a0 2024-04-23T14:44:03.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.334+0000 7fcd40e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd3c077c00 0x7fcd3c19ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:03.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.334+0000 7fcd40e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd3c077c00 0x7fcd3c19ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:37014/0 (socket says 172.21.15.28:37014) 2024-04-23T14:44:03.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.334+0000 7fcd40e00640 1 -- 172.21.15.28:0/4217275573 learned_addr learned my addr 172.21.15.28:0/4217275573 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:03.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.334+0000 7fcd41800640 1 --2- 172.21.15.28:0/4217275573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd3c0769c0 0x7fcd3c19e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:03.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.334+0000 7fcd42200640 1 --2- 172.21.15.28:0/4217275573 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd3c0785a0 0x7fcd3c1a5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:03.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.334+0000 7fcd40e00640 1 -- 172.21.15.28:0/4217275573 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd3c0785a0 msgr2=0x7fcd3c1a5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:03.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.334+0000 7fcd40e00640 1 --2- 172.21.15.28:0/4217275573 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd3c0785a0 0x7fcd3c1a5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:03.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.334+0000 7fcd40e00640 1 -- 172.21.15.28:0/4217275573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd3c0769c0 msgr2=0x7fcd3c19e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:03.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.334+0000 7fcd40e00640 1 --2- 172.21.15.28:0/4217275573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd3c0769c0 0x7fcd3c19e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:03.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.334+0000 7fcd40e00640 1 -- 172.21.15.28:0/4217275573 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd3c1a6570 con 0x7fcd3c077c00 2024-04-23T14:44:03.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.334+0000 7fcd40e00640 1 --2- 172.21.15.28:0/4217275573 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd3c077c00 0x7fcd3c19ee90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcd28031440 tx=0x7fcd28032ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:03.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.335+0000 7fcd3a000640 1 -- 172.21.15.28:0/4217275573 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd2803f030 con 0x7fcd3c077c00 2024-04-23T14:44:03.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.335+0000 7fcd3a000640 1 -- 172.21.15.28:0/4217275573 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd28003ce0 con 0x7fcd3c077c00 2024-04-23T14:44:03.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.335+0000 7fcd3a000640 1 -- 172.21.15.28:0/4217275573 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd28035070 con 0x7fcd3c077c00 2024-04-23T14:44:03.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.335+0000 7fcd44200640 1 -- 172.21.15.28:0/4217275573 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd3c1a6800 con 0x7fcd3c077c00 2024-04-23T14:44:03.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.335+0000 7fcd44200640 1 -- 172.21.15.28:0/4217275573 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd3c1a6ca0 con 0x7fcd3c077c00 2024-04-23T14:44:03.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.336+0000 7fcd44200640 1 -- 172.21.15.28:0/4217275573 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd3c110bf0 con 0x7fcd3c077c00 2024-04-23T14:44:03.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.338+0000 7fcd3a000640 1 -- 172.21.15.28:0/4217275573 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 23) ==== 138458+0+0 (secure 0 0 0) 0x7fcd28042030 con 0x7fcd3c077c00 2024-04-23T14:44:03.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.339+0000 7fcd3a000640 1 --2- 172.21.15.28:0/4217275573 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fcd240a3a30 0x7fcd240a5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:03.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.339+0000 7fcd3a000640 1 -- 172.21.15.28:0/4217275573 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fcd280f0550 con 0x7fcd3c077c00 2024-04-23T14:44:03.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.339+0000 7fcd41800640 1 --2- 172.21.15.28:0/4217275573 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fcd240a3a30 0x7fcd240a5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:03.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.340+0000 7fcd41800640 1 --2- 172.21.15.28:0/4217275573 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fcd240a3a30 0x7fcd240a5ef0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcd300095f0 tx=0x7fcd30005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:03.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.343+0000 7fcd3a000640 1 -- 172.21.15.28:0/4217275573 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcd280bca80 con 0x7fcd3c077c00 2024-04-23T14:44:03.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.560+0000 7fcd44200640 1 -- 172.21.15.28:0/4217275573 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fcd3c062c00 con 0x7fcd3c077c00 2024-04-23T14:44:03.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.565+0000 7fcd3a000640 1 -- 172.21.15.28:0/4217275573 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7fcd280c0d80 con 0x7fcd3c077c00 2024-04-23T14:44:03.565 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-23T14:44:03.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.568+0000 7fcd44200640 1 -- 172.21.15.28:0/4217275573 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fcd240a3a30 msgr2=0x7fcd240a5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:03.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.568+0000 7fcd44200640 1 --2- 172.21.15.28:0/4217275573 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fcd240a3a30 0x7fcd240a5ef0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcd300095f0 tx=0x7fcd30005f90 comp rx=0 tx=0).stop 2024-04-23T14:44:03.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.568+0000 7fcd44200640 1 -- 172.21.15.28:0/4217275573 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd3c077c00 msgr2=0x7fcd3c19ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:03.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.568+0000 7fcd44200640 1 --2- 172.21.15.28:0/4217275573 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd3c077c00 0x7fcd3c19ee90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcd28031440 tx=0x7fcd28032ce0 comp rx=0 tx=0).stop 2024-04-23T14:44:03.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.568+0000 7fcd44200640 1 -- 172.21.15.28:0/4217275573 shutdown_connections 2024-04-23T14:44:03.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.568+0000 7fcd44200640 1 --2- 172.21.15.28:0/4217275573 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fcd240a3a30 0x7fcd240a5ef0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:03.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.568+0000 7fcd44200640 1 --2- 172.21.15.28:0/4217275573 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd3c077c00 0x7fcd3c19ee90 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:03.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.568+0000 7fcd44200640 1 --2- 172.21.15.28:0/4217275573 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd3c0785a0 0x7fcd3c1a5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:03.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.568+0000 7fcd44200640 1 --2- 172.21.15.28:0/4217275573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd3c0769c0 0x7fcd3c19e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:03.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.568+0000 7fcd44200640 1 -- 172.21.15.28:0/4217275573 >> 172.21.15.28:0/4217275573 conn(0x7fcd3c0ff7f0 msgr2=0x7fcd3c1012f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:03.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.568+0000 7fcd44200640 1 -- 172.21.15.28:0/4217275573 shutdown_connections 2024-04-23T14:44:03.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.568+0000 7fcd44200640 1 -- 172.21.15.28:0/4217275573 wait complete. 2024-04-23T14:44:03.593 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T14:44:03.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.776+0000 7fd0f9000640 1 -- 172.21.15.28:0/1769449451 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0f4105cf0 msgr2=0x7fd0f410c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:03.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.776+0000 7fd0f9000640 1 --2- 172.21.15.28:0/1769449451 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0f4105cf0 0x7fd0f410c580 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec009ca0 tx=0x7fd0ec02a8c0 comp rx=0 tx=0).stop 2024-04-23T14:44:03.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.776+0000 7fd0f9000640 1 -- 172.21.15.28:0/1769449451 shutdown_connections 2024-04-23T14:44:03.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.776+0000 7fd0f9000640 1 --2- 172.21.15.28:0/1769449451 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0f4105cf0 0x7fd0f410c580 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:03.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.776+0000 7fd0f9000640 1 --2- 172.21.15.28:0/1769449451 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0f4105330 0x7fd0f41057b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:03.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.776+0000 7fd0f9000640 1 --2- 172.21.15.28:0/1769449451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0f4104140 0x7fd0f4104560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:03.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.776+0000 7fd0f9000640 1 -- 172.21.15.28:0/1769449451 >> 172.21.15.28:0/1769449451 conn(0x7fd0f40ff6b0 msgr2=0x7fd0f4101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:03.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.777+0000 7fd0f9000640 1 -- 172.21.15.28:0/1769449451 shutdown_connections 2024-04-23T14:44:03.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.777+0000 7fd0f9000640 1 -- 172.21.15.28:0/1769449451 wait complete. 2024-04-23T14:44:03.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.777+0000 7fd0f9000640 1 Processor -- start 2024-04-23T14:44:03.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.778+0000 7fd0f9000640 1 -- start start 2024-04-23T14:44:03.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.778+0000 7fd0f9000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0f4104140 0x7fd0f419a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:03.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.778+0000 7fd0f9000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0f4105330 0x7fd0f419a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:03.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.778+0000 7fd0f9000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0f4105cf0 0x7fd0f41a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:03.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.778+0000 7fd0f9000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f410fda0 con 0x7fd0f4105330 2024-04-23T14:44:03.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.778+0000 7fd0f9000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f410fc20 con 0x7fd0f4104140 2024-04-23T14:44:03.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.778+0000 7fd0f9000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f410ff20 con 0x7fd0f4105cf0 2024-04-23T14:44:03.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.778+0000 7fd0f3400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0f4104140 0x7fd0f419a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:03.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.778+0000 7fd0f3400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0f4104140 0x7fd0f419a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:37050/0 (socket says 172.21.15.28:37050) 2024-04-23T14:44:03.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.778+0000 7fd0f3400640 1 -- 172.21.15.28:0/518493619 learned_addr learned my addr 172.21.15.28:0/518493619 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:03.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.778+0000 7fd0f3e00640 1 --2- 172.21.15.28:0/518493619 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0f4105cf0 0x7fd0f41a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:03.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.778+0000 7fd0ebe00640 1 --2- 172.21.15.28:0/518493619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0f4105330 0x7fd0f419a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:03.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.779+0000 7fd0f3400640 1 -- 172.21.15.28:0/518493619 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0f4105cf0 msgr2=0x7fd0f41a19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:03.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.779+0000 7fd0f3400640 1 --2- 172.21.15.28:0/518493619 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0f4105cf0 0x7fd0f41a19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:03.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.779+0000 7fd0f3400640 1 -- 172.21.15.28:0/518493619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0f4105330 msgr2=0x7fd0f419a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:03.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.779+0000 7fd0f3400640 1 --2- 172.21.15.28:0/518493619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0f4105330 0x7fd0f419a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:03.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.779+0000 7fd0f3400640 1 -- 172.21.15.28:0/518493619 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0f41a2020 con 0x7fd0f4104140 2024-04-23T14:44:03.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.779+0000 7fd0ebe00640 1 --2- 172.21.15.28:0/518493619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0f4105330 0x7fd0f419a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:44:03.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.779+0000 7fd0f3400640 1 --2- 172.21.15.28:0/518493619 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0f4104140 0x7fd0f419a400 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd0e400e9c0 tx=0x7fd0e400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:03.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.779+0000 7fd0f0c00640 1 -- 172.21.15.28:0/518493619 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e400cde0 con 0x7fd0f4104140 2024-04-23T14:44:03.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.779+0000 7fd0f9000640 1 -- 172.21.15.28:0/518493619 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0f41a2310 con 0x7fd0f4104140 2024-04-23T14:44:03.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.779+0000 7fd0f0c00640 1 -- 172.21.15.28:0/518493619 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0e4010910 con 0x7fd0f4104140 2024-04-23T14:44:03.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.779+0000 7fd0f0c00640 1 -- 172.21.15.28:0/518493619 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4018a10 con 0x7fd0f4104140 2024-04-23T14:44:03.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.779+0000 7fd0f9000640 1 -- 172.21.15.28:0/518493619 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0f41a2860 con 0x7fd0f4104140 2024-04-23T14:44:03.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.780+0000 7fd0f9000640 1 -- 172.21.15.28:0/518493619 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0f410b400 con 0x7fd0f4104140 2024-04-23T14:44:03.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.782+0000 7fd0f0c00640 1 -- 172.21.15.28:0/518493619 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) ==== 138766+0+0 (secure 0 0 0) 0x7fd0e401f020 con 0x7fd0f4104140 2024-04-23T14:44:03.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.784+0000 7fd0f0c00640 1 --2- 172.21.15.28:0/518493619 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd0d80a3d20 0x7fd0d80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:03.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.784+0000 7fd0f0c00640 1 -- 172.21.15.28:0/518493619 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fd0e40d0c50 con 0x7fd0f4104140 2024-04-23T14:44:03.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.784+0000 7fd0ebe00640 1 --2- 172.21.15.28:0/518493619 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd0d80a3d20 0x7fd0d80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:03.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.785+0000 7fd0ebe00640 1 --2- 172.21.15.28:0/518493619 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd0d80a3d20 0x7fd0d80a61e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd0f419b910 tx=0x7fd0dc005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:03.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.786+0000 7fd0f0c00640 1 -- 172.21.15.28:0/518493619 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd0e409d330 con 0x7fd0f4104140 2024-04-23T14:44:03.867 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:03 smithi028 ceph-mon[27810]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:44:03.867 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:03 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:03.867 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:03 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:03.867 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:03 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2156419162' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:44:03.867 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:03 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2558556868' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:44:03.867 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:03 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:03.867 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:03 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/4217275573' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T14:44:03.868 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:03 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:03.943 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:03 smithi038 ceph-mon[32348]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:44:03.943 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:03 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:03.944 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:03 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:03.944 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:03 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2156419162' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:44:03.944 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:03 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2558556868' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:44:03.944 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:03 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:03.944 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:03 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/4217275573' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T14:44:03.944 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:03 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:03.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.988+0000 7fd0f9000640 1 -- 172.21.15.28:0/518493619 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd0f4062c00 con 0x7fd0f4104140 2024-04-23T14:44:03.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.989+0000 7fd0f0c00640 1 -- 172.21.15.28:0/518493619 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7fd0e409cc30 con 0x7fd0f4104140 2024-04-23T14:44:03.989 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-23T14:44:03.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.991+0000 7fd0f9000640 1 -- 172.21.15.28:0/518493619 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd0d80a3d20 msgr2=0x7fd0d80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:03.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.991+0000 7fd0f9000640 1 --2- 172.21.15.28:0/518493619 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd0d80a3d20 0x7fd0d80a61e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd0f419b910 tx=0x7fd0dc005f90 comp rx=0 tx=0).stop 2024-04-23T14:44:03.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.991+0000 7fd0f9000640 1 -- 172.21.15.28:0/518493619 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0f4104140 msgr2=0x7fd0f419a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:03.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.991+0000 7fd0f9000640 1 --2- 172.21.15.28:0/518493619 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0f4104140 0x7fd0f419a400 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd0e400e9c0 tx=0x7fd0e400ee80 comp rx=0 tx=0).stop 2024-04-23T14:44:03.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.991+0000 7fd0f9000640 1 -- 172.21.15.28:0/518493619 shutdown_connections 2024-04-23T14:44:03.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.991+0000 7fd0f9000640 1 --2- 172.21.15.28:0/518493619 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd0d80a3d20 0x7fd0d80a61e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:03.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.991+0000 7fd0f9000640 1 --2- 172.21.15.28:0/518493619 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0f4105cf0 0x7fd0f41a19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:03.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.991+0000 7fd0f9000640 1 --2- 172.21.15.28:0/518493619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0f4105330 0x7fd0f419a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:03.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.991+0000 7fd0f9000640 1 --2- 172.21.15.28:0/518493619 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0f4104140 0x7fd0f419a400 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:03.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.991+0000 7fd0f9000640 1 -- 172.21.15.28:0/518493619 >> 172.21.15.28:0/518493619 conn(0x7fd0f40ff6b0 msgr2=0x7fd0f41085d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:03.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.992+0000 7fd0f9000640 1 -- 172.21.15.28:0/518493619 shutdown_connections 2024-04-23T14:44:03.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:03.992+0000 7fd0f9000640 1 -- 172.21.15.28:0/518493619 wait complete. 2024-04-23T14:44:04.011 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-04-23T14:44:04.011 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:44:04.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:03 smithi105 ceph-mon[32555]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:44:04.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:03 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:04.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:03 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:04.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:03 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2156419162' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:44:04.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:03 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2558556868' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:44:04.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:03 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:04.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:03 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/4217275573' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T14:44:04.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:03 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:04.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.197+0000 7f4254000640 1 -- 172.21.15.28:0/472378674 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f424c109850 msgr2=0x7f424c109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:04.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.197+0000 7f4254000640 1 --2- 172.21.15.28:0/472378674 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f424c109850 0x7f424c109cd0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4238009900 tx=0x7f423802a410 comp rx=0 tx=0).stop 2024-04-23T14:44:04.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.197+0000 7f4254000640 1 -- 172.21.15.28:0/472378674 shutdown_connections 2024-04-23T14:44:04.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.197+0000 7f4254000640 1 --2- 172.21.15.28:0/472378674 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f424c109850 0x7f424c109cd0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:04.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.197+0000 7f4254000640 1 --2- 172.21.15.28:0/472378674 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f424c10a210 0x7f424c110aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:04.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.197+0000 7f4254000640 1 --2- 172.21.15.28:0/472378674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f424c108660 0x7f424c108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:04.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.197+0000 7f4254000640 1 -- 172.21.15.28:0/472378674 >> 172.21.15.28:0/472378674 conn(0x7f424c077620 msgr2=0x7f424c079a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:04.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.198+0000 7f4254000640 1 -- 172.21.15.28:0/472378674 shutdown_connections 2024-04-23T14:44:04.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.198+0000 7f4254000640 1 -- 172.21.15.28:0/472378674 wait complete. 2024-04-23T14:44:04.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.199+0000 7f4254000640 1 Processor -- start 2024-04-23T14:44:04.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.199+0000 7f4254000640 1 -- start start 2024-04-23T14:44:04.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.199+0000 7f4254000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f424c108660 0x7f424c19e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:04.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.199+0000 7f4254000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f424c109850 0x7f424c19eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:04.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.199+0000 7f4254000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f424c10a210 0x7f424c1a5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:04.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.199+0000 7f4254000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f424c114310 con 0x7f424c109850 2024-04-23T14:44:04.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.199+0000 7f4254000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f424c114190 con 0x7f424c10a210 2024-04-23T14:44:04.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.199+0000 7f4254000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f424c114490 con 0x7f424c108660 2024-04-23T14:44:04.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.200+0000 7f4251600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f424c108660 0x7f424c19e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:04.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.200+0000 7f4251600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f424c108660 0x7f424c19e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:44230/0 (socket says 172.21.15.28:44230) 2024-04-23T14:44:04.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.200+0000 7f4251600640 1 -- 172.21.15.28:0/1357640533 learned_addr learned my addr 172.21.15.28:0/1357640533 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:04.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.200+0000 7f4250c00640 1 --2- 172.21.15.28:0/1357640533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f424c109850 0x7f424c19eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:04.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.200+0000 7f4252000640 1 --2- 172.21.15.28:0/1357640533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f424c10a210 0x7f424c1a5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:04.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.200+0000 7f4251600640 1 -- 172.21.15.28:0/1357640533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f424c10a210 msgr2=0x7f424c1a5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:04.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.200+0000 7f4251600640 1 --2- 172.21.15.28:0/1357640533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f424c10a210 0x7f424c1a5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:04.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.200+0000 7f4251600640 1 -- 172.21.15.28:0/1357640533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f424c109850 msgr2=0x7f424c19eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:04.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.200+0000 7f4251600640 1 --2- 172.21.15.28:0/1357640533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f424c109850 0x7f424c19eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:04.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.200+0000 7f4251600640 1 -- 172.21.15.28:0/1357640533 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f424c1a65d0 con 0x7f424c108660 2024-04-23T14:44:04.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.200+0000 7f4251600640 1 --2- 172.21.15.28:0/1357640533 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f424c108660 0x7f424c19e9b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f424000ebc0 tx=0x7f424000c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:04.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.200+0000 7f424a000640 1 -- 172.21.15.28:0/1357640533 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4240003970 con 0x7f424c108660 2024-04-23T14:44:04.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.201+0000 7f424a000640 1 -- 172.21.15.28:0/1357640533 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4240004030 con 0x7f424c108660 2024-04-23T14:44:04.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.201+0000 7f4254000640 1 -- 172.21.15.28:0/1357640533 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f424c1a68c0 con 0x7f424c108660 2024-04-23T14:44:04.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.201+0000 7f424a000640 1 -- 172.21.15.28:0/1357640533 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42400197b0 con 0x7f424c108660 2024-04-23T14:44:04.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.201+0000 7f4254000640 1 -- 172.21.15.28:0/1357640533 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f424c1a6de0 con 0x7f424c108660 2024-04-23T14:44:04.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.202+0000 7f4254000640 1 -- 172.21.15.28:0/1357640533 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f424c10f980 con 0x7f424c108660 2024-04-23T14:44:04.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.204+0000 7f424a000640 1 -- 172.21.15.28:0/1357640533 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138766+0+0 (secure 0 0 0) 0x7f42400041e0 con 0x7f424c108660 2024-04-23T14:44:04.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.205+0000 7f424a000640 1 --2- 172.21.15.28:0/1357640533 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f42340a3d20 0x7f42340a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:04.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.205+0000 7f424a000640 1 -- 172.21.15.28:0/1357640533 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f42400d1140 con 0x7f424c108660 2024-04-23T14:44:04.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.205+0000 7f4250c00640 1 --2- 172.21.15.28:0/1357640533 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f42340a3d20 0x7f42340a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:04.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.206+0000 7f4250c00640 1 --2- 172.21.15.28:0/1357640533 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f42340a3d20 0x7f42340a61e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4238031440 tx=0x7f42380315b0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:04.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.209+0000 7f424a000640 1 -- 172.21.15.28:0/1357640533 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f424009d790 con 0x7f424c108660 2024-04-23T14:44:04.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.465+0000 7f4254000640 1 -- 172.21.15.28:0/1357640533 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f424c02d620 con 0x7f424c108660 2024-04-23T14:44:04.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.466+0000 7f424a000640 1 -- 172.21.15.28:0/1357640533 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7f424009d090 con 0x7f424c108660 2024-04-23T14:44:04.466 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:44:04.466 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:44:04.466 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:44:04.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.468+0000 7f4254000640 1 -- 172.21.15.28:0/1357640533 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f42340a3d20 msgr2=0x7f42340a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:04.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.468+0000 7f4254000640 1 --2- 172.21.15.28:0/1357640533 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f42340a3d20 0x7f42340a61e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4238031440 tx=0x7f42380315b0 comp rx=0 tx=0).stop 2024-04-23T14:44:04.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.468+0000 7f4254000640 1 -- 172.21.15.28:0/1357640533 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f424c108660 msgr2=0x7f424c19e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.468+0000 7f4254000640 1 --2- 172.21.15.28:0/1357640533 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f424c108660 0x7f424c19e9b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f424000ebc0 tx=0x7f424000c2d0 comp rx=0 tx=0).stop 2024-04-23T14:44:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.468+0000 7f4254000640 1 -- 172.21.15.28:0/1357640533 shutdown_connections 2024-04-23T14:44:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.468+0000 7f4254000640 1 --2- 172.21.15.28:0/1357640533 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f42340a3d20 0x7f42340a61e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.468+0000 7f4254000640 1 --2- 172.21.15.28:0/1357640533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f424c10a210 0x7f424c1a5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.468+0000 7f4254000640 1 --2- 172.21.15.28:0/1357640533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f424c109850 0x7f424c19eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.468+0000 7f4254000640 1 --2- 172.21.15.28:0/1357640533 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f424c108660 0x7f424c19e9b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.468+0000 7f4254000640 1 -- 172.21.15.28:0/1357640533 >> 172.21.15.28:0/1357640533 conn(0x7f424c077620 msgr2=0x7f424c10cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.468+0000 7f4254000640 1 -- 172.21.15.28:0/1357640533 shutdown_connections 2024-04-23T14:44:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.468+0000 7f4254000640 1 -- 172.21.15.28:0/1357640533 wait complete. 2024-04-23T14:44:04.487 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-04-23T14:44:04.487 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-04-23T14:44:04.487 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T14:44:04.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.673+0000 7f3723600640 1 -- 172.21.15.28:0/3145353793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f371c103c10 msgr2=0x7f371c10a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:04.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.673+0000 7f3723600640 1 --2- 172.21.15.28:0/3145353793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f371c103c10 0x7f371c10a4a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3714009ca0 tx=0x7f371402a8c0 comp rx=0 tx=0).stop 2024-04-23T14:44:04.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.673+0000 7f3723600640 1 -- 172.21.15.28:0/3145353793 shutdown_connections 2024-04-23T14:44:04.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.673+0000 7f3723600640 1 --2- 172.21.15.28:0/3145353793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f371c103250 0x7f371c1036d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:04.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.673+0000 7f3723600640 1 --2- 172.21.15.28:0/3145353793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f371c103c10 0x7f371c10a4a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:04.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.673+0000 7f3723600640 1 --2- 172.21.15.28:0/3145353793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f371c102060 0x7f371c102480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:04.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.673+0000 7f3723600640 1 -- 172.21.15.28:0/3145353793 >> 172.21.15.28:0/3145353793 conn(0x7f371c0fd5b0 msgr2=0x7f371c0ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:04.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.673+0000 7f3723600640 1 -- 172.21.15.28:0/3145353793 shutdown_connections 2024-04-23T14:44:04.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.673+0000 7f3723600640 1 -- 172.21.15.28:0/3145353793 wait complete. 2024-04-23T14:44:04.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.675+0000 7f3723600640 1 Processor -- start 2024-04-23T14:44:04.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.675+0000 7f3723600640 1 -- start start 2024-04-23T14:44:04.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.675+0000 7f3723600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f371c102060 0x7f371c075db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:04.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.675+0000 7f3723600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f371c103250 0x7f371c0743f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:04.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.675+0000 7f3723600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f371c103c10 0x7f371c074930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:04.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.676+0000 7f3723600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f371c10dd00 con 0x7f371c102060 2024-04-23T14:44:04.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.676+0000 7f3723600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f371c10db80 con 0x7f371c103250 2024-04-23T14:44:04.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.676+0000 7f3723600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f371c10de80 con 0x7f371c103c10 2024-04-23T14:44:04.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.676+0000 7f3720c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f371c102060 0x7f371c075db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:04.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.676+0000 7f3720c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f371c102060 0x7f371c075db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36308/0 (socket says 172.21.15.28:36308) 2024-04-23T14:44:04.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.676+0000 7f3720c00640 1 -- 172.21.15.28:0/1381911998 learned_addr learned my addr 172.21.15.28:0/1381911998 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:04.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.676+0000 7f3721600640 1 --2- 172.21.15.28:0/1381911998 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f371c103c10 0x7f371c074930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:04.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.676+0000 7f3713e00640 1 --2- 172.21.15.28:0/1381911998 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f371c103250 0x7f371c0743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:04.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.676+0000 7f3720c00640 1 -- 172.21.15.28:0/1381911998 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f371c103c10 msgr2=0x7f371c074930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:04.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.676+0000 7f3720c00640 1 --2- 172.21.15.28:0/1381911998 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f371c103c10 0x7f371c074930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:04.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.676+0000 7f3720c00640 1 -- 172.21.15.28:0/1381911998 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f371c103250 msgr2=0x7f371c0743f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:04.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.676+0000 7f3720c00640 1 --2- 172.21.15.28:0/1381911998 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f371c103250 0x7f371c0743f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:04.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.677+0000 7f3720c00640 1 -- 172.21.15.28:0/1381911998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f371c0751c0 con 0x7f371c102060 2024-04-23T14:44:04.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.677+0000 7f3713e00640 1 --2- 172.21.15.28:0/1381911998 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f371c103250 0x7f371c0743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:44:04.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.677+0000 7f3720c00640 1 --2- 172.21.15.28:0/1381911998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f371c102060 0x7f371c075db0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f370c00d950 tx=0x7f370c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:04.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.682+0000 7f371a000640 1 -- 172.21.15.28:0/1381911998 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f370c017020 con 0x7f371c102060 2024-04-23T14:44:04.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.682+0000 7f371a000640 1 -- 172.21.15.28:0/1381911998 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f370c009c30 con 0x7f371c102060 2024-04-23T14:44:04.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.682+0000 7f371a000640 1 -- 172.21.15.28:0/1381911998 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f370c00f720 con 0x7f371c102060 2024-04-23T14:44:04.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.682+0000 7f3723600640 1 -- 172.21.15.28:0/1381911998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f371c1a23e0 con 0x7f371c102060 2024-04-23T14:44:04.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.682+0000 7f3723600640 1 -- 172.21.15.28:0/1381911998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f371c1a2980 con 0x7f371c102060 2024-04-23T14:44:04.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.684+0000 7f3723600640 1 -- 172.21.15.28:0/1381911998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f371c109320 con 0x7f371c102060 2024-04-23T14:44:04.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.686+0000 7f371a000640 1 -- 172.21.15.28:0/1381911998 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138766+0+0 (secure 0 0 0) 0x7f370c01a020 con 0x7f371c102060 2024-04-23T14:44:04.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.688+0000 7f371a000640 1 --2- 172.21.15.28:0/1381911998 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f37000a3da0 0x7f37000a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:04.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.689+0000 7f371a000640 1 -- 172.21.15.28:0/1381911998 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f370c0d06d0 con 0x7f371c102060 2024-04-23T14:44:04.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.689+0000 7f3713e00640 1 --2- 172.21.15.28:0/1381911998 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f37000a3da0 0x7f37000a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:04.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.690+0000 7f3713e00640 1 --2- 172.21.15.28:0/1381911998 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f37000a3da0 0x7f37000a6260 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f371c075b20 tx=0x7f3708005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:04.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.692+0000 7f371a000640 1 -- 172.21.15.28:0/1381911998 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f370c021020 con 0x7f371c102060 2024-04-23T14:44:04.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:04 smithi028 ceph-mon[27810]: [23/Apr/2024:14:44:02] ENGINE Bus STARTING 2024-04-23T14:44:04.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:04 smithi028 ceph-mon[27810]: [23/Apr/2024:14:44:02] ENGINE Serving on http://172.21.15.28:8765 2024-04-23T14:44:04.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:04 smithi028 ceph-mon[27810]: [23/Apr/2024:14:44:03] ENGINE Serving on https://172.21.15.28:7150 2024-04-23T14:44:04.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:04 smithi028 ceph-mon[27810]: [23/Apr/2024:14:44:03] ENGINE Bus STARTED 2024-04-23T14:44:04.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:04 smithi028 ceph-mon[27810]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:44:04.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:04 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:04.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:04 smithi028 ceph-mon[27810]: mgrmap e24: x(active, since 2s), standbys: z, y 2024-04-23T14:44:04.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:04 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:04.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:04 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/518493619' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T14:44:04.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:04 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1357640533' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:44:04.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:04 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:04.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.901+0000 7f3723600640 1 -- 172.21.15.28:0/1381911998 --> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f371c062440 con 0x7f37000a3da0 2024-04-23T14:44:04.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.912+0000 7f371a000640 1 -- 172.21.15.28:0/1381911998 <== mgr.25153 v2:172.21.15.28:6800/1615546338 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257706 (secure 0 0 0) 0x7f371c062440 con 0x7f37000a3da0 2024-04-23T14:44:04.916 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:44:04.918 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-23T14:44:04.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.917+0000 7f3723600640 1 -- 172.21.15.28:0/1381911998 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f37000a3da0 msgr2=0x7f37000a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:04.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.917+0000 7f3723600640 1 --2- 172.21.15.28:0/1381911998 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f37000a3da0 0x7f37000a6260 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f371c075b20 tx=0x7f3708005f90 comp rx=0 tx=0).stop 2024-04-23T14:44:04.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.917+0000 7f3723600640 1 -- 172.21.15.28:0/1381911998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f371c102060 msgr2=0x7f371c075db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:04.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.917+0000 7f3723600640 1 --2- 172.21.15.28:0/1381911998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f371c102060 0x7f371c075db0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f370c00d950 tx=0x7f370c00de10 comp rx=0 tx=0).stop 2024-04-23T14:44:04.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.918+0000 7f3723600640 1 -- 172.21.15.28:0/1381911998 shutdown_connections 2024-04-23T14:44:04.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.918+0000 7f3723600640 1 --2- 172.21.15.28:0/1381911998 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f37000a3da0 0x7f37000a6260 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:04.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.918+0000 7f3723600640 1 --2- 172.21.15.28:0/1381911998 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f371c103250 0x7f371c0743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:04.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.918+0000 7f3723600640 1 --2- 172.21.15.28:0/1381911998 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f371c103c10 0x7f371c074930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:04.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.918+0000 7f3723600640 1 --2- 172.21.15.28:0/1381911998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f371c102060 0x7f371c075db0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:04.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.918+0000 7f3723600640 1 -- 172.21.15.28:0/1381911998 >> 172.21.15.28:0/1381911998 conn(0x7f371c0fd5b0 msgr2=0x7f371c1064f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:04.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.918+0000 7f3723600640 1 -- 172.21.15.28:0/1381911998 shutdown_connections 2024-04-23T14:44:04.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:04.918+0000 7f3723600640 1 -- 172.21.15.28:0/1381911998 wait complete. 2024-04-23T14:44:04.936 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:04 smithi038 ceph-mon[32348]: [23/Apr/2024:14:44:02] ENGINE Bus STARTING 2024-04-23T14:44:04.936 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:04 smithi038 ceph-mon[32348]: [23/Apr/2024:14:44:02] ENGINE Serving on http://172.21.15.28:8765 2024-04-23T14:44:04.936 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:04 smithi038 ceph-mon[32348]: [23/Apr/2024:14:44:03] ENGINE Serving on https://172.21.15.28:7150 2024-04-23T14:44:04.936 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:04 smithi038 ceph-mon[32348]: [23/Apr/2024:14:44:03] ENGINE Bus STARTED 2024-04-23T14:44:04.936 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:04 smithi038 ceph-mon[32348]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:44:04.936 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:04 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:04.936 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:04 smithi038 ceph-mon[32348]: mgrmap e24: x(active, since 2s), standbys: z, y 2024-04-23T14:44:04.937 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:04 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:04.937 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:04 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/518493619' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T14:44:04.937 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:04 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1357640533' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:44:04.937 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:04 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:04.942 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":4,"stamp":"2024-04-23T14:44:03.321881+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1184826,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":522,"num_read_kb":409,"num_write":383,"num_write_kb":4345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":787,"ondisk_log_size":787,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":380880,"kb_used_data":14808,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124560944,"statfs":{"total":1151940427776,"available":1151550406656,"internally_reserved":0,"allocated":15163392,"data_stored":9052616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.991286"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.317096+0000","last_change":"2024-04-23T14:41:40.033061+0000","last_active":"2024-04-23T14:44:01.317096+0000","last_peered":"2024-04-23T14:44:01.317096+0000","last_clean":"2024-04-23T14:44:01.317096+0000","last_became_active":"2024-04-23T14:41:40.032750+0000","last_became_peered":"2024-04-23T14:41:40.032750+0000","last_unstale":"2024-04-23T14:44:01.317096+0000","last_undegraded":"2024-04-23T14:44:01.317096+0000","last_fullsized":"2024-04-23T14:44:01.317096+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:58:03.451539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.306532+0000","last_change":"2024-04-23T14:41:40.037435+0000","last_active":"2024-04-23T14:44:01.306532+0000","last_peered":"2024-04-23T14:44:01.306532+0000","last_clean":"2024-04-23T14:44:01.306532+0000","last_became_active":"2024-04-23T14:41:40.036655+0000","last_became_peered":"2024-04-23T14:41:40.036655+0000","last_unstale":"2024-04-23T14:44:01.306532+0000","last_undegraded":"2024-04-23T14:44:01.306532+0000","last_fullsized":"2024-04-23T14:44:01.306532+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:51:58.847850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.306139+0000","last_change":"2024-04-23T14:41:40.024977+0000","last_active":"2024-04-23T14:44:01.306139+0000","last_peered":"2024-04-23T14:44:01.306139+0000","last_clean":"2024-04-23T14:44:01.306139+0000","last_became_active":"2024-04-23T14:41:40.023879+0000","last_became_peered":"2024-04-23T14:41:40.023879+0000","last_unstale":"2024-04-23T14:44:01.306139+0000","last_undegraded":"2024-04-23T14:44:01.306139+0000","last_fullsized":"2024-04-23T14:44:01.306139+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:03:40.672661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315735+0000","last_change":"2024-04-23T14:41:40.017757+0000","last_active":"2024-04-23T14:44:01.315735+0000","last_peered":"2024-04-23T14:44:01.315735+0000","last_clean":"2024-04-23T14:44:01.315735+0000","last_became_active":"2024-04-23T14:41:40.013996+0000","last_became_peered":"2024-04-23T14:41:40.013996+0000","last_unstale":"2024-04-23T14:44:01.315735+0000","last_undegraded":"2024-04-23T14:44:01.315735+0000","last_fullsized":"2024-04-23T14:44:01.315735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:20:03.376733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314329+0000","last_change":"2024-04-23T14:41:40.038247+0000","last_active":"2024-04-23T14:44:01.314329+0000","last_peered":"2024-04-23T14:44:01.314329+0000","last_clean":"2024-04-23T14:44:01.314329+0000","last_became_active":"2024-04-23T14:41:40.037934+0000","last_became_peered":"2024-04-23T14:41:40.037934+0000","last_unstale":"2024-04-23T14:44:01.314329+0000","last_undegraded":"2024-04-23T14:44:01.314329+0000","last_fullsized":"2024-04-23T14:44:01.314329+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:15.637733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315407+0000","last_change":"2024-04-23T14:41:40.015341+0000","last_active":"2024-04-23T14:44:01.315407+0000","last_peered":"2024-04-23T14:44:01.315407+0000","last_clean":"2024-04-23T14:44:01.315407+0000","last_became_active":"2024-04-23T14:41:40.013627+0000","last_became_peered":"2024-04-23T14:41:40.013627+0000","last_unstale":"2024-04-23T14:44:01.315407+0000","last_undegraded":"2024-04-23T14:44:01.315407+0000","last_fullsized":"2024-04-23T14:44:01.315407+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:05:36.248107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748243+0000","last_change":"2024-04-23T14:41:40.010950+0000","last_active":"2024-04-23T14:44:01.748243+0000","last_peered":"2024-04-23T14:44:01.748243+0000","last_clean":"2024-04-23T14:44:01.748243+0000","last_became_active":"2024-04-23T14:41:40.010584+0000","last_became_peered":"2024-04-23T14:41:40.010584+0000","last_unstale":"2024-04-23T14:44:01.748243+0000","last_undegraded":"2024-04-23T14:44:01.748243+0000","last_fullsized":"2024-04-23T14:44:01.748243+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:46:25.379243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.318133+0000","last_change":"2024-04-23T14:41:40.031198+0000","last_active":"2024-04-23T14:44:01.318133+0000","last_peered":"2024-04-23T14:44:01.318133+0000","last_clean":"2024-04-23T14:44:01.318133+0000","last_became_active":"2024-04-23T14:41:40.030652+0000","last_became_peered":"2024-04-23T14:41:40.030652+0000","last_unstale":"2024-04-23T14:44:01.318133+0000","last_undegraded":"2024-04-23T14:44:01.318133+0000","last_fullsized":"2024-04-23T14:44:01.318133+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:26:00.520182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315379+0000","last_change":"2024-04-23T14:41:40.009820+0000","last_active":"2024-04-23T14:44:01.315379+0000","last_peered":"2024-04-23T14:44:01.315379+0000","last_clean":"2024-04-23T14:44:01.315379+0000","last_became_active":"2024-04-23T14:41:40.009468+0000","last_became_peered":"2024-04-23T14:41:40.009468+0000","last_unstale":"2024-04-23T14:44:01.315379+0000","last_undegraded":"2024-04-23T14:44:01.315379+0000","last_fullsized":"2024-04-23T14:44:01.315379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:03:50.178050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316126+0000","last_change":"2024-04-23T14:41:40.008363+0000","last_active":"2024-04-23T14:44:01.316126+0000","last_peered":"2024-04-23T14:44:01.316126+0000","last_clean":"2024-04-23T14:44:01.316126+0000","last_became_active":"2024-04-23T14:41:40.006823+0000","last_became_peered":"2024-04-23T14:41:40.006823+0000","last_unstale":"2024-04-23T14:44:01.316126+0000","last_undegraded":"2024-04-23T14:44:01.316126+0000","last_fullsized":"2024-04-23T14:44:01.316126+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:12:33.718815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748598+0000","last_change":"2024-04-23T14:41:40.021069+0000","last_active":"2024-04-23T14:44:01.748598+0000","last_peered":"2024-04-23T14:44:01.748598+0000","last_clean":"2024-04-23T14:44:01.748598+0000","last_became_active":"2024-04-23T14:41:40.020334+0000","last_became_peered":"2024-04-23T14:41:40.020334+0000","last_unstale":"2024-04-23T14:44:01.748598+0000","last_undegraded":"2024-04-23T14:44:01.748598+0000","last_fullsized":"2024-04-23T14:44:01.748598+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:50:56.560928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315582+0000","last_change":"2024-04-23T14:41:40.017746+0000","last_active":"2024-04-23T14:44:01.315582+0000","last_peered":"2024-04-23T14:44:01.315582+0000","last_clean":"2024-04-23T14:44:01.315582+0000","last_became_active":"2024-04-23T14:41:40.014495+0000","last_became_peered":"2024-04-23T14:41:40.014495+0000","last_unstale":"2024-04-23T14:44:01.315582+0000","last_undegraded":"2024-04-23T14:44:01.315582+0000","last_fullsized":"2024-04-23T14:44:01.315582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:23:22.188110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316167+0000","last_change":"2024-04-23T14:41:40.021429+0000","last_active":"2024-04-23T14:44:01.316167+0000","last_peered":"2024-04-23T14:44:01.316167+0000","last_clean":"2024-04-23T14:44:01.316167+0000","last_became_active":"2024-04-23T14:41:40.021087+0000","last_became_peered":"2024-04-23T14:41:40.021087+0000","last_unstale":"2024-04-23T14:44:01.316167+0000","last_undegraded":"2024-04-23T14:44:01.316167+0000","last_fullsized":"2024-04-23T14:44:01.316167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:51:16.633253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.303276+0000","last_change":"2024-04-23T14:41:40.031839+0000","last_active":"2024-04-23T14:44:01.303276+0000","last_peered":"2024-04-23T14:44:01.303276+0000","last_clean":"2024-04-23T14:44:01.303276+0000","last_became_active":"2024-04-23T14:41:40.031463+0000","last_became_peered":"2024-04-23T14:41:40.031463+0000","last_unstale":"2024-04-23T14:44:01.303276+0000","last_undegraded":"2024-04-23T14:44:01.303276+0000","last_fullsized":"2024-04-23T14:44:01.303276+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:51:51.200463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.302427+0000","last_change":"2024-04-23T14:41:40.042575+0000","last_active":"2024-04-23T14:44:01.302427+0000","last_peered":"2024-04-23T14:44:01.302427+0000","last_clean":"2024-04-23T14:44:01.302427+0000","last_became_active":"2024-04-23T14:41:40.033758+0000","last_became_peered":"2024-04-23T14:41:40.033758+0000","last_unstale":"2024-04-23T14:44:01.302427+0000","last_undegraded":"2024-04-23T14:44:01.302427+0000","last_fullsized":"2024-04-23T14:44:01.302427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:55:17.338169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314525+0000","last_change":"2024-04-23T14:41:40.036214+0000","last_active":"2024-04-23T14:44:01.314525+0000","last_peered":"2024-04-23T14:44:01.314525+0000","last_clean":"2024-04-23T14:44:01.314525+0000","last_became_active":"2024-04-23T14:41:40.035892+0000","last_became_peered":"2024-04-23T14:41:40.035892+0000","last_unstale":"2024-04-23T14:44:01.314525+0000","last_undegraded":"2024-04-23T14:44:01.314525+0000","last_fullsized":"2024-04-23T14:44:01.314525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:29:18.668690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.317251+0000","last_change":"2024-04-23T14:41:40.029837+0000","last_active":"2024-04-23T14:44:01.317251+0000","last_peered":"2024-04-23T14:44:01.317251+0000","last_clean":"2024-04-23T14:44:01.317251+0000","last_became_active":"2024-04-23T14:41:40.029472+0000","last_became_peered":"2024-04-23T14:41:40.029472+0000","last_unstale":"2024-04-23T14:44:01.317251+0000","last_undegraded":"2024-04-23T14:44:01.317251+0000","last_fullsized":"2024-04-23T14:44:01.317251+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:28:52.194898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.304455+0000","last_change":"2024-04-23T14:41:40.015631+0000","last_active":"2024-04-23T14:44:01.304455+0000","last_peered":"2024-04-23T14:44:01.304455+0000","last_clean":"2024-04-23T14:44:01.304455+0000","last_became_active":"2024-04-23T14:41:40.015211+0000","last_became_peered":"2024-04-23T14:41:40.015211+0000","last_unstale":"2024-04-23T14:44:01.304455+0000","last_undegraded":"2024-04-23T14:44:01.304455+0000","last_fullsized":"2024-04-23T14:44:01.304455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:03:33.485364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315489+0000","last_change":"2024-04-23T14:42:43.680530+0000","last_active":"2024-04-23T14:44:01.315489+0000","last_peered":"2024-04-23T14:44:01.315489+0000","last_clean":"2024-04-23T14:44:01.315489+0000","last_became_active":"2024-04-23T14:42:43.680249+0000","last_became_peered":"2024-04-23T14:42:43.680249+0000","last_unstale":"2024-04-23T14:44:01.315489+0000","last_undegraded":"2024-04-23T14:44:01.315489+0000","last_fullsized":"2024-04-23T14:44:01.315489+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:55:36.334572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,6],"acting":[2,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.304507+0000","last_change":"2024-04-23T14:41:39.017031+0000","last_active":"2024-04-23T14:44:01.304507+0000","last_peered":"2024-04-23T14:44:01.304507+0000","last_clean":"2024-04-23T14:44:01.304507+0000","last_became_active":"2024-04-23T14:41:39.016699+0000","last_became_peered":"2024-04-23T14:41:39.016699+0000","last_unstale":"2024-04-23T14:44:01.304507+0000","last_undegraded":"2024-04-23T14:44:01.304507+0000","last_fullsized":"2024-04-23T14:44:01.304507+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:17:49.812601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.319643+0000","last_change":"2024-04-23T14:41:40.031957+0000","last_active":"2024-04-23T14:44:01.319643+0000","last_peered":"2024-04-23T14:44:01.319643+0000","last_clean":"2024-04-23T14:44:01.319643+0000","last_became_active":"2024-04-23T14:41:40.028151+0000","last_became_peered":"2024-04-23T14:41:40.028151+0000","last_unstale":"2024-04-23T14:44:01.319643+0000","last_undegraded":"2024-04-23T14:44:01.319643+0000","last_fullsized":"2024-04-23T14:44:01.319643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:46:48.455032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.319788+0000","last_change":"2024-04-23T14:41:39.022503+0000","last_active":"2024-04-23T14:44:01.319788+0000","last_peered":"2024-04-23T14:44:01.319788+0000","last_clean":"2024-04-23T14:44:01.319788+0000","last_became_active":"2024-04-23T14:41:39.022093+0000","last_became_peered":"2024-04-23T14:41:39.022093+0000","last_unstale":"2024-04-23T14:44:01.319788+0000","last_undegraded":"2024-04-23T14:44:01.319788+0000","last_fullsized":"2024-04-23T14:44:01.319788+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:48:27.861140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.306043+0000","last_change":"2024-04-23T14:41:40.032878+0000","last_active":"2024-04-23T14:44:01.306043+0000","last_peered":"2024-04-23T14:44:01.306043+0000","last_clean":"2024-04-23T14:44:01.306043+0000","last_became_active":"2024-04-23T14:41:40.027473+0000","last_became_peered":"2024-04-23T14:41:40.027473+0000","last_unstale":"2024-04-23T14:44:01.306043+0000","last_undegraded":"2024-04-23T14:44:01.306043+0000","last_fullsized":"2024-04-23T14:44:01.306043+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:00:17.677306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316901+0000","last_change":"2024-04-23T14:42:43.448728+0000","last_active":"2024-04-23T14:44:01.316901+0000","last_peered":"2024-04-23T14:44:01.316901+0000","last_clean":"2024-04-23T14:44:01.316901+0000","last_became_active":"2024-04-23T14:42:43.443993+0000","last_became_peered":"2024-04-23T14:42:43.443993+0000","last_unstale":"2024-04-23T14:44:01.316901+0000","last_undegraded":"2024-04-23T14:44:01.316901+0000","last_fullsized":"2024-04-23T14:44:01.316901+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:04:16.430822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314260+0000","last_change":"2024-04-23T14:41:40.019955+0000","last_active":"2024-04-23T14:44:01.314260+0000","last_peered":"2024-04-23T14:44:01.314260+0000","last_clean":"2024-04-23T14:44:01.314260+0000","last_became_active":"2024-04-23T14:41:40.019579+0000","last_became_peered":"2024-04-23T14:41:40.019579+0000","last_unstale":"2024-04-23T14:44:01.314260+0000","last_undegraded":"2024-04-23T14:44:01.314260+0000","last_fullsized":"2024-04-23T14:44:01.314260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:13:14.330051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316794+0000","last_change":"2024-04-23T14:42:43.446652+0000","last_active":"2024-04-23T14:44:01.316794+0000","last_peered":"2024-04-23T14:44:01.316794+0000","last_clean":"2024-04-23T14:44:01.316794+0000","last_became_active":"2024-04-23T14:42:43.443549+0000","last_became_peered":"2024-04-23T14:42:43.443549+0000","last_unstale":"2024-04-23T14:44:01.316794+0000","last_undegraded":"2024-04-23T14:44:01.316794+0000","last_fullsized":"2024-04-23T14:44:01.316794+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:31:12.927663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315219+0000","last_change":"2024-04-23T14:41:40.015592+0000","last_active":"2024-04-23T14:44:01.315219+0000","last_peered":"2024-04-23T14:44:01.315219+0000","last_clean":"2024-04-23T14:44:01.315219+0000","last_became_active":"2024-04-23T14:41:40.015292+0000","last_became_peered":"2024-04-23T14:41:40.015292+0000","last_unstale":"2024-04-23T14:44:01.315219+0000","last_undegraded":"2024-04-23T14:44:01.315219+0000","last_fullsized":"2024-04-23T14:44:01.315219+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:52:01.641962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314999+0000","last_change":"2024-04-23T14:41:39.008526+0000","last_active":"2024-04-23T14:44:01.314999+0000","last_peered":"2024-04-23T14:44:01.314999+0000","last_clean":"2024-04-23T14:44:01.314999+0000","last_became_active":"2024-04-23T14:41:39.008222+0000","last_became_peered":"2024-04-23T14:41:39.008222+0000","last_unstale":"2024-04-23T14:44:01.314999+0000","last_undegraded":"2024-04-23T14:44:01.314999+0000","last_fullsized":"2024-04-23T14:44:01.314999+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:23:09.191426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314352+0000","last_change":"2024-04-23T14:42:43.677846+0000","last_active":"2024-04-23T14:44:01.314352+0000","last_peered":"2024-04-23T14:44:01.314352+0000","last_clean":"2024-04-23T14:44:01.314352+0000","last_became_active":"2024-04-23T14:42:43.677413+0000","last_became_peered":"2024-04-23T14:42:43.677413+0000","last_unstale":"2024-04-23T14:44:01.314352+0000","last_undegraded":"2024-04-23T14:44:01.314352+0000","last_fullsized":"2024-04-23T14:44:01.314352+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:35:55.039607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315086+0000","last_change":"2024-04-23T14:41:39.028566+0000","last_active":"2024-04-23T14:44:01.315086+0000","last_peered":"2024-04-23T14:44:01.315086+0000","last_clean":"2024-04-23T14:44:01.315086+0000","last_became_active":"2024-04-23T14:41:39.028174+0000","last_became_peered":"2024-04-23T14:41:39.028174+0000","last_unstale":"2024-04-23T14:44:01.315086+0000","last_undegraded":"2024-04-23T14:44:01.315086+0000","last_fullsized":"2024-04-23T14:44:01.315086+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:50:55.424741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315025+0000","last_change":"2024-04-23T14:42:43.530152+0000","last_active":"2024-04-23T14:44:01.315025+0000","last_peered":"2024-04-23T14:44:01.315025+0000","last_clean":"2024-04-23T14:44:01.315025+0000","last_became_active":"2024-04-23T14:42:43.529776+0000","last_became_peered":"2024-04-23T14:42:43.529776+0000","last_unstale":"2024-04-23T14:44:01.315025+0000","last_undegraded":"2024-04-23T14:44:01.315025+0000","last_fullsized":"2024-04-23T14:44:01.315025+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:53:29.597081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315842+0000","last_change":"2024-04-23T14:41:39.028618+0000","last_active":"2024-04-23T14:44:01.315842+0000","last_peered":"2024-04-23T14:44:01.315842+0000","last_clean":"2024-04-23T14:44:01.315842+0000","last_became_active":"2024-04-23T14:41:39.028274+0000","last_became_peered":"2024-04-23T14:41:39.028274+0000","last_unstale":"2024-04-23T14:44:01.315842+0000","last_undegraded":"2024-04-23T14:44:01.315842+0000","last_fullsized":"2024-04-23T14:44:01.315842+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:14:31.571760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.318144+0000","last_change":"2024-04-23T14:41:40.020687+0000","last_active":"2024-04-23T14:44:01.318144+0000","last_peered":"2024-04-23T14:44:01.318144+0000","last_clean":"2024-04-23T14:44:01.318144+0000","last_became_active":"2024-04-23T14:41:40.020352+0000","last_became_peered":"2024-04-23T14:41:40.020352+0000","last_unstale":"2024-04-23T14:44:01.318144+0000","last_undegraded":"2024-04-23T14:44:01.318144+0000","last_fullsized":"2024-04-23T14:44:01.318144+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:29:37.013486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314616+0000","last_change":"2024-04-23T14:41:39.006002+0000","last_active":"2024-04-23T14:44:01.314616+0000","last_peered":"2024-04-23T14:44:01.314616+0000","last_clean":"2024-04-23T14:44:01.314616+0000","last_became_active":"2024-04-23T14:41:39.005415+0000","last_became_peered":"2024-04-23T14:41:39.005415+0000","last_unstale":"2024-04-23T14:44:01.314616+0000","last_undegraded":"2024-04-23T14:44:01.314616+0000","last_fullsized":"2024-04-23T14:44:01.314616+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:13:55.491931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.303902+0000","last_change":"2024-04-23T14:41:40.029393+0000","last_active":"2024-04-23T14:44:01.303902+0000","last_peered":"2024-04-23T14:44:01.303902+0000","last_clean":"2024-04-23T14:44:01.303902+0000","last_became_active":"2024-04-23T14:41:40.029039+0000","last_became_peered":"2024-04-23T14:41:40.029039+0000","last_unstale":"2024-04-23T14:44:01.303902+0000","last_undegraded":"2024-04-23T14:44:01.303902+0000","last_fullsized":"2024-04-23T14:44:01.303902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:39:24.461260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316197+0000","last_change":"2024-04-23T14:41:39.014638+0000","last_active":"2024-04-23T14:44:01.316197+0000","last_peered":"2024-04-23T14:44:01.316197+0000","last_clean":"2024-04-23T14:44:01.316197+0000","last_became_active":"2024-04-23T14:41:39.014289+0000","last_became_peered":"2024-04-23T14:41:39.014289+0000","last_unstale":"2024-04-23T14:44:01.316197+0000","last_undegraded":"2024-04-23T14:44:01.316197+0000","last_fullsized":"2024-04-23T14:44:01.316197+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:15:15.855399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315648+0000","last_change":"2024-04-23T14:41:40.015781+0000","last_active":"2024-04-23T14:44:01.315648+0000","last_peered":"2024-04-23T14:44:01.315648+0000","last_clean":"2024-04-23T14:44:01.315648+0000","last_became_active":"2024-04-23T14:41:40.015423+0000","last_became_peered":"2024-04-23T14:41:40.015423+0000","last_unstale":"2024-04-23T14:44:01.315648+0000","last_undegraded":"2024-04-23T14:44:01.315648+0000","last_fullsized":"2024-04-23T14:44:01.315648+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:56:55.947585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748648+0000","last_change":"2024-04-23T14:41:39.011021+0000","last_active":"2024-04-23T14:44:01.748648+0000","last_peered":"2024-04-23T14:44:01.748648+0000","last_clean":"2024-04-23T14:44:01.748648+0000","last_became_active":"2024-04-23T14:41:39.010637+0000","last_became_peered":"2024-04-23T14:41:39.010637+0000","last_unstale":"2024-04-23T14:44:01.748648+0000","last_undegraded":"2024-04-23T14:44:01.748648+0000","last_fullsized":"2024-04-23T14:44:01.748648+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:22:22.647170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.305882+0000","last_change":"2024-04-23T14:41:40.042797+0000","last_active":"2024-04-23T14:44:01.305882+0000","last_peered":"2024-04-23T14:44:01.305882+0000","last_clean":"2024-04-23T14:44:01.305882+0000","last_became_active":"2024-04-23T14:41:40.037797+0000","last_became_peered":"2024-04-23T14:41:40.037797+0000","last_unstale":"2024-04-23T14:44:01.305882+0000","last_undegraded":"2024-04-23T14:44:01.305882+0000","last_fullsized":"2024-04-23T14:44:01.305882+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:37:32.604276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315453+0000","last_change":"2024-04-23T14:41:39.010585+0000","last_active":"2024-04-23T14:44:01.315453+0000","last_peered":"2024-04-23T14:44:01.315453+0000","last_clean":"2024-04-23T14:44:01.315453+0000","last_became_active":"2024-04-23T14:41:39.010294+0000","last_became_peered":"2024-04-23T14:41:39.010294+0000","last_unstale":"2024-04-23T14:44:01.315453+0000","last_undegraded":"2024-04-23T14:44:01.315453+0000","last_fullsized":"2024-04-23T14:44:01.315453+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:38:03.146904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314580+0000","last_change":"2024-04-23T14:41:40.013150+0000","last_active":"2024-04-23T14:44:01.314580+0000","last_peered":"2024-04-23T14:44:01.314580+0000","last_clean":"2024-04-23T14:44:01.314580+0000","last_became_active":"2024-04-23T14:41:40.012310+0000","last_became_peered":"2024-04-23T14:41:40.012310+0000","last_unstale":"2024-04-23T14:44:01.314580+0000","last_undegraded":"2024-04-23T14:44:01.314580+0000","last_fullsized":"2024-04-23T14:44:01.314580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:45:25.777200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'33","reported_seq":343,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.706603+0000","last_change":"2024-04-23T14:41:39.004256+0000","last_active":"2024-04-23T14:44:01.706603+0000","last_peered":"2024-04-23T14:44:01.706603+0000","last_clean":"2024-04-23T14:44:01.706603+0000","last_became_active":"2024-04-23T14:41:39.003874+0000","last_became_peered":"2024-04-23T14:41:39.003874+0000","last_unstale":"2024-04-23T14:44:01.706603+0000","last_undegraded":"2024-04-23T14:44:01.706603+0000","last_fullsized":"2024-04-23T14:44:01.706603+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:38:30.727149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68610,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":288,"num_read_kb":205,"num_write":33,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314522+0000","last_change":"2024-04-23T14:41:40.012131+0000","last_active":"2024-04-23T14:44:01.314522+0000","last_peered":"2024-04-23T14:44:01.314522+0000","last_clean":"2024-04-23T14:44:01.314522+0000","last_became_active":"2024-04-23T14:41:40.011225+0000","last_became_peered":"2024-04-23T14:41:40.011225+0000","last_unstale":"2024-04-23T14:44:01.314522+0000","last_undegraded":"2024-04-23T14:44:01.314522+0000","last_fullsized":"2024-04-23T14:44:01.314522+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:36:12.085228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315400+0000","last_change":"2024-04-23T14:41:39.001935+0000","last_active":"2024-04-23T14:44:01.315400+0000","last_peered":"2024-04-23T14:44:01.315400+0000","last_clean":"2024-04-23T14:44:01.315400+0000","last_became_active":"2024-04-23T14:41:39.001583+0000","last_became_peered":"2024-04-23T14:41:39.001583+0000","last_unstale":"2024-04-23T14:44:01.315400+0000","last_undegraded":"2024-04-23T14:44:01.315400+0000","last_fullsized":"2024-04-23T14:44:01.315400+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:47:40.986280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315917+0000","last_change":"2024-04-23T14:41:40.036515+0000","last_active":"2024-04-23T14:44:01.315917+0000","last_peered":"2024-04-23T14:44:01.315917+0000","last_clean":"2024-04-23T14:44:01.315917+0000","last_became_active":"2024-04-23T14:41:40.036125+0000","last_became_peered":"2024-04-23T14:41:40.036125+0000","last_unstale":"2024-04-23T14:44:01.315917+0000","last_undegraded":"2024-04-23T14:44:01.315917+0000","last_fullsized":"2024-04-23T14:44:01.315917+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:55:28.465378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314793+0000","last_change":"2024-04-23T14:41:39.006127+0000","last_active":"2024-04-23T14:44:01.314793+0000","last_peered":"2024-04-23T14:44:01.314793+0000","last_clean":"2024-04-23T14:44:01.314793+0000","last_became_active":"2024-04-23T14:41:39.005761+0000","last_became_peered":"2024-04-23T14:41:39.005761+0000","last_unstale":"2024-04-23T14:44:01.314793+0000","last_undegraded":"2024-04-23T14:44:01.314793+0000","last_fullsized":"2024-04-23T14:44:01.314793+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:10:24.050016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748673+0000","last_change":"2024-04-23T14:42:43.678799+0000","last_active":"2024-04-23T14:44:01.748673+0000","last_peered":"2024-04-23T14:44:01.748673+0000","last_clean":"2024-04-23T14:44:01.748673+0000","last_became_active":"2024-04-23T14:42:43.678318+0000","last_became_peered":"2024-04-23T14:42:43.678318+0000","last_unstale":"2024-04-23T14:44:01.748673+0000","last_undegraded":"2024-04-23T14:44:01.748673+0000","last_fullsized":"2024-04-23T14:44:01.748673+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:26:21.827247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315145+0000","last_change":"2024-04-23T14:41:39.005873+0000","last_active":"2024-04-23T14:44:01.315145+0000","last_peered":"2024-04-23T14:44:01.315145+0000","last_clean":"2024-04-23T14:44:01.315145+0000","last_became_active":"2024-04-23T14:41:39.005481+0000","last_became_peered":"2024-04-23T14:41:39.005481+0000","last_unstale":"2024-04-23T14:44:01.315145+0000","last_undegraded":"2024-04-23T14:44:01.315145+0000","last_fullsized":"2024-04-23T14:44:01.315145+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:49:59.117451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748238+0000","last_change":"2024-04-23T14:41:40.009346+0000","last_active":"2024-04-23T14:44:01.748238+0000","last_peered":"2024-04-23T14:44:01.748238+0000","last_clean":"2024-04-23T14:44:01.748238+0000","last_became_active":"2024-04-23T14:41:40.008927+0000","last_became_peered":"2024-04-23T14:41:40.008927+0000","last_unstale":"2024-04-23T14:44:01.748238+0000","last_undegraded":"2024-04-23T14:44:01.748238+0000","last_fullsized":"2024-04-23T14:44:01.748238+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:33:36.513349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314285+0000","last_change":"2024-04-23T14:42:43.452178+0000","last_active":"2024-04-23T14:44:01.314285+0000","last_peered":"2024-04-23T14:44:01.314285+0000","last_clean":"2024-04-23T14:44:01.314285+0000","last_became_active":"2024-04-23T14:42:43.444786+0000","last_became_peered":"2024-04-23T14:42:43.444786+0000","last_unstale":"2024-04-23T14:44:01.314285+0000","last_undegraded":"2024-04-23T14:44:01.314285+0000","last_fullsized":"2024-04-23T14:44:01.314285+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:00:21.945663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,1],"acting":[8,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314546+0000","last_change":"2024-04-23T14:41:40.016845+0000","last_active":"2024-04-23T14:44:01.314546+0000","last_peered":"2024-04-23T14:44:01.314546+0000","last_clean":"2024-04-23T14:44:01.314546+0000","last_became_active":"2024-04-23T14:41:40.016543+0000","last_became_peered":"2024-04-23T14:41:40.016543+0000","last_unstale":"2024-04-23T14:44:01.314546+0000","last_undegraded":"2024-04-23T14:44:01.314546+0000","last_fullsized":"2024-04-23T14:44:01.314546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:52:41.054454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314792+0000","last_change":"2024-04-23T14:41:38.999968+0000","last_active":"2024-04-23T14:44:01.314792+0000","last_peered":"2024-04-23T14:44:01.314792+0000","last_clean":"2024-04-23T14:44:01.314792+0000","last_became_active":"2024-04-23T14:41:38.999658+0000","last_became_peered":"2024-04-23T14:41:38.999658+0000","last_unstale":"2024-04-23T14:44:01.314792+0000","last_undegraded":"2024-04-23T14:44:01.314792+0000","last_fullsized":"2024-04-23T14:44:01.314792+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:14:20.990331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.317006+0000","last_change":"2024-04-23T14:41:40.011753+0000","last_active":"2024-04-23T14:44:01.317006+0000","last_peered":"2024-04-23T14:44:01.317006+0000","last_clean":"2024-04-23T14:44:01.317006+0000","last_became_active":"2024-04-23T14:41:40.011264+0000","last_became_peered":"2024-04-23T14:41:40.011264+0000","last_unstale":"2024-04-23T14:44:01.317006+0000","last_undegraded":"2024-04-23T14:44:01.317006+0000","last_fullsized":"2024-04-23T14:44:01.317006+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:52:22.812046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.304344+0000","last_change":"2024-04-23T14:41:39.027384+0000","last_active":"2024-04-23T14:44:01.304344+0000","last_peered":"2024-04-23T14:44:01.304344+0000","last_clean":"2024-04-23T14:44:01.304344+0000","last_became_active":"2024-04-23T14:41:39.027018+0000","last_became_peered":"2024-04-23T14:41:39.027018+0000","last_unstale":"2024-04-23T14:44:01.304344+0000","last_undegraded":"2024-04-23T14:44:01.304344+0000","last_fullsized":"2024-04-23T14:44:01.304344+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:13:54.685704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315414+0000","last_change":"2024-04-23T14:41:40.041216+0000","last_active":"2024-04-23T14:44:01.315414+0000","last_peered":"2024-04-23T14:44:01.315414+0000","last_clean":"2024-04-23T14:44:01.315414+0000","last_became_active":"2024-04-23T14:41:40.040789+0000","last_became_peered":"2024-04-23T14:41:40.040789+0000","last_unstale":"2024-04-23T14:44:01.315414+0000","last_undegraded":"2024-04-23T14:44:01.315414+0000","last_fullsized":"2024-04-23T14:44:01.315414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:01:13.156704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748318+0000","last_change":"2024-04-23T14:41:39.024399+0000","last_active":"2024-04-23T14:44:01.748318+0000","last_peered":"2024-04-23T14:44:01.748318+0000","last_clean":"2024-04-23T14:44:01.748318+0000","last_became_active":"2024-04-23T14:41:39.024141+0000","last_became_peered":"2024-04-23T14:41:39.024141+0000","last_unstale":"2024-04-23T14:44:01.748318+0000","last_undegraded":"2024-04-23T14:44:01.748318+0000","last_fullsized":"2024-04-23T14:44:01.748318+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:34:27.516614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315148+0000","last_change":"2024-04-23T14:42:43.680818+0000","last_active":"2024-04-23T14:44:01.315148+0000","last_peered":"2024-04-23T14:44:01.315148+0000","last_clean":"2024-04-23T14:44:01.315148+0000","last_became_active":"2024-04-23T14:42:43.680463+0000","last_became_peered":"2024-04-23T14:42:43.680463+0000","last_unstale":"2024-04-23T14:44:01.315148+0000","last_undegraded":"2024-04-23T14:44:01.315148+0000","last_fullsized":"2024-04-23T14:44:01.315148+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:34:36.964359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314817+0000","last_change":"2024-04-23T14:41:40.014689+0000","last_active":"2024-04-23T14:44:01.314817+0000","last_peered":"2024-04-23T14:44:01.314817+0000","last_clean":"2024-04-23T14:44:01.314817+0000","last_became_active":"2024-04-23T14:41:40.014371+0000","last_became_peered":"2024-04-23T14:41:40.014371+0000","last_unstale":"2024-04-23T14:44:01.314817+0000","last_undegraded":"2024-04-23T14:44:01.314817+0000","last_fullsized":"2024-04-23T14:44:01.314817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:56:41.284144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315704+0000","last_change":"2024-04-23T14:41:39.031327+0000","last_active":"2024-04-23T14:44:01.315704+0000","last_peered":"2024-04-23T14:44:01.315704+0000","last_clean":"2024-04-23T14:44:01.315704+0000","last_became_active":"2024-04-23T14:41:39.030864+0000","last_became_peered":"2024-04-23T14:41:39.030864+0000","last_unstale":"2024-04-23T14:44:01.315704+0000","last_undegraded":"2024-04-23T14:44:01.315704+0000","last_fullsized":"2024-04-23T14:44:01.315704+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:58:24.048207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.319239+0000","last_change":"2024-04-23T14:41:40.035104+0000","last_active":"2024-04-23T14:44:01.319239+0000","last_peered":"2024-04-23T14:44:01.319239+0000","last_clean":"2024-04-23T14:44:01.319239+0000","last_became_active":"2024-04-23T14:41:40.034678+0000","last_became_peered":"2024-04-23T14:41:40.034678+0000","last_unstale":"2024-04-23T14:44:01.319239+0000","last_undegraded":"2024-04-23T14:44:01.319239+0000","last_fullsized":"2024-04-23T14:44:01.319239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:29:53.040424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314795+0000","last_change":"2024-04-23T14:41:39.016684+0000","last_active":"2024-04-23T14:44:01.314795+0000","last_peered":"2024-04-23T14:44:01.314795+0000","last_clean":"2024-04-23T14:44:01.314795+0000","last_became_active":"2024-04-23T14:41:39.016375+0000","last_became_peered":"2024-04-23T14:41:39.016375+0000","last_unstale":"2024-04-23T14:44:01.314795+0000","last_undegraded":"2024-04-23T14:44:01.314795+0000","last_fullsized":"2024-04-23T14:44:01.314795+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:09.449590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":357,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748262+0000","last_change":"2024-04-23T14:40:22.305413+0000","last_active":"2024-04-23T14:44:01.748262+0000","last_peered":"2024-04-23T14:44:01.748262+0000","last_clean":"2024-04-23T14:44:01.748262+0000","last_became_active":"2024-04-23T14:40:22.304977+0000","last_became_peered":"2024-04-23T14:40:22.304977+0000","last_unstale":"2024-04-23T14:44:01.748262+0000","last_undegraded":"2024-04-23T14:44:01.748262+0000","last_fullsized":"2024-04-23T14:44:01.748262+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:36:28.336753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:36:28.336753+0000","last_clean_scrub_stamp":"2024-04-23T14:36:28.336753+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:42:27.611770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315629+0000","last_change":"2024-04-23T14:41:40.023300+0000","last_active":"2024-04-23T14:44:01.315629+0000","last_peered":"2024-04-23T14:44:01.315629+0000","last_clean":"2024-04-23T14:44:01.315629+0000","last_became_active":"2024-04-23T14:41:40.022888+0000","last_became_peered":"2024-04-23T14:41:40.022888+0000","last_unstale":"2024-04-23T14:44:01.315629+0000","last_undegraded":"2024-04-23T14:44:01.315629+0000","last_fullsized":"2024-04-23T14:44:01.315629+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:31:48.705005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748412+0000","last_change":"2024-04-23T14:41:39.005482+0000","last_active":"2024-04-23T14:44:01.748412+0000","last_peered":"2024-04-23T14:44:01.748412+0000","last_clean":"2024-04-23T14:44:01.748412+0000","last_became_active":"2024-04-23T14:41:39.004552+0000","last_became_peered":"2024-04-23T14:41:39.004552+0000","last_unstale":"2024-04-23T14:44:01.748412+0000","last_undegraded":"2024-04-23T14:44:01.748412+0000","last_fullsized":"2024-04-23T14:44:01.748412+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:05:51.411210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314496+0000","last_change":"2024-04-23T14:41:39.015605+0000","last_active":"2024-04-23T14:44:01.314496+0000","last_peered":"2024-04-23T14:44:01.314496+0000","last_clean":"2024-04-23T14:44:01.314496+0000","last_became_active":"2024-04-23T14:41:39.015260+0000","last_became_peered":"2024-04-23T14:41:39.015260+0000","last_unstale":"2024-04-23T14:44:01.314496+0000","last_undegraded":"2024-04-23T14:44:01.314496+0000","last_fullsized":"2024-04-23T14:44:01.314496+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:23:21.251483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316641+0000","last_change":"2024-04-23T14:41:40.020619+0000","last_active":"2024-04-23T14:44:01.316641+0000","last_peered":"2024-04-23T14:44:01.316641+0000","last_clean":"2024-04-23T14:44:01.316641+0000","last_became_active":"2024-04-23T14:41:40.020191+0000","last_became_peered":"2024-04-23T14:41:40.020191+0000","last_unstale":"2024-04-23T14:44:01.316641+0000","last_undegraded":"2024-04-23T14:44:01.316641+0000","last_fullsized":"2024-04-23T14:44:01.316641+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:54:34.036464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'45","reported_seq":90,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:02.019133+0000","last_change":"2024-04-23T14:41:39.019507+0000","last_active":"2024-04-23T14:44:02.019133+0000","last_peered":"2024-04-23T14:44:02.019133+0000","last_clean":"2024-04-23T14:44:02.019133+0000","last_became_active":"2024-04-23T14:41:39.019116+0000","last_became_peered":"2024-04-23T14:41:39.019116+0000","last_unstale":"2024-04-23T14:44:02.019133+0000","last_undegraded":"2024-04-23T14:44:02.019133+0000","last_fullsized":"2024-04-23T14:44:02.019133+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:06:05.777908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":81,"num_write_kb":838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.318810+0000","last_change":"2024-04-23T14:41:40.032765+0000","last_active":"2024-04-23T14:44:01.318810+0000","last_peered":"2024-04-23T14:44:01.318810+0000","last_clean":"2024-04-23T14:44:01.318810+0000","last_became_active":"2024-04-23T14:41:40.027896+0000","last_became_peered":"2024-04-23T14:41:40.027896+0000","last_unstale":"2024-04-23T14:44:01.318810+0000","last_undegraded":"2024-04-23T14:44:01.318810+0000","last_fullsized":"2024-04-23T14:44:01.318810+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:36:16.225444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315304+0000","last_change":"2024-04-23T14:41:39.021681+0000","last_active":"2024-04-23T14:44:01.315304+0000","last_peered":"2024-04-23T14:44:01.315304+0000","last_clean":"2024-04-23T14:44:01.315304+0000","last_became_active":"2024-04-23T14:41:39.021423+0000","last_became_peered":"2024-04-23T14:41:39.021423+0000","last_unstale":"2024-04-23T14:44:01.315304+0000","last_undegraded":"2024-04-23T14:44:01.315304+0000","last_fullsized":"2024-04-23T14:44:01.315304+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:42:55.351296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.306731+0000","last_change":"2024-04-23T14:41:40.037250+0000","last_active":"2024-04-23T14:44:01.306731+0000","last_peered":"2024-04-23T14:44:01.306731+0000","last_clean":"2024-04-23T14:44:01.306731+0000","last_became_active":"2024-04-23T14:41:40.036645+0000","last_became_peered":"2024-04-23T14:41:40.036645+0000","last_unstale":"2024-04-23T14:44:01.306731+0000","last_undegraded":"2024-04-23T14:44:01.306731+0000","last_fullsized":"2024-04-23T14:44:01.306731+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:50:35.556841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314324+0000","last_change":"2024-04-23T14:41:39.007764+0000","last_active":"2024-04-23T14:44:01.314324+0000","last_peered":"2024-04-23T14:44:01.314324+0000","last_clean":"2024-04-23T14:44:01.314324+0000","last_became_active":"2024-04-23T14:41:39.007312+0000","last_became_peered":"2024-04-23T14:41:39.007312+0000","last_unstale":"2024-04-23T14:44:01.314324+0000","last_undegraded":"2024-04-23T14:44:01.314324+0000","last_fullsized":"2024-04-23T14:44:01.314324+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:38:56.021238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314309+0000","last_change":"2024-04-23T14:41:40.004343+0000","last_active":"2024-04-23T14:44:01.314309+0000","last_peered":"2024-04-23T14:44:01.314309+0000","last_clean":"2024-04-23T14:44:01.314309+0000","last_became_active":"2024-04-23T14:41:40.004015+0000","last_became_peered":"2024-04-23T14:41:40.004015+0000","last_unstale":"2024-04-23T14:44:01.314309+0000","last_undegraded":"2024-04-23T14:44:01.314309+0000","last_fullsized":"2024-04-23T14:44:01.314309+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:05:33.863692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.319064+0000","last_change":"2024-04-23T14:41:39.016271+0000","last_active":"2024-04-23T14:44:01.319064+0000","last_peered":"2024-04-23T14:44:01.319064+0000","last_clean":"2024-04-23T14:44:01.319064+0000","last_became_active":"2024-04-23T14:41:39.015851+0000","last_became_peered":"2024-04-23T14:41:39.015851+0000","last_unstale":"2024-04-23T14:44:01.319064+0000","last_undegraded":"2024-04-23T14:44:01.319064+0000","last_fullsized":"2024-04-23T14:44:01.319064+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:50:52.943534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316750+0000","last_change":"2024-04-23T14:41:40.029815+0000","last_active":"2024-04-23T14:44:01.316750+0000","last_peered":"2024-04-23T14:44:01.316750+0000","last_clean":"2024-04-23T14:44:01.316750+0000","last_became_active":"2024-04-23T14:41:40.029496+0000","last_became_peered":"2024-04-23T14:41:40.029496+0000","last_unstale":"2024-04-23T14:44:01.316750+0000","last_undegraded":"2024-04-23T14:44:01.316750+0000","last_fullsized":"2024-04-23T14:44:01.316750+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:36:09.228993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315232+0000","last_change":"2024-04-23T14:41:39.001946+0000","last_active":"2024-04-23T14:44:01.315232+0000","last_peered":"2024-04-23T14:44:01.315232+0000","last_clean":"2024-04-23T14:44:01.315232+0000","last_became_active":"2024-04-23T14:41:39.001619+0000","last_became_peered":"2024-04-23T14:41:39.001619+0000","last_unstale":"2024-04-23T14:44:01.315232+0000","last_undegraded":"2024-04-23T14:44:01.315232+0000","last_fullsized":"2024-04-23T14:44:01.315232+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:43:57.680195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.318758+0000","last_change":"2024-04-23T14:41:40.045728+0000","last_active":"2024-04-23T14:44:01.318758+0000","last_peered":"2024-04-23T14:44:01.318758+0000","last_clean":"2024-04-23T14:44:01.318758+0000","last_became_active":"2024-04-23T14:41:40.045293+0000","last_became_peered":"2024-04-23T14:41:40.045293+0000","last_unstale":"2024-04-23T14:44:01.318758+0000","last_undegraded":"2024-04-23T14:44:01.318758+0000","last_fullsized":"2024-04-23T14:44:01.318758+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:35:54.340644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314538+0000","last_change":"2024-04-23T14:42:43.823943+0000","last_active":"2024-04-23T14:44:01.314538+0000","last_peered":"2024-04-23T14:44:01.314538+0000","last_clean":"2024-04-23T14:44:01.314538+0000","last_became_active":"2024-04-23T14:42:43.821663+0000","last_became_peered":"2024-04-23T14:42:43.821663+0000","last_unstale":"2024-04-23T14:44:01.314538+0000","last_undegraded":"2024-04-23T14:44:01.314538+0000","last_fullsized":"2024-04-23T14:44:01.314538+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:18:54.895943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748380+0000","last_change":"2024-04-23T14:41:40.028897+0000","last_active":"2024-04-23T14:44:01.748380+0000","last_peered":"2024-04-23T14:44:01.748380+0000","last_clean":"2024-04-23T14:44:01.748380+0000","last_became_active":"2024-04-23T14:41:40.028557+0000","last_became_peered":"2024-04-23T14:41:40.028557+0000","last_unstale":"2024-04-23T14:44:01.748380+0000","last_undegraded":"2024-04-23T14:44:01.748380+0000","last_fullsized":"2024-04-23T14:44:01.748380+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:26:57.871013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315623+0000","last_change":"2024-04-23T14:41:39.012861+0000","last_active":"2024-04-23T14:44:01.315623+0000","last_peered":"2024-04-23T14:44:01.315623+0000","last_clean":"2024-04-23T14:44:01.315623+0000","last_became_active":"2024-04-23T14:41:39.012503+0000","last_became_peered":"2024-04-23T14:41:39.012503+0000","last_unstale":"2024-04-23T14:44:01.315623+0000","last_undegraded":"2024-04-23T14:44:01.315623+0000","last_fullsized":"2024-04-23T14:44:01.315623+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:41:57.347221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748833+0000","last_change":"2024-04-23T14:41:40.021580+0000","last_active":"2024-04-23T14:44:01.748833+0000","last_peered":"2024-04-23T14:44:01.748833+0000","last_clean":"2024-04-23T14:44:01.748833+0000","last_became_active":"2024-04-23T14:41:40.020706+0000","last_became_peered":"2024-04-23T14:41:40.020706+0000","last_unstale":"2024-04-23T14:44:01.748833+0000","last_undegraded":"2024-04-23T14:44:01.748833+0000","last_fullsized":"2024-04-23T14:44:01.748833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:59:42.576276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'39","reported_seq":69,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.599085+0000","last_change":"2024-04-23T14:41:39.006425+0000","last_active":"2024-04-23T14:44:01.599085+0000","last_peered":"2024-04-23T14:44:01.599085+0000","last_clean":"2024-04-23T14:44:01.599085+0000","last_became_active":"2024-04-23T14:41:39.006123+0000","last_became_peered":"2024-04-23T14:41:39.006123+0000","last_unstale":"2024-04-23T14:44:01.599085+0000","last_undegraded":"2024-04-23T14:44:01.599085+0000","last_fullsized":"2024-04-23T14:44:01.599085+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:08:37.261092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748644+0000","last_change":"2024-04-23T14:41:40.034063+0000","last_active":"2024-04-23T14:44:01.748644+0000","last_peered":"2024-04-23T14:44:01.748644+0000","last_clean":"2024-04-23T14:44:01.748644+0000","last_became_active":"2024-04-23T14:41:40.033669+0000","last_became_peered":"2024-04-23T14:41:40.033669+0000","last_unstale":"2024-04-23T14:44:01.748644+0000","last_undegraded":"2024-04-23T14:44:01.748644+0000","last_fullsized":"2024-04-23T14:44:01.748644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:23:50.608041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316265+0000","last_change":"2024-04-23T14:41:39.025975+0000","last_active":"2024-04-23T14:44:01.316265+0000","last_peered":"2024-04-23T14:44:01.316265+0000","last_clean":"2024-04-23T14:44:01.316265+0000","last_became_active":"2024-04-23T14:41:39.025629+0000","last_became_peered":"2024-04-23T14:41:39.025629+0000","last_unstale":"2024-04-23T14:44:01.316265+0000","last_undegraded":"2024-04-23T14:44:01.316265+0000","last_fullsized":"2024-04-23T14:44:01.316265+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:00:03.753547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748873+0000","last_change":"2024-04-23T14:41:40.013030+0000","last_active":"2024-04-23T14:44:01.748873+0000","last_peered":"2024-04-23T14:44:01.748873+0000","last_clean":"2024-04-23T14:44:01.748873+0000","last_became_active":"2024-04-23T14:41:40.012708+0000","last_became_peered":"2024-04-23T14:41:40.012708+0000","last_unstale":"2024-04-23T14:44:01.748873+0000","last_undegraded":"2024-04-23T14:44:01.748873+0000","last_fullsized":"2024-04-23T14:44:01.748873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:05:29.217273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316834+0000","last_change":"2024-04-23T14:41:40.010551+0000","last_active":"2024-04-23T14:44:01.316834+0000","last_peered":"2024-04-23T14:44:01.316834+0000","last_clean":"2024-04-23T14:44:01.316834+0000","last_became_active":"2024-04-23T14:41:40.010244+0000","last_became_peered":"2024-04-23T14:41:40.010244+0000","last_unstale":"2024-04-23T14:44:01.316834+0000","last_undegraded":"2024-04-23T14:44:01.316834+0000","last_fullsized":"2024-04-23T14:44:01.316834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:03:57.223815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315281+0000","last_change":"2024-04-23T14:41:40.009132+0000","last_active":"2024-04-23T14:44:01.315281+0000","last_peered":"2024-04-23T14:44:01.315281+0000","last_clean":"2024-04-23T14:44:01.315281+0000","last_became_active":"2024-04-23T14:41:40.008822+0000","last_became_peered":"2024-04-23T14:41:40.008822+0000","last_unstale":"2024-04-23T14:44:01.315281+0000","last_undegraded":"2024-04-23T14:44:01.315281+0000","last_fullsized":"2024-04-23T14:44:01.315281+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:10:28.513846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315222+0000","last_change":"2024-04-23T14:41:40.041193+0000","last_active":"2024-04-23T14:44:01.315222+0000","last_peered":"2024-04-23T14:44:01.315222+0000","last_clean":"2024-04-23T14:44:01.315222+0000","last_became_active":"2024-04-23T14:41:40.040851+0000","last_became_peered":"2024-04-23T14:41:40.040851+0000","last_unstale":"2024-04-23T14:44:01.315222+0000","last_undegraded":"2024-04-23T14:44:01.315222+0000","last_fullsized":"2024-04-23T14:44:01.315222+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:13:30.090015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.304852+0000","last_change":"2024-04-23T14:41:40.042703+0000","last_active":"2024-04-23T14:44:01.304852+0000","last_peered":"2024-04-23T14:44:01.304852+0000","last_clean":"2024-04-23T14:44:01.304852+0000","last_became_active":"2024-04-23T14:41:40.037666+0000","last_became_peered":"2024-04-23T14:41:40.037666+0000","last_unstale":"2024-04-23T14:44:01.304852+0000","last_undegraded":"2024-04-23T14:44:01.304852+0000","last_fullsized":"2024-04-23T14:44:01.304852+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:04:04.240689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316760+0000","last_change":"2024-04-23T14:41:40.033487+0000","last_active":"2024-04-23T14:44:01.316760+0000","last_peered":"2024-04-23T14:44:01.316760+0000","last_clean":"2024-04-23T14:44:01.316760+0000","last_became_active":"2024-04-23T14:41:40.031799+0000","last_became_peered":"2024-04-23T14:41:40.031799+0000","last_unstale":"2024-04-23T14:44:01.316760+0000","last_undegraded":"2024-04-23T14:44:01.316760+0000","last_fullsized":"2024-04-23T14:44:01.316760+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:36:03.930264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316943+0000","last_change":"2024-04-23T14:41:40.012833+0000","last_active":"2024-04-23T14:44:01.316943+0000","last_peered":"2024-04-23T14:44:01.316943+0000","last_clean":"2024-04-23T14:44:01.316943+0000","last_became_active":"2024-04-23T14:41:40.012251+0000","last_became_peered":"2024-04-23T14:41:40.012251+0000","last_unstale":"2024-04-23T14:44:01.316943+0000","last_undegraded":"2024-04-23T14:44:01.316943+0000","last_fullsized":"2024-04-23T14:44:01.316943+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:17:09.415998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315679+0000","last_change":"2024-04-23T14:41:40.013336+0000","last_active":"2024-04-23T14:44:01.315679+0000","last_peered":"2024-04-23T14:44:01.315679+0000","last_clean":"2024-04-23T14:44:01.315679+0000","last_became_active":"2024-04-23T14:41:40.013043+0000","last_became_peered":"2024-04-23T14:41:40.013043+0000","last_unstale":"2024-04-23T14:44:01.315679+0000","last_undegraded":"2024-04-23T14:44:01.315679+0000","last_fullsized":"2024-04-23T14:44:01.315679+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:27:59.118986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.318677+0000","last_change":"2024-04-23T14:41:40.028236+0000","last_active":"2024-04-23T14:44:01.318677+0000","last_peered":"2024-04-23T14:44:01.318677+0000","last_clean":"2024-04-23T14:44:01.318677+0000","last_became_active":"2024-04-23T14:41:40.027504+0000","last_became_peered":"2024-04-23T14:41:40.027504+0000","last_unstale":"2024-04-23T14:44:01.318677+0000","last_undegraded":"2024-04-23T14:44:01.318677+0000","last_fullsized":"2024-04-23T14:44:01.318677+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:04:31.540482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314699+0000","last_change":"2024-04-23T14:41:40.029252+0000","last_active":"2024-04-23T14:44:01.314699+0000","last_peered":"2024-04-23T14:44:01.314699+0000","last_clean":"2024-04-23T14:44:01.314699+0000","last_became_active":"2024-04-23T14:41:40.028937+0000","last_became_peered":"2024-04-23T14:41:40.028937+0000","last_unstale":"2024-04-23T14:44:01.314699+0000","last_undegraded":"2024-04-23T14:44:01.314699+0000","last_fullsized":"2024-04-23T14:44:01.314699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:46:44.441319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.318839+0000","last_change":"2024-04-23T14:41:40.027705+0000","last_active":"2024-04-23T14:44:01.318839+0000","last_peered":"2024-04-23T14:44:01.318839+0000","last_clean":"2024-04-23T14:44:01.318839+0000","last_became_active":"2024-04-23T14:41:40.027260+0000","last_became_peered":"2024-04-23T14:41:40.027260+0000","last_unstale":"2024-04-23T14:44:01.318839+0000","last_undegraded":"2024-04-23T14:44:01.318839+0000","last_fullsized":"2024-04-23T14:44:01.318839+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:33:42.461219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314679+0000","last_change":"2024-04-23T14:41:40.026490+0000","last_active":"2024-04-23T14:44:01.314679+0000","last_peered":"2024-04-23T14:44:01.314679+0000","last_clean":"2024-04-23T14:44:01.314679+0000","last_became_active":"2024-04-23T14:41:40.026117+0000","last_became_peered":"2024-04-23T14:41:40.026117+0000","last_unstale":"2024-04-23T14:44:01.314679+0000","last_undegraded":"2024-04-23T14:44:01.314679+0000","last_fullsized":"2024-04-23T14:44:01.314679+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:06:17.626663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316132+0000","last_change":"2024-04-23T14:41:40.038489+0000","last_active":"2024-04-23T14:44:01.316132+0000","last_peered":"2024-04-23T14:44:01.316132+0000","last_clean":"2024-04-23T14:44:01.316132+0000","last_became_active":"2024-04-23T14:41:40.038078+0000","last_became_peered":"2024-04-23T14:41:40.038078+0000","last_unstale":"2024-04-23T14:44:01.316132+0000","last_undegraded":"2024-04-23T14:44:01.316132+0000","last_fullsized":"2024-04-23T14:44:01.316132+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:58:59.440468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316974+0000","last_change":"2024-04-23T14:41:40.009463+0000","last_active":"2024-04-23T14:44:01.316974+0000","last_peered":"2024-04-23T14:44:01.316974+0000","last_clean":"2024-04-23T14:44:01.316974+0000","last_became_active":"2024-04-23T14:41:40.009154+0000","last_became_peered":"2024-04-23T14:41:40.009154+0000","last_unstale":"2024-04-23T14:44:01.316974+0000","last_undegraded":"2024-04-23T14:44:01.316974+0000","last_fullsized":"2024-04-23T14:44:01.316974+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:13:47.641863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":594224,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":386,"num_read_kb":290,"num_write":200,"num_write_kb":1405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1880064,"data_stored":1782672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383721,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":962560,"data_stored":464666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645358,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28760,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716392,"statfs":{"total":95995035648,"available":95965585408,"internally_reserved":0,"allocated":1785856,"data_stored":1255555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808883,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36668,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708484,"statfs":{"total":95995035648,"available":95957487616,"internally_reserved":0,"allocated":1359872,"data_stored":858464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070522,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28352,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716800,"statfs":{"total":95995035648,"available":95966003200,"internally_reserved":0,"allocated":1363968,"data_stored":858486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332160,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32708,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712444,"statfs":{"total":95995035648,"available":95961542656,"internally_reserved":0,"allocated":1564672,"data_stored":1055733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495685,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":464783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574946,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1036288,"data_stored":527121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738471,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36284,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708868,"statfs":{"total":95995035648,"available":95957880832,"internally_reserved":0,"allocated":966656,"data_stored":464756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411421,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1040384,"data_stored":527155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182486,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28488,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716664,"statfs":{"total":95995035648,"available":95965863936,"internally_reserved":0,"allocated":1441792,"data_stored":924276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920849,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":966656,"data_stored":464688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757323,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28864,"kb_used_data":1668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716288,"statfs":{"total":95995035648,"available":95965478912,"internally_reserved":0,"allocated":1708032,"data_stored":1186933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":68610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":68644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":536576,"data_stored":524920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":131782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":393820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":397312,"data_stored":393798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":200404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T14:44:04.946 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T14:44:05.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:04 smithi105 ceph-mon[32555]: [23/Apr/2024:14:44:02] ENGINE Bus STARTING 2024-04-23T14:44:05.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:04 smithi105 ceph-mon[32555]: [23/Apr/2024:14:44:02] ENGINE Serving on http://172.21.15.28:8765 2024-04-23T14:44:05.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:04 smithi105 ceph-mon[32555]: [23/Apr/2024:14:44:03] ENGINE Serving on https://172.21.15.28:7150 2024-04-23T14:44:05.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:04 smithi105 ceph-mon[32555]: [23/Apr/2024:14:44:03] ENGINE Bus STARTED 2024-04-23T14:44:05.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:04 smithi105 ceph-mon[32555]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:44:05.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:04 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:05.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:04 smithi105 ceph-mon[32555]: mgrmap e24: x(active, since 2s), standbys: z, y 2024-04-23T14:44:05.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:04 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:05.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:04 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/518493619' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T14:44:05.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:04 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1357640533' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:44:05.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:04 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:05.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.177+0000 7f886ea00640 1 -- 172.21.15.28:0/3632313831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8868104260 msgr2=0x7f8868104680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:05.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.177+0000 7f886ea00640 1 --2- 172.21.15.28:0/3632313831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8868104260 0x7f8868104680 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8858009900 tx=0x7f885802a410 comp rx=0 tx=0).stop 2024-04-23T14:44:05.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.177+0000 7f886ea00640 1 -- 172.21.15.28:0/3632313831 shutdown_connections 2024-04-23T14:44:05.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.177+0000 7f886ea00640 1 --2- 172.21.15.28:0/3632313831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8868105e10 0x7f886810c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:05.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.177+0000 7f886ea00640 1 --2- 172.21.15.28:0/3632313831 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8868105450 0x7f88681058d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:05.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.177+0000 7f886ea00640 1 --2- 172.21.15.28:0/3632313831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8868104260 0x7f8868104680 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:05.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.177+0000 7f886ea00640 1 -- 172.21.15.28:0/3632313831 >> 172.21.15.28:0/3632313831 conn(0x7f88680ff7f0 msgr2=0x7f8868101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:05.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.177+0000 7f886ea00640 1 -- 172.21.15.28:0/3632313831 shutdown_connections 2024-04-23T14:44:05.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.177+0000 7f886ea00640 1 -- 172.21.15.28:0/3632313831 wait complete. 2024-04-23T14:44:05.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.179+0000 7f886ea00640 1 Processor -- start 2024-04-23T14:44:05.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.179+0000 7f886ea00640 1 -- start start 2024-04-23T14:44:05.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.179+0000 7f886ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8868105450 0x7f886819a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:05.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.179+0000 7f886ea00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8868105e10 0x7f886819ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:05.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.179+0000 7f886ea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f886819fd80 0x7f88681a2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:05.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.179+0000 7f886ea00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f886810ff40 con 0x7f8868105450 2024-04-23T14:44:05.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.179+0000 7f886ea00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f886810fdc0 con 0x7f886819fd80 2024-04-23T14:44:05.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.179+0000 7f886ea00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f88681100c0 con 0x7f8868105e10 2024-04-23T14:44:05.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.180+0000 7f8867e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8868105450 0x7f886819a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:05.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.180+0000 7f8867400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8868105e10 0x7f886819ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:05.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.180+0000 7f8867e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8868105450 0x7f886819a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36326/0 (socket says 172.21.15.28:36326) 2024-04-23T14:44:05.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.180+0000 7f8867e00640 1 -- 172.21.15.28:0/1358227672 learned_addr learned my addr 172.21.15.28:0/1358227672 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:05.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.180+0000 7f886ca00640 1 --2- 172.21.15.28:0/1358227672 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f886819fd80 0x7f88681a2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:05.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.180+0000 7f8867400640 1 -- 172.21.15.28:0/1358227672 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f886819fd80 msgr2=0x7f88681a2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:05.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.180+0000 7f8867400640 1 --2- 172.21.15.28:0/1358227672 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f886819fd80 0x7f88681a2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:05.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.180+0000 7f8867400640 1 -- 172.21.15.28:0/1358227672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8868105450 msgr2=0x7f886819a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:05.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.180+0000 7f8867400640 1 --2- 172.21.15.28:0/1358227672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8868105450 0x7f886819a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:05.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.180+0000 7f8867400640 1 -- 172.21.15.28:0/1358227672 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88681a27e0 con 0x7f8868105e10 2024-04-23T14:44:05.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.180+0000 7f8867400640 1 --2- 172.21.15.28:0/1358227672 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8868105e10 0x7f886819ad20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f885400e9f0 tx=0x7f885400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:05.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.181+0000 7f8864c00640 1 -- 172.21.15.28:0/1358227672 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885400ccc0 con 0x7f8868105e10 2024-04-23T14:44:05.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.181+0000 7f886ea00640 1 -- 172.21.15.28:0/1358227672 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88681a2ad0 con 0x7f8868105e10 2024-04-23T14:44:05.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.181+0000 7f8864c00640 1 -- 172.21.15.28:0/1358227672 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8854005020 con 0x7f8868105e10 2024-04-23T14:44:05.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.181+0000 7f8864c00640 1 -- 172.21.15.28:0/1358227672 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88540049e0 con 0x7f8868105e10 2024-04-23T14:44:05.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.181+0000 7f886ea00640 1 -- 172.21.15.28:0/1358227672 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88681a3020 con 0x7f8868105e10 2024-04-23T14:44:05.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.182+0000 7f886ea00640 1 -- 172.21.15.28:0/1358227672 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f886810b6f0 con 0x7f8868105e10 2024-04-23T14:44:05.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.184+0000 7f8864c00640 1 -- 172.21.15.28:0/1358227672 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138766+0+0 (secure 0 0 0) 0x7f8854005560 con 0x7f8868105e10 2024-04-23T14:44:05.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.185+0000 7f8864c00640 1 --2- 172.21.15.28:0/1358227672 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f884c0a3d20 0x7f884c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:05.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.185+0000 7f8864c00640 1 -- 172.21.15.28:0/1358227672 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f88540d0870 con 0x7f8868105e10 2024-04-23T14:44:05.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.186+0000 7f8867e00640 1 --2- 172.21.15.28:0/1358227672 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f884c0a3d20 0x7f884c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:05.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.186+0000 7f8867e00640 1 --2- 172.21.15.28:0/1358227672 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f884c0a3d20 0x7f884c0a61e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8858004770 tx=0x7f88580315d0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:05.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.189+0000 7f8864c00640 1 -- 172.21.15.28:0/1358227672 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f88540051d0 con 0x7f8868105e10 2024-04-23T14:44:05.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.398+0000 7f886ea00640 1 -- 172.21.15.28:0/1358227672 --> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8868062440 con 0x7f884c0a3d20 2024-04-23T14:44:05.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.409+0000 7f8864c00640 1 -- 172.21.15.28:0/1358227672 <== mgr.25153 v2:172.21.15.28:6800/1615546338 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257706 (secure 0 0 0) 0x7f8868062440 con 0x7f884c0a3d20 2024-04-23T14:44:05.413 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:44:05.414 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-23T14:44:05.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.415+0000 7f886ea00640 1 -- 172.21.15.28:0/1358227672 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f884c0a3d20 msgr2=0x7f884c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:05.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.415+0000 7f886ea00640 1 --2- 172.21.15.28:0/1358227672 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f884c0a3d20 0x7f884c0a61e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8858004770 tx=0x7f88580315d0 comp rx=0 tx=0).stop 2024-04-23T14:44:05.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.415+0000 7f886ea00640 1 -- 172.21.15.28:0/1358227672 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8868105e10 msgr2=0x7f886819ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:05.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.415+0000 7f886ea00640 1 --2- 172.21.15.28:0/1358227672 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8868105e10 0x7f886819ad20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f885400e9f0 tx=0x7f885400eeb0 comp rx=0 tx=0).stop 2024-04-23T14:44:05.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.415+0000 7f886ea00640 1 -- 172.21.15.28:0/1358227672 shutdown_connections 2024-04-23T14:44:05.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.415+0000 7f886ea00640 1 --2- 172.21.15.28:0/1358227672 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f884c0a3d20 0x7f884c0a61e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:05.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.415+0000 7f886ea00640 1 --2- 172.21.15.28:0/1358227672 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f886819fd80 0x7f88681a2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:05.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.415+0000 7f886ea00640 1 --2- 172.21.15.28:0/1358227672 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8868105e10 0x7f886819ad20 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:05.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.415+0000 7f886ea00640 1 --2- 172.21.15.28:0/1358227672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8868105450 0x7f886819a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:05.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.415+0000 7f886ea00640 1 -- 172.21.15.28:0/1358227672 >> 172.21.15.28:0/1358227672 conn(0x7f88680ff7f0 msgr2=0x7f88681088c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:05.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.415+0000 7f886ea00640 1 -- 172.21.15.28:0/1358227672 shutdown_connections 2024-04-23T14:44:05.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.415+0000 7f886ea00640 1 -- 172.21.15.28:0/1358227672 wait complete. 2024-04-23T14:44:05.433 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-23T14:44:05.322311+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1184826,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":522,"num_read_kb":409,"num_write":383,"num_write_kb":4345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":787,"ondisk_log_size":787,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":380880,"kb_used_data":14808,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124560944,"statfs":{"total":1151940427776,"available":1151550406656,"internally_reserved":0,"allocated":15163392,"data_stored":9052616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.991716"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.317096+0000","last_change":"2024-04-23T14:41:40.033061+0000","last_active":"2024-04-23T14:44:01.317096+0000","last_peered":"2024-04-23T14:44:01.317096+0000","last_clean":"2024-04-23T14:44:01.317096+0000","last_became_active":"2024-04-23T14:41:40.032750+0000","last_became_peered":"2024-04-23T14:41:40.032750+0000","last_unstale":"2024-04-23T14:44:01.317096+0000","last_undegraded":"2024-04-23T14:44:01.317096+0000","last_fullsized":"2024-04-23T14:44:01.317096+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:58:03.451539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.306532+0000","last_change":"2024-04-23T14:41:40.037435+0000","last_active":"2024-04-23T14:44:01.306532+0000","last_peered":"2024-04-23T14:44:01.306532+0000","last_clean":"2024-04-23T14:44:01.306532+0000","last_became_active":"2024-04-23T14:41:40.036655+0000","last_became_peered":"2024-04-23T14:41:40.036655+0000","last_unstale":"2024-04-23T14:44:01.306532+0000","last_undegraded":"2024-04-23T14:44:01.306532+0000","last_fullsized":"2024-04-23T14:44:01.306532+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:51:58.847850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.306139+0000","last_change":"2024-04-23T14:41:40.024977+0000","last_active":"2024-04-23T14:44:01.306139+0000","last_peered":"2024-04-23T14:44:01.306139+0000","last_clean":"2024-04-23T14:44:01.306139+0000","last_became_active":"2024-04-23T14:41:40.023879+0000","last_became_peered":"2024-04-23T14:41:40.023879+0000","last_unstale":"2024-04-23T14:44:01.306139+0000","last_undegraded":"2024-04-23T14:44:01.306139+0000","last_fullsized":"2024-04-23T14:44:01.306139+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:03:40.672661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315735+0000","last_change":"2024-04-23T14:41:40.017757+0000","last_active":"2024-04-23T14:44:01.315735+0000","last_peered":"2024-04-23T14:44:01.315735+0000","last_clean":"2024-04-23T14:44:01.315735+0000","last_became_active":"2024-04-23T14:41:40.013996+0000","last_became_peered":"2024-04-23T14:41:40.013996+0000","last_unstale":"2024-04-23T14:44:01.315735+0000","last_undegraded":"2024-04-23T14:44:01.315735+0000","last_fullsized":"2024-04-23T14:44:01.315735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:20:03.376733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314329+0000","last_change":"2024-04-23T14:41:40.038247+0000","last_active":"2024-04-23T14:44:01.314329+0000","last_peered":"2024-04-23T14:44:01.314329+0000","last_clean":"2024-04-23T14:44:01.314329+0000","last_became_active":"2024-04-23T14:41:40.037934+0000","last_became_peered":"2024-04-23T14:41:40.037934+0000","last_unstale":"2024-04-23T14:44:01.314329+0000","last_undegraded":"2024-04-23T14:44:01.314329+0000","last_fullsized":"2024-04-23T14:44:01.314329+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:15.637733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315407+0000","last_change":"2024-04-23T14:41:40.015341+0000","last_active":"2024-04-23T14:44:01.315407+0000","last_peered":"2024-04-23T14:44:01.315407+0000","last_clean":"2024-04-23T14:44:01.315407+0000","last_became_active":"2024-04-23T14:41:40.013627+0000","last_became_peered":"2024-04-23T14:41:40.013627+0000","last_unstale":"2024-04-23T14:44:01.315407+0000","last_undegraded":"2024-04-23T14:44:01.315407+0000","last_fullsized":"2024-04-23T14:44:01.315407+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:05:36.248107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748243+0000","last_change":"2024-04-23T14:41:40.010950+0000","last_active":"2024-04-23T14:44:01.748243+0000","last_peered":"2024-04-23T14:44:01.748243+0000","last_clean":"2024-04-23T14:44:01.748243+0000","last_became_active":"2024-04-23T14:41:40.010584+0000","last_became_peered":"2024-04-23T14:41:40.010584+0000","last_unstale":"2024-04-23T14:44:01.748243+0000","last_undegraded":"2024-04-23T14:44:01.748243+0000","last_fullsized":"2024-04-23T14:44:01.748243+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:46:25.379243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.318133+0000","last_change":"2024-04-23T14:41:40.031198+0000","last_active":"2024-04-23T14:44:01.318133+0000","last_peered":"2024-04-23T14:44:01.318133+0000","last_clean":"2024-04-23T14:44:01.318133+0000","last_became_active":"2024-04-23T14:41:40.030652+0000","last_became_peered":"2024-04-23T14:41:40.030652+0000","last_unstale":"2024-04-23T14:44:01.318133+0000","last_undegraded":"2024-04-23T14:44:01.318133+0000","last_fullsized":"2024-04-23T14:44:01.318133+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:26:00.520182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315379+0000","last_change":"2024-04-23T14:41:40.009820+0000","last_active":"2024-04-23T14:44:01.315379+0000","last_peered":"2024-04-23T14:44:01.315379+0000","last_clean":"2024-04-23T14:44:01.315379+0000","last_became_active":"2024-04-23T14:41:40.009468+0000","last_became_peered":"2024-04-23T14:41:40.009468+0000","last_unstale":"2024-04-23T14:44:01.315379+0000","last_undegraded":"2024-04-23T14:44:01.315379+0000","last_fullsized":"2024-04-23T14:44:01.315379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:03:50.178050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316126+0000","last_change":"2024-04-23T14:41:40.008363+0000","last_active":"2024-04-23T14:44:01.316126+0000","last_peered":"2024-04-23T14:44:01.316126+0000","last_clean":"2024-04-23T14:44:01.316126+0000","last_became_active":"2024-04-23T14:41:40.006823+0000","last_became_peered":"2024-04-23T14:41:40.006823+0000","last_unstale":"2024-04-23T14:44:01.316126+0000","last_undegraded":"2024-04-23T14:44:01.316126+0000","last_fullsized":"2024-04-23T14:44:01.316126+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:12:33.718815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748598+0000","last_change":"2024-04-23T14:41:40.021069+0000","last_active":"2024-04-23T14:44:01.748598+0000","last_peered":"2024-04-23T14:44:01.748598+0000","last_clean":"2024-04-23T14:44:01.748598+0000","last_became_active":"2024-04-23T14:41:40.020334+0000","last_became_peered":"2024-04-23T14:41:40.020334+0000","last_unstale":"2024-04-23T14:44:01.748598+0000","last_undegraded":"2024-04-23T14:44:01.748598+0000","last_fullsized":"2024-04-23T14:44:01.748598+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:50:56.560928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315582+0000","last_change":"2024-04-23T14:41:40.017746+0000","last_active":"2024-04-23T14:44:01.315582+0000","last_peered":"2024-04-23T14:44:01.315582+0000","last_clean":"2024-04-23T14:44:01.315582+0000","last_became_active":"2024-04-23T14:41:40.014495+0000","last_became_peered":"2024-04-23T14:41:40.014495+0000","last_unstale":"2024-04-23T14:44:01.315582+0000","last_undegraded":"2024-04-23T14:44:01.315582+0000","last_fullsized":"2024-04-23T14:44:01.315582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:23:22.188110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316167+0000","last_change":"2024-04-23T14:41:40.021429+0000","last_active":"2024-04-23T14:44:01.316167+0000","last_peered":"2024-04-23T14:44:01.316167+0000","last_clean":"2024-04-23T14:44:01.316167+0000","last_became_active":"2024-04-23T14:41:40.021087+0000","last_became_peered":"2024-04-23T14:41:40.021087+0000","last_unstale":"2024-04-23T14:44:01.316167+0000","last_undegraded":"2024-04-23T14:44:01.316167+0000","last_fullsized":"2024-04-23T14:44:01.316167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:51:16.633253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.303276+0000","last_change":"2024-04-23T14:41:40.031839+0000","last_active":"2024-04-23T14:44:01.303276+0000","last_peered":"2024-04-23T14:44:01.303276+0000","last_clean":"2024-04-23T14:44:01.303276+0000","last_became_active":"2024-04-23T14:41:40.031463+0000","last_became_peered":"2024-04-23T14:41:40.031463+0000","last_unstale":"2024-04-23T14:44:01.303276+0000","last_undegraded":"2024-04-23T14:44:01.303276+0000","last_fullsized":"2024-04-23T14:44:01.303276+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:51:51.200463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.302427+0000","last_change":"2024-04-23T14:41:40.042575+0000","last_active":"2024-04-23T14:44:01.302427+0000","last_peered":"2024-04-23T14:44:01.302427+0000","last_clean":"2024-04-23T14:44:01.302427+0000","last_became_active":"2024-04-23T14:41:40.033758+0000","last_became_peered":"2024-04-23T14:41:40.033758+0000","last_unstale":"2024-04-23T14:44:01.302427+0000","last_undegraded":"2024-04-23T14:44:01.302427+0000","last_fullsized":"2024-04-23T14:44:01.302427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:55:17.338169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314525+0000","last_change":"2024-04-23T14:41:40.036214+0000","last_active":"2024-04-23T14:44:01.314525+0000","last_peered":"2024-04-23T14:44:01.314525+0000","last_clean":"2024-04-23T14:44:01.314525+0000","last_became_active":"2024-04-23T14:41:40.035892+0000","last_became_peered":"2024-04-23T14:41:40.035892+0000","last_unstale":"2024-04-23T14:44:01.314525+0000","last_undegraded":"2024-04-23T14:44:01.314525+0000","last_fullsized":"2024-04-23T14:44:01.314525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:29:18.668690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.317251+0000","last_change":"2024-04-23T14:41:40.029837+0000","last_active":"2024-04-23T14:44:01.317251+0000","last_peered":"2024-04-23T14:44:01.317251+0000","last_clean":"2024-04-23T14:44:01.317251+0000","last_became_active":"2024-04-23T14:41:40.029472+0000","last_became_peered":"2024-04-23T14:41:40.029472+0000","last_unstale":"2024-04-23T14:44:01.317251+0000","last_undegraded":"2024-04-23T14:44:01.317251+0000","last_fullsized":"2024-04-23T14:44:01.317251+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:28:52.194898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.304455+0000","last_change":"2024-04-23T14:41:40.015631+0000","last_active":"2024-04-23T14:44:01.304455+0000","last_peered":"2024-04-23T14:44:01.304455+0000","last_clean":"2024-04-23T14:44:01.304455+0000","last_became_active":"2024-04-23T14:41:40.015211+0000","last_became_peered":"2024-04-23T14:41:40.015211+0000","last_unstale":"2024-04-23T14:44:01.304455+0000","last_undegraded":"2024-04-23T14:44:01.304455+0000","last_fullsized":"2024-04-23T14:44:01.304455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:03:33.485364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315489+0000","last_change":"2024-04-23T14:42:43.680530+0000","last_active":"2024-04-23T14:44:01.315489+0000","last_peered":"2024-04-23T14:44:01.315489+0000","last_clean":"2024-04-23T14:44:01.315489+0000","last_became_active":"2024-04-23T14:42:43.680249+0000","last_became_peered":"2024-04-23T14:42:43.680249+0000","last_unstale":"2024-04-23T14:44:01.315489+0000","last_undegraded":"2024-04-23T14:44:01.315489+0000","last_fullsized":"2024-04-23T14:44:01.315489+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:55:36.334572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,6],"acting":[2,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.304507+0000","last_change":"2024-04-23T14:41:39.017031+0000","last_active":"2024-04-23T14:44:01.304507+0000","last_peered":"2024-04-23T14:44:01.304507+0000","last_clean":"2024-04-23T14:44:01.304507+0000","last_became_active":"2024-04-23T14:41:39.016699+0000","last_became_peered":"2024-04-23T14:41:39.016699+0000","last_unstale":"2024-04-23T14:44:01.304507+0000","last_undegraded":"2024-04-23T14:44:01.304507+0000","last_fullsized":"2024-04-23T14:44:01.304507+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:17:49.812601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.319643+0000","last_change":"2024-04-23T14:41:40.031957+0000","last_active":"2024-04-23T14:44:01.319643+0000","last_peered":"2024-04-23T14:44:01.319643+0000","last_clean":"2024-04-23T14:44:01.319643+0000","last_became_active":"2024-04-23T14:41:40.028151+0000","last_became_peered":"2024-04-23T14:41:40.028151+0000","last_unstale":"2024-04-23T14:44:01.319643+0000","last_undegraded":"2024-04-23T14:44:01.319643+0000","last_fullsized":"2024-04-23T14:44:01.319643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:46:48.455032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.319788+0000","last_change":"2024-04-23T14:41:39.022503+0000","last_active":"2024-04-23T14:44:01.319788+0000","last_peered":"2024-04-23T14:44:01.319788+0000","last_clean":"2024-04-23T14:44:01.319788+0000","last_became_active":"2024-04-23T14:41:39.022093+0000","last_became_peered":"2024-04-23T14:41:39.022093+0000","last_unstale":"2024-04-23T14:44:01.319788+0000","last_undegraded":"2024-04-23T14:44:01.319788+0000","last_fullsized":"2024-04-23T14:44:01.319788+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:48:27.861140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.306043+0000","last_change":"2024-04-23T14:41:40.032878+0000","last_active":"2024-04-23T14:44:01.306043+0000","last_peered":"2024-04-23T14:44:01.306043+0000","last_clean":"2024-04-23T14:44:01.306043+0000","last_became_active":"2024-04-23T14:41:40.027473+0000","last_became_peered":"2024-04-23T14:41:40.027473+0000","last_unstale":"2024-04-23T14:44:01.306043+0000","last_undegraded":"2024-04-23T14:44:01.306043+0000","last_fullsized":"2024-04-23T14:44:01.306043+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:00:17.677306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316901+0000","last_change":"2024-04-23T14:42:43.448728+0000","last_active":"2024-04-23T14:44:01.316901+0000","last_peered":"2024-04-23T14:44:01.316901+0000","last_clean":"2024-04-23T14:44:01.316901+0000","last_became_active":"2024-04-23T14:42:43.443993+0000","last_became_peered":"2024-04-23T14:42:43.443993+0000","last_unstale":"2024-04-23T14:44:01.316901+0000","last_undegraded":"2024-04-23T14:44:01.316901+0000","last_fullsized":"2024-04-23T14:44:01.316901+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:04:16.430822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314260+0000","last_change":"2024-04-23T14:41:40.019955+0000","last_active":"2024-04-23T14:44:01.314260+0000","last_peered":"2024-04-23T14:44:01.314260+0000","last_clean":"2024-04-23T14:44:01.314260+0000","last_became_active":"2024-04-23T14:41:40.019579+0000","last_became_peered":"2024-04-23T14:41:40.019579+0000","last_unstale":"2024-04-23T14:44:01.314260+0000","last_undegraded":"2024-04-23T14:44:01.314260+0000","last_fullsized":"2024-04-23T14:44:01.314260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:13:14.330051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316794+0000","last_change":"2024-04-23T14:42:43.446652+0000","last_active":"2024-04-23T14:44:01.316794+0000","last_peered":"2024-04-23T14:44:01.316794+0000","last_clean":"2024-04-23T14:44:01.316794+0000","last_became_active":"2024-04-23T14:42:43.443549+0000","last_became_peered":"2024-04-23T14:42:43.443549+0000","last_unstale":"2024-04-23T14:44:01.316794+0000","last_undegraded":"2024-04-23T14:44:01.316794+0000","last_fullsized":"2024-04-23T14:44:01.316794+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:31:12.927663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315219+0000","last_change":"2024-04-23T14:41:40.015592+0000","last_active":"2024-04-23T14:44:01.315219+0000","last_peered":"2024-04-23T14:44:01.315219+0000","last_clean":"2024-04-23T14:44:01.315219+0000","last_became_active":"2024-04-23T14:41:40.015292+0000","last_became_peered":"2024-04-23T14:41:40.015292+0000","last_unstale":"2024-04-23T14:44:01.315219+0000","last_undegraded":"2024-04-23T14:44:01.315219+0000","last_fullsized":"2024-04-23T14:44:01.315219+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:52:01.641962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314999+0000","last_change":"2024-04-23T14:41:39.008526+0000","last_active":"2024-04-23T14:44:01.314999+0000","last_peered":"2024-04-23T14:44:01.314999+0000","last_clean":"2024-04-23T14:44:01.314999+0000","last_became_active":"2024-04-23T14:41:39.008222+0000","last_became_peered":"2024-04-23T14:41:39.008222+0000","last_unstale":"2024-04-23T14:44:01.314999+0000","last_undegraded":"2024-04-23T14:44:01.314999+0000","last_fullsized":"2024-04-23T14:44:01.314999+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:23:09.191426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314352+0000","last_change":"2024-04-23T14:42:43.677846+0000","last_active":"2024-04-23T14:44:01.314352+0000","last_peered":"2024-04-23T14:44:01.314352+0000","last_clean":"2024-04-23T14:44:01.314352+0000","last_became_active":"2024-04-23T14:42:43.677413+0000","last_became_peered":"2024-04-23T14:42:43.677413+0000","last_unstale":"2024-04-23T14:44:01.314352+0000","last_undegraded":"2024-04-23T14:44:01.314352+0000","last_fullsized":"2024-04-23T14:44:01.314352+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:35:55.039607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315086+0000","last_change":"2024-04-23T14:41:39.028566+0000","last_active":"2024-04-23T14:44:01.315086+0000","last_peered":"2024-04-23T14:44:01.315086+0000","last_clean":"2024-04-23T14:44:01.315086+0000","last_became_active":"2024-04-23T14:41:39.028174+0000","last_became_peered":"2024-04-23T14:41:39.028174+0000","last_unstale":"2024-04-23T14:44:01.315086+0000","last_undegraded":"2024-04-23T14:44:01.315086+0000","last_fullsized":"2024-04-23T14:44:01.315086+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:50:55.424741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315025+0000","last_change":"2024-04-23T14:42:43.530152+0000","last_active":"2024-04-23T14:44:01.315025+0000","last_peered":"2024-04-23T14:44:01.315025+0000","last_clean":"2024-04-23T14:44:01.315025+0000","last_became_active":"2024-04-23T14:42:43.529776+0000","last_became_peered":"2024-04-23T14:42:43.529776+0000","last_unstale":"2024-04-23T14:44:01.315025+0000","last_undegraded":"2024-04-23T14:44:01.315025+0000","last_fullsized":"2024-04-23T14:44:01.315025+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:53:29.597081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315842+0000","last_change":"2024-04-23T14:41:39.028618+0000","last_active":"2024-04-23T14:44:01.315842+0000","last_peered":"2024-04-23T14:44:01.315842+0000","last_clean":"2024-04-23T14:44:01.315842+0000","last_became_active":"2024-04-23T14:41:39.028274+0000","last_became_peered":"2024-04-23T14:41:39.028274+0000","last_unstale":"2024-04-23T14:44:01.315842+0000","last_undegraded":"2024-04-23T14:44:01.315842+0000","last_fullsized":"2024-04-23T14:44:01.315842+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:14:31.571760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.318144+0000","last_change":"2024-04-23T14:41:40.020687+0000","last_active":"2024-04-23T14:44:01.318144+0000","last_peered":"2024-04-23T14:44:01.318144+0000","last_clean":"2024-04-23T14:44:01.318144+0000","last_became_active":"2024-04-23T14:41:40.020352+0000","last_became_peered":"2024-04-23T14:41:40.020352+0000","last_unstale":"2024-04-23T14:44:01.318144+0000","last_undegraded":"2024-04-23T14:44:01.318144+0000","last_fullsized":"2024-04-23T14:44:01.318144+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:29:37.013486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314616+0000","last_change":"2024-04-23T14:41:39.006002+0000","last_active":"2024-04-23T14:44:01.314616+0000","last_peered":"2024-04-23T14:44:01.314616+0000","last_clean":"2024-04-23T14:44:01.314616+0000","last_became_active":"2024-04-23T14:41:39.005415+0000","last_became_peered":"2024-04-23T14:41:39.005415+0000","last_unstale":"2024-04-23T14:44:01.314616+0000","last_undegraded":"2024-04-23T14:44:01.314616+0000","last_fullsized":"2024-04-23T14:44:01.314616+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:13:55.491931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.303902+0000","last_change":"2024-04-23T14:41:40.029393+0000","last_active":"2024-04-23T14:44:01.303902+0000","last_peered":"2024-04-23T14:44:01.303902+0000","last_clean":"2024-04-23T14:44:01.303902+0000","last_became_active":"2024-04-23T14:41:40.029039+0000","last_became_peered":"2024-04-23T14:41:40.029039+0000","last_unstale":"2024-04-23T14:44:01.303902+0000","last_undegraded":"2024-04-23T14:44:01.303902+0000","last_fullsized":"2024-04-23T14:44:01.303902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:39:24.461260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316197+0000","last_change":"2024-04-23T14:41:39.014638+0000","last_active":"2024-04-23T14:44:01.316197+0000","last_peered":"2024-04-23T14:44:01.316197+0000","last_clean":"2024-04-23T14:44:01.316197+0000","last_became_active":"2024-04-23T14:41:39.014289+0000","last_became_peered":"2024-04-23T14:41:39.014289+0000","last_unstale":"2024-04-23T14:44:01.316197+0000","last_undegraded":"2024-04-23T14:44:01.316197+0000","last_fullsized":"2024-04-23T14:44:01.316197+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:15:15.855399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315648+0000","last_change":"2024-04-23T14:41:40.015781+0000","last_active":"2024-04-23T14:44:01.315648+0000","last_peered":"2024-04-23T14:44:01.315648+0000","last_clean":"2024-04-23T14:44:01.315648+0000","last_became_active":"2024-04-23T14:41:40.015423+0000","last_became_peered":"2024-04-23T14:41:40.015423+0000","last_unstale":"2024-04-23T14:44:01.315648+0000","last_undegraded":"2024-04-23T14:44:01.315648+0000","last_fullsized":"2024-04-23T14:44:01.315648+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:56:55.947585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748648+0000","last_change":"2024-04-23T14:41:39.011021+0000","last_active":"2024-04-23T14:44:01.748648+0000","last_peered":"2024-04-23T14:44:01.748648+0000","last_clean":"2024-04-23T14:44:01.748648+0000","last_became_active":"2024-04-23T14:41:39.010637+0000","last_became_peered":"2024-04-23T14:41:39.010637+0000","last_unstale":"2024-04-23T14:44:01.748648+0000","last_undegraded":"2024-04-23T14:44:01.748648+0000","last_fullsized":"2024-04-23T14:44:01.748648+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:22:22.647170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.305882+0000","last_change":"2024-04-23T14:41:40.042797+0000","last_active":"2024-04-23T14:44:01.305882+0000","last_peered":"2024-04-23T14:44:01.305882+0000","last_clean":"2024-04-23T14:44:01.305882+0000","last_became_active":"2024-04-23T14:41:40.037797+0000","last_became_peered":"2024-04-23T14:41:40.037797+0000","last_unstale":"2024-04-23T14:44:01.305882+0000","last_undegraded":"2024-04-23T14:44:01.305882+0000","last_fullsized":"2024-04-23T14:44:01.305882+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:37:32.604276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315453+0000","last_change":"2024-04-23T14:41:39.010585+0000","last_active":"2024-04-23T14:44:01.315453+0000","last_peered":"2024-04-23T14:44:01.315453+0000","last_clean":"2024-04-23T14:44:01.315453+0000","last_became_active":"2024-04-23T14:41:39.010294+0000","last_became_peered":"2024-04-23T14:41:39.010294+0000","last_unstale":"2024-04-23T14:44:01.315453+0000","last_undegraded":"2024-04-23T14:44:01.315453+0000","last_fullsized":"2024-04-23T14:44:01.315453+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:38:03.146904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314580+0000","last_change":"2024-04-23T14:41:40.013150+0000","last_active":"2024-04-23T14:44:01.314580+0000","last_peered":"2024-04-23T14:44:01.314580+0000","last_clean":"2024-04-23T14:44:01.314580+0000","last_became_active":"2024-04-23T14:41:40.012310+0000","last_became_peered":"2024-04-23T14:41:40.012310+0000","last_unstale":"2024-04-23T14:44:01.314580+0000","last_undegraded":"2024-04-23T14:44:01.314580+0000","last_fullsized":"2024-04-23T14:44:01.314580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:45:25.777200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'33","reported_seq":343,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.706603+0000","last_change":"2024-04-23T14:41:39.004256+0000","last_active":"2024-04-23T14:44:01.706603+0000","last_peered":"2024-04-23T14:44:01.706603+0000","last_clean":"2024-04-23T14:44:01.706603+0000","last_became_active":"2024-04-23T14:41:39.003874+0000","last_became_peered":"2024-04-23T14:41:39.003874+0000","last_unstale":"2024-04-23T14:44:01.706603+0000","last_undegraded":"2024-04-23T14:44:01.706603+0000","last_fullsized":"2024-04-23T14:44:01.706603+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:38:30.727149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68610,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":288,"num_read_kb":205,"num_write":33,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314522+0000","last_change":"2024-04-23T14:41:40.012131+0000","last_active":"2024-04-23T14:44:01.314522+0000","last_peered":"2024-04-23T14:44:01.314522+0000","last_clean":"2024-04-23T14:44:01.314522+0000","last_became_active":"2024-04-23T14:41:40.011225+0000","last_became_peered":"2024-04-23T14:41:40.011225+0000","last_unstale":"2024-04-23T14:44:01.314522+0000","last_undegraded":"2024-04-23T14:44:01.314522+0000","last_fullsized":"2024-04-23T14:44:01.314522+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:36:12.085228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315400+0000","last_change":"2024-04-23T14:41:39.001935+0000","last_active":"2024-04-23T14:44:01.315400+0000","last_peered":"2024-04-23T14:44:01.315400+0000","last_clean":"2024-04-23T14:44:01.315400+0000","last_became_active":"2024-04-23T14:41:39.001583+0000","last_became_peered":"2024-04-23T14:41:39.001583+0000","last_unstale":"2024-04-23T14:44:01.315400+0000","last_undegraded":"2024-04-23T14:44:01.315400+0000","last_fullsized":"2024-04-23T14:44:01.315400+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:47:40.986280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315917+0000","last_change":"2024-04-23T14:41:40.036515+0000","last_active":"2024-04-23T14:44:01.315917+0000","last_peered":"2024-04-23T14:44:01.315917+0000","last_clean":"2024-04-23T14:44:01.315917+0000","last_became_active":"2024-04-23T14:41:40.036125+0000","last_became_peered":"2024-04-23T14:41:40.036125+0000","last_unstale":"2024-04-23T14:44:01.315917+0000","last_undegraded":"2024-04-23T14:44:01.315917+0000","last_fullsized":"2024-04-23T14:44:01.315917+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:55:28.465378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314793+0000","last_change":"2024-04-23T14:41:39.006127+0000","last_active":"2024-04-23T14:44:01.314793+0000","last_peered":"2024-04-23T14:44:01.314793+0000","last_clean":"2024-04-23T14:44:01.314793+0000","last_became_active":"2024-04-23T14:41:39.005761+0000","last_became_peered":"2024-04-23T14:41:39.005761+0000","last_unstale":"2024-04-23T14:44:01.314793+0000","last_undegraded":"2024-04-23T14:44:01.314793+0000","last_fullsized":"2024-04-23T14:44:01.314793+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:10:24.050016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748673+0000","last_change":"2024-04-23T14:42:43.678799+0000","last_active":"2024-04-23T14:44:01.748673+0000","last_peered":"2024-04-23T14:44:01.748673+0000","last_clean":"2024-04-23T14:44:01.748673+0000","last_became_active":"2024-04-23T14:42:43.678318+0000","last_became_peered":"2024-04-23T14:42:43.678318+0000","last_unstale":"2024-04-23T14:44:01.748673+0000","last_undegraded":"2024-04-23T14:44:01.748673+0000","last_fullsized":"2024-04-23T14:44:01.748673+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:26:21.827247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315145+0000","last_change":"2024-04-23T14:41:39.005873+0000","last_active":"2024-04-23T14:44:01.315145+0000","last_peered":"2024-04-23T14:44:01.315145+0000","last_clean":"2024-04-23T14:44:01.315145+0000","last_became_active":"2024-04-23T14:41:39.005481+0000","last_became_peered":"2024-04-23T14:41:39.005481+0000","last_unstale":"2024-04-23T14:44:01.315145+0000","last_undegraded":"2024-04-23T14:44:01.315145+0000","last_fullsized":"2024-04-23T14:44:01.315145+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:49:59.117451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748238+0000","last_change":"2024-04-23T14:41:40.009346+0000","last_active":"2024-04-23T14:44:01.748238+0000","last_peered":"2024-04-23T14:44:01.748238+0000","last_clean":"2024-04-23T14:44:01.748238+0000","last_became_active":"2024-04-23T14:41:40.008927+0000","last_became_peered":"2024-04-23T14:41:40.008927+0000","last_unstale":"2024-04-23T14:44:01.748238+0000","last_undegraded":"2024-04-23T14:44:01.748238+0000","last_fullsized":"2024-04-23T14:44:01.748238+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:33:36.513349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314285+0000","last_change":"2024-04-23T14:42:43.452178+0000","last_active":"2024-04-23T14:44:01.314285+0000","last_peered":"2024-04-23T14:44:01.314285+0000","last_clean":"2024-04-23T14:44:01.314285+0000","last_became_active":"2024-04-23T14:42:43.444786+0000","last_became_peered":"2024-04-23T14:42:43.444786+0000","last_unstale":"2024-04-23T14:44:01.314285+0000","last_undegraded":"2024-04-23T14:44:01.314285+0000","last_fullsized":"2024-04-23T14:44:01.314285+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:00:21.945663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,1],"acting":[8,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314546+0000","last_change":"2024-04-23T14:41:40.016845+0000","last_active":"2024-04-23T14:44:01.314546+0000","last_peered":"2024-04-23T14:44:01.314546+0000","last_clean":"2024-04-23T14:44:01.314546+0000","last_became_active":"2024-04-23T14:41:40.016543+0000","last_became_peered":"2024-04-23T14:41:40.016543+0000","last_unstale":"2024-04-23T14:44:01.314546+0000","last_undegraded":"2024-04-23T14:44:01.314546+0000","last_fullsized":"2024-04-23T14:44:01.314546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:52:41.054454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314792+0000","last_change":"2024-04-23T14:41:38.999968+0000","last_active":"2024-04-23T14:44:01.314792+0000","last_peered":"2024-04-23T14:44:01.314792+0000","last_clean":"2024-04-23T14:44:01.314792+0000","last_became_active":"2024-04-23T14:41:38.999658+0000","last_became_peered":"2024-04-23T14:41:38.999658+0000","last_unstale":"2024-04-23T14:44:01.314792+0000","last_undegraded":"2024-04-23T14:44:01.314792+0000","last_fullsized":"2024-04-23T14:44:01.314792+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:14:20.990331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.317006+0000","last_change":"2024-04-23T14:41:40.011753+0000","last_active":"2024-04-23T14:44:01.317006+0000","last_peered":"2024-04-23T14:44:01.317006+0000","last_clean":"2024-04-23T14:44:01.317006+0000","last_became_active":"2024-04-23T14:41:40.011264+0000","last_became_peered":"2024-04-23T14:41:40.011264+0000","last_unstale":"2024-04-23T14:44:01.317006+0000","last_undegraded":"2024-04-23T14:44:01.317006+0000","last_fullsized":"2024-04-23T14:44:01.317006+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:52:22.812046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.304344+0000","last_change":"2024-04-23T14:41:39.027384+0000","last_active":"2024-04-23T14:44:01.304344+0000","last_peered":"2024-04-23T14:44:01.304344+0000","last_clean":"2024-04-23T14:44:01.304344+0000","last_became_active":"2024-04-23T14:41:39.027018+0000","last_became_peered":"2024-04-23T14:41:39.027018+0000","last_unstale":"2024-04-23T14:44:01.304344+0000","last_undegraded":"2024-04-23T14:44:01.304344+0000","last_fullsized":"2024-04-23T14:44:01.304344+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:13:54.685704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315414+0000","last_change":"2024-04-23T14:41:40.041216+0000","last_active":"2024-04-23T14:44:01.315414+0000","last_peered":"2024-04-23T14:44:01.315414+0000","last_clean":"2024-04-23T14:44:01.315414+0000","last_became_active":"2024-04-23T14:41:40.040789+0000","last_became_peered":"2024-04-23T14:41:40.040789+0000","last_unstale":"2024-04-23T14:44:01.315414+0000","last_undegraded":"2024-04-23T14:44:01.315414+0000","last_fullsized":"2024-04-23T14:44:01.315414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:01:13.156704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748318+0000","last_change":"2024-04-23T14:41:39.024399+0000","last_active":"2024-04-23T14:44:01.748318+0000","last_peered":"2024-04-23T14:44:01.748318+0000","last_clean":"2024-04-23T14:44:01.748318+0000","last_became_active":"2024-04-23T14:41:39.024141+0000","last_became_peered":"2024-04-23T14:41:39.024141+0000","last_unstale":"2024-04-23T14:44:01.748318+0000","last_undegraded":"2024-04-23T14:44:01.748318+0000","last_fullsized":"2024-04-23T14:44:01.748318+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:34:27.516614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315148+0000","last_change":"2024-04-23T14:42:43.680818+0000","last_active":"2024-04-23T14:44:01.315148+0000","last_peered":"2024-04-23T14:44:01.315148+0000","last_clean":"2024-04-23T14:44:01.315148+0000","last_became_active":"2024-04-23T14:42:43.680463+0000","last_became_peered":"2024-04-23T14:42:43.680463+0000","last_unstale":"2024-04-23T14:44:01.315148+0000","last_undegraded":"2024-04-23T14:44:01.315148+0000","last_fullsized":"2024-04-23T14:44:01.315148+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:34:36.964359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314817+0000","last_change":"2024-04-23T14:41:40.014689+0000","last_active":"2024-04-23T14:44:01.314817+0000","last_peered":"2024-04-23T14:44:01.314817+0000","last_clean":"2024-04-23T14:44:01.314817+0000","last_became_active":"2024-04-23T14:41:40.014371+0000","last_became_peered":"2024-04-23T14:41:40.014371+0000","last_unstale":"2024-04-23T14:44:01.314817+0000","last_undegraded":"2024-04-23T14:44:01.314817+0000","last_fullsized":"2024-04-23T14:44:01.314817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:56:41.284144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315704+0000","last_change":"2024-04-23T14:41:39.031327+0000","last_active":"2024-04-23T14:44:01.315704+0000","last_peered":"2024-04-23T14:44:01.315704+0000","last_clean":"2024-04-23T14:44:01.315704+0000","last_became_active":"2024-04-23T14:41:39.030864+0000","last_became_peered":"2024-04-23T14:41:39.030864+0000","last_unstale":"2024-04-23T14:44:01.315704+0000","last_undegraded":"2024-04-23T14:44:01.315704+0000","last_fullsized":"2024-04-23T14:44:01.315704+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:58:24.048207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.319239+0000","last_change":"2024-04-23T14:41:40.035104+0000","last_active":"2024-04-23T14:44:01.319239+0000","last_peered":"2024-04-23T14:44:01.319239+0000","last_clean":"2024-04-23T14:44:01.319239+0000","last_became_active":"2024-04-23T14:41:40.034678+0000","last_became_peered":"2024-04-23T14:41:40.034678+0000","last_unstale":"2024-04-23T14:44:01.319239+0000","last_undegraded":"2024-04-23T14:44:01.319239+0000","last_fullsized":"2024-04-23T14:44:01.319239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:29:53.040424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314795+0000","last_change":"2024-04-23T14:41:39.016684+0000","last_active":"2024-04-23T14:44:01.314795+0000","last_peered":"2024-04-23T14:44:01.314795+0000","last_clean":"2024-04-23T14:44:01.314795+0000","last_became_active":"2024-04-23T14:41:39.016375+0000","last_became_peered":"2024-04-23T14:41:39.016375+0000","last_unstale":"2024-04-23T14:44:01.314795+0000","last_undegraded":"2024-04-23T14:44:01.314795+0000","last_fullsized":"2024-04-23T14:44:01.314795+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:09.449590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":357,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748262+0000","last_change":"2024-04-23T14:40:22.305413+0000","last_active":"2024-04-23T14:44:01.748262+0000","last_peered":"2024-04-23T14:44:01.748262+0000","last_clean":"2024-04-23T14:44:01.748262+0000","last_became_active":"2024-04-23T14:40:22.304977+0000","last_became_peered":"2024-04-23T14:40:22.304977+0000","last_unstale":"2024-04-23T14:44:01.748262+0000","last_undegraded":"2024-04-23T14:44:01.748262+0000","last_fullsized":"2024-04-23T14:44:01.748262+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:36:28.336753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:36:28.336753+0000","last_clean_scrub_stamp":"2024-04-23T14:36:28.336753+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:42:27.611770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315629+0000","last_change":"2024-04-23T14:41:40.023300+0000","last_active":"2024-04-23T14:44:01.315629+0000","last_peered":"2024-04-23T14:44:01.315629+0000","last_clean":"2024-04-23T14:44:01.315629+0000","last_became_active":"2024-04-23T14:41:40.022888+0000","last_became_peered":"2024-04-23T14:41:40.022888+0000","last_unstale":"2024-04-23T14:44:01.315629+0000","last_undegraded":"2024-04-23T14:44:01.315629+0000","last_fullsized":"2024-04-23T14:44:01.315629+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:31:48.705005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748412+0000","last_change":"2024-04-23T14:41:39.005482+0000","last_active":"2024-04-23T14:44:01.748412+0000","last_peered":"2024-04-23T14:44:01.748412+0000","last_clean":"2024-04-23T14:44:01.748412+0000","last_became_active":"2024-04-23T14:41:39.004552+0000","last_became_peered":"2024-04-23T14:41:39.004552+0000","last_unstale":"2024-04-23T14:44:01.748412+0000","last_undegraded":"2024-04-23T14:44:01.748412+0000","last_fullsized":"2024-04-23T14:44:01.748412+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:05:51.411210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314496+0000","last_change":"2024-04-23T14:41:39.015605+0000","last_active":"2024-04-23T14:44:01.314496+0000","last_peered":"2024-04-23T14:44:01.314496+0000","last_clean":"2024-04-23T14:44:01.314496+0000","last_became_active":"2024-04-23T14:41:39.015260+0000","last_became_peered":"2024-04-23T14:41:39.015260+0000","last_unstale":"2024-04-23T14:44:01.314496+0000","last_undegraded":"2024-04-23T14:44:01.314496+0000","last_fullsized":"2024-04-23T14:44:01.314496+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:23:21.251483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316641+0000","last_change":"2024-04-23T14:41:40.020619+0000","last_active":"2024-04-23T14:44:01.316641+0000","last_peered":"2024-04-23T14:44:01.316641+0000","last_clean":"2024-04-23T14:44:01.316641+0000","last_became_active":"2024-04-23T14:41:40.020191+0000","last_became_peered":"2024-04-23T14:41:40.020191+0000","last_unstale":"2024-04-23T14:44:01.316641+0000","last_undegraded":"2024-04-23T14:44:01.316641+0000","last_fullsized":"2024-04-23T14:44:01.316641+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:54:34.036464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'45","reported_seq":90,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:02.019133+0000","last_change":"2024-04-23T14:41:39.019507+0000","last_active":"2024-04-23T14:44:02.019133+0000","last_peered":"2024-04-23T14:44:02.019133+0000","last_clean":"2024-04-23T14:44:02.019133+0000","last_became_active":"2024-04-23T14:41:39.019116+0000","last_became_peered":"2024-04-23T14:41:39.019116+0000","last_unstale":"2024-04-23T14:44:02.019133+0000","last_undegraded":"2024-04-23T14:44:02.019133+0000","last_fullsized":"2024-04-23T14:44:02.019133+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:06:05.777908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":81,"num_write_kb":838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.318810+0000","last_change":"2024-04-23T14:41:40.032765+0000","last_active":"2024-04-23T14:44:01.318810+0000","last_peered":"2024-04-23T14:44:01.318810+0000","last_clean":"2024-04-23T14:44:01.318810+0000","last_became_active":"2024-04-23T14:41:40.027896+0000","last_became_peered":"2024-04-23T14:41:40.027896+0000","last_unstale":"2024-04-23T14:44:01.318810+0000","last_undegraded":"2024-04-23T14:44:01.318810+0000","last_fullsized":"2024-04-23T14:44:01.318810+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:36:16.225444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315304+0000","last_change":"2024-04-23T14:41:39.021681+0000","last_active":"2024-04-23T14:44:01.315304+0000","last_peered":"2024-04-23T14:44:01.315304+0000","last_clean":"2024-04-23T14:44:01.315304+0000","last_became_active":"2024-04-23T14:41:39.021423+0000","last_became_peered":"2024-04-23T14:41:39.021423+0000","last_unstale":"2024-04-23T14:44:01.315304+0000","last_undegraded":"2024-04-23T14:44:01.315304+0000","last_fullsized":"2024-04-23T14:44:01.315304+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:42:55.351296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.306731+0000","last_change":"2024-04-23T14:41:40.037250+0000","last_active":"2024-04-23T14:44:01.306731+0000","last_peered":"2024-04-23T14:44:01.306731+0000","last_clean":"2024-04-23T14:44:01.306731+0000","last_became_active":"2024-04-23T14:41:40.036645+0000","last_became_peered":"2024-04-23T14:41:40.036645+0000","last_unstale":"2024-04-23T14:44:01.306731+0000","last_undegraded":"2024-04-23T14:44:01.306731+0000","last_fullsized":"2024-04-23T14:44:01.306731+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:50:35.556841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314324+0000","last_change":"2024-04-23T14:41:39.007764+0000","last_active":"2024-04-23T14:44:01.314324+0000","last_peered":"2024-04-23T14:44:01.314324+0000","last_clean":"2024-04-23T14:44:01.314324+0000","last_became_active":"2024-04-23T14:41:39.007312+0000","last_became_peered":"2024-04-23T14:41:39.007312+0000","last_unstale":"2024-04-23T14:44:01.314324+0000","last_undegraded":"2024-04-23T14:44:01.314324+0000","last_fullsized":"2024-04-23T14:44:01.314324+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:38:56.021238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314309+0000","last_change":"2024-04-23T14:41:40.004343+0000","last_active":"2024-04-23T14:44:01.314309+0000","last_peered":"2024-04-23T14:44:01.314309+0000","last_clean":"2024-04-23T14:44:01.314309+0000","last_became_active":"2024-04-23T14:41:40.004015+0000","last_became_peered":"2024-04-23T14:41:40.004015+0000","last_unstale":"2024-04-23T14:44:01.314309+0000","last_undegraded":"2024-04-23T14:44:01.314309+0000","last_fullsized":"2024-04-23T14:44:01.314309+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:05:33.863692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.319064+0000","last_change":"2024-04-23T14:41:39.016271+0000","last_active":"2024-04-23T14:44:01.319064+0000","last_peered":"2024-04-23T14:44:01.319064+0000","last_clean":"2024-04-23T14:44:01.319064+0000","last_became_active":"2024-04-23T14:41:39.015851+0000","last_became_peered":"2024-04-23T14:41:39.015851+0000","last_unstale":"2024-04-23T14:44:01.319064+0000","last_undegraded":"2024-04-23T14:44:01.319064+0000","last_fullsized":"2024-04-23T14:44:01.319064+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:50:52.943534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316750+0000","last_change":"2024-04-23T14:41:40.029815+0000","last_active":"2024-04-23T14:44:01.316750+0000","last_peered":"2024-04-23T14:44:01.316750+0000","last_clean":"2024-04-23T14:44:01.316750+0000","last_became_active":"2024-04-23T14:41:40.029496+0000","last_became_peered":"2024-04-23T14:41:40.029496+0000","last_unstale":"2024-04-23T14:44:01.316750+0000","last_undegraded":"2024-04-23T14:44:01.316750+0000","last_fullsized":"2024-04-23T14:44:01.316750+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:36:09.228993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315232+0000","last_change":"2024-04-23T14:41:39.001946+0000","last_active":"2024-04-23T14:44:01.315232+0000","last_peered":"2024-04-23T14:44:01.315232+0000","last_clean":"2024-04-23T14:44:01.315232+0000","last_became_active":"2024-04-23T14:41:39.001619+0000","last_became_peered":"2024-04-23T14:41:39.001619+0000","last_unstale":"2024-04-23T14:44:01.315232+0000","last_undegraded":"2024-04-23T14:44:01.315232+0000","last_fullsized":"2024-04-23T14:44:01.315232+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:43:57.680195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.318758+0000","last_change":"2024-04-23T14:41:40.045728+0000","last_active":"2024-04-23T14:44:01.318758+0000","last_peered":"2024-04-23T14:44:01.318758+0000","last_clean":"2024-04-23T14:44:01.318758+0000","last_became_active":"2024-04-23T14:41:40.045293+0000","last_became_peered":"2024-04-23T14:41:40.045293+0000","last_unstale":"2024-04-23T14:44:01.318758+0000","last_undegraded":"2024-04-23T14:44:01.318758+0000","last_fullsized":"2024-04-23T14:44:01.318758+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:35:54.340644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314538+0000","last_change":"2024-04-23T14:42:43.823943+0000","last_active":"2024-04-23T14:44:01.314538+0000","last_peered":"2024-04-23T14:44:01.314538+0000","last_clean":"2024-04-23T14:44:01.314538+0000","last_became_active":"2024-04-23T14:42:43.821663+0000","last_became_peered":"2024-04-23T14:42:43.821663+0000","last_unstale":"2024-04-23T14:44:01.314538+0000","last_undegraded":"2024-04-23T14:44:01.314538+0000","last_fullsized":"2024-04-23T14:44:01.314538+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:18:54.895943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748380+0000","last_change":"2024-04-23T14:41:40.028897+0000","last_active":"2024-04-23T14:44:01.748380+0000","last_peered":"2024-04-23T14:44:01.748380+0000","last_clean":"2024-04-23T14:44:01.748380+0000","last_became_active":"2024-04-23T14:41:40.028557+0000","last_became_peered":"2024-04-23T14:41:40.028557+0000","last_unstale":"2024-04-23T14:44:01.748380+0000","last_undegraded":"2024-04-23T14:44:01.748380+0000","last_fullsized":"2024-04-23T14:44:01.748380+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:26:57.871013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315623+0000","last_change":"2024-04-23T14:41:39.012861+0000","last_active":"2024-04-23T14:44:01.315623+0000","last_peered":"2024-04-23T14:44:01.315623+0000","last_clean":"2024-04-23T14:44:01.315623+0000","last_became_active":"2024-04-23T14:41:39.012503+0000","last_became_peered":"2024-04-23T14:41:39.012503+0000","last_unstale":"2024-04-23T14:44:01.315623+0000","last_undegraded":"2024-04-23T14:44:01.315623+0000","last_fullsized":"2024-04-23T14:44:01.315623+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:41:57.347221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748833+0000","last_change":"2024-04-23T14:41:40.021580+0000","last_active":"2024-04-23T14:44:01.748833+0000","last_peered":"2024-04-23T14:44:01.748833+0000","last_clean":"2024-04-23T14:44:01.748833+0000","last_became_active":"2024-04-23T14:41:40.020706+0000","last_became_peered":"2024-04-23T14:41:40.020706+0000","last_unstale":"2024-04-23T14:44:01.748833+0000","last_undegraded":"2024-04-23T14:44:01.748833+0000","last_fullsized":"2024-04-23T14:44:01.748833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:59:42.576276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'39","reported_seq":69,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.599085+0000","last_change":"2024-04-23T14:41:39.006425+0000","last_active":"2024-04-23T14:44:01.599085+0000","last_peered":"2024-04-23T14:44:01.599085+0000","last_clean":"2024-04-23T14:44:01.599085+0000","last_became_active":"2024-04-23T14:41:39.006123+0000","last_became_peered":"2024-04-23T14:41:39.006123+0000","last_unstale":"2024-04-23T14:44:01.599085+0000","last_undegraded":"2024-04-23T14:44:01.599085+0000","last_fullsized":"2024-04-23T14:44:01.599085+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:08:37.261092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748644+0000","last_change":"2024-04-23T14:41:40.034063+0000","last_active":"2024-04-23T14:44:01.748644+0000","last_peered":"2024-04-23T14:44:01.748644+0000","last_clean":"2024-04-23T14:44:01.748644+0000","last_became_active":"2024-04-23T14:41:40.033669+0000","last_became_peered":"2024-04-23T14:41:40.033669+0000","last_unstale":"2024-04-23T14:44:01.748644+0000","last_undegraded":"2024-04-23T14:44:01.748644+0000","last_fullsized":"2024-04-23T14:44:01.748644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:23:50.608041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316265+0000","last_change":"2024-04-23T14:41:39.025975+0000","last_active":"2024-04-23T14:44:01.316265+0000","last_peered":"2024-04-23T14:44:01.316265+0000","last_clean":"2024-04-23T14:44:01.316265+0000","last_became_active":"2024-04-23T14:41:39.025629+0000","last_became_peered":"2024-04-23T14:41:39.025629+0000","last_unstale":"2024-04-23T14:44:01.316265+0000","last_undegraded":"2024-04-23T14:44:01.316265+0000","last_fullsized":"2024-04-23T14:44:01.316265+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:00:03.753547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748873+0000","last_change":"2024-04-23T14:41:40.013030+0000","last_active":"2024-04-23T14:44:01.748873+0000","last_peered":"2024-04-23T14:44:01.748873+0000","last_clean":"2024-04-23T14:44:01.748873+0000","last_became_active":"2024-04-23T14:41:40.012708+0000","last_became_peered":"2024-04-23T14:41:40.012708+0000","last_unstale":"2024-04-23T14:44:01.748873+0000","last_undegraded":"2024-04-23T14:44:01.748873+0000","last_fullsized":"2024-04-23T14:44:01.748873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:05:29.217273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316834+0000","last_change":"2024-04-23T14:41:40.010551+0000","last_active":"2024-04-23T14:44:01.316834+0000","last_peered":"2024-04-23T14:44:01.316834+0000","last_clean":"2024-04-23T14:44:01.316834+0000","last_became_active":"2024-04-23T14:41:40.010244+0000","last_became_peered":"2024-04-23T14:41:40.010244+0000","last_unstale":"2024-04-23T14:44:01.316834+0000","last_undegraded":"2024-04-23T14:44:01.316834+0000","last_fullsized":"2024-04-23T14:44:01.316834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:03:57.223815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315281+0000","last_change":"2024-04-23T14:41:40.009132+0000","last_active":"2024-04-23T14:44:01.315281+0000","last_peered":"2024-04-23T14:44:01.315281+0000","last_clean":"2024-04-23T14:44:01.315281+0000","last_became_active":"2024-04-23T14:41:40.008822+0000","last_became_peered":"2024-04-23T14:41:40.008822+0000","last_unstale":"2024-04-23T14:44:01.315281+0000","last_undegraded":"2024-04-23T14:44:01.315281+0000","last_fullsized":"2024-04-23T14:44:01.315281+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:10:28.513846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315222+0000","last_change":"2024-04-23T14:41:40.041193+0000","last_active":"2024-04-23T14:44:01.315222+0000","last_peered":"2024-04-23T14:44:01.315222+0000","last_clean":"2024-04-23T14:44:01.315222+0000","last_became_active":"2024-04-23T14:41:40.040851+0000","last_became_peered":"2024-04-23T14:41:40.040851+0000","last_unstale":"2024-04-23T14:44:01.315222+0000","last_undegraded":"2024-04-23T14:44:01.315222+0000","last_fullsized":"2024-04-23T14:44:01.315222+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:13:30.090015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.304852+0000","last_change":"2024-04-23T14:41:40.042703+0000","last_active":"2024-04-23T14:44:01.304852+0000","last_peered":"2024-04-23T14:44:01.304852+0000","last_clean":"2024-04-23T14:44:01.304852+0000","last_became_active":"2024-04-23T14:41:40.037666+0000","last_became_peered":"2024-04-23T14:41:40.037666+0000","last_unstale":"2024-04-23T14:44:01.304852+0000","last_undegraded":"2024-04-23T14:44:01.304852+0000","last_fullsized":"2024-04-23T14:44:01.304852+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:04:04.240689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316760+0000","last_change":"2024-04-23T14:41:40.033487+0000","last_active":"2024-04-23T14:44:01.316760+0000","last_peered":"2024-04-23T14:44:01.316760+0000","last_clean":"2024-04-23T14:44:01.316760+0000","last_became_active":"2024-04-23T14:41:40.031799+0000","last_became_peered":"2024-04-23T14:41:40.031799+0000","last_unstale":"2024-04-23T14:44:01.316760+0000","last_undegraded":"2024-04-23T14:44:01.316760+0000","last_fullsized":"2024-04-23T14:44:01.316760+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:36:03.930264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316943+0000","last_change":"2024-04-23T14:41:40.012833+0000","last_active":"2024-04-23T14:44:01.316943+0000","last_peered":"2024-04-23T14:44:01.316943+0000","last_clean":"2024-04-23T14:44:01.316943+0000","last_became_active":"2024-04-23T14:41:40.012251+0000","last_became_peered":"2024-04-23T14:41:40.012251+0000","last_unstale":"2024-04-23T14:44:01.316943+0000","last_undegraded":"2024-04-23T14:44:01.316943+0000","last_fullsized":"2024-04-23T14:44:01.316943+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:17:09.415998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315679+0000","last_change":"2024-04-23T14:41:40.013336+0000","last_active":"2024-04-23T14:44:01.315679+0000","last_peered":"2024-04-23T14:44:01.315679+0000","last_clean":"2024-04-23T14:44:01.315679+0000","last_became_active":"2024-04-23T14:41:40.013043+0000","last_became_peered":"2024-04-23T14:41:40.013043+0000","last_unstale":"2024-04-23T14:44:01.315679+0000","last_undegraded":"2024-04-23T14:44:01.315679+0000","last_fullsized":"2024-04-23T14:44:01.315679+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:27:59.118986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.318677+0000","last_change":"2024-04-23T14:41:40.028236+0000","last_active":"2024-04-23T14:44:01.318677+0000","last_peered":"2024-04-23T14:44:01.318677+0000","last_clean":"2024-04-23T14:44:01.318677+0000","last_became_active":"2024-04-23T14:41:40.027504+0000","last_became_peered":"2024-04-23T14:41:40.027504+0000","last_unstale":"2024-04-23T14:44:01.318677+0000","last_undegraded":"2024-04-23T14:44:01.318677+0000","last_fullsized":"2024-04-23T14:44:01.318677+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:04:31.540482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314699+0000","last_change":"2024-04-23T14:41:40.029252+0000","last_active":"2024-04-23T14:44:01.314699+0000","last_peered":"2024-04-23T14:44:01.314699+0000","last_clean":"2024-04-23T14:44:01.314699+0000","last_became_active":"2024-04-23T14:41:40.028937+0000","last_became_peered":"2024-04-23T14:41:40.028937+0000","last_unstale":"2024-04-23T14:44:01.314699+0000","last_undegraded":"2024-04-23T14:44:01.314699+0000","last_fullsized":"2024-04-23T14:44:01.314699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:46:44.441319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.318839+0000","last_change":"2024-04-23T14:41:40.027705+0000","last_active":"2024-04-23T14:44:01.318839+0000","last_peered":"2024-04-23T14:44:01.318839+0000","last_clean":"2024-04-23T14:44:01.318839+0000","last_became_active":"2024-04-23T14:41:40.027260+0000","last_became_peered":"2024-04-23T14:41:40.027260+0000","last_unstale":"2024-04-23T14:44:01.318839+0000","last_undegraded":"2024-04-23T14:44:01.318839+0000","last_fullsized":"2024-04-23T14:44:01.318839+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:33:42.461219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314679+0000","last_change":"2024-04-23T14:41:40.026490+0000","last_active":"2024-04-23T14:44:01.314679+0000","last_peered":"2024-04-23T14:44:01.314679+0000","last_clean":"2024-04-23T14:44:01.314679+0000","last_became_active":"2024-04-23T14:41:40.026117+0000","last_became_peered":"2024-04-23T14:41:40.026117+0000","last_unstale":"2024-04-23T14:44:01.314679+0000","last_undegraded":"2024-04-23T14:44:01.314679+0000","last_fullsized":"2024-04-23T14:44:01.314679+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:06:17.626663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316132+0000","last_change":"2024-04-23T14:41:40.038489+0000","last_active":"2024-04-23T14:44:01.316132+0000","last_peered":"2024-04-23T14:44:01.316132+0000","last_clean":"2024-04-23T14:44:01.316132+0000","last_became_active":"2024-04-23T14:41:40.038078+0000","last_became_peered":"2024-04-23T14:41:40.038078+0000","last_unstale":"2024-04-23T14:44:01.316132+0000","last_undegraded":"2024-04-23T14:44:01.316132+0000","last_fullsized":"2024-04-23T14:44:01.316132+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:58:59.440468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316974+0000","last_change":"2024-04-23T14:41:40.009463+0000","last_active":"2024-04-23T14:44:01.316974+0000","last_peered":"2024-04-23T14:44:01.316974+0000","last_clean":"2024-04-23T14:44:01.316974+0000","last_became_active":"2024-04-23T14:41:40.009154+0000","last_became_peered":"2024-04-23T14:41:40.009154+0000","last_unstale":"2024-04-23T14:44:01.316974+0000","last_undegraded":"2024-04-23T14:44:01.316974+0000","last_fullsized":"2024-04-23T14:44:01.316974+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:13:47.641863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":594224,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":386,"num_read_kb":290,"num_write":200,"num_write_kb":1405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1880064,"data_stored":1782672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383721,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":962560,"data_stored":464666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645358,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28760,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716392,"statfs":{"total":95995035648,"available":95965585408,"internally_reserved":0,"allocated":1785856,"data_stored":1255555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808883,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36668,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708484,"statfs":{"total":95995035648,"available":95957487616,"internally_reserved":0,"allocated":1359872,"data_stored":858464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070522,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28352,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716800,"statfs":{"total":95995035648,"available":95966003200,"internally_reserved":0,"allocated":1363968,"data_stored":858486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332160,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32708,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712444,"statfs":{"total":95995035648,"available":95961542656,"internally_reserved":0,"allocated":1564672,"data_stored":1055733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495685,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":464783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574946,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1036288,"data_stored":527121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738471,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36284,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708868,"statfs":{"total":95995035648,"available":95957880832,"internally_reserved":0,"allocated":966656,"data_stored":464756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411421,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1040384,"data_stored":527155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182486,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28488,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716664,"statfs":{"total":95995035648,"available":95965863936,"internally_reserved":0,"allocated":1441792,"data_stored":924276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920849,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":966656,"data_stored":464688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757323,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28864,"kb_used_data":1668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716288,"statfs":{"total":95995035648,"available":95965478912,"internally_reserved":0,"allocated":1708032,"data_stored":1186933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":68610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":68644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":536576,"data_stored":524920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":131782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":393820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":397312,"data_stored":393798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":200404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T14:44:05.436 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-04-23T14:44:05.436 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T14:44:05.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.661+0000 7fd304600640 1 -- 172.21.15.28:0/2368442208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2fc102070 msgr2=0x7fd2fc102490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:05.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.661+0000 7fd304600640 1 --2- 172.21.15.28:0/2368442208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2fc102070 0x7fd2fc102490 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0009900 tx=0x7fd2f002a3f0 comp rx=0 tx=0).stop 2024-04-23T14:44:05.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.661+0000 7fd304600640 1 -- 172.21.15.28:0/2368442208 shutdown_connections 2024-04-23T14:44:05.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.661+0000 7fd304600640 1 --2- 172.21.15.28:0/2368442208 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd2fc103c20 0x7fd2fc10a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:05.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.661+0000 7fd304600640 1 --2- 172.21.15.28:0/2368442208 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd2fc103260 0x7fd2fc1036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:05.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.661+0000 7fd304600640 1 --2- 172.21.15.28:0/2368442208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2fc102070 0x7fd2fc102490 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:05.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.661+0000 7fd304600640 1 -- 172.21.15.28:0/2368442208 >> 172.21.15.28:0/2368442208 conn(0x7fd2fc0fd600 msgr2=0x7fd2fc0ffa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:05.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.661+0000 7fd304600640 1 -- 172.21.15.28:0/2368442208 shutdown_connections 2024-04-23T14:44:05.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.661+0000 7fd304600640 1 -- 172.21.15.28:0/2368442208 wait complete. 2024-04-23T14:44:05.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.662+0000 7fd304600640 1 Processor -- start 2024-04-23T14:44:05.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.663+0000 7fd304600640 1 -- start start 2024-04-23T14:44:05.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.663+0000 7fd304600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2fc102070 0x7fd2fc19a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:05.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.663+0000 7fd304600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd2fc103260 0x7fd2fc19ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:05.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.663+0000 7fd304600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd2fc103c20 0x7fd2fc1a1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:05.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.663+0000 7fd304600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd2fc10dd50 con 0x7fd2fc102070 2024-04-23T14:44:05.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.663+0000 7fd304600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd2fc10dbd0 con 0x7fd2fc103260 2024-04-23T14:44:05.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.663+0000 7fd304600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd2fc10ded0 con 0x7fd2fc103c20 2024-04-23T14:44:05.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.663+0000 7fd301c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2fc102070 0x7fd2fc19a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:05.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.663+0000 7fd301c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2fc102070 0x7fd2fc19a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36338/0 (socket says 172.21.15.28:36338) 2024-04-23T14:44:05.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.663+0000 7fd301200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd2fc103260 0x7fd2fc19ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:05.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.663+0000 7fd301c00640 1 -- 172.21.15.28:0/3863035119 learned_addr learned my addr 172.21.15.28:0/3863035119 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:05.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.664+0000 7fd302600640 1 --2- 172.21.15.28:0/3863035119 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd2fc103c20 0x7fd2fc1a1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:05.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.664+0000 7fd301c00640 1 -- 172.21.15.28:0/3863035119 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd2fc103c20 msgr2=0x7fd2fc1a1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:05.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.664+0000 7fd301c00640 1 --2- 172.21.15.28:0/3863035119 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd2fc103c20 0x7fd2fc1a1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:05.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.664+0000 7fd301c00640 1 -- 172.21.15.28:0/3863035119 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd2fc103260 msgr2=0x7fd2fc19ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:05.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.664+0000 7fd301c00640 1 --2- 172.21.15.28:0/3863035119 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd2fc103260 0x7fd2fc19ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:05.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.664+0000 7fd301c00640 1 -- 172.21.15.28:0/3863035119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2fc1a21e0 con 0x7fd2fc102070 2024-04-23T14:44:05.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.664+0000 7fd301200640 1 --2- 172.21.15.28:0/3863035119 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd2fc103260 0x7fd2fc19ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:44:05.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.664+0000 7fd301c00640 1 --2- 172.21.15.28:0/3863035119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2fc102070 0x7fd2fc19a5c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0031440 tx=0x7fd2f0032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:05.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.664+0000 7fd2fa000640 1 -- 172.21.15.28:0/3863035119 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f003f030 con 0x7fd2fc102070 2024-04-23T14:44:05.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.665+0000 7fd2fa000640 1 -- 172.21.15.28:0/3863035119 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2f0003ce0 con 0x7fd2fc102070 2024-04-23T14:44:05.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.665+0000 7fd304600640 1 -- 172.21.15.28:0/3863035119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2fc1a2470 con 0x7fd2fc102070 2024-04-23T14:44:05.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.665+0000 7fd2fa000640 1 -- 172.21.15.28:0/3863035119 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f0035070 con 0x7fd2fc102070 2024-04-23T14:44:05.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.665+0000 7fd304600640 1 -- 172.21.15.28:0/3863035119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2fc1a2960 con 0x7fd2fc102070 2024-04-23T14:44:05.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.666+0000 7fd304600640 1 -- 172.21.15.28:0/3863035119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2fc1a2bc0 con 0x7fd2fc102070 2024-04-23T14:44:05.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.668+0000 7fd2fa000640 1 -- 172.21.15.28:0/3863035119 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138766+0+0 (secure 0 0 0) 0x7fd2f0042030 con 0x7fd2fc102070 2024-04-23T14:44:05.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.669+0000 7fd2fa000640 1 --2- 172.21.15.28:0/3863035119 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd2e40a3d20 0x7fd2e40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:05.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.669+0000 7fd2fa000640 1 -- 172.21.15.28:0/3863035119 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fd2f00f0ca0 con 0x7fd2fc102070 2024-04-23T14:44:05.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.670+0000 7fd301200640 1 --2- 172.21.15.28:0/3863035119 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd2e40a3d20 0x7fd2e40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:05.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.671+0000 7fd301200640 1 --2- 172.21.15.28:0/3863035119 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd2e40a3d20 0x7fd2e40a61e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc19bad0 tx=0x7fd2e8005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:05.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.674+0000 7fd2fa000640 1 -- 172.21.15.28:0/3863035119 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd2f00bcca0 con 0x7fd2fc102070 2024-04-23T14:44:05.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:05 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:05.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:05 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:05.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:05 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:05.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:05 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:05.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:05 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:05.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:05 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:05.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:05 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:05.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:05 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:05.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:05 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:05.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:05 smithi028 ceph-mon[27810]: Adjusting osd_memory_target on smithi105 to 176.2M 2024-04-23T14:44:05.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:05 smithi028 ceph-mon[27810]: Unable to set osd_memory_target on smithi105 to 184783872: error parsing value: Value '184783872' is below minimum 939524096 2024-04-23T14:44:05.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:05 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:05.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:05 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:05.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:05 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:05.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:05 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:05.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:05 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:05.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:05 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:05.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:05 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:05.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:05 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:05.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:05 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:05.877 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:05 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:05.877 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:05 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:05.877 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:05 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:05.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.890+0000 7fd304600640 1 -- 172.21.15.28:0/3863035119 --> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd2fc075950 con 0x7fd2e40a3d20 2024-04-23T14:44:05.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.916+0000 7fd2fa000640 1 -- 172.21.15.28:0/3863035119 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7fd2f0003800 con 0x7fd2fc102070 2024-04-23T14:44:05.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.919+0000 7fd2fa000640 1 -- 172.21.15.28:0/3863035119 <== mgr.25153 v2:172.21.15.28:6800/1615546338 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257706 (secure 0 0 0) 0x7fd2fc075950 con 0x7fd2e40a3d20 2024-04-23T14:44:05.923 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:44:05.925 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-23T14:44:05.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.925+0000 7fd304600640 1 -- 172.21.15.28:0/3863035119 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd2e40a3d20 msgr2=0x7fd2e40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:05.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.925+0000 7fd304600640 1 --2- 172.21.15.28:0/3863035119 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd2e40a3d20 0x7fd2e40a61e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc19bad0 tx=0x7fd2e8005f90 comp rx=0 tx=0).stop 2024-04-23T14:44:05.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.925+0000 7fd304600640 1 -- 172.21.15.28:0/3863035119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2fc102070 msgr2=0x7fd2fc19a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:05.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.925+0000 7fd304600640 1 --2- 172.21.15.28:0/3863035119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2fc102070 0x7fd2fc19a5c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0031440 tx=0x7fd2f0032ce0 comp rx=0 tx=0).stop 2024-04-23T14:44:05.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.925+0000 7fd304600640 1 -- 172.21.15.28:0/3863035119 shutdown_connections 2024-04-23T14:44:05.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.925+0000 7fd304600640 1 --2- 172.21.15.28:0/3863035119 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd2e40a3d20 0x7fd2e40a61e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:05.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.925+0000 7fd304600640 1 --2- 172.21.15.28:0/3863035119 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd2fc103260 0x7fd2fc19ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:05.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.925+0000 7fd304600640 1 --2- 172.21.15.28:0/3863035119 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd2fc103c20 0x7fd2fc1a1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:05.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.925+0000 7fd304600640 1 --2- 172.21.15.28:0/3863035119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2fc102070 0x7fd2fc19a5c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:05.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.925+0000 7fd304600640 1 -- 172.21.15.28:0/3863035119 >> 172.21.15.28:0/3863035119 conn(0x7fd2fc0fd600 msgr2=0x7fd2fc074230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:05.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.926+0000 7fd304600640 1 -- 172.21.15.28:0/3863035119 shutdown_connections 2024-04-23T14:44:05.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:05.926+0000 7fd304600640 1 -- 172.21.15.28:0/3863035119 wait complete. 2024-04-23T14:44:05.945 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-23T14:44:05.322311+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1184826,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":522,"num_read_kb":409,"num_write":383,"num_write_kb":4345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":787,"ondisk_log_size":787,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":380880,"kb_used_data":14808,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124560944,"statfs":{"total":1151940427776,"available":1151550406656,"internally_reserved":0,"allocated":15163392,"data_stored":9052616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.991716"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.317096+0000","last_change":"2024-04-23T14:41:40.033061+0000","last_active":"2024-04-23T14:44:01.317096+0000","last_peered":"2024-04-23T14:44:01.317096+0000","last_clean":"2024-04-23T14:44:01.317096+0000","last_became_active":"2024-04-23T14:41:40.032750+0000","last_became_peered":"2024-04-23T14:41:40.032750+0000","last_unstale":"2024-04-23T14:44:01.317096+0000","last_undegraded":"2024-04-23T14:44:01.317096+0000","last_fullsized":"2024-04-23T14:44:01.317096+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:58:03.451539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.306532+0000","last_change":"2024-04-23T14:41:40.037435+0000","last_active":"2024-04-23T14:44:01.306532+0000","last_peered":"2024-04-23T14:44:01.306532+0000","last_clean":"2024-04-23T14:44:01.306532+0000","last_became_active":"2024-04-23T14:41:40.036655+0000","last_became_peered":"2024-04-23T14:41:40.036655+0000","last_unstale":"2024-04-23T14:44:01.306532+0000","last_undegraded":"2024-04-23T14:44:01.306532+0000","last_fullsized":"2024-04-23T14:44:01.306532+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:51:58.847850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.306139+0000","last_change":"2024-04-23T14:41:40.024977+0000","last_active":"2024-04-23T14:44:01.306139+0000","last_peered":"2024-04-23T14:44:01.306139+0000","last_clean":"2024-04-23T14:44:01.306139+0000","last_became_active":"2024-04-23T14:41:40.023879+0000","last_became_peered":"2024-04-23T14:41:40.023879+0000","last_unstale":"2024-04-23T14:44:01.306139+0000","last_undegraded":"2024-04-23T14:44:01.306139+0000","last_fullsized":"2024-04-23T14:44:01.306139+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:03:40.672661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315735+0000","last_change":"2024-04-23T14:41:40.017757+0000","last_active":"2024-04-23T14:44:01.315735+0000","last_peered":"2024-04-23T14:44:01.315735+0000","last_clean":"2024-04-23T14:44:01.315735+0000","last_became_active":"2024-04-23T14:41:40.013996+0000","last_became_peered":"2024-04-23T14:41:40.013996+0000","last_unstale":"2024-04-23T14:44:01.315735+0000","last_undegraded":"2024-04-23T14:44:01.315735+0000","last_fullsized":"2024-04-23T14:44:01.315735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:20:03.376733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314329+0000","last_change":"2024-04-23T14:41:40.038247+0000","last_active":"2024-04-23T14:44:01.314329+0000","last_peered":"2024-04-23T14:44:01.314329+0000","last_clean":"2024-04-23T14:44:01.314329+0000","last_became_active":"2024-04-23T14:41:40.037934+0000","last_became_peered":"2024-04-23T14:41:40.037934+0000","last_unstale":"2024-04-23T14:44:01.314329+0000","last_undegraded":"2024-04-23T14:44:01.314329+0000","last_fullsized":"2024-04-23T14:44:01.314329+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:15.637733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315407+0000","last_change":"2024-04-23T14:41:40.015341+0000","last_active":"2024-04-23T14:44:01.315407+0000","last_peered":"2024-04-23T14:44:01.315407+0000","last_clean":"2024-04-23T14:44:01.315407+0000","last_became_active":"2024-04-23T14:41:40.013627+0000","last_became_peered":"2024-04-23T14:41:40.013627+0000","last_unstale":"2024-04-23T14:44:01.315407+0000","last_undegraded":"2024-04-23T14:44:01.315407+0000","last_fullsized":"2024-04-23T14:44:01.315407+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:05:36.248107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748243+0000","last_change":"2024-04-23T14:41:40.010950+0000","last_active":"2024-04-23T14:44:01.748243+0000","last_peered":"2024-04-23T14:44:01.748243+0000","last_clean":"2024-04-23T14:44:01.748243+0000","last_became_active":"2024-04-23T14:41:40.010584+0000","last_became_peered":"2024-04-23T14:41:40.010584+0000","last_unstale":"2024-04-23T14:44:01.748243+0000","last_undegraded":"2024-04-23T14:44:01.748243+0000","last_fullsized":"2024-04-23T14:44:01.748243+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:46:25.379243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.318133+0000","last_change":"2024-04-23T14:41:40.031198+0000","last_active":"2024-04-23T14:44:01.318133+0000","last_peered":"2024-04-23T14:44:01.318133+0000","last_clean":"2024-04-23T14:44:01.318133+0000","last_became_active":"2024-04-23T14:41:40.030652+0000","last_became_peered":"2024-04-23T14:41:40.030652+0000","last_unstale":"2024-04-23T14:44:01.318133+0000","last_undegraded":"2024-04-23T14:44:01.318133+0000","last_fullsized":"2024-04-23T14:44:01.318133+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:26:00.520182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315379+0000","last_change":"2024-04-23T14:41:40.009820+0000","last_active":"2024-04-23T14:44:01.315379+0000","last_peered":"2024-04-23T14:44:01.315379+0000","last_clean":"2024-04-23T14:44:01.315379+0000","last_became_active":"2024-04-23T14:41:40.009468+0000","last_became_peered":"2024-04-23T14:41:40.009468+0000","last_unstale":"2024-04-23T14:44:01.315379+0000","last_undegraded":"2024-04-23T14:44:01.315379+0000","last_fullsized":"2024-04-23T14:44:01.315379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:03:50.178050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316126+0000","last_change":"2024-04-23T14:41:40.008363+0000","last_active":"2024-04-23T14:44:01.316126+0000","last_peered":"2024-04-23T14:44:01.316126+0000","last_clean":"2024-04-23T14:44:01.316126+0000","last_became_active":"2024-04-23T14:41:40.006823+0000","last_became_peered":"2024-04-23T14:41:40.006823+0000","last_unstale":"2024-04-23T14:44:01.316126+0000","last_undegraded":"2024-04-23T14:44:01.316126+0000","last_fullsized":"2024-04-23T14:44:01.316126+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:12:33.718815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748598+0000","last_change":"2024-04-23T14:41:40.021069+0000","last_active":"2024-04-23T14:44:01.748598+0000","last_peered":"2024-04-23T14:44:01.748598+0000","last_clean":"2024-04-23T14:44:01.748598+0000","last_became_active":"2024-04-23T14:41:40.020334+0000","last_became_peered":"2024-04-23T14:41:40.020334+0000","last_unstale":"2024-04-23T14:44:01.748598+0000","last_undegraded":"2024-04-23T14:44:01.748598+0000","last_fullsized":"2024-04-23T14:44:01.748598+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:50:56.560928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315582+0000","last_change":"2024-04-23T14:41:40.017746+0000","last_active":"2024-04-23T14:44:01.315582+0000","last_peered":"2024-04-23T14:44:01.315582+0000","last_clean":"2024-04-23T14:44:01.315582+0000","last_became_active":"2024-04-23T14:41:40.014495+0000","last_became_peered":"2024-04-23T14:41:40.014495+0000","last_unstale":"2024-04-23T14:44:01.315582+0000","last_undegraded":"2024-04-23T14:44:01.315582+0000","last_fullsized":"2024-04-23T14:44:01.315582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:23:22.188110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316167+0000","last_change":"2024-04-23T14:41:40.021429+0000","last_active":"2024-04-23T14:44:01.316167+0000","last_peered":"2024-04-23T14:44:01.316167+0000","last_clean":"2024-04-23T14:44:01.316167+0000","last_became_active":"2024-04-23T14:41:40.021087+0000","last_became_peered":"2024-04-23T14:41:40.021087+0000","last_unstale":"2024-04-23T14:44:01.316167+0000","last_undegraded":"2024-04-23T14:44:01.316167+0000","last_fullsized":"2024-04-23T14:44:01.316167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:51:16.633253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.303276+0000","last_change":"2024-04-23T14:41:40.031839+0000","last_active":"2024-04-23T14:44:01.303276+0000","last_peered":"2024-04-23T14:44:01.303276+0000","last_clean":"2024-04-23T14:44:01.303276+0000","last_became_active":"2024-04-23T14:41:40.031463+0000","last_became_peered":"2024-04-23T14:41:40.031463+0000","last_unstale":"2024-04-23T14:44:01.303276+0000","last_undegraded":"2024-04-23T14:44:01.303276+0000","last_fullsized":"2024-04-23T14:44:01.303276+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:51:51.200463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.302427+0000","last_change":"2024-04-23T14:41:40.042575+0000","last_active":"2024-04-23T14:44:01.302427+0000","last_peered":"2024-04-23T14:44:01.302427+0000","last_clean":"2024-04-23T14:44:01.302427+0000","last_became_active":"2024-04-23T14:41:40.033758+0000","last_became_peered":"2024-04-23T14:41:40.033758+0000","last_unstale":"2024-04-23T14:44:01.302427+0000","last_undegraded":"2024-04-23T14:44:01.302427+0000","last_fullsized":"2024-04-23T14:44:01.302427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:55:17.338169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314525+0000","last_change":"2024-04-23T14:41:40.036214+0000","last_active":"2024-04-23T14:44:01.314525+0000","last_peered":"2024-04-23T14:44:01.314525+0000","last_clean":"2024-04-23T14:44:01.314525+0000","last_became_active":"2024-04-23T14:41:40.035892+0000","last_became_peered":"2024-04-23T14:41:40.035892+0000","last_unstale":"2024-04-23T14:44:01.314525+0000","last_undegraded":"2024-04-23T14:44:01.314525+0000","last_fullsized":"2024-04-23T14:44:01.314525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:29:18.668690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.317251+0000","last_change":"2024-04-23T14:41:40.029837+0000","last_active":"2024-04-23T14:44:01.317251+0000","last_peered":"2024-04-23T14:44:01.317251+0000","last_clean":"2024-04-23T14:44:01.317251+0000","last_became_active":"2024-04-23T14:41:40.029472+0000","last_became_peered":"2024-04-23T14:41:40.029472+0000","last_unstale":"2024-04-23T14:44:01.317251+0000","last_undegraded":"2024-04-23T14:44:01.317251+0000","last_fullsized":"2024-04-23T14:44:01.317251+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:28:52.194898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.304455+0000","last_change":"2024-04-23T14:41:40.015631+0000","last_active":"2024-04-23T14:44:01.304455+0000","last_peered":"2024-04-23T14:44:01.304455+0000","last_clean":"2024-04-23T14:44:01.304455+0000","last_became_active":"2024-04-23T14:41:40.015211+0000","last_became_peered":"2024-04-23T14:41:40.015211+0000","last_unstale":"2024-04-23T14:44:01.304455+0000","last_undegraded":"2024-04-23T14:44:01.304455+0000","last_fullsized":"2024-04-23T14:44:01.304455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:03:33.485364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315489+0000","last_change":"2024-04-23T14:42:43.680530+0000","last_active":"2024-04-23T14:44:01.315489+0000","last_peered":"2024-04-23T14:44:01.315489+0000","last_clean":"2024-04-23T14:44:01.315489+0000","last_became_active":"2024-04-23T14:42:43.680249+0000","last_became_peered":"2024-04-23T14:42:43.680249+0000","last_unstale":"2024-04-23T14:44:01.315489+0000","last_undegraded":"2024-04-23T14:44:01.315489+0000","last_fullsized":"2024-04-23T14:44:01.315489+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:55:36.334572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,6],"acting":[2,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.304507+0000","last_change":"2024-04-23T14:41:39.017031+0000","last_active":"2024-04-23T14:44:01.304507+0000","last_peered":"2024-04-23T14:44:01.304507+0000","last_clean":"2024-04-23T14:44:01.304507+0000","last_became_active":"2024-04-23T14:41:39.016699+0000","last_became_peered":"2024-04-23T14:41:39.016699+0000","last_unstale":"2024-04-23T14:44:01.304507+0000","last_undegraded":"2024-04-23T14:44:01.304507+0000","last_fullsized":"2024-04-23T14:44:01.304507+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:17:49.812601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.319643+0000","last_change":"2024-04-23T14:41:40.031957+0000","last_active":"2024-04-23T14:44:01.319643+0000","last_peered":"2024-04-23T14:44:01.319643+0000","last_clean":"2024-04-23T14:44:01.319643+0000","last_became_active":"2024-04-23T14:41:40.028151+0000","last_became_peered":"2024-04-23T14:41:40.028151+0000","last_unstale":"2024-04-23T14:44:01.319643+0000","last_undegraded":"2024-04-23T14:44:01.319643+0000","last_fullsized":"2024-04-23T14:44:01.319643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:46:48.455032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.319788+0000","last_change":"2024-04-23T14:41:39.022503+0000","last_active":"2024-04-23T14:44:01.319788+0000","last_peered":"2024-04-23T14:44:01.319788+0000","last_clean":"2024-04-23T14:44:01.319788+0000","last_became_active":"2024-04-23T14:41:39.022093+0000","last_became_peered":"2024-04-23T14:41:39.022093+0000","last_unstale":"2024-04-23T14:44:01.319788+0000","last_undegraded":"2024-04-23T14:44:01.319788+0000","last_fullsized":"2024-04-23T14:44:01.319788+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:48:27.861140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.306043+0000","last_change":"2024-04-23T14:41:40.032878+0000","last_active":"2024-04-23T14:44:01.306043+0000","last_peered":"2024-04-23T14:44:01.306043+0000","last_clean":"2024-04-23T14:44:01.306043+0000","last_became_active":"2024-04-23T14:41:40.027473+0000","last_became_peered":"2024-04-23T14:41:40.027473+0000","last_unstale":"2024-04-23T14:44:01.306043+0000","last_undegraded":"2024-04-23T14:44:01.306043+0000","last_fullsized":"2024-04-23T14:44:01.306043+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:00:17.677306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316901+0000","last_change":"2024-04-23T14:42:43.448728+0000","last_active":"2024-04-23T14:44:01.316901+0000","last_peered":"2024-04-23T14:44:01.316901+0000","last_clean":"2024-04-23T14:44:01.316901+0000","last_became_active":"2024-04-23T14:42:43.443993+0000","last_became_peered":"2024-04-23T14:42:43.443993+0000","last_unstale":"2024-04-23T14:44:01.316901+0000","last_undegraded":"2024-04-23T14:44:01.316901+0000","last_fullsized":"2024-04-23T14:44:01.316901+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:04:16.430822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314260+0000","last_change":"2024-04-23T14:41:40.019955+0000","last_active":"2024-04-23T14:44:01.314260+0000","last_peered":"2024-04-23T14:44:01.314260+0000","last_clean":"2024-04-23T14:44:01.314260+0000","last_became_active":"2024-04-23T14:41:40.019579+0000","last_became_peered":"2024-04-23T14:41:40.019579+0000","last_unstale":"2024-04-23T14:44:01.314260+0000","last_undegraded":"2024-04-23T14:44:01.314260+0000","last_fullsized":"2024-04-23T14:44:01.314260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:13:14.330051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316794+0000","last_change":"2024-04-23T14:42:43.446652+0000","last_active":"2024-04-23T14:44:01.316794+0000","last_peered":"2024-04-23T14:44:01.316794+0000","last_clean":"2024-04-23T14:44:01.316794+0000","last_became_active":"2024-04-23T14:42:43.443549+0000","last_became_peered":"2024-04-23T14:42:43.443549+0000","last_unstale":"2024-04-23T14:44:01.316794+0000","last_undegraded":"2024-04-23T14:44:01.316794+0000","last_fullsized":"2024-04-23T14:44:01.316794+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:31:12.927663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315219+0000","last_change":"2024-04-23T14:41:40.015592+0000","last_active":"2024-04-23T14:44:01.315219+0000","last_peered":"2024-04-23T14:44:01.315219+0000","last_clean":"2024-04-23T14:44:01.315219+0000","last_became_active":"2024-04-23T14:41:40.015292+0000","last_became_peered":"2024-04-23T14:41:40.015292+0000","last_unstale":"2024-04-23T14:44:01.315219+0000","last_undegraded":"2024-04-23T14:44:01.315219+0000","last_fullsized":"2024-04-23T14:44:01.315219+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:52:01.641962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314999+0000","last_change":"2024-04-23T14:41:39.008526+0000","last_active":"2024-04-23T14:44:01.314999+0000","last_peered":"2024-04-23T14:44:01.314999+0000","last_clean":"2024-04-23T14:44:01.314999+0000","last_became_active":"2024-04-23T14:41:39.008222+0000","last_became_peered":"2024-04-23T14:41:39.008222+0000","last_unstale":"2024-04-23T14:44:01.314999+0000","last_undegraded":"2024-04-23T14:44:01.314999+0000","last_fullsized":"2024-04-23T14:44:01.314999+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:23:09.191426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314352+0000","last_change":"2024-04-23T14:42:43.677846+0000","last_active":"2024-04-23T14:44:01.314352+0000","last_peered":"2024-04-23T14:44:01.314352+0000","last_clean":"2024-04-23T14:44:01.314352+0000","last_became_active":"2024-04-23T14:42:43.677413+0000","last_became_peered":"2024-04-23T14:42:43.677413+0000","last_unstale":"2024-04-23T14:44:01.314352+0000","last_undegraded":"2024-04-23T14:44:01.314352+0000","last_fullsized":"2024-04-23T14:44:01.314352+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:35:55.039607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315086+0000","last_change":"2024-04-23T14:41:39.028566+0000","last_active":"2024-04-23T14:44:01.315086+0000","last_peered":"2024-04-23T14:44:01.315086+0000","last_clean":"2024-04-23T14:44:01.315086+0000","last_became_active":"2024-04-23T14:41:39.028174+0000","last_became_peered":"2024-04-23T14:41:39.028174+0000","last_unstale":"2024-04-23T14:44:01.315086+0000","last_undegraded":"2024-04-23T14:44:01.315086+0000","last_fullsized":"2024-04-23T14:44:01.315086+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:50:55.424741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315025+0000","last_change":"2024-04-23T14:42:43.530152+0000","last_active":"2024-04-23T14:44:01.315025+0000","last_peered":"2024-04-23T14:44:01.315025+0000","last_clean":"2024-04-23T14:44:01.315025+0000","last_became_active":"2024-04-23T14:42:43.529776+0000","last_became_peered":"2024-04-23T14:42:43.529776+0000","last_unstale":"2024-04-23T14:44:01.315025+0000","last_undegraded":"2024-04-23T14:44:01.315025+0000","last_fullsized":"2024-04-23T14:44:01.315025+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:53:29.597081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315842+0000","last_change":"2024-04-23T14:41:39.028618+0000","last_active":"2024-04-23T14:44:01.315842+0000","last_peered":"2024-04-23T14:44:01.315842+0000","last_clean":"2024-04-23T14:44:01.315842+0000","last_became_active":"2024-04-23T14:41:39.028274+0000","last_became_peered":"2024-04-23T14:41:39.028274+0000","last_unstale":"2024-04-23T14:44:01.315842+0000","last_undegraded":"2024-04-23T14:44:01.315842+0000","last_fullsized":"2024-04-23T14:44:01.315842+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:14:31.571760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.318144+0000","last_change":"2024-04-23T14:41:40.020687+0000","last_active":"2024-04-23T14:44:01.318144+0000","last_peered":"2024-04-23T14:44:01.318144+0000","last_clean":"2024-04-23T14:44:01.318144+0000","last_became_active":"2024-04-23T14:41:40.020352+0000","last_became_peered":"2024-04-23T14:41:40.020352+0000","last_unstale":"2024-04-23T14:44:01.318144+0000","last_undegraded":"2024-04-23T14:44:01.318144+0000","last_fullsized":"2024-04-23T14:44:01.318144+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:29:37.013486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314616+0000","last_change":"2024-04-23T14:41:39.006002+0000","last_active":"2024-04-23T14:44:01.314616+0000","last_peered":"2024-04-23T14:44:01.314616+0000","last_clean":"2024-04-23T14:44:01.314616+0000","last_became_active":"2024-04-23T14:41:39.005415+0000","last_became_peered":"2024-04-23T14:41:39.005415+0000","last_unstale":"2024-04-23T14:44:01.314616+0000","last_undegraded":"2024-04-23T14:44:01.314616+0000","last_fullsized":"2024-04-23T14:44:01.314616+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:13:55.491931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.303902+0000","last_change":"2024-04-23T14:41:40.029393+0000","last_active":"2024-04-23T14:44:01.303902+0000","last_peered":"2024-04-23T14:44:01.303902+0000","last_clean":"2024-04-23T14:44:01.303902+0000","last_became_active":"2024-04-23T14:41:40.029039+0000","last_became_peered":"2024-04-23T14:41:40.029039+0000","last_unstale":"2024-04-23T14:44:01.303902+0000","last_undegraded":"2024-04-23T14:44:01.303902+0000","last_fullsized":"2024-04-23T14:44:01.303902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:39:24.461260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316197+0000","last_change":"2024-04-23T14:41:39.014638+0000","last_active":"2024-04-23T14:44:01.316197+0000","last_peered":"2024-04-23T14:44:01.316197+0000","last_clean":"2024-04-23T14:44:01.316197+0000","last_became_active":"2024-04-23T14:41:39.014289+0000","last_became_peered":"2024-04-23T14:41:39.014289+0000","last_unstale":"2024-04-23T14:44:01.316197+0000","last_undegraded":"2024-04-23T14:44:01.316197+0000","last_fullsized":"2024-04-23T14:44:01.316197+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:15:15.855399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315648+0000","last_change":"2024-04-23T14:41:40.015781+0000","last_active":"2024-04-23T14:44:01.315648+0000","last_peered":"2024-04-23T14:44:01.315648+0000","last_clean":"2024-04-23T14:44:01.315648+0000","last_became_active":"2024-04-23T14:41:40.015423+0000","last_became_peered":"2024-04-23T14:41:40.015423+0000","last_unstale":"2024-04-23T14:44:01.315648+0000","last_undegraded":"2024-04-23T14:44:01.315648+0000","last_fullsized":"2024-04-23T14:44:01.315648+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:56:55.947585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748648+0000","last_change":"2024-04-23T14:41:39.011021+0000","last_active":"2024-04-23T14:44:01.748648+0000","last_peered":"2024-04-23T14:44:01.748648+0000","last_clean":"2024-04-23T14:44:01.748648+0000","last_became_active":"2024-04-23T14:41:39.010637+0000","last_became_peered":"2024-04-23T14:41:39.010637+0000","last_unstale":"2024-04-23T14:44:01.748648+0000","last_undegraded":"2024-04-23T14:44:01.748648+0000","last_fullsized":"2024-04-23T14:44:01.748648+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:22:22.647170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.305882+0000","last_change":"2024-04-23T14:41:40.042797+0000","last_active":"2024-04-23T14:44:01.305882+0000","last_peered":"2024-04-23T14:44:01.305882+0000","last_clean":"2024-04-23T14:44:01.305882+0000","last_became_active":"2024-04-23T14:41:40.037797+0000","last_became_peered":"2024-04-23T14:41:40.037797+0000","last_unstale":"2024-04-23T14:44:01.305882+0000","last_undegraded":"2024-04-23T14:44:01.305882+0000","last_fullsized":"2024-04-23T14:44:01.305882+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:37:32.604276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315453+0000","last_change":"2024-04-23T14:41:39.010585+0000","last_active":"2024-04-23T14:44:01.315453+0000","last_peered":"2024-04-23T14:44:01.315453+0000","last_clean":"2024-04-23T14:44:01.315453+0000","last_became_active":"2024-04-23T14:41:39.010294+0000","last_became_peered":"2024-04-23T14:41:39.010294+0000","last_unstale":"2024-04-23T14:44:01.315453+0000","last_undegraded":"2024-04-23T14:44:01.315453+0000","last_fullsized":"2024-04-23T14:44:01.315453+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:38:03.146904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314580+0000","last_change":"2024-04-23T14:41:40.013150+0000","last_active":"2024-04-23T14:44:01.314580+0000","last_peered":"2024-04-23T14:44:01.314580+0000","last_clean":"2024-04-23T14:44:01.314580+0000","last_became_active":"2024-04-23T14:41:40.012310+0000","last_became_peered":"2024-04-23T14:41:40.012310+0000","last_unstale":"2024-04-23T14:44:01.314580+0000","last_undegraded":"2024-04-23T14:44:01.314580+0000","last_fullsized":"2024-04-23T14:44:01.314580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:45:25.777200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'33","reported_seq":343,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.706603+0000","last_change":"2024-04-23T14:41:39.004256+0000","last_active":"2024-04-23T14:44:01.706603+0000","last_peered":"2024-04-23T14:44:01.706603+0000","last_clean":"2024-04-23T14:44:01.706603+0000","last_became_active":"2024-04-23T14:41:39.003874+0000","last_became_peered":"2024-04-23T14:41:39.003874+0000","last_unstale":"2024-04-23T14:44:01.706603+0000","last_undegraded":"2024-04-23T14:44:01.706603+0000","last_fullsized":"2024-04-23T14:44:01.706603+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:38:30.727149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68610,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":288,"num_read_kb":205,"num_write":33,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314522+0000","last_change":"2024-04-23T14:41:40.012131+0000","last_active":"2024-04-23T14:44:01.314522+0000","last_peered":"2024-04-23T14:44:01.314522+0000","last_clean":"2024-04-23T14:44:01.314522+0000","last_became_active":"2024-04-23T14:41:40.011225+0000","last_became_peered":"2024-04-23T14:41:40.011225+0000","last_unstale":"2024-04-23T14:44:01.314522+0000","last_undegraded":"2024-04-23T14:44:01.314522+0000","last_fullsized":"2024-04-23T14:44:01.314522+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:36:12.085228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315400+0000","last_change":"2024-04-23T14:41:39.001935+0000","last_active":"2024-04-23T14:44:01.315400+0000","last_peered":"2024-04-23T14:44:01.315400+0000","last_clean":"2024-04-23T14:44:01.315400+0000","last_became_active":"2024-04-23T14:41:39.001583+0000","last_became_peered":"2024-04-23T14:41:39.001583+0000","last_unstale":"2024-04-23T14:44:01.315400+0000","last_undegraded":"2024-04-23T14:44:01.315400+0000","last_fullsized":"2024-04-23T14:44:01.315400+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:47:40.986280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315917+0000","last_change":"2024-04-23T14:41:40.036515+0000","last_active":"2024-04-23T14:44:01.315917+0000","last_peered":"2024-04-23T14:44:01.315917+0000","last_clean":"2024-04-23T14:44:01.315917+0000","last_became_active":"2024-04-23T14:41:40.036125+0000","last_became_peered":"2024-04-23T14:41:40.036125+0000","last_unstale":"2024-04-23T14:44:01.315917+0000","last_undegraded":"2024-04-23T14:44:01.315917+0000","last_fullsized":"2024-04-23T14:44:01.315917+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:55:28.465378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314793+0000","last_change":"2024-04-23T14:41:39.006127+0000","last_active":"2024-04-23T14:44:01.314793+0000","last_peered":"2024-04-23T14:44:01.314793+0000","last_clean":"2024-04-23T14:44:01.314793+0000","last_became_active":"2024-04-23T14:41:39.005761+0000","last_became_peered":"2024-04-23T14:41:39.005761+0000","last_unstale":"2024-04-23T14:44:01.314793+0000","last_undegraded":"2024-04-23T14:44:01.314793+0000","last_fullsized":"2024-04-23T14:44:01.314793+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:10:24.050016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748673+0000","last_change":"2024-04-23T14:42:43.678799+0000","last_active":"2024-04-23T14:44:01.748673+0000","last_peered":"2024-04-23T14:44:01.748673+0000","last_clean":"2024-04-23T14:44:01.748673+0000","last_became_active":"2024-04-23T14:42:43.678318+0000","last_became_peered":"2024-04-23T14:42:43.678318+0000","last_unstale":"2024-04-23T14:44:01.748673+0000","last_undegraded":"2024-04-23T14:44:01.748673+0000","last_fullsized":"2024-04-23T14:44:01.748673+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:26:21.827247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315145+0000","last_change":"2024-04-23T14:41:39.005873+0000","last_active":"2024-04-23T14:44:01.315145+0000","last_peered":"2024-04-23T14:44:01.315145+0000","last_clean":"2024-04-23T14:44:01.315145+0000","last_became_active":"2024-04-23T14:41:39.005481+0000","last_became_peered":"2024-04-23T14:41:39.005481+0000","last_unstale":"2024-04-23T14:44:01.315145+0000","last_undegraded":"2024-04-23T14:44:01.315145+0000","last_fullsized":"2024-04-23T14:44:01.315145+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:49:59.117451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748238+0000","last_change":"2024-04-23T14:41:40.009346+0000","last_active":"2024-04-23T14:44:01.748238+0000","last_peered":"2024-04-23T14:44:01.748238+0000","last_clean":"2024-04-23T14:44:01.748238+0000","last_became_active":"2024-04-23T14:41:40.008927+0000","last_became_peered":"2024-04-23T14:41:40.008927+0000","last_unstale":"2024-04-23T14:44:01.748238+0000","last_undegraded":"2024-04-23T14:44:01.748238+0000","last_fullsized":"2024-04-23T14:44:01.748238+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:33:36.513349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314285+0000","last_change":"2024-04-23T14:42:43.452178+0000","last_active":"2024-04-23T14:44:01.314285+0000","last_peered":"2024-04-23T14:44:01.314285+0000","last_clean":"2024-04-23T14:44:01.314285+0000","last_became_active":"2024-04-23T14:42:43.444786+0000","last_became_peered":"2024-04-23T14:42:43.444786+0000","last_unstale":"2024-04-23T14:44:01.314285+0000","last_undegraded":"2024-04-23T14:44:01.314285+0000","last_fullsized":"2024-04-23T14:44:01.314285+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:00:21.945663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,1],"acting":[8,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314546+0000","last_change":"2024-04-23T14:41:40.016845+0000","last_active":"2024-04-23T14:44:01.314546+0000","last_peered":"2024-04-23T14:44:01.314546+0000","last_clean":"2024-04-23T14:44:01.314546+0000","last_became_active":"2024-04-23T14:41:40.016543+0000","last_became_peered":"2024-04-23T14:41:40.016543+0000","last_unstale":"2024-04-23T14:44:01.314546+0000","last_undegraded":"2024-04-23T14:44:01.314546+0000","last_fullsized":"2024-04-23T14:44:01.314546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:52:41.054454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314792+0000","last_change":"2024-04-23T14:41:38.999968+0000","last_active":"2024-04-23T14:44:01.314792+0000","last_peered":"2024-04-23T14:44:01.314792+0000","last_clean":"2024-04-23T14:44:01.314792+0000","last_became_active":"2024-04-23T14:41:38.999658+0000","last_became_peered":"2024-04-23T14:41:38.999658+0000","last_unstale":"2024-04-23T14:44:01.314792+0000","last_undegraded":"2024-04-23T14:44:01.314792+0000","last_fullsized":"2024-04-23T14:44:01.314792+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:14:20.990331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.317006+0000","last_change":"2024-04-23T14:41:40.011753+0000","last_active":"2024-04-23T14:44:01.317006+0000","last_peered":"2024-04-23T14:44:01.317006+0000","last_clean":"2024-04-23T14:44:01.317006+0000","last_became_active":"2024-04-23T14:41:40.011264+0000","last_became_peered":"2024-04-23T14:41:40.011264+0000","last_unstale":"2024-04-23T14:44:01.317006+0000","last_undegraded":"2024-04-23T14:44:01.317006+0000","last_fullsized":"2024-04-23T14:44:01.317006+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:52:22.812046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.304344+0000","last_change":"2024-04-23T14:41:39.027384+0000","last_active":"2024-04-23T14:44:01.304344+0000","last_peered":"2024-04-23T14:44:01.304344+0000","last_clean":"2024-04-23T14:44:01.304344+0000","last_became_active":"2024-04-23T14:41:39.027018+0000","last_became_peered":"2024-04-23T14:41:39.027018+0000","last_unstale":"2024-04-23T14:44:01.304344+0000","last_undegraded":"2024-04-23T14:44:01.304344+0000","last_fullsized":"2024-04-23T14:44:01.304344+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:13:54.685704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315414+0000","last_change":"2024-04-23T14:41:40.041216+0000","last_active":"2024-04-23T14:44:01.315414+0000","last_peered":"2024-04-23T14:44:01.315414+0000","last_clean":"2024-04-23T14:44:01.315414+0000","last_became_active":"2024-04-23T14:41:40.040789+0000","last_became_peered":"2024-04-23T14:41:40.040789+0000","last_unstale":"2024-04-23T14:44:01.315414+0000","last_undegraded":"2024-04-23T14:44:01.315414+0000","last_fullsized":"2024-04-23T14:44:01.315414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:01:13.156704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748318+0000","last_change":"2024-04-23T14:41:39.024399+0000","last_active":"2024-04-23T14:44:01.748318+0000","last_peered":"2024-04-23T14:44:01.748318+0000","last_clean":"2024-04-23T14:44:01.748318+0000","last_became_active":"2024-04-23T14:41:39.024141+0000","last_became_peered":"2024-04-23T14:41:39.024141+0000","last_unstale":"2024-04-23T14:44:01.748318+0000","last_undegraded":"2024-04-23T14:44:01.748318+0000","last_fullsized":"2024-04-23T14:44:01.748318+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:34:27.516614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315148+0000","last_change":"2024-04-23T14:42:43.680818+0000","last_active":"2024-04-23T14:44:01.315148+0000","last_peered":"2024-04-23T14:44:01.315148+0000","last_clean":"2024-04-23T14:44:01.315148+0000","last_became_active":"2024-04-23T14:42:43.680463+0000","last_became_peered":"2024-04-23T14:42:43.680463+0000","last_unstale":"2024-04-23T14:44:01.315148+0000","last_undegraded":"2024-04-23T14:44:01.315148+0000","last_fullsized":"2024-04-23T14:44:01.315148+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:34:36.964359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314817+0000","last_change":"2024-04-23T14:41:40.014689+0000","last_active":"2024-04-23T14:44:01.314817+0000","last_peered":"2024-04-23T14:44:01.314817+0000","last_clean":"2024-04-23T14:44:01.314817+0000","last_became_active":"2024-04-23T14:41:40.014371+0000","last_became_peered":"2024-04-23T14:41:40.014371+0000","last_unstale":"2024-04-23T14:44:01.314817+0000","last_undegraded":"2024-04-23T14:44:01.314817+0000","last_fullsized":"2024-04-23T14:44:01.314817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:56:41.284144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315704+0000","last_change":"2024-04-23T14:41:39.031327+0000","last_active":"2024-04-23T14:44:01.315704+0000","last_peered":"2024-04-23T14:44:01.315704+0000","last_clean":"2024-04-23T14:44:01.315704+0000","last_became_active":"2024-04-23T14:41:39.030864+0000","last_became_peered":"2024-04-23T14:41:39.030864+0000","last_unstale":"2024-04-23T14:44:01.315704+0000","last_undegraded":"2024-04-23T14:44:01.315704+0000","last_fullsized":"2024-04-23T14:44:01.315704+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:58:24.048207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.319239+0000","last_change":"2024-04-23T14:41:40.035104+0000","last_active":"2024-04-23T14:44:01.319239+0000","last_peered":"2024-04-23T14:44:01.319239+0000","last_clean":"2024-04-23T14:44:01.319239+0000","last_became_active":"2024-04-23T14:41:40.034678+0000","last_became_peered":"2024-04-23T14:41:40.034678+0000","last_unstale":"2024-04-23T14:44:01.319239+0000","last_undegraded":"2024-04-23T14:44:01.319239+0000","last_fullsized":"2024-04-23T14:44:01.319239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:29:53.040424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314795+0000","last_change":"2024-04-23T14:41:39.016684+0000","last_active":"2024-04-23T14:44:01.314795+0000","last_peered":"2024-04-23T14:44:01.314795+0000","last_clean":"2024-04-23T14:44:01.314795+0000","last_became_active":"2024-04-23T14:41:39.016375+0000","last_became_peered":"2024-04-23T14:41:39.016375+0000","last_unstale":"2024-04-23T14:44:01.314795+0000","last_undegraded":"2024-04-23T14:44:01.314795+0000","last_fullsized":"2024-04-23T14:44:01.314795+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:00:09.449590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":357,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748262+0000","last_change":"2024-04-23T14:40:22.305413+0000","last_active":"2024-04-23T14:44:01.748262+0000","last_peered":"2024-04-23T14:44:01.748262+0000","last_clean":"2024-04-23T14:44:01.748262+0000","last_became_active":"2024-04-23T14:40:22.304977+0000","last_became_peered":"2024-04-23T14:40:22.304977+0000","last_unstale":"2024-04-23T14:44:01.748262+0000","last_undegraded":"2024-04-23T14:44:01.748262+0000","last_fullsized":"2024-04-23T14:44:01.748262+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:36:28.336753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:36:28.336753+0000","last_clean_scrub_stamp":"2024-04-23T14:36:28.336753+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:42:27.611770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315629+0000","last_change":"2024-04-23T14:41:40.023300+0000","last_active":"2024-04-23T14:44:01.315629+0000","last_peered":"2024-04-23T14:44:01.315629+0000","last_clean":"2024-04-23T14:44:01.315629+0000","last_became_active":"2024-04-23T14:41:40.022888+0000","last_became_peered":"2024-04-23T14:41:40.022888+0000","last_unstale":"2024-04-23T14:44:01.315629+0000","last_undegraded":"2024-04-23T14:44:01.315629+0000","last_fullsized":"2024-04-23T14:44:01.315629+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:31:48.705005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748412+0000","last_change":"2024-04-23T14:41:39.005482+0000","last_active":"2024-04-23T14:44:01.748412+0000","last_peered":"2024-04-23T14:44:01.748412+0000","last_clean":"2024-04-23T14:44:01.748412+0000","last_became_active":"2024-04-23T14:41:39.004552+0000","last_became_peered":"2024-04-23T14:41:39.004552+0000","last_unstale":"2024-04-23T14:44:01.748412+0000","last_undegraded":"2024-04-23T14:44:01.748412+0000","last_fullsized":"2024-04-23T14:44:01.748412+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:05:51.411210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314496+0000","last_change":"2024-04-23T14:41:39.015605+0000","last_active":"2024-04-23T14:44:01.314496+0000","last_peered":"2024-04-23T14:44:01.314496+0000","last_clean":"2024-04-23T14:44:01.314496+0000","last_became_active":"2024-04-23T14:41:39.015260+0000","last_became_peered":"2024-04-23T14:41:39.015260+0000","last_unstale":"2024-04-23T14:44:01.314496+0000","last_undegraded":"2024-04-23T14:44:01.314496+0000","last_fullsized":"2024-04-23T14:44:01.314496+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:23:21.251483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316641+0000","last_change":"2024-04-23T14:41:40.020619+0000","last_active":"2024-04-23T14:44:01.316641+0000","last_peered":"2024-04-23T14:44:01.316641+0000","last_clean":"2024-04-23T14:44:01.316641+0000","last_became_active":"2024-04-23T14:41:40.020191+0000","last_became_peered":"2024-04-23T14:41:40.020191+0000","last_unstale":"2024-04-23T14:44:01.316641+0000","last_undegraded":"2024-04-23T14:44:01.316641+0000","last_fullsized":"2024-04-23T14:44:01.316641+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:54:34.036464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'45","reported_seq":90,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:02.019133+0000","last_change":"2024-04-23T14:41:39.019507+0000","last_active":"2024-04-23T14:44:02.019133+0000","last_peered":"2024-04-23T14:44:02.019133+0000","last_clean":"2024-04-23T14:44:02.019133+0000","last_became_active":"2024-04-23T14:41:39.019116+0000","last_became_peered":"2024-04-23T14:41:39.019116+0000","last_unstale":"2024-04-23T14:44:02.019133+0000","last_undegraded":"2024-04-23T14:44:02.019133+0000","last_fullsized":"2024-04-23T14:44:02.019133+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:06:05.777908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":81,"num_write_kb":838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.318810+0000","last_change":"2024-04-23T14:41:40.032765+0000","last_active":"2024-04-23T14:44:01.318810+0000","last_peered":"2024-04-23T14:44:01.318810+0000","last_clean":"2024-04-23T14:44:01.318810+0000","last_became_active":"2024-04-23T14:41:40.027896+0000","last_became_peered":"2024-04-23T14:41:40.027896+0000","last_unstale":"2024-04-23T14:44:01.318810+0000","last_undegraded":"2024-04-23T14:44:01.318810+0000","last_fullsized":"2024-04-23T14:44:01.318810+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:36:16.225444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315304+0000","last_change":"2024-04-23T14:41:39.021681+0000","last_active":"2024-04-23T14:44:01.315304+0000","last_peered":"2024-04-23T14:44:01.315304+0000","last_clean":"2024-04-23T14:44:01.315304+0000","last_became_active":"2024-04-23T14:41:39.021423+0000","last_became_peered":"2024-04-23T14:41:39.021423+0000","last_unstale":"2024-04-23T14:44:01.315304+0000","last_undegraded":"2024-04-23T14:44:01.315304+0000","last_fullsized":"2024-04-23T14:44:01.315304+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:42:55.351296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.306731+0000","last_change":"2024-04-23T14:41:40.037250+0000","last_active":"2024-04-23T14:44:01.306731+0000","last_peered":"2024-04-23T14:44:01.306731+0000","last_clean":"2024-04-23T14:44:01.306731+0000","last_became_active":"2024-04-23T14:41:40.036645+0000","last_became_peered":"2024-04-23T14:41:40.036645+0000","last_unstale":"2024-04-23T14:44:01.306731+0000","last_undegraded":"2024-04-23T14:44:01.306731+0000","last_fullsized":"2024-04-23T14:44:01.306731+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:50:35.556841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314324+0000","last_change":"2024-04-23T14:41:39.007764+0000","last_active":"2024-04-23T14:44:01.314324+0000","last_peered":"2024-04-23T14:44:01.314324+0000","last_clean":"2024-04-23T14:44:01.314324+0000","last_became_active":"2024-04-23T14:41:39.007312+0000","last_became_peered":"2024-04-23T14:41:39.007312+0000","last_unstale":"2024-04-23T14:44:01.314324+0000","last_undegraded":"2024-04-23T14:44:01.314324+0000","last_fullsized":"2024-04-23T14:44:01.314324+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:38:56.021238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314309+0000","last_change":"2024-04-23T14:41:40.004343+0000","last_active":"2024-04-23T14:44:01.314309+0000","last_peered":"2024-04-23T14:44:01.314309+0000","last_clean":"2024-04-23T14:44:01.314309+0000","last_became_active":"2024-04-23T14:41:40.004015+0000","last_became_peered":"2024-04-23T14:41:40.004015+0000","last_unstale":"2024-04-23T14:44:01.314309+0000","last_undegraded":"2024-04-23T14:44:01.314309+0000","last_fullsized":"2024-04-23T14:44:01.314309+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:05:33.863692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.319064+0000","last_change":"2024-04-23T14:41:39.016271+0000","last_active":"2024-04-23T14:44:01.319064+0000","last_peered":"2024-04-23T14:44:01.319064+0000","last_clean":"2024-04-23T14:44:01.319064+0000","last_became_active":"2024-04-23T14:41:39.015851+0000","last_became_peered":"2024-04-23T14:41:39.015851+0000","last_unstale":"2024-04-23T14:44:01.319064+0000","last_undegraded":"2024-04-23T14:44:01.319064+0000","last_fullsized":"2024-04-23T14:44:01.319064+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:50:52.943534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316750+0000","last_change":"2024-04-23T14:41:40.029815+0000","last_active":"2024-04-23T14:44:01.316750+0000","last_peered":"2024-04-23T14:44:01.316750+0000","last_clean":"2024-04-23T14:44:01.316750+0000","last_became_active":"2024-04-23T14:41:40.029496+0000","last_became_peered":"2024-04-23T14:41:40.029496+0000","last_unstale":"2024-04-23T14:44:01.316750+0000","last_undegraded":"2024-04-23T14:44:01.316750+0000","last_fullsized":"2024-04-23T14:44:01.316750+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:36:09.228993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315232+0000","last_change":"2024-04-23T14:41:39.001946+0000","last_active":"2024-04-23T14:44:01.315232+0000","last_peered":"2024-04-23T14:44:01.315232+0000","last_clean":"2024-04-23T14:44:01.315232+0000","last_became_active":"2024-04-23T14:41:39.001619+0000","last_became_peered":"2024-04-23T14:41:39.001619+0000","last_unstale":"2024-04-23T14:44:01.315232+0000","last_undegraded":"2024-04-23T14:44:01.315232+0000","last_fullsized":"2024-04-23T14:44:01.315232+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:43:57.680195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.318758+0000","last_change":"2024-04-23T14:41:40.045728+0000","last_active":"2024-04-23T14:44:01.318758+0000","last_peered":"2024-04-23T14:44:01.318758+0000","last_clean":"2024-04-23T14:44:01.318758+0000","last_became_active":"2024-04-23T14:41:40.045293+0000","last_became_peered":"2024-04-23T14:41:40.045293+0000","last_unstale":"2024-04-23T14:44:01.318758+0000","last_undegraded":"2024-04-23T14:44:01.318758+0000","last_fullsized":"2024-04-23T14:44:01.318758+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:35:54.340644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314538+0000","last_change":"2024-04-23T14:42:43.823943+0000","last_active":"2024-04-23T14:44:01.314538+0000","last_peered":"2024-04-23T14:44:01.314538+0000","last_clean":"2024-04-23T14:44:01.314538+0000","last_became_active":"2024-04-23T14:42:43.821663+0000","last_became_peered":"2024-04-23T14:42:43.821663+0000","last_unstale":"2024-04-23T14:44:01.314538+0000","last_undegraded":"2024-04-23T14:44:01.314538+0000","last_fullsized":"2024-04-23T14:44:01.314538+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:18:54.895943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748380+0000","last_change":"2024-04-23T14:41:40.028897+0000","last_active":"2024-04-23T14:44:01.748380+0000","last_peered":"2024-04-23T14:44:01.748380+0000","last_clean":"2024-04-23T14:44:01.748380+0000","last_became_active":"2024-04-23T14:41:40.028557+0000","last_became_peered":"2024-04-23T14:41:40.028557+0000","last_unstale":"2024-04-23T14:44:01.748380+0000","last_undegraded":"2024-04-23T14:44:01.748380+0000","last_fullsized":"2024-04-23T14:44:01.748380+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:26:57.871013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315623+0000","last_change":"2024-04-23T14:41:39.012861+0000","last_active":"2024-04-23T14:44:01.315623+0000","last_peered":"2024-04-23T14:44:01.315623+0000","last_clean":"2024-04-23T14:44:01.315623+0000","last_became_active":"2024-04-23T14:41:39.012503+0000","last_became_peered":"2024-04-23T14:41:39.012503+0000","last_unstale":"2024-04-23T14:44:01.315623+0000","last_undegraded":"2024-04-23T14:44:01.315623+0000","last_fullsized":"2024-04-23T14:44:01.315623+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:41:57.347221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748833+0000","last_change":"2024-04-23T14:41:40.021580+0000","last_active":"2024-04-23T14:44:01.748833+0000","last_peered":"2024-04-23T14:44:01.748833+0000","last_clean":"2024-04-23T14:44:01.748833+0000","last_became_active":"2024-04-23T14:41:40.020706+0000","last_became_peered":"2024-04-23T14:41:40.020706+0000","last_unstale":"2024-04-23T14:44:01.748833+0000","last_undegraded":"2024-04-23T14:44:01.748833+0000","last_fullsized":"2024-04-23T14:44:01.748833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:59:42.576276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'39","reported_seq":69,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.599085+0000","last_change":"2024-04-23T14:41:39.006425+0000","last_active":"2024-04-23T14:44:01.599085+0000","last_peered":"2024-04-23T14:44:01.599085+0000","last_clean":"2024-04-23T14:44:01.599085+0000","last_became_active":"2024-04-23T14:41:39.006123+0000","last_became_peered":"2024-04-23T14:41:39.006123+0000","last_unstale":"2024-04-23T14:44:01.599085+0000","last_undegraded":"2024-04-23T14:44:01.599085+0000","last_fullsized":"2024-04-23T14:44:01.599085+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:08:37.261092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748644+0000","last_change":"2024-04-23T14:41:40.034063+0000","last_active":"2024-04-23T14:44:01.748644+0000","last_peered":"2024-04-23T14:44:01.748644+0000","last_clean":"2024-04-23T14:44:01.748644+0000","last_became_active":"2024-04-23T14:41:40.033669+0000","last_became_peered":"2024-04-23T14:41:40.033669+0000","last_unstale":"2024-04-23T14:44:01.748644+0000","last_undegraded":"2024-04-23T14:44:01.748644+0000","last_fullsized":"2024-04-23T14:44:01.748644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:23:50.608041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316265+0000","last_change":"2024-04-23T14:41:39.025975+0000","last_active":"2024-04-23T14:44:01.316265+0000","last_peered":"2024-04-23T14:44:01.316265+0000","last_clean":"2024-04-23T14:44:01.316265+0000","last_became_active":"2024-04-23T14:41:39.025629+0000","last_became_peered":"2024-04-23T14:41:39.025629+0000","last_unstale":"2024-04-23T14:44:01.316265+0000","last_undegraded":"2024-04-23T14:44:01.316265+0000","last_fullsized":"2024-04-23T14:44:01.316265+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:37.967525+0000","last_clean_scrub_stamp":"2024-04-23T14:41:37.967525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:00:03.753547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.748873+0000","last_change":"2024-04-23T14:41:40.013030+0000","last_active":"2024-04-23T14:44:01.748873+0000","last_peered":"2024-04-23T14:44:01.748873+0000","last_clean":"2024-04-23T14:44:01.748873+0000","last_became_active":"2024-04-23T14:41:40.012708+0000","last_became_peered":"2024-04-23T14:41:40.012708+0000","last_unstale":"2024-04-23T14:44:01.748873+0000","last_undegraded":"2024-04-23T14:44:01.748873+0000","last_fullsized":"2024-04-23T14:44:01.748873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:05:29.217273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316834+0000","last_change":"2024-04-23T14:41:40.010551+0000","last_active":"2024-04-23T14:44:01.316834+0000","last_peered":"2024-04-23T14:44:01.316834+0000","last_clean":"2024-04-23T14:44:01.316834+0000","last_became_active":"2024-04-23T14:41:40.010244+0000","last_became_peered":"2024-04-23T14:41:40.010244+0000","last_unstale":"2024-04-23T14:44:01.316834+0000","last_undegraded":"2024-04-23T14:44:01.316834+0000","last_fullsized":"2024-04-23T14:44:01.316834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:03:57.223815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315281+0000","last_change":"2024-04-23T14:41:40.009132+0000","last_active":"2024-04-23T14:44:01.315281+0000","last_peered":"2024-04-23T14:44:01.315281+0000","last_clean":"2024-04-23T14:44:01.315281+0000","last_became_active":"2024-04-23T14:41:40.008822+0000","last_became_peered":"2024-04-23T14:41:40.008822+0000","last_unstale":"2024-04-23T14:44:01.315281+0000","last_undegraded":"2024-04-23T14:44:01.315281+0000","last_fullsized":"2024-04-23T14:44:01.315281+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:10:28.513846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315222+0000","last_change":"2024-04-23T14:41:40.041193+0000","last_active":"2024-04-23T14:44:01.315222+0000","last_peered":"2024-04-23T14:44:01.315222+0000","last_clean":"2024-04-23T14:44:01.315222+0000","last_became_active":"2024-04-23T14:41:40.040851+0000","last_became_peered":"2024-04-23T14:41:40.040851+0000","last_unstale":"2024-04-23T14:44:01.315222+0000","last_undegraded":"2024-04-23T14:44:01.315222+0000","last_fullsized":"2024-04-23T14:44:01.315222+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:13:30.090015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.304852+0000","last_change":"2024-04-23T14:41:40.042703+0000","last_active":"2024-04-23T14:44:01.304852+0000","last_peered":"2024-04-23T14:44:01.304852+0000","last_clean":"2024-04-23T14:44:01.304852+0000","last_became_active":"2024-04-23T14:41:40.037666+0000","last_became_peered":"2024-04-23T14:41:40.037666+0000","last_unstale":"2024-04-23T14:44:01.304852+0000","last_undegraded":"2024-04-23T14:44:01.304852+0000","last_fullsized":"2024-04-23T14:44:01.304852+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:04:04.240689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316760+0000","last_change":"2024-04-23T14:41:40.033487+0000","last_active":"2024-04-23T14:44:01.316760+0000","last_peered":"2024-04-23T14:44:01.316760+0000","last_clean":"2024-04-23T14:44:01.316760+0000","last_became_active":"2024-04-23T14:41:40.031799+0000","last_became_peered":"2024-04-23T14:41:40.031799+0000","last_unstale":"2024-04-23T14:44:01.316760+0000","last_undegraded":"2024-04-23T14:44:01.316760+0000","last_fullsized":"2024-04-23T14:44:01.316760+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:36:03.930264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316943+0000","last_change":"2024-04-23T14:41:40.012833+0000","last_active":"2024-04-23T14:44:01.316943+0000","last_peered":"2024-04-23T14:44:01.316943+0000","last_clean":"2024-04-23T14:44:01.316943+0000","last_became_active":"2024-04-23T14:41:40.012251+0000","last_became_peered":"2024-04-23T14:41:40.012251+0000","last_unstale":"2024-04-23T14:44:01.316943+0000","last_undegraded":"2024-04-23T14:44:01.316943+0000","last_fullsized":"2024-04-23T14:44:01.316943+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:17:09.415998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.315679+0000","last_change":"2024-04-23T14:41:40.013336+0000","last_active":"2024-04-23T14:44:01.315679+0000","last_peered":"2024-04-23T14:44:01.315679+0000","last_clean":"2024-04-23T14:44:01.315679+0000","last_became_active":"2024-04-23T14:41:40.013043+0000","last_became_peered":"2024-04-23T14:41:40.013043+0000","last_unstale":"2024-04-23T14:44:01.315679+0000","last_undegraded":"2024-04-23T14:44:01.315679+0000","last_fullsized":"2024-04-23T14:44:01.315679+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:27:59.118986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.318677+0000","last_change":"2024-04-23T14:41:40.028236+0000","last_active":"2024-04-23T14:44:01.318677+0000","last_peered":"2024-04-23T14:44:01.318677+0000","last_clean":"2024-04-23T14:44:01.318677+0000","last_became_active":"2024-04-23T14:41:40.027504+0000","last_became_peered":"2024-04-23T14:41:40.027504+0000","last_unstale":"2024-04-23T14:44:01.318677+0000","last_undegraded":"2024-04-23T14:44:01.318677+0000","last_fullsized":"2024-04-23T14:44:01.318677+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:04:31.540482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314699+0000","last_change":"2024-04-23T14:41:40.029252+0000","last_active":"2024-04-23T14:44:01.314699+0000","last_peered":"2024-04-23T14:44:01.314699+0000","last_clean":"2024-04-23T14:44:01.314699+0000","last_became_active":"2024-04-23T14:41:40.028937+0000","last_became_peered":"2024-04-23T14:41:40.028937+0000","last_unstale":"2024-04-23T14:44:01.314699+0000","last_undegraded":"2024-04-23T14:44:01.314699+0000","last_fullsized":"2024-04-23T14:44:01.314699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:46:44.441319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.318839+0000","last_change":"2024-04-23T14:41:40.027705+0000","last_active":"2024-04-23T14:44:01.318839+0000","last_peered":"2024-04-23T14:44:01.318839+0000","last_clean":"2024-04-23T14:44:01.318839+0000","last_became_active":"2024-04-23T14:41:40.027260+0000","last_became_peered":"2024-04-23T14:41:40.027260+0000","last_unstale":"2024-04-23T14:44:01.318839+0000","last_undegraded":"2024-04-23T14:44:01.318839+0000","last_fullsized":"2024-04-23T14:44:01.318839+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:33:42.461219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.314679+0000","last_change":"2024-04-23T14:41:40.026490+0000","last_active":"2024-04-23T14:44:01.314679+0000","last_peered":"2024-04-23T14:44:01.314679+0000","last_clean":"2024-04-23T14:44:01.314679+0000","last_became_active":"2024-04-23T14:41:40.026117+0000","last_became_peered":"2024-04-23T14:41:40.026117+0000","last_unstale":"2024-04-23T14:44:01.314679+0000","last_undegraded":"2024-04-23T14:44:01.314679+0000","last_fullsized":"2024-04-23T14:44:01.314679+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:06:17.626663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316132+0000","last_change":"2024-04-23T14:41:40.038489+0000","last_active":"2024-04-23T14:44:01.316132+0000","last_peered":"2024-04-23T14:44:01.316132+0000","last_clean":"2024-04-23T14:44:01.316132+0000","last_became_active":"2024-04-23T14:41:40.038078+0000","last_became_peered":"2024-04-23T14:41:40.038078+0000","last_unstale":"2024-04-23T14:44:01.316132+0000","last_undegraded":"2024-04-23T14:44:01.316132+0000","last_fullsized":"2024-04-23T14:44:01.316132+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:58:59.440468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-23T14:44:01.316974+0000","last_change":"2024-04-23T14:41:40.009463+0000","last_active":"2024-04-23T14:44:01.316974+0000","last_peered":"2024-04-23T14:44:01.316974+0000","last_clean":"2024-04-23T14:44:01.316974+0000","last_became_active":"2024-04-23T14:41:40.009154+0000","last_became_peered":"2024-04-23T14:41:40.009154+0000","last_unstale":"2024-04-23T14:44:01.316974+0000","last_undegraded":"2024-04-23T14:44:01.316974+0000","last_fullsized":"2024-04-23T14:44:01.316974+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T14:41:38.970844+0000","last_clean_scrub_stamp":"2024-04-23T14:41:38.970844+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:13:47.641863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":594224,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":386,"num_read_kb":290,"num_write":200,"num_write_kb":1405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1880064,"data_stored":1782672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383721,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":962560,"data_stored":464666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645358,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28760,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716392,"statfs":{"total":95995035648,"available":95965585408,"internally_reserved":0,"allocated":1785856,"data_stored":1255555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808883,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36668,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708484,"statfs":{"total":95995035648,"available":95957487616,"internally_reserved":0,"allocated":1359872,"data_stored":858464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070522,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28352,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716800,"statfs":{"total":95995035648,"available":95966003200,"internally_reserved":0,"allocated":1363968,"data_stored":858486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332160,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32708,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712444,"statfs":{"total":95995035648,"available":95961542656,"internally_reserved":0,"allocated":1564672,"data_stored":1055733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495685,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":464783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574946,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1036288,"data_stored":527121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738471,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36284,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708868,"statfs":{"total":95995035648,"available":95957880832,"internally_reserved":0,"allocated":966656,"data_stored":464756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411421,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1040384,"data_stored":527155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182486,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28488,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716664,"statfs":{"total":95995035648,"available":95965863936,"internally_reserved":0,"allocated":1441792,"data_stored":924276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920849,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":966656,"data_stored":464688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757323,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28864,"kb_used_data":1668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716288,"statfs":{"total":95995035648,"available":95965478912,"internally_reserved":0,"allocated":1708032,"data_stored":1186933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":68610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":68644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":536576,"data_stored":524920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":131782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":393820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":397312,"data_stored":393798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":200404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T14:44:05.950 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:44:06.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:05 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:06.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:05 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:06.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:05 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:06.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:05 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:06.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:05 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:06.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:05 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:06.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:05 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:06.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:05 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:06.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:05 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:06.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:05 smithi105 ceph-mon[32555]: Adjusting osd_memory_target on smithi105 to 176.2M 2024-04-23T14:44:06.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:05 smithi105 ceph-mon[32555]: Unable to set osd_memory_target on smithi105 to 184783872: error parsing value: Value '184783872' is below minimum 939524096 2024-04-23T14:44:06.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:05 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:06.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:05 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:06.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:05 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:06.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:05 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:06.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:05 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:06.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:05 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:06.038 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:05 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:06.038 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:05 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:06.038 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:05 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:06.038 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:05 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:06.038 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:05 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:06.038 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:05 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:06.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:05 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:06.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:05 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:06.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:05 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:06.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:05 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:06.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:05 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:06.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:05 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:06.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:05 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:06.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:05 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:06.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:05 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:06.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:05 smithi038 ceph-mon[32348]: Adjusting osd_memory_target on smithi105 to 176.2M 2024-04-23T14:44:06.124 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:05 smithi038 ceph-mon[32348]: Unable to set osd_memory_target on smithi105 to 184783872: error parsing value: Value '184783872' is below minimum 939524096 2024-04-23T14:44:06.124 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:05 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:06.124 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:05 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:06.124 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:05 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:06.124 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:05 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:06.124 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:05 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:06.124 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:05 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:06.124 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:05 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:06.124 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:05 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:06.124 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:05 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:06.124 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:05 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:06.124 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:05 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:06.124 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:05 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:06.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.169+0000 7f371b200640 1 -- 172.21.15.28:0/2581936474 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f371c101ea0 msgr2=0x7f371c1022c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:06.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.169+0000 7f371b200640 1 --2- 172.21.15.28:0/2581936474 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f371c101ea0 0x7f371c1022c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f370c009900 tx=0x7f370c02a410 comp rx=0 tx=0).stop 2024-04-23T14:44:06.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.169+0000 7f371b200640 1 -- 172.21.15.28:0/2581936474 shutdown_connections 2024-04-23T14:44:06.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.169+0000 7f371b200640 1 --2- 172.21.15.28:0/2581936474 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f371c1039b0 0x7f371c10a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:06.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.169+0000 7f371b200640 1 --2- 172.21.15.28:0/2581936474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f371c102ff0 0x7f371c103470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:06.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.169+0000 7f371b200640 1 --2- 172.21.15.28:0/2581936474 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f371c101ea0 0x7f371c1022c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:06.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.169+0000 7f371b200640 1 -- 172.21.15.28:0/2581936474 >> 172.21.15.28:0/2581936474 conn(0x7f371c0fd410 msgr2=0x7f371c0ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:06.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.169+0000 7f371b200640 1 -- 172.21.15.28:0/2581936474 shutdown_connections 2024-04-23T14:44:06.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.169+0000 7f371b200640 1 -- 172.21.15.28:0/2581936474 wait complete. 2024-04-23T14:44:06.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.170+0000 7f371b200640 1 Processor -- start 2024-04-23T14:44:06.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.170+0000 7f371b200640 1 -- start start 2024-04-23T14:44:06.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.171+0000 7f371b200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f371c101ea0 0x7f371c19a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:06.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.171+0000 7f371b200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f371c102ff0 0x7f371c19a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:06.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.171+0000 7f371b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f371c1039b0 0x7f371c1a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:06.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.171+0000 7f371b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f371c10db90 con 0x7f371c1039b0 2024-04-23T14:44:06.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.171+0000 7f371b200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f371c10da10 con 0x7f371c102ff0 2024-04-23T14:44:06.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.171+0000 7f371b200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f371c10dd10 con 0x7f371c101ea0 2024-04-23T14:44:06.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.171+0000 7f371a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f371c1039b0 0x7f371c1a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:06.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.171+0000 7f3719400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f371c102ff0 0x7f371c19a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:06.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.171+0000 7f3719e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f371c101ea0 0x7f371c19a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:06.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.171+0000 7f3719400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f371c102ff0 0x7f371c19a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:37146/0 (socket says 172.21.15.28:37146) 2024-04-23T14:44:06.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.171+0000 7f3719400640 1 -- 172.21.15.28:0/2270023503 learned_addr learned my addr 172.21.15.28:0/2270023503 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:06.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.171+0000 7f3719400640 1 -- 172.21.15.28:0/2270023503 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f371c101ea0 msgr2=0x7f371c19a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:06.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.171+0000 7f3719400640 1 --2- 172.21.15.28:0/2270023503 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f371c101ea0 0x7f371c19a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:06.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.171+0000 7f3719400640 1 -- 172.21.15.28:0/2270023503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f371c1039b0 msgr2=0x7f371c1a1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:06.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.172+0000 7f3719400640 1 --2- 172.21.15.28:0/2270023503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f371c1039b0 0x7f371c1a1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:06.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.172+0000 7f3719400640 1 -- 172.21.15.28:0/2270023503 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f371c1a1fc0 con 0x7f371c102ff0 2024-04-23T14:44:06.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.172+0000 7f371a800640 1 --2- 172.21.15.28:0/2270023503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f371c1039b0 0x7f371c1a1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:44:06.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.172+0000 7f3719400640 1 --2- 172.21.15.28:0/2270023503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f371c102ff0 0x7f371c19a8e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f370800e9c0 tx=0x7f370800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:06.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.172+0000 7f3712a00640 1 -- 172.21.15.28:0/2270023503 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f370800cd80 con 0x7f371c102ff0 2024-04-23T14:44:06.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.172+0000 7f3712a00640 1 -- 172.21.15.28:0/2270023503 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37080035f0 con 0x7f371c102ff0 2024-04-23T14:44:06.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.172+0000 7f3712a00640 1 -- 172.21.15.28:0/2270023503 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3708010690 con 0x7f371c102ff0 2024-04-23T14:44:06.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.172+0000 7f371b200640 1 -- 172.21.15.28:0/2270023503 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f371c1a22b0 con 0x7f371c102ff0 2024-04-23T14:44:06.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.172+0000 7f371b200640 1 -- 172.21.15.28:0/2270023503 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f371c1a2800 con 0x7f371c102ff0 2024-04-23T14:44:06.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.173+0000 7f371b200640 1 -- 172.21.15.28:0/2270023503 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f371c1a2ac0 con 0x7f371c102ff0 2024-04-23T14:44:06.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.176+0000 7f3712a00640 1 -- 172.21.15.28:0/2270023503 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7f37080037a0 con 0x7f371c102ff0 2024-04-23T14:44:06.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.178+0000 7f3712a00640 1 --2- 172.21.15.28:0/2270023503 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f37000a3db0 0x7f37000a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:06.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.178+0000 7f3712a00640 1 -- 172.21.15.28:0/2270023503 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f37080d0eb0 con 0x7f371c102ff0 2024-04-23T14:44:06.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.178+0000 7f3719e00640 1 --2- 172.21.15.28:0/2270023503 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f37000a3db0 0x7f37000a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:06.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.179+0000 7f3719e00640 1 --2- 172.21.15.28:0/2270023503 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f37000a3db0 0x7f37000a6270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f370c031440 tx=0x7f370c0315b0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:06.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.179+0000 7f3712a00640 1 -- 172.21.15.28:0/2270023503 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f370809d390 con 0x7f371c102ff0 2024-04-23T14:44:06.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.379+0000 7f371b200640 1 -- 172.21.15.28:0/2270023503 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f371c1a2c90 con 0x7f371c102ff0 2024-04-23T14:44:06.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.381+0000 7f3712a00640 1 -- 172.21.15.28:0/2270023503 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+23026 (secure 0 0 0) 0x7f370809cc90 con 0x7f371c102ff0 2024-04-23T14:44:06.382 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:44:06.382 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":88,"fsid":"1ee6e996-017e-11ef-bc93-c7b262605968","created":"2024-04-23T14:32:34.581082+0000","modified":"2024-04-23T14:44:01.286339+0000","last_up_change":"2024-04-23T14:40:43.375846+0000","last_in_change":"2024-04-23T14:40:23.576939+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:36:27.698112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:41:37.030341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:41:38.444742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e0b427bc-be65-4543-99cc-2f53ad98ddad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6803","nonce":2993445336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6805","nonce":2993445336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6809","nonce":2993445336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6807","nonce":2993445336}]},"public_addr":"172.21.15.28:6803/2993445336","cluster_addr":"172.21.15.28:6805/2993445336","heartbeat_back_addr":"172.21.15.28:6809/2993445336","heartbeat_front_addr":"172.21.15.28:6807/2993445336","state":["exists","up"]},{"osd":1,"uuid":"ef527ec9-1bf3-4bb0-8df3-f5a015af557c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6801","nonce":4024093383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6803","nonce":4024093383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6807","nonce":4024093383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6805","nonce":4024093383}]},"public_addr":"172.21.15.38:6801/4024093383","cluster_addr":"172.21.15.38:6803/4024093383","heartbeat_back_addr":"172.21.15.38:6807/4024093383","heartbeat_front_addr":"172.21.15.38:6805/4024093383","state":["exists","up"]},{"osd":2,"uuid":"b33b23d9-adce-4c11-b304-2e117bdf6bfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6801","nonce":570739029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6803","nonce":570739029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6807","nonce":570739029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6805","nonce":570739029}]},"public_addr":"172.21.15.105:6801/570739029","cluster_addr":"172.21.15.105:6803/570739029","heartbeat_back_addr":"172.21.15.105:6807/570739029","heartbeat_front_addr":"172.21.15.105:6805/570739029","state":["exists","up"]},{"osd":3,"uuid":"28bd5608-b426-434e-9db1-bdaa9828bdb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6811","nonce":2480400247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6813","nonce":2480400247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6817","nonce":2480400247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6815","nonce":2480400247}]},"public_addr":"172.21.15.28:6811/2480400247","cluster_addr":"172.21.15.28:6813/2480400247","heartbeat_back_addr":"172.21.15.28:6817/2480400247","heartbeat_front_addr":"172.21.15.28:6815/2480400247","state":["exists","up"]},{"osd":4,"uuid":"e5d1c89b-025f-4d57-8807-59d75c787b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6809","nonce":3772189246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6811","nonce":3772189246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6815","nonce":3772189246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6813","nonce":3772189246}]},"public_addr":"172.21.15.38:6809/3772189246","cluster_addr":"172.21.15.38:6811/3772189246","heartbeat_back_addr":"172.21.15.38:6815/3772189246","heartbeat_front_addr":"172.21.15.38:6813/3772189246","state":["exists","up"]},{"osd":5,"uuid":"9cf9cf4c-aafd-4ed3-bfb1-0527710a0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6809","nonce":928755894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6811","nonce":928755894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6815","nonce":928755894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6813","nonce":928755894}]},"public_addr":"172.21.15.105:6809/928755894","cluster_addr":"172.21.15.105:6811/928755894","heartbeat_back_addr":"172.21.15.105:6815/928755894","heartbeat_front_addr":"172.21.15.105:6813/928755894","state":["exists","up"]},{"osd":6,"uuid":"304db525-d24d-4687-bfa0-f92bdf66d1e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6819","nonce":897979441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6821","nonce":897979441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6825","nonce":897979441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6823","nonce":897979441}]},"public_addr":"172.21.15.28:6819/897979441","cluster_addr":"172.21.15.28:6821/897979441","heartbeat_back_addr":"172.21.15.28:6825/897979441","heartbeat_front_addr":"172.21.15.28:6823/897979441","state":["exists","up"]},{"osd":7,"uuid":"08e01d59-03f3-47b1-b45f-35ec5a50992c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6817","nonce":2400251786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6819","nonce":2400251786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6823","nonce":2400251786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6821","nonce":2400251786}]},"public_addr":"172.21.15.38:6817/2400251786","cluster_addr":"172.21.15.38:6819/2400251786","heartbeat_back_addr":"172.21.15.38:6823/2400251786","heartbeat_front_addr":"172.21.15.38:6821/2400251786","state":["exists","up"]},{"osd":8,"uuid":"64ee1e95-0756-48b1-9c00-472d0c3cbd29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6817","nonce":1682312269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6819","nonce":1682312269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6823","nonce":1682312269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6821","nonce":1682312269}]},"public_addr":"172.21.15.105:6817/1682312269","cluster_addr":"172.21.15.105:6819/1682312269","heartbeat_back_addr":"172.21.15.105:6823/1682312269","heartbeat_front_addr":"172.21.15.105:6821/1682312269","state":["exists","up"]},{"osd":9,"uuid":"18f60893-3ffd-48c2-9db5-a67d35611d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6827","nonce":1292942516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6829","nonce":1292942516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6833","nonce":1292942516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6831","nonce":1292942516}]},"public_addr":"172.21.15.28:6827/1292942516","cluster_addr":"172.21.15.28:6829/1292942516","heartbeat_back_addr":"172.21.15.28:6833/1292942516","heartbeat_front_addr":"172.21.15.28:6831/1292942516","state":["exists","up"]},{"osd":10,"uuid":"76c06725-7040-4118-8894-fe808a87f999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6825","nonce":2444616468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6827","nonce":2444616468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6831","nonce":2444616468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6829","nonce":2444616468}]},"public_addr":"172.21.15.38:6825/2444616468","cluster_addr":"172.21.15.38:6827/2444616468","heartbeat_back_addr":"172.21.15.38:6831/2444616468","heartbeat_front_addr":"172.21.15.38:6829/2444616468","state":["exists","up"]},{"osd":11,"uuid":"e8fc9454-5540-4e2e-91bf-3ba251dbd1a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6825","nonce":3400299562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6827","nonce":3400299562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6831","nonce":3400299562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6829","nonce":3400299562}]},"public_addr":"172.21.15.105:6825/3400299562","cluster_addr":"172.21.15.105:6827/3400299562","heartbeat_back_addr":"172.21.15.105:6831/3400299562","heartbeat_front_addr":"172.21.15.105:6829/3400299562","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:35:33.615395+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:00.545293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:24.940211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:57.684887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:24.519061+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:49.010804+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:22.638478+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:49.559104+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:14.011771+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:47.859950+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:16.298445+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:41.619069+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.2d","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/38494154":"2024-04-24T14:44:01.286264+0000","172.21.15.28:0/905362079":"2024-04-24T14:44:01.286264+0000","172.21.15.28:6801/1997136173":"2024-04-24T14:44:01.286264+0000","172.21.15.28:6800/1997136173":"2024-04-24T14:44:01.286264+0000","172.21.15.28:0/590495740":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/1163172512":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/4290328460":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/4145020172":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/1997097543":"2024-04-24T14:44:01.286264+0000","172.21.15.28:0/3847772687":"2024-04-24T14:44:01.286264+0000","172.21.15.28:6800/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/720626958":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/948450262":"2024-04-24T14:33:07.576363+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:44:06.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.384+0000 7f371b200640 1 -- 172.21.15.28:0/2270023503 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f37000a3db0 msgr2=0x7f37000a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:06.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.384+0000 7f371b200640 1 --2- 172.21.15.28:0/2270023503 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f37000a3db0 0x7f37000a6270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f370c031440 tx=0x7f370c0315b0 comp rx=0 tx=0).stop 2024-04-23T14:44:06.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.384+0000 7f371b200640 1 -- 172.21.15.28:0/2270023503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f371c102ff0 msgr2=0x7f371c19a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:06.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.384+0000 7f371b200640 1 --2- 172.21.15.28:0/2270023503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f371c102ff0 0x7f371c19a8e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f370800e9c0 tx=0x7f370800ee80 comp rx=0 tx=0).stop 2024-04-23T14:44:06.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.384+0000 7f371b200640 1 -- 172.21.15.28:0/2270023503 shutdown_connections 2024-04-23T14:44:06.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.384+0000 7f371b200640 1 --2- 172.21.15.28:0/2270023503 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f37000a3db0 0x7f37000a6270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:06.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.384+0000 7f371b200640 1 --2- 172.21.15.28:0/2270023503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f371c102ff0 0x7f371c19a8e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:06.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.384+0000 7f371b200640 1 --2- 172.21.15.28:0/2270023503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f371c1039b0 0x7f371c1a1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:06.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.384+0000 7f371b200640 1 --2- 172.21.15.28:0/2270023503 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f371c101ea0 0x7f371c19a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:06.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.384+0000 7f371b200640 1 -- 172.21.15.28:0/2270023503 >> 172.21.15.28:0/2270023503 conn(0x7f371c0fd410 msgr2=0x7f371c074230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:06.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.384+0000 7f371b200640 1 -- 172.21.15.28:0/2270023503 shutdown_connections 2024-04-23T14:44:06.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.384+0000 7f371b200640 1 -- 172.21.15.28:0/2270023503 wait complete. 2024-04-23T14:44:06.403 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T14:36:27.698112+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T14:41:37.030341+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T14:41:38.444742+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T14:44:06.403 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T14:44:06.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.593+0000 7f87ef200640 1 -- 172.21.15.28:0/1035967269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e8108540 msgr2=0x7f87e8108960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:06.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.593+0000 7f87ef200640 1 --2- 172.21.15.28:0/1035967269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e8108540 0x7f87e8108960 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f87dc009900 tx=0x7f87dc02a410 comp rx=0 tx=0).stop 2024-04-23T14:44:06.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.593+0000 7f87ef200640 1 -- 172.21.15.28:0/1035967269 shutdown_connections 2024-04-23T14:44:06.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.593+0000 7f87ef200640 1 --2- 172.21.15.28:0/1035967269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87e810a0f0 0x7f87e8110980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:06.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.593+0000 7f87ef200640 1 --2- 172.21.15.28:0/1035967269 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f87e8109730 0x7f87e8109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:06.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.593+0000 7f87ef200640 1 --2- 172.21.15.28:0/1035967269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e8108540 0x7f87e8108960 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:06.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.593+0000 7f87ef200640 1 -- 172.21.15.28:0/1035967269 >> 172.21.15.28:0/1035967269 conn(0x7f87e80774d0 msgr2=0x7f87e8079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:06.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.593+0000 7f87ef200640 1 -- 172.21.15.28:0/1035967269 shutdown_connections 2024-04-23T14:44:06.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.593+0000 7f87ef200640 1 -- 172.21.15.28:0/1035967269 wait complete. 2024-04-23T14:44:06.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.594+0000 7f87ef200640 1 Processor -- start 2024-04-23T14:44:06.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.594+0000 7f87ef200640 1 -- start start 2024-04-23T14:44:06.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.594+0000 7f87ef200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e8108540 0x7f87e819e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:06.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.594+0000 7f87ef200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87e8109730 0x7f87e819ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:06.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.594+0000 7f87ef200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f87e810a0f0 0x7f87e81a5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:06.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.594+0000 7f87ef200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f87e81141f0 con 0x7f87e8108540 2024-04-23T14:44:06.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.594+0000 7f87ef200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f87e8114070 con 0x7f87e8109730 2024-04-23T14:44:06.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.594+0000 7f87ef200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f87e8114370 con 0x7f87e810a0f0 2024-04-23T14:44:06.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.595+0000 7f87ede00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e8108540 0x7f87e819e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:06.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.595+0000 7f87ede00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e8108540 0x7f87e819e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36384/0 (socket says 172.21.15.28:36384) 2024-04-23T14:44:06.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.595+0000 7f87ede00640 1 -- 172.21.15.28:0/3360904293 learned_addr learned my addr 172.21.15.28:0/3360904293 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:06.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.595+0000 7f87ee800640 1 --2- 172.21.15.28:0/3360904293 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f87e810a0f0 0x7f87e81a5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:06.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.595+0000 7f87ed400640 1 --2- 172.21.15.28:0/3360904293 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87e8109730 0x7f87e819ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:06.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.595+0000 7f87ede00640 1 -- 172.21.15.28:0/3360904293 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f87e810a0f0 msgr2=0x7f87e81a5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:06.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.595+0000 7f87ede00640 1 --2- 172.21.15.28:0/3360904293 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f87e810a0f0 0x7f87e81a5dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:06.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.595+0000 7f87ede00640 1 -- 172.21.15.28:0/3360904293 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87e8109730 msgr2=0x7f87e819ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:06.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.595+0000 7f87ede00640 1 --2- 172.21.15.28:0/3360904293 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87e8109730 0x7f87e819ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:06.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.595+0000 7f87ede00640 1 -- 172.21.15.28:0/3360904293 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87e81a6430 con 0x7f87e8108540 2024-04-23T14:44:06.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.596+0000 7f87ede00640 1 --2- 172.21.15.28:0/3360904293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e8108540 0x7f87e819e810 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f87dc031440 tx=0x7f87dc032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:06.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.596+0000 7f87e2a00640 1 -- 172.21.15.28:0/3360904293 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87dc03f030 con 0x7f87e8108540 2024-04-23T14:44:06.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.596+0000 7f87e2a00640 1 -- 172.21.15.28:0/3360904293 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87dc003ce0 con 0x7f87e8108540 2024-04-23T14:44:06.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.596+0000 7f87ef200640 1 -- 172.21.15.28:0/3360904293 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87e81a66c0 con 0x7f87e8108540 2024-04-23T14:44:06.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.596+0000 7f87e2a00640 1 -- 172.21.15.28:0/3360904293 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87dc035070 con 0x7f87e8108540 2024-04-23T14:44:06.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.597+0000 7f87ef200640 1 -- 172.21.15.28:0/3360904293 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87e81a6c00 con 0x7f87e8108540 2024-04-23T14:44:06.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.598+0000 7f87ef200640 1 -- 172.21.15.28:0/3360904293 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87e810f860 con 0x7f87e8108540 2024-04-23T14:44:06.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.600+0000 7f87e2a00640 1 -- 172.21.15.28:0/3360904293 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7f87dc042030 con 0x7f87e8108540 2024-04-23T14:44:06.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.601+0000 7f87e2a00640 1 --2- 172.21.15.28:0/3360904293 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f87d00a3e30 0x7f87d00a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:06.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.601+0000 7f87e2a00640 1 -- 172.21.15.28:0/3360904293 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f87dc0f0da0 con 0x7f87e8108540 2024-04-23T14:44:06.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.602+0000 7f87ed400640 1 --2- 172.21.15.28:0/3360904293 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f87d00a3e30 0x7f87d00a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:06.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.602+0000 7f87ed400640 1 --2- 172.21.15.28:0/3360904293 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f87d00a3e30 0x7f87d00a62f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f87e819fd20 tx=0x7f87d8005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:06.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.605+0000 7f87e2a00640 1 -- 172.21.15.28:0/3360904293 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f87dc0bcc30 con 0x7f87e8108540 2024-04-23T14:44:06.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.814+0000 7f87ef200640 1 -- 172.21.15.28:0/3360904293 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f87e802d620 con 0x7f87e8108540 2024-04-23T14:44:06.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.815+0000 7f87e2a00640 1 -- 172.21.15.28:0/3360904293 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f87dc0c0f30 con 0x7f87e8108540 2024-04-23T14:44:06.816 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-23T14:44:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.817+0000 7f87ef200640 1 -- 172.21.15.28:0/3360904293 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f87d00a3e30 msgr2=0x7f87d00a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.817+0000 7f87ef200640 1 --2- 172.21.15.28:0/3360904293 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f87d00a3e30 0x7f87d00a62f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f87e819fd20 tx=0x7f87d8005f90 comp rx=0 tx=0).stop 2024-04-23T14:44:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.817+0000 7f87ef200640 1 -- 172.21.15.28:0/3360904293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e8108540 msgr2=0x7f87e819e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.817+0000 7f87ef200640 1 --2- 172.21.15.28:0/3360904293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e8108540 0x7f87e819e810 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f87dc031440 tx=0x7f87dc032ce0 comp rx=0 tx=0).stop 2024-04-23T14:44:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.817+0000 7f87ef200640 1 -- 172.21.15.28:0/3360904293 shutdown_connections 2024-04-23T14:44:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.817+0000 7f87ef200640 1 --2- 172.21.15.28:0/3360904293 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f87d00a3e30 0x7f87d00a62f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.817+0000 7f87ef200640 1 --2- 172.21.15.28:0/3360904293 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87e8109730 0x7f87e819ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.817+0000 7f87ef200640 1 --2- 172.21.15.28:0/3360904293 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f87e810a0f0 0x7f87e81a5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.817+0000 7f87ef200640 1 --2- 172.21.15.28:0/3360904293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e8108540 0x7f87e819e810 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.817+0000 7f87ef200640 1 -- 172.21.15.28:0/3360904293 >> 172.21.15.28:0/3360904293 conn(0x7f87e80774d0 msgr2=0x7f87e810ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.817+0000 7f87ef200640 1 -- 172.21.15.28:0/3360904293 shutdown_connections 2024-04-23T14:44:06.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:06.817+0000 7f87ef200640 1 -- 172.21.15.28:0/3360904293 wait complete. 2024-04-23T14:44:06.837 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T14:44:06.980 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:06 smithi028 ceph-mon[27810]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T14:44:06.980 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:06 smithi028 ceph-mon[27810]: Adjusting osd_memory_target on smithi038 to 177.2M 2024-04-23T14:44:06.980 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:06 smithi028 ceph-mon[27810]: Unable to set osd_memory_target on smithi038 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-23T14:44:06.980 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:06 smithi028 ceph-mon[27810]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:44:06.980 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:06 smithi028 ceph-mon[27810]: from='client.25270 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T14:44:06.980 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:06 smithi028 ceph-mon[27810]: mgrmap e25: x(active, since 4s), standbys: z, y 2024-04-23T14:44:06.980 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:06 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2270023503' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:44:07.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:06 smithi105 ceph-mon[32555]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T14:44:07.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:06 smithi105 ceph-mon[32555]: Adjusting osd_memory_target on smithi038 to 177.2M 2024-04-23T14:44:07.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:06 smithi105 ceph-mon[32555]: Unable to set osd_memory_target on smithi038 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-23T14:44:07.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:06 smithi105 ceph-mon[32555]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:44:07.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:06 smithi105 ceph-mon[32555]: from='client.25270 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T14:44:07.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:06 smithi105 ceph-mon[32555]: mgrmap e25: x(active, since 4s), standbys: z, y 2024-04-23T14:44:07.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:06 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2270023503' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:44:07.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.038+0000 7f2213000640 1 -- 172.21.15.28:0/2822900958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f220c0769c0 msgr2=0x7f220c076de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:07.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.038+0000 7f2213000640 1 --2- 172.21.15.28:0/2822900958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f220c0769c0 0x7f220c076de0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f21fc009900 tx=0x7f21fc02a410 comp rx=0 tx=0).stop 2024-04-23T14:44:07.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.038+0000 7f2213000640 1 -- 172.21.15.28:0/2822900958 shutdown_connections 2024-04-23T14:44:07.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.038+0000 7f2213000640 1 --2- 172.21.15.28:0/2822900958 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f220c077c00 0x7f220c078060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:07.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.038+0000 7f2213000640 1 --2- 172.21.15.28:0/2822900958 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f220c0785a0 0x7f220c078e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:07.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.038+0000 7f2213000640 1 --2- 172.21.15.28:0/2822900958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f220c0769c0 0x7f220c076de0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:07.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.038+0000 7f2213000640 1 -- 172.21.15.28:0/2822900958 >> 172.21.15.28:0/2822900958 conn(0x7f220c0ff7b0 msgr2=0x7f220c101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:07.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.038+0000 7f2213000640 1 -- 172.21.15.28:0/2822900958 shutdown_connections 2024-04-23T14:44:07.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.038+0000 7f2213000640 1 -- 172.21.15.28:0/2822900958 wait complete. 2024-04-23T14:44:07.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.039+0000 7f2213000640 1 Processor -- start 2024-04-23T14:44:07.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.039+0000 7f2213000640 1 -- start start 2024-04-23T14:44:07.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.039+0000 7f2213000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f220c0769c0 0x7f220c19e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:07.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.039+0000 7f2213000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f220c077c00 0x7f220c19ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:07.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.039+0000 7f2213000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f220c0785a0 0x7f220c1a5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:07.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.039+0000 7f2213000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f220c1142f0 con 0x7f220c0769c0 2024-04-23T14:44:07.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.039+0000 7f2213000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f220c114170 con 0x7f220c0785a0 2024-04-23T14:44:07.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.039+0000 7f2213000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f220c114470 con 0x7f220c077c00 2024-04-23T14:44:07.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.040+0000 7f220be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f220c0769c0 0x7f220c19e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:07.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.040+0000 7f220be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f220c0769c0 0x7f220c19e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36398/0 (socket says 172.21.15.28:36398) 2024-04-23T14:44:07.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.040+0000 7f220be00640 1 -- 172.21.15.28:0/3618987788 learned_addr learned my addr 172.21.15.28:0/3618987788 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:07.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.040+0000 7f2203e00640 1 --2- 172.21.15.28:0/3618987788 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f220c077c00 0x7f220c19ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:07.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.040+0000 7f2211000640 1 --2- 172.21.15.28:0/3618987788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f220c0785a0 0x7f220c1a5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:07.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.040+0000 7f220be00640 1 -- 172.21.15.28:0/3618987788 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f220c077c00 msgr2=0x7f220c19ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:07.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.040+0000 7f220be00640 1 --2- 172.21.15.28:0/3618987788 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f220c077c00 0x7f220c19ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:07.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.040+0000 7f220be00640 1 -- 172.21.15.28:0/3618987788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f220c0785a0 msgr2=0x7f220c1a5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:07.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.040+0000 7f220be00640 1 --2- 172.21.15.28:0/3618987788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f220c0785a0 0x7f220c1a5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:07.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.040+0000 7f220be00640 1 -- 172.21.15.28:0/3618987788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f220c1a6560 con 0x7f220c0769c0 2024-04-23T14:44:07.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.040+0000 7f2203e00640 1 --2- 172.21.15.28:0/3618987788 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f220c077c00 0x7f220c19ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:44:07.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.040+0000 7f220be00640 1 --2- 172.21.15.28:0/3618987788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f220c0769c0 0x7f220c19e940 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f21fc009230 tx=0x7f21fc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:07.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.041+0000 7f2209600640 1 -- 172.21.15.28:0/3618987788 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21fc03f030 con 0x7f220c0769c0 2024-04-23T14:44:07.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.041+0000 7f2209600640 1 -- 172.21.15.28:0/3618987788 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21fc003ce0 con 0x7f220c0769c0 2024-04-23T14:44:07.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.041+0000 7f2213000640 1 -- 172.21.15.28:0/3618987788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f220c1a67f0 con 0x7f220c0769c0 2024-04-23T14:44:07.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.041+0000 7f2209600640 1 -- 172.21.15.28:0/3618987788 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21fc035070 con 0x7f220c0769c0 2024-04-23T14:44:07.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.041+0000 7f2213000640 1 -- 172.21.15.28:0/3618987788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f220c1a6d30 con 0x7f220c0769c0 2024-04-23T14:44:07.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.042+0000 7f2213000640 1 -- 172.21.15.28:0/3618987788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f220c110d10 con 0x7f220c0769c0 2024-04-23T14:44:07.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.046+0000 7f2209600640 1 -- 172.21.15.28:0/3618987788 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7f21fc042030 con 0x7f220c0769c0 2024-04-23T14:44:07.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.047+0000 7f2209600640 1 --2- 172.21.15.28:0/3618987788 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f21f00a3e30 0x7f21f00a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:07.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.047+0000 7f2209600640 1 -- 172.21.15.28:0/3618987788 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f21fc0f0de0 con 0x7f220c0769c0 2024-04-23T14:44:07.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.047+0000 7f2203e00640 1 --2- 172.21.15.28:0/3618987788 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f21f00a3e30 0x7f21f00a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:07.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.048+0000 7f2203e00640 1 --2- 172.21.15.28:0/3618987788 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f21f00a3e30 0x7f21f00a62f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f21f40095f0 tx=0x7f21f4005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:07.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.052+0000 7f2209600640 1 -- 172.21.15.28:0/3618987788 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f21fc0bcc70 con 0x7f220c0769c0 2024-04-23T14:44:07.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:06 smithi038 ceph-mon[32348]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T14:44:07.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:06 smithi038 ceph-mon[32348]: Adjusting osd_memory_target on smithi038 to 177.2M 2024-04-23T14:44:07.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:06 smithi038 ceph-mon[32348]: Unable to set osd_memory_target on smithi038 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-23T14:44:07.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:06 smithi038 ceph-mon[32348]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:44:07.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:06 smithi038 ceph-mon[32348]: from='client.25270 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T14:44:07.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:06 smithi038 ceph-mon[32348]: mgrmap e25: x(active, since 4s), standbys: z, y 2024-04-23T14:44:07.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:06 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2270023503' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:44:07.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.259+0000 7f2213000640 1 -- 172.21.15.28:0/3618987788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f220c062c00 con 0x7f220c0769c0 2024-04-23T14:44:07.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.260+0000 7f2209600640 1 -- 172.21.15.28:0/3618987788 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f21fc0c0f70 con 0x7f220c0769c0 2024-04-23T14:44:07.261 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-23T14:44:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.262+0000 7f2213000640 1 -- 172.21.15.28:0/3618987788 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f21f00a3e30 msgr2=0x7f21f00a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.262+0000 7f2213000640 1 --2- 172.21.15.28:0/3618987788 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f21f00a3e30 0x7f21f00a62f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f21f40095f0 tx=0x7f21f4005f90 comp rx=0 tx=0).stop 2024-04-23T14:44:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.262+0000 7f2213000640 1 -- 172.21.15.28:0/3618987788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f220c0769c0 msgr2=0x7f220c19e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.262+0000 7f2213000640 1 --2- 172.21.15.28:0/3618987788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f220c0769c0 0x7f220c19e940 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f21fc009230 tx=0x7f21fc0042f0 comp rx=0 tx=0).stop 2024-04-23T14:44:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.263+0000 7f2213000640 1 -- 172.21.15.28:0/3618987788 shutdown_connections 2024-04-23T14:44:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.263+0000 7f2213000640 1 --2- 172.21.15.28:0/3618987788 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f21f00a3e30 0x7f21f00a62f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:07.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.263+0000 7f2213000640 1 --2- 172.21.15.28:0/3618987788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f220c0785a0 0x7f220c1a5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:07.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.263+0000 7f2213000640 1 --2- 172.21.15.28:0/3618987788 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f220c077c00 0x7f220c19ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:07.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.263+0000 7f2213000640 1 --2- 172.21.15.28:0/3618987788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f220c0769c0 0x7f220c19e940 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:07.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.263+0000 7f2213000640 1 -- 172.21.15.28:0/3618987788 >> 172.21.15.28:0/3618987788 conn(0x7f220c0ff7b0 msgr2=0x7f220c100f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:07.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.263+0000 7f2213000640 1 -- 172.21.15.28:0/3618987788 shutdown_connections 2024-04-23T14:44:07.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.263+0000 7f2213000640 1 -- 172.21.15.28:0/3618987788 wait complete. 2024-04-23T14:44:07.282 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T14:44:07.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.466+0000 7f5f13a00640 1 -- 172.21.15.28:0/1250508345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0c108660 msgr2=0x7f5f0c108a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:07.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.466+0000 7f5f13a00640 1 --2- 172.21.15.28:0/1250508345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0c108660 0x7f5f0c108a80 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5f00009860 tx=0x7f5f0002a340 comp rx=0 tx=0).stop 2024-04-23T14:44:07.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.466+0000 7f5f13a00640 1 -- 172.21.15.28:0/1250508345 shutdown_connections 2024-04-23T14:44:07.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.466+0000 7f5f13a00640 1 --2- 172.21.15.28:0/1250508345 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f0c10a210 0x7f5f0c110aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:07.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.466+0000 7f5f13a00640 1 --2- 172.21.15.28:0/1250508345 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5f0c109850 0x7f5f0c109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:07.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.466+0000 7f5f13a00640 1 --2- 172.21.15.28:0/1250508345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0c108660 0x7f5f0c108a80 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:07.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.466+0000 7f5f13a00640 1 -- 172.21.15.28:0/1250508345 >> 172.21.15.28:0/1250508345 conn(0x7f5f0c077600 msgr2=0x7f5f0c079a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:07.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.466+0000 7f5f13a00640 1 -- 172.21.15.28:0/1250508345 shutdown_connections 2024-04-23T14:44:07.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.466+0000 7f5f13a00640 1 -- 172.21.15.28:0/1250508345 wait complete. 2024-04-23T14:44:07.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.467+0000 7f5f13a00640 1 Processor -- start 2024-04-23T14:44:07.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.467+0000 7f5f13a00640 1 -- start start 2024-04-23T14:44:07.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.468+0000 7f5f13a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5f0c108660 0x7f5f0c19ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:07.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.468+0000 7f5f13a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0c109850 0x7f5f0c19efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:07.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.468+0000 7f5f13a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f0c10a210 0x7f5f0c1a5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:07.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.468+0000 7f5f13a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5f0c1142e0 con 0x7f5f0c109850 2024-04-23T14:44:07.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.468+0000 7f5f13a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5f0c114160 con 0x7f5f0c10a210 2024-04-23T14:44:07.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.468+0000 7f5f13a00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5f0c114460 con 0x7f5f0c108660 2024-04-23T14:44:07.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.468+0000 7f5f0be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0c109850 0x7f5f0c19efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:07.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.468+0000 7f5f0be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0c109850 0x7f5f0c19efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36402/0 (socket says 172.21.15.28:36402) 2024-04-23T14:44:07.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.468+0000 7f5f0be00640 1 -- 172.21.15.28:0/250165951 learned_addr learned my addr 172.21.15.28:0/250165951 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:07.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.468+0000 7f5f11000640 1 --2- 172.21.15.28:0/250165951 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5f0c108660 0x7f5f0c19ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:07.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.468+0000 7f5f11a00640 1 --2- 172.21.15.28:0/250165951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f0c10a210 0x7f5f0c1a5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:07.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.468+0000 7f5f0be00640 1 -- 172.21.15.28:0/250165951 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5f0c108660 msgr2=0x7f5f0c19ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:07.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.468+0000 7f5f0be00640 1 --2- 172.21.15.28:0/250165951 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5f0c108660 0x7f5f0c19ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:07.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.468+0000 7f5f0be00640 1 -- 172.21.15.28:0/250165951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f0c10a210 msgr2=0x7f5f0c1a5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:07.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.468+0000 7f5f0be00640 1 --2- 172.21.15.28:0/250165951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f0c10a210 0x7f5f0c1a5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:07.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.468+0000 7f5f0be00640 1 -- 172.21.15.28:0/250165951 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f0c1a65f0 con 0x7f5f0c109850 2024-04-23T14:44:07.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.469+0000 7f5f11000640 1 --2- 172.21.15.28:0/250165951 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5f0c108660 0x7f5f0c19ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:44:07.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.469+0000 7f5f11a00640 1 --2- 172.21.15.28:0/250165951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f0c10a210 0x7f5f0c1a5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:44:07.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.469+0000 7f5f0be00640 1 --2- 172.21.15.28:0/250165951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0c109850 0x7f5f0c19efb0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5ef800e9f0 tx=0x7f5ef800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:07.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.469+0000 7f5f09600640 1 -- 172.21.15.28:0/250165951 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef800cd80 con 0x7f5f0c109850 2024-04-23T14:44:07.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.469+0000 7f5f09600640 1 -- 172.21.15.28:0/250165951 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ef8005500 con 0x7f5f0c109850 2024-04-23T14:44:07.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.469+0000 7f5f09600640 1 -- 172.21.15.28:0/250165951 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef8010690 con 0x7f5f0c109850 2024-04-23T14:44:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.469+0000 7f5f13a00640 1 -- 172.21.15.28:0/250165951 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f0c1a68e0 con 0x7f5f0c109850 2024-04-23T14:44:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.469+0000 7f5f13a00640 1 -- 172.21.15.28:0/250165951 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f0c1a6de0 con 0x7f5f0c109850 2024-04-23T14:44:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.470+0000 7f5f13a00640 1 -- 172.21.15.28:0/250165951 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f0c10f9a0 con 0x7f5f0c109850 2024-04-23T14:44:07.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.473+0000 7f5f09600640 1 -- 172.21.15.28:0/250165951 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7f5ef80046c0 con 0x7f5f0c109850 2024-04-23T14:44:07.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.474+0000 7f5f09600640 1 --2- 172.21.15.28:0/250165951 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5ef40a3db0 0x7f5ef40a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:07.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.475+0000 7f5f09600640 1 -- 172.21.15.28:0/250165951 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f5ef80d0e00 con 0x7f5f0c109850 2024-04-23T14:44:07.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.475+0000 7f5f11000640 1 --2- 172.21.15.28:0/250165951 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5ef40a3db0 0x7f5ef40a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:07.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.476+0000 7f5f11000640 1 --2- 172.21.15.28:0/250165951 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5ef40a3db0 0x7f5ef40a6270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5f00030410 tx=0x7f5f00030550 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:07.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.479+0000 7f5f09600640 1 -- 172.21.15.28:0/250165951 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5ef809d2e0 con 0x7f5f0c109850 2024-04-23T14:44:07.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.697+0000 7f5f13a00640 1 -- 172.21.15.28:0/250165951 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5f0c02d870 con 0x7f5f0c109850 2024-04-23T14:44:07.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.697+0000 7f5f09600640 1 -- 172.21.15.28:0/250165951 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f5ef809cbe0 con 0x7f5f0c109850 2024-04-23T14:44:07.698 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-23T14:44:07.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.700+0000 7f5f13a00640 1 -- 172.21.15.28:0/250165951 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5ef40a3db0 msgr2=0x7f5ef40a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:07.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.700+0000 7f5f13a00640 1 --2- 172.21.15.28:0/250165951 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5ef40a3db0 0x7f5ef40a6270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5f00030410 tx=0x7f5f00030550 comp rx=0 tx=0).stop 2024-04-23T14:44:07.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.700+0000 7f5f13a00640 1 -- 172.21.15.28:0/250165951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0c109850 msgr2=0x7f5f0c19efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:07.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.700+0000 7f5f13a00640 1 --2- 172.21.15.28:0/250165951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0c109850 0x7f5f0c19efb0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5ef800e9f0 tx=0x7f5ef800edf0 comp rx=0 tx=0).stop 2024-04-23T14:44:07.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.700+0000 7f5f13a00640 1 -- 172.21.15.28:0/250165951 shutdown_connections 2024-04-23T14:44:07.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.700+0000 7f5f13a00640 1 --2- 172.21.15.28:0/250165951 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5ef40a3db0 0x7f5ef40a6270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:07.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.700+0000 7f5f13a00640 1 --2- 172.21.15.28:0/250165951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f0c10a210 0x7f5f0c1a5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:07.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.700+0000 7f5f13a00640 1 --2- 172.21.15.28:0/250165951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0c109850 0x7f5f0c19efb0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:07.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.700+0000 7f5f13a00640 1 --2- 172.21.15.28:0/250165951 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5f0c108660 0x7f5f0c19ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:07.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.700+0000 7f5f13a00640 1 -- 172.21.15.28:0/250165951 >> 172.21.15.28:0/250165951 conn(0x7f5f0c077600 msgr2=0x7f5f0c10cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:07.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.700+0000 7f5f13a00640 1 -- 172.21.15.28:0/250165951 shutdown_connections 2024-04-23T14:44:07.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.700+0000 7f5f13a00640 1 -- 172.21.15.28:0/250165951 wait complete. 2024-04-23T14:44:07.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:44:07.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.916+0000 7f2f1d800640 1 -- 172.21.15.28:0/965910128 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f18104080 msgr2=0x7f2f181044a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:07.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.916+0000 7f2f1d800640 1 --2- 172.21.15.28:0/965910128 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f18104080 0x7f2f181044a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2f08009900 tx=0x7f2f0802a3f0 comp rx=0 tx=0).stop 2024-04-23T14:44:07.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.916+0000 7f2f1d800640 1 -- 172.21.15.28:0/965910128 shutdown_connections 2024-04-23T14:44:07.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.916+0000 7f2f1d800640 1 --2- 172.21.15.28:0/965910128 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f18105c30 0x7f2f1810c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:07.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.916+0000 7f2f1d800640 1 --2- 172.21.15.28:0/965910128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f18105270 0x7f2f181056f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:07.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.916+0000 7f2f1d800640 1 --2- 172.21.15.28:0/965910128 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f18104080 0x7f2f181044a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:07.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.916+0000 7f2f1d800640 1 -- 172.21.15.28:0/965910128 >> 172.21.15.28:0/965910128 conn(0x7f2f180ff610 msgr2=0x7f2f18101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:07.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.916+0000 7f2f1d800640 1 -- 172.21.15.28:0/965910128 shutdown_connections 2024-04-23T14:44:07.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.917+0000 7f2f1d800640 1 -- 172.21.15.28:0/965910128 wait complete. 2024-04-23T14:44:07.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.917+0000 7f2f1d800640 1 Processor -- start 2024-04-23T14:44:07.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.918+0000 7f2f1d800640 1 -- start start 2024-04-23T14:44:07.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.918+0000 7f2f1d800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f18104080 0x7f2f1819a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:07.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.918+0000 7f2f1d800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f18105270 0x7f2f1819a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:07.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.918+0000 7f2f1d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f18105c30 0x7f2f181a1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:07.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.918+0000 7f2f1d800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1810fcd0 con 0x7f2f18105c30 2024-04-23T14:44:07.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.918+0000 7f2f1d800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1810fb50 con 0x7f2f18105270 2024-04-23T14:44:07.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.918+0000 7f2f1d800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1810fe50 con 0x7f2f18104080 2024-04-23T14:44:07.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.918+0000 7f2f17400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f18105270 0x7f2f1819a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:07.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.918+0000 7f2f17e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f18104080 0x7f2f1819a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:07.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.918+0000 7f2f17400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f18105270 0x7f2f1819a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:42592/0 (socket says 172.21.15.28:42592) 2024-04-23T14:44:07.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.918+0000 7f2f17400640 1 -- 172.21.15.28:0/4225550200 learned_addr learned my addr 172.21.15.28:0/4225550200 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:07.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.918+0000 7f2f1ce00640 1 --2- 172.21.15.28:0/4225550200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f18105c30 0x7f2f181a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:07.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.919+0000 7f2f17400640 1 -- 172.21.15.28:0/4225550200 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f18104080 msgr2=0x7f2f1819a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:07.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.919+0000 7f2f17400640 1 --2- 172.21.15.28:0/4225550200 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f18104080 0x7f2f1819a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:07.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.919+0000 7f2f17400640 1 -- 172.21.15.28:0/4225550200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f18105c30 msgr2=0x7f2f181a1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:07.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.919+0000 7f2f17400640 1 --2- 172.21.15.28:0/4225550200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f18105c30 0x7f2f181a1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:07.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.919+0000 7f2f17400640 1 -- 172.21.15.28:0/4225550200 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f181a1fd0 con 0x7f2f18105270 2024-04-23T14:44:07.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.919+0000 7f2f17400640 1 --2- 172.21.15.28:0/4225550200 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f18105270 0x7f2f1819a940 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2f0400e9f0 tx=0x7f2f0400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:07.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.919+0000 7f2f14c00640 1 -- 172.21.15.28:0/4225550200 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f0400cd80 con 0x7f2f18105270 2024-04-23T14:44:07.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.919+0000 7f2f14c00640 1 -- 172.21.15.28:0/4225550200 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f04010910 con 0x7f2f18105270 2024-04-23T14:44:07.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.919+0000 7f2f14c00640 1 -- 172.21.15.28:0/4225550200 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f04018990 con 0x7f2f18105270 2024-04-23T14:44:07.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.919+0000 7f2f1d800640 1 -- 172.21.15.28:0/4225550200 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f181a22c0 con 0x7f2f18105270 2024-04-23T14:44:07.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.919+0000 7f2f1d800640 1 -- 172.21.15.28:0/4225550200 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f181a2810 con 0x7f2f18105270 2024-04-23T14:44:07.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.920+0000 7f2f1d800640 1 -- 172.21.15.28:0/4225550200 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f1810b4c0 con 0x7f2f18105270 2024-04-23T14:44:07.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.923+0000 7f2f14c00640 1 -- 172.21.15.28:0/4225550200 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7f2f04010ac0 con 0x7f2f18105270 2024-04-23T14:44:07.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.925+0000 7f2f14c00640 1 --2- 172.21.15.28:0/4225550200 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f2efc0a3db0 0x7f2efc0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:07.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.925+0000 7f2f14c00640 1 -- 172.21.15.28:0/4225550200 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f2f040d0d20 con 0x7f2f18105270 2024-04-23T14:44:07.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.925+0000 7f2f17e00640 1 --2- 172.21.15.28:0/4225550200 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f2efc0a3db0 0x7f2efc0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:07.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.926+0000 7f2f17e00640 1 --2- 172.21.15.28:0/4225550200 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f2efc0a3db0 0x7f2efc0a6270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2f08005ab0 tx=0x7f2f08000bc0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:07.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:07.929+0000 7f2f14c00640 1 -- 172.21.15.28:0/4225550200 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2f0409cbb0 con 0x7f2f18105270 2024-04-23T14:44:08.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:07 smithi105 ceph-mon[32555]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T14:44:08.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:07 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3360904293' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:44:08.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:07 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3618987788' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T14:44:08.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:07 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:08.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:07 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:08.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:07 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:08.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:07 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:08.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:07 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:08.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:07 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:08.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:07 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:08.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:07 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:08.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:07 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:08.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:07 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:08.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:07 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:44:08.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:07 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:44:08.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:07 smithi038 ceph-mon[32348]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T14:44:08.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:07 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3360904293' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:44:08.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:07 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3618987788' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T14:44:08.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:07 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:08.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:07 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:08.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:07 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:08.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:07 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:08.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:07 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:08.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:07 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:08.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:07 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:08.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:07 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:08.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:07 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:08.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:07 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:08.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:07 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:44:08.124 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:07 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:44:08.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:07 smithi028 ceph-mon[27810]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T14:44:08.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:07 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3360904293' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:44:08.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:07 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3618987788' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T14:44:08.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:07 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:08.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:07 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:08.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:07 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:08.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:07 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:08.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:07 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:08.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:07 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:08.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:07 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:08.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:07 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:08.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:07 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:08.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:07 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:44:08.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:07 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:44:08.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:07 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:44:08.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.322+0000 7f2f1d800640 1 -- 172.21.15.28:0/4225550200 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2f1802d870 con 0x7f2f18105270 2024-04-23T14:44:08.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.323+0000 7f2f14c00640 1 -- 172.21.15.28:0/4225550200 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7f2f040a0eb0 con 0x7f2f18105270 2024-04-23T14:44:08.324 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:44:08.324 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:44:08.324 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:44:08.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.326+0000 7f2f1d800640 1 -- 172.21.15.28:0/4225550200 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f2efc0a3db0 msgr2=0x7f2efc0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:08.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.326+0000 7f2f1d800640 1 --2- 172.21.15.28:0/4225550200 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f2efc0a3db0 0x7f2efc0a6270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2f08005ab0 tx=0x7f2f08000bc0 comp rx=0 tx=0).stop 2024-04-23T14:44:08.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.326+0000 7f2f1d800640 1 -- 172.21.15.28:0/4225550200 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f18105270 msgr2=0x7f2f1819a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:08.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.326+0000 7f2f1d800640 1 --2- 172.21.15.28:0/4225550200 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f18105270 0x7f2f1819a940 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2f0400e9f0 tx=0x7f2f0400eeb0 comp rx=0 tx=0).stop 2024-04-23T14:44:08.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.326+0000 7f2f1d800640 1 -- 172.21.15.28:0/4225550200 shutdown_connections 2024-04-23T14:44:08.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.326+0000 7f2f1d800640 1 --2- 172.21.15.28:0/4225550200 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f2efc0a3db0 0x7f2efc0a6270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:08.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.326+0000 7f2f1d800640 1 --2- 172.21.15.28:0/4225550200 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f18105270 0x7f2f1819a940 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:08.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.326+0000 7f2f1d800640 1 --2- 172.21.15.28:0/4225550200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f18105c30 0x7f2f181a1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:08.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.326+0000 7f2f1d800640 1 --2- 172.21.15.28:0/4225550200 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f18104080 0x7f2f1819a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:08.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.326+0000 7f2f1d800640 1 -- 172.21.15.28:0/4225550200 >> 172.21.15.28:0/4225550200 conn(0x7f2f180ff610 msgr2=0x7f2f18108690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:08.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.326+0000 7f2f1d800640 1 -- 172.21.15.28:0/4225550200 shutdown_connections 2024-04-23T14:44:08.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.326+0000 7f2f1d800640 1 -- 172.21.15.28:0/4225550200 wait complete. 2024-04-23T14:44:08.347 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:44:08.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.547+0000 7efe46a00640 1 -- 172.21.15.28:0/1339767428 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efe400769c0 msgr2=0x7efe40076de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:08.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.547+0000 7efe46a00640 1 --2- 172.21.15.28:0/1339767428 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efe400769c0 0x7efe40076de0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7efe34009900 tx=0x7efe3402a3f0 comp rx=0 tx=0).stop 2024-04-23T14:44:08.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.548+0000 7efe46a00640 1 -- 172.21.15.28:0/1339767428 shutdown_connections 2024-04-23T14:44:08.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.548+0000 7efe46a00640 1 --2- 172.21.15.28:0/1339767428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe400785a0 0x7efe40078e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:08.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.548+0000 7efe46a00640 1 --2- 172.21.15.28:0/1339767428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe40077c00 0x7efe40078060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:08.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.548+0000 7efe46a00640 1 --2- 172.21.15.28:0/1339767428 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efe400769c0 0x7efe40076de0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:08.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.548+0000 7efe46a00640 1 -- 172.21.15.28:0/1339767428 >> 172.21.15.28:0/1339767428 conn(0x7efe400ff610 msgr2=0x7efe40101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:08.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.548+0000 7efe46a00640 1 -- 172.21.15.28:0/1339767428 shutdown_connections 2024-04-23T14:44:08.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.548+0000 7efe46a00640 1 -- 172.21.15.28:0/1339767428 wait complete. 2024-04-23T14:44:08.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.549+0000 7efe46a00640 1 Processor -- start 2024-04-23T14:44:08.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.550+0000 7efe46a00640 1 -- start start 2024-04-23T14:44:08.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.550+0000 7efe46a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe400769c0 0x7efe4019e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:08.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.550+0000 7efe46a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe40077c00 0x7efe4019ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:08.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.550+0000 7efe46a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efe400785a0 0x7efe401a5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:08.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.550+0000 7efe46a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efe40114170 con 0x7efe40077c00 2024-04-23T14:44:08.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.550+0000 7efe46a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efe40113ff0 con 0x7efe400769c0 2024-04-23T14:44:08.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.550+0000 7efe46a00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7efe401142f0 con 0x7efe400785a0 2024-04-23T14:44:08.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.550+0000 7efe44c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe40077c00 0x7efe4019ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:08.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.550+0000 7efe44c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe40077c00 0x7efe4019ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42910/0 (socket says 172.21.15.28:42910) 2024-04-23T14:44:08.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.550+0000 7efe44c00640 1 -- 172.21.15.28:0/3841187774 learned_addr learned my addr 172.21.15.28:0/3841187774 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:08.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.550+0000 7efe45600640 1 --2- 172.21.15.28:0/3841187774 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe400769c0 0x7efe4019e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:08.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.550+0000 7efe46000640 1 --2- 172.21.15.28:0/3841187774 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efe400785a0 0x7efe401a5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:08.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.551+0000 7efe44c00640 1 -- 172.21.15.28:0/3841187774 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efe400785a0 msgr2=0x7efe401a5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:08.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.551+0000 7efe44c00640 1 --2- 172.21.15.28:0/3841187774 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efe400785a0 0x7efe401a5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:08.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.551+0000 7efe44c00640 1 -- 172.21.15.28:0/3841187774 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe400769c0 msgr2=0x7efe4019e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:08.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.551+0000 7efe44c00640 1 --2- 172.21.15.28:0/3841187774 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe400769c0 0x7efe4019e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:08.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.551+0000 7efe44c00640 1 -- 172.21.15.28:0/3841187774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe401a63a0 con 0x7efe40077c00 2024-04-23T14:44:08.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.551+0000 7efe46000640 1 --2- 172.21.15.28:0/3841187774 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efe400785a0 0x7efe401a5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:44:08.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.551+0000 7efe44c00640 1 --2- 172.21.15.28:0/3841187774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe40077c00 0x7efe4019ed10 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7efe2c00d950 tx=0x7efe2c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:08.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.552+0000 7efe3e000640 1 -- 172.21.15.28:0/3841187774 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe2c017070 con 0x7efe40077c00 2024-04-23T14:44:08.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.552+0000 7efe3e000640 1 -- 172.21.15.28:0/3841187774 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe2c009c30 con 0x7efe40077c00 2024-04-23T14:44:08.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.552+0000 7efe3e000640 1 -- 172.21.15.28:0/3841187774 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe2c00f6b0 con 0x7efe40077c00 2024-04-23T14:44:08.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.552+0000 7efe46a00640 1 -- 172.21.15.28:0/3841187774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe401a6690 con 0x7efe40077c00 2024-04-23T14:44:08.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.552+0000 7efe46a00640 1 -- 172.21.15.28:0/3841187774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe401a6c30 con 0x7efe40077c00 2024-04-23T14:44:08.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.553+0000 7efe46a00640 1 -- 172.21.15.28:0/3841187774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe40110a10 con 0x7efe40077c00 2024-04-23T14:44:08.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.555+0000 7efe3e000640 1 -- 172.21.15.28:0/3841187774 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7efe2c0109d0 con 0x7efe40077c00 2024-04-23T14:44:08.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.557+0000 7efe3e000640 1 --2- 172.21.15.28:0/3841187774 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7efe280a3e30 0x7efe280a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:08.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.557+0000 7efe3e000640 1 -- 172.21.15.28:0/3841187774 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7efe2c0d04f0 con 0x7efe40077c00 2024-04-23T14:44:08.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.557+0000 7efe45600640 1 --2- 172.21.15.28:0/3841187774 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7efe280a3e30 0x7efe280a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:08.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.558+0000 7efe45600640 1 --2- 172.21.15.28:0/3841187774 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7efe280a3e30 0x7efe280a62f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7efe34031440 tx=0x7efe340315b0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:08.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.563+0000 7efe3e000640 1 -- 172.21.15.28:0/3841187774 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efe2c009de0 con 0x7efe40077c00 2024-04-23T14:44:08.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.780+0000 7efe46a00640 1 -- 172.21.15.28:0/3841187774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efe4002d870 con 0x7efe40077c00 2024-04-23T14:44:08.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.782+0000 7efe3e000640 1 -- 172.21.15.28:0/3841187774 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+23026 (secure 0 0 0) 0x7efe2c09c380 con 0x7efe40077c00 2024-04-23T14:44:08.784 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:44:08.784 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":88,"fsid":"1ee6e996-017e-11ef-bc93-c7b262605968","created":"2024-04-23T14:32:34.581082+0000","modified":"2024-04-23T14:44:01.286339+0000","last_up_change":"2024-04-23T14:40:43.375846+0000","last_in_change":"2024-04-23T14:40:23.576939+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:36:27.698112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:41:37.030341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:41:38.444742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e0b427bc-be65-4543-99cc-2f53ad98ddad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6803","nonce":2993445336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6805","nonce":2993445336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6809","nonce":2993445336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6807","nonce":2993445336}]},"public_addr":"172.21.15.28:6803/2993445336","cluster_addr":"172.21.15.28:6805/2993445336","heartbeat_back_addr":"172.21.15.28:6809/2993445336","heartbeat_front_addr":"172.21.15.28:6807/2993445336","state":["exists","up"]},{"osd":1,"uuid":"ef527ec9-1bf3-4bb0-8df3-f5a015af557c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6801","nonce":4024093383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6803","nonce":4024093383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6807","nonce":4024093383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6805","nonce":4024093383}]},"public_addr":"172.21.15.38:6801/4024093383","cluster_addr":"172.21.15.38:6803/4024093383","heartbeat_back_addr":"172.21.15.38:6807/4024093383","heartbeat_front_addr":"172.21.15.38:6805/4024093383","state":["exists","up"]},{"osd":2,"uuid":"b33b23d9-adce-4c11-b304-2e117bdf6bfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6801","nonce":570739029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6803","nonce":570739029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6807","nonce":570739029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6805","nonce":570739029}]},"public_addr":"172.21.15.105:6801/570739029","cluster_addr":"172.21.15.105:6803/570739029","heartbeat_back_addr":"172.21.15.105:6807/570739029","heartbeat_front_addr":"172.21.15.105:6805/570739029","state":["exists","up"]},{"osd":3,"uuid":"28bd5608-b426-434e-9db1-bdaa9828bdb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6811","nonce":2480400247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6813","nonce":2480400247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6817","nonce":2480400247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6815","nonce":2480400247}]},"public_addr":"172.21.15.28:6811/2480400247","cluster_addr":"172.21.15.28:6813/2480400247","heartbeat_back_addr":"172.21.15.28:6817/2480400247","heartbeat_front_addr":"172.21.15.28:6815/2480400247","state":["exists","up"]},{"osd":4,"uuid":"e5d1c89b-025f-4d57-8807-59d75c787b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6809","nonce":3772189246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6811","nonce":3772189246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6815","nonce":3772189246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6813","nonce":3772189246}]},"public_addr":"172.21.15.38:6809/3772189246","cluster_addr":"172.21.15.38:6811/3772189246","heartbeat_back_addr":"172.21.15.38:6815/3772189246","heartbeat_front_addr":"172.21.15.38:6813/3772189246","state":["exists","up"]},{"osd":5,"uuid":"9cf9cf4c-aafd-4ed3-bfb1-0527710a0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6809","nonce":928755894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6811","nonce":928755894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6815","nonce":928755894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6813","nonce":928755894}]},"public_addr":"172.21.15.105:6809/928755894","cluster_addr":"172.21.15.105:6811/928755894","heartbeat_back_addr":"172.21.15.105:6815/928755894","heartbeat_front_addr":"172.21.15.105:6813/928755894","state":["exists","up"]},{"osd":6,"uuid":"304db525-d24d-4687-bfa0-f92bdf66d1e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6819","nonce":897979441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6821","nonce":897979441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6825","nonce":897979441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6823","nonce":897979441}]},"public_addr":"172.21.15.28:6819/897979441","cluster_addr":"172.21.15.28:6821/897979441","heartbeat_back_addr":"172.21.15.28:6825/897979441","heartbeat_front_addr":"172.21.15.28:6823/897979441","state":["exists","up"]},{"osd":7,"uuid":"08e01d59-03f3-47b1-b45f-35ec5a50992c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6817","nonce":2400251786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6819","nonce":2400251786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6823","nonce":2400251786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6821","nonce":2400251786}]},"public_addr":"172.21.15.38:6817/2400251786","cluster_addr":"172.21.15.38:6819/2400251786","heartbeat_back_addr":"172.21.15.38:6823/2400251786","heartbeat_front_addr":"172.21.15.38:6821/2400251786","state":["exists","up"]},{"osd":8,"uuid":"64ee1e95-0756-48b1-9c00-472d0c3cbd29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6817","nonce":1682312269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6819","nonce":1682312269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6823","nonce":1682312269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6821","nonce":1682312269}]},"public_addr":"172.21.15.105:6817/1682312269","cluster_addr":"172.21.15.105:6819/1682312269","heartbeat_back_addr":"172.21.15.105:6823/1682312269","heartbeat_front_addr":"172.21.15.105:6821/1682312269","state":["exists","up"]},{"osd":9,"uuid":"18f60893-3ffd-48c2-9db5-a67d35611d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6827","nonce":1292942516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6829","nonce":1292942516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6833","nonce":1292942516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6831","nonce":1292942516}]},"public_addr":"172.21.15.28:6827/1292942516","cluster_addr":"172.21.15.28:6829/1292942516","heartbeat_back_addr":"172.21.15.28:6833/1292942516","heartbeat_front_addr":"172.21.15.28:6831/1292942516","state":["exists","up"]},{"osd":10,"uuid":"76c06725-7040-4118-8894-fe808a87f999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6825","nonce":2444616468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6827","nonce":2444616468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6831","nonce":2444616468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6829","nonce":2444616468}]},"public_addr":"172.21.15.38:6825/2444616468","cluster_addr":"172.21.15.38:6827/2444616468","heartbeat_back_addr":"172.21.15.38:6831/2444616468","heartbeat_front_addr":"172.21.15.38:6829/2444616468","state":["exists","up"]},{"osd":11,"uuid":"e8fc9454-5540-4e2e-91bf-3ba251dbd1a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6825","nonce":3400299562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6827","nonce":3400299562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6831","nonce":3400299562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6829","nonce":3400299562}]},"public_addr":"172.21.15.105:6825/3400299562","cluster_addr":"172.21.15.105:6827/3400299562","heartbeat_back_addr":"172.21.15.105:6831/3400299562","heartbeat_front_addr":"172.21.15.105:6829/3400299562","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:35:33.615395+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:00.545293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:24.940211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:57.684887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:24.519061+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:49.010804+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:22.638478+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:49.559104+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:14.011771+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:47.859950+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:16.298445+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:41.619069+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.2d","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/38494154":"2024-04-24T14:44:01.286264+0000","172.21.15.28:0/905362079":"2024-04-24T14:44:01.286264+0000","172.21.15.28:6801/1997136173":"2024-04-24T14:44:01.286264+0000","172.21.15.28:6800/1997136173":"2024-04-24T14:44:01.286264+0000","172.21.15.28:0/590495740":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/1163172512":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/4290328460":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/4145020172":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/1997097543":"2024-04-24T14:44:01.286264+0000","172.21.15.28:0/3847772687":"2024-04-24T14:44:01.286264+0000","172.21.15.28:6800/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/720626958":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/948450262":"2024-04-24T14:33:07.576363+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:44:08.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.785+0000 7efe46a00640 1 -- 172.21.15.28:0/3841187774 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7efe280a3e30 msgr2=0x7efe280a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:08.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.785+0000 7efe46a00640 1 --2- 172.21.15.28:0/3841187774 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7efe280a3e30 0x7efe280a62f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7efe34031440 tx=0x7efe340315b0 comp rx=0 tx=0).stop 2024-04-23T14:44:08.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.785+0000 7efe46a00640 1 -- 172.21.15.28:0/3841187774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe40077c00 msgr2=0x7efe4019ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:08.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.785+0000 7efe46a00640 1 --2- 172.21.15.28:0/3841187774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe40077c00 0x7efe4019ed10 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7efe2c00d950 tx=0x7efe2c00de10 comp rx=0 tx=0).stop 2024-04-23T14:44:08.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.785+0000 7efe46a00640 1 -- 172.21.15.28:0/3841187774 shutdown_connections 2024-04-23T14:44:08.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.785+0000 7efe46a00640 1 --2- 172.21.15.28:0/3841187774 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7efe280a3e30 0x7efe280a62f0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:08.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.785+0000 7efe46a00640 1 --2- 172.21.15.28:0/3841187774 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efe400785a0 0x7efe401a5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:08.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.785+0000 7efe46a00640 1 --2- 172.21.15.28:0/3841187774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe40077c00 0x7efe4019ed10 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:08.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.785+0000 7efe46a00640 1 --2- 172.21.15.28:0/3841187774 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe400769c0 0x7efe4019e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:08.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.785+0000 7efe46a00640 1 -- 172.21.15.28:0/3841187774 >> 172.21.15.28:0/3841187774 conn(0x7efe400ff610 msgr2=0x7efe40101110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:08.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.785+0000 7efe46a00640 1 -- 172.21.15.28:0/3841187774 shutdown_connections 2024-04-23T14:44:08.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:08.785+0000 7efe46a00640 1 -- 172.21.15.28:0/3841187774 wait complete. 2024-04-23T14:44:08.810 DEBUG:tasks.fwd_scrub:Yielding 2024-04-23T14:44:08.810 INFO:teuthology.run_tasks:Running task exec... 2024-04-23T14:44:08.819 INFO:teuthology.task.exec:Executing custom commands... 2024-04-23T14:44:08.820 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-04-23T14:44:08.820 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-23T14:44:08.821 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-04-23T14:44:08.821 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-23T14:44:08.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:44:08.822 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-23T14:44:09.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.001+0000 7f7c77e00640 1 -- 172.21.15.28:0/4218810195 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c781040c0 msgr2=0x7f7c781044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:09.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.001+0000 7f7c77e00640 1 --2- 172.21.15.28:0/4218810195 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c781040c0 0x7f7c781044e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7c68009860 tx=0x7f7c6802a340 comp rx=0 tx=0).stop 2024-04-23T14:44:09.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.001+0000 7f7c77e00640 1 -- 172.21.15.28:0/4218810195 shutdown_connections 2024-04-23T14:44:09.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.001+0000 7f7c77e00640 1 --2- 172.21.15.28:0/4218810195 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c78105c70 0x7f7c7810c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.001+0000 7f7c77e00640 1 --2- 172.21.15.28:0/4218810195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c781052b0 0x7f7c78105730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.001+0000 7f7c77e00640 1 --2- 172.21.15.28:0/4218810195 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c781040c0 0x7f7c781044e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.001+0000 7f7c77e00640 1 -- 172.21.15.28:0/4218810195 >> 172.21.15.28:0/4218810195 conn(0x7f7c780ff630 msgr2=0x7f7c78101a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:09.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.002+0000 7f7c77e00640 1 -- 172.21.15.28:0/4218810195 shutdown_connections 2024-04-23T14:44:09.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.002+0000 7f7c77e00640 1 -- 172.21.15.28:0/4218810195 wait complete. 2024-04-23T14:44:09.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.003+0000 7f7c77e00640 1 Processor -- start 2024-04-23T14:44:09.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.003+0000 7f7c77e00640 1 -- start start 2024-04-23T14:44:09.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.003+0000 7f7c77e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c781040c0 0x7f7c7819a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.003+0000 7f7c77e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c781052b0 0x7f7c7819a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.003+0000 7f7c77e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c78105c70 0x7f7c781a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.003+0000 7f7c77e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7c7810fd50 con 0x7f7c781040c0 2024-04-23T14:44:09.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.003+0000 7f7c77e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7c7810fbd0 con 0x7f7c781052b0 2024-04-23T14:44:09.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.003+0000 7f7c77e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7c7810fed0 con 0x7f7c78105c70 2024-04-23T14:44:09.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.004+0000 7f7c77400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c78105c70 0x7f7c781a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.004+0000 7f7c76000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c781052b0 0x7f7c7819a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.004+0000 7f7c77400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c78105c70 0x7f7c781a1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:40292/0 (socket says 172.21.15.28:40292) 2024-04-23T14:44:09.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.004+0000 7f7c76a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c781040c0 0x7f7c7819a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.004+0000 7f7c77400640 1 -- 172.21.15.28:0/3543873454 learned_addr learned my addr 172.21.15.28:0/3543873454 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:09.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.004+0000 7f7c77400640 1 -- 172.21.15.28:0/3543873454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c781052b0 msgr2=0x7f7c7819a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:09.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.004+0000 7f7c77400640 1 --2- 172.21.15.28:0/3543873454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c781052b0 0x7f7c7819a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.004+0000 7f7c77400640 1 -- 172.21.15.28:0/3543873454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c781040c0 msgr2=0x7f7c7819a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:09.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.004+0000 7f7c77400640 1 --2- 172.21.15.28:0/3543873454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c781040c0 0x7f7c7819a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.004+0000 7f7c77400640 1 -- 172.21.15.28:0/3543873454 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c781a1fc0 con 0x7f7c78105c70 2024-04-23T14:44:09.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.004+0000 7f7c76000640 1 --2- 172.21.15.28:0/3543873454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c781052b0 0x7f7c7819a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:44:09.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.005+0000 7f7c77400640 1 --2- 172.21.15.28:0/3543873454 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c78105c70 0x7f7c781a1950 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7c7000eb10 tx=0x7f7c7000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:09.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.005+0000 7f7c7ca00640 1 -- 172.21.15.28:0/3543873454 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c7000ce20 con 0x7f7c78105c70 2024-04-23T14:44:09.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.005+0000 7f7c7ca00640 1 -- 172.21.15.28:0/3543873454 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c70005020 con 0x7f7c78105c70 2024-04-23T14:44:09.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.005+0000 7f7c7ca00640 1 -- 172.21.15.28:0/3543873454 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c70010690 con 0x7f7c78105c70 2024-04-23T14:44:09.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.005+0000 7f7c77e00640 1 -- 172.21.15.28:0/3543873454 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c781a22b0 con 0x7f7c78105c70 2024-04-23T14:44:09.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.005+0000 7f7c77e00640 1 -- 172.21.15.28:0/3543873454 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c781a27b0 con 0x7f7c78105c70 2024-04-23T14:44:09.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.007+0000 7f7c77e00640 1 -- 172.21.15.28:0/3543873454 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c7810b430 con 0x7f7c78105c70 2024-04-23T14:44:09.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.010+0000 7f7c7ca00640 1 -- 172.21.15.28:0/3543873454 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7f7c70005540 con 0x7f7c78105c70 2024-04-23T14:44:09.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.011+0000 7f7c7ca00640 1 --2- 172.21.15.28:0/3543873454 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7c5c0a3e20 0x7f7c5c0a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.011+0000 7f7c7ca00640 1 -- 172.21.15.28:0/3543873454 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f7c700d0ff0 con 0x7f7c78105c70 2024-04-23T14:44:09.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.011+0000 7f7c76a00640 1 --2- 172.21.15.28:0/3543873454 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7c5c0a3e20 0x7f7c5c0a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.012+0000 7f7c76a00640 1 --2- 172.21.15.28:0/3543873454 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7c5c0a3e20 0x7f7c5c0a62e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7c68009830 tx=0x7f7c68030570 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.015+0000 7f7c7ca00640 1 -- 172.21.15.28:0/3543873454 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7c7009d510 con 0x7f7c78105c70 2024-04-23T14:44:09.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.024+0000 7fa088200640 1 -- 172.21.15.28:0/3460418764 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa080074a60 msgr2=0x7fa080074ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:09.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.024+0000 7fa088200640 1 --2- 172.21.15.28:0/3460418764 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa080074a60 0x7fa080074ee0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa06c009900 tx=0x7fa06c02a3f0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.024+0000 7fa088200640 1 -- 172.21.15.28:0/3460418764 shutdown_connections 2024-04-23T14:44:09.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.024+0000 7fa088200640 1 --2- 172.21.15.28:0/3460418764 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa080109f10 0x7fa08010c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.024+0000 7fa088200640 1 --2- 172.21.15.28:0/3460418764 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa080074a60 0x7fa080074ee0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.024+0000 7fa088200640 1 --2- 172.21.15.28:0/3460418764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa080076040 0x7fa080074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.024+0000 7fa088200640 1 -- 172.21.15.28:0/3460418764 >> 172.21.15.28:0/3460418764 conn(0x7fa0800fd5e0 msgr2=0x7fa0800ffa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:09.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.024+0000 7fa088200640 1 -- 172.21.15.28:0/3460418764 shutdown_connections 2024-04-23T14:44:09.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.025+0000 7fa088200640 1 -- 172.21.15.28:0/3460418764 wait complete. 2024-04-23T14:44:09.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.025+0000 7fa088200640 1 Processor -- start 2024-04-23T14:44:09.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.026+0000 7fa088200640 1 -- start start 2024-04-23T14:44:09.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.026+0000 7fa088200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa080074a60 0x7fa08019a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.026+0000 7fa088200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa080076040 0x7fa08019ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.026+0000 7fa088200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa080109f10 0x7fa0801a1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.026+0000 7fa088200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa08010fee0 con 0x7fa080076040 2024-04-23T14:44:09.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.026+0000 7fa088200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa08010fd60 con 0x7fa080109f10 2024-04-23T14:44:09.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.026+0000 7fa088200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa080110060 con 0x7fa080074a60 2024-04-23T14:44:09.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.026+0000 7fa085800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa080074a60 0x7fa08019a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.026+0000 7fa084e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa080076040 0x7fa08019ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.026+0000 7fa086200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa080109f10 0x7fa0801a1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.026+0000 7fa085800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa080074a60 0x7fa08019a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:40316/0 (socket says 172.21.15.28:40316) 2024-04-23T14:44:09.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.026+0000 7fa085800640 1 -- 172.21.15.28:0/1745305913 learned_addr learned my addr 172.21.15.28:0/1745305913 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:09.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.027+0000 7fa086200640 1 -- 172.21.15.28:0/1745305913 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa080074a60 msgr2=0x7fa08019a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:09.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.027+0000 7fa086200640 1 --2- 172.21.15.28:0/1745305913 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa080074a60 0x7fa08019a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.027+0000 7fa086200640 1 -- 172.21.15.28:0/1745305913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa080076040 msgr2=0x7fa08019ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:09.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.027+0000 7fa086200640 1 --2- 172.21.15.28:0/1745305913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa080076040 0x7fa08019ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.027+0000 7fa086200640 1 -- 172.21.15.28:0/1745305913 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0801a21f0 con 0x7fa080109f10 2024-04-23T14:44:09.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.027+0000 7fa086200640 1 --2- 172.21.15.28:0/1745305913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa080109f10 0x7fa0801a1b80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa07800ec50 tx=0x7fa07800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:09.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.027+0000 7fa07e000640 1 -- 172.21.15.28:0/1745305913 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa078003b70 con 0x7fa080109f10 2024-04-23T14:44:09.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.027+0000 7fa07e000640 1 -- 172.21.15.28:0/1745305913 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa078003d10 con 0x7fa080109f10 2024-04-23T14:44:09.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.027+0000 7fa088200640 1 -- 172.21.15.28:0/1745305913 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0801a24e0 con 0x7fa080109f10 2024-04-23T14:44:09.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.027+0000 7fa07e000640 1 -- 172.21.15.28:0/1745305913 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa078010890 con 0x7fa080109f10 2024-04-23T14:44:09.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.028+0000 7fa088200640 1 -- 172.21.15.28:0/1745305913 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0801a29b0 con 0x7fa080109f10 2024-04-23T14:44:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.031+0000 7fa07e000640 1 -- 172.21.15.28:0/1745305913 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7fa078003fc0 con 0x7fa080109f10 2024-04-23T14:44:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.032+0000 7fa08af99740 1 Processor -- start 2024-04-23T14:44:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.032+0000 7fa08af99740 1 -- start start 2024-04-23T14:44:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.032+0000 7fa08af99740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56035fc09d40 0x56035fc0a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.032+0000 7fa08af99740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56035fc0a730 0x56035fc0f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.032+0000 7fa08af99740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035fc0f8a0 0x56035fc11d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.032+0000 7fa08af99740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56035fb6ee70 con 0x56035fc0f8a0 2024-04-23T14:44:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.032+0000 7fa08af99740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56035fb15380 con 0x56035fc09d40 2024-04-23T14:44:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.032+0000 7fa08af99740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56035fb6ae30 con 0x56035fc0a730 2024-04-23T14:44:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.032+0000 7fa085800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56035fc0a730 0x56035fc0f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.032+0000 7fa07e000640 1 --2- 172.21.15.28:0/1745305913 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa0680a3db0 0x7fa0680a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.032+0000 7fa085800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56035fc0a730 0x56035fc0f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:40318/0 (socket says 172.21.15.28:40318) 2024-04-23T14:44:09.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.032+0000 7fa085800640 1 -- 172.21.15.28:0/2469437476 learned_addr learned my addr 172.21.15.28:0/2469437476 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:09.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.032+0000 7fa084e00640 1 --2- 172.21.15.28:0/2469437476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56035fc09d40 0x56035fc0a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.033+0000 7fa07e000640 1 -- 172.21.15.28:0/1745305913 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fa0780d0f80 con 0x7fa080109f10 2024-04-23T14:44:09.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.033+0000 7fa084e00640 1 --2- 172.21.15.28:0/2469437476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035fc0f8a0 0x56035fc11d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.033+0000 7fa085800640 1 -- 172.21.15.28:0/2469437476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56035fc09d40 msgr2=0x56035fc0a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:09.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.033+0000 7fa085800640 1 --2- 172.21.15.28:0/2469437476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56035fc09d40 0x56035fc0a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.033+0000 7fa085800640 1 -- 172.21.15.28:0/2469437476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035fc0f8a0 msgr2=0x56035fc11d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:09.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.033+0000 7fa085800640 1 --2- 172.21.15.28:0/2469437476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035fc0f8a0 0x56035fc11d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.033+0000 7fa085800640 1 -- 172.21.15.28:0/2469437476 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56035f9497e0 con 0x56035fc0a730 2024-04-23T14:44:09.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.033+0000 7fa086200640 1 --2- 172.21.15.28:0/1745305913 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa0680a3db0 0x7fa0680a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.033+0000 7fa084e00640 1 --2- 172.21.15.28:0/2469437476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035fc0f8a0 0x56035fc11d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:44:09.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.033+0000 7fa085800640 1 --2- 172.21.15.28:0/2469437476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56035fc0a730 0x56035fc0f230 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa074009dc0 tx=0x7fa074011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1543b151c983a47 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:09.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.034+0000 7fa086200640 1 --2- 172.21.15.28:0/1745305913 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa0680a3db0 0x7fa0680a6270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa0780ce620 tx=0x7fa07800ef50 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:09.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.034+0000 7fa072a00640 1 -- 172.21.15.28:0/2469437476 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa074005220 con 0x56035fc0a730 2024-04-23T14:44:09.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.034+0000 7fa072a00640 1 -- 172.21.15.28:0/2469437476 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa074004c80 con 0x56035fc0a730 2024-04-23T14:44:09.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.034+0000 7fa072a00640 1 -- 172.21.15.28:0/2469437476 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa074019970 con 0x56035fc0a730 2024-04-23T14:44:09.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:08 smithi105 ceph-mon[32555]: Adjusting osd_memory_target on smithi028 to 177.5M 2024-04-23T14:44:09.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:08 smithi105 ceph-mon[32555]: Unable to set osd_memory_target on smithi028 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-23T14:44:09.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:08 smithi105 ceph-mon[32555]: Updating smithi028:/etc/ceph/ceph.conf 2024-04-23T14:44:09.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:08 smithi105 ceph-mon[32555]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-23T14:44:09.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:08 smithi105 ceph-mon[32555]: Updating smithi105:/etc/ceph/ceph.conf 2024-04-23T14:44:09.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:08 smithi105 ceph-mon[32555]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 0 B/s wr, 26 op/s 2024-04-23T14:44:09.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:08 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/250165951' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T14:44:09.038 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:08 smithi105 ceph-mon[32555]: Updating smithi038:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T14:44:09.038 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:08 smithi105 ceph-mon[32555]: Updating smithi105:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T14:44:09.038 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:08 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/4225550200' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:44:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.035+0000 7fa08af99740 1 -- 172.21.15.28:0/2469437476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56035fc0a730 msgr2=0x56035fc0f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.035+0000 7fa08af99740 1 --2- 172.21.15.28:0/2469437476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56035fc0a730 0x56035fc0f230 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa074009dc0 tx=0x7fa074011810 comp rx=0 tx=0).stop 2024-04-23T14:44:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.035+0000 7fa08af99740 1 -- 172.21.15.28:0/2469437476 shutdown_connections 2024-04-23T14:44:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.035+0000 7fa08af99740 1 --2- 172.21.15.28:0/2469437476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035fc0f8a0 0x56035fc11d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.035+0000 7fa08af99740 1 --2- 172.21.15.28:0/2469437476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56035fc0a730 0x56035fc0f230 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.035+0000 7fa08af99740 1 --2- 172.21.15.28:0/2469437476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56035fc09d40 0x56035fc0a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.035+0000 7fa08af99740 1 -- 172.21.15.28:0/2469437476 >> 172.21.15.28:0/2469437476 conn(0x56035fc053b0 msgr2=0x56035fc057b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.035+0000 7fa08af99740 1 -- 172.21.15.28:0/2469437476 shutdown_connections 2024-04-23T14:44:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.035+0000 7fa08af99740 1 -- 172.21.15.28:0/2469437476 wait complete. 2024-04-23T14:44:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.035+0000 7fa08af99740 1 -- start start 2024-04-23T14:44:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.036+0000 7fa08af99740 1 Processor -- start 2024-04-23T14:44:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.036+0000 7fa08af99740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035fc09d40 0x56035fc1ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.036+0000 7fa08af99740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56035fc0a730 0x56035fc1cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.036+0000 7fa08af99740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56035fc0f8a0 0x56035fc23fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.036+0000 7fa08af99740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56035fc12850 con 0x56035fc09d40 2024-04-23T14:44:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.036+0000 7fa08af99740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56035fb28660 con 0x56035fc0a730 2024-04-23T14:44:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.036+0000 7fa08af99740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56035fc129d0 con 0x56035fc0f8a0 2024-04-23T14:44:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.036+0000 7fa08af99740 10 client.0 _ll_register_callbacks cb 0x56035fb16aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:44:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.036+0000 7fa084e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035fc09d40 0x56035fc1ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.036+0000 7fa084e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035fc09d40 0x56035fc1ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42978/0 (socket says 172.21.15.28:42978) 2024-04-23T14:44:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.036+0000 7fa084e00640 1 -- 172.21.15.28:0/2115411970 learned_addr learned my addr 172.21.15.28:0/2115411970 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.037+0000 7fa085800640 1 --2- 172.21.15.28:0/2115411970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56035fc0a730 0x56035fc1cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.037+0000 7fa084e00640 1 --2- 172.21.15.28:0/2115411970 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56035fc0f8a0 0x56035fc23fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.037+0000 7fa085800640 1 -- 172.21.15.28:0/2115411970 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56035fc0f8a0 msgr2=0x56035fc23fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.037+0000 7fa085800640 1 --2- 172.21.15.28:0/2115411970 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56035fc0f8a0 0x56035fc23fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.037+0000 7fa085800640 1 -- 172.21.15.28:0/2115411970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035fc09d40 msgr2=0x56035fc1ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.037+0000 7fa085800640 1 --2- 172.21.15.28:0/2115411970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035fc09d40 0x56035fc1ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.037+0000 7fa085800640 1 -- 172.21.15.28:0/2115411970 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56035f9a9f70 con 0x56035fc0a730 2024-04-23T14:44:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.037+0000 7fa084e00640 1 --2- 172.21.15.28:0/2115411970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035fc09d40 0x56035fc1ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:44:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.037+0000 7fa085800640 1 --2- 172.21.15.28:0/2115411970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56035fc0a730 0x56035fc1cf70 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x56035fc06280 tx=0x7fa074004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec93f2f4181adb56 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.037+0000 7fa067400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:44:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.038+0000 7fa067400640 1 -- 172.21.15.28:0/2115411970 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa074005030 con 0x56035fc0a730 2024-04-23T14:44:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.038+0000 7fa067400640 1 -- 172.21.15.28:0/2115411970 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fa0740051d0 con 0x56035fc0a730 2024-04-23T14:44:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.038+0000 7fa08af99740 1 -- 172.21.15.28:0/2115411970 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa074004c80 con 0x56035fc0a730 2024-04-23T14:44:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.038+0000 7fa067400640 1 -- 172.21.15.28:0/2115411970 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fa074032050 con 0x56035fc0a730 2024-04-23T14:44:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.038+0000 7fa08af99740 10 client.25325 fetch_fsmap learned FSMap version 20 2024-04-23T14:44:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.038+0000 7fa08af99740 1 -- 172.21.15.28:0/2115411970 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x56035f841170 con 0x56035fc0a730 2024-04-23T14:44:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.038+0000 7fa067400640 1 -- 172.21.15.28:0/2115411970 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0740099d0 con 0x56035fc0a730 2024-04-23T14:44:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.038+0000 7fa067400640 1 -- 172.21.15.28:0/2115411970 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fa074003950 con 0x56035fc0a730 2024-04-23T14:44:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.038+0000 7fa08af99740 10 client.25325 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:44:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.038+0000 7fa08af99740 10 client.25325 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:44:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.039+0000 7fa08af99740 20 client.25325 populate_metadata read hostname 'smithi028' 2024-04-23T14:44:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.039+0000 7fa08af99740 1 --2- 172.21.15.28:0/2115411970 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56035fc284e0 0x56035fc2a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.039+0000 7fa08af99740 4 client.25325 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:44:09.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.039+0000 7fa08af99740 1 -- 172.21.15.28:0/2115411970 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa074004c80 con 0x56035fc284e0 2024-04-23T14:44:09.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.039+0000 7fa084e00640 1 --2- 172.21.15.28:0/2115411970 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56035fc284e0 0x56035fc2a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.040+0000 7fa084e00640 1 --2- 172.21.15.28:0/2115411970 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56035fc284e0 0x56035fc2a910 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4737a9cb91fb86f9 server_cookie=583f04bcc1a576c0 in_seq=0 out_seq=0 2024-04-23T14:44:09.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.040+0000 7fa067400640 10 client.25325 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:44:09.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.041+0000 7fa067400640 1 -- 172.21.15.28:0/2115411970 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fa074004c80 con 0x56035fc284e0 2024-04-23T14:44:09.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.041+0000 7fa067400640 10 client.25325 handle_command_reply: tid=0 2024-04-23T14:44:09.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.041+0000 7fa08af99740 1 client.25325 shutdown 2024-04-23T14:44:09.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.041+0000 7fa08af99740 1 -- 172.21.15.28:0/2115411970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56035fc0a730 msgr2=0x56035fc1cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:09.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.041+0000 7fa08af99740 1 --2- 172.21.15.28:0/2115411970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56035fc0a730 0x56035fc1cf70 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x56035fc06280 tx=0x7fa074004270 comp rx=0 tx=0).stop 2024-04-23T14:44:09.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.042+0000 7fa08af99740 1 -- 172.21.15.28:0/2115411970 shutdown_connections 2024-04-23T14:44:09.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.042+0000 7fa08af99740 1 --2- 172.21.15.28:0/2115411970 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56035fc284e0 0x56035fc2a910 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.042+0000 7fa08af99740 1 --2- 172.21.15.28:0/2115411970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56035fc0a730 0x56035fc1cf70 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.042+0000 7fa08af99740 1 --2- 172.21.15.28:0/2115411970 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56035fc0f8a0 0x56035fc23fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.042+0000 7fa08af99740 1 --2- 172.21.15.28:0/2115411970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035fc09d40 0x56035fc1ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.042+0000 7fa08af99740 1 -- 172.21.15.28:0/2115411970 >> 172.21.15.28:0/2115411970 conn(0x56035fc053b0 msgr2=0x56035fc04510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:09.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.042+0000 7fa067400640 0 client.25325 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:44:09.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.042+0000 7fa08af99740 1 -- 172.21.15.28:0/2115411970 shutdown_connections 2024-04-23T14:44:09.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.042+0000 7fa08af99740 1 -- 172.21.15.28:0/2115411970 wait complete. 2024-04-23T14:44:09.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.042+0000 7fa08af99740 20 client.25325 trim_cache size 0 max 16384 2024-04-23T14:44:09.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.065+0000 7fa08af99740 1 Processor -- start 2024-04-23T14:44:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.065+0000 7fa08af99740 1 -- start start 2024-04-23T14:44:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.065+0000 7fa08af99740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56035fc09d40 0x56035fc082c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.065+0000 7fa08af99740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56035fc0a730 0x56035fc23f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.065+0000 7fa08af99740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035fc0f8a0 0x56035fc08800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.066+0000 7fa08af99740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa0801a56e0 con 0x56035fc0f8a0 2024-04-23T14:44:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.066+0000 7fa08af99740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa0801a7f60 con 0x56035fc09d40 2024-04-23T14:44:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.066+0000 7fa08af99740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa0801a5d50 con 0x56035fc0a730 2024-04-23T14:44:09.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.066+0000 7fa084e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56035fc09d40 0x56035fc082c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.066+0000 7fa085800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56035fc0a730 0x56035fc23f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.066+0000 7fa085800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56035fc0a730 0x56035fc23f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:40326/0 (socket says 172.21.15.28:40326) 2024-04-23T14:44:09.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.066+0000 7fa085800640 1 -- 172.21.15.28:0/4118577759 learned_addr learned my addr 172.21.15.28:0/4118577759 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:09.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.066+0000 7fa084e00640 1 --2- 172.21.15.28:0/4118577759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035fc0f8a0 0x56035fc08800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.066+0000 7fa085800640 1 -- 172.21.15.28:0/4118577759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56035fc09d40 msgr2=0x56035fc082c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:09.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.066+0000 7fa085800640 1 --2- 172.21.15.28:0/4118577759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56035fc09d40 0x56035fc082c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.066+0000 7fa085800640 1 -- 172.21.15.28:0/4118577759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035fc0f8a0 msgr2=0x56035fc08800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:09.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.066+0000 7fa085800640 1 --2- 172.21.15.28:0/4118577759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035fc0f8a0 0x56035fc08800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.066+0000 7fa085800640 1 -- 172.21.15.28:0/4118577759 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56035fc12730 con 0x56035fc0a730 2024-04-23T14:44:09.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.067+0000 7fa085800640 1 --2- 172.21.15.28:0/4118577759 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56035fc0a730 0x56035fc23f70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa074003f30 tx=0x7fa074004b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0245259a0a6eff3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:09.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.067+0000 7fa072000640 1 -- 172.21.15.28:0/4118577759 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa074005620 con 0x56035fc0a730 2024-04-23T14:44:09.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.067+0000 7fa072000640 1 -- 172.21.15.28:0/4118577759 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa074003ad0 con 0x56035fc0a730 2024-04-23T14:44:09.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.067+0000 7fa072000640 1 -- 172.21.15.28:0/4118577759 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa07402eb30 con 0x56035fc0a730 2024-04-23T14:44:09.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.068+0000 7fa08af99740 1 -- 172.21.15.28:0/4118577759 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56035fc0a730 msgr2=0x56035fc23f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:09.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.068+0000 7fa08af99740 1 --2- 172.21.15.28:0/4118577759 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56035fc0a730 0x56035fc23f70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa074003f30 tx=0x7fa074004b70 comp rx=0 tx=0).stop 2024-04-23T14:44:09.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.068+0000 7fa08af99740 1 -- 172.21.15.28:0/4118577759 shutdown_connections 2024-04-23T14:44:09.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.068+0000 7fa08af99740 1 --2- 172.21.15.28:0/4118577759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035fc0f8a0 0x56035fc08800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.068+0000 7fa08af99740 1 --2- 172.21.15.28:0/4118577759 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56035fc0a730 0x56035fc23f70 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.068+0000 7fa08af99740 1 --2- 172.21.15.28:0/4118577759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56035fc09d40 0x56035fc082c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.068+0000 7fa08af99740 1 -- 172.21.15.28:0/4118577759 >> 172.21.15.28:0/4118577759 conn(0x56035fc053b0 msgr2=0x56035fc0bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:09.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.068+0000 7fa08af99740 1 -- 172.21.15.28:0/4118577759 shutdown_connections 2024-04-23T14:44:09.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.068+0000 7fa08af99740 1 -- 172.21.15.28:0/4118577759 wait complete. 2024-04-23T14:44:09.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.068+0000 7fa08af99740 1 -- start start 2024-04-23T14:44:09.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.069+0000 7fa08af99740 1 Processor -- start 2024-04-23T14:44:09.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.069+0000 7fa08af99740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56035fc09d40 0x56035fc21a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.069+0000 7fa08af99740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56035fc0a730 0x56035fc21f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.069+0000 7fa08af99740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035fc0f8a0 0x56035fc22f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.069+0000 7fa08af99740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa0801a6420 con 0x56035fc0f8a0 2024-04-23T14:44:09.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.069+0000 7fa08af99740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa0801a7890 con 0x56035fc0a730 2024-04-23T14:44:09.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.069+0000 7fa08af99740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa0801a6af0 con 0x56035fc09d40 2024-04-23T14:44:09.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.069+0000 7fa08af99740 10 client.0 _ll_register_callbacks cb 0x7fa074009c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:44:09.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.070+0000 7fa084e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56035fc09d40 0x56035fc21a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.070+0000 7fa085800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56035fc0a730 0x56035fc21f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.070+0000 7fa085800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56035fc0a730 0x56035fc21f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:42682/0 (socket says 172.21.15.28:42682) 2024-04-23T14:44:09.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.070+0000 7fa085800640 1 -- 172.21.15.28:0/2760065503 learned_addr learned my addr 172.21.15.28:0/2760065503 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:09.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.070+0000 7fa084e00640 1 --2- 172.21.15.28:0/2760065503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035fc0f8a0 0x56035fc22f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.070+0000 7fa085800640 1 -- 172.21.15.28:0/2760065503 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56035fc09d40 msgr2=0x56035fc21a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:09.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.070+0000 7fa085800640 1 --2- 172.21.15.28:0/2760065503 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56035fc09d40 0x56035fc21a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.070+0000 7fa085800640 1 -- 172.21.15.28:0/2760065503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035fc0f8a0 msgr2=0x56035fc22f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:09.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.070+0000 7fa085800640 1 --2- 172.21.15.28:0/2760065503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035fc0f8a0 0x56035fc22f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.070+0000 7fa085800640 1 -- 172.21.15.28:0/2760065503 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56035fc23540 con 0x56035fc0a730 2024-04-23T14:44:09.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.070+0000 7fa084e00640 1 --2- 172.21.15.28:0/2760065503 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56035fc09d40 0x56035fc21a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:44:09.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.070+0000 7fa085800640 1 --2- 172.21.15.28:0/2760065503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56035fc0a730 0x56035fc21f90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa074003fd0 tx=0x7fa07402e6b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5f0b794c58917e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:09.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.070+0000 7fa067400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:44:09.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.071+0000 7fa067400640 1 -- 172.21.15.28:0/2760065503 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0740035f0 con 0x56035fc0a730 2024-04-23T14:44:09.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.071+0000 7fa067400640 1 -- 172.21.15.28:0/2760065503 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fa074003790 con 0x56035fc0a730 2024-04-23T14:44:09.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.071+0000 7fa08af99740 1 -- 172.21.15.28:0/2760065503 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa074003ad0 con 0x56035fc0a730 2024-04-23T14:44:09.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.071+0000 7fa067400640 1 -- 172.21.15.28:0/2760065503 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fa07402e800 con 0x56035fc0a730 2024-04-23T14:44:09.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.071+0000 7fa08af99740 10 client.25331 fetch_fsmap learned FSMap version 20 2024-04-23T14:44:09.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.071+0000 7fa08af99740 1 -- 172.21.15.28:0/2760065503 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x56035fc8ee60 con 0x56035fc0a730 2024-04-23T14:44:09.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.071+0000 7fa067400640 1 -- 172.21.15.28:0/2760065503 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa074003ad0 con 0x56035fc0a730 2024-04-23T14:44:09.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.072+0000 7fa067400640 1 -- 172.21.15.28:0/2760065503 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fa074023460 con 0x56035fc0a730 2024-04-23T14:44:09.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.072+0000 7fa08af99740 10 client.25331 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:44:09.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.072+0000 7fa08af99740 10 client.25331 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:44:09.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.072+0000 7fa08af99740 20 client.25331 populate_metadata read hostname 'smithi028' 2024-04-23T14:44:09.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.072+0000 7fa08af99740 1 --2- 172.21.15.28:0/2760065503 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56035fc8f7b0 0x56035fc91bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.072+0000 7fa08af99740 4 client.25331 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-23T14:44:09.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.072+0000 7fa08af99740 1 -- 172.21.15.28:0/2760065503 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56035fb7c890 con 0x56035fc8f7b0 2024-04-23T14:44:09.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.072+0000 7fa084e00640 1 --2- 172.21.15.28:0/2760065503 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56035fc8f7b0 0x56035fc91bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.073+0000 7fa084e00640 1 --2- 172.21.15.28:0/2760065503 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56035fc8f7b0 0x56035fc91bd0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc5f089a94e29c38 server_cookie=6bd23eb6c0df3b18 in_seq=0 out_seq=0 2024-04-23T14:44:09.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.073+0000 7fa067400640 10 client.25331 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:44:09.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.074+0000 7fa067400640 1 -- 172.21.15.28:0/2760065503 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56035fb7c890 con 0x56035fc8f7b0 2024-04-23T14:44:09.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.074+0000 7fa067400640 10 client.25331 handle_command_reply: tid=0 2024-04-23T14:44:09.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.074+0000 7fa08af99740 1 client.25331 shutdown 2024-04-23T14:44:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.074+0000 7fa08af99740 1 -- 172.21.15.28:0/2760065503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56035fc0a730 msgr2=0x56035fc21f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.074+0000 7fa08af99740 1 --2- 172.21.15.28:0/2760065503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56035fc0a730 0x56035fc21f90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa074003fd0 tx=0x7fa07402e6b0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.074+0000 7fa08af99740 1 -- 172.21.15.28:0/2760065503 shutdown_connections 2024-04-23T14:44:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.074+0000 7fa08af99740 1 --2- 172.21.15.28:0/2760065503 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56035fc8f7b0 0x56035fc91bd0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.074+0000 7fa08af99740 1 --2- 172.21.15.28:0/2760065503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56035fc0a730 0x56035fc21f90 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.074+0000 7fa08af99740 1 --2- 172.21.15.28:0/2760065503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56035fc0f8a0 0x56035fc22f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.074+0000 7fa08af99740 1 --2- 172.21.15.28:0/2760065503 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56035fc09d40 0x56035fc21a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.074+0000 7fa08af99740 1 -- 172.21.15.28:0/2760065503 >> 172.21.15.28:0/2760065503 conn(0x56035fc053b0 msgr2=0x56035fc082c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.074+0000 7fa067400640 0 client.25331 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:44:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.075+0000 7fa08af99740 1 -- 172.21.15.28:0/2760065503 shutdown_connections 2024-04-23T14:44:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.075+0000 7fa08af99740 1 -- 172.21.15.28:0/2760065503 wait complete. 2024-04-23T14:44:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.075+0000 7fa08af99740 20 client.25331 trim_cache size 0 max 16384 2024-04-23T14:44:09.078 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:44:09.078 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-23T14:44:09.078 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "8664f60c-bb6f-4943-8d5c-d35b29eeeda6", 2024-04-23T14:44:09.078 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-23T14:44:09.079 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:44:09.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.075+0000 7fa088200640 1 -- 172.21.15.28:0/1745305913 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa0680a3db0 msgr2=0x7fa0680a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:09.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.075+0000 7fa088200640 1 --2- 172.21.15.28:0/1745305913 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa0680a3db0 0x7fa0680a6270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa0780ce620 tx=0x7fa07800ef50 comp rx=0 tx=0).stop 2024-04-23T14:44:09.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.075+0000 7fa088200640 1 -- 172.21.15.28:0/1745305913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa080109f10 msgr2=0x7fa0801a1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:09.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.075+0000 7fa088200640 1 --2- 172.21.15.28:0/1745305913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa080109f10 0x7fa0801a1b80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa07800ec50 tx=0x7fa07800c6a0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.076+0000 7fa088200640 1 -- 172.21.15.28:0/1745305913 shutdown_connections 2024-04-23T14:44:09.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.076+0000 7fa088200640 1 --2- 172.21.15.28:0/1745305913 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa0680a3db0 0x7fa0680a6270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.076+0000 7fa088200640 1 --2- 172.21.15.28:0/1745305913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa080109f10 0x7fa0801a1b80 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.076+0000 7fa088200640 1 --2- 172.21.15.28:0/1745305913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa080076040 0x7fa08019ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.076+0000 7fa088200640 1 --2- 172.21.15.28:0/1745305913 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa080074a60 0x7fa08019a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.076+0000 7fa088200640 1 -- 172.21.15.28:0/1745305913 >> 172.21.15.28:0/1745305913 conn(0x7fa0800fd5e0 msgr2=0x7fa080105e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:09.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.076+0000 7fa088200640 1 -- 172.21.15.28:0/1745305913 shutdown_connections 2024-04-23T14:44:09.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.076+0000 7fa088200640 1 -- 172.21.15.28:0/1745305913 wait complete. 2024-04-23T14:44:09.093 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:08 smithi028 ceph-mon[27810]: Adjusting osd_memory_target on smithi028 to 177.5M 2024-04-23T14:44:09.093 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:08 smithi028 ceph-mon[27810]: Unable to set osd_memory_target on smithi028 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-23T14:44:09.093 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:08 smithi028 ceph-mon[27810]: Updating smithi028:/etc/ceph/ceph.conf 2024-04-23T14:44:09.093 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:08 smithi028 ceph-mon[27810]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-23T14:44:09.093 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:08 smithi028 ceph-mon[27810]: Updating smithi105:/etc/ceph/ceph.conf 2024-04-23T14:44:09.093 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:08 smithi028 ceph-mon[27810]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 0 B/s wr, 26 op/s 2024-04-23T14:44:09.093 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:08 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/250165951' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T14:44:09.094 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:08 smithi028 ceph-mon[27810]: Updating smithi038:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T14:44:09.094 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:08 smithi028 ceph-mon[27810]: Updating smithi105:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T14:44:09.094 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:08 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/4225550200' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:44:09.096 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:44:09.096 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:44:09.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:08 smithi038 ceph-mon[32348]: Adjusting osd_memory_target on smithi028 to 177.5M 2024-04-23T14:44:09.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:08 smithi038 ceph-mon[32348]: Unable to set osd_memory_target on smithi028 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-23T14:44:09.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:08 smithi038 ceph-mon[32348]: Updating smithi028:/etc/ceph/ceph.conf 2024-04-23T14:44:09.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:08 smithi038 ceph-mon[32348]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-23T14:44:09.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:08 smithi038 ceph-mon[32348]: Updating smithi105:/etc/ceph/ceph.conf 2024-04-23T14:44:09.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:08 smithi038 ceph-mon[32348]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 0 B/s wr, 26 op/s 2024-04-23T14:44:09.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:08 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/250165951' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T14:44:09.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:08 smithi038 ceph-mon[32348]: Updating smithi038:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T14:44:09.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:08 smithi038 ceph-mon[32348]: Updating smithi105:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T14:44:09.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:08 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/4225550200' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:44:09.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.280+0000 7f7c77e00640 1 -- 172.21.15.28:0/3543873454 --> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f7c78062440 con 0x7f7c5c0a3e20 2024-04-23T14:44:09.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.283+0000 7fbd02600640 1 -- 172.21.15.28:0/4212211422 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbcfc076e50 msgr2=0x7fbcfc1129a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:09.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.283+0000 7fbd02600640 1 --2- 172.21.15.28:0/4212211422 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbcfc076e50 0x7fbcfc1129a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbce4009890 tx=0x7fbce402a380 comp rx=0 tx=0).stop 2024-04-23T14:44:09.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.283+0000 7fbd02600640 1 -- 172.21.15.28:0/4212211422 shutdown_connections 2024-04-23T14:44:09.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.283+0000 7fbd02600640 1 --2- 172.21.15.28:0/4212211422 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbcfc076e50 0x7fbcfc1129a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.283+0000 7fbd02600640 1 --2- 172.21.15.28:0/4212211422 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbcfc112ee0 0x7fbcfc1152d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.283+0000 7fbd02600640 1 --2- 172.21.15.28:0/4212211422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcfc076530 0x7fbcfc076910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.283+0000 7fbd02600640 1 -- 172.21.15.28:0/4212211422 >> 172.21.15.28:0/4212211422 conn(0x7fbcfc0ffd90 msgr2=0x7fbcfc1021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:09.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.284+0000 7fbd02600640 1 -- 172.21.15.28:0/4212211422 shutdown_connections 2024-04-23T14:44:09.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.284+0000 7fbd02600640 1 -- 172.21.15.28:0/4212211422 wait complete. 2024-04-23T14:44:09.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.285+0000 7fbd02600640 1 Processor -- start 2024-04-23T14:44:09.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.285+0000 7fbd02600640 1 -- start start 2024-04-23T14:44:09.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.285+0000 7fbd02600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcfc076530 0x7fbcfc19f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.285+0000 7fbd02600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbcfc076e50 0x7fbcfc19f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.286+0000 7fbd02600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbcfc112ee0 0x7fbcfc1a38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.286+0000 7fbd02600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbcfc118a40 con 0x7fbcfc076530 2024-04-23T14:44:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.286+0000 7fbd02600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbcfc1188c0 con 0x7fbcfc112ee0 2024-04-23T14:44:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.286+0000 7fbd02600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fbcfc118bc0 con 0x7fbcfc076e50 2024-04-23T14:44:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.286+0000 7fbcfb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcfc076530 0x7fbcfc19f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.286+0000 7fbcfb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcfc076530 0x7fbcfc19f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43026/0 (socket says 172.21.15.28:43026) 2024-04-23T14:44:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.286+0000 7fbcfb400640 1 -- 172.21.15.28:0/1399059373 learned_addr learned my addr 172.21.15.28:0/1399059373 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.286+0000 7fbcfbe00640 1 --2- 172.21.15.28:0/1399059373 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbcfc112ee0 0x7fbcfc1a38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.286+0000 7fbcf2a00640 1 --2- 172.21.15.28:0/1399059373 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbcfc076e50 0x7fbcfc19f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.286+0000 7fbcfb400640 1 -- 172.21.15.28:0/1399059373 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbcfc076e50 msgr2=0x7fbcfc19f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.286+0000 7fbcfb400640 1 --2- 172.21.15.28:0/1399059373 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbcfc076e50 0x7fbcfc19f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.286+0000 7fbcfb400640 1 -- 172.21.15.28:0/1399059373 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbcfc112ee0 msgr2=0x7fbcfc1a38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.286+0000 7fbcfb400640 1 --2- 172.21.15.28:0/1399059373 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbcfc112ee0 0x7fbcfc1a38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.286+0000 7fbcfb400640 1 -- 172.21.15.28:0/1399059373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcfc1a4000 con 0x7fbcfc076530 2024-04-23T14:44:09.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.287+0000 7fbcfb400640 1 --2- 172.21.15.28:0/1399059373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcfc076530 0x7fbcfc19f060 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbcec00efb0 tx=0x7fbcec00c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:09.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.287+0000 7fbcf8c00640 1 -- 172.21.15.28:0/1399059373 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcec01b070 con 0x7fbcfc076530 2024-04-23T14:44:09.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.287+0000 7fbcf8c00640 1 -- 172.21.15.28:0/1399059373 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcec00ba50 con 0x7fbcfc076530 2024-04-23T14:44:09.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.287+0000 7fbd02600640 1 -- 172.21.15.28:0/1399059373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcfc1a42f0 con 0x7fbcfc076530 2024-04-23T14:44:09.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.287+0000 7fbcf8c00640 1 -- 172.21.15.28:0/1399059373 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcec016b50 con 0x7fbcfc076530 2024-04-23T14:44:09.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.287+0000 7fbd02600640 1 -- 172.21.15.28:0/1399059373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcfc1abb80 con 0x7fbcfc076530 2024-04-23T14:44:09.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.291+0000 7fbd05578740 1 Processor -- start 2024-04-23T14:44:09.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.292+0000 7fbd05578740 1 -- start start 2024-04-23T14:44:09.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.292+0000 7fbcf8c00640 1 -- 172.21.15.28:0/1399059373 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7fbcec01e070 con 0x7fbcfc076530 2024-04-23T14:44:09.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.292+0000 7fbd05578740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562cb18be860 0x562cb18bec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.292+0000 7fbd05578740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562cb18bf180 0x562cb18c9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.292+0000 7fbd05578740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562cb18c9850 0x562cb18cbc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.292+0000 7fbd05578740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562cb18241f0 con 0x562cb18c9850 2024-04-23T14:44:09.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.292+0000 7fbd05578740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562cb17ca350 con 0x562cb18be860 2024-04-23T14:44:09.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.292+0000 7fbd05578740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562cb18201b0 con 0x562cb18bf180 2024-04-23T14:44:09.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.292+0000 7fbcfb400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562cb18bf180 0x562cb18c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.292+0000 7fbcfbe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562cb18be860 0x562cb18bec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.292+0000 7fbcfbe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562cb18be860 0x562cb18bec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:42716/0 (socket says 172.21.15.28:42716) 2024-04-23T14:44:09.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.292+0000 7fbcfbe00640 1 -- 172.21.15.28:0/832395274 learned_addr learned my addr 172.21.15.28:0/832395274 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:09.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.293+0000 7fbcf2a00640 1 --2- 172.21.15.28:0/832395274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562cb18c9850 0x562cb18cbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.293+0000 7fbcfb400640 1 -- 172.21.15.28:0/832395274 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562cb18be860 msgr2=0x562cb18bec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:09.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.293+0000 7fbcfb400640 1 --2- 172.21.15.28:0/832395274 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562cb18be860 0x562cb18bec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.293+0000 7fbcfb400640 1 -- 172.21.15.28:0/832395274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562cb18c9850 msgr2=0x562cb18cbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:09.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.293+0000 7fbcfb400640 1 --2- 172.21.15.28:0/832395274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562cb18c9850 0x562cb18cbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.293+0000 7fbcfb400640 1 -- 172.21.15.28:0/832395274 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562cb181ef50 con 0x562cb18bf180 2024-04-23T14:44:09.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.293+0000 7fbcfbe00640 1 --2- 172.21.15.28:0/832395274 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562cb18be860 0x562cb18bec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:44:09.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.293+0000 7fbcfb400640 1 --2- 172.21.15.28:0/832395274 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562cb18bf180 0x562cb18c9310 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbcec0cded0 tx=0x7fbcec0d6e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6892bfaf52bf693 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.294+0000 7fbcf0c00640 1 -- 172.21.15.28:0/832395274 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcec0df070 con 0x562cb18bf180 2024-04-23T14:44:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.294+0000 7fbcf0c00640 1 -- 172.21.15.28:0/832395274 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcec0d8d20 con 0x562cb18bf180 2024-04-23T14:44:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.294+0000 7fbcf0c00640 1 -- 172.21.15.28:0/832395274 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcec0db7e0 con 0x562cb18bf180 2024-04-23T14:44:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.294+0000 7fbcf8c00640 1 --2- 172.21.15.28:0/1399059373 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fbce00a3e30 0x7fbce00a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.294+0000 7fbcf8c00640 1 -- 172.21.15.28:0/1399059373 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fbcec0d0890 con 0x7fbcfc076530 2024-04-23T14:44:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.294+0000 7fbcfbe00640 1 --2- 172.21.15.28:0/1399059373 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fbce00a3e30 0x7fbce00a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.295+0000 7fbd05578740 1 -- 172.21.15.28:0/832395274 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562cb18bf180 msgr2=0x562cb18c9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.295+0000 7fbd05578740 1 --2- 172.21.15.28:0/832395274 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562cb18bf180 0x562cb18c9310 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbcec0cded0 tx=0x7fbcec0d6e40 comp rx=0 tx=0).stop 2024-04-23T14:44:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.295+0000 7fbd05578740 1 -- 172.21.15.28:0/832395274 shutdown_connections 2024-04-23T14:44:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.295+0000 7fbd05578740 1 --2- 172.21.15.28:0/832395274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562cb18c9850 0x562cb18cbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.295+0000 7fbd05578740 1 --2- 172.21.15.28:0/832395274 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562cb18bf180 0x562cb18c9310 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.295+0000 7fbd05578740 1 --2- 172.21.15.28:0/832395274 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562cb18be860 0x562cb18bec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.295+0000 7fbd05578740 1 -- 172.21.15.28:0/832395274 >> 172.21.15.28:0/832395274 conn(0x562cb18bab80 msgr2=0x562cb18baf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:09.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.295+0000 7fbd05578740 1 -- 172.21.15.28:0/832395274 shutdown_connections 2024-04-23T14:44:09.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.295+0000 7fbcfbe00640 1 --2- 172.21.15.28:0/1399059373 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fbce00a3e30 0x7fbce00a62f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4009fc0 tx=0x7fbcf4005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:09.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.295+0000 7fbd05578740 1 -- 172.21.15.28:0/832395274 wait complete. 2024-04-23T14:44:09.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.296+0000 7fbd05578740 1 -- start start 2024-04-23T14:44:09.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.296+0000 7fbd05578740 1 Processor -- start 2024-04-23T14:44:09.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.297+0000 7fbd05578740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562cb18be860 0x562cb18c6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.297+0000 7fbd05578740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562cb18bf180 0x562cb18c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.297+0000 7fbd05578740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562cb18c9850 0x562cb18c72b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.297+0000 7fbd05578740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562cb18cc640 con 0x562cb18be860 2024-04-23T14:44:09.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.297+0000 7fbd05578740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562cb17dd690 con 0x562cb18c9850 2024-04-23T14:44:09.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.297+0000 7fbd05578740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562cb18cc7c0 con 0x562cb18bf180 2024-04-23T14:44:09.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.297+0000 7fbd05578740 10 client.0 _ll_register_callbacks cb 0x562cb1819650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:44:09.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.297+0000 7fbcf2a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562cb18bf180 0x562cb18c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.297+0000 7fbcf2a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562cb18bf180 0x562cb18c6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:40358/0 (socket says 172.21.15.28:40358) 2024-04-23T14:44:09.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.297+0000 7fbcf2a00640 1 -- 172.21.15.28:0/4160914933 learned_addr learned my addr 172.21.15.28:0/4160914933 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:09.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.297+0000 7fbcfbe00640 1 --2- 172.21.15.28:0/4160914933 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562cb18c9850 0x562cb18c72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.297+0000 7fbcfb400640 1 --2- 172.21.15.28:0/4160914933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562cb18be860 0x562cb18c6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.298+0000 7fbcf2a00640 1 -- 172.21.15.28:0/4160914933 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562cb18c9850 msgr2=0x562cb18c72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:09.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.298+0000 7fbcf2a00640 1 --2- 172.21.15.28:0/4160914933 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562cb18c9850 0x562cb18c72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.298+0000 7fbcf2a00640 1 -- 172.21.15.28:0/4160914933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562cb18be860 msgr2=0x562cb18c6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:09.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.298+0000 7fbcf2a00640 1 --2- 172.21.15.28:0/4160914933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562cb18be860 0x562cb18c6830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.298+0000 7fbcf2a00640 1 -- 172.21.15.28:0/4160914933 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562cb17dc920 con 0x562cb18bf180 2024-04-23T14:44:09.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.298+0000 7fbcf2a00640 1 --2- 172.21.15.28:0/4160914933 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562cb18bf180 0x562cb18c6d70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fbce4034000 tx=0x7fbce4009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2dc9d741057f2e0e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:09.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.298+0000 7fbce9600640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:44:09.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.299+0000 7fbce9600640 1 -- 172.21.15.28:0/4160914933 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce4037040 con 0x562cb18bf180 2024-04-23T14:44:09.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.299+0000 7fbce9600640 1 -- 172.21.15.28:0/4160914933 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fbce4004460 con 0x562cb18bf180 2024-04-23T14:44:09.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.299+0000 7fbd05578740 1 -- 172.21.15.28:0/4160914933 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbcec0d8d20 con 0x562cb18bf180 2024-04-23T14:44:09.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.299+0000 7fbce9600640 1 -- 172.21.15.28:0/4160914933 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fbce404b050 con 0x562cb18bf180 2024-04-23T14:44:09.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.299+0000 7fbd05578740 10 client.25348 fetch_fsmap learned FSMap version 20 2024-04-23T14:44:09.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.299+0000 7fbd05578740 1 -- 172.21.15.28:0/4160914933 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x562cb16007d0 con 0x562cb18bf180 2024-04-23T14:44:09.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.299+0000 7fbce9600640 1 -- 172.21.15.28:0/4160914933 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcec0d8d20 con 0x562cb18bf180 2024-04-23T14:44:09.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.300+0000 7fbce9600640 1 -- 172.21.15.28:0/4160914933 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fbce403d450 con 0x562cb18bf180 2024-04-23T14:44:09.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.300+0000 7fbd05578740 10 client.25348 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:44:09.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.300+0000 7fbd05578740 10 client.25348 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:44:09.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.300+0000 7fbd05578740 20 client.25348 populate_metadata read hostname 'smithi028' 2024-04-23T14:44:09.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.300+0000 7fbd05578740 1 --2- 172.21.15.28:0/4160914933 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562cb18d9c20 0x562cb18dc070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.300+0000 7fbd05578740 4 client.25348 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:44:09.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.300+0000 7fbd05578740 1 -- 172.21.15.28:0/4160914933 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562cb1831c10 con 0x562cb18d9c20 2024-04-23T14:44:09.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.301+0000 7fbcfb400640 1 --2- 172.21.15.28:0/4160914933 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562cb18d9c20 0x562cb18dc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.301+0000 7fbcfb400640 1 --2- 172.21.15.28:0/4160914933 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562cb18d9c20 0x562cb18dc070 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76a102d1aefe08a2 server_cookie=ef12de0443140085 in_seq=0 out_seq=0 2024-04-23T14:44:09.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.301+0000 7fbce9600640 10 client.25348 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:44:09.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.302+0000 7fbce9600640 1 -- 172.21.15.28:0/4160914933 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562cb1831c10 con 0x562cb18d9c20 2024-04-23T14:44:09.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.302+0000 7fbce9600640 10 client.25348 handle_command_reply: tid=0 2024-04-23T14:44:09.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.302+0000 7fbd05578740 1 client.25348 shutdown 2024-04-23T14:44:09.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.302+0000 7fbd05578740 1 -- 172.21.15.28:0/4160914933 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562cb18bf180 msgr2=0x562cb18c6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:09.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.302+0000 7fbd05578740 1 --2- 172.21.15.28:0/4160914933 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562cb18bf180 0x562cb18c6d70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fbce4034000 tx=0x7fbce4009500 comp rx=0 tx=0).stop 2024-04-23T14:44:09.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.303+0000 7fbd05578740 1 -- 172.21.15.28:0/4160914933 shutdown_connections 2024-04-23T14:44:09.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.303+0000 7fbd05578740 1 --2- 172.21.15.28:0/4160914933 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562cb18d9c20 0x562cb18dc070 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.303+0000 7fbd05578740 1 --2- 172.21.15.28:0/4160914933 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562cb18c9850 0x562cb18c72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.303+0000 7fbd05578740 1 --2- 172.21.15.28:0/4160914933 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562cb18bf180 0x562cb18c6d70 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.303+0000 7fbd05578740 1 --2- 172.21.15.28:0/4160914933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562cb18be860 0x562cb18c6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.303+0000 7fbce9600640 0 client.25348 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:44:09.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.303+0000 7fbd05578740 1 -- 172.21.15.28:0/4160914933 >> 172.21.15.28:0/4160914933 conn(0x562cb18bab80 msgr2=0x562cb18c0980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:09.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.303+0000 7fbd05578740 1 -- 172.21.15.28:0/4160914933 shutdown_connections 2024-04-23T14:44:09.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.303+0000 7fbd05578740 1 -- 172.21.15.28:0/4160914933 wait complete. 2024-04-23T14:44:09.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.304+0000 7fbd05578740 20 client.25348 trim_cache size 0 max 16384 2024-04-23T14:44:09.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.333+0000 7fbd05578740 1 Processor -- start 2024-04-23T14:44:09.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.333+0000 7fbd05578740 1 -- start start 2024-04-23T14:44:09.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.333+0000 7fbd05578740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562cb18bab80 0x562cb18d5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.333+0000 7fbd05578740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562cb18be860 0x562cb18c1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.333+0000 7fbd05578740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562cb18bf180 0x562cb18e1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.333+0000 7fbd05578740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbcfc1b2370 con 0x562cb18be860 2024-04-23T14:44:09.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.333+0000 7fbd05578740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbcfc1b4bf0 con 0x562cb18bf180 2024-04-23T14:44:09.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.333+0000 7fbd05578740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fbcfc1b29e0 con 0x562cb18bab80 2024-04-23T14:44:09.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.334+0000 7fbcfbe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562cb18bf180 0x562cb18e1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.334+0000 7fbcfb400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562cb18bab80 0x562cb18d5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.334+0000 7fbcfb400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562cb18bab80 0x562cb18d5500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:40364/0 (socket says 172.21.15.28:40364) 2024-04-23T14:44:09.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.334+0000 7fbcfbe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562cb18bf180 0x562cb18e1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:42734/0 (socket says 172.21.15.28:42734) 2024-04-23T14:44:09.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.334+0000 7fbcfb400640 1 -- 172.21.15.28:0/518639336 learned_addr learned my addr 172.21.15.28:0/518639336 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:09.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.334+0000 7fbcf2a00640 1 --2- 172.21.15.28:0/518639336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562cb18be860 0x562cb18c1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.334+0000 7fbcfb400640 1 -- 172.21.15.28:0/518639336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562cb18bf180 msgr2=0x562cb18e1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:09.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.334+0000 7fbcfb400640 1 --2- 172.21.15.28:0/518639336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562cb18bf180 0x562cb18e1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.334+0000 7fbcfb400640 1 -- 172.21.15.28:0/518639336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562cb18be860 msgr2=0x562cb18c1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:09.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.334+0000 7fbcfb400640 1 --2- 172.21.15.28:0/518639336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562cb18be860 0x562cb18c1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.334+0000 7fbcfb400640 1 -- 172.21.15.28:0/518639336 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562cb15363f0 con 0x562cb18bab80 2024-04-23T14:44:09.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.334+0000 7fbcfb400640 1 --2- 172.21.15.28:0/518639336 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562cb18bab80 0x562cb18d5500 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fbcec09c6b0 tx=0x7fbcec00bf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7327bdef1faff273 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:09.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.335+0000 7fbcebe00640 1 -- 172.21.15.28:0/518639336 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcec0df070 con 0x562cb18bab80 2024-04-23T14:44:09.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.335+0000 7fbcebe00640 1 -- 172.21.15.28:0/518639336 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcec09c720 con 0x562cb18bab80 2024-04-23T14:44:09.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.335+0000 7fbcebe00640 1 -- 172.21.15.28:0/518639336 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcec0a52e0 con 0x562cb18bab80 2024-04-23T14:44:09.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.336+0000 7fbd05578740 1 -- 172.21.15.28:0/518639336 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562cb18bab80 msgr2=0x562cb18d5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:09.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.336+0000 7fbd05578740 1 --2- 172.21.15.28:0/518639336 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562cb18bab80 0x562cb18d5500 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fbcec09c6b0 tx=0x7fbcec00bf90 comp rx=0 tx=0).stop 2024-04-23T14:44:09.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.336+0000 7fbd05578740 1 -- 172.21.15.28:0/518639336 shutdown_connections 2024-04-23T14:44:09.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.336+0000 7fbd05578740 1 --2- 172.21.15.28:0/518639336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562cb18bf180 0x562cb18e1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.336+0000 7fbd05578740 1 --2- 172.21.15.28:0/518639336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562cb18be860 0x562cb18c1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.336+0000 7fbd05578740 1 --2- 172.21.15.28:0/518639336 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562cb18bab80 0x562cb18d5500 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.336+0000 7fbd05578740 1 -- 172.21.15.28:0/518639336 >> 172.21.15.28:0/518639336 conn(0x562cb18c5470 msgr2=0x562cb18c7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:09.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.336+0000 7fbd05578740 1 -- 172.21.15.28:0/518639336 shutdown_connections 2024-04-23T14:44:09.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.336+0000 7fbd05578740 1 -- 172.21.15.28:0/518639336 wait complete. 2024-04-23T14:44:09.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.337+0000 7fbd05578740 1 -- start start 2024-04-23T14:44:09.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.337+0000 7fbd05578740 1 Processor -- start 2024-04-23T14:44:09.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.338+0000 7fbd05578740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562cb18bab80 0x562cb18cd650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.338+0000 7fbd05578740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562cb18be860 0x562cb18cdb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.338+0000 7fbd05578740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562cb18bf180 0x562cb18ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.338+0000 7fbd05578740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbcfc1b30b0 con 0x562cb18bf180 2024-04-23T14:44:09.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.338+0000 7fbd05578740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbcfc1b4520 con 0x562cb18bab80 2024-04-23T14:44:09.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.338+0000 7fbd05578740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fbcfc1b3780 con 0x562cb18be860 2024-04-23T14:44:09.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.338+0000 7fbd05578740 10 client.0 _ll_register_callbacks cb 0x7fbcec009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:44:09.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.338+0000 7fbcf2a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562cb18be860 0x562cb18cdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.338+0000 7fbcf2a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562cb18be860 0x562cb18cdb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:40380/0 (socket says 172.21.15.28:40380) 2024-04-23T14:44:09.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.338+0000 7fbcf2a00640 1 -- 172.21.15.28:0/3920645390 learned_addr learned my addr 172.21.15.28:0/3920645390 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:09.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.338+0000 7fbcfb400640 1 --2- 172.21.15.28:0/3920645390 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562cb18bab80 0x562cb18cd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.338+0000 7fbcf2a00640 1 -- 172.21.15.28:0/3920645390 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562cb18bab80 msgr2=0x562cb18cd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:09.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.338+0000 7fbcf2a00640 1 --2- 172.21.15.28:0/3920645390 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562cb18bab80 0x562cb18cd650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.338+0000 7fbcf2a00640 1 -- 172.21.15.28:0/3920645390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562cb18bf180 msgr2=0x562cb18ce0d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T14:44:09.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.339+0000 7fbcf2a00640 1 --2- 172.21.15.28:0/3920645390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562cb18bf180 0x562cb18ce0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.339+0000 7fbcf2a00640 1 -- 172.21.15.28:0/3920645390 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562cb14f60f0 con 0x562cb18be860 2024-04-23T14:44:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.339+0000 7fbcf2a00640 1 --2- 172.21.15.28:0/3920645390 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562cb18be860 0x562cb18cdb90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbce403d870 tx=0x7fbce403d660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9190ef833e9f654 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.339+0000 7fbce9600640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:44:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.339+0000 7fbce9600640 1 -- 172.21.15.28:0/3920645390 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce4037040 con 0x562cb18be860 2024-04-23T14:44:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.339+0000 7fbce9600640 1 -- 172.21.15.28:0/3920645390 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fbce4004460 con 0x562cb18be860 2024-04-23T14:44:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.339+0000 7fbd05578740 1 -- 172.21.15.28:0/3920645390 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbcec09c720 con 0x562cb18be860 2024-04-23T14:44:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.340+0000 7fbce9600640 1 -- 172.21.15.28:0/3920645390 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fbce405d050 con 0x562cb18be860 2024-04-23T14:44:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.340+0000 7fbd05578740 10 client.25354 fetch_fsmap learned FSMap version 20 2024-04-23T14:44:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.340+0000 7fbd05578740 1 -- 172.21.15.28:0/3920645390 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x562cb17e9810 con 0x562cb18be860 2024-04-23T14:44:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.340+0000 7fbce9600640 1 -- 172.21.15.28:0/3920645390 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcec09c720 con 0x562cb18be860 2024-04-23T14:44:09.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.340+0000 7fbce9600640 1 -- 172.21.15.28:0/3920645390 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fbce40036d0 con 0x562cb18be860 2024-04-23T14:44:09.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.340+0000 7fbd05578740 10 client.25354 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:44:09.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.340+0000 7fbd05578740 10 client.25354 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:44:09.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.340+0000 7fbd05578740 20 client.25354 populate_metadata read hostname 'smithi028' 2024-04-23T14:44:09.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.340+0000 7fbd05578740 1 --2- 172.21.15.28:0/3920645390 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562cb18c9850 0x562cb1942a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.340+0000 7fbd05578740 4 client.25354 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:44:09.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.340+0000 7fbd05578740 1 -- 172.21.15.28:0/3920645390 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562cb182ba80 con 0x562cb18c9850 2024-04-23T14:44:09.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.341+0000 7fbcfb400640 1 --2- 172.21.15.28:0/3920645390 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562cb18c9850 0x562cb1942a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.341+0000 7fbcfb400640 1 --2- 172.21.15.28:0/3920645390 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562cb18c9850 0x562cb1942a80 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85abe7e51b5d8bda server_cookie=d19c4e5f934ced9a in_seq=0 out_seq=0 2024-04-23T14:44:09.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.341+0000 7fbce9600640 10 client.25354 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:44:09.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.341+0000 7fbce9600640 1 -- 172.21.15.28:0/3920645390 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x562cb182ba80 con 0x562cb18c9850 2024-04-23T14:44:09.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.341+0000 7fbce9600640 10 client.25354 handle_command_reply: tid=0 2024-04-23T14:44:09.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.341+0000 7fbd05578740 1 client.25354 shutdown 2024-04-23T14:44:09.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.342+0000 7fbd05578740 1 -- 172.21.15.28:0/3920645390 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562cb18be860 msgr2=0x562cb18cdb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:09.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.342+0000 7fbd05578740 1 --2- 172.21.15.28:0/3920645390 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562cb18be860 0x562cb18cdb90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbce403d870 tx=0x7fbce403d660 comp rx=0 tx=0).stop 2024-04-23T14:44:09.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.343+0000 7fbd05578740 1 -- 172.21.15.28:0/3920645390 shutdown_connections 2024-04-23T14:44:09.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.343+0000 7fbd05578740 1 --2- 172.21.15.28:0/3920645390 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562cb18c9850 0x562cb1942a80 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.343+0000 7fbd05578740 1 --2- 172.21.15.28:0/3920645390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562cb18bf180 0x562cb18ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.343+0000 7fbd05578740 1 --2- 172.21.15.28:0/3920645390 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562cb18be860 0x562cb18cdb90 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.343+0000 7fbd05578740 1 --2- 172.21.15.28:0/3920645390 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562cb18bab80 0x562cb18cd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.343+0000 7fbce9600640 0 client.25354 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:44:09.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.343+0000 7fbd05578740 1 -- 172.21.15.28:0/3920645390 >> 172.21.15.28:0/3920645390 conn(0x562cb18c5470 msgr2=0x562cb18b9390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:09.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.343+0000 7fbd05578740 1 -- 172.21.15.28:0/3920645390 shutdown_connections 2024-04-23T14:44:09.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.343+0000 7fbd05578740 1 -- 172.21.15.28:0/3920645390 wait complete. 2024-04-23T14:44:09.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.343+0000 7fbd05578740 20 client.25354 trim_cache size 0 max 16384 2024-04-23T14:44:09.346 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:44:09.346 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-23T14:44:09.346 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-23T14:44:09.346 INFO:teuthology.orchestra.run.smithi028.stdout: "8664f60c-bb6f-4943-8d5c-d35b29eeeda6": { 2024-04-23T14:44:09.346 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-23T14:44:09.346 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "8664f60c-bb6f-4943-8d5c-d35b29eeeda6", 2024-04-23T14:44:09.346 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-23T14:44:09.346 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:44:09.346 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:44:09.346 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:44:09.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.344+0000 7fbd02600640 1 -- 172.21.15.28:0/1399059373 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fbce00a3e30 msgr2=0x7fbce00a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:09.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.344+0000 7fbd02600640 1 --2- 172.21.15.28:0/1399059373 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fbce00a3e30 0x7fbce00a62f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4009fc0 tx=0x7fbcf4005f90 comp rx=0 tx=0).stop 2024-04-23T14:44:09.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.344+0000 7fbd02600640 1 -- 172.21.15.28:0/1399059373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcfc076530 msgr2=0x7fbcfc19f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:09.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.344+0000 7fbd02600640 1 --2- 172.21.15.28:0/1399059373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcfc076530 0x7fbcfc19f060 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbcec00efb0 tx=0x7fbcec00c740 comp rx=0 tx=0).stop 2024-04-23T14:44:09.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.345+0000 7fbd02600640 1 -- 172.21.15.28:0/1399059373 shutdown_connections 2024-04-23T14:44:09.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.345+0000 7fbd02600640 1 --2- 172.21.15.28:0/1399059373 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fbce00a3e30 0x7fbce00a62f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.345+0000 7fbd02600640 1 --2- 172.21.15.28:0/1399059373 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbcfc112ee0 0x7fbcfc1a38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.345+0000 7fbd02600640 1 --2- 172.21.15.28:0/1399059373 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbcfc076e50 0x7fbcfc19f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.345+0000 7fbd02600640 1 --2- 172.21.15.28:0/1399059373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcfc076530 0x7fbcfc19f060 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.345+0000 7fbd02600640 1 -- 172.21.15.28:0/1399059373 >> 172.21.15.28:0/1399059373 conn(0x7fbcfc0ffd90 msgr2=0x7fbcfc114600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:09.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.345+0000 7fbd02600640 1 -- 172.21.15.28:0/1399059373 shutdown_connections 2024-04-23T14:44:09.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.345+0000 7fbd02600640 1 -- 172.21.15.28:0/1399059373 wait complete. 2024-04-23T14:44:09.367 INFO:tasks.cephfs.filesystem:scrub status for tag:8664f60c-bb6f-4943-8d5c-d35b29eeeda6 - {'path': '/', 'tag': '8664f60c-bb6f-4943-8d5c-d35b29eeeda6', 'options': 'recursive,force'} 2024-04-23T14:44:09.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.371+0000 7f7c7ca00640 1 -- 172.21.15.28:0/3543873454 <== mgr.25153 v2:172.21.15.28:6800/1615546338 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f7c78062440 con 0x7f7c5c0a3e20 2024-04-23T14:44:09.372 INFO:teuthology.orchestra.run.smithi028.stdout:Schedule set for path / 2024-04-23T14:44:09.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.373+0000 7f7c77e00640 1 -- 172.21.15.28:0/3543873454 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7c5c0a3e20 msgr2=0x7f7c5c0a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:09.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.373+0000 7f7c77e00640 1 --2- 172.21.15.28:0/3543873454 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7c5c0a3e20 0x7f7c5c0a62e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7c68009830 tx=0x7f7c68030570 comp rx=0 tx=0).stop 2024-04-23T14:44:09.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.373+0000 7f7c77e00640 1 -- 172.21.15.28:0/3543873454 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c78105c70 msgr2=0x7f7c781a1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:09.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.373+0000 7f7c77e00640 1 --2- 172.21.15.28:0/3543873454 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c78105c70 0x7f7c781a1950 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7c7000eb10 tx=0x7f7c7000ef10 comp rx=0 tx=0).stop 2024-04-23T14:44:09.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.373+0000 7f7c77e00640 1 -- 172.21.15.28:0/3543873454 shutdown_connections 2024-04-23T14:44:09.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.373+0000 7f7c77e00640 1 --2- 172.21.15.28:0/3543873454 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7c5c0a3e20 0x7f7c5c0a62e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.373+0000 7f7c77e00640 1 --2- 172.21.15.28:0/3543873454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c781052b0 0x7f7c7819a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.373+0000 7f7c77e00640 1 --2- 172.21.15.28:0/3543873454 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c78105c70 0x7f7c781a1950 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.373+0000 7f7c77e00640 1 --2- 172.21.15.28:0/3543873454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c781040c0 0x7f7c7819a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.373+0000 7f7c77e00640 1 -- 172.21.15.28:0/3543873454 >> 172.21.15.28:0/3543873454 conn(0x7f7c780ff630 msgr2=0x7f7c78108600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:09.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.373+0000 7f7c77e00640 1 -- 172.21.15.28:0/3543873454 shutdown_connections 2024-04-23T14:44:09.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.374+0000 7f7c77e00640 1 -- 172.21.15.28:0/3543873454 wait complete. 2024-04-23T14:44:09.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-23T14:44:09.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.610+0000 7fb3c3200640 1 -- 172.21.15.28:0/2279571627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3bc104460 msgr2=0x7fb3bc104840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:09.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.610+0000 7fb3c3200640 1 --2- 172.21.15.28:0/2279571627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3bc104460 0x7fb3bc104840 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac009900 tx=0x7fb3ac02a3f0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.610+0000 7fb3c3200640 1 -- 172.21.15.28:0/2279571627 shutdown_connections 2024-04-23T14:44:09.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.610+0000 7fb3c3200640 1 --2- 172.21.15.28:0/2279571627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3bc109530 0x7fb3bc111100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.610+0000 7fb3c3200640 1 --2- 172.21.15.28:0/2279571627 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb3bc104e10 0x7fb3bc108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.610+0000 7fb3c3200640 1 --2- 172.21.15.28:0/2279571627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3bc104460 0x7fb3bc104840 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.610+0000 7fb3c3200640 1 -- 172.21.15.28:0/2279571627 >> 172.21.15.28:0/2279571627 conn(0x7fb3bc0ffd00 msgr2=0x7fb3bc102120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:09.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.610+0000 7fb3c3200640 1 -- 172.21.15.28:0/2279571627 shutdown_connections 2024-04-23T14:44:09.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.610+0000 7fb3c3200640 1 -- 172.21.15.28:0/2279571627 wait complete. 2024-04-23T14:44:09.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.612+0000 7fb3c3200640 1 Processor -- start 2024-04-23T14:44:09.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.612+0000 7fb3c3200640 1 -- start start 2024-04-23T14:44:09.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.612+0000 7fb3c3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3bc104460 0x7fb3bc1a0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.612+0000 7fb3c3200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3bc109530 0x7fb3bc1a0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.612+0000 7fb3c3200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb3bc19a8b0 0x7fb3bc19ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.612+0000 7fb3c3200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb3bc114630 con 0x7fb3bc104460 2024-04-23T14:44:09.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.612+0000 7fb3c3200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb3bc1144b0 con 0x7fb3bc109530 2024-04-23T14:44:09.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.612+0000 7fb3c3200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb3bc1147b0 con 0x7fb3bc19a8b0 2024-04-23T14:44:09.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.613+0000 7fb3bbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3bc104460 0x7fb3bc1a0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.613+0000 7fb3bbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3bc104460 0x7fb3bc1a0700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43070/0 (socket says 172.21.15.28:43070) 2024-04-23T14:44:09.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.613+0000 7fb3bbe00640 1 -- 172.21.15.28:0/327188836 learned_addr learned my addr 172.21.15.28:0/327188836 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:09.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.613+0000 7fb3c1200640 1 --2- 172.21.15.28:0/327188836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb3bc19a8b0 0x7fb3bc19ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.613+0000 7fb3bb400640 1 --2- 172.21.15.28:0/327188836 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3bc109530 0x7fb3bc1a0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.613+0000 7fb3bbe00640 1 -- 172.21.15.28:0/327188836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb3bc19a8b0 msgr2=0x7fb3bc19ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:09.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.613+0000 7fb3bbe00640 1 --2- 172.21.15.28:0/327188836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb3bc19a8b0 0x7fb3bc19ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.613+0000 7fb3bbe00640 1 -- 172.21.15.28:0/327188836 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3bc109530 msgr2=0x7fb3bc1a0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:09.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.613+0000 7fb3bbe00640 1 --2- 172.21.15.28:0/327188836 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3bc109530 0x7fb3bc1a0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.613+0000 7fb3bbe00640 1 -- 172.21.15.28:0/327188836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3bc19b3d0 con 0x7fb3bc104460 2024-04-23T14:44:09.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.614+0000 7fb3bbe00640 1 --2- 172.21.15.28:0/327188836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3bc104460 0x7fb3bc1a0700 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac004770 tx=0x7fb3ac003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:09.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.614+0000 7fb3b8c00640 1 -- 172.21.15.28:0/327188836 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3ac03f070 con 0x7fb3bc104460 2024-04-23T14:44:09.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.614+0000 7fb3b8c00640 1 -- 172.21.15.28:0/327188836 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3ac031ca0 con 0x7fb3bc104460 2024-04-23T14:44:09.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.614+0000 7fb3b8c00640 1 -- 172.21.15.28:0/327188836 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3ac035030 con 0x7fb3bc104460 2024-04-23T14:44:09.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.614+0000 7fb3c3200640 1 -- 172.21.15.28:0/327188836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3bc19b660 con 0x7fb3bc104460 2024-04-23T14:44:09.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.614+0000 7fb3c3200640 1 -- 172.21.15.28:0/327188836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3bc1a7800 con 0x7fb3bc104460 2024-04-23T14:44:09.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.616+0000 7fb3c3200640 1 -- 172.21.15.28:0/327188836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3bc04fd90 con 0x7fb3bc104460 2024-04-23T14:44:09.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.618+0000 7fb3b8c00640 1 -- 172.21.15.28:0/327188836 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7fb3ac042070 con 0x7fb3bc104460 2024-04-23T14:44:09.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.620+0000 7fb3b8c00640 1 --2- 172.21.15.28:0/327188836 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb3a00a3e60 0x7fb3a00a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:09.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.620+0000 7fb3b8c00640 1 -- 172.21.15.28:0/327188836 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fb3ac0f0fa0 con 0x7fb3bc104460 2024-04-23T14:44:09.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.620+0000 7fb3bb400640 1 --2- 172.21.15.28:0/327188836 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb3a00a3e60 0x7fb3a00a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:09.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.621+0000 7fb3bb400640 1 --2- 172.21.15.28:0/327188836 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb3a00a3e60 0x7fb3a00a6320 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc1033e0 tx=0x7fb3a8005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:09.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.624+0000 7fb3b8c00640 1 -- 172.21.15.28:0/327188836 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb3ac0bcef0 con 0x7fb3bc104460 2024-04-23T14:44:09.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.867+0000 7fb3c3200640 1 -- 172.21.15.28:0/327188836 --> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7fb3bc104eb0 con 0x7fb3a00a3e60 2024-04-23T14:44:09.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.984+0000 7fb3b8c00640 1 -- 172.21.15.28:0/327188836 <== mgr.25153 v2:172.21.15.28:6800/1615546338 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7fb3bc104eb0 con 0x7fb3a00a3e60 2024-04-23T14:44:09.986 INFO:teuthology.orchestra.run.smithi028.stdout:Retention added to path / 2024-04-23T14:44:09.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.987+0000 7fb3c3200640 1 -- 172.21.15.28:0/327188836 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb3a00a3e60 msgr2=0x7fb3a00a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:09.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.987+0000 7fb3c3200640 1 --2- 172.21.15.28:0/327188836 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb3a00a3e60 0x7fb3a00a6320 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc1033e0 tx=0x7fb3a8005f90 comp rx=0 tx=0).stop 2024-04-23T14:44:09.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.987+0000 7fb3c3200640 1 -- 172.21.15.28:0/327188836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3bc104460 msgr2=0x7fb3bc1a0700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:09.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.987+0000 7fb3c3200640 1 --2- 172.21.15.28:0/327188836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3bc104460 0x7fb3bc1a0700 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac004770 tx=0x7fb3ac003c10 comp rx=0 tx=0).stop 2024-04-23T14:44:09.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.988+0000 7fb3c3200640 1 -- 172.21.15.28:0/327188836 shutdown_connections 2024-04-23T14:44:09.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.988+0000 7fb3c3200640 1 --2- 172.21.15.28:0/327188836 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb3a00a3e60 0x7fb3a00a6320 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.988+0000 7fb3c3200640 1 --2- 172.21.15.28:0/327188836 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3bc109530 0x7fb3bc1a0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.988+0000 7fb3c3200640 1 --2- 172.21.15.28:0/327188836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb3bc19a8b0 0x7fb3bc19ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.988+0000 7fb3c3200640 1 --2- 172.21.15.28:0/327188836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3bc104460 0x7fb3bc1a0700 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:09.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.988+0000 7fb3c3200640 1 -- 172.21.15.28:0/327188836 >> 172.21.15.28:0/327188836 conn(0x7fb3bc0ffd00 msgr2=0x7fb3bc101ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:09.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.988+0000 7fb3c3200640 1 -- 172.21.15.28:0/327188836 shutdown_connections 2024-04-23T14:44:09.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:09.988+0000 7fb3c3200640 1 -- 172.21.15.28:0/327188836 wait complete. 2024-04-23T14:44:10.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:09 smithi028 ceph-mon[27810]: Updating smithi028:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T14:44:10.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:09 smithi028 ceph-mon[27810]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-04-23T14:44:10.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:09 smithi028 ceph-mon[27810]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-04-23T14:44:10.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:09 smithi028 ceph-mon[27810]: Updating smithi038:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T14:44:10.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:09 smithi028 ceph-mon[27810]: Updating smithi105:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T14:44:10.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:09 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:10.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:09 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:10.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:09 smithi028 ceph-mon[27810]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-04-23T14:44:10.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:09 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:10.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:09 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:10.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:09 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3841187774' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:44:10.010 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-23T14:44:10.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:09 smithi105 ceph-mon[32555]: Updating smithi028:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T14:44:10.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:09 smithi105 ceph-mon[32555]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-04-23T14:44:10.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:09 smithi105 ceph-mon[32555]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-04-23T14:44:10.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:09 smithi105 ceph-mon[32555]: Updating smithi038:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T14:44:10.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:09 smithi105 ceph-mon[32555]: Updating smithi105:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T14:44:10.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:09 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:10.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:09 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:10.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:09 smithi105 ceph-mon[32555]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-04-23T14:44:10.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:09 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:10.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:09 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:10.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:09 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3841187774' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:44:10.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:09 smithi038 ceph-mon[32348]: Updating smithi028:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T14:44:10.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:09 smithi038 ceph-mon[32348]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-04-23T14:44:10.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:09 smithi038 ceph-mon[32348]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-04-23T14:44:10.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:09 smithi038 ceph-mon[32348]: Updating smithi038:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T14:44:10.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:09 smithi038 ceph-mon[32348]: Updating smithi105:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T14:44:10.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:09 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:10.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:09 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:10.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:09 smithi038 ceph-mon[32348]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-04-23T14:44:10.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:09 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:10.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:09 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:10.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:09 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3841187774' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:44:10.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.189+0000 7f716e000640 1 -- 172.21.15.28:0/2605197304 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f71681042b0 msgr2=0x7f7168104690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:10.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.189+0000 7f716e000640 1 --2- 172.21.15.28:0/2605197304 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f71681042b0 0x7f7168104690 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f715c009900 tx=0x7f715c02a3f0 comp rx=0 tx=0).stop 2024-04-23T14:44:10.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.189+0000 7f716e000640 1 -- 172.21.15.28:0/2605197304 shutdown_connections 2024-04-23T14:44:10.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.189+0000 7f716e000640 1 --2- 172.21.15.28:0/2605197304 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7168104c60 0x7f7168108ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:10.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.189+0000 7f716e000640 1 --2- 172.21.15.28:0/2605197304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7168109380 0x7f7168110f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:10.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.189+0000 7f716e000640 1 --2- 172.21.15.28:0/2605197304 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f71681042b0 0x7f7168104690 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:10.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.189+0000 7f716e000640 1 -- 172.21.15.28:0/2605197304 >> 172.21.15.28:0/2605197304 conn(0x7f71680ffb10 msgr2=0x7f7168101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:10.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.190+0000 7f716e000640 1 -- 172.21.15.28:0/2605197304 shutdown_connections 2024-04-23T14:44:10.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.190+0000 7f716e000640 1 -- 172.21.15.28:0/2605197304 wait complete. 2024-04-23T14:44:10.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.191+0000 7f716e000640 1 Processor -- start 2024-04-23T14:44:10.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.191+0000 7f716e000640 1 -- start start 2024-04-23T14:44:10.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.192+0000 7f716e000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f71681042b0 0x7f71681a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:10.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.192+0000 7f716e000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7168109380 0x7f71681a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:10.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.192+0000 7f716e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f716819a760 0x7f716819abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:10.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.192+0000 7f716e000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7168114460 con 0x7f716819a760 2024-04-23T14:44:10.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.192+0000 7f716e000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f71681142e0 con 0x7f7168109380 2024-04-23T14:44:10.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.192+0000 7f716e000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f71681145e0 con 0x7f71681042b0 2024-04-23T14:44:10.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.192+0000 7f716cc00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f71681042b0 0x7f71681a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:10.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.192+0000 7f716cc00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f71681042b0 0x7f71681a06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:40430/0 (socket says 172.21.15.28:40430) 2024-04-23T14:44:10.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.192+0000 7f716cc00640 1 -- 172.21.15.28:0/2107357214 learned_addr learned my addr 172.21.15.28:0/2107357214 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:10.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.192+0000 7f716d600640 1 --2- 172.21.15.28:0/2107357214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f716819a760 0x7f716819abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:10.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.192+0000 7f7167e00640 1 --2- 172.21.15.28:0/2107357214 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7168109380 0x7f71681a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:10.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.192+0000 7f716cc00640 1 -- 172.21.15.28:0/2107357214 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7168109380 msgr2=0x7f71681a0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:10.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.192+0000 7f716cc00640 1 --2- 172.21.15.28:0/2107357214 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7168109380 0x7f71681a0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:10.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.192+0000 7f716cc00640 1 -- 172.21.15.28:0/2107357214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f716819a760 msgr2=0x7f716819abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:10.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.192+0000 7f716cc00640 1 --2- 172.21.15.28:0/2107357214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f716819a760 0x7f716819abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:10.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.192+0000 7f716cc00640 1 -- 172.21.15.28:0/2107357214 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f716819b360 con 0x7f71681042b0 2024-04-23T14:44:10.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.192+0000 7f716d600640 1 --2- 172.21.15.28:0/2107357214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f716819a760 0x7f716819abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:44:10.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.193+0000 7f716cc00640 1 --2- 172.21.15.28:0/2107357214 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f71681042b0 0x7f71681a06a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f715c004770 tx=0x7f715c003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:10.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.193+0000 7f7165600640 1 -- 172.21.15.28:0/2107357214 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715c03f070 con 0x7f71681042b0 2024-04-23T14:44:10.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.193+0000 7f7165600640 1 -- 172.21.15.28:0/2107357214 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f715c031ca0 con 0x7f71681042b0 2024-04-23T14:44:10.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.193+0000 7f7165600640 1 -- 172.21.15.28:0/2107357214 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715c035030 con 0x7f71681042b0 2024-04-23T14:44:10.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.193+0000 7f716e000640 1 -- 172.21.15.28:0/2107357214 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f716819b5f0 con 0x7f71681042b0 2024-04-23T14:44:10.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.193+0000 7f716e000640 1 -- 172.21.15.28:0/2107357214 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71681a76e0 con 0x7f71681042b0 2024-04-23T14:44:10.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.194+0000 7f716e000640 1 -- 172.21.15.28:0/2107357214 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7168105780 con 0x7f71681042b0 2024-04-23T14:44:10.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.197+0000 7f7165600640 1 -- 172.21.15.28:0/2107357214 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7f715c042070 con 0x7f71681042b0 2024-04-23T14:44:10.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.198+0000 7f7165600640 1 --2- 172.21.15.28:0/2107357214 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f71500a3e20 0x7f71500a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:10.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.198+0000 7f7165600640 1 -- 172.21.15.28:0/2107357214 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f715c0f1680 con 0x7f71681042b0 2024-04-23T14:44:10.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.198+0000 7f7167e00640 1 --2- 172.21.15.28:0/2107357214 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f71500a3e20 0x7f71500a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:10.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.199+0000 7f7167e00640 1 --2- 172.21.15.28:0/2107357214 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f71500a3e20 0x7f71500a62e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f71540095f0 tx=0x7f7154005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:10.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.200+0000 7f7165600640 1 -- 172.21.15.28:0/2107357214 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f715c0bdb60 con 0x7f71681042b0 2024-04-23T14:44:10.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.433+0000 7f716e000640 1 -- 172.21.15.28:0/2107357214 --> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f7168104c60 con 0x7f71500a3e20 2024-04-23T14:44:10.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.452+0000 7f7165600640 1 -- 172.21.15.28:0/2107357214 <== mgr.25153 v2:172.21.15.28:6800/1615546338 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f7168104c60 con 0x7f71500a3e20 2024-04-23T14:44:10.453 INFO:teuthology.orchestra.run.smithi028.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-23T00:00:00", "created": "2024-04-23T14:44:09", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-23T14:44:10.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.455+0000 7f716e000640 1 -- 172.21.15.28:0/2107357214 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f71500a3e20 msgr2=0x7f71500a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:10.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.455+0000 7f716e000640 1 --2- 172.21.15.28:0/2107357214 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f71500a3e20 0x7f71500a62e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f71540095f0 tx=0x7f7154005f90 comp rx=0 tx=0).stop 2024-04-23T14:44:10.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.455+0000 7f716e000640 1 -- 172.21.15.28:0/2107357214 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f71681042b0 msgr2=0x7f71681a06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:10.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.455+0000 7f716e000640 1 --2- 172.21.15.28:0/2107357214 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f71681042b0 0x7f71681a06a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f715c004770 tx=0x7f715c003c10 comp rx=0 tx=0).stop 2024-04-23T14:44:10.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.455+0000 7f716e000640 1 -- 172.21.15.28:0/2107357214 shutdown_connections 2024-04-23T14:44:10.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.455+0000 7f716e000640 1 --2- 172.21.15.28:0/2107357214 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f71500a3e20 0x7f71500a62e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:10.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.455+0000 7f716e000640 1 --2- 172.21.15.28:0/2107357214 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7168109380 0x7f71681a0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:10.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.455+0000 7f716e000640 1 --2- 172.21.15.28:0/2107357214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f716819a760 0x7f716819abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:10.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.455+0000 7f716e000640 1 --2- 172.21.15.28:0/2107357214 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f71681042b0 0x7f71681a06a0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:10.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.455+0000 7f716e000640 1 -- 172.21.15.28:0/2107357214 >> 172.21.15.28:0/2107357214 conn(0x7f71680ffb10 msgr2=0x7f7168101e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:10.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.455+0000 7f716e000640 1 -- 172.21.15.28:0/2107357214 shutdown_connections 2024-04-23T14:44:10.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.455+0000 7f716e000640 1 -- 172.21.15.28:0/2107357214 wait complete. 2024-04-23T14:44:10.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-23T14:44:10.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.687+0000 7f4672c00640 1 -- 172.21.15.28:0/302278842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f466c10e8f0 msgr2=0x7f466c110ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:10.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.687+0000 7f4672c00640 1 --2- 172.21.15.28:0/302278842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f466c10e8f0 0x7f466c110ce0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4664009ca0 tx=0x7f466402a950 comp rx=0 tx=0).stop 2024-04-23T14:44:10.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.687+0000 7f4672c00640 1 -- 172.21.15.28:0/302278842 shutdown_connections 2024-04-23T14:44:10.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.687+0000 7f4672c00640 1 --2- 172.21.15.28:0/302278842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f466c10e8f0 0x7f466c110ce0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:10.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.687+0000 7f4672c00640 1 --2- 172.21.15.28:0/302278842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f466c074c20 0x7f466c10e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:10.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.687+0000 7f4672c00640 1 --2- 172.21.15.28:0/302278842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f466c074300 0x7f466c0746e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:10.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.687+0000 7f4672c00640 1 -- 172.21.15.28:0/302278842 >> 172.21.15.28:0/302278842 conn(0x7f466c079490 msgr2=0x7f466c0798a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:10.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.687+0000 7f4672c00640 1 -- 172.21.15.28:0/302278842 shutdown_connections 2024-04-23T14:44:10.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.687+0000 7f4672c00640 1 -- 172.21.15.28:0/302278842 wait complete. 2024-04-23T14:44:10.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.688+0000 7f4672c00640 1 Processor -- start 2024-04-23T14:44:10.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.688+0000 7f4672c00640 1 -- start start 2024-04-23T14:44:10.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.689+0000 7f4672c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f466c074300 0x7f466c1a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:10.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.689+0000 7f4672c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f466c074c20 0x7f466c1a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:10.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.689+0000 7f4672c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f466c10e8f0 0x7f466c19a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:10.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.689+0000 7f4672c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f466c114610 con 0x7f466c10e8f0 2024-04-23T14:44:10.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.689+0000 7f4672c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f466c114490 con 0x7f466c074300 2024-04-23T14:44:10.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.689+0000 7f4672c00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f466c114790 con 0x7f466c074c20 2024-04-23T14:44:10.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.689+0000 7f466be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f466c074300 0x7f466c1a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:10.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.689+0000 7f466be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f466c074300 0x7f466c1a0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:42784/0 (socket says 172.21.15.28:42784) 2024-04-23T14:44:10.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.689+0000 7f466be00640 1 -- 172.21.15.28:0/2199291557 learned_addr learned my addr 172.21.15.28:0/2199291557 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:10.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.689+0000 7f4670c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f466c10e8f0 0x7f466c19a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:10.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.689+0000 7f4663400640 1 --2- 172.21.15.28:0/2199291557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f466c074c20 0x7f466c1a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:10.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.689+0000 7f466be00640 1 -- 172.21.15.28:0/2199291557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f466c074c20 msgr2=0x7f466c1a0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:10.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.689+0000 7f466be00640 1 --2- 172.21.15.28:0/2199291557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f466c074c20 0x7f466c1a0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:10.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.689+0000 7f466be00640 1 -- 172.21.15.28:0/2199291557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f466c10e8f0 msgr2=0x7f466c19a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:10.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.690+0000 7f466be00640 1 --2- 172.21.15.28:0/2199291557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f466c10e8f0 0x7f466c19a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:10.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.690+0000 7f466be00640 1 -- 172.21.15.28:0/2199291557 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f466c19aff0 con 0x7f466c074300 2024-04-23T14:44:10.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.690+0000 7f466be00640 1 --2- 172.21.15.28:0/2199291557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f466c074300 0x7f466c1a0670 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f465c00d950 tx=0x7f465c00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:10.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.690+0000 7f4669600640 1 -- 172.21.15.28:0/2199291557 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f465c017070 con 0x7f466c074300 2024-04-23T14:44:10.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.690+0000 7f4669600640 1 -- 172.21.15.28:0/2199291557 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f465c009c30 con 0x7f466c074300 2024-04-23T14:44:10.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.690+0000 7f4672c00640 1 -- 172.21.15.28:0/2199291557 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f466c19b2e0 con 0x7f466c074300 2024-04-23T14:44:10.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.690+0000 7f4669600640 1 -- 172.21.15.28:0/2199291557 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f465c00f6b0 con 0x7f466c074300 2024-04-23T14:44:10.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.690+0000 7f4672c00640 1 -- 172.21.15.28:0/2199291557 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f466c1a75f0 con 0x7f466c074300 2024-04-23T14:44:10.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.691+0000 7f4672c00640 1 -- 172.21.15.28:0/2199291557 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f466c19b620 con 0x7f466c074300 2024-04-23T14:44:10.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.693+0000 7f4669600640 1 -- 172.21.15.28:0/2199291557 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7f465c0109d0 con 0x7f466c074300 2024-04-23T14:44:10.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.695+0000 7f4669600640 1 --2- 172.21.15.28:0/2199291557 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f46500a3e60 0x7f46500a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:10.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.695+0000 7f4669600640 1 -- 172.21.15.28:0/2199291557 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f465c0d0630 con 0x7f466c074300 2024-04-23T14:44:10.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.695+0000 7f4663400640 1 --2- 172.21.15.28:0/2199291557 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f46500a3e60 0x7f46500a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:10.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.696+0000 7f4663400640 1 --2- 172.21.15.28:0/2199291557 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f46500a3e60 0x7f46500a6320 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f466c19bf80 tx=0x7f4654005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:10.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.703+0000 7f4669600640 1 -- 172.21.15.28:0/2199291557 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f465c009de0 con 0x7f466c074300 2024-04-23T14:44:10.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:10 smithi028 ceph-mon[27810]: scrub queued for path: / 2024-04-23T14:44:10.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:10 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:44:10.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:10 smithi028 ceph-mon[27810]: scrub summary: active paths [/] 2024-04-23T14:44:10.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:10 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:44:10.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:10 smithi028 ceph-mon[27810]: from='client.25318 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:44:10.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:10 smithi028 ceph-mon[27810]: Updating smithi028:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T14:44:10.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:10 smithi028 ceph-mon[27810]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 0 B/s wr, 18 op/s 2024-04-23T14:44:10.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:10 smithi028 ceph-mon[27810]: scrub summary: idle 2024-04-23T14:44:10.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:10 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:10.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:10 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:10.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:10 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:10.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:10 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:44:10.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.942+0000 7f4672c00640 1 -- 172.21.15.28:0/2199291557 --> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f466c0ff510 con 0x7f46500a3e60 2024-04-23T14:44:10.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.952+0000 7f4669600640 1 -- 172.21.15.28:0/2199291557 <== mgr.25153 v2:172.21.15.28:6800/1615546338 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f466c0ff510 con 0x7f46500a3e60 2024-04-23T14:44:10.953 INFO:teuthology.orchestra.run.smithi028.stdout:/ 1m 6m3h 2024-04-23T14:44:10.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.955+0000 7f4672c00640 1 -- 172.21.15.28:0/2199291557 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f46500a3e60 msgr2=0x7f46500a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:10.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.955+0000 7f4672c00640 1 --2- 172.21.15.28:0/2199291557 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f46500a3e60 0x7f46500a6320 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f466c19bf80 tx=0x7f4654005f90 comp rx=0 tx=0).stop 2024-04-23T14:44:10.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.955+0000 7f4672c00640 1 -- 172.21.15.28:0/2199291557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f466c074300 msgr2=0x7f466c1a0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:10.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.955+0000 7f4672c00640 1 --2- 172.21.15.28:0/2199291557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f466c074300 0x7f466c1a0670 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f465c00d950 tx=0x7f465c00de10 comp rx=0 tx=0).stop 2024-04-23T14:44:10.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.955+0000 7f4672c00640 1 -- 172.21.15.28:0/2199291557 shutdown_connections 2024-04-23T14:44:10.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.955+0000 7f4672c00640 1 --2- 172.21.15.28:0/2199291557 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f46500a3e60 0x7f46500a6320 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:10.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.955+0000 7f4672c00640 1 --2- 172.21.15.28:0/2199291557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f466c10e8f0 0x7f466c19a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:10.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.955+0000 7f4672c00640 1 --2- 172.21.15.28:0/2199291557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f466c074c20 0x7f466c1a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:10.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.955+0000 7f4672c00640 1 --2- 172.21.15.28:0/2199291557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f466c074300 0x7f466c1a0670 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:10.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.955+0000 7f4672c00640 1 -- 172.21.15.28:0/2199291557 >> 172.21.15.28:0/2199291557 conn(0x7f466c079490 msgr2=0x7f466c0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:10.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.955+0000 7f4672c00640 1 -- 172.21.15.28:0/2199291557 shutdown_connections 2024-04-23T14:44:10.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:10.955+0000 7f4672c00640 1 -- 172.21.15.28:0/2199291557 wait complete. 2024-04-23T14:44:10.975 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-23T14:44:11.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:10 smithi105 ceph-mon[32555]: scrub queued for path: / 2024-04-23T14:44:11.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:10 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:44:11.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:10 smithi105 ceph-mon[32555]: scrub summary: active paths [/] 2024-04-23T14:44:11.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:10 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:44:11.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:10 smithi105 ceph-mon[32555]: from='client.25318 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:44:11.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:10 smithi105 ceph-mon[32555]: Updating smithi028:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T14:44:11.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:10 smithi105 ceph-mon[32555]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 0 B/s wr, 18 op/s 2024-04-23T14:44:11.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:10 smithi105 ceph-mon[32555]: scrub summary: idle 2024-04-23T14:44:11.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:10 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:11.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:10 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:11.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:10 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:11.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:10 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:44:11.057 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-23T14:44:11.067 INFO:teuthology.run_tasks:Running task workunit... 2024-04-23T14:44:11.079 INFO:tasks.workunit:Pulling workunits from ref d44fddd910b5e94da0f2357af4be37d7bd495ae0 2024-04-23T14:44:11.079 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-23T14:44:11.080 INFO:tasks.workunit:timeout=3h 2024-04-23T14:44:11.080 INFO:tasks.workunit:cleanup=True 2024-04-23T14:44:11.080 DEBUG:teuthology.orchestra.run.smithi028:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-23T14:44:11.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:10 smithi038 ceph-mon[32348]: scrub queued for path: / 2024-04-23T14:44:11.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:10 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:44:11.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:10 smithi038 ceph-mon[32348]: scrub summary: active paths [/] 2024-04-23T14:44:11.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:10 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:44:11.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:10 smithi038 ceph-mon[32348]: from='client.25318 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:44:11.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:10 smithi038 ceph-mon[32348]: Updating smithi028:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T14:44:11.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:10 smithi038 ceph-mon[32348]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 0 B/s wr, 18 op/s 2024-04-23T14:44:11.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:10 smithi038 ceph-mon[32348]: scrub summary: idle 2024-04-23T14:44:11.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:10 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:11.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:10 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:11.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:10 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:11.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:10 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:44:11.135 INFO:teuthology.orchestra.run.smithi028.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-23T14:44:11.136 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-23T14:44:11.136 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 72h/114d Inode: 1099511628283 Links: 2 2024-04-23T14:44:11.136 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T14:44:11.136 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-23T14:44:11.136 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-04-23 14:41:46.481399116 +0000 2024-04-23T14:44:11.136 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-04-23 14:41:46.481399116 +0000 2024-04-23T14:44:11.136 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-04-23 14:43:59.868190716 +0000 2024-04-23T14:44:11.136 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-04-23 14:41:46.481399116 +0000 2024-04-23T14:44:11.136 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-23T14:44:11.136 DEBUG:teuthology.orchestra.run.smithi028:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-23T14:44:11.234 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d44fddd910b5e94da0f2357af4be37d7bd495ae0 2024-04-23T14:44:11.299 INFO:tasks.workunit.client.0.smithi028.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-23T14:44:12.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:11 smithi028 ceph-mon[27810]: from='client.15222 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:44:12.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:11 smithi028 ceph-mon[27810]: from='client.25366 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:44:12.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:11 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:12.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:11 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:12.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:11 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:44:12.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:11 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:44:12.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:11 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:12.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:11 smithi105 ceph-mon[32555]: from='client.15222 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:44:12.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:11 smithi105 ceph-mon[32555]: from='client.25366 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:44:12.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:11 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:12.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:11 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:12.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:11 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:44:12.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:11 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:44:12.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:11 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:12.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:11 smithi038 ceph-mon[32348]: from='client.15222 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:44:12.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:11 smithi038 ceph-mon[32348]: from='client.25366 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:44:12.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:11 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:12.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:11 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:12.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:11 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:44:12.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:11 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:44:12.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:11 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:44:13.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:12 smithi105 ceph-mon[32555]: from='client.25367 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:44:13.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:12 smithi105 ceph-mon[32555]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-23T14:44:13.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:12 smithi038 ceph-mon[32348]: from='client.25367 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:44:13.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:12 smithi038 ceph-mon[32348]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-23T14:44:13.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:12 smithi028 ceph-mon[27810]: from='client.25367 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:44:13.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:12 smithi028 ceph-mon[27810]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-23T14:44:15.012 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:14 smithi038 ceph-mon[32348]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 84 KiB/s wr, 47 op/s 2024-04-23T14:44:15.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:14 smithi105 ceph-mon[32555]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 84 KiB/s wr, 47 op/s 2024-04-23T14:44:15.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:14 smithi028 ceph-mon[27810]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 84 KiB/s wr, 47 op/s 2024-04-23T14:44:17.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:16 smithi105 ceph-mon[32555]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 43 op/s 2024-04-23T14:44:17.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:16 smithi038 ceph-mon[32348]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 43 op/s 2024-04-23T14:44:17.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:16 smithi028 ceph-mon[27810]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 43 op/s 2024-04-23T14:44:19.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:18 smithi105 ceph-mon[32555]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-04-23T14:44:19.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:18 smithi038 ceph-mon[32348]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-04-23T14:44:19.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:18 smithi028 ceph-mon[27810]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-04-23T14:44:21.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:20 smithi105 ceph-mon[32555]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 77 KiB/s wr, 33 op/s 2024-04-23T14:44:21.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:20 smithi038 ceph-mon[32348]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 77 KiB/s wr, 33 op/s 2024-04-23T14:44:21.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:20 smithi028 ceph-mon[27810]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 77 KiB/s wr, 33 op/s 2024-04-23T14:44:23.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:22 smithi105 ceph-mon[32555]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 77 KiB/s wr, 33 op/s 2024-04-23T14:44:23.046 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:22 smithi028 ceph-mon[27810]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 77 KiB/s wr, 33 op/s 2024-04-23T14:44:23.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:22 smithi038 ceph-mon[32348]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 77 KiB/s wr, 33 op/s 2024-04-23T14:44:25.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:24 smithi105 ceph-mon[32555]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 77 KiB/s wr, 34 op/s 2024-04-23T14:44:25.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:24 smithi038 ceph-mon[32348]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 77 KiB/s wr, 34 op/s 2024-04-23T14:44:25.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:24 smithi028 ceph-mon[27810]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 77 KiB/s wr, 34 op/s 2024-04-23T14:44:27.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:26 smithi105 ceph-mon[32555]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:44:27.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:26 smithi038 ceph-mon[32348]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:44:27.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:26 smithi028 ceph-mon[27810]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:44:29.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:28 smithi105 ceph-mon[32555]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:44:29.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:28 smithi038 ceph-mon[32348]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:44:29.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:28 smithi028 ceph-mon[27810]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:44:30.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:29 smithi105 ceph-mon[32555]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:44:30.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:29 smithi038 ceph-mon[32348]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:44:30.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:29 smithi028 ceph-mon[27810]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:44:32.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:32 smithi105 ceph-mon[32555]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:44:32.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:32 smithi038 ceph-mon[32348]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:44:32.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:32 smithi028 ceph-mon[27810]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:44:34.675 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:34 smithi038 ceph-mon[32348]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:44:34.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:34 smithi105 ceph-mon[32555]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:44:34.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:34 smithi028 ceph-mon[27810]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:44:36.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:36 smithi105 ceph-mon[32555]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:44:36.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:36 smithi038 ceph-mon[32348]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:44:36.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:36 smithi028 ceph-mon[27810]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:44:38.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:38 smithi105 ceph-mon[32555]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:44:38.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:38 smithi038 ceph-mon[32348]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:44:38.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:38 smithi028 ceph-mon[27810]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-23T14:44:39.369 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:44:39.369 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:44:39.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.597+0000 7fc27a200640 1 -- 172.21.15.28:0/2746693803 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2740743a0 msgr2=0x7fc274074780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:39.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.597+0000 7fc27a200640 1 --2- 172.21.15.28:0/2746693803 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2740743a0 0x7fc274074780 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc25c009900 tx=0x7fc25c02a480 comp rx=0 tx=0).stop 2024-04-23T14:44:39.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.598+0000 7fc27a200640 1 -- 172.21.15.28:0/2746693803 shutdown_connections 2024-04-23T14:44:39.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.598+0000 7fc27a200640 1 --2- 172.21.15.28:0/2746693803 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc274112fd0 0x7fc2741153c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.598+0000 7fc27a200640 1 --2- 172.21.15.28:0/2746693803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc274074cc0 0x7fc274112a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.598+0000 7fc27a200640 1 --2- 172.21.15.28:0/2746693803 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2740743a0 0x7fc274074780 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.598+0000 7fc27a200640 1 -- 172.21.15.28:0/2746693803 >> 172.21.15.28:0/2746693803 conn(0x7fc2740fda70 msgr2=0x7fc2740ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:39.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.598+0000 7fc27a200640 1 -- 172.21.15.28:0/2746693803 shutdown_connections 2024-04-23T14:44:39.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.598+0000 7fc27a200640 1 -- 172.21.15.28:0/2746693803 wait complete. 2024-04-23T14:44:39.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.599+0000 7fc27a200640 1 Processor -- start 2024-04-23T14:44:39.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.599+0000 7fc27a200640 1 -- start start 2024-04-23T14:44:39.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.600+0000 7fc27a200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2740743a0 0x7fc27419ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:39.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.600+0000 7fc27a200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc274074cc0 0x7fc27419f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:39.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.600+0000 7fc27a200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc274112fd0 0x7fc2741a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:39.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.600+0000 7fc27a200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc274118a20 con 0x7fc274074cc0 2024-04-23T14:44:39.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.600+0000 7fc27a200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc2741188a0 con 0x7fc274112fd0 2024-04-23T14:44:39.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.600+0000 7fc27a200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc274118ba0 con 0x7fc2740743a0 2024-04-23T14:44:39.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.600+0000 7fc272a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc274074cc0 0x7fc27419f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:39.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.600+0000 7fc272a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc274074cc0 0x7fc27419f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43644/0 (socket says 172.21.15.28:43644) 2024-04-23T14:44:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.600+0000 7fc272a00640 1 -- 172.21.15.28:0/3169053397 learned_addr learned my addr 172.21.15.28:0/3169053397 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.600+0000 7fc273e00640 1 --2- 172.21.15.28:0/3169053397 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc274112fd0 0x7fc2741a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.600+0000 7fc273400640 1 --2- 172.21.15.28:0/3169053397 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2740743a0 0x7fc27419ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.601+0000 7fc272a00640 1 -- 172.21.15.28:0/3169053397 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2740743a0 msgr2=0x7fc27419ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.601+0000 7fc272a00640 1 --2- 172.21.15.28:0/3169053397 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2740743a0 0x7fc27419ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.601+0000 7fc272a00640 1 -- 172.21.15.28:0/3169053397 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc274112fd0 msgr2=0x7fc2741a37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.601+0000 7fc272a00640 1 --2- 172.21.15.28:0/3169053397 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc274112fd0 0x7fc2741a37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.601+0000 7fc272a00640 1 -- 172.21.15.28:0/3169053397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2741a3f10 con 0x7fc274074cc0 2024-04-23T14:44:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.601+0000 7fc273e00640 1 --2- 172.21.15.28:0/3169053397 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc274112fd0 0x7fc2741a37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:44:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.601+0000 7fc272a00640 1 --2- 172.21.15.28:0/3169053397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc274074cc0 0x7fc27419f460 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc264009d50 tx=0x7fc26400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.601+0000 7fc26be00640 1 -- 172.21.15.28:0/3169053397 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc264003b70 con 0x7fc274074cc0 2024-04-23T14:44:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.601+0000 7fc27a200640 1 -- 172.21.15.28:0/3169053397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2741a4200 con 0x7fc274074cc0 2024-04-23T14:44:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.601+0000 7fc26be00640 1 -- 172.21.15.28:0/3169053397 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc264003d10 con 0x7fc274074cc0 2024-04-23T14:44:39.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.601+0000 7fc26be00640 1 -- 172.21.15.28:0/3169053397 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc264010a20 con 0x7fc274074cc0 2024-04-23T14:44:39.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.602+0000 7fc27a200640 1 -- 172.21.15.28:0/3169053397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2741aba90 con 0x7fc274074cc0 2024-04-23T14:44:39.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.605+0000 7fc27d073740 1 Processor -- start 2024-04-23T14:44:39.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.605+0000 7fc27d073740 1 -- start start 2024-04-23T14:44:39.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.605+0000 7fc26be00640 1 -- 172.21.15.28:0/3169053397 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7fc264003ec0 con 0x7fc274074cc0 2024-04-23T14:44:39.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.605+0000 7fc27d073740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56162f89b860 0x56162f89bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:39.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.606+0000 7fc27d073740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162f89c180 0x56162f8a6310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:39.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.606+0000 7fc27d073740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56162f8a6850 0x56162f8a8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:39.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.606+0000 7fc27d073740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56162f8011f0 con 0x56162f89c180 2024-04-23T14:44:39.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.606+0000 7fc27d073740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56162f7a7350 con 0x56162f89b860 2024-04-23T14:44:39.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.606+0000 7fc27d073740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56162f7fd1b0 con 0x56162f8a6850 2024-04-23T14:44:39.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.606+0000 7fc272a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56162f8a6850 0x56162f8a8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:39.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.606+0000 7fc273400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162f89c180 0x56162f8a6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:39.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.606+0000 7fc273e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56162f89b860 0x56162f89bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:39.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.606+0000 7fc273e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56162f89b860 0x56162f89bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:42922/0 (socket says 172.21.15.28:42922) 2024-04-23T14:44:39.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.606+0000 7fc272a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56162f8a6850 0x56162f8a8c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:44250/0 (socket says 172.21.15.28:44250) 2024-04-23T14:44:39.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.606+0000 7fc272a00640 1 -- 172.21.15.28:0/2701283517 learned_addr learned my addr 172.21.15.28:0/2701283517 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:39.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.606+0000 7fc273400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162f89c180 0x56162f8a6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43654/0 (socket says 172.21.15.28:43654) 2024-04-23T14:44:39.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.606+0000 7fc273e00640 1 -- 172.21.15.28:0/2701283517 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56162f8a6850 msgr2=0x56162f8a8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:39.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.606+0000 7fc273e00640 1 --2- 172.21.15.28:0/2701283517 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56162f8a6850 0x56162f8a8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.606+0000 7fc273e00640 1 -- 172.21.15.28:0/2701283517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162f89c180 msgr2=0x56162f8a6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:39.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.606+0000 7fc273e00640 1 --2- 172.21.15.28:0/2701283517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162f89c180 0x56162f8a6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.606+0000 7fc273e00640 1 -- 172.21.15.28:0/2701283517 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56162f7fbf50 con 0x56162f89b860 2024-04-23T14:44:39.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.607+0000 7fc273e00640 1 --2- 172.21.15.28:0/2701283517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56162f89b860 0x56162f89bc40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc26c009f00 tx=0x7fc26c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=419bd13b331cf5f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:39.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.607+0000 7fc26be00640 1 --2- 172.21.15.28:0/3169053397 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc2580a3e30 0x7fc2580a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:39.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.607+0000 7fc263e00640 1 -- 172.21.15.28:0/2701283517 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc26c004310 con 0x56162f89b860 2024-04-23T14:44:39.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.607+0000 7fc263e00640 1 -- 172.21.15.28:0/2701283517 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc26c0044b0 con 0x56162f89b860 2024-04-23T14:44:39.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.607+0000 7fc263e00640 1 -- 172.21.15.28:0/2701283517 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc26c0035f0 con 0x56162f89b860 2024-04-23T14:44:39.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.607+0000 7fc26be00640 1 -- 172.21.15.28:0/3169053397 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fc2640d0ff0 con 0x7fc274074cc0 2024-04-23T14:44:39.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.607+0000 7fc273400640 1 --2- 172.21.15.28:0/3169053397 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc2580a3e30 0x7fc2580a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:39.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.608+0000 7fc27d073740 1 -- 172.21.15.28:0/2701283517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56162f89b860 msgr2=0x56162f89bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:39.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.608+0000 7fc27d073740 1 --2- 172.21.15.28:0/2701283517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56162f89b860 0x56162f89bc40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc26c009f00 tx=0x7fc26c011810 comp rx=0 tx=0).stop 2024-04-23T14:44:39.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.608+0000 7fc27d073740 1 -- 172.21.15.28:0/2701283517 shutdown_connections 2024-04-23T14:44:39.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.608+0000 7fc27d073740 1 --2- 172.21.15.28:0/2701283517 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56162f8a6850 0x56162f8a8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.608+0000 7fc27d073740 1 --2- 172.21.15.28:0/2701283517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162f89c180 0x56162f8a6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.608+0000 7fc27d073740 1 --2- 172.21.15.28:0/2701283517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56162f89b860 0x56162f89bc40 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.608+0000 7fc27d073740 1 -- 172.21.15.28:0/2701283517 >> 172.21.15.28:0/2701283517 conn(0x56162f897b80 msgr2=0x56162f897f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:39.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.608+0000 7fc273400640 1 --2- 172.21.15.28:0/3169053397 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc2580a3e30 0x7fc2580a62f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc25c000c00 tx=0x7fc25c005b50 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:39.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.608+0000 7fc27d073740 1 -- 172.21.15.28:0/2701283517 shutdown_connections 2024-04-23T14:44:39.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.608+0000 7fc27d073740 1 -- 172.21.15.28:0/2701283517 wait complete. 2024-04-23T14:44:39.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.609+0000 7fc27d073740 1 -- start start 2024-04-23T14:44:39.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.609+0000 7fc27d073740 1 Processor -- start 2024-04-23T14:44:39.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.610+0000 7fc27d073740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56162f89b860 0x56162f8a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:39.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.610+0000 7fc27d073740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56162f89c180 0x56162f8a3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:39.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.610+0000 7fc27d073740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162f8a6850 0x56162f8a4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:39.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.610+0000 7fc27d073740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56162f8a9640 con 0x56162f8a6850 2024-04-23T14:44:39.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.610+0000 7fc27d073740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56162f7ba690 con 0x56162f89c180 2024-04-23T14:44:39.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.610+0000 7fc27d073740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56162f8a97c0 con 0x56162f89b860 2024-04-23T14:44:39.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.610+0000 7fc27d073740 10 client.0 _ll_register_callbacks cb 0x56162f7f6650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:44:39.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.610+0000 7fc273400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56162f89b860 0x56162f8a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:39.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.610+0000 7fc273e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162f8a6850 0x56162f8a4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:39.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.610+0000 7fc273400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56162f89b860 0x56162f8a37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:44262/0 (socket says 172.21.15.28:44262) 2024-04-23T14:44:39.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.610+0000 7fc273400640 1 -- 172.21.15.28:0/3584763507 learned_addr learned my addr 172.21.15.28:0/3584763507 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:39.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.610+0000 7fc272a00640 1 --2- 172.21.15.28:0/3584763507 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56162f89c180 0x56162f8a3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:39.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.610+0000 7fc273400640 1 -- 172.21.15.28:0/3584763507 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56162f89c180 msgr2=0x56162f8a3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:39.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.610+0000 7fc273400640 1 --2- 172.21.15.28:0/3584763507 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56162f89c180 0x56162f8a3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.610+0000 7fc273400640 1 -- 172.21.15.28:0/3584763507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162f8a6850 msgr2=0x56162f8a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:39.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.610+0000 7fc273400640 1 --2- 172.21.15.28:0/3584763507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162f8a6850 0x56162f8a4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.610+0000 7fc273400640 1 -- 172.21.15.28:0/3584763507 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56162f7b9920 con 0x56162f89b860 2024-04-23T14:44:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.611+0000 7fc273400640 1 --2- 172.21.15.28:0/3584763507 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56162f89b860 0x56162f8a37f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc25c033000 tx=0x7fc25c009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed44cb78ca053fd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.611+0000 7fc260c00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:44:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.611+0000 7fc260c00640 1 -- 172.21.15.28:0/3584763507 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc25c04e070 con 0x56162f89b860 2024-04-23T14:44:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.611+0000 7fc27d073740 1 -- 172.21.15.28:0/3584763507 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc26c0044b0 con 0x56162f89b860 2024-04-23T14:44:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.611+0000 7fc260c00640 1 -- 172.21.15.28:0/3584763507 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fc25c004c50 con 0x56162f89b860 2024-04-23T14:44:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.611+0000 7fc260c00640 1 -- 172.21.15.28:0/3584763507 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fc25c05a050 con 0x56162f89b860 2024-04-23T14:44:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.611+0000 7fc27d073740 10 client.25384 fetch_fsmap learned FSMap version 20 2024-04-23T14:44:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.611+0000 7fc27d073740 1 -- 172.21.15.28:0/3584763507 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x56162f5dd7d0 con 0x56162f89b860 2024-04-23T14:44:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.612+0000 7fc260c00640 1 -- 172.21.15.28:0/3584763507 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc26c0044b0 con 0x56162f89b860 2024-04-23T14:44:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.612+0000 7fc260c00640 1 -- 172.21.15.28:0/3584763507 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fc25c003710 con 0x56162f89b860 2024-04-23T14:44:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.612+0000 7fc27d073740 10 client.25384 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:44:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.612+0000 7fc27d073740 10 client.25384 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:44:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.612+0000 7fc27d073740 20 client.25384 populate_metadata read hostname 'smithi028' 2024-04-23T14:44:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.612+0000 7fc27d073740 1 --2- 172.21.15.28:0/3584763507 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56162f8b6c10 0x56162f8b9040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.612+0000 7fc27d073740 4 client.25384 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:44:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.612+0000 7fc27d073740 1 -- 172.21.15.28:0/3584763507 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56162f80ec10 con 0x56162f8b6c10 2024-04-23T14:44:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.612+0000 7fc272a00640 1 --2- 172.21.15.28:0/3584763507 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56162f8b6c10 0x56162f8b9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.613+0000 7fc272a00640 1 --2- 172.21.15.28:0/3584763507 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56162f8b6c10 0x56162f8b9040 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89db1d2af945c861 server_cookie=ca637c7be978cfef in_seq=0 out_seq=0 2024-04-23T14:44:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.613+0000 7fc260c00640 10 client.25384 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:44:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.614+0000 7fc260c00640 1 -- 172.21.15.28:0/3584763507 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56162f80ec10 con 0x56162f8b6c10 2024-04-23T14:44:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.614+0000 7fc260c00640 10 client.25384 handle_command_reply: tid=0 2024-04-23T14:44:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.614+0000 7fc27d073740 1 client.25384 shutdown 2024-04-23T14:44:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.614+0000 7fc27d073740 1 -- 172.21.15.28:0/3584763507 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56162f89b860 msgr2=0x56162f8a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.614+0000 7fc27d073740 1 --2- 172.21.15.28:0/3584763507 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56162f89b860 0x56162f8a37f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc25c033000 tx=0x7fc25c009260 comp rx=0 tx=0).stop 2024-04-23T14:44:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.614+0000 7fc27d073740 1 -- 172.21.15.28:0/3584763507 shutdown_connections 2024-04-23T14:44:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.614+0000 7fc27d073740 1 --2- 172.21.15.28:0/3584763507 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56162f8b6c10 0x56162f8b9040 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.614+0000 7fc27d073740 1 --2- 172.21.15.28:0/3584763507 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56162f89c180 0x56162f8a3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.614+0000 7fc27d073740 1 --2- 172.21.15.28:0/3584763507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162f8a6850 0x56162f8a4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.614+0000 7fc27d073740 1 --2- 172.21.15.28:0/3584763507 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56162f89b860 0x56162f8a37f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.614+0000 7fc27d073740 1 -- 172.21.15.28:0/3584763507 >> 172.21.15.28:0/3584763507 conn(0x56162f897b80 msgr2=0x56162f89d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.614+0000 7fc260c00640 0 client.25384 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:44:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.614+0000 7fc27d073740 1 -- 172.21.15.28:0/3584763507 shutdown_connections 2024-04-23T14:44:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.614+0000 7fc27d073740 1 -- 172.21.15.28:0/3584763507 wait complete. 2024-04-23T14:44:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.614+0000 7fc27d073740 20 client.25384 trim_cache size 0 max 16384 2024-04-23T14:44:39.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.633+0000 7fc27d073740 1 Processor -- start 2024-04-23T14:44:39.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.633+0000 7fc27d073740 1 -- start start 2024-04-23T14:44:39.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.634+0000 7fc27d073740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56162f89b860 0x56162f8a4040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:39.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.634+0000 7fc27d073740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162f89c180 0x56162f89dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:39.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.634+0000 7fc27d073740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56162f8a6850 0x56162f8be910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:39.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.634+0000 7fc27d073740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc2741b2250 con 0x56162f89c180 2024-04-23T14:44:39.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.634+0000 7fc27d073740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc2741b4ad0 con 0x56162f89b860 2024-04-23T14:44:39.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.634+0000 7fc27d073740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc2741b28c0 con 0x56162f8a6850 2024-04-23T14:44:39.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.634+0000 7fc272a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162f89c180 0x56162f89dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:39.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.634+0000 7fc272a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162f89c180 0x56162f89dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43672/0 (socket says 172.21.15.28:43672) 2024-04-23T14:44:39.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.634+0000 7fc272a00640 1 -- 172.21.15.28:0/1881353605 learned_addr learned my addr 172.21.15.28:0/1881353605 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:39.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.634+0000 7fc273400640 1 --2- 172.21.15.28:0/1881353605 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56162f89b860 0x56162f8a4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:39.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.634+0000 7fc273e00640 1 --2- 172.21.15.28:0/1881353605 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56162f8a6850 0x56162f8be910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:39.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.634+0000 7fc272a00640 1 -- 172.21.15.28:0/1881353605 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56162f8a6850 msgr2=0x56162f8be910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.634+0000 7fc272a00640 1 --2- 172.21.15.28:0/1881353605 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56162f8a6850 0x56162f8be910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.635+0000 7fc272a00640 1 -- 172.21.15.28:0/1881353605 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56162f89b860 msgr2=0x56162f8a4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.635+0000 7fc272a00640 1 --2- 172.21.15.28:0/1881353605 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56162f89b860 0x56162f8a4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.635+0000 7fc272a00640 1 -- 172.21.15.28:0/1881353605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56162f5133f0 con 0x56162f89c180 2024-04-23T14:44:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.635+0000 7fc273400640 1 --2- 172.21.15.28:0/1881353605 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56162f89b860 0x56162f8a4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:44:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.635+0000 7fc272a00640 1 --2- 172.21.15.28:0/1881353605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162f89c180 0x56162f89dfc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc26400cba0 tx=0x7fc2640d3440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad4eb596dfaf819c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.635+0000 7fc263400640 1 -- 172.21.15.28:0/1881353605 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc26409c730 con 0x56162f89c180 2024-04-23T14:44:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.635+0000 7fc263400640 1 -- 172.21.15.28:0/1881353605 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc264009820 con 0x56162f89c180 2024-04-23T14:44:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.635+0000 7fc263400640 1 -- 172.21.15.28:0/1881353605 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc264004620 con 0x56162f89c180 2024-04-23T14:44:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.636+0000 7fc27d073740 1 -- 172.21.15.28:0/1881353605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162f89c180 msgr2=0x56162f89dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.636+0000 7fc27d073740 1 --2- 172.21.15.28:0/1881353605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162f89c180 0x56162f89dfc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc26400cba0 tx=0x7fc2640d3440 comp rx=0 tx=0).stop 2024-04-23T14:44:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.636+0000 7fc27d073740 1 -- 172.21.15.28:0/1881353605 shutdown_connections 2024-04-23T14:44:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.636+0000 7fc27d073740 1 --2- 172.21.15.28:0/1881353605 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56162f8a6850 0x56162f8be910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.636+0000 7fc27d073740 1 --2- 172.21.15.28:0/1881353605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162f89c180 0x56162f89dfc0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.636+0000 7fc27d073740 1 --2- 172.21.15.28:0/1881353605 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56162f89b860 0x56162f8a4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.636+0000 7fc27d073740 1 -- 172.21.15.28:0/1881353605 >> 172.21.15.28:0/1881353605 conn(0x56162f897b80 msgr2=0x56162f896390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.636+0000 7fc27d073740 1 -- 172.21.15.28:0/1881353605 shutdown_connections 2024-04-23T14:44:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.636+0000 7fc27d073740 1 -- 172.21.15.28:0/1881353605 wait complete. 2024-04-23T14:44:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.636+0000 7fc27d073740 1 -- start start 2024-04-23T14:44:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.637+0000 7fc27d073740 1 Processor -- start 2024-04-23T14:44:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.637+0000 7fc27d073740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56162f89b860 0x56162f8aec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.637+0000 7fc27d073740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56162f89c180 0x56162f8aba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.637+0000 7fc27d073740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162f8a6850 0x56162f8abf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.637+0000 7fc27d073740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc2741b2f90 con 0x56162f8a6850 2024-04-23T14:44:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.637+0000 7fc27d073740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc2741b4400 con 0x56162f89b860 2024-04-23T14:44:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.637+0000 7fc27d073740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc2741b3660 con 0x56162f89c180 2024-04-23T14:44:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.637+0000 7fc27d073740 10 client.0 _ll_register_callbacks cb 0x7fc25c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:44:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.637+0000 7fc273400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56162f89b860 0x56162f8aec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.637+0000 7fc273400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56162f89b860 0x56162f8aec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:42934/0 (socket says 172.21.15.28:42934) 2024-04-23T14:44:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.637+0000 7fc272a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56162f89c180 0x56162f8aba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.637+0000 7fc273400640 1 -- 172.21.15.28:0/767943655 learned_addr learned my addr 172.21.15.28:0/767943655 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.637+0000 7fc273e00640 1 --2- 172.21.15.28:0/767943655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162f8a6850 0x56162f8abf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.638+0000 7fc273400640 1 -- 172.21.15.28:0/767943655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56162f89c180 msgr2=0x56162f8aba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.638+0000 7fc273400640 1 --2- 172.21.15.28:0/767943655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56162f89c180 0x56162f8aba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.638+0000 7fc273400640 1 -- 172.21.15.28:0/767943655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162f8a6850 msgr2=0x56162f8abf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.638+0000 7fc273400640 1 --2- 172.21.15.28:0/767943655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162f8a6850 0x56162f8abf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.638+0000 7fc273400640 1 -- 172.21.15.28:0/767943655 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56162f4d30f0 con 0x56162f89b860 2024-04-23T14:44:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.638+0000 7fc273400640 1 --2- 172.21.15.28:0/767943655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56162f89b860 0x56162f8aec30 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc25c031550 tx=0x7fc25c0045b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=582464e71a28c0c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.638+0000 7fc260c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:44:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.638+0000 7fc260c00640 1 -- 172.21.15.28:0/767943655 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc25c062070 con 0x56162f89b860 2024-04-23T14:44:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.638+0000 7fc27d073740 1 -- 172.21.15.28:0/767943655 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc264009820 con 0x56162f89b860 2024-04-23T14:44:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.638+0000 7fc260c00640 1 -- 172.21.15.28:0/767943655 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fc25c0037f0 con 0x56162f89b860 2024-04-23T14:44:39.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.639+0000 7fc260c00640 1 -- 172.21.15.28:0/767943655 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fc25c06e050 con 0x56162f89b860 2024-04-23T14:44:39.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.639+0000 7fc27d073740 10 client.25385 fetch_fsmap learned FSMap version 20 2024-04-23T14:44:39.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.639+0000 7fc27d073740 1 -- 172.21.15.28:0/767943655 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x56162f7c6810 con 0x56162f89b860 2024-04-23T14:44:39.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.639+0000 7fc260c00640 1 -- 172.21.15.28:0/767943655 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc264009820 con 0x56162f89b860 2024-04-23T14:44:39.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.639+0000 7fc260c00640 1 -- 172.21.15.28:0/767943655 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fc25c032bd0 con 0x56162f89b860 2024-04-23T14:44:39.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.639+0000 7fc27d073740 10 client.25385 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:44:39.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.639+0000 7fc27d073740 10 client.25385 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:44:39.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.639+0000 7fc27d073740 20 client.25385 populate_metadata read hostname 'smithi028' 2024-04-23T14:44:39.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.639+0000 7fc27d073740 1 --2- 172.21.15.28:0/767943655 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56162f91e230 0x56162f920650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:39.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.639+0000 7fc27d073740 4 client.25385 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:44:39.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.639+0000 7fc27d073740 1 -- 172.21.15.28:0/767943655 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56162f808a80 con 0x56162f91e230 2024-04-23T14:44:39.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.640+0000 7fc272a00640 1 --2- 172.21.15.28:0/767943655 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56162f91e230 0x56162f920650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:39.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.640+0000 7fc272a00640 1 --2- 172.21.15.28:0/767943655 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56162f91e230 0x56162f920650 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47b85eb913c55ca4 server_cookie=5f1f3e40fbe4a9f5 in_seq=0 out_seq=0 2024-04-23T14:44:39.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.640+0000 7fc260c00640 10 client.25385 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:44:39.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.640+0000 7fc260c00640 1 -- 172.21.15.28:0/767943655 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56162f808a80 con 0x56162f91e230 2024-04-23T14:44:39.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.640+0000 7fc260c00640 10 client.25385 handle_command_reply: tid=0 2024-04-23T14:44:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.640+0000 7fc27d073740 1 client.25385 shutdown 2024-04-23T14:44:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.641+0000 7fc27d073740 1 -- 172.21.15.28:0/767943655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56162f89b860 msgr2=0x56162f8aec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.641+0000 7fc27d073740 1 --2- 172.21.15.28:0/767943655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56162f89b860 0x56162f8aec30 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc25c031550 tx=0x7fc25c0045b0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.641+0000 7fc27d073740 1 -- 172.21.15.28:0/767943655 shutdown_connections 2024-04-23T14:44:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.641+0000 7fc27d073740 1 --2- 172.21.15.28:0/767943655 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56162f91e230 0x56162f920650 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.641+0000 7fc27d073740 1 --2- 172.21.15.28:0/767943655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162f8a6850 0x56162f8abf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.641+0000 7fc27d073740 1 --2- 172.21.15.28:0/767943655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56162f89c180 0x56162f8aba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.641+0000 7fc27d073740 1 --2- 172.21.15.28:0/767943655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56162f89b860 0x56162f8aec30 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.641+0000 7fc260c00640 0 client.25385 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:44:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.641+0000 7fc27d073740 1 -- 172.21.15.28:0/767943655 >> 172.21.15.28:0/767943655 conn(0x56162f897b80 msgr2=0x56162f8a3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.641+0000 7fc27d073740 1 -- 172.21.15.28:0/767943655 shutdown_connections 2024-04-23T14:44:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.641+0000 7fc27d073740 1 -- 172.21.15.28:0/767943655 wait complete. 2024-04-23T14:44:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.641+0000 7fc27d073740 20 client.25385 trim_cache size 0 max 16384 2024-04-23T14:44:39.646 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:44:39.646 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-23T14:44:39.646 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-23T14:44:39.646 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:44:39.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.642+0000 7fc27a200640 1 -- 172.21.15.28:0/3169053397 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc2580a3e30 msgr2=0x7fc2580a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:39.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.642+0000 7fc27a200640 1 --2- 172.21.15.28:0/3169053397 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc2580a3e30 0x7fc2580a62f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc25c000c00 tx=0x7fc25c005b50 comp rx=0 tx=0).stop 2024-04-23T14:44:39.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.642+0000 7fc27a200640 1 -- 172.21.15.28:0/3169053397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc274074cc0 msgr2=0x7fc27419f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:39.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.642+0000 7fc27a200640 1 --2- 172.21.15.28:0/3169053397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc274074cc0 0x7fc27419f460 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc264009d50 tx=0x7fc26400c6a0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.642+0000 7fc27a200640 1 -- 172.21.15.28:0/3169053397 shutdown_connections 2024-04-23T14:44:39.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.642+0000 7fc27a200640 1 --2- 172.21.15.28:0/3169053397 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc2580a3e30 0x7fc2580a62f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.642+0000 7fc27a200640 1 --2- 172.21.15.28:0/3169053397 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc274112fd0 0x7fc2741a37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.642+0000 7fc27a200640 1 --2- 172.21.15.28:0/3169053397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc274074cc0 0x7fc27419f460 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.642+0000 7fc27a200640 1 --2- 172.21.15.28:0/3169053397 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2740743a0 0x7fc27419ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.642+0000 7fc27a200640 1 -- 172.21.15.28:0/3169053397 >> 172.21.15.28:0/3169053397 conn(0x7fc2740fda70 msgr2=0x7fc2740ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:39.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.642+0000 7fc27a200640 1 -- 172.21.15.28:0/3169053397 shutdown_connections 2024-04-23T14:44:39.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.642+0000 7fc27a200640 1 -- 172.21.15.28:0/3169053397 wait complete. 2024-04-23T14:44:39.659 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-23T14:44:39.659 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:44:39.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.848+0000 7fdc5ee00640 1 -- 172.21.15.28:0/2998039363 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdc58104c10 msgr2=0x7fdc58108c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:39.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.848+0000 7fdc5ee00640 1 --2- 172.21.15.28:0/2998039363 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdc58104c10 0x7fdc58108c50 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fdc48009900 tx=0x7fdc4802a3f0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.848+0000 7fdc5ee00640 1 -- 172.21.15.28:0/2998039363 shutdown_connections 2024-04-23T14:44:39.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.848+0000 7fdc5ee00640 1 --2- 172.21.15.28:0/2998039363 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdc58104c10 0x7fdc58108c50 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.848+0000 7fdc5ee00640 1 --2- 172.21.15.28:0/2998039363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc58109330 0x7fdc58110f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.848+0000 7fdc5ee00640 1 --2- 172.21.15.28:0/2998039363 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc58104260 0x7fdc58104640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.848+0000 7fdc5ee00640 1 -- 172.21.15.28:0/2998039363 >> 172.21.15.28:0/2998039363 conn(0x7fdc580ffaa0 msgr2=0x7fdc58101ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:39.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.848+0000 7fdc5ee00640 1 -- 172.21.15.28:0/2998039363 shutdown_connections 2024-04-23T14:44:39.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.848+0000 7fdc5ee00640 1 -- 172.21.15.28:0/2998039363 wait complete. 2024-04-23T14:44:39.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.850+0000 7fdc5ee00640 1 Processor -- start 2024-04-23T14:44:39.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.850+0000 7fdc5ee00640 1 -- start start 2024-04-23T14:44:39.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.850+0000 7fdc5ee00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdc58104260 0x7fdc581a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:39.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.850+0000 7fdc5ee00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc58109330 0x7fdc581a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:39.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.850+0000 7fdc5ee00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc5819a770 0x7fdc5819abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:39.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.850+0000 7fdc5ee00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdc581143f0 con 0x7fdc5819a770 2024-04-23T14:44:39.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.850+0000 7fdc5ee00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdc58114270 con 0x7fdc58104260 2024-04-23T14:44:39.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.850+0000 7fdc5ee00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdc58114570 con 0x7fdc58109330 2024-04-23T14:44:39.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.851+0000 7fdc5da00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdc58104260 0x7fdc581a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:39.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.851+0000 7fdc5d000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc58109330 0x7fdc581a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:39.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.851+0000 7fdc5d000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc58109330 0x7fdc581a0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:44296/0 (socket says 172.21.15.28:44296) 2024-04-23T14:44:39.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.851+0000 7fdc5d000640 1 -- 172.21.15.28:0/3267071323 learned_addr learned my addr 172.21.15.28:0/3267071323 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:39.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.851+0000 7fdc5e400640 1 --2- 172.21.15.28:0/3267071323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc5819a770 0x7fdc5819abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:39.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.851+0000 7fdc5da00640 1 -- 172.21.15.28:0/3267071323 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc58109330 msgr2=0x7fdc581a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:39.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.851+0000 7fdc5da00640 1 --2- 172.21.15.28:0/3267071323 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc58109330 0x7fdc581a0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.851+0000 7fdc5da00640 1 -- 172.21.15.28:0/3267071323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc5819a770 msgr2=0x7fdc5819abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:39.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.851+0000 7fdc5da00640 1 --2- 172.21.15.28:0/3267071323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc5819a770 0x7fdc5819abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:39.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.851+0000 7fdc5da00640 1 -- 172.21.15.28:0/3267071323 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc5819b370 con 0x7fdc58104260 2024-04-23T14:44:39.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.851+0000 7fdc5d000640 1 --2- 172.21.15.28:0/3267071323 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc58109330 0x7fdc581a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:44:39.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.852+0000 7fdc5da00640 1 --2- 172.21.15.28:0/3267071323 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdc58104260 0x7fdc581a0660 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c00e9f0 tx=0x7fdc4c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:39.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.852+0000 7fdc56000640 1 -- 172.21.15.28:0/3267071323 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c00ccc0 con 0x7fdc58104260 2024-04-23T14:44:39.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.852+0000 7fdc5ee00640 1 -- 172.21.15.28:0/3267071323 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc581a7400 con 0x7fdc58104260 2024-04-23T14:44:39.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.852+0000 7fdc56000640 1 -- 172.21.15.28:0/3267071323 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc4c010910 con 0x7fdc58104260 2024-04-23T14:44:39.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.852+0000 7fdc56000640 1 -- 172.21.15.28:0/3267071323 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c018790 con 0x7fdc58104260 2024-04-23T14:44:39.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.852+0000 7fdc5ee00640 1 -- 172.21.15.28:0/3267071323 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc581a7860 con 0x7fdc58104260 2024-04-23T14:44:39.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.853+0000 7fdc5ee00640 1 -- 172.21.15.28:0/3267071323 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc58105730 con 0x7fdc58104260 2024-04-23T14:44:39.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.856+0000 7fdc56000640 1 -- 172.21.15.28:0/3267071323 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7fdc4c010ac0 con 0x7fdc58104260 2024-04-23T14:44:39.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.857+0000 7fdc56000640 1 --2- 172.21.15.28:0/3267071323 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fdc400a3db0 0x7fdc400a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:39.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.857+0000 7fdc56000640 1 -- 172.21.15.28:0/3267071323 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fdc4c0d1220 con 0x7fdc58104260 2024-04-23T14:44:39.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.857+0000 7fdc5d000640 1 --2- 172.21.15.28:0/3267071323 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fdc400a3db0 0x7fdc400a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:39.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.858+0000 7fdc5d000640 1 --2- 172.21.15.28:0/3267071323 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fdc400a3db0 0x7fdc400a6270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdc5819bf70 tx=0x7fdc48031510 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:39.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:39.860+0000 7fdc56000640 1 -- 172.21.15.28:0/3267071323 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdc4c09d700 con 0x7fdc58104260 2024-04-23T14:44:40.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.110+0000 7fdc5ee00640 1 -- 172.21.15.28:0/3267071323 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdc58104c10 con 0x7fdc58104260 2024-04-23T14:44:40.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.111+0000 7fdc56000640 1 -- 172.21.15.28:0/3267071323 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7fdc4c09d000 con 0x7fdc58104260 2024-04-23T14:44:40.112 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:44:40.112 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:44:40.112 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:44:40.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.114+0000 7fdc5ee00640 1 -- 172.21.15.28:0/3267071323 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fdc400a3db0 msgr2=0x7fdc400a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:40.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.114+0000 7fdc5ee00640 1 --2- 172.21.15.28:0/3267071323 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fdc400a3db0 0x7fdc400a6270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdc5819bf70 tx=0x7fdc48031510 comp rx=0 tx=0).stop 2024-04-23T14:44:40.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.114+0000 7fdc5ee00640 1 -- 172.21.15.28:0/3267071323 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdc58104260 msgr2=0x7fdc581a0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:40.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.114+0000 7fdc5ee00640 1 --2- 172.21.15.28:0/3267071323 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdc58104260 0x7fdc581a0660 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c00e9f0 tx=0x7fdc4c00edf0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.114+0000 7fdc5ee00640 1 -- 172.21.15.28:0/3267071323 shutdown_connections 2024-04-23T14:44:40.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.114+0000 7fdc5ee00640 1 --2- 172.21.15.28:0/3267071323 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fdc400a3db0 0x7fdc400a6270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.114+0000 7fdc5ee00640 1 --2- 172.21.15.28:0/3267071323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc5819a770 0x7fdc5819abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.114+0000 7fdc5ee00640 1 --2- 172.21.15.28:0/3267071323 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc58109330 0x7fdc581a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.114+0000 7fdc5ee00640 1 --2- 172.21.15.28:0/3267071323 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdc58104260 0x7fdc581a0660 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.114+0000 7fdc5ee00640 1 -- 172.21.15.28:0/3267071323 >> 172.21.15.28:0/3267071323 conn(0x7fdc580ffaa0 msgr2=0x7fdc58100db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:40.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.115+0000 7fdc5ee00640 1 -- 172.21.15.28:0/3267071323 shutdown_connections 2024-04-23T14:44:40.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.115+0000 7fdc5ee00640 1 -- 172.21.15.28:0/3267071323 wait complete. 2024-04-23T14:44:40.135 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-23T14:44:40.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.351+0000 7fcfe9600640 1 -- 172.21.15.28:0/3200576293 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfe4104450 msgr2=0x7fcfe4104830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:40.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.351+0000 7fcfe9600640 1 --2- 172.21.15.28:0/3200576293 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfe4104450 0x7fcfe4104830 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4009900 tx=0x7fcfd402a3f0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.351+0000 7fcfe9600640 1 -- 172.21.15.28:0/3200576293 shutdown_connections 2024-04-23T14:44:40.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.351+0000 7fcfe9600640 1 --2- 172.21.15.28:0/3200576293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfe4109520 0x7fcfe41110f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.351+0000 7fcfe9600640 1 --2- 172.21.15.28:0/3200576293 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfe4104e00 0x7fcfe4108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.351+0000 7fcfe9600640 1 --2- 172.21.15.28:0/3200576293 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfe4104450 0x7fcfe4104830 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.351+0000 7fcfe9600640 1 -- 172.21.15.28:0/3200576293 >> 172.21.15.28:0/3200576293 conn(0x7fcfe40ffc70 msgr2=0x7fcfe4102090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:40.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.351+0000 7fcfe9600640 1 -- 172.21.15.28:0/3200576293 shutdown_connections 2024-04-23T14:44:40.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.352+0000 7fcfe9600640 1 -- 172.21.15.28:0/3200576293 wait complete. 2024-04-23T14:44:40.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.353+0000 7fcfe9600640 1 Processor -- start 2024-04-23T14:44:40.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.353+0000 7fcfe9600640 1 -- start start 2024-04-23T14:44:40.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.353+0000 7fcfe9600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfe4104450 0x7fcfe41a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:40.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.353+0000 7fcfe9600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfe4109520 0x7fcfe41a0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:40.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.353+0000 7fcfe9600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfe419a9c0 0x7fcfe419ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:40.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.353+0000 7fcfe9600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe4114600 con 0x7fcfe4104450 2024-04-23T14:44:40.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.353+0000 7fcfe9600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe4114480 con 0x7fcfe4109520 2024-04-23T14:44:40.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.353+0000 7fcfe9600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe4114780 con 0x7fcfe419a9c0 2024-04-23T14:44:40.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.354+0000 7fcfe2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfe4104450 0x7fcfe41a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:40.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.354+0000 7fcfe2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfe4104450 0x7fcfe41a07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43714/0 (socket says 172.21.15.28:43714) 2024-04-23T14:44:40.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.354+0000 7fcfe2a00640 1 -- 172.21.15.28:0/3753737509 learned_addr learned my addr 172.21.15.28:0/3753737509 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:40.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.354+0000 7fcfe3400640 1 --2- 172.21.15.28:0/3753737509 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfe419a9c0 0x7fcfe419ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:40.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.354+0000 7fcfda000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfe4109520 0x7fcfe41a0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:40.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.354+0000 7fcfe3400640 1 -- 172.21.15.28:0/3753737509 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfe4109520 msgr2=0x7fcfe41a0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:40.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.354+0000 7fcfe3400640 1 --2- 172.21.15.28:0/3753737509 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfe4109520 0x7fcfe41a0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.354+0000 7fcfe3400640 1 -- 172.21.15.28:0/3753737509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfe4104450 msgr2=0x7fcfe41a07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:40.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.354+0000 7fcfe3400640 1 --2- 172.21.15.28:0/3753737509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfe4104450 0x7fcfe41a07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.354+0000 7fcfe3400640 1 -- 172.21.15.28:0/3753737509 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfe419b4e0 con 0x7fcfe419a9c0 2024-04-23T14:44:40.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.354+0000 7fcfe3400640 1 --2- 172.21.15.28:0/3753737509 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfe419a9c0 0x7fcfe419ae70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc009fd0 tx=0x7fcfdc00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:40.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.355+0000 7fcfdbe00640 1 -- 172.21.15.28:0/3753737509 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc00ce20 con 0x7fcfe419a9c0 2024-04-23T14:44:40.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.355+0000 7fcfdbe00640 1 -- 172.21.15.28:0/3753737509 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfdc005020 con 0x7fcfe419a9c0 2024-04-23T14:44:40.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.355+0000 7fcfdbe00640 1 -- 172.21.15.28:0/3753737509 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc010720 con 0x7fcfe419a9c0 2024-04-23T14:44:40.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.355+0000 7fcfe9600640 1 -- 172.21.15.28:0/3753737509 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfe419b7d0 con 0x7fcfe419a9c0 2024-04-23T14:44:40.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.355+0000 7fcfe9600640 1 -- 172.21.15.28:0/3753737509 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfe41a7980 con 0x7fcfe419a9c0 2024-04-23T14:44:40.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.359+0000 7fcfec3f3740 1 Processor -- start 2024-04-23T14:44:40.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.360+0000 7fcfdbe00640 1 -- 172.21.15.28:0/3753737509 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7fcfdc005540 con 0x7fcfe419a9c0 2024-04-23T14:44:40.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.360+0000 7fcfec3f3740 1 -- start start 2024-04-23T14:44:40.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.360+0000 7fcfec3f3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7a8b31950 0x55c7a8b31d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:40.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.360+0000 7fcfec3f3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7a8b32270 0x55c7a8b3c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:40.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.360+0000 7fcfec3f3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a8b3c940 0x55c7a8b3ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:40.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.360+0000 7fcfec3f3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c7a8a971d0 con 0x55c7a8b31950 2024-04-23T14:44:40.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.360+0000 7fcfec3f3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c7a8a3d450 con 0x55c7a8b3c940 2024-04-23T14:44:40.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.360+0000 7fcfec3f3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c7a8a93190 con 0x55c7a8b32270 2024-04-23T14:44:40.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.361+0000 7fcfe2a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7a8b32270 0x55c7a8b3c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:40.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.361+0000 7fcfda000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7a8b31950 0x55c7a8b31d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:40.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.361+0000 7fcfe2a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7a8b32270 0x55c7a8b3c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:44308/0 (socket says 172.21.15.28:44308) 2024-04-23T14:44:40.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.361+0000 7fcfe2a00640 1 -- 172.21.15.28:0/3814575160 learned_addr learned my addr 172.21.15.28:0/3814575160 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:40.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.361+0000 7fcfda000640 1 --2- 172.21.15.28:0/3814575160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a8b3c940 0x55c7a8b3ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:40.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.361+0000 7fcfe2a00640 1 -- 172.21.15.28:0/3814575160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a8b3c940 msgr2=0x55c7a8b3ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:40.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.361+0000 7fcfe2a00640 1 --2- 172.21.15.28:0/3814575160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a8b3c940 0x55c7a8b3ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.361+0000 7fcfe2a00640 1 -- 172.21.15.28:0/3814575160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7a8b31950 msgr2=0x55c7a8b31d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:40.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.361+0000 7fcfe2a00640 1 --2- 172.21.15.28:0/3814575160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7a8b31950 0x55c7a8b31d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.361+0000 7fcfe2a00640 1 -- 172.21.15.28:0/3814575160 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7a8afa310 con 0x55c7a8b32270 2024-04-23T14:44:40.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.361+0000 7fcfda000640 1 --2- 172.21.15.28:0/3814575160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a8b3c940 0x55c7a8b3ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:44:40.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.361+0000 7fcfe2a00640 1 --2- 172.21.15.28:0/3814575160 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7a8b32270 0x55c7a8b3c400 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4006010 tx=0x7fcfd4005dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b8cb46b36ffe9f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:40.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.362+0000 7fcfdbe00640 1 --2- 172.21.15.28:0/3753737509 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fcfc80a3db0 0x7fcfc80a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:40.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.362+0000 7fcfd3400640 1 -- 172.21.15.28:0/3814575160 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd4040070 con 0x55c7a8b32270 2024-04-23T14:44:40.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.362+0000 7fcfd3400640 1 -- 172.21.15.28:0/3814575160 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfd4004400 con 0x55c7a8b32270 2024-04-23T14:44:40.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.362+0000 7fcfd3400640 1 -- 172.21.15.28:0/3814575160 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd4035040 con 0x55c7a8b32270 2024-04-23T14:44:40.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.362+0000 7fcfdbe00640 1 -- 172.21.15.28:0/3753737509 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fcfdc0d1060 con 0x7fcfe419a9c0 2024-04-23T14:44:40.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.362+0000 7fcfda000640 1 --2- 172.21.15.28:0/3753737509 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fcfc80a3db0 0x7fcfc80a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:40.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.363+0000 7fcfec3f3740 1 -- 172.21.15.28:0/3814575160 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7a8b32270 msgr2=0x55c7a8b3c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:40.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.363+0000 7fcfec3f3740 1 --2- 172.21.15.28:0/3814575160 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7a8b32270 0x55c7a8b3c400 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4006010 tx=0x7fcfd4005dd0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.363+0000 7fcfda000640 1 --2- 172.21.15.28:0/3753737509 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fcfc80a3db0 0x7fcfc80a6270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x55c7a8b2ee00 tx=0x7fcfcc005850 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:40.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.363+0000 7fcfec3f3740 1 -- 172.21.15.28:0/3814575160 shutdown_connections 2024-04-23T14:44:40.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.363+0000 7fcfec3f3740 1 --2- 172.21.15.28:0/3814575160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a8b3c940 0x55c7a8b3ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.363+0000 7fcfec3f3740 1 --2- 172.21.15.28:0/3814575160 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7a8b32270 0x55c7a8b3c400 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.363+0000 7fcfec3f3740 1 --2- 172.21.15.28:0/3814575160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7a8b31950 0x55c7a8b31d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.363+0000 7fcfec3f3740 1 -- 172.21.15.28:0/3814575160 >> 172.21.15.28:0/3814575160 conn(0x55c7a8b2dc70 msgr2=0x55c7a8b2e050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:40.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.363+0000 7fcfec3f3740 1 -- 172.21.15.28:0/3814575160 shutdown_connections 2024-04-23T14:44:40.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.363+0000 7fcfec3f3740 1 -- 172.21.15.28:0/3814575160 wait complete. 2024-04-23T14:44:40.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.363+0000 7fcfec3f3740 1 -- start start 2024-04-23T14:44:40.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.364+0000 7fcfec3f3740 1 Processor -- start 2024-04-23T14:44:40.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.364+0000 7fcfec3f3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a8b31950 0x55c7a8b3c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:40.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.364+0000 7fcfec3f3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7a8b32270 0x55c7a8b39690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:40.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.364+0000 7fcfec3f3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7a8b3c940 0x55c7a8b39db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:40.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.364+0000 7fcfec3f3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c7a8b3f770 con 0x55c7a8b3c940 2024-04-23T14:44:40.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.364+0000 7fcfec3f3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c7a8a50810 con 0x55c7a8b31950 2024-04-23T14:44:40.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.364+0000 7fcfec3f3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c7a8b3f8f0 con 0x55c7a8b32270 2024-04-23T14:44:40.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.364+0000 7fcfec3f3740 10 client.0 _ll_register_callbacks cb 0x55c7a8a1e140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:44:40.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.364+0000 7fcfe2a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a8b31950 0x55c7a8b3c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:40.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.364+0000 7fcfe2a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a8b31950 0x55c7a8b3c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:42982/0 (socket says 172.21.15.28:42982) 2024-04-23T14:44:40.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.364+0000 7fcfe2a00640 1 -- 172.21.15.28:0/3263602130 learned_addr learned my addr 172.21.15.28:0/3263602130 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:40.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.364+0000 7fcfda000640 1 --2- 172.21.15.28:0/3263602130 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7a8b32270 0x55c7a8b39690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:40.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.364+0000 7fcfe3400640 1 --2- 172.21.15.28:0/3263602130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7a8b3c940 0x55c7a8b39db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:40.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.365+0000 7fcfe2a00640 1 -- 172.21.15.28:0/3263602130 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7a8b32270 msgr2=0x55c7a8b39690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:40.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.365+0000 7fcfe2a00640 1 --2- 172.21.15.28:0/3263602130 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7a8b32270 0x55c7a8b39690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.365+0000 7fcfe2a00640 1 -- 172.21.15.28:0/3263602130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7a8b3c940 msgr2=0x55c7a8b39db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:40.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.365+0000 7fcfe2a00640 1 --2- 172.21.15.28:0/3263602130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7a8b3c940 0x55c7a8b39db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.365+0000 7fcfe2a00640 1 -- 172.21.15.28:0/3263602130 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7a8a613f0 con 0x55c7a8b31950 2024-04-23T14:44:40.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.365+0000 7fcfe2a00640 1 --2- 172.21.15.28:0/3263602130 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a8b31950 0x55c7a8b3c3a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4005040 tx=0x7fcfd4004990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d26519948d6eb7d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:40.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.365+0000 7fcfc7e00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:44:40.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.365+0000 7fcfc7e00640 1 -- 172.21.15.28:0/3263602130 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd4035070 con 0x55c7a8b31950 2024-04-23T14:44:40.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.365+0000 7fcfc7e00640 1 -- 172.21.15.28:0/3263602130 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fcfd4005430 con 0x55c7a8b31950 2024-04-23T14:44:40.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.365+0000 7fcfec3f3740 1 -- 172.21.15.28:0/3263602130 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfd4004400 con 0x55c7a8b31950 2024-04-23T14:44:40.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.366+0000 7fcfc7e00640 1 -- 172.21.15.28:0/3263602130 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fcfd4040040 con 0x55c7a8b31950 2024-04-23T14:44:40.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.366+0000 7fcfec3f3740 10 client.25400 fetch_fsmap learned FSMap version 20 2024-04-23T14:44:40.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.366+0000 7fcfec3f3740 1 -- 172.21.15.28:0/3263602130 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55c7a87a93f0 con 0x55c7a8b31950 2024-04-23T14:44:40.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.366+0000 7fcfc7e00640 1 -- 172.21.15.28:0/3263602130 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfd4004400 con 0x55c7a8b31950 2024-04-23T14:44:40.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.366+0000 7fcfc7e00640 1 -- 172.21.15.28:0/3263602130 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fcfd4031c20 con 0x55c7a8b31950 2024-04-23T14:44:40.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.366+0000 7fcfec3f3740 10 client.25400 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:44:40.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.366+0000 7fcfec3f3740 10 client.25400 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:44:40.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.366+0000 7fcfec3f3740 20 client.25400 populate_metadata read hostname 'smithi028' 2024-04-23T14:44:40.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.367+0000 7fcfec3f3740 1 --2- 172.21.15.28:0/3263602130 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c7a8b4c900 0x55c7a8b4ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:40.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.367+0000 7fcfec3f3740 4 client.25400 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:44:40.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.367+0000 7fcfec3f3740 1 -- 172.21.15.28:0/3263602130 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c7a8aa4bf0 con 0x55c7a8b4c900 2024-04-23T14:44:40.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.367+0000 7fcfda000640 1 --2- 172.21.15.28:0/3263602130 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c7a8b4c900 0x55c7a8b4ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:40.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.367+0000 7fcfda000640 1 --2- 172.21.15.28:0/3263602130 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c7a8b4c900 0x55c7a8b4ecf0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a22b750b1d77ad1 server_cookie=c915b895bdbf3a77 in_seq=0 out_seq=0 2024-04-23T14:44:40.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.367+0000 7fcfc7e00640 10 client.25400 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:44:40.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.368+0000 7fcfc7e00640 1 -- 172.21.15.28:0/3263602130 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c7a8aa4bf0 con 0x55c7a8b4c900 2024-04-23T14:44:40.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.368+0000 7fcfc7e00640 10 client.25400 handle_command_reply: tid=0 2024-04-23T14:44:40.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.368+0000 7fcfec3f3740 1 client.25400 shutdown 2024-04-23T14:44:40.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.369+0000 7fcfec3f3740 1 -- 172.21.15.28:0/3263602130 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a8b31950 msgr2=0x55c7a8b3c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:40.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.369+0000 7fcfec3f3740 1 --2- 172.21.15.28:0/3263602130 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a8b31950 0x55c7a8b3c3a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4005040 tx=0x7fcfd4004990 comp rx=0 tx=0).stop 2024-04-23T14:44:40.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.369+0000 7fcfec3f3740 1 -- 172.21.15.28:0/3263602130 shutdown_connections 2024-04-23T14:44:40.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.369+0000 7fcfec3f3740 1 --2- 172.21.15.28:0/3263602130 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c7a8b4c900 0x55c7a8b4ecf0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.369+0000 7fcfec3f3740 1 --2- 172.21.15.28:0/3263602130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7a8b3c940 0x55c7a8b39db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.369+0000 7fcfec3f3740 1 --2- 172.21.15.28:0/3263602130 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7a8b32270 0x55c7a8b39690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.369+0000 7fcfec3f3740 1 --2- 172.21.15.28:0/3263602130 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a8b31950 0x55c7a8b3c3a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.369+0000 7fcfec3f3740 1 -- 172.21.15.28:0/3263602130 >> 172.21.15.28:0/3263602130 conn(0x55c7a8b2dc70 msgr2=0x55c7a8b33930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:40.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.369+0000 7fcfc7e00640 0 client.25400 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:44:40.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.369+0000 7fcfec3f3740 1 -- 172.21.15.28:0/3263602130 shutdown_connections 2024-04-23T14:44:40.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.369+0000 7fcfec3f3740 1 -- 172.21.15.28:0/3263602130 wait complete. 2024-04-23T14:44:40.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.369+0000 7fcfec3f3740 20 client.25400 trim_cache size 0 max 16384 2024-04-23T14:44:40.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.384+0000 7fcfec3f3740 1 Processor -- start 2024-04-23T14:44:40.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.384+0000 7fcfec3f3740 1 -- start start 2024-04-23T14:44:40.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.384+0000 7fcfec3f3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7a8b31950 0x55c7a8b48660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:40.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.384+0000 7fcfec3f3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a8b32270 0x55c7a8b3ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:40.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.384+0000 7fcfec3f3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7a8b3c940 0x55c7a8b3bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:40.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.384+0000 7fcfec3f3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe41adf40 con 0x55c7a8b31950 2024-04-23T14:44:40.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.384+0000 7fcfec3f3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe41b07c0 con 0x55c7a8b32270 2024-04-23T14:44:40.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.384+0000 7fcfec3f3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe41ae5b0 con 0x55c7a8b3c940 2024-04-23T14:44:40.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.384+0000 7fcfe2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7a8b31950 0x55c7a8b48660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:40.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.385+0000 7fcfda000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a8b32270 0x55c7a8b3ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:40.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.385+0000 7fcfe2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7a8b31950 0x55c7a8b48660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43750/0 (socket says 172.21.15.28:43750) 2024-04-23T14:44:40.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.385+0000 7fcfe2a00640 1 -- 172.21.15.28:0/2607623789 learned_addr learned my addr 172.21.15.28:0/2607623789 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:40.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.385+0000 7fcfe3400640 1 --2- 172.21.15.28:0/2607623789 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7a8b3c940 0x55c7a8b3bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:40.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.385+0000 7fcfe2a00640 1 -- 172.21.15.28:0/2607623789 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7a8b3c940 msgr2=0x55c7a8b3bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:40.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.385+0000 7fcfe2a00640 1 --2- 172.21.15.28:0/2607623789 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7a8b3c940 0x55c7a8b3bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.385+0000 7fcfe2a00640 1 -- 172.21.15.28:0/2607623789 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a8b32270 msgr2=0x55c7a8b3ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:40.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.385+0000 7fcfe2a00640 1 --2- 172.21.15.28:0/2607623789 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a8b32270 0x55c7a8b3ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.385+0000 7fcfe2a00640 1 -- 172.21.15.28:0/2607623789 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7a8b320a0 con 0x55c7a8b31950 2024-04-23T14:44:40.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.385+0000 7fcfda000640 1 --2- 172.21.15.28:0/2607623789 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a8b32270 0x55c7a8b3ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:44:40.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.385+0000 7fcfe2a00640 1 --2- 172.21.15.28:0/2607623789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7a8b31950 0x55c7a8b48660 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcfd402a9e0 tx=0x7fcfd4005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c7daf6d4d1a7f80 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:40.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.386+0000 7fcfd2a00640 1 -- 172.21.15.28:0/2607623789 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd4035070 con 0x55c7a8b31950 2024-04-23T14:44:40.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.386+0000 7fcfd2a00640 1 -- 172.21.15.28:0/2607623789 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfd4009520 con 0x55c7a8b31950 2024-04-23T14:44:40.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.386+0000 7fcfd2a00640 1 -- 172.21.15.28:0/2607623789 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd4003ec0 con 0x55c7a8b31950 2024-04-23T14:44:40.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.386+0000 7fcfec3f3740 1 -- 172.21.15.28:0/2607623789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7a8b31950 msgr2=0x55c7a8b48660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:40.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.386+0000 7fcfec3f3740 1 --2- 172.21.15.28:0/2607623789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7a8b31950 0x55c7a8b48660 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcfd402a9e0 tx=0x7fcfd4005fd0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.386+0000 7fcfec3f3740 1 -- 172.21.15.28:0/2607623789 shutdown_connections 2024-04-23T14:44:40.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.386+0000 7fcfec3f3740 1 --2- 172.21.15.28:0/2607623789 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a8b32270 0x55c7a8b3ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.386+0000 7fcfec3f3740 1 --2- 172.21.15.28:0/2607623789 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7a8b3c940 0x55c7a8b3bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.386+0000 7fcfec3f3740 1 --2- 172.21.15.28:0/2607623789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7a8b31950 0x55c7a8b48660 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.386+0000 7fcfec3f3740 1 -- 172.21.15.28:0/2607623789 >> 172.21.15.28:0/2607623789 conn(0x55c7a8b2dc70 msgr2=0x55c7a8b2c560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:40.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.386+0000 7fcfec3f3740 1 -- 172.21.15.28:0/2607623789 shutdown_connections 2024-04-23T14:44:40.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.386+0000 7fcfec3f3740 1 -- 172.21.15.28:0/2607623789 wait complete. 2024-04-23T14:44:40.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.387+0000 7fcfec3f3740 1 -- start start 2024-04-23T14:44:40.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.387+0000 7fcfec3f3740 1 Processor -- start 2024-04-23T14:44:40.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.387+0000 7fcfec3f3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7a8b31950 0x55c7a8b3d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:40.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.387+0000 7fcfec3f3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7a8b32270 0x55c7a8b3da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:40.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.388+0000 7fcfec3f3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a8b3c940 0x55c7a8b43b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:40.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.388+0000 7fcfec3f3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe41aec80 con 0x55c7a8b31950 2024-04-23T14:44:40.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.388+0000 7fcfec3f3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe41b00f0 con 0x55c7a8b3c940 2024-04-23T14:44:40.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.388+0000 7fcfec3f3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe41af350 con 0x55c7a8b32270 2024-04-23T14:44:40.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.388+0000 7fcfec3f3740 10 client.0 _ll_register_callbacks cb 0x7fcfcc0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:44:40.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.388+0000 7fcfe2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7a8b31950 0x55c7a8b3d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:40.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.388+0000 7fcfda000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7a8b32270 0x55c7a8b3da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:40.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.388+0000 7fcfda000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7a8b32270 0x55c7a8b3da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:44328/0 (socket says 172.21.15.28:44328) 2024-04-23T14:44:40.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.388+0000 7fcfda000640 1 -- 172.21.15.28:0/1968595061 learned_addr learned my addr 172.21.15.28:0/1968595061 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:40.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.388+0000 7fcfe3400640 1 --2- 172.21.15.28:0/1968595061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a8b3c940 0x55c7a8b43b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:40.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.388+0000 7fcfda000640 1 -- 172.21.15.28:0/1968595061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a8b3c940 msgr2=0x55c7a8b43b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:40.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.388+0000 7fcfda000640 1 --2- 172.21.15.28:0/1968595061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a8b3c940 0x55c7a8b43b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.388+0000 7fcfda000640 1 -- 172.21.15.28:0/1968595061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7a8b31950 msgr2=0x55c7a8b3d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:40.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.388+0000 7fcfda000640 1 --2- 172.21.15.28:0/1968595061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7a8b31950 0x55c7a8b3d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.388+0000 7fcfda000640 1 -- 172.21.15.28:0/1968595061 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7a88737d0 con 0x55c7a8b32270 2024-04-23T14:44:40.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.388+0000 7fcfe2a00640 1 --2- 172.21.15.28:0/1968595061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7a8b31950 0x55c7a8b3d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:44:40.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.389+0000 7fcfda000640 1 --2- 172.21.15.28:0/1968595061 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7a8b32270 0x55c7a8b3da20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc0042f0 tx=0x7fcfcc004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6b910c5651766be server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:40.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.389+0000 7fcfc7e00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:44:40.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.389+0000 7fcfc7e00640 1 -- 172.21.15.28:0/1968595061 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc030070 con 0x55c7a8b32270 2024-04-23T14:44:40.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.389+0000 7fcfc7e00640 1 -- 172.21.15.28:0/1968595061 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fcfcc020a80 con 0x55c7a8b32270 2024-04-23T14:44:40.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.389+0000 7fcfec3f3740 1 -- 172.21.15.28:0/1968595061 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfd4009520 con 0x55c7a8b32270 2024-04-23T14:44:40.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.389+0000 7fcfc7e00640 1 -- 172.21.15.28:0/1968595061 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fcfcc03c050 con 0x55c7a8b32270 2024-04-23T14:44:40.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.389+0000 7fcfec3f3740 10 client.25414 fetch_fsmap learned FSMap version 20 2024-04-23T14:44:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.389+0000 7fcfec3f3740 1 -- 172.21.15.28:0/1968595061 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55c7a8b45070 con 0x55c7a8b32270 2024-04-23T14:44:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.389+0000 7fcfc7e00640 1 -- 172.21.15.28:0/1968595061 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfd4009520 con 0x55c7a8b32270 2024-04-23T14:44:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.390+0000 7fcfc7e00640 1 -- 172.21.15.28:0/1968595061 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fcfcc003bc0 con 0x55c7a8b32270 2024-04-23T14:44:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.390+0000 7fcfec3f3740 10 client.25414 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:44:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.390+0000 7fcfec3f3740 10 client.25414 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:44:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.390+0000 7fcfec3f3740 20 client.25414 populate_metadata read hostname 'smithi028' 2024-04-23T14:44:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.390+0000 7fcfec3f3740 1 --2- 172.21.15.28:0/1968595061 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c7a8bb2d20 0x55c7a8bb5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.390+0000 7fcfec3f3740 4 client.25414 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-23T14:44:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.390+0000 7fcfec3f3740 1 -- 172.21.15.28:0/1968595061 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c7a8a9ea60 con 0x55c7a8bb2d20 2024-04-23T14:44:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.390+0000 7fcfe2a00640 1 --2- 172.21.15.28:0/1968595061 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c7a8bb2d20 0x55c7a8bb5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.391+0000 7fcfe2a00640 1 --2- 172.21.15.28:0/1968595061 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c7a8bb2d20 0x55c7a8bb5140 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=896ae2f86b0d1dd5 server_cookie=49d3a001220e159d in_seq=0 out_seq=0 2024-04-23T14:44:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.391+0000 7fcfc7e00640 10 client.25414 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:44:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.391+0000 7fcfc7e00640 1 -- 172.21.15.28:0/1968595061 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c7a8a9ea60 con 0x55c7a8bb2d20 2024-04-23T14:44:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.391+0000 7fcfc7e00640 10 client.25414 handle_command_reply: tid=0 2024-04-23T14:44:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.391+0000 7fcfec3f3740 1 client.25414 shutdown 2024-04-23T14:44:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.391+0000 7fcfec3f3740 1 -- 172.21.15.28:0/1968595061 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7a8b32270 msgr2=0x55c7a8b3da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:40.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.391+0000 7fcfec3f3740 1 --2- 172.21.15.28:0/1968595061 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7a8b32270 0x55c7a8b3da20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc0042f0 tx=0x7fcfcc004730 comp rx=0 tx=0).stop 2024-04-23T14:44:40.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.391+0000 7fcfec3f3740 1 -- 172.21.15.28:0/1968595061 shutdown_connections 2024-04-23T14:44:40.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.391+0000 7fcfec3f3740 1 --2- 172.21.15.28:0/1968595061 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c7a8bb2d20 0x55c7a8bb5140 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.391+0000 7fcfec3f3740 1 --2- 172.21.15.28:0/1968595061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a8b3c940 0x55c7a8b43b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.391+0000 7fcfec3f3740 1 --2- 172.21.15.28:0/1968595061 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c7a8b32270 0x55c7a8b3da20 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.391+0000 7fcfec3f3740 1 --2- 172.21.15.28:0/1968595061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7a8b31950 0x55c7a8b3d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.391+0000 7fcfc7e00640 0 client.25414 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:44:40.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.391+0000 7fcfec3f3740 1 -- 172.21.15.28:0/1968595061 >> 172.21.15.28:0/1968595061 conn(0x55c7a8b2dc70 msgr2=0x55c7a8b2cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:40.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.392+0000 7fcfec3f3740 1 -- 172.21.15.28:0/1968595061 shutdown_connections 2024-04-23T14:44:40.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.392+0000 7fcfec3f3740 1 -- 172.21.15.28:0/1968595061 wait complete. 2024-04-23T14:44:40.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.392+0000 7fcfec3f3740 20 client.25414 trim_cache size 0 max 16384 2024-04-23T14:44:40.395 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-23T14:44:40.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.392+0000 7fcfe9600640 1 -- 172.21.15.28:0/3753737509 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fcfc80a3db0 msgr2=0x7fcfc80a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:40.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.392+0000 7fcfe9600640 1 --2- 172.21.15.28:0/3753737509 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fcfc80a3db0 0x7fcfc80a6270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x55c7a8b2ee00 tx=0x7fcfcc005850 comp rx=0 tx=0).stop 2024-04-23T14:44:40.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.392+0000 7fcfe9600640 1 -- 172.21.15.28:0/3753737509 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfe419a9c0 msgr2=0x7fcfe419ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:40.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.392+0000 7fcfe9600640 1 --2- 172.21.15.28:0/3753737509 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfe419a9c0 0x7fcfe419ae70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc009fd0 tx=0x7fcfdc00ef10 comp rx=0 tx=0).stop 2024-04-23T14:44:40.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.392+0000 7fcfe9600640 1 -- 172.21.15.28:0/3753737509 shutdown_connections 2024-04-23T14:44:40.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.392+0000 7fcfe9600640 1 --2- 172.21.15.28:0/3753737509 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fcfc80a3db0 0x7fcfc80a6270 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.392+0000 7fcfe9600640 1 --2- 172.21.15.28:0/3753737509 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfe4109520 0x7fcfe41a0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.392+0000 7fcfe9600640 1 --2- 172.21.15.28:0/3753737509 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfe419a9c0 0x7fcfe419ae70 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.392+0000 7fcfe9600640 1 --2- 172.21.15.28:0/3753737509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfe4104450 0x7fcfe41a07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:40.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.392+0000 7fcfe9600640 1 -- 172.21.15.28:0/3753737509 >> 172.21.15.28:0/3753737509 conn(0x7fcfe40ffc70 msgr2=0x7fcfe4101fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:40.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.393+0000 7fcfe9600640 1 -- 172.21.15.28:0/3753737509 shutdown_connections 2024-04-23T14:44:40.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:40.393+0000 7fcfe9600640 1 -- 172.21.15.28:0/3753737509 wait complete. 2024-04-23T14:44:40.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:40 smithi105 ceph-mon[32555]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:44:40.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:40 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3267071323' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:44:40.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:40 smithi038 ceph-mon[32348]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:44:40.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:40 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3267071323' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:44:40.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:40 smithi028 ceph-mon[27810]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:44:40.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:40 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3267071323' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:44:41.412 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-23T14:44:41.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:44:41.412 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-23T14:44:41.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.638+0000 7fad5be00640 1 -- 172.21.15.28:0/1275753380 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad5c076e50 msgr2=0x7fad5c112810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:41.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.638+0000 7fad5be00640 1 --2- 172.21.15.28:0/1275753380 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad5c076e50 0x7fad5c112810 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fad44009930 tx=0x7fad4402a440 comp rx=0 tx=0).stop 2024-04-23T14:44:41.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.638+0000 7fad5be00640 1 -- 172.21.15.28:0/1275753380 shutdown_connections 2024-04-23T14:44:41.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.638+0000 7fad5be00640 1 --2- 172.21.15.28:0/1275753380 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad5c076e50 0x7fad5c112810 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.638+0000 7fad5be00640 1 --2- 172.21.15.28:0/1275753380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad5c112d50 0x7fad5c115140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.638+0000 7fad5be00640 1 --2- 172.21.15.28:0/1275753380 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fad5c076530 0x7fad5c076910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.638+0000 7fad5be00640 1 -- 172.21.15.28:0/1275753380 >> 172.21.15.28:0/1275753380 conn(0x7fad5c0ffb40 msgr2=0x7fad5c101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:41.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.638+0000 7fad5be00640 1 -- 172.21.15.28:0/1275753380 shutdown_connections 2024-04-23T14:44:41.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.638+0000 7fad5be00640 1 -- 172.21.15.28:0/1275753380 wait complete. 2024-04-23T14:44:41.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.639+0000 7fad5be00640 1 Processor -- start 2024-04-23T14:44:41.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.639+0000 7fad5be00640 1 -- start start 2024-04-23T14:44:41.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.639+0000 7fad5be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad5c076530 0x7fad5c19ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.639+0000 7fad5be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad5c076e50 0x7fad5c19f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.639+0000 7fad5be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fad5c112d50 0x7fad5c1a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.639+0000 7fad5be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fad5c1188c0 con 0x7fad5c076e50 2024-04-23T14:44:41.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.639+0000 7fad5be00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fad5c118740 con 0x7fad5c076530 2024-04-23T14:44:41.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.639+0000 7fad5be00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fad5c118a40 con 0x7fad5c112d50 2024-04-23T14:44:41.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.640+0000 7fad5a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad5c076e50 0x7fad5c19f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.640+0000 7fad5aa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad5c076530 0x7fad5c19ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.640+0000 7fad5a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad5c076e50 0x7fad5c19f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43766/0 (socket says 172.21.15.28:43766) 2024-04-23T14:44:41.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.640+0000 7fad5a000640 1 -- 172.21.15.28:0/2630865073 learned_addr learned my addr 172.21.15.28:0/2630865073 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:41.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.640+0000 7fad5b400640 1 --2- 172.21.15.28:0/2630865073 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fad5c112d50 0x7fad5c1a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.640+0000 7fad5aa00640 1 -- 172.21.15.28:0/2630865073 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fad5c112d50 msgr2=0x7fad5c1a36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:41.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.640+0000 7fad5aa00640 1 --2- 172.21.15.28:0/2630865073 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fad5c112d50 0x7fad5c1a36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.640+0000 7fad5aa00640 1 -- 172.21.15.28:0/2630865073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad5c076e50 msgr2=0x7fad5c19f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:41.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.640+0000 7fad5aa00640 1 --2- 172.21.15.28:0/2630865073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad5c076e50 0x7fad5c19f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.640+0000 7fad5aa00640 1 -- 172.21.15.28:0/2630865073 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad5c1a3e00 con 0x7fad5c076530 2024-04-23T14:44:41.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.641+0000 7fad5aa00640 1 --2- 172.21.15.28:0/2630865073 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad5c076530 0x7fad5c19ee60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fad54009fd0 tx=0x7fad5400ef90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:41.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.641+0000 7fad53400640 1 -- 172.21.15.28:0/2630865073 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad54004b20 con 0x7fad5c076530 2024-04-23T14:44:41.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.641+0000 7fad53400640 1 -- 172.21.15.28:0/2630865073 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad54004cc0 con 0x7fad5c076530 2024-04-23T14:44:41.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.641+0000 7fad5be00640 1 -- 172.21.15.28:0/2630865073 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad5c1a40f0 con 0x7fad5c076530 2024-04-23T14:44:41.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.641+0000 7fad53400640 1 -- 172.21.15.28:0/2630865073 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad540107b0 con 0x7fad5c076530 2024-04-23T14:44:41.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.641+0000 7fad5be00640 1 -- 172.21.15.28:0/2630865073 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad5c1ab980 con 0x7fad5c076530 2024-04-23T14:44:41.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.644+0000 7fad649e0740 1 Processor -- start 2024-04-23T14:44:41.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.645+0000 7fad649e0740 1 -- start start 2024-04-23T14:44:41.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.645+0000 7fad53400640 1 -- 172.21.15.28:0/2630865073 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7fad54005020 con 0x7fad5c076530 2024-04-23T14:44:41.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.645+0000 7fad649e0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560895a15270 0x560895a15650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.645+0000 7fad649e0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560895a15b90 0x560895a1fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.645+0000 7fad649e0740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560895a20260 0x560895a22650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.645+0000 7fad649e0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56089597ae70 con 0x560895a15270 2024-04-23T14:44:41.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.645+0000 7fad649e0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560895921380 con 0x560895a15b90 2024-04-23T14:44:41.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.645+0000 7fad649e0740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560895976e30 con 0x560895a20260 2024-04-23T14:44:41.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.645+0000 7fad5aa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560895a15b90 0x560895a1fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.645+0000 7fad5b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560895a15270 0x560895a15650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.645+0000 7fad5b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560895a15270 0x560895a15650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43776/0 (socket says 172.21.15.28:43776) 2024-04-23T14:44:41.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.645+0000 7fad5b400640 1 -- 172.21.15.28:0/1462689506 learned_addr learned my addr 172.21.15.28:0/1462689506 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:41.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.645+0000 7fad5a000640 1 --2- 172.21.15.28:0/1462689506 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560895a20260 0x560895a22650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.646+0000 7fad5aa00640 1 -- 172.21.15.28:0/1462689506 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560895a20260 msgr2=0x560895a22650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:41.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.646+0000 7fad5aa00640 1 --2- 172.21.15.28:0/1462689506 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560895a20260 0x560895a22650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.646+0000 7fad5aa00640 1 -- 172.21.15.28:0/1462689506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560895a15270 msgr2=0x560895a15650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:41.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.646+0000 7fad5aa00640 1 --2- 172.21.15.28:0/1462689506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560895a15270 0x560895a15650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.646+0000 7fad5aa00640 1 -- 172.21.15.28:0/1462689506 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608957557e0 con 0x560895a15b90 2024-04-23T14:44:41.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.646+0000 7fad5b400640 1 --2- 172.21.15.28:0/1462689506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560895a15270 0x560895a15650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:44:41.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.646+0000 7fad5aa00640 1 --2- 172.21.15.28:0/1462689506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560895a15b90 0x560895a1fd20 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fad540ce790 tx=0x7fad540d9320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b945a38eb0d63f92 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:41.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.646+0000 7fad4be00640 1 -- 172.21.15.28:0/1462689506 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad540e4070 con 0x560895a15b90 2024-04-23T14:44:41.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.647+0000 7fad4be00640 1 -- 172.21.15.28:0/1462689506 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad540d1540 con 0x560895a15b90 2024-04-23T14:44:41.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.647+0000 7fad4be00640 1 -- 172.21.15.28:0/1462689506 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad540df660 con 0x560895a15b90 2024-04-23T14:44:41.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.647+0000 7fad53400640 1 --2- 172.21.15.28:0/2630865073 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fad400a3db0 0x7fad400a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.647+0000 7fad53400640 1 -- 172.21.15.28:0/2630865073 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fad540d1150 con 0x7fad5c076530 2024-04-23T14:44:41.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.647+0000 7fad5b400640 1 --2- 172.21.15.28:0/2630865073 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fad400a3db0 0x7fad400a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.647+0000 7fad649e0740 1 -- 172.21.15.28:0/1462689506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560895a15b90 msgr2=0x560895a1fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:41.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.647+0000 7fad649e0740 1 --2- 172.21.15.28:0/1462689506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560895a15b90 0x560895a1fd20 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fad540ce790 tx=0x7fad540d9320 comp rx=0 tx=0).stop 2024-04-23T14:44:41.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.647+0000 7fad649e0740 1 -- 172.21.15.28:0/1462689506 shutdown_connections 2024-04-23T14:44:41.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.647+0000 7fad649e0740 1 --2- 172.21.15.28:0/1462689506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560895a15b90 0x560895a1fd20 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.647+0000 7fad649e0740 1 --2- 172.21.15.28:0/1462689506 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560895a20260 0x560895a22650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.647+0000 7fad649e0740 1 --2- 172.21.15.28:0/1462689506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560895a15270 0x560895a15650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.647+0000 7fad649e0740 1 -- 172.21.15.28:0/1462689506 >> 172.21.15.28:0/1462689506 conn(0x560895a11590 msgr2=0x560895a11970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:41.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.648+0000 7fad649e0740 1 -- 172.21.15.28:0/1462689506 shutdown_connections 2024-04-23T14:44:41.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.648+0000 7fad649e0740 1 -- 172.21.15.28:0/1462689506 wait complete. 2024-04-23T14:44:41.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.648+0000 7fad5b400640 1 --2- 172.21.15.28:0/2630865073 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fad400a3db0 0x7fad400a6270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fad4c009fb0 tx=0x7fad4c005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:41.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.648+0000 7fad649e0740 1 -- start start 2024-04-23T14:44:41.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.648+0000 7fad649e0740 1 Processor -- start 2024-04-23T14:44:41.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.649+0000 7fad649e0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560895a15270 0x560895a1d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.649+0000 7fad649e0740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560895a15b90 0x560895a1d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.649+0000 7fad649e0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560895a20260 0x560895a1dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.649+0000 7fad649e0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560895a23050 con 0x560895a15270 2024-04-23T14:44:41.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.649+0000 7fad649e0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560895934660 con 0x560895a20260 2024-04-23T14:44:41.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.649+0000 7fad649e0740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560895a231d0 con 0x560895a15b90 2024-04-23T14:44:41.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.649+0000 7fad649e0740 10 client.0 _ll_register_callbacks cb 0x560895922aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:44:41.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.649+0000 7fad5aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560895a15270 0x560895a1d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.649+0000 7fad5b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560895a20260 0x560895a1dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.649+0000 7fad5b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560895a20260 0x560895a1dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:43046/0 (socket says 172.21.15.28:43046) 2024-04-23T14:44:41.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.649+0000 7fad5aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560895a15270 0x560895a1d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43780/0 (socket says 172.21.15.28:43780) 2024-04-23T14:44:41.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.649+0000 7fad5b400640 1 -- 172.21.15.28:0/810495624 learned_addr learned my addr 172.21.15.28:0/810495624 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:41.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.649+0000 7fad5a000640 1 --2- 172.21.15.28:0/810495624 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560895a15b90 0x560895a1d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.650+0000 7fad5b400640 1 -- 172.21.15.28:0/810495624 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560895a15b90 msgr2=0x560895a1d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:41.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.650+0000 7fad5b400640 1 --2- 172.21.15.28:0/810495624 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560895a15b90 0x560895a1d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.650+0000 7fad5b400640 1 -- 172.21.15.28:0/810495624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560895a15270 msgr2=0x560895a1d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:41.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.650+0000 7fad5b400640 1 --2- 172.21.15.28:0/810495624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560895a15270 0x560895a1d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.650+0000 7fad5b400640 1 -- 172.21.15.28:0/810495624 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608957b5f70 con 0x560895a20260 2024-04-23T14:44:41.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.650+0000 7fad5aa00640 1 --2- 172.21.15.28:0/810495624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560895a15270 0x560895a1d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:44:41.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.650+0000 7fad5b400640 1 --2- 172.21.15.28:0/810495624 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560895a20260 0x560895a1dce0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fad4c0045d0 tx=0x7fad4c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1ea41094d016cef server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:41.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.650+0000 7fad48c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:44:41.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.650+0000 7fad48c00640 1 -- 172.21.15.28:0/810495624 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad4c012e40 con 0x560895a20260 2024-04-23T14:44:41.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.650+0000 7fad48c00640 1 -- 172.21.15.28:0/810495624 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fad4c01f640 con 0x560895a20260 2024-04-23T14:44:41.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.650+0000 7fad649e0740 1 -- 172.21.15.28:0/810495624 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad540d1540 con 0x560895a20260 2024-04-23T14:44:41.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.651+0000 7fad48c00640 1 -- 172.21.15.28:0/810495624 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fad4c01f840 con 0x560895a20260 2024-04-23T14:44:41.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.651+0000 7fad649e0740 10 client.25418 fetch_fsmap learned FSMap version 20 2024-04-23T14:44:41.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.651+0000 7fad649e0740 1 -- 172.21.15.28:0/810495624 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x56089564d170 con 0x560895a20260 2024-04-23T14:44:41.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.651+0000 7fad48c00640 1 -- 172.21.15.28:0/810495624 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad540d1540 con 0x560895a20260 2024-04-23T14:44:41.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.651+0000 7fad48c00640 1 -- 172.21.15.28:0/810495624 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fad4c027070 con 0x560895a20260 2024-04-23T14:44:41.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.651+0000 7fad649e0740 10 client.25418 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:44:41.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.651+0000 7fad649e0740 10 client.25418 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:44:41.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.651+0000 7fad649e0740 20 client.25418 populate_metadata read hostname 'smithi028' 2024-04-23T14:44:41.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.651+0000 7fad649e0740 1 --2- 172.21.15.28:0/810495624 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560895a306f0 0x560895a32b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.651+0000 7fad649e0740 4 client.25418 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:44:41.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.651+0000 7fad649e0740 1 -- 172.21.15.28:0/810495624 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560895988890 con 0x560895a306f0 2024-04-23T14:44:41.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.652+0000 7fad5aa00640 1 --2- 172.21.15.28:0/810495624 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560895a306f0 0x560895a32b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.652+0000 7fad5aa00640 1 --2- 172.21.15.28:0/810495624 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560895a306f0 0x560895a32b20 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b81dc5c974d412cb server_cookie=fdd24ec67d1e2f5f in_seq=0 out_seq=0 2024-04-23T14:44:41.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.652+0000 7fad48c00640 10 client.25418 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:44:41.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.653+0000 7fad48c00640 1 -- 172.21.15.28:0/810495624 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560895988890 con 0x560895a306f0 2024-04-23T14:44:41.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.653+0000 7fad48c00640 10 client.25418 handle_command_reply: tid=0 2024-04-23T14:44:41.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.653+0000 7fad649e0740 1 client.25418 shutdown 2024-04-23T14:44:41.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.653+0000 7fad649e0740 1 -- 172.21.15.28:0/810495624 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560895a20260 msgr2=0x560895a1dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:41.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.653+0000 7fad649e0740 1 --2- 172.21.15.28:0/810495624 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560895a20260 0x560895a1dce0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fad4c0045d0 tx=0x7fad4c004730 comp rx=0 tx=0).stop 2024-04-23T14:44:41.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.653+0000 7fad649e0740 1 -- 172.21.15.28:0/810495624 shutdown_connections 2024-04-23T14:44:41.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.653+0000 7fad649e0740 1 --2- 172.21.15.28:0/810495624 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560895a306f0 0x560895a32b20 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.653+0000 7fad649e0740 1 --2- 172.21.15.28:0/810495624 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560895a20260 0x560895a1dce0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.653+0000 7fad649e0740 1 --2- 172.21.15.28:0/810495624 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560895a15b90 0x560895a1d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.653+0000 7fad649e0740 1 --2- 172.21.15.28:0/810495624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560895a15270 0x560895a1d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.653+0000 7fad649e0740 1 -- 172.21.15.28:0/810495624 >> 172.21.15.28:0/810495624 conn(0x560895a11590 msgr2=0x560895a17390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:41.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.653+0000 7fad48c00640 0 client.25418 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:44:41.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.653+0000 7fad649e0740 1 -- 172.21.15.28:0/810495624 shutdown_connections 2024-04-23T14:44:41.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.653+0000 7fad649e0740 1 -- 172.21.15.28:0/810495624 wait complete. 2024-04-23T14:44:41.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.654+0000 7fad649e0740 20 client.25418 trim_cache size 0 max 16384 2024-04-23T14:44:41.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.673+0000 7fad649e0740 1 Processor -- start 2024-04-23T14:44:41.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.673+0000 7fad649e0740 1 -- start start 2024-04-23T14:44:41.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.673+0000 7fad649e0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560895a15270 0x560895a1fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.673+0000 7fad649e0740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560895a15b90 0x560895a1cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.673+0000 7fad649e0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560895a20260 0x560895a10e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.673+0000 7fad649e0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fad5c1b2110 con 0x560895a15270 2024-04-23T14:44:41.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.673+0000 7fad649e0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fad5c1b4990 con 0x560895a20260 2024-04-23T14:44:41.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.673+0000 7fad649e0740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fad5c1b2780 con 0x560895a15b90 2024-04-23T14:44:41.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.674+0000 7fad5a000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560895a15b90 0x560895a1cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.674+0000 7fad5a000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560895a15b90 0x560895a1cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:44386/0 (socket says 172.21.15.28:44386) 2024-04-23T14:44:41.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.674+0000 7fad5a000640 1 -- 172.21.15.28:0/1141838531 learned_addr learned my addr 172.21.15.28:0/1141838531 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:41.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.674+0000 7fad5b400640 1 --2- 172.21.15.28:0/1141838531 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560895a20260 0x560895a10e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.674+0000 7fad5aa00640 1 --2- 172.21.15.28:0/1141838531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560895a15270 0x560895a1fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.674+0000 7fad5a000640 1 -- 172.21.15.28:0/1141838531 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560895a20260 msgr2=0x560895a10e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:41.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.674+0000 7fad5a000640 1 --2- 172.21.15.28:0/1141838531 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560895a20260 0x560895a10e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.674+0000 7fad5a000640 1 -- 172.21.15.28:0/1141838531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560895a15270 msgr2=0x560895a1fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:41.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.674+0000 7fad5a000640 1 --2- 172.21.15.28:0/1141838531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560895a15270 0x560895a1fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.674+0000 7fad5a000640 1 -- 172.21.15.28:0/1141838531 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560895a113b0 con 0x560895a15b90 2024-04-23T14:44:41.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.675+0000 7fad5a000640 1 --2- 172.21.15.28:0/1141838531 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560895a15b90 0x560895a1cb60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fad44009230 tx=0x7fad44031880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51de11accd3dfd95 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:41.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.675+0000 7fad4b400640 1 -- 172.21.15.28:0/1141838531 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad44035040 con 0x560895a15b90 2024-04-23T14:44:41.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.675+0000 7fad4b400640 1 -- 172.21.15.28:0/1141838531 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad44031970 con 0x560895a15b90 2024-04-23T14:44:41.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.675+0000 7fad4b400640 1 -- 172.21.15.28:0/1141838531 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad44004a00 con 0x560895a15b90 2024-04-23T14:44:41.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.676+0000 7fad649e0740 1 -- 172.21.15.28:0/1141838531 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560895a15b90 msgr2=0x560895a1cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:41.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.676+0000 7fad649e0740 1 --2- 172.21.15.28:0/1141838531 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560895a15b90 0x560895a1cb60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fad44009230 tx=0x7fad44031880 comp rx=0 tx=0).stop 2024-04-23T14:44:41.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.676+0000 7fad649e0740 1 -- 172.21.15.28:0/1141838531 shutdown_connections 2024-04-23T14:44:41.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.676+0000 7fad649e0740 1 --2- 172.21.15.28:0/1141838531 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560895a20260 0x560895a10e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.676+0000 7fad649e0740 1 --2- 172.21.15.28:0/1141838531 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560895a15b90 0x560895a1cb60 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.676+0000 7fad649e0740 1 --2- 172.21.15.28:0/1141838531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560895a15270 0x560895a1fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.676+0000 7fad649e0740 1 -- 172.21.15.28:0/1141838531 >> 172.21.15.28:0/1141838531 conn(0x560895a11590 msgr2=0x560895a1f320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:41.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.676+0000 7fad649e0740 1 -- 172.21.15.28:0/1141838531 shutdown_connections 2024-04-23T14:44:41.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.676+0000 7fad649e0740 1 -- 172.21.15.28:0/1141838531 wait complete. 2024-04-23T14:44:41.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.676+0000 7fad649e0740 1 -- start start 2024-04-23T14:44:41.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.677+0000 7fad649e0740 1 Processor -- start 2024-04-23T14:44:41.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.677+0000 7fad649e0740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560895a15270 0x560895a28810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.677+0000 7fad649e0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560895a15b90 0x560895a19ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.677+0000 7fad649e0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560895a20260 0x560895a1a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.677+0000 7fad649e0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fad5c1b2e50 con 0x560895a20260 2024-04-23T14:44:41.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.677+0000 7fad649e0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fad5c1b42c0 con 0x560895a15b90 2024-04-23T14:44:41.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.677+0000 7fad649e0740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fad5c1b3520 con 0x560895a15270 2024-04-23T14:44:41.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.677+0000 7fad649e0740 10 client.0 _ll_register_callbacks cb 0x7fad4c009c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:44:41.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.677+0000 7fad5b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560895a20260 0x560895a1a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.677+0000 7fad5b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560895a20260 0x560895a1a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43796/0 (socket says 172.21.15.28:43796) 2024-04-23T14:44:41.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.677+0000 7fad5b400640 1 -- 172.21.15.28:0/618666961 learned_addr learned my addr 172.21.15.28:0/618666961 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:41.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.677+0000 7fad5aa00640 1 --2- 172.21.15.28:0/618666961 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560895a15270 0x560895a28810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.678+0000 7fad5a000640 1 --2- 172.21.15.28:0/618666961 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560895a15b90 0x560895a19ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.678+0000 7fad5aa00640 1 -- 172.21.15.28:0/618666961 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560895a15b90 msgr2=0x560895a19ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:41.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.678+0000 7fad5aa00640 1 --2- 172.21.15.28:0/618666961 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560895a15b90 0x560895a19ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.678+0000 7fad5aa00640 1 -- 172.21.15.28:0/618666961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560895a20260 msgr2=0x560895a1a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:41.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.678+0000 7fad5aa00640 1 --2- 172.21.15.28:0/618666961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560895a20260 0x560895a1a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.678+0000 7fad5aa00640 1 -- 172.21.15.28:0/618666961 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560895a1aa50 con 0x560895a15270 2024-04-23T14:44:41.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.678+0000 7fad5b400640 1 --2- 172.21.15.28:0/618666961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560895a20260 0x560895a1a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:44:41.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.678+0000 7fad5aa00640 1 --2- 172.21.15.28:0/618666961 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560895a15270 0x560895a28810 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fad5409cc30 tx=0x7fad540d1ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef3dcca8c79b6b64 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:41.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.678+0000 7fad48c00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:44:41.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.678+0000 7fad48c00640 1 -- 172.21.15.28:0/618666961 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad540991d0 con 0x560895a15270 2024-04-23T14:44:41.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.678+0000 7fad48c00640 1 -- 172.21.15.28:0/618666961 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fad54005310 con 0x560895a15270 2024-04-23T14:44:41.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.678+0000 7fad649e0740 1 -- 172.21.15.28:0/618666961 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad44031970 con 0x560895a15270 2024-04-23T14:44:41.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.679+0000 7fad48c00640 1 -- 172.21.15.28:0/618666961 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fad5409c890 con 0x560895a15270 2024-04-23T14:44:41.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.679+0000 7fad649e0740 10 client.25432 fetch_fsmap learned FSMap version 20 2024-04-23T14:44:41.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.679+0000 7fad649e0740 1 -- 172.21.15.28:0/618666961 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x560895a96d00 con 0x560895a15270 2024-04-23T14:44:41.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.679+0000 7fad48c00640 1 -- 172.21.15.28:0/618666961 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad44031970 con 0x560895a15270 2024-04-23T14:44:41.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.679+0000 7fad48c00640 1 -- 172.21.15.28:0/618666961 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fad540995a0 con 0x560895a15270 2024-04-23T14:44:41.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.679+0000 7fad649e0740 10 client.25432 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:44:41.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.679+0000 7fad649e0740 10 client.25432 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:44:41.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.679+0000 7fad649e0740 20 client.25432 populate_metadata read hostname 'smithi028' 2024-04-23T14:44:41.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.679+0000 7fad649e0740 1 --2- 172.21.15.28:0/618666961 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560895a97050 0x560895a99440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.679+0000 7fad649e0740 4 client.25432 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-23T14:44:41.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.679+0000 7fad649e0740 1 -- 172.21.15.28:0/618666961 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560895982700 con 0x560895a97050 2024-04-23T14:44:41.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.680+0000 7fad5a000640 1 --2- 172.21.15.28:0/618666961 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560895a97050 0x560895a99440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.680+0000 7fad5a000640 1 --2- 172.21.15.28:0/618666961 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560895a97050 0x560895a99440 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b093cce82c96b560 server_cookie=47d3cf71a6625b95 in_seq=0 out_seq=0 2024-04-23T14:44:41.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.680+0000 7fad48c00640 10 client.25432 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:44:41.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.681+0000 7fad48c00640 1 -- 172.21.15.28:0/618666961 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560895982700 con 0x560895a97050 2024-04-23T14:44:41.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.681+0000 7fad48c00640 10 client.25432 handle_command_reply: tid=0 2024-04-23T14:44:41.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.681+0000 7fad649e0740 1 client.25432 shutdown 2024-04-23T14:44:41.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.681+0000 7fad649e0740 1 -- 172.21.15.28:0/618666961 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560895a15270 msgr2=0x560895a28810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:41.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.681+0000 7fad649e0740 1 --2- 172.21.15.28:0/618666961 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560895a15270 0x560895a28810 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fad5409cc30 tx=0x7fad540d1ec0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.681+0000 7fad649e0740 1 -- 172.21.15.28:0/618666961 shutdown_connections 2024-04-23T14:44:41.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.681+0000 7fad649e0740 1 --2- 172.21.15.28:0/618666961 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560895a97050 0x560895a99440 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.681+0000 7fad649e0740 1 --2- 172.21.15.28:0/618666961 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560895a15b90 0x560895a19ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.681+0000 7fad649e0740 1 --2- 172.21.15.28:0/618666961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560895a20260 0x560895a1a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.681+0000 7fad649e0740 1 --2- 172.21.15.28:0/618666961 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560895a15270 0x560895a28810 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.681+0000 7fad48c00640 0 client.25432 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:44:41.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.681+0000 7fad649e0740 1 -- 172.21.15.28:0/618666961 >> 172.21.15.28:0/618666961 conn(0x560895a11590 msgr2=0x560895a1bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:41.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.682+0000 7fad649e0740 1 -- 172.21.15.28:0/618666961 shutdown_connections 2024-04-23T14:44:41.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.682+0000 7fad649e0740 1 -- 172.21.15.28:0/618666961 wait complete. 2024-04-23T14:44:41.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.682+0000 7fad649e0740 20 client.25432 trim_cache size 0 max 16384 2024-04-23T14:44:41.686 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:44:41.686 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-23T14:44:41.686 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "6fd9d4d6-0c2f-45ce-99e1-d6fd9e726c23", 2024-04-23T14:44:41.686 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-23T14:44:41.686 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:44:41.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.682+0000 7fad5be00640 1 -- 172.21.15.28:0/2630865073 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fad400a3db0 msgr2=0x7fad400a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:41.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.682+0000 7fad5be00640 1 --2- 172.21.15.28:0/2630865073 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fad400a3db0 0x7fad400a6270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fad4c009fb0 tx=0x7fad4c005f90 comp rx=0 tx=0).stop 2024-04-23T14:44:41.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.682+0000 7fad5be00640 1 -- 172.21.15.28:0/2630865073 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad5c076530 msgr2=0x7fad5c19ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:41.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.682+0000 7fad5be00640 1 --2- 172.21.15.28:0/2630865073 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad5c076530 0x7fad5c19ee60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fad54009fd0 tx=0x7fad5400ef90 comp rx=0 tx=0).stop 2024-04-23T14:44:41.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.683+0000 7fad5be00640 1 -- 172.21.15.28:0/2630865073 shutdown_connections 2024-04-23T14:44:41.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.683+0000 7fad5be00640 1 --2- 172.21.15.28:0/2630865073 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fad400a3db0 0x7fad400a6270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.683+0000 7fad5be00640 1 --2- 172.21.15.28:0/2630865073 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fad5c112d50 0x7fad5c1a36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.683+0000 7fad5be00640 1 --2- 172.21.15.28:0/2630865073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad5c076e50 0x7fad5c19f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.683+0000 7fad5be00640 1 --2- 172.21.15.28:0/2630865073 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad5c076530 0x7fad5c19ee60 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.683+0000 7fad5be00640 1 -- 172.21.15.28:0/2630865073 >> 172.21.15.28:0/2630865073 conn(0x7fad5c0ffb40 msgr2=0x7fad5c101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:41.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.683+0000 7fad5be00640 1 -- 172.21.15.28:0/2630865073 shutdown_connections 2024-04-23T14:44:41.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.683+0000 7fad5be00640 1 -- 172.21.15.28:0/2630865073 wait complete. 2024-04-23T14:44:41.701 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:44:41.701 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:44:41.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.908+0000 7f7c7f200640 1 -- 172.21.15.28:0/2675476731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c8010d500 msgr2=0x7f7c8010d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:41.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.908+0000 7f7c7f200640 1 --2- 172.21.15.28:0/2675476731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c8010d500 0x7f7c8010d8e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7c68009930 tx=0x7f7c6802a420 comp rx=0 tx=0).stop 2024-04-23T14:44:41.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.908+0000 7f7c7f200640 1 -- 172.21.15.28:0/2675476731 shutdown_connections 2024-04-23T14:44:41.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.908+0000 7f7c7f200640 1 --2- 172.21.15.28:0/2675476731 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c80108580 0x7f7c80108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.908+0000 7f7c7f200640 1 --2- 172.21.15.28:0/2675476731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c80108f40 0x7f7c801153b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.908+0000 7f7c7f200640 1 --2- 172.21.15.28:0/2675476731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c8010d500 0x7f7c8010d8e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.908+0000 7f7c7f200640 1 -- 172.21.15.28:0/2675476731 >> 172.21.15.28:0/2675476731 conn(0x7f7c80077e20 msgr2=0x7f7c80106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:41.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.908+0000 7f7c7f200640 1 -- 172.21.15.28:0/2675476731 shutdown_connections 2024-04-23T14:44:41.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.908+0000 7f7c7f200640 1 -- 172.21.15.28:0/2675476731 wait complete. 2024-04-23T14:44:41.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.910+0000 7f7c7f200640 1 Processor -- start 2024-04-23T14:44:41.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.910+0000 7f7c7f200640 1 -- start start 2024-04-23T14:44:41.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.910+0000 7f7c7f200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c80108580 0x7f7c8019ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.911+0000 7f7c7f200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c80108f40 0x7f7c8019f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.911+0000 7f7c7f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c8010d500 0x7f7c801a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.911+0000 7f7c7f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7c80118910 con 0x7f7c8010d500 2024-04-23T14:44:41.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.911+0000 7f7c7f200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7c80118790 con 0x7f7c80108f40 2024-04-23T14:44:41.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.911+0000 7f7c7f200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7c80118a90 con 0x7f7c80108580 2024-04-23T14:44:41.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.911+0000 7f7c7e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c8010d500 0x7f7c801a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.911+0000 7f7c7e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c8010d500 0x7f7c801a35e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43820/0 (socket says 172.21.15.28:43820) 2024-04-23T14:44:41.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.911+0000 7f7c7d400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c80108f40 0x7f7c8019f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.911+0000 7f7c7e800640 1 -- 172.21.15.28:0/136004410 learned_addr learned my addr 172.21.15.28:0/136004410 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:41.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.911+0000 7f7c7de00640 1 --2- 172.21.15.28:0/136004410 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c80108580 0x7f7c8019ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.911+0000 7f7c7e800640 1 -- 172.21.15.28:0/136004410 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c80108580 msgr2=0x7f7c8019ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:41.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.911+0000 7f7c7e800640 1 --2- 172.21.15.28:0/136004410 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c80108580 0x7f7c8019ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.911+0000 7f7c7e800640 1 -- 172.21.15.28:0/136004410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c80108f40 msgr2=0x7f7c8019f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:41.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.911+0000 7f7c7e800640 1 --2- 172.21.15.28:0/136004410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c80108f40 0x7f7c8019f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.911+0000 7f7c7e800640 1 -- 172.21.15.28:0/136004410 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c801a3d10 con 0x7f7c8010d500 2024-04-23T14:44:41.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.912+0000 7f7c7e800640 1 --2- 172.21.15.28:0/136004410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c8010d500 0x7f7c801a35e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7c7800ea60 tx=0x7f7c7800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:41.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.912+0000 7f7c76a00640 1 -- 172.21.15.28:0/136004410 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c7800ce40 con 0x7f7c8010d500 2024-04-23T14:44:41.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.912+0000 7f7c76a00640 1 -- 172.21.15.28:0/136004410 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c78005020 con 0x7f7c8010d500 2024-04-23T14:44:41.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.912+0000 7f7c7f200640 1 -- 172.21.15.28:0/136004410 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c801a4000 con 0x7f7c8010d500 2024-04-23T14:44:41.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.912+0000 7f7c76a00640 1 -- 172.21.15.28:0/136004410 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c78010690 con 0x7f7c8010d500 2024-04-23T14:44:41.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.912+0000 7f7c7f200640 1 -- 172.21.15.28:0/136004410 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c801ab890 con 0x7f7c8010d500 2024-04-23T14:44:41.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.915+0000 7f7c87813740 1 Processor -- start 2024-04-23T14:44:41.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.915+0000 7f7c76a00640 1 -- 172.21.15.28:0/136004410 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138822+0+0 (secure 0 0 0) 0x7f7c78005520 con 0x7f7c8010d500 2024-04-23T14:44:41.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.916+0000 7f7c87813740 1 -- start start 2024-04-23T14:44:41.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.916+0000 7f7c87813740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efff9eea30 0x55efff9eee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.916+0000 7f7c87813740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efff9ef350 0x55efff9f94e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.916+0000 7f7c87813740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efff9f9a20 0x55efff9fbe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.916+0000 7f7c87813740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55efff9541f0 con 0x55efff9eea30 2024-04-23T14:44:41.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.916+0000 7f7c87813740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55efff8fa350 con 0x55efff9ef350 2024-04-23T14:44:41.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.916+0000 7f7c87813740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55efff9501b0 con 0x55efff9f9a20 2024-04-23T14:44:41.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.916+0000 7f7c7d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efff9eea30 0x55efff9eee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.916+0000 7f7c7d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efff9eea30 0x55efff9eee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43824/0 (socket says 172.21.15.28:43824) 2024-04-23T14:44:41.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.916+0000 7f7c7d400640 1 -- 172.21.15.28:0/3657317311 learned_addr learned my addr 172.21.15.28:0/3657317311 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:41.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.917+0000 7f7c7de00640 1 --2- 172.21.15.28:0/3657317311 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efff9ef350 0x55efff9f94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.917+0000 7f7c7d400640 1 --2- 172.21.15.28:0/3657317311 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efff9f9a20 0x55efff9fbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.917+0000 7f7c7d400640 1 -- 172.21.15.28:0/3657317311 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efff9f9a20 msgr2=0x55efff9fbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:41.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.917+0000 7f7c7d400640 1 --2- 172.21.15.28:0/3657317311 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efff9f9a20 0x55efff9fbe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.917+0000 7f7c7d400640 1 -- 172.21.15.28:0/3657317311 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efff9ef350 msgr2=0x55efff9f94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:41.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.917+0000 7f7c7d400640 1 --2- 172.21.15.28:0/3657317311 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efff9ef350 0x55efff9f94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.917+0000 7f7c7d400640 1 -- 172.21.15.28:0/3657317311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efff94ef50 con 0x55efff9eea30 2024-04-23T14:44:41.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.917+0000 7f7c7d400640 1 --2- 172.21.15.28:0/3657317311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efff9eea30 0x55efff9eee10 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7c7000bef0 tx=0x7f7c7000b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63f042a2558dd795 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:41.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.917+0000 7f7c76a00640 1 --2- 172.21.15.28:0/136004410 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7c640a3db0 0x7f7c640a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.917+0000 7f7c6f000640 1 -- 172.21.15.28:0/3657317311 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c7000e490 con 0x55efff9eea30 2024-04-23T14:44:41.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.918+0000 7f7c6f000640 1 -- 172.21.15.28:0/3657317311 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c7000eb10 con 0x55efff9eea30 2024-04-23T14:44:41.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.918+0000 7f7c6f000640 1 -- 172.21.15.28:0/3657317311 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c70010900 con 0x55efff9eea30 2024-04-23T14:44:41.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.918+0000 7f7c76a00640 1 -- 172.21.15.28:0/136004410 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f7c780d0e00 con 0x7f7c8010d500 2024-04-23T14:44:41.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.918+0000 7f7c7de00640 1 --2- 172.21.15.28:0/136004410 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7c640a3db0 0x7f7c640a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.918+0000 7f7c87813740 1 -- 172.21.15.28:0/3657317311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efff9eea30 msgr2=0x55efff9eee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:41.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.918+0000 7f7c87813740 1 --2- 172.21.15.28:0/3657317311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efff9eea30 0x55efff9eee10 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7c7000bef0 tx=0x7f7c7000b410 comp rx=0 tx=0).stop 2024-04-23T14:44:41.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.918+0000 7f7c87813740 1 -- 172.21.15.28:0/3657317311 shutdown_connections 2024-04-23T14:44:41.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.919+0000 7f7c87813740 1 --2- 172.21.15.28:0/3657317311 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efff9ef350 0x55efff9f94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.919+0000 7f7c87813740 1 --2- 172.21.15.28:0/3657317311 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efff9f9a20 0x55efff9fbe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.918+0000 7f7c7de00640 1 --2- 172.21.15.28:0/136004410 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7c640a3db0 0x7f7c640a6270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7c68009900 tx=0x7f7c68000bc0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.919+0000 7f7c87813740 1 --2- 172.21.15.28:0/3657317311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efff9eea30 0x55efff9eee10 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.919+0000 7f7c87813740 1 -- 172.21.15.28:0/3657317311 >> 172.21.15.28:0/3657317311 conn(0x55efff9ead50 msgr2=0x55efff9eb130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.919+0000 7f7c87813740 1 -- 172.21.15.28:0/3657317311 shutdown_connections 2024-04-23T14:44:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.919+0000 7f7c87813740 1 -- 172.21.15.28:0/3657317311 wait complete. 2024-04-23T14:44:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.919+0000 7f7c87813740 1 -- start start 2024-04-23T14:44:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.919+0000 7f7c87813740 1 Processor -- start 2024-04-23T14:44:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.920+0000 7f7c87813740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efff9eea30 0x55efff9f6900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.920+0000 7f7c87813740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efff9ef350 0x55efff9f6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.920+0000 7f7c87813740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efff9f9a20 0x55efff9f7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.920+0000 7f7c87813740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55efff9fc810 con 0x55efff9ef350 2024-04-23T14:44:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.920+0000 7f7c87813740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55efff90d690 con 0x55efff9eea30 2024-04-23T14:44:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.920+0000 7f7c87813740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55efff9fc990 con 0x55efff9f9a20 2024-04-23T14:44:41.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.920+0000 7f7c87813740 10 client.0 _ll_register_callbacks cb 0x55efff949650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:44:41.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.920+0000 7f7c7d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efff9ef350 0x55efff9f6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.920+0000 7f7c7d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efff9ef350 0x55efff9f6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43832/0 (socket says 172.21.15.28:43832) 2024-04-23T14:44:41.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.920+0000 7f7c7d400640 1 -- 172.21.15.28:0/1747080400 learned_addr learned my addr 172.21.15.28:0/1747080400 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:41.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.920+0000 7f7c7de00640 1 --2- 172.21.15.28:0/1747080400 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efff9eea30 0x55efff9f6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.920+0000 7f7c7e800640 1 --2- 172.21.15.28:0/1747080400 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efff9f9a20 0x55efff9f7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.921+0000 7f7c7d400640 1 -- 172.21.15.28:0/1747080400 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efff9f9a20 msgr2=0x55efff9f7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:41.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.921+0000 7f7c7d400640 1 --2- 172.21.15.28:0/1747080400 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efff9f9a20 0x55efff9f7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.921+0000 7f7c7d400640 1 -- 172.21.15.28:0/1747080400 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efff9eea30 msgr2=0x55efff9f6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:41.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.921+0000 7f7c7d400640 1 --2- 172.21.15.28:0/1747080400 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efff9eea30 0x55efff9f6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.921+0000 7f7c7d400640 1 -- 172.21.15.28:0/1747080400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efff90c920 con 0x55efff9ef350 2024-04-23T14:44:41.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.921+0000 7f7c7e800640 1 --2- 172.21.15.28:0/1747080400 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efff9f9a20 0x55efff9f7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:44:41.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.921+0000 7f7c7de00640 1 --2- 172.21.15.28:0/1747080400 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efff9eea30 0x55efff9f6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:44:41.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.921+0000 7f7c7d400640 1 --2- 172.21.15.28:0/1747080400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efff9ef350 0x55efff9f6e40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7c70010460 tx=0x7f7c70012190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0310e25c86eef07 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:41.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.921+0000 7f7c63400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-23T14:44:41.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.921+0000 7f7c63400640 1 -- 172.21.15.28:0/1747080400 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c7000e8a0 con 0x55efff9ef350 2024-04-23T14:44:41.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.921+0000 7f7c87813740 1 -- 172.21.15.28:0/1747080400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c7000eb10 con 0x55efff9ef350 2024-04-23T14:44:41.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.921+0000 7f7c63400640 1 -- 172.21.15.28:0/1747080400 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f7c70005040 con 0x55efff9ef350 2024-04-23T14:44:41.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.922+0000 7f7c63400640 1 -- 172.21.15.28:0/1747080400 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f7c700123b0 con 0x55efff9ef350 2024-04-23T14:44:41.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.922+0000 7f7c87813740 10 client.15306 fetch_fsmap learned FSMap version 20 2024-04-23T14:44:41.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.922+0000 7f7c87813740 1 -- 172.21.15.28:0/1747080400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55efff7307d0 con 0x55efff9ef350 2024-04-23T14:44:41.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.922+0000 7f7c63400640 1 -- 172.21.15.28:0/1747080400 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c7000eb10 con 0x55efff9ef350 2024-04-23T14:44:41.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.922+0000 7f7c63400640 1 -- 172.21.15.28:0/1747080400 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f7c700046c0 con 0x55efff9ef350 2024-04-23T14:44:41.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.922+0000 7f7c87813740 10 client.15306 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:44:41.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.922+0000 7f7c87813740 10 client.15306 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:44:41.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.922+0000 7f7c87813740 20 client.15306 populate_metadata read hostname 'smithi028' 2024-04-23T14:44:41.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.922+0000 7f7c87813740 1 --2- 172.21.15.28:0/1747080400 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55efffa09b70 0x55efffa0bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.922+0000 7f7c87813740 4 client.15306 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:44:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.922+0000 7f7c87813740 1 -- 172.21.15.28:0/1747080400 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55efff961c10 con 0x55efffa09b70 2024-04-23T14:44:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.923+0000 7f7c7de00640 1 --2- 172.21.15.28:0/1747080400 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55efffa09b70 0x55efffa0bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.923+0000 7f7c7de00640 1 --2- 172.21.15.28:0/1747080400 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55efffa09b70 0x55efffa0bf60 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbcc271a4108aafc server_cookie=3c2e04f4f1d8958 in_seq=0 out_seq=0 2024-04-23T14:44:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.923+0000 7f7c63400640 10 client.15306 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:44:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.924+0000 7f7c63400640 1 -- 172.21.15.28:0/1747080400 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55efff961c10 con 0x55efffa09b70 2024-04-23T14:44:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.924+0000 7f7c63400640 10 client.15306 handle_command_reply: tid=0 2024-04-23T14:44:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.924+0000 7f7c87813740 1 client.15306 shutdown 2024-04-23T14:44:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.924+0000 7f7c87813740 1 -- 172.21.15.28:0/1747080400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efff9ef350 msgr2=0x55efff9f6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.924+0000 7f7c87813740 1 --2- 172.21.15.28:0/1747080400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efff9ef350 0x55efff9f6e40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7c70010460 tx=0x7f7c70012190 comp rx=0 tx=0).stop 2024-04-23T14:44:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.924+0000 7f7c87813740 1 -- 172.21.15.28:0/1747080400 shutdown_connections 2024-04-23T14:44:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.924+0000 7f7c87813740 1 --2- 172.21.15.28:0/1747080400 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55efffa09b70 0x55efffa0bf60 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.924+0000 7f7c87813740 1 --2- 172.21.15.28:0/1747080400 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efff9f9a20 0x55efff9f7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.924+0000 7f7c87813740 1 --2- 172.21.15.28:0/1747080400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efff9ef350 0x55efff9f6e40 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.924+0000 7f7c87813740 1 --2- 172.21.15.28:0/1747080400 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efff9eea30 0x55efff9f6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.924+0000 7f7c63400640 0 client.15306 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:44:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.924+0000 7f7c87813740 1 -- 172.21.15.28:0/1747080400 >> 172.21.15.28:0/1747080400 conn(0x55efff9ead50 msgr2=0x55efff9e9fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.924+0000 7f7c87813740 1 -- 172.21.15.28:0/1747080400 shutdown_connections 2024-04-23T14:44:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.924+0000 7f7c87813740 1 -- 172.21.15.28:0/1747080400 wait complete. 2024-04-23T14:44:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.925+0000 7f7c87813740 20 client.15306 trim_cache size 0 max 16384 2024-04-23T14:44:41.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.944+0000 7f7c87813740 1 Processor -- start 2024-04-23T14:44:41.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.944+0000 7f7c87813740 1 -- start start 2024-04-23T14:44:41.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.944+0000 7f7c87813740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efff9eea30 0x55efff9f1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.944+0000 7f7c87813740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efff9ef350 0x55efffa117b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.944+0000 7f7c87813740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efff9f9a20 0x55efffa11cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.944+0000 7f7c87813740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7c801b2100 con 0x55efff9eea30 2024-04-23T14:44:41.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.944+0000 7f7c87813740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7c801b4980 con 0x55efff9f9a20 2024-04-23T14:44:41.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.944+0000 7f7c87813740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7c801b2770 con 0x55efff9ef350 2024-04-23T14:44:41.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.945+0000 7f7c7de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efff9eea30 0x55efff9f1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.945+0000 7f7c7de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efff9eea30 0x55efff9f1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43838/0 (socket says 172.21.15.28:43838) 2024-04-23T14:44:41.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.945+0000 7f7c7de00640 1 -- 172.21.15.28:0/3536145049 learned_addr learned my addr 172.21.15.28:0/3536145049 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:41.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.945+0000 7f7c7d400640 1 --2- 172.21.15.28:0/3536145049 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efff9ef350 0x55efffa117b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.945+0000 7f7c7e800640 1 --2- 172.21.15.28:0/3536145049 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efff9f9a20 0x55efffa11cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.945+0000 7f7c7de00640 1 -- 172.21.15.28:0/3536145049 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efff9ef350 msgr2=0x55efffa117b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:41.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.945+0000 7f7c7de00640 1 --2- 172.21.15.28:0/3536145049 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efff9ef350 0x55efffa117b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.945+0000 7f7c7de00640 1 -- 172.21.15.28:0/3536145049 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efff9f9a20 msgr2=0x55efffa11cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:41.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.945+0000 7f7c7de00640 1 --2- 172.21.15.28:0/3536145049 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efff9f9a20 0x55efffa11cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.945+0000 7f7c7de00640 1 -- 172.21.15.28:0/3536145049 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efff6663f0 con 0x55efff9eea30 2024-04-23T14:44:41.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.945+0000 7f7c7d400640 1 --2- 172.21.15.28:0/3536145049 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efff9ef350 0x55efffa117b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:44:41.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.946+0000 7f7c7de00640 1 --2- 172.21.15.28:0/3536145049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efff9eea30 0x55efff9f1190 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7c68005100 tx=0x7f7c6803f070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aa43ff080334c78 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:41.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.946+0000 7f7c6e600640 1 -- 172.21.15.28:0/3536145049 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c68058070 con 0x55efff9eea30 2024-04-23T14:44:41.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.946+0000 7f7c6e600640 1 -- 172.21.15.28:0/3536145049 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c68052e10 con 0x55efff9eea30 2024-04-23T14:44:41.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.946+0000 7f7c6e600640 1 -- 172.21.15.28:0/3536145049 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c68035040 con 0x55efff9eea30 2024-04-23T14:44:41.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.947+0000 7f7c87813740 1 -- 172.21.15.28:0/3536145049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efff9eea30 msgr2=0x55efff9f1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:41.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.947+0000 7f7c87813740 1 --2- 172.21.15.28:0/3536145049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efff9eea30 0x55efff9f1190 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7c68005100 tx=0x7f7c6803f070 comp rx=0 tx=0).stop 2024-04-23T14:44:41.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.947+0000 7f7c87813740 1 -- 172.21.15.28:0/3536145049 shutdown_connections 2024-04-23T14:44:41.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.947+0000 7f7c87813740 1 --2- 172.21.15.28:0/3536145049 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efff9f9a20 0x55efffa11cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.947+0000 7f7c87813740 1 --2- 172.21.15.28:0/3536145049 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efff9ef350 0x55efffa117b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.947+0000 7f7c87813740 1 --2- 172.21.15.28:0/3536145049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efff9eea30 0x55efff9f1190 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.947+0000 7f7c87813740 1 -- 172.21.15.28:0/3536145049 >> 172.21.15.28:0/3536145049 conn(0x55efff9ead50 msgr2=0x55efff9e9740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:41.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.947+0000 7f7c87813740 1 -- 172.21.15.28:0/3536145049 shutdown_connections 2024-04-23T14:44:41.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.947+0000 7f7c87813740 1 -- 172.21.15.28:0/3536145049 wait complete. 2024-04-23T14:44:41.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.947+0000 7f7c87813740 1 -- start start 2024-04-23T14:44:41.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.947+0000 7f7c87813740 1 Processor -- start 2024-04-23T14:44:41.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.948+0000 7f7c87813740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efff9eea30 0x55efffa020b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.948+0000 7f7c87813740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efff9ef350 0x55efff9f3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.948+0000 7f7c87813740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efff9f9a20 0x55efff9f3a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.948+0000 7f7c87813740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7c801b2e40 con 0x55efff9f9a20 2024-04-23T14:44:41.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.948+0000 7f7c87813740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7c801b42b0 con 0x55efff9eea30 2024-04-23T14:44:41.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.948+0000 7f7c87813740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7c801b3510 con 0x55efff9ef350 2024-04-23T14:44:41.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.948+0000 7f7c87813740 10 client.0 _ll_register_callbacks cb 0x7f7c68009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:44:41.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.948+0000 7f7c7e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efff9f9a20 0x55efff9f3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.948+0000 7f7c7e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efff9f9a20 0x55efff9f3a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43846/0 (socket says 172.21.15.28:43846) 2024-04-23T14:44:41.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.948+0000 7f7c7e800640 1 -- 172.21.15.28:0/2266873594 learned_addr learned my addr 172.21.15.28:0/2266873594 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:44:41.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.948+0000 7f7c7d400640 1 --2- 172.21.15.28:0/2266873594 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efff9ef350 0x55efff9f3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.948+0000 7f7c7de00640 1 --2- 172.21.15.28:0/2266873594 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efff9eea30 0x55efffa020b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.949+0000 7f7c7de00640 1 -- 172.21.15.28:0/2266873594 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efff9ef350 msgr2=0x55efff9f3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:41.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.949+0000 7f7c7de00640 1 --2- 172.21.15.28:0/2266873594 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efff9ef350 0x55efff9f3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.949+0000 7f7c7de00640 1 -- 172.21.15.28:0/2266873594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efff9f9a20 msgr2=0x55efff9f3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:44:41.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.949+0000 7f7c7de00640 1 --2- 172.21.15.28:0/2266873594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efff9f9a20 0x55efff9f3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.949+0000 7f7c7de00640 1 -- 172.21.15.28:0/2266873594 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efff6260f0 con 0x55efff9eea30 2024-04-23T14:44:41.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.949+0000 7f7c7e800640 1 --2- 172.21.15.28:0/2266873594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efff9f9a20 0x55efff9f3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:44:41.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.949+0000 7f7c7d400640 1 --2- 172.21.15.28:0/2266873594 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efff9ef350 0x55efff9f3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:44:41.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.949+0000 7f7c7de00640 1 --2- 172.21.15.28:0/2266873594 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efff9eea30 0x55efffa020b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7c68053fd0 tx=0x7f7c68005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=310db880b361cc42 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:44:41.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.949+0000 7f7c63400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:44:41.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.949+0000 7f7c63400640 1 -- 172.21.15.28:0/2266873594 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c68061060 con 0x55efff9eea30 2024-04-23T14:44:41.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.949+0000 7f7c63400640 1 -- 172.21.15.28:0/2266873594 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f7c68004c50 con 0x55efff9eea30 2024-04-23T14:44:41.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.949+0000 7f7c87813740 1 -- 172.21.15.28:0/2266873594 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c68052e10 con 0x55efff9eea30 2024-04-23T14:44:41.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.950+0000 7f7c63400640 1 -- 172.21.15.28:0/2266873594 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f7c68058070 con 0x55efff9eea30 2024-04-23T14:44:41.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.950+0000 7f7c87813740 10 client.25442 fetch_fsmap learned FSMap version 20 2024-04-23T14:44:41.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.950+0000 7f7c87813740 1 -- 172.21.15.28:0/2266873594 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55efff919810 con 0x55efff9eea30 2024-04-23T14:44:41.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.950+0000 7f7c63400640 1 -- 172.21.15.28:0/2266873594 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c68052e10 con 0x55efff9eea30 2024-04-23T14:44:41.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.951+0000 7f7c63400640 1 -- 172.21.15.28:0/2266873594 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f7c68032820 con 0x55efff9eea30 2024-04-23T14:44:41.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.951+0000 7f7c87813740 10 client.25442 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:44:41.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.951+0000 7f7c87813740 10 client.25442 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:44:41.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.951+0000 7f7c87813740 20 client.25442 populate_metadata read hostname 'smithi028' 2024-04-23T14:44:41.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.951+0000 7f7c87813740 1 --2- 172.21.15.28:0/2266873594 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55efffa70450 0x55efffa72870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:44:41.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.951+0000 7f7c87813740 4 client.25442 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:44:41.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.951+0000 7f7c87813740 1 -- 172.21.15.28:0/2266873594 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55efff95ba80 con 0x55efffa70450 2024-04-23T14:44:41.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.951+0000 7f7c7d400640 1 --2- 172.21.15.28:0/2266873594 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55efffa70450 0x55efffa72870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:44:41.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.952+0000 7f7c7d400640 1 --2- 172.21.15.28:0/2266873594 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55efffa70450 0x55efffa72870 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e546dd0e099dcd5d server_cookie=ca919b499090e005 in_seq=0 out_seq=0 2024-04-23T14:44:41.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.952+0000 7f7c63400640 10 client.25442 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:44:41.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.952+0000 7f7c63400640 1 -- 172.21.15.28:0/2266873594 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55efff95ba80 con 0x55efffa70450 2024-04-23T14:44:41.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.952+0000 7f7c63400640 10 client.25442 handle_command_reply: tid=0 2024-04-23T14:44:41.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.952+0000 7f7c87813740 1 client.25442 shutdown 2024-04-23T14:44:41.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.952+0000 7f7c87813740 1 -- 172.21.15.28:0/2266873594 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efff9eea30 msgr2=0x55efffa020b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:41.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.952+0000 7f7c87813740 1 --2- 172.21.15.28:0/2266873594 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efff9eea30 0x55efffa020b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7c68053fd0 tx=0x7f7c68005fd0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.952+0000 7f7c87813740 1 -- 172.21.15.28:0/2266873594 shutdown_connections 2024-04-23T14:44:41.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.952+0000 7f7c87813740 1 --2- 172.21.15.28:0/2266873594 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55efffa70450 0x55efffa72870 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.952+0000 7f7c87813740 1 --2- 172.21.15.28:0/2266873594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55efff9f9a20 0x55efff9f3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.952+0000 7f7c87813740 1 --2- 172.21.15.28:0/2266873594 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efff9ef350 0x55efff9f3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.952+0000 7f7c63400640 0 client.25442 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:44:41.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.952+0000 7f7c87813740 1 --2- 172.21.15.28:0/2266873594 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efff9eea30 0x55efffa020b0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.952+0000 7f7c87813740 1 -- 172.21.15.28:0/2266873594 >> 172.21.15.28:0/2266873594 conn(0x55efff9ead50 msgr2=0x55efff9e9fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:41.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.952+0000 7f7c87813740 1 -- 172.21.15.28:0/2266873594 shutdown_connections 2024-04-23T14:44:41.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.952+0000 7f7c87813740 1 -- 172.21.15.28:0/2266873594 wait complete. 2024-04-23T14:44:41.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.952+0000 7f7c87813740 20 client.25442 trim_cache size 0 max 16384 2024-04-23T14:44:41.962 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:44:41.962 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-23T14:44:41.962 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-23T14:44:41.962 INFO:teuthology.orchestra.run.smithi028.stdout: "6fd9d4d6-0c2f-45ce-99e1-d6fd9e726c23": { 2024-04-23T14:44:41.962 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-23T14:44:41.963 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "6fd9d4d6-0c2f-45ce-99e1-d6fd9e726c23", 2024-04-23T14:44:41.963 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-23T14:44:41.963 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:44:41.963 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:44:41.963 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:44:41.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.953+0000 7f7c7f200640 1 -- 172.21.15.28:0/136004410 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7c640a3db0 msgr2=0x7f7c640a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:41.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.953+0000 7f7c7f200640 1 --2- 172.21.15.28:0/136004410 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7c640a3db0 0x7f7c640a6270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7c68009900 tx=0x7f7c68000bc0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.953+0000 7f7c7f200640 1 -- 172.21.15.28:0/136004410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c8010d500 msgr2=0x7f7c801a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:44:41.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.953+0000 7f7c7f200640 1 --2- 172.21.15.28:0/136004410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c8010d500 0x7f7c801a35e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7c7800ea60 tx=0x7f7c7800ef20 comp rx=0 tx=0).stop 2024-04-23T14:44:41.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.953+0000 7f7c7f200640 1 -- 172.21.15.28:0/136004410 shutdown_connections 2024-04-23T14:44:41.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.953+0000 7f7c7f200640 1 --2- 172.21.15.28:0/136004410 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7c640a3db0 0x7f7c640a6270 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.953+0000 7f7c7f200640 1 --2- 172.21.15.28:0/136004410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c80108f40 0x7f7c8019f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.953+0000 7f7c7f200640 1 --2- 172.21.15.28:0/136004410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c8010d500 0x7f7c801a35e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.953+0000 7f7c7f200640 1 --2- 172.21.15.28:0/136004410 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c80108580 0x7f7c8019ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:44:41.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.953+0000 7f7c7f200640 1 -- 172.21.15.28:0/136004410 >> 172.21.15.28:0/136004410 conn(0x7f7c80077e20 msgr2=0x7f7c80078340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:44:41.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.953+0000 7f7c7f200640 1 -- 172.21.15.28:0/136004410 shutdown_connections 2024-04-23T14:44:41.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:44:41.953+0000 7f7c7f200640 1 -- 172.21.15.28:0/136004410 wait complete. 2024-04-23T14:44:41.972 INFO:tasks.cephfs.filesystem:scrub status for tag:6fd9d4d6-0c2f-45ce-99e1-d6fd9e726c23 - {'path': '/', 'tag': '6fd9d4d6-0c2f-45ce-99e1-d6fd9e726c23', 'options': 'recursive,force'} 2024-04-23T14:44:42.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:42 smithi105 ceph-mon[32555]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:44:42.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:42 smithi105 ceph-mon[32555]: scrub queued for path: / 2024-04-23T14:44:42.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:42 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:44:42.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:42 smithi105 ceph-mon[32555]: scrub summary: active paths [/] 2024-04-23T14:44:42.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:42 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:44:42.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:42 smithi038 ceph-mon[32348]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:44:42.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:42 smithi038 ceph-mon[32348]: scrub queued for path: / 2024-04-23T14:44:42.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:42 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:44:42.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:42 smithi038 ceph-mon[32348]: scrub summary: active paths [/] 2024-04-23T14:44:42.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:42 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:44:42.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:42 smithi028 ceph-mon[27810]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:44:42.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:42 smithi028 ceph-mon[27810]: scrub queued for path: / 2024-04-23T14:44:42.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:42 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:44:42.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:42 smithi028 ceph-mon[27810]: scrub summary: active paths [/] 2024-04-23T14:44:42.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:42 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:44:44.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:44 smithi105 ceph-mon[32555]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s 2024-04-23T14:44:44.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:44 smithi038 ceph-mon[32348]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s 2024-04-23T14:44:44.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:44 smithi028 ceph-mon[27810]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s 2024-04-23T14:44:45.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:45 smithi105 ceph-mon[32555]: scrub summary: idle 2024-04-23T14:44:45.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:45 smithi038 ceph-mon[32348]: scrub summary: idle 2024-04-23T14:44:45.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:45 smithi028 ceph-mon[27810]: scrub summary: idle 2024-04-23T14:44:46.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:46 smithi105 ceph-mon[32555]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-23T14:44:46.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:46 smithi038 ceph-mon[32348]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-23T14:44:46.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:46 smithi028 ceph-mon[27810]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-23T14:44:48.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:48 smithi105 ceph-mon[32555]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 682 B/s wr, 3 op/s 2024-04-23T14:44:48.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:48 smithi038 ceph-mon[32348]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 682 B/s wr, 3 op/s 2024-04-23T14:44:48.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:48 smithi028 ceph-mon[27810]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 682 B/s wr, 3 op/s 2024-04-23T14:44:50.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:50 smithi105 ceph-mon[32555]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 682 B/s wr, 2 op/s 2024-04-23T14:44:50.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:50 smithi038 ceph-mon[32348]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 682 B/s wr, 2 op/s 2024-04-23T14:44:50.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:50 smithi028 ceph-mon[27810]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 682 B/s wr, 2 op/s 2024-04-23T14:44:52.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:52 smithi105 ceph-mon[32555]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 682 B/s wr, 2 op/s 2024-04-23T14:44:52.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:52 smithi038 ceph-mon[32348]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 682 B/s wr, 2 op/s 2024-04-23T14:44:52.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:52 smithi028 ceph-mon[27810]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 682 B/s wr, 2 op/s 2024-04-23T14:44:54.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:54 smithi105 ceph-mon[32555]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 682 B/s wr, 3 op/s 2024-04-23T14:44:54.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:54 smithi038 ceph-mon[32348]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 682 B/s wr, 3 op/s 2024-04-23T14:44:54.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:54 smithi028 ceph-mon[27810]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 682 B/s wr, 3 op/s 2024-04-23T14:44:56.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:56 smithi105 ceph-mon[32555]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 682 B/s wr, 2 op/s 2024-04-23T14:44:56.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:56 smithi038 ceph-mon[32348]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 682 B/s wr, 2 op/s 2024-04-23T14:44:56.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:56 smithi028 ceph-mon[27810]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 682 B/s wr, 2 op/s 2024-04-23T14:44:58.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:44:58 smithi105 ceph-mon[32555]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 682 B/s wr, 2 op/s 2024-04-23T14:44:58.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:44:58 smithi038 ceph-mon[32348]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 682 B/s wr, 2 op/s 2024-04-23T14:44:58.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:44:58 smithi028 ceph-mon[27810]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 682 B/s wr, 2 op/s 2024-04-23T14:45:00.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:00 smithi105 ceph-mon[32555]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:45:00.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:00 smithi038 ceph-mon[32348]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:45:00.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:00 smithi028 ceph-mon[27810]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:45:01.375 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:45:01 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:45:01.008+0000 7f0a38600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T14:45:01.375 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:45:01 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:45:01.008+0000 7f0a38600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T14:45:01.375 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:45:01 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:45:01.008+0000 7f0a38600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T14:45:01.375 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:45:01 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:45:01.008+0000 7f0a38600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T14:45:01.375 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:45:01 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:45:01.008+0000 7f0a38600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T14:45:02.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:02 smithi105 ceph-mon[32555]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:45:02.787 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:02 smithi105 ceph-mon[32555]: mgrmap e26: x(active, since 60s), standbys: z, y 2024-04-23T14:45:02.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:02 smithi038 ceph-mon[32348]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:45:02.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:02 smithi038 ceph-mon[32348]: mgrmap e26: x(active, since 60s), standbys: z, y 2024-04-23T14:45:02.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:02 smithi028 ceph-mon[27810]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-23T14:45:02.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:02 smithi028 ceph-mon[27810]: mgrmap e26: x(active, since 60s), standbys: z, y 2024-04-23T14:45:04.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:04 smithi105 ceph-mon[32555]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-04-23T14:45:04.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:04 smithi038 ceph-mon[32348]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-04-23T14:45:04.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:04 smithi028 ceph-mon[27810]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-04-23T14:45:06.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:06 smithi105 ceph-mon[32555]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-04-23T14:45:06.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:06 smithi038 ceph-mon[32348]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-04-23T14:45:06.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:06 smithi028 ceph-mon[27810]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-04-23T14:45:08.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:08 smithi105 ceph-mon[32555]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-04-23T14:45:08.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:08 smithi038 ceph-mon[32348]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-04-23T14:45:08.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:08 smithi028 ceph-mon[27810]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-04-23T14:45:10.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:10 smithi038 ceph-mon[32348]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-04-23T14:45:10.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:10 smithi028 ceph-mon[27810]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-04-23T14:45:11.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:10 smithi105 ceph-mon[32555]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-04-23T14:45:11.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:11 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:45:11.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:11 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:45:11.974 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:45:11.974 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:45:12.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:11 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:45:12.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.203+0000 7f92a7200640 1 -- 172.21.15.28:0/3023727906 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92a8104cd0 msgr2=0x7f92a8108d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:12.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.203+0000 7f92a7200640 1 --2- 172.21.15.28:0/3023727906 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92a8104cd0 0x7f92a8108d10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9294009900 tx=0x7f929402a3f0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.203+0000 7f92a7200640 1 -- 172.21.15.28:0/3023727906 shutdown_connections 2024-04-23T14:45:12.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.203+0000 7f92a7200640 1 --2- 172.21.15.28:0/3023727906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92a81093f0 0x7f92a8110fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.203+0000 7f92a7200640 1 --2- 172.21.15.28:0/3023727906 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92a8104cd0 0x7f92a8108d10 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.203+0000 7f92a7200640 1 --2- 172.21.15.28:0/3023727906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92a8104320 0x7f92a8104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.203+0000 7f92a7200640 1 -- 172.21.15.28:0/3023727906 >> 172.21.15.28:0/3023727906 conn(0x7f92a80ffb40 msgr2=0x7f92a8101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:12.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.203+0000 7f92a7200640 1 -- 172.21.15.28:0/3023727906 shutdown_connections 2024-04-23T14:45:12.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.204+0000 7f92a7200640 1 -- 172.21.15.28:0/3023727906 wait complete. 2024-04-23T14:45:12.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.205+0000 7f92a7200640 1 Processor -- start 2024-04-23T14:45:12.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.205+0000 7f92a7200640 1 -- start start 2024-04-23T14:45:12.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.206+0000 7f92a7200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92a8104320 0x7f92a81a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:12.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.206+0000 7f92a7200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92a81093f0 0x7f92a81a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:12.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.206+0000 7f92a7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92a819a7d0 0x7f92a819ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:12.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.206+0000 7f92a7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f92a81144b0 con 0x7f92a819a7d0 2024-04-23T14:45:12.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.206+0000 7f92a7200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f92a8114330 con 0x7f92a8104320 2024-04-23T14:45:12.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.206+0000 7f92a7200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f92a8114630 con 0x7f92a81093f0 2024-04-23T14:45:12.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.206+0000 7f92a5400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92a81093f0 0x7f92a81a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:12.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.206+0000 7f92a5400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92a81093f0 0x7f92a81a0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:39874/0 (socket says 172.21.15.28:39874) 2024-04-23T14:45:12.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.206+0000 7f92a5400640 1 -- 172.21.15.28:0/760221870 learned_addr learned my addr 172.21.15.28:0/760221870 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:12.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.206+0000 7f92a5e00640 1 --2- 172.21.15.28:0/760221870 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92a8104320 0x7f92a81a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:12.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.206+0000 7f92a6800640 1 --2- 172.21.15.28:0/760221870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92a819a7d0 0x7f92a819ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:12.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.207+0000 7f92a5400640 1 -- 172.21.15.28:0/760221870 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92a8104320 msgr2=0x7f92a81a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:12.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.207+0000 7f92a5400640 1 --2- 172.21.15.28:0/760221870 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92a8104320 0x7f92a81a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.207+0000 7f92a5400640 1 -- 172.21.15.28:0/760221870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92a819a7d0 msgr2=0x7f92a819ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:12.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.207+0000 7f92a5400640 1 --2- 172.21.15.28:0/760221870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92a819a7d0 0x7f92a819ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.207+0000 7f92a5400640 1 -- 172.21.15.28:0/760221870 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92a819b360 con 0x7f92a81093f0 2024-04-23T14:45:12.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.207+0000 7f92a6800640 1 --2- 172.21.15.28:0/760221870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92a819a7d0 0x7f92a819ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:45:12.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.207+0000 7f92a5400640 1 --2- 172.21.15.28:0/760221870 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92a81093f0 0x7f92a81a0ba0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f929402a910 tx=0x7f9294003b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:12.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.207+0000 7f929ea00640 1 -- 172.21.15.28:0/760221870 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f929403f070 con 0x7f92a81093f0 2024-04-23T14:45:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.207+0000 7f929ea00640 1 -- 172.21.15.28:0/760221870 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9294031c90 con 0x7f92a81093f0 2024-04-23T14:45:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.207+0000 7f92a7200640 1 -- 172.21.15.28:0/760221870 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92a819b5f0 con 0x7f92a81093f0 2024-04-23T14:45:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.208+0000 7f929ea00640 1 -- 172.21.15.28:0/760221870 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9294035030 con 0x7f92a81093f0 2024-04-23T14:45:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.208+0000 7f92a7200640 1 -- 172.21.15.28:0/760221870 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92a81a77d0 con 0x7f92a81093f0 2024-04-23T14:45:12.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.212+0000 7f92af839740 1 Processor -- start 2024-04-23T14:45:12.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.213+0000 7f92af839740 1 -- start start 2024-04-23T14:45:12.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.213+0000 7f929ea00640 1 -- 172.21.15.28:0/760221870 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138879+0+0 (secure 0 0 0) 0x7f9294042070 con 0x7f92a81093f0 2024-04-23T14:45:12.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.213+0000 7f92af839740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd6bac5a30 0x55cd6bac5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:12.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.213+0000 7f92af839740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd6bac6350 0x55cd6bad04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:12.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.213+0000 7f92af839740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd6bad0a20 0x55cd6bad2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:12.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.213+0000 7f92af839740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cd6ba2b1f0 con 0x55cd6bac5a30 2024-04-23T14:45:12.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.213+0000 7f92af839740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55cd6b9d1350 con 0x55cd6bad0a20 2024-04-23T14:45:12.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.213+0000 7f92af839740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55cd6ba271b0 con 0x55cd6bac6350 2024-04-23T14:45:12.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.213+0000 7f92a5e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd6bac6350 0x55cd6bad04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:12.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.213+0000 7f92a5e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd6bac6350 0x55cd6bad04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:39888/0 (socket says 172.21.15.28:39888) 2024-04-23T14:45:12.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.213+0000 7f92a5e00640 1 -- 172.21.15.28:0/16498778 learned_addr learned my addr 172.21.15.28:0/16498778 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:12.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.214+0000 7f92a5400640 1 --2- 172.21.15.28:0/16498778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd6bad0a20 0x55cd6bad2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:12.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.214+0000 7f92a6800640 1 --2- 172.21.15.28:0/16498778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd6bac5a30 0x55cd6bac5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:12.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.214+0000 7f92a5e00640 1 -- 172.21.15.28:0/16498778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd6bad0a20 msgr2=0x55cd6bad2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:12.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.214+0000 7f92a5e00640 1 --2- 172.21.15.28:0/16498778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd6bad0a20 0x55cd6bad2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.214+0000 7f92a5e00640 1 -- 172.21.15.28:0/16498778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd6bac5a30 msgr2=0x55cd6bac5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:12.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.215+0000 7f92a5e00640 1 --2- 172.21.15.28:0/16498778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd6bac5a30 0x55cd6bac5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.215+0000 7f92a5e00640 1 -- 172.21.15.28:0/16498778 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd6ba25f50 con 0x55cd6bac6350 2024-04-23T14:45:12.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.215+0000 7f92a6800640 1 --2- 172.21.15.28:0/16498778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd6bac5a30 0x55cd6bac5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:45:12.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.215+0000 7f929ea00640 1 --2- 172.21.15.28:0/760221870 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f928c0a3f40 0x7f928c0a6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:12.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.215+0000 7f929ea00640 1 -- 172.21.15.28:0/760221870 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f92940f1120 con 0x7f92a81093f0 2024-04-23T14:45:12.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.215+0000 7f92a5e00640 1 --2- 172.21.15.28:0/16498778 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd6bac6350 0x55cd6bad04e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9298009c80 tx=0x7f92980117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5d1e421f66d261d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:12.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.216+0000 7f9293000640 1 -- 172.21.15.28:0/16498778 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9298012450 con 0x55cd6bac6350 2024-04-23T14:45:12.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.216+0000 7f92a6800640 1 --2- 172.21.15.28:0/760221870 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f928c0a3f40 0x7f928c0a6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:12.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.216+0000 7f9293000640 1 -- 172.21.15.28:0/16498778 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9298012ad0 con 0x55cd6bac6350 2024-04-23T14:45:12.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.216+0000 7f9293000640 1 -- 172.21.15.28:0/16498778 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f929800fc00 con 0x55cd6bac6350 2024-04-23T14:45:12.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.217+0000 7f92af839740 1 -- 172.21.15.28:0/16498778 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd6bac6350 msgr2=0x55cd6bad04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:12.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.217+0000 7f92af839740 1 --2- 172.21.15.28:0/16498778 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd6bac6350 0x55cd6bad04e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9298009c80 tx=0x7f92980117b0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.217+0000 7f92af839740 1 -- 172.21.15.28:0/16498778 shutdown_connections 2024-04-23T14:45:12.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.217+0000 7f92af839740 1 --2- 172.21.15.28:0/16498778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd6bad0a20 0x55cd6bad2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.217+0000 7f92af839740 1 --2- 172.21.15.28:0/16498778 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd6bac6350 0x55cd6bad04e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.217+0000 7f92af839740 1 --2- 172.21.15.28:0/16498778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd6bac5a30 0x55cd6bac5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.217+0000 7f92af839740 1 -- 172.21.15.28:0/16498778 >> 172.21.15.28:0/16498778 conn(0x55cd6bac1d50 msgr2=0x55cd6bac2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:12.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.217+0000 7f92a6800640 1 --2- 172.21.15.28:0/760221870 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f928c0a3f40 0x7f928c0a6400 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f92a0005fd0 tx=0x7f92a0009500 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:12.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.218+0000 7f92af839740 1 -- 172.21.15.28:0/16498778 shutdown_connections 2024-04-23T14:45:12.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.218+0000 7f92af839740 1 -- 172.21.15.28:0/16498778 wait complete. 2024-04-23T14:45:12.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.218+0000 7f92af839740 1 -- start start 2024-04-23T14:45:12.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.219+0000 7f92af839740 1 Processor -- start 2024-04-23T14:45:12.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.219+0000 7f92af839740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd6bac5a30 0x55cd6bacd980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:12.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.219+0000 7f92af839740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd6bac6350 0x55cd6bacdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:12.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.219+0000 7f92af839740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd6bad0a20 0x55cd6bace400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:12.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.219+0000 7f92af839740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cd6badc020 con 0x55cd6bac5a30 2024-04-23T14:45:12.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.219+0000 7f92af839740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55cd6b9e4690 con 0x55cd6bad0a20 2024-04-23T14:45:12.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.219+0000 7f92af839740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55cd6badc1a0 con 0x55cd6bac6350 2024-04-23T14:45:12.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.219+0000 7f92af839740 10 client.0 _ll_register_callbacks cb 0x55cd6ba20650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:45:12.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.220+0000 7f92a5400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd6bac6350 0x55cd6bacdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:12.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.220+0000 7f92a5400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd6bac6350 0x55cd6bacdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:39900/0 (socket says 172.21.15.28:39900) 2024-04-23T14:45:12.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.220+0000 7f92a6800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd6bad0a20 0x55cd6bace400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:12.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.220+0000 7f92a5400640 1 -- 172.21.15.28:0/4102683995 learned_addr learned my addr 172.21.15.28:0/4102683995 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:12.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.220+0000 7f92a5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd6bac5a30 0x55cd6bacd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:12.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.220+0000 7f92a5400640 1 -- 172.21.15.28:0/4102683995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd6bad0a20 msgr2=0x55cd6bace400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:12.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.220+0000 7f92a5400640 1 --2- 172.21.15.28:0/4102683995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd6bad0a20 0x55cd6bace400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.220+0000 7f92a5400640 1 -- 172.21.15.28:0/4102683995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd6bac5a30 msgr2=0x55cd6bacd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:12.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.220+0000 7f92a5400640 1 --2- 172.21.15.28:0/4102683995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd6bac5a30 0x55cd6bacd980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.220+0000 7f92a5400640 1 -- 172.21.15.28:0/4102683995 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd6b9e3920 con 0x55cd6bac6350 2024-04-23T14:45:12.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.220+0000 7f92a5400640 1 --2- 172.21.15.28:0/4102683995 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd6bac6350 0x55cd6bacdec0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9294006010 tx=0x7f929403b720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d789569b951d51e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:12.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.220+0000 7f9287400640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:45:12.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.221+0000 7f9287400640 1 -- 172.21.15.28:0/4102683995 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9294049050 con 0x55cd6bac6350 2024-04-23T14:45:12.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.221+0000 7f92af839740 1 -- 172.21.15.28:0/4102683995 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9298012ad0 con 0x55cd6bac6350 2024-04-23T14:45:12.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.221+0000 7f9287400640 1 -- 172.21.15.28:0/4102683995 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f92940f30d0 con 0x55cd6bac6350 2024-04-23T14:45:12.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.221+0000 7f9287400640 1 -- 172.21.15.28:0/4102683995 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f92940bca30 con 0x55cd6bac6350 2024-04-23T14:45:12.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.221+0000 7f92af839740 10 client.25468 fetch_fsmap learned FSMap version 20 2024-04-23T14:45:12.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.221+0000 7f92af839740 1 -- 172.21.15.28:0/4102683995 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55cd6b8077d0 con 0x55cd6bac6350 2024-04-23T14:45:12.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.221+0000 7f9287400640 1 -- 172.21.15.28:0/4102683995 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9298012ad0 con 0x55cd6bac6350 2024-04-23T14:45:12.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.222+0000 7f9287400640 1 -- 172.21.15.28:0/4102683995 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f92940d6240 con 0x55cd6bac6350 2024-04-23T14:45:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.222+0000 7f92af839740 10 client.25468 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:45:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.222+0000 7f92af839740 10 client.25468 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:45:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.222+0000 7f92af839740 20 client.25468 populate_metadata read hostname 'smithi028' 2024-04-23T14:45:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.222+0000 7f92af839740 1 --2- 172.21.15.28:0/4102683995 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55cd6bae94c0 0x55cd6baeb930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.222+0000 7f92af839740 4 client.25468 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:45:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.222+0000 7f92af839740 1 -- 172.21.15.28:0/4102683995 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cd6ba38c10 con 0x55cd6bae94c0 2024-04-23T14:45:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.223+0000 7f92a5e00640 1 --2- 172.21.15.28:0/4102683995 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55cd6bae94c0 0x55cd6baeb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.223+0000 7f9287400640 10 client.25468 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:45:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.223+0000 7f92a5e00640 1 --2- 172.21.15.28:0/4102683995 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55cd6bae94c0 0x55cd6baeb930 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62f98a124dcc989 server_cookie=e5ae7522aa62414 in_seq=0 out_seq=0 2024-04-23T14:45:12.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.224+0000 7f9287400640 1 -- 172.21.15.28:0/4102683995 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cd6ba38c10 con 0x55cd6bae94c0 2024-04-23T14:45:12.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.224+0000 7f9287400640 10 client.25468 handle_command_reply: tid=0 2024-04-23T14:45:12.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.224+0000 7f92af839740 1 client.25468 shutdown 2024-04-23T14:45:12.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.225+0000 7f92af839740 1 -- 172.21.15.28:0/4102683995 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd6bac6350 msgr2=0x55cd6bacdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:12.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.225+0000 7f92af839740 1 --2- 172.21.15.28:0/4102683995 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd6bac6350 0x55cd6bacdec0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9294006010 tx=0x7f929403b720 comp rx=0 tx=0).stop 2024-04-23T14:45:12.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.225+0000 7f92af839740 1 -- 172.21.15.28:0/4102683995 shutdown_connections 2024-04-23T14:45:12.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.225+0000 7f92af839740 1 --2- 172.21.15.28:0/4102683995 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55cd6bae94c0 0x55cd6baeb930 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.225+0000 7f92af839740 1 --2- 172.21.15.28:0/4102683995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd6bad0a20 0x55cd6bace400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.225+0000 7f9287400640 0 client.25468 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:45:12.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.225+0000 7f92af839740 1 --2- 172.21.15.28:0/4102683995 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd6bac6350 0x55cd6bacdec0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.225+0000 7f92af839740 1 --2- 172.21.15.28:0/4102683995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd6bac5a30 0x55cd6bacd980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.225+0000 7f92af839740 1 -- 172.21.15.28:0/4102683995 >> 172.21.15.28:0/4102683995 conn(0x55cd6bac1d50 msgr2=0x55cd6bac0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:12.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.226+0000 7f92af839740 1 -- 172.21.15.28:0/4102683995 shutdown_connections 2024-04-23T14:45:12.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.226+0000 7f92af839740 1 -- 172.21.15.28:0/4102683995 wait complete. 2024-04-23T14:45:12.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.226+0000 7f92af839740 20 client.25468 trim_cache size 0 max 16384 2024-04-23T14:45:12.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.263+0000 7f92af839740 1 Processor -- start 2024-04-23T14:45:12.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.264+0000 7f92af839740 1 -- start start 2024-04-23T14:45:12.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.264+0000 7f92af839740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd6bac5a30 0x55cd6bac81d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:12.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.264+0000 7f92af839740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd6bac6350 0x55cd6baf11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:12.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.264+0000 7f92af839740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd6bad0a20 0x55cd6baf1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:12.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.264+0000 7f92af839740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f92a81a98f0 con 0x55cd6bad0a20 2024-04-23T14:45:12.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.264+0000 7f92af839740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f92a81ac150 con 0x55cd6bac5a30 2024-04-23T14:45:12.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.264+0000 7f92af839740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f92a81a9f40 con 0x55cd6bac6350 2024-04-23T14:45:12.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.265+0000 7f92a5400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd6bac6350 0x55cd6baf11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:12.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.265+0000 7f92a5e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd6bac5a30 0x55cd6bac81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:12.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.265+0000 7f92a5e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd6bac5a30 0x55cd6bac81d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:54858/0 (socket says 172.21.15.28:54858) 2024-04-23T14:45:12.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.265+0000 7f92a5e00640 1 -- 172.21.15.28:0/3901324827 learned_addr learned my addr 172.21.15.28:0/3901324827 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:12.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.265+0000 7f92a6800640 1 --2- 172.21.15.28:0/3901324827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd6bad0a20 0x55cd6baf1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:12.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.266+0000 7f92a5400640 1 -- 172.21.15.28:0/3901324827 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd6bac5a30 msgr2=0x55cd6bac81d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:12.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.266+0000 7f92a5400640 1 --2- 172.21.15.28:0/3901324827 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd6bac5a30 0x55cd6bac81d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.266+0000 7f92a5400640 1 -- 172.21.15.28:0/3901324827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd6bad0a20 msgr2=0x55cd6baf1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:12.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.266+0000 7f92a5400640 1 --2- 172.21.15.28:0/3901324827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd6bad0a20 0x55cd6baf1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.266+0000 7f92a5400640 1 -- 172.21.15.28:0/3901324827 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd6bac6180 con 0x55cd6bac6350 2024-04-23T14:45:12.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.266+0000 7f92a5e00640 1 --2- 172.21.15.28:0/3901324827 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd6bac5a30 0x55cd6bac81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:45:12.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.266+0000 7f92a6800640 1 --2- 172.21.15.28:0/3901324827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd6bad0a20 0x55cd6baf1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:45:12.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.266+0000 7f92a5400640 1 --2- 172.21.15.28:0/3901324827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd6bac6350 0x55cd6baf11d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9294031f80 tx=0x7f9294005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a721675867b086e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:12.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.267+0000 7f9292600640 1 -- 172.21.15.28:0/3901324827 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9294049070 con 0x55cd6bac6350 2024-04-23T14:45:12.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.267+0000 7f9292600640 1 -- 172.21.15.28:0/3901324827 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92940bd5f0 con 0x55cd6bac6350 2024-04-23T14:45:12.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.267+0000 7f9292600640 1 -- 172.21.15.28:0/3901324827 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92940bc850 con 0x55cd6bac6350 2024-04-23T14:45:12.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.268+0000 7f92af839740 1 -- 172.21.15.28:0/3901324827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd6bac6350 msgr2=0x55cd6baf11d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:12.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.268+0000 7f92af839740 1 --2- 172.21.15.28:0/3901324827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd6bac6350 0x55cd6baf11d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9294031f80 tx=0x7f9294005d50 comp rx=0 tx=0).stop 2024-04-23T14:45:12.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.268+0000 7f92af839740 1 -- 172.21.15.28:0/3901324827 shutdown_connections 2024-04-23T14:45:12.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.268+0000 7f92af839740 1 --2- 172.21.15.28:0/3901324827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd6bad0a20 0x55cd6baf1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.268+0000 7f92af839740 1 --2- 172.21.15.28:0/3901324827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd6bac6350 0x55cd6baf11d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.268+0000 7f92af839740 1 --2- 172.21.15.28:0/3901324827 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd6bac5a30 0x55cd6bac81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.268+0000 7f92af839740 1 -- 172.21.15.28:0/3901324827 >> 172.21.15.28:0/3901324827 conn(0x55cd6bac1d50 msgr2=0x55cd6bac0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:12.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.268+0000 7f92af839740 1 -- 172.21.15.28:0/3901324827 shutdown_connections 2024-04-23T14:45:12.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.268+0000 7f92af839740 1 -- 172.21.15.28:0/3901324827 wait complete. 2024-04-23T14:45:12.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.269+0000 7f92af839740 1 -- start start 2024-04-23T14:45:12.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.269+0000 7f92af839740 1 Processor -- start 2024-04-23T14:45:12.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.270+0000 7f92af839740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd6bac5a30 0x55cd6bae13c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:12.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.270+0000 7f92af839740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd6bad0a20 0x55cd6bae1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:12.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.270+0000 7f92af839740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd6bae94c0 0x55cd6bade390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:12.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.270+0000 7f92af839740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f92a81aa610 con 0x55cd6bae94c0 2024-04-23T14:45:12.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.270+0000 7f92af839740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f92a81aba80 con 0x55cd6bac5a30 2024-04-23T14:45:12.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.270+0000 7f92af839740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f92a81aace0 con 0x55cd6bad0a20 2024-04-23T14:45:12.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.270+0000 7f92af839740 10 client.0 _ll_register_callbacks cb 0x7f92940ee730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:45:12.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.270+0000 7f92a5e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd6bac5a30 0x55cd6bae13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:12.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.270+0000 7f92a5e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd6bac5a30 0x55cd6bae13c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:54866/0 (socket says 172.21.15.28:54866) 2024-04-23T14:45:12.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.270+0000 7f92a5e00640 1 -- 172.21.15.28:0/1724713079 learned_addr learned my addr 172.21.15.28:0/1724713079 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:12.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.270+0000 7f92a5400640 1 --2- 172.21.15.28:0/1724713079 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd6bad0a20 0x55cd6bae1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:12.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.271+0000 7f92a6800640 1 --2- 172.21.15.28:0/1724713079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd6bae94c0 0x55cd6bade390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:12.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.271+0000 7f92a5e00640 1 -- 172.21.15.28:0/1724713079 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd6bad0a20 msgr2=0x55cd6bae1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:12.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.271+0000 7f92a5e00640 1 --2- 172.21.15.28:0/1724713079 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd6bad0a20 0x55cd6bae1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.271+0000 7f92a5e00640 1 -- 172.21.15.28:0/1724713079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd6bae94c0 msgr2=0x55cd6bade390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:12.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.271+0000 7f92a5e00640 1 --2- 172.21.15.28:0/1724713079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd6bae94c0 0x55cd6bade390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.271+0000 7f92a5e00640 1 -- 172.21.15.28:0/1724713079 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd6b73d3f0 con 0x55cd6bac5a30 2024-04-23T14:45:12.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.271+0000 7f92a6800640 1 --2- 172.21.15.28:0/1724713079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd6bae94c0 0x55cd6bade390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:45:12.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.272+0000 7f92a5e00640 1 --2- 172.21.15.28:0/1724713079 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd6bac5a30 0x55cd6bae13c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f929801c040 tx=0x7f9298005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81fad3f16cf7e788 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:12.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.272+0000 7f9287400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:45:12.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.272+0000 7f9287400640 1 -- 172.21.15.28:0/1724713079 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9298005230 con 0x55cd6bac5a30 2024-04-23T14:45:12.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.273+0000 7f92af839740 1 -- 172.21.15.28:0/1724713079 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92940bd5f0 con 0x55cd6bac5a30 2024-04-23T14:45:12.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.273+0000 7f9287400640 1 -- 172.21.15.28:0/1724713079 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f92980053d0 con 0x55cd6bac5a30 2024-04-23T14:45:12.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.275+0000 7f9287400640 1 -- 172.21.15.28:0/1724713079 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f92980055d0 con 0x55cd6bac5a30 2024-04-23T14:45:12.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.275+0000 7f92af839740 10 client.25466 fetch_fsmap learned FSMap version 20 2024-04-23T14:45:12.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.275+0000 7f92af839740 1 -- 172.21.15.28:0/1724713079 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55cd6b6fd0f0 con 0x55cd6bac5a30 2024-04-23T14:45:12.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.275+0000 7f9287400640 1 -- 172.21.15.28:0/1724713079 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92940bd5f0 con 0x55cd6bac5a30 2024-04-23T14:45:12.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.276+0000 7f9287400640 1 -- 172.21.15.28:0/1724713079 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f929802a710 con 0x55cd6bac5a30 2024-04-23T14:45:12.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.276+0000 7f92af839740 10 client.25466 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:45:12.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.276+0000 7f92af839740 10 client.25466 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:45:12.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.276+0000 7f92af839740 20 client.25466 populate_metadata read hostname 'smithi028' 2024-04-23T14:45:12.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.276+0000 7f92af839740 1 --2- 172.21.15.28:0/1724713079 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55cd6bb50690 0x55cd6bb52ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:12.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.276+0000 7f92af839740 4 client.25466 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:45:12.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.276+0000 7f92af839740 1 -- 172.21.15.28:0/1724713079 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55cd6ba32a80 con 0x55cd6bb50690 2024-04-23T14:45:12.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.276+0000 7f92a5400640 1 --2- 172.21.15.28:0/1724713079 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55cd6bb50690 0x55cd6bb52ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:12.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.277+0000 7f92a5400640 1 --2- 172.21.15.28:0/1724713079 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55cd6bb50690 0x55cd6bb52ab0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a912904d5f381985 server_cookie=954bc9c0f0287979 in_seq=0 out_seq=0 2024-04-23T14:45:12.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.277+0000 7f9287400640 10 client.25466 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:45:12.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.277+0000 7f9287400640 1 -- 172.21.15.28:0/1724713079 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55cd6ba32a80 con 0x55cd6bb50690 2024-04-23T14:45:12.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.277+0000 7f9287400640 10 client.25466 handle_command_reply: tid=0 2024-04-23T14:45:12.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.278+0000 7f92af839740 1 client.25466 shutdown 2024-04-23T14:45:12.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.278+0000 7f92af839740 1 -- 172.21.15.28:0/1724713079 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd6bac5a30 msgr2=0x55cd6bae13c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:12.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.278+0000 7f92af839740 1 --2- 172.21.15.28:0/1724713079 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd6bac5a30 0x55cd6bae13c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f929801c040 tx=0x7f9298005840 comp rx=0 tx=0).stop 2024-04-23T14:45:12.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.278+0000 7f92af839740 1 -- 172.21.15.28:0/1724713079 shutdown_connections 2024-04-23T14:45:12.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.278+0000 7f92af839740 1 --2- 172.21.15.28:0/1724713079 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55cd6bb50690 0x55cd6bb52ab0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.278+0000 7f92af839740 1 --2- 172.21.15.28:0/1724713079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd6bae94c0 0x55cd6bade390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.278+0000 7f92af839740 1 --2- 172.21.15.28:0/1724713079 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cd6bad0a20 0x55cd6bae1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.278+0000 7f92af839740 1 --2- 172.21.15.28:0/1724713079 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cd6bac5a30 0x55cd6bae13c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.278+0000 7f92af839740 1 -- 172.21.15.28:0/1724713079 >> 172.21.15.28:0/1724713079 conn(0x55cd6bac1d50 msgr2=0x55cd6bac0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:12.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.278+0000 7f9287400640 0 client.25466 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:45:12.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.278+0000 7f92af839740 1 -- 172.21.15.28:0/1724713079 shutdown_connections 2024-04-23T14:45:12.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.278+0000 7f92af839740 1 -- 172.21.15.28:0/1724713079 wait complete. 2024-04-23T14:45:12.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.278+0000 7f92af839740 20 client.25466 trim_cache size 0 max 16384 2024-04-23T14:45:12.280 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:45:12.280 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-23T14:45:12.280 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-23T14:45:12.280 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:45:12.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.279+0000 7f92a7200640 1 -- 172.21.15.28:0/760221870 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f928c0a3f40 msgr2=0x7f928c0a6400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:12.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.279+0000 7f92a7200640 1 --2- 172.21.15.28:0/760221870 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f928c0a3f40 0x7f928c0a6400 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f92a0005fd0 tx=0x7f92a0009500 comp rx=0 tx=0).stop 2024-04-23T14:45:12.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.279+0000 7f92a7200640 1 -- 172.21.15.28:0/760221870 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92a81093f0 msgr2=0x7f92a81a0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:12.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.279+0000 7f92a7200640 1 --2- 172.21.15.28:0/760221870 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92a81093f0 0x7f92a81a0ba0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f929402a910 tx=0x7f9294003b20 comp rx=0 tx=0).stop 2024-04-23T14:45:12.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.279+0000 7f92a7200640 1 -- 172.21.15.28:0/760221870 shutdown_connections 2024-04-23T14:45:12.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.279+0000 7f92a7200640 1 --2- 172.21.15.28:0/760221870 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f928c0a3f40 0x7f928c0a6400 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.279+0000 7f92a7200640 1 --2- 172.21.15.28:0/760221870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92a819a7d0 0x7f92a819ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.279+0000 7f92a7200640 1 --2- 172.21.15.28:0/760221870 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92a81093f0 0x7f92a81a0ba0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.279+0000 7f92a7200640 1 --2- 172.21.15.28:0/760221870 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92a8104320 0x7f92a81a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.279+0000 7f92a7200640 1 -- 172.21.15.28:0/760221870 >> 172.21.15.28:0/760221870 conn(0x7f92a80ffb40 msgr2=0x7f92a8101e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:12.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.280+0000 7f92a7200640 1 -- 172.21.15.28:0/760221870 shutdown_connections 2024-04-23T14:45:12.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.280+0000 7f92a7200640 1 -- 172.21.15.28:0/760221870 wait complete. 2024-04-23T14:45:12.299 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-23T14:45:12.300 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:45:12.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.517+0000 7f648ec00640 1 -- 172.21.15.28:0/1580560705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6488102230 msgr2=0x7f6488102610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:12.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.517+0000 7f648ec00640 1 --2- 172.21.15.28:0/1580560705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6488102230 0x7f6488102610 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6478009860 tx=0x7f647802a340 comp rx=0 tx=0).stop 2024-04-23T14:45:12.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.517+0000 7f648ec00640 1 -- 172.21.15.28:0/1580560705 shutdown_connections 2024-04-23T14:45:12.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.517+0000 7f648ec00640 1 --2- 172.21.15.28:0/1580560705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6488107300 0x7f648810eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.517+0000 7f648ec00640 1 --2- 172.21.15.28:0/1580560705 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6488102be0 0x7f6488106c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.517+0000 7f648ec00640 1 --2- 172.21.15.28:0/1580560705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6488102230 0x7f6488102610 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.517+0000 7f648ec00640 1 -- 172.21.15.28:0/1580560705 >> 172.21.15.28:0/1580560705 conn(0x7f6488079400 msgr2=0x7f6488079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:12.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.517+0000 7f648ec00640 1 -- 172.21.15.28:0/1580560705 shutdown_connections 2024-04-23T14:45:12.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.517+0000 7f648ec00640 1 -- 172.21.15.28:0/1580560705 wait complete. 2024-04-23T14:45:12.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.518+0000 7f648ec00640 1 Processor -- start 2024-04-23T14:45:12.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.518+0000 7f648ec00640 1 -- start start 2024-04-23T14:45:12.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.519+0000 7f648ec00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6488102230 0x7f6488074300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:12.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.519+0000 7f648ec00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6488107300 0x7f6488074840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:12.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.519+0000 7f648ec00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6488074d80 0x7f64881a6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:12.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.519+0000 7f648ec00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f64881123b0 con 0x7f6488074d80 2024-04-23T14:45:12.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.519+0000 7f648ec00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6488112230 con 0x7f6488102230 2024-04-23T14:45:12.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.519+0000 7f648ec00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6488112530 con 0x7f6488107300 2024-04-23T14:45:12.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.519+0000 7f6487e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6488102230 0x7f6488074300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:12.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.519+0000 7f6487e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6488102230 0x7f6488074300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:54878/0 (socket says 172.21.15.28:54878) 2024-04-23T14:45:12.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.519+0000 7f6487e00640 1 -- 172.21.15.28:0/2660996868 learned_addr learned my addr 172.21.15.28:0/2660996868 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:12.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.519+0000 7f6487400640 1 --2- 172.21.15.28:0/2660996868 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6488107300 0x7f6488074840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:12.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.519+0000 7f648cc00640 1 --2- 172.21.15.28:0/2660996868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6488074d80 0x7f64881a6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:12.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.520+0000 7f6487e00640 1 -- 172.21.15.28:0/2660996868 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6488107300 msgr2=0x7f6488074840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:12.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.520+0000 7f6487e00640 1 --2- 172.21.15.28:0/2660996868 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6488107300 0x7f6488074840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.520+0000 7f6487e00640 1 -- 172.21.15.28:0/2660996868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6488074d80 msgr2=0x7f64881a6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:12.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.520+0000 7f6487e00640 1 --2- 172.21.15.28:0/2660996868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6488074d80 0x7f64881a6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.520+0000 7f6487e00640 1 -- 172.21.15.28:0/2660996868 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64881a7200 con 0x7f6488102230 2024-04-23T14:45:12.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.520+0000 7f6487400640 1 --2- 172.21.15.28:0/2660996868 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6488107300 0x7f6488074840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:45:12.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.520+0000 7f648cc00640 1 --2- 172.21.15.28:0/2660996868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6488074d80 0x7f64881a6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:45:12.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.520+0000 7f6487e00640 1 --2- 172.21.15.28:0/2660996868 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6488102230 0x7f6488074300 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6478035040 tx=0x7f6478033d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:12.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.520+0000 7f6484c00640 1 -- 172.21.15.28:0/2660996868 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647803f020 con 0x7f6488102230 2024-04-23T14:45:12.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.521+0000 7f6484c00640 1 -- 172.21.15.28:0/2660996868 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6478032770 con 0x7f6488102230 2024-04-23T14:45:12.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.521+0000 7f6484c00640 1 -- 172.21.15.28:0/2660996868 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6478037040 con 0x7f6488102230 2024-04-23T14:45:12.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.521+0000 7f648ec00640 1 -- 172.21.15.28:0/2660996868 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64881a7490 con 0x7f6488102230 2024-04-23T14:45:12.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.521+0000 7f648ec00640 1 -- 172.21.15.28:0/2660996868 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64881a7930 con 0x7f6488102230 2024-04-23T14:45:12.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.522+0000 7f648ec00640 1 -- 172.21.15.28:0/2660996868 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6488069270 con 0x7f6488102230 2024-04-23T14:45:12.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.524+0000 7f6484c00640 1 -- 172.21.15.28:0/2660996868 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138879+0+0 (secure 0 0 0) 0x7f6478042020 con 0x7f6488102230 2024-04-23T14:45:12.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.525+0000 7f6484c00640 1 --2- 172.21.15.28:0/2660996868 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f646c0a3ef0 0x7f646c0a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:12.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.526+0000 7f6484c00640 1 -- 172.21.15.28:0/2660996868 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f64780f0f20 con 0x7f6488102230 2024-04-23T14:45:12.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.526+0000 7f6487400640 1 --2- 172.21.15.28:0/2660996868 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f646c0a3ef0 0x7f646c0a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:12.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.527+0000 7f6487400640 1 --2- 172.21.15.28:0/2660996868 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f646c0a3ef0 0x7f646c0a63b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6488075d90 tx=0x7f6474005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:12.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.528+0000 7f6484c00640 1 -- 172.21.15.28:0/2660996868 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6478049020 con 0x7f6488102230 2024-04-23T14:45:12.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.906+0000 7f648ec00640 1 -- 172.21.15.28:0/2660996868 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f648810a530 con 0x7f6488102230 2024-04-23T14:45:12.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.907+0000 7f6484c00640 1 -- 172.21.15.28:0/2660996868 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7f64780bd4b0 con 0x7f6488102230 2024-04-23T14:45:12.907 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:45:12.907 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:45:12.908 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:45:12.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.910+0000 7f648ec00640 1 -- 172.21.15.28:0/2660996868 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f646c0a3ef0 msgr2=0x7f646c0a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:12.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.910+0000 7f648ec00640 1 --2- 172.21.15.28:0/2660996868 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f646c0a3ef0 0x7f646c0a63b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6488075d90 tx=0x7f6474005f90 comp rx=0 tx=0).stop 2024-04-23T14:45:12.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.910+0000 7f648ec00640 1 -- 172.21.15.28:0/2660996868 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6488102230 msgr2=0x7f6488074300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:12.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.910+0000 7f648ec00640 1 --2- 172.21.15.28:0/2660996868 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6488102230 0x7f6488074300 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6478035040 tx=0x7f6478033d90 comp rx=0 tx=0).stop 2024-04-23T14:45:12.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.910+0000 7f648ec00640 1 -- 172.21.15.28:0/2660996868 shutdown_connections 2024-04-23T14:45:12.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.910+0000 7f648ec00640 1 --2- 172.21.15.28:0/2660996868 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f646c0a3ef0 0x7f646c0a63b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.910+0000 7f648ec00640 1 --2- 172.21.15.28:0/2660996868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6488074d80 0x7f64881a6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.910+0000 7f648ec00640 1 --2- 172.21.15.28:0/2660996868 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6488107300 0x7f6488074840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.910+0000 7f648ec00640 1 --2- 172.21.15.28:0/2660996868 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6488102230 0x7f6488074300 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:12.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.910+0000 7f648ec00640 1 -- 172.21.15.28:0/2660996868 >> 172.21.15.28:0/2660996868 conn(0x7f6488079400 msgr2=0x7f6488100250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:12.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.910+0000 7f648ec00640 1 -- 172.21.15.28:0/2660996868 shutdown_connections 2024-04-23T14:45:12.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:12.910+0000 7f648ec00640 1 -- 172.21.15.28:0/2660996868 wait complete. 2024-04-23T14:45:12.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:12 smithi028 ceph-mon[27810]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-04-23T14:45:12.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:12 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:45:12.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:12 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:45:12.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:12 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:45:12.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:12 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:45:12.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:12 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:45:12.936 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-23T14:45:13.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:12 smithi105 ceph-mon[32555]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-04-23T14:45:13.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:12 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:45:13.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:12 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:45:13.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:12 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:45:13.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:12 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:45:13.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:12 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:45:13.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:12 smithi038 ceph-mon[32348]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-04-23T14:45:13.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:12 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:45:13.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:12 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:45:13.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:12 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:45:13.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:12 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:45:13.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:12 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:45:13.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.153+0000 7fc7c1600640 1 -- 172.21.15.28:0/4101347603 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7bc104320 msgr2=0x7fc7bc104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:13.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.153+0000 7fc7c1600640 1 --2- 172.21.15.28:0/4101347603 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7bc104320 0x7fc7bc104700 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac009900 tx=0x7fc7ac02a3f0 comp rx=0 tx=0).stop 2024-04-23T14:45:13.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.153+0000 7fc7c1600640 1 -- 172.21.15.28:0/4101347603 shutdown_connections 2024-04-23T14:45:13.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.153+0000 7fc7c1600640 1 --2- 172.21.15.28:0/4101347603 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7bc1093f0 0x7fc7bc110fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:13.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.153+0000 7fc7c1600640 1 --2- 172.21.15.28:0/4101347603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7bc104cd0 0x7fc7bc108d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:13.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.153+0000 7fc7c1600640 1 --2- 172.21.15.28:0/4101347603 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7bc104320 0x7fc7bc104700 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:13.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.154+0000 7fc7c1600640 1 -- 172.21.15.28:0/4101347603 >> 172.21.15.28:0/4101347603 conn(0x7fc7bc0ffb40 msgr2=0x7fc7bc101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:13.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.154+0000 7fc7c1600640 1 -- 172.21.15.28:0/4101347603 shutdown_connections 2024-04-23T14:45:13.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.154+0000 7fc7c1600640 1 -- 172.21.15.28:0/4101347603 wait complete. 2024-04-23T14:45:13.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.155+0000 7fc7c1600640 1 Processor -- start 2024-04-23T14:45:13.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.155+0000 7fc7c1600640 1 -- start start 2024-04-23T14:45:13.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.155+0000 7fc7c1600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7bc104320 0x7fc7bc1a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:13.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.155+0000 7fc7c1600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7bc1093f0 0x7fc7bc1a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:13.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.155+0000 7fc7c1600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7bc19a820 0x7fc7bc19acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:13.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.155+0000 7fc7c1600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc7bc114480 con 0x7fc7bc19a820 2024-04-23T14:45:13.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.155+0000 7fc7c1600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc7bc114300 con 0x7fc7bc1093f0 2024-04-23T14:45:13.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.155+0000 7fc7c1600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc7bc114600 con 0x7fc7bc104320 2024-04-23T14:45:13.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.155+0000 7fc7bbe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7bc104320 0x7fc7bc1a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:13.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.155+0000 7fc7bbe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7bc104320 0x7fc7bc1a0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:39948/0 (socket says 172.21.15.28:39948) 2024-04-23T14:45:13.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.155+0000 7fc7bbe00640 1 -- 172.21.15.28:0/3574620524 learned_addr learned my addr 172.21.15.28:0/3574620524 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:13.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.156+0000 7fc7bb400640 1 --2- 172.21.15.28:0/3574620524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7bc1093f0 0x7fc7bc1a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:13.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.156+0000 7fc7c0c00640 1 --2- 172.21.15.28:0/3574620524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7bc19a820 0x7fc7bc19acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:13.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.156+0000 7fc7bbe00640 1 -- 172.21.15.28:0/3574620524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7bc1093f0 msgr2=0x7fc7bc1a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:13.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.156+0000 7fc7bbe00640 1 --2- 172.21.15.28:0/3574620524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7bc1093f0 0x7fc7bc1a0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:13.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.156+0000 7fc7bbe00640 1 -- 172.21.15.28:0/3574620524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7bc19a820 msgr2=0x7fc7bc19acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:13.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.156+0000 7fc7bbe00640 1 --2- 172.21.15.28:0/3574620524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7bc19a820 0x7fc7bc19acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:13.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.156+0000 7fc7bbe00640 1 -- 172.21.15.28:0/3574620524 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7bc19b3b0 con 0x7fc7bc104320 2024-04-23T14:45:13.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.156+0000 7fc7bb400640 1 --2- 172.21.15.28:0/3574620524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7bc1093f0 0x7fc7bc1a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:45:13.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.156+0000 7fc7bbe00640 1 --2- 172.21.15.28:0/3574620524 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7bc104320 0x7fc7bc1a0660 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac006010 tx=0x7fc7ac004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:13.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.157+0000 7fc7b8c00640 1 -- 172.21.15.28:0/3574620524 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac03f030 con 0x7fc7bc104320 2024-04-23T14:45:13.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.157+0000 7fc7b8c00640 1 -- 172.21.15.28:0/3574620524 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7ac004440 con 0x7fc7bc104320 2024-04-23T14:45:13.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.157+0000 7fc7b8c00640 1 -- 172.21.15.28:0/3574620524 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac035070 con 0x7fc7bc104320 2024-04-23T14:45:13.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.157+0000 7fc7c1600640 1 -- 172.21.15.28:0/3574620524 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7bc19b640 con 0x7fc7bc104320 2024-04-23T14:45:13.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.157+0000 7fc7c1600640 1 -- 172.21.15.28:0/3574620524 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7bc1a7750 con 0x7fc7bc104320 2024-04-23T14:45:13.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.161+0000 7fc7c5a4c740 1 Processor -- start 2024-04-23T14:45:13.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.162+0000 7fc7c5a4c740 1 -- start start 2024-04-23T14:45:13.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.162+0000 7fc7b8c00640 1 -- 172.21.15.28:0/3574620524 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138879+0+0 (secure 0 0 0) 0x7fc7ac042030 con 0x7fc7bc104320 2024-04-23T14:45:13.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.162+0000 7fc7c5a4c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c842c7350 0x556c842c7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:13.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.162+0000 7fc7c5a4c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c842c7c70 0x556c842d1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:13.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.162+0000 7fc7c5a4c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556c842d2340 0x556c842d4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:13.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.162+0000 7fc7c5a4c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556c842241d0 con 0x556c842c7c70 2024-04-23T14:45:13.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.162+0000 7fc7c5a4c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x556c841ca450 con 0x556c842d2340 2024-04-23T14:45:13.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.162+0000 7fc7c5a4c740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556c84220190 con 0x556c842c7350 2024-04-23T14:45:13.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.163+0000 7fc7bb400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556c842d2340 0x556c842d4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:13.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.163+0000 7fc7bbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c842c7c70 0x556c842d1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:13.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.163+0000 7fc7bb400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556c842d2340 0x556c842d4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:54910/0 (socket says 172.21.15.28:54910) 2024-04-23T14:45:13.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.163+0000 7fc7bb400640 1 -- 172.21.15.28:0/3273333465 learned_addr learned my addr 172.21.15.28:0/3273333465 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:13.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.163+0000 7fc7c0c00640 1 --2- 172.21.15.28:0/3273333465 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c842c7350 0x556c842c7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:13.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.163+0000 7fc7bb400640 1 -- 172.21.15.28:0/3273333465 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c842c7350 msgr2=0x556c842c7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:13.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.163+0000 7fc7bb400640 1 --2- 172.21.15.28:0/3273333465 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c842c7350 0x556c842c7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:13.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.163+0000 7fc7bb400640 1 -- 172.21.15.28:0/3273333465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c842c7c70 msgr2=0x556c842d1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:13.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.163+0000 7fc7bb400640 1 --2- 172.21.15.28:0/3273333465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c842c7c70 0x556c842d1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:13.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.163+0000 7fc7bb400640 1 -- 172.21.15.28:0/3273333465 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c84287310 con 0x556c842d2340 2024-04-23T14:45:13.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.163+0000 7fc7bbe00640 1 --2- 172.21.15.28:0/3273333465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c842c7c70 0x556c842d1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:45:13.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.164+0000 7fc7bb400640 1 --2- 172.21.15.28:0/3273333465 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556c842d2340 0x556c842d4730 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc7a8009c70 tx=0x7fc7a80117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98cde6ad8891deef server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:13.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.164+0000 7fc7b8c00640 1 --2- 172.21.15.28:0/3574620524 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc7a00a3ec0 0x7fc7a00a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:13.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.164+0000 7fc7b8c00640 1 -- 172.21.15.28:0/3574620524 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fc7ac0f0de0 con 0x7fc7bc104320 2024-04-23T14:45:13.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.164+0000 7fc7b1200640 1 -- 172.21.15.28:0/3273333465 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a8012450 con 0x556c842d2340 2024-04-23T14:45:13.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.164+0000 7fc7b1200640 1 -- 172.21.15.28:0/3273333465 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7a8012ad0 con 0x556c842d2340 2024-04-23T14:45:13.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.164+0000 7fc7c0c00640 1 --2- 172.21.15.28:0/3574620524 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc7a00a3ec0 0x7fc7a00a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:13.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.164+0000 7fc7b1200640 1 -- 172.21.15.28:0/3273333465 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a800fc30 con 0x556c842d2340 2024-04-23T14:45:13.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.165+0000 7fc7c5a4c740 1 -- 172.21.15.28:0/3273333465 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556c842d2340 msgr2=0x556c842d4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:13.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.165+0000 7fc7c5a4c740 1 --2- 172.21.15.28:0/3273333465 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556c842d2340 0x556c842d4730 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc7a8009c70 tx=0x7fc7a80117b0 comp rx=0 tx=0).stop 2024-04-23T14:45:13.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.165+0000 7fc7c5a4c740 1 -- 172.21.15.28:0/3273333465 shutdown_connections 2024-04-23T14:45:13.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.165+0000 7fc7c5a4c740 1 --2- 172.21.15.28:0/3273333465 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556c842d2340 0x556c842d4730 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:13.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.165+0000 7fc7c5a4c740 1 --2- 172.21.15.28:0/3273333465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c842c7c70 0x556c842d1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:13.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.165+0000 7fc7c5a4c740 1 --2- 172.21.15.28:0/3273333465 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c842c7350 0x556c842c7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:13.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.165+0000 7fc7c5a4c740 1 -- 172.21.15.28:0/3273333465 >> 172.21.15.28:0/3273333465 conn(0x556c842c3670 msgr2=0x556c842c3a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:13.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.165+0000 7fc7c5a4c740 1 -- 172.21.15.28:0/3273333465 shutdown_connections 2024-04-23T14:45:13.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.165+0000 7fc7c0c00640 1 --2- 172.21.15.28:0/3574620524 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc7a00a3ec0 0x7fc7a00a6380 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4005fd0 tx=0x7fc7b4009500 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:13.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.165+0000 7fc7c5a4c740 1 -- 172.21.15.28:0/3273333465 wait complete. 2024-04-23T14:45:13.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.166+0000 7fc7c5a4c740 1 -- start start 2024-04-23T14:45:13.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.166+0000 7fc7c5a4c740 1 Processor -- start 2024-04-23T14:45:13.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.167+0000 7fc7c5a4c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c842c7350 0x556c842d1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:13.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.167+0000 7fc7c5a4c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556c842c7c70 0x556c842cf030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:13.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.167+0000 7fc7c5a4c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c842d2340 0x556c842cf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:13.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.167+0000 7fc7c5a4c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556c842d5220 con 0x556c842d2340 2024-04-23T14:45:13.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.167+0000 7fc7c5a4c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x556c841dd810 con 0x556c842c7c70 2024-04-23T14:45:13.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.167+0000 7fc7c5a4c740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556c842d53a0 con 0x556c842c7350 2024-04-23T14:45:13.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.167+0000 7fc7c5a4c740 10 client.0 _ll_register_callbacks cb 0x556c841ab140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:45:13.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.167+0000 7fc7bbe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c842c7350 0x556c842d1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:13.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.167+0000 7fc7bbe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c842c7350 0x556c842d1d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:39960/0 (socket says 172.21.15.28:39960) 2024-04-23T14:45:13.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.167+0000 7fc7bbe00640 1 -- 172.21.15.28:0/3904366187 learned_addr learned my addr 172.21.15.28:0/3904366187 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:13.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.167+0000 7fc7c0c00640 1 --2- 172.21.15.28:0/3904366187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c842d2340 0x556c842cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:13.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.167+0000 7fc7bb400640 1 --2- 172.21.15.28:0/3904366187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556c842c7c70 0x556c842cf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:13.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.168+0000 7fc7c0c00640 1 -- 172.21.15.28:0/3904366187 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c842c7350 msgr2=0x556c842d1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:13.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.168+0000 7fc7c0c00640 1 --2- 172.21.15.28:0/3904366187 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c842c7350 0x556c842d1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:13.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.168+0000 7fc7c0c00640 1 -- 172.21.15.28:0/3904366187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556c842c7c70 msgr2=0x556c842cf030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:13.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.168+0000 7fc7c0c00640 1 --2- 172.21.15.28:0/3904366187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556c842c7c70 0x556c842cf030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:13.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.168+0000 7fc7c0c00640 1 -- 172.21.15.28:0/3904366187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c841ee3f0 con 0x556c842d2340 2024-04-23T14:45:13.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.168+0000 7fc7c0c00640 1 --2- 172.21.15.28:0/3904366187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c842d2340 0x556c842cf640 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4012750 tx=0x7fc7b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=447eea143e7932a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:13.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.168+0000 7fc7a5600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-23T14:45:13.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.168+0000 7fc7a5600640 1 -- 172.21.15.28:0/3904366187 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7b4004d20 con 0x556c842d2340 2024-04-23T14:45:13.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.168+0000 7fc7c5a4c740 1 -- 172.21.15.28:0/3904366187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7a8012ad0 con 0x556c842d2340 2024-04-23T14:45:13.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.168+0000 7fc7a5600640 1 -- 172.21.15.28:0/3904366187 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fc7b40108e0 con 0x556c842d2340 2024-04-23T14:45:13.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.169+0000 7fc7a5600640 1 -- 172.21.15.28:0/3904366187 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fc7b4010ae0 con 0x556c842d2340 2024-04-23T14:45:13.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.169+0000 7fc7c5a4c740 10 client.15348 fetch_fsmap learned FSMap version 20 2024-04-23T14:45:13.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.169+0000 7fc7c5a4c740 1 -- 172.21.15.28:0/3904366187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x556c83f363f0 con 0x556c842d2340 2024-04-23T14:45:13.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.169+0000 7fc7a5600640 1 -- 172.21.15.28:0/3904366187 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7a8012ad0 con 0x556c842d2340 2024-04-23T14:45:13.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.169+0000 7fc7a5600640 1 -- 172.21.15.28:0/3904366187 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fc7b4024070 con 0x556c842d2340 2024-04-23T14:45:13.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.170+0000 7fc7c5a4c740 10 client.15348 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:45:13.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.170+0000 7fc7c5a4c740 10 client.15348 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:45:13.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.170+0000 7fc7c5a4c740 20 client.15348 populate_metadata read hostname 'smithi028' 2024-04-23T14:45:13.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.170+0000 7fc7c5a4c740 1 --2- 172.21.15.28:0/3904366187 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x556c842e25b0 0x556c842e4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:13.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.170+0000 7fc7c5a4c740 4 client.15348 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:45:13.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.170+0000 7fc7c5a4c740 1 -- 172.21.15.28:0/3904366187 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556c84231bf0 con 0x556c842e25b0 2024-04-23T14:45:13.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.170+0000 7fc7bbe00640 1 --2- 172.21.15.28:0/3904366187 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x556c842e25b0 0x556c842e4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:13.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.170+0000 7fc7bbe00640 1 --2- 172.21.15.28:0/3904366187 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x556c842e25b0 0x556c842e4a00 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=409a5a7c28ece4b5 server_cookie=14b9c5d28dee3d42 in_seq=0 out_seq=0 2024-04-23T14:45:13.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.170+0000 7fc7a5600640 10 client.15348 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:45:13.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.171+0000 7fc7a5600640 1 -- 172.21.15.28:0/3904366187 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556c84231bf0 con 0x556c842e25b0 2024-04-23T14:45:13.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.171+0000 7fc7a5600640 10 client.15348 handle_command_reply: tid=0 2024-04-23T14:45:13.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.171+0000 7fc7c5a4c740 1 client.15348 shutdown 2024-04-23T14:45:13.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.172+0000 7fc7c5a4c740 1 -- 172.21.15.28:0/3904366187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c842d2340 msgr2=0x556c842cf640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:13.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.172+0000 7fc7c5a4c740 1 --2- 172.21.15.28:0/3904366187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c842d2340 0x556c842cf640 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4012750 tx=0x7fc7b4004a40 comp rx=0 tx=0).stop 2024-04-23T14:45:13.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.172+0000 7fc7c5a4c740 1 -- 172.21.15.28:0/3904366187 shutdown_connections 2024-04-23T14:45:13.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.172+0000 7fc7c5a4c740 1 --2- 172.21.15.28:0/3904366187 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x556c842e25b0 0x556c842e4a00 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:13.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.172+0000 7fc7c5a4c740 1 --2- 172.21.15.28:0/3904366187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556c842c7c70 0x556c842cf030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:13.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.172+0000 7fc7c5a4c740 1 --2- 172.21.15.28:0/3904366187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c842d2340 0x556c842cf640 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:13.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.172+0000 7fc7a5600640 0 client.15348 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:45:13.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.172+0000 7fc7c5a4c740 1 --2- 172.21.15.28:0/3904366187 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c842c7350 0x556c842d1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:13.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.172+0000 7fc7c5a4c740 1 -- 172.21.15.28:0/3904366187 >> 172.21.15.28:0/3904366187 conn(0x556c842c3670 msgr2=0x556c842c9470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:13.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.172+0000 7fc7c5a4c740 1 -- 172.21.15.28:0/3904366187 shutdown_connections 2024-04-23T14:45:13.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.172+0000 7fc7c5a4c740 1 -- 172.21.15.28:0/3904366187 wait complete. 2024-04-23T14:45:13.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.172+0000 7fc7c5a4c740 20 client.15348 trim_cache size 0 max 16384 2024-04-23T14:45:13.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.200+0000 7fc7c5a4c740 1 Processor -- start 2024-04-23T14:45:13.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.200+0000 7fc7c5a4c740 1 -- start start 2024-04-23T14:45:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.200+0000 7fc7c5a4c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c842c7350 0x556c842de0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.200+0000 7fc7c5a4c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c842c7c70 0x556c842d1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.200+0000 7fc7c5a4c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556c842d2340 0x556c842ea2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.201+0000 7fc7c5a4c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc7bc1a9890 con 0x556c842c7350 2024-04-23T14:45:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.201+0000 7fc7c5a4c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc7bc1ac110 con 0x556c842d2340 2024-04-23T14:45:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.201+0000 7fc7c5a4c740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc7bc1a9f00 con 0x556c842c7c70 2024-04-23T14:45:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.201+0000 7fc7bbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c842c7350 0x556c842de0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.201+0000 7fc7bbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c842c7350 0x556c842de0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44606/0 (socket says 172.21.15.28:44606) 2024-04-23T14:45:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.201+0000 7fc7bbe00640 1 -- 172.21.15.28:0/920652842 learned_addr learned my addr 172.21.15.28:0/920652842 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.201+0000 7fc7bb400640 1 --2- 172.21.15.28:0/920652842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c842c7c70 0x556c842d1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.201+0000 7fc7c0c00640 1 --2- 172.21.15.28:0/920652842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556c842d2340 0x556c842ea2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.201+0000 7fc7bbe00640 1 -- 172.21.15.28:0/920652842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c842c7c70 msgr2=0x556c842d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.201+0000 7fc7bbe00640 1 --2- 172.21.15.28:0/920652842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c842c7c70 0x556c842d1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.201+0000 7fc7bbe00640 1 -- 172.21.15.28:0/920652842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556c842d2340 msgr2=0x556c842ea2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.201+0000 7fc7bbe00640 1 --2- 172.21.15.28:0/920652842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556c842d2340 0x556c842ea2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.201+0000 7fc7bbe00640 1 -- 172.21.15.28:0/920652842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c840007d0 con 0x556c842c7350 2024-04-23T14:45:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.202+0000 7fc7bbe00640 1 --2- 172.21.15.28:0/920652842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c842c7350 0x556c842de0e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac000c00 tx=0x7fc7ac0f48f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ebf37e0bcdfb1d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.202+0000 7fc7a7e00640 1 -- 172.21.15.28:0/920652842 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac049040 con 0x556c842c7350 2024-04-23T14:45:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.202+0000 7fc7a7e00640 1 -- 172.21.15.28:0/920652842 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7ac0bd310 con 0x556c842c7350 2024-04-23T14:45:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.202+0000 7fc7a7e00640 1 -- 172.21.15.28:0/920652842 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac003cd0 con 0x556c842c7350 2024-04-23T14:45:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.203+0000 7fc7c5a4c740 1 -- 172.21.15.28:0/920652842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c842c7350 msgr2=0x556c842de0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.203+0000 7fc7c5a4c740 1 --2- 172.21.15.28:0/920652842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c842c7350 0x556c842de0e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac000c00 tx=0x7fc7ac0f48f0 comp rx=0 tx=0).stop 2024-04-23T14:45:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.203+0000 7fc7c5a4c740 1 -- 172.21.15.28:0/920652842 shutdown_connections 2024-04-23T14:45:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.203+0000 7fc7c5a4c740 1 --2- 172.21.15.28:0/920652842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556c842d2340 0x556c842ea2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.203+0000 7fc7c5a4c740 1 --2- 172.21.15.28:0/920652842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c842c7c70 0x556c842d1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.203+0000 7fc7c5a4c740 1 --2- 172.21.15.28:0/920652842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c842c7350 0x556c842de0e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.203+0000 7fc7c5a4c740 1 -- 172.21.15.28:0/920652842 >> 172.21.15.28:0/920652842 conn(0x556c842c3670 msgr2=0x556c842ceff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.203+0000 7fc7c5a4c740 1 -- 172.21.15.28:0/920652842 shutdown_connections 2024-04-23T14:45:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.203+0000 7fc7c5a4c740 1 -- 172.21.15.28:0/920652842 wait complete. 2024-04-23T14:45:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.204+0000 7fc7c5a4c740 1 -- start start 2024-04-23T14:45:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.204+0000 7fc7c5a4c740 1 Processor -- start 2024-04-23T14:45:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.205+0000 7fc7c5a4c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556c842c7350 0x556c842d3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.205+0000 7fc7c5a4c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c842c7c70 0x556c842d3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.205+0000 7fc7c5a4c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c842d2340 0x556c842d8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.205+0000 7fc7c5a4c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc7bc1aa5d0 con 0x556c842d2340 2024-04-23T14:45:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.205+0000 7fc7c5a4c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc7bc1aba40 con 0x556c842c7350 2024-04-23T14:45:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.205+0000 7fc7c5a4c740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc7bc1aaca0 con 0x556c842c7c70 2024-04-23T14:45:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.205+0000 7fc7c5a4c740 10 client.0 _ll_register_callbacks cb 0x7fc7b4003f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:45:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.205+0000 7fc7bbe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556c842c7350 0x556c842d3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.205+0000 7fc7bbe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556c842c7350 0x556c842d3450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:54924/0 (socket says 172.21.15.28:54924) 2024-04-23T14:45:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.205+0000 7fc7c0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c842d2340 0x556c842d8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.205+0000 7fc7bbe00640 1 -- 172.21.15.28:0/2257861942 learned_addr learned my addr 172.21.15.28:0/2257861942 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.205+0000 7fc7bb400640 1 --2- 172.21.15.28:0/2257861942 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c842c7c70 0x556c842d3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:13.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.206+0000 7fc7bbe00640 1 -- 172.21.15.28:0/2257861942 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c842c7c70 msgr2=0x556c842d3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:13.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.206+0000 7fc7bbe00640 1 --2- 172.21.15.28:0/2257861942 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c842c7c70 0x556c842d3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:13.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.206+0000 7fc7bbe00640 1 -- 172.21.15.28:0/2257861942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c842d2340 msgr2=0x556c842d8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:13.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.206+0000 7fc7bbe00640 1 --2- 172.21.15.28:0/2257861942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c842d2340 0x556c842d8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:13.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.206+0000 7fc7bbe00640 1 -- 172.21.15.28:0/2257861942 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c842d90c0 con 0x556c842c7350 2024-04-23T14:45:13.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.206+0000 7fc7bbe00640 1 --2- 172.21.15.28:0/2257861942 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556c842c7350 0x556c842d3450 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac0bcba0 tx=0x7fc7ac0d6230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb11713f0001cce1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:13.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.206+0000 7fc7a5600640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:45:13.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.206+0000 7fc7a5600640 1 -- 172.21.15.28:0/2257861942 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac049030 con 0x556c842c7350 2024-04-23T14:45:13.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.206+0000 7fc7a5600640 1 -- 172.21.15.28:0/2257861942 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fc7ac0b9280 con 0x556c842c7350 2024-04-23T14:45:13.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.206+0000 7fc7c5a4c740 1 -- 172.21.15.28:0/2257861942 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7ac0bd310 con 0x556c842c7350 2024-04-23T14:45:13.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.207+0000 7fc7a5600640 1 -- 172.21.15.28:0/2257861942 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fc7ac0bcc10 con 0x556c842c7350 2024-04-23T14:45:13.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.207+0000 7fc7c5a4c740 10 client.25490 fetch_fsmap learned FSMap version 20 2024-04-23T14:45:13.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.207+0000 7fc7c5a4c740 1 -- 172.21.15.28:0/2257861942 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x556c84349200 con 0x556c842c7350 2024-04-23T14:45:13.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.207+0000 7fc7a5600640 1 -- 172.21.15.28:0/2257861942 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7ac0bd310 con 0x556c842c7350 2024-04-23T14:45:13.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.207+0000 7fc7a5600640 1 -- 172.21.15.28:0/2257861942 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fc7ac031bd0 con 0x556c842c7350 2024-04-23T14:45:13.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.207+0000 7fc7c5a4c740 10 client.25490 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:45:13.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.207+0000 7fc7c5a4c740 10 client.25490 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:45:13.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.207+0000 7fc7c5a4c740 20 client.25490 populate_metadata read hostname 'smithi028' 2024-04-23T14:45:13.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.207+0000 7fc7c5a4c740 1 --2- 172.21.15.28:0/2257861942 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x556c843498c0 0x556c8434bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:13.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.207+0000 7fc7c5a4c740 4 client.25490 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-23T14:45:13.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.207+0000 7fc7c5a4c740 1 -- 172.21.15.28:0/2257861942 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556c8422ba60 con 0x556c843498c0 2024-04-23T14:45:13.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.207+0000 7fc7bb400640 1 --2- 172.21.15.28:0/2257861942 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x556c843498c0 0x556c8434bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:13.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.208+0000 7fc7bb400640 1 --2- 172.21.15.28:0/2257861942 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x556c843498c0 0x556c8434bcb0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cc060a421123753 server_cookie=fbe085c7335f96ab in_seq=0 out_seq=0 2024-04-23T14:45:13.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.208+0000 7fc7a5600640 10 client.25490 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:45:13.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.208+0000 7fc7a5600640 1 -- 172.21.15.28:0/2257861942 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556c8422ba60 con 0x556c843498c0 2024-04-23T14:45:13.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.208+0000 7fc7a5600640 10 client.25490 handle_command_reply: tid=0 2024-04-23T14:45:13.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.208+0000 7fc7c5a4c740 1 client.25490 shutdown 2024-04-23T14:45:13.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.209+0000 7fc7c5a4c740 1 -- 172.21.15.28:0/2257861942 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556c842c7350 msgr2=0x556c842d3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:13.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.209+0000 7fc7c5a4c740 1 --2- 172.21.15.28:0/2257861942 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556c842c7350 0x556c842d3450 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac0bcba0 tx=0x7fc7ac0d6230 comp rx=0 tx=0).stop 2024-04-23T14:45:13.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.209+0000 7fc7c5a4c740 1 -- 172.21.15.28:0/2257861942 shutdown_connections 2024-04-23T14:45:13.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.209+0000 7fc7c5a4c740 1 --2- 172.21.15.28:0/2257861942 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x556c843498c0 0x556c8434bcb0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:13.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.209+0000 7fc7c5a4c740 1 --2- 172.21.15.28:0/2257861942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c842d2340 0x556c842d8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:13.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.209+0000 7fc7c5a4c740 1 --2- 172.21.15.28:0/2257861942 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556c842c7c70 0x556c842d3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:13.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.209+0000 7fc7c5a4c740 1 --2- 172.21.15.28:0/2257861942 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556c842c7350 0x556c842d3450 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:13.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.209+0000 7fc7a5600640 0 client.25490 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:45:13.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.209+0000 7fc7c5a4c740 1 -- 172.21.15.28:0/2257861942 >> 172.21.15.28:0/2257861942 conn(0x556c842c3670 msgr2=0x556c842b96c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:13.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.209+0000 7fc7c5a4c740 1 -- 172.21.15.28:0/2257861942 shutdown_connections 2024-04-23T14:45:13.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.209+0000 7fc7c5a4c740 1 -- 172.21.15.28:0/2257861942 wait complete. 2024-04-23T14:45:13.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.209+0000 7fc7c5a4c740 20 client.25490 trim_cache size 0 max 16384 2024-04-23T14:45:13.212 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-23T14:45:13.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.210+0000 7fc7c1600640 1 -- 172.21.15.28:0/3574620524 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc7a00a3ec0 msgr2=0x7fc7a00a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:13.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.210+0000 7fc7c1600640 1 --2- 172.21.15.28:0/3574620524 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc7a00a3ec0 0x7fc7a00a6380 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4005fd0 tx=0x7fc7b4009500 comp rx=0 tx=0).stop 2024-04-23T14:45:13.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.210+0000 7fc7c1600640 1 -- 172.21.15.28:0/3574620524 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7bc104320 msgr2=0x7fc7bc1a0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:13.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.210+0000 7fc7c1600640 1 --2- 172.21.15.28:0/3574620524 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7bc104320 0x7fc7bc1a0660 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac006010 tx=0x7fc7ac004040 comp rx=0 tx=0).stop 2024-04-23T14:45:13.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.210+0000 7fc7c1600640 1 -- 172.21.15.28:0/3574620524 shutdown_connections 2024-04-23T14:45:13.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.210+0000 7fc7c1600640 1 --2- 172.21.15.28:0/3574620524 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc7a00a3ec0 0x7fc7a00a6380 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:13.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.210+0000 7fc7c1600640 1 --2- 172.21.15.28:0/3574620524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7bc1093f0 0x7fc7bc1a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:13.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.210+0000 7fc7c1600640 1 --2- 172.21.15.28:0/3574620524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7bc19a820 0x7fc7bc19acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:13.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.210+0000 7fc7c1600640 1 --2- 172.21.15.28:0/3574620524 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7bc104320 0x7fc7bc1a0660 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:13.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.210+0000 7fc7c1600640 1 -- 172.21.15.28:0/3574620524 >> 172.21.15.28:0/3574620524 conn(0x7fc7bc0ffb40 msgr2=0x7fc7bc100f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:13.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.210+0000 7fc7c1600640 1 -- 172.21.15.28:0/3574620524 shutdown_connections 2024-04-23T14:45:13.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:13.210+0000 7fc7c1600640 1 -- 172.21.15.28:0/3574620524 wait complete. 2024-04-23T14:45:14.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:13 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2660996868' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:45:14.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:13 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2660996868' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:45:14.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:13 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2660996868' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:45:14.227 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-23T14:45:14.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:45:14.228 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-23T14:45:14.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.429+0000 7fd769200640 1 -- 172.21.15.28:0/2595330597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd764104cd0 msgr2=0x7fd764108d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:14.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.429+0000 7fd769200640 1 --2- 172.21.15.28:0/2595330597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd764104cd0 0x7fd764108d10 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd750009900 tx=0x7fd75002a400 comp rx=0 tx=0).stop 2024-04-23T14:45:14.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.429+0000 7fd769200640 1 -- 172.21.15.28:0/2595330597 shutdown_connections 2024-04-23T14:45:14.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.429+0000 7fd769200640 1 --2- 172.21.15.28:0/2595330597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7641093f0 0x7fd764110fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.429+0000 7fd769200640 1 --2- 172.21.15.28:0/2595330597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd764104cd0 0x7fd764108d10 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.429+0000 7fd769200640 1 --2- 172.21.15.28:0/2595330597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd764104320 0x7fd764104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.429+0000 7fd769200640 1 -- 172.21.15.28:0/2595330597 >> 172.21.15.28:0/2595330597 conn(0x7fd7640ffb40 msgr2=0x7fd764101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:14.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.429+0000 7fd769200640 1 -- 172.21.15.28:0/2595330597 shutdown_connections 2024-04-23T14:45:14.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.429+0000 7fd769200640 1 -- 172.21.15.28:0/2595330597 wait complete. 2024-04-23T14:45:14.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.430+0000 7fd769200640 1 Processor -- start 2024-04-23T14:45:14.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.431+0000 7fd769200640 1 -- start start 2024-04-23T14:45:14.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.431+0000 7fd769200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd764104320 0x7fd7641a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.431+0000 7fd769200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd7641093f0 0x7fd7641a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.431+0000 7fd769200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd76419a810 0x7fd76419acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.431+0000 7fd769200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd7641144d0 con 0x7fd764104320 2024-04-23T14:45:14.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.431+0000 7fd769200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd764114350 con 0x7fd76419a810 2024-04-23T14:45:14.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.431+0000 7fd769200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd764114650 con 0x7fd7641093f0 2024-04-23T14:45:14.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.432+0000 7fd763400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd764104320 0x7fd7641a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:14.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.432+0000 7fd763400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd764104320 0x7fd7641a0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44630/0 (socket says 172.21.15.28:44630) 2024-04-23T14:45:14.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.432+0000 7fd763400640 1 -- 172.21.15.28:0/1368278648 learned_addr learned my addr 172.21.15.28:0/1368278648 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:14.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.432+0000 7fd762a00640 1 --2- 172.21.15.28:0/1368278648 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd7641093f0 0x7fd7641a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:14.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.432+0000 7fd763e00640 1 --2- 172.21.15.28:0/1368278648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd76419a810 0x7fd76419acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:14.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.432+0000 7fd763400640 1 -- 172.21.15.28:0/1368278648 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd7641093f0 msgr2=0x7fd7641a0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:14.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.432+0000 7fd763400640 1 --2- 172.21.15.28:0/1368278648 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd7641093f0 0x7fd7641a0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.432+0000 7fd763400640 1 -- 172.21.15.28:0/1368278648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd76419a810 msgr2=0x7fd76419acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:14.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.432+0000 7fd763400640 1 --2- 172.21.15.28:0/1368278648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd76419a810 0x7fd76419acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.432+0000 7fd763400640 1 -- 172.21.15.28:0/1368278648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd76419b3a0 con 0x7fd764104320 2024-04-23T14:45:14.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.432+0000 7fd763e00640 1 --2- 172.21.15.28:0/1368278648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd76419a810 0x7fd76419acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:45:14.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.432+0000 7fd763400640 1 --2- 172.21.15.28:0/1368278648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd764104320 0x7fd7641a0650 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd75400e9f0 tx=0x7fd75400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:14.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.433+0000 7fd75be00640 1 -- 172.21.15.28:0/1368278648 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd75400cd80 con 0x7fd764104320 2024-04-23T14:45:14.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.433+0000 7fd769200640 1 -- 172.21.15.28:0/1368278648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd76419b690 con 0x7fd764104320 2024-04-23T14:45:14.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.433+0000 7fd75be00640 1 -- 172.21.15.28:0/1368278648 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd754010910 con 0x7fd764104320 2024-04-23T14:45:14.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.433+0000 7fd75be00640 1 -- 172.21.15.28:0/1368278648 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd754018990 con 0x7fd764104320 2024-04-23T14:45:14.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.433+0000 7fd769200640 1 -- 172.21.15.28:0/1368278648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7641a7840 con 0x7fd764104320 2024-04-23T14:45:14.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.437+0000 7fd75be00640 1 -- 172.21.15.28:0/1368278648 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138879+0+0 (secure 0 0 0) 0x7fd754018b30 con 0x7fd764104320 2024-04-23T14:45:14.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.437+0000 7fd76d660740 1 Processor -- start 2024-04-23T14:45:14.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.437+0000 7fd76d660740 1 -- start start 2024-04-23T14:45:14.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.437+0000 7fd76d660740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585de3ca270 0x5585de3ca650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.437+0000 7fd76d660740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585de3cab90 0x5585de3d4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.437+0000 7fd76d660740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585de3d5260 0x5585de3d7650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.437+0000 7fd76d660740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5585de32fe70 con 0x5585de3d5260 2024-04-23T14:45:14.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.437+0000 7fd76d660740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5585de2d6380 con 0x5585de3ca270 2024-04-23T14:45:14.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.437+0000 7fd76d660740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5585de32be30 con 0x5585de3cab90 2024-04-23T14:45:14.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.438+0000 7fd763e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585de3ca270 0x5585de3ca650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:14.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.438+0000 7fd763e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585de3ca270 0x5585de3ca650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:54962/0 (socket says 172.21.15.28:54962) 2024-04-23T14:45:14.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.438+0000 7fd763e00640 1 -- 172.21.15.28:0/516717608 learned_addr learned my addr 172.21.15.28:0/516717608 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:14.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.438+0000 7fd763e00640 1 -- 172.21.15.28:0/516717608 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585de3cab90 msgr2=0x5585de3d4d20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T14:45:14.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.438+0000 7fd763e00640 1 --2- 172.21.15.28:0/516717608 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585de3cab90 0x5585de3d4d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.438+0000 7fd763e00640 1 -- 172.21.15.28:0/516717608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585de3d5260 msgr2=0x5585de3d7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.438+0000 7fd762a00640 1 --2- 172.21.15.28:0/516717608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585de3d5260 0x5585de3d7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.438+0000 7fd75be00640 1 --2- 172.21.15.28:0/1368278648 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd7480a3f40 0x7fd7480a6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.438+0000 7fd763e00640 1 --2- 172.21.15.28:0/516717608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585de3d5260 0x5585de3d7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.438+0000 7fd763e00640 1 -- 172.21.15.28:0/516717608 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585de10a7e0 con 0x5585de3ca270 2024-04-23T14:45:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.438+0000 7fd762a00640 1 --2- 172.21.15.28:0/516717608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585de3d5260 0x5585de3d7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:45:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.438+0000 7fd75be00640 1 -- 172.21.15.28:0/1368278648 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fd7540d0d60 con 0x7fd764104320 2024-04-23T14:45:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.438+0000 7fd763e00640 1 --2- 172.21.15.28:0/516717608 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585de3ca270 0x5585de3ca650 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd75c009f10 tx=0x7fd75c00f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b55d6bacbb1d6e3c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:14.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.439+0000 7fd763400640 1 --2- 172.21.15.28:0/1368278648 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd7480a3f40 0x7fd7480a6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:14.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.439+0000 7fd758c00640 1 -- 172.21.15.28:0/516717608 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd75c003b70 con 0x5585de3ca270 2024-04-23T14:45:14.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.439+0000 7fd758c00640 1 -- 172.21.15.28:0/516717608 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd75c003d10 con 0x5585de3ca270 2024-04-23T14:45:14.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.439+0000 7fd758c00640 1 -- 172.21.15.28:0/516717608 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd75c011a70 con 0x5585de3ca270 2024-04-23T14:45:14.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.439+0000 7fd76d660740 1 -- 172.21.15.28:0/516717608 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585de3ca270 msgr2=0x5585de3ca650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:14.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.439+0000 7fd76d660740 1 --2- 172.21.15.28:0/516717608 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585de3ca270 0x5585de3ca650 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd75c009f10 tx=0x7fd75c00f730 comp rx=0 tx=0).stop 2024-04-23T14:45:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.439+0000 7fd763400640 1 --2- 172.21.15.28:0/1368278648 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd7480a3f40 0x7fd7480a6400 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd7540ce340 tx=0x7fd754009310 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.440+0000 7fd76d660740 1 -- 172.21.15.28:0/516717608 shutdown_connections 2024-04-23T14:45:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.440+0000 7fd76d660740 1 --2- 172.21.15.28:0/516717608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585de3d5260 0x5585de3d7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.440+0000 7fd76d660740 1 --2- 172.21.15.28:0/516717608 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585de3cab90 0x5585de3d4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.440+0000 7fd76d660740 1 --2- 172.21.15.28:0/516717608 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585de3ca270 0x5585de3ca650 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.440+0000 7fd76d660740 1 -- 172.21.15.28:0/516717608 >> 172.21.15.28:0/516717608 conn(0x5585de3c6590 msgr2=0x5585de3c6970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.440+0000 7fd76d660740 1 -- 172.21.15.28:0/516717608 shutdown_connections 2024-04-23T14:45:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.440+0000 7fd76d660740 1 -- 172.21.15.28:0/516717608 wait complete. 2024-04-23T14:45:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.440+0000 7fd76d660740 1 -- start start 2024-04-23T14:45:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.441+0000 7fd76d660740 1 Processor -- start 2024-04-23T14:45:14.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.441+0000 7fd76d660740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585de3ca270 0x5585de3d2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.441+0000 7fd76d660740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585de3cab90 0x5585de3d2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.441+0000 7fd76d660740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585de3d5260 0x5585de3d2c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.441+0000 7fd76d660740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5585de3d7e60 con 0x5585de3d5260 2024-04-23T14:45:14.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.441+0000 7fd76d660740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5585de2e9660 con 0x5585de3cab90 2024-04-23T14:45:14.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.441+0000 7fd76d660740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5585de3d7fe0 con 0x5585de3ca270 2024-04-23T14:45:14.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.441+0000 7fd76d660740 10 client.0 _ll_register_callbacks cb 0x5585de2d7aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:45:14.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.442+0000 7fd763e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585de3d5260 0x5585de3d2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:14.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.442+0000 7fd762a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585de3cab90 0x5585de3d2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:14.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.442+0000 7fd763e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585de3d5260 0x5585de3d2c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44646/0 (socket says 172.21.15.28:44646) 2024-04-23T14:45:14.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.442+0000 7fd763e00640 1 -- 172.21.15.28:0/2502841164 learned_addr learned my addr 172.21.15.28:0/2502841164 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:14.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.442+0000 7fd763e00640 1 --2- 172.21.15.28:0/2502841164 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585de3ca270 0x5585de3d2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:14.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.442+0000 7fd763e00640 1 -- 172.21.15.28:0/2502841164 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585de3ca270 msgr2=0x5585de3d2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:14.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.442+0000 7fd763e00640 1 --2- 172.21.15.28:0/2502841164 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585de3ca270 0x5585de3d2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.442+0000 7fd763e00640 1 -- 172.21.15.28:0/2502841164 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585de3cab90 msgr2=0x5585de3d2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:14.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.442+0000 7fd763e00640 1 --2- 172.21.15.28:0/2502841164 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585de3cab90 0x5585de3d2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.442+0000 7fd763e00640 1 -- 172.21.15.28:0/2502841164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585de16af70 con 0x5585de3d5260 2024-04-23T14:45:14.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.442+0000 7fd762a00640 1 --2- 172.21.15.28:0/2502841164 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585de3cab90 0x5585de3d2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:45:14.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.442+0000 7fd763e00640 1 --2- 172.21.15.28:0/2502841164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585de3d5260 0x5585de3d2c80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd75c009370 tx=0x7fd75c0092a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86f05003fa6bc4d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:14.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.443+0000 7fd74d600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-23T14:45:14.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.443+0000 7fd74d600640 1 -- 172.21.15.28:0/2502841164 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd75c024040 con 0x5585de3d5260 2024-04-23T14:45:14.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.443+0000 7fd74d600640 1 -- 172.21.15.28:0/2502841164 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fd75c012c30 con 0x5585de3d5260 2024-04-23T14:45:14.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.443+0000 7fd76d660740 1 -- 172.21.15.28:0/2502841164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd75c003d10 con 0x5585de3d5260 2024-04-23T14:45:14.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.443+0000 7fd74d600640 1 -- 172.21.15.28:0/2502841164 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fd75c012e30 con 0x5585de3d5260 2024-04-23T14:45:14.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.443+0000 7fd76d660740 10 client.15363 fetch_fsmap learned FSMap version 20 2024-04-23T14:45:14.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.443+0000 7fd76d660740 1 -- 172.21.15.28:0/2502841164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5585de002170 con 0x5585de3d5260 2024-04-23T14:45:14.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.444+0000 7fd74d600640 1 -- 172.21.15.28:0/2502841164 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd75c003d10 con 0x5585de3d5260 2024-04-23T14:45:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.444+0000 7fd74d600640 1 -- 172.21.15.28:0/2502841164 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fd75c035750 con 0x5585de3d5260 2024-04-23T14:45:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.444+0000 7fd76d660740 10 client.15363 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:45:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.444+0000 7fd76d660740 10 client.15363 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:45:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.444+0000 7fd76d660740 20 client.15363 populate_metadata read hostname 'smithi028' 2024-04-23T14:45:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.444+0000 7fd76d660740 1 --2- 172.21.15.28:0/2502841164 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5585de3e5290 0x5585de3e76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.444+0000 7fd76d660740 4 client.15363 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:45:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.444+0000 7fd76d660740 1 -- 172.21.15.28:0/2502841164 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5585de33d890 con 0x5585de3e5290 2024-04-23T14:45:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.445+0000 7fd762a00640 1 --2- 172.21.15.28:0/2502841164 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5585de3e5290 0x5585de3e76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.445+0000 7fd762a00640 1 --2- 172.21.15.28:0/2502841164 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5585de3e5290 0x5585de3e76f0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4d5976df23ecca9 server_cookie=1f13fa3f34d53663 in_seq=0 out_seq=0 2024-04-23T14:45:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.445+0000 7fd74d600640 10 client.15363 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:45:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.446+0000 7fd74d600640 1 -- 172.21.15.28:0/2502841164 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5585de33d890 con 0x5585de3e5290 2024-04-23T14:45:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.446+0000 7fd74d600640 10 client.15363 handle_command_reply: tid=0 2024-04-23T14:45:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.446+0000 7fd76d660740 1 client.15363 shutdown 2024-04-23T14:45:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.447+0000 7fd76d660740 1 -- 172.21.15.28:0/2502841164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585de3d5260 msgr2=0x5585de3d2c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:14.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.447+0000 7fd76d660740 1 --2- 172.21.15.28:0/2502841164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585de3d5260 0x5585de3d2c80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd75c009370 tx=0x7fd75c0092a0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.447+0000 7fd76d660740 1 -- 172.21.15.28:0/2502841164 shutdown_connections 2024-04-23T14:45:14.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.447+0000 7fd76d660740 1 --2- 172.21.15.28:0/2502841164 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5585de3e5290 0x5585de3e76f0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.447+0000 7fd76d660740 1 --2- 172.21.15.28:0/2502841164 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585de3cab90 0x5585de3d2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.447+0000 7fd76d660740 1 --2- 172.21.15.28:0/2502841164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585de3d5260 0x5585de3d2c80 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.447+0000 7fd76d660740 1 --2- 172.21.15.28:0/2502841164 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585de3ca270 0x5585de3d2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.447+0000 7fd76d660740 1 -- 172.21.15.28:0/2502841164 >> 172.21.15.28:0/2502841164 conn(0x5585de3c6590 msgr2=0x5585de3cc290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:14.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.447+0000 7fd74d600640 0 client.15363 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:45:14.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.447+0000 7fd76d660740 1 -- 172.21.15.28:0/2502841164 shutdown_connections 2024-04-23T14:45:14.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.447+0000 7fd76d660740 1 -- 172.21.15.28:0/2502841164 wait complete. 2024-04-23T14:45:14.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.447+0000 7fd76d660740 20 client.15363 trim_cache size 0 max 16384 2024-04-23T14:45:14.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.467+0000 7fd76d660740 1 Processor -- start 2024-04-23T14:45:14.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.467+0000 7fd76d660740 1 -- start start 2024-04-23T14:45:14.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.467+0000 7fd76d660740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585de3ca270 0x5585de3e0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.467+0000 7fd76d660740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585de3cab90 0x5585de3e85e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.467+0000 7fd76d660740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585de3d5260 0x5585de3cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.467+0000 7fd76d660740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd74400c820 con 0x5585de3d5260 2024-04-23T14:45:14.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.467+0000 7fd76d660740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd74400f0a0 con 0x5585de3cab90 2024-04-23T14:45:14.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.467+0000 7fd76d660740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd74400ce90 con 0x5585de3ca270 2024-04-23T14:45:14.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.468+0000 7fd763e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585de3d5260 0x5585de3cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:14.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.468+0000 7fd763e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585de3d5260 0x5585de3cc9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44652/0 (socket says 172.21.15.28:44652) 2024-04-23T14:45:14.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.468+0000 7fd763e00640 1 -- 172.21.15.28:0/3374037780 learned_addr learned my addr 172.21.15.28:0/3374037780 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:14.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.468+0000 7fd763e00640 1 --2- 172.21.15.28:0/3374037780 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585de3ca270 0x5585de3e0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:14.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.468+0000 7fd762a00640 1 --2- 172.21.15.28:0/3374037780 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585de3cab90 0x5585de3e85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:14.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.468+0000 7fd763e00640 1 -- 172.21.15.28:0/3374037780 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585de3ca270 msgr2=0x5585de3e0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:14.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.468+0000 7fd763e00640 1 --2- 172.21.15.28:0/3374037780 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585de3ca270 0x5585de3e0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.468+0000 7fd763e00640 1 -- 172.21.15.28:0/3374037780 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585de3cab90 msgr2=0x5585de3e85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:14.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.468+0000 7fd763e00640 1 --2- 172.21.15.28:0/3374037780 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585de3cab90 0x5585de3e85e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.468+0000 7fd763e00640 1 -- 172.21.15.28:0/3374037780 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585de3d0750 con 0x5585de3d5260 2024-04-23T14:45:14.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.469+0000 7fd763e00640 1 --2- 172.21.15.28:0/3374037780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585de3d5260 0x5585de3cc9d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x5585de3d7fc0 tx=0x7fd75c009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eda1219e24cda67 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:14.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.469+0000 7fd74fe00640 1 -- 172.21.15.28:0/3374037780 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd75c03b030 con 0x5585de3d5260 2024-04-23T14:45:14.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.469+0000 7fd74fe00640 1 -- 172.21.15.28:0/3374037780 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd75c00fc30 con 0x5585de3d5260 2024-04-23T14:45:14.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.469+0000 7fd74fe00640 1 -- 172.21.15.28:0/3374037780 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd75c035420 con 0x5585de3d5260 2024-04-23T14:45:14.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.470+0000 7fd76d660740 1 -- 172.21.15.28:0/3374037780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585de3d5260 msgr2=0x5585de3cc9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:14.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.470+0000 7fd76d660740 1 --2- 172.21.15.28:0/3374037780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585de3d5260 0x5585de3cc9d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x5585de3d7fc0 tx=0x7fd75c009500 comp rx=0 tx=0).stop 2024-04-23T14:45:14.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.470+0000 7fd76d660740 1 -- 172.21.15.28:0/3374037780 shutdown_connections 2024-04-23T14:45:14.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.470+0000 7fd76d660740 1 --2- 172.21.15.28:0/3374037780 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585de3cab90 0x5585de3e85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.470+0000 7fd76d660740 1 --2- 172.21.15.28:0/3374037780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585de3d5260 0x5585de3cc9d0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.470+0000 7fd76d660740 1 --2- 172.21.15.28:0/3374037780 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585de3ca270 0x5585de3e0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.470+0000 7fd76d660740 1 -- 172.21.15.28:0/3374037780 >> 172.21.15.28:0/3374037780 conn(0x5585de3c6590 msgr2=0x5585de3c4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:14.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.470+0000 7fd76d660740 1 -- 172.21.15.28:0/3374037780 shutdown_connections 2024-04-23T14:45:14.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.470+0000 7fd76d660740 1 -- 172.21.15.28:0/3374037780 wait complete. 2024-04-23T14:45:14.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.470+0000 7fd76d660740 1 -- start start 2024-04-23T14:45:14.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.471+0000 7fd76d660740 1 Processor -- start 2024-04-23T14:45:14.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.471+0000 7fd76d660740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585de3ca270 0x5585de3d8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.471+0000 7fd76d660740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585de3cab90 0x5585de3d8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.471+0000 7fd76d660740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585de3d5260 0x5585de3d8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.471+0000 7fd76d660740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd74400d560 con 0x5585de3cab90 2024-04-23T14:45:14.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.471+0000 7fd76d660740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd74400e9d0 con 0x5585de3ca270 2024-04-23T14:45:14.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.471+0000 7fd76d660740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd74400dc30 con 0x5585de3d5260 2024-04-23T14:45:14.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.471+0000 7fd76d660740 10 client.0 _ll_register_callbacks cb 0x7fd75c015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:45:14.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.471+0000 7fd762a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585de3cab90 0x5585de3d8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:14.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.472+0000 7fd763e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585de3d5260 0x5585de3d8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:14.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.472+0000 7fd763e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585de3d5260 0x5585de3d8ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:40040/0 (socket says 172.21.15.28:40040) 2024-04-23T14:45:14.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.472+0000 7fd763e00640 1 -- 172.21.15.28:0/1263361788 learned_addr learned my addr 172.21.15.28:0/1263361788 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:14.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.472+0000 7fd763e00640 1 --2- 172.21.15.28:0/1263361788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585de3ca270 0x5585de3d8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:14.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.472+0000 7fd763e00640 1 -- 172.21.15.28:0/1263361788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585de3ca270 msgr2=0x5585de3d8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:14.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.472+0000 7fd763e00640 1 --2- 172.21.15.28:0/1263361788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585de3ca270 0x5585de3d8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.472+0000 7fd763e00640 1 -- 172.21.15.28:0/1263361788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585de3cab90 msgr2=0x5585de3d8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:14.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.472+0000 7fd763e00640 1 --2- 172.21.15.28:0/1263361788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585de3cab90 0x5585de3d8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.472+0000 7fd763e00640 1 -- 172.21.15.28:0/1263361788 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585de44a880 con 0x5585de3d5260 2024-04-23T14:45:14.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.472+0000 7fd763e00640 1 --2- 172.21.15.28:0/1263361788 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585de3d5260 0x5585de3d8ca0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd75c005d70 tx=0x7fd75c009e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd7fa70db96595ae server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:14.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.472+0000 7fd74d600640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:45:14.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.473+0000 7fd74d600640 1 -- 172.21.15.28:0/1263361788 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd75c03b040 con 0x5585de3d5260 2024-04-23T14:45:14.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.473+0000 7fd76d660740 1 -- 172.21.15.28:0/1263361788 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd75c00fc30 con 0x5585de3d5260 2024-04-23T14:45:14.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.473+0000 7fd74d600640 1 -- 172.21.15.28:0/1263361788 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fd75c004960 con 0x5585de3d5260 2024-04-23T14:45:14.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.473+0000 7fd74d600640 1 -- 172.21.15.28:0/1263361788 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fd75c035420 con 0x5585de3d5260 2024-04-23T14:45:14.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.473+0000 7fd76d660740 10 client.25513 fetch_fsmap learned FSMap version 20 2024-04-23T14:45:14.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.473+0000 7fd76d660740 1 -- 172.21.15.28:0/1263361788 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5585de44bba0 con 0x5585de3d5260 2024-04-23T14:45:14.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.473+0000 7fd74d600640 1 -- 172.21.15.28:0/1263361788 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd75c00fc30 con 0x5585de3d5260 2024-04-23T14:45:14.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.473+0000 7fd74d600640 1 -- 172.21.15.28:0/1263361788 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fd75c01b550 con 0x5585de3d5260 2024-04-23T14:45:14.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.474+0000 7fd76d660740 10 client.25513 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:45:14.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.474+0000 7fd76d660740 10 client.25513 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:45:14.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.474+0000 7fd76d660740 20 client.25513 populate_metadata read hostname 'smithi028' 2024-04-23T14:45:14.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.474+0000 7fd76d660740 1 --2- 172.21.15.28:0/1263361788 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5585de44c300 0x5585de44e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.474+0000 7fd76d660740 4 client.25513 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-23T14:45:14.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.474+0000 7fd76d660740 1 -- 172.21.15.28:0/1263361788 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5585de337700 con 0x5585de44c300 2024-04-23T14:45:14.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.474+0000 7fd762a00640 1 --2- 172.21.15.28:0/1263361788 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5585de44c300 0x5585de44e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:14.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.474+0000 7fd762a00640 1 --2- 172.21.15.28:0/1263361788 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5585de44c300 0x5585de44e720 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b480adf80644336 server_cookie=6d783a5c2aa0702 in_seq=0 out_seq=0 2024-04-23T14:45:14.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.474+0000 7fd74d600640 10 client.25513 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:45:14.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.476+0000 7fd74d600640 1 -- 172.21.15.28:0/1263361788 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5585de337700 con 0x5585de44c300 2024-04-23T14:45:14.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.476+0000 7fd74d600640 10 client.25513 handle_command_reply: tid=0 2024-04-23T14:45:14.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.476+0000 7fd76d660740 1 client.25513 shutdown 2024-04-23T14:45:14.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.476+0000 7fd76d660740 1 -- 172.21.15.28:0/1263361788 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585de3d5260 msgr2=0x5585de3d8ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:14.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.476+0000 7fd76d660740 1 --2- 172.21.15.28:0/1263361788 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585de3d5260 0x5585de3d8ca0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd75c005d70 tx=0x7fd75c009e90 comp rx=0 tx=0).stop 2024-04-23T14:45:14.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.476+0000 7fd76d660740 1 -- 172.21.15.28:0/1263361788 shutdown_connections 2024-04-23T14:45:14.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.476+0000 7fd76d660740 1 --2- 172.21.15.28:0/1263361788 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5585de44c300 0x5585de44e720 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.476+0000 7fd76d660740 1 --2- 172.21.15.28:0/1263361788 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585de3d5260 0x5585de3d8ca0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.476+0000 7fd76d660740 1 --2- 172.21.15.28:0/1263361788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585de3cab90 0x5585de3d8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.476+0000 7fd76d660740 1 --2- 172.21.15.28:0/1263361788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585de3ca270 0x5585de3d8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.476+0000 7fd76d660740 1 -- 172.21.15.28:0/1263361788 >> 172.21.15.28:0/1263361788 conn(0x5585de3c6590 msgr2=0x5585de3d2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:14.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.476+0000 7fd74d600640 0 client.25513 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:45:14.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.476+0000 7fd76d660740 1 -- 172.21.15.28:0/1263361788 shutdown_connections 2024-04-23T14:45:14.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.476+0000 7fd76d660740 1 -- 172.21.15.28:0/1263361788 wait complete. 2024-04-23T14:45:14.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.476+0000 7fd76d660740 20 client.25513 trim_cache size 0 max 16384 2024-04-23T14:45:14.483 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:45:14.483 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-23T14:45:14.484 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "6866b68b-8da6-4507-b07a-b500f2a1b353", 2024-04-23T14:45:14.484 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-23T14:45:14.484 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:45:14.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.477+0000 7fd769200640 1 -- 172.21.15.28:0/1368278648 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd7480a3f40 msgr2=0x7fd7480a6400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:14.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.477+0000 7fd769200640 1 --2- 172.21.15.28:0/1368278648 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd7480a3f40 0x7fd7480a6400 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd7540ce340 tx=0x7fd754009310 comp rx=0 tx=0).stop 2024-04-23T14:45:14.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.477+0000 7fd769200640 1 -- 172.21.15.28:0/1368278648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd764104320 msgr2=0x7fd7641a0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:14.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.477+0000 7fd769200640 1 --2- 172.21.15.28:0/1368278648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd764104320 0x7fd7641a0650 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd75400e9f0 tx=0x7fd75400eeb0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.477+0000 7fd769200640 1 -- 172.21.15.28:0/1368278648 shutdown_connections 2024-04-23T14:45:14.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.477+0000 7fd769200640 1 --2- 172.21.15.28:0/1368278648 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd7480a3f40 0x7fd7480a6400 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.477+0000 7fd769200640 1 --2- 172.21.15.28:0/1368278648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd76419a810 0x7fd76419acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.477+0000 7fd769200640 1 --2- 172.21.15.28:0/1368278648 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd7641093f0 0x7fd7641a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.477+0000 7fd769200640 1 --2- 172.21.15.28:0/1368278648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd764104320 0x7fd7641a0650 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.477+0000 7fd769200640 1 -- 172.21.15.28:0/1368278648 >> 172.21.15.28:0/1368278648 conn(0x7fd7640ffb40 msgr2=0x7fd764101ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:14.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.477+0000 7fd769200640 1 -- 172.21.15.28:0/1368278648 shutdown_connections 2024-04-23T14:45:14.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.477+0000 7fd769200640 1 -- 172.21.15.28:0/1368278648 wait complete. 2024-04-23T14:45:14.497 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:45:14.497 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:45:14.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.720+0000 7f143b200640 1 -- 172.21.15.28:0/1714019941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f143c104320 msgr2=0x7f143c104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:14.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.720+0000 7f143b200640 1 --2- 172.21.15.28:0/1714019941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f143c104320 0x7f143c104700 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f142c009900 tx=0x7f142c02a3f0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.720+0000 7f143b200640 1 -- 172.21.15.28:0/1714019941 shutdown_connections 2024-04-23T14:45:14.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.720+0000 7f143b200640 1 --2- 172.21.15.28:0/1714019941 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f143c104cd0 0x7f143c108d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.720+0000 7f143b200640 1 --2- 172.21.15.28:0/1714019941 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f143c1093f0 0x7f143c110fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.720+0000 7f143b200640 1 --2- 172.21.15.28:0/1714019941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f143c104320 0x7f143c104700 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.720+0000 7f143b200640 1 -- 172.21.15.28:0/1714019941 >> 172.21.15.28:0/1714019941 conn(0x7f143c0ffb40 msgr2=0x7f143c101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:14.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.720+0000 7f143b200640 1 -- 172.21.15.28:0/1714019941 shutdown_connections 2024-04-23T14:45:14.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.721+0000 7f143b200640 1 -- 172.21.15.28:0/1714019941 wait complete. 2024-04-23T14:45:14.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.721+0000 7f143b200640 1 Processor -- start 2024-04-23T14:45:14.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.722+0000 7f143b200640 1 -- start start 2024-04-23T14:45:14.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.722+0000 7f143b200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f143c104320 0x7f143c1a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.722+0000 7f143b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f143c1093f0 0x7f143c1a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.722+0000 7f143b200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f143c19a7d0 0x7f143c19ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.722+0000 7f143b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f143c1144d0 con 0x7f143c1093f0 2024-04-23T14:45:14.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.722+0000 7f143b200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f143c114350 con 0x7f143c19a7d0 2024-04-23T14:45:14.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.722+0000 7f143b200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f143c114650 con 0x7f143c104320 2024-04-23T14:45:14.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.722+0000 7f1439400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f143c1093f0 0x7f143c1a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:14.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.722+0000 7f1439400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f143c1093f0 0x7f143c1a0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44680/0 (socket says 172.21.15.28:44680) 2024-04-23T14:45:14.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.722+0000 7f1439400640 1 -- 172.21.15.28:0/3079356281 learned_addr learned my addr 172.21.15.28:0/3079356281 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:14.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.722+0000 7f1439e00640 1 --2- 172.21.15.28:0/3079356281 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f143c104320 0x7f143c1a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:14.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.723+0000 7f143a800640 1 --2- 172.21.15.28:0/3079356281 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f143c19a7d0 0x7f143c19ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:14.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.723+0000 7f1439e00640 1 -- 172.21.15.28:0/3079356281 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f143c19a7d0 msgr2=0x7f143c19ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:14.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.723+0000 7f1439e00640 1 --2- 172.21.15.28:0/3079356281 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f143c19a7d0 0x7f143c19ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.723+0000 7f1439e00640 1 -- 172.21.15.28:0/3079356281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f143c1093f0 msgr2=0x7f143c1a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:14.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.723+0000 7f1439e00640 1 --2- 172.21.15.28:0/3079356281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f143c1093f0 0x7f143c1a0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.723+0000 7f1439e00640 1 -- 172.21.15.28:0/3079356281 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f143c19b360 con 0x7f143c104320 2024-04-23T14:45:14.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.723+0000 7f1439400640 1 --2- 172.21.15.28:0/3079356281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f143c1093f0 0x7f143c1a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:45:14.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.723+0000 7f1439e00640 1 --2- 172.21.15.28:0/3079356281 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f143c104320 0x7f143c1a0660 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f142c02a9e0 tx=0x7f142c003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:14.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.724+0000 7f1432a00640 1 -- 172.21.15.28:0/3079356281 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f142c03f070 con 0x7f143c104320 2024-04-23T14:45:14.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.724+0000 7f143b200640 1 -- 172.21.15.28:0/3079356281 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f143c19b5f0 con 0x7f143c104320 2024-04-23T14:45:14.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.724+0000 7f143b200640 1 -- 172.21.15.28:0/3079356281 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f143c1a77a0 con 0x7f143c104320 2024-04-23T14:45:14.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.724+0000 7f1432a00640 1 -- 172.21.15.28:0/3079356281 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f142c031c90 con 0x7f143c104320 2024-04-23T14:45:14.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.724+0000 7f1432a00640 1 -- 172.21.15.28:0/3079356281 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f142c035030 con 0x7f143c104320 2024-04-23T14:45:14.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.727+0000 7f1432a00640 1 -- 172.21.15.28:0/3079356281 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138879+0+0 (secure 0 0 0) 0x7f142c042070 con 0x7f143c104320 2024-04-23T14:45:14.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.728+0000 7f1443dcd740 1 Processor -- start 2024-04-23T14:45:14.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.728+0000 7f1443dcd740 1 -- start start 2024-04-23T14:45:14.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.728+0000 7f1443dcd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da3d961a30 0x55da3d961e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.728+0000 7f1443dcd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da3d962350 0x55da3d96c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.728+0000 7f1443dcd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da3d96ca20 0x55da3d96ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.729+0000 7f1443dcd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55da3d8c71f0 con 0x55da3d96ca20 2024-04-23T14:45:14.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.729+0000 7f1443dcd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55da3d86d350 con 0x55da3d962350 2024-04-23T14:45:14.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.729+0000 7f1443dcd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55da3d8c31b0 con 0x55da3d961a30 2024-04-23T14:45:14.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.729+0000 7f1439e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da3d962350 0x55da3d96c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:14.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.729+0000 7f143a800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da3d961a30 0x55da3d961e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:14.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.729+0000 7f143a800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da3d961a30 0x55da3d961e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:40070/0 (socket says 172.21.15.28:40070) 2024-04-23T14:45:14.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.729+0000 7f143a800640 1 -- 172.21.15.28:0/129552097 learned_addr learned my addr 172.21.15.28:0/129552097 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:14.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.729+0000 7f1439400640 1 --2- 172.21.15.28:0/129552097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da3d96ca20 0x55da3d96ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:14.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.729+0000 7f1432a00640 1 --2- 172.21.15.28:0/3079356281 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f14200a3f10 0x7f14200a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.729+0000 7f1439e00640 1 -- 172.21.15.28:0/129552097 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da3d961a30 msgr2=0x55da3d961e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:14.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.729+0000 7f1439e00640 1 --2- 172.21.15.28:0/129552097 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da3d961a30 0x55da3d961e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.729+0000 7f1439e00640 1 -- 172.21.15.28:0/129552097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da3d96ca20 msgr2=0x55da3d96ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:14.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.729+0000 7f1439e00640 1 --2- 172.21.15.28:0/129552097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da3d96ca20 0x55da3d96ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.729+0000 7f1439e00640 1 -- 172.21.15.28:0/129552097 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da3d8c1f50 con 0x55da3d962350 2024-04-23T14:45:14.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.730+0000 7f1439e00640 1 --2- 172.21.15.28:0/129552097 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da3d962350 0x55da3d96c4e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f142c0ee870 tx=0x7f142c0bbdc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d684be536fc72b2b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:14.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.730+0000 7f143a800640 1 --2- 172.21.15.28:0/3079356281 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f14200a3f10 0x7f14200a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:14.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.730+0000 7f1427400640 1 -- 172.21.15.28:0/129552097 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f142c0c2c00 con 0x55da3d962350 2024-04-23T14:45:14.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.730+0000 7f1432a00640 1 -- 172.21.15.28:0/3079356281 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f142c0f12f0 con 0x7f143c104320 2024-04-23T14:45:14.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.730+0000 7f1427400640 1 -- 172.21.15.28:0/129552097 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f142c031e40 con 0x55da3d962350 2024-04-23T14:45:14.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.730+0000 7f1427400640 1 -- 172.21.15.28:0/129552097 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f142c0c1b10 con 0x55da3d962350 2024-04-23T14:45:14.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.731+0000 7f143a800640 1 --2- 172.21.15.28:0/3079356281 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f14200a3f10 0x7f14200a63d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f14340105d0 tx=0x7f1434004a40 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:14.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.731+0000 7f1443dcd740 1 -- 172.21.15.28:0/129552097 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da3d962350 msgr2=0x55da3d96c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:14.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.731+0000 7f1443dcd740 1 --2- 172.21.15.28:0/129552097 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da3d962350 0x55da3d96c4e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f142c0ee870 tx=0x7f142c0bbdc0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.731+0000 7f1443dcd740 1 -- 172.21.15.28:0/129552097 shutdown_connections 2024-04-23T14:45:14.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.731+0000 7f1443dcd740 1 --2- 172.21.15.28:0/129552097 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da3d962350 0x55da3d96c4e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.731+0000 7f1443dcd740 1 --2- 172.21.15.28:0/129552097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da3d96ca20 0x55da3d96ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.731+0000 7f1443dcd740 1 --2- 172.21.15.28:0/129552097 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da3d961a30 0x55da3d961e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.731+0000 7f1443dcd740 1 -- 172.21.15.28:0/129552097 >> 172.21.15.28:0/129552097 conn(0x55da3d95dd50 msgr2=0x55da3d95e130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:14.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.732+0000 7f1443dcd740 1 -- 172.21.15.28:0/129552097 shutdown_connections 2024-04-23T14:45:14.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.732+0000 7f1443dcd740 1 -- 172.21.15.28:0/129552097 wait complete. 2024-04-23T14:45:14.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.732+0000 7f1443dcd740 1 -- start start 2024-04-23T14:45:14.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.732+0000 7f1443dcd740 1 Processor -- start 2024-04-23T14:45:14.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.733+0000 7f1443dcd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da3d961a30 0x55da3d969a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.733+0000 7f1443dcd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da3d962350 0x55da3d969f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.733+0000 7f1443dcd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da3d96ca20 0x55da3d96a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.733+0000 7f1443dcd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55da3d96f5d0 con 0x55da3d962350 2024-04-23T14:45:14.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.733+0000 7f1443dcd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55da3d880690 con 0x55da3d961a30 2024-04-23T14:45:14.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.733+0000 7f1443dcd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55da3d96f750 con 0x55da3d96ca20 2024-04-23T14:45:14.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.733+0000 7f1443dcd740 10 client.0 _ll_register_callbacks cb 0x55da3d8bc650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:45:14.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.733+0000 7f1439e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da3d961a30 0x55da3d969a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:14.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.733+0000 7f1439e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da3d961a30 0x55da3d969a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:55050/0 (socket says 172.21.15.28:55050) 2024-04-23T14:45:14.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.733+0000 7f1439e00640 1 -- 172.21.15.28:0/942045191 learned_addr learned my addr 172.21.15.28:0/942045191 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:14.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.733+0000 7f143a800640 1 --2- 172.21.15.28:0/942045191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da3d96ca20 0x55da3d96a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:14.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.733+0000 7f1439400640 1 --2- 172.21.15.28:0/942045191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da3d962350 0x55da3d969f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:14.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.734+0000 7f1439e00640 1 -- 172.21.15.28:0/942045191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da3d96ca20 msgr2=0x55da3d96a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:14.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.734+0000 7f1439e00640 1 --2- 172.21.15.28:0/942045191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da3d96ca20 0x55da3d96a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.734+0000 7f1439e00640 1 -- 172.21.15.28:0/942045191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da3d962350 msgr2=0x55da3d969f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:14.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.734+0000 7f1439e00640 1 --2- 172.21.15.28:0/942045191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da3d962350 0x55da3d969f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.734+0000 7f1439e00640 1 -- 172.21.15.28:0/942045191 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da3d87f920 con 0x55da3d961a30 2024-04-23T14:45:14.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.734+0000 7f1439e00640 1 --2- 172.21.15.28:0/942045191 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da3d961a30 0x55da3d969a20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f142c0b9a90 tx=0x7f142c03b6b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49fa993b9edb102e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:14.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.734+0000 7f141be00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:45:14.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.734+0000 7f141be00640 1 -- 172.21.15.28:0/942045191 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f142c049040 con 0x55da3d961a30 2024-04-23T14:45:14.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.734+0000 7f141be00640 1 -- 172.21.15.28:0/942045191 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f142c0c5ab0 con 0x55da3d961a30 2024-04-23T14:45:14.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.734+0000 7f1443dcd740 1 -- 172.21.15.28:0/942045191 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55da3d8d4c10 con 0x55da3d961a30 2024-04-23T14:45:14.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.735+0000 7f141be00640 1 -- 172.21.15.28:0/942045191 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f142c0c1d30 con 0x55da3d961a30 2024-04-23T14:45:14.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.735+0000 7f1443dcd740 10 client.25517 fetch_fsmap learned FSMap version 20 2024-04-23T14:45:14.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.735+0000 7f1443dcd740 1 -- 172.21.15.28:0/942045191 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55da3d6a37d0 con 0x55da3d961a30 2024-04-23T14:45:14.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.735+0000 7f141be00640 1 -- 172.21.15.28:0/942045191 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55da3d8d4c10 con 0x55da3d961a30 2024-04-23T14:45:14.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.735+0000 7f141be00640 1 -- 172.21.15.28:0/942045191 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f142c0fd070 con 0x55da3d961a30 2024-04-23T14:45:14.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.735+0000 7f1443dcd740 10 client.25517 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:45:14.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.735+0000 7f1443dcd740 10 client.25517 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:45:14.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.735+0000 7f1443dcd740 20 client.25517 populate_metadata read hostname 'smithi028' 2024-04-23T14:45:14.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.735+0000 7f1443dcd740 1 --2- 172.21.15.28:0/942045191 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55da3d97c9c0 0x55da3d97edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.735+0000 7f1443dcd740 4 client.25517 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:45:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.735+0000 7f1443dcd740 1 -- 172.21.15.28:0/942045191 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da3d8cea80 con 0x55da3d97c9c0 2024-04-23T14:45:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.736+0000 7f1439400640 1 --2- 172.21.15.28:0/942045191 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55da3d97c9c0 0x55da3d97edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.737+0000 7f1439400640 1 --2- 172.21.15.28:0/942045191 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55da3d97c9c0 0x55da3d97edb0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d01ede5774b671c3 server_cookie=f9f793c672331d6b in_seq=0 out_seq=0 2024-04-23T14:45:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.737+0000 7f141be00640 10 client.25517 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:45:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.737+0000 7f141be00640 1 -- 172.21.15.28:0/942045191 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55da3d8cea80 con 0x55da3d97c9c0 2024-04-23T14:45:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.737+0000 7f141be00640 10 client.25517 handle_command_reply: tid=0 2024-04-23T14:45:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.738+0000 7f1443dcd740 1 client.25517 shutdown 2024-04-23T14:45:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.738+0000 7f1443dcd740 1 -- 172.21.15.28:0/942045191 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da3d961a30 msgr2=0x55da3d969a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.738+0000 7f1443dcd740 1 --2- 172.21.15.28:0/942045191 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da3d961a30 0x55da3d969a20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f142c0b9a90 tx=0x7f142c03b6b0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.738+0000 7f1443dcd740 1 -- 172.21.15.28:0/942045191 shutdown_connections 2024-04-23T14:45:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.738+0000 7f1443dcd740 1 --2- 172.21.15.28:0/942045191 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55da3d97c9c0 0x55da3d97edb0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.738+0000 7f1443dcd740 1 --2- 172.21.15.28:0/942045191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da3d96ca20 0x55da3d96a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.738+0000 7f141be00640 0 client.25517 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:45:14.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.738+0000 7f1443dcd740 1 --2- 172.21.15.28:0/942045191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da3d962350 0x55da3d969f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.738+0000 7f1443dcd740 1 --2- 172.21.15.28:0/942045191 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da3d961a30 0x55da3d969a20 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.738+0000 7f1443dcd740 1 -- 172.21.15.28:0/942045191 >> 172.21.15.28:0/942045191 conn(0x55da3d95dd50 msgr2=0x55da3d95cfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:14.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.738+0000 7f1443dcd740 1 -- 172.21.15.28:0/942045191 shutdown_connections 2024-04-23T14:45:14.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.738+0000 7f1443dcd740 1 -- 172.21.15.28:0/942045191 wait complete. 2024-04-23T14:45:14.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.738+0000 7f1443dcd740 20 client.25517 trim_cache size 0 max 16384 2024-04-23T14:45:14.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.762+0000 7f1443dcd740 1 Processor -- start 2024-04-23T14:45:14.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.762+0000 7f1443dcd740 1 -- start start 2024-04-23T14:45:14.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.762+0000 7f1443dcd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da3d961a30 0x55da3d978400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.762+0000 7f1443dcd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da3d962350 0x55da3d984510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.762+0000 7f1443dcd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da3d96ca20 0x55da3d984a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.762+0000 7f1443dcd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f143c1adcf0 con 0x55da3d962350 2024-04-23T14:45:14.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.762+0000 7f1443dcd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f143c1b0570 con 0x55da3d961a30 2024-04-23T14:45:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.762+0000 7f1443dcd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f143c1ae360 con 0x55da3d96ca20 2024-04-23T14:45:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.763+0000 7f1439400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da3d962350 0x55da3d984510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.763+0000 7f1439e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da3d961a30 0x55da3d978400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.763+0000 7f143a800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da3d96ca20 0x55da3d984a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.763+0000 7f1439e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da3d961a30 0x55da3d978400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:55064/0 (socket says 172.21.15.28:55064) 2024-04-23T14:45:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.763+0000 7f1439e00640 1 -- 172.21.15.28:0/1195345184 learned_addr learned my addr 172.21.15.28:0/1195345184 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.763+0000 7f1439400640 1 -- 172.21.15.28:0/1195345184 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da3d96ca20 msgr2=0x55da3d984a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.763+0000 7f1439400640 1 --2- 172.21.15.28:0/1195345184 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da3d96ca20 0x55da3d984a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.763+0000 7f1439400640 1 -- 172.21.15.28:0/1195345184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da3d961a30 msgr2=0x55da3d978400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.763+0000 7f1439400640 1 --2- 172.21.15.28:0/1195345184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da3d961a30 0x55da3d978400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.763+0000 7f1439400640 1 -- 172.21.15.28:0/1195345184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da3d5d93f0 con 0x55da3d962350 2024-04-23T14:45:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.763+0000 7f143a800640 1 --2- 172.21.15.28:0/1195345184 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da3d96ca20 0x55da3d984a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:45:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.763+0000 7f1439e00640 1 --2- 172.21.15.28:0/1195345184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da3d961a30 0x55da3d978400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:45:14.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.764+0000 7f1439400640 1 --2- 172.21.15.28:0/1195345184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da3d962350 0x55da3d984510 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x55da3d95eee0 tx=0x7f1428005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=633563117f224591 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:14.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.764+0000 7f1426a00640 1 -- 172.21.15.28:0/1195345184 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1428024070 con 0x55da3d962350 2024-04-23T14:45:14.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.764+0000 7f1426a00640 1 -- 172.21.15.28:0/1195345184 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1428010800 con 0x55da3d962350 2024-04-23T14:45:14.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.764+0000 7f1426a00640 1 -- 172.21.15.28:0/1195345184 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1428004bc0 con 0x55da3d962350 2024-04-23T14:45:14.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.765+0000 7f1443dcd740 1 -- 172.21.15.28:0/1195345184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da3d962350 msgr2=0x55da3d984510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:14.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.765+0000 7f1443dcd740 1 --2- 172.21.15.28:0/1195345184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da3d962350 0x55da3d984510 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x55da3d95eee0 tx=0x7f1428005f90 comp rx=0 tx=0).stop 2024-04-23T14:45:14.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.765+0000 7f1443dcd740 1 -- 172.21.15.28:0/1195345184 shutdown_connections 2024-04-23T14:45:14.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.765+0000 7f1443dcd740 1 --2- 172.21.15.28:0/1195345184 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da3d96ca20 0x55da3d984a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.765+0000 7f1443dcd740 1 --2- 172.21.15.28:0/1195345184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da3d962350 0x55da3d984510 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.765+0000 7f1443dcd740 1 --2- 172.21.15.28:0/1195345184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da3d961a30 0x55da3d978400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.765+0000 7f1443dcd740 1 -- 172.21.15.28:0/1195345184 >> 172.21.15.28:0/1195345184 conn(0x55da3d95dd50 msgr2=0x55da3d95cfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:14.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.765+0000 7f1443dcd740 1 -- 172.21.15.28:0/1195345184 shutdown_connections 2024-04-23T14:45:14.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.765+0000 7f1443dcd740 1 -- 172.21.15.28:0/1195345184 wait complete. 2024-04-23T14:45:14.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.766+0000 7f1443dcd740 1 -- start start 2024-04-23T14:45:14.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.766+0000 7f1443dcd740 1 Processor -- start 2024-04-23T14:45:14.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.767+0000 7f1443dcd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da3d961a30 0x55da3d970390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.767+0000 7f1443dcd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da3d962350 0x55da3d9708d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.767+0000 7f1443dcd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da3d96ca20 0x55da3d970e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.767+0000 7f1443dcd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f143c1aea30 con 0x55da3d96ca20 2024-04-23T14:45:14.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.767+0000 7f1443dcd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f143c1afea0 con 0x55da3d962350 2024-04-23T14:45:14.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.767+0000 7f1443dcd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f143c1af100 con 0x55da3d961a30 2024-04-23T14:45:14.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.767+0000 7f1443dcd740 10 client.0 _ll_register_callbacks cb 0x7f1428009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:45:14.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.767+0000 7f1439e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da3d961a30 0x55da3d970390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:14.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.767+0000 7f1439400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da3d962350 0x55da3d9708d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:14.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.767+0000 7f1439e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da3d961a30 0x55da3d970390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:40096/0 (socket says 172.21.15.28:40096) 2024-04-23T14:45:14.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.767+0000 7f1439e00640 1 -- 172.21.15.28:0/2292937213 learned_addr learned my addr 172.21.15.28:0/2292937213 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:14.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.767+0000 7f143a800640 1 --2- 172.21.15.28:0/2292937213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da3d96ca20 0x55da3d970e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:14.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.768+0000 7f1439e00640 1 -- 172.21.15.28:0/2292937213 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da3d962350 msgr2=0x55da3d9708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:14.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.768+0000 7f1439e00640 1 --2- 172.21.15.28:0/2292937213 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da3d962350 0x55da3d9708d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.768+0000 7f1439e00640 1 -- 172.21.15.28:0/2292937213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da3d96ca20 msgr2=0x55da3d970e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:14.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.768+0000 7f1439e00640 1 --2- 172.21.15.28:0/2292937213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da3d96ca20 0x55da3d970e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.768+0000 7f1439e00640 1 -- 172.21.15.28:0/2292937213 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da3d5990f0 con 0x55da3d961a30 2024-04-23T14:45:14.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.768+0000 7f1439400640 1 --2- 172.21.15.28:0/2292937213 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da3d962350 0x55da3d9708d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:45:14.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.768+0000 7f1439e00640 1 --2- 172.21.15.28:0/2292937213 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da3d961a30 0x55da3d970390 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f142c009900 tx=0x7f142c02a3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af59d198a63a60a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:14.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.768+0000 7f141be00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:45:14.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.768+0000 7f141be00640 1 -- 172.21.15.28:0/2292937213 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f142c0d6060 con 0x55da3d961a30 2024-04-23T14:45:14.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.769+0000 7f1443dcd740 1 -- 172.21.15.28:0/2292937213 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1428010800 con 0x55da3d961a30 2024-04-23T14:45:14.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.769+0000 7f141be00640 1 -- 172.21.15.28:0/2292937213 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f142c0bd320 con 0x55da3d961a30 2024-04-23T14:45:14.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.769+0000 7f141be00640 1 -- 172.21.15.28:0/2292937213 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f142c049070 con 0x55da3d961a30 2024-04-23T14:45:14.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.769+0000 7f1443dcd740 10 client.25531 fetch_fsmap learned FSMap version 20 2024-04-23T14:45:14.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.769+0000 7f1443dcd740 1 -- 172.21.15.28:0/2292937213 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55da3d88c810 con 0x55da3d961a30 2024-04-23T14:45:14.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.769+0000 7f141be00640 1 -- 172.21.15.28:0/2292937213 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1428010800 con 0x55da3d961a30 2024-04-23T14:45:14.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.769+0000 7f141be00640 1 -- 172.21.15.28:0/2292937213 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f142c0fd040 con 0x55da3d961a30 2024-04-23T14:45:14.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.769+0000 7f1443dcd740 10 client.25531 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:45:14.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.769+0000 7f1443dcd740 10 client.25531 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:45:14.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.769+0000 7f1443dcd740 20 client.25531 populate_metadata read hostname 'smithi028' 2024-04-23T14:45:14.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.770+0000 7f1443dcd740 1 --2- 172.21.15.28:0/2292937213 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55da3d9e30c0 0x55da3d9e54b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:14.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.770+0000 7f1443dcd740 4 client.25531 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:45:14.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.770+0000 7f1443dcd740 1 -- 172.21.15.28:0/2292937213 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55da3d84a360 con 0x55da3d9e30c0 2024-04-23T14:45:14.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.770+0000 7f1439400640 1 --2- 172.21.15.28:0/2292937213 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55da3d9e30c0 0x55da3d9e54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:14.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.771+0000 7f1439400640 1 --2- 172.21.15.28:0/2292937213 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55da3d9e30c0 0x55da3d9e54b0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10456d5ebb094b28 server_cookie=5dddb84517643b79 in_seq=0 out_seq=0 2024-04-23T14:45:14.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.771+0000 7f141be00640 10 client.25531 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:45:14.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.771+0000 7f141be00640 1 -- 172.21.15.28:0/2292937213 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55da3d84a360 con 0x55da3d9e30c0 2024-04-23T14:45:14.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.771+0000 7f141be00640 10 client.25531 handle_command_reply: tid=0 2024-04-23T14:45:14.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.771+0000 7f1443dcd740 1 client.25531 shutdown 2024-04-23T14:45:14.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.771+0000 7f1443dcd740 1 -- 172.21.15.28:0/2292937213 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da3d961a30 msgr2=0x55da3d970390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:14.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.771+0000 7f1443dcd740 1 --2- 172.21.15.28:0/2292937213 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da3d961a30 0x55da3d970390 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f142c009900 tx=0x7f142c02a3f0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.771+0000 7f1443dcd740 1 -- 172.21.15.28:0/2292937213 shutdown_connections 2024-04-23T14:45:14.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.771+0000 7f1443dcd740 1 --2- 172.21.15.28:0/2292937213 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55da3d9e30c0 0x55da3d9e54b0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.771+0000 7f1443dcd740 1 --2- 172.21.15.28:0/2292937213 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da3d962350 0x55da3d9708d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.771+0000 7f1443dcd740 1 --2- 172.21.15.28:0/2292937213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da3d96ca20 0x55da3d970e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.771+0000 7f1443dcd740 1 --2- 172.21.15.28:0/2292937213 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55da3d961a30 0x55da3d970390 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.771+0000 7f141be00640 0 client.25531 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:45:14.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.771+0000 7f1443dcd740 1 -- 172.21.15.28:0/2292937213 >> 172.21.15.28:0/2292937213 conn(0x55da3d95dd50 msgr2=0x55da3d963180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:14.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.771+0000 7f1443dcd740 1 -- 172.21.15.28:0/2292937213 shutdown_connections 2024-04-23T14:45:14.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.772+0000 7f1443dcd740 1 -- 172.21.15.28:0/2292937213 wait complete. 2024-04-23T14:45:14.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.772+0000 7f1443dcd740 20 client.25531 trim_cache size 0 max 16384 2024-04-23T14:45:14.774 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:45:14.774 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-23T14:45:14.774 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-23T14:45:14.775 INFO:teuthology.orchestra.run.smithi028.stdout: "6866b68b-8da6-4507-b07a-b500f2a1b353": { 2024-04-23T14:45:14.775 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-23T14:45:14.775 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "6866b68b-8da6-4507-b07a-b500f2a1b353", 2024-04-23T14:45:14.775 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-23T14:45:14.775 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:45:14.775 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:45:14.775 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:45:14.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.772+0000 7f143b200640 1 -- 172.21.15.28:0/3079356281 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f14200a3f10 msgr2=0x7f14200a63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:14.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.772+0000 7f143b200640 1 --2- 172.21.15.28:0/3079356281 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f14200a3f10 0x7f14200a63d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f14340105d0 tx=0x7f1434004a40 comp rx=0 tx=0).stop 2024-04-23T14:45:14.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.772+0000 7f143b200640 1 -- 172.21.15.28:0/3079356281 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f143c104320 msgr2=0x7f143c1a0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:14.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.772+0000 7f143b200640 1 --2- 172.21.15.28:0/3079356281 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f143c104320 0x7f143c1a0660 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f142c02a9e0 tx=0x7f142c003b90 comp rx=0 tx=0).stop 2024-04-23T14:45:14.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.773+0000 7f143b200640 1 -- 172.21.15.28:0/3079356281 shutdown_connections 2024-04-23T14:45:14.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.773+0000 7f143b200640 1 --2- 172.21.15.28:0/3079356281 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f14200a3f10 0x7f14200a63d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.773+0000 7f143b200640 1 --2- 172.21.15.28:0/3079356281 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f143c19a7d0 0x7f143c19ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.773+0000 7f143b200640 1 --2- 172.21.15.28:0/3079356281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f143c1093f0 0x7f143c1a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.773+0000 7f143b200640 1 --2- 172.21.15.28:0/3079356281 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f143c104320 0x7f143c1a0660 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:14.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.773+0000 7f143b200640 1 -- 172.21.15.28:0/3079356281 >> 172.21.15.28:0/3079356281 conn(0x7f143c0ffb40 msgr2=0x7f143c101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:14.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.773+0000 7f143b200640 1 -- 172.21.15.28:0/3079356281 shutdown_connections 2024-04-23T14:45:14.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:14.773+0000 7f143b200640 1 -- 172.21.15.28:0/3079356281 wait complete. 2024-04-23T14:45:14.794 INFO:tasks.cephfs.filesystem:scrub status for tag:6866b68b-8da6-4507-b07a-b500f2a1b353 - {'path': '/', 'tag': '6866b68b-8da6-4507-b07a-b500f2a1b353', 'options': 'recursive,force'} 2024-04-23T14:45:14.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:14 smithi028 ceph-mon[27810]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-04-23T14:45:15.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:14 smithi105 ceph-mon[32555]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-04-23T14:45:15.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:14 smithi038 ceph-mon[32348]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-04-23T14:45:16.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:15 smithi105 ceph-mon[32555]: scrub queued for path: / 2024-04-23T14:45:16.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:15 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:45:16.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:15 smithi105 ceph-mon[32555]: scrub summary: active paths [/] 2024-04-23T14:45:16.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:15 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:45:16.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:15 smithi105 ceph-mon[32555]: scrub summary: idle 2024-04-23T14:45:16.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:15 smithi038 ceph-mon[32348]: scrub queued for path: / 2024-04-23T14:45:16.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:15 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:45:16.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:15 smithi038 ceph-mon[32348]: scrub summary: active paths [/] 2024-04-23T14:45:16.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:15 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:45:16.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:15 smithi038 ceph-mon[32348]: scrub summary: idle 2024-04-23T14:45:16.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:15 smithi028 ceph-mon[27810]: scrub queued for path: / 2024-04-23T14:45:16.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:15 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:45:16.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:15 smithi028 ceph-mon[27810]: scrub summary: active paths [/] 2024-04-23T14:45:16.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:15 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:45:16.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:15 smithi028 ceph-mon[27810]: scrub summary: idle 2024-04-23T14:45:17.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:16 smithi105 ceph-mon[32555]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-04-23T14:45:17.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:16 smithi038 ceph-mon[32348]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-04-23T14:45:17.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:16 smithi028 ceph-mon[27810]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-04-23T14:45:19.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:18 smithi105 ceph-mon[32555]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 170 B/s wr, 2 op/s 2024-04-23T14:45:19.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:18 smithi038 ceph-mon[32348]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 170 B/s wr, 2 op/s 2024-04-23T14:45:19.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:18 smithi028 ceph-mon[27810]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 170 B/s wr, 2 op/s 2024-04-23T14:45:21.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:20 smithi105 ceph-mon[32555]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 85 B/s wr, 1 op/s 2024-04-23T14:45:21.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:20 smithi038 ceph-mon[32348]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 85 B/s wr, 1 op/s 2024-04-23T14:45:21.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:20 smithi028 ceph-mon[27810]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 85 B/s wr, 1 op/s 2024-04-23T14:45:23.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:22 smithi105 ceph-mon[32555]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 85 B/s wr, 1 op/s 2024-04-23T14:45:23.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:22 smithi038 ceph-mon[32348]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 85 B/s wr, 1 op/s 2024-04-23T14:45:23.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:22 smithi028 ceph-mon[27810]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 85 B/s wr, 1 op/s 2024-04-23T14:45:25.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:24 smithi105 ceph-mon[32555]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 85 B/s wr, 2 op/s 2024-04-23T14:45:25.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:24 smithi038 ceph-mon[32348]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 85 B/s wr, 2 op/s 2024-04-23T14:45:25.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:24 smithi028 ceph-mon[27810]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 85 B/s wr, 2 op/s 2024-04-23T14:45:27.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:26 smithi105 ceph-mon[32555]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-23T14:45:27.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:26 smithi038 ceph-mon[32348]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-23T14:45:27.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:26 smithi028 ceph-mon[27810]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-23T14:45:29.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:28 smithi105 ceph-mon[32555]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s 2024-04-23T14:45:29.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:28 smithi038 ceph-mon[32348]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s 2024-04-23T14:45:29.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:28 smithi028 ceph-mon[27810]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s 2024-04-23T14:45:31.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:30 smithi105 ceph-mon[32555]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-23T14:45:31.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:30 smithi038 ceph-mon[32348]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-23T14:45:31.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:30 smithi028 ceph-mon[27810]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-23T14:45:33.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:32 smithi105 ceph-mon[32555]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-23T14:45:33.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:32 smithi038 ceph-mon[32348]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-23T14:45:33.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:32 smithi028 ceph-mon[27810]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-23T14:45:35.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:34 smithi105 ceph-mon[32555]: pgmap v49: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 341 B/s wr, 2 op/s 2024-04-23T14:45:35.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:34 smithi105 ceph-mon[32555]: mgrmap e27: x(active, since 92s), standbys: z, y 2024-04-23T14:45:35.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:34 smithi038 ceph-mon[32348]: pgmap v49: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 341 B/s wr, 2 op/s 2024-04-23T14:45:35.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:34 smithi038 ceph-mon[32348]: mgrmap e27: x(active, since 92s), standbys: z, y 2024-04-23T14:45:35.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:34 smithi028 ceph-mon[27810]: pgmap v49: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 341 B/s wr, 2 op/s 2024-04-23T14:45:35.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:34 smithi028 ceph-mon[27810]: mgrmap e27: x(active, since 92s), standbys: z, y 2024-04-23T14:45:37.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:36 smithi105 ceph-mon[32555]: pgmap v50: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 341 B/s wr, 2 op/s 2024-04-23T14:45:37.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:36 smithi038 ceph-mon[32348]: pgmap v50: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 341 B/s wr, 2 op/s 2024-04-23T14:45:37.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:36 smithi028 ceph-mon[27810]: pgmap v50: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 341 B/s wr, 2 op/s 2024-04-23T14:45:39.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:38 smithi105 ceph-mon[32555]: pgmap v51: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 341 B/s wr, 2 op/s 2024-04-23T14:45:39.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:38 smithi038 ceph-mon[32348]: pgmap v51: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 341 B/s wr, 2 op/s 2024-04-23T14:45:39.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:38 smithi028 ceph-mon[27810]: pgmap v51: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 341 B/s wr, 2 op/s 2024-04-23T14:45:41.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:40 smithi105 ceph-mon[32555]: pgmap v52: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 341 B/s wr, 2 op/s 2024-04-23T14:45:41.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:40 smithi038 ceph-mon[32348]: pgmap v52: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 341 B/s wr, 2 op/s 2024-04-23T14:45:41.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:40 smithi028 ceph-mon[27810]: pgmap v52: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 341 B/s wr, 2 op/s 2024-04-23T14:45:42.303 INFO:tasks.workunit.client.0.smithi028.stderr:Updating files: 58% (7189/12271) Updating files: 59% (7240/12271) Updating files: 60% (7363/12271) Updating files: 61% (7486/12271) Updating files: 62% (7609/12271) Updating files: 63% (7731/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 95% (11757/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-04-23T14:45:42.985 INFO:tasks.workunit.client.0.smithi028.stderr:Note: switching to 'd44fddd910b5e94da0f2357af4be37d7bd495ae0'. 2024-04-23T14:45:42.985 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-04-23T14:45:42.985 INFO:tasks.workunit.client.0.smithi028.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-23T14:45:42.985 INFO:tasks.workunit.client.0.smithi028.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-23T14:45:42.985 INFO:tasks.workunit.client.0.smithi028.stderr:state without impacting any branches by switching back to a branch. 2024-04-23T14:45:42.985 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-04-23T14:45:42.985 INFO:tasks.workunit.client.0.smithi028.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-23T14:45:42.986 INFO:tasks.workunit.client.0.smithi028.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-23T14:45:42.986 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-04-23T14:45:42.986 INFO:tasks.workunit.client.0.smithi028.stderr: git switch -c 2024-04-23T14:45:42.986 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-04-23T14:45:42.986 INFO:tasks.workunit.client.0.smithi028.stderr:Or undo this operation with: 2024-04-23T14:45:42.986 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-04-23T14:45:42.986 INFO:tasks.workunit.client.0.smithi028.stderr: git switch - 2024-04-23T14:45:42.986 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-04-23T14:45:42.986 INFO:tasks.workunit.client.0.smithi028.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-23T14:45:42.986 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-04-23T14:45:42.986 INFO:tasks.workunit.client.0.smithi028.stderr:HEAD is now at d44fddd910b DNM: trigger builds again 2024-04-23T14:45:43.000 DEBUG:teuthology.orchestra.run.smithi028:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-23T14:45:43.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:42 smithi105 ceph-mon[32555]: pgmap v53: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 341 B/s wr, 2 op/s 2024-04-23T14:45:43.059 INFO:tasks.workunit.client.0.smithi028.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-23T14:45:43.065 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-23T14:45:43.065 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-23T14:45:43.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:42 smithi038 ceph-mon[32348]: pgmap v53: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 341 B/s wr, 2 op/s 2024-04-23T14:45:43.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:42 smithi028 ceph-mon[27810]: pgmap v53: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 341 B/s wr, 2 op/s 2024-04-23T14:45:43.905 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-23T14:45:44.012 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-23T14:45:44.102 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-23T14:45:44.107 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-23T14:45:44.107 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-23T14:45:44.197 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-23T14:45:44.207 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-23T14:45:44.207 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-23T14:45:44.260 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-04-23T14:45:44.261 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-04-23T14:45:44.262 DEBUG:teuthology.orchestra.run.smithi028:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d44fddd910b5e94da0f2357af4be37d7bd495ae0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-04-23T14:45:44.336 INFO:tasks.workunit.client.0.smithi028.stderr:+ dbench 1 2024-04-23T14:45:44.350 INFO:tasks.workunit.client.0.smithi028.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-23T14:45:44.351 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-23T14:45:44.351 INFO:tasks.workunit.client.0.smithi028.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-04-23T14:45:44.357 INFO:tasks.workunit.client.0.smithi028.stdout:0 of 1 processes prepared for launch 0 sec 2024-04-23T14:45:44.457 INFO:tasks.workunit.client.0.smithi028.stdout:1 of 1 processes prepared for launch 0 sec 2024-04-23T14:45:44.457 INFO:tasks.workunit.client.0.smithi028.stdout:releasing clients 2024-04-23T14:45:44.795 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:45:44.795 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:45:45.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:44 smithi105 ceph-mon[32555]: pgmap v54: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 341 B/s wr, 2 op/s 2024-04-23T14:45:45.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.035+0000 7f84ebe00640 1 -- 172.21.15.28:0/3052284758 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84ec10c480 msgr2=0x7f84ec10e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:45.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.035+0000 7f84ebe00640 1 --2- 172.21.15.28:0/3052284758 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84ec10c480 0x7f84ec10e920 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f84e4009ca0 tx=0x7f84e402a950 comp rx=0 tx=0).stop 2024-04-23T14:45:45.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.036+0000 7f84ebe00640 1 -- 172.21.15.28:0/3052284758 shutdown_connections 2024-04-23T14:45:45.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.036+0000 7f84ebe00640 1 --2- 172.21.15.28:0/3052284758 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84ec103980 0x7f84ec10be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.036+0000 7f84ebe00640 1 --2- 172.21.15.28:0/3052284758 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84ec10c480 0x7f84ec10e920 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.036+0000 7f84ebe00640 1 --2- 172.21.15.28:0/3052284758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84ec103060 0x7f84ec103440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.036+0000 7f84ebe00640 1 -- 172.21.15.28:0/3052284758 >> 172.21.15.28:0/3052284758 conn(0x7f84ec0fc880 msgr2=0x7f84ec0feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:45.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.036+0000 7f84ebe00640 1 -- 172.21.15.28:0/3052284758 shutdown_connections 2024-04-23T14:45:45.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.036+0000 7f84ebe00640 1 -- 172.21.15.28:0/3052284758 wait complete. 2024-04-23T14:45:45.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.037+0000 7f84ebe00640 1 Processor -- start 2024-04-23T14:45:45.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.037+0000 7f84ebe00640 1 -- start start 2024-04-23T14:45:45.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.037+0000 7f84ebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84ec103060 0x7f84ec198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.037+0000 7f84ebe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84ec103980 0x7f84ec198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.037+0000 7f84ebe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84ec10c480 0x7f84ec19cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.037+0000 7f84ebe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f84ec112290 con 0x7f84ec103060 2024-04-23T14:45:45.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.037+0000 7f84ebe00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f84ec112110 con 0x7f84ec103980 2024-04-23T14:45:45.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.037+0000 7f84ebe00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f84ec112410 con 0x7f84ec10c480 2024-04-23T14:45:45.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.038+0000 7f84eaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84ec103060 0x7f84ec198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.038+0000 7f84e2000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84ec103980 0x7f84ec198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.038+0000 7f84e2000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84ec103980 0x7f84ec198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:53520/0 (socket says 172.21.15.28:53520) 2024-04-23T14:45:45.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.038+0000 7f84e2000640 1 -- 172.21.15.28:0/1881843827 learned_addr learned my addr 172.21.15.28:0/1881843827 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:45.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.038+0000 7f84eb400640 1 --2- 172.21.15.28:0/1881843827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84ec10c480 0x7f84ec19cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.038+0000 7f84e2000640 1 -- 172.21.15.28:0/1881843827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84ec10c480 msgr2=0x7f84ec19cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:45.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.038+0000 7f84e2000640 1 --2- 172.21.15.28:0/1881843827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84ec10c480 0x7f84ec19cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.038+0000 7f84e2000640 1 -- 172.21.15.28:0/1881843827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84ec103060 msgr2=0x7f84ec198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:45.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.038+0000 7f84e2000640 1 --2- 172.21.15.28:0/1881843827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84ec103060 0x7f84ec198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.038+0000 7f84e2000640 1 -- 172.21.15.28:0/1881843827 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84ec19d700 con 0x7f84ec103980 2024-04-23T14:45:45.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.038+0000 7f84e2000640 1 --2- 172.21.15.28:0/1881843827 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84ec103980 0x7f84ec198c50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f84d400e9f0 tx=0x7f84d400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:45.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.039+0000 7f84e3e00640 1 -- 172.21.15.28:0/1881843827 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d400cd80 con 0x7f84ec103980 2024-04-23T14:45:45.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.039+0000 7f84ebe00640 1 -- 172.21.15.28:0/1881843827 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84ec19d9f0 con 0x7f84ec103980 2024-04-23T14:45:45.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.039+0000 7f84e3e00640 1 -- 172.21.15.28:0/1881843827 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84d40035f0 con 0x7f84ec103980 2024-04-23T14:45:45.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.039+0000 7f84e3e00640 1 -- 172.21.15.28:0/1881843827 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d4010670 con 0x7f84ec103980 2024-04-23T14:45:45.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.039+0000 7f84ebe00640 1 -- 172.21.15.28:0/1881843827 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84ec1a5280 con 0x7f84ec103980 2024-04-23T14:45:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.042+0000 7f84f49c5740 1 Processor -- start 2024-04-23T14:45:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.042+0000 7f84e3e00640 1 -- 172.21.15.28:0/1881843827 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f84d40037a0 con 0x7f84ec103980 2024-04-23T14:45:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.042+0000 7f84f49c5740 1 -- start start 2024-04-23T14:45:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.043+0000 7f84f49c5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d32486ea30 0x55d32486ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.043+0000 7f84f49c5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d32486f350 0x55d3248794e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.043+0000 7f84f49c5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d324879a20 0x55d32487be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.043+0000 7f84f49c5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d3247d41f0 con 0x55d324879a20 2024-04-23T14:45:45.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.043+0000 7f84f49c5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d32477a350 con 0x55d32486f350 2024-04-23T14:45:45.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.043+0000 7f84f49c5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d3247d01b0 con 0x55d32486ea30 2024-04-23T14:45:45.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.043+0000 7f84eb400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d32486ea30 0x55d32486ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.043+0000 7f84eaa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d32486f350 0x55d3248794e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.043+0000 7f84eb400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d32486ea30 0x55d32486ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:52890/0 (socket says 172.21.15.28:52890) 2024-04-23T14:45:45.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.043+0000 7f84eb400640 1 -- 172.21.15.28:0/3405369677 learned_addr learned my addr 172.21.15.28:0/3405369677 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:45.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.043+0000 7f84e2000640 1 --2- 172.21.15.28:0/3405369677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d324879a20 0x55d32487be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.044+0000 7f84eaa00640 1 -- 172.21.15.28:0/3405369677 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d32486ea30 msgr2=0x55d32486ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:45.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.044+0000 7f84eaa00640 1 --2- 172.21.15.28:0/3405369677 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d32486ea30 0x55d32486ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.044+0000 7f84eaa00640 1 -- 172.21.15.28:0/3405369677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d324879a20 msgr2=0x55d32487be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:45.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.044+0000 7f84eaa00640 1 --2- 172.21.15.28:0/3405369677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d324879a20 0x55d32487be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.044+0000 7f84eaa00640 1 -- 172.21.15.28:0/3405369677 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3247cef50 con 0x55d32486f350 2024-04-23T14:45:45.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.044+0000 7f84eaa00640 1 --2- 172.21.15.28:0/3405369677 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d32486f350 0x55d3248794e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00bbf0 tx=0x7f84dc0107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3962795bfacff5b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:45.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.044+0000 7f84e3e00640 1 --2- 172.21.15.28:0/1881843827 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f84d00a3d20 0x7f84d00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.044+0000 7f84dbe00640 1 -- 172.21.15.28:0/3405369677 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84dc004030 con 0x55d32486f350 2024-04-23T14:45:45.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.044+0000 7f84dbe00640 1 -- 172.21.15.28:0/3405369677 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84dc0041d0 con 0x55d32486f350 2024-04-23T14:45:45.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.044+0000 7f84dbe00640 1 -- 172.21.15.28:0/3405369677 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84dc0117b0 con 0x55d32486f350 2024-04-23T14:45:45.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.045+0000 7f84e3e00640 1 -- 172.21.15.28:0/1881843827 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f84d40d0aa0 con 0x7f84ec103980 2024-04-23T14:45:45.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.045+0000 7f84eb400640 1 --2- 172.21.15.28:0/1881843827 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f84d00a3d20 0x7f84d00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.045+0000 7f84f49c5740 1 -- 172.21.15.28:0/3405369677 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d32486f350 msgr2=0x55d3248794e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:45.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.045+0000 7f84f49c5740 1 --2- 172.21.15.28:0/3405369677 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d32486f350 0x55d3248794e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00bbf0 tx=0x7f84dc0107a0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.045+0000 7f84f49c5740 1 -- 172.21.15.28:0/3405369677 shutdown_connections 2024-04-23T14:45:45.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.045+0000 7f84f49c5740 1 --2- 172.21.15.28:0/3405369677 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d32486f350 0x55d3248794e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.045+0000 7f84f49c5740 1 --2- 172.21.15.28:0/3405369677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d324879a20 0x55d32487be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.045+0000 7f84f49c5740 1 --2- 172.21.15.28:0/3405369677 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d32486ea30 0x55d32486ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.045+0000 7f84f49c5740 1 -- 172.21.15.28:0/3405369677 >> 172.21.15.28:0/3405369677 conn(0x55d32486ad50 msgr2=0x55d32486b130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:45.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.045+0000 7f84eb400640 1 --2- 172.21.15.28:0/1881843827 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f84d00a3d20 0x7f84d00a61e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f84e4009a60 tx=0x7f84e4005d80 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:45.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.045+0000 7f84f49c5740 1 -- 172.21.15.28:0/3405369677 shutdown_connections 2024-04-23T14:45:45.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.046+0000 7f84f49c5740 1 -- 172.21.15.28:0/3405369677 wait complete. 2024-04-23T14:45:45.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.046+0000 7f84f49c5740 1 -- start start 2024-04-23T14:45:45.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.046+0000 7f84f49c5740 1 Processor -- start 2024-04-23T14:45:45.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.047+0000 7f84f49c5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d32486ea30 0x55d324876a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.047+0000 7f84f49c5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d32486f350 0x55d324876f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.047+0000 7f84f49c5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d324879a20 0x55d324877480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.047+0000 7f84f49c5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d32487c810 con 0x55d32486ea30 2024-04-23T14:45:45.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.047+0000 7f84f49c5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d32478d690 con 0x55d324879a20 2024-04-23T14:45:45.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.047+0000 7f84f49c5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d32487c990 con 0x55d32486f350 2024-04-23T14:45:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.047+0000 7f84f49c5740 10 client.0 _ll_register_callbacks cb 0x55d3247c9650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:45:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.047+0000 7f84eb400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d324879a20 0x55d324877480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.047+0000 7f84eaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d32486ea30 0x55d324876a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.047+0000 7f84eb400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d324879a20 0x55d324877480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:53536/0 (socket says 172.21.15.28:53536) 2024-04-23T14:45:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.047+0000 7f84eb400640 1 -- 172.21.15.28:0/3120487785 learned_addr learned my addr 172.21.15.28:0/3120487785 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.047+0000 7f84e2000640 1 --2- 172.21.15.28:0/3120487785 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d32486f350 0x55d324876f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.048+0000 7f84eaa00640 1 -- 172.21.15.28:0/3120487785 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d32486f350 msgr2=0x55d324876f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.048+0000 7f84eaa00640 1 --2- 172.21.15.28:0/3120487785 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d32486f350 0x55d324876f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.048+0000 7f84eaa00640 1 -- 172.21.15.28:0/3120487785 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d324879a20 msgr2=0x55d324877480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.048+0000 7f84eaa00640 1 --2- 172.21.15.28:0/3120487785 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d324879a20 0x55d324877480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.048+0000 7f84eaa00640 1 -- 172.21.15.28:0/3120487785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d32478c920 con 0x55d32486ea30 2024-04-23T14:45:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.048+0000 7f84eb400640 1 --2- 172.21.15.28:0/3120487785 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d324879a20 0x55d324877480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:45:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.048+0000 7f84eaa00640 1 --2- 172.21.15.28:0/3120487785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d32486ea30 0x55d324876a00 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f84dc010ca0 tx=0x7f84dc00bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0044081ce409171 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:45.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.048+0000 7f84d8c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-23T14:45:45.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.048+0000 7f84d8c00640 1 -- 172.21.15.28:0/3120487785 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84dc02a060 con 0x55d32486ea30 2024-04-23T14:45:45.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.049+0000 7f84f49c5740 1 -- 172.21.15.28:0/3120487785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84dc0041d0 con 0x55d32486ea30 2024-04-23T14:45:45.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.049+0000 7f84d8c00640 1 -- 172.21.15.28:0/3120487785 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f84dc004380 con 0x55d32486ea30 2024-04-23T14:45:45.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.049+0000 7f84d8c00640 1 -- 172.21.15.28:0/3120487785 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f84dc0035f0 con 0x55d32486ea30 2024-04-23T14:45:45.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.049+0000 7f84f49c5740 10 client.15396 fetch_fsmap learned FSMap version 20 2024-04-23T14:45:45.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.049+0000 7f84f49c5740 1 -- 172.21.15.28:0/3120487785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55d3245b07d0 con 0x55d32486ea30 2024-04-23T14:45:45.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.049+0000 7f84d8c00640 1 -- 172.21.15.28:0/3120487785 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84dc0041d0 con 0x55d32486ea30 2024-04-23T14:45:45.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.049+0000 7f84d8c00640 1 -- 172.21.15.28:0/3120487785 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f84dc004800 con 0x55d32486ea30 2024-04-23T14:45:45.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.049+0000 7f84f49c5740 10 client.15396 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:45:45.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.050+0000 7f84f49c5740 10 client.15396 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:45:45.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.050+0000 7f84f49c5740 20 client.15396 populate_metadata read hostname 'smithi028' 2024-04-23T14:45:45.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.050+0000 7f84f49c5740 1 --2- 172.21.15.28:0/3120487785 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d324889d30 0x55d32488c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.050+0000 7f84f49c5740 4 client.15396 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:45:45.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.050+0000 7f84f49c5740 1 -- 172.21.15.28:0/3120487785 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d3247e1c10 con 0x55d324889d30 2024-04-23T14:45:45.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.050+0000 7f84e2000640 1 --2- 172.21.15.28:0/3120487785 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d324889d30 0x55d32488c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.050+0000 7f84e2000640 1 --2- 172.21.15.28:0/3120487785 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d324889d30 0x55d32488c120 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3dd5e04c51c8222 server_cookie=dacaba897c11b929 in_seq=0 out_seq=0 2024-04-23T14:45:45.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.050+0000 7f84d8c00640 10 client.15396 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:45:45.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.051+0000 7f84d8c00640 1 -- 172.21.15.28:0/3120487785 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d3247e1c10 con 0x55d324889d30 2024-04-23T14:45:45.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.051+0000 7f84d8c00640 10 client.15396 handle_command_reply: tid=0 2024-04-23T14:45:45.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.051+0000 7f84f49c5740 1 client.15396 shutdown 2024-04-23T14:45:45.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.052+0000 7f84f49c5740 1 -- 172.21.15.28:0/3120487785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d32486ea30 msgr2=0x55d324876a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:45.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.052+0000 7f84f49c5740 1 --2- 172.21.15.28:0/3120487785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d32486ea30 0x55d324876a00 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f84dc010ca0 tx=0x7f84dc00bfd0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.052+0000 7f84f49c5740 1 -- 172.21.15.28:0/3120487785 shutdown_connections 2024-04-23T14:45:45.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.052+0000 7f84f49c5740 1 --2- 172.21.15.28:0/3120487785 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d324889d30 0x55d32488c120 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.052+0000 7f84f49c5740 1 --2- 172.21.15.28:0/3120487785 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d324879a20 0x55d324877480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.052+0000 7f84f49c5740 1 --2- 172.21.15.28:0/3120487785 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d32486f350 0x55d324876f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.052+0000 7f84f49c5740 1 --2- 172.21.15.28:0/3120487785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d32486ea30 0x55d324876a00 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.052+0000 7f84f49c5740 1 -- 172.21.15.28:0/3120487785 >> 172.21.15.28:0/3120487785 conn(0x55d32486ad50 msgr2=0x55d324869fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:45.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.052+0000 7f84d8c00640 0 client.15396 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:45:45.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.052+0000 7f84f49c5740 1 -- 172.21.15.28:0/3120487785 shutdown_connections 2024-04-23T14:45:45.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.052+0000 7f84f49c5740 1 -- 172.21.15.28:0/3120487785 wait complete. 2024-04-23T14:45:45.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.052+0000 7f84f49c5740 20 client.15396 trim_cache size 0 max 16384 2024-04-23T14:45:45.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.072+0000 7f84f49c5740 1 Processor -- start 2024-04-23T14:45:45.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.072+0000 7f84f49c5740 1 -- start start 2024-04-23T14:45:45.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.072+0000 7f84f49c5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d32486ea30 0x55d3248856d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.072+0000 7f84f49c5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d32486f350 0x55d32486b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.072+0000 7f84f49c5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d324879a20 0x55d324869e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.072+0000 7f84f49c5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f84ec1abbf0 con 0x55d32486ea30 2024-04-23T14:45:45.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.072+0000 7f84f49c5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f84ec1ae470 con 0x55d32486f350 2024-04-23T14:45:45.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.072+0000 7f84f49c5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f84ec1ac260 con 0x55d324879a20 2024-04-23T14:45:45.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.072+0000 7f84e2000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d32486f350 0x55d32486b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.072+0000 7f84eaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d32486ea30 0x55d3248856d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.072+0000 7f84eb400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d324879a20 0x55d324869e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.072+0000 7f84e2000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d32486f350 0x55d32486b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:53546/0 (socket says 172.21.15.28:53546) 2024-04-23T14:45:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.072+0000 7f84e2000640 1 -- 172.21.15.28:0/3986826471 learned_addr learned my addr 172.21.15.28:0/3986826471 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.073+0000 7f84e2000640 1 -- 172.21.15.28:0/3986826471 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d324879a20 msgr2=0x55d324869e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.073+0000 7f84e2000640 1 --2- 172.21.15.28:0/3986826471 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d324879a20 0x55d324869e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.073+0000 7f84e2000640 1 -- 172.21.15.28:0/3986826471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d32486ea30 msgr2=0x55d3248856d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.073+0000 7f84e2000640 1 --2- 172.21.15.28:0/3986826471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d32486ea30 0x55d3248856d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.073+0000 7f84e2000640 1 -- 172.21.15.28:0/3986826471 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3244e63f0 con 0x55d32486f350 2024-04-23T14:45:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.073+0000 7f84eaa00640 1 --2- 172.21.15.28:0/3986826471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d32486ea30 0x55d3248856d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:45:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.073+0000 7f84eb400640 1 --2- 172.21.15.28:0/3986826471 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d324879a20 0x55d324869e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:45:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.073+0000 7f84e2000640 1 --2- 172.21.15.28:0/3986826471 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d32486f350 0x55d32486b130 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f84d40993c0 tx=0x7f84d40d2ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56530cb1f3822099 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.073+0000 7f84db400640 1 -- 172.21.15.28:0/3986826471 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d409d0f0 con 0x55d32486f350 2024-04-23T14:45:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.073+0000 7f84db400640 1 -- 172.21.15.28:0/3986826471 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84d409c9f0 con 0x55d32486f350 2024-04-23T14:45:45.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.073+0000 7f84db400640 1 -- 172.21.15.28:0/3986826471 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d40ce420 con 0x55d32486f350 2024-04-23T14:45:45.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.074+0000 7f84f49c5740 1 -- 172.21.15.28:0/3986826471 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d32486f350 msgr2=0x55d32486b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:45.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.074+0000 7f84f49c5740 1 --2- 172.21.15.28:0/3986826471 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d32486f350 0x55d32486b130 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f84d40993c0 tx=0x7f84d40d2ef0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.074+0000 7f84f49c5740 1 -- 172.21.15.28:0/3986826471 shutdown_connections 2024-04-23T14:45:45.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.074+0000 7f84f49c5740 1 --2- 172.21.15.28:0/3986826471 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d32486f350 0x55d32486b130 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.074+0000 7f84f49c5740 1 --2- 172.21.15.28:0/3986826471 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d324879a20 0x55d324869e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.074+0000 7f84f49c5740 1 --2- 172.21.15.28:0/3986826471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d32486ea30 0x55d3248856d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.074+0000 7f84f49c5740 1 -- 172.21.15.28:0/3986826471 >> 172.21.15.28:0/3986826471 conn(0x55d32486ad50 msgr2=0x55d324869740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:45.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.074+0000 7f84f49c5740 1 -- 172.21.15.28:0/3986826471 shutdown_connections 2024-04-23T14:45:45.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.074+0000 7f84f49c5740 1 -- 172.21.15.28:0/3986826471 wait complete. 2024-04-23T14:45:45.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.075+0000 7f84f49c5740 1 -- start start 2024-04-23T14:45:45.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.075+0000 7f84f49c5740 1 Processor -- start 2024-04-23T14:45:45.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.075+0000 7f84f49c5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d32486ea30 0x55d324873800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.075+0000 7f84f49c5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d32486f350 0x55d324873d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.075+0000 7f84f49c5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d324879a20 0x55d324874280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.075+0000 7f84f49c5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f84ec1ac930 con 0x55d32486f350 2024-04-23T14:45:45.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.075+0000 7f84f49c5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f84ec1adda0 con 0x55d324879a20 2024-04-23T14:45:45.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.075+0000 7f84f49c5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f84ec1ad000 con 0x55d32486ea30 2024-04-23T14:45:45.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.075+0000 7f84f49c5740 10 client.0 _ll_register_callbacks cb 0x7f84e4009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:45:45.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.076+0000 7f84eb400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d324879a20 0x55d324874280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.076+0000 7f84e2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d32486f350 0x55d324873d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.076+0000 7f84eb400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d324879a20 0x55d324874280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:53558/0 (socket says 172.21.15.28:53558) 2024-04-23T14:45:45.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.076+0000 7f84eb400640 1 -- 172.21.15.28:0/141641881 learned_addr learned my addr 172.21.15.28:0/141641881 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:45.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.076+0000 7f84eaa00640 1 --2- 172.21.15.28:0/141641881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d32486ea30 0x55d324873800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.076+0000 7f84eaa00640 1 -- 172.21.15.28:0/141641881 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d324879a20 msgr2=0x55d324874280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:45.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.076+0000 7f84eaa00640 1 --2- 172.21.15.28:0/141641881 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d324879a20 0x55d324874280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.076+0000 7f84eaa00640 1 -- 172.21.15.28:0/141641881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d32486f350 msgr2=0x55d324873d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:45.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.076+0000 7f84eaa00640 1 --2- 172.21.15.28:0/141641881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d32486f350 0x55d324873d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.076+0000 7f84eaa00640 1 -- 172.21.15.28:0/141641881 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3244a60f0 con 0x55d32486ea30 2024-04-23T14:45:45.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.076+0000 7f84eaa00640 1 --2- 172.21.15.28:0/141641881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d32486ea30 0x55d324873800 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f84dc003f60 tx=0x7f84dc00b5e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e068bc8af6c517db server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:45.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.076+0000 7f84d8c00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:45:45.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.077+0000 7f84d8c00640 1 -- 172.21.15.28:0/141641881 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84dc02a040 con 0x55d32486ea30 2024-04-23T14:45:45.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.077+0000 7f84d8c00640 1 -- 172.21.15.28:0/141641881 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f84dc00b770 con 0x55d32486ea30 2024-04-23T14:45:45.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.077+0000 7f84f49c5740 1 -- 172.21.15.28:0/141641881 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84d409c9f0 con 0x55d32486ea30 2024-04-23T14:45:45.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.077+0000 7f84d8c00640 1 -- 172.21.15.28:0/141641881 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f84dc00ba40 con 0x55d32486ea30 2024-04-23T14:45:45.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.077+0000 7f84f49c5740 10 client.25549 fetch_fsmap learned FSMap version 20 2024-04-23T14:45:45.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.077+0000 7f84f49c5740 1 -- 172.21.15.28:0/141641881 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55d324799810 con 0x55d32486ea30 2024-04-23T14:45:45.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.077+0000 7f84d8c00640 1 -- 172.21.15.28:0/141641881 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84d409c9f0 con 0x55d32486ea30 2024-04-23T14:45:45.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.078+0000 7f84d8c00640 1 -- 172.21.15.28:0/141641881 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f84dc0044d0 con 0x55d32486ea30 2024-04-23T14:45:45.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.078+0000 7f84f49c5740 10 client.25549 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:45:45.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.078+0000 7f84f49c5740 10 client.25549 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:45:45.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.078+0000 7f84f49c5740 20 client.25549 populate_metadata read hostname 'smithi028' 2024-04-23T14:45:45.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.078+0000 7f84f49c5740 1 --2- 172.21.15.28:0/141641881 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d3248f0760 0x55d3248f2b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.078+0000 7f84f49c5740 4 client.25549 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:45:45.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.078+0000 7f84f49c5740 1 -- 172.21.15.28:0/141641881 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d3247dba80 con 0x55d3248f0760 2024-04-23T14:45:45.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.078+0000 7f84e2000640 1 --2- 172.21.15.28:0/141641881 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d3248f0760 0x55d3248f2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.079+0000 7f84e2000640 1 --2- 172.21.15.28:0/141641881 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d3248f0760 0x55d3248f2b80 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ea110da862fb3d9 server_cookie=9c5f61fc02b8b2c9 in_seq=0 out_seq=0 2024-04-23T14:45:45.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.079+0000 7f84d8c00640 10 client.25549 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:45:45.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.079+0000 7f84d8c00640 1 -- 172.21.15.28:0/141641881 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d3247dba80 con 0x55d3248f0760 2024-04-23T14:45:45.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.079+0000 7f84d8c00640 10 client.25549 handle_command_reply: tid=0 2024-04-23T14:45:45.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.079+0000 7f84f49c5740 1 client.25549 shutdown 2024-04-23T14:45:45.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.079+0000 7f84f49c5740 1 -- 172.21.15.28:0/141641881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d32486ea30 msgr2=0x55d324873800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:45.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.079+0000 7f84f49c5740 1 --2- 172.21.15.28:0/141641881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d32486ea30 0x55d324873800 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f84dc003f60 tx=0x7f84dc00b5e0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.079+0000 7f84f49c5740 1 -- 172.21.15.28:0/141641881 shutdown_connections 2024-04-23T14:45:45.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.079+0000 7f84f49c5740 1 --2- 172.21.15.28:0/141641881 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d3248f0760 0x55d3248f2b80 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.079+0000 7f84f49c5740 1 --2- 172.21.15.28:0/141641881 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d324879a20 0x55d324874280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.079+0000 7f84f49c5740 1 --2- 172.21.15.28:0/141641881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d32486f350 0x55d324873d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.079+0000 7f84f49c5740 1 --2- 172.21.15.28:0/141641881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d32486ea30 0x55d324873800 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.079+0000 7f84f49c5740 1 -- 172.21.15.28:0/141641881 >> 172.21.15.28:0/141641881 conn(0x55d32486ad50 msgr2=0x55d324870ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:45.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.079+0000 7f84d8c00640 0 client.25549 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:45:45.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.079+0000 7f84f49c5740 1 -- 172.21.15.28:0/141641881 shutdown_connections 2024-04-23T14:45:45.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.079+0000 7f84f49c5740 1 -- 172.21.15.28:0/141641881 wait complete. 2024-04-23T14:45:45.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.079+0000 7f84f49c5740 20 client.25549 trim_cache size 0 max 16384 2024-04-23T14:45:45.084 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:45:45.084 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-23T14:45:45.084 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-23T14:45:45.084 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:45:45.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.080+0000 7f84ebe00640 1 -- 172.21.15.28:0/1881843827 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f84d00a3d20 msgr2=0x7f84d00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:45.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.080+0000 7f84ebe00640 1 --2- 172.21.15.28:0/1881843827 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f84d00a3d20 0x7f84d00a61e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f84e4009a60 tx=0x7f84e4005d80 comp rx=0 tx=0).stop 2024-04-23T14:45:45.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.080+0000 7f84ebe00640 1 -- 172.21.15.28:0/1881843827 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84ec103980 msgr2=0x7f84ec198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:45.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.080+0000 7f84ebe00640 1 --2- 172.21.15.28:0/1881843827 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84ec103980 0x7f84ec198c50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f84d400e9f0 tx=0x7f84d400eeb0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.080+0000 7f84ebe00640 1 -- 172.21.15.28:0/1881843827 shutdown_connections 2024-04-23T14:45:45.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.080+0000 7f84ebe00640 1 --2- 172.21.15.28:0/1881843827 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f84d00a3d20 0x7f84d00a61e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.080+0000 7f84ebe00640 1 --2- 172.21.15.28:0/1881843827 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84ec103980 0x7f84ec198c50 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.080+0000 7f84ebe00640 1 --2- 172.21.15.28:0/1881843827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84ec10c480 0x7f84ec19cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.080+0000 7f84ebe00640 1 --2- 172.21.15.28:0/1881843827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84ec103060 0x7f84ec198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.080+0000 7f84ebe00640 1 -- 172.21.15.28:0/1881843827 >> 172.21.15.28:0/1881843827 conn(0x7f84ec0fc880 msgr2=0x7f84ec0fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:45.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.081+0000 7f84ebe00640 1 -- 172.21.15.28:0/1881843827 shutdown_connections 2024-04-23T14:45:45.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.081+0000 7f84ebe00640 1 -- 172.21.15.28:0/1881843827 wait complete. 2024-04-23T14:45:45.098 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:44 smithi028 ceph-mon[27810]: pgmap v54: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 341 B/s wr, 2 op/s 2024-04-23T14:45:45.101 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-23T14:45:45.101 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:45:45.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:44 smithi038 ceph-mon[32348]: pgmap v54: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 341 B/s wr, 2 op/s 2024-04-23T14:45:45.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.312+0000 7f22afe00640 1 -- 172.21.15.28:0/2109808391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22b0069400 msgr2=0x7f22b00697e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:45.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.312+0000 7f22afe00640 1 --2- 172.21.15.28:0/2109808391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22b0069400 0x7f22b00697e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f22a0009ca0 tx=0x7f22a002a8c0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.312+0000 7f22afe00640 1 -- 172.21.15.28:0/2109808391 shutdown_connections 2024-04-23T14:45:45.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.312+0000 7f22afe00640 1 --2- 172.21.15.28:0/2109808391 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22b01071a0 0x7f22b010ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.312+0000 7f22afe00640 1 --2- 172.21.15.28:0/2109808391 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f22b0069db0 0x7f22b0106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.312+0000 7f22afe00640 1 --2- 172.21.15.28:0/2109808391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22b0069400 0x7f22b00697e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.312+0000 7f22afe00640 1 -- 172.21.15.28:0/2109808391 >> 172.21.15.28:0/2109808391 conn(0x7f22b0076840 msgr2=0x7f22b0076c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:45.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.312+0000 7f22afe00640 1 -- 172.21.15.28:0/2109808391 shutdown_connections 2024-04-23T14:45:45.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.312+0000 7f22afe00640 1 -- 172.21.15.28:0/2109808391 wait complete. 2024-04-23T14:45:45.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.313+0000 7f22afe00640 1 Processor -- start 2024-04-23T14:45:45.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.313+0000 7f22afe00640 1 -- start start 2024-04-23T14:45:45.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.314+0000 7f22afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22b0069400 0x7f22b01a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.314+0000 7f22afe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f22b01071a0 0x7f22b01a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.314+0000 7f22afe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22b019a740 0x7f22b019abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.314+0000 7f22afe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f22b0112200 con 0x7f22b0069400 2024-04-23T14:45:45.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.314+0000 7f22afe00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f22b0112080 con 0x7f22b019a740 2024-04-23T14:45:45.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.314+0000 7f22afe00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f22b0112380 con 0x7f22b01071a0 2024-04-23T14:45:45.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.314+0000 7f22af400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22b019a740 0x7f22b019abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.314+0000 7f22af400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22b019a740 0x7f22b019abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:53588/0 (socket says 172.21.15.28:53588) 2024-04-23T14:45:45.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.314+0000 7f22af400640 1 -- 172.21.15.28:0/1588453066 learned_addr learned my addr 172.21.15.28:0/1588453066 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:45.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.314+0000 7f22aea00640 1 --2- 172.21.15.28:0/1588453066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22b0069400 0x7f22b01a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.314+0000 7f22a6000640 1 --2- 172.21.15.28:0/1588453066 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f22b01071a0 0x7f22b01a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.314+0000 7f22af400640 1 -- 172.21.15.28:0/1588453066 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f22b01071a0 msgr2=0x7f22b01a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:45.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.314+0000 7f22af400640 1 --2- 172.21.15.28:0/1588453066 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f22b01071a0 0x7f22b01a0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.314+0000 7f22af400640 1 -- 172.21.15.28:0/1588453066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22b0069400 msgr2=0x7f22b01a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:45.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.314+0000 7f22af400640 1 --2- 172.21.15.28:0/1588453066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22b0069400 0x7f22b01a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.314+0000 7f22af400640 1 -- 172.21.15.28:0/1588453066 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22b019b340 con 0x7f22b019a740 2024-04-23T14:45:45.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.315+0000 7f22af400640 1 --2- 172.21.15.28:0/1588453066 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22b019a740 0x7f22b019abc0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f22a800e9b0 tx=0x7f22a800ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:45.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.315+0000 7f22b4a00640 1 -- 172.21.15.28:0/1588453066 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a800cd80 con 0x7f22b019a740 2024-04-23T14:45:45.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.315+0000 7f22b4a00640 1 -- 172.21.15.28:0/1588453066 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22a80035f0 con 0x7f22b019a740 2024-04-23T14:45:45.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.315+0000 7f22b4a00640 1 -- 172.21.15.28:0/1588453066 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a8010670 con 0x7f22b019a740 2024-04-23T14:45:45.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.315+0000 7f22afe00640 1 -- 172.21.15.28:0/1588453066 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22b01a7420 con 0x7f22b019a740 2024-04-23T14:45:45.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.315+0000 7f22afe00640 1 -- 172.21.15.28:0/1588453066 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22b01a77d0 con 0x7f22b019a740 2024-04-23T14:45:45.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.316+0000 7f22afe00640 1 -- 172.21.15.28:0/1588453066 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22b0103550 con 0x7f22b019a740 2024-04-23T14:45:45.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.318+0000 7f22b4a00640 1 -- 172.21.15.28:0/1588453066 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f22a80037a0 con 0x7f22b019a740 2024-04-23T14:45:45.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.320+0000 7f22b4a00640 1 --2- 172.21.15.28:0/1588453066 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f22940a3da0 0x7f22940a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.320+0000 7f22b4a00640 1 -- 172.21.15.28:0/1588453066 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f22a80d0a30 con 0x7f22b019a740 2024-04-23T14:45:45.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.320+0000 7f22aea00640 1 --2- 172.21.15.28:0/1588453066 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f22940a3da0 0x7f22940a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.321+0000 7f22aea00640 1 --2- 172.21.15.28:0/1588453066 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f22940a3da0 0x7f22940a6260 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f22a0006010 tx=0x7f22a0009360 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:45.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.326+0000 7f22b4a00640 1 -- 172.21.15.28:0/1588453066 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f22a809d080 con 0x7f22b019a740 2024-04-23T14:45:45.458 INFO:tasks.workunit.client.0.smithi028.stdout: 1 2368 27.89 MB/sec warmup 1 sec latency 12.733 ms 2024-04-23T14:45:45.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.586+0000 7f22afe00640 1 -- 172.21.15.28:0/1588453066 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f22b0069e80 con 0x7f22b019a740 2024-04-23T14:45:45.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.587+0000 7f22b4a00640 1 -- 172.21.15.28:0/1588453066 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7f22a809c980 con 0x7f22b019a740 2024-04-23T14:45:45.589 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:45:45.589 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:45:45.589 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:45:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.590+0000 7f22afe00640 1 -- 172.21.15.28:0/1588453066 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f22940a3da0 msgr2=0x7f22940a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.590+0000 7f22afe00640 1 --2- 172.21.15.28:0/1588453066 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f22940a3da0 0x7f22940a6260 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f22a0006010 tx=0x7f22a0009360 comp rx=0 tx=0).stop 2024-04-23T14:45:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.590+0000 7f22afe00640 1 -- 172.21.15.28:0/1588453066 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22b019a740 msgr2=0x7f22b019abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.590+0000 7f22afe00640 1 --2- 172.21.15.28:0/1588453066 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22b019a740 0x7f22b019abc0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f22a800e9b0 tx=0x7f22a800ee70 comp rx=0 tx=0).stop 2024-04-23T14:45:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.591+0000 7f22afe00640 1 -- 172.21.15.28:0/1588453066 shutdown_connections 2024-04-23T14:45:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.591+0000 7f22afe00640 1 --2- 172.21.15.28:0/1588453066 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f22940a3da0 0x7f22940a6260 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.591+0000 7f22afe00640 1 --2- 172.21.15.28:0/1588453066 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22b019a740 0x7f22b019abc0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.591+0000 7f22afe00640 1 --2- 172.21.15.28:0/1588453066 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f22b01071a0 0x7f22b01a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.591+0000 7f22afe00640 1 --2- 172.21.15.28:0/1588453066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22b0069400 0x7f22b01a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.591+0000 7f22afe00640 1 -- 172.21.15.28:0/1588453066 >> 172.21.15.28:0/1588453066 conn(0x7f22b0076840 msgr2=0x7f22b00ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:45.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.591+0000 7f22afe00640 1 -- 172.21.15.28:0/1588453066 shutdown_connections 2024-04-23T14:45:45.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.591+0000 7f22afe00640 1 -- 172.21.15.28:0/1588453066 wait complete. 2024-04-23T14:45:45.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-23T14:45:45.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.834+0000 7f447be00640 1 -- 172.21.15.28:0/1938708673 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f447c100c40 msgr2=0x7f447c1010c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:45.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.834+0000 7f447be00640 1 --2- 172.21.15.28:0/1938708673 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f447c100c40 0x7f447c1010c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4468009930 tx=0x7f446802a440 comp rx=0 tx=0).stop 2024-04-23T14:45:45.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.834+0000 7f447be00640 1 -- 172.21.15.28:0/1938708673 shutdown_connections 2024-04-23T14:45:45.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.834+0000 7f447be00640 1 --2- 172.21.15.28:0/1938708673 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f447c100c40 0x7f447c1010c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.834+0000 7f447be00640 1 --2- 172.21.15.28:0/1938708673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f447c101600 0x7f447c10d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.834+0000 7f447be00640 1 --2- 172.21.15.28:0/1938708673 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f447c106630 0x7f447c106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.834+0000 7f447be00640 1 -- 172.21.15.28:0/1938708673 >> 172.21.15.28:0/1938708673 conn(0x7f447c075d00 msgr2=0x7f447c076110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:45.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.834+0000 7f447be00640 1 -- 172.21.15.28:0/1938708673 shutdown_connections 2024-04-23T14:45:45.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.834+0000 7f447be00640 1 -- 172.21.15.28:0/1938708673 wait complete. 2024-04-23T14:45:45.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.835+0000 7f447be00640 1 Processor -- start 2024-04-23T14:45:45.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.836+0000 7f447be00640 1 -- start start 2024-04-23T14:45:45.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.836+0000 7f447be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f447c100c40 0x7f447c198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.836+0000 7f447be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f447c101600 0x7f447c198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.836+0000 7f447be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f447c106630 0x7f447c19cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.836+0000 7f447be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f447c110080 con 0x7f447c100c40 2024-04-23T14:45:45.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.836+0000 7f447be00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f447c10ff00 con 0x7f447c101600 2024-04-23T14:45:45.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.836+0000 7f447be00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f447c110200 con 0x7f447c106630 2024-04-23T14:45:45.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.836+0000 7f447aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f447c100c40 0x7f447c198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.836+0000 7f447aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f447c100c40 0x7f447c198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36148/0 (socket says 172.21.15.28:36148) 2024-04-23T14:45:45.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.836+0000 7f447aa00640 1 -- 172.21.15.28:0/4184193852 learned_addr learned my addr 172.21.15.28:0/4184193852 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:45.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.836+0000 7f447a000640 1 --2- 172.21.15.28:0/4184193852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f447c101600 0x7f447c198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.836+0000 7f447b400640 1 --2- 172.21.15.28:0/4184193852 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f447c106630 0x7f447c19cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.837+0000 7f447a000640 1 -- 172.21.15.28:0/4184193852 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f447c106630 msgr2=0x7f447c19cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:45.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.837+0000 7f447a000640 1 --2- 172.21.15.28:0/4184193852 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f447c106630 0x7f447c19cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.837+0000 7f447a000640 1 -- 172.21.15.28:0/4184193852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f447c100c40 msgr2=0x7f447c198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:45.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.837+0000 7f447a000640 1 --2- 172.21.15.28:0/4184193852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f447c100c40 0x7f447c198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.837+0000 7f447a000640 1 -- 172.21.15.28:0/4184193852 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f447c19d700 con 0x7f447c101600 2024-04-23T14:45:45.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.837+0000 7f447aa00640 1 --2- 172.21.15.28:0/4184193852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f447c100c40 0x7f447c198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:45:45.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.837+0000 7f447b400640 1 --2- 172.21.15.28:0/4184193852 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f447c106630 0x7f447c19cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:45:45.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.837+0000 7f447a000640 1 --2- 172.21.15.28:0/4184193852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f447c101600 0x7f447c198c50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f446802a960 tx=0x7f4468003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:45.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.837+0000 7f4473400640 1 -- 172.21.15.28:0/4184193852 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f446803f070 con 0x7f447c101600 2024-04-23T14:45:45.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.837+0000 7f4473400640 1 -- 172.21.15.28:0/4184193852 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4468031c90 con 0x7f447c101600 2024-04-23T14:45:45.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.838+0000 7f447be00640 1 -- 172.21.15.28:0/4184193852 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f447c19d990 con 0x7f447c101600 2024-04-23T14:45:45.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.838+0000 7f4473400640 1 -- 172.21.15.28:0/4184193852 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4468035030 con 0x7f447c101600 2024-04-23T14:45:45.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.838+0000 7f447be00640 1 -- 172.21.15.28:0/4184193852 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f447c1a5280 con 0x7f447c101600 2024-04-23T14:45:45.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.841+0000 7f4473400640 1 -- 172.21.15.28:0/4184193852 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f4468042070 con 0x7f447c101600 2024-04-23T14:45:45.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.841+0000 7f448498e740 1 Processor -- start 2024-04-23T14:45:45.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.841+0000 7f448498e740 1 -- start start 2024-04-23T14:45:45.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.842+0000 7f448498e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568fcf43b40 0x5568fcf43f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.842+0000 7f448498e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5568fcf44460 0x5568fcf4e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.842+0000 7f448498e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568fcf4eb30 0x5568fcf50f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.842+0000 7f448498e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5568fcea91d0 con 0x5568fcf4eb30 2024-04-23T14:45:45.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.842+0000 7f448498e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5568fce4f450 con 0x5568fcf44460 2024-04-23T14:45:45.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.842+0000 7f448498e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5568fcea5190 con 0x5568fcf43b40 2024-04-23T14:45:45.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.842+0000 7f447b400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568fcf43b40 0x5568fcf43f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.842+0000 7f447aa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5568fcf44460 0x5568fcf4e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.842+0000 7f447b400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568fcf43b40 0x5568fcf43f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:52976/0 (socket says 172.21.15.28:52976) 2024-04-23T14:45:45.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.842+0000 7f447aa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5568fcf44460 0x5568fcf4e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:53614/0 (socket says 172.21.15.28:53614) 2024-04-23T14:45:45.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.842+0000 7f447b400640 1 -- 172.21.15.28:0/668425655 learned_addr learned my addr 172.21.15.28:0/668425655 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:45.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.842+0000 7f447a000640 1 --2- 172.21.15.28:0/668425655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568fcf4eb30 0x5568fcf50f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.842+0000 7f447b400640 1 -- 172.21.15.28:0/668425655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5568fcf44460 msgr2=0x5568fcf4e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:45.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.842+0000 7f447b400640 1 --2- 172.21.15.28:0/668425655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5568fcf44460 0x5568fcf4e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.842+0000 7f447b400640 1 -- 172.21.15.28:0/668425655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568fcf4eb30 msgr2=0x5568fcf50f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:45.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.842+0000 7f447b400640 1 --2- 172.21.15.28:0/668425655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568fcf4eb30 0x5568fcf50f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.842+0000 7f447b400640 1 -- 172.21.15.28:0/668425655 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568fcf0c310 con 0x5568fcf43b40 2024-04-23T14:45:45.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.842+0000 7f447aa00640 1 --2- 172.21.15.28:0/668425655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5568fcf44460 0x5568fcf4e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:45:45.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.843+0000 7f447b400640 1 --2- 172.21.15.28:0/668425655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568fcf43b40 0x5568fcf43f20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f44740049d0 tx=0x7f447400ec20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adc1d8d96c02bacb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:45.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.843+0000 7f4473400640 1 --2- 172.21.15.28:0/4184193852 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f44600a3da0 0x7f44600a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.843+0000 7f4467e00640 1 -- 172.21.15.28:0/668425655 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f447401c070 con 0x5568fcf43b40 2024-04-23T14:45:45.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.843+0000 7f4467e00640 1 -- 172.21.15.28:0/668425655 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f447400dcf0 con 0x5568fcf43b40 2024-04-23T14:45:45.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.843+0000 7f4467e00640 1 -- 172.21.15.28:0/668425655 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f447400c720 con 0x5568fcf43b40 2024-04-23T14:45:45.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.843+0000 7f4473400640 1 -- 172.21.15.28:0/4184193852 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f44680f0e20 con 0x7f447c101600 2024-04-23T14:45:45.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.843+0000 7f447aa00640 1 --2- 172.21.15.28:0/4184193852 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f44600a3da0 0x7f44600a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.844+0000 7f448498e740 1 -- 172.21.15.28:0/668425655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568fcf43b40 msgr2=0x5568fcf43f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:45.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.844+0000 7f448498e740 1 --2- 172.21.15.28:0/668425655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568fcf43b40 0x5568fcf43f20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f44740049d0 tx=0x7f447400ec20 comp rx=0 tx=0).stop 2024-04-23T14:45:45.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.844+0000 7f447aa00640 1 --2- 172.21.15.28:0/4184193852 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f44600a3da0 0x7f44600a6260 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f446c00afd0 tx=0x7f446c00a380 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:45.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.844+0000 7f448498e740 1 -- 172.21.15.28:0/668425655 shutdown_connections 2024-04-23T14:45:45.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.844+0000 7f448498e740 1 --2- 172.21.15.28:0/668425655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5568fcf44460 0x5568fcf4e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.844+0000 7f448498e740 1 --2- 172.21.15.28:0/668425655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568fcf4eb30 0x5568fcf50f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.844+0000 7f448498e740 1 --2- 172.21.15.28:0/668425655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568fcf43b40 0x5568fcf43f20 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.844+0000 7f448498e740 1 -- 172.21.15.28:0/668425655 >> 172.21.15.28:0/668425655 conn(0x5568fcf3fe60 msgr2=0x5568fcf40240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:45.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.844+0000 7f448498e740 1 -- 172.21.15.28:0/668425655 shutdown_connections 2024-04-23T14:45:45.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.844+0000 7f448498e740 1 -- 172.21.15.28:0/668425655 wait complete. 2024-04-23T14:45:45.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.845+0000 7f448498e740 1 -- start start 2024-04-23T14:45:45.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.845+0000 7f448498e740 1 Processor -- start 2024-04-23T14:45:45.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.845+0000 7f448498e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568fcf43b40 0x5568fcf58bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.845+0000 7f448498e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568fcf44460 0x5568fcf59110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.845+0000 7f448498e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5568fcf4eb30 0x5568fcf52f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.845+0000 7f448498e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5568fcf519c0 con 0x5568fcf43b40 2024-04-23T14:45:45.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.846+0000 7f448498e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5568fce62810 con 0x5568fcf4eb30 2024-04-23T14:45:45.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.846+0000 7f448498e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5568fcf51b40 con 0x5568fcf44460 2024-04-23T14:45:45.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.846+0000 7f448498e740 10 client.0 _ll_register_callbacks cb 0x5568fce30140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:45:45.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.846+0000 7f447aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568fcf43b40 0x5568fcf58bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.846+0000 7f447b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5568fcf4eb30 0x5568fcf52f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.846+0000 7f447b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5568fcf4eb30 0x5568fcf52f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:53618/0 (socket says 172.21.15.28:53618) 2024-04-23T14:45:45.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.846+0000 7f447a000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568fcf44460 0x5568fcf59110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.846+0000 7f447b400640 1 -- 172.21.15.28:0/3455844983 learned_addr learned my addr 172.21.15.28:0/3455844983 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:45.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.846+0000 7f447b400640 1 -- 172.21.15.28:0/3455844983 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568fcf44460 msgr2=0x5568fcf59110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:45.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.846+0000 7f447b400640 1 --2- 172.21.15.28:0/3455844983 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568fcf44460 0x5568fcf59110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.846+0000 7f447b400640 1 -- 172.21.15.28:0/3455844983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568fcf43b40 msgr2=0x5568fcf58bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:45.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.846+0000 7f447b400640 1 --2- 172.21.15.28:0/3455844983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568fcf43b40 0x5568fcf58bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.846+0000 7f447b400640 1 -- 172.21.15.28:0/3455844983 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568fce733f0 con 0x5568fcf4eb30 2024-04-23T14:45:45.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.847+0000 7f447b400640 1 --2- 172.21.15.28:0/3455844983 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5568fcf4eb30 0x5568fcf52f40 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f447400d810 tx=0x7f44740176c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=540d9d5643741b52 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:45.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.847+0000 7f4464c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:45:45.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.847+0000 7f4464c00640 1 -- 172.21.15.28:0/3455844983 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4474024020 con 0x5568fcf4eb30 2024-04-23T14:45:45.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.847+0000 7f4464c00640 1 -- 172.21.15.28:0/3455844983 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f4474017830 con 0x5568fcf4eb30 2024-04-23T14:45:45.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.847+0000 7f448498e740 1 -- 172.21.15.28:0/3455844983 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f447400dcf0 con 0x5568fcf4eb30 2024-04-23T14:45:45.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.848+0000 7f4464c00640 1 -- 172.21.15.28:0/3455844983 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f447402f050 con 0x5568fcf4eb30 2024-04-23T14:45:45.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.848+0000 7f448498e740 10 client.25559 fetch_fsmap learned FSMap version 20 2024-04-23T14:45:45.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.848+0000 7f448498e740 1 -- 172.21.15.28:0/3455844983 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5568fcbbb3f0 con 0x5568fcf4eb30 2024-04-23T14:45:45.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.848+0000 7f4464c00640 1 -- 172.21.15.28:0/3455844983 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f447400dcf0 con 0x5568fcf4eb30 2024-04-23T14:45:45.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.848+0000 7f4464c00640 1 -- 172.21.15.28:0/3455844983 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f447401f030 con 0x5568fcf4eb30 2024-04-23T14:45:45.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.848+0000 7f448498e740 10 client.25559 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:45:45.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.848+0000 7f448498e740 10 client.25559 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:45:45.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.848+0000 7f448498e740 20 client.25559 populate_metadata read hostname 'smithi028' 2024-04-23T14:45:45.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.848+0000 7f448498e740 1 --2- 172.21.15.28:0/3455844983 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5568fcf630e0 0x5568fcf654d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.848+0000 7f448498e740 4 client.25559 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:45:45.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.848+0000 7f448498e740 1 -- 172.21.15.28:0/3455844983 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5568fceb6bf0 con 0x5568fcf630e0 2024-04-23T14:45:45.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.849+0000 7f447aa00640 1 --2- 172.21.15.28:0/3455844983 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5568fcf630e0 0x5568fcf654d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.849+0000 7f447aa00640 1 --2- 172.21.15.28:0/3455844983 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5568fcf630e0 0x5568fcf654d0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da248806402e15a1 server_cookie=367e2899cf3f45b6 in_seq=0 out_seq=0 2024-04-23T14:45:45.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.849+0000 7f4464c00640 10 client.25559 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:45:45.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.851+0000 7f4464c00640 1 -- 172.21.15.28:0/3455844983 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5568fceb6bf0 con 0x5568fcf630e0 2024-04-23T14:45:45.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.851+0000 7f4464c00640 10 client.25559 handle_command_reply: tid=0 2024-04-23T14:45:45.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.851+0000 7f448498e740 1 client.25559 shutdown 2024-04-23T14:45:45.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.851+0000 7f448498e740 1 -- 172.21.15.28:0/3455844983 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5568fcf4eb30 msgr2=0x5568fcf52f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:45.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.851+0000 7f448498e740 1 --2- 172.21.15.28:0/3455844983 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5568fcf4eb30 0x5568fcf52f40 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f447400d810 tx=0x7f44740176c0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.851+0000 7f448498e740 1 -- 172.21.15.28:0/3455844983 shutdown_connections 2024-04-23T14:45:45.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.851+0000 7f448498e740 1 --2- 172.21.15.28:0/3455844983 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5568fcf630e0 0x5568fcf654d0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.851+0000 7f4464c00640 0 client.25559 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:45:45.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.851+0000 7f448498e740 1 --2- 172.21.15.28:0/3455844983 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5568fcf4eb30 0x5568fcf52f40 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.851+0000 7f448498e740 1 --2- 172.21.15.28:0/3455844983 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568fcf44460 0x5568fcf59110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.851+0000 7f448498e740 1 --2- 172.21.15.28:0/3455844983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568fcf43b40 0x5568fcf58bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.851+0000 7f448498e740 1 -- 172.21.15.28:0/3455844983 >> 172.21.15.28:0/3455844983 conn(0x5568fcf3fe60 msgr2=0x5568fcf45c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:45.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.851+0000 7f448498e740 1 -- 172.21.15.28:0/3455844983 shutdown_connections 2024-04-23T14:45:45.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.851+0000 7f448498e740 1 -- 172.21.15.28:0/3455844983 wait complete. 2024-04-23T14:45:45.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.851+0000 7f448498e740 20 client.25559 trim_cache size 0 max 16384 2024-04-23T14:45:45.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.866+0000 7f448498e740 1 Processor -- start 2024-04-23T14:45:45.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.866+0000 7f448498e740 1 -- start start 2024-04-23T14:45:45.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.866+0000 7f448498e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568fcf43b40 0x5568fcf58ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.867+0000 7f448498e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5568fcf44460 0x5568fcf6ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.867+0000 7f448498e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568fcf4eb30 0x5568fcf6b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.867+0000 7f448498e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f447c1a9910 con 0x5568fcf43b40 2024-04-23T14:45:45.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.867+0000 7f448498e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f447c1ac170 con 0x5568fcf44460 2024-04-23T14:45:45.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.867+0000 7f448498e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f447c1a9f60 con 0x5568fcf4eb30 2024-04-23T14:45:45.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.867+0000 7f447b400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568fcf4eb30 0x5568fcf6b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.867+0000 7f447a000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5568fcf44460 0x5568fcf6ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.867+0000 7f447b400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568fcf4eb30 0x5568fcf6b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:52992/0 (socket says 172.21.15.28:52992) 2024-04-23T14:45:45.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.867+0000 7f447b400640 1 -- 172.21.15.28:0/1676880834 learned_addr learned my addr 172.21.15.28:0/1676880834 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:45.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.867+0000 7f447aa00640 1 --2- 172.21.15.28:0/1676880834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568fcf43b40 0x5568fcf58ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.867+0000 7f447b400640 1 -- 172.21.15.28:0/1676880834 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5568fcf44460 msgr2=0x5568fcf6ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:45.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.867+0000 7f447b400640 1 --2- 172.21.15.28:0/1676880834 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5568fcf44460 0x5568fcf6ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.867+0000 7f447b400640 1 -- 172.21.15.28:0/1676880834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568fcf43b40 msgr2=0x5568fcf58ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:45.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.867+0000 7f447b400640 1 --2- 172.21.15.28:0/1676880834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568fcf43b40 0x5568fcf58ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.867+0000 7f447b400640 1 -- 172.21.15.28:0/1676880834 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568fcc857d0 con 0x5568fcf4eb30 2024-04-23T14:45:45.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.867+0000 7f447aa00640 1 --2- 172.21.15.28:0/1676880834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568fcf43b40 0x5568fcf58ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:45:45.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.868+0000 7f447b400640 1 --2- 172.21.15.28:0/1676880834 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568fcf4eb30 0x5568fcf6b040 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4474017f50 tx=0x7f4474004980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=944446ed4c0a6cc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:45.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.868+0000 7f4467400640 1 -- 172.21.15.28:0/1676880834 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4474038070 con 0x5568fcf4eb30 2024-04-23T14:45:45.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.868+0000 7f4467400640 1 -- 172.21.15.28:0/1676880834 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4474005970 con 0x5568fcf4eb30 2024-04-23T14:45:45.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.868+0000 7f4467400640 1 -- 172.21.15.28:0/1676880834 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4474024040 con 0x5568fcf4eb30 2024-04-23T14:45:45.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.869+0000 7f448498e740 1 -- 172.21.15.28:0/1676880834 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568fcf4eb30 msgr2=0x5568fcf6b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:45.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.869+0000 7f448498e740 1 --2- 172.21.15.28:0/1676880834 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568fcf4eb30 0x5568fcf6b040 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4474017f50 tx=0x7f4474004980 comp rx=0 tx=0).stop 2024-04-23T14:45:45.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.869+0000 7f448498e740 1 -- 172.21.15.28:0/1676880834 shutdown_connections 2024-04-23T14:45:45.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.869+0000 7f448498e740 1 --2- 172.21.15.28:0/1676880834 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5568fcf44460 0x5568fcf6ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.869+0000 7f448498e740 1 --2- 172.21.15.28:0/1676880834 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568fcf4eb30 0x5568fcf6b040 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.869+0000 7f448498e740 1 --2- 172.21.15.28:0/1676880834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568fcf43b40 0x5568fcf58ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.869+0000 7f448498e740 1 -- 172.21.15.28:0/1676880834 >> 172.21.15.28:0/1676880834 conn(0x5568fcf3fe60 msgr2=0x5568fcf58790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.869+0000 7f448498e740 1 -- 172.21.15.28:0/1676880834 shutdown_connections 2024-04-23T14:45:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.869+0000 7f448498e740 1 -- 172.21.15.28:0/1676880834 wait complete. 2024-04-23T14:45:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.869+0000 7f448498e740 1 -- start start 2024-04-23T14:45:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.869+0000 7f448498e740 1 Processor -- start 2024-04-23T14:45:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.870+0000 7f448498e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5568fcf43b40 0x5568fcf5f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.870+0000 7f448498e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568fcf44460 0x5568fcf4ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.870+0000 7f448498e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568fcf4eb30 0x5568fcf4d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.870+0000 7f448498e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f447c1aa630 con 0x5568fcf44460 2024-04-23T14:45:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.870+0000 7f448498e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f447c1abaa0 con 0x5568fcf43b40 2024-04-23T14:45:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.870+0000 7f448498e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f447c1aad00 con 0x5568fcf4eb30 2024-04-23T14:45:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.870+0000 7f448498e740 10 client.0 _ll_register_callbacks cb 0x7f447401a070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:45:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.870+0000 7f447aa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5568fcf43b40 0x5568fcf5f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.870+0000 7f447aa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5568fcf43b40 0x5568fcf5f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:53630/0 (socket says 172.21.15.28:53630) 2024-04-23T14:45:45.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.870+0000 7f447aa00640 1 -- 172.21.15.28:0/2746140938 learned_addr learned my addr 172.21.15.28:0/2746140938 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:45.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.870+0000 7f447a000640 1 --2- 172.21.15.28:0/2746140938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568fcf44460 0x5568fcf4ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.870+0000 7f447b400640 1 --2- 172.21.15.28:0/2746140938 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568fcf4eb30 0x5568fcf4d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.870+0000 7f447aa00640 1 -- 172.21.15.28:0/2746140938 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568fcf4eb30 msgr2=0x5568fcf4d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:45.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.870+0000 7f447aa00640 1 --2- 172.21.15.28:0/2746140938 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568fcf4eb30 0x5568fcf4d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.871+0000 7f447aa00640 1 -- 172.21.15.28:0/2746140938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568fcf44460 msgr2=0x5568fcf4ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:45.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.871+0000 7f447aa00640 1 --2- 172.21.15.28:0/2746140938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568fcf44460 0x5568fcf4ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.871+0000 7f447aa00640 1 -- 172.21.15.28:0/2746140938 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568fcf4d8b0 con 0x5568fcf43b40 2024-04-23T14:45:45.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.871+0000 7f447b400640 1 --2- 172.21.15.28:0/2746140938 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568fcf4eb30 0x5568fcf4d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:45:45.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.871+0000 7f447aa00640 1 --2- 172.21.15.28:0/2746140938 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5568fcf43b40 0x5568fcf5f420 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f446c010520 tx=0x7f446c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea6cc048e9d8b2b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:45.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.871+0000 7f4464c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:45:45.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.871+0000 7f4464c00640 1 -- 172.21.15.28:0/2746140938 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f446c029340 con 0x5568fcf43b40 2024-04-23T14:45:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.871+0000 7f4464c00640 1 -- 172.21.15.28:0/2746140938 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f446c016030 con 0x5568fcf43b40 2024-04-23T14:45:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.871+0000 7f448498e740 1 -- 172.21.15.28:0/2746140938 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4474005970 con 0x5568fcf43b40 2024-04-23T14:45:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.871+0000 7f4464c00640 1 -- 172.21.15.28:0/2746140938 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f446c038050 con 0x5568fcf43b40 2024-04-23T14:45:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.871+0000 7f448498e740 10 client.25565 fetch_fsmap learned FSMap version 20 2024-04-23T14:45:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.871+0000 7f448498e740 1 -- 172.21.15.28:0/2746140938 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5568fcfc9210 con 0x5568fcf43b40 2024-04-23T14:45:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.872+0000 7f4464c00640 1 -- 172.21.15.28:0/2746140938 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4474005970 con 0x5568fcf43b40 2024-04-23T14:45:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.872+0000 7f4464c00640 1 -- 172.21.15.28:0/2746140938 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f446c01bb80 con 0x5568fcf43b40 2024-04-23T14:45:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.872+0000 7f448498e740 10 client.25565 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:45:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.872+0000 7f448498e740 10 client.25565 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:45:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.872+0000 7f448498e740 20 client.25565 populate_metadata read hostname 'smithi028' 2024-04-23T14:45:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.872+0000 7f448498e740 1 --2- 172.21.15.28:0/2746140938 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5568fcf630e0 0x5568fcfcba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.872+0000 7f448498e740 4 client.25565 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-23T14:45:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.872+0000 7f448498e740 1 -- 172.21.15.28:0/2746140938 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5568fceb0a60 con 0x5568fcf630e0 2024-04-23T14:45:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.872+0000 7f447a000640 1 --2- 172.21.15.28:0/2746140938 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5568fcf630e0 0x5568fcfcba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:45.881 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:45 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1588453066' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:45:45.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.873+0000 7f447a000640 1 --2- 172.21.15.28:0/2746140938 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5568fcf630e0 0x5568fcfcba30 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33a53362f70d3be4 server_cookie=4325fcafd94d7a3 in_seq=0 out_seq=0 2024-04-23T14:45:45.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.873+0000 7f4464c00640 10 client.25565 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:45:45.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.873+0000 7f4464c00640 1 -- 172.21.15.28:0/2746140938 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5568fceb0a60 con 0x5568fcf630e0 2024-04-23T14:45:45.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.873+0000 7f4464c00640 10 client.25565 handle_command_reply: tid=0 2024-04-23T14:45:45.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.873+0000 7f448498e740 1 client.25565 shutdown 2024-04-23T14:45:45.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.873+0000 7f448498e740 1 -- 172.21.15.28:0/2746140938 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5568fcf43b40 msgr2=0x5568fcf5f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:45.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.873+0000 7f448498e740 1 --2- 172.21.15.28:0/2746140938 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5568fcf43b40 0x5568fcf5f420 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f446c010520 tx=0x7f446c0046c0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.874+0000 7f448498e740 1 -- 172.21.15.28:0/2746140938 shutdown_connections 2024-04-23T14:45:45.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.874+0000 7f448498e740 1 --2- 172.21.15.28:0/2746140938 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5568fcf630e0 0x5568fcfcba30 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.874+0000 7f448498e740 1 --2- 172.21.15.28:0/2746140938 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568fcf4eb30 0x5568fcf4d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.874+0000 7f448498e740 1 --2- 172.21.15.28:0/2746140938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568fcf44460 0x5568fcf4ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.874+0000 7f448498e740 1 --2- 172.21.15.28:0/2746140938 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5568fcf43b40 0x5568fcf5f420 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.874+0000 7f448498e740 1 -- 172.21.15.28:0/2746140938 >> 172.21.15.28:0/2746140938 conn(0x5568fcf3fe60 msgr2=0x5568fcf52eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:45.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.874+0000 7f4464c00640 0 client.25565 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:45:45.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.874+0000 7f448498e740 1 -- 172.21.15.28:0/2746140938 shutdown_connections 2024-04-23T14:45:45.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.874+0000 7f448498e740 1 -- 172.21.15.28:0/2746140938 wait complete. 2024-04-23T14:45:45.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.874+0000 7f448498e740 20 client.25565 trim_cache size 0 max 16384 2024-04-23T14:45:45.884 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-23T14:45:45.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.875+0000 7f447be00640 1 -- 172.21.15.28:0/4184193852 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f44600a3da0 msgr2=0x7f44600a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:45.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.875+0000 7f447be00640 1 --2- 172.21.15.28:0/4184193852 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f44600a3da0 0x7f44600a6260 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f446c00afd0 tx=0x7f446c00a380 comp rx=0 tx=0).stop 2024-04-23T14:45:45.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.875+0000 7f447be00640 1 -- 172.21.15.28:0/4184193852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f447c101600 msgr2=0x7f447c198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:45.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.875+0000 7f447be00640 1 --2- 172.21.15.28:0/4184193852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f447c101600 0x7f447c198c50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f446802a960 tx=0x7f4468003b90 comp rx=0 tx=0).stop 2024-04-23T14:45:45.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.875+0000 7f447be00640 1 -- 172.21.15.28:0/4184193852 shutdown_connections 2024-04-23T14:45:45.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.875+0000 7f447be00640 1 --2- 172.21.15.28:0/4184193852 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f44600a3da0 0x7f44600a6260 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.875+0000 7f447be00640 1 --2- 172.21.15.28:0/4184193852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f447c101600 0x7f447c198c50 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.875+0000 7f447be00640 1 --2- 172.21.15.28:0/4184193852 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f447c106630 0x7f447c19cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.875+0000 7f447be00640 1 --2- 172.21.15.28:0/4184193852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f447c100c40 0x7f447c198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:45.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.875+0000 7f447be00640 1 -- 172.21.15.28:0/4184193852 >> 172.21.15.28:0/4184193852 conn(0x7f447c075d00 msgr2=0x7f447c0fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:45.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.875+0000 7f447be00640 1 -- 172.21.15.28:0/4184193852 shutdown_connections 2024-04-23T14:45:45.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:45.875+0000 7f447be00640 1 -- 172.21.15.28:0/4184193852 wait complete. 2024-04-23T14:45:46.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:45 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1588453066' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:45:46.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:45 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1588453066' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:45:46.458 INFO:tasks.workunit.client.0.smithi028.stdout: 1 4367 20.29 MB/sec warmup 2 sec latency 19.213 ms 2024-04-23T14:45:46.894 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-23T14:45:46.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:45:46.895 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-23T14:45:46.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:46 smithi028 ceph-mon[27810]: pgmap v55: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-23T14:45:47.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:46 smithi105 ceph-mon[32555]: pgmap v55: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-23T14:45:47.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:46 smithi038 ceph-mon[32348]: pgmap v55: 97 pgs: 97 active+clean; 1.3 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-23T14:45:47.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.136+0000 7f3c83e00640 1 -- 172.21.15.28:0/2461821915 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3c84100e50 msgr2=0x7f3c84101230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:47.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.136+0000 7f3c83e00640 1 --2- 172.21.15.28:0/2461821915 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3c84100e50 0x7f3c84101230 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3c74009860 tx=0x7f3c7402a3d0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.137+0000 7f3c83e00640 1 -- 172.21.15.28:0/2461821915 shutdown_connections 2024-04-23T14:45:47.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.137+0000 7f3c83e00640 1 --2- 172.21.15.28:0/2461821915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c84106250 0x7f3c841066e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.137+0000 7f3c83e00640 1 --2- 172.21.15.28:0/2461821915 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c84101770 0x7f3c84105c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.137+0000 7f3c83e00640 1 --2- 172.21.15.28:0/2461821915 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3c84100e50 0x7f3c84101230 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.137+0000 7f3c83e00640 1 -- 172.21.15.28:0/2461821915 >> 172.21.15.28:0/2461821915 conn(0x7f3c840fc880 msgr2=0x7f3c840feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:47.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.137+0000 7f3c83e00640 1 -- 172.21.15.28:0/2461821915 shutdown_connections 2024-04-23T14:45:47.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.137+0000 7f3c83e00640 1 -- 172.21.15.28:0/2461821915 wait complete. 2024-04-23T14:45:47.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.138+0000 7f3c83e00640 1 Processor -- start 2024-04-23T14:45:47.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.138+0000 7f3c83e00640 1 -- start start 2024-04-23T14:45:47.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.138+0000 7f3c83e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c84100e50 0x7f3c8419ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.139+0000 7f3c83e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3c84101770 0x7f3c8419f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.139+0000 7f3c83e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c84106250 0x7f3c841a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.139+0000 7f3c83e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3c841187f0 con 0x7f3c84106250 2024-04-23T14:45:47.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.139+0000 7f3c83e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3c84118670 con 0x7f3c84101770 2024-04-23T14:45:47.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.139+0000 7f3c83e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3c84118970 con 0x7f3c84100e50 2024-04-23T14:45:47.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.139+0000 7f3c82a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c84100e50 0x7f3c8419ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.139+0000 7f3c82000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3c84101770 0x7f3c8419f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.139+0000 7f3c82a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c84100e50 0x7f3c8419ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:53018/0 (socket says 172.21.15.28:53018) 2024-04-23T14:45:47.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.139+0000 7f3c82a00640 1 -- 172.21.15.28:0/1706919796 learned_addr learned my addr 172.21.15.28:0/1706919796 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:47.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.139+0000 7f3c83400640 1 --2- 172.21.15.28:0/1706919796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c84106250 0x7f3c841a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.139+0000 7f3c82a00640 1 -- 172.21.15.28:0/1706919796 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3c84101770 msgr2=0x7f3c8419f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:47.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.139+0000 7f3c82a00640 1 --2- 172.21.15.28:0/1706919796 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3c84101770 0x7f3c8419f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.139+0000 7f3c82a00640 1 -- 172.21.15.28:0/1706919796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c84106250 msgr2=0x7f3c841a3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:47.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.139+0000 7f3c82a00640 1 --2- 172.21.15.28:0/1706919796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c84106250 0x7f3c841a3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.139+0000 7f3c82a00640 1 -- 172.21.15.28:0/1706919796 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c841a3d80 con 0x7f3c84100e50 2024-04-23T14:45:47.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.139+0000 7f3c82000640 1 --2- 172.21.15.28:0/1706919796 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3c84101770 0x7f3c8419f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:45:47.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.140+0000 7f3c82a00640 1 --2- 172.21.15.28:0/1706919796 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c84100e50 0x7f3c8419ed90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3c7402a8f0 tx=0x7f3c74030790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:47.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.140+0000 7f3c7b400640 1 -- 172.21.15.28:0/1706919796 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c74040020 con 0x7f3c84100e50 2024-04-23T14:45:47.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.140+0000 7f3c7b400640 1 -- 172.21.15.28:0/1706919796 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c74004440 con 0x7f3c84100e50 2024-04-23T14:45:47.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.140+0000 7f3c83e00640 1 -- 172.21.15.28:0/1706919796 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c841a4010 con 0x7f3c84100e50 2024-04-23T14:45:47.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.140+0000 7f3c7b400640 1 -- 172.21.15.28:0/1706919796 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c740046c0 con 0x7f3c84100e50 2024-04-23T14:45:47.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.140+0000 7f3c83e00640 1 -- 172.21.15.28:0/1706919796 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c841ab900 con 0x7f3c84100e50 2024-04-23T14:45:47.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.143+0000 7f3c8c363740 1 Processor -- start 2024-04-23T14:45:47.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.143+0000 7f3c7b400640 1 -- 172.21.15.28:0/1706919796 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f3c74043020 con 0x7f3c84100e50 2024-04-23T14:45:47.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.144+0000 7f3c8c363740 1 -- start start 2024-04-23T14:45:47.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.144+0000 7f3c8c363740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d4fd60270 0x564d4fd60650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.144+0000 7f3c8c363740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4fd60b90 0x564d4fd6ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.144+0000 7f3c8c363740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564d4fd6b260 0x564d4fd6d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.144+0000 7f3c8c363740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564d4fcc5e70 con 0x564d4fd60b90 2024-04-23T14:45:47.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.144+0000 7f3c8c363740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564d4fc6c380 con 0x564d4fd60270 2024-04-23T14:45:47.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.144+0000 7f3c8c363740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x564d4fcc1e30 con 0x564d4fd6b260 2024-04-23T14:45:47.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.144+0000 7f3c83400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d4fd60270 0x564d4fd60650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.144+0000 7f3c83400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d4fd60270 0x564d4fd60650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:53652/0 (socket says 172.21.15.28:53652) 2024-04-23T14:45:47.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.144+0000 7f3c83400640 1 -- 172.21.15.28:0/4229582267 learned_addr learned my addr 172.21.15.28:0/4229582267 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:47.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.144+0000 7f3c82a00640 1 --2- 172.21.15.28:0/4229582267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4fd60b90 0x564d4fd6ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.144+0000 7f3c82000640 1 --2- 172.21.15.28:0/4229582267 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564d4fd6b260 0x564d4fd6d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.145+0000 7f3c83400640 1 -- 172.21.15.28:0/4229582267 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564d4fd6b260 msgr2=0x564d4fd6d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:47.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.145+0000 7f3c83400640 1 --2- 172.21.15.28:0/4229582267 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564d4fd6b260 0x564d4fd6d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.145+0000 7f3c83400640 1 -- 172.21.15.28:0/4229582267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4fd60b90 msgr2=0x564d4fd6ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:47.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.145+0000 7f3c83400640 1 --2- 172.21.15.28:0/4229582267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4fd60b90 0x564d4fd6ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.145+0000 7f3c83400640 1 -- 172.21.15.28:0/4229582267 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d4faa07e0 con 0x564d4fd60270 2024-04-23T14:45:47.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.145+0000 7f3c82000640 1 --2- 172.21.15.28:0/4229582267 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564d4fd6b260 0x564d4fd6d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:45:47.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.145+0000 7f3c83400640 1 --2- 172.21.15.28:0/4229582267 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d4fd60270 0x564d4fd60650 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c009f90 tx=0x7f3c7c0107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc737d2cd273a8a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:47.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.145+0000 7f3c7b400640 1 --2- 172.21.15.28:0/1706919796 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f3c680a3d20 0x7f3c680a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.145+0000 7f3c73400640 1 -- 172.21.15.28:0/4229582267 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c011640 con 0x564d4fd60270 2024-04-23T14:45:47.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.145+0000 7f3c73400640 1 -- 172.21.15.28:0/4229582267 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c7c011cc0 con 0x564d4fd60270 2024-04-23T14:45:47.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.145+0000 7f3c73400640 1 -- 172.21.15.28:0/4229582267 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c019d50 con 0x564d4fd60270 2024-04-23T14:45:47.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.146+0000 7f3c7b400640 1 -- 172.21.15.28:0/1706919796 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f3c740f1720 con 0x7f3c84100e50 2024-04-23T14:45:47.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.146+0000 7f3c82a00640 1 --2- 172.21.15.28:0/1706919796 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f3c680a3d20 0x7f3c680a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.146+0000 7f3c8c363740 1 -- 172.21.15.28:0/4229582267 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d4fd60270 msgr2=0x564d4fd60650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:47.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.146+0000 7f3c8c363740 1 --2- 172.21.15.28:0/4229582267 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d4fd60270 0x564d4fd60650 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c009f90 tx=0x7f3c7c0107a0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.146+0000 7f3c82a00640 1 --2- 172.21.15.28:0/1706919796 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f3c680a3d20 0x7f3c680a61e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3c740eef30 tx=0x7f3c740f7550 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:47.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.146+0000 7f3c8c363740 1 -- 172.21.15.28:0/4229582267 shutdown_connections 2024-04-23T14:45:47.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.147+0000 7f3c8c363740 1 --2- 172.21.15.28:0/4229582267 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564d4fd6b260 0x564d4fd6d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.147+0000 7f3c8c363740 1 --2- 172.21.15.28:0/4229582267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4fd60b90 0x564d4fd6ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.147+0000 7f3c8c363740 1 --2- 172.21.15.28:0/4229582267 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d4fd60270 0x564d4fd60650 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.147+0000 7f3c8c363740 1 -- 172.21.15.28:0/4229582267 >> 172.21.15.28:0/4229582267 conn(0x564d4fd5c590 msgr2=0x564d4fd5c970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:47.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.147+0000 7f3c8c363740 1 -- 172.21.15.28:0/4229582267 shutdown_connections 2024-04-23T14:45:47.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.147+0000 7f3c8c363740 1 -- 172.21.15.28:0/4229582267 wait complete. 2024-04-23T14:45:47.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.147+0000 7f3c8c363740 1 -- start start 2024-04-23T14:45:47.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.147+0000 7f3c8c363740 1 Processor -- start 2024-04-23T14:45:47.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.148+0000 7f3c8c363740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4fd60270 0x564d4fd681f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.148+0000 7f3c8c363740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d4fd60b90 0x564d4fd68730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.148+0000 7f3c8c363740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564d4fd6b260 0x564d4fd68c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.148+0000 7f3c8c363740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564d4fd6e050 con 0x564d4fd60270 2024-04-23T14:45:47.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.148+0000 7f3c8c363740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564d4fc7f660 con 0x564d4fd60b90 2024-04-23T14:45:47.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.148+0000 7f3c8c363740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x564d4fd6e1d0 con 0x564d4fd6b260 2024-04-23T14:45:47.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.148+0000 7f3c8c363740 10 client.0 _ll_register_callbacks cb 0x564d4fc6daa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:45:47.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.148+0000 7f3c83400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4fd60270 0x564d4fd681f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.148+0000 7f3c82000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d4fd60b90 0x564d4fd68730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.148+0000 7f3c83400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4fd60270 0x564d4fd681f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36204/0 (socket says 172.21.15.28:36204) 2024-04-23T14:45:47.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.148+0000 7f3c83400640 1 -- 172.21.15.28:0/3258267207 learned_addr learned my addr 172.21.15.28:0/3258267207 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:47.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.149+0000 7f3c83400640 1 --2- 172.21.15.28:0/3258267207 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564d4fd6b260 0x564d4fd68c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.149+0000 7f3c83400640 1 -- 172.21.15.28:0/3258267207 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564d4fd6b260 msgr2=0x564d4fd68c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:47.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.149+0000 7f3c83400640 1 --2- 172.21.15.28:0/3258267207 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564d4fd6b260 0x564d4fd68c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.149+0000 7f3c83400640 1 -- 172.21.15.28:0/3258267207 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d4fd60b90 msgr2=0x564d4fd68730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:47.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.149+0000 7f3c83400640 1 --2- 172.21.15.28:0/3258267207 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d4fd60b90 0x564d4fd68730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.149+0000 7f3c83400640 1 -- 172.21.15.28:0/3258267207 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d4fb00f70 con 0x564d4fd60270 2024-04-23T14:45:47.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.149+0000 7f3c83400640 1 --2- 172.21.15.28:0/3258267207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4fd60270 0x564d4fd681f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c009990 tx=0x7f3c7c009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cc749cbcc612554 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:47.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.149+0000 7f3c67e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-23T14:45:47.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.150+0000 7f3c67e00640 1 -- 172.21.15.28:0/3258267207 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c019cc0 con 0x564d4fd60270 2024-04-23T14:45:47.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.150+0000 7f3c67e00640 1 -- 172.21.15.28:0/3258267207 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f3c7c004a40 con 0x564d4fd60270 2024-04-23T14:45:47.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.150+0000 7f3c8c363740 1 -- 172.21.15.28:0/3258267207 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c7c011cc0 con 0x564d4fd60270 2024-04-23T14:45:47.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.150+0000 7f3c67e00640 1 -- 172.21.15.28:0/3258267207 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f3c7c019e60 con 0x564d4fd60270 2024-04-23T14:45:47.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.150+0000 7f3c8c363740 10 client.15435 fetch_fsmap learned FSMap version 20 2024-04-23T14:45:47.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.150+0000 7f3c8c363740 1 -- 172.21.15.28:0/3258267207 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x564d4f998170 con 0x564d4fd60270 2024-04-23T14:45:47.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.150+0000 7f3c67e00640 1 -- 172.21.15.28:0/3258267207 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c7c011cc0 con 0x564d4fd60270 2024-04-23T14:45:47.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.150+0000 7f3c67e00640 1 -- 172.21.15.28:0/3258267207 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f3c7c025030 con 0x564d4fd60270 2024-04-23T14:45:47.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.151+0000 7f3c8c363740 10 client.15435 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:45:47.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.151+0000 7f3c8c363740 10 client.15435 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:45:47.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.151+0000 7f3c8c363740 20 client.15435 populate_metadata read hostname 'smithi028' 2024-04-23T14:45:47.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.151+0000 7f3c8c363740 1 --2- 172.21.15.28:0/3258267207 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564d4fd7b470 0x564d4fd7d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.151+0000 7f3c8c363740 4 client.15435 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:45:47.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.151+0000 7f3c8c363740 1 -- 172.21.15.28:0/3258267207 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564d4fcd3890 con 0x564d4fd7b470 2024-04-23T14:45:47.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.151+0000 7f3c82000640 1 --2- 172.21.15.28:0/3258267207 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564d4fd7b470 0x564d4fd7d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.151+0000 7f3c82000640 1 --2- 172.21.15.28:0/3258267207 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564d4fd7b470 0x564d4fd7d860 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91c255a914049ee8 server_cookie=f4b43b00764e1cc9 in_seq=0 out_seq=0 2024-04-23T14:45:47.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.151+0000 7f3c67e00640 10 client.15435 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:45:47.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.153+0000 7f3c67e00640 1 -- 172.21.15.28:0/3258267207 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564d4fcd3890 con 0x564d4fd7b470 2024-04-23T14:45:47.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.153+0000 7f3c67e00640 10 client.15435 handle_command_reply: tid=0 2024-04-23T14:45:47.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.153+0000 7f3c8c363740 1 client.15435 shutdown 2024-04-23T14:45:47.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.154+0000 7f3c8c363740 1 -- 172.21.15.28:0/3258267207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4fd60270 msgr2=0x564d4fd681f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:47.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.154+0000 7f3c8c363740 1 --2- 172.21.15.28:0/3258267207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4fd60270 0x564d4fd681f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c009990 tx=0x7f3c7c009900 comp rx=0 tx=0).stop 2024-04-23T14:45:47.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.154+0000 7f3c8c363740 1 -- 172.21.15.28:0/3258267207 shutdown_connections 2024-04-23T14:45:47.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.154+0000 7f3c8c363740 1 --2- 172.21.15.28:0/3258267207 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564d4fd7b470 0x564d4fd7d860 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.154+0000 7f3c8c363740 1 --2- 172.21.15.28:0/3258267207 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d4fd60b90 0x564d4fd68730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.154+0000 7f3c8c363740 1 --2- 172.21.15.28:0/3258267207 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564d4fd6b260 0x564d4fd68c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.154+0000 7f3c67e00640 0 client.15435 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:45:47.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.154+0000 7f3c8c363740 1 --2- 172.21.15.28:0/3258267207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4fd60270 0x564d4fd681f0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.154+0000 7f3c8c363740 1 -- 172.21.15.28:0/3258267207 >> 172.21.15.28:0/3258267207 conn(0x564d4fd5c590 msgr2=0x564d4fd62390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:47.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.154+0000 7f3c8c363740 1 -- 172.21.15.28:0/3258267207 shutdown_connections 2024-04-23T14:45:47.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.154+0000 7f3c8c363740 1 -- 172.21.15.28:0/3258267207 wait complete. 2024-04-23T14:45:47.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.154+0000 7f3c8c363740 20 client.15435 trim_cache size 0 max 16384 2024-04-23T14:45:47.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.175+0000 7f3c8c363740 1 Processor -- start 2024-04-23T14:45:47.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.175+0000 7f3c8c363740 1 -- start start 2024-04-23T14:45:47.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.175+0000 7f3c8c363740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d4fd60270 0x564d4fd76f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.175+0000 7f3c8c363740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4fd60b90 0x564d4fd677d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.175+0000 7f3c8c363740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564d4fd6b260 0x564d4fd62110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.175+0000 7f3c8c363740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3c841b2180 con 0x564d4fd60b90 2024-04-23T14:45:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.175+0000 7f3c8c363740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3c841b4a00 con 0x564d4fd60270 2024-04-23T14:45:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.175+0000 7f3c8c363740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3c841b27f0 con 0x564d4fd6b260 2024-04-23T14:45:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.175+0000 7f3c83400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d4fd60270 0x564d4fd76f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.175+0000 7f3c83400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d4fd60270 0x564d4fd76f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:53658/0 (socket says 172.21.15.28:53658) 2024-04-23T14:45:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.175+0000 7f3c83400640 1 -- 172.21.15.28:0/2548403393 learned_addr learned my addr 172.21.15.28:0/2548403393 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.175+0000 7f3c82000640 1 --2- 172.21.15.28:0/2548403393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4fd60b90 0x564d4fd677d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.175+0000 7f3c83400640 1 --2- 172.21.15.28:0/2548403393 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564d4fd6b260 0x564d4fd62110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.176+0000 7f3c83400640 1 -- 172.21.15.28:0/2548403393 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564d4fd6b260 msgr2=0x564d4fd62110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.176+0000 7f3c83400640 1 --2- 172.21.15.28:0/2548403393 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564d4fd6b260 0x564d4fd62110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.176+0000 7f3c83400640 1 -- 172.21.15.28:0/2548403393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4fd60b90 msgr2=0x564d4fd677d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:47.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.176+0000 7f3c83400640 1 --2- 172.21.15.28:0/2548403393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4fd60b90 0x564d4fd677d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.176+0000 7f3c83400640 1 -- 172.21.15.28:0/2548403393 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d4fd57ef0 con 0x564d4fd60270 2024-04-23T14:45:47.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.176+0000 7f3c83400640 1 --2- 172.21.15.28:0/2548403393 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d4fd60270 0x564d4fd76f10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c009a80 tx=0x7f3c7c0107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=759bfe99d2622766 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:47.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.176+0000 7f3c72a00640 1 -- 172.21.15.28:0/2548403393 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c036030 con 0x564d4fd60270 2024-04-23T14:45:47.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.176+0000 7f3c72a00640 1 -- 172.21.15.28:0/2548403393 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c7c037040 con 0x564d4fd60270 2024-04-23T14:45:47.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.176+0000 7f3c72a00640 1 -- 172.21.15.28:0/2548403393 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c0195a0 con 0x564d4fd60270 2024-04-23T14:45:47.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.177+0000 7f3c8c363740 1 -- 172.21.15.28:0/2548403393 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d4fd60270 msgr2=0x564d4fd76f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:47.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.177+0000 7f3c8c363740 1 --2- 172.21.15.28:0/2548403393 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d4fd60270 0x564d4fd76f10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c009a80 tx=0x7f3c7c0107a0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.177+0000 7f3c8c363740 1 -- 172.21.15.28:0/2548403393 shutdown_connections 2024-04-23T14:45:47.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.177+0000 7f3c8c363740 1 --2- 172.21.15.28:0/2548403393 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564d4fd6b260 0x564d4fd62110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.177+0000 7f3c8c363740 1 --2- 172.21.15.28:0/2548403393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4fd60b90 0x564d4fd677d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.177+0000 7f3c8c363740 1 --2- 172.21.15.28:0/2548403393 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d4fd60270 0x564d4fd76f10 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.177+0000 7f3c8c363740 1 -- 172.21.15.28:0/2548403393 >> 172.21.15.28:0/2548403393 conn(0x564d4fd5c590 msgr2=0x564d4fd61a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:47.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.177+0000 7f3c8c363740 1 -- 172.21.15.28:0/2548403393 shutdown_connections 2024-04-23T14:45:47.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.177+0000 7f3c8c363740 1 -- 172.21.15.28:0/2548403393 wait complete. 2024-04-23T14:45:47.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.178+0000 7f3c8c363740 1 -- start start 2024-04-23T14:45:47.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.178+0000 7f3c8c363740 1 Processor -- start 2024-04-23T14:45:47.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.178+0000 7f3c8c363740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564d4fd60270 0x564d4fd736e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.178+0000 7f3c8c363740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4fd60b90 0x564d4fd64be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.178+0000 7f3c8c363740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d4fd6b260 0x564d4fd65120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.178+0000 7f3c8c363740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3c841b2ec0 con 0x564d4fd60b90 2024-04-23T14:45:47.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.178+0000 7f3c8c363740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3c841b4330 con 0x564d4fd6b260 2024-04-23T14:45:47.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.178+0000 7f3c8c363740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3c841b3590 con 0x564d4fd60270 2024-04-23T14:45:47.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.179+0000 7f3c8c363740 10 client.0 _ll_register_callbacks cb 0x7f3c740f7420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:45:47.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.179+0000 7f3c83400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564d4fd60270 0x564d4fd736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.179+0000 7f3c82000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4fd60b90 0x564d4fd64be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.179+0000 7f3c83400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564d4fd60270 0x564d4fd736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:53054/0 (socket says 172.21.15.28:53054) 2024-04-23T14:45:47.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.179+0000 7f3c83400640 1 -- 172.21.15.28:0/2780191739 learned_addr learned my addr 172.21.15.28:0/2780191739 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:47.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.179+0000 7f3c83400640 1 --2- 172.21.15.28:0/2780191739 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d4fd6b260 0x564d4fd65120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.179+0000 7f3c83400640 1 -- 172.21.15.28:0/2780191739 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d4fd6b260 msgr2=0x564d4fd65120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:47.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.179+0000 7f3c83400640 1 --2- 172.21.15.28:0/2780191739 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d4fd6b260 0x564d4fd65120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.179+0000 7f3c83400640 1 -- 172.21.15.28:0/2780191739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4fd60b90 msgr2=0x564d4fd64be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:47.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.179+0000 7f3c83400640 1 --2- 172.21.15.28:0/2780191739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4fd60b90 0x564d4fd64be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.179+0000 7f3c83400640 1 -- 172.21.15.28:0/2780191739 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d4fd65860 con 0x564d4fd60270 2024-04-23T14:45:47.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.179+0000 7f3c83400640 1 --2- 172.21.15.28:0/2780191739 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564d4fd60270 0x564d4fd736e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c019910 tx=0x7f3c7c0097d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d18fe11633c8b9ce server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:47.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.179+0000 7f3c67e00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:45:47.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.179+0000 7f3c67e00640 1 -- 172.21.15.28:0/2780191739 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c036040 con 0x564d4fd60270 2024-04-23T14:45:47.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.180+0000 7f3c8c363740 1 -- 172.21.15.28:0/2780191739 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c7c037040 con 0x564d4fd60270 2024-04-23T14:45:47.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.180+0000 7f3c67e00640 1 -- 172.21.15.28:0/2780191739 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f3c7c003d80 con 0x564d4fd60270 2024-04-23T14:45:47.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.180+0000 7f3c67e00640 1 -- 172.21.15.28:0/2780191739 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f3c7c005350 con 0x564d4fd60270 2024-04-23T14:45:47.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.180+0000 7f3c8c363740 10 client.25588 fetch_fsmap learned FSMap version 20 2024-04-23T14:45:47.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.180+0000 7f3c8c363740 1 -- 172.21.15.28:0/2780191739 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x564d4fde1740 con 0x564d4fd60270 2024-04-23T14:45:47.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.180+0000 7f3c67e00640 1 -- 172.21.15.28:0/2780191739 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c7c037040 con 0x564d4fd60270 2024-04-23T14:45:47.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.180+0000 7f3c67e00640 1 -- 172.21.15.28:0/2780191739 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f3c7c03c070 con 0x564d4fd60270 2024-04-23T14:45:47.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.180+0000 7f3c8c363740 10 client.25588 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:45:47.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.180+0000 7f3c8c363740 10 client.25588 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:45:47.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.180+0000 7f3c8c363740 20 client.25588 populate_metadata read hostname 'smithi028' 2024-04-23T14:45:47.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.181+0000 7f3c8c363740 1 --2- 172.21.15.28:0/2780191739 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564d4fde1d50 0x564d4fde4140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.181+0000 7f3c8c363740 4 client.25588 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-23T14:45:47.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.181+0000 7f3c8c363740 1 -- 172.21.15.28:0/2780191739 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564d4fccd700 con 0x564d4fde1d50 2024-04-23T14:45:47.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.181+0000 7f3c82000640 1 --2- 172.21.15.28:0/2780191739 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564d4fde1d50 0x564d4fde4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.181+0000 7f3c82000640 1 --2- 172.21.15.28:0/2780191739 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564d4fde1d50 0x564d4fde4140 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=207f2442744ee5b5 server_cookie=2e9e60780525fdce in_seq=0 out_seq=0 2024-04-23T14:45:47.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.181+0000 7f3c67e00640 10 client.25588 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:45:47.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.182+0000 7f3c67e00640 1 -- 172.21.15.28:0/2780191739 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564d4fccd700 con 0x564d4fde1d50 2024-04-23T14:45:47.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.182+0000 7f3c67e00640 10 client.25588 handle_command_reply: tid=0 2024-04-23T14:45:47.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.182+0000 7f3c8c363740 1 client.25588 shutdown 2024-04-23T14:45:47.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.182+0000 7f3c8c363740 1 -- 172.21.15.28:0/2780191739 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564d4fd60270 msgr2=0x564d4fd736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:47.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.182+0000 7f3c8c363740 1 --2- 172.21.15.28:0/2780191739 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564d4fd60270 0x564d4fd736e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c019910 tx=0x7f3c7c0097d0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.183+0000 7f3c8c363740 1 -- 172.21.15.28:0/2780191739 shutdown_connections 2024-04-23T14:45:47.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.183+0000 7f3c8c363740 1 --2- 172.21.15.28:0/2780191739 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564d4fde1d50 0x564d4fde4140 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.183+0000 7f3c8c363740 1 --2- 172.21.15.28:0/2780191739 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d4fd6b260 0x564d4fd65120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.183+0000 7f3c8c363740 1 --2- 172.21.15.28:0/2780191739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4fd60b90 0x564d4fd64be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.183+0000 7f3c8c363740 1 --2- 172.21.15.28:0/2780191739 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564d4fd60270 0x564d4fd736e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.183+0000 7f3c67e00640 0 client.25588 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:45:47.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.183+0000 7f3c8c363740 1 -- 172.21.15.28:0/2780191739 >> 172.21.15.28:0/2780191739 conn(0x564d4fd5c590 msgr2=0x564d4fd68a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:47.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.183+0000 7f3c8c363740 1 -- 172.21.15.28:0/2780191739 shutdown_connections 2024-04-23T14:45:47.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.183+0000 7f3c8c363740 1 -- 172.21.15.28:0/2780191739 wait complete. 2024-04-23T14:45:47.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.183+0000 7f3c8c363740 20 client.25588 trim_cache size 0 max 16384 2024-04-23T14:45:47.188 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:45:47.188 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-23T14:45:47.188 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "e273475a-9889-4ab0-9a0b-3f5909d3f63c", 2024-04-23T14:45:47.189 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-23T14:45:47.189 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:45:47.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.184+0000 7f3c83e00640 1 -- 172.21.15.28:0/1706919796 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f3c680a3d20 msgr2=0x7f3c680a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:47.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.184+0000 7f3c83e00640 1 --2- 172.21.15.28:0/1706919796 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f3c680a3d20 0x7f3c680a61e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3c740eef30 tx=0x7f3c740f7550 comp rx=0 tx=0).stop 2024-04-23T14:45:47.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.184+0000 7f3c83e00640 1 -- 172.21.15.28:0/1706919796 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c84100e50 msgr2=0x7f3c8419ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:47.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.184+0000 7f3c83e00640 1 --2- 172.21.15.28:0/1706919796 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c84100e50 0x7f3c8419ed90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3c7402a8f0 tx=0x7f3c74030790 comp rx=0 tx=0).stop 2024-04-23T14:45:47.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.184+0000 7f3c83e00640 1 -- 172.21.15.28:0/1706919796 shutdown_connections 2024-04-23T14:45:47.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.184+0000 7f3c83e00640 1 --2- 172.21.15.28:0/1706919796 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f3c680a3d20 0x7f3c680a61e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.184+0000 7f3c83e00640 1 --2- 172.21.15.28:0/1706919796 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3c84101770 0x7f3c8419f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.184+0000 7f3c83e00640 1 --2- 172.21.15.28:0/1706919796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c84106250 0x7f3c841a3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.184+0000 7f3c83e00640 1 --2- 172.21.15.28:0/1706919796 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c84100e50 0x7f3c8419ed90 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.184+0000 7f3c83e00640 1 -- 172.21.15.28:0/1706919796 >> 172.21.15.28:0/1706919796 conn(0x7f3c840fc880 msgr2=0x7f3c840fd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:47.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.184+0000 7f3c83e00640 1 -- 172.21.15.28:0/1706919796 shutdown_connections 2024-04-23T14:45:47.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.184+0000 7f3c83e00640 1 -- 172.21.15.28:0/1706919796 wait complete. 2024-04-23T14:45:47.202 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:45:47.203 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:45:47.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.411+0000 7f7de3e00640 1 -- 172.21.15.28:0/2894654882 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7de4102aa0 msgr2=0x7f7de4106ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:47.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.411+0000 7f7de3e00640 1 --2- 172.21.15.28:0/2894654882 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7de4102aa0 0x7f7de4106ae0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7dd000fb60 tx=0x7f7dd002e4c0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.411+0000 7f7de3e00640 1 -- 172.21.15.28:0/2894654882 shutdown_connections 2024-04-23T14:45:47.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.411+0000 7f7de3e00640 1 --2- 172.21.15.28:0/2894654882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7de41071c0 0x7f7de410ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.411+0000 7f7de3e00640 1 --2- 172.21.15.28:0/2894654882 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7de4102aa0 0x7f7de4106ae0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.411+0000 7f7de3e00640 1 --2- 172.21.15.28:0/2894654882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7de41020f0 0x7f7de41024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.411+0000 7f7de3e00640 1 -- 172.21.15.28:0/2894654882 >> 172.21.15.28:0/2894654882 conn(0x7f7de40fd9d0 msgr2=0x7f7de40ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:47.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.411+0000 7f7de3e00640 1 -- 172.21.15.28:0/2894654882 shutdown_connections 2024-04-23T14:45:47.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.411+0000 7f7de3e00640 1 -- 172.21.15.28:0/2894654882 wait complete. 2024-04-23T14:45:47.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.412+0000 7f7de3e00640 1 Processor -- start 2024-04-23T14:45:47.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.412+0000 7f7de3e00640 1 -- start start 2024-04-23T14:45:47.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.413+0000 7f7de3e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7de41020f0 0x7f7de41965a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.413+0000 7f7de3e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7de41071c0 0x7f7de4196ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.413+0000 7f7de3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7de419ae10 0x7f7de419b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.413+0000 7f7de3e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7de4110060 con 0x7f7de419ae10 2024-04-23T14:45:47.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.413+0000 7f7de3e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7de410fee0 con 0x7f7de41071c0 2024-04-23T14:45:47.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.413+0000 7f7de3e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7de41101e0 con 0x7f7de41020f0 2024-04-23T14:45:47.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.413+0000 7f7de2000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7de41071c0 0x7f7de4196ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.413+0000 7f7de3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7de419ae10 0x7f7de419b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.413+0000 7f7de2000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7de41071c0 0x7f7de4196ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:53692/0 (socket says 172.21.15.28:53692) 2024-04-23T14:45:47.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.413+0000 7f7de2000640 1 -- 172.21.15.28:0/2773280541 learned_addr learned my addr 172.21.15.28:0/2773280541 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:47.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.413+0000 7f7de2a00640 1 --2- 172.21.15.28:0/2773280541 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7de41020f0 0x7f7de41965a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.413+0000 7f7de2000640 1 -- 172.21.15.28:0/2773280541 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7de41020f0 msgr2=0x7f7de41965a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:47.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.413+0000 7f7de2000640 1 --2- 172.21.15.28:0/2773280541 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7de41020f0 0x7f7de41965a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.414+0000 7f7de2000640 1 -- 172.21.15.28:0/2773280541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7de419ae10 msgr2=0x7f7de419b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:47.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.414+0000 7f7de2000640 1 --2- 172.21.15.28:0/2773280541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7de419ae10 0x7f7de419b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.414+0000 7f7de2000640 1 -- 172.21.15.28:0/2773280541 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7de419b900 con 0x7f7de41071c0 2024-04-23T14:45:47.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.414+0000 7f7de2000640 1 --2- 172.21.15.28:0/2773280541 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7de41071c0 0x7f7de4196ae0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7dd002e9e0 tx=0x7f7dd0034650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:47.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.414+0000 7f7ddb400640 1 -- 172.21.15.28:0/2773280541 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd0044020 con 0x7f7de41071c0 2024-04-23T14:45:47.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.414+0000 7f7ddb400640 1 -- 172.21.15.28:0/2773280541 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7dd0005510 con 0x7f7de41071c0 2024-04-23T14:45:47.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.414+0000 7f7ddb400640 1 -- 172.21.15.28:0/2773280541 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd00046c0 con 0x7f7de41071c0 2024-04-23T14:45:47.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.414+0000 7f7de3e00640 1 -- 172.21.15.28:0/2773280541 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7de419bb90 con 0x7f7de41071c0 2024-04-23T14:45:47.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.415+0000 7f7de3e00640 1 -- 172.21.15.28:0/2773280541 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7de41a3300 con 0x7f7de41071c0 2024-04-23T14:45:47.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.418+0000 7f7dec3f9740 1 Processor -- start 2024-04-23T14:45:47.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.418+0000 7f7ddb400640 1 -- 172.21.15.28:0/2773280541 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f7dd0047020 con 0x7f7de41071c0 2024-04-23T14:45:47.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.419+0000 7f7dec3f9740 1 -- start start 2024-04-23T14:45:47.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.419+0000 7f7dec3f9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fc920fa30 0x563fc920fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.419+0000 7f7dec3f9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fc9210350 0x563fc921a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.419+0000 7f7dec3f9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563fc921aa20 0x563fc921ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.419+0000 7f7dec3f9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563fc91751f0 con 0x563fc9210350 2024-04-23T14:45:47.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.419+0000 7f7dec3f9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563fc911b350 con 0x563fc921aa20 2024-04-23T14:45:47.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.419+0000 7f7dec3f9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x563fc91711b0 con 0x563fc920fa30 2024-04-23T14:45:47.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.419+0000 7f7de2000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563fc921aa20 0x563fc921ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.419+0000 7f7de2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fc9210350 0x563fc921a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.419+0000 7f7de3400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fc920fa30 0x563fc920fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.419+0000 7f7de2000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563fc921aa20 0x563fc921ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:53706/0 (socket says 172.21.15.28:53706) 2024-04-23T14:45:47.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.419+0000 7f7de2000640 1 -- 172.21.15.28:0/1353532583 learned_addr learned my addr 172.21.15.28:0/1353532583 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:47.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.420+0000 7f7de2000640 1 -- 172.21.15.28:0/1353532583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fc920fa30 msgr2=0x563fc920fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:47.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.420+0000 7f7de2000640 1 --2- 172.21.15.28:0/1353532583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fc920fa30 0x563fc920fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.420+0000 7f7de2000640 1 -- 172.21.15.28:0/1353532583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fc9210350 msgr2=0x563fc921a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:47.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.420+0000 7f7de2000640 1 --2- 172.21.15.28:0/1353532583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fc9210350 0x563fc921a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.420+0000 7f7de2000640 1 -- 172.21.15.28:0/1353532583 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563fc916ff50 con 0x563fc921aa20 2024-04-23T14:45:47.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.420+0000 7f7de3400640 1 --2- 172.21.15.28:0/1353532583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fc920fa30 0x563fc920fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:45:47.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.420+0000 7f7de2000640 1 --2- 172.21.15.28:0/1353532583 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563fc921aa20 0x563fc921ce10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7dd00f2900 tx=0x7f7dd00fd540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=219552931e6bbd3d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:47.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.420+0000 7f7ddb400640 1 --2- 172.21.15.28:0/2773280541 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7dc80a3da0 0x7f7dc80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.420+0000 7f7dcf400640 1 -- 172.21.15.28:0/1353532583 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd00bd4b0 con 0x563fc921aa20 2024-04-23T14:45:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.421+0000 7f7dcf400640 1 -- 172.21.15.28:0/1353532583 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7dd0102030 con 0x563fc921aa20 2024-04-23T14:45:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.421+0000 7f7dcf400640 1 -- 172.21.15.28:0/1353532583 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd00c0990 con 0x563fc921aa20 2024-04-23T14:45:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.421+0000 7f7ddb400640 1 -- 172.21.15.28:0/2773280541 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f7dd00f52c0 con 0x7f7de41071c0 2024-04-23T14:45:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.421+0000 7f7de3400640 1 --2- 172.21.15.28:0/2773280541 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7dc80a3da0 0x7f7dc80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.421+0000 7f7dec3f9740 1 -- 172.21.15.28:0/1353532583 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563fc921aa20 msgr2=0x563fc921ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.421+0000 7f7dec3f9740 1 --2- 172.21.15.28:0/1353532583 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563fc921aa20 0x563fc921ce10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7dd00f2900 tx=0x7f7dd00fd540 comp rx=0 tx=0).stop 2024-04-23T14:45:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.421+0000 7f7de3400640 1 --2- 172.21.15.28:0/2773280541 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7dc80a3da0 0x7f7dc80a6260 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc009fd0 tx=0x7f7ddc009500 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.422+0000 7f7dec3f9740 1 -- 172.21.15.28:0/1353532583 shutdown_connections 2024-04-23T14:45:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.422+0000 7f7dec3f9740 1 --2- 172.21.15.28:0/1353532583 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563fc921aa20 0x563fc921ce10 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.422+0000 7f7dec3f9740 1 --2- 172.21.15.28:0/1353532583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fc9210350 0x563fc921a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.422+0000 7f7dec3f9740 1 --2- 172.21.15.28:0/1353532583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fc920fa30 0x563fc920fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.422+0000 7f7dec3f9740 1 -- 172.21.15.28:0/1353532583 >> 172.21.15.28:0/1353532583 conn(0x563fc920bd50 msgr2=0x563fc920c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:47.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.422+0000 7f7dec3f9740 1 -- 172.21.15.28:0/1353532583 shutdown_connections 2024-04-23T14:45:47.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.422+0000 7f7dec3f9740 1 -- 172.21.15.28:0/1353532583 wait complete. 2024-04-23T14:45:47.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.422+0000 7f7dec3f9740 1 -- start start 2024-04-23T14:45:47.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.422+0000 7f7dec3f9740 1 Processor -- start 2024-04-23T14:45:47.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.423+0000 7f7dec3f9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fc920fa30 0x563fc9217980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.423+0000 7f7dec3f9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fc9210350 0x563fc9217ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.423+0000 7f7dec3f9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563fc921aa20 0x563fc9218400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.423+0000 7f7dec3f9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563fc921d5d0 con 0x563fc920fa30 2024-04-23T14:45:47.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.423+0000 7f7dec3f9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563fc912e690 con 0x563fc921aa20 2024-04-23T14:45:47.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.423+0000 7f7dec3f9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x563fc921d750 con 0x563fc9210350 2024-04-23T14:45:47.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.423+0000 7f7dec3f9740 10 client.0 _ll_register_callbacks cb 0x563fc916a650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:45:47.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.423+0000 7f7de3400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563fc921aa20 0x563fc9218400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.423+0000 7f7de2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fc920fa30 0x563fc9217980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.423+0000 7f7de3400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563fc921aa20 0x563fc9218400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:53708/0 (socket says 172.21.15.28:53708) 2024-04-23T14:45:47.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.423+0000 7f7de3400640 1 -- 172.21.15.28:0/1934272980 learned_addr learned my addr 172.21.15.28:0/1934272980 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:47.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.423+0000 7f7de2000640 1 --2- 172.21.15.28:0/1934272980 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fc9210350 0x563fc9217ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.424+0000 7f7de3400640 1 -- 172.21.15.28:0/1934272980 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fc9210350 msgr2=0x563fc9217ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:47.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.424+0000 7f7de3400640 1 --2- 172.21.15.28:0/1934272980 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fc9210350 0x563fc9217ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.424+0000 7f7de3400640 1 -- 172.21.15.28:0/1934272980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fc920fa30 msgr2=0x563fc9217980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:47.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.424+0000 7f7de3400640 1 --2- 172.21.15.28:0/1934272980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fc920fa30 0x563fc9217980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.424+0000 7f7de3400640 1 -- 172.21.15.28:0/1934272980 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563fc912d920 con 0x563fc921aa20 2024-04-23T14:45:47.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.424+0000 7f7de3400640 1 --2- 172.21.15.28:0/1934272980 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563fc921aa20 0x563fc9218400 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc00f6a0 tx=0x7f7ddc004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9df4fe8e0740fca server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:47.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.424+0000 7f7dc3e00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:45:47.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.424+0000 7f7dc3e00640 1 -- 172.21.15.28:0/1934272980 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc02c060 con 0x563fc921aa20 2024-04-23T14:45:47.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.424+0000 7f7dec3f9740 1 -- 172.21.15.28:0/1934272980 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7dd0102030 con 0x563fc921aa20 2024-04-23T14:45:47.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.424+0000 7f7dc3e00640 1 -- 172.21.15.28:0/1934272980 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f7ddc017030 con 0x563fc921aa20 2024-04-23T14:45:47.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.425+0000 7f7dc3e00640 1 -- 172.21.15.28:0/1934272980 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f7ddc01f800 con 0x563fc921aa20 2024-04-23T14:45:47.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.425+0000 7f7dec3f9740 10 client.25595 fetch_fsmap learned FSMap version 20 2024-04-23T14:45:47.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.425+0000 7f7dec3f9740 1 -- 172.21.15.28:0/1934272980 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x563fc8f517d0 con 0x563fc921aa20 2024-04-23T14:45:47.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.425+0000 7f7dc3e00640 1 -- 172.21.15.28:0/1934272980 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7dd0102030 con 0x563fc921aa20 2024-04-23T14:45:47.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.425+0000 7f7dc3e00640 1 -- 172.21.15.28:0/1934272980 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f7ddc010c80 con 0x563fc921aa20 2024-04-23T14:45:47.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.425+0000 7f7dec3f9740 10 client.25595 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:45:47.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.425+0000 7f7dec3f9740 10 client.25595 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:45:47.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.425+0000 7f7dec3f9740 20 client.25595 populate_metadata read hostname 'smithi028' 2024-04-23T14:45:47.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.425+0000 7f7dec3f9740 1 --2- 172.21.15.28:0/1934272980 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x563fc922aaf0 0x563fc922cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.425+0000 7f7dec3f9740 4 client.25595 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:45:47.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.425+0000 7f7dec3f9740 1 -- 172.21.15.28:0/1934272980 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563fc9182c10 con 0x563fc922aaf0 2024-04-23T14:45:47.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.426+0000 7f7de2a00640 1 --2- 172.21.15.28:0/1934272980 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x563fc922aaf0 0x563fc922cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.426+0000 7f7de2a00640 1 --2- 172.21.15.28:0/1934272980 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x563fc922aaf0 0x563fc922cee0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a10baf7f8854cda server_cookie=26c9c04c73b966b3 in_seq=0 out_seq=0 2024-04-23T14:45:47.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.426+0000 7f7dc3e00640 10 client.25595 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:45:47.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.427+0000 7f7dc3e00640 1 -- 172.21.15.28:0/1934272980 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563fc9182c10 con 0x563fc922aaf0 2024-04-23T14:45:47.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.427+0000 7f7dc3e00640 10 client.25595 handle_command_reply: tid=0 2024-04-23T14:45:47.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.427+0000 7f7dec3f9740 1 client.25595 shutdown 2024-04-23T14:45:47.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.427+0000 7f7dec3f9740 1 -- 172.21.15.28:0/1934272980 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563fc921aa20 msgr2=0x563fc9218400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:47.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.427+0000 7f7dec3f9740 1 --2- 172.21.15.28:0/1934272980 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563fc921aa20 0x563fc9218400 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc00f6a0 tx=0x7f7ddc004ab0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.427+0000 7f7dec3f9740 1 -- 172.21.15.28:0/1934272980 shutdown_connections 2024-04-23T14:45:47.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.427+0000 7f7dec3f9740 1 --2- 172.21.15.28:0/1934272980 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x563fc922aaf0 0x563fc922cee0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.427+0000 7f7dec3f9740 1 --2- 172.21.15.28:0/1934272980 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563fc921aa20 0x563fc9218400 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.427+0000 7f7dec3f9740 1 --2- 172.21.15.28:0/1934272980 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fc9210350 0x563fc9217ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.427+0000 7f7dec3f9740 1 --2- 172.21.15.28:0/1934272980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fc920fa30 0x563fc9217980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.427+0000 7f7dc3e00640 0 client.25595 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:45:47.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.427+0000 7f7dec3f9740 1 -- 172.21.15.28:0/1934272980 >> 172.21.15.28:0/1934272980 conn(0x563fc920bd50 msgr2=0x563fc920afd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:47.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.428+0000 7f7dec3f9740 1 -- 172.21.15.28:0/1934272980 shutdown_connections 2024-04-23T14:45:47.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.428+0000 7f7dec3f9740 1 -- 172.21.15.28:0/1934272980 wait complete. 2024-04-23T14:45:47.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.428+0000 7f7dec3f9740 20 client.25595 trim_cache size 0 max 16384 2024-04-23T14:45:47.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.449+0000 7f7dec3f9740 1 Processor -- start 2024-04-23T14:45:47.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.449+0000 7f7dec3f9740 1 -- start start 2024-04-23T14:45:47.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.450+0000 7f7dec3f9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fc920fa30 0x563fc920b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.450+0000 7f7dec3f9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563fc9210350 0x563fc92181d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.450+0000 7f7dec3f9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fc921aa20 0x563fc9212190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.450+0000 7f7dec3f9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7de41a9960 con 0x563fc920fa30 2024-04-23T14:45:47.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.450+0000 7f7dec3f9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7de41ac1c0 con 0x563fc9210350 2024-04-23T14:45:47.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.450+0000 7f7dec3f9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7de41a9fb0 con 0x563fc921aa20 2024-04-23T14:45:47.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.450+0000 7f7de3400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fc921aa20 0x563fc9212190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.450+0000 7f7de3400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fc921aa20 0x563fc9212190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:53088/0 (socket says 172.21.15.28:53088) 2024-04-23T14:45:47.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.450+0000 7f7de2000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563fc9210350 0x563fc92181d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.450+0000 7f7de3400640 1 -- 172.21.15.28:0/2097683088 learned_addr learned my addr 172.21.15.28:0/2097683088 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:47.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.450+0000 7f7de2a00640 1 --2- 172.21.15.28:0/2097683088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fc920fa30 0x563fc920b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.450+0000 7f7de3400640 1 -- 172.21.15.28:0/2097683088 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563fc9210350 msgr2=0x563fc92181d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:47.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.450+0000 7f7de3400640 1 --2- 172.21.15.28:0/2097683088 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563fc9210350 0x563fc92181d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.450+0000 7f7de3400640 1 -- 172.21.15.28:0/2097683088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fc920fa30 msgr2=0x563fc920b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:47.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.450+0000 7f7de3400640 1 --2- 172.21.15.28:0/2097683088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fc920fa30 0x563fc920b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.450+0000 7f7de3400640 1 -- 172.21.15.28:0/2097683088 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563fc922d250 con 0x563fc921aa20 2024-04-23T14:45:47.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.450+0000 7f7de2000640 1 --2- 172.21.15.28:0/2097683088 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563fc9210350 0x563fc92181d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:45:47.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.451+0000 7f7de3400640 1 --2- 172.21.15.28:0/2097683088 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fc921aa20 0x563fc9212190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc027530 tx=0x7f7ddc0278b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75b3a29e7225de4b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:47.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.451+0000 7f7dcea00640 1 -- 172.21.15.28:0/2097683088 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc01f230 con 0x563fc921aa20 2024-04-23T14:45:47.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.451+0000 7f7dcea00640 1 -- 172.21.15.28:0/2097683088 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ddc01f9a0 con 0x563fc921aa20 2024-04-23T14:45:47.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.451+0000 7f7dcea00640 1 -- 172.21.15.28:0/2097683088 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc01e680 con 0x563fc921aa20 2024-04-23T14:45:47.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.452+0000 7f7dec3f9740 1 -- 172.21.15.28:0/2097683088 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fc921aa20 msgr2=0x563fc9212190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:47.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.452+0000 7f7dec3f9740 1 --2- 172.21.15.28:0/2097683088 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fc921aa20 0x563fc9212190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc027530 tx=0x7f7ddc0278b0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.452+0000 7f7dec3f9740 1 -- 172.21.15.28:0/2097683088 shutdown_connections 2024-04-23T14:45:47.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.452+0000 7f7dec3f9740 1 --2- 172.21.15.28:0/2097683088 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563fc9210350 0x563fc92181d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.452+0000 7f7dec3f9740 1 --2- 172.21.15.28:0/2097683088 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fc921aa20 0x563fc9212190 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.452+0000 7f7dec3f9740 1 --2- 172.21.15.28:0/2097683088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fc920fa30 0x563fc920b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.452+0000 7f7dec3f9740 1 -- 172.21.15.28:0/2097683088 >> 172.21.15.28:0/2097683088 conn(0x563fc920bd50 msgr2=0x563fc92191b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:47.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.452+0000 7f7dec3f9740 1 -- 172.21.15.28:0/2097683088 shutdown_connections 2024-04-23T14:45:47.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.452+0000 7f7dec3f9740 1 -- 172.21.15.28:0/2097683088 wait complete. 2024-04-23T14:45:47.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.452+0000 7f7dec3f9740 1 -- start start 2024-04-23T14:45:47.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.453+0000 7f7dec3f9740 1 Processor -- start 2024-04-23T14:45:47.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.453+0000 7f7dec3f9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fc920fa30 0x563fc9214a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.453+0000 7f7dec3f9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563fc9210350 0x563fc9214f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.453+0000 7f7dec3f9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fc921aa20 0x563fc9215480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.453+0000 7f7dec3f9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7de41aa680 con 0x563fc921aa20 2024-04-23T14:45:47.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.453+0000 7f7dec3f9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7de41abaf0 con 0x563fc9210350 2024-04-23T14:45:47.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.453+0000 7f7dec3f9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7de41aad50 con 0x563fc920fa30 2024-04-23T14:45:47.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.453+0000 7f7dec3f9740 10 client.0 _ll_register_callbacks cb 0x7f7ddc009be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:45:47.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.453+0000 7f7de2000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563fc9210350 0x563fc9214f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.453+0000 7f7de2000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563fc9210350 0x563fc9214f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:53734/0 (socket says 172.21.15.28:53734) 2024-04-23T14:45:47.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.453+0000 7f7de2000640 1 -- 172.21.15.28:0/2801384422 learned_addr learned my addr 172.21.15.28:0/2801384422 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:45:47.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.453+0000 7f7de2a00640 1 --2- 172.21.15.28:0/2801384422 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fc920fa30 0x563fc9214a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.453+0000 7f7de3400640 1 --2- 172.21.15.28:0/2801384422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fc921aa20 0x563fc9215480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.454+0000 7f7de2000640 1 -- 172.21.15.28:0/2801384422 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fc920fa30 msgr2=0x563fc9214a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:47.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.454+0000 7f7de2000640 1 --2- 172.21.15.28:0/2801384422 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fc920fa30 0x563fc9214a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.454+0000 7f7de2000640 1 -- 172.21.15.28:0/2801384422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fc921aa20 msgr2=0x563fc9215480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:45:47.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.454+0000 7f7de2000640 1 --2- 172.21.15.28:0/2801384422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fc921aa20 0x563fc9215480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.454+0000 7f7de2000640 1 -- 172.21.15.28:0/2801384422 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563fc8e873f0 con 0x563fc9210350 2024-04-23T14:45:47.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.454+0000 7f7de3400640 1 --2- 172.21.15.28:0/2801384422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fc921aa20 0x563fc9215480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:45:47.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.454+0000 7f7de2a00640 1 --2- 172.21.15.28:0/2801384422 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fc920fa30 0x563fc9214a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:45:47.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.454+0000 7f7de2000640 1 --2- 172.21.15.28:0/2801384422 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563fc9210350 0x563fc9214f40 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7dd00c0f10 tx=0x7f7dd0005b10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ed43edda5873e74 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:45:47.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.454+0000 7f7dc3e00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:45:47.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.454+0000 7f7dc3e00640 1 -- 172.21.15.28:0/2801384422 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd00f8e60 con 0x563fc9210350 2024-04-23T14:45:47.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.454+0000 7f7dc3e00640 1 -- 172.21.15.28:0/2801384422 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f7dd00bf2b0 con 0x563fc9210350 2024-04-23T14:45:47.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.454+0000 7f7dec3f9740 1 -- 172.21.15.28:0/2801384422 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ddc01f9a0 con 0x563fc9210350 2024-04-23T14:45:47.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.454+0000 7f7dc3e00640 1 -- 172.21.15.28:0/2801384422 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f7dd00c0b70 con 0x563fc9210350 2024-04-23T14:45:47.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.455+0000 7f7dec3f9740 10 client.25601 fetch_fsmap learned FSMap version 20 2024-04-23T14:45:47.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.455+0000 7f7dec3f9740 1 -- 172.21.15.28:0/2801384422 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x563fc8e470f0 con 0x563fc9210350 2024-04-23T14:45:47.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.455+0000 7f7dc3e00640 1 -- 172.21.15.28:0/2801384422 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ddc01f9a0 con 0x563fc9210350 2024-04-23T14:45:47.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.455+0000 7f7dc3e00640 1 -- 172.21.15.28:0/2801384422 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f7dd0051060 con 0x563fc9210350 2024-04-23T14:45:47.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.455+0000 7f7dec3f9740 10 client.25601 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:45:47.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.455+0000 7f7dec3f9740 10 client.25601 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:45:47.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.455+0000 7f7dec3f9740 20 client.25601 populate_metadata read hostname 'smithi028' 2024-04-23T14:45:47.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.455+0000 7f7dec3f9740 1 --2- 172.21.15.28:0/2801384422 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x563fc9291660 0x563fc9293a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:45:47.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.455+0000 7f7dec3f9740 4 client.25601 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:45:47.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.455+0000 7f7dec3f9740 1 -- 172.21.15.28:0/2801384422 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563fc917ca80 con 0x563fc9291660 2024-04-23T14:45:47.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.455+0000 7f7de2a00640 1 --2- 172.21.15.28:0/2801384422 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x563fc9291660 0x563fc9293a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:45:47.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.456+0000 7f7de2a00640 1 --2- 172.21.15.28:0/2801384422 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x563fc9291660 0x563fc9293a50 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a251a248b1a9885 server_cookie=52ed8cbb07afbab7 in_seq=0 out_seq=0 2024-04-23T14:45:47.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.456+0000 7f7dc3e00640 10 client.25601 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:45:47.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.456+0000 7f7dc3e00640 1 -- 172.21.15.28:0/2801384422 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x563fc917ca80 con 0x563fc9291660 2024-04-23T14:45:47.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.456+0000 7f7dc3e00640 10 client.25601 handle_command_reply: tid=0 2024-04-23T14:45:47.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.456+0000 7f7dec3f9740 1 client.25601 shutdown 2024-04-23T14:45:47.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.456+0000 7f7dec3f9740 1 -- 172.21.15.28:0/2801384422 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563fc9210350 msgr2=0x563fc9214f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:47.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.456+0000 7f7dec3f9740 1 --2- 172.21.15.28:0/2801384422 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563fc9210350 0x563fc9214f40 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7dd00c0f10 tx=0x7f7dd0005b10 comp rx=0 tx=0).stop 2024-04-23T14:45:47.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.456+0000 7f7dec3f9740 1 -- 172.21.15.28:0/2801384422 shutdown_connections 2024-04-23T14:45:47.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.456+0000 7f7dec3f9740 1 --2- 172.21.15.28:0/2801384422 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x563fc9291660 0x563fc9293a50 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.456+0000 7f7dec3f9740 1 --2- 172.21.15.28:0/2801384422 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563fc9210350 0x563fc9214f40 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.456+0000 7f7dec3f9740 1 --2- 172.21.15.28:0/2801384422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fc921aa20 0x563fc9215480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.456+0000 7f7dec3f9740 1 --2- 172.21.15.28:0/2801384422 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fc920fa30 0x563fc9214a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.457+0000 7f7dc3e00640 0 client.25601 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:45:47.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.457+0000 7f7dec3f9740 1 -- 172.21.15.28:0/2801384422 >> 172.21.15.28:0/2801384422 conn(0x563fc920bd50 msgr2=0x563fc9218050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:47.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.457+0000 7f7dec3f9740 1 -- 172.21.15.28:0/2801384422 shutdown_connections 2024-04-23T14:45:47.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.457+0000 7f7dec3f9740 1 -- 172.21.15.28:0/2801384422 wait complete. 2024-04-23T14:45:47.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.457+0000 7f7dec3f9740 20 client.25601 trim_cache size 0 max 16384 2024-04-23T14:45:47.461 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:45:47.461 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-23T14:45:47.461 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-23T14:45:47.461 INFO:teuthology.orchestra.run.smithi028.stdout: "e273475a-9889-4ab0-9a0b-3f5909d3f63c": { 2024-04-23T14:45:47.461 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-23T14:45:47.461 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "e273475a-9889-4ab0-9a0b-3f5909d3f63c", 2024-04-23T14:45:47.461 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-23T14:45:47.462 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:45:47.462 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:45:47.462 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:45:47.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.458+0000 7f7de3e00640 1 -- 172.21.15.28:0/2773280541 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7dc80a3da0 msgr2=0x7f7dc80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:47.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.458+0000 7f7de3e00640 1 --2- 172.21.15.28:0/2773280541 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7dc80a3da0 0x7f7dc80a6260 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc009fd0 tx=0x7f7ddc009500 comp rx=0 tx=0).stop 2024-04-23T14:45:47.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.458+0000 7f7de3e00640 1 -- 172.21.15.28:0/2773280541 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7de41071c0 msgr2=0x7f7de4196ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:45:47.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.458+0000 7f7de3e00640 1 --2- 172.21.15.28:0/2773280541 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7de41071c0 0x7f7de4196ae0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7dd002e9e0 tx=0x7f7dd0034650 comp rx=0 tx=0).stop 2024-04-23T14:45:47.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.458+0000 7f7de3e00640 1 -- 172.21.15.28:0/2773280541 shutdown_connections 2024-04-23T14:45:47.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.458+0000 7f7de3e00640 1 --2- 172.21.15.28:0/2773280541 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7dc80a3da0 0x7f7dc80a6260 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.458+0000 7f7de3e00640 1 --2- 172.21.15.28:0/2773280541 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7de41071c0 0x7f7de4196ae0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.458+0000 7f7de3e00640 1 --2- 172.21.15.28:0/2773280541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7de419ae10 0x7f7de419b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.458+0000 7f7de3e00640 1 --2- 172.21.15.28:0/2773280541 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7de41020f0 0x7f7de41965a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:45:47.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.458+0000 7f7de3e00640 1 -- 172.21.15.28:0/2773280541 >> 172.21.15.28:0/2773280541 conn(0x7f7de40fd9d0 msgr2=0x7f7de40ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:45:47.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.458+0000 7f7de3e00640 1 -- 172.21.15.28:0/2773280541 shutdown_connections 2024-04-23T14:45:47.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:45:47.458+0000 7f7de3e00640 1 -- 172.21.15.28:0/2773280541 wait complete. 2024-04-23T14:45:47.463 INFO:tasks.workunit.client.0.smithi028.stdout: 1 7345 19.62 MB/sec warmup 3 sec latency 9.271 ms 2024-04-23T14:45:47.477 INFO:tasks.cephfs.filesystem:scrub status for tag:e273475a-9889-4ab0-9a0b-3f5909d3f63c - {'path': '/', 'tag': 'e273475a-9889-4ab0-9a0b-3f5909d3f63c', 'options': 'recursive,force'} 2024-04-23T14:45:48.458 INFO:tasks.workunit.client.0.smithi028.stdout: 1 9492 16.55 MB/sec warmup 4 sec latency 10.925 ms 2024-04-23T14:45:49.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:48 smithi105 ceph-mon[32555]: scrub queued for path: / 2024-04-23T14:45:49.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:48 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:45:49.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:48 smithi105 ceph-mon[32555]: scrub summary: active paths [/] 2024-04-23T14:45:49.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:48 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:45:49.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:48 smithi105 ceph-mon[32555]: pgmap v56: 97 pgs: 97 active+clean; 4.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 343 KiB/s wr, 41 op/s 2024-04-23T14:45:49.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:48 smithi038 ceph-mon[32348]: scrub queued for path: / 2024-04-23T14:45:49.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:48 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:45:49.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:48 smithi038 ceph-mon[32348]: scrub summary: active paths [/] 2024-04-23T14:45:49.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:48 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:45:49.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:48 smithi038 ceph-mon[32348]: pgmap v56: 97 pgs: 97 active+clean; 4.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 343 KiB/s wr, 41 op/s 2024-04-23T14:45:49.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:48 smithi028 ceph-mon[27810]: scrub queued for path: / 2024-04-23T14:45:49.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:48 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:45:49.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:48 smithi028 ceph-mon[27810]: scrub summary: active paths [/] 2024-04-23T14:45:49.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:48 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:45:49.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:48 smithi028 ceph-mon[27810]: pgmap v56: 97 pgs: 97 active+clean; 4.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 343 KiB/s wr, 41 op/s 2024-04-23T14:45:49.458 INFO:tasks.workunit.client.0.smithi028.stdout: 1 11381 15.73 MB/sec warmup 5 sec latency 24.164 ms 2024-04-23T14:45:50.459 INFO:tasks.workunit.client.0.smithi028.stdout: 1 13992 15.45 MB/sec warmup 6 sec latency 10.675 ms 2024-04-23T14:45:51.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:50 smithi105 ceph-mon[32555]: pgmap v57: 97 pgs: 97 active+clean; 4.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 343 KiB/s wr, 40 op/s 2024-04-23T14:45:51.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:50 smithi105 ceph-mon[32555]: scrub summary: idle 2024-04-23T14:45:51.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:50 smithi038 ceph-mon[32348]: pgmap v57: 97 pgs: 97 active+clean; 4.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 343 KiB/s wr, 40 op/s 2024-04-23T14:45:51.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:50 smithi038 ceph-mon[32348]: scrub summary: idle 2024-04-23T14:45:51.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:50 smithi028 ceph-mon[27810]: pgmap v57: 97 pgs: 97 active+clean; 4.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 343 KiB/s wr, 40 op/s 2024-04-23T14:45:51.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:50 smithi028 ceph-mon[27810]: scrub summary: idle 2024-04-23T14:45:51.459 INFO:tasks.workunit.client.0.smithi028.stdout: 1 15754 14.65 MB/sec warmup 7 sec latency 10.954 ms 2024-04-23T14:45:52.459 INFO:tasks.workunit.client.0.smithi028.stdout: 1 18220 14.57 MB/sec warmup 8 sec latency 8.117 ms 2024-04-23T14:45:53.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:52 smithi105 ceph-mon[32555]: pgmap v58: 97 pgs: 97 active+clean; 4.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 343 KiB/s wr, 40 op/s 2024-04-23T14:45:53.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:52 smithi038 ceph-mon[32348]: pgmap v58: 97 pgs: 97 active+clean; 4.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 343 KiB/s wr, 40 op/s 2024-04-23T14:45:53.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:52 smithi028 ceph-mon[27810]: pgmap v58: 97 pgs: 97 active+clean; 4.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 343 KiB/s wr, 40 op/s 2024-04-23T14:45:53.459 INFO:tasks.workunit.client.0.smithi028.stdout: 1 20792 14.35 MB/sec warmup 9 sec latency 12.141 ms 2024-04-23T14:45:54.459 INFO:tasks.workunit.client.0.smithi028.stdout: 1 22709 14.07 MB/sec warmup 10 sec latency 35.805 ms 2024-04-23T14:45:55.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:54 smithi105 ceph-mon[32555]: pgmap v59: 97 pgs: 97 active+clean; 10 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 891 KiB/s wr, 106 op/s 2024-04-23T14:45:55.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:54 smithi038 ceph-mon[32348]: pgmap v59: 97 pgs: 97 active+clean; 10 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 891 KiB/s wr, 106 op/s 2024-04-23T14:45:55.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:54 smithi028 ceph-mon[27810]: pgmap v59: 97 pgs: 97 active+clean; 10 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 891 KiB/s wr, 106 op/s 2024-04-23T14:45:55.459 INFO:tasks.workunit.client.0.smithi028.stdout: 1 25246 14.07 MB/sec warmup 11 sec latency 13.358 ms 2024-04-23T14:45:56.459 INFO:tasks.workunit.client.0.smithi028.stdout: 1 27469 13.57 MB/sec warmup 12 sec latency 12.290 ms 2024-04-23T14:45:57.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:56 smithi105 ceph-mon[32555]: pgmap v60: 97 pgs: 97 active+clean; 10 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 890 KiB/s wr, 105 op/s 2024-04-23T14:45:57.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:56 smithi038 ceph-mon[32348]: pgmap v60: 97 pgs: 97 active+clean; 10 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 890 KiB/s wr, 105 op/s 2024-04-23T14:45:57.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:56 smithi028 ceph-mon[27810]: pgmap v60: 97 pgs: 97 active+clean; 10 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 890 KiB/s wr, 105 op/s 2024-04-23T14:45:57.459 INFO:tasks.workunit.client.0.smithi028.stdout: 1 29406 13.45 MB/sec warmup 13 sec latency 11.612 ms 2024-04-23T14:45:58.460 INFO:tasks.workunit.client.0.smithi028.stdout: 1 32203 13.77 MB/sec warmup 14 sec latency 9.015 ms 2024-04-23T14:45:59.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:45:58 smithi105 ceph-mon[32555]: pgmap v61: 97 pgs: 97 active+clean; 15 MiB data, 434 MiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 1.4 MiB/s wr, 171 op/s 2024-04-23T14:45:59.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:45:58 smithi038 ceph-mon[32348]: pgmap v61: 97 pgs: 97 active+clean; 15 MiB data, 434 MiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 1.4 MiB/s wr, 171 op/s 2024-04-23T14:45:59.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:45:58 smithi028 ceph-mon[27810]: pgmap v61: 97 pgs: 97 active+clean; 15 MiB data, 434 MiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 1.4 MiB/s wr, 171 op/s 2024-04-23T14:45:59.460 INFO:tasks.workunit.client.0.smithi028.stdout: 1 34365 13.34 MB/sec warmup 15 sec latency 15.758 ms 2024-04-23T14:46:00.460 INFO:tasks.workunit.client.0.smithi028.stdout: 1 36229 13.28 MB/sec warmup 16 sec latency 12.273 ms 2024-04-23T14:46:01.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:00 smithi105 ceph-mon[32555]: pgmap v62: 97 pgs: 97 active+clean; 15 MiB data, 434 MiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 1.1 MiB/s wr, 132 op/s 2024-04-23T14:46:01.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:00 smithi038 ceph-mon[32348]: pgmap v62: 97 pgs: 97 active+clean; 15 MiB data, 434 MiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 1.1 MiB/s wr, 132 op/s 2024-04-23T14:46:01.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:00 smithi028 ceph-mon[27810]: pgmap v62: 97 pgs: 97 active+clean; 15 MiB data, 434 MiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 1.1 MiB/s wr, 132 op/s 2024-04-23T14:46:01.460 INFO:tasks.workunit.client.0.smithi028.stdout: 1 37369 12.80 MB/sec warmup 17 sec latency 64.696 ms 2024-04-23T14:46:02.460 INFO:tasks.workunit.client.0.smithi028.stdout: 1 39486 12.85 MB/sec warmup 18 sec latency 10.170 ms 2024-04-23T14:46:03.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:02 smithi105 ceph-mon[32555]: pgmap v63: 97 pgs: 97 active+clean; 15 MiB data, 434 MiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 1.1 MiB/s wr, 132 op/s 2024-04-23T14:46:03.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:02 smithi038 ceph-mon[32348]: pgmap v63: 97 pgs: 97 active+clean; 15 MiB data, 434 MiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 1.1 MiB/s wr, 132 op/s 2024-04-23T14:46:03.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:02 smithi028 ceph-mon[27810]: pgmap v63: 97 pgs: 97 active+clean; 15 MiB data, 434 MiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 1.1 MiB/s wr, 132 op/s 2024-04-23T14:46:03.460 INFO:tasks.workunit.client.0.smithi028.stdout: 1 41525 12.58 MB/sec warmup 19 sec latency 12.243 ms 2024-04-23T14:46:04.461 INFO:tasks.workunit.client.0.smithi028.stdout: 1 43172 12.52 MB/sec warmup 20 sec latency 10.774 ms 2024-04-23T14:46:05.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:04 smithi105 ceph-mon[32555]: pgmap v64: 97 pgs: 97 active+clean; 21 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 844 KiB/s rd, 1.7 MiB/s wr, 226 op/s 2024-04-23T14:46:05.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:04 smithi038 ceph-mon[32348]: pgmap v64: 97 pgs: 97 active+clean; 21 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 844 KiB/s rd, 1.7 MiB/s wr, 226 op/s 2024-04-23T14:46:05.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:04 smithi028 ceph-mon[27810]: pgmap v64: 97 pgs: 97 active+clean; 21 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 844 KiB/s rd, 1.7 MiB/s wr, 226 op/s 2024-04-23T14:46:05.460 INFO:tasks.workunit.client.0.smithi028.stdout: 1 45226 12.32 MB/sec warmup 21 sec latency 23.013 ms 2024-04-23T14:46:06.461 INFO:tasks.workunit.client.0.smithi028.stdout: 1 46870 12.26 MB/sec warmup 22 sec latency 16.004 ms 2024-04-23T14:46:07.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:06 smithi105 ceph-mon[32555]: pgmap v65: 97 pgs: 97 active+clean; 21 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 1.2 MiB/s wr, 160 op/s 2024-04-23T14:46:07.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:06 smithi038 ceph-mon[32348]: pgmap v65: 97 pgs: 97 active+clean; 21 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 1.2 MiB/s wr, 160 op/s 2024-04-23T14:46:07.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:06 smithi028 ceph-mon[27810]: pgmap v65: 97 pgs: 97 active+clean; 21 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 1.2 MiB/s wr, 160 op/s 2024-04-23T14:46:07.461 INFO:tasks.workunit.client.0.smithi028.stdout: 1 49039 12.23 MB/sec warmup 23 sec latency 10.644 ms 2024-04-23T14:46:08.461 INFO:tasks.workunit.client.0.smithi028.stdout: 1 50426 12.04 MB/sec warmup 24 sec latency 16.175 ms 2024-04-23T14:46:09.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:08 smithi105 ceph-mon[32555]: pgmap v66: 97 pgs: 97 active+clean; 26 MiB data, 495 MiB used, 1.0 TiB / 1.0 TiB avail; 848 KiB/s rd, 1.7 MiB/s wr, 218 op/s 2024-04-23T14:46:09.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:08 smithi038 ceph-mon[32348]: pgmap v66: 97 pgs: 97 active+clean; 26 MiB data, 495 MiB used, 1.0 TiB / 1.0 TiB avail; 848 KiB/s rd, 1.7 MiB/s wr, 218 op/s 2024-04-23T14:46:09.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:08 smithi028 ceph-mon[27810]: pgmap v66: 97 pgs: 97 active+clean; 26 MiB data, 495 MiB used, 1.0 TiB / 1.0 TiB avail; 848 KiB/s rd, 1.7 MiB/s wr, 218 op/s 2024-04-23T14:46:09.461 INFO:tasks.workunit.client.0.smithi028.stdout: 1 52604 12.05 MB/sec warmup 25 sec latency 10.776 ms 2024-04-23T14:46:10.461 INFO:tasks.workunit.client.0.smithi028.stdout: 1 54027 11.85 MB/sec warmup 26 sec latency 35.848 ms 2024-04-23T14:46:11.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:10 smithi105 ceph-mon[32555]: pgmap v67: 97 pgs: 97 active+clean; 26 MiB data, 495 MiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 1.1 MiB/s wr, 151 op/s 2024-04-23T14:46:11.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:10 smithi038 ceph-mon[32348]: pgmap v67: 97 pgs: 97 active+clean; 26 MiB data, 495 MiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 1.1 MiB/s wr, 151 op/s 2024-04-23T14:46:11.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:10 smithi028 ceph-mon[27810]: pgmap v67: 97 pgs: 97 active+clean; 26 MiB data, 495 MiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 1.1 MiB/s wr, 151 op/s 2024-04-23T14:46:11.461 INFO:tasks.workunit.client.0.smithi028.stdout: 1 56425 11.89 MB/sec warmup 27 sec latency 11.621 ms 2024-04-23T14:46:12.462 INFO:tasks.workunit.client.0.smithi028.stdout: 1 57760 11.71 MB/sec warmup 28 sec latency 22.439 ms 2024-04-23T14:46:13.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:12 smithi105 ceph-mon[32555]: pgmap v68: 97 pgs: 97 active+clean; 26 MiB data, 495 MiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 1.1 MiB/s wr, 151 op/s 2024-04-23T14:46:13.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:12 smithi038 ceph-mon[32348]: pgmap v68: 97 pgs: 97 active+clean; 26 MiB data, 495 MiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 1.1 MiB/s wr, 151 op/s 2024-04-23T14:46:13.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:12 smithi028 ceph-mon[27810]: pgmap v68: 97 pgs: 97 active+clean; 26 MiB data, 495 MiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 1.1 MiB/s wr, 151 op/s 2024-04-23T14:46:13.462 INFO:tasks.workunit.client.0.smithi028.stdout: 1 60402 11.88 MB/sec warmup 29 sec latency 9.478 ms 2024-04-23T14:46:14.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:13 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:46:14.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:13 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:46:14.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:13 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:46:14.462 INFO:tasks.workunit.client.0.smithi028.stdout: 1 61848 11.70 MB/sec warmup 30 sec latency 14.957 ms 2024-04-23T14:46:15.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:14 smithi105 ceph-mon[32555]: pgmap v69: 97 pgs: 97 active+clean; 31 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 878 KiB/s rd, 1.6 MiB/s wr, 207 op/s 2024-04-23T14:46:15.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:14 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:46:15.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:14 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:46:15.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:14 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:46:15.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:14 smithi038 ceph-mon[32348]: pgmap v69: 97 pgs: 97 active+clean; 31 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 878 KiB/s rd, 1.6 MiB/s wr, 207 op/s 2024-04-23T14:46:15.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:14 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:46:15.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:14 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:46:15.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:14 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:46:15.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:14 smithi028 ceph-mon[27810]: pgmap v69: 97 pgs: 97 active+clean; 31 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 878 KiB/s rd, 1.6 MiB/s wr, 207 op/s 2024-04-23T14:46:15.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:14 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:46:15.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:14 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:46:15.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:14 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:46:15.463 INFO:tasks.workunit.client.0.smithi028.stdout: 1 63960 11.73 MB/sec warmup 31 sec latency 30.336 ms 2024-04-23T14:46:16.464 INFO:tasks.workunit.client.0.smithi028.stdout: 1 64843 11.44 MB/sec warmup 32 sec latency 56.924 ms 2024-04-23T14:46:17.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:16 smithi038 ceph-mon[32348]: pgmap v70: 97 pgs: 97 active+clean; 31 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 1.0 MiB/s wr, 113 op/s 2024-04-23T14:46:17.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:16 smithi028 ceph-mon[27810]: pgmap v70: 97 pgs: 97 active+clean; 31 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 1.0 MiB/s wr, 113 op/s 2024-04-23T14:46:17.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:16 smithi105 ceph-mon[32555]: pgmap v70: 97 pgs: 97 active+clean; 31 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 1.0 MiB/s wr, 113 op/s 2024-04-23T14:46:17.464 INFO:tasks.workunit.client.0.smithi028.stdout: 1 66868 11.45 MB/sec warmup 33 sec latency 18.090 ms 2024-04-23T14:46:17.478 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:46:17.478 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:46:17.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.715+0000 7f10fee00640 1 -- 172.21.15.28:0/3391094693 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10f8109580 msgr2=0x7f10f8111150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:17.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.715+0000 7f10fee00640 1 --2- 172.21.15.28:0/3391094693 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10f8109580 0x7f10f8111150 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f10f0009be0 tx=0x7f10f002a6c0 comp rx=0 tx=0).stop 2024-04-23T14:46:17.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.715+0000 7f10fee00640 1 -- 172.21.15.28:0/3391094693 shutdown_connections 2024-04-23T14:46:17.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.715+0000 7f10fee00640 1 --2- 172.21.15.28:0/3391094693 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10f8109580 0x7f10f8111150 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:17.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.715+0000 7f10fee00640 1 --2- 172.21.15.28:0/3391094693 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f10f8104e60 0x7f10f8108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:17.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.715+0000 7f10fee00640 1 --2- 172.21.15.28:0/3391094693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10f81044b0 0x7f10f8104890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:17.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.715+0000 7f10fee00640 1 -- 172.21.15.28:0/3391094693 >> 172.21.15.28:0/3391094693 conn(0x7f10f80ffd90 msgr2=0x7f10f81021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:17.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.715+0000 7f10fee00640 1 -- 172.21.15.28:0/3391094693 shutdown_connections 2024-04-23T14:46:17.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.715+0000 7f10fee00640 1 -- 172.21.15.28:0/3391094693 wait complete. 2024-04-23T14:46:17.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.716+0000 7f10fee00640 1 Processor -- start 2024-04-23T14:46:17.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.716+0000 7f10fee00640 1 -- start start 2024-04-23T14:46:17.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.716+0000 7f10fee00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f10f81044b0 0x7f10f81a07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:17.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.717+0000 7f10fee00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10f8109580 0x7f10f81a0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:17.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.717+0000 7f10fee00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10f819a9a0 0x7f10f819ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:17.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.717+0000 7f10fee00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f10f8114620 con 0x7f10f8109580 2024-04-23T14:46:17.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.717+0000 7f10fee00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f10f81144a0 con 0x7f10f819a9a0 2024-04-23T14:46:17.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.717+0000 7f10fee00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f10f81147a0 con 0x7f10f81044b0 2024-04-23T14:46:17.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.717+0000 7f10fce00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10f819a9a0 0x7f10f819ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:17.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.717+0000 7f10fce00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10f819a9a0 0x7f10f819ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60078/0 (socket says 172.21.15.28:60078) 2024-04-23T14:46:17.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.717+0000 7f10fce00640 1 -- 172.21.15.28:0/2261922438 learned_addr learned my addr 172.21.15.28:0/2261922438 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:17.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.717+0000 7f10f7e00640 1 --2- 172.21.15.28:0/2261922438 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f10f81044b0 0x7f10f81a07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:17.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.717+0000 7f10efe00640 1 --2- 172.21.15.28:0/2261922438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10f8109580 0x7f10f81a0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:17.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.718+0000 7f10f7e00640 1 -- 172.21.15.28:0/2261922438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10f819a9a0 msgr2=0x7f10f819ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:17.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.718+0000 7f10f7e00640 1 --2- 172.21.15.28:0/2261922438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10f819a9a0 0x7f10f819ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:17.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.718+0000 7f10f7e00640 1 -- 172.21.15.28:0/2261922438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10f8109580 msgr2=0x7f10f81a0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:17.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.718+0000 7f10f7e00640 1 --2- 172.21.15.28:0/2261922438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10f8109580 0x7f10f81a0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:17.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.718+0000 7f10f7e00640 1 -- 172.21.15.28:0/2261922438 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10f819b4c0 con 0x7f10f81044b0 2024-04-23T14:46:17.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.718+0000 7f10fce00640 1 --2- 172.21.15.28:0/2261922438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10f819a9a0 0x7f10f819ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:46:17.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.718+0000 7f10f7e00640 1 --2- 172.21.15.28:0/2261922438 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f10f81044b0 0x7f10f81a07a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f10e800e9f0 tx=0x7f10e800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:17.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.718+0000 7f10f5600640 1 -- 172.21.15.28:0/2261922438 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e800ccc0 con 0x7f10f81044b0 2024-04-23T14:46:17.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.718+0000 7f10f5600640 1 -- 172.21.15.28:0/2261922438 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10e8010910 con 0x7f10f81044b0 2024-04-23T14:46:17.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.718+0000 7f10f5600640 1 -- 172.21.15.28:0/2261922438 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e8018960 con 0x7f10f81044b0 2024-04-23T14:46:17.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.718+0000 7f10fee00640 1 -- 172.21.15.28:0/2261922438 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10f819b7b0 con 0x7f10f81044b0 2024-04-23T14:46:17.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.718+0000 7f10fee00640 1 -- 172.21.15.28:0/2261922438 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10f81a7960 con 0x7f10f81044b0 2024-04-23T14:46:17.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.722+0000 7f10f5600640 1 -- 172.21.15.28:0/2261922438 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f10e8010ac0 con 0x7f10f81044b0 2024-04-23T14:46:17.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.722+0000 7f1101c32740 1 Processor -- start 2024-04-23T14:46:17.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.722+0000 7f1101c32740 1 -- start start 2024-04-23T14:46:17.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.723+0000 7f1101c32740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ad5829860 0x558ad5829c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:17.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.723+0000 7f1101c32740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558ad582a180 0x558ad5834310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:17.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.723+0000 7f1101c32740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ad5834850 0x558ad5836c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:17.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.723+0000 7f1101c32740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558ad578f1f0 con 0x558ad5829860 2024-04-23T14:46:17.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.723+0000 7f1101c32740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558ad5735350 con 0x558ad582a180 2024-04-23T14:46:17.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.723+0000 7f1101c32740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558ad578b1b0 con 0x558ad5834850 2024-04-23T14:46:17.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.723+0000 7f10f7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558ad582a180 0x558ad5834310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:17.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.723+0000 7f10fce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ad5829860 0x558ad5829c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:17.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.723+0000 7f10f7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558ad582a180 0x558ad5834310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60088/0 (socket says 172.21.15.28:60088) 2024-04-23T14:46:17.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.723+0000 7f10f7e00640 1 -- 172.21.15.28:0/2379557027 learned_addr learned my addr 172.21.15.28:0/2379557027 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:17.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.723+0000 7f10efe00640 1 --2- 172.21.15.28:0/2379557027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ad5834850 0x558ad5836c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:17.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.724+0000 7f10f7e00640 1 -- 172.21.15.28:0/2379557027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ad5834850 msgr2=0x558ad5836c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:17.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.724+0000 7f10f7e00640 1 --2- 172.21.15.28:0/2379557027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ad5834850 0x558ad5836c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:17.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.724+0000 7f10f7e00640 1 -- 172.21.15.28:0/2379557027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ad5829860 msgr2=0x558ad5829c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:17.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.724+0000 7f10f7e00640 1 --2- 172.21.15.28:0/2379557027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ad5829860 0x558ad5829c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:17.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.724+0000 7f10f7e00640 1 -- 172.21.15.28:0/2379557027 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ad5789f50 con 0x558ad582a180 2024-04-23T14:46:17.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.724+0000 7f10fce00640 1 --2- 172.21.15.28:0/2379557027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ad5829860 0x558ad5829c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:46:17.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.724+0000 7f10efe00640 1 --2- 172.21.15.28:0/2379557027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ad5834850 0x558ad5836c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:46:17.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.724+0000 7f10f7e00640 1 --2- 172.21.15.28:0/2379557027 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558ad582a180 0x558ad5834310 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f10e80ce250 tx=0x7f10e80d96d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d5f76539725f289 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:17.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.724+0000 7f10f5600640 1 --2- 172.21.15.28:0/2261922438 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f10dc0a3da0 0x7f10dc0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:17.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.724+0000 7f10ed600640 1 -- 172.21.15.28:0/2379557027 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e80e0020 con 0x558ad582a180 2024-04-23T14:46:17.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.725+0000 7f10ed600640 1 -- 172.21.15.28:0/2379557027 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10e8010750 con 0x558ad582a180 2024-04-23T14:46:17.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.725+0000 7f10ed600640 1 -- 172.21.15.28:0/2379557027 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e809c4c0 con 0x558ad582a180 2024-04-23T14:46:17.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.725+0000 7f10f5600640 1 -- 172.21.15.28:0/2261922438 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f10e80d0c10 con 0x7f10f81044b0 2024-04-23T14:46:17.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.725+0000 7f10fce00640 1 --2- 172.21.15.28:0/2261922438 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f10dc0a3da0 0x7f10dc0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:17.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.725+0000 7f1101c32740 1 -- 172.21.15.28:0/2379557027 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558ad582a180 msgr2=0x558ad5834310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:17.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.725+0000 7f1101c32740 1 --2- 172.21.15.28:0/2379557027 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558ad582a180 0x558ad5834310 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f10e80ce250 tx=0x7f10e80d96d0 comp rx=0 tx=0).stop 2024-04-23T14:46:17.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.726+0000 7f1101c32740 1 -- 172.21.15.28:0/2379557027 shutdown_connections 2024-04-23T14:46:17.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.726+0000 7f1101c32740 1 --2- 172.21.15.28:0/2379557027 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558ad582a180 0x558ad5834310 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:17.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.726+0000 7f1101c32740 1 --2- 172.21.15.28:0/2379557027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ad5834850 0x558ad5836c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:17.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.726+0000 7f1101c32740 1 --2- 172.21.15.28:0/2379557027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ad5829860 0x558ad5829c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:17.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.726+0000 7f1101c32740 1 -- 172.21.15.28:0/2379557027 >> 172.21.15.28:0/2379557027 conn(0x558ad5825b80 msgr2=0x558ad5825f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:17.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.726+0000 7f10fce00640 1 --2- 172.21.15.28:0/2261922438 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f10dc0a3da0 0x7f10dc0a6260 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f10f0004af0 tx=0x7f10f0009900 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:17.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.726+0000 7f1101c32740 1 -- 172.21.15.28:0/2379557027 shutdown_connections 2024-04-23T14:46:17.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.726+0000 7f1101c32740 1 -- 172.21.15.28:0/2379557027 wait complete. 2024-04-23T14:46:17.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.726+0000 7f1101c32740 1 -- start start 2024-04-23T14:46:17.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.726+0000 7f1101c32740 1 Processor -- start 2024-04-23T14:46:17.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.727+0000 7f1101c32740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ad5829860 0x558ad5831740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:17.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.727+0000 7f1101c32740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558ad582a180 0x558ad5831c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:17.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.727+0000 7f1101c32740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ad5834850 0x558ad58321c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:17.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.727+0000 7f1101c32740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558ad5837640 con 0x558ad5829860 2024-04-23T14:46:17.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.727+0000 7f1101c32740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558ad5748690 con 0x558ad582a180 2024-04-23T14:46:17.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.727+0000 7f1101c32740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558ad58377c0 con 0x558ad5834850 2024-04-23T14:46:17.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.727+0000 7f1101c32740 10 client.0 _ll_register_callbacks cb 0x558ad5784650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:46:17.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.727+0000 7f10efe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558ad582a180 0x558ad5831c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:17.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.727+0000 7f10f7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ad5829860 0x558ad5831740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:17.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.727+0000 7f10efe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558ad582a180 0x558ad5831c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60100/0 (socket says 172.21.15.28:60100) 2024-04-23T14:46:17.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.727+0000 7f10efe00640 1 -- 172.21.15.28:0/2021217744 learned_addr learned my addr 172.21.15.28:0/2021217744 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:17.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.727+0000 7f10fce00640 1 --2- 172.21.15.28:0/2021217744 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ad5834850 0x558ad58321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:17.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.728+0000 7f10efe00640 1 -- 172.21.15.28:0/2021217744 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ad5834850 msgr2=0x558ad58321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:17.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.728+0000 7f10efe00640 1 --2- 172.21.15.28:0/2021217744 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ad5834850 0x558ad58321c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:17.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.728+0000 7f10efe00640 1 -- 172.21.15.28:0/2021217744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ad5829860 msgr2=0x558ad5831740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:17.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.728+0000 7f10efe00640 1 --2- 172.21.15.28:0/2021217744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ad5829860 0x558ad5831740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:17.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.728+0000 7f10efe00640 1 -- 172.21.15.28:0/2021217744 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ad5747920 con 0x558ad582a180 2024-04-23T14:46:17.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.728+0000 7f10fce00640 1 --2- 172.21.15.28:0/2021217744 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ad5834850 0x558ad58321c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:46:17.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.728+0000 7f10efe00640 1 --2- 172.21.15.28:0/2021217744 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558ad582a180 0x558ad5831c80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f10e0012750 tx=0x7f10e0012c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ffbf85b26ecad591 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:17.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.728+0000 7f10e6000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:46:17.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.728+0000 7f10e6000640 1 -- 172.21.15.28:0/2021217744 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e0010dc0 con 0x558ad582a180 2024-04-23T14:46:17.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.728+0000 7f10e6000640 1 -- 172.21.15.28:0/2021217744 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f10e0004020 con 0x558ad582a180 2024-04-23T14:46:17.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.728+0000 7f1101c32740 1 -- 172.21.15.28:0/2021217744 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558ad579cc10 con 0x558ad582a180 2024-04-23T14:46:17.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.729+0000 7f10e6000640 1 -- 172.21.15.28:0/2021217744 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f10e0004220 con 0x558ad582a180 2024-04-23T14:46:17.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.729+0000 7f1101c32740 10 client.25613 fetch_fsmap learned FSMap version 20 2024-04-23T14:46:17.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.729+0000 7f1101c32740 1 -- 172.21.15.28:0/2021217744 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x558ad556b7d0 con 0x558ad582a180 2024-04-23T14:46:17.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.729+0000 7f10e6000640 1 -- 172.21.15.28:0/2021217744 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558ad579cc10 con 0x558ad582a180 2024-04-23T14:46:17.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.729+0000 7f10e6000640 1 -- 172.21.15.28:0/2021217744 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f10e0004a90 con 0x558ad582a180 2024-04-23T14:46:17.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.729+0000 7f1101c32740 10 client.25613 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:46:17.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.729+0000 7f1101c32740 10 client.25613 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:46:17.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.729+0000 7f1101c32740 20 client.25613 populate_metadata read hostname 'smithi028' 2024-04-23T14:46:17.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.729+0000 7f1101c32740 1 --2- 172.21.15.28:0/2021217744 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558ad5844cf0 0x558ad5847160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:17.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.729+0000 7f1101c32740 4 client.25613 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:46:17.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.729+0000 7f1101c32740 1 -- 172.21.15.28:0/2021217744 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ad5796a80 con 0x558ad5844cf0 2024-04-23T14:46:17.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.730+0000 7f10f7e00640 1 --2- 172.21.15.28:0/2021217744 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558ad5844cf0 0x558ad5847160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:17.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.730+0000 7f10f7e00640 1 --2- 172.21.15.28:0/2021217744 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558ad5844cf0 0x558ad5847160 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1cc6ffa9a65a4450 server_cookie=9faf71596fb9ffb6 in_seq=0 out_seq=0 2024-04-23T14:46:17.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.730+0000 7f10e6000640 10 client.25613 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:46:17.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.732+0000 7f10e6000640 1 -- 172.21.15.28:0/2021217744 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558ad5796a80 con 0x558ad5844cf0 2024-04-23T14:46:17.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.732+0000 7f10e6000640 10 client.25613 handle_command_reply: tid=0 2024-04-23T14:46:17.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.732+0000 7f1101c32740 1 client.25613 shutdown 2024-04-23T14:46:17.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.732+0000 7f1101c32740 1 -- 172.21.15.28:0/2021217744 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558ad582a180 msgr2=0x558ad5831c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:17.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.732+0000 7f1101c32740 1 --2- 172.21.15.28:0/2021217744 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558ad582a180 0x558ad5831c80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f10e0012750 tx=0x7f10e0012c10 comp rx=0 tx=0).stop 2024-04-23T14:46:17.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.733+0000 7f1101c32740 1 -- 172.21.15.28:0/2021217744 shutdown_connections 2024-04-23T14:46:17.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.733+0000 7f1101c32740 1 --2- 172.21.15.28:0/2021217744 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558ad5844cf0 0x558ad5847160 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:17.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.733+0000 7f1101c32740 1 --2- 172.21.15.28:0/2021217744 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558ad582a180 0x558ad5831c80 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:17.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.733+0000 7f1101c32740 1 --2- 172.21.15.28:0/2021217744 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ad5834850 0x558ad58321c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:17.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.733+0000 7f1101c32740 1 --2- 172.21.15.28:0/2021217744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ad5829860 0x558ad5831740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:17.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.733+0000 7f1101c32740 1 -- 172.21.15.28:0/2021217744 >> 172.21.15.28:0/2021217744 conn(0x558ad5825b80 msgr2=0x558ad582b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:17.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.733+0000 7f10e6000640 0 client.25613 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:46:17.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.733+0000 7f1101c32740 1 -- 172.21.15.28:0/2021217744 shutdown_connections 2024-04-23T14:46:17.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.733+0000 7f1101c32740 1 -- 172.21.15.28:0/2021217744 wait complete. 2024-04-23T14:46:17.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.733+0000 7f1101c32740 20 client.25613 trim_cache size 0 max 16384 2024-04-23T14:46:17.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.757+0000 7f1101c32740 1 Processor -- start 2024-04-23T14:46:17.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.757+0000 7f1101c32740 1 -- start start 2024-04-23T14:46:17.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.757+0000 7f1101c32740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ad5829860 0x558ad5830680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:17.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.757+0000 7f1101c32740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ad582a180 0x558ad5840500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:17.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.757+0000 7f1101c32740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558ad5834850 0x558ad584ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:17.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.757+0000 7f1101c32740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f10f81ae000 con 0x558ad5829860 2024-04-23T14:46:17.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.757+0000 7f1101c32740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f10f81b0880 con 0x558ad5834850 2024-04-23T14:46:17.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.757+0000 7f1101c32740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f10f81ae670 con 0x558ad582a180 2024-04-23T14:46:17.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.758+0000 7f10fce00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558ad5834850 0x558ad584ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:17.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.758+0000 7f10f7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ad5829860 0x558ad5830680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:17.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.758+0000 7f10fce00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558ad5834850 0x558ad584ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60104/0 (socket says 172.21.15.28:60104) 2024-04-23T14:46:17.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.758+0000 7f10fce00640 1 -- 172.21.15.28:0/2941579394 learned_addr learned my addr 172.21.15.28:0/2941579394 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:17.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.758+0000 7f10efe00640 1 --2- 172.21.15.28:0/2941579394 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ad582a180 0x558ad5840500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:17.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.758+0000 7f10fce00640 1 -- 172.21.15.28:0/2941579394 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ad582a180 msgr2=0x558ad5840500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:17.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.758+0000 7f10fce00640 1 --2- 172.21.15.28:0/2941579394 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ad582a180 0x558ad5840500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:17.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.758+0000 7f10fce00640 1 -- 172.21.15.28:0/2941579394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ad5829860 msgr2=0x558ad5830680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:17.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.758+0000 7f10fce00640 1 --2- 172.21.15.28:0/2941579394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ad5829860 0x558ad5830680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:17.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.758+0000 7f10fce00640 1 -- 172.21.15.28:0/2941579394 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ad54a13f0 con 0x558ad5834850 2024-04-23T14:46:17.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.758+0000 7f10fce00640 1 --2- 172.21.15.28:0/2941579394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558ad5834850 0x558ad584ca70 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f10f0030790 tx=0x7f10f003f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a515a571cbed9dd5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:17.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.759+0000 7f10ecc00640 1 -- 172.21.15.28:0/2941579394 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f0050070 con 0x558ad5834850 2024-04-23T14:46:17.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.759+0000 7f10ecc00640 1 -- 172.21.15.28:0/2941579394 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10f0004690 con 0x558ad5834850 2024-04-23T14:46:17.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.759+0000 7f10ecc00640 1 -- 172.21.15.28:0/2941579394 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f0036040 con 0x558ad5834850 2024-04-23T14:46:17.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.760+0000 7f1101c32740 1 -- 172.21.15.28:0/2941579394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558ad5834850 msgr2=0x558ad584ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:17.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.760+0000 7f1101c32740 1 --2- 172.21.15.28:0/2941579394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558ad5834850 0x558ad584ca70 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f10f0030790 tx=0x7f10f003f000 comp rx=0 tx=0).stop 2024-04-23T14:46:17.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.760+0000 7f1101c32740 1 -- 172.21.15.28:0/2941579394 shutdown_connections 2024-04-23T14:46:17.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.760+0000 7f1101c32740 1 --2- 172.21.15.28:0/2941579394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558ad5834850 0x558ad584ca70 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:17.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.760+0000 7f1101c32740 1 --2- 172.21.15.28:0/2941579394 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ad582a180 0x558ad5840500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:17.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.760+0000 7f1101c32740 1 --2- 172.21.15.28:0/2941579394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ad5829860 0x558ad5830680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:17.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.760+0000 7f1101c32740 1 -- 172.21.15.28:0/2941579394 >> 172.21.15.28:0/2941579394 conn(0x558ad5825b80 msgr2=0x558ad5831f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:17.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.760+0000 7f1101c32740 1 -- 172.21.15.28:0/2941579394 shutdown_connections 2024-04-23T14:46:17.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.760+0000 7f1101c32740 1 -- 172.21.15.28:0/2941579394 wait complete. 2024-04-23T14:46:17.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.760+0000 7f1101c32740 1 -- start start 2024-04-23T14:46:17.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.761+0000 7f1101c32740 1 Processor -- start 2024-04-23T14:46:17.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.761+0000 7f1101c32740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ad5829860 0x558ad5838780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:17.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.761+0000 7f1101c32740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558ad582a180 0x558ad5838cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:17.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.761+0000 7f1101c32740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ad5834850 0x558ad58aa2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:17.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.761+0000 7f1101c32740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f10f81aed40 con 0x558ad5829860 2024-04-23T14:46:17.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.761+0000 7f1101c32740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f10f81b01b0 con 0x558ad582a180 2024-04-23T14:46:17.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.761+0000 7f1101c32740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f10f81af410 con 0x558ad5834850 2024-04-23T14:46:17.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.761+0000 7f1101c32740 10 client.0 _ll_register_callbacks cb 0x7f10e0009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:46:17.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.761+0000 7f10efe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558ad582a180 0x558ad5838cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:17.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.761+0000 7f10f7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ad5829860 0x558ad5838780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:17.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.761+0000 7f10fce00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ad5834850 0x558ad58aa2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:17.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.761+0000 7f10efe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558ad582a180 0x558ad5838cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60108/0 (socket says 172.21.15.28:60108) 2024-04-23T14:46:17.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.761+0000 7f10efe00640 1 -- 172.21.15.28:0/90775191 learned_addr learned my addr 172.21.15.28:0/90775191 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:17.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.762+0000 7f10f7e00640 1 -- 172.21.15.28:0/90775191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ad5834850 msgr2=0x558ad58aa2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:17.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.762+0000 7f10f7e00640 1 --2- 172.21.15.28:0/90775191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ad5834850 0x558ad58aa2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:17.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.762+0000 7f10f7e00640 1 -- 172.21.15.28:0/90775191 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558ad582a180 msgr2=0x558ad5838cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:17.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.762+0000 7f10f7e00640 1 --2- 172.21.15.28:0/90775191 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558ad582a180 0x558ad5838cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:17.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.762+0000 7f10f7e00640 1 -- 172.21.15.28:0/90775191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ad54610f0 con 0x558ad5829860 2024-04-23T14:46:17.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.762+0000 7f10efe00640 1 --2- 172.21.15.28:0/90775191 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558ad582a180 0x558ad5838cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:46:17.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.762+0000 7f10f7e00640 1 --2- 172.21.15.28:0/90775191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ad5829860 0x558ad5838780 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f10e800e9c0 tx=0x7f10e800e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a69174ae94e869c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:17.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.762+0000 7f10e6000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-23T14:46:17.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.762+0000 7f10e6000640 1 -- 172.21.15.28:0/90775191 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e80225c0 con 0x558ad5829860 2024-04-23T14:46:17.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.762+0000 7f10e6000640 1 -- 172.21.15.28:0/90775191 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f10e80d4860 con 0x558ad5829860 2024-04-23T14:46:17.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.763+0000 7f1101c32740 1 -- 172.21.15.28:0/90775191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10f0004690 con 0x558ad5829860 2024-04-23T14:46:17.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.763+0000 7f10e6000640 1 -- 172.21.15.28:0/90775191 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f10e80227a0 con 0x558ad5829860 2024-04-23T14:46:17.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.763+0000 7f1101c32740 10 client.15477 fetch_fsmap learned FSMap version 20 2024-04-23T14:46:17.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.763+0000 7f1101c32740 1 -- 172.21.15.28:0/90775191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x558ad5754810 con 0x558ad5829860 2024-04-23T14:46:17.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.763+0000 7f10e6000640 1 -- 172.21.15.28:0/90775191 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10f0004690 con 0x558ad5829860 2024-04-23T14:46:17.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.763+0000 7f10e6000640 1 -- 172.21.15.28:0/90775191 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f10e80991d0 con 0x558ad5829860 2024-04-23T14:46:17.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.763+0000 7f1101c32740 10 client.15477 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:46:17.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.763+0000 7f1101c32740 10 client.15477 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:46:17.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.763+0000 7f1101c32740 20 client.15477 populate_metadata read hostname 'smithi028' 2024-04-23T14:46:17.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.763+0000 7f1101c32740 1 --2- 172.21.15.28:0/90775191 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558ad58ab880 0x558ad58adc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:17.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.763+0000 7f1101c32740 4 client.15477 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:46:17.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.763+0000 7f1101c32740 1 -- 172.21.15.28:0/90775191 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558ad5712360 con 0x558ad58ab880 2024-04-23T14:46:17.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.764+0000 7f10efe00640 1 --2- 172.21.15.28:0/90775191 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558ad58ab880 0x558ad58adc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:17.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.764+0000 7f10efe00640 1 --2- 172.21.15.28:0/90775191 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558ad58ab880 0x558ad58adc70 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=440f590486fe5536 server_cookie=fd014684c978471a in_seq=0 out_seq=0 2024-04-23T14:46:17.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.764+0000 7f10e6000640 10 client.15477 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:46:17.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.764+0000 7f10e6000640 1 -- 172.21.15.28:0/90775191 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558ad5712360 con 0x558ad58ab880 2024-04-23T14:46:17.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.764+0000 7f10e6000640 10 client.15477 handle_command_reply: tid=0 2024-04-23T14:46:17.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.764+0000 7f1101c32740 1 client.15477 shutdown 2024-04-23T14:46:17.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.765+0000 7f1101c32740 1 -- 172.21.15.28:0/90775191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ad5829860 msgr2=0x558ad5838780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:17.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.765+0000 7f1101c32740 1 --2- 172.21.15.28:0/90775191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ad5829860 0x558ad5838780 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f10e800e9c0 tx=0x7f10e800e940 comp rx=0 tx=0).stop 2024-04-23T14:46:17.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.765+0000 7f1101c32740 1 -- 172.21.15.28:0/90775191 shutdown_connections 2024-04-23T14:46:17.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.765+0000 7f1101c32740 1 --2- 172.21.15.28:0/90775191 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558ad58ab880 0x558ad58adc70 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:17.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.765+0000 7f1101c32740 1 --2- 172.21.15.28:0/90775191 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558ad582a180 0x558ad5838cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:17.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.765+0000 7f1101c32740 1 --2- 172.21.15.28:0/90775191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ad5834850 0x558ad58aa2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:17.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.765+0000 7f1101c32740 1 --2- 172.21.15.28:0/90775191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ad5829860 0x558ad5838780 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:17.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.765+0000 7f1101c32740 1 -- 172.21.15.28:0/90775191 >> 172.21.15.28:0/90775191 conn(0x558ad5825b80 msgr2=0x558ad5824390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:17.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.765+0000 7f10e6000640 0 client.15477 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:46:17.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.765+0000 7f1101c32740 1 -- 172.21.15.28:0/90775191 shutdown_connections 2024-04-23T14:46:17.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.765+0000 7f1101c32740 1 -- 172.21.15.28:0/90775191 wait complete. 2024-04-23T14:46:17.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.765+0000 7f1101c32740 20 client.15477 trim_cache size 0 max 16384 2024-04-23T14:46:17.770 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:46:17.770 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-23T14:46:17.770 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-23T14:46:17.770 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:46:17.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.766+0000 7f10fee00640 1 -- 172.21.15.28:0/2261922438 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f10dc0a3da0 msgr2=0x7f10dc0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:17.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.766+0000 7f10fee00640 1 --2- 172.21.15.28:0/2261922438 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f10dc0a3da0 0x7f10dc0a6260 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f10f0004af0 tx=0x7f10f0009900 comp rx=0 tx=0).stop 2024-04-23T14:46:17.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.766+0000 7f10fee00640 1 -- 172.21.15.28:0/2261922438 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f10f81044b0 msgr2=0x7f10f81a07a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:17.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.766+0000 7f10fee00640 1 --2- 172.21.15.28:0/2261922438 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f10f81044b0 0x7f10f81a07a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f10e800e9f0 tx=0x7f10e800eeb0 comp rx=0 tx=0).stop 2024-04-23T14:46:17.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.766+0000 7f10fee00640 1 -- 172.21.15.28:0/2261922438 shutdown_connections 2024-04-23T14:46:17.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.766+0000 7f10fee00640 1 --2- 172.21.15.28:0/2261922438 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f10dc0a3da0 0x7f10dc0a6260 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:17.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.766+0000 7f10fee00640 1 --2- 172.21.15.28:0/2261922438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10f819a9a0 0x7f10f819ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:17.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.766+0000 7f10fee00640 1 --2- 172.21.15.28:0/2261922438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10f8109580 0x7f10f81a0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:17.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.766+0000 7f10fee00640 1 --2- 172.21.15.28:0/2261922438 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f10f81044b0 0x7f10f81a07a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:17.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.766+0000 7f10fee00640 1 -- 172.21.15.28:0/2261922438 >> 172.21.15.28:0/2261922438 conn(0x7f10f80ffd90 msgr2=0x7f10f81020e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:17.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.766+0000 7f10fee00640 1 -- 172.21.15.28:0/2261922438 shutdown_connections 2024-04-23T14:46:17.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.766+0000 7f10fee00640 1 -- 172.21.15.28:0/2261922438 wait complete. 2024-04-23T14:46:17.785 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-23T14:46:17.786 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:46:18.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.999+0000 7f3711e00640 1 -- 172.21.15.28:0/3761557226 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f370c104df0 msgr2=0x7f370c108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:18.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.999+0000 7f3711e00640 1 --2- 172.21.15.28:0/3761557226 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f370c104df0 0x7f370c108e30 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f36f8009900 tx=0x7f36f802a3f0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.999+0000 7f3711e00640 1 -- 172.21.15.28:0/3761557226 shutdown_connections 2024-04-23T14:46:18.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.999+0000 7f3711e00640 1 --2- 172.21.15.28:0/3761557226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f370c109510 0x7f370c1110e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.999+0000 7f3711e00640 1 --2- 172.21.15.28:0/3761557226 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f370c104df0 0x7f370c108e30 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.999+0000 7f3711e00640 1 --2- 172.21.15.28:0/3761557226 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f370c104440 0x7f370c104820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.999+0000 7f3711e00640 1 -- 172.21.15.28:0/3761557226 >> 172.21.15.28:0/3761557226 conn(0x7f370c0ffd00 msgr2=0x7f370c102120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:18.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.999+0000 7f3711e00640 1 -- 172.21.15.28:0/3761557226 shutdown_connections 2024-04-23T14:46:18.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:17.999+0000 7f3711e00640 1 -- 172.21.15.28:0/3761557226 wait complete. 2024-04-23T14:46:18.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.000+0000 7f3711e00640 1 Processor -- start 2024-04-23T14:46:18.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.000+0000 7f3711e00640 1 -- start start 2024-04-23T14:46:18.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.001+0000 7f3711e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f370c104440 0x7f370c1a0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:18.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.001+0000 7f3711e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f370c109510 0x7f370c1a0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:18.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.001+0000 7f3711e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f370c19a930 0x7f370c19ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:18.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.001+0000 7f3711e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f370c1145f0 con 0x7f370c19a930 2024-04-23T14:46:18.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.001+0000 7f3711e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f370c114470 con 0x7f370c109510 2024-04-23T14:46:18.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.001+0000 7f3711e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f370c114770 con 0x7f370c104440 2024-04-23T14:46:18.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.001+0000 7f370a000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f370c109510 0x7f370c1a0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:18.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.001+0000 7f370a000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f370c109510 0x7f370c1a0c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:33824/0 (socket says 172.21.15.28:33824) 2024-04-23T14:46:18.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.001+0000 7f370a000640 1 -- 172.21.15.28:0/896677852 learned_addr learned my addr 172.21.15.28:0/896677852 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:18.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.001+0000 7f370b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f370c19a930 0x7f370c19ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:18.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.001+0000 7f370aa00640 1 --2- 172.21.15.28:0/896677852 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f370c104440 0x7f370c1a0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:18.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.002+0000 7f370a000640 1 -- 172.21.15.28:0/896677852 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f370c104440 msgr2=0x7f370c1a0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:18.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.002+0000 7f370a000640 1 --2- 172.21.15.28:0/896677852 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f370c104440 0x7f370c1a0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.002+0000 7f370a000640 1 -- 172.21.15.28:0/896677852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f370c19a930 msgr2=0x7f370c19ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:18.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.002+0000 7f370a000640 1 --2- 172.21.15.28:0/896677852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f370c19a930 0x7f370c19ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.002+0000 7f370a000640 1 -- 172.21.15.28:0/896677852 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f370c19b450 con 0x7f370c109510 2024-04-23T14:46:18.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.002+0000 7f370a000640 1 --2- 172.21.15.28:0/896677852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f370c109510 0x7f370c1a0c70 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f36f802a910 tx=0x7f36f80045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:18.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.003+0000 7f3710a00640 1 -- 172.21.15.28:0/896677852 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f8035070 con 0x7f370c109510 2024-04-23T14:46:18.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.003+0000 7f3711e00640 1 -- 172.21.15.28:0/896677852 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f370c19b6e0 con 0x7f370c109510 2024-04-23T14:46:18.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.003+0000 7f3711e00640 1 -- 172.21.15.28:0/896677852 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f370c076730 con 0x7f370c109510 2024-04-23T14:46:18.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.003+0000 7f3710a00640 1 -- 172.21.15.28:0/896677852 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36f8003de0 con 0x7f370c109510 2024-04-23T14:46:18.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.003+0000 7f3710a00640 1 -- 172.21.15.28:0/896677852 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f8047620 con 0x7f370c109510 2024-04-23T14:46:18.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.004+0000 7f3711e00640 1 -- 172.21.15.28:0/896677852 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f370c105910 con 0x7f370c109510 2024-04-23T14:46:18.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.006+0000 7f3710a00640 1 -- 172.21.15.28:0/896677852 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f36f8031790 con 0x7f370c109510 2024-04-23T14:46:18.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.008+0000 7f3710a00640 1 --2- 172.21.15.28:0/896677852 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f36f00a3d20 0x7f36f00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:18.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.008+0000 7f3710a00640 1 -- 172.21.15.28:0/896677852 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f36f80f8d10 con 0x7f370c109510 2024-04-23T14:46:18.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.008+0000 7f370aa00640 1 --2- 172.21.15.28:0/896677852 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f36f00a3d20 0x7f36f00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:18.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.009+0000 7f370aa00640 1 --2- 172.21.15.28:0/896677852 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f36f00a3d20 0x7f36f00a61e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f370c1001a0 tx=0x7f36fc005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:18.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.013+0000 7f3710a00640 1 -- 172.21.15.28:0/896677852 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f36f80fe020 con 0x7f370c109510 2024-04-23T14:46:18.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.310+0000 7f3711e00640 1 -- 172.21.15.28:0/896677852 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f370c10c740 con 0x7f370c109510 2024-04-23T14:46:18.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.311+0000 7f3710a00640 1 -- 172.21.15.28:0/896677852 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7f36f80e3e20 con 0x7f370c109510 2024-04-23T14:46:18.313 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:46:18.313 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:46:18.313 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:46:18.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.314+0000 7f3711e00640 1 -- 172.21.15.28:0/896677852 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f36f00a3d20 msgr2=0x7f36f00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:18.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.314+0000 7f3711e00640 1 --2- 172.21.15.28:0/896677852 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f36f00a3d20 0x7f36f00a61e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f370c1001a0 tx=0x7f36fc005f90 comp rx=0 tx=0).stop 2024-04-23T14:46:18.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.314+0000 7f3711e00640 1 -- 172.21.15.28:0/896677852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f370c109510 msgr2=0x7f370c1a0c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:18.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.314+0000 7f3711e00640 1 --2- 172.21.15.28:0/896677852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f370c109510 0x7f370c1a0c70 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f36f802a910 tx=0x7f36f80045e0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.315+0000 7f3711e00640 1 -- 172.21.15.28:0/896677852 shutdown_connections 2024-04-23T14:46:18.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.315+0000 7f3711e00640 1 --2- 172.21.15.28:0/896677852 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f36f00a3d20 0x7f36f00a61e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.315+0000 7f3711e00640 1 --2- 172.21.15.28:0/896677852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f370c109510 0x7f370c1a0c70 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.315+0000 7f3711e00640 1 --2- 172.21.15.28:0/896677852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f370c19a930 0x7f370c19ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.315+0000 7f3711e00640 1 --2- 172.21.15.28:0/896677852 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f370c104440 0x7f370c1a0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.315+0000 7f3711e00640 1 -- 172.21.15.28:0/896677852 >> 172.21.15.28:0/896677852 conn(0x7f370c0ffd00 msgr2=0x7f370c1020a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:18.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.315+0000 7f3711e00640 1 -- 172.21.15.28:0/896677852 shutdown_connections 2024-04-23T14:46:18.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.315+0000 7f3711e00640 1 -- 172.21.15.28:0/896677852 wait complete. 2024-04-23T14:46:18.339 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-23T14:46:18.463 INFO:tasks.workunit.client.0.smithi028.stdout: 1 68492 11.36 MB/sec warmup 34 sec latency 10.807 ms 2024-04-23T14:46:18.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.567+0000 7fe87b200640 1 -- 172.21.15.28:0/131518023 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe87c100c70 msgr2=0x7fe87c101050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:18.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.567+0000 7fe87b200640 1 --2- 172.21.15.28:0/131518023 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe87c100c70 0x7fe87c101050 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe86c009860 tx=0x7fe86c02a340 comp rx=0 tx=0).stop 2024-04-23T14:46:18.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.567+0000 7fe87b200640 1 -- 172.21.15.28:0/131518023 shutdown_connections 2024-04-23T14:46:18.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.567+0000 7fe87b200640 1 --2- 172.21.15.28:0/131518023 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe87c10e820 0x7fe87c110c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.567+0000 7fe87b200640 1 --2- 172.21.15.28:0/131518023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe87c101620 0x7fe87c10e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.567+0000 7fe87b200640 1 --2- 172.21.15.28:0/131518023 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe87c100c70 0x7fe87c101050 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.567+0000 7fe87b200640 1 -- 172.21.15.28:0/131518023 >> 172.21.15.28:0/131518023 conn(0x7fe87c0fc870 msgr2=0x7fe87c0fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:18.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.568+0000 7fe87b200640 1 -- 172.21.15.28:0/131518023 shutdown_connections 2024-04-23T14:46:18.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.568+0000 7fe87b200640 1 -- 172.21.15.28:0/131518023 wait complete. 2024-04-23T14:46:18.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.569+0000 7fe87b200640 1 Processor -- start 2024-04-23T14:46:18.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.569+0000 7fe87b200640 1 -- start start 2024-04-23T14:46:18.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.569+0000 7fe87b200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe87c100c70 0x7fe87c1a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:18.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.569+0000 7fe87b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe87c101620 0x7fe87c1a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:18.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.569+0000 7fe87b200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe87c10e820 0x7fe87c19a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:18.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.569+0000 7fe87b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c1144b0 con 0x7fe87c101620 2024-04-23T14:46:18.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.569+0000 7fe87b200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c114330 con 0x7fe87c10e820 2024-04-23T14:46:18.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.569+0000 7fe87b200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c114630 con 0x7fe87c100c70 2024-04-23T14:46:18.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.570+0000 7fe879e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe87c100c70 0x7fe87c1a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:18.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.570+0000 7fe879e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe87c100c70 0x7fe87c1a06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:49750/0 (socket says 172.21.15.28:49750) 2024-04-23T14:46:18.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.570+0000 7fe879e00640 1 -- 172.21.15.28:0/3456883585 learned_addr learned my addr 172.21.15.28:0/3456883585 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:18.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.570+0000 7fe87a800640 1 --2- 172.21.15.28:0/3456883585 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe87c10e820 0x7fe87c19a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:18.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.570+0000 7fe879400640 1 --2- 172.21.15.28:0/3456883585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe87c101620 0x7fe87c1a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:18.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.570+0000 7fe879e00640 1 -- 172.21.15.28:0/3456883585 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe87c10e820 msgr2=0x7fe87c19a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:18.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.570+0000 7fe879e00640 1 --2- 172.21.15.28:0/3456883585 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe87c10e820 0x7fe87c19a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.570+0000 7fe879e00640 1 -- 172.21.15.28:0/3456883585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe87c101620 msgr2=0x7fe87c1a0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:18.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.570+0000 7fe879e00640 1 --2- 172.21.15.28:0/3456883585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe87c101620 0x7fe87c1a0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.570+0000 7fe879e00640 1 -- 172.21.15.28:0/3456883585 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe87c19b000 con 0x7fe87c100c70 2024-04-23T14:46:18.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.570+0000 7fe879400640 1 --2- 172.21.15.28:0/3456883585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe87c101620 0x7fe87c1a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:46:18.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.570+0000 7fe879e00640 1 --2- 172.21.15.28:0/3456883585 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe87c100c70 0x7fe87c1a06c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe86c035040 tx=0x7fe86c033d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:18.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.571+0000 7fe872a00640 1 -- 172.21.15.28:0/3456883585 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c037040 con 0x7fe87c100c70 2024-04-23T14:46:18.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.571+0000 7fe872a00640 1 -- 172.21.15.28:0/3456883585 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe86c040070 con 0x7fe87c100c70 2024-04-23T14:46:18.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.571+0000 7fe87b200640 1 -- 172.21.15.28:0/3456883585 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe87c19b290 con 0x7fe87c100c70 2024-04-23T14:46:18.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.571+0000 7fe872a00640 1 -- 172.21.15.28:0/3456883585 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c003a50 con 0x7fe87c100c70 2024-04-23T14:46:18.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.571+0000 7fe87b200640 1 -- 172.21.15.28:0/3456883585 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe87c1a7510 con 0x7fe87c100c70 2024-04-23T14:46:18.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.574+0000 7fe872a00640 1 -- 172.21.15.28:0/3456883585 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fe86c003bf0 con 0x7fe87c100c70 2024-04-23T14:46:18.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.574+0000 7fe883973740 1 Processor -- start 2024-04-23T14:46:18.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.575+0000 7fe883973740 1 -- start start 2024-04-23T14:46:18.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.575+0000 7fe883973740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca8480db40 0x55ca8480df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:18.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.575+0000 7fe883973740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca8480e460 0x55ca848185f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:18.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.575+0000 7fe883973740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca84818b30 0x55ca8481af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:18.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.575+0000 7fe883973740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ca847731d0 con 0x55ca8480e460 2024-04-23T14:46:18.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.575+0000 7fe883973740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ca84719450 con 0x55ca8480db40 2024-04-23T14:46:18.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.575+0000 7fe883973740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55ca8476f190 con 0x55ca84818b30 2024-04-23T14:46:18.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.575+0000 7fe87a800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca8480db40 0x55ca8480df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:18.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.575+0000 7fe879400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca84818b30 0x55ca8481af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:18.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.575+0000 7fe87a800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca8480db40 0x55ca8480df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:33860/0 (socket says 172.21.15.28:33860) 2024-04-23T14:46:18.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.575+0000 7fe87a800640 1 -- 172.21.15.28:0/4217463435 learned_addr learned my addr 172.21.15.28:0/4217463435 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:18.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.576+0000 7fe87a800640 1 -- 172.21.15.28:0/4217463435 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca84818b30 msgr2=0x55ca8481af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:18.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.576+0000 7fe879e00640 1 --2- 172.21.15.28:0/4217463435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca8480e460 0x55ca848185f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:18.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.576+0000 7fe87a800640 1 --2- 172.21.15.28:0/4217463435 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca84818b30 0x55ca8481af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.576+0000 7fe87a800640 1 -- 172.21.15.28:0/4217463435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca8480e460 msgr2=0x55ca848185f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:18.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.576+0000 7fe87a800640 1 --2- 172.21.15.28:0/4217463435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca8480e460 0x55ca848185f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.576+0000 7fe87a800640 1 -- 172.21.15.28:0/4217463435 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca847d6310 con 0x55ca8480db40 2024-04-23T14:46:18.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.576+0000 7fe879400640 1 --2- 172.21.15.28:0/4217463435 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca84818b30 0x55ca8481af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:46:18.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.576+0000 7fe879e00640 1 --2- 172.21.15.28:0/4217463435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca8480e460 0x55ca848185f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:46:18.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.576+0000 7fe87a800640 1 --2- 172.21.15.28:0/4217463435 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca8480db40 0x55ca8480df20 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe874009fc0 tx=0x7fe8740095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee7a269f71168f5c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:18.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.577+0000 7fe872a00640 1 --2- 172.21.15.28:0/3456883585 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe8600a3da0 0x7fe8600a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:18.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.577+0000 7fe867000640 1 -- 172.21.15.28:0/4217463435 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe874004b20 con 0x55ca8480db40 2024-04-23T14:46:18.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.577+0000 7fe867000640 1 -- 172.21.15.28:0/4217463435 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe874004cc0 con 0x55ca8480db40 2024-04-23T14:46:18.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.577+0000 7fe867000640 1 -- 172.21.15.28:0/4217463435 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe874005580 con 0x55ca8480db40 2024-04-23T14:46:18.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.577+0000 7fe872a00640 1 -- 172.21.15.28:0/3456883585 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fe86c0f0780 con 0x7fe87c100c70 2024-04-23T14:46:18.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.577+0000 7fe879e00640 1 --2- 172.21.15.28:0/3456883585 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe8600a3da0 0x7fe8600a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:18.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.577+0000 7fe883973740 1 -- 172.21.15.28:0/4217463435 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca8480db40 msgr2=0x55ca8480df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:18.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.577+0000 7fe883973740 1 --2- 172.21.15.28:0/4217463435 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca8480db40 0x55ca8480df20 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe874009fc0 tx=0x7fe8740095c0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.578+0000 7fe879e00640 1 --2- 172.21.15.28:0/3456883585 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe8600a3da0 0x7fe8600a6260 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x55ca8480f100 tx=0x7fe86c02a340 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:18.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.578+0000 7fe883973740 1 -- 172.21.15.28:0/4217463435 shutdown_connections 2024-04-23T14:46:18.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.578+0000 7fe883973740 1 --2- 172.21.15.28:0/4217463435 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca84818b30 0x55ca8481af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.578+0000 7fe883973740 1 --2- 172.21.15.28:0/4217463435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca8480e460 0x55ca848185f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.578+0000 7fe883973740 1 --2- 172.21.15.28:0/4217463435 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca8480db40 0x55ca8480df20 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.578+0000 7fe883973740 1 -- 172.21.15.28:0/4217463435 >> 172.21.15.28:0/4217463435 conn(0x55ca84809e60 msgr2=0x55ca8480a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:18.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.578+0000 7fe883973740 1 -- 172.21.15.28:0/4217463435 shutdown_connections 2024-04-23T14:46:18.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.578+0000 7fe883973740 1 -- 172.21.15.28:0/4217463435 wait complete. 2024-04-23T14:46:18.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.578+0000 7fe883973740 1 -- start start 2024-04-23T14:46:18.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.578+0000 7fe883973740 1 Processor -- start 2024-04-23T14:46:18.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.579+0000 7fe883973740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca8480db40 0x55ca84818570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:18.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.579+0000 7fe883973740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca8480e460 0x55ca84815860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:18.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.579+0000 7fe883973740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca84818b30 0x55ca84815fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:18.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.579+0000 7fe883973740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ca8481b9c0 con 0x55ca84818b30 2024-04-23T14:46:18.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.579+0000 7fe883973740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ca8472c810 con 0x55ca8480db40 2024-04-23T14:46:18.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.579+0000 7fe883973740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55ca8481bb40 con 0x55ca8480e460 2024-04-23T14:46:18.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.579+0000 7fe883973740 10 client.0 _ll_register_callbacks cb 0x55ca846fa140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:46:18.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.579+0000 7fe87a800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca8480db40 0x55ca84818570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.579+0000 7fe879400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca8480e460 0x55ca84815860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.579+0000 7fe87a800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca8480db40 0x55ca84818570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:33874/0 (socket says 172.21.15.28:33874) 2024-04-23T14:46:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.579+0000 7fe87a800640 1 -- 172.21.15.28:0/3396637766 learned_addr learned my addr 172.21.15.28:0/3396637766 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.579+0000 7fe879400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca8480e460 0x55ca84815860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:49764/0 (socket says 172.21.15.28:49764) 2024-04-23T14:46:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.579+0000 7fe87a800640 1 --2- 172.21.15.28:0/3396637766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca84818b30 0x55ca84815fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.580+0000 7fe879400640 1 -- 172.21.15.28:0/3396637766 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca8480db40 msgr2=0x55ca84818570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.580+0000 7fe879400640 1 --2- 172.21.15.28:0/3396637766 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca8480db40 0x55ca84818570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.580+0000 7fe879400640 1 -- 172.21.15.28:0/3396637766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca84818b30 msgr2=0x55ca84815fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.580+0000 7fe879400640 1 --2- 172.21.15.28:0/3396637766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca84818b30 0x55ca84815fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.580+0000 7fe879400640 1 -- 172.21.15.28:0/3396637766 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca8473d3f0 con 0x55ca8480e460 2024-04-23T14:46:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.580+0000 7fe87a800640 1 --2- 172.21.15.28:0/3396637766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca84818b30 0x55ca84815fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:46:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.580+0000 7fe879400640 1 --2- 172.21.15.28:0/3396637766 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca8480e460 0x55ca84815860 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe868005fd0 tx=0x7fe868012820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de4839cd36785266 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.580+0000 7fe85b400640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:46:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.580+0000 7fe85b400640 1 -- 172.21.15.28:0/3396637766 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe868010960 con 0x55ca8480e460 2024-04-23T14:46:18.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.580+0000 7fe85b400640 1 -- 172.21.15.28:0/3396637766 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fe868010b00 con 0x55ca8480e460 2024-04-23T14:46:18.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.580+0000 7fe883973740 1 -- 172.21.15.28:0/3396637766 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe874004cc0 con 0x55ca8480e460 2024-04-23T14:46:18.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.581+0000 7fe85b400640 1 -- 172.21.15.28:0/3396637766 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fe868010d00 con 0x55ca8480e460 2024-04-23T14:46:18.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.581+0000 7fe883973740 10 client.25642 fetch_fsmap learned FSMap version 20 2024-04-23T14:46:18.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.581+0000 7fe883973740 1 -- 172.21.15.28:0/3396637766 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55ca844853f0 con 0x55ca8480e460 2024-04-23T14:46:18.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.581+0000 7fe85b400640 1 -- 172.21.15.28:0/3396637766 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe874004cc0 con 0x55ca8480e460 2024-04-23T14:46:18.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.581+0000 7fe85b400640 1 -- 172.21.15.28:0/3396637766 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fe868019070 con 0x55ca8480e460 2024-04-23T14:46:18.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.581+0000 7fe883973740 10 client.25642 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:46:18.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.581+0000 7fe883973740 10 client.25642 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:46:18.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.581+0000 7fe883973740 20 client.25642 populate_metadata read hostname 'smithi028' 2024-04-23T14:46:18.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.581+0000 7fe883973740 1 --2- 172.21.15.28:0/3396637766 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55ca84828ec0 0x55ca8482b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:18.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.581+0000 7fe883973740 4 client.25642 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:46:18.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.581+0000 7fe883973740 1 -- 172.21.15.28:0/3396637766 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca84780bf0 con 0x55ca84828ec0 2024-04-23T14:46:18.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.582+0000 7fe87a800640 1 --2- 172.21.15.28:0/3396637766 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55ca84828ec0 0x55ca8482b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:18.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.582+0000 7fe87a800640 1 --2- 172.21.15.28:0/3396637766 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55ca84828ec0 0x55ca8482b300 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc6c30ad5431a91e server_cookie=35604683b1eaa835 in_seq=0 out_seq=0 2024-04-23T14:46:18.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.582+0000 7fe85b400640 10 client.25642 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:46:18.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.583+0000 7fe85b400640 1 -- 172.21.15.28:0/3396637766 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ca84780bf0 con 0x55ca84828ec0 2024-04-23T14:46:18.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.583+0000 7fe85b400640 10 client.25642 handle_command_reply: tid=0 2024-04-23T14:46:18.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.583+0000 7fe883973740 1 client.25642 shutdown 2024-04-23T14:46:18.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.583+0000 7fe883973740 1 -- 172.21.15.28:0/3396637766 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca8480e460 msgr2=0x55ca84815860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:18.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.583+0000 7fe883973740 1 --2- 172.21.15.28:0/3396637766 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca8480e460 0x55ca84815860 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe868005fd0 tx=0x7fe868012820 comp rx=0 tx=0).stop 2024-04-23T14:46:18.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.583+0000 7fe883973740 1 -- 172.21.15.28:0/3396637766 shutdown_connections 2024-04-23T14:46:18.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.583+0000 7fe883973740 1 --2- 172.21.15.28:0/3396637766 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55ca84828ec0 0x55ca8482b300 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.583+0000 7fe883973740 1 --2- 172.21.15.28:0/3396637766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca84818b30 0x55ca84815fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.583+0000 7fe883973740 1 --2- 172.21.15.28:0/3396637766 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca8480e460 0x55ca84815860 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.583+0000 7fe883973740 1 --2- 172.21.15.28:0/3396637766 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca8480db40 0x55ca84818570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.583+0000 7fe85b400640 0 client.25642 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:46:18.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.583+0000 7fe883973740 1 -- 172.21.15.28:0/3396637766 >> 172.21.15.28:0/3396637766 conn(0x55ca84809e60 msgr2=0x55ca8480fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:18.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.583+0000 7fe883973740 1 -- 172.21.15.28:0/3396637766 shutdown_connections 2024-04-23T14:46:18.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.584+0000 7fe883973740 1 -- 172.21.15.28:0/3396637766 wait complete. 2024-04-23T14:46:18.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.584+0000 7fe883973740 20 client.25642 trim_cache size 0 max 16384 2024-04-23T14:46:18.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.600+0000 7fe883973740 1 Processor -- start 2024-04-23T14:46:18.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.600+0000 7fe883973740 1 -- start start 2024-04-23T14:46:18.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.600+0000 7fe883973740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca8480db40 0x55ca848248e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:18.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.600+0000 7fe883973740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca8480e460 0x55ca848180a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:18.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.600+0000 7fe883973740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca84818b30 0x55ca848185e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:18.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.600+0000 7fe883973740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c1add60 con 0x55ca8480e460 2024-04-23T14:46:18.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.600+0000 7fe883973740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c1b05e0 con 0x55ca8480db40 2024-04-23T14:46:18.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.600+0000 7fe883973740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c1ae3d0 con 0x55ca84818b30 2024-04-23T14:46:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.600+0000 7fe87a800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca8480db40 0x55ca848248e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.600+0000 7fe87a800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca8480db40 0x55ca848248e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:33890/0 (socket says 172.21.15.28:33890) 2024-04-23T14:46:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.600+0000 7fe87a800640 1 -- 172.21.15.28:0/3153714441 learned_addr learned my addr 172.21.15.28:0/3153714441 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.601+0000 7fe879400640 1 --2- 172.21.15.28:0/3153714441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca8480e460 0x55ca848180a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.601+0000 7fe87a800640 1 --2- 172.21.15.28:0/3153714441 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca84818b30 0x55ca848185e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.601+0000 7fe87a800640 1 -- 172.21.15.28:0/3153714441 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca84818b30 msgr2=0x55ca848185e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.601+0000 7fe87a800640 1 --2- 172.21.15.28:0/3153714441 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca84818b30 0x55ca848185e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.601+0000 7fe87a800640 1 -- 172.21.15.28:0/3153714441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca8480e460 msgr2=0x55ca848180a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.601+0000 7fe87a800640 1 --2- 172.21.15.28:0/3153714441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca8480e460 0x55ca848180a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.601+0000 7fe87a800640 1 -- 172.21.15.28:0/3153714441 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca8454f7d0 con 0x55ca8480db40 2024-04-23T14:46:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.601+0000 7fe87a800640 1 --2- 172.21.15.28:0/3153714441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca8480db40 0x55ca848248e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fe874010010 tx=0x7fe874009be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6bc809ce850e0ba server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.601+0000 7fe866600640 1 -- 172.21.15.28:0/3153714441 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87402f040 con 0x55ca8480db40 2024-04-23T14:46:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.601+0000 7fe866600640 1 -- 172.21.15.28:0/3153714441 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe874003e50 con 0x55ca8480db40 2024-04-23T14:46:18.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.602+0000 7fe866600640 1 -- 172.21.15.28:0/3153714441 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe874005890 con 0x55ca8480db40 2024-04-23T14:46:18.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.602+0000 7fe883973740 1 -- 172.21.15.28:0/3153714441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca8480db40 msgr2=0x55ca848248e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:18.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.602+0000 7fe883973740 1 --2- 172.21.15.28:0/3153714441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca8480db40 0x55ca848248e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fe874010010 tx=0x7fe874009be0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.602+0000 7fe883973740 1 -- 172.21.15.28:0/3153714441 shutdown_connections 2024-04-23T14:46:18.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.602+0000 7fe883973740 1 --2- 172.21.15.28:0/3153714441 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca84818b30 0x55ca848185e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.602+0000 7fe883973740 1 --2- 172.21.15.28:0/3153714441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca8480e460 0x55ca848180a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.602+0000 7fe883973740 1 --2- 172.21.15.28:0/3153714441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca8480db40 0x55ca848248e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.602+0000 7fe883973740 1 -- 172.21.15.28:0/3153714441 >> 172.21.15.28:0/3153714441 conn(0x55ca84809e60 msgr2=0x55ca84810b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:18.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.602+0000 7fe883973740 1 -- 172.21.15.28:0/3153714441 shutdown_connections 2024-04-23T14:46:18.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.602+0000 7fe883973740 1 -- 172.21.15.28:0/3153714441 wait complete. 2024-04-23T14:46:18.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.603+0000 7fe883973740 1 -- start start 2024-04-23T14:46:18.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.603+0000 7fe883973740 1 Processor -- start 2024-04-23T14:46:18.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.604+0000 7fe883973740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca8480db40 0x55ca84821280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:18.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.604+0000 7fe883973740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca8480e460 0x55ca84812f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:18.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.604+0000 7fe883973740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca84818b30 0x55ca84813530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:18.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.604+0000 7fe883973740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c1aeaa0 con 0x55ca8480db40 2024-04-23T14:46:18.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.604+0000 7fe883973740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c1aff10 con 0x55ca84818b30 2024-04-23T14:46:18.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.604+0000 7fe883973740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c1af170 con 0x55ca8480e460 2024-04-23T14:46:18.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.604+0000 7fe883973740 10 client.0 _ll_register_callbacks cb 0x7fe86800b8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:46:18.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.604+0000 7fe879400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca8480e460 0x55ca84812f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:18.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.604+0000 7fe879400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca8480e460 0x55ca84812f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:49784/0 (socket says 172.21.15.28:49784) 2024-04-23T14:46:18.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.604+0000 7fe879400640 1 -- 172.21.15.28:0/4276231712 learned_addr learned my addr 172.21.15.28:0/4276231712 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:18.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.604+0000 7fe87a800640 1 --2- 172.21.15.28:0/4276231712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca8480db40 0x55ca84821280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:18.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.604+0000 7fe87a800640 1 --2- 172.21.15.28:0/4276231712 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca84818b30 0x55ca84813530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:18.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.604+0000 7fe879400640 1 -- 172.21.15.28:0/4276231712 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca84818b30 msgr2=0x55ca84813530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:18.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.604+0000 7fe879400640 1 --2- 172.21.15.28:0/4276231712 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca84818b30 0x55ca84813530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.604+0000 7fe879400640 1 -- 172.21.15.28:0/4276231712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca8480db40 msgr2=0x55ca84821280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:18.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.604+0000 7fe879400640 1 --2- 172.21.15.28:0/4276231712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca8480db40 0x55ca84821280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.604+0000 7fe879400640 1 -- 172.21.15.28:0/4276231712 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca84819dd0 con 0x55ca8480e460 2024-04-23T14:46:18.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.605+0000 7fe879400640 1 --2- 172.21.15.28:0/4276231712 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca8480e460 0x55ca84812f20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x55ca84816e30 tx=0x7fe868023a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2dd04cdbd58b7de3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:18.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.605+0000 7fe85b400640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:46:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.605+0000 7fe85b400640 1 -- 172.21.15.28:0/4276231712 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe868010ea0 con 0x55ca8480e460 2024-04-23T14:46:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.605+0000 7fe85b400640 1 -- 172.21.15.28:0/4276231712 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fe868023dc0 con 0x55ca8480e460 2024-04-23T14:46:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.605+0000 7fe883973740 1 -- 172.21.15.28:0/4276231712 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe874003e50 con 0x55ca8480e460 2024-04-23T14:46:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.606+0000 7fe85b400640 1 -- 172.21.15.28:0/4276231712 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fe868005030 con 0x55ca8480e460 2024-04-23T14:46:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.606+0000 7fe883973740 10 client.25648 fetch_fsmap learned FSMap version 20 2024-04-23T14:46:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.606+0000 7fe883973740 1 -- 172.21.15.28:0/4276231712 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55ca8488f9d0 con 0x55ca8480e460 2024-04-23T14:46:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.606+0000 7fe85b400640 1 -- 172.21.15.28:0/4276231712 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe874003e50 con 0x55ca8480e460 2024-04-23T14:46:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.606+0000 7fe85b400640 1 -- 172.21.15.28:0/4276231712 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fe868019040 con 0x55ca8480e460 2024-04-23T14:46:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.606+0000 7fe883973740 10 client.25648 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:46:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.606+0000 7fe883973740 10 client.25648 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:46:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.606+0000 7fe883973740 20 client.25648 populate_metadata read hostname 'smithi028' 2024-04-23T14:46:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.606+0000 7fe883973740 1 --2- 172.21.15.28:0/4276231712 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55ca84890170 0x55ca84892590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.606+0000 7fe883973740 4 client.25648 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-23T14:46:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.606+0000 7fe883973740 1 -- 172.21.15.28:0/4276231712 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ca8477aa60 con 0x55ca84890170 2024-04-23T14:46:18.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.606+0000 7fe87a800640 1 --2- 172.21.15.28:0/4276231712 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55ca84890170 0x55ca84892590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:18.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.607+0000 7fe87a800640 1 --2- 172.21.15.28:0/4276231712 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55ca84890170 0x55ca84892590 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=662e6fa7cc90adf9 server_cookie=9f64ef54b9a81c9d in_seq=0 out_seq=0 2024-04-23T14:46:18.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.607+0000 7fe85b400640 10 client.25648 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:46:18.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.608+0000 7fe85b400640 1 -- 172.21.15.28:0/4276231712 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ca8477aa60 con 0x55ca84890170 2024-04-23T14:46:18.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.608+0000 7fe85b400640 10 client.25648 handle_command_reply: tid=0 2024-04-23T14:46:18.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.608+0000 7fe883973740 1 client.25648 shutdown 2024-04-23T14:46:18.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.608+0000 7fe883973740 1 -- 172.21.15.28:0/4276231712 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca8480e460 msgr2=0x55ca84812f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:18.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.608+0000 7fe883973740 1 --2- 172.21.15.28:0/4276231712 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca8480e460 0x55ca84812f20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x55ca84816e30 tx=0x7fe868023a40 comp rx=0 tx=0).stop 2024-04-23T14:46:18.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.608+0000 7fe883973740 1 -- 172.21.15.28:0/4276231712 shutdown_connections 2024-04-23T14:46:18.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.608+0000 7fe883973740 1 --2- 172.21.15.28:0/4276231712 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55ca84890170 0x55ca84892590 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.608+0000 7fe883973740 1 --2- 172.21.15.28:0/4276231712 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca84818b30 0x55ca84813530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.608+0000 7fe883973740 1 --2- 172.21.15.28:0/4276231712 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ca8480e460 0x55ca84812f20 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.608+0000 7fe883973740 1 --2- 172.21.15.28:0/4276231712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca8480db40 0x55ca84821280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.608+0000 7fe883973740 1 -- 172.21.15.28:0/4276231712 >> 172.21.15.28:0/4276231712 conn(0x55ca84809e60 msgr2=0x55ca848086c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:18.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.608+0000 7fe85b400640 0 client.25648 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:46:18.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.609+0000 7fe883973740 1 -- 172.21.15.28:0/4276231712 shutdown_connections 2024-04-23T14:46:18.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.609+0000 7fe883973740 1 -- 172.21.15.28:0/4276231712 wait complete. 2024-04-23T14:46:18.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.609+0000 7fe883973740 20 client.25648 trim_cache size 0 max 16384 2024-04-23T14:46:18.612 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-23T14:46:18.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.609+0000 7fe87b200640 1 -- 172.21.15.28:0/3456883585 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe8600a3da0 msgr2=0x7fe8600a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:18.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.609+0000 7fe87b200640 1 --2- 172.21.15.28:0/3456883585 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe8600a3da0 0x7fe8600a6260 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x55ca8480f100 tx=0x7fe86c02a340 comp rx=0 tx=0).stop 2024-04-23T14:46:18.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.609+0000 7fe87b200640 1 -- 172.21.15.28:0/3456883585 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe87c100c70 msgr2=0x7fe87c1a06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:18.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.609+0000 7fe87b200640 1 --2- 172.21.15.28:0/3456883585 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe87c100c70 0x7fe87c1a06c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe86c035040 tx=0x7fe86c033d90 comp rx=0 tx=0).stop 2024-04-23T14:46:18.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.610+0000 7fe87b200640 1 -- 172.21.15.28:0/3456883585 shutdown_connections 2024-04-23T14:46:18.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.610+0000 7fe87b200640 1 --2- 172.21.15.28:0/3456883585 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe8600a3da0 0x7fe8600a6260 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.610+0000 7fe87b200640 1 --2- 172.21.15.28:0/3456883585 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe87c10e820 0x7fe87c19a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.610+0000 7fe87b200640 1 --2- 172.21.15.28:0/3456883585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe87c101620 0x7fe87c1a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.610+0000 7fe87b200640 1 --2- 172.21.15.28:0/3456883585 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe87c100c70 0x7fe87c1a06c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:18.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.610+0000 7fe87b200640 1 -- 172.21.15.28:0/3456883585 >> 172.21.15.28:0/3456883585 conn(0x7fe87c0fc870 msgr2=0x7fe87c0fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:18.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.610+0000 7fe87b200640 1 -- 172.21.15.28:0/3456883585 shutdown_connections 2024-04-23T14:46:18.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:18.610+0000 7fe87b200640 1 -- 172.21.15.28:0/3456883585 wait complete. 2024-04-23T14:46:19.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:18 smithi038 ceph-mon[32348]: pgmap v71: 97 pgs: 97 active+clean; 56 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 3.3 MiB/s wr, 181 op/s 2024-04-23T14:46:19.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:18 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/896677852' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:46:19.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:18 smithi028 ceph-mon[27810]: pgmap v71: 97 pgs: 97 active+clean; 56 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 3.3 MiB/s wr, 181 op/s 2024-04-23T14:46:19.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:18 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/896677852' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:46:19.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:18 smithi105 ceph-mon[32555]: pgmap v71: 97 pgs: 97 active+clean; 56 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 3.3 MiB/s wr, 181 op/s 2024-04-23T14:46:19.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:18 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/896677852' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:46:19.464 INFO:tasks.workunit.client.0.smithi028.stdout: 1 70592 11.35 MB/sec warmup 35 sec latency 12.867 ms 2024-04-23T14:46:19.630 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-23T14:46:19.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:46:19.631 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-23T14:46:19.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.868+0000 7fdd2b800640 1 -- 172.21.15.28:0/4501741 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd24074bf0 msgr2=0x7fdd24112b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:19.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.868+0000 7fdd2b800640 1 --2- 172.21.15.28:0/4501741 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd24074bf0 0x7fdd24112b70 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdd10009900 tx=0x7fdd1002a480 comp rx=0 tx=0).stop 2024-04-23T14:46:19.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.868+0000 7fdd2b800640 1 -- 172.21.15.28:0/4501741 shutdown_connections 2024-04-23T14:46:19.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.868+0000 7fdd2b800640 1 --2- 172.21.15.28:0/4501741 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd24074bf0 0x7fdd24112b70 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:19.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.868+0000 7fdd2b800640 1 --2- 172.21.15.28:0/4501741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd241130b0 0x7fdd241154a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:19.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.868+0000 7fdd2b800640 1 --2- 172.21.15.28:0/4501741 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd240742d0 0x7fdd240746b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:19.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.868+0000 7fdd2b800640 1 -- 172.21.15.28:0/4501741 >> 172.21.15.28:0/4501741 conn(0x7fdd24079450 msgr2=0x7fdd24079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:19.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.868+0000 7fdd2b800640 1 -- 172.21.15.28:0/4501741 shutdown_connections 2024-04-23T14:46:19.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.868+0000 7fdd2b800640 1 -- 172.21.15.28:0/4501741 wait complete. 2024-04-23T14:46:19.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.870+0000 7fdd2b800640 1 Processor -- start 2024-04-23T14:46:19.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.870+0000 7fdd2b800640 1 -- start start 2024-04-23T14:46:19.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.870+0000 7fdd2b800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd240742d0 0x7fdd2419ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:19.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.870+0000 7fdd2b800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd24074bf0 0x7fdd2419f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:19.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.870+0000 7fdd2b800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd241130b0 0x7fdd241a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:19.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.870+0000 7fdd2b800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdd24118a20 con 0x7fdd240742d0 2024-04-23T14:46:19.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.870+0000 7fdd2b800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdd241188a0 con 0x7fdd241130b0 2024-04-23T14:46:19.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.870+0000 7fdd2b800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdd24118ba0 con 0x7fdd24074bf0 2024-04-23T14:46:19.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.871+0000 7fdd28e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd240742d0 0x7fdd2419ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:19.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.871+0000 7fdd28e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd240742d0 0x7fdd2419ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36294/0 (socket says 172.21.15.28:36294) 2024-04-23T14:46:19.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.871+0000 7fdd28e00640 1 -- 172.21.15.28:0/1365529891 learned_addr learned my addr 172.21.15.28:0/1365529891 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:19.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.871+0000 7fdd23e00640 1 --2- 172.21.15.28:0/1365529891 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd24074bf0 0x7fdd2419f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:19.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.871+0000 7fdd29800640 1 --2- 172.21.15.28:0/1365529891 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd241130b0 0x7fdd241a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:19.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.872+0000 7fdd28e00640 1 -- 172.21.15.28:0/1365529891 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd24074bf0 msgr2=0x7fdd2419f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:19.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.872+0000 7fdd28e00640 1 --2- 172.21.15.28:0/1365529891 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd24074bf0 0x7fdd2419f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:19.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.872+0000 7fdd28e00640 1 -- 172.21.15.28:0/1365529891 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd241130b0 msgr2=0x7fdd241a37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:19.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.872+0000 7fdd28e00640 1 --2- 172.21.15.28:0/1365529891 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd241130b0 0x7fdd241a37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:19.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.872+0000 7fdd28e00640 1 -- 172.21.15.28:0/1365529891 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd241a3f10 con 0x7fdd240742d0 2024-04-23T14:46:19.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.872+0000 7fdd28e00640 1 --2- 172.21.15.28:0/1365529891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd240742d0 0x7fdd2419ef20 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fdd1800ea60 tx=0x7fdd1800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:19.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.872+0000 7fdd21600640 1 -- 172.21.15.28:0/1365529891 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1800cde0 con 0x7fdd240742d0 2024-04-23T14:46:19.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.872+0000 7fdd21600640 1 -- 172.21.15.28:0/1365529891 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd180035f0 con 0x7fdd240742d0 2024-04-23T14:46:19.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.872+0000 7fdd21600640 1 -- 172.21.15.28:0/1365529891 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd18010670 con 0x7fdd240742d0 2024-04-23T14:46:19.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.872+0000 7fdd2b800640 1 -- 172.21.15.28:0/1365529891 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd241a4200 con 0x7fdd240742d0 2024-04-23T14:46:19.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.873+0000 7fdd2b800640 1 -- 172.21.15.28:0/1365529891 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd241aba90 con 0x7fdd240742d0 2024-04-23T14:46:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.878+0000 7fdd2e5ca740 1 Processor -- start 2024-04-23T14:46:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.878+0000 7fdd21600640 1 -- 172.21.15.28:0/1365529891 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fdd180037a0 con 0x7fdd240742d0 2024-04-23T14:46:19.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.879+0000 7fdd2e5ca740 1 -- start start 2024-04-23T14:46:19.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.879+0000 7fdd2e5ca740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14e245090 0x55d14e245470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:19.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.879+0000 7fdd2e5ca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d14e2459b0 0x55d14e24fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:19.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.879+0000 7fdd2e5ca740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14e250080 0x55d14e252470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:19.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.879+0000 7fdd2e5ca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d14e1aae70 con 0x55d14e2459b0 2024-04-23T14:46:19.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.879+0000 7fdd2e5ca740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d14e151380 con 0x55d14e245090 2024-04-23T14:46:19.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.879+0000 7fdd2e5ca740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d14e1a6e30 con 0x55d14e250080 2024-04-23T14:46:19.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.880+0000 7fdd29800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14e245090 0x55d14e245470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:19.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.880+0000 7fdd23e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14e250080 0x55d14e252470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:19.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.880+0000 7fdd29800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14e245090 0x55d14e245470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:33912/0 (socket says 172.21.15.28:33912) 2024-04-23T14:46:19.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.880+0000 7fdd29800640 1 -- 172.21.15.28:0/3953773548 learned_addr learned my addr 172.21.15.28:0/3953773548 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:19.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.880+0000 7fdd28e00640 1 --2- 172.21.15.28:0/3953773548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d14e2459b0 0x55d14e24fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:19.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.880+0000 7fdd29800640 1 -- 172.21.15.28:0/3953773548 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14e250080 msgr2=0x55d14e252470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:19.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.880+0000 7fdd29800640 1 --2- 172.21.15.28:0/3953773548 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14e250080 0x55d14e252470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:19.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.880+0000 7fdd29800640 1 -- 172.21.15.28:0/3953773548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d14e2459b0 msgr2=0x55d14e24fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:19.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.880+0000 7fdd29800640 1 --2- 172.21.15.28:0/3953773548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d14e2459b0 0x55d14e24fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:19.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.880+0000 7fdd29800640 1 -- 172.21.15.28:0/3953773548 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d14df857e0 con 0x55d14e245090 2024-04-23T14:46:19.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.880+0000 7fdd23e00640 1 --2- 172.21.15.28:0/3953773548 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14e250080 0x55d14e252470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:46:19.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.880+0000 7fdd28e00640 1 --2- 172.21.15.28:0/3953773548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d14e2459b0 0x55d14e24fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:46:19.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.881+0000 7fdd29800640 1 --2- 172.21.15.28:0/3953773548 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14e245090 0x55d14e245470 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c009f30 tx=0x7fdd1c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cce682092f2f9444 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:19.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.881+0000 7fdd21600640 1 --2- 172.21.15.28:0/1365529891 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fdd0c0a3d20 0x7fdd0c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:19.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.881+0000 7fdd21600640 1 -- 172.21.15.28:0/1365529891 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fdd180d0ab0 con 0x7fdd240742d0 2024-04-23T14:46:19.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.881+0000 7fdd28e00640 1 --2- 172.21.15.28:0/1365529891 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fdd0c0a3d20 0x7fdd0c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:19.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.882+0000 7fdd16000640 1 -- 172.21.15.28:0/3953773548 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c004d20 con 0x55d14e245090 2024-04-23T14:46:19.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.882+0000 7fdd16000640 1 -- 172.21.15.28:0/3953773548 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd1c004050 con 0x55d14e245090 2024-04-23T14:46:19.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.882+0000 7fdd16000640 1 -- 172.21.15.28:0/3953773548 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c005260 con 0x55d14e245090 2024-04-23T14:46:19.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.883+0000 7fdd2e5ca740 1 -- 172.21.15.28:0/3953773548 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14e245090 msgr2=0x55d14e245470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:19.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.883+0000 7fdd28e00640 1 --2- 172.21.15.28:0/1365529891 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fdd0c0a3d20 0x7fdd0c0a61e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdd180ce2c0 tx=0x7fdd1800ef90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:19.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.883+0000 7fdd2e5ca740 1 --2- 172.21.15.28:0/3953773548 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14e245090 0x55d14e245470 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c009f30 tx=0x7fdd1c0118d0 comp rx=0 tx=0).stop 2024-04-23T14:46:19.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.883+0000 7fdd2e5ca740 1 -- 172.21.15.28:0/3953773548 shutdown_connections 2024-04-23T14:46:19.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.883+0000 7fdd2e5ca740 1 --2- 172.21.15.28:0/3953773548 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14e250080 0x55d14e252470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:19.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.883+0000 7fdd2e5ca740 1 --2- 172.21.15.28:0/3953773548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d14e2459b0 0x55d14e24fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:19.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.883+0000 7fdd2e5ca740 1 --2- 172.21.15.28:0/3953773548 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14e245090 0x55d14e245470 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:19.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.883+0000 7fdd2e5ca740 1 -- 172.21.15.28:0/3953773548 >> 172.21.15.28:0/3953773548 conn(0x55d14e2413b0 msgr2=0x55d14e241790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:19.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.883+0000 7fdd2e5ca740 1 -- 172.21.15.28:0/3953773548 shutdown_connections 2024-04-23T14:46:19.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.883+0000 7fdd2e5ca740 1 -- 172.21.15.28:0/3953773548 wait complete. 2024-04-23T14:46:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.884+0000 7fdd2e5ca740 1 -- start start 2024-04-23T14:46:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.884+0000 7fdd2e5ca740 1 Processor -- start 2024-04-23T14:46:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.885+0000 7fdd2e5ca740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14e245090 0x55d14e24cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.885+0000 7fdd2e5ca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d14e2459b0 0x55d14e24d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.885+0000 7fdd2e5ca740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14e250080 0x55d14e24da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.885+0000 7fdd2e5ca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d14e252e70 con 0x55d14e2459b0 2024-04-23T14:46:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.885+0000 7fdd2e5ca740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d14e164660 con 0x55d14e250080 2024-04-23T14:46:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.885+0000 7fdd2e5ca740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d14e252ff0 con 0x55d14e245090 2024-04-23T14:46:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.885+0000 7fdd2e5ca740 10 client.0 _ll_register_callbacks cb 0x55d14e152aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:46:19.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.885+0000 7fdd23e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d14e2459b0 0x55d14e24d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:19.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.885+0000 7fdd29800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14e250080 0x55d14e24da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:19.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.885+0000 7fdd23e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d14e2459b0 0x55d14e24d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36312/0 (socket says 172.21.15.28:36312) 2024-04-23T14:46:19.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.885+0000 7fdd23e00640 1 -- 172.21.15.28:0/884619992 learned_addr learned my addr 172.21.15.28:0/884619992 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:19.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.885+0000 7fdd29800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14e250080 0x55d14e24da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:33922/0 (socket says 172.21.15.28:33922) 2024-04-23T14:46:19.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.885+0000 7fdd29800640 1 --2- 172.21.15.28:0/884619992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14e245090 0x55d14e24cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:19.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.886+0000 7fdd29800640 1 -- 172.21.15.28:0/884619992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14e245090 msgr2=0x55d14e24cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:19.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.886+0000 7fdd29800640 1 --2- 172.21.15.28:0/884619992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14e245090 0x55d14e24cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:19.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.886+0000 7fdd29800640 1 -- 172.21.15.28:0/884619992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d14e2459b0 msgr2=0x55d14e24d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:19.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.886+0000 7fdd29800640 1 --2- 172.21.15.28:0/884619992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d14e2459b0 0x55d14e24d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:19.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.886+0000 7fdd29800640 1 -- 172.21.15.28:0/884619992 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d14dfe5f70 con 0x55d14e250080 2024-04-23T14:46:19.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.886+0000 7fdd29800640 1 --2- 172.21.15.28:0/884619992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14e250080 0x55d14e24da60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c005fd0 tx=0x7fdd1c000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fddf1f9e8fc9b42 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:19.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.886+0000 7fdd0aa00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:46:19.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.886+0000 7fdd0aa00640 1 -- 172.21.15.28:0/884619992 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c02d060 con 0x55d14e250080 2024-04-23T14:46:19.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.886+0000 7fdd2e5ca740 1 -- 172.21.15.28:0/884619992 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd1c004050 con 0x55d14e250080 2024-04-23T14:46:19.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.886+0000 7fdd0aa00640 1 -- 172.21.15.28:0/884619992 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fdd1c00f480 con 0x55d14e250080 2024-04-23T14:46:19.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.887+0000 7fdd0aa00640 1 -- 172.21.15.28:0/884619992 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fdd1c004610 con 0x55d14e250080 2024-04-23T14:46:19.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.887+0000 7fdd2e5ca740 10 client.25652 fetch_fsmap learned FSMap version 20 2024-04-23T14:46:19.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.887+0000 7fdd2e5ca740 1 -- 172.21.15.28:0/884619992 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55d14de7d170 con 0x55d14e250080 2024-04-23T14:46:19.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.887+0000 7fdd0aa00640 1 -- 172.21.15.28:0/884619992 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd1c004050 con 0x55d14e250080 2024-04-23T14:46:19.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.887+0000 7fdd0aa00640 1 -- 172.21.15.28:0/884619992 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fdd1c025040 con 0x55d14e250080 2024-04-23T14:46:19.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.887+0000 7fdd2e5ca740 10 client.25652 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:46:19.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.887+0000 7fdd2e5ca740 10 client.25652 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:46:19.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.887+0000 7fdd2e5ca740 20 client.25652 populate_metadata read hostname 'smithi028' 2024-04-23T14:46:19.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.887+0000 7fdd2e5ca740 1 --2- 172.21.15.28:0/884619992 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d14e2601c0 0x55d14e2625b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:19.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.887+0000 7fdd2e5ca740 4 client.25652 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:46:19.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.887+0000 7fdd2e5ca740 1 -- 172.21.15.28:0/884619992 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d14e1b8890 con 0x55d14e2601c0 2024-04-23T14:46:19.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.887+0000 7fdd23e00640 1 --2- 172.21.15.28:0/884619992 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d14e2601c0 0x55d14e2625b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:19.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.888+0000 7fdd23e00640 1 --2- 172.21.15.28:0/884619992 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d14e2601c0 0x55d14e2625b0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9612b43673a260bc server_cookie=ae12450eaaa1216f in_seq=0 out_seq=0 2024-04-23T14:46:19.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.888+0000 7fdd0aa00640 10 client.25652 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:46:19.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.890+0000 7fdd0aa00640 1 -- 172.21.15.28:0/884619992 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d14e1b8890 con 0x55d14e2601c0 2024-04-23T14:46:19.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.890+0000 7fdd0aa00640 10 client.25652 handle_command_reply: tid=0 2024-04-23T14:46:19.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.890+0000 7fdd2e5ca740 1 client.25652 shutdown 2024-04-23T14:46:19.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.890+0000 7fdd2e5ca740 1 -- 172.21.15.28:0/884619992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14e250080 msgr2=0x55d14e24da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:19.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.890+0000 7fdd2e5ca740 1 --2- 172.21.15.28:0/884619992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14e250080 0x55d14e24da60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c005fd0 tx=0x7fdd1c000f40 comp rx=0 tx=0).stop 2024-04-23T14:46:19.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.890+0000 7fdd2e5ca740 1 -- 172.21.15.28:0/884619992 shutdown_connections 2024-04-23T14:46:19.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.890+0000 7fdd2e5ca740 1 --2- 172.21.15.28:0/884619992 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d14e2601c0 0x55d14e2625b0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:19.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.890+0000 7fdd2e5ca740 1 --2- 172.21.15.28:0/884619992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14e250080 0x55d14e24da60 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:19.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.890+0000 7fdd2e5ca740 1 --2- 172.21.15.28:0/884619992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d14e2459b0 0x55d14e24d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:19.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.890+0000 7fdd2e5ca740 1 --2- 172.21.15.28:0/884619992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14e245090 0x55d14e24cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:19.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.890+0000 7fdd2e5ca740 1 -- 172.21.15.28:0/884619992 >> 172.21.15.28:0/884619992 conn(0x55d14e2413b0 msgr2=0x55d14e2470b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:19.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.890+0000 7fdd0aa00640 0 client.25652 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:46:19.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.891+0000 7fdd2e5ca740 1 -- 172.21.15.28:0/884619992 shutdown_connections 2024-04-23T14:46:19.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.891+0000 7fdd2e5ca740 1 -- 172.21.15.28:0/884619992 wait complete. 2024-04-23T14:46:19.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.891+0000 7fdd2e5ca740 20 client.25652 trim_cache size 0 max 16384 2024-04-23T14:46:19.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.912+0000 7fdd2e5ca740 1 Processor -- start 2024-04-23T14:46:19.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.912+0000 7fdd2e5ca740 1 -- start start 2024-04-23T14:46:19.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.912+0000 7fdd2e5ca740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14e245090 0x55d14e25bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:19.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.912+0000 7fdd2e5ca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d14e2459b0 0x55d14e24d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:19.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.912+0000 7fdd2e5ca740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14e250080 0x55d14e267cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:19.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.912+0000 7fdd2e5ca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdd241b23a0 con 0x55d14e2459b0 2024-04-23T14:46:19.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.912+0000 7fdd2e5ca740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdd241b4c20 con 0x55d14e250080 2024-04-23T14:46:19.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.912+0000 7fdd2e5ca740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdd241b2a10 con 0x55d14e245090 2024-04-23T14:46:19.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.913+0000 7fdd23e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d14e2459b0 0x55d14e24d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:19.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.913+0000 7fdd29800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14e245090 0x55d14e25bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:19.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.913+0000 7fdd23e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d14e2459b0 0x55d14e24d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36328/0 (socket says 172.21.15.28:36328) 2024-04-23T14:46:19.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.913+0000 7fdd23e00640 1 -- 172.21.15.28:0/1115133490 learned_addr learned my addr 172.21.15.28:0/1115133490 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:19.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.913+0000 7fdd29800640 1 --2- 172.21.15.28:0/1115133490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14e250080 0x55d14e267cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:19.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.913+0000 7fdd23e00640 1 -- 172.21.15.28:0/1115133490 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14e245090 msgr2=0x55d14e25bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:19.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.913+0000 7fdd23e00640 1 --2- 172.21.15.28:0/1115133490 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14e245090 0x55d14e25bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:19.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.913+0000 7fdd23e00640 1 -- 172.21.15.28:0/1115133490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14e250080 msgr2=0x55d14e267cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:19.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.913+0000 7fdd23e00640 1 --2- 172.21.15.28:0/1115133490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14e250080 0x55d14e267cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:19.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.913+0000 7fdd23e00640 1 -- 172.21.15.28:0/1115133490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d14e2457e0 con 0x55d14e2459b0 2024-04-23T14:46:19.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.914+0000 7fdd23e00640 1 --2- 172.21.15.28:0/1115133490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d14e2459b0 0x55d14e24d880 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fdd10009230 tx=0x7fdd10005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e4e521ed6a9fab3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:19.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.914+0000 7fdd15600640 1 -- 172.21.15.28:0/1115133490 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd10035040 con 0x55d14e2459b0 2024-04-23T14:46:19.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.914+0000 7fdd15600640 1 -- 172.21.15.28:0/1115133490 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd1004b070 con 0x55d14e2459b0 2024-04-23T14:46:19.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.914+0000 7fdd15600640 1 -- 172.21.15.28:0/1115133490 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd10005600 con 0x55d14e2459b0 2024-04-23T14:46:19.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.915+0000 7fdd2e5ca740 1 -- 172.21.15.28:0/1115133490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d14e2459b0 msgr2=0x55d14e24d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:19.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.915+0000 7fdd2e5ca740 1 --2- 172.21.15.28:0/1115133490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d14e2459b0 0x55d14e24d880 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fdd10009230 tx=0x7fdd10005b50 comp rx=0 tx=0).stop 2024-04-23T14:46:19.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.915+0000 7fdd2e5ca740 1 -- 172.21.15.28:0/1115133490 shutdown_connections 2024-04-23T14:46:19.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.915+0000 7fdd2e5ca740 1 --2- 172.21.15.28:0/1115133490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14e250080 0x55d14e267cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:19.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.915+0000 7fdd2e5ca740 1 --2- 172.21.15.28:0/1115133490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d14e2459b0 0x55d14e24d880 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:19.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.915+0000 7fdd2e5ca740 1 --2- 172.21.15.28:0/1115133490 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14e245090 0x55d14e25bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:19.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.915+0000 7fdd2e5ca740 1 -- 172.21.15.28:0/1115133490 >> 172.21.15.28:0/1115133490 conn(0x55d14e2413b0 msgr2=0x55d14e2470b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:19.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.915+0000 7fdd2e5ca740 1 -- 172.21.15.28:0/1115133490 shutdown_connections 2024-04-23T14:46:19.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.915+0000 7fdd2e5ca740 1 -- 172.21.15.28:0/1115133490 wait complete. 2024-04-23T14:46:19.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.915+0000 7fdd2e5ca740 1 -- start start 2024-04-23T14:46:19.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.916+0000 7fdd2e5ca740 1 Processor -- start 2024-04-23T14:46:19.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.916+0000 7fdd2e5ca740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14e245090 0x55d14e258690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:19.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.916+0000 7fdd2e5ca740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14e2459b0 0x55d14e253a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:19.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.916+0000 7fdd2e5ca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d14e250080 0x55d14e253fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:19.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.916+0000 7fdd2e5ca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdd241b30e0 con 0x55d14e250080 2024-04-23T14:46:19.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.916+0000 7fdd2e5ca740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdd241b4550 con 0x55d14e2459b0 2024-04-23T14:46:19.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.916+0000 7fdd2e5ca740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdd241b37b0 con 0x55d14e245090 2024-04-23T14:46:19.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.916+0000 7fdd2e5ca740 10 client.0 _ll_register_callbacks cb 0x7fdd18005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:46:19.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.916+0000 7fdd23e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14e2459b0 0x55d14e253a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:19.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.916+0000 7fdd29800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14e245090 0x55d14e258690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:19.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.916+0000 7fdd23e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14e2459b0 0x55d14e253a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:33930/0 (socket says 172.21.15.28:33930) 2024-04-23T14:46:19.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.917+0000 7fdd23e00640 1 -- 172.21.15.28:0/3153128616 learned_addr learned my addr 172.21.15.28:0/3153128616 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:19.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.917+0000 7fdd29800640 1 --2- 172.21.15.28:0/3153128616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d14e250080 0x55d14e253fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:19.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.917+0000 7fdd29800640 1 -- 172.21.15.28:0/3153128616 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14e2459b0 msgr2=0x55d14e253a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:19.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.917+0000 7fdd29800640 1 --2- 172.21.15.28:0/3153128616 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14e2459b0 0x55d14e253a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:19.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.917+0000 7fdd29800640 1 -- 172.21.15.28:0/3153128616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d14e250080 msgr2=0x55d14e253fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:19.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.917+0000 7fdd29800640 1 --2- 172.21.15.28:0/3153128616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d14e250080 0x55d14e253fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:19.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.917+0000 7fdd29800640 1 -- 172.21.15.28:0/3153128616 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d14e254800 con 0x55d14e245090 2024-04-23T14:46:19.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.917+0000 7fdd23e00640 1 --2- 172.21.15.28:0/3153128616 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14e2459b0 0x55d14e253a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:46:19.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.917+0000 7fdd29800640 1 --2- 172.21.15.28:0/3153128616 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14e245090 0x55d14e258690 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c003e50 tx=0x7fdd1c011dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=663a4d0a624813fe server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:19.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.917+0000 7fdd0aa00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:46:19.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.917+0000 7fdd0aa00640 1 -- 172.21.15.28:0/3153128616 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c02d040 con 0x55d14e245090 2024-04-23T14:46:19.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.917+0000 7fdd0aa00640 1 -- 172.21.15.28:0/3153128616 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fdd1c004410 con 0x55d14e245090 2024-04-23T14:46:19.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.918+0000 7fdd2e5ca740 1 -- 172.21.15.28:0/3153128616 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd1004b070 con 0x55d14e245090 2024-04-23T14:46:19.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.918+0000 7fdd0aa00640 1 -- 172.21.15.28:0/3153128616 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fdd1c0286b0 con 0x55d14e245090 2024-04-23T14:46:19.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.918+0000 7fdd2e5ca740 10 client.25666 fetch_fsmap learned FSMap version 20 2024-04-23T14:46:19.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.918+0000 7fdd2e5ca740 1 -- 172.21.15.28:0/3153128616 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55d14e2c6640 con 0x55d14e245090 2024-04-23T14:46:19.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.918+0000 7fdd0aa00640 1 -- 172.21.15.28:0/3153128616 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd1004b070 con 0x55d14e245090 2024-04-23T14:46:19.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.918+0000 7fdd0aa00640 1 -- 172.21.15.28:0/3153128616 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fdd1c005020 con 0x55d14e245090 2024-04-23T14:46:19.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.918+0000 7fdd2e5ca740 10 client.25666 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:46:19.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.918+0000 7fdd2e5ca740 10 client.25666 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:46:19.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.918+0000 7fdd2e5ca740 20 client.25666 populate_metadata read hostname 'smithi028' 2024-04-23T14:46:19.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.919+0000 7fdd2e5ca740 1 --2- 172.21.15.28:0/3153128616 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d14e2c6d70 0x55d14e2c9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:19.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.919+0000 7fdd2e5ca740 4 client.25666 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-23T14:46:19.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.919+0000 7fdd2e5ca740 1 -- 172.21.15.28:0/3153128616 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d14e1b2700 con 0x55d14e2c6d70 2024-04-23T14:46:19.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.919+0000 7fdd23e00640 1 --2- 172.21.15.28:0/3153128616 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d14e2c6d70 0x55d14e2c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:19.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.919+0000 7fdd23e00640 1 --2- 172.21.15.28:0/3153128616 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d14e2c6d70 0x55d14e2c9190 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47ebd0b275f8c71 server_cookie=33ef1e235f8b9955 in_seq=0 out_seq=0 2024-04-23T14:46:19.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.919+0000 7fdd0aa00640 10 client.25666 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:46:19.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.921+0000 7fdd0aa00640 1 -- 172.21.15.28:0/3153128616 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d14e1b2700 con 0x55d14e2c6d70 2024-04-23T14:46:19.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.921+0000 7fdd0aa00640 10 client.25666 handle_command_reply: tid=0 2024-04-23T14:46:19.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.921+0000 7fdd2e5ca740 1 client.25666 shutdown 2024-04-23T14:46:19.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.921+0000 7fdd2e5ca740 1 -- 172.21.15.28:0/3153128616 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14e245090 msgr2=0x55d14e258690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:19.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.921+0000 7fdd2e5ca740 1 --2- 172.21.15.28:0/3153128616 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14e245090 0x55d14e258690 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c003e50 tx=0x7fdd1c011dd0 comp rx=0 tx=0).stop 2024-04-23T14:46:19.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.922+0000 7fdd2e5ca740 1 -- 172.21.15.28:0/3153128616 shutdown_connections 2024-04-23T14:46:19.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.922+0000 7fdd2e5ca740 1 --2- 172.21.15.28:0/3153128616 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d14e2c6d70 0x55d14e2c9190 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:19.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.922+0000 7fdd2e5ca740 1 --2- 172.21.15.28:0/3153128616 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14e2459b0 0x55d14e253a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:19.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.922+0000 7fdd2e5ca740 1 --2- 172.21.15.28:0/3153128616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d14e250080 0x55d14e253fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:19.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.922+0000 7fdd2e5ca740 1 --2- 172.21.15.28:0/3153128616 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14e245090 0x55d14e258690 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:19.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.922+0000 7fdd0aa00640 0 client.25666 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:46:19.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.922+0000 7fdd2e5ca740 1 -- 172.21.15.28:0/3153128616 >> 172.21.15.28:0/3153128616 conn(0x55d14e2413b0 msgr2=0x55d14e23fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:19.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.922+0000 7fdd2e5ca740 1 -- 172.21.15.28:0/3153128616 shutdown_connections 2024-04-23T14:46:19.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.922+0000 7fdd2e5ca740 1 -- 172.21.15.28:0/3153128616 wait complete. 2024-04-23T14:46:19.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.922+0000 7fdd2e5ca740 20 client.25666 trim_cache size 0 max 16384 2024-04-23T14:46:19.925 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:46:19.925 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-23T14:46:19.925 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "be072548-93ec-49ad-af92-0e7903d67b90", 2024-04-23T14:46:19.925 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-23T14:46:19.925 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:46:19.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.923+0000 7fdd2b800640 1 -- 172.21.15.28:0/1365529891 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fdd0c0a3d20 msgr2=0x7fdd0c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:19.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.923+0000 7fdd2b800640 1 --2- 172.21.15.28:0/1365529891 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fdd0c0a3d20 0x7fdd0c0a61e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdd180ce2c0 tx=0x7fdd1800ef90 comp rx=0 tx=0).stop 2024-04-23T14:46:19.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.923+0000 7fdd2b800640 1 -- 172.21.15.28:0/1365529891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd240742d0 msgr2=0x7fdd2419ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:19.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.923+0000 7fdd2b800640 1 --2- 172.21.15.28:0/1365529891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd240742d0 0x7fdd2419ef20 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fdd1800ea60 tx=0x7fdd1800ef20 comp rx=0 tx=0).stop 2024-04-23T14:46:19.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.923+0000 7fdd2b800640 1 -- 172.21.15.28:0/1365529891 shutdown_connections 2024-04-23T14:46:19.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.923+0000 7fdd2b800640 1 --2- 172.21.15.28:0/1365529891 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fdd0c0a3d20 0x7fdd0c0a61e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:19.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.923+0000 7fdd2b800640 1 --2- 172.21.15.28:0/1365529891 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd241130b0 0x7fdd241a37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:19.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.923+0000 7fdd2b800640 1 --2- 172.21.15.28:0/1365529891 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd24074bf0 0x7fdd2419f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:19.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.923+0000 7fdd2b800640 1 --2- 172.21.15.28:0/1365529891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd240742d0 0x7fdd2419ef20 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:19.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.923+0000 7fdd2b800640 1 -- 172.21.15.28:0/1365529891 >> 172.21.15.28:0/1365529891 conn(0x7fdd24079450 msgr2=0x7fdd240fef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:19.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.924+0000 7fdd2b800640 1 -- 172.21.15.28:0/1365529891 shutdown_connections 2024-04-23T14:46:19.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:19.924+0000 7fdd2b800640 1 -- 172.21.15.28:0/1365529891 wait complete. 2024-04-23T14:46:19.945 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:46:19.945 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:46:20.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:19 smithi028 ceph-mon[27810]: pgmap v72: 97 pgs: 97 active+clean; 56 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 2.8 MiB/s wr, 124 op/s 2024-04-23T14:46:20.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:19 smithi038 ceph-mon[32348]: pgmap v72: 97 pgs: 97 active+clean; 56 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 2.8 MiB/s wr, 124 op/s 2024-04-23T14:46:20.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.155+0000 7f526e400640 1 -- 172.21.15.28:0/3307490551 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5268074cd0 msgr2=0x7f526810e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:20.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.155+0000 7f526e400640 1 --2- 172.21.15.28:0/3307490551 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5268074cd0 0x7f526810e220 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5250009900 tx=0x7f525002a480 comp rx=0 tx=0).stop 2024-04-23T14:46:20.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.155+0000 7f526e400640 1 -- 172.21.15.28:0/3307490551 shutdown_connections 2024-04-23T14:46:20.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.155+0000 7f526e400640 1 --2- 172.21.15.28:0/3307490551 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5268074cd0 0x7f526810e220 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:20.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.155+0000 7f526e400640 1 --2- 172.21.15.28:0/3307490551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f526810e760 0x7f5268110b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:20.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.155+0000 7f526e400640 1 --2- 172.21.15.28:0/3307490551 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52680743b0 0x7f5268074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:20.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.155+0000 7f526e400640 1 -- 172.21.15.28:0/3307490551 >> 172.21.15.28:0/3307490551 conn(0x7f52680fd9d0 msgr2=0x7f52680ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:20.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.155+0000 7f526e400640 1 -- 172.21.15.28:0/3307490551 shutdown_connections 2024-04-23T14:46:20.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.155+0000 7f526e400640 1 -- 172.21.15.28:0/3307490551 wait complete. 2024-04-23T14:46:20.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.157+0000 7f526e400640 1 Processor -- start 2024-04-23T14:46:20.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.157+0000 7f526e400640 1 -- start start 2024-04-23T14:46:20.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.158+0000 7f526e400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52680743b0 0x7f52681a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:20.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.158+0000 7f526e400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5268074cd0 0x7f52681a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:20.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.158+0000 7f526e400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f526810e760 0x7f526819a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:20.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.158+0000 7f526e400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f52681144a0 con 0x7f52680743b0 2024-04-23T14:46:20.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.158+0000 7f526e400640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5268114320 con 0x7f5268074cd0 2024-04-23T14:46:20.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.158+0000 7f526e400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5268114620 con 0x7f526810e760 2024-04-23T14:46:20.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.158+0000 7f5267e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5268074cd0 0x7f52681a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:20.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.158+0000 7f526d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52680743b0 0x7f52681a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:20.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.158+0000 7f5267e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5268074cd0 0x7f52681a0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:33940/0 (socket says 172.21.15.28:33940) 2024-04-23T14:46:20.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.158+0000 7f5267e00640 1 -- 172.21.15.28:0/1957970465 learned_addr learned my addr 172.21.15.28:0/1957970465 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:20.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.158+0000 7f526da00640 1 --2- 172.21.15.28:0/1957970465 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f526810e760 0x7f526819a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:20.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.159+0000 7f5267e00640 1 -- 172.21.15.28:0/1957970465 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f526810e760 msgr2=0x7f526819a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:20.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.159+0000 7f5267e00640 1 --2- 172.21.15.28:0/1957970465 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f526810e760 0x7f526819a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:20.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.159+0000 7f5267e00640 1 -- 172.21.15.28:0/1957970465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52680743b0 msgr2=0x7f52681a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:20.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.159+0000 7f5267e00640 1 --2- 172.21.15.28:0/1957970465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52680743b0 0x7f52681a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:20.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.159+0000 7f5267e00640 1 -- 172.21.15.28:0/1957970465 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f526819aff0 con 0x7f5268074cd0 2024-04-23T14:46:20.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.159+0000 7f526da00640 1 --2- 172.21.15.28:0/1957970465 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f526810e760 0x7f526819a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:46:20.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.159+0000 7f5267e00640 1 --2- 172.21.15.28:0/1957970465 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5268074cd0 0x7f52681a0ba0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5250004770 tx=0x7f5250003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:20.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.159+0000 7f5265600640 1 -- 172.21.15.28:0/1957970465 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f525003f070 con 0x7f5268074cd0 2024-04-23T14:46:20.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.159+0000 7f5265600640 1 -- 172.21.15.28:0/1957970465 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5250031ca0 con 0x7f5268074cd0 2024-04-23T14:46:20.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.159+0000 7f526e400640 1 -- 172.21.15.28:0/1957970465 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f526819b280 con 0x7f5268074cd0 2024-04-23T14:46:20.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.159+0000 7f5265600640 1 -- 172.21.15.28:0/1957970465 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5250035030 con 0x7f5268074cd0 2024-04-23T14:46:20.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.159+0000 7f526e400640 1 -- 172.21.15.28:0/1957970465 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52681a74b0 con 0x7f5268074cd0 2024-04-23T14:46:20.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.163+0000 7f5265600640 1 -- 172.21.15.28:0/1957970465 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f5250042070 con 0x7f5268074cd0 2024-04-23T14:46:20.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.163+0000 7f52729d4740 1 Processor -- start 2024-04-23T14:46:20.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.163+0000 7f52729d4740 1 -- start start 2024-04-23T14:46:20.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.163+0000 7f52729d4740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564165d63a30 0x564165d63e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:20.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.163+0000 7f52729d4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564165d64350 0x564165d6e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:20.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.163+0000 7f52729d4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564165d6ea20 0x564165d70e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:20.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.163+0000 7f52729d4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564165cc91f0 con 0x564165d64350 2024-04-23T14:46:20.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.163+0000 7f52729d4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564165c6f350 con 0x564165d6ea20 2024-04-23T14:46:20.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.163+0000 7f52729d4740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x564165cc51b0 con 0x564165d63a30 2024-04-23T14:46:20.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.164+0000 7f526da00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564165d63a30 0x564165d63e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:20.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.164+0000 7f526d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564165d64350 0x564165d6e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:20.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.164+0000 7f526da00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564165d63a30 0x564165d63e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:49874/0 (socket says 172.21.15.28:49874) 2024-04-23T14:46:20.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.164+0000 7f526da00640 1 -- 172.21.15.28:0/3187050177 learned_addr learned my addr 172.21.15.28:0/3187050177 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:20.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.164+0000 7f5267e00640 1 --2- 172.21.15.28:0/3187050177 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564165d6ea20 0x564165d70e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:20.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.164+0000 7f526da00640 1 -- 172.21.15.28:0/3187050177 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564165d6ea20 msgr2=0x564165d70e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:20.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.164+0000 7f526da00640 1 --2- 172.21.15.28:0/3187050177 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564165d6ea20 0x564165d70e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:20.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.164+0000 7f526da00640 1 -- 172.21.15.28:0/3187050177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564165d64350 msgr2=0x564165d6e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:20.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.164+0000 7f526da00640 1 --2- 172.21.15.28:0/3187050177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564165d64350 0x564165d6e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:20.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.164+0000 7f526da00640 1 -- 172.21.15.28:0/3187050177 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564165cc3f50 con 0x564165d63a30 2024-04-23T14:46:20.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.164+0000 7f526da00640 1 --2- 172.21.15.28:0/3187050177 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564165d63a30 0x564165d63e10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5258009c20 tx=0x7f52580117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2e5004005562ae7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:20.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.165+0000 7f5265600640 1 --2- 172.21.15.28:0/1957970465 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f524c0a3d20 0x7f524c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:20.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.165+0000 7f525e000640 1 -- 172.21.15.28:0/3187050177 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5258012450 con 0x564165d63a30 2024-04-23T14:46:20.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.165+0000 7f525e000640 1 -- 172.21.15.28:0/3187050177 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5258012ad0 con 0x564165d63a30 2024-04-23T14:46:20.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.165+0000 7f525e000640 1 -- 172.21.15.28:0/3187050177 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f525800fc10 con 0x564165d63a30 2024-04-23T14:46:20.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.165+0000 7f5265600640 1 -- 172.21.15.28:0/1957970465 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f52500f0c30 con 0x7f5268074cd0 2024-04-23T14:46:20.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.165+0000 7f526d000640 1 --2- 172.21.15.28:0/1957970465 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f524c0a3d20 0x7f524c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:20.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.166+0000 7f52729d4740 1 -- 172.21.15.28:0/3187050177 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564165d63a30 msgr2=0x564165d63e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:20.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.166+0000 7f52729d4740 1 --2- 172.21.15.28:0/3187050177 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564165d63a30 0x564165d63e10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5258009c20 tx=0x7f52580117b0 comp rx=0 tx=0).stop 2024-04-23T14:46:20.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.166+0000 7f52729d4740 1 -- 172.21.15.28:0/3187050177 shutdown_connections 2024-04-23T14:46:20.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.166+0000 7f52729d4740 1 --2- 172.21.15.28:0/3187050177 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564165d6ea20 0x564165d70e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:20.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.166+0000 7f52729d4740 1 --2- 172.21.15.28:0/3187050177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564165d64350 0x564165d6e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:20.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.166+0000 7f52729d4740 1 --2- 172.21.15.28:0/3187050177 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564165d63a30 0x564165d63e10 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:20.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.166+0000 7f52729d4740 1 -- 172.21.15.28:0/3187050177 >> 172.21.15.28:0/3187050177 conn(0x564165d5fd50 msgr2=0x564165d60130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:20.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.166+0000 7f526d000640 1 --2- 172.21.15.28:0/1957970465 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f524c0a3d20 0x7f524c0a61e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x564165d64ff0 tx=0x7f5260009680 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:20.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.166+0000 7f52729d4740 1 -- 172.21.15.28:0/3187050177 shutdown_connections 2024-04-23T14:46:20.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.166+0000 7f52729d4740 1 -- 172.21.15.28:0/3187050177 wait complete. 2024-04-23T14:46:20.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.166+0000 7f52729d4740 1 -- start start 2024-04-23T14:46:20.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.167+0000 7f52729d4740 1 Processor -- start 2024-04-23T14:46:20.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.167+0000 7f52729d4740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564165d63a30 0x564165d6ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:20.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.167+0000 7f52729d4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564165d64350 0x564165d6bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:20.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.167+0000 7f52729d4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564165d6ea20 0x564165d6c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:20.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.167+0000 7f52729d4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564165d71810 con 0x564165d6ea20 2024-04-23T14:46:20.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.167+0000 7f52729d4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564165c82690 con 0x564165d64350 2024-04-23T14:46:20.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.167+0000 7f52729d4740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x564165d71990 con 0x564165d63a30 2024-04-23T14:46:20.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.167+0000 7f52729d4740 10 client.0 _ll_register_callbacks cb 0x564165cbe650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:46:20.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.167+0000 7f526d000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564165d63a30 0x564165d6ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:20.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.167+0000 7f5267e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564165d64350 0x564165d6bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:20.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.167+0000 7f526d000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564165d63a30 0x564165d6ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:49878/0 (socket says 172.21.15.28:49878) 2024-04-23T14:46:20.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.167+0000 7f526d000640 1 -- 172.21.15.28:0/2765463849 learned_addr learned my addr 172.21.15.28:0/2765463849 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:20.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.167+0000 7f526da00640 1 --2- 172.21.15.28:0/2765463849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564165d6ea20 0x564165d6c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:20.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.168+0000 7f526d000640 1 -- 172.21.15.28:0/2765463849 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564165d64350 msgr2=0x564165d6bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:20.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.168+0000 7f526d000640 1 --2- 172.21.15.28:0/2765463849 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564165d64350 0x564165d6bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:20.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.168+0000 7f526d000640 1 -- 172.21.15.28:0/2765463849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564165d6ea20 msgr2=0x564165d6c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:20.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.168+0000 7f526d000640 1 --2- 172.21.15.28:0/2765463849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564165d6ea20 0x564165d6c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:20.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.168+0000 7f526d000640 1 -- 172.21.15.28:0/2765463849 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564165c81920 con 0x564165d63a30 2024-04-23T14:46:20.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.168+0000 7f526d000640 1 --2- 172.21.15.28:0/2765463849 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564165d63a30 0x564165d6ba40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5260013c20 tx=0x7f5260004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=237872c7e38debd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:20.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.168+0000 7f5256a00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:46:20.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.168+0000 7f5256a00640 1 -- 172.21.15.28:0/2765463849 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5260013e50 con 0x564165d63a30 2024-04-23T14:46:20.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.168+0000 7f5256a00640 1 -- 172.21.15.28:0/2765463849 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f526001c460 con 0x564165d63a30 2024-04-23T14:46:20.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.168+0000 7f52729d4740 1 -- 172.21.15.28:0/2765463849 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5258012ad0 con 0x564165d63a30 2024-04-23T14:46:20.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.169+0000 7f5256a00640 1 -- 172.21.15.28:0/2765463849 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f526001c660 con 0x564165d63a30 2024-04-23T14:46:20.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.169+0000 7f52729d4740 10 client.25678 fetch_fsmap learned FSMap version 20 2024-04-23T14:46:20.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.169+0000 7f52729d4740 1 -- 172.21.15.28:0/2765463849 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x564165aa57d0 con 0x564165d63a30 2024-04-23T14:46:20.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.169+0000 7f5256a00640 1 -- 172.21.15.28:0/2765463849 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5258012ad0 con 0x564165d63a30 2024-04-23T14:46:20.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.169+0000 7f5256a00640 1 -- 172.21.15.28:0/2765463849 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f5260025070 con 0x564165d63a30 2024-04-23T14:46:20.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.169+0000 7f52729d4740 10 client.25678 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:46:20.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.169+0000 7f52729d4740 10 client.25678 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:46:20.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.170+0000 7f52729d4740 20 client.25678 populate_metadata read hostname 'smithi028' 2024-04-23T14:46:20.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.170+0000 7f52729d4740 1 --2- 172.21.15.28:0/2765463849 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564165d7ee90 0x564165d81310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:20.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.170+0000 7f52729d4740 4 client.25678 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:46:20.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.170+0000 7f52729d4740 1 -- 172.21.15.28:0/2765463849 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564165cd6c10 con 0x564165d7ee90 2024-04-23T14:46:20.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.170+0000 7f5267e00640 1 --2- 172.21.15.28:0/2765463849 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564165d7ee90 0x564165d81310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:20.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.170+0000 7f5267e00640 1 --2- 172.21.15.28:0/2765463849 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564165d7ee90 0x564165d81310 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da33fc03ad005578 server_cookie=51466f8809702f47 in_seq=0 out_seq=0 2024-04-23T14:46:20.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.170+0000 7f5256a00640 10 client.25678 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:46:20.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.173+0000 7f5256a00640 1 -- 172.21.15.28:0/2765463849 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564165cd6c10 con 0x564165d7ee90 2024-04-23T14:46:20.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.173+0000 7f5256a00640 10 client.25678 handle_command_reply: tid=0 2024-04-23T14:46:20.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.173+0000 7f52729d4740 1 client.25678 shutdown 2024-04-23T14:46:20.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.173+0000 7f52729d4740 1 -- 172.21.15.28:0/2765463849 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564165d63a30 msgr2=0x564165d6ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:20.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.173+0000 7f52729d4740 1 --2- 172.21.15.28:0/2765463849 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564165d63a30 0x564165d6ba40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5260013c20 tx=0x7f5260004a40 comp rx=0 tx=0).stop 2024-04-23T14:46:20.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.173+0000 7f52729d4740 1 -- 172.21.15.28:0/2765463849 shutdown_connections 2024-04-23T14:46:20.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.173+0000 7f52729d4740 1 --2- 172.21.15.28:0/2765463849 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564165d7ee90 0x564165d81310 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:20.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.173+0000 7f52729d4740 1 --2- 172.21.15.28:0/2765463849 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564165d64350 0x564165d6bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:20.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.173+0000 7f52729d4740 1 --2- 172.21.15.28:0/2765463849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564165d6ea20 0x564165d6c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:20.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.173+0000 7f52729d4740 1 --2- 172.21.15.28:0/2765463849 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564165d63a30 0x564165d6ba40 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:20.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.173+0000 7f5256a00640 0 client.25678 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:46:20.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.173+0000 7f52729d4740 1 -- 172.21.15.28:0/2765463849 >> 172.21.15.28:0/2765463849 conn(0x564165d5fd50 msgr2=0x564165d65b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:20.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.173+0000 7f52729d4740 1 -- 172.21.15.28:0/2765463849 shutdown_connections 2024-04-23T14:46:20.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.173+0000 7f52729d4740 1 -- 172.21.15.28:0/2765463849 wait complete. 2024-04-23T14:46:20.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.173+0000 7f52729d4740 20 client.25678 trim_cache size 0 max 16384 2024-04-23T14:46:20.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.194+0000 7f52729d4740 1 Processor -- start 2024-04-23T14:46:20.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.194+0000 7f52729d4740 1 -- start start 2024-04-23T14:46:20.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.195+0000 7f52729d4740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564165d63a30 0x564165d7a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:20.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.195+0000 7f52729d4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564165d64350 0x564165d662a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:20.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.195+0000 7f52729d4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564165d6ea20 0x564165d5ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:20.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.195+0000 7f52729d4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f52681addf0 con 0x564165d6ea20 2024-04-23T14:46:20.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.195+0000 7f52729d4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f52681b0670 con 0x564165d64350 2024-04-23T14:46:20.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.195+0000 7f52729d4740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f52681ae460 con 0x564165d63a30 2024-04-23T14:46:20.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.195+0000 7f5267e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564165d64350 0x564165d662a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:20.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.195+0000 7f5267e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564165d64350 0x564165d662a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:33972/0 (socket says 172.21.15.28:33972) 2024-04-23T14:46:20.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.195+0000 7f5267e00640 1 -- 172.21.15.28:0/2905812668 learned_addr learned my addr 172.21.15.28:0/2905812668 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:20.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.195+0000 7f526da00640 1 --2- 172.21.15.28:0/2905812668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564165d6ea20 0x564165d5ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:20.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.195+0000 7f526d000640 1 --2- 172.21.15.28:0/2905812668 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564165d63a30 0x564165d7a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:20.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.196+0000 7f5267e00640 1 -- 172.21.15.28:0/2905812668 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564165d63a30 msgr2=0x564165d7a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:20.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.196+0000 7f5267e00640 1 --2- 172.21.15.28:0/2905812668 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564165d63a30 0x564165d7a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:20.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.196+0000 7f5267e00640 1 -- 172.21.15.28:0/2905812668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564165d6ea20 msgr2=0x564165d5ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:20.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.196+0000 7f5267e00640 1 --2- 172.21.15.28:0/2905812668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564165d6ea20 0x564165d5ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:20.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.196+0000 7f5267e00640 1 -- 172.21.15.28:0/2905812668 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641659db3f0 con 0x564165d64350 2024-04-23T14:46:20.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.196+0000 7f526d000640 1 --2- 172.21.15.28:0/2905812668 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564165d63a30 0x564165d7a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:46:20.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.196+0000 7f5267e00640 1 --2- 172.21.15.28:0/2905812668 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564165d64350 0x564165d662a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f52500ee390 tx=0x7f52500dd040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51399f8cf74e9fac server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:20.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.196+0000 7f525d600640 1 -- 172.21.15.28:0/2905812668 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5250049040 con 0x564165d64350 2024-04-23T14:46:20.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.196+0000 7f525d600640 1 -- 172.21.15.28:0/2905812668 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5250031e50 con 0x564165d64350 2024-04-23T14:46:20.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.196+0000 7f525d600640 1 -- 172.21.15.28:0/2905812668 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52500c57f0 con 0x564165d64350 2024-04-23T14:46:20.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.197+0000 7f52729d4740 1 -- 172.21.15.28:0/2905812668 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564165d64350 msgr2=0x564165d662a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:20.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.197+0000 7f52729d4740 1 --2- 172.21.15.28:0/2905812668 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564165d64350 0x564165d662a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f52500ee390 tx=0x7f52500dd040 comp rx=0 tx=0).stop 2024-04-23T14:46:20.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.197+0000 7f52729d4740 1 -- 172.21.15.28:0/2905812668 shutdown_connections 2024-04-23T14:46:20.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.197+0000 7f52729d4740 1 --2- 172.21.15.28:0/2905812668 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564165d64350 0x564165d662a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:20.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.197+0000 7f52729d4740 1 --2- 172.21.15.28:0/2905812668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564165d6ea20 0x564165d5ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:20.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.197+0000 7f52729d4740 1 --2- 172.21.15.28:0/2905812668 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564165d63a30 0x564165d7a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:20.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.198+0000 7f52729d4740 1 -- 172.21.15.28:0/2905812668 >> 172.21.15.28:0/2905812668 conn(0x564165d5fd50 msgr2=0x564165d5e740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:20.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.198+0000 7f52729d4740 1 -- 172.21.15.28:0/2905812668 shutdown_connections 2024-04-23T14:46:20.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.198+0000 7f52729d4740 1 -- 172.21.15.28:0/2905812668 wait complete. 2024-04-23T14:46:20.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.198+0000 7f52729d4740 1 -- start start 2024-04-23T14:46:20.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.198+0000 7f52729d4740 1 Processor -- start 2024-04-23T14:46:20.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.199+0000 7f52729d4740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564165d63a30 0x564165d769a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:20.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.199+0000 7f52729d4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564165d64350 0x564165d76ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:20.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.199+0000 7f52729d4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564165d6ea20 0x564165d71da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:20.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.199+0000 7f52729d4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f52681aeb30 con 0x564165d6ea20 2024-04-23T14:46:20.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.199+0000 7f52729d4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f52681affa0 con 0x564165d64350 2024-04-23T14:46:20.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.199+0000 7f52729d4740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f52681af200 con 0x564165d63a30 2024-04-23T14:46:20.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.199+0000 7f52729d4740 10 client.0 _ll_register_callbacks cb 0x7f52600138d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:46:20.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.199+0000 7f5267e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564165d64350 0x564165d76ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:20.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.199+0000 7f526d000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564165d63a30 0x564165d769a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:20.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.199+0000 7f5267e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564165d64350 0x564165d76ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:33980/0 (socket says 172.21.15.28:33980) 2024-04-23T14:46:20.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.199+0000 7f526d000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564165d63a30 0x564165d769a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:49902/0 (socket says 172.21.15.28:49902) 2024-04-23T14:46:20.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.199+0000 7f5267e00640 1 -- 172.21.15.28:0/3573573013 learned_addr learned my addr 172.21.15.28:0/3573573013 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:20.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.199+0000 7f526da00640 1 --2- 172.21.15.28:0/3573573013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564165d6ea20 0x564165d71da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:20.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.200+0000 7f526d000640 1 -- 172.21.15.28:0/3573573013 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564165d64350 msgr2=0x564165d76ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:20.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.200+0000 7f526d000640 1 --2- 172.21.15.28:0/3573573013 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564165d64350 0x564165d76ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:20.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.200+0000 7f526d000640 1 -- 172.21.15.28:0/3573573013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564165d6ea20 msgr2=0x564165d71da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:20.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.200+0000 7f526d000640 1 --2- 172.21.15.28:0/3573573013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564165d6ea20 0x564165d71da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:20.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.200+0000 7f526d000640 1 -- 172.21.15.28:0/3573573013 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56416599b0f0 con 0x564165d63a30 2024-04-23T14:46:20.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.200+0000 7f5267e00640 1 --2- 172.21.15.28:0/3573573013 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564165d64350 0x564165d76ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:46:20.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.200+0000 7f526d000640 1 --2- 172.21.15.28:0/3573573013 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564165d63a30 0x564165d769a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5260027d10 tx=0x7f5260004d60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d02aad8a03381ac server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:20.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.200+0000 7f5256a00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:46:20.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.200+0000 7f5256a00640 1 -- 172.21.15.28:0/3573573013 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f526003c060 con 0x564165d63a30 2024-04-23T14:46:20.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.200+0000 7f52729d4740 1 -- 172.21.15.28:0/3573573013 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5250031e50 con 0x564165d63a30 2024-04-23T14:46:20.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.200+0000 7f5256a00640 1 -- 172.21.15.28:0/3573573013 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f526001c460 con 0x564165d63a30 2024-04-23T14:46:20.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.200+0000 7f5256a00640 1 -- 172.21.15.28:0/3573573013 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f5260004870 con 0x564165d63a30 2024-04-23T14:46:20.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.200+0000 7f52729d4740 10 client.25684 fetch_fsmap learned FSMap version 20 2024-04-23T14:46:20.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.201+0000 7f52729d4740 1 -- 172.21.15.28:0/3573573013 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x564165c8e810 con 0x564165d63a30 2024-04-23T14:46:20.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.201+0000 7f5256a00640 1 -- 172.21.15.28:0/3573573013 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5250031e50 con 0x564165d63a30 2024-04-23T14:46:20.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.201+0000 7f5256a00640 1 -- 172.21.15.28:0/3573573013 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f5260025040 con 0x564165d63a30 2024-04-23T14:46:20.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.201+0000 7f52729d4740 10 client.25684 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:46:20.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.201+0000 7f52729d4740 10 client.25684 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:46:20.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.201+0000 7f52729d4740 20 client.25684 populate_metadata read hostname 'smithi028' 2024-04-23T14:46:20.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.201+0000 7f52729d4740 1 --2- 172.21.15.28:0/3573573013 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564165deda80 0x564165defe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:20.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.201+0000 7f52729d4740 4 client.25684 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:46:20.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.201+0000 7f52729d4740 1 -- 172.21.15.28:0/3573573013 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564165cd0a80 con 0x564165deda80 2024-04-23T14:46:20.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.201+0000 7f5267e00640 1 --2- 172.21.15.28:0/3573573013 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564165deda80 0x564165defe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:20.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.202+0000 7f5267e00640 1 --2- 172.21.15.28:0/3573573013 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564165deda80 0x564165defe70 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba0762228d5cd17b server_cookie=b0b4b8108e7068b7 in_seq=0 out_seq=0 2024-04-23T14:46:20.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.202+0000 7f5256a00640 10 client.25684 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:46:20.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.204+0000 7f5256a00640 1 -- 172.21.15.28:0/3573573013 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564165cd0a80 con 0x564165deda80 2024-04-23T14:46:20.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.204+0000 7f5256a00640 10 client.25684 handle_command_reply: tid=0 2024-04-23T14:46:20.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.204+0000 7f52729d4740 1 client.25684 shutdown 2024-04-23T14:46:20.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.204+0000 7f52729d4740 1 -- 172.21.15.28:0/3573573013 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564165d63a30 msgr2=0x564165d769a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:20.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.204+0000 7f52729d4740 1 --2- 172.21.15.28:0/3573573013 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564165d63a30 0x564165d769a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5260027d10 tx=0x7f5260004d60 comp rx=0 tx=0).stop 2024-04-23T14:46:20.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.204+0000 7f52729d4740 1 -- 172.21.15.28:0/3573573013 shutdown_connections 2024-04-23T14:46:20.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.204+0000 7f52729d4740 1 --2- 172.21.15.28:0/3573573013 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564165deda80 0x564165defe70 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:20.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.204+0000 7f52729d4740 1 --2- 172.21.15.28:0/3573573013 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564165d64350 0x564165d76ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:20.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.204+0000 7f52729d4740 1 --2- 172.21.15.28:0/3573573013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564165d6ea20 0x564165d71da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:20.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.204+0000 7f52729d4740 1 --2- 172.21.15.28:0/3573573013 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564165d63a30 0x564165d769a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:20.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.204+0000 7f5256a00640 0 client.25684 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:46:20.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.204+0000 7f52729d4740 1 -- 172.21.15.28:0/3573573013 >> 172.21.15.28:0/3573573013 conn(0x564165d5fd50 msgr2=0x564165d5efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:20.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.204+0000 7f52729d4740 1 -- 172.21.15.28:0/3573573013 shutdown_connections 2024-04-23T14:46:20.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.204+0000 7f52729d4740 1 -- 172.21.15.28:0/3573573013 wait complete. 2024-04-23T14:46:20.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.204+0000 7f52729d4740 20 client.25684 trim_cache size 0 max 16384 2024-04-23T14:46:20.208 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:46:20.208 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-23T14:46:20.208 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-23T14:46:20.208 INFO:teuthology.orchestra.run.smithi028.stdout: "be072548-93ec-49ad-af92-0e7903d67b90": { 2024-04-23T14:46:20.208 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-23T14:46:20.208 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "be072548-93ec-49ad-af92-0e7903d67b90", 2024-04-23T14:46:20.208 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-23T14:46:20.208 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:46:20.208 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:46:20.208 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:46:20.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.205+0000 7f526e400640 1 -- 172.21.15.28:0/1957970465 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f524c0a3d20 msgr2=0x7f524c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:20.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.205+0000 7f526e400640 1 --2- 172.21.15.28:0/1957970465 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f524c0a3d20 0x7f524c0a61e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x564165d64ff0 tx=0x7f5260009680 comp rx=0 tx=0).stop 2024-04-23T14:46:20.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.205+0000 7f526e400640 1 -- 172.21.15.28:0/1957970465 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5268074cd0 msgr2=0x7f52681a0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:20.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.205+0000 7f526e400640 1 --2- 172.21.15.28:0/1957970465 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5268074cd0 0x7f52681a0ba0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5250004770 tx=0x7f5250003c10 comp rx=0 tx=0).stop 2024-04-23T14:46:20.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.206+0000 7f526e400640 1 -- 172.21.15.28:0/1957970465 shutdown_connections 2024-04-23T14:46:20.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.206+0000 7f526e400640 1 --2- 172.21.15.28:0/1957970465 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f524c0a3d20 0x7f524c0a61e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:20.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.206+0000 7f526e400640 1 --2- 172.21.15.28:0/1957970465 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5268074cd0 0x7f52681a0ba0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:20.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.206+0000 7f526e400640 1 --2- 172.21.15.28:0/1957970465 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f526810e760 0x7f526819a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:20.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.206+0000 7f526e400640 1 --2- 172.21.15.28:0/1957970465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52680743b0 0x7f52681a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:20.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.206+0000 7f526e400640 1 -- 172.21.15.28:0/1957970465 >> 172.21.15.28:0/1957970465 conn(0x7f52680fd9d0 msgr2=0x7f52680fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:20.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.206+0000 7f526e400640 1 -- 172.21.15.28:0/1957970465 shutdown_connections 2024-04-23T14:46:20.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:20.206+0000 7f526e400640 1 -- 172.21.15.28:0/1957970465 wait complete. 2024-04-23T14:46:20.226 INFO:tasks.cephfs.filesystem:scrub status for tag:be072548-93ec-49ad-af92-0e7903d67b90 - {'path': '/', 'tag': 'be072548-93ec-49ad-af92-0e7903d67b90', 'options': 'recursive,force'} 2024-04-23T14:46:20.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:19 smithi105 ceph-mon[32555]: pgmap v72: 97 pgs: 97 active+clean; 56 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 2.8 MiB/s wr, 124 op/s 2024-04-23T14:46:20.464 INFO:tasks.workunit.client.0.smithi028.stdout: 1 71942 11.23 MB/sec warmup 36 sec latency 14.120 ms 2024-04-23T14:46:21.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:20 smithi105 ceph-mon[32555]: scrub queued for path: / 2024-04-23T14:46:21.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:20 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:46:21.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:20 smithi105 ceph-mon[32555]: scrub summary: active paths [/] 2024-04-23T14:46:21.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:20 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:46:21.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:20 smithi038 ceph-mon[32348]: scrub queued for path: / 2024-04-23T14:46:21.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:20 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:46:21.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:20 smithi038 ceph-mon[32348]: scrub summary: active paths [/] 2024-04-23T14:46:21.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:20 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:46:21.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:20 smithi028 ceph-mon[27810]: scrub queued for path: / 2024-04-23T14:46:21.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:20 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:46:21.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:20 smithi028 ceph-mon[27810]: scrub summary: active paths [/] 2024-04-23T14:46:21.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:20 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:46:21.464 INFO:tasks.workunit.client.0.smithi028.stdout: 1 74387 11.30 MB/sec warmup 37 sec latency 44.559 ms 2024-04-23T14:46:22.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:21 smithi105 ceph-mon[32555]: pgmap v73: 97 pgs: 97 active+clean; 56 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 2.8 MiB/s wr, 124 op/s 2024-04-23T14:46:22.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:21 smithi038 ceph-mon[32348]: pgmap v73: 97 pgs: 97 active+clean; 56 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 2.8 MiB/s wr, 124 op/s 2024-04-23T14:46:22.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:21 smithi028 ceph-mon[27810]: pgmap v73: 97 pgs: 97 active+clean; 56 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 2.8 MiB/s wr, 124 op/s 2024-04-23T14:46:22.464 INFO:tasks.workunit.client.0.smithi028.stdout: 1 76192 11.26 MB/sec warmup 38 sec latency 11.864 ms 2024-04-23T14:46:23.464 INFO:tasks.workunit.client.0.smithi028.stdout: 1 78513 11.32 MB/sec warmup 39 sec latency 10.704 ms 2024-04-23T14:46:24.464 INFO:tasks.workunit.client.0.smithi028.stdout: 1 80859 11.32 MB/sec warmup 40 sec latency 11.267 ms 2024-04-23T14:46:24.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:24 smithi105 ceph-mon[32555]: pgmap v74: 97 pgs: 97 active+clean; 60 MiB data, 594 MiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 4.2 MiB/s wr, 211 op/s 2024-04-23T14:46:24.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:24 smithi038 ceph-mon[32348]: pgmap v74: 97 pgs: 97 active+clean; 60 MiB data, 594 MiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 4.2 MiB/s wr, 211 op/s 2024-04-23T14:46:24.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:24 smithi028 ceph-mon[27810]: pgmap v74: 97 pgs: 97 active+clean; 60 MiB data, 594 MiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 4.2 MiB/s wr, 211 op/s 2024-04-23T14:46:25.464 INFO:tasks.workunit.client.0.smithi028.stdout: 1 82614 11.26 MB/sec warmup 41 sec latency 12.156 ms 2024-04-23T14:46:25.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:25 smithi105 ceph-mon[32555]: scrub summary: idle 2024-04-23T14:46:25.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:25 smithi038 ceph-mon[32348]: scrub summary: idle 2024-04-23T14:46:25.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:25 smithi028 ceph-mon[27810]: scrub summary: idle 2024-04-23T14:46:26.465 INFO:tasks.workunit.client.0.smithi028.stdout: 1 84984 11.31 MB/sec warmup 42 sec latency 98.362 ms 2024-04-23T14:46:26.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:26 smithi105 ceph-mon[32555]: pgmap v75: 97 pgs: 97 active+clean; 60 MiB data, 594 MiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 3.7 MiB/s wr, 156 op/s 2024-04-23T14:46:26.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:26 smithi038 ceph-mon[32348]: pgmap v75: 97 pgs: 97 active+clean; 60 MiB data, 594 MiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 3.7 MiB/s wr, 156 op/s 2024-04-23T14:46:26.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:26 smithi028 ceph-mon[27810]: pgmap v75: 97 pgs: 97 active+clean; 60 MiB data, 594 MiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 3.7 MiB/s wr, 156 op/s 2024-04-23T14:46:27.465 INFO:tasks.workunit.client.0.smithi028.stdout: 1 86629 11.28 MB/sec warmup 43 sec latency 33.427 ms 2024-04-23T14:46:28.465 INFO:tasks.workunit.client.0.smithi028.stdout: 1 89113 11.34 MB/sec warmup 44 sec latency 10.139 ms 2024-04-23T14:46:28.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:28 smithi105 ceph-mon[32555]: pgmap v76: 97 pgs: 97 active+clean; 64 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 1012 KiB/s rd, 5.1 MiB/s wr, 245 op/s 2024-04-23T14:46:28.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:28 smithi038 ceph-mon[32348]: pgmap v76: 97 pgs: 97 active+clean; 64 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 1012 KiB/s rd, 5.1 MiB/s wr, 245 op/s 2024-04-23T14:46:28.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:28 smithi028 ceph-mon[27810]: pgmap v76: 97 pgs: 97 active+clean; 64 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 1012 KiB/s rd, 5.1 MiB/s wr, 245 op/s 2024-04-23T14:46:29.465 INFO:tasks.workunit.client.0.smithi028.stdout: 1 91322 11.27 MB/sec warmup 45 sec latency 12.603 ms 2024-04-23T14:46:30.465 INFO:tasks.workunit.client.0.smithi028.stdout: 1 93014 11.27 MB/sec warmup 46 sec latency 58.534 ms 2024-04-23T14:46:30.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:30 smithi105 ceph-mon[32555]: pgmap v77: 97 pgs: 97 active+clean; 64 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 2.8 MiB/s wr, 176 op/s 2024-04-23T14:46:30.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:30 smithi038 ceph-mon[32348]: pgmap v77: 97 pgs: 97 active+clean; 64 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 2.8 MiB/s wr, 176 op/s 2024-04-23T14:46:30.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:30 smithi028 ceph-mon[27810]: pgmap v77: 97 pgs: 97 active+clean; 64 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 2.8 MiB/s wr, 176 op/s 2024-04-23T14:46:31.465 INFO:tasks.workunit.client.0.smithi028.stdout: 1 95263 11.29 MB/sec warmup 47 sec latency 82.869 ms 2024-04-23T14:46:32.465 INFO:tasks.workunit.client.0.smithi028.stdout: 1 97067 11.24 MB/sec warmup 48 sec latency 14.491 ms 2024-04-23T14:46:32.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:32 smithi105 ceph-mon[32555]: pgmap v78: 97 pgs: 97 active+clean; 64 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 2.8 MiB/s wr, 176 op/s 2024-04-23T14:46:32.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:32 smithi038 ceph-mon[32348]: pgmap v78: 97 pgs: 97 active+clean; 64 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 2.8 MiB/s wr, 176 op/s 2024-04-23T14:46:32.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:32 smithi028 ceph-mon[27810]: pgmap v78: 97 pgs: 97 active+clean; 64 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 2.8 MiB/s wr, 176 op/s 2024-04-23T14:46:33.466 INFO:tasks.workunit.client.0.smithi028.stdout: 1 99693 11.35 MB/sec warmup 49 sec latency 16.589 ms 2024-04-23T14:46:34.466 INFO:tasks.workunit.client.0.smithi028.stdout: 1 101795 11.27 MB/sec warmup 50 sec latency 18.015 ms 2024-04-23T14:46:34.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:34 smithi105 ceph-mon[32555]: pgmap v79: 97 pgs: 97 active+clean; 69 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.8 MiB/s wr, 262 op/s 2024-04-23T14:46:34.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:34 smithi038 ceph-mon[32348]: pgmap v79: 97 pgs: 97 active+clean; 69 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.8 MiB/s wr, 262 op/s 2024-04-23T14:46:34.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:34 smithi028 ceph-mon[27810]: pgmap v79: 97 pgs: 97 active+clean; 69 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.8 MiB/s wr, 262 op/s 2024-04-23T14:46:35.466 INFO:tasks.workunit.client.0.smithi028.stdout: 1 103485 11.28 MB/sec warmup 51 sec latency 10.594 ms 2024-04-23T14:46:36.466 INFO:tasks.workunit.client.0.smithi028.stdout: 1 105359 11.21 MB/sec warmup 52 sec latency 51.167 ms 2024-04-23T14:46:36.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:36 smithi105 ceph-mon[32555]: pgmap v80: 97 pgs: 97 active+clean; 69 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 2.4 MiB/s wr, 175 op/s 2024-04-23T14:46:36.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:36 smithi038 ceph-mon[32348]: pgmap v80: 97 pgs: 97 active+clean; 69 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 2.4 MiB/s wr, 175 op/s 2024-04-23T14:46:36.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:36 smithi028 ceph-mon[27810]: pgmap v80: 97 pgs: 97 active+clean; 69 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 2.4 MiB/s wr, 175 op/s 2024-04-23T14:46:37.466 INFO:tasks.workunit.client.0.smithi028.stdout: 1 106983 11.22 MB/sec warmup 53 sec latency 10.741 ms 2024-04-23T14:46:38.466 INFO:tasks.workunit.client.0.smithi028.stdout: 1 109026 11.16 MB/sec warmup 54 sec latency 14.882 ms 2024-04-23T14:46:38.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:38 smithi105 ceph-mon[32555]: pgmap v81: 97 pgs: 97 active+clean; 72 MiB data, 698 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.6 MiB/s wr, 256 op/s 2024-04-23T14:46:38.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:38 smithi038 ceph-mon[32348]: pgmap v81: 97 pgs: 97 active+clean; 72 MiB data, 698 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.6 MiB/s wr, 256 op/s 2024-04-23T14:46:38.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:38 smithi028 ceph-mon[27810]: pgmap v81: 97 pgs: 97 active+clean; 72 MiB data, 698 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.6 MiB/s wr, 256 op/s 2024-04-23T14:46:39.466 INFO:tasks.workunit.client.0.smithi028.stdout: 1 110540 11.16 MB/sec warmup 55 sec latency 28.450 ms 2024-04-23T14:46:40.466 INFO:tasks.workunit.client.0.smithi028.stdout: 1 112478 11.10 MB/sec warmup 56 sec latency 24.778 ms 2024-04-23T14:46:40.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:40 smithi105 ceph-mon[32555]: pgmap v82: 97 pgs: 97 active+clean; 72 MiB data, 698 MiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 2.3 MiB/s wr, 167 op/s 2024-04-23T14:46:40.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:40 smithi038 ceph-mon[32348]: pgmap v82: 97 pgs: 97 active+clean; 72 MiB data, 698 MiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 2.3 MiB/s wr, 167 op/s 2024-04-23T14:46:40.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:40 smithi028 ceph-mon[27810]: pgmap v82: 97 pgs: 97 active+clean; 72 MiB data, 698 MiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 2.3 MiB/s wr, 167 op/s 2024-04-23T14:46:41.467 INFO:tasks.workunit.client.0.smithi028.stdout: 1 113951 11.10 MB/sec warmup 57 sec latency 83.416 ms 2024-04-23T14:46:42.467 INFO:tasks.workunit.client.0.smithi028.stdout: 1 115946 11.03 MB/sec warmup 58 sec latency 14.697 ms 2024-04-23T14:46:42.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:42 smithi105 ceph-mon[32555]: pgmap v83: 97 pgs: 97 active+clean; 72 MiB data, 698 MiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 2.3 MiB/s wr, 167 op/s 2024-04-23T14:46:42.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:42 smithi038 ceph-mon[32348]: pgmap v83: 97 pgs: 97 active+clean; 72 MiB data, 698 MiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 2.3 MiB/s wr, 167 op/s 2024-04-23T14:46:42.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:42 smithi028 ceph-mon[27810]: pgmap v83: 97 pgs: 97 active+clean; 72 MiB data, 698 MiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 2.3 MiB/s wr, 167 op/s 2024-04-23T14:46:43.467 INFO:tasks.workunit.client.0.smithi028.stdout: 1 117616 11.05 MB/sec warmup 59 sec latency 9.347 ms 2024-04-23T14:46:44.467 INFO:tasks.workunit.client.0.smithi028.stdout: 1 119681 11.00 MB/sec warmup 60 sec latency 14.739 ms 2024-04-23T14:46:44.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:44 smithi105 ceph-mon[32555]: pgmap v84: 97 pgs: 97 active+clean; 76 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 998 KiB/s rd, 3.3 MiB/s wr, 243 op/s 2024-04-23T14:46:44.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:44 smithi038 ceph-mon[32348]: pgmap v84: 97 pgs: 97 active+clean; 76 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 998 KiB/s rd, 3.3 MiB/s wr, 243 op/s 2024-04-23T14:46:44.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:44 smithi028 ceph-mon[27810]: pgmap v84: 97 pgs: 97 active+clean; 76 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 998 KiB/s rd, 3.3 MiB/s wr, 243 op/s 2024-04-23T14:46:45.467 INFO:tasks.workunit.client.0.smithi028.stdout: 1 121344 11.01 MB/sec warmup 61 sec latency 15.450 ms 2024-04-23T14:46:46.467 INFO:tasks.workunit.client.0.smithi028.stdout: 1 123561 11.03 MB/sec warmup 62 sec latency 39.148 ms 2024-04-23T14:46:46.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:46 smithi105 ceph-mon[32555]: pgmap v85: 97 pgs: 97 active+clean; 76 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 2.2 MiB/s wr, 157 op/s 2024-04-23T14:46:46.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:46 smithi038 ceph-mon[32348]: pgmap v85: 97 pgs: 97 active+clean; 76 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 2.2 MiB/s wr, 157 op/s 2024-04-23T14:46:46.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:46 smithi028 ceph-mon[27810]: pgmap v85: 97 pgs: 97 active+clean; 76 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 2.2 MiB/s wr, 157 op/s 2024-04-23T14:46:47.467 INFO:tasks.workunit.client.0.smithi028.stdout: 1 125228 10.99 MB/sec warmup 63 sec latency 20.154 ms 2024-04-23T14:46:48.467 INFO:tasks.workunit.client.0.smithi028.stdout: 1 127756 11.06 MB/sec warmup 64 sec latency 10.005 ms 2024-04-23T14:46:48.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:48 smithi105 ceph-mon[32555]: pgmap v86: 97 pgs: 97 active+clean; 81 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 3.5 MiB/s wr, 237 op/s 2024-04-23T14:46:48.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:48 smithi038 ceph-mon[32348]: pgmap v86: 97 pgs: 97 active+clean; 81 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 3.5 MiB/s wr, 237 op/s 2024-04-23T14:46:48.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:48 smithi028 ceph-mon[27810]: pgmap v86: 97 pgs: 97 active+clean; 81 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 3.5 MiB/s wr, 237 op/s 2024-04-23T14:46:49.468 INFO:tasks.workunit.client.0.smithi028.stdout: 1 129285 10.99 MB/sec warmup 65 sec latency 12.757 ms 2024-04-23T14:46:50.227 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:46:50.227 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:46:50.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.448+0000 7f3484c00640 1 -- 172.21.15.28:0/1699848490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3480074d50 msgr2=0x7f348010e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:50.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.448+0000 7f3484c00640 1 --2- 172.21.15.28:0/1699848490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3480074d50 0x7f348010e1a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3468009900 tx=0x7f346802a480 comp rx=0 tx=0).stop 2024-04-23T14:46:50.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.448+0000 7f3484c00640 1 -- 172.21.15.28:0/1699848490 shutdown_connections 2024-04-23T14:46:50.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.448+0000 7f3484c00640 1 --2- 172.21.15.28:0/1699848490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3480074d50 0x7f348010e1a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.448+0000 7f3484c00640 1 --2- 172.21.15.28:0/1699848490 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f348010e6e0 0x7f3480110b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.448+0000 7f3484c00640 1 --2- 172.21.15.28:0/1699848490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3480074430 0x7f3480074810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.448+0000 7f3484c00640 1 -- 172.21.15.28:0/1699848490 >> 172.21.15.28:0/1699848490 conn(0x7f34800fd9d0 msgr2=0x7f34800ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:50.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.448+0000 7f3484c00640 1 -- 172.21.15.28:0/1699848490 shutdown_connections 2024-04-23T14:46:50.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.448+0000 7f3484c00640 1 -- 172.21.15.28:0/1699848490 wait complete. 2024-04-23T14:46:50.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.449+0000 7f3484c00640 1 Processor -- start 2024-04-23T14:46:50.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.449+0000 7f3484c00640 1 -- start start 2024-04-23T14:46:50.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.450+0000 7f3484c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3480074430 0x7f34801a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:50.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.450+0000 7f3484c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3480074d50 0x7f34801a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:50.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.450+0000 7f3484c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f348010e6e0 0x7f348019a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:50.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.450+0000 7f3484c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f34801144c0 con 0x7f3480074430 2024-04-23T14:46:50.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.450+0000 7f3484c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3480114340 con 0x7f348010e6e0 2024-04-23T14:46:50.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.450+0000 7f3484c00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3480114640 con 0x7f3480074d50 2024-04-23T14:46:50.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.450+0000 7f3476a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3480074d50 0x7f34801a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:50.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.450+0000 7f347f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3480074430 0x7f34801a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:50.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.450+0000 7f3476a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3480074d50 0x7f34801a0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:42758/0 (socket says 172.21.15.28:42758) 2024-04-23T14:46:50.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.450+0000 7f3476a00640 1 -- 172.21.15.28:0/2244203732 learned_addr learned my addr 172.21.15.28:0/2244203732 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:50.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.450+0000 7f347fe00640 1 --2- 172.21.15.28:0/2244203732 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f348010e6e0 0x7f348019a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:50.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.450+0000 7f3476a00640 1 -- 172.21.15.28:0/2244203732 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f348010e6e0 msgr2=0x7f348019a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:50.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.450+0000 7f3476a00640 1 --2- 172.21.15.28:0/2244203732 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f348010e6e0 0x7f348019a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.451+0000 7f3476a00640 1 -- 172.21.15.28:0/2244203732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3480074430 msgr2=0x7f34801a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:50.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.451+0000 7f3476a00640 1 --2- 172.21.15.28:0/2244203732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3480074430 0x7f34801a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.451+0000 7f3476a00640 1 -- 172.21.15.28:0/2244203732 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f348019afd0 con 0x7f3480074d50 2024-04-23T14:46:50.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.451+0000 7f3476a00640 1 --2- 172.21.15.28:0/2244203732 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3480074d50 0x7f34801a0b80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3468004770 tx=0x7f3468003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:50.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.451+0000 7f347cc00640 1 -- 172.21.15.28:0/2244203732 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f346803f070 con 0x7f3480074d50 2024-04-23T14:46:50.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.451+0000 7f3484c00640 1 -- 172.21.15.28:0/2244203732 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f348019b260 con 0x7f3480074d50 2024-04-23T14:46:50.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.451+0000 7f347cc00640 1 -- 172.21.15.28:0/2244203732 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3468031ca0 con 0x7f3480074d50 2024-04-23T14:46:50.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.451+0000 7f347cc00640 1 -- 172.21.15.28:0/2244203732 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3468035030 con 0x7f3480074d50 2024-04-23T14:46:50.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.451+0000 7f3484c00640 1 -- 172.21.15.28:0/2244203732 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34801a7490 con 0x7f3480074d50 2024-04-23T14:46:50.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.455+0000 7f347cc00640 1 -- 172.21.15.28:0/2244203732 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f3468042070 con 0x7f3480074d50 2024-04-23T14:46:50.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.455+0000 7f3489082740 1 Processor -- start 2024-04-23T14:46:50.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.455+0000 7f3489082740 1 -- start start 2024-04-23T14:46:50.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.455+0000 7f3489082740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56043bd21a30 0x56043bd21e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:50.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.455+0000 7f3489082740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56043bd22350 0x56043bd2c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:50.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.455+0000 7f3489082740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56043bd2ca20 0x56043bd2ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:50.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.455+0000 7f3489082740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56043bc871f0 con 0x56043bd2ca20 2024-04-23T14:46:50.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.455+0000 7f3489082740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56043bc2d350 con 0x56043bd21a30 2024-04-23T14:46:50.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.455+0000 7f3489082740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56043bc831b0 con 0x56043bd22350 2024-04-23T14:46:50.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.456+0000 7f347f400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56043bd22350 0x56043bd2c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:50.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.456+0000 7f347fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56043bd21a30 0x56043bd21e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:50.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.456+0000 7f347fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56043bd21a30 0x56043bd21e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60312/0 (socket says 172.21.15.28:60312) 2024-04-23T14:46:50.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.456+0000 7f347f400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56043bd22350 0x56043bd2c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:42766/0 (socket says 172.21.15.28:42766) 2024-04-23T14:46:50.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.456+0000 7f347fe00640 1 -- 172.21.15.28:0/3614333364 learned_addr learned my addr 172.21.15.28:0/3614333364 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:50.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.456+0000 7f3476a00640 1 --2- 172.21.15.28:0/3614333364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56043bd2ca20 0x56043bd2ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:50.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.456+0000 7f347f400640 1 -- 172.21.15.28:0/3614333364 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56043bd21a30 msgr2=0x56043bd21e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:50.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.456+0000 7f347f400640 1 --2- 172.21.15.28:0/3614333364 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56043bd21a30 0x56043bd21e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.456+0000 7f347f400640 1 -- 172.21.15.28:0/3614333364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56043bd2ca20 msgr2=0x56043bd2ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:50.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.456+0000 7f347f400640 1 --2- 172.21.15.28:0/3614333364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56043bd2ca20 0x56043bd2ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.456+0000 7f347f400640 1 -- 172.21.15.28:0/3614333364 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56043bc81f50 con 0x56043bd22350 2024-04-23T14:46:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.456+0000 7f347f400640 1 --2- 172.21.15.28:0/3614333364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56043bd22350 0x56043bd2c4e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3470009fa0 tx=0x7f34700118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db87f1e58617ce83 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.456+0000 7f346fe00640 1 -- 172.21.15.28:0/3614333364 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3470003c30 con 0x56043bd22350 2024-04-23T14:46:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.457+0000 7f346fe00640 1 -- 172.21.15.28:0/3614333364 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3470003dd0 con 0x56043bd22350 2024-04-23T14:46:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.457+0000 7f346fe00640 1 -- 172.21.15.28:0/3614333364 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f347000fb90 con 0x56043bd22350 2024-04-23T14:46:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.457+0000 7f347cc00640 1 --2- 172.21.15.28:0/2244203732 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f34640a3d20 0x7f34640a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.457+0000 7f347cc00640 1 -- 172.21.15.28:0/2244203732 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f34680f0df0 con 0x7f3480074d50 2024-04-23T14:46:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.457+0000 7f347fe00640 1 --2- 172.21.15.28:0/2244203732 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f34640a3d20 0x7f34640a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.457+0000 7f3489082740 1 -- 172.21.15.28:0/3614333364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56043bd22350 msgr2=0x56043bd2c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.457+0000 7f3489082740 1 --2- 172.21.15.28:0/3614333364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56043bd22350 0x56043bd2c4e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3470009fa0 tx=0x7f34700118d0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.457+0000 7f3489082740 1 -- 172.21.15.28:0/3614333364 shutdown_connections 2024-04-23T14:46:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.457+0000 7f3489082740 1 --2- 172.21.15.28:0/3614333364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56043bd2ca20 0x56043bd2ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.457+0000 7f3489082740 1 --2- 172.21.15.28:0/3614333364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56043bd22350 0x56043bd2c4e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.457+0000 7f3489082740 1 --2- 172.21.15.28:0/3614333364 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56043bd21a30 0x56043bd21e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.457+0000 7f3489082740 1 -- 172.21.15.28:0/3614333364 >> 172.21.15.28:0/3614333364 conn(0x56043bd1dd50 msgr2=0x56043bd1e130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.458+0000 7f3489082740 1 -- 172.21.15.28:0/3614333364 shutdown_connections 2024-04-23T14:46:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.458+0000 7f3489082740 1 -- 172.21.15.28:0/3614333364 wait complete. 2024-04-23T14:46:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.458+0000 7f347fe00640 1 --2- 172.21.15.28:0/2244203732 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f34640a3d20 0x7f34640a61e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3478009e60 tx=0x7f3478005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:50.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.458+0000 7f3489082740 1 -- start start 2024-04-23T14:46:50.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.458+0000 7f3489082740 1 Processor -- start 2024-04-23T14:46:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.459+0000 7f3489082740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56043bd21a30 0x56043bd299c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.459+0000 7f3489082740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56043bd22350 0x56043bd29f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.459+0000 7f3489082740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56043bd2ca20 0x56043bd2a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.459+0000 7f3489082740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56043bd2f810 con 0x56043bd21a30 2024-04-23T14:46:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.459+0000 7f3489082740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56043bc40690 con 0x56043bd22350 2024-04-23T14:46:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.459+0000 7f3489082740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56043bd2f990 con 0x56043bd2ca20 2024-04-23T14:46:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.459+0000 7f3489082740 10 client.0 _ll_register_callbacks cb 0x56043bc7c650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:46:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.459+0000 7f3476a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56043bd22350 0x56043bd29f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.459+0000 7f3476a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56043bd22350 0x56043bd29f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60324/0 (socket says 172.21.15.28:60324) 2024-04-23T14:46:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.459+0000 7f3476a00640 1 -- 172.21.15.28:0/3776698626 learned_addr learned my addr 172.21.15.28:0/3776698626 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.459+0000 7f347f400640 1 --2- 172.21.15.28:0/3776698626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56043bd21a30 0x56043bd299c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.459+0000 7f347fe00640 1 --2- 172.21.15.28:0/3776698626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56043bd2ca20 0x56043bd2a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.460+0000 7f3476a00640 1 -- 172.21.15.28:0/3776698626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56043bd2ca20 msgr2=0x56043bd2a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.460+0000 7f3476a00640 1 --2- 172.21.15.28:0/3776698626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56043bd2ca20 0x56043bd2a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.460+0000 7f3476a00640 1 -- 172.21.15.28:0/3776698626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56043bd21a30 msgr2=0x56043bd299c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.460+0000 7f3476a00640 1 --2- 172.21.15.28:0/3776698626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56043bd21a30 0x56043bd299c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.460+0000 7f3476a00640 1 -- 172.21.15.28:0/3776698626 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56043bc3f920 con 0x56043bd22350 2024-04-23T14:46:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.460+0000 7f3476a00640 1 --2- 172.21.15.28:0/3776698626 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56043bd22350 0x56043bd29f00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3468031550 tx=0x7f346802a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c957acfe90e19c15 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.460+0000 7f346cc00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:46:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.460+0000 7f346cc00640 1 -- 172.21.15.28:0/3776698626 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3468049050 con 0x56043bd22350 2024-04-23T14:46:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.460+0000 7f346cc00640 1 -- 172.21.15.28:0/3776698626 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f34680b9520 con 0x56043bd22350 2024-04-23T14:46:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.460+0000 7f3489082740 1 -- 172.21.15.28:0/3776698626 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3470003dd0 con 0x56043bd22350 2024-04-23T14:46:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.460+0000 7f346cc00640 1 -- 172.21.15.28:0/3776698626 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f3468005d50 con 0x56043bd22350 2024-04-23T14:46:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.460+0000 7f3489082740 10 client.25688 fetch_fsmap learned FSMap version 20 2024-04-23T14:46:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.460+0000 7f3489082740 1 -- 172.21.15.28:0/3776698626 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x56043ba637d0 con 0x56043bd22350 2024-04-23T14:46:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.461+0000 7f346cc00640 1 -- 172.21.15.28:0/3776698626 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3470003dd0 con 0x56043bd22350 2024-04-23T14:46:50.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.461+0000 7f346cc00640 1 -- 172.21.15.28:0/3776698626 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f34680ee770 con 0x56043bd22350 2024-04-23T14:46:50.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.461+0000 7f3489082740 10 client.25688 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:46:50.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.461+0000 7f3489082740 10 client.25688 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:46:50.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.461+0000 7f3489082740 20 client.25688 populate_metadata read hostname 'smithi028' 2024-04-23T14:46:50.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.461+0000 7f3489082740 1 --2- 172.21.15.28:0/3776698626 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56043bd3cd90 0x56043bd3f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:50.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.461+0000 7f3489082740 4 client.25688 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:46:50.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.461+0000 7f3489082740 1 -- 172.21.15.28:0/3776698626 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56043bc94c10 con 0x56043bd3cd90 2024-04-23T14:46:50.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.461+0000 7f347f400640 1 --2- 172.21.15.28:0/3776698626 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56043bd3cd90 0x56043bd3f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:50.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.462+0000 7f347f400640 1 --2- 172.21.15.28:0/3776698626 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56043bd3cd90 0x56043bd3f1b0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0971df8e008dde6 server_cookie=10a6a6e21795084 in_seq=0 out_seq=0 2024-04-23T14:46:50.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.462+0000 7f346cc00640 10 client.25688 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:46:50.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.463+0000 7f346cc00640 1 -- 172.21.15.28:0/3776698626 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56043bc94c10 con 0x56043bd3cd90 2024-04-23T14:46:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.463+0000 7f346cc00640 10 client.25688 handle_command_reply: tid=0 2024-04-23T14:46:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.463+0000 7f3489082740 1 client.25688 shutdown 2024-04-23T14:46:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.463+0000 7f3489082740 1 -- 172.21.15.28:0/3776698626 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56043bd22350 msgr2=0x56043bd29f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.463+0000 7f3489082740 1 --2- 172.21.15.28:0/3776698626 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56043bd22350 0x56043bd29f00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3468031550 tx=0x7f346802a480 comp rx=0 tx=0).stop 2024-04-23T14:46:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.463+0000 7f3489082740 1 -- 172.21.15.28:0/3776698626 shutdown_connections 2024-04-23T14:46:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.463+0000 7f3489082740 1 --2- 172.21.15.28:0/3776698626 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56043bd3cd90 0x56043bd3f1b0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.463+0000 7f3489082740 1 --2- 172.21.15.28:0/3776698626 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56043bd22350 0x56043bd29f00 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.464+0000 7f3489082740 1 --2- 172.21.15.28:0/3776698626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56043bd2ca20 0x56043bd2a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.464+0000 7f346cc00640 0 client.25688 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:46:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.464+0000 7f3489082740 1 --2- 172.21.15.28:0/3776698626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56043bd21a30 0x56043bd299c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.464+0000 7f3489082740 1 -- 172.21.15.28:0/3776698626 >> 172.21.15.28:0/3776698626 conn(0x56043bd1dd50 msgr2=0x56043bd1cfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.464+0000 7f3489082740 1 -- 172.21.15.28:0/3776698626 shutdown_connections 2024-04-23T14:46:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.464+0000 7f3489082740 1 -- 172.21.15.28:0/3776698626 wait complete. 2024-04-23T14:46:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.464+0000 7f3489082740 20 client.25688 trim_cache size 0 max 16384 2024-04-23T14:46:50.469 INFO:tasks.workunit.client.0.smithi028.stdout: 1 131708 11.04 MB/sec warmup 66 sec latency 9.052 ms 2024-04-23T14:46:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.484+0000 7f3489082740 1 Processor -- start 2024-04-23T14:46:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.484+0000 7f3489082740 1 -- start start 2024-04-23T14:46:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.485+0000 7f3489082740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56043bd21a30 0x56043bd38700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.485+0000 7f3489082740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56043bd22350 0x56043bd44a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.485+0000 7f3489082740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56043bd2ca20 0x56043bd44fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.485+0000 7f3489082740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f34801a9970 con 0x56043bd22350 2024-04-23T14:46:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.485+0000 7f3489082740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f34801ac1f0 con 0x56043bd21a30 2024-04-23T14:46:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.485+0000 7f3489082740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f34801a9fe0 con 0x56043bd2ca20 2024-04-23T14:46:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.485+0000 7f347f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56043bd21a30 0x56043bd38700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.485+0000 7f3476a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56043bd22350 0x56043bd44a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:50.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.485+0000 7f347f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56043bd21a30 0x56043bd38700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60328/0 (socket says 172.21.15.28:60328) 2024-04-23T14:46:50.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.485+0000 7f347f400640 1 -- 172.21.15.28:0/3828930349 learned_addr learned my addr 172.21.15.28:0/3828930349 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:50.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.485+0000 7f347fe00640 1 --2- 172.21.15.28:0/3828930349 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56043bd2ca20 0x56043bd44fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:50.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.485+0000 7f347f400640 1 -- 172.21.15.28:0/3828930349 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56043bd2ca20 msgr2=0x56043bd44fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:50.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.485+0000 7f347f400640 1 --2- 172.21.15.28:0/3828930349 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56043bd2ca20 0x56043bd44fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.485+0000 7f347f400640 1 -- 172.21.15.28:0/3828930349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56043bd22350 msgr2=0x56043bd44a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:50.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.485+0000 7f347f400640 1 --2- 172.21.15.28:0/3828930349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56043bd22350 0x56043bd44a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.485+0000 7f347f400640 1 -- 172.21.15.28:0/3828930349 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56043b9993f0 con 0x56043bd21a30 2024-04-23T14:46:50.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.485+0000 7f3476a00640 1 --2- 172.21.15.28:0/3828930349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56043bd22350 0x56043bd44a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:46:50.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.486+0000 7f347f400640 1 --2- 172.21.15.28:0/3828930349 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56043bd21a30 0x56043bd38700 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3470009fa0 tx=0x7f3470000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=243f18b1c3e0a049 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:50.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.486+0000 7f346f400640 1 -- 172.21.15.28:0/3828930349 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3470031070 con 0x56043bd21a30 2024-04-23T14:46:50.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.486+0000 7f346f400640 1 -- 172.21.15.28:0/3828930349 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3470003d30 con 0x56043bd21a30 2024-04-23T14:46:50.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.486+0000 7f346f400640 1 -- 172.21.15.28:0/3828930349 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f347000f9e0 con 0x56043bd21a30 2024-04-23T14:46:50.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.487+0000 7f3489082740 1 -- 172.21.15.28:0/3828930349 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56043bd21a30 msgr2=0x56043bd38700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:50.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.487+0000 7f3489082740 1 --2- 172.21.15.28:0/3828930349 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56043bd21a30 0x56043bd38700 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3470009fa0 tx=0x7f3470000f40 comp rx=0 tx=0).stop 2024-04-23T14:46:50.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.487+0000 7f3489082740 1 -- 172.21.15.28:0/3828930349 shutdown_connections 2024-04-23T14:46:50.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.487+0000 7f3489082740 1 --2- 172.21.15.28:0/3828930349 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56043bd2ca20 0x56043bd44fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.487+0000 7f3489082740 1 --2- 172.21.15.28:0/3828930349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56043bd22350 0x56043bd44a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.487+0000 7f3489082740 1 --2- 172.21.15.28:0/3828930349 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56043bd21a30 0x56043bd38700 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.487+0000 7f3489082740 1 -- 172.21.15.28:0/3828930349 >> 172.21.15.28:0/3828930349 conn(0x56043bd1dd50 msgr2=0x56043bd23ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:50.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.487+0000 7f3489082740 1 -- 172.21.15.28:0/3828930349 shutdown_connections 2024-04-23T14:46:50.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.487+0000 7f3489082740 1 -- 172.21.15.28:0/3828930349 wait complete. 2024-04-23T14:46:50.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.487+0000 7f3489082740 1 -- start start 2024-04-23T14:46:50.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.487+0000 7f3489082740 1 Processor -- start 2024-04-23T14:46:50.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.488+0000 7f3489082740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56043bd21a30 0x56043bd26d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:50.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.488+0000 7f3489082740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56043bd22350 0x56043bd27260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:50.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.488+0000 7f3489082740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56043bd2ca20 0x56043bda22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.488+0000 7f3489082740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f34801aa6b0 con 0x56043bd2ca20 2024-04-23T14:46:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.488+0000 7f3489082740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f34801abb20 con 0x56043bd22350 2024-04-23T14:46:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.488+0000 7f3489082740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f34801aad80 con 0x56043bd21a30 2024-04-23T14:46:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.488+0000 7f3489082740 10 client.0 _ll_register_callbacks cb 0x7f34680f43d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:46:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.488+0000 7f3476a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56043bd22350 0x56043bd27260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.488+0000 7f3476a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56043bd22350 0x56043bd27260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60340/0 (socket says 172.21.15.28:60340) 2024-04-23T14:46:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.488+0000 7f3476a00640 1 -- 172.21.15.28:0/3617234753 learned_addr learned my addr 172.21.15.28:0/3617234753 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.488+0000 7f347f400640 1 --2- 172.21.15.28:0/3617234753 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56043bd21a30 0x56043bd26d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.488+0000 7f347fe00640 1 --2- 172.21.15.28:0/3617234753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56043bd2ca20 0x56043bda22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.488+0000 7f3476a00640 1 -- 172.21.15.28:0/3617234753 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56043bd21a30 msgr2=0x56043bd26d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.488+0000 7f3476a00640 1 --2- 172.21.15.28:0/3617234753 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56043bd21a30 0x56043bd26d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.488+0000 7f3476a00640 1 -- 172.21.15.28:0/3617234753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56043bd2ca20 msgr2=0x56043bda22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.488+0000 7f3476a00640 1 --2- 172.21.15.28:0/3617234753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56043bd2ca20 0x56043bda22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.488+0000 7f3476a00640 1 -- 172.21.15.28:0/3617234753 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56043b9590f0 con 0x56043bd22350 2024-04-23T14:46:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.489+0000 7f3476a00640 1 --2- 172.21.15.28:0/3617234753 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56043bd22350 0x56043bd27260 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f34680c5650 tx=0x7f34680bb3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91ff1bd03efc1d12 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.489+0000 7f346cc00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:46:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.489+0000 7f346cc00640 1 -- 172.21.15.28:0/3617234753 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3468049070 con 0x56043bd22350 2024-04-23T14:46:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.489+0000 7f346cc00640 1 -- 172.21.15.28:0/3617234753 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f346803b4f0 con 0x56043bd22350 2024-04-23T14:46:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.489+0000 7f3489082740 1 -- 172.21.15.28:0/3617234753 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3470003d30 con 0x56043bd22350 2024-04-23T14:46:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.489+0000 7f346cc00640 1 -- 172.21.15.28:0/3617234753 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f34680bc9f0 con 0x56043bd22350 2024-04-23T14:46:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.489+0000 7f3489082740 10 client.25694 fetch_fsmap learned FSMap version 20 2024-04-23T14:46:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.489+0000 7f3489082740 1 -- 172.21.15.28:0/3617234753 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x56043bc4c810 con 0x56043bd22350 2024-04-23T14:46:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.489+0000 7f346cc00640 1 -- 172.21.15.28:0/3617234753 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3470003d30 con 0x56043bd22350 2024-04-23T14:46:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.490+0000 7f346cc00640 1 -- 172.21.15.28:0/3617234753 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f34680dd0b0 con 0x56043bd22350 2024-04-23T14:46:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.490+0000 7f3489082740 10 client.25694 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:46:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.490+0000 7f3489082740 10 client.25694 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:46:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.490+0000 7f3489082740 20 client.25694 populate_metadata read hostname 'smithi028' 2024-04-23T14:46:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.490+0000 7f3489082740 1 --2- 172.21.15.28:0/3617234753 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56043bda3cf0 0x56043bda6110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.490+0000 7f3489082740 4 client.25694 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:46:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.490+0000 7f3489082740 1 -- 172.21.15.28:0/3617234753 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56043bc8ea80 con 0x56043bda3cf0 2024-04-23T14:46:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.490+0000 7f347f400640 1 --2- 172.21.15.28:0/3617234753 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56043bda3cf0 0x56043bda6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.490+0000 7f347f400640 1 --2- 172.21.15.28:0/3617234753 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56043bda3cf0 0x56043bda6110 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15c087e8cf33dc51 server_cookie=f313799abadf4347 in_seq=0 out_seq=0 2024-04-23T14:46:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.490+0000 7f346cc00640 10 client.25694 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:46:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.492+0000 7f346cc00640 1 -- 172.21.15.28:0/3617234753 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56043bc8ea80 con 0x56043bda3cf0 2024-04-23T14:46:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.492+0000 7f346cc00640 10 client.25694 handle_command_reply: tid=0 2024-04-23T14:46:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.492+0000 7f3489082740 1 client.25694 shutdown 2024-04-23T14:46:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.492+0000 7f3489082740 1 -- 172.21.15.28:0/3617234753 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56043bd22350 msgr2=0x56043bd27260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.492+0000 7f3489082740 1 --2- 172.21.15.28:0/3617234753 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56043bd22350 0x56043bd27260 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f34680c5650 tx=0x7f34680bb3c0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.493+0000 7f3489082740 1 -- 172.21.15.28:0/3617234753 shutdown_connections 2024-04-23T14:46:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.493+0000 7f3489082740 1 --2- 172.21.15.28:0/3617234753 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56043bda3cf0 0x56043bda6110 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.493+0000 7f3489082740 1 --2- 172.21.15.28:0/3617234753 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56043bd22350 0x56043bd27260 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.493+0000 7f3489082740 1 --2- 172.21.15.28:0/3617234753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56043bd2ca20 0x56043bda22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.493+0000 7f3489082740 1 --2- 172.21.15.28:0/3617234753 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56043bd21a30 0x56043bd26d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.493+0000 7f3489082740 1 -- 172.21.15.28:0/3617234753 >> 172.21.15.28:0/3617234753 conn(0x56043bd1dd50 msgr2=0x56043bd29980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:50.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.493+0000 7f346cc00640 0 client.25694 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:46:50.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.493+0000 7f3489082740 1 -- 172.21.15.28:0/3617234753 shutdown_connections 2024-04-23T14:46:50.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.493+0000 7f3489082740 1 -- 172.21.15.28:0/3617234753 wait complete. 2024-04-23T14:46:50.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.493+0000 7f3489082740 20 client.25694 trim_cache size 0 max 16384 2024-04-23T14:46:50.497 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:46:50.497 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-23T14:46:50.497 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-23T14:46:50.498 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:46:50.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.494+0000 7f3484c00640 1 -- 172.21.15.28:0/2244203732 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f34640a3d20 msgr2=0x7f34640a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:50.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.494+0000 7f3484c00640 1 --2- 172.21.15.28:0/2244203732 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f34640a3d20 0x7f34640a61e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3478009e60 tx=0x7f3478005f90 comp rx=0 tx=0).stop 2024-04-23T14:46:50.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.494+0000 7f3484c00640 1 -- 172.21.15.28:0/2244203732 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3480074d50 msgr2=0x7f34801a0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.494+0000 7f3484c00640 1 --2- 172.21.15.28:0/2244203732 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3480074d50 0x7f34801a0b80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3468004770 tx=0x7f3468003c10 comp rx=0 tx=0).stop 2024-04-23T14:46:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.494+0000 7f3484c00640 1 -- 172.21.15.28:0/2244203732 shutdown_connections 2024-04-23T14:46:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.494+0000 7f3484c00640 1 --2- 172.21.15.28:0/2244203732 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f34640a3d20 0x7f34640a61e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.494+0000 7f3484c00640 1 --2- 172.21.15.28:0/2244203732 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f348010e6e0 0x7f348019a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.494+0000 7f3484c00640 1 --2- 172.21.15.28:0/2244203732 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3480074d50 0x7f34801a0b80 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.494+0000 7f3484c00640 1 --2- 172.21.15.28:0/2244203732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3480074430 0x7f34801a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.494+0000 7f3484c00640 1 -- 172.21.15.28:0/2244203732 >> 172.21.15.28:0/2244203732 conn(0x7f34800fd9d0 msgr2=0x7f34800fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.494+0000 7f3484c00640 1 -- 172.21.15.28:0/2244203732 shutdown_connections 2024-04-23T14:46:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.494+0000 7f3484c00640 1 -- 172.21.15.28:0/2244203732 wait complete. 2024-04-23T14:46:50.513 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-23T14:46:50.513 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:46:50.583 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:50 smithi028 ceph-mon[27810]: pgmap v87: 97 pgs: 97 active+clean; 81 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 2.3 MiB/s wr, 156 op/s 2024-04-23T14:46:50.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.746+0000 7f020ec00640 1 -- 172.21.15.28:0/4061925888 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0208104df0 msgr2=0x7f0208108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:50.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.746+0000 7f020ec00640 1 --2- 172.21.15.28:0/4061925888 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0208104df0 0x7f0208108e30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f01f0009900 tx=0x7f01f002a410 comp rx=0 tx=0).stop 2024-04-23T14:46:50.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.746+0000 7f020ec00640 1 -- 172.21.15.28:0/4061925888 shutdown_connections 2024-04-23T14:46:50.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.746+0000 7f020ec00640 1 --2- 172.21.15.28:0/4061925888 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0208104df0 0x7f0208108e30 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.746+0000 7f020ec00640 1 --2- 172.21.15.28:0/4061925888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0208109510 0x7f02081110e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.746+0000 7f020ec00640 1 --2- 172.21.15.28:0/4061925888 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0208104440 0x7f0208104820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.746+0000 7f020ec00640 1 -- 172.21.15.28:0/4061925888 >> 172.21.15.28:0/4061925888 conn(0x7f02080ffd00 msgr2=0x7f0208102120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:50.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.746+0000 7f020ec00640 1 -- 172.21.15.28:0/4061925888 shutdown_connections 2024-04-23T14:46:50.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.747+0000 7f020ec00640 1 -- 172.21.15.28:0/4061925888 wait complete. 2024-04-23T14:46:50.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.747+0000 7f020ec00640 1 Processor -- start 2024-04-23T14:46:50.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.748+0000 7f020ec00640 1 -- start start 2024-04-23T14:46:50.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.748+0000 7f020ec00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0208104440 0x7f02081a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:50.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.748+0000 7f020ec00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0208109510 0x7f02081a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:50.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.748+0000 7f020ec00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f020819aa10 0x7f020819aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:50.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.748+0000 7f020ec00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f02081145d0 con 0x7f0208109510 2024-04-23T14:46:50.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.748+0000 7f020ec00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0208114450 con 0x7f020819aa10 2024-04-23T14:46:50.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.748+0000 7f020ec00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0208114750 con 0x7f0208104440 2024-04-23T14:46:50.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.748+0000 7f0207e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0208104440 0x7f02081a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:50.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.748+0000 7f0207e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0208104440 0x7f02081a0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:42790/0 (socket says 172.21.15.28:42790) 2024-04-23T14:46:50.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.748+0000 7f0207e00640 1 -- 172.21.15.28:0/2293019656 learned_addr learned my addr 172.21.15.28:0/2293019656 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:50.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.748+0000 7f01ff400640 1 --2- 172.21.15.28:0/2293019656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0208109510 0x7f02081a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:50.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.749+0000 7f020cc00640 1 --2- 172.21.15.28:0/2293019656 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f020819aa10 0x7f020819aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:50.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.749+0000 7f0207e00640 1 -- 172.21.15.28:0/2293019656 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f020819aa10 msgr2=0x7f020819aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:50.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.749+0000 7f0207e00640 1 --2- 172.21.15.28:0/2293019656 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f020819aa10 0x7f020819aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.749+0000 7f0207e00640 1 -- 172.21.15.28:0/2293019656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0208109510 msgr2=0x7f02081a0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:50.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.749+0000 7f0207e00640 1 --2- 172.21.15.28:0/2293019656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0208109510 0x7f02081a0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:50.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.749+0000 7f0207e00640 1 -- 172.21.15.28:0/2293019656 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f020819b530 con 0x7f0208104440 2024-04-23T14:46:50.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.749+0000 7f0207e00640 1 --2- 172.21.15.28:0/2293019656 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0208104440 0x7f02081a0810 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f01f800ebc0 tx=0x7f01f800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:50.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.749+0000 7f0205600640 1 -- 172.21.15.28:0/2293019656 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f8010630 con 0x7f0208104440 2024-04-23T14:46:50.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.749+0000 7f0205600640 1 -- 172.21.15.28:0/2293019656 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01f8010cb0 con 0x7f0208104440 2024-04-23T14:46:50.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.749+0000 7f020ec00640 1 -- 172.21.15.28:0/2293019656 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f020819b820 con 0x7f0208104440 2024-04-23T14:46:50.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.749+0000 7f0205600640 1 -- 172.21.15.28:0/2293019656 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f8018bd0 con 0x7f0208104440 2024-04-23T14:46:50.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.750+0000 7f020ec00640 1 -- 172.21.15.28:0/2293019656 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02081a7a20 con 0x7f0208104440 2024-04-23T14:46:50.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.751+0000 7f020ec00640 1 -- 172.21.15.28:0/2293019656 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0208069270 con 0x7f0208104440 2024-04-23T14:46:50.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.753+0000 7f0205600640 1 -- 172.21.15.28:0/2293019656 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f01f80107d0 con 0x7f0208104440 2024-04-23T14:46:50.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.755+0000 7f0205600640 1 --2- 172.21.15.28:0/2293019656 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f01ec0a3d20 0x7f01ec0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:50.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.755+0000 7f0205600640 1 -- 172.21.15.28:0/2293019656 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f01f80d0f70 con 0x7f0208104440 2024-04-23T14:46:50.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.755+0000 7f01ff400640 1 --2- 172.21.15.28:0/2293019656 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f01ec0a3d20 0x7f01ec0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:50.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.756+0000 7f01ff400640 1 --2- 172.21.15.28:0/2293019656 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f01ec0a3d20 0x7f01ec0a61e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f01f0006010 tx=0x7f01f0005a20 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:50.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:50.758+0000 7f0205600640 1 -- 172.21.15.28:0/2293019656 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f01f809d5c0 con 0x7f0208104440 2024-04-23T14:46:50.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:50 smithi105 ceph-mon[32555]: pgmap v87: 97 pgs: 97 active+clean; 81 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 2.3 MiB/s wr, 156 op/s 2024-04-23T14:46:50.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:50 smithi038 ceph-mon[32348]: pgmap v87: 97 pgs: 97 active+clean; 81 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 2.3 MiB/s wr, 156 op/s 2024-04-23T14:46:51.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.019+0000 7f020ec00640 1 -- 172.21.15.28:0/2293019656 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f020819bef0 con 0x7f0208104440 2024-04-23T14:46:51.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.020+0000 7f0205600640 1 -- 172.21.15.28:0/2293019656 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7f01f809cec0 con 0x7f0208104440 2024-04-23T14:46:51.021 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:46:51.021 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:46:51.022 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:46:51.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.022+0000 7f020ec00640 1 -- 172.21.15.28:0/2293019656 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f01ec0a3d20 msgr2=0x7f01ec0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:51.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.022+0000 7f020ec00640 1 --2- 172.21.15.28:0/2293019656 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f01ec0a3d20 0x7f01ec0a61e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f01f0006010 tx=0x7f01f0005a20 comp rx=0 tx=0).stop 2024-04-23T14:46:51.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.023+0000 7f020ec00640 1 -- 172.21.15.28:0/2293019656 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0208104440 msgr2=0x7f02081a0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:51.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.023+0000 7f020ec00640 1 --2- 172.21.15.28:0/2293019656 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0208104440 0x7f02081a0810 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f01f800ebc0 tx=0x7f01f800c6a0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.023+0000 7f020ec00640 1 -- 172.21.15.28:0/2293019656 shutdown_connections 2024-04-23T14:46:51.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.023+0000 7f020ec00640 1 --2- 172.21.15.28:0/2293019656 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f01ec0a3d20 0x7f01ec0a61e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.023+0000 7f020ec00640 1 --2- 172.21.15.28:0/2293019656 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f020819aa10 0x7f020819aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.023+0000 7f020ec00640 1 --2- 172.21.15.28:0/2293019656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0208109510 0x7f02081a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.023+0000 7f020ec00640 1 --2- 172.21.15.28:0/2293019656 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0208104440 0x7f02081a0810 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.023+0000 7f020ec00640 1 -- 172.21.15.28:0/2293019656 >> 172.21.15.28:0/2293019656 conn(0x7f02080ffd00 msgr2=0x7f0208101070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:51.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.023+0000 7f020ec00640 1 -- 172.21.15.28:0/2293019656 shutdown_connections 2024-04-23T14:46:51.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.023+0000 7f020ec00640 1 -- 172.21.15.28:0/2293019656 wait complete. 2024-04-23T14:46:51.047 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-23T14:46:51.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.294+0000 7f13a8a00640 1 -- 172.21.15.28:0/2037995696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13a411b7e0 msgr2=0x7f13a411dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:51.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.294+0000 7f13a8a00640 1 --2- 172.21.15.28:0/2037995696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13a411b7e0 0x7f13a411dbd0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1394009890 tx=0x7f139402a3b0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.296+0000 7f13a8a00640 1 -- 172.21.15.28:0/2037995696 shutdown_connections 2024-04-23T14:46:51.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.296+0000 7f13a8a00640 1 --2- 172.21.15.28:0/2037995696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13a411b7e0 0x7f13a411dbd0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.296+0000 7f13a8a00640 1 --2- 172.21.15.28:0/2037995696 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f13a410e880 0x7f13a410ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.297+0000 7f13a8a00640 1 --2- 172.21.15.28:0/2037995696 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13a4110850 0x7f13a410e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.297+0000 7f13a8a00640 1 -- 172.21.15.28:0/2037995696 >> 172.21.15.28:0/2037995696 conn(0x7f13a406d110 msgr2=0x7f13a406d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:51.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.297+0000 7f13a8a00640 1 -- 172.21.15.28:0/2037995696 shutdown_connections 2024-04-23T14:46:51.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.298+0000 7f13a8a00640 1 -- 172.21.15.28:0/2037995696 wait complete. 2024-04-23T14:46:51.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.301+0000 7f13a8a00640 1 Processor -- start 2024-04-23T14:46:51.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.301+0000 7f13a8a00640 1 -- start start 2024-04-23T14:46:51.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.301+0000 7f13a8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13a410e880 0x7f13a411af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:51.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.302+0000 7f13a8a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13a4110850 0x7f13a4116fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:51.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.302+0000 7f13a8a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f13a411b7e0 0x7f13a4117500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:51.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.302+0000 7f13a8a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f13a4113990 con 0x7f13a410e880 2024-04-23T14:46:51.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.302+0000 7f13a8a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f13a4113810 con 0x7f13a4110850 2024-04-23T14:46:51.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.302+0000 7f13a8a00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f13a4113b10 con 0x7f13a411b7e0 2024-04-23T14:46:51.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.302+0000 7f13a1400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13a4110850 0x7f13a4116fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:51.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.302+0000 7f13a1400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13a4110850 0x7f13a4116fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60384/0 (socket says 172.21.15.28:60384) 2024-04-23T14:46:51.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.302+0000 7f13a1400640 1 -- 172.21.15.28:0/2140956044 learned_addr learned my addr 172.21.15.28:0/2140956044 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:51.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.303+0000 7f13a1400640 1 -- 172.21.15.28:0/2140956044 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f13a411b7e0 msgr2=0x7f13a4117500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T14:46:51.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.303+0000 7f13a1400640 1 --2- 172.21.15.28:0/2140956044 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f13a411b7e0 0x7f13a4117500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.303+0000 7f13a1400640 1 -- 172.21.15.28:0/2140956044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13a410e880 msgr2=0x7f13a411af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:51.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.303+0000 7f13a1400640 1 --2- 172.21.15.28:0/2140956044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13a410e880 0x7f13a411af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.303+0000 7f13a1400640 1 -- 172.21.15.28:0/2140956044 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13a4117c30 con 0x7f13a4110850 2024-04-23T14:46:51.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.303+0000 7f13a1400640 1 --2- 172.21.15.28:0/2140956044 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13a4110850 0x7f13a4116fc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f139001fa60 tx=0x7f139001ff20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:51.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.305+0000 7f139aa00640 1 -- 172.21.15.28:0/2140956044 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139001dc50 con 0x7f13a4110850 2024-04-23T14:46:51.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.305+0000 7f139aa00640 1 -- 172.21.15.28:0/2140956044 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f139001ddf0 con 0x7f13a4110850 2024-04-23T14:46:51.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.305+0000 7f139aa00640 1 -- 172.21.15.28:0/2140956044 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13900216a0 con 0x7f13a4110850 2024-04-23T14:46:51.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.306+0000 7f13a8a00640 1 -- 172.21.15.28:0/2140956044 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13a41ab570 con 0x7f13a4110850 2024-04-23T14:46:51.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.307+0000 7f13a8a00640 1 -- 172.21.15.28:0/2140956044 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13a41aba00 con 0x7f13a4110850 2024-04-23T14:46:51.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.313+0000 7f13ab852740 1 Processor -- start 2024-04-23T14:46:51.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.313+0000 7f139aa00640 1 -- 172.21.15.28:0/2140956044 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f13900046c0 con 0x7f13a4110850 2024-04-23T14:46:51.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.313+0000 7f13ab852740 1 -- start start 2024-04-23T14:46:51.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.313+0000 7f13ab852740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593d61a7950 0x5593d61a7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:51.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.314+0000 7f13ab852740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593d61a8270 0x5593d61b2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:51.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.314+0000 7f13ab852740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593d61b2940 0x5593d61b4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:51.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.314+0000 7f13ab852740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5593d610d1d0 con 0x5593d61a7950 2024-04-23T14:46:51.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.314+0000 7f13ab852740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5593d60b3450 con 0x5593d61a8270 2024-04-23T14:46:51.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.314+0000 7f13ab852740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5593d6109190 con 0x5593d61b2940 2024-04-23T14:46:51.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.314+0000 7f13a2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593d61a7950 0x5593d61a7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:51.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.314+0000 7f13a2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593d61a7950 0x5593d61a7d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42130/0 (socket says 172.21.15.28:42130) 2024-04-23T14:46:51.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.314+0000 7f13a2800640 1 -- 172.21.15.28:0/3700413323 learned_addr learned my addr 172.21.15.28:0/3700413323 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:51.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.315+0000 7f13a1e00640 1 --2- 172.21.15.28:0/3700413323 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593d61a8270 0x5593d61b2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:51.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.315+0000 7f13a1400640 1 --2- 172.21.15.28:0/3700413323 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593d61b2940 0x5593d61b4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:51.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.316+0000 7f13a2800640 1 -- 172.21.15.28:0/3700413323 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593d61b2940 msgr2=0x5593d61b4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:51.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.316+0000 7f13a2800640 1 --2- 172.21.15.28:0/3700413323 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593d61b2940 0x5593d61b4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.316+0000 7f13a2800640 1 -- 172.21.15.28:0/3700413323 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593d61a8270 msgr2=0x5593d61b2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:51.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.316+0000 7f13a2800640 1 --2- 172.21.15.28:0/3700413323 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593d61a8270 0x5593d61b2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.316+0000 7f139aa00640 1 --2- 172.21.15.28:0/2140956044 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f13880a3ea0 0x7f13880a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:51.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.316+0000 7f13a2800640 1 -- 172.21.15.28:0/3700413323 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593d6170310 con 0x5593d61a7950 2024-04-23T14:46:51.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.316+0000 7f139aa00640 1 -- 172.21.15.28:0/2140956044 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f13900e1ad0 con 0x7f13a4110850 2024-04-23T14:46:51.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.317+0000 7f13a1400640 1 --2- 172.21.15.28:0/2140956044 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f13880a3ea0 0x7f13880a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:51.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.317+0000 7f13a2800640 1 --2- 172.21.15.28:0/3700413323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593d61a7950 0x5593d61a7d30 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f139402a920 tx=0x7f1394009430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2cb9f354156554 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:51.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.318+0000 7f138f000640 1 -- 172.21.15.28:0/3700413323 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1394037040 con 0x5593d61a7950 2024-04-23T14:46:51.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.318+0000 7f138f000640 1 -- 172.21.15.28:0/3700413323 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1394030cc0 con 0x5593d61a7950 2024-04-23T14:46:51.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.318+0000 7f138f000640 1 -- 172.21.15.28:0/3700413323 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1394005290 con 0x5593d61a7950 2024-04-23T14:46:51.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.319+0000 7f13ab852740 1 -- 172.21.15.28:0/3700413323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593d61a7950 msgr2=0x5593d61a7d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:51.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.319+0000 7f13ab852740 1 --2- 172.21.15.28:0/3700413323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593d61a7950 0x5593d61a7d30 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f139402a920 tx=0x7f1394009430 comp rx=0 tx=0).stop 2024-04-23T14:46:51.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.319+0000 7f13a1400640 1 --2- 172.21.15.28:0/2140956044 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f13880a3ea0 0x7f13880a6360 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f13900df2e0 tx=0x7f13900e5700 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:51.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.319+0000 7f13ab852740 1 -- 172.21.15.28:0/3700413323 shutdown_connections 2024-04-23T14:46:51.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.319+0000 7f13ab852740 1 --2- 172.21.15.28:0/3700413323 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593d61a8270 0x5593d61b2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.319+0000 7f13ab852740 1 --2- 172.21.15.28:0/3700413323 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593d61b2940 0x5593d61b4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.319+0000 7f13ab852740 1 --2- 172.21.15.28:0/3700413323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593d61a7950 0x5593d61a7d30 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.319+0000 7f13ab852740 1 -- 172.21.15.28:0/3700413323 >> 172.21.15.28:0/3700413323 conn(0x5593d61a3c70 msgr2=0x5593d61a4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:51.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.320+0000 7f13ab852740 1 -- 172.21.15.28:0/3700413323 shutdown_connections 2024-04-23T14:46:51.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.320+0000 7f13ab852740 1 -- 172.21.15.28:0/3700413323 wait complete. 2024-04-23T14:46:51.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.320+0000 7f13ab852740 1 -- start start 2024-04-23T14:46:51.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.321+0000 7f13ab852740 1 Processor -- start 2024-04-23T14:46:51.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.322+0000 7f13ab852740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593d61a7950 0x5593d61b2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:51.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.322+0000 7f13ab852740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593d61a8270 0x5593d61af650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:51.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.322+0000 7f13ab852740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593d61b2940 0x5593d61afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:51.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.322+0000 7f13ab852740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5593d61b5770 con 0x5593d61a8270 2024-04-23T14:46:51.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.322+0000 7f13ab852740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5593d60c6810 con 0x5593d61b2940 2024-04-23T14:46:51.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.322+0000 7f13ab852740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5593d61b58f0 con 0x5593d61a7950 2024-04-23T14:46:51.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.322+0000 7f13ab852740 10 client.0 _ll_register_callbacks cb 0x5593d6094140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:46:51.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.323+0000 7f13a2800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593d61a7950 0x5593d61b2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:51.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.323+0000 7f13a2800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593d61a7950 0x5593d61b2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:42824/0 (socket says 172.21.15.28:42824) 2024-04-23T14:46:51.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.323+0000 7f13a2800640 1 -- 172.21.15.28:0/1961495657 learned_addr learned my addr 172.21.15.28:0/1961495657 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:51.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.323+0000 7f13a2800640 1 --2- 172.21.15.28:0/1961495657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593d61b2940 0x5593d61afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:51.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.323+0000 7f13a1e00640 1 --2- 172.21.15.28:0/1961495657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593d61a8270 0x5593d61af650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:51.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.324+0000 7f13a2800640 1 -- 172.21.15.28:0/1961495657 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593d61a7950 msgr2=0x5593d61b2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:51.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.324+0000 7f13a2800640 1 --2- 172.21.15.28:0/1961495657 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593d61a7950 0x5593d61b2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.324+0000 7f13a2800640 1 -- 172.21.15.28:0/1961495657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593d61a8270 msgr2=0x5593d61af650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:51.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.324+0000 7f13a2800640 1 --2- 172.21.15.28:0/1961495657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593d61a8270 0x5593d61af650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.324+0000 7f13a2800640 1 -- 172.21.15.28:0/1961495657 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593d60d73f0 con 0x5593d61b2940 2024-04-23T14:46:51.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.324+0000 7f13a1e00640 1 --2- 172.21.15.28:0/1961495657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593d61a8270 0x5593d61af650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:46:51.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.325+0000 7f13a2800640 1 --2- 172.21.15.28:0/1961495657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593d61b2940 0x5593d61afc60 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f13940057c0 tx=0x7f139402a8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75e440d16127b23b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:51.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.325+0000 7f1383400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:46:51.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.325+0000 7f1383400640 1 -- 172.21.15.28:0/1961495657 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1394037050 con 0x5593d61b2940 2024-04-23T14:46:51.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.325+0000 7f1383400640 1 -- 172.21.15.28:0/1961495657 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f1394004040 con 0x5593d61b2940 2024-04-23T14:46:51.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.325+0000 7f13ab852740 1 -- 172.21.15.28:0/1961495657 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1394030cc0 con 0x5593d61b2940 2024-04-23T14:46:51.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.325+0000 7f1383400640 1 -- 172.21.15.28:0/1961495657 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f1394004240 con 0x5593d61b2940 2024-04-23T14:46:51.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.326+0000 7f13ab852740 10 client.25709 fetch_fsmap learned FSMap version 20 2024-04-23T14:46:51.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.326+0000 7f13ab852740 1 -- 172.21.15.28:0/1961495657 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5593d5e1f3f0 con 0x5593d61b2940 2024-04-23T14:46:51.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.326+0000 7f1383400640 1 -- 172.21.15.28:0/1961495657 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13940043e0 con 0x5593d61b2940 2024-04-23T14:46:51.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.326+0000 7f1383400640 1 -- 172.21.15.28:0/1961495657 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f139403a070 con 0x5593d61b2940 2024-04-23T14:46:51.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.326+0000 7f13ab852740 10 client.25709 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:46:51.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.326+0000 7f13ab852740 10 client.25709 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:46:51.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.326+0000 7f13ab852740 20 client.25709 populate_metadata read hostname 'smithi028' 2024-04-23T14:46:51.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.326+0000 7f13ab852740 1 --2- 172.21.15.28:0/1961495657 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5593d61c2950 0x5593d61c4d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:51.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.327+0000 7f13ab852740 4 client.25709 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:46:51.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.327+0000 7f13ab852740 1 -- 172.21.15.28:0/1961495657 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f1394030cc0 con 0x5593d61c2950 2024-04-23T14:46:51.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.327+0000 7f13a1e00640 1 --2- 172.21.15.28:0/1961495657 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5593d61c2950 0x5593d61c4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:51.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.327+0000 7f13a1e00640 1 --2- 172.21.15.28:0/1961495657 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5593d61c2950 0x5593d61c4d40 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0c29737cba472c1 server_cookie=7aa0d858c135dac2 in_seq=0 out_seq=0 2024-04-23T14:46:51.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.327+0000 7f1383400640 10 client.25709 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:46:51.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.329+0000 7f1383400640 1 -- 172.21.15.28:0/1961495657 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f1394030cc0 con 0x5593d61c2950 2024-04-23T14:46:51.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.329+0000 7f1383400640 10 client.25709 handle_command_reply: tid=0 2024-04-23T14:46:51.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.329+0000 7f13ab852740 1 client.25709 shutdown 2024-04-23T14:46:51.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.329+0000 7f13ab852740 1 -- 172.21.15.28:0/1961495657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593d61b2940 msgr2=0x5593d61afc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:51.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.329+0000 7f13ab852740 1 --2- 172.21.15.28:0/1961495657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593d61b2940 0x5593d61afc60 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f13940057c0 tx=0x7f139402a8b0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.329+0000 7f13ab852740 1 -- 172.21.15.28:0/1961495657 shutdown_connections 2024-04-23T14:46:51.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.329+0000 7f13ab852740 1 --2- 172.21.15.28:0/1961495657 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5593d61c2950 0x5593d61c4d40 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.329+0000 7f13ab852740 1 --2- 172.21.15.28:0/1961495657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593d61b2940 0x5593d61afc60 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.329+0000 7f13ab852740 1 --2- 172.21.15.28:0/1961495657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593d61a8270 0x5593d61af650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.329+0000 7f13ab852740 1 --2- 172.21.15.28:0/1961495657 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593d61a7950 0x5593d61b2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.329+0000 7f13ab852740 1 -- 172.21.15.28:0/1961495657 >> 172.21.15.28:0/1961495657 conn(0x5593d61a3c70 msgr2=0x5593d61a9950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:51.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.330+0000 7f1383400640 0 client.25709 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:46:51.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.330+0000 7f13ab852740 1 -- 172.21.15.28:0/1961495657 shutdown_connections 2024-04-23T14:46:51.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.330+0000 7f13ab852740 1 -- 172.21.15.28:0/1961495657 wait complete. 2024-04-23T14:46:51.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.330+0000 7f13ab852740 20 client.25709 trim_cache size 0 max 16384 2024-04-23T14:46:51.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.347+0000 7f13ab852740 1 Processor -- start 2024-04-23T14:46:51.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.347+0000 7f13ab852740 1 -- start start 2024-04-23T14:46:51.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.347+0000 7f13ab852740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593d61a7950 0x5593d61aa020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:51.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.347+0000 7f13ab852740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593d61a8270 0x5593d61b1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:51.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.347+0000 7f13ab852740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593d61b2940 0x5593d61ca310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:51.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.347+0000 7f13ab852740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f13a41b2320 con 0x5593d61a7950 2024-04-23T14:46:51.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.347+0000 7f13ab852740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f13a41b4ba0 con 0x5593d61a8270 2024-04-23T14:46:51.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.347+0000 7f13ab852740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f13a41b2990 con 0x5593d61b2940 2024-04-23T14:46:51.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.348+0000 7f13a1e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593d61a8270 0x5593d61b1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:51.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.348+0000 7f13a1e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593d61a8270 0x5593d61b1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60416/0 (socket says 172.21.15.28:60416) 2024-04-23T14:46:51.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.348+0000 7f13a1e00640 1 -- 172.21.15.28:0/1341806660 learned_addr learned my addr 172.21.15.28:0/1341806660 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:51.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.348+0000 7f13a2800640 1 --2- 172.21.15.28:0/1341806660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593d61a7950 0x5593d61aa020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:51.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.348+0000 7f13a2800640 1 --2- 172.21.15.28:0/1341806660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593d61b2940 0x5593d61ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:51.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.348+0000 7f13a1e00640 1 -- 172.21.15.28:0/1341806660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593d61b2940 msgr2=0x5593d61ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:51.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.348+0000 7f13a1e00640 1 --2- 172.21.15.28:0/1341806660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593d61b2940 0x5593d61ca310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.348+0000 7f13a1e00640 1 -- 172.21.15.28:0/1341806660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593d61a7950 msgr2=0x5593d61aa020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:51.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.348+0000 7f13a1e00640 1 --2- 172.21.15.28:0/1341806660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593d61a7950 0x5593d61aa020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.348+0000 7f13a1e00640 1 -- 172.21.15.28:0/1341806660 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593d5ee97d0 con 0x5593d61a8270 2024-04-23T14:46:51.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.348+0000 7f13a2800640 1 --2- 172.21.15.28:0/1341806660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593d61b2940 0x5593d61ca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:46:51.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.348+0000 7f13a1e00640 1 --2- 172.21.15.28:0/1341806660 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593d61a8270 0x5593d61b1e90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f139c000f80 tx=0x7f139c009650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e06f8fea76e80ffd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:51.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.349+0000 7f138e600640 1 -- 172.21.15.28:0/1341806660 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139c017ac0 con 0x5593d61a8270 2024-04-23T14:46:51.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.349+0000 7f138e600640 1 -- 172.21.15.28:0/1341806660 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f139c00ae30 con 0x5593d61a8270 2024-04-23T14:46:51.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.349+0000 7f138e600640 1 -- 172.21.15.28:0/1341806660 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139c0247b0 con 0x5593d61a8270 2024-04-23T14:46:51.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.349+0000 7f13ab852740 1 -- 172.21.15.28:0/1341806660 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593d61a8270 msgr2=0x5593d61b1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:51.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.349+0000 7f13ab852740 1 --2- 172.21.15.28:0/1341806660 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593d61a8270 0x5593d61b1e90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f139c000f80 tx=0x7f139c009650 comp rx=0 tx=0).stop 2024-04-23T14:46:51.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.350+0000 7f13ab852740 1 -- 172.21.15.28:0/1341806660 shutdown_connections 2024-04-23T14:46:51.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.350+0000 7f13ab852740 1 --2- 172.21.15.28:0/1341806660 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593d61a8270 0x5593d61b1e90 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.350+0000 7f13ab852740 1 --2- 172.21.15.28:0/1341806660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593d61b2940 0x5593d61ca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.350+0000 7f13ab852740 1 --2- 172.21.15.28:0/1341806660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593d61a7950 0x5593d61aa020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.350+0000 7f13ab852740 1 -- 172.21.15.28:0/1341806660 >> 172.21.15.28:0/1341806660 conn(0x5593d61a3c70 msgr2=0x5593d61a2df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:51.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.350+0000 7f13ab852740 1 -- 172.21.15.28:0/1341806660 shutdown_connections 2024-04-23T14:46:51.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.350+0000 7f13ab852740 1 -- 172.21.15.28:0/1341806660 wait complete. 2024-04-23T14:46:51.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.350+0000 7f13ab852740 1 -- start start 2024-04-23T14:46:51.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.350+0000 7f13ab852740 1 Processor -- start 2024-04-23T14:46:51.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.351+0000 7f13ab852740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593d61a7950 0x5593d61baff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:51.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.351+0000 7f13ab852740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593d61a8270 0x5593d61acbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:51.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.351+0000 7f13ab852740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593d61b2940 0x5593d61ad1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:51.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.351+0000 7f13ab852740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f13a41b3060 con 0x5593d61a8270 2024-04-23T14:46:51.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.351+0000 7f13ab852740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f13a41b44d0 con 0x5593d61a7950 2024-04-23T14:46:51.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.351+0000 7f13ab852740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f13a41b3730 con 0x5593d61b2940 2024-04-23T14:46:51.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.351+0000 7f13ab852740 10 client.0 _ll_register_callbacks cb 0x7f13900e55d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:46:51.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.351+0000 7f13a2800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593d61a7950 0x5593d61baff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:51.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.351+0000 7f13a1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593d61a8270 0x5593d61acbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:51.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.351+0000 7f13a1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593d61a8270 0x5593d61acbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42150/0 (socket says 172.21.15.28:42150) 2024-04-23T14:46:51.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.351+0000 7f13a1e00640 1 -- 172.21.15.28:0/2976252222 learned_addr learned my addr 172.21.15.28:0/2976252222 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:51.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.351+0000 7f13a2800640 1 --2- 172.21.15.28:0/2976252222 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593d61b2940 0x5593d61ad1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:51.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.351+0000 7f13a2800640 1 -- 172.21.15.28:0/2976252222 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593d61b2940 msgr2=0x5593d61ad1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:51.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.351+0000 7f13a2800640 1 --2- 172.21.15.28:0/2976252222 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593d61b2940 0x5593d61ad1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.351+0000 7f13a2800640 1 -- 172.21.15.28:0/2976252222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593d61a8270 msgr2=0x5593d61acbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.351+0000 7f13a2800640 1 --2- 172.21.15.28:0/2976252222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593d61a8270 0x5593d61acbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.351+0000 7f13a2800640 1 -- 172.21.15.28:0/2976252222 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593d61b3930 con 0x5593d61a7950 2024-04-23T14:46:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.352+0000 7f13a1e00640 1 --2- 172.21.15.28:0/2976252222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593d61a8270 0x5593d61acbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:46:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.352+0000 7f13a2800640 1 --2- 172.21.15.28:0/2976252222 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593d61a7950 0x5593d61baff0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f139402a9a0 tx=0x7f1394005b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=446c1574963a1df5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.352+0000 7f1383400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:46:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.352+0000 7f1383400640 1 -- 172.21.15.28:0/2976252222 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1394037040 con 0x5593d61a7950 2024-04-23T14:46:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.352+0000 7f1383400640 1 -- 172.21.15.28:0/2976252222 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f13940305c0 con 0x5593d61a7950 2024-04-23T14:46:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.352+0000 7f13ab852740 1 -- 172.21.15.28:0/2976252222 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f139c00ae30 con 0x5593d61a7950 2024-04-23T14:46:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.352+0000 7f1383400640 1 -- 172.21.15.28:0/2976252222 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f1394063050 con 0x5593d61a7950 2024-04-23T14:46:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.353+0000 7f13ab852740 10 client.25715 fetch_fsmap learned FSMap version 20 2024-04-23T14:46:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.353+0000 7f13ab852740 1 -- 172.21.15.28:0/2976252222 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5593d6228ee0 con 0x5593d61a7950 2024-04-23T14:46:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.353+0000 7f1383400640 1 -- 172.21.15.28:0/2976252222 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f139c00ae30 con 0x5593d61a7950 2024-04-23T14:46:51.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.353+0000 7f1383400640 1 -- 172.21.15.28:0/2976252222 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f139403a040 con 0x5593d61a7950 2024-04-23T14:46:51.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.353+0000 7f13ab852740 10 client.25715 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:46:51.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.353+0000 7f13ab852740 10 client.25715 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:46:51.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.353+0000 7f13ab852740 20 client.25715 populate_metadata read hostname 'smithi028' 2024-04-23T14:46:51.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.353+0000 7f13ab852740 1 --2- 172.21.15.28:0/2976252222 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5593d6229610 0x5593d622ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:51.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.353+0000 7f13ab852740 4 client.25715 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-23T14:46:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.353+0000 7f13ab852740 1 -- 172.21.15.28:0/2976252222 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5593d611abf0 con 0x5593d6229610 2024-04-23T14:46:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.354+0000 7f13a1e00640 1 --2- 172.21.15.28:0/2976252222 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5593d6229610 0x5593d622ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.354+0000 7f13a1e00640 1 --2- 172.21.15.28:0/2976252222 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5593d6229610 0x5593d622ba30 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db83574fa9a89ec8 server_cookie=19c5f96141d756c6 in_seq=0 out_seq=0 2024-04-23T14:46:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.354+0000 7f1383400640 10 client.25715 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:46:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.355+0000 7f1383400640 1 -- 172.21.15.28:0/2976252222 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5593d611abf0 con 0x5593d6229610 2024-04-23T14:46:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.355+0000 7f1383400640 10 client.25715 handle_command_reply: tid=0 2024-04-23T14:46:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.355+0000 7f13ab852740 1 client.25715 shutdown 2024-04-23T14:46:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.355+0000 7f13ab852740 1 -- 172.21.15.28:0/2976252222 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593d61a7950 msgr2=0x5593d61baff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.355+0000 7f13ab852740 1 --2- 172.21.15.28:0/2976252222 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593d61a7950 0x5593d61baff0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f139402a9a0 tx=0x7f1394005b20 comp rx=0 tx=0).stop 2024-04-23T14:46:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.355+0000 7f13ab852740 1 -- 172.21.15.28:0/2976252222 shutdown_connections 2024-04-23T14:46:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.355+0000 7f13ab852740 1 --2- 172.21.15.28:0/2976252222 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5593d6229610 0x5593d622ba30 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.355+0000 7f13ab852740 1 --2- 172.21.15.28:0/2976252222 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5593d61b2940 0x5593d61ad1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.355+0000 7f13ab852740 1 --2- 172.21.15.28:0/2976252222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593d61a8270 0x5593d61acbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.355+0000 7f13ab852740 1 --2- 172.21.15.28:0/2976252222 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593d61a7950 0x5593d61baff0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.355+0000 7f13ab852740 1 -- 172.21.15.28:0/2976252222 >> 172.21.15.28:0/2976252222 conn(0x5593d61a3c70 msgr2=0x5593d61aeeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:51.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.355+0000 7f1383400640 0 client.25715 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:46:51.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.356+0000 7f13ab852740 1 -- 172.21.15.28:0/2976252222 shutdown_connections 2024-04-23T14:46:51.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.356+0000 7f13ab852740 1 -- 172.21.15.28:0/2976252222 wait complete. 2024-04-23T14:46:51.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.356+0000 7f13ab852740 20 client.25715 trim_cache size 0 max 16384 2024-04-23T14:46:51.360 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-23T14:46:51.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.356+0000 7f13a8a00640 1 -- 172.21.15.28:0/2140956044 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f13880a3ea0 msgr2=0x7f13880a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:51.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.356+0000 7f13a8a00640 1 --2- 172.21.15.28:0/2140956044 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f13880a3ea0 0x7f13880a6360 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f13900df2e0 tx=0x7f13900e5700 comp rx=0 tx=0).stop 2024-04-23T14:46:51.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.356+0000 7f13a8a00640 1 -- 172.21.15.28:0/2140956044 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13a4110850 msgr2=0x7f13a4116fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:51.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.356+0000 7f13a8a00640 1 --2- 172.21.15.28:0/2140956044 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13a4110850 0x7f13a4116fc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f139001fa60 tx=0x7f139001ff20 comp rx=0 tx=0).stop 2024-04-23T14:46:51.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.357+0000 7f13a8a00640 1 -- 172.21.15.28:0/2140956044 shutdown_connections 2024-04-23T14:46:51.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.357+0000 7f13a8a00640 1 --2- 172.21.15.28:0/2140956044 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f13880a3ea0 0x7f13880a6360 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.357+0000 7f13a8a00640 1 --2- 172.21.15.28:0/2140956044 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13a4110850 0x7f13a4116fc0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.357+0000 7f13a8a00640 1 --2- 172.21.15.28:0/2140956044 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f13a411b7e0 0x7f13a4117500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.357+0000 7f13a8a00640 1 --2- 172.21.15.28:0/2140956044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13a410e880 0x7f13a411af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:51.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.357+0000 7f13a8a00640 1 -- 172.21.15.28:0/2140956044 >> 172.21.15.28:0/2140956044 conn(0x7f13a406d110 msgr2=0x7f13a411cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:51.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.357+0000 7f13a8a00640 1 -- 172.21.15.28:0/2140956044 shutdown_connections 2024-04-23T14:46:51.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:51.357+0000 7f13a8a00640 1 -- 172.21.15.28:0/2140956044 wait complete. 2024-04-23T14:46:51.468 INFO:tasks.workunit.client.0.smithi028.stdout: 1 133695 10.98 MB/sec warmup 67 sec latency 29.798 ms 2024-04-23T14:46:51.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:51 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2293019656' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:46:51.874 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:51 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2293019656' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:46:51.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:51 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2293019656' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:46:52.377 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-23T14:46:52.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:46:52.377 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-23T14:46:52.468 INFO:tasks.workunit.client.0.smithi028.stdout: 1 135276 10.99 MB/sec warmup 68 sec latency 39.336 ms 2024-04-23T14:46:52.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.579+0000 7f9caa800640 1 -- 172.21.15.28:0/723419587 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ca4104320 msgr2=0x7f9ca4104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:52.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.579+0000 7f9caa800640 1 --2- 172.21.15.28:0/723419587 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ca4104320 0x7f9ca4104700 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9c90009900 tx=0x7f9c9002a3f0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.579+0000 7f9caa800640 1 -- 172.21.15.28:0/723419587 shutdown_connections 2024-04-23T14:46:52.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.579+0000 7f9caa800640 1 --2- 172.21.15.28:0/723419587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ca41093f0 0x7f9ca4110fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.579+0000 7f9caa800640 1 --2- 172.21.15.28:0/723419587 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9ca4104cd0 0x7f9ca4108d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.579+0000 7f9caa800640 1 --2- 172.21.15.28:0/723419587 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ca4104320 0x7f9ca4104700 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.579+0000 7f9caa800640 1 -- 172.21.15.28:0/723419587 >> 172.21.15.28:0/723419587 conn(0x7f9ca40ffb40 msgr2=0x7f9ca4101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:52.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.579+0000 7f9caa800640 1 -- 172.21.15.28:0/723419587 shutdown_connections 2024-04-23T14:46:52.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.579+0000 7f9caa800640 1 -- 172.21.15.28:0/723419587 wait complete. 2024-04-23T14:46:52.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.580+0000 7f9caa800640 1 Processor -- start 2024-04-23T14:46:52.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.580+0000 7f9caa800640 1 -- start start 2024-04-23T14:46:52.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.580+0000 7f9caa800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9ca4104320 0x7f9ca41a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.581+0000 7f9caa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ca41093f0 0x7f9ca41a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.581+0000 7f9caa800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ca419a7d0 0x7f9ca419ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.581+0000 7f9caa800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca41144d0 con 0x7f9ca41093f0 2024-04-23T14:46:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.581+0000 7f9caa800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca4114350 con 0x7f9ca419a7d0 2024-04-23T14:46:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.581+0000 7f9caa800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca4114650 con 0x7f9ca4104320 2024-04-23T14:46:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.581+0000 7f9ca9e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ca419a7d0 0x7f9ca419ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.581+0000 7f9ca8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ca41093f0 0x7f9ca41a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.581+0000 7f9ca9e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ca419a7d0 0x7f9ca419ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60442/0 (socket says 172.21.15.28:60442) 2024-04-23T14:46:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.581+0000 7f9ca9400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9ca4104320 0x7f9ca41a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.581+0000 7f9ca9e00640 1 -- 172.21.15.28:0/680505303 learned_addr learned my addr 172.21.15.28:0/680505303 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.581+0000 7f9ca9e00640 1 -- 172.21.15.28:0/680505303 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9ca4104320 msgr2=0x7f9ca41a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.581+0000 7f9ca9e00640 1 --2- 172.21.15.28:0/680505303 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9ca4104320 0x7f9ca41a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.581+0000 7f9ca9e00640 1 -- 172.21.15.28:0/680505303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ca41093f0 msgr2=0x7f9ca41a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.581+0000 7f9ca9e00640 1 --2- 172.21.15.28:0/680505303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ca41093f0 0x7f9ca41a0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.581+0000 7f9ca9e00640 1 -- 172.21.15.28:0/680505303 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca419b360 con 0x7f9ca419a7d0 2024-04-23T14:46:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.581+0000 7f9ca8a00640 1 --2- 172.21.15.28:0/680505303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ca41093f0 0x7f9ca41a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:46:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.582+0000 7f9ca9e00640 1 --2- 172.21.15.28:0/680505303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ca419a7d0 0x7f9ca419ac80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c00eb10 tx=0x7f9c9c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.582+0000 7f9ca2000640 1 -- 172.21.15.28:0/680505303 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9c004b20 con 0x7f9ca419a7d0 2024-04-23T14:46:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.582+0000 7f9ca2000640 1 -- 172.21.15.28:0/680505303 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c9c004cc0 con 0x7f9ca419a7d0 2024-04-23T14:46:52.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.582+0000 7f9ca2000640 1 -- 172.21.15.28:0/680505303 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9c010760 con 0x7f9ca419a7d0 2024-04-23T14:46:52.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.582+0000 7f9caa800640 1 -- 172.21.15.28:0/680505303 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ca419b650 con 0x7f9ca419a7d0 2024-04-23T14:46:52.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.582+0000 7f9caa800640 1 -- 172.21.15.28:0/680505303 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ca41a77e0 con 0x7f9ca419a7d0 2024-04-23T14:46:52.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.585+0000 7f9caed67740 1 Processor -- start 2024-04-23T14:46:52.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.586+0000 7f9ca2000640 1 -- 172.21.15.28:0/680505303 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f9c9c005020 con 0x7f9ca419a7d0 2024-04-23T14:46:52.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.586+0000 7f9caed67740 1 -- start start 2024-04-23T14:46:52.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.587+0000 7f9caed67740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6ecbfd270 0x55b6ecbfd650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.587+0000 7f9caed67740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6ecbfdb90 0x55b6ecc07d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.587+0000 7f9caed67740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6ecc08260 0x55b6ecc0a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.587+0000 7f9caed67740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b6ecb62e70 con 0x55b6ecc08260 2024-04-23T14:46:52.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.587+0000 7f9caed67740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b6ecb09380 con 0x55b6ecbfdb90 2024-04-23T14:46:52.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.587+0000 7f9caed67740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b6ecb5ee30 con 0x55b6ecbfd270 2024-04-23T14:46:52.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.587+0000 7f9ca9400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6ecbfdb90 0x55b6ecc07d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.587+0000 7f9ca9400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6ecbfdb90 0x55b6ecc07d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60446/0 (socket says 172.21.15.28:60446) 2024-04-23T14:46:52.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.587+0000 7f9ca9400640 1 -- 172.21.15.28:0/4048917607 learned_addr learned my addr 172.21.15.28:0/4048917607 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:52.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.587+0000 7f9ca8a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6ecbfd270 0x55b6ecbfd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.587+0000 7f9ca8a00640 1 --2- 172.21.15.28:0/4048917607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6ecc08260 0x55b6ecc0a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.587+0000 7f9ca8a00640 1 -- 172.21.15.28:0/4048917607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6ecbfdb90 msgr2=0x55b6ecc07d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.587+0000 7f9ca8a00640 1 --2- 172.21.15.28:0/4048917607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6ecbfdb90 0x55b6ecc07d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.587+0000 7f9ca8a00640 1 -- 172.21.15.28:0/4048917607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6ecc08260 msgr2=0x55b6ecc0a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.587+0000 7f9ca8a00640 1 --2- 172.21.15.28:0/4048917607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6ecc08260 0x55b6ecc0a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.587+0000 7f9ca8a00640 1 -- 172.21.15.28:0/4048917607 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6ec93d7e0 con 0x55b6ecbfd270 2024-04-23T14:46:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.587+0000 7f9ca9400640 1 --2- 172.21.15.28:0/4048917607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6ecbfdb90 0x55b6ecc07d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:46:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.588+0000 7f9ca8a00640 1 --2- 172.21.15.28:0/4048917607 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6ecbfd270 0x55b6ecbfd650 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9ca419bb20 tx=0x7f9c980046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=832646b7ff42eb42 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.588+0000 7f9c96600640 1 -- 172.21.15.28:0/4048917607 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c98004030 con 0x55b6ecbfd270 2024-04-23T14:46:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.588+0000 7f9c96600640 1 -- 172.21.15.28:0/4048917607 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c98005500 con 0x55b6ecbfd270 2024-04-23T14:46:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.588+0000 7f9c96600640 1 -- 172.21.15.28:0/4048917607 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9800fe30 con 0x55b6ecbfd270 2024-04-23T14:46:52.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.588+0000 7f9ca2000640 1 --2- 172.21.15.28:0/680505303 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f9c8c0a3da0 0x7f9c8c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.588+0000 7f9ca2000640 1 -- 172.21.15.28:0/680505303 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f9c9c0d0d00 con 0x7f9ca419a7d0 2024-04-23T14:46:52.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.589+0000 7f9ca9400640 1 --2- 172.21.15.28:0/680505303 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f9c8c0a3da0 0x7f9c8c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.589+0000 7f9caed67740 1 -- 172.21.15.28:0/4048917607 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6ecbfd270 msgr2=0x55b6ecbfd650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:52.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.589+0000 7f9caed67740 1 --2- 172.21.15.28:0/4048917607 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6ecbfd270 0x55b6ecbfd650 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9ca419bb20 tx=0x7f9c980046c0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.589+0000 7f9caed67740 1 -- 172.21.15.28:0/4048917607 shutdown_connections 2024-04-23T14:46:52.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.589+0000 7f9caed67740 1 --2- 172.21.15.28:0/4048917607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6ecbfdb90 0x55b6ecc07d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.589+0000 7f9caed67740 1 --2- 172.21.15.28:0/4048917607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6ecc08260 0x55b6ecc0a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.589+0000 7f9caed67740 1 --2- 172.21.15.28:0/4048917607 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6ecbfd270 0x55b6ecbfd650 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.589+0000 7f9caed67740 1 -- 172.21.15.28:0/4048917607 >> 172.21.15.28:0/4048917607 conn(0x55b6ecbf9590 msgr2=0x55b6ecbf9970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:52.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.589+0000 7f9caed67740 1 -- 172.21.15.28:0/4048917607 shutdown_connections 2024-04-23T14:46:52.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.589+0000 7f9caed67740 1 -- 172.21.15.28:0/4048917607 wait complete. 2024-04-23T14:46:52.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.589+0000 7f9ca9400640 1 --2- 172.21.15.28:0/680505303 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f9c8c0a3da0 0x7f9c8c0a6260 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9c90005c40 tx=0x7f9c90005b50 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:52.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.590+0000 7f9caed67740 1 -- start start 2024-04-23T14:46:52.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.590+0000 7f9caed67740 1 Processor -- start 2024-04-23T14:46:52.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.590+0000 7f9caed67740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6ecbfd270 0x55b6ecc05180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.591+0000 7f9caed67740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6ecbfdb90 0x55b6ecc056c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.591+0000 7f9caed67740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6ecc08260 0x55b6ecc05c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.591+0000 7f9caed67740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b6ecc0b050 con 0x55b6ecc08260 2024-04-23T14:46:52.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.591+0000 7f9caed67740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b6ecb1c660 con 0x55b6ecbfd270 2024-04-23T14:46:52.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.591+0000 7f9caed67740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b6ecc0b1d0 con 0x55b6ecbfdb90 2024-04-23T14:46:52.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.591+0000 7f9caed67740 10 client.0 _ll_register_callbacks cb 0x55b6ecb0aaa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:46:52.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.591+0000 7f9ca9400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6ecbfd270 0x55b6ecc05180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.591+0000 7f9ca8a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6ecbfdb90 0x55b6ecc056c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.591+0000 7f9ca9400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6ecbfd270 0x55b6ecc05180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60456/0 (socket says 172.21.15.28:60456) 2024-04-23T14:46:52.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.591+0000 7f9ca9400640 1 -- 172.21.15.28:0/3149510847 learned_addr learned my addr 172.21.15.28:0/3149510847 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:52.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.591+0000 7f9ca9e00640 1 --2- 172.21.15.28:0/3149510847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6ecc08260 0x55b6ecc05c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.591+0000 7f9ca8a00640 1 -- 172.21.15.28:0/3149510847 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6ecbfd270 msgr2=0x55b6ecc05180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:52.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.591+0000 7f9ca8a00640 1 --2- 172.21.15.28:0/3149510847 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6ecbfd270 0x55b6ecc05180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.591+0000 7f9ca8a00640 1 -- 172.21.15.28:0/3149510847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6ecc08260 msgr2=0x55b6ecc05c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:52.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.591+0000 7f9ca8a00640 1 --2- 172.21.15.28:0/3149510847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6ecc08260 0x55b6ecc05c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.591+0000 7f9ca8a00640 1 -- 172.21.15.28:0/3149510847 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6ec99df70 con 0x55b6ecbfdb90 2024-04-23T14:46:52.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.591+0000 7f9ca9e00640 1 --2- 172.21.15.28:0/3149510847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6ecc08260 0x55b6ecc05c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:46:52.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.592+0000 7f9ca8a00640 1 --2- 172.21.15.28:0/3149510847 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6ecbfdb90 0x55b6ecc056c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9c98004600 tx=0x7f9c9800ffb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=553f52e80bbc0d1a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:52.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.592+0000 7f9c8aa00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:46:52.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.592+0000 7f9c8aa00640 1 -- 172.21.15.28:0/3149510847 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9802c060 con 0x55b6ecbfdb90 2024-04-23T14:46:52.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.592+0000 7f9c8aa00640 1 -- 172.21.15.28:0/3149510847 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f9c98005230 con 0x55b6ecbfdb90 2024-04-23T14:46:52.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.592+0000 7f9caed67740 1 -- 172.21.15.28:0/3149510847 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c98005500 con 0x55b6ecbfdb90 2024-04-23T14:46:52.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.592+0000 7f9c8aa00640 1 -- 172.21.15.28:0/3149510847 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f9c98004730 con 0x55b6ecbfdb90 2024-04-23T14:46:52.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.592+0000 7f9caed67740 10 client.25732 fetch_fsmap learned FSMap version 20 2024-04-23T14:46:52.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.592+0000 7f9caed67740 1 -- 172.21.15.28:0/3149510847 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55b6ec835170 con 0x55b6ecbfdb90 2024-04-23T14:46:52.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.592+0000 7f9c8aa00640 1 -- 172.21.15.28:0/3149510847 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c98005500 con 0x55b6ecbfdb90 2024-04-23T14:46:52.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.593+0000 7f9c8aa00640 1 -- 172.21.15.28:0/3149510847 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f9c9801a050 con 0x55b6ecbfdb90 2024-04-23T14:46:52.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.593+0000 7f9caed67740 10 client.25732 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:46:52.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.593+0000 7f9caed67740 10 client.25732 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:46:52.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.593+0000 7f9caed67740 20 client.25732 populate_metadata read hostname 'smithi028' 2024-04-23T14:46:52.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.593+0000 7f9caed67740 1 --2- 172.21.15.28:0/3149510847 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55b6ecc184f0 0x55b6ecc1a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.593+0000 7f9caed67740 4 client.25732 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:46:52.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.593+0000 7f9caed67740 1 -- 172.21.15.28:0/3149510847 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b6ecb70890 con 0x55b6ecc184f0 2024-04-23T14:46:52.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.593+0000 7f9ca9400640 1 --2- 172.21.15.28:0/3149510847 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55b6ecc184f0 0x55b6ecc1a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.594+0000 7f9ca9400640 1 --2- 172.21.15.28:0/3149510847 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55b6ecc184f0 0x55b6ecc1a8e0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=332ab468a4aa577d server_cookie=6d4ae0f968c66cf9 in_seq=0 out_seq=0 2024-04-23T14:46:52.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.594+0000 7f9c8aa00640 10 client.25732 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:46:52.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.595+0000 7f9c8aa00640 1 -- 172.21.15.28:0/3149510847 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b6ecb70890 con 0x55b6ecc184f0 2024-04-23T14:46:52.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.595+0000 7f9c8aa00640 10 client.25732 handle_command_reply: tid=0 2024-04-23T14:46:52.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.595+0000 7f9caed67740 1 client.25732 shutdown 2024-04-23T14:46:52.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.595+0000 7f9caed67740 1 -- 172.21.15.28:0/3149510847 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6ecbfdb90 msgr2=0x55b6ecc056c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:52.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.595+0000 7f9caed67740 1 --2- 172.21.15.28:0/3149510847 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6ecbfdb90 0x55b6ecc056c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9c98004600 tx=0x7f9c9800ffb0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.595+0000 7f9caed67740 1 -- 172.21.15.28:0/3149510847 shutdown_connections 2024-04-23T14:46:52.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.595+0000 7f9caed67740 1 --2- 172.21.15.28:0/3149510847 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55b6ecc184f0 0x55b6ecc1a8e0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.595+0000 7f9caed67740 1 --2- 172.21.15.28:0/3149510847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6ecc08260 0x55b6ecc05c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.595+0000 7f9caed67740 1 --2- 172.21.15.28:0/3149510847 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6ecbfdb90 0x55b6ecc056c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.595+0000 7f9c8aa00640 0 client.25732 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:46:52.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.595+0000 7f9caed67740 1 --2- 172.21.15.28:0/3149510847 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6ecbfd270 0x55b6ecc05180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.596+0000 7f9caed67740 1 -- 172.21.15.28:0/3149510847 >> 172.21.15.28:0/3149510847 conn(0x55b6ecbf9590 msgr2=0x55b6ecbff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:52.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.596+0000 7f9caed67740 1 -- 172.21.15.28:0/3149510847 shutdown_connections 2024-04-23T14:46:52.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.596+0000 7f9caed67740 1 -- 172.21.15.28:0/3149510847 wait complete. 2024-04-23T14:46:52.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.596+0000 7f9caed67740 20 client.25732 trim_cache size 0 max 16384 2024-04-23T14:46:52.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.617+0000 7f9caed67740 1 Processor -- start 2024-04-23T14:46:52.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.617+0000 7f9caed67740 1 -- start start 2024-04-23T14:46:52.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.618+0000 7f9caed67740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6ecbfd270 0x55b6ecc13f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.618+0000 7f9caed67740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6ecbfdb90 0x55b6ecc1ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.618+0000 7f9caed67740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6ecc08260 0x55b6ecc20450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.618+0000 7f9caed67740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca41add50 con 0x55b6ecbfd270 2024-04-23T14:46:52.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.618+0000 7f9caed67740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca41b05d0 con 0x55b6ecbfdb90 2024-04-23T14:46:52.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.618+0000 7f9caed67740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca41ae3c0 con 0x55b6ecc08260 2024-04-23T14:46:52.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.618+0000 7f9ca8a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6ecbfdb90 0x55b6ecc1ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.618+0000 7f9ca9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6ecbfd270 0x55b6ecc13f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.618+0000 7f9ca8a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6ecbfdb90 0x55b6ecc1ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60466/0 (socket says 172.21.15.28:60466) 2024-04-23T14:46:52.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.618+0000 7f9ca8a00640 1 -- 172.21.15.28:0/80253689 learned_addr learned my addr 172.21.15.28:0/80253689 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:52.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.618+0000 7f9ca9e00640 1 --2- 172.21.15.28:0/80253689 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6ecc08260 0x55b6ecc20450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.619+0000 7f9ca8a00640 1 -- 172.21.15.28:0/80253689 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6ecc08260 msgr2=0x55b6ecc20450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:52.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.619+0000 7f9ca8a00640 1 --2- 172.21.15.28:0/80253689 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6ecc08260 0x55b6ecc20450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.619+0000 7f9ca8a00640 1 -- 172.21.15.28:0/80253689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6ecbfd270 msgr2=0x55b6ecc13f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:52.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.619+0000 7f9ca8a00640 1 --2- 172.21.15.28:0/80253689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6ecbfd270 0x55b6ecc13f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.619+0000 7f9ca8a00640 1 -- 172.21.15.28:0/80253689 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6ecbfd980 con 0x55b6ecbfdb90 2024-04-23T14:46:52.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.619+0000 7f9ca8a00640 1 --2- 172.21.15.28:0/80253689 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6ecbfdb90 0x55b6ecc1ff10 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9c9800f960 tx=0x7f9c98009370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=223a759f4f17035e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:52.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.619+0000 7f9c95c00640 1 -- 172.21.15.28:0/80253689 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9802c070 con 0x55b6ecbfdb90 2024-04-23T14:46:52.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.619+0000 7f9c95c00640 1 -- 172.21.15.28:0/80253689 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c98009c00 con 0x55b6ecbfdb90 2024-04-23T14:46:52.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.619+0000 7f9c95c00640 1 -- 172.21.15.28:0/80253689 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c980048d0 con 0x55b6ecbfdb90 2024-04-23T14:46:52.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.620+0000 7f9caed67740 1 -- 172.21.15.28:0/80253689 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6ecbfdb90 msgr2=0x55b6ecc1ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:52.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.620+0000 7f9caed67740 1 --2- 172.21.15.28:0/80253689 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6ecbfdb90 0x55b6ecc1ff10 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9c9800f960 tx=0x7f9c98009370 comp rx=0 tx=0).stop 2024-04-23T14:46:52.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.620+0000 7f9caed67740 1 -- 172.21.15.28:0/80253689 shutdown_connections 2024-04-23T14:46:52.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.620+0000 7f9caed67740 1 --2- 172.21.15.28:0/80253689 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6ecbfdb90 0x55b6ecc1ff10 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.620+0000 7f9caed67740 1 --2- 172.21.15.28:0/80253689 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6ecc08260 0x55b6ecc20450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.620+0000 7f9caed67740 1 --2- 172.21.15.28:0/80253689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6ecbfd270 0x55b6ecc13f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.620+0000 7f9caed67740 1 -- 172.21.15.28:0/80253689 >> 172.21.15.28:0/80253689 conn(0x55b6ecbf9590 msgr2=0x55b6ecbf9970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:52.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.620+0000 7f9caed67740 1 -- 172.21.15.28:0/80253689 shutdown_connections 2024-04-23T14:46:52.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.620+0000 7f9caed67740 1 -- 172.21.15.28:0/80253689 wait complete. 2024-04-23T14:46:52.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.621+0000 7f9caed67740 1 -- start start 2024-04-23T14:46:52.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.621+0000 7f9caed67740 1 Processor -- start 2024-04-23T14:46:52.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.621+0000 7f9caed67740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6ecbfd270 0x55b6ecc02520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.621+0000 7f9caed67740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6ecbfdb90 0x55b6ecc02a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.621+0000 7f9caed67740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6ecc08260 0x55b6ecc7d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.622+0000 7f9caed67740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca41aea90 con 0x55b6ecc08260 2024-04-23T14:46:52.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.622+0000 7f9caed67740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca41aff00 con 0x55b6ecbfd270 2024-04-23T14:46:52.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.622+0000 7f9caed67740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca41af160 con 0x55b6ecbfdb90 2024-04-23T14:46:52.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.622+0000 7f9caed67740 10 client.0 _ll_register_callbacks cb 0x7f9c98005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:46:52.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.622+0000 7f9ca9400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6ecbfd270 0x55b6ecc02520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.622+0000 7f9ca9400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6ecbfd270 0x55b6ecc02520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60468/0 (socket says 172.21.15.28:60468) 2024-04-23T14:46:52.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.622+0000 7f9ca9400640 1 -- 172.21.15.28:0/1575688414 learned_addr learned my addr 172.21.15.28:0/1575688414 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:52.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.622+0000 7f9ca9e00640 1 --2- 172.21.15.28:0/1575688414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6ecc08260 0x55b6ecc7d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.622+0000 7f9ca8a00640 1 --2- 172.21.15.28:0/1575688414 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6ecbfdb90 0x55b6ecc02a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.622+0000 7f9ca9400640 1 -- 172.21.15.28:0/1575688414 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6ecbfdb90 msgr2=0x55b6ecc02a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:52.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.622+0000 7f9ca9400640 1 --2- 172.21.15.28:0/1575688414 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6ecbfdb90 0x55b6ecc02a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.622+0000 7f9ca9400640 1 -- 172.21.15.28:0/1575688414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6ecc08260 msgr2=0x55b6ecc7d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:52.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.622+0000 7f9ca9400640 1 --2- 172.21.15.28:0/1575688414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6ecc08260 0x55b6ecc7d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.622+0000 7f9ca9400640 1 -- 172.21.15.28:0/1575688414 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6ecc7dcc0 con 0x55b6ecbfd270 2024-04-23T14:46:52.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.623+0000 7f9ca9400640 1 --2- 172.21.15.28:0/1575688414 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6ecbfd270 0x55b6ecc02520 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9c90009230 tx=0x7f9c90009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82aa2406d5a00ef3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:52.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.623+0000 7f9c8aa00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:46:52.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.623+0000 7f9c8aa00640 1 -- 172.21.15.28:0/1575688414 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c90035040 con 0x55b6ecbfd270 2024-04-23T14:46:52.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.623+0000 7f9c8aa00640 1 -- 172.21.15.28:0/1575688414 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f9c90054500 con 0x55b6ecbfd270 2024-04-23T14:46:52.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.623+0000 7f9caed67740 1 -- 172.21.15.28:0/1575688414 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c98009c00 con 0x55b6ecbfd270 2024-04-23T14:46:52.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.623+0000 7f9c8aa00640 1 -- 172.21.15.28:0/1575688414 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f9c90054700 con 0x55b6ecbfd270 2024-04-23T14:46:52.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.623+0000 7f9caed67740 10 client.25733 fetch_fsmap learned FSMap version 20 2024-04-23T14:46:52.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.623+0000 7f9caed67740 1 -- 172.21.15.28:0/1575688414 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55b6ecc7f000 con 0x55b6ecbfd270 2024-04-23T14:46:52.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.623+0000 7f9c8aa00640 1 -- 172.21.15.28:0/1575688414 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c98009c00 con 0x55b6ecbfd270 2024-04-23T14:46:52.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.624+0000 7f9c8aa00640 1 -- 172.21.15.28:0/1575688414 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f9c9005b070 con 0x55b6ecbfd270 2024-04-23T14:46:52.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.624+0000 7f9caed67740 10 client.25733 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:46:52.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.624+0000 7f9caed67740 10 client.25733 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:46:52.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.624+0000 7f9caed67740 20 client.25733 populate_metadata read hostname 'smithi028' 2024-04-23T14:46:52.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.624+0000 7f9caed67740 1 --2- 172.21.15.28:0/1575688414 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55b6ecc7f700 0x55b6ecc81b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.624+0000 7f9caed67740 4 client.25733 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-23T14:46:52.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.624+0000 7f9caed67740 1 -- 172.21.15.28:0/1575688414 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b6ecb6a700 con 0x55b6ecc7f700 2024-04-23T14:46:52.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.624+0000 7f9ca8a00640 1 --2- 172.21.15.28:0/1575688414 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55b6ecc7f700 0x55b6ecc81b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.624+0000 7f9ca8a00640 1 --2- 172.21.15.28:0/1575688414 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55b6ecc7f700 0x55b6ecc81b20 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f14c3f8fc496219 server_cookie=41509ed62296b574 in_seq=0 out_seq=0 2024-04-23T14:46:52.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.624+0000 7f9c8aa00640 10 client.25733 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:46:52.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.625+0000 7f9c8aa00640 1 -- 172.21.15.28:0/1575688414 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b6ecb6a700 con 0x55b6ecc7f700 2024-04-23T14:46:52.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.625+0000 7f9c8aa00640 10 client.25733 handle_command_reply: tid=0 2024-04-23T14:46:52.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.625+0000 7f9caed67740 1 client.25733 shutdown 2024-04-23T14:46:52.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.626+0000 7f9caed67740 1 -- 172.21.15.28:0/1575688414 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6ecbfd270 msgr2=0x55b6ecc02520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:52.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.626+0000 7f9caed67740 1 --2- 172.21.15.28:0/1575688414 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6ecbfd270 0x55b6ecc02520 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9c90009230 tx=0x7f9c90009260 comp rx=0 tx=0).stop 2024-04-23T14:46:52.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.626+0000 7f9caed67740 1 -- 172.21.15.28:0/1575688414 shutdown_connections 2024-04-23T14:46:52.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.626+0000 7f9caed67740 1 --2- 172.21.15.28:0/1575688414 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55b6ecc7f700 0x55b6ecc81b20 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.626+0000 7f9caed67740 1 --2- 172.21.15.28:0/1575688414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6ecc08260 0x55b6ecc7d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.626+0000 7f9caed67740 1 --2- 172.21.15.28:0/1575688414 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b6ecbfdb90 0x55b6ecc02a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.626+0000 7f9caed67740 1 --2- 172.21.15.28:0/1575688414 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6ecbfd270 0x55b6ecc02520 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.626+0000 7f9c8aa00640 0 client.25733 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:46:52.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.626+0000 7f9caed67740 1 -- 172.21.15.28:0/1575688414 >> 172.21.15.28:0/1575688414 conn(0x55b6ecbf9590 msgr2=0x55b6ecc05120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:52.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.626+0000 7f9caed67740 1 -- 172.21.15.28:0/1575688414 shutdown_connections 2024-04-23T14:46:52.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.626+0000 7f9caed67740 1 -- 172.21.15.28:0/1575688414 wait complete. 2024-04-23T14:46:52.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.626+0000 7f9caed67740 20 client.25733 trim_cache size 0 max 16384 2024-04-23T14:46:52.629 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:46:52.629 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-23T14:46:52.630 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "ae24cd23-44a8-4fcf-b69d-9c09dda0f02a", 2024-04-23T14:46:52.630 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-23T14:46:52.630 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:46:52.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.627+0000 7f9caa800640 1 -- 172.21.15.28:0/680505303 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f9c8c0a3da0 msgr2=0x7f9c8c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:52.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.627+0000 7f9caa800640 1 --2- 172.21.15.28:0/680505303 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f9c8c0a3da0 0x7f9c8c0a6260 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9c90005c40 tx=0x7f9c90005b50 comp rx=0 tx=0).stop 2024-04-23T14:46:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.627+0000 7f9caa800640 1 -- 172.21.15.28:0/680505303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ca419a7d0 msgr2=0x7f9ca419ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.627+0000 7f9caa800640 1 --2- 172.21.15.28:0/680505303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ca419a7d0 0x7f9ca419ac80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c00eb10 tx=0x7f9c9c00c2d0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.628+0000 7f9caa800640 1 -- 172.21.15.28:0/680505303 shutdown_connections 2024-04-23T14:46:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.628+0000 7f9caa800640 1 --2- 172.21.15.28:0/680505303 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f9c8c0a3da0 0x7f9c8c0a6260 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.628+0000 7f9caa800640 1 --2- 172.21.15.28:0/680505303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ca419a7d0 0x7f9ca419ac80 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.628+0000 7f9caa800640 1 --2- 172.21.15.28:0/680505303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ca41093f0 0x7f9ca41a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.628+0000 7f9caa800640 1 --2- 172.21.15.28:0/680505303 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9ca4104320 0x7f9ca41a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.628+0000 7f9caa800640 1 -- 172.21.15.28:0/680505303 >> 172.21.15.28:0/680505303 conn(0x7f9ca40ffb40 msgr2=0x7f9ca4101ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.628+0000 7f9caa800640 1 -- 172.21.15.28:0/680505303 shutdown_connections 2024-04-23T14:46:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.628+0000 7f9caa800640 1 -- 172.21.15.28:0/680505303 wait complete. 2024-04-23T14:46:52.650 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:46:52.651 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:46:52.718 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:52 smithi028 ceph-mon[27810]: pgmap v88: 97 pgs: 97 active+clean; 81 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 2.3 MiB/s wr, 156 op/s 2024-04-23T14:46:52.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:52 smithi105 ceph-mon[32555]: pgmap v88: 97 pgs: 97 active+clean; 81 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 2.3 MiB/s wr, 156 op/s 2024-04-23T14:46:52.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.861+0000 7fa559a00640 1 -- 172.21.15.28:0/3447453889 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa554108fd0 msgr2=0x7fa554115490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:52.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.861+0000 7fa559a00640 1 --2- 172.21.15.28:0/3447453889 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa554108fd0 0x7fa554115490 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa54c009cd0 tx=0x7fa54c02a8f0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.861+0000 7fa559a00640 1 -- 172.21.15.28:0/3447453889 shutdown_connections 2024-04-23T14:46:52.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.861+0000 7fa559a00640 1 --2- 172.21.15.28:0/3447453889 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa554108fd0 0x7fa554115490 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.861+0000 7fa559a00640 1 --2- 172.21.15.28:0/3447453889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa554108610 0x7fa554108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.861+0000 7fa559a00640 1 --2- 172.21.15.28:0/3447453889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa55410e660 0x7fa55410ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.861+0000 7fa559a00640 1 -- 172.21.15.28:0/3447453889 >> 172.21.15.28:0/3447453889 conn(0x7fa554076540 msgr2=0x7fa554078960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:52.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.861+0000 7fa559a00640 1 -- 172.21.15.28:0/3447453889 shutdown_connections 2024-04-23T14:46:52.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.861+0000 7fa559a00640 1 -- 172.21.15.28:0/3447453889 wait complete. 2024-04-23T14:46:52.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.862+0000 7fa559a00640 1 Processor -- start 2024-04-23T14:46:52.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.862+0000 7fa559a00640 1 -- start start 2024-04-23T14:46:52.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.863+0000 7fa559a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa554108610 0x7fa55419ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.863+0000 7fa559a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa554108fd0 0x7fa55419f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.863+0000 7fa559a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa55410e660 0x7fa5541a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.863+0000 7fa559a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa5541189f0 con 0x7fa55410e660 2024-04-23T14:46:52.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.863+0000 7fa559a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa554118870 con 0x7fa554108fd0 2024-04-23T14:46:52.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.863+0000 7fa559a00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa554118b70 con 0x7fa554108610 2024-04-23T14:46:52.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.863+0000 7fa552a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa554108610 0x7fa55419ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.863+0000 7fa54be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa554108fd0 0x7fa55419f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.863+0000 7fa552a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa554108610 0x7fa55419ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:42922/0 (socket says 172.21.15.28:42922) 2024-04-23T14:46:52.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.863+0000 7fa54be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa554108fd0 0x7fa55419f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60488/0 (socket says 172.21.15.28:60488) 2024-04-23T14:46:52.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.863+0000 7fa552a00640 1 -- 172.21.15.28:0/3184363071 learned_addr learned my addr 172.21.15.28:0/3184363071 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:52.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.863+0000 7fa553400640 1 --2- 172.21.15.28:0/3184363071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa55410e660 0x7fa5541a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.863+0000 7fa54be00640 1 -- 172.21.15.28:0/3184363071 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa554108610 msgr2=0x7fa55419ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:52.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.863+0000 7fa54be00640 1 --2- 172.21.15.28:0/3184363071 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa554108610 0x7fa55419ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.863+0000 7fa54be00640 1 -- 172.21.15.28:0/3184363071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa55410e660 msgr2=0x7fa5541a36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:52.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.863+0000 7fa54be00640 1 --2- 172.21.15.28:0/3184363071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa55410e660 0x7fa5541a36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.863+0000 7fa54be00640 1 -- 172.21.15.28:0/3184363071 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5541a3e00 con 0x7fa554108fd0 2024-04-23T14:46:52.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.863+0000 7fa552a00640 1 --2- 172.21.15.28:0/3184363071 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa554108610 0x7fa55419ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:46:52.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.863+0000 7fa553400640 1 --2- 172.21.15.28:0/3184363071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa55410e660 0x7fa5541a36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:46:52.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.864+0000 7fa54be00640 1 --2- 172.21.15.28:0/3184363071 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa554108fd0 0x7fa55419f350 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa53c00e9f0 tx=0x7fa53c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:52.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.864+0000 7fa54b400640 1 -- 172.21.15.28:0/3184363071 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa53c00ccc0 con 0x7fa554108fd0 2024-04-23T14:46:52.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.864+0000 7fa54b400640 1 -- 172.21.15.28:0/3184363071 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa53c010910 con 0x7fa554108fd0 2024-04-23T14:46:52.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.864+0000 7fa54b400640 1 -- 172.21.15.28:0/3184363071 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa53c018960 con 0x7fa554108fd0 2024-04-23T14:46:52.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.864+0000 7fa559a00640 1 -- 172.21.15.28:0/3184363071 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5541a40f0 con 0x7fa554108fd0 2024-04-23T14:46:52.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.864+0000 7fa559a00640 1 -- 172.21.15.28:0/3184363071 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5541ab980 con 0x7fa554108fd0 2024-04-23T14:46:52.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.868+0000 7fa55c878740 1 Processor -- start 2024-04-23T14:46:52.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.869+0000 7fa55c878740 1 -- start start 2024-04-23T14:46:52.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.869+0000 7fa55c878740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562622b65860 0x562622b65c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.869+0000 7fa55c878740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562622b66180 0x562622b70310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.869+0000 7fa55c878740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562622b70850 0x562622b72c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.869+0000 7fa55c878740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562622acb1f0 con 0x562622b66180 2024-04-23T14:46:52.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.869+0000 7fa55c878740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562622a71350 con 0x562622b65860 2024-04-23T14:46:52.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.869+0000 7fa55c878740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562622ac71b0 con 0x562622b70850 2024-04-23T14:46:52.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.869+0000 7fa54b400640 1 -- 172.21.15.28:0/3184363071 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fa53c010ac0 con 0x7fa554108fd0 2024-04-23T14:46:52.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.869+0000 7fa553400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562622b65860 0x562622b65c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.869+0000 7fa553400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562622b65860 0x562622b65c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60504/0 (socket says 172.21.15.28:60504) 2024-04-23T14:46:52.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.869+0000 7fa553400640 1 -- 172.21.15.28:0/2748723938 learned_addr learned my addr 172.21.15.28:0/2748723938 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.870+0000 7fa552a00640 1 --2- 172.21.15.28:0/2748723938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562622b66180 0x562622b70310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.870+0000 7fa54be00640 1 --2- 172.21.15.28:0/2748723938 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562622b70850 0x562622b72c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.870+0000 7fa553400640 1 -- 172.21.15.28:0/2748723938 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562622b70850 msgr2=0x562622b72c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.870+0000 7fa553400640 1 --2- 172.21.15.28:0/2748723938 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562622b70850 0x562622b72c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.870+0000 7fa553400640 1 -- 172.21.15.28:0/2748723938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562622b66180 msgr2=0x562622b70310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.870+0000 7fa553400640 1 --2- 172.21.15.28:0/2748723938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562622b66180 0x562622b70310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.870+0000 7fa553400640 1 -- 172.21.15.28:0/2748723938 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562622ac5f50 con 0x562622b65860 2024-04-23T14:46:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.870+0000 7fa54be00640 1 --2- 172.21.15.28:0/2748723938 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562622b70850 0x562622b72c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:46:52.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:52 smithi038 ceph-mon[32348]: pgmap v88: 97 pgs: 97 active+clean; 81 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 2.3 MiB/s wr, 156 op/s 2024-04-23T14:46:52.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.870+0000 7fa553400640 1 --2- 172.21.15.28:0/2748723938 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562622b65860 0x562622b65c40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa54c009cd0 tx=0x7fa54c009a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbf14d284e3ed52e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:52.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.871+0000 7fa543e00640 1 -- 172.21.15.28:0/2748723938 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa54c030720 con 0x562622b65860 2024-04-23T14:46:52.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.871+0000 7fa543e00640 1 -- 172.21.15.28:0/2748723938 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa54c036040 con 0x562622b65860 2024-04-23T14:46:52.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.871+0000 7fa543e00640 1 -- 172.21.15.28:0/2748723938 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa54c004460 con 0x562622b65860 2024-04-23T14:46:52.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.871+0000 7fa54b400640 1 --2- 172.21.15.28:0/3184363071 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa5380a3da0 0x7fa5380a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.871+0000 7fa54b400640 1 -- 172.21.15.28:0/3184363071 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fa53c0d0be0 con 0x7fa554108fd0 2024-04-23T14:46:52.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.871+0000 7fa55c878740 1 -- 172.21.15.28:0/2748723938 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562622b65860 msgr2=0x562622b65c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:52.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.871+0000 7fa55c878740 1 --2- 172.21.15.28:0/2748723938 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562622b65860 0x562622b65c40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa54c009cd0 tx=0x7fa54c009a20 comp rx=0 tx=0).stop 2024-04-23T14:46:52.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.871+0000 7fa552a00640 1 --2- 172.21.15.28:0/3184363071 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa5380a3da0 0x7fa5380a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.872+0000 7fa55c878740 1 -- 172.21.15.28:0/2748723938 shutdown_connections 2024-04-23T14:46:52.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.872+0000 7fa55c878740 1 --2- 172.21.15.28:0/2748723938 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562622b70850 0x562622b72c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.872+0000 7fa55c878740 1 --2- 172.21.15.28:0/2748723938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562622b66180 0x562622b70310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.872+0000 7fa55c878740 1 --2- 172.21.15.28:0/2748723938 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562622b65860 0x562622b65c40 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.872+0000 7fa55c878740 1 -- 172.21.15.28:0/2748723938 >> 172.21.15.28:0/2748723938 conn(0x562622b61b80 msgr2=0x562622b61f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:52.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.872+0000 7fa55c878740 1 -- 172.21.15.28:0/2748723938 shutdown_connections 2024-04-23T14:46:52.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.872+0000 7fa55c878740 1 -- 172.21.15.28:0/2748723938 wait complete. 2024-04-23T14:46:52.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.872+0000 7fa552a00640 1 --2- 172.21.15.28:0/3184363071 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa5380a3da0 0x7fa5380a6260 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa544009fd0 tx=0x7fa544005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:52.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.872+0000 7fa55c878740 1 -- start start 2024-04-23T14:46:52.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.872+0000 7fa55c878740 1 Processor -- start 2024-04-23T14:46:52.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.873+0000 7fa55c878740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562622b65860 0x562622b6d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.873+0000 7fa55c878740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562622b66180 0x562622b6dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.873+0000 7fa55c878740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562622b70850 0x562622b6e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.873+0000 7fa55c878740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562622b735e0 con 0x562622b65860 2024-04-23T14:46:52.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.873+0000 7fa55c878740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562622a84690 con 0x562622b66180 2024-04-23T14:46:52.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.873+0000 7fa55c878740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562622b73760 con 0x562622b70850 2024-04-23T14:46:52.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.873+0000 7fa55c878740 10 client.0 _ll_register_callbacks cb 0x562622ac0650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:46:52.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.873+0000 7fa54be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562622b66180 0x562622b6dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.873+0000 7fa553400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562622b70850 0x562622b6e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.873+0000 7fa54be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562622b66180 0x562622b6dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60518/0 (socket says 172.21.15.28:60518) 2024-04-23T14:46:52.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.873+0000 7fa54be00640 1 -- 172.21.15.28:0/1520429992 learned_addr learned my addr 172.21.15.28:0/1520429992 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:52.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.873+0000 7fa552a00640 1 --2- 172.21.15.28:0/1520429992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562622b65860 0x562622b6d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.874+0000 7fa54be00640 1 -- 172.21.15.28:0/1520429992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562622b70850 msgr2=0x562622b6e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:52.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.874+0000 7fa54be00640 1 --2- 172.21.15.28:0/1520429992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562622b70850 0x562622b6e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.874+0000 7fa54be00640 1 -- 172.21.15.28:0/1520429992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562622b65860 msgr2=0x562622b6d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:52.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.874+0000 7fa54be00640 1 --2- 172.21.15.28:0/1520429992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562622b65860 0x562622b6d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.874+0000 7fa54be00640 1 -- 172.21.15.28:0/1520429992 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562622a83920 con 0x562622b66180 2024-04-23T14:46:52.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.874+0000 7fa552a00640 1 --2- 172.21.15.28:0/1520429992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562622b65860 0x562622b6d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:46:52.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.874+0000 7fa54be00640 1 --2- 172.21.15.28:0/1520429992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562622b66180 0x562622b6dd40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa53c0ce340 tx=0x7fa53c09d230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cebb478c17c2b703 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:52.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.874+0000 7fa540c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:46:52.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.874+0000 7fa540c00640 1 -- 172.21.15.28:0/1520429992 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa53c09de60 con 0x562622b66180 2024-04-23T14:46:52.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.874+0000 7fa540c00640 1 -- 172.21.15.28:0/1520429992 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fa53c09bf80 con 0x562622b66180 2024-04-23T14:46:52.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.874+0000 7fa55c878740 1 -- 172.21.15.28:0/1520429992 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa54c036040 con 0x562622b66180 2024-04-23T14:46:52.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.874+0000 7fa540c00640 1 -- 172.21.15.28:0/1520429992 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fa53c0ce760 con 0x562622b66180 2024-04-23T14:46:52.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.874+0000 7fa55c878740 10 client.25745 fetch_fsmap learned FSMap version 20 2024-04-23T14:46:52.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.874+0000 7fa55c878740 1 -- 172.21.15.28:0/1520429992 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5626228a77d0 con 0x562622b66180 2024-04-23T14:46:52.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.875+0000 7fa540c00640 1 -- 172.21.15.28:0/1520429992 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa54c036040 con 0x562622b66180 2024-04-23T14:46:52.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.875+0000 7fa540c00640 1 -- 172.21.15.28:0/1520429992 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fa53c098880 con 0x562622b66180 2024-04-23T14:46:52.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.875+0000 7fa55c878740 10 client.25745 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:46:52.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.875+0000 7fa55c878740 10 client.25745 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:46:52.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.875+0000 7fa55c878740 20 client.25745 populate_metadata read hostname 'smithi028' 2024-04-23T14:46:52.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.875+0000 7fa55c878740 1 --2- 172.21.15.28:0/1520429992 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562622b80a80 0x562622b82e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.875+0000 7fa55c878740 4 client.25745 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:46:52.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.875+0000 7fa55c878740 1 -- 172.21.15.28:0/1520429992 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562622ad8c10 con 0x562622b80a80 2024-04-23T14:46:52.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.875+0000 7fa552a00640 1 --2- 172.21.15.28:0/1520429992 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562622b80a80 0x562622b82e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.876+0000 7fa540c00640 10 client.25745 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:46:52.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.876+0000 7fa552a00640 1 --2- 172.21.15.28:0/1520429992 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562622b80a80 0x562622b82e70 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7e668fd105699f5 server_cookie=45d1f5ae1f6a5c83 in_seq=0 out_seq=0 2024-04-23T14:46:52.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.877+0000 7fa540c00640 1 -- 172.21.15.28:0/1520429992 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562622ad8c10 con 0x562622b80a80 2024-04-23T14:46:52.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.877+0000 7fa540c00640 10 client.25745 handle_command_reply: tid=0 2024-04-23T14:46:52.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.877+0000 7fa55c878740 1 client.25745 shutdown 2024-04-23T14:46:52.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.877+0000 7fa55c878740 1 -- 172.21.15.28:0/1520429992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562622b66180 msgr2=0x562622b6dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:52.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.877+0000 7fa55c878740 1 --2- 172.21.15.28:0/1520429992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562622b66180 0x562622b6dd40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa53c0ce340 tx=0x7fa53c09d230 comp rx=0 tx=0).stop 2024-04-23T14:46:52.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.877+0000 7fa55c878740 1 -- 172.21.15.28:0/1520429992 shutdown_connections 2024-04-23T14:46:52.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.877+0000 7fa55c878740 1 --2- 172.21.15.28:0/1520429992 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562622b80a80 0x562622b82e70 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.877+0000 7fa540c00640 0 client.25745 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:46:52.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.877+0000 7fa55c878740 1 --2- 172.21.15.28:0/1520429992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562622b66180 0x562622b6dd40 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.877+0000 7fa55c878740 1 --2- 172.21.15.28:0/1520429992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562622b70850 0x562622b6e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.877+0000 7fa55c878740 1 --2- 172.21.15.28:0/1520429992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562622b65860 0x562622b6d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.877+0000 7fa55c878740 1 -- 172.21.15.28:0/1520429992 >> 172.21.15.28:0/1520429992 conn(0x562622b61b80 msgr2=0x562622b67880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:52.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.877+0000 7fa55c878740 1 -- 172.21.15.28:0/1520429992 shutdown_connections 2024-04-23T14:46:52.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.877+0000 7fa55c878740 1 -- 172.21.15.28:0/1520429992 wait complete. 2024-04-23T14:46:52.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.877+0000 7fa55c878740 20 client.25745 trim_cache size 0 max 16384 2024-04-23T14:46:52.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.898+0000 7fa55c878740 1 Processor -- start 2024-04-23T14:46:52.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.898+0000 7fa55c878740 1 -- start start 2024-04-23T14:46:52.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.899+0000 7fa55c878740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562622b61b80 0x562622b7c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.899+0000 7fa55c878740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562622b65860 0x562622b68000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.899+0000 7fa55c878740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562622b66180 0x562622b88720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.899+0000 7fa55c878740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa5541b22f0 con 0x562622b65860 2024-04-23T14:46:52.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.899+0000 7fa55c878740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa5541b4b70 con 0x562622b61b80 2024-04-23T14:46:52.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.899+0000 7fa55c878740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa5541b2960 con 0x562622b66180 2024-04-23T14:46:52.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.899+0000 7fa553400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562622b66180 0x562622b88720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.899+0000 7fa552a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562622b61b80 0x562622b7c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.899+0000 7fa552a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562622b61b80 0x562622b7c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60524/0 (socket says 172.21.15.28:60524) 2024-04-23T14:46:52.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.899+0000 7fa552a00640 1 -- 172.21.15.28:0/862595398 learned_addr learned my addr 172.21.15.28:0/862595398 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:52.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.899+0000 7fa54be00640 1 --2- 172.21.15.28:0/862595398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562622b65860 0x562622b68000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.899+0000 7fa553400640 1 -- 172.21.15.28:0/862595398 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562622b61b80 msgr2=0x562622b7c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:52.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.899+0000 7fa553400640 1 --2- 172.21.15.28:0/862595398 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562622b61b80 0x562622b7c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.899+0000 7fa553400640 1 -- 172.21.15.28:0/862595398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562622b65860 msgr2=0x562622b68000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:52.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.899+0000 7fa553400640 1 --2- 172.21.15.28:0/862595398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562622b65860 0x562622b68000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.899+0000 7fa553400640 1 -- 172.21.15.28:0/862595398 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562622b65fb0 con 0x562622b66180 2024-04-23T14:46:52.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.900+0000 7fa553400640 1 --2- 172.21.15.28:0/862595398 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562622b66180 0x562622b88720 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa54c02aee0 tx=0x7fa54c004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b7ecb7a34c899e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:52.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.900+0000 7fa543400640 1 -- 172.21.15.28:0/862595398 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa54c053070 con 0x562622b66180 2024-04-23T14:46:52.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.900+0000 7fa543400640 1 -- 172.21.15.28:0/862595398 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa54c036040 con 0x562622b66180 2024-04-23T14:46:52.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.900+0000 7fa543400640 1 -- 172.21.15.28:0/862595398 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa54c004040 con 0x562622b66180 2024-04-23T14:46:52.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.901+0000 7fa55c878740 1 -- 172.21.15.28:0/862595398 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562622b66180 msgr2=0x562622b88720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:52.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.901+0000 7fa55c878740 1 --2- 172.21.15.28:0/862595398 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562622b66180 0x562622b88720 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa54c02aee0 tx=0x7fa54c004ab0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.901+0000 7fa55c878740 1 -- 172.21.15.28:0/862595398 shutdown_connections 2024-04-23T14:46:52.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.901+0000 7fa55c878740 1 --2- 172.21.15.28:0/862595398 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562622b66180 0x562622b88720 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.901+0000 7fa55c878740 1 --2- 172.21.15.28:0/862595398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562622b65860 0x562622b68000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.901+0000 7fa55c878740 1 --2- 172.21.15.28:0/862595398 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562622b61b80 0x562622b7c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.901+0000 7fa55c878740 1 -- 172.21.15.28:0/862595398 >> 172.21.15.28:0/862595398 conn(0x562622b6c440 msgr2=0x562622b6e050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:52.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.901+0000 7fa55c878740 1 -- 172.21.15.28:0/862595398 shutdown_connections 2024-04-23T14:46:52.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.901+0000 7fa55c878740 1 -- 172.21.15.28:0/862595398 wait complete. 2024-04-23T14:46:52.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.901+0000 7fa55c878740 1 -- start start 2024-04-23T14:46:52.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.902+0000 7fa55c878740 1 Processor -- start 2024-04-23T14:46:52.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.902+0000 7fa55c878740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562622b61b80 0x562622b6a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.902+0000 7fa55c878740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562622b65860 0x562622b6ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.902+0000 7fa55c878740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562622b66180 0x562622be5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.902+0000 7fa55c878740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa5541b3030 con 0x562622b61b80 2024-04-23T14:46:52.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.902+0000 7fa55c878740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa5541b44a0 con 0x562622b65860 2024-04-23T14:46:52.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.902+0000 7fa55c878740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa5541b3700 con 0x562622b66180 2024-04-23T14:46:52.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.902+0000 7fa55c878740 10 client.0 _ll_register_callbacks cb 0x7fa53c010e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:46:52.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.902+0000 7fa553400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562622b66180 0x562622be5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.902+0000 7fa54be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562622b65860 0x562622b6ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.902+0000 7fa552a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562622b61b80 0x562622b6a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.902+0000 7fa553400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562622b66180 0x562622be5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:42976/0 (socket says 172.21.15.28:42976) 2024-04-23T14:46:52.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.902+0000 7fa553400640 1 -- 172.21.15.28:0/3292252236 learned_addr learned my addr 172.21.15.28:0/3292252236 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:46:52.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.903+0000 7fa54be00640 1 -- 172.21.15.28:0/3292252236 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562622b66180 msgr2=0x562622be5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:52.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.903+0000 7fa54be00640 1 --2- 172.21.15.28:0/3292252236 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562622b66180 0x562622be5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.903+0000 7fa54be00640 1 -- 172.21.15.28:0/3292252236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562622b61b80 msgr2=0x562622b6a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:46:52.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.903+0000 7fa54be00640 1 --2- 172.21.15.28:0/3292252236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562622b61b80 0x562622b6a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.903+0000 7fa54be00640 1 -- 172.21.15.28:0/3292252236 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626227dd3f0 con 0x562622b65860 2024-04-23T14:46:52.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.903+0000 7fa553400640 1 --2- 172.21.15.28:0/3292252236 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562622b66180 0x562622be5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:46:52.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.903+0000 7fa54be00640 1 --2- 172.21.15.28:0/3292252236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562622b65860 0x562622b6ad80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa53c098f90 tx=0x7fa53c005b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33eb6d0ffadd6f2e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:46:52.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.903+0000 7fa540c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:46:52.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.903+0000 7fa540c00640 1 -- 172.21.15.28:0/3292252236 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa53c09c670 con 0x562622b65860 2024-04-23T14:46:52.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.903+0000 7fa540c00640 1 -- 172.21.15.28:0/3292252236 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fa53c09bf80 con 0x562622b65860 2024-04-23T14:46:52.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.903+0000 7fa55c878740 1 -- 172.21.15.28:0/3292252236 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa54c036040 con 0x562622b65860 2024-04-23T14:46:52.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.903+0000 7fa540c00640 1 -- 172.21.15.28:0/3292252236 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fa53c098dd0 con 0x562622b65860 2024-04-23T14:46:52.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.903+0000 7fa55c878740 10 client.25751 fetch_fsmap learned FSMap version 20 2024-04-23T14:46:52.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.903+0000 7fa55c878740 1 -- 172.21.15.28:0/3292252236 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x56262279d0f0 con 0x562622b65860 2024-04-23T14:46:52.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.904+0000 7fa540c00640 1 -- 172.21.15.28:0/3292252236 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa54c036040 con 0x562622b65860 2024-04-23T14:46:52.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.904+0000 7fa540c00640 1 -- 172.21.15.28:0/3292252236 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fa53c09b9f0 con 0x562622b65860 2024-04-23T14:46:52.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.904+0000 7fa55c878740 10 client.25751 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:46:52.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.904+0000 7fa55c878740 10 client.25751 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:46:52.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.904+0000 7fa55c878740 20 client.25751 populate_metadata read hostname 'smithi028' 2024-04-23T14:46:52.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.904+0000 7fa55c878740 1 --2- 172.21.15.28:0/3292252236 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562622b70850 0x562622be9a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:46:52.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.904+0000 7fa55c878740 4 client.25751 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:46:52.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.904+0000 7fa55c878740 1 -- 172.21.15.28:0/3292252236 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562622ad2a80 con 0x562622b70850 2024-04-23T14:46:52.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.904+0000 7fa552a00640 1 --2- 172.21.15.28:0/3292252236 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562622b70850 0x562622be9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:46:52.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.905+0000 7fa552a00640 1 --2- 172.21.15.28:0/3292252236 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562622b70850 0x562622be9a40 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d73dc1f6d23781f server_cookie=37223c5d74ca0761 in_seq=0 out_seq=0 2024-04-23T14:46:52.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.905+0000 7fa540c00640 10 client.25751 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:46:52.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.905+0000 7fa540c00640 1 -- 172.21.15.28:0/3292252236 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x562622ad2a80 con 0x562622b70850 2024-04-23T14:46:52.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.905+0000 7fa540c00640 10 client.25751 handle_command_reply: tid=0 2024-04-23T14:46:52.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.905+0000 7fa55c878740 1 client.25751 shutdown 2024-04-23T14:46:52.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.905+0000 7fa55c878740 1 -- 172.21.15.28:0/3292252236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562622b65860 msgr2=0x562622b6ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:52.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.905+0000 7fa55c878740 1 --2- 172.21.15.28:0/3292252236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562622b65860 0x562622b6ad80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa53c098f90 tx=0x7fa53c005b20 comp rx=0 tx=0).stop 2024-04-23T14:46:52.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.905+0000 7fa55c878740 1 -- 172.21.15.28:0/3292252236 shutdown_connections 2024-04-23T14:46:52.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.905+0000 7fa55c878740 1 --2- 172.21.15.28:0/3292252236 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562622b70850 0x562622be9a40 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.905+0000 7fa55c878740 1 --2- 172.21.15.28:0/3292252236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562622b65860 0x562622b6ad80 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.905+0000 7fa55c878740 1 --2- 172.21.15.28:0/3292252236 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562622b66180 0x562622be5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.905+0000 7fa55c878740 1 --2- 172.21.15.28:0/3292252236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562622b61b80 0x562622b6a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.905+0000 7fa540c00640 0 client.25751 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:46:52.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.905+0000 7fa55c878740 1 -- 172.21.15.28:0/3292252236 >> 172.21.15.28:0/3292252236 conn(0x562622b6c440 msgr2=0x562622b6e050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:52.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.906+0000 7fa55c878740 1 -- 172.21.15.28:0/3292252236 shutdown_connections 2024-04-23T14:46:52.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.906+0000 7fa55c878740 1 -- 172.21.15.28:0/3292252236 wait complete. 2024-04-23T14:46:52.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.906+0000 7fa55c878740 20 client.25751 trim_cache size 0 max 16384 2024-04-23T14:46:52.911 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:46:52.911 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-23T14:46:52.911 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-23T14:46:52.911 INFO:teuthology.orchestra.run.smithi028.stdout: "ae24cd23-44a8-4fcf-b69d-9c09dda0f02a": { 2024-04-23T14:46:52.911 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-23T14:46:52.911 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "ae24cd23-44a8-4fcf-b69d-9c09dda0f02a", 2024-04-23T14:46:52.912 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-23T14:46:52.912 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:46:52.912 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:46:52.912 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:46:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.906+0000 7fa559a00640 1 -- 172.21.15.28:0/3184363071 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa5380a3da0 msgr2=0x7fa5380a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.906+0000 7fa559a00640 1 --2- 172.21.15.28:0/3184363071 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa5380a3da0 0x7fa5380a6260 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa544009fd0 tx=0x7fa544005f90 comp rx=0 tx=0).stop 2024-04-23T14:46:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.906+0000 7fa559a00640 1 -- 172.21.15.28:0/3184363071 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa554108fd0 msgr2=0x7fa55419f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:46:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.906+0000 7fa559a00640 1 --2- 172.21.15.28:0/3184363071 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa554108fd0 0x7fa55419f350 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa53c00e9f0 tx=0x7fa53c00eeb0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.906+0000 7fa559a00640 1 -- 172.21.15.28:0/3184363071 shutdown_connections 2024-04-23T14:46:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.906+0000 7fa559a00640 1 --2- 172.21.15.28:0/3184363071 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa5380a3da0 0x7fa5380a6260 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.906+0000 7fa559a00640 1 --2- 172.21.15.28:0/3184363071 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa554108fd0 0x7fa55419f350 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.906+0000 7fa559a00640 1 --2- 172.21.15.28:0/3184363071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa55410e660 0x7fa5541a36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.906+0000 7fa559a00640 1 --2- 172.21.15.28:0/3184363071 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa554108610 0x7fa55419ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:46:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.906+0000 7fa559a00640 1 -- 172.21.15.28:0/3184363071 >> 172.21.15.28:0/3184363071 conn(0x7fa554076540 msgr2=0x7fa554077b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:46:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.907+0000 7fa559a00640 1 -- 172.21.15.28:0/3184363071 shutdown_connections 2024-04-23T14:46:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:46:52.907+0000 7fa559a00640 1 -- 172.21.15.28:0/3184363071 wait complete. 2024-04-23T14:46:52.927 INFO:tasks.cephfs.filesystem:scrub status for tag:ae24cd23-44a8-4fcf-b69d-9c09dda0f02a - {'path': '/', 'tag': 'ae24cd23-44a8-4fcf-b69d-9c09dda0f02a', 'options': 'recursive,force'} 2024-04-23T14:46:53.468 INFO:tasks.workunit.client.0.smithi028.stdout: 1 137428 10.95 MB/sec warmup 69 sec latency 12.859 ms 2024-04-23T14:46:53.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:53 smithi105 ceph-mon[32555]: scrub queued for path: / 2024-04-23T14:46:53.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:53 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:46:53.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:53 smithi105 ceph-mon[32555]: scrub summary: active paths [/] 2024-04-23T14:46:53.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:53 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:46:53.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:53 smithi038 ceph-mon[32348]: scrub queued for path: / 2024-04-23T14:46:53.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:53 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:46:53.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:53 smithi038 ceph-mon[32348]: scrub summary: active paths [/] 2024-04-23T14:46:53.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:53 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:46:53.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:53 smithi028 ceph-mon[27810]: scrub queued for path: / 2024-04-23T14:46:53.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:53 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:46:53.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:53 smithi028 ceph-mon[27810]: scrub summary: active paths [/] 2024-04-23T14:46:53.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:53 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:46:54.468 INFO:tasks.workunit.client.0.smithi028.stdout: 1 138950 10.95 MB/sec warmup 70 sec latency 11.462 ms 2024-04-23T14:46:54.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:54 smithi038 ceph-mon[32348]: pgmap v89: 97 pgs: 97 active+clean; 85 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.5 MiB/s wr, 240 op/s 2024-04-23T14:46:54.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:54 smithi028 ceph-mon[27810]: pgmap v89: 97 pgs: 97 active+clean; 85 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.5 MiB/s wr, 240 op/s 2024-04-23T14:46:55.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:54 smithi105 ceph-mon[32555]: pgmap v89: 97 pgs: 97 active+clean; 85 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.5 MiB/s wr, 240 op/s 2024-04-23T14:46:55.468 INFO:tasks.workunit.client.0.smithi028.stdout: 1 140925 10.91 MB/sec warmup 71 sec latency 43.605 ms 2024-04-23T14:46:55.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:55 smithi038 ceph-mon[32348]: scrub summary: idle 2024-04-23T14:46:55.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:55 smithi028 ceph-mon[27810]: scrub summary: idle 2024-04-23T14:46:56.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:55 smithi105 ceph-mon[32555]: scrub summary: idle 2024-04-23T14:46:56.468 INFO:tasks.workunit.client.0.smithi028.stdout: 1 142499 10.91 MB/sec warmup 72 sec latency 37.410 ms 2024-04-23T14:46:56.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:56 smithi038 ceph-mon[32348]: pgmap v90: 97 pgs: 97 active+clean; 85 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 2.5 MiB/s wr, 163 op/s 2024-04-23T14:46:56.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:56 smithi028 ceph-mon[27810]: pgmap v90: 97 pgs: 97 active+clean; 85 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 2.5 MiB/s wr, 163 op/s 2024-04-23T14:46:57.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:56 smithi105 ceph-mon[32555]: pgmap v90: 97 pgs: 97 active+clean; 85 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 2.5 MiB/s wr, 163 op/s 2024-04-23T14:46:57.468 INFO:tasks.workunit.client.0.smithi028.stdout: 1 144833 10.94 MB/sec warmup 73 sec latency 37.652 ms 2024-04-23T14:46:58.469 INFO:tasks.workunit.client.0.smithi028.stdout: 1 146647 10.91 MB/sec warmup 74 sec latency 12.616 ms 2024-04-23T14:46:58.837 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:46:58 smithi105 ceph-mon[32555]: pgmap v91: 97 pgs: 97 active+clean; 89 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 1005 KiB/s rd, 3.7 MiB/s wr, 244 op/s 2024-04-23T14:46:58.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:46:58 smithi038 ceph-mon[32348]: pgmap v91: 97 pgs: 97 active+clean; 89 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 1005 KiB/s rd, 3.7 MiB/s wr, 244 op/s 2024-04-23T14:46:58.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:46:58 smithi028 ceph-mon[27810]: pgmap v91: 97 pgs: 97 active+clean; 89 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 1005 KiB/s rd, 3.7 MiB/s wr, 244 op/s 2024-04-23T14:46:59.468 INFO:tasks.workunit.client.0.smithi028.stdout: 1 149293 10.98 MB/sec warmup 75 sec latency 8.506 ms 2024-04-23T14:47:00.469 INFO:tasks.workunit.client.0.smithi028.stdout: 1 151265 10.92 MB/sec warmup 76 sec latency 44.985 ms 2024-04-23T14:47:00.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:00 smithi038 ceph-mon[32348]: pgmap v92: 97 pgs: 97 active+clean; 89 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 2.4 MiB/s wr, 164 op/s 2024-04-23T14:47:00.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:00 smithi028 ceph-mon[27810]: pgmap v92: 97 pgs: 97 active+clean; 89 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 2.4 MiB/s wr, 164 op/s 2024-04-23T14:47:01.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:00 smithi105 ceph-mon[32555]: pgmap v92: 97 pgs: 97 active+clean; 89 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 2.4 MiB/s wr, 164 op/s 2024-04-23T14:47:01.469 INFO:tasks.workunit.client.0.smithi028.stdout: 1 152512 10.93 MB/sec warmup 77 sec latency 50.322 ms 2024-04-23T14:47:02.469 INFO:tasks.workunit.client.0.smithi028.stdout: 1 153586 10.83 MB/sec warmup 78 sec latency 40.461 ms 2024-04-23T14:47:02.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:02 smithi038 ceph-mon[32348]: pgmap v93: 97 pgs: 97 active+clean; 89 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 2.4 MiB/s wr, 164 op/s 2024-04-23T14:47:02.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:02 smithi028 ceph-mon[27810]: pgmap v93: 97 pgs: 97 active+clean; 89 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 2.4 MiB/s wr, 164 op/s 2024-04-23T14:47:03.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:02 smithi105 ceph-mon[32555]: pgmap v93: 97 pgs: 97 active+clean; 89 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 2.4 MiB/s wr, 164 op/s 2024-04-23T14:47:03.469 INFO:tasks.workunit.client.0.smithi028.stdout: 1 155919 10.86 MB/sec warmup 79 sec latency 10.637 ms 2024-04-23T14:47:04.469 INFO:tasks.workunit.client.0.smithi028.stdout: 1 157280 10.80 MB/sec warmup 80 sec latency 17.535 ms 2024-04-23T14:47:04.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:04 smithi038 ceph-mon[32348]: pgmap v94: 97 pgs: 97 active+clean; 96 MiB data, 796 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.8 MiB/s wr, 277 op/s 2024-04-23T14:47:04.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:04 smithi028 ceph-mon[27810]: pgmap v94: 97 pgs: 97 active+clean; 96 MiB data, 796 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.8 MiB/s wr, 277 op/s 2024-04-23T14:47:05.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:04 smithi105 ceph-mon[32555]: pgmap v94: 97 pgs: 97 active+clean; 96 MiB data, 796 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.8 MiB/s wr, 277 op/s 2024-04-23T14:47:05.469 INFO:tasks.workunit.client.0.smithi028.stdout: 1 159650 10.86 MB/sec warmup 81 sec latency 10.651 ms 2024-04-23T14:47:06.469 INFO:tasks.workunit.client.0.smithi028.stdout: 1 160901 10.77 MB/sec warmup 82 sec latency 43.940 ms 2024-04-23T14:47:06.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:06 smithi038 ceph-mon[32348]: pgmap v95: 97 pgs: 97 active+clean; 96 MiB data, 796 MiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 2.6 MiB/s wr, 193 op/s 2024-04-23T14:47:06.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:06 smithi028 ceph-mon[27810]: pgmap v95: 97 pgs: 97 active+clean; 96 MiB data, 796 MiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 2.6 MiB/s wr, 193 op/s 2024-04-23T14:47:07.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:06 smithi105 ceph-mon[32555]: pgmap v95: 97 pgs: 97 active+clean; 96 MiB data, 796 MiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 2.6 MiB/s wr, 193 op/s 2024-04-23T14:47:07.470 INFO:tasks.workunit.client.0.smithi028.stdout: 1 163149 10.83 MB/sec warmup 83 sec latency 93.519 ms 2024-04-23T14:47:08.470 INFO:tasks.workunit.client.0.smithi028.stdout: 1 164362 10.75 MB/sec warmup 84 sec latency 16.112 ms 2024-04-23T14:47:08.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:08 smithi038 ceph-mon[32348]: pgmap v96: 97 pgs: 97 active+clean; 103 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 271 op/s 2024-04-23T14:47:08.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:08 smithi028 ceph-mon[27810]: pgmap v96: 97 pgs: 97 active+clean; 103 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 271 op/s 2024-04-23T14:47:09.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:08 smithi105 ceph-mon[32555]: pgmap v96: 97 pgs: 97 active+clean; 103 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 271 op/s 2024-04-23T14:47:09.470 INFO:tasks.workunit.client.0.smithi028.stdout: 1 166740 10.80 MB/sec warmup 85 sec latency 13.705 ms 2024-04-23T14:47:10.470 INFO:tasks.workunit.client.0.smithi028.stdout: 1 167954 10.72 MB/sec warmup 86 sec latency 19.047 ms 2024-04-23T14:47:10.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:10 smithi038 ceph-mon[32348]: pgmap v97: 97 pgs: 97 active+clean; 103 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 2.6 MiB/s wr, 190 op/s 2024-04-23T14:47:10.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:10 smithi028 ceph-mon[27810]: pgmap v97: 97 pgs: 97 active+clean; 103 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 2.6 MiB/s wr, 190 op/s 2024-04-23T14:47:11.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:10 smithi105 ceph-mon[32555]: pgmap v97: 97 pgs: 97 active+clean; 103 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 2.6 MiB/s wr, 190 op/s 2024-04-23T14:47:11.470 INFO:tasks.workunit.client.0.smithi028.stdout: 1 170392 10.77 MB/sec warmup 87 sec latency 10.773 ms 2024-04-23T14:47:12.470 INFO:tasks.workunit.client.0.smithi028.stdout: 1 171345 10.69 MB/sec warmup 88 sec latency 36.577 ms 2024-04-23T14:47:12.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:12 smithi038 ceph-mon[32348]: pgmap v98: 97 pgs: 97 active+clean; 103 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 2.6 MiB/s wr, 190 op/s 2024-04-23T14:47:12.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:12 smithi028 ceph-mon[27810]: pgmap v98: 97 pgs: 97 active+clean; 103 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 2.6 MiB/s wr, 190 op/s 2024-04-23T14:47:13.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:12 smithi105 ceph-mon[32555]: pgmap v98: 97 pgs: 97 active+clean; 103 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 2.6 MiB/s wr, 190 op/s 2024-04-23T14:47:13.470 INFO:tasks.workunit.client.0.smithi028.stdout: 1 173817 10.74 MB/sec warmup 89 sec latency 10.695 ms 2024-04-23T14:47:14.470 INFO:tasks.workunit.client.0.smithi028.stdout: 1 175232 10.70 MB/sec warmup 90 sec latency 15.471 ms 2024-04-23T14:47:14.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:14 smithi038 ceph-mon[32348]: pgmap v99: 97 pgs: 97 active+clean; 108 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 263 op/s 2024-04-23T14:47:14.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:14 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:47:14.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:14 smithi028 ceph-mon[27810]: pgmap v99: 97 pgs: 97 active+clean; 108 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 263 op/s 2024-04-23T14:47:14.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:14 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:47:15.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:14 smithi105 ceph-mon[32555]: pgmap v99: 97 pgs: 97 active+clean; 108 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 263 op/s 2024-04-23T14:47:15.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:14 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:47:15.470 INFO:tasks.workunit.client.0.smithi028.stdout: 1 177692 10.73 MB/sec warmup 91 sec latency 15.035 ms 2024-04-23T14:47:16.471 INFO:tasks.workunit.client.0.smithi028.stdout: 1 179863 10.70 MB/sec warmup 92 sec latency 12.968 ms 2024-04-23T14:47:16.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:16 smithi038 ceph-mon[32348]: pgmap v100: 97 pgs: 97 active+clean; 108 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 951 KiB/s rd, 2.3 MiB/s wr, 150 op/s 2024-04-23T14:47:16.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:16 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:47:16.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:16 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:47:16.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:16 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:47:16.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:16 smithi028 ceph-mon[27810]: pgmap v100: 97 pgs: 97 active+clean; 108 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 951 KiB/s rd, 2.3 MiB/s wr, 150 op/s 2024-04-23T14:47:16.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:16 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:47:16.877 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:16 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:47:16.877 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:16 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:47:17.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:16 smithi105 ceph-mon[32555]: pgmap v100: 97 pgs: 97 active+clean; 108 MiB data, 844 MiB used, 1.0 TiB / 1.0 TiB avail; 951 KiB/s rd, 2.3 MiB/s wr, 150 op/s 2024-04-23T14:47:17.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:16 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:47:17.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:16 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:47:17.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:16 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:47:17.471 INFO:tasks.workunit.client.0.smithi028.stdout: 1 181375 10.71 MB/sec warmup 93 sec latency 92.762 ms 2024-04-23T14:47:18.473 INFO:tasks.workunit.client.0.smithi028.stdout: 1 183807 10.72 MB/sec warmup 94 sec latency 13.495 ms 2024-04-23T14:47:18.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:18 smithi038 ceph-mon[32348]: pgmap v101: 97 pgs: 97 active+clean; 117 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 230 op/s 2024-04-23T14:47:18.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:18 smithi028 ceph-mon[27810]: pgmap v101: 97 pgs: 97 active+clean; 117 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 230 op/s 2024-04-23T14:47:19.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:18 smithi105 ceph-mon[32555]: pgmap v101: 97 pgs: 97 active+clean; 117 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 230 op/s 2024-04-23T14:47:19.471 INFO:tasks.workunit.client.0.smithi028.stdout: 1 185141 10.69 MB/sec warmup 95 sec latency 17.368 ms 2024-04-23T14:47:20.471 INFO:tasks.workunit.client.0.smithi028.stdout: 1 187014 10.66 MB/sec warmup 96 sec latency 20.850 ms 2024-04-23T14:47:20.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:20 smithi038 ceph-mon[32348]: pgmap v102: 97 pgs: 97 active+clean; 117 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 2.5 MiB/s wr, 152 op/s 2024-04-23T14:47:20.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:20 smithi028 ceph-mon[27810]: pgmap v102: 97 pgs: 97 active+clean; 117 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 2.5 MiB/s wr, 152 op/s 2024-04-23T14:47:21.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:20 smithi105 ceph-mon[32555]: pgmap v102: 97 pgs: 97 active+clean; 117 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 2.5 MiB/s wr, 152 op/s 2024-04-23T14:47:21.471 INFO:tasks.workunit.client.0.smithi028.stdout: 1 188447 10.66 MB/sec warmup 97 sec latency 28.657 ms 2024-04-23T14:47:22.471 INFO:tasks.workunit.client.0.smithi028.stdout: 1 189655 10.61 MB/sec warmup 98 sec latency 61.977 ms 2024-04-23T14:47:22.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:22 smithi038 ceph-mon[32348]: pgmap v103: 97 pgs: 97 active+clean; 117 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 2.5 MiB/s wr, 152 op/s 2024-04-23T14:47:22.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:22 smithi028 ceph-mon[27810]: pgmap v103: 97 pgs: 97 active+clean; 117 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 2.5 MiB/s wr, 152 op/s 2024-04-23T14:47:22.928 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:47:22.929 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:47:23.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:22 smithi105 ceph-mon[32555]: pgmap v103: 97 pgs: 97 active+clean; 117 MiB data, 871 MiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 2.5 MiB/s wr, 152 op/s 2024-04-23T14:47:23.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.168+0000 7f89dbe00640 1 -- 172.21.15.28:0/2282127039 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89dc10c480 msgr2=0x7f89dc10e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:23.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.168+0000 7f89dbe00640 1 --2- 172.21.15.28:0/2282127039 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89dc10c480 0x7f89dc10e920 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f89d4009900 tx=0x7f89d402a480 comp rx=0 tx=0).stop 2024-04-23T14:47:23.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.168+0000 7f89dbe00640 1 -- 172.21.15.28:0/2282127039 shutdown_connections 2024-04-23T14:47:23.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.168+0000 7f89dbe00640 1 --2- 172.21.15.28:0/2282127039 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89dc10c480 0x7f89dc10e920 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.168+0000 7f89dbe00640 1 --2- 172.21.15.28:0/2282127039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89dc103980 0x7f89dc10be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.168+0000 7f89dbe00640 1 --2- 172.21.15.28:0/2282127039 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89dc103060 0x7f89dc103440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.168+0000 7f89dbe00640 1 -- 172.21.15.28:0/2282127039 >> 172.21.15.28:0/2282127039 conn(0x7f89dc0fc880 msgr2=0x7f89dc0feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:23.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.169+0000 7f89dbe00640 1 -- 172.21.15.28:0/2282127039 shutdown_connections 2024-04-23T14:47:23.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.169+0000 7f89dbe00640 1 -- 172.21.15.28:0/2282127039 wait complete. 2024-04-23T14:47:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.170+0000 7f89dbe00640 1 Processor -- start 2024-04-23T14:47:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.170+0000 7f89dbe00640 1 -- start start 2024-04-23T14:47:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.170+0000 7f89dbe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89dc103060 0x7f89dc1090c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.170+0000 7f89dbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89dc103980 0x7f89dc109600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.170+0000 7f89dbe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89dc10c480 0x7f89dc109b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.170+0000 7f89dbe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc102900 con 0x7f89dc103980 2024-04-23T14:47:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.170+0000 7f89dbe00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc102780 con 0x7f89dc103060 2024-04-23T14:47:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.170+0000 7f89dbe00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc102a80 con 0x7f89dc10c480 2024-04-23T14:47:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.170+0000 7f89daa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89dc103060 0x7f89dc1090c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.170+0000 7f89daa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89dc103060 0x7f89dc1090c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:43300/0 (socket says 172.21.15.28:43300) 2024-04-23T14:47:23.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.170+0000 7f89daa00640 1 -- 172.21.15.28:0/653597188 learned_addr learned my addr 172.21.15.28:0/653597188 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:23.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.171+0000 7f89db400640 1 --2- 172.21.15.28:0/653597188 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89dc10c480 0x7f89dc109b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.171+0000 7f89da000640 1 --2- 172.21.15.28:0/653597188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89dc103980 0x7f89dc109600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.171+0000 7f89daa00640 1 -- 172.21.15.28:0/653597188 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89dc10c480 msgr2=0x7f89dc109b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:23.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.171+0000 7f89daa00640 1 --2- 172.21.15.28:0/653597188 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89dc10c480 0x7f89dc109b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.171+0000 7f89daa00640 1 -- 172.21.15.28:0/653597188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89dc103980 msgr2=0x7f89dc109600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:23.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.171+0000 7f89daa00640 1 --2- 172.21.15.28:0/653597188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89dc103980 0x7f89dc109600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.171+0000 7f89daa00640 1 -- 172.21.15.28:0/653597188 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89dc1aafd0 con 0x7f89dc103060 2024-04-23T14:47:23.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.171+0000 7f89daa00640 1 --2- 172.21.15.28:0/653597188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89dc103060 0x7f89dc1090c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f89cc00eab0 tx=0x7f89cc00ef70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:23.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.172+0000 7f89d3400640 1 -- 172.21.15.28:0/653597188 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89cc00ce40 con 0x7f89dc103060 2024-04-23T14:47:23.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.172+0000 7f89d3400640 1 -- 172.21.15.28:0/653597188 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89cc005020 con 0x7f89dc103060 2024-04-23T14:47:23.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.172+0000 7f89d3400640 1 -- 172.21.15.28:0/653597188 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89cc011760 con 0x7f89dc103060 2024-04-23T14:47:23.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.172+0000 7f89dbe00640 1 -- 172.21.15.28:0/653597188 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89dc1ab2c0 con 0x7f89dc103060 2024-04-23T14:47:23.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.172+0000 7f89dbe00640 1 -- 172.21.15.28:0/653597188 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89dc1ab7c0 con 0x7f89dc103060 2024-04-23T14:47:23.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.176+0000 7f89e4793740 1 Processor -- start 2024-04-23T14:47:23.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.176+0000 7f89d3400640 1 -- 172.21.15.28:0/653597188 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f89cc0051d0 con 0x7f89dc103060 2024-04-23T14:47:23.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.176+0000 7f89e4793740 1 -- start start 2024-04-23T14:47:23.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.177+0000 7f89e4793740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56489cbf1a30 0x56489cbf1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.177+0000 7f89e4793740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56489cbf2350 0x56489cbfc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.177+0000 7f89e4793740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56489cbfca20 0x56489cbfee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.177+0000 7f89e4793740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56489cb571f0 con 0x56489cbf2350 2024-04-23T14:47:23.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.177+0000 7f89e4793740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56489cafd350 con 0x56489cbfca20 2024-04-23T14:47:23.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.177+0000 7f89e4793740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56489cb531b0 con 0x56489cbf1a30 2024-04-23T14:47:23.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.177+0000 7f89db400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56489cbf1a30 0x56489cbf1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.177+0000 7f89da000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56489cbfca20 0x56489cbfee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.177+0000 7f89db400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56489cbf1a30 0x56489cbf1e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:58208/0 (socket says 172.21.15.28:58208) 2024-04-23T14:47:23.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.177+0000 7f89db400640 1 -- 172.21.15.28:0/2333618690 learned_addr learned my addr 172.21.15.28:0/2333618690 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:23.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.177+0000 7f89daa00640 1 --2- 172.21.15.28:0/2333618690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56489cbf2350 0x56489cbfc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.178+0000 7f89db400640 1 -- 172.21.15.28:0/2333618690 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56489cbfca20 msgr2=0x56489cbfee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:23.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.178+0000 7f89db400640 1 --2- 172.21.15.28:0/2333618690 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56489cbfca20 0x56489cbfee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.178+0000 7f89db400640 1 -- 172.21.15.28:0/2333618690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56489cbf2350 msgr2=0x56489cbfc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:23.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.178+0000 7f89db400640 1 --2- 172.21.15.28:0/2333618690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56489cbf2350 0x56489cbfc4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.178+0000 7f89db400640 1 -- 172.21.15.28:0/2333618690 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56489cb51f50 con 0x56489cbf1a30 2024-04-23T14:47:23.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.178+0000 7f89daa00640 1 --2- 172.21.15.28:0/2333618690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56489cbf2350 0x56489cbfc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:47:23.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.178+0000 7f89da000640 1 --2- 172.21.15.28:0/2333618690 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56489cbfca20 0x56489cbfee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:47:23.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.178+0000 7f89db400640 1 --2- 172.21.15.28:0/2333618690 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56489cbf1a30 0x56489cbf1e10 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f89d402aa70 tx=0x7f89d4005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12c447aadd698d55 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:23.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.178+0000 7f89d3400640 1 --2- 172.21.15.28:0/653597188 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f89c00a3da0 0x7f89c00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.179+0000 7f89cbe00640 1 -- 172.21.15.28:0/2333618690 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d4040070 con 0x56489cbf1a30 2024-04-23T14:47:23.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.179+0000 7f89cbe00640 1 -- 172.21.15.28:0/2333618690 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89d4004400 con 0x56489cbf1a30 2024-04-23T14:47:23.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.179+0000 7f89cbe00640 1 -- 172.21.15.28:0/2333618690 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d4035040 con 0x56489cbf1a30 2024-04-23T14:47:23.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.179+0000 7f89daa00640 1 --2- 172.21.15.28:0/653597188 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f89c00a3da0 0x7f89c00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.179+0000 7f89d3400640 1 -- 172.21.15.28:0/653597188 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f89cc0d0ad0 con 0x7f89dc103060 2024-04-23T14:47:23.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.179+0000 7f89e4793740 1 -- 172.21.15.28:0/2333618690 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56489cbf1a30 msgr2=0x56489cbf1e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.179+0000 7f89e4793740 1 --2- 172.21.15.28:0/2333618690 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56489cbf1a30 0x56489cbf1e10 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f89d402aa70 tx=0x7f89d4005c00 comp rx=0 tx=0).stop 2024-04-23T14:47:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.180+0000 7f89daa00640 1 --2- 172.21.15.28:0/653597188 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f89c00a3da0 0x7f89c00a6260 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x56489cbf2ff0 tx=0x7f89cc00c8b0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.180+0000 7f89e4793740 1 -- 172.21.15.28:0/2333618690 shutdown_connections 2024-04-23T14:47:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.180+0000 7f89e4793740 1 --2- 172.21.15.28:0/2333618690 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56489cbfca20 0x56489cbfee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.180+0000 7f89e4793740 1 --2- 172.21.15.28:0/2333618690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56489cbf2350 0x56489cbfc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.180+0000 7f89e4793740 1 --2- 172.21.15.28:0/2333618690 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56489cbf1a30 0x56489cbf1e10 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.180+0000 7f89e4793740 1 -- 172.21.15.28:0/2333618690 >> 172.21.15.28:0/2333618690 conn(0x56489cbedd50 msgr2=0x56489cbee130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.180+0000 7f89e4793740 1 -- 172.21.15.28:0/2333618690 shutdown_connections 2024-04-23T14:47:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.180+0000 7f89e4793740 1 -- 172.21.15.28:0/2333618690 wait complete. 2024-04-23T14:47:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.180+0000 7f89e4793740 1 -- start start 2024-04-23T14:47:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.180+0000 7f89e4793740 1 Processor -- start 2024-04-23T14:47:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.181+0000 7f89e4793740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56489cbf1a30 0x56489cbf9a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.181+0000 7f89e4793740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56489cbf2350 0x56489cbf9f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.181+0000 7f89e4793740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56489cbfca20 0x56489cbfa4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.181+0000 7f89e4793740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56489cbff7b0 con 0x56489cbf1a30 2024-04-23T14:47:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.181+0000 7f89e4793740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56489cb10690 con 0x56489cbf2350 2024-04-23T14:47:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.181+0000 7f89e4793740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56489cbff930 con 0x56489cbfca20 2024-04-23T14:47:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.181+0000 7f89e4793740 10 client.0 _ll_register_callbacks cb 0x56489cb4c650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:47:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.181+0000 7f89da000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56489cbf2350 0x56489cbf9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.181+0000 7f89da000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56489cbf2350 0x56489cbf9f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:43314/0 (socket says 172.21.15.28:43314) 2024-04-23T14:47:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.181+0000 7f89da000640 1 -- 172.21.15.28:0/1201145285 learned_addr learned my addr 172.21.15.28:0/1201145285 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.181+0000 7f89db400640 1 --2- 172.21.15.28:0/1201145285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56489cbf1a30 0x56489cbf9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.182+0000 7f89da000640 1 -- 172.21.15.28:0/1201145285 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56489cbfca20 msgr2=0x56489cbfa4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.182+0000 7f89da000640 1 --2- 172.21.15.28:0/1201145285 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56489cbfca20 0x56489cbfa4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.182+0000 7f89da000640 1 -- 172.21.15.28:0/1201145285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56489cbf1a30 msgr2=0x56489cbf9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.182+0000 7f89da000640 1 --2- 172.21.15.28:0/1201145285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56489cbf1a30 0x56489cbf9a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.182+0000 7f89da000640 1 -- 172.21.15.28:0/1201145285 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56489cb0f920 con 0x56489cbf2350 2024-04-23T14:47:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.182+0000 7f89db400640 1 --2- 172.21.15.28:0/1201145285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56489cbf1a30 0x56489cbf9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:47:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.182+0000 7f89da000640 1 --2- 172.21.15.28:0/1201145285 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56489cbf2350 0x56489cbf9f80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f89c4012700 tx=0x7f89c4012bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef73d03d7b0d7208 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.182+0000 7f89c8c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:47:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.182+0000 7f89c8c00640 1 -- 172.21.15.28:0/1201145285 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c4011bb0 con 0x56489cbf2350 2024-04-23T14:47:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.182+0000 7f89e4793740 1 -- 172.21.15.28:0/1201145285 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89d4004400 con 0x56489cbf2350 2024-04-23T14:47:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.182+0000 7f89c8c00640 1 -- 172.21.15.28:0/1201145285 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f89c4011d50 con 0x56489cbf2350 2024-04-23T14:47:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.182+0000 7f89c8c00640 1 -- 172.21.15.28:0/1201145285 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f89c4004030 con 0x56489cbf2350 2024-04-23T14:47:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.182+0000 7f89e4793740 10 client.25763 fetch_fsmap learned FSMap version 20 2024-04-23T14:47:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.182+0000 7f89e4793740 1 -- 172.21.15.28:0/1201145285 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x56489c9337d0 con 0x56489cbf2350 2024-04-23T14:47:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.183+0000 7f89c8c00640 1 -- 172.21.15.28:0/1201145285 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89d4004400 con 0x56489cbf2350 2024-04-23T14:47:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.183+0000 7f89c8c00640 1 -- 172.21.15.28:0/1201145285 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f89c4025470 con 0x56489cbf2350 2024-04-23T14:47:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.183+0000 7f89e4793740 10 client.25763 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:47:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.183+0000 7f89e4793740 10 client.25763 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:47:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.183+0000 7f89e4793740 20 client.25763 populate_metadata read hostname 'smithi028' 2024-04-23T14:47:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.183+0000 7f89e4793740 1 --2- 172.21.15.28:0/1201145285 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56489cc0cdd0 0x56489cc0f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.183+0000 7f89e4793740 4 client.25763 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:47:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.183+0000 7f89e4793740 1 -- 172.21.15.28:0/1201145285 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56489cb64c10 con 0x56489cc0cdd0 2024-04-23T14:47:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.183+0000 7f89db400640 1 --2- 172.21.15.28:0/1201145285 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56489cc0cdd0 0x56489cc0f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.184+0000 7f89db400640 1 --2- 172.21.15.28:0/1201145285 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56489cc0cdd0 0x56489cc0f230 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=224e9e15259f178f server_cookie=7dcaf4c3b7ecd5ef in_seq=0 out_seq=0 2024-04-23T14:47:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.184+0000 7f89c8c00640 10 client.25763 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:47:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.185+0000 7f89c8c00640 1 -- 172.21.15.28:0/1201145285 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56489cb64c10 con 0x56489cc0cdd0 2024-04-23T14:47:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.185+0000 7f89c8c00640 10 client.25763 handle_command_reply: tid=0 2024-04-23T14:47:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.185+0000 7f89e4793740 1 client.25763 shutdown 2024-04-23T14:47:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.185+0000 7f89e4793740 1 -- 172.21.15.28:0/1201145285 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56489cbf2350 msgr2=0x56489cbf9f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.185+0000 7f89e4793740 1 --2- 172.21.15.28:0/1201145285 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56489cbf2350 0x56489cbf9f80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f89c4012700 tx=0x7f89c4012bc0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.185+0000 7f89e4793740 1 -- 172.21.15.28:0/1201145285 shutdown_connections 2024-04-23T14:47:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.185+0000 7f89e4793740 1 --2- 172.21.15.28:0/1201145285 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56489cc0cdd0 0x56489cc0f230 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.185+0000 7f89e4793740 1 --2- 172.21.15.28:0/1201145285 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56489cbf2350 0x56489cbf9f80 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.185+0000 7f89e4793740 1 --2- 172.21.15.28:0/1201145285 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56489cbfca20 0x56489cbfa4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.185+0000 7f89c8c00640 0 client.25763 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:47:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.185+0000 7f89e4793740 1 --2- 172.21.15.28:0/1201145285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56489cbf1a30 0x56489cbf9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.185+0000 7f89e4793740 1 -- 172.21.15.28:0/1201145285 >> 172.21.15.28:0/1201145285 conn(0x56489cbedd50 msgr2=0x56489cbf3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:23.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.185+0000 7f89e4793740 1 -- 172.21.15.28:0/1201145285 shutdown_connections 2024-04-23T14:47:23.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.186+0000 7f89e4793740 1 -- 172.21.15.28:0/1201145285 wait complete. 2024-04-23T14:47:23.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.186+0000 7f89e4793740 20 client.25763 trim_cache size 0 max 16384 2024-04-23T14:47:23.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.208+0000 7f89e4793740 1 Processor -- start 2024-04-23T14:47:23.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.208+0000 7f89e4793740 1 -- start start 2024-04-23T14:47:23.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.208+0000 7f89e4793740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56489cbf1a30 0x56489cc08670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.208+0000 7f89e4793740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56489cbf2350 0x56489cbee130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.208+0000 7f89e4793740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56489cbfca20 0x56489cbece50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.208+0000 7f89e4793740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc1b20e0 con 0x56489cbf1a30 2024-04-23T14:47:23.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.208+0000 7f89e4793740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc1b4960 con 0x56489cbf2350 2024-04-23T14:47:23.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.208+0000 7f89e4793740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc1b2750 con 0x56489cbfca20 2024-04-23T14:47:23.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.208+0000 7f89da000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56489cbf2350 0x56489cbee130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.208+0000 7f89da000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56489cbf2350 0x56489cbee130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:43322/0 (socket says 172.21.15.28:43322) 2024-04-23T14:47:23.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.208+0000 7f89da000640 1 -- 172.21.15.28:0/3214773364 learned_addr learned my addr 172.21.15.28:0/3214773364 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:23.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.208+0000 7f89db400640 1 --2- 172.21.15.28:0/3214773364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56489cbf1a30 0x56489cc08670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.209+0000 7f89db400640 1 --2- 172.21.15.28:0/3214773364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56489cbfca20 0x56489cbece50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.209+0000 7f89da000640 1 -- 172.21.15.28:0/3214773364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56489cbfca20 msgr2=0x56489cbece50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:23.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.209+0000 7f89da000640 1 --2- 172.21.15.28:0/3214773364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56489cbfca20 0x56489cbece50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.209+0000 7f89da000640 1 -- 172.21.15.28:0/3214773364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56489cbf1a30 msgr2=0x56489cc08670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:23.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.209+0000 7f89da000640 1 --2- 172.21.15.28:0/3214773364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56489cbf1a30 0x56489cc08670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.209+0000 7f89da000640 1 -- 172.21.15.28:0/3214773364 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56489cbf2180 con 0x56489cbf2350 2024-04-23T14:47:23.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.209+0000 7f89db400640 1 --2- 172.21.15.28:0/3214773364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56489cbfca20 0x56489cbece50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:47:23.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.209+0000 7f89da000640 1 --2- 172.21.15.28:0/3214773364 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56489cbf2350 0x56489cbee130 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f89c4025810 tx=0x7f89c4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bd3ea9b7c4fc1a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:23.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.209+0000 7f89cb400640 1 -- 172.21.15.28:0/3214773364 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c4025470 con 0x56489cbf2350 2024-04-23T14:47:23.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.209+0000 7f89cb400640 1 -- 172.21.15.28:0/3214773364 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89c4025610 con 0x56489cbf2350 2024-04-23T14:47:23.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.209+0000 7f89cb400640 1 -- 172.21.15.28:0/3214773364 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c40036e0 con 0x56489cbf2350 2024-04-23T14:47:23.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.210+0000 7f89e4793740 1 -- 172.21.15.28:0/3214773364 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56489cbf2350 msgr2=0x56489cbee130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:23.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.210+0000 7f89e4793740 1 --2- 172.21.15.28:0/3214773364 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56489cbf2350 0x56489cbee130 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f89c4025810 tx=0x7f89c4011810 comp rx=0 tx=0).stop 2024-04-23T14:47:23.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.210+0000 7f89e4793740 1 -- 172.21.15.28:0/3214773364 shutdown_connections 2024-04-23T14:47:23.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.210+0000 7f89e4793740 1 --2- 172.21.15.28:0/3214773364 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56489cbf2350 0x56489cbee130 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.210+0000 7f89e4793740 1 --2- 172.21.15.28:0/3214773364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56489cbfca20 0x56489cbece50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.210+0000 7f89e4793740 1 --2- 172.21.15.28:0/3214773364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56489cbf1a30 0x56489cc08670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.210+0000 7f89e4793740 1 -- 172.21.15.28:0/3214773364 >> 172.21.15.28:0/3214773364 conn(0x56489cbedd50 msgr2=0x56489cbec740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:23.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.210+0000 7f89e4793740 1 -- 172.21.15.28:0/3214773364 shutdown_connections 2024-04-23T14:47:23.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.210+0000 7f89e4793740 1 -- 172.21.15.28:0/3214773364 wait complete. 2024-04-23T14:47:23.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.211+0000 7f89e4793740 1 -- start start 2024-04-23T14:47:23.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.211+0000 7f89e4793740 1 Processor -- start 2024-04-23T14:47:23.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.211+0000 7f89e4793740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56489cbf1a30 0x56489cc04d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.211+0000 7f89e4793740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56489cbf2350 0x56489cc000e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.211+0000 7f89e4793740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56489cbfca20 0x56489cc00620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.211+0000 7f89e4793740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc1b2e20 con 0x56489cbf1a30 2024-04-23T14:47:23.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.211+0000 7f89e4793740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc1b4290 con 0x56489cbf2350 2024-04-23T14:47:23.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.211+0000 7f89e4793740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc1b34f0 con 0x56489cbfca20 2024-04-23T14:47:23.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.212+0000 7f89e4793740 10 client.0 _ll_register_callbacks cb 0x7f89c4009710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:47:23.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.212+0000 7f89db400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56489cbf1a30 0x56489cc04d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.212+0000 7f89da000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56489cbf2350 0x56489cc000e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.212+0000 7f89da000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56489cbf2350 0x56489cc000e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:43330/0 (socket says 172.21.15.28:43330) 2024-04-23T14:47:23.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.212+0000 7f89da000640 1 -- 172.21.15.28:0/429482945 learned_addr learned my addr 172.21.15.28:0/429482945 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:23.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.212+0000 7f89db400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56489cbf1a30 0x56489cc04d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57496/0 (socket says 172.21.15.28:57496) 2024-04-23T14:47:23.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.212+0000 7f89db400640 1 --2- 172.21.15.28:0/429482945 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56489cbfca20 0x56489cc00620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.212+0000 7f89da000640 1 -- 172.21.15.28:0/429482945 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56489cbfca20 msgr2=0x56489cc00620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:23.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.212+0000 7f89da000640 1 --2- 172.21.15.28:0/429482945 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56489cbfca20 0x56489cc00620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.212+0000 7f89da000640 1 -- 172.21.15.28:0/429482945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56489cbf1a30 msgr2=0x56489cc04d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:23.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.212+0000 7f89da000640 1 --2- 172.21.15.28:0/429482945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56489cbf1a30 0x56489cc04d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.212+0000 7f89da000640 1 -- 172.21.15.28:0/429482945 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56489c8693f0 con 0x56489cbf2350 2024-04-23T14:47:23.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.212+0000 7f89db400640 1 --2- 172.21.15.28:0/429482945 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56489cbfca20 0x56489cc00620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:47:23.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.213+0000 7f89da000640 1 --2- 172.21.15.28:0/429482945 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56489cbf2350 0x56489cc000e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f89c4025930 tx=0x7f89c4011de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=917fab0e6375525 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:23.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.213+0000 7f89c8c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:47:23.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.213+0000 7f89c8c00640 1 -- 172.21.15.28:0/429482945 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c4025d80 con 0x56489cbf2350 2024-04-23T14:47:23.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.213+0000 7f89c8c00640 1 -- 172.21.15.28:0/429482945 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f89c4005040 con 0x56489cbf2350 2024-04-23T14:47:23.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.213+0000 7f89e4793740 1 -- 172.21.15.28:0/429482945 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89c4025610 con 0x56489cbf2350 2024-04-23T14:47:23.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.213+0000 7f89c8c00640 1 -- 172.21.15.28:0/429482945 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f89c4005240 con 0x56489cbf2350 2024-04-23T14:47:23.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.213+0000 7f89e4793740 10 client.25769 fetch_fsmap learned FSMap version 20 2024-04-23T14:47:23.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.213+0000 7f89e4793740 1 -- 172.21.15.28:0/429482945 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x56489c8290f0 con 0x56489cbf2350 2024-04-23T14:47:23.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.213+0000 7f89c8c00640 1 -- 172.21.15.28:0/429482945 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89c4025610 con 0x56489cbf2350 2024-04-23T14:47:23.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.214+0000 7f89c8c00640 1 -- 172.21.15.28:0/429482945 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f89c4037480 con 0x56489cbf2350 2024-04-23T14:47:23.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.214+0000 7f89e4793740 10 client.25769 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:47:23.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.214+0000 7f89e4793740 10 client.25769 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:47:23.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.214+0000 7f89e4793740 20 client.25769 populate_metadata read hostname 'smithi028' 2024-04-23T14:47:23.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.214+0000 7f89e4793740 1 --2- 172.21.15.28:0/429482945 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56489cc733b0 0x56489cc757d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.214+0000 7f89e4793740 4 client.25769 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:47:23.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.214+0000 7f89e4793740 1 -- 172.21.15.28:0/429482945 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56489cb5ea80 con 0x56489cc733b0 2024-04-23T14:47:23.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.214+0000 7f89db400640 1 --2- 172.21.15.28:0/429482945 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56489cc733b0 0x56489cc757d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.214+0000 7f89db400640 1 --2- 172.21.15.28:0/429482945 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56489cc733b0 0x56489cc757d0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24114616c3dc6e4c server_cookie=cd0d605a1e2f4ad6 in_seq=0 out_seq=0 2024-04-23T14:47:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.214+0000 7f89c8c00640 10 client.25769 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:47:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.215+0000 7f89c8c00640 1 -- 172.21.15.28:0/429482945 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56489cb5ea80 con 0x56489cc733b0 2024-04-23T14:47:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.215+0000 7f89c8c00640 10 client.25769 handle_command_reply: tid=0 2024-04-23T14:47:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.215+0000 7f89e4793740 1 client.25769 shutdown 2024-04-23T14:47:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.215+0000 7f89e4793740 1 -- 172.21.15.28:0/429482945 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56489cbf2350 msgr2=0x56489cc000e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.215+0000 7f89e4793740 1 --2- 172.21.15.28:0/429482945 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56489cbf2350 0x56489cc000e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f89c4025930 tx=0x7f89c4011de0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.215+0000 7f89e4793740 1 -- 172.21.15.28:0/429482945 shutdown_connections 2024-04-23T14:47:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.215+0000 7f89e4793740 1 --2- 172.21.15.28:0/429482945 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56489cc733b0 0x56489cc757d0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.215+0000 7f89e4793740 1 --2- 172.21.15.28:0/429482945 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56489cbf2350 0x56489cc000e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.215+0000 7f89e4793740 1 --2- 172.21.15.28:0/429482945 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56489cbfca20 0x56489cc00620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.215+0000 7f89e4793740 1 --2- 172.21.15.28:0/429482945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56489cbf1a30 0x56489cc04d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.215+0000 7f89c8c00640 0 client.25769 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:47:23.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.215+0000 7f89e4793740 1 -- 172.21.15.28:0/429482945 >> 172.21.15.28:0/429482945 conn(0x56489cbedd50 msgr2=0x56489cbecfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:23.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.215+0000 7f89e4793740 1 -- 172.21.15.28:0/429482945 shutdown_connections 2024-04-23T14:47:23.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.215+0000 7f89e4793740 1 -- 172.21.15.28:0/429482945 wait complete. 2024-04-23T14:47:23.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.215+0000 7f89e4793740 20 client.25769 trim_cache size 0 max 16384 2024-04-23T14:47:23.221 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:47:23.221 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-23T14:47:23.221 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-23T14:47:23.221 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:47:23.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.216+0000 7f89dbe00640 1 -- 172.21.15.28:0/653597188 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f89c00a3da0 msgr2=0x7f89c00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:23.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.216+0000 7f89dbe00640 1 --2- 172.21.15.28:0/653597188 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f89c00a3da0 0x7f89c00a6260 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x56489cbf2ff0 tx=0x7f89cc00c8b0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.216+0000 7f89dbe00640 1 -- 172.21.15.28:0/653597188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89dc103060 msgr2=0x7f89dc1090c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:23.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.216+0000 7f89dbe00640 1 --2- 172.21.15.28:0/653597188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89dc103060 0x7f89dc1090c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f89cc00eab0 tx=0x7f89cc00ef70 comp rx=0 tx=0).stop 2024-04-23T14:47:23.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.216+0000 7f89dbe00640 1 -- 172.21.15.28:0/653597188 shutdown_connections 2024-04-23T14:47:23.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.216+0000 7f89dbe00640 1 --2- 172.21.15.28:0/653597188 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f89c00a3da0 0x7f89c00a6260 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.216+0000 7f89dbe00640 1 --2- 172.21.15.28:0/653597188 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89dc10c480 0x7f89dc109b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.216+0000 7f89dbe00640 1 --2- 172.21.15.28:0/653597188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89dc103980 0x7f89dc109600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.216+0000 7f89dbe00640 1 --2- 172.21.15.28:0/653597188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89dc103060 0x7f89dc1090c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.217+0000 7f89dbe00640 1 -- 172.21.15.28:0/653597188 >> 172.21.15.28:0/653597188 conn(0x7f89dc0fc880 msgr2=0x7f89dc0fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:23.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.217+0000 7f89dbe00640 1 -- 172.21.15.28:0/653597188 shutdown_connections 2024-04-23T14:47:23.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.217+0000 7f89dbe00640 1 -- 172.21.15.28:0/653597188 wait complete. 2024-04-23T14:47:23.237 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-23T14:47:23.237 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:47:23.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.451+0000 7fccda200640 1 -- 172.21.15.28:0/651646200 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccd4102090 msgr2=0x7fccd4102470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:23.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.451+0000 7fccda200640 1 --2- 172.21.15.28:0/651646200 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccd4102090 0x7fccd4102470 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fccc4009900 tx=0x7fccc402a3f0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.452+0000 7fccda200640 1 -- 172.21.15.28:0/651646200 shutdown_connections 2024-04-23T14:47:23.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.452+0000 7fccda200640 1 --2- 172.21.15.28:0/651646200 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fccd4107160 0x7fccd410ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.452+0000 7fccda200640 1 --2- 172.21.15.28:0/651646200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccd4102a40 0x7fccd4106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.452+0000 7fccda200640 1 --2- 172.21.15.28:0/651646200 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccd4102090 0x7fccd4102470 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.452+0000 7fccda200640 1 -- 172.21.15.28:0/651646200 >> 172.21.15.28:0/651646200 conn(0x7fccd40fd930 msgr2=0x7fccd40ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:23.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.452+0000 7fccda200640 1 -- 172.21.15.28:0/651646200 shutdown_connections 2024-04-23T14:47:23.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.452+0000 7fccda200640 1 -- 172.21.15.28:0/651646200 wait complete. 2024-04-23T14:47:23.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.453+0000 7fccda200640 1 Processor -- start 2024-04-23T14:47:23.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.453+0000 7fccda200640 1 -- start start 2024-04-23T14:47:23.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.454+0000 7fccda200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fccd4102090 0x7fccd41a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.454+0000 7fccda200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccd4107160 0x7fccd41a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.454+0000 7fccda200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccd419a790 0x7fccd419ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.454+0000 7fccda200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fccd41121f0 con 0x7fccd419a790 2024-04-23T14:47:23.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.454+0000 7fccda200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fccd4112070 con 0x7fccd4107160 2024-04-23T14:47:23.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.454+0000 7fccda200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fccd4112370 con 0x7fccd4102090 2024-04-23T14:47:23.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.454+0000 7fccd8e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fccd4102090 0x7fccd41a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.454+0000 7fccd8e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fccd4102090 0x7fccd41a0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:58240/0 (socket says 172.21.15.28:58240) 2024-04-23T14:47:23.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.454+0000 7fccd8e00640 1 -- 172.21.15.28:0/894352934 learned_addr learned my addr 172.21.15.28:0/894352934 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:23.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.454+0000 7fccd9800640 1 --2- 172.21.15.28:0/894352934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccd419a790 0x7fccd419ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.454+0000 7fcccbe00640 1 --2- 172.21.15.28:0/894352934 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccd4107160 0x7fccd41a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.454+0000 7fccd8e00640 1 -- 172.21.15.28:0/894352934 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccd4107160 msgr2=0x7fccd41a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:23.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.454+0000 7fccd8e00640 1 --2- 172.21.15.28:0/894352934 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccd4107160 0x7fccd41a0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.454+0000 7fccd8e00640 1 -- 172.21.15.28:0/894352934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccd419a790 msgr2=0x7fccd419ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:23.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.454+0000 7fccd8e00640 1 --2- 172.21.15.28:0/894352934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccd419a790 0x7fccd419ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.454+0000 7fccd8e00640 1 -- 172.21.15.28:0/894352934 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccd419b320 con 0x7fccd4102090 2024-04-23T14:47:23.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.455+0000 7fccd8e00640 1 --2- 172.21.15.28:0/894352934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fccd4102090 0x7fccd41a0620 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fccc4006010 tx=0x7fccc4004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:23.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.455+0000 7fccd2000640 1 -- 172.21.15.28:0/894352934 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc403f030 con 0x7fccd4102090 2024-04-23T14:47:23.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.455+0000 7fccd2000640 1 -- 172.21.15.28:0/894352934 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fccc4004440 con 0x7fccd4102090 2024-04-23T14:47:23.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.455+0000 7fccd2000640 1 -- 172.21.15.28:0/894352934 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc4035070 con 0x7fccd4102090 2024-04-23T14:47:23.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.455+0000 7fccda200640 1 -- 172.21.15.28:0/894352934 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccd419b5b0 con 0x7fccd4102090 2024-04-23T14:47:23.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.455+0000 7fccda200640 1 -- 172.21.15.28:0/894352934 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccd41a7760 con 0x7fccd4102090 2024-04-23T14:47:23.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.456+0000 7fccda200640 1 -- 172.21.15.28:0/894352934 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccd4103560 con 0x7fccd4102090 2024-04-23T14:47:23.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.459+0000 7fccd2000640 1 -- 172.21.15.28:0/894352934 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fccc4042030 con 0x7fccd4102090 2024-04-23T14:47:23.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.460+0000 7fccd2000640 1 --2- 172.21.15.28:0/894352934 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fccb80a3d20 0x7fccb80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.460+0000 7fccd2000640 1 -- 172.21.15.28:0/894352934 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fccc40f0f60 con 0x7fccd4102090 2024-04-23T14:47:23.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.460+0000 7fcccbe00640 1 --2- 172.21.15.28:0/894352934 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fccb80a3d20 0x7fccb80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.461+0000 7fcccbe00640 1 --2- 172.21.15.28:0/894352934 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fccb80a3d20 0x7fccb80a61e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fccd410f180 tx=0x7fccbc005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:23.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.462+0000 7fccd2000640 1 -- 172.21.15.28:0/894352934 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fccc40bd5b0 con 0x7fccd4102090 2024-04-23T14:47:23.471 INFO:tasks.workunit.client.0.smithi028.stdout: 1 191860 10.64 MB/sec warmup 99 sec latency 16.521 ms 2024-04-23T14:47:23.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.725+0000 7fccda200640 1 -- 172.21.15.28:0/894352934 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fccd4068d40 con 0x7fccd4102090 2024-04-23T14:47:23.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.726+0000 7fccd2000640 1 -- 172.21.15.28:0/894352934 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7fccc40bceb0 con 0x7fccd4102090 2024-04-23T14:47:23.727 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:47:23.728 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:47:23.728 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:47:23.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.729+0000 7fccda200640 1 -- 172.21.15.28:0/894352934 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fccb80a3d20 msgr2=0x7fccb80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:23.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.729+0000 7fccda200640 1 --2- 172.21.15.28:0/894352934 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fccb80a3d20 0x7fccb80a61e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fccd410f180 tx=0x7fccbc005f90 comp rx=0 tx=0).stop 2024-04-23T14:47:23.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.729+0000 7fccda200640 1 -- 172.21.15.28:0/894352934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fccd4102090 msgr2=0x7fccd41a0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:23.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.729+0000 7fccda200640 1 --2- 172.21.15.28:0/894352934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fccd4102090 0x7fccd41a0620 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fccc4006010 tx=0x7fccc4004040 comp rx=0 tx=0).stop 2024-04-23T14:47:23.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.729+0000 7fccda200640 1 -- 172.21.15.28:0/894352934 shutdown_connections 2024-04-23T14:47:23.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.729+0000 7fccda200640 1 --2- 172.21.15.28:0/894352934 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fccb80a3d20 0x7fccb80a61e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.729+0000 7fccda200640 1 --2- 172.21.15.28:0/894352934 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccd4107160 0x7fccd41a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.729+0000 7fccda200640 1 --2- 172.21.15.28:0/894352934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccd419a790 0x7fccd419ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.729+0000 7fccda200640 1 --2- 172.21.15.28:0/894352934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fccd4102090 0x7fccd41a0620 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.729+0000 7fccda200640 1 -- 172.21.15.28:0/894352934 >> 172.21.15.28:0/894352934 conn(0x7fccd40fd930 msgr2=0x7fccd40ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:23.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.729+0000 7fccda200640 1 -- 172.21.15.28:0/894352934 shutdown_connections 2024-04-23T14:47:23.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.729+0000 7fccda200640 1 -- 172.21.15.28:0/894352934 wait complete. 2024-04-23T14:47:23.751 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-23T14:47:23.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.958+0000 7f7ba7200640 1 -- 172.21.15.28:0/2496059124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ba0102240 msgr2=0x7f7ba0102620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:23.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.958+0000 7f7ba7200640 1 --2- 172.21.15.28:0/2496059124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ba0102240 0x7f7ba0102620 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7b90009ca0 tx=0x7f7b9002a8c0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.958+0000 7f7ba7200640 1 -- 172.21.15.28:0/2496059124 shutdown_connections 2024-04-23T14:47:23.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.958+0000 7f7ba7200640 1 --2- 172.21.15.28:0/2496059124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ba0107310 0x7f7ba010eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.958+0000 7f7ba7200640 1 --2- 172.21.15.28:0/2496059124 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7ba0102bf0 0x7f7ba0106c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.958+0000 7f7ba7200640 1 --2- 172.21.15.28:0/2496059124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ba0102240 0x7f7ba0102620 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.958+0000 7f7ba7200640 1 -- 172.21.15.28:0/2496059124 >> 172.21.15.28:0/2496059124 conn(0x7f7ba0079450 msgr2=0x7f7ba0079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:23.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.958+0000 7f7ba7200640 1 -- 172.21.15.28:0/2496059124 shutdown_connections 2024-04-23T14:47:23.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.958+0000 7f7ba7200640 1 -- 172.21.15.28:0/2496059124 wait complete. 2024-04-23T14:47:23.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.959+0000 7f7ba7200640 1 Processor -- start 2024-04-23T14:47:23.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.960+0000 7f7ba7200640 1 -- start start 2024-04-23T14:47:23.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.960+0000 7f7ba7200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ba0102240 0x7f7ba01a0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.960+0000 7f7ba7200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7ba0107310 0x7f7ba01a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.960+0000 7f7ba7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ba019a8f0 0x7f7ba019ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.960+0000 7f7ba7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba01123f0 con 0x7f7ba019a8f0 2024-04-23T14:47:23.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.960+0000 7f7ba7200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba0112270 con 0x7f7ba0102240 2024-04-23T14:47:23.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.960+0000 7f7ba7200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba0112570 con 0x7f7ba0107310 2024-04-23T14:47:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.960+0000 7f7b9fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ba0102240 0x7f7ba01a0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.960+0000 7f7b9fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ba0102240 0x7f7ba01a0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:43358/0 (socket says 172.21.15.28:43358) 2024-04-23T14:47:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.960+0000 7f7b9fe00640 1 -- 172.21.15.28:0/732413112 learned_addr learned my addr 172.21.15.28:0/732413112 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.960+0000 7f7b9f400640 1 --2- 172.21.15.28:0/732413112 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7ba0107310 0x7f7ba01a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.961+0000 7f7ba5200640 1 --2- 172.21.15.28:0/732413112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ba019a8f0 0x7f7ba019ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.961+0000 7f7b9fe00640 1 -- 172.21.15.28:0/732413112 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7ba0107310 msgr2=0x7f7ba01a0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.961+0000 7f7b9fe00640 1 --2- 172.21.15.28:0/732413112 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7ba0107310 0x7f7ba01a0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.961+0000 7f7b9fe00640 1 -- 172.21.15.28:0/732413112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ba019a8f0 msgr2=0x7f7ba019ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.961+0000 7f7b9fe00640 1 --2- 172.21.15.28:0/732413112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ba019a8f0 0x7f7ba019ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.961+0000 7f7b9fe00640 1 -- 172.21.15.28:0/732413112 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ba019b410 con 0x7f7ba0102240 2024-04-23T14:47:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.961+0000 7f7b9fe00640 1 --2- 172.21.15.28:0/732413112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ba0102240 0x7f7ba01a0740 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7b90005e30 tx=0x7f7b90003f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.961+0000 7f7b9cc00640 1 -- 172.21.15.28:0/732413112 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b90031650 con 0x7f7ba0102240 2024-04-23T14:47:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.961+0000 7f7b9cc00640 1 -- 172.21.15.28:0/732413112 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b900317f0 con 0x7f7ba0102240 2024-04-23T14:47:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.961+0000 7f7b9cc00640 1 -- 172.21.15.28:0/732413112 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b9002fa30 con 0x7f7ba0102240 2024-04-23T14:47:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.961+0000 7f7ba7200640 1 -- 172.21.15.28:0/732413112 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ba019b6a0 con 0x7f7ba0102240 2024-04-23T14:47:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.962+0000 7f7ba7200640 1 -- 172.21.15.28:0/732413112 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ba01a7940 con 0x7f7ba0102240 2024-04-23T14:47:23.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.965+0000 7f7baa1aa740 1 Processor -- start 2024-04-23T14:47:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.965+0000 7f7baa1aa740 1 -- start start 2024-04-23T14:47:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.965+0000 7f7b9cc00640 1 -- 172.21.15.28:0/732413112 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f7b9002fc30 con 0x7f7ba0102240 2024-04-23T14:47:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.965+0000 7f7baa1aa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c99cfa6a0 0x555c99cfaa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.965+0000 7f7baa1aa740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555c99cfb050 0x555c99d0dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.965+0000 7f7baa1aa740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c99d0e150 0x555c99d10540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.965+0000 7f7baa1aa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555c99c601d0 con 0x555c99cfa6a0 2024-04-23T14:47:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.965+0000 7f7baa1aa740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555c99c06450 con 0x555c99d0e150 2024-04-23T14:47:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.965+0000 7f7baa1aa740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x555c99c5c190 con 0x555c99cfb050 2024-04-23T14:47:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.966+0000 7f7b9fe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555c99cfb050 0x555c99d0dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.966+0000 7f7ba5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c99cfa6a0 0x555c99cfaa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.966+0000 7f7b9f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c99d0e150 0x555c99d10540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.966+0000 7f7b9fe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555c99cfb050 0x555c99d0dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:58266/0 (socket says 172.21.15.28:58266) 2024-04-23T14:47:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.966+0000 7f7b9fe00640 1 -- 172.21.15.28:0/1709178500 learned_addr learned my addr 172.21.15.28:0/1709178500 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.966+0000 7f7b9fe00640 1 -- 172.21.15.28:0/1709178500 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c99d0e150 msgr2=0x555c99d10540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.966+0000 7f7b9fe00640 1 --2- 172.21.15.28:0/1709178500 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c99d0e150 0x555c99d10540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.966+0000 7f7b9fe00640 1 -- 172.21.15.28:0/1709178500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c99cfa6a0 msgr2=0x555c99cfaa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.966+0000 7f7b9fe00640 1 --2- 172.21.15.28:0/1709178500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c99cfa6a0 0x555c99cfaa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.966+0000 7f7b9fe00640 1 -- 172.21.15.28:0/1709178500 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c99cc3310 con 0x555c99cfb050 2024-04-23T14:47:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.966+0000 7f7b9fe00640 1 --2- 172.21.15.28:0/1709178500 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555c99cfb050 0x555c99d0dc10 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7b900ef1f0 tx=0x7f7b900f5a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eed0d52ee57873b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.967+0000 7f7b95600640 1 -- 172.21.15.28:0/1709178500 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b90105070 con 0x555c99cfb050 2024-04-23T14:47:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.967+0000 7f7b95600640 1 -- 172.21.15.28:0/1709178500 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b900fa050 con 0x555c99cfb050 2024-04-23T14:47:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.967+0000 7f7b95600640 1 -- 172.21.15.28:0/1709178500 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b900fe960 con 0x555c99cfb050 2024-04-23T14:47:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.967+0000 7f7b9cc00640 1 --2- 172.21.15.28:0/732413112 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7b840a3da0 0x7f7b840a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.967+0000 7f7b9cc00640 1 -- 172.21.15.28:0/732413112 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f7b900f1aa0 con 0x7f7ba0102240 2024-04-23T14:47:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.967+0000 7f7ba5200640 1 --2- 172.21.15.28:0/732413112 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7b840a3da0 0x7f7b840a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.968+0000 7f7baa1aa740 1 -- 172.21.15.28:0/1709178500 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555c99cfb050 msgr2=0x555c99d0dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.968+0000 7f7baa1aa740 1 --2- 172.21.15.28:0/1709178500 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555c99cfb050 0x555c99d0dc10 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7b900ef1f0 tx=0x7f7b900f5a90 comp rx=0 tx=0).stop 2024-04-23T14:47:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.968+0000 7f7baa1aa740 1 -- 172.21.15.28:0/1709178500 shutdown_connections 2024-04-23T14:47:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.968+0000 7f7baa1aa740 1 --2- 172.21.15.28:0/1709178500 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c99d0e150 0x555c99d10540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.968+0000 7f7baa1aa740 1 --2- 172.21.15.28:0/1709178500 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555c99cfb050 0x555c99d0dc10 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.968+0000 7f7baa1aa740 1 --2- 172.21.15.28:0/1709178500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c99cfa6a0 0x555c99cfaa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.968+0000 7f7baa1aa740 1 -- 172.21.15.28:0/1709178500 >> 172.21.15.28:0/1709178500 conn(0x555c99cf6c70 msgr2=0x555c99cf7050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.968+0000 7f7baa1aa740 1 -- 172.21.15.28:0/1709178500 shutdown_connections 2024-04-23T14:47:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.968+0000 7f7ba5200640 1 --2- 172.21.15.28:0/732413112 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7b840a3da0 0x7f7b840a6260 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7b98009c70 tx=0x7f7b98005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.968+0000 7f7baa1aa740 1 -- 172.21.15.28:0/1709178500 wait complete. 2024-04-23T14:47:23.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.969+0000 7f7baa1aa740 1 -- start start 2024-04-23T14:47:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.969+0000 7f7baa1aa740 1 Processor -- start 2024-04-23T14:47:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.969+0000 7f7baa1aa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c99cfa6a0 0x555c99d0db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.969+0000 7f7baa1aa740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c99cfb050 0x555c99d0ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.969+0000 7f7baa1aa740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555c99d0e150 0x555c99d0b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.969+0000 7f7baa1aa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555c99c5c190 con 0x555c99cfa6a0 2024-04-23T14:47:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.969+0000 7f7baa1aa740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555c99c19810 con 0x555c99cfb050 2024-04-23T14:47:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.969+0000 7f7baa1aa740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x555c99d10cd0 con 0x555c99d0e150 2024-04-23T14:47:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.969+0000 7f7baa1aa740 10 client.0 _ll_register_callbacks cb 0x555c99be7140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:47:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.970+0000 7f7b9f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c99cfb050 0x555c99d0ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.970+0000 7f7ba5200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555c99d0e150 0x555c99d0b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.970+0000 7f7b9f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c99cfb050 0x555c99d0ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:43366/0 (socket says 172.21.15.28:43366) 2024-04-23T14:47:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.970+0000 7f7b9fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c99cfa6a0 0x555c99d0db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.970+0000 7f7b9f400640 1 -- 172.21.15.28:0/1819442147 learned_addr learned my addr 172.21.15.28:0/1819442147 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:23.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.970+0000 7f7b9f400640 1 -- 172.21.15.28:0/1819442147 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555c99d0e150 msgr2=0x555c99d0b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:23.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.970+0000 7f7b9f400640 1 --2- 172.21.15.28:0/1819442147 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555c99d0e150 0x555c99d0b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.970+0000 7f7b9f400640 1 -- 172.21.15.28:0/1819442147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c99cfa6a0 msgr2=0x555c99d0db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:23.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.970+0000 7f7b9f400640 1 --2- 172.21.15.28:0/1819442147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c99cfa6a0 0x555c99d0db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.970+0000 7f7b9f400640 1 -- 172.21.15.28:0/1819442147 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c99c2a3f0 con 0x555c99cfb050 2024-04-23T14:47:23.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.970+0000 7f7b9f400640 1 --2- 172.21.15.28:0/1819442147 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c99cfb050 0x555c99d0ae80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7b880126d0 tx=0x7f7b88012b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=283597f30c3ff1c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:23.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.970+0000 7f7b8e000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:47:23.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.971+0000 7f7b8e000640 1 -- 172.21.15.28:0/1819442147 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b8800f8c0 con 0x555c99cfb050 2024-04-23T14:47:23.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.971+0000 7f7b8e000640 1 -- 172.21.15.28:0/1819442147 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f7b8800fa60 con 0x555c99cfb050 2024-04-23T14:47:23.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.971+0000 7f7baa1aa740 1 -- 172.21.15.28:0/1819442147 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b900fa050 con 0x555c99cfb050 2024-04-23T14:47:23.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.971+0000 7f7b8e000640 1 -- 172.21.15.28:0/1819442147 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f7b8800fc60 con 0x555c99cfb050 2024-04-23T14:47:23.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.971+0000 7f7baa1aa740 10 client.25787 fetch_fsmap learned FSMap version 20 2024-04-23T14:47:23.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.971+0000 7f7baa1aa740 1 -- 172.21.15.28:0/1819442147 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x555c999723f0 con 0x555c99cfb050 2024-04-23T14:47:23.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.971+0000 7f7b8e000640 1 -- 172.21.15.28:0/1819442147 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b900fa050 con 0x555c99cfb050 2024-04-23T14:47:23.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.971+0000 7f7b8e000640 1 -- 172.21.15.28:0/1819442147 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f7b8801c420 con 0x555c99cfb050 2024-04-23T14:47:23.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.972+0000 7f7baa1aa740 10 client.25787 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:47:23.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.972+0000 7f7baa1aa740 10 client.25787 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:47:23.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.972+0000 7f7baa1aa740 20 client.25787 populate_metadata read hostname 'smithi028' 2024-04-23T14:47:23.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.972+0000 7f7baa1aa740 1 --2- 172.21.15.28:0/1819442147 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x555c99d1df40 0x555c99d203e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.972+0000 7f7baa1aa740 4 client.25787 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:47:23.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.972+0000 7f7baa1aa740 1 -- 172.21.15.28:0/1819442147 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555c99c6dbf0 con 0x555c99d1df40 2024-04-23T14:47:23.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.972+0000 7f7b9fe00640 1 --2- 172.21.15.28:0/1819442147 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x555c99d1df40 0x555c99d203e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.972+0000 7f7b9fe00640 1 --2- 172.21.15.28:0/1819442147 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x555c99d1df40 0x555c99d203e0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a3abe9fa4885d67 server_cookie=ebefcacd681b8c0d in_seq=0 out_seq=0 2024-04-23T14:47:23.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.972+0000 7f7b8e000640 10 client.25787 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:47:23.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.973+0000 7f7b8e000640 1 -- 172.21.15.28:0/1819442147 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555c99c6dbf0 con 0x555c99d1df40 2024-04-23T14:47:23.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.973+0000 7f7b8e000640 10 client.25787 handle_command_reply: tid=0 2024-04-23T14:47:23.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.973+0000 7f7baa1aa740 1 client.25787 shutdown 2024-04-23T14:47:23.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.973+0000 7f7baa1aa740 1 -- 172.21.15.28:0/1819442147 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c99cfb050 msgr2=0x555c99d0ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:23.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.973+0000 7f7baa1aa740 1 --2- 172.21.15.28:0/1819442147 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c99cfb050 0x555c99d0ae80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7b880126d0 tx=0x7f7b88012b90 comp rx=0 tx=0).stop 2024-04-23T14:47:23.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.973+0000 7f7baa1aa740 1 -- 172.21.15.28:0/1819442147 shutdown_connections 2024-04-23T14:47:23.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.973+0000 7f7baa1aa740 1 --2- 172.21.15.28:0/1819442147 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x555c99d1df40 0x555c99d203e0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.973+0000 7f7baa1aa740 1 --2- 172.21.15.28:0/1819442147 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c99cfb050 0x555c99d0ae80 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.973+0000 7f7baa1aa740 1 --2- 172.21.15.28:0/1819442147 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555c99d0e150 0x555c99d0b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.973+0000 7f7baa1aa740 1 --2- 172.21.15.28:0/1819442147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c99cfa6a0 0x555c99d0db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.973+0000 7f7b8e000640 0 client.25787 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:47:23.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.973+0000 7f7baa1aa740 1 -- 172.21.15.28:0/1819442147 >> 172.21.15.28:0/1819442147 conn(0x555c99cf6c70 msgr2=0x555c99cfca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:23.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.974+0000 7f7baa1aa740 1 -- 172.21.15.28:0/1819442147 shutdown_connections 2024-04-23T14:47:23.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.974+0000 7f7baa1aa740 1 -- 172.21.15.28:0/1819442147 wait complete. 2024-04-23T14:47:23.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.974+0000 7f7baa1aa740 20 client.25787 trim_cache size 0 max 16384 2024-04-23T14:47:23.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.989+0000 7f7baa1aa740 1 Processor -- start 2024-04-23T14:47:23.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.989+0000 7f7baa1aa740 1 -- start start 2024-04-23T14:47:23.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.990+0000 7f7baa1aa740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c99cfa6a0 0x555c99cf6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.990+0000 7f7baa1aa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c99cfb050 0x555c99d0d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.990+0000 7f7baa1aa740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555c99d0e150 0x555c99d0dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.990+0000 7f7baa1aa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba01a9c80 con 0x555c99cfb050 2024-04-23T14:47:23.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.990+0000 7f7baa1aa740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba01ac500 con 0x555c99cfa6a0 2024-04-23T14:47:23.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.990+0000 7f7baa1aa740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba01aa2f0 con 0x555c99d0e150 2024-04-23T14:47:23.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.990+0000 7f7b9fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c99cfa6a0 0x555c99cf6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.990+0000 7f7b9f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c99cfb050 0x555c99d0d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.990+0000 7f7b9fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c99cfa6a0 0x555c99cf6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:43374/0 (socket says 172.21.15.28:43374) 2024-04-23T14:47:23.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.990+0000 7f7b9fe00640 1 -- 172.21.15.28:0/2931483063 learned_addr learned my addr 172.21.15.28:0/2931483063 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:23.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.990+0000 7f7ba5200640 1 --2- 172.21.15.28:0/2931483063 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555c99d0e150 0x555c99d0dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.990+0000 7f7b9fe00640 1 -- 172.21.15.28:0/2931483063 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555c99d0e150 msgr2=0x555c99d0dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:23.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.990+0000 7f7b9fe00640 1 --2- 172.21.15.28:0/2931483063 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555c99d0e150 0x555c99d0dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.990+0000 7f7b9fe00640 1 -- 172.21.15.28:0/2931483063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c99cfb050 msgr2=0x555c99d0d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:23.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.990+0000 7f7b9fe00640 1 --2- 172.21.15.28:0/2931483063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c99cfb050 0x555c99d0d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.990+0000 7f7b9fe00640 1 -- 172.21.15.28:0/2931483063 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c99a3c7d0 con 0x555c99cfa6a0 2024-04-23T14:47:23.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.990+0000 7f7b9fe00640 1 --2- 172.21.15.28:0/2931483063 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c99cfa6a0 0x555c99cf6570 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7b900fe4f0 tx=0x7f7b900fe520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c8913ce24c7c45 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:23.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.991+0000 7f7b94c00640 1 -- 172.21.15.28:0/2931483063 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b900b9da0 con 0x555c99cfa6a0 2024-04-23T14:47:23.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.991+0000 7f7b94c00640 1 -- 172.21.15.28:0/2931483063 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b900fa070 con 0x555c99cfa6a0 2024-04-23T14:47:23.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.991+0000 7f7b94c00640 1 -- 172.21.15.28:0/2931483063 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b900bd460 con 0x555c99cfa6a0 2024-04-23T14:47:23.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.991+0000 7f7baa1aa740 1 -- 172.21.15.28:0/2931483063 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c99cfa6a0 msgr2=0x555c99cf6570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:23.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.991+0000 7f7baa1aa740 1 --2- 172.21.15.28:0/2931483063 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c99cfa6a0 0x555c99cf6570 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7b900fe4f0 tx=0x7f7b900fe520 comp rx=0 tx=0).stop 2024-04-23T14:47:23.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.991+0000 7f7baa1aa740 1 -- 172.21.15.28:0/2931483063 shutdown_connections 2024-04-23T14:47:23.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.991+0000 7f7baa1aa740 1 --2- 172.21.15.28:0/2931483063 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555c99d0e150 0x555c99d0dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.991+0000 7f7baa1aa740 1 --2- 172.21.15.28:0/2931483063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c99cfb050 0x555c99d0d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.991+0000 7f7baa1aa740 1 --2- 172.21.15.28:0/2931483063 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c99cfa6a0 0x555c99cf6570 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.991+0000 7f7baa1aa740 1 -- 172.21.15.28:0/2931483063 >> 172.21.15.28:0/2931483063 conn(0x555c99cf6c70 msgr2=0x555c99d061f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:23.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.992+0000 7f7baa1aa740 1 -- 172.21.15.28:0/2931483063 shutdown_connections 2024-04-23T14:47:23.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.992+0000 7f7baa1aa740 1 -- 172.21.15.28:0/2931483063 wait complete. 2024-04-23T14:47:23.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.992+0000 7f7baa1aa740 1 -- start start 2024-04-23T14:47:23.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.992+0000 7f7baa1aa740 1 Processor -- start 2024-04-23T14:47:23.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.993+0000 7f7baa1aa740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c99cfa6a0 0x555c99d16460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.993+0000 7f7baa1aa740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555c99cfb050 0x555c99d08550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.993+0000 7f7baa1aa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c99d0e150 0x555c99d08bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:23.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.993+0000 7f7baa1aa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba01aa9c0 con 0x555c99d0e150 2024-04-23T14:47:23.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.993+0000 7f7baa1aa740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba01abe30 con 0x555c99cfa6a0 2024-04-23T14:47:23.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.993+0000 7f7baa1aa740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba01ab090 con 0x555c99cfb050 2024-04-23T14:47:23.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.993+0000 7f7baa1aa740 10 client.0 _ll_register_callbacks cb 0x7f7b88009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:47:23.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.993+0000 7f7b9fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c99cfa6a0 0x555c99d16460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.993+0000 7f7b9f400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555c99cfb050 0x555c99d08550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.993+0000 7f7b9fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c99cfa6a0 0x555c99d16460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:43390/0 (socket says 172.21.15.28:43390) 2024-04-23T14:47:23.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.993+0000 7f7b9fe00640 1 -- 172.21.15.28:0/3862186201 learned_addr learned my addr 172.21.15.28:0/3862186201 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:23.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.993+0000 7f7ba5200640 1 --2- 172.21.15.28:0/3862186201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c99d0e150 0x555c99d08bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:23.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.993+0000 7f7b9f400640 1 -- 172.21.15.28:0/3862186201 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c99cfa6a0 msgr2=0x555c99d16460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:23.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.993+0000 7f7b9f400640 1 --2- 172.21.15.28:0/3862186201 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c99cfa6a0 0x555c99d16460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.993+0000 7f7b9f400640 1 -- 172.21.15.28:0/3862186201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c99d0e150 msgr2=0x555c99d08bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:23.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.993+0000 7f7b9f400640 1 --2- 172.21.15.28:0/3862186201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c99d0e150 0x555c99d08bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:23.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.993+0000 7f7b9f400640 1 -- 172.21.15.28:0/3862186201 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c99d0f460 con 0x555c99cfb050 2024-04-23T14:47:23.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.993+0000 7f7ba5200640 1 --2- 172.21.15.28:0/3862186201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c99d0e150 0x555c99d08bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:47:23.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.994+0000 7f7b9f400640 1 --2- 172.21.15.28:0/3862186201 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555c99cfb050 0x555c99d08550 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7b8801ca80 tx=0x7f7b880094f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=446eb62e6f3bc1bc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:23.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.994+0000 7f7b8e000640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:47:23.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.994+0000 7f7b8e000640 1 -- 172.21.15.28:0/3862186201 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b8800f9a0 con 0x555c99cfb050 2024-04-23T14:47:23.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.994+0000 7f7baa1aa740 1 -- 172.21.15.28:0/3862186201 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b900fa070 con 0x555c99cfb050 2024-04-23T14:47:23.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.994+0000 7f7b8e000640 1 -- 172.21.15.28:0/3862186201 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f7b8801cb00 con 0x555c99cfb050 2024-04-23T14:47:23.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.994+0000 7f7b8e000640 1 -- 172.21.15.28:0/3862186201 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f7b88034050 con 0x555c99cfb050 2024-04-23T14:47:23.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.994+0000 7f7baa1aa740 10 client.25786 fetch_fsmap learned FSMap version 20 2024-04-23T14:47:23.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.994+0000 7f7baa1aa740 1 -- 172.21.15.28:0/3862186201 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x555c99d84ab0 con 0x555c99cfb050 2024-04-23T14:47:23.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.994+0000 7f7b8e000640 1 -- 172.21.15.28:0/3862186201 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b900fa070 con 0x555c99cfb050 2024-04-23T14:47:24.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.994+0000 7f7b8e000640 1 -- 172.21.15.28:0/3862186201 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f7b88003710 con 0x555c99cfb050 2024-04-23T14:47:24.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.995+0000 7f7baa1aa740 10 client.25786 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:47:24.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.995+0000 7f7baa1aa740 10 client.25786 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:47:24.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.995+0000 7f7baa1aa740 20 client.25786 populate_metadata read hostname 'smithi028' 2024-04-23T14:47:24.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.995+0000 7f7baa1aa740 1 --2- 172.21.15.28:0/3862186201 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x555c99d85170 0x555c99d87590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:24.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.995+0000 7f7baa1aa740 4 client.25786 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-23T14:47:24.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.995+0000 7f7baa1aa740 1 -- 172.21.15.28:0/3862186201 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555c99c67a60 con 0x555c99d85170 2024-04-23T14:47:24.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.995+0000 7f7b9fe00640 1 --2- 172.21.15.28:0/3862186201 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x555c99d85170 0x555c99d87590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:24.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.995+0000 7f7b9fe00640 1 --2- 172.21.15.28:0/3862186201 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x555c99d85170 0x555c99d87590 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60677386ce06e382 server_cookie=29a4186781ecb26e in_seq=0 out_seq=0 2024-04-23T14:47:24.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.995+0000 7f7b8e000640 10 client.25786 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:47:24.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.996+0000 7f7b8e000640 1 -- 172.21.15.28:0/3862186201 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555c99c67a60 con 0x555c99d85170 2024-04-23T14:47:24.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.996+0000 7f7b8e000640 10 client.25786 handle_command_reply: tid=0 2024-04-23T14:47:24.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.996+0000 7f7baa1aa740 1 client.25786 shutdown 2024-04-23T14:47:24.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.996+0000 7f7baa1aa740 1 -- 172.21.15.28:0/3862186201 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555c99cfb050 msgr2=0x555c99d08550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:24.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.996+0000 7f7baa1aa740 1 --2- 172.21.15.28:0/3862186201 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555c99cfb050 0x555c99d08550 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7b8801ca80 tx=0x7f7b880094f0 comp rx=0 tx=0).stop 2024-04-23T14:47:24.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.996+0000 7f7baa1aa740 1 -- 172.21.15.28:0/3862186201 shutdown_connections 2024-04-23T14:47:24.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.996+0000 7f7baa1aa740 1 --2- 172.21.15.28:0/3862186201 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x555c99d85170 0x555c99d87590 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:24.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.996+0000 7f7baa1aa740 1 --2- 172.21.15.28:0/3862186201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c99d0e150 0x555c99d08bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:24.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.996+0000 7f7baa1aa740 1 --2- 172.21.15.28:0/3862186201 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555c99cfb050 0x555c99d08550 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:24.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.996+0000 7f7baa1aa740 1 --2- 172.21.15.28:0/3862186201 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c99cfa6a0 0x555c99d16460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:24.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.996+0000 7f7baa1aa740 1 -- 172.21.15.28:0/3862186201 >> 172.21.15.28:0/3862186201 conn(0x555c99cf6c70 msgr2=0x555c99cfc1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:24.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.996+0000 7f7b8e000640 0 client.25786 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:47:24.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.996+0000 7f7baa1aa740 1 -- 172.21.15.28:0/3862186201 shutdown_connections 2024-04-23T14:47:24.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.996+0000 7f7baa1aa740 1 -- 172.21.15.28:0/3862186201 wait complete. 2024-04-23T14:47:24.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.996+0000 7f7baa1aa740 20 client.25786 trim_cache size 0 max 16384 2024-04-23T14:47:24.001 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-23T14:47:24.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.997+0000 7f7ba7200640 1 -- 172.21.15.28:0/732413112 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7b840a3da0 msgr2=0x7f7b840a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:24.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.997+0000 7f7ba7200640 1 --2- 172.21.15.28:0/732413112 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7b840a3da0 0x7f7b840a6260 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7b98009c70 tx=0x7f7b98005f90 comp rx=0 tx=0).stop 2024-04-23T14:47:24.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.997+0000 7f7ba7200640 1 -- 172.21.15.28:0/732413112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ba0102240 msgr2=0x7f7ba01a0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:24.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.997+0000 7f7ba7200640 1 --2- 172.21.15.28:0/732413112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ba0102240 0x7f7ba01a0740 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7b90005e30 tx=0x7f7b90003f90 comp rx=0 tx=0).stop 2024-04-23T14:47:24.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.997+0000 7f7ba7200640 1 -- 172.21.15.28:0/732413112 shutdown_connections 2024-04-23T14:47:24.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.997+0000 7f7ba7200640 1 --2- 172.21.15.28:0/732413112 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7b840a3da0 0x7f7b840a6260 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:24.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.997+0000 7f7ba7200640 1 --2- 172.21.15.28:0/732413112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ba019a8f0 0x7f7ba019ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:24.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.997+0000 7f7ba7200640 1 --2- 172.21.15.28:0/732413112 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7ba0107310 0x7f7ba01a0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:24.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.997+0000 7f7ba7200640 1 --2- 172.21.15.28:0/732413112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ba0102240 0x7f7ba01a0740 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:24.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.997+0000 7f7ba7200640 1 -- 172.21.15.28:0/732413112 >> 172.21.15.28:0/732413112 conn(0x7f7ba0079450 msgr2=0x7f7ba0100010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:24.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.997+0000 7f7ba7200640 1 -- 172.21.15.28:0/732413112 shutdown_connections 2024-04-23T14:47:24.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:23.997+0000 7f7ba7200640 1 -- 172.21.15.28:0/732413112 wait complete. 2024-04-23T14:47:24.471 INFO:tasks.workunit.client.0.smithi028.stdout: 1 194118 10.61 MB/sec warmup 100 sec latency 12.621 ms 2024-04-23T14:47:24.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:24 smithi038 ceph-mon[32348]: pgmap v104: 97 pgs: 97 active+clean; 120 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 231 op/s 2024-04-23T14:47:24.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:24 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/894352934' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:47:24.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:24 smithi028 ceph-mon[27810]: pgmap v104: 97 pgs: 97 active+clean; 120 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 231 op/s 2024-04-23T14:47:24.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:24 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/894352934' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:47:25.018 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-23T14:47:25.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:47:25.018 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-23T14:47:25.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:24 smithi105 ceph-mon[32555]: pgmap v104: 97 pgs: 97 active+clean; 120 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 231 op/s 2024-04-23T14:47:25.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:24 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/894352934' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:47:25.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.242+0000 7f5f41a00640 1 -- 172.21.15.28:0/1263328195 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f3c076e50 msgr2=0x7f5f3c112800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:25.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.242+0000 7f5f41a00640 1 --2- 172.21.15.28:0/1263328195 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f3c076e50 0x7f5f3c112800 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5f24009930 tx=0x7f5f2402a420 comp rx=0 tx=0).stop 2024-04-23T14:47:25.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.242+0000 7f5f41a00640 1 -- 172.21.15.28:0/1263328195 shutdown_connections 2024-04-23T14:47:25.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.242+0000 7f5f41a00640 1 --2- 172.21.15.28:0/1263328195 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f3c076e50 0x7f5f3c112800 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.242+0000 7f5f41a00640 1 --2- 172.21.15.28:0/1263328195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f3c112d40 0x7f5f3c115130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.242+0000 7f5f41a00640 1 --2- 172.21.15.28:0/1263328195 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5f3c076530 0x7f5f3c076910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.242+0000 7f5f41a00640 1 -- 172.21.15.28:0/1263328195 >> 172.21.15.28:0/1263328195 conn(0x7f5f3c0ffb10 msgr2=0x7f5f3c101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:25.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.243+0000 7f5f41a00640 1 -- 172.21.15.28:0/1263328195 shutdown_connections 2024-04-23T14:47:25.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.243+0000 7f5f41a00640 1 -- 172.21.15.28:0/1263328195 wait complete. 2024-04-23T14:47:25.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.244+0000 7f5f41a00640 1 Processor -- start 2024-04-23T14:47:25.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.244+0000 7f5f41a00640 1 -- start start 2024-04-23T14:47:25.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.244+0000 7f5f41a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f3c076530 0x7f5f3c19ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.244+0000 7f5f41a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f3c076e50 0x7f5f3c19f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.244+0000 7f5f41a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5f3c112d40 0x7f5f3c1a36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.244+0000 7f5f41a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3c118900 con 0x7f5f3c076530 2024-04-23T14:47:25.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.244+0000 7f5f41a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3c118780 con 0x7f5f3c076e50 2024-04-23T14:47:25.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.244+0000 7f5f41a00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3c118a80 con 0x7f5f3c112d40 2024-04-23T14:47:25.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.244+0000 7f5f3b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f3c076e50 0x7f5f3c19f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.244+0000 7f5f3b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f3c076e50 0x7f5f3c19f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:43418/0 (socket says 172.21.15.28:43418) 2024-04-23T14:47:25.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.244+0000 7f5f3b400640 1 -- 172.21.15.28:0/1093898580 learned_addr learned my addr 172.21.15.28:0/1093898580 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:25.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.244+0000 7f5f3be00640 1 --2- 172.21.15.28:0/1093898580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f3c076530 0x7f5f3c19ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.245+0000 7f5f41000640 1 --2- 172.21.15.28:0/1093898580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5f3c112d40 0x7f5f3c1a36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.245+0000 7f5f3b400640 1 -- 172.21.15.28:0/1093898580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5f3c112d40 msgr2=0x7f5f3c1a36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:25.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.245+0000 7f5f3b400640 1 --2- 172.21.15.28:0/1093898580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5f3c112d40 0x7f5f3c1a36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.245+0000 7f5f3b400640 1 -- 172.21.15.28:0/1093898580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f3c076530 msgr2=0x7f5f3c19ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:25.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.245+0000 7f5f3b400640 1 --2- 172.21.15.28:0/1093898580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f3c076530 0x7f5f3c19ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.245+0000 7f5f3b400640 1 -- 172.21.15.28:0/1093898580 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f3c1a3df0 con 0x7f5f3c076e50 2024-04-23T14:47:25.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.245+0000 7f5f41000640 1 --2- 172.21.15.28:0/1093898580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5f3c112d40 0x7f5f3c1a36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:47:25.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.245+0000 7f5f3b400640 1 --2- 172.21.15.28:0/1093898580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f3c076e50 0x7f5f3c19f340 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5f2402a940 tx=0x7f5f24003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:25.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.245+0000 7f5f38c00640 1 -- 172.21.15.28:0/1093898580 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2403f070 con 0x7f5f3c076e50 2024-04-23T14:47:25.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.245+0000 7f5f38c00640 1 -- 172.21.15.28:0/1093898580 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f24031c90 con 0x7f5f3c076e50 2024-04-23T14:47:25.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.245+0000 7f5f41a00640 1 -- 172.21.15.28:0/1093898580 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f3c1a4080 con 0x7f5f3c076e50 2024-04-23T14:47:25.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.245+0000 7f5f38c00640 1 -- 172.21.15.28:0/1093898580 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f24035030 con 0x7f5f3c076e50 2024-04-23T14:47:25.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.246+0000 7f5f41a00640 1 -- 172.21.15.28:0/1093898580 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f3c1ab970 con 0x7f5f3c076e50 2024-04-23T14:47:25.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.249+0000 7f5f38c00640 1 -- 172.21.15.28:0/1093898580 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f5f24042070 con 0x7f5f3c076e50 2024-04-23T14:47:25.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.249+0000 7f5f45e6e740 1 Processor -- start 2024-04-23T14:47:25.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.249+0000 7f5f45e6e740 1 -- start start 2024-04-23T14:47:25.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.249+0000 7f5f45e6e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564b611c4270 0x564b611c4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.249+0000 7f5f45e6e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b611c4b90 0x564b611ced20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.249+0000 7f5f45e6e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b611cf260 0x564b611d1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.249+0000 7f5f45e6e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564b61129e70 con 0x564b611cf260 2024-04-23T14:47:25.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.249+0000 7f5f45e6e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564b610d0380 con 0x564b611c4b90 2024-04-23T14:47:25.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.249+0000 7f5f45e6e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x564b61125e30 con 0x564b611c4270 2024-04-23T14:47:25.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.249+0000 7f5f41000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564b611c4270 0x564b611c4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.249+0000 7f5f3be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b611c4b90 0x564b611ced20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.250+0000 7f5f3be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b611c4b90 0x564b611ced20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:43424/0 (socket says 172.21.15.28:43424) 2024-04-23T14:47:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.250+0000 7f5f41000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564b611c4270 0x564b611c4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:58326/0 (socket says 172.21.15.28:58326) 2024-04-23T14:47:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.250+0000 7f5f3be00640 1 -- 172.21.15.28:0/128679463 learned_addr learned my addr 172.21.15.28:0/128679463 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.250+0000 7f5f3b400640 1 --2- 172.21.15.28:0/128679463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b611cf260 0x564b611d1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.250+0000 7f5f3be00640 1 -- 172.21.15.28:0/128679463 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564b611c4270 msgr2=0x564b611c4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.250+0000 7f5f3be00640 1 --2- 172.21.15.28:0/128679463 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564b611c4270 0x564b611c4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.250+0000 7f5f3be00640 1 -- 172.21.15.28:0/128679463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b611cf260 msgr2=0x564b611d1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.250+0000 7f5f3be00640 1 --2- 172.21.15.28:0/128679463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b611cf260 0x564b611d1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.250+0000 7f5f3be00640 1 -- 172.21.15.28:0/128679463 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b60f047e0 con 0x564b611c4b90 2024-04-23T14:47:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.250+0000 7f5f3be00640 1 --2- 172.21.15.28:0/128679463 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b611c4b90 0x564b611ced20 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c0114f0 tx=0x7f5f2c0119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5630351a951d482a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.250+0000 7f5f31600640 1 -- 172.21.15.28:0/128679463 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c004d20 con 0x564b611c4b90 2024-04-23T14:47:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.250+0000 7f5f38c00640 1 --2- 172.21.15.28:0/1093898580 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5f200a3d20 0x7f5f200a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.250+0000 7f5f31600640 1 -- 172.21.15.28:0/128679463 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f2c004050 con 0x564b611c4b90 2024-04-23T14:47:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.250+0000 7f5f31600640 1 -- 172.21.15.28:0/128679463 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c00f7c0 con 0x564b611c4b90 2024-04-23T14:47:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.251+0000 7f5f38c00640 1 -- 172.21.15.28:0/1093898580 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f5f240f0f80 con 0x7f5f3c076e50 2024-04-23T14:47:25.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.251+0000 7f5f41000640 1 --2- 172.21.15.28:0/1093898580 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5f200a3d20 0x7f5f200a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.251+0000 7f5f45e6e740 1 -- 172.21.15.28:0/128679463 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b611c4b90 msgr2=0x564b611ced20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:25.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.251+0000 7f5f45e6e740 1 --2- 172.21.15.28:0/128679463 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b611c4b90 0x564b611ced20 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c0114f0 tx=0x7f5f2c0119b0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.251+0000 7f5f45e6e740 1 -- 172.21.15.28:0/128679463 shutdown_connections 2024-04-23T14:47:25.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.251+0000 7f5f45e6e740 1 --2- 172.21.15.28:0/128679463 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b611c4b90 0x564b611ced20 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.251+0000 7f5f45e6e740 1 --2- 172.21.15.28:0/128679463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b611cf260 0x564b611d1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.251+0000 7f5f45e6e740 1 --2- 172.21.15.28:0/128679463 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564b611c4270 0x564b611c4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.251+0000 7f5f45e6e740 1 -- 172.21.15.28:0/128679463 >> 172.21.15.28:0/128679463 conn(0x564b611c0590 msgr2=0x564b611c0970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:25.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.251+0000 7f5f45e6e740 1 -- 172.21.15.28:0/128679463 shutdown_connections 2024-04-23T14:47:25.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.252+0000 7f5f45e6e740 1 -- 172.21.15.28:0/128679463 wait complete. 2024-04-23T14:47:25.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.252+0000 7f5f41000640 1 --2- 172.21.15.28:0/1093898580 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5f200a3d20 0x7f5f200a61e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5f34005fd0 tx=0x7f5f3400d4c0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:25.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.252+0000 7f5f45e6e740 1 -- start start 2024-04-23T14:47:25.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.252+0000 7f5f45e6e740 1 Processor -- start 2024-04-23T14:47:25.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.253+0000 7f5f45e6e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b611c4270 0x564b611cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.253+0000 7f5f45e6e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b611c4b90 0x564b611cc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.253+0000 7f5f45e6e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564b611cf260 0x564b611ccbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.253+0000 7f5f45e6e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564b611d2050 con 0x564b611c4270 2024-04-23T14:47:25.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.253+0000 7f5f45e6e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564b610e3660 con 0x564b611c4b90 2024-04-23T14:47:25.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.253+0000 7f5f45e6e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x564b611d21d0 con 0x564b611cf260 2024-04-23T14:47:25.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.253+0000 7f5f45e6e740 10 client.0 _ll_register_callbacks cb 0x564b610d1aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:47:25.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.253+0000 7f5f3be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b611c4270 0x564b611cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.253+0000 7f5f3b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b611c4b90 0x564b611cc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.253+0000 7f5f3be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b611c4270 0x564b611cc160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57600/0 (socket says 172.21.15.28:57600) 2024-04-23T14:47:25.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.253+0000 7f5f3be00640 1 -- 172.21.15.28:0/3315598009 learned_addr learned my addr 172.21.15.28:0/3315598009 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:25.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.253+0000 7f5f3b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b611c4b90 0x564b611cc6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:43430/0 (socket says 172.21.15.28:43430) 2024-04-23T14:47:25.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.253+0000 7f5f41000640 1 --2- 172.21.15.28:0/3315598009 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564b611cf260 0x564b611ccbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.254+0000 7f5f3b400640 1 -- 172.21.15.28:0/3315598009 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564b611cf260 msgr2=0x564b611ccbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:25.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.254+0000 7f5f3b400640 1 --2- 172.21.15.28:0/3315598009 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564b611cf260 0x564b611ccbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.254+0000 7f5f3b400640 1 -- 172.21.15.28:0/3315598009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b611c4270 msgr2=0x564b611cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.254+0000 7f5f3b400640 1 --2- 172.21.15.28:0/3315598009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b611c4270 0x564b611cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.254+0000 7f5f3b400640 1 -- 172.21.15.28:0/3315598009 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b60f64f70 con 0x564b611c4b90 2024-04-23T14:47:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.254+0000 7f5f3b400640 1 --2- 172.21.15.28:0/3315598009 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b611c4b90 0x564b611cc6a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5f240ee680 tx=0x7f5f24005e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3136987f2a20c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.254+0000 7f5f2a000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:47:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.254+0000 7f5f2a000640 1 -- 172.21.15.28:0/3315598009 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f24049070 con 0x564b611c4b90 2024-04-23T14:47:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.254+0000 7f5f2a000640 1 -- 172.21.15.28:0/3315598009 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f5f240ee900 con 0x564b611c4b90 2024-04-23T14:47:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.254+0000 7f5f45e6e740 1 -- 172.21.15.28:0/3315598009 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f2c004050 con 0x564b611c4b90 2024-04-23T14:47:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.255+0000 7f5f2a000640 1 -- 172.21.15.28:0/3315598009 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f5f240bc830 con 0x564b611c4b90 2024-04-23T14:47:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.255+0000 7f5f45e6e740 10 client.25805 fetch_fsmap learned FSMap version 20 2024-04-23T14:47:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.255+0000 7f5f45e6e740 1 -- 172.21.15.28:0/3315598009 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x564b60dfc170 con 0x564b611c4b90 2024-04-23T14:47:25.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.255+0000 7f5f2a000640 1 -- 172.21.15.28:0/3315598009 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f2c004050 con 0x564b611c4b90 2024-04-23T14:47:25.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.255+0000 7f5f2a000640 1 -- 172.21.15.28:0/3315598009 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f5f240c56b0 con 0x564b611c4b90 2024-04-23T14:47:25.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.255+0000 7f5f45e6e740 10 client.25805 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:47:25.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.255+0000 7f5f45e6e740 10 client.25805 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:47:25.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.255+0000 7f5f45e6e740 20 client.25805 populate_metadata read hostname 'smithi028' 2024-04-23T14:47:25.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.255+0000 7f5f45e6e740 1 --2- 172.21.15.28:0/3315598009 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564b611df4c0 0x564b611e1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.255+0000 7f5f45e6e740 4 client.25805 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:47:25.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.255+0000 7f5f45e6e740 1 -- 172.21.15.28:0/3315598009 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564b61137890 con 0x564b611df4c0 2024-04-23T14:47:25.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.255+0000 7f5f3be00640 1 --2- 172.21.15.28:0/3315598009 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564b611df4c0 0x564b611e1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.256+0000 7f5f3be00640 1 --2- 172.21.15.28:0/3315598009 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564b611df4c0 0x564b611e1930 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=926dec7113c68d68 server_cookie=4ea939b65574698a in_seq=0 out_seq=0 2024-04-23T14:47:25.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.256+0000 7f5f2a000640 10 client.25805 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:47:25.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.257+0000 7f5f2a000640 1 -- 172.21.15.28:0/3315598009 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564b61137890 con 0x564b611df4c0 2024-04-23T14:47:25.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.257+0000 7f5f2a000640 10 client.25805 handle_command_reply: tid=0 2024-04-23T14:47:25.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.257+0000 7f5f45e6e740 1 client.25805 shutdown 2024-04-23T14:47:25.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.257+0000 7f5f45e6e740 1 -- 172.21.15.28:0/3315598009 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b611c4b90 msgr2=0x564b611cc6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:25.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.257+0000 7f5f45e6e740 1 --2- 172.21.15.28:0/3315598009 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b611c4b90 0x564b611cc6a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5f240ee680 tx=0x7f5f24005e50 comp rx=0 tx=0).stop 2024-04-23T14:47:25.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.257+0000 7f5f45e6e740 1 -- 172.21.15.28:0/3315598009 shutdown_connections 2024-04-23T14:47:25.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.257+0000 7f5f45e6e740 1 --2- 172.21.15.28:0/3315598009 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564b611df4c0 0x564b611e1930 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.257+0000 7f5f45e6e740 1 --2- 172.21.15.28:0/3315598009 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b611c4b90 0x564b611cc6a0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.257+0000 7f5f2a000640 0 client.25805 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:47:25.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.257+0000 7f5f45e6e740 1 --2- 172.21.15.28:0/3315598009 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564b611cf260 0x564b611ccbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.257+0000 7f5f45e6e740 1 --2- 172.21.15.28:0/3315598009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b611c4270 0x564b611cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.258+0000 7f5f45e6e740 1 -- 172.21.15.28:0/3315598009 >> 172.21.15.28:0/3315598009 conn(0x564b611c0590 msgr2=0x564b611c6390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:25.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.258+0000 7f5f45e6e740 1 -- 172.21.15.28:0/3315598009 shutdown_connections 2024-04-23T14:47:25.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.258+0000 7f5f45e6e740 1 -- 172.21.15.28:0/3315598009 wait complete. 2024-04-23T14:47:25.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.258+0000 7f5f45e6e740 20 client.25805 trim_cache size 0 max 16384 2024-04-23T14:47:25.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.279+0000 7f5f45e6e740 1 Processor -- start 2024-04-23T14:47:25.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.279+0000 7f5f45e6e740 1 -- start start 2024-04-23T14:47:25.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.280+0000 7f5f45e6e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b611c4270 0x564b611daf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.280+0000 7f5f45e6e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564b611c4b90 0x564b611c6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.280+0000 7f5f45e6e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b611cf260 0x564b611e7240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.280+0000 7f5f45e6e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3c1b2110 con 0x564b611cf260 2024-04-23T14:47:25.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.280+0000 7f5f45e6e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3c1b4990 con 0x564b611c4270 2024-04-23T14:47:25.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.280+0000 7f5f45e6e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3c1b2780 con 0x564b611c4b90 2024-04-23T14:47:25.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.280+0000 7f5f41000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b611cf260 0x564b611e7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.280+0000 7f5f3b400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564b611c4b90 0x564b611c6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.280+0000 7f5f3b400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564b611c4b90 0x564b611c6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:58348/0 (socket says 172.21.15.28:58348) 2024-04-23T14:47:25.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.280+0000 7f5f3b400640 1 -- 172.21.15.28:0/700130662 learned_addr learned my addr 172.21.15.28:0/700130662 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:25.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.280+0000 7f5f41000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b611cf260 0x564b611e7240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57606/0 (socket says 172.21.15.28:57606) 2024-04-23T14:47:25.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.280+0000 7f5f3be00640 1 --2- 172.21.15.28:0/700130662 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b611c4270 0x564b611daf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.280+0000 7f5f3b400640 1 -- 172.21.15.28:0/700130662 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b611c4270 msgr2=0x564b611daf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:25.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.280+0000 7f5f3b400640 1 --2- 172.21.15.28:0/700130662 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b611c4270 0x564b611daf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.280+0000 7f5f3b400640 1 -- 172.21.15.28:0/700130662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b611cf260 msgr2=0x564b611e7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:25.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.280+0000 7f5f3b400640 1 --2- 172.21.15.28:0/700130662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b611cf260 0x564b611e7240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.280+0000 7f5f3b400640 1 -- 172.21.15.28:0/700130662 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b611cb110 con 0x564b611c4b90 2024-04-23T14:47:25.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.280+0000 7f5f3be00640 1 --2- 172.21.15.28:0/700130662 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b611c4270 0x564b611daf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:47:25.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.281+0000 7f5f3b400640 1 --2- 172.21.15.28:0/700130662 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564b611c4b90 0x564b611c6a10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5f240bce60 tx=0x7f5f2402a420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5791388d9fec21d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:25.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.281+0000 7f5f30c00640 1 -- 172.21.15.28:0/700130662 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f24049040 con 0x564b611c4b90 2024-04-23T14:47:25.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.281+0000 7f5f30c00640 1 -- 172.21.15.28:0/700130662 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f240bd5d0 con 0x564b611c4b90 2024-04-23T14:47:25.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.281+0000 7f5f30c00640 1 -- 172.21.15.28:0/700130662 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f240d6150 con 0x564b611c4b90 2024-04-23T14:47:25.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.282+0000 7f5f45e6e740 1 -- 172.21.15.28:0/700130662 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564b611c4b90 msgr2=0x564b611c6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:25.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.282+0000 7f5f45e6e740 1 --2- 172.21.15.28:0/700130662 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564b611c4b90 0x564b611c6a10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5f240bce60 tx=0x7f5f2402a420 comp rx=0 tx=0).stop 2024-04-23T14:47:25.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.282+0000 7f5f45e6e740 1 -- 172.21.15.28:0/700130662 shutdown_connections 2024-04-23T14:47:25.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.282+0000 7f5f45e6e740 1 --2- 172.21.15.28:0/700130662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b611cf260 0x564b611e7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.282+0000 7f5f45e6e740 1 --2- 172.21.15.28:0/700130662 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564b611c4b90 0x564b611c6a10 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.282+0000 7f5f45e6e740 1 --2- 172.21.15.28:0/700130662 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b611c4270 0x564b611daf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.282+0000 7f5f45e6e740 1 -- 172.21.15.28:0/700130662 >> 172.21.15.28:0/700130662 conn(0x564b611c0590 msgr2=0x564b611cc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:25.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.282+0000 7f5f45e6e740 1 -- 172.21.15.28:0/700130662 shutdown_connections 2024-04-23T14:47:25.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.282+0000 7f5f45e6e740 1 -- 172.21.15.28:0/700130662 wait complete. 2024-04-23T14:47:25.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.282+0000 7f5f45e6e740 1 -- start start 2024-04-23T14:47:25.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.283+0000 7f5f45e6e740 1 Processor -- start 2024-04-23T14:47:25.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.283+0000 7f5f45e6e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b611c4270 0x564b611d75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.283+0000 7f5f45e6e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b611c4b90 0x564b611d3970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.283+0000 7f5f45e6e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564b611cf260 0x564b611d3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.283+0000 7f5f45e6e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3c1b2e50 con 0x564b611c4270 2024-04-23T14:47:25.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.283+0000 7f5f45e6e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3c1b42c0 con 0x564b611c4b90 2024-04-23T14:47:25.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.283+0000 7f5f45e6e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3c1b3520 con 0x564b611cf260 2024-04-23T14:47:25.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.283+0000 7f5f45e6e740 10 client.0 _ll_register_callbacks cb 0x7f5f240f4560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:47:25.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.283+0000 7f5f3be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b611c4270 0x564b611d75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.283+0000 7f5f3b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b611c4b90 0x564b611d3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.283+0000 7f5f3be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b611c4270 0x564b611d75f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57612/0 (socket says 172.21.15.28:57612) 2024-04-23T14:47:25.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.283+0000 7f5f3be00640 1 -- 172.21.15.28:0/1992695738 learned_addr learned my addr 172.21.15.28:0/1992695738 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:25.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.284+0000 7f5f41000640 1 --2- 172.21.15.28:0/1992695738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564b611cf260 0x564b611d3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.284+0000 7f5f3b400640 1 -- 172.21.15.28:0/1992695738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564b611cf260 msgr2=0x564b611d3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:25.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.284+0000 7f5f3b400640 1 --2- 172.21.15.28:0/1992695738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564b611cf260 0x564b611d3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.284+0000 7f5f3b400640 1 -- 172.21.15.28:0/1992695738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b611c4270 msgr2=0x564b611d75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:25.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.284+0000 7f5f3b400640 1 --2- 172.21.15.28:0/1992695738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b611c4270 0x564b611d75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.284+0000 7f5f3b400640 1 -- 172.21.15.28:0/1992695738 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b611d4470 con 0x564b611c4b90 2024-04-23T14:47:25.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.284+0000 7f5f3be00640 1 --2- 172.21.15.28:0/1992695738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b611c4270 0x564b611d75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:47:25.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.284+0000 7f5f41000640 1 --2- 172.21.15.28:0/1992695738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564b611cf260 0x564b611d3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:47:25.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.284+0000 7f5f3b400640 1 --2- 172.21.15.28:0/1992695738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b611c4b90 0x564b611d3970 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5f240f5000 tx=0x7f5f24009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec27d7ef27077c75 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:25.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.284+0000 7f5f2a000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:47:25.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.284+0000 7f5f2a000640 1 -- 172.21.15.28:0/1992695738 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f240bcca0 con 0x564b611c4b90 2024-04-23T14:47:25.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.284+0000 7f5f2a000640 1 -- 172.21.15.28:0/1992695738 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f5f240d61a0 con 0x564b611c4b90 2024-04-23T14:47:25.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.284+0000 7f5f45e6e740 1 -- 172.21.15.28:0/1992695738 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f240bd5d0 con 0x564b611c4b90 2024-04-23T14:47:25.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.285+0000 7f5f2a000640 1 -- 172.21.15.28:0/1992695738 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f5f240d63e0 con 0x564b611c4b90 2024-04-23T14:47:25.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.285+0000 7f5f45e6e740 10 client.25811 fetch_fsmap learned FSMap version 20 2024-04-23T14:47:25.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.285+0000 7f5f45e6e740 1 -- 172.21.15.28:0/1992695738 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x564b611d4470 con 0x564b611c4b90 2024-04-23T14:47:25.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.285+0000 7f5f2a000640 1 -- 172.21.15.28:0/1992695738 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f240bd5d0 con 0x564b611c4b90 2024-04-23T14:47:25.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.285+0000 7f5f2a000640 1 -- 172.21.15.28:0/1992695738 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f5f240bc3b0 con 0x564b611c4b90 2024-04-23T14:47:25.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.285+0000 7f5f45e6e740 10 client.25811 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:47:25.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.285+0000 7f5f45e6e740 10 client.25811 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:47:25.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.285+0000 7f5f45e6e740 20 client.25811 populate_metadata read hostname 'smithi028' 2024-04-23T14:47:25.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.285+0000 7f5f45e6e740 1 --2- 172.21.15.28:0/1992695738 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564b61246c00 0x564b61249020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.285+0000 7f5f45e6e740 4 client.25811 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-23T14:47:25.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.285+0000 7f5f45e6e740 1 -- 172.21.15.28:0/1992695738 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564b61131700 con 0x564b61246c00 2024-04-23T14:47:25.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.286+0000 7f5f3be00640 1 --2- 172.21.15.28:0/1992695738 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564b61246c00 0x564b61249020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.286+0000 7f5f3be00640 1 --2- 172.21.15.28:0/1992695738 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564b61246c00 0x564b61249020 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea06d325a408fb87 server_cookie=626b2766482442ae in_seq=0 out_seq=0 2024-04-23T14:47:25.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.286+0000 7f5f2a000640 10 client.25811 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:47:25.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.300+0000 7f5f2a000640 1 -- 172.21.15.28:0/1992695738 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564b61131700 con 0x564b61246c00 2024-04-23T14:47:25.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.300+0000 7f5f2a000640 10 client.25811 handle_command_reply: tid=0 2024-04-23T14:47:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.300+0000 7f5f45e6e740 1 client.25811 shutdown 2024-04-23T14:47:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.300+0000 7f5f45e6e740 1 -- 172.21.15.28:0/1992695738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b611c4b90 msgr2=0x564b611d3970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.300+0000 7f5f45e6e740 1 --2- 172.21.15.28:0/1992695738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b611c4b90 0x564b611d3970 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5f240f5000 tx=0x7f5f24009260 comp rx=0 tx=0).stop 2024-04-23T14:47:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.300+0000 7f5f45e6e740 1 -- 172.21.15.28:0/1992695738 shutdown_connections 2024-04-23T14:47:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.300+0000 7f5f45e6e740 1 --2- 172.21.15.28:0/1992695738 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564b61246c00 0x564b61249020 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.300+0000 7f5f45e6e740 1 --2- 172.21.15.28:0/1992695738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b611c4b90 0x564b611d3970 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.300+0000 7f5f45e6e740 1 --2- 172.21.15.28:0/1992695738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564b611cf260 0x564b611d3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.300+0000 7f5f45e6e740 1 --2- 172.21.15.28:0/1992695738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b611c4270 0x564b611d75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.300+0000 7f5f2a000640 0 client.25811 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:47:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.300+0000 7f5f45e6e740 1 -- 172.21.15.28:0/1992695738 >> 172.21.15.28:0/1992695738 conn(0x564b611c0590 msgr2=0x564b611bee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.300+0000 7f5f45e6e740 1 -- 172.21.15.28:0/1992695738 shutdown_connections 2024-04-23T14:47:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.300+0000 7f5f45e6e740 1 -- 172.21.15.28:0/1992695738 wait complete. 2024-04-23T14:47:25.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.300+0000 7f5f45e6e740 20 client.25811 trim_cache size 0 max 16384 2024-04-23T14:47:25.303 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:47:25.303 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-23T14:47:25.303 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "ab069c6b-1232-40ff-a0b2-bb61138dcf1b", 2024-04-23T14:47:25.303 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-23T14:47:25.303 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:47:25.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.301+0000 7f5f41a00640 1 -- 172.21.15.28:0/1093898580 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5f200a3d20 msgr2=0x7f5f200a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:25.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.301+0000 7f5f41a00640 1 --2- 172.21.15.28:0/1093898580 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5f200a3d20 0x7f5f200a61e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5f34005fd0 tx=0x7f5f3400d4c0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.301+0000 7f5f41a00640 1 -- 172.21.15.28:0/1093898580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f3c076e50 msgr2=0x7f5f3c19f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:25.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.301+0000 7f5f41a00640 1 --2- 172.21.15.28:0/1093898580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f3c076e50 0x7f5f3c19f340 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5f2402a940 tx=0x7f5f24003b90 comp rx=0 tx=0).stop 2024-04-23T14:47:25.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.301+0000 7f5f41a00640 1 -- 172.21.15.28:0/1093898580 shutdown_connections 2024-04-23T14:47:25.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.301+0000 7f5f41a00640 1 --2- 172.21.15.28:0/1093898580 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5f200a3d20 0x7f5f200a61e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.301+0000 7f5f41a00640 1 --2- 172.21.15.28:0/1093898580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f3c076e50 0x7f5f3c19f340 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.301+0000 7f5f41a00640 1 --2- 172.21.15.28:0/1093898580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5f3c112d40 0x7f5f3c1a36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.301+0000 7f5f41a00640 1 --2- 172.21.15.28:0/1093898580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f3c076530 0x7f5f3c19ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.301+0000 7f5f41a00640 1 -- 172.21.15.28:0/1093898580 >> 172.21.15.28:0/1093898580 conn(0x7f5f3c0ffb10 msgr2=0x7f5f3c114360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:25.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.301+0000 7f5f41a00640 1 -- 172.21.15.28:0/1093898580 shutdown_connections 2024-04-23T14:47:25.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.301+0000 7f5f41a00640 1 -- 172.21.15.28:0/1093898580 wait complete. 2024-04-23T14:47:25.321 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:47:25.321 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:47:25.471 INFO:tasks.workunit.client.0.smithi028.stdout: 1 196077 10.63 MB/sec warmup 101 sec latency 13.673 ms 2024-04-23T14:47:25.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.540+0000 7f04e7200640 1 -- 172.21.15.28:0/4099894136 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04e8102ad0 msgr2=0x7f04e8106b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:25.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.540+0000 7f04e7200640 1 --2- 172.21.15.28:0/4099894136 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04e8102ad0 0x7f04e8106b10 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f04d4009860 tx=0x7f04d402a340 comp rx=0 tx=0).stop 2024-04-23T14:47:25.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.540+0000 7f04e7200640 1 -- 172.21.15.28:0/4099894136 shutdown_connections 2024-04-23T14:47:25.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.540+0000 7f04e7200640 1 --2- 172.21.15.28:0/4099894136 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04e8102ad0 0x7f04e8106b10 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.540+0000 7f04e7200640 1 --2- 172.21.15.28:0/4099894136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04e81071f0 0x7f04e810edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.540+0000 7f04e7200640 1 --2- 172.21.15.28:0/4099894136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04e8102120 0x7f04e8102500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.540+0000 7f04e7200640 1 -- 172.21.15.28:0/4099894136 >> 172.21.15.28:0/4099894136 conn(0x7f04e80fd9a0 msgr2=0x7f04e80ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:25.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.540+0000 7f04e7200640 1 -- 172.21.15.28:0/4099894136 shutdown_connections 2024-04-23T14:47:25.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.540+0000 7f04e7200640 1 -- 172.21.15.28:0/4099894136 wait complete. 2024-04-23T14:47:25.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.541+0000 7f04e7200640 1 Processor -- start 2024-04-23T14:47:25.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.541+0000 7f04e7200640 1 -- start start 2024-04-23T14:47:25.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.542+0000 7f04e7200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04e8102120 0x7f04e81a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.542+0000 7f04e7200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04e81071f0 0x7f04e81a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.542+0000 7f04e7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04e819a840 0x7f04e819acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.542+0000 7f04e7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f04e8112260 con 0x7f04e819a840 2024-04-23T14:47:25.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.542+0000 7f04e7200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f04e81120e0 con 0x7f04e8102120 2024-04-23T14:47:25.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.542+0000 7f04e7200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f04e81123e0 con 0x7f04e81071f0 2024-04-23T14:47:25.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.542+0000 7f04e5400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04e81071f0 0x7f04e81a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.542+0000 7f04e6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04e819a840 0x7f04e819acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.542+0000 7f04e5400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04e81071f0 0x7f04e81a0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:58372/0 (socket says 172.21.15.28:58372) 2024-04-23T14:47:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.542+0000 7f04e5400640 1 -- 172.21.15.28:0/3334966513 learned_addr learned my addr 172.21.15.28:0/3334966513 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.542+0000 7f04e5e00640 1 --2- 172.21.15.28:0/3334966513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04e8102120 0x7f04e81a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.543+0000 7f04e5400640 1 -- 172.21.15.28:0/3334966513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04e8102120 msgr2=0x7f04e81a06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.543+0000 7f04e5400640 1 --2- 172.21.15.28:0/3334966513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04e8102120 0x7f04e81a06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.543+0000 7f04e5400640 1 -- 172.21.15.28:0/3334966513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04e819a840 msgr2=0x7f04e819acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.543+0000 7f04e5400640 1 --2- 172.21.15.28:0/3334966513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04e819a840 0x7f04e819acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.543+0000 7f04e5400640 1 -- 172.21.15.28:0/3334966513 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04e819b3d0 con 0x7f04e81071f0 2024-04-23T14:47:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.543+0000 7f04e5e00640 1 --2- 172.21.15.28:0/3334966513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04e8102120 0x7f04e81a06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:47:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.543+0000 7f04e5400640 1 --2- 172.21.15.28:0/3334966513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04e81071f0 0x7f04e81a0c10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f04d4035040 tx=0x7f04d4033d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:25.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.543+0000 7f04dea00640 1 -- 172.21.15.28:0/3334966513 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d4037040 con 0x7f04e81071f0 2024-04-23T14:47:25.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.543+0000 7f04dea00640 1 -- 172.21.15.28:0/3334966513 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04d4040070 con 0x7f04e81071f0 2024-04-23T14:47:25.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.543+0000 7f04dea00640 1 -- 172.21.15.28:0/3334966513 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d40039b0 con 0x7f04e81071f0 2024-04-23T14:47:25.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.543+0000 7f04e7200640 1 -- 172.21.15.28:0/3334966513 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04e819b660 con 0x7f04e81071f0 2024-04-23T14:47:25.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.544+0000 7f04e7200640 1 -- 172.21.15.28:0/3334966513 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04e81a7840 con 0x7f04e81071f0 2024-04-23T14:47:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.547+0000 7f04efb9c740 1 Processor -- start 2024-04-23T14:47:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.547+0000 7f04dea00640 1 -- 172.21.15.28:0/3334966513 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f04d4003b50 con 0x7f04e81071f0 2024-04-23T14:47:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.547+0000 7f04efb9c740 1 -- start start 2024-04-23T14:47:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.547+0000 7f04efb9c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5632dc83ca30 0x5632dc83ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.547+0000 7f04efb9c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632dc83d350 0x5632dc8474e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.547+0000 7f04efb9c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632dc847a20 0x5632dc849e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.548+0000 7f04efb9c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5632dc7a21f0 con 0x5632dc83d350 2024-04-23T14:47:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.548+0000 7f04efb9c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5632dc748350 con 0x5632dc83ca30 2024-04-23T14:47:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.548+0000 7f04efb9c740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5632dc79e1b0 con 0x5632dc847a20 2024-04-23T14:47:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.548+0000 7f04e6800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5632dc83ca30 0x5632dc83ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.548+0000 7f04e6800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5632dc83ca30 0x5632dc83ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:43482/0 (socket says 172.21.15.28:43482) 2024-04-23T14:47:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.548+0000 7f04e6800640 1 -- 172.21.15.28:0/1740236346 learned_addr learned my addr 172.21.15.28:0/1740236346 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.548+0000 7f04e5400640 1 --2- 172.21.15.28:0/1740236346 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632dc847a20 0x5632dc849e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.548+0000 7f04e5e00640 1 --2- 172.21.15.28:0/1740236346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632dc83d350 0x5632dc8474e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.548+0000 7f04e6800640 1 -- 172.21.15.28:0/1740236346 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632dc847a20 msgr2=0x5632dc849e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.548+0000 7f04e6800640 1 --2- 172.21.15.28:0/1740236346 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632dc847a20 0x5632dc849e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.548+0000 7f04e6800640 1 -- 172.21.15.28:0/1740236346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632dc83d350 msgr2=0x5632dc8474e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.548+0000 7f04e6800640 1 --2- 172.21.15.28:0/1740236346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632dc83d350 0x5632dc8474e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.548+0000 7f04e6800640 1 -- 172.21.15.28:0/1740236346 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632dc79cf50 con 0x5632dc83ca30 2024-04-23T14:47:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.549+0000 7f04e6800640 1 --2- 172.21.15.28:0/1740236346 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5632dc83ca30 0x5632dc83ce10 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f04e0009f70 tx=0x7f04e0010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcf41eff0a714850 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.549+0000 7f04d3400640 1 -- 172.21.15.28:0/1740236346 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e0016070 con 0x5632dc83ca30 2024-04-23T14:47:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.549+0000 7f04d3400640 1 -- 172.21.15.28:0/1740236346 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04e0004d20 con 0x5632dc83ca30 2024-04-23T14:47:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.549+0000 7f04d3400640 1 -- 172.21.15.28:0/1740236346 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e0012950 con 0x5632dc83ca30 2024-04-23T14:47:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.549+0000 7f04dea00640 1 --2- 172.21.15.28:0/3334966513 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f04cc0a3da0 0x7f04cc0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.549+0000 7f04dea00640 1 -- 172.21.15.28:0/3334966513 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f04d40f07b0 con 0x7f04e81071f0 2024-04-23T14:47:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.549+0000 7f04e5e00640 1 --2- 172.21.15.28:0/3334966513 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f04cc0a3da0 0x7f04cc0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.550+0000 7f04efb9c740 1 -- 172.21.15.28:0/1740236346 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5632dc83ca30 msgr2=0x5632dc83ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:25.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.550+0000 7f04efb9c740 1 --2- 172.21.15.28:0/1740236346 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5632dc83ca30 0x5632dc83ce10 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f04e0009f70 tx=0x7f04e0010800 comp rx=0 tx=0).stop 2024-04-23T14:47:25.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.550+0000 7f04efb9c740 1 -- 172.21.15.28:0/1740236346 shutdown_connections 2024-04-23T14:47:25.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.550+0000 7f04efb9c740 1 --2- 172.21.15.28:0/1740236346 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632dc847a20 0x5632dc849e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.550+0000 7f04efb9c740 1 --2- 172.21.15.28:0/1740236346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632dc83d350 0x5632dc8474e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.550+0000 7f04efb9c740 1 --2- 172.21.15.28:0/1740236346 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5632dc83ca30 0x5632dc83ce10 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.550+0000 7f04efb9c740 1 -- 172.21.15.28:0/1740236346 >> 172.21.15.28:0/1740236346 conn(0x5632dc838d50 msgr2=0x5632dc839130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:25.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.550+0000 7f04efb9c740 1 -- 172.21.15.28:0/1740236346 shutdown_connections 2024-04-23T14:47:25.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.550+0000 7f04efb9c740 1 -- 172.21.15.28:0/1740236346 wait complete. 2024-04-23T14:47:25.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.550+0000 7f04e5e00640 1 --2- 172.21.15.28:0/3334966513 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f04cc0a3da0 0x7f04cc0a6260 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x5632dc83dff0 tx=0x7f04d8009260 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:25.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.550+0000 7f04efb9c740 1 -- start start 2024-04-23T14:47:25.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.551+0000 7f04efb9c740 1 Processor -- start 2024-04-23T14:47:25.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.551+0000 7f04efb9c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632dc83ca30 0x5632dc844a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.551+0000 7f04efb9c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5632dc83d350 0x5632dc844f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.551+0000 7f04efb9c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632dc847a20 0x5632dc8454b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.551+0000 7f04efb9c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5632dc84a810 con 0x5632dc847a20 2024-04-23T14:47:25.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.551+0000 7f04efb9c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5632dc75b690 con 0x5632dc83d350 2024-04-23T14:47:25.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.551+0000 7f04efb9c740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5632dc84a990 con 0x5632dc83ca30 2024-04-23T14:47:25.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.551+0000 7f04efb9c740 10 client.0 _ll_register_callbacks cb 0x5632dc797650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:47:25.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.551+0000 7f04e5e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632dc83ca30 0x5632dc844a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.552+0000 7f04e5400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5632dc83d350 0x5632dc844f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.552+0000 7f04e6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632dc847a20 0x5632dc8454b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.552+0000 7f04e5e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632dc83ca30 0x5632dc844a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:58396/0 (socket says 172.21.15.28:58396) 2024-04-23T14:47:25.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.552+0000 7f04e5e00640 1 -- 172.21.15.28:0/3490949576 learned_addr learned my addr 172.21.15.28:0/3490949576 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:25.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.552+0000 7f04e5400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5632dc83d350 0x5632dc844f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:43486/0 (socket says 172.21.15.28:43486) 2024-04-23T14:47:25.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.552+0000 7f04e5400640 1 -- 172.21.15.28:0/3490949576 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632dc83ca30 msgr2=0x5632dc844a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:25.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.552+0000 7f04e5400640 1 --2- 172.21.15.28:0/3490949576 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632dc83ca30 0x5632dc844a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.552+0000 7f04e5400640 1 -- 172.21.15.28:0/3490949576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632dc847a20 msgr2=0x5632dc8454b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:25.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.552+0000 7f04e5400640 1 --2- 172.21.15.28:0/3490949576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632dc847a20 0x5632dc8454b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.552+0000 7f04e5400640 1 -- 172.21.15.28:0/3490949576 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632dc75a920 con 0x5632dc83d350 2024-04-23T14:47:25.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.552+0000 7f04e5400640 1 --2- 172.21.15.28:0/3490949576 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5632dc83d350 0x5632dc844f70 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f04d40edfc0 tx=0x7f04d40b84b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4299aec564dc9edb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:25.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.552+0000 7f04c7e00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:47:25.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.552+0000 7f04c7e00640 1 -- 172.21.15.28:0/3490949576 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d40bc350 con 0x5632dc83d350 2024-04-23T14:47:25.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.553+0000 7f04c7e00640 1 -- 172.21.15.28:0/3490949576 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f04d40ee130 con 0x5632dc83d350 2024-04-23T14:47:25.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.553+0000 7f04efb9c740 1 -- 172.21.15.28:0/3490949576 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04e0004d20 con 0x5632dc83d350 2024-04-23T14:47:25.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.553+0000 7f04c7e00640 1 -- 172.21.15.28:0/3490949576 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f04d40bc170 con 0x5632dc83d350 2024-04-23T14:47:25.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.553+0000 7f04efb9c740 10 client.25823 fetch_fsmap learned FSMap version 20 2024-04-23T14:47:25.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.553+0000 7f04efb9c740 1 -- 172.21.15.28:0/3490949576 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5632dc57e7d0 con 0x5632dc83d350 2024-04-23T14:47:25.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.553+0000 7f04c7e00640 1 -- 172.21.15.28:0/3490949576 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04e0004d20 con 0x5632dc83d350 2024-04-23T14:47:25.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.553+0000 7f04c7e00640 1 -- 172.21.15.28:0/3490949576 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f04d40f2ce0 con 0x5632dc83d350 2024-04-23T14:47:25.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.553+0000 7f04efb9c740 10 client.25823 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:47:25.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.553+0000 7f04efb9c740 10 client.25823 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:47:25.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.553+0000 7f04efb9c740 20 client.25823 populate_metadata read hostname 'smithi028' 2024-04-23T14:47:25.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.553+0000 7f04efb9c740 1 --2- 172.21.15.28:0/3490949576 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5632dc857e70 0x5632dc85a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.553+0000 7f04efb9c740 4 client.25823 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:47:25.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.553+0000 7f04efb9c740 1 -- 172.21.15.28:0/3490949576 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5632dc7afc10 con 0x5632dc857e70 2024-04-23T14:47:25.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.554+0000 7f04e5e00640 1 --2- 172.21.15.28:0/3490949576 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5632dc857e70 0x5632dc85a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.554+0000 7f04c7e00640 10 client.25823 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:47:25.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.554+0000 7f04e5e00640 1 --2- 172.21.15.28:0/3490949576 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5632dc857e70 0x5632dc85a2d0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55a12879b84d7fb3 server_cookie=95055d9a52758290 in_seq=0 out_seq=0 2024-04-23T14:47:25.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.555+0000 7f04c7e00640 1 -- 172.21.15.28:0/3490949576 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5632dc7afc10 con 0x5632dc857e70 2024-04-23T14:47:25.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.555+0000 7f04c7e00640 10 client.25823 handle_command_reply: tid=0 2024-04-23T14:47:25.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.555+0000 7f04efb9c740 1 client.25823 shutdown 2024-04-23T14:47:25.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.556+0000 7f04efb9c740 1 -- 172.21.15.28:0/3490949576 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5632dc83d350 msgr2=0x5632dc844f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:25.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.556+0000 7f04efb9c740 1 --2- 172.21.15.28:0/3490949576 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5632dc83d350 0x5632dc844f70 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f04d40edfc0 tx=0x7f04d40b84b0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.556+0000 7f04efb9c740 1 -- 172.21.15.28:0/3490949576 shutdown_connections 2024-04-23T14:47:25.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.556+0000 7f04efb9c740 1 --2- 172.21.15.28:0/3490949576 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5632dc857e70 0x5632dc85a2d0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.556+0000 7f04efb9c740 1 --2- 172.21.15.28:0/3490949576 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5632dc83d350 0x5632dc844f70 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.556+0000 7f04efb9c740 1 --2- 172.21.15.28:0/3490949576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632dc847a20 0x5632dc8454b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.556+0000 7f04efb9c740 1 --2- 172.21.15.28:0/3490949576 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632dc83ca30 0x5632dc844a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.556+0000 7f04c7e00640 0 client.25823 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:47:25.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.556+0000 7f04efb9c740 1 -- 172.21.15.28:0/3490949576 >> 172.21.15.28:0/3490949576 conn(0x5632dc838d50 msgr2=0x5632dc837fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:25.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.556+0000 7f04efb9c740 1 -- 172.21.15.28:0/3490949576 shutdown_connections 2024-04-23T14:47:25.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.556+0000 7f04efb9c740 1 -- 172.21.15.28:0/3490949576 wait complete. 2024-04-23T14:47:25.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.556+0000 7f04efb9c740 20 client.25823 trim_cache size 0 max 16384 2024-04-23T14:47:25.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.577+0000 7f04efb9c740 1 Processor -- start 2024-04-23T14:47:25.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.577+0000 7f04efb9c740 1 -- start start 2024-04-23T14:47:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.578+0000 7f04efb9c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5632dc83ca30 0x5632dc838750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.578+0000 7f04efb9c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632dc83d350 0x5632dc8536d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.578+0000 7f04efb9c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632dc847a20 0x5632dc83e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.578+0000 7f04efb9c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f04e81add30 con 0x5632dc847a20 2024-04-23T14:47:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.578+0000 7f04efb9c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f04e81b05b0 con 0x5632dc83ca30 2024-04-23T14:47:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.578+0000 7f04efb9c740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f04e81ae3a0 con 0x5632dc83d350 2024-04-23T14:47:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.578+0000 7f04e5e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5632dc83ca30 0x5632dc838750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.578+0000 7f04e5e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5632dc83ca30 0x5632dc838750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:43500/0 (socket says 172.21.15.28:43500) 2024-04-23T14:47:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.578+0000 7f04e5e00640 1 -- 172.21.15.28:0/3878387258 learned_addr learned my addr 172.21.15.28:0/3878387258 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.578+0000 7f04e5400640 1 --2- 172.21.15.28:0/3878387258 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632dc83d350 0x5632dc8536d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.578+0000 7f04e6800640 1 --2- 172.21.15.28:0/3878387258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632dc847a20 0x5632dc83e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.578+0000 7f04e5e00640 1 -- 172.21.15.28:0/3878387258 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632dc83d350 msgr2=0x5632dc8536d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:25.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.578+0000 7f04e5e00640 1 --2- 172.21.15.28:0/3878387258 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632dc83d350 0x5632dc8536d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.578+0000 7f04e5e00640 1 -- 172.21.15.28:0/3878387258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632dc847a20 msgr2=0x5632dc83e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:25.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.578+0000 7f04e5e00640 1 --2- 172.21.15.28:0/3878387258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632dc847a20 0x5632dc83e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.578+0000 7f04e5e00640 1 -- 172.21.15.28:0/3878387258 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632dc4b43f0 con 0x5632dc83ca30 2024-04-23T14:47:25.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.578+0000 7f04e5400640 1 --2- 172.21.15.28:0/3878387258 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632dc83d350 0x5632dc8536d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:47:25.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.579+0000 7f04e5e00640 1 --2- 172.21.15.28:0/3878387258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5632dc83ca30 0x5632dc838750 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f04d8004630 tx=0x7f04d801b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78090fa1d5a02e33 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:25.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.579+0000 7f04d2a00640 1 -- 172.21.15.28:0/3878387258 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d802a4f0 con 0x5632dc83ca30 2024-04-23T14:47:25.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.579+0000 7f04d2a00640 1 -- 172.21.15.28:0/3878387258 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04d802f070 con 0x5632dc83ca30 2024-04-23T14:47:25.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.579+0000 7f04d2a00640 1 -- 172.21.15.28:0/3878387258 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d801e650 con 0x5632dc83ca30 2024-04-23T14:47:25.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.580+0000 7f04efb9c740 1 -- 172.21.15.28:0/3878387258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5632dc83ca30 msgr2=0x5632dc838750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:25.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.580+0000 7f04efb9c740 1 --2- 172.21.15.28:0/3878387258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5632dc83ca30 0x5632dc838750 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f04d8004630 tx=0x7f04d801b000 comp rx=0 tx=0).stop 2024-04-23T14:47:25.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.580+0000 7f04efb9c740 1 -- 172.21.15.28:0/3878387258 shutdown_connections 2024-04-23T14:47:25.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.580+0000 7f04efb9c740 1 --2- 172.21.15.28:0/3878387258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632dc847a20 0x5632dc83e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.580+0000 7f04efb9c740 1 --2- 172.21.15.28:0/3878387258 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632dc83d350 0x5632dc8536d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.580+0000 7f04efb9c740 1 --2- 172.21.15.28:0/3878387258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5632dc83ca30 0x5632dc838750 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.580+0000 7f04efb9c740 1 -- 172.21.15.28:0/3878387258 >> 172.21.15.28:0/3878387258 conn(0x5632dc838d50 msgr2=0x5632dc83e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:25.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.580+0000 7f04efb9c740 1 -- 172.21.15.28:0/3878387258 shutdown_connections 2024-04-23T14:47:25.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.580+0000 7f04efb9c740 1 -- 172.21.15.28:0/3878387258 wait complete. 2024-04-23T14:47:25.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.580+0000 7f04efb9c740 1 -- start start 2024-04-23T14:47:25.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.580+0000 7f04efb9c740 1 Processor -- start 2024-04-23T14:47:25.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.581+0000 7f04efb9c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632dc83ca30 0x5632dc850050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.581+0000 7f04efb9c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5632dc83d350 0x5632dc8414a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.581+0000 7f04efb9c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632dc847a20 0x5632dc841a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.581+0000 7f04efb9c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f04e81aea70 con 0x5632dc83ca30 2024-04-23T14:47:25.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.581+0000 7f04efb9c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f04e81afee0 con 0x5632dc83d350 2024-04-23T14:47:25.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.581+0000 7f04efb9c740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f04e81af140 con 0x5632dc847a20 2024-04-23T14:47:25.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.581+0000 7f04efb9c740 10 client.0 _ll_register_callbacks cb 0x7f04d40097b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:47:25.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.581+0000 7f04e5400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5632dc83d350 0x5632dc8414a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.581+0000 7f04e5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632dc83ca30 0x5632dc850050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.581+0000 7f04e6800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632dc847a20 0x5632dc841a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.581+0000 7f04e5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632dc83ca30 0x5632dc850050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57672/0 (socket says 172.21.15.28:57672) 2024-04-23T14:47:25.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.581+0000 7f04e5400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5632dc83d350 0x5632dc8414a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:43516/0 (socket says 172.21.15.28:43516) 2024-04-23T14:47:25.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.581+0000 7f04e5e00640 1 -- 172.21.15.28:0/2982777920 learned_addr learned my addr 172.21.15.28:0/2982777920 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:25.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.582+0000 7f04e6800640 1 -- 172.21.15.28:0/2982777920 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5632dc83d350 msgr2=0x5632dc8414a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:25.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.582+0000 7f04e6800640 1 --2- 172.21.15.28:0/2982777920 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5632dc83d350 0x5632dc8414a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.582+0000 7f04e6800640 1 -- 172.21.15.28:0/2982777920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632dc83ca30 msgr2=0x5632dc850050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:25.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.582+0000 7f04e6800640 1 --2- 172.21.15.28:0/2982777920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632dc83ca30 0x5632dc850050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.582+0000 7f04e6800640 1 -- 172.21.15.28:0/2982777920 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632dc4740f0 con 0x5632dc847a20 2024-04-23T14:47:25.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.582+0000 7f04e5400640 1 --2- 172.21.15.28:0/2982777920 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5632dc83d350 0x5632dc8414a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:47:25.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.582+0000 7f04e5e00640 1 --2- 172.21.15.28:0/2982777920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632dc83ca30 0x5632dc850050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:47:25.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.582+0000 7f04e6800640 1 --2- 172.21.15.28:0/2982777920 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632dc847a20 0x5632dc841a00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f04e0010d00 tx=0x7f04e0004b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5276b4fc94eee1ac server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:25.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.582+0000 7f04c7e00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:47:25.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.582+0000 7f04c7e00640 1 -- 172.21.15.28:0/2982777920 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e0016040 con 0x5632dc847a20 2024-04-23T14:47:25.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.583+0000 7f04efb9c740 1 -- 172.21.15.28:0/2982777920 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04d802f070 con 0x5632dc847a20 2024-04-23T14:47:25.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.583+0000 7f04c7e00640 1 -- 172.21.15.28:0/2982777920 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f04e0012760 con 0x5632dc847a20 2024-04-23T14:47:25.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.583+0000 7f04c7e00640 1 -- 172.21.15.28:0/2982777920 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f04e0032050 con 0x5632dc847a20 2024-04-23T14:47:25.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.583+0000 7f04efb9c740 10 client.25819 fetch_fsmap learned FSMap version 20 2024-04-23T14:47:25.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.583+0000 7f04efb9c740 1 -- 172.21.15.28:0/2982777920 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5632dc767810 con 0x5632dc847a20 2024-04-23T14:47:25.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.583+0000 7f04c7e00640 1 -- 172.21.15.28:0/2982777920 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04d802f070 con 0x5632dc847a20 2024-04-23T14:47:25.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.583+0000 7f04c7e00640 1 -- 172.21.15.28:0/2982777920 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f04e002d030 con 0x5632dc847a20 2024-04-23T14:47:25.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.583+0000 7f04efb9c740 10 client.25819 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:47:25.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.583+0000 7f04efb9c740 10 client.25819 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:47:25.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.583+0000 7f04efb9c740 20 client.25819 populate_metadata read hostname 'smithi028' 2024-04-23T14:47:25.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.583+0000 7f04efb9c740 1 --2- 172.21.15.28:0/2982777920 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5632dc8be540 0x5632dc8c0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:25.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.584+0000 7f04efb9c740 4 client.25819 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:47:25.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.584+0000 7f04efb9c740 1 -- 172.21.15.28:0/2982777920 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5632dc7a9a80 con 0x5632dc8be540 2024-04-23T14:47:25.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.584+0000 7f04e5e00640 1 --2- 172.21.15.28:0/2982777920 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5632dc8be540 0x5632dc8c0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:25.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.584+0000 7f04e5e00640 1 --2- 172.21.15.28:0/2982777920 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5632dc8be540 0x5632dc8c0960 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c0c82a9a27b31fa server_cookie=458d5a33f005fcf9 in_seq=0 out_seq=0 2024-04-23T14:47:25.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.584+0000 7f04c7e00640 10 client.25819 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:47:25.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.585+0000 7f04c7e00640 1 -- 172.21.15.28:0/2982777920 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5632dc7a9a80 con 0x5632dc8be540 2024-04-23T14:47:25.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.585+0000 7f04c7e00640 10 client.25819 handle_command_reply: tid=0 2024-04-23T14:47:25.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.585+0000 7f04efb9c740 1 client.25819 shutdown 2024-04-23T14:47:25.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.585+0000 7f04efb9c740 1 -- 172.21.15.28:0/2982777920 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632dc847a20 msgr2=0x5632dc841a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:25.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.585+0000 7f04efb9c740 1 --2- 172.21.15.28:0/2982777920 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632dc847a20 0x5632dc841a00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f04e0010d00 tx=0x7f04e0004b90 comp rx=0 tx=0).stop 2024-04-23T14:47:25.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.586+0000 7f04efb9c740 1 -- 172.21.15.28:0/2982777920 shutdown_connections 2024-04-23T14:47:25.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.586+0000 7f04efb9c740 1 --2- 172.21.15.28:0/2982777920 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5632dc8be540 0x5632dc8c0960 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.586+0000 7f04efb9c740 1 --2- 172.21.15.28:0/2982777920 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5632dc83d350 0x5632dc8414a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.586+0000 7f04efb9c740 1 --2- 172.21.15.28:0/2982777920 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5632dc847a20 0x5632dc841a00 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.586+0000 7f04efb9c740 1 --2- 172.21.15.28:0/2982777920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632dc83ca30 0x5632dc850050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.586+0000 7f04c7e00640 0 client.25819 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:47:25.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.586+0000 7f04efb9c740 1 -- 172.21.15.28:0/2982777920 >> 172.21.15.28:0/2982777920 conn(0x5632dc838d50 msgr2=0x5632dc83ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:25.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.586+0000 7f04efb9c740 1 -- 172.21.15.28:0/2982777920 shutdown_connections 2024-04-23T14:47:25.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.586+0000 7f04efb9c740 1 -- 172.21.15.28:0/2982777920 wait complete. 2024-04-23T14:47:25.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.586+0000 7f04efb9c740 20 client.25819 trim_cache size 0 max 16384 2024-04-23T14:47:25.591 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:47:25.591 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-23T14:47:25.591 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-23T14:47:25.591 INFO:teuthology.orchestra.run.smithi028.stdout: "ab069c6b-1232-40ff-a0b2-bb61138dcf1b": { 2024-04-23T14:47:25.591 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-23T14:47:25.591 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "ab069c6b-1232-40ff-a0b2-bb61138dcf1b", 2024-04-23T14:47:25.591 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-23T14:47:25.591 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:47:25.591 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:47:25.591 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:47:25.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.587+0000 7f04e7200640 1 -- 172.21.15.28:0/3334966513 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f04cc0a3da0 msgr2=0x7f04cc0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:25.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.587+0000 7f04e7200640 1 --2- 172.21.15.28:0/3334966513 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f04cc0a3da0 0x7f04cc0a6260 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x5632dc83dff0 tx=0x7f04d8009260 comp rx=0 tx=0).stop 2024-04-23T14:47:25.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.587+0000 7f04e7200640 1 -- 172.21.15.28:0/3334966513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04e81071f0 msgr2=0x7f04e81a0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:25.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.587+0000 7f04e7200640 1 --2- 172.21.15.28:0/3334966513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04e81071f0 0x7f04e81a0c10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f04d4035040 tx=0x7f04d4033d90 comp rx=0 tx=0).stop 2024-04-23T14:47:25.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.587+0000 7f04e7200640 1 -- 172.21.15.28:0/3334966513 shutdown_connections 2024-04-23T14:47:25.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.587+0000 7f04e7200640 1 --2- 172.21.15.28:0/3334966513 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f04cc0a3da0 0x7f04cc0a6260 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.587+0000 7f04e7200640 1 --2- 172.21.15.28:0/3334966513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04e819a840 0x7f04e819acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.587+0000 7f04e7200640 1 --2- 172.21.15.28:0/3334966513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04e81071f0 0x7f04e81a0c10 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.587+0000 7f04e7200640 1 --2- 172.21.15.28:0/3334966513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04e8102120 0x7f04e81a06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:25.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.587+0000 7f04e7200640 1 -- 172.21.15.28:0/3334966513 >> 172.21.15.28:0/3334966513 conn(0x7f04e80fd9a0 msgr2=0x7f04e80fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:25.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.587+0000 7f04e7200640 1 -- 172.21.15.28:0/3334966513 shutdown_connections 2024-04-23T14:47:25.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:25.587+0000 7f04e7200640 1 -- 172.21.15.28:0/3334966513 wait complete. 2024-04-23T14:47:25.607 INFO:tasks.cephfs.filesystem:scrub status for tag:ab069c6b-1232-40ff-a0b2-bb61138dcf1b - {'path': '/', 'tag': 'ab069c6b-1232-40ff-a0b2-bb61138dcf1b', 'options': 'recursive,force'} 2024-04-23T14:47:26.472 INFO:tasks.workunit.client.0.smithi028.stdout: 1 198756 10.69 MB/sec warmup 102 sec latency 9.171 ms 2024-04-23T14:47:27.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:26 smithi105 ceph-mon[32555]: scrub queued for path: / 2024-04-23T14:47:27.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:26 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:47:27.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:26 smithi105 ceph-mon[32555]: scrub summary: active paths [/] 2024-04-23T14:47:27.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:26 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:47:27.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:26 smithi105 ceph-mon[32555]: pgmap v105: 97 pgs: 97 active+clean; 120 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 801 KiB/s rd, 2.6 MiB/s wr, 159 op/s 2024-04-23T14:47:27.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:26 smithi038 ceph-mon[32348]: scrub queued for path: / 2024-04-23T14:47:27.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:26 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:47:27.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:26 smithi038 ceph-mon[32348]: scrub summary: active paths [/] 2024-04-23T14:47:27.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:26 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:47:27.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:26 smithi038 ceph-mon[32348]: pgmap v105: 97 pgs: 97 active+clean; 120 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 801 KiB/s rd, 2.6 MiB/s wr, 159 op/s 2024-04-23T14:47:27.128 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:26 smithi028 ceph-mon[27810]: scrub queued for path: / 2024-04-23T14:47:27.128 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:26 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:47:27.128 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:26 smithi028 ceph-mon[27810]: scrub summary: active paths [/] 2024-04-23T14:47:27.128 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:26 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:47:27.128 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:26 smithi028 ceph-mon[27810]: pgmap v105: 97 pgs: 97 active+clean; 120 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 801 KiB/s rd, 2.6 MiB/s wr, 159 op/s 2024-04-23T14:47:27.471 INFO:tasks.workunit.client.0.smithi028.stdout: 1 200087 10.65 MB/sec warmup 103 sec latency 73.817 ms 2024-04-23T14:47:28.472 INFO:tasks.workunit.client.0.smithi028.stdout: 1 202571 10.68 MB/sec warmup 104 sec latency 9.020 ms 2024-04-23T14:47:29.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:28 smithi105 ceph-mon[32555]: pgmap v106: 97 pgs: 97 active+clean; 124 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.8 MiB/s wr, 243 op/s 2024-04-23T14:47:29.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:28 smithi038 ceph-mon[32348]: pgmap v106: 97 pgs: 97 active+clean; 124 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.8 MiB/s wr, 243 op/s 2024-04-23T14:47:29.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:28 smithi028 ceph-mon[27810]: pgmap v106: 97 pgs: 97 active+clean; 124 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.8 MiB/s wr, 243 op/s 2024-04-23T14:47:29.472 INFO:tasks.workunit.client.0.smithi028.stdout: 1 204683 10.65 MB/sec warmup 105 sec latency 12.644 ms 2024-04-23T14:47:30.472 INFO:tasks.workunit.client.0.smithi028.stdout: 1 206537 10.66 MB/sec warmup 106 sec latency 13.349 ms 2024-04-23T14:47:31.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:30 smithi105 ceph-mon[32555]: pgmap v107: 97 pgs: 97 active+clean; 124 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 2.4 MiB/s wr, 164 op/s 2024-04-23T14:47:31.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:30 smithi105 ceph-mon[32555]: scrub summary: idle 2024-04-23T14:47:31.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:30 smithi038 ceph-mon[32348]: pgmap v107: 97 pgs: 97 active+clean; 124 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 2.4 MiB/s wr, 164 op/s 2024-04-23T14:47:31.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:30 smithi038 ceph-mon[32348]: scrub summary: idle 2024-04-23T14:47:31.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:30 smithi028 ceph-mon[27810]: pgmap v107: 97 pgs: 97 active+clean; 124 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 2.4 MiB/s wr, 164 op/s 2024-04-23T14:47:31.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:30 smithi028 ceph-mon[27810]: scrub summary: idle 2024-04-23T14:47:31.472 INFO:tasks.workunit.client.0.smithi028.stdout: 1 209347 10.73 MB/sec warmup 107 sec latency 9.492 ms 2024-04-23T14:47:32.473 INFO:tasks.workunit.client.0.smithi028.stdout: 1 210731 10.69 MB/sec warmup 108 sec latency 59.512 ms 2024-04-23T14:47:33.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:32 smithi105 ceph-mon[32555]: pgmap v108: 97 pgs: 97 active+clean; 124 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 2.4 MiB/s wr, 164 op/s 2024-04-23T14:47:33.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:32 smithi038 ceph-mon[32348]: pgmap v108: 97 pgs: 97 active+clean; 124 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 2.4 MiB/s wr, 164 op/s 2024-04-23T14:47:33.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:32 smithi028 ceph-mon[27810]: pgmap v108: 97 pgs: 97 active+clean; 124 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 2.4 MiB/s wr, 164 op/s 2024-04-23T14:47:33.472 INFO:tasks.workunit.client.0.smithi028.stdout: 1 213214 10.72 MB/sec warmup 109 sec latency 15.946 ms 2024-04-23T14:47:34.473 INFO:tasks.workunit.client.0.smithi028.stdout: 1 215310 10.69 MB/sec warmup 110 sec latency 24.946 ms 2024-04-23T14:47:34.924 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:34 smithi038 ceph-mon[32348]: pgmap v109: 97 pgs: 97 active+clean; 131 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 250 op/s 2024-04-23T14:47:35.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:34 smithi105 ceph-mon[32555]: pgmap v109: 97 pgs: 97 active+clean; 131 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 250 op/s 2024-04-23T14:47:35.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:34 smithi028 ceph-mon[27810]: pgmap v109: 97 pgs: 97 active+clean; 131 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 250 op/s 2024-04-23T14:47:35.472 INFO:tasks.workunit.client.0.smithi028.stdout: 1 217249 10.70 MB/sec warmup 111 sec latency 11.924 ms 2024-04-23T14:47:36.473 INFO:tasks.workunit.client.0.smithi028.stdout: 1 219694 10.73 MB/sec warmup 112 sec latency 12.424 ms 2024-04-23T14:47:37.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:36 smithi105 ceph-mon[32555]: pgmap v110: 97 pgs: 97 active+clean; 131 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 2.5 MiB/s wr, 170 op/s 2024-04-23T14:47:37.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:36 smithi038 ceph-mon[32348]: pgmap v110: 97 pgs: 97 active+clean; 131 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 2.5 MiB/s wr, 170 op/s 2024-04-23T14:47:37.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:36 smithi028 ceph-mon[27810]: pgmap v110: 97 pgs: 97 active+clean; 131 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 2.5 MiB/s wr, 170 op/s 2024-04-23T14:47:37.473 INFO:tasks.workunit.client.0.smithi028.stdout: 1 220902 10.68 MB/sec warmup 113 sec latency 65.773 ms 2024-04-23T14:47:38.473 INFO:tasks.workunit.client.0.smithi028.stdout: 1 223328 10.71 MB/sec warmup 114 sec latency 62.059 ms 2024-04-23T14:47:39.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:38 smithi105 ceph-mon[32555]: pgmap v111: 97 pgs: 97 active+clean; 129 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 243 op/s 2024-04-23T14:47:39.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:38 smithi038 ceph-mon[32348]: pgmap v111: 97 pgs: 97 active+clean; 129 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 243 op/s 2024-04-23T14:47:39.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:38 smithi028 ceph-mon[27810]: pgmap v111: 97 pgs: 97 active+clean; 129 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 243 op/s 2024-04-23T14:47:39.473 INFO:tasks.workunit.client.0.smithi028.stdout: 1 224764 10.68 MB/sec warmup 115 sec latency 15.661 ms 2024-04-23T14:47:40.473 INFO:tasks.workunit.client.0.smithi028.stdout: 1 227088 10.72 MB/sec warmup 116 sec latency 22.059 ms 2024-04-23T14:47:41.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:40 smithi105 ceph-mon[32555]: pgmap v112: 97 pgs: 97 active+clean; 129 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 2.1 MiB/s wr, 159 op/s 2024-04-23T14:47:41.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:40 smithi038 ceph-mon[32348]: pgmap v112: 97 pgs: 97 active+clean; 129 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 2.1 MiB/s wr, 159 op/s 2024-04-23T14:47:41.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:40 smithi028 ceph-mon[27810]: pgmap v112: 97 pgs: 97 active+clean; 129 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 2.1 MiB/s wr, 159 op/s 2024-04-23T14:47:41.473 INFO:tasks.workunit.client.0.smithi028.stdout: 1 228238 10.66 MB/sec warmup 117 sec latency 21.265 ms 2024-04-23T14:47:42.474 INFO:tasks.workunit.client.0.smithi028.stdout: 1 230629 10.70 MB/sec warmup 118 sec latency 19.971 ms 2024-04-23T14:47:42.928 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:42 smithi028 ceph-mon[27810]: pgmap v113: 97 pgs: 97 active+clean; 129 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 2.1 MiB/s wr, 159 op/s 2024-04-23T14:47:43.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:42 smithi105 ceph-mon[32555]: pgmap v113: 97 pgs: 97 active+clean; 129 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 2.1 MiB/s wr, 159 op/s 2024-04-23T14:47:43.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:42 smithi038 ceph-mon[32348]: pgmap v113: 97 pgs: 97 active+clean; 129 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 2.1 MiB/s wr, 159 op/s 2024-04-23T14:47:43.474 INFO:tasks.workunit.client.0.smithi028.stdout: 1 231672 10.64 MB/sec warmup 119 sec latency 47.249 ms 2024-04-23T14:47:45.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:44 smithi105 ceph-mon[32555]: pgmap v114: 97 pgs: 97 active+clean; 132 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 236 op/s 2024-04-23T14:47:45.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:44 smithi038 ceph-mon[32348]: pgmap v114: 97 pgs: 97 active+clean; 132 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 236 op/s 2024-04-23T14:47:45.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:44 smithi028 ceph-mon[27810]: pgmap v114: 97 pgs: 97 active+clean; 132 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 236 op/s 2024-04-23T14:47:45.474 INFO:tasks.workunit.client.0.smithi028.stdout: 1 236021 6.76 MB/sec execute 1 sec latency 15.772 ms 2024-04-23T14:47:46.474 INFO:tasks.workunit.client.0.smithi028.stdout: 1 238244 10.24 MB/sec execute 2 sec latency 9.977 ms 2024-04-23T14:47:47.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:46 smithi105 ceph-mon[32555]: pgmap v115: 97 pgs: 97 active+clean; 132 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 1.9 MiB/s wr, 150 op/s 2024-04-23T14:47:47.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:46 smithi038 ceph-mon[32348]: pgmap v115: 97 pgs: 97 active+clean; 132 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 1.9 MiB/s wr, 150 op/s 2024-04-23T14:47:47.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:46 smithi028 ceph-mon[27810]: pgmap v115: 97 pgs: 97 active+clean; 132 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 1.9 MiB/s wr, 150 op/s 2024-04-23T14:47:47.474 INFO:tasks.workunit.client.0.smithi028.stdout: 1 240675 11.01 MB/sec execute 3 sec latency 12.664 ms 2024-04-23T14:47:48.474 INFO:tasks.workunit.client.0.smithi028.stdout: 1 242041 10.02 MB/sec execute 4 sec latency 58.942 ms 2024-04-23T14:47:49.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:48 smithi105 ceph-mon[32555]: pgmap v116: 97 pgs: 97 active+clean; 137 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 230 op/s 2024-04-23T14:47:49.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:48 smithi038 ceph-mon[32348]: pgmap v116: 97 pgs: 97 active+clean; 137 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 230 op/s 2024-04-23T14:47:49.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:48 smithi028 ceph-mon[27810]: pgmap v116: 97 pgs: 97 active+clean; 137 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 230 op/s 2024-04-23T14:47:49.474 INFO:tasks.workunit.client.0.smithi028.stdout: 1 244615 10.82 MB/sec execute 5 sec latency 9.165 ms 2024-04-23T14:47:50.475 INFO:tasks.workunit.client.0.smithi028.stdout: 1 245994 10.14 MB/sec execute 6 sec latency 43.180 ms 2024-04-23T14:47:51.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:50 smithi105 ceph-mon[32555]: pgmap v117: 97 pgs: 97 active+clean; 137 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 829 KiB/s rd, 2.3 MiB/s wr, 157 op/s 2024-04-23T14:47:51.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:50 smithi038 ceph-mon[32348]: pgmap v117: 97 pgs: 97 active+clean; 137 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 829 KiB/s rd, 2.3 MiB/s wr, 157 op/s 2024-04-23T14:47:51.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:50 smithi028 ceph-mon[27810]: pgmap v117: 97 pgs: 97 active+clean; 137 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 829 KiB/s rd, 2.3 MiB/s wr, 157 op/s 2024-04-23T14:47:51.475 INFO:tasks.workunit.client.0.smithi028.stdout: 1 248565 11.07 MB/sec execute 7 sec latency 9.261 ms 2024-04-23T14:47:52.475 INFO:tasks.workunit.client.0.smithi028.stdout: 1 250221 10.41 MB/sec execute 8 sec latency 21.022 ms 2024-04-23T14:47:53.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:52 smithi105 ceph-mon[32555]: pgmap v118: 97 pgs: 97 active+clean; 137 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 829 KiB/s rd, 2.3 MiB/s wr, 157 op/s 2024-04-23T14:47:53.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:52 smithi038 ceph-mon[32348]: pgmap v118: 97 pgs: 97 active+clean; 137 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 829 KiB/s rd, 2.3 MiB/s wr, 157 op/s 2024-04-23T14:47:53.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:52 smithi028 ceph-mon[27810]: pgmap v118: 97 pgs: 97 active+clean; 137 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 829 KiB/s rd, 2.3 MiB/s wr, 157 op/s 2024-04-23T14:47:53.476 INFO:tasks.workunit.client.0.smithi028.stdout: 1 251953 10.61 MB/sec execute 9 sec latency 89.202 ms 2024-04-23T14:47:54.475 INFO:tasks.workunit.client.0.smithi028.stdout: 1 253063 9.90 MB/sec execute 10 sec latency 55.165 ms 2024-04-23T14:47:55.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:54 smithi105 ceph-mon[32555]: pgmap v119: 97 pgs: 97 active+clean; 142 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.7 MiB/s wr, 241 op/s 2024-04-23T14:47:55.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:54 smithi038 ceph-mon[32348]: pgmap v119: 97 pgs: 97 active+clean; 142 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.7 MiB/s wr, 241 op/s 2024-04-23T14:47:55.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:54 smithi028 ceph-mon[27810]: pgmap v119: 97 pgs: 97 active+clean; 142 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.7 MiB/s wr, 241 op/s 2024-04-23T14:47:55.475 INFO:tasks.workunit.client.0.smithi028.stdout: 1 255265 10.13 MB/sec execute 11 sec latency 13.756 ms 2024-04-23T14:47:55.608 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:47:55.609 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:47:55.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.833+0000 7fb8c2a00640 1 -- 172.21.15.28:0/568065631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8bc102c40 msgr2=0x7fb8bc106c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:55.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.833+0000 7fb8c2a00640 1 --2- 172.21.15.28:0/568065631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8bc102c40 0x7fb8bc106c80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb8a4009900 tx=0x7fb8a402a410 comp rx=0 tx=0).stop 2024-04-23T14:47:55.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.834+0000 7fb8c2a00640 1 -- 172.21.15.28:0/568065631 shutdown_connections 2024-04-23T14:47:55.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.834+0000 7fb8c2a00640 1 --2- 172.21.15.28:0/568065631 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8bc107360 0x7fb8bc10ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:55.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.834+0000 7fb8c2a00640 1 --2- 172.21.15.28:0/568065631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8bc102c40 0x7fb8bc106c80 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:55.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.834+0000 7fb8c2a00640 1 --2- 172.21.15.28:0/568065631 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8bc102290 0x7fb8bc102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:55.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.834+0000 7fb8c2a00640 1 -- 172.21.15.28:0/568065631 >> 172.21.15.28:0/568065631 conn(0x7fb8bc0fdb10 msgr2=0x7fb8bc0fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:55.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.834+0000 7fb8c2a00640 1 -- 172.21.15.28:0/568065631 shutdown_connections 2024-04-23T14:47:55.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.834+0000 7fb8c2a00640 1 -- 172.21.15.28:0/568065631 wait complete. 2024-04-23T14:47:55.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.835+0000 7fb8c2a00640 1 Processor -- start 2024-04-23T14:47:55.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.835+0000 7fb8c2a00640 1 -- start start 2024-04-23T14:47:55.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.836+0000 7fb8c2a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8bc102290 0x7fb8bc19e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:55.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.836+0000 7fb8c2a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8bc107360 0x7fb8bc19ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:55.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.836+0000 7fb8c2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8bc1987e0 0x7fb8bc198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:55.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.836+0000 7fb8c2a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc112440 con 0x7fb8bc1987e0 2024-04-23T14:47:55.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.836+0000 7fb8c2a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc1122c0 con 0x7fb8bc102290 2024-04-23T14:47:55.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.836+0000 7fb8c2a00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc1125c0 con 0x7fb8bc107360 2024-04-23T14:47:55.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.836+0000 7fb8bbe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8bc102290 0x7fb8bc19e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:55.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.836+0000 7fb8bbe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8bc102290 0x7fb8bc19e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48190/0 (socket says 172.21.15.28:48190) 2024-04-23T14:47:55.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.836+0000 7fb8bbe00640 1 -- 172.21.15.28:0/3497111016 learned_addr learned my addr 172.21.15.28:0/3497111016 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:55.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.836+0000 7fb8c0a00640 1 --2- 172.21.15.28:0/3497111016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8bc1987e0 0x7fb8bc198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:55.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.836+0000 7fb8bb400640 1 --2- 172.21.15.28:0/3497111016 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8bc107360 0x7fb8bc19ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:55.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.837+0000 7fb8bbe00640 1 -- 172.21.15.28:0/3497111016 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8bc107360 msgr2=0x7fb8bc19ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:55.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.837+0000 7fb8bbe00640 1 --2- 172.21.15.28:0/3497111016 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8bc107360 0x7fb8bc19ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:55.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.837+0000 7fb8bbe00640 1 -- 172.21.15.28:0/3497111016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8bc1987e0 msgr2=0x7fb8bc198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:55.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.837+0000 7fb8bbe00640 1 --2- 172.21.15.28:0/3497111016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8bc1987e0 0x7fb8bc198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:55.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.837+0000 7fb8bbe00640 1 -- 172.21.15.28:0/3497111016 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8bc199370 con 0x7fb8bc102290 2024-04-23T14:47:55.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.837+0000 7fb8bb400640 1 --2- 172.21.15.28:0/3497111016 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8bc107360 0x7fb8bc19ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:47:55.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.837+0000 7fb8bbe00640 1 --2- 172.21.15.28:0/3497111016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8bc102290 0x7fb8bc19e670 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac00eb90 tx=0x7fb8ac00ef90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:55.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.837+0000 7fb8b8c00640 1 -- 172.21.15.28:0/3497111016 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac004b20 con 0x7fb8bc102290 2024-04-23T14:47:55.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.837+0000 7fb8c2a00640 1 -- 172.21.15.28:0/3497111016 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8bc199660 con 0x7fb8bc102290 2024-04-23T14:47:55.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.837+0000 7fb8b8c00640 1 -- 172.21.15.28:0/3497111016 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8ac004cc0 con 0x7fb8bc102290 2024-04-23T14:47:55.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.837+0000 7fb8b8c00640 1 -- 172.21.15.28:0/3497111016 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac010720 con 0x7fb8bc102290 2024-04-23T14:47:55.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.837+0000 7fb8c2a00640 1 -- 172.21.15.28:0/3497111016 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8bc074580 con 0x7fb8bc102290 2024-04-23T14:47:55.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.842+0000 7fb8c5786740 1 Processor -- start 2024-04-23T14:47:55.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.842+0000 7fb8c5786740 1 -- start start 2024-04-23T14:47:55.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.842+0000 7fb8b8c00640 1 -- 172.21.15.28:0/3497111016 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fb8ac005020 con 0x7fb8bc102290 2024-04-23T14:47:55.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.843+0000 7fb8c5786740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e4160895b0 0x55e416089990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:55.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.843+0000 7fb8c5786740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e416089f60 0x55e41609cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:55.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.843+0000 7fb8c5786740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e41609d060 0x55e41609f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:55.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.843+0000 7fb8c5786740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e415fef1f0 con 0x55e41609d060 2024-04-23T14:47:55.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.843+0000 7fb8c5786740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e415f95350 con 0x55e416089f60 2024-04-23T14:47:55.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.843+0000 7fb8c5786740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e415feb1b0 con 0x55e4160895b0 2024-04-23T14:47:55.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.843+0000 7fb8bbe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e416089f60 0x55e41609cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:55.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.843+0000 7fb8bbe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e416089f60 0x55e41609cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48198/0 (socket says 172.21.15.28:48198) 2024-04-23T14:47:55.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.843+0000 7fb8bbe00640 1 -- 172.21.15.28:0/2947707535 learned_addr learned my addr 172.21.15.28:0/2947707535 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:55.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.843+0000 7fb8bb400640 1 --2- 172.21.15.28:0/2947707535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e41609d060 0x55e41609f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:55.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.843+0000 7fb8c0a00640 1 --2- 172.21.15.28:0/2947707535 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e4160895b0 0x55e416089990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:55.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.843+0000 7fb8bbe00640 1 -- 172.21.15.28:0/2947707535 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e4160895b0 msgr2=0x55e416089990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:55.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.843+0000 7fb8bbe00640 1 --2- 172.21.15.28:0/2947707535 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e4160895b0 0x55e416089990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:55.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.843+0000 7fb8bbe00640 1 -- 172.21.15.28:0/2947707535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e41609d060 msgr2=0x55e41609f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:55.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.843+0000 7fb8bbe00640 1 --2- 172.21.15.28:0/2947707535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e41609d060 0x55e41609f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:55.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.844+0000 7fb8bbe00640 1 -- 172.21.15.28:0/2947707535 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e415fe9f50 con 0x55e416089f60 2024-04-23T14:47:55.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.844+0000 7fb8c0a00640 1 --2- 172.21.15.28:0/2947707535 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e4160895b0 0x55e416089990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:47:55.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.844+0000 7fb8bbe00640 1 --2- 172.21.15.28:0/2947707535 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e416089f60 0x55e41609cb20 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac0ce4e0 tx=0x7fb8ac0d4ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b101628cdfe22bc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:55.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.844+0000 7fb8b1200640 1 -- 172.21.15.28:0/2947707535 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac0e4070 con 0x55e416089f60 2024-04-23T14:47:55.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.844+0000 7fb8b1200640 1 -- 172.21.15.28:0/2947707535 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8ac0d9bc0 con 0x55e416089f60 2024-04-23T14:47:55.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.844+0000 7fb8b1200640 1 -- 172.21.15.28:0/2947707535 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac0de600 con 0x55e416089f60 2024-04-23T14:47:55.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.844+0000 7fb8b8c00640 1 --2- 172.21.15.28:0/3497111016 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb8a00a3da0 0x7fb8a00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:55.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.845+0000 7fb8b8c00640 1 -- 172.21.15.28:0/3497111016 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fb8ac0d0e10 con 0x7fb8bc102290 2024-04-23T14:47:55.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.845+0000 7fb8c0a00640 1 --2- 172.21.15.28:0/3497111016 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb8a00a3da0 0x7fb8a00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:55.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.845+0000 7fb8c5786740 1 -- 172.21.15.28:0/2947707535 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e416089f60 msgr2=0x55e41609cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:55.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.845+0000 7fb8c5786740 1 --2- 172.21.15.28:0/2947707535 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e416089f60 0x55e41609cb20 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac0ce4e0 tx=0x7fb8ac0d4ec0 comp rx=0 tx=0).stop 2024-04-23T14:47:55.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.845+0000 7fb8c5786740 1 -- 172.21.15.28:0/2947707535 shutdown_connections 2024-04-23T14:47:55.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.845+0000 7fb8c5786740 1 --2- 172.21.15.28:0/2947707535 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e416089f60 0x55e41609cb20 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:55.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.845+0000 7fb8c5786740 1 --2- 172.21.15.28:0/2947707535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e41609d060 0x55e41609f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:55.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.845+0000 7fb8c5786740 1 --2- 172.21.15.28:0/2947707535 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e4160895b0 0x55e416089990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:55.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.845+0000 7fb8c5786740 1 -- 172.21.15.28:0/2947707535 >> 172.21.15.28:0/2947707535 conn(0x55e416085b80 msgr2=0x55e416085f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:55.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.846+0000 7fb8c5786740 1 -- 172.21.15.28:0/2947707535 shutdown_connections 2024-04-23T14:47:55.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.846+0000 7fb8c5786740 1 -- 172.21.15.28:0/2947707535 wait complete. 2024-04-23T14:47:55.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.846+0000 7fb8c0a00640 1 --2- 172.21.15.28:0/3497111016 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb8a00a3da0 0x7fb8a00a6260 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4010970 tx=0x7fb8b4012040 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:55.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.846+0000 7fb8c5786740 1 -- start start 2024-04-23T14:47:55.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.846+0000 7fb8c5786740 1 Processor -- start 2024-04-23T14:47:55.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.847+0000 7fb8c5786740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4160895b0 0x55e41609a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:55.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.847+0000 7fb8c5786740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e416089f60 0x55e41609a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:55.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.847+0000 7fb8c5786740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e41609d060 0x55e41609aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:55.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.847+0000 7fb8c5786740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e41609fe20 con 0x55e4160895b0 2024-04-23T14:47:55.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.847+0000 7fb8c5786740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e415fa8690 con 0x55e41609d060 2024-04-23T14:47:55.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.847+0000 7fb8c5786740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e41609ffa0 con 0x55e416089f60 2024-04-23T14:47:55.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.847+0000 7fb8c5786740 10 client.0 _ll_register_callbacks cb 0x55e415fe4650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:47:55.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.847+0000 7fb8bbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4160895b0 0x55e41609a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:55.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.847+0000 7fb8c0a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e41609d060 0x55e41609aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:55.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.847+0000 7fb8bbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4160895b0 0x55e41609a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58322/0 (socket says 172.21.15.28:58322) 2024-04-23T14:47:55.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.847+0000 7fb8bbe00640 1 -- 172.21.15.28:0/1422157709 learned_addr learned my addr 172.21.15.28:0/1422157709 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:55.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.847+0000 7fb8bb400640 1 --2- 172.21.15.28:0/1422157709 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e416089f60 0x55e41609a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:55.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.848+0000 7fb8bbe00640 1 -- 172.21.15.28:0/1422157709 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e416089f60 msgr2=0x55e41609a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:55.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.848+0000 7fb8bbe00640 1 --2- 172.21.15.28:0/1422157709 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e416089f60 0x55e41609a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:55.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.848+0000 7fb8bbe00640 1 -- 172.21.15.28:0/1422157709 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e41609d060 msgr2=0x55e41609aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:55.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.848+0000 7fb8bbe00640 1 --2- 172.21.15.28:0/1422157709 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e41609d060 0x55e41609aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:55.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.848+0000 7fb8bbe00640 1 -- 172.21.15.28:0/1422157709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e415fa7920 con 0x55e4160895b0 2024-04-23T14:47:55.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.848+0000 7fb8c0a00640 1 --2- 172.21.15.28:0/1422157709 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e41609d060 0x55e41609aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:47:55.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.848+0000 7fb8bbe00640 1 --2- 172.21.15.28:0/1422157709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4160895b0 0x55e41609a040 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac099790 tx=0x7fb8ac0dfe60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15bc35dd97f9492 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:55.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.848+0000 7fb8a9600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-23T14:47:55.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.848+0000 7fb8a9600640 1 -- 172.21.15.28:0/1422157709 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac099060 con 0x55e4160895b0 2024-04-23T14:47:55.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.848+0000 7fb8a9600640 1 -- 172.21.15.28:0/1422157709 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fb8ac09b0c0 con 0x55e4160895b0 2024-04-23T14:47:55.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.848+0000 7fb8c5786740 1 -- 172.21.15.28:0/1422157709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8ac0d9bc0 con 0x55e4160895b0 2024-04-23T14:47:55.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.849+0000 7fb8a9600640 1 -- 172.21.15.28:0/1422157709 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fb8ac09c720 con 0x55e4160895b0 2024-04-23T14:47:55.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.849+0000 7fb8c5786740 10 client.15690 fetch_fsmap learned FSMap version 20 2024-04-23T14:47:55.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.849+0000 7fb8c5786740 1 -- 172.21.15.28:0/1422157709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55e415dcb7d0 con 0x55e4160895b0 2024-04-23T14:47:55.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.849+0000 7fb8a9600640 1 -- 172.21.15.28:0/1422157709 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8ac0d9bc0 con 0x55e4160895b0 2024-04-23T14:47:55.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.849+0000 7fb8a9600640 1 -- 172.21.15.28:0/1422157709 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fb8ac099430 con 0x55e4160895b0 2024-04-23T14:47:55.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.849+0000 7fb8c5786740 10 client.15690 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:47:55.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.849+0000 7fb8c5786740 10 client.15690 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:47:55.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.849+0000 7fb8c5786740 20 client.15690 populate_metadata read hostname 'smithi028' 2024-04-23T14:47:55.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.849+0000 7fb8c5786740 1 --2- 172.21.15.28:0/1422157709 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55e4160ad2d0 0x55e4160af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:55.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.849+0000 7fb8c5786740 4 client.15690 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:47:55.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.849+0000 7fb8c5786740 1 -- 172.21.15.28:0/1422157709 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e415ffcc10 con 0x55e4160ad2d0 2024-04-23T14:47:55.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.850+0000 7fb8bb400640 1 --2- 172.21.15.28:0/1422157709 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55e4160ad2d0 0x55e4160af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:55.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.850+0000 7fb8a9600640 10 client.15690 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:47:55.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.850+0000 7fb8bb400640 1 --2- 172.21.15.28:0/1422157709 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55e4160ad2d0 0x55e4160af6c0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=988fa161991bf2b7 server_cookie=5e95916ee0a93eb4 in_seq=0 out_seq=0 2024-04-23T14:47:55.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.851+0000 7fb8a9600640 1 -- 172.21.15.28:0/1422157709 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e415ffcc10 con 0x55e4160ad2d0 2024-04-23T14:47:55.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.851+0000 7fb8a9600640 10 client.15690 handle_command_reply: tid=0 2024-04-23T14:47:55.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.851+0000 7fb8c5786740 1 client.15690 shutdown 2024-04-23T14:47:55.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.852+0000 7fb8c5786740 1 -- 172.21.15.28:0/1422157709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4160895b0 msgr2=0x55e41609a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:55.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.852+0000 7fb8c5786740 1 --2- 172.21.15.28:0/1422157709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4160895b0 0x55e41609a040 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac099790 tx=0x7fb8ac0dfe60 comp rx=0 tx=0).stop 2024-04-23T14:47:55.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.852+0000 7fb8c5786740 1 -- 172.21.15.28:0/1422157709 shutdown_connections 2024-04-23T14:47:55.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.852+0000 7fb8c5786740 1 --2- 172.21.15.28:0/1422157709 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55e4160ad2d0 0x55e4160af6c0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:55.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.852+0000 7fb8c5786740 1 --2- 172.21.15.28:0/1422157709 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e41609d060 0x55e41609aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:55.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.852+0000 7fb8c5786740 1 --2- 172.21.15.28:0/1422157709 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e416089f60 0x55e41609a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:55.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.852+0000 7fb8c5786740 1 --2- 172.21.15.28:0/1422157709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4160895b0 0x55e41609a040 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:55.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.852+0000 7fb8a9600640 0 client.15690 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:47:55.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.852+0000 7fb8c5786740 1 -- 172.21.15.28:0/1422157709 >> 172.21.15.28:0/1422157709 conn(0x55e416085b80 msgr2=0x55e41608b9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:55.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.852+0000 7fb8c5786740 1 -- 172.21.15.28:0/1422157709 shutdown_connections 2024-04-23T14:47:55.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.852+0000 7fb8c5786740 1 -- 172.21.15.28:0/1422157709 wait complete. 2024-04-23T14:47:55.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.852+0000 7fb8c5786740 20 client.15690 trim_cache size 0 max 16384 2024-04-23T14:47:55.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.873+0000 7fb8c5786740 1 Processor -- start 2024-04-23T14:47:55.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.873+0000 7fb8c5786740 1 -- start start 2024-04-23T14:47:55.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.874+0000 7fb8c5786740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4160895b0 0x55e4160972c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:55.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.874+0000 7fb8c5786740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e416089f60 0x55e416097800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:55.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.874+0000 7fb8c5786740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e41609d060 0x55e4160a8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:55.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.874+0000 7fb8c5786740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc1a9bf0 con 0x55e4160895b0 2024-04-23T14:47:55.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.874+0000 7fb8c5786740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc1ac470 con 0x55e416089f60 2024-04-23T14:47:55.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.874+0000 7fb8c5786740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc1aa260 con 0x55e41609d060 2024-04-23T14:47:55.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.874+0000 7fb8bb400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e416089f60 0x55e416097800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:55.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.874+0000 7fb8bbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4160895b0 0x55e4160972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:55.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.874+0000 7fb8bb400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e416089f60 0x55e416097800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48218/0 (socket says 172.21.15.28:48218) 2024-04-23T14:47:55.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.874+0000 7fb8bb400640 1 -- 172.21.15.28:0/1405054192 learned_addr learned my addr 172.21.15.28:0/1405054192 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:55.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.874+0000 7fb8c0a00640 1 --2- 172.21.15.28:0/1405054192 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e41609d060 0x55e4160a8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:55.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.874+0000 7fb8bb400640 1 -- 172.21.15.28:0/1405054192 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e41609d060 msgr2=0x55e4160a8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:55.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.875+0000 7fb8bb400640 1 --2- 172.21.15.28:0/1405054192 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e41609d060 0x55e4160a8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:55.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.875+0000 7fb8bb400640 1 -- 172.21.15.28:0/1405054192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4160895b0 msgr2=0x55e4160972c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:55.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.875+0000 7fb8bb400640 1 --2- 172.21.15.28:0/1405054192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4160895b0 0x55e4160972c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:55.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.875+0000 7fb8bb400640 1 -- 172.21.15.28:0/1405054192 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e415d013f0 con 0x55e416089f60 2024-04-23T14:47:55.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.875+0000 7fb8c0a00640 1 --2- 172.21.15.28:0/1405054192 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e41609d060 0x55e4160a8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:47:55.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.875+0000 7fb8bb400640 1 --2- 172.21.15.28:0/1405054192 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e416089f60 0x55e416097800 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fb8a4006010 tx=0x7fb8a4005b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0192c7f290f91b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:55.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.875+0000 7fb8abe00640 1 -- 172.21.15.28:0/1405054192 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a4035040 con 0x55e416089f60 2024-04-23T14:47:55.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.875+0000 7fb8abe00640 1 -- 172.21.15.28:0/1405054192 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8a4045860 con 0x55e416089f60 2024-04-23T14:47:55.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.875+0000 7fb8abe00640 1 -- 172.21.15.28:0/1405054192 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a4005500 con 0x55e416089f60 2024-04-23T14:47:55.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.876+0000 7fb8c5786740 1 -- 172.21.15.28:0/1405054192 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e416089f60 msgr2=0x55e416097800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:55.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.876+0000 7fb8c5786740 1 --2- 172.21.15.28:0/1405054192 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e416089f60 0x55e416097800 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fb8a4006010 tx=0x7fb8a4005b70 comp rx=0 tx=0).stop 2024-04-23T14:47:55.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.876+0000 7fb8c5786740 1 -- 172.21.15.28:0/1405054192 shutdown_connections 2024-04-23T14:47:55.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.876+0000 7fb8c5786740 1 --2- 172.21.15.28:0/1405054192 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e416089f60 0x55e416097800 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:55.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.876+0000 7fb8c5786740 1 --2- 172.21.15.28:0/1405054192 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e41609d060 0x55e4160a8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:55.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.876+0000 7fb8c5786740 1 --2- 172.21.15.28:0/1405054192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4160895b0 0x55e4160972c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:55.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.876+0000 7fb8c5786740 1 -- 172.21.15.28:0/1405054192 >> 172.21.15.28:0/1405054192 conn(0x55e416085b80 msgr2=0x55e41608b980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:55.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.876+0000 7fb8c5786740 1 -- 172.21.15.28:0/1405054192 shutdown_connections 2024-04-23T14:47:55.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.876+0000 7fb8c5786740 1 -- 172.21.15.28:0/1405054192 wait complete. 2024-04-23T14:47:55.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.877+0000 7fb8c5786740 1 -- start start 2024-04-23T14:47:55.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.877+0000 7fb8c5786740 1 Processor -- start 2024-04-23T14:47:55.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.878+0000 7fb8c5786740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e4160895b0 0x55e4160b2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:55.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.878+0000 7fb8c5786740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e416089f60 0x55e4160afff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:55.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.878+0000 7fb8c5786740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e41609d060 0x55e4160b06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:55.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.878+0000 7fb8c5786740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc1aa930 con 0x55e416089f60 2024-04-23T14:47:55.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.878+0000 7fb8c5786740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc1abda0 con 0x55e4160895b0 2024-04-23T14:47:55.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.878+0000 7fb8c5786740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc1ab000 con 0x55e41609d060 2024-04-23T14:47:55.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.878+0000 7fb8c5786740 10 client.0 _ll_register_callbacks cb 0x7fb8ac0e2070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:47:55.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.878+0000 7fb8bbe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e4160895b0 0x55e4160b2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:55.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.878+0000 7fb8bbe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e4160895b0 0x55e4160b2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48222/0 (socket says 172.21.15.28:48222) 2024-04-23T14:47:55.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.878+0000 7fb8bbe00640 1 -- 172.21.15.28:0/292022423 learned_addr learned my addr 172.21.15.28:0/292022423 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:55.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.878+0000 7fb8c0a00640 1 --2- 172.21.15.28:0/292022423 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e41609d060 0x55e4160b06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:55.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.878+0000 7fb8bb400640 1 --2- 172.21.15.28:0/292022423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e416089f60 0x55e4160afff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:55.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.878+0000 7fb8bbe00640 1 -- 172.21.15.28:0/292022423 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e41609d060 msgr2=0x55e4160b06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:55.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.879+0000 7fb8bbe00640 1 --2- 172.21.15.28:0/292022423 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e41609d060 0x55e4160b06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:55.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.879+0000 7fb8bbe00640 1 -- 172.21.15.28:0/292022423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e416089f60 msgr2=0x55e4160afff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:55.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.879+0000 7fb8bbe00640 1 --2- 172.21.15.28:0/292022423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e416089f60 0x55e4160afff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:55.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.879+0000 7fb8bbe00640 1 -- 172.21.15.28:0/292022423 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e415cc10f0 con 0x55e4160895b0 2024-04-23T14:47:55.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.879+0000 7fb8c0a00640 1 --2- 172.21.15.28:0/292022423 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e41609d060 0x55e4160b06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:47:55.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.879+0000 7fb8bb400640 1 --2- 172.21.15.28:0/292022423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e416089f60 0x55e4160afff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:47:55.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.879+0000 7fb8bbe00640 1 --2- 172.21.15.28:0/292022423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e4160895b0 0x55e4160b2c60 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac0dff30 tx=0x7fb8ac009a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb089f8afa2612d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:55.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.879+0000 7fb8a9600640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:47:55.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.879+0000 7fb8a9600640 1 -- 172.21.15.28:0/292022423 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac09c900 con 0x55e4160895b0 2024-04-23T14:47:55.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.879+0000 7fb8a9600640 1 -- 172.21.15.28:0/292022423 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fb8ac0d99c0 con 0x55e4160895b0 2024-04-23T14:47:55.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.879+0000 7fb8c5786740 1 -- 172.21.15.28:0/292022423 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8a4045860 con 0x55e4160895b0 2024-04-23T14:47:55.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.879+0000 7fb8a9600640 1 -- 172.21.15.28:0/292022423 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fb8ac09cb90 con 0x55e4160895b0 2024-04-23T14:47:55.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.879+0000 7fb8c5786740 10 client.25847 fetch_fsmap learned FSMap version 20 2024-04-23T14:47:55.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.879+0000 7fb8c5786740 1 -- 172.21.15.28:0/292022423 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55e415fb4810 con 0x55e4160895b0 2024-04-23T14:47:55.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.880+0000 7fb8a9600640 1 -- 172.21.15.28:0/292022423 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8a4045860 con 0x55e4160895b0 2024-04-23T14:47:55.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.880+0000 7fb8a9600640 1 -- 172.21.15.28:0/292022423 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fb8ac07bbe0 con 0x55e4160895b0 2024-04-23T14:47:55.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.880+0000 7fb8c5786740 10 client.25847 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:47:55.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.880+0000 7fb8c5786740 10 client.25847 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:47:55.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.880+0000 7fb8c5786740 20 client.25847 populate_metadata read hostname 'smithi028' 2024-04-23T14:47:55.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.880+0000 7fb8c5786740 1 --2- 172.21.15.28:0/292022423 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55e416113980 0x55e416115d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:55.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.880+0000 7fb8c5786740 4 client.25847 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:47:55.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.880+0000 7fb8c5786740 1 -- 172.21.15.28:0/292022423 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e415ff6a80 con 0x55e416113980 2024-04-23T14:47:55.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.880+0000 7fb8bb400640 1 --2- 172.21.15.28:0/292022423 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55e416113980 0x55e416115d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:55.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.881+0000 7fb8bb400640 1 --2- 172.21.15.28:0/292022423 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55e416113980 0x55e416115d70 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4bb650022df05d server_cookie=89bf1b691c470030 in_seq=0 out_seq=0 2024-04-23T14:47:55.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.881+0000 7fb8a9600640 10 client.25847 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:47:55.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.881+0000 7fb8a9600640 1 -- 172.21.15.28:0/292022423 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e415ff6a80 con 0x55e416113980 2024-04-23T14:47:55.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.881+0000 7fb8a9600640 10 client.25847 handle_command_reply: tid=0 2024-04-23T14:47:55.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.881+0000 7fb8c5786740 1 client.25847 shutdown 2024-04-23T14:47:55.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.881+0000 7fb8c5786740 1 -- 172.21.15.28:0/292022423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e4160895b0 msgr2=0x55e4160b2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:55.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.881+0000 7fb8c5786740 1 --2- 172.21.15.28:0/292022423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e4160895b0 0x55e4160b2c60 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac0dff30 tx=0x7fb8ac009a70 comp rx=0 tx=0).stop 2024-04-23T14:47:55.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.881+0000 7fb8c5786740 1 -- 172.21.15.28:0/292022423 shutdown_connections 2024-04-23T14:47:55.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.881+0000 7fb8c5786740 1 --2- 172.21.15.28:0/292022423 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55e416113980 0x55e416115d70 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:55.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.881+0000 7fb8c5786740 1 --2- 172.21.15.28:0/292022423 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e41609d060 0x55e4160b06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:55.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.881+0000 7fb8c5786740 1 --2- 172.21.15.28:0/292022423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e416089f60 0x55e4160afff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:55.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.881+0000 7fb8a9600640 0 client.25847 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:47:55.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.881+0000 7fb8c5786740 1 --2- 172.21.15.28:0/292022423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e4160895b0 0x55e4160b2c60 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:55.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.881+0000 7fb8c5786740 1 -- 172.21.15.28:0/292022423 >> 172.21.15.28:0/292022423 conn(0x55e416085b80 msgr2=0x55e416084390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:55.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.882+0000 7fb8c5786740 1 -- 172.21.15.28:0/292022423 shutdown_connections 2024-04-23T14:47:55.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.882+0000 7fb8c5786740 1 -- 172.21.15.28:0/292022423 wait complete. 2024-04-23T14:47:55.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.882+0000 7fb8c5786740 20 client.25847 trim_cache size 0 max 16384 2024-04-23T14:47:55.886 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:47:55.886 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-23T14:47:55.886 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-23T14:47:55.886 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:47:55.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.882+0000 7fb8c2a00640 1 -- 172.21.15.28:0/3497111016 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb8a00a3da0 msgr2=0x7fb8a00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:55.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.882+0000 7fb8c2a00640 1 --2- 172.21.15.28:0/3497111016 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb8a00a3da0 0x7fb8a00a6260 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4010970 tx=0x7fb8b4012040 comp rx=0 tx=0).stop 2024-04-23T14:47:55.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.882+0000 7fb8c2a00640 1 -- 172.21.15.28:0/3497111016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8bc102290 msgr2=0x7fb8bc19e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:55.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.882+0000 7fb8c2a00640 1 --2- 172.21.15.28:0/3497111016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8bc102290 0x7fb8bc19e670 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac00eb90 tx=0x7fb8ac00ef90 comp rx=0 tx=0).stop 2024-04-23T14:47:55.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.883+0000 7fb8c2a00640 1 -- 172.21.15.28:0/3497111016 shutdown_connections 2024-04-23T14:47:55.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.883+0000 7fb8c2a00640 1 --2- 172.21.15.28:0/3497111016 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb8a00a3da0 0x7fb8a00a6260 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:55.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.883+0000 7fb8c2a00640 1 --2- 172.21.15.28:0/3497111016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8bc1987e0 0x7fb8bc198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:55.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.883+0000 7fb8c2a00640 1 --2- 172.21.15.28:0/3497111016 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8bc107360 0x7fb8bc19ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:55.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.883+0000 7fb8c2a00640 1 --2- 172.21.15.28:0/3497111016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8bc102290 0x7fb8bc19e670 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:55.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.883+0000 7fb8c2a00640 1 -- 172.21.15.28:0/3497111016 >> 172.21.15.28:0/3497111016 conn(0x7fb8bc0fdb10 msgr2=0x7fb8bc0ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:55.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.883+0000 7fb8c2a00640 1 -- 172.21.15.28:0/3497111016 shutdown_connections 2024-04-23T14:47:55.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:55.883+0000 7fb8c2a00640 1 -- 172.21.15.28:0/3497111016 wait complete. 2024-04-23T14:47:55.904 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-23T14:47:55.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:47:56.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.117+0000 7f5ca1800640 1 -- 172.21.15.28:0/2388679243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c9c104c10 msgr2=0x7f5c9c108c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:56.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.117+0000 7f5ca1800640 1 --2- 172.21.15.28:0/2388679243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c9c104c10 0x7f5c9c108c50 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f5c84009900 tx=0x7f5c8402a3f0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.118+0000 7f5ca1800640 1 -- 172.21.15.28:0/2388679243 shutdown_connections 2024-04-23T14:47:56.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.118+0000 7f5ca1800640 1 --2- 172.21.15.28:0/2388679243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c9c104c10 0x7f5c9c108c50 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.118+0000 7f5ca1800640 1 --2- 172.21.15.28:0/2388679243 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5c9c109330 0x7f5c9c110f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.118+0000 7f5ca1800640 1 --2- 172.21.15.28:0/2388679243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c9c104260 0x7f5c9c104640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.118+0000 7f5ca1800640 1 -- 172.21.15.28:0/2388679243 >> 172.21.15.28:0/2388679243 conn(0x7f5c9c0ffaa0 msgr2=0x7f5c9c101ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:56.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.118+0000 7f5ca1800640 1 -- 172.21.15.28:0/2388679243 shutdown_connections 2024-04-23T14:47:56.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.118+0000 7f5ca1800640 1 -- 172.21.15.28:0/2388679243 wait complete. 2024-04-23T14:47:56.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.119+0000 7f5ca1800640 1 Processor -- start 2024-04-23T14:47:56.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.119+0000 7f5ca1800640 1 -- start start 2024-04-23T14:47:56.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.120+0000 7f5ca1800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5c9c104260 0x7f5c9c1a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:56.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.120+0000 7f5ca1800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c9c109330 0x7f5c9c1a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:56.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.120+0000 7f5ca1800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c9c19a770 0x7f5c9c19abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:56.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.120+0000 7f5ca1800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5c9c1143f0 con 0x7f5c9c19a770 2024-04-23T14:47:56.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.120+0000 7f5ca1800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5c9c114270 con 0x7f5c9c109330 2024-04-23T14:47:56.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.120+0000 7f5ca1800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5c9c114570 con 0x7f5c9c104260 2024-04-23T14:47:56.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.120+0000 7f5c9b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c9c109330 0x7f5c9c1a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:56.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.120+0000 7f5ca0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c9c19a770 0x7f5c9c19abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:56.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.120+0000 7f5c9b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c9c109330 0x7f5c9c1a0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48240/0 (socket says 172.21.15.28:48240) 2024-04-23T14:47:56.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.120+0000 7f5c9b400640 1 -- 172.21.15.28:0/3684962810 learned_addr learned my addr 172.21.15.28:0/3684962810 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:56.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.120+0000 7f5c9be00640 1 --2- 172.21.15.28:0/3684962810 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5c9c104260 0x7f5c9c1a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:56.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.120+0000 7f5c9b400640 1 -- 172.21.15.28:0/3684962810 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5c9c104260 msgr2=0x7f5c9c1a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:56.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.120+0000 7f5c9b400640 1 --2- 172.21.15.28:0/3684962810 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5c9c104260 0x7f5c9c1a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.120+0000 7f5c9b400640 1 -- 172.21.15.28:0/3684962810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c9c19a770 msgr2=0x7f5c9c19abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:56.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.120+0000 7f5c9b400640 1 --2- 172.21.15.28:0/3684962810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c9c19a770 0x7f5c9c19abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.120+0000 7f5c9b400640 1 -- 172.21.15.28:0/3684962810 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c9c19b370 con 0x7f5c9c109330 2024-04-23T14:47:56.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.121+0000 7f5c9b400640 1 --2- 172.21.15.28:0/3684962810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c9c109330 0x7f5c9c1a0ba0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f5c8402a910 tx=0x7f5c84003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:56.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.121+0000 7f5c98c00640 1 -- 172.21.15.28:0/3684962810 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c8403f070 con 0x7f5c9c109330 2024-04-23T14:47:56.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.121+0000 7f5c98c00640 1 -- 172.21.15.28:0/3684962810 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c84031c90 con 0x7f5c9c109330 2024-04-23T14:47:56.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.121+0000 7f5ca1800640 1 -- 172.21.15.28:0/3684962810 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c9c19b600 con 0x7f5c9c109330 2024-04-23T14:47:56.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.121+0000 7f5c98c00640 1 -- 172.21.15.28:0/3684962810 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c84035030 con 0x7f5c9c109330 2024-04-23T14:47:56.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.121+0000 7f5ca1800640 1 -- 172.21.15.28:0/3684962810 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c9c1a76f0 con 0x7f5c9c109330 2024-04-23T14:47:56.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.123+0000 7f5ca1800640 1 -- 172.21.15.28:0/3684962810 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c9c105730 con 0x7f5c9c109330 2024-04-23T14:47:56.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.125+0000 7f5c98c00640 1 -- 172.21.15.28:0/3684962810 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f5c84042070 con 0x7f5c9c109330 2024-04-23T14:47:56.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.127+0000 7f5c98c00640 1 --2- 172.21.15.28:0/3684962810 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5c800a3d20 0x7f5c800a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:56.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.127+0000 7f5c98c00640 1 -- 172.21.15.28:0/3684962810 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f5c840f0f10 con 0x7f5c9c109330 2024-04-23T14:47:56.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.127+0000 7f5c9be00640 1 --2- 172.21.15.28:0/3684962810 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5c800a3d20 0x7f5c800a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:56.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.128+0000 7f5c9be00640 1 --2- 172.21.15.28:0/3684962810 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5c800a3d20 0x7f5c800a61e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c005fd0 tx=0x7f5c8c009500 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:56.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.132+0000 7f5c98c00640 1 -- 172.21.15.28:0/3684962810 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5c840bcf10 con 0x7f5c9c109330 2024-04-23T14:47:56.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.436+0000 7f5ca1800640 1 -- 172.21.15.28:0/3684962810 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5c9c10c560 con 0x7f5c9c109330 2024-04-23T14:47:56.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.437+0000 7f5c98c00640 1 -- 172.21.15.28:0/3684962810 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7f5c840c1210 con 0x7f5c9c109330 2024-04-23T14:47:56.438 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:47:56.439 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:47:56.439 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:47:56.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.440+0000 7f5ca1800640 1 -- 172.21.15.28:0/3684962810 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5c800a3d20 msgr2=0x7f5c800a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:56.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.440+0000 7f5ca1800640 1 --2- 172.21.15.28:0/3684962810 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5c800a3d20 0x7f5c800a61e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c005fd0 tx=0x7f5c8c009500 comp rx=0 tx=0).stop 2024-04-23T14:47:56.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.440+0000 7f5ca1800640 1 -- 172.21.15.28:0/3684962810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c9c109330 msgr2=0x7f5c9c1a0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:56.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.440+0000 7f5ca1800640 1 --2- 172.21.15.28:0/3684962810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c9c109330 0x7f5c9c1a0ba0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f5c8402a910 tx=0x7f5c84003b90 comp rx=0 tx=0).stop 2024-04-23T14:47:56.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.440+0000 7f5ca1800640 1 -- 172.21.15.28:0/3684962810 shutdown_connections 2024-04-23T14:47:56.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.440+0000 7f5ca1800640 1 --2- 172.21.15.28:0/3684962810 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5c800a3d20 0x7f5c800a61e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.440+0000 7f5ca1800640 1 --2- 172.21.15.28:0/3684962810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5c9c109330 0x7f5c9c1a0ba0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.440+0000 7f5ca1800640 1 --2- 172.21.15.28:0/3684962810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c9c19a770 0x7f5c9c19abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.440+0000 7f5ca1800640 1 --2- 172.21.15.28:0/3684962810 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5c9c104260 0x7f5c9c1a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.440+0000 7f5ca1800640 1 -- 172.21.15.28:0/3684962810 >> 172.21.15.28:0/3684962810 conn(0x7f5c9c0ffaa0 msgr2=0x7f5c9c100db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:56.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.440+0000 7f5ca1800640 1 -- 172.21.15.28:0/3684962810 shutdown_connections 2024-04-23T14:47:56.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.440+0000 7f5ca1800640 1 -- 172.21.15.28:0/3684962810 wait complete. 2024-04-23T14:47:56.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-23T14:47:56.479 INFO:tasks.workunit.client.0.smithi028.stdout: 1 256652 9.85 MB/sec execute 12 sec latency 27.206 ms 2024-04-23T14:47:56.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.681+0000 7f9a9b200640 1 -- 172.21.15.28:0/2453798872 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a9c112eb0 msgr2=0x7f9a9c1152a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:56.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.681+0000 7f9a9b200640 1 --2- 172.21.15.28:0/2453798872 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a9c112eb0 0x7f9a9c1152a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9a94009ca0 tx=0x7f9a9402a950 comp rx=0 tx=0).stop 2024-04-23T14:47:56.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.681+0000 7f9a9b200640 1 -- 172.21.15.28:0/2453798872 shutdown_connections 2024-04-23T14:47:56.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.681+0000 7f9a9b200640 1 --2- 172.21.15.28:0/2453798872 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a9c112eb0 0x7f9a9c1152a0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.681+0000 7f9a9b200640 1 --2- 172.21.15.28:0/2453798872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a9c074cc0 0x7f9a9c112970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.681+0000 7f9a9b200640 1 --2- 172.21.15.28:0/2453798872 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a9c0743a0 0x7f9a9c074780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.681+0000 7f9a9b200640 1 -- 172.21.15.28:0/2453798872 >> 172.21.15.28:0/2453798872 conn(0x7f9a9c0fd9d0 msgr2=0x7f9a9c0ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:56.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.681+0000 7f9a9b200640 1 -- 172.21.15.28:0/2453798872 shutdown_connections 2024-04-23T14:47:56.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.681+0000 7f9a9b200640 1 -- 172.21.15.28:0/2453798872 wait complete. 2024-04-23T14:47:56.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.683+0000 7f9a9b200640 1 Processor -- start 2024-04-23T14:47:56.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.683+0000 7f9a9b200640 1 -- start start 2024-04-23T14:47:56.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.683+0000 7f9a9b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a9c0743a0 0x7f9a9c19ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:56.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.683+0000 7f9a9b200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a9c074cc0 0x7f9a9c19f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:56.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.683+0000 7f9a9b200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a9c112eb0 0x7f9a9c1a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:56.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.683+0000 7f9a9b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9c118870 con 0x7f9a9c0743a0 2024-04-23T14:47:56.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.683+0000 7f9a9b200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9c1186f0 con 0x7f9a9c074cc0 2024-04-23T14:47:56.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.683+0000 7f9a9b200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9c1189f0 con 0x7f9a9c112eb0 2024-04-23T14:47:56.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.684+0000 7f9a99e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a9c0743a0 0x7f9a9c19ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:56.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.684+0000 7f9a99400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a9c074cc0 0x7f9a9c19f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:56.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.684+0000 7f9a99e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a9c0743a0 0x7f9a9c19ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58390/0 (socket says 172.21.15.28:58390) 2024-04-23T14:47:56.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.684+0000 7f9a99e00640 1 -- 172.21.15.28:0/1649607717 learned_addr learned my addr 172.21.15.28:0/1649607717 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:56.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.684+0000 7f9a9a800640 1 --2- 172.21.15.28:0/1649607717 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a9c112eb0 0x7f9a9c1a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:56.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.684+0000 7f9a99e00640 1 -- 172.21.15.28:0/1649607717 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a9c112eb0 msgr2=0x7f9a9c1a3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:56.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.684+0000 7f9a99e00640 1 --2- 172.21.15.28:0/1649607717 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a9c112eb0 0x7f9a9c1a3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.684+0000 7f9a99e00640 1 -- 172.21.15.28:0/1649607717 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a9c074cc0 msgr2=0x7f9a9c19f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:56.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.684+0000 7f9a99e00640 1 --2- 172.21.15.28:0/1649607717 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a9c074cc0 0x7f9a9c19f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.684+0000 7f9a99e00640 1 -- 172.21.15.28:0/1649607717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a9c1a3d80 con 0x7f9a9c0743a0 2024-04-23T14:47:56.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.684+0000 7f9a99400640 1 --2- 172.21.15.28:0/1649607717 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a9c074cc0 0x7f9a9c19f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:47:56.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.685+0000 7f9a99e00640 1 --2- 172.21.15.28:0/1649607717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a9c0743a0 0x7f9a9c19ed90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c00e9f0 tx=0x7f9a8c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:56.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.685+0000 7f9a92a00640 1 -- 172.21.15.28:0/1649607717 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c00cd80 con 0x7f9a9c0743a0 2024-04-23T14:47:56.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.685+0000 7f9a9b200640 1 -- 172.21.15.28:0/1649607717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a9c1a4070 con 0x7f9a9c0743a0 2024-04-23T14:47:56.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.685+0000 7f9a92a00640 1 -- 172.21.15.28:0/1649607717 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a8c005500 con 0x7f9a9c0743a0 2024-04-23T14:47:56.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.685+0000 7f9a92a00640 1 -- 172.21.15.28:0/1649607717 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c018630 con 0x7f9a9c0743a0 2024-04-23T14:47:56.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.685+0000 7f9a9b200640 1 -- 172.21.15.28:0/1649607717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a9c1ab900 con 0x7f9a9c0743a0 2024-04-23T14:47:56.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.688+0000 7f9aa3dfc740 1 Processor -- start 2024-04-23T14:47:56.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.688+0000 7f9a92a00640 1 -- 172.21.15.28:0/1649607717 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f9a8c0046c0 con 0x7f9a9c0743a0 2024-04-23T14:47:56.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.689+0000 7f9aa3dfc740 1 -- start start 2024-04-23T14:47:56.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.689+0000 7f9aa3dfc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fae5ab40 0x55f3fae5af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:56.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.689+0000 7f9aa3dfc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3fae5b460 0x55f3fae655f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:56.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.689+0000 7f9aa3dfc740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3fae65b30 0x55f3fae67f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:56.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.689+0000 7f9aa3dfc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f3fadc01d0 con 0x55f3fae5ab40 2024-04-23T14:47:56.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.689+0000 7f9aa3dfc740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f3fad66450 con 0x55f3fae65b30 2024-04-23T14:47:56.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.689+0000 7f9aa3dfc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f3fadbc190 con 0x55f3fae5b460 2024-04-23T14:47:56.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.689+0000 7f9a99400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3fae65b30 0x55f3fae67f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:56.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.690+0000 7f9a9a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fae5ab40 0x55f3fae5af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:56.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.690+0000 7f9a99400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3fae65b30 0x55f3fae67f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48266/0 (socket says 172.21.15.28:48266) 2024-04-23T14:47:56.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.690+0000 7f9a99400640 1 -- 172.21.15.28:0/505851793 learned_addr learned my addr 172.21.15.28:0/505851793 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:56.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.690+0000 7f9a99e00640 1 --2- 172.21.15.28:0/505851793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3fae5b460 0x55f3fae655f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:56.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.690+0000 7f9a99400640 1 -- 172.21.15.28:0/505851793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3fae5b460 msgr2=0x55f3fae655f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:56.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.690+0000 7f9a99400640 1 --2- 172.21.15.28:0/505851793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3fae5b460 0x55f3fae655f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.690+0000 7f9a99400640 1 -- 172.21.15.28:0/505851793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fae5ab40 msgr2=0x55f3fae5af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:56.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.690+0000 7f9a99400640 1 --2- 172.21.15.28:0/505851793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fae5ab40 0x55f3fae5af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.690+0000 7f9a99400640 1 -- 172.21.15.28:0/505851793 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3fae23310 con 0x55f3fae65b30 2024-04-23T14:47:56.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.690+0000 7f9a9a800640 1 --2- 172.21.15.28:0/505851793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fae5ab40 0x55f3fae5af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:47:56.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.690+0000 7f9a99400640 1 --2- 172.21.15.28:0/505851793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3fae65b30 0x55f3fae67f20 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9a88011470 tx=0x7f9a88011930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78c0347b426c224e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:56.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.690+0000 7f9a87400640 1 -- 172.21.15.28:0/505851793 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8801e070 con 0x55f3fae65b30 2024-04-23T14:47:56.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.691+0000 7f9a87400640 1 -- 172.21.15.28:0/505851793 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a880035f0 con 0x55f3fae65b30 2024-04-23T14:47:56.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.691+0000 7f9a87400640 1 -- 172.21.15.28:0/505851793 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8800f860 con 0x55f3fae65b30 2024-04-23T14:47:56.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.691+0000 7f9a92a00640 1 --2- 172.21.15.28:0/1649607717 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f9a800a3d20 0x7f9a800a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:56.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.691+0000 7f9a92a00640 1 -- 172.21.15.28:0/1649607717 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f9a8c0d0d90 con 0x7f9a9c0743a0 2024-04-23T14:47:56.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.691+0000 7f9a9a800640 1 --2- 172.21.15.28:0/1649607717 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f9a800a3d20 0x7f9a800a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:56.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.691+0000 7f9aa3dfc740 1 -- 172.21.15.28:0/505851793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3fae65b30 msgr2=0x55f3fae67f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:56.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.691+0000 7f9aa3dfc740 1 --2- 172.21.15.28:0/505851793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3fae65b30 0x55f3fae67f20 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9a88011470 tx=0x7f9a88011930 comp rx=0 tx=0).stop 2024-04-23T14:47:56.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.691+0000 7f9aa3dfc740 1 -- 172.21.15.28:0/505851793 shutdown_connections 2024-04-23T14:47:56.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.691+0000 7f9aa3dfc740 1 --2- 172.21.15.28:0/505851793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3fae65b30 0x55f3fae67f20 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.691+0000 7f9aa3dfc740 1 --2- 172.21.15.28:0/505851793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3fae5b460 0x55f3fae655f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.691+0000 7f9aa3dfc740 1 --2- 172.21.15.28:0/505851793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fae5ab40 0x55f3fae5af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.691+0000 7f9aa3dfc740 1 -- 172.21.15.28:0/505851793 >> 172.21.15.28:0/505851793 conn(0x55f3fae56e60 msgr2=0x55f3fae57240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:56.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.692+0000 7f9aa3dfc740 1 -- 172.21.15.28:0/505851793 shutdown_connections 2024-04-23T14:47:56.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.692+0000 7f9aa3dfc740 1 -- 172.21.15.28:0/505851793 wait complete. 2024-04-23T14:47:56.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.692+0000 7f9a9a800640 1 --2- 172.21.15.28:0/1649607717 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f9a800a3d20 0x7f9a800a61e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9a94005e30 tx=0x7f9a94005c50 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:56.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.692+0000 7f9aa3dfc740 1 -- start start 2024-04-23T14:47:56.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.692+0000 7f9aa3dfc740 1 Processor -- start 2024-04-23T14:47:56.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.693+0000 7f9aa3dfc740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3fae5ab40 0x55f3fae655d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:56.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.693+0000 7f9aa3dfc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fae5b460 0x55f3fae628c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:56.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.693+0000 7f9aa3dfc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3fae65b30 0x55f3fae62fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:56.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.693+0000 7f9aa3dfc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f3fae689c0 con 0x55f3fae5b460 2024-04-23T14:47:56.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.693+0000 7f9aa3dfc740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f3fad79810 con 0x55f3fae5ab40 2024-04-23T14:47:56.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.693+0000 7f9aa3dfc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f3fae68b40 con 0x55f3fae65b30 2024-04-23T14:47:56.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.693+0000 7f9aa3dfc740 10 client.0 _ll_register_callbacks cb 0x55f3fad47140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:47:56.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.693+0000 7f9a99e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3fae5ab40 0x55f3fae655d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:56.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.694+0000 7f9a99e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3fae5ab40 0x55f3fae655d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48276/0 (socket says 172.21.15.28:48276) 2024-04-23T14:47:56.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.694+0000 7f9a99e00640 1 -- 172.21.15.28:0/3122926012 learned_addr learned my addr 172.21.15.28:0/3122926012 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:56.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.694+0000 7f9a99400640 1 --2- 172.21.15.28:0/3122926012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fae5b460 0x55f3fae628c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:56.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.694+0000 7f9a99e00640 1 -- 172.21.15.28:0/3122926012 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3fae65b30 msgr2=0x55f3fae62fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:56.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.694+0000 7f9a99e00640 1 --2- 172.21.15.28:0/3122926012 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3fae65b30 0x55f3fae62fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.694+0000 7f9a99e00640 1 -- 172.21.15.28:0/3122926012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fae5b460 msgr2=0x55f3fae628c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:56.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.694+0000 7f9a99e00640 1 --2- 172.21.15.28:0/3122926012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fae5b460 0x55f3fae628c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.694+0000 7f9a99e00640 1 -- 172.21.15.28:0/3122926012 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3fad8a3f0 con 0x55f3fae5ab40 2024-04-23T14:47:56.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.694+0000 7f9a99e00640 1 --2- 172.21.15.28:0/3122926012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3fae5ab40 0x55f3fae655d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c00c6a0 tx=0x7f9a8c09cba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af52d10b0eebcf35 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:56.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.694+0000 7f9a7be00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:47:56.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.694+0000 7f9a7be00640 1 -- 172.21.15.28:0/3122926012 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c0ce920 con 0x55f3fae5ab40 2024-04-23T14:47:56.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.695+0000 7f9a7be00640 1 -- 172.21.15.28:0/3122926012 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f9a8c0cede0 con 0x55f3fae5ab40 2024-04-23T14:47:56.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.695+0000 7f9aa3dfc740 1 -- 172.21.15.28:0/3122926012 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a880035f0 con 0x55f3fae5ab40 2024-04-23T14:47:56.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.695+0000 7f9a7be00640 1 -- 172.21.15.28:0/3122926012 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f9a8c0b6250 con 0x55f3fae5ab40 2024-04-23T14:47:56.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.695+0000 7f9aa3dfc740 10 client.25865 fetch_fsmap learned FSMap version 20 2024-04-23T14:47:56.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.695+0000 7f9aa3dfc740 1 -- 172.21.15.28:0/3122926012 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55f3faad23f0 con 0x55f3fae5ab40 2024-04-23T14:47:56.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.695+0000 7f9a7be00640 1 -- 172.21.15.28:0/3122926012 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a880035f0 con 0x55f3fae5ab40 2024-04-23T14:47:56.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.695+0000 7f9a7be00640 1 -- 172.21.15.28:0/3122926012 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f9a8c09c130 con 0x55f3fae5ab40 2024-04-23T14:47:56.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.695+0000 7f9aa3dfc740 10 client.25865 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:47:56.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.695+0000 7f9aa3dfc740 10 client.25865 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:47:56.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.695+0000 7f9aa3dfc740 20 client.25865 populate_metadata read hostname 'smithi028' 2024-04-23T14:47:56.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.695+0000 7f9aa3dfc740 1 --2- 172.21.15.28:0/3122926012 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f3fae75e20 0x55f3fae78210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:56.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.695+0000 7f9aa3dfc740 4 client.25865 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:47:56.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.695+0000 7f9aa3dfc740 1 -- 172.21.15.28:0/3122926012 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f3fadcdbf0 con 0x55f3fae75e20 2024-04-23T14:47:56.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.696+0000 7f9a99400640 1 --2- 172.21.15.28:0/3122926012 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f3fae75e20 0x55f3fae78210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:56.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.696+0000 7f9a99400640 1 --2- 172.21.15.28:0/3122926012 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f3fae75e20 0x55f3fae78210 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f80940666ea7f58 server_cookie=2ace03cdcb7ff11b in_seq=0 out_seq=0 2024-04-23T14:47:56.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.696+0000 7f9a7be00640 10 client.25865 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:47:56.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.699+0000 7f9a7be00640 1 -- 172.21.15.28:0/3122926012 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f3fadcdbf0 con 0x55f3fae75e20 2024-04-23T14:47:56.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.699+0000 7f9a7be00640 10 client.25865 handle_command_reply: tid=0 2024-04-23T14:47:56.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.699+0000 7f9aa3dfc740 1 client.25865 shutdown 2024-04-23T14:47:56.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.699+0000 7f9aa3dfc740 1 -- 172.21.15.28:0/3122926012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3fae5ab40 msgr2=0x55f3fae655d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:56.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.699+0000 7f9aa3dfc740 1 --2- 172.21.15.28:0/3122926012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3fae5ab40 0x55f3fae655d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c00c6a0 tx=0x7f9a8c09cba0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.699+0000 7f9aa3dfc740 1 -- 172.21.15.28:0/3122926012 shutdown_connections 2024-04-23T14:47:56.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.699+0000 7f9aa3dfc740 1 --2- 172.21.15.28:0/3122926012 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f3fae75e20 0x55f3fae78210 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.699+0000 7f9aa3dfc740 1 --2- 172.21.15.28:0/3122926012 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3fae65b30 0x55f3fae62fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.699+0000 7f9aa3dfc740 1 --2- 172.21.15.28:0/3122926012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fae5b460 0x55f3fae628c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.699+0000 7f9aa3dfc740 1 --2- 172.21.15.28:0/3122926012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3fae5ab40 0x55f3fae655d0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.699+0000 7f9a7be00640 0 client.25865 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:47:56.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.699+0000 7f9aa3dfc740 1 -- 172.21.15.28:0/3122926012 >> 172.21.15.28:0/3122926012 conn(0x55f3fae56e60 msgr2=0x55f3fae5cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:56.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.700+0000 7f9aa3dfc740 1 -- 172.21.15.28:0/3122926012 shutdown_connections 2024-04-23T14:47:56.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.700+0000 7f9aa3dfc740 1 -- 172.21.15.28:0/3122926012 wait complete. 2024-04-23T14:47:56.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.700+0000 7f9aa3dfc740 20 client.25865 trim_cache size 0 max 16384 2024-04-23T14:47:56.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.717+0000 7f9aa3dfc740 1 Processor -- start 2024-04-23T14:47:56.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.717+0000 7f9aa3dfc740 1 -- start start 2024-04-23T14:47:56.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.717+0000 7f9aa3dfc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3fae5ab40 0x55f3fae718b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:56.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.717+0000 7f9aa3dfc740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3fae5b460 0x55f3fae65100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:56.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.717+0000 7f9aa3dfc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fae65b30 0x55f3fae7dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:56.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.717+0000 7f9aa3dfc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9c1b2140 con 0x55f3fae65b30 2024-04-23T14:47:56.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.717+0000 7f9aa3dfc740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9c1b49c0 con 0x55f3fae5b460 2024-04-23T14:47:56.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.717+0000 7f9aa3dfc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9c1b27b0 con 0x55f3fae5ab40 2024-04-23T14:47:56.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.717+0000 7f9a99400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3fae5b460 0x55f3fae65100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:56.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.718+0000 7f9a99400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3fae5b460 0x55f3fae65100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48290/0 (socket says 172.21.15.28:48290) 2024-04-23T14:47:56.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.718+0000 7f9a99400640 1 -- 172.21.15.28:0/2002436124 learned_addr learned my addr 172.21.15.28:0/2002436124 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:56.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.718+0000 7f9a99e00640 1 --2- 172.21.15.28:0/2002436124 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3fae5ab40 0x55f3fae718b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:56.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.718+0000 7f9a9a800640 1 --2- 172.21.15.28:0/2002436124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fae65b30 0x55f3fae7dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:56.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.718+0000 7f9a99400640 1 -- 172.21.15.28:0/2002436124 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3fae5ab40 msgr2=0x55f3fae718b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:56.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.718+0000 7f9a99400640 1 --2- 172.21.15.28:0/2002436124 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3fae5ab40 0x55f3fae718b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.718+0000 7f9a99400640 1 -- 172.21.15.28:0/2002436124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fae65b30 msgr2=0x55f3fae7dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:56.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.718+0000 7f9a99400640 1 --2- 172.21.15.28:0/2002436124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fae65b30 0x55f3fae7dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.718+0000 7f9a99400640 1 -- 172.21.15.28:0/2002436124 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3fab9c7d0 con 0x55f3fae5b460 2024-04-23T14:47:56.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.718+0000 7f9a99400640 1 --2- 172.21.15.28:0/2002436124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3fae5b460 0x55f3fae65100 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9a8801c040 tx=0x7f9a88005d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2eea8985142b82de server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:56.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.718+0000 7f9a86a00640 1 -- 172.21.15.28:0/2002436124 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8802d070 con 0x55f3fae5b460 2024-04-23T14:47:56.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.718+0000 7f9a86a00640 1 -- 172.21.15.28:0/2002436124 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a88028e50 con 0x55f3fae5b460 2024-04-23T14:47:56.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.718+0000 7f9a86a00640 1 -- 172.21.15.28:0/2002436124 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8801e040 con 0x55f3fae5b460 2024-04-23T14:47:56.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.719+0000 7f9aa3dfc740 1 -- 172.21.15.28:0/2002436124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3fae5b460 msgr2=0x55f3fae65100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:56.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.719+0000 7f9aa3dfc740 1 --2- 172.21.15.28:0/2002436124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3fae5b460 0x55f3fae65100 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9a8801c040 tx=0x7f9a88005d30 comp rx=0 tx=0).stop 2024-04-23T14:47:56.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.719+0000 7f9aa3dfc740 1 -- 172.21.15.28:0/2002436124 shutdown_connections 2024-04-23T14:47:56.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.719+0000 7f9aa3dfc740 1 --2- 172.21.15.28:0/2002436124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3fae5b460 0x55f3fae65100 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.719+0000 7f9aa3dfc740 1 --2- 172.21.15.28:0/2002436124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fae65b30 0x55f3fae7dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.719+0000 7f9aa3dfc740 1 --2- 172.21.15.28:0/2002436124 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3fae5ab40 0x55f3fae718b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.719+0000 7f9aa3dfc740 1 -- 172.21.15.28:0/2002436124 >> 172.21.15.28:0/2002436124 conn(0x55f3fae56e60 msgr2=0x55f3fae5dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:56.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.719+0000 7f9aa3dfc740 1 -- 172.21.15.28:0/2002436124 shutdown_connections 2024-04-23T14:47:56.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.719+0000 7f9aa3dfc740 1 -- 172.21.15.28:0/2002436124 wait complete. 2024-04-23T14:47:56.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.720+0000 7f9aa3dfc740 1 -- start start 2024-04-23T14:47:56.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.720+0000 7f9aa3dfc740 1 Processor -- start 2024-04-23T14:47:56.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.720+0000 7f9aa3dfc740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3fae5ab40 0x55f3fae66af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:56.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.720+0000 7f9aa3dfc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fae5b460 0x55f3fae67030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:56.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.720+0000 7f9aa3dfc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3fae65b30 0x55f3fae67570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:56.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.721+0000 7f9aa3dfc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9c1b2e80 con 0x55f3fae5b460 2024-04-23T14:47:56.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.721+0000 7f9aa3dfc740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9c1b42f0 con 0x55f3fae5ab40 2024-04-23T14:47:56.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.721+0000 7f9aa3dfc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9c1b3550 con 0x55f3fae65b30 2024-04-23T14:47:56.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.721+0000 7f9aa3dfc740 10 client.0 _ll_register_callbacks cb 0x7f9a8c009720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:47:56.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.721+0000 7f9a99e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3fae5ab40 0x55f3fae66af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:56.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.721+0000 7f9a99e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3fae5ab40 0x55f3fae66af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48306/0 (socket says 172.21.15.28:48306) 2024-04-23T14:47:56.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.721+0000 7f9a99400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fae5b460 0x55f3fae67030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:56.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.721+0000 7f9a99e00640 1 -- 172.21.15.28:0/318775191 learned_addr learned my addr 172.21.15.28:0/318775191 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:56.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.721+0000 7f9a9a800640 1 --2- 172.21.15.28:0/318775191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3fae65b30 0x55f3fae67570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:56.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.721+0000 7f9a99e00640 1 -- 172.21.15.28:0/318775191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3fae65b30 msgr2=0x55f3fae67570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:56.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.721+0000 7f9a99e00640 1 --2- 172.21.15.28:0/318775191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3fae65b30 0x55f3fae67570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.721+0000 7f9a99e00640 1 -- 172.21.15.28:0/318775191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fae5b460 msgr2=0x55f3fae67030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:56.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.721+0000 7f9a99e00640 1 --2- 172.21.15.28:0/318775191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fae5b460 0x55f3fae67030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.721+0000 7f9a99e00640 1 -- 172.21.15.28:0/318775191 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3fae6c3d0 con 0x55f3fae5ab40 2024-04-23T14:47:56.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.721+0000 7f9a9a800640 1 --2- 172.21.15.28:0/318775191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3fae65b30 0x55f3fae67570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:47:56.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.722+0000 7f9a99e00640 1 --2- 172.21.15.28:0/318775191 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3fae5ab40 0x55f3fae66af0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c09bba0 tx=0x7f9a8c0cee90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=896434547dc0b666 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:56.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.722+0000 7f9a7be00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:47:56.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.722+0000 7f9a7be00640 1 -- 172.21.15.28:0/318775191 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c098f80 con 0x55f3fae5ab40 2024-04-23T14:47:56.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.722+0000 7f9a7be00640 1 -- 172.21.15.28:0/318775191 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f9a8c0a5ba0 con 0x55f3fae5ab40 2024-04-23T14:47:56.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.722+0000 7f9aa3dfc740 1 -- 172.21.15.28:0/318775191 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a88028e50 con 0x55f3fae5ab40 2024-04-23T14:47:56.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.722+0000 7f9a7be00640 1 -- 172.21.15.28:0/318775191 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f9a8c0b6250 con 0x55f3fae5ab40 2024-04-23T14:47:56.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.722+0000 7f9aa3dfc740 10 client.25871 fetch_fsmap learned FSMap version 20 2024-04-23T14:47:56.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.722+0000 7f9aa3dfc740 1 -- 172.21.15.28:0/318775191 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55f3faedc7f0 con 0x55f3fae5ab40 2024-04-23T14:47:56.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.722+0000 7f9a7be00640 1 -- 172.21.15.28:0/318775191 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a88028e50 con 0x55f3fae5ab40 2024-04-23T14:47:56.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.722+0000 7f9a7be00640 1 -- 172.21.15.28:0/318775191 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f9a8c0a2220 con 0x55f3fae5ab40 2024-04-23T14:47:56.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.723+0000 7f9aa3dfc740 10 client.25871 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:47:56.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.723+0000 7f9aa3dfc740 10 client.25871 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:47:56.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.723+0000 7f9aa3dfc740 20 client.25871 populate_metadata read hostname 'smithi028' 2024-04-23T14:47:56.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.723+0000 7f9aa3dfc740 1 --2- 172.21.15.28:0/318775191 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f3faedceb0 0x55f3faedf2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:56.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.723+0000 7f9aa3dfc740 4 client.25871 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-23T14:47:56.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.723+0000 7f9aa3dfc740 1 -- 172.21.15.28:0/318775191 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f3fadc7a60 con 0x55f3faedceb0 2024-04-23T14:47:56.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.723+0000 7f9a99400640 1 --2- 172.21.15.28:0/318775191 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f3faedceb0 0x55f3faedf2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:56.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.724+0000 7f9a99400640 1 --2- 172.21.15.28:0/318775191 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f3faedceb0 0x55f3faedf2d0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3441b09c385ddba4 server_cookie=7c25673906a3c910 in_seq=0 out_seq=0 2024-04-23T14:47:56.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.724+0000 7f9a7be00640 10 client.25871 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:47:56.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.724+0000 7f9a7be00640 1 -- 172.21.15.28:0/318775191 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f3fadc7a60 con 0x55f3faedceb0 2024-04-23T14:47:56.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.724+0000 7f9a7be00640 10 client.25871 handle_command_reply: tid=0 2024-04-23T14:47:56.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.724+0000 7f9aa3dfc740 1 client.25871 shutdown 2024-04-23T14:47:56.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.725+0000 7f9aa3dfc740 1 -- 172.21.15.28:0/318775191 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3fae5ab40 msgr2=0x55f3fae66af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:56.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.725+0000 7f9aa3dfc740 1 --2- 172.21.15.28:0/318775191 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3fae5ab40 0x55f3fae66af0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c09bba0 tx=0x7f9a8c0cee90 comp rx=0 tx=0).stop 2024-04-23T14:47:56.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.725+0000 7f9aa3dfc740 1 -- 172.21.15.28:0/318775191 shutdown_connections 2024-04-23T14:47:56.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.725+0000 7f9aa3dfc740 1 --2- 172.21.15.28:0/318775191 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f3faedceb0 0x55f3faedf2d0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.725+0000 7f9aa3dfc740 1 --2- 172.21.15.28:0/318775191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f3fae65b30 0x55f3fae67570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.725+0000 7f9aa3dfc740 1 --2- 172.21.15.28:0/318775191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3fae5b460 0x55f3fae67030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.725+0000 7f9aa3dfc740 1 --2- 172.21.15.28:0/318775191 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3fae5ab40 0x55f3fae66af0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.725+0000 7f9a7be00640 0 client.25871 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:47:56.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.725+0000 7f9aa3dfc740 1 -- 172.21.15.28:0/318775191 >> 172.21.15.28:0/318775191 conn(0x55f3fae56e60 msgr2=0x55f3fae556c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:56.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.725+0000 7f9aa3dfc740 1 -- 172.21.15.28:0/318775191 shutdown_connections 2024-04-23T14:47:56.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.725+0000 7f9aa3dfc740 1 -- 172.21.15.28:0/318775191 wait complete. 2024-04-23T14:47:56.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.725+0000 7f9aa3dfc740 20 client.25871 trim_cache size 0 max 16384 2024-04-23T14:47:56.730 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-23T14:47:56.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.726+0000 7f9a9b200640 1 -- 172.21.15.28:0/1649607717 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f9a800a3d20 msgr2=0x7f9a800a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:56.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.726+0000 7f9a9b200640 1 --2- 172.21.15.28:0/1649607717 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f9a800a3d20 0x7f9a800a61e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9a94005e30 tx=0x7f9a94005c50 comp rx=0 tx=0).stop 2024-04-23T14:47:56.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.726+0000 7f9a9b200640 1 -- 172.21.15.28:0/1649607717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a9c0743a0 msgr2=0x7f9a9c19ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:56.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.726+0000 7f9a9b200640 1 --2- 172.21.15.28:0/1649607717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a9c0743a0 0x7f9a9c19ed90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c00e9f0 tx=0x7f9a8c00edf0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.726+0000 7f9a9b200640 1 -- 172.21.15.28:0/1649607717 shutdown_connections 2024-04-23T14:47:56.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.726+0000 7f9a9b200640 1 --2- 172.21.15.28:0/1649607717 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f9a800a3d20 0x7f9a800a61e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.726+0000 7f9a9b200640 1 --2- 172.21.15.28:0/1649607717 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a9c074cc0 0x7f9a9c19f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.726+0000 7f9a9b200640 1 --2- 172.21.15.28:0/1649607717 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a9c112eb0 0x7f9a9c1a3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.726+0000 7f9a9b200640 1 --2- 172.21.15.28:0/1649607717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a9c0743a0 0x7f9a9c19ed90 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:56.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.726+0000 7f9a9b200640 1 -- 172.21.15.28:0/1649607717 >> 172.21.15.28:0/1649607717 conn(0x7f9a9c0fd9d0 msgr2=0x7f9a9c0ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:56.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.727+0000 7f9a9b200640 1 -- 172.21.15.28:0/1649607717 shutdown_connections 2024-04-23T14:47:56.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:56.727+0000 7f9a9b200640 1 -- 172.21.15.28:0/1649607717 wait complete. 2024-04-23T14:47:56.744 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:56 smithi028 ceph-mon[27810]: pgmap v120: 97 pgs: 97 active+clean; 142 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 2.5 MiB/s wr, 164 op/s 2024-04-23T14:47:56.744 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:56 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3684962810' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:47:57.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:56 smithi105 ceph-mon[32555]: pgmap v120: 97 pgs: 97 active+clean; 142 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 2.5 MiB/s wr, 164 op/s 2024-04-23T14:47:57.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:56 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3684962810' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:47:57.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:56 smithi038 ceph-mon[32348]: pgmap v120: 97 pgs: 97 active+clean; 142 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 2.5 MiB/s wr, 164 op/s 2024-04-23T14:47:57.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:56 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3684962810' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:47:57.475 INFO:tasks.workunit.client.0.smithi028.stdout: 1 258972 10.27 MB/sec execute 13 sec latency 29.287 ms 2024-04-23T14:47:57.747 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-23T14:47:57.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:47:57.748 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-23T14:47:58.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.001+0000 7ff3e6e00640 1 -- 172.21.15.28:0/2388238163 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3e0100d60 msgr2=0x7ff3e010a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:58.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.001+0000 7ff3e6e00640 1 --2- 172.21.15.28:0/2388238163 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3e0100d60 0x7ff3e010a120 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8009900 tx=0x7ff3c802a3f0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.001+0000 7ff3e6e00640 1 -- 172.21.15.28:0/2388238163 shutdown_connections 2024-04-23T14:47:58.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.001+0000 7ff3e6e00640 1 --2- 172.21.15.28:0/2388238163 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3e0100d60 0x7ff3e010a120 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.001+0000 7ff3e6e00640 1 --2- 172.21.15.28:0/2388238163 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3e010a800 0x7ff3e01111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.001+0000 7ff3e6e00640 1 --2- 172.21.15.28:0/2388238163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3e01003b0 0x7ff3e0100790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.001+0000 7ff3e6e00640 1 -- 172.21.15.28:0/2388238163 >> 172.21.15.28:0/2388238163 conn(0x7ff3e0077060 msgr2=0x7ff3e00fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:58.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.001+0000 7ff3e6e00640 1 -- 172.21.15.28:0/2388238163 shutdown_connections 2024-04-23T14:47:58.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.002+0000 7ff3e6e00640 1 -- 172.21.15.28:0/2388238163 wait complete. 2024-04-23T14:47:58.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.002+0000 7ff3e6e00640 1 Processor -- start 2024-04-23T14:47:58.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.003+0000 7ff3e6e00640 1 -- start start 2024-04-23T14:47:58.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.003+0000 7ff3e6e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3e01003b0 0x7ff3e01a0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.003+0000 7ff3e6e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3e010a800 0x7ff3e01a0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.003+0000 7ff3e6e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3e019a990 0x7ff3e019ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.003+0000 7ff3e6e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e0114670 con 0x7ff3e010a800 2024-04-23T14:47:58.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.003+0000 7ff3e6e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e01144f0 con 0x7ff3e019a990 2024-04-23T14:47:58.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.003+0000 7ff3e6e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e01147f0 con 0x7ff3e01003b0 2024-04-23T14:47:58.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.003+0000 7ff3d7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3e010a800 0x7ff3e01a0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.003+0000 7ff3e4e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3e019a990 0x7ff3e019ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.003+0000 7ff3d7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3e010a800 0x7ff3e01a0cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32854/0 (socket says 172.21.15.28:32854) 2024-04-23T14:47:58.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.003+0000 7ff3d7e00640 1 -- 172.21.15.28:0/2989974410 learned_addr learned my addr 172.21.15.28:0/2989974410 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:58.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.004+0000 7ff3dfe00640 1 --2- 172.21.15.28:0/2989974410 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3e01003b0 0x7ff3e01a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.004+0000 7ff3e4e00640 1 -- 172.21.15.28:0/2989974410 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3e01003b0 msgr2=0x7ff3e01a0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:58.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.004+0000 7ff3e4e00640 1 --2- 172.21.15.28:0/2989974410 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3e01003b0 0x7ff3e01a0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.004+0000 7ff3e4e00640 1 -- 172.21.15.28:0/2989974410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3e010a800 msgr2=0x7ff3e01a0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:58.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.004+0000 7ff3e4e00640 1 --2- 172.21.15.28:0/2989974410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3e010a800 0x7ff3e01a0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.004+0000 7ff3e4e00640 1 -- 172.21.15.28:0/2989974410 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3e019b4b0 con 0x7ff3e019a990 2024-04-23T14:47:58.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.004+0000 7ff3e4e00640 1 --2- 172.21.15.28:0/2989974410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3e019a990 0x7ff3e019ae40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8005fd0 tx=0x7ff3d800ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:58.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.004+0000 7ff3dd600640 1 -- 172.21.15.28:0/2989974410 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d800cd80 con 0x7ff3e019a990 2024-04-23T14:47:58.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.004+0000 7ff3dd600640 1 -- 172.21.15.28:0/2989974410 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3d8005020 con 0x7ff3e019a990 2024-04-23T14:47:58.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.004+0000 7ff3e6e00640 1 -- 172.21.15.28:0/2989974410 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3e019b7a0 con 0x7ff3e019a990 2024-04-23T14:47:58.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.004+0000 7ff3dd600640 1 -- 172.21.15.28:0/2989974410 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d8010720 con 0x7ff3e019a990 2024-04-23T14:47:58.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.005+0000 7ff3e6e00640 1 -- 172.21.15.28:0/2989974410 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3e01a79f0 con 0x7ff3e019a990 2024-04-23T14:47:58.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.008+0000 7ff3e9bc5740 1 Processor -- start 2024-04-23T14:47:58.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.008+0000 7ff3dd600640 1 -- 172.21.15.28:0/2989974410 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7ff3d80051d0 con 0x7ff3e019a990 2024-04-23T14:47:58.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.008+0000 7ff3e9bc5740 1 -- start start 2024-04-23T14:47:58.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.009+0000 7ff3e9bc5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55732b410090 0x55732b410470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.009+0000 7ff3e9bc5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55732b4109b0 0x55732b41ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.009+0000 7ff3e9bc5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55732b41b080 0x55732b41d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.009+0000 7ff3e9bc5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55732b375e70 con 0x55732b4109b0 2024-04-23T14:47:58.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.009+0000 7ff3e9bc5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55732b31c380 con 0x55732b410090 2024-04-23T14:47:58.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.009+0000 7ff3e9bc5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55732b371e30 con 0x55732b41b080 2024-04-23T14:47:58.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.009+0000 7ff3d7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55732b410090 0x55732b410470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.009+0000 7ff3d7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55732b410090 0x55732b410470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:37444/0 (socket says 172.21.15.28:37444) 2024-04-23T14:47:58.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.009+0000 7ff3d7e00640 1 -- 172.21.15.28:0/4097887356 learned_addr learned my addr 172.21.15.28:0/4097887356 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:58.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.009+0000 7ff3d7e00640 1 --2- 172.21.15.28:0/4097887356 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55732b41b080 0x55732b41d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.009+0000 7ff3dfe00640 1 --2- 172.21.15.28:0/4097887356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55732b4109b0 0x55732b41ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.009+0000 7ff3d7e00640 1 -- 172.21.15.28:0/4097887356 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55732b41b080 msgr2=0x55732b41d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:58.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.009+0000 7ff3d7e00640 1 --2- 172.21.15.28:0/4097887356 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55732b41b080 0x55732b41d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.009+0000 7ff3d7e00640 1 -- 172.21.15.28:0/4097887356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55732b4109b0 msgr2=0x55732b41ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:58.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.009+0000 7ff3d7e00640 1 --2- 172.21.15.28:0/4097887356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55732b4109b0 0x55732b41ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.010+0000 7ff3d7e00640 1 -- 172.21.15.28:0/4097887356 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55732b1507e0 con 0x55732b410090 2024-04-23T14:47:58.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.010+0000 7ff3dd600640 1 --2- 172.21.15.28:0/2989974410 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7ff3c40a3da0 0x7ff3c40a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.010+0000 7ff3d7e00640 1 --2- 172.21.15.28:0/4097887356 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55732b410090 0x55732b410470 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff3c80096c0 tx=0x7ff3c8005b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=687c9bedf7d19038 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:58.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.010+0000 7ff3dd600640 1 -- 172.21.15.28:0/2989974410 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7ff3d80d0d50 con 0x7ff3e019a990 2024-04-23T14:47:58.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.010+0000 7ff3dfe00640 1 --2- 172.21.15.28:0/2989974410 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7ff3c40a3da0 0x7ff3c40a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.010+0000 7ff3d5600640 1 -- 172.21.15.28:0/4097887356 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c8035030 con 0x55732b410090 2024-04-23T14:47:58.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.010+0000 7ff3d5600640 1 -- 172.21.15.28:0/4097887356 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3c80046c0 con 0x55732b410090 2024-04-23T14:47:58.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.010+0000 7ff3d5600640 1 -- 172.21.15.28:0/4097887356 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c8003850 con 0x55732b410090 2024-04-23T14:47:58.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.011+0000 7ff3e9bc5740 1 -- 172.21.15.28:0/4097887356 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55732b410090 msgr2=0x55732b410470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:58.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.011+0000 7ff3dfe00640 1 --2- 172.21.15.28:0/2989974410 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7ff3c40a3da0 0x7ff3c40a6260 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x55732b411650 tx=0x7ff3d0005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.011+0000 7ff3e9bc5740 1 --2- 172.21.15.28:0/4097887356 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55732b410090 0x55732b410470 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff3c80096c0 tx=0x7ff3c8005b50 comp rx=0 tx=0).stop 2024-04-23T14:47:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.011+0000 7ff3e9bc5740 1 -- 172.21.15.28:0/4097887356 shutdown_connections 2024-04-23T14:47:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.011+0000 7ff3e9bc5740 1 --2- 172.21.15.28:0/4097887356 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55732b41b080 0x55732b41d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.011+0000 7ff3e9bc5740 1 --2- 172.21.15.28:0/4097887356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55732b4109b0 0x55732b41ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.011+0000 7ff3e9bc5740 1 --2- 172.21.15.28:0/4097887356 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55732b410090 0x55732b410470 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.011+0000 7ff3e9bc5740 1 -- 172.21.15.28:0/4097887356 >> 172.21.15.28:0/4097887356 conn(0x55732b40c3b0 msgr2=0x55732b40c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.011+0000 7ff3e9bc5740 1 -- 172.21.15.28:0/4097887356 shutdown_connections 2024-04-23T14:47:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.011+0000 7ff3e9bc5740 1 -- 172.21.15.28:0/4097887356 wait complete. 2024-04-23T14:47:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.012+0000 7ff3e9bc5740 1 -- start start 2024-04-23T14:47:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.012+0000 7ff3e9bc5740 1 Processor -- start 2024-04-23T14:47:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.012+0000 7ff3e9bc5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55732b410090 0x55732b418060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.013+0000 7ff3e9bc5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55732b4109b0 0x55732b4185a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.013+0000 7ff3e9bc5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55732b41b080 0x55732b418ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.013+0000 7ff3e9bc5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55732b41de10 con 0x55732b41b080 2024-04-23T14:47:58.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.013+0000 7ff3e9bc5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55732b32f660 con 0x55732b4109b0 2024-04-23T14:47:58.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.013+0000 7ff3e9bc5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55732b41df90 con 0x55732b410090 2024-04-23T14:47:58.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.013+0000 7ff3e9bc5740 10 client.0 _ll_register_callbacks cb 0x55732b31daa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:47:58.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.013+0000 7ff3d7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55732b4109b0 0x55732b4185a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.013+0000 7ff3d7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55732b4109b0 0x55732b4185a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:37454/0 (socket says 172.21.15.28:37454) 2024-04-23T14:47:58.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.013+0000 7ff3d7e00640 1 -- 172.21.15.28:0/2943895828 learned_addr learned my addr 172.21.15.28:0/2943895828 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:58.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.013+0000 7ff3dfe00640 1 --2- 172.21.15.28:0/2943895828 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55732b410090 0x55732b418060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.013+0000 7ff3e4e00640 1 --2- 172.21.15.28:0/2943895828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55732b41b080 0x55732b418ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.013+0000 7ff3d7e00640 1 -- 172.21.15.28:0/2943895828 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55732b410090 msgr2=0x55732b418060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:58.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.013+0000 7ff3d7e00640 1 --2- 172.21.15.28:0/2943895828 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55732b410090 0x55732b418060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.013+0000 7ff3d7e00640 1 -- 172.21.15.28:0/2943895828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55732b41b080 msgr2=0x55732b418ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:58.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.013+0000 7ff3d7e00640 1 --2- 172.21.15.28:0/2943895828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55732b41b080 0x55732b418ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.013+0000 7ff3d7e00640 1 -- 172.21.15.28:0/2943895828 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55732b1b0f70 con 0x55732b4109b0 2024-04-23T14:47:58.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.014+0000 7ff3d7e00640 1 --2- 172.21.15.28:0/2943895828 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55732b4109b0 0x55732b4185a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8004ad0 tx=0x7ff3c802a3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4aa87b6d4bddc7af server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:58.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.014+0000 7ff3ce000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:47:58.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.014+0000 7ff3ce000640 1 -- 172.21.15.28:0/2943895828 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c804e070 con 0x55732b4109b0 2024-04-23T14:47:58.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.014+0000 7ff3ce000640 1 -- 172.21.15.28:0/2943895828 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7ff3c8005230 con 0x55732b4109b0 2024-04-23T14:47:58.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.014+0000 7ff3e9bc5740 1 -- 172.21.15.28:0/2943895828 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3c80046c0 con 0x55732b4109b0 2024-04-23T14:47:58.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.014+0000 7ff3ce000640 1 -- 172.21.15.28:0/2943895828 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7ff3c805a050 con 0x55732b4109b0 2024-04-23T14:47:58.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.014+0000 7ff3e9bc5740 10 client.25883 fetch_fsmap learned FSMap version 20 2024-04-23T14:47:58.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.014+0000 7ff3e9bc5740 1 -- 172.21.15.28:0/2943895828 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55732b048170 con 0x55732b4109b0 2024-04-23T14:47:58.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.015+0000 7ff3ce000640 1 -- 172.21.15.28:0/2943895828 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3c80046c0 con 0x55732b4109b0 2024-04-23T14:47:58.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.015+0000 7ff3ce000640 1 -- 172.21.15.28:0/2943895828 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7ff3c80035f0 con 0x55732b4109b0 2024-04-23T14:47:58.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.015+0000 7ff3e9bc5740 10 client.25883 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:47:58.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.015+0000 7ff3e9bc5740 10 client.25883 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:47:58.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.015+0000 7ff3e9bc5740 20 client.25883 populate_metadata read hostname 'smithi028' 2024-04-23T14:47:58.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.015+0000 7ff3e9bc5740 1 --2- 172.21.15.28:0/2943895828 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55732b42b280 0x55732b42d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.015+0000 7ff3e9bc5740 4 client.25883 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:47:58.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.015+0000 7ff3e9bc5740 1 -- 172.21.15.28:0/2943895828 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55732b383890 con 0x55732b42b280 2024-04-23T14:47:58.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.015+0000 7ff3dfe00640 1 --2- 172.21.15.28:0/2943895828 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55732b42b280 0x55732b42d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.016+0000 7ff3dfe00640 1 --2- 172.21.15.28:0/2943895828 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55732b42b280 0x55732b42d670 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22fba944e45da27b server_cookie=d6672df4465ad297 in_seq=0 out_seq=0 2024-04-23T14:47:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.016+0000 7ff3ce000640 10 client.25883 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:47:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.017+0000 7ff3ce000640 1 -- 172.21.15.28:0/2943895828 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55732b383890 con 0x55732b42b280 2024-04-23T14:47:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.017+0000 7ff3ce000640 10 client.25883 handle_command_reply: tid=0 2024-04-23T14:47:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.017+0000 7ff3e9bc5740 1 client.25883 shutdown 2024-04-23T14:47:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.018+0000 7ff3e9bc5740 1 -- 172.21.15.28:0/2943895828 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55732b4109b0 msgr2=0x55732b4185a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.018+0000 7ff3e9bc5740 1 --2- 172.21.15.28:0/2943895828 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55732b4109b0 0x55732b4185a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8004ad0 tx=0x7ff3c802a3f0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.018+0000 7ff3e9bc5740 1 -- 172.21.15.28:0/2943895828 shutdown_connections 2024-04-23T14:47:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.018+0000 7ff3e9bc5740 1 --2- 172.21.15.28:0/2943895828 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55732b42b280 0x55732b42d670 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.018+0000 7ff3e9bc5740 1 --2- 172.21.15.28:0/2943895828 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55732b4109b0 0x55732b4185a0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.018+0000 7ff3e9bc5740 1 --2- 172.21.15.28:0/2943895828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55732b41b080 0x55732b418ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.018+0000 7ff3e9bc5740 1 --2- 172.21.15.28:0/2943895828 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55732b410090 0x55732b418060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.018+0000 7ff3ce000640 0 client.25883 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:47:58.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.018+0000 7ff3e9bc5740 1 -- 172.21.15.28:0/2943895828 >> 172.21.15.28:0/2943895828 conn(0x55732b40c3b0 msgr2=0x55732b412090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:58.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.018+0000 7ff3e9bc5740 1 -- 172.21.15.28:0/2943895828 shutdown_connections 2024-04-23T14:47:58.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.018+0000 7ff3e9bc5740 1 -- 172.21.15.28:0/2943895828 wait complete. 2024-04-23T14:47:58.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.018+0000 7ff3e9bc5740 20 client.25883 trim_cache size 0 max 16384 2024-04-23T14:47:58.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.039+0000 7ff3e9bc5740 1 Processor -- start 2024-04-23T14:47:58.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.039+0000 7ff3e9bc5740 1 -- start start 2024-04-23T14:47:58.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.040+0000 7ff3e9bc5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55732b410090 0x55732b4127f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.040+0000 7ff3e9bc5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55732b4109b0 0x55732b40b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.040+0000 7ff3e9bc5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55732b41b080 0x55732b430690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.040+0000 7ff3e9bc5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e01adf00 con 0x55732b410090 2024-04-23T14:47:58.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.040+0000 7ff3e9bc5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e01b0780 con 0x55732b4109b0 2024-04-23T14:47:58.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.040+0000 7ff3e9bc5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e01ae570 con 0x55732b41b080 2024-04-23T14:47:58.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.040+0000 7ff3d7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55732b4109b0 0x55732b40b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.040+0000 7ff3dfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55732b410090 0x55732b4127f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.040+0000 7ff3d7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55732b4109b0 0x55732b40b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:37458/0 (socket says 172.21.15.28:37458) 2024-04-23T14:47:58.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.040+0000 7ff3d7e00640 1 -- 172.21.15.28:0/147104563 learned_addr learned my addr 172.21.15.28:0/147104563 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:58.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.040+0000 7ff3e4e00640 1 --2- 172.21.15.28:0/147104563 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55732b41b080 0x55732b430690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.041+0000 7ff3dfe00640 1 -- 172.21.15.28:0/147104563 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55732b41b080 msgr2=0x55732b430690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:58.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.041+0000 7ff3dfe00640 1 --2- 172.21.15.28:0/147104563 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55732b41b080 0x55732b430690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.041+0000 7ff3dfe00640 1 -- 172.21.15.28:0/147104563 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55732b4109b0 msgr2=0x55732b40b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:58.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.041+0000 7ff3dfe00640 1 --2- 172.21.15.28:0/147104563 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55732b4109b0 0x55732b40b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.041+0000 7ff3dfe00640 1 -- 172.21.15.28:0/147104563 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55732b412d30 con 0x55732b410090 2024-04-23T14:47:58.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.041+0000 7ff3d7e00640 1 --2- 172.21.15.28:0/147104563 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55732b4109b0 0x55732b40b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:47:58.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.041+0000 7ff3dfe00640 1 --2- 172.21.15.28:0/147104563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55732b410090 0x55732b4127f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x55732b41e030 tx=0x7ff3d001b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c0b59e6aa4af942 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:58.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.041+0000 7ff3d4c00640 1 -- 172.21.15.28:0/147104563 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d0030070 con 0x55732b410090 2024-04-23T14:47:58.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.041+0000 7ff3d4c00640 1 -- 172.21.15.28:0/147104563 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3d002b900 con 0x55732b410090 2024-04-23T14:47:58.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.041+0000 7ff3d4c00640 1 -- 172.21.15.28:0/147104563 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d0003b90 con 0x55732b410090 2024-04-23T14:47:58.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.042+0000 7ff3e9bc5740 1 -- 172.21.15.28:0/147104563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55732b410090 msgr2=0x55732b4127f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:58.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.042+0000 7ff3e9bc5740 1 --2- 172.21.15.28:0/147104563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55732b410090 0x55732b4127f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x55732b41e030 tx=0x7ff3d001b000 comp rx=0 tx=0).stop 2024-04-23T14:47:58.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.042+0000 7ff3e9bc5740 1 -- 172.21.15.28:0/147104563 shutdown_connections 2024-04-23T14:47:58.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.042+0000 7ff3e9bc5740 1 --2- 172.21.15.28:0/147104563 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55732b4109b0 0x55732b40b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.042+0000 7ff3e9bc5740 1 --2- 172.21.15.28:0/147104563 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55732b41b080 0x55732b430690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.042+0000 7ff3e9bc5740 1 --2- 172.21.15.28:0/147104563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55732b410090 0x55732b4127f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.042+0000 7ff3e9bc5740 1 -- 172.21.15.28:0/147104563 >> 172.21.15.28:0/147104563 conn(0x55732b40c3b0 msgr2=0x55732b40ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:58.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.042+0000 7ff3e9bc5740 1 -- 172.21.15.28:0/147104563 shutdown_connections 2024-04-23T14:47:58.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.042+0000 7ff3e9bc5740 1 -- 172.21.15.28:0/147104563 wait complete. 2024-04-23T14:47:58.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.043+0000 7ff3e9bc5740 1 -- start start 2024-04-23T14:47:58.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.043+0000 7ff3e9bc5740 1 Processor -- start 2024-04-23T14:47:58.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.043+0000 7ff3e9bc5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55732b410090 0x55732b42e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.043+0000 7ff3e9bc5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55732b4109b0 0x55732b42e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.043+0000 7ff3e9bc5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55732b41b080 0x55732b490460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.043+0000 7ff3e9bc5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e01aec40 con 0x55732b41b080 2024-04-23T14:47:58.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.043+0000 7ff3e9bc5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e01b00b0 con 0x55732b4109b0 2024-04-23T14:47:58.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.043+0000 7ff3e9bc5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e01af310 con 0x55732b410090 2024-04-23T14:47:58.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.043+0000 7ff3e9bc5740 10 client.0 _ll_register_callbacks cb 0x7ff3d0009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:47:58.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.044+0000 7ff3dfe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55732b410090 0x55732b42e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.044+0000 7ff3dfe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55732b410090 0x55732b42e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:35282/0 (socket says 172.21.15.28:35282) 2024-04-23T14:47:58.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.044+0000 7ff3dfe00640 1 -- 172.21.15.28:0/2096799218 learned_addr learned my addr 172.21.15.28:0/2096799218 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:58.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.044+0000 7ff3d7e00640 1 --2- 172.21.15.28:0/2096799218 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55732b4109b0 0x55732b42e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.044+0000 7ff3e4e00640 1 --2- 172.21.15.28:0/2096799218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55732b41b080 0x55732b490460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.044+0000 7ff3dfe00640 1 -- 172.21.15.28:0/2096799218 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55732b4109b0 msgr2=0x55732b42e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:58.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.044+0000 7ff3dfe00640 1 --2- 172.21.15.28:0/2096799218 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55732b4109b0 0x55732b42e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.044+0000 7ff3dfe00640 1 -- 172.21.15.28:0/2096799218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55732b41b080 msgr2=0x55732b490460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:58.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.044+0000 7ff3dfe00640 1 --2- 172.21.15.28:0/2096799218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55732b41b080 0x55732b490460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.044+0000 7ff3dfe00640 1 -- 172.21.15.28:0/2096799218 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55732b42f2b0 con 0x55732b410090 2024-04-23T14:47:58.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.044+0000 7ff3e4e00640 1 --2- 172.21.15.28:0/2096799218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55732b41b080 0x55732b490460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:47:58.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.044+0000 7ff3dfe00640 1 --2- 172.21.15.28:0/2096799218 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55732b410090 0x55732b42e470 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0005890 tx=0x7ff3d0009f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=541135a21b6cc3a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:58.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.044+0000 7ff3ce000640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:47:58.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.045+0000 7ff3ce000640 1 -- 172.21.15.28:0/2096799218 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d0030040 con 0x55732b410090 2024-04-23T14:47:58.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.045+0000 7ff3ce000640 1 -- 172.21.15.28:0/2096799218 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7ff3d002b420 con 0x55732b410090 2024-04-23T14:47:58.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.045+0000 7ff3e9bc5740 1 -- 172.21.15.28:0/2096799218 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3d002b900 con 0x55732b410090 2024-04-23T14:47:58.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.045+0000 7ff3ce000640 1 -- 172.21.15.28:0/2096799218 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7ff3d0042050 con 0x55732b410090 2024-04-23T14:47:58.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.045+0000 7ff3e9bc5740 10 client.25876 fetch_fsmap learned FSMap version 20 2024-04-23T14:47:58.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.045+0000 7ff3e9bc5740 1 -- 172.21.15.28:0/2096799218 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55732b491ae0 con 0x55732b410090 2024-04-23T14:47:58.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.045+0000 7ff3ce000640 1 -- 172.21.15.28:0/2096799218 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3d002b900 con 0x55732b410090 2024-04-23T14:47:58.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.046+0000 7ff3ce000640 1 -- 172.21.15.28:0/2096799218 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7ff3d0033070 con 0x55732b410090 2024-04-23T14:47:58.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.046+0000 7ff3e9bc5740 10 client.25876 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:47:58.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.046+0000 7ff3e9bc5740 10 client.25876 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:47:58.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.046+0000 7ff3e9bc5740 20 client.25876 populate_metadata read hostname 'smithi028' 2024-04-23T14:47:58.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.046+0000 7ff3e9bc5740 1 --2- 172.21.15.28:0/2096799218 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55732b492130 0x55732b494550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.046+0000 7ff3e9bc5740 4 client.25876 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-23T14:47:58.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.046+0000 7ff3e9bc5740 1 -- 172.21.15.28:0/2096799218 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55732b37d700 con 0x55732b492130 2024-04-23T14:47:58.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.046+0000 7ff3d7e00640 1 --2- 172.21.15.28:0/2096799218 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55732b492130 0x55732b494550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.046+0000 7ff3d7e00640 1 --2- 172.21.15.28:0/2096799218 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55732b492130 0x55732b494550 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfd5b96011a76d7c server_cookie=9fb525fedcd0e813 in_seq=0 out_seq=0 2024-04-23T14:47:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.046+0000 7ff3ce000640 10 client.25876 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:47:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.047+0000 7ff3ce000640 1 -- 172.21.15.28:0/2096799218 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55732b37d700 con 0x55732b492130 2024-04-23T14:47:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.047+0000 7ff3ce000640 10 client.25876 handle_command_reply: tid=0 2024-04-23T14:47:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.047+0000 7ff3e9bc5740 1 client.25876 shutdown 2024-04-23T14:47:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.048+0000 7ff3e9bc5740 1 -- 172.21.15.28:0/2096799218 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55732b410090 msgr2=0x55732b42e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.048+0000 7ff3e9bc5740 1 --2- 172.21.15.28:0/2096799218 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55732b410090 0x55732b42e470 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0005890 tx=0x7ff3d0009f70 comp rx=0 tx=0).stop 2024-04-23T14:47:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.048+0000 7ff3e9bc5740 1 -- 172.21.15.28:0/2096799218 shutdown_connections 2024-04-23T14:47:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.048+0000 7ff3e9bc5740 1 --2- 172.21.15.28:0/2096799218 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55732b492130 0x55732b494550 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.048+0000 7ff3e9bc5740 1 --2- 172.21.15.28:0/2096799218 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55732b4109b0 0x55732b42e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.048+0000 7ff3ce000640 0 client.25876 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:47:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.048+0000 7ff3e9bc5740 1 --2- 172.21.15.28:0/2096799218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55732b41b080 0x55732b490460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.048+0000 7ff3e9bc5740 1 --2- 172.21.15.28:0/2096799218 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55732b410090 0x55732b42e470 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.048+0000 7ff3e9bc5740 1 -- 172.21.15.28:0/2096799218 >> 172.21.15.28:0/2096799218 conn(0x55732b40c3b0 msgr2=0x55732b40b510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.048+0000 7ff3e9bc5740 1 -- 172.21.15.28:0/2096799218 shutdown_connections 2024-04-23T14:47:58.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.048+0000 7ff3e9bc5740 1 -- 172.21.15.28:0/2096799218 wait complete. 2024-04-23T14:47:58.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.048+0000 7ff3e9bc5740 20 client.25876 trim_cache size 0 max 16384 2024-04-23T14:47:58.052 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:47:58.052 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-23T14:47:58.052 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "fcd88ff1-6c17-4111-bcd3-b274dc4c8c1b", 2024-04-23T14:47:58.052 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-23T14:47:58.052 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:47:58.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.049+0000 7ff3e6e00640 1 -- 172.21.15.28:0/2989974410 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7ff3c40a3da0 msgr2=0x7ff3c40a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:58.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.049+0000 7ff3e6e00640 1 --2- 172.21.15.28:0/2989974410 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7ff3c40a3da0 0x7ff3c40a6260 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x55732b411650 tx=0x7ff3d0005f90 comp rx=0 tx=0).stop 2024-04-23T14:47:58.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.049+0000 7ff3e6e00640 1 -- 172.21.15.28:0/2989974410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3e019a990 msgr2=0x7ff3e019ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:58.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.049+0000 7ff3e6e00640 1 --2- 172.21.15.28:0/2989974410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3e019a990 0x7ff3e019ae40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8005fd0 tx=0x7ff3d800ee60 comp rx=0 tx=0).stop 2024-04-23T14:47:58.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.049+0000 7ff3e6e00640 1 -- 172.21.15.28:0/2989974410 shutdown_connections 2024-04-23T14:47:58.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.049+0000 7ff3e6e00640 1 --2- 172.21.15.28:0/2989974410 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7ff3c40a3da0 0x7ff3c40a6260 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.049+0000 7ff3e6e00640 1 --2- 172.21.15.28:0/2989974410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3e019a990 0x7ff3e019ae40 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.049+0000 7ff3e6e00640 1 --2- 172.21.15.28:0/2989974410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3e010a800 0x7ff3e01a0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.049+0000 7ff3e6e00640 1 --2- 172.21.15.28:0/2989974410 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3e01003b0 0x7ff3e01a0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.049+0000 7ff3e6e00640 1 -- 172.21.15.28:0/2989974410 >> 172.21.15.28:0/2989974410 conn(0x7ff3e0077060 msgr2=0x7ff3e00fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:58.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.049+0000 7ff3e6e00640 1 -- 172.21.15.28:0/2989974410 shutdown_connections 2024-04-23T14:47:58.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.049+0000 7ff3e6e00640 1 -- 172.21.15.28:0/2989974410 wait complete. 2024-04-23T14:47:58.070 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:47:58.070 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:47:58.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.281+0000 7f2173000640 1 -- 172.21.15.28:0/87841431 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f216c104cd0 msgr2=0x7f216c108d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:58.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.281+0000 7f2173000640 1 --2- 172.21.15.28:0/87841431 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f216c104cd0 0x7f216c108d10 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2158009900 tx=0x7f215802a3f0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.281+0000 7f2173000640 1 -- 172.21.15.28:0/87841431 shutdown_connections 2024-04-23T14:47:58.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.282+0000 7f2173000640 1 --2- 172.21.15.28:0/87841431 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f216c104cd0 0x7f216c108d10 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.282+0000 7f2173000640 1 --2- 172.21.15.28:0/87841431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f216c1093f0 0x7f216c110fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.282+0000 7f2173000640 1 --2- 172.21.15.28:0/87841431 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f216c104320 0x7f216c104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.282+0000 7f2173000640 1 -- 172.21.15.28:0/87841431 >> 172.21.15.28:0/87841431 conn(0x7f216c0ffb40 msgr2=0x7f216c101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:58.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.282+0000 7f2173000640 1 -- 172.21.15.28:0/87841431 shutdown_connections 2024-04-23T14:47:58.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.282+0000 7f2173000640 1 -- 172.21.15.28:0/87841431 wait complete. 2024-04-23T14:47:58.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.283+0000 7f2173000640 1 Processor -- start 2024-04-23T14:47:58.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.283+0000 7f2173000640 1 -- start start 2024-04-23T14:47:58.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.284+0000 7f2173000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f216c104320 0x7f216c1a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.284+0000 7f2173000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f216c1093f0 0x7f216c1a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.284+0000 7f2173000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f216c19a7b0 0x7f216c19ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.284+0000 7f2173000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f216c1144b0 con 0x7f216c1093f0 2024-04-23T14:47:58.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.284+0000 7f2173000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f216c114330 con 0x7f216c19a7b0 2024-04-23T14:47:58.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.284+0000 7f2173000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f216c114630 con 0x7f216c104320 2024-04-23T14:47:58.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.284+0000 7f2171c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f216c104320 0x7f216c1a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.284+0000 7f2171c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f216c104320 0x7f216c1a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:35292/0 (socket says 172.21.15.28:35292) 2024-04-23T14:47:58.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.284+0000 7f2171c00640 1 -- 172.21.15.28:0/3713636062 learned_addr learned my addr 172.21.15.28:0/3713636062 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:58.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.284+0000 7f2172600640 1 --2- 172.21.15.28:0/3713636062 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f216c19a7b0 0x7f216c19ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.284+0000 7f2171200640 1 --2- 172.21.15.28:0/3713636062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f216c1093f0 0x7f216c1a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.284+0000 7f2171c00640 1 -- 172.21.15.28:0/3713636062 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f216c19a7b0 msgr2=0x7f216c19ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:58.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.284+0000 7f2171c00640 1 --2- 172.21.15.28:0/3713636062 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f216c19a7b0 0x7f216c19ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.284+0000 7f2171c00640 1 -- 172.21.15.28:0/3713636062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f216c1093f0 msgr2=0x7f216c1a0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:58.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.284+0000 7f2171c00640 1 --2- 172.21.15.28:0/3713636062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f216c1093f0 0x7f216c1a0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.285+0000 7f2171c00640 1 -- 172.21.15.28:0/3713636062 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f216c19b340 con 0x7f216c104320 2024-04-23T14:47:58.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.285+0000 7f2171c00640 1 --2- 172.21.15.28:0/3713636062 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f216c104320 0x7f216c1a0640 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f216000e9f0 tx=0x7f216000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:58.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.285+0000 7f216a000640 1 -- 172.21.15.28:0/3713636062 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216000cd80 con 0x7f216c104320 2024-04-23T14:47:58.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.285+0000 7f216a000640 1 -- 172.21.15.28:0/3713636062 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2160010910 con 0x7f216c104320 2024-04-23T14:47:58.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.285+0000 7f2173000640 1 -- 172.21.15.28:0/3713636062 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f216c19b630 con 0x7f216c104320 2024-04-23T14:47:58.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.285+0000 7f216a000640 1 -- 172.21.15.28:0/3713636062 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21600188f0 con 0x7f216c104320 2024-04-23T14:47:58.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.285+0000 7f2173000640 1 -- 172.21.15.28:0/3713636062 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f216c1a77e0 con 0x7f216c104320 2024-04-23T14:47:58.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.288+0000 7f216a000640 1 -- 172.21.15.28:0/3713636062 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f2160010ac0 con 0x7f216c104320 2024-04-23T14:47:58.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.289+0000 7f21773bd740 1 Processor -- start 2024-04-23T14:47:58.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.289+0000 7f21773bd740 1 -- start start 2024-04-23T14:47:58.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.290+0000 7f21773bd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d8f5531a30 0x55d8f5531e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.290+0000 7f21773bd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8f5532350 0x55d8f553c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.290+0000 7f21773bd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d8f553ca20 0x55d8f553ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.290+0000 7f21773bd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d8f54971f0 con 0x55d8f5532350 2024-04-23T14:47:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.290+0000 7f21773bd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d8f543d350 con 0x55d8f5531a30 2024-04-23T14:47:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.290+0000 7f21773bd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d8f54931b0 con 0x55d8f553ca20 2024-04-23T14:47:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.290+0000 7f2172600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d8f5531a30 0x55d8f5531e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.290+0000 7f2172600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d8f5531a30 0x55d8f5531e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:37486/0 (socket says 172.21.15.28:37486) 2024-04-23T14:47:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.290+0000 7f2172600640 1 -- 172.21.15.28:0/2662820925 learned_addr learned my addr 172.21.15.28:0/2662820925 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.290+0000 7f2171c00640 1 --2- 172.21.15.28:0/2662820925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8f5532350 0x55d8f553c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.290+0000 7f2171200640 1 --2- 172.21.15.28:0/2662820925 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d8f553ca20 0x55d8f553ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.290+0000 7f216a000640 1 --2- 172.21.15.28:0/3713636062 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f21540a3d20 0x7f21540a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.290+0000 7f2172600640 1 -- 172.21.15.28:0/2662820925 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d8f553ca20 msgr2=0x55d8f553ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.291+0000 7f2172600640 1 --2- 172.21.15.28:0/2662820925 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d8f553ca20 0x55d8f553ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.291+0000 7f2172600640 1 -- 172.21.15.28:0/2662820925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8f5532350 msgr2=0x55d8f553c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.291+0000 7f2172600640 1 --2- 172.21.15.28:0/2662820925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8f5532350 0x55d8f553c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.291+0000 7f2172600640 1 -- 172.21.15.28:0/2662820925 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8f5491f50 con 0x55d8f5531a30 2024-04-23T14:47:58.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.291+0000 7f216a000640 1 -- 172.21.15.28:0/3713636062 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f21600d0bf0 con 0x7f216c104320 2024-04-23T14:47:58.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.291+0000 7f2172600640 1 --2- 172.21.15.28:0/3713636062 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f21540a3d20 0x7f21540a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.291+0000 7f2172600640 1 --2- 172.21.15.28:0/2662820925 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d8f5531a30 0x55d8f5531e10 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2164009f50 tx=0x7f21640118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd1084db1ac58913 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:58.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.291+0000 7f215ea00640 1 -- 172.21.15.28:0/2662820925 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2164004250 con 0x55d8f5531a30 2024-04-23T14:47:58.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.291+0000 7f215ea00640 1 -- 172.21.15.28:0/2662820925 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f216400f480 con 0x55d8f5531a30 2024-04-23T14:47:58.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.291+0000 7f215ea00640 1 -- 172.21.15.28:0/2662820925 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216401b650 con 0x55d8f5531a30 2024-04-23T14:47:58.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.292+0000 7f21773bd740 1 -- 172.21.15.28:0/2662820925 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d8f5531a30 msgr2=0x55d8f5531e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:58.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.292+0000 7f21773bd740 1 --2- 172.21.15.28:0/2662820925 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d8f5531a30 0x55d8f5531e10 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2164009f50 tx=0x7f21640118d0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.292+0000 7f2172600640 1 --2- 172.21.15.28:0/3713636062 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f21540a3d20 0x7f21540a61e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2164012fd0 tx=0x7f2164004990 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:58.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.292+0000 7f21773bd740 1 -- 172.21.15.28:0/2662820925 shutdown_connections 2024-04-23T14:47:58.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.292+0000 7f21773bd740 1 --2- 172.21.15.28:0/2662820925 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d8f553ca20 0x55d8f553ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.292+0000 7f21773bd740 1 --2- 172.21.15.28:0/2662820925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8f5532350 0x55d8f553c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.292+0000 7f21773bd740 1 --2- 172.21.15.28:0/2662820925 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d8f5531a30 0x55d8f5531e10 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.292+0000 7f21773bd740 1 -- 172.21.15.28:0/2662820925 >> 172.21.15.28:0/2662820925 conn(0x55d8f552dd50 msgr2=0x55d8f552e130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:58.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.292+0000 7f21773bd740 1 -- 172.21.15.28:0/2662820925 shutdown_connections 2024-04-23T14:47:58.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.292+0000 7f21773bd740 1 -- 172.21.15.28:0/2662820925 wait complete. 2024-04-23T14:47:58.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.293+0000 7f21773bd740 1 -- start start 2024-04-23T14:47:58.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.293+0000 7f21773bd740 1 Processor -- start 2024-04-23T14:47:58.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.294+0000 7f21773bd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8f5531a30 0x55d8f5539910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.294+0000 7f21773bd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d8f5532350 0x55d8f5539e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.294+0000 7f21773bd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d8f553ca20 0x55d8f553a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.294+0000 7f21773bd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d8f553f810 con 0x55d8f5531a30 2024-04-23T14:47:58.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.294+0000 7f21773bd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d8f5450690 con 0x55d8f553ca20 2024-04-23T14:47:58.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.294+0000 7f21773bd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d8f553f990 con 0x55d8f5532350 2024-04-23T14:47:58.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.294+0000 7f21773bd740 10 client.0 _ll_register_callbacks cb 0x55d8f548c650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:47:58.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.294+0000 7f2171c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8f5531a30 0x55d8f5539910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.294+0000 7f2172600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d8f553ca20 0x55d8f553a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.294+0000 7f2171200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d8f5532350 0x55d8f5539e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.294+0000 7f2172600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d8f553ca20 0x55d8f553a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:37490/0 (socket says 172.21.15.28:37490) 2024-04-23T14:47:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.294+0000 7f2171c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8f5531a30 0x55d8f5539910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32930/0 (socket says 172.21.15.28:32930) 2024-04-23T14:47:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.294+0000 7f2171200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d8f5532350 0x55d8f5539e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:35306/0 (socket says 172.21.15.28:35306) 2024-04-23T14:47:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.294+0000 7f2172600640 1 -- 172.21.15.28:0/4174833826 learned_addr learned my addr 172.21.15.28:0/4174833826 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.294+0000 7f2171200640 1 -- 172.21.15.28:0/4174833826 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d8f553ca20 msgr2=0x55d8f553a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.294+0000 7f2171200640 1 --2- 172.21.15.28:0/4174833826 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d8f553ca20 0x55d8f553a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.294+0000 7f2171200640 1 -- 172.21.15.28:0/4174833826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8f5531a30 msgr2=0x55d8f5539910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.294+0000 7f2171200640 1 --2- 172.21.15.28:0/4174833826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8f5531a30 0x55d8f5539910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.294+0000 7f2171200640 1 -- 172.21.15.28:0/4174833826 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8f544f920 con 0x55d8f5532350 2024-04-23T14:47:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.295+0000 7f2171200640 1 --2- 172.21.15.28:0/4174833826 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d8f5532350 0x55d8f5539e50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f215802a910 tx=0x7f21580043b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6987f4981378306e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.295+0000 7f2153400640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:47:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.295+0000 7f2153400640 1 -- 172.21.15.28:0/4174833826 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2158035040 con 0x55d8f5532350 2024-04-23T14:47:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.295+0000 7f21773bd740 1 -- 172.21.15.28:0/4174833826 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f216400f480 con 0x55d8f5532350 2024-04-23T14:47:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.295+0000 7f2153400640 1 -- 172.21.15.28:0/4174833826 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f2158005450 con 0x55d8f5532350 2024-04-23T14:47:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.296+0000 7f2153400640 1 -- 172.21.15.28:0/4174833826 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f2158005650 con 0x55d8f5532350 2024-04-23T14:47:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.296+0000 7f21773bd740 10 client.25888 fetch_fsmap learned FSMap version 20 2024-04-23T14:47:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.296+0000 7f21773bd740 1 -- 172.21.15.28:0/4174833826 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55d8f52737d0 con 0x55d8f5532350 2024-04-23T14:47:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.296+0000 7f2153400640 1 -- 172.21.15.28:0/4174833826 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f216400f480 con 0x55d8f5532350 2024-04-23T14:47:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.296+0000 7f2153400640 1 -- 172.21.15.28:0/4174833826 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f21580318e0 con 0x55d8f5532350 2024-04-23T14:47:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.296+0000 7f21773bd740 10 client.25888 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:47:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.296+0000 7f21773bd740 10 client.25888 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:47:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.296+0000 7f21773bd740 20 client.25888 populate_metadata read hostname 'smithi028' 2024-04-23T14:47:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.296+0000 7f21773bd740 1 --2- 172.21.15.28:0/4174833826 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d8f554cd00 0x55d8f554f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.296+0000 7f21773bd740 4 client.25888 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:47:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.296+0000 7f21773bd740 1 -- 172.21.15.28:0/4174833826 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d8f54a4c10 con 0x55d8f554cd00 2024-04-23T14:47:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.297+0000 7f2171c00640 1 --2- 172.21.15.28:0/4174833826 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d8f554cd00 0x55d8f554f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.297+0000 7f2171c00640 1 --2- 172.21.15.28:0/4174833826 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d8f554cd00 0x55d8f554f180 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36d494492154f3a4 server_cookie=ea8d6f40acd3ecdf in_seq=0 out_seq=0 2024-04-23T14:47:58.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.297+0000 7f2153400640 10 client.25888 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:47:58.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.298+0000 7f2153400640 1 -- 172.21.15.28:0/4174833826 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d8f54a4c10 con 0x55d8f554cd00 2024-04-23T14:47:58.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.298+0000 7f2153400640 10 client.25888 handle_command_reply: tid=0 2024-04-23T14:47:58.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.298+0000 7f21773bd740 1 client.25888 shutdown 2024-04-23T14:47:58.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.299+0000 7f21773bd740 1 -- 172.21.15.28:0/4174833826 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d8f5532350 msgr2=0x55d8f5539e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:58.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.299+0000 7f21773bd740 1 --2- 172.21.15.28:0/4174833826 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d8f5532350 0x55d8f5539e50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f215802a910 tx=0x7f21580043b0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.299+0000 7f21773bd740 1 -- 172.21.15.28:0/4174833826 shutdown_connections 2024-04-23T14:47:58.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.299+0000 7f21773bd740 1 --2- 172.21.15.28:0/4174833826 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d8f554cd00 0x55d8f554f180 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.299+0000 7f21773bd740 1 --2- 172.21.15.28:0/4174833826 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d8f553ca20 0x55d8f553a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.299+0000 7f2153400640 0 client.25888 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:47:58.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.299+0000 7f21773bd740 1 --2- 172.21.15.28:0/4174833826 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d8f5532350 0x55d8f5539e50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.299+0000 7f21773bd740 1 --2- 172.21.15.28:0/4174833826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8f5531a30 0x55d8f5539910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.299+0000 7f21773bd740 1 -- 172.21.15.28:0/4174833826 >> 172.21.15.28:0/4174833826 conn(0x55d8f552dd50 msgr2=0x55d8f5533b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:58.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.299+0000 7f21773bd740 1 -- 172.21.15.28:0/4174833826 shutdown_connections 2024-04-23T14:47:58.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.299+0000 7f21773bd740 1 -- 172.21.15.28:0/4174833826 wait complete. 2024-04-23T14:47:58.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.299+0000 7f21773bd740 20 client.25888 trim_cache size 0 max 16384 2024-04-23T14:47:58.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.319+0000 7f21773bd740 1 Processor -- start 2024-04-23T14:47:58.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.319+0000 7f21773bd740 1 -- start start 2024-04-23T14:47:58.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.320+0000 7f21773bd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d8f5531a30 0x55d8f55486d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.320+0000 7f21773bd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8f5532350 0x55d8f5528e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.320+0000 7f21773bd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d8f553ca20 0x55d8f5554a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.320+0000 7f21773bd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f216c1add90 con 0x55d8f5532350 2024-04-23T14:47:58.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.320+0000 7f21773bd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f216c1b05f0 con 0x55d8f5531a30 2024-04-23T14:47:58.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.320+0000 7f21773bd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f216c1ae3e0 con 0x55d8f553ca20 2024-04-23T14:47:58.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.320+0000 7f2171c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d8f5531a30 0x55d8f55486d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.320+0000 7f2172600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d8f553ca20 0x55d8f5554a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.320+0000 7f2171c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d8f5531a30 0x55d8f55486d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:37502/0 (socket says 172.21.15.28:37502) 2024-04-23T14:47:58.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.320+0000 7f2171c00640 1 -- 172.21.15.28:0/296759525 learned_addr learned my addr 172.21.15.28:0/296759525 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:58.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.320+0000 7f2171200640 1 --2- 172.21.15.28:0/296759525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8f5532350 0x55d8f5528e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.321+0000 7f2172600640 1 -- 172.21.15.28:0/296759525 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d8f5531a30 msgr2=0x55d8f55486d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:58.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.321+0000 7f2172600640 1 --2- 172.21.15.28:0/296759525 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d8f5531a30 0x55d8f55486d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.321+0000 7f2172600640 1 -- 172.21.15.28:0/296759525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8f5532350 msgr2=0x55d8f5528e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:58.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.321+0000 7f2172600640 1 --2- 172.21.15.28:0/296759525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8f5532350 0x55d8f5528e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.321+0000 7f2172600640 1 -- 172.21.15.28:0/296759525 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8f55388c0 con 0x55d8f553ca20 2024-04-23T14:47:58.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.321+0000 7f2172600640 1 --2- 172.21.15.28:0/296759525 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d8f553ca20 0x55d8f5554a50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2164019780 tx=0x7f2164029000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b94c9b6b43afbad6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:58.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.321+0000 7f215e000640 1 -- 172.21.15.28:0/296759525 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2164009b30 con 0x55d8f553ca20 2024-04-23T14:47:58.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.321+0000 7f215e000640 1 -- 172.21.15.28:0/296759525 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2164019950 con 0x55d8f553ca20 2024-04-23T14:47:58.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.321+0000 7f215e000640 1 -- 172.21.15.28:0/296759525 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2164004140 con 0x55d8f553ca20 2024-04-23T14:47:58.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.322+0000 7f21773bd740 1 -- 172.21.15.28:0/296759525 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d8f553ca20 msgr2=0x55d8f5554a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:58.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.322+0000 7f21773bd740 1 --2- 172.21.15.28:0/296759525 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d8f553ca20 0x55d8f5554a50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2164019780 tx=0x7f2164029000 comp rx=0 tx=0).stop 2024-04-23T14:47:58.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.322+0000 7f21773bd740 1 -- 172.21.15.28:0/296759525 shutdown_connections 2024-04-23T14:47:58.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.322+0000 7f21773bd740 1 --2- 172.21.15.28:0/296759525 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d8f553ca20 0x55d8f5554a50 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.322+0000 7f21773bd740 1 --2- 172.21.15.28:0/296759525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8f5532350 0x55d8f5528e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.322+0000 7f21773bd740 1 --2- 172.21.15.28:0/296759525 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d8f5531a30 0x55d8f55486d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.322+0000 7f21773bd740 1 -- 172.21.15.28:0/296759525 >> 172.21.15.28:0/296759525 conn(0x55d8f552dd50 msgr2=0x55d8f5533b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:58.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.322+0000 7f21773bd740 1 -- 172.21.15.28:0/296759525 shutdown_connections 2024-04-23T14:47:58.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.322+0000 7f21773bd740 1 -- 172.21.15.28:0/296759525 wait complete. 2024-04-23T14:47:58.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.323+0000 7f21773bd740 1 -- start start 2024-04-23T14:47:58.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.323+0000 7f21773bd740 1 Processor -- start 2024-04-23T14:47:58.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.323+0000 7f21773bd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8f5531a30 0x55d8f5540a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.323+0000 7f21773bd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d8f5532350 0x55d8f5540f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.323+0000 7f21773bd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d8f553ca20 0x55d8f55b22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.323+0000 7f21773bd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f216c1aeab0 con 0x55d8f5531a30 2024-04-23T14:47:58.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.323+0000 7f21773bd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f216c1aff20 con 0x55d8f5532350 2024-04-23T14:47:58.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.323+0000 7f21773bd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f216c1af180 con 0x55d8f553ca20 2024-04-23T14:47:58.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.323+0000 7f21773bd740 10 client.0 _ll_register_callbacks cb 0x7f2158009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:47:58.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.324+0000 7f2172600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d8f553ca20 0x55d8f55b22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.324+0000 7f2171200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d8f5532350 0x55d8f5540f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.324+0000 7f2171c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8f5531a30 0x55d8f5540a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.324+0000 7f2172600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d8f553ca20 0x55d8f55b22c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:35330/0 (socket says 172.21.15.28:35330) 2024-04-23T14:47:58.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.324+0000 7f2171200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d8f5532350 0x55d8f5540f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:37510/0 (socket says 172.21.15.28:37510) 2024-04-23T14:47:58.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.324+0000 7f2172600640 1 -- 172.21.15.28:0/4227188612 learned_addr learned my addr 172.21.15.28:0/4227188612 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:47:58.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.324+0000 7f2172600640 1 -- 172.21.15.28:0/4227188612 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d8f5532350 msgr2=0x55d8f5540f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:58.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.324+0000 7f2172600640 1 --2- 172.21.15.28:0/4227188612 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d8f5532350 0x55d8f5540f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.324+0000 7f2172600640 1 -- 172.21.15.28:0/4227188612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8f5531a30 msgr2=0x55d8f5540a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:47:58.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.324+0000 7f2172600640 1 --2- 172.21.15.28:0/4227188612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8f5531a30 0x55d8f5540a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.324+0000 7f2172600640 1 -- 172.21.15.28:0/4227188612 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8f51a93f0 con 0x55d8f553ca20 2024-04-23T14:47:58.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.324+0000 7f2172600640 1 --2- 172.21.15.28:0/4227188612 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d8f553ca20 0x55d8f55b22c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2164019f50 tx=0x7f2164019f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf6610660de802a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:47:58.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.324+0000 7f2153400640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:47:58.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.324+0000 7f2153400640 1 -- 172.21.15.28:0/4227188612 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216401b430 con 0x55d8f553ca20 2024-04-23T14:47:58.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.325+0000 7f21773bd740 1 -- 172.21.15.28:0/4227188612 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2164019950 con 0x55d8f553ca20 2024-04-23T14:47:58.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.325+0000 7f2153400640 1 -- 172.21.15.28:0/4227188612 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f216401b5d0 con 0x55d8f553ca20 2024-04-23T14:47:58.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.325+0000 7f2153400640 1 -- 172.21.15.28:0/4227188612 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f216401b7d0 con 0x55d8f553ca20 2024-04-23T14:47:58.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.325+0000 7f21773bd740 10 client.25894 fetch_fsmap learned FSMap version 20 2024-04-23T14:47:58.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.325+0000 7f21773bd740 1 -- 172.21.15.28:0/4227188612 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55d8f51690f0 con 0x55d8f553ca20 2024-04-23T14:47:58.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.325+0000 7f2153400640 1 -- 172.21.15.28:0/4227188612 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2164019950 con 0x55d8f553ca20 2024-04-23T14:47:58.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.325+0000 7f2153400640 1 -- 172.21.15.28:0/4227188612 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f216400f850 con 0x55d8f553ca20 2024-04-23T14:47:58.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.325+0000 7f21773bd740 10 client.25894 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:47:58.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.325+0000 7f21773bd740 10 client.25894 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:47:58.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.325+0000 7f21773bd740 20 client.25894 populate_metadata read hostname 'smithi028' 2024-04-23T14:47:58.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.325+0000 7f21773bd740 1 --2- 172.21.15.28:0/4227188612 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d8f554cd00 0x55d8f55b5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:47:58.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.325+0000 7f21773bd740 4 client.25894 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:47:58.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.325+0000 7f21773bd740 1 -- 172.21.15.28:0/4227188612 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d8f549ea80 con 0x55d8f554cd00 2024-04-23T14:47:58.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.326+0000 7f2171c00640 1 --2- 172.21.15.28:0/4227188612 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d8f554cd00 0x55d8f55b5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:47:58.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.326+0000 7f2171c00640 1 --2- 172.21.15.28:0/4227188612 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d8f554cd00 0x55d8f55b5bf0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3a70a9508b6a740 server_cookie=94eb00b7cd4ec17c in_seq=0 out_seq=0 2024-04-23T14:47:58.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.326+0000 7f2153400640 10 client.25894 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:47:58.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.326+0000 7f2153400640 1 -- 172.21.15.28:0/4227188612 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55d8f549ea80 con 0x55d8f554cd00 2024-04-23T14:47:58.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.326+0000 7f2153400640 10 client.25894 handle_command_reply: tid=0 2024-04-23T14:47:58.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.327+0000 7f21773bd740 1 client.25894 shutdown 2024-04-23T14:47:58.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.327+0000 7f21773bd740 1 -- 172.21.15.28:0/4227188612 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d8f553ca20 msgr2=0x55d8f55b22c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:58.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.327+0000 7f21773bd740 1 --2- 172.21.15.28:0/4227188612 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d8f553ca20 0x55d8f55b22c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2164019f50 tx=0x7f2164019f80 comp rx=0 tx=0).stop 2024-04-23T14:47:58.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.327+0000 7f21773bd740 1 -- 172.21.15.28:0/4227188612 shutdown_connections 2024-04-23T14:47:58.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.327+0000 7f21773bd740 1 --2- 172.21.15.28:0/4227188612 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d8f554cd00 0x55d8f55b5bf0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.327+0000 7f21773bd740 1 --2- 172.21.15.28:0/4227188612 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d8f5532350 0x55d8f5540f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.327+0000 7f21773bd740 1 --2- 172.21.15.28:0/4227188612 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d8f553ca20 0x55d8f55b22c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.327+0000 7f2153400640 0 client.25894 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:47:58.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.327+0000 7f21773bd740 1 --2- 172.21.15.28:0/4227188612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8f5531a30 0x55d8f5540a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.327+0000 7f21773bd740 1 -- 172.21.15.28:0/4227188612 >> 172.21.15.28:0/4227188612 conn(0x55d8f552dd50 msgr2=0x55d8f55331d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:58.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.327+0000 7f21773bd740 1 -- 172.21.15.28:0/4227188612 shutdown_connections 2024-04-23T14:47:58.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.327+0000 7f21773bd740 1 -- 172.21.15.28:0/4227188612 wait complete. 2024-04-23T14:47:58.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.327+0000 7f21773bd740 20 client.25894 trim_cache size 0 max 16384 2024-04-23T14:47:58.332 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:47:58.332 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-23T14:47:58.332 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-23T14:47:58.332 INFO:teuthology.orchestra.run.smithi028.stdout: "fcd88ff1-6c17-4111-bcd3-b274dc4c8c1b": { 2024-04-23T14:47:58.332 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-23T14:47:58.332 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "fcd88ff1-6c17-4111-bcd3-b274dc4c8c1b", 2024-04-23T14:47:58.332 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-23T14:47:58.332 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:47:58.332 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:47:58.333 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:47:58.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.328+0000 7f2173000640 1 -- 172.21.15.28:0/3713636062 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f21540a3d20 msgr2=0x7f21540a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:58.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.328+0000 7f2173000640 1 --2- 172.21.15.28:0/3713636062 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f21540a3d20 0x7f21540a61e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2164012fd0 tx=0x7f2164004990 comp rx=0 tx=0).stop 2024-04-23T14:47:58.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.328+0000 7f2173000640 1 -- 172.21.15.28:0/3713636062 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f216c104320 msgr2=0x7f216c1a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:47:58.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.328+0000 7f2173000640 1 --2- 172.21.15.28:0/3713636062 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f216c104320 0x7f216c1a0640 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f216000e9f0 tx=0x7f216000eeb0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.328+0000 7f2173000640 1 -- 172.21.15.28:0/3713636062 shutdown_connections 2024-04-23T14:47:58.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.328+0000 7f2173000640 1 --2- 172.21.15.28:0/3713636062 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f21540a3d20 0x7f21540a61e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.328+0000 7f2173000640 1 --2- 172.21.15.28:0/3713636062 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f216c19a7b0 0x7f216c19ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.328+0000 7f2173000640 1 --2- 172.21.15.28:0/3713636062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f216c1093f0 0x7f216c1a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.328+0000 7f2173000640 1 --2- 172.21.15.28:0/3713636062 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f216c104320 0x7f216c1a0640 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:47:58.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.328+0000 7f2173000640 1 -- 172.21.15.28:0/3713636062 >> 172.21.15.28:0/3713636062 conn(0x7f216c0ffb40 msgr2=0x7f216c100f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:47:58.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.328+0000 7f2173000640 1 -- 172.21.15.28:0/3713636062 shutdown_connections 2024-04-23T14:47:58.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:47:58.328+0000 7f2173000640 1 -- 172.21.15.28:0/3713636062 wait complete. 2024-04-23T14:47:58.347 INFO:tasks.cephfs.filesystem:scrub status for tag:fcd88ff1-6c17-4111-bcd3-b274dc4c8c1b - {'path': '/', 'tag': 'fcd88ff1-6c17-4111-bcd3-b274dc4c8c1b', 'options': 'recursive,force'} 2024-04-23T14:47:58.475 INFO:tasks.workunit.client.0.smithi028.stdout: 1 260219 9.81 MB/sec execute 14 sec latency 50.643 ms 2024-04-23T14:47:59.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:58 smithi105 ceph-mon[32555]: pgmap v121: 97 pgs: 97 active+clean; 146 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 241 op/s 2024-04-23T14:47:59.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:58 smithi038 ceph-mon[32348]: pgmap v121: 97 pgs: 97 active+clean; 146 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 241 op/s 2024-04-23T14:47:59.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:58 smithi028 ceph-mon[27810]: pgmap v121: 97 pgs: 97 active+clean; 146 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 241 op/s 2024-04-23T14:47:59.475 INFO:tasks.workunit.client.0.smithi028.stdout: 1 262852 10.29 MB/sec execute 15 sec latency 19.865 ms 2024-04-23T14:48:00.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:59 smithi105 ceph-mon[32555]: scrub queued for path: / 2024-04-23T14:48:00.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:59 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:48:00.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:59 smithi105 ceph-mon[32555]: scrub summary: active paths [/] 2024-04-23T14:48:00.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:47:59 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:48:00.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:59 smithi038 ceph-mon[32348]: scrub queued for path: / 2024-04-23T14:48:00.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:59 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:48:00.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:59 smithi038 ceph-mon[32348]: scrub summary: active paths [/] 2024-04-23T14:48:00.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:47:59 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:48:00.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:59 smithi028 ceph-mon[27810]: scrub queued for path: / 2024-04-23T14:48:00.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:59 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:48:00.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:59 smithi028 ceph-mon[27810]: scrub summary: active paths [/] 2024-04-23T14:48:00.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:47:59 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:48:00.476 INFO:tasks.workunit.client.0.smithi028.stdout: 1 264958 10.09 MB/sec execute 16 sec latency 13.052 ms 2024-04-23T14:48:01.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:00 smithi105 ceph-mon[32555]: pgmap v122: 97 pgs: 97 active+clean; 146 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 161 op/s 2024-04-23T14:48:01.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:00 smithi105 ceph-mon[32555]: scrub summary: idle 2024-04-23T14:48:01.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:00 smithi038 ceph-mon[32348]: pgmap v122: 97 pgs: 97 active+clean; 146 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 161 op/s 2024-04-23T14:48:01.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:00 smithi038 ceph-mon[32348]: scrub summary: idle 2024-04-23T14:48:01.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:00 smithi028 ceph-mon[27810]: pgmap v122: 97 pgs: 97 active+clean; 146 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 161 op/s 2024-04-23T14:48:01.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:00 smithi028 ceph-mon[27810]: scrub summary: idle 2024-04-23T14:48:01.476 INFO:tasks.workunit.client.0.smithi028.stdout: 1 266196 10.12 MB/sec execute 17 sec latency 59.205 ms 2024-04-23T14:48:02.476 INFO:tasks.workunit.client.0.smithi028.stdout: 1 267495 9.92 MB/sec execute 18 sec latency 16.348 ms 2024-04-23T14:48:03.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:02 smithi105 ceph-mon[32555]: pgmap v123: 97 pgs: 97 active+clean; 146 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 161 op/s 2024-04-23T14:48:03.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:02 smithi038 ceph-mon[32348]: pgmap v123: 97 pgs: 97 active+clean; 146 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 161 op/s 2024-04-23T14:48:03.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:02 smithi028 ceph-mon[27810]: pgmap v123: 97 pgs: 97 active+clean; 146 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 161 op/s 2024-04-23T14:48:03.479 INFO:tasks.workunit.client.0.smithi028.stdout: 1 269422 9.89 MB/sec execute 19 sec latency 83.959 ms 2024-04-23T14:48:04.478 INFO:tasks.workunit.client.0.smithi028.stdout: 1 270581 9.67 MB/sec execute 20 sec latency 22.128 ms 2024-04-23T14:48:05.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:04 smithi105 ceph-mon[32555]: pgmap v124: 97 pgs: 97 active+clean; 151 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.9 MiB/s wr, 277 op/s 2024-04-23T14:48:05.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:04 smithi038 ceph-mon[32348]: pgmap v124: 97 pgs: 97 active+clean; 151 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.9 MiB/s wr, 277 op/s 2024-04-23T14:48:05.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:04 smithi028 ceph-mon[27810]: pgmap v124: 97 pgs: 97 active+clean; 151 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.9 MiB/s wr, 277 op/s 2024-04-23T14:48:05.478 INFO:tasks.workunit.client.0.smithi028.stdout: 1 272600 9.77 MB/sec execute 21 sec latency 29.311 ms 2024-04-23T14:48:06.478 INFO:tasks.workunit.client.0.smithi028.stdout: 1 273981 9.64 MB/sec execute 22 sec latency 16.066 ms 2024-04-23T14:48:07.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:06 smithi105 ceph-mon[32555]: pgmap v125: 97 pgs: 97 active+clean; 151 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 192 op/s 2024-04-23T14:48:07.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:06 smithi038 ceph-mon[32348]: pgmap v125: 97 pgs: 97 active+clean; 151 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 192 op/s 2024-04-23T14:48:07.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:06 smithi028 ceph-mon[27810]: pgmap v125: 97 pgs: 97 active+clean; 151 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 192 op/s 2024-04-23T14:48:07.480 INFO:tasks.workunit.client.0.smithi028.stdout: 1 276148 9.75 MB/sec execute 23 sec latency 11.091 ms 2024-04-23T14:48:08.480 INFO:tasks.workunit.client.0.smithi028.stdout: 1 277431 9.63 MB/sec execute 24 sec latency 60.098 ms 2024-04-23T14:48:09.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:08 smithi105 ceph-mon[32555]: pgmap v126: 97 pgs: 97 active+clean; 155 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.9 MiB/s wr, 269 op/s 2024-04-23T14:48:09.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:08 smithi038 ceph-mon[32348]: pgmap v126: 97 pgs: 97 active+clean; 155 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.9 MiB/s wr, 269 op/s 2024-04-23T14:48:09.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:08 smithi028 ceph-mon[27810]: pgmap v126: 97 pgs: 97 active+clean; 155 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.9 MiB/s wr, 269 op/s 2024-04-23T14:48:09.480 INFO:tasks.workunit.client.0.smithi028.stdout: 1 279680 9.73 MB/sec execute 25 sec latency 13.653 ms 2024-04-23T14:48:10.480 INFO:tasks.workunit.client.0.smithi028.stdout: 1 281302 9.68 MB/sec execute 26 sec latency 16.016 ms 2024-04-23T14:48:11.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:10 smithi105 ceph-mon[32555]: pgmap v127: 97 pgs: 97 active+clean; 155 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 2.8 MiB/s wr, 192 op/s 2024-04-23T14:48:11.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:10 smithi038 ceph-mon[32348]: pgmap v127: 97 pgs: 97 active+clean; 155 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 2.8 MiB/s wr, 192 op/s 2024-04-23T14:48:11.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:10 smithi028 ceph-mon[27810]: pgmap v127: 97 pgs: 97 active+clean; 155 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 2.8 MiB/s wr, 192 op/s 2024-04-23T14:48:11.480 INFO:tasks.workunit.client.0.smithi028.stdout: 1 283984 9.95 MB/sec execute 27 sec latency 9.242 ms 2024-04-23T14:48:12.480 INFO:tasks.workunit.client.0.smithi028.stdout: 1 285877 9.81 MB/sec execute 28 sec latency 12.299 ms 2024-04-23T14:48:13.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:12 smithi105 ceph-mon[32555]: pgmap v128: 97 pgs: 97 active+clean; 155 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 2.8 MiB/s wr, 192 op/s 2024-04-23T14:48:13.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:12 smithi038 ceph-mon[32348]: pgmap v128: 97 pgs: 97 active+clean; 155 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 2.8 MiB/s wr, 192 op/s 2024-04-23T14:48:13.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:12 smithi028 ceph-mon[27810]: pgmap v128: 97 pgs: 97 active+clean; 155 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 2.8 MiB/s wr, 192 op/s 2024-04-23T14:48:13.480 INFO:tasks.workunit.client.0.smithi028.stdout: 1 287693 9.93 MB/sec execute 29 sec latency 126.669 ms 2024-04-23T14:48:14.481 INFO:tasks.workunit.client.0.smithi028.stdout: 1 289193 9.78 MB/sec execute 30 sec latency 25.208 ms 2024-04-23T14:48:15.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:14 smithi105 ceph-mon[32555]: pgmap v129: 97 pgs: 97 active+clean; 161 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.1 MiB/s wr, 278 op/s 2024-04-23T14:48:15.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:14 smithi038 ceph-mon[32348]: pgmap v129: 97 pgs: 97 active+clean; 161 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.1 MiB/s wr, 278 op/s 2024-04-23T14:48:15.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:14 smithi028 ceph-mon[27810]: pgmap v129: 97 pgs: 97 active+clean; 161 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.1 MiB/s wr, 278 op/s 2024-04-23T14:48:15.481 INFO:tasks.workunit.client.0.smithi028.stdout: 1 291167 9.90 MB/sec execute 31 sec latency 15.726 ms 2024-04-23T14:48:16.481 INFO:tasks.workunit.client.0.smithi028.stdout: 1 292781 9.77 MB/sec execute 32 sec latency 23.539 ms 2024-04-23T14:48:17.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:16 smithi105 ceph-mon[32555]: pgmap v130: 97 pgs: 97 active+clean; 161 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 2.6 MiB/s wr, 162 op/s 2024-04-23T14:48:17.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:16 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:48:17.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:16 smithi038 ceph-mon[32348]: pgmap v130: 97 pgs: 97 active+clean; 161 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 2.6 MiB/s wr, 162 op/s 2024-04-23T14:48:17.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:16 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:48:17.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:16 smithi028 ceph-mon[27810]: pgmap v130: 97 pgs: 97 active+clean; 161 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 2.6 MiB/s wr, 162 op/s 2024-04-23T14:48:17.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:16 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:48:17.481 INFO:tasks.workunit.client.0.smithi028.stdout: 1 294822 9.89 MB/sec execute 33 sec latency 14.518 ms 2024-04-23T14:48:18.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:17 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:48:18.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:17 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:48:18.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:17 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:48:18.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:17 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:48:18.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:17 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:48:18.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:17 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:48:18.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:17 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:48:18.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:17 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:48:18.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:17 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:48:18.481 INFO:tasks.workunit.client.0.smithi028.stdout: 1 296251 9.76 MB/sec execute 34 sec latency 41.204 ms 2024-04-23T14:48:19.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:18 smithi105 ceph-mon[32555]: pgmap v131: 97 pgs: 97 active+clean; 166 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.9 MiB/s wr, 242 op/s 2024-04-23T14:48:19.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:18 smithi038 ceph-mon[32348]: pgmap v131: 97 pgs: 97 active+clean; 166 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.9 MiB/s wr, 242 op/s 2024-04-23T14:48:19.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:18 smithi028 ceph-mon[27810]: pgmap v131: 97 pgs: 97 active+clean; 166 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.9 MiB/s wr, 242 op/s 2024-04-23T14:48:19.481 INFO:tasks.workunit.client.0.smithi028.stdout: 1 298260 9.86 MB/sec execute 35 sec latency 25.444 ms 2024-04-23T14:48:20.481 INFO:tasks.workunit.client.0.smithi028.stdout: 1 299749 9.74 MB/sec execute 36 sec latency 42.829 ms 2024-04-23T14:48:21.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:20 smithi038 ceph-mon[32348]: pgmap v132: 97 pgs: 97 active+clean; 166 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 2.5 MiB/s wr, 166 op/s 2024-04-23T14:48:21.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:20 smithi028 ceph-mon[27810]: pgmap v132: 97 pgs: 97 active+clean; 166 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 2.5 MiB/s wr, 166 op/s 2024-04-23T14:48:21.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:20 smithi105 ceph-mon[32555]: pgmap v132: 97 pgs: 97 active+clean; 166 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 2.5 MiB/s wr, 166 op/s 2024-04-23T14:48:21.481 INFO:tasks.workunit.client.0.smithi028.stdout: 1 301848 9.85 MB/sec execute 37 sec latency 11.495 ms 2024-04-23T14:48:22.482 INFO:tasks.workunit.client.0.smithi028.stdout: 1 303557 9.74 MB/sec execute 38 sec latency 15.418 ms 2024-04-23T14:48:23.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:22 smithi038 ceph-mon[32348]: pgmap v133: 97 pgs: 97 active+clean; 166 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 2.5 MiB/s wr, 166 op/s 2024-04-23T14:48:23.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:22 smithi028 ceph-mon[27810]: pgmap v133: 97 pgs: 97 active+clean; 166 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 2.5 MiB/s wr, 166 op/s 2024-04-23T14:48:23.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:22 smithi105 ceph-mon[32555]: pgmap v133: 97 pgs: 97 active+clean; 166 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 2.5 MiB/s wr, 166 op/s 2024-04-23T14:48:23.481 INFO:tasks.workunit.client.0.smithi028.stdout: 1 305405 9.84 MB/sec execute 39 sec latency 18.752 ms 2024-04-23T14:48:24.482 INFO:tasks.workunit.client.0.smithi028.stdout: 1 306553 9.72 MB/sec execute 40 sec latency 70.105 ms 2024-04-23T14:48:25.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:24 smithi038 ceph-mon[32348]: pgmap v134: 97 pgs: 97 active+clean; 171 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.9 MiB/s wr, 247 op/s 2024-04-23T14:48:25.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:24 smithi028 ceph-mon[27810]: pgmap v134: 97 pgs: 97 active+clean; 171 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.9 MiB/s wr, 247 op/s 2024-04-23T14:48:25.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:24 smithi105 ceph-mon[32555]: pgmap v134: 97 pgs: 97 active+clean; 171 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.9 MiB/s wr, 247 op/s 2024-04-23T14:48:25.482 INFO:tasks.workunit.client.0.smithi028.stdout: 1 308826 9.82 MB/sec execute 41 sec latency 27.416 ms 2024-04-23T14:48:26.482 INFO:tasks.workunit.client.0.smithi028.stdout: 1 309865 9.64 MB/sec execute 42 sec latency 18.457 ms 2024-04-23T14:48:27.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:26 smithi038 ceph-mon[32348]: pgmap v135: 97 pgs: 97 active+clean; 171 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 2.7 MiB/s wr, 161 op/s 2024-04-23T14:48:27.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:26 smithi028 ceph-mon[27810]: pgmap v135: 97 pgs: 97 active+clean; 171 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 2.7 MiB/s wr, 161 op/s 2024-04-23T14:48:27.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:26 smithi105 ceph-mon[32555]: pgmap v135: 97 pgs: 97 active+clean; 171 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 2.7 MiB/s wr, 161 op/s 2024-04-23T14:48:27.482 INFO:tasks.workunit.client.0.smithi028.stdout: 1 312154 9.77 MB/sec execute 43 sec latency 12.534 ms 2024-04-23T14:48:28.347 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:48:28.348 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:48:28.482 INFO:tasks.workunit.client.0.smithi028.stdout: 1 313214 9.63 MB/sec execute 44 sec latency 17.548 ms 2024-04-23T14:48:28.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.584+0000 7fa280800640 1 -- 172.21.15.28:0/1826560738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa278109270 msgr2=0x7fa278109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:28.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.584+0000 7fa280800640 1 --2- 172.21.15.28:0/1826560738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa278109270 0x7fa278109650 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa26c009900 tx=0x7fa26c02a3f0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.584+0000 7fa280800640 1 -- 172.21.15.28:0/1826560738 shutdown_connections 2024-04-23T14:48:28.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.584+0000 7fa280800640 1 --2- 172.21.15.28:0/1826560738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa27810dd30 0x7fa278115900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.584+0000 7fa280800640 1 --2- 172.21.15.28:0/1826560738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa278109c20 0x7fa27810d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.584+0000 7fa280800640 1 --2- 172.21.15.28:0/1826560738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa278109270 0x7fa278109650 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.584+0000 7fa280800640 1 -- 172.21.15.28:0/1826560738 >> 172.21.15.28:0/1826560738 conn(0x7fa27806d110 msgr2=0x7fa27806d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:48:28.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.586+0000 7fa280800640 1 -- 172.21.15.28:0/1826560738 shutdown_connections 2024-04-23T14:48:28.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.586+0000 7fa280800640 1 -- 172.21.15.28:0/1826560738 wait complete. 2024-04-23T14:48:28.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.588+0000 7fa280800640 1 Processor -- start 2024-04-23T14:48:28.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.591+0000 7fa280800640 1 -- start start 2024-04-23T14:48:28.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.591+0000 7fa280800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa278109270 0x7fa2781a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:28.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.591+0000 7fa280800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa27810dd30 0x7fa2781a1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:28.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.591+0000 7fa280800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2781a59e0 0x7fa2781a5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:28.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.591+0000 7fa280800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa27811ac20 con 0x7fa278109270 2024-04-23T14:48:28.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.591+0000 7fa280800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa27811aaa0 con 0x7fa27810dd30 2024-04-23T14:48:28.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.591+0000 7fa280800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa27811ada0 con 0x7fa2781a59e0 2024-04-23T14:48:28.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.592+0000 7fa27e800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2781a59e0 0x7fa2781a5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:28.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.592+0000 7fa27e800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2781a59e0 0x7fa2781a5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:34434/0 (socket says 172.21.15.28:34434) 2024-04-23T14:48:28.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.592+0000 7fa27e800640 1 -- 172.21.15.28:0/1332346881 learned_addr learned my addr 172.21.15.28:0/1332346881 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:48:28.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.592+0000 7fa27d400640 1 --2- 172.21.15.28:0/1332346881 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa27810dd30 0x7fa2781a1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:28.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.592+0000 7fa27de00640 1 --2- 172.21.15.28:0/1332346881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa278109270 0x7fa2781a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:28.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.593+0000 7fa27e800640 1 -- 172.21.15.28:0/1332346881 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa27810dd30 msgr2=0x7fa2781a1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:28.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.593+0000 7fa27e800640 1 --2- 172.21.15.28:0/1332346881 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa27810dd30 0x7fa2781a1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.593+0000 7fa27e800640 1 -- 172.21.15.28:0/1332346881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa278109270 msgr2=0x7fa2781a1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:28.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.593+0000 7fa27e800640 1 --2- 172.21.15.28:0/1332346881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa278109270 0x7fa2781a1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.593+0000 7fa27e800640 1 -- 172.21.15.28:0/1332346881 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2781a64d0 con 0x7fa2781a59e0 2024-04-23T14:48:28.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.594+0000 7fa27de00640 1 --2- 172.21.15.28:0/1332346881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa278109270 0x7fa2781a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:48:28.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.596+0000 7fa27e800640 1 --2- 172.21.15.28:0/1332346881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2781a59e0 0x7fa2781a5e90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa27400eaa0 tx=0x7fa27400ef60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:48:28.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.597+0000 7fa272a00640 1 -- 172.21.15.28:0/1332346881 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa27400cdb0 con 0x7fa2781a59e0 2024-04-23T14:48:28.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.597+0000 7fa272a00640 1 -- 172.21.15.28:0/1332346881 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa274005020 con 0x7fa2781a59e0 2024-04-23T14:48:28.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.597+0000 7fa280800640 1 -- 172.21.15.28:0/1332346881 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2781a67c0 con 0x7fa2781a59e0 2024-04-23T14:48:28.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.597+0000 7fa272a00640 1 -- 172.21.15.28:0/1332346881 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2740116b0 con 0x7fa2781a59e0 2024-04-23T14:48:28.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.598+0000 7fa280800640 1 -- 172.21.15.28:0/1332346881 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa278117aa0 con 0x7fa2781a59e0 2024-04-23T14:48:28.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.605+0000 7fa272a00640 1 -- 172.21.15.28:0/1332346881 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fa274005560 con 0x7fa2781a59e0 2024-04-23T14:48:28.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.605+0000 7fa28358f740 1 Processor -- start 2024-04-23T14:48:28.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.605+0000 7fa28358f740 1 -- start start 2024-04-23T14:48:28.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.607+0000 7fa28358f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557efa51f860 0x557efa51fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:28.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.607+0000 7fa28358f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557efa520180 0x557efa52a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:28.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.607+0000 7fa28358f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557efa52a850 0x557efa52cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:28.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.607+0000 7fa28358f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557efa4851f0 con 0x557efa51f860 2024-04-23T14:48:28.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.607+0000 7fa28358f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x557efa42b350 con 0x557efa520180 2024-04-23T14:48:28.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.607+0000 7fa28358f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x557efa4811b0 con 0x557efa52a850 2024-04-23T14:48:28.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.607+0000 7fa27d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557efa51f860 0x557efa51fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:28.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.607+0000 7fa27de00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557efa520180 0x557efa52a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:28.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.607+0000 7fa27d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557efa51f860 0x557efa51fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59796/0 (socket says 172.21.15.28:59796) 2024-04-23T14:48:28.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.607+0000 7fa27d400640 1 -- 172.21.15.28:0/3309900435 learned_addr learned my addr 172.21.15.28:0/3309900435 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:48:28.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.608+0000 7fa27d400640 1 --2- 172.21.15.28:0/3309900435 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557efa52a850 0x557efa52cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.608+0000 7fa272a00640 1 --2- 172.21.15.28:0/1332346881 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa2600a3e70 0x7fa2600a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.608+0000 7fa27d400640 1 -- 172.21.15.28:0/3309900435 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557efa52a850 msgr2=0x557efa52cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:48:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.608+0000 7fa27d400640 1 --2- 172.21.15.28:0/3309900435 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557efa52a850 0x557efa52cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.608+0000 7fa27d400640 1 -- 172.21.15.28:0/3309900435 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557efa520180 msgr2=0x557efa52a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:48:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.608+0000 7fa27d400640 1 --2- 172.21.15.28:0/3309900435 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557efa520180 0x557efa52a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.608+0000 7fa272a00640 1 -- 172.21.15.28:0/1332346881 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fa2740d0950 con 0x7fa2781a59e0 2024-04-23T14:48:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.608+0000 7fa27d400640 1 -- 172.21.15.28:0/3309900435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557efa47ff50 con 0x557efa51f860 2024-04-23T14:48:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.609+0000 7fa27e800640 1 --2- 172.21.15.28:0/1332346881 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa2600a3e70 0x7fa2600a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.609+0000 7fa27d400640 1 --2- 172.21.15.28:0/3309900435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557efa51f860 0x557efa51fc40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa2680137e0 tx=0x7fa26800a390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3d32e5ecbc96e3b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:48:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.610+0000 7fa267000640 1 -- 172.21.15.28:0/3309900435 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa268004220 con 0x557efa51f860 2024-04-23T14:48:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.610+0000 7fa27e800640 1 --2- 172.21.15.28:0/1332346881 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa2600a3e70 0x7fa2600a6330 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa2740a4fe0 tx=0x7fa2740058d0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:48:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.610+0000 7fa267000640 1 -- 172.21.15.28:0/3309900435 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2680043c0 con 0x557efa51f860 2024-04-23T14:48:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.610+0000 7fa267000640 1 -- 172.21.15.28:0/3309900435 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa26800e6f0 con 0x557efa51f860 2024-04-23T14:48:28.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.611+0000 7fa28358f740 1 -- 172.21.15.28:0/3309900435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557efa51f860 msgr2=0x557efa51fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:28.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.611+0000 7fa28358f740 1 --2- 172.21.15.28:0/3309900435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557efa51f860 0x557efa51fc40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa2680137e0 tx=0x7fa26800a390 comp rx=0 tx=0).stop 2024-04-23T14:48:28.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.611+0000 7fa28358f740 1 -- 172.21.15.28:0/3309900435 shutdown_connections 2024-04-23T14:48:28.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.611+0000 7fa28358f740 1 --2- 172.21.15.28:0/3309900435 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557efa520180 0x557efa52a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.611+0000 7fa28358f740 1 --2- 172.21.15.28:0/3309900435 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557efa52a850 0x557efa52cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.611+0000 7fa28358f740 1 --2- 172.21.15.28:0/3309900435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557efa51f860 0x557efa51fc40 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.611+0000 7fa28358f740 1 -- 172.21.15.28:0/3309900435 >> 172.21.15.28:0/3309900435 conn(0x557efa51bb80 msgr2=0x557efa51bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:48:28.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.611+0000 7fa28358f740 1 -- 172.21.15.28:0/3309900435 shutdown_connections 2024-04-23T14:48:28.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.611+0000 7fa28358f740 1 -- 172.21.15.28:0/3309900435 wait complete. 2024-04-23T14:48:28.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.612+0000 7fa28358f740 1 -- start start 2024-04-23T14:48:28.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.612+0000 7fa28358f740 1 Processor -- start 2024-04-23T14:48:28.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.612+0000 7fa28358f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557efa51f860 0x557efa527870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:28.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.612+0000 7fa28358f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557efa520180 0x557efa527db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:28.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.612+0000 7fa28358f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557efa52a850 0x557efa5282f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:28.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.612+0000 7fa28358f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557efa52d640 con 0x557efa520180 2024-04-23T14:48:28.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.612+0000 7fa28358f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x557efa43e690 con 0x557efa52a850 2024-04-23T14:48:28.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.613+0000 7fa28358f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x557efa52d7c0 con 0x557efa51f860 2024-04-23T14:48:28.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.613+0000 7fa28358f740 10 client.0 _ll_register_callbacks cb 0x557efa47a650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:48:28.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.613+0000 7fa27d400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557efa51f860 0x557efa527870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:28.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.613+0000 7fa27d400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557efa51f860 0x557efa527870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:34450/0 (socket says 172.21.15.28:34450) 2024-04-23T14:48:28.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.613+0000 7fa27d400640 1 -- 172.21.15.28:0/1190547260 learned_addr learned my addr 172.21.15.28:0/1190547260 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:48:28.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.613+0000 7fa27d400640 1 --2- 172.21.15.28:0/1190547260 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557efa52a850 0x557efa5282f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:28.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.613+0000 7fa27de00640 1 --2- 172.21.15.28:0/1190547260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557efa520180 0x557efa527db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:28.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.613+0000 7fa27d400640 1 -- 172.21.15.28:0/1190547260 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557efa52a850 msgr2=0x557efa5282f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:48:28.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.613+0000 7fa27d400640 1 --2- 172.21.15.28:0/1190547260 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557efa52a850 0x557efa5282f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.613+0000 7fa27d400640 1 -- 172.21.15.28:0/1190547260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557efa520180 msgr2=0x557efa527db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:48:28.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.613+0000 7fa27d400640 1 --2- 172.21.15.28:0/1190547260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557efa520180 0x557efa527db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.613+0000 7fa27d400640 1 -- 172.21.15.28:0/1190547260 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557efa43d920 con 0x557efa51f860 2024-04-23T14:48:28.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.614+0000 7fa27d400640 1 --2- 172.21.15.28:0/1190547260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557efa51f860 0x557efa527870 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x557efa5175d0 tx=0x7fa268004b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23194f3c18dd42a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:48:28.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.614+0000 7fa25b400640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:48:28.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.614+0000 7fa25b400640 1 -- 172.21.15.28:0/1190547260 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2680046c0 con 0x557efa51f860 2024-04-23T14:48:28.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.614+0000 7fa25b400640 1 -- 172.21.15.28:0/1190547260 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fa268004860 con 0x557efa51f860 2024-04-23T14:48:28.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.614+0000 7fa28358f740 1 -- 172.21.15.28:0/1190547260 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2680043c0 con 0x557efa51f860 2024-04-23T14:48:28.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.615+0000 7fa25b400640 1 -- 172.21.15.28:0/1190547260 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fa268034050 con 0x557efa51f860 2024-04-23T14:48:28.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.615+0000 7fa28358f740 10 client.25906 fetch_fsmap learned FSMap version 20 2024-04-23T14:48:28.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.615+0000 7fa28358f740 1 -- 172.21.15.28:0/1190547260 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x557efa2617d0 con 0x557efa51f860 2024-04-23T14:48:28.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.615+0000 7fa25b400640 1 -- 172.21.15.28:0/1190547260 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2680043c0 con 0x557efa51f860 2024-04-23T14:48:28.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.615+0000 7fa25b400640 1 -- 172.21.15.28:0/1190547260 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fa26800f040 con 0x557efa51f860 2024-04-23T14:48:28.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.615+0000 7fa28358f740 10 client.25906 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:48:28.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.615+0000 7fa28358f740 10 client.25906 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:48:28.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.615+0000 7fa28358f740 20 client.25906 populate_metadata read hostname 'smithi028' 2024-04-23T14:48:28.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.615+0000 7fa28358f740 1 --2- 172.21.15.28:0/1190547260 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x557efa53aaf0 0x557efa53cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:28.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.615+0000 7fa28358f740 4 client.25906 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:48:28.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.615+0000 7fa28358f740 1 -- 172.21.15.28:0/1190547260 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557efa492c10 con 0x557efa53aaf0 2024-04-23T14:48:28.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.616+0000 7fa27de00640 1 --2- 172.21.15.28:0/1190547260 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x557efa53aaf0 0x557efa53cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:28.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.616+0000 7fa27de00640 1 --2- 172.21.15.28:0/1190547260 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x557efa53aaf0 0x557efa53cee0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2509f40b4cc2d578 server_cookie=66400a8d53eae4e2 in_seq=0 out_seq=0 2024-04-23T14:48:28.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.616+0000 7fa25b400640 10 client.25906 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:48:28.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.617+0000 7fa25b400640 1 -- 172.21.15.28:0/1190547260 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557efa492c10 con 0x557efa53aaf0 2024-04-23T14:48:28.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.617+0000 7fa25b400640 10 client.25906 handle_command_reply: tid=0 2024-04-23T14:48:28.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.617+0000 7fa28358f740 1 client.25906 shutdown 2024-04-23T14:48:28.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.618+0000 7fa28358f740 1 -- 172.21.15.28:0/1190547260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557efa51f860 msgr2=0x557efa527870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:28.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.618+0000 7fa28358f740 1 --2- 172.21.15.28:0/1190547260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557efa51f860 0x557efa527870 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x557efa5175d0 tx=0x7fa268004b30 comp rx=0 tx=0).stop 2024-04-23T14:48:28.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.618+0000 7fa28358f740 1 -- 172.21.15.28:0/1190547260 shutdown_connections 2024-04-23T14:48:28.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.618+0000 7fa28358f740 1 --2- 172.21.15.28:0/1190547260 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x557efa53aaf0 0x557efa53cee0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.618+0000 7fa28358f740 1 --2- 172.21.15.28:0/1190547260 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557efa52a850 0x557efa5282f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.618+0000 7fa28358f740 1 --2- 172.21.15.28:0/1190547260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557efa520180 0x557efa527db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.618+0000 7fa28358f740 1 --2- 172.21.15.28:0/1190547260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557efa51f860 0x557efa527870 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.618+0000 7fa25b400640 0 client.25906 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:48:28.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.618+0000 7fa28358f740 1 -- 172.21.15.28:0/1190547260 >> 172.21.15.28:0/1190547260 conn(0x557efa51bb80 msgr2=0x557efa521980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:48:28.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.618+0000 7fa28358f740 1 -- 172.21.15.28:0/1190547260 shutdown_connections 2024-04-23T14:48:28.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.618+0000 7fa28358f740 1 -- 172.21.15.28:0/1190547260 wait complete. 2024-04-23T14:48:28.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.618+0000 7fa28358f740 20 client.25906 trim_cache size 0 max 16384 2024-04-23T14:48:28.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.640+0000 7fa28358f740 1 Processor -- start 2024-04-23T14:48:28.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.640+0000 7fa28358f740 1 -- start start 2024-04-23T14:48:28.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.641+0000 7fa28358f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557efa51f860 0x557efa5267d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:28.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.641+0000 7fa28358f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557efa520180 0x557efa53ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:28.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.641+0000 7fa28358f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557efa52a850 0x557efa540440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:28.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.641+0000 7fa28358f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa2781b67c0 con 0x557efa51f860 2024-04-23T14:48:28.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.641+0000 7fa28358f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa2781b9040 con 0x557efa52a850 2024-04-23T14:48:28.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.641+0000 7fa28358f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa2781b6e30 con 0x557efa520180 2024-04-23T14:48:28.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.641+0000 7fa27d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557efa51f860 0x557efa5267d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:28.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.641+0000 7fa27d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557efa51f860 0x557efa5267d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59816/0 (socket says 172.21.15.28:59816) 2024-04-23T14:48:28.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.641+0000 7fa27de00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557efa520180 0x557efa53ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:28.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.641+0000 7fa27d400640 1 -- 172.21.15.28:0/960101420 learned_addr learned my addr 172.21.15.28:0/960101420 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:48:28.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.641+0000 7fa27d400640 1 --2- 172.21.15.28:0/960101420 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557efa52a850 0x557efa540440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:28.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.641+0000 7fa27de00640 1 -- 172.21.15.28:0/960101420 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557efa52a850 msgr2=0x557efa540440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:48:28.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.641+0000 7fa27de00640 1 --2- 172.21.15.28:0/960101420 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557efa52a850 0x557efa540440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.641+0000 7fa27de00640 1 -- 172.21.15.28:0/960101420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557efa51f860 msgr2=0x557efa5267d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:48:28.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.641+0000 7fa27de00640 1 --2- 172.21.15.28:0/960101420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557efa51f860 0x557efa5267d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.641+0000 7fa27de00640 1 -- 172.21.15.28:0/960101420 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557efa1973f0 con 0x557efa520180 2024-04-23T14:48:28.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.641+0000 7fa27d400640 1 --2- 172.21.15.28:0/960101420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557efa51f860 0x557efa5267d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:48:28.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.642+0000 7fa27de00640 1 --2- 172.21.15.28:0/960101420 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557efa520180 0x557efa53ff00 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa26c02a910 tx=0x7fa26c005500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9fd0a813f89207f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:48:28.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.642+0000 7fa266600640 1 -- 172.21.15.28:0/960101420 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa26c035040 con 0x557efa520180 2024-04-23T14:48:28.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.642+0000 7fa266600640 1 -- 172.21.15.28:0/960101420 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa26c047630 con 0x557efa520180 2024-04-23T14:48:28.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.642+0000 7fa266600640 1 -- 172.21.15.28:0/960101420 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa26c0315b0 con 0x557efa520180 2024-04-23T14:48:28.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.643+0000 7fa28358f740 1 -- 172.21.15.28:0/960101420 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557efa520180 msgr2=0x557efa53ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:28.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.643+0000 7fa28358f740 1 --2- 172.21.15.28:0/960101420 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557efa520180 0x557efa53ff00 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa26c02a910 tx=0x7fa26c005500 comp rx=0 tx=0).stop 2024-04-23T14:48:28.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.643+0000 7fa28358f740 1 -- 172.21.15.28:0/960101420 shutdown_connections 2024-04-23T14:48:28.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.643+0000 7fa28358f740 1 --2- 172.21.15.28:0/960101420 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557efa52a850 0x557efa540440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.643+0000 7fa28358f740 1 --2- 172.21.15.28:0/960101420 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557efa520180 0x557efa53ff00 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.643+0000 7fa28358f740 1 --2- 172.21.15.28:0/960101420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557efa51f860 0x557efa5267d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.643+0000 7fa28358f740 1 -- 172.21.15.28:0/960101420 >> 172.21.15.28:0/960101420 conn(0x557efa51bb80 msgr2=0x557efa5362a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:48:28.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.643+0000 7fa28358f740 1 -- 172.21.15.28:0/960101420 shutdown_connections 2024-04-23T14:48:28.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.643+0000 7fa28358f740 1 -- 172.21.15.28:0/960101420 wait complete. 2024-04-23T14:48:28.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.643+0000 7fa28358f740 1 -- start start 2024-04-23T14:48:28.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.644+0000 7fa28358f740 1 Processor -- start 2024-04-23T14:48:28.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.644+0000 7fa28358f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557efa51f860 0x557efa5401b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:28.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.644+0000 7fa28358f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557efa520180 0x557efa53d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:28.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.644+0000 7fa28358f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557efa52a850 0x557efa53da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:28.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.644+0000 7fa28358f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa2781b7500 con 0x557efa520180 2024-04-23T14:48:28.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.644+0000 7fa28358f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa2781b8970 con 0x557efa51f860 2024-04-23T14:48:28.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.644+0000 7fa28358f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa2781b7bd0 con 0x557efa52a850 2024-04-23T14:48:28.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.644+0000 7fa28358f740 10 client.0 _ll_register_callbacks cb 0x7fa26c009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:48:28.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.644+0000 7fa27d400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557efa51f860 0x557efa5401b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:28.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.644+0000 7fa27de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557efa520180 0x557efa53d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:28.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.644+0000 7fa27d400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557efa51f860 0x557efa5401b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:55362/0 (socket says 172.21.15.28:55362) 2024-04-23T14:48:28.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.644+0000 7fa27d400640 1 -- 172.21.15.28:0/2192542812 learned_addr learned my addr 172.21.15.28:0/2192542812 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:48:28.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.645+0000 7fa27d400640 1 --2- 172.21.15.28:0/2192542812 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557efa52a850 0x557efa53da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:28.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.645+0000 7fa27d400640 1 -- 172.21.15.28:0/2192542812 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557efa52a850 msgr2=0x557efa53da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:48:28.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.645+0000 7fa27d400640 1 --2- 172.21.15.28:0/2192542812 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557efa52a850 0x557efa53da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.645+0000 7fa27d400640 1 -- 172.21.15.28:0/2192542812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557efa520180 msgr2=0x557efa53d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:48:28.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.645+0000 7fa27d400640 1 --2- 172.21.15.28:0/2192542812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557efa520180 0x557efa53d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.645+0000 7fa27d400640 1 -- 172.21.15.28:0/2192542812 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557efa1570f0 con 0x557efa51f860 2024-04-23T14:48:28.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.645+0000 7fa27de00640 1 --2- 172.21.15.28:0/2192542812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557efa520180 0x557efa53d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:48:28.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.645+0000 7fa27d400640 1 --2- 172.21.15.28:0/2192542812 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557efa51f860 0x557efa5401b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa268005d90 tx=0x7fa26800a390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c6c4a3261460b60 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:48:28.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.645+0000 7fa25b400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:48:28.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.645+0000 7fa25b400640 1 -- 172.21.15.28:0/2192542812 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa268004020 con 0x557efa51f860 2024-04-23T14:48:28.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.646+0000 7fa25b400640 1 -- 172.21.15.28:0/2192542812 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fa2680041c0 con 0x557efa51f860 2024-04-23T14:48:28.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.646+0000 7fa28358f740 1 -- 172.21.15.28:0/2192542812 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa26c047630 con 0x557efa51f860 2024-04-23T14:48:28.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.646+0000 7fa25b400640 1 -- 172.21.15.28:0/2192542812 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fa2680043c0 con 0x557efa51f860 2024-04-23T14:48:28.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.646+0000 7fa28358f740 10 client.25925 fetch_fsmap learned FSMap version 20 2024-04-23T14:48:28.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.646+0000 7fa28358f740 1 -- 172.21.15.28:0/2192542812 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x557efa44a810 con 0x557efa51f860 2024-04-23T14:48:28.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.646+0000 7fa25b400640 1 -- 172.21.15.28:0/2192542812 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa26c047630 con 0x557efa51f860 2024-04-23T14:48:28.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.646+0000 7fa25b400640 1 -- 172.21.15.28:0/2192542812 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fa26801e570 con 0x557efa51f860 2024-04-23T14:48:28.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.646+0000 7fa28358f740 10 client.25925 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:48:28.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.646+0000 7fa28358f740 10 client.25925 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:48:28.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.646+0000 7fa28358f740 20 client.25925 populate_metadata read hostname 'smithi028' 2024-04-23T14:48:28.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.646+0000 7fa28358f740 1 --2- 172.21.15.28:0/2192542812 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x557efa5a0ef0 0x557efa5a32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:28.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.646+0000 7fa28358f740 4 client.25925 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:48:28.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.646+0000 7fa28358f740 1 -- 172.21.15.28:0/2192542812 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557efa48ca80 con 0x557efa5a0ef0 2024-04-23T14:48:28.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.647+0000 7fa27de00640 1 --2- 172.21.15.28:0/2192542812 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x557efa5a0ef0 0x557efa5a32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:28.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.647+0000 7fa27de00640 1 --2- 172.21.15.28:0/2192542812 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x557efa5a0ef0 0x557efa5a32e0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e6bdafd5c3f6369 server_cookie=9d389f2d6bd4b877 in_seq=0 out_seq=0 2024-04-23T14:48:28.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.647+0000 7fa25b400640 10 client.25925 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:48:28.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.649+0000 7fa25b400640 1 -- 172.21.15.28:0/2192542812 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x557efa48ca80 con 0x557efa5a0ef0 2024-04-23T14:48:28.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.649+0000 7fa25b400640 10 client.25925 handle_command_reply: tid=0 2024-04-23T14:48:28.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.649+0000 7fa28358f740 1 client.25925 shutdown 2024-04-23T14:48:28.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.649+0000 7fa28358f740 1 -- 172.21.15.28:0/2192542812 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557efa51f860 msgr2=0x557efa5401b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:28.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.649+0000 7fa28358f740 1 --2- 172.21.15.28:0/2192542812 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557efa51f860 0x557efa5401b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa268005d90 tx=0x7fa26800a390 comp rx=0 tx=0).stop 2024-04-23T14:48:28.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.649+0000 7fa28358f740 1 -- 172.21.15.28:0/2192542812 shutdown_connections 2024-04-23T14:48:28.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.649+0000 7fa28358f740 1 --2- 172.21.15.28:0/2192542812 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x557efa5a0ef0 0x557efa5a32e0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.649+0000 7fa28358f740 1 --2- 172.21.15.28:0/2192542812 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557efa52a850 0x557efa53da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.649+0000 7fa28358f740 1 --2- 172.21.15.28:0/2192542812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557efa520180 0x557efa53d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.649+0000 7fa28358f740 1 --2- 172.21.15.28:0/2192542812 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557efa51f860 0x557efa5401b0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.649+0000 7fa28358f740 1 -- 172.21.15.28:0/2192542812 >> 172.21.15.28:0/2192542812 conn(0x557efa51bb80 msgr2=0x557efa51a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:48:28.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.649+0000 7fa25b400640 0 client.25925 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:48:28.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.649+0000 7fa28358f740 1 -- 172.21.15.28:0/2192542812 shutdown_connections 2024-04-23T14:48:28.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.649+0000 7fa28358f740 1 -- 172.21.15.28:0/2192542812 wait complete. 2024-04-23T14:48:28.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.649+0000 7fa28358f740 20 client.25925 trim_cache size 0 max 16384 2024-04-23T14:48:28.653 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:48:28.653 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-23T14:48:28.653 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-23T14:48:28.653 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:48:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.650+0000 7fa280800640 1 -- 172.21.15.28:0/1332346881 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa2600a3e70 msgr2=0x7fa2600a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.650+0000 7fa280800640 1 --2- 172.21.15.28:0/1332346881 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa2600a3e70 0x7fa2600a6330 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa2740a4fe0 tx=0x7fa2740058d0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.650+0000 7fa280800640 1 -- 172.21.15.28:0/1332346881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2781a59e0 msgr2=0x7fa2781a5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.650+0000 7fa280800640 1 --2- 172.21.15.28:0/1332346881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2781a59e0 0x7fa2781a5e90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa27400eaa0 tx=0x7fa27400ef60 comp rx=0 tx=0).stop 2024-04-23T14:48:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.650+0000 7fa280800640 1 -- 172.21.15.28:0/1332346881 shutdown_connections 2024-04-23T14:48:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.650+0000 7fa280800640 1 --2- 172.21.15.28:0/1332346881 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa2600a3e70 0x7fa2600a6330 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.650+0000 7fa280800640 1 --2- 172.21.15.28:0/1332346881 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa27810dd30 0x7fa2781a1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.650+0000 7fa280800640 1 --2- 172.21.15.28:0/1332346881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2781a59e0 0x7fa2781a5e90 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.650+0000 7fa280800640 1 --2- 172.21.15.28:0/1332346881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa278109270 0x7fa2781a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.650+0000 7fa280800640 1 -- 172.21.15.28:0/1332346881 >> 172.21.15.28:0/1332346881 conn(0x7fa27806d110 msgr2=0x7fa278071df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:48:28.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.650+0000 7fa280800640 1 -- 172.21.15.28:0/1332346881 shutdown_connections 2024-04-23T14:48:28.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.650+0000 7fa280800640 1 -- 172.21.15.28:0/1332346881 wait complete. 2024-04-23T14:48:28.669 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-23T14:48:28.670 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:48:28.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.862+0000 7f3722600640 1 -- 172.21.15.28:0/4006151604 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f371c103c00 msgr2=0x7f371c10c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:28.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.862+0000 7f3722600640 1 --2- 172.21.15.28:0/4006151604 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f371c103c00 0x7f371c10c0e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3708009860 tx=0x7f370802a3d0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.863+0000 7f3722600640 1 -- 172.21.15.28:0/4006151604 shutdown_connections 2024-04-23T14:48:28.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.863+0000 7f3722600640 1 --2- 172.21.15.28:0/4006151604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f371c10c620 0x7f371c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.863+0000 7f3722600640 1 --2- 172.21.15.28:0/4006151604 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f371c103c00 0x7f371c10c0e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.863+0000 7f3722600640 1 --2- 172.21.15.28:0/4006151604 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f371c1032e0 0x7f371c1036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.863+0000 7f3722600640 1 -- 172.21.15.28:0/4006151604 >> 172.21.15.28:0/4006151604 conn(0x7f371c078060 msgr2=0x7f371c078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:48:28.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.863+0000 7f3722600640 1 -- 172.21.15.28:0/4006151604 shutdown_connections 2024-04-23T14:48:28.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.863+0000 7f3722600640 1 -- 172.21.15.28:0/4006151604 wait complete. 2024-04-23T14:48:28.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.864+0000 7f3722600640 1 Processor -- start 2024-04-23T14:48:28.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.864+0000 7f3722600640 1 -- start start 2024-04-23T14:48:28.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.865+0000 7f3722600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f371c1032e0 0x7f371c1a0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:28.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.865+0000 7f3722600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f371c103c00 0x7f371c1a0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:28.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.865+0000 7f3722600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f371c10c620 0x7f371c19a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:28.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.865+0000 7f3722600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f371c1123b0 con 0x7f371c103c00 2024-04-23T14:48:28.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.865+0000 7f3722600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f371c112230 con 0x7f371c1032e0 2024-04-23T14:48:28.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.865+0000 7f3722600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f371c112530 con 0x7f371c10c620 2024-04-23T14:48:28.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.865+0000 7f371b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f371c1032e0 0x7f371c1a0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:28.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.865+0000 7f371aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f371c103c00 0x7f371c1a0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:28.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.865+0000 7f371b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f371c1032e0 0x7f371c1a0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:55374/0 (socket says 172.21.15.28:55374) 2024-04-23T14:48:28.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.865+0000 7f371b400640 1 -- 172.21.15.28:0/2032627513 learned_addr learned my addr 172.21.15.28:0/2032627513 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:48:28.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.866+0000 7f371be00640 1 --2- 172.21.15.28:0/2032627513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f371c10c620 0x7f371c19a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:28.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.866+0000 7f371b400640 1 -- 172.21.15.28:0/2032627513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f371c10c620 msgr2=0x7f371c19a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:28.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.866+0000 7f371b400640 1 --2- 172.21.15.28:0/2032627513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f371c10c620 0x7f371c19a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.866+0000 7f371b400640 1 -- 172.21.15.28:0/2032627513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f371c103c00 msgr2=0x7f371c1a0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:28.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.866+0000 7f371b400640 1 --2- 172.21.15.28:0/2032627513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f371c103c00 0x7f371c1a0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:28.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.866+0000 7f371b400640 1 -- 172.21.15.28:0/2032627513 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f371c19b090 con 0x7f371c1032e0 2024-04-23T14:48:28.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.866+0000 7f371b400640 1 --2- 172.21.15.28:0/2032627513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f371c1032e0 0x7f371c1a0760 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f370c00e9c0 tx=0x7f370c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:48:28.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.866+0000 7f3713e00640 1 -- 172.21.15.28:0/2032627513 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f370c00cde0 con 0x7f371c1032e0 2024-04-23T14:48:28.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.867+0000 7f3722600640 1 -- 172.21.15.28:0/2032627513 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f371c19b380 con 0x7f371c1032e0 2024-04-23T14:48:28.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.867+0000 7f3713e00640 1 -- 172.21.15.28:0/2032627513 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f370c0035f0 con 0x7f371c1032e0 2024-04-23T14:48:28.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.867+0000 7f3713e00640 1 -- 172.21.15.28:0/2032627513 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f370c010690 con 0x7f371c1032e0 2024-04-23T14:48:28.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.867+0000 7f3722600640 1 -- 172.21.15.28:0/2032627513 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f371c1a7620 con 0x7f371c1032e0 2024-04-23T14:48:28.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.868+0000 7f3722600640 1 -- 172.21.15.28:0/2032627513 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f371c069270 con 0x7f371c1032e0 2024-04-23T14:48:28.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.870+0000 7f3713e00640 1 -- 172.21.15.28:0/2032627513 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f370c0037a0 con 0x7f371c1032e0 2024-04-23T14:48:28.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.871+0000 7f3713e00640 1 --2- 172.21.15.28:0/2032627513 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f37000a3d20 0x7f37000a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:28.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.871+0000 7f3713e00640 1 -- 172.21.15.28:0/2032627513 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f370c0d0c20 con 0x7f371c1032e0 2024-04-23T14:48:28.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.872+0000 7f371aa00640 1 --2- 172.21.15.28:0/2032627513 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f37000a3d20 0x7f37000a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:28.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.872+0000 7f371aa00640 1 --2- 172.21.15.28:0/2032627513 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f37000a3d20 0x7f37000a61e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f371c19c100 tx=0x7f37080094d0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:48:28.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:28.875+0000 7f3713e00640 1 -- 172.21.15.28:0/2032627513 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f370c09d270 con 0x7f371c1032e0 2024-04-23T14:48:29.088 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:28 smithi105 ceph-mon[32555]: pgmap v136: 97 pgs: 97 active+clean; 176 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.9 MiB/s wr, 236 op/s 2024-04-23T14:48:29.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:28 smithi038 ceph-mon[32348]: pgmap v136: 97 pgs: 97 active+clean; 176 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.9 MiB/s wr, 236 op/s 2024-04-23T14:48:29.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:28 smithi028 ceph-mon[27810]: pgmap v136: 97 pgs: 97 active+clean; 176 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.9 MiB/s wr, 236 op/s 2024-04-23T14:48:29.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.159+0000 7f3722600640 1 -- 172.21.15.28:0/2032627513 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f371c19be40 con 0x7f371c1032e0 2024-04-23T14:48:29.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.160+0000 7f3713e00640 1 -- 172.21.15.28:0/2032627513 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7f370c09cb70 con 0x7f371c1032e0 2024-04-23T14:48:29.161 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:48:29.161 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:48:29.162 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:48:29.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.162+0000 7f3722600640 1 -- 172.21.15.28:0/2032627513 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f37000a3d20 msgr2=0x7f37000a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:29.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.162+0000 7f3722600640 1 --2- 172.21.15.28:0/2032627513 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f37000a3d20 0x7f37000a61e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f371c19c100 tx=0x7f37080094d0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.162+0000 7f3722600640 1 -- 172.21.15.28:0/2032627513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f371c1032e0 msgr2=0x7f371c1a0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:29.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.162+0000 7f3722600640 1 --2- 172.21.15.28:0/2032627513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f371c1032e0 0x7f371c1a0760 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f370c00e9c0 tx=0x7f370c00ee80 comp rx=0 tx=0).stop 2024-04-23T14:48:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.162+0000 7f3722600640 1 -- 172.21.15.28:0/2032627513 shutdown_connections 2024-04-23T14:48:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.162+0000 7f3722600640 1 --2- 172.21.15.28:0/2032627513 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f37000a3d20 0x7f37000a61e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.162+0000 7f3722600640 1 --2- 172.21.15.28:0/2032627513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f371c10c620 0x7f371c19a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.162+0000 7f3722600640 1 --2- 172.21.15.28:0/2032627513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f371c103c00 0x7f371c1a0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.162+0000 7f3722600640 1 --2- 172.21.15.28:0/2032627513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f371c1032e0 0x7f371c1a0760 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.162+0000 7f3722600640 1 -- 172.21.15.28:0/2032627513 >> 172.21.15.28:0/2032627513 conn(0x7f371c078060 msgr2=0x7f371c108530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:48:29.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.163+0000 7f3722600640 1 -- 172.21.15.28:0/2032627513 shutdown_connections 2024-04-23T14:48:29.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.163+0000 7f3722600640 1 -- 172.21.15.28:0/2032627513 wait complete. 2024-04-23T14:48:29.185 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-23T14:48:29.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.405+0000 7f66cf200640 1 -- 172.21.15.28:0/3961760986 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66c8100e20 msgr2=0x7f66c8101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:29.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.405+0000 7f66cf200640 1 --2- 172.21.15.28:0/3961760986 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66c8100e20 0x7f66c8101200 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f66b8009900 tx=0x7f66b802a3f0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.405+0000 7f66cf200640 1 -- 172.21.15.28:0/3961760986 shutdown_connections 2024-04-23T14:48:29.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.405+0000 7f66cf200640 1 --2- 172.21.15.28:0/3961760986 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66c810e9d0 0x7f66c8110dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.405+0000 7f66cf200640 1 --2- 172.21.15.28:0/3961760986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66c8101740 0x7f66c810e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.405+0000 7f66cf200640 1 --2- 172.21.15.28:0/3961760986 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66c8100e20 0x7f66c8101200 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.405+0000 7f66cf200640 1 -- 172.21.15.28:0/3961760986 >> 172.21.15.28:0/3961760986 conn(0x7f66c8078060 msgr2=0x7f66c8078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:48:29.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.405+0000 7f66cf200640 1 -- 172.21.15.28:0/3961760986 shutdown_connections 2024-04-23T14:48:29.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.405+0000 7f66cf200640 1 -- 172.21.15.28:0/3961760986 wait complete. 2024-04-23T14:48:29.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.407+0000 7f66cf200640 1 Processor -- start 2024-04-23T14:48:29.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.407+0000 7f66cf200640 1 -- start start 2024-04-23T14:48:29.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.407+0000 7f66cf200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66c8100e20 0x7f66c81a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:29.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.407+0000 7f66cf200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66c8101740 0x7f66c81a0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:29.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.407+0000 7f66cf200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66c810e9d0 0x7f66c819a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:29.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.407+0000 7f66cf200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f66c8114650 con 0x7f66c8101740 2024-04-23T14:48:29.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.407+0000 7f66cf200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f66c81144d0 con 0x7f66c8100e20 2024-04-23T14:48:29.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.407+0000 7f66cf200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f66c81147d0 con 0x7f66c810e9d0 2024-04-23T14:48:29.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.408+0000 7f66c7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66c8100e20 0x7f66c81a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:29.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.408+0000 7f66c7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66c8100e20 0x7f66c81a0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:55406/0 (socket says 172.21.15.28:55406) 2024-04-23T14:48:29.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.408+0000 7f66c7e00640 1 -- 172.21.15.28:0/2189201864 learned_addr learned my addr 172.21.15.28:0/2189201864 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:48:29.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.408+0000 7f66cd200640 1 --2- 172.21.15.28:0/2189201864 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66c810e9d0 0x7f66c819a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:29.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.408+0000 7f66c7400640 1 --2- 172.21.15.28:0/2189201864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66c8101740 0x7f66c81a0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:29.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.408+0000 7f66c7e00640 1 -- 172.21.15.28:0/2189201864 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66c810e9d0 msgr2=0x7f66c819a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:29.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.408+0000 7f66c7e00640 1 --2- 172.21.15.28:0/2189201864 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66c810e9d0 0x7f66c819a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.408+0000 7f66c7e00640 1 -- 172.21.15.28:0/2189201864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66c8101740 msgr2=0x7f66c81a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:29.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.408+0000 7f66c7e00640 1 --2- 172.21.15.28:0/2189201864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66c8101740 0x7f66c81a0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.408+0000 7f66c7e00640 1 -- 172.21.15.28:0/2189201864 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66c819b0d0 con 0x7f66c8100e20 2024-04-23T14:48:29.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.409+0000 7f66c7400640 1 --2- 172.21.15.28:0/2189201864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66c8101740 0x7f66c81a0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:48:29.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.409+0000 7f66c7e00640 1 --2- 172.21.15.28:0/2189201864 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66c8100e20 0x7f66c81a0750 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f66b8004770 tx=0x7f66b8003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:48:29.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.409+0000 7f66c4c00640 1 -- 172.21.15.28:0/2189201864 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b803f070 con 0x7f66c8100e20 2024-04-23T14:48:29.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.409+0000 7f66c4c00640 1 -- 172.21.15.28:0/2189201864 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66b8031ca0 con 0x7f66c8100e20 2024-04-23T14:48:29.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.409+0000 7f66cf200640 1 -- 172.21.15.28:0/2189201864 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66c819b360 con 0x7f66c8100e20 2024-04-23T14:48:29.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.409+0000 7f66c4c00640 1 -- 172.21.15.28:0/2189201864 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b8035030 con 0x7f66c8100e20 2024-04-23T14:48:29.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.409+0000 7f66cf200640 1 -- 172.21.15.28:0/2189201864 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66c81a7610 con 0x7f66c8100e20 2024-04-23T14:48:29.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.413+0000 7f66d2197740 1 Processor -- start 2024-04-23T14:48:29.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.413+0000 7f66c4c00640 1 -- 172.21.15.28:0/2189201864 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f66b8042070 con 0x7f66c8100e20 2024-04-23T14:48:29.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.413+0000 7f66d2197740 1 -- start start 2024-04-23T14:48:29.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.414+0000 7f66d2197740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f495999950 0x55f495999d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:29.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.414+0000 7f66d2197740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f49599a270 0x55f4959a4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:29.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.414+0000 7f66d2197740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4959a4940 0x55f4959a6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:29.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.414+0000 7f66d2197740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f4958ff1d0 con 0x55f495999950 2024-04-23T14:48:29.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.414+0000 7f66d2197740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f4958a5450 con 0x55f4959a4940 2024-04-23T14:48:29.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.414+0000 7f66d2197740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f4958fb190 con 0x55f49599a270 2024-04-23T14:48:29.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.414+0000 7f66c7400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4959a4940 0x55f4959a6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:29.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.414+0000 7f66c7400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4959a4940 0x55f4959a6d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:55412/0 (socket says 172.21.15.28:55412) 2024-04-23T14:48:29.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.414+0000 7f66c7400640 1 -- 172.21.15.28:0/1398117836 learned_addr learned my addr 172.21.15.28:0/1398117836 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:48:29.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.414+0000 7f66cd200640 1 --2- 172.21.15.28:0/1398117836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f495999950 0x55f495999d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:29.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.414+0000 7f66c7e00640 1 --2- 172.21.15.28:0/1398117836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f49599a270 0x55f4959a4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:29.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.414+0000 7f66c7400640 1 -- 172.21.15.28:0/1398117836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f49599a270 msgr2=0x55f4959a4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:48:29.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.414+0000 7f66c7400640 1 --2- 172.21.15.28:0/1398117836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f49599a270 0x55f4959a4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.414+0000 7f66c7400640 1 -- 172.21.15.28:0/1398117836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f495999950 msgr2=0x55f495999d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:48:29.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.414+0000 7f66c7400640 1 --2- 172.21.15.28:0/1398117836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f495999950 0x55f495999d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.414+0000 7f66c7400640 1 -- 172.21.15.28:0/1398117836 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f495962310 con 0x55f4959a4940 2024-04-23T14:48:29.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.415+0000 7f66c7400640 1 --2- 172.21.15.28:0/1398117836 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4959a4940 0x55f4959a6d30 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f66b000bc80 tx=0x7f66b000f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87804b1651844ce4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:48:29.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.415+0000 7f66c4c00640 1 --2- 172.21.15.28:0/2189201864 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f66ac0a3d20 0x7f66ac0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:29.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.415+0000 7f66c4c00640 1 -- 172.21.15.28:0/2189201864 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f66b80f10b0 con 0x7f66c8100e20 2024-04-23T14:48:29.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.416+0000 7f66cd200640 1 --2- 172.21.15.28:0/2189201864 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f66ac0a3d20 0x7f66ac0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:29.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.416+0000 7f66bd600640 1 -- 172.21.15.28:0/1398117836 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b0011440 con 0x55f4959a4940 2024-04-23T14:48:29.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.416+0000 7f66bd600640 1 -- 172.21.15.28:0/1398117836 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66b0011ac0 con 0x55f4959a4940 2024-04-23T14:48:29.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.416+0000 7f66bd600640 1 -- 172.21.15.28:0/1398117836 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b001bac0 con 0x55f4959a4940 2024-04-23T14:48:29.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.417+0000 7f66d2197740 1 -- 172.21.15.28:0/1398117836 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4959a4940 msgr2=0x55f4959a6d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:29.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.417+0000 7f66d2197740 1 --2- 172.21.15.28:0/1398117836 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4959a4940 0x55f4959a6d30 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f66b000bc80 tx=0x7f66b000f730 comp rx=0 tx=0).stop 2024-04-23T14:48:29.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.417+0000 7f66cd200640 1 --2- 172.21.15.28:0/2189201864 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f66ac0a3d20 0x7f66ac0a61e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f66c806a4a0 tx=0x7f66c00096e0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:48:29.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.417+0000 7f66d2197740 1 -- 172.21.15.28:0/1398117836 shutdown_connections 2024-04-23T14:48:29.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.417+0000 7f66d2197740 1 --2- 172.21.15.28:0/1398117836 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4959a4940 0x55f4959a6d30 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.417+0000 7f66d2197740 1 --2- 172.21.15.28:0/1398117836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f49599a270 0x55f4959a4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.417+0000 7f66d2197740 1 --2- 172.21.15.28:0/1398117836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f495999950 0x55f495999d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.417+0000 7f66d2197740 1 -- 172.21.15.28:0/1398117836 >> 172.21.15.28:0/1398117836 conn(0x55f495995c70 msgr2=0x55f495996050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:48:29.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.417+0000 7f66d2197740 1 -- 172.21.15.28:0/1398117836 shutdown_connections 2024-04-23T14:48:29.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.417+0000 7f66d2197740 1 -- 172.21.15.28:0/1398117836 wait complete. 2024-04-23T14:48:29.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.418+0000 7f66d2197740 1 -- start start 2024-04-23T14:48:29.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.418+0000 7f66d2197740 1 Processor -- start 2024-04-23T14:48:29.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.418+0000 7f66d2197740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f495999950 0x55f4959a43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:29.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.418+0000 7f66d2197740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f49599a270 0x55f4959a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:29.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.418+0000 7f66d2197740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4959a4940 0x55f4959a1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:29.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.418+0000 7f66d2197740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f4959a77d0 con 0x55f4959a4940 2024-04-23T14:48:29.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.418+0000 7f66d2197740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f4958b8810 con 0x55f495999950 2024-04-23T14:48:29.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.418+0000 7f66d2197740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f4959a7950 con 0x55f49599a270 2024-04-23T14:48:29.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.419+0000 7f66d2197740 10 client.0 _ll_register_callbacks cb 0x55f495886140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:48:29.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.419+0000 7f66c7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f495999950 0x55f4959a43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:29.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.419+0000 7f66c7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f495999950 0x55f4959a43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:55420/0 (socket says 172.21.15.28:55420) 2024-04-23T14:48:29.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.419+0000 7f66c7e00640 1 -- 172.21.15.28:0/818941617 learned_addr learned my addr 172.21.15.28:0/818941617 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:48:29.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.419+0000 7f66c7400640 1 --2- 172.21.15.28:0/818941617 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f49599a270 0x55f4959a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:29.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.419+0000 7f66cd200640 1 --2- 172.21.15.28:0/818941617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4959a4940 0x55f4959a1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:29.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.419+0000 7f66c7e00640 1 -- 172.21.15.28:0/818941617 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f49599a270 msgr2=0x55f4959a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:48:29.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.419+0000 7f66c7e00640 1 --2- 172.21.15.28:0/818941617 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f49599a270 0x55f4959a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.419+0000 7f66c7e00640 1 -- 172.21.15.28:0/818941617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4959a4940 msgr2=0x55f4959a1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:48:29.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.419+0000 7f66c7e00640 1 --2- 172.21.15.28:0/818941617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4959a4940 0x55f4959a1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.419+0000 7f66c7e00640 1 -- 172.21.15.28:0/818941617 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4958c93f0 con 0x55f495999950 2024-04-23T14:48:29.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.419+0000 7f66c7e00640 1 --2- 172.21.15.28:0/818941617 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f495999950 0x55f4959a43f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f66b80ee810 tx=0x7f66b802a3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4f8cc24a0657d77 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:48:29.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.419+0000 7f66b6000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:48:29.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.420+0000 7f66b6000640 1 -- 172.21.15.28:0/818941617 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b8049050 con 0x55f495999950 2024-04-23T14:48:29.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.420+0000 7f66b6000640 1 -- 172.21.15.28:0/818941617 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f66b80eea30 con 0x55f495999950 2024-04-23T14:48:29.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.420+0000 7f66d2197740 1 -- 172.21.15.28:0/818941617 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66b0011ac0 con 0x55f495999950 2024-04-23T14:48:29.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.420+0000 7f66b6000640 1 -- 172.21.15.28:0/818941617 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f66b80eec30 con 0x55f495999950 2024-04-23T14:48:29.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.420+0000 7f66d2197740 10 client.25943 fetch_fsmap learned FSMap version 20 2024-04-23T14:48:29.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.420+0000 7f66d2197740 1 -- 172.21.15.28:0/818941617 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55f4956113f0 con 0x55f495999950 2024-04-23T14:48:29.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.420+0000 7f66b6000640 1 -- 172.21.15.28:0/818941617 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66b0011ac0 con 0x55f495999950 2024-04-23T14:48:29.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.420+0000 7f66b6000640 1 -- 172.21.15.28:0/818941617 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f66b80f3110 con 0x55f495999950 2024-04-23T14:48:29.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.420+0000 7f66d2197740 10 client.25943 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:48:29.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.420+0000 7f66d2197740 10 client.25943 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:48:29.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.421+0000 7f66d2197740 20 client.25943 populate_metadata read hostname 'smithi028' 2024-04-23T14:48:29.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.421+0000 7f66d2197740 1 --2- 172.21.15.28:0/818941617 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f4959b4a00 0x55f4959b6e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:29.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.421+0000 7f66d2197740 4 client.25943 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:48:29.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.421+0000 7f66d2197740 1 -- 172.21.15.28:0/818941617 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f49590cbf0 con 0x55f4959b4a00 2024-04-23T14:48:29.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.421+0000 7f66c7400640 1 --2- 172.21.15.28:0/818941617 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f4959b4a00 0x55f4959b6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:29.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.421+0000 7f66c7400640 1 --2- 172.21.15.28:0/818941617 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f4959b4a00 0x55f4959b6e80 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c672b4f10fc2fcb6 server_cookie=72c6ef2d79e729a1 in_seq=0 out_seq=0 2024-04-23T14:48:29.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.421+0000 7f66b6000640 10 client.25943 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:48:29.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.422+0000 7f66b6000640 1 -- 172.21.15.28:0/818941617 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f49590cbf0 con 0x55f4959b4a00 2024-04-23T14:48:29.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.422+0000 7f66b6000640 10 client.25943 handle_command_reply: tid=0 2024-04-23T14:48:29.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.422+0000 7f66d2197740 1 client.25943 shutdown 2024-04-23T14:48:29.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.423+0000 7f66d2197740 1 -- 172.21.15.28:0/818941617 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f495999950 msgr2=0x55f4959a43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:29.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.423+0000 7f66d2197740 1 --2- 172.21.15.28:0/818941617 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f495999950 0x55f4959a43f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f66b80ee810 tx=0x7f66b802a3f0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.423+0000 7f66d2197740 1 -- 172.21.15.28:0/818941617 shutdown_connections 2024-04-23T14:48:29.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.423+0000 7f66d2197740 1 --2- 172.21.15.28:0/818941617 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f4959b4a00 0x55f4959b6e80 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.423+0000 7f66d2197740 1 --2- 172.21.15.28:0/818941617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4959a4940 0x55f4959a1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.423+0000 7f66b6000640 0 client.25943 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:48:29.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.423+0000 7f66d2197740 1 --2- 172.21.15.28:0/818941617 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f49599a270 0x55f4959a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.423+0000 7f66d2197740 1 --2- 172.21.15.28:0/818941617 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f495999950 0x55f4959a43f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.423+0000 7f66d2197740 1 -- 172.21.15.28:0/818941617 >> 172.21.15.28:0/818941617 conn(0x55f495995c70 msgr2=0x55f49599ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:48:29.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.423+0000 7f66d2197740 1 -- 172.21.15.28:0/818941617 shutdown_connections 2024-04-23T14:48:29.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.423+0000 7f66d2197740 1 -- 172.21.15.28:0/818941617 wait complete. 2024-04-23T14:48:29.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.423+0000 7f66d2197740 20 client.25943 trim_cache size 0 max 16384 2024-04-23T14:48:29.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.440+0000 7f66d2197740 1 Processor -- start 2024-04-23T14:48:29.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.441+0000 7f66d2197740 1 -- start start 2024-04-23T14:48:29.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.441+0000 7f66d2197740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f495999950 0x55f4959b0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:29.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.441+0000 7f66d2197740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f49599a270 0x55f4959a3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:29.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.441+0000 7f66d2197740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4959a4940 0x55f4959bc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:29.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.441+0000 7f66d2197740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f66c81adf30 con 0x55f49599a270 2024-04-23T14:48:29.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.441+0000 7f66d2197740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f66c81b07b0 con 0x55f4959a4940 2024-04-23T14:48:29.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.441+0000 7f66d2197740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f66c81ae5a0 con 0x55f495999950 2024-04-23T14:48:29.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.441+0000 7f66cd200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4959a4940 0x55f4959bc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:29.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.441+0000 7f66cd200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4959a4940 0x55f4959bc7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:55428/0 (socket says 172.21.15.28:55428) 2024-04-23T14:48:29.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.441+0000 7f66c7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f49599a270 0x55f4959a3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:29.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.441+0000 7f66cd200640 1 -- 172.21.15.28:0/1396796696 learned_addr learned my addr 172.21.15.28:0/1396796696 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:48:29.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.442+0000 7f66c7e00640 1 --2- 172.21.15.28:0/1396796696 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f495999950 0x55f4959b0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:29.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.442+0000 7f66cd200640 1 -- 172.21.15.28:0/1396796696 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f495999950 msgr2=0x55f4959b0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:48:29.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.442+0000 7f66cd200640 1 --2- 172.21.15.28:0/1396796696 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f495999950 0x55f4959b0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.442+0000 7f66cd200640 1 -- 172.21.15.28:0/1396796696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f49599a270 msgr2=0x55f4959a3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:48:29.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.442+0000 7f66cd200640 1 --2- 172.21.15.28:0/1396796696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f49599a270 0x55f4959a3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.442+0000 7f66cd200640 1 -- 172.21.15.28:0/1396796696 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f49599a0a0 con 0x55f4959a4940 2024-04-23T14:48:29.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.442+0000 7f66c7400640 1 --2- 172.21.15.28:0/1396796696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f49599a270 0x55f4959a3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:48:29.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.442+0000 7f66cd200640 1 --2- 172.21.15.28:0/1396796696 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4959a4940 0x55f4959bc7a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f66c001f490 tx=0x7f66c0015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd15ba51c6a2a0d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:48:29.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.443+0000 7f66bcc00640 1 -- 172.21.15.28:0/1396796696 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c0029630 con 0x55f4959a4940 2024-04-23T14:48:29.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.443+0000 7f66bcc00640 1 -- 172.21.15.28:0/1396796696 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66c00208e0 con 0x55f4959a4940 2024-04-23T14:48:29.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.443+0000 7f66bcc00640 1 -- 172.21.15.28:0/1396796696 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c0028a90 con 0x55f4959a4940 2024-04-23T14:48:29.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.444+0000 7f66d2197740 1 -- 172.21.15.28:0/1396796696 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4959a4940 msgr2=0x55f4959bc7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:29.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.444+0000 7f66d2197740 1 --2- 172.21.15.28:0/1396796696 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4959a4940 0x55f4959bc7a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f66c001f490 tx=0x7f66c0015000 comp rx=0 tx=0).stop 2024-04-23T14:48:29.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.444+0000 7f66d2197740 1 -- 172.21.15.28:0/1396796696 shutdown_connections 2024-04-23T14:48:29.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.444+0000 7f66d2197740 1 --2- 172.21.15.28:0/1396796696 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4959a4940 0x55f4959bc7a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.444+0000 7f66d2197740 1 --2- 172.21.15.28:0/1396796696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f49599a270 0x55f4959a3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.444+0000 7f66d2197740 1 --2- 172.21.15.28:0/1396796696 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f495999950 0x55f4959b0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.444+0000 7f66d2197740 1 -- 172.21.15.28:0/1396796696 >> 172.21.15.28:0/1396796696 conn(0x55f495995c70 msgr2=0x55f495996050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:48:29.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.444+0000 7f66d2197740 1 -- 172.21.15.28:0/1396796696 shutdown_connections 2024-04-23T14:48:29.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.444+0000 7f66d2197740 1 -- 172.21.15.28:0/1396796696 wait complete. 2024-04-23T14:48:29.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.444+0000 7f66d2197740 1 -- start start 2024-04-23T14:48:29.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.445+0000 7f66d2197740 1 Processor -- start 2024-04-23T14:48:29.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.445+0000 7f66d2197740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f495999950 0x55f4959acba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:29.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.445+0000 7f66d2197740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f49599a270 0x55f4959ad0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:29.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.445+0000 7f66d2197740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f4959a4940 0x55f49599d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:29.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.445+0000 7f66d2197740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f66c81aec70 con 0x55f49599a270 2024-04-23T14:48:29.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.445+0000 7f66d2197740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f66c81b00e0 con 0x55f495999950 2024-04-23T14:48:29.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.445+0000 7f66d2197740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f66c81af340 con 0x55f4959a4940 2024-04-23T14:48:29.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.445+0000 7f66d2197740 10 client.0 _ll_register_callbacks cb 0x7f66b80f4690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:48:29.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.445+0000 7f66c7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f495999950 0x55f4959acba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:29.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.445+0000 7f66c7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f49599a270 0x55f4959ad0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:29.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.446+0000 7f66c7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f495999950 0x55f4959acba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:55430/0 (socket says 172.21.15.28:55430) 2024-04-23T14:48:29.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.446+0000 7f66c7e00640 1 -- 172.21.15.28:0/2067719669 learned_addr learned my addr 172.21.15.28:0/2067719669 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:48:29.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.446+0000 7f66cd200640 1 --2- 172.21.15.28:0/2067719669 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f4959a4940 0x55f49599d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:29.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.446+0000 7f66c7e00640 1 -- 172.21.15.28:0/2067719669 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f4959a4940 msgr2=0x55f49599d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:48:29.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.446+0000 7f66c7e00640 1 --2- 172.21.15.28:0/2067719669 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f4959a4940 0x55f49599d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.446+0000 7f66c7e00640 1 -- 172.21.15.28:0/2067719669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f49599a270 msgr2=0x55f4959ad0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:48:29.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.446+0000 7f66c7e00640 1 --2- 172.21.15.28:0/2067719669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f49599a270 0x55f4959ad0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.446+0000 7f66c7e00640 1 -- 172.21.15.28:0/2067719669 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4956db7d0 con 0x55f495999950 2024-04-23T14:48:29.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.446+0000 7f66cd200640 1 --2- 172.21.15.28:0/2067719669 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f4959a4940 0x55f49599d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:48:29.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.446+0000 7f66c7e00640 1 --2- 172.21.15.28:0/2067719669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f495999950 0x55f4959acba0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f66b80b99d0 tx=0x7f66b80c5ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=913cc44b14057f33 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:48:29.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.446+0000 7f66b6000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:48:29.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.446+0000 7f66b6000640 1 -- 172.21.15.28:0/2067719669 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b80da030 con 0x55f495999950 2024-04-23T14:48:29.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.446+0000 7f66b6000640 1 -- 172.21.15.28:0/2067719669 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f66b80b7b70 con 0x55f495999950 2024-04-23T14:48:29.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.446+0000 7f66d2197740 1 -- 172.21.15.28:0/2067719669 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66c00208e0 con 0x55f495999950 2024-04-23T14:48:29.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.447+0000 7f66b6000640 1 -- 172.21.15.28:0/2067719669 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f66b80bcb40 con 0x55f495999950 2024-04-23T14:48:29.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.447+0000 7f66d2197740 10 client.25949 fetch_fsmap learned FSMap version 20 2024-04-23T14:48:29.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.447+0000 7f66d2197740 1 -- 172.21.15.28:0/2067719669 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55f49599f290 con 0x55f495999950 2024-04-23T14:48:29.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.447+0000 7f66b6000640 1 -- 172.21.15.28:0/2067719669 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66c00208e0 con 0x55f495999950 2024-04-23T14:48:29.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.447+0000 7f66b6000640 1 -- 172.21.15.28:0/2067719669 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f66b80bc450 con 0x55f495999950 2024-04-23T14:48:29.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.447+0000 7f66d2197740 10 client.25949 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:48:29.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.447+0000 7f66d2197740 10 client.25949 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:48:29.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.447+0000 7f66d2197740 20 client.25949 populate_metadata read hostname 'smithi028' 2024-04-23T14:48:29.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.447+0000 7f66d2197740 1 --2- 172.21.15.28:0/2067719669 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f495a1c020 0x55f495a1e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:29.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.447+0000 7f66d2197740 4 client.25949 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-23T14:48:29.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.447+0000 7f66d2197740 1 -- 172.21.15.28:0/2067719669 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f495906a60 con 0x55f495a1c020 2024-04-23T14:48:29.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.447+0000 7f66c7400640 1 --2- 172.21.15.28:0/2067719669 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f495a1c020 0x55f495a1e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:29.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.448+0000 7f66c7400640 1 --2- 172.21.15.28:0/2067719669 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f495a1c020 0x55f495a1e410 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8ee9451aadff2a7 server_cookie=855ee6781d605b09 in_seq=0 out_seq=0 2024-04-23T14:48:29.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.448+0000 7f66b6000640 10 client.25949 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:48:29.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.448+0000 7f66b6000640 1 -- 172.21.15.28:0/2067719669 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f495906a60 con 0x55f495a1c020 2024-04-23T14:48:29.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.448+0000 7f66b6000640 10 client.25949 handle_command_reply: tid=0 2024-04-23T14:48:29.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.448+0000 7f66d2197740 1 client.25949 shutdown 2024-04-23T14:48:29.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.449+0000 7f66d2197740 1 -- 172.21.15.28:0/2067719669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f495999950 msgr2=0x55f4959acba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:29.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.449+0000 7f66d2197740 1 --2- 172.21.15.28:0/2067719669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f495999950 0x55f4959acba0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f66b80b99d0 tx=0x7f66b80c5ba0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.449+0000 7f66d2197740 1 -- 172.21.15.28:0/2067719669 shutdown_connections 2024-04-23T14:48:29.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.449+0000 7f66d2197740 1 --2- 172.21.15.28:0/2067719669 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f495a1c020 0x55f495a1e410 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.449+0000 7f66d2197740 1 --2- 172.21.15.28:0/2067719669 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f4959a4940 0x55f49599d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.449+0000 7f66d2197740 1 --2- 172.21.15.28:0/2067719669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f49599a270 0x55f4959ad0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.449+0000 7f66b6000640 0 client.25949 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:48:29.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.449+0000 7f66d2197740 1 --2- 172.21.15.28:0/2067719669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f495999950 0x55f4959acba0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.449+0000 7f66d2197740 1 -- 172.21.15.28:0/2067719669 >> 172.21.15.28:0/2067719669 conn(0x55f495995c70 msgr2=0x55f495994560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:48:29.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.449+0000 7f66d2197740 1 -- 172.21.15.28:0/2067719669 shutdown_connections 2024-04-23T14:48:29.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.449+0000 7f66d2197740 1 -- 172.21.15.28:0/2067719669 wait complete. 2024-04-23T14:48:29.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.449+0000 7f66d2197740 20 client.25949 trim_cache size 0 max 16384 2024-04-23T14:48:29.454 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-23T14:48:29.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.450+0000 7f66cf200640 1 -- 172.21.15.28:0/2189201864 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f66ac0a3d20 msgr2=0x7f66ac0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:29.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.450+0000 7f66cf200640 1 --2- 172.21.15.28:0/2189201864 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f66ac0a3d20 0x7f66ac0a61e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f66c806a4a0 tx=0x7f66c00096e0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.450+0000 7f66cf200640 1 -- 172.21.15.28:0/2189201864 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66c8100e20 msgr2=0x7f66c81a0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:29.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.450+0000 7f66cf200640 1 --2- 172.21.15.28:0/2189201864 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66c8100e20 0x7f66c81a0750 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f66b8004770 tx=0x7f66b8003c10 comp rx=0 tx=0).stop 2024-04-23T14:48:29.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.450+0000 7f66cf200640 1 -- 172.21.15.28:0/2189201864 shutdown_connections 2024-04-23T14:48:29.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.450+0000 7f66cf200640 1 --2- 172.21.15.28:0/2189201864 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f66ac0a3d20 0x7f66ac0a61e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.450+0000 7f66cf200640 1 --2- 172.21.15.28:0/2189201864 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66c810e9d0 0x7f66c819a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.450+0000 7f66cf200640 1 --2- 172.21.15.28:0/2189201864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66c8101740 0x7f66c81a0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.450+0000 7f66cf200640 1 --2- 172.21.15.28:0/2189201864 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66c8100e20 0x7f66c81a0750 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:29.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.450+0000 7f66cf200640 1 -- 172.21.15.28:0/2189201864 >> 172.21.15.28:0/2189201864 conn(0x7f66c8078060 msgr2=0x7f66c810ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:48:29.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.450+0000 7f66cf200640 1 -- 172.21.15.28:0/2189201864 shutdown_connections 2024-04-23T14:48:29.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:29.450+0000 7f66cf200640 1 -- 172.21.15.28:0/2189201864 wait complete. 2024-04-23T14:48:29.482 INFO:tasks.workunit.client.0.smithi028.stdout: 1 315706 9.76 MB/sec execute 45 sec latency 58.766 ms 2024-04-23T14:48:30.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:29 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2032627513' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:48:30.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:29 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2032627513' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:48:30.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:29 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2032627513' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:48:30.475 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-23T14:48:30.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:48:30.475 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-23T14:48:30.482 INFO:tasks.workunit.client.0.smithi028.stdout: 1 316840 9.63 MB/sec execute 46 sec latency 45.639 ms 2024-04-23T14:48:30.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.709+0000 7fa3b5400640 1 -- 172.21.15.28:0/2782747514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3b0101070 msgr2=0x7fa3b0101450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:30.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.709+0000 7fa3b5400640 1 --2- 172.21.15.28:0/2782747514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3b0101070 0x7fa3b0101450 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0009ca0 tx=0x7fa3a002a9e0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.709+0000 7fa3b5400640 1 -- 172.21.15.28:0/2782747514 shutdown_connections 2024-04-23T14:48:30.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.709+0000 7fa3b5400640 1 --2- 172.21.15.28:0/2782747514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3b0101990 0x7fa3b0105e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.709+0000 7fa3b5400640 1 --2- 172.21.15.28:0/2782747514 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa3b0106390 0x7fa3b0106820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.709+0000 7fa3b5400640 1 --2- 172.21.15.28:0/2782747514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3b0101070 0x7fa3b0101450 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.709+0000 7fa3b5400640 1 -- 172.21.15.28:0/2782747514 >> 172.21.15.28:0/2782747514 conn(0x7fa3b0078000 msgr2=0x7fa3b0078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:48:30.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.709+0000 7fa3b5400640 1 -- 172.21.15.28:0/2782747514 shutdown_connections 2024-04-23T14:48:30.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.709+0000 7fa3b5400640 1 -- 172.21.15.28:0/2782747514 wait complete. 2024-04-23T14:48:30.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.711+0000 7fa3b5400640 1 Processor -- start 2024-04-23T14:48:30.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.711+0000 7fa3b5400640 1 -- start start 2024-04-23T14:48:30.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.712+0000 7fa3b5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3b0101070 0x7fa3b019eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:30.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.712+0000 7fa3b5400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa3b0101990 0x7fa3b019f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:30.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.712+0000 7fa3b5400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3b0106390 0x7fa3b01a3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:30.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.712+0000 7fa3b5400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b01189c0 con 0x7fa3b0101070 2024-04-23T14:48:30.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.712+0000 7fa3b5400640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b0118840 con 0x7fa3b0106390 2024-04-23T14:48:30.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.712+0000 7fa3b5400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b0118b40 con 0x7fa3b0101990 2024-04-23T14:48:30.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.712+0000 7fa3ae800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3b0101070 0x7fa3b019eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:30.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.712+0000 7fa3ae800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3b0101070 0x7fa3b019eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59886/0 (socket says 172.21.15.28:59886) 2024-04-23T14:48:30.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.712+0000 7fa3ae800640 1 -- 172.21.15.28:0/2831012117 learned_addr learned my addr 172.21.15.28:0/2831012117 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:48:30.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.712+0000 7fa3ade00640 1 --2- 172.21.15.28:0/2831012117 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa3b0101990 0x7fa3b019f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:30.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.712+0000 7fa3af200640 1 --2- 172.21.15.28:0/2831012117 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3b0106390 0x7fa3b01a3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:30.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.713+0000 7fa3ae800640 1 -- 172.21.15.28:0/2831012117 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa3b0101990 msgr2=0x7fa3b019f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:30.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.713+0000 7fa3ae800640 1 --2- 172.21.15.28:0/2831012117 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa3b0101990 0x7fa3b019f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.713+0000 7fa3ae800640 1 -- 172.21.15.28:0/2831012117 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3b0106390 msgr2=0x7fa3b01a3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:30.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.713+0000 7fa3ae800640 1 --2- 172.21.15.28:0/2831012117 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3b0106390 0x7fa3b01a3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.713+0000 7fa3ae800640 1 -- 172.21.15.28:0/2831012117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3b01a3eb0 con 0x7fa3b0101070 2024-04-23T14:48:30.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.713+0000 7fa3ade00640 1 --2- 172.21.15.28:0/2831012117 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa3b0101990 0x7fa3b019f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:48:30.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.713+0000 7fa3ae800640 1 --2- 172.21.15.28:0/2831012117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3b0101070 0x7fa3b019eec0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0004af0 tx=0x7fa3a0003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:48:30.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.713+0000 7fa3a7400640 1 -- 172.21.15.28:0/2831012117 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a0047070 con 0x7fa3b0101070 2024-04-23T14:48:30.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.713+0000 7fa3b5400640 1 -- 172.21.15.28:0/2831012117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3b01a4140 con 0x7fa3b0101070 2024-04-23T14:48:30.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.713+0000 7fa3a7400640 1 -- 172.21.15.28:0/2831012117 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3a002fb20 con 0x7fa3b0101070 2024-04-23T14:48:30.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.713+0000 7fa3a7400640 1 -- 172.21.15.28:0/2831012117 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a0033050 con 0x7fa3b0101070 2024-04-23T14:48:30.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.714+0000 7fa3b5400640 1 -- 172.21.15.28:0/2831012117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3b01aba30 con 0x7fa3b0101070 2024-04-23T14:48:30.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.717+0000 7fa3a7400640 1 -- 172.21.15.28:0/2831012117 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fa3a004a070 con 0x7fa3b0101070 2024-04-23T14:48:30.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.717+0000 7fa3b817d740 1 Processor -- start 2024-04-23T14:48:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.718+0000 7fa3b817d740 1 -- start start 2024-04-23T14:48:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.718+0000 7fa3b817d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f71d2e090 0x562f71d2e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.718+0000 7fa3b817d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f71d2e9b0 0x562f71d38b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.718+0000 7fa3b817d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f71d39080 0x562f71d3b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.718+0000 7fa3b817d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562f71c93e70 con 0x562f71d39080 2024-04-23T14:48:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.718+0000 7fa3b817d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562f71c3a380 con 0x562f71d2e090 2024-04-23T14:48:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.718+0000 7fa3b817d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562f71c8fe30 con 0x562f71d2e9b0 2024-04-23T14:48:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.718+0000 7fa3af200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f71d2e090 0x562f71d2e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.718+0000 7fa3ae800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f71d2e9b0 0x562f71d38b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.718+0000 7fa3af200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f71d2e090 0x562f71d2e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:55454/0 (socket says 172.21.15.28:55454) 2024-04-23T14:48:30.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.718+0000 7fa3af200640 1 -- 172.21.15.28:0/2372495406 learned_addr learned my addr 172.21.15.28:0/2372495406 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:48:30.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.719+0000 7fa3ade00640 1 --2- 172.21.15.28:0/2372495406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f71d39080 0x562f71d3b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:30.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.719+0000 7fa3af200640 1 -- 172.21.15.28:0/2372495406 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f71d2e9b0 msgr2=0x562f71d38b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:48:30.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.719+0000 7fa3af200640 1 --2- 172.21.15.28:0/2372495406 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f71d2e9b0 0x562f71d38b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.719+0000 7fa3af200640 1 -- 172.21.15.28:0/2372495406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f71d39080 msgr2=0x562f71d3b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:48:30.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.719+0000 7fa3af200640 1 --2- 172.21.15.28:0/2372495406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f71d39080 0x562f71d3b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.719+0000 7fa3af200640 1 -- 172.21.15.28:0/2372495406 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f71a6e7e0 con 0x562f71d2e090 2024-04-23T14:48:30.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.719+0000 7fa3af200640 1 --2- 172.21.15.28:0/2372495406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f71d2e090 0x562f71d2e470 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8009c20 tx=0x7fa3a8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e7dc0d90a0a20d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:48:30.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.719+0000 7fa3a7400640 1 --2- 172.21.15.28:0/2831012117 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa3940a3da0 0x7fa3940a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:30.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.719+0000 7fa39f400640 1 -- 172.21.15.28:0/2372495406 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a801e070 con 0x562f71d2e090 2024-04-23T14:48:30.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.720+0000 7fa39f400640 1 -- 172.21.15.28:0/2372495406 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3a8019b10 con 0x562f71d2e090 2024-04-23T14:48:30.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.720+0000 7fa39f400640 1 -- 172.21.15.28:0/2372495406 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a800eae0 con 0x562f71d2e090 2024-04-23T14:48:30.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.720+0000 7fa3a7400640 1 -- 172.21.15.28:0/2831012117 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fa3a00f1310 con 0x7fa3b0101070 2024-04-23T14:48:30.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.720+0000 7fa3ae800640 1 --2- 172.21.15.28:0/2831012117 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa3940a3da0 0x7fa3940a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:30.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.721+0000 7fa3b817d740 1 -- 172.21.15.28:0/2372495406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f71d2e090 msgr2=0x562f71d2e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:30.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.721+0000 7fa3ae800640 1 --2- 172.21.15.28:0/2831012117 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa3940a3da0 0x7fa3940a6260 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x562f71d2f650 tx=0x7fa3a00f4dc0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:48:30.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.721+0000 7fa3b817d740 1 --2- 172.21.15.28:0/2372495406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f71d2e090 0x562f71d2e470 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8009c20 tx=0x7fa3a8011810 comp rx=0 tx=0).stop 2024-04-23T14:48:30.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.721+0000 7fa3b817d740 1 -- 172.21.15.28:0/2372495406 shutdown_connections 2024-04-23T14:48:30.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.721+0000 7fa3b817d740 1 --2- 172.21.15.28:0/2372495406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f71d39080 0x562f71d3b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.721+0000 7fa3b817d740 1 --2- 172.21.15.28:0/2372495406 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f71d2e9b0 0x562f71d38b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.721+0000 7fa3b817d740 1 --2- 172.21.15.28:0/2372495406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f71d2e090 0x562f71d2e470 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.721+0000 7fa3b817d740 1 -- 172.21.15.28:0/2372495406 >> 172.21.15.28:0/2372495406 conn(0x562f71d2a3b0 msgr2=0x562f71d2a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:48:30.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.721+0000 7fa3b817d740 1 -- 172.21.15.28:0/2372495406 shutdown_connections 2024-04-23T14:48:30.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.721+0000 7fa3b817d740 1 -- 172.21.15.28:0/2372495406 wait complete. 2024-04-23T14:48:30.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.721+0000 7fa3b817d740 1 -- start start 2024-04-23T14:48:30.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.722+0000 7fa3b817d740 1 Processor -- start 2024-04-23T14:48:30.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.722+0000 7fa3b817d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f71d2e090 0x562f71d36030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:30.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.722+0000 7fa3b817d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f71d2e9b0 0x562f71d36570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:30.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.722+0000 7fa3b817d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f71d39080 0x562f71d36ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:30.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.722+0000 7fa3b817d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562f71d3be70 con 0x562f71d39080 2024-04-23T14:48:30.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.722+0000 7fa3b817d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562f71c4d660 con 0x562f71d2e090 2024-04-23T14:48:30.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.722+0000 7fa3b817d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562f71d3bff0 con 0x562f71d2e9b0 2024-04-23T14:48:30.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.722+0000 7fa3b817d740 10 client.0 _ll_register_callbacks cb 0x562f71c3baa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:48:30.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.722+0000 7fa3af200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f71d2e090 0x562f71d36030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:30.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.722+0000 7fa3af200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f71d2e090 0x562f71d36030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:55468/0 (socket says 172.21.15.28:55468) 2024-04-23T14:48:30.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.723+0000 7fa3af200640 1 -- 172.21.15.28:0/168344807 learned_addr learned my addr 172.21.15.28:0/168344807 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:48:30.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.723+0000 7fa3ade00640 1 --2- 172.21.15.28:0/168344807 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f71d2e9b0 0x562f71d36570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:30.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.723+0000 7fa3af200640 1 --2- 172.21.15.28:0/168344807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f71d39080 0x562f71d36ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:30.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.723+0000 7fa3ade00640 1 -- 172.21.15.28:0/168344807 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f71d2e090 msgr2=0x562f71d36030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:48:30.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.723+0000 7fa3ade00640 1 --2- 172.21.15.28:0/168344807 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f71d2e090 0x562f71d36030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.723+0000 7fa3ade00640 1 -- 172.21.15.28:0/168344807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f71d39080 msgr2=0x562f71d36ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:48:30.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.723+0000 7fa3ade00640 1 --2- 172.21.15.28:0/168344807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f71d39080 0x562f71d36ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.723+0000 7fa3ade00640 1 -- 172.21.15.28:0/168344807 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f71acef70 con 0x562f71d2e9b0 2024-04-23T14:48:30.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.723+0000 7fa3ade00640 1 --2- 172.21.15.28:0/168344807 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f71d2e9b0 0x562f71d36570 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa3980129e0 tx=0x7fa398012ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6c7d41b37507d31 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:48:30.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.723+0000 7fa393e00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:48:30.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.724+0000 7fa393e00640 1 -- 172.21.15.28:0/168344807 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa39800fa10 con 0x562f71d2e9b0 2024-04-23T14:48:30.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.724+0000 7fa393e00640 1 -- 172.21.15.28:0/168344807 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fa39800fbb0 con 0x562f71d2e9b0 2024-04-23T14:48:30.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.724+0000 7fa3b817d740 1 -- 172.21.15.28:0/168344807 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3a8019b10 con 0x562f71d2e9b0 2024-04-23T14:48:30.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.724+0000 7fa393e00640 1 -- 172.21.15.28:0/168344807 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fa39800fdf0 con 0x562f71d2e9b0 2024-04-23T14:48:30.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.724+0000 7fa3b817d740 10 client.25948 fetch_fsmap learned FSMap version 20 2024-04-23T14:48:30.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.724+0000 7fa3b817d740 1 -- 172.21.15.28:0/168344807 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x562f71966170 con 0x562f71d2e9b0 2024-04-23T14:48:30.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.724+0000 7fa393e00640 1 -- 172.21.15.28:0/168344807 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3a8019b10 con 0x562f71d2e9b0 2024-04-23T14:48:30.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.724+0000 7fa393e00640 1 -- 172.21.15.28:0/168344807 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fa398010430 con 0x562f71d2e9b0 2024-04-23T14:48:30.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.725+0000 7fa3b817d740 10 client.25948 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:48:30.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.725+0000 7fa3b817d740 10 client.25948 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:48:30.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.725+0000 7fa3b817d740 20 client.25948 populate_metadata read hostname 'smithi028' 2024-04-23T14:48:30.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.725+0000 7fa3b817d740 1 --2- 172.21.15.28:0/168344807 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562f71d49500 0x562f71d4b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:30.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.725+0000 7fa3b817d740 4 client.25948 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:48:30.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.725+0000 7fa3b817d740 1 -- 172.21.15.28:0/168344807 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562f71ca1890 con 0x562f71d49500 2024-04-23T14:48:30.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.725+0000 7fa3af200640 1 --2- 172.21.15.28:0/168344807 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562f71d49500 0x562f71d4b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:30.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.726+0000 7fa3af200640 1 --2- 172.21.15.28:0/168344807 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562f71d49500 0x562f71d4b8f0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78844b9888eea772 server_cookie=75b6753664be1d6 in_seq=0 out_seq=0 2024-04-23T14:48:30.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.726+0000 7fa393e00640 10 client.25948 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:48:30.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.727+0000 7fa393e00640 1 -- 172.21.15.28:0/168344807 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562f71ca1890 con 0x562f71d49500 2024-04-23T14:48:30.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.727+0000 7fa393e00640 10 client.25948 handle_command_reply: tid=0 2024-04-23T14:48:30.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.727+0000 7fa3b817d740 1 client.25948 shutdown 2024-04-23T14:48:30.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.727+0000 7fa3b817d740 1 -- 172.21.15.28:0/168344807 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f71d2e9b0 msgr2=0x562f71d36570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:30.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.727+0000 7fa3b817d740 1 --2- 172.21.15.28:0/168344807 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f71d2e9b0 0x562f71d36570 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa3980129e0 tx=0x7fa398012ea0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.727+0000 7fa3b817d740 1 -- 172.21.15.28:0/168344807 shutdown_connections 2024-04-23T14:48:30.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.727+0000 7fa3b817d740 1 --2- 172.21.15.28:0/168344807 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562f71d49500 0x562f71d4b8f0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.727+0000 7fa3b817d740 1 --2- 172.21.15.28:0/168344807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f71d39080 0x562f71d36ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.727+0000 7fa3b817d740 1 --2- 172.21.15.28:0/168344807 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f71d2e9b0 0x562f71d36570 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.727+0000 7fa3b817d740 1 --2- 172.21.15.28:0/168344807 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f71d2e090 0x562f71d36030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.727+0000 7fa393e00640 0 client.25948 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:48:30.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.727+0000 7fa3b817d740 1 -- 172.21.15.28:0/168344807 >> 172.21.15.28:0/168344807 conn(0x562f71d2a3b0 msgr2=0x562f71d301b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:48:30.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.728+0000 7fa3b817d740 1 -- 172.21.15.28:0/168344807 shutdown_connections 2024-04-23T14:48:30.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.728+0000 7fa3b817d740 1 -- 172.21.15.28:0/168344807 wait complete. 2024-04-23T14:48:30.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.728+0000 7fa3b817d740 20 client.25948 trim_cache size 0 max 16384 2024-04-23T14:48:30.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.749+0000 7fa3b817d740 1 Processor -- start 2024-04-23T14:48:30.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.749+0000 7fa3b817d740 1 -- start start 2024-04-23T14:48:30.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.749+0000 7fa3b817d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f71d2e090 0x562f71d44d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:30.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.749+0000 7fa3b817d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f71d2e9b0 0x562f71d368d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:30.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.749+0000 7fa3b817d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f71d39080 0x562f71d51200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:30.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.749+0000 7fa3b817d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b01b22c0 con 0x562f71d2e090 2024-04-23T14:48:30.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.749+0000 7fa3b817d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b01b4b40 con 0x562f71d2e9b0 2024-04-23T14:48:30.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.749+0000 7fa3b817d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b01b2930 con 0x562f71d39080 2024-04-23T14:48:30.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.750+0000 7fa3ade00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f71d2e9b0 0x562f71d368d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:30.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.750+0000 7fa3af200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f71d2e090 0x562f71d44d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:30.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.750+0000 7fa3ade00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f71d2e9b0 0x562f71d368d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:55472/0 (socket says 172.21.15.28:55472) 2024-04-23T14:48:30.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.750+0000 7fa3ade00640 1 -- 172.21.15.28:0/1287338617 learned_addr learned my addr 172.21.15.28:0/1287338617 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:48:30.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.750+0000 7fa3af200640 1 --2- 172.21.15.28:0/1287338617 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f71d39080 0x562f71d51200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:30.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.750+0000 7fa3ade00640 1 -- 172.21.15.28:0/1287338617 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f71d39080 msgr2=0x562f71d51200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:48:30.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.750+0000 7fa3ade00640 1 --2- 172.21.15.28:0/1287338617 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f71d39080 0x562f71d51200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.750+0000 7fa3ade00640 1 -- 172.21.15.28:0/1287338617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f71d2e090 msgr2=0x562f71d44d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:48:30.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.750+0000 7fa3ade00640 1 --2- 172.21.15.28:0/1287338617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f71d2e090 0x562f71d44d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.750+0000 7fa3ade00640 1 -- 172.21.15.28:0/1287338617 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f71d31c30 con 0x562f71d2e9b0 2024-04-23T14:48:30.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.750+0000 7fa3af200640 1 --2- 172.21.15.28:0/1287338617 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f71d39080 0x562f71d51200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:48:30.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.750+0000 7fa3ade00640 1 --2- 172.21.15.28:0/1287338617 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f71d2e9b0 0x562f71d368d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa3980108a0 tx=0x7fa398010640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b0754859060230f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:48:30.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.751+0000 7fa39ea00640 1 -- 172.21.15.28:0/1287338617 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa398029070 con 0x562f71d2e9b0 2024-04-23T14:48:30.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.751+0000 7fa39ea00640 1 -- 172.21.15.28:0/1287338617 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3980055c0 con 0x562f71d2e9b0 2024-04-23T14:48:30.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.751+0000 7fa39ea00640 1 -- 172.21.15.28:0/1287338617 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa398010e60 con 0x562f71d2e9b0 2024-04-23T14:48:30.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.752+0000 7fa3b817d740 1 -- 172.21.15.28:0/1287338617 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f71d2e9b0 msgr2=0x562f71d368d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:30.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.752+0000 7fa3b817d740 1 --2- 172.21.15.28:0/1287338617 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f71d2e9b0 0x562f71d368d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa3980108a0 tx=0x7fa398010640 comp rx=0 tx=0).stop 2024-04-23T14:48:30.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.752+0000 7fa3b817d740 1 -- 172.21.15.28:0/1287338617 shutdown_connections 2024-04-23T14:48:30.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.752+0000 7fa3b817d740 1 --2- 172.21.15.28:0/1287338617 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f71d2e9b0 0x562f71d368d0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.752+0000 7fa3b817d740 1 --2- 172.21.15.28:0/1287338617 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f71d39080 0x562f71d51200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.752+0000 7fa3b817d740 1 --2- 172.21.15.28:0/1287338617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f71d2e090 0x562f71d44d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.752+0000 7fa3b817d740 1 -- 172.21.15.28:0/1287338617 >> 172.21.15.28:0/1287338617 conn(0x562f71d2a3b0 msgr2=0x562f71d29510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:48:30.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.752+0000 7fa3b817d740 1 -- 172.21.15.28:0/1287338617 shutdown_connections 2024-04-23T14:48:30.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.752+0000 7fa3b817d740 1 -- 172.21.15.28:0/1287338617 wait complete. 2024-04-23T14:48:30.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.752+0000 7fa3b817d740 1 -- start start 2024-04-23T14:48:30.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.753+0000 7fa3b817d740 1 Processor -- start 2024-04-23T14:48:30.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.753+0000 7fa3b817d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f71d2e090 0x562f71d32e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:30.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.753+0000 7fa3b817d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f71d2e9b0 0x562f71d33390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:30.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.753+0000 7fa3b817d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f71d39080 0x562f71d338d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:30.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.753+0000 7fa3b817d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b01b3000 con 0x562f71d2e9b0 2024-04-23T14:48:30.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.753+0000 7fa3b817d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b01b4470 con 0x562f71d39080 2024-04-23T14:48:30.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.753+0000 7fa3b817d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b01b36d0 con 0x562f71d2e090 2024-04-23T14:48:30.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.753+0000 7fa3b817d740 10 client.0 _ll_register_callbacks cb 0x7fa398009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:48:30.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.753+0000 7fa3ade00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f71d2e9b0 0x562f71d33390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:30.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.753+0000 7fa3af200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f71d2e090 0x562f71d32e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:30.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.753+0000 7fa3ade00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f71d2e9b0 0x562f71d33390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59928/0 (socket says 172.21.15.28:59928) 2024-04-23T14:48:30.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.753+0000 7fa3ade00640 1 -- 172.21.15.28:0/1432626068 learned_addr learned my addr 172.21.15.28:0/1432626068 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:48:30.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.754+0000 7fa3af200640 1 --2- 172.21.15.28:0/1432626068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f71d39080 0x562f71d338d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:30.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.754+0000 7fa3ade00640 1 -- 172.21.15.28:0/1432626068 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f71d2e090 msgr2=0x562f71d32e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:48:30.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.754+0000 7fa3ade00640 1 --2- 172.21.15.28:0/1432626068 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f71d2e090 0x562f71d32e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.754+0000 7fa3ade00640 1 -- 172.21.15.28:0/1432626068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f71d39080 msgr2=0x562f71d338d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:48:30.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.754+0000 7fa3ade00640 1 --2- 172.21.15.28:0/1432626068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f71d39080 0x562f71d338d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.754+0000 7fa3ade00640 1 -- 172.21.15.28:0/1432626068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f71daebd0 con 0x562f71d2e9b0 2024-04-23T14:48:30.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.754+0000 7fa3ade00640 1 --2- 172.21.15.28:0/1432626068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f71d2e9b0 0x562f71d33390 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa398010990 tx=0x7fa398005da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cd0e744fabcf138 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:48:30.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.754+0000 7fa393e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-23T14:48:30.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.754+0000 7fa393e00640 1 -- 172.21.15.28:0/1432626068 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa398031070 con 0x562f71d2e9b0 2024-04-23T14:48:30.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.755+0000 7fa393e00640 1 -- 172.21.15.28:0/1432626068 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fa398005390 con 0x562f71d2e9b0 2024-04-23T14:48:30.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.755+0000 7fa3b817d740 1 -- 172.21.15.28:0/1432626068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3980055c0 con 0x562f71d2e9b0 2024-04-23T14:48:30.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.755+0000 7fa393e00640 1 -- 172.21.15.28:0/1432626068 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fa39803c050 con 0x562f71d2e9b0 2024-04-23T14:48:30.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.755+0000 7fa3b817d740 10 client.15813 fetch_fsmap learned FSMap version 20 2024-04-23T14:48:30.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.755+0000 7fa3b817d740 1 -- 172.21.15.28:0/1432626068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x562f71dafe00 con 0x562f71d2e9b0 2024-04-23T14:48:30.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.755+0000 7fa393e00640 1 -- 172.21.15.28:0/1432626068 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3980055c0 con 0x562f71d2e9b0 2024-04-23T14:48:30.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.755+0000 7fa393e00640 1 -- 172.21.15.28:0/1432626068 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fa39801c420 con 0x562f71d2e9b0 2024-04-23T14:48:30.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.756+0000 7fa3b817d740 10 client.15813 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:48:30.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.756+0000 7fa3b817d740 10 client.15813 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:48:30.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.756+0000 7fa3b817d740 20 client.15813 populate_metadata read hostname 'smithi028' 2024-04-23T14:48:30.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.756+0000 7fa3b817d740 1 --2- 172.21.15.28:0/1432626068 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562f71db0470 0x562f71db2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:30.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.756+0000 7fa3b817d740 4 client.15813 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-23T14:48:30.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.756+0000 7fa3b817d740 1 -- 172.21.15.28:0/1432626068 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x562f71c9b700 con 0x562f71db0470 2024-04-23T14:48:30.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.756+0000 7fa3af200640 1 --2- 172.21.15.28:0/1432626068 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562f71db0470 0x562f71db2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:30.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.756+0000 7fa3af200640 1 --2- 172.21.15.28:0/1432626068 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562f71db0470 0x562f71db2890 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=457ce2007dc0e385 server_cookie=882224f8d2ce3dc in_seq=0 out_seq=0 2024-04-23T14:48:30.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.756+0000 7fa393e00640 10 client.15813 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:48:30.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.758+0000 7fa393e00640 1 -- 172.21.15.28:0/1432626068 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x562f71c9b700 con 0x562f71db0470 2024-04-23T14:48:30.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.758+0000 7fa393e00640 10 client.15813 handle_command_reply: tid=0 2024-04-23T14:48:30.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.758+0000 7fa3b817d740 1 client.15813 shutdown 2024-04-23T14:48:30.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.758+0000 7fa3b817d740 1 -- 172.21.15.28:0/1432626068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f71d2e9b0 msgr2=0x562f71d33390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:30.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.758+0000 7fa3b817d740 1 --2- 172.21.15.28:0/1432626068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f71d2e9b0 0x562f71d33390 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa398010990 tx=0x7fa398005da0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.758+0000 7fa3b817d740 1 -- 172.21.15.28:0/1432626068 shutdown_connections 2024-04-23T14:48:30.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.758+0000 7fa3b817d740 1 --2- 172.21.15.28:0/1432626068 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562f71db0470 0x562f71db2890 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.758+0000 7fa3b817d740 1 --2- 172.21.15.28:0/1432626068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f71d39080 0x562f71d338d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.758+0000 7fa3b817d740 1 --2- 172.21.15.28:0/1432626068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f71d2e9b0 0x562f71d33390 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.758+0000 7fa3b817d740 1 --2- 172.21.15.28:0/1432626068 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f71d2e090 0x562f71d32e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.758+0000 7fa393e00640 0 client.15813 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:48:30.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.758+0000 7fa3b817d740 1 -- 172.21.15.28:0/1432626068 >> 172.21.15.28:0/1432626068 conn(0x562f71d2a3b0 msgr2=0x562f71d28c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:48:30.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.758+0000 7fa3b817d740 1 -- 172.21.15.28:0/1432626068 shutdown_connections 2024-04-23T14:48:30.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.758+0000 7fa3b817d740 1 -- 172.21.15.28:0/1432626068 wait complete. 2024-04-23T14:48:30.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.758+0000 7fa3b817d740 20 client.15813 trim_cache size 0 max 16384 2024-04-23T14:48:30.762 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:48:30.762 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-23T14:48:30.763 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "2c10311a-1b77-4209-817b-2d6d8a633f09", 2024-04-23T14:48:30.763 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-23T14:48:30.763 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:48:30.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.759+0000 7fa3b5400640 1 -- 172.21.15.28:0/2831012117 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa3940a3da0 msgr2=0x7fa3940a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:30.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.759+0000 7fa3b5400640 1 --2- 172.21.15.28:0/2831012117 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa3940a3da0 0x7fa3940a6260 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x562f71d2f650 tx=0x7fa3a00f4dc0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.759+0000 7fa3b5400640 1 -- 172.21.15.28:0/2831012117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3b0101070 msgr2=0x7fa3b019eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:30.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.759+0000 7fa3b5400640 1 --2- 172.21.15.28:0/2831012117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3b0101070 0x7fa3b019eec0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0004af0 tx=0x7fa3a0003ba0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.759+0000 7fa3b5400640 1 -- 172.21.15.28:0/2831012117 shutdown_connections 2024-04-23T14:48:30.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.759+0000 7fa3b5400640 1 --2- 172.21.15.28:0/2831012117 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa3940a3da0 0x7fa3940a6260 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.759+0000 7fa3b5400640 1 --2- 172.21.15.28:0/2831012117 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3b0106390 0x7fa3b01a3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.759+0000 7fa3b5400640 1 --2- 172.21.15.28:0/2831012117 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa3b0101990 0x7fa3b019f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.759+0000 7fa3b5400640 1 --2- 172.21.15.28:0/2831012117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3b0101070 0x7fa3b019eec0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:30.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.759+0000 7fa3b5400640 1 -- 172.21.15.28:0/2831012117 >> 172.21.15.28:0/2831012117 conn(0x7fa3b0078000 msgr2=0x7fa3b0103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:48:30.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.759+0000 7fa3b5400640 1 -- 172.21.15.28:0/2831012117 shutdown_connections 2024-04-23T14:48:30.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:30.759+0000 7fa3b5400640 1 -- 172.21.15.28:0/2831012117 wait complete. 2024-04-23T14:48:30.779 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:48:30.779 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:48:31.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.039+0000 7f96be200640 1 -- 172.21.15.28:0/1323977684 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96b8106210 msgr2=0x7f96b810dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:31.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.039+0000 7f96be200640 1 --2- 172.21.15.28:0/1323977684 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96b8106210 0x7f96b810dde0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f96b0009be0 tx=0x7f96b002a6c0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.040+0000 7f96be200640 1 -- 172.21.15.28:0/1323977684 shutdown_connections 2024-04-23T14:48:31.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.040+0000 7f96be200640 1 --2- 172.21.15.28:0/1323977684 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96b8106210 0x7f96b810dde0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.040+0000 7f96be200640 1 --2- 172.21.15.28:0/1323977684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96b8101af0 0x7f96b8105b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.040+0000 7f96be200640 1 --2- 172.21.15.28:0/1323977684 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f96b8101140 0x7f96b8101520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.040+0000 7f96be200640 1 -- 172.21.15.28:0/1323977684 >> 172.21.15.28:0/1323977684 conn(0x7f96b8078000 msgr2=0x7f96b8078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:48:31.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.040+0000 7f96be200640 1 -- 172.21.15.28:0/1323977684 shutdown_connections 2024-04-23T14:48:31.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.040+0000 7f96be200640 1 -- 172.21.15.28:0/1323977684 wait complete. 2024-04-23T14:48:31.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.041+0000 7f96be200640 1 Processor -- start 2024-04-23T14:48:31.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.041+0000 7f96be200640 1 -- start start 2024-04-23T14:48:31.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.041+0000 7f96be200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96b8101140 0x7f96b8196660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:31.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.041+0000 7f96be200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96b8106210 0x7f96b8196ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:31.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.041+0000 7f96be200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f96b819af20 0x7f96b819b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:31.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.041+0000 7f96be200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f96b81101c0 con 0x7f96b8106210 2024-04-23T14:48:31.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.041+0000 7f96be200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f96b8110040 con 0x7f96b8101140 2024-04-23T14:48:31.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.041+0000 7f96be200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f96b8110340 con 0x7f96b819af20 2024-04-23T14:48:31.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.042+0000 7f96b7400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96b8101140 0x7f96b8196660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:31.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.042+0000 7f96b7e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f96b819af20 0x7f96b819b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:31.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.042+0000 7f96b7400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96b8101140 0x7f96b8196660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:55502/0 (socket says 172.21.15.28:55502) 2024-04-23T14:48:31.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.042+0000 7f96b7400640 1 -- 172.21.15.28:0/1545817322 learned_addr learned my addr 172.21.15.28:0/1545817322 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:48:31.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.042+0000 7f96b6a00640 1 --2- 172.21.15.28:0/1545817322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96b8106210 0x7f96b8196ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:31.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.042+0000 7f96b7e00640 1 -- 172.21.15.28:0/1545817322 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96b8101140 msgr2=0x7f96b8196660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:31.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.042+0000 7f96b7e00640 1 --2- 172.21.15.28:0/1545817322 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96b8101140 0x7f96b8196660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.042+0000 7f96b7e00640 1 -- 172.21.15.28:0/1545817322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96b8106210 msgr2=0x7f96b8196ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:31.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.042+0000 7f96b7e00640 1 --2- 172.21.15.28:0/1545817322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96b8106210 0x7f96b8196ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.042+0000 7f96b7e00640 1 -- 172.21.15.28:0/1545817322 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96b819ba10 con 0x7f96b819af20 2024-04-23T14:48:31.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.042+0000 7f96b6a00640 1 --2- 172.21.15.28:0/1545817322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96b8106210 0x7f96b8196ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:48:31.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.042+0000 7f96b7400640 1 --2- 172.21.15.28:0/1545817322 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96b8101140 0x7f96b8196660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:48:31.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.042+0000 7f96b7e00640 1 --2- 172.21.15.28:0/1545817322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f96b819af20 0x7f96b819b3d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f96b0009b30 tx=0x7f96b00305d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:48:31.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.043+0000 7f96afe00640 1 -- 172.21.15.28:0/1545817322 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b0041020 con 0x7f96b819af20 2024-04-23T14:48:31.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.043+0000 7f96be200640 1 -- 172.21.15.28:0/1545817322 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96b819bca0 con 0x7f96b819af20 2024-04-23T14:48:31.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.043+0000 7f96afe00640 1 -- 172.21.15.28:0/1545817322 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96b0003e50 con 0x7f96b819af20 2024-04-23T14:48:31.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.043+0000 7f96afe00640 1 -- 172.21.15.28:0/1545817322 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b0036040 con 0x7f96b819af20 2024-04-23T14:48:31.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.043+0000 7f96be200640 1 -- 172.21.15.28:0/1545817322 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96b81a3460 con 0x7f96b819af20 2024-04-23T14:48:31.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.047+0000 7f96c0f7d740 1 Processor -- start 2024-04-23T14:48:31.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.048+0000 7f96afe00640 1 -- 172.21.15.28:0/1545817322 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f96b0044020 con 0x7f96b819af20 2024-04-23T14:48:31.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.048+0000 7f96c0f7d740 1 -- start start 2024-04-23T14:48:31.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.048+0000 7f96c0f7d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a1ff895860 0x55a1ff895c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:31.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.048+0000 7f96c0f7d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a1ff896180 0x55a1ff8a0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:31.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.048+0000 7f96c0f7d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1ff8a0850 0x55a1ff8a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:31.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.048+0000 7f96c0f7d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a1ff7fb1f0 con 0x55a1ff8a0850 2024-04-23T14:48:31.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.048+0000 7f96c0f7d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a1ff7a1350 con 0x55a1ff895860 2024-04-23T14:48:31.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.048+0000 7f96c0f7d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a1ff7f71b0 con 0x55a1ff896180 2024-04-23T14:48:31.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.049+0000 7f96b7400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a1ff896180 0x55a1ff8a0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:31.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.049+0000 7f96b7400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a1ff896180 0x55a1ff8a0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:34628/0 (socket says 172.21.15.28:34628) 2024-04-23T14:48:31.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.049+0000 7f96b6a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a1ff895860 0x55a1ff895c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:31.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.049+0000 7f96b7400640 1 -- 172.21.15.28:0/2966643719 learned_addr learned my addr 172.21.15.28:0/2966643719 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:48:31.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.049+0000 7f96b6a00640 1 --2- 172.21.15.28:0/2966643719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1ff8a0850 0x55a1ff8a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:31.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.049+0000 7f96b7400640 1 -- 172.21.15.28:0/2966643719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a1ff895860 msgr2=0x55a1ff895c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:48:31.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.049+0000 7f96b7400640 1 --2- 172.21.15.28:0/2966643719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a1ff895860 0x55a1ff895c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.049+0000 7f96b7400640 1 -- 172.21.15.28:0/2966643719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1ff8a0850 msgr2=0x55a1ff8a2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:48:31.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.049+0000 7f96b7400640 1 --2- 172.21.15.28:0/2966643719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1ff8a0850 0x55a1ff8a2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.049+0000 7f96b7400640 1 -- 172.21.15.28:0/2966643719 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1ff7f5f50 con 0x55a1ff896180 2024-04-23T14:48:31.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.049+0000 7f96b7400640 1 --2- 172.21.15.28:0/2966643719 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a1ff896180 0x55a1ff8a0310 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f96a800d3a0 tx=0x7f96a8005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7eb9871a91c5a90a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:48:31.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.050+0000 7f96a3e00640 1 -- 172.21.15.28:0/2966643719 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a8012900 con 0x55a1ff896180 2024-04-23T14:48:31.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.050+0000 7f96afe00640 1 --2- 172.21.15.28:0/1545817322 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f969c0a3da0 0x7f969c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:31.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.050+0000 7f96a3e00640 1 -- 172.21.15.28:0/2966643719 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96a8020070 con 0x55a1ff896180 2024-04-23T14:48:31.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.050+0000 7f96a3e00640 1 -- 172.21.15.28:0/2966643719 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a8019450 con 0x55a1ff896180 2024-04-23T14:48:31.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.050+0000 7f96afe00640 1 -- 172.21.15.28:0/1545817322 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f96b00f1060 con 0x7f96b819af20 2024-04-23T14:48:31.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.050+0000 7f96b6a00640 1 --2- 172.21.15.28:0/1545817322 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f969c0a3da0 0x7f969c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:31.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.051+0000 7f96c0f7d740 1 -- 172.21.15.28:0/2966643719 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a1ff896180 msgr2=0x55a1ff8a0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:31.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.051+0000 7f96c0f7d740 1 --2- 172.21.15.28:0/2966643719 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a1ff896180 0x55a1ff8a0310 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f96a800d3a0 tx=0x7f96a8005f90 comp rx=0 tx=0).stop 2024-04-23T14:48:31.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.051+0000 7f96c0f7d740 1 -- 172.21.15.28:0/2966643719 shutdown_connections 2024-04-23T14:48:31.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.051+0000 7f96b6a00640 1 --2- 172.21.15.28:0/1545817322 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f969c0a3da0 0x7f969c0a6260 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x55a1ff892d10 tx=0x7f96a400d040 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:48:31.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.051+0000 7f96c0f7d740 1 --2- 172.21.15.28:0/2966643719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1ff8a0850 0x55a1ff8a2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.051+0000 7f96c0f7d740 1 --2- 172.21.15.28:0/2966643719 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a1ff896180 0x55a1ff8a0310 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.051+0000 7f96c0f7d740 1 --2- 172.21.15.28:0/2966643719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a1ff895860 0x55a1ff895c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.051+0000 7f96c0f7d740 1 -- 172.21.15.28:0/2966643719 >> 172.21.15.28:0/2966643719 conn(0x55a1ff891b80 msgr2=0x55a1ff891f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:48:31.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.051+0000 7f96c0f7d740 1 -- 172.21.15.28:0/2966643719 shutdown_connections 2024-04-23T14:48:31.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.051+0000 7f96c0f7d740 1 -- 172.21.15.28:0/2966643719 wait complete. 2024-04-23T14:48:31.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.051+0000 7f96c0f7d740 1 -- start start 2024-04-23T14:48:31.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.052+0000 7f96c0f7d740 1 Processor -- start 2024-04-23T14:48:31.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.052+0000 7f96c0f7d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a1ff895860 0x55a1ff89d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:31.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.053+0000 7f96c0f7d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1ff896180 0x55a1ff89dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:31.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.053+0000 7f96c0f7d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a1ff8a0850 0x55a1ff89e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:31.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.053+0000 7f96c0f7d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a1ff8a3640 con 0x55a1ff896180 2024-04-23T14:48:31.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.053+0000 7f96c0f7d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a1ff7b4690 con 0x55a1ff895860 2024-04-23T14:48:31.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.053+0000 7f96c0f7d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a1ff8a37c0 con 0x55a1ff8a0850 2024-04-23T14:48:31.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.053+0000 7f96c0f7d740 10 client.0 _ll_register_callbacks cb 0x55a1ff7f0650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:48:31.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.053+0000 7f96b7400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a1ff895860 0x55a1ff89d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:31.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.053+0000 7f96b7400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a1ff895860 0x55a1ff89d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:55512/0 (socket says 172.21.15.28:55512) 2024-04-23T14:48:31.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.053+0000 7f96b7400640 1 -- 172.21.15.28:0/3959163919 learned_addr learned my addr 172.21.15.28:0/3959163919 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:48:31.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.053+0000 7f96b6a00640 1 --2- 172.21.15.28:0/3959163919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1ff896180 0x55a1ff89dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:31.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.053+0000 7f96b7e00640 1 --2- 172.21.15.28:0/3959163919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a1ff8a0850 0x55a1ff89e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:31.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.053+0000 7f96b7400640 1 -- 172.21.15.28:0/3959163919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a1ff8a0850 msgr2=0x55a1ff89e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:48:31.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.053+0000 7f96b7400640 1 --2- 172.21.15.28:0/3959163919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a1ff8a0850 0x55a1ff89e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.053+0000 7f96b7400640 1 -- 172.21.15.28:0/3959163919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1ff896180 msgr2=0x55a1ff89dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:48:31.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.053+0000 7f96b7400640 1 --2- 172.21.15.28:0/3959163919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1ff896180 0x55a1ff89dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.053+0000 7f96b7400640 1 -- 172.21.15.28:0/3959163919 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1ff7b3920 con 0x55a1ff895860 2024-04-23T14:48:31.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.053+0000 7f96b7e00640 1 --2- 172.21.15.28:0/3959163919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a1ff8a0850 0x55a1ff89e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:48:31.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.054+0000 7f96b7400640 1 --2- 172.21.15.28:0/3959163919 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a1ff895860 0x55a1ff89d720 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f96a800f010 tx=0x7f96a801b7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe710cdf6647ee3c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:48:31.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.054+0000 7f96a0c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:48:31.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.054+0000 7f96a0c00640 1 -- 172.21.15.28:0/3959163919 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a8028030 con 0x55a1ff895860 2024-04-23T14:48:31.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.054+0000 7f96a0c00640 1 -- 172.21.15.28:0/3959163919 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f96a80046c0 con 0x55a1ff895860 2024-04-23T14:48:31.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.054+0000 7f96c0f7d740 1 -- 172.21.15.28:0/3959163919 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96a8020070 con 0x55a1ff895860 2024-04-23T14:48:31.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.054+0000 7f96a0c00640 1 -- 172.21.15.28:0/3959163919 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f96a8005600 con 0x55a1ff895860 2024-04-23T14:48:31.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.054+0000 7f96c0f7d740 10 client.25979 fetch_fsmap learned FSMap version 20 2024-04-23T14:48:31.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.054+0000 7f96c0f7d740 1 -- 172.21.15.28:0/3959163919 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55a1ff5d77d0 con 0x55a1ff895860 2024-04-23T14:48:31.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.055+0000 7f96a0c00640 1 -- 172.21.15.28:0/3959163919 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96a8020070 con 0x55a1ff895860 2024-04-23T14:48:31.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.055+0000 7f96a0c00640 1 -- 172.21.15.28:0/3959163919 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f96a8012ad0 con 0x55a1ff895860 2024-04-23T14:48:31.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.055+0000 7f96c0f7d740 10 client.25979 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:48:31.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.055+0000 7f96c0f7d740 10 client.25979 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:48:31.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.055+0000 7f96c0f7d740 20 client.25979 populate_metadata read hostname 'smithi028' 2024-04-23T14:48:31.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.055+0000 7f96c0f7d740 1 --2- 172.21.15.28:0/3959163919 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55a1ff8b0a70 0x55a1ff8b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:31.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.055+0000 7f96c0f7d740 4 client.25979 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:48:31.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.055+0000 7f96c0f7d740 1 -- 172.21.15.28:0/3959163919 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a1ff808c10 con 0x55a1ff8b0a70 2024-04-23T14:48:31.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.055+0000 7f96b6a00640 1 --2- 172.21.15.28:0/3959163919 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55a1ff8b0a70 0x55a1ff8b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:31.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.056+0000 7f96a0c00640 10 client.25979 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:48:31.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.056+0000 7f96b6a00640 1 --2- 172.21.15.28:0/3959163919 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55a1ff8b0a70 0x55a1ff8b2e60 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8bcc3e8fed3a709 server_cookie=f2faa50efeee2eb5 in_seq=0 out_seq=0 2024-04-23T14:48:31.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.057+0000 7f96a0c00640 1 -- 172.21.15.28:0/3959163919 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a1ff808c10 con 0x55a1ff8b0a70 2024-04-23T14:48:31.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.057+0000 7f96a0c00640 10 client.25979 handle_command_reply: tid=0 2024-04-23T14:48:31.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.057+0000 7f96c0f7d740 1 client.25979 shutdown 2024-04-23T14:48:31.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.057+0000 7f96c0f7d740 1 -- 172.21.15.28:0/3959163919 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a1ff895860 msgr2=0x55a1ff89d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:31.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.057+0000 7f96c0f7d740 1 --2- 172.21.15.28:0/3959163919 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a1ff895860 0x55a1ff89d720 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f96a800f010 tx=0x7f96a801b7d0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.057+0000 7f96c0f7d740 1 -- 172.21.15.28:0/3959163919 shutdown_connections 2024-04-23T14:48:31.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.057+0000 7f96c0f7d740 1 --2- 172.21.15.28:0/3959163919 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55a1ff8b0a70 0x55a1ff8b2e60 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.057+0000 7f96c0f7d740 1 --2- 172.21.15.28:0/3959163919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a1ff8a0850 0x55a1ff89e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.057+0000 7f96c0f7d740 1 --2- 172.21.15.28:0/3959163919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1ff896180 0x55a1ff89dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.057+0000 7f96c0f7d740 1 --2- 172.21.15.28:0/3959163919 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a1ff895860 0x55a1ff89d720 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.057+0000 7f96a0c00640 0 client.25979 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:48:31.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.057+0000 7f96c0f7d740 1 -- 172.21.15.28:0/3959163919 >> 172.21.15.28:0/3959163919 conn(0x55a1ff891b80 msgr2=0x55a1ff897980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:48:31.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.058+0000 7f96c0f7d740 1 -- 172.21.15.28:0/3959163919 shutdown_connections 2024-04-23T14:48:31.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.058+0000 7f96c0f7d740 1 -- 172.21.15.28:0/3959163919 wait complete. 2024-04-23T14:48:31.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.058+0000 7f96c0f7d740 20 client.25979 trim_cache size 0 max 16384 2024-04-23T14:48:31.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.079+0000 7f96c0f7d740 1 Processor -- start 2024-04-23T14:48:31.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.079+0000 7f96c0f7d740 1 -- start start 2024-04-23T14:48:31.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.079+0000 7f96c0f7d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a1ff895860 0x55a1ff8ac530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:31.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.079+0000 7f96c0f7d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a1ff896180 0x55a1ff898000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:31.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.079+0000 7f96c0f7d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1ff8a0850 0x55a1ff890aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:31.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.079+0000 7f96c0f7d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f96b81a9a20 con 0x55a1ff8a0850 2024-04-23T14:48:31.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.079+0000 7f96c0f7d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f96b81ac280 con 0x55a1ff896180 2024-04-23T14:48:31.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.079+0000 7f96c0f7d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f96b81aa070 con 0x55a1ff895860 2024-04-23T14:48:31.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.080+0000 7f96b6a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a1ff896180 0x55a1ff898000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:31.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.080+0000 7f96b7400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a1ff895860 0x55a1ff8ac530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:31.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.080+0000 7f96b6a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a1ff896180 0x55a1ff898000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:55514/0 (socket says 172.21.15.28:55514) 2024-04-23T14:48:31.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.080+0000 7f96b6a00640 1 -- 172.21.15.28:0/2802177554 learned_addr learned my addr 172.21.15.28:0/2802177554 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:48:31.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.080+0000 7f96b7e00640 1 --2- 172.21.15.28:0/2802177554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1ff8a0850 0x55a1ff890aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:31.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.080+0000 7f96b6a00640 1 -- 172.21.15.28:0/2802177554 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a1ff895860 msgr2=0x55a1ff8ac530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:48:31.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.080+0000 7f96b6a00640 1 --2- 172.21.15.28:0/2802177554 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a1ff895860 0x55a1ff8ac530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.080+0000 7f96b6a00640 1 -- 172.21.15.28:0/2802177554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1ff8a0850 msgr2=0x55a1ff890aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:48:31.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.080+0000 7f96b6a00640 1 --2- 172.21.15.28:0/2802177554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1ff8a0850 0x55a1ff890aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.080+0000 7f96b6a00640 1 -- 172.21.15.28:0/2802177554 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1ff50d3f0 con 0x55a1ff896180 2024-04-23T14:48:31.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.080+0000 7f96b7400640 1 --2- 172.21.15.28:0/2802177554 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a1ff895860 0x55a1ff8ac530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:48:31.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.080+0000 7f96b7e00640 1 --2- 172.21.15.28:0/2802177554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1ff8a0850 0x55a1ff890aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:48:31.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.080+0000 7f96b6a00640 1 --2- 172.21.15.28:0/2802177554 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a1ff896180 0x55a1ff898000 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f96b806b790 tx=0x7f96a40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e52129c0340fee9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:48:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.081+0000 7f96a3400640 1 -- 172.21.15.28:0/2802177554 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a402e2d0 con 0x55a1ff896180 2024-04-23T14:48:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.081+0000 7f96a3400640 1 -- 172.21.15.28:0/2802177554 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96a4034070 con 0x55a1ff896180 2024-04-23T14:48:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.081+0000 7f96a3400640 1 -- 172.21.15.28:0/2802177554 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a4027450 con 0x55a1ff896180 2024-04-23T14:48:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.081+0000 7f96c0f7d740 1 -- 172.21.15.28:0/2802177554 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a1ff896180 msgr2=0x55a1ff898000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.081+0000 7f96c0f7d740 1 --2- 172.21.15.28:0/2802177554 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a1ff896180 0x55a1ff898000 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f96b806b790 tx=0x7f96a40046c0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.082+0000 7f96c0f7d740 1 -- 172.21.15.28:0/2802177554 shutdown_connections 2024-04-23T14:48:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.082+0000 7f96c0f7d740 1 --2- 172.21.15.28:0/2802177554 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a1ff896180 0x55a1ff898000 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.082+0000 7f96c0f7d740 1 --2- 172.21.15.28:0/2802177554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1ff8a0850 0x55a1ff890aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.082+0000 7f96c0f7d740 1 --2- 172.21.15.28:0/2802177554 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a1ff895860 0x55a1ff8ac530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.082+0000 7f96c0f7d740 1 -- 172.21.15.28:0/2802177554 >> 172.21.15.28:0/2802177554 conn(0x55a1ff891b80 msgr2=0x55a1ff890390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:48:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.082+0000 7f96c0f7d740 1 -- 172.21.15.28:0/2802177554 shutdown_connections 2024-04-23T14:48:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.082+0000 7f96c0f7d740 1 -- 172.21.15.28:0/2802177554 wait complete. 2024-04-23T14:48:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.082+0000 7f96c0f7d740 1 -- start start 2024-04-23T14:48:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.082+0000 7f96c0f7d740 1 Processor -- start 2024-04-23T14:48:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.083+0000 7f96c0f7d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1ff895860 0x55a1ff8a8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.083+0000 7f96c0f7d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a1ff896180 0x55a1ff8a8f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.083+0000 7f96c0f7d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a1ff8a0850 0x55a1ff899d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.083+0000 7f96c0f7d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f96b81aa740 con 0x55a1ff895860 2024-04-23T14:48:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.083+0000 7f96c0f7d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f96b81abbb0 con 0x55a1ff8a0850 2024-04-23T14:48:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.083+0000 7f96c0f7d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f96b81aae10 con 0x55a1ff896180 2024-04-23T14:48:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.083+0000 7f96c0f7d740 10 client.0 _ll_register_callbacks cb 0x7f96a400b920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:48:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.083+0000 7f96b7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a1ff8a0850 0x55a1ff899d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.083+0000 7f96b7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1ff895860 0x55a1ff8a8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.083+0000 7f96b6a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a1ff896180 0x55a1ff8a8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.083+0000 7f96b7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1ff895860 0x55a1ff8a8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59988/0 (socket says 172.21.15.28:59988) 2024-04-23T14:48:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.083+0000 7f96b7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a1ff8a0850 0x55a1ff899d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:55518/0 (socket says 172.21.15.28:55518) 2024-04-23T14:48:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.083+0000 7f96b7400640 1 -- 172.21.15.28:0/2060264912 learned_addr learned my addr 172.21.15.28:0/2060264912 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:48:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.084+0000 7f96b7e00640 1 -- 172.21.15.28:0/2060264912 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a1ff896180 msgr2=0x55a1ff8a8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:48:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.084+0000 7f96b7e00640 1 --2- 172.21.15.28:0/2060264912 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a1ff896180 0x55a1ff8a8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.084+0000 7f96b7e00640 1 -- 172.21.15.28:0/2060264912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1ff895860 msgr2=0x55a1ff8a8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:48:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.084+0000 7f96b7e00640 1 --2- 172.21.15.28:0/2060264912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1ff895860 0x55a1ff8a8a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.084+0000 7f96b7e00640 1 -- 172.21.15.28:0/2060264912 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1ff4cd0f0 con 0x55a1ff8a0850 2024-04-23T14:48:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.084+0000 7f96b7e00640 1 --2- 172.21.15.28:0/2060264912 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a1ff8a0850 0x55a1ff899d80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f96b00ee760 tx=0x7f96b00bcbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=968dd80f1a92874f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:48:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.084+0000 7f96a0c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:48:31.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.084+0000 7f96a0c00640 1 -- 172.21.15.28:0/2060264912 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b00dd040 con 0x55a1ff8a0850 2024-04-23T14:48:31.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.084+0000 7f96a0c00640 1 -- 172.21.15.28:0/2060264912 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f96b00315d0 con 0x55a1ff8a0850 2024-04-23T14:48:31.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.084+0000 7f96c0f7d740 1 -- 172.21.15.28:0/2060264912 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96a4034070 con 0x55a1ff8a0850 2024-04-23T14:48:31.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.085+0000 7f96a0c00640 1 -- 172.21.15.28:0/2060264912 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f96b00c1360 con 0x55a1ff8a0850 2024-04-23T14:48:31.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.085+0000 7f96c0f7d740 10 client.25985 fetch_fsmap learned FSMap version 20 2024-04-23T14:48:31.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.085+0000 7f96c0f7d740 1 -- 172.21.15.28:0/2060264912 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55a1ff7c0810 con 0x55a1ff8a0850 2024-04-23T14:48:31.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.085+0000 7f96a0c00640 1 -- 172.21.15.28:0/2060264912 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96a4034070 con 0x55a1ff8a0850 2024-04-23T14:48:31.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.085+0000 7f96a0c00640 1 -- 172.21.15.28:0/2060264912 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f96b00d6260 con 0x55a1ff8a0850 2024-04-23T14:48:31.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.085+0000 7f96c0f7d740 10 client.25985 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:48:31.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.085+0000 7f96c0f7d740 10 client.25985 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:48:31.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.085+0000 7f96c0f7d740 20 client.25985 populate_metadata read hostname 'smithi028' 2024-04-23T14:48:31.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.085+0000 7f96c0f7d740 1 --2- 172.21.15.28:0/2060264912 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55a1ff91f410 0x55a1ff921830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:48:31.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.085+0000 7f96c0f7d740 4 client.25985 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:48:31.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.085+0000 7f96c0f7d740 1 -- 172.21.15.28:0/2060264912 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a1ff802a80 con 0x55a1ff91f410 2024-04-23T14:48:31.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.085+0000 7f96b7400640 1 --2- 172.21.15.28:0/2060264912 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55a1ff91f410 0x55a1ff921830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:48:31.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.086+0000 7f96b7400640 1 --2- 172.21.15.28:0/2060264912 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55a1ff91f410 0x55a1ff921830 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=712ac8281acc0fe3 server_cookie=c69fda9fb7182cd5 in_seq=0 out_seq=0 2024-04-23T14:48:31.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.086+0000 7f96a0c00640 10 client.25985 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:48:31.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.086+0000 7f96a0c00640 1 -- 172.21.15.28:0/2060264912 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55a1ff802a80 con 0x55a1ff91f410 2024-04-23T14:48:31.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.086+0000 7f96a0c00640 10 client.25985 handle_command_reply: tid=0 2024-04-23T14:48:31.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.086+0000 7f96c0f7d740 1 client.25985 shutdown 2024-04-23T14:48:31.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.086+0000 7f96c0f7d740 1 -- 172.21.15.28:0/2060264912 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a1ff8a0850 msgr2=0x55a1ff899d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:31.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.086+0000 7f96c0f7d740 1 --2- 172.21.15.28:0/2060264912 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a1ff8a0850 0x55a1ff899d80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f96b00ee760 tx=0x7f96b00bcbb0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.086+0000 7f96c0f7d740 1 -- 172.21.15.28:0/2060264912 shutdown_connections 2024-04-23T14:48:31.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.086+0000 7f96c0f7d740 1 --2- 172.21.15.28:0/2060264912 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55a1ff91f410 0x55a1ff921830 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.087+0000 7f96c0f7d740 1 --2- 172.21.15.28:0/2060264912 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a1ff8a0850 0x55a1ff899d80 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.087+0000 7f96c0f7d740 1 --2- 172.21.15.28:0/2060264912 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a1ff896180 0x55a1ff8a8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.087+0000 7f96a0c00640 0 client.25985 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:48:31.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.087+0000 7f96c0f7d740 1 --2- 172.21.15.28:0/2060264912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1ff895860 0x55a1ff8a8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.087+0000 7f96c0f7d740 1 -- 172.21.15.28:0/2060264912 >> 172.21.15.28:0/2060264912 conn(0x55a1ff891b80 msgr2=0x55a1ff890c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:48:31.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.087+0000 7f96c0f7d740 1 -- 172.21.15.28:0/2060264912 shutdown_connections 2024-04-23T14:48:31.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.087+0000 7f96c0f7d740 1 -- 172.21.15.28:0/2060264912 wait complete. 2024-04-23T14:48:31.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.087+0000 7f96c0f7d740 20 client.25985 trim_cache size 0 max 16384 2024-04-23T14:48:31.093 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:48:31.093 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-23T14:48:31.093 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-23T14:48:31.093 INFO:teuthology.orchestra.run.smithi028.stdout: "2c10311a-1b77-4209-817b-2d6d8a633f09": { 2024-04-23T14:48:31.093 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-23T14:48:31.094 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "2c10311a-1b77-4209-817b-2d6d8a633f09", 2024-04-23T14:48:31.094 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-23T14:48:31.094 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:48:31.094 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:48:31.094 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:48:31.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.088+0000 7f96be200640 1 -- 172.21.15.28:0/1545817322 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f969c0a3da0 msgr2=0x7f969c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:31.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.088+0000 7f96be200640 1 --2- 172.21.15.28:0/1545817322 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f969c0a3da0 0x7f969c0a6260 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x55a1ff892d10 tx=0x7f96a400d040 comp rx=0 tx=0).stop 2024-04-23T14:48:31.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.088+0000 7f96be200640 1 -- 172.21.15.28:0/1545817322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f96b819af20 msgr2=0x7f96b819b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:48:31.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.088+0000 7f96be200640 1 --2- 172.21.15.28:0/1545817322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f96b819af20 0x7f96b819b3d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f96b0009b30 tx=0x7f96b00305d0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.088+0000 7f96be200640 1 -- 172.21.15.28:0/1545817322 shutdown_connections 2024-04-23T14:48:31.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.088+0000 7f96be200640 1 --2- 172.21.15.28:0/1545817322 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f969c0a3da0 0x7f969c0a6260 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.088+0000 7f96be200640 1 --2- 172.21.15.28:0/1545817322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f96b819af20 0x7f96b819b3d0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.088+0000 7f96be200640 1 --2- 172.21.15.28:0/1545817322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96b8106210 0x7f96b8196ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.088+0000 7f96be200640 1 --2- 172.21.15.28:0/1545817322 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96b8101140 0x7f96b8196660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:48:31.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.088+0000 7f96be200640 1 -- 172.21.15.28:0/1545817322 >> 172.21.15.28:0/1545817322 conn(0x7f96b8078000 msgr2=0x7f96b80fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:48:31.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.088+0000 7f96be200640 1 -- 172.21.15.28:0/1545817322 shutdown_connections 2024-04-23T14:48:31.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:48:31.088+0000 7f96be200640 1 -- 172.21.15.28:0/1545817322 wait complete. 2024-04-23T14:48:31.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:30 smithi028 ceph-mon[27810]: pgmap v137: 97 pgs: 97 active+clean; 176 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 2.6 MiB/s wr, 155 op/s 2024-04-23T14:48:31.107 INFO:tasks.cephfs.filesystem:scrub status for tag:2c10311a-1b77-4209-817b-2d6d8a633f09 - {'path': '/', 'tag': '2c10311a-1b77-4209-817b-2d6d8a633f09', 'options': 'recursive,force'} 2024-04-23T14:48:31.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:30 smithi038 ceph-mon[32348]: pgmap v137: 97 pgs: 97 active+clean; 176 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 2.6 MiB/s wr, 155 op/s 2024-04-23T14:48:31.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:30 smithi105 ceph-mon[32555]: pgmap v137: 97 pgs: 97 active+clean; 176 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 2.6 MiB/s wr, 155 op/s 2024-04-23T14:48:31.483 INFO:tasks.workunit.client.0.smithi028.stdout: 1 319475 9.78 MB/sec execute 47 sec latency 8.853 ms 2024-04-23T14:48:32.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:31 smithi038 ceph-mon[32348]: scrub queued for path: / 2024-04-23T14:48:32.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:31 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:48:32.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:31 smithi038 ceph-mon[32348]: scrub summary: active paths [/] 2024-04-23T14:48:32.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:31 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:48:32.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:31 smithi028 ceph-mon[27810]: scrub queued for path: / 2024-04-23T14:48:32.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:31 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:48:32.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:31 smithi028 ceph-mon[27810]: scrub summary: active paths [/] 2024-04-23T14:48:32.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:31 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:48:32.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:31 smithi105 ceph-mon[32555]: scrub queued for path: / 2024-04-23T14:48:32.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:31 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:48:32.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:31 smithi105 ceph-mon[32555]: scrub summary: active paths [/] 2024-04-23T14:48:32.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:31 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:48:32.483 INFO:tasks.workunit.client.0.smithi028.stdout: 1 320952 9.70 MB/sec execute 48 sec latency 15.287 ms 2024-04-23T14:48:33.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:32 smithi038 ceph-mon[32348]: pgmap v138: 97 pgs: 97 active+clean; 176 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 2.6 MiB/s wr, 156 op/s 2024-04-23T14:48:33.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:32 smithi028 ceph-mon[27810]: pgmap v138: 97 pgs: 97 active+clean; 176 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 2.6 MiB/s wr, 156 op/s 2024-04-23T14:48:33.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:32 smithi105 ceph-mon[32555]: pgmap v138: 97 pgs: 97 active+clean; 176 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 2.6 MiB/s wr, 156 op/s 2024-04-23T14:48:33.483 INFO:tasks.workunit.client.0.smithi028.stdout: 1 323179 9.78 MB/sec execute 49 sec latency 20.132 ms 2024-04-23T14:48:34.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:33 smithi038 ceph-mon[32348]: pgmap v139: 97 pgs: 97 active+clean; 181 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.8 MiB/s wr, 235 op/s 2024-04-23T14:48:34.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:33 smithi028 ceph-mon[27810]: pgmap v139: 97 pgs: 97 active+clean; 181 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.8 MiB/s wr, 235 op/s 2024-04-23T14:48:34.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:33 smithi105 ceph-mon[32555]: pgmap v139: 97 pgs: 97 active+clean; 181 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.8 MiB/s wr, 235 op/s 2024-04-23T14:48:34.483 INFO:tasks.workunit.client.0.smithi028.stdout: 1 325055 9.70 MB/sec execute 50 sec latency 54.015 ms 2024-04-23T14:48:35.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:34 smithi105 ceph-mon[32555]: scrub summary: idle 2024-04-23T14:48:35.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:34 smithi038 ceph-mon[32348]: scrub summary: idle 2024-04-23T14:48:35.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:34 smithi028 ceph-mon[27810]: scrub summary: idle 2024-04-23T14:48:35.483 INFO:tasks.workunit.client.0.smithi028.stdout: 1 326938 9.77 MB/sec execute 51 sec latency 20.840 ms 2024-04-23T14:48:36.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:35 smithi105 ceph-mon[32555]: pgmap v140: 97 pgs: 97 active+clean; 181 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 2.4 MiB/s wr, 154 op/s 2024-04-23T14:48:36.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:35 smithi038 ceph-mon[32348]: pgmap v140: 97 pgs: 97 active+clean; 181 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 2.4 MiB/s wr, 154 op/s 2024-04-23T14:48:36.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:35 smithi028 ceph-mon[27810]: pgmap v140: 97 pgs: 97 active+clean; 181 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 2.4 MiB/s wr, 154 op/s 2024-04-23T14:48:36.483 INFO:tasks.workunit.client.0.smithi028.stdout: 1 329331 9.83 MB/sec execute 52 sec latency 12.114 ms 2024-04-23T14:48:37.483 INFO:tasks.workunit.client.0.smithi028.stdout: 1 330927 9.80 MB/sec execute 53 sec latency 14.827 ms 2024-04-23T14:48:38.484 INFO:tasks.workunit.client.0.smithi028.stdout: 1 333260 9.85 MB/sec execute 54 sec latency 29.020 ms 2024-04-23T14:48:38.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:38 smithi105 ceph-mon[32555]: pgmap v141: 97 pgs: 97 active+clean; 187 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.7 MiB/s wr, 236 op/s 2024-04-23T14:48:38.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:38 smithi038 ceph-mon[32348]: pgmap v141: 97 pgs: 97 active+clean; 187 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.7 MiB/s wr, 236 op/s 2024-04-23T14:48:38.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:38 smithi028 ceph-mon[27810]: pgmap v141: 97 pgs: 97 active+clean; 187 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.7 MiB/s wr, 236 op/s 2024-04-23T14:48:39.484 INFO:tasks.workunit.client.0.smithi028.stdout: 1 334352 9.77 MB/sec execute 55 sec latency 44.643 ms 2024-04-23T14:48:40.484 INFO:tasks.workunit.client.0.smithi028.stdout: 1 336437 9.81 MB/sec execute 56 sec latency 37.135 ms 2024-04-23T14:48:40.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:40 smithi105 ceph-mon[32555]: pgmap v142: 97 pgs: 97 active+clean; 187 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 951 KiB/s rd, 2.5 MiB/s wr, 161 op/s 2024-04-23T14:48:40.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:40 smithi038 ceph-mon[32348]: pgmap v142: 97 pgs: 97 active+clean; 187 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 951 KiB/s rd, 2.5 MiB/s wr, 161 op/s 2024-04-23T14:48:40.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:40 smithi028 ceph-mon[27810]: pgmap v142: 97 pgs: 97 active+clean; 187 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 951 KiB/s rd, 2.5 MiB/s wr, 161 op/s 2024-04-23T14:48:41.484 INFO:tasks.workunit.client.0.smithi028.stdout: 1 337782 9.76 MB/sec execute 57 sec latency 19.817 ms 2024-04-23T14:48:42.484 INFO:tasks.workunit.client.0.smithi028.stdout: 1 340176 9.81 MB/sec execute 58 sec latency 9.606 ms 2024-04-23T14:48:42.786 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:42 smithi105 ceph-mon[32555]: pgmap v143: 97 pgs: 97 active+clean; 187 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 951 KiB/s rd, 2.5 MiB/s wr, 161 op/s 2024-04-23T14:48:42.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:42 smithi038 ceph-mon[32348]: pgmap v143: 97 pgs: 97 active+clean; 187 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 951 KiB/s rd, 2.5 MiB/s wr, 161 op/s 2024-04-23T14:48:42.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:42 smithi028 ceph-mon[27810]: pgmap v143: 97 pgs: 97 active+clean; 187 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 951 KiB/s rd, 2.5 MiB/s wr, 161 op/s 2024-04-23T14:48:43.484 INFO:tasks.workunit.client.0.smithi028.stdout: 1 341558 9.78 MB/sec execute 59 sec latency 13.523 ms 2024-04-23T14:48:44.484 INFO:tasks.workunit.client.0.smithi028.stdout: 1 343692 9.80 MB/sec execute 60 sec latency 52.192 ms 2024-04-23T14:48:44.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:44 smithi038 ceph-mon[32348]: pgmap v144: 97 pgs: 97 active+clean; 192 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 235 op/s 2024-04-23T14:48:44.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:44 smithi028 ceph-mon[27810]: pgmap v144: 97 pgs: 97 active+clean; 192 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 235 op/s 2024-04-23T14:48:45.038 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:44 smithi105 ceph-mon[32555]: pgmap v144: 97 pgs: 97 active+clean; 192 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 235 op/s 2024-04-23T14:48:45.484 INFO:tasks.workunit.client.0.smithi028.stdout: 1 344941 9.75 MB/sec execute 61 sec latency 48.578 ms 2024-04-23T14:48:46.484 INFO:tasks.workunit.client.0.smithi028.stdout: 1 346995 9.78 MB/sec execute 62 sec latency 11.987 ms 2024-04-23T14:48:46.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:46 smithi038 ceph-mon[32348]: pgmap v145: 97 pgs: 97 active+clean; 192 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 2.4 MiB/s wr, 156 op/s 2024-04-23T14:48:46.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:46 smithi028 ceph-mon[27810]: pgmap v145: 97 pgs: 97 active+clean; 192 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 2.4 MiB/s wr, 156 op/s 2024-04-23T14:48:47.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:46 smithi105 ceph-mon[32555]: pgmap v145: 97 pgs: 97 active+clean; 192 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 2.4 MiB/s wr, 156 op/s 2024-04-23T14:48:47.484 INFO:tasks.workunit.client.0.smithi028.stdout: 1 348391 9.74 MB/sec execute 63 sec latency 18.918 ms 2024-04-23T14:48:48.484 INFO:tasks.workunit.client.0.smithi028.stdout: 1 350510 9.77 MB/sec execute 64 sec latency 12.441 ms 2024-04-23T14:48:48.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:48 smithi038 ceph-mon[32348]: pgmap v146: 97 pgs: 97 active+clean; 196 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 232 op/s 2024-04-23T14:48:48.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:48 smithi028 ceph-mon[27810]: pgmap v146: 97 pgs: 97 active+clean; 196 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 232 op/s 2024-04-23T14:48:49.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:48 smithi105 ceph-mon[32555]: pgmap v146: 97 pgs: 97 active+clean; 196 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 232 op/s 2024-04-23T14:48:49.485 INFO:tasks.workunit.client.0.smithi028.stdout: 1 351644 9.73 MB/sec execute 65 sec latency 71.837 ms 2024-04-23T14:48:50.485 INFO:tasks.workunit.client.0.smithi028.stdout: 1 353600 9.69 MB/sec execute 66 sec latency 64.974 ms 2024-04-23T14:48:50.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:50 smithi038 ceph-mon[32348]: pgmap v147: 97 pgs: 97 active+clean; 196 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 2.4 MiB/s wr, 150 op/s 2024-04-23T14:48:50.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:50 smithi028 ceph-mon[27810]: pgmap v147: 97 pgs: 97 active+clean; 196 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 2.4 MiB/s wr, 150 op/s 2024-04-23T14:48:51.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:50 smithi105 ceph-mon[32555]: pgmap v147: 97 pgs: 97 active+clean; 196 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 2.4 MiB/s wr, 150 op/s 2024-04-23T14:48:51.485 INFO:tasks.workunit.client.0.smithi028.stdout: 1 355096 9.72 MB/sec execute 67 sec latency 12.195 ms 2024-04-23T14:48:52.485 INFO:tasks.workunit.client.0.smithi028.stdout: 1 356840 9.66 MB/sec execute 68 sec latency 18.950 ms 2024-04-23T14:48:52.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:52 smithi038 ceph-mon[32348]: pgmap v148: 97 pgs: 97 active+clean; 196 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 2.4 MiB/s wr, 150 op/s 2024-04-23T14:48:52.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:52 smithi028 ceph-mon[27810]: pgmap v148: 97 pgs: 97 active+clean; 196 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 2.4 MiB/s wr, 150 op/s 2024-04-23T14:48:53.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:52 smithi105 ceph-mon[32555]: pgmap v148: 97 pgs: 97 active+clean; 196 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 2.4 MiB/s wr, 150 op/s 2024-04-23T14:48:53.485 INFO:tasks.workunit.client.0.smithi028.stdout: 1 358586 9.72 MB/sec execute 69 sec latency 10.830 ms 2024-04-23T14:48:54.485 INFO:tasks.workunit.client.0.smithi028.stdout: 1 360113 9.66 MB/sec execute 70 sec latency 51.615 ms 2024-04-23T14:48:54.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:54 smithi038 ceph-mon[32348]: pgmap v149: 97 pgs: 97 active+clean; 201 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 226 op/s 2024-04-23T14:48:54.877 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:54 smithi028 ceph-mon[27810]: pgmap v149: 97 pgs: 97 active+clean; 201 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 226 op/s 2024-04-23T14:48:55.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:54 smithi105 ceph-mon[32555]: pgmap v149: 97 pgs: 97 active+clean; 201 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 226 op/s 2024-04-23T14:48:55.485 INFO:tasks.workunit.client.0.smithi028.stdout: 1 362187 9.71 MB/sec execute 71 sec latency 65.045 ms 2024-04-23T14:48:56.486 INFO:tasks.workunit.client.0.smithi028.stdout: 1 364344 9.69 MB/sec execute 72 sec latency 12.887 ms 2024-04-23T14:48:56.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:56 smithi038 ceph-mon[32348]: pgmap v150: 97 pgs: 97 active+clean; 201 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 2.6 MiB/s wr, 152 op/s 2024-04-23T14:48:56.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:56 smithi028 ceph-mon[27810]: pgmap v150: 97 pgs: 97 active+clean; 201 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 2.6 MiB/s wr, 152 op/s 2024-04-23T14:48:57.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:56 smithi105 ceph-mon[32555]: pgmap v150: 97 pgs: 97 active+clean; 201 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 2.6 MiB/s wr, 152 op/s 2024-04-23T14:48:57.486 INFO:tasks.workunit.client.0.smithi028.stdout: 1 366175 9.72 MB/sec execute 73 sec latency 13.032 ms 2024-04-23T14:48:58.486 INFO:tasks.workunit.client.0.smithi028.stdout: 1 368361 9.75 MB/sec execute 74 sec latency 10.524 ms 2024-04-23T14:48:58.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:48:58 smithi038 ceph-mon[32348]: pgmap v151: 97 pgs: 97 active+clean; 206 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 226 op/s 2024-04-23T14:48:58.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:48:58 smithi028 ceph-mon[27810]: pgmap v151: 97 pgs: 97 active+clean; 206 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 226 op/s 2024-04-23T14:48:59.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:48:58 smithi105 ceph-mon[32555]: pgmap v151: 97 pgs: 97 active+clean; 206 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 226 op/s 2024-04-23T14:48:59.486 INFO:tasks.workunit.client.0.smithi028.stdout: 1 369668 9.71 MB/sec execute 75 sec latency 58.213 ms 2024-04-23T14:49:00.486 INFO:tasks.workunit.client.0.smithi028.stdout: 1 371847 9.74 MB/sec execute 76 sec latency 51.174 ms 2024-04-23T14:49:00.838 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:00 smithi105 ceph-mon[32555]: pgmap v152: 97 pgs: 97 active+clean; 206 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 974 KiB/s rd, 2.5 MiB/s wr, 150 op/s 2024-04-23T14:49:00.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:00 smithi038 ceph-mon[32348]: pgmap v152: 97 pgs: 97 active+clean; 206 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 974 KiB/s rd, 2.5 MiB/s wr, 150 op/s 2024-04-23T14:49:00.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:00 smithi028 ceph-mon[27810]: pgmap v152: 97 pgs: 97 active+clean; 206 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 974 KiB/s rd, 2.5 MiB/s wr, 150 op/s 2024-04-23T14:49:01.108 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:49:01.108 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:49:01.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.317+0000 7f5e47600640 1 -- 172.21.15.28:0/1632007648 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5e40109580 msgr2=0x7f5e40111150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:01.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.317+0000 7f5e47600640 1 --2- 172.21.15.28:0/1632007648 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5e40109580 0x7f5e40111150 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5e30009900 tx=0x7f5e3002a3f0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.317+0000 7f5e47600640 1 -- 172.21.15.28:0/1632007648 shutdown_connections 2024-04-23T14:49:01.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.317+0000 7f5e47600640 1 --2- 172.21.15.28:0/1632007648 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5e40109580 0x7f5e40111150 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.317+0000 7f5e47600640 1 --2- 172.21.15.28:0/1632007648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e40104e60 0x7f5e40108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.317+0000 7f5e47600640 1 --2- 172.21.15.28:0/1632007648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e401044b0 0x7f5e40104890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.317+0000 7f5e47600640 1 -- 172.21.15.28:0/1632007648 >> 172.21.15.28:0/1632007648 conn(0x7f5e400ffd90 msgr2=0x7f5e401021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:01.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.317+0000 7f5e47600640 1 -- 172.21.15.28:0/1632007648 shutdown_connections 2024-04-23T14:49:01.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.317+0000 7f5e47600640 1 -- 172.21.15.28:0/1632007648 wait complete. 2024-04-23T14:49:01.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.318+0000 7f5e47600640 1 Processor -- start 2024-04-23T14:49:01.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.319+0000 7f5e47600640 1 -- start start 2024-04-23T14:49:01.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.319+0000 7f5e47600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5e401044b0 0x7f5e401a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:01.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.319+0000 7f5e47600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e40109580 0x7f5e401a0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:01.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.319+0000 7f5e47600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e4019a9b0 0x7f5e4019ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:01.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.319+0000 7f5e47600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5e40114690 con 0x7f5e40109580 2024-04-23T14:49:01.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.319+0000 7f5e47600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5e40114510 con 0x7f5e4019a9b0 2024-04-23T14:49:01.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.319+0000 7f5e47600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5e40114810 con 0x7f5e401044b0 2024-04-23T14:49:01.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.319+0000 7f5e44c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5e401044b0 0x7f5e401a07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:01.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.319+0000 7f5e3fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e40109580 0x7f5e401a0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:01.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.319+0000 7f5e44c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5e401044b0 0x7f5e401a07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:47292/0 (socket says 172.21.15.28:47292) 2024-04-23T14:49:01.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.319+0000 7f5e44c00640 1 -- 172.21.15.28:0/1866011223 learned_addr learned my addr 172.21.15.28:0/1866011223 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:01.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.319+0000 7f5e45600640 1 --2- 172.21.15.28:0/1866011223 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e4019a9b0 0x7f5e4019ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:01.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.320+0000 7f5e44c00640 1 -- 172.21.15.28:0/1866011223 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e4019a9b0 msgr2=0x7f5e4019ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:01.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.320+0000 7f5e44c00640 1 --2- 172.21.15.28:0/1866011223 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e4019a9b0 0x7f5e4019ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.320+0000 7f5e44c00640 1 -- 172.21.15.28:0/1866011223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e40109580 msgr2=0x7f5e401a0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:01.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.320+0000 7f5e44c00640 1 --2- 172.21.15.28:0/1866011223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e40109580 0x7f5e401a0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.320+0000 7f5e44c00640 1 -- 172.21.15.28:0/1866011223 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e4019b4d0 con 0x7f5e401044b0 2024-04-23T14:49:01.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.320+0000 7f5e3fe00640 1 --2- 172.21.15.28:0/1866011223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e40109580 0x7f5e401a0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:49:01.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.320+0000 7f5e45600640 1 --2- 172.21.15.28:0/1866011223 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e4019a9b0 0x7f5e4019ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:49:01.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.320+0000 7f5e44c00640 1 --2- 172.21.15.28:0/1866011223 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5e401044b0 0x7f5e401a07b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5e3800eb90 tx=0x7f5e3800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:01.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.320+0000 7f5e3d600640 1 -- 172.21.15.28:0/1866011223 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e38004d20 con 0x7f5e401044b0 2024-04-23T14:49:01.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.321+0000 7f5e3d600640 1 -- 172.21.15.28:0/1866011223 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e3800ee90 con 0x7f5e401044b0 2024-04-23T14:49:01.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.321+0000 7f5e3d600640 1 -- 172.21.15.28:0/1866011223 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e38010830 con 0x7f5e401044b0 2024-04-23T14:49:01.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.321+0000 7f5e47600640 1 -- 172.21.15.28:0/1866011223 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e4019b7c0 con 0x7f5e401044b0 2024-04-23T14:49:01.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.321+0000 7f5e47600640 1 -- 172.21.15.28:0/1866011223 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e401a7970 con 0x7f5e401044b0 2024-04-23T14:49:01.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.324+0000 7f5e4a453740 1 Processor -- start 2024-04-23T14:49:01.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.324+0000 7f5e4a453740 1 -- start start 2024-04-23T14:49:01.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.324+0000 7f5e3d600640 1 -- 172.21.15.28:0/1866011223 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f5e38005020 con 0x7f5e401044b0 2024-04-23T14:49:01.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.324+0000 7f5e4a453740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9c1068860 0x55d9c1068c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:01.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.324+0000 7f5e4a453740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9c1069180 0x55d9c1073310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:01.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.324+0000 7f5e4a453740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9c1073850 0x55d9c1075c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:01.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.324+0000 7f5e4a453740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d9c0fce1f0 con 0x55d9c1068860 2024-04-23T14:49:01.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.324+0000 7f5e4a453740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d9c0f74350 con 0x55d9c1073850 2024-04-23T14:49:01.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.324+0000 7f5e4a453740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d9c0fca1b0 con 0x55d9c1069180 2024-04-23T14:49:01.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.325+0000 7f5e44c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9c1069180 0x55d9c1073310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:01.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.325+0000 7f5e45600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9c1068860 0x55d9c1068c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:01.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.325+0000 7f5e44c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9c1069180 0x55d9c1073310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:47294/0 (socket says 172.21.15.28:47294) 2024-04-23T14:49:01.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.325+0000 7f5e45600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9c1068860 0x55d9c1068c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34354/0 (socket says 172.21.15.28:34354) 2024-04-23T14:49:01.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.325+0000 7f5e44c00640 1 -- 172.21.15.28:0/3907705336 learned_addr learned my addr 172.21.15.28:0/3907705336 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:01.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.325+0000 7f5e44c00640 1 -- 172.21.15.28:0/3907705336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9c1073850 msgr2=0x55d9c1075c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:01.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.325+0000 7f5e44c00640 1 --2- 172.21.15.28:0/3907705336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9c1073850 0x55d9c1075c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.325+0000 7f5e44c00640 1 -- 172.21.15.28:0/3907705336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9c1068860 msgr2=0x55d9c1068c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:01.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.325+0000 7f5e44c00640 1 --2- 172.21.15.28:0/3907705336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9c1068860 0x55d9c1068c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.325+0000 7f5e44c00640 1 -- 172.21.15.28:0/3907705336 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9c0fc8f50 con 0x55d9c1069180 2024-04-23T14:49:01.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.326+0000 7f5e44c00640 1 --2- 172.21.15.28:0/3907705336 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9c1069180 0x55d9c1073310 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5e380ce480 tx=0x7f5e380d4e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ac1a90055777f56 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:01.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.326+0000 7f5e35c00640 1 -- 172.21.15.28:0/3907705336 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e380e4070 con 0x55d9c1069180 2024-04-23T14:49:01.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.326+0000 7f5e35c00640 1 -- 172.21.15.28:0/3907705336 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e380d9b60 con 0x55d9c1069180 2024-04-23T14:49:01.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.326+0000 7f5e35c00640 1 -- 172.21.15.28:0/3907705336 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e380de5e0 con 0x55d9c1069180 2024-04-23T14:49:01.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.326+0000 7f5e3d600640 1 --2- 172.21.15.28:0/1866011223 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5e240a3da0 0x7f5e240a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:01.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.326+0000 7f5e3d600640 1 -- 172.21.15.28:0/1866011223 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f5e380d0db0 con 0x7f5e401044b0 2024-04-23T14:49:01.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.326+0000 7f5e45600640 1 --2- 172.21.15.28:0/1866011223 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5e240a3da0 0x7f5e240a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:01.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.327+0000 7f5e4a453740 1 -- 172.21.15.28:0/3907705336 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9c1069180 msgr2=0x55d9c1073310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:01.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.327+0000 7f5e4a453740 1 --2- 172.21.15.28:0/3907705336 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9c1069180 0x55d9c1073310 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5e380ce480 tx=0x7f5e380d4e60 comp rx=0 tx=0).stop 2024-04-23T14:49:01.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.327+0000 7f5e4a453740 1 -- 172.21.15.28:0/3907705336 shutdown_connections 2024-04-23T14:49:01.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.327+0000 7f5e4a453740 1 --2- 172.21.15.28:0/3907705336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9c1073850 0x55d9c1075c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.327+0000 7f5e4a453740 1 --2- 172.21.15.28:0/3907705336 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9c1069180 0x55d9c1073310 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.327+0000 7f5e4a453740 1 --2- 172.21.15.28:0/3907705336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9c1068860 0x55d9c1068c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.327+0000 7f5e4a453740 1 -- 172.21.15.28:0/3907705336 >> 172.21.15.28:0/3907705336 conn(0x55d9c1064b80 msgr2=0x55d9c1064f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:01.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.327+0000 7f5e4a453740 1 -- 172.21.15.28:0/3907705336 shutdown_connections 2024-04-23T14:49:01.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.327+0000 7f5e4a453740 1 -- 172.21.15.28:0/3907705336 wait complete. 2024-04-23T14:49:01.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.327+0000 7f5e45600640 1 --2- 172.21.15.28:0/1866011223 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5e240a3da0 0x7f5e240a6260 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5e30031550 tx=0x7f5e300046c0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:01.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.328+0000 7f5e4a453740 1 -- start start 2024-04-23T14:49:01.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.328+0000 7f5e4a453740 1 Processor -- start 2024-04-23T14:49:01.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.328+0000 7f5e4a453740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9c1068860 0x55d9c10707a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:01.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.328+0000 7f5e4a453740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9c1069180 0x55d9c1070ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:01.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.328+0000 7f5e4a453740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9c1073850 0x55d9c1071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:01.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.329+0000 7f5e4a453740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d9c1076640 con 0x55d9c1073850 2024-04-23T14:49:01.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.329+0000 7f5e4a453740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d9c0f87690 con 0x55d9c1069180 2024-04-23T14:49:01.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.329+0000 7f5e4a453740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d9c10767c0 con 0x55d9c1068860 2024-04-23T14:49:01.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.329+0000 7f5e4a453740 10 client.0 _ll_register_callbacks cb 0x55d9c0fc3650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:49:01.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.329+0000 7f5e44c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9c1068860 0x55d9c10707a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:01.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.329+0000 7f5e44c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9c1068860 0x55d9c10707a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:47302/0 (socket says 172.21.15.28:47302) 2024-04-23T14:49:01.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.329+0000 7f5e3fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9c1069180 0x55d9c1070ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:01.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.329+0000 7f5e44c00640 1 -- 172.21.15.28:0/1971172324 learned_addr learned my addr 172.21.15.28:0/1971172324 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:01.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.329+0000 7f5e45600640 1 --2- 172.21.15.28:0/1971172324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9c1073850 0x55d9c1071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:01.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.329+0000 7f5e44c00640 1 -- 172.21.15.28:0/1971172324 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9c1069180 msgr2=0x55d9c1070ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:01.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.329+0000 7f5e44c00640 1 --2- 172.21.15.28:0/1971172324 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9c1069180 0x55d9c1070ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.329+0000 7f5e44c00640 1 -- 172.21.15.28:0/1971172324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9c1073850 msgr2=0x55d9c1071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:01.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.329+0000 7f5e44c00640 1 --2- 172.21.15.28:0/1971172324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9c1073850 0x55d9c1071220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.329+0000 7f5e44c00640 1 -- 172.21.15.28:0/1971172324 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9c0f86920 con 0x55d9c1068860 2024-04-23T14:49:01.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.329+0000 7f5e3fe00640 1 --2- 172.21.15.28:0/1971172324 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9c1069180 0x55d9c1070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:49:01.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.329+0000 7f5e44c00640 1 --2- 172.21.15.28:0/1971172324 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9c1068860 0x55d9c10707a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5e38099730 tx=0x7f5e380dfed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9eefca1225ba4f6c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:01.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.330+0000 7f5e2e000640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:49:01.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.330+0000 7f5e2e000640 1 -- 172.21.15.28:0/1971172324 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e38099000 con 0x55d9c1068860 2024-04-23T14:49:01.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.330+0000 7f5e2e000640 1 -- 172.21.15.28:0/1971172324 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f5e3809b060 con 0x55d9c1068860 2024-04-23T14:49:01.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.330+0000 7f5e4a453740 1 -- 172.21.15.28:0/1971172324 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e380d9b60 con 0x55d9c1068860 2024-04-23T14:49:01.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.330+0000 7f5e2e000640 1 -- 172.21.15.28:0/1971172324 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f5e3809c6c0 con 0x55d9c1068860 2024-04-23T14:49:01.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.330+0000 7f5e4a453740 10 client.25981 fetch_fsmap learned FSMap version 20 2024-04-23T14:49:01.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.330+0000 7f5e4a453740 1 -- 172.21.15.28:0/1971172324 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55d9c0daa7d0 con 0x55d9c1068860 2024-04-23T14:49:01.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.330+0000 7f5e2e000640 1 -- 172.21.15.28:0/1971172324 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e380d9b60 con 0x55d9c1068860 2024-04-23T14:49:01.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.330+0000 7f5e2e000640 1 -- 172.21.15.28:0/1971172324 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f5e380d9d10 con 0x55d9c1068860 2024-04-23T14:49:01.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.331+0000 7f5e4a453740 10 client.25981 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:49:01.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.331+0000 7f5e4a453740 10 client.25981 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:49:01.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.331+0000 7f5e4a453740 20 client.25981 populate_metadata read hostname 'smithi028' 2024-04-23T14:49:01.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.331+0000 7f5e4a453740 1 --2- 172.21.15.28:0/1971172324 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d9c1083a10 0x55d9c1085e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:01.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.331+0000 7f5e4a453740 4 client.25981 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:49:01.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.331+0000 7f5e4a453740 1 -- 172.21.15.28:0/1971172324 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9c0fdbc10 con 0x55d9c1083a10 2024-04-23T14:49:01.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.331+0000 7f5e3fe00640 1 --2- 172.21.15.28:0/1971172324 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d9c1083a10 0x55d9c1085e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:01.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.331+0000 7f5e3fe00640 1 --2- 172.21.15.28:0/1971172324 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d9c1083a10 0x55d9c1085e00 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23bbe046f813d17a server_cookie=9613442a3d75d945 in_seq=0 out_seq=0 2024-04-23T14:49:01.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.331+0000 7f5e2e000640 10 client.25981 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:49:01.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.332+0000 7f5e2e000640 1 -- 172.21.15.28:0/1971172324 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d9c0fdbc10 con 0x55d9c1083a10 2024-04-23T14:49:01.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.332+0000 7f5e2e000640 10 client.25981 handle_command_reply: tid=0 2024-04-23T14:49:01.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.332+0000 7f5e4a453740 1 client.25981 shutdown 2024-04-23T14:49:01.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.332+0000 7f5e4a453740 1 -- 172.21.15.28:0/1971172324 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9c1068860 msgr2=0x55d9c10707a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:01.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.332+0000 7f5e4a453740 1 --2- 172.21.15.28:0/1971172324 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9c1068860 0x55d9c10707a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5e38099730 tx=0x7f5e380dfed0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.332+0000 7f5e4a453740 1 -- 172.21.15.28:0/1971172324 shutdown_connections 2024-04-23T14:49:01.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.332+0000 7f5e4a453740 1 --2- 172.21.15.28:0/1971172324 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d9c1083a10 0x55d9c1085e00 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.332+0000 7f5e4a453740 1 --2- 172.21.15.28:0/1971172324 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9c1069180 0x55d9c1070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.332+0000 7f5e4a453740 1 --2- 172.21.15.28:0/1971172324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9c1073850 0x55d9c1071220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.332+0000 7f5e4a453740 1 --2- 172.21.15.28:0/1971172324 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9c1068860 0x55d9c10707a0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.332+0000 7f5e4a453740 1 -- 172.21.15.28:0/1971172324 >> 172.21.15.28:0/1971172324 conn(0x55d9c1064b80 msgr2=0x55d9c106a980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:01.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.332+0000 7f5e2e000640 0 client.25981 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:49:01.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.333+0000 7f5e4a453740 1 -- 172.21.15.28:0/1971172324 shutdown_connections 2024-04-23T14:49:01.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.333+0000 7f5e4a453740 1 -- 172.21.15.28:0/1971172324 wait complete. 2024-04-23T14:49:01.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.333+0000 7f5e4a453740 20 client.25981 trim_cache size 0 max 16384 2024-04-23T14:49:01.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.352+0000 7f5e4a453740 1 Processor -- start 2024-04-23T14:49:01.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.352+0000 7f5e4a453740 1 -- start start 2024-04-23T14:49:01.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.352+0000 7f5e4a453740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9c1068860 0x55d9c106b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:01.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.352+0000 7f5e4a453740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9c1069180 0x55d9c1088e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:01.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.352+0000 7f5e4a453740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9c1073850 0x55d9c1089360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:01.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.352+0000 7f5e4a453740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5e401adff0 con 0x55d9c1069180 2024-04-23T14:49:01.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.352+0000 7f5e4a453740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5e401b0870 con 0x55d9c1068860 2024-04-23T14:49:01.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.352+0000 7f5e4a453740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5e401ae660 con 0x55d9c1073850 2024-04-23T14:49:01.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.353+0000 7f5e44c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9c1068860 0x55d9c106b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:01.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.353+0000 7f5e45600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9c1073850 0x55d9c1089360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:01.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.353+0000 7f5e44c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9c1068860 0x55d9c106b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:58846/0 (socket says 172.21.15.28:58846) 2024-04-23T14:49:01.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.353+0000 7f5e44c00640 1 -- 172.21.15.28:0/1002320354 learned_addr learned my addr 172.21.15.28:0/1002320354 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:01.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.353+0000 7f5e3fe00640 1 --2- 172.21.15.28:0/1002320354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9c1069180 0x55d9c1088e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:01.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.353+0000 7f5e44c00640 1 -- 172.21.15.28:0/1002320354 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9c1073850 msgr2=0x55d9c1089360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:01.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.353+0000 7f5e44c00640 1 --2- 172.21.15.28:0/1002320354 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9c1073850 0x55d9c1089360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.353+0000 7f5e44c00640 1 -- 172.21.15.28:0/1002320354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9c1069180 msgr2=0x55d9c1088e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:01.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.353+0000 7f5e44c00640 1 --2- 172.21.15.28:0/1002320354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9c1069180 0x55d9c1088e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.353+0000 7f5e44c00640 1 -- 172.21.15.28:0/1002320354 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9c0ce03f0 con 0x55d9c1068860 2024-04-23T14:49:01.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.353+0000 7f5e44c00640 1 --2- 172.21.15.28:0/1002320354 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9c1068860 0x55d9c106b000 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f5e380991c0 tx=0x7f5e380d4940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b598c8fdad06dae3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:01.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.354+0000 7f5e35200640 1 -- 172.21.15.28:0/1002320354 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e3809c8a0 con 0x55d9c1068860 2024-04-23T14:49:01.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.354+0000 7f5e35200640 1 -- 172.21.15.28:0/1002320354 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e380d46d0 con 0x55d9c1068860 2024-04-23T14:49:01.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.354+0000 7f5e35200640 1 -- 172.21.15.28:0/1002320354 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e3809cb30 con 0x55d9c1068860 2024-04-23T14:49:01.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.354+0000 7f5e4a453740 1 -- 172.21.15.28:0/1002320354 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9c1068860 msgr2=0x55d9c106b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:01.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.354+0000 7f5e4a453740 1 --2- 172.21.15.28:0/1002320354 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9c1068860 0x55d9c106b000 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f5e380991c0 tx=0x7f5e380d4940 comp rx=0 tx=0).stop 2024-04-23T14:49:01.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.354+0000 7f5e4a453740 1 -- 172.21.15.28:0/1002320354 shutdown_connections 2024-04-23T14:49:01.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.354+0000 7f5e4a453740 1 --2- 172.21.15.28:0/1002320354 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9c1073850 0x55d9c1089360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.354+0000 7f5e4a453740 1 --2- 172.21.15.28:0/1002320354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9c1069180 0x55d9c1088e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.354+0000 7f5e4a453740 1 --2- 172.21.15.28:0/1002320354 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9c1068860 0x55d9c106b000 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.354+0000 7f5e4a453740 1 -- 172.21.15.28:0/1002320354 >> 172.21.15.28:0/1002320354 conn(0x55d9c1064b80 msgr2=0x55d9c1063c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:01.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.354+0000 7f5e4a453740 1 -- 172.21.15.28:0/1002320354 shutdown_connections 2024-04-23T14:49:01.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.355+0000 7f5e4a453740 1 -- 172.21.15.28:0/1002320354 wait complete. 2024-04-23T14:49:01.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.355+0000 7f5e4a453740 1 -- start start 2024-04-23T14:49:01.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.355+0000 7f5e4a453740 1 Processor -- start 2024-04-23T14:49:01.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.356+0000 7f5e4a453740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9c1068860 0x55d9c1088e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:01.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.356+0000 7f5e4a453740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9c1069180 0x55d9c1086250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:01.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.356+0000 7f5e4a453740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9c1073850 0x55d9c1086790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:01.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.356+0000 7f5e4a453740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5e401aed30 con 0x55d9c1073850 2024-04-23T14:49:01.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.356+0000 7f5e4a453740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5e401b01a0 con 0x55d9c1068860 2024-04-23T14:49:01.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.356+0000 7f5e4a453740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5e401af400 con 0x55d9c1069180 2024-04-23T14:49:01.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.356+0000 7f5e4a453740 10 client.0 _ll_register_callbacks cb 0x7f5e380e2070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:49:01.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.356+0000 7f5e44c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9c1068860 0x55d9c1088e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:01.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.356+0000 7f5e44c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9c1068860 0x55d9c1088e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:58848/0 (socket says 172.21.15.28:58848) 2024-04-23T14:49:01.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.356+0000 7f5e3fe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9c1069180 0x55d9c1086250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:01.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.356+0000 7f5e44c00640 1 -- 172.21.15.28:0/3731408550 learned_addr learned my addr 172.21.15.28:0/3731408550 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:01.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.356+0000 7f5e45600640 1 --2- 172.21.15.28:0/3731408550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9c1073850 0x55d9c1086790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:01.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.356+0000 7f5e3fe00640 1 -- 172.21.15.28:0/3731408550 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9c1068860 msgr2=0x55d9c1088e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:01.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.356+0000 7f5e3fe00640 1 --2- 172.21.15.28:0/3731408550 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9c1068860 0x55d9c1088e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.356+0000 7f5e3fe00640 1 -- 172.21.15.28:0/3731408550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9c1073850 msgr2=0x55d9c1086790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:01.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.356+0000 7f5e3fe00640 1 --2- 172.21.15.28:0/3731408550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9c1073850 0x55d9c1086790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.356+0000 7f5e3fe00640 1 -- 172.21.15.28:0/3731408550 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9c0ca00f0 con 0x55d9c1069180 2024-04-23T14:49:01.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.356+0000 7f5e45600640 1 --2- 172.21.15.28:0/3731408550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9c1073850 0x55d9c1086790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:49:01.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.357+0000 7f5e3fe00640 1 --2- 172.21.15.28:0/3731408550 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9c1069180 0x55d9c1086250 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x55d9c1070000 tx=0x7f5e280123e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e85d48c2e73691da server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:01.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.357+0000 7f5e2e000640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:49:01.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.357+0000 7f5e2e000640 1 -- 172.21.15.28:0/3731408550 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e28004460 con 0x55d9c1069180 2024-04-23T14:49:01.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.357+0000 7f5e4a453740 1 -- 172.21.15.28:0/3731408550 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e380d46d0 con 0x55d9c1069180 2024-04-23T14:49:01.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.357+0000 7f5e2e000640 1 -- 172.21.15.28:0/3731408550 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f5e28010960 con 0x55d9c1069180 2024-04-23T14:49:01.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.357+0000 7f5e2e000640 1 -- 172.21.15.28:0/3731408550 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f5e2802f050 con 0x55d9c1069180 2024-04-23T14:49:01.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.357+0000 7f5e4a453740 10 client.25987 fetch_fsmap learned FSMap version 20 2024-04-23T14:49:01.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.357+0000 7f5e4a453740 1 -- 172.21.15.28:0/3731408550 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55d9c0f93810 con 0x55d9c1069180 2024-04-23T14:49:01.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.358+0000 7f5e2e000640 1 -- 172.21.15.28:0/3731408550 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e380d46d0 con 0x55d9c1069180 2024-04-23T14:49:01.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.358+0000 7f5e2e000640 1 -- 172.21.15.28:0/3731408550 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f5e280038a0 con 0x55d9c1069180 2024-04-23T14:49:01.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.358+0000 7f5e4a453740 10 client.25987 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:49:01.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.358+0000 7f5e4a453740 10 client.25987 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:49:01.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.358+0000 7f5e4a453740 20 client.25987 populate_metadata read hostname 'smithi028' 2024-04-23T14:49:01.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.358+0000 7f5e4a453740 1 --2- 172.21.15.28:0/3731408550 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d9c10e9c90 0x55d9c10ec0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:01.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.358+0000 7f5e4a453740 4 client.25987 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:49:01.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.358+0000 7f5e4a453740 1 -- 172.21.15.28:0/3731408550 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d9c0fd5a80 con 0x55d9c10e9c90 2024-04-23T14:49:01.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.358+0000 7f5e44c00640 1 --2- 172.21.15.28:0/3731408550 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d9c10e9c90 0x55d9c10ec0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:01.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.359+0000 7f5e44c00640 1 --2- 172.21.15.28:0/3731408550 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d9c10e9c90 0x55d9c10ec0b0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5057e313b29bf725 server_cookie=566b80527dbeae97 in_seq=0 out_seq=0 2024-04-23T14:49:01.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.359+0000 7f5e2e000640 10 client.25987 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:49:01.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.359+0000 7f5e2e000640 1 -- 172.21.15.28:0/3731408550 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d9c0fd5a80 con 0x55d9c10e9c90 2024-04-23T14:49:01.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.359+0000 7f5e2e000640 10 client.25987 handle_command_reply: tid=0 2024-04-23T14:49:01.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.359+0000 7f5e4a453740 1 client.25987 shutdown 2024-04-23T14:49:01.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.359+0000 7f5e4a453740 1 -- 172.21.15.28:0/3731408550 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9c1069180 msgr2=0x55d9c1086250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:01.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.359+0000 7f5e4a453740 1 --2- 172.21.15.28:0/3731408550 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9c1069180 0x55d9c1086250 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x55d9c1070000 tx=0x7f5e280123e0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.359+0000 7f5e4a453740 1 -- 172.21.15.28:0/3731408550 shutdown_connections 2024-04-23T14:49:01.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.359+0000 7f5e4a453740 1 --2- 172.21.15.28:0/3731408550 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d9c10e9c90 0x55d9c10ec0b0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.359+0000 7f5e4a453740 1 --2- 172.21.15.28:0/3731408550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9c1073850 0x55d9c1086790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.359+0000 7f5e4a453740 1 --2- 172.21.15.28:0/3731408550 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9c1069180 0x55d9c1086250 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.359+0000 7f5e2e000640 0 client.25987 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:49:01.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.359+0000 7f5e4a453740 1 --2- 172.21.15.28:0/3731408550 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9c1068860 0x55d9c1088e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.359+0000 7f5e4a453740 1 -- 172.21.15.28:0/3731408550 >> 172.21.15.28:0/3731408550 conn(0x55d9c1064b80 msgr2=0x55d9c107ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:01.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.359+0000 7f5e4a453740 1 -- 172.21.15.28:0/3731408550 shutdown_connections 2024-04-23T14:49:01.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.359+0000 7f5e4a453740 1 -- 172.21.15.28:0/3731408550 wait complete. 2024-04-23T14:49:01.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.360+0000 7f5e4a453740 20 client.25987 trim_cache size 0 max 16384 2024-04-23T14:49:01.364 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:49:01.364 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-23T14:49:01.364 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-23T14:49:01.364 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:49:01.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.360+0000 7f5e47600640 1 -- 172.21.15.28:0/1866011223 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5e240a3da0 msgr2=0x7f5e240a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:01.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.360+0000 7f5e47600640 1 --2- 172.21.15.28:0/1866011223 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5e240a3da0 0x7f5e240a6260 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5e30031550 tx=0x7f5e300046c0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.360+0000 7f5e47600640 1 -- 172.21.15.28:0/1866011223 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5e401044b0 msgr2=0x7f5e401a07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:01.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.360+0000 7f5e47600640 1 --2- 172.21.15.28:0/1866011223 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5e401044b0 0x7f5e401a07b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5e3800eb90 tx=0x7f5e3800c6a0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.360+0000 7f5e47600640 1 -- 172.21.15.28:0/1866011223 shutdown_connections 2024-04-23T14:49:01.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.360+0000 7f5e47600640 1 --2- 172.21.15.28:0/1866011223 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5e240a3da0 0x7f5e240a6260 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.360+0000 7f5e47600640 1 --2- 172.21.15.28:0/1866011223 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e4019a9b0 0x7f5e4019ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.360+0000 7f5e47600640 1 --2- 172.21.15.28:0/1866011223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e40109580 0x7f5e401a0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.360+0000 7f5e47600640 1 --2- 172.21.15.28:0/1866011223 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5e401044b0 0x7f5e401a07b0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.360+0000 7f5e47600640 1 -- 172.21.15.28:0/1866011223 >> 172.21.15.28:0/1866011223 conn(0x7f5e400ffd90 msgr2=0x7f5e40102040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:01.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.360+0000 7f5e47600640 1 -- 172.21.15.28:0/1866011223 shutdown_connections 2024-04-23T14:49:01.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.361+0000 7f5e47600640 1 -- 172.21.15.28:0/1866011223 wait complete. 2024-04-23T14:49:01.381 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-23T14:49:01.381 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:49:01.486 INFO:tasks.workunit.client.0.smithi028.stdout: 1 372723 9.70 MB/sec execute 77 sec latency 60.441 ms 2024-04-23T14:49:01.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.600+0000 7f4cb6a00640 1 -- 172.21.15.28:0/1277893589 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cb0105d30 msgr2=0x7f4cb0106110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:01.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.600+0000 7f4cb6a00640 1 --2- 172.21.15.28:0/1277893589 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cb0105d30 0x7f4cb0106110 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4009930 tx=0x7f4ca402a4d0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.600+0000 7f4cb6a00640 1 -- 172.21.15.28:0/1277893589 shutdown_connections 2024-04-23T14:49:01.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.600+0000 7f4cb6a00640 1 --2- 172.21.15.28:0/1277893589 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cb0100ff0 0x7f4cb0101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.600+0000 7f4cb6a00640 1 --2- 172.21.15.28:0/1277893589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cb01019b0 0x7f4cb010dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.600+0000 7f4cb6a00640 1 --2- 172.21.15.28:0/1277893589 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cb0105d30 0x7f4cb0106110 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.600+0000 7f4cb6a00640 1 -- 172.21.15.28:0/1277893589 >> 172.21.15.28:0/1277893589 conn(0x7f4cb0076840 msgr2=0x7f4cb0076c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:01.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.600+0000 7f4cb6a00640 1 -- 172.21.15.28:0/1277893589 shutdown_connections 2024-04-23T14:49:01.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.600+0000 7f4cb6a00640 1 -- 172.21.15.28:0/1277893589 wait complete. 2024-04-23T14:49:01.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.601+0000 7f4cb6a00640 1 Processor -- start 2024-04-23T14:49:01.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.601+0000 7f4cb6a00640 1 -- start start 2024-04-23T14:49:01.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.601+0000 7f4cb6a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cb0100ff0 0x7f4cb01a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:01.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.602+0000 7f4cb6a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cb01019b0 0x7f4cb01a0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:01.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.602+0000 7f4cb6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cb0105d30 0x7f4cb019a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:01.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.602+0000 7f4cb6a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb0111130 con 0x7f4cb0105d30 2024-04-23T14:49:01.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.602+0000 7f4cb6a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb0110fb0 con 0x7f4cb0100ff0 2024-04-23T14:49:01.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.602+0000 7f4cb6a00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb01112b0 con 0x7f4cb01019b0 2024-04-23T14:49:01.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.602+0000 7f4cb5600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cb0100ff0 0x7f4cb01a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:01.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.602+0000 7f4cb4c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cb01019b0 0x7f4cb01a0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:01.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.602+0000 7f4cb5600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cb0100ff0 0x7f4cb01a05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:58862/0 (socket says 172.21.15.28:58862) 2024-04-23T14:49:01.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.602+0000 7f4cb5600640 1 -- 172.21.15.28:0/3775437572 learned_addr learned my addr 172.21.15.28:0/3775437572 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:01.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.602+0000 7f4cb4c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cb01019b0 0x7f4cb01a0b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:47324/0 (socket says 172.21.15.28:47324) 2024-04-23T14:49:01.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.602+0000 7f4cb6000640 1 --2- 172.21.15.28:0/3775437572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cb0105d30 0x7f4cb019a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:01.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.602+0000 7f4cb5600640 1 -- 172.21.15.28:0/3775437572 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cb01019b0 msgr2=0x7f4cb01a0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:01.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.602+0000 7f4cb5600640 1 --2- 172.21.15.28:0/3775437572 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cb01019b0 0x7f4cb01a0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.602+0000 7f4cb5600640 1 -- 172.21.15.28:0/3775437572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cb0105d30 msgr2=0x7f4cb019a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:01.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.602+0000 7f4cb5600640 1 --2- 172.21.15.28:0/3775437572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cb0105d30 0x7f4cb019a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.602+0000 7f4cb5600640 1 -- 172.21.15.28:0/3775437572 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cb019af70 con 0x7f4cb0100ff0 2024-04-23T14:49:01.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.603+0000 7f4cb5600640 1 --2- 172.21.15.28:0/3775437572 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cb0100ff0 0x7f4cb01a05f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f4ca402a9f0 tx=0x7f4ca4003b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:01.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.603+0000 7f4cae000640 1 -- 172.21.15.28:0/3775437572 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca403f070 con 0x7f4cb0100ff0 2024-04-23T14:49:01.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.603+0000 7f4cb6a00640 1 -- 172.21.15.28:0/3775437572 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cb019b200 con 0x7f4cb0100ff0 2024-04-23T14:49:01.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.603+0000 7f4cae000640 1 -- 172.21.15.28:0/3775437572 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ca4031c90 con 0x7f4cb0100ff0 2024-04-23T14:49:01.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.603+0000 7f4cae000640 1 -- 172.21.15.28:0/3775437572 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca4035030 con 0x7f4cb0100ff0 2024-04-23T14:49:01.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.603+0000 7f4cb6a00640 1 -- 172.21.15.28:0/3775437572 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cb01a7390 con 0x7f4cb0100ff0 2024-04-23T14:49:01.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.604+0000 7f4cb6a00640 1 -- 172.21.15.28:0/3775437572 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cb0102460 con 0x7f4cb0100ff0 2024-04-23T14:49:01.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.607+0000 7f4cae000640 1 -- 172.21.15.28:0/3775437572 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f4ca4042070 con 0x7f4cb0100ff0 2024-04-23T14:49:01.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.608+0000 7f4cae000640 1 --2- 172.21.15.28:0/3775437572 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f4c980a3d20 0x7f4c980a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:01.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.608+0000 7f4cae000640 1 -- 172.21.15.28:0/3775437572 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f4ca40f1150 con 0x7f4cb0100ff0 2024-04-23T14:49:01.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.608+0000 7f4cb4c00640 1 --2- 172.21.15.28:0/3775437572 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f4c980a3d20 0x7f4c980a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:01.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.609+0000 7f4cb4c00640 1 --2- 172.21.15.28:0/3775437572 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f4c980a3d20 0x7f4c980a61e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4cb019bf00 tx=0x7f4c9c005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:01.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.613+0000 7f4cae000640 1 -- 172.21.15.28:0/3775437572 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4ca40bd1b0 con 0x7f4cb0100ff0 2024-04-23T14:49:01.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.872+0000 7f4cb6a00640 1 -- 172.21.15.28:0/3775437572 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4cb0104060 con 0x7f4cb0100ff0 2024-04-23T14:49:01.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.873+0000 7f4cae000640 1 -- 172.21.15.28:0/3775437572 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7f4ca40c14b0 con 0x7f4cb0100ff0 2024-04-23T14:49:01.875 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:49:01.875 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:49:01.875 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:49:01.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.876+0000 7f4cb6a00640 1 -- 172.21.15.28:0/3775437572 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f4c980a3d20 msgr2=0x7f4c980a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:01.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.876+0000 7f4cb6a00640 1 --2- 172.21.15.28:0/3775437572 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f4c980a3d20 0x7f4c980a61e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4cb019bf00 tx=0x7f4c9c005f90 comp rx=0 tx=0).stop 2024-04-23T14:49:01.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.876+0000 7f4cb6a00640 1 -- 172.21.15.28:0/3775437572 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cb0100ff0 msgr2=0x7f4cb01a05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:01.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.876+0000 7f4cb6a00640 1 --2- 172.21.15.28:0/3775437572 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cb0100ff0 0x7f4cb01a05f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f4ca402a9f0 tx=0x7f4ca4003b20 comp rx=0 tx=0).stop 2024-04-23T14:49:01.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.876+0000 7f4cb6a00640 1 -- 172.21.15.28:0/3775437572 shutdown_connections 2024-04-23T14:49:01.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.876+0000 7f4cb6a00640 1 --2- 172.21.15.28:0/3775437572 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f4c980a3d20 0x7f4c980a61e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.876+0000 7f4cb6a00640 1 --2- 172.21.15.28:0/3775437572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cb0105d30 0x7f4cb019a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.876+0000 7f4cb6a00640 1 --2- 172.21.15.28:0/3775437572 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cb01019b0 0x7f4cb01a0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.876+0000 7f4cb6a00640 1 --2- 172.21.15.28:0/3775437572 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cb0100ff0 0x7f4cb01a05f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:01.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.876+0000 7f4cb6a00640 1 -- 172.21.15.28:0/3775437572 >> 172.21.15.28:0/3775437572 conn(0x7f4cb0076840 msgr2=0x7f4cb010c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:01.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.876+0000 7f4cb6a00640 1 -- 172.21.15.28:0/3775437572 shutdown_connections 2024-04-23T14:49:01.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:01.876+0000 7f4cb6a00640 1 -- 172.21.15.28:0/3775437572 wait complete. 2024-04-23T14:49:01.898 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-23T14:49:02.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.112+0000 7fee63e00640 1 -- 172.21.15.28:0/2845217610 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee64106250 msgr2=0x7fee641066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.112+0000 7fee63e00640 1 --2- 172.21.15.28:0/2845217610 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee64106250 0x7fee641066e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fee5c009860 tx=0x7fee5c02a3c0 comp rx=0 tx=0).stop 2024-04-23T14:49:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.112+0000 7fee63e00640 1 -- 172.21.15.28:0/2845217610 shutdown_connections 2024-04-23T14:49:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.112+0000 7fee63e00640 1 --2- 172.21.15.28:0/2845217610 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee64106250 0x7fee641066e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.112+0000 7fee63e00640 1 --2- 172.21.15.28:0/2845217610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee64101770 0x7fee64105c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.112+0000 7fee63e00640 1 --2- 172.21.15.28:0/2845217610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee64100e50 0x7fee64101230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.112+0000 7fee63e00640 1 -- 172.21.15.28:0/2845217610 >> 172.21.15.28:0/2845217610 conn(0x7fee640fc880 msgr2=0x7fee640feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.112+0000 7fee63e00640 1 -- 172.21.15.28:0/2845217610 shutdown_connections 2024-04-23T14:49:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.112+0000 7fee63e00640 1 -- 172.21.15.28:0/2845217610 wait complete. 2024-04-23T14:49:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.113+0000 7fee63e00640 1 Processor -- start 2024-04-23T14:49:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.113+0000 7fee63e00640 1 -- start start 2024-04-23T14:49:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.114+0000 7fee63e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee64100e50 0x7fee641a4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.114+0000 7fee63e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee64101770 0x7fee641a4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.114+0000 7fee63e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee64106250 0x7fee6419eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.114+0000 7fee63e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fee64118850 con 0x7fee64106250 2024-04-23T14:49:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.114+0000 7fee63e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fee641186d0 con 0x7fee64100e50 2024-04-23T14:49:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.114+0000 7fee63e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fee641189d0 con 0x7fee64101770 2024-04-23T14:49:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.114+0000 7fee62a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee64100e50 0x7fee641a4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.114+0000 7fee5a000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee64101770 0x7fee641a4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.114+0000 7fee62a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee64100e50 0x7fee641a4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:58874/0 (socket says 172.21.15.28:58874) 2024-04-23T14:49:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.114+0000 7fee62a00640 1 -- 172.21.15.28:0/3451809068 learned_addr learned my addr 172.21.15.28:0/3451809068 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.114+0000 7fee63400640 1 --2- 172.21.15.28:0/3451809068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee64106250 0x7fee6419eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.114+0000 7fee62a00640 1 -- 172.21.15.28:0/3451809068 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee64101770 msgr2=0x7fee641a4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.114+0000 7fee62a00640 1 --2- 172.21.15.28:0/3451809068 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee64101770 0x7fee641a4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.115+0000 7fee62a00640 1 -- 172.21.15.28:0/3451809068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee64106250 msgr2=0x7fee6419eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.115+0000 7fee62a00640 1 --2- 172.21.15.28:0/3451809068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee64106250 0x7fee6419eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.115+0000 7fee62a00640 1 -- 172.21.15.28:0/3451809068 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee6419f3f0 con 0x7fee64100e50 2024-04-23T14:49:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.115+0000 7fee63400640 1 --2- 172.21.15.28:0/3451809068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee64106250 0x7fee6419eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:49:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.115+0000 7fee5a000640 1 --2- 172.21.15.28:0/3451809068 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee64101770 0x7fee641a4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:49:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.115+0000 7fee62a00640 1 --2- 172.21.15.28:0/3451809068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee64100e50 0x7fee641a4a70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fee5400c420 tx=0x7fee54009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.115+0000 7fee5be00640 1 -- 172.21.15.28:0/3451809068 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee54003970 con 0x7fee64100e50 2024-04-23T14:49:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.115+0000 7fee63e00640 1 -- 172.21.15.28:0/3451809068 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee6419f6e0 con 0x7fee64100e50 2024-04-23T14:49:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.115+0000 7fee5be00640 1 -- 172.21.15.28:0/3451809068 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee54004020 con 0x7fee64100e50 2024-04-23T14:49:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.115+0000 7fee5be00640 1 -- 172.21.15.28:0/3451809068 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee5400bad0 con 0x7fee64100e50 2024-04-23T14:49:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.115+0000 7fee63e00640 1 -- 172.21.15.28:0/3451809068 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee641ab9f0 con 0x7fee64100e50 2024-04-23T14:49:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.119+0000 7fee6c9bf740 1 Processor -- start 2024-04-23T14:49:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.120+0000 7fee5be00640 1 -- 172.21.15.28:0/3451809068 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fee5401d020 con 0x7fee64100e50 2024-04-23T14:49:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.120+0000 7fee6c9bf740 1 -- start start 2024-04-23T14:49:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.120+0000 7fee6c9bf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aab5d9db40 0x55aab5d9df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.120+0000 7fee6c9bf740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aab5d9e460 0x55aab5da85f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.120+0000 7fee6c9bf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aab5da8b30 0x55aab5daaf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.120+0000 7fee6c9bf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55aab5d031d0 con 0x55aab5da8b30 2024-04-23T14:49:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.120+0000 7fee6c9bf740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55aab5ca9450 con 0x55aab5d9e460 2024-04-23T14:49:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.120+0000 7fee6c9bf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55aab5cff190 con 0x55aab5d9db40 2024-04-23T14:49:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.121+0000 7fee63400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aab5d9db40 0x55aab5d9df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.121+0000 7fee62a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aab5d9e460 0x55aab5da85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.121+0000 7fee63400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aab5d9db40 0x55aab5d9df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:47342/0 (socket says 172.21.15.28:47342) 2024-04-23T14:49:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.121+0000 7fee63400640 1 -- 172.21.15.28:0/2886467308 learned_addr learned my addr 172.21.15.28:0/2886467308 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.121+0000 7fee5a000640 1 --2- 172.21.15.28:0/2886467308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aab5da8b30 0x55aab5daaf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.121+0000 7fee63400640 1 -- 172.21.15.28:0/2886467308 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aab5d9e460 msgr2=0x55aab5da85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.121+0000 7fee63400640 1 --2- 172.21.15.28:0/2886467308 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aab5d9e460 0x55aab5da85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.121+0000 7fee63400640 1 -- 172.21.15.28:0/2886467308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aab5da8b30 msgr2=0x55aab5daaf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.121+0000 7fee63400640 1 --2- 172.21.15.28:0/2886467308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aab5da8b30 0x55aab5daaf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.121+0000 7fee63400640 1 -- 172.21.15.28:0/2886467308 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aab5d66310 con 0x55aab5d9db40 2024-04-23T14:49:02.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.121+0000 7fee63400640 1 --2- 172.21.15.28:0/2886467308 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aab5d9db40 0x55aab5d9df20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fee5c009860 tx=0x7fee5c005dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2494c196b248479 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:02.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.122+0000 7fee53e00640 1 -- 172.21.15.28:0/2886467308 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee5c037040 con 0x55aab5d9db40 2024-04-23T14:49:02.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.122+0000 7fee53e00640 1 -- 172.21.15.28:0/2886467308 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee5c041070 con 0x55aab5d9db40 2024-04-23T14:49:02.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.122+0000 7fee53e00640 1 -- 172.21.15.28:0/2886467308 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee5c004b50 con 0x55aab5d9db40 2024-04-23T14:49:02.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.122+0000 7fee5be00640 1 --2- 172.21.15.28:0/3451809068 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fee480a3d20 0x7fee480a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:02.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.122+0000 7fee5be00640 1 -- 172.21.15.28:0/3451809068 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fee540d0760 con 0x7fee64100e50 2024-04-23T14:49:02.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.122+0000 7fee62a00640 1 --2- 172.21.15.28:0/3451809068 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fee480a3d20 0x7fee480a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:02.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.123+0000 7fee6c9bf740 1 -- 172.21.15.28:0/2886467308 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aab5d9db40 msgr2=0x55aab5d9df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:02.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.123+0000 7fee6c9bf740 1 --2- 172.21.15.28:0/2886467308 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aab5d9db40 0x55aab5d9df20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fee5c009860 tx=0x7fee5c005dc0 comp rx=0 tx=0).stop 2024-04-23T14:49:02.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.123+0000 7fee6c9bf740 1 -- 172.21.15.28:0/2886467308 shutdown_connections 2024-04-23T14:49:02.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.123+0000 7fee6c9bf740 1 --2- 172.21.15.28:0/2886467308 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aab5d9e460 0x55aab5da85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:02.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.123+0000 7fee6c9bf740 1 --2- 172.21.15.28:0/2886467308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aab5da8b30 0x55aab5daaf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:02.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.123+0000 7fee6c9bf740 1 --2- 172.21.15.28:0/2886467308 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aab5d9db40 0x55aab5d9df20 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:02.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.123+0000 7fee6c9bf740 1 -- 172.21.15.28:0/2886467308 >> 172.21.15.28:0/2886467308 conn(0x55aab5d99e60 msgr2=0x55aab5d9a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:02.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.123+0000 7fee6c9bf740 1 -- 172.21.15.28:0/2886467308 shutdown_connections 2024-04-23T14:49:02.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.123+0000 7fee6c9bf740 1 -- 172.21.15.28:0/2886467308 wait complete. 2024-04-23T14:49:02.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.123+0000 7fee62a00640 1 --2- 172.21.15.28:0/3451809068 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fee480a3d20 0x7fee480a61e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x55aab5d9f100 tx=0x7fee540d6530 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:02.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.124+0000 7fee6c9bf740 1 -- start start 2024-04-23T14:49:02.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.124+0000 7fee6c9bf740 1 Processor -- start 2024-04-23T14:49:02.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.124+0000 7fee6c9bf740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aab5d9db40 0x55aab5da58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:02.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.124+0000 7fee6c9bf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aab5d9e460 0x55aab5da5e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:02.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.124+0000 7fee6c9bf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aab5da8b30 0x55aab5da6370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:02.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.125+0000 7fee6c9bf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55aab5dab960 con 0x55aab5d9e460 2024-04-23T14:49:02.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.125+0000 7fee6c9bf740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55aab5cbc810 con 0x55aab5d9db40 2024-04-23T14:49:02.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.125+0000 7fee6c9bf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55aab5dabae0 con 0x55aab5da8b30 2024-04-23T14:49:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.125+0000 7fee6c9bf740 10 client.0 _ll_register_callbacks cb 0x55aab5c8a140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:49:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.125+0000 7fee5a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aab5d9e460 0x55aab5da5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.125+0000 7fee5a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aab5d9e460 0x55aab5da5e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34444/0 (socket says 172.21.15.28:34444) 2024-04-23T14:49:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.125+0000 7fee5a000640 1 -- 172.21.15.28:0/2051086698 learned_addr learned my addr 172.21.15.28:0/2051086698 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.125+0000 7fee63400640 1 --2- 172.21.15.28:0/2051086698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aab5da8b30 0x55aab5da6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.125+0000 7fee63400640 1 --2- 172.21.15.28:0/2051086698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aab5d9db40 0x55aab5da58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.126+0000 7fee5a000640 1 -- 172.21.15.28:0/2051086698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aab5da8b30 msgr2=0x55aab5da6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.126+0000 7fee5a000640 1 --2- 172.21.15.28:0/2051086698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aab5da8b30 0x55aab5da6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.126+0000 7fee5a000640 1 -- 172.21.15.28:0/2051086698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aab5d9db40 msgr2=0x55aab5da58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.126+0000 7fee5a000640 1 --2- 172.21.15.28:0/2051086698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aab5d9db40 0x55aab5da58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.126+0000 7fee5a000640 1 -- 172.21.15.28:0/2051086698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aab5ccd3f0 con 0x55aab5d9e460 2024-04-23T14:49:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.126+0000 7fee63400640 1 --2- 172.21.15.28:0/2051086698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aab5d9db40 0x55aab5da58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:49:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.126+0000 7fee5a000640 1 --2- 172.21.15.28:0/2051086698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aab5d9e460 0x55aab5da5e30 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fee4c0129d0 tx=0x7fee4c012e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9512bd7faeb3e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:02.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.126+0000 7fee50c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-23T14:49:02.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.127+0000 7fee50c00640 1 -- 172.21.15.28:0/2051086698 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4c00fa40 con 0x55aab5d9e460 2024-04-23T14:49:02.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.127+0000 7fee50c00640 1 -- 172.21.15.28:0/2051086698 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fee4c00fbe0 con 0x55aab5d9e460 2024-04-23T14:49:02.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.127+0000 7fee6c9bf740 1 -- 172.21.15.28:0/2051086698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee5c041070 con 0x55aab5d9e460 2024-04-23T14:49:02.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.127+0000 7fee50c00640 1 -- 172.21.15.28:0/2051086698 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fee4c00fde0 con 0x55aab5d9e460 2024-04-23T14:49:02.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.127+0000 7fee6c9bf740 10 client.15867 fetch_fsmap learned FSMap version 20 2024-04-23T14:49:02.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.127+0000 7fee6c9bf740 1 -- 172.21.15.28:0/2051086698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55aab5a153f0 con 0x55aab5d9e460 2024-04-23T14:49:02.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.128+0000 7fee50c00640 1 -- 172.21.15.28:0/2051086698 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee5c041070 con 0x55aab5d9e460 2024-04-23T14:49:02.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.128+0000 7fee50c00640 1 -- 172.21.15.28:0/2051086698 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fee4c025470 con 0x55aab5d9e460 2024-04-23T14:49:02.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.128+0000 7fee6c9bf740 10 client.15867 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:49:02.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.128+0000 7fee6c9bf740 10 client.15867 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:49:02.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.128+0000 7fee6c9bf740 20 client.15867 populate_metadata read hostname 'smithi028' 2024-04-23T14:49:02.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.128+0000 7fee6c9bf740 1 --2- 172.21.15.28:0/2051086698 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55aab5db8cd0 0x55aab5dbb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:02.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.128+0000 7fee6c9bf740 4 client.15867 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:49:02.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.128+0000 7fee6c9bf740 1 -- 172.21.15.28:0/2051086698 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aab5d10bf0 con 0x55aab5db8cd0 2024-04-23T14:49:02.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.128+0000 7fee63400640 1 --2- 172.21.15.28:0/2051086698 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55aab5db8cd0 0x55aab5dbb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:02.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.129+0000 7fee63400640 1 --2- 172.21.15.28:0/2051086698 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55aab5db8cd0 0x55aab5dbb0c0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d32f54d444359a server_cookie=690cb0415bde2b6f in_seq=0 out_seq=0 2024-04-23T14:49:02.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.129+0000 7fee50c00640 10 client.15867 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:49:02.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.130+0000 7fee50c00640 1 -- 172.21.15.28:0/2051086698 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55aab5d10bf0 con 0x55aab5db8cd0 2024-04-23T14:49:02.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.130+0000 7fee50c00640 10 client.15867 handle_command_reply: tid=0 2024-04-23T14:49:02.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.131+0000 7fee6c9bf740 1 client.15867 shutdown 2024-04-23T14:49:02.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.131+0000 7fee6c9bf740 1 -- 172.21.15.28:0/2051086698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aab5d9e460 msgr2=0x55aab5da5e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:02.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.131+0000 7fee6c9bf740 1 --2- 172.21.15.28:0/2051086698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aab5d9e460 0x55aab5da5e30 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fee4c0129d0 tx=0x7fee4c012e90 comp rx=0 tx=0).stop 2024-04-23T14:49:02.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.131+0000 7fee6c9bf740 1 -- 172.21.15.28:0/2051086698 shutdown_connections 2024-04-23T14:49:02.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.131+0000 7fee6c9bf740 1 --2- 172.21.15.28:0/2051086698 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55aab5db8cd0 0x55aab5dbb0c0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:02.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.131+0000 7fee6c9bf740 1 --2- 172.21.15.28:0/2051086698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aab5da8b30 0x55aab5da6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:02.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.131+0000 7fee6c9bf740 1 --2- 172.21.15.28:0/2051086698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aab5d9e460 0x55aab5da5e30 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:02.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.131+0000 7fee6c9bf740 1 --2- 172.21.15.28:0/2051086698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aab5d9db40 0x55aab5da58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:02.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.131+0000 7fee50c00640 0 client.15867 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:49:02.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.131+0000 7fee6c9bf740 1 -- 172.21.15.28:0/2051086698 >> 172.21.15.28:0/2051086698 conn(0x55aab5d99e60 msgr2=0x55aab5d9fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:02.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.131+0000 7fee6c9bf740 1 -- 172.21.15.28:0/2051086698 shutdown_connections 2024-04-23T14:49:02.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.131+0000 7fee6c9bf740 1 -- 172.21.15.28:0/2051086698 wait complete. 2024-04-23T14:49:02.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.131+0000 7fee6c9bf740 20 client.15867 trim_cache size 0 max 16384 2024-04-23T14:49:02.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.147+0000 7fee6c9bf740 1 Processor -- start 2024-04-23T14:49:02.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.147+0000 7fee6c9bf740 1 -- start start 2024-04-23T14:49:02.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.147+0000 7fee6c9bf740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aab5d9db40 0x55aab5da02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:02.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.147+0000 7fee6c9bf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aab5d9e460 0x55aab5db4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:02.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.147+0000 7fee6c9bf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aab5da8b30 0x55aab5da4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:02.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.147+0000 7fee6c9bf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fee641b21b0 con 0x55aab5d9e460 2024-04-23T14:49:02.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.147+0000 7fee6c9bf740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fee641b4a30 con 0x55aab5d9db40 2024-04-23T14:49:02.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.147+0000 7fee6c9bf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fee641b2820 con 0x55aab5da8b30 2024-04-23T14:49:02.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.147+0000 7fee63400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aab5d9db40 0x55aab5da02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:02.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.147+0000 7fee5a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aab5d9e460 0x55aab5db4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:02.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.148+0000 7fee5a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aab5d9e460 0x55aab5db4820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34454/0 (socket says 172.21.15.28:34454) 2024-04-23T14:49:02.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.147+0000 7fee63400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aab5d9db40 0x55aab5da02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:58904/0 (socket says 172.21.15.28:58904) 2024-04-23T14:49:02.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.148+0000 7fee5a000640 1 -- 172.21.15.28:0/1532792877 learned_addr learned my addr 172.21.15.28:0/1532792877 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:02.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.148+0000 7fee63400640 1 --2- 172.21.15.28:0/1532792877 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aab5da8b30 0x55aab5da4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:02.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.148+0000 7fee5a000640 1 -- 172.21.15.28:0/1532792877 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aab5da8b30 msgr2=0x55aab5da4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:02.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.148+0000 7fee5a000640 1 --2- 172.21.15.28:0/1532792877 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aab5da8b30 0x55aab5da4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:02.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.148+0000 7fee5a000640 1 -- 172.21.15.28:0/1532792877 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aab5d9db40 msgr2=0x55aab5da02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:02.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.148+0000 7fee5a000640 1 --2- 172.21.15.28:0/1532792877 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aab5d9db40 0x55aab5da02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:02.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.148+0000 7fee5a000640 1 -- 172.21.15.28:0/1532792877 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aab5adf7d0 con 0x55aab5d9e460 2024-04-23T14:49:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.148+0000 7fee63400640 1 --2- 172.21.15.28:0/1532792877 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aab5da8b30 0x55aab5da4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:49:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.148+0000 7fee5a000640 1 --2- 172.21.15.28:0/1532792877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aab5d9e460 0x55aab5db4820 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fee4c025a00 tx=0x7fee4c012520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5211d47058de54cd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.149+0000 7fee53400640 1 -- 172.21.15.28:0/1532792877 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4c025470 con 0x55aab5d9e460 2024-04-23T14:49:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.149+0000 7fee53400640 1 -- 172.21.15.28:0/1532792877 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee4c025610 con 0x55aab5d9e460 2024-04-23T14:49:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.149+0000 7fee53400640 1 -- 172.21.15.28:0/1532792877 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4c005410 con 0x55aab5d9e460 2024-04-23T14:49:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.150+0000 7fee6c9bf740 1 -- 172.21.15.28:0/1532792877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aab5d9e460 msgr2=0x55aab5db4820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.150+0000 7fee6c9bf740 1 --2- 172.21.15.28:0/1532792877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aab5d9e460 0x55aab5db4820 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fee4c025a00 tx=0x7fee4c012520 comp rx=0 tx=0).stop 2024-04-23T14:49:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.150+0000 7fee6c9bf740 1 -- 172.21.15.28:0/1532792877 shutdown_connections 2024-04-23T14:49:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.150+0000 7fee6c9bf740 1 --2- 172.21.15.28:0/1532792877 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aab5da8b30 0x55aab5da4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.150+0000 7fee6c9bf740 1 --2- 172.21.15.28:0/1532792877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aab5d9e460 0x55aab5db4820 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.150+0000 7fee6c9bf740 1 --2- 172.21.15.28:0/1532792877 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aab5d9db40 0x55aab5da02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.150+0000 7fee6c9bf740 1 -- 172.21.15.28:0/1532792877 >> 172.21.15.28:0/1532792877 conn(0x55aab5d99e60 msgr2=0x55aab5d986c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.150+0000 7fee6c9bf740 1 -- 172.21.15.28:0/1532792877 shutdown_connections 2024-04-23T14:49:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.150+0000 7fee6c9bf740 1 -- 172.21.15.28:0/1532792877 wait complete. 2024-04-23T14:49:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.150+0000 7fee6c9bf740 1 -- start start 2024-04-23T14:49:02.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.150+0000 7fee6c9bf740 1 Processor -- start 2024-04-23T14:49:02.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.151+0000 7fee6c9bf740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aab5d9db40 0x55aab5da9320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:02.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.151+0000 7fee6c9bf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aab5d9e460 0x55aab5da9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:02.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.151+0000 7fee6c9bf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aab5da8b30 0x55aab5dafd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:02.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.151+0000 7fee6c9bf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fee641b2ef0 con 0x55aab5d9e460 2024-04-23T14:49:02.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.151+0000 7fee6c9bf740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fee641b4360 con 0x55aab5d9db40 2024-04-23T14:49:02.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.151+0000 7fee6c9bf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fee641b35c0 con 0x55aab5da8b30 2024-04-23T14:49:02.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.151+0000 7fee6c9bf740 10 client.0 _ll_register_callbacks cb 0x7fee4c009c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:49:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.151+0000 7fee5a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aab5d9e460 0x55aab5da9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.151+0000 7fee63400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aab5d9db40 0x55aab5da9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.151+0000 7fee5a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aab5d9e460 0x55aab5da9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34458/0 (socket says 172.21.15.28:34458) 2024-04-23T14:49:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.151+0000 7fee5a000640 1 -- 172.21.15.28:0/895545720 learned_addr learned my addr 172.21.15.28:0/895545720 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.151+0000 7fee63400640 1 --2- 172.21.15.28:0/895545720 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aab5da8b30 0x55aab5dafd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.152+0000 7fee5a000640 1 -- 172.21.15.28:0/895545720 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aab5da8b30 msgr2=0x55aab5dafd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.152+0000 7fee5a000640 1 --2- 172.21.15.28:0/895545720 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aab5da8b30 0x55aab5dafd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.152+0000 7fee5a000640 1 -- 172.21.15.28:0/895545720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aab5d9db40 msgr2=0x55aab5da9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.152+0000 7fee5a000640 1 --2- 172.21.15.28:0/895545720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aab5d9db40 0x55aab5da9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.152+0000 7fee5a000640 1 -- 172.21.15.28:0/895545720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aab5db0270 con 0x55aab5d9e460 2024-04-23T14:49:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.152+0000 7fee63400640 1 --2- 172.21.15.28:0/895545720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aab5d9db40 0x55aab5da9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:49:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.152+0000 7fee5a000640 1 --2- 172.21.15.28:0/895545720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aab5d9e460 0x55aab5da9860 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fee4c012420 tx=0x7fee4c012680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df7ec602f9fc3b9b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.152+0000 7fee50c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-23T14:49:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.152+0000 7fee50c00640 1 -- 172.21.15.28:0/895545720 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4c025470 con 0x55aab5d9e460 2024-04-23T14:49:02.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.152+0000 7fee6c9bf740 1 -- 172.21.15.28:0/895545720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee4c025610 con 0x55aab5d9e460 2024-04-23T14:49:02.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.152+0000 7fee50c00640 1 -- 172.21.15.28:0/895545720 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fee4c025a80 con 0x55aab5d9e460 2024-04-23T14:49:02.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.152+0000 7fee50c00640 1 -- 172.21.15.28:0/895545720 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fee4c03d050 con 0x55aab5d9e460 2024-04-23T14:49:02.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.152+0000 7fee6c9bf740 10 client.15873 fetch_fsmap learned FSMap version 20 2024-04-23T14:49:02.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.153+0000 7fee6c9bf740 1 -- 172.21.15.28:0/895545720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55aab5db1570 con 0x55aab5d9e460 2024-04-23T14:49:02.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.153+0000 7fee50c00640 1 -- 172.21.15.28:0/895545720 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee4c025610 con 0x55aab5d9e460 2024-04-23T14:49:02.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.153+0000 7fee50c00640 1 -- 172.21.15.28:0/895545720 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fee4c010430 con 0x55aab5d9e460 2024-04-23T14:49:02.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.153+0000 7fee6c9bf740 10 client.15873 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:49:02.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.153+0000 7fee6c9bf740 10 client.15873 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:49:02.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.153+0000 7fee6c9bf740 20 client.15873 populate_metadata read hostname 'smithi028' 2024-04-23T14:49:02.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.153+0000 7fee6c9bf740 1 --2- 172.21.15.28:0/895545720 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55aab5e1f5e0 0x55aab5e21a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:02.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.153+0000 7fee6c9bf740 4 client.15873 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-23T14:49:02.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.153+0000 7fee6c9bf740 1 -- 172.21.15.28:0/895545720 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55aab5d0aa60 con 0x55aab5e1f5e0 2024-04-23T14:49:02.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.153+0000 7fee63400640 1 --2- 172.21.15.28:0/895545720 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55aab5e1f5e0 0x55aab5e21a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:02.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.154+0000 7fee63400640 1 --2- 172.21.15.28:0/895545720 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55aab5e1f5e0 0x55aab5e21a00 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92f5bfd6bc36db70 server_cookie=e22b5bcf9087729c in_seq=0 out_seq=0 2024-04-23T14:49:02.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.154+0000 7fee50c00640 10 client.15873 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:49:02.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.154+0000 7fee50c00640 1 -- 172.21.15.28:0/895545720 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55aab5d0aa60 con 0x55aab5e1f5e0 2024-04-23T14:49:02.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.154+0000 7fee50c00640 10 client.15873 handle_command_reply: tid=0 2024-04-23T14:49:02.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.154+0000 7fee6c9bf740 1 client.15873 shutdown 2024-04-23T14:49:02.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.154+0000 7fee6c9bf740 1 -- 172.21.15.28:0/895545720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aab5d9e460 msgr2=0x55aab5da9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:02.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.154+0000 7fee6c9bf740 1 --2- 172.21.15.28:0/895545720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aab5d9e460 0x55aab5da9860 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fee4c012420 tx=0x7fee4c012680 comp rx=0 tx=0).stop 2024-04-23T14:49:02.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.154+0000 7fee6c9bf740 1 -- 172.21.15.28:0/895545720 shutdown_connections 2024-04-23T14:49:02.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.154+0000 7fee6c9bf740 1 --2- 172.21.15.28:0/895545720 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55aab5e1f5e0 0x55aab5e21a00 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:02.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.154+0000 7fee6c9bf740 1 --2- 172.21.15.28:0/895545720 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aab5da8b30 0x55aab5dafd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:02.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.154+0000 7fee6c9bf740 1 --2- 172.21.15.28:0/895545720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aab5d9e460 0x55aab5da9860 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:02.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.154+0000 7fee6c9bf740 1 --2- 172.21.15.28:0/895545720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aab5d9db40 0x55aab5da9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:02.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.154+0000 7fee50c00640 0 client.15873 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:49:02.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.154+0000 7fee6c9bf740 1 -- 172.21.15.28:0/895545720 >> 172.21.15.28:0/895545720 conn(0x55aab5d99e60 msgr2=0x55aab5d986c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:02.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.154+0000 7fee6c9bf740 1 -- 172.21.15.28:0/895545720 shutdown_connections 2024-04-23T14:49:02.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.155+0000 7fee6c9bf740 1 -- 172.21.15.28:0/895545720 wait complete. 2024-04-23T14:49:02.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.155+0000 7fee6c9bf740 20 client.15873 trim_cache size 0 max 16384 2024-04-23T14:49:02.160 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-23T14:49:02.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.155+0000 7fee63e00640 1 -- 172.21.15.28:0/3451809068 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fee480a3d20 msgr2=0x7fee480a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:02.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.155+0000 7fee63e00640 1 --2- 172.21.15.28:0/3451809068 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fee480a3d20 0x7fee480a61e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x55aab5d9f100 tx=0x7fee540d6530 comp rx=0 tx=0).stop 2024-04-23T14:49:02.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.155+0000 7fee63e00640 1 -- 172.21.15.28:0/3451809068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee64100e50 msgr2=0x7fee641a4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:02.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.155+0000 7fee63e00640 1 --2- 172.21.15.28:0/3451809068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee64100e50 0x7fee641a4a70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fee5400c420 tx=0x7fee54009f90 comp rx=0 tx=0).stop 2024-04-23T14:49:02.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.155+0000 7fee63e00640 1 -- 172.21.15.28:0/3451809068 shutdown_connections 2024-04-23T14:49:02.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.155+0000 7fee63e00640 1 --2- 172.21.15.28:0/3451809068 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fee480a3d20 0x7fee480a61e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:02.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.155+0000 7fee63e00640 1 --2- 172.21.15.28:0/3451809068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee64106250 0x7fee6419eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:02.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.155+0000 7fee63e00640 1 --2- 172.21.15.28:0/3451809068 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee64101770 0x7fee641a4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:02.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.155+0000 7fee63e00640 1 --2- 172.21.15.28:0/3451809068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee64100e50 0x7fee641a4a70 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:02.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.155+0000 7fee63e00640 1 -- 172.21.15.28:0/3451809068 >> 172.21.15.28:0/3451809068 conn(0x7fee640fc880 msgr2=0x7fee640fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:02.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.156+0000 7fee63e00640 1 -- 172.21.15.28:0/3451809068 shutdown_connections 2024-04-23T14:49:02.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:02.156+0000 7fee63e00640 1 -- 172.21.15.28:0/3451809068 wait complete. 2024-04-23T14:49:02.486 INFO:tasks.workunit.client.0.smithi028.stdout: 1 374809 9.67 MB/sec execute 78 sec latency 9.412 ms 2024-04-23T14:49:02.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:02 smithi038 ceph-mon[32348]: pgmap v153: 97 pgs: 97 active+clean; 206 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 974 KiB/s rd, 2.5 MiB/s wr, 150 op/s 2024-04-23T14:49:02.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:02 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3775437572' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:49:02.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:02 smithi028 ceph-mon[27810]: pgmap v153: 97 pgs: 97 active+clean; 206 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 974 KiB/s rd, 2.5 MiB/s wr, 150 op/s 2024-04-23T14:49:02.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:02 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3775437572' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:49:03.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:02 smithi105 ceph-mon[32555]: pgmap v153: 97 pgs: 97 active+clean; 206 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 974 KiB/s rd, 2.5 MiB/s wr, 150 op/s 2024-04-23T14:49:03.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:02 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3775437572' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:49:03.175 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-23T14:49:03.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:49:03.175 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-23T14:49:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.394+0000 7fb2ada00640 1 -- 172.21.15.28:0/3027761177 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2a8100da0 msgr2=0x7fb2a8106d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.394+0000 7fb2ada00640 1 --2- 172.21.15.28:0/3027761177 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2a8100da0 0x7fb2a8106d60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb294009900 tx=0x7fb29402a400 comp rx=0 tx=0).stop 2024-04-23T14:49:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.394+0000 7fb2ada00640 1 -- 172.21.15.28:0/3027761177 shutdown_connections 2024-04-23T14:49:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.394+0000 7fb2ada00640 1 --2- 172.21.15.28:0/3027761177 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2a8100da0 0x7fb2a8106d60 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.394+0000 7fb2ada00640 1 --2- 172.21.15.28:0/3027761177 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb2a8107490 0x7fb2a810ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.394+0000 7fb2ada00640 1 --2- 172.21.15.28:0/3027761177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2a81003f0 0x7fb2a81007d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.394+0000 7fb2ada00640 1 -- 172.21.15.28:0/3027761177 >> 172.21.15.28:0/3027761177 conn(0x7fb2a8077060 msgr2=0x7fb2a80fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.394+0000 7fb2ada00640 1 -- 172.21.15.28:0/3027761177 shutdown_connections 2024-04-23T14:49:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.394+0000 7fb2ada00640 1 -- 172.21.15.28:0/3027761177 wait complete. 2024-04-23T14:49:03.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.395+0000 7fb2ada00640 1 Processor -- start 2024-04-23T14:49:03.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.396+0000 7fb2ada00640 1 -- start start 2024-04-23T14:49:03.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.396+0000 7fb2ada00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2a81003f0 0x7fb2a81a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.396+0000 7fb2ada00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb2a8107490 0x7fb2a81a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.396+0000 7fb2ada00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2a819a980 0x7fb2a819ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.396+0000 7fb2ada00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a8112460 con 0x7fb2a81003f0 2024-04-23T14:49:03.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.396+0000 7fb2ada00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a81122e0 con 0x7fb2a819a980 2024-04-23T14:49:03.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.396+0000 7fb2ada00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a81125e0 con 0x7fb2a8107490 2024-04-23T14:49:03.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.396+0000 7fb2a7400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2a819a980 0x7fb2a819ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.396+0000 7fb2a7400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2a819a980 0x7fb2a819ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:58924/0 (socket says 172.21.15.28:58924) 2024-04-23T14:49:03.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.396+0000 7fb2a7400640 1 -- 172.21.15.28:0/832065686 learned_addr learned my addr 172.21.15.28:0/832065686 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:03.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.396+0000 7fb2a6a00640 1 --2- 172.21.15.28:0/832065686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2a81003f0 0x7fb2a81a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.397+0000 7fb2a6000640 1 --2- 172.21.15.28:0/832065686 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb2a8107490 0x7fb2a81a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.397+0000 7fb2a7400640 1 -- 172.21.15.28:0/832065686 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb2a8107490 msgr2=0x7fb2a81a0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:03.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.397+0000 7fb2a7400640 1 --2- 172.21.15.28:0/832065686 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb2a8107490 0x7fb2a81a0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.397+0000 7fb2a7400640 1 -- 172.21.15.28:0/832065686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2a81003f0 msgr2=0x7fb2a81a07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:03.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.397+0000 7fb2a7400640 1 --2- 172.21.15.28:0/832065686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2a81003f0 0x7fb2a81a07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.397+0000 7fb2a7400640 1 -- 172.21.15.28:0/832065686 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2a819b4a0 con 0x7fb2a819a980 2024-04-23T14:49:03.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.397+0000 7fb2a7400640 1 --2- 172.21.15.28:0/832065686 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2a819a980 0x7fb2a819ae30 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb2a000ea60 tx=0x7fb2a000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:03.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.397+0000 7fb29f400640 1 -- 172.21.15.28:0/832065686 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a000ce40 con 0x7fb2a819a980 2024-04-23T14:49:03.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.398+0000 7fb29f400640 1 -- 172.21.15.28:0/832065686 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2a0005020 con 0x7fb2a819a980 2024-04-23T14:49:03.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.398+0000 7fb29f400640 1 -- 172.21.15.28:0/832065686 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a0010690 con 0x7fb2a819a980 2024-04-23T14:49:03.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.398+0000 7fb2ada00640 1 -- 172.21.15.28:0/832065686 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2a819b790 con 0x7fb2a819a980 2024-04-23T14:49:03.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.398+0000 7fb2ada00640 1 -- 172.21.15.28:0/832065686 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2a81a79e0 con 0x7fb2a819a980 2024-04-23T14:49:03.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.402+0000 7fb2b0841740 1 Processor -- start 2024-04-23T14:49:03.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.402+0000 7fb29f400640 1 -- 172.21.15.28:0/832065686 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fb2a0005520 con 0x7fb2a819a980 2024-04-23T14:49:03.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.403+0000 7fb2b0841740 1 -- start start 2024-04-23T14:49:03.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.403+0000 7fb2b0841740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b54dea1090 0x55b54dea1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.403+0000 7fb2b0841740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b54dea19b0 0x55b54deabb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.403+0000 7fb2b0841740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b54deac080 0x55b54deae470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.403+0000 7fb2b0841740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b54de06e70 con 0x55b54dea1090 2024-04-23T14:49:03.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.403+0000 7fb2b0841740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b54ddad380 con 0x55b54deac080 2024-04-23T14:49:03.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.403+0000 7fb2b0841740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b54de02e30 con 0x55b54dea19b0 2024-04-23T14:49:03.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.403+0000 7fb2a6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b54dea1090 0x55b54dea1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.403+0000 7fb2a6a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b54dea19b0 0x55b54deabb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.404+0000 7fb2a6a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b54dea19b0 0x55b54deabb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:47402/0 (socket says 172.21.15.28:47402) 2024-04-23T14:49:03.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.404+0000 7fb2a6a00640 1 -- 172.21.15.28:0/1614156341 learned_addr learned my addr 172.21.15.28:0/1614156341 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:03.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.404+0000 7fb2a6000640 1 --2- 172.21.15.28:0/1614156341 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b54deac080 0x55b54deae470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.404+0000 7fb2a6a00640 1 -- 172.21.15.28:0/1614156341 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b54deac080 msgr2=0x55b54deae470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:03.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.404+0000 7fb2a6a00640 1 --2- 172.21.15.28:0/1614156341 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b54deac080 0x55b54deae470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.404+0000 7fb2a6a00640 1 -- 172.21.15.28:0/1614156341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b54dea1090 msgr2=0x55b54dea1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:03.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.404+0000 7fb2a6a00640 1 --2- 172.21.15.28:0/1614156341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b54dea1090 0x55b54dea1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.404+0000 7fb2a6a00640 1 -- 172.21.15.28:0/1614156341 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b54dbe17e0 con 0x55b54dea19b0 2024-04-23T14:49:03.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.404+0000 7fb2a6a00640 1 --2- 172.21.15.28:0/1614156341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b54dea19b0 0x55b54deabb40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb298009c90 tx=0x7fb298011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=341e26f57ea79c39 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:03.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.404+0000 7fb29f400640 1 --2- 172.21.15.28:0/832065686 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb28c0a3da0 0x7fb28c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.405+0000 7fb293e00640 1 -- 172.21.15.28:0/1614156341 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb298005220 con 0x55b54dea19b0 2024-04-23T14:49:03.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.405+0000 7fb29f400640 1 -- 172.21.15.28:0/832065686 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fb2a00d0c90 con 0x7fb2a819a980 2024-04-23T14:49:03.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.405+0000 7fb293e00640 1 -- 172.21.15.28:0/1614156341 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb298004c80 con 0x55b54dea19b0 2024-04-23T14:49:03.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.405+0000 7fb293e00640 1 -- 172.21.15.28:0/1614156341 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2980197b0 con 0x55b54dea19b0 2024-04-23T14:49:03.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.405+0000 7fb2a6000640 1 --2- 172.21.15.28:0/832065686 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb28c0a3da0 0x7fb28c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.405+0000 7fb2b0841740 1 -- 172.21.15.28:0/1614156341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b54dea19b0 msgr2=0x55b54deabb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:03.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.405+0000 7fb2b0841740 1 --2- 172.21.15.28:0/1614156341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b54dea19b0 0x55b54deabb40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb298009c90 tx=0x7fb298011810 comp rx=0 tx=0).stop 2024-04-23T14:49:03.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.405+0000 7fb2a6000640 1 --2- 172.21.15.28:0/832065686 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb28c0a3da0 0x7fb28c0a6260 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb29402a920 tx=0x7fb294041040 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:03.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.406+0000 7fb2b0841740 1 -- 172.21.15.28:0/1614156341 shutdown_connections 2024-04-23T14:49:03.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.406+0000 7fb2b0841740 1 --2- 172.21.15.28:0/1614156341 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b54deac080 0x55b54deae470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.406+0000 7fb2b0841740 1 --2- 172.21.15.28:0/1614156341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b54dea19b0 0x55b54deabb40 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.406+0000 7fb2b0841740 1 --2- 172.21.15.28:0/1614156341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b54dea1090 0x55b54dea1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.406+0000 7fb2b0841740 1 -- 172.21.15.28:0/1614156341 >> 172.21.15.28:0/1614156341 conn(0x55b54de9d3b0 msgr2=0x55b54de9d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:03.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.406+0000 7fb2b0841740 1 -- 172.21.15.28:0/1614156341 shutdown_connections 2024-04-23T14:49:03.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.406+0000 7fb2b0841740 1 -- 172.21.15.28:0/1614156341 wait complete. 2024-04-23T14:49:03.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.406+0000 7fb2b0841740 1 -- start start 2024-04-23T14:49:03.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.406+0000 7fb2b0841740 1 Processor -- start 2024-04-23T14:49:03.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.407+0000 7fb2b0841740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b54dea1090 0x55b54dea8f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.407+0000 7fb2b0841740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b54dea19b0 0x55b54dea94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.407+0000 7fb2b0841740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b54deac080 0x55b54dea9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.407+0000 7fb2b0841740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b54deaee70 con 0x55b54dea19b0 2024-04-23T14:49:03.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.407+0000 7fb2b0841740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b54ddc0660 con 0x55b54dea1090 2024-04-23T14:49:03.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.407+0000 7fb2b0841740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b54deaeff0 con 0x55b54deac080 2024-04-23T14:49:03.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.407+0000 7fb2b0841740 10 client.0 _ll_register_callbacks cb 0x55b54ddaeaa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:49:03.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.407+0000 7fb2a6a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b54dea1090 0x55b54dea8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.407+0000 7fb2a6a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b54dea1090 0x55b54dea8f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:58940/0 (socket says 172.21.15.28:58940) 2024-04-23T14:49:03.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.407+0000 7fb2a6a00640 1 -- 172.21.15.28:0/3919637992 learned_addr learned my addr 172.21.15.28:0/3919637992 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:03.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.407+0000 7fb2a6000640 1 --2- 172.21.15.28:0/3919637992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b54dea19b0 0x55b54dea94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.407+0000 7fb2a7400640 1 --2- 172.21.15.28:0/3919637992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b54deac080 0x55b54dea9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.408+0000 7fb2a7400640 1 -- 172.21.15.28:0/3919637992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b54dea1090 msgr2=0x55b54dea8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:03.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.408+0000 7fb2a7400640 1 --2- 172.21.15.28:0/3919637992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b54dea1090 0x55b54dea8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.408+0000 7fb2a7400640 1 -- 172.21.15.28:0/3919637992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b54dea19b0 msgr2=0x55b54dea94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:03.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.408+0000 7fb2a7400640 1 --2- 172.21.15.28:0/3919637992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b54dea19b0 0x55b54dea94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.408+0000 7fb2a7400640 1 -- 172.21.15.28:0/3919637992 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b54dc41f70 con 0x55b54deac080 2024-04-23T14:49:03.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.408+0000 7fb2a6a00640 1 --2- 172.21.15.28:0/3919637992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b54dea1090 0x55b54dea8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:49:03.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.408+0000 7fb2a7400640 1 --2- 172.21.15.28:0/3919637992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b54deac080 0x55b54dea9a10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb2a00ce390 tx=0x7fb2a009cca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=321f2dfc718d7f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:03.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.408+0000 7fb290c00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:49:03.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.408+0000 7fb290c00640 1 -- 172.21.15.28:0/3919637992 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a00b6060 con 0x55b54deac080 2024-04-23T14:49:03.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.408+0000 7fb290c00640 1 -- 172.21.15.28:0/3919637992 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fb2a00bd070 con 0x55b54deac080 2024-04-23T14:49:03.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.408+0000 7fb2b0841740 1 -- 172.21.15.28:0/3919637992 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb298004c80 con 0x55b54deac080 2024-04-23T14:49:03.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.408+0000 7fb290c00640 1 -- 172.21.15.28:0/3919637992 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fb2a00a53c0 con 0x55b54deac080 2024-04-23T14:49:03.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.408+0000 7fb2b0841740 10 client.26023 fetch_fsmap learned FSMap version 20 2024-04-23T14:49:03.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.409+0000 7fb2b0841740 1 -- 172.21.15.28:0/3919637992 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55b54dad9170 con 0x55b54deac080 2024-04-23T14:49:03.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.409+0000 7fb290c00640 1 -- 172.21.15.28:0/3919637992 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb298004c80 con 0x55b54deac080 2024-04-23T14:49:03.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.409+0000 7fb290c00640 1 -- 172.21.15.28:0/3919637992 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fb2a009c150 con 0x55b54deac080 2024-04-23T14:49:03.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.409+0000 7fb2b0841740 10 client.26023 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:49:03.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.409+0000 7fb2b0841740 10 client.26023 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:49:03.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.409+0000 7fb2b0841740 20 client.26023 populate_metadata read hostname 'smithi028' 2024-04-23T14:49:03.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.409+0000 7fb2b0841740 1 --2- 172.21.15.28:0/3919637992 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55b54debc3e0 0x55b54debe7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.409+0000 7fb2b0841740 4 client.26023 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:49:03.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.409+0000 7fb2b0841740 1 -- 172.21.15.28:0/3919637992 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b54de14890 con 0x55b54debc3e0 2024-04-23T14:49:03.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.409+0000 7fb2a6a00640 1 --2- 172.21.15.28:0/3919637992 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55b54debc3e0 0x55b54debe7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.410+0000 7fb2a6a00640 1 --2- 172.21.15.28:0/3919637992 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55b54debc3e0 0x55b54debe7d0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fecbc963ef09df0 server_cookie=c89c03ced30f2374 in_seq=0 out_seq=0 2024-04-23T14:49:03.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.410+0000 7fb290c00640 10 client.26023 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:49:03.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.411+0000 7fb290c00640 1 -- 172.21.15.28:0/3919637992 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b54de14890 con 0x55b54debc3e0 2024-04-23T14:49:03.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.411+0000 7fb290c00640 10 client.26023 handle_command_reply: tid=0 2024-04-23T14:49:03.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.411+0000 7fb2b0841740 1 client.26023 shutdown 2024-04-23T14:49:03.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.411+0000 7fb2b0841740 1 -- 172.21.15.28:0/3919637992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b54deac080 msgr2=0x55b54dea9a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:03.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.411+0000 7fb2b0841740 1 --2- 172.21.15.28:0/3919637992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b54deac080 0x55b54dea9a10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb2a00ce390 tx=0x7fb2a009cca0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.411+0000 7fb2b0841740 1 -- 172.21.15.28:0/3919637992 shutdown_connections 2024-04-23T14:49:03.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.411+0000 7fb2b0841740 1 --2- 172.21.15.28:0/3919637992 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55b54debc3e0 0x55b54debe7d0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.411+0000 7fb2b0841740 1 --2- 172.21.15.28:0/3919637992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b54deac080 0x55b54dea9a10 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.411+0000 7fb2b0841740 1 --2- 172.21.15.28:0/3919637992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b54dea19b0 0x55b54dea94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.411+0000 7fb290c00640 0 client.26023 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:49:03.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.411+0000 7fb2b0841740 1 --2- 172.21.15.28:0/3919637992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b54dea1090 0x55b54dea8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.411+0000 7fb2b0841740 1 -- 172.21.15.28:0/3919637992 >> 172.21.15.28:0/3919637992 conn(0x55b54de9d3b0 msgr2=0x55b54dea31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:03.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.411+0000 7fb2b0841740 1 -- 172.21.15.28:0/3919637992 shutdown_connections 2024-04-23T14:49:03.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.412+0000 7fb2b0841740 1 -- 172.21.15.28:0/3919637992 wait complete. 2024-04-23T14:49:03.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.412+0000 7fb2b0841740 20 client.26023 trim_cache size 0 max 16384 2024-04-23T14:49:03.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.431+0000 7fb2b0841740 1 Processor -- start 2024-04-23T14:49:03.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.431+0000 7fb2b0841740 1 -- start start 2024-04-23T14:49:03.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.431+0000 7fb2b0841740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b54dea1090 0x55b54deb7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.431+0000 7fb2b0841740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b54dea19b0 0x55b54dea37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.431+0000 7fb2b0841740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b54deac080 0x55b54deaaed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.432+0000 7fb2b0841740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a81adf10 con 0x55b54deac080 2024-04-23T14:49:03.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.432+0000 7fb2b0841740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a81b0790 con 0x55b54dea19b0 2024-04-23T14:49:03.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.432+0000 7fb2b0841740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a81ae580 con 0x55b54dea1090 2024-04-23T14:49:03.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.432+0000 7fb2a6a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b54dea1090 0x55b54deb7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.432+0000 7fb2a6a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b54dea1090 0x55b54deb7d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:47420/0 (socket says 172.21.15.28:47420) 2024-04-23T14:49:03.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.432+0000 7fb2a6a00640 1 -- 172.21.15.28:0/3219020520 learned_addr learned my addr 172.21.15.28:0/3219020520 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:03.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.432+0000 7fb2a6000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b54dea19b0 0x55b54dea37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.432+0000 7fb2a7400640 1 --2- 172.21.15.28:0/3219020520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b54deac080 0x55b54deaaed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.432+0000 7fb2a6a00640 1 -- 172.21.15.28:0/3219020520 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b54dea19b0 msgr2=0x55b54dea37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:03.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.432+0000 7fb2a6a00640 1 --2- 172.21.15.28:0/3219020520 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b54dea19b0 0x55b54dea37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.432+0000 7fb2a6a00640 1 -- 172.21.15.28:0/3219020520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b54deac080 msgr2=0x55b54deaaed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:03.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.432+0000 7fb2a6a00640 1 --2- 172.21.15.28:0/3219020520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b54deac080 0x55b54deaaed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.432+0000 7fb2a6a00640 1 -- 172.21.15.28:0/3219020520 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b54dea7f40 con 0x55b54dea1090 2024-04-23T14:49:03.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.432+0000 7fb2a7400640 1 --2- 172.21.15.28:0/3219020520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b54deac080 0x55b54deaaed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:49:03.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.432+0000 7fb2a6a00640 1 --2- 172.21.15.28:0/3219020520 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b54dea1090 0x55b54deb7d30 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb298009950 tx=0x7fb298019430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19be894c715631ac server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:03.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.433+0000 7fb293400640 1 -- 172.21.15.28:0/3219020520 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29802bd50 con 0x55b54dea1090 2024-04-23T14:49:03.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.433+0000 7fb293400640 1 -- 172.21.15.28:0/3219020520 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb29802b2e0 con 0x55b54dea1090 2024-04-23T14:49:03.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.433+0000 7fb293400640 1 -- 172.21.15.28:0/3219020520 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb298004520 con 0x55b54dea1090 2024-04-23T14:49:03.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.433+0000 7fb2b0841740 1 -- 172.21.15.28:0/3219020520 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b54dea1090 msgr2=0x55b54deb7d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:03.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.433+0000 7fb2b0841740 1 --2- 172.21.15.28:0/3219020520 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b54dea1090 0x55b54deb7d30 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb298009950 tx=0x7fb298019430 comp rx=0 tx=0).stop 2024-04-23T14:49:03.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.434+0000 7fb2b0841740 1 -- 172.21.15.28:0/3219020520 shutdown_connections 2024-04-23T14:49:03.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.434+0000 7fb2b0841740 1 --2- 172.21.15.28:0/3219020520 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b54dea19b0 0x55b54dea37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.434+0000 7fb2b0841740 1 --2- 172.21.15.28:0/3219020520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b54deac080 0x55b54deaaed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.434+0000 7fb2b0841740 1 --2- 172.21.15.28:0/3219020520 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b54dea1090 0x55b54deb7d30 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.434+0000 7fb2b0841740 1 -- 172.21.15.28:0/3219020520 >> 172.21.15.28:0/3219020520 conn(0x55b54de9d3b0 msgr2=0x55b54deaa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:03.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.434+0000 7fb2b0841740 1 -- 172.21.15.28:0/3219020520 shutdown_connections 2024-04-23T14:49:03.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.434+0000 7fb2b0841740 1 -- 172.21.15.28:0/3219020520 wait complete. 2024-04-23T14:49:03.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.434+0000 7fb2b0841740 1 -- start start 2024-04-23T14:49:03.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.434+0000 7fb2b0841740 1 Processor -- start 2024-04-23T14:49:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.435+0000 7fb2b0841740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b54dea1090 0x55b54dea5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.435+0000 7fb2b0841740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b54dea19b0 0x55b54dea6160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.435+0000 7fb2b0841740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b54deac080 0x55b54dea66a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.435+0000 7fb2b0841740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a81aec50 con 0x55b54dea1090 2024-04-23T14:49:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.435+0000 7fb2b0841740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a81b00c0 con 0x55b54dea19b0 2024-04-23T14:49:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.435+0000 7fb2b0841740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a81af320 con 0x55b54deac080 2024-04-23T14:49:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.435+0000 7fb2b0841740 10 client.0 _ll_register_callbacks cb 0x7fb2a0098e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:49:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.435+0000 7fb2a6000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b54dea19b0 0x55b54dea6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.435+0000 7fb2a6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b54dea1090 0x55b54dea5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.435+0000 7fb2a6000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b54dea19b0 0x55b54dea6160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:58972/0 (socket says 172.21.15.28:58972) 2024-04-23T14:49:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.435+0000 7fb2a6000640 1 -- 172.21.15.28:0/1789104573 learned_addr learned my addr 172.21.15.28:0/1789104573 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.435+0000 7fb2a7400640 1 --2- 172.21.15.28:0/1789104573 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b54deac080 0x55b54dea66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.436+0000 7fb2a6000640 1 -- 172.21.15.28:0/1789104573 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b54deac080 msgr2=0x55b54dea66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:03.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.436+0000 7fb2a6000640 1 --2- 172.21.15.28:0/1789104573 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b54deac080 0x55b54dea66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.436+0000 7fb2a6000640 1 -- 172.21.15.28:0/1789104573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b54dea1090 msgr2=0x55b54dea5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:03.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.436+0000 7fb2a6000640 1 --2- 172.21.15.28:0/1789104573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b54dea1090 0x55b54dea5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.436+0000 7fb2a6000640 1 -- 172.21.15.28:0/1789104573 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b54df21910 con 0x55b54dea19b0 2024-04-23T14:49:03.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.436+0000 7fb2a6a00640 1 --2- 172.21.15.28:0/1789104573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b54dea1090 0x55b54dea5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:49:03.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.436+0000 7fb2a6000640 1 --2- 172.21.15.28:0/1789104573 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b54dea19b0 0x55b54dea6160 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb29402a9f0 tx=0x7fb29402a400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=627b7c74a8634470 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:03.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.436+0000 7fb290c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:49:03.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.436+0000 7fb290c00640 1 -- 172.21.15.28:0/1789104573 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb294046070 con 0x55b54dea19b0 2024-04-23T14:49:03.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.436+0000 7fb290c00640 1 -- 172.21.15.28:0/1789104573 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fb294003c80 con 0x55b54dea19b0 2024-04-23T14:49:03.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.436+0000 7fb2b0841740 1 -- 172.21.15.28:0/1789104573 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb29802b2e0 con 0x55b54dea19b0 2024-04-23T14:49:03.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.437+0000 7fb290c00640 1 -- 172.21.15.28:0/1789104573 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fb294003e80 con 0x55b54dea19b0 2024-04-23T14:49:03.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.437+0000 7fb2b0841740 10 client.26042 fetch_fsmap learned FSMap version 20 2024-04-23T14:49:03.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.437+0000 7fb2b0841740 1 -- 172.21.15.28:0/1789104573 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55b54df22b70 con 0x55b54dea19b0 2024-04-23T14:49:03.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.437+0000 7fb290c00640 1 -- 172.21.15.28:0/1789104573 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb29802b2e0 con 0x55b54dea19b0 2024-04-23T14:49:03.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.437+0000 7fb290c00640 1 -- 172.21.15.28:0/1789104573 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fb294055020 con 0x55b54dea19b0 2024-04-23T14:49:03.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.437+0000 7fb2b0841740 10 client.26042 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:49:03.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.437+0000 7fb2b0841740 10 client.26042 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:49:03.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.437+0000 7fb2b0841740 20 client.26042 populate_metadata read hostname 'smithi028' 2024-04-23T14:49:03.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.437+0000 7fb2b0841740 1 --2- 172.21.15.28:0/1789104573 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55b54df232d0 0x55b54df256f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.437+0000 7fb2b0841740 4 client.26042 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-23T14:49:03.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.437+0000 7fb2b0841740 1 -- 172.21.15.28:0/1789104573 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b54de0e700 con 0x55b54df232d0 2024-04-23T14:49:03.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.437+0000 7fb2a6a00640 1 --2- 172.21.15.28:0/1789104573 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55b54df232d0 0x55b54df256f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.438+0000 7fb2a6a00640 1 --2- 172.21.15.28:0/1789104573 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55b54df232d0 0x55b54df256f0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4737e51db067a2ad server_cookie=1c9149ae1ab3f6e6 in_seq=0 out_seq=0 2024-04-23T14:49:03.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.438+0000 7fb290c00640 10 client.26042 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:49:03.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.439+0000 7fb290c00640 1 -- 172.21.15.28:0/1789104573 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b54de0e700 con 0x55b54df232d0 2024-04-23T14:49:03.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.439+0000 7fb290c00640 10 client.26042 handle_command_reply: tid=0 2024-04-23T14:49:03.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.439+0000 7fb2b0841740 1 client.26042 shutdown 2024-04-23T14:49:03.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.439+0000 7fb2b0841740 1 -- 172.21.15.28:0/1789104573 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b54dea19b0 msgr2=0x55b54dea6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:03.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.439+0000 7fb2b0841740 1 --2- 172.21.15.28:0/1789104573 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b54dea19b0 0x55b54dea6160 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb29402a9f0 tx=0x7fb29402a400 comp rx=0 tx=0).stop 2024-04-23T14:49:03.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.439+0000 7fb2b0841740 1 -- 172.21.15.28:0/1789104573 shutdown_connections 2024-04-23T14:49:03.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.439+0000 7fb2b0841740 1 --2- 172.21.15.28:0/1789104573 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55b54df232d0 0x55b54df256f0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.439+0000 7fb2b0841740 1 --2- 172.21.15.28:0/1789104573 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b54dea19b0 0x55b54dea6160 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.439+0000 7fb2b0841740 1 --2- 172.21.15.28:0/1789104573 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b54deac080 0x55b54dea66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.439+0000 7fb2b0841740 1 --2- 172.21.15.28:0/1789104573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b54dea1090 0x55b54dea5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.439+0000 7fb2b0841740 1 -- 172.21.15.28:0/1789104573 >> 172.21.15.28:0/1789104573 conn(0x55b54de9d3b0 msgr2=0x55b54de9c510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:03.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.439+0000 7fb290c00640 0 client.26042 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:49:03.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.439+0000 7fb2b0841740 1 -- 172.21.15.28:0/1789104573 shutdown_connections 2024-04-23T14:49:03.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.439+0000 7fb2b0841740 1 -- 172.21.15.28:0/1789104573 wait complete. 2024-04-23T14:49:03.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.439+0000 7fb2b0841740 20 client.26042 trim_cache size 0 max 16384 2024-04-23T14:49:03.445 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:49:03.445 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-23T14:49:03.445 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "1504808e-9ee1-48b1-b966-4ae84fa5c87f", 2024-04-23T14:49:03.445 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-23T14:49:03.445 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:49:03.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.440+0000 7fb2ada00640 1 -- 172.21.15.28:0/832065686 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb28c0a3da0 msgr2=0x7fb28c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:03.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.440+0000 7fb2ada00640 1 --2- 172.21.15.28:0/832065686 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb28c0a3da0 0x7fb28c0a6260 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb29402a920 tx=0x7fb294041040 comp rx=0 tx=0).stop 2024-04-23T14:49:03.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.440+0000 7fb2ada00640 1 -- 172.21.15.28:0/832065686 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2a819a980 msgr2=0x7fb2a819ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:03.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.440+0000 7fb2ada00640 1 --2- 172.21.15.28:0/832065686 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2a819a980 0x7fb2a819ae30 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb2a000ea60 tx=0x7fb2a000ef20 comp rx=0 tx=0).stop 2024-04-23T14:49:03.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.440+0000 7fb2ada00640 1 -- 172.21.15.28:0/832065686 shutdown_connections 2024-04-23T14:49:03.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.440+0000 7fb2ada00640 1 --2- 172.21.15.28:0/832065686 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb28c0a3da0 0x7fb28c0a6260 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.440+0000 7fb2ada00640 1 --2- 172.21.15.28:0/832065686 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2a819a980 0x7fb2a819ae30 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.440+0000 7fb2ada00640 1 --2- 172.21.15.28:0/832065686 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb2a8107490 0x7fb2a81a0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.440+0000 7fb2ada00640 1 --2- 172.21.15.28:0/832065686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2a81003f0 0x7fb2a81a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.440+0000 7fb2ada00640 1 -- 172.21.15.28:0/832065686 >> 172.21.15.28:0/832065686 conn(0x7fb2a8077060 msgr2=0x7fb2a81058a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:03.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.440+0000 7fb2ada00640 1 -- 172.21.15.28:0/832065686 shutdown_connections 2024-04-23T14:49:03.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.440+0000 7fb2ada00640 1 -- 172.21.15.28:0/832065686 wait complete. 2024-04-23T14:49:03.459 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:49:03.460 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:49:03.486 INFO:tasks.workunit.client.0.smithi028.stdout: 1 376641 9.70 MB/sec execute 79 sec latency 14.160 ms 2024-04-23T14:49:03.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.671+0000 7f9484400640 1 -- 172.21.15.28:0/3748715756 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f947c104450 msgr2=0x7f947c104830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:03.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.671+0000 7f9484400640 1 --2- 172.21.15.28:0/3748715756 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f947c104450 0x7f947c104830 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f9470009900 tx=0x7f947002a3f0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.671+0000 7f9484400640 1 -- 172.21.15.28:0/3748715756 shutdown_connections 2024-04-23T14:49:03.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.671+0000 7f9484400640 1 --2- 172.21.15.28:0/3748715756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f947c109520 0x7f947c1110f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.671+0000 7f9484400640 1 --2- 172.21.15.28:0/3748715756 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f947c104e00 0x7f947c108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.671+0000 7f9484400640 1 --2- 172.21.15.28:0/3748715756 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f947c104450 0x7f947c104830 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.671+0000 7f9484400640 1 -- 172.21.15.28:0/3748715756 >> 172.21.15.28:0/3748715756 conn(0x7f947c0ffc70 msgr2=0x7f947c102090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:03.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.671+0000 7f9484400640 1 -- 172.21.15.28:0/3748715756 shutdown_connections 2024-04-23T14:49:03.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.671+0000 7f9484400640 1 -- 172.21.15.28:0/3748715756 wait complete. 2024-04-23T14:49:03.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.672+0000 7f9484400640 1 Processor -- start 2024-04-23T14:49:03.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.672+0000 7f9484400640 1 -- start start 2024-04-23T14:49:03.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.672+0000 7f9484400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f947c104450 0x7f947c1a3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.672+0000 7f9484400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f947c109520 0x7f947c1a4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.673+0000 7f9484400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f947c10fb70 0x7f947c10fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.673+0000 7f9484400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f947c077760 con 0x7f947c104450 2024-04-23T14:49:03.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.673+0000 7f9484400640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f947c0775e0 con 0x7f947c109520 2024-04-23T14:49:03.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.673+0000 7f9484400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f947c0778e0 con 0x7f947c10fb70 2024-04-23T14:49:03.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.673+0000 7f9481000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f947c109520 0x7f947c1a4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.673+0000 7f9481a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f947c104450 0x7f947c1a3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.673+0000 7f9482400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f947c10fb70 0x7f947c10fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.673+0000 7f9481000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f947c109520 0x7f947c1a4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:58988/0 (socket says 172.21.15.28:58988) 2024-04-23T14:49:03.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.673+0000 7f9481000640 1 -- 172.21.15.28:0/1825444328 learned_addr learned my addr 172.21.15.28:0/1825444328 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:03.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.673+0000 7f9481000640 1 -- 172.21.15.28:0/1825444328 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f947c10fb70 msgr2=0x7f947c10fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:03.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.673+0000 7f9481000640 1 --2- 172.21.15.28:0/1825444328 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f947c10fb70 0x7f947c10fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.673+0000 7f9481000640 1 -- 172.21.15.28:0/1825444328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f947c104450 msgr2=0x7f947c1a3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:03.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.673+0000 7f9481000640 1 --2- 172.21.15.28:0/1825444328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f947c104450 0x7f947c1a3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.673+0000 7f9481000640 1 -- 172.21.15.28:0/1825444328 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f947c1108b0 con 0x7f947c109520 2024-04-23T14:49:03.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.673+0000 7f9482400640 1 --2- 172.21.15.28:0/1825444328 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f947c10fb70 0x7f947c10fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:49:03.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.674+0000 7f9481000640 1 --2- 172.21.15.28:0/1825444328 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f947c109520 0x7f947c1a4170 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f946800e9c0 tx=0x7f946800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:03.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.674+0000 7f947a000640 1 -- 172.21.15.28:0/1825444328 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f946800cd80 con 0x7f947c109520 2024-04-23T14:49:03.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.674+0000 7f9484400640 1 -- 172.21.15.28:0/1825444328 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f947c1ab910 con 0x7f947c109520 2024-04-23T14:49:03.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.674+0000 7f947a000640 1 -- 172.21.15.28:0/1825444328 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9468005500 con 0x7f947c109520 2024-04-23T14:49:03.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.674+0000 7f947a000640 1 -- 172.21.15.28:0/1825444328 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9468010690 con 0x7f947c109520 2024-04-23T14:49:03.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.674+0000 7f9484400640 1 -- 172.21.15.28:0/1825444328 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f947c1abe00 con 0x7f947c109520 2024-04-23T14:49:03.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.678+0000 7f9487206740 1 Processor -- start 2024-04-23T14:49:03.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.679+0000 7f9487206740 1 -- start start 2024-04-23T14:49:03.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.679+0000 7f9487206740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f02ef8e860 0x55f02ef8ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.679+0000 7f9487206740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f02ef8f180 0x55f02ef99310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.679+0000 7f9487206740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f02ef99850 0x55f02ef9bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.679+0000 7f9487206740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f02eef41f0 con 0x55f02ef8f180 2024-04-23T14:49:03.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.679+0000 7f9487206740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f02ee9a350 con 0x55f02ef99850 2024-04-23T14:49:03.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.679+0000 7f9487206740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f02eef01b0 con 0x55f02ef8e860 2024-04-23T14:49:03.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.680+0000 7f9481a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f02ef8f180 0x55f02ef99310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.680+0000 7f9481a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f02ef8f180 0x55f02ef99310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34560/0 (socket says 172.21.15.28:34560) 2024-04-23T14:49:03.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.680+0000 7f947a000640 1 -- 172.21.15.28:0/1825444328 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f94680046c0 con 0x7f947c109520 2024-04-23T14:49:03.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.680+0000 7f9481a00640 1 -- 172.21.15.28:0/3264279393 learned_addr learned my addr 172.21.15.28:0/3264279393 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:03.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.680+0000 7f9482400640 1 --2- 172.21.15.28:0/3264279393 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f02ef8e860 0x55f02ef8ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.680+0000 7f9481000640 1 --2- 172.21.15.28:0/3264279393 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f02ef99850 0x55f02ef9bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.680+0000 7f9482400640 1 -- 172.21.15.28:0/3264279393 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f02ef99850 msgr2=0x55f02ef9bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:03.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.680+0000 7f9482400640 1 --2- 172.21.15.28:0/3264279393 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f02ef99850 0x55f02ef9bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.680+0000 7f9482400640 1 -- 172.21.15.28:0/3264279393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f02ef8f180 msgr2=0x55f02ef99310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:03.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.680+0000 7f9482400640 1 --2- 172.21.15.28:0/3264279393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f02ef8f180 0x55f02ef99310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.680+0000 7f9482400640 1 -- 172.21.15.28:0/3264279393 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f02eeeef50 con 0x55f02ef8e860 2024-04-23T14:49:03.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.680+0000 7f9481a00640 1 --2- 172.21.15.28:0/3264279393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f02ef8f180 0x55f02ef99310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:49:03.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.681+0000 7f9482400640 1 --2- 172.21.15.28:0/3264279393 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f02ef8e860 0x55f02ef8ec40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9474005fd0 tx=0x7f9474011bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5cc4c2af80bc348 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:03.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.681+0000 7f946ea00640 1 -- 172.21.15.28:0/3264279393 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9474004d20 con 0x55f02ef8e860 2024-04-23T14:49:03.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.681+0000 7f946ea00640 1 -- 172.21.15.28:0/3264279393 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f947400f9a0 con 0x55f02ef8e860 2024-04-23T14:49:03.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.681+0000 7f946ea00640 1 -- 172.21.15.28:0/3264279393 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9474004430 con 0x55f02ef8e860 2024-04-23T14:49:03.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.681+0000 7f947a000640 1 --2- 172.21.15.28:0/1825444328 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f94640a3da0 0x7f94640a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.682+0000 7f9487206740 1 -- 172.21.15.28:0/3264279393 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f02ef8e860 msgr2=0x55f02ef8ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:03.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.682+0000 7f9487206740 1 --2- 172.21.15.28:0/3264279393 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f02ef8e860 0x55f02ef8ec40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9474005fd0 tx=0x7f9474011bf0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.682+0000 7f947a000640 1 -- 172.21.15.28:0/1825444328 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f94680d0ac0 con 0x7f947c109520 2024-04-23T14:49:03.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.682+0000 7f9481a00640 1 --2- 172.21.15.28:0/1825444328 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f94640a3da0 0x7f94640a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.682+0000 7f9487206740 1 -- 172.21.15.28:0/3264279393 shutdown_connections 2024-04-23T14:49:03.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.682+0000 7f9487206740 1 --2- 172.21.15.28:0/3264279393 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f02ef99850 0x55f02ef9bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.682+0000 7f9487206740 1 --2- 172.21.15.28:0/3264279393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f02ef8f180 0x55f02ef99310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.682+0000 7f9487206740 1 --2- 172.21.15.28:0/3264279393 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f02ef8e860 0x55f02ef8ec40 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.682+0000 7f9487206740 1 -- 172.21.15.28:0/3264279393 >> 172.21.15.28:0/3264279393 conn(0x55f02ef8ab80 msgr2=0x55f02ef8af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.682+0000 7f9487206740 1 -- 172.21.15.28:0/3264279393 shutdown_connections 2024-04-23T14:49:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.682+0000 7f9487206740 1 -- 172.21.15.28:0/3264279393 wait complete. 2024-04-23T14:49:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.683+0000 7f9481a00640 1 --2- 172.21.15.28:0/1825444328 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f94640a3da0 0x7f94640a6260 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f94700096c0 tx=0x7f9470000bc0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.683+0000 7f9487206740 1 -- start start 2024-04-23T14:49:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.683+0000 7f9487206740 1 Processor -- start 2024-04-23T14:49:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.684+0000 7f9487206740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f02ef8e860 0x55f02ef96760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.684+0000 7f9487206740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f02ef8f180 0x55f02ef96ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.684+0000 7f9487206740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f02ef99850 0x55f02ef971e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.684+0000 7f9487206740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f02ef9c640 con 0x55f02ef99850 2024-04-23T14:49:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.684+0000 7f9487206740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f02eead690 con 0x55f02ef8e860 2024-04-23T14:49:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.684+0000 7f9487206740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f02ef9c7c0 con 0x55f02ef8f180 2024-04-23T14:49:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.684+0000 7f9487206740 10 client.0 _ll_register_callbacks cb 0x55f02eee9650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:49:03.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.684+0000 7f9481000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f02ef8f180 0x55f02ef96ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.684+0000 7f9481a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f02ef8e860 0x55f02ef96760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.684+0000 7f9481a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f02ef8e860 0x55f02ef96760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:59006/0 (socket says 172.21.15.28:59006) 2024-04-23T14:49:03.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.684+0000 7f9481a00640 1 -- 172.21.15.28:0/3198984041 learned_addr learned my addr 172.21.15.28:0/3198984041 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:03.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.684+0000 7f9481000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f02ef8f180 0x55f02ef96ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:47468/0 (socket says 172.21.15.28:47468) 2024-04-23T14:49:03.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.684+0000 7f9482400640 1 --2- 172.21.15.28:0/3198984041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f02ef99850 0x55f02ef971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.684+0000 7f9481a00640 1 -- 172.21.15.28:0/3198984041 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f02ef8f180 msgr2=0x55f02ef96ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:03.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.684+0000 7f9481a00640 1 --2- 172.21.15.28:0/3198984041 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f02ef8f180 0x55f02ef96ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.685+0000 7f9481a00640 1 -- 172.21.15.28:0/3198984041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f02ef99850 msgr2=0x55f02ef971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:03.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.685+0000 7f9481a00640 1 --2- 172.21.15.28:0/3198984041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f02ef99850 0x55f02ef971e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.685+0000 7f9481a00640 1 -- 172.21.15.28:0/3198984041 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f02eeac920 con 0x55f02ef8e860 2024-04-23T14:49:03.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.685+0000 7f9481000640 1 --2- 172.21.15.28:0/3198984041 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f02ef8f180 0x55f02ef96ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:49:03.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.685+0000 7f9481a00640 1 --2- 172.21.15.28:0/3198984041 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f02ef8e860 0x55f02ef96760 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f9470005090 tx=0x7f9470009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=280fa7979749a4af server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:03.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.685+0000 7f9463400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:49:03.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.685+0000 7f9463400640 1 -- 172.21.15.28:0/3198984041 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947004e070 con 0x55f02ef8e860 2024-04-23T14:49:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.685+0000 7f9463400640 1 -- 172.21.15.28:0/3198984041 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f9470004b90 con 0x55f02ef8e860 2024-04-23T14:49:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.685+0000 7f9487206740 1 -- 172.21.15.28:0/3198984041 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f947400f9a0 con 0x55f02ef8e860 2024-04-23T14:49:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.686+0000 7f9463400640 1 -- 172.21.15.28:0/3198984041 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f947005a050 con 0x55f02ef8e860 2024-04-23T14:49:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.686+0000 7f9487206740 10 client.26054 fetch_fsmap learned FSMap version 20 2024-04-23T14:49:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.686+0000 7f9487206740 1 -- 172.21.15.28:0/3198984041 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55f02ecd07d0 con 0x55f02ef8e860 2024-04-23T14:49:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.686+0000 7f9463400640 1 -- 172.21.15.28:0/3198984041 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f947400f9a0 con 0x55f02ef8e860 2024-04-23T14:49:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.686+0000 7f9463400640 1 -- 172.21.15.28:0/3198984041 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f94700037b0 con 0x55f02ef8e860 2024-04-23T14:49:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.686+0000 7f9487206740 10 client.26054 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:49:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.686+0000 7f9487206740 10 client.26054 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:49:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.686+0000 7f9487206740 20 client.26054 populate_metadata read hostname 'smithi028' 2024-04-23T14:49:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.686+0000 7f9487206740 1 --2- 172.21.15.28:0/3198984041 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f02efa9b50 0x55f02efabfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.686+0000 7f9487206740 4 client.26054 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:49:03.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.686+0000 7f9487206740 1 -- 172.21.15.28:0/3198984041 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f02ef01c10 con 0x55f02efa9b50 2024-04-23T14:49:03.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.686+0000 7f9481000640 1 --2- 172.21.15.28:0/3198984041 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f02efa9b50 0x55f02efabfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.687+0000 7f9481000640 1 --2- 172.21.15.28:0/3198984041 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f02efa9b50 0x55f02efabfa0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15804ec55f662d4c server_cookie=b657fedfbcdf14fb in_seq=0 out_seq=0 2024-04-23T14:49:03.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.687+0000 7f9463400640 10 client.26054 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:49:03.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.688+0000 7f9463400640 1 -- 172.21.15.28:0/3198984041 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f02ef01c10 con 0x55f02efa9b50 2024-04-23T14:49:03.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.688+0000 7f9463400640 10 client.26054 handle_command_reply: tid=0 2024-04-23T14:49:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.688+0000 7f9487206740 1 client.26054 shutdown 2024-04-23T14:49:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.688+0000 7f9487206740 1 -- 172.21.15.28:0/3198984041 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f02ef8e860 msgr2=0x55f02ef96760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.688+0000 7f9487206740 1 --2- 172.21.15.28:0/3198984041 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f02ef8e860 0x55f02ef96760 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f9470005090 tx=0x7f9470009260 comp rx=0 tx=0).stop 2024-04-23T14:49:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.688+0000 7f9487206740 1 -- 172.21.15.28:0/3198984041 shutdown_connections 2024-04-23T14:49:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.688+0000 7f9487206740 1 --2- 172.21.15.28:0/3198984041 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f02efa9b50 0x55f02efabfa0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.688+0000 7f9487206740 1 --2- 172.21.15.28:0/3198984041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f02ef99850 0x55f02ef971e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.688+0000 7f9487206740 1 --2- 172.21.15.28:0/3198984041 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f02ef8f180 0x55f02ef96ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.688+0000 7f9463400640 0 client.26054 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:49:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.688+0000 7f9487206740 1 --2- 172.21.15.28:0/3198984041 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f02ef8e860 0x55f02ef96760 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.688+0000 7f9487206740 1 -- 172.21.15.28:0/3198984041 >> 172.21.15.28:0/3198984041 conn(0x55f02ef8ab80 msgr2=0x55f02ef90880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.689+0000 7f9487206740 1 -- 172.21.15.28:0/3198984041 shutdown_connections 2024-04-23T14:49:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.689+0000 7f9487206740 1 -- 172.21.15.28:0/3198984041 wait complete. 2024-04-23T14:49:03.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.689+0000 7f9487206740 20 client.26054 trim_cache size 0 max 16384 2024-04-23T14:49:03.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.709+0000 7f9487206740 1 Processor -- start 2024-04-23T14:49:03.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.709+0000 7f9487206740 1 -- start start 2024-04-23T14:49:03.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.709+0000 7f9487206740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f02ef8e860 0x55f02efa5560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.710+0000 7f9487206740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f02ef8f180 0x55f02ef956a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.710+0000 7f9487206740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f02ef99850 0x55f02ef91000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.710+0000 7f9487206740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f947c1b2330 con 0x55f02ef8e860 2024-04-23T14:49:03.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.710+0000 7f9487206740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f947c1b4bb0 con 0x55f02ef8f180 2024-04-23T14:49:03.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.710+0000 7f9487206740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f947c1b29a0 con 0x55f02ef99850 2024-04-23T14:49:03.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.710+0000 7f9481000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f02ef8f180 0x55f02ef956a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.710+0000 7f9481000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f02ef8f180 0x55f02ef956a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:59020/0 (socket says 172.21.15.28:59020) 2024-04-23T14:49:03.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.710+0000 7f9481000640 1 -- 172.21.15.28:0/3093495641 learned_addr learned my addr 172.21.15.28:0/3093495641 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:03.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.710+0000 7f9482400640 1 --2- 172.21.15.28:0/3093495641 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f02ef99850 0x55f02ef91000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.710+0000 7f9481a00640 1 --2- 172.21.15.28:0/3093495641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f02ef8e860 0x55f02efa5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.710+0000 7f9481000640 1 -- 172.21.15.28:0/3093495641 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f02ef99850 msgr2=0x55f02ef91000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:03.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.710+0000 7f9481000640 1 --2- 172.21.15.28:0/3093495641 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f02ef99850 0x55f02ef91000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.710+0000 7f9481000640 1 -- 172.21.15.28:0/3093495641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f02ef8e860 msgr2=0x55f02efa5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:03.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.710+0000 7f9481000640 1 --2- 172.21.15.28:0/3093495641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f02ef8e860 0x55f02efa5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.710+0000 7f9481000640 1 -- 172.21.15.28:0/3093495641 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f02ec063f0 con 0x55f02ef8f180 2024-04-23T14:49:03.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.710+0000 7f9481a00640 1 --2- 172.21.15.28:0/3093495641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f02ef8e860 0x55f02efa5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:49:03.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.711+0000 7f9481000640 1 --2- 172.21.15.28:0/3093495641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f02ef8f180 0x55f02ef956a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f94680ce220 tx=0x7f94680bd040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94ab5eaaaafd00d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:03.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.711+0000 7f946e000640 1 -- 172.21.15.28:0/3093495641 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f946809c550 con 0x55f02ef8f180 2024-04-23T14:49:03.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.711+0000 7f946e000640 1 -- 172.21.15.28:0/3093495641 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f946809d110 con 0x55f02ef8f180 2024-04-23T14:49:03.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.711+0000 7f946e000640 1 -- 172.21.15.28:0/3093495641 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f946809c370 con 0x55f02ef8f180 2024-04-23T14:49:03.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.712+0000 7f9487206740 1 -- 172.21.15.28:0/3093495641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f02ef8f180 msgr2=0x55f02ef956a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:03.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.712+0000 7f9487206740 1 --2- 172.21.15.28:0/3093495641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f02ef8f180 0x55f02ef956a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f94680ce220 tx=0x7f94680bd040 comp rx=0 tx=0).stop 2024-04-23T14:49:03.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.712+0000 7f9487206740 1 -- 172.21.15.28:0/3093495641 shutdown_connections 2024-04-23T14:49:03.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.712+0000 7f9487206740 1 --2- 172.21.15.28:0/3093495641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f02ef8f180 0x55f02ef956a0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.712+0000 7f9487206740 1 --2- 172.21.15.28:0/3093495641 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f02ef99850 0x55f02ef91000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.712+0000 7f9487206740 1 --2- 172.21.15.28:0/3093495641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f02ef8e860 0x55f02efa5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.712+0000 7f9487206740 1 -- 172.21.15.28:0/3093495641 >> 172.21.15.28:0/3093495641 conn(0x55f02ef8ab80 msgr2=0x55f02ef89c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:03.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.712+0000 7f9487206740 1 -- 172.21.15.28:0/3093495641 shutdown_connections 2024-04-23T14:49:03.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.712+0000 7f9487206740 1 -- 172.21.15.28:0/3093495641 wait complete. 2024-04-23T14:49:03.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.712+0000 7f9487206740 1 -- start start 2024-04-23T14:49:03.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.713+0000 7f9487206740 1 Processor -- start 2024-04-23T14:49:03.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.713+0000 7f9487206740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f02ef8e860 0x55f02efa1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.713+0000 7f9487206740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f02ef8f180 0x55f02ef9d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.713+0000 7f9487206740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f02ef99850 0x55f02ef9d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.713+0000 7f9487206740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f947c1b3070 con 0x55f02ef99850 2024-04-23T14:49:03.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.713+0000 7f9487206740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f947c1b44e0 con 0x55f02ef8e860 2024-04-23T14:49:03.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.713+0000 7f9487206740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f947c1b3740 con 0x55f02ef8f180 2024-04-23T14:49:03.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.713+0000 7f9487206740 10 client.0 _ll_register_callbacks cb 0x7f9470009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:49:03.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.713+0000 7f9481a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f02ef8e860 0x55f02efa1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.713+0000 7f9482400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f02ef99850 0x55f02ef9d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.713+0000 7f9481000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f02ef8f180 0x55f02ef9d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.713+0000 7f9481a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f02ef8e860 0x55f02efa1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:59032/0 (socket says 172.21.15.28:59032) 2024-04-23T14:49:03.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.713+0000 7f9481a00640 1 -- 172.21.15.28:0/3230490903 learned_addr learned my addr 172.21.15.28:0/3230490903 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:03.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.714+0000 7f9481a00640 1 -- 172.21.15.28:0/3230490903 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f02ef8f180 msgr2=0x55f02ef9d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:03.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.714+0000 7f9481a00640 1 --2- 172.21.15.28:0/3230490903 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f02ef8f180 0x55f02ef9d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.714+0000 7f9481a00640 1 -- 172.21.15.28:0/3230490903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f02ef99850 msgr2=0x55f02ef9d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:03.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.714+0000 7f9481a00640 1 --2- 172.21.15.28:0/3230490903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f02ef99850 0x55f02ef9d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.714+0000 7f9481a00640 1 -- 172.21.15.28:0/3230490903 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f02ebc60f0 con 0x55f02ef8e860 2024-04-23T14:49:03.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.714+0000 7f9481000640 1 --2- 172.21.15.28:0/3230490903 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f02ef8f180 0x55f02ef9d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:49:03.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.714+0000 7f9481a00640 1 --2- 172.21.15.28:0/3230490903 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f02ef8e860 0x55f02efa1d90 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9470009230 tx=0x7f9470004320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e095989578694e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:03.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.714+0000 7f9463400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:49:03.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.714+0000 7f9463400640 1 -- 172.21.15.28:0/3230490903 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9470062070 con 0x55f02ef8e860 2024-04-23T14:49:03.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.714+0000 7f9463400640 1 -- 172.21.15.28:0/3230490903 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f9470004990 con 0x55f02ef8e860 2024-04-23T14:49:03.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.714+0000 7f9487206740 1 -- 172.21.15.28:0/3230490903 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f946809d110 con 0x55f02ef8e860 2024-04-23T14:49:03.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.715+0000 7f9463400640 1 -- 172.21.15.28:0/3230490903 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f947004e040 con 0x55f02ef8e860 2024-04-23T14:49:03.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.715+0000 7f9487206740 10 client.26060 fetch_fsmap learned FSMap version 20 2024-04-23T14:49:03.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.715+0000 7f9487206740 1 -- 172.21.15.28:0/3230490903 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55f02eeb9810 con 0x55f02ef8e860 2024-04-23T14:49:03.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.715+0000 7f9463400640 1 -- 172.21.15.28:0/3230490903 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f946809d110 con 0x55f02ef8e860 2024-04-23T14:49:03.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.715+0000 7f9463400640 1 -- 172.21.15.28:0/3230490903 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f9470032bc0 con 0x55f02ef8e860 2024-04-23T14:49:03.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.715+0000 7f9487206740 10 client.26060 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:49:03.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.715+0000 7f9487206740 10 client.26060 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:49:03.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.715+0000 7f9487206740 20 client.26060 populate_metadata read hostname 'smithi028' 2024-04-23T14:49:03.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.715+0000 7f9487206740 1 --2- 172.21.15.28:0/3230490903 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f02f010380 0x55f02f012770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:03.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.715+0000 7f9487206740 4 client.26060 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:49:03.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.715+0000 7f9487206740 1 -- 172.21.15.28:0/3230490903 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f02eefba80 con 0x55f02f010380 2024-04-23T14:49:03.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.716+0000 7f9481000640 1 --2- 172.21.15.28:0/3230490903 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f02f010380 0x55f02f012770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:03.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.716+0000 7f9481000640 1 --2- 172.21.15.28:0/3230490903 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f02f010380 0x55f02f012770 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b52396c592736c95 server_cookie=2e9be678c000fb70 in_seq=0 out_seq=0 2024-04-23T14:49:03.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.716+0000 7f9463400640 10 client.26060 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:49:03.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.718+0000 7f9463400640 1 -- 172.21.15.28:0/3230490903 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f02eefba80 con 0x55f02f010380 2024-04-23T14:49:03.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.718+0000 7f9463400640 10 client.26060 handle_command_reply: tid=0 2024-04-23T14:49:03.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.718+0000 7f9487206740 1 client.26060 shutdown 2024-04-23T14:49:03.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.718+0000 7f9487206740 1 -- 172.21.15.28:0/3230490903 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f02ef8e860 msgr2=0x55f02efa1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:03.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.718+0000 7f9487206740 1 --2- 172.21.15.28:0/3230490903 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f02ef8e860 0x55f02efa1d90 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9470009230 tx=0x7f9470004320 comp rx=0 tx=0).stop 2024-04-23T14:49:03.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.718+0000 7f9487206740 1 -- 172.21.15.28:0/3230490903 shutdown_connections 2024-04-23T14:49:03.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.718+0000 7f9487206740 1 --2- 172.21.15.28:0/3230490903 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f02f010380 0x55f02f012770 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.718+0000 7f9487206740 1 --2- 172.21.15.28:0/3230490903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f02ef99850 0x55f02ef9d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.718+0000 7f9487206740 1 --2- 172.21.15.28:0/3230490903 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f02ef8f180 0x55f02ef9d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.718+0000 7f9487206740 1 --2- 172.21.15.28:0/3230490903 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f02ef8e860 0x55f02efa1d90 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.718+0000 7f9463400640 0 client.26060 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:49:03.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.718+0000 7f9487206740 1 -- 172.21.15.28:0/3230490903 >> 172.21.15.28:0/3230490903 conn(0x55f02ef8ab80 msgr2=0x55f02ef96700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:03.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.718+0000 7f9487206740 1 -- 172.21.15.28:0/3230490903 shutdown_connections 2024-04-23T14:49:03.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.718+0000 7f9487206740 1 -- 172.21.15.28:0/3230490903 wait complete. 2024-04-23T14:49:03.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.718+0000 7f9487206740 20 client.26060 trim_cache size 0 max 16384 2024-04-23T14:49:03.722 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:49:03.723 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-23T14:49:03.723 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-23T14:49:03.723 INFO:teuthology.orchestra.run.smithi028.stdout: "1504808e-9ee1-48b1-b966-4ae84fa5c87f": { 2024-04-23T14:49:03.723 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-23T14:49:03.723 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "1504808e-9ee1-48b1-b966-4ae84fa5c87f", 2024-04-23T14:49:03.723 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-23T14:49:03.723 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:49:03.723 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:49:03.723 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:49:03.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.719+0000 7f9484400640 1 -- 172.21.15.28:0/1825444328 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f94640a3da0 msgr2=0x7f94640a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:03.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.719+0000 7f9484400640 1 --2- 172.21.15.28:0/1825444328 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f94640a3da0 0x7f94640a6260 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f94700096c0 tx=0x7f9470000bc0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.719+0000 7f9484400640 1 -- 172.21.15.28:0/1825444328 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f947c109520 msgr2=0x7f947c1a4170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:03.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.719+0000 7f9484400640 1 --2- 172.21.15.28:0/1825444328 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f947c109520 0x7f947c1a4170 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f946800e9c0 tx=0x7f946800ee80 comp rx=0 tx=0).stop 2024-04-23T14:49:03.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.719+0000 7f9484400640 1 -- 172.21.15.28:0/1825444328 shutdown_connections 2024-04-23T14:49:03.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.719+0000 7f9484400640 1 --2- 172.21.15.28:0/1825444328 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f94640a3da0 0x7f94640a6260 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.719+0000 7f9484400640 1 --2- 172.21.15.28:0/1825444328 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f947c109520 0x7f947c1a4170 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.719+0000 7f9484400640 1 --2- 172.21.15.28:0/1825444328 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f947c10fb70 0x7f947c10fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.719+0000 7f9484400640 1 --2- 172.21.15.28:0/1825444328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f947c104450 0x7f947c1a3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:03.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.719+0000 7f9484400640 1 -- 172.21.15.28:0/1825444328 >> 172.21.15.28:0/1825444328 conn(0x7f947c0ffc70 msgr2=0x7f947c100f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:03.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.719+0000 7f9484400640 1 -- 172.21.15.28:0/1825444328 shutdown_connections 2024-04-23T14:49:03.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:03.720+0000 7f9484400640 1 -- 172.21.15.28:0/1825444328 wait complete. 2024-04-23T14:49:03.741 INFO:tasks.cephfs.filesystem:scrub status for tag:1504808e-9ee1-48b1-b966-4ae84fa5c87f - {'path': '/', 'tag': '1504808e-9ee1-48b1-b966-4ae84fa5c87f', 'options': 'recursive,force'} 2024-04-23T14:49:04.486 INFO:tasks.workunit.client.0.smithi028.stdout: 1 378933 9.73 MB/sec execute 80 sec latency 27.215 ms 2024-04-23T14:49:04.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:04 smithi038 ceph-mon[32348]: scrub queued for path: / 2024-04-23T14:49:04.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:04 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:49:04.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:04 smithi038 ceph-mon[32348]: scrub summary: active paths [/] 2024-04-23T14:49:04.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:04 smithi038 ceph-mon[32348]: pgmap v154: 97 pgs: 97 active+clean; 211 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.8 MiB/s wr, 261 op/s 2024-04-23T14:49:04.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:04 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:49:04.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:04 smithi028 ceph-mon[27810]: scrub queued for path: / 2024-04-23T14:49:04.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:04 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:49:04.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:04 smithi028 ceph-mon[27810]: scrub summary: active paths [/] 2024-04-23T14:49:04.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:04 smithi028 ceph-mon[27810]: pgmap v154: 97 pgs: 97 active+clean; 211 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.8 MiB/s wr, 261 op/s 2024-04-23T14:49:04.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:04 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:49:05.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:04 smithi105 ceph-mon[32555]: scrub queued for path: / 2024-04-23T14:49:05.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:04 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:49:05.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:04 smithi105 ceph-mon[32555]: scrub summary: active paths [/] 2024-04-23T14:49:05.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:04 smithi105 ceph-mon[32555]: pgmap v154: 97 pgs: 97 active+clean; 211 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.8 MiB/s wr, 261 op/s 2024-04-23T14:49:05.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:04 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:49:05.486 INFO:tasks.workunit.client.0.smithi028.stdout: 1 380347 9.70 MB/sec execute 81 sec latency 56.811 ms 2024-04-23T14:49:05.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:05 smithi038 ceph-mon[32348]: scrub summary: idle 2024-04-23T14:49:05.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:05 smithi028 ceph-mon[27810]: scrub summary: idle 2024-04-23T14:49:06.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:05 smithi105 ceph-mon[32555]: scrub summary: idle 2024-04-23T14:49:06.487 INFO:tasks.workunit.client.0.smithi028.stdout: 1 382533 9.73 MB/sec execute 82 sec latency 12.410 ms 2024-04-23T14:49:06.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:06 smithi038 ceph-mon[32348]: pgmap v155: 97 pgs: 97 active+clean; 211 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 2.5 MiB/s wr, 185 op/s 2024-04-23T14:49:06.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:06 smithi028 ceph-mon[27810]: pgmap v155: 97 pgs: 97 active+clean; 211 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 2.5 MiB/s wr, 185 op/s 2024-04-23T14:49:07.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:06 smithi105 ceph-mon[32555]: pgmap v155: 97 pgs: 97 active+clean; 211 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 2.5 MiB/s wr, 185 op/s 2024-04-23T14:49:07.487 INFO:tasks.workunit.client.0.smithi028.stdout: 1 383859 9.70 MB/sec execute 83 sec latency 23.567 ms 2024-04-23T14:49:08.487 INFO:tasks.workunit.client.0.smithi028.stdout: 1 386126 9.73 MB/sec execute 84 sec latency 19.205 ms 2024-04-23T14:49:08.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:08 smithi038 ceph-mon[32348]: pgmap v156: 97 pgs: 97 active+clean; 217 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 263 op/s 2024-04-23T14:49:08.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:08 smithi028 ceph-mon[27810]: pgmap v156: 97 pgs: 97 active+clean; 217 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 263 op/s 2024-04-23T14:49:09.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:08 smithi105 ceph-mon[32555]: pgmap v156: 97 pgs: 97 active+clean; 217 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.7 MiB/s wr, 263 op/s 2024-04-23T14:49:09.487 INFO:tasks.workunit.client.0.smithi028.stdout: 1 387431 9.69 MB/sec execute 85 sec latency 18.076 ms 2024-04-23T14:49:10.487 INFO:tasks.workunit.client.0.smithi028.stdout: 1 389125 9.66 MB/sec execute 86 sec latency 48.991 ms 2024-04-23T14:49:10.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:10 smithi038 ceph-mon[32348]: pgmap v157: 97 pgs: 97 active+clean; 217 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 189 op/s 2024-04-23T14:49:10.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:10 smithi028 ceph-mon[27810]: pgmap v157: 97 pgs: 97 active+clean; 217 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 189 op/s 2024-04-23T14:49:11.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:10 smithi105 ceph-mon[32555]: pgmap v157: 97 pgs: 97 active+clean; 217 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 189 op/s 2024-04-23T14:49:11.487 INFO:tasks.workunit.client.0.smithi028.stdout: 1 390653 9.69 MB/sec execute 87 sec latency 19.179 ms 2024-04-23T14:49:12.487 INFO:tasks.workunit.client.0.smithi028.stdout: 1 392514 9.65 MB/sec execute 88 sec latency 20.218 ms 2024-04-23T14:49:12.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:12 smithi038 ceph-mon[32348]: pgmap v158: 97 pgs: 97 active+clean; 217 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 189 op/s 2024-04-23T14:49:12.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:12 smithi028 ceph-mon[27810]: pgmap v158: 97 pgs: 97 active+clean; 217 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 189 op/s 2024-04-23T14:49:13.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:12 smithi105 ceph-mon[32555]: pgmap v158: 97 pgs: 97 active+clean; 217 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 189 op/s 2024-04-23T14:49:13.487 INFO:tasks.workunit.client.0.smithi028.stdout: 1 394158 9.68 MB/sec execute 89 sec latency 11.046 ms 2024-04-23T14:49:14.488 INFO:tasks.workunit.client.0.smithi028.stdout: 1 396103 9.65 MB/sec execute 90 sec latency 21.431 ms 2024-04-23T14:49:14.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:14 smithi038 ceph-mon[32348]: pgmap v159: 97 pgs: 97 active+clean; 222 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.9 MiB/s wr, 265 op/s 2024-04-23T14:49:14.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:14 smithi028 ceph-mon[27810]: pgmap v159: 97 pgs: 97 active+clean; 222 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.9 MiB/s wr, 265 op/s 2024-04-23T14:49:15.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:14 smithi105 ceph-mon[32555]: pgmap v159: 97 pgs: 97 active+clean; 222 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.9 MiB/s wr, 265 op/s 2024-04-23T14:49:15.488 INFO:tasks.workunit.client.0.smithi028.stdout: 1 397581 9.68 MB/sec execute 91 sec latency 82.805 ms 2024-04-23T14:49:16.488 INFO:tasks.workunit.client.0.smithi028.stdout: 1 399088 9.63 MB/sec execute 92 sec latency 31.343 ms 2024-04-23T14:49:17.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:16 smithi105 ceph-mon[32555]: pgmap v160: 97 pgs: 97 active+clean; 222 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 154 op/s 2024-04-23T14:49:17.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:16 smithi038 ceph-mon[32348]: pgmap v160: 97 pgs: 97 active+clean; 222 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 154 op/s 2024-04-23T14:49:17.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:16 smithi028 ceph-mon[27810]: pgmap v160: 97 pgs: 97 active+clean; 222 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 154 op/s 2024-04-23T14:49:17.488 INFO:tasks.workunit.client.0.smithi028.stdout: 1 401297 9.68 MB/sec execute 93 sec latency 10.394 ms 2024-04-23T14:49:18.488 INFO:tasks.workunit.client.0.smithi028.stdout: 1 403617 9.69 MB/sec execute 94 sec latency 12.012 ms 2024-04-23T14:49:19.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:18 smithi105 ceph-mon[32555]: pgmap v161: 97 pgs: 97 active+clean; 230 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.1 MiB/s wr, 235 op/s 2024-04-23T14:49:19.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:18 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:49:19.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:18 smithi038 ceph-mon[32348]: pgmap v161: 97 pgs: 97 active+clean; 230 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.1 MiB/s wr, 235 op/s 2024-04-23T14:49:19.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:18 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:49:19.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:18 smithi028 ceph-mon[27810]: pgmap v161: 97 pgs: 97 active+clean; 230 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.1 MiB/s wr, 235 op/s 2024-04-23T14:49:19.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:18 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:49:19.488 INFO:tasks.workunit.client.0.smithi028.stdout: 1 405380 9.68 MB/sec execute 95 sec latency 29.214 ms 2024-04-23T14:49:20.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:19 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:49:20.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:19 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:49:20.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:19 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:49:20.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:19 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:49:20.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:19 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:49:20.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:19 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:49:20.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:19 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:49:20.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:19 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:49:20.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:19 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:49:20.488 INFO:tasks.workunit.client.0.smithi028.stdout: 1 407777 9.72 MB/sec execute 96 sec latency 42.912 ms 2024-04-23T14:49:21.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:20 smithi105 ceph-mon[32555]: pgmap v162: 97 pgs: 97 active+clean; 230 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 2.9 MiB/s wr, 157 op/s 2024-04-23T14:49:21.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:20 smithi038 ceph-mon[32348]: pgmap v162: 97 pgs: 97 active+clean; 230 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 2.9 MiB/s wr, 157 op/s 2024-04-23T14:49:21.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:20 smithi028 ceph-mon[27810]: pgmap v162: 97 pgs: 97 active+clean; 230 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 2.9 MiB/s wr, 157 op/s 2024-04-23T14:49:21.488 INFO:tasks.workunit.client.0.smithi028.stdout: 1 409479 9.73 MB/sec execute 97 sec latency 18.315 ms 2024-04-23T14:49:22.489 INFO:tasks.workunit.client.0.smithi028.stdout: 1 411857 9.77 MB/sec execute 98 sec latency 8.739 ms 2024-04-23T14:49:23.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:22 smithi105 ceph-mon[32555]: pgmap v163: 97 pgs: 97 active+clean; 230 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 2.9 MiB/s wr, 157 op/s 2024-04-23T14:49:23.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:22 smithi038 ceph-mon[32348]: pgmap v163: 97 pgs: 97 active+clean; 230 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 2.9 MiB/s wr, 157 op/s 2024-04-23T14:49:23.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:22 smithi028 ceph-mon[27810]: pgmap v163: 97 pgs: 97 active+clean; 230 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 2.9 MiB/s wr, 157 op/s 2024-04-23T14:49:23.489 INFO:tasks.workunit.client.0.smithi028.stdout: 1 413527 9.73 MB/sec execute 99 sec latency 15.521 ms 2024-04-23T14:49:24.489 INFO:tasks.workunit.client.0.smithi028.stdout: 1 415378 9.76 MB/sec execute 100 sec latency 10.388 ms 2024-04-23T14:49:25.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:24 smithi105 ceph-mon[32555]: pgmap v164: 97 pgs: 97 active+clean; 237 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.2 MiB/s wr, 240 op/s 2024-04-23T14:49:25.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:24 smithi038 ceph-mon[32348]: pgmap v164: 97 pgs: 97 active+clean; 237 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.2 MiB/s wr, 240 op/s 2024-04-23T14:49:25.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:24 smithi028 ceph-mon[27810]: pgmap v164: 97 pgs: 97 active+clean; 237 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.2 MiB/s wr, 240 op/s 2024-04-23T14:49:25.489 INFO:tasks.workunit.client.0.smithi028.stdout: 1 416721 9.72 MB/sec execute 101 sec latency 48.618 ms 2024-04-23T14:49:26.489 INFO:tasks.workunit.client.0.smithi028.stdout: 1 419034 9.76 MB/sec execute 102 sec latency 19.384 ms 2024-04-23T14:49:27.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:26 smithi105 ceph-mon[32555]: pgmap v165: 97 pgs: 97 active+clean; 237 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 163 op/s 2024-04-23T14:49:27.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:26 smithi038 ceph-mon[32348]: pgmap v165: 97 pgs: 97 active+clean; 237 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 163 op/s 2024-04-23T14:49:27.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:26 smithi028 ceph-mon[27810]: pgmap v165: 97 pgs: 97 active+clean; 237 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 163 op/s 2024-04-23T14:49:27.489 INFO:tasks.workunit.client.0.smithi028.stdout: 1 421476 9.78 MB/sec execute 103 sec latency 13.307 ms 2024-04-23T14:49:28.489 INFO:tasks.workunit.client.0.smithi028.stdout: 1 423170 9.78 MB/sec execute 104 sec latency 11.817 ms 2024-04-23T14:49:29.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:28 smithi105 ceph-mon[32555]: pgmap v166: 97 pgs: 97 active+clean; 244 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.2 MiB/s wr, 244 op/s 2024-04-23T14:49:29.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:28 smithi038 ceph-mon[32348]: pgmap v166: 97 pgs: 97 active+clean; 244 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.2 MiB/s wr, 244 op/s 2024-04-23T14:49:29.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:28 smithi028 ceph-mon[27810]: pgmap v166: 97 pgs: 97 active+clean; 244 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.2 MiB/s wr, 244 op/s 2024-04-23T14:49:29.489 INFO:tasks.workunit.client.0.smithi028.stdout: 1 425874 9.84 MB/sec execute 105 sec latency 17.312 ms 2024-04-23T14:49:30.490 INFO:tasks.workunit.client.0.smithi028.stdout: 1 427332 9.80 MB/sec execute 106 sec latency 56.252 ms 2024-04-23T14:49:31.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:30 smithi105 ceph-mon[32555]: pgmap v167: 97 pgs: 97 active+clean; 244 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 164 op/s 2024-04-23T14:49:31.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:30 smithi038 ceph-mon[32348]: pgmap v167: 97 pgs: 97 active+clean; 244 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 164 op/s 2024-04-23T14:49:31.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:30 smithi028 ceph-mon[27810]: pgmap v167: 97 pgs: 97 active+clean; 244 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 164 op/s 2024-04-23T14:49:31.490 INFO:tasks.workunit.client.0.smithi028.stdout: 1 429595 9.84 MB/sec execute 107 sec latency 29.911 ms 2024-04-23T14:49:32.490 INFO:tasks.workunit.client.0.smithi028.stdout: 1 431001 9.80 MB/sec execute 108 sec latency 18.682 ms 2024-04-23T14:49:33.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:32 smithi105 ceph-mon[32555]: pgmap v168: 97 pgs: 97 active+clean; 244 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 164 op/s 2024-04-23T14:49:33.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:32 smithi038 ceph-mon[32348]: pgmap v168: 97 pgs: 97 active+clean; 244 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 164 op/s 2024-04-23T14:49:33.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:32 smithi028 ceph-mon[27810]: pgmap v168: 97 pgs: 97 active+clean; 244 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 164 op/s 2024-04-23T14:49:33.490 INFO:tasks.workunit.client.0.smithi028.stdout: 1 432987 9.83 MB/sec execute 109 sec latency 12.827 ms 2024-04-23T14:49:33.742 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:49:33.742 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:49:33.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.968+0000 7f2997e00640 1 -- 172.21.15.28:0/1146904311 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2998101620 msgr2=0x7f299810e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:33.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.968+0000 7f2997e00640 1 --2- 172.21.15.28:0/1146904311 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2998101620 0x7f299810e250 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2984009900 tx=0x7f298402a400 comp rx=0 tx=0).stop 2024-04-23T14:49:33.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.968+0000 7f2997e00640 1 -- 172.21.15.28:0/1146904311 shutdown_connections 2024-04-23T14:49:33.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.968+0000 7f2997e00640 1 --2- 172.21.15.28:0/1146904311 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2998101620 0x7f299810e250 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:33.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.968+0000 7f2997e00640 1 --2- 172.21.15.28:0/1146904311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f299810e820 0x7f2998110c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:33.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.968+0000 7f2997e00640 1 --2- 172.21.15.28:0/1146904311 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2998100c70 0x7f2998101050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:33.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.968+0000 7f2997e00640 1 -- 172.21.15.28:0/1146904311 >> 172.21.15.28:0/1146904311 conn(0x7f29980fc880 msgr2=0x7f29980feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:33.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.968+0000 7f2997e00640 1 -- 172.21.15.28:0/1146904311 shutdown_connections 2024-04-23T14:49:33.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.968+0000 7f2997e00640 1 -- 172.21.15.28:0/1146904311 wait complete. 2024-04-23T14:49:33.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.969+0000 7f2997e00640 1 Processor -- start 2024-04-23T14:49:33.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.969+0000 7f2997e00640 1 -- start start 2024-04-23T14:49:33.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.970+0000 7f2997e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2998100c70 0x7f29981a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:33.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.970+0000 7f2997e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2998101620 0x7f29981a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:33.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.970+0000 7f2997e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f299810e820 0x7f299819a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:33.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.970+0000 7f2997e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f29981144f0 con 0x7f2998100c70 2024-04-23T14:49:33.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.970+0000 7f2997e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2998114370 con 0x7f299810e820 2024-04-23T14:49:33.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.970+0000 7f2997e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2998114670 con 0x7f2998101620 2024-04-23T14:49:33.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.970+0000 7f2996a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2998100c70 0x7f29981a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:33.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.970+0000 7f2997400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f299810e820 0x7f299819a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:33.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.970+0000 7f2997400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f299810e820 0x7f299819a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:54164/0 (socket says 172.21.15.28:54164) 2024-04-23T14:49:33.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.970+0000 7f2997400640 1 -- 172.21.15.28:0/1908760869 learned_addr learned my addr 172.21.15.28:0/1908760869 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:33.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.970+0000 7f2996a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2998100c70 0x7f29981a0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51758/0 (socket says 172.21.15.28:51758) 2024-04-23T14:49:33.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.970+0000 7f2996000640 1 --2- 172.21.15.28:0/1908760869 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2998101620 0x7f29981a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:33.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.971+0000 7f2997400640 1 -- 172.21.15.28:0/1908760869 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2998101620 msgr2=0x7f29981a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.971+0000 7f2997400640 1 --2- 172.21.15.28:0/1908760869 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2998101620 0x7f29981a0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.971+0000 7f2997400640 1 -- 172.21.15.28:0/1908760869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2998100c70 msgr2=0x7f29981a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.971+0000 7f2997400640 1 --2- 172.21.15.28:0/1908760869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2998100c70 0x7f29981a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.971+0000 7f2997400640 1 -- 172.21.15.28:0/1908760869 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f299819afa0 con 0x7f299810e820 2024-04-23T14:49:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.971+0000 7f2997400640 1 --2- 172.21.15.28:0/1908760869 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f299810e820 0x7f299819a7d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2990009fd0 tx=0x7f299000ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.971+0000 7f298f400640 1 -- 172.21.15.28:0/1908760869 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2990004d20 con 0x7f299810e820 2024-04-23T14:49:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.971+0000 7f298f400640 1 -- 172.21.15.28:0/1908760869 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2990005020 con 0x7f299810e820 2024-04-23T14:49:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.971+0000 7f298f400640 1 -- 172.21.15.28:0/1908760869 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29900116d0 con 0x7f299810e820 2024-04-23T14:49:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.971+0000 7f2997e00640 1 -- 172.21.15.28:0/1908760869 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f299819b290 con 0x7f299810e820 2024-04-23T14:49:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.971+0000 7f2997e00640 1 -- 172.21.15.28:0/1908760869 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29981a74b0 con 0x7f299810e820 2024-04-23T14:49:33.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.974+0000 7f29a0793740 1 Processor -- start 2024-04-23T14:49:33.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.974+0000 7f298f400640 1 -- 172.21.15.28:0/1908760869 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f29900051d0 con 0x7f299810e820 2024-04-23T14:49:33.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.975+0000 7f29a0793740 1 -- start start 2024-04-23T14:49:33.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.975+0000 7f29a0793740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a0b6a1a30 0x563a0b6a1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:33.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.975+0000 7f29a0793740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563a0b6a2350 0x563a0b6ac4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:33.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.975+0000 7f29a0793740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a0b6aca20 0x563a0b6aee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:33.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.975+0000 7f29a0793740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563a0b6071f0 con 0x563a0b6aca20 2024-04-23T14:49:33.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.975+0000 7f29a0793740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563a0b5ad350 con 0x563a0b6a2350 2024-04-23T14:49:33.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.975+0000 7f29a0793740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x563a0b6031b0 con 0x563a0b6a1a30 2024-04-23T14:49:33.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.975+0000 7f2996a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563a0b6a2350 0x563a0b6ac4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:33.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.975+0000 7f2996a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563a0b6a2350 0x563a0b6ac4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:54180/0 (socket says 172.21.15.28:54180) 2024-04-23T14:49:33.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.975+0000 7f2996a00640 1 -- 172.21.15.28:0/1135871681 learned_addr learned my addr 172.21.15.28:0/1135871681 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:33.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.975+0000 7f2996000640 1 --2- 172.21.15.28:0/1135871681 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a0b6a1a30 0x563a0b6a1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:33.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.976+0000 7f2996000640 1 --2- 172.21.15.28:0/1135871681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a0b6aca20 0x563a0b6aee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:33.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.976+0000 7f2996a00640 1 -- 172.21.15.28:0/1135871681 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a0b6a1a30 msgr2=0x563a0b6a1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:33.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.976+0000 7f2996a00640 1 --2- 172.21.15.28:0/1135871681 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a0b6a1a30 0x563a0b6a1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:33.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.976+0000 7f2996a00640 1 -- 172.21.15.28:0/1135871681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a0b6aca20 msgr2=0x563a0b6aee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:33.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.976+0000 7f2996a00640 1 --2- 172.21.15.28:0/1135871681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a0b6aca20 0x563a0b6aee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:33.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.976+0000 7f2996a00640 1 -- 172.21.15.28:0/1135871681 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a0b601f50 con 0x563a0b6a2350 2024-04-23T14:49:33.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.976+0000 7f2996000640 1 --2- 172.21.15.28:0/1135871681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a0b6aca20 0x563a0b6aee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:49:33.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.976+0000 7f2996a00640 1 --2- 172.21.15.28:0/1135871681 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563a0b6a2350 0x563a0b6ac4e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f2988009cf0 tx=0x7f2988011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9450725609c57289 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:33.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.976+0000 7f2983e00640 1 -- 172.21.15.28:0/1135871681 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2988004230 con 0x563a0b6a2350 2024-04-23T14:49:33.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.976+0000 7f2983e00640 1 -- 172.21.15.28:0/1135871681 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29880043d0 con 0x563a0b6a2350 2024-04-23T14:49:33.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.976+0000 7f2983e00640 1 -- 172.21.15.28:0/1135871681 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2988019960 con 0x563a0b6a2350 2024-04-23T14:49:33.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.976+0000 7f298f400640 1 --2- 172.21.15.28:0/1908760869 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f297c0a3de0 0x7f297c0a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:33.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.977+0000 7f298f400640 1 -- 172.21.15.28:0/1908760869 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f29900d0c50 con 0x7f299810e820 2024-04-23T14:49:33.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.977+0000 7f2996000640 1 --2- 172.21.15.28:0/1908760869 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f297c0a3de0 0x7f297c0a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:33.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.977+0000 7f29a0793740 1 -- 172.21.15.28:0/1135871681 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563a0b6a2350 msgr2=0x563a0b6ac4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:33.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.977+0000 7f29a0793740 1 --2- 172.21.15.28:0/1135871681 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563a0b6a2350 0x563a0b6ac4e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f2988009cf0 tx=0x7f2988011810 comp rx=0 tx=0).stop 2024-04-23T14:49:33.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.977+0000 7f29a0793740 1 -- 172.21.15.28:0/1135871681 shutdown_connections 2024-04-23T14:49:33.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.977+0000 7f29a0793740 1 --2- 172.21.15.28:0/1135871681 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563a0b6a2350 0x563a0b6ac4e0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:33.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.977+0000 7f29a0793740 1 --2- 172.21.15.28:0/1135871681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a0b6aca20 0x563a0b6aee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:33.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.977+0000 7f29a0793740 1 --2- 172.21.15.28:0/1135871681 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a0b6a1a30 0x563a0b6a1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:33.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.977+0000 7f29a0793740 1 -- 172.21.15.28:0/1135871681 >> 172.21.15.28:0/1135871681 conn(0x563a0b69dd50 msgr2=0x563a0b69e130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:33.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.977+0000 7f29a0793740 1 -- 172.21.15.28:0/1135871681 shutdown_connections 2024-04-23T14:49:33.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.977+0000 7f29a0793740 1 -- 172.21.15.28:0/1135871681 wait complete. 2024-04-23T14:49:33.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.977+0000 7f2996000640 1 --2- 172.21.15.28:0/1908760869 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f297c0a3de0 0x7f297c0a62a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2984005ab0 tx=0x7f298402a900 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:33.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.978+0000 7f29a0793740 1 -- start start 2024-04-23T14:49:33.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.978+0000 7f29a0793740 1 Processor -- start 2024-04-23T14:49:33.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.978+0000 7f29a0793740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a0b6a1a30 0x563a0b6a99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:33.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.978+0000 7f29a0793740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563a0b6a2350 0x563a0b6a9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:33.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.979+0000 7f29a0793740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a0b6aca20 0x563a0b6aa430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:33.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.979+0000 7f29a0793740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563a0b6af810 con 0x563a0b6a1a30 2024-04-23T14:49:33.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.979+0000 7f29a0793740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563a0b5c0690 con 0x563a0b6a2350 2024-04-23T14:49:33.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.979+0000 7f29a0793740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x563a0b6af990 con 0x563a0b6aca20 2024-04-23T14:49:33.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.979+0000 7f29a0793740 10 client.0 _ll_register_callbacks cb 0x563a0b5fc650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:49:33.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.979+0000 7f2996000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563a0b6a2350 0x563a0b6a9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:33.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.979+0000 7f2996000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563a0b6a2350 0x563a0b6a9ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:54196/0 (socket says 172.21.15.28:54196) 2024-04-23T14:49:33.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.979+0000 7f2997400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a0b6aca20 0x563a0b6aa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:33.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.979+0000 7f2996000640 1 -- 172.21.15.28:0/3629179092 learned_addr learned my addr 172.21.15.28:0/3629179092 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:33.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.979+0000 7f2996a00640 1 --2- 172.21.15.28:0/3629179092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a0b6a1a30 0x563a0b6a99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:33.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.979+0000 7f2997400640 1 -- 172.21.15.28:0/3629179092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563a0b6a2350 msgr2=0x563a0b6a9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:33.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.979+0000 7f2997400640 1 --2- 172.21.15.28:0/3629179092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563a0b6a2350 0x563a0b6a9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:33.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.979+0000 7f2997400640 1 -- 172.21.15.28:0/3629179092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a0b6a1a30 msgr2=0x563a0b6a99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:33.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.979+0000 7f2997400640 1 --2- 172.21.15.28:0/3629179092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a0b6a1a30 0x563a0b6a99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:33.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.979+0000 7f2997400640 1 -- 172.21.15.28:0/3629179092 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a0b5bf920 con 0x563a0b6aca20 2024-04-23T14:49:33.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.979+0000 7f2996a00640 1 --2- 172.21.15.28:0/3629179092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a0b6a1a30 0x563a0b6a99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:49:33.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.979+0000 7f2997400640 1 --2- 172.21.15.28:0/3629179092 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a0b6aca20 0x563a0b6aa430 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f29900ce350 tx=0x7f299009cd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=600a003fe1063b9c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:33.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.979+0000 7f2980c00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:49:33.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.980+0000 7f2980c00640 1 -- 172.21.15.28:0/3629179092 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f299001f030 con 0x563a0b6aca20 2024-04-23T14:49:33.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.980+0000 7f29a0793740 1 -- 172.21.15.28:0/3629179092 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29880043d0 con 0x563a0b6aca20 2024-04-23T14:49:33.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.980+0000 7f2980c00640 1 -- 172.21.15.28:0/3629179092 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f29900bd070 con 0x563a0b6aca20 2024-04-23T14:49:33.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.980+0000 7f2980c00640 1 -- 172.21.15.28:0/3629179092 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f29900a5380 con 0x563a0b6aca20 2024-04-23T14:49:33.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.980+0000 7f29a0793740 10 client.26059 fetch_fsmap learned FSMap version 20 2024-04-23T14:49:33.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.980+0000 7f29a0793740 1 -- 172.21.15.28:0/3629179092 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x563a0b3e37d0 con 0x563a0b6aca20 2024-04-23T14:49:33.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.980+0000 7f2980c00640 1 -- 172.21.15.28:0/3629179092 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29880043d0 con 0x563a0b6aca20 2024-04-23T14:49:33.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.980+0000 7f2980c00640 1 -- 172.21.15.28:0/3629179092 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f299000f040 con 0x563a0b6aca20 2024-04-23T14:49:33.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.981+0000 7f29a0793740 10 client.26059 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:49:33.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.981+0000 7f29a0793740 10 client.26059 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:49:33.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.981+0000 7f29a0793740 20 client.26059 populate_metadata read hostname 'smithi028' 2024-04-23T14:49:33.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.981+0000 7f29a0793740 1 --2- 172.21.15.28:0/3629179092 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x563a0b6bccb0 0x563a0b6bf0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:33.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.981+0000 7f29a0793740 4 client.26059 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:49:33.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.981+0000 7f29a0793740 1 -- 172.21.15.28:0/3629179092 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a0b614c10 con 0x563a0b6bccb0 2024-04-23T14:49:33.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.981+0000 7f2996a00640 1 --2- 172.21.15.28:0/3629179092 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x563a0b6bccb0 0x563a0b6bf0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:33.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.981+0000 7f2996a00640 1 --2- 172.21.15.28:0/3629179092 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x563a0b6bccb0 0x563a0b6bf0e0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8310b8e871767d0b server_cookie=190251236a30fa11 in_seq=0 out_seq=0 2024-04-23T14:49:33.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.981+0000 7f2980c00640 10 client.26059 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:49:33.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.984+0000 7f2980c00640 1 -- 172.21.15.28:0/3629179092 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563a0b614c10 con 0x563a0b6bccb0 2024-04-23T14:49:33.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.984+0000 7f2980c00640 10 client.26059 handle_command_reply: tid=0 2024-04-23T14:49:33.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.984+0000 7f29a0793740 1 client.26059 shutdown 2024-04-23T14:49:33.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.985+0000 7f29a0793740 1 -- 172.21.15.28:0/3629179092 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a0b6aca20 msgr2=0x563a0b6aa430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:33.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.985+0000 7f29a0793740 1 --2- 172.21.15.28:0/3629179092 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a0b6aca20 0x563a0b6aa430 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f29900ce350 tx=0x7f299009cd80 comp rx=0 tx=0).stop 2024-04-23T14:49:33.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.985+0000 7f29a0793740 1 -- 172.21.15.28:0/3629179092 shutdown_connections 2024-04-23T14:49:33.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.985+0000 7f29a0793740 1 --2- 172.21.15.28:0/3629179092 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x563a0b6bccb0 0x563a0b6bf0e0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:33.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.985+0000 7f29a0793740 1 --2- 172.21.15.28:0/3629179092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563a0b6a2350 0x563a0b6a9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:33.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.985+0000 7f29a0793740 1 --2- 172.21.15.28:0/3629179092 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a0b6aca20 0x563a0b6aa430 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:33.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.985+0000 7f29a0793740 1 --2- 172.21.15.28:0/3629179092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a0b6a1a30 0x563a0b6a99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:33.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.985+0000 7f2980c00640 0 client.26059 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:49:33.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.985+0000 7f29a0793740 1 -- 172.21.15.28:0/3629179092 >> 172.21.15.28:0/3629179092 conn(0x563a0b69dd50 msgr2=0x563a0b69cfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:33.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.985+0000 7f29a0793740 1 -- 172.21.15.28:0/3629179092 shutdown_connections 2024-04-23T14:49:33.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.985+0000 7f29a0793740 1 -- 172.21.15.28:0/3629179092 wait complete. 2024-04-23T14:49:33.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:33.985+0000 7f29a0793740 20 client.26059 trim_cache size 0 max 16384 2024-04-23T14:49:34.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.006+0000 7f29a0793740 1 Processor -- start 2024-04-23T14:49:34.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.006+0000 7f29a0793740 1 -- start start 2024-04-23T14:49:34.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.006+0000 7f29a0793740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563a0b6a1a30 0x563a0b6b86d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:34.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.006+0000 7f29a0793740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a0b6a2350 0x563a0b6a8280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:34.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.006+0000 7f29a0793740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a0b6aca20 0x563a0b6aa250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:34.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.006+0000 7f29a0793740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f29981addf0 con 0x563a0b6aca20 2024-04-23T14:49:34.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.006+0000 7f29a0793740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f29981b0670 con 0x563a0b6a1a30 2024-04-23T14:49:34.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.006+0000 7f29a0793740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f29981ae460 con 0x563a0b6a2350 2024-04-23T14:49:34.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.006+0000 7f2996a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563a0b6a1a30 0x563a0b6b86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:34.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.007+0000 7f2996a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563a0b6a1a30 0x563a0b6b86d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:54202/0 (socket says 172.21.15.28:54202) 2024-04-23T14:49:34.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.007+0000 7f2996a00640 1 -- 172.21.15.28:0/1675162456 learned_addr learned my addr 172.21.15.28:0/1675162456 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:34.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.007+0000 7f2996000640 1 --2- 172.21.15.28:0/1675162456 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a0b6a2350 0x563a0b6a8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:34.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.007+0000 7f2997400640 1 --2- 172.21.15.28:0/1675162456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a0b6aca20 0x563a0b6aa250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:34.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.007+0000 7f2996a00640 1 -- 172.21.15.28:0/1675162456 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a0b6a2350 msgr2=0x563a0b6a8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:34.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.007+0000 7f2996a00640 1 --2- 172.21.15.28:0/1675162456 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a0b6a2350 0x563a0b6a8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.007+0000 7f2996a00640 1 -- 172.21.15.28:0/1675162456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a0b6aca20 msgr2=0x563a0b6aa250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:34.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.007+0000 7f2996a00640 1 --2- 172.21.15.28:0/1675162456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a0b6aca20 0x563a0b6aa250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.007+0000 7f2996a00640 1 -- 172.21.15.28:0/1675162456 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a0b3193f0 con 0x563a0b6a1a30 2024-04-23T14:49:34.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.007+0000 7f2997400640 1 --2- 172.21.15.28:0/1675162456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a0b6aca20 0x563a0b6aa250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:49:34.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.007+0000 7f2996a00640 1 --2- 172.21.15.28:0/1675162456 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563a0b6a1a30 0x563a0b6b86d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x563a0b6a2ff0 tx=0x7f2988019420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bb038db2e0b2d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:34.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.007+0000 7f2983400640 1 -- 172.21.15.28:0/1675162456 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298802cb50 con 0x563a0b6a1a30 2024-04-23T14:49:34.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.008+0000 7f2983400640 1 -- 172.21.15.28:0/1675162456 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f298802ccf0 con 0x563a0b6a1a30 2024-04-23T14:49:34.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.008+0000 7f2983400640 1 -- 172.21.15.28:0/1675162456 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2988005620 con 0x563a0b6a1a30 2024-04-23T14:49:34.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.008+0000 7f29a0793740 1 -- 172.21.15.28:0/1675162456 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563a0b6a1a30 msgr2=0x563a0b6b86d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:34.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.008+0000 7f29a0793740 1 --2- 172.21.15.28:0/1675162456 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563a0b6a1a30 0x563a0b6b86d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x563a0b6a2ff0 tx=0x7f2988019420 comp rx=0 tx=0).stop 2024-04-23T14:49:34.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.008+0000 7f29a0793740 1 -- 172.21.15.28:0/1675162456 shutdown_connections 2024-04-23T14:49:34.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.008+0000 7f29a0793740 1 --2- 172.21.15.28:0/1675162456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a0b6aca20 0x563a0b6aa250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.008+0000 7f29a0793740 1 --2- 172.21.15.28:0/1675162456 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a0b6a2350 0x563a0b6a8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.008+0000 7f29a0793740 1 --2- 172.21.15.28:0/1675162456 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563a0b6a1a30 0x563a0b6b86d0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.008+0000 7f29a0793740 1 -- 172.21.15.28:0/1675162456 >> 172.21.15.28:0/1675162456 conn(0x563a0b69dd50 msgr2=0x563a0b6ab1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:34.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.008+0000 7f29a0793740 1 -- 172.21.15.28:0/1675162456 shutdown_connections 2024-04-23T14:49:34.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.009+0000 7f29a0793740 1 -- 172.21.15.28:0/1675162456 wait complete. 2024-04-23T14:49:34.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.009+0000 7f29a0793740 1 -- start start 2024-04-23T14:49:34.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.009+0000 7f29a0793740 1 Processor -- start 2024-04-23T14:49:34.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.010+0000 7f29a0793740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a0b6a1a30 0x563a0b6b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:34.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.010+0000 7f29a0793740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a0b6a2350 0x563a0b6b0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:34.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.010+0000 7f29a0793740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563a0b6aca20 0x563a0b6b0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:34.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.010+0000 7f29a0793740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f29981aeb30 con 0x563a0b6a2350 2024-04-23T14:49:34.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.010+0000 7f29a0793740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f29981affa0 con 0x563a0b6aca20 2024-04-23T14:49:34.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.010+0000 7f29a0793740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f29981af200 con 0x563a0b6a1a30 2024-04-23T14:49:34.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.010+0000 7f29a0793740 10 client.0 _ll_register_callbacks cb 0x7f2990098ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:49:34.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.010+0000 7f2997400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563a0b6aca20 0x563a0b6b0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:34.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.010+0000 7f2997400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563a0b6aca20 0x563a0b6b0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:54208/0 (socket says 172.21.15.28:54208) 2024-04-23T14:49:34.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.010+0000 7f2997400640 1 -- 172.21.15.28:0/1586668196 learned_addr learned my addr 172.21.15.28:0/1586668196 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:34.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.010+0000 7f2996a00640 1 --2- 172.21.15.28:0/1586668196 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a0b6a1a30 0x563a0b6b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:34.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.010+0000 7f2996000640 1 --2- 172.21.15.28:0/1586668196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a0b6a2350 0x563a0b6b0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:34.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.010+0000 7f2997400640 1 -- 172.21.15.28:0/1586668196 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a0b6a1a30 msgr2=0x563a0b6b4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:34.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.010+0000 7f2997400640 1 --2- 172.21.15.28:0/1586668196 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a0b6a1a30 0x563a0b6b4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.010+0000 7f2997400640 1 -- 172.21.15.28:0/1586668196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a0b6a2350 msgr2=0x563a0b6b0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:34.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.010+0000 7f2997400640 1 --2- 172.21.15.28:0/1586668196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a0b6a2350 0x563a0b6b0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.010+0000 7f2997400640 1 -- 172.21.15.28:0/1586668196 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a0b2d90f0 con 0x563a0b6aca20 2024-04-23T14:49:34.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.010+0000 7f2996000640 1 --2- 172.21.15.28:0/1586668196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a0b6a2350 0x563a0b6b0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:49:34.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.011+0000 7f2997400640 1 --2- 172.21.15.28:0/1586668196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563a0b6aca20 0x563a0b6b0890 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f2990098e90 tx=0x7f29900a1010 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da9bbc36e257fda0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:34.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.011+0000 7f2980c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:49:34.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.011+0000 7f2980c00640 1 -- 172.21.15.28:0/1586668196 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f299001f070 con 0x563a0b6aca20 2024-04-23T14:49:34.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.011+0000 7f29a0793740 1 -- 172.21.15.28:0/1586668196 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f298802ccf0 con 0x563a0b6aca20 2024-04-23T14:49:34.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.011+0000 7f2980c00640 1 -- 172.21.15.28:0/1586668196 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f299009c0a0 con 0x563a0b6aca20 2024-04-23T14:49:34.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.011+0000 7f2980c00640 1 -- 172.21.15.28:0/1586668196 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f2990098bf0 con 0x563a0b6aca20 2024-04-23T14:49:34.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.011+0000 7f29a0793740 10 client.26078 fetch_fsmap learned FSMap version 20 2024-04-23T14:49:34.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.011+0000 7f29a0793740 1 -- 172.21.15.28:0/1586668196 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x563a0b5cc810 con 0x563a0b6aca20 2024-04-23T14:49:34.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.011+0000 7f2980c00640 1 -- 172.21.15.28:0/1586668196 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f298802ccf0 con 0x563a0b6aca20 2024-04-23T14:49:34.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.011+0000 7f2980c00640 1 -- 172.21.15.28:0/1586668196 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f29900ceaa0 con 0x563a0b6aca20 2024-04-23T14:49:34.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.011+0000 7f29a0793740 10 client.26078 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:49:34.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.011+0000 7f29a0793740 10 client.26078 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:49:34.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.011+0000 7f29a0793740 20 client.26078 populate_metadata read hostname 'smithi028' 2024-04-23T14:49:34.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.011+0000 7f29a0793740 1 --2- 172.21.15.28:0/1586668196 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x563a0b72bc50 0x563a0b72e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:34.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.011+0000 7f29a0793740 4 client.26078 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:49:34.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.011+0000 7f29a0793740 1 -- 172.21.15.28:0/1586668196 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563a0b60ea80 con 0x563a0b72bc50 2024-04-23T14:49:34.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.012+0000 7f2996a00640 1 --2- 172.21.15.28:0/1586668196 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x563a0b72bc50 0x563a0b72e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:34.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.012+0000 7f2996a00640 1 --2- 172.21.15.28:0/1586668196 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x563a0b72bc50 0x563a0b72e040 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6de043fb059ed081 server_cookie=d51c976dbbc47ae3 in_seq=0 out_seq=0 2024-04-23T14:49:34.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.012+0000 7f2980c00640 10 client.26078 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:49:34.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.016+0000 7f2980c00640 1 -- 172.21.15.28:0/1586668196 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563a0b60ea80 con 0x563a0b72bc50 2024-04-23T14:49:34.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.016+0000 7f2980c00640 10 client.26078 handle_command_reply: tid=0 2024-04-23T14:49:34.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.016+0000 7f29a0793740 1 client.26078 shutdown 2024-04-23T14:49:34.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.016+0000 7f29a0793740 1 -- 172.21.15.28:0/1586668196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563a0b6aca20 msgr2=0x563a0b6b0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:34.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.016+0000 7f29a0793740 1 --2- 172.21.15.28:0/1586668196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563a0b6aca20 0x563a0b6b0890 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f2990098e90 tx=0x7f29900a1010 comp rx=0 tx=0).stop 2024-04-23T14:49:34.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.016+0000 7f29a0793740 1 -- 172.21.15.28:0/1586668196 shutdown_connections 2024-04-23T14:49:34.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.016+0000 7f29a0793740 1 --2- 172.21.15.28:0/1586668196 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x563a0b72bc50 0x563a0b72e040 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.016+0000 7f29a0793740 1 --2- 172.21.15.28:0/1586668196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563a0b6aca20 0x563a0b6b0890 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.016+0000 7f29a0793740 1 --2- 172.21.15.28:0/1586668196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a0b6a2350 0x563a0b6b0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.016+0000 7f29a0793740 1 --2- 172.21.15.28:0/1586668196 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563a0b6a1a30 0x563a0b6b4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.016+0000 7f2980c00640 0 client.26078 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:49:34.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.016+0000 7f29a0793740 1 -- 172.21.15.28:0/1586668196 >> 172.21.15.28:0/1586668196 conn(0x563a0b69dd50 msgr2=0x563a0b6aba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:34.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.016+0000 7f29a0793740 1 -- 172.21.15.28:0/1586668196 shutdown_connections 2024-04-23T14:49:34.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.016+0000 7f29a0793740 1 -- 172.21.15.28:0/1586668196 wait complete. 2024-04-23T14:49:34.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.016+0000 7f29a0793740 20 client.26078 trim_cache size 0 max 16384 2024-04-23T14:49:34.019 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:49:34.020 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-23T14:49:34.020 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-23T14:49:34.020 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:49:34.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.017+0000 7f2997e00640 1 -- 172.21.15.28:0/1908760869 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f297c0a3de0 msgr2=0x7f297c0a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:34.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.017+0000 7f2997e00640 1 --2- 172.21.15.28:0/1908760869 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f297c0a3de0 0x7f297c0a62a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2984005ab0 tx=0x7f298402a900 comp rx=0 tx=0).stop 2024-04-23T14:49:34.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.017+0000 7f2997e00640 1 -- 172.21.15.28:0/1908760869 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f299810e820 msgr2=0x7f299819a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:34.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.017+0000 7f2997e00640 1 --2- 172.21.15.28:0/1908760869 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f299810e820 0x7f299819a7d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2990009fd0 tx=0x7f299000ee60 comp rx=0 tx=0).stop 2024-04-23T14:49:34.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.017+0000 7f2997e00640 1 -- 172.21.15.28:0/1908760869 shutdown_connections 2024-04-23T14:49:34.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.017+0000 7f2997e00640 1 --2- 172.21.15.28:0/1908760869 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f297c0a3de0 0x7f297c0a62a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.017+0000 7f2997e00640 1 --2- 172.21.15.28:0/1908760869 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f299810e820 0x7f299819a7d0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.017+0000 7f2997e00640 1 --2- 172.21.15.28:0/1908760869 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2998101620 0x7f29981a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.017+0000 7f2997e00640 1 --2- 172.21.15.28:0/1908760869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2998100c70 0x7f29981a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.017+0000 7f2997e00640 1 -- 172.21.15.28:0/1908760869 >> 172.21.15.28:0/1908760869 conn(0x7f29980fc880 msgr2=0x7f299810fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:34.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.017+0000 7f2997e00640 1 -- 172.21.15.28:0/1908760869 shutdown_connections 2024-04-23T14:49:34.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.017+0000 7f2997e00640 1 -- 172.21.15.28:0/1908760869 wait complete. 2024-04-23T14:49:34.040 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-23T14:49:34.040 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:49:34.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.270+0000 7f8737800640 1 -- 172.21.15.28:0/3583317360 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8730076e50 msgr2=0x7f8730112930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:34.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.270+0000 7f8737800640 1 --2- 172.21.15.28:0/3583317360 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8730076e50 0x7f8730112930 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f871c009930 tx=0x7f871c02a440 comp rx=0 tx=0).stop 2024-04-23T14:49:34.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.270+0000 7f8737800640 1 -- 172.21.15.28:0/3583317360 shutdown_connections 2024-04-23T14:49:34.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.270+0000 7f8737800640 1 --2- 172.21.15.28:0/3583317360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8730112e70 0x7f8730115260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.270+0000 7f8737800640 1 --2- 172.21.15.28:0/3583317360 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8730076e50 0x7f8730112930 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.270+0000 7f8737800640 1 --2- 172.21.15.28:0/3583317360 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8730076530 0x7f8730076910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.270+0000 7f8737800640 1 -- 172.21.15.28:0/3583317360 >> 172.21.15.28:0/3583317360 conn(0x7f87300ffd00 msgr2=0x7f8730102120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:34.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.270+0000 7f8737800640 1 -- 172.21.15.28:0/3583317360 shutdown_connections 2024-04-23T14:49:34.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.270+0000 7f8737800640 1 -- 172.21.15.28:0/3583317360 wait complete. 2024-04-23T14:49:34.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.271+0000 7f8737800640 1 Processor -- start 2024-04-23T14:49:34.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.271+0000 7f8737800640 1 -- start start 2024-04-23T14:49:34.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.271+0000 7f8737800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8730076530 0x7f873019efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:34.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.272+0000 7f8737800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8730076e50 0x7f873019f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:34.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.272+0000 7f8737800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8730112e70 0x7f87301a38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:34.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.272+0000 7f8737800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8730118a20 con 0x7f8730112e70 2024-04-23T14:49:34.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.272+0000 7f8737800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f87301188a0 con 0x7f8730076e50 2024-04-23T14:49:34.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.272+0000 7f8737800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8730118ba0 con 0x7f8730076530 2024-04-23T14:49:34.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.272+0000 7f8734e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8730076530 0x7f873019efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:34.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.272+0000 7f872fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8730076e50 0x7f873019f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:34.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.272+0000 7f8734e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8730076530 0x7f873019efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:59656/0 (socket says 172.21.15.28:59656) 2024-04-23T14:49:34.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.272+0000 7f872fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8730076e50 0x7f873019f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:54226/0 (socket says 172.21.15.28:54226) 2024-04-23T14:49:34.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.272+0000 7f8734e00640 1 -- 172.21.15.28:0/2998165367 learned_addr learned my addr 172.21.15.28:0/2998165367 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:34.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.272+0000 7f8735800640 1 --2- 172.21.15.28:0/2998165367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8730112e70 0x7f87301a38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:34.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.272+0000 7f8734e00640 1 -- 172.21.15.28:0/2998165367 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8730076e50 msgr2=0x7f873019f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:34.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.272+0000 7f8734e00640 1 --2- 172.21.15.28:0/2998165367 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8730076e50 0x7f873019f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.272+0000 7f8734e00640 1 -- 172.21.15.28:0/2998165367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8730112e70 msgr2=0x7f87301a38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:34.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.272+0000 7f8734e00640 1 --2- 172.21.15.28:0/2998165367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8730112e70 0x7f87301a38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.272+0000 7f8734e00640 1 -- 172.21.15.28:0/2998165367 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87301a3fd0 con 0x7f8730076530 2024-04-23T14:49:34.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.273+0000 7f8734e00640 1 --2- 172.21.15.28:0/2998165367 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8730076530 0x7f873019efe0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f872400ea60 tx=0x7f872400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:34.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.273+0000 7f872d600640 1 -- 172.21.15.28:0/2998165367 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f872400cde0 con 0x7f8730076530 2024-04-23T14:49:34.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.273+0000 7f8737800640 1 -- 172.21.15.28:0/2998165367 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87301a42c0 con 0x7f8730076530 2024-04-23T14:49:34.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.273+0000 7f872d600640 1 -- 172.21.15.28:0/2998165367 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87240035f0 con 0x7f8730076530 2024-04-23T14:49:34.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.273+0000 7f872d600640 1 -- 172.21.15.28:0/2998165367 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8724010670 con 0x7f8730076530 2024-04-23T14:49:34.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.273+0000 7f8737800640 1 -- 172.21.15.28:0/2998165367 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87301abb50 con 0x7f8730076530 2024-04-23T14:49:34.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.275+0000 7f8737800640 1 -- 172.21.15.28:0/2998165367 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8730069270 con 0x7f8730076530 2024-04-23T14:49:34.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.276+0000 7f872d600640 1 -- 172.21.15.28:0/2998165367 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f8724004020 con 0x7f8730076530 2024-04-23T14:49:34.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.278+0000 7f872d600640 1 --2- 172.21.15.28:0/2998165367 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f87180a3d20 0x7f87180a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:34.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.278+0000 7f872d600640 1 -- 172.21.15.28:0/2998165367 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f87240d0c10 con 0x7f8730076530 2024-04-23T14:49:34.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.278+0000 7f872fe00640 1 --2- 172.21.15.28:0/2998165367 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f87180a3d20 0x7f87180a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:34.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.279+0000 7f872fe00640 1 --2- 172.21.15.28:0/2998165367 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f87180a3d20 0x7f87180a61e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f871c006010 tx=0x7f871c000bc0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:34.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.282+0000 7f872d600640 1 -- 172.21.15.28:0/2998165367 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f872409d260 con 0x7f8730076530 2024-04-23T14:49:34.490 INFO:tasks.workunit.client.0.smithi028.stdout: 1 434285 9.79 MB/sec execute 110 sec latency 15.109 ms 2024-04-23T14:49:34.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.558+0000 7f8737800640 1 -- 172.21.15.28:0/2998165367 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f873010e000 con 0x7f8730076530 2024-04-23T14:49:34.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.559+0000 7f872d600640 1 -- 172.21.15.28:0/2998165367 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7f872409cb60 con 0x7f8730076530 2024-04-23T14:49:34.560 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:49:34.560 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:49:34.561 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:49:34.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.561+0000 7f8737800640 1 -- 172.21.15.28:0/2998165367 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f87180a3d20 msgr2=0x7f87180a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:34.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.561+0000 7f8737800640 1 --2- 172.21.15.28:0/2998165367 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f87180a3d20 0x7f87180a61e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f871c006010 tx=0x7f871c000bc0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.561+0000 7f8737800640 1 -- 172.21.15.28:0/2998165367 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8730076530 msgr2=0x7f873019efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:34.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.561+0000 7f8737800640 1 --2- 172.21.15.28:0/2998165367 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8730076530 0x7f873019efe0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f872400ea60 tx=0x7f872400ef20 comp rx=0 tx=0).stop 2024-04-23T14:49:34.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.562+0000 7f8737800640 1 -- 172.21.15.28:0/2998165367 shutdown_connections 2024-04-23T14:49:34.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.562+0000 7f8737800640 1 --2- 172.21.15.28:0/2998165367 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f87180a3d20 0x7f87180a61e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.562+0000 7f8737800640 1 --2- 172.21.15.28:0/2998165367 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8730076e50 0x7f873019f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.562+0000 7f8737800640 1 --2- 172.21.15.28:0/2998165367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8730112e70 0x7f87301a38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.562+0000 7f8737800640 1 --2- 172.21.15.28:0/2998165367 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8730076530 0x7f873019efe0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.562+0000 7f8737800640 1 -- 172.21.15.28:0/2998165367 >> 172.21.15.28:0/2998165367 conn(0x7f87300ffd00 msgr2=0x7f8730114560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:34.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.562+0000 7f8737800640 1 -- 172.21.15.28:0/2998165367 shutdown_connections 2024-04-23T14:49:34.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.562+0000 7f8737800640 1 -- 172.21.15.28:0/2998165367 wait complete. 2024-04-23T14:49:34.585 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-23T14:49:34.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.794+0000 7fe372200640 1 -- 172.21.15.28:0/3104071890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe36c100e00 msgr2=0x7fe36c1011e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:34.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.794+0000 7fe372200640 1 --2- 172.21.15.28:0/3104071890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe36c100e00 0x7fe36c1011e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe35c009860 tx=0x7fe35c02a340 comp rx=0 tx=0).stop 2024-04-23T14:49:34.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.795+0000 7fe372200640 1 -- 172.21.15.28:0/3104071890 shutdown_connections 2024-04-23T14:49:34.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.795+0000 7fe372200640 1 --2- 172.21.15.28:0/3104071890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe36c101720 0x7fe36c10e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.795+0000 7fe372200640 1 --2- 172.21.15.28:0/3104071890 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe36c10e9b0 0x7fe36c110da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.795+0000 7fe372200640 1 --2- 172.21.15.28:0/3104071890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe36c100e00 0x7fe36c1011e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.795+0000 7fe372200640 1 -- 172.21.15.28:0/3104071890 >> 172.21.15.28:0/3104071890 conn(0x7fe36c078000 msgr2=0x7fe36c078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:34.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.795+0000 7fe372200640 1 -- 172.21.15.28:0/3104071890 shutdown_connections 2024-04-23T14:49:34.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.795+0000 7fe372200640 1 -- 172.21.15.28:0/3104071890 wait complete. 2024-04-23T14:49:34.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.796+0000 7fe372200640 1 Processor -- start 2024-04-23T14:49:34.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.796+0000 7fe372200640 1 -- start start 2024-04-23T14:49:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.797+0000 7fe372200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe36c100e00 0x7fe36c1a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.797+0000 7fe372200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe36c101720 0x7fe36c1a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.797+0000 7fe372200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe36c10e9b0 0x7fe36c19a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.797+0000 7fe372200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe36c114610 con 0x7fe36c10e9b0 2024-04-23T14:49:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.797+0000 7fe372200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe36c114490 con 0x7fe36c100e00 2024-04-23T14:49:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.797+0000 7fe372200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe36c114790 con 0x7fe36c101720 2024-04-23T14:49:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.797+0000 7fe36b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe36c100e00 0x7fe36c1a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.797+0000 7fe36b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe36c100e00 0x7fe36c1a0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:54248/0 (socket says 172.21.15.28:54248) 2024-04-23T14:49:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.797+0000 7fe36b400640 1 -- 172.21.15.28:0/4255284435 learned_addr learned my addr 172.21.15.28:0/4255284435 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.797+0000 7fe36be00640 1 --2- 172.21.15.28:0/4255284435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe36c10e9b0 0x7fe36c19a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.798+0000 7fe36aa00640 1 --2- 172.21.15.28:0/4255284435 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe36c101720 0x7fe36c1a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.798+0000 7fe36b400640 1 -- 172.21.15.28:0/4255284435 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe36c101720 msgr2=0x7fe36c1a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.798+0000 7fe36b400640 1 --2- 172.21.15.28:0/4255284435 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe36c101720 0x7fe36c1a0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.798+0000 7fe36b400640 1 -- 172.21.15.28:0/4255284435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe36c10e9b0 msgr2=0x7fe36c19a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.798+0000 7fe36b400640 1 --2- 172.21.15.28:0/4255284435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe36c10e9b0 0x7fe36c19a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.798+0000 7fe36b400640 1 -- 172.21.15.28:0/4255284435 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe36c19b170 con 0x7fe36c100e00 2024-04-23T14:49:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.798+0000 7fe36be00640 1 --2- 172.21.15.28:0/4255284435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe36c10e9b0 0x7fe36c19a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:49:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.798+0000 7fe36b400640 1 --2- 172.21.15.28:0/4255284435 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe36c100e00 0x7fe36c1a0840 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe35c02a8b0 tx=0x7fe35c033d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.798+0000 7fe363e00640 1 -- 172.21.15.28:0/4255284435 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe35c037040 con 0x7fe36c100e00 2024-04-23T14:49:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.798+0000 7fe363e00640 1 -- 172.21.15.28:0/4255284435 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe35c030550 con 0x7fe36c100e00 2024-04-23T14:49:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.798+0000 7fe363e00640 1 -- 172.21.15.28:0/4255284435 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe35c0039f0 con 0x7fe36c100e00 2024-04-23T14:49:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.798+0000 7fe372200640 1 -- 172.21.15.28:0/4255284435 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe36c19b400 con 0x7fe36c100e00 2024-04-23T14:49:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.799+0000 7fe372200640 1 -- 172.21.15.28:0/4255284435 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe36c1a7700 con 0x7fe36c100e00 2024-04-23T14:49:34.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.803+0000 7fe3751c7740 1 Processor -- start 2024-04-23T14:49:34.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.803+0000 7fe363e00640 1 -- 172.21.15.28:0/4255284435 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fe35c042020 con 0x7fe36c100e00 2024-04-23T14:49:34.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.803+0000 7fe3751c7740 1 -- start start 2024-04-23T14:49:34.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.803+0000 7fe3751c7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e068182950 0x55e068182d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:34.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.803+0000 7fe3751c7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e068183270 0x55e06818d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:34.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.803+0000 7fe3751c7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e06818d940 0x55e06818fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:34.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.803+0000 7fe3751c7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e0680e81d0 con 0x55e068182950 2024-04-23T14:49:34.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.803+0000 7fe3751c7740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e06808e450 con 0x55e068183270 2024-04-23T14:49:34.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.803+0000 7fe3751c7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e0680e4190 con 0x55e06818d940 2024-04-23T14:49:34.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.804+0000 7fe36aa00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e06818d940 0x55e06818fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:34.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.804+0000 7fe36be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e068182950 0x55e068182d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:34.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.804+0000 7fe36aa00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e06818d940 0x55e06818fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:59692/0 (socket says 172.21.15.28:59692) 2024-04-23T14:49:34.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.804+0000 7fe36aa00640 1 -- 172.21.15.28:0/2101314798 learned_addr learned my addr 172.21.15.28:0/2101314798 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:34.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.804+0000 7fe36b400640 1 --2- 172.21.15.28:0/2101314798 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e068183270 0x55e06818d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:34.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.804+0000 7fe36aa00640 1 -- 172.21.15.28:0/2101314798 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e068183270 msgr2=0x55e06818d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:34.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.804+0000 7fe36aa00640 1 --2- 172.21.15.28:0/2101314798 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e068183270 0x55e06818d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.804+0000 7fe36aa00640 1 -- 172.21.15.28:0/2101314798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e068182950 msgr2=0x55e068182d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:34.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.804+0000 7fe36aa00640 1 --2- 172.21.15.28:0/2101314798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e068182950 0x55e068182d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.804+0000 7fe36aa00640 1 -- 172.21.15.28:0/2101314798 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e06814b310 con 0x55e06818d940 2024-04-23T14:49:34.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.805+0000 7fe36aa00640 1 --2- 172.21.15.28:0/2101314798 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e06818d940 0x55e06818fd30 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe358009bf0 tx=0x7fe3580117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2518a8ec57924191 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:34.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.805+0000 7fe357e00640 1 -- 172.21.15.28:0/2101314798 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe358005020 con 0x55e06818d940 2024-04-23T14:49:34.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.805+0000 7fe357e00640 1 -- 172.21.15.28:0/2101314798 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3580047a0 con 0x55e06818d940 2024-04-23T14:49:34.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.805+0000 7fe357e00640 1 -- 172.21.15.28:0/2101314798 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe358012740 con 0x55e06818d940 2024-04-23T14:49:34.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.805+0000 7fe363e00640 1 --2- 172.21.15.28:0/4255284435 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe3500a3de0 0x7fe3500a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:34.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.805+0000 7fe363e00640 1 -- 172.21.15.28:0/4255284435 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fe35c0efdd0 con 0x7fe36c100e00 2024-04-23T14:49:34.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.805+0000 7fe36be00640 1 --2- 172.21.15.28:0/4255284435 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe3500a3de0 0x7fe3500a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:34.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.806+0000 7fe3751c7740 1 -- 172.21.15.28:0/2101314798 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e06818d940 msgr2=0x55e06818fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:34.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.806+0000 7fe3751c7740 1 --2- 172.21.15.28:0/2101314798 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e06818d940 0x55e06818fd30 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe358009bf0 tx=0x7fe3580117b0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.806+0000 7fe3751c7740 1 -- 172.21.15.28:0/2101314798 shutdown_connections 2024-04-23T14:49:34.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.806+0000 7fe3751c7740 1 --2- 172.21.15.28:0/2101314798 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e068183270 0x55e06818d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.806+0000 7fe3751c7740 1 --2- 172.21.15.28:0/2101314798 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e06818d940 0x55e06818fd30 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.806+0000 7fe3751c7740 1 --2- 172.21.15.28:0/2101314798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e068182950 0x55e068182d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.806+0000 7fe3751c7740 1 -- 172.21.15.28:0/2101314798 >> 172.21.15.28:0/2101314798 conn(0x55e06817ec70 msgr2=0x55e06817f050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:34.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.806+0000 7fe36be00640 1 --2- 172.21.15.28:0/4255284435 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe3500a3de0 0x7fe3500a62a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe36c06b790 tx=0x7fe364011040 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:34.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.806+0000 7fe3751c7740 1 -- 172.21.15.28:0/2101314798 shutdown_connections 2024-04-23T14:49:34.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.806+0000 7fe3751c7740 1 -- 172.21.15.28:0/2101314798 wait complete. 2024-04-23T14:49:34.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.807+0000 7fe3751c7740 1 -- start start 2024-04-23T14:49:34.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.807+0000 7fe3751c7740 1 Processor -- start 2024-04-23T14:49:34.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.807+0000 7fe3751c7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e068182950 0x55e06818d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:34.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.807+0000 7fe3751c7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e068183270 0x55e06818a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:34.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.807+0000 7fe3751c7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e06818d940 0x55e06818ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:34.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.807+0000 7fe3751c7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e0681907d0 con 0x55e068182950 2024-04-23T14:49:34.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.807+0000 7fe3751c7740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e0680a1810 con 0x55e068183270 2024-04-23T14:49:34.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.807+0000 7fe3751c7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e068190950 con 0x55e06818d940 2024-04-23T14:49:34.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.808+0000 7fe3751c7740 10 client.0 _ll_register_callbacks cb 0x55e06806f140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:49:34.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.808+0000 7fe36aa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e068183270 0x55e06818a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:34.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.808+0000 7fe36aa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e068183270 0x55e06818a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:54270/0 (socket says 172.21.15.28:54270) 2024-04-23T14:49:34.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.808+0000 7fe36aa00640 1 -- 172.21.15.28:0/3574105614 learned_addr learned my addr 172.21.15.28:0/3574105614 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:34.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.808+0000 7fe36b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e068182950 0x55e06818d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:34.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.808+0000 7fe36be00640 1 --2- 172.21.15.28:0/3574105614 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e06818d940 0x55e06818ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:34.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.808+0000 7fe36aa00640 1 -- 172.21.15.28:0/3574105614 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e06818d940 msgr2=0x55e06818ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:34.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.808+0000 7fe36aa00640 1 --2- 172.21.15.28:0/3574105614 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e06818d940 0x55e06818ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.808+0000 7fe36aa00640 1 -- 172.21.15.28:0/3574105614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e068182950 msgr2=0x55e06818d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:34.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.808+0000 7fe36aa00640 1 --2- 172.21.15.28:0/3574105614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e068182950 0x55e06818d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.808+0000 7fe36aa00640 1 -- 172.21.15.28:0/3574105614 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0680b23f0 con 0x55e068183270 2024-04-23T14:49:34.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.808+0000 7fe36aa00640 1 --2- 172.21.15.28:0/3574105614 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e068183270 0x55e06818a610 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe358009970 tx=0x7fe358004680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64cb2ca89a96d1d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:34.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.808+0000 7fe354c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:49:34.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.809+0000 7fe354c00640 1 -- 172.21.15.28:0/3574105614 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe358028020 con 0x55e068183270 2024-04-23T14:49:34.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.809+0000 7fe354c00640 1 -- 172.21.15.28:0/3574105614 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fe358004fe0 con 0x55e068183270 2024-04-23T14:49:34.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.809+0000 7fe3751c7740 1 -- 172.21.15.28:0/3574105614 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3580047a0 con 0x55e068183270 2024-04-23T14:49:34.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.809+0000 7fe354c00640 1 -- 172.21.15.28:0/3574105614 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fe358033050 con 0x55e068183270 2024-04-23T14:49:34.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.809+0000 7fe3751c7740 10 client.26096 fetch_fsmap learned FSMap version 20 2024-04-23T14:49:34.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.809+0000 7fe3751c7740 1 -- 172.21.15.28:0/3574105614 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55e067dfa3f0 con 0x55e068183270 2024-04-23T14:49:34.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.809+0000 7fe354c00640 1 -- 172.21.15.28:0/3574105614 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3580047a0 con 0x55e068183270 2024-04-23T14:49:34.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.810+0000 7fe354c00640 1 -- 172.21.15.28:0/3574105614 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fe358024410 con 0x55e068183270 2024-04-23T14:49:34.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.810+0000 7fe3751c7740 10 client.26096 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:49:34.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.810+0000 7fe3751c7740 10 client.26096 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:49:34.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.810+0000 7fe3751c7740 20 client.26096 populate_metadata read hostname 'smithi028' 2024-04-23T14:49:34.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.810+0000 7fe3751c7740 1 --2- 172.21.15.28:0/3574105614 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55e06819dab0 0x55e06819fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:34.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.810+0000 7fe3751c7740 4 client.26096 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:49:34.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.810+0000 7fe3751c7740 1 -- 172.21.15.28:0/3574105614 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e0680f5bf0 con 0x55e06819dab0 2024-04-23T14:49:34.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.810+0000 7fe36b400640 1 --2- 172.21.15.28:0/3574105614 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55e06819dab0 0x55e06819fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:34.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.811+0000 7fe36b400640 1 --2- 172.21.15.28:0/3574105614 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55e06819dab0 0x55e06819fea0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d885bc3c05fe8bee server_cookie=1f39e4b9a6c7ad73 in_seq=0 out_seq=0 2024-04-23T14:49:34.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.811+0000 7fe354c00640 10 client.26096 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:49:34.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.812+0000 7fe354c00640 1 -- 172.21.15.28:0/3574105614 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e0680f5bf0 con 0x55e06819dab0 2024-04-23T14:49:34.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.812+0000 7fe354c00640 10 client.26096 handle_command_reply: tid=0 2024-04-23T14:49:34.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.812+0000 7fe3751c7740 1 client.26096 shutdown 2024-04-23T14:49:34.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.812+0000 7fe3751c7740 1 -- 172.21.15.28:0/3574105614 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e068183270 msgr2=0x55e06818a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:34.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.812+0000 7fe3751c7740 1 --2- 172.21.15.28:0/3574105614 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e068183270 0x55e06818a610 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe358009970 tx=0x7fe358004680 comp rx=0 tx=0).stop 2024-04-23T14:49:34.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.812+0000 7fe3751c7740 1 -- 172.21.15.28:0/3574105614 shutdown_connections 2024-04-23T14:49:34.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.812+0000 7fe3751c7740 1 --2- 172.21.15.28:0/3574105614 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55e06819dab0 0x55e06819fea0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.812+0000 7fe3751c7740 1 --2- 172.21.15.28:0/3574105614 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e068183270 0x55e06818a610 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.812+0000 7fe3751c7740 1 --2- 172.21.15.28:0/3574105614 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e06818d940 0x55e06818ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.812+0000 7fe3751c7740 1 --2- 172.21.15.28:0/3574105614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e068182950 0x55e06818d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.812+0000 7fe3751c7740 1 -- 172.21.15.28:0/3574105614 >> 172.21.15.28:0/3574105614 conn(0x55e06817ec70 msgr2=0x55e068184970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:34.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.812+0000 7fe354c00640 0 client.26096 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:49:34.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.812+0000 7fe3751c7740 1 -- 172.21.15.28:0/3574105614 shutdown_connections 2024-04-23T14:49:34.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.812+0000 7fe3751c7740 1 -- 172.21.15.28:0/3574105614 wait complete. 2024-04-23T14:49:34.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.813+0000 7fe3751c7740 20 client.26096 trim_cache size 0 max 16384 2024-04-23T14:49:34.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.829+0000 7fe3751c7740 1 Processor -- start 2024-04-23T14:49:34.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.829+0000 7fe3751c7740 1 -- start start 2024-04-23T14:49:34.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.829+0000 7fe3751c7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e068182950 0x55e0681996f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:34.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.829+0000 7fe3751c7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06818d940 0x55e0681895f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:34.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.829+0000 7fe3751c7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e06818ce50 0x55e06818d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:34.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.829+0000 7fe3751c7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe36c1adfe0 con 0x55e06818d940 2024-04-23T14:49:34.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.829+0000 7fe3751c7740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe36c1b0860 con 0x55e06818ce50 2024-04-23T14:49:34.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.829+0000 7fe3751c7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe36c1ae650 con 0x55e068182950 2024-04-23T14:49:34.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.829+0000 7fe36be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e06818ce50 0x55e06818d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:34.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.829+0000 7fe36b400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e068182950 0x55e0681996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:34.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.829+0000 7fe36be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e06818ce50 0x55e06818d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:54284/0 (socket says 172.21.15.28:54284) 2024-04-23T14:49:34.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.829+0000 7fe36aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06818d940 0x55e0681895f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:34.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.829+0000 7fe36be00640 1 -- 172.21.15.28:0/2106046845 learned_addr learned my addr 172.21.15.28:0/2106046845 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:34.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.830+0000 7fe36be00640 1 -- 172.21.15.28:0/2106046845 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e068182950 msgr2=0x55e0681996f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:34.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.830+0000 7fe36be00640 1 --2- 172.21.15.28:0/2106046845 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e068182950 0x55e0681996f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.830+0000 7fe36be00640 1 -- 172.21.15.28:0/2106046845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06818d940 msgr2=0x55e0681895f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:34.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.830+0000 7fe36be00640 1 --2- 172.21.15.28:0/2106046845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06818d940 0x55e0681895f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.830+0000 7fe36be00640 1 -- 172.21.15.28:0/2106046845 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e067ec47d0 con 0x55e06818ce50 2024-04-23T14:49:34.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.830+0000 7fe36b400640 1 --2- 172.21.15.28:0/2106046845 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e068182950 0x55e0681996f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:49:34.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.830+0000 7fe36be00640 1 --2- 172.21.15.28:0/2106046845 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e06818ce50 0x55e06818d2b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe36401b6a0 tx=0x7fe364017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7419dd04cb9531c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:34.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.830+0000 7fe357400640 1 -- 172.21.15.28:0/2106046845 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe364021040 con 0x55e06818ce50 2024-04-23T14:49:34.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.830+0000 7fe357400640 1 -- 172.21.15.28:0/2106046845 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe364003e50 con 0x55e06818ce50 2024-04-23T14:49:34.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.830+0000 7fe357400640 1 -- 172.21.15.28:0/2106046845 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe364019880 con 0x55e06818ce50 2024-04-23T14:49:34.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.831+0000 7fe3751c7740 1 -- 172.21.15.28:0/2106046845 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e06818ce50 msgr2=0x55e06818d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:34.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.831+0000 7fe3751c7740 1 --2- 172.21.15.28:0/2106046845 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e06818ce50 0x55e06818d2b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe36401b6a0 tx=0x7fe364017000 comp rx=0 tx=0).stop 2024-04-23T14:49:34.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.831+0000 7fe3751c7740 1 -- 172.21.15.28:0/2106046845 shutdown_connections 2024-04-23T14:49:34.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.831+0000 7fe3751c7740 1 --2- 172.21.15.28:0/2106046845 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e06818ce50 0x55e06818d2b0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.831+0000 7fe3751c7740 1 --2- 172.21.15.28:0/2106046845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06818d940 0x55e0681895f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.831+0000 7fe3751c7740 1 --2- 172.21.15.28:0/2106046845 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e068182950 0x55e0681996f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.831+0000 7fe3751c7740 1 -- 172.21.15.28:0/2106046845 >> 172.21.15.28:0/2106046845 conn(0x55e06817ec70 msgr2=0x55e068188ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:34.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.831+0000 7fe3751c7740 1 -- 172.21.15.28:0/2106046845 shutdown_connections 2024-04-23T14:49:34.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.831+0000 7fe3751c7740 1 -- 172.21.15.28:0/2106046845 wait complete. 2024-04-23T14:49:34.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.832+0000 7fe3751c7740 1 -- start start 2024-04-23T14:49:34.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.832+0000 7fe3751c7740 1 Processor -- start 2024-04-23T14:49:34.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.833+0000 7fe3751c7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e068182950 0x55e068196120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:34.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.833+0000 7fe3751c7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06818ce50 0x55e068187a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:34.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.833+0000 7fe3751c7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e06818d940 0x55e068188190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:34.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.833+0000 7fe3751c7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe36c1aed20 con 0x55e06818ce50 2024-04-23T14:49:34.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.833+0000 7fe3751c7740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe36c1b0190 con 0x55e068182950 2024-04-23T14:49:34.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.833+0000 7fe3751c7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe36c1af3f0 con 0x55e06818d940 2024-04-23T14:49:34.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.833+0000 7fe3751c7740 10 client.0 _ll_register_callbacks cb 0x7fe3640099e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:49:34.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.833+0000 7fe36b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e068182950 0x55e068196120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:34.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.833+0000 7fe36b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e068182950 0x55e068196120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:54290/0 (socket says 172.21.15.28:54290) 2024-04-23T14:49:34.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.833+0000 7fe36b400640 1 -- 172.21.15.28:0/731059860 learned_addr learned my addr 172.21.15.28:0/731059860 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:34.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.833+0000 7fe36aa00640 1 --2- 172.21.15.28:0/731059860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06818ce50 0x55e068187a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:34.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.833+0000 7fe36be00640 1 --2- 172.21.15.28:0/731059860 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e06818d940 0x55e068188190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:34.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.833+0000 7fe36b400640 1 -- 172.21.15.28:0/731059860 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e06818d940 msgr2=0x55e068188190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:34.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.833+0000 7fe36b400640 1 --2- 172.21.15.28:0/731059860 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e06818d940 0x55e068188190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.833+0000 7fe36b400640 1 -- 172.21.15.28:0/731059860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06818ce50 msgr2=0x55e068187a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:34.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.833+0000 7fe36b400640 1 --2- 172.21.15.28:0/731059860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06818ce50 0x55e068187a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.833+0000 7fe36b400640 1 -- 172.21.15.28:0/731059860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e06818e690 con 0x55e068182950 2024-04-23T14:49:34.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.834+0000 7fe36b400640 1 --2- 172.21.15.28:0/731059860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e068182950 0x55e068196120 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe35c0b8290 tx=0x7fe35c032820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53586c52d6ff2515 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:34.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.834+0000 7fe354c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:49:34.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.834+0000 7fe354c00640 1 -- 172.21.15.28:0/731059860 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe35c0d5060 con 0x55e068182950 2024-04-23T14:49:34.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.834+0000 7fe354c00640 1 -- 172.21.15.28:0/731059860 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fe35c0b84a0 con 0x55e068182950 2024-04-23T14:49:34.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.834+0000 7fe3751c7740 1 -- 172.21.15.28:0/731059860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe364003e50 con 0x55e068182950 2024-04-23T14:49:34.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.834+0000 7fe354c00640 1 -- 172.21.15.28:0/731059860 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fe35c0bb790 con 0x55e068182950 2024-04-23T14:49:34.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.834+0000 7fe3751c7740 10 client.26102 fetch_fsmap learned FSMap version 20 2024-04-23T14:49:34.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.834+0000 7fe3751c7740 1 -- 172.21.15.28:0/731059860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55e068204060 con 0x55e068182950 2024-04-23T14:49:34.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.835+0000 7fe354c00640 1 -- 172.21.15.28:0/731059860 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe364003e50 con 0x55e068182950 2024-04-23T14:49:34.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.835+0000 7fe354c00640 1 -- 172.21.15.28:0/731059860 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fe35c03a070 con 0x55e068182950 2024-04-23T14:49:34.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.835+0000 7fe3751c7740 10 client.26102 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:49:34.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.835+0000 7fe3751c7740 10 client.26102 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:49:34.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.835+0000 7fe3751c7740 20 client.26102 populate_metadata read hostname 'smithi028' 2024-04-23T14:49:34.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.835+0000 7fe3751c7740 1 --2- 172.21.15.28:0/731059860 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55e068204710 0x55e068206b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:34.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.835+0000 7fe3751c7740 4 client.26102 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-23T14:49:34.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.835+0000 7fe3751c7740 1 -- 172.21.15.28:0/731059860 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e0680efa60 con 0x55e068204710 2024-04-23T14:49:34.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.835+0000 7fe36aa00640 1 --2- 172.21.15.28:0/731059860 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55e068204710 0x55e068206b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:34.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.836+0000 7fe36aa00640 1 --2- 172.21.15.28:0/731059860 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55e068204710 0x55e068206b30 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=420febb13e398f1 server_cookie=a6a9ea8d2026abe7 in_seq=0 out_seq=0 2024-04-23T14:49:34.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.836+0000 7fe354c00640 10 client.26102 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:49:34.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.838+0000 7fe354c00640 1 -- 172.21.15.28:0/731059860 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e0680efa60 con 0x55e068204710 2024-04-23T14:49:34.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.838+0000 7fe354c00640 10 client.26102 handle_command_reply: tid=0 2024-04-23T14:49:34.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.838+0000 7fe3751c7740 1 client.26102 shutdown 2024-04-23T14:49:34.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.838+0000 7fe3751c7740 1 -- 172.21.15.28:0/731059860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e068182950 msgr2=0x55e068196120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:34.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.838+0000 7fe3751c7740 1 --2- 172.21.15.28:0/731059860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e068182950 0x55e068196120 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe35c0b8290 tx=0x7fe35c032820 comp rx=0 tx=0).stop 2024-04-23T14:49:34.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.838+0000 7fe3751c7740 1 -- 172.21.15.28:0/731059860 shutdown_connections 2024-04-23T14:49:34.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.838+0000 7fe3751c7740 1 --2- 172.21.15.28:0/731059860 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55e068204710 0x55e068206b30 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.838+0000 7fe3751c7740 1 --2- 172.21.15.28:0/731059860 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e06818d940 0x55e068188190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.838+0000 7fe3751c7740 1 --2- 172.21.15.28:0/731059860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06818ce50 0x55e068187a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.838+0000 7fe3751c7740 1 --2- 172.21.15.28:0/731059860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e068182950 0x55e068196120 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.838+0000 7fe354c00640 0 client.26102 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:49:34.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.838+0000 7fe3751c7740 1 -- 172.21.15.28:0/731059860 >> 172.21.15.28:0/731059860 conn(0x55e06817ec70 msgr2=0x55e06818be40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:34.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.838+0000 7fe3751c7740 1 -- 172.21.15.28:0/731059860 shutdown_connections 2024-04-23T14:49:34.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.838+0000 7fe3751c7740 1 -- 172.21.15.28:0/731059860 wait complete. 2024-04-23T14:49:34.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.838+0000 7fe3751c7740 20 client.26102 trim_cache size 0 max 16384 2024-04-23T14:49:34.842 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-23T14:49:34.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.839+0000 7fe372200640 1 -- 172.21.15.28:0/4255284435 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe3500a3de0 msgr2=0x7fe3500a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:34.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.839+0000 7fe372200640 1 --2- 172.21.15.28:0/4255284435 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe3500a3de0 0x7fe3500a62a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe36c06b790 tx=0x7fe364011040 comp rx=0 tx=0).stop 2024-04-23T14:49:34.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.839+0000 7fe372200640 1 -- 172.21.15.28:0/4255284435 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe36c100e00 msgr2=0x7fe36c1a0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:34.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.839+0000 7fe372200640 1 --2- 172.21.15.28:0/4255284435 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe36c100e00 0x7fe36c1a0840 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe35c02a8b0 tx=0x7fe35c033d10 comp rx=0 tx=0).stop 2024-04-23T14:49:34.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.839+0000 7fe372200640 1 -- 172.21.15.28:0/4255284435 shutdown_connections 2024-04-23T14:49:34.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.839+0000 7fe372200640 1 --2- 172.21.15.28:0/4255284435 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe3500a3de0 0x7fe3500a62a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.839+0000 7fe372200640 1 --2- 172.21.15.28:0/4255284435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe36c10e9b0 0x7fe36c19a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.839+0000 7fe372200640 1 --2- 172.21.15.28:0/4255284435 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe36c101720 0x7fe36c1a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.839+0000 7fe372200640 1 --2- 172.21.15.28:0/4255284435 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe36c100e00 0x7fe36c1a0840 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:34.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.839+0000 7fe372200640 1 -- 172.21.15.28:0/4255284435 >> 172.21.15.28:0/4255284435 conn(0x7fe36c078000 msgr2=0x7fe36c110050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:34.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.839+0000 7fe372200640 1 -- 172.21.15.28:0/4255284435 shutdown_connections 2024-04-23T14:49:34.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:34.839+0000 7fe372200640 1 -- 172.21.15.28:0/4255284435 wait complete. 2024-04-23T14:49:34.860 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:34 smithi028 ceph-mon[27810]: pgmap v169: 97 pgs: 97 active+clean; 251 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.1 MiB/s wr, 248 op/s 2024-04-23T14:49:34.860 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:34 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2998165367' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:49:34.923 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:34 smithi038 ceph-mon[32348]: pgmap v169: 97 pgs: 97 active+clean; 251 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.1 MiB/s wr, 248 op/s 2024-04-23T14:49:34.924 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:34 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2998165367' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:49:35.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:34 smithi105 ceph-mon[32555]: pgmap v169: 97 pgs: 97 active+clean; 251 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.1 MiB/s wr, 248 op/s 2024-04-23T14:49:35.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:34 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2998165367' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:49:35.490 INFO:tasks.workunit.client.0.smithi028.stdout: 1 436307 9.82 MB/sec execute 111 sec latency 54.902 ms 2024-04-23T14:49:35.861 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-23T14:49:35.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:49:35.862 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-23T14:49:36.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.073+0000 7f04b3200640 1 -- 172.21.15.28:0/77021293 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04b4101aa0 msgr2=0x7f04b410dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:36.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.073+0000 7f04b3200640 1 --2- 172.21.15.28:0/77021293 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04b4101aa0 0x7f04b410dc70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f04ac009ca0 tx=0x7f04ac02a8c0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.074+0000 7f04b3200640 1 -- 172.21.15.28:0/77021293 shutdown_connections 2024-04-23T14:49:36.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.074+0000 7f04b3200640 1 --2- 172.21.15.28:0/77021293 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04b4101aa0 0x7f04b410dc70 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.074+0000 7f04b3200640 1 --2- 172.21.15.28:0/77021293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04b41010e0 0x7f04b4101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.074+0000 7f04b3200640 1 --2- 172.21.15.28:0/77021293 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04b4106ed0 0x7f04b41072b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.074+0000 7f04b3200640 1 -- 172.21.15.28:0/77021293 >> 172.21.15.28:0/77021293 conn(0x7f04b4077c90 msgr2=0x7f04b40780a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:36.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.074+0000 7f04b3200640 1 -- 172.21.15.28:0/77021293 shutdown_connections 2024-04-23T14:49:36.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.074+0000 7f04b3200640 1 -- 172.21.15.28:0/77021293 wait complete. 2024-04-23T14:49:36.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.075+0000 7f04b3200640 1 Processor -- start 2024-04-23T14:49:36.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.075+0000 7f04b3200640 1 -- start start 2024-04-23T14:49:36.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.075+0000 7f04b3200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04b41010e0 0x7f04b4102aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.075+0000 7f04b3200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04b4101aa0 0x7f04b4102fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.075+0000 7f04b3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04b4106ed0 0x7f04b4103520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.075+0000 7f04b3200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f04b4111180 con 0x7f04b4106ed0 2024-04-23T14:49:36.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.075+0000 7f04b3200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f04b4111000 con 0x7f04b41010e0 2024-04-23T14:49:36.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.075+0000 7f04b3200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f04b4111300 con 0x7f04b4101aa0 2024-04-23T14:49:36.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.075+0000 7f04b1e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04b41010e0 0x7f04b4102aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.076+0000 7f04b1e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04b41010e0 0x7f04b4102aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:54300/0 (socket says 172.21.15.28:54300) 2024-04-23T14:49:36.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.076+0000 7f04b1e00640 1 -- 172.21.15.28:0/242011022 learned_addr learned my addr 172.21.15.28:0/242011022 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:36.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.076+0000 7f04b2800640 1 --2- 172.21.15.28:0/242011022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04b4106ed0 0x7f04b4103520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.076+0000 7f04abe00640 1 --2- 172.21.15.28:0/242011022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04b4101aa0 0x7f04b4102fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.076+0000 7f04abe00640 1 -- 172.21.15.28:0/242011022 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04b41010e0 msgr2=0x7f04b4102aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:36.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.076+0000 7f04abe00640 1 --2- 172.21.15.28:0/242011022 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04b41010e0 0x7f04b4102aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.076+0000 7f04abe00640 1 -- 172.21.15.28:0/242011022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04b4106ed0 msgr2=0x7f04b4103520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:36.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.076+0000 7f04abe00640 1 --2- 172.21.15.28:0/242011022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04b4106ed0 0x7f04b4103520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.076+0000 7f04abe00640 1 -- 172.21.15.28:0/242011022 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04b41a6be0 con 0x7f04b4101aa0 2024-04-23T14:49:36.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.076+0000 7f04b1e00640 1 --2- 172.21.15.28:0/242011022 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04b41010e0 0x7f04b4102aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:49:36.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.076+0000 7f04abe00640 1 --2- 172.21.15.28:0/242011022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04b4101aa0 0x7f04b4102fe0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f049c00d6d0 tx=0x7f049c00db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:36.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.077+0000 7f04aaa00640 1 -- 172.21.15.28:0/242011022 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049c017070 con 0x7f04b4101aa0 2024-04-23T14:49:36.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.077+0000 7f04aaa00640 1 -- 172.21.15.28:0/242011022 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f049c009820 con 0x7f04b4101aa0 2024-04-23T14:49:36.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.077+0000 7f04aaa00640 1 -- 172.21.15.28:0/242011022 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049c004d00 con 0x7f04b4101aa0 2024-04-23T14:49:36.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.077+0000 7f04b3200640 1 -- 172.21.15.28:0/242011022 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04b41a6ed0 con 0x7f04b4101aa0 2024-04-23T14:49:36.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.077+0000 7f04b3200640 1 -- 172.21.15.28:0/242011022 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04b41a73e0 con 0x7f04b4101aa0 2024-04-23T14:49:36.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.080+0000 7f04aaa00640 1 -- 172.21.15.28:0/242011022 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f049c0099d0 con 0x7f04b4101aa0 2024-04-23T14:49:36.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.080+0000 7f04bbe7a740 1 Processor -- start 2024-04-23T14:49:36.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.080+0000 7f04bbe7a740 1 -- start start 2024-04-23T14:49:36.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.081+0000 7f04bbe7a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e23da89270 0x55e23da89650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.081+0000 7f04bbe7a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e23da89b90 0x55e23da93d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.081+0000 7f04bbe7a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e23da94260 0x55e23da96650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.081+0000 7f04bbe7a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e23d9eee70 con 0x55e23da89b90 2024-04-23T14:49:36.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.081+0000 7f04bbe7a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e23d995380 con 0x55e23da89270 2024-04-23T14:49:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.081+0000 7f04bbe7a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e23d9eae30 con 0x55e23da94260 2024-04-23T14:49:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.081+0000 7f04b2800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e23da89270 0x55e23da89650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.081+0000 7f04b1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e23da89b90 0x55e23da93d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.081+0000 7f04b2800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e23da89270 0x55e23da89650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:54310/0 (socket says 172.21.15.28:54310) 2024-04-23T14:49:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.081+0000 7f04b2800640 1 -- 172.21.15.28:0/2242823968 learned_addr learned my addr 172.21.15.28:0/2242823968 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.081+0000 7f04abe00640 1 --2- 172.21.15.28:0/2242823968 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e23da94260 0x55e23da96650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.081+0000 7f04b2800640 1 -- 172.21.15.28:0/2242823968 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e23da94260 msgr2=0x55e23da96650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.081+0000 7f04b2800640 1 --2- 172.21.15.28:0/2242823968 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e23da94260 0x55e23da96650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.081+0000 7f04b2800640 1 -- 172.21.15.28:0/2242823968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e23da89b90 msgr2=0x55e23da93d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.081+0000 7f04b2800640 1 --2- 172.21.15.28:0/2242823968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e23da89b90 0x55e23da93d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.081+0000 7f04b2800640 1 -- 172.21.15.28:0/2242823968 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e23d7c97e0 con 0x55e23da89270 2024-04-23T14:49:36.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.081+0000 7f04b1e00640 1 --2- 172.21.15.28:0/2242823968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e23da89b90 0x55e23da93d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:49:36.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.081+0000 7f04abe00640 1 --2- 172.21.15.28:0/2242823968 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e23da94260 0x55e23da96650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:49:36.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.082+0000 7f04b2800640 1 --2- 172.21.15.28:0/2242823968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e23da89270 0x55e23da89650 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f04ac035000 tx=0x7f04ac005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b4e92a3b9c1c413 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:36.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.082+0000 7f04aaa00640 1 --2- 172.21.15.28:0/242011022 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f04980a3da0 0x7f04980a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.082+0000 7f04a3400640 1 -- 172.21.15.28:0/2242823968 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04ac031da0 con 0x55e23da89270 2024-04-23T14:49:36.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.082+0000 7f04a3400640 1 -- 172.21.15.28:0/2242823968 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04ac036030 con 0x55e23da89270 2024-04-23T14:49:36.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.082+0000 7f04a3400640 1 -- 172.21.15.28:0/2242823968 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04ac004330 con 0x55e23da89270 2024-04-23T14:49:36.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.082+0000 7f04aaa00640 1 -- 172.21.15.28:0/242011022 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f049c0d0280 con 0x7f04b4101aa0 2024-04-23T14:49:36.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.082+0000 7f04b1e00640 1 --2- 172.21.15.28:0/242011022 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f04980a3da0 0x7f04980a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.083+0000 7f04bbe7a740 1 -- 172.21.15.28:0/2242823968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e23da89270 msgr2=0x55e23da89650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:36.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.083+0000 7f04bbe7a740 1 --2- 172.21.15.28:0/2242823968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e23da89270 0x55e23da89650 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f04ac035000 tx=0x7f04ac005ef0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.083+0000 7f04bbe7a740 1 -- 172.21.15.28:0/2242823968 shutdown_connections 2024-04-23T14:49:36.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.083+0000 7f04bbe7a740 1 --2- 172.21.15.28:0/2242823968 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e23da94260 0x55e23da96650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.083+0000 7f04bbe7a740 1 --2- 172.21.15.28:0/2242823968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e23da89b90 0x55e23da93d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.083+0000 7f04bbe7a740 1 --2- 172.21.15.28:0/2242823968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e23da89270 0x55e23da89650 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.083+0000 7f04bbe7a740 1 -- 172.21.15.28:0/2242823968 >> 172.21.15.28:0/2242823968 conn(0x55e23da85590 msgr2=0x55e23da85970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:36.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.083+0000 7f04b1e00640 1 --2- 172.21.15.28:0/242011022 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f04980a3da0 0x7f04980a6260 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x55e23da8a830 tx=0x7f04a4005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:36.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.083+0000 7f04bbe7a740 1 -- 172.21.15.28:0/2242823968 shutdown_connections 2024-04-23T14:49:36.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.083+0000 7f04bbe7a740 1 -- 172.21.15.28:0/2242823968 wait complete. 2024-04-23T14:49:36.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.084+0000 7f04bbe7a740 1 -- start start 2024-04-23T14:49:36.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.084+0000 7f04bbe7a740 1 Processor -- start 2024-04-23T14:49:36.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.084+0000 7f04bbe7a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e23da89270 0x55e23da91220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.084+0000 7f04bbe7a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e23da89b90 0x55e23da91760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.084+0000 7f04bbe7a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e23da94260 0x55e23da91ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.084+0000 7f04bbe7a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e23da96e70 con 0x55e23da89270 2024-04-23T14:49:36.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.085+0000 7f04bbe7a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e23d9a8660 con 0x55e23da89b90 2024-04-23T14:49:36.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.085+0000 7f04bbe7a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e23da96ff0 con 0x55e23da94260 2024-04-23T14:49:36.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.085+0000 7f04bbe7a740 10 client.0 _ll_register_callbacks cb 0x55e23d996aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:49:36.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.085+0000 7f04abe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e23da89b90 0x55e23da91760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.085+0000 7f04b1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e23da89270 0x55e23da91220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.085+0000 7f04abe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e23da89b90 0x55e23da91760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:54320/0 (socket says 172.21.15.28:54320) 2024-04-23T14:49:36.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.085+0000 7f04abe00640 1 -- 172.21.15.28:0/931473106 learned_addr learned my addr 172.21.15.28:0/931473106 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:36.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.085+0000 7f04b2800640 1 --2- 172.21.15.28:0/931473106 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e23da94260 0x55e23da91ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.085+0000 7f04abe00640 1 -- 172.21.15.28:0/931473106 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e23da94260 msgr2=0x55e23da91ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:36.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.085+0000 7f04abe00640 1 --2- 172.21.15.28:0/931473106 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e23da94260 0x55e23da91ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.085+0000 7f04abe00640 1 -- 172.21.15.28:0/931473106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e23da89270 msgr2=0x55e23da91220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:36.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.085+0000 7f04abe00640 1 --2- 172.21.15.28:0/931473106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e23da89270 0x55e23da91220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.085+0000 7f04abe00640 1 -- 172.21.15.28:0/931473106 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e23d829f70 con 0x55e23da89b90 2024-04-23T14:49:36.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.085+0000 7f04abe00640 1 --2- 172.21.15.28:0/931473106 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e23da89b90 0x55e23da91760 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f049c0cd9e0 tx=0x7f049c09c140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9305b83c6d2d1e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:36.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.085+0000 7f0497e00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:49:36.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.086+0000 7f0497e00640 1 -- 172.21.15.28:0/931473106 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049c0d0030 con 0x55e23da89b90 2024-04-23T14:49:36.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.086+0000 7f0497e00640 1 -- 172.21.15.28:0/931473106 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f049c0988f0 con 0x55e23da89b90 2024-04-23T14:49:36.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.086+0000 7f04bbe7a740 1 -- 172.21.15.28:0/931473106 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04ac036030 con 0x55e23da89b90 2024-04-23T14:49:36.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.086+0000 7f0497e00640 1 -- 172.21.15.28:0/931473106 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f049c020040 con 0x55e23da89b90 2024-04-23T14:49:36.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.086+0000 7f04bbe7a740 10 client.26114 fetch_fsmap learned FSMap version 20 2024-04-23T14:49:36.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.086+0000 7f04bbe7a740 1 -- 172.21.15.28:0/931473106 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55e23d6c1170 con 0x55e23da89b90 2024-04-23T14:49:36.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.086+0000 7f0497e00640 1 -- 172.21.15.28:0/931473106 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04ac036030 con 0x55e23da89b90 2024-04-23T14:49:36.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.086+0000 7f0497e00640 1 -- 172.21.15.28:0/931473106 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f049c0a0110 con 0x55e23da89b90 2024-04-23T14:49:36.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.086+0000 7f04bbe7a740 10 client.26114 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:49:36.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.086+0000 7f04bbe7a740 10 client.26114 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:49:36.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.086+0000 7f04bbe7a740 20 client.26114 populate_metadata read hostname 'smithi028' 2024-04-23T14:49:36.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.086+0000 7f04bbe7a740 1 --2- 172.21.15.28:0/931473106 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55e23daa4430 0x55e23daa6860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.086+0000 7f04bbe7a740 4 client.26114 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:49:36.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.086+0000 7f04bbe7a740 1 -- 172.21.15.28:0/931473106 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e23d9fc890 con 0x55e23daa4430 2024-04-23T14:49:36.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.087+0000 7f04b1e00640 1 --2- 172.21.15.28:0/931473106 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55e23daa4430 0x55e23daa6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.087+0000 7f04b1e00640 1 --2- 172.21.15.28:0/931473106 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55e23daa4430 0x55e23daa6860 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be49bde9bd827405 server_cookie=bb36d8cd2ecea121 in_seq=0 out_seq=0 2024-04-23T14:49:36.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.087+0000 7f0497e00640 10 client.26114 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:49:36.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.089+0000 7f0497e00640 1 -- 172.21.15.28:0/931473106 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e23d9fc890 con 0x55e23daa4430 2024-04-23T14:49:36.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.089+0000 7f0497e00640 10 client.26114 handle_command_reply: tid=0 2024-04-23T14:49:36.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.089+0000 7f04bbe7a740 1 client.26114 shutdown 2024-04-23T14:49:36.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.089+0000 7f04bbe7a740 1 -- 172.21.15.28:0/931473106 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e23da89b90 msgr2=0x55e23da91760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:36.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.089+0000 7f04bbe7a740 1 --2- 172.21.15.28:0/931473106 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e23da89b90 0x55e23da91760 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f049c0cd9e0 tx=0x7f049c09c140 comp rx=0 tx=0).stop 2024-04-23T14:49:36.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.090+0000 7f04bbe7a740 1 -- 172.21.15.28:0/931473106 shutdown_connections 2024-04-23T14:49:36.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.090+0000 7f04bbe7a740 1 --2- 172.21.15.28:0/931473106 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55e23daa4430 0x55e23daa6860 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.090+0000 7f04bbe7a740 1 --2- 172.21.15.28:0/931473106 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e23da89b90 0x55e23da91760 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.090+0000 7f04bbe7a740 1 --2- 172.21.15.28:0/931473106 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e23da94260 0x55e23da91ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.090+0000 7f04bbe7a740 1 --2- 172.21.15.28:0/931473106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e23da89270 0x55e23da91220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.090+0000 7f04bbe7a740 1 -- 172.21.15.28:0/931473106 >> 172.21.15.28:0/931473106 conn(0x55e23da85590 msgr2=0x55e23da8b290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:36.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.090+0000 7f0497e00640 0 client.26114 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:49:36.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.090+0000 7f04bbe7a740 1 -- 172.21.15.28:0/931473106 shutdown_connections 2024-04-23T14:49:36.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.090+0000 7f04bbe7a740 1 -- 172.21.15.28:0/931473106 wait complete. 2024-04-23T14:49:36.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.090+0000 7f04bbe7a740 20 client.26114 trim_cache size 0 max 16384 2024-04-23T14:49:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.112+0000 7f04bbe7a740 1 Processor -- start 2024-04-23T14:49:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.113+0000 7f04bbe7a740 1 -- start start 2024-04-23T14:49:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.113+0000 7f04bbe7a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e23da89270 0x55e23da9fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.113+0000 7f04bbe7a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e23da89b90 0x55e23da8b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.113+0000 7f04bbe7a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e23da94260 0x55e23daac130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.113+0000 7f04bbe7a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f04b41adcc0 con 0x55e23da89b90 2024-04-23T14:49:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.113+0000 7f04bbe7a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f04b41b0540 con 0x55e23da94260 2024-04-23T14:49:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.113+0000 7f04bbe7a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f04b41ae330 con 0x55e23da89270 2024-04-23T14:49:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.113+0000 7f04b2800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e23da94260 0x55e23daac130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.113+0000 7f04b1e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e23da89270 0x55e23da9fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.113+0000 7f04abe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e23da89b90 0x55e23da8b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.113+0000 7f04b2800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e23da94260 0x55e23daac130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:54336/0 (socket says 172.21.15.28:54336) 2024-04-23T14:49:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.113+0000 7f04b2800640 1 -- 172.21.15.28:0/2037127356 learned_addr learned my addr 172.21.15.28:0/2037127356 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.114+0000 7f04b2800640 1 -- 172.21.15.28:0/2037127356 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e23da89270 msgr2=0x55e23da9fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.114+0000 7f04b2800640 1 --2- 172.21.15.28:0/2037127356 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e23da89270 0x55e23da9fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.114+0000 7f04b2800640 1 -- 172.21.15.28:0/2037127356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e23da89b90 msgr2=0x55e23da8b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.114+0000 7f04b2800640 1 --2- 172.21.15.28:0/2037127356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e23da89b90 0x55e23da8b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.114+0000 7f04b2800640 1 -- 172.21.15.28:0/2037127356 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e23da8f770 con 0x55e23da94260 2024-04-23T14:49:36.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.114+0000 7f04b2800640 1 --2- 172.21.15.28:0/2037127356 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e23da94260 0x55e23daac130 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f04ac035000 tx=0x7f04ac000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2da6dcdb9540dc13 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:36.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.114+0000 7f04a2a00640 1 -- 172.21.15.28:0/2037127356 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04ac031ba0 con 0x55e23da94260 2024-04-23T14:49:36.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.115+0000 7f04a2a00640 1 -- 172.21.15.28:0/2037127356 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04ac036040 con 0x55e23da94260 2024-04-23T14:49:36.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.115+0000 7f04a2a00640 1 -- 172.21.15.28:0/2037127356 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04ac004040 con 0x55e23da94260 2024-04-23T14:49:36.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.115+0000 7f04bbe7a740 1 -- 172.21.15.28:0/2037127356 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e23da94260 msgr2=0x55e23daac130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:36.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.115+0000 7f04bbe7a740 1 --2- 172.21.15.28:0/2037127356 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e23da94260 0x55e23daac130 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f04ac035000 tx=0x7f04ac000f40 comp rx=0 tx=0).stop 2024-04-23T14:49:36.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.115+0000 7f04bbe7a740 1 -- 172.21.15.28:0/2037127356 shutdown_connections 2024-04-23T14:49:36.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.115+0000 7f04bbe7a740 1 --2- 172.21.15.28:0/2037127356 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e23da94260 0x55e23daac130 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.115+0000 7f04bbe7a740 1 --2- 172.21.15.28:0/2037127356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e23da89b90 0x55e23da8b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.115+0000 7f04bbe7a740 1 --2- 172.21.15.28:0/2037127356 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e23da89270 0x55e23da9fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.115+0000 7f04bbe7a740 1 -- 172.21.15.28:0/2037127356 >> 172.21.15.28:0/2037127356 conn(0x55e23da85590 msgr2=0x55e23da91a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:36.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.115+0000 7f04bbe7a740 1 -- 172.21.15.28:0/2037127356 shutdown_connections 2024-04-23T14:49:36.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.116+0000 7f04bbe7a740 1 -- 172.21.15.28:0/2037127356 wait complete. 2024-04-23T14:49:36.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.116+0000 7f04bbe7a740 1 -- start start 2024-04-23T14:49:36.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.116+0000 7f04bbe7a740 1 Processor -- start 2024-04-23T14:49:36.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.117+0000 7f04bbe7a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e23da89270 0x55e23da8e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.117+0000 7f04bbe7a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e23da89b90 0x55e23da8e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.117+0000 7f04bbe7a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e23da94260 0x55e23da8eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.117+0000 7f04bbe7a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f04b41aea00 con 0x55e23da89b90 2024-04-23T14:49:36.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.117+0000 7f04bbe7a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f04b41afe70 con 0x55e23da89270 2024-04-23T14:49:36.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.117+0000 7f04bbe7a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f04b41af0d0 con 0x55e23da94260 2024-04-23T14:49:36.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.117+0000 7f04bbe7a740 10 client.0 _ll_register_callbacks cb 0x7f049c098610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:49:36.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.117+0000 7f04abe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e23da89b90 0x55e23da8e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.117+0000 7f04b2800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e23da94260 0x55e23da8eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.117+0000 7f04abe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e23da89b90 0x55e23da8e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51926/0 (socket says 172.21.15.28:51926) 2024-04-23T14:49:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.117+0000 7f04b2800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e23da94260 0x55e23da8eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:59758/0 (socket says 172.21.15.28:59758) 2024-04-23T14:49:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.117+0000 7f04abe00640 1 -- 172.21.15.28:0/4018022679 learned_addr learned my addr 172.21.15.28:0/4018022679 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.117+0000 7f04b1e00640 1 --2- 172.21.15.28:0/4018022679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e23da89270 0x55e23da8e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.117+0000 7f04b2800640 1 -- 172.21.15.28:0/4018022679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e23da89270 msgr2=0x55e23da8e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.117+0000 7f04b2800640 1 --2- 172.21.15.28:0/4018022679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e23da89270 0x55e23da8e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.117+0000 7f04b2800640 1 -- 172.21.15.28:0/4018022679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e23da89b90 msgr2=0x55e23da8e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.117+0000 7f04b2800640 1 --2- 172.21.15.28:0/4018022679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e23da89b90 0x55e23da8e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.117+0000 7f04b2800640 1 -- 172.21.15.28:0/4018022679 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e23db09b00 con 0x55e23da94260 2024-04-23T14:49:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.118+0000 7f04abe00640 1 --2- 172.21.15.28:0/4018022679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e23da89b90 0x55e23da8e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:49:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.118+0000 7f04b2800640 1 --2- 172.21.15.28:0/4018022679 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e23da94260 0x55e23da8eb20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f04ac005e30 tx=0x7f04ac009600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c8f2fbfa6d48374 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.118+0000 7f0497e00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:49:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.118+0000 7f0497e00640 1 -- 172.21.15.28:0/4018022679 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04ac059030 con 0x55e23da94260 2024-04-23T14:49:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.118+0000 7f04bbe7a740 1 -- 172.21.15.28:0/4018022679 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04ac036040 con 0x55e23da94260 2024-04-23T14:49:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.118+0000 7f0497e00640 1 -- 172.21.15.28:0/4018022679 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f04ac0318d0 con 0x55e23da94260 2024-04-23T14:49:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.118+0000 7f0497e00640 1 -- 172.21.15.28:0/4018022679 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f04ac003970 con 0x55e23da94260 2024-04-23T14:49:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.118+0000 7f04bbe7a740 10 client.26107 fetch_fsmap learned FSMap version 20 2024-04-23T14:49:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.118+0000 7f04bbe7a740 1 -- 172.21.15.28:0/4018022679 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55e23db0ae20 con 0x55e23da94260 2024-04-23T14:49:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.119+0000 7f0497e00640 1 -- 172.21.15.28:0/4018022679 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04ac036040 con 0x55e23da94260 2024-04-23T14:49:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.119+0000 7f0497e00640 1 -- 172.21.15.28:0/4018022679 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f04ac043070 con 0x55e23da94260 2024-04-23T14:49:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.119+0000 7f04bbe7a740 10 client.26107 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:49:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.119+0000 7f04bbe7a740 10 client.26107 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:49:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.119+0000 7f04bbe7a740 20 client.26107 populate_metadata read hostname 'smithi028' 2024-04-23T14:49:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.119+0000 7f04bbe7a740 1 --2- 172.21.15.28:0/4018022679 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55e23db0b210 0x55e23db0d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.119+0000 7f04bbe7a740 4 client.26107 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-23T14:49:36.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.119+0000 7f04bbe7a740 1 -- 172.21.15.28:0/4018022679 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55e23d9f6700 con 0x55e23db0b210 2024-04-23T14:49:36.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.119+0000 7f04b1e00640 1 --2- 172.21.15.28:0/4018022679 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55e23db0b210 0x55e23db0d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.120+0000 7f04b1e00640 1 --2- 172.21.15.28:0/4018022679 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55e23db0b210 0x55e23db0d600 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=adfbe786b24d8008 server_cookie=a662d367aac1ca47 in_seq=0 out_seq=0 2024-04-23T14:49:36.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.120+0000 7f0497e00640 10 client.26107 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:49:36.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.123+0000 7f0497e00640 1 -- 172.21.15.28:0/4018022679 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e23d9f6700 con 0x55e23db0b210 2024-04-23T14:49:36.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.123+0000 7f0497e00640 10 client.26107 handle_command_reply: tid=0 2024-04-23T14:49:36.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.123+0000 7f04bbe7a740 1 client.26107 shutdown 2024-04-23T14:49:36.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.123+0000 7f04bbe7a740 1 -- 172.21.15.28:0/4018022679 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e23da94260 msgr2=0x55e23da8eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:36.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.123+0000 7f04bbe7a740 1 --2- 172.21.15.28:0/4018022679 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e23da94260 0x55e23da8eb20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f04ac005e30 tx=0x7f04ac009600 comp rx=0 tx=0).stop 2024-04-23T14:49:36.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.123+0000 7f04bbe7a740 1 -- 172.21.15.28:0/4018022679 shutdown_connections 2024-04-23T14:49:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.123+0000 7f04bbe7a740 1 --2- 172.21.15.28:0/4018022679 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55e23db0b210 0x55e23db0d600 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.123+0000 7f04bbe7a740 1 --2- 172.21.15.28:0/4018022679 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e23da94260 0x55e23da8eb20 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.123+0000 7f0497e00640 0 client.26107 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:49:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.123+0000 7f04bbe7a740 1 --2- 172.21.15.28:0/4018022679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e23da89b90 0x55e23da8e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.123+0000 7f04bbe7a740 1 --2- 172.21.15.28:0/4018022679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e23da89270 0x55e23da8e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.123+0000 7f04bbe7a740 1 -- 172.21.15.28:0/4018022679 >> 172.21.15.28:0/4018022679 conn(0x55e23da85590 msgr2=0x55e23da83e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.124+0000 7f04bbe7a740 1 -- 172.21.15.28:0/4018022679 shutdown_connections 2024-04-23T14:49:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.124+0000 7f04bbe7a740 1 -- 172.21.15.28:0/4018022679 wait complete. 2024-04-23T14:49:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.124+0000 7f04bbe7a740 20 client.26107 trim_cache size 0 max 16384 2024-04-23T14:49:36.127 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:49:36.128 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-23T14:49:36.128 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "20b79495-89c4-495b-9ec8-3aa0922b0318", 2024-04-23T14:49:36.128 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-23T14:49:36.128 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:49:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.124+0000 7f04b3200640 1 -- 172.21.15.28:0/242011022 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f04980a3da0 msgr2=0x7f04980a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.124+0000 7f04b3200640 1 --2- 172.21.15.28:0/242011022 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f04980a3da0 0x7f04980a6260 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x55e23da8a830 tx=0x7f04a4005f90 comp rx=0 tx=0).stop 2024-04-23T14:49:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.124+0000 7f04b3200640 1 -- 172.21.15.28:0/242011022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04b4101aa0 msgr2=0x7f04b4102fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.124+0000 7f04b3200640 1 --2- 172.21.15.28:0/242011022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04b4101aa0 0x7f04b4102fe0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f049c00d6d0 tx=0x7f049c00db90 comp rx=0 tx=0).stop 2024-04-23T14:49:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.124+0000 7f04b3200640 1 -- 172.21.15.28:0/242011022 shutdown_connections 2024-04-23T14:49:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.125+0000 7f04b3200640 1 --2- 172.21.15.28:0/242011022 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f04980a3da0 0x7f04980a6260 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.125+0000 7f04b3200640 1 --2- 172.21.15.28:0/242011022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04b4106ed0 0x7f04b4103520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.125+0000 7f04b3200640 1 --2- 172.21.15.28:0/242011022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04b4101aa0 0x7f04b4102fe0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.125+0000 7f04b3200640 1 --2- 172.21.15.28:0/242011022 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04b41010e0 0x7f04b4102aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.125+0000 7f04b3200640 1 -- 172.21.15.28:0/242011022 >> 172.21.15.28:0/242011022 conn(0x7f04b4077c90 msgr2=0x7f04b406a880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.125+0000 7f04b3200640 1 -- 172.21.15.28:0/242011022 shutdown_connections 2024-04-23T14:49:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.125+0000 7f04b3200640 1 -- 172.21.15.28:0/242011022 wait complete. 2024-04-23T14:49:36.150 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:49:36.150 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:49:36.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.357+0000 7ff2a2600640 1 -- 172.21.15.28:0/2564988194 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff29c108080 msgr2=0x7ff29c108460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:36.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.357+0000 7ff2a2600640 1 --2- 172.21.15.28:0/2564988194 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff29c108080 0x7ff29c108460 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff290009900 tx=0x7ff29002a400 comp rx=0 tx=0).stop 2024-04-23T14:49:36.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.357+0000 7ff2a2600640 1 -- 172.21.15.28:0/2564988194 shutdown_connections 2024-04-23T14:49:36.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.357+0000 7ff2a2600640 1 --2- 172.21.15.28:0/2564988194 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff29c06af20 0x7ff29c10ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.357+0000 7ff2a2600640 1 --2- 172.21.15.28:0/2564988194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29c06a580 0x7ff29c06a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.357+0000 7ff2a2600640 1 --2- 172.21.15.28:0/2564988194 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff29c108080 0x7ff29c108460 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.357+0000 7ff2a2600640 1 -- 172.21.15.28:0/2564988194 >> 172.21.15.28:0/2564988194 conn(0x7ff29c077cd0 msgr2=0x7ff29c0780e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:36.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.357+0000 7ff2a2600640 1 -- 172.21.15.28:0/2564988194 shutdown_connections 2024-04-23T14:49:36.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.357+0000 7ff2a2600640 1 -- 172.21.15.28:0/2564988194 wait complete. 2024-04-23T14:49:36.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.358+0000 7ff2a2600640 1 Processor -- start 2024-04-23T14:49:36.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.358+0000 7ff2a2600640 1 -- start start 2024-04-23T14:49:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.359+0000 7ff2a2600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff29c06a580 0x7ff29c1a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.359+0000 7ff2a2600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff29c06af20 0x7ff29c1a0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.359+0000 7ff2a2600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29c108080 0x7ff29c19a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.359+0000 7ff2a2600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff29c112390 con 0x7ff29c108080 2024-04-23T14:49:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.359+0000 7ff2a2600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff29c112210 con 0x7ff29c06af20 2024-04-23T14:49:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.359+0000 7ff2a2600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff29c112510 con 0x7ff29c06a580 2024-04-23T14:49:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.359+0000 7ff2a1200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff29c06a580 0x7ff29c1a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.359+0000 7ff29be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff29c06af20 0x7ff29c1a0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.359+0000 7ff2a1200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff29c06a580 0x7ff29c1a0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:59764/0 (socket says 172.21.15.28:59764) 2024-04-23T14:49:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.359+0000 7ff2a1200640 1 -- 172.21.15.28:0/376565306 learned_addr learned my addr 172.21.15.28:0/376565306 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.359+0000 7ff2a1c00640 1 --2- 172.21.15.28:0/376565306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29c108080 0x7ff29c19a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.360+0000 7ff29be00640 1 -- 172.21.15.28:0/376565306 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff29c06a580 msgr2=0x7ff29c1a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:36.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.360+0000 7ff29be00640 1 --2- 172.21.15.28:0/376565306 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff29c06a580 0x7ff29c1a0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.360+0000 7ff29be00640 1 -- 172.21.15.28:0/376565306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29c108080 msgr2=0x7ff29c19a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:36.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.360+0000 7ff29be00640 1 --2- 172.21.15.28:0/376565306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29c108080 0x7ff29c19a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.360+0000 7ff29be00640 1 -- 172.21.15.28:0/376565306 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff29c19b0b0 con 0x7ff29c06af20 2024-04-23T14:49:36.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.360+0000 7ff2a1200640 1 --2- 172.21.15.28:0/376565306 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff29c06a580 0x7ff29c1a0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:49:36.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.360+0000 7ff29be00640 1 --2- 172.21.15.28:0/376565306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff29c06af20 0x7ff29c1a0cc0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff28c00d6d0 tx=0x7ff28c00db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:36.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.360+0000 7ff299600640 1 -- 172.21.15.28:0/376565306 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff28c017070 con 0x7ff29c06af20 2024-04-23T14:49:36.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.360+0000 7ff2a2600640 1 -- 172.21.15.28:0/376565306 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff29c19b3a0 con 0x7ff29c06af20 2024-04-23T14:49:36.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.360+0000 7ff299600640 1 -- 172.21.15.28:0/376565306 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff28c009820 con 0x7ff29c06af20 2024-04-23T14:49:36.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.360+0000 7ff299600640 1 -- 172.21.15.28:0/376565306 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff28c004cd0 con 0x7ff29c06af20 2024-04-23T14:49:36.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.361+0000 7ff2a2600640 1 -- 172.21.15.28:0/376565306 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff29c1a7640 con 0x7ff29c06af20 2024-04-23T14:49:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.364+0000 7ff299600640 1 -- 172.21.15.28:0/376565306 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7ff28c0099d0 con 0x7ff29c06af20 2024-04-23T14:49:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.364+0000 7ff2a6a27740 1 Processor -- start 2024-04-23T14:49:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.364+0000 7ff2a6a27740 1 -- start start 2024-04-23T14:49:36.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.365+0000 7ff2a6a27740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556965afaa30 0x556965afae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.365+0000 7ff2a6a27740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556965afb350 0x556965b054e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.365+0000 7ff2a6a27740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556965b05a20 0x556965b07e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.365+0000 7ff2a6a27740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556965a601f0 con 0x556965b05a20 2024-04-23T14:49:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.365+0000 7ff2a6a27740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x556965a06350 con 0x556965afaa30 2024-04-23T14:49:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.365+0000 7ff2a6a27740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556965a5c1b0 con 0x556965afb350 2024-04-23T14:49:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.365+0000 7ff2a1c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556965afaa30 0x556965afae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.365+0000 7ff2a1c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556965afaa30 0x556965afae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:54370/0 (socket says 172.21.15.28:54370) 2024-04-23T14:49:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.365+0000 7ff2a1200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556965afb350 0x556965b054e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.365+0000 7ff2a1c00640 1 -- 172.21.15.28:0/3258145459 learned_addr learned my addr 172.21.15.28:0/3258145459 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.365+0000 7ff29be00640 1 --2- 172.21.15.28:0/3258145459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556965b05a20 0x556965b07e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.365+0000 7ff2a1c00640 1 -- 172.21.15.28:0/3258145459 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556965afb350 msgr2=0x556965b054e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.365+0000 7ff2a1c00640 1 --2- 172.21.15.28:0/3258145459 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556965afb350 0x556965b054e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.365+0000 7ff2a1c00640 1 -- 172.21.15.28:0/3258145459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556965b05a20 msgr2=0x556965b07e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.365+0000 7ff2a1c00640 1 --2- 172.21.15.28:0/3258145459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556965b05a20 0x556965b07e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.365+0000 7ff2a1c00640 1 -- 172.21.15.28:0/3258145459 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556965a5af50 con 0x556965afaa30 2024-04-23T14:49:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.366+0000 7ff2a1c00640 1 --2- 172.21.15.28:0/3258145459 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556965afaa30 0x556965afae10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff294009f40 tx=0x7ff2940118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6c3c9242488ecc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.366+0000 7ff299600640 1 --2- 172.21.15.28:0/376565306 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7ff2840a3d20 0x7ff2840a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.366+0000 7ff28a000640 1 -- 172.21.15.28:0/3258145459 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff294004d20 con 0x556965afaa30 2024-04-23T14:49:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.366+0000 7ff28a000640 1 -- 172.21.15.28:0/3258145459 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff294004050 con 0x556965afaa30 2024-04-23T14:49:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.366+0000 7ff28a000640 1 -- 172.21.15.28:0/3258145459 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff294005240 con 0x556965afaa30 2024-04-23T14:49:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.366+0000 7ff299600640 1 -- 172.21.15.28:0/376565306 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7ff28c0d0330 con 0x7ff29c06af20 2024-04-23T14:49:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.366+0000 7ff2a1200640 1 --2- 172.21.15.28:0/376565306 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7ff2840a3d20 0x7ff2840a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.367+0000 7ff2a6a27740 1 -- 172.21.15.28:0/3258145459 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556965afaa30 msgr2=0x556965afae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.367+0000 7ff2a6a27740 1 --2- 172.21.15.28:0/3258145459 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556965afaa30 0x556965afae10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff294009f40 tx=0x7ff2940118d0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.367+0000 7ff2a6a27740 1 -- 172.21.15.28:0/3258145459 shutdown_connections 2024-04-23T14:49:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.367+0000 7ff2a1200640 1 --2- 172.21.15.28:0/376565306 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7ff2840a3d20 0x7ff2840a61e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff290005770 tx=0x7ff290005a20 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.367+0000 7ff2a6a27740 1 --2- 172.21.15.28:0/3258145459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556965b05a20 0x556965b07e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.367+0000 7ff2a6a27740 1 --2- 172.21.15.28:0/3258145459 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556965afb350 0x556965b054e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.367+0000 7ff2a6a27740 1 --2- 172.21.15.28:0/3258145459 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556965afaa30 0x556965afae10 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.367+0000 7ff2a6a27740 1 -- 172.21.15.28:0/3258145459 >> 172.21.15.28:0/3258145459 conn(0x556965af6d50 msgr2=0x556965af7130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:36.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.367+0000 7ff2a6a27740 1 -- 172.21.15.28:0/3258145459 shutdown_connections 2024-04-23T14:49:36.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.367+0000 7ff2a6a27740 1 -- 172.21.15.28:0/3258145459 wait complete. 2024-04-23T14:49:36.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.368+0000 7ff2a6a27740 1 -- start start 2024-04-23T14:49:36.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.368+0000 7ff2a6a27740 1 Processor -- start 2024-04-23T14:49:36.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.368+0000 7ff2a6a27740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556965afaa30 0x556965b02950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.368+0000 7ff2a6a27740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556965afb350 0x556965b02e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.368+0000 7ff2a6a27740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556965b05a20 0x556965b033d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.368+0000 7ff2a6a27740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556965b08810 con 0x556965b05a20 2024-04-23T14:49:36.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.368+0000 7ff2a6a27740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x556965a19690 con 0x556965afaa30 2024-04-23T14:49:36.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.368+0000 7ff2a6a27740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556965b08990 con 0x556965afb350 2024-04-23T14:49:36.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.368+0000 7ff2a6a27740 10 client.0 _ll_register_callbacks cb 0x556965a55650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:49:36.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.369+0000 7ff2a1200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556965afaa30 0x556965b02950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.369+0000 7ff29be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556965afb350 0x556965b02e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.369+0000 7ff2a1200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556965afaa30 0x556965b02950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:54384/0 (socket says 172.21.15.28:54384) 2024-04-23T14:49:36.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.369+0000 7ff2a1200640 1 -- 172.21.15.28:0/560787103 learned_addr learned my addr 172.21.15.28:0/560787103 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:36.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.369+0000 7ff2a1c00640 1 --2- 172.21.15.28:0/560787103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556965b05a20 0x556965b033d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.369+0000 7ff29be00640 1 -- 172.21.15.28:0/560787103 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556965afaa30 msgr2=0x556965b02950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:36.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.369+0000 7ff29be00640 1 --2- 172.21.15.28:0/560787103 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556965afaa30 0x556965b02950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.369+0000 7ff29be00640 1 -- 172.21.15.28:0/560787103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556965b05a20 msgr2=0x556965b033d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:36.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.369+0000 7ff29be00640 1 --2- 172.21.15.28:0/560787103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556965b05a20 0x556965b033d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.369+0000 7ff29be00640 1 -- 172.21.15.28:0/560787103 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556965a18920 con 0x556965afb350 2024-04-23T14:49:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.369+0000 7ff29be00640 1 --2- 172.21.15.28:0/560787103 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556965afb350 0x556965b02e90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff28c0cda30 tx=0x7ff28c09bef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3330713095ed4153 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.369+0000 7ff27ea00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:49:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.369+0000 7ff27ea00640 1 -- 172.21.15.28:0/560787103 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff28c020040 con 0x556965afb350 2024-04-23T14:49:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.369+0000 7ff27ea00640 1 -- 172.21.15.28:0/560787103 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7ff28c0cdcb0 con 0x556965afb350 2024-04-23T14:49:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.370+0000 7ff2a6a27740 1 -- 172.21.15.28:0/560787103 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff294004050 con 0x556965afb350 2024-04-23T14:49:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.370+0000 7ff27ea00640 1 -- 172.21.15.28:0/560787103 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7ff28c0cdeb0 con 0x556965afb350 2024-04-23T14:49:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.370+0000 7ff2a6a27740 10 client.26119 fetch_fsmap learned FSMap version 20 2024-04-23T14:49:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.370+0000 7ff2a6a27740 1 -- 172.21.15.28:0/560787103 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55696583c7d0 con 0x556965afb350 2024-04-23T14:49:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.370+0000 7ff27ea00640 1 -- 172.21.15.28:0/560787103 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff294004050 con 0x556965afb350 2024-04-23T14:49:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.370+0000 7ff27ea00640 1 -- 172.21.15.28:0/560787103 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7ff28c0a4a60 con 0x556965afb350 2024-04-23T14:49:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.370+0000 7ff2a6a27740 10 client.26119 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:49:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.370+0000 7ff2a6a27740 10 client.26119 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:49:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.370+0000 7ff2a6a27740 20 client.26119 populate_metadata read hostname 'smithi028' 2024-04-23T14:49:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.370+0000 7ff2a6a27740 1 --2- 172.21.15.28:0/560787103 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x556965b15c90 0x556965b180a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.370+0000 7ff2a6a27740 4 client.26119 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:49:36.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.370+0000 7ff2a6a27740 1 -- 172.21.15.28:0/560787103 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556965a6dc10 con 0x556965b15c90 2024-04-23T14:49:36.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.371+0000 7ff2a1200640 1 --2- 172.21.15.28:0/560787103 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x556965b15c90 0x556965b180a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.371+0000 7ff2a1200640 1 --2- 172.21.15.28:0/560787103 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x556965b15c90 0x556965b180a0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8eacdef0340f9f44 server_cookie=5f62d0a723576f5c in_seq=0 out_seq=0 2024-04-23T14:49:36.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.371+0000 7ff27ea00640 10 client.26119 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:49:36.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.373+0000 7ff27ea00640 1 -- 172.21.15.28:0/560787103 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556965a6dc10 con 0x556965b15c90 2024-04-23T14:49:36.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.373+0000 7ff27ea00640 10 client.26119 handle_command_reply: tid=0 2024-04-23T14:49:36.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.373+0000 7ff2a6a27740 1 client.26119 shutdown 2024-04-23T14:49:36.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.373+0000 7ff2a6a27740 1 -- 172.21.15.28:0/560787103 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556965afb350 msgr2=0x556965b02e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:36.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.373+0000 7ff2a6a27740 1 --2- 172.21.15.28:0/560787103 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556965afb350 0x556965b02e90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff28c0cda30 tx=0x7ff28c09bef0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.373+0000 7ff2a6a27740 1 -- 172.21.15.28:0/560787103 shutdown_connections 2024-04-23T14:49:36.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.373+0000 7ff2a6a27740 1 --2- 172.21.15.28:0/560787103 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x556965b15c90 0x556965b180a0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.373+0000 7ff2a6a27740 1 --2- 172.21.15.28:0/560787103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556965b05a20 0x556965b033d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.373+0000 7ff2a6a27740 1 --2- 172.21.15.28:0/560787103 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556965afb350 0x556965b02e90 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.373+0000 7ff2a6a27740 1 --2- 172.21.15.28:0/560787103 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556965afaa30 0x556965b02950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.373+0000 7ff27ea00640 0 client.26119 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:49:36.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.373+0000 7ff2a6a27740 1 -- 172.21.15.28:0/560787103 >> 172.21.15.28:0/560787103 conn(0x556965af6d50 msgr2=0x556965afcb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:36.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.373+0000 7ff2a6a27740 1 -- 172.21.15.28:0/560787103 shutdown_connections 2024-04-23T14:49:36.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.373+0000 7ff2a6a27740 1 -- 172.21.15.28:0/560787103 wait complete. 2024-04-23T14:49:36.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.374+0000 7ff2a6a27740 20 client.26119 trim_cache size 0 max 16384 2024-04-23T14:49:36.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.393+0000 7ff2a6a27740 1 Processor -- start 2024-04-23T14:49:36.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.393+0000 7ff2a6a27740 1 -- start start 2024-04-23T14:49:36.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.394+0000 7ff2a6a27740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556965afaa30 0x556965afd280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.394+0000 7ff2a6a27740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556965afb350 0x556965b03020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.394+0000 7ff2a6a27740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556965b05a20 0x556965b1d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.394+0000 7ff2a6a27740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff29c1adde0 con 0x556965b05a20 2024-04-23T14:49:36.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.394+0000 7ff2a6a27740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff29c1b0660 con 0x556965afaa30 2024-04-23T14:49:36.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.394+0000 7ff2a6a27740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff29c1ae450 con 0x556965afb350 2024-04-23T14:49:36.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.394+0000 7ff2a1200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556965afaa30 0x556965afd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.394+0000 7ff2a1200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556965afaa30 0x556965afd280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:54386/0 (socket says 172.21.15.28:54386) 2024-04-23T14:49:36.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.394+0000 7ff2a1200640 1 -- 172.21.15.28:0/1910248907 learned_addr learned my addr 172.21.15.28:0/1910248907 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:36.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.394+0000 7ff2a1c00640 1 --2- 172.21.15.28:0/1910248907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556965b05a20 0x556965b1d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.394+0000 7ff29be00640 1 --2- 172.21.15.28:0/1910248907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556965afb350 0x556965b03020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.394+0000 7ff2a1200640 1 -- 172.21.15.28:0/1910248907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556965afb350 msgr2=0x556965b03020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:36.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.394+0000 7ff2a1200640 1 --2- 172.21.15.28:0/1910248907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556965afb350 0x556965b03020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.394+0000 7ff2a1200640 1 -- 172.21.15.28:0/1910248907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556965b05a20 msgr2=0x556965b1d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:36.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.394+0000 7ff2a1200640 1 --2- 172.21.15.28:0/1910248907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556965b05a20 0x556965b1d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.394+0000 7ff2a1200640 1 -- 172.21.15.28:0/1910248907 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569657723f0 con 0x556965afaa30 2024-04-23T14:49:36.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.394+0000 7ff2a1c00640 1 --2- 172.21.15.28:0/1910248907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556965b05a20 0x556965b1d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:49:36.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.395+0000 7ff2a1200640 1 --2- 172.21.15.28:0/1910248907 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556965afaa30 0x556965afd280 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff290005ab0 tx=0x7ff290040000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7dbfdf2188dfdeb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:36.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.395+0000 7ff289600640 1 -- 172.21.15.28:0/1910248907 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff290035040 con 0x556965afaa30 2024-04-23T14:49:36.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.395+0000 7ff289600640 1 -- 172.21.15.28:0/1910248907 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2900546e0 con 0x556965afaa30 2024-04-23T14:49:36.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.395+0000 7ff289600640 1 -- 172.21.15.28:0/1910248907 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff29003d810 con 0x556965afaa30 2024-04-23T14:49:36.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.396+0000 7ff2a6a27740 1 -- 172.21.15.28:0/1910248907 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556965afaa30 msgr2=0x556965afd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:36.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.396+0000 7ff2a6a27740 1 --2- 172.21.15.28:0/1910248907 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556965afaa30 0x556965afd280 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff290005ab0 tx=0x7ff290040000 comp rx=0 tx=0).stop 2024-04-23T14:49:36.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.396+0000 7ff2a6a27740 1 -- 172.21.15.28:0/1910248907 shutdown_connections 2024-04-23T14:49:36.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.396+0000 7ff2a6a27740 1 --2- 172.21.15.28:0/1910248907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556965b05a20 0x556965b1d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.396+0000 7ff2a6a27740 1 --2- 172.21.15.28:0/1910248907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556965afb350 0x556965b03020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.396+0000 7ff2a6a27740 1 --2- 172.21.15.28:0/1910248907 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556965afaa30 0x556965afd280 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.396+0000 7ff2a6a27740 1 -- 172.21.15.28:0/1910248907 >> 172.21.15.28:0/1910248907 conn(0x556965af6d50 msgr2=0x556965b02910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:36.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.396+0000 7ff2a6a27740 1 -- 172.21.15.28:0/1910248907 shutdown_connections 2024-04-23T14:49:36.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.396+0000 7ff2a6a27740 1 -- 172.21.15.28:0/1910248907 wait complete. 2024-04-23T14:49:36.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.396+0000 7ff2a6a27740 1 -- start start 2024-04-23T14:49:36.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.396+0000 7ff2a6a27740 1 Processor -- start 2024-04-23T14:49:36.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.397+0000 7ff2a6a27740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556965afaa30 0x556965b09f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.397+0000 7ff2a6a27740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556965afb350 0x556965b0a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.397+0000 7ff2a6a27740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556965b05a20 0x556965b75220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.397+0000 7ff2a6a27740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff29c1aeb20 con 0x556965afaa30 2024-04-23T14:49:36.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.397+0000 7ff2a6a27740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff29c1aff90 con 0x556965afb350 2024-04-23T14:49:36.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.397+0000 7ff2a6a27740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff29c1af1f0 con 0x556965b05a20 2024-04-23T14:49:36.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.397+0000 7ff2a6a27740 10 client.0 _ll_register_callbacks cb 0x7ff28c0057f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:49:36.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.397+0000 7ff2a1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556965afaa30 0x556965b09f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.397+0000 7ff29be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556965afb350 0x556965b0a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.397+0000 7ff29be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556965afb350 0x556965b0a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:54390/0 (socket says 172.21.15.28:54390) 2024-04-23T14:49:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.397+0000 7ff29be00640 1 -- 172.21.15.28:0/1196594669 learned_addr learned my addr 172.21.15.28:0/1196594669 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:49:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.397+0000 7ff2a1c00640 1 --2- 172.21.15.28:0/1196594669 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556965b05a20 0x556965b75220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.398+0000 7ff29be00640 1 -- 172.21.15.28:0/1196594669 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556965b05a20 msgr2=0x556965b75220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.398+0000 7ff29be00640 1 --2- 172.21.15.28:0/1196594669 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556965b05a20 0x556965b75220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.398+0000 7ff29be00640 1 -- 172.21.15.28:0/1196594669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556965afaa30 msgr2=0x556965b09f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:49:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.398+0000 7ff29be00640 1 --2- 172.21.15.28:0/1196594669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556965afaa30 0x556965b09f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.398+0000 7ff29be00640 1 -- 172.21.15.28:0/1196594669 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569657320f0 con 0x556965afb350 2024-04-23T14:49:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.398+0000 7ff2a1200640 1 --2- 172.21.15.28:0/1196594669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556965afaa30 0x556965b09f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:49:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.398+0000 7ff29be00640 1 --2- 172.21.15.28:0/1196594669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556965afb350 0x556965b0a450 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff28c09bfe0 tx=0x7ff28c09bdc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a9aec31f7a63965 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:49:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.398+0000 7ff27ea00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:49:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.398+0000 7ff27ea00640 1 -- 172.21.15.28:0/1196594669 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff28c014070 con 0x556965afb350 2024-04-23T14:49:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.398+0000 7ff27ea00640 1 -- 172.21.15.28:0/1196594669 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7ff28c0cdcb0 con 0x556965afb350 2024-04-23T14:49:36.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.398+0000 7ff2a6a27740 1 -- 172.21.15.28:0/1196594669 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2900546e0 con 0x556965afb350 2024-04-23T14:49:36.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.398+0000 7ff27ea00640 1 -- 172.21.15.28:0/1196594669 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7ff28c09c100 con 0x556965afb350 2024-04-23T14:49:36.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.398+0000 7ff2a6a27740 10 client.26138 fetch_fsmap learned FSMap version 20 2024-04-23T14:49:36.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.398+0000 7ff2a6a27740 1 -- 172.21.15.28:0/1196594669 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x556965a25810 con 0x556965afb350 2024-04-23T14:49:36.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.399+0000 7ff27ea00640 1 -- 172.21.15.28:0/1196594669 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2900546e0 con 0x556965afb350 2024-04-23T14:49:36.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.399+0000 7ff27ea00640 1 -- 172.21.15.28:0/1196594669 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7ff28c005220 con 0x556965afb350 2024-04-23T14:49:36.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.399+0000 7ff2a6a27740 10 client.26138 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:49:36.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.399+0000 7ff2a6a27740 10 client.26138 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:49:36.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.399+0000 7ff2a6a27740 20 client.26138 populate_metadata read hostname 'smithi028' 2024-04-23T14:49:36.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.399+0000 7ff2a6a27740 1 --2- 172.21.15.28:0/1196594669 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x556965b7ce60 0x556965b7f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:49:36.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.399+0000 7ff2a6a27740 4 client.26138 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:49:36.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.399+0000 7ff2a6a27740 1 -- 172.21.15.28:0/1196594669 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556965a67a80 con 0x556965b7ce60 2024-04-23T14:49:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.399+0000 7ff2a1200640 1 --2- 172.21.15.28:0/1196594669 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x556965b7ce60 0x556965b7f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:49:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.400+0000 7ff2a1200640 1 --2- 172.21.15.28:0/1196594669 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x556965b7ce60 0x556965b7f280 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72b4b68421a40c0f server_cookie=323e41c461c549 in_seq=0 out_seq=0 2024-04-23T14:49:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.400+0000 7ff27ea00640 10 client.26138 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:49:36.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.401+0000 7ff27ea00640 1 -- 172.21.15.28:0/1196594669 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x556965a67a80 con 0x556965b7ce60 2024-04-23T14:49:36.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.401+0000 7ff27ea00640 10 client.26138 handle_command_reply: tid=0 2024-04-23T14:49:36.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.401+0000 7ff2a6a27740 1 client.26138 shutdown 2024-04-23T14:49:36.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.401+0000 7ff2a6a27740 1 -- 172.21.15.28:0/1196594669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556965afb350 msgr2=0x556965b0a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:36.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.401+0000 7ff2a6a27740 1 --2- 172.21.15.28:0/1196594669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556965afb350 0x556965b0a450 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff28c09bfe0 tx=0x7ff28c09bdc0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.401+0000 7ff2a6a27740 1 -- 172.21.15.28:0/1196594669 shutdown_connections 2024-04-23T14:49:36.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.401+0000 7ff2a6a27740 1 --2- 172.21.15.28:0/1196594669 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x556965b7ce60 0x556965b7f280 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.401+0000 7ff2a6a27740 1 --2- 172.21.15.28:0/1196594669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556965afb350 0x556965b0a450 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.401+0000 7ff2a6a27740 1 --2- 172.21.15.28:0/1196594669 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556965b05a20 0x556965b75220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.401+0000 7ff2a6a27740 1 --2- 172.21.15.28:0/1196594669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556965afaa30 0x556965b09f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.401+0000 7ff27ea00640 0 client.26138 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:49:36.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.401+0000 7ff2a6a27740 1 -- 172.21.15.28:0/1196594669 >> 172.21.15.28:0/1196594669 conn(0x556965af6d50 msgr2=0x556965b031a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:36.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.402+0000 7ff2a6a27740 1 -- 172.21.15.28:0/1196594669 shutdown_connections 2024-04-23T14:49:36.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.402+0000 7ff2a6a27740 1 -- 172.21.15.28:0/1196594669 wait complete. 2024-04-23T14:49:36.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.402+0000 7ff2a6a27740 20 client.26138 trim_cache size 0 max 16384 2024-04-23T14:49:36.407 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:49:36.408 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-23T14:49:36.408 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-23T14:49:36.408 INFO:teuthology.orchestra.run.smithi028.stdout: "20b79495-89c4-495b-9ec8-3aa0922b0318": { 2024-04-23T14:49:36.408 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-23T14:49:36.408 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "20b79495-89c4-495b-9ec8-3aa0922b0318", 2024-04-23T14:49:36.408 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-23T14:49:36.408 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:49:36.408 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:49:36.408 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:49:36.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.402+0000 7ff2a2600640 1 -- 172.21.15.28:0/376565306 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7ff2840a3d20 msgr2=0x7ff2840a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:36.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.402+0000 7ff2a2600640 1 --2- 172.21.15.28:0/376565306 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7ff2840a3d20 0x7ff2840a61e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff290005770 tx=0x7ff290005a20 comp rx=0 tx=0).stop 2024-04-23T14:49:36.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.403+0000 7ff2a2600640 1 -- 172.21.15.28:0/376565306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff29c06af20 msgr2=0x7ff29c1a0cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:49:36.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.403+0000 7ff2a2600640 1 --2- 172.21.15.28:0/376565306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff29c06af20 0x7ff29c1a0cc0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff28c00d6d0 tx=0x7ff28c00db90 comp rx=0 tx=0).stop 2024-04-23T14:49:36.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.403+0000 7ff2a2600640 1 -- 172.21.15.28:0/376565306 shutdown_connections 2024-04-23T14:49:36.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.403+0000 7ff2a2600640 1 --2- 172.21.15.28:0/376565306 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7ff2840a3d20 0x7ff2840a61e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.403+0000 7ff2a2600640 1 --2- 172.21.15.28:0/376565306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff29c06af20 0x7ff29c1a0cc0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.403+0000 7ff2a2600640 1 --2- 172.21.15.28:0/376565306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29c108080 0x7ff29c19a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.403+0000 7ff2a2600640 1 --2- 172.21.15.28:0/376565306 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff29c06a580 0x7ff29c1a0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:49:36.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.403+0000 7ff2a2600640 1 -- 172.21.15.28:0/376565306 >> 172.21.15.28:0/376565306 conn(0x7ff29c077cd0 msgr2=0x7ff29c10d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:49:36.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.403+0000 7ff2a2600640 1 -- 172.21.15.28:0/376565306 shutdown_connections 2024-04-23T14:49:36.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:49:36.403+0000 7ff2a2600640 1 -- 172.21.15.28:0/376565306 wait complete. 2024-04-23T14:49:36.423 INFO:tasks.cephfs.filesystem:scrub status for tag:20b79495-89c4-495b-9ec8-3aa0922b0318 - {'path': '/', 'tag': '20b79495-89c4-495b-9ec8-3aa0922b0318', 'options': 'recursive,force'} 2024-04-23T14:49:36.490 INFO:tasks.workunit.client.0.smithi028.stdout: 1 437875 9.79 MB/sec execute 112 sec latency 29.486 ms 2024-04-23T14:49:37.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:36 smithi105 ceph-mon[32555]: pgmap v170: 97 pgs: 97 active+clean; 251 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.8 MiB/s wr, 165 op/s 2024-04-23T14:49:37.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:36 smithi038 ceph-mon[32348]: pgmap v170: 97 pgs: 97 active+clean; 251 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.8 MiB/s wr, 165 op/s 2024-04-23T14:49:37.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:36 smithi028 ceph-mon[27810]: pgmap v170: 97 pgs: 97 active+clean; 251 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.8 MiB/s wr, 165 op/s 2024-04-23T14:49:37.490 INFO:tasks.workunit.client.0.smithi028.stdout: 1 440247 9.83 MB/sec execute 113 sec latency 10.120 ms 2024-04-23T14:49:38.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:37 smithi105 ceph-mon[32555]: scrub queued for path: / 2024-04-23T14:49:38.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:37 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:49:38.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:37 smithi105 ceph-mon[32555]: scrub summary: active paths [/] 2024-04-23T14:49:38.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:37 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:49:38.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:37 smithi038 ceph-mon[32348]: scrub queued for path: / 2024-04-23T14:49:38.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:37 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:49:38.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:37 smithi038 ceph-mon[32348]: scrub summary: active paths [/] 2024-04-23T14:49:38.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:37 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:49:38.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:37 smithi028 ceph-mon[27810]: scrub queued for path: / 2024-04-23T14:49:38.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:37 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:49:38.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:37 smithi028 ceph-mon[27810]: scrub summary: active paths [/] 2024-04-23T14:49:38.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:37 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:49:38.491 INFO:tasks.workunit.client.0.smithi028.stdout: 1 442250 9.80 MB/sec execute 114 sec latency 85.527 ms 2024-04-23T14:49:39.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:38 smithi105 ceph-mon[32555]: pgmap v171: 97 pgs: 97 active+clean; 257 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.9 MiB/s wr, 238 op/s 2024-04-23T14:49:39.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:38 smithi038 ceph-mon[32348]: pgmap v171: 97 pgs: 97 active+clean; 257 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.9 MiB/s wr, 238 op/s 2024-04-23T14:49:39.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:38 smithi028 ceph-mon[27810]: pgmap v171: 97 pgs: 97 active+clean; 257 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.9 MiB/s wr, 238 op/s 2024-04-23T14:49:39.491 INFO:tasks.workunit.client.0.smithi028.stdout: 1 443971 9.82 MB/sec execute 115 sec latency 9.227 ms 2024-04-23T14:49:40.491 INFO:tasks.workunit.client.0.smithi028.stdout: 1 445812 9.80 MB/sec execute 116 sec latency 39.227 ms 2024-04-23T14:49:41.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:40 smithi105 ceph-mon[32555]: pgmap v172: 97 pgs: 97 active+clean; 257 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 157 op/s 2024-04-23T14:49:41.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:40 smithi105 ceph-mon[32555]: scrub summary: idle 2024-04-23T14:49:41.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:40 smithi038 ceph-mon[32348]: pgmap v172: 97 pgs: 97 active+clean; 257 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 157 op/s 2024-04-23T14:49:41.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:40 smithi038 ceph-mon[32348]: scrub summary: idle 2024-04-23T14:49:41.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:40 smithi028 ceph-mon[27810]: pgmap v172: 97 pgs: 97 active+clean; 257 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 157 op/s 2024-04-23T14:49:41.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:40 smithi028 ceph-mon[27810]: scrub summary: idle 2024-04-23T14:49:41.491 INFO:tasks.workunit.client.0.smithi028.stdout: 1 447404 9.82 MB/sec execute 117 sec latency 25.693 ms 2024-04-23T14:49:42.491 INFO:tasks.workunit.client.0.smithi028.stdout: 1 449372 9.79 MB/sec execute 118 sec latency 18.379 ms 2024-04-23T14:49:43.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:42 smithi105 ceph-mon[32555]: pgmap v173: 97 pgs: 97 active+clean; 257 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 157 op/s 2024-04-23T14:49:43.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:42 smithi038 ceph-mon[32348]: pgmap v173: 97 pgs: 97 active+clean; 257 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 157 op/s 2024-04-23T14:49:43.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:42 smithi028 ceph-mon[27810]: pgmap v173: 97 pgs: 97 active+clean; 257 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 157 op/s 2024-04-23T14:49:43.492 INFO:tasks.workunit.client.0.smithi028.stdout: 1 450942 9.81 MB/sec execute 119 sec latency 9.693 ms 2024-04-23T14:49:44.492 INFO:tasks.workunit.client.0.smithi028.stdout: 1 452884 9.79 MB/sec execute 120 sec latency 16.309 ms 2024-04-23T14:49:45.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:44 smithi105 ceph-mon[32555]: pgmap v174: 97 pgs: 97 active+clean; 262 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 235 op/s 2024-04-23T14:49:45.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:44 smithi038 ceph-mon[32348]: pgmap v174: 97 pgs: 97 active+clean; 262 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 235 op/s 2024-04-23T14:49:45.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:44 smithi028 ceph-mon[27810]: pgmap v174: 97 pgs: 97 active+clean; 262 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 235 op/s 2024-04-23T14:49:45.492 INFO:tasks.workunit.client.0.smithi028.stdout: 1 454415 9.81 MB/sec execute 121 sec latency 35.351 ms 2024-04-23T14:49:46.492 INFO:tasks.workunit.client.0.smithi028.stdout: 1 456017 9.77 MB/sec execute 122 sec latency 47.329 ms 2024-04-23T14:49:47.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:46 smithi105 ceph-mon[32555]: pgmap v175: 97 pgs: 97 active+clean; 262 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 151 op/s 2024-04-23T14:49:47.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:46 smithi038 ceph-mon[32348]: pgmap v175: 97 pgs: 97 active+clean; 262 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 151 op/s 2024-04-23T14:49:47.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:46 smithi028 ceph-mon[27810]: pgmap v175: 97 pgs: 97 active+clean; 262 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 151 op/s 2024-04-23T14:49:47.492 INFO:tasks.workunit.client.0.smithi028.stdout: 1 457904 9.80 MB/sec execute 123 sec latency 10.979 ms 2024-04-23T14:49:48.492 INFO:tasks.workunit.client.0.smithi028.stdout: 1 458809 9.86 MB/sec execute 124 sec latency 61.481 ms 2024-04-23T14:49:49.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:48 smithi105 ceph-mon[32555]: pgmap v176: 97 pgs: 97 active+clean; 268 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.7 MiB/s wr, 224 op/s 2024-04-23T14:49:49.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:48 smithi038 ceph-mon[32348]: pgmap v176: 97 pgs: 97 active+clean; 268 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.7 MiB/s wr, 224 op/s 2024-04-23T14:49:49.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:48 smithi028 ceph-mon[27810]: pgmap v176: 97 pgs: 97 active+clean; 268 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.7 MiB/s wr, 224 op/s 2024-04-23T14:49:49.492 INFO:tasks.workunit.client.0.smithi028.stdout: 1 460791 9.87 MB/sec execute 125 sec latency 12.375 ms 2024-04-23T14:49:50.492 INFO:tasks.workunit.client.0.smithi028.stdout: 1 462488 9.89 MB/sec execute 126 sec latency 29.617 ms 2024-04-23T14:49:51.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:50 smithi105 ceph-mon[32555]: pgmap v177: 97 pgs: 97 active+clean; 268 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 939 KiB/s rd, 2.5 MiB/s wr, 151 op/s 2024-04-23T14:49:51.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:50 smithi038 ceph-mon[32348]: pgmap v177: 97 pgs: 97 active+clean; 268 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 939 KiB/s rd, 2.5 MiB/s wr, 151 op/s 2024-04-23T14:49:51.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:50 smithi028 ceph-mon[27810]: pgmap v177: 97 pgs: 97 active+clean; 268 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 939 KiB/s rd, 2.5 MiB/s wr, 151 op/s 2024-04-23T14:49:51.492 INFO:tasks.workunit.client.0.smithi028.stdout: 1 464332 9.87 MB/sec execute 127 sec latency 61.653 ms 2024-04-23T14:49:52.492 INFO:tasks.workunit.client.0.smithi028.stdout: 1 466167 9.89 MB/sec execute 128 sec latency 14.097 ms 2024-04-23T14:49:53.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:52 smithi105 ceph-mon[32555]: pgmap v178: 97 pgs: 97 active+clean; 268 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 939 KiB/s rd, 2.5 MiB/s wr, 151 op/s 2024-04-23T14:49:53.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:52 smithi038 ceph-mon[32348]: pgmap v178: 97 pgs: 97 active+clean; 268 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 939 KiB/s rd, 2.5 MiB/s wr, 151 op/s 2024-04-23T14:49:53.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:52 smithi028 ceph-mon[27810]: pgmap v178: 97 pgs: 97 active+clean; 268 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 939 KiB/s rd, 2.5 MiB/s wr, 151 op/s 2024-04-23T14:49:53.493 INFO:tasks.workunit.client.0.smithi028.stdout: 1 467937 9.87 MB/sec execute 129 sec latency 31.454 ms 2024-04-23T14:49:54.493 INFO:tasks.workunit.client.0.smithi028.stdout: 1 469408 9.88 MB/sec execute 130 sec latency 11.186 ms 2024-04-23T14:49:55.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:54 smithi105 ceph-mon[32555]: pgmap v179: 97 pgs: 97 active+clean; 274 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.9 MiB/s wr, 231 op/s 2024-04-23T14:49:55.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:54 smithi038 ceph-mon[32348]: pgmap v179: 97 pgs: 97 active+clean; 274 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.9 MiB/s wr, 231 op/s 2024-04-23T14:49:55.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:54 smithi028 ceph-mon[27810]: pgmap v179: 97 pgs: 97 active+clean; 274 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.9 MiB/s wr, 231 op/s 2024-04-23T14:49:55.493 INFO:tasks.workunit.client.0.smithi028.stdout: 1 470769 9.85 MB/sec execute 131 sec latency 19.368 ms 2024-04-23T14:49:56.493 INFO:tasks.workunit.client.0.smithi028.stdout: 1 472371 9.84 MB/sec execute 132 sec latency 102.021 ms 2024-04-23T14:49:57.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:56 smithi105 ceph-mon[32555]: pgmap v180: 97 pgs: 97 active+clean; 274 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 2.7 MiB/s wr, 153 op/s 2024-04-23T14:49:57.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:56 smithi038 ceph-mon[32348]: pgmap v180: 97 pgs: 97 active+clean; 274 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 2.7 MiB/s wr, 153 op/s 2024-04-23T14:49:57.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:56 smithi028 ceph-mon[27810]: pgmap v180: 97 pgs: 97 active+clean; 274 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 2.7 MiB/s wr, 153 op/s 2024-04-23T14:49:57.493 INFO:tasks.workunit.client.0.smithi028.stdout: 1 473484 9.81 MB/sec execute 133 sec latency 24.247 ms 2024-04-23T14:49:58.493 INFO:tasks.workunit.client.0.smithi028.stdout: 1 475289 9.81 MB/sec execute 134 sec latency 17.777 ms 2024-04-23T14:49:59.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:49:58 smithi105 ceph-mon[32555]: pgmap v181: 97 pgs: 97 active+clean; 278 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.9 MiB/s wr, 224 op/s 2024-04-23T14:49:59.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:49:58 smithi038 ceph-mon[32348]: pgmap v181: 97 pgs: 97 active+clean; 278 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.9 MiB/s wr, 224 op/s 2024-04-23T14:49:59.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:49:58 smithi028 ceph-mon[27810]: pgmap v181: 97 pgs: 97 active+clean; 278 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.9 MiB/s wr, 224 op/s 2024-04-23T14:49:59.493 INFO:tasks.workunit.client.0.smithi028.stdout: 1 476754 9.80 MB/sec execute 135 sec latency 16.548 ms 2024-04-23T14:50:00.493 INFO:tasks.workunit.client.0.smithi028.stdout: 1 478474 9.78 MB/sec execute 136 sec latency 18.781 ms 2024-04-23T14:50:01.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:00 smithi105 ceph-mon[32555]: pgmap v182: 97 pgs: 97 active+clean; 278 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1009 KiB/s rd, 2.6 MiB/s wr, 151 op/s 2024-04-23T14:50:01.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:00 smithi105 ceph-mon[32555]: overall HEALTH_OK 2024-04-23T14:50:01.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:00 smithi038 ceph-mon[32348]: pgmap v182: 97 pgs: 97 active+clean; 278 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1009 KiB/s rd, 2.6 MiB/s wr, 151 op/s 2024-04-23T14:50:01.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:00 smithi038 ceph-mon[32348]: overall HEALTH_OK 2024-04-23T14:50:01.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:00 smithi028 ceph-mon[27810]: pgmap v182: 97 pgs: 97 active+clean; 278 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1009 KiB/s rd, 2.6 MiB/s wr, 151 op/s 2024-04-23T14:50:01.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:00 smithi028 ceph-mon[27810]: overall HEALTH_OK 2024-04-23T14:50:01.493 INFO:tasks.workunit.client.0.smithi028.stdout: 1 479381 9.76 MB/sec execute 137 sec latency 144.177 ms 2024-04-23T14:50:02.494 INFO:tasks.workunit.client.0.smithi028.stdout: 1 480379 9.73 MB/sec execute 138 sec latency 68.127 ms 2024-04-23T14:50:03.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:02 smithi105 ceph-mon[32555]: pgmap v183: 97 pgs: 97 active+clean; 278 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1009 KiB/s rd, 2.6 MiB/s wr, 151 op/s 2024-04-23T14:50:03.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:02 smithi038 ceph-mon[32348]: pgmap v183: 97 pgs: 97 active+clean; 278 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1009 KiB/s rd, 2.6 MiB/s wr, 151 op/s 2024-04-23T14:50:03.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:02 smithi028 ceph-mon[27810]: pgmap v183: 97 pgs: 97 active+clean; 278 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1009 KiB/s rd, 2.6 MiB/s wr, 151 op/s 2024-04-23T14:50:03.494 INFO:tasks.workunit.client.0.smithi028.stdout: 1 482277 9.71 MB/sec execute 139 sec latency 16.203 ms 2024-04-23T14:50:04.494 INFO:tasks.workunit.client.0.smithi028.stdout: 1 484026 9.72 MB/sec execute 140 sec latency 17.989 ms 2024-04-23T14:50:05.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:04 smithi105 ceph-mon[32555]: pgmap v184: 97 pgs: 97 active+clean; 282 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.9 MiB/s wr, 254 op/s 2024-04-23T14:50:05.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:04 smithi038 ceph-mon[32348]: pgmap v184: 97 pgs: 97 active+clean; 282 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.9 MiB/s wr, 254 op/s 2024-04-23T14:50:05.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:04 smithi028 ceph-mon[27810]: pgmap v184: 97 pgs: 97 active+clean; 282 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.9 MiB/s wr, 254 op/s 2024-04-23T14:50:05.494 INFO:tasks.workunit.client.0.smithi028.stdout: 1 485975 9.73 MB/sec execute 141 sec latency 21.710 ms 2024-04-23T14:50:06.423 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:50:06.424 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:50:06.494 INFO:tasks.workunit.client.0.smithi028.stdout: 1 487380 9.72 MB/sec execute 142 sec latency 35.900 ms 2024-04-23T14:50:06.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.659+0000 7f4c5b800640 1 -- 172.21.15.28:0/3093246168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5406a680 msgr2=0x7f4c5406aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:06.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.659+0000 7f4c5b800640 1 --2- 172.21.15.28:0/3093246168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5406a680 0x7f4c5406aa60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4c44009900 tx=0x7f4c4402a3f0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.659+0000 7f4c5b800640 1 -- 172.21.15.28:0/3093246168 shutdown_connections 2024-04-23T14:50:06.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.659+0000 7f4c5b800640 1 --2- 172.21.15.28:0/3093246168 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c5406afa0 0x7f4c5410e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.659+0000 7f4c5b800640 1 --2- 172.21.15.28:0/3093246168 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c5410ea10 0x7f4c54110e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.659+0000 7f4c5b800640 1 --2- 172.21.15.28:0/3093246168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5406a680 0x7f4c5406aa60 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.659+0000 7f4c5b800640 1 -- 172.21.15.28:0/3093246168 >> 172.21.15.28:0/3093246168 conn(0x7f4c54077f60 msgr2=0x7f4c54078370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:06.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.659+0000 7f4c5b800640 1 -- 172.21.15.28:0/3093246168 shutdown_connections 2024-04-23T14:50:06.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.659+0000 7f4c5b800640 1 -- 172.21.15.28:0/3093246168 wait complete. 2024-04-23T14:50:06.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.660+0000 7f4c5b800640 1 Processor -- start 2024-04-23T14:50:06.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.660+0000 7f4c5b800640 1 -- start start 2024-04-23T14:50:06.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.661+0000 7f4c5b800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c5406a680 0x7f4c541a0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:06.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.661+0000 7f4c5b800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5406afa0 0x7f4c541a0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:06.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.661+0000 7f4c5b800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c5410ea10 0x7f4c5419a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:06.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.661+0000 7f4c5b800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4c54114610 con 0x7f4c5406afa0 2024-04-23T14:50:06.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.661+0000 7f4c5b800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4c54114490 con 0x7f4c5410ea10 2024-04-23T14:50:06.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.661+0000 7f4c5b800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4c54114790 con 0x7f4c5406a680 2024-04-23T14:50:06.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.661+0000 7f4c58e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c5406a680 0x7f4c541a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:06.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.661+0000 7f4c58e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c5406a680 0x7f4c541a0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:49612/0 (socket says 172.21.15.28:49612) 2024-04-23T14:50:06.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.661+0000 7f4c58e00640 1 -- 172.21.15.28:0/1223271363 learned_addr learned my addr 172.21.15.28:0/1223271363 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:06.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.661+0000 7f4c59800640 1 --2- 172.21.15.28:0/1223271363 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c5410ea10 0x7f4c5419a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:06.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.661+0000 7f4c4be00640 1 --2- 172.21.15.28:0/1223271363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5406afa0 0x7f4c541a0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:06.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.661+0000 7f4c58e00640 1 -- 172.21.15.28:0/1223271363 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c5410ea10 msgr2=0x7f4c5419a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:06.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.661+0000 7f4c58e00640 1 --2- 172.21.15.28:0/1223271363 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c5410ea10 0x7f4c5419a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.661+0000 7f4c58e00640 1 -- 172.21.15.28:0/1223271363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5406afa0 msgr2=0x7f4c541a0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:06.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.661+0000 7f4c58e00640 1 --2- 172.21.15.28:0/1223271363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5406afa0 0x7f4c541a0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.661+0000 7f4c58e00640 1 -- 172.21.15.28:0/1223271363 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c5419b110 con 0x7f4c5406a680 2024-04-23T14:50:06.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.662+0000 7f4c58e00640 1 --2- 172.21.15.28:0/1223271363 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c5406a680 0x7f4c541a0790 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4c44004770 tx=0x7f4c44003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:06.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.662+0000 7f4c52000640 1 -- 172.21.15.28:0/1223271363 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c4403f070 con 0x7f4c5406a680 2024-04-23T14:50:06.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.662+0000 7f4c52000640 1 -- 172.21.15.28:0/1223271363 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c44031ca0 con 0x7f4c5406a680 2024-04-23T14:50:06.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.662+0000 7f4c52000640 1 -- 172.21.15.28:0/1223271363 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c44035030 con 0x7f4c5406a680 2024-04-23T14:50:06.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.662+0000 7f4c5b800640 1 -- 172.21.15.28:0/1223271363 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c5419b3a0 con 0x7f4c5406a680 2024-04-23T14:50:06.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.662+0000 7f4c5b800640 1 -- 172.21.15.28:0/1223271363 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c541a7650 con 0x7f4c5406a680 2024-04-23T14:50:06.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.666+0000 7f4c52000640 1 -- 172.21.15.28:0/1223271363 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f4c44042070 con 0x7f4c5406a680 2024-04-23T14:50:06.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.666+0000 7f4c5e578740 1 Processor -- start 2024-04-23T14:50:06.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.666+0000 7f4c5e578740 1 -- start start 2024-04-23T14:50:06.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.667+0000 7f4c5e578740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7f475c860 0x55f7f475cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:06.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.667+0000 7f4c5e578740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7f475d180 0x55f7f4767310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:06.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.667+0000 7f4c5e578740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7f4767850 0x55f7f4769c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:06.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.667+0000 7f4c5e578740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f7f46c21f0 con 0x55f7f475c860 2024-04-23T14:50:06.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.667+0000 7f4c5e578740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f7f4668350 con 0x55f7f4767850 2024-04-23T14:50:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.667+0000 7f4c5e578740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f7f46be1b0 con 0x55f7f475d180 2024-04-23T14:50:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.667+0000 7f4c59800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7f475c860 0x55f7f475cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.667+0000 7f4c4be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7f4767850 0x55f7f4769c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.667+0000 7f4c4be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7f4767850 0x55f7f4769c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:33610/0 (socket says 172.21.15.28:33610) 2024-04-23T14:50:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.667+0000 7f4c59800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7f475c860 0x55f7f475cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48126/0 (socket says 172.21.15.28:48126) 2024-04-23T14:50:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.667+0000 7f4c59800640 1 -- 172.21.15.28:0/2242693842 learned_addr learned my addr 172.21.15.28:0/2242693842 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.667+0000 7f4c58e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7f475d180 0x55f7f4767310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.667+0000 7f4c4be00640 1 -- 172.21.15.28:0/2242693842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7f475d180 msgr2=0x55f7f4767310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.667+0000 7f4c4be00640 1 --2- 172.21.15.28:0/2242693842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7f475d180 0x55f7f4767310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.667+0000 7f4c4be00640 1 -- 172.21.15.28:0/2242693842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7f475c860 msgr2=0x55f7f475cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.667+0000 7f4c4be00640 1 --2- 172.21.15.28:0/2242693842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7f475c860 0x55f7f475cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.667+0000 7f4c4be00640 1 -- 172.21.15.28:0/2242693842 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7f46bcf50 con 0x55f7f4767850 2024-04-23T14:50:06.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.668+0000 7f4c4be00640 1 --2- 172.21.15.28:0/2242693842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7f4767850 0x55f7f4769c40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f4c5419c130 tx=0x7f4c40011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0f0c62f3b070f74 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:06.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.668+0000 7f4c4a000640 1 -- 172.21.15.28:0/2242693842 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c4001e070 con 0x55f7f4767850 2024-04-23T14:50:06.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.668+0000 7f4c52000640 1 --2- 172.21.15.28:0/1223271363 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f4c380a3d20 0x7f4c380a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:06.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.668+0000 7f4c4a000640 1 -- 172.21.15.28:0/2242693842 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c40009d70 con 0x55f7f4767850 2024-04-23T14:50:06.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.668+0000 7f4c4a000640 1 -- 172.21.15.28:0/2242693842 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c400187d0 con 0x55f7f4767850 2024-04-23T14:50:06.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.668+0000 7f4c52000640 1 -- 172.21.15.28:0/1223271363 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f4c440f0e90 con 0x7f4c5406a680 2024-04-23T14:50:06.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.668+0000 7f4c59800640 1 --2- 172.21.15.28:0/1223271363 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f4c380a3d20 0x7f4c380a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:06.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.669+0000 7f4c5e578740 1 -- 172.21.15.28:0/2242693842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7f4767850 msgr2=0x55f7f4769c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:06.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.669+0000 7f4c5e578740 1 --2- 172.21.15.28:0/2242693842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7f4767850 0x55f7f4769c40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f4c5419c130 tx=0x7f4c40011810 comp rx=0 tx=0).stop 2024-04-23T14:50:06.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.669+0000 7f4c59800640 1 --2- 172.21.15.28:0/1223271363 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f4c380a3d20 0x7f4c380a61e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009fd0 tx=0x7f4c4c0096a0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:06.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.669+0000 7f4c5e578740 1 -- 172.21.15.28:0/2242693842 shutdown_connections 2024-04-23T14:50:06.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.669+0000 7f4c5e578740 1 --2- 172.21.15.28:0/2242693842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7f4767850 0x55f7f4769c40 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.669+0000 7f4c5e578740 1 --2- 172.21.15.28:0/2242693842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7f475d180 0x55f7f4767310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.669+0000 7f4c5e578740 1 --2- 172.21.15.28:0/2242693842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7f475c860 0x55f7f475cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.669+0000 7f4c5e578740 1 -- 172.21.15.28:0/2242693842 >> 172.21.15.28:0/2242693842 conn(0x55f7f4758b80 msgr2=0x55f7f4758f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:06.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.669+0000 7f4c5e578740 1 -- 172.21.15.28:0/2242693842 shutdown_connections 2024-04-23T14:50:06.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.669+0000 7f4c5e578740 1 -- 172.21.15.28:0/2242693842 wait complete. 2024-04-23T14:50:06.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.670+0000 7f4c5e578740 1 -- start start 2024-04-23T14:50:06.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.670+0000 7f4c5e578740 1 Processor -- start 2024-04-23T14:50:06.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.670+0000 7f4c5e578740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7f475c860 0x55f7f4764800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:06.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.670+0000 7f4c5e578740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7f475d180 0x55f7f4764d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:06.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.670+0000 7f4c5e578740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7f4767850 0x55f7f4765280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:06.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.670+0000 7f4c5e578740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f7f476a640 con 0x55f7f475c860 2024-04-23T14:50:06.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.670+0000 7f4c5e578740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f7f467b690 con 0x55f7f475d180 2024-04-23T14:50:06.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.670+0000 7f4c5e578740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f7f476a7c0 con 0x55f7f4767850 2024-04-23T14:50:06.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.671+0000 7f4c5e578740 10 client.0 _ll_register_callbacks cb 0x55f7f46b7650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:50:06.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.671+0000 7f4c4be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7f475d180 0x55f7f4764d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:06.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.671+0000 7f4c58e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7f475c860 0x55f7f4764800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:06.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.671+0000 7f4c59800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7f4767850 0x55f7f4765280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:06.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.671+0000 7f4c4be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7f475d180 0x55f7f4764d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:33618/0 (socket says 172.21.15.28:33618) 2024-04-23T14:50:06.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.671+0000 7f4c4be00640 1 -- 172.21.15.28:0/1675015102 learned_addr learned my addr 172.21.15.28:0/1675015102 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:06.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.671+0000 7f4c4be00640 1 -- 172.21.15.28:0/1675015102 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7f4767850 msgr2=0x55f7f4765280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:06.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.671+0000 7f4c4be00640 1 --2- 172.21.15.28:0/1675015102 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7f4767850 0x55f7f4765280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.671+0000 7f4c4be00640 1 -- 172.21.15.28:0/1675015102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7f475c860 msgr2=0x55f7f4764800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:06.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.671+0000 7f4c4be00640 1 --2- 172.21.15.28:0/1675015102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7f475c860 0x55f7f4764800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.671+0000 7f4c4be00640 1 -- 172.21.15.28:0/1675015102 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7f467a920 con 0x55f7f475d180 2024-04-23T14:50:06.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.672+0000 7f4c4be00640 1 --2- 172.21.15.28:0/1675015102 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7f475d180 0x55f7f4764d40 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4c40019f60 tx=0x7f4c40009fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d95f451c15eabd8d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:06.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.672+0000 7f4c3ea00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:50:06.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.672+0000 7f4c3ea00640 1 -- 172.21.15.28:0/1675015102 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c4001e040 con 0x55f7f475d180 2024-04-23T14:50:06.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.672+0000 7f4c5e578740 1 -- 172.21.15.28:0/1675015102 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c40009d70 con 0x55f7f475d180 2024-04-23T14:50:06.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.672+0000 7f4c3ea00640 1 -- 172.21.15.28:0/1675015102 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f4c40005040 con 0x55f7f475d180 2024-04-23T14:50:06.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.672+0000 7f4c3ea00640 1 -- 172.21.15.28:0/1675015102 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f4c40032050 con 0x55f7f475d180 2024-04-23T14:50:06.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.672+0000 7f4c5e578740 10 client.26150 fetch_fsmap learned FSMap version 20 2024-04-23T14:50:06.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.672+0000 7f4c5e578740 1 -- 172.21.15.28:0/1675015102 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55f7f449e7d0 con 0x55f7f475d180 2024-04-23T14:50:06.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.672+0000 7f4c3ea00640 1 -- 172.21.15.28:0/1675015102 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c40009d70 con 0x55f7f475d180 2024-04-23T14:50:06.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.673+0000 7f4c3ea00640 1 -- 172.21.15.28:0/1675015102 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f4c40019410 con 0x55f7f475d180 2024-04-23T14:50:06.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.673+0000 7f4c5e578740 10 client.26150 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:50:06.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.673+0000 7f4c5e578740 10 client.26150 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:50:06.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.673+0000 7f4c5e578740 20 client.26150 populate_metadata read hostname 'smithi028' 2024-04-23T14:50:06.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.673+0000 7f4c5e578740 1 --2- 172.21.15.28:0/1675015102 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f7f4777b60 0x55f7f4779f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:06.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.673+0000 7f4c5e578740 4 client.26150 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:50:06.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.673+0000 7f4c5e578740 1 -- 172.21.15.28:0/1675015102 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f7f46cfc10 con 0x55f7f4777b60 2024-04-23T14:50:06.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.673+0000 7f4c58e00640 1 --2- 172.21.15.28:0/1675015102 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f7f4777b60 0x55f7f4779f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:06.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.674+0000 7f4c58e00640 1 --2- 172.21.15.28:0/1675015102 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f7f4777b60 0x55f7f4779f50 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=881d2848c40d159e server_cookie=b6c1901560b16f5e in_seq=0 out_seq=0 2024-04-23T14:50:06.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.674+0000 7f4c3ea00640 10 client.26150 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:50:06.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.675+0000 7f4c3ea00640 1 -- 172.21.15.28:0/1675015102 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f7f46cfc10 con 0x55f7f4777b60 2024-04-23T14:50:06.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.675+0000 7f4c3ea00640 10 client.26150 handle_command_reply: tid=0 2024-04-23T14:50:06.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.675+0000 7f4c5e578740 1 client.26150 shutdown 2024-04-23T14:50:06.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.675+0000 7f4c5e578740 1 -- 172.21.15.28:0/1675015102 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7f475d180 msgr2=0x55f7f4764d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:06.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.675+0000 7f4c5e578740 1 --2- 172.21.15.28:0/1675015102 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7f475d180 0x55f7f4764d40 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4c40019f60 tx=0x7f4c40009fc0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.675+0000 7f4c5e578740 1 -- 172.21.15.28:0/1675015102 shutdown_connections 2024-04-23T14:50:06.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.675+0000 7f4c5e578740 1 --2- 172.21.15.28:0/1675015102 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f7f4777b60 0x55f7f4779f50 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.675+0000 7f4c5e578740 1 --2- 172.21.15.28:0/1675015102 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7f475d180 0x55f7f4764d40 secure :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4c40019f60 tx=0x7f4c40009fc0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.675+0000 7f4c5e578740 1 --2- 172.21.15.28:0/1675015102 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7f4767850 0x55f7f4765280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.675+0000 7f4c5e578740 1 --2- 172.21.15.28:0/1675015102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7f475c860 0x55f7f4764800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.675+0000 7f4c3ea00640 0 client.26150 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:50:06.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.675+0000 7f4c5e578740 1 -- 172.21.15.28:0/1675015102 >> 172.21.15.28:0/1675015102 conn(0x55f7f4758b80 msgr2=0x55f7f475e880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:06.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.675+0000 7f4c5e578740 1 -- 172.21.15.28:0/1675015102 shutdown_connections 2024-04-23T14:50:06.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.675+0000 7f4c5e578740 1 -- 172.21.15.28:0/1675015102 wait complete. 2024-04-23T14:50:06.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.676+0000 7f4c5e578740 20 client.26150 trim_cache size 0 max 16384 2024-04-23T14:50:06.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.697+0000 7f4c5e578740 1 Processor -- start 2024-04-23T14:50:06.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.697+0000 7f4c5e578740 1 -- start start 2024-04-23T14:50:06.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.697+0000 7f4c5e578740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7f4758b80 0x55f7f47583a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:06.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.697+0000 7f4c5e578740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7f475c860 0x55f7f4764ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:06.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.697+0000 7f4c5e578740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7f4767850 0x55f7f477f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:06.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.697+0000 7f4c5e578740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4c541adf60 con 0x55f7f4767850 2024-04-23T14:50:06.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.697+0000 7f4c5e578740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4c541b07e0 con 0x55f7f475c860 2024-04-23T14:50:06.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.697+0000 7f4c5e578740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4c541ae5d0 con 0x55f7f4758b80 2024-04-23T14:50:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.698+0000 7f4c4be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7f475c860 0x55f7f4764ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.698+0000 7f4c4be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7f475c860 0x55f7f4764ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:33624/0 (socket says 172.21.15.28:33624) 2024-04-23T14:50:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.698+0000 7f4c4be00640 1 -- 172.21.15.28:0/632701322 learned_addr learned my addr 172.21.15.28:0/632701322 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.698+0000 7f4c59800640 1 --2- 172.21.15.28:0/632701322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7f4767850 0x55f7f477f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.698+0000 7f4c58e00640 1 --2- 172.21.15.28:0/632701322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7f4758b80 0x55f7f47583a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.698+0000 7f4c4be00640 1 -- 172.21.15.28:0/632701322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7f4758b80 msgr2=0x55f7f47583a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.698+0000 7f4c4be00640 1 --2- 172.21.15.28:0/632701322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7f4758b80 0x55f7f47583a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.698+0000 7f4c4be00640 1 -- 172.21.15.28:0/632701322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7f4767850 msgr2=0x55f7f477f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.698+0000 7f4c4be00640 1 --2- 172.21.15.28:0/632701322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7f4767850 0x55f7f477f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.698+0000 7f4c4be00640 1 -- 172.21.15.28:0/632701322 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7f475cfb0 con 0x55f7f475c860 2024-04-23T14:50:06.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.698+0000 7f4c59800640 1 --2- 172.21.15.28:0/632701322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7f4767850 0x55f7f477f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:50:06.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.698+0000 7f4c4be00640 1 --2- 172.21.15.28:0/632701322 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7f475c860 0x55f7f4764ed0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4c40019860 tx=0x7f4c40011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddaa96e1de88b731 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:06.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.699+0000 7f4c49600640 1 -- 172.21.15.28:0/632701322 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c4003b070 con 0x55f7f475c860 2024-04-23T14:50:06.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.699+0000 7f4c49600640 1 -- 172.21.15.28:0/632701322 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c40016070 con 0x55f7f475c860 2024-04-23T14:50:06.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.699+0000 7f4c49600640 1 -- 172.21.15.28:0/632701322 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c4001e050 con 0x55f7f475c860 2024-04-23T14:50:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.699+0000 7f4c5e578740 1 -- 172.21.15.28:0/632701322 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7f475c860 msgr2=0x55f7f4764ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.699+0000 7f4c5e578740 1 --2- 172.21.15.28:0/632701322 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7f475c860 0x55f7f4764ed0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4c40019860 tx=0x7f4c40011810 comp rx=0 tx=0).stop 2024-04-23T14:50:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.700+0000 7f4c5e578740 1 -- 172.21.15.28:0/632701322 shutdown_connections 2024-04-23T14:50:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.700+0000 7f4c5e578740 1 --2- 172.21.15.28:0/632701322 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7f475c860 0x55f7f4764ed0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.700+0000 7f4c5e578740 1 --2- 172.21.15.28:0/632701322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7f4767850 0x55f7f477f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.700+0000 7f4c5e578740 1 --2- 172.21.15.28:0/632701322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7f4758b80 0x55f7f47583a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.700+0000 7f4c5e578740 1 -- 172.21.15.28:0/632701322 >> 172.21.15.28:0/632701322 conn(0x55f7f4763440 msgr2=0x55f7f47647c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.700+0000 7f4c5e578740 1 -- 172.21.15.28:0/632701322 shutdown_connections 2024-04-23T14:50:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.700+0000 7f4c5e578740 1 -- 172.21.15.28:0/632701322 wait complete. 2024-04-23T14:50:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.700+0000 7f4c5e578740 1 -- start start 2024-04-23T14:50:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.700+0000 7f4c5e578740 1 Processor -- start 2024-04-23T14:50:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.701+0000 7f4c5e578740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7f4758b80 0x55f7f476fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.701+0000 7f4c5e578740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7f475c860 0x55f7f476ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.701+0000 7f4c5e578740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7f4767850 0x55f7f476ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.701+0000 7f4c5e578740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4c541aeca0 con 0x55f7f4758b80 2024-04-23T14:50:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.701+0000 7f4c5e578740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4c541b0110 con 0x55f7f4767850 2024-04-23T14:50:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.701+0000 7f4c5e578740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4c541af370 con 0x55f7f475c860 2024-04-23T14:50:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.701+0000 7f4c5e578740 10 client.0 _ll_register_callbacks cb 0x7f4c4001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:50:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.701+0000 7f4c59800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7f4767850 0x55f7f476ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.701+0000 7f4c4be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7f475c860 0x55f7f476ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.701+0000 7f4c58e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7f4758b80 0x55f7f476fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.701+0000 7f4c4be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7f475c860 0x55f7f476ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:49652/0 (socket says 172.21.15.28:49652) 2024-04-23T14:50:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.701+0000 7f4c59800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7f4767850 0x55f7f476ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:33636/0 (socket says 172.21.15.28:33636) 2024-04-23T14:50:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.701+0000 7f4c4be00640 1 -- 172.21.15.28:0/1089693652 learned_addr learned my addr 172.21.15.28:0/1089693652 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:06.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.701+0000 7f4c4be00640 1 -- 172.21.15.28:0/1089693652 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7f4767850 msgr2=0x55f7f476ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:06.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.701+0000 7f4c4be00640 1 --2- 172.21.15.28:0/1089693652 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7f4767850 0x55f7f476ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.701+0000 7f4c4be00640 1 -- 172.21.15.28:0/1089693652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7f4758b80 msgr2=0x55f7f476fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:06.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.701+0000 7f4c4be00640 1 --2- 172.21.15.28:0/1089693652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7f4758b80 0x55f7f476fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.702+0000 7f4c4be00640 1 -- 172.21.15.28:0/1089693652 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7f43d43f0 con 0x55f7f475c860 2024-04-23T14:50:06.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.702+0000 7f4c4be00640 1 --2- 172.21.15.28:0/1089693652 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7f475c860 0x55f7f476ff80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4c400199d0 tx=0x7f4c40009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bae1a1317ace8c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:06.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.702+0000 7f4c3ea00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:50:06.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.702+0000 7f4c3ea00640 1 -- 172.21.15.28:0/1089693652 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c4001e070 con 0x55f7f475c860 2024-04-23T14:50:06.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.702+0000 7f4c3ea00640 1 -- 172.21.15.28:0/1089693652 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f4c40019410 con 0x55f7f475c860 2024-04-23T14:50:06.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.702+0000 7f4c5e578740 1 -- 172.21.15.28:0/1089693652 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c40016070 con 0x55f7f475c860 2024-04-23T14:50:06.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.702+0000 7f4c3ea00640 1 -- 172.21.15.28:0/1089693652 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f4c40046050 con 0x55f7f475c860 2024-04-23T14:50:06.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.702+0000 7f4c5e578740 10 client.26143 fetch_fsmap learned FSMap version 20 2024-04-23T14:50:06.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.702+0000 7f4c5e578740 1 -- 172.21.15.28:0/1089693652 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55f7f43940f0 con 0x55f7f475c860 2024-04-23T14:50:06.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.703+0000 7f4c3ea00640 1 -- 172.21.15.28:0/1089693652 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c40016070 con 0x55f7f475c860 2024-04-23T14:50:06.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.703+0000 7f4c3ea00640 1 -- 172.21.15.28:0/1089693652 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f4c400053c0 con 0x55f7f475c860 2024-04-23T14:50:06.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.703+0000 7f4c5e578740 10 client.26143 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:50:06.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.703+0000 7f4c5e578740 10 client.26143 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:50:06.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.703+0000 7f4c5e578740 20 client.26143 populate_metadata read hostname 'smithi028' 2024-04-23T14:50:06.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.703+0000 7f4c5e578740 1 --2- 172.21.15.28:0/1089693652 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f7f47debc0 0x55f7f47e0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:06.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.703+0000 7f4c5e578740 4 client.26143 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:50:06.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.703+0000 7f4c5e578740 1 -- 172.21.15.28:0/1089693652 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f7f46c9a80 con 0x55f7f47debc0 2024-04-23T14:50:06.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.703+0000 7f4c58e00640 1 --2- 172.21.15.28:0/1089693652 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f7f47debc0 0x55f7f47e0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:06.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.704+0000 7f4c58e00640 1 --2- 172.21.15.28:0/1089693652 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f7f47debc0 0x55f7f47e0fe0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ea1b0931818e636 server_cookie=13363336c9e04842 in_seq=0 out_seq=0 2024-04-23T14:50:06.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.704+0000 7f4c3ea00640 10 client.26143 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:50:06.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.704+0000 7f4c3ea00640 1 -- 172.21.15.28:0/1089693652 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f7f46c9a80 con 0x55f7f47debc0 2024-04-23T14:50:06.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.704+0000 7f4c3ea00640 10 client.26143 handle_command_reply: tid=0 2024-04-23T14:50:06.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.704+0000 7f4c5e578740 1 client.26143 shutdown 2024-04-23T14:50:06.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.704+0000 7f4c5e578740 1 -- 172.21.15.28:0/1089693652 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7f475c860 msgr2=0x55f7f476ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:06.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.704+0000 7f4c5e578740 1 --2- 172.21.15.28:0/1089693652 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7f475c860 0x55f7f476ff80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4c400199d0 tx=0x7f4c40009260 comp rx=0 tx=0).stop 2024-04-23T14:50:06.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.704+0000 7f4c5e578740 1 -- 172.21.15.28:0/1089693652 shutdown_connections 2024-04-23T14:50:06.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.704+0000 7f4c5e578740 1 --2- 172.21.15.28:0/1089693652 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f7f47debc0 0x55f7f47e0fe0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.704+0000 7f4c5e578740 1 --2- 172.21.15.28:0/1089693652 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7f4767850 0x55f7f476ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.704+0000 7f4c5e578740 1 --2- 172.21.15.28:0/1089693652 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7f475c860 0x55f7f476ff80 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.704+0000 7f4c3ea00640 0 client.26143 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:50:06.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.704+0000 7f4c5e578740 1 --2- 172.21.15.28:0/1089693652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7f4758b80 0x55f7f476fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.704+0000 7f4c5e578740 1 -- 172.21.15.28:0/1089693652 >> 172.21.15.28:0/1089693652 conn(0x55f7f4763440 msgr2=0x55f7f4765030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:06.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.704+0000 7f4c5e578740 1 -- 172.21.15.28:0/1089693652 shutdown_connections 2024-04-23T14:50:06.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.704+0000 7f4c5e578740 1 -- 172.21.15.28:0/1089693652 wait complete. 2024-04-23T14:50:06.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.704+0000 7f4c5e578740 20 client.26143 trim_cache size 0 max 16384 2024-04-23T14:50:06.710 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:50:06.710 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-23T14:50:06.710 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-23T14:50:06.710 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:50:06.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.705+0000 7f4c5b800640 1 -- 172.21.15.28:0/1223271363 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f4c380a3d20 msgr2=0x7f4c380a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:06.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.705+0000 7f4c5b800640 1 --2- 172.21.15.28:0/1223271363 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f4c380a3d20 0x7f4c380a61e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009fd0 tx=0x7f4c4c0096a0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.705+0000 7f4c5b800640 1 -- 172.21.15.28:0/1223271363 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c5406a680 msgr2=0x7f4c541a0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:06.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.705+0000 7f4c5b800640 1 --2- 172.21.15.28:0/1223271363 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c5406a680 0x7f4c541a0790 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4c44004770 tx=0x7f4c44003c10 comp rx=0 tx=0).stop 2024-04-23T14:50:06.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.705+0000 7f4c5b800640 1 -- 172.21.15.28:0/1223271363 shutdown_connections 2024-04-23T14:50:06.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.705+0000 7f4c5b800640 1 --2- 172.21.15.28:0/1223271363 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f4c380a3d20 0x7f4c380a61e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.705+0000 7f4c5b800640 1 --2- 172.21.15.28:0/1223271363 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c5410ea10 0x7f4c5419a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.705+0000 7f4c5b800640 1 --2- 172.21.15.28:0/1223271363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5406afa0 0x7f4c541a0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.705+0000 7f4c5b800640 1 --2- 172.21.15.28:0/1223271363 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c5406a680 0x7f4c541a0790 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.705+0000 7f4c5b800640 1 -- 172.21.15.28:0/1223271363 >> 172.21.15.28:0/1223271363 conn(0x7f4c54077f60 msgr2=0x7f4c5410ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:06.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.705+0000 7f4c5b800640 1 -- 172.21.15.28:0/1223271363 shutdown_connections 2024-04-23T14:50:06.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.705+0000 7f4c5b800640 1 -- 172.21.15.28:0/1223271363 wait complete. 2024-04-23T14:50:06.724 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-23T14:50:06.724 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:50:06.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.938+0000 7f46afe00640 1 -- 172.21.15.28:0/2700978767 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46b0069db0 msgr2=0x7f46b0106a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:06.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.938+0000 7f46afe00640 1 --2- 172.21.15.28:0/2700978767 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46b0069db0 0x7f46b0106a70 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f4698009900 tx=0x7f469802a3f0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.938+0000 7f46afe00640 1 -- 172.21.15.28:0/2700978767 shutdown_connections 2024-04-23T14:50:06.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.938+0000 7f46afe00640 1 --2- 172.21.15.28:0/2700978767 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46b0069db0 0x7f46b0106a70 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.938+0000 7f46afe00640 1 --2- 172.21.15.28:0/2700978767 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46b01071a0 0x7f46b010ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.938+0000 7f46afe00640 1 --2- 172.21.15.28:0/2700978767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46b0069400 0x7f46b00697e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.938+0000 7f46afe00640 1 -- 172.21.15.28:0/2700978767 >> 172.21.15.28:0/2700978767 conn(0x7f46b0076840 msgr2=0x7f46b0076c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:06.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.938+0000 7f46afe00640 1 -- 172.21.15.28:0/2700978767 shutdown_connections 2024-04-23T14:50:06.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.938+0000 7f46afe00640 1 -- 172.21.15.28:0/2700978767 wait complete. 2024-04-23T14:50:06.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.939+0000 7f46afe00640 1 Processor -- start 2024-04-23T14:50:06.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.940+0000 7f46afe00640 1 -- start start 2024-04-23T14:50:06.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.940+0000 7f46afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46b0069400 0x7f46b01a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:06.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.940+0000 7f46afe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46b01071a0 0x7f46b01a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:06.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.940+0000 7f46afe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46b019a760 0x7f46b019abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:06.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.940+0000 7f46afe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f46b01121c0 con 0x7f46b0069400 2024-04-23T14:50:06.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.940+0000 7f46afe00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f46b0112040 con 0x7f46b019a760 2024-04-23T14:50:06.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.940+0000 7f46afe00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f46b0112340 con 0x7f46b01071a0 2024-04-23T14:50:06.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.940+0000 7f46aea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46b0069400 0x7f46b01a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:06.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.940+0000 7f46af400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46b019a760 0x7f46b019abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:06.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.940+0000 7f46aea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46b0069400 0x7f46b01a06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48176/0 (socket says 172.21.15.28:48176) 2024-04-23T14:50:06.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.940+0000 7f46aea00640 1 -- 172.21.15.28:0/4133745734 learned_addr learned my addr 172.21.15.28:0/4133745734 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:06.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.940+0000 7f46a7e00640 1 --2- 172.21.15.28:0/4133745734 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46b01071a0 0x7f46b01a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:06.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.941+0000 7f46af400640 1 -- 172.21.15.28:0/4133745734 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46b01071a0 msgr2=0x7f46b01a0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:06.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.941+0000 7f46af400640 1 --2- 172.21.15.28:0/4133745734 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46b01071a0 0x7f46b01a0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.941+0000 7f46af400640 1 -- 172.21.15.28:0/4133745734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46b0069400 msgr2=0x7f46b01a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:06.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.941+0000 7f46af400640 1 --2- 172.21.15.28:0/4133745734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46b0069400 0x7f46b01a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:06.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.941+0000 7f46af400640 1 -- 172.21.15.28:0/4133745734 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46b019b360 con 0x7f46b019a760 2024-04-23T14:50:06.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.941+0000 7f46af400640 1 --2- 172.21.15.28:0/4133745734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46b019a760 0x7f46b019abe0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f46a8009fd0 tx=0x7f46a800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:06.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.941+0000 7f46a7400640 1 -- 172.21.15.28:0/4133745734 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a800ce20 con 0x7f46b019a760 2024-04-23T14:50:06.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.941+0000 7f46a7400640 1 -- 172.21.15.28:0/4133745734 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46a8005020 con 0x7f46b019a760 2024-04-23T14:50:06.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.941+0000 7f46afe00640 1 -- 172.21.15.28:0/4133745734 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46b01a7440 con 0x7f46b019a760 2024-04-23T14:50:06.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.941+0000 7f46a7400640 1 -- 172.21.15.28:0/4133745734 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a8010720 con 0x7f46b019a760 2024-04-23T14:50:06.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.942+0000 7f46afe00640 1 -- 172.21.15.28:0/4133745734 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46b01a77f0 con 0x7f46b019a760 2024-04-23T14:50:06.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.943+0000 7f46afe00640 1 -- 172.21.15.28:0/4133745734 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46b0103550 con 0x7f46b019a760 2024-04-23T14:50:06.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.945+0000 7f46a7400640 1 -- 172.21.15.28:0/4133745734 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f46a8005540 con 0x7f46b019a760 2024-04-23T14:50:06.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.946+0000 7f46a7400640 1 --2- 172.21.15.28:0/4133745734 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f46940a3d20 0x7f46940a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:06.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.946+0000 7f46a7400640 1 -- 172.21.15.28:0/4133745734 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f46a80d0ec0 con 0x7f46b019a760 2024-04-23T14:50:06.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.946+0000 7f46aea00640 1 --2- 172.21.15.28:0/4133745734 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f46940a3d20 0x7f46940a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:06.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.947+0000 7f46aea00640 1 --2- 172.21.15.28:0/4133745734 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f46940a3d20 0x7f46940a61e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f46a0009c90 tx=0x7f46a0005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:06.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:06.950+0000 7f46a7400640 1 -- 172.21.15.28:0/4133745734 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f46a809d510 con 0x7f46b019a760 2024-04-23T14:50:07.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:06 smithi105 ceph-mon[32555]: pgmap v185: 97 pgs: 97 active+clean; 282 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 2.5 MiB/s wr, 173 op/s 2024-04-23T14:50:07.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:06 smithi038 ceph-mon[32348]: pgmap v185: 97 pgs: 97 active+clean; 282 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 2.5 MiB/s wr, 173 op/s 2024-04-23T14:50:07.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:06 smithi028 ceph-mon[27810]: pgmap v185: 97 pgs: 97 active+clean; 282 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 2.5 MiB/s wr, 173 op/s 2024-04-23T14:50:07.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.219+0000 7f46afe00640 1 -- 172.21.15.28:0/4133745734 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f46b0068d40 con 0x7f46b019a760 2024-04-23T14:50:07.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.220+0000 7f46a7400640 1 -- 172.21.15.28:0/4133745734 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7f46a809ce10 con 0x7f46b019a760 2024-04-23T14:50:07.221 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:50:07.221 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:50:07.222 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:50:07.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.223+0000 7f46afe00640 1 -- 172.21.15.28:0/4133745734 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f46940a3d20 msgr2=0x7f46940a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:07.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.223+0000 7f46afe00640 1 --2- 172.21.15.28:0/4133745734 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f46940a3d20 0x7f46940a61e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f46a0009c90 tx=0x7f46a0005f90 comp rx=0 tx=0).stop 2024-04-23T14:50:07.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.223+0000 7f46afe00640 1 -- 172.21.15.28:0/4133745734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46b019a760 msgr2=0x7f46b019abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:07.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.223+0000 7f46afe00640 1 --2- 172.21.15.28:0/4133745734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46b019a760 0x7f46b019abe0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f46a8009fd0 tx=0x7f46a800ef10 comp rx=0 tx=0).stop 2024-04-23T14:50:07.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.223+0000 7f46afe00640 1 -- 172.21.15.28:0/4133745734 shutdown_connections 2024-04-23T14:50:07.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.223+0000 7f46afe00640 1 --2- 172.21.15.28:0/4133745734 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f46940a3d20 0x7f46940a61e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.223+0000 7f46afe00640 1 --2- 172.21.15.28:0/4133745734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46b019a760 0x7f46b019abe0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.223+0000 7f46afe00640 1 --2- 172.21.15.28:0/4133745734 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46b01071a0 0x7f46b01a0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.223+0000 7f46afe00640 1 --2- 172.21.15.28:0/4133745734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46b0069400 0x7f46b01a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.223+0000 7f46afe00640 1 -- 172.21.15.28:0/4133745734 >> 172.21.15.28:0/4133745734 conn(0x7f46b0076840 msgr2=0x7f46b010d230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:07.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.223+0000 7f46afe00640 1 -- 172.21.15.28:0/4133745734 shutdown_connections 2024-04-23T14:50:07.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.223+0000 7f46afe00640 1 -- 172.21.15.28:0/4133745734 wait complete. 2024-04-23T14:50:07.245 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-23T14:50:07.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.467+0000 7fed7fe00640 1 -- 172.21.15.28:0/2774736523 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed80104cd0 msgr2=0x7fed80108d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:07.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.467+0000 7fed7fe00640 1 --2- 172.21.15.28:0/2774736523 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed80104cd0 0x7fed80108d10 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fed68009900 tx=0x7fed6802a3f0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.467+0000 7fed7fe00640 1 -- 172.21.15.28:0/2774736523 shutdown_connections 2024-04-23T14:50:07.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.467+0000 7fed7fe00640 1 --2- 172.21.15.28:0/2774736523 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed80104cd0 0x7fed80108d10 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.467+0000 7fed7fe00640 1 --2- 172.21.15.28:0/2774736523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed801093f0 0x7fed80110fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.467+0000 7fed7fe00640 1 --2- 172.21.15.28:0/2774736523 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed80104320 0x7fed80104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.467+0000 7fed7fe00640 1 -- 172.21.15.28:0/2774736523 >> 172.21.15.28:0/2774736523 conn(0x7fed800ffb40 msgr2=0x7fed80101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:07.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.467+0000 7fed7fe00640 1 -- 172.21.15.28:0/2774736523 shutdown_connections 2024-04-23T14:50:07.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.467+0000 7fed7fe00640 1 -- 172.21.15.28:0/2774736523 wait complete. 2024-04-23T14:50:07.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.468+0000 7fed7fe00640 1 Processor -- start 2024-04-23T14:50:07.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.468+0000 7fed7fe00640 1 -- start start 2024-04-23T14:50:07.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.469+0000 7fed7fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed80104320 0x7fed801a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:07.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.469+0000 7fed7fe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed801093f0 0x7fed801a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:07.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.469+0000 7fed7fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed8019a870 0x7fed8019ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:07.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.469+0000 7fed7fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fed801144d0 con 0x7fed8019a870 2024-04-23T14:50:07.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.469+0000 7fed7fe00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fed80114350 con 0x7fed80104320 2024-04-23T14:50:07.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.469+0000 7fed7fe00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fed80114650 con 0x7fed801093f0 2024-04-23T14:50:07.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.469+0000 7fed7ea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed80104320 0x7fed801a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:07.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.469+0000 7fed7ea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed80104320 0x7fed801a06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:33680/0 (socket says 172.21.15.28:33680) 2024-04-23T14:50:07.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.469+0000 7fed7ea00640 1 -- 172.21.15.28:0/2005984438 learned_addr learned my addr 172.21.15.28:0/2005984438 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:07.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.469+0000 7fed7f400640 1 --2- 172.21.15.28:0/2005984438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed8019a870 0x7fed8019ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:07.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.469+0000 7fed7e000640 1 --2- 172.21.15.28:0/2005984438 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed801093f0 0x7fed801a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:07.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.470+0000 7fed7ea00640 1 -- 172.21.15.28:0/2005984438 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed801093f0 msgr2=0x7fed801a0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:07.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.470+0000 7fed7ea00640 1 --2- 172.21.15.28:0/2005984438 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed801093f0 0x7fed801a0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.470+0000 7fed7ea00640 1 -- 172.21.15.28:0/2005984438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed8019a870 msgr2=0x7fed8019ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:07.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.470+0000 7fed7ea00640 1 --2- 172.21.15.28:0/2005984438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed8019a870 0x7fed8019ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.470+0000 7fed7ea00640 1 -- 172.21.15.28:0/2005984438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed8019b390 con 0x7fed80104320 2024-04-23T14:50:07.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.470+0000 7fed7ea00640 1 --2- 172.21.15.28:0/2005984438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed80104320 0x7fed801a06c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fed7000e9f0 tx=0x7fed7000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:07.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.470+0000 7fed77400640 1 -- 172.21.15.28:0/2005984438 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed7000cd80 con 0x7fed80104320 2024-04-23T14:50:07.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.470+0000 7fed77400640 1 -- 172.21.15.28:0/2005984438 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed70005500 con 0x7fed80104320 2024-04-23T14:50:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.470+0000 7fed7fe00640 1 -- 172.21.15.28:0/2005984438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed8019b680 con 0x7fed80104320 2024-04-23T14:50:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.470+0000 7fed77400640 1 -- 172.21.15.28:0/2005984438 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed70010690 con 0x7fed80104320 2024-04-23T14:50:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.470+0000 7fed7fe00640 1 -- 172.21.15.28:0/2005984438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed801a78d0 con 0x7fed80104320 2024-04-23T14:50:07.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.474+0000 7fed77400640 1 -- 172.21.15.28:0/2005984438 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fed700046c0 con 0x7fed80104320 2024-04-23T14:50:07.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.474+0000 7fed885bd740 1 Processor -- start 2024-04-23T14:50:07.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.474+0000 7fed885bd740 1 -- start start 2024-04-23T14:50:07.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.475+0000 7fed885bd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7483f6b40 0x55f7483f6f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:07.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.475+0000 7fed885bd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7483f7460 0x55f7484015f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:07.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.475+0000 7fed885bd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f748401b30 0x55f748403f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:07.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.475+0000 7fed885bd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f74835c1d0 con 0x55f7483f7460 2024-04-23T14:50:07.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.475+0000 7fed885bd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f748302450 con 0x55f7483f6b40 2024-04-23T14:50:07.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.475+0000 7fed885bd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f748358190 con 0x55f748401b30 2024-04-23T14:50:07.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.475+0000 7fed7f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7483f6b40 0x55f7483f6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:07.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.475+0000 7fed7f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7483f6b40 0x55f7483f6f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:33688/0 (socket says 172.21.15.28:33688) 2024-04-23T14:50:07.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.475+0000 7fed7f400640 1 -- 172.21.15.28:0/547735385 learned_addr learned my addr 172.21.15.28:0/547735385 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:07.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.475+0000 7fed7ea00640 1 --2- 172.21.15.28:0/547735385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7483f7460 0x55f7484015f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:07.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.476+0000 7fed7f400640 1 -- 172.21.15.28:0/547735385 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f748401b30 msgr2=0x55f748403f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:07.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.476+0000 7fed7f400640 1 --2- 172.21.15.28:0/547735385 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f748401b30 0x55f748403f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.476+0000 7fed7f400640 1 -- 172.21.15.28:0/547735385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7483f7460 msgr2=0x55f7484015f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:07.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.476+0000 7fed7f400640 1 --2- 172.21.15.28:0/547735385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7483f7460 0x55f7484015f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.476+0000 7fed7f400640 1 -- 172.21.15.28:0/547735385 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7483bf310 con 0x55f7483f6b40 2024-04-23T14:50:07.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.476+0000 7fed7f400640 1 --2- 172.21.15.28:0/547735385 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7483f6b40 0x55f7483f6f20 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fed78009f50 tx=0x7fed780118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d4f280dba6286d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:07.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.476+0000 7fed77400640 1 --2- 172.21.15.28:0/2005984438 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fed640a3da0 0x7fed640a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:07.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.476+0000 7fed6fe00640 1 -- 172.21.15.28:0/547735385 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed78004b20 con 0x55f7483f6b40 2024-04-23T14:50:07.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.477+0000 7fed77400640 1 -- 172.21.15.28:0/2005984438 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fed700d0b80 con 0x7fed80104320 2024-04-23T14:50:07.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.477+0000 7fed6fe00640 1 -- 172.21.15.28:0/547735385 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed78004cc0 con 0x55f7483f6b40 2024-04-23T14:50:07.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.477+0000 7fed6fe00640 1 -- 172.21.15.28:0/547735385 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed78005020 con 0x55f7483f6b40 2024-04-23T14:50:07.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.477+0000 7fed7ea00640 1 --2- 172.21.15.28:0/2005984438 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fed640a3da0 0x7fed640a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:07.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.477+0000 7fed885bd740 1 -- 172.21.15.28:0/547735385 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7483f6b40 msgr2=0x55f7483f6f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:07.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.477+0000 7fed7ea00640 1 --2- 172.21.15.28:0/2005984438 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fed640a3da0 0x7fed640a6260 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fed700ce390 tx=0x7fed70004a40 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:07.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.477+0000 7fed885bd740 1 --2- 172.21.15.28:0/547735385 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7483f6b40 0x55f7483f6f20 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fed78009f50 tx=0x7fed780118d0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.478+0000 7fed885bd740 1 -- 172.21.15.28:0/547735385 shutdown_connections 2024-04-23T14:50:07.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.478+0000 7fed885bd740 1 --2- 172.21.15.28:0/547735385 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f748401b30 0x55f748403f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.478+0000 7fed885bd740 1 --2- 172.21.15.28:0/547735385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7483f7460 0x55f7484015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.478+0000 7fed885bd740 1 --2- 172.21.15.28:0/547735385 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7483f6b40 0x55f7483f6f20 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.478+0000 7fed885bd740 1 -- 172.21.15.28:0/547735385 >> 172.21.15.28:0/547735385 conn(0x55f7483f2e60 msgr2=0x55f7483f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:07.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.478+0000 7fed885bd740 1 -- 172.21.15.28:0/547735385 shutdown_connections 2024-04-23T14:50:07.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.478+0000 7fed885bd740 1 -- 172.21.15.28:0/547735385 wait complete. 2024-04-23T14:50:07.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.478+0000 7fed885bd740 1 -- start start 2024-04-23T14:50:07.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.478+0000 7fed885bd740 1 Processor -- start 2024-04-23T14:50:07.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.479+0000 7fed885bd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7483f6b40 0x55f7484015f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:07.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.479+0000 7fed885bd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7483f7460 0x55f7483fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:07.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.479+0000 7fed885bd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f748401b30 0x55f7483ff020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:07.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.479+0000 7fed885bd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f7484049c0 con 0x55f748401b30 2024-04-23T14:50:07.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.479+0000 7fed885bd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f748315810 con 0x55f7483f6b40 2024-04-23T14:50:07.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.479+0000 7fed885bd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f748404b40 con 0x55f7483f7460 2024-04-23T14:50:07.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.479+0000 7fed885bd740 10 client.0 _ll_register_callbacks cb 0x55f7482e3140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:50:07.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.479+0000 7fed7f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7483f6b40 0x55f7484015f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:07.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.479+0000 7fed7f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7483f6b40 0x55f7484015f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:33700/0 (socket says 172.21.15.28:33700) 2024-04-23T14:50:07.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.479+0000 7fed7f400640 1 -- 172.21.15.28:0/2754188502 learned_addr learned my addr 172.21.15.28:0/2754188502 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:07.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.479+0000 7fed7e000640 1 --2- 172.21.15.28:0/2754188502 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7483f7460 0x55f7483fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:07.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.480+0000 7fed7f400640 1 --2- 172.21.15.28:0/2754188502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f748401b30 0x55f7483ff020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:07.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.480+0000 7fed7e000640 1 -- 172.21.15.28:0/2754188502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7483f6b40 msgr2=0x55f7484015f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:07.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.480+0000 7fed7e000640 1 --2- 172.21.15.28:0/2754188502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7483f6b40 0x55f7484015f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.480+0000 7fed7e000640 1 -- 172.21.15.28:0/2754188502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f748401b30 msgr2=0x55f7483ff020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:07.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.480+0000 7fed7e000640 1 --2- 172.21.15.28:0/2754188502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f748401b30 0x55f7483ff020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.480+0000 7fed7e000640 1 -- 172.21.15.28:0/2754188502 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7483263f0 con 0x55f7483f7460 2024-04-23T14:50:07.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.480+0000 7fed7f400640 1 --2- 172.21.15.28:0/2754188502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f748401b30 0x55f7483ff020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:50:07.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.480+0000 7fed7e000640 1 --2- 172.21.15.28:0/2754188502 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7483f7460 0x55f7483fe8e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fed6802a910 tx=0x7fed68004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8feacb12bff38b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:07.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.480+0000 7fed6cc00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:50:07.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.481+0000 7fed6cc00640 1 -- 172.21.15.28:0/2754188502 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed68035040 con 0x55f7483f7460 2024-04-23T14:50:07.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.481+0000 7fed6cc00640 1 -- 172.21.15.28:0/2754188502 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fed68005240 con 0x55f7483f7460 2024-04-23T14:50:07.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.481+0000 7fed885bd740 1 -- 172.21.15.28:0/2754188502 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed78004cc0 con 0x55f7483f7460 2024-04-23T14:50:07.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.481+0000 7fed6cc00640 1 -- 172.21.15.28:0/2754188502 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fed68005440 con 0x55f7483f7460 2024-04-23T14:50:07.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.481+0000 7fed885bd740 10 client.26158 fetch_fsmap learned FSMap version 20 2024-04-23T14:50:07.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.481+0000 7fed885bd740 1 -- 172.21.15.28:0/2754188502 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55f74806e3f0 con 0x55f7483f7460 2024-04-23T14:50:07.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.481+0000 7fed6cc00640 1 -- 172.21.15.28:0/2754188502 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed78004cc0 con 0x55f7483f7460 2024-04-23T14:50:07.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.481+0000 7fed6cc00640 1 -- 172.21.15.28:0/2754188502 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fed68003850 con 0x55f7483f7460 2024-04-23T14:50:07.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.482+0000 7fed885bd740 10 client.26158 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:50:07.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.482+0000 7fed885bd740 10 client.26158 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:50:07.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.482+0000 7fed885bd740 20 client.26158 populate_metadata read hostname 'smithi028' 2024-04-23T14:50:07.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.482+0000 7fed885bd740 1 --2- 172.21.15.28:0/2754188502 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f748411cb0 0x55f7484140c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:07.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.482+0000 7fed885bd740 4 client.26158 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:50:07.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.482+0000 7fed885bd740 1 -- 172.21.15.28:0/2754188502 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f748369bf0 con 0x55f748411cb0 2024-04-23T14:50:07.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.483+0000 7fed7f400640 1 --2- 172.21.15.28:0/2754188502 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f748411cb0 0x55f7484140c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:07.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.483+0000 7fed7f400640 1 --2- 172.21.15.28:0/2754188502 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f748411cb0 0x55f7484140c0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53c63d2e289d95d3 server_cookie=ee8b6bdc4c82ca10 in_seq=0 out_seq=0 2024-04-23T14:50:07.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.483+0000 7fed6cc00640 10 client.26158 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:50:07.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.484+0000 7fed6cc00640 1 -- 172.21.15.28:0/2754188502 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f748369bf0 con 0x55f748411cb0 2024-04-23T14:50:07.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.484+0000 7fed6cc00640 10 client.26158 handle_command_reply: tid=0 2024-04-23T14:50:07.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.484+0000 7fed885bd740 1 client.26158 shutdown 2024-04-23T14:50:07.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.484+0000 7fed885bd740 1 -- 172.21.15.28:0/2754188502 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7483f7460 msgr2=0x55f7483fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:07.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.484+0000 7fed885bd740 1 --2- 172.21.15.28:0/2754188502 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7483f7460 0x55f7483fe8e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fed6802a910 tx=0x7fed68004270 comp rx=0 tx=0).stop 2024-04-23T14:50:07.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.484+0000 7fed885bd740 1 -- 172.21.15.28:0/2754188502 shutdown_connections 2024-04-23T14:50:07.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.484+0000 7fed885bd740 1 --2- 172.21.15.28:0/2754188502 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f748411cb0 0x55f7484140c0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.485+0000 7fed885bd740 1 --2- 172.21.15.28:0/2754188502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f748401b30 0x55f7483ff020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.485+0000 7fed885bd740 1 --2- 172.21.15.28:0/2754188502 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7483f7460 0x55f7483fe8e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.485+0000 7fed885bd740 1 --2- 172.21.15.28:0/2754188502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7483f6b40 0x55f7484015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.485+0000 7fed885bd740 1 -- 172.21.15.28:0/2754188502 >> 172.21.15.28:0/2754188502 conn(0x55f7483f2e60 msgr2=0x55f7483f8b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:07.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.485+0000 7fed6cc00640 0 client.26158 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:50:07.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.485+0000 7fed885bd740 1 -- 172.21.15.28:0/2754188502 shutdown_connections 2024-04-23T14:50:07.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.485+0000 7fed885bd740 1 -- 172.21.15.28:0/2754188502 wait complete. 2024-04-23T14:50:07.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.485+0000 7fed885bd740 20 client.26158 trim_cache size 0 max 16384 2024-04-23T14:50:07.494 INFO:tasks.workunit.client.0.smithi028.stdout: 1 489252 9.71 MB/sec execute 143 sec latency 34.731 ms 2024-04-23T14:50:07.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.503+0000 7fed885bd740 1 Processor -- start 2024-04-23T14:50:07.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.503+0000 7fed885bd740 1 -- start start 2024-04-23T14:50:07.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.503+0000 7fed885bd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7483f6b40 0x55f74840d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:07.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.503+0000 7fed885bd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7483f7460 0x55f748401120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:07.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.503+0000 7fed885bd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f748401b30 0x55f748419990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:07.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.503+0000 7fed885bd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fed801add90 con 0x55f7483f7460 2024-04-23T14:50:07.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.503+0000 7fed885bd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fed801b05f0 con 0x55f748401b30 2024-04-23T14:50:07.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.503+0000 7fed885bd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fed801ae3e0 con 0x55f7483f6b40 2024-04-23T14:50:07.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.503+0000 7fed7e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7483f7460 0x55f748401120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:07.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.503+0000 7fed7f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f748401b30 0x55f748419990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:07.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.503+0000 7fed7e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7483f7460 0x55f748401120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48208/0 (socket says 172.21.15.28:48208) 2024-04-23T14:50:07.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.504+0000 7fed7e000640 1 -- 172.21.15.28:0/2422162646 learned_addr learned my addr 172.21.15.28:0/2422162646 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:07.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.504+0000 7fed7f400640 1 --2- 172.21.15.28:0/2422162646 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7483f6b40 0x55f74840d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:07.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.504+0000 7fed7f400640 1 -- 172.21.15.28:0/2422162646 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7483f6b40 msgr2=0x55f74840d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:07.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.504+0000 7fed7f400640 1 --2- 172.21.15.28:0/2422162646 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7483f6b40 0x55f74840d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.504+0000 7fed7f400640 1 -- 172.21.15.28:0/2422162646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7483f7460 msgr2=0x55f748401120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:07.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.504+0000 7fed7f400640 1 --2- 172.21.15.28:0/2422162646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7483f7460 0x55f748401120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.504+0000 7fed7f400640 1 -- 172.21.15.28:0/2422162646 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7481387d0 con 0x55f748401b30 2024-04-23T14:50:07.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.504+0000 7fed7e000640 1 --2- 172.21.15.28:0/2422162646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7483f7460 0x55f748401120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:50:07.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.504+0000 7fed7f400640 1 --2- 172.21.15.28:0/2422162646 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f748401b30 0x55f748419990 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fed78009270 tx=0x7fed78005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8613159a05946196 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:07.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.505+0000 7fed6f400640 1 -- 172.21.15.28:0/2422162646 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed78004670 con 0x55f748401b30 2024-04-23T14:50:07.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.505+0000 7fed6f400640 1 -- 172.21.15.28:0/2422162646 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed78004810 con 0x55f748401b30 2024-04-23T14:50:07.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.505+0000 7fed6f400640 1 -- 172.21.15.28:0/2422162646 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed7802e200 con 0x55f748401b30 2024-04-23T14:50:07.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.506+0000 7fed885bd740 1 -- 172.21.15.28:0/2422162646 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f748401b30 msgr2=0x55f748419990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:07.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.506+0000 7fed885bd740 1 --2- 172.21.15.28:0/2422162646 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f748401b30 0x55f748419990 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fed78009270 tx=0x7fed78005fd0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.506+0000 7fed885bd740 1 -- 172.21.15.28:0/2422162646 shutdown_connections 2024-04-23T14:50:07.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.506+0000 7fed885bd740 1 --2- 172.21.15.28:0/2422162646 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f748401b30 0x55f748419990 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.506+0000 7fed885bd740 1 --2- 172.21.15.28:0/2422162646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7483f7460 0x55f748401120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.506+0000 7fed885bd740 1 --2- 172.21.15.28:0/2422162646 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7483f6b40 0x55f74840d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.506+0000 7fed885bd740 1 -- 172.21.15.28:0/2422162646 >> 172.21.15.28:0/2422162646 conn(0x55f7483f2e60 msgr2=0x55f7483f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:07.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.506+0000 7fed885bd740 1 -- 172.21.15.28:0/2422162646 shutdown_connections 2024-04-23T14:50:07.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.506+0000 7fed885bd740 1 -- 172.21.15.28:0/2422162646 wait complete. 2024-04-23T14:50:07.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.506+0000 7fed885bd740 1 -- start start 2024-04-23T14:50:07.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.507+0000 7fed885bd740 1 Processor -- start 2024-04-23T14:50:07.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.507+0000 7fed885bd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7483f6b40 0x55f748403000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:07.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.507+0000 7fed885bd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7483f7460 0x55f748403540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:07.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.507+0000 7fed885bd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f748401b30 0x55f748408450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:07.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.507+0000 7fed885bd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fed801aeab0 con 0x55f748401b30 2024-04-23T14:50:07.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.507+0000 7fed885bd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fed801aff20 con 0x55f7483f6b40 2024-04-23T14:50:07.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.507+0000 7fed885bd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fed801af180 con 0x55f7483f7460 2024-04-23T14:50:07.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.508+0000 7fed885bd740 10 client.0 _ll_register_callbacks cb 0x7fed70005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:50:07.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.508+0000 7fed7e000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7483f7460 0x55f748403540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:07.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.508+0000 7fed7e000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7483f7460 0x55f748403540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:49698/0 (socket says 172.21.15.28:49698) 2024-04-23T14:50:07.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.508+0000 7fed7e000640 1 -- 172.21.15.28:0/2112159668 learned_addr learned my addr 172.21.15.28:0/2112159668 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:07.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.508+0000 7fed7f400640 1 --2- 172.21.15.28:0/2112159668 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7483f6b40 0x55f748403000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:07.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.508+0000 7fed7f400640 1 --2- 172.21.15.28:0/2112159668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f748401b30 0x55f748408450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:07.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.508+0000 7fed7e000640 1 -- 172.21.15.28:0/2112159668 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7483f6b40 msgr2=0x55f748403000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:07.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.508+0000 7fed7e000640 1 --2- 172.21.15.28:0/2112159668 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7483f6b40 0x55f748403000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.508+0000 7fed7e000640 1 -- 172.21.15.28:0/2112159668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f748401b30 msgr2=0x55f748408450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:07.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.508+0000 7fed7e000640 1 --2- 172.21.15.28:0/2112159668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f748401b30 0x55f748408450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.508+0000 7fed7e000640 1 -- 172.21.15.28:0/2112159668 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f748408990 con 0x55f7483f7460 2024-04-23T14:50:07.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.508+0000 7fed7f400640 1 --2- 172.21.15.28:0/2112159668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f748401b30 0x55f748408450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:50:07.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.508+0000 7fed7e000640 1 --2- 172.21.15.28:0/2112159668 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7483f7460 0x55f748403540 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fed6802a910 tx=0x7fed6802a3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=acf79c46348dbcd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:07.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.508+0000 7fed6cc00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:50:07.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.509+0000 7fed6cc00640 1 -- 172.21.15.28:0/2112159668 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed68035070 con 0x55f7483f7460 2024-04-23T14:50:07.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.509+0000 7fed6cc00640 1 -- 172.21.15.28:0/2112159668 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fed680317c0 con 0x55f7483f7460 2024-04-23T14:50:07.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.509+0000 7fed885bd740 1 -- 172.21.15.28:0/2112159668 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed78004810 con 0x55f7483f7460 2024-04-23T14:50:07.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.509+0000 7fed6cc00640 1 -- 172.21.15.28:0/2112159668 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fed68053040 con 0x55f7483f7460 2024-04-23T14:50:07.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.509+0000 7fed885bd740 10 client.26164 fetch_fsmap learned FSMap version 20 2024-04-23T14:50:07.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.509+0000 7fed885bd740 1 -- 172.21.15.28:0/2112159668 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55f7484789f0 con 0x55f7483f7460 2024-04-23T14:50:07.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.509+0000 7fed6cc00640 1 -- 172.21.15.28:0/2112159668 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed78004810 con 0x55f7483f7460 2024-04-23T14:50:07.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.509+0000 7fed6cc00640 1 -- 172.21.15.28:0/2112159668 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fed6804d470 con 0x55f7483f7460 2024-04-23T14:50:07.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.509+0000 7fed885bd740 10 client.26164 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:50:07.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.509+0000 7fed885bd740 10 client.26164 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:50:07.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.509+0000 7fed885bd740 20 client.26164 populate_metadata read hostname 'smithi028' 2024-04-23T14:50:07.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.510+0000 7fed885bd740 1 --2- 172.21.15.28:0/2112159668 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f7484791f0 0x55f74847b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:07.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.510+0000 7fed885bd740 4 client.26164 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-23T14:50:07.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.510+0000 7fed885bd740 1 -- 172.21.15.28:0/2112159668 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f748363a60 con 0x55f7484791f0 2024-04-23T14:50:07.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.510+0000 7fed7f400640 1 --2- 172.21.15.28:0/2112159668 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f7484791f0 0x55f74847b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:07.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.510+0000 7fed7f400640 1 --2- 172.21.15.28:0/2112159668 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f7484791f0 0x55f74847b610 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2198590d2b9c0a84 server_cookie=737728ed5a651bd6 in_seq=0 out_seq=0 2024-04-23T14:50:07.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.510+0000 7fed6cc00640 10 client.26164 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:50:07.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.511+0000 7fed6cc00640 1 -- 172.21.15.28:0/2112159668 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f748363a60 con 0x55f7484791f0 2024-04-23T14:50:07.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.511+0000 7fed6cc00640 10 client.26164 handle_command_reply: tid=0 2024-04-23T14:50:07.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.511+0000 7fed885bd740 1 client.26164 shutdown 2024-04-23T14:50:07.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.511+0000 7fed885bd740 1 -- 172.21.15.28:0/2112159668 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7483f7460 msgr2=0x55f748403540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:07.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.511+0000 7fed885bd740 1 --2- 172.21.15.28:0/2112159668 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7483f7460 0x55f748403540 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fed6802a910 tx=0x7fed6802a3f0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.511+0000 7fed885bd740 1 -- 172.21.15.28:0/2112159668 shutdown_connections 2024-04-23T14:50:07.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.511+0000 7fed885bd740 1 --2- 172.21.15.28:0/2112159668 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f7484791f0 0x55f74847b610 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.511+0000 7fed885bd740 1 --2- 172.21.15.28:0/2112159668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f748401b30 0x55f748408450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.511+0000 7fed885bd740 1 --2- 172.21.15.28:0/2112159668 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f7483f7460 0x55f748403540 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.511+0000 7fed885bd740 1 --2- 172.21.15.28:0/2112159668 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7483f6b40 0x55f748403000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.511+0000 7fed885bd740 1 -- 172.21.15.28:0/2112159668 >> 172.21.15.28:0/2112159668 conn(0x55f7483f2e60 msgr2=0x55f7483f16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:07.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.511+0000 7fed6cc00640 0 client.26164 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:50:07.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.511+0000 7fed885bd740 1 -- 172.21.15.28:0/2112159668 shutdown_connections 2024-04-23T14:50:07.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.511+0000 7fed885bd740 1 -- 172.21.15.28:0/2112159668 wait complete. 2024-04-23T14:50:07.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.512+0000 7fed885bd740 20 client.26164 trim_cache size 0 max 16384 2024-04-23T14:50:07.518 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-23T14:50:07.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.513+0000 7fed7fe00640 1 -- 172.21.15.28:0/2005984438 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fed640a3da0 msgr2=0x7fed640a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:07.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.513+0000 7fed7fe00640 1 --2- 172.21.15.28:0/2005984438 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fed640a3da0 0x7fed640a6260 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fed700ce390 tx=0x7fed70004a40 comp rx=0 tx=0).stop 2024-04-23T14:50:07.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.513+0000 7fed7fe00640 1 -- 172.21.15.28:0/2005984438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed80104320 msgr2=0x7fed801a06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:07.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.513+0000 7fed7fe00640 1 --2- 172.21.15.28:0/2005984438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed80104320 0x7fed801a06c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fed7000e9f0 tx=0x7fed7000edf0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.513+0000 7fed7fe00640 1 -- 172.21.15.28:0/2005984438 shutdown_connections 2024-04-23T14:50:07.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.513+0000 7fed7fe00640 1 --2- 172.21.15.28:0/2005984438 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fed640a3da0 0x7fed640a6260 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.513+0000 7fed7fe00640 1 --2- 172.21.15.28:0/2005984438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed8019a870 0x7fed8019ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.513+0000 7fed7fe00640 1 --2- 172.21.15.28:0/2005984438 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed801093f0 0x7fed801a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.513+0000 7fed7fe00640 1 --2- 172.21.15.28:0/2005984438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed80104320 0x7fed801a06c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:07.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.513+0000 7fed7fe00640 1 -- 172.21.15.28:0/2005984438 >> 172.21.15.28:0/2005984438 conn(0x7fed800ffb40 msgr2=0x7fed80101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:07.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.513+0000 7fed7fe00640 1 -- 172.21.15.28:0/2005984438 shutdown_connections 2024-04-23T14:50:07.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:07.513+0000 7fed7fe00640 1 -- 172.21.15.28:0/2005984438 wait complete. 2024-04-23T14:50:07.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:07 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/4133745734' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:50:08.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:07 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/4133745734' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:50:08.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:07 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/4133745734' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:50:08.494 INFO:tasks.workunit.client.0.smithi028.stdout: 1 490764 9.72 MB/sec execute 144 sec latency 12.885 ms 2024-04-23T14:50:08.534 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-23T14:50:08.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:50:08.534 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-23T14:50:08.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.782+0000 7f7c06800640 1 -- 172.21.15.28:0/2211291131 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c00076530 msgr2=0x7f7c00076910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:08.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.782+0000 7f7c06800640 1 --2- 172.21.15.28:0/2211291131 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c00076530 0x7f7c00076910 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7bf0009930 tx=0x7f7bf002a420 comp rx=0 tx=0).stop 2024-04-23T14:50:08.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.782+0000 7f7c06800640 1 -- 172.21.15.28:0/2211291131 shutdown_connections 2024-04-23T14:50:08.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.782+0000 7f7c06800640 1 --2- 172.21.15.28:0/2211291131 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c00112d50 0x7f7c00115140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:08.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.782+0000 7f7c06800640 1 --2- 172.21.15.28:0/2211291131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c00076e50 0x7f7c00112810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:08.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.782+0000 7f7c06800640 1 --2- 172.21.15.28:0/2211291131 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c00076530 0x7f7c00076910 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:08.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.782+0000 7f7c06800640 1 -- 172.21.15.28:0/2211291131 >> 172.21.15.28:0/2211291131 conn(0x7f7c000ffb40 msgr2=0x7f7c00101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:08.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.782+0000 7f7c06800640 1 -- 172.21.15.28:0/2211291131 shutdown_connections 2024-04-23T14:50:08.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.782+0000 7f7c06800640 1 -- 172.21.15.28:0/2211291131 wait complete. 2024-04-23T14:50:08.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.783+0000 7f7c06800640 1 Processor -- start 2024-04-23T14:50:08.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.783+0000 7f7c06800640 1 -- start start 2024-04-23T14:50:08.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.784+0000 7f7c06800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c00076530 0x7f7c0019ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:08.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.784+0000 7f7c06800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c00076e50 0x7f7c0019f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:08.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.784+0000 7f7c06800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c00112d50 0x7f7c001a36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:08.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.784+0000 7f7c06800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7c00118910 con 0x7f7c00076530 2024-04-23T14:50:08.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.784+0000 7f7c06800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7c00118790 con 0x7f7c00112d50 2024-04-23T14:50:08.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.784+0000 7f7c06800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7c00118a90 con 0x7f7c00076e50 2024-04-23T14:50:08.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.784+0000 7f7c05400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c00076530 0x7f7c0019ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:08.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.784+0000 7f7c05e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c00112d50 0x7f7c001a36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:08.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.784+0000 7f7c05e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c00112d50 0x7f7c001a36c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:46970/0 (socket says 172.21.15.28:46970) 2024-04-23T14:50:08.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.784+0000 7f7c05400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c00076530 0x7f7c0019ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55374/0 (socket says 172.21.15.28:55374) 2024-04-23T14:50:08.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.784+0000 7f7c05e00640 1 -- 172.21.15.28:0/2661716782 learned_addr learned my addr 172.21.15.28:0/2661716782 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:08.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.784+0000 7f7c04a00640 1 --2- 172.21.15.28:0/2661716782 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c00076e50 0x7f7c0019f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:08.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.785+0000 7f7c05e00640 1 -- 172.21.15.28:0/2661716782 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c00076e50 msgr2=0x7f7c0019f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:08.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.785+0000 7f7c05e00640 1 --2- 172.21.15.28:0/2661716782 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c00076e50 0x7f7c0019f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:08.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.785+0000 7f7c05e00640 1 -- 172.21.15.28:0/2661716782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c00076530 msgr2=0x7f7c0019ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:08.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.785+0000 7f7c05e00640 1 --2- 172.21.15.28:0/2661716782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c00076530 0x7f7c0019ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:08.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.785+0000 7f7c05e00640 1 -- 172.21.15.28:0/2661716782 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c001a3df0 con 0x7f7c00112d50 2024-04-23T14:50:08.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.785+0000 7f7c05400640 1 --2- 172.21.15.28:0/2661716782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c00076530 0x7f7c0019ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:50:08.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.785+0000 7f7c05e00640 1 --2- 172.21.15.28:0/2661716782 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c00112d50 0x7f7c001a36c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7bf800eb40 tx=0x7f7bf800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:08.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.785+0000 7f7bfe000640 1 -- 172.21.15.28:0/2661716782 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf8004b20 con 0x7f7c00112d50 2024-04-23T14:50:08.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.785+0000 7f7bfe000640 1 -- 172.21.15.28:0/2661716782 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bf8004cc0 con 0x7f7c00112d50 2024-04-23T14:50:08.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.785+0000 7f7c06800640 1 -- 172.21.15.28:0/2661716782 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c001a40e0 con 0x7f7c00112d50 2024-04-23T14:50:08.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.786+0000 7f7c06800640 1 -- 172.21.15.28:0/2661716782 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c001ab970 con 0x7f7c00112d50 2024-04-23T14:50:08.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.786+0000 7f7bfe000640 1 -- 172.21.15.28:0/2661716782 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf8010850 con 0x7f7c00112d50 2024-04-23T14:50:08.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.789+0000 7f7bfe000640 1 -- 172.21.15.28:0/2661716782 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f7bf80109f0 con 0x7f7c00112d50 2024-04-23T14:50:08.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.790+0000 7f7c0ac43740 1 Processor -- start 2024-04-23T14:50:08.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.790+0000 7f7c0ac43740 1 -- start start 2024-04-23T14:50:08.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.791+0000 7f7c0ac43740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590278c0270 0x5590278c0650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:08.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.791+0000 7f7c0ac43740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590278c0b90 0x5590278cad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:08.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.791+0000 7f7bfe000640 1 --2- 172.21.15.28:0/2661716782 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7be40a3e20 0x7f7be40a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:08.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.791+0000 7f7c0ac43740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590278cb260 0x5590278cd650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:08.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.791+0000 7f7c0ac43740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559027825e70 con 0x5590278c0b90 2024-04-23T14:50:08.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.791+0000 7f7c0ac43740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5590277cc380 con 0x5590278cb260 2024-04-23T14:50:08.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.791+0000 7f7c0ac43740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559027821e30 con 0x5590278c0270 2024-04-23T14:50:08.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.792+0000 7f7bfe000640 1 -- 172.21.15.28:0/2661716782 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f7bf80d0da0 con 0x7f7c00112d50 2024-04-23T14:50:08.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.792+0000 7f7c05e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590278cb260 0x5590278cd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:08.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.792+0000 7f7c04a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590278c0270 0x5590278c0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:08.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.792+0000 7f7c05e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590278cb260 0x5590278cd650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:46972/0 (socket says 172.21.15.28:46972) 2024-04-23T14:50:08.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.792+0000 7f7c05e00640 1 -- 172.21.15.28:0/4090984449 learned_addr learned my addr 172.21.15.28:0/4090984449 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:08.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.792+0000 7f7c04a00640 1 --2- 172.21.15.28:0/2661716782 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7be40a3e20 0x7f7be40a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:08.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.792+0000 7f7c05e00640 1 -- 172.21.15.28:0/4090984449 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590278c0270 msgr2=0x5590278c0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:08.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.792+0000 7f7c05e00640 1 --2- 172.21.15.28:0/4090984449 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590278c0270 0x5590278c0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:08.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.792+0000 7f7c05e00640 1 -- 172.21.15.28:0/4090984449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590278c0b90 msgr2=0x5590278cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:08.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.792+0000 7f7c05e00640 1 --2- 172.21.15.28:0/4090984449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590278c0b90 0x5590278cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:08.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.792+0000 7f7c05e00640 1 -- 172.21.15.28:0/4090984449 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590276007e0 con 0x5590278cb260 2024-04-23T14:50:08.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.792+0000 7f7c04a00640 1 --2- 172.21.15.28:0/4090984449 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590278c0270 0x5590278c0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:50:08.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.793+0000 7f7c05e00640 1 --2- 172.21.15.28:0/4090984449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590278cb260 0x5590278cd650 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f7bf80996c0 tx=0x7f7bf80afdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ab8b4cdbdeb1134 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:08.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.793+0000 7f7bf6600640 1 -- 172.21.15.28:0/4090984449 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf809ccf0 con 0x5590278cb260 2024-04-23T14:50:08.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.793+0000 7f7bf6600640 1 -- 172.21.15.28:0/4090984449 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bf80a10a0 con 0x5590278cb260 2024-04-23T14:50:08.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.793+0000 7f7bf6600640 1 -- 172.21.15.28:0/4090984449 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf80b60d0 con 0x5590278cb260 2024-04-23T14:50:08.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.794+0000 7f7c0ac43740 1 -- 172.21.15.28:0/4090984449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590278cb260 msgr2=0x5590278cd650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:08.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.794+0000 7f7c0ac43740 1 --2- 172.21.15.28:0/4090984449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590278cb260 0x5590278cd650 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f7bf80996c0 tx=0x7f7bf80afdb0 comp rx=0 tx=0).stop 2024-04-23T14:50:08.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.794+0000 7f7c04a00640 1 --2- 172.21.15.28:0/2661716782 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7be40a3e20 0x7f7be40a62e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7bec0058c0 tx=0x7f7bec005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:08.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.794+0000 7f7c0ac43740 1 -- 172.21.15.28:0/4090984449 shutdown_connections 2024-04-23T14:50:08.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.794+0000 7f7c0ac43740 1 --2- 172.21.15.28:0/4090984449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590278cb260 0x5590278cd650 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:08.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.794+0000 7f7c0ac43740 1 --2- 172.21.15.28:0/4090984449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590278c0b90 0x5590278cad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:08.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.794+0000 7f7c0ac43740 1 --2- 172.21.15.28:0/4090984449 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590278c0270 0x5590278c0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:08.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.794+0000 7f7c0ac43740 1 -- 172.21.15.28:0/4090984449 >> 172.21.15.28:0/4090984449 conn(0x5590278bc590 msgr2=0x5590278bc970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:08.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.795+0000 7f7c0ac43740 1 -- 172.21.15.28:0/4090984449 shutdown_connections 2024-04-23T14:50:08.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.795+0000 7f7c0ac43740 1 -- 172.21.15.28:0/4090984449 wait complete. 2024-04-23T14:50:08.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.795+0000 7f7c0ac43740 1 -- start start 2024-04-23T14:50:08.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.796+0000 7f7c0ac43740 1 Processor -- start 2024-04-23T14:50:08.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.796+0000 7f7c0ac43740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590278c0270 0x5590278c81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:08.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.796+0000 7f7c0ac43740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590278c0b90 0x5590278c8730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:08.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.796+0000 7f7c0ac43740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590278cb260 0x5590278c8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:08.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.796+0000 7f7c0ac43740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5590278cdff0 con 0x5590278c0b90 2024-04-23T14:50:08.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.796+0000 7f7c0ac43740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5590277df660 con 0x5590278c0270 2024-04-23T14:50:08.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.796+0000 7f7c0ac43740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5590278ce170 con 0x5590278cb260 2024-04-23T14:50:08.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.796+0000 7f7c0ac43740 10 client.0 _ll_register_callbacks cb 0x5590277cdaa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:50:08.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.796+0000 7f7c05400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590278c0270 0x5590278c81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:08.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.796+0000 7f7c05400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590278c0270 0x5590278c81f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:46988/0 (socket says 172.21.15.28:46988) 2024-04-23T14:50:08.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.796+0000 7f7c05400640 1 -- 172.21.15.28:0/1270945953 learned_addr learned my addr 172.21.15.28:0/1270945953 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:08.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.796+0000 7f7c04a00640 1 --2- 172.21.15.28:0/1270945953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590278c0b90 0x5590278c8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:08.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.797+0000 7f7c05e00640 1 --2- 172.21.15.28:0/1270945953 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590278cb260 0x5590278c8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:08.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.797+0000 7f7c05400640 1 -- 172.21.15.28:0/1270945953 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590278cb260 msgr2=0x5590278c8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:08.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.797+0000 7f7c05400640 1 --2- 172.21.15.28:0/1270945953 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590278cb260 0x5590278c8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:08.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.797+0000 7f7c05400640 1 -- 172.21.15.28:0/1270945953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590278c0b90 msgr2=0x5590278c8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:08.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.797+0000 7f7c05400640 1 --2- 172.21.15.28:0/1270945953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590278c0b90 0x5590278c8730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:08.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.797+0000 7f7c05400640 1 -- 172.21.15.28:0/1270945953 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559027660f70 con 0x5590278c0270 2024-04-23T14:50:08.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.797+0000 7f7c05e00640 1 --2- 172.21.15.28:0/1270945953 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590278cb260 0x5590278c8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:50:08.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.797+0000 7f7c05400640 1 --2- 172.21.15.28:0/1270945953 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590278c0270 0x5590278c81f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f7bf0009930 tx=0x7f7bf0005bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=827256beb772dfea server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:08.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.797+0000 7f7beaa00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:50:08.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.798+0000 7f7beaa00640 1 -- 172.21.15.28:0/1270945953 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf0035040 con 0x5590278c0270 2024-04-23T14:50:08.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.798+0000 7f7beaa00640 1 -- 172.21.15.28:0/1270945953 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f7bf0005590 con 0x5590278c0270 2024-04-23T14:50:08.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.798+0000 7f7c0ac43740 1 -- 172.21.15.28:0/1270945953 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bf80a10a0 con 0x5590278c0270 2024-04-23T14:50:08.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.799+0000 7f7beaa00640 1 -- 172.21.15.28:0/1270945953 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f7bf0004760 con 0x5590278c0270 2024-04-23T14:50:08.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.799+0000 7f7c0ac43740 10 client.26192 fetch_fsmap learned FSMap version 20 2024-04-23T14:50:08.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.799+0000 7f7c0ac43740 1 -- 172.21.15.28:0/1270945953 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5590274f8170 con 0x5590278c0270 2024-04-23T14:50:08.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.799+0000 7f7beaa00640 1 -- 172.21.15.28:0/1270945953 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bf80a10a0 con 0x5590278c0270 2024-04-23T14:50:08.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.799+0000 7f7beaa00640 1 -- 172.21.15.28:0/1270945953 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f7bf00315b0 con 0x5590278c0270 2024-04-23T14:50:08.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.799+0000 7f7c0ac43740 10 client.26192 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:50:08.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.799+0000 7f7c0ac43740 10 client.26192 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:50:08.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.799+0000 7f7c0ac43740 20 client.26192 populate_metadata read hostname 'smithi028' 2024-04-23T14:50:08.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.799+0000 7f7c0ac43740 1 --2- 172.21.15.28:0/1270945953 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5590278db5d0 0x5590278dda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:08.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.800+0000 7f7c0ac43740 4 client.26192 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:50:08.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.800+0000 7f7c0ac43740 1 -- 172.21.15.28:0/1270945953 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559027833890 con 0x5590278db5d0 2024-04-23T14:50:08.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.800+0000 7f7c04a00640 1 --2- 172.21.15.28:0/1270945953 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5590278db5d0 0x5590278dda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:08.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.800+0000 7f7c04a00640 1 --2- 172.21.15.28:0/1270945953 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5590278db5d0 0x5590278dda00 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9ed1cfb9461df1a server_cookie=643a80c468cf33b5 in_seq=0 out_seq=0 2024-04-23T14:50:08.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.800+0000 7f7beaa00640 10 client.26192 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:50:08.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.801+0000 7f7beaa00640 1 -- 172.21.15.28:0/1270945953 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559027833890 con 0x5590278db5d0 2024-04-23T14:50:08.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.801+0000 7f7beaa00640 10 client.26192 handle_command_reply: tid=0 2024-04-23T14:50:08.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.801+0000 7f7c0ac43740 1 client.26192 shutdown 2024-04-23T14:50:08.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.802+0000 7f7c0ac43740 1 -- 172.21.15.28:0/1270945953 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590278c0270 msgr2=0x5590278c81f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:08.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.802+0000 7f7c0ac43740 1 --2- 172.21.15.28:0/1270945953 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590278c0270 0x5590278c81f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f7bf0009930 tx=0x7f7bf0005bf0 comp rx=0 tx=0).stop 2024-04-23T14:50:08.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.802+0000 7f7c0ac43740 1 -- 172.21.15.28:0/1270945953 shutdown_connections 2024-04-23T14:50:08.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.802+0000 7f7c0ac43740 1 --2- 172.21.15.28:0/1270945953 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5590278db5d0 0x5590278dda00 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:08.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.802+0000 7f7beaa00640 0 client.26192 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:50:08.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.802+0000 7f7c0ac43740 1 --2- 172.21.15.28:0/1270945953 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590278cb260 0x5590278c8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:08.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.802+0000 7f7c0ac43740 1 --2- 172.21.15.28:0/1270945953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590278c0b90 0x5590278c8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:08.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.802+0000 7f7c0ac43740 1 --2- 172.21.15.28:0/1270945953 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590278c0270 0x5590278c81f0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:08.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.802+0000 7f7c0ac43740 1 -- 172.21.15.28:0/1270945953 >> 172.21.15.28:0/1270945953 conn(0x5590278bc590 msgr2=0x5590278c2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:08.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.802+0000 7f7c0ac43740 1 -- 172.21.15.28:0/1270945953 shutdown_connections 2024-04-23T14:50:08.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.802+0000 7f7c0ac43740 1 -- 172.21.15.28:0/1270945953 wait complete. 2024-04-23T14:50:08.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.802+0000 7f7c0ac43740 20 client.26192 trim_cache size 0 max 16384 2024-04-23T14:50:08.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.826+0000 7f7c0ac43740 1 Processor -- start 2024-04-23T14:50:08.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.826+0000 7f7c0ac43740 1 -- start start 2024-04-23T14:50:08.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.826+0000 7f7c0ac43740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590278c0270 0x5590278d6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:08.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.826+0000 7f7c0ac43740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590278c0b90 0x5590278c3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:08.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.826+0000 7f7c0ac43740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590278cb260 0x5590278c2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:08.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.826+0000 7f7c0ac43740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7be000c720 con 0x5590278cb260 2024-04-23T14:50:08.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.826+0000 7f7c0ac43740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7be000efa0 con 0x5590278c0b90 2024-04-23T14:50:08.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.826+0000 7f7c0ac43740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7be000cd90 con 0x5590278c0270 2024-04-23T14:50:08.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.827+0000 7f7c05400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590278c0270 0x5590278d6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:08.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.827+0000 7f7c04a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590278c0b90 0x5590278c3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:08.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.827+0000 7f7c05400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590278c0270 0x5590278d6ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:52334/0 (socket says 172.21.15.28:52334) 2024-04-23T14:50:08.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.827+0000 7f7c05400640 1 -- 172.21.15.28:0/1763610952 learned_addr learned my addr 172.21.15.28:0/1763610952 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:08.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.827+0000 7f7c05e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590278cb260 0x5590278c2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:08.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.827+0000 7f7c04a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590278c0b90 0x5590278c3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:47004/0 (socket says 172.21.15.28:47004) 2024-04-23T14:50:08.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.827+0000 7f7c05400640 1 -- 172.21.15.28:0/1763610952 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590278c0b90 msgr2=0x5590278c3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:08.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.827+0000 7f7c05400640 1 --2- 172.21.15.28:0/1763610952 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590278c0b90 0x5590278c3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:08.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.827+0000 7f7c05400640 1 -- 172.21.15.28:0/1763610952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590278cb260 msgr2=0x5590278c2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:08.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.827+0000 7f7c05400640 1 --2- 172.21.15.28:0/1763610952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590278cb260 0x5590278c2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:08.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.827+0000 7f7c05400640 1 -- 172.21.15.28:0/1763610952 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590278c6740 con 0x5590278c0270 2024-04-23T14:50:08.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.827+0000 7f7c04a00640 1 --2- 172.21.15.28:0/1763610952 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590278c0b90 0x5590278c3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:50:08.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.827+0000 7f7c05400640 1 --2- 172.21.15.28:0/1763610952 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590278c0270 0x5590278d6ee0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7bf002a940 tx=0x7f7bf0031510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a450fad7efc7c02d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:08.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.828+0000 7f7bf5c00640 1 -- 172.21.15.28:0/1763610952 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf0035070 con 0x5590278c0270 2024-04-23T14:50:08.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.828+0000 7f7bf5c00640 1 -- 172.21.15.28:0/1763610952 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bf0005850 con 0x5590278c0270 2024-04-23T14:50:08.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.828+0000 7f7bf5c00640 1 -- 172.21.15.28:0/1763610952 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf00055f0 con 0x5590278c0270 2024-04-23T14:50:08.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.828+0000 7f7c0ac43740 1 -- 172.21.15.28:0/1763610952 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590278c0270 msgr2=0x5590278d6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:08.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.828+0000 7f7c0ac43740 1 --2- 172.21.15.28:0/1763610952 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590278c0270 0x5590278d6ee0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7bf002a940 tx=0x7f7bf0031510 comp rx=0 tx=0).stop 2024-04-23T14:50:08.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.829+0000 7f7c0ac43740 1 -- 172.21.15.28:0/1763610952 shutdown_connections 2024-04-23T14:50:08.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.829+0000 7f7c0ac43740 1 --2- 172.21.15.28:0/1763610952 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590278c0b90 0x5590278c3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:08.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.829+0000 7f7c0ac43740 1 --2- 172.21.15.28:0/1763610952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590278cb260 0x5590278c2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:08.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.829+0000 7f7c0ac43740 1 --2- 172.21.15.28:0/1763610952 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590278c0270 0x5590278d6ee0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:08.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.829+0000 7f7c0ac43740 1 -- 172.21.15.28:0/1763610952 >> 172.21.15.28:0/1763610952 conn(0x5590278bc590 msgr2=0x5590278c1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:08.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.829+0000 7f7c0ac43740 1 -- 172.21.15.28:0/1763610952 shutdown_connections 2024-04-23T14:50:08.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.829+0000 7f7c0ac43740 1 -- 172.21.15.28:0/1763610952 wait complete. 2024-04-23T14:50:08.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.829+0000 7f7c0ac43740 1 -- start start 2024-04-23T14:50:08.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.829+0000 7f7c0ac43740 1 Processor -- start 2024-04-23T14:50:08.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.830+0000 7f7c0ac43740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590278c0270 0x5590278cf620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:08.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.830+0000 7f7c0ac43740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590278c0b90 0x5590278cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:08.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.830+0000 7f7c0ac43740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590278cb260 0x55902793ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:08.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.830+0000 7f7c0ac43740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7be000d460 con 0x5590278c0270 2024-04-23T14:50:08.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.830+0000 7f7c0ac43740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7be000e8d0 con 0x5590278c0b90 2024-04-23T14:50:08.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.830+0000 7f7c0ac43740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7be000db30 con 0x5590278cb260 2024-04-23T14:50:08.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.830+0000 7f7c0ac43740 10 client.0 _ll_register_callbacks cb 0x7f7bf80093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:50:08.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.830+0000 7f7c05400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590278c0270 0x5590278cf620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:08.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.830+0000 7f7c05400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590278c0270 0x5590278cf620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55414/0 (socket says 172.21.15.28:55414) 2024-04-23T14:50:08.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.830+0000 7f7c05400640 1 -- 172.21.15.28:0/2809218821 learned_addr learned my addr 172.21.15.28:0/2809218821 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:08.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.830+0000 7f7c04a00640 1 --2- 172.21.15.28:0/2809218821 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590278c0b90 0x5590278cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:08.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.830+0000 7f7c05e00640 1 --2- 172.21.15.28:0/2809218821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590278cb260 0x55902793ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:08.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.831+0000 7f7c05400640 1 -- 172.21.15.28:0/2809218821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590278cb260 msgr2=0x55902793ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:08.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.831+0000 7f7c05400640 1 --2- 172.21.15.28:0/2809218821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590278cb260 0x55902793ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:08.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.831+0000 7f7c05400640 1 -- 172.21.15.28:0/2809218821 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590278c0b90 msgr2=0x5590278cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:08.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.831+0000 7f7c05400640 1 --2- 172.21.15.28:0/2809218821 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590278c0b90 0x5590278cfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:08.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.831+0000 7f7c05400640 1 -- 172.21.15.28:0/2809218821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55902793b0d0 con 0x5590278c0270 2024-04-23T14:50:08.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.831+0000 7f7c04a00640 1 --2- 172.21.15.28:0/2809218821 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590278c0b90 0x5590278cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:50:08.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.831+0000 7f7c05400640 1 --2- 172.21.15.28:0/2809218821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590278c0270 0x5590278cf620 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7bf0031ab0 tx=0x7f7bf0031ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa05d761a2e9e246 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:08.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.831+0000 7f7beaa00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-23T14:50:08.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.832+0000 7f7beaa00640 1 -- 172.21.15.28:0/2809218821 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf0035070 con 0x5590278c0270 2024-04-23T14:50:08.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.832+0000 7f7beaa00640 1 -- 172.21.15.28:0/2809218821 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f7bf0004b00 con 0x5590278c0270 2024-04-23T14:50:08.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.832+0000 7f7c0ac43740 1 -- 172.21.15.28:0/2809218821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bf0005850 con 0x5590278c0270 2024-04-23T14:50:08.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.832+0000 7f7beaa00640 1 -- 172.21.15.28:0/2809218821 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f7bf0064050 con 0x5590278c0270 2024-04-23T14:50:08.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.832+0000 7f7c0ac43740 10 client.16032 fetch_fsmap learned FSMap version 20 2024-04-23T14:50:08.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.832+0000 7f7c0ac43740 1 -- 172.21.15.28:0/2809218821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5590279424a0 con 0x5590278c0270 2024-04-23T14:50:08.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.833+0000 7f7beaa00640 1 -- 172.21.15.28:0/2809218821 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bf0005850 con 0x5590278c0270 2024-04-23T14:50:08.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.833+0000 7f7beaa00640 1 -- 172.21.15.28:0/2809218821 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f7bf005f030 con 0x5590278c0270 2024-04-23T14:50:08.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.833+0000 7f7c0ac43740 10 client.16032 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:50:08.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.833+0000 7f7c0ac43740 10 client.16032 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:50:08.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.833+0000 7f7c0ac43740 20 client.16032 populate_metadata read hostname 'smithi028' 2024-04-23T14:50:08.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.833+0000 7f7c0ac43740 1 --2- 172.21.15.28:0/2809218821 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5590278db5d0 0x559027944ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:08.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.833+0000 7f7c0ac43740 4 client.16032 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-23T14:50:08.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.833+0000 7f7c0ac43740 1 -- 172.21.15.28:0/2809218821 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55902782d700 con 0x5590278db5d0 2024-04-23T14:50:08.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.833+0000 7f7c04a00640 1 --2- 172.21.15.28:0/2809218821 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5590278db5d0 0x559027944ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:08.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.834+0000 7f7c04a00640 1 --2- 172.21.15.28:0/2809218821 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5590278db5d0 0x559027944ac0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8806dff25697e906 server_cookie=8d27ccc92b4e696f in_seq=0 out_seq=0 2024-04-23T14:50:08.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.834+0000 7f7beaa00640 10 client.16032 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:50:08.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.835+0000 7f7beaa00640 1 -- 172.21.15.28:0/2809218821 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55902782d700 con 0x5590278db5d0 2024-04-23T14:50:08.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.835+0000 7f7beaa00640 10 client.16032 handle_command_reply: tid=0 2024-04-23T14:50:08.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.835+0000 7f7c0ac43740 1 client.16032 shutdown 2024-04-23T14:50:08.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.836+0000 7f7c0ac43740 1 -- 172.21.15.28:0/2809218821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590278c0270 msgr2=0x5590278cf620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:08.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.836+0000 7f7c0ac43740 1 --2- 172.21.15.28:0/2809218821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590278c0270 0x5590278cf620 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7bf0031ab0 tx=0x7f7bf0031ae0 comp rx=0 tx=0).stop 2024-04-23T14:50:08.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.836+0000 7f7c0ac43740 1 -- 172.21.15.28:0/2809218821 shutdown_connections 2024-04-23T14:50:08.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.836+0000 7f7c0ac43740 1 --2- 172.21.15.28:0/2809218821 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5590278db5d0 0x559027944ac0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:08.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.836+0000 7f7c0ac43740 1 --2- 172.21.15.28:0/2809218821 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590278c0b90 0x5590278cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:08.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.836+0000 7f7c0ac43740 1 --2- 172.21.15.28:0/2809218821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590278cb260 0x55902793ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:08.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.836+0000 7f7c0ac43740 1 --2- 172.21.15.28:0/2809218821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590278c0270 0x5590278cf620 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:08.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.836+0000 7f7beaa00640 0 client.16032 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:50:08.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.836+0000 7f7c0ac43740 1 -- 172.21.15.28:0/2809218821 >> 172.21.15.28:0/2809218821 conn(0x5590278bc590 msgr2=0x5590278c8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:08.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.836+0000 7f7c0ac43740 1 -- 172.21.15.28:0/2809218821 shutdown_connections 2024-04-23T14:50:08.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.836+0000 7f7c0ac43740 1 -- 172.21.15.28:0/2809218821 wait complete. 2024-04-23T14:50:08.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.836+0000 7f7c0ac43740 20 client.16032 trim_cache size 0 max 16384 2024-04-23T14:50:08.840 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:50:08.840 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-23T14:50:08.840 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "74724afe-8998-4b90-a254-f9bfcbd06ad8", 2024-04-23T14:50:08.840 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-23T14:50:08.840 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:50:08.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.837+0000 7f7c06800640 1 -- 172.21.15.28:0/2661716782 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7be40a3e20 msgr2=0x7f7be40a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:08.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.837+0000 7f7c06800640 1 --2- 172.21.15.28:0/2661716782 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7be40a3e20 0x7f7be40a62e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7bec0058c0 tx=0x7f7bec005f90 comp rx=0 tx=0).stop 2024-04-23T14:50:08.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.837+0000 7f7c06800640 1 -- 172.21.15.28:0/2661716782 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c00112d50 msgr2=0x7f7c001a36c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:08.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.837+0000 7f7c06800640 1 --2- 172.21.15.28:0/2661716782 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c00112d50 0x7f7c001a36c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7bf800eb40 tx=0x7f7bf800c2d0 comp rx=0 tx=0).stop 2024-04-23T14:50:08.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.838+0000 7f7c06800640 1 -- 172.21.15.28:0/2661716782 shutdown_connections 2024-04-23T14:50:08.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.838+0000 7f7c06800640 1 --2- 172.21.15.28:0/2661716782 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7be40a3e20 0x7f7be40a62e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:08.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.838+0000 7f7c06800640 1 --2- 172.21.15.28:0/2661716782 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c00112d50 0x7f7c001a36c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:08.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.838+0000 7f7c06800640 1 --2- 172.21.15.28:0/2661716782 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c00076e50 0x7f7c0019f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:08.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.838+0000 7f7c06800640 1 --2- 172.21.15.28:0/2661716782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c00076530 0x7f7c0019ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:08.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.838+0000 7f7c06800640 1 -- 172.21.15.28:0/2661716782 >> 172.21.15.28:0/2661716782 conn(0x7f7c000ffb40 msgr2=0x7f7c00114470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:08.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.838+0000 7f7c06800640 1 -- 172.21.15.28:0/2661716782 shutdown_connections 2024-04-23T14:50:08.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:08.838+0000 7f7c06800640 1 -- 172.21.15.28:0/2661716782 wait complete. 2024-04-23T14:50:08.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:08 smithi028 ceph-mon[27810]: pgmap v186: 97 pgs: 97 active+clean; 289 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.9 MiB/s wr, 247 op/s 2024-04-23T14:50:08.860 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:50:08.860 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:50:09.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:08 smithi105 ceph-mon[32555]: pgmap v186: 97 pgs: 97 active+clean; 289 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.9 MiB/s wr, 247 op/s 2024-04-23T14:50:09.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.063+0000 7f75c6400640 1 -- 172.21.15.28:0/3761470846 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f75c0074cf0 msgr2=0x7f75c010e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:09.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.063+0000 7f75c6400640 1 --2- 172.21.15.28:0/3761470846 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f75c0074cf0 0x7f75c010e200 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f75ac009900 tx=0x7f75ac02a480 comp rx=0 tx=0).stop 2024-04-23T14:50:09.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.063+0000 7f75c6400640 1 -- 172.21.15.28:0/3761470846 shutdown_connections 2024-04-23T14:50:09.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.063+0000 7f75c6400640 1 --2- 172.21.15.28:0/3761470846 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75c010e740 0x7f75c0110b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:09.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.063+0000 7f75c6400640 1 --2- 172.21.15.28:0/3761470846 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f75c0074cf0 0x7f75c010e200 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:09.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.063+0000 7f75c6400640 1 --2- 172.21.15.28:0/3761470846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75c00743d0 0x7f75c00747b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:09.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.063+0000 7f75c6400640 1 -- 172.21.15.28:0/3761470846 >> 172.21.15.28:0/3761470846 conn(0x7f75c00fd9d0 msgr2=0x7f75c00ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:09.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.063+0000 7f75c6400640 1 -- 172.21.15.28:0/3761470846 shutdown_connections 2024-04-23T14:50:09.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.063+0000 7f75c6400640 1 -- 172.21.15.28:0/3761470846 wait complete. 2024-04-23T14:50:09.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.064+0000 7f75c6400640 1 Processor -- start 2024-04-23T14:50:09.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.064+0000 7f75c6400640 1 -- start start 2024-04-23T14:50:09.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.064+0000 7f75c6400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75c00743d0 0x7f75c01a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:09.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.064+0000 7f75c6400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f75c0074cf0 0x7f75c01a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.064+0000 7f75c6400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75c010e740 0x7f75c019a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.064+0000 7f75c6400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f75c0114510 con 0x7f75c00743d0 2024-04-23T14:50:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.064+0000 7f75c6400640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f75c0114390 con 0x7f75c010e740 2024-04-23T14:50:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.064+0000 7f75c6400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f75c0114690 con 0x7f75c0074cf0 2024-04-23T14:50:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.065+0000 7f75c5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75c00743d0 0x7f75c01a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.065+0000 7f75c5a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75c010e740 0x7f75c019a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.065+0000 7f75c5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75c00743d0 0x7f75c01a0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55436/0 (socket says 172.21.15.28:55436) 2024-04-23T14:50:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.065+0000 7f75c5000640 1 -- 172.21.15.28:0/2164327583 learned_addr learned my addr 172.21.15.28:0/2164327583 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.065+0000 7f75bfe00640 1 --2- 172.21.15.28:0/2164327583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f75c0074cf0 0x7f75c01a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.065+0000 7f75c5a00640 1 -- 172.21.15.28:0/2164327583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f75c0074cf0 msgr2=0x7f75c01a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.065+0000 7f75c5a00640 1 --2- 172.21.15.28:0/2164327583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f75c0074cf0 0x7f75c01a0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.065+0000 7f75c5a00640 1 -- 172.21.15.28:0/2164327583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75c00743d0 msgr2=0x7f75c01a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.065+0000 7f75c5a00640 1 --2- 172.21.15.28:0/2164327583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75c00743d0 0x7f75c01a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.065+0000 7f75c5a00640 1 -- 172.21.15.28:0/2164327583 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75c019afc0 con 0x7f75c010e740 2024-04-23T14:50:09.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.065+0000 7f75c5000640 1 --2- 172.21.15.28:0/2164327583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75c00743d0 0x7f75c01a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:50:09.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.065+0000 7f75c5a00640 1 --2- 172.21.15.28:0/2164327583 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75c010e740 0x7f75c019a7f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f75c0075320 tx=0x7f75b800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:09.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.066+0000 7f75bd600640 1 -- 172.21.15.28:0/2164327583 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b800cdf0 con 0x7f75c010e740 2024-04-23T14:50:09.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.066+0000 7f75bd600640 1 -- 172.21.15.28:0/2164327583 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75b8005020 con 0x7f75c010e740 2024-04-23T14:50:09.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.066+0000 7f75bd600640 1 -- 172.21.15.28:0/2164327583 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b8010670 con 0x7f75c010e740 2024-04-23T14:50:09.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.066+0000 7f75c6400640 1 -- 172.21.15.28:0/2164327583 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75c019b2b0 con 0x7f75c010e740 2024-04-23T14:50:09.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.066+0000 7f75c6400640 1 -- 172.21.15.28:0/2164327583 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75c01a74d0 con 0x7f75c010e740 2024-04-23T14:50:09.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.069+0000 7f75bd600640 1 -- 172.21.15.28:0/2164327583 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f75b80054d0 con 0x7f75c010e740 2024-04-23T14:50:09.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.070+0000 7f75ca9b8740 1 Processor -- start 2024-04-23T14:50:09.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.070+0000 7f75ca9b8740 1 -- start start 2024-04-23T14:50:09.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.070+0000 7f75ca9b8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f55453a30 0x560f55453e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:09.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.070+0000 7f75ca9b8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f55454350 0x560f5545e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:09.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.070+0000 7f75ca9b8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f5545ea20 0x560f55460e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:09.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.070+0000 7f75ca9b8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560f553b91f0 con 0x560f5545ea20 2024-04-23T14:50:09.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.070+0000 7f75ca9b8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560f5535f350 con 0x560f55453a30 2024-04-23T14:50:09.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.070+0000 7f75ca9b8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560f553b51b0 con 0x560f55454350 2024-04-23T14:50:09.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.071+0000 7f75bd600640 1 --2- 172.21.15.28:0/2164327583 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f75a80a3de0 0x7f75a80a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:09.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.071+0000 7f75c5000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f55454350 0x560f5545e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:09.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.071+0000 7f75c5000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f55454350 0x560f5545e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:52366/0 (socket says 172.21.15.28:52366) 2024-04-23T14:50:09.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.071+0000 7f75c5000640 1 -- 172.21.15.28:0/3958133567 learned_addr learned my addr 172.21.15.28:0/3958133567 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:09.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.071+0000 7f75bfe00640 1 --2- 172.21.15.28:0/3958133567 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f55453a30 0x560f55453e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:09.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.071+0000 7f75bd600640 1 -- 172.21.15.28:0/2164327583 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f75b80d0db0 con 0x7f75c010e740 2024-04-23T14:50:09.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.071+0000 7f75bfe00640 1 --2- 172.21.15.28:0/3958133567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f5545ea20 0x560f55460e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:09.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.071+0000 7f75c5a00640 1 --2- 172.21.15.28:0/2164327583 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f75a80a3de0 0x7f75a80a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:09.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.071+0000 7f75c5000640 1 -- 172.21.15.28:0/3958133567 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f55453a30 msgr2=0x560f55453e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:09.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.071+0000 7f75c5000640 1 --2- 172.21.15.28:0/3958133567 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f55453a30 0x560f55453e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:09.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.071+0000 7f75c5000640 1 -- 172.21.15.28:0/3958133567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f5545ea20 msgr2=0x560f55460e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:09.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.071+0000 7f75c5000640 1 --2- 172.21.15.28:0/3958133567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f5545ea20 0x560f55460e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:09.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.071+0000 7f75c5000640 1 -- 172.21.15.28:0/3958133567 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f553b3f50 con 0x560f55454350 2024-04-23T14:50:09.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.072+0000 7f75c5000640 1 --2- 172.21.15.28:0/3958133567 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f55454350 0x560f5545e4e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f75b4011650 tx=0x7f75b4009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8fcb361a8da7fce server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:09.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.072+0000 7f75b2000640 1 -- 172.21.15.28:0/3958133567 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b401e070 con 0x560f55454350 2024-04-23T14:50:09.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.072+0000 7f75c5a00640 1 --2- 172.21.15.28:0/2164327583 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f75a80a3de0 0x7f75a80a62a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f75b8005fd0 tx=0x7f75b80058a0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.072+0000 7f75b2000640 1 -- 172.21.15.28:0/3958133567 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75b400fba0 con 0x560f55454350 2024-04-23T14:50:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.072+0000 7f75b2000640 1 -- 172.21.15.28:0/3958133567 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b4019cd0 con 0x560f55454350 2024-04-23T14:50:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.073+0000 7f75ca9b8740 1 -- 172.21.15.28:0/3958133567 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f55454350 msgr2=0x560f5545e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.073+0000 7f75ca9b8740 1 --2- 172.21.15.28:0/3958133567 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f55454350 0x560f5545e4e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f75b4011650 tx=0x7f75b4009f90 comp rx=0 tx=0).stop 2024-04-23T14:50:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.073+0000 7f75ca9b8740 1 -- 172.21.15.28:0/3958133567 shutdown_connections 2024-04-23T14:50:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.073+0000 7f75ca9b8740 1 --2- 172.21.15.28:0/3958133567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f5545ea20 0x560f55460e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.073+0000 7f75ca9b8740 1 --2- 172.21.15.28:0/3958133567 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f55454350 0x560f5545e4e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.073+0000 7f75ca9b8740 1 --2- 172.21.15.28:0/3958133567 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f55453a30 0x560f55453e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.073+0000 7f75ca9b8740 1 -- 172.21.15.28:0/3958133567 >> 172.21.15.28:0/3958133567 conn(0x560f5544fd50 msgr2=0x560f55450130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.074+0000 7f75ca9b8740 1 -- 172.21.15.28:0/3958133567 shutdown_connections 2024-04-23T14:50:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.074+0000 7f75ca9b8740 1 -- 172.21.15.28:0/3958133567 wait complete. 2024-04-23T14:50:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.074+0000 7f75ca9b8740 1 -- start start 2024-04-23T14:50:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.074+0000 7f75ca9b8740 1 Processor -- start 2024-04-23T14:50:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.075+0000 7f75ca9b8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f55453a30 0x560f5545b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.075+0000 7f75ca9b8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f55454350 0x560f5545bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.075+0000 7f75ca9b8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f5545ea20 0x560f5545c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.075+0000 7f75ca9b8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560f55461810 con 0x560f55453a30 2024-04-23T14:50:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.075+0000 7f75ca9b8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560f55372690 con 0x560f5545ea20 2024-04-23T14:50:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.075+0000 7f75ca9b8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560f55461990 con 0x560f55454350 2024-04-23T14:50:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.075+0000 7f75ca9b8740 10 client.0 _ll_register_callbacks cb 0x560f553ae650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:50:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.075+0000 7f75bfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f55453a30 0x560f5545b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.075+0000 7f75c5000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f55454350 0x560f5545bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.075+0000 7f75bfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f55453a30 0x560f5545b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55462/0 (socket says 172.21.15.28:55462) 2024-04-23T14:50:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.075+0000 7f75bfe00640 1 -- 172.21.15.28:0/3972697228 learned_addr learned my addr 172.21.15.28:0/3972697228 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.075+0000 7f75bfe00640 1 --2- 172.21.15.28:0/3972697228 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f5545ea20 0x560f5545c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:09.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.076+0000 7f75c5000640 1 -- 172.21.15.28:0/3972697228 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f5545ea20 msgr2=0x560f5545c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:09.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.076+0000 7f75c5000640 1 --2- 172.21.15.28:0/3972697228 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f5545ea20 0x560f5545c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:09.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.076+0000 7f75c5000640 1 -- 172.21.15.28:0/3972697228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f55453a30 msgr2=0x560f5545b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:09.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.076+0000 7f75c5000640 1 --2- 172.21.15.28:0/3972697228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f55453a30 0x560f5545b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:09.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.076+0000 7f75c5000640 1 -- 172.21.15.28:0/3972697228 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f55371920 con 0x560f55454350 2024-04-23T14:50:09.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.076+0000 7f75c5000640 1 --2- 172.21.15.28:0/3972697228 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f55454350 0x560f5545bea0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f75b401c040 tx=0x7f75b4019670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a19f9fc4e6e86f46 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:09.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.076+0000 7f75a6a00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:50:09.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.076+0000 7f75a6a00640 1 -- 172.21.15.28:0/3972697228 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b401e040 con 0x560f55454350 2024-04-23T14:50:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.076+0000 7f75a6a00640 1 -- 172.21.15.28:0/3972697228 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f75b4016040 con 0x560f55454350 2024-04-23T14:50:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.076+0000 7f75ca9b8740 1 -- 172.21.15.28:0/3972697228 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75b400fba0 con 0x560f55454350 2024-04-23T14:50:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.077+0000 7f75a6a00640 1 -- 172.21.15.28:0/3972697228 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f75b402f050 con 0x560f55454350 2024-04-23T14:50:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.077+0000 7f75ca9b8740 10 client.26194 fetch_fsmap learned FSMap version 20 2024-04-23T14:50:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.077+0000 7f75ca9b8740 1 -- 172.21.15.28:0/3972697228 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x560f551957d0 con 0x560f55454350 2024-04-23T14:50:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.077+0000 7f75a6a00640 1 -- 172.21.15.28:0/3972697228 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75b400fba0 con 0x560f55454350 2024-04-23T14:50:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.077+0000 7f75a6a00640 1 -- 172.21.15.28:0/3972697228 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f75b4021070 con 0x560f55454350 2024-04-23T14:50:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.077+0000 7f75ca9b8740 10 client.26194 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:50:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.077+0000 7f75ca9b8740 10 client.26194 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:50:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.077+0000 7f75ca9b8740 20 client.26194 populate_metadata read hostname 'smithi028' 2024-04-23T14:50:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.077+0000 7f75ca9b8740 1 --2- 172.21.15.28:0/3972697228 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560f5546ec90 0x560f55471080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.077+0000 7f75ca9b8740 4 client.26194 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:50:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.077+0000 7f75ca9b8740 1 -- 172.21.15.28:0/3972697228 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f553c6c10 con 0x560f5546ec90 2024-04-23T14:50:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.078+0000 7f75bfe00640 1 --2- 172.21.15.28:0/3972697228 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560f5546ec90 0x560f55471080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.078+0000 7f75bfe00640 1 --2- 172.21.15.28:0/3972697228 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560f5546ec90 0x560f55471080 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=617313290e13310e server_cookie=f1f4a7b9a2903271 in_seq=0 out_seq=0 2024-04-23T14:50:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.078+0000 7f75a6a00640 10 client.26194 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:50:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.079+0000 7f75a6a00640 1 -- 172.21.15.28:0/3972697228 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560f553c6c10 con 0x560f5546ec90 2024-04-23T14:50:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.079+0000 7f75a6a00640 10 client.26194 handle_command_reply: tid=0 2024-04-23T14:50:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.079+0000 7f75ca9b8740 1 client.26194 shutdown 2024-04-23T14:50:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.080+0000 7f75ca9b8740 1 -- 172.21.15.28:0/3972697228 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f55454350 msgr2=0x560f5545bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.080+0000 7f75ca9b8740 1 --2- 172.21.15.28:0/3972697228 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f55454350 0x560f5545bea0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f75b401c040 tx=0x7f75b4019670 comp rx=0 tx=0).stop 2024-04-23T14:50:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.080+0000 7f75ca9b8740 1 -- 172.21.15.28:0/3972697228 shutdown_connections 2024-04-23T14:50:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.080+0000 7f75ca9b8740 1 --2- 172.21.15.28:0/3972697228 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560f5546ec90 0x560f55471080 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.080+0000 7f75ca9b8740 1 --2- 172.21.15.28:0/3972697228 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f5545ea20 0x560f5545c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.080+0000 7f75ca9b8740 1 --2- 172.21.15.28:0/3972697228 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f55454350 0x560f5545bea0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.080+0000 7f75ca9b8740 1 --2- 172.21.15.28:0/3972697228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f55453a30 0x560f5545b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.080+0000 7f75ca9b8740 1 -- 172.21.15.28:0/3972697228 >> 172.21.15.28:0/3972697228 conn(0x560f5544fd50 msgr2=0x560f5544efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.080+0000 7f75a6a00640 0 client.26194 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:50:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.080+0000 7f75ca9b8740 1 -- 172.21.15.28:0/3972697228 shutdown_connections 2024-04-23T14:50:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.080+0000 7f75ca9b8740 1 -- 172.21.15.28:0/3972697228 wait complete. 2024-04-23T14:50:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.080+0000 7f75ca9b8740 20 client.26194 trim_cache size 0 max 16384 2024-04-23T14:50:09.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.101+0000 7f75ca9b8740 1 Processor -- start 2024-04-23T14:50:09.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.101+0000 7f75ca9b8740 1 -- start start 2024-04-23T14:50:09.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.101+0000 7f75ca9b8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f55453a30 0x560f5546a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:09.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.101+0000 7f75ca9b8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f55454350 0x560f554767c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:09.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.101+0000 7f75ca9b8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f5545ea20 0x560f55476d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:09.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.101+0000 7f75ca9b8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f75c01add70 con 0x560f5545ea20 2024-04-23T14:50:09.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.101+0000 7f75ca9b8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f75c01b05f0 con 0x560f55454350 2024-04-23T14:50:09.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.101+0000 7f75ca9b8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f75c01ae3e0 con 0x560f55453a30 2024-04-23T14:50:09.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.102+0000 7f75c5000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f55454350 0x560f554767c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:09.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.102+0000 7f75bfe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f55453a30 0x560f5546a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:09.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.102+0000 7f75c5000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f55454350 0x560f554767c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:47052/0 (socket says 172.21.15.28:47052) 2024-04-23T14:50:09.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.102+0000 7f75c5000640 1 -- 172.21.15.28:0/514181514 learned_addr learned my addr 172.21.15.28:0/514181514 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:09.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.102+0000 7f75bfe00640 1 --2- 172.21.15.28:0/514181514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f5545ea20 0x560f55476d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:09.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.102+0000 7f75c5000640 1 -- 172.21.15.28:0/514181514 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f55453a30 msgr2=0x560f5546a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:09.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.102+0000 7f75c5000640 1 --2- 172.21.15.28:0/514181514 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f55453a30 0x560f5546a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:09.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.102+0000 7f75c5000640 1 -- 172.21.15.28:0/514181514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f5545ea20 msgr2=0x560f55476d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:09.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.102+0000 7f75c5000640 1 --2- 172.21.15.28:0/514181514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f5545ea20 0x560f55476d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:09.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.102+0000 7f75c5000640 1 -- 172.21.15.28:0/514181514 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f550cb3f0 con 0x560f55454350 2024-04-23T14:50:09.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.102+0000 7f75bfe00640 1 --2- 172.21.15.28:0/514181514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f5545ea20 0x560f55476d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:50:09.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.102+0000 7f75c5000640 1 --2- 172.21.15.28:0/514181514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f55454350 0x560f554767c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f75b4009780 tx=0x7f75b400f440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82dc244f7a397f41 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:09.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.102+0000 7f75b1600640 1 -- 172.21.15.28:0/514181514 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b401e070 con 0x560f55454350 2024-04-23T14:50:09.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.102+0000 7f75b1600640 1 -- 172.21.15.28:0/514181514 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75b4009e50 con 0x560f55454350 2024-04-23T14:50:09.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.102+0000 7f75b1600640 1 -- 172.21.15.28:0/514181514 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b4003a10 con 0x560f55454350 2024-04-23T14:50:09.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.103+0000 7f75ca9b8740 1 -- 172.21.15.28:0/514181514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f55454350 msgr2=0x560f554767c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:09.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.103+0000 7f75ca9b8740 1 --2- 172.21.15.28:0/514181514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f55454350 0x560f554767c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f75b4009780 tx=0x7f75b400f440 comp rx=0 tx=0).stop 2024-04-23T14:50:09.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.103+0000 7f75ca9b8740 1 -- 172.21.15.28:0/514181514 shutdown_connections 2024-04-23T14:50:09.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.103+0000 7f75ca9b8740 1 --2- 172.21.15.28:0/514181514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f55454350 0x560f554767c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:09.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.103+0000 7f75ca9b8740 1 --2- 172.21.15.28:0/514181514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f5545ea20 0x560f55476d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:09.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.103+0000 7f75ca9b8740 1 --2- 172.21.15.28:0/514181514 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f55453a30 0x560f5546a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:09.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.103+0000 7f75ca9b8740 1 -- 172.21.15.28:0/514181514 >> 172.21.15.28:0/514181514 conn(0x560f5544fd50 msgr2=0x560f55455b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:09.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.103+0000 7f75ca9b8740 1 -- 172.21.15.28:0/514181514 shutdown_connections 2024-04-23T14:50:09.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.103+0000 7f75ca9b8740 1 -- 172.21.15.28:0/514181514 wait complete. 2024-04-23T14:50:09.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.104+0000 7f75ca9b8740 1 -- start start 2024-04-23T14:50:09.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.104+0000 7f75ca9b8740 1 Processor -- start 2024-04-23T14:50:09.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.105+0000 7f75ca9b8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f55453a30 0x560f55466da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:09.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.105+0000 7f75ca9b8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f55454350 0x560f554581f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:09.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.105+0000 7f75ca9b8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f5545ea20 0x560f55458730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:09.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.105+0000 7f75ca9b8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f75c01aeab0 con 0x560f55454350 2024-04-23T14:50:09.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.105+0000 7f75ca9b8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f75c01aff20 con 0x560f55453a30 2024-04-23T14:50:09.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.105+0000 7f75ca9b8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f75c01af180 con 0x560f5545ea20 2024-04-23T14:50:09.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.105+0000 7f75ca9b8740 10 client.0 _ll_register_callbacks cb 0x7f75b401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:50:09.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.105+0000 7f75bfe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f55453a30 0x560f55466da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:09.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.105+0000 7f75c5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f55454350 0x560f554581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:09.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.105+0000 7f75bfe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f55453a30 0x560f55466da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:47060/0 (socket says 172.21.15.28:47060) 2024-04-23T14:50:09.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.105+0000 7f75bfe00640 1 -- 172.21.15.28:0/2140282635 learned_addr learned my addr 172.21.15.28:0/2140282635 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:09.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.105+0000 7f75bfe00640 1 --2- 172.21.15.28:0/2140282635 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f5545ea20 0x560f55458730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:09.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.105+0000 7f75c5000640 1 -- 172.21.15.28:0/2140282635 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f5545ea20 msgr2=0x560f55458730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:09.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.105+0000 7f75c5000640 1 --2- 172.21.15.28:0/2140282635 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f5545ea20 0x560f55458730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:09.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.105+0000 7f75c5000640 1 -- 172.21.15.28:0/2140282635 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f55453a30 msgr2=0x560f55466da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:09.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.105+0000 7f75c5000640 1 --2- 172.21.15.28:0/2140282635 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f55453a30 0x560f55466da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:09.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.105+0000 7f75c5000640 1 -- 172.21.15.28:0/2140282635 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f5508b0f0 con 0x560f55454350 2024-04-23T14:50:09.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.105+0000 7f75bfe00640 1 --2- 172.21.15.28:0/2140282635 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f55453a30 0x560f55466da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:50:09.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.106+0000 7f75c5000640 1 --2- 172.21.15.28:0/2140282635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f55454350 0x560f554581f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f75b40056a0 tx=0x7f75b4019620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d8438c3f1467da6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:09.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.106+0000 7f75a6a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-23T14:50:09.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.106+0000 7f75a6a00640 1 -- 172.21.15.28:0/2140282635 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b401e070 con 0x560f55454350 2024-04-23T14:50:09.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.106+0000 7f75a6a00640 1 -- 172.21.15.28:0/2140282635 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f75b400fa60 con 0x560f55454350 2024-04-23T14:50:09.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.106+0000 7f75ca9b8740 1 -- 172.21.15.28:0/2140282635 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75b4009e50 con 0x560f55454350 2024-04-23T14:50:09.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.106+0000 7f75a6a00640 1 -- 172.21.15.28:0/2140282635 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f75b400fca0 con 0x560f55454350 2024-04-23T14:50:09.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.106+0000 7f75ca9b8740 10 client.16050 fetch_fsmap learned FSMap version 20 2024-04-23T14:50:09.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.106+0000 7f75ca9b8740 1 -- 172.21.15.28:0/2140282635 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x560f5537e810 con 0x560f55454350 2024-04-23T14:50:09.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.107+0000 7f75a6a00640 1 -- 172.21.15.28:0/2140282635 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75b4009e50 con 0x560f55454350 2024-04-23T14:50:09.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.107+0000 7f75a6a00640 1 -- 172.21.15.28:0/2140282635 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f75b4021050 con 0x560f55454350 2024-04-23T14:50:09.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.107+0000 7f75ca9b8740 10 client.16050 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:50:09.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.107+0000 7f75ca9b8740 10 client.16050 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:50:09.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.107+0000 7f75ca9b8740 20 client.16050 populate_metadata read hostname 'smithi028' 2024-04-23T14:50:09.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.107+0000 7f75ca9b8740 1 --2- 172.21.15.28:0/2140282635 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560f554d5160 0x560f554d7580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:09.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.107+0000 7f75ca9b8740 4 client.16050 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:50:09.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.107+0000 7f75ca9b8740 1 -- 172.21.15.28:0/2140282635 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560f553c0a80 con 0x560f554d5160 2024-04-23T14:50:09.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.107+0000 7f75bfe00640 1 --2- 172.21.15.28:0/2140282635 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560f554d5160 0x560f554d7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:09.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.108+0000 7f75bfe00640 1 --2- 172.21.15.28:0/2140282635 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560f554d5160 0x560f554d7580 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46637abb409079da server_cookie=a7967491ef27214c in_seq=0 out_seq=0 2024-04-23T14:50:09.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.108+0000 7f75a6a00640 10 client.16050 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:50:09.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.109+0000 7f75a6a00640 1 -- 172.21.15.28:0/2140282635 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x560f553c0a80 con 0x560f554d5160 2024-04-23T14:50:09.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.109+0000 7f75a6a00640 10 client.16050 handle_command_reply: tid=0 2024-04-23T14:50:09.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.109+0000 7f75ca9b8740 1 client.16050 shutdown 2024-04-23T14:50:09.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.109+0000 7f75ca9b8740 1 -- 172.21.15.28:0/2140282635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f55454350 msgr2=0x560f554581f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:09.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.109+0000 7f75ca9b8740 1 --2- 172.21.15.28:0/2140282635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f55454350 0x560f554581f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f75b40056a0 tx=0x7f75b4019620 comp rx=0 tx=0).stop 2024-04-23T14:50:09.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.109+0000 7f75ca9b8740 1 -- 172.21.15.28:0/2140282635 shutdown_connections 2024-04-23T14:50:09.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.109+0000 7f75ca9b8740 1 --2- 172.21.15.28:0/2140282635 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560f554d5160 0x560f554d7580 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:09.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.109+0000 7f75ca9b8740 1 --2- 172.21.15.28:0/2140282635 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f5545ea20 0x560f55458730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:09.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.109+0000 7f75ca9b8740 1 --2- 172.21.15.28:0/2140282635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f55454350 0x560f554581f0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:09.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.109+0000 7f75ca9b8740 1 --2- 172.21.15.28:0/2140282635 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f55453a30 0x560f55466da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:09.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.109+0000 7f75ca9b8740 1 -- 172.21.15.28:0/2140282635 >> 172.21.15.28:0/2140282635 conn(0x560f5544fd50 msgr2=0x560f5544e740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:09.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.109+0000 7f75a6a00640 0 client.16050 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:50:09.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.109+0000 7f75ca9b8740 1 -- 172.21.15.28:0/2140282635 shutdown_connections 2024-04-23T14:50:09.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.109+0000 7f75ca9b8740 1 -- 172.21.15.28:0/2140282635 wait complete. 2024-04-23T14:50:09.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.109+0000 7f75ca9b8740 20 client.16050 trim_cache size 0 max 16384 2024-04-23T14:50:09.114 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:50:09.114 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-23T14:50:09.114 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-23T14:50:09.114 INFO:teuthology.orchestra.run.smithi028.stdout: "74724afe-8998-4b90-a254-f9bfcbd06ad8": { 2024-04-23T14:50:09.114 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-23T14:50:09.114 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "74724afe-8998-4b90-a254-f9bfcbd06ad8", 2024-04-23T14:50:09.114 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-23T14:50:09.114 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:50:09.114 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:50:09.114 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:50:09.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.110+0000 7f75c6400640 1 -- 172.21.15.28:0/2164327583 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f75a80a3de0 msgr2=0x7f75a80a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:09.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.110+0000 7f75c6400640 1 --2- 172.21.15.28:0/2164327583 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f75a80a3de0 0x7f75a80a62a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f75b8005fd0 tx=0x7f75b80058a0 comp rx=0 tx=0).stop 2024-04-23T14:50:09.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.110+0000 7f75c6400640 1 -- 172.21.15.28:0/2164327583 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75c010e740 msgr2=0x7f75c019a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:09.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.110+0000 7f75c6400640 1 --2- 172.21.15.28:0/2164327583 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75c010e740 0x7f75c019a7f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f75c0075320 tx=0x7f75b800ef10 comp rx=0 tx=0).stop 2024-04-23T14:50:09.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.110+0000 7f75c6400640 1 -- 172.21.15.28:0/2164327583 shutdown_connections 2024-04-23T14:50:09.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.110+0000 7f75c6400640 1 --2- 172.21.15.28:0/2164327583 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f75a80a3de0 0x7f75a80a62a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:09.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.110+0000 7f75c6400640 1 --2- 172.21.15.28:0/2164327583 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75c010e740 0x7f75c019a7f0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:09.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.110+0000 7f75c6400640 1 --2- 172.21.15.28:0/2164327583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f75c0074cf0 0x7f75c01a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:09.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.110+0000 7f75c6400640 1 --2- 172.21.15.28:0/2164327583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75c00743d0 0x7f75c01a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:09.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.110+0000 7f75c6400640 1 -- 172.21.15.28:0/2164327583 >> 172.21.15.28:0/2164327583 conn(0x7f75c00fd9d0 msgr2=0x7f75c00fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:09.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.110+0000 7f75c6400640 1 -- 172.21.15.28:0/2164327583 shutdown_connections 2024-04-23T14:50:09.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:09.110+0000 7f75c6400640 1 -- 172.21.15.28:0/2164327583 wait complete. 2024-04-23T14:50:09.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:08 smithi038 ceph-mon[32348]: pgmap v186: 97 pgs: 97 active+clean; 289 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.9 MiB/s wr, 247 op/s 2024-04-23T14:50:09.131 INFO:tasks.cephfs.filesystem:scrub status for tag:74724afe-8998-4b90-a254-f9bfcbd06ad8 - {'path': '/', 'tag': '74724afe-8998-4b90-a254-f9bfcbd06ad8', 'options': 'recursive,force'} 2024-04-23T14:50:09.494 INFO:tasks.workunit.client.0.smithi028.stdout: 1 492223 9.69 MB/sec execute 145 sec latency 21.665 ms 2024-04-23T14:50:10.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:09 smithi105 ceph-mon[32555]: scrub queued for path: / 2024-04-23T14:50:10.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:09 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:50:10.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:09 smithi105 ceph-mon[32555]: scrub summary: active paths [/] 2024-04-23T14:50:10.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:09 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:50:10.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:09 smithi038 ceph-mon[32348]: scrub queued for path: / 2024-04-23T14:50:10.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:09 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:50:10.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:09 smithi038 ceph-mon[32348]: scrub summary: active paths [/] 2024-04-23T14:50:10.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:09 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:50:10.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:09 smithi028 ceph-mon[27810]: scrub queued for path: / 2024-04-23T14:50:10.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:09 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:50:10.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:09 smithi028 ceph-mon[27810]: scrub summary: active paths [/] 2024-04-23T14:50:10.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:09 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:50:10.495 INFO:tasks.workunit.client.0.smithi028.stdout: 1 494053 9.71 MB/sec execute 146 sec latency 32.510 ms 2024-04-23T14:50:11.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:10 smithi038 ceph-mon[32348]: pgmap v187: 97 pgs: 97 active+clean; 289 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 929 KiB/s rd, 2.8 MiB/s wr, 176 op/s 2024-04-23T14:50:11.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:10 smithi038 ceph-mon[32348]: scrub summary: idle 2024-04-23T14:50:11.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:10 smithi028 ceph-mon[27810]: pgmap v187: 97 pgs: 97 active+clean; 289 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 929 KiB/s rd, 2.8 MiB/s wr, 176 op/s 2024-04-23T14:50:11.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:10 smithi028 ceph-mon[27810]: scrub summary: idle 2024-04-23T14:50:11.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:10 smithi105 ceph-mon[32555]: pgmap v187: 97 pgs: 97 active+clean; 289 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 929 KiB/s rd, 2.8 MiB/s wr, 176 op/s 2024-04-23T14:50:11.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:10 smithi105 ceph-mon[32555]: scrub summary: idle 2024-04-23T14:50:11.495 INFO:tasks.workunit.client.0.smithi028.stdout: 1 494911 9.66 MB/sec execute 147 sec latency 42.682 ms 2024-04-23T14:50:12.495 INFO:tasks.workunit.client.0.smithi028.stdout: 1 496447 9.64 MB/sec execute 148 sec latency 59.893 ms 2024-04-23T14:50:13.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:12 smithi038 ceph-mon[32348]: pgmap v188: 97 pgs: 97 active+clean; 289 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 929 KiB/s rd, 2.8 MiB/s wr, 176 op/s 2024-04-23T14:50:13.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:12 smithi028 ceph-mon[27810]: pgmap v188: 97 pgs: 97 active+clean; 289 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 929 KiB/s rd, 2.8 MiB/s wr, 176 op/s 2024-04-23T14:50:13.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:12 smithi105 ceph-mon[32555]: pgmap v188: 97 pgs: 97 active+clean; 289 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 929 KiB/s rd, 2.8 MiB/s wr, 176 op/s 2024-04-23T14:50:13.495 INFO:tasks.workunit.client.0.smithi028.stdout: 1 498001 9.65 MB/sec execute 149 sec latency 9.817 ms 2024-04-23T14:50:14.495 INFO:tasks.workunit.client.0.smithi028.stdout: 1 499492 9.62 MB/sec execute 150 sec latency 22.439 ms 2024-04-23T14:50:15.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:14 smithi038 ceph-mon[32348]: pgmap v189: 97 pgs: 97 active+clean; 294 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.0 MiB/s wr, 247 op/s 2024-04-23T14:50:15.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:14 smithi028 ceph-mon[27810]: pgmap v189: 97 pgs: 97 active+clean; 294 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.0 MiB/s wr, 247 op/s 2024-04-23T14:50:15.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:14 smithi105 ceph-mon[32555]: pgmap v189: 97 pgs: 97 active+clean; 294 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.0 MiB/s wr, 247 op/s 2024-04-23T14:50:15.495 INFO:tasks.workunit.client.0.smithi028.stdout: 1 501230 9.64 MB/sec execute 151 sec latency 22.307 ms 2024-04-23T14:50:16.495 INFO:tasks.workunit.client.0.smithi028.stdout: 1 502376 9.62 MB/sec execute 152 sec latency 46.664 ms 2024-04-23T14:50:17.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:16 smithi038 ceph-mon[32348]: pgmap v190: 97 pgs: 97 active+clean; 294 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 2.7 MiB/s wr, 145 op/s 2024-04-23T14:50:17.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:16 smithi028 ceph-mon[27810]: pgmap v190: 97 pgs: 97 active+clean; 294 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 2.7 MiB/s wr, 145 op/s 2024-04-23T14:50:17.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:16 smithi105 ceph-mon[32555]: pgmap v190: 97 pgs: 97 active+clean; 294 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 2.7 MiB/s wr, 145 op/s 2024-04-23T14:50:17.495 INFO:tasks.workunit.client.0.smithi028.stdout: 1 504124 9.61 MB/sec execute 153 sec latency 37.367 ms 2024-04-23T14:50:18.495 INFO:tasks.workunit.client.0.smithi028.stdout: 1 505375 9.59 MB/sec execute 154 sec latency 18.593 ms 2024-04-23T14:50:19.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:18 smithi038 ceph-mon[32348]: pgmap v191: 97 pgs: 97 active+clean; 300 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.9 MiB/s wr, 212 op/s 2024-04-23T14:50:19.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:18 smithi028 ceph-mon[27810]: pgmap v191: 97 pgs: 97 active+clean; 300 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.9 MiB/s wr, 212 op/s 2024-04-23T14:50:19.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:18 smithi105 ceph-mon[32555]: pgmap v191: 97 pgs: 97 active+clean; 300 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.9 MiB/s wr, 212 op/s 2024-04-23T14:50:19.495 INFO:tasks.workunit.client.0.smithi028.stdout: 1 507502 9.60 MB/sec execute 155 sec latency 14.797 ms 2024-04-23T14:50:20.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:19 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:50:20.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:19 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:50:20.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:19 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:50:20.496 INFO:tasks.workunit.client.0.smithi028.stdout: 1 509252 9.60 MB/sec execute 156 sec latency 17.958 ms 2024-04-23T14:50:21.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:20 smithi038 ceph-mon[32348]: pgmap v192: 97 pgs: 97 active+clean; 300 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 980 KiB/s rd, 2.5 MiB/s wr, 138 op/s 2024-04-23T14:50:21.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:20 smithi028 ceph-mon[27810]: pgmap v192: 97 pgs: 97 active+clean; 300 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 980 KiB/s rd, 2.5 MiB/s wr, 138 op/s 2024-04-23T14:50:21.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:20 smithi105 ceph-mon[32555]: pgmap v192: 97 pgs: 97 active+clean; 300 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 980 KiB/s rd, 2.5 MiB/s wr, 138 op/s 2024-04-23T14:50:21.496 INFO:tasks.workunit.client.0.smithi028.stdout: 1 510815 9.59 MB/sec execute 157 sec latency 156.573 ms 2024-04-23T14:50:22.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:21 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:50:22.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:21 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:50:22.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:21 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:50:22.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:21 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:50:22.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:21 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:50:22.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:21 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:50:22.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:21 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:50:22.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:21 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:50:22.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:21 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:50:22.496 INFO:tasks.workunit.client.0.smithi028.stdout: 1 512241 9.58 MB/sec execute 158 sec latency 31.418 ms 2024-04-23T14:50:23.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:22 smithi038 ceph-mon[32348]: pgmap v193: 97 pgs: 97 active+clean; 300 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 980 KiB/s rd, 2.5 MiB/s wr, 138 op/s 2024-04-23T14:50:23.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:22 smithi028 ceph-mon[27810]: pgmap v193: 97 pgs: 97 active+clean; 300 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 980 KiB/s rd, 2.5 MiB/s wr, 138 op/s 2024-04-23T14:50:23.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:22 smithi105 ceph-mon[32555]: pgmap v193: 97 pgs: 97 active+clean; 300 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 980 KiB/s rd, 2.5 MiB/s wr, 138 op/s 2024-04-23T14:50:23.496 INFO:tasks.workunit.client.0.smithi028.stdout: 1 513724 9.56 MB/sec execute 159 sec latency 20.829 ms 2024-04-23T14:50:24.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:23 smithi038 ceph-mon[32348]: pgmap v194: 97 pgs: 97 active+clean; 325 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.2 MiB/s wr, 215 op/s 2024-04-23T14:50:24.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:23 smithi028 ceph-mon[27810]: pgmap v194: 97 pgs: 97 active+clean; 325 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.2 MiB/s wr, 215 op/s 2024-04-23T14:50:24.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:23 smithi105 ceph-mon[32555]: pgmap v194: 97 pgs: 97 active+clean; 325 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.2 MiB/s wr, 215 op/s 2024-04-23T14:50:24.496 INFO:tasks.workunit.client.0.smithi028.stdout: 1 515488 9.58 MB/sec execute 160 sec latency 14.248 ms 2024-04-23T14:50:25.496 INFO:tasks.workunit.client.0.smithi028.stdout: 1 516950 9.56 MB/sec execute 161 sec latency 15.177 ms 2024-04-23T14:50:26.496 INFO:tasks.workunit.client.0.smithi028.stdout: 1 519060 9.58 MB/sec execute 162 sec latency 16.389 ms 2024-04-23T14:50:26.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:26 smithi038 ceph-mon[32348]: pgmap v195: 97 pgs: 97 active+clean; 325 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 144 op/s 2024-04-23T14:50:26.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:26 smithi028 ceph-mon[27810]: pgmap v195: 97 pgs: 97 active+clean; 325 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 144 op/s 2024-04-23T14:50:27.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:26 smithi105 ceph-mon[32555]: pgmap v195: 97 pgs: 97 active+clean; 325 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 144 op/s 2024-04-23T14:50:27.496 INFO:tasks.workunit.client.0.smithi028.stdout: 1 520322 9.55 MB/sec execute 163 sec latency 65.366 ms 2024-04-23T14:50:28.496 INFO:tasks.workunit.client.0.smithi028.stdout: 1 522323 9.57 MB/sec execute 164 sec latency 41.603 ms 2024-04-23T14:50:28.804 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:28 smithi105 ceph-mon[32555]: pgmap v196: 97 pgs: 97 active+clean; 341 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.3 MiB/s wr, 204 op/s 2024-04-23T14:50:28.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:28 smithi038 ceph-mon[32348]: pgmap v196: 97 pgs: 97 active+clean; 341 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.3 MiB/s wr, 204 op/s 2024-04-23T14:50:28.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:28 smithi028 ceph-mon[27810]: pgmap v196: 97 pgs: 97 active+clean; 341 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.3 MiB/s wr, 204 op/s 2024-04-23T14:50:29.497 INFO:tasks.workunit.client.0.smithi028.stdout: 1 523397 9.54 MB/sec execute 165 sec latency 19.634 ms 2024-04-23T14:50:30.497 INFO:tasks.workunit.client.0.smithi028.stdout: 1 525122 9.54 MB/sec execute 166 sec latency 31.237 ms 2024-04-23T14:50:30.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:30 smithi038 ceph-mon[32348]: pgmap v197: 97 pgs: 97 active+clean; 341 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1004 KiB/s rd, 4.1 MiB/s wr, 137 op/s 2024-04-23T14:50:30.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:30 smithi028 ceph-mon[27810]: pgmap v197: 97 pgs: 97 active+clean; 341 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1004 KiB/s rd, 4.1 MiB/s wr, 137 op/s 2024-04-23T14:50:31.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:30 smithi105 ceph-mon[32555]: pgmap v197: 97 pgs: 97 active+clean; 341 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1004 KiB/s rd, 4.1 MiB/s wr, 137 op/s 2024-04-23T14:50:31.497 INFO:tasks.workunit.client.0.smithi028.stdout: 1 526486 9.52 MB/sec execute 167 sec latency 18.053 ms 2024-04-23T14:50:32.497 INFO:tasks.workunit.client.0.smithi028.stdout: 1 527904 9.50 MB/sec execute 168 sec latency 66.674 ms 2024-04-23T14:50:32.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:32 smithi038 ceph-mon[32348]: pgmap v198: 97 pgs: 97 active+clean; 341 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1004 KiB/s rd, 4.1 MiB/s wr, 137 op/s 2024-04-23T14:50:32.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:32 smithi028 ceph-mon[27810]: pgmap v198: 97 pgs: 97 active+clean; 341 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1004 KiB/s rd, 4.1 MiB/s wr, 137 op/s 2024-04-23T14:50:33.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:32 smithi105 ceph-mon[32555]: pgmap v198: 97 pgs: 97 active+clean; 341 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1004 KiB/s rd, 4.1 MiB/s wr, 137 op/s 2024-04-23T14:50:33.497 INFO:tasks.workunit.client.0.smithi028.stdout: 1 529593 9.52 MB/sec execute 169 sec latency 28.933 ms 2024-04-23T14:50:34.497 INFO:tasks.workunit.client.0.smithi028.stdout: 1 530760 9.50 MB/sec execute 170 sec latency 22.171 ms 2024-04-23T14:50:34.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:34 smithi038 ceph-mon[32348]: pgmap v199: 97 pgs: 97 active+clean; 343 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.2 MiB/s wr, 202 op/s 2024-04-23T14:50:34.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:34 smithi028 ceph-mon[27810]: pgmap v199: 97 pgs: 97 active+clean; 343 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.2 MiB/s wr, 202 op/s 2024-04-23T14:50:35.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:34 smithi105 ceph-mon[32555]: pgmap v199: 97 pgs: 97 active+clean; 343 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.2 MiB/s wr, 202 op/s 2024-04-23T14:50:35.497 INFO:tasks.workunit.client.0.smithi028.stdout: 1 532506 9.49 MB/sec execute 171 sec latency 19.336 ms 2024-04-23T14:50:36.498 INFO:tasks.workunit.client.0.smithi028.stdout: 1 533579 9.47 MB/sec execute 172 sec latency 20.333 ms 2024-04-23T14:50:36.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:36 smithi038 ceph-mon[32348]: pgmap v200: 97 pgs: 97 active+clean; 343 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 929 KiB/s rd, 2.4 MiB/s wr, 125 op/s 2024-04-23T14:50:36.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:36 smithi028 ceph-mon[27810]: pgmap v200: 97 pgs: 97 active+clean; 343 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 929 KiB/s rd, 2.4 MiB/s wr, 125 op/s 2024-04-23T14:50:37.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:36 smithi105 ceph-mon[32555]: pgmap v200: 97 pgs: 97 active+clean; 343 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 929 KiB/s rd, 2.4 MiB/s wr, 125 op/s 2024-04-23T14:50:37.498 INFO:tasks.workunit.client.0.smithi028.stdout: 1 534854 9.45 MB/sec execute 173 sec latency 80.249 ms 2024-04-23T14:50:38.498 INFO:tasks.workunit.client.0.smithi028.stdout: 1 536659 9.47 MB/sec execute 174 sec latency 46.477 ms 2024-04-23T14:50:38.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:38 smithi038 ceph-mon[32348]: pgmap v201: 97 pgs: 97 active+clean; 349 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.5 MiB/s wr, 185 op/s 2024-04-23T14:50:38.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:38 smithi028 ceph-mon[27810]: pgmap v201: 97 pgs: 97 active+clean; 349 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.5 MiB/s wr, 185 op/s 2024-04-23T14:50:39.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:38 smithi105 ceph-mon[32555]: pgmap v201: 97 pgs: 97 active+clean; 349 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.5 MiB/s wr, 185 op/s 2024-04-23T14:50:39.132 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:50:39.132 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:50:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.358+0000 7f74b4a00640 1 -- 172.21.15.28:0/1287351355 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74b0100570 msgr2=0x7f74b01009f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.358+0000 7f74b4a00640 1 --2- 172.21.15.28:0/1287351355 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74b0100570 0x7f74b01009f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f749c00aa40 tx=0x7f749c02b760 comp rx=0 tx=0).stop 2024-04-23T14:50:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.358+0000 7f74b4a00640 1 -- 172.21.15.28:0/1287351355 shutdown_connections 2024-04-23T14:50:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.358+0000 7f74b4a00640 1 --2- 172.21.15.28:0/1287351355 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74b0100570 0x7f74b01009f0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.358+0000 7f74b4a00640 1 --2- 172.21.15.28:0/1287351355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74b0100f30 0x7f74b010d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.358+0000 7f74b4a00640 1 --2- 172.21.15.28:0/1287351355 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f74b01065e0 0x7f74b01069c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.358+0000 7f74b4a00640 1 -- 172.21.15.28:0/1287351355 >> 172.21.15.28:0/1287351355 conn(0x7f74b0077390 msgr2=0x7f74b00fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.359+0000 7f74b4a00640 1 -- 172.21.15.28:0/1287351355 shutdown_connections 2024-04-23T14:50:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.359+0000 7f74b4a00640 1 -- 172.21.15.28:0/1287351355 wait complete. 2024-04-23T14:50:39.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.360+0000 7f74b4a00640 1 Processor -- start 2024-04-23T14:50:39.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.360+0000 7f74b4a00640 1 -- start start 2024-04-23T14:50:39.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.360+0000 7f74b4a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f74b0100570 0x7f74b01a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:39.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.360+0000 7f74b4a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74b0100f30 0x7f74b01a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:39.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.361+0000 7f74b4a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74b01065e0 0x7f74b019a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:39.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.361+0000 7f74b4a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f74b0110090 con 0x7f74b01065e0 2024-04-23T14:50:39.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.361+0000 7f74b4a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f74b010ff10 con 0x7f74b0100f30 2024-04-23T14:50:39.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.361+0000 7f74b4a00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f74b0110210 con 0x7f74b0100570 2024-04-23T14:50:39.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.361+0000 7f74ade00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74b0100f30 0x7f74b01a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:39.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.361+0000 7f74ade00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74b0100f30 0x7f74b01a0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:52768/0 (socket says 172.21.15.28:52768) 2024-04-23T14:50:39.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.361+0000 7f74ade00640 1 -- 172.21.15.28:0/3552991015 learned_addr learned my addr 172.21.15.28:0/3552991015 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:39.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.361+0000 7f74af200640 1 --2- 172.21.15.28:0/3552991015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74b01065e0 0x7f74b019a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:39.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.361+0000 7f74ae800640 1 --2- 172.21.15.28:0/3552991015 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f74b0100570 0x7f74b01a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:39.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.361+0000 7f74ade00640 1 -- 172.21.15.28:0/3552991015 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f74b0100570 msgr2=0x7f74b01a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:39.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.361+0000 7f74ade00640 1 --2- 172.21.15.28:0/3552991015 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f74b0100570 0x7f74b01a0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.361+0000 7f74ade00640 1 -- 172.21.15.28:0/3552991015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74b01065e0 msgr2=0x7f74b019a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:39.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.361+0000 7f74ade00640 1 --2- 172.21.15.28:0/3552991015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74b01065e0 0x7f74b019a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.361+0000 7f74ade00640 1 -- 172.21.15.28:0/3552991015 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74b019afd0 con 0x7f74b0100f30 2024-04-23T14:50:39.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.362+0000 7f74ade00640 1 --2- 172.21.15.28:0/3552991015 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74b0100f30 0x7f74b01a0bd0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f749c02bc80 tx=0x7f749c030bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:39.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.362+0000 7f74a7400640 1 -- 172.21.15.28:0/3552991015 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749c03f070 con 0x7f74b0100f30 2024-04-23T14:50:39.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.362+0000 7f74a7400640 1 -- 172.21.15.28:0/3552991015 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f749c003cd0 con 0x7f74b0100f30 2024-04-23T14:50:39.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.362+0000 7f74a7400640 1 -- 172.21.15.28:0/3552991015 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749c036040 con 0x7f74b0100f30 2024-04-23T14:50:39.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.362+0000 7f74b4a00640 1 -- 172.21.15.28:0/3552991015 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74b019b260 con 0x7f74b0100f30 2024-04-23T14:50:39.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.362+0000 7f74b4a00640 1 -- 172.21.15.28:0/3552991015 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74b01a74e0 con 0x7f74b0100f30 2024-04-23T14:50:39.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.366+0000 7f74a7400640 1 -- 172.21.15.28:0/3552991015 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f749c042070 con 0x7f74b0100f30 2024-04-23T14:50:39.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.366+0000 7f74b8272740 1 Processor -- start 2024-04-23T14:50:39.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.367+0000 7f74b8272740 1 -- start start 2024-04-23T14:50:39.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.367+0000 7f74b8272740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbbd466a30 0x55bbbd466e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:39.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.367+0000 7f74b8272740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bbbd467350 0x55bbbd4714e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:39.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.367+0000 7f74b8272740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbbd471a20 0x55bbbd473e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:39.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.367+0000 7f74b8272740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bbbd3cc1f0 con 0x55bbbd471a20 2024-04-23T14:50:39.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.367+0000 7f74b8272740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55bbbd372350 con 0x55bbbd467350 2024-04-23T14:50:39.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.367+0000 7f74b8272740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55bbbd3c81b0 con 0x55bbbd466a30 2024-04-23T14:50:39.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.367+0000 7f74ae800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bbbd467350 0x55bbbd4714e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:39.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.367+0000 7f74ae800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bbbd467350 0x55bbbd4714e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:52774/0 (socket says 172.21.15.28:52774) 2024-04-23T14:50:39.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.367+0000 7f74ae800640 1 -- 172.21.15.28:0/3972544188 learned_addr learned my addr 172.21.15.28:0/3972544188 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:39.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.367+0000 7f74af200640 1 --2- 172.21.15.28:0/3972544188 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbbd466a30 0x55bbbd466e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:39.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.367+0000 7f74ade00640 1 --2- 172.21.15.28:0/3972544188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbbd471a20 0x55bbbd473e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:39.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.368+0000 7f74ae800640 1 -- 172.21.15.28:0/3972544188 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbbd466a30 msgr2=0x55bbbd466e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:39.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.368+0000 7f74ae800640 1 --2- 172.21.15.28:0/3972544188 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbbd466a30 0x55bbbd466e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.368+0000 7f74ae800640 1 -- 172.21.15.28:0/3972544188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbbd471a20 msgr2=0x55bbbd473e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:39.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.368+0000 7f74ae800640 1 --2- 172.21.15.28:0/3972544188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbbd471a20 0x55bbbd473e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.368+0000 7f74ae800640 1 -- 172.21.15.28:0/3972544188 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bbbd3c6f50 con 0x55bbbd467350 2024-04-23T14:50:39.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.368+0000 7f74ae800640 1 --2- 172.21.15.28:0/3972544188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bbbd467350 0x55bbbd4714e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f74a0009ca0 tx=0x7f74a0011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf8ade300dda2f40 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:39.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.368+0000 7f749b400640 1 -- 172.21.15.28:0/3972544188 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a001e070 con 0x55bbbd467350 2024-04-23T14:50:39.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.368+0000 7f749b400640 1 -- 172.21.15.28:0/3972544188 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74a0019b10 con 0x55bbbd467350 2024-04-23T14:50:39.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.368+0000 7f749b400640 1 -- 172.21.15.28:0/3972544188 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a000eae0 con 0x55bbbd467350 2024-04-23T14:50:39.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.368+0000 7f74a7400640 1 --2- 172.21.15.28:0/3552991015 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f74940a3da0 0x7f74940a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:39.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.369+0000 7f74a7400640 1 -- 172.21.15.28:0/3552991015 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f749c0f08f0 con 0x7f74b0100f30 2024-04-23T14:50:39.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.369+0000 7f74af200640 1 --2- 172.21.15.28:0/3552991015 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f74940a3da0 0x7f74940a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:39.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.369+0000 7f74b8272740 1 -- 172.21.15.28:0/3972544188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bbbd467350 msgr2=0x55bbbd4714e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:39.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.369+0000 7f74b8272740 1 --2- 172.21.15.28:0/3972544188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bbbd467350 0x55bbbd4714e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f74a0009ca0 tx=0x7f74a0011810 comp rx=0 tx=0).stop 2024-04-23T14:50:39.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.369+0000 7f74b8272740 1 -- 172.21.15.28:0/3972544188 shutdown_connections 2024-04-23T14:50:39.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.369+0000 7f74b8272740 1 --2- 172.21.15.28:0/3972544188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bbbd467350 0x55bbbd4714e0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.369+0000 7f74b8272740 1 --2- 172.21.15.28:0/3972544188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbbd471a20 0x55bbbd473e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.369+0000 7f74b8272740 1 --2- 172.21.15.28:0/3972544188 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbbd466a30 0x55bbbd466e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.369+0000 7f74b8272740 1 -- 172.21.15.28:0/3972544188 >> 172.21.15.28:0/3972544188 conn(0x55bbbd462d50 msgr2=0x55bbbd463130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.369+0000 7f74af200640 1 --2- 172.21.15.28:0/3552991015 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f74940a3da0 0x7f74940a6260 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f74a8005fd0 tx=0x7f74a8009500 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.370+0000 7f74b8272740 1 -- 172.21.15.28:0/3972544188 shutdown_connections 2024-04-23T14:50:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.370+0000 7f74b8272740 1 -- 172.21.15.28:0/3972544188 wait complete. 2024-04-23T14:50:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.370+0000 7f74b8272740 1 -- start start 2024-04-23T14:50:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.370+0000 7f74b8272740 1 Processor -- start 2024-04-23T14:50:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.371+0000 7f74b8272740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbbd466a30 0x55bbbd46ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.371+0000 7f74b8272740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbbd467350 0x55bbbd46ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.371+0000 7f74b8272740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bbbd471a20 0x55bbbd46f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.371+0000 7f74b8272740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bbbd474810 con 0x55bbbd467350 2024-04-23T14:50:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.371+0000 7f74b8272740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55bbbd385690 con 0x55bbbd471a20 2024-04-23T14:50:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.371+0000 7f74b8272740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55bbbd474990 con 0x55bbbd466a30 2024-04-23T14:50:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.371+0000 7f74b8272740 10 client.0 _ll_register_callbacks cb 0x55bbbd3c1650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:50:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.371+0000 7f74af200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bbbd471a20 0x55bbbd46f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.371+0000 7f74ade00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbbd467350 0x55bbbd46ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.371+0000 7f74af200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bbbd471a20 0x55bbbd46f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:52776/0 (socket says 172.21.15.28:52776) 2024-04-23T14:50:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.371+0000 7f74af200640 1 -- 172.21.15.28:0/2741029019 learned_addr learned my addr 172.21.15.28:0/2741029019 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.371+0000 7f74ae800640 1 --2- 172.21.15.28:0/2741029019 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbbd466a30 0x55bbbd46ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.372+0000 7f74ae800640 1 -- 172.21.15.28:0/2741029019 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bbbd471a20 msgr2=0x55bbbd46f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.372+0000 7f74ae800640 1 --2- 172.21.15.28:0/2741029019 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bbbd471a20 0x55bbbd46f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.372+0000 7f74ae800640 1 -- 172.21.15.28:0/2741029019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbbd467350 msgr2=0x55bbbd46ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.372+0000 7f74ae800640 1 --2- 172.21.15.28:0/2741029019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbbd467350 0x55bbbd46ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.372+0000 7f74ae800640 1 -- 172.21.15.28:0/2741029019 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bbbd384920 con 0x55bbbd466a30 2024-04-23T14:50:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.372+0000 7f74af200640 1 --2- 172.21.15.28:0/2741029019 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bbbd471a20 0x55bbbd46f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:50:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.372+0000 7f74ae800640 1 --2- 172.21.15.28:0/2741029019 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbbd466a30 0x55bbbd46ea10 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f74a0005890 tx=0x7f74a0009980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2de601867744f6c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.372+0000 7f748fe00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:50:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.372+0000 7f748fe00640 1 -- 172.21.15.28:0/2741029019 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a0026030 con 0x55bbbd466a30 2024-04-23T14:50:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.372+0000 7f748fe00640 1 -- 172.21.15.28:0/2741029019 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f74a0019640 con 0x55bbbd466a30 2024-04-23T14:50:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.372+0000 7f74b8272740 1 -- 172.21.15.28:0/2741029019 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74a0019b10 con 0x55bbbd466a30 2024-04-23T14:50:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.373+0000 7f748fe00640 1 -- 172.21.15.28:0/2741029019 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f74a0030050 con 0x55bbbd466a30 2024-04-23T14:50:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.373+0000 7f74b8272740 10 client.26212 fetch_fsmap learned FSMap version 20 2024-04-23T14:50:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.373+0000 7f74b8272740 1 -- 172.21.15.28:0/2741029019 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55bbbd1a87d0 con 0x55bbbd466a30 2024-04-23T14:50:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.373+0000 7f748fe00640 1 -- 172.21.15.28:0/2741029019 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74a0019b10 con 0x55bbbd466a30 2024-04-23T14:50:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.374+0000 7f748fe00640 1 -- 172.21.15.28:0/2741029019 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f74a000e7e0 con 0x55bbbd466a30 2024-04-23T14:50:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.374+0000 7f74b8272740 10 client.26212 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:50:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.374+0000 7f74b8272740 10 client.26212 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:50:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.374+0000 7f74b8272740 20 client.26212 populate_metadata read hostname 'smithi028' 2024-04-23T14:50:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.374+0000 7f74b8272740 1 --2- 172.21.15.28:0/2741029019 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55bbbd481dd0 0x55bbbd4841c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:39.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.374+0000 7f74b8272740 4 client.26212 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:50:39.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.374+0000 7f74b8272740 1 -- 172.21.15.28:0/2741029019 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bbbd3d9c10 con 0x55bbbd481dd0 2024-04-23T14:50:39.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.374+0000 7f74ade00640 1 --2- 172.21.15.28:0/2741029019 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55bbbd481dd0 0x55bbbd4841c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:39.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.375+0000 7f74ade00640 1 --2- 172.21.15.28:0/2741029019 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55bbbd481dd0 0x55bbbd4841c0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc0abce2d9c69749 server_cookie=3ab8edfe9b3f083f in_seq=0 out_seq=0 2024-04-23T14:50:39.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.375+0000 7f748fe00640 10 client.26212 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:50:39.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.377+0000 7f748fe00640 1 -- 172.21.15.28:0/2741029019 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bbbd3d9c10 con 0x55bbbd481dd0 2024-04-23T14:50:39.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.377+0000 7f748fe00640 10 client.26212 handle_command_reply: tid=0 2024-04-23T14:50:39.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.377+0000 7f74b8272740 1 client.26212 shutdown 2024-04-23T14:50:39.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.378+0000 7f74b8272740 1 -- 172.21.15.28:0/2741029019 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbbd466a30 msgr2=0x55bbbd46ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:39.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.378+0000 7f74b8272740 1 --2- 172.21.15.28:0/2741029019 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbbd466a30 0x55bbbd46ea10 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f74a0005890 tx=0x7f74a0009980 comp rx=0 tx=0).stop 2024-04-23T14:50:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.378+0000 7f74b8272740 1 -- 172.21.15.28:0/2741029019 shutdown_connections 2024-04-23T14:50:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.378+0000 7f74b8272740 1 --2- 172.21.15.28:0/2741029019 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55bbbd481dd0 0x55bbbd4841c0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.378+0000 7f74b8272740 1 --2- 172.21.15.28:0/2741029019 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bbbd471a20 0x55bbbd46f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.378+0000 7f74b8272740 1 --2- 172.21.15.28:0/2741029019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbbd467350 0x55bbbd46ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.378+0000 7f74b8272740 1 --2- 172.21.15.28:0/2741029019 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbbd466a30 0x55bbbd46ea10 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.378+0000 7f74b8272740 1 -- 172.21.15.28:0/2741029019 >> 172.21.15.28:0/2741029019 conn(0x55bbbd462d50 msgr2=0x55bbbd461fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.378+0000 7f748fe00640 0 client.26212 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:50:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.378+0000 7f74b8272740 1 -- 172.21.15.28:0/2741029019 shutdown_connections 2024-04-23T14:50:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.378+0000 7f74b8272740 1 -- 172.21.15.28:0/2741029019 wait complete. 2024-04-23T14:50:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.378+0000 7f74b8272740 20 client.26212 trim_cache size 0 max 16384 2024-04-23T14:50:39.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.400+0000 7f74b8272740 1 Processor -- start 2024-04-23T14:50:39.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.400+0000 7f74b8272740 1 -- start start 2024-04-23T14:50:39.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.400+0000 7f74b8272740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbbd466a30 0x55bbbd47d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:39.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.400+0000 7f74b8272740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bbbd467350 0x55bbbd489900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:39.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.400+0000 7f74b8272740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbbd471a20 0x55bbbd489e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:39.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.400+0000 7f74b8272740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f74b01ade70 con 0x55bbbd471a20 2024-04-23T14:50:39.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.400+0000 7f74b8272740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f74b01b06f0 con 0x55bbbd467350 2024-04-23T14:50:39.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.400+0000 7f74b8272740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f74b01ae4e0 con 0x55bbbd466a30 2024-04-23T14:50:39.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.401+0000 7f74ade00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bbbd467350 0x55bbbd489900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:39.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.401+0000 7f74ade00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bbbd467350 0x55bbbd489900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:52790/0 (socket says 172.21.15.28:52790) 2024-04-23T14:50:39.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.401+0000 7f74ade00640 1 -- 172.21.15.28:0/5558286 learned_addr learned my addr 172.21.15.28:0/5558286 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:39.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.401+0000 7f74ae800640 1 --2- 172.21.15.28:0/5558286 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbbd466a30 0x55bbbd47d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:39.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.401+0000 7f74af200640 1 --2- 172.21.15.28:0/5558286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbbd471a20 0x55bbbd489e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:39.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.401+0000 7f74ade00640 1 -- 172.21.15.28:0/5558286 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbbd466a30 msgr2=0x55bbbd47d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:39.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.401+0000 7f74ade00640 1 --2- 172.21.15.28:0/5558286 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbbd466a30 0x55bbbd47d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.401+0000 7f74ade00640 1 -- 172.21.15.28:0/5558286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbbd471a20 msgr2=0x55bbbd489e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:39.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.401+0000 7f74ade00640 1 --2- 172.21.15.28:0/5558286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbbd471a20 0x55bbbd489e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.401+0000 7f74ade00640 1 -- 172.21.15.28:0/5558286 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bbbd0de3f0 con 0x55bbbd467350 2024-04-23T14:50:39.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.401+0000 7f74af200640 1 --2- 172.21.15.28:0/5558286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbbd471a20 0x55bbbd489e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:50:39.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.401+0000 7f74ade00640 1 --2- 172.21.15.28:0/5558286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bbbd467350 0x55bbbd489900 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f749c0b8d90 tx=0x7f749c0a7040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fe46b3f1aeaee69 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:39.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.401+0000 7f749aa00640 1 -- 172.21.15.28:0/5558286 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749c0f2e60 con 0x55bbbd467350 2024-04-23T14:50:39.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.402+0000 7f749aa00640 1 -- 172.21.15.28:0/5558286 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f749c0bc930 con 0x55bbbd467350 2024-04-23T14:50:39.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.402+0000 7f749aa00640 1 -- 172.21.15.28:0/5558286 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749c049040 con 0x55bbbd467350 2024-04-23T14:50:39.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.402+0000 7f74b8272740 1 -- 172.21.15.28:0/5558286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bbbd467350 msgr2=0x55bbbd489900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:39.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.402+0000 7f74b8272740 1 --2- 172.21.15.28:0/5558286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bbbd467350 0x55bbbd489900 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f749c0b8d90 tx=0x7f749c0a7040 comp rx=0 tx=0).stop 2024-04-23T14:50:39.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.402+0000 7f74b8272740 1 -- 172.21.15.28:0/5558286 shutdown_connections 2024-04-23T14:50:39.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.402+0000 7f74b8272740 1 --2- 172.21.15.28:0/5558286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bbbd467350 0x55bbbd489900 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.402+0000 7f74b8272740 1 --2- 172.21.15.28:0/5558286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbbd471a20 0x55bbbd489e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.402+0000 7f74b8272740 1 --2- 172.21.15.28:0/5558286 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbbd466a30 0x55bbbd47d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.402+0000 7f74b8272740 1 -- 172.21.15.28:0/5558286 >> 172.21.15.28:0/5558286 conn(0x55bbbd462d50 msgr2=0x55bbbd461fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:39.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.403+0000 7f74b8272740 1 -- 172.21.15.28:0/5558286 shutdown_connections 2024-04-23T14:50:39.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.403+0000 7f74b8272740 1 -- 172.21.15.28:0/5558286 wait complete. 2024-04-23T14:50:39.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.403+0000 7f74b8272740 1 -- start start 2024-04-23T14:50:39.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.403+0000 7f74b8272740 1 Processor -- start 2024-04-23T14:50:39.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.404+0000 7f74b8272740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbbd466a30 0x55bbbd479c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:39.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.404+0000 7f74b8272740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbbd467350 0x55bbbd47a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:39.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.404+0000 7f74b8272740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bbbd471a20 0x55bbbd476be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:39.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.404+0000 7f74b8272740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f74b01aebb0 con 0x55bbbd466a30 2024-04-23T14:50:39.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.404+0000 7f74b8272740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f74b01b0020 con 0x55bbbd471a20 2024-04-23T14:50:39.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.404+0000 7f74b8272740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f74b01af280 con 0x55bbbd467350 2024-04-23T14:50:39.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.404+0000 7f74b8272740 10 client.0 _ll_register_callbacks cb 0x7f74a001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:50:39.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.404+0000 7f74ae800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbbd466a30 0x55bbbd479c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:39.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.404+0000 7f74ae800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbbd466a30 0x55bbbd479c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56880/0 (socket says 172.21.15.28:56880) 2024-04-23T14:50:39.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.404+0000 7f74ade00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbbd467350 0x55bbbd47a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:39.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.404+0000 7f74ae800640 1 -- 172.21.15.28:0/4188130444 learned_addr learned my addr 172.21.15.28:0/4188130444 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:39.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.404+0000 7f74af200640 1 --2- 172.21.15.28:0/4188130444 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bbbd471a20 0x55bbbd476be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:39.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.405+0000 7f74ade00640 1 -- 172.21.15.28:0/4188130444 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bbbd471a20 msgr2=0x55bbbd476be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:39.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.405+0000 7f74ade00640 1 --2- 172.21.15.28:0/4188130444 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bbbd471a20 0x55bbbd476be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.405+0000 7f74ade00640 1 -- 172.21.15.28:0/4188130444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbbd466a30 msgr2=0x55bbbd479c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:39.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.405+0000 7f74ade00640 1 --2- 172.21.15.28:0/4188130444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbbd466a30 0x55bbbd479c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.405+0000 7f74ade00640 1 -- 172.21.15.28:0/4188130444 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bbbd09e0f0 con 0x55bbbd467350 2024-04-23T14:50:39.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.405+0000 7f74ae800640 1 --2- 172.21.15.28:0/4188130444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbbd466a30 0x55bbbd479c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:50:39.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.405+0000 7f74ade00640 1 --2- 172.21.15.28:0/4188130444 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbbd467350 0x55bbbd47a150 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x55bbbd3d9c60 tx=0x7f749c0c0cc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ac08ce843c411cd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:39.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.405+0000 7f748fe00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:50:39.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.405+0000 7f748fe00640 1 -- 172.21.15.28:0/4188130444 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749c049040 con 0x55bbbd467350 2024-04-23T14:50:39.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.406+0000 7f748fe00640 1 -- 172.21.15.28:0/4188130444 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f749c032400 con 0x55bbbd467350 2024-04-23T14:50:39.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.406+0000 7f74b8272740 1 -- 172.21.15.28:0/4188130444 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f749c0bc930 con 0x55bbbd467350 2024-04-23T14:50:39.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.406+0000 7f748fe00640 1 -- 172.21.15.28:0/4188130444 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f749c0ee2b0 con 0x55bbbd467350 2024-04-23T14:50:39.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.406+0000 7f74b8272740 10 client.26218 fetch_fsmap learned FSMap version 20 2024-04-23T14:50:39.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.406+0000 7f74b8272740 1 -- 172.21.15.28:0/4188130444 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55bbbd391810 con 0x55bbbd467350 2024-04-23T14:50:39.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.406+0000 7f748fe00640 1 -- 172.21.15.28:0/4188130444 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f749c0bc930 con 0x55bbbd467350 2024-04-23T14:50:39.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.406+0000 7f748fe00640 1 -- 172.21.15.28:0/4188130444 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f749c04c060 con 0x55bbbd467350 2024-04-23T14:50:39.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.406+0000 7f74b8272740 10 client.26218 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:50:39.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.406+0000 7f74b8272740 10 client.26218 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:50:39.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.406+0000 7f74b8272740 20 client.26218 populate_metadata read hostname 'smithi028' 2024-04-23T14:50:39.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.407+0000 7f74b8272740 1 --2- 172.21.15.28:0/4188130444 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55bbbd4e8e80 0x55bbbd4eb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:39.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.407+0000 7f74b8272740 4 client.26218 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:50:39.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.407+0000 7f74b8272740 1 -- 172.21.15.28:0/4188130444 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bbbd3d3a80 con 0x55bbbd4e8e80 2024-04-23T14:50:39.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.407+0000 7f74ae800640 1 --2- 172.21.15.28:0/4188130444 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55bbbd4e8e80 0x55bbbd4eb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:39.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.407+0000 7f74ae800640 1 --2- 172.21.15.28:0/4188130444 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55bbbd4e8e80 0x55bbbd4eb2a0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e385a681290098c server_cookie=d508b42a08b8fd26 in_seq=0 out_seq=0 2024-04-23T14:50:39.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.407+0000 7f748fe00640 10 client.26218 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:50:39.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.409+0000 7f748fe00640 1 -- 172.21.15.28:0/4188130444 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55bbbd3d3a80 con 0x55bbbd4e8e80 2024-04-23T14:50:39.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.409+0000 7f748fe00640 10 client.26218 handle_command_reply: tid=0 2024-04-23T14:50:39.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.409+0000 7f74b8272740 1 client.26218 shutdown 2024-04-23T14:50:39.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.409+0000 7f74b8272740 1 -- 172.21.15.28:0/4188130444 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbbd467350 msgr2=0x55bbbd47a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:39.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.409+0000 7f74b8272740 1 --2- 172.21.15.28:0/4188130444 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbbd467350 0x55bbbd47a150 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x55bbbd3d9c60 tx=0x7f749c0c0cc0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.409+0000 7f74b8272740 1 -- 172.21.15.28:0/4188130444 shutdown_connections 2024-04-23T14:50:39.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.409+0000 7f74b8272740 1 --2- 172.21.15.28:0/4188130444 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55bbbd4e8e80 0x55bbbd4eb2a0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.409+0000 7f74b8272740 1 --2- 172.21.15.28:0/4188130444 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bbbd471a20 0x55bbbd476be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.409+0000 7f74b8272740 1 --2- 172.21.15.28:0/4188130444 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbbd467350 0x55bbbd47a150 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.409+0000 7f748fe00640 0 client.26218 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:50:39.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.409+0000 7f74b8272740 1 --2- 172.21.15.28:0/4188130444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbbd466a30 0x55bbbd479c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.409+0000 7f74b8272740 1 -- 172.21.15.28:0/4188130444 >> 172.21.15.28:0/4188130444 conn(0x55bbbd462d50 msgr2=0x55bbbd468180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:39.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.409+0000 7f74b8272740 1 -- 172.21.15.28:0/4188130444 shutdown_connections 2024-04-23T14:50:39.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.409+0000 7f74b8272740 1 -- 172.21.15.28:0/4188130444 wait complete. 2024-04-23T14:50:39.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.410+0000 7f74b8272740 20 client.26218 trim_cache size 0 max 16384 2024-04-23T14:50:39.414 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:50:39.414 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-23T14:50:39.414 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-23T14:50:39.414 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:50:39.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.410+0000 7f74b4a00640 1 -- 172.21.15.28:0/3552991015 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f74940a3da0 msgr2=0x7f74940a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:39.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.410+0000 7f74b4a00640 1 --2- 172.21.15.28:0/3552991015 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f74940a3da0 0x7f74940a6260 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f74a8005fd0 tx=0x7f74a8009500 comp rx=0 tx=0).stop 2024-04-23T14:50:39.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.410+0000 7f74b4a00640 1 -- 172.21.15.28:0/3552991015 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74b0100f30 msgr2=0x7f74b01a0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:39.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.410+0000 7f74b4a00640 1 --2- 172.21.15.28:0/3552991015 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74b0100f30 0x7f74b01a0bd0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f749c02bc80 tx=0x7f749c030bf0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.411+0000 7f74b4a00640 1 -- 172.21.15.28:0/3552991015 shutdown_connections 2024-04-23T14:50:39.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.411+0000 7f74b4a00640 1 --2- 172.21.15.28:0/3552991015 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f74940a3da0 0x7f74940a6260 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.411+0000 7f74b4a00640 1 --2- 172.21.15.28:0/3552991015 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74b0100f30 0x7f74b01a0bd0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.411+0000 7f74b4a00640 1 --2- 172.21.15.28:0/3552991015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74b01065e0 0x7f74b019a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.411+0000 7f74b4a00640 1 --2- 172.21.15.28:0/3552991015 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f74b0100570 0x7f74b01a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.411+0000 7f74b4a00640 1 -- 172.21.15.28:0/3552991015 >> 172.21.15.28:0/3552991015 conn(0x7f74b0077390 msgr2=0x7f74b010ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:39.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.411+0000 7f74b4a00640 1 -- 172.21.15.28:0/3552991015 shutdown_connections 2024-04-23T14:50:39.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.411+0000 7f74b4a00640 1 -- 172.21.15.28:0/3552991015 wait complete. 2024-04-23T14:50:39.430 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-23T14:50:39.431 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:50:39.498 INFO:tasks.workunit.client.0.smithi028.stdout: 1 537815 9.44 MB/sec execute 175 sec latency 18.259 ms 2024-04-23T14:50:39.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.635+0000 7feea9800640 1 -- 172.21.15.28:0/3418160776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feea410c5d0 msgr2=0x7feea410ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:39.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.635+0000 7feea9800640 1 --2- 172.21.15.28:0/3418160776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feea410c5d0 0x7feea410ea70 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fee9c009ca0 tx=0x7fee9c02a9e0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.635+0000 7feea9800640 1 -- 172.21.15.28:0/3418160776 shutdown_connections 2024-04-23T14:50:39.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.635+0000 7feea9800640 1 --2- 172.21.15.28:0/3418160776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feea410c5d0 0x7feea410ea70 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.635+0000 7feea9800640 1 --2- 172.21.15.28:0/3418160776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feea4103bb0 0x7feea410c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.635+0000 7feea9800640 1 --2- 172.21.15.28:0/3418160776 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feea4103290 0x7feea4103670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.635+0000 7feea9800640 1 -- 172.21.15.28:0/3418160776 >> 172.21.15.28:0/3418160776 conn(0x7feea4078000 msgr2=0x7feea4078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:39.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.636+0000 7feea9800640 1 -- 172.21.15.28:0/3418160776 shutdown_connections 2024-04-23T14:50:39.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.636+0000 7feea9800640 1 -- 172.21.15.28:0/3418160776 wait complete. 2024-04-23T14:50:39.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.637+0000 7feea9800640 1 Processor -- start 2024-04-23T14:50:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.637+0000 7feea9800640 1 -- start start 2024-04-23T14:50:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.637+0000 7feea9800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feea4103290 0x7feea4100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.637+0000 7feea9800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feea4103bb0 0x7feea4101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.637+0000 7feea9800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feea410c5d0 0x7feea4101960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.637+0000 7feea9800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7feea4112400 con 0x7feea4103bb0 2024-04-23T14:50:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.637+0000 7feea9800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7feea4112280 con 0x7feea410c5d0 2024-04-23T14:50:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.637+0000 7feea9800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7feea4112580 con 0x7feea4103290 2024-04-23T14:50:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.637+0000 7feea3400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feea410c5d0 0x7feea4101960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.637+0000 7feea2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feea4103bb0 0x7feea4101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.637+0000 7feea3400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feea410c5d0 0x7feea4101960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:52830/0 (socket says 172.21.15.28:52830) 2024-04-23T14:50:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.637+0000 7feea3400640 1 -- 172.21.15.28:0/2720753821 learned_addr learned my addr 172.21.15.28:0/2720753821 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.638+0000 7feea2a00640 1 --2- 172.21.15.28:0/2720753821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feea4103290 0x7feea4100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.638+0000 7feea3400640 1 -- 172.21.15.28:0/2720753821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feea4103290 msgr2=0x7feea4100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.638+0000 7feea3400640 1 --2- 172.21.15.28:0/2720753821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feea4103290 0x7feea4100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.638+0000 7feea3400640 1 -- 172.21.15.28:0/2720753821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feea4103bb0 msgr2=0x7feea4101420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.638+0000 7feea3400640 1 --2- 172.21.15.28:0/2720753821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feea4103bb0 0x7feea4101420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.638+0000 7feea3400640 1 -- 172.21.15.28:0/2720753821 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feea41a6ca0 con 0x7feea410c5d0 2024-04-23T14:50:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.638+0000 7feea3400640 1 --2- 172.21.15.28:0/2720753821 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feea410c5d0 0x7feea4101960 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fee9c004af0 tx=0x7fee9c003f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.638+0000 7fee9b400640 1 -- 172.21.15.28:0/2720753821 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee9c047070 con 0x7feea410c5d0 2024-04-23T14:50:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.638+0000 7fee9b400640 1 -- 172.21.15.28:0/2720753821 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee9c02e870 con 0x7feea410c5d0 2024-04-23T14:50:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.639+0000 7feea9800640 1 -- 172.21.15.28:0/2720753821 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feea41a6f30 con 0x7feea410c5d0 2024-04-23T14:50:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.639+0000 7fee9b400640 1 -- 172.21.15.28:0/2720753821 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee9c033050 con 0x7feea410c5d0 2024-04-23T14:50:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.639+0000 7feea9800640 1 -- 172.21.15.28:0/2720753821 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feea41a7440 con 0x7feea410c5d0 2024-04-23T14:50:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.640+0000 7feea9800640 1 -- 172.21.15.28:0/2720753821 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feea4068e20 con 0x7feea410c5d0 2024-04-23T14:50:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.642+0000 7fee9b400640 1 -- 172.21.15.28:0/2720753821 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fee9c04a070 con 0x7feea410c5d0 2024-04-23T14:50:39.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.643+0000 7fee9b400640 1 --2- 172.21.15.28:0/2720753821 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fee880a3dd0 0x7fee880a6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:39.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.643+0000 7fee9b400640 1 -- 172.21.15.28:0/2720753821 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fee9c0f1280 con 0x7feea410c5d0 2024-04-23T14:50:39.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.643+0000 7feea2a00640 1 --2- 172.21.15.28:0/2720753821 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fee880a3dd0 0x7fee880a6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.644+0000 7feea2a00640 1 --2- 172.21.15.28:0/2720753821 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fee880a3dd0 0x7fee880a6290 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fee94009bd0 tx=0x7fee94005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:39.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.647+0000 7fee9b400640 1 -- 172.21.15.28:0/2720753821 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fee9c07eff0 con 0x7feea410c5d0 2024-04-23T14:50:39.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.956+0000 7feea9800640 1 -- 172.21.15.28:0/2720753821 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feea41a7670 con 0x7feea410c5d0 2024-04-23T14:50:39.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.957+0000 7fee9b400640 1 -- 172.21.15.28:0/2720753821 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7fee9c0bd8d0 con 0x7feea410c5d0 2024-04-23T14:50:39.958 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:50:39.959 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:50:39.959 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:50:39.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.961+0000 7feea9800640 1 -- 172.21.15.28:0/2720753821 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fee880a3dd0 msgr2=0x7fee880a6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:39.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.961+0000 7feea9800640 1 --2- 172.21.15.28:0/2720753821 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fee880a3dd0 0x7fee880a6290 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fee94009bd0 tx=0x7fee94005f90 comp rx=0 tx=0).stop 2024-04-23T14:50:39.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.961+0000 7feea9800640 1 -- 172.21.15.28:0/2720753821 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feea410c5d0 msgr2=0x7feea4101960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:39.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.961+0000 7feea9800640 1 --2- 172.21.15.28:0/2720753821 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feea410c5d0 0x7feea4101960 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fee9c004af0 tx=0x7fee9c003f30 comp rx=0 tx=0).stop 2024-04-23T14:50:39.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.961+0000 7feea9800640 1 -- 172.21.15.28:0/2720753821 shutdown_connections 2024-04-23T14:50:39.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.961+0000 7feea9800640 1 --2- 172.21.15.28:0/2720753821 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fee880a3dd0 0x7fee880a6290 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.961+0000 7feea9800640 1 --2- 172.21.15.28:0/2720753821 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feea410c5d0 0x7feea4101960 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.961+0000 7feea9800640 1 --2- 172.21.15.28:0/2720753821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feea4103bb0 0x7feea4101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.961+0000 7feea9800640 1 --2- 172.21.15.28:0/2720753821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feea4103290 0x7feea4100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:39.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.961+0000 7feea9800640 1 -- 172.21.15.28:0/2720753821 >> 172.21.15.28:0/2720753821 conn(0x7feea4078000 msgr2=0x7feea40fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:39.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.961+0000 7feea9800640 1 -- 172.21.15.28:0/2720753821 shutdown_connections 2024-04-23T14:50:39.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:39.962+0000 7feea9800640 1 -- 172.21.15.28:0/2720753821 wait complete. 2024-04-23T14:50:39.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-23T14:50:40.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.191+0000 7fc0cfa00640 1 -- 172.21.15.28:0/1381760525 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0c8104e60 msgr2=0x7fc0c8108ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:40.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.191+0000 7fc0cfa00640 1 --2- 172.21.15.28:0/1381760525 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0c8104e60 0x7fc0c8108ea0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc0b0009900 tx=0x7fc0b002a3f0 comp rx=0 tx=0).stop 2024-04-23T14:50:40.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.191+0000 7fc0cfa00640 1 -- 172.21.15.28:0/1381760525 shutdown_connections 2024-04-23T14:50:40.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.191+0000 7fc0cfa00640 1 --2- 172.21.15.28:0/1381760525 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0c8104e60 0x7fc0c8108ea0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:40.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.191+0000 7fc0cfa00640 1 --2- 172.21.15.28:0/1381760525 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc0c8109580 0x7fc0c8111150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:40.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.191+0000 7fc0cfa00640 1 --2- 172.21.15.28:0/1381760525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c81044b0 0x7fc0c8104890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:40.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.191+0000 7fc0cfa00640 1 -- 172.21.15.28:0/1381760525 >> 172.21.15.28:0/1381760525 conn(0x7fc0c80ffd90 msgr2=0x7fc0c81021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:40.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.191+0000 7fc0cfa00640 1 -- 172.21.15.28:0/1381760525 shutdown_connections 2024-04-23T14:50:40.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.191+0000 7fc0cfa00640 1 -- 172.21.15.28:0/1381760525 wait complete. 2024-04-23T14:50:40.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.192+0000 7fc0cfa00640 1 Processor -- start 2024-04-23T14:50:40.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.192+0000 7fc0cfa00640 1 -- start start 2024-04-23T14:50:40.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.193+0000 7fc0cfa00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc0c81044b0 0x7fc0c81a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:40.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.193+0000 7fc0cfa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0c8109580 0x7fc0c81a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:40.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.193+0000 7fc0cfa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c819a9c0 0x7fc0c819ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:40.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.193+0000 7fc0cfa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c8114640 con 0x7fc0c819a9c0 2024-04-23T14:50:40.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.193+0000 7fc0cfa00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c81144c0 con 0x7fc0c8109580 2024-04-23T14:50:40.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.193+0000 7fc0cfa00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c81147c0 con 0x7fc0c81044b0 2024-04-23T14:50:40.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.193+0000 7fc0bfe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0c8109580 0x7fc0c81a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:40.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.193+0000 7fc0cd000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc0c81044b0 0x7fc0c81a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:40.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.193+0000 7fc0bfe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0c8109580 0x7fc0c81a0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:52844/0 (socket says 172.21.15.28:52844) 2024-04-23T14:50:40.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.193+0000 7fc0cda00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c819a9c0 0x7fc0c819ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:40.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.193+0000 7fc0bfe00640 1 -- 172.21.15.28:0/3231667903 learned_addr learned my addr 172.21.15.28:0/3231667903 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:40.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.193+0000 7fc0cd000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc0c81044b0 0x7fc0c81a0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:55714/0 (socket says 172.21.15.28:55714) 2024-04-23T14:50:40.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.193+0000 7fc0bfe00640 1 -- 172.21.15.28:0/3231667903 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc0c81044b0 msgr2=0x7fc0c81a0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:40.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.193+0000 7fc0bfe00640 1 --2- 172.21.15.28:0/3231667903 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc0c81044b0 0x7fc0c81a0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:40.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.193+0000 7fc0bfe00640 1 -- 172.21.15.28:0/3231667903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c819a9c0 msgr2=0x7fc0c819ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:40.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.194+0000 7fc0bfe00640 1 --2- 172.21.15.28:0/3231667903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c819a9c0 0x7fc0c819ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:40.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.194+0000 7fc0bfe00640 1 -- 172.21.15.28:0/3231667903 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0c819b4e0 con 0x7fc0c8109580 2024-04-23T14:50:40.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.194+0000 7fc0bfe00640 1 --2- 172.21.15.28:0/3231667903 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0c8109580 0x7fc0c81a0d50 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc0b002a910 tx=0x7fc0b0003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:40.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.194+0000 7fc0c6000640 1 -- 172.21.15.28:0/3231667903 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b003f070 con 0x7fc0c8109580 2024-04-23T14:50:40.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.194+0000 7fc0c6000640 1 -- 172.21.15.28:0/3231667903 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0b0031c90 con 0x7fc0c8109580 2024-04-23T14:50:40.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.194+0000 7fc0c6000640 1 -- 172.21.15.28:0/3231667903 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b0035030 con 0x7fc0c8109580 2024-04-23T14:50:40.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.194+0000 7fc0cfa00640 1 -- 172.21.15.28:0/3231667903 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0c819b770 con 0x7fc0c8109580 2024-04-23T14:50:40.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.195+0000 7fc0cfa00640 1 -- 172.21.15.28:0/3231667903 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0c81a79c0 con 0x7fc0c8109580 2024-04-23T14:50:40.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.198+0000 7fc0c6000640 1 -- 172.21.15.28:0/3231667903 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fc0b0042070 con 0x7fc0c8109580 2024-04-23T14:50:40.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.198+0000 7fc0d29d5740 1 Processor -- start 2024-04-23T14:50:40.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.198+0000 7fc0d29d5740 1 -- start start 2024-04-23T14:50:40.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.199+0000 7fc0d29d5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585043d7950 0x5585043d7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:40.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.199+0000 7fc0d29d5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585043d8270 0x5585043e2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:40.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.199+0000 7fc0d29d5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585043e2940 0x5585043e4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:40.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.199+0000 7fc0d29d5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55850433d1d0 con 0x5585043d8270 2024-04-23T14:50:40.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.199+0000 7fc0d29d5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5585042e3450 con 0x5585043e2940 2024-04-23T14:50:40.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.199+0000 7fc0d29d5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558504339190 con 0x5585043d7950 2024-04-23T14:50:40.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.199+0000 7fc0bfe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585043e2940 0x5585043e4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:40.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.199+0000 7fc0cd000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585043d8270 0x5585043e2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:40.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.199+0000 7fc0cda00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585043d7950 0x5585043d7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:40.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.199+0000 7fc0bfe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585043e2940 0x5585043e4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:52848/0 (socket says 172.21.15.28:52848) 2024-04-23T14:50:40.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.199+0000 7fc0bfe00640 1 -- 172.21.15.28:0/791867808 learned_addr learned my addr 172.21.15.28:0/791867808 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:40.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.200+0000 7fc0bfe00640 1 -- 172.21.15.28:0/791867808 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585043d7950 msgr2=0x5585043d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:40.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.200+0000 7fc0bfe00640 1 --2- 172.21.15.28:0/791867808 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585043d7950 0x5585043d7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:40.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.200+0000 7fc0bfe00640 1 -- 172.21.15.28:0/791867808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585043d8270 msgr2=0x5585043e2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:40.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.200+0000 7fc0bfe00640 1 --2- 172.21.15.28:0/791867808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585043d8270 0x5585043e2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:40.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.200+0000 7fc0bfe00640 1 -- 172.21.15.28:0/791867808 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585043a0310 con 0x5585043e2940 2024-04-23T14:50:40.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.200+0000 7fc0bfe00640 1 --2- 172.21.15.28:0/791867808 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585043e2940 0x5585043e4d30 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc0b00ee440 tx=0x7fc0b00f7470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23535089584c0a22 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:40.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.200+0000 7fc0c6000640 1 --2- 172.21.15.28:0/3231667903 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc0ac0a3da0 0x7fc0ac0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:40.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.200+0000 7fc0be000640 1 -- 172.21.15.28:0/791867808 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b00f70d0 con 0x5585043e2940 2024-04-23T14:50:40.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.200+0000 7fc0be000640 1 -- 172.21.15.28:0/791867808 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0b00bd450 con 0x5585043e2940 2024-04-23T14:50:40.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.200+0000 7fc0be000640 1 -- 172.21.15.28:0/791867808 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b00968d0 con 0x5585043e2940 2024-04-23T14:50:40.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.201+0000 7fc0c6000640 1 -- 172.21.15.28:0/3231667903 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fc0b00f0e00 con 0x7fc0c8109580 2024-04-23T14:50:40.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.201+0000 7fc0cda00640 1 --2- 172.21.15.28:0/3231667903 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc0ac0a3da0 0x7fc0ac0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:40.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.201+0000 7fc0d29d5740 1 -- 172.21.15.28:0/791867808 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585043e2940 msgr2=0x5585043e4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:40.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.201+0000 7fc0d29d5740 1 --2- 172.21.15.28:0/791867808 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585043e2940 0x5585043e4d30 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc0b00ee440 tx=0x7fc0b00f7470 comp rx=0 tx=0).stop 2024-04-23T14:50:40.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.202+0000 7fc0cda00640 1 --2- 172.21.15.28:0/3231667903 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc0ac0a3da0 0x7fc0ac0a6260 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0005fd0 tx=0x7fc0c00096a0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:40.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.202+0000 7fc0d29d5740 1 -- 172.21.15.28:0/791867808 shutdown_connections 2024-04-23T14:50:40.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.202+0000 7fc0d29d5740 1 --2- 172.21.15.28:0/791867808 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585043e2940 0x5585043e4d30 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:40.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.202+0000 7fc0d29d5740 1 --2- 172.21.15.28:0/791867808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585043d8270 0x5585043e2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:40.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.202+0000 7fc0d29d5740 1 --2- 172.21.15.28:0/791867808 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585043d7950 0x5585043d7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:40.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.202+0000 7fc0d29d5740 1 -- 172.21.15.28:0/791867808 >> 172.21.15.28:0/791867808 conn(0x5585043d3c70 msgr2=0x5585043d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:40.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.202+0000 7fc0d29d5740 1 -- 172.21.15.28:0/791867808 shutdown_connections 2024-04-23T14:50:40.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.202+0000 7fc0d29d5740 1 -- 172.21.15.28:0/791867808 wait complete. 2024-04-23T14:50:40.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.203+0000 7fc0d29d5740 1 -- start start 2024-04-23T14:50:40.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.203+0000 7fc0d29d5740 1 Processor -- start 2024-04-23T14:50:40.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.203+0000 7fc0d29d5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585043d7950 0x5585043e2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:40.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.203+0000 7fc0d29d5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585043d8270 0x5585043df560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:40.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.203+0000 7fc0d29d5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585043e2940 0x5585043dfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:40.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.203+0000 7fc0d29d5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5585043e54f0 con 0x5585043e2940 2024-04-23T14:50:40.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.204+0000 7fc0d29d5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5585042f6810 con 0x5585043d8270 2024-04-23T14:50:40.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.204+0000 7fc0d29d5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5585043e5670 con 0x5585043d7950 2024-04-23T14:50:40.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.204+0000 7fc0d29d5740 10 client.0 _ll_register_callbacks cb 0x5585042c4140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:50:40.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.204+0000 7fc0bfe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585043d8270 0x5585043df560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:40.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.204+0000 7fc0bfe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585043d8270 0x5585043df560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:52858/0 (socket says 172.21.15.28:52858) 2024-04-23T14:50:40.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.204+0000 7fc0bfe00640 1 -- 172.21.15.28:0/3443168827 learned_addr learned my addr 172.21.15.28:0/3443168827 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:40.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.204+0000 7fc0cd000640 1 --2- 172.21.15.28:0/3443168827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585043d7950 0x5585043e2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:40.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.204+0000 7fc0cda00640 1 --2- 172.21.15.28:0/3443168827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585043e2940 0x5585043dfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:40.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.204+0000 7fc0bfe00640 1 -- 172.21.15.28:0/3443168827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585043d7950 msgr2=0x5585043e2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:40.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.204+0000 7fc0bfe00640 1 --2- 172.21.15.28:0/3443168827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585043d7950 0x5585043e2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:40.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.204+0000 7fc0bfe00640 1 -- 172.21.15.28:0/3443168827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585043e2940 msgr2=0x5585043dfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:40.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.204+0000 7fc0bfe00640 1 --2- 172.21.15.28:0/3443168827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585043e2940 0x5585043dfaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:40.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.204+0000 7fc0bfe00640 1 -- 172.21.15.28:0/3443168827 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585043073f0 con 0x5585043d8270 2024-04-23T14:50:40.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.204+0000 7fc0bfe00640 1 --2- 172.21.15.28:0/3443168827 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585043d8270 0x5585043df560 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fc0b00f7970 tx=0x7fc0b0005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ea48db27015aaf5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:40.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.204+0000 7fc0b6a00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:50:40.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.205+0000 7fc0b6a00640 1 -- 172.21.15.28:0/3443168827 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b0095780 con 0x5585043d8270 2024-04-23T14:50:40.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.205+0000 7fc0b6a00640 1 -- 172.21.15.28:0/3443168827 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fc0b00bb050 con 0x5585043d8270 2024-04-23T14:50:40.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.205+0000 7fc0d29d5740 1 -- 172.21.15.28:0/3443168827 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0b00bd450 con 0x5585043d8270 2024-04-23T14:50:40.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.205+0000 7fc0b6a00640 1 -- 172.21.15.28:0/3443168827 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fc0b0095970 con 0x5585043d8270 2024-04-23T14:50:40.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.205+0000 7fc0d29d5740 10 client.26252 fetch_fsmap learned FSMap version 20 2024-04-23T14:50:40.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.205+0000 7fc0d29d5740 1 -- 172.21.15.28:0/3443168827 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55850404f3f0 con 0x5585043d8270 2024-04-23T14:50:40.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.205+0000 7fc0b6a00640 1 -- 172.21.15.28:0/3443168827 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0b00bd450 con 0x5585043d8270 2024-04-23T14:50:40.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.205+0000 7fc0b6a00640 1 -- 172.21.15.28:0/3443168827 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fc0b00b93c0 con 0x5585043d8270 2024-04-23T14:50:40.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.205+0000 7fc0d29d5740 10 client.26252 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:50:40.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.205+0000 7fc0d29d5740 10 client.26252 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:50:40.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.205+0000 7fc0d29d5740 20 client.26252 populate_metadata read hostname 'smithi028' 2024-04-23T14:50:40.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.206+0000 7fc0d29d5740 1 --2- 172.21.15.28:0/3443168827 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5585043f25f0 0x5585043f49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:40.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.206+0000 7fc0d29d5740 4 client.26252 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:50:40.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.206+0000 7fc0d29d5740 1 -- 172.21.15.28:0/3443168827 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55850434abf0 con 0x5585043f25f0 2024-04-23T14:50:40.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.206+0000 7fc0cd000640 1 --2- 172.21.15.28:0/3443168827 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5585043f25f0 0x5585043f49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:40.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.206+0000 7fc0cd000640 1 --2- 172.21.15.28:0/3443168827 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5585043f25f0 0x5585043f49e0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=727e91025ccbefe9 server_cookie=fc0189122199eb7b in_seq=0 out_seq=0 2024-04-23T14:50:40.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.206+0000 7fc0b6a00640 10 client.26252 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:50:40.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.208+0000 7fc0b6a00640 1 -- 172.21.15.28:0/3443168827 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55850434abf0 con 0x5585043f25f0 2024-04-23T14:50:40.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.208+0000 7fc0b6a00640 10 client.26252 handle_command_reply: tid=0 2024-04-23T14:50:40.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.208+0000 7fc0d29d5740 1 client.26252 shutdown 2024-04-23T14:50:40.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.208+0000 7fc0d29d5740 1 -- 172.21.15.28:0/3443168827 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585043d8270 msgr2=0x5585043df560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:40.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.208+0000 7fc0d29d5740 1 --2- 172.21.15.28:0/3443168827 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585043d8270 0x5585043df560 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fc0b00f7970 tx=0x7fc0b0005dc0 comp rx=0 tx=0).stop 2024-04-23T14:50:40.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.208+0000 7fc0d29d5740 1 -- 172.21.15.28:0/3443168827 shutdown_connections 2024-04-23T14:50:40.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.208+0000 7fc0d29d5740 1 --2- 172.21.15.28:0/3443168827 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5585043f25f0 0x5585043f49e0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:40.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.208+0000 7fc0d29d5740 1 --2- 172.21.15.28:0/3443168827 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585043d8270 0x5585043df560 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:40.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.208+0000 7fc0d29d5740 1 --2- 172.21.15.28:0/3443168827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585043e2940 0x5585043dfaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:40.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.208+0000 7fc0d29d5740 1 --2- 172.21.15.28:0/3443168827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585043d7950 0x5585043e2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:40.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.208+0000 7fc0d29d5740 1 -- 172.21.15.28:0/3443168827 >> 172.21.15.28:0/3443168827 conn(0x5585043d3c70 msgr2=0x5585043d2df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:40.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.208+0000 7fc0b6a00640 0 client.26252 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:50:40.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.208+0000 7fc0d29d5740 1 -- 172.21.15.28:0/3443168827 shutdown_connections 2024-04-23T14:50:40.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.208+0000 7fc0d29d5740 1 -- 172.21.15.28:0/3443168827 wait complete. 2024-04-23T14:50:40.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.209+0000 7fc0d29d5740 20 client.26252 trim_cache size 0 max 16384 2024-04-23T14:50:40.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.226+0000 7fc0d29d5740 1 Processor -- start 2024-04-23T14:50:40.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.226+0000 7fc0d29d5740 1 -- start start 2024-04-23T14:50:40.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.226+0000 7fc0d29d5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585043d7950 0x5585043d3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:40.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.227+0000 7fc0d29d5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585043d8270 0x5585043e1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:40.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.227+0000 7fc0d29d5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585043e2940 0x5585043fa1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:40.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.227+0000 7fc0d29d5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c81ae020 con 0x5585043d7950 2024-04-23T14:50:40.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.227+0000 7fc0d29d5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c81b0880 con 0x5585043e2940 2024-04-23T14:50:40.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.227+0000 7fc0d29d5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c81ae670 con 0x5585043d8270 2024-04-23T14:50:40.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.227+0000 7fc0cda00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585043e2940 0x5585043fa1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:40.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.227+0000 7fc0cd000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585043d7950 0x5585043d3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:40.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.227+0000 7fc0cda00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585043e2940 0x5585043fa1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:52862/0 (socket says 172.21.15.28:52862) 2024-04-23T14:50:40.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.227+0000 7fc0bfe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585043d8270 0x5585043e1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:40.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.227+0000 7fc0cda00640 1 -- 172.21.15.28:0/816502099 learned_addr learned my addr 172.21.15.28:0/816502099 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:40.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.227+0000 7fc0bfe00640 1 -- 172.21.15.28:0/816502099 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585043e2940 msgr2=0x5585043fa1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:40.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.227+0000 7fc0bfe00640 1 --2- 172.21.15.28:0/816502099 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585043e2940 0x5585043fa1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:40.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.227+0000 7fc0bfe00640 1 -- 172.21.15.28:0/816502099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585043d7950 msgr2=0x5585043d3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:40.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.227+0000 7fc0bfe00640 1 --2- 172.21.15.28:0/816502099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585043d7950 0x5585043d3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:40.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.227+0000 7fc0bfe00640 1 -- 172.21.15.28:0/816502099 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585041197d0 con 0x5585043d8270 2024-04-23T14:50:40.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.228+0000 7fc0bfe00640 1 --2- 172.21.15.28:0/816502099 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585043d8270 0x5585043e1da0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc0b00bca50 tx=0x7fc0b0009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8e3cd4c43eaec94 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:40.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.228+0000 7fc0bd600640 1 -- 172.21.15.28:0/816502099 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b0095b60 con 0x5585043d8270 2024-04-23T14:50:40.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.228+0000 7fc0bd600640 1 -- 172.21.15.28:0/816502099 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0b00f4380 con 0x5585043d8270 2024-04-23T14:50:40.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.228+0000 7fc0bd600640 1 -- 172.21.15.28:0/816502099 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b0095d50 con 0x5585043d8270 2024-04-23T14:50:40.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.229+0000 7fc0d29d5740 1 -- 172.21.15.28:0/816502099 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585043d8270 msgr2=0x5585043e1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:40.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.229+0000 7fc0d29d5740 1 --2- 172.21.15.28:0/816502099 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585043d8270 0x5585043e1da0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc0b00bca50 tx=0x7fc0b0009260 comp rx=0 tx=0).stop 2024-04-23T14:50:40.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.229+0000 7fc0d29d5740 1 -- 172.21.15.28:0/816502099 shutdown_connections 2024-04-23T14:50:40.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.229+0000 7fc0d29d5740 1 --2- 172.21.15.28:0/816502099 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585043e2940 0x5585043fa1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:40.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.229+0000 7fc0d29d5740 1 --2- 172.21.15.28:0/816502099 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585043d8270 0x5585043e1da0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:40.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.229+0000 7fc0d29d5740 1 --2- 172.21.15.28:0/816502099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585043d7950 0x5585043d3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:40.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.229+0000 7fc0d29d5740 1 -- 172.21.15.28:0/816502099 >> 172.21.15.28:0/816502099 conn(0x5585043d3c70 msgr2=0x5585043da800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:40.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.229+0000 7fc0d29d5740 1 -- 172.21.15.28:0/816502099 shutdown_connections 2024-04-23T14:50:40.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.229+0000 7fc0d29d5740 1 -- 172.21.15.28:0/816502099 wait complete. 2024-04-23T14:50:40.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.230+0000 7fc0d29d5740 1 -- start start 2024-04-23T14:50:40.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.230+0000 7fc0d29d5740 1 Processor -- start 2024-04-23T14:50:40.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.230+0000 7fc0d29d5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585043d7950 0x5585043dc290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:40.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.230+0000 7fc0d29d5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585043d8270 0x5585043dc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:40.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.230+0000 7fc0d29d5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585043e2940 0x5585043dcd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:40.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.230+0000 7fc0d29d5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c81aed40 con 0x5585043e2940 2024-04-23T14:50:40.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.230+0000 7fc0d29d5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c81b01b0 con 0x5585043d8270 2024-04-23T14:50:40.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.230+0000 7fc0d29d5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c81af410 con 0x5585043d7950 2024-04-23T14:50:40.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.231+0000 7fc0d29d5740 10 client.0 _ll_register_callbacks cb 0x7fc0b80097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:50:40.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.231+0000 7fc0bfe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585043d8270 0x5585043dc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:40.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.231+0000 7fc0bfe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585043d8270 0x5585043dc7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:52870/0 (socket says 172.21.15.28:52870) 2024-04-23T14:50:40.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.231+0000 7fc0bfe00640 1 -- 172.21.15.28:0/2167544175 learned_addr learned my addr 172.21.15.28:0/2167544175 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:40.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.231+0000 7fc0cd000640 1 --2- 172.21.15.28:0/2167544175 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585043d7950 0x5585043dc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:40.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.231+0000 7fc0cda00640 1 --2- 172.21.15.28:0/2167544175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585043e2940 0x5585043dcd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:40.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.231+0000 7fc0bfe00640 1 -- 172.21.15.28:0/2167544175 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585043d7950 msgr2=0x5585043dc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:40.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.231+0000 7fc0bfe00640 1 --2- 172.21.15.28:0/2167544175 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585043d7950 0x5585043dc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:40.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.231+0000 7fc0bfe00640 1 -- 172.21.15.28:0/2167544175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585043e2940 msgr2=0x5585043dcd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:40.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.231+0000 7fc0bfe00640 1 --2- 172.21.15.28:0/2167544175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585043e2940 0x5585043dcd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:40.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.231+0000 7fc0bfe00640 1 -- 172.21.15.28:0/2167544175 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558504457b60 con 0x5585043d8270 2024-04-23T14:50:40.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.231+0000 7fc0cd000640 1 --2- 172.21.15.28:0/2167544175 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585043d7950 0x5585043dc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:50:40.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.231+0000 7fc0bfe00640 1 --2- 172.21.15.28:0/2167544175 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585043d8270 0x5585043dc7d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc0b00f4ec0 tx=0x7fc0b0005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=983640f5c2ef10c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:40.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.231+0000 7fc0b6a00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:50:40.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.232+0000 7fc0b6a00640 1 -- 172.21.15.28:0/2167544175 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b0095f40 con 0x5585043d8270 2024-04-23T14:50:40.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.232+0000 7fc0d29d5740 1 -- 172.21.15.28:0/2167544175 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558504344a60 con 0x5585043d8270 2024-04-23T14:50:40.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.232+0000 7fc0b6a00640 1 -- 172.21.15.28:0/2167544175 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fc0b00ee780 con 0x5585043d8270 2024-04-23T14:50:40.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.232+0000 7fc0b6a00640 1 -- 172.21.15.28:0/2167544175 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fc0b00964d0 con 0x5585043d8270 2024-04-23T14:50:40.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.232+0000 7fc0d29d5740 10 client.26258 fetch_fsmap learned FSMap version 20 2024-04-23T14:50:40.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.232+0000 7fc0d29d5740 1 -- 172.21.15.28:0/2167544175 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x558504457b60 con 0x5585043d8270 2024-04-23T14:50:40.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.232+0000 7fc0b6a00640 1 -- 172.21.15.28:0/2167544175 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558504344a60 con 0x5585043d8270 2024-04-23T14:50:40.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.232+0000 7fc0b6a00640 1 -- 172.21.15.28:0/2167544175 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fc0b00960e0 con 0x5585043d8270 2024-04-23T14:50:40.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.233+0000 7fc0d29d5740 10 client.26258 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:50:40.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.233+0000 7fc0d29d5740 10 client.26258 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:50:40.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.233+0000 7fc0d29d5740 20 client.26258 populate_metadata read hostname 'smithi028' 2024-04-23T14:50:40.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.233+0000 7fc0d29d5740 1 --2- 172.21.15.28:0/2167544175 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558504459180 0x55850445b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:40.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.233+0000 7fc0d29d5740 4 client.26258 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-23T14:50:40.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.233+0000 7fc0d29d5740 1 -- 172.21.15.28:0/2167544175 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5585042c04d0 con 0x558504459180 2024-04-23T14:50:40.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.233+0000 7fc0cd000640 1 --2- 172.21.15.28:0/2167544175 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558504459180 0x55850445b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:40.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.234+0000 7fc0cd000640 1 --2- 172.21.15.28:0/2167544175 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558504459180 0x55850445b5a0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17456f4bc206697e server_cookie=63fc1aa7e88519b8 in_seq=0 out_seq=0 2024-04-23T14:50:40.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.234+0000 7fc0b6a00640 10 client.26258 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:50:40.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.234+0000 7fc0b6a00640 1 -- 172.21.15.28:0/2167544175 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5585042c04d0 con 0x558504459180 2024-04-23T14:50:40.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.234+0000 7fc0b6a00640 10 client.26258 handle_command_reply: tid=0 2024-04-23T14:50:40.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.234+0000 7fc0d29d5740 1 client.26258 shutdown 2024-04-23T14:50:40.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.234+0000 7fc0d29d5740 1 -- 172.21.15.28:0/2167544175 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585043d8270 msgr2=0x5585043dc7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:40.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.234+0000 7fc0d29d5740 1 --2- 172.21.15.28:0/2167544175 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585043d8270 0x5585043dc7d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc0b00f4ec0 tx=0x7fc0b0005dc0 comp rx=0 tx=0).stop 2024-04-23T14:50:40.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.234+0000 7fc0d29d5740 1 -- 172.21.15.28:0/2167544175 shutdown_connections 2024-04-23T14:50:40.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.234+0000 7fc0d29d5740 1 --2- 172.21.15.28:0/2167544175 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558504459180 0x55850445b5a0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:40.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.234+0000 7fc0d29d5740 1 --2- 172.21.15.28:0/2167544175 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5585043d8270 0x5585043dc7d0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:40.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.234+0000 7fc0d29d5740 1 --2- 172.21.15.28:0/2167544175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585043e2940 0x5585043dcd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:40.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.234+0000 7fc0b6a00640 0 client.26258 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:50:40.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.234+0000 7fc0d29d5740 1 --2- 172.21.15.28:0/2167544175 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5585043d7950 0x5585043dc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:40.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.234+0000 7fc0d29d5740 1 -- 172.21.15.28:0/2167544175 >> 172.21.15.28:0/2167544175 conn(0x5585043d3c70 msgr2=0x5585043d90e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:40.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.235+0000 7fc0d29d5740 1 -- 172.21.15.28:0/2167544175 shutdown_connections 2024-04-23T14:50:40.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.235+0000 7fc0d29d5740 1 -- 172.21.15.28:0/2167544175 wait complete. 2024-04-23T14:50:40.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.235+0000 7fc0d29d5740 20 client.26258 trim_cache size 0 max 16384 2024-04-23T14:50:40.239 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-23T14:50:40.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.235+0000 7fc0cfa00640 1 -- 172.21.15.28:0/3231667903 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc0ac0a3da0 msgr2=0x7fc0ac0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:40.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.235+0000 7fc0cfa00640 1 --2- 172.21.15.28:0/3231667903 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc0ac0a3da0 0x7fc0ac0a6260 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0005fd0 tx=0x7fc0c00096a0 comp rx=0 tx=0).stop 2024-04-23T14:50:40.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.235+0000 7fc0cfa00640 1 -- 172.21.15.28:0/3231667903 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0c8109580 msgr2=0x7fc0c81a0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:40.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.235+0000 7fc0cfa00640 1 --2- 172.21.15.28:0/3231667903 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0c8109580 0x7fc0c81a0d50 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc0b002a910 tx=0x7fc0b0003b90 comp rx=0 tx=0).stop 2024-04-23T14:50:40.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.235+0000 7fc0cfa00640 1 -- 172.21.15.28:0/3231667903 shutdown_connections 2024-04-23T14:50:40.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.235+0000 7fc0cfa00640 1 --2- 172.21.15.28:0/3231667903 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc0ac0a3da0 0x7fc0ac0a6260 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:40.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.235+0000 7fc0cfa00640 1 --2- 172.21.15.28:0/3231667903 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0c8109580 0x7fc0c81a0d50 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:40.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.235+0000 7fc0cfa00640 1 --2- 172.21.15.28:0/3231667903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c819a9c0 0x7fc0c819ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:40.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.235+0000 7fc0cfa00640 1 --2- 172.21.15.28:0/3231667903 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc0c81044b0 0x7fc0c81a0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:40.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.235+0000 7fc0cfa00640 1 -- 172.21.15.28:0/3231667903 >> 172.21.15.28:0/3231667903 conn(0x7fc0c80ffd90 msgr2=0x7fc0c81020f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:40.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.236+0000 7fc0cfa00640 1 -- 172.21.15.28:0/3231667903 shutdown_connections 2024-04-23T14:50:40.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:40.236+0000 7fc0cfa00640 1 -- 172.21.15.28:0/3231667903 wait complete. 2024-04-23T14:50:40.498 INFO:tasks.workunit.client.0.smithi028.stdout: 1 539524 9.44 MB/sec execute 176 sec latency 26.458 ms 2024-04-23T14:50:40.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:40 smithi038 ceph-mon[32348]: pgmap v202: 97 pgs: 97 active+clean; 349 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 988 KiB/s rd, 2.1 MiB/s wr, 125 op/s 2024-04-23T14:50:40.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:40 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2720753821' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:50:40.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:40 smithi028 ceph-mon[27810]: pgmap v202: 97 pgs: 97 active+clean; 349 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 988 KiB/s rd, 2.1 MiB/s wr, 125 op/s 2024-04-23T14:50:40.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:40 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2720753821' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:50:41.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:40 smithi105 ceph-mon[32555]: pgmap v202: 97 pgs: 97 active+clean; 349 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 988 KiB/s rd, 2.1 MiB/s wr, 125 op/s 2024-04-23T14:50:41.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:40 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2720753821' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:50:41.259 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-23T14:50:41.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:50:41.260 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-23T14:50:41.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.491+0000 7fd661200640 1 -- 172.21.15.28:0/587692152 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd65c104320 msgr2=0x7fd65c104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:41.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.491+0000 7fd661200640 1 --2- 172.21.15.28:0/587692152 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd65c104320 0x7fd65c104700 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd64c009900 tx=0x7fd64c02a410 comp rx=0 tx=0).stop 2024-04-23T14:50:41.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.492+0000 7fd661200640 1 -- 172.21.15.28:0/587692152 shutdown_connections 2024-04-23T14:50:41.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.492+0000 7fd661200640 1 --2- 172.21.15.28:0/587692152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd65c1093f0 0x7fd65c110fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.492+0000 7fd661200640 1 --2- 172.21.15.28:0/587692152 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd65c104cd0 0x7fd65c108d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.492+0000 7fd661200640 1 --2- 172.21.15.28:0/587692152 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd65c104320 0x7fd65c104700 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.492+0000 7fd661200640 1 -- 172.21.15.28:0/587692152 >> 172.21.15.28:0/587692152 conn(0x7fd65c0ffb40 msgr2=0x7fd65c101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:41.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.492+0000 7fd661200640 1 -- 172.21.15.28:0/587692152 shutdown_connections 2024-04-23T14:50:41.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.492+0000 7fd661200640 1 -- 172.21.15.28:0/587692152 wait complete. 2024-04-23T14:50:41.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.493+0000 7fd661200640 1 Processor -- start 2024-04-23T14:50:41.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.494+0000 7fd661200640 1 -- start start 2024-04-23T14:50:41.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.494+0000 7fd661200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd65c104320 0x7fd65c1a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.494+0000 7fd661200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd65c1093f0 0x7fd65c1a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.495+0000 7fd661200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd65c19a830 0x7fd65c19ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.495+0000 7fd661200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd65c1144f0 con 0x7fd65c104320 2024-04-23T14:50:41.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.495+0000 7fd661200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd65c114370 con 0x7fd65c19a830 2024-04-23T14:50:41.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.495+0000 7fd661200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd65c114670 con 0x7fd65c1093f0 2024-04-23T14:50:41.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.495+0000 7fd65be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd65c19a830 0x7fd65c19ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.495+0000 7fd65be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd65c19a830 0x7fd65c19ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:52890/0 (socket says 172.21.15.28:52890) 2024-04-23T14:50:41.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.495+0000 7fd65be00640 1 -- 172.21.15.28:0/1442144639 learned_addr learned my addr 172.21.15.28:0/1442144639 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:41.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.495+0000 7fd65aa00640 1 --2- 172.21.15.28:0/1442144639 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd65c1093f0 0x7fd65c1a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.495+0000 7fd65b400640 1 --2- 172.21.15.28:0/1442144639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd65c104320 0x7fd65c1a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.495+0000 7fd65be00640 1 -- 172.21.15.28:0/1442144639 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd65c1093f0 msgr2=0x7fd65c1a0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:41.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.495+0000 7fd65be00640 1 --2- 172.21.15.28:0/1442144639 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd65c1093f0 0x7fd65c1a0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.495+0000 7fd65be00640 1 -- 172.21.15.28:0/1442144639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd65c104320 msgr2=0x7fd65c1a06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:41.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.495+0000 7fd65be00640 1 --2- 172.21.15.28:0/1442144639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd65c104320 0x7fd65c1a06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.495+0000 7fd65be00640 1 -- 172.21.15.28:0/1442144639 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd65c19b3c0 con 0x7fd65c19a830 2024-04-23T14:50:41.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.496+0000 7fd65be00640 1 --2- 172.21.15.28:0/1442144639 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd65c19a830 0x7fd65c19ace0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd65400eb10 tx=0x7fd65400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:41.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.496+0000 7fd653e00640 1 -- 172.21.15.28:0/1442144639 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd654004b20 con 0x7fd65c19a830 2024-04-23T14:50:41.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.496+0000 7fd653e00640 1 -- 172.21.15.28:0/1442144639 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd654004cc0 con 0x7fd65c19a830 2024-04-23T14:50:41.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.496+0000 7fd661200640 1 -- 172.21.15.28:0/1442144639 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd65c19b6b0 con 0x7fd65c19a830 2024-04-23T14:50:41.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.496+0000 7fd653e00640 1 -- 172.21.15.28:0/1442144639 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd654010760 con 0x7fd65c19a830 2024-04-23T14:50:41.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.497+0000 7fd661200640 1 -- 172.21.15.28:0/1442144639 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd65c1a78e0 con 0x7fd65c19a830 2024-04-23T14:50:41.499 INFO:tasks.workunit.client.0.smithi028.stdout: 1 540724 9.42 MB/sec execute 177 sec latency 20.389 ms 2024-04-23T14:50:41.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.501+0000 7fd653e00640 1 -- 172.21.15.28:0/1442144639 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fd654010900 con 0x7fd65c19a830 2024-04-23T14:50:41.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.502+0000 7fd66562b740 1 Processor -- start 2024-04-23T14:50:41.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.502+0000 7fd66562b740 1 -- start start 2024-04-23T14:50:41.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.502+0000 7fd66562b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557aeca12a80 0x557aeca12e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.502+0000 7fd66562b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aeca133a0 0x557aeca1d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.502+0000 7fd66562b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557aeca1da70 0x557aeca1fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.502+0000 7fd66562b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557aec96fe70 con 0x557aeca133a0 2024-04-23T14:50:41.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.502+0000 7fd66562b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x557aec916380 con 0x557aeca12a80 2024-04-23T14:50:41.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.502+0000 7fd66562b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x557aec96be30 con 0x557aeca1da70 2024-04-23T14:50:41.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.503+0000 7fd65b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aeca133a0 0x557aeca1d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.503+0000 7fd65b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aeca133a0 0x557aeca1d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56968/0 (socket says 172.21.15.28:56968) 2024-04-23T14:50:41.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.503+0000 7fd65b400640 1 -- 172.21.15.28:0/3431327686 learned_addr learned my addr 172.21.15.28:0/3431327686 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:41.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.503+0000 7fd65aa00640 1 --2- 172.21.15.28:0/3431327686 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557aeca1da70 0x557aeca1fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.503+0000 7fd65aa00640 1 --2- 172.21.15.28:0/3431327686 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557aeca12a80 0x557aeca12e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.503+0000 7fd65b400640 1 -- 172.21.15.28:0/3431327686 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557aeca1da70 msgr2=0x557aeca1fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:41.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.503+0000 7fd65b400640 1 --2- 172.21.15.28:0/3431327686 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557aeca1da70 0x557aeca1fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.503+0000 7fd65b400640 1 -- 172.21.15.28:0/3431327686 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557aeca12a80 msgr2=0x557aeca12e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:41.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.503+0000 7fd65b400640 1 --2- 172.21.15.28:0/3431327686 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557aeca12a80 0x557aeca12e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.503+0000 7fd65b400640 1 -- 172.21.15.28:0/3431327686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557aec74a7e0 con 0x557aeca133a0 2024-04-23T14:50:41.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.503+0000 7fd65aa00640 1 --2- 172.21.15.28:0/3431327686 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557aeca12a80 0x557aeca12e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:50:41.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.503+0000 7fd653e00640 1 --2- 172.21.15.28:0/1442144639 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd6400a3da0 0x7fd6400a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.504+0000 7fd653e00640 1 -- 172.21.15.28:0/1442144639 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fd6540d0ed0 con 0x7fd65c19a830 2024-04-23T14:50:41.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.504+0000 7fd65b400640 1 --2- 172.21.15.28:0/3431327686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aeca133a0 0x557aeca1d530 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd64c009900 tx=0x7fd64c005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae243201dc103a65 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:41.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.504+0000 7fd65aa00640 1 --2- 172.21.15.28:0/1442144639 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd6400a3da0 0x7fd6400a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.505+0000 7fd650c00640 1 -- 172.21.15.28:0/3431327686 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64c035030 con 0x557aeca133a0 2024-04-23T14:50:41.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.505+0000 7fd650c00640 1 -- 172.21.15.28:0/3431327686 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd64c0043c0 con 0x557aeca133a0 2024-04-23T14:50:41.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.505+0000 7fd650c00640 1 -- 172.21.15.28:0/3431327686 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64c0035f0 con 0x557aeca133a0 2024-04-23T14:50:41.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.506+0000 7fd66562b740 1 -- 172.21.15.28:0/3431327686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aeca133a0 msgr2=0x557aeca1d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:41.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.506+0000 7fd66562b740 1 --2- 172.21.15.28:0/3431327686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aeca133a0 0x557aeca1d530 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd64c009900 tx=0x7fd64c005e60 comp rx=0 tx=0).stop 2024-04-23T14:50:41.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.506+0000 7fd65aa00640 1 --2- 172.21.15.28:0/1442144639 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd6400a3da0 0x7fd6400a6260 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x557aeca0ff30 tx=0x7fd648013040 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:41.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.506+0000 7fd66562b740 1 -- 172.21.15.28:0/3431327686 shutdown_connections 2024-04-23T14:50:41.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.506+0000 7fd66562b740 1 --2- 172.21.15.28:0/3431327686 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557aeca1da70 0x557aeca1fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.506+0000 7fd66562b740 1 --2- 172.21.15.28:0/3431327686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aeca133a0 0x557aeca1d530 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.506+0000 7fd66562b740 1 --2- 172.21.15.28:0/3431327686 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557aeca12a80 0x557aeca12e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.506+0000 7fd66562b740 1 -- 172.21.15.28:0/3431327686 >> 172.21.15.28:0/3431327686 conn(0x557aeca0eda0 msgr2=0x557aeca0f180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:41.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.506+0000 7fd66562b740 1 -- 172.21.15.28:0/3431327686 shutdown_connections 2024-04-23T14:50:41.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.506+0000 7fd66562b740 1 -- 172.21.15.28:0/3431327686 wait complete. 2024-04-23T14:50:41.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.507+0000 7fd66562b740 1 -- start start 2024-04-23T14:50:41.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.507+0000 7fd66562b740 1 Processor -- start 2024-04-23T14:50:41.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.508+0000 7fd66562b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557aeca12a80 0x557aeca1a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.508+0000 7fd66562b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aeca133a0 0x557aeca1af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.508+0000 7fd66562b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557aeca1da70 0x557aeca1b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.508+0000 7fd66562b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557aeca20800 con 0x557aeca133a0 2024-04-23T14:50:41.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.508+0000 7fd66562b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x557aec929660 con 0x557aeca1da70 2024-04-23T14:50:41.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.508+0000 7fd66562b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x557aeca20980 con 0x557aeca12a80 2024-04-23T14:50:41.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.508+0000 7fd66562b740 10 client.0 _ll_register_callbacks cb 0x557aec917aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:50:41.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.508+0000 7fd65be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557aeca1da70 0x557aeca1b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.508+0000 7fd65b400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557aeca12a80 0x557aeca1a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.508+0000 7fd65be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557aeca1da70 0x557aeca1b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:52904/0 (socket says 172.21.15.28:52904) 2024-04-23T14:50:41.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.508+0000 7fd65be00640 1 -- 172.21.15.28:0/2880413170 learned_addr learned my addr 172.21.15.28:0/2880413170 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:41.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.508+0000 7fd65aa00640 1 --2- 172.21.15.28:0/2880413170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aeca133a0 0x557aeca1af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.508+0000 7fd65be00640 1 -- 172.21.15.28:0/2880413170 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557aeca12a80 msgr2=0x557aeca1a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:41.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.508+0000 7fd65be00640 1 --2- 172.21.15.28:0/2880413170 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557aeca12a80 0x557aeca1a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.508+0000 7fd65be00640 1 -- 172.21.15.28:0/2880413170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aeca133a0 msgr2=0x557aeca1af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:41.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.508+0000 7fd65be00640 1 --2- 172.21.15.28:0/2880413170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aeca133a0 0x557aeca1af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.508+0000 7fd65be00640 1 -- 172.21.15.28:0/2880413170 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557aec7aaf70 con 0x557aeca1da70 2024-04-23T14:50:41.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.509+0000 7fd65be00640 1 --2- 172.21.15.28:0/2880413170 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557aeca1da70 0x557aeca1b450 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd6540a55c0 tx=0x7fd654004f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e2d58fa6da54f1c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:41.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.509+0000 7fd645600640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:50:41.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.509+0000 7fd645600640 1 -- 172.21.15.28:0/2880413170 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6540b5020 con 0x557aeca1da70 2024-04-23T14:50:41.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.509+0000 7fd645600640 1 -- 172.21.15.28:0/2880413170 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fd6540a5660 con 0x557aeca1da70 2024-04-23T14:50:41.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.509+0000 7fd66562b740 1 -- 172.21.15.28:0/2880413170 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd64c0043c0 con 0x557aeca1da70 2024-04-23T14:50:41.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.509+0000 7fd645600640 1 -- 172.21.15.28:0/2880413170 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fd65409ce80 con 0x557aeca1da70 2024-04-23T14:50:41.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.509+0000 7fd66562b740 10 client.26270 fetch_fsmap learned FSMap version 20 2024-04-23T14:50:41.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.509+0000 7fd66562b740 1 -- 172.21.15.28:0/2880413170 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x557aec642170 con 0x557aeca1da70 2024-04-23T14:50:41.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.509+0000 7fd645600640 1 -- 172.21.15.28:0/2880413170 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd64c0043c0 con 0x557aeca1da70 2024-04-23T14:50:41.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.510+0000 7fd645600640 1 -- 172.21.15.28:0/2880413170 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fd65409c460 con 0x557aeca1da70 2024-04-23T14:50:41.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.510+0000 7fd66562b740 10 client.26270 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:50:41.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.510+0000 7fd66562b740 10 client.26270 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:50:41.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.510+0000 7fd66562b740 20 client.26270 populate_metadata read hostname 'smithi028' 2024-04-23T14:50:41.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.510+0000 7fd66562b740 1 --2- 172.21.15.28:0/2880413170 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x557aeca2dda0 0x557aeca30240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.510+0000 7fd66562b740 4 client.26270 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:50:41.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.510+0000 7fd66562b740 1 -- 172.21.15.28:0/2880413170 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557aec97d890 con 0x557aeca2dda0 2024-04-23T14:50:41.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.510+0000 7fd65b400640 1 --2- 172.21.15.28:0/2880413170 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x557aeca2dda0 0x557aeca30240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.511+0000 7fd65b400640 1 --2- 172.21.15.28:0/2880413170 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x557aeca2dda0 0x557aeca30240 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59256ef6ec9bda70 server_cookie=34e3d4575f84e158 in_seq=0 out_seq=0 2024-04-23T14:50:41.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.511+0000 7fd645600640 10 client.26270 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:50:41.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.512+0000 7fd645600640 1 -- 172.21.15.28:0/2880413170 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557aec97d890 con 0x557aeca2dda0 2024-04-23T14:50:41.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.512+0000 7fd645600640 10 client.26270 handle_command_reply: tid=0 2024-04-23T14:50:41.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.512+0000 7fd66562b740 1 client.26270 shutdown 2024-04-23T14:50:41.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.512+0000 7fd66562b740 1 -- 172.21.15.28:0/2880413170 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557aeca1da70 msgr2=0x557aeca1b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:41.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.512+0000 7fd66562b740 1 --2- 172.21.15.28:0/2880413170 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557aeca1da70 0x557aeca1b450 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd6540a55c0 tx=0x7fd654004f20 comp rx=0 tx=0).stop 2024-04-23T14:50:41.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.512+0000 7fd66562b740 1 -- 172.21.15.28:0/2880413170 shutdown_connections 2024-04-23T14:50:41.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.513+0000 7fd66562b740 1 --2- 172.21.15.28:0/2880413170 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x557aeca2dda0 0x557aeca30240 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.513+0000 7fd66562b740 1 --2- 172.21.15.28:0/2880413170 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557aeca1da70 0x557aeca1b450 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.513+0000 7fd66562b740 1 --2- 172.21.15.28:0/2880413170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aeca133a0 0x557aeca1af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.513+0000 7fd66562b740 1 --2- 172.21.15.28:0/2880413170 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557aeca12a80 0x557aeca1a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.513+0000 7fd66562b740 1 -- 172.21.15.28:0/2880413170 >> 172.21.15.28:0/2880413170 conn(0x557aeca0eda0 msgr2=0x557aeca056f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:41.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.513+0000 7fd645600640 0 client.26270 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:50:41.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.513+0000 7fd66562b740 1 -- 172.21.15.28:0/2880413170 shutdown_connections 2024-04-23T14:50:41.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.513+0000 7fd66562b740 1 -- 172.21.15.28:0/2880413170 wait complete. 2024-04-23T14:50:41.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.513+0000 7fd66562b740 20 client.26270 trim_cache size 0 max 16384 2024-04-23T14:50:41.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.536+0000 7fd66562b740 1 Processor -- start 2024-04-23T14:50:41.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.536+0000 7fd66562b740 1 -- start start 2024-04-23T14:50:41.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.536+0000 7fd66562b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aeca12a80 0x557aeca29710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.536+0000 7fd66562b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557aeca133a0 0x557aeca19b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.536+0000 7fd66562b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557aeca1da70 0x557aeca05570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.536+0000 7fd66562b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd65c1a99a0 con 0x557aeca12a80 2024-04-23T14:50:41.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.536+0000 7fd66562b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd65c1ac220 con 0x557aeca133a0 2024-04-23T14:50:41.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.536+0000 7fd66562b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd65c1aa010 con 0x557aeca1da70 2024-04-23T14:50:41.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.537+0000 7fd65aa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557aeca133a0 0x557aeca19b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.537+0000 7fd65aa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557aeca133a0 0x557aeca19b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:52918/0 (socket says 172.21.15.28:52918) 2024-04-23T14:50:41.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.537+0000 7fd65aa00640 1 -- 172.21.15.28:0/3852999081 learned_addr learned my addr 172.21.15.28:0/3852999081 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:41.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.537+0000 7fd65b400640 1 --2- 172.21.15.28:0/3852999081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aeca12a80 0x557aeca29710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.537+0000 7fd65be00640 1 --2- 172.21.15.28:0/3852999081 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557aeca1da70 0x557aeca05570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.537+0000 7fd65aa00640 1 -- 172.21.15.28:0/3852999081 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557aeca1da70 msgr2=0x557aeca05570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:41.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.537+0000 7fd65aa00640 1 --2- 172.21.15.28:0/3852999081 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557aeca1da70 0x557aeca05570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.537+0000 7fd65aa00640 1 -- 172.21.15.28:0/3852999081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aeca12a80 msgr2=0x557aeca29710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:41.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.537+0000 7fd65aa00640 1 --2- 172.21.15.28:0/3852999081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aeca12a80 0x557aeca29710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.537+0000 7fd65aa00640 1 -- 172.21.15.28:0/3852999081 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557aeca01e80 con 0x557aeca133a0 2024-04-23T14:50:41.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.538+0000 7fd65aa00640 1 --2- 172.21.15.28:0/3852999081 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557aeca133a0 0x557aeca19b30 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd648004140 tx=0x7fd648005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3a2d13e3e263d05 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:41.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.538+0000 7fd647e00640 1 -- 172.21.15.28:0/3852999081 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64802f070 con 0x557aeca133a0 2024-04-23T14:50:41.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.538+0000 7fd647e00640 1 -- 172.21.15.28:0/3852999081 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd648003b30 con 0x557aeca133a0 2024-04-23T14:50:41.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.538+0000 7fd647e00640 1 -- 172.21.15.28:0/3852999081 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd648020440 con 0x557aeca133a0 2024-04-23T14:50:41.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.539+0000 7fd66562b740 1 -- 172.21.15.28:0/3852999081 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557aeca133a0 msgr2=0x557aeca19b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:41.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.539+0000 7fd66562b740 1 --2- 172.21.15.28:0/3852999081 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557aeca133a0 0x557aeca19b30 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd648004140 tx=0x7fd648005f90 comp rx=0 tx=0).stop 2024-04-23T14:50:41.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.539+0000 7fd66562b740 1 -- 172.21.15.28:0/3852999081 shutdown_connections 2024-04-23T14:50:41.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.539+0000 7fd66562b740 1 --2- 172.21.15.28:0/3852999081 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557aeca133a0 0x557aeca19b30 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.539+0000 7fd66562b740 1 --2- 172.21.15.28:0/3852999081 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557aeca1da70 0x557aeca05570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.539+0000 7fd66562b740 1 --2- 172.21.15.28:0/3852999081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aeca12a80 0x557aeca29710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.539+0000 7fd66562b740 1 -- 172.21.15.28:0/3852999081 >> 172.21.15.28:0/3852999081 conn(0x557aeca0eda0 msgr2=0x557aeca04e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:41.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.539+0000 7fd66562b740 1 -- 172.21.15.28:0/3852999081 shutdown_connections 2024-04-23T14:50:41.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.540+0000 7fd66562b740 1 -- 172.21.15.28:0/3852999081 wait complete. 2024-04-23T14:50:41.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.540+0000 7fd66562b740 1 -- start start 2024-04-23T14:50:41.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.540+0000 7fd66562b740 1 Processor -- start 2024-04-23T14:50:41.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.541+0000 7fd66562b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aeca12a80 0x557aeca25ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.541+0000 7fd66562b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557aeca133a0 0x557aeca21310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.541+0000 7fd66562b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557aeca1da70 0x557aeca21850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.541+0000 7fd66562b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd65c1aa6e0 con 0x557aeca12a80 2024-04-23T14:50:41.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.541+0000 7fd66562b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd65c1abb50 con 0x557aeca133a0 2024-04-23T14:50:41.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.541+0000 7fd66562b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd65c1aadb0 con 0x557aeca1da70 2024-04-23T14:50:41.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.541+0000 7fd66562b740 10 client.0 _ll_register_callbacks cb 0x7fd65409cc50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:50:41.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.541+0000 7fd65b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aeca12a80 0x557aeca25ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.541+0000 7fd65b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aeca12a80 0x557aeca25ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56986/0 (socket says 172.21.15.28:56986) 2024-04-23T14:50:41.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.541+0000 7fd65b400640 1 -- 172.21.15.28:0/1989615198 learned_addr learned my addr 172.21.15.28:0/1989615198 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:41.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.542+0000 7fd65aa00640 1 --2- 172.21.15.28:0/1989615198 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557aeca133a0 0x557aeca21310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.542+0000 7fd65be00640 1 --2- 172.21.15.28:0/1989615198 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557aeca1da70 0x557aeca21850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.542+0000 7fd65b400640 1 -- 172.21.15.28:0/1989615198 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557aeca1da70 msgr2=0x557aeca21850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:41.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.542+0000 7fd65b400640 1 --2- 172.21.15.28:0/1989615198 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557aeca1da70 0x557aeca21850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.542+0000 7fd65b400640 1 -- 172.21.15.28:0/1989615198 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557aeca133a0 msgr2=0x557aeca21310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:41.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.542+0000 7fd65b400640 1 --2- 172.21.15.28:0/1989615198 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557aeca133a0 0x557aeca21310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.542+0000 7fd65b400640 1 -- 172.21.15.28:0/1989615198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557aeca22080 con 0x557aeca12a80 2024-04-23T14:50:41.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.542+0000 7fd65aa00640 1 --2- 172.21.15.28:0/1989615198 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557aeca133a0 0x557aeca21310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:50:41.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.543+0000 7fd65b400640 1 --2- 172.21.15.28:0/1989615198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aeca12a80 0x557aeca25ff0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd64c009900 tx=0x7fd64c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed1546ae6b557f44 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:41.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.543+0000 7fd645600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-23T14:50:41.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.544+0000 7fd645600640 1 -- 172.21.15.28:0/1989615198 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64c035040 con 0x557aeca12a80 2024-04-23T14:50:41.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.544+0000 7fd645600640 1 -- 172.21.15.28:0/1989615198 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7fd64c0530f0 con 0x557aeca12a80 2024-04-23T14:50:41.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.544+0000 7fd66562b740 1 -- 172.21.15.28:0/1989615198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd648003b30 con 0x557aeca12a80 2024-04-23T14:50:41.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.545+0000 7fd645600640 1 -- 172.21.15.28:0/1989615198 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fd64c060050 con 0x557aeca12a80 2024-04-23T14:50:41.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.545+0000 7fd66562b740 10 client.16104 fetch_fsmap learned FSMap version 20 2024-04-23T14:50:41.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.545+0000 7fd66562b740 1 -- 172.21.15.28:0/1989615198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x557aeca94410 con 0x557aeca12a80 2024-04-23T14:50:41.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.545+0000 7fd645600640 1 -- 172.21.15.28:0/1989615198 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd648003b30 con 0x557aeca12a80 2024-04-23T14:50:41.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.546+0000 7fd645600640 1 -- 172.21.15.28:0/1989615198 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fd64c031a20 con 0x557aeca12a80 2024-04-23T14:50:41.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.546+0000 7fd66562b740 10 client.16104 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:50:41.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.546+0000 7fd66562b740 10 client.16104 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:50:41.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.546+0000 7fd66562b740 20 client.16104 populate_metadata read hostname 'smithi028' 2024-04-23T14:50:41.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.546+0000 7fd66562b740 1 --2- 172.21.15.28:0/1989615198 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x557aeca94750 0x557aeca96b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.546+0000 7fd66562b740 4 client.16104 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-23T14:50:41.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.546+0000 7fd66562b740 1 -- 172.21.15.28:0/1989615198 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557aec977700 con 0x557aeca94750 2024-04-23T14:50:41.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.546+0000 7fd65aa00640 1 --2- 172.21.15.28:0/1989615198 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x557aeca94750 0x557aeca96b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.548+0000 7fd645600640 10 client.16104 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:50:41.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.548+0000 7fd65aa00640 1 --2- 172.21.15.28:0/1989615198 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x557aeca94750 0x557aeca96b40 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76564a615d032650 server_cookie=bb9b2c4522852a38 in_seq=0 out_seq=0 2024-04-23T14:50:41.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.550+0000 7fd645600640 1 -- 172.21.15.28:0/1989615198 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557aec977700 con 0x557aeca94750 2024-04-23T14:50:41.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.550+0000 7fd645600640 10 client.16104 handle_command_reply: tid=0 2024-04-23T14:50:41.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.550+0000 7fd66562b740 1 client.16104 shutdown 2024-04-23T14:50:41.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.551+0000 7fd66562b740 1 -- 172.21.15.28:0/1989615198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aeca12a80 msgr2=0x557aeca25ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:41.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.551+0000 7fd66562b740 1 --2- 172.21.15.28:0/1989615198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aeca12a80 0x557aeca25ff0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd64c009900 tx=0x7fd64c004740 comp rx=0 tx=0).stop 2024-04-23T14:50:41.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.551+0000 7fd66562b740 1 -- 172.21.15.28:0/1989615198 shutdown_connections 2024-04-23T14:50:41.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.551+0000 7fd66562b740 1 --2- 172.21.15.28:0/1989615198 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x557aeca94750 0x557aeca96b40 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.551+0000 7fd66562b740 1 --2- 172.21.15.28:0/1989615198 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557aeca133a0 0x557aeca21310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.551+0000 7fd66562b740 1 --2- 172.21.15.28:0/1989615198 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557aeca1da70 0x557aeca21850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.551+0000 7fd66562b740 1 --2- 172.21.15.28:0/1989615198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aeca12a80 0x557aeca25ff0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.551+0000 7fd645600640 0 client.16104 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:50:41.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.551+0000 7fd66562b740 1 -- 172.21.15.28:0/1989615198 >> 172.21.15.28:0/1989615198 conn(0x557aeca0eda0 msgr2=0x557aeca04e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:41.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.551+0000 7fd66562b740 1 -- 172.21.15.28:0/1989615198 shutdown_connections 2024-04-23T14:50:41.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.551+0000 7fd66562b740 1 -- 172.21.15.28:0/1989615198 wait complete. 2024-04-23T14:50:41.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.551+0000 7fd66562b740 20 client.16104 trim_cache size 0 max 16384 2024-04-23T14:50:41.553 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:50:41.554 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-23T14:50:41.554 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "5fc41a35-4762-4b86-9e56-e89ffc1676ef", 2024-04-23T14:50:41.554 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-23T14:50:41.554 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:50:41.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.552+0000 7fd661200640 1 -- 172.21.15.28:0/1442144639 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd6400a3da0 msgr2=0x7fd6400a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:41.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.552+0000 7fd661200640 1 --2- 172.21.15.28:0/1442144639 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd6400a3da0 0x7fd6400a6260 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x557aeca0ff30 tx=0x7fd648013040 comp rx=0 tx=0).stop 2024-04-23T14:50:41.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.552+0000 7fd661200640 1 -- 172.21.15.28:0/1442144639 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd65c19a830 msgr2=0x7fd65c19ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:41.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.552+0000 7fd661200640 1 --2- 172.21.15.28:0/1442144639 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd65c19a830 0x7fd65c19ace0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd65400eb10 tx=0x7fd65400c2d0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.552+0000 7fd661200640 1 -- 172.21.15.28:0/1442144639 shutdown_connections 2024-04-23T14:50:41.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.552+0000 7fd661200640 1 --2- 172.21.15.28:0/1442144639 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd6400a3da0 0x7fd6400a6260 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.552+0000 7fd661200640 1 --2- 172.21.15.28:0/1442144639 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd65c19a830 0x7fd65c19ace0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.552+0000 7fd661200640 1 --2- 172.21.15.28:0/1442144639 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd65c1093f0 0x7fd65c1a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.552+0000 7fd661200640 1 --2- 172.21.15.28:0/1442144639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd65c104320 0x7fd65c1a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.552+0000 7fd661200640 1 -- 172.21.15.28:0/1442144639 >> 172.21.15.28:0/1442144639 conn(0x7fd65c0ffb40 msgr2=0x7fd65c101ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:41.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.552+0000 7fd661200640 1 -- 172.21.15.28:0/1442144639 shutdown_connections 2024-04-23T14:50:41.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.552+0000 7fd661200640 1 -- 172.21.15.28:0/1442144639 wait complete. 2024-04-23T14:50:41.574 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:50:41.574 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:50:41.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.782+0000 7f0c20200640 1 -- 172.21.15.28:0/2513925498 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c18104450 msgr2=0x7f0c18104830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:41.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.782+0000 7f0c20200640 1 --2- 172.21.15.28:0/2513925498 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c18104450 0x7f0c18104830 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c009ca0 tx=0x7f0c0c02a8c0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.782+0000 7f0c20200640 1 -- 172.21.15.28:0/2513925498 shutdown_connections 2024-04-23T14:50:41.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.782+0000 7f0c20200640 1 --2- 172.21.15.28:0/2513925498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c18109520 0x7f0c181110f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.782+0000 7f0c20200640 1 --2- 172.21.15.28:0/2513925498 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c18104e00 0x7f0c18108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.782+0000 7f0c20200640 1 --2- 172.21.15.28:0/2513925498 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c18104450 0x7f0c18104830 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.782+0000 7f0c20200640 1 -- 172.21.15.28:0/2513925498 >> 172.21.15.28:0/2513925498 conn(0x7f0c180ffc70 msgr2=0x7f0c18102090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:41.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.783+0000 7f0c20200640 1 -- 172.21.15.28:0/2513925498 shutdown_connections 2024-04-23T14:50:41.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.783+0000 7f0c20200640 1 -- 172.21.15.28:0/2513925498 wait complete. 2024-04-23T14:50:41.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.784+0000 7f0c20200640 1 Processor -- start 2024-04-23T14:50:41.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.784+0000 7f0c20200640 1 -- start start 2024-04-23T14:50:41.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.784+0000 7f0c20200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c18104450 0x7f0c181a0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.784+0000 7f0c20200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c18109520 0x7f0c181a0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.784+0000 7f0c20200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c1819a940 0x7f0c1819adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.784+0000 7f0c20200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0c18114600 con 0x7f0c18104450 2024-04-23T14:50:41.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.784+0000 7f0c20200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0c18114480 con 0x7f0c18109520 2024-04-23T14:50:41.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.784+0000 7f0c20200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0c18114780 con 0x7f0c1819a940 2024-04-23T14:50:41.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.785+0000 7f0c1d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c18104450 0x7f0c181a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.785+0000 7f0c1ce00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c18109520 0x7f0c181a0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.785+0000 7f0c1d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c18104450 0x7f0c181a0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57008/0 (socket says 172.21.15.28:57008) 2024-04-23T14:50:41.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.785+0000 7f0c1e200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c1819a940 0x7f0c1819adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.785+0000 7f0c1d800640 1 -- 172.21.15.28:0/670283010 learned_addr learned my addr 172.21.15.28:0/670283010 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:41.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.785+0000 7f0c1ce00640 1 -- 172.21.15.28:0/670283010 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c1819a940 msgr2=0x7f0c1819adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:41.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.785+0000 7f0c1ce00640 1 --2- 172.21.15.28:0/670283010 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c1819a940 0x7f0c1819adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.785+0000 7f0c1ce00640 1 -- 172.21.15.28:0/670283010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c18104450 msgr2=0x7f0c181a0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:41.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.785+0000 7f0c1ce00640 1 --2- 172.21.15.28:0/670283010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c18104450 0x7f0c181a0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.785+0000 7f0c1ce00640 1 -- 172.21.15.28:0/670283010 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c1819b460 con 0x7f0c18109520 2024-04-23T14:50:41.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.785+0000 7f0c1e200640 1 --2- 172.21.15.28:0/670283010 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c1819a940 0x7f0c1819adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:50:41.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.785+0000 7f0c1ce00640 1 --2- 172.21.15.28:0/670283010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c18109520 0x7f0c181a0cd0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f0c0400d6a0 tx=0x7f0c0400db60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:41.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.786+0000 7f0c16000640 1 -- 172.21.15.28:0/670283010 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c04017070 con 0x7f0c18109520 2024-04-23T14:50:41.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.786+0000 7f0c16000640 1 -- 172.21.15.28:0/670283010 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c04009820 con 0x7f0c18109520 2024-04-23T14:50:41.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.786+0000 7f0c20200640 1 -- 172.21.15.28:0/670283010 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c1819b750 con 0x7f0c18109520 2024-04-23T14:50:41.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.786+0000 7f0c16000640 1 -- 172.21.15.28:0/670283010 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c04004d00 con 0x7f0c18109520 2024-04-23T14:50:41.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.786+0000 7f0c20200640 1 -- 172.21.15.28:0/670283010 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c181a79f0 con 0x7f0c18109520 2024-04-23T14:50:41.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.790+0000 7f0c2301b740 1 Processor -- start 2024-04-23T14:50:41.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.790+0000 7f0c16000640 1 -- 172.21.15.28:0/670283010 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f0c04005220 con 0x7f0c18109520 2024-04-23T14:50:41.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.790+0000 7f0c2301b740 1 -- start start 2024-04-23T14:50:41.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.791+0000 7f0c2301b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561c97f8860 0x5561c97f8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.791+0000 7f0c2301b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561c97f9180 0x5561c9803310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.791+0000 7f0c2301b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5561c9803850 0x5561c9805c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.791+0000 7f0c2301b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5561c975e1f0 con 0x5561c97f8860 2024-04-23T14:50:41.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.791+0000 7f0c2301b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5561c9704350 con 0x5561c9803850 2024-04-23T14:50:41.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.791+0000 7f0c2301b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5561c975a1b0 con 0x5561c97f9180 2024-04-23T14:50:41.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.791+0000 7f0c1ce00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5561c9803850 0x5561c9805c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.791+0000 7f0c1e200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561c97f8860 0x5561c97f8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.791+0000 7f0c1ce00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5561c9803850 0x5561c9805c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:52966/0 (socket says 172.21.15.28:52966) 2024-04-23T14:50:41.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.791+0000 7f0c1ce00640 1 -- 172.21.15.28:0/1137928450 learned_addr learned my addr 172.21.15.28:0/1137928450 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:41.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.791+0000 7f0c1d800640 1 --2- 172.21.15.28:0/1137928450 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561c97f9180 0x5561c9803310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.792+0000 7f0c1ce00640 1 -- 172.21.15.28:0/1137928450 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561c97f9180 msgr2=0x5561c9803310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:41.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.792+0000 7f0c1ce00640 1 --2- 172.21.15.28:0/1137928450 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561c97f9180 0x5561c9803310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.792+0000 7f0c1ce00640 1 -- 172.21.15.28:0/1137928450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561c97f8860 msgr2=0x5561c97f8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:41.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.792+0000 7f0c1ce00640 1 --2- 172.21.15.28:0/1137928450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561c97f8860 0x5561c97f8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.792+0000 7f0c1ce00640 1 -- 172.21.15.28:0/1137928450 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561c9758f50 con 0x5561c9803850 2024-04-23T14:50:41.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.792+0000 7f0c1ce00640 1 --2- 172.21.15.28:0/1137928450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5561c9803850 0x5561c9805c40 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0c040cd370 tx=0x7f0c040d3f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7eeb98ed3f89a32 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:41.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.792+0000 7f0c0aa00640 1 -- 172.21.15.28:0/1137928450 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c040e1440 con 0x5561c9803850 2024-04-23T14:50:41.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.792+0000 7f0c0aa00640 1 -- 172.21.15.28:0/1137928450 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c040d58e0 con 0x5561c9803850 2024-04-23T14:50:41.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.792+0000 7f0c0aa00640 1 -- 172.21.15.28:0/1137928450 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c040e0a10 con 0x5561c9803850 2024-04-23T14:50:41.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.792+0000 7f0c16000640 1 --2- 172.21.15.28:0/670283010 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f0c000a3da0 0x7f0c000a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.793+0000 7f0c16000640 1 -- 172.21.15.28:0/670283010 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f0c040cfd30 con 0x7f0c18109520 2024-04-23T14:50:41.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.793+0000 7f0c1e200640 1 --2- 172.21.15.28:0/670283010 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f0c000a3da0 0x7f0c000a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.793+0000 7f0c2301b740 1 -- 172.21.15.28:0/1137928450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5561c9803850 msgr2=0x5561c9805c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:41.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.793+0000 7f0c2301b740 1 --2- 172.21.15.28:0/1137928450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5561c9803850 0x5561c9805c40 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0c040cd370 tx=0x7f0c040d3f60 comp rx=0 tx=0).stop 2024-04-23T14:50:41.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.793+0000 7f0c2301b740 1 -- 172.21.15.28:0/1137928450 shutdown_connections 2024-04-23T14:50:41.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.793+0000 7f0c2301b740 1 --2- 172.21.15.28:0/1137928450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5561c9803850 0x5561c9805c40 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.793+0000 7f0c2301b740 1 --2- 172.21.15.28:0/1137928450 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561c97f9180 0x5561c9803310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.793+0000 7f0c2301b740 1 --2- 172.21.15.28:0/1137928450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561c97f8860 0x5561c97f8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.793+0000 7f0c2301b740 1 -- 172.21.15.28:0/1137928450 >> 172.21.15.28:0/1137928450 conn(0x5561c97f4b80 msgr2=0x5561c97f4f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:41.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.793+0000 7f0c2301b740 1 -- 172.21.15.28:0/1137928450 shutdown_connections 2024-04-23T14:50:41.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.793+0000 7f0c2301b740 1 -- 172.21.15.28:0/1137928450 wait complete. 2024-04-23T14:50:41.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.793+0000 7f0c1e200640 1 --2- 172.21.15.28:0/670283010 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f0c000a3da0 0x7f0c000a6260 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0c10012560 tx=0x7f0c10005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:41.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.794+0000 7f0c2301b740 1 -- start start 2024-04-23T14:50:41.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.794+0000 7f0c2301b740 1 Processor -- start 2024-04-23T14:50:41.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.795+0000 7f0c2301b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561c97f8860 0x5561c9800780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.795+0000 7f0c2301b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561c97f9180 0x5561c9800cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.795+0000 7f0c2301b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5561c9803850 0x5561c9801200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.795+0000 7f0c2301b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5561c9806640 con 0x5561c97f8860 2024-04-23T14:50:41.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.795+0000 7f0c2301b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5561c9717690 con 0x5561c9803850 2024-04-23T14:50:41.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.795+0000 7f0c2301b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5561c98067c0 con 0x5561c97f9180 2024-04-23T14:50:41.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.795+0000 7f0c2301b740 10 client.0 _ll_register_callbacks cb 0x5561c9753650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:50:41.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.795+0000 7f0c1d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561c97f8860 0x5561c9800780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.795+0000 7f0c1e200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5561c9803850 0x5561c9801200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.795+0000 7f0c1ce00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561c97f9180 0x5561c9800cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.795+0000 7f0c1d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561c97f8860 0x5561c9800780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57034/0 (socket says 172.21.15.28:57034) 2024-04-23T14:50:41.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.795+0000 7f0c1d800640 1 -- 172.21.15.28:0/3405767577 learned_addr learned my addr 172.21.15.28:0/3405767577 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:41.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.796+0000 7f0c1ce00640 1 -- 172.21.15.28:0/3405767577 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5561c9803850 msgr2=0x5561c9801200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:41.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.796+0000 7f0c1ce00640 1 --2- 172.21.15.28:0/3405767577 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5561c9803850 0x5561c9801200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.796+0000 7f0c1ce00640 1 -- 172.21.15.28:0/3405767577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561c97f8860 msgr2=0x5561c9800780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.796+0000 7f0c1ce00640 1 --2- 172.21.15.28:0/3405767577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561c97f8860 0x5561c9800780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.796+0000 7f0c1ce00640 1 -- 172.21.15.28:0/3405767577 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561c9716920 con 0x5561c97f9180 2024-04-23T14:50:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.796+0000 7f0c1ce00640 1 --2- 172.21.15.28:0/3405767577 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561c97f9180 0x5561c9800cc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0c04098190 tx=0x7f0c040e0510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12403b02a4c3aa98 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.796+0000 7f0bff400640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:50:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.796+0000 7f0bff400640 1 -- 172.21.15.28:0/3405767577 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c04020070 con 0x5561c97f9180 2024-04-23T14:50:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.796+0000 7f0c2301b740 1 -- 172.21.15.28:0/3405767577 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c040d58e0 con 0x5561c97f9180 2024-04-23T14:50:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.796+0000 7f0bff400640 1 -- 172.21.15.28:0/3405767577 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f0c0409a030 con 0x5561c97f9180 2024-04-23T14:50:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.796+0000 7f0bff400640 1 -- 172.21.15.28:0/3405767577 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f0c04097fd0 con 0x5561c97f9180 2024-04-23T14:50:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.797+0000 7f0c2301b740 10 client.26272 fetch_fsmap learned FSMap version 20 2024-04-23T14:50:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.797+0000 7f0c2301b740 1 -- 172.21.15.28:0/3405767577 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5561c953a7d0 con 0x5561c97f9180 2024-04-23T14:50:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.797+0000 7f0bff400640 1 -- 172.21.15.28:0/3405767577 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c040d58e0 con 0x5561c97f9180 2024-04-23T14:50:41.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.797+0000 7f0bff400640 1 -- 172.21.15.28:0/3405767577 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f0c040df070 con 0x5561c97f9180 2024-04-23T14:50:41.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.797+0000 7f0c2301b740 10 client.26272 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:50:41.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.797+0000 7f0c2301b740 10 client.26272 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:50:41.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.797+0000 7f0c2301b740 20 client.26272 populate_metadata read hostname 'smithi028' 2024-04-23T14:50:41.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.797+0000 7f0c2301b740 1 --2- 172.21.15.28:0/3405767577 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5561c9813a00 0x5561c9815df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.797+0000 7f0c2301b740 4 client.26272 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:50:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.797+0000 7f0c2301b740 1 -- 172.21.15.28:0/3405767577 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5561c976bc10 con 0x5561c9813a00 2024-04-23T14:50:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.797+0000 7f0c1d800640 1 --2- 172.21.15.28:0/3405767577 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5561c9813a00 0x5561c9815df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.798+0000 7f0c1d800640 1 --2- 172.21.15.28:0/3405767577 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5561c9813a00 0x5561c9815df0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed18af8eaafc7007 server_cookie=9579d5b38d950541 in_seq=0 out_seq=0 2024-04-23T14:50:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.798+0000 7f0bff400640 10 client.26272 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:50:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.800+0000 7f0bff400640 1 -- 172.21.15.28:0/3405767577 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5561c976bc10 con 0x5561c9813a00 2024-04-23T14:50:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.800+0000 7f0bff400640 10 client.26272 handle_command_reply: tid=0 2024-04-23T14:50:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.800+0000 7f0c2301b740 1 client.26272 shutdown 2024-04-23T14:50:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.800+0000 7f0c2301b740 1 -- 172.21.15.28:0/3405767577 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561c97f9180 msgr2=0x5561c9800cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.800+0000 7f0c2301b740 1 --2- 172.21.15.28:0/3405767577 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561c97f9180 0x5561c9800cc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0c04098190 tx=0x7f0c040e0510 comp rx=0 tx=0).stop 2024-04-23T14:50:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.800+0000 7f0c2301b740 1 -- 172.21.15.28:0/3405767577 shutdown_connections 2024-04-23T14:50:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.800+0000 7f0c2301b740 1 --2- 172.21.15.28:0/3405767577 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5561c9813a00 0x5561c9815df0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.800+0000 7f0c2301b740 1 --2- 172.21.15.28:0/3405767577 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5561c9803850 0x5561c9801200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.800+0000 7f0c2301b740 1 --2- 172.21.15.28:0/3405767577 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561c97f9180 0x5561c9800cc0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.800+0000 7f0bff400640 0 client.26272 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:50:41.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.800+0000 7f0c2301b740 1 --2- 172.21.15.28:0/3405767577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561c97f8860 0x5561c9800780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.800+0000 7f0c2301b740 1 -- 172.21.15.28:0/3405767577 >> 172.21.15.28:0/3405767577 conn(0x5561c97f4b80 msgr2=0x5561c97fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:41.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.800+0000 7f0c2301b740 1 -- 172.21.15.28:0/3405767577 shutdown_connections 2024-04-23T14:50:41.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.800+0000 7f0c2301b740 1 -- 172.21.15.28:0/3405767577 wait complete. 2024-04-23T14:50:41.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.800+0000 7f0c2301b740 20 client.26272 trim_cache size 0 max 16384 2024-04-23T14:50:41.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.824+0000 7f0c2301b740 1 Processor -- start 2024-04-23T14:50:41.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.824+0000 7f0c2301b740 1 -- start start 2024-04-23T14:50:41.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.824+0000 7f0c2301b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5561c97f8860 0x5561c97ff700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.824+0000 7f0c2301b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561c97f9180 0x5561c97fb000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.824+0000 7f0c2301b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561c9803850 0x5561c9818e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.825+0000 7f0c2301b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0c181ae070 con 0x5561c97f9180 2024-04-23T14:50:41.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.825+0000 7f0c2301b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0c181b08f0 con 0x5561c97f8860 2024-04-23T14:50:41.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.825+0000 7f0c2301b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0c181ae6e0 con 0x5561c9803850 2024-04-23T14:50:41.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.825+0000 7f0c1d800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5561c97f8860 0x5561c97ff700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.825+0000 7f0c1d800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5561c97f8860 0x5561c97ff700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:52978/0 (socket says 172.21.15.28:52978) 2024-04-23T14:50:41.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.825+0000 7f0c1ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561c97f9180 0x5561c97fb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.825+0000 7f0c1d800640 1 -- 172.21.15.28:0/2305681620 learned_addr learned my addr 172.21.15.28:0/2305681620 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:41.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.825+0000 7f0c1e200640 1 --2- 172.21.15.28:0/2305681620 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561c9803850 0x5561c9818e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.825+0000 7f0c1d800640 1 -- 172.21.15.28:0/2305681620 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561c9803850 msgr2=0x5561c9818e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:41.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.825+0000 7f0c1d800640 1 --2- 172.21.15.28:0/2305681620 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561c9803850 0x5561c9818e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.825+0000 7f0c1d800640 1 -- 172.21.15.28:0/2305681620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561c97f9180 msgr2=0x5561c97fb000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:41.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.825+0000 7f0c1d800640 1 --2- 172.21.15.28:0/2305681620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561c97f9180 0x5561c97fb000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.825+0000 7f0c1d800640 1 -- 172.21.15.28:0/2305681620 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561c97f8fb0 con 0x5561c97f8860 2024-04-23T14:50:41.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.825+0000 7f0c1ce00640 1 --2- 172.21.15.28:0/2305681620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561c97f9180 0x5561c97fb000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:50:41.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.826+0000 7f0c1d800640 1 --2- 172.21.15.28:0/2305681620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5561c97f8860 0x5561c97ff700 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c005e30 tx=0x7f0c0c005d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5033f769c6d5654 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:41.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.826+0000 7f0c0a000640 1 -- 172.21.15.28:0/2305681620 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c032ea0 con 0x5561c97f8860 2024-04-23T14:50:41.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.826+0000 7f0c0a000640 1 -- 172.21.15.28:0/2305681620 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c0c04e020 con 0x5561c97f8860 2024-04-23T14:50:41.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.826+0000 7f0c0a000640 1 -- 172.21.15.28:0/2305681620 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c004330 con 0x5561c97f8860 2024-04-23T14:50:41.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.827+0000 7f0c2301b740 1 -- 172.21.15.28:0/2305681620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5561c97f8860 msgr2=0x5561c97ff700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:41.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.827+0000 7f0c2301b740 1 --2- 172.21.15.28:0/2305681620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5561c97f8860 0x5561c97ff700 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c005e30 tx=0x7f0c0c005d80 comp rx=0 tx=0).stop 2024-04-23T14:50:41.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.827+0000 7f0c2301b740 1 -- 172.21.15.28:0/2305681620 shutdown_connections 2024-04-23T14:50:41.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.827+0000 7f0c2301b740 1 --2- 172.21.15.28:0/2305681620 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561c9803850 0x5561c9818e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.827+0000 7f0c2301b740 1 --2- 172.21.15.28:0/2305681620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561c97f9180 0x5561c97fb000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.827+0000 7f0c2301b740 1 --2- 172.21.15.28:0/2305681620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5561c97f8860 0x5561c97ff700 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.827+0000 7f0c2301b740 1 -- 172.21.15.28:0/2305681620 >> 172.21.15.28:0/2305681620 conn(0x5561c97f4b80 msgr2=0x5561c980f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:41.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.827+0000 7f0c2301b740 1 -- 172.21.15.28:0/2305681620 shutdown_connections 2024-04-23T14:50:41.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.827+0000 7f0c2301b740 1 -- 172.21.15.28:0/2305681620 wait complete. 2024-04-23T14:50:41.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.828+0000 7f0c2301b740 1 -- start start 2024-04-23T14:50:41.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.828+0000 7f0c2301b740 1 Processor -- start 2024-04-23T14:50:41.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.828+0000 7f0c2301b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561c97f8860 0x5561c9819300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.828+0000 7f0c2301b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5561c97f9180 0x5561c9816690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.828+0000 7f0c2301b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561c9803850 0x5561c9816ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.828+0000 7f0c2301b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0c181aedb0 con 0x5561c97f8860 2024-04-23T14:50:41.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.828+0000 7f0c2301b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0c181b0220 con 0x5561c97f9180 2024-04-23T14:50:41.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.828+0000 7f0c2301b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0c181af480 con 0x5561c9803850 2024-04-23T14:50:41.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.828+0000 7f0c2301b740 10 client.0 _ll_register_callbacks cb 0x7f0c10009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:50:41.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.829+0000 7f0c1ce00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5561c97f9180 0x5561c9816690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.829+0000 7f0c1ce00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5561c97f9180 0x5561c9816690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:52980/0 (socket says 172.21.15.28:52980) 2024-04-23T14:50:41.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.829+0000 7f0c1ce00640 1 -- 172.21.15.28:0/3529232492 learned_addr learned my addr 172.21.15.28:0/3529232492 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:50:41.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.829+0000 7f0c1e200640 1 --2- 172.21.15.28:0/3529232492 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561c9803850 0x5561c9816ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.829+0000 7f0c1d800640 1 --2- 172.21.15.28:0/3529232492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561c97f8860 0x5561c9819300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.829+0000 7f0c1ce00640 1 -- 172.21.15.28:0/3529232492 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561c9803850 msgr2=0x5561c9816ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:41.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.829+0000 7f0c1ce00640 1 --2- 172.21.15.28:0/3529232492 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561c9803850 0x5561c9816ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.829+0000 7f0c1ce00640 1 -- 172.21.15.28:0/3529232492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561c97f8860 msgr2=0x5561c9819300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:50:41.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.829+0000 7f0c1ce00640 1 --2- 172.21.15.28:0/3529232492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561c97f8860 0x5561c9819300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.829+0000 7f0c1ce00640 1 -- 172.21.15.28:0/3529232492 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561c94703f0 con 0x5561c97f9180 2024-04-23T14:50:41.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.829+0000 7f0c1d800640 1 --2- 172.21.15.28:0/3529232492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561c97f8860 0x5561c9819300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:50:41.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.829+0000 7f0c1ce00640 1 --2- 172.21.15.28:0/3529232492 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5561c97f9180 0x5561c9816690 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f0c04098700 tx=0x7f0c040b5120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=256ea129ee73bdd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:50:41.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.829+0000 7f0bff400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:50:41.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.830+0000 7f0bff400640 1 -- 172.21.15.28:0/3529232492 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c04020070 con 0x5561c97f9180 2024-04-23T14:50:41.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.830+0000 7f0bff400640 1 -- 172.21.15.28:0/3529232492 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8498+0+0 (secure 0 0 0) 0x7f0c040cd6b0 con 0x5561c97f9180 2024-04-23T14:50:41.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.830+0000 7f0c2301b740 1 -- 172.21.15.28:0/3529232492 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c0c04e020 con 0x5561c97f9180 2024-04-23T14:50:41.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.830+0000 7f0bff400640 1 -- 172.21.15.28:0/3529232492 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f0c0409b690 con 0x5561c97f9180 2024-04-23T14:50:41.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.830+0000 7f0c2301b740 10 client.26294 fetch_fsmap learned FSMap version 20 2024-04-23T14:50:41.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.830+0000 7f0c2301b740 1 -- 172.21.15.28:0/3529232492 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5561c94300f0 con 0x5561c97f9180 2024-04-23T14:50:41.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.830+0000 7f0bff400640 1 -- 172.21.15.28:0/3529232492 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c0c04e020 con 0x5561c97f9180 2024-04-23T14:50:41.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.830+0000 7f0bff400640 1 -- 172.21.15.28:0/3529232492 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f0c040df040 con 0x5561c97f9180 2024-04-23T14:50:41.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.830+0000 7f0c2301b740 10 client.26294 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:50:41.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.830+0000 7f0c2301b740 10 client.26294 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:50:41.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.830+0000 7f0c2301b740 20 client.26294 populate_metadata read hostname 'smithi028' 2024-04-23T14:50:41.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.831+0000 7f0c2301b740 1 --2- 172.21.15.28:0/3529232492 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5561c9879e80 0x5561c987c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:50:41.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.831+0000 7f0c2301b740 4 client.26294 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:50:41.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.831+0000 7f0c2301b740 1 -- 172.21.15.28:0/3529232492 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5561c9765a80 con 0x5561c9879e80 2024-04-23T14:50:41.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.831+0000 7f0c1d800640 1 --2- 172.21.15.28:0/3529232492 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5561c9879e80 0x5561c987c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:50:41.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.832+0000 7f0c1d800640 1 --2- 172.21.15.28:0/3529232492 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5561c9879e80 0x5561c987c2a0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=748d6df91029f4c6 server_cookie=534e20650f2a683e in_seq=0 out_seq=0 2024-04-23T14:50:41.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.832+0000 7f0bff400640 10 client.26294 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:50:41.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.832+0000 7f0bff400640 1 -- 172.21.15.28:0/3529232492 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5561c9765a80 con 0x5561c9879e80 2024-04-23T14:50:41.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.832+0000 7f0bff400640 10 client.26294 handle_command_reply: tid=0 2024-04-23T14:50:41.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.832+0000 7f0c2301b740 1 client.26294 shutdown 2024-04-23T14:50:41.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.832+0000 7f0c2301b740 1 -- 172.21.15.28:0/3529232492 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5561c97f9180 msgr2=0x5561c9816690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:41.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.832+0000 7f0c2301b740 1 --2- 172.21.15.28:0/3529232492 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5561c97f9180 0x5561c9816690 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f0c04098700 tx=0x7f0c040b5120 comp rx=0 tx=0).stop 2024-04-23T14:50:41.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.832+0000 7f0c2301b740 1 -- 172.21.15.28:0/3529232492 shutdown_connections 2024-04-23T14:50:41.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.832+0000 7f0c2301b740 1 --2- 172.21.15.28:0/3529232492 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5561c9879e80 0x5561c987c2a0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.832+0000 7f0c2301b740 1 --2- 172.21.15.28:0/3529232492 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5561c97f9180 0x5561c9816690 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.832+0000 7f0c2301b740 1 --2- 172.21.15.28:0/3529232492 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5561c9803850 0x5561c9816ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.832+0000 7f0c2301b740 1 --2- 172.21.15.28:0/3529232492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561c97f8860 0x5561c9819300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.832+0000 7f0bff400640 0 client.26294 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:50:41.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.832+0000 7f0c2301b740 1 -- 172.21.15.28:0/3529232492 >> 172.21.15.28:0/3529232492 conn(0x5561c97f4b80 msgr2=0x5561c9800720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:41.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.832+0000 7f0c2301b740 1 -- 172.21.15.28:0/3529232492 shutdown_connections 2024-04-23T14:50:41.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.832+0000 7f0c2301b740 1 -- 172.21.15.28:0/3529232492 wait complete. 2024-04-23T14:50:41.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.832+0000 7f0c2301b740 20 client.26294 trim_cache size 0 max 16384 2024-04-23T14:50:41.837 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:50:41.837 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-23T14:50:41.837 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-23T14:50:41.838 INFO:teuthology.orchestra.run.smithi028.stdout: "5fc41a35-4762-4b86-9e56-e89ffc1676ef": { 2024-04-23T14:50:41.838 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-23T14:50:41.838 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "5fc41a35-4762-4b86-9e56-e89ffc1676ef", 2024-04-23T14:50:41.838 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-23T14:50:41.838 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:50:41.838 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:50:41.838 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:50:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.833+0000 7f0c20200640 1 -- 172.21.15.28:0/670283010 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f0c000a3da0 msgr2=0x7f0c000a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.833+0000 7f0c20200640 1 --2- 172.21.15.28:0/670283010 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f0c000a3da0 0x7f0c000a6260 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0c10012560 tx=0x7f0c10005f90 comp rx=0 tx=0).stop 2024-04-23T14:50:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.833+0000 7f0c20200640 1 -- 172.21.15.28:0/670283010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c18109520 msgr2=0x7f0c181a0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:50:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.833+0000 7f0c20200640 1 --2- 172.21.15.28:0/670283010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c18109520 0x7f0c181a0cd0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f0c0400d6a0 tx=0x7f0c0400db60 comp rx=0 tx=0).stop 2024-04-23T14:50:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.833+0000 7f0c20200640 1 -- 172.21.15.28:0/670283010 shutdown_connections 2024-04-23T14:50:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.833+0000 7f0c20200640 1 --2- 172.21.15.28:0/670283010 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f0c000a3da0 0x7f0c000a6260 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.833+0000 7f0c20200640 1 --2- 172.21.15.28:0/670283010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c18109520 0x7f0c181a0cd0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.833+0000 7f0c20200640 1 --2- 172.21.15.28:0/670283010 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c1819a940 0x7f0c1819adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.833+0000 7f0c20200640 1 --2- 172.21.15.28:0/670283010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c18104450 0x7f0c181a0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:50:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.833+0000 7f0c20200640 1 -- 172.21.15.28:0/670283010 >> 172.21.15.28:0/670283010 conn(0x7f0c180ffc70 msgr2=0x7f0c18101f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:50:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.833+0000 7f0c20200640 1 -- 172.21.15.28:0/670283010 shutdown_connections 2024-04-23T14:50:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:50:41.834+0000 7f0c20200640 1 -- 172.21.15.28:0/670283010 wait complete. 2024-04-23T14:50:41.857 INFO:tasks.cephfs.filesystem:scrub status for tag:5fc41a35-4762-4b86-9e56-e89ffc1676ef - {'path': '/', 'tag': '5fc41a35-4762-4b86-9e56-e89ffc1676ef', 'options': 'recursive,force'} 2024-04-23T14:50:42.499 INFO:tasks.workunit.client.0.smithi028.stdout: 1 542124 9.40 MB/sec execute 178 sec latency 97.743 ms 2024-04-23T14:50:42.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:42 smithi038 ceph-mon[32348]: pgmap v203: 97 pgs: 97 active+clean; 349 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 988 KiB/s rd, 2.1 MiB/s wr, 125 op/s 2024-04-23T14:50:42.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:42 smithi038 ceph-mon[32348]: scrub queued for path: / 2024-04-23T14:50:42.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:42 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:50:42.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:42 smithi038 ceph-mon[32348]: scrub summary: active paths [/] 2024-04-23T14:50:42.873 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:42 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:50:42.877 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:42 smithi028 ceph-mon[27810]: pgmap v203: 97 pgs: 97 active+clean; 349 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 988 KiB/s rd, 2.1 MiB/s wr, 125 op/s 2024-04-23T14:50:42.877 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:42 smithi028 ceph-mon[27810]: scrub queued for path: / 2024-04-23T14:50:42.877 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:42 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:50:42.877 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:42 smithi028 ceph-mon[27810]: scrub summary: active paths [/] 2024-04-23T14:50:42.877 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:42 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:50:43.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:42 smithi105 ceph-mon[32555]: pgmap v203: 97 pgs: 97 active+clean; 349 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 988 KiB/s rd, 2.1 MiB/s wr, 125 op/s 2024-04-23T14:50:43.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:42 smithi105 ceph-mon[32555]: scrub queued for path: / 2024-04-23T14:50:43.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:42 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:50:43.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:42 smithi105 ceph-mon[32555]: scrub summary: active paths [/] 2024-04-23T14:50:43.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:42 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:50:43.499 INFO:tasks.workunit.client.0.smithi028.stdout: 1 543809 9.42 MB/sec execute 179 sec latency 44.649 ms 2024-04-23T14:50:44.500 INFO:tasks.workunit.client.0.smithi028.stdout: 1 545160 9.40 MB/sec execute 180 sec latency 18.573 ms 2024-04-23T14:50:44.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:44 smithi038 ceph-mon[32348]: pgmap v204: 97 pgs: 97 active+clean; 354 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.2 MiB/s wr, 190 op/s 2024-04-23T14:50:44.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:44 smithi028 ceph-mon[27810]: pgmap v204: 97 pgs: 97 active+clean; 354 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.2 MiB/s wr, 190 op/s 2024-04-23T14:50:45.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:44 smithi105 ceph-mon[32555]: pgmap v204: 97 pgs: 97 active+clean; 354 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.2 MiB/s wr, 190 op/s 2024-04-23T14:50:45.500 INFO:tasks.workunit.client.0.smithi028.stdout: 1 547064 9.41 MB/sec execute 181 sec latency 22.825 ms 2024-04-23T14:50:45.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:45 smithi038 ceph-mon[32348]: scrub summary: idle 2024-04-23T14:50:45.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:45 smithi028 ceph-mon[27810]: scrub summary: idle 2024-04-23T14:50:46.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:45 smithi105 ceph-mon[32555]: scrub summary: idle 2024-04-23T14:50:46.500 INFO:tasks.workunit.client.0.smithi028.stdout: 1 548253 9.38 MB/sec execute 182 sec latency 16.140 ms 2024-04-23T14:50:46.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:46 smithi038 ceph-mon[32348]: pgmap v205: 97 pgs: 97 active+clean; 354 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.2 MiB/s wr, 125 op/s 2024-04-23T14:50:46.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:46 smithi028 ceph-mon[27810]: pgmap v205: 97 pgs: 97 active+clean; 354 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.2 MiB/s wr, 125 op/s 2024-04-23T14:50:47.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:46 smithi105 ceph-mon[32555]: pgmap v205: 97 pgs: 97 active+clean; 354 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.2 MiB/s wr, 125 op/s 2024-04-23T14:50:47.500 INFO:tasks.workunit.client.0.smithi028.stdout: 1 549946 9.38 MB/sec execute 183 sec latency 71.439 ms 2024-04-23T14:50:48.500 INFO:tasks.workunit.client.0.smithi028.stdout: 1 551521 9.37 MB/sec execute 184 sec latency 29.648 ms 2024-04-23T14:50:48.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:48 smithi038 ceph-mon[32348]: pgmap v206: 97 pgs: 97 active+clean; 360 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 188 op/s 2024-04-23T14:50:48.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:48 smithi028 ceph-mon[27810]: pgmap v206: 97 pgs: 97 active+clean; 360 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 188 op/s 2024-04-23T14:50:49.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:48 smithi105 ceph-mon[32555]: pgmap v206: 97 pgs: 97 active+clean; 360 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 188 op/s 2024-04-23T14:50:49.500 INFO:tasks.workunit.client.0.smithi028.stdout: 1 553565 9.39 MB/sec execute 185 sec latency 17.725 ms 2024-04-23T14:50:50.500 INFO:tasks.workunit.client.0.smithi028.stdout: 1 555075 9.38 MB/sec execute 186 sec latency 15.605 ms 2024-04-23T14:50:50.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:50 smithi038 ceph-mon[32348]: pgmap v207: 97 pgs: 97 active+clean; 360 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 2.2 MiB/s wr, 128 op/s 2024-04-23T14:50:50.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:50 smithi028 ceph-mon[27810]: pgmap v207: 97 pgs: 97 active+clean; 360 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 2.2 MiB/s wr, 128 op/s 2024-04-23T14:50:51.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:50 smithi105 ceph-mon[32555]: pgmap v207: 97 pgs: 97 active+clean; 360 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 2.2 MiB/s wr, 128 op/s 2024-04-23T14:50:51.500 INFO:tasks.workunit.client.0.smithi028.stdout: 1 556590 9.36 MB/sec execute 187 sec latency 49.705 ms 2024-04-23T14:50:52.501 INFO:tasks.workunit.client.0.smithi028.stdout: 1 557989 9.37 MB/sec execute 188 sec latency 37.536 ms 2024-04-23T14:50:53.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:52 smithi105 ceph-mon[32555]: pgmap v208: 97 pgs: 97 active+clean; 360 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 2.2 MiB/s wr, 128 op/s 2024-04-23T14:50:53.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:52 smithi038 ceph-mon[32348]: pgmap v208: 97 pgs: 97 active+clean; 360 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 2.2 MiB/s wr, 128 op/s 2024-04-23T14:50:53.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:52 smithi028 ceph-mon[27810]: pgmap v208: 97 pgs: 97 active+clean; 360 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 2.2 MiB/s wr, 128 op/s 2024-04-23T14:50:53.501 INFO:tasks.workunit.client.0.smithi028.stdout: 1 559140 9.35 MB/sec execute 189 sec latency 32.394 ms 2024-04-23T14:50:54.501 INFO:tasks.workunit.client.0.smithi028.stdout: 1 560870 9.35 MB/sec execute 190 sec latency 22.545 ms 2024-04-23T14:50:55.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:54 smithi105 ceph-mon[32555]: pgmap v209: 97 pgs: 97 active+clean; 359 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 189 op/s 2024-04-23T14:50:55.099 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:54 smithi028 ceph-mon[27810]: pgmap v209: 97 pgs: 97 active+clean; 359 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 189 op/s 2024-04-23T14:50:55.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:54 smithi038 ceph-mon[32348]: pgmap v209: 97 pgs: 97 active+clean; 359 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 189 op/s 2024-04-23T14:50:55.501 INFO:tasks.workunit.client.0.smithi028.stdout: 1 562056 9.33 MB/sec execute 191 sec latency 25.458 ms 2024-04-23T14:50:56.501 INFO:tasks.workunit.client.0.smithi028.stdout: 1 563709 9.32 MB/sec execute 192 sec latency 17.876 ms 2024-04-23T14:50:57.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:56 smithi105 ceph-mon[32555]: pgmap v210: 97 pgs: 97 active+clean; 359 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 1.8 MiB/s wr, 124 op/s 2024-04-23T14:50:57.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:56 smithi038 ceph-mon[32348]: pgmap v210: 97 pgs: 97 active+clean; 359 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 1.8 MiB/s wr, 124 op/s 2024-04-23T14:50:57.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:56 smithi028 ceph-mon[27810]: pgmap v210: 97 pgs: 97 active+clean; 359 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 1.8 MiB/s wr, 124 op/s 2024-04-23T14:50:57.501 INFO:tasks.workunit.client.0.smithi028.stdout: 1 565265 9.33 MB/sec execute 193 sec latency 9.810 ms 2024-04-23T14:50:58.501 INFO:tasks.workunit.client.0.smithi028.stdout: 1 566573 9.31 MB/sec execute 194 sec latency 42.342 ms 2024-04-23T14:50:59.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:50:58 smithi105 ceph-mon[32555]: pgmap v211: 97 pgs: 97 active+clean; 366 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 188 op/s 2024-04-23T14:50:59.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:50:58 smithi038 ceph-mon[32348]: pgmap v211: 97 pgs: 97 active+clean; 366 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 188 op/s 2024-04-23T14:50:59.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:50:58 smithi028 ceph-mon[27810]: pgmap v211: 97 pgs: 97 active+clean; 366 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 188 op/s 2024-04-23T14:50:59.501 INFO:tasks.workunit.client.0.smithi028.stdout: 1 568585 9.33 MB/sec execute 195 sec latency 14.776 ms 2024-04-23T14:51:00.501 INFO:tasks.workunit.client.0.smithi028.stdout: 1 569800 9.31 MB/sec execute 196 sec latency 19.376 ms 2024-04-23T14:51:01.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:00 smithi105 ceph-mon[32555]: pgmap v212: 97 pgs: 97 active+clean; 366 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 125 op/s 2024-04-23T14:51:01.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:00 smithi038 ceph-mon[32348]: pgmap v212: 97 pgs: 97 active+clean; 366 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 125 op/s 2024-04-23T14:51:01.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:00 smithi028 ceph-mon[27810]: pgmap v212: 97 pgs: 97 active+clean; 366 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 125 op/s 2024-04-23T14:51:01.501 INFO:tasks.workunit.client.0.smithi028.stdout: 1 570632 9.27 MB/sec execute 197 sec latency 73.406 ms 2024-04-23T14:51:02.502 INFO:tasks.workunit.client.0.smithi028.stdout: 1 571584 9.26 MB/sec execute 198 sec latency 78.270 ms 2024-04-23T14:51:03.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:02 smithi105 ceph-mon[32555]: pgmap v213: 97 pgs: 97 active+clean; 366 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 125 op/s 2024-04-23T14:51:03.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:02 smithi038 ceph-mon[32348]: pgmap v213: 97 pgs: 97 active+clean; 366 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 125 op/s 2024-04-23T14:51:03.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:02 smithi028 ceph-mon[27810]: pgmap v213: 97 pgs: 97 active+clean; 366 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 125 op/s 2024-04-23T14:51:03.502 INFO:tasks.workunit.client.0.smithi028.stdout: 1 572536 9.24 MB/sec execute 199 sec latency 71.019 ms 2024-04-23T14:51:04.502 INFO:tasks.workunit.client.0.smithi028.stdout: 1 573782 9.22 MB/sec execute 200 sec latency 23.626 ms 2024-04-23T14:51:04.923 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:04 smithi038 ceph-mon[32348]: pgmap v214: 97 pgs: 97 active+clean; 371 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.1 MiB/s wr, 227 op/s 2024-04-23T14:51:05.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:04 smithi105 ceph-mon[32555]: pgmap v214: 97 pgs: 97 active+clean; 371 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.1 MiB/s wr, 227 op/s 2024-04-23T14:51:05.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:04 smithi028 ceph-mon[27810]: pgmap v214: 97 pgs: 97 active+clean; 371 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.1 MiB/s wr, 227 op/s 2024-04-23T14:51:05.502 INFO:tasks.workunit.client.0.smithi028.stdout: 1 575648 9.24 MB/sec execute 201 sec latency 17.513 ms 2024-04-23T14:51:06.502 INFO:tasks.workunit.client.0.smithi028.stdout: 1 576859 9.22 MB/sec execute 202 sec latency 17.039 ms 2024-04-23T14:51:07.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:06 smithi105 ceph-mon[32555]: pgmap v215: 97 pgs: 97 active+clean; 371 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 2.5 MiB/s wr, 166 op/s 2024-04-23T14:51:07.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:06 smithi105 ceph-mon[32555]: osdmap e89: 12 total, 12 up, 12 in 2024-04-23T14:51:07.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:06 smithi038 ceph-mon[32348]: pgmap v215: 97 pgs: 97 active+clean; 371 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 2.5 MiB/s wr, 166 op/s 2024-04-23T14:51:07.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:06 smithi038 ceph-mon[32348]: osdmap e89: 12 total, 12 up, 12 in 2024-04-23T14:51:07.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:06 smithi028 ceph-mon[27810]: pgmap v215: 97 pgs: 97 active+clean; 371 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 2.5 MiB/s wr, 166 op/s 2024-04-23T14:51:07.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:06 smithi028 ceph-mon[27810]: osdmap e89: 12 total, 12 up, 12 in 2024-04-23T14:51:07.503 INFO:tasks.workunit.client.0.smithi028.stdout: 1 578724 9.22 MB/sec execute 203 sec latency 17.369 ms 2024-04-23T14:51:08.502 INFO:tasks.workunit.client.0.smithi028.stdout: 1 579661 9.20 MB/sec execute 204 sec latency 96.774 ms 2024-04-23T14:51:09.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:08 smithi105 ceph-mon[32555]: pgmap v217: 97 pgs: 97 active+clean; 377 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 208 op/s 2024-04-23T14:51:09.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:08 smithi038 ceph-mon[32348]: pgmap v217: 97 pgs: 97 active+clean; 377 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 208 op/s 2024-04-23T14:51:09.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:08 smithi028 ceph-mon[27810]: pgmap v217: 97 pgs: 97 active+clean; 377 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 208 op/s 2024-04-23T14:51:09.503 INFO:tasks.workunit.client.0.smithi028.stdout: 1 581530 9.20 MB/sec execute 205 sec latency 18.553 ms 2024-04-23T14:51:10.503 INFO:tasks.workunit.client.0.smithi028.stdout: 1 583316 9.21 MB/sec execute 206 sec latency 13.574 ms 2024-04-23T14:51:11.038 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:10 smithi105 ceph-mon[32555]: pgmap v218: 97 pgs: 97 active+clean; 377 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 208 op/s 2024-04-23T14:51:11.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:10 smithi038 ceph-mon[32348]: pgmap v218: 97 pgs: 97 active+clean; 377 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 208 op/s 2024-04-23T14:51:11.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:10 smithi028 ceph-mon[27810]: pgmap v218: 97 pgs: 97 active+clean; 377 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 208 op/s 2024-04-23T14:51:11.503 INFO:tasks.workunit.client.0.smithi028.stdout: 1 585298 9.21 MB/sec execute 207 sec latency 16.116 ms 2024-04-23T14:51:11.857 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:51:11.858 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:51:12.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.088+0000 7f086dc00640 1 -- 172.21.15.28:0/1333107411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0868102ac0 msgr2=0x7f0868106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:12.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.088+0000 7f086dc00640 1 --2- 172.21.15.28:0/1333107411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0868102ac0 0x7f0868106b00 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f0850009ca0 tx=0x7f085002a8c0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.088+0000 7f086dc00640 1 -- 172.21.15.28:0/1333107411 shutdown_connections 2024-04-23T14:51:12.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.088+0000 7f086dc00640 1 --2- 172.21.15.28:0/1333107411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0868102ac0 0x7f0868106b00 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.088+0000 7f086dc00640 1 --2- 172.21.15.28:0/1333107411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08681071e0 0x7f086810edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.088+0000 7f086dc00640 1 --2- 172.21.15.28:0/1333107411 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0868102110 0x7f08681024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.088+0000 7f086dc00640 1 -- 172.21.15.28:0/1333107411 >> 172.21.15.28:0/1333107411 conn(0x7f08680fd9d0 msgr2=0x7f08680ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:12.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.088+0000 7f086dc00640 1 -- 172.21.15.28:0/1333107411 shutdown_connections 2024-04-23T14:51:12.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.089+0000 7f086dc00640 1 -- 172.21.15.28:0/1333107411 wait complete. 2024-04-23T14:51:12.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.090+0000 7f086dc00640 1 Processor -- start 2024-04-23T14:51:12.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.090+0000 7f086dc00640 1 -- start start 2024-04-23T14:51:12.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.090+0000 7f086dc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0868102110 0x7f08681a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.090+0000 7f086dc00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08681071e0 0x7f08681a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.090+0000 7f086dc00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f086819a840 0x7f086819acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.090+0000 7f086dc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f08681122a0 con 0x7f0868102110 2024-04-23T14:51:12.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.090+0000 7f086dc00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0868112120 con 0x7f08681071e0 2024-04-23T14:51:12.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.090+0000 7f086dc00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0868112420 con 0x7f086819a840 2024-04-23T14:51:12.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.090+0000 7f0867e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0868102110 0x7f08681a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.090+0000 7f0867e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0868102110 0x7f08681a06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44028/0 (socket says 172.21.15.28:44028) 2024-04-23T14:51:12.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.090+0000 7f0867e00640 1 -- 172.21.15.28:0/3216035096 learned_addr learned my addr 172.21.15.28:0/3216035096 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:12.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.091+0000 7f086d200640 1 --2- 172.21.15.28:0/3216035096 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f086819a840 0x7f086819acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.091+0000 7f0867400640 1 --2- 172.21.15.28:0/3216035096 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08681071e0 0x7f08681a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.091+0000 7f0867e00640 1 -- 172.21.15.28:0/3216035096 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f086819a840 msgr2=0x7f086819acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:12.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.091+0000 7f0867e00640 1 --2- 172.21.15.28:0/3216035096 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f086819a840 0x7f086819acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.091+0000 7f0867e00640 1 -- 172.21.15.28:0/3216035096 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08681071e0 msgr2=0x7f08681a0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:12.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.091+0000 7f0867e00640 1 --2- 172.21.15.28:0/3216035096 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08681071e0 0x7f08681a0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.091+0000 7f0867e00640 1 -- 172.21.15.28:0/3216035096 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f086819b3d0 con 0x7f0868102110 2024-04-23T14:51:12.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.091+0000 7f0867400640 1 --2- 172.21.15.28:0/3216035096 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08681071e0 0x7f08681a0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:51:12.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.091+0000 7f0867e00640 1 --2- 172.21.15.28:0/3216035096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0868102110 0x7f08681a06d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f085800ea60 tx=0x7f085800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:12.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.092+0000 7f0864c00640 1 -- 172.21.15.28:0/3216035096 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085800cde0 con 0x7f0868102110 2024-04-23T14:51:12.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.092+0000 7f086dc00640 1 -- 172.21.15.28:0/3216035096 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f086819b6c0 con 0x7f0868102110 2024-04-23T14:51:12.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.092+0000 7f0864c00640 1 -- 172.21.15.28:0/3216035096 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08580035f0 con 0x7f0868102110 2024-04-23T14:51:12.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.092+0000 7f0864c00640 1 -- 172.21.15.28:0/3216035096 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0858010670 con 0x7f0868102110 2024-04-23T14:51:12.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.092+0000 7f086dc00640 1 -- 172.21.15.28:0/3216035096 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08681a78f0 con 0x7f0868102110 2024-04-23T14:51:12.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.095+0000 7f087200e740 1 Processor -- start 2024-04-23T14:51:12.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.095+0000 7f087200e740 1 -- start start 2024-04-23T14:51:12.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.096+0000 7f0864c00640 1 -- 172.21.15.28:0/3216035096 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f08580037a0 con 0x7f0868102110 2024-04-23T14:51:12.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.096+0000 7f087200e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55956f7b0a30 0x55956f7b0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.096+0000 7f087200e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55956f7b1350 0x55956f7bb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.096+0000 7f087200e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55956f7bba20 0x55956f7bde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.096+0000 7f087200e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55956f7161f0 con 0x55956f7b1350 2024-04-23T14:51:12.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.096+0000 7f087200e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55956f6bc350 con 0x55956f7bba20 2024-04-23T14:51:12.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.096+0000 7f087200e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55956f7121b0 con 0x55956f7b0a30 2024-04-23T14:51:12.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.096+0000 7f0867e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55956f7b1350 0x55956f7bb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.096+0000 7f0867400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55956f7bba20 0x55956f7bde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.096+0000 7f0867e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55956f7b1350 0x55956f7bb4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44040/0 (socket says 172.21.15.28:44040) 2024-04-23T14:51:12.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.096+0000 7f0867e00640 1 -- 172.21.15.28:0/2206705354 learned_addr learned my addr 172.21.15.28:0/2206705354 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:12.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.096+0000 7f086d200640 1 --2- 172.21.15.28:0/2206705354 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55956f7b0a30 0x55956f7b0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.097+0000 7f086d200640 1 -- 172.21.15.28:0/2206705354 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55956f7bba20 msgr2=0x55956f7bde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:12.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.097+0000 7f086d200640 1 --2- 172.21.15.28:0/2206705354 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55956f7bba20 0x55956f7bde10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.097+0000 7f086d200640 1 -- 172.21.15.28:0/2206705354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55956f7b1350 msgr2=0x55956f7bb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:12.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.097+0000 7f086d200640 1 --2- 172.21.15.28:0/2206705354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55956f7b1350 0x55956f7bb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.097+0000 7f086d200640 1 -- 172.21.15.28:0/2206705354 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55956f710f50 con 0x55956f7b0a30 2024-04-23T14:51:12.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.097+0000 7f0867400640 1 --2- 172.21.15.28:0/2206705354 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55956f7bba20 0x55956f7bde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:51:12.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.097+0000 7f086d200640 1 --2- 172.21.15.28:0/2206705354 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55956f7b0a30 0x55956f7b0e10 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0860009b80 tx=0x7f0860011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fc1e43865d5440f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:12.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.097+0000 7f085d600640 1 -- 172.21.15.28:0/2206705354 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0860009d30 con 0x55956f7b0a30 2024-04-23T14:51:12.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.097+0000 7f085d600640 1 -- 172.21.15.28:0/2206705354 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0860005020 con 0x55956f7b0a30 2024-04-23T14:51:12.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.097+0000 7f085d600640 1 -- 172.21.15.28:0/2206705354 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0860004b20 con 0x55956f7b0a30 2024-04-23T14:51:12.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.097+0000 7f0864c00640 1 --2- 172.21.15.28:0/3216035096 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f08480a3d20 0x7f08480a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.098+0000 7f0864c00640 1 -- 172.21.15.28:0/3216035096 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8542+0+0 (secure 0 0 0) 0x7f08580d0be0 con 0x7f0868102110 2024-04-23T14:51:12.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.098+0000 7f0867e00640 1 --2- 172.21.15.28:0/3216035096 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f08480a3d20 0x7f08480a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.098+0000 7f087200e740 1 -- 172.21.15.28:0/2206705354 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55956f7b0a30 msgr2=0x55956f7b0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:12.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.098+0000 7f087200e740 1 --2- 172.21.15.28:0/2206705354 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55956f7b0a30 0x55956f7b0e10 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0860009b80 tx=0x7f0860011810 comp rx=0 tx=0).stop 2024-04-23T14:51:12.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.098+0000 7f087200e740 1 -- 172.21.15.28:0/2206705354 shutdown_connections 2024-04-23T14:51:12.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.098+0000 7f087200e740 1 --2- 172.21.15.28:0/2206705354 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55956f7bba20 0x55956f7bde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.098+0000 7f087200e740 1 --2- 172.21.15.28:0/2206705354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55956f7b1350 0x55956f7bb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.098+0000 7f087200e740 1 --2- 172.21.15.28:0/2206705354 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55956f7b0a30 0x55956f7b0e10 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.098+0000 7f087200e740 1 -- 172.21.15.28:0/2206705354 >> 172.21.15.28:0/2206705354 conn(0x55956f7acd50 msgr2=0x55956f7ad130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:12.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.098+0000 7f087200e740 1 -- 172.21.15.28:0/2206705354 shutdown_connections 2024-04-23T14:51:12.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.098+0000 7f087200e740 1 -- 172.21.15.28:0/2206705354 wait complete. 2024-04-23T14:51:12.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.099+0000 7f0867e00640 1 --2- 172.21.15.28:0/3216035096 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f08480a3d20 0x7f08480a61e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f085800d010 tx=0x7f08580d4690 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:12.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.099+0000 7f087200e740 1 -- start start 2024-04-23T14:51:12.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.099+0000 7f087200e740 1 Processor -- start 2024-04-23T14:51:12.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.100+0000 7f087200e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55956f7b0a30 0x55956f7b89d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.100+0000 7f087200e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55956f7b1350 0x55956f7b8f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.100+0000 7f087200e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55956f7bba20 0x55956f7b9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.100+0000 7f087200e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55956f7be810 con 0x55956f7b1350 2024-04-23T14:51:12.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.100+0000 7f087200e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55956f6cf690 con 0x55956f7bba20 2024-04-23T14:51:12.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.100+0000 7f087200e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55956f7be990 con 0x55956f7b0a30 2024-04-23T14:51:12.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.100+0000 7f087200e740 10 client.0 _ll_register_callbacks cb 0x55956f70b650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:51:12.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.100+0000 7f0867400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55956f7b1350 0x55956f7b8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.100+0000 7f086d200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55956f7b0a30 0x55956f7b89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.100+0000 7f0867400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55956f7b1350 0x55956f7b8f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44056/0 (socket says 172.21.15.28:44056) 2024-04-23T14:51:12.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.100+0000 7f086d200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55956f7b0a30 0x55956f7b89d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:34986/0 (socket says 172.21.15.28:34986) 2024-04-23T14:51:12.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.100+0000 7f0867400640 1 -- 172.21.15.28:0/3146155723 learned_addr learned my addr 172.21.15.28:0/3146155723 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:12.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.100+0000 7f086d200640 1 --2- 172.21.15.28:0/3146155723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55956f7bba20 0x55956f7b9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.101+0000 7f0867400640 1 -- 172.21.15.28:0/3146155723 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55956f7b0a30 msgr2=0x55956f7b89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:12.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.101+0000 7f0867400640 1 --2- 172.21.15.28:0/3146155723 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55956f7b0a30 0x55956f7b89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.101+0000 7f0867400640 1 -- 172.21.15.28:0/3146155723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55956f7bba20 msgr2=0x55956f7b9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:12.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.101+0000 7f0867400640 1 --2- 172.21.15.28:0/3146155723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55956f7bba20 0x55956f7b9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.101+0000 7f0867400640 1 -- 172.21.15.28:0/3146155723 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55956f6ce920 con 0x55956f7b1350 2024-04-23T14:51:12.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.101+0000 7f0867400640 1 --2- 172.21.15.28:0/3146155723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55956f7b1350 0x55956f7b8f10 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f086806b560 tx=0x7f0850009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e52c5d8bf2b56bae server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:12.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.101+0000 7f0856000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-23T14:51:12.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.101+0000 7f0856000640 1 -- 172.21.15.28:0/3146155723 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0850049060 con 0x55956f7b1350 2024-04-23T14:51:12.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.101+0000 7f087200e740 1 -- 172.21.15.28:0/3146155723 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0860005020 con 0x55956f7b1350 2024-04-23T14:51:12.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.101+0000 7f0856000640 1 -- 172.21.15.28:0/3146155723 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8542+0+0 (secure 0 0 0) 0x7f0850003b80 con 0x55956f7b1350 2024-04-23T14:51:12.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.102+0000 7f0856000640 1 -- 172.21.15.28:0/3146155723 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f0850003d80 con 0x55956f7b1350 2024-04-23T14:51:12.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.102+0000 7f087200e740 10 client.16134 fetch_fsmap learned FSMap version 20 2024-04-23T14:51:12.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.102+0000 7f087200e740 1 -- 172.21.15.28:0/3146155723 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55956f4f27d0 con 0x55956f7b1350 2024-04-23T14:51:12.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.102+0000 7f0856000640 1 -- 172.21.15.28:0/3146155723 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0860005020 con 0x55956f7b1350 2024-04-23T14:51:12.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.102+0000 7f0856000640 1 -- 172.21.15.28:0/3146155723 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f08500046c0 con 0x55956f7b1350 2024-04-23T14:51:12.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.102+0000 7f087200e740 10 client.16134 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:51:12.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.102+0000 7f087200e740 10 client.16134 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:51:12.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.102+0000 7f087200e740 20 client.16134 populate_metadata read hostname 'smithi028' 2024-04-23T14:51:12.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.102+0000 7f087200e740 1 --2- 172.21.15.28:0/3146155723 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55956f7cbd60 0x55956f7ce180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.102+0000 7f087200e740 4 client.16134 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:51:12.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.102+0000 7f087200e740 1 -- 172.21.15.28:0/3146155723 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55956f723c10 con 0x55956f7cbd60 2024-04-23T14:51:12.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.103+0000 7f086d200640 1 --2- 172.21.15.28:0/3146155723 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55956f7cbd60 0x55956f7ce180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.103+0000 7f086d200640 1 --2- 172.21.15.28:0/3146155723 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55956f7cbd60 0x55956f7ce180 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cb52e60631d2545 server_cookie=e391cdc6647588cc in_seq=0 out_seq=0 2024-04-23T14:51:12.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.103+0000 7f0856000640 10 client.16134 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:51:12.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.104+0000 7f0856000640 1 -- 172.21.15.28:0/3146155723 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55956f723c10 con 0x55956f7cbd60 2024-04-23T14:51:12.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.104+0000 7f0856000640 10 client.16134 handle_command_reply: tid=0 2024-04-23T14:51:12.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.104+0000 7f087200e740 1 client.16134 shutdown 2024-04-23T14:51:12.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.104+0000 7f087200e740 1 -- 172.21.15.28:0/3146155723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55956f7b1350 msgr2=0x55956f7b8f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:12.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.104+0000 7f087200e740 1 --2- 172.21.15.28:0/3146155723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55956f7b1350 0x55956f7b8f10 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f086806b560 tx=0x7f0850009800 comp rx=0 tx=0).stop 2024-04-23T14:51:12.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.104+0000 7f087200e740 1 -- 172.21.15.28:0/3146155723 shutdown_connections 2024-04-23T14:51:12.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.104+0000 7f087200e740 1 --2- 172.21.15.28:0/3146155723 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55956f7cbd60 0x55956f7ce180 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.104+0000 7f087200e740 1 --2- 172.21.15.28:0/3146155723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55956f7bba20 0x55956f7b9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.104+0000 7f087200e740 1 --2- 172.21.15.28:0/3146155723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55956f7b1350 0x55956f7b8f10 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.104+0000 7f087200e740 1 --2- 172.21.15.28:0/3146155723 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55956f7b0a30 0x55956f7b89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.104+0000 7f0856000640 0 client.16134 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:51:12.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.104+0000 7f087200e740 1 -- 172.21.15.28:0/3146155723 >> 172.21.15.28:0/3146155723 conn(0x55956f7acd50 msgr2=0x55956f7b2b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:12.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.105+0000 7f087200e740 1 -- 172.21.15.28:0/3146155723 shutdown_connections 2024-04-23T14:51:12.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.105+0000 7f087200e740 1 -- 172.21.15.28:0/3146155723 wait complete. 2024-04-23T14:51:12.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.105+0000 7f087200e740 20 client.16134 trim_cache size 0 max 16384 2024-04-23T14:51:12.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.127+0000 7f087200e740 1 Processor -- start 2024-04-23T14:51:12.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.127+0000 7f087200e740 1 -- start start 2024-04-23T14:51:12.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.127+0000 7f087200e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55956f7b0a30 0x55956f7bb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.127+0000 7f087200e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55956f7b1350 0x55956f7c76d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.127+0000 7f087200e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55956f7bba20 0x55956f7b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.127+0000 7f087200e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f08681ade80 con 0x55956f7bba20 2024-04-23T14:51:12.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.127+0000 7f087200e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f08681b06e0 con 0x55956f7b0a30 2024-04-23T14:51:12.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.127+0000 7f087200e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f08681ae4d0 con 0x55956f7b1350 2024-04-23T14:51:12.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.127+0000 7f086d200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55956f7b0a30 0x55956f7bb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.127+0000 7f0867400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55956f7b1350 0x55956f7c76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.127+0000 7f086d200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55956f7b0a30 0x55956f7bb1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:57760/0 (socket says 172.21.15.28:57760) 2024-04-23T14:51:12.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.127+0000 7f0867400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55956f7b1350 0x55956f7c76d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:34996/0 (socket says 172.21.15.28:34996) 2024-04-23T14:51:12.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.127+0000 7f086d200640 1 -- 172.21.15.28:0/996272333 learned_addr learned my addr 172.21.15.28:0/996272333 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:12.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.127+0000 7f086d200640 1 --2- 172.21.15.28:0/996272333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55956f7bba20 0x55956f7b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.128+0000 7f086d200640 1 -- 172.21.15.28:0/996272333 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55956f7b1350 msgr2=0x55956f7c76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:12.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.128+0000 7f086d200640 1 --2- 172.21.15.28:0/996272333 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55956f7b1350 0x55956f7c76d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.128+0000 7f086d200640 1 -- 172.21.15.28:0/996272333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55956f7bba20 msgr2=0x55956f7b31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:12.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.128+0000 7f086d200640 1 --2- 172.21.15.28:0/996272333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55956f7bba20 0x55956f7b31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.128+0000 7f086d200640 1 -- 172.21.15.28:0/996272333 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55956f4283f0 con 0x55956f7b0a30 2024-04-23T14:51:12.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.128+0000 7f0867400640 1 --2- 172.21.15.28:0/996272333 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55956f7b1350 0x55956f7c76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:51:12.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.128+0000 7f086d200640 1 --2- 172.21.15.28:0/996272333 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55956f7b0a30 0x55956f7bb1a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f086001b040 tx=0x7f0860011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ecd4210f02533c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:12.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.128+0000 7f085cc00640 1 -- 172.21.15.28:0/996272333 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0860004970 con 0x55956f7b0a30 2024-04-23T14:51:12.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.128+0000 7f085cc00640 1 -- 172.21.15.28:0/996272333 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0860004b10 con 0x55956f7b0a30 2024-04-23T14:51:12.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.128+0000 7f085cc00640 1 -- 172.21.15.28:0/996272333 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0860028910 con 0x55956f7b0a30 2024-04-23T14:51:12.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.129+0000 7f087200e740 1 -- 172.21.15.28:0/996272333 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55956f7b0a30 msgr2=0x55956f7bb1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:12.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.129+0000 7f087200e740 1 --2- 172.21.15.28:0/996272333 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55956f7b0a30 0x55956f7bb1a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f086001b040 tx=0x7f0860011810 comp rx=0 tx=0).stop 2024-04-23T14:51:12.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.129+0000 7f087200e740 1 -- 172.21.15.28:0/996272333 shutdown_connections 2024-04-23T14:51:12.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.129+0000 7f087200e740 1 --2- 172.21.15.28:0/996272333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55956f7bba20 0x55956f7b31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.129+0000 7f087200e740 1 --2- 172.21.15.28:0/996272333 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55956f7b1350 0x55956f7c76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.129+0000 7f087200e740 1 --2- 172.21.15.28:0/996272333 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55956f7b0a30 0x55956f7bb1a0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.129+0000 7f087200e740 1 -- 172.21.15.28:0/996272333 >> 172.21.15.28:0/996272333 conn(0x55956f7acd50 msgr2=0x55956f7baa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:12.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.129+0000 7f087200e740 1 -- 172.21.15.28:0/996272333 shutdown_connections 2024-04-23T14:51:12.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.129+0000 7f087200e740 1 -- 172.21.15.28:0/996272333 wait complete. 2024-04-23T14:51:12.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.130+0000 7f087200e740 1 -- start start 2024-04-23T14:51:12.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.130+0000 7f087200e740 1 Processor -- start 2024-04-23T14:51:12.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.130+0000 7f087200e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55956f7b0a30 0x55956f7c3be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.130+0000 7f087200e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55956f7b1350 0x55956f7c4120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.130+0000 7f087200e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55956f7bba20 0x55956f7c0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.130+0000 7f087200e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f08681aeba0 con 0x55956f7bba20 2024-04-23T14:51:12.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.130+0000 7f087200e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f08681b0010 con 0x55956f7b0a30 2024-04-23T14:51:12.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.130+0000 7f087200e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f08681af270 con 0x55956f7b1350 2024-04-23T14:51:12.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.130+0000 7f087200e740 10 client.0 _ll_register_callbacks cb 0x7f0858005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:51:12.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.131+0000 7f086d200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55956f7b0a30 0x55956f7c3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.131+0000 7f0867400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55956f7b1350 0x55956f7c4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.131+0000 7f086d200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55956f7b0a30 0x55956f7c3be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:57776/0 (socket says 172.21.15.28:57776) 2024-04-23T14:51:12.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.131+0000 7f086d200640 1 -- 172.21.15.28:0/2933426774 learned_addr learned my addr 172.21.15.28:0/2933426774 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:12.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.131+0000 7f086d200640 1 --2- 172.21.15.28:0/2933426774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55956f7bba20 0x55956f7c0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.131+0000 7f0867400640 1 -- 172.21.15.28:0/2933426774 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55956f7b0a30 msgr2=0x55956f7c3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:12.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.131+0000 7f0867400640 1 --2- 172.21.15.28:0/2933426774 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55956f7b0a30 0x55956f7c3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.131+0000 7f0867400640 1 -- 172.21.15.28:0/2933426774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55956f7bba20 msgr2=0x55956f7c0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:12.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.131+0000 7f0867400640 1 --2- 172.21.15.28:0/2933426774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55956f7bba20 0x55956f7c0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.131+0000 7f0867400640 1 -- 172.21.15.28:0/2933426774 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55956f3e80f0 con 0x55956f7b1350 2024-04-23T14:51:12.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.131+0000 7f0867400640 1 --2- 172.21.15.28:0/2933426774 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55956f7b1350 0x55956f7c4120 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f085002f610 tx=0x7f0850000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e22477f607f9bfa server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:12.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.131+0000 7f0856000640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:51:12.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.131+0000 7f0856000640 1 -- 172.21.15.28:0/2933426774 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0850049070 con 0x55956f7b1350 2024-04-23T14:51:12.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.132+0000 7f087200e740 1 -- 172.21.15.28:0/2933426774 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0860004b10 con 0x55956f7b1350 2024-04-23T14:51:12.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.132+0000 7f0856000640 1 -- 172.21.15.28:0/2933426774 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8542+0+0 (secure 0 0 0) 0x7f0850003b80 con 0x55956f7b1350 2024-04-23T14:51:12.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.132+0000 7f0856000640 1 -- 172.21.15.28:0/2933426774 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f085002af40 con 0x55956f7b1350 2024-04-23T14:51:12.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.132+0000 7f087200e740 10 client.26296 fetch_fsmap learned FSMap version 20 2024-04-23T14:51:12.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.132+0000 7f087200e740 1 -- 172.21.15.28:0/2933426774 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55956f6db810 con 0x55956f7b1350 2024-04-23T14:51:12.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.132+0000 7f0856000640 1 -- 172.21.15.28:0/2933426774 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0860004b10 con 0x55956f7b1350 2024-04-23T14:51:12.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.132+0000 7f0856000640 1 -- 172.21.15.28:0/2933426774 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f0850030a00 con 0x55956f7b1350 2024-04-23T14:51:12.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.132+0000 7f087200e740 10 client.26296 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:51:12.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.132+0000 7f087200e740 10 client.26296 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:51:12.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.133+0000 7f087200e740 20 client.26296 populate_metadata read hostname 'smithi028' 2024-04-23T14:51:12.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.133+0000 7f087200e740 1 --2- 172.21.15.28:0/2933426774 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55956f833190 0x55956f8355b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.133+0000 7f087200e740 4 client.26296 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:51:12.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.133+0000 7f087200e740 1 -- 172.21.15.28:0/2933426774 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55956f71da80 con 0x55956f833190 2024-04-23T14:51:12.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.133+0000 7f086d200640 1 --2- 172.21.15.28:0/2933426774 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55956f833190 0x55956f8355b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.133+0000 7f086d200640 1 --2- 172.21.15.28:0/2933426774 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55956f833190 0x55956f8355b0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f49361f39dbdb7e5 server_cookie=b54893960a833280 in_seq=0 out_seq=0 2024-04-23T14:51:12.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.133+0000 7f0856000640 10 client.26296 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:51:12.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.133+0000 7f0856000640 1 -- 172.21.15.28:0/2933426774 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55956f71da80 con 0x55956f833190 2024-04-23T14:51:12.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.133+0000 7f0856000640 10 client.26296 handle_command_reply: tid=0 2024-04-23T14:51:12.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.133+0000 7f087200e740 1 client.26296 shutdown 2024-04-23T14:51:12.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.134+0000 7f087200e740 1 -- 172.21.15.28:0/2933426774 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55956f7b1350 msgr2=0x55956f7c4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:12.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.134+0000 7f087200e740 1 --2- 172.21.15.28:0/2933426774 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55956f7b1350 0x55956f7c4120 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f085002f610 tx=0x7f0850000f40 comp rx=0 tx=0).stop 2024-04-23T14:51:12.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.134+0000 7f087200e740 1 -- 172.21.15.28:0/2933426774 shutdown_connections 2024-04-23T14:51:12.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.134+0000 7f087200e740 1 --2- 172.21.15.28:0/2933426774 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55956f833190 0x55956f8355b0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.134+0000 7f087200e740 1 --2- 172.21.15.28:0/2933426774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55956f7bba20 0x55956f7c0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.134+0000 7f087200e740 1 --2- 172.21.15.28:0/2933426774 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55956f7b1350 0x55956f7c4120 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.134+0000 7f087200e740 1 --2- 172.21.15.28:0/2933426774 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55956f7b0a30 0x55956f7c3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.134+0000 7f087200e740 1 -- 172.21.15.28:0/2933426774 >> 172.21.15.28:0/2933426774 conn(0x55956f7acd50 msgr2=0x55956f7c66c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:12.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.134+0000 7f0856000640 0 client.26296 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:51:12.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.134+0000 7f087200e740 1 -- 172.21.15.28:0/2933426774 shutdown_connections 2024-04-23T14:51:12.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.134+0000 7f087200e740 1 -- 172.21.15.28:0/2933426774 wait complete. 2024-04-23T14:51:12.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.134+0000 7f087200e740 20 client.26296 trim_cache size 0 max 16384 2024-04-23T14:51:12.140 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:51:12.141 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-23T14:51:12.141 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-23T14:51:12.141 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:51:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.135+0000 7f086dc00640 1 -- 172.21.15.28:0/3216035096 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f08480a3d20 msgr2=0x7f08480a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.135+0000 7f086dc00640 1 --2- 172.21.15.28:0/3216035096 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f08480a3d20 0x7f08480a61e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f085800d010 tx=0x7f08580d4690 comp rx=0 tx=0).stop 2024-04-23T14:51:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.135+0000 7f086dc00640 1 -- 172.21.15.28:0/3216035096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0868102110 msgr2=0x7f08681a06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.135+0000 7f086dc00640 1 --2- 172.21.15.28:0/3216035096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0868102110 0x7f08681a06d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f085800ea60 tx=0x7f085800ef20 comp rx=0 tx=0).stop 2024-04-23T14:51:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.135+0000 7f086dc00640 1 -- 172.21.15.28:0/3216035096 shutdown_connections 2024-04-23T14:51:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.135+0000 7f086dc00640 1 --2- 172.21.15.28:0/3216035096 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f08480a3d20 0x7f08480a61e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.135+0000 7f086dc00640 1 --2- 172.21.15.28:0/3216035096 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08681071e0 0x7f08681a0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.135+0000 7f086dc00640 1 --2- 172.21.15.28:0/3216035096 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f086819a840 0x7f086819acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.135+0000 7f086dc00640 1 --2- 172.21.15.28:0/3216035096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0868102110 0x7f08681a06d0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.135+0000 7f086dc00640 1 -- 172.21.15.28:0/3216035096 >> 172.21.15.28:0/3216035096 conn(0x7f08680fd9d0 msgr2=0x7f08680fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.135+0000 7f086dc00640 1 -- 172.21.15.28:0/3216035096 shutdown_connections 2024-04-23T14:51:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.135+0000 7f086dc00640 1 -- 172.21.15.28:0/3216035096 wait complete. 2024-04-23T14:51:12.153 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-23T14:51:12.153 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:51:12.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.346+0000 7f7fce200640 1 -- 172.21.15.28:0/2345236234 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fc8101780 msgr2=0x7f7fc810e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:12.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.346+0000 7f7fce200640 1 --2- 172.21.15.28:0/2345236234 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fc8101780 0x7f7fc810e4d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4009900 tx=0x7f7fb402a410 comp rx=0 tx=0).stop 2024-04-23T14:51:12.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.346+0000 7f7fce200640 1 -- 172.21.15.28:0/2345236234 shutdown_connections 2024-04-23T14:51:12.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.346+0000 7f7fce200640 1 --2- 172.21.15.28:0/2345236234 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fc8101780 0x7f7fc810e4d0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.346+0000 7f7fce200640 1 --2- 172.21.15.28:0/2345236234 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7fc810ea10 0x7f7fc8110e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.346+0000 7f7fce200640 1 --2- 172.21.15.28:0/2345236234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fc8100e60 0x7f7fc8101240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.346+0000 7f7fce200640 1 -- 172.21.15.28:0/2345236234 >> 172.21.15.28:0/2345236234 conn(0x7f7fc8078060 msgr2=0x7f7fc8078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:12.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.347+0000 7f7fce200640 1 -- 172.21.15.28:0/2345236234 shutdown_connections 2024-04-23T14:51:12.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.347+0000 7f7fce200640 1 -- 172.21.15.28:0/2345236234 wait complete. 2024-04-23T14:51:12.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.347+0000 7f7fce200640 1 Processor -- start 2024-04-23T14:51:12.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.348+0000 7f7fce200640 1 -- start start 2024-04-23T14:51:12.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.348+0000 7f7fce200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fc8100e60 0x7f7fc81a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.348+0000 7f7fce200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7fc8101780 0x7f7fc81a0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.348+0000 7f7fce200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fc810ea10 0x7f7fc819aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.348+0000 7f7fce200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7fc8114660 con 0x7f7fc810ea10 2024-04-23T14:51:12.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.348+0000 7f7fce200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7fc81144e0 con 0x7f7fc8100e60 2024-04-23T14:51:12.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.348+0000 7f7fce200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7fc81147e0 con 0x7f7fc8101780 2024-04-23T14:51:12.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.348+0000 7f7fc7400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fc8100e60 0x7f7fc81a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.348+0000 7f7fc7400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fc8100e60 0x7f7fc81a0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:57796/0 (socket says 172.21.15.28:57796) 2024-04-23T14:51:12.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.348+0000 7f7fc7400640 1 -- 172.21.15.28:0/1891839693 learned_addr learned my addr 172.21.15.28:0/1891839693 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:12.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.348+0000 7f7fc6a00640 1 --2- 172.21.15.28:0/1891839693 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7fc8101780 0x7f7fc81a0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.348+0000 7f7fc7e00640 1 --2- 172.21.15.28:0/1891839693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fc810ea10 0x7f7fc819aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.349+0000 7f7fc7400640 1 -- 172.21.15.28:0/1891839693 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7fc8101780 msgr2=0x7f7fc81a0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:12.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.349+0000 7f7fc7400640 1 --2- 172.21.15.28:0/1891839693 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7fc8101780 0x7f7fc81a0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.349+0000 7f7fc7400640 1 -- 172.21.15.28:0/1891839693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fc810ea10 msgr2=0x7f7fc819aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:12.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.349+0000 7f7fc7400640 1 --2- 172.21.15.28:0/1891839693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fc810ea10 0x7f7fc819aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.349+0000 7f7fc7400640 1 -- 172.21.15.28:0/1891839693 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fc819b180 con 0x7f7fc8100e60 2024-04-23T14:51:12.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.349+0000 7f7fc7400640 1 --2- 172.21.15.28:0/1891839693 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fc8100e60 0x7f7fc81a0850 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f7fb800e9f0 tx=0x7f7fb800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:12.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.349+0000 7f7fbfe00640 1 -- 172.21.15.28:0/1891839693 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb800cd80 con 0x7f7fc8100e60 2024-04-23T14:51:12.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.349+0000 7f7fbfe00640 1 -- 172.21.15.28:0/1891839693 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fb8010910 con 0x7f7fc8100e60 2024-04-23T14:51:12.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.349+0000 7f7fbfe00640 1 -- 172.21.15.28:0/1891839693 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb8018990 con 0x7f7fc8100e60 2024-04-23T14:51:12.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.349+0000 7f7fce200640 1 -- 172.21.15.28:0/1891839693 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fc819b470 con 0x7f7fc8100e60 2024-04-23T14:51:12.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.349+0000 7f7fce200640 1 -- 172.21.15.28:0/1891839693 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fc81a7710 con 0x7f7fc8100e60 2024-04-23T14:51:12.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.351+0000 7f7fce200640 1 -- 172.21.15.28:0/1891839693 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fc8069270 con 0x7f7fc8100e60 2024-04-23T14:51:12.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.353+0000 7f7fbfe00640 1 -- 172.21.15.28:0/1891839693 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f7fb8010ac0 con 0x7f7fc8100e60 2024-04-23T14:51:12.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.354+0000 7f7fbfe00640 1 --2- 172.21.15.28:0/1891839693 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7fac0a3da0 0x7f7fac0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.354+0000 7f7fbfe00640 1 -- 172.21.15.28:0/1891839693 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8542+0+0 (secure 0 0 0) 0x7f7fb80d0af0 con 0x7f7fc8100e60 2024-04-23T14:51:12.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.354+0000 7f7fc6a00640 1 --2- 172.21.15.28:0/1891839693 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7fac0a3da0 0x7f7fac0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.355+0000 7f7fc6a00640 1 --2- 172.21.15.28:0/1891839693 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7fac0a3da0 0x7f7fac0a6260 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4000c00 tx=0x7f7fb4005b50 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:12.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.359+0000 7f7fbfe00640 1 -- 172.21.15.28:0/1891839693 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7fb809cad0 con 0x7f7fc8100e60 2024-04-23T14:51:12.503 INFO:tasks.workunit.client.0.smithi028.stdout: 1 586995 9.21 MB/sec execute 208 sec latency 14.460 ms 2024-04-23T14:51:12.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.634+0000 7f7fce200640 1 -- 172.21.15.28:0/1891839693 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7fc819bf30 con 0x7f7fc8100e60 2024-04-23T14:51:12.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.635+0000 7f7fbfe00640 1 -- 172.21.15.28:0/1891839693 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7f7fb80a0dd0 con 0x7f7fc8100e60 2024-04-23T14:51:12.637 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:51:12.637 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:51:12.637 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:51:12.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.638+0000 7f7fce200640 1 -- 172.21.15.28:0/1891839693 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7fac0a3da0 msgr2=0x7f7fac0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:12.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.638+0000 7f7fce200640 1 --2- 172.21.15.28:0/1891839693 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7fac0a3da0 0x7f7fac0a6260 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4000c00 tx=0x7f7fb4005b50 comp rx=0 tx=0).stop 2024-04-23T14:51:12.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.638+0000 7f7fce200640 1 -- 172.21.15.28:0/1891839693 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fc8100e60 msgr2=0x7f7fc81a0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:12.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.638+0000 7f7fce200640 1 --2- 172.21.15.28:0/1891839693 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fc8100e60 0x7f7fc81a0850 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f7fb800e9f0 tx=0x7f7fb800eeb0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.638+0000 7f7fce200640 1 -- 172.21.15.28:0/1891839693 shutdown_connections 2024-04-23T14:51:12.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.638+0000 7f7fce200640 1 --2- 172.21.15.28:0/1891839693 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7fac0a3da0 0x7f7fac0a6260 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.638+0000 7f7fce200640 1 --2- 172.21.15.28:0/1891839693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fc810ea10 0x7f7fc819aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.638+0000 7f7fce200640 1 --2- 172.21.15.28:0/1891839693 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7fc8101780 0x7f7fc81a0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.638+0000 7f7fce200640 1 --2- 172.21.15.28:0/1891839693 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fc8100e60 0x7f7fc81a0850 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.638+0000 7f7fce200640 1 -- 172.21.15.28:0/1891839693 >> 172.21.15.28:0/1891839693 conn(0x7f7fc8078060 msgr2=0x7f7fc810fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:12.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.638+0000 7f7fce200640 1 -- 172.21.15.28:0/1891839693 shutdown_connections 2024-04-23T14:51:12.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.639+0000 7f7fce200640 1 -- 172.21.15.28:0/1891839693 wait complete. 2024-04-23T14:51:12.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-23T14:51:12.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:12 smithi038 ceph-mon[32348]: pgmap v219: 97 pgs: 97 active+clean; 377 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 208 op/s 2024-04-23T14:51:12.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:12 smithi038 ceph-mon[32348]: osdmap e90: 12 total, 12 up, 12 in 2024-04-23T14:51:12.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:12 smithi028 ceph-mon[27810]: pgmap v219: 97 pgs: 97 active+clean; 377 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 208 op/s 2024-04-23T14:51:12.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:12 smithi028 ceph-mon[27810]: osdmap e90: 12 total, 12 up, 12 in 2024-04-23T14:51:12.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.878+0000 7f7ca9000640 1 -- 172.21.15.28:0/361175873 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ca4102ac0 msgr2=0x7f7ca4106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:12.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.878+0000 7f7ca9000640 1 --2- 172.21.15.28:0/361175873 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ca4102ac0 0x7f7ca4106b00 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f7c90009900 tx=0x7f7c9002a3f0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.878+0000 7f7ca9000640 1 -- 172.21.15.28:0/361175873 shutdown_connections 2024-04-23T14:51:12.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.878+0000 7f7ca9000640 1 --2- 172.21.15.28:0/361175873 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ca4102ac0 0x7f7ca4106b00 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.878+0000 7f7ca9000640 1 --2- 172.21.15.28:0/361175873 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7ca41071e0 0x7f7ca410edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.878+0000 7f7ca9000640 1 --2- 172.21.15.28:0/361175873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ca4102110 0x7f7ca41024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.878+0000 7f7ca9000640 1 -- 172.21.15.28:0/361175873 >> 172.21.15.28:0/361175873 conn(0x7f7ca40fd9d0 msgr2=0x7f7ca40ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:12.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.878+0000 7f7ca9000640 1 -- 172.21.15.28:0/361175873 shutdown_connections 2024-04-23T14:51:12.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.878+0000 7f7ca9000640 1 -- 172.21.15.28:0/361175873 wait complete. 2024-04-23T14:51:12.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.880+0000 7f7ca9000640 1 Processor -- start 2024-04-23T14:51:12.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.880+0000 7f7ca9000640 1 -- start start 2024-04-23T14:51:12.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.880+0000 7f7ca9000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ca4102110 0x7f7ca41a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.880+0000 7f7ca9000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ca41071e0 0x7f7ca41a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.880+0000 7f7ca9000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7ca419a840 0x7f7ca419acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.880+0000 7f7ca9000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca41122a0 con 0x7f7ca4102110 2024-04-23T14:51:12.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.880+0000 7f7ca9000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca4112120 con 0x7f7ca41071e0 2024-04-23T14:51:12.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.880+0000 7f7ca9000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca4112420 con 0x7f7ca419a840 2024-04-23T14:51:12.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.881+0000 7f7ca2a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ca41071e0 0x7f7ca41a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.881+0000 7f7ca3e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7ca419a840 0x7f7ca419acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.881+0000 7f7ca3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ca4102110 0x7f7ca41a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.881+0000 7f7ca2a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ca41071e0 0x7f7ca41a0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:57816/0 (socket says 172.21.15.28:57816) 2024-04-23T14:51:12.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.881+0000 7f7ca3e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7ca419a840 0x7f7ca419acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:35046/0 (socket says 172.21.15.28:35046) 2024-04-23T14:51:12.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.881+0000 7f7ca2a00640 1 -- 172.21.15.28:0/807511322 learned_addr learned my addr 172.21.15.28:0/807511322 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:12.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.881+0000 7f7ca2a00640 1 -- 172.21.15.28:0/807511322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7ca419a840 msgr2=0x7f7ca419acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:12.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.881+0000 7f7ca2a00640 1 --2- 172.21.15.28:0/807511322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7ca419a840 0x7f7ca419acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.881+0000 7f7ca2a00640 1 -- 172.21.15.28:0/807511322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ca4102110 msgr2=0x7f7ca41a06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:12.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.881+0000 7f7ca2a00640 1 --2- 172.21.15.28:0/807511322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ca4102110 0x7f7ca41a06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.881+0000 7f7ca2a00640 1 -- 172.21.15.28:0/807511322 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ca419b3d0 con 0x7f7ca41071e0 2024-04-23T14:51:12.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.881+0000 7f7ca3400640 1 --2- 172.21.15.28:0/807511322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ca4102110 0x7f7ca41a06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:51:12.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.881+0000 7f7ca3e00640 1 --2- 172.21.15.28:0/807511322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7ca419a840 0x7f7ca419acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:51:12.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.882+0000 7f7ca2a00640 1 --2- 172.21.15.28:0/807511322 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ca41071e0 0x7f7ca41a0c10 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f7c9002a910 tx=0x7f7c90003b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:12.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.882+0000 7f7c9be00640 1 -- 172.21.15.28:0/807511322 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c9003f070 con 0x7f7ca41071e0 2024-04-23T14:51:12.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.882+0000 7f7c9be00640 1 -- 172.21.15.28:0/807511322 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c90031c90 con 0x7f7ca41071e0 2024-04-23T14:51:12.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.882+0000 7f7c9be00640 1 -- 172.21.15.28:0/807511322 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c90035030 con 0x7f7ca41071e0 2024-04-23T14:51:12.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.882+0000 7f7ca9000640 1 -- 172.21.15.28:0/807511322 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ca419b660 con 0x7f7ca41071e0 2024-04-23T14:51:12.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.882+0000 7f7ca9000640 1 -- 172.21.15.28:0/807511322 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ca41a77c0 con 0x7f7ca41071e0 2024-04-23T14:51:12.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.886+0000 7f7cad3c9740 1 Processor -- start 2024-04-23T14:51:12.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.886+0000 7f7cad3c9740 1 -- start start 2024-04-23T14:51:12.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.886+0000 7f7c9be00640 1 -- 172.21.15.28:0/807511322 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f7c90042070 con 0x7f7ca41071e0 2024-04-23T14:51:12.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.887+0000 7f7cad3c9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631a341ab40 0x5631a341af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.887+0000 7f7cad3c9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631a341b460 0x5631a34255f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.887+0000 7f7cad3c9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631a3425b30 0x5631a3427f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.887+0000 7f7cad3c9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5631a33801d0 con 0x5631a341ab40 2024-04-23T14:51:12.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.887+0000 7f7cad3c9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5631a3326450 con 0x5631a3425b30 2024-04-23T14:51:12.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.887+0000 7f7cad3c9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5631a337c190 con 0x5631a341b460 2024-04-23T14:51:12.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.887+0000 7f7ca3400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631a341b460 0x5631a34255f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.887+0000 7f7ca3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631a341ab40 0x5631a341af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.887+0000 7f7ca2a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631a3425b30 0x5631a3427f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.887+0000 7f7ca3400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631a341b460 0x5631a34255f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:35058/0 (socket says 172.21.15.28:35058) 2024-04-23T14:51:12.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.887+0000 7f7ca3400640 1 -- 172.21.15.28:0/3126860821 learned_addr learned my addr 172.21.15.28:0/3126860821 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:12.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.887+0000 7f7ca3400640 1 -- 172.21.15.28:0/3126860821 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631a3425b30 msgr2=0x5631a3427f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:12.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.887+0000 7f7ca3400640 1 --2- 172.21.15.28:0/3126860821 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631a3425b30 0x5631a3427f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.887+0000 7f7ca3400640 1 -- 172.21.15.28:0/3126860821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631a341ab40 msgr2=0x5631a341af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:12.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.887+0000 7f7ca3400640 1 --2- 172.21.15.28:0/3126860821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631a341ab40 0x5631a341af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.887+0000 7f7ca3400640 1 -- 172.21.15.28:0/3126860821 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631a33e3310 con 0x5631a341b460 2024-04-23T14:51:12.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.888+0000 7f7ca3400640 1 --2- 172.21.15.28:0/3126860821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631a341b460 0x5631a34255f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7c940115f0 tx=0x7f7c94011ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6bbf74871dfa7aca server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:12.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.888+0000 7f7c98c00640 1 -- 172.21.15.28:0/3126860821 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c9401e070 con 0x5631a341b460 2024-04-23T14:51:12.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.888+0000 7f7c98c00640 1 -- 172.21.15.28:0/3126860821 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c9400fb20 con 0x5631a341b460 2024-04-23T14:51:12.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.888+0000 7f7c98c00640 1 -- 172.21.15.28:0/3126860821 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c94019c40 con 0x5631a341b460 2024-04-23T14:51:12.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.889+0000 7f7cad3c9740 1 -- 172.21.15.28:0/3126860821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631a341b460 msgr2=0x5631a34255f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:12.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.889+0000 7f7cad3c9740 1 --2- 172.21.15.28:0/3126860821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631a341b460 0x5631a34255f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7c940115f0 tx=0x7f7c94011ab0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.889+0000 7f7cad3c9740 1 -- 172.21.15.28:0/3126860821 shutdown_connections 2024-04-23T14:51:12.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.889+0000 7f7cad3c9740 1 --2- 172.21.15.28:0/3126860821 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631a3425b30 0x5631a3427f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.889+0000 7f7cad3c9740 1 --2- 172.21.15.28:0/3126860821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631a341b460 0x5631a34255f0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.889+0000 7f7cad3c9740 1 --2- 172.21.15.28:0/3126860821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631a341ab40 0x5631a341af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.889+0000 7f7cad3c9740 1 -- 172.21.15.28:0/3126860821 >> 172.21.15.28:0/3126860821 conn(0x5631a3416e60 msgr2=0x5631a3417240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:12.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.889+0000 7f7cad3c9740 1 -- 172.21.15.28:0/3126860821 shutdown_connections 2024-04-23T14:51:12.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.889+0000 7f7c9be00640 1 --2- 172.21.15.28:0/807511322 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7c880a3d50 0x7f7c880a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.889+0000 7f7cad3c9740 1 -- 172.21.15.28:0/3126860821 wait complete. 2024-04-23T14:51:12.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.889+0000 7f7c9be00640 1 -- 172.21.15.28:0/807511322 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8542+0+0 (secure 0 0 0) 0x7f7c900f0da0 con 0x7f7ca41071e0 2024-04-23T14:51:12.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.889+0000 7f7ca3400640 1 --2- 172.21.15.28:0/807511322 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7c880a3d50 0x7f7c880a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.890+0000 7f7cad3c9740 1 -- start start 2024-04-23T14:51:12.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.890+0000 7f7cad3c9740 1 Processor -- start 2024-04-23T14:51:12.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.890+0000 7f7ca3400640 1 --2- 172.21.15.28:0/807511322 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7c880a3d50 0x7f7c880a6210 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7c94019720 tx=0x7f7c94000bc0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:12.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.891+0000 7f7cad3c9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631a341ab40 0x5631a3425550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.891+0000 7f7cad3c9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631a341b460 0x5631a3422840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.891+0000 7f7cad3c9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631a3425b30 0x5631a3422e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.891+0000 7f7cad3c9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5631a34289c0 con 0x5631a341ab40 2024-04-23T14:51:12.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.891+0000 7f7cad3c9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5631a3339810 con 0x5631a341b460 2024-04-23T14:51:12.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.891+0000 7f7cad3c9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5631a3428b40 con 0x5631a3425b30 2024-04-23T14:51:12.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.891+0000 7f7cad3c9740 10 client.0 _ll_register_callbacks cb 0x5631a3307140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:51:12.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.891+0000 7f7ca3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631a341ab40 0x5631a3425550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.892+0000 7f7ca3e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631a3425b30 0x5631a3422e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.892+0000 7f7ca2a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631a341b460 0x5631a3422840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.892+0000 7f7ca3e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631a3425b30 0x5631a3422e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:35062/0 (socket says 172.21.15.28:35062) 2024-04-23T14:51:12.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.892+0000 7f7ca3e00640 1 -- 172.21.15.28:0/1970909042 learned_addr learned my addr 172.21.15.28:0/1970909042 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:12.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.892+0000 7f7ca2a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631a341b460 0x5631a3422840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:57836/0 (socket says 172.21.15.28:57836) 2024-04-23T14:51:12.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.892+0000 7f7ca3e00640 1 -- 172.21.15.28:0/1970909042 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631a341b460 msgr2=0x5631a3422840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:12.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.892+0000 7f7ca3e00640 1 --2- 172.21.15.28:0/1970909042 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631a341b460 0x5631a3422840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.892+0000 7f7ca3e00640 1 -- 172.21.15.28:0/1970909042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631a341ab40 msgr2=0x5631a3425550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:12.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.892+0000 7f7ca3e00640 1 --2- 172.21.15.28:0/1970909042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631a341ab40 0x5631a3425550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.892+0000 7f7ca3e00640 1 -- 172.21.15.28:0/1970909042 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631a334a3f0 con 0x5631a3425b30 2024-04-23T14:51:12.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.892+0000 7f7ca3400640 1 --2- 172.21.15.28:0/1970909042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631a341ab40 0x5631a3425550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:51:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.893+0000 7f7ca3e00640 1 --2- 172.21.15.28:0/1970909042 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631a3425b30 0x5631a3422e50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c00fe20 tx=0x7f7c9c012820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a9bbd6bc2f51d4b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.893+0000 7f7c8d600640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:51:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.893+0000 7f7c8d600640 1 -- 172.21.15.28:0/1970909042 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c003970 con 0x5631a3425b30 2024-04-23T14:51:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.893+0000 7f7c8d600640 1 -- 172.21.15.28:0/1970909042 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8542+0+0 (secure 0 0 0) 0x7f7c9c003b10 con 0x5631a3425b30 2024-04-23T14:51:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.893+0000 7f7cad3c9740 1 -- 172.21.15.28:0/1970909042 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c9400fb20 con 0x5631a3425b30 2024-04-23T14:51:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.894+0000 7f7c8d600640 1 -- 172.21.15.28:0/1970909042 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f7c9c003d10 con 0x5631a3425b30 2024-04-23T14:51:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.894+0000 7f7cad3c9740 10 client.26314 fetch_fsmap learned FSMap version 20 2024-04-23T14:51:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.894+0000 7f7cad3c9740 1 -- 172.21.15.28:0/1970909042 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5631a30923f0 con 0x5631a3425b30 2024-04-23T14:51:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.894+0000 7f7c8d600640 1 -- 172.21.15.28:0/1970909042 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c9400fb20 con 0x5631a3425b30 2024-04-23T14:51:12.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.895+0000 7f7c8d600640 1 -- 172.21.15.28:0/1970909042 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f7c9c01c620 con 0x5631a3425b30 2024-04-23T14:51:12.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.895+0000 7f7cad3c9740 10 client.26314 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:51:12.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.895+0000 7f7cad3c9740 10 client.26314 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:51:12.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.895+0000 7f7cad3c9740 20 client.26314 populate_metadata read hostname 'smithi028' 2024-04-23T14:51:12.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.895+0000 7f7cad3c9740 1 --2- 172.21.15.28:0/1970909042 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5631a3435da0 0x5631a34381f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.895+0000 7f7cad3c9740 4 client.26314 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:51:12.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.895+0000 7f7cad3c9740 1 -- 172.21.15.28:0/1970909042 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5631a338dbf0 con 0x5631a3435da0 2024-04-23T14:51:12.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.897+0000 7f7ca3400640 1 --2- 172.21.15.28:0/1970909042 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5631a3435da0 0x5631a34381f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.902+0000 7f7c8d600640 10 client.26314 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:51:12.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.902+0000 7f7ca3400640 1 --2- 172.21.15.28:0/1970909042 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5631a3435da0 0x5631a34381f0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a74adf28abd8f3ff server_cookie=bda53bf9df7713fc in_seq=0 out_seq=0 2024-04-23T14:51:12.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.904+0000 7f7c8d600640 1 -- 172.21.15.28:0/1970909042 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5631a338dbf0 con 0x5631a3435da0 2024-04-23T14:51:12.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.904+0000 7f7c8d600640 10 client.26314 handle_command_reply: tid=0 2024-04-23T14:51:12.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.904+0000 7f7cad3c9740 1 client.26314 shutdown 2024-04-23T14:51:12.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.909+0000 7f7cad3c9740 1 -- 172.21.15.28:0/1970909042 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631a3425b30 msgr2=0x5631a3422e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:12.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.909+0000 7f7cad3c9740 1 --2- 172.21.15.28:0/1970909042 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631a3425b30 0x5631a3422e50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c00fe20 tx=0x7f7c9c012820 comp rx=0 tx=0).stop 2024-04-23T14:51:12.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.910+0000 7f7cad3c9740 1 -- 172.21.15.28:0/1970909042 shutdown_connections 2024-04-23T14:51:12.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.910+0000 7f7cad3c9740 1 --2- 172.21.15.28:0/1970909042 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5631a3435da0 0x5631a34381f0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.910+0000 7f7cad3c9740 1 --2- 172.21.15.28:0/1970909042 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631a341b460 0x5631a3422840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.910+0000 7f7cad3c9740 1 --2- 172.21.15.28:0/1970909042 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631a3425b30 0x5631a3422e50 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.910+0000 7f7cad3c9740 1 --2- 172.21.15.28:0/1970909042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631a341ab40 0x5631a3425550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.910+0000 7f7cad3c9740 1 -- 172.21.15.28:0/1970909042 >> 172.21.15.28:0/1970909042 conn(0x5631a3416e60 msgr2=0x5631a341cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:12.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.910+0000 7f7c8d600640 0 client.26314 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:51:12.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.911+0000 7f7cad3c9740 1 -- 172.21.15.28:0/1970909042 shutdown_connections 2024-04-23T14:51:12.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.911+0000 7f7cad3c9740 1 -- 172.21.15.28:0/1970909042 wait complete. 2024-04-23T14:51:12.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.911+0000 7f7cad3c9740 20 client.26314 trim_cache size 0 max 16384 2024-04-23T14:51:12.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.944+0000 7f7cad3c9740 1 Processor -- start 2024-04-23T14:51:12.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.944+0000 7f7cad3c9740 1 -- start start 2024-04-23T14:51:12.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.944+0000 7f7cad3c9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631a341ab40 0x5631a34318e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.944+0000 7f7cad3c9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631a341b460 0x5631a3425080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.944+0000 7f7cad3c9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631a3425b30 0x5631a343dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.945+0000 7f7cad3c9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca41add60 con 0x5631a341ab40 2024-04-23T14:51:12.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.945+0000 7f7cad3c9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca41b05c0 con 0x5631a341b460 2024-04-23T14:51:12.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.945+0000 7f7cad3c9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca41ae3b0 con 0x5631a3425b30 2024-04-23T14:51:12.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.945+0000 7f7ca3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631a341ab40 0x5631a34318e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.945+0000 7f7ca3e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631a3425b30 0x5631a343dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.945+0000 7f7ca2a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631a341b460 0x5631a3425080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.945+0000 7f7ca3e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631a3425b30 0x5631a343dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:35068/0 (socket says 172.21.15.28:35068) 2024-04-23T14:51:12.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.945+0000 7f7ca2a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631a341b460 0x5631a3425080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:57852/0 (socket says 172.21.15.28:57852) 2024-04-23T14:51:12.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.945+0000 7f7ca3e00640 1 -- 172.21.15.28:0/3072401225 learned_addr learned my addr 172.21.15.28:0/3072401225 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:12.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.945+0000 7f7ca3e00640 1 -- 172.21.15.28:0/3072401225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631a341b460 msgr2=0x5631a3425080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:12.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.945+0000 7f7ca3e00640 1 --2- 172.21.15.28:0/3072401225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631a341b460 0x5631a3425080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.945+0000 7f7ca3e00640 1 -- 172.21.15.28:0/3072401225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631a341ab40 msgr2=0x5631a34318e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:12.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.945+0000 7f7ca3e00640 1 --2- 172.21.15.28:0/3072401225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631a341ab40 0x5631a34318e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.945+0000 7f7ca3e00640 1 -- 172.21.15.28:0/3072401225 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631a315c7d0 con 0x5631a3425b30 2024-04-23T14:51:12.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.945+0000 7f7ca2a00640 1 --2- 172.21.15.28:0/3072401225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631a341b460 0x5631a3425080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:51:12.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.946+0000 7f7ca3e00640 1 --2- 172.21.15.28:0/3072401225 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631a3425b30 0x5631a343dac0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c01ce10 tx=0x7f7c9c01ce40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc9283d53bd3cef5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:12.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.946+0000 7f7c8fe00640 1 -- 172.21.15.28:0/3072401225 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c029070 con 0x5631a3425b30 2024-04-23T14:51:12.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.946+0000 7f7c8fe00640 1 -- 172.21.15.28:0/3072401225 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c9c005b70 con 0x5631a3425b30 2024-04-23T14:51:12.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.946+0000 7f7c8fe00640 1 -- 172.21.15.28:0/3072401225 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c0054a0 con 0x5631a3425b30 2024-04-23T14:51:12.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.947+0000 7f7cad3c9740 1 -- 172.21.15.28:0/3072401225 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631a3425b30 msgr2=0x5631a343dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:12.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.947+0000 7f7cad3c9740 1 --2- 172.21.15.28:0/3072401225 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631a3425b30 0x5631a343dac0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c01ce10 tx=0x7f7c9c01ce40 comp rx=0 tx=0).stop 2024-04-23T14:51:12.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.947+0000 7f7cad3c9740 1 -- 172.21.15.28:0/3072401225 shutdown_connections 2024-04-23T14:51:12.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.947+0000 7f7cad3c9740 1 --2- 172.21.15.28:0/3072401225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631a341b460 0x5631a3425080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.947+0000 7f7cad3c9740 1 --2- 172.21.15.28:0/3072401225 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631a3425b30 0x5631a343dac0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.947+0000 7f7cad3c9740 1 --2- 172.21.15.28:0/3072401225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631a341ab40 0x5631a34318e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.947+0000 7f7cad3c9740 1 -- 172.21.15.28:0/3072401225 >> 172.21.15.28:0/3072401225 conn(0x5631a3416e60 msgr2=0x5631a341db30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:12.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.947+0000 7f7cad3c9740 1 -- 172.21.15.28:0/3072401225 shutdown_connections 2024-04-23T14:51:12.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.947+0000 7f7cad3c9740 1 -- 172.21.15.28:0/3072401225 wait complete. 2024-04-23T14:51:12.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.947+0000 7f7cad3c9740 1 -- start start 2024-04-23T14:51:12.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.948+0000 7f7cad3c9740 1 Processor -- start 2024-04-23T14:51:12.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.948+0000 7f7cad3c9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631a341ab40 0x5631a342dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.948+0000 7f7cad3c9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631a341b460 0x5631a341fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.948+0000 7f7cad3c9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631a3425b30 0x5631a3420020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.948+0000 7f7cad3c9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca41aea80 con 0x5631a341ab40 2024-04-23T14:51:12.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.948+0000 7f7cad3c9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca41afef0 con 0x5631a341b460 2024-04-23T14:51:12.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.948+0000 7f7cad3c9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca41af150 con 0x5631a3425b30 2024-04-23T14:51:12.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.948+0000 7f7cad3c9740 10 client.0 _ll_register_callbacks cb 0x7f7c9c00f4a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:51:12.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.948+0000 7f7ca2a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631a341b460 0x5631a341fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.948+0000 7f7ca3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631a341ab40 0x5631a342dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.948+0000 7f7ca2a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631a341b460 0x5631a341fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:57864/0 (socket says 172.21.15.28:57864) 2024-04-23T14:51:12.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.948+0000 7f7ca2a00640 1 -- 172.21.15.28:0/3234364824 learned_addr learned my addr 172.21.15.28:0/3234364824 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:12.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.948+0000 7f7ca3e00640 1 --2- 172.21.15.28:0/3234364824 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631a3425b30 0x5631a3420020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.949+0000 7f7ca2a00640 1 -- 172.21.15.28:0/3234364824 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631a3425b30 msgr2=0x5631a3420020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:12.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.949+0000 7f7ca2a00640 1 --2- 172.21.15.28:0/3234364824 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631a3425b30 0x5631a3420020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.949+0000 7f7ca2a00640 1 -- 172.21.15.28:0/3234364824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631a341ab40 msgr2=0x5631a342dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:12.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.949+0000 7f7ca2a00640 1 --2- 172.21.15.28:0/3234364824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631a341ab40 0x5631a342dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.949+0000 7f7ca2a00640 1 -- 172.21.15.28:0/3234364824 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631a3420610 con 0x5631a341b460 2024-04-23T14:51:12.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.949+0000 7f7ca3400640 1 --2- 172.21.15.28:0/3234364824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631a341ab40 0x5631a342dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:51:12.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.949+0000 7f7ca2a00640 1 --2- 172.21.15.28:0/3234364824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631a341b460 0x5631a341fae0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f7c900b9220 tx=0x7f7c90005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83037c73a7b365d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:12.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.949+0000 7f7c8d600640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:51:12.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.949+0000 7f7c8d600640 1 -- 172.21.15.28:0/3234364824 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c90049040 con 0x5631a341b460 2024-04-23T14:51:12.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.950+0000 7f7c8d600640 1 -- 172.21.15.28:0/3234364824 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8542+0+0 (secure 0 0 0) 0x7f7c900b9430 con 0x5631a341b460 2024-04-23T14:51:12.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.950+0000 7f7cad3c9740 1 -- 172.21.15.28:0/3234364824 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c9c005b70 con 0x5631a341b460 2024-04-23T14:51:12.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.950+0000 7f7c8d600640 1 -- 172.21.15.28:0/3234364824 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f7c900bc900 con 0x5631a341b460 2024-04-23T14:51:12.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.950+0000 7f7cad3c9740 10 client.26333 fetch_fsmap learned FSMap version 20 2024-04-23T14:51:12.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.950+0000 7f7cad3c9740 1 -- 172.21.15.28:0/3234364824 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5631a3427360 con 0x5631a341b460 2024-04-23T14:51:12.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.950+0000 7f7c8d600640 1 -- 172.21.15.28:0/3234364824 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c9c005b70 con 0x5631a341b460 2024-04-23T14:51:12.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.950+0000 7f7c8d600640 1 -- 172.21.15.28:0/3234364824 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f7c900ee740 con 0x5631a341b460 2024-04-23T14:51:12.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.950+0000 7f7cad3c9740 10 client.26333 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:51:12.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.950+0000 7f7cad3c9740 10 client.26333 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:51:12.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.950+0000 7f7cad3c9740 20 client.26333 populate_metadata read hostname 'smithi028' 2024-04-23T14:51:12.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.950+0000 7f7cad3c9740 1 --2- 172.21.15.28:0/3234364824 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5631a349c980 0x5631a349ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:12.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.950+0000 7f7cad3c9740 4 client.26333 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-23T14:51:12.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.950+0000 7f7cad3c9740 1 -- 172.21.15.28:0/3234364824 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5631a3387a60 con 0x5631a349c980 2024-04-23T14:51:12.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.951+0000 7f7ca3400640 1 --2- 172.21.15.28:0/3234364824 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5631a349c980 0x5631a349ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:12.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.951+0000 7f7ca3400640 1 --2- 172.21.15.28:0/3234364824 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5631a349c980 0x5631a349ed70 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=837337fc74f6b6de server_cookie=773c919d76077294 in_seq=0 out_seq=0 2024-04-23T14:51:12.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.951+0000 7f7c8d600640 10 client.26333 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:51:12.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.953+0000 7f7c8d600640 1 -- 172.21.15.28:0/3234364824 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5631a3387a60 con 0x5631a349c980 2024-04-23T14:51:12.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.953+0000 7f7c8d600640 10 client.26333 handle_command_reply: tid=0 2024-04-23T14:51:12.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.953+0000 7f7cad3c9740 1 client.26333 shutdown 2024-04-23T14:51:12.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.954+0000 7f7cad3c9740 1 -- 172.21.15.28:0/3234364824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631a341b460 msgr2=0x5631a341fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:12.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.954+0000 7f7cad3c9740 1 --2- 172.21.15.28:0/3234364824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631a341b460 0x5631a341fae0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f7c900b9220 tx=0x7f7c90005d50 comp rx=0 tx=0).stop 2024-04-23T14:51:12.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.954+0000 7f7cad3c9740 1 -- 172.21.15.28:0/3234364824 shutdown_connections 2024-04-23T14:51:12.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.954+0000 7f7cad3c9740 1 --2- 172.21.15.28:0/3234364824 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5631a349c980 0x5631a349ed70 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.954+0000 7f7cad3c9740 1 --2- 172.21.15.28:0/3234364824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631a341b460 0x5631a341fae0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.954+0000 7f7cad3c9740 1 --2- 172.21.15.28:0/3234364824 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5631a3425b30 0x5631a3420020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.954+0000 7f7c8d600640 0 client.26333 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:51:12.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.954+0000 7f7cad3c9740 1 --2- 172.21.15.28:0/3234364824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5631a341ab40 0x5631a342dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.954+0000 7f7cad3c9740 1 -- 172.21.15.28:0/3234364824 >> 172.21.15.28:0/3234364824 conn(0x5631a3416e60 msgr2=0x5631a34210b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:12.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.954+0000 7f7cad3c9740 1 -- 172.21.15.28:0/3234364824 shutdown_connections 2024-04-23T14:51:12.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.954+0000 7f7cad3c9740 1 -- 172.21.15.28:0/3234364824 wait complete. 2024-04-23T14:51:12.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.954+0000 7f7cad3c9740 20 client.26333 trim_cache size 0 max 16384 2024-04-23T14:51:12.958 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-23T14:51:12.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.955+0000 7f7ca9000640 1 -- 172.21.15.28:0/807511322 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7c880a3d50 msgr2=0x7f7c880a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:12.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.955+0000 7f7ca9000640 1 --2- 172.21.15.28:0/807511322 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7c880a3d50 0x7f7c880a6210 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7c94019720 tx=0x7f7c94000bc0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.955+0000 7f7ca9000640 1 -- 172.21.15.28:0/807511322 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ca41071e0 msgr2=0x7f7ca41a0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:12.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.955+0000 7f7ca9000640 1 --2- 172.21.15.28:0/807511322 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ca41071e0 0x7f7ca41a0c10 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f7c9002a910 tx=0x7f7c90003b20 comp rx=0 tx=0).stop 2024-04-23T14:51:12.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.955+0000 7f7ca9000640 1 -- 172.21.15.28:0/807511322 shutdown_connections 2024-04-23T14:51:12.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.955+0000 7f7ca9000640 1 --2- 172.21.15.28:0/807511322 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7c880a3d50 0x7f7c880a6210 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.955+0000 7f7ca9000640 1 --2- 172.21.15.28:0/807511322 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ca41071e0 0x7f7ca41a0c10 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.955+0000 7f7ca9000640 1 --2- 172.21.15.28:0/807511322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7ca419a840 0x7f7ca419acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.955+0000 7f7ca9000640 1 --2- 172.21.15.28:0/807511322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ca4102110 0x7f7ca41a06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:12.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.955+0000 7f7ca9000640 1 -- 172.21.15.28:0/807511322 >> 172.21.15.28:0/807511322 conn(0x7f7ca40fd9d0 msgr2=0x7f7ca40fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:12.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.955+0000 7f7ca9000640 1 -- 172.21.15.28:0/807511322 shutdown_connections 2024-04-23T14:51:12.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:12.955+0000 7f7ca9000640 1 -- 172.21.15.28:0/807511322 wait complete. 2024-04-23T14:51:13.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:12 smithi105 ceph-mon[32555]: pgmap v219: 97 pgs: 97 active+clean; 377 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 208 op/s 2024-04-23T14:51:13.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:12 smithi105 ceph-mon[32555]: osdmap e90: 12 total, 12 up, 12 in 2024-04-23T14:51:13.503 INFO:tasks.workunit.client.0.smithi028.stdout: 1 588654 9.20 MB/sec execute 209 sec latency 75.896 ms 2024-04-23T14:51:13.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:13 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1891839693' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:51:13.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:13 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1891839693' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:51:13.974 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-23T14:51:13.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:51:13.975 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-23T14:51:14.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:13 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1891839693' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:51:14.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.165+0000 7f1f68600640 1 -- 172.21.15.28:0/3166283489 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f60107360 msgr2=0x7f1f6010ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:14.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.165+0000 7f1f68600640 1 --2- 172.21.15.28:0/3166283489 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f60107360 0x7f1f6010ef30 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1f58009860 tx=0x7f1f5802a340 comp rx=0 tx=0).stop 2024-04-23T14:51:14.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.165+0000 7f1f68600640 1 -- 172.21.15.28:0/3166283489 shutdown_connections 2024-04-23T14:51:14.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.165+0000 7f1f68600640 1 --2- 172.21.15.28:0/3166283489 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f60107360 0x7f1f6010ef30 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.165+0000 7f1f68600640 1 --2- 172.21.15.28:0/3166283489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f60102c40 0x7f1f60106c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.165+0000 7f1f68600640 1 --2- 172.21.15.28:0/3166283489 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f60102290 0x7f1f60102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.165+0000 7f1f68600640 1 -- 172.21.15.28:0/3166283489 >> 172.21.15.28:0/3166283489 conn(0x7f1f600fdb10 msgr2=0x7f1f600fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.165+0000 7f1f68600640 1 -- 172.21.15.28:0/3166283489 shutdown_connections 2024-04-23T14:51:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.165+0000 7f1f68600640 1 -- 172.21.15.28:0/3166283489 wait complete. 2024-04-23T14:51:14.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.166+0000 7f1f68600640 1 Processor -- start 2024-04-23T14:51:14.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.167+0000 7f1f68600640 1 -- start start 2024-04-23T14:51:14.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.167+0000 7f1f68600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f60102290 0x7f1f601a0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.167+0000 7f1f68600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f60107360 0x7f1f601a0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.167+0000 7f1f68600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f6019a940 0x7f1f6019adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.167+0000 7f1f68600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1f60112400 con 0x7f1f60107360 2024-04-23T14:51:14.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.167+0000 7f1f68600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1f60112280 con 0x7f1f60102290 2024-04-23T14:51:14.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.167+0000 7f1f68600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1f60112580 con 0x7f1f6019a940 2024-04-23T14:51:14.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.167+0000 7f1f65c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f60102290 0x7f1f601a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.167+0000 7f1f65c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f60102290 0x7f1f601a0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:57876/0 (socket says 172.21.15.28:57876) 2024-04-23T14:51:14.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.167+0000 7f1f65c00640 1 -- 172.21.15.28:0/2555316945 learned_addr learned my addr 172.21.15.28:0/2555316945 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:14.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.167+0000 7f1f65200640 1 --2- 172.21.15.28:0/2555316945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f60107360 0x7f1f601a0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.167+0000 7f1f66600640 1 --2- 172.21.15.28:0/2555316945 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f6019a940 0x7f1f6019adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.168+0000 7f1f65c00640 1 -- 172.21.15.28:0/2555316945 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f6019a940 msgr2=0x7f1f6019adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:14.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.168+0000 7f1f65c00640 1 --2- 172.21.15.28:0/2555316945 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f6019a940 0x7f1f6019adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.168+0000 7f1f65c00640 1 -- 172.21.15.28:0/2555316945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f60107360 msgr2=0x7f1f601a0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:14.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.168+0000 7f1f65c00640 1 --2- 172.21.15.28:0/2555316945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f60107360 0x7f1f601a0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.168+0000 7f1f65c00640 1 -- 172.21.15.28:0/2555316945 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f6019b460 con 0x7f1f60102290 2024-04-23T14:51:14.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.168+0000 7f1f65200640 1 --2- 172.21.15.28:0/2555316945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f60107360 0x7f1f601a0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:51:14.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.168+0000 7f1f65c00640 1 --2- 172.21.15.28:0/2555316945 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f60102290 0x7f1f601a0790 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f1f5000c420 tx=0x7f1f50009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:14.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.168+0000 7f1f5e000640 1 -- 172.21.15.28:0/2555316945 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f50003970 con 0x7f1f60102290 2024-04-23T14:51:14.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.168+0000 7f1f5e000640 1 -- 172.21.15.28:0/2555316945 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f50004020 con 0x7f1f60102290 2024-04-23T14:51:14.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.168+0000 7f1f68600640 1 -- 172.21.15.28:0/2555316945 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f6019b750 con 0x7f1f60102290 2024-04-23T14:51:14.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.168+0000 7f1f5e000640 1 -- 172.21.15.28:0/2555316945 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f5000bad0 con 0x7f1f60102290 2024-04-23T14:51:14.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.168+0000 7f1f68600640 1 -- 172.21.15.28:0/2555316945 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f601a79f0 con 0x7f1f60102290 2024-04-23T14:51:14.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.171+0000 7f1f5e000640 1 -- 172.21.15.28:0/2555316945 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f1f5001d020 con 0x7f1f60102290 2024-04-23T14:51:14.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.172+0000 7f1f6b438740 1 Processor -- start 2024-04-23T14:51:14.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.172+0000 7f1f6b438740 1 -- start start 2024-04-23T14:51:14.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.172+0000 7f1f6b438740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d143383090 0x55d143383470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.173+0000 7f1f6b438740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1433839b0 0x55d14338db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.173+0000 7f1f6b438740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14338e080 0x55d143390470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.173+0000 7f1f6b438740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d1432e8e70 con 0x55d1433839b0 2024-04-23T14:51:14.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.173+0000 7f1f6b438740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d14328f380 con 0x55d143383090 2024-04-23T14:51:14.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.173+0000 7f1f6b438740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d1432e4e30 con 0x55d14338e080 2024-04-23T14:51:14.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.173+0000 7f1f66600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d143383090 0x55d143383470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.173+0000 7f1f66600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d143383090 0x55d143383470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:57890/0 (socket says 172.21.15.28:57890) 2024-04-23T14:51:14.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.173+0000 7f1f66600640 1 -- 172.21.15.28:0/3687450142 learned_addr learned my addr 172.21.15.28:0/3687450142 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:14.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.173+0000 7f1f65c00640 1 --2- 172.21.15.28:0/3687450142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1433839b0 0x55d14338db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.173+0000 7f1f65200640 1 --2- 172.21.15.28:0/3687450142 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14338e080 0x55d143390470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.173+0000 7f1f5e000640 1 --2- 172.21.15.28:0/2555316945 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f1f440a3da0 0x7f1f440a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.173+0000 7f1f66600640 1 -- 172.21.15.28:0/3687450142 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14338e080 msgr2=0x55d143390470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:14.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.173+0000 7f1f66600640 1 --2- 172.21.15.28:0/3687450142 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14338e080 0x55d143390470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.173+0000 7f1f66600640 1 -- 172.21.15.28:0/3687450142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1433839b0 msgr2=0x55d14338db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:14.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.173+0000 7f1f66600640 1 --2- 172.21.15.28:0/3687450142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1433839b0 0x55d14338db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.173+0000 7f1f66600640 1 -- 172.21.15.28:0/3687450142 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1430c37e0 con 0x55d143383090 2024-04-23T14:51:14.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.173+0000 7f1f65200640 1 --2- 172.21.15.28:0/3687450142 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14338e080 0x55d143390470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:51:14.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.174+0000 7f1f5e000640 1 -- 172.21.15.28:0/2555316945 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8542+0+0 (secure 0 0 0) 0x7f1f500d06f0 con 0x7f1f60102290 2024-04-23T14:51:14.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.174+0000 7f1f66600640 1 --2- 172.21.15.28:0/2555316945 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f1f440a3da0 0x7f1f440a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.174+0000 7f1f66600640 1 --2- 172.21.15.28:0/3687450142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d143383090 0x55d143383470 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f1f58035040 tx=0x7f1f58009490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=389a5e765af6af1d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:14.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.175+0000 7f1f56a00640 1 -- 172.21.15.28:0/3687450142 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f5804f070 con 0x55d143383090 2024-04-23T14:51:14.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.175+0000 7f1f56a00640 1 -- 172.21.15.28:0/3687450142 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f58030750 con 0x55d143383090 2024-04-23T14:51:14.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.175+0000 7f1f56a00640 1 -- 172.21.15.28:0/3687450142 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f58037040 con 0x55d143383090 2024-04-23T14:51:14.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.175+0000 7f1f6b438740 1 -- 172.21.15.28:0/3687450142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d143383090 msgr2=0x55d143383470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:14.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.175+0000 7f1f6b438740 1 --2- 172.21.15.28:0/3687450142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d143383090 0x55d143383470 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f1f58035040 tx=0x7f1f58009490 comp rx=0 tx=0).stop 2024-04-23T14:51:14.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.175+0000 7f1f66600640 1 --2- 172.21.15.28:0/2555316945 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f1f440a3da0 0x7f1f440a6260 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1f58049000 tx=0x7f1f5804c000 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:14.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.176+0000 7f1f6b438740 1 -- 172.21.15.28:0/3687450142 shutdown_connections 2024-04-23T14:51:14.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.176+0000 7f1f6b438740 1 --2- 172.21.15.28:0/3687450142 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14338e080 0x55d143390470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.176+0000 7f1f6b438740 1 --2- 172.21.15.28:0/3687450142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1433839b0 0x55d14338db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.176+0000 7f1f6b438740 1 --2- 172.21.15.28:0/3687450142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d143383090 0x55d143383470 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.176+0000 7f1f6b438740 1 -- 172.21.15.28:0/3687450142 >> 172.21.15.28:0/3687450142 conn(0x55d14337f3b0 msgr2=0x55d14337f790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:14.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.176+0000 7f1f6b438740 1 -- 172.21.15.28:0/3687450142 shutdown_connections 2024-04-23T14:51:14.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.176+0000 7f1f6b438740 1 -- 172.21.15.28:0/3687450142 wait complete. 2024-04-23T14:51:14.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.176+0000 7f1f6b438740 1 -- start start 2024-04-23T14:51:14.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.176+0000 7f1f6b438740 1 Processor -- start 2024-04-23T14:51:14.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.177+0000 7f1f6b438740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d143383090 0x55d14338afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.177+0000 7f1f6b438740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1433839b0 0x55d14338b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.177+0000 7f1f6b438740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14338e080 0x55d14338ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.177+0000 7f1f6b438740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d143390e70 con 0x55d1433839b0 2024-04-23T14:51:14.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.177+0000 7f1f6b438740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d1432a2660 con 0x55d143383090 2024-04-23T14:51:14.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.177+0000 7f1f6b438740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d143390ff0 con 0x55d14338e080 2024-04-23T14:51:14.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.177+0000 7f1f6b438740 10 client.0 _ll_register_callbacks cb 0x55d143290aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:51:14.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.177+0000 7f1f65c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d143383090 0x55d14338afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.177+0000 7f1f65c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d143383090 0x55d14338afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:57902/0 (socket says 172.21.15.28:57902) 2024-04-23T14:51:14.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.177+0000 7f1f65c00640 1 -- 172.21.15.28:0/718964152 learned_addr learned my addr 172.21.15.28:0/718964152 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:14.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.177+0000 7f1f65200640 1 --2- 172.21.15.28:0/718964152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1433839b0 0x55d14338b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.177+0000 7f1f66600640 1 --2- 172.21.15.28:0/718964152 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14338e080 0x55d14338ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.178+0000 7f1f65c00640 1 -- 172.21.15.28:0/718964152 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14338e080 msgr2=0x55d14338ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:14.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.178+0000 7f1f65c00640 1 --2- 172.21.15.28:0/718964152 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14338e080 0x55d14338ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.178+0000 7f1f65c00640 1 -- 172.21.15.28:0/718964152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1433839b0 msgr2=0x55d14338b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:14.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.178+0000 7f1f65c00640 1 --2- 172.21.15.28:0/718964152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1433839b0 0x55d14338b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.178+0000 7f1f65c00640 1 -- 172.21.15.28:0/718964152 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d143123f70 con 0x55d143383090 2024-04-23T14:51:14.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.178+0000 7f1f65c00640 1 --2- 172.21.15.28:0/718964152 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d143383090 0x55d14338afb0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f1f5000c3f0 tx=0x7f1f500988c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78c5703ff184f6a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:14.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.178+0000 7f1f4b400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:51:14.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.178+0000 7f1f4b400640 1 -- 172.21.15.28:0/718964152 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f5009bf80 con 0x55d143383090 2024-04-23T14:51:14.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.178+0000 7f1f4b400640 1 -- 172.21.15.28:0/718964152 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8542+0+0 (secure 0 0 0) 0x7f1f500a4e00 con 0x55d143383090 2024-04-23T14:51:14.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.178+0000 7f1f6b438740 1 -- 172.21.15.28:0/718964152 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f58030750 con 0x55d143383090 2024-04-23T14:51:14.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.179+0000 7f1f4b400640 1 -- 172.21.15.28:0/718964152 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f1f500ce250 con 0x55d143383090 2024-04-23T14:51:14.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.179+0000 7f1f6b438740 10 client.26345 fetch_fsmap learned FSMap version 20 2024-04-23T14:51:14.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.179+0000 7f1f6b438740 1 -- 172.21.15.28:0/718964152 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55d142fbb170 con 0x55d143383090 2024-04-23T14:51:14.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.179+0000 7f1f4b400640 1 -- 172.21.15.28:0/718964152 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f58030750 con 0x55d143383090 2024-04-23T14:51:14.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.179+0000 7f1f4b400640 1 -- 172.21.15.28:0/718964152 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f1f5000b040 con 0x55d143383090 2024-04-23T14:51:14.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.179+0000 7f1f6b438740 10 client.26345 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:51:14.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.179+0000 7f1f6b438740 10 client.26345 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:51:14.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.179+0000 7f1f6b438740 20 client.26345 populate_metadata read hostname 'smithi028' 2024-04-23T14:51:14.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.179+0000 7f1f6b438740 1 --2- 172.21.15.28:0/718964152 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d14339e400 0x55d1433a0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.179+0000 7f1f6b438740 4 client.26345 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:51:14.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.179+0000 7f1f6b438740 1 -- 172.21.15.28:0/718964152 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d1432f6890 con 0x55d14339e400 2024-04-23T14:51:14.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.180+0000 7f1f65200640 1 --2- 172.21.15.28:0/718964152 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d14339e400 0x55d1433a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.180+0000 7f1f65200640 1 --2- 172.21.15.28:0/718964152 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d14339e400 0x55d1433a0870 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f02ca5b045bc2d3 server_cookie=37a77772c8891ca1 in_seq=0 out_seq=0 2024-04-23T14:51:14.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.180+0000 7f1f4b400640 10 client.26345 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:51:14.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.181+0000 7f1f4b400640 1 -- 172.21.15.28:0/718964152 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d1432f6890 con 0x55d14339e400 2024-04-23T14:51:14.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.181+0000 7f1f4b400640 10 client.26345 handle_command_reply: tid=0 2024-04-23T14:51:14.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.181+0000 7f1f6b438740 1 client.26345 shutdown 2024-04-23T14:51:14.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.182+0000 7f1f6b438740 1 -- 172.21.15.28:0/718964152 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d143383090 msgr2=0x55d14338afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:14.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.182+0000 7f1f6b438740 1 --2- 172.21.15.28:0/718964152 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d143383090 0x55d14338afb0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f1f5000c3f0 tx=0x7f1f500988c0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.182+0000 7f1f6b438740 1 -- 172.21.15.28:0/718964152 shutdown_connections 2024-04-23T14:51:14.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.182+0000 7f1f6b438740 1 --2- 172.21.15.28:0/718964152 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d14339e400 0x55d1433a0870 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.182+0000 7f1f6b438740 1 --2- 172.21.15.28:0/718964152 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14338e080 0x55d14338ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.182+0000 7f1f6b438740 1 --2- 172.21.15.28:0/718964152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1433839b0 0x55d14338b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.182+0000 7f1f6b438740 1 --2- 172.21.15.28:0/718964152 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d143383090 0x55d14338afb0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.182+0000 7f1f4b400640 0 client.26345 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:51:14.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.182+0000 7f1f6b438740 1 -- 172.21.15.28:0/718964152 >> 172.21.15.28:0/718964152 conn(0x55d14337f3b0 msgr2=0x55d1433850b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:14.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.182+0000 7f1f6b438740 1 -- 172.21.15.28:0/718964152 shutdown_connections 2024-04-23T14:51:14.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.182+0000 7f1f6b438740 1 -- 172.21.15.28:0/718964152 wait complete. 2024-04-23T14:51:14.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.182+0000 7f1f6b438740 20 client.26345 trim_cache size 0 max 16384 2024-04-23T14:51:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.203+0000 7f1f6b438740 1 Processor -- start 2024-04-23T14:51:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.203+0000 7f1f6b438740 1 -- start start 2024-04-23T14:51:14.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.203+0000 7f1f6b438740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d143383090 0x55d143399d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.203+0000 7f1f6b438740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1433839b0 0x55d1433a6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.203+0000 7f1f6b438740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14338e080 0x55d1433a6680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.203+0000 7f1f6b438740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1f601ae030 con 0x55d143383090 2024-04-23T14:51:14.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.203+0000 7f1f6b438740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1f601b0890 con 0x55d1433839b0 2024-04-23T14:51:14.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.203+0000 7f1f6b438740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1f601ae680 con 0x55d14338e080 2024-04-23T14:51:14.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.203+0000 7f1f65c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d143383090 0x55d143399d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.204+0000 7f1f66600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14338e080 0x55d1433a6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.204+0000 7f1f65200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1433839b0 0x55d1433a6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.204+0000 7f1f65c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d143383090 0x55d143399d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44210/0 (socket says 172.21.15.28:44210) 2024-04-23T14:51:14.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.204+0000 7f1f65c00640 1 -- 172.21.15.28:0/688366802 learned_addr learned my addr 172.21.15.28:0/688366802 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:14.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.204+0000 7f1f66600640 1 -- 172.21.15.28:0/688366802 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1433839b0 msgr2=0x55d1433a6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:14.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.204+0000 7f1f66600640 1 --2- 172.21.15.28:0/688366802 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1433839b0 0x55d1433a6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.204+0000 7f1f66600640 1 -- 172.21.15.28:0/688366802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d143383090 msgr2=0x55d143399d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:14.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.204+0000 7f1f66600640 1 --2- 172.21.15.28:0/688366802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d143383090 0x55d143399d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.204+0000 7f1f66600640 1 -- 172.21.15.28:0/688366802 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d14337ad10 con 0x55d14338e080 2024-04-23T14:51:14.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.204+0000 7f1f66600640 1 --2- 172.21.15.28:0/688366802 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14338e080 0x55d1433a6680 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1f580057c0 tx=0x7f1f58050000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca828eead8e2f125 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:14.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.204+0000 7f1f56000640 1 -- 172.21.15.28:0/688366802 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f58037040 con 0x55d14338e080 2024-04-23T14:51:14.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.205+0000 7f1f56000640 1 -- 172.21.15.28:0/688366802 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f58003ad0 con 0x55d14338e080 2024-04-23T14:51:14.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.205+0000 7f1f56000640 1 -- 172.21.15.28:0/688366802 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f58032730 con 0x55d14338e080 2024-04-23T14:51:14.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.205+0000 7f1f6b438740 1 -- 172.21.15.28:0/688366802 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14338e080 msgr2=0x55d1433a6680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:14.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.205+0000 7f1f6b438740 1 --2- 172.21.15.28:0/688366802 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14338e080 0x55d1433a6680 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1f580057c0 tx=0x7f1f58050000 comp rx=0 tx=0).stop 2024-04-23T14:51:14.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.205+0000 7f1f6b438740 1 -- 172.21.15.28:0/688366802 shutdown_connections 2024-04-23T14:51:14.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.205+0000 7f1f6b438740 1 --2- 172.21.15.28:0/688366802 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1433839b0 0x55d1433a6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.205+0000 7f1f6b438740 1 --2- 172.21.15.28:0/688366802 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d14338e080 0x55d1433a6680 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.205+0000 7f1f6b438740 1 --2- 172.21.15.28:0/688366802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d143383090 0x55d143399d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.205+0000 7f1f6b438740 1 -- 172.21.15.28:0/688366802 >> 172.21.15.28:0/688366802 conn(0x55d14337f3b0 msgr2=0x55d14337e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:14.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.206+0000 7f1f6b438740 1 -- 172.21.15.28:0/688366802 shutdown_connections 2024-04-23T14:51:14.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.206+0000 7f1f6b438740 1 -- 172.21.15.28:0/688366802 wait complete. 2024-04-23T14:51:14.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.206+0000 7f1f6b438740 1 -- start start 2024-04-23T14:51:14.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.206+0000 7f1f6b438740 1 Processor -- start 2024-04-23T14:51:14.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.207+0000 7f1f6b438740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d143383090 0x55d143391eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.207+0000 7f1f6b438740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1433839b0 0x55d1433923f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.207+0000 7f1f6b438740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14338e080 0x55d143392930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.207+0000 7f1f6b438740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1f601aed50 con 0x55d1433839b0 2024-04-23T14:51:14.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.207+0000 7f1f6b438740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1f601b01c0 con 0x55d14338e080 2024-04-23T14:51:14.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.207+0000 7f1f6b438740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1f601af420 con 0x55d143383090 2024-04-23T14:51:14.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.207+0000 7f1f6b438740 10 client.0 _ll_register_callbacks cb 0x7f1f50098200 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:51:14.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.207+0000 7f1f65c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d143383090 0x55d143391eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.207+0000 7f1f65c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d143383090 0x55d143391eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:35144/0 (socket says 172.21.15.28:35144) 2024-04-23T14:51:14.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.207+0000 7f1f65c00640 1 -- 172.21.15.28:0/1508315203 learned_addr learned my addr 172.21.15.28:0/1508315203 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:14.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.207+0000 7f1f66600640 1 --2- 172.21.15.28:0/1508315203 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14338e080 0x55d143392930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.207+0000 7f1f65200640 1 --2- 172.21.15.28:0/1508315203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1433839b0 0x55d1433923f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.207+0000 7f1f65c00640 1 -- 172.21.15.28:0/1508315203 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14338e080 msgr2=0x55d143392930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:14.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.207+0000 7f1f65c00640 1 --2- 172.21.15.28:0/1508315203 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14338e080 0x55d143392930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.207+0000 7f1f65c00640 1 -- 172.21.15.28:0/1508315203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1433839b0 msgr2=0x55d1433923f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:14.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.207+0000 7f1f65c00640 1 --2- 172.21.15.28:0/1508315203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1433839b0 0x55d1433923f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.207+0000 7f1f65c00640 1 -- 172.21.15.28:0/1508315203 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d143403c00 con 0x55d143383090 2024-04-23T14:51:14.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.207+0000 7f1f65200640 1 --2- 172.21.15.28:0/1508315203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1433839b0 0x55d1433923f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:51:14.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.208+0000 7f1f65c00640 1 --2- 172.21.15.28:0/1508315203 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d143383090 0x55d143391eb0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1f500870b0 tx=0x7f1f5009c290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d77b09c9bf969fe9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:14.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.208+0000 7f1f4b400640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:51:14.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.208+0000 7f1f4b400640 1 -- 172.21.15.28:0/1508315203 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f5009c3f0 con 0x55d143383090 2024-04-23T14:51:14.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.208+0000 7f1f4b400640 1 -- 172.21.15.28:0/1508315203 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8542+0+0 (secure 0 0 0) 0x7f1f500a5000 con 0x55d143383090 2024-04-23T14:51:14.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.208+0000 7f1f6b438740 1 -- 172.21.15.28:0/1508315203 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f58003ad0 con 0x55d143383090 2024-04-23T14:51:14.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.208+0000 7f1f4b400640 1 -- 172.21.15.28:0/1508315203 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f1f500ce260 con 0x55d143383090 2024-04-23T14:51:14.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.208+0000 7f1f6b438740 10 client.26338 fetch_fsmap learned FSMap version 20 2024-04-23T14:51:14.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.208+0000 7f1f6b438740 1 -- 172.21.15.28:0/1508315203 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55d143404f40 con 0x55d143383090 2024-04-23T14:51:14.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.208+0000 7f1f4b400640 1 -- 172.21.15.28:0/1508315203 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f58003ad0 con 0x55d143383090 2024-04-23T14:51:14.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.208+0000 7f1f4b400640 1 -- 172.21.15.28:0/1508315203 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f1f50085bd0 con 0x55d143383090 2024-04-23T14:51:14.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.209+0000 7f1f6b438740 10 client.26338 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:51:14.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.209+0000 7f1f6b438740 10 client.26338 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:51:14.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.209+0000 7f1f6b438740 20 client.26338 populate_metadata read hostname 'smithi028' 2024-04-23T14:51:14.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.209+0000 7f1f6b438740 1 --2- 172.21.15.28:0/1508315203 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d143405600 0x55d1434079f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.209+0000 7f1f6b438740 4 client.26338 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-23T14:51:14.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.209+0000 7f1f6b438740 1 -- 172.21.15.28:0/1508315203 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d1432f0700 con 0x55d143405600 2024-04-23T14:51:14.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.209+0000 7f1f65200640 1 --2- 172.21.15.28:0/1508315203 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d143405600 0x55d1434079f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.209+0000 7f1f65200640 1 --2- 172.21.15.28:0/1508315203 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d143405600 0x55d1434079f0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2f3909a5d69a0e9 server_cookie=44492a412dc6c66e in_seq=0 out_seq=0 2024-04-23T14:51:14.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.209+0000 7f1f4b400640 10 client.26338 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:51:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.211+0000 7f1f4b400640 1 -- 172.21.15.28:0/1508315203 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d1432f0700 con 0x55d143405600 2024-04-23T14:51:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.211+0000 7f1f4b400640 10 client.26338 handle_command_reply: tid=0 2024-04-23T14:51:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.211+0000 7f1f6b438740 1 client.26338 shutdown 2024-04-23T14:51:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.211+0000 7f1f6b438740 1 -- 172.21.15.28:0/1508315203 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d143383090 msgr2=0x55d143391eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.211+0000 7f1f6b438740 1 --2- 172.21.15.28:0/1508315203 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d143383090 0x55d143391eb0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1f500870b0 tx=0x7f1f5009c290 comp rx=0 tx=0).stop 2024-04-23T14:51:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.211+0000 7f1f6b438740 1 -- 172.21.15.28:0/1508315203 shutdown_connections 2024-04-23T14:51:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.211+0000 7f1f6b438740 1 --2- 172.21.15.28:0/1508315203 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d143405600 0x55d1434079f0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.211+0000 7f1f6b438740 1 --2- 172.21.15.28:0/1508315203 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14338e080 0x55d143392930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.211+0000 7f1f6b438740 1 --2- 172.21.15.28:0/1508315203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1433839b0 0x55d1433923f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.211+0000 7f1f6b438740 1 --2- 172.21.15.28:0/1508315203 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d143383090 0x55d143391eb0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.211+0000 7f1f6b438740 1 -- 172.21.15.28:0/1508315203 >> 172.21.15.28:0/1508315203 conn(0x55d14337f3b0 msgr2=0x55d14337dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:14.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.211+0000 7f1f4b400640 0 client.26338 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:51:14.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.211+0000 7f1f6b438740 1 -- 172.21.15.28:0/1508315203 shutdown_connections 2024-04-23T14:51:14.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.211+0000 7f1f6b438740 1 -- 172.21.15.28:0/1508315203 wait complete. 2024-04-23T14:51:14.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.211+0000 7f1f6b438740 20 client.26338 trim_cache size 0 max 16384 2024-04-23T14:51:14.219 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:51:14.219 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-23T14:51:14.220 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "05d4bf37-627d-41e1-a2c3-519ba56a1bb9", 2024-04-23T14:51:14.220 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-23T14:51:14.220 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:51:14.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.212+0000 7f1f68600640 1 -- 172.21.15.28:0/2555316945 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f1f440a3da0 msgr2=0x7f1f440a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:14.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.212+0000 7f1f68600640 1 --2- 172.21.15.28:0/2555316945 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f1f440a3da0 0x7f1f440a6260 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1f58049000 tx=0x7f1f5804c000 comp rx=0 tx=0).stop 2024-04-23T14:51:14.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.212+0000 7f1f68600640 1 -- 172.21.15.28:0/2555316945 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f60102290 msgr2=0x7f1f601a0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:14.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.212+0000 7f1f68600640 1 --2- 172.21.15.28:0/2555316945 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f60102290 0x7f1f601a0790 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f1f5000c420 tx=0x7f1f50009f90 comp rx=0 tx=0).stop 2024-04-23T14:51:14.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.212+0000 7f1f68600640 1 -- 172.21.15.28:0/2555316945 shutdown_connections 2024-04-23T14:51:14.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.212+0000 7f1f68600640 1 --2- 172.21.15.28:0/2555316945 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f1f440a3da0 0x7f1f440a6260 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.212+0000 7f1f68600640 1 --2- 172.21.15.28:0/2555316945 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f6019a940 0x7f1f6019adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.212+0000 7f1f68600640 1 --2- 172.21.15.28:0/2555316945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f60107360 0x7f1f601a0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.212+0000 7f1f68600640 1 --2- 172.21.15.28:0/2555316945 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f60102290 0x7f1f601a0790 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.212+0000 7f1f68600640 1 -- 172.21.15.28:0/2555316945 >> 172.21.15.28:0/2555316945 conn(0x7f1f600fdb10 msgr2=0x7f1f600fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:14.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.213+0000 7f1f68600640 1 -- 172.21.15.28:0/2555316945 shutdown_connections 2024-04-23T14:51:14.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.213+0000 7f1f68600640 1 -- 172.21.15.28:0/2555316945 wait complete. 2024-04-23T14:51:14.232 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:51:14.232 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:51:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.441+0000 7f41af200640 1 -- 172.21.15.28:0/572168632 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41b0102ac0 msgr2=0x7f41b0106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.441+0000 7f41af200640 1 --2- 172.21.15.28:0/572168632 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41b0102ac0 0x7f41b0106b00 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f419c009900 tx=0x7f419c02a3f0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.441+0000 7f41af200640 1 -- 172.21.15.28:0/572168632 shutdown_connections 2024-04-23T14:51:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.441+0000 7f41af200640 1 --2- 172.21.15.28:0/572168632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41b01071e0 0x7f41b010edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.441+0000 7f41af200640 1 --2- 172.21.15.28:0/572168632 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41b0102ac0 0x7f41b0106b00 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.441+0000 7f41af200640 1 --2- 172.21.15.28:0/572168632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41b0102110 0x7f41b01024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.441+0000 7f41af200640 1 -- 172.21.15.28:0/572168632 >> 172.21.15.28:0/572168632 conn(0x7f41b00fd9d0 msgr2=0x7f41b00ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.441+0000 7f41af200640 1 -- 172.21.15.28:0/572168632 shutdown_connections 2024-04-23T14:51:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.441+0000 7f41af200640 1 -- 172.21.15.28:0/572168632 wait complete. 2024-04-23T14:51:14.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.443+0000 7f41af200640 1 Processor -- start 2024-04-23T14:51:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.443+0000 7f41af200640 1 -- start start 2024-04-23T14:51:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.443+0000 7f41af200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41b0102110 0x7f41b0198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.443+0000 7f41af200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41b01071e0 0x7f41b0198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.443+0000 7f41af200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41b019d020 0x7f41b019d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.443+0000 7f41af200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f41b01122a0 con 0x7f41b019d020 2024-04-23T14:51:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.443+0000 7f41af200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f41b0112120 con 0x7f41b01071e0 2024-04-23T14:51:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.443+0000 7f41af200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f41b0112420 con 0x7f41b0102110 2024-04-23T14:51:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.444+0000 7f41ade00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41b0102110 0x7f41b0198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.444+0000 7f41ad400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41b01071e0 0x7f41b0198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.444+0000 7f41ade00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41b0102110 0x7f41b0198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:35162/0 (socket says 172.21.15.28:35162) 2024-04-23T14:51:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.444+0000 7f41ade00640 1 -- 172.21.15.28:0/45184682 learned_addr learned my addr 172.21.15.28:0/45184682 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.444+0000 7f41ae800640 1 --2- 172.21.15.28:0/45184682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41b019d020 0x7f41b019d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.444+0000 7f41ad400640 1 -- 172.21.15.28:0/45184682 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41b0102110 msgr2=0x7f41b0198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:14.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.444+0000 7f41ad400640 1 --2- 172.21.15.28:0/45184682 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41b0102110 0x7f41b0198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.444+0000 7f41ad400640 1 -- 172.21.15.28:0/45184682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41b019d020 msgr2=0x7f41b019d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:14.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.444+0000 7f41ad400640 1 --2- 172.21.15.28:0/45184682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41b019d020 0x7f41b019d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.444+0000 7f41ad400640 1 -- 172.21.15.28:0/45184682 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41b019db10 con 0x7f41b01071e0 2024-04-23T14:51:14.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.444+0000 7f41ade00640 1 --2- 172.21.15.28:0/45184682 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41b0102110 0x7f41b0198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:51:14.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.445+0000 7f41ad400640 1 --2- 172.21.15.28:0/45184682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41b01071e0 0x7f41b0198ca0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f419c02a910 tx=0x7f419c003b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:14.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.445+0000 7f41a6a00640 1 -- 172.21.15.28:0/45184682 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f419c03f070 con 0x7f41b01071e0 2024-04-23T14:51:14.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.445+0000 7f41a6a00640 1 -- 172.21.15.28:0/45184682 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f419c031c90 con 0x7f41b01071e0 2024-04-23T14:51:14.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.445+0000 7f41a6a00640 1 -- 172.21.15.28:0/45184682 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f419c035030 con 0x7f41b01071e0 2024-04-23T14:51:14.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.445+0000 7f41af200640 1 -- 172.21.15.28:0/45184682 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41b019dda0 con 0x7f41b01071e0 2024-04-23T14:51:14.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.445+0000 7f41af200640 1 -- 172.21.15.28:0/45184682 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41b01a55c0 con 0x7f41b01071e0 2024-04-23T14:51:14.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.449+0000 7f41b7fe3740 1 Processor -- start 2024-04-23T14:51:14.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.449+0000 7f41b7fe3740 1 -- start start 2024-04-23T14:51:14.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.449+0000 7f41a6a00640 1 -- 172.21.15.28:0/45184682 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f419c042070 con 0x7f41b01071e0 2024-04-23T14:51:14.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.449+0000 7f41b7fe3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f1699f760 0x560f1699fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.449+0000 7f41b7fe3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f169a0110 0x560f169ae8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.450+0000 7f41b7fe3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f169aee30 0x560f169b1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.450+0000 7f41b7fe3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560f169051f0 con 0x560f169aee30 2024-04-23T14:51:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.450+0000 7f41b7fe3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560f168ab350 con 0x560f1699f760 2024-04-23T14:51:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.450+0000 7f41b7fe3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560f169011b0 con 0x560f169a0110 2024-04-23T14:51:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.450+0000 7f41ae800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f1699f760 0x560f1699fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.450+0000 7f41ade00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f169a0110 0x560f169ae8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.450+0000 7f41ae800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f1699f760 0x560f1699fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:57942/0 (socket says 172.21.15.28:57942) 2024-04-23T14:51:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.450+0000 7f41ae800640 1 -- 172.21.15.28:0/1232914560 learned_addr learned my addr 172.21.15.28:0/1232914560 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.450+0000 7f41ade00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f169a0110 0x560f169ae8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:35168/0 (socket says 172.21.15.28:35168) 2024-04-23T14:51:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.450+0000 7f41ad400640 1 --2- 172.21.15.28:0/1232914560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f169aee30 0x560f169b1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.450+0000 7f41ade00640 1 -- 172.21.15.28:0/1232914560 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f1699f760 msgr2=0x560f1699fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.450+0000 7f41ade00640 1 --2- 172.21.15.28:0/1232914560 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f1699f760 0x560f1699fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.450+0000 7f41ade00640 1 -- 172.21.15.28:0/1232914560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f169aee30 msgr2=0x560f169b1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.450+0000 7f41ade00640 1 --2- 172.21.15.28:0/1232914560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f169aee30 0x560f169b1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.450+0000 7f41ade00640 1 -- 172.21.15.28:0/1232914560 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f168fff50 con 0x560f169a0110 2024-04-23T14:51:14.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.450+0000 7f41ad400640 1 --2- 172.21.15.28:0/1232914560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f169aee30 0x560f169b1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:51:14.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.451+0000 7f41ade00640 1 --2- 172.21.15.28:0/1232914560 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f169a0110 0x560f169ae8f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f41a0011520 tx=0x7f41a00119e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ad8390e67b346e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:14.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.451+0000 7f419b400640 1 -- 172.21.15.28:0/1232914560 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a001e070 con 0x560f169a0110 2024-04-23T14:51:14.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.451+0000 7f419b400640 1 -- 172.21.15.28:0/1232914560 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41a00035f0 con 0x560f169a0110 2024-04-23T14:51:14.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.451+0000 7f419b400640 1 -- 172.21.15.28:0/1232914560 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a000f8d0 con 0x560f169a0110 2024-04-23T14:51:14.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.451+0000 7f41a6a00640 1 --2- 172.21.15.28:0/45184682 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f41940a3da0 0x7f41940a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.451+0000 7f41a6a00640 1 -- 172.21.15.28:0/45184682 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8542+0+0 (secure 0 0 0) 0x7f419c0f1050 con 0x7f41b01071e0 2024-04-23T14:51:14.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.452+0000 7f41ae800640 1 --2- 172.21.15.28:0/45184682 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f41940a3da0 0x7f41940a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.452+0000 7f41b7fe3740 1 -- 172.21.15.28:0/1232914560 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f169a0110 msgr2=0x560f169ae8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:14.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.452+0000 7f41b7fe3740 1 --2- 172.21.15.28:0/1232914560 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f169a0110 0x560f169ae8f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f41a0011520 tx=0x7f41a00119e0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.452+0000 7f41b7fe3740 1 -- 172.21.15.28:0/1232914560 shutdown_connections 2024-04-23T14:51:14.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.452+0000 7f41b7fe3740 1 --2- 172.21.15.28:0/1232914560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f169aee30 0x560f169b1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.452+0000 7f41b7fe3740 1 --2- 172.21.15.28:0/1232914560 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f169a0110 0x560f169ae8f0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.452+0000 7f41b7fe3740 1 --2- 172.21.15.28:0/1232914560 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f1699f760 0x560f1699fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.452+0000 7f41b7fe3740 1 -- 172.21.15.28:0/1232914560 >> 172.21.15.28:0/1232914560 conn(0x560f1699bd50 msgr2=0x560f1699c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:14.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.452+0000 7f41b7fe3740 1 -- 172.21.15.28:0/1232914560 shutdown_connections 2024-04-23T14:51:14.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.452+0000 7f41ae800640 1 --2- 172.21.15.28:0/45184682 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f41940a3da0 0x7f41940a6260 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f41a8011630 tx=0x7f41a8009500 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:14.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.452+0000 7f41b7fe3740 1 -- 172.21.15.28:0/1232914560 wait complete. 2024-04-23T14:51:14.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.453+0000 7f41b7fe3740 1 -- start start 2024-04-23T14:51:14.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.453+0000 7f41b7fe3740 1 Processor -- start 2024-04-23T14:51:14.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.454+0000 7f41b7fe3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f1699f760 0x560f169abe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.454+0000 7f41b7fe3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f169a0110 0x560f169ac370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.454+0000 7f41b7fe3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f169aee30 0x560f169ac8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.454+0000 7f41b7fe3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560f169b1bf0 con 0x560f1699f760 2024-04-23T14:51:14.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.454+0000 7f41b7fe3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560f168be690 con 0x560f169aee30 2024-04-23T14:51:14.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.454+0000 7f41b7fe3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560f169b1d70 con 0x560f169a0110 2024-04-23T14:51:14.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.454+0000 7f41b7fe3740 10 client.0 _ll_register_callbacks cb 0x560f168fa650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:51:14.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.454+0000 7f41ae800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f169aee30 0x560f169ac8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.454+0000 7f41ade00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f1699f760 0x560f169abe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.454+0000 7f41ad400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f169a0110 0x560f169ac370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.454+0000 7f41ae800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f169aee30 0x560f169ac8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:57954/0 (socket says 172.21.15.28:57954) 2024-04-23T14:51:14.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.454+0000 7f41ae800640 1 -- 172.21.15.28:0/2213853101 learned_addr learned my addr 172.21.15.28:0/2213853101 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:14.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.454+0000 7f41ad400640 1 -- 172.21.15.28:0/2213853101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f169aee30 msgr2=0x560f169ac8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:14.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.454+0000 7f41ad400640 1 --2- 172.21.15.28:0/2213853101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f169aee30 0x560f169ac8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.454+0000 7f41ad400640 1 -- 172.21.15.28:0/2213853101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f1699f760 msgr2=0x560f169abe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:14.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.454+0000 7f41ad400640 1 --2- 172.21.15.28:0/2213853101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f1699f760 0x560f169abe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.454+0000 7f41ad400640 1 -- 172.21.15.28:0/2213853101 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f168bd920 con 0x560f169a0110 2024-04-23T14:51:14.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.454+0000 7f41ae800640 1 --2- 172.21.15.28:0/2213853101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f169aee30 0x560f169ac8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:51:14.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.454+0000 7f41ade00640 1 --2- 172.21.15.28:0/2213853101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f1699f760 0x560f169abe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:51:14.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.455+0000 7f41ad400640 1 --2- 172.21.15.28:0/2213853101 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f169a0110 0x560f169ac370 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f419c0ee790 tx=0x7f419c005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24695420ec7f9b8a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:14.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.455+0000 7f418fe00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:51:14.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.455+0000 7f418fe00640 1 -- 172.21.15.28:0/2213853101 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f419c049070 con 0x560f169a0110 2024-04-23T14:51:14.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.455+0000 7f418fe00640 1 -- 172.21.15.28:0/2213853101 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8542+0+0 (secure 0 0 0) 0x7f419c0b9700 con 0x560f169a0110 2024-04-23T14:51:14.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.455+0000 7f41b7fe3740 1 -- 172.21.15.28:0/2213853101 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41a00035f0 con 0x560f169a0110 2024-04-23T14:51:14.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.455+0000 7f418fe00640 1 -- 172.21.15.28:0/2213853101 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f419c0bc9f0 con 0x560f169a0110 2024-04-23T14:51:14.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.455+0000 7f41b7fe3740 10 client.26350 fetch_fsmap learned FSMap version 20 2024-04-23T14:51:14.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.455+0000 7f41b7fe3740 1 -- 172.21.15.28:0/2213853101 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x560f166e17d0 con 0x560f169a0110 2024-04-23T14:51:14.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.455+0000 7f418fe00640 1 -- 172.21.15.28:0/2213853101 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41a00035f0 con 0x560f169a0110 2024-04-23T14:51:14.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.456+0000 7f418fe00640 1 -- 172.21.15.28:0/2213853101 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f419c0f3110 con 0x560f169a0110 2024-04-23T14:51:14.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.456+0000 7f41b7fe3740 10 client.26350 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:51:14.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.456+0000 7f41b7fe3740 10 client.26350 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:51:14.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.456+0000 7f41b7fe3740 20 client.26350 populate_metadata read hostname 'smithi028' 2024-04-23T14:51:14.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.456+0000 7f41b7fe3740 1 --2- 172.21.15.28:0/2213853101 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560f169bf100 0x560f169c15c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.456+0000 7f41b7fe3740 4 client.26350 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:51:14.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.456+0000 7f41b7fe3740 1 -- 172.21.15.28:0/2213853101 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f16912c10 con 0x560f169bf100 2024-04-23T14:51:14.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.456+0000 7f41ade00640 1 --2- 172.21.15.28:0/2213853101 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560f169bf100 0x560f169c15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.457+0000 7f41ade00640 1 --2- 172.21.15.28:0/2213853101 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560f169bf100 0x560f169c15c0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9be163136512ea6 server_cookie=d3e7b34751bad305 in_seq=0 out_seq=0 2024-04-23T14:51:14.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.457+0000 7f418fe00640 10 client.26350 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:51:14.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.457+0000 7f418fe00640 1 -- 172.21.15.28:0/2213853101 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560f16912c10 con 0x560f169bf100 2024-04-23T14:51:14.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.457+0000 7f418fe00640 10 client.26350 handle_command_reply: tid=0 2024-04-23T14:51:14.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.457+0000 7f41b7fe3740 1 client.26350 shutdown 2024-04-23T14:51:14.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.458+0000 7f41b7fe3740 1 -- 172.21.15.28:0/2213853101 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f169a0110 msgr2=0x560f169ac370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:14.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.458+0000 7f41b7fe3740 1 --2- 172.21.15.28:0/2213853101 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f169a0110 0x560f169ac370 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f419c0ee790 tx=0x7f419c005d50 comp rx=0 tx=0).stop 2024-04-23T14:51:14.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.458+0000 7f41b7fe3740 1 -- 172.21.15.28:0/2213853101 shutdown_connections 2024-04-23T14:51:14.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.458+0000 7f41b7fe3740 1 --2- 172.21.15.28:0/2213853101 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560f169bf100 0x560f169c15c0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.458+0000 7f41b7fe3740 1 --2- 172.21.15.28:0/2213853101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f169aee30 0x560f169ac8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.458+0000 7f41b7fe3740 1 --2- 172.21.15.28:0/2213853101 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f169a0110 0x560f169ac370 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.458+0000 7f41b7fe3740 1 --2- 172.21.15.28:0/2213853101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f1699f760 0x560f169abe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.458+0000 7f418fe00640 0 client.26350 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:51:14.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.458+0000 7f41b7fe3740 1 -- 172.21.15.28:0/2213853101 >> 172.21.15.28:0/2213853101 conn(0x560f1699bd50 msgr2=0x560f169a1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:14.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.458+0000 7f41b7fe3740 1 -- 172.21.15.28:0/2213853101 shutdown_connections 2024-04-23T14:51:14.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.458+0000 7f41b7fe3740 1 -- 172.21.15.28:0/2213853101 wait complete. 2024-04-23T14:51:14.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.458+0000 7f41b7fe3740 20 client.26350 trim_cache size 0 max 16384 2024-04-23T14:51:14.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.483+0000 7f41b7fe3740 1 Processor -- start 2024-04-23T14:51:14.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.483+0000 7f41b7fe3740 1 -- start start 2024-04-23T14:51:14.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.484+0000 7f41b7fe3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f1699f760 0x560f169baaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.484+0000 7f41b7fe3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f169a0110 0x560f169aaf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.484+0000 7f41b7fe3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f169aee30 0x560f1699ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.484+0000 7f41b7fe3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f41b01a9910 con 0x560f169a0110 2024-04-23T14:51:14.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.484+0000 7f41b7fe3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f41b01ac170 con 0x560f1699f760 2024-04-23T14:51:14.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.484+0000 7f41b7fe3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f41b01a9f60 con 0x560f169aee30 2024-04-23T14:51:14.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.484+0000 7f41ade00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f1699f760 0x560f169baaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.484+0000 7f41ad400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f169a0110 0x560f169aaf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.484+0000 7f41ade00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f1699f760 0x560f169baaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:57962/0 (socket says 172.21.15.28:57962) 2024-04-23T14:51:14.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.484+0000 7f41ade00640 1 -- 172.21.15.28:0/729067702 learned_addr learned my addr 172.21.15.28:0/729067702 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:14.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.484+0000 7f41ae800640 1 --2- 172.21.15.28:0/729067702 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f169aee30 0x560f1699ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.484+0000 7f41ade00640 1 -- 172.21.15.28:0/729067702 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f169aee30 msgr2=0x560f1699ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:14.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.484+0000 7f41ade00640 1 --2- 172.21.15.28:0/729067702 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f169aee30 0x560f1699ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.484+0000 7f41ade00640 1 -- 172.21.15.28:0/729067702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f169a0110 msgr2=0x560f169aaf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:14.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.484+0000 7f41ade00640 1 --2- 172.21.15.28:0/729067702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f169a0110 0x560f169aaf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.484+0000 7f41ade00640 1 -- 172.21.15.28:0/729067702 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f1699feb0 con 0x560f1699f760 2024-04-23T14:51:14.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.485+0000 7f41ade00640 1 --2- 172.21.15.28:0/729067702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f1699f760 0x560f169baaf0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f41a001c040 tx=0x7f41a0005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a66001803887a6d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:14.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.485+0000 7f419aa00640 1 -- 172.21.15.28:0/729067702 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a001e040 con 0x560f1699f760 2024-04-23T14:51:14.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.485+0000 7f419aa00640 1 -- 172.21.15.28:0/729067702 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41a0005250 con 0x560f1699f760 2024-04-23T14:51:14.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.485+0000 7f419aa00640 1 -- 172.21.15.28:0/729067702 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a0026750 con 0x560f1699f760 2024-04-23T14:51:14.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.486+0000 7f41b7fe3740 1 -- 172.21.15.28:0/729067702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f1699f760 msgr2=0x560f169baaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:14.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.486+0000 7f41b7fe3740 1 --2- 172.21.15.28:0/729067702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f1699f760 0x560f169baaf0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f41a001c040 tx=0x7f41a0005f90 comp rx=0 tx=0).stop 2024-04-23T14:51:14.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.486+0000 7f41b7fe3740 1 -- 172.21.15.28:0/729067702 shutdown_connections 2024-04-23T14:51:14.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.486+0000 7f41b7fe3740 1 --2- 172.21.15.28:0/729067702 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f169aee30 0x560f1699ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.486+0000 7f41b7fe3740 1 --2- 172.21.15.28:0/729067702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f169a0110 0x560f169aaf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.486+0000 7f41b7fe3740 1 --2- 172.21.15.28:0/729067702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f1699f760 0x560f169baaf0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.486+0000 7f41b7fe3740 1 -- 172.21.15.28:0/729067702 >> 172.21.15.28:0/729067702 conn(0x560f1699bd50 msgr2=0x560f1699a740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:14.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.486+0000 7f41b7fe3740 1 -- 172.21.15.28:0/729067702 shutdown_connections 2024-04-23T14:51:14.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.486+0000 7f41b7fe3740 1 -- 172.21.15.28:0/729067702 wait complete. 2024-04-23T14:51:14.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.487+0000 7f41b7fe3740 1 -- start start 2024-04-23T14:51:14.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.487+0000 7f41b7fe3740 1 Processor -- start 2024-04-23T14:51:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.487+0000 7f41b7fe3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f1699f760 0x560f169b21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.487+0000 7f41b7fe3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f169a0110 0x560f169b2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.487+0000 7f41b7fe3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f169aee30 0x560f16a236f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.487+0000 7f41b7fe3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f41b01aa630 con 0x560f169a0110 2024-04-23T14:51:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.487+0000 7f41b7fe3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f41b01abaa0 con 0x560f169aee30 2024-04-23T14:51:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.487+0000 7f41b7fe3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f41b01aad00 con 0x560f1699f760 2024-04-23T14:51:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.487+0000 7f41b7fe3740 10 client.0 _ll_register_callbacks cb 0x7f419c005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:51:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.488+0000 7f41ade00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f1699f760 0x560f169b21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.488+0000 7f41ae800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f169aee30 0x560f16a236f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.488+0000 7f41ade00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f1699f760 0x560f169b21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:35192/0 (socket says 172.21.15.28:35192) 2024-04-23T14:51:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.488+0000 7f41ade00640 1 -- 172.21.15.28:0/774150307 learned_addr learned my addr 172.21.15.28:0/774150307 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.488+0000 7f41ad400640 1 --2- 172.21.15.28:0/774150307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f169a0110 0x560f169b2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.488+0000 7f41ade00640 1 -- 172.21.15.28:0/774150307 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f169aee30 msgr2=0x560f16a236f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.488+0000 7f41ade00640 1 --2- 172.21.15.28:0/774150307 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f169aee30 0x560f16a236f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.488+0000 7f41ade00640 1 -- 172.21.15.28:0/774150307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f169a0110 msgr2=0x560f169b2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.488+0000 7f41ade00640 1 --2- 172.21.15.28:0/774150307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f169a0110 0x560f169b2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.488+0000 7f41ade00640 1 -- 172.21.15.28:0/774150307 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f166173f0 con 0x560f1699f760 2024-04-23T14:51:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.488+0000 7f41ade00640 1 --2- 172.21.15.28:0/774150307 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f1699f760 0x560f169b21d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f41a000a010 tx=0x7f41a00093a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17555abc6c1b0993 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.488+0000 7f418fe00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:51:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.489+0000 7f418fe00640 1 -- 172.21.15.28:0/774150307 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a0034060 con 0x560f1699f760 2024-04-23T14:51:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.489+0000 7f418fe00640 1 -- 172.21.15.28:0/774150307 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8542+0+0 (secure 0 0 0) 0x7f41a0005400 con 0x560f1699f760 2024-04-23T14:51:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.489+0000 7f41b7fe3740 1 -- 172.21.15.28:0/774150307 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41a0005250 con 0x560f1699f760 2024-04-23T14:51:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.489+0000 7f418fe00640 1 -- 172.21.15.28:0/774150307 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f41a001e030 con 0x560f1699f760 2024-04-23T14:51:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.489+0000 7f41b7fe3740 10 client.26356 fetch_fsmap learned FSMap version 20 2024-04-23T14:51:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.489+0000 7f41b7fe3740 1 -- 172.21.15.28:0/774150307 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x560f165d70f0 con 0x560f1699f760 2024-04-23T14:51:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.489+0000 7f418fe00640 1 -- 172.21.15.28:0/774150307 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41a0005250 con 0x560f1699f760 2024-04-23T14:51:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.489+0000 7f418fe00640 1 -- 172.21.15.28:0/774150307 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f41a00035f0 con 0x560f1699f760 2024-04-23T14:51:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.489+0000 7f41b7fe3740 10 client.26356 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:51:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.489+0000 7f41b7fe3740 10 client.26356 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:51:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.489+0000 7f41b7fe3740 20 client.26356 populate_metadata read hostname 'smithi028' 2024-04-23T14:51:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.489+0000 7f41b7fe3740 1 --2- 172.21.15.28:0/774150307 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560f16a24fd0 0x560f16a273c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.490+0000 7f41b7fe3740 4 client.26356 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:51:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.490+0000 7f41b7fe3740 1 -- 172.21.15.28:0/774150307 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560f1690ca80 con 0x560f16a24fd0 2024-04-23T14:51:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.490+0000 7f41ad400640 1 --2- 172.21.15.28:0/774150307 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560f16a24fd0 0x560f16a273c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.490+0000 7f41ad400640 1 --2- 172.21.15.28:0/774150307 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560f16a24fd0 0x560f16a273c0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d38b9d2c17cc5c33 server_cookie=7abbbbb98b58038 in_seq=0 out_seq=0 2024-04-23T14:51:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.490+0000 7f418fe00640 10 client.26356 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:51:14.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.492+0000 7f418fe00640 1 -- 172.21.15.28:0/774150307 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x560f1690ca80 con 0x560f16a24fd0 2024-04-23T14:51:14.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.492+0000 7f418fe00640 10 client.26356 handle_command_reply: tid=0 2024-04-23T14:51:14.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.492+0000 7f41b7fe3740 1 client.26356 shutdown 2024-04-23T14:51:14.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.492+0000 7f41b7fe3740 1 -- 172.21.15.28:0/774150307 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f1699f760 msgr2=0x560f169b21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:14.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.492+0000 7f41b7fe3740 1 --2- 172.21.15.28:0/774150307 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f1699f760 0x560f169b21d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f41a000a010 tx=0x7f41a00093a0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.492+0000 7f41b7fe3740 1 -- 172.21.15.28:0/774150307 shutdown_connections 2024-04-23T14:51:14.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.492+0000 7f41b7fe3740 1 --2- 172.21.15.28:0/774150307 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560f16a24fd0 0x560f16a273c0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.492+0000 7f41b7fe3740 1 --2- 172.21.15.28:0/774150307 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f169aee30 0x560f16a236f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.492+0000 7f41b7fe3740 1 --2- 172.21.15.28:0/774150307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f169a0110 0x560f169b2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.492+0000 7f418fe00640 0 client.26356 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:51:14.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.492+0000 7f41b7fe3740 1 --2- 172.21.15.28:0/774150307 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560f1699f760 0x560f169b21d0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.492+0000 7f41b7fe3740 1 -- 172.21.15.28:0/774150307 >> 172.21.15.28:0/774150307 conn(0x560f1699bd50 msgr2=0x560f169ac680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:14.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.493+0000 7f41b7fe3740 1 -- 172.21.15.28:0/774150307 shutdown_connections 2024-04-23T14:51:14.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.493+0000 7f41b7fe3740 1 -- 172.21.15.28:0/774150307 wait complete. 2024-04-23T14:51:14.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.493+0000 7f41b7fe3740 20 client.26356 trim_cache size 0 max 16384 2024-04-23T14:51:14.497 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:51:14.497 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-23T14:51:14.497 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-23T14:51:14.497 INFO:teuthology.orchestra.run.smithi028.stdout: "05d4bf37-627d-41e1-a2c3-519ba56a1bb9": { 2024-04-23T14:51:14.498 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-23T14:51:14.498 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "05d4bf37-627d-41e1-a2c3-519ba56a1bb9", 2024-04-23T14:51:14.498 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-23T14:51:14.498 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:51:14.498 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:51:14.498 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:51:14.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.493+0000 7f41af200640 1 -- 172.21.15.28:0/45184682 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f41940a3da0 msgr2=0x7f41940a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:14.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.494+0000 7f41af200640 1 --2- 172.21.15.28:0/45184682 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f41940a3da0 0x7f41940a6260 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f41a8011630 tx=0x7f41a8009500 comp rx=0 tx=0).stop 2024-04-23T14:51:14.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.494+0000 7f41af200640 1 -- 172.21.15.28:0/45184682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41b01071e0 msgr2=0x7f41b0198ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:14.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.494+0000 7f41af200640 1 --2- 172.21.15.28:0/45184682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41b01071e0 0x7f41b0198ca0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f419c02a910 tx=0x7f419c003b20 comp rx=0 tx=0).stop 2024-04-23T14:51:14.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.494+0000 7f41af200640 1 -- 172.21.15.28:0/45184682 shutdown_connections 2024-04-23T14:51:14.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.494+0000 7f41af200640 1 --2- 172.21.15.28:0/45184682 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f41940a3da0 0x7f41940a6260 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.494+0000 7f41af200640 1 --2- 172.21.15.28:0/45184682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41b01071e0 0x7f41b0198ca0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.494+0000 7f41af200640 1 --2- 172.21.15.28:0/45184682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41b019d020 0x7f41b019d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.494+0000 7f41af200640 1 --2- 172.21.15.28:0/45184682 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41b0102110 0x7f41b0198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:14.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.494+0000 7f41af200640 1 -- 172.21.15.28:0/45184682 >> 172.21.15.28:0/45184682 conn(0x7f41b00fd9d0 msgr2=0x7f41b00ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:14.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.494+0000 7f41af200640 1 -- 172.21.15.28:0/45184682 shutdown_connections 2024-04-23T14:51:14.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:14.494+0000 7f41af200640 1 -- 172.21.15.28:0/45184682 wait complete. 2024-04-23T14:51:14.503 INFO:tasks.workunit.client.0.smithi028.stdout: 1 590411 9.21 MB/sec execute 210 sec latency 24.045 ms 2024-04-23T14:51:14.514 INFO:tasks.cephfs.filesystem:scrub status for tag:05d4bf37-627d-41e1-a2c3-519ba56a1bb9 - {'path': '/', 'tag': '05d4bf37-627d-41e1-a2c3-519ba56a1bb9', 'options': 'recursive,force'} 2024-04-23T14:51:14.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:14 smithi038 ceph-mon[32348]: pgmap v221: 97 pgs: 97 active+clean; 384 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.7 MiB/s wr, 214 op/s 2024-04-23T14:51:14.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:14 smithi028 ceph-mon[27810]: pgmap v221: 97 pgs: 97 active+clean; 384 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.7 MiB/s wr, 214 op/s 2024-04-23T14:51:15.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:14 smithi105 ceph-mon[32555]: pgmap v221: 97 pgs: 97 active+clean; 384 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.7 MiB/s wr, 214 op/s 2024-04-23T14:51:15.503 INFO:tasks.workunit.client.0.smithi028.stdout: 1 592064 9.20 MB/sec execute 211 sec latency 29.464 ms 2024-04-23T14:51:16.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:15 smithi105 ceph-mon[32555]: scrub queued for path: / 2024-04-23T14:51:16.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:15 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:51:16.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:15 smithi105 ceph-mon[32555]: scrub summary: active paths [/] 2024-04-23T14:51:16.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:15 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:51:16.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:15 smithi105 ceph-mon[32555]: scrub summary: idle 2024-04-23T14:51:16.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:15 smithi038 ceph-mon[32348]: scrub queued for path: / 2024-04-23T14:51:16.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:15 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:51:16.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:15 smithi038 ceph-mon[32348]: scrub summary: active paths [/] 2024-04-23T14:51:16.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:15 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:51:16.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:15 smithi038 ceph-mon[32348]: scrub summary: idle 2024-04-23T14:51:16.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:15 smithi028 ceph-mon[27810]: scrub queued for path: / 2024-04-23T14:51:16.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:15 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:51:16.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:15 smithi028 ceph-mon[27810]: scrub summary: active paths [/] 2024-04-23T14:51:16.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:15 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:51:16.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:15 smithi028 ceph-mon[27810]: scrub summary: idle 2024-04-23T14:51:16.503 INFO:tasks.workunit.client.0.smithi028.stdout: 1 593634 9.22 MB/sec execute 212 sec latency 11.164 ms 2024-04-23T14:51:17.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:16 smithi105 ceph-mon[32555]: pgmap v222: 97 pgs: 97 active+clean; 384 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 174 op/s 2024-04-23T14:51:17.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:16 smithi038 ceph-mon[32348]: pgmap v222: 97 pgs: 97 active+clean; 384 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 174 op/s 2024-04-23T14:51:17.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:16 smithi028 ceph-mon[27810]: pgmap v222: 97 pgs: 97 active+clean; 384 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 174 op/s 2024-04-23T14:51:17.503 INFO:tasks.workunit.client.0.smithi028.stdout: 1 595143 9.20 MB/sec execute 213 sec latency 17.724 ms 2024-04-23T14:51:18.503 INFO:tasks.workunit.client.0.smithi028.stdout: 1 596913 9.22 MB/sec execute 214 sec latency 82.857 ms 2024-04-23T14:51:19.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:18 smithi105 ceph-mon[32555]: pgmap v223: 97 pgs: 97 active+clean; 390 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 174 op/s 2024-04-23T14:51:19.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:18 smithi038 ceph-mon[32348]: pgmap v223: 97 pgs: 97 active+clean; 390 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 174 op/s 2024-04-23T14:51:19.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:18 smithi028 ceph-mon[27810]: pgmap v223: 97 pgs: 97 active+clean; 390 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 174 op/s 2024-04-23T14:51:19.503 INFO:tasks.workunit.client.0.smithi028.stdout: 1 598121 9.20 MB/sec execute 215 sec latency 25.107 ms 2024-04-23T14:51:20.504 INFO:tasks.workunit.client.0.smithi028.stdout: 1 599886 9.19 MB/sec execute 216 sec latency 30.767 ms 2024-04-23T14:51:21.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:20 smithi105 ceph-mon[32555]: pgmap v224: 97 pgs: 97 active+clean; 390 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 174 op/s 2024-04-23T14:51:21.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:20 smithi038 ceph-mon[32348]: pgmap v224: 97 pgs: 97 active+clean; 390 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 174 op/s 2024-04-23T14:51:21.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:20 smithi028 ceph-mon[27810]: pgmap v224: 97 pgs: 97 active+clean; 390 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 174 op/s 2024-04-23T14:51:21.504 INFO:tasks.workunit.client.0.smithi028.stdout: 1 601081 9.18 MB/sec execute 217 sec latency 26.262 ms 2024-04-23T14:51:22.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:21 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:51:22.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:21 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:51:22.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:21 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:51:22.504 INFO:tasks.workunit.client.0.smithi028.stdout: 1 602733 9.17 MB/sec execute 218 sec latency 19.565 ms 2024-04-23T14:51:23.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:22 smithi105 ceph-mon[32555]: pgmap v225: 97 pgs: 97 active+clean; 390 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 174 op/s 2024-04-23T14:51:23.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:22 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:51:23.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:22 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:51:23.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:22 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:51:23.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:22 smithi038 ceph-mon[32348]: pgmap v225: 97 pgs: 97 active+clean; 390 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 174 op/s 2024-04-23T14:51:23.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:22 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:51:23.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:22 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:51:23.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:22 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:51:23.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:22 smithi028 ceph-mon[27810]: pgmap v225: 97 pgs: 97 active+clean; 390 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 174 op/s 2024-04-23T14:51:23.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:22 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:51:23.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:22 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:51:23.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:22 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:51:23.504 INFO:tasks.workunit.client.0.smithi028.stdout: 1 604181 9.18 MB/sec execute 219 sec latency 44.449 ms 2024-04-23T14:51:24.504 INFO:tasks.workunit.client.0.smithi028.stdout: 1 605462 9.17 MB/sec execute 220 sec latency 43.448 ms 2024-04-23T14:51:25.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:24 smithi105 ceph-mon[32555]: pgmap v226: 97 pgs: 97 active+clean; 397 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 213 op/s 2024-04-23T14:51:25.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:24 smithi038 ceph-mon[32348]: pgmap v226: 97 pgs: 97 active+clean; 397 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 213 op/s 2024-04-23T14:51:25.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:24 smithi028 ceph-mon[27810]: pgmap v226: 97 pgs: 97 active+clean; 397 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 213 op/s 2024-04-23T14:51:25.504 INFO:tasks.workunit.client.0.smithi028.stdout: 1 607633 9.18 MB/sec execute 221 sec latency 14.365 ms 2024-04-23T14:51:26.504 INFO:tasks.workunit.client.0.smithi028.stdout: 1 609039 9.17 MB/sec execute 222 sec latency 17.784 ms 2024-04-23T14:51:27.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:26 smithi105 ceph-mon[32555]: pgmap v227: 97 pgs: 97 active+clean; 397 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 140 op/s 2024-04-23T14:51:27.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:26 smithi038 ceph-mon[32348]: pgmap v227: 97 pgs: 97 active+clean; 397 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 140 op/s 2024-04-23T14:51:27.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:26 smithi028 ceph-mon[27810]: pgmap v227: 97 pgs: 97 active+clean; 397 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 140 op/s 2024-04-23T14:51:27.504 INFO:tasks.workunit.client.0.smithi028.stdout: 1 611154 9.19 MB/sec execute 223 sec latency 13.776 ms 2024-04-23T14:51:28.504 INFO:tasks.workunit.client.0.smithi028.stdout: 1 612279 9.17 MB/sec execute 224 sec latency 69.615 ms 2024-04-23T14:51:29.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:28 smithi105 ceph-mon[32555]: pgmap v228: 97 pgs: 97 active+clean; 403 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 210 op/s 2024-04-23T14:51:29.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:28 smithi038 ceph-mon[32348]: pgmap v228: 97 pgs: 97 active+clean; 403 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 210 op/s 2024-04-23T14:51:29.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:28 smithi028 ceph-mon[27810]: pgmap v228: 97 pgs: 97 active+clean; 403 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 210 op/s 2024-04-23T14:51:29.505 INFO:tasks.workunit.client.0.smithi028.stdout: 1 613831 9.16 MB/sec execute 225 sec latency 67.670 ms 2024-04-23T14:51:30.505 INFO:tasks.workunit.client.0.smithi028.stdout: 1 615130 9.15 MB/sec execute 226 sec latency 26.657 ms 2024-04-23T14:51:31.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:30 smithi105 ceph-mon[32555]: pgmap v229: 97 pgs: 97 active+clean; 403 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 136 op/s 2024-04-23T14:51:31.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:30 smithi038 ceph-mon[32348]: pgmap v229: 97 pgs: 97 active+clean; 403 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 136 op/s 2024-04-23T14:51:31.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:30 smithi028 ceph-mon[27810]: pgmap v229: 97 pgs: 97 active+clean; 403 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 136 op/s 2024-04-23T14:51:31.505 INFO:tasks.workunit.client.0.smithi028.stdout: 1 617008 9.15 MB/sec execute 227 sec latency 16.039 ms 2024-04-23T14:51:32.505 INFO:tasks.workunit.client.0.smithi028.stdout: 1 618705 9.16 MB/sec execute 228 sec latency 16.984 ms 2024-04-23T14:51:33.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:32 smithi105 ceph-mon[32555]: pgmap v230: 97 pgs: 97 active+clean; 403 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 136 op/s 2024-04-23T14:51:33.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:32 smithi038 ceph-mon[32348]: pgmap v230: 97 pgs: 97 active+clean; 403 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 136 op/s 2024-04-23T14:51:33.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:32 smithi028 ceph-mon[27810]: pgmap v230: 97 pgs: 97 active+clean; 403 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 136 op/s 2024-04-23T14:51:33.505 INFO:tasks.workunit.client.0.smithi028.stdout: 1 620260 9.14 MB/sec execute 229 sec latency 75.093 ms 2024-04-23T14:51:34.505 INFO:tasks.workunit.client.0.smithi028.stdout: 1 622084 9.16 MB/sec execute 230 sec latency 18.818 ms 2024-04-23T14:51:35.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:34 smithi105 ceph-mon[32555]: pgmap v231: 97 pgs: 97 active+clean; 410 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 206 op/s 2024-04-23T14:51:35.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:34 smithi038 ceph-mon[32348]: pgmap v231: 97 pgs: 97 active+clean; 410 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 206 op/s 2024-04-23T14:51:35.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:34 smithi028 ceph-mon[27810]: pgmap v231: 97 pgs: 97 active+clean; 410 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 206 op/s 2024-04-23T14:51:35.505 INFO:tasks.workunit.client.0.smithi028.stdout: 1 623930 9.15 MB/sec execute 231 sec latency 18.440 ms 2024-04-23T14:51:36.506 INFO:tasks.workunit.client.0.smithi028.stdout: 1 625743 9.16 MB/sec execute 232 sec latency 17.092 ms 2024-04-23T14:51:37.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:36 smithi105 ceph-mon[32555]: pgmap v232: 97 pgs: 97 active+clean; 410 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 139 op/s 2024-04-23T14:51:37.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:36 smithi038 ceph-mon[32348]: pgmap v232: 97 pgs: 97 active+clean; 410 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 139 op/s 2024-04-23T14:51:37.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:36 smithi028 ceph-mon[27810]: pgmap v232: 97 pgs: 97 active+clean; 410 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 139 op/s 2024-04-23T14:51:37.506 INFO:tasks.workunit.client.0.smithi028.stdout: 1 626992 9.15 MB/sec execute 233 sec latency 28.700 ms 2024-04-23T14:51:38.506 INFO:tasks.workunit.client.0.smithi028.stdout: 1 628849 9.16 MB/sec execute 234 sec latency 14.956 ms 2024-04-23T14:51:39.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:38 smithi105 ceph-mon[32555]: pgmap v233: 97 pgs: 97 active+clean; 417 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.7 MiB/s wr, 205 op/s 2024-04-23T14:51:39.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:38 smithi038 ceph-mon[32348]: pgmap v233: 97 pgs: 97 active+clean; 417 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.7 MiB/s wr, 205 op/s 2024-04-23T14:51:39.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:38 smithi028 ceph-mon[27810]: pgmap v233: 97 pgs: 97 active+clean; 417 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.7 MiB/s wr, 205 op/s 2024-04-23T14:51:39.506 INFO:tasks.workunit.client.0.smithi028.stdout: 1 630027 9.15 MB/sec execute 235 sec latency 39.725 ms 2024-04-23T14:51:40.506 INFO:tasks.workunit.client.0.smithi028.stdout: 1 631623 9.14 MB/sec execute 236 sec latency 47.440 ms 2024-04-23T14:51:41.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:40 smithi105 ceph-mon[32555]: pgmap v234: 97 pgs: 97 active+clean; 417 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 135 op/s 2024-04-23T14:51:41.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:40 smithi038 ceph-mon[32348]: pgmap v234: 97 pgs: 97 active+clean; 417 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 135 op/s 2024-04-23T14:51:41.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:40 smithi028 ceph-mon[27810]: pgmap v234: 97 pgs: 97 active+clean; 417 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 135 op/s 2024-04-23T14:51:41.506 INFO:tasks.workunit.client.0.smithi028.stdout: 1 632889 9.13 MB/sec execute 237 sec latency 36.970 ms 2024-04-23T14:51:42.506 INFO:tasks.workunit.client.0.smithi028.stdout: 1 634501 9.12 MB/sec execute 238 sec latency 21.144 ms 2024-04-23T14:51:43.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:42 smithi105 ceph-mon[32555]: pgmap v235: 97 pgs: 97 active+clean; 417 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 135 op/s 2024-04-23T14:51:43.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:42 smithi038 ceph-mon[32348]: pgmap v235: 97 pgs: 97 active+clean; 417 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 135 op/s 2024-04-23T14:51:43.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:42 smithi028 ceph-mon[27810]: pgmap v235: 97 pgs: 97 active+clean; 417 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 135 op/s 2024-04-23T14:51:43.506 INFO:tasks.workunit.client.0.smithi028.stdout: 1 636077 9.13 MB/sec execute 239 sec latency 21.127 ms 2024-04-23T14:51:44.506 INFO:tasks.workunit.client.0.smithi028.stdout: 1 637197 9.12 MB/sec execute 240 sec latency 48.148 ms 2024-04-23T14:51:44.515 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:51:44.515 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:51:44.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.744+0000 7f3867e00640 1 -- 172.21.15.28:0/2832747739 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3868102ac0 msgr2=0x7f3868106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:44.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.744+0000 7f3867e00640 1 --2- 172.21.15.28:0/2832747739 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3868102ac0 0x7f3868106b00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3850009860 tx=0x7f385002a340 comp rx=0 tx=0).stop 2024-04-23T14:51:44.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.745+0000 7f3867e00640 1 -- 172.21.15.28:0/2832747739 shutdown_connections 2024-04-23T14:51:44.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.745+0000 7f3867e00640 1 --2- 172.21.15.28:0/2832747739 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f38681071e0 0x7f386810edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:44.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.745+0000 7f3867e00640 1 --2- 172.21.15.28:0/2832747739 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3868102ac0 0x7f3868106b00 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:44.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.745+0000 7f3867e00640 1 --2- 172.21.15.28:0/2832747739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3868102110 0x7f38681024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:44.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.745+0000 7f3867e00640 1 -- 172.21.15.28:0/2832747739 >> 172.21.15.28:0/2832747739 conn(0x7f38680fd9d0 msgr2=0x7f38680ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:44.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.745+0000 7f3867e00640 1 -- 172.21.15.28:0/2832747739 shutdown_connections 2024-04-23T14:51:44.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.745+0000 7f3867e00640 1 -- 172.21.15.28:0/2832747739 wait complete. 2024-04-23T14:51:44.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.746+0000 7f3867e00640 1 Processor -- start 2024-04-23T14:51:44.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.746+0000 7f3867e00640 1 -- start start 2024-04-23T14:51:44.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.746+0000 7f3867e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3868102110 0x7f386819e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:44.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.746+0000 7f3867e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f38681071e0 0x7f386819e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:44.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.746+0000 7f3867e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38681985f0 0x7f3868198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:44.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.746+0000 7f3867e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3868112250 con 0x7f38681985f0 2024-04-23T14:51:44.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.746+0000 7f3867e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f38681120d0 con 0x7f3868102110 2024-04-23T14:51:44.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.746+0000 7f3867e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f38681123d0 con 0x7f38681071e0 2024-04-23T14:51:44.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.747+0000 7f3867400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38681985f0 0x7f3868198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:44.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.747+0000 7f3866a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3868102110 0x7f386819e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:44.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.747+0000 7f3866a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3868102110 0x7f386819e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:45464/0 (socket says 172.21.15.28:45464) 2024-04-23T14:51:44.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.747+0000 7f3866a00640 1 -- 172.21.15.28:0/3249435357 learned_addr learned my addr 172.21.15.28:0/3249435357 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:44.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.747+0000 7f3866000640 1 --2- 172.21.15.28:0/3249435357 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f38681071e0 0x7f386819e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:44.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.747+0000 7f3866a00640 1 -- 172.21.15.28:0/3249435357 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f38681071e0 msgr2=0x7f386819e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:44.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.747+0000 7f3866a00640 1 --2- 172.21.15.28:0/3249435357 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f38681071e0 0x7f386819e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:44.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.747+0000 7f3866a00640 1 -- 172.21.15.28:0/3249435357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38681985f0 msgr2=0x7f3868198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:44.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.747+0000 7f3866a00640 1 --2- 172.21.15.28:0/3249435357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38681985f0 0x7f3868198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:44.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.747+0000 7f3866a00640 1 -- 172.21.15.28:0/3249435357 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3868199110 con 0x7f3868102110 2024-04-23T14:51:44.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.748+0000 7f3866a00640 1 --2- 172.21.15.28:0/3249435357 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3868102110 0x7f386819e3f0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f385800ebc0 tx=0x7f385800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.748+0000 7f386ca00640 1 -- 172.21.15.28:0/3249435357 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3858010430 con 0x7f3868102110 2024-04-23T14:51:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.748+0000 7f3867e00640 1 -- 172.21.15.28:0/3249435357 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3868199400 con 0x7f3868102110 2024-04-23T14:51:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.748+0000 7f386ca00640 1 -- 172.21.15.28:0/3249435357 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3858010ab0 con 0x7f3868102110 2024-04-23T14:51:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.748+0000 7f386ca00640 1 -- 172.21.15.28:0/3249435357 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3858018b30 con 0x7f3868102110 2024-04-23T14:51:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.748+0000 7f3867e00640 1 -- 172.21.15.28:0/3249435357 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38681a5650 con 0x7f3868102110 2024-04-23T14:51:44.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.751+0000 7f386ca00640 1 -- 172.21.15.28:0/3249435357 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f3858018cd0 con 0x7f3868102110 2024-04-23T14:51:44.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.752+0000 7f3870dbf740 1 Processor -- start 2024-04-23T14:51:44.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.752+0000 7f3870dbf740 1 -- start start 2024-04-23T14:51:44.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.752+0000 7f3870dbf740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fb24b27af0 0x55fb24b27ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:44.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.752+0000 7f3870dbf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb24b23350 0x55fb24b237d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:44.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.752+0000 7f3870dbf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb24b23d10 0x55fb24b2fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:44.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.752+0000 7f3870dbf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fb24a7f1f0 con 0x55fb24b23350 2024-04-23T14:51:44.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.752+0000 7f3870dbf740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55fb24a25350 con 0x55fb24b27af0 2024-04-23T14:51:44.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.752+0000 7f3870dbf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55fb24a7b1b0 con 0x55fb24b23d10 2024-04-23T14:51:44.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.752+0000 7f3867400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fb24b27af0 0x55fb24b27ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:44.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.753+0000 7f3866a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb24b23350 0x55fb24b237d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:44.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.753+0000 7f3866000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb24b23d10 0x55fb24b2fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:44.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.753+0000 7f3867400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fb24b27af0 0x55fb24b27ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:45478/0 (socket says 172.21.15.28:45478) 2024-04-23T14:51:44.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.753+0000 7f3867400640 1 -- 172.21.15.28:0/36975425 learned_addr learned my addr 172.21.15.28:0/36975425 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:44.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.753+0000 7f3867400640 1 -- 172.21.15.28:0/36975425 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb24b23d10 msgr2=0x55fb24b2fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:44.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.753+0000 7f3867400640 1 --2- 172.21.15.28:0/36975425 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb24b23d10 0x55fb24b2fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:44.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.753+0000 7f3867400640 1 -- 172.21.15.28:0/36975425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb24b23350 msgr2=0x55fb24b237d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:44.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.753+0000 7f3867400640 1 --2- 172.21.15.28:0/36975425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb24b23350 0x55fb24b237d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:44.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.753+0000 7f3867400640 1 -- 172.21.15.28:0/36975425 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb24a79f50 con 0x55fb24b27af0 2024-04-23T14:51:44.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.753+0000 7f386ca00640 1 --2- 172.21.15.28:0/3249435357 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f384c0a3da0 0x7f384c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:44.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.753+0000 7f3866a00640 1 --2- 172.21.15.28:0/36975425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb24b23350 0x55fb24b237d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:51:44.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.753+0000 7f386ca00640 1 -- 172.21.15.28:0/3249435357 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8542+0+0 (secure 0 0 0) 0x7f38580d0ee0 con 0x7f3868102110 2024-04-23T14:51:44.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.754+0000 7f3867400640 1 --2- 172.21.15.28:0/36975425 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fb24b27af0 0x55fb24b27ed0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3860009be0 tx=0x7f3860011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf7fb3dad3f673a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:44.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.754+0000 7f3866a00640 1 --2- 172.21.15.28:0/3249435357 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f384c0a3da0 0x7f384c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:44.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.754+0000 7f3857e00640 1 -- 172.21.15.28:0/36975425 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386001e070 con 0x55fb24b27af0 2024-04-23T14:51:44.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.754+0000 7f3857e00640 1 -- 172.21.15.28:0/36975425 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3860009d90 con 0x55fb24b27af0 2024-04-23T14:51:44.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.754+0000 7f3857e00640 1 -- 172.21.15.28:0/36975425 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3860018810 con 0x55fb24b27af0 2024-04-23T14:51:44.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.755+0000 7f3870dbf740 1 -- 172.21.15.28:0/36975425 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fb24b27af0 msgr2=0x55fb24b27ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:44.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.755+0000 7f3870dbf740 1 --2- 172.21.15.28:0/36975425 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fb24b27af0 0x55fb24b27ed0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3860009be0 tx=0x7f3860011810 comp rx=0 tx=0).stop 2024-04-23T14:51:44.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.755+0000 7f3866a00640 1 --2- 172.21.15.28:0/3249435357 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f384c0a3da0 0x7f384c0a6260 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x55fb24b19340 tx=0x7f38580d4990 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:44.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.755+0000 7f3870dbf740 1 -- 172.21.15.28:0/36975425 shutdown_connections 2024-04-23T14:51:44.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.755+0000 7f3870dbf740 1 --2- 172.21.15.28:0/36975425 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb24b23d10 0x55fb24b2fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:44.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.755+0000 7f3870dbf740 1 --2- 172.21.15.28:0/36975425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb24b23350 0x55fb24b237d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:44.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.755+0000 7f3870dbf740 1 --2- 172.21.15.28:0/36975425 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fb24b27af0 0x55fb24b27ed0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:44.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.755+0000 7f3870dbf740 1 -- 172.21.15.28:0/36975425 >> 172.21.15.28:0/36975425 conn(0x55fb24b15800 msgr2=0x55fb24b15be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:44.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.755+0000 7f3870dbf740 1 -- 172.21.15.28:0/36975425 shutdown_connections 2024-04-23T14:51:44.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.755+0000 7f3870dbf740 1 -- 172.21.15.28:0/36975425 wait complete. 2024-04-23T14:51:44.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.756+0000 7f3870dbf740 1 -- start start 2024-04-23T14:51:44.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.756+0000 7f3870dbf740 1 Processor -- start 2024-04-23T14:51:44.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.756+0000 7f3870dbf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb24b23350 0x55fb24b2ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:44.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.756+0000 7f3870dbf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb24b23d10 0x55fb24b2d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:44.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.756+0000 7f3870dbf740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fb24b27af0 0x55fb24b2d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:44.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.756+0000 7f3870dbf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fb24b30040 con 0x55fb24b23350 2024-04-23T14:51:44.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.756+0000 7f3870dbf740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55fb24a38690 con 0x55fb24b27af0 2024-04-23T14:51:44.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.756+0000 7f3870dbf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55fb24b301c0 con 0x55fb24b23d10 2024-04-23T14:51:44.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.757+0000 7f3870dbf740 10 client.0 _ll_register_callbacks cb 0x55fb24a74650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:51:44.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.757+0000 7f3866000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb24b23d10 0x55fb24b2d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:44.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.757+0000 7f3866000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb24b23d10 0x55fb24b2d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:59600/0 (socket says 172.21.15.28:59600) 2024-04-23T14:51:44.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.757+0000 7f3866000640 1 -- 172.21.15.28:0/830628109 learned_addr learned my addr 172.21.15.28:0/830628109 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:44.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.757+0000 7f3867400640 1 --2- 172.21.15.28:0/830628109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb24b23350 0x55fb24b2ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:44.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.757+0000 7f3866000640 1 -- 172.21.15.28:0/830628109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fb24b27af0 msgr2=0x55fb24b2d930 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-23T14:51:44.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.757+0000 7f3866000640 1 --2- 172.21.15.28:0/830628109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fb24b27af0 0x55fb24b2d930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:44.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.757+0000 7f3866000640 1 -- 172.21.15.28:0/830628109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb24b23350 msgr2=0x55fb24b2ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:44.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.757+0000 7f3866000640 1 --2- 172.21.15.28:0/830628109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb24b23350 0x55fb24b2ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:44.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.757+0000 7f3866000640 1 -- 172.21.15.28:0/830628109 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb24a37920 con 0x55fb24b23d10 2024-04-23T14:51:44.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.757+0000 7f3866000640 1 --2- 172.21.15.28:0/830628109 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb24b23d10 0x55fb24b2d3f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f386806b560 tx=0x7f3850030340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b068ff4238fb5138 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:44.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.757+0000 7f3854c00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:51:44.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.758+0000 7f3854c00640 1 -- 172.21.15.28:0/830628109 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3850041070 con 0x55fb24b23d10 2024-04-23T14:51:44.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.758+0000 7f3854c00640 1 -- 172.21.15.28:0/830628109 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8542+0+0 (secure 0 0 0) 0x7f3850030550 con 0x55fb24b23d10 2024-04-23T14:51:44.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.758+0000 7f3870dbf740 1 -- 172.21.15.28:0/830628109 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3860009d90 con 0x55fb24b23d10 2024-04-23T14:51:44.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.758+0000 7f3854c00640 1 -- 172.21.15.28:0/830628109 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f385004d050 con 0x55fb24b23d10 2024-04-23T14:51:44.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.758+0000 7f3870dbf740 10 client.26368 fetch_fsmap learned FSMap version 20 2024-04-23T14:51:44.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.758+0000 7f3870dbf740 1 -- 172.21.15.28:0/830628109 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55fb2485b7d0 con 0x55fb24b23d10 2024-04-23T14:51:44.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.758+0000 7f3854c00640 1 -- 172.21.15.28:0/830628109 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3860009d90 con 0x55fb24b23d10 2024-04-23T14:51:44.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.759+0000 7f3854c00640 1 -- 172.21.15.28:0/830628109 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f385003d450 con 0x55fb24b23d10 2024-04-23T14:51:44.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.759+0000 7f3870dbf740 10 client.26368 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:51:44.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.759+0000 7f3870dbf740 10 client.26368 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:51:44.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.759+0000 7f3870dbf740 20 client.26368 populate_metadata read hostname 'smithi028' 2024-04-23T14:51:44.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.759+0000 7f3870dbf740 1 --2- 172.21.15.28:0/830628109 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55fb24b3d500 0x55fb24b3f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:44.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.759+0000 7f3870dbf740 4 client.26368 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:51:44.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.759+0000 7f3870dbf740 1 -- 172.21.15.28:0/830628109 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fb24a8cc10 con 0x55fb24b3d500 2024-04-23T14:51:44.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.759+0000 7f3867400640 1 --2- 172.21.15.28:0/830628109 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55fb24b3d500 0x55fb24b3f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:44.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.760+0000 7f3867400640 1 --2- 172.21.15.28:0/830628109 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55fb24b3d500 0x55fb24b3f950 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ab3632c039c990 server_cookie=5be441085d438ad0 in_seq=0 out_seq=0 2024-04-23T14:51:44.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.760+0000 7f3854c00640 10 client.26368 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:51:44.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.761+0000 7f3854c00640 1 -- 172.21.15.28:0/830628109 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fb24a8cc10 con 0x55fb24b3d500 2024-04-23T14:51:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.761+0000 7f3854c00640 10 client.26368 handle_command_reply: tid=0 2024-04-23T14:51:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.761+0000 7f3870dbf740 1 client.26368 shutdown 2024-04-23T14:51:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.761+0000 7f3870dbf740 1 -- 172.21.15.28:0/830628109 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb24b23d10 msgr2=0x55fb24b2d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.761+0000 7f3870dbf740 1 --2- 172.21.15.28:0/830628109 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb24b23d10 0x55fb24b2d3f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f386806b560 tx=0x7f3850030340 comp rx=0 tx=0).stop 2024-04-23T14:51:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.761+0000 7f3870dbf740 1 -- 172.21.15.28:0/830628109 shutdown_connections 2024-04-23T14:51:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.761+0000 7f3870dbf740 1 --2- 172.21.15.28:0/830628109 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55fb24b3d500 0x55fb24b3f950 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.761+0000 7f3870dbf740 1 --2- 172.21.15.28:0/830628109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fb24b27af0 0x55fb24b2d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.761+0000 7f3870dbf740 1 --2- 172.21.15.28:0/830628109 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb24b23d10 0x55fb24b2d3f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.761+0000 7f3854c00640 0 client.26368 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:51:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.761+0000 7f3870dbf740 1 --2- 172.21.15.28:0/830628109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb24b23350 0x55fb24b2ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.761+0000 7f3870dbf740 1 -- 172.21.15.28:0/830628109 >> 172.21.15.28:0/830628109 conn(0x55fb24b15800 msgr2=0x55fb24b22bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.761+0000 7f3870dbf740 1 -- 172.21.15.28:0/830628109 shutdown_connections 2024-04-23T14:51:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.762+0000 7f3870dbf740 1 -- 172.21.15.28:0/830628109 wait complete. 2024-04-23T14:51:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.762+0000 7f3870dbf740 20 client.26368 trim_cache size 0 max 16384 2024-04-23T14:51:44.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.783+0000 7f3870dbf740 1 Processor -- start 2024-04-23T14:51:44.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.784+0000 7f3870dbf740 1 -- start start 2024-04-23T14:51:44.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.784+0000 7f3870dbf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb24b23350 0x55fb24b38f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:44.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.784+0000 7f3870dbf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb24b23d10 0x55fb24b25670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:44.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.784+0000 7f3870dbf740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fb24b27af0 0x55fb24b45220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:44.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.784+0000 7f3870dbf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f38681a99c0 con 0x55fb24b23d10 2024-04-23T14:51:44.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.784+0000 7f3870dbf740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f38681ac220 con 0x55fb24b27af0 2024-04-23T14:51:44.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.784+0000 7f3870dbf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f38681aa010 con 0x55fb24b23350 2024-04-23T14:51:44.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.784+0000 7f3867400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb24b23350 0x55fb24b38f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:44.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.784+0000 7f3866000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb24b23d10 0x55fb24b25670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:44.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.784+0000 7f3867400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb24b23350 0x55fb24b38f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:59602/0 (socket says 172.21.15.28:59602) 2024-04-23T14:51:44.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.784+0000 7f3867400640 1 -- 172.21.15.28:0/1413860742 learned_addr learned my addr 172.21.15.28:0/1413860742 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:44.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.785+0000 7f3867400640 1 --2- 172.21.15.28:0/1413860742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fb24b27af0 0x55fb24b45220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:44.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.785+0000 7f3867400640 1 -- 172.21.15.28:0/1413860742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fb24b27af0 msgr2=0x55fb24b45220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:44.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.785+0000 7f3867400640 1 --2- 172.21.15.28:0/1413860742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fb24b27af0 0x55fb24b45220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:44.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.785+0000 7f3867400640 1 -- 172.21.15.28:0/1413860742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb24b23d10 msgr2=0x55fb24b25670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:44.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.785+0000 7f3867400640 1 --2- 172.21.15.28:0/1413860742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb24b23d10 0x55fb24b25670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:44.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.785+0000 7f3867400640 1 -- 172.21.15.28:0/1413860742 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb24b28240 con 0x55fb24b23350 2024-04-23T14:51:44.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.785+0000 7f3867400640 1 --2- 172.21.15.28:0/1413860742 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb24b23350 0x55fb24b38f40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3860009670 tx=0x7f3860019f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=736501338ad171ff server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:44.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.785+0000 7f3857400640 1 -- 172.21.15.28:0/1413860742 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386001e040 con 0x55fb24b23350 2024-04-23T14:51:44.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.785+0000 7f3857400640 1 -- 172.21.15.28:0/1413860742 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38600035f0 con 0x55fb24b23350 2024-04-23T14:51:44.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.785+0000 7f3857400640 1 -- 172.21.15.28:0/1413860742 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3860019670 con 0x55fb24b23350 2024-04-23T14:51:44.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.786+0000 7f3870dbf740 1 -- 172.21.15.28:0/1413860742 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb24b23350 msgr2=0x55fb24b38f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:44.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.786+0000 7f3870dbf740 1 --2- 172.21.15.28:0/1413860742 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb24b23350 0x55fb24b38f40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3860009670 tx=0x7f3860019f50 comp rx=0 tx=0).stop 2024-04-23T14:51:44.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.786+0000 7f3870dbf740 1 -- 172.21.15.28:0/1413860742 shutdown_connections 2024-04-23T14:51:44.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.786+0000 7f3870dbf740 1 --2- 172.21.15.28:0/1413860742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fb24b27af0 0x55fb24b45220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:44.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.786+0000 7f3870dbf740 1 --2- 172.21.15.28:0/1413860742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb24b23d10 0x55fb24b25670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:44.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.786+0000 7f3870dbf740 1 --2- 172.21.15.28:0/1413860742 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb24b23350 0x55fb24b38f40 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:44.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.786+0000 7f3870dbf740 1 -- 172.21.15.28:0/1413860742 >> 172.21.15.28:0/1413860742 conn(0x55fb24b15800 msgr2=0x55fb24b270a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:44.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.787+0000 7f3870dbf740 1 -- 172.21.15.28:0/1413860742 shutdown_connections 2024-04-23T14:51:44.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.787+0000 7f3870dbf740 1 -- 172.21.15.28:0/1413860742 wait complete. 2024-04-23T14:51:44.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.787+0000 7f3870dbf740 1 -- start start 2024-04-23T14:51:44.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.787+0000 7f3870dbf740 1 Processor -- start 2024-04-23T14:51:44.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.788+0000 7f3870dbf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb24b23350 0x55fb24b447b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:44.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.788+0000 7f3870dbf740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fb24b23d10 0x55fb24b44cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:44.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.788+0000 7f3870dbf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb24b27af0 0x55fb24b45230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:44.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.788+0000 7f3870dbf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f38681aa6e0 con 0x55fb24b23350 2024-04-23T14:51:44.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.788+0000 7f3870dbf740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f38681abb50 con 0x55fb24b23d10 2024-04-23T14:51:44.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.788+0000 7f3870dbf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f38681aadb0 con 0x55fb24b27af0 2024-04-23T14:51:44.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.788+0000 7f3870dbf740 10 client.0 _ll_register_callbacks cb 0x7f3860005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:51:44.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.788+0000 7f3866000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fb24b23d10 0x55fb24b44cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:44.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.788+0000 7f3866000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fb24b23d10 0x55fb24b44cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:45518/0 (socket says 172.21.15.28:45518) 2024-04-23T14:51:44.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.788+0000 7f3866000640 1 -- 172.21.15.28:0/2643558347 learned_addr learned my addr 172.21.15.28:0/2643558347 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:44.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.788+0000 7f3867400640 1 --2- 172.21.15.28:0/2643558347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb24b23350 0x55fb24b447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:44.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.788+0000 7f3867400640 1 --2- 172.21.15.28:0/2643558347 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb24b27af0 0x55fb24b45230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:44.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.788+0000 7f3866000640 1 -- 172.21.15.28:0/2643558347 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb24b27af0 msgr2=0x55fb24b45230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:44.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.788+0000 7f3866000640 1 --2- 172.21.15.28:0/2643558347 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb24b27af0 0x55fb24b45230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:44.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.788+0000 7f3866000640 1 -- 172.21.15.28:0/2643558347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb24b23350 msgr2=0x55fb24b447b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:44.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.788+0000 7f3866000640 1 --2- 172.21.15.28:0/2643558347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb24b23350 0x55fb24b447b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:44.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.788+0000 7f3866000640 1 -- 172.21.15.28:0/2643558347 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb247913f0 con 0x55fb24b23d10 2024-04-23T14:51:44.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.788+0000 7f3867400640 1 --2- 172.21.15.28:0/2643558347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb24b23350 0x55fb24b447b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:51:44.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.789+0000 7f3866000640 1 --2- 172.21.15.28:0/2643558347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fb24b23d10 0x55fb24b44cf0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f385003d8a0 tx=0x7f38500045c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e08d765b4c9e322 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:44.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.789+0000 7f3854c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:51:44.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.789+0000 7f3854c00640 1 -- 172.21.15.28:0/2643558347 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3850041040 con 0x55fb24b23d10 2024-04-23T14:51:44.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.789+0000 7f3870dbf740 1 -- 172.21.15.28:0/2643558347 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38600035f0 con 0x55fb24b23d10 2024-04-23T14:51:44.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.789+0000 7f3854c00640 1 -- 172.21.15.28:0/2643558347 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8542+0+0 (secure 0 0 0) 0x7f3850030550 con 0x55fb24b23d10 2024-04-23T14:51:44.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.789+0000 7f3854c00640 1 -- 172.21.15.28:0/2643558347 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f3850000e60 con 0x55fb24b23d10 2024-04-23T14:51:44.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.789+0000 7f3870dbf740 10 client.26384 fetch_fsmap learned FSMap version 20 2024-04-23T14:51:44.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.789+0000 7f3870dbf740 1 -- 172.21.15.28:0/2643558347 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55fb247510f0 con 0x55fb24b23d10 2024-04-23T14:51:44.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.790+0000 7f3854c00640 1 -- 172.21.15.28:0/2643558347 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38600035f0 con 0x55fb24b23d10 2024-04-23T14:51:44.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.790+0000 7f3854c00640 1 -- 172.21.15.28:0/2643558347 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f3850003ad0 con 0x55fb24b23d10 2024-04-23T14:51:44.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.790+0000 7f3870dbf740 10 client.26384 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:51:44.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.790+0000 7f3870dbf740 10 client.26384 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:51:44.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.790+0000 7f3870dbf740 20 client.26384 populate_metadata read hostname 'smithi028' 2024-04-23T14:51:44.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.790+0000 7f3870dbf740 1 --2- 172.21.15.28:0/2643558347 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55fb24ba4220 0x55fb24ba6640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:44.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.790+0000 7f3870dbf740 4 client.26384 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:51:44.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.790+0000 7f3870dbf740 1 -- 172.21.15.28:0/2643558347 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fb24a86a80 con 0x55fb24ba4220 2024-04-23T14:51:44.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.790+0000 7f3867400640 1 --2- 172.21.15.28:0/2643558347 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55fb24ba4220 0x55fb24ba6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:44.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.791+0000 7f3867400640 1 --2- 172.21.15.28:0/2643558347 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55fb24ba4220 0x55fb24ba6640 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f360b84fd84a02e8 server_cookie=341ebd18cba72cfd in_seq=0 out_seq=0 2024-04-23T14:51:44.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.791+0000 7f3854c00640 10 client.26384 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:51:44.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.791+0000 7f3854c00640 1 -- 172.21.15.28:0/2643558347 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55fb24a86a80 con 0x55fb24ba4220 2024-04-23T14:51:44.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.791+0000 7f3854c00640 10 client.26384 handle_command_reply: tid=0 2024-04-23T14:51:44.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.791+0000 7f3870dbf740 1 client.26384 shutdown 2024-04-23T14:51:44.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.791+0000 7f3870dbf740 1 -- 172.21.15.28:0/2643558347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fb24b23d10 msgr2=0x55fb24b44cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:44.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.791+0000 7f3870dbf740 1 --2- 172.21.15.28:0/2643558347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fb24b23d10 0x55fb24b44cf0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f385003d8a0 tx=0x7f38500045c0 comp rx=0 tx=0).stop 2024-04-23T14:51:44.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.791+0000 7f3870dbf740 1 -- 172.21.15.28:0/2643558347 shutdown_connections 2024-04-23T14:51:44.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.791+0000 7f3870dbf740 1 --2- 172.21.15.28:0/2643558347 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55fb24ba4220 0x55fb24ba6640 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:44.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.791+0000 7f3870dbf740 1 --2- 172.21.15.28:0/2643558347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fb24b23d10 0x55fb24b44cf0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:44.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.791+0000 7f3870dbf740 1 --2- 172.21.15.28:0/2643558347 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fb24b27af0 0x55fb24b45230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:44.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.791+0000 7f3854c00640 0 client.26384 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:51:44.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.791+0000 7f3870dbf740 1 --2- 172.21.15.28:0/2643558347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb24b23350 0x55fb24b447b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:44.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.791+0000 7f3870dbf740 1 -- 172.21.15.28:0/2643558347 >> 172.21.15.28:0/2643558347 conn(0x55fb24b15800 msgr2=0x55fb24b14740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:44.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.791+0000 7f3870dbf740 1 -- 172.21.15.28:0/2643558347 shutdown_connections 2024-04-23T14:51:44.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.792+0000 7f3870dbf740 1 -- 172.21.15.28:0/2643558347 wait complete. 2024-04-23T14:51:44.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.792+0000 7f3870dbf740 20 client.26384 trim_cache size 0 max 16384 2024-04-23T14:51:44.799 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:51:44.799 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-23T14:51:44.799 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-23T14:51:44.800 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:51:44.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.792+0000 7f3867e00640 1 -- 172.21.15.28:0/3249435357 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f384c0a3da0 msgr2=0x7f384c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:44.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.792+0000 7f3867e00640 1 --2- 172.21.15.28:0/3249435357 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f384c0a3da0 0x7f384c0a6260 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x55fb24b19340 tx=0x7f38580d4990 comp rx=0 tx=0).stop 2024-04-23T14:51:44.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.792+0000 7f3867e00640 1 -- 172.21.15.28:0/3249435357 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3868102110 msgr2=0x7f386819e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:44.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.792+0000 7f3867e00640 1 --2- 172.21.15.28:0/3249435357 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3868102110 0x7f386819e3f0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f385800ebc0 tx=0x7f385800c2d0 comp rx=0 tx=0).stop 2024-04-23T14:51:44.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.792+0000 7f3867e00640 1 -- 172.21.15.28:0/3249435357 shutdown_connections 2024-04-23T14:51:44.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.792+0000 7f3867e00640 1 --2- 172.21.15.28:0/3249435357 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f384c0a3da0 0x7f384c0a6260 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:44.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.792+0000 7f3867e00640 1 --2- 172.21.15.28:0/3249435357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38681985f0 0x7f3868198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:44.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.792+0000 7f3867e00640 1 --2- 172.21.15.28:0/3249435357 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f38681071e0 0x7f386819e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:44.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.792+0000 7f3867e00640 1 --2- 172.21.15.28:0/3249435357 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3868102110 0x7f386819e3f0 secure :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f385800ebc0 tx=0x7f385800c2d0 comp rx=0 tx=0).stop 2024-04-23T14:51:44.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.792+0000 7f3867e00640 1 -- 172.21.15.28:0/3249435357 >> 172.21.15.28:0/3249435357 conn(0x7f38680fd9d0 msgr2=0x7f38680fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:44.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.793+0000 7f3867e00640 1 -- 172.21.15.28:0/3249435357 shutdown_connections 2024-04-23T14:51:44.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:44.793+0000 7f3867e00640 1 -- 172.21.15.28:0/3249435357 wait complete. 2024-04-23T14:51:44.810 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-23T14:51:44.811 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:51:44.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:44 smithi028 ceph-mon[27810]: pgmap v236: 97 pgs: 97 active+clean; 424 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 199 op/s 2024-04-23T14:51:45.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.009+0000 7fd0bba00640 1 -- 172.21.15.28:0/2198659441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0b406a660 msgr2=0x7fd0b406aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:45.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.009+0000 7fd0bba00640 1 --2- 172.21.15.28:0/2198659441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0b406a660 0x7fd0b406aac0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd09c009900 tx=0x7fd09c02a3f0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.009+0000 7fd0bba00640 1 -- 172.21.15.28:0/2198659441 shutdown_connections 2024-04-23T14:51:45.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.009+0000 7fd0bba00640 1 --2- 172.21.15.28:0/2198659441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0b406a660 0x7fd0b406aac0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.009+0000 7fd0bba00640 1 --2- 172.21.15.28:0/2198659441 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0b406b000 0x7fd0b410ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.009+0000 7fd0bba00640 1 --2- 172.21.15.28:0/2198659441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0b4108070 0x7fd0b4108450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.009+0000 7fd0bba00640 1 -- 172.21.15.28:0/2198659441 >> 172.21.15.28:0/2198659441 conn(0x7fd0b4077010 msgr2=0x7fd0b4077420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:45.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.009+0000 7fd0bba00640 1 -- 172.21.15.28:0/2198659441 shutdown_connections 2024-04-23T14:51:45.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.009+0000 7fd0bba00640 1 -- 172.21.15.28:0/2198659441 wait complete. 2024-04-23T14:51:45.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.010+0000 7fd0bba00640 1 Processor -- start 2024-04-23T14:51:45.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.010+0000 7fd0bba00640 1 -- start start 2024-04-23T14:51:45.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.010+0000 7fd0bba00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0b406a660 0x7fd0b41a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:45.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.011+0000 7fd0bba00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0b406b000 0x7fd0b41a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:45.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.011+0000 7fd0bba00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0b4108070 0x7fd0b419a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:45.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.011+0000 7fd0bba00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b41123c0 con 0x7fd0b4108070 2024-04-23T14:51:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.011+0000 7fd0bba00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b4112240 con 0x7fd0b406b000 2024-04-23T14:51:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.011+0000 7fd0bba00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b4112540 con 0x7fd0b406a660 2024-04-23T14:51:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.011+0000 7fd0b9000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0b406a660 0x7fd0b41a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.011+0000 7fd0b9a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0b4108070 0x7fd0b419a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.011+0000 7fd0abe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0b406b000 0x7fd0b41a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.011+0000 7fd0b9000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0b406a660 0x7fd0b41a06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:59634/0 (socket says 172.21.15.28:59634) 2024-04-23T14:51:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.011+0000 7fd0b9000640 1 -- 172.21.15.28:0/3775068867 learned_addr learned my addr 172.21.15.28:0/3775068867 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.011+0000 7fd0b9000640 1 -- 172.21.15.28:0/3775068867 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0b406b000 msgr2=0x7fd0b41a0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.011+0000 7fd0b9000640 1 --2- 172.21.15.28:0/3775068867 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0b406b000 0x7fd0b41a0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.011+0000 7fd0b9000640 1 -- 172.21.15.28:0/3775068867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0b4108070 msgr2=0x7fd0b419a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.011+0000 7fd0b9000640 1 --2- 172.21.15.28:0/3775068867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0b4108070 0x7fd0b419a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.011+0000 7fd0b9000640 1 -- 172.21.15.28:0/3775068867 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0b419aff0 con 0x7fd0b406a660 2024-04-23T14:51:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.012+0000 7fd0b9000640 1 --2- 172.21.15.28:0/3775068867 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0b406a660 0x7fd0b41a06c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd0a400e9f0 tx=0x7fd0a400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.012+0000 7fd0b2000640 1 -- 172.21.15.28:0/3775068867 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a400cd80 con 0x7fd0b406a660 2024-04-23T14:51:45.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.012+0000 7fd0b2000640 1 -- 172.21.15.28:0/3775068867 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0a4010910 con 0x7fd0b406a660 2024-04-23T14:51:45.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.012+0000 7fd0bba00640 1 -- 172.21.15.28:0/3775068867 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0b419b2e0 con 0x7fd0b406a660 2024-04-23T14:51:45.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.012+0000 7fd0b2000640 1 -- 172.21.15.28:0/3775068867 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a4018990 con 0x7fd0b406a660 2024-04-23T14:51:45.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.012+0000 7fd0bba00640 1 -- 172.21.15.28:0/3775068867 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0b41a7580 con 0x7fd0b406a660 2024-04-23T14:51:45.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.013+0000 7fd0bba00640 1 -- 172.21.15.28:0/3775068867 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0b4103710 con 0x7fd0b406a660 2024-04-23T14:51:45.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.015+0000 7fd0b2000640 1 -- 172.21.15.28:0/3775068867 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fd0a4010ac0 con 0x7fd0b406a660 2024-04-23T14:51:45.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.017+0000 7fd0b2000640 1 --2- 172.21.15.28:0/3775068867 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd0980a3da0 0x7fd0980a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:45.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.017+0000 7fd0b2000640 1 -- 172.21.15.28:0/3775068867 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8542+0+0 (secure 0 0 0) 0x7fd0a40d0af0 con 0x7fd0b406a660 2024-04-23T14:51:45.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.017+0000 7fd0abe00640 1 --2- 172.21.15.28:0/3775068867 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd0980a3da0 0x7fd0980a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:45.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.018+0000 7fd0abe00640 1 --2- 172.21.15.28:0/3775068867 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd0980a3da0 0x7fd0980a6260 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd09c000c00 tx=0x7fd09c005b50 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:45.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.021+0000 7fd0b2000640 1 -- 172.21.15.28:0/3775068867 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd0a409cad0 con 0x7fd0b406a660 2024-04-23T14:51:45.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:44 smithi105 ceph-mon[32555]: pgmap v236: 97 pgs: 97 active+clean; 424 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 199 op/s 2024-04-23T14:51:45.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:44 smithi038 ceph-mon[32348]: pgmap v236: 97 pgs: 97 active+clean; 424 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 199 op/s 2024-04-23T14:51:45.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.297+0000 7fd0bba00640 1 -- 172.21.15.28:0/3775068867 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd0b41a77b0 con 0x7fd0b406a660 2024-04-23T14:51:45.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.298+0000 7fd0b2000640 1 -- 172.21.15.28:0/3775068867 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7fd0a40a0dd0 con 0x7fd0b406a660 2024-04-23T14:51:45.298 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:51:45.299 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:51:45.299 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:51:45.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.301+0000 7fd0bba00640 1 -- 172.21.15.28:0/3775068867 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd0980a3da0 msgr2=0x7fd0980a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:45.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.301+0000 7fd0bba00640 1 --2- 172.21.15.28:0/3775068867 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd0980a3da0 0x7fd0980a6260 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd09c000c00 tx=0x7fd09c005b50 comp rx=0 tx=0).stop 2024-04-23T14:51:45.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.301+0000 7fd0bba00640 1 -- 172.21.15.28:0/3775068867 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0b406a660 msgr2=0x7fd0b41a06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:45.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.301+0000 7fd0bba00640 1 --2- 172.21.15.28:0/3775068867 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0b406a660 0x7fd0b41a06c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd0a400e9f0 tx=0x7fd0a400eeb0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.302+0000 7fd0bba00640 1 -- 172.21.15.28:0/3775068867 shutdown_connections 2024-04-23T14:51:45.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.302+0000 7fd0bba00640 1 --2- 172.21.15.28:0/3775068867 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd0980a3da0 0x7fd0980a6260 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.302+0000 7fd0bba00640 1 --2- 172.21.15.28:0/3775068867 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0b406b000 0x7fd0b41a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.302+0000 7fd0bba00640 1 --2- 172.21.15.28:0/3775068867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0b4108070 0x7fd0b419a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.302+0000 7fd0bba00640 1 --2- 172.21.15.28:0/3775068867 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0b406a660 0x7fd0b41a06c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.302+0000 7fd0bba00640 1 -- 172.21.15.28:0/3775068867 >> 172.21.15.28:0/3775068867 conn(0x7fd0b4077010 msgr2=0x7fd0b40ffa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:45.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.302+0000 7fd0bba00640 1 -- 172.21.15.28:0/3775068867 shutdown_connections 2024-04-23T14:51:45.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.302+0000 7fd0bba00640 1 -- 172.21.15.28:0/3775068867 wait complete. 2024-04-23T14:51:45.322 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-23T14:51:45.507 INFO:tasks.workunit.client.0.smithi028.stdout: 1 639262 9.13 MB/sec execute 241 sec latency 18.295 ms 2024-04-23T14:51:45.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.545+0000 7f7f33e00640 1 -- 172.21.15.28:0/1847837940 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f34102ac0 msgr2=0x7f7f34106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:45.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.545+0000 7f7f33e00640 1 --2- 172.21.15.28:0/1847837940 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f34102ac0 0x7f7f34106b00 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7f20009900 tx=0x7f7f2002a3f0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.545+0000 7f7f33e00640 1 -- 172.21.15.28:0/1847837940 shutdown_connections 2024-04-23T14:51:45.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.545+0000 7f7f33e00640 1 --2- 172.21.15.28:0/1847837940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f341071e0 0x7f7f3410edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.545+0000 7f7f33e00640 1 --2- 172.21.15.28:0/1847837940 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f34102ac0 0x7f7f34106b00 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.545+0000 7f7f33e00640 1 --2- 172.21.15.28:0/1847837940 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f34102110 0x7f7f341024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.545+0000 7f7f33e00640 1 -- 172.21.15.28:0/1847837940 >> 172.21.15.28:0/1847837940 conn(0x7f7f340fd9d0 msgr2=0x7f7f340ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:45.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.545+0000 7f7f33e00640 1 -- 172.21.15.28:0/1847837940 shutdown_connections 2024-04-23T14:51:45.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.545+0000 7f7f33e00640 1 -- 172.21.15.28:0/1847837940 wait complete. 2024-04-23T14:51:45.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.546+0000 7f7f33e00640 1 Processor -- start 2024-04-23T14:51:45.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.546+0000 7f7f33e00640 1 -- start start 2024-04-23T14:51:45.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.547+0000 7f7f33e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f34102110 0x7f7f3419e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:45.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.547+0000 7f7f33e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f341071e0 0x7f7f3419e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:45.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.547+0000 7f7f33e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f34198620 0x7f7f34198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:45.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.547+0000 7f7f33e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7f341122c0 con 0x7f7f341071e0 2024-04-23T14:51:45.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.547+0000 7f7f33e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7f34112140 con 0x7f7f34198620 2024-04-23T14:51:45.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.547+0000 7f7f33e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7f34112440 con 0x7f7f34102110 2024-04-23T14:51:45.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.547+0000 7f7f32a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f34102110 0x7f7f3419e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:45.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.547+0000 7f7f32a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f34102110 0x7f7f3419e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:59652/0 (socket says 172.21.15.28:59652) 2024-04-23T14:51:45.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.547+0000 7f7f32a00640 1 -- 172.21.15.28:0/3091156611 learned_addr learned my addr 172.21.15.28:0/3091156611 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:45.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.547+0000 7f7f33400640 1 --2- 172.21.15.28:0/3091156611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f34198620 0x7f7f34198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:45.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.547+0000 7f7f32000640 1 --2- 172.21.15.28:0/3091156611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f341071e0 0x7f7f3419e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:45.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.547+0000 7f7f32a00640 1 -- 172.21.15.28:0/3091156611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f34198620 msgr2=0x7f7f34198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:45.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.547+0000 7f7f32a00640 1 --2- 172.21.15.28:0/3091156611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f34198620 0x7f7f34198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.547+0000 7f7f32a00640 1 -- 172.21.15.28:0/3091156611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f341071e0 msgr2=0x7f7f3419e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:45.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.547+0000 7f7f32a00640 1 --2- 172.21.15.28:0/3091156611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f341071e0 0x7f7f3419e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.547+0000 7f7f32a00640 1 -- 172.21.15.28:0/3091156611 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f34199140 con 0x7f7f34102110 2024-04-23T14:51:45.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.547+0000 7f7f32000640 1 --2- 172.21.15.28:0/3091156611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f341071e0 0x7f7f3419e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:51:45.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.548+0000 7f7f32a00640 1 --2- 172.21.15.28:0/3091156611 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f34102110 0x7f7f3419e470 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7f2400e9f0 tx=0x7f7f2400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:45.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.548+0000 7f7f2b400640 1 -- 172.21.15.28:0/3091156611 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f2400cd80 con 0x7f7f34102110 2024-04-23T14:51:45.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.548+0000 7f7f2b400640 1 -- 172.21.15.28:0/3091156611 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f24005500 con 0x7f7f34102110 2024-04-23T14:51:45.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.548+0000 7f7f2b400640 1 -- 172.21.15.28:0/3091156611 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f24010690 con 0x7f7f34102110 2024-04-23T14:51:45.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.548+0000 7f7f33e00640 1 -- 172.21.15.28:0/3091156611 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f34199430 con 0x7f7f34102110 2024-04-23T14:51:45.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.548+0000 7f7f33e00640 1 -- 172.21.15.28:0/3091156611 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f341a5680 con 0x7f7f34102110 2024-04-23T14:51:45.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.552+0000 7f7f3ca32740 1 Processor -- start 2024-04-23T14:51:45.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.552+0000 7f7f3ca32740 1 -- start start 2024-04-23T14:51:45.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.552+0000 7f7f2b400640 1 -- 172.21.15.28:0/3091156611 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f7f240046c0 con 0x7f7f34102110 2024-04-23T14:51:45.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.552+0000 7f7f3ca32740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559190d70b40 0x559190d70f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:45.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.552+0000 7f7f3ca32740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559190d71460 0x559190d7b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:45.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.552+0000 7f7f3ca32740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559190d7bb30 0x559190d7df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:45.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.552+0000 7f7f3ca32740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559190cd61d0 con 0x559190d71460 2024-04-23T14:51:45.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.552+0000 7f7f3ca32740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559190c7c450 con 0x559190d70b40 2024-04-23T14:51:45.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.552+0000 7f7f3ca32740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559190cd2190 con 0x559190d7bb30 2024-04-23T14:51:45.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.552+0000 7f7f33400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559190d70b40 0x559190d70f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:45.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.552+0000 7f7f33400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559190d70b40 0x559190d70f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:45564/0 (socket says 172.21.15.28:45564) 2024-04-23T14:51:45.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.552+0000 7f7f33400640 1 -- 172.21.15.28:0/3788766006 learned_addr learned my addr 172.21.15.28:0/3788766006 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:45.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.553+0000 7f7f32a00640 1 --2- 172.21.15.28:0/3788766006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559190d71460 0x559190d7b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:45.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.553+0000 7f7f32000640 1 --2- 172.21.15.28:0/3788766006 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559190d7bb30 0x559190d7df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:45.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.553+0000 7f7f33400640 1 -- 172.21.15.28:0/3788766006 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559190d7bb30 msgr2=0x559190d7df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:45.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.553+0000 7f7f33400640 1 --2- 172.21.15.28:0/3788766006 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559190d7bb30 0x559190d7df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.553+0000 7f7f33400640 1 -- 172.21.15.28:0/3788766006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559190d71460 msgr2=0x559190d7b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:45.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.553+0000 7f7f33400640 1 --2- 172.21.15.28:0/3788766006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559190d71460 0x559190d7b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.553+0000 7f7f33400640 1 -- 172.21.15.28:0/3788766006 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559190d39310 con 0x559190d70b40 2024-04-23T14:51:45.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.553+0000 7f7f33400640 1 --2- 172.21.15.28:0/3788766006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559190d70b40 0x559190d70f20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c009f90 tx=0x7f7f2c010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c3f74ba83148238 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:45.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.554+0000 7f7f1fe00640 1 -- 172.21.15.28:0/3788766006 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c011700 con 0x559190d70b40 2024-04-23T14:51:45.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.554+0000 7f7f1fe00640 1 -- 172.21.15.28:0/3788766006 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f2c011d80 con 0x559190d70b40 2024-04-23T14:51:45.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.554+0000 7f7f1fe00640 1 -- 172.21.15.28:0/3788766006 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c018380 con 0x559190d70b40 2024-04-23T14:51:45.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.554+0000 7f7f2b400640 1 --2- 172.21.15.28:0/3091156611 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7f180a3da0 0x7f7f180a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:45.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.554+0000 7f7f2b400640 1 -- 172.21.15.28:0/3091156611 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8542+0+0 (secure 0 0 0) 0x7f7f240d0cb0 con 0x7f7f34102110 2024-04-23T14:51:45.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.554+0000 7f7f32a00640 1 --2- 172.21.15.28:0/3091156611 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7f180a3da0 0x7f7f180a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:45.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.554+0000 7f7f3ca32740 1 -- 172.21.15.28:0/3788766006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559190d70b40 msgr2=0x559190d70f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:45.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.554+0000 7f7f3ca32740 1 --2- 172.21.15.28:0/3788766006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559190d70b40 0x559190d70f20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c009f90 tx=0x7f7f2c010860 comp rx=0 tx=0).stop 2024-04-23T14:51:45.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.555+0000 7f7f3ca32740 1 -- 172.21.15.28:0/3788766006 shutdown_connections 2024-04-23T14:51:45.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.555+0000 7f7f3ca32740 1 --2- 172.21.15.28:0/3788766006 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559190d7bb30 0x559190d7df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.555+0000 7f7f3ca32740 1 --2- 172.21.15.28:0/3788766006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559190d71460 0x559190d7b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.555+0000 7f7f3ca32740 1 --2- 172.21.15.28:0/3788766006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559190d70b40 0x559190d70f20 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.555+0000 7f7f3ca32740 1 -- 172.21.15.28:0/3788766006 >> 172.21.15.28:0/3788766006 conn(0x559190d6ce60 msgr2=0x559190d6d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:45.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.555+0000 7f7f3ca32740 1 -- 172.21.15.28:0/3788766006 shutdown_connections 2024-04-23T14:51:45.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.555+0000 7f7f3ca32740 1 -- 172.21.15.28:0/3788766006 wait complete. 2024-04-23T14:51:45.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.555+0000 7f7f32a00640 1 --2- 172.21.15.28:0/3091156611 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7f180a3da0 0x7f7f180a6260 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x559190d72100 tx=0x7f7f240d4760 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:45.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.555+0000 7f7f3ca32740 1 -- start start 2024-04-23T14:51:45.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.556+0000 7f7f3ca32740 1 Processor -- start 2024-04-23T14:51:45.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.556+0000 7f7f3ca32740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559190d70b40 0x559190d7b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:45.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.556+0000 7f7f3ca32740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559190d71460 0x559190d788b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:45.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.556+0000 7f7f3ca32740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559190d7bb30 0x559190d78fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:45.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.556+0000 7f7f3ca32740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559190d7e9c0 con 0x559190d70b40 2024-04-23T14:51:45.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.556+0000 7f7f3ca32740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559190c8f810 con 0x559190d71460 2024-04-23T14:51:45.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.556+0000 7f7f3ca32740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559190d7eb40 con 0x559190d7bb30 2024-04-23T14:51:45.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.556+0000 7f7f3ca32740 10 client.0 _ll_register_callbacks cb 0x559190c5d140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:51:45.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.556+0000 7f7f32000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559190d71460 0x559190d788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:45.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.556+0000 7f7f32000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559190d71460 0x559190d788b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:45572/0 (socket says 172.21.15.28:45572) 2024-04-23T14:51:45.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.556+0000 7f7f33400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559190d70b40 0x559190d7b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:45.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.556+0000 7f7f32000640 1 -- 172.21.15.28:0/850364742 learned_addr learned my addr 172.21.15.28:0/850364742 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:45.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.557+0000 7f7f33400640 1 --2- 172.21.15.28:0/850364742 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559190d7bb30 0x559190d78fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:45.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.557+0000 7f7f32000640 1 -- 172.21.15.28:0/850364742 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559190d7bb30 msgr2=0x559190d78fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:45.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.557+0000 7f7f32000640 1 --2- 172.21.15.28:0/850364742 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559190d7bb30 0x559190d78fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.557+0000 7f7f32000640 1 -- 172.21.15.28:0/850364742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559190d70b40 msgr2=0x559190d7b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:45.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.557+0000 7f7f32000640 1 --2- 172.21.15.28:0/850364742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559190d70b40 0x559190d7b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.557+0000 7f7f32000640 1 -- 172.21.15.28:0/850364742 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559190ca03f0 con 0x559190d71460 2024-04-23T14:51:45.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.557+0000 7f7f33400640 1 --2- 172.21.15.28:0/850364742 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559190d7bb30 0x559190d78fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:51:45.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.557+0000 7f7f32000640 1 --2- 172.21.15.28:0/850364742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559190d71460 0x559190d788b0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7f20005c40 tx=0x7f7f20005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe5db2d3799bc2b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:45.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.557+0000 7f7f1cc00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:51:45.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.557+0000 7f7f1cc00640 1 -- 172.21.15.28:0/850364742 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f20035040 con 0x559190d71460 2024-04-23T14:51:45.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.557+0000 7f7f3ca32740 1 -- 172.21.15.28:0/850364742 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f2c011d80 con 0x559190d71460 2024-04-23T14:51:45.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.557+0000 7f7f1cc00640 1 -- 172.21.15.28:0/850364742 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8542+0+0 (secure 0 0 0) 0x7f7f200048c0 con 0x559190d71460 2024-04-23T14:51:45.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.558+0000 7f7f1cc00640 1 -- 172.21.15.28:0/850364742 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f7f20005d50 con 0x559190d71460 2024-04-23T14:51:45.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.558+0000 7f7f3ca32740 10 client.26402 fetch_fsmap learned FSMap version 20 2024-04-23T14:51:45.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.558+0000 7f7f3ca32740 1 -- 172.21.15.28:0/850364742 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5591909e83f0 con 0x559190d71460 2024-04-23T14:51:45.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.558+0000 7f7f1cc00640 1 -- 172.21.15.28:0/850364742 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f2c011d80 con 0x559190d71460 2024-04-23T14:51:45.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.558+0000 7f7f1cc00640 1 -- 172.21.15.28:0/850364742 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f7f200315b0 con 0x559190d71460 2024-04-23T14:51:45.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.558+0000 7f7f3ca32740 10 client.26402 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:51:45.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.558+0000 7f7f3ca32740 10 client.26402 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:51:45.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.558+0000 7f7f3ca32740 20 client.26402 populate_metadata read hostname 'smithi028' 2024-04-23T14:51:45.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.558+0000 7f7f3ca32740 1 --2- 172.21.15.28:0/850364742 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x559190d8bce0 0x559190d8e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:45.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.558+0000 7f7f3ca32740 4 client.26402 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:51:45.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.558+0000 7f7f3ca32740 1 -- 172.21.15.28:0/850364742 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559190ce3bf0 con 0x559190d8bce0 2024-04-23T14:51:45.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.559+0000 7f7f33400640 1 --2- 172.21.15.28:0/850364742 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x559190d8bce0 0x559190d8e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:45.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.559+0000 7f7f33400640 1 --2- 172.21.15.28:0/850364742 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x559190d8bce0 0x559190d8e0d0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60951b02b7edc189 server_cookie=2a139b30b2155bfa in_seq=0 out_seq=0 2024-04-23T14:51:45.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.559+0000 7f7f1cc00640 10 client.26402 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:51:45.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.560+0000 7f7f1cc00640 1 -- 172.21.15.28:0/850364742 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559190ce3bf0 con 0x559190d8bce0 2024-04-23T14:51:45.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.560+0000 7f7f1cc00640 10 client.26402 handle_command_reply: tid=0 2024-04-23T14:51:45.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.560+0000 7f7f3ca32740 1 client.26402 shutdown 2024-04-23T14:51:45.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.560+0000 7f7f3ca32740 1 -- 172.21.15.28:0/850364742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559190d71460 msgr2=0x559190d788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:45.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.560+0000 7f7f3ca32740 1 --2- 172.21.15.28:0/850364742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559190d71460 0x559190d788b0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7f20005c40 tx=0x7f7f20005270 comp rx=0 tx=0).stop 2024-04-23T14:51:45.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.560+0000 7f7f3ca32740 1 -- 172.21.15.28:0/850364742 shutdown_connections 2024-04-23T14:51:45.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.560+0000 7f7f3ca32740 1 --2- 172.21.15.28:0/850364742 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x559190d8bce0 0x559190d8e0d0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.560+0000 7f7f3ca32740 1 --2- 172.21.15.28:0/850364742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559190d71460 0x559190d788b0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.560+0000 7f7f3ca32740 1 --2- 172.21.15.28:0/850364742 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559190d7bb30 0x559190d78fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.560+0000 7f7f3ca32740 1 --2- 172.21.15.28:0/850364742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559190d70b40 0x559190d7b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.560+0000 7f7f3ca32740 1 -- 172.21.15.28:0/850364742 >> 172.21.15.28:0/850364742 conn(0x559190d6ce60 msgr2=0x559190d72c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:45.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.560+0000 7f7f1cc00640 0 client.26402 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:51:45.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.561+0000 7f7f3ca32740 1 -- 172.21.15.28:0/850364742 shutdown_connections 2024-04-23T14:51:45.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.561+0000 7f7f3ca32740 1 -- 172.21.15.28:0/850364742 wait complete. 2024-04-23T14:51:45.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.561+0000 7f7f3ca32740 20 client.26402 trim_cache size 0 max 16384 2024-04-23T14:51:45.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.578+0000 7f7f3ca32740 1 Processor -- start 2024-04-23T14:51:45.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.578+0000 7f7f3ca32740 1 -- start start 2024-04-23T14:51:45.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.578+0000 7f7f3ca32740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559190d70b40 0x559190d87880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:45.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.578+0000 7f7f3ca32740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559190d71460 0x559190d7b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:45.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.578+0000 7f7f3ca32740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559190d7bb30 0x559190d939a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:45.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.578+0000 7f7f3ca32740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7f341abbd0 con 0x559190d7bb30 2024-04-23T14:51:45.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.578+0000 7f7f3ca32740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7f341ae430 con 0x559190d71460 2024-04-23T14:51:45.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.578+0000 7f7f3ca32740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7f341ac220 con 0x559190d70b40 2024-04-23T14:51:45.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.578+0000 7f7f32000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559190d71460 0x559190d7b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:45.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.578+0000 7f7f32000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559190d71460 0x559190d7b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:45576/0 (socket says 172.21.15.28:45576) 2024-04-23T14:51:45.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.578+0000 7f7f32000640 1 -- 172.21.15.28:0/705608090 learned_addr learned my addr 172.21.15.28:0/705608090 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:45.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.579+0000 7f7f33400640 1 --2- 172.21.15.28:0/705608090 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559190d70b40 0x559190d87880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:45.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.579+0000 7f7f33400640 1 --2- 172.21.15.28:0/705608090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559190d7bb30 0x559190d939a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:45.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.579+0000 7f7f32000640 1 -- 172.21.15.28:0/705608090 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559190d70b40 msgr2=0x559190d87880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:45.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.579+0000 7f7f32000640 1 --2- 172.21.15.28:0/705608090 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559190d70b40 0x559190d87880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.579+0000 7f7f32000640 1 -- 172.21.15.28:0/705608090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559190d7bb30 msgr2=0x559190d939a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:45.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.579+0000 7f7f32000640 1 --2- 172.21.15.28:0/705608090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559190d7bb30 0x559190d939a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.579+0000 7f7f32000640 1 -- 172.21.15.28:0/705608090 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559190ab27d0 con 0x559190d71460 2024-04-23T14:51:45.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.579+0000 7f7f33400640 1 --2- 172.21.15.28:0/705608090 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559190d70b40 0x559190d87880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:51:45.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.579+0000 7f7f32000640 1 --2- 172.21.15.28:0/705608090 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559190d71460 0x559190d7b0f0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7f20031550 tx=0x7f7f20000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6848a1ce4fcce2c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:45.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.579+0000 7f7f1f400640 1 -- 172.21.15.28:0/705608090 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f20052070 con 0x559190d71460 2024-04-23T14:51:45.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.579+0000 7f7f1f400640 1 -- 172.21.15.28:0/705608090 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f20040070 con 0x559190d71460 2024-04-23T14:51:45.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.580+0000 7f7f1f400640 1 -- 172.21.15.28:0/705608090 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f20035040 con 0x559190d71460 2024-04-23T14:51:45.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.580+0000 7f7f3ca32740 1 -- 172.21.15.28:0/705608090 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559190d71460 msgr2=0x559190d7b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:45.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.580+0000 7f7f3ca32740 1 --2- 172.21.15.28:0/705608090 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559190d71460 0x559190d7b0f0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7f20031550 tx=0x7f7f20000bc0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.580+0000 7f7f3ca32740 1 -- 172.21.15.28:0/705608090 shutdown_connections 2024-04-23T14:51:45.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.580+0000 7f7f3ca32740 1 --2- 172.21.15.28:0/705608090 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559190d71460 0x559190d7b0f0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.580+0000 7f7f3ca32740 1 --2- 172.21.15.28:0/705608090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559190d7bb30 0x559190d939a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.580+0000 7f7f3ca32740 1 --2- 172.21.15.28:0/705608090 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559190d70b40 0x559190d87880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.581+0000 7f7f3ca32740 1 -- 172.21.15.28:0/705608090 >> 172.21.15.28:0/705608090 conn(0x559190d6ce60 msgr2=0x559190d73b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:45.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.581+0000 7f7f3ca32740 1 -- 172.21.15.28:0/705608090 shutdown_connections 2024-04-23T14:51:45.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.581+0000 7f7f3ca32740 1 -- 172.21.15.28:0/705608090 wait complete. 2024-04-23T14:51:45.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.581+0000 7f7f3ca32740 1 -- start start 2024-04-23T14:51:45.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.581+0000 7f7f3ca32740 1 Processor -- start 2024-04-23T14:51:45.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.582+0000 7f7f3ca32740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559190d70b40 0x559190d7c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:45.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.582+0000 7f7f3ca32740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559190d71460 0x559190d7ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:45.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.582+0000 7f7f3ca32740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559190d7bb30 0x559190d7d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:45.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.582+0000 7f7f3ca32740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7f341ac8f0 con 0x559190d70b40 2024-04-23T14:51:45.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.582+0000 7f7f3ca32740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7f341add60 con 0x559190d71460 2024-04-23T14:51:45.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.582+0000 7f7f3ca32740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7f341acfc0 con 0x559190d7bb30 2024-04-23T14:51:45.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.582+0000 7f7f3ca32740 10 client.0 _ll_register_callbacks cb 0x7f7f20009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:51:45.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.582+0000 7f7f32000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559190d71460 0x559190d7ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:45.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.582+0000 7f7f32000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559190d71460 0x559190d7ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:45582/0 (socket says 172.21.15.28:45582) 2024-04-23T14:51:45.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.582+0000 7f7f33400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559190d70b40 0x559190d7c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:45.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.582+0000 7f7f32000640 1 -- 172.21.15.28:0/1453119212 learned_addr learned my addr 172.21.15.28:0/1453119212 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:45.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.583+0000 7f7f32000640 1 -- 172.21.15.28:0/1453119212 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559190d7bb30 msgr2=0x559190d7d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:45.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.583+0000 7f7f32000640 1 --2- 172.21.15.28:0/1453119212 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559190d7bb30 0x559190d7d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.583+0000 7f7f32000640 1 -- 172.21.15.28:0/1453119212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559190d70b40 msgr2=0x559190d7c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:45.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.583+0000 7f7f32000640 1 --2- 172.21.15.28:0/1453119212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559190d70b40 0x559190d7c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.583+0000 7f7f32000640 1 -- 172.21.15.28:0/1453119212 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559190d81ed0 con 0x559190d71460 2024-04-23T14:51:45.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.583+0000 7f7f32000640 1 --2- 172.21.15.28:0/1453119212 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559190d71460 0x559190d7ce20 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f7f20009900 tx=0x7f7f20032ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=570ec82a3ab4fbaf server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:45.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.583+0000 7f7f1cc00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:51:45.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.583+0000 7f7f1cc00640 1 -- 172.21.15.28:0/1453119212 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f20035070 con 0x559190d71460 2024-04-23T14:51:45.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.583+0000 7f7f1cc00640 1 -- 172.21.15.28:0/1453119212 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8542+0+0 (secure 0 0 0) 0x7f7f20003800 con 0x559190d71460 2024-04-23T14:51:45.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.583+0000 7f7f3ca32740 1 -- 172.21.15.28:0/1453119212 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f20040070 con 0x559190d71460 2024-04-23T14:51:45.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.583+0000 7f7f1cc00640 1 -- 172.21.15.28:0/1453119212 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f7f20065050 con 0x559190d71460 2024-04-23T14:51:45.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.583+0000 7f7f3ca32740 10 client.26408 fetch_fsmap learned FSMap version 20 2024-04-23T14:51:45.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.583+0000 7f7f3ca32740 1 -- 172.21.15.28:0/1453119212 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x559190df6880 con 0x559190d71460 2024-04-23T14:51:45.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.584+0000 7f7f1cc00640 1 -- 172.21.15.28:0/1453119212 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f20040070 con 0x559190d71460 2024-04-23T14:51:45.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.584+0000 7f7f1cc00640 1 -- 172.21.15.28:0/1453119212 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f7f20045470 con 0x559190d71460 2024-04-23T14:51:45.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.584+0000 7f7f3ca32740 10 client.26408 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:51:45.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.584+0000 7f7f3ca32740 10 client.26408 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:51:45.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.584+0000 7f7f3ca32740 20 client.26408 populate_metadata read hostname 'smithi028' 2024-04-23T14:51:45.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.584+0000 7f7f3ca32740 1 --2- 172.21.15.28:0/1453119212 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x559190df6fb0 0x559190df93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:45.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.584+0000 7f7f3ca32740 4 client.26408 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-23T14:51:45.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.584+0000 7f7f3ca32740 1 -- 172.21.15.28:0/1453119212 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559190cdda60 con 0x559190df6fb0 2024-04-23T14:51:45.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.584+0000 7f7f33400640 1 --2- 172.21.15.28:0/1453119212 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x559190df6fb0 0x559190df93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:45.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.585+0000 7f7f33400640 1 --2- 172.21.15.28:0/1453119212 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x559190df6fb0 0x559190df93d0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70c327dff5d2e2a9 server_cookie=f711642cd14dcb93 in_seq=0 out_seq=0 2024-04-23T14:51:45.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.585+0000 7f7f1cc00640 10 client.26408 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:51:45.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.585+0000 7f7f1cc00640 1 -- 172.21.15.28:0/1453119212 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559190cdda60 con 0x559190df6fb0 2024-04-23T14:51:45.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.585+0000 7f7f1cc00640 10 client.26408 handle_command_reply: tid=0 2024-04-23T14:51:45.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.585+0000 7f7f3ca32740 1 client.26408 shutdown 2024-04-23T14:51:45.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.585+0000 7f7f3ca32740 1 -- 172.21.15.28:0/1453119212 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559190d71460 msgr2=0x559190d7ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:45.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.585+0000 7f7f3ca32740 1 --2- 172.21.15.28:0/1453119212 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559190d71460 0x559190d7ce20 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f7f20009900 tx=0x7f7f20032ea0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.586+0000 7f7f3ca32740 1 -- 172.21.15.28:0/1453119212 shutdown_connections 2024-04-23T14:51:45.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.586+0000 7f7f3ca32740 1 --2- 172.21.15.28:0/1453119212 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x559190df6fb0 0x559190df93d0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.586+0000 7f7f3ca32740 1 --2- 172.21.15.28:0/1453119212 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559190d71460 0x559190d7ce20 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.586+0000 7f7f3ca32740 1 --2- 172.21.15.28:0/1453119212 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559190d7bb30 0x559190d7d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.586+0000 7f7f1cc00640 0 client.26408 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:51:45.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.586+0000 7f7f3ca32740 1 --2- 172.21.15.28:0/1453119212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559190d70b40 0x559190d7c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.586+0000 7f7f3ca32740 1 -- 172.21.15.28:0/1453119212 >> 172.21.15.28:0/1453119212 conn(0x559190d6ce60 msgr2=0x559190d6b6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:45.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.586+0000 7f7f3ca32740 1 -- 172.21.15.28:0/1453119212 shutdown_connections 2024-04-23T14:51:45.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.586+0000 7f7f3ca32740 1 -- 172.21.15.28:0/1453119212 wait complete. 2024-04-23T14:51:45.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.586+0000 7f7f3ca32740 20 client.26408 trim_cache size 0 max 16384 2024-04-23T14:51:45.590 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-23T14:51:45.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.587+0000 7f7f33e00640 1 -- 172.21.15.28:0/3091156611 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7f180a3da0 msgr2=0x7f7f180a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:45.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.587+0000 7f7f33e00640 1 --2- 172.21.15.28:0/3091156611 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7f180a3da0 0x7f7f180a6260 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x559190d72100 tx=0x7f7f240d4760 comp rx=0 tx=0).stop 2024-04-23T14:51:45.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.587+0000 7f7f33e00640 1 -- 172.21.15.28:0/3091156611 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f34102110 msgr2=0x7f7f3419e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:45.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.587+0000 7f7f33e00640 1 --2- 172.21.15.28:0/3091156611 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f34102110 0x7f7f3419e470 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7f2400e9f0 tx=0x7f7f2400edf0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.587+0000 7f7f33e00640 1 -- 172.21.15.28:0/3091156611 shutdown_connections 2024-04-23T14:51:45.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.587+0000 7f7f33e00640 1 --2- 172.21.15.28:0/3091156611 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7f180a3da0 0x7f7f180a6260 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.587+0000 7f7f33e00640 1 --2- 172.21.15.28:0/3091156611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f34198620 0x7f7f34198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.587+0000 7f7f33e00640 1 --2- 172.21.15.28:0/3091156611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f341071e0 0x7f7f3419e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.587+0000 7f7f33e00640 1 --2- 172.21.15.28:0/3091156611 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f34102110 0x7f7f3419e470 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.587+0000 7f7f33e00640 1 -- 172.21.15.28:0/3091156611 >> 172.21.15.28:0/3091156611 conn(0x7f7f340fd9d0 msgr2=0x7f7f340ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.587+0000 7f7f33e00640 1 -- 172.21.15.28:0/3091156611 shutdown_connections 2024-04-23T14:51:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:45.587+0000 7f7f33e00640 1 -- 172.21.15.28:0/3091156611 wait complete. 2024-04-23T14:51:45.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:45 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3775068867' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:51:46.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:45 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3775068867' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:51:46.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:45 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3775068867' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:51:46.507 INFO:tasks.workunit.client.0.smithi028.stdout: 1 640444 9.11 MB/sec execute 242 sec latency 19.555 ms 2024-04-23T14:51:46.609 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-23T14:51:46.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:51:46.609 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-23T14:51:46.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.835+0000 7f4368a00640 1 -- 172.21.15.28:0/2206111997 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4364104320 msgr2=0x7f4364104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:46.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.835+0000 7f4368a00640 1 --2- 172.21.15.28:0/2206111997 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4364104320 0x7f4364104700 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f4354009900 tx=0x7f435402a410 comp rx=0 tx=0).stop 2024-04-23T14:51:46.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.835+0000 7f4368a00640 1 -- 172.21.15.28:0/2206111997 shutdown_connections 2024-04-23T14:51:46.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.835+0000 7f4368a00640 1 --2- 172.21.15.28:0/2206111997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43641093f0 0x7f4364110fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:46.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.835+0000 7f4368a00640 1 --2- 172.21.15.28:0/2206111997 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4364104cd0 0x7f4364108d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:46.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.835+0000 7f4368a00640 1 --2- 172.21.15.28:0/2206111997 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4364104320 0x7f4364104700 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:46.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.835+0000 7f4368a00640 1 -- 172.21.15.28:0/2206111997 >> 172.21.15.28:0/2206111997 conn(0x7f43640ffb40 msgr2=0x7f4364101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:46.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.835+0000 7f4368a00640 1 -- 172.21.15.28:0/2206111997 shutdown_connections 2024-04-23T14:51:46.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.835+0000 7f4368a00640 1 -- 172.21.15.28:0/2206111997 wait complete. 2024-04-23T14:51:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.836+0000 7f4368a00640 1 Processor -- start 2024-04-23T14:51:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.836+0000 7f4368a00640 1 -- start start 2024-04-23T14:51:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.837+0000 7f4368a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4364104320 0x7f43641a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.837+0000 7f4368a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f43641093f0 0x7f43641a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.837+0000 7f4368a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f436419a810 0x7f436419acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.837+0000 7f4368a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f43641144f0 con 0x7f436419a810 2024-04-23T14:51:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.837+0000 7f4368a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4364114370 con 0x7f4364104320 2024-04-23T14:51:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.837+0000 7f4368a00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4364114670 con 0x7f43641093f0 2024-04-23T14:51:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.837+0000 7f4363400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4364104320 0x7f43641a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.837+0000 7f4363400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4364104320 0x7f43641a06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:45608/0 (socket says 172.21.15.28:45608) 2024-04-23T14:51:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.837+0000 7f4363400640 1 -- 172.21.15.28:0/2583625001 learned_addr learned my addr 172.21.15.28:0/2583625001 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.837+0000 7f4363e00640 1 --2- 172.21.15.28:0/2583625001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f436419a810 0x7f436419acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.837+0000 7f4362a00640 1 --2- 172.21.15.28:0/2583625001 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f43641093f0 0x7f43641a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.837+0000 7f4363400640 1 -- 172.21.15.28:0/2583625001 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f43641093f0 msgr2=0x7f43641a0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.837+0000 7f4363400640 1 --2- 172.21.15.28:0/2583625001 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f43641093f0 0x7f43641a0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.837+0000 7f4363400640 1 -- 172.21.15.28:0/2583625001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f436419a810 msgr2=0x7f436419acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:46.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.837+0000 7f4363400640 1 --2- 172.21.15.28:0/2583625001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f436419a810 0x7f436419acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:46.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.837+0000 7f4363400640 1 -- 172.21.15.28:0/2583625001 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f436419b3a0 con 0x7f4364104320 2024-04-23T14:51:46.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.837+0000 7f4363e00640 1 --2- 172.21.15.28:0/2583625001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f436419a810 0x7f436419acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:51:46.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.837+0000 7f4362a00640 1 --2- 172.21.15.28:0/2583625001 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f43641093f0 0x7f43641a0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:51:46.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.838+0000 7f4363400640 1 --2- 172.21.15.28:0/2583625001 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4364104320 0x7f43641a06a0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f4354004770 tx=0x7f4354003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:46.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.838+0000 7f435be00640 1 -- 172.21.15.28:0/2583625001 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f435403f070 con 0x7f4364104320 2024-04-23T14:51:46.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.838+0000 7f435be00640 1 -- 172.21.15.28:0/2583625001 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4354031ca0 con 0x7f4364104320 2024-04-23T14:51:46.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.838+0000 7f4368a00640 1 -- 172.21.15.28:0/2583625001 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f436419b630 con 0x7f4364104320 2024-04-23T14:51:46.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.838+0000 7f435be00640 1 -- 172.21.15.28:0/2583625001 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4354035030 con 0x7f4364104320 2024-04-23T14:51:46.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.838+0000 7f4368a00640 1 -- 172.21.15.28:0/2583625001 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43641a7810 con 0x7f4364104320 2024-04-23T14:51:46.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.842+0000 7f435be00640 1 -- 172.21.15.28:0/2583625001 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f4354042070 con 0x7f4364104320 2024-04-23T14:51:46.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.842+0000 7f436cfb5740 1 Processor -- start 2024-04-23T14:51:46.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.842+0000 7f436cfb5740 1 -- start start 2024-04-23T14:51:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.842+0000 7f436cfb5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c5b8fa8270 0x55c5b8fa8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.842+0000 7f436cfb5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5b8fa8b90 0x55c5b8fb2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.842+0000 7f436cfb5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5b8fb3260 0x55c5b8fb5650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.842+0000 7f436cfb5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c5b8f0de70 con 0x55c5b8fa8b90 2024-04-23T14:51:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.842+0000 7f436cfb5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c5b8eb4380 con 0x55c5b8fa8270 2024-04-23T14:51:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.842+0000 7f436cfb5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c5b8f09e30 con 0x55c5b8fb3260 2024-04-23T14:51:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.843+0000 7f4363e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c5b8fa8270 0x55c5b8fa8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.843+0000 7f4363e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c5b8fa8270 0x55c5b8fa8650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:45618/0 (socket says 172.21.15.28:45618) 2024-04-23T14:51:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.843+0000 7f4363e00640 1 -- 172.21.15.28:0/3027072428 learned_addr learned my addr 172.21.15.28:0/3027072428 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.843+0000 7f4362a00640 1 --2- 172.21.15.28:0/3027072428 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5b8fb3260 0x55c5b8fb5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.843+0000 7f4363400640 1 --2- 172.21.15.28:0/3027072428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5b8fa8b90 0x55c5b8fb2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.843+0000 7f4363e00640 1 -- 172.21.15.28:0/3027072428 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5b8fb3260 msgr2=0x55c5b8fb5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:46.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.843+0000 7f4363e00640 1 --2- 172.21.15.28:0/3027072428 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5b8fb3260 0x55c5b8fb5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:46.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.843+0000 7f4363e00640 1 -- 172.21.15.28:0/3027072428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5b8fa8b90 msgr2=0x55c5b8fb2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:46.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.843+0000 7f4363e00640 1 --2- 172.21.15.28:0/3027072428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5b8fa8b90 0x55c5b8fb2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:46.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.843+0000 7f4363e00640 1 -- 172.21.15.28:0/3027072428 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5b8ce87e0 con 0x55c5b8fa8270 2024-04-23T14:51:46.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.843+0000 7f435be00640 1 --2- 172.21.15.28:0/2583625001 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f43480a3da0 0x7f43480a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:46.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.844+0000 7f4363e00640 1 --2- 172.21.15.28:0/3027072428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c5b8fa8270 0x55c5b8fa8650 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f435c009fd0 tx=0x7f435c010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d50a8024f49f64b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:46.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.844+0000 7f435be00640 1 -- 172.21.15.28:0/2583625001 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8542+0+0 (secure 0 0 0) 0x7f43540f0ef0 con 0x7f4364104320 2024-04-23T14:51:46.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.844+0000 7f4363400640 1 --2- 172.21.15.28:0/2583625001 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f43480a3da0 0x7f43480a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:46.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.844+0000 7f434fe00640 1 -- 172.21.15.28:0/3027072428 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f435c011700 con 0x55c5b8fa8270 2024-04-23T14:51:46.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.844+0000 7f434fe00640 1 -- 172.21.15.28:0/3027072428 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f435c011d80 con 0x55c5b8fa8270 2024-04-23T14:51:46.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.844+0000 7f434fe00640 1 -- 172.21.15.28:0/3027072428 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f435c00e3c0 con 0x55c5b8fa8270 2024-04-23T14:51:46.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.845+0000 7f436cfb5740 1 -- 172.21.15.28:0/3027072428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c5b8fa8270 msgr2=0x55c5b8fa8650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:46.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.845+0000 7f436cfb5740 1 --2- 172.21.15.28:0/3027072428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c5b8fa8270 0x55c5b8fa8650 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f435c009fd0 tx=0x7f435c010860 comp rx=0 tx=0).stop 2024-04-23T14:51:46.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.845+0000 7f436cfb5740 1 -- 172.21.15.28:0/3027072428 shutdown_connections 2024-04-23T14:51:46.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.845+0000 7f436cfb5740 1 --2- 172.21.15.28:0/3027072428 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5b8fb3260 0x55c5b8fb5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:46.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.845+0000 7f436cfb5740 1 --2- 172.21.15.28:0/3027072428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5b8fa8b90 0x55c5b8fb2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:46.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.845+0000 7f436cfb5740 1 --2- 172.21.15.28:0/3027072428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c5b8fa8270 0x55c5b8fa8650 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:46.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.845+0000 7f436cfb5740 1 -- 172.21.15.28:0/3027072428 >> 172.21.15.28:0/3027072428 conn(0x55c5b8fa4590 msgr2=0x55c5b8fa4970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:46.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.845+0000 7f436cfb5740 1 -- 172.21.15.28:0/3027072428 shutdown_connections 2024-04-23T14:51:46.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.845+0000 7f436cfb5740 1 -- 172.21.15.28:0/3027072428 wait complete. 2024-04-23T14:51:46.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.845+0000 7f4363400640 1 --2- 172.21.15.28:0/2583625001 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f43480a3da0 0x7f43480a6260 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x55c5b8fa9830 tx=0x7f4354009260 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:46.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.846+0000 7f436cfb5740 1 -- start start 2024-04-23T14:51:46.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.846+0000 7f436cfb5740 1 Processor -- start 2024-04-23T14:51:46.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.846+0000 7f436cfb5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5b8fa8270 0x55c5b8fb0280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:46.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.846+0000 7f436cfb5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5b8fa8b90 0x55c5b8fb07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:46.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.847+0000 7f436cfb5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c5b8fb3260 0x55c5b8fb0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:46.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.847+0000 7f436cfb5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c5b8fb6050 con 0x55c5b8fa8270 2024-04-23T14:51:46.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.847+0000 7f436cfb5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c5b8ec7660 con 0x55c5b8fb3260 2024-04-23T14:51:46.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.847+0000 7f436cfb5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c5b8fb61d0 con 0x55c5b8fa8b90 2024-04-23T14:51:46.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.847+0000 7f436cfb5740 10 client.0 _ll_register_callbacks cb 0x55c5b8eb5aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:51:46.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.847+0000 7f4362a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5b8fa8b90 0x55c5b8fb07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:46.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.847+0000 7f4363e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5b8fa8270 0x55c5b8fb0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:46.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.847+0000 7f4362a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5b8fa8b90 0x55c5b8fb07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:59734/0 (socket says 172.21.15.28:59734) 2024-04-23T14:51:46.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.847+0000 7f4362a00640 1 -- 172.21.15.28:0/1255064923 learned_addr learned my addr 172.21.15.28:0/1255064923 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:46.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.847+0000 7f4363e00640 1 --2- 172.21.15.28:0/1255064923 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c5b8fb3260 0x55c5b8fb0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:46.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.847+0000 7f4362a00640 1 -- 172.21.15.28:0/1255064923 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c5b8fb3260 msgr2=0x55c5b8fb0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:46.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.847+0000 7f4362a00640 1 --2- 172.21.15.28:0/1255064923 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c5b8fb3260 0x55c5b8fb0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:46.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.847+0000 7f4362a00640 1 -- 172.21.15.28:0/1255064923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5b8fa8270 msgr2=0x55c5b8fb0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:46.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.847+0000 7f4362a00640 1 --2- 172.21.15.28:0/1255064923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5b8fa8270 0x55c5b8fb0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:46.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.847+0000 7f4362a00640 1 -- 172.21.15.28:0/1255064923 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5b8d48f70 con 0x55c5b8fa8b90 2024-04-23T14:51:46.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.847+0000 7f4363e00640 1 --2- 172.21.15.28:0/1255064923 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c5b8fb3260 0x55c5b8fb0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:51:46.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.848+0000 7f4362a00640 1 --2- 172.21.15.28:0/1255064923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5b8fa8b90 0x55c5b8fb07c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4350013790 tx=0x7f4350013c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8be8f5d8a75a7ee server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:46.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.848+0000 7f434cc00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:51:46.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.848+0000 7f434cc00640 1 -- 172.21.15.28:0/1255064923 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4350020060 con 0x55c5b8fa8b90 2024-04-23T14:51:46.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.848+0000 7f434cc00640 1 -- 172.21.15.28:0/1255064923 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8542+0+0 (secure 0 0 0) 0x7f4350011dd0 con 0x55c5b8fa8b90 2024-04-23T14:51:46.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.848+0000 7f436cfb5740 1 -- 172.21.15.28:0/1255064923 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f435c011d80 con 0x55c5b8fa8b90 2024-04-23T14:51:46.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.848+0000 7f434cc00640 1 -- 172.21.15.28:0/1255064923 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f435000f450 con 0x55c5b8fa8b90 2024-04-23T14:51:46.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.848+0000 7f436cfb5740 10 client.26404 fetch_fsmap learned FSMap version 20 2024-04-23T14:51:46.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.848+0000 7f436cfb5740 1 -- 172.21.15.28:0/1255064923 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55c5b8be0170 con 0x55c5b8fa8b90 2024-04-23T14:51:46.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.849+0000 7f434cc00640 1 -- 172.21.15.28:0/1255064923 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f435c011d80 con 0x55c5b8fa8b90 2024-04-23T14:51:46.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.849+0000 7f434cc00640 1 -- 172.21.15.28:0/1255064923 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f435001a030 con 0x55c5b8fa8b90 2024-04-23T14:51:46.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.849+0000 7f436cfb5740 10 client.26404 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:51:46.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.849+0000 7f436cfb5740 10 client.26404 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:51:46.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.849+0000 7f436cfb5740 20 client.26404 populate_metadata read hostname 'smithi028' 2024-04-23T14:51:46.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.849+0000 7f436cfb5740 1 --2- 172.21.15.28:0/1255064923 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c5b8fc36d0 0x55c5b8fc5b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:46.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.849+0000 7f436cfb5740 4 client.26404 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:51:46.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.849+0000 7f436cfb5740 1 -- 172.21.15.28:0/1255064923 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c5b8f1b890 con 0x55c5b8fc36d0 2024-04-23T14:51:46.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.849+0000 7f4363e00640 1 --2- 172.21.15.28:0/1255064923 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c5b8fc36d0 0x55c5b8fc5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:46.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.850+0000 7f4363e00640 1 --2- 172.21.15.28:0/1255064923 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c5b8fc36d0 0x55c5b8fc5b20 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6302706a5e3c6a71 server_cookie=67d8f07207c189b8 in_seq=0 out_seq=0 2024-04-23T14:51:46.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.850+0000 7f434cc00640 10 client.26404 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:51:46.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.850+0000 7f434cc00640 1 -- 172.21.15.28:0/1255064923 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c5b8f1b890 con 0x55c5b8fc36d0 2024-04-23T14:51:46.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.850+0000 7f434cc00640 10 client.26404 handle_command_reply: tid=0 2024-04-23T14:51:46.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.851+0000 7f436cfb5740 1 client.26404 shutdown 2024-04-23T14:51:46.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.851+0000 7f436cfb5740 1 -- 172.21.15.28:0/1255064923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5b8fa8b90 msgr2=0x55c5b8fb07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:46.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.851+0000 7f436cfb5740 1 --2- 172.21.15.28:0/1255064923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5b8fa8b90 0x55c5b8fb07c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4350013790 tx=0x7f4350013c50 comp rx=0 tx=0).stop 2024-04-23T14:51:46.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.851+0000 7f436cfb5740 1 -- 172.21.15.28:0/1255064923 shutdown_connections 2024-04-23T14:51:46.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.851+0000 7f436cfb5740 1 --2- 172.21.15.28:0/1255064923 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c5b8fc36d0 0x55c5b8fc5b20 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:46.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.851+0000 7f436cfb5740 1 --2- 172.21.15.28:0/1255064923 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c5b8fb3260 0x55c5b8fb0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:46.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.851+0000 7f436cfb5740 1 --2- 172.21.15.28:0/1255064923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5b8fa8b90 0x55c5b8fb07c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:46.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.851+0000 7f436cfb5740 1 --2- 172.21.15.28:0/1255064923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5b8fa8270 0x55c5b8fb0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:46.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.851+0000 7f434cc00640 0 client.26404 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:51:46.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.851+0000 7f436cfb5740 1 -- 172.21.15.28:0/1255064923 >> 172.21.15.28:0/1255064923 conn(0x55c5b8fa4590 msgr2=0x55c5b8faa270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:46.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.851+0000 7f436cfb5740 1 -- 172.21.15.28:0/1255064923 shutdown_connections 2024-04-23T14:51:46.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.851+0000 7f436cfb5740 1 -- 172.21.15.28:0/1255064923 wait complete. 2024-04-23T14:51:46.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.851+0000 7f436cfb5740 20 client.26404 trim_cache size 0 max 16384 2024-04-23T14:51:46.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.874+0000 7f436cfb5740 1 Processor -- start 2024-04-23T14:51:46.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.874+0000 7f436cfb5740 1 -- start start 2024-04-23T14:51:46.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.874+0000 7f436cfb5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c5b8fa8270 0x55c5b8fbef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:46.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.874+0000 7f436cfb5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5b8fa8b90 0x55c5b8faa9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:46.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.874+0000 7f436cfb5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5b8fb3260 0x55c5b8fcb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:46.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.874+0000 7f436cfb5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f43641ade40 con 0x55c5b8fb3260 2024-04-23T14:51:46.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.874+0000 7f436cfb5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f43641b06a0 con 0x55c5b8fa8270 2024-04-23T14:51:46.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.874+0000 7f436cfb5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f43641ae490 con 0x55c5b8fa8b90 2024-04-23T14:51:46.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.875+0000 7f4362a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5b8fa8b90 0x55c5b8faa9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:46.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.875+0000 7f4363e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c5b8fa8270 0x55c5b8fbef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:46.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.875+0000 7f4362a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5b8fa8b90 0x55c5b8faa9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:59750/0 (socket says 172.21.15.28:59750) 2024-04-23T14:51:46.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.875+0000 7f4362a00640 1 -- 172.21.15.28:0/4263759282 learned_addr learned my addr 172.21.15.28:0/4263759282 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:46.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.875+0000 7f4363e00640 1 --2- 172.21.15.28:0/4263759282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5b8fb3260 0x55c5b8fcb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:46.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.875+0000 7f4362a00640 1 -- 172.21.15.28:0/4263759282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c5b8fa8270 msgr2=0x55c5b8fbef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:46.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.875+0000 7f4362a00640 1 --2- 172.21.15.28:0/4263759282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c5b8fa8270 0x55c5b8fbef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:46.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.875+0000 7f4362a00640 1 -- 172.21.15.28:0/4263759282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5b8fb3260 msgr2=0x55c5b8fcb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:46.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.875+0000 7f4362a00640 1 --2- 172.21.15.28:0/4263759282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5b8fb3260 0x55c5b8fcb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:46.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.875+0000 7f4362a00640 1 -- 172.21.15.28:0/4263759282 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5b8fa4ed0 con 0x55c5b8fa8b90 2024-04-23T14:51:46.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.875+0000 7f4363e00640 1 --2- 172.21.15.28:0/4263759282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5b8fb3260 0x55c5b8fcb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:51:46.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.876+0000 7f4362a00640 1 --2- 172.21.15.28:0/4263759282 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5b8fa8b90 0x55c5b8faa9d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4350023aa0 tx=0x7f4350009da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e93d5557ce6a332c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:46.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.876+0000 7f434f400640 1 -- 172.21.15.28:0/4263759282 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4350031070 con 0x55c5b8fa8b90 2024-04-23T14:51:46.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.876+0000 7f434f400640 1 -- 172.21.15.28:0/4263759282 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4350023d20 con 0x55c5b8fa8b90 2024-04-23T14:51:46.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.876+0000 7f434f400640 1 -- 172.21.15.28:0/4263759282 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4350005530 con 0x55c5b8fa8b90 2024-04-23T14:51:46.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.877+0000 7f436cfb5740 1 -- 172.21.15.28:0/4263759282 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5b8fa8b90 msgr2=0x55c5b8faa9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:46.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.877+0000 7f436cfb5740 1 --2- 172.21.15.28:0/4263759282 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5b8fa8b90 0x55c5b8faa9d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4350023aa0 tx=0x7f4350009da0 comp rx=0 tx=0).stop 2024-04-23T14:51:46.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.877+0000 7f436cfb5740 1 -- 172.21.15.28:0/4263759282 shutdown_connections 2024-04-23T14:51:46.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.877+0000 7f436cfb5740 1 --2- 172.21.15.28:0/4263759282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5b8fb3260 0x55c5b8fcb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:46.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.877+0000 7f436cfb5740 1 --2- 172.21.15.28:0/4263759282 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5b8fa8b90 0x55c5b8faa9d0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:46.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.877+0000 7f436cfb5740 1 --2- 172.21.15.28:0/4263759282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c5b8fa8270 0x55c5b8fbef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:46.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.877+0000 7f436cfb5740 1 -- 172.21.15.28:0/4263759282 >> 172.21.15.28:0/4263759282 conn(0x55c5b8fa4590 msgr2=0x55c5b8fb1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:46.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.877+0000 7f436cfb5740 1 -- 172.21.15.28:0/4263759282 shutdown_connections 2024-04-23T14:51:46.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.877+0000 7f436cfb5740 1 -- 172.21.15.28:0/4263759282 wait complete. 2024-04-23T14:51:46.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.877+0000 7f436cfb5740 1 -- start start 2024-04-23T14:51:46.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.877+0000 7f436cfb5740 1 Processor -- start 2024-04-23T14:51:46.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.878+0000 7f436cfb5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5b8fa8270 0x55c5b8fbb610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:46.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.878+0000 7f436cfb5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5b8fa8b90 0x55c5b8fb83f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:46.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.878+0000 7f436cfb5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c5b8fb3260 0x55c5b8fb8930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:46.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.878+0000 7f436cfb5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f43641aeb60 con 0x55c5b8fa8b90 2024-04-23T14:51:46.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.878+0000 7f436cfb5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f43641affd0 con 0x55c5b8fb3260 2024-04-23T14:51:46.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.878+0000 7f436cfb5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f43641af230 con 0x55c5b8fa8270 2024-04-23T14:51:46.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.878+0000 7f436cfb5740 10 client.0 _ll_register_callbacks cb 0x7f4350009650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:51:46.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.878+0000 7f4363e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5b8fa8270 0x55c5b8fbb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:46.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.878+0000 7f4363e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5b8fa8270 0x55c5b8fbb610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:59756/0 (socket says 172.21.15.28:59756) 2024-04-23T14:51:46.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.878+0000 7f4363e00640 1 -- 172.21.15.28:0/2211184950 learned_addr learned my addr 172.21.15.28:0/2211184950 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:46.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.878+0000 7f4363e00640 1 --2- 172.21.15.28:0/2211184950 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c5b8fb3260 0x55c5b8fb8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:46.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.878+0000 7f4362a00640 1 --2- 172.21.15.28:0/2211184950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5b8fa8b90 0x55c5b8fb83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:46.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.879+0000 7f4363e00640 1 -- 172.21.15.28:0/2211184950 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c5b8fb3260 msgr2=0x55c5b8fb8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:46.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.879+0000 7f4363e00640 1 --2- 172.21.15.28:0/2211184950 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c5b8fb3260 0x55c5b8fb8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:46.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.879+0000 7f4363e00640 1 -- 172.21.15.28:0/2211184950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5b8fa8b90 msgr2=0x55c5b8fb83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:46.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.879+0000 7f4363e00640 1 --2- 172.21.15.28:0/2211184950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5b8fa8b90 0x55c5b8fb83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:46.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.879+0000 7f4363e00640 1 -- 172.21.15.28:0/2211184950 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5b8fb8e70 con 0x55c5b8fa8270 2024-04-23T14:51:46.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.879+0000 7f4363e00640 1 --2- 172.21.15.28:0/2211184950 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5b8fa8270 0x55c5b8fbb610 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f435c013010 tx=0x7f435c02d040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53241d04034acb09 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:46.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.879+0000 7f434cc00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:51:46.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.879+0000 7f434cc00640 1 -- 172.21.15.28:0/2211184950 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f435c01b520 con 0x55c5b8fa8270 2024-04-23T14:51:46.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.879+0000 7f436cfb5740 1 -- 172.21.15.28:0/2211184950 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4350023d20 con 0x55c5b8fa8270 2024-04-23T14:51:46.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.879+0000 7f434cc00640 1 -- 172.21.15.28:0/2211184950 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8542+0+0 (secure 0 0 0) 0x7f435c0049c0 con 0x55c5b8fa8270 2024-04-23T14:51:46.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.880+0000 7f434cc00640 1 -- 172.21.15.28:0/2211184950 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f435c004bc0 con 0x55c5b8fa8270 2024-04-23T14:51:46.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.880+0000 7f436cfb5740 10 client.26410 fetch_fsmap learned FSMap version 20 2024-04-23T14:51:46.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.880+0000 7f436cfb5740 1 -- 172.21.15.28:0/2211184950 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55c5b902aa00 con 0x55c5b8fa8270 2024-04-23T14:51:46.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.880+0000 7f434cc00640 1 -- 172.21.15.28:0/2211184950 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4350023d20 con 0x55c5b8fa8270 2024-04-23T14:51:46.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.880+0000 7f434cc00640 1 -- 172.21.15.28:0/2211184950 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f435c029040 con 0x55c5b8fa8270 2024-04-23T14:51:46.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.880+0000 7f436cfb5740 10 client.26410 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:51:46.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.880+0000 7f436cfb5740 10 client.26410 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:51:46.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.880+0000 7f436cfb5740 20 client.26410 populate_metadata read hostname 'smithi028' 2024-04-23T14:51:46.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.880+0000 7f436cfb5740 1 --2- 172.21.15.28:0/2211184950 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c5b902ae00 0x55c5b902d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:46.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.880+0000 7f436cfb5740 4 client.26410 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-23T14:51:46.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.880+0000 7f436cfb5740 1 -- 172.21.15.28:0/2211184950 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c5b8f15700 con 0x55c5b902ae00 2024-04-23T14:51:46.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.880+0000 7f4362a00640 1 --2- 172.21.15.28:0/2211184950 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c5b902ae00 0x55c5b902d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:46.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.881+0000 7f4362a00640 1 --2- 172.21.15.28:0/2211184950 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c5b902ae00 0x55c5b902d1f0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51dbcadb22580ede server_cookie=2a31517a6ef3eb7d in_seq=0 out_seq=0 2024-04-23T14:51:46.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.881+0000 7f434cc00640 10 client.26410 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:51:46.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.882+0000 7f434cc00640 1 -- 172.21.15.28:0/2211184950 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c5b8f15700 con 0x55c5b902ae00 2024-04-23T14:51:46.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.882+0000 7f434cc00640 10 client.26410 handle_command_reply: tid=0 2024-04-23T14:51:46.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.882+0000 7f436cfb5740 1 client.26410 shutdown 2024-04-23T14:51:46.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.882+0000 7f436cfb5740 1 -- 172.21.15.28:0/2211184950 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5b8fa8270 msgr2=0x55c5b8fbb610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:46.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.882+0000 7f436cfb5740 1 --2- 172.21.15.28:0/2211184950 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5b8fa8270 0x55c5b8fbb610 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f435c013010 tx=0x7f435c02d040 comp rx=0 tx=0).stop 2024-04-23T14:51:46.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.882+0000 7f436cfb5740 1 -- 172.21.15.28:0/2211184950 shutdown_connections 2024-04-23T14:51:46.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.882+0000 7f436cfb5740 1 --2- 172.21.15.28:0/2211184950 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c5b902ae00 0x55c5b902d1f0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:46.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.882+0000 7f436cfb5740 1 --2- 172.21.15.28:0/2211184950 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c5b8fb3260 0x55c5b8fb8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:46.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.882+0000 7f436cfb5740 1 --2- 172.21.15.28:0/2211184950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c5b8fa8b90 0x55c5b8fb83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:46.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.882+0000 7f434cc00640 0 client.26410 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:51:46.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.882+0000 7f436cfb5740 1 --2- 172.21.15.28:0/2211184950 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5b8fa8270 0x55c5b8fbb610 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:46.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.882+0000 7f436cfb5740 1 -- 172.21.15.28:0/2211184950 >> 172.21.15.28:0/2211184950 conn(0x55c5b8fa4590 msgr2=0x55c5b8fb2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:46.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.882+0000 7f436cfb5740 1 -- 172.21.15.28:0/2211184950 shutdown_connections 2024-04-23T14:51:46.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.882+0000 7f436cfb5740 1 -- 172.21.15.28:0/2211184950 wait complete. 2024-04-23T14:51:46.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.882+0000 7f436cfb5740 20 client.26410 trim_cache size 0 max 16384 2024-04-23T14:51:46.887 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:51:46.887 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-23T14:51:46.887 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "c381e5c7-159e-4f54-b397-a9a32e09d526", 2024-04-23T14:51:46.887 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-23T14:51:46.887 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:51:46.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.883+0000 7f4368a00640 1 -- 172.21.15.28:0/2583625001 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f43480a3da0 msgr2=0x7f43480a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:46.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.883+0000 7f4368a00640 1 --2- 172.21.15.28:0/2583625001 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f43480a3da0 0x7f43480a6260 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x55c5b8fa9830 tx=0x7f4354009260 comp rx=0 tx=0).stop 2024-04-23T14:51:46.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.883+0000 7f4368a00640 1 -- 172.21.15.28:0/2583625001 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4364104320 msgr2=0x7f43641a06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:46.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.883+0000 7f4368a00640 1 --2- 172.21.15.28:0/2583625001 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4364104320 0x7f43641a06a0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f4354004770 tx=0x7f4354003c10 comp rx=0 tx=0).stop 2024-04-23T14:51:46.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.883+0000 7f4368a00640 1 -- 172.21.15.28:0/2583625001 shutdown_connections 2024-04-23T14:51:46.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.883+0000 7f4368a00640 1 --2- 172.21.15.28:0/2583625001 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f43480a3da0 0x7f43480a6260 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:46.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.883+0000 7f4368a00640 1 --2- 172.21.15.28:0/2583625001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f436419a810 0x7f436419acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:46.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.883+0000 7f4368a00640 1 --2- 172.21.15.28:0/2583625001 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f43641093f0 0x7f43641a0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:46.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.883+0000 7f4368a00640 1 --2- 172.21.15.28:0/2583625001 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4364104320 0x7f43641a06a0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:46.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.883+0000 7f4368a00640 1 -- 172.21.15.28:0/2583625001 >> 172.21.15.28:0/2583625001 conn(0x7f43640ffb40 msgr2=0x7f4364101ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:46.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.883+0000 7f4368a00640 1 -- 172.21.15.28:0/2583625001 shutdown_connections 2024-04-23T14:51:46.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:46.883+0000 7f4368a00640 1 -- 172.21.15.28:0/2583625001 wait complete. 2024-04-23T14:51:46.904 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:51:46.904 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:51:46.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:46 smithi028 ceph-mon[27810]: pgmap v237: 97 pgs: 97 active+clean; 424 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 129 op/s 2024-04-23T14:51:47.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:46 smithi105 ceph-mon[32555]: pgmap v237: 97 pgs: 97 active+clean; 424 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 129 op/s 2024-04-23T14:51:47.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:46 smithi038 ceph-mon[32348]: pgmap v237: 97 pgs: 97 active+clean; 424 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 129 op/s 2024-04-23T14:51:47.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.123+0000 7fa64d200640 1 -- 172.21.15.28:0/679766037 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa648104320 msgr2=0x7fa648104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:47.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.123+0000 7fa64d200640 1 --2- 172.21.15.28:0/679766037 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa648104320 0x7fa648104700 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa638009900 tx=0x7fa63802a400 comp rx=0 tx=0).stop 2024-04-23T14:51:47.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.123+0000 7fa64d200640 1 -- 172.21.15.28:0/679766037 shutdown_connections 2024-04-23T14:51:47.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.123+0000 7fa64d200640 1 --2- 172.21.15.28:0/679766037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa648104cd0 0x7fa648108d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:47.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.123+0000 7fa64d200640 1 --2- 172.21.15.28:0/679766037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6481093f0 0x7fa648110fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:47.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.123+0000 7fa64d200640 1 --2- 172.21.15.28:0/679766037 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa648104320 0x7fa648104700 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:47.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.123+0000 7fa64d200640 1 -- 172.21.15.28:0/679766037 >> 172.21.15.28:0/679766037 conn(0x7fa6480ffb40 msgr2=0x7fa648101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:47.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.124+0000 7fa64d200640 1 -- 172.21.15.28:0/679766037 shutdown_connections 2024-04-23T14:51:47.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.124+0000 7fa64d200640 1 -- 172.21.15.28:0/679766037 wait complete. 2024-04-23T14:51:47.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.125+0000 7fa64d200640 1 Processor -- start 2024-04-23T14:51:47.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.125+0000 7fa64d200640 1 -- start start 2024-04-23T14:51:47.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.125+0000 7fa64d200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa648104320 0x7fa6481a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:47.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.125+0000 7fa64d200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa6481093f0 0x7fa6481a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:47.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.125+0000 7fa64d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa64819a7d0 0x7fa64819ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:47.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.125+0000 7fa64d200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa6481144d0 con 0x7fa64819a7d0 2024-04-23T14:51:47.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.125+0000 7fa64d200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa648114350 con 0x7fa648104320 2024-04-23T14:51:47.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.125+0000 7fa64d200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa648114650 con 0x7fa6481093f0 2024-04-23T14:51:47.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.125+0000 7fa647400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa648104320 0x7fa6481a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:47.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.125+0000 7fa646a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa6481093f0 0x7fa6481a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:47.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.125+0000 7fa647400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa648104320 0x7fa6481a0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:45656/0 (socket says 172.21.15.28:45656) 2024-04-23T14:51:47.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.125+0000 7fa647400640 1 -- 172.21.15.28:0/344385694 learned_addr learned my addr 172.21.15.28:0/344385694 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:47.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.126+0000 7fa647e00640 1 --2- 172.21.15.28:0/344385694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa64819a7d0 0x7fa64819ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:47.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.126+0000 7fa647400640 1 -- 172.21.15.28:0/344385694 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa6481093f0 msgr2=0x7fa6481a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:47.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.126+0000 7fa647400640 1 --2- 172.21.15.28:0/344385694 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa6481093f0 0x7fa6481a0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:47.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.126+0000 7fa647400640 1 -- 172.21.15.28:0/344385694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa64819a7d0 msgr2=0x7fa64819ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:47.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.126+0000 7fa647400640 1 --2- 172.21.15.28:0/344385694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa64819a7d0 0x7fa64819ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:47.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.126+0000 7fa647400640 1 -- 172.21.15.28:0/344385694 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa64819b360 con 0x7fa648104320 2024-04-23T14:51:47.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.126+0000 7fa647400640 1 --2- 172.21.15.28:0/344385694 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa648104320 0x7fa6481a0660 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fa638004770 tx=0x7fa638003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:47.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.127+0000 7fa63fe00640 1 -- 172.21.15.28:0/344385694 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63803f070 con 0x7fa648104320 2024-04-23T14:51:47.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.127+0000 7fa63fe00640 1 -- 172.21.15.28:0/344385694 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa638031ca0 con 0x7fa648104320 2024-04-23T14:51:47.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.127+0000 7fa64d200640 1 -- 172.21.15.28:0/344385694 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa64819b5f0 con 0x7fa648104320 2024-04-23T14:51:47.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.127+0000 7fa63fe00640 1 -- 172.21.15.28:0/344385694 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa638035030 con 0x7fa648104320 2024-04-23T14:51:47.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.127+0000 7fa64d200640 1 -- 172.21.15.28:0/344385694 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6481a77a0 con 0x7fa648104320 2024-04-23T14:51:47.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.130+0000 7fa63fe00640 1 -- 172.21.15.28:0/344385694 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fa638042070 con 0x7fa648104320 2024-04-23T14:51:47.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.130+0000 7fa651614740 1 Processor -- start 2024-04-23T14:51:47.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.131+0000 7fa651614740 1 -- start start 2024-04-23T14:51:47.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.131+0000 7fa651614740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c724ec7a30 0x55c724ec7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:47.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.131+0000 7fa651614740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c724ec8350 0x55c724ed24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:47.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.131+0000 7fa651614740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c724ed2a20 0x55c724ed4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:47.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.131+0000 7fa651614740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c724e2d1f0 con 0x55c724ec8350 2024-04-23T14:51:47.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.131+0000 7fa651614740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c724dd3350 con 0x55c724ed2a20 2024-04-23T14:51:47.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.131+0000 7fa651614740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c724e291b0 con 0x55c724ec7a30 2024-04-23T14:51:47.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.131+0000 7fa647e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c724ec7a30 0x55c724ec7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:47.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.131+0000 7fa646a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c724ed2a20 0x55c724ed4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:47.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.131+0000 7fa647e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c724ec7a30 0x55c724ec7e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:59782/0 (socket says 172.21.15.28:59782) 2024-04-23T14:51:47.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.131+0000 7fa647e00640 1 -- 172.21.15.28:0/1792473275 learned_addr learned my addr 172.21.15.28:0/1792473275 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:47.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.131+0000 7fa647400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c724ec8350 0x55c724ed24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:47.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.132+0000 7fa647e00640 1 -- 172.21.15.28:0/1792473275 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c724ed2a20 msgr2=0x55c724ed4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:47.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.132+0000 7fa647e00640 1 --2- 172.21.15.28:0/1792473275 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c724ed2a20 0x55c724ed4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:47.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.132+0000 7fa647e00640 1 -- 172.21.15.28:0/1792473275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c724ec8350 msgr2=0x55c724ed24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:47.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.132+0000 7fa647e00640 1 --2- 172.21.15.28:0/1792473275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c724ec8350 0x55c724ed24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:47.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.132+0000 7fa647e00640 1 -- 172.21.15.28:0/1792473275 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c724e27f50 con 0x55c724ec7a30 2024-04-23T14:51:47.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.132+0000 7fa647400640 1 --2- 172.21.15.28:0/1792473275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c724ec8350 0x55c724ed24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:51:47.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.132+0000 7fa63fe00640 1 --2- 172.21.15.28:0/344385694 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa62c0a3d20 0x7fa62c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:47.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.132+0000 7fa647e00640 1 --2- 172.21.15.28:0/1792473275 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c724ec7a30 0x55c724ec7e10 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa640009f50 tx=0x7fa6400118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66c6e0d57a8beff server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:47.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.132+0000 7fa63fe00640 1 -- 172.21.15.28:0/344385694 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8542+0+0 (secure 0 0 0) 0x7fa6380f0ae0 con 0x7fa648104320 2024-04-23T14:51:47.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.132+0000 7fa647400640 1 --2- 172.21.15.28:0/344385694 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa62c0a3d20 0x7fa62c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:47.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.132+0000 7fa63cc00640 1 -- 172.21.15.28:0/1792473275 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa640004d20 con 0x55c724ec7a30 2024-04-23T14:51:47.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.132+0000 7fa63cc00640 1 -- 172.21.15.28:0/1792473275 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa640004050 con 0x55c724ec7a30 2024-04-23T14:51:47.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.132+0000 7fa63cc00640 1 -- 172.21.15.28:0/1792473275 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa640005280 con 0x55c724ec7a30 2024-04-23T14:51:47.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.133+0000 7fa651614740 1 -- 172.21.15.28:0/1792473275 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c724ec7a30 msgr2=0x55c724ec7e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:47.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.133+0000 7fa651614740 1 --2- 172.21.15.28:0/1792473275 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c724ec7a30 0x55c724ec7e10 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa640009f50 tx=0x7fa6400118d0 comp rx=0 tx=0).stop 2024-04-23T14:51:47.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.133+0000 7fa647400640 1 --2- 172.21.15.28:0/344385694 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa62c0a3d20 0x7fa62c0a61e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x55c724ec8ff0 tx=0x7fa63803b890 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:47.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.133+0000 7fa651614740 1 -- 172.21.15.28:0/1792473275 shutdown_connections 2024-04-23T14:51:47.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.133+0000 7fa651614740 1 --2- 172.21.15.28:0/1792473275 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c724ed2a20 0x55c724ed4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:47.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.133+0000 7fa651614740 1 --2- 172.21.15.28:0/1792473275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c724ec8350 0x55c724ed24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:47.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.133+0000 7fa651614740 1 --2- 172.21.15.28:0/1792473275 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c724ec7a30 0x55c724ec7e10 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:47.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.133+0000 7fa651614740 1 -- 172.21.15.28:0/1792473275 >> 172.21.15.28:0/1792473275 conn(0x55c724ec3d50 msgr2=0x55c724ec4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:47.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.133+0000 7fa651614740 1 -- 172.21.15.28:0/1792473275 shutdown_connections 2024-04-23T14:51:47.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.134+0000 7fa651614740 1 -- 172.21.15.28:0/1792473275 wait complete. 2024-04-23T14:51:47.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.134+0000 7fa651614740 1 -- start start 2024-04-23T14:51:47.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.134+0000 7fa651614740 1 Processor -- start 2024-04-23T14:51:47.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.135+0000 7fa651614740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c724ec7a30 0x55c724ecf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:47.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.135+0000 7fa651614740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c724ec8350 0x55c724ecff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:47.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.135+0000 7fa651614740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c724ed2a20 0x55c724ed0450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:47.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.135+0000 7fa651614740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c724ed5810 con 0x55c724ec7a30 2024-04-23T14:51:47.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.135+0000 7fa651614740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c724de6690 con 0x55c724ed2a20 2024-04-23T14:51:47.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.135+0000 7fa651614740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c724ed5990 con 0x55c724ec8350 2024-04-23T14:51:47.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.135+0000 7fa651614740 10 client.0 _ll_register_callbacks cb 0x55c724e22650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:51:47.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.135+0000 7fa646a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c724ec8350 0x55c724ecff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:47.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.135+0000 7fa647e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c724ec7a30 0x55c724ecf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:47.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.135+0000 7fa646a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c724ec8350 0x55c724ecff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:59790/0 (socket says 172.21.15.28:59790) 2024-04-23T14:51:47.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.135+0000 7fa646a00640 1 -- 172.21.15.28:0/3821677527 learned_addr learned my addr 172.21.15.28:0/3821677527 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:47.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.135+0000 7fa647e00640 1 --2- 172.21.15.28:0/3821677527 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c724ed2a20 0x55c724ed0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:47.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.136+0000 7fa646a00640 1 -- 172.21.15.28:0/3821677527 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c724ed2a20 msgr2=0x55c724ed0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:47.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.136+0000 7fa646a00640 1 --2- 172.21.15.28:0/3821677527 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c724ed2a20 0x55c724ed0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:47.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.136+0000 7fa646a00640 1 -- 172.21.15.28:0/3821677527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c724ec7a30 msgr2=0x55c724ecf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:47.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.136+0000 7fa646a00640 1 --2- 172.21.15.28:0/3821677527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c724ec7a30 0x55c724ecf9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:47.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.136+0000 7fa646a00640 1 -- 172.21.15.28:0/3821677527 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c724de5920 con 0x55c724ec8350 2024-04-23T14:51:47.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.136+0000 7fa647e00640 1 --2- 172.21.15.28:0/3821677527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c724ec7a30 0x55c724ecf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:51:47.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.136+0000 7fa646a00640 1 --2- 172.21.15.28:0/3821677527 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c724ec8350 0x55c724ecff10 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa634012730 tx=0x7fa634012bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3922724524701b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:47.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.136+0000 7fa631600640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:51:47.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.136+0000 7fa631600640 1 -- 172.21.15.28:0/3821677527 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63400f920 con 0x55c724ec8350 2024-04-23T14:51:47.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.136+0000 7fa631600640 1 -- 172.21.15.28:0/3821677527 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8542+0+0 (secure 0 0 0) 0x7fa63400fac0 con 0x55c724ec8350 2024-04-23T14:51:47.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.136+0000 7fa651614740 1 -- 172.21.15.28:0/3821677527 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa640004050 con 0x55c724ec8350 2024-04-23T14:51:47.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.137+0000 7fa631600640 1 -- 172.21.15.28:0/3821677527 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fa63400fcc0 con 0x55c724ec8350 2024-04-23T14:51:47.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.137+0000 7fa651614740 10 client.26422 fetch_fsmap learned FSMap version 20 2024-04-23T14:51:47.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.137+0000 7fa651614740 1 -- 172.21.15.28:0/3821677527 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55c724c097d0 con 0x55c724ec8350 2024-04-23T14:51:47.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.137+0000 7fa631600640 1 -- 172.21.15.28:0/3821677527 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa640004050 con 0x55c724ec8350 2024-04-23T14:51:47.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.138+0000 7fa631600640 1 -- 172.21.15.28:0/3821677527 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fa63401c420 con 0x55c724ec8350 2024-04-23T14:51:47.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.138+0000 7fa651614740 10 client.26422 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:51:47.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.138+0000 7fa651614740 10 client.26422 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:51:47.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.138+0000 7fa651614740 20 client.26422 populate_metadata read hostname 'smithi028' 2024-04-23T14:51:47.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.138+0000 7fa651614740 1 --2- 172.21.15.28:0/3821677527 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c724ee2e50 0x55c724ee52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:47.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.138+0000 7fa651614740 4 client.26422 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:51:47.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.138+0000 7fa651614740 1 -- 172.21.15.28:0/3821677527 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c724e3ac10 con 0x55c724ee2e50 2024-04-23T14:51:47.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.138+0000 7fa647e00640 1 --2- 172.21.15.28:0/3821677527 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c724ee2e50 0x55c724ee52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:47.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.139+0000 7fa647e00640 1 --2- 172.21.15.28:0/3821677527 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c724ee2e50 0x55c724ee52b0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fed90068b5f3b0e6 server_cookie=eb68bfd82676c00b in_seq=0 out_seq=0 2024-04-23T14:51:47.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.139+0000 7fa631600640 10 client.26422 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:51:47.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.140+0000 7fa631600640 1 -- 172.21.15.28:0/3821677527 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c724e3ac10 con 0x55c724ee2e50 2024-04-23T14:51:47.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.140+0000 7fa631600640 10 client.26422 handle_command_reply: tid=0 2024-04-23T14:51:47.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.140+0000 7fa651614740 1 client.26422 shutdown 2024-04-23T14:51:47.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.140+0000 7fa651614740 1 -- 172.21.15.28:0/3821677527 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c724ec8350 msgr2=0x55c724ecff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:47.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.140+0000 7fa651614740 1 --2- 172.21.15.28:0/3821677527 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c724ec8350 0x55c724ecff10 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa634012730 tx=0x7fa634012bf0 comp rx=0 tx=0).stop 2024-04-23T14:51:47.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.140+0000 7fa651614740 1 -- 172.21.15.28:0/3821677527 shutdown_connections 2024-04-23T14:51:47.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.140+0000 7fa651614740 1 --2- 172.21.15.28:0/3821677527 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c724ee2e50 0x55c724ee52b0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:47.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.140+0000 7fa651614740 1 --2- 172.21.15.28:0/3821677527 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c724ed2a20 0x55c724ed0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:47.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.140+0000 7fa651614740 1 --2- 172.21.15.28:0/3821677527 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c724ec8350 0x55c724ecff10 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:47.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.140+0000 7fa651614740 1 --2- 172.21.15.28:0/3821677527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c724ec7a30 0x55c724ecf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:47.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.140+0000 7fa631600640 0 client.26422 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:51:47.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.140+0000 7fa651614740 1 -- 172.21.15.28:0/3821677527 >> 172.21.15.28:0/3821677527 conn(0x55c724ec3d50 msgr2=0x55c724ec2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:47.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.140+0000 7fa651614740 1 -- 172.21.15.28:0/3821677527 shutdown_connections 2024-04-23T14:51:47.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.140+0000 7fa651614740 1 -- 172.21.15.28:0/3821677527 wait complete. 2024-04-23T14:51:47.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.141+0000 7fa651614740 20 client.26422 trim_cache size 0 max 16384 2024-04-23T14:51:47.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.162+0000 7fa651614740 1 Processor -- start 2024-04-23T14:51:47.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.162+0000 7fa651614740 1 -- start start 2024-04-23T14:51:47.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.162+0000 7fa651614740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c724ec7a30 0x55c724ede700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:47.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.162+0000 7fa651614740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c724ec8350 0x55c724eeab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:47.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.162+0000 7fa651614740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c724ed2a20 0x55c724eeb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:47.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.162+0000 7fa651614740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa6481ade70 con 0x55c724ec8350 2024-04-23T14:51:47.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.162+0000 7fa651614740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa6481b06d0 con 0x55c724ec7a30 2024-04-23T14:51:47.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.162+0000 7fa651614740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa6481ae4c0 con 0x55c724ed2a20 2024-04-23T14:51:47.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.162+0000 7fa647e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c724ec7a30 0x55c724ede700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:47.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.162+0000 7fa647e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c724ec7a30 0x55c724ede700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:45672/0 (socket says 172.21.15.28:45672) 2024-04-23T14:51:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.162+0000 7fa647e00640 1 -- 172.21.15.28:0/52973360 learned_addr learned my addr 172.21.15.28:0/52973360 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.163+0000 7fa646a00640 1 --2- 172.21.15.28:0/52973360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c724ec8350 0x55c724eeab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.163+0000 7fa647e00640 1 --2- 172.21.15.28:0/52973360 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c724ed2a20 0x55c724eeb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.163+0000 7fa647e00640 1 -- 172.21.15.28:0/52973360 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c724ed2a20 msgr2=0x55c724eeb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.163+0000 7fa647e00640 1 --2- 172.21.15.28:0/52973360 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c724ed2a20 0x55c724eeb0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.163+0000 7fa647e00640 1 -- 172.21.15.28:0/52973360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c724ec8350 msgr2=0x55c724eeab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.163+0000 7fa647e00640 1 --2- 172.21.15.28:0/52973360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c724ec8350 0x55c724eeab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.163+0000 7fa647e00640 1 -- 172.21.15.28:0/52973360 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c724b3f3f0 con 0x55c724ec7a30 2024-04-23T14:51:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.163+0000 7fa647e00640 1 --2- 172.21.15.28:0/52973360 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c724ec7a30 0x55c724ede700 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa640009730 tx=0x7fa640009c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b2ca08d510f3f57 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:47.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.163+0000 7fa633e00640 1 -- 172.21.15.28:0/52973360 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa640004930 con 0x55c724ec7a30 2024-04-23T14:51:47.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.164+0000 7fa633e00640 1 -- 172.21.15.28:0/52973360 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa640027440 con 0x55c724ec7a30 2024-04-23T14:51:47.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.164+0000 7fa633e00640 1 -- 172.21.15.28:0/52973360 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa64000f6e0 con 0x55c724ec7a30 2024-04-23T14:51:47.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.164+0000 7fa651614740 1 -- 172.21.15.28:0/52973360 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c724ec7a30 msgr2=0x55c724ede700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:47.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.164+0000 7fa651614740 1 --2- 172.21.15.28:0/52973360 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c724ec7a30 0x55c724ede700 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa640009730 tx=0x7fa640009c60 comp rx=0 tx=0).stop 2024-04-23T14:51:47.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.164+0000 7fa651614740 1 -- 172.21.15.28:0/52973360 shutdown_connections 2024-04-23T14:51:47.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.164+0000 7fa651614740 1 --2- 172.21.15.28:0/52973360 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c724ed2a20 0x55c724eeb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:47.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.164+0000 7fa651614740 1 --2- 172.21.15.28:0/52973360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c724ec8350 0x55c724eeab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:47.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.164+0000 7fa651614740 1 --2- 172.21.15.28:0/52973360 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c724ec7a30 0x55c724ede700 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:47.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.164+0000 7fa651614740 1 -- 172.21.15.28:0/52973360 >> 172.21.15.28:0/52973360 conn(0x55c724ec3d50 msgr2=0x55c724ed0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:47.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.164+0000 7fa651614740 1 -- 172.21.15.28:0/52973360 shutdown_connections 2024-04-23T14:51:47.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.165+0000 7fa651614740 1 -- 172.21.15.28:0/52973360 wait complete. 2024-04-23T14:51:47.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.165+0000 7fa651614740 1 -- start start 2024-04-23T14:51:47.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.165+0000 7fa651614740 1 Processor -- start 2024-04-23T14:51:47.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.166+0000 7fa651614740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c724ec7a30 0x55c724edaf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:47.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.166+0000 7fa651614740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c724ec8350 0x55c724ecc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:47.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.166+0000 7fa651614740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c724ed2a20 0x55c724ecc800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:47.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.166+0000 7fa651614740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa6481aeb90 con 0x55c724ed2a20 2024-04-23T14:51:47.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.166+0000 7fa651614740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa6481b0000 con 0x55c724ec8350 2024-04-23T14:51:47.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.166+0000 7fa651614740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa6481af260 con 0x55c724ec7a30 2024-04-23T14:51:47.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.166+0000 7fa651614740 10 client.0 _ll_register_callbacks cb 0x7fa634009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:51:47.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.166+0000 7fa647e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c724ec7a30 0x55c724edaf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:47.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.166+0000 7fa646a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c724ec8350 0x55c724ecc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:47.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.166+0000 7fa647e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c724ed2a20 0x55c724ecc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:47.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.166+0000 7fa646a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c724ec8350 0x55c724ecc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:45674/0 (socket says 172.21.15.28:45674) 2024-04-23T14:51:47.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.166+0000 7fa646a00640 1 -- 172.21.15.28:0/465835495 learned_addr learned my addr 172.21.15.28:0/465835495 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:51:47.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.167+0000 7fa646a00640 1 -- 172.21.15.28:0/465835495 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c724ec7a30 msgr2=0x55c724edaf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:47.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.167+0000 7fa646a00640 1 --2- 172.21.15.28:0/465835495 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c724ec7a30 0x55c724edaf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:47.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.167+0000 7fa646a00640 1 -- 172.21.15.28:0/465835495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c724ed2a20 msgr2=0x55c724ecc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:51:47.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.167+0000 7fa646a00640 1 --2- 172.21.15.28:0/465835495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c724ed2a20 0x55c724ecc800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:47.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.167+0000 7fa646a00640 1 -- 172.21.15.28:0/465835495 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c724aff0f0 con 0x55c724ec8350 2024-04-23T14:51:47.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.167+0000 7fa646a00640 1 --2- 172.21.15.28:0/465835495 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c724ec8350 0x55c724ecc2c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x55c724ecf230 tx=0x7fa63400f800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9cacbb7dae1cd23 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:51:47.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.167+0000 7fa631600640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:51:47.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.167+0000 7fa631600640 1 -- 172.21.15.28:0/465835495 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63400fe60 con 0x55c724ec8350 2024-04-23T14:51:47.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.167+0000 7fa631600640 1 -- 172.21.15.28:0/465835495 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8542+0+0 (secure 0 0 0) 0x7fa63401c630 con 0x55c724ec8350 2024-04-23T14:51:47.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.167+0000 7fa651614740 1 -- 172.21.15.28:0/465835495 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa640027440 con 0x55c724ec8350 2024-04-23T14:51:47.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.168+0000 7fa631600640 1 -- 172.21.15.28:0/465835495 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fa634033050 con 0x55c724ec8350 2024-04-23T14:51:47.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.168+0000 7fa651614740 10 client.26444 fetch_fsmap learned FSMap version 20 2024-04-23T14:51:47.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.168+0000 7fa651614740 1 -- 172.21.15.28:0/465835495 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55c724df2810 con 0x55c724ec8350 2024-04-23T14:51:47.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.168+0000 7fa631600640 1 -- 172.21.15.28:0/465835495 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa640027440 con 0x55c724ec8350 2024-04-23T14:51:47.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.168+0000 7fa631600640 1 -- 172.21.15.28:0/465835495 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fa63402b020 con 0x55c724ec8350 2024-04-23T14:51:47.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.168+0000 7fa651614740 10 client.26444 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:51:47.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.168+0000 7fa651614740 10 client.26444 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:51:47.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.168+0000 7fa651614740 20 client.26444 populate_metadata read hostname 'smithi028' 2024-04-23T14:51:47.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.168+0000 7fa651614740 1 --2- 172.21.15.28:0/465835495 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c724f497f0 0x55c724f4bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:51:47.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.168+0000 7fa651614740 4 client.26444 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:51:47.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.168+0000 7fa651614740 1 -- 172.21.15.28:0/465835495 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c724e34a80 con 0x55c724f497f0 2024-04-23T14:51:47.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.168+0000 7fa647e00640 1 --2- 172.21.15.28:0/465835495 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c724f497f0 0x55c724f4bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:51:47.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.169+0000 7fa647e00640 1 --2- 172.21.15.28:0/465835495 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c724f497f0 0x55c724f4bc10 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=764619b00f6a3e70 server_cookie=39295a802f44a894 in_seq=0 out_seq=0 2024-04-23T14:51:47.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.169+0000 7fa631600640 10 client.26444 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:51:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.169+0000 7fa631600640 1 -- 172.21.15.28:0/465835495 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55c724e34a80 con 0x55c724f497f0 2024-04-23T14:51:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.169+0000 7fa631600640 10 client.26444 handle_command_reply: tid=0 2024-04-23T14:51:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.169+0000 7fa651614740 1 client.26444 shutdown 2024-04-23T14:51:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.170+0000 7fa651614740 1 -- 172.21.15.28:0/465835495 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c724ec8350 msgr2=0x55c724ecc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.170+0000 7fa651614740 1 --2- 172.21.15.28:0/465835495 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c724ec8350 0x55c724ecc2c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x55c724ecf230 tx=0x7fa63400f800 comp rx=0 tx=0).stop 2024-04-23T14:51:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.170+0000 7fa651614740 1 -- 172.21.15.28:0/465835495 shutdown_connections 2024-04-23T14:51:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.170+0000 7fa651614740 1 --2- 172.21.15.28:0/465835495 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c724f497f0 0x55c724f4bc10 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.170+0000 7fa651614740 1 --2- 172.21.15.28:0/465835495 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c724ec8350 0x55c724ecc2c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.170+0000 7fa651614740 1 --2- 172.21.15.28:0/465835495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c724ed2a20 0x55c724ecc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.170+0000 7fa651614740 1 --2- 172.21.15.28:0/465835495 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c724ec7a30 0x55c724edaf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.170+0000 7fa651614740 1 -- 172.21.15.28:0/465835495 >> 172.21.15.28:0/465835495 conn(0x55c724ec3d50 msgr2=0x55c724ec9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.170+0000 7fa631600640 0 client.26444 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:51:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.170+0000 7fa651614740 1 -- 172.21.15.28:0/465835495 shutdown_connections 2024-04-23T14:51:47.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.170+0000 7fa651614740 1 -- 172.21.15.28:0/465835495 wait complete. 2024-04-23T14:51:47.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.170+0000 7fa651614740 20 client.26444 trim_cache size 0 max 16384 2024-04-23T14:51:47.175 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:51:47.175 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-23T14:51:47.175 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-23T14:51:47.175 INFO:teuthology.orchestra.run.smithi028.stdout: "c381e5c7-159e-4f54-b397-a9a32e09d526": { 2024-04-23T14:51:47.175 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-23T14:51:47.175 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "c381e5c7-159e-4f54-b397-a9a32e09d526", 2024-04-23T14:51:47.175 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-23T14:51:47.175 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:51:47.176 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:51:47.176 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:51:47.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.171+0000 7fa64d200640 1 -- 172.21.15.28:0/344385694 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa62c0a3d20 msgr2=0x7fa62c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:47.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.171+0000 7fa64d200640 1 --2- 172.21.15.28:0/344385694 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa62c0a3d20 0x7fa62c0a61e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x55c724ec8ff0 tx=0x7fa63803b890 comp rx=0 tx=0).stop 2024-04-23T14:51:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.171+0000 7fa64d200640 1 -- 172.21.15.28:0/344385694 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa648104320 msgr2=0x7fa6481a0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:51:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.171+0000 7fa64d200640 1 --2- 172.21.15.28:0/344385694 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa648104320 0x7fa6481a0660 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fa638004770 tx=0x7fa638003c10 comp rx=0 tx=0).stop 2024-04-23T14:51:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.171+0000 7fa64d200640 1 -- 172.21.15.28:0/344385694 shutdown_connections 2024-04-23T14:51:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.171+0000 7fa64d200640 1 --2- 172.21.15.28:0/344385694 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa62c0a3d20 0x7fa62c0a61e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.171+0000 7fa64d200640 1 --2- 172.21.15.28:0/344385694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa64819a7d0 0x7fa64819ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.171+0000 7fa64d200640 1 --2- 172.21.15.28:0/344385694 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa6481093f0 0x7fa6481a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.171+0000 7fa64d200640 1 --2- 172.21.15.28:0/344385694 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa648104320 0x7fa6481a0660 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:51:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.171+0000 7fa64d200640 1 -- 172.21.15.28:0/344385694 >> 172.21.15.28:0/344385694 conn(0x7fa6480ffb40 msgr2=0x7fa648101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:51:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.171+0000 7fa64d200640 1 -- 172.21.15.28:0/344385694 shutdown_connections 2024-04-23T14:51:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:51:47.171+0000 7fa64d200640 1 -- 172.21.15.28:0/344385694 wait complete. 2024-04-23T14:51:47.191 INFO:tasks.cephfs.filesystem:scrub status for tag:c381e5c7-159e-4f54-b397-a9a32e09d526 - {'path': '/', 'tag': 'c381e5c7-159e-4f54-b397-a9a32e09d526', 'options': 'recursive,force'} 2024-04-23T14:51:47.507 INFO:tasks.workunit.client.0.smithi028.stdout: 1 642278 9.11 MB/sec execute 243 sec latency 57.236 ms 2024-04-23T14:51:48.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:47 smithi105 ceph-mon[32555]: scrub queued for path: / 2024-04-23T14:51:48.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:47 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:51:48.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:47 smithi105 ceph-mon[32555]: scrub summary: active paths [/] 2024-04-23T14:51:48.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:47 smithi038 ceph-mon[32348]: scrub queued for path: / 2024-04-23T14:51:48.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:47 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:51:48.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:47 smithi038 ceph-mon[32348]: scrub summary: active paths [/] 2024-04-23T14:51:48.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:47 smithi028 ceph-mon[27810]: scrub queued for path: / 2024-04-23T14:51:48.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:47 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:51:48.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:47 smithi028 ceph-mon[27810]: scrub summary: active paths [/] 2024-04-23T14:51:48.507 INFO:tasks.workunit.client.0.smithi028.stdout: 1 643667 9.11 MB/sec execute 244 sec latency 21.787 ms 2024-04-23T14:51:49.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:48 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:51:49.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:48 smithi105 ceph-mon[32555]: pgmap v238: 97 pgs: 97 active+clean; 430 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.8 MiB/s wr, 201 op/s 2024-04-23T14:51:49.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:48 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:51:49.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:48 smithi038 ceph-mon[32348]: pgmap v238: 97 pgs: 97 active+clean; 430 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.8 MiB/s wr, 201 op/s 2024-04-23T14:51:49.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:48 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:51:49.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:48 smithi028 ceph-mon[27810]: pgmap v238: 97 pgs: 97 active+clean; 430 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.8 MiB/s wr, 201 op/s 2024-04-23T14:51:49.507 INFO:tasks.workunit.client.0.smithi028.stdout: 1 645135 9.09 MB/sec execute 245 sec latency 46.836 ms 2024-04-23T14:51:50.507 INFO:tasks.workunit.client.0.smithi028.stdout: 1 646764 9.11 MB/sec execute 246 sec latency 13.318 ms 2024-04-23T14:51:51.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:50 smithi105 ceph-mon[32555]: pgmap v239: 97 pgs: 97 active+clean; 430 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 136 op/s 2024-04-23T14:51:51.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:50 smithi105 ceph-mon[32555]: scrub summary: idle 2024-04-23T14:51:51.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:50 smithi038 ceph-mon[32348]: pgmap v239: 97 pgs: 97 active+clean; 430 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 136 op/s 2024-04-23T14:51:51.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:50 smithi038 ceph-mon[32348]: scrub summary: idle 2024-04-23T14:51:51.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:50 smithi028 ceph-mon[27810]: pgmap v239: 97 pgs: 97 active+clean; 430 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 136 op/s 2024-04-23T14:51:51.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:50 smithi028 ceph-mon[27810]: scrub summary: idle 2024-04-23T14:51:51.507 INFO:tasks.workunit.client.0.smithi028.stdout: 1 648131 9.09 MB/sec execute 247 sec latency 18.190 ms 2024-04-23T14:51:52.507 INFO:tasks.workunit.client.0.smithi028.stdout: 1 650139 9.11 MB/sec execute 248 sec latency 14.075 ms 2024-04-23T14:51:53.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:52 smithi105 ceph-mon[32555]: pgmap v240: 97 pgs: 97 active+clean; 430 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 136 op/s 2024-04-23T14:51:53.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:52 smithi038 ceph-mon[32348]: pgmap v240: 97 pgs: 97 active+clean; 430 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 136 op/s 2024-04-23T14:51:53.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:52 smithi028 ceph-mon[27810]: pgmap v240: 97 pgs: 97 active+clean; 430 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 136 op/s 2024-04-23T14:51:53.508 INFO:tasks.workunit.client.0.smithi028.stdout: 1 651422 9.10 MB/sec execute 249 sec latency 17.292 ms 2024-04-23T14:51:54.508 INFO:tasks.workunit.client.0.smithi028.stdout: 1 653262 9.10 MB/sec execute 250 sec latency 42.490 ms 2024-04-23T14:51:55.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:54 smithi105 ceph-mon[32555]: pgmap v241: 97 pgs: 97 active+clean; 408 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 203 op/s 2024-04-23T14:51:55.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:54 smithi038 ceph-mon[32348]: pgmap v241: 97 pgs: 97 active+clean; 408 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 203 op/s 2024-04-23T14:51:55.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:54 smithi028 ceph-mon[27810]: pgmap v241: 97 pgs: 97 active+clean; 408 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 203 op/s 2024-04-23T14:51:55.508 INFO:tasks.workunit.client.0.smithi028.stdout: 1 654479 9.09 MB/sec execute 251 sec latency 17.572 ms 2024-04-23T14:51:56.508 INFO:tasks.workunit.client.0.smithi028.stdout: 1 656205 9.09 MB/sec execute 252 sec latency 20.705 ms 2024-04-23T14:51:57.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:56 smithi105 ceph-mon[32555]: pgmap v242: 97 pgs: 97 active+clean; 408 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 139 op/s 2024-04-23T14:51:57.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:56 smithi038 ceph-mon[32348]: pgmap v242: 97 pgs: 97 active+clean; 408 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 139 op/s 2024-04-23T14:51:57.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:56 smithi028 ceph-mon[27810]: pgmap v242: 97 pgs: 97 active+clean; 408 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 139 op/s 2024-04-23T14:51:57.508 INFO:tasks.workunit.client.0.smithi028.stdout: 1 657576 9.08 MB/sec execute 253 sec latency 11.279 ms 2024-04-23T14:51:58.508 INFO:tasks.workunit.client.0.smithi028.stdout: 1 658963 9.07 MB/sec execute 254 sec latency 35.002 ms 2024-04-23T14:51:59.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:51:58 smithi105 ceph-mon[32555]: pgmap v243: 97 pgs: 97 active+clean; 414 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.7 MiB/s wr, 203 op/s 2024-04-23T14:51:59.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:51:58 smithi038 ceph-mon[32348]: pgmap v243: 97 pgs: 97 active+clean; 414 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.7 MiB/s wr, 203 op/s 2024-04-23T14:51:59.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:51:58 smithi028 ceph-mon[27810]: pgmap v243: 97 pgs: 97 active+clean; 414 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.7 MiB/s wr, 203 op/s 2024-04-23T14:51:59.508 INFO:tasks.workunit.client.0.smithi028.stdout: 1 660799 9.08 MB/sec execute 255 sec latency 71.898 ms 2024-04-23T14:52:00.510 INFO:tasks.workunit.client.0.smithi028.stdout: 1 662171 9.07 MB/sec execute 256 sec latency 21.065 ms 2024-04-23T14:52:01.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:00 smithi105 ceph-mon[32555]: pgmap v244: 97 pgs: 97 active+clean; 414 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 130 op/s 2024-04-23T14:52:01.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:00 smithi038 ceph-mon[32348]: pgmap v244: 97 pgs: 97 active+clean; 414 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 130 op/s 2024-04-23T14:52:01.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:00 smithi028 ceph-mon[27810]: pgmap v244: 97 pgs: 97 active+clean; 414 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 130 op/s 2024-04-23T14:52:01.508 INFO:tasks.workunit.client.0.smithi028.stdout: 1 662820 9.04 MB/sec execute 257 sec latency 71.567 ms 2024-04-23T14:52:02.509 INFO:tasks.workunit.client.0.smithi028.stdout: 1 664472 9.05 MB/sec execute 258 sec latency 54.978 ms 2024-04-23T14:52:03.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:02 smithi105 ceph-mon[32555]: pgmap v245: 97 pgs: 97 active+clean; 414 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 130 op/s 2024-04-23T14:52:03.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:02 smithi038 ceph-mon[32348]: pgmap v245: 97 pgs: 97 active+clean; 414 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 130 op/s 2024-04-23T14:52:03.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:02 smithi028 ceph-mon[27810]: pgmap v245: 97 pgs: 97 active+clean; 414 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 130 op/s 2024-04-23T14:52:03.509 INFO:tasks.workunit.client.0.smithi028.stdout: 1 665722 9.04 MB/sec execute 259 sec latency 21.667 ms 2024-04-23T14:52:04.509 INFO:tasks.workunit.client.0.smithi028.stdout: 1 667428 9.04 MB/sec execute 260 sec latency 39.546 ms 2024-04-23T14:52:05.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:04 smithi105 ceph-mon[32555]: pgmap v246: 97 pgs: 97 active+clean; 420 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 235 op/s 2024-04-23T14:52:05.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:04 smithi038 ceph-mon[32348]: pgmap v246: 97 pgs: 97 active+clean; 420 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 235 op/s 2024-04-23T14:52:05.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:04 smithi028 ceph-mon[27810]: pgmap v246: 97 pgs: 97 active+clean; 420 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 235 op/s 2024-04-23T14:52:05.509 INFO:tasks.workunit.client.0.smithi028.stdout: 1 668601 9.03 MB/sec execute 261 sec latency 41.863 ms 2024-04-23T14:52:06.509 INFO:tasks.workunit.client.0.smithi028.stdout: 1 670119 9.02 MB/sec execute 262 sec latency 36.627 ms 2024-04-23T14:52:07.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:06 smithi038 ceph-mon[32348]: pgmap v247: 97 pgs: 97 active+clean; 420 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 168 op/s 2024-04-23T14:52:07.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:06 smithi038 ceph-mon[32348]: osdmap e91: 12 total, 12 up, 12 in 2024-04-23T14:52:07.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:06 smithi028 ceph-mon[27810]: pgmap v247: 97 pgs: 97 active+clean; 420 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 168 op/s 2024-04-23T14:52:07.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:06 smithi028 ceph-mon[27810]: osdmap e91: 12 total, 12 up, 12 in 2024-04-23T14:52:07.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:06 smithi105 ceph-mon[32555]: pgmap v247: 97 pgs: 97 active+clean; 420 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 168 op/s 2024-04-23T14:52:07.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:06 smithi105 ceph-mon[32555]: osdmap e91: 12 total, 12 up, 12 in 2024-04-23T14:52:07.509 INFO:tasks.workunit.client.0.smithi028.stdout: 1 671695 9.03 MB/sec execute 263 sec latency 9.581 ms 2024-04-23T14:52:08.509 INFO:tasks.workunit.client.0.smithi028.stdout: 1 673197 9.01 MB/sec execute 264 sec latency 18.639 ms 2024-04-23T14:52:09.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:08 smithi038 ceph-mon[32348]: pgmap v249: 97 pgs: 97 active+clean; 423 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.3 MiB/s wr, 204 op/s 2024-04-23T14:52:09.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:08 smithi028 ceph-mon[27810]: pgmap v249: 97 pgs: 97 active+clean; 423 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.3 MiB/s wr, 204 op/s 2024-04-23T14:52:09.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:08 smithi105 ceph-mon[32555]: pgmap v249: 97 pgs: 97 active+clean; 423 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.3 MiB/s wr, 204 op/s 2024-04-23T14:52:09.510 INFO:tasks.workunit.client.0.smithi028.stdout: 1 674999 9.03 MB/sec execute 265 sec latency 41.540 ms 2024-04-23T14:52:10.510 INFO:tasks.workunit.client.0.smithi028.stdout: 1 676191 9.02 MB/sec execute 266 sec latency 60.892 ms 2024-04-23T14:52:11.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:10 smithi038 ceph-mon[32348]: pgmap v250: 97 pgs: 97 active+clean; 423 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.3 MiB/s wr, 204 op/s 2024-04-23T14:52:11.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:10 smithi028 ceph-mon[27810]: pgmap v250: 97 pgs: 97 active+clean; 423 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.3 MiB/s wr, 204 op/s 2024-04-23T14:52:11.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:10 smithi105 ceph-mon[32555]: pgmap v250: 97 pgs: 97 active+clean; 423 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.3 MiB/s wr, 204 op/s 2024-04-23T14:52:11.510 INFO:tasks.workunit.client.0.smithi028.stdout: 1 677698 9.01 MB/sec execute 267 sec latency 26.662 ms 2024-04-23T14:52:12.510 INFO:tasks.workunit.client.0.smithi028.stdout: 1 678929 9.00 MB/sec execute 268 sec latency 17.719 ms 2024-04-23T14:52:12.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:12 smithi038 ceph-mon[32348]: pgmap v251: 97 pgs: 97 active+clean; 423 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.3 MiB/s wr, 204 op/s 2024-04-23T14:52:12.872 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:12 smithi038 ceph-mon[32348]: osdmap e92: 12 total, 12 up, 12 in 2024-04-23T14:52:12.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:12 smithi028 ceph-mon[27810]: pgmap v251: 97 pgs: 97 active+clean; 423 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.3 MiB/s wr, 204 op/s 2024-04-23T14:52:12.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:12 smithi028 ceph-mon[27810]: osdmap e92: 12 total, 12 up, 12 in 2024-04-23T14:52:13.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:12 smithi105 ceph-mon[32555]: pgmap v251: 97 pgs: 97 active+clean; 423 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.3 MiB/s wr, 204 op/s 2024-04-23T14:52:13.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:12 smithi105 ceph-mon[32555]: osdmap e92: 12 total, 12 up, 12 in 2024-04-23T14:52:13.510 INFO:tasks.workunit.client.0.smithi028.stdout: 1 680401 8.99 MB/sec execute 269 sec latency 21.011 ms 2024-04-23T14:52:14.510 INFO:tasks.workunit.client.0.smithi028.stdout: 1 682079 9.01 MB/sec execute 270 sec latency 42.548 ms 2024-04-23T14:52:15.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:14 smithi105 ceph-mon[32555]: pgmap v253: 97 pgs: 97 active+clean; 430 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.9 MiB/s wr, 195 op/s 2024-04-23T14:52:15.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:14 smithi038 ceph-mon[32348]: pgmap v253: 97 pgs: 97 active+clean; 430 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.9 MiB/s wr, 195 op/s 2024-04-23T14:52:15.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:14 smithi028 ceph-mon[27810]: pgmap v253: 97 pgs: 97 active+clean; 430 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.9 MiB/s wr, 195 op/s 2024-04-23T14:52:15.510 INFO:tasks.workunit.client.0.smithi028.stdout: 1 683282 8.99 MB/sec execute 271 sec latency 42.595 ms 2024-04-23T14:52:16.510 INFO:tasks.workunit.client.0.smithi028.stdout: 1 684867 8.99 MB/sec execute 272 sec latency 32.031 ms 2024-04-23T14:52:17.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:16 smithi105 ceph-mon[32555]: pgmap v254: 97 pgs: 97 active+clean; 430 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 160 op/s 2024-04-23T14:52:17.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:16 smithi038 ceph-mon[32348]: pgmap v254: 97 pgs: 97 active+clean; 430 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 160 op/s 2024-04-23T14:52:17.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:16 smithi028 ceph-mon[27810]: pgmap v254: 97 pgs: 97 active+clean; 430 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 160 op/s 2024-04-23T14:52:17.193 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:52:17.193 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:52:17.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.419+0000 7f3b36600640 1 -- 172.21.15.28:0/2715684444 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b30074bf0 msgr2=0x7f3b30112b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:17.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.419+0000 7f3b36600640 1 --2- 172.21.15.28:0/2715684444 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b30074bf0 0x7f3b30112b70 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c009900 tx=0x7f3b1c02a480 comp rx=0 tx=0).stop 2024-04-23T14:52:17.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.419+0000 7f3b36600640 1 -- 172.21.15.28:0/2715684444 shutdown_connections 2024-04-23T14:52:17.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.419+0000 7f3b36600640 1 --2- 172.21.15.28:0/2715684444 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b30074bf0 0x7f3b30112b70 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.419+0000 7f3b36600640 1 --2- 172.21.15.28:0/2715684444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b301130b0 0x7f3b301154a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.419+0000 7f3b36600640 1 --2- 172.21.15.28:0/2715684444 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b300742d0 0x7f3b300746b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.419+0000 7f3b36600640 1 -- 172.21.15.28:0/2715684444 >> 172.21.15.28:0/2715684444 conn(0x7f3b30079450 msgr2=0x7f3b30079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:17.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.419+0000 7f3b36600640 1 -- 172.21.15.28:0/2715684444 shutdown_connections 2024-04-23T14:52:17.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.419+0000 7f3b36600640 1 -- 172.21.15.28:0/2715684444 wait complete. 2024-04-23T14:52:17.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.420+0000 7f3b36600640 1 Processor -- start 2024-04-23T14:52:17.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.420+0000 7f3b36600640 1 -- start start 2024-04-23T14:52:17.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.421+0000 7f3b36600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b300742d0 0x7f3b3019ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:17.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.421+0000 7f3b36600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b30074bf0 0x7f3b3019f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:17.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.421+0000 7f3b36600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b301130b0 0x7f3b301a3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:17.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.421+0000 7f3b36600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3b30118a20 con 0x7f3b30074bf0 2024-04-23T14:52:17.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.421+0000 7f3b36600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3b301188a0 con 0x7f3b301130b0 2024-04-23T14:52:17.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.421+0000 7f3b36600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3b30118ba0 con 0x7f3b300742d0 2024-04-23T14:52:17.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.421+0000 7f3b2f400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b300742d0 0x7f3b3019ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:17.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.421+0000 7f3b2fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b301130b0 0x7f3b301a3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:17.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.421+0000 7f3b2f400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b300742d0 0x7f3b3019ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:56736/0 (socket says 172.21.15.28:56736) 2024-04-23T14:52:17.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.421+0000 7f3b2f400640 1 -- 172.21.15.28:0/46909004 learned_addr learned my addr 172.21.15.28:0/46909004 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:17.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.421+0000 7f3b2ea00640 1 --2- 172.21.15.28:0/46909004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b30074bf0 0x7f3b3019f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:17.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.422+0000 7f3b2f400640 1 -- 172.21.15.28:0/46909004 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b301130b0 msgr2=0x7f3b301a3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:17.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.422+0000 7f3b2f400640 1 --2- 172.21.15.28:0/46909004 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b301130b0 0x7f3b301a3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.422+0000 7f3b2f400640 1 -- 172.21.15.28:0/46909004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b30074bf0 msgr2=0x7f3b3019f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:17.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.422+0000 7f3b2f400640 1 --2- 172.21.15.28:0/46909004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b30074bf0 0x7f3b3019f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.422+0000 7f3b2f400640 1 -- 172.21.15.28:0/46909004 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b301a3f50 con 0x7f3b300742d0 2024-04-23T14:52:17.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.422+0000 7f3b2f400640 1 --2- 172.21.15.28:0/46909004 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b300742d0 0x7f3b3019ef60 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3b2000e9f0 tx=0x7f3b2000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:17.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.422+0000 7f3b27e00640 1 -- 172.21.15.28:0/46909004 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b2000cd80 con 0x7f3b300742d0 2024-04-23T14:52:17.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.423+0000 7f3b27e00640 1 -- 172.21.15.28:0/46909004 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b20005500 con 0x7f3b300742d0 2024-04-23T14:52:17.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.423+0000 7f3b36600640 1 -- 172.21.15.28:0/46909004 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b301a4240 con 0x7f3b300742d0 2024-04-23T14:52:17.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.423+0000 7f3b27e00640 1 -- 172.21.15.28:0/46909004 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b20010690 con 0x7f3b300742d0 2024-04-23T14:52:17.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.423+0000 7f3b36600640 1 -- 172.21.15.28:0/46909004 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b301abad0 con 0x7f3b300742d0 2024-04-23T14:52:17.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.426+0000 7f3b393d3740 1 Processor -- start 2024-04-23T14:52:17.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.426+0000 7f3b27e00640 1 -- 172.21.15.28:0/46909004 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f3b200046c0 con 0x7f3b300742d0 2024-04-23T14:52:17.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.426+0000 7f3b393d3740 1 -- start start 2024-04-23T14:52:17.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.427+0000 7f3b393d3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55990205e860 0x55990205ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:17.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.427+0000 7f3b393d3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55990205f180 0x559902069310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:17.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.427+0000 7f3b393d3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559902069850 0x55990206bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:17.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.427+0000 7f3b393d3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559901fc41f0 con 0x559902069850 2024-04-23T14:52:17.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.427+0000 7f3b393d3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559901f6a350 con 0x55990205e860 2024-04-23T14:52:17.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.427+0000 7f3b393d3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559901fc01b0 con 0x55990205f180 2024-04-23T14:52:17.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.427+0000 7f3b2f400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55990205f180 0x559902069310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:17.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.427+0000 7f3b2fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55990205e860 0x55990205ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:17.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.427+0000 7f3b2f400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55990205f180 0x559902069310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:56742/0 (socket says 172.21.15.28:56742) 2024-04-23T14:52:17.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.427+0000 7f3b2fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55990205e860 0x55990205ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:54618/0 (socket says 172.21.15.28:54618) 2024-04-23T14:52:17.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.427+0000 7f3b2f400640 1 -- 172.21.15.28:0/1945307445 learned_addr learned my addr 172.21.15.28:0/1945307445 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:17.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.427+0000 7f3b2ea00640 1 --2- 172.21.15.28:0/1945307445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559902069850 0x55990206bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:17.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.428+0000 7f3b2fe00640 1 -- 172.21.15.28:0/1945307445 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55990205f180 msgr2=0x559902069310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:17.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.428+0000 7f3b2fe00640 1 --2- 172.21.15.28:0/1945307445 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55990205f180 0x559902069310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.428+0000 7f3b2fe00640 1 -- 172.21.15.28:0/1945307445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559902069850 msgr2=0x55990206bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:17.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.428+0000 7f3b2fe00640 1 --2- 172.21.15.28:0/1945307445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559902069850 0x55990206bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.428+0000 7f3b2fe00640 1 -- 172.21.15.28:0/1945307445 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559901fbef50 con 0x55990205e860 2024-04-23T14:52:17.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.428+0000 7f3b2ea00640 1 --2- 172.21.15.28:0/1945307445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559902069850 0x55990206bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:52:17.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.428+0000 7f3b2f400640 1 --2- 172.21.15.28:0/1945307445 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55990205f180 0x559902069310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:52:17.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.428+0000 7f3b2fe00640 1 --2- 172.21.15.28:0/1945307445 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55990205e860 0x55990205ec40 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3b28005fd0 tx=0x7f3b280107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=755d6370b6e3524c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:17.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.428+0000 7f3b27e00640 1 --2- 172.21.15.28:0/46909004 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f3b140a3d20 0x7f3b140a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:17.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.429+0000 7f3b27e00640 1 -- 172.21.15.28:0/46909004 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7f3b200d0c60 con 0x7f3b300742d0 2024-04-23T14:52:17.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.429+0000 7f3b2f400640 1 --2- 172.21.15.28:0/46909004 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f3b140a3d20 0x7f3b140a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:17.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.429+0000 7f3b24c00640 1 -- 172.21.15.28:0/1945307445 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b28004d20 con 0x55990205e860 2024-04-23T14:52:17.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.429+0000 7f3b24c00640 1 -- 172.21.15.28:0/1945307445 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b28015070 con 0x55990205e860 2024-04-23T14:52:17.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.429+0000 7f3b24c00640 1 -- 172.21.15.28:0/1945307445 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b28005020 con 0x55990205e860 2024-04-23T14:52:17.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.430+0000 7f3b393d3740 1 -- 172.21.15.28:0/1945307445 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55990205e860 msgr2=0x55990205ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:17.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.430+0000 7f3b393d3740 1 --2- 172.21.15.28:0/1945307445 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55990205e860 0x55990205ec40 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3b28005fd0 tx=0x7f3b280107a0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.430+0000 7f3b2f400640 1 --2- 172.21.15.28:0/46909004 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f3b140a3d20 0x7f3b140a61e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3b2000c6a0 tx=0x7f3b200d4710 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:17.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.430+0000 7f3b393d3740 1 -- 172.21.15.28:0/1945307445 shutdown_connections 2024-04-23T14:52:17.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.430+0000 7f3b393d3740 1 --2- 172.21.15.28:0/1945307445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559902069850 0x55990206bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.430+0000 7f3b393d3740 1 --2- 172.21.15.28:0/1945307445 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55990205f180 0x559902069310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.430+0000 7f3b393d3740 1 --2- 172.21.15.28:0/1945307445 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55990205e860 0x55990205ec40 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.430+0000 7f3b393d3740 1 -- 172.21.15.28:0/1945307445 >> 172.21.15.28:0/1945307445 conn(0x55990205ab80 msgr2=0x55990205af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:17.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.431+0000 7f3b393d3740 1 -- 172.21.15.28:0/1945307445 shutdown_connections 2024-04-23T14:52:17.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.431+0000 7f3b393d3740 1 -- 172.21.15.28:0/1945307445 wait complete. 2024-04-23T14:52:17.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.431+0000 7f3b393d3740 1 -- start start 2024-04-23T14:52:17.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.432+0000 7f3b393d3740 1 Processor -- start 2024-04-23T14:52:17.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.433+0000 7f3b393d3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55990205e860 0x559902066870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:17.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.433+0000 7f3b393d3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55990205f180 0x559902066db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:17.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.433+0000 7f3b393d3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559902069850 0x5599020672f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:17.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.433+0000 7f3b393d3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55990206c640 con 0x55990205f180 2024-04-23T14:52:17.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.433+0000 7f3b393d3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559901f7d690 con 0x559902069850 2024-04-23T14:52:17.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.433+0000 7f3b393d3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55990206c7c0 con 0x55990205e860 2024-04-23T14:52:17.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.433+0000 7f3b393d3740 10 client.0 _ll_register_callbacks cb 0x559901fb9650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:52:17.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.433+0000 7f3b2fe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55990205e860 0x559902066870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:17.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.433+0000 7f3b2fe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55990205e860 0x559902066870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:56750/0 (socket says 172.21.15.28:56750) 2024-04-23T14:52:17.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.433+0000 7f3b2fe00640 1 -- 172.21.15.28:0/2655172910 learned_addr learned my addr 172.21.15.28:0/2655172910 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:17.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.433+0000 7f3b2fe00640 1 --2- 172.21.15.28:0/2655172910 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559902069850 0x5599020672f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:17.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.433+0000 7f3b2ea00640 1 --2- 172.21.15.28:0/2655172910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55990205f180 0x559902066db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:17.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.433+0000 7f3b2fe00640 1 -- 172.21.15.28:0/2655172910 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559902069850 msgr2=0x5599020672f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:17.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.434+0000 7f3b2fe00640 1 --2- 172.21.15.28:0/2655172910 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559902069850 0x5599020672f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.434+0000 7f3b2fe00640 1 -- 172.21.15.28:0/2655172910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55990205f180 msgr2=0x559902066db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:17.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.434+0000 7f3b2fe00640 1 --2- 172.21.15.28:0/2655172910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55990205f180 0x559902066db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.434+0000 7f3b2fe00640 1 -- 172.21.15.28:0/2655172910 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559901f7c920 con 0x55990205e860 2024-04-23T14:52:17.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.434+0000 7f3b2fe00640 1 --2- 172.21.15.28:0/2655172910 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55990205e860 0x559902066870 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3b28009c50 tx=0x7f3b28004590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=600f752a2d6e06eb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:17.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.434+0000 7f3b19600640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:52:17.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.434+0000 7f3b19600640 1 -- 172.21.15.28:0/2655172910 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b28004170 con 0x55990205e860 2024-04-23T14:52:17.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.434+0000 7f3b19600640 1 -- 172.21.15.28:0/2655172910 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7f3b28003970 con 0x55990205e860 2024-04-23T14:52:17.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.435+0000 7f3b393d3740 1 -- 172.21.15.28:0/2655172910 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b28015070 con 0x55990205e860 2024-04-23T14:52:17.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.435+0000 7f3b19600640 1 -- 172.21.15.28:0/2655172910 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f3b28003b70 con 0x55990205e860 2024-04-23T14:52:17.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.435+0000 7f3b393d3740 10 client.26440 fetch_fsmap learned FSMap version 20 2024-04-23T14:52:17.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.435+0000 7f3b393d3740 1 -- 172.21.15.28:0/2655172910 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x559901da07d0 con 0x55990205e860 2024-04-23T14:52:17.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.435+0000 7f3b19600640 1 -- 172.21.15.28:0/2655172910 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b28015070 con 0x55990205e860 2024-04-23T14:52:17.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.436+0000 7f3b19600640 1 -- 172.21.15.28:0/2655172910 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f3b28033480 con 0x55990205e860 2024-04-23T14:52:17.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.436+0000 7f3b393d3740 10 client.26440 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:52:17.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.436+0000 7f3b393d3740 10 client.26440 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:52:17.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.436+0000 7f3b393d3740 20 client.26440 populate_metadata read hostname 'smithi028' 2024-04-23T14:52:17.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.436+0000 7f3b393d3740 1 --2- 172.21.15.28:0/2655172910 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x559902079b50 0x55990207bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:17.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.436+0000 7f3b393d3740 4 client.26440 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:52:17.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.436+0000 7f3b393d3740 1 -- 172.21.15.28:0/2655172910 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559901fd1c10 con 0x559902079b50 2024-04-23T14:52:17.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.436+0000 7f3b2ea00640 1 --2- 172.21.15.28:0/2655172910 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x559902079b50 0x55990207bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:17.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.437+0000 7f3b2ea00640 1 --2- 172.21.15.28:0/2655172910 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x559902079b50 0x55990207bf40 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b766f38bb265d3c9 server_cookie=1e3158cffab67cbf in_seq=0 out_seq=0 2024-04-23T14:52:17.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.437+0000 7f3b19600640 10 client.26440 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:17.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.438+0000 7f3b19600640 1 -- 172.21.15.28:0/2655172910 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559901fd1c10 con 0x559902079b50 2024-04-23T14:52:17.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.438+0000 7f3b19600640 10 client.26440 handle_command_reply: tid=0 2024-04-23T14:52:17.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.438+0000 7f3b393d3740 1 client.26440 shutdown 2024-04-23T14:52:17.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.438+0000 7f3b393d3740 1 -- 172.21.15.28:0/2655172910 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55990205e860 msgr2=0x559902066870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:17.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.438+0000 7f3b393d3740 1 --2- 172.21.15.28:0/2655172910 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55990205e860 0x559902066870 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3b28009c50 tx=0x7f3b28004590 comp rx=0 tx=0).stop 2024-04-23T14:52:17.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.438+0000 7f3b393d3740 1 -- 172.21.15.28:0/2655172910 shutdown_connections 2024-04-23T14:52:17.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.438+0000 7f3b393d3740 1 --2- 172.21.15.28:0/2655172910 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x559902079b50 0x55990207bf40 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.438+0000 7f3b393d3740 1 --2- 172.21.15.28:0/2655172910 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559902069850 0x5599020672f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.438+0000 7f3b393d3740 1 --2- 172.21.15.28:0/2655172910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55990205f180 0x559902066db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.438+0000 7f3b19600640 0 client.26440 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:17.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.438+0000 7f3b393d3740 1 --2- 172.21.15.28:0/2655172910 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55990205e860 0x559902066870 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.438+0000 7f3b393d3740 1 -- 172.21.15.28:0/2655172910 >> 172.21.15.28:0/2655172910 conn(0x55990205ab80 msgr2=0x559902060980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:17.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.439+0000 7f3b393d3740 1 -- 172.21.15.28:0/2655172910 shutdown_connections 2024-04-23T14:52:17.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.439+0000 7f3b393d3740 1 -- 172.21.15.28:0/2655172910 wait complete. 2024-04-23T14:52:17.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.439+0000 7f3b393d3740 20 client.26440 trim_cache size 0 max 16384 2024-04-23T14:52:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.460+0000 7f3b393d3740 1 Processor -- start 2024-04-23T14:52:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.460+0000 7f3b393d3740 1 -- start start 2024-04-23T14:52:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.460+0000 7f3b393d3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55990205e860 0x559902075500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.460+0000 7f3b393d3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55990205f180 0x559902060fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.460+0000 7f3b393d3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559902069850 0x559902059aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.460+0000 7f3b393d3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3b301b23d0 con 0x559902069850 2024-04-23T14:52:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.460+0000 7f3b393d3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3b301b4c30 con 0x55990205f180 2024-04-23T14:52:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.460+0000 7f3b393d3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3b301b2a20 con 0x55990205e860 2024-04-23T14:52:17.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.460+0000 7f3b2ea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55990205f180 0x559902060fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:17.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.460+0000 7f3b2fe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55990205e860 0x559902075500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:17.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.460+0000 7f3b2ea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55990205f180 0x559902060fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:54642/0 (socket says 172.21.15.28:54642) 2024-04-23T14:52:17.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.460+0000 7f3b2ea00640 1 -- 172.21.15.28:0/4031380860 learned_addr learned my addr 172.21.15.28:0/4031380860 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:17.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.461+0000 7f3b2fe00640 1 --2- 172.21.15.28:0/4031380860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559902069850 0x559902059aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:17.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.461+0000 7f3b2ea00640 1 -- 172.21.15.28:0/4031380860 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55990205e860 msgr2=0x559902075500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:17.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.461+0000 7f3b2ea00640 1 --2- 172.21.15.28:0/4031380860 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55990205e860 0x559902075500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.461+0000 7f3b2ea00640 1 -- 172.21.15.28:0/4031380860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559902069850 msgr2=0x559902059aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:17.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.461+0000 7f3b2ea00640 1 --2- 172.21.15.28:0/4031380860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559902069850 0x559902059aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.461+0000 7f3b2ea00640 1 -- 172.21.15.28:0/4031380860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559901cd63f0 con 0x55990205f180 2024-04-23T14:52:17.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.461+0000 7f3b2ea00640 1 --2- 172.21.15.28:0/4031380860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55990205f180 0x559902060fc0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c009230 tx=0x7f3b1c005dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=130711061dc152e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:17.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.462+0000 7f3b1be00640 1 -- 172.21.15.28:0/4031380860 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c035040 con 0x55990205f180 2024-04-23T14:52:17.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.462+0000 7f3b1be00640 1 -- 172.21.15.28:0/4031380860 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b1c04a070 con 0x55990205f180 2024-04-23T14:52:17.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.462+0000 7f3b1be00640 1 -- 172.21.15.28:0/4031380860 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c005620 con 0x55990205f180 2024-04-23T14:52:17.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.463+0000 7f3b393d3740 1 -- 172.21.15.28:0/4031380860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55990205f180 msgr2=0x559902060fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:17.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.463+0000 7f3b393d3740 1 --2- 172.21.15.28:0/4031380860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55990205f180 0x559902060fc0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c009230 tx=0x7f3b1c005dd0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.463+0000 7f3b393d3740 1 -- 172.21.15.28:0/4031380860 shutdown_connections 2024-04-23T14:52:17.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.463+0000 7f3b393d3740 1 --2- 172.21.15.28:0/4031380860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55990205f180 0x559902060fc0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.463+0000 7f3b393d3740 1 --2- 172.21.15.28:0/4031380860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559902069850 0x559902059aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.463+0000 7f3b393d3740 1 --2- 172.21.15.28:0/4031380860 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55990205e860 0x559902075500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.463+0000 7f3b393d3740 1 -- 172.21.15.28:0/4031380860 >> 172.21.15.28:0/4031380860 conn(0x55990205ab80 msgr2=0x559902059390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:17.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.463+0000 7f3b393d3740 1 -- 172.21.15.28:0/4031380860 shutdown_connections 2024-04-23T14:52:17.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.463+0000 7f3b393d3740 1 -- 172.21.15.28:0/4031380860 wait complete. 2024-04-23T14:52:17.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.463+0000 7f3b393d3740 1 -- start start 2024-04-23T14:52:17.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.464+0000 7f3b393d3740 1 Processor -- start 2024-04-23T14:52:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.464+0000 7f3b393d3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55990205e860 0x559902071c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.464+0000 7f3b393d3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55990205f180 0x55990206cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.464+0000 7f3b393d3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559902069850 0x55990206d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.464+0000 7f3b393d3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3b301b30f0 con 0x55990205e860 2024-04-23T14:52:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.464+0000 7f3b393d3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3b301b4560 con 0x559902069850 2024-04-23T14:52:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.464+0000 7f3b393d3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3b301b37c0 con 0x55990205f180 2024-04-23T14:52:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.464+0000 7f3b393d3740 10 client.0 _ll_register_callbacks cb 0x7f3b200d45e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:52:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.465+0000 7f3b2fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55990205e860 0x559902071c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.465+0000 7f3b2fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55990205e860 0x559902071c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46186/0 (socket says 172.21.15.28:46186) 2024-04-23T14:52:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.465+0000 7f3b2fe00640 1 -- 172.21.15.28:0/3751624660 learned_addr learned my addr 172.21.15.28:0/3751624660 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.465+0000 7f3b2ea00640 1 --2- 172.21.15.28:0/3751624660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55990205f180 0x55990206cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.465+0000 7f3b2fe00640 1 --2- 172.21.15.28:0/3751624660 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559902069850 0x55990206d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.465+0000 7f3b2ea00640 1 -- 172.21.15.28:0/3751624660 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559902069850 msgr2=0x55990206d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.465+0000 7f3b2ea00640 1 --2- 172.21.15.28:0/3751624660 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559902069850 0x55990206d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.465+0000 7f3b2ea00640 1 -- 172.21.15.28:0/3751624660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55990205e860 msgr2=0x559902071c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.465+0000 7f3b2ea00640 1 --2- 172.21.15.28:0/3751624660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55990205e860 0x559902071c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.465+0000 7f3b2ea00640 1 -- 172.21.15.28:0/3751624660 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559901c960f0 con 0x55990205f180 2024-04-23T14:52:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.465+0000 7f3b2fe00640 1 --2- 172.21.15.28:0/3751624660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55990205e860 0x559902071c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:52:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.466+0000 7f3b2ea00640 1 --2- 172.21.15.28:0/3751624660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55990205f180 0x55990206cfd0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x55990205a7e0 tx=0x7f3b1c02a480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4d72c73a1f55eb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.466+0000 7f3b19600640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:52:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.466+0000 7f3b19600640 1 -- 172.21.15.28:0/3751624660 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c052030 con 0x55990205f180 2024-04-23T14:52:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.466+0000 7f3b19600640 1 -- 172.21.15.28:0/3751624660 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7f3b1c0053a0 con 0x55990205f180 2024-04-23T14:52:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.466+0000 7f3b393d3740 1 -- 172.21.15.28:0/3751624660 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b1c04a070 con 0x55990205f180 2024-04-23T14:52:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.466+0000 7f3b19600640 1 -- 172.21.15.28:0/3751624660 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f3b1c035050 con 0x55990205f180 2024-04-23T14:52:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.466+0000 7f3b393d3740 10 client.26446 fetch_fsmap learned FSMap version 20 2024-04-23T14:52:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.466+0000 7f3b393d3740 1 -- 172.21.15.28:0/3751624660 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x559901f89810 con 0x55990205f180 2024-04-23T14:52:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.467+0000 7f3b19600640 1 -- 172.21.15.28:0/3751624660 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b1c04a070 con 0x55990205f180 2024-04-23T14:52:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.467+0000 7f3b19600640 1 -- 172.21.15.28:0/3751624660 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f3b1c003790 con 0x55990205f180 2024-04-23T14:52:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.467+0000 7f3b393d3740 10 client.26446 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:52:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.467+0000 7f3b393d3740 10 client.26446 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:52:17.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.467+0000 7f3b393d3740 20 client.26446 populate_metadata read hostname 'smithi028' 2024-04-23T14:52:17.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.467+0000 7f3b393d3740 1 --2- 172.21.15.28:0/3751624660 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5599020dffa0 0x5599020e23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:17.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.467+0000 7f3b393d3740 4 client.26446 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:52:17.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.467+0000 7f3b393d3740 1 -- 172.21.15.28:0/3751624660 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559901fcba80 con 0x5599020dffa0 2024-04-23T14:52:17.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.467+0000 7f3b2fe00640 1 --2- 172.21.15.28:0/3751624660 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5599020dffa0 0x5599020e23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:17.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.468+0000 7f3b2fe00640 1 --2- 172.21.15.28:0/3751624660 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5599020dffa0 0x5599020e23c0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab10b7d700eae5cf server_cookie=835c34a7279aff8f in_seq=0 out_seq=0 2024-04-23T14:52:17.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.468+0000 7f3b19600640 10 client.26446 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:17.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.468+0000 7f3b19600640 1 -- 172.21.15.28:0/3751624660 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x559901fcba80 con 0x5599020dffa0 2024-04-23T14:52:17.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.468+0000 7f3b19600640 10 client.26446 handle_command_reply: tid=0 2024-04-23T14:52:17.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.468+0000 7f3b393d3740 1 client.26446 shutdown 2024-04-23T14:52:17.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.468+0000 7f3b393d3740 1 -- 172.21.15.28:0/3751624660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55990205f180 msgr2=0x55990206cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:17.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.468+0000 7f3b393d3740 1 --2- 172.21.15.28:0/3751624660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55990205f180 0x55990206cfd0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x55990205a7e0 tx=0x7f3b1c02a480 comp rx=0 tx=0).stop 2024-04-23T14:52:17.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.468+0000 7f3b393d3740 1 -- 172.21.15.28:0/3751624660 shutdown_connections 2024-04-23T14:52:17.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.468+0000 7f3b393d3740 1 --2- 172.21.15.28:0/3751624660 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5599020dffa0 0x5599020e23c0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.468+0000 7f3b393d3740 1 --2- 172.21.15.28:0/3751624660 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559902069850 0x55990206d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.468+0000 7f3b393d3740 1 --2- 172.21.15.28:0/3751624660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55990205f180 0x55990206cfd0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.468+0000 7f3b393d3740 1 --2- 172.21.15.28:0/3751624660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55990205e860 0x559902071c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.468+0000 7f3b19600640 0 client.26446 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:17.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.468+0000 7f3b393d3740 1 -- 172.21.15.28:0/3751624660 >> 172.21.15.28:0/3751624660 conn(0x55990205ab80 msgr2=0x5599020670c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:17.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.468+0000 7f3b393d3740 1 -- 172.21.15.28:0/3751624660 shutdown_connections 2024-04-23T14:52:17.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.469+0000 7f3b393d3740 1 -- 172.21.15.28:0/3751624660 wait complete. 2024-04-23T14:52:17.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.469+0000 7f3b393d3740 20 client.26446 trim_cache size 0 max 16384 2024-04-23T14:52:17.471 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:52:17.471 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-23T14:52:17.471 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-23T14:52:17.472 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:52:17.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.469+0000 7f3b36600640 1 -- 172.21.15.28:0/46909004 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f3b140a3d20 msgr2=0x7f3b140a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:17.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.469+0000 7f3b36600640 1 --2- 172.21.15.28:0/46909004 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f3b140a3d20 0x7f3b140a61e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3b2000c6a0 tx=0x7f3b200d4710 comp rx=0 tx=0).stop 2024-04-23T14:52:17.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.469+0000 7f3b36600640 1 -- 172.21.15.28:0/46909004 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b300742d0 msgr2=0x7f3b3019ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:17.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.469+0000 7f3b36600640 1 --2- 172.21.15.28:0/46909004 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b300742d0 0x7f3b3019ef60 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3b2000e9f0 tx=0x7f3b2000edf0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.470+0000 7f3b36600640 1 -- 172.21.15.28:0/46909004 shutdown_connections 2024-04-23T14:52:17.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.470+0000 7f3b36600640 1 --2- 172.21.15.28:0/46909004 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f3b140a3d20 0x7f3b140a61e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.470+0000 7f3b36600640 1 --2- 172.21.15.28:0/46909004 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b301130b0 0x7f3b301a3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.470+0000 7f3b36600640 1 --2- 172.21.15.28:0/46909004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b30074bf0 0x7f3b3019f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.470+0000 7f3b36600640 1 --2- 172.21.15.28:0/46909004 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b300742d0 0x7f3b3019ef60 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.470+0000 7f3b36600640 1 -- 172.21.15.28:0/46909004 >> 172.21.15.28:0/46909004 conn(0x7f3b30079450 msgr2=0x7f3b300fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:17.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.470+0000 7f3b36600640 1 -- 172.21.15.28:0/46909004 shutdown_connections 2024-04-23T14:52:17.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.470+0000 7f3b36600640 1 -- 172.21.15.28:0/46909004 wait complete. 2024-04-23T14:52:17.492 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-23T14:52:17.492 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:52:17.510 INFO:tasks.workunit.client.0.smithi028.stdout: 1 686200 8.98 MB/sec execute 273 sec latency 18.043 ms 2024-04-23T14:52:17.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.699+0000 7f040be00640 1 -- 172.21.15.28:0/3911413061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f040c074430 msgr2=0x7f040c074810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:17.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.699+0000 7f040be00640 1 --2- 172.21.15.28:0/3911413061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f040c074430 0x7f040c074810 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f03fc009900 tx=0x7f03fc02a480 comp rx=0 tx=0).stop 2024-04-23T14:52:17.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.699+0000 7f040be00640 1 -- 172.21.15.28:0/3911413061 shutdown_connections 2024-04-23T14:52:17.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.699+0000 7f040be00640 1 --2- 172.21.15.28:0/3911413061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f040c10e660 0x7f040c110ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.699+0000 7f040be00640 1 --2- 172.21.15.28:0/3911413061 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f040c074d50 0x7f040c10e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.699+0000 7f040be00640 1 --2- 172.21.15.28:0/3911413061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f040c074430 0x7f040c074810 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.699+0000 7f040be00640 1 -- 172.21.15.28:0/3911413061 >> 172.21.15.28:0/3911413061 conn(0x7f040c0fd930 msgr2=0x7f040c0ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:17.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.700+0000 7f040be00640 1 -- 172.21.15.28:0/3911413061 shutdown_connections 2024-04-23T14:52:17.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.700+0000 7f040be00640 1 -- 172.21.15.28:0/3911413061 wait complete. 2024-04-23T14:52:17.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.701+0000 7f040be00640 1 Processor -- start 2024-04-23T14:52:17.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.701+0000 7f040be00640 1 -- start start 2024-04-23T14:52:17.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.701+0000 7f040be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f040c074430 0x7f040c1a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:17.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.701+0000 7f040be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f040c074d50 0x7f040c1a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:17.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.701+0000 7f040be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f040c10e660 0x7f040c19a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:17.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.702+0000 7f040be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f040c114440 con 0x7f040c074d50 2024-04-23T14:52:17.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.702+0000 7f040be00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f040c1142c0 con 0x7f040c10e660 2024-04-23T14:52:17.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.702+0000 7f040be00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f040c1145c0 con 0x7f040c074430 2024-04-23T14:52:17.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.702+0000 7f040aa00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f040c074430 0x7f040c1a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:17.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.702+0000 7f040b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f040c10e660 0x7f040c19a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:17.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.702+0000 7f040aa00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f040c074430 0x7f040c1a0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:56784/0 (socket says 172.21.15.28:56784) 2024-04-23T14:52:17.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.702+0000 7f040b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f040c10e660 0x7f040c19a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:54666/0 (socket says 172.21.15.28:54666) 2024-04-23T14:52:17.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.702+0000 7f040a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f040c074d50 0x7f040c1a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:17.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.702+0000 7f040aa00640 1 -- 172.21.15.28:0/273809091 learned_addr learned my addr 172.21.15.28:0/273809091 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:17.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.702+0000 7f040aa00640 1 -- 172.21.15.28:0/273809091 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f040c10e660 msgr2=0x7f040c19a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:17.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.702+0000 7f040aa00640 1 --2- 172.21.15.28:0/273809091 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f040c10e660 0x7f040c19a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.702+0000 7f040aa00640 1 -- 172.21.15.28:0/273809091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f040c074d50 msgr2=0x7f040c1a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:17.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.702+0000 7f040aa00640 1 --2- 172.21.15.28:0/273809091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f040c074d50 0x7f040c1a0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.702+0000 7f040aa00640 1 -- 172.21.15.28:0/273809091 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f040c19af50 con 0x7f040c074430 2024-04-23T14:52:17.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.702+0000 7f040b400640 1 --2- 172.21.15.28:0/273809091 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f040c10e660 0x7f040c19a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:52:17.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.703+0000 7f040aa00640 1 --2- 172.21.15.28:0/273809091 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f040c074430 0x7f040c1a0610 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f03fc004770 tx=0x7f03fc003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:17.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.703+0000 7f0403400640 1 -- 172.21.15.28:0/273809091 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03fc03f070 con 0x7f040c074430 2024-04-23T14:52:17.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.703+0000 7f0403400640 1 -- 172.21.15.28:0/273809091 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03fc031ca0 con 0x7f040c074430 2024-04-23T14:52:17.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.703+0000 7f040be00640 1 -- 172.21.15.28:0/273809091 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f040c19b1e0 con 0x7f040c074430 2024-04-23T14:52:17.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.703+0000 7f0403400640 1 -- 172.21.15.28:0/273809091 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03fc035030 con 0x7f040c074430 2024-04-23T14:52:17.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.703+0000 7f040be00640 1 -- 172.21.15.28:0/273809091 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f040c1a7460 con 0x7f040c074430 2024-04-23T14:52:17.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.704+0000 7f040be00640 1 -- 172.21.15.28:0/273809091 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f040c075ae0 con 0x7f040c074430 2024-04-23T14:52:17.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.706+0000 7f0403400640 1 -- 172.21.15.28:0/273809091 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f03fc042070 con 0x7f040c074430 2024-04-23T14:52:17.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.708+0000 7f0403400640 1 --2- 172.21.15.28:0/273809091 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f03f00a3da0 0x7f03f00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:17.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.708+0000 7f0403400640 1 -- 172.21.15.28:0/273809091 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7f03fc0f1280 con 0x7f040c074430 2024-04-23T14:52:17.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.708+0000 7f040a000640 1 --2- 172.21.15.28:0/273809091 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f03f00a3da0 0x7f03f00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:17.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.709+0000 7f040a000640 1 --2- 172.21.15.28:0/273809091 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f03f00a3da0 0x7f03f00a6260 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f040c101500 tx=0x7f03f8005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:17.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.710+0000 7f0403400640 1 -- 172.21.15.28:0/273809091 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f03fc0bd260 con 0x7f040c074430 2024-04-23T14:52:17.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.981+0000 7f040be00640 1 -- 172.21.15.28:0/273809091 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f040c068d40 con 0x7f040c074430 2024-04-23T14:52:17.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.983+0000 7f0403400640 1 -- 172.21.15.28:0/273809091 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7f03fc0c1560 con 0x7f040c074430 2024-04-23T14:52:17.983 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:52:17.983 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:52:17.983 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:52:17.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.985+0000 7f040be00640 1 -- 172.21.15.28:0/273809091 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f03f00a3da0 msgr2=0x7f03f00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:17.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.986+0000 7f040be00640 1 --2- 172.21.15.28:0/273809091 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f03f00a3da0 0x7f03f00a6260 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f040c101500 tx=0x7f03f8005f90 comp rx=0 tx=0).stop 2024-04-23T14:52:17.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.986+0000 7f040be00640 1 -- 172.21.15.28:0/273809091 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f040c074430 msgr2=0x7f040c1a0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:17.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.986+0000 7f040be00640 1 --2- 172.21.15.28:0/273809091 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f040c074430 0x7f040c1a0610 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f03fc004770 tx=0x7f03fc003c10 comp rx=0 tx=0).stop 2024-04-23T14:52:17.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.986+0000 7f040be00640 1 -- 172.21.15.28:0/273809091 shutdown_connections 2024-04-23T14:52:17.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.986+0000 7f040be00640 1 --2- 172.21.15.28:0/273809091 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f03f00a3da0 0x7f03f00a6260 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.986+0000 7f040be00640 1 --2- 172.21.15.28:0/273809091 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f040c10e660 0x7f040c19a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.986+0000 7f040be00640 1 --2- 172.21.15.28:0/273809091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f040c074d50 0x7f040c1a0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.986+0000 7f040be00640 1 --2- 172.21.15.28:0/273809091 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f040c074430 0x7f040c1a0610 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:17.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.986+0000 7f040be00640 1 -- 172.21.15.28:0/273809091 >> 172.21.15.28:0/273809091 conn(0x7f040c0fd930 msgr2=0x7f040c0fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:17.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.986+0000 7f040be00640 1 -- 172.21.15.28:0/273809091 shutdown_connections 2024-04-23T14:52:17.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:17.986+0000 7f040be00640 1 -- 172.21.15.28:0/273809091 wait complete. 2024-04-23T14:52:18.007 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-23T14:52:18.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.261+0000 7fa263e00640 1 -- 172.21.15.28:0/489377130 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa264104320 msgr2=0x7fa264104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:18.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.261+0000 7fa263e00640 1 --2- 172.21.15.28:0/489377130 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa264104320 0x7fa264104700 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fa254009900 tx=0x7fa25402a3f0 comp rx=0 tx=0).stop 2024-04-23T14:52:18.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.261+0000 7fa263e00640 1 -- 172.21.15.28:0/489377130 shutdown_connections 2024-04-23T14:52:18.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.261+0000 7fa263e00640 1 --2- 172.21.15.28:0/489377130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2641093f0 0x7fa264110fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:18.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.261+0000 7fa263e00640 1 --2- 172.21.15.28:0/489377130 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa264104cd0 0x7fa264108d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:18.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.261+0000 7fa263e00640 1 --2- 172.21.15.28:0/489377130 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa264104320 0x7fa264104700 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:18.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.261+0000 7fa263e00640 1 -- 172.21.15.28:0/489377130 >> 172.21.15.28:0/489377130 conn(0x7fa2640ffb40 msgr2=0x7fa264101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:18.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.261+0000 7fa263e00640 1 -- 172.21.15.28:0/489377130 shutdown_connections 2024-04-23T14:52:18.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.261+0000 7fa263e00640 1 -- 172.21.15.28:0/489377130 wait complete. 2024-04-23T14:52:18.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.263+0000 7fa263e00640 1 Processor -- start 2024-04-23T14:52:18.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.263+0000 7fa263e00640 1 -- start start 2024-04-23T14:52:18.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.263+0000 7fa263e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa264104320 0x7fa2641a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:18.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.263+0000 7fa263e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2641093f0 0x7fa2641a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:18.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.263+0000 7fa263e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa26419a7f0 0x7fa26419aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:18.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.263+0000 7fa263e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa2641144d0 con 0x7fa264104320 2024-04-23T14:52:18.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.263+0000 7fa263e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa264114350 con 0x7fa26419a7f0 2024-04-23T14:52:18.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.263+0000 7fa263e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa264114650 con 0x7fa2641093f0 2024-04-23T14:52:18.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.263+0000 7fa262a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa264104320 0x7fa2641a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:18.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.264+0000 7fa263400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa26419a7f0 0x7fa26419aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:18.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.264+0000 7fa262000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2641093f0 0x7fa2641a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:18.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.264+0000 7fa263400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa26419a7f0 0x7fa26419aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40240/0 (socket says 172.21.15.28:40240) 2024-04-23T14:52:18.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.264+0000 7fa263400640 1 -- 172.21.15.28:0/500203362 learned_addr learned my addr 172.21.15.28:0/500203362 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:18.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.264+0000 7fa262000640 1 -- 172.21.15.28:0/500203362 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa26419a7f0 msgr2=0x7fa26419aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:18.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.264+0000 7fa262000640 1 --2- 172.21.15.28:0/500203362 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa26419a7f0 0x7fa26419aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:18.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.264+0000 7fa262000640 1 -- 172.21.15.28:0/500203362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa264104320 msgr2=0x7fa2641a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:18.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.264+0000 7fa262000640 1 --2- 172.21.15.28:0/500203362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa264104320 0x7fa2641a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:18.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.264+0000 7fa262000640 1 -- 172.21.15.28:0/500203362 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa26419b380 con 0x7fa2641093f0 2024-04-23T14:52:18.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.264+0000 7fa263400640 1 --2- 172.21.15.28:0/500203362 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa26419a7f0 0x7fa26419aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:52:18.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.264+0000 7fa262000640 1 --2- 172.21.15.28:0/500203362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2641093f0 0x7fa2641a0bc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa24c005fd0 tx=0x7fa24c00df40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:18.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.265+0000 7fa25b400640 1 -- 172.21.15.28:0/500203362 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24c017030 con 0x7fa2641093f0 2024-04-23T14:52:18.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.265+0000 7fa25b400640 1 -- 172.21.15.28:0/500203362 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa24c003e50 con 0x7fa2641093f0 2024-04-23T14:52:18.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.265+0000 7fa25b400640 1 -- 172.21.15.28:0/500203362 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24c010460 con 0x7fa2641093f0 2024-04-23T14:52:18.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.265+0000 7fa263e00640 1 -- 172.21.15.28:0/500203362 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa26419b670 con 0x7fa2641093f0 2024-04-23T14:52:18.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.265+0000 7fa263e00640 1 -- 172.21.15.28:0/500203362 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2641a7800 con 0x7fa2641093f0 2024-04-23T14:52:18.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.268+0000 7fa26ca24740 1 Processor -- start 2024-04-23T14:52:18.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.268+0000 7fa26ca24740 1 -- start start 2024-04-23T14:52:18.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.269+0000 7fa26ca24740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55adfac2db40 0x55adfac2df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:18.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.269+0000 7fa26ca24740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfac2e460 0x55adfac385f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:18.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.269+0000 7fa26ca24740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55adfac38b30 0x55adfac3af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:18.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.269+0000 7fa26ca24740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55adfab931d0 con 0x55adfac2e460 2024-04-23T14:52:18.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.269+0000 7fa26ca24740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55adfab39450 con 0x55adfac2db40 2024-04-23T14:52:18.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.269+0000 7fa26ca24740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55adfab8f190 con 0x55adfac38b30 2024-04-23T14:52:18.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.269+0000 7fa25b400640 1 -- 172.21.15.28:0/500203362 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fa24c01a070 con 0x7fa2641093f0 2024-04-23T14:52:18.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.269+0000 7fa263400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55adfac2db40 0x55adfac2df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:18.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.269+0000 7fa263400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55adfac2db40 0x55adfac2df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40252/0 (socket says 172.21.15.28:40252) 2024-04-23T14:52:18.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.269+0000 7fa263400640 1 -- 172.21.15.28:0/2843734886 learned_addr learned my addr 172.21.15.28:0/2843734886 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:18.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.269+0000 7fa262a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfac2e460 0x55adfac385f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:18.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.269+0000 7fa263400640 1 -- 172.21.15.28:0/2843734886 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55adfac38b30 msgr2=0x55adfac3af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:18.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.269+0000 7fa263400640 1 --2- 172.21.15.28:0/2843734886 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55adfac38b30 0x55adfac3af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:18.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.269+0000 7fa263400640 1 -- 172.21.15.28:0/2843734886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfac2e460 msgr2=0x55adfac385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:18.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.269+0000 7fa263400640 1 --2- 172.21.15.28:0/2843734886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfac2e460 0x55adfac385f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:18.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.269+0000 7fa263400640 1 -- 172.21.15.28:0/2843734886 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adfabf6310 con 0x55adfac2db40 2024-04-23T14:52:18.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.270+0000 7fa263400640 1 --2- 172.21.15.28:0/2843734886 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55adfac2db40 0x55adfac2df20 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa25c0104d0 tx=0x7fa25c0108d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c298cac16feeb6e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:18.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.270+0000 7fa253e00640 1 -- 172.21.15.28:0/2843734886 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa25c011760 con 0x55adfac2db40 2024-04-23T14:52:18.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.270+0000 7fa253e00640 1 -- 172.21.15.28:0/2843734886 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa25c011de0 con 0x55adfac2db40 2024-04-23T14:52:18.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.270+0000 7fa253e00640 1 -- 172.21.15.28:0/2843734886 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa25c018380 con 0x55adfac2db40 2024-04-23T14:52:18.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.271+0000 7fa25b400640 1 --2- 172.21.15.28:0/500203362 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa2440a3d20 0x7fa2440a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:18.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.271+0000 7fa26ca24740 1 -- 172.21.15.28:0/2843734886 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55adfac2db40 msgr2=0x55adfac2df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:18.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.271+0000 7fa26ca24740 1 --2- 172.21.15.28:0/2843734886 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55adfac2db40 0x55adfac2df20 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa25c0104d0 tx=0x7fa25c0108d0 comp rx=0 tx=0).stop 2024-04-23T14:52:18.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.271+0000 7fa25b400640 1 -- 172.21.15.28:0/500203362 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7fa24c0d05b0 con 0x7fa2641093f0 2024-04-23T14:52:18.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.271+0000 7fa26ca24740 1 -- 172.21.15.28:0/2843734886 shutdown_connections 2024-04-23T14:52:18.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.271+0000 7fa26ca24740 1 --2- 172.21.15.28:0/2843734886 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55adfac38b30 0x55adfac3af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:18.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.271+0000 7fa26ca24740 1 --2- 172.21.15.28:0/2843734886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfac2e460 0x55adfac385f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:18.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.271+0000 7fa26ca24740 1 --2- 172.21.15.28:0/2843734886 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55adfac2db40 0x55adfac2df20 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:18.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.271+0000 7fa26ca24740 1 -- 172.21.15.28:0/2843734886 >> 172.21.15.28:0/2843734886 conn(0x55adfac29e60 msgr2=0x55adfac2a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:18.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.271+0000 7fa262a00640 1 --2- 172.21.15.28:0/500203362 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa2440a3d20 0x7fa2440a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:18.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.271+0000 7fa26ca24740 1 -- 172.21.15.28:0/2843734886 shutdown_connections 2024-04-23T14:52:18.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.271+0000 7fa26ca24740 1 -- 172.21.15.28:0/2843734886 wait complete. 2024-04-23T14:52:18.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.272+0000 7fa26ca24740 1 -- start start 2024-04-23T14:52:18.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.272+0000 7fa262a00640 1 --2- 172.21.15.28:0/500203362 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa2440a3d20 0x7fa2440a61e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa254000c00 tx=0x7fa254005c00 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:18.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.272+0000 7fa26ca24740 1 Processor -- start 2024-04-23T14:52:18.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.272+0000 7fa26ca24740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfac2db40 0x55adfac35910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:18.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.272+0000 7fa26ca24740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55adfac2e460 0x55adfac35e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:18.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.272+0000 7fa26ca24740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55adfac38b30 0x55adfac36390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:18.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.272+0000 7fa26ca24740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55adfac3b9c0 con 0x55adfac2db40 2024-04-23T14:52:18.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.272+0000 7fa26ca24740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55adfab4c810 con 0x55adfac2e460 2024-04-23T14:52:18.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.272+0000 7fa26ca24740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55adfac3bb40 con 0x55adfac38b30 2024-04-23T14:52:18.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.273+0000 7fa26ca24740 10 client.0 _ll_register_callbacks cb 0x55adfab1a140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:52:18.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.273+0000 7fa262a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfac2db40 0x55adfac35910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:18.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.273+0000 7fa262000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55adfac2e460 0x55adfac35e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:18.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.273+0000 7fa262a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfac2db40 0x55adfac35910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52908/0 (socket says 172.21.15.28:52908) 2024-04-23T14:52:18.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.273+0000 7fa262000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55adfac2e460 0x55adfac35e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40260/0 (socket says 172.21.15.28:40260) 2024-04-23T14:52:18.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.273+0000 7fa262a00640 1 -- 172.21.15.28:0/1612469884 learned_addr learned my addr 172.21.15.28:0/1612469884 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:18.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.273+0000 7fa263400640 1 --2- 172.21.15.28:0/1612469884 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55adfac38b30 0x55adfac36390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:18.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.273+0000 7fa262000640 1 -- 172.21.15.28:0/1612469884 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55adfac38b30 msgr2=0x55adfac36390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:18.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.273+0000 7fa262000640 1 --2- 172.21.15.28:0/1612469884 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55adfac38b30 0x55adfac36390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:18.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.273+0000 7fa262000640 1 -- 172.21.15.28:0/1612469884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfac2db40 msgr2=0x55adfac35910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:18.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.273+0000 7fa262000640 1 --2- 172.21.15.28:0/1612469884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfac2db40 0x55adfac35910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:18.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.273+0000 7fa262000640 1 -- 172.21.15.28:0/1612469884 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adfab5d3f0 con 0x55adfac2e460 2024-04-23T14:52:18.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.273+0000 7fa262000640 1 --2- 172.21.15.28:0/1612469884 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55adfac2e460 0x55adfac35e50 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa24c0cdc90 tx=0x7fa24c09c110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5207979f98c9cd58 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:18.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.274+0000 7fa250c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:52:18.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.274+0000 7fa250c00640 1 -- 172.21.15.28:0/1612469884 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24c0a0890 con 0x55adfac2e460 2024-04-23T14:52:18.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.274+0000 7fa250c00640 1 -- 172.21.15.28:0/1612469884 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7fa24c00bc30 con 0x55adfac2e460 2024-04-23T14:52:18.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.274+0000 7fa26ca24740 1 -- 172.21.15.28:0/1612469884 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa25c011de0 con 0x55adfac2e460 2024-04-23T14:52:18.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.274+0000 7fa250c00640 1 -- 172.21.15.28:0/1612469884 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fa24c0a5150 con 0x55adfac2e460 2024-04-23T14:52:18.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.274+0000 7fa26ca24740 10 client.26480 fetch_fsmap learned FSMap version 20 2024-04-23T14:52:18.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.274+0000 7fa26ca24740 1 -- 172.21.15.28:0/1612469884 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55adfa8a53f0 con 0x55adfac2e460 2024-04-23T14:52:18.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.274+0000 7fa250c00640 1 -- 172.21.15.28:0/1612469884 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa25c011de0 con 0x55adfac2e460 2024-04-23T14:52:18.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.274+0000 7fa250c00640 1 -- 172.21.15.28:0/1612469884 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fa24c024060 con 0x55adfac2e460 2024-04-23T14:52:18.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.275+0000 7fa26ca24740 10 client.26480 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:52:18.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.275+0000 7fa26ca24740 10 client.26480 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:52:18.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.275+0000 7fa26ca24740 20 client.26480 populate_metadata read hostname 'smithi028' 2024-04-23T14:52:18.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.275+0000 7fa26ca24740 1 --2- 172.21.15.28:0/1612469884 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55adfac48da0 0x55adfac4b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:18.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.275+0000 7fa26ca24740 4 client.26480 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:52:18.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.275+0000 7fa26ca24740 1 -- 172.21.15.28:0/1612469884 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55adfaba0bf0 con 0x55adfac48da0 2024-04-23T14:52:18.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.275+0000 7fa262a00640 1 --2- 172.21.15.28:0/1612469884 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55adfac48da0 0x55adfac4b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:18.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.275+0000 7fa262a00640 1 --2- 172.21.15.28:0/1612469884 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55adfac48da0 0x55adfac4b190 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a08a2cbe1cd8dac9 server_cookie=45ae593adc10087b in_seq=0 out_seq=0 2024-04-23T14:52:18.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.275+0000 7fa250c00640 10 client.26480 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:18.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.276+0000 7fa250c00640 1 -- 172.21.15.28:0/1612469884 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55adfaba0bf0 con 0x55adfac48da0 2024-04-23T14:52:18.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.276+0000 7fa250c00640 10 client.26480 handle_command_reply: tid=0 2024-04-23T14:52:18.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.276+0000 7fa26ca24740 1 client.26480 shutdown 2024-04-23T14:52:18.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.277+0000 7fa26ca24740 1 -- 172.21.15.28:0/1612469884 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55adfac2e460 msgr2=0x55adfac35e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:18.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.277+0000 7fa26ca24740 1 --2- 172.21.15.28:0/1612469884 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55adfac2e460 0x55adfac35e50 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa24c0cdc90 tx=0x7fa24c09c110 comp rx=0 tx=0).stop 2024-04-23T14:52:18.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.277+0000 7fa26ca24740 1 -- 172.21.15.28:0/1612469884 shutdown_connections 2024-04-23T14:52:18.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.277+0000 7fa26ca24740 1 --2- 172.21.15.28:0/1612469884 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55adfac48da0 0x55adfac4b190 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:18.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.277+0000 7fa26ca24740 1 --2- 172.21.15.28:0/1612469884 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55adfac2e460 0x55adfac35e50 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:18.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.277+0000 7fa26ca24740 1 --2- 172.21.15.28:0/1612469884 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55adfac38b30 0x55adfac36390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:18.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.277+0000 7fa250c00640 0 client.26480 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:18.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.277+0000 7fa26ca24740 1 --2- 172.21.15.28:0/1612469884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfac2db40 0x55adfac35910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:18.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.277+0000 7fa26ca24740 1 -- 172.21.15.28:0/1612469884 >> 172.21.15.28:0/1612469884 conn(0x55adfac29e60 msgr2=0x55adfac2fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:18.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.277+0000 7fa26ca24740 1 -- 172.21.15.28:0/1612469884 shutdown_connections 2024-04-23T14:52:18.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.277+0000 7fa26ca24740 1 -- 172.21.15.28:0/1612469884 wait complete. 2024-04-23T14:52:18.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.277+0000 7fa26ca24740 20 client.26480 trim_cache size 0 max 16384 2024-04-23T14:52:18.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.292+0000 7fa26ca24740 1 Processor -- start 2024-04-23T14:52:18.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.293+0000 7fa26ca24740 1 -- start start 2024-04-23T14:52:18.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.293+0000 7fa26ca24740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfac2db40 0x55adfac44880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:18.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.293+0000 7fa26ca24740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55adfac2e460 0x55adfac50950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:18.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.293+0000 7fa26ca24740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55adfac38b30 0x55adfac50e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:18.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.293+0000 7fa26ca24740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa2641ade80 con 0x55adfac2db40 2024-04-23T14:52:18.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.293+0000 7fa26ca24740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa2641b06e0 con 0x55adfac38b30 2024-04-23T14:52:18.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.293+0000 7fa26ca24740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa2641ae4d0 con 0x55adfac2e460 2024-04-23T14:52:18.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.293+0000 7fa262a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfac2db40 0x55adfac44880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:18.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.293+0000 7fa262a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfac2db40 0x55adfac44880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52914/0 (socket says 172.21.15.28:52914) 2024-04-23T14:52:18.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.293+0000 7fa262a00640 1 -- 172.21.15.28:0/310370919 learned_addr learned my addr 172.21.15.28:0/310370919 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:18.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.293+0000 7fa263400640 1 --2- 172.21.15.28:0/310370919 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55adfac38b30 0x55adfac50e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:18.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.293+0000 7fa262000640 1 --2- 172.21.15.28:0/310370919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55adfac2e460 0x55adfac50950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:18.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.294+0000 7fa262a00640 1 -- 172.21.15.28:0/310370919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55adfac2e460 msgr2=0x55adfac50950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:18.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.294+0000 7fa262a00640 1 --2- 172.21.15.28:0/310370919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55adfac2e460 0x55adfac50950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:18.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.294+0000 7fa262a00640 1 -- 172.21.15.28:0/310370919 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55adfac38b30 msgr2=0x55adfac50e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:18.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.294+0000 7fa262a00640 1 --2- 172.21.15.28:0/310370919 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55adfac38b30 0x55adfac50e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:18.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.294+0000 7fa262a00640 1 -- 172.21.15.28:0/310370919 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adfa96f7d0 con 0x55adfac2db40 2024-04-23T14:52:18.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.294+0000 7fa262a00640 1 --2- 172.21.15.28:0/310370919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfac2db40 0x55adfac44880 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa254006010 tx=0x7fa25403f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=731544b526507b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:18.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.294+0000 7fa253400640 1 -- 172.21.15.28:0/310370919 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa254035040 con 0x55adfac2db40 2024-04-23T14:52:18.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.294+0000 7fa253400640 1 -- 172.21.15.28:0/310370919 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa25405a070 con 0x55adfac2db40 2024-04-23T14:52:18.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.294+0000 7fa253400640 1 -- 172.21.15.28:0/310370919 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa254032820 con 0x55adfac2db40 2024-04-23T14:52:18.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.295+0000 7fa26ca24740 1 -- 172.21.15.28:0/310370919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfac2db40 msgr2=0x55adfac44880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:18.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.295+0000 7fa26ca24740 1 --2- 172.21.15.28:0/310370919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfac2db40 0x55adfac44880 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa254006010 tx=0x7fa25403f000 comp rx=0 tx=0).stop 2024-04-23T14:52:18.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.295+0000 7fa26ca24740 1 -- 172.21.15.28:0/310370919 shutdown_connections 2024-04-23T14:52:18.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.295+0000 7fa26ca24740 1 --2- 172.21.15.28:0/310370919 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55adfac38b30 0x55adfac50e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:18.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.295+0000 7fa26ca24740 1 --2- 172.21.15.28:0/310370919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55adfac2e460 0x55adfac50950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:18.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.295+0000 7fa26ca24740 1 --2- 172.21.15.28:0/310370919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfac2db40 0x55adfac44880 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:18.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.295+0000 7fa26ca24740 1 -- 172.21.15.28:0/310370919 >> 172.21.15.28:0/310370919 conn(0x55adfac29e60 msgr2=0x55adfac28f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:18.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.295+0000 7fa26ca24740 1 -- 172.21.15.28:0/310370919 shutdown_connections 2024-04-23T14:52:18.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.295+0000 7fa26ca24740 1 -- 172.21.15.28:0/310370919 wait complete. 2024-04-23T14:52:18.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.296+0000 7fa26ca24740 1 -- start start 2024-04-23T14:52:18.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.296+0000 7fa26ca24740 1 Processor -- start 2024-04-23T14:52:18.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.296+0000 7fa26ca24740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55adfac2db40 0x55adfac335f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:18.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.296+0000 7fa26ca24740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfac2e460 0x55adfac40d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:18.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.296+0000 7fa26ca24740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55adfac38b30 0x55adfac39d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:18.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.296+0000 7fa26ca24740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa2641aeba0 con 0x55adfac2e460 2024-04-23T14:52:18.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.296+0000 7fa26ca24740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa2641b0010 con 0x55adfac38b30 2024-04-23T14:52:18.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.296+0000 7fa26ca24740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa2641af270 con 0x55adfac2db40 2024-04-23T14:52:18.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.296+0000 7fa26ca24740 10 client.0 _ll_register_callbacks cb 0x7fa24c004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:52:18.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.297+0000 7fa262a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55adfac2db40 0x55adfac335f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:18.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.297+0000 7fa263400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55adfac38b30 0x55adfac39d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:18.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.297+0000 7fa262000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfac2e460 0x55adfac40d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:18.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.297+0000 7fa262a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55adfac2db40 0x55adfac335f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:42092/0 (socket says 172.21.15.28:42092) 2024-04-23T14:52:18.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.297+0000 7fa262a00640 1 -- 172.21.15.28:0/1015624374 learned_addr learned my addr 172.21.15.28:0/1015624374 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:18.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.297+0000 7fa262a00640 1 -- 172.21.15.28:0/1015624374 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55adfac38b30 msgr2=0x55adfac39d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:18.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.297+0000 7fa262a00640 1 --2- 172.21.15.28:0/1015624374 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55adfac38b30 0x55adfac39d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:18.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.297+0000 7fa262a00640 1 -- 172.21.15.28:0/1015624374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfac2e460 msgr2=0x55adfac40d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:18.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.297+0000 7fa262a00640 1 --2- 172.21.15.28:0/1015624374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfac2e460 0x55adfac40d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:18.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.297+0000 7fa262a00640 1 -- 172.21.15.28:0/1015624374 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adfac3a270 con 0x55adfac2db40 2024-04-23T14:52:18.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.297+0000 7fa263400640 1 --2- 172.21.15.28:0/1015624374 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55adfac38b30 0x55adfac39d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:52:18.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.297+0000 7fa262000640 1 --2- 172.21.15.28:0/1015624374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfac2e460 0x55adfac40d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:52:18.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.297+0000 7fa262a00640 1 --2- 172.21.15.28:0/1015624374 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55adfac2db40 0x55adfac335f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa25402a910 tx=0x7fa254009880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9daa5302bbb8f523 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:18.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.297+0000 7fa250c00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:52:18.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.298+0000 7fa250c00640 1 -- 172.21.15.28:0/1015624374 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa254063070 con 0x55adfac2db40 2024-04-23T14:52:18.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.298+0000 7fa250c00640 1 -- 172.21.15.28:0/1015624374 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7fa254004ad0 con 0x55adfac2db40 2024-04-23T14:52:18.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.298+0000 7fa26ca24740 1 -- 172.21.15.28:0/1015624374 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa25405a070 con 0x55adfac2db40 2024-04-23T14:52:18.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.298+0000 7fa250c00640 1 -- 172.21.15.28:0/1015624374 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fa254004ea0 con 0x55adfac2db40 2024-04-23T14:52:18.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.298+0000 7fa26ca24740 10 client.26467 fetch_fsmap learned FSMap version 20 2024-04-23T14:52:18.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.298+0000 7fa26ca24740 1 -- 172.21.15.28:0/1015624374 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55adfacaf840 con 0x55adfac2db40 2024-04-23T14:52:18.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.299+0000 7fa250c00640 1 -- 172.21.15.28:0/1015624374 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa25405a070 con 0x55adfac2db40 2024-04-23T14:52:18.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.299+0000 7fa250c00640 1 -- 172.21.15.28:0/1015624374 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fa2540035f0 con 0x55adfac2db40 2024-04-23T14:52:18.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.299+0000 7fa26ca24740 10 client.26467 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:52:18.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.299+0000 7fa26ca24740 10 client.26467 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:52:18.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.299+0000 7fa26ca24740 20 client.26467 populate_metadata read hostname 'smithi028' 2024-04-23T14:52:18.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.299+0000 7fa26ca24740 1 --2- 172.21.15.28:0/1015624374 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55adfacaff60 0x55adfacb2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:18.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.299+0000 7fa26ca24740 4 client.26467 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-23T14:52:18.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.299+0000 7fa26ca24740 1 -- 172.21.15.28:0/1015624374 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55adfab9aa60 con 0x55adfacaff60 2024-04-23T14:52:18.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.299+0000 7fa262000640 1 --2- 172.21.15.28:0/1015624374 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55adfacaff60 0x55adfacb2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:18.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.300+0000 7fa262000640 1 --2- 172.21.15.28:0/1015624374 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55adfacaff60 0x55adfacb2380 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d8c8588b2f78ae server_cookie=23a48a910078e8c6 in_seq=0 out_seq=0 2024-04-23T14:52:18.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.300+0000 7fa250c00640 10 client.26467 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:18.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.301+0000 7fa250c00640 1 -- 172.21.15.28:0/1015624374 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55adfab9aa60 con 0x55adfacaff60 2024-04-23T14:52:18.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.301+0000 7fa250c00640 10 client.26467 handle_command_reply: tid=0 2024-04-23T14:52:18.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.301+0000 7fa26ca24740 1 client.26467 shutdown 2024-04-23T14:52:18.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.301+0000 7fa26ca24740 1 -- 172.21.15.28:0/1015624374 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55adfac2db40 msgr2=0x55adfac335f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:18.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.301+0000 7fa26ca24740 1 --2- 172.21.15.28:0/1015624374 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55adfac2db40 0x55adfac335f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa25402a910 tx=0x7fa254009880 comp rx=0 tx=0).stop 2024-04-23T14:52:18.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.301+0000 7fa26ca24740 1 -- 172.21.15.28:0/1015624374 shutdown_connections 2024-04-23T14:52:18.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.301+0000 7fa26ca24740 1 --2- 172.21.15.28:0/1015624374 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55adfacaff60 0x55adfacb2380 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:18.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.301+0000 7fa26ca24740 1 --2- 172.21.15.28:0/1015624374 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55adfac38b30 0x55adfac39d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:18.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.301+0000 7fa26ca24740 1 --2- 172.21.15.28:0/1015624374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfac2e460 0x55adfac40d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:18.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.301+0000 7fa26ca24740 1 --2- 172.21.15.28:0/1015624374 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55adfac2db40 0x55adfac335f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:18.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.301+0000 7fa26ca24740 1 -- 172.21.15.28:0/1015624374 >> 172.21.15.28:0/1015624374 conn(0x55adfac29e60 msgr2=0x55adfac358d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:18.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.301+0000 7fa250c00640 0 client.26467 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:18.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.301+0000 7fa26ca24740 1 -- 172.21.15.28:0/1015624374 shutdown_connections 2024-04-23T14:52:18.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.301+0000 7fa26ca24740 1 -- 172.21.15.28:0/1015624374 wait complete. 2024-04-23T14:52:18.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.301+0000 7fa26ca24740 20 client.26467 trim_cache size 0 max 16384 2024-04-23T14:52:18.309 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-23T14:52:18.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.302+0000 7fa263e00640 1 -- 172.21.15.28:0/500203362 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa2440a3d20 msgr2=0x7fa2440a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:18.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.302+0000 7fa263e00640 1 --2- 172.21.15.28:0/500203362 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa2440a3d20 0x7fa2440a61e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa254000c00 tx=0x7fa254005c00 comp rx=0 tx=0).stop 2024-04-23T14:52:18.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.302+0000 7fa263e00640 1 -- 172.21.15.28:0/500203362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2641093f0 msgr2=0x7fa2641a0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:18.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.302+0000 7fa263e00640 1 --2- 172.21.15.28:0/500203362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2641093f0 0x7fa2641a0bc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa24c005fd0 tx=0x7fa24c00df40 comp rx=0 tx=0).stop 2024-04-23T14:52:18.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.302+0000 7fa263e00640 1 -- 172.21.15.28:0/500203362 shutdown_connections 2024-04-23T14:52:18.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.302+0000 7fa263e00640 1 --2- 172.21.15.28:0/500203362 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa2440a3d20 0x7fa2440a61e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:18.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.302+0000 7fa263e00640 1 --2- 172.21.15.28:0/500203362 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa26419a7f0 0x7fa26419aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:18.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.302+0000 7fa263e00640 1 --2- 172.21.15.28:0/500203362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2641093f0 0x7fa2641a0bc0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:18.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.302+0000 7fa263e00640 1 --2- 172.21.15.28:0/500203362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa264104320 0x7fa2641a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:18.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.302+0000 7fa263e00640 1 -- 172.21.15.28:0/500203362 >> 172.21.15.28:0/500203362 conn(0x7fa2640ffb40 msgr2=0x7fa264101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:18.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.302+0000 7fa263e00640 1 -- 172.21.15.28:0/500203362 shutdown_connections 2024-04-23T14:52:18.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:18.302+0000 7fa263e00640 1 -- 172.21.15.28:0/500203362 wait complete. 2024-04-23T14:52:18.510 INFO:tasks.workunit.client.0.smithi028.stdout: 1 687995 8.97 MB/sec execute 274 sec latency 24.315 ms 2024-04-23T14:52:19.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:18 smithi105 ceph-mon[32555]: pgmap v255: 97 pgs: 97 active+clean; 436 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 154 op/s 2024-04-23T14:52:19.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:18 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/273809091' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:52:19.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:18 smithi038 ceph-mon[32348]: pgmap v255: 97 pgs: 97 active+clean; 436 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 154 op/s 2024-04-23T14:52:19.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:18 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/273809091' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:52:19.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:18 smithi028 ceph-mon[27810]: pgmap v255: 97 pgs: 97 active+clean; 436 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 154 op/s 2024-04-23T14:52:19.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:18 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/273809091' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:52:19.321 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-23T14:52:19.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:52:19.321 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-23T14:52:19.510 INFO:tasks.workunit.client.0.smithi028.stdout: 1 689473 8.98 MB/sec execute 275 sec latency 19.530 ms 2024-04-23T14:52:19.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.584+0000 7fdf0da00640 1 -- 172.21.15.28:0/3109816647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf081044b0 msgr2=0x7fdf08104890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:19.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.584+0000 7fdf0da00640 1 --2- 172.21.15.28:0/3109816647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf081044b0 0x7fdf08104890 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fdef8009900 tx=0x7fdef802a3f0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.584+0000 7fdf0da00640 1 -- 172.21.15.28:0/3109816647 shutdown_connections 2024-04-23T14:52:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.584+0000 7fdf0da00640 1 --2- 172.21.15.28:0/3109816647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf08109580 0x7fdf08111150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.584+0000 7fdf0da00640 1 --2- 172.21.15.28:0/3109816647 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf08104e60 0x7fdf08108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.584+0000 7fdf0da00640 1 --2- 172.21.15.28:0/3109816647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf081044b0 0x7fdf08104890 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.584+0000 7fdf0da00640 1 -- 172.21.15.28:0/3109816647 >> 172.21.15.28:0/3109816647 conn(0x7fdf080ffd90 msgr2=0x7fdf081021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.584+0000 7fdf0da00640 1 -- 172.21.15.28:0/3109816647 shutdown_connections 2024-04-23T14:52:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.584+0000 7fdf0da00640 1 -- 172.21.15.28:0/3109816647 wait complete. 2024-04-23T14:52:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.585+0000 7fdf0da00640 1 Processor -- start 2024-04-23T14:52:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.585+0000 7fdf0da00640 1 -- start start 2024-04-23T14:52:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.585+0000 7fdf0da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf081044b0 0x7fdf081a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.585+0000 7fdf0da00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf08109580 0x7fdf081a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.585+0000 7fdf0da00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf0819a9f0 0x7fdf0819aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.585+0000 7fdf0da00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdf08114630 con 0x7fdf081044b0 2024-04-23T14:52:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.585+0000 7fdf0da00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdf081144b0 con 0x7fdf0819a9f0 2024-04-23T14:52:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.585+0000 7fdf0da00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdf081147b0 con 0x7fdf08109580 2024-04-23T14:52:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.586+0000 7fdf06a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf081044b0 0x7fdf081a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.586+0000 7fdeffe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf08109580 0x7fdf081a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.586+0000 7fdf06a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf081044b0 0x7fdf081a0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52940/0 (socket says 172.21.15.28:52940) 2024-04-23T14:52:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.586+0000 7fdf06a00640 1 -- 172.21.15.28:0/2603296434 learned_addr learned my addr 172.21.15.28:0/2603296434 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.586+0000 7fdf07400640 1 --2- 172.21.15.28:0/2603296434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf0819a9f0 0x7fdf0819aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.586+0000 7fdeffe00640 1 -- 172.21.15.28:0/2603296434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf0819a9f0 msgr2=0x7fdf0819aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.586+0000 7fdeffe00640 1 --2- 172.21.15.28:0/2603296434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf0819a9f0 0x7fdf0819aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.586+0000 7fdeffe00640 1 -- 172.21.15.28:0/2603296434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf081044b0 msgr2=0x7fdf081a0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.586+0000 7fdeffe00640 1 --2- 172.21.15.28:0/2603296434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf081044b0 0x7fdf081a0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.586+0000 7fdeffe00640 1 -- 172.21.15.28:0/2603296434 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf0819b510 con 0x7fdf08109580 2024-04-23T14:52:19.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.586+0000 7fdeffe00640 1 --2- 172.21.15.28:0/2603296434 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf08109580 0x7fdf081a0d80 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fdef000e9c0 tx=0x7fdef000edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:19.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.587+0000 7fdeff400640 1 -- 172.21.15.28:0/2603296434 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef000ccc0 con 0x7fdf08109580 2024-04-23T14:52:19.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.587+0000 7fdf0da00640 1 -- 172.21.15.28:0/2603296434 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf0819b800 con 0x7fdf08109580 2024-04-23T14:52:19.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.587+0000 7fdeff400640 1 -- 172.21.15.28:0/2603296434 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdef0005020 con 0x7fdf08109580 2024-04-23T14:52:19.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.587+0000 7fdeff400640 1 -- 172.21.15.28:0/2603296434 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef0004a00 con 0x7fdf08109580 2024-04-23T14:52:19.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.587+0000 7fdf0da00640 1 -- 172.21.15.28:0/2603296434 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf081a7a50 con 0x7fdf08109580 2024-04-23T14:52:19.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.590+0000 7fdf107cc740 1 Processor -- start 2024-04-23T14:52:19.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.590+0000 7fdeff400640 1 -- 172.21.15.28:0/2603296434 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fdef00051d0 con 0x7fdf08109580 2024-04-23T14:52:19.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.590+0000 7fdf107cc740 1 -- start start 2024-04-23T14:52:19.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.591+0000 7fdf107cc740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558b02001090 0x558b02001470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.591+0000 7fdf107cc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b020019b0 0x558b0200bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.591+0000 7fdf107cc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b0200c080 0x558b0200e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.591+0000 7fdf107cc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558b01f66e70 con 0x558b020019b0 2024-04-23T14:52:19.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.591+0000 7fdf107cc740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558b01f0d380 con 0x558b02001090 2024-04-23T14:52:19.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.591+0000 7fdf107cc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558b01f62e30 con 0x558b0200c080 2024-04-23T14:52:19.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.591+0000 7fdf07400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558b02001090 0x558b02001470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.591+0000 7fdf07400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558b02001090 0x558b02001470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40308/0 (socket says 172.21.15.28:40308) 2024-04-23T14:52:19.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.591+0000 7fdf07400640 1 -- 172.21.15.28:0/3554061671 learned_addr learned my addr 172.21.15.28:0/3554061671 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:19.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.591+0000 7fdf06a00640 1 --2- 172.21.15.28:0/3554061671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b020019b0 0x558b0200bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.591+0000 7fdeffe00640 1 --2- 172.21.15.28:0/3554061671 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b0200c080 0x558b0200e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.592+0000 7fdf07400640 1 -- 172.21.15.28:0/3554061671 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b0200c080 msgr2=0x558b0200e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:19.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.592+0000 7fdf07400640 1 --2- 172.21.15.28:0/3554061671 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b0200c080 0x558b0200e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.592+0000 7fdf07400640 1 -- 172.21.15.28:0/3554061671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b020019b0 msgr2=0x558b0200bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:19.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.592+0000 7fdf07400640 1 --2- 172.21.15.28:0/3554061671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b020019b0 0x558b0200bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.592+0000 7fdf07400640 1 -- 172.21.15.28:0/3554061671 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b01d417e0 con 0x558b02001090 2024-04-23T14:52:19.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.592+0000 7fdf06a00640 1 --2- 172.21.15.28:0/3554061671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b020019b0 0x558b0200bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:52:19.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.592+0000 7fdf07400640 1 --2- 172.21.15.28:0/3554061671 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558b02001090 0x558b02001470 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fdf00009f60 tx=0x7fdf00009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f4dfd31751be692 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:19.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.592+0000 7fdeff400640 1 --2- 172.21.15.28:0/2603296434 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fdeec0a3da0 0x7fdeec0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.593+0000 7fdeff400640 1 -- 172.21.15.28:0/2603296434 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7fdef00d0760 con 0x7fdf08109580 2024-04-23T14:52:19.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.593+0000 7fdef7e00640 1 -- 172.21.15.28:0/3554061671 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf0001e070 con 0x558b02001090 2024-04-23T14:52:19.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.593+0000 7fdef7e00640 1 -- 172.21.15.28:0/3554061671 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf00003c30 con 0x558b02001090 2024-04-23T14:52:19.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.593+0000 7fdef7e00640 1 -- 172.21.15.28:0/3554061671 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf0000fa20 con 0x558b02001090 2024-04-23T14:52:19.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.593+0000 7fdf06a00640 1 --2- 172.21.15.28:0/2603296434 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fdeec0a3da0 0x7fdeec0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.594+0000 7fdf06a00640 1 --2- 172.21.15.28:0/2603296434 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fdeec0a3da0 0x7fdeec0a6260 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x558b02002650 tx=0x7fdef80058d0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:19.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.594+0000 7fdf107cc740 1 -- 172.21.15.28:0/3554061671 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558b02001090 msgr2=0x558b02001470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:19.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.594+0000 7fdf107cc740 1 --2- 172.21.15.28:0/3554061671 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558b02001090 0x558b02001470 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fdf00009f60 tx=0x7fdf00009f90 comp rx=0 tx=0).stop 2024-04-23T14:52:19.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.594+0000 7fdf107cc740 1 -- 172.21.15.28:0/3554061671 shutdown_connections 2024-04-23T14:52:19.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.594+0000 7fdf107cc740 1 --2- 172.21.15.28:0/3554061671 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b0200c080 0x558b0200e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.594+0000 7fdf107cc740 1 --2- 172.21.15.28:0/3554061671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b020019b0 0x558b0200bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.594+0000 7fdf107cc740 1 --2- 172.21.15.28:0/3554061671 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558b02001090 0x558b02001470 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.594+0000 7fdf107cc740 1 -- 172.21.15.28:0/3554061671 >> 172.21.15.28:0/3554061671 conn(0x558b01ffd3b0 msgr2=0x558b01ffd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:19.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.594+0000 7fdf107cc740 1 -- 172.21.15.28:0/3554061671 shutdown_connections 2024-04-23T14:52:19.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.594+0000 7fdf107cc740 1 -- 172.21.15.28:0/3554061671 wait complete. 2024-04-23T14:52:19.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.594+0000 7fdf107cc740 1 -- start start 2024-04-23T14:52:19.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.595+0000 7fdf107cc740 1 Processor -- start 2024-04-23T14:52:19.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.595+0000 7fdf107cc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b02001090 0x558b02009050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.595+0000 7fdf107cc740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558b020019b0 0x558b02009590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.595+0000 7fdf107cc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b0200c080 0x558b02009ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.595+0000 7fdf107cc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558b0200ee70 con 0x558b02001090 2024-04-23T14:52:19.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.595+0000 7fdf107cc740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558b01f20660 con 0x558b020019b0 2024-04-23T14:52:19.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.595+0000 7fdf107cc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558b0200eff0 con 0x558b0200c080 2024-04-23T14:52:19.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.595+0000 7fdf107cc740 10 client.0 _ll_register_callbacks cb 0x558b01f0eaa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:52:19.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.595+0000 7fdf06a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b02001090 0x558b02009050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.595+0000 7fdeffe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558b020019b0 0x558b02009590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.595+0000 7fdeffe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558b020019b0 0x558b02009590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40310/0 (socket says 172.21.15.28:40310) 2024-04-23T14:52:19.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.596+0000 7fdeffe00640 1 -- 172.21.15.28:0/4283267584 learned_addr learned my addr 172.21.15.28:0/4283267584 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:19.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.596+0000 7fdf07400640 1 --2- 172.21.15.28:0/4283267584 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b0200c080 0x558b02009ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.596+0000 7fdeffe00640 1 -- 172.21.15.28:0/4283267584 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b0200c080 msgr2=0x558b02009ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:19.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.596+0000 7fdeffe00640 1 --2- 172.21.15.28:0/4283267584 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b0200c080 0x558b02009ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.596+0000 7fdeffe00640 1 -- 172.21.15.28:0/4283267584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b02001090 msgr2=0x558b02009050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:19.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.596+0000 7fdeffe00640 1 --2- 172.21.15.28:0/4283267584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b02001090 0x558b02009050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.596+0000 7fdeffe00640 1 -- 172.21.15.28:0/4283267584 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b01da1f70 con 0x558b020019b0 2024-04-23T14:52:19.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.596+0000 7fdeffe00640 1 --2- 172.21.15.28:0/4283267584 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558b020019b0 0x558b02009590 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fdef000c6a0 tx=0x7fdef00989f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b95377034809fccd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:19.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.596+0000 7fdef4c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:52:19.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.596+0000 7fdef4c00640 1 -- 172.21.15.28:0/4283267584 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef001c070 con 0x558b020019b0 2024-04-23T14:52:19.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.596+0000 7fdef4c00640 1 -- 172.21.15.28:0/4283267584 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7fdef0085bd0 con 0x558b020019b0 2024-04-23T14:52:19.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.596+0000 7fdf107cc740 1 -- 172.21.15.28:0/4283267584 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf00003c30 con 0x558b020019b0 2024-04-23T14:52:19.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.597+0000 7fdef4c00640 1 -- 172.21.15.28:0/4283267584 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fdef00ce2c0 con 0x558b020019b0 2024-04-23T14:52:19.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.597+0000 7fdf107cc740 10 client.26498 fetch_fsmap learned FSMap version 20 2024-04-23T14:52:19.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.597+0000 7fdf107cc740 1 -- 172.21.15.28:0/4283267584 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x558b01c39170 con 0x558b020019b0 2024-04-23T14:52:19.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.597+0000 7fdef4c00640 1 -- 172.21.15.28:0/4283267584 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf00003c30 con 0x558b020019b0 2024-04-23T14:52:19.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.597+0000 7fdef4c00640 1 -- 172.21.15.28:0/4283267584 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fdef00a4e70 con 0x558b020019b0 2024-04-23T14:52:19.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.597+0000 7fdf107cc740 10 client.26498 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:52:19.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.597+0000 7fdf107cc740 10 client.26498 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:52:19.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.597+0000 7fdf107cc740 20 client.26498 populate_metadata read hostname 'smithi028' 2024-04-23T14:52:19.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.597+0000 7fdf107cc740 1 --2- 172.21.15.28:0/4283267584 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558b0201c3a0 0x558b0201e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.597+0000 7fdf107cc740 4 client.26498 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:52:19.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.597+0000 7fdf107cc740 1 -- 172.21.15.28:0/4283267584 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558b01f74890 con 0x558b0201c3a0 2024-04-23T14:52:19.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.598+0000 7fdf06a00640 1 --2- 172.21.15.28:0/4283267584 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558b0201c3a0 0x558b0201e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.598+0000 7fdf06a00640 1 --2- 172.21.15.28:0/4283267584 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558b0201c3a0 0x558b0201e7b0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b880a9fcac306494 server_cookie=6ce01b0c26a0ad64 in_seq=0 out_seq=0 2024-04-23T14:52:19.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.598+0000 7fdef4c00640 10 client.26498 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:19.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.600+0000 7fdef4c00640 1 -- 172.21.15.28:0/4283267584 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558b01f74890 con 0x558b0201c3a0 2024-04-23T14:52:19.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.600+0000 7fdef4c00640 10 client.26498 handle_command_reply: tid=0 2024-04-23T14:52:19.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.600+0000 7fdf107cc740 1 client.26498 shutdown 2024-04-23T14:52:19.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.601+0000 7fdf107cc740 1 -- 172.21.15.28:0/4283267584 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558b020019b0 msgr2=0x558b02009590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:19.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.601+0000 7fdf107cc740 1 --2- 172.21.15.28:0/4283267584 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558b020019b0 0x558b02009590 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fdef000c6a0 tx=0x7fdef00989f0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.601+0000 7fdf107cc740 1 -- 172.21.15.28:0/4283267584 shutdown_connections 2024-04-23T14:52:19.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.601+0000 7fdf107cc740 1 --2- 172.21.15.28:0/4283267584 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558b0201c3a0 0x558b0201e7b0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.601+0000 7fdf107cc740 1 --2- 172.21.15.28:0/4283267584 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558b020019b0 0x558b02009590 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.601+0000 7fdf107cc740 1 --2- 172.21.15.28:0/4283267584 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b0200c080 0x558b02009ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.601+0000 7fdf107cc740 1 --2- 172.21.15.28:0/4283267584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b02001090 0x558b02009050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.601+0000 7fdef4c00640 0 client.26498 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:19.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.601+0000 7fdf107cc740 1 -- 172.21.15.28:0/4283267584 >> 172.21.15.28:0/4283267584 conn(0x558b01ffd3b0 msgr2=0x558b020030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:19.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.601+0000 7fdf107cc740 1 -- 172.21.15.28:0/4283267584 shutdown_connections 2024-04-23T14:52:19.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.601+0000 7fdf107cc740 1 -- 172.21.15.28:0/4283267584 wait complete. 2024-04-23T14:52:19.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.601+0000 7fdf107cc740 20 client.26498 trim_cache size 0 max 16384 2024-04-23T14:52:19.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.622+0000 7fdf107cc740 1 Processor -- start 2024-04-23T14:52:19.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.622+0000 7fdf107cc740 1 -- start start 2024-04-23T14:52:19.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.622+0000 7fdf107cc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b02001090 0x558b02017d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.622+0000 7fdf107cc740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558b020019b0 0x558b02003830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.622+0000 7fdf107cc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b0200c080 0x558b02024080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.622+0000 7fdf107cc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdf081ae030 con 0x558b0200c080 2024-04-23T14:52:19.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.622+0000 7fdf107cc740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdf081b0890 con 0x558b020019b0 2024-04-23T14:52:19.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.622+0000 7fdf107cc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdf081ae680 con 0x558b02001090 2024-04-23T14:52:19.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.622+0000 7fdeffe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558b020019b0 0x558b02003830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.622+0000 7fdeffe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558b020019b0 0x558b02003830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40324/0 (socket says 172.21.15.28:40324) 2024-04-23T14:52:19.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.622+0000 7fdeffe00640 1 -- 172.21.15.28:0/3792834923 learned_addr learned my addr 172.21.15.28:0/3792834923 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:19.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.622+0000 7fdf07400640 1 --2- 172.21.15.28:0/3792834923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b0200c080 0x558b02024080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.622+0000 7fdf06a00640 1 --2- 172.21.15.28:0/3792834923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b02001090 0x558b02017d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.623+0000 7fdeffe00640 1 -- 172.21.15.28:0/3792834923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b02001090 msgr2=0x558b02017d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:19.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.623+0000 7fdeffe00640 1 --2- 172.21.15.28:0/3792834923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b02001090 0x558b02017d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.623+0000 7fdeffe00640 1 -- 172.21.15.28:0/3792834923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b0200c080 msgr2=0x558b02024080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:19.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.623+0000 7fdeffe00640 1 --2- 172.21.15.28:0/3792834923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b0200c080 0x558b02024080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.623+0000 7fdeffe00640 1 -- 172.21.15.28:0/3792834923 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b020075a0 con 0x558b020019b0 2024-04-23T14:52:19.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.623+0000 7fdeffe00640 1 --2- 172.21.15.28:0/3792834923 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558b020019b0 0x558b02003830 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fdef009a940 tx=0x7fdef009aba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1528f8c6155dbba server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:19.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.623+0000 7fdef7400640 1 -- 172.21.15.28:0/3792834923 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef00a0a40 con 0x558b020019b0 2024-04-23T14:52:19.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.623+0000 7fdef7400640 1 -- 172.21.15.28:0/3792834923 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdef009cd90 con 0x558b020019b0 2024-04-23T14:52:19.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.623+0000 7fdef7400640 1 -- 172.21.15.28:0/3792834923 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef00ce0c0 con 0x558b020019b0 2024-04-23T14:52:19.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.624+0000 7fdf107cc740 1 -- 172.21.15.28:0/3792834923 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558b020019b0 msgr2=0x558b02003830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:19.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.624+0000 7fdf107cc740 1 --2- 172.21.15.28:0/3792834923 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558b020019b0 0x558b02003830 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fdef009a940 tx=0x7fdef009aba0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.624+0000 7fdf107cc740 1 -- 172.21.15.28:0/3792834923 shutdown_connections 2024-04-23T14:52:19.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.624+0000 7fdf107cc740 1 --2- 172.21.15.28:0/3792834923 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558b020019b0 0x558b02003830 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.624+0000 7fdf107cc740 1 --2- 172.21.15.28:0/3792834923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b0200c080 0x558b02024080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.624+0000 7fdf107cc740 1 --2- 172.21.15.28:0/3792834923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b02001090 0x558b02017d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.624+0000 7fdf107cc740 1 -- 172.21.15.28:0/3792834923 >> 172.21.15.28:0/3792834923 conn(0x558b01ffd3b0 msgr2=0x558b01ffc510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:19.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.624+0000 7fdf107cc740 1 -- 172.21.15.28:0/3792834923 shutdown_connections 2024-04-23T14:52:19.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.624+0000 7fdf107cc740 1 -- 172.21.15.28:0/3792834923 wait complete. 2024-04-23T14:52:19.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.624+0000 7fdf107cc740 1 -- start start 2024-04-23T14:52:19.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.625+0000 7fdf107cc740 1 Processor -- start 2024-04-23T14:52:19.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.625+0000 7fdf107cc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b02001090 0x558b0200fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.625+0000 7fdf107cc740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558b020019b0 0x558b02010390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.625+0000 7fdf107cc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b0200c080 0x558b020108d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.625+0000 7fdf107cc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdf081aed50 con 0x558b0200c080 2024-04-23T14:52:19.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.625+0000 7fdf107cc740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdf081b01c0 con 0x558b020019b0 2024-04-23T14:52:19.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.625+0000 7fdf107cc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdf081af420 con 0x558b02001090 2024-04-23T14:52:19.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.625+0000 7fdf107cc740 10 client.0 _ll_register_callbacks cb 0x7fdef000cf30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:52:19.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.625+0000 7fdeffe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558b020019b0 0x558b02010390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.625+0000 7fdf07400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b0200c080 0x558b020108d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.625+0000 7fdf06a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b02001090 0x558b0200fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.625+0000 7fdeffe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558b020019b0 0x558b02010390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40330/0 (socket says 172.21.15.28:40330) 2024-04-23T14:52:19.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.626+0000 7fdeffe00640 1 -- 172.21.15.28:0/3252961839 learned_addr learned my addr 172.21.15.28:0/3252961839 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:19.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.626+0000 7fdeffe00640 1 -- 172.21.15.28:0/3252961839 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b02001090 msgr2=0x558b0200fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:19.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.626+0000 7fdeffe00640 1 --2- 172.21.15.28:0/3252961839 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b02001090 0x558b0200fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.626+0000 7fdeffe00640 1 -- 172.21.15.28:0/3252961839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b0200c080 msgr2=0x558b020108d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:19.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.626+0000 7fdeffe00640 1 --2- 172.21.15.28:0/3252961839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b0200c080 0x558b020108d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.626+0000 7fdeffe00640 1 -- 172.21.15.28:0/3252961839 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b02081af0 con 0x558b020019b0 2024-04-23T14:52:19.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.626+0000 7fdeffe00640 1 --2- 172.21.15.28:0/3252961839 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558b020019b0 0x558b02010390 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fdef0099010 tx=0x7fdef00d6530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f81311317753a0c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:19.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.626+0000 7fdef4c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:52:19.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.626+0000 7fdef4c00640 1 -- 172.21.15.28:0/3252961839 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef0080060 con 0x558b020019b0 2024-04-23T14:52:19.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.626+0000 7fdef4c00640 1 -- 172.21.15.28:0/3252961839 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7fdef00ce0c0 con 0x558b020019b0 2024-04-23T14:52:19.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.626+0000 7fdf107cc740 1 -- 172.21.15.28:0/3252961839 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdef009cd90 con 0x558b020019b0 2024-04-23T14:52:19.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.627+0000 7fdef4c00640 1 -- 172.21.15.28:0/3252961839 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fdef00ba070 con 0x558b020019b0 2024-04-23T14:52:19.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.627+0000 7fdf107cc740 10 client.26504 fetch_fsmap learned FSMap version 20 2024-04-23T14:52:19.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.627+0000 7fdf107cc740 1 -- 172.21.15.28:0/3252961839 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x558b02082e10 con 0x558b020019b0 2024-04-23T14:52:19.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.627+0000 7fdef4c00640 1 -- 172.21.15.28:0/3252961839 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdef009cd90 con 0x558b020019b0 2024-04-23T14:52:19.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.627+0000 7fdef4c00640 1 -- 172.21.15.28:0/3252961839 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fdef00983e0 con 0x558b020019b0 2024-04-23T14:52:19.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.627+0000 7fdf107cc740 10 client.26504 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:52:19.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.627+0000 7fdf107cc740 10 client.26504 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:52:19.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.627+0000 7fdf107cc740 20 client.26504 populate_metadata read hostname 'smithi028' 2024-04-23T14:52:19.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.627+0000 7fdf107cc740 1 --2- 172.21.15.28:0/3252961839 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558b02083250 0x558b02085670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.627+0000 7fdf107cc740 4 client.26504 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-23T14:52:19.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.627+0000 7fdf107cc740 1 -- 172.21.15.28:0/3252961839 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558b01f6e700 con 0x558b02083250 2024-04-23T14:52:19.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.627+0000 7fdf06a00640 1 --2- 172.21.15.28:0/3252961839 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558b02083250 0x558b02085670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.628+0000 7fdf06a00640 1 --2- 172.21.15.28:0/3252961839 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558b02083250 0x558b02085670 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=452d5bbbb50ec441 server_cookie=75c67cff1d585fc3 in_seq=0 out_seq=0 2024-04-23T14:52:19.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.628+0000 7fdef4c00640 10 client.26504 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:19.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.662+0000 7fdef4c00640 1 -- 172.21.15.28:0/3252961839 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558b01f6e700 con 0x558b02083250 2024-04-23T14:52:19.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.662+0000 7fdef4c00640 10 client.26504 handle_command_reply: tid=0 2024-04-23T14:52:19.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.662+0000 7fdf107cc740 1 client.26504 shutdown 2024-04-23T14:52:19.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.662+0000 7fdf107cc740 1 -- 172.21.15.28:0/3252961839 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558b020019b0 msgr2=0x558b02010390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:19.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.662+0000 7fdf107cc740 1 --2- 172.21.15.28:0/3252961839 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558b020019b0 0x558b02010390 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fdef0099010 tx=0x7fdef00d6530 comp rx=0 tx=0).stop 2024-04-23T14:52:19.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.662+0000 7fdf107cc740 1 -- 172.21.15.28:0/3252961839 shutdown_connections 2024-04-23T14:52:19.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.662+0000 7fdf107cc740 1 --2- 172.21.15.28:0/3252961839 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558b02083250 0x558b02085670 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.662+0000 7fdf107cc740 1 --2- 172.21.15.28:0/3252961839 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558b020019b0 0x558b02010390 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.662+0000 7fdf107cc740 1 --2- 172.21.15.28:0/3252961839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b0200c080 0x558b020108d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.662+0000 7fdf107cc740 1 --2- 172.21.15.28:0/3252961839 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b02001090 0x558b0200fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.662+0000 7fdef4c00640 0 client.26504 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:19.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.662+0000 7fdf107cc740 1 -- 172.21.15.28:0/3252961839 >> 172.21.15.28:0/3252961839 conn(0x558b01ffd3b0 msgr2=0x558b02009010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:19.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.663+0000 7fdf107cc740 1 -- 172.21.15.28:0/3252961839 shutdown_connections 2024-04-23T14:52:19.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.663+0000 7fdf107cc740 1 -- 172.21.15.28:0/3252961839 wait complete. 2024-04-23T14:52:19.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.663+0000 7fdf107cc740 20 client.26504 trim_cache size 0 max 16384 2024-04-23T14:52:19.681 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:52:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-23T14:52:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "a8d05ae0-e5c0-430a-b310-048e299a5f97", 2024-04-23T14:52:19.681 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-23T14:52:19.682 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:52:19.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.664+0000 7fdf0da00640 1 -- 172.21.15.28:0/2603296434 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fdeec0a3da0 msgr2=0x7fdeec0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:19.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.664+0000 7fdf0da00640 1 --2- 172.21.15.28:0/2603296434 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fdeec0a3da0 0x7fdeec0a6260 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x558b02002650 tx=0x7fdef80058d0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.664+0000 7fdf0da00640 1 -- 172.21.15.28:0/2603296434 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf08109580 msgr2=0x7fdf081a0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:19.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.664+0000 7fdf0da00640 1 --2- 172.21.15.28:0/2603296434 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf08109580 0x7fdf081a0d80 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fdef000e9c0 tx=0x7fdef000edc0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.664+0000 7fdf0da00640 1 -- 172.21.15.28:0/2603296434 shutdown_connections 2024-04-23T14:52:19.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.664+0000 7fdf0da00640 1 --2- 172.21.15.28:0/2603296434 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fdeec0a3da0 0x7fdeec0a6260 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.664+0000 7fdf0da00640 1 --2- 172.21.15.28:0/2603296434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf0819a9f0 0x7fdf0819aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.664+0000 7fdf0da00640 1 --2- 172.21.15.28:0/2603296434 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdf08109580 0x7fdf081a0d80 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.664+0000 7fdf0da00640 1 --2- 172.21.15.28:0/2603296434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf081044b0 0x7fdf081a0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.664+0000 7fdf0da00640 1 -- 172.21.15.28:0/2603296434 >> 172.21.15.28:0/2603296434 conn(0x7fdf080ffd90 msgr2=0x7fdf08101110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:19.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.664+0000 7fdf0da00640 1 -- 172.21.15.28:0/2603296434 shutdown_connections 2024-04-23T14:52:19.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.664+0000 7fdf0da00640 1 -- 172.21.15.28:0/2603296434 wait complete. 2024-04-23T14:52:19.684 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:52:19.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:52:19.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.890+0000 7f6675e00640 1 -- 172.21.15.28:0/2769610982 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6670104e60 msgr2=0x7f6670108ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:19.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.890+0000 7f6675e00640 1 --2- 172.21.15.28:0/2769610982 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6670104e60 0x7f6670108ea0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f6658009900 tx=0x7f665802a3f0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.890+0000 7f6675e00640 1 -- 172.21.15.28:0/2769610982 shutdown_connections 2024-04-23T14:52:19.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.890+0000 7f6675e00640 1 --2- 172.21.15.28:0/2769610982 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6670104e60 0x7f6670108ea0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.890+0000 7f6675e00640 1 --2- 172.21.15.28:0/2769610982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6670109580 0x7f6670111150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.890+0000 7f6675e00640 1 --2- 172.21.15.28:0/2769610982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66701044b0 0x7f6670104890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.890+0000 7f6675e00640 1 -- 172.21.15.28:0/2769610982 >> 172.21.15.28:0/2769610982 conn(0x7f66700ffd90 msgr2=0x7f66701021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:19.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.890+0000 7f6675e00640 1 -- 172.21.15.28:0/2769610982 shutdown_connections 2024-04-23T14:52:19.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.890+0000 7f6675e00640 1 -- 172.21.15.28:0/2769610982 wait complete. 2024-04-23T14:52:19.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.891+0000 7f6675e00640 1 Processor -- start 2024-04-23T14:52:19.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.892+0000 7f6675e00640 1 -- start start 2024-04-23T14:52:19.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.892+0000 7f6675e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66701044b0 0x7f6670076510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.892+0000 7f6675e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6670109580 0x7f6670076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.892+0000 7f6675e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6670076f90 0x7f66701ab190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.892+0000 7f6675e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6670114640 con 0x7f6670076f90 2024-04-23T14:52:19.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.892+0000 7f6675e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f66701144c0 con 0x7f66701044b0 2024-04-23T14:52:19.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.892+0000 7f6675e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f66701147c0 con 0x7f6670109580 2024-04-23T14:52:19.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.893+0000 7f666ea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66701044b0 0x7f6670076510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.893+0000 7f666f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6670076f90 0x7f66701ab190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.893+0000 7f666ea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66701044b0 0x7f6670076510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40350/0 (socket says 172.21.15.28:40350) 2024-04-23T14:52:19.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.893+0000 7f666ea00640 1 -- 172.21.15.28:0/4027348836 learned_addr learned my addr 172.21.15.28:0/4027348836 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:19.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.893+0000 7f6666000640 1 --2- 172.21.15.28:0/4027348836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6670109580 0x7f6670076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.893+0000 7f666ea00640 1 -- 172.21.15.28:0/4027348836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6670109580 msgr2=0x7f6670076a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:19.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.893+0000 7f666ea00640 1 --2- 172.21.15.28:0/4027348836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6670109580 0x7f6670076a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.893+0000 7f666ea00640 1 -- 172.21.15.28:0/4027348836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6670076f90 msgr2=0x7f66701ab190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:19.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.893+0000 7f666ea00640 1 --2- 172.21.15.28:0/4027348836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6670076f90 0x7f66701ab190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.893+0000 7f666ea00640 1 -- 172.21.15.28:0/4027348836 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66701ab6d0 con 0x7f66701044b0 2024-04-23T14:52:19.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.893+0000 7f6666000640 1 --2- 172.21.15.28:0/4027348836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6670109580 0x7f6670076a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:52:19.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.894+0000 7f666ea00640 1 --2- 172.21.15.28:0/4027348836 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66701044b0 0x7f6670076510 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f666000ebc0 tx=0x7f666000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:19.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.894+0000 7f6674a00640 1 -- 172.21.15.28:0/4027348836 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6660010630 con 0x7f66701044b0 2024-04-23T14:52:19.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.894+0000 7f6674a00640 1 -- 172.21.15.28:0/4027348836 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6660010cb0 con 0x7f66701044b0 2024-04-23T14:52:19.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.894+0000 7f6675e00640 1 -- 172.21.15.28:0/4027348836 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66701ab960 con 0x7f66701044b0 2024-04-23T14:52:19.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.894+0000 7f6674a00640 1 -- 172.21.15.28:0/4027348836 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6660018bd0 con 0x7f66701044b0 2024-04-23T14:52:19.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.894+0000 7f6675e00640 1 -- 172.21.15.28:0/4027348836 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66701abe80 con 0x7f66701044b0 2024-04-23T14:52:19.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.898+0000 7f6674a00640 1 -- 172.21.15.28:0/4027348836 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f66600107d0 con 0x7f66701044b0 2024-04-23T14:52:19.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.898+0000 7f6678d86740 1 Processor -- start 2024-04-23T14:52:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.898+0000 7f6678d86740 1 -- start start 2024-04-23T14:52:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.899+0000 7f6678d86740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c1cf53d860 0x55c1cf53dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.899+0000 7f6678d86740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1cf53e180 0x55c1cf548310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.899+0000 7f6678d86740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1cf548850 0x55c1cf54ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.899+0000 7f6678d86740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c1cf4a31f0 con 0x55c1cf548850 2024-04-23T14:52:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.899+0000 7f6678d86740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c1cf449350 con 0x55c1cf53e180 2024-04-23T14:52:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.899+0000 7f6678d86740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c1cf49f1b0 con 0x55c1cf53d860 2024-04-23T14:52:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.899+0000 7f666ea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1cf53e180 0x55c1cf548310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.899+0000 7f666f400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c1cf53d860 0x55c1cf53dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.899+0000 7f6674a00640 1 --2- 172.21.15.28:0/4027348836 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f66540a3d20 0x7f66540a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.899+0000 7f666ea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1cf53e180 0x55c1cf548310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40354/0 (socket says 172.21.15.28:40354) 2024-04-23T14:52:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.899+0000 7f666ea00640 1 -- 172.21.15.28:0/582992111 learned_addr learned my addr 172.21.15.28:0/582992111 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.899+0000 7f6666000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1cf548850 0x55c1cf54ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.900+0000 7f6674a00640 1 -- 172.21.15.28:0/4027348836 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7f66600d0e60 con 0x7f66701044b0 2024-04-23T14:52:19.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.900+0000 7f666f400640 1 --2- 172.21.15.28:0/4027348836 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f66540a3d20 0x7f66540a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.900+0000 7f666ea00640 1 -- 172.21.15.28:0/582992111 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c1cf53d860 msgr2=0x55c1cf53dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:19.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.900+0000 7f666ea00640 1 --2- 172.21.15.28:0/582992111 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c1cf53d860 0x55c1cf53dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.900+0000 7f666ea00640 1 -- 172.21.15.28:0/582992111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1cf548850 msgr2=0x55c1cf54ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:19.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.900+0000 7f666ea00640 1 --2- 172.21.15.28:0/582992111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1cf548850 0x55c1cf54ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.900+0000 7f666ea00640 1 -- 172.21.15.28:0/582992111 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1cf49df50 con 0x55c1cf53e180 2024-04-23T14:52:19.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.900+0000 7f666ea00640 1 --2- 172.21.15.28:0/582992111 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1cf53e180 0x55c1cf548310 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f666000eb90 tx=0x7f6660000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b316049b08bb6ab1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:19.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.900+0000 7f665fe00640 1 -- 172.21.15.28:0/582992111 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6660010e60 con 0x55c1cf53e180 2024-04-23T14:52:19.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.900+0000 7f665fe00640 1 -- 172.21.15.28:0/582992111 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66600a1140 con 0x55c1cf53e180 2024-04-23T14:52:19.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.900+0000 7f665fe00640 1 -- 172.21.15.28:0/582992111 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666009c290 con 0x55c1cf53e180 2024-04-23T14:52:19.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.901+0000 7f666f400640 1 --2- 172.21.15.28:0/4027348836 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f66540a3d20 0x7f66540a61e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6668009fd0 tx=0x7f6668004730 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:19.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.901+0000 7f6678d86740 1 -- 172.21.15.28:0/582992111 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1cf53e180 msgr2=0x55c1cf548310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:19.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.901+0000 7f6678d86740 1 --2- 172.21.15.28:0/582992111 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1cf53e180 0x55c1cf548310 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f666000eb90 tx=0x7f6660000f40 comp rx=0 tx=0).stop 2024-04-23T14:52:19.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.901+0000 7f6678d86740 1 -- 172.21.15.28:0/582992111 shutdown_connections 2024-04-23T14:52:19.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.901+0000 7f6678d86740 1 --2- 172.21.15.28:0/582992111 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1cf53e180 0x55c1cf548310 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.901+0000 7f6678d86740 1 --2- 172.21.15.28:0/582992111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1cf548850 0x55c1cf54ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.901+0000 7f6678d86740 1 --2- 172.21.15.28:0/582992111 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c1cf53d860 0x55c1cf53dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.901+0000 7f6678d86740 1 -- 172.21.15.28:0/582992111 >> 172.21.15.28:0/582992111 conn(0x55c1cf539b80 msgr2=0x55c1cf539f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:19.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.901+0000 7f6678d86740 1 -- 172.21.15.28:0/582992111 shutdown_connections 2024-04-23T14:52:19.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.901+0000 7f6678d86740 1 -- 172.21.15.28:0/582992111 wait complete. 2024-04-23T14:52:19.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.902+0000 7f6678d86740 1 -- start start 2024-04-23T14:52:19.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.902+0000 7f6678d86740 1 Processor -- start 2024-04-23T14:52:19.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.903+0000 7f6678d86740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1cf53d860 0x55c1cf545780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.903+0000 7f6678d86740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1cf53e180 0x55c1cf545cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.903+0000 7f6678d86740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c1cf548850 0x55c1cf546200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.903+0000 7f6678d86740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c1cf54b640 con 0x55c1cf53e180 2024-04-23T14:52:19.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.903+0000 7f6678d86740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c1cf45c690 con 0x55c1cf53d860 2024-04-23T14:52:19.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.903+0000 7f6678d86740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c1cf54b7c0 con 0x55c1cf548850 2024-04-23T14:52:19.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.903+0000 7f6678d86740 10 client.0 _ll_register_callbacks cb 0x55c1cf498650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:52:19.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.903+0000 7f6666000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1cf53e180 0x55c1cf545cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.903+0000 7f666ea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1cf53d860 0x55c1cf545780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.903+0000 7f666ea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1cf53d860 0x55c1cf545780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40360/0 (socket says 172.21.15.28:40360) 2024-04-23T14:52:19.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.903+0000 7f666ea00640 1 -- 172.21.15.28:0/2868363080 learned_addr learned my addr 172.21.15.28:0/2868363080 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:19.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.903+0000 7f666f400640 1 --2- 172.21.15.28:0/2868363080 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c1cf548850 0x55c1cf546200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.904+0000 7f666ea00640 1 -- 172.21.15.28:0/2868363080 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c1cf548850 msgr2=0x55c1cf546200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:19.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.904+0000 7f666ea00640 1 --2- 172.21.15.28:0/2868363080 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c1cf548850 0x55c1cf546200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.904+0000 7f666ea00640 1 -- 172.21.15.28:0/2868363080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1cf53e180 msgr2=0x55c1cf545cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:19.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.904+0000 7f666ea00640 1 --2- 172.21.15.28:0/2868363080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1cf53e180 0x55c1cf545cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.904+0000 7f666ea00640 1 -- 172.21.15.28:0/2868363080 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1cf45b920 con 0x55c1cf53d860 2024-04-23T14:52:19.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.904+0000 7f6666000640 1 --2- 172.21.15.28:0/2868363080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1cf53e180 0x55c1cf545cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:52:19.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.904+0000 7f666ea00640 1 --2- 172.21.15.28:0/2868363080 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1cf53d860 0x55c1cf545780 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f66600ce5a0 tx=0x7f66600cead0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbd06b35711f3266 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:19.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.904+0000 7f665cc00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:52:19.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.904+0000 7f665cc00640 1 -- 172.21.15.28:0/2868363080 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6660010e60 con 0x55c1cf53d860 2024-04-23T14:52:19.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.904+0000 7f665cc00640 1 -- 172.21.15.28:0/2868363080 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7f666009c4a0 con 0x55c1cf53d860 2024-04-23T14:52:19.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.904+0000 7f6678d86740 1 -- 172.21.15.28:0/2868363080 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66600a1140 con 0x55c1cf53d860 2024-04-23T14:52:19.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.905+0000 7f665cc00640 1 -- 172.21.15.28:0/2868363080 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f66600a5a00 con 0x55c1cf53d860 2024-04-23T14:52:19.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.905+0000 7f6678d86740 10 client.26516 fetch_fsmap learned FSMap version 20 2024-04-23T14:52:19.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.905+0000 7f6678d86740 1 -- 172.21.15.28:0/2868363080 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55c1cf27f7d0 con 0x55c1cf53d860 2024-04-23T14:52:19.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.905+0000 7f665cc00640 1 -- 172.21.15.28:0/2868363080 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66600a1140 con 0x55c1cf53d860 2024-04-23T14:52:19.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.905+0000 7f665cc00640 1 -- 172.21.15.28:0/2868363080 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f666009bd00 con 0x55c1cf53d860 2024-04-23T14:52:19.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.905+0000 7f6678d86740 10 client.26516 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:52:19.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.905+0000 7f6678d86740 10 client.26516 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:52:19.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.905+0000 7f6678d86740 20 client.26516 populate_metadata read hostname 'smithi028' 2024-04-23T14:52:19.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.905+0000 7f6678d86740 1 --2- 172.21.15.28:0/2868363080 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c1cf558a00 0x55c1cf55adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.905+0000 7f6678d86740 4 client.26516 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:52:19.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.905+0000 7f6678d86740 1 -- 172.21.15.28:0/2868363080 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c1cf4b0c10 con 0x55c1cf558a00 2024-04-23T14:52:19.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.906+0000 7f6666000640 1 --2- 172.21.15.28:0/2868363080 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c1cf558a00 0x55c1cf55adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.906+0000 7f6666000640 1 --2- 172.21.15.28:0/2868363080 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c1cf558a00 0x55c1cf55adf0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87d965ef84035c9f server_cookie=44303e7f65ee5a40 in_seq=0 out_seq=0 2024-04-23T14:52:19.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.906+0000 7f665cc00640 10 client.26516 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:19.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.908+0000 7f665cc00640 1 -- 172.21.15.28:0/2868363080 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c1cf4b0c10 con 0x55c1cf558a00 2024-04-23T14:52:19.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.908+0000 7f665cc00640 10 client.26516 handle_command_reply: tid=0 2024-04-23T14:52:19.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.908+0000 7f6678d86740 1 client.26516 shutdown 2024-04-23T14:52:19.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.908+0000 7f6678d86740 1 -- 172.21.15.28:0/2868363080 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1cf53d860 msgr2=0x55c1cf545780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:19.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.908+0000 7f6678d86740 1 --2- 172.21.15.28:0/2868363080 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1cf53d860 0x55c1cf545780 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f66600ce5a0 tx=0x7f66600cead0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.909+0000 7f6678d86740 1 -- 172.21.15.28:0/2868363080 shutdown_connections 2024-04-23T14:52:19.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.909+0000 7f6678d86740 1 --2- 172.21.15.28:0/2868363080 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c1cf558a00 0x55c1cf55adf0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.909+0000 7f6678d86740 1 --2- 172.21.15.28:0/2868363080 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c1cf548850 0x55c1cf546200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.909+0000 7f6678d86740 1 --2- 172.21.15.28:0/2868363080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1cf53e180 0x55c1cf545cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.909+0000 7f6678d86740 1 --2- 172.21.15.28:0/2868363080 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1cf53d860 0x55c1cf545780 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.909+0000 7f665cc00640 0 client.26516 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:19.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.909+0000 7f6678d86740 1 -- 172.21.15.28:0/2868363080 >> 172.21.15.28:0/2868363080 conn(0x55c1cf539b80 msgr2=0x55c1cf53f980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:19.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.909+0000 7f6678d86740 1 -- 172.21.15.28:0/2868363080 shutdown_connections 2024-04-23T14:52:19.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.909+0000 7f6678d86740 1 -- 172.21.15.28:0/2868363080 wait complete. 2024-04-23T14:52:19.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.909+0000 7f6678d86740 20 client.26516 trim_cache size 0 max 16384 2024-04-23T14:52:19.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.930+0000 7f6678d86740 1 Processor -- start 2024-04-23T14:52:19.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.930+0000 7f6678d86740 1 -- start start 2024-04-23T14:52:19.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.930+0000 7f6678d86740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1cf53d860 0x55c1cf5476c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.930+0000 7f6678d86740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c1cf53e180 0x55c1cf55de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.930+0000 7f6678d86740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1cf548850 0x55c1cf55e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.930+0000 7f6678d86740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f66701b2420 con 0x55c1cf548850 2024-04-23T14:52:19.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.930+0000 7f6678d86740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f66701b4c80 con 0x55c1cf53d860 2024-04-23T14:52:19.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.930+0000 7f6678d86740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f66701b2a70 con 0x55c1cf53e180 2024-04-23T14:52:19.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.930+0000 7f666ea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1cf53d860 0x55c1cf5476c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.930+0000 7f666ea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1cf53d860 0x55c1cf5476c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40374/0 (socket says 172.21.15.28:40374) 2024-04-23T14:52:19.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.931+0000 7f666ea00640 1 -- 172.21.15.28:0/2032594598 learned_addr learned my addr 172.21.15.28:0/2032594598 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:19.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.931+0000 7f6666000640 1 --2- 172.21.15.28:0/2032594598 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c1cf53e180 0x55c1cf55de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.931+0000 7f666f400640 1 --2- 172.21.15.28:0/2032594598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1cf548850 0x55c1cf55e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.931+0000 7f666ea00640 1 -- 172.21.15.28:0/2032594598 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c1cf53e180 msgr2=0x55c1cf55de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:19.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.931+0000 7f666ea00640 1 --2- 172.21.15.28:0/2032594598 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c1cf53e180 0x55c1cf55de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.931+0000 7f666ea00640 1 -- 172.21.15.28:0/2032594598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1cf548850 msgr2=0x55c1cf55e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:19.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.931+0000 7f666ea00640 1 --2- 172.21.15.28:0/2032594598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1cf548850 0x55c1cf55e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.931+0000 7f666ea00640 1 -- 172.21.15.28:0/2032594598 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1cf1b53f0 con 0x55c1cf53d860 2024-04-23T14:52:19.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.931+0000 7f6666000640 1 --2- 172.21.15.28:0/2032594598 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c1cf53e180 0x55c1cf55de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:52:19.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.931+0000 7f666f400640 1 --2- 172.21.15.28:0/2032594598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1cf548850 0x55c1cf55e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:52:19.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.931+0000 7f666ea00640 1 --2- 172.21.15.28:0/2032594598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1cf53d860 0x55c1cf5476c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f66600ceb40 tx=0x7f666009cfc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9879beaa5ff707e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:19.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.932+0000 7f665f400640 1 -- 172.21.15.28:0/2032594598 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6660010e60 con 0x55c1cf53d860 2024-04-23T14:52:19.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.932+0000 7f665f400640 1 -- 172.21.15.28:0/2032594598 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66600a1140 con 0x55c1cf53d860 2024-04-23T14:52:19.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.932+0000 7f665f400640 1 -- 172.21.15.28:0/2032594598 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666009bd00 con 0x55c1cf53d860 2024-04-23T14:52:19.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.932+0000 7f6678d86740 1 -- 172.21.15.28:0/2032594598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1cf53d860 msgr2=0x55c1cf5476c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:19.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.932+0000 7f6678d86740 1 --2- 172.21.15.28:0/2032594598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1cf53d860 0x55c1cf5476c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f66600ceb40 tx=0x7f666009cfc0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.933+0000 7f6678d86740 1 -- 172.21.15.28:0/2032594598 shutdown_connections 2024-04-23T14:52:19.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.933+0000 7f6678d86740 1 --2- 172.21.15.28:0/2032594598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1cf548850 0x55c1cf55e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.933+0000 7f6678d86740 1 --2- 172.21.15.28:0/2032594598 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c1cf53e180 0x55c1cf55de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.933+0000 7f6678d86740 1 --2- 172.21.15.28:0/2032594598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1cf53d860 0x55c1cf5476c0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.933+0000 7f6678d86740 1 -- 172.21.15.28:0/2032594598 >> 172.21.15.28:0/2032594598 conn(0x55c1cf539b80 msgr2=0x55c1cf546fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:19.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.933+0000 7f6678d86740 1 -- 172.21.15.28:0/2032594598 shutdown_connections 2024-04-23T14:52:19.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.933+0000 7f6678d86740 1 -- 172.21.15.28:0/2032594598 wait complete. 2024-04-23T14:52:19.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.933+0000 7f6678d86740 1 -- start start 2024-04-23T14:52:19.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.933+0000 7f6678d86740 1 Processor -- start 2024-04-23T14:52:19.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.934+0000 7f6678d86740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c1cf53e180 0x55c1cf55b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.934+0000 7f6678d86740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1cf548850 0x55c1cf55bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.934+0000 7f6678d86740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1cf55c130 0x55c1cf5bdbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.934+0000 7f6678d86740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f66701b3140 con 0x55c1cf548850 2024-04-23T14:52:19.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.934+0000 7f6678d86740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f66701b45b0 con 0x55c1cf55c130 2024-04-23T14:52:19.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.934+0000 7f6678d86740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f66701b3810 con 0x55c1cf53e180 2024-04-23T14:52:19.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.934+0000 7f6678d86740 10 client.0 _ll_register_callbacks cb 0x7f66600bbf50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:52:19.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.934+0000 7f666f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1cf55c130 0x55c1cf5bdbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.934+0000 7f666f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1cf55c130 0x55c1cf5bdbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40378/0 (socket says 172.21.15.28:40378) 2024-04-23T14:52:19.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.934+0000 7f666f400640 1 -- 172.21.15.28:0/3065932961 learned_addr learned my addr 172.21.15.28:0/3065932961 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:19.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.934+0000 7f6666000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1cf548850 0x55c1cf55bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.934+0000 7f666ea00640 1 --2- 172.21.15.28:0/3065932961 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c1cf53e180 0x55c1cf55b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.935+0000 7f666f400640 1 -- 172.21.15.28:0/3065932961 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c1cf53e180 msgr2=0x55c1cf55b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:19.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.935+0000 7f666f400640 1 --2- 172.21.15.28:0/3065932961 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c1cf53e180 0x55c1cf55b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.935+0000 7f666f400640 1 -- 172.21.15.28:0/3065932961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1cf548850 msgr2=0x55c1cf55bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:19.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.935+0000 7f666f400640 1 --2- 172.21.15.28:0/3065932961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1cf548850 0x55c1cf55bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.935+0000 7f666f400640 1 -- 172.21.15.28:0/3065932961 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1cf1750f0 con 0x55c1cf55c130 2024-04-23T14:52:19.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.935+0000 7f6666000640 1 --2- 172.21.15.28:0/3065932961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1cf548850 0x55c1cf55bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:52:19.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.935+0000 7f666f400640 1 --2- 172.21.15.28:0/3065932961 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1cf55c130 0x55c1cf5bdbe0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6668010780 tx=0x7f66680046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9b59e1d1e6b98b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:19.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.935+0000 7f665cc00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:52:19.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.935+0000 7f665cc00640 1 -- 172.21.15.28:0/3065932961 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6668020980 con 0x55c1cf55c130 2024-04-23T14:52:19.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.935+0000 7f665cc00640 1 -- 172.21.15.28:0/3065932961 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7f6668020b20 con 0x55c1cf55c130 2024-04-23T14:52:19.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.935+0000 7f6678d86740 1 -- 172.21.15.28:0/3065932961 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66600a1140 con 0x55c1cf55c130 2024-04-23T14:52:19.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.936+0000 7f665cc00640 1 -- 172.21.15.28:0/3065932961 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f6668020d40 con 0x55c1cf55c130 2024-04-23T14:52:19.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.936+0000 7f6678d86740 10 client.26522 fetch_fsmap learned FSMap version 20 2024-04-23T14:52:19.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.936+0000 7f6678d86740 1 -- 172.21.15.28:0/3065932961 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55c1cf468810 con 0x55c1cf55c130 2024-04-23T14:52:19.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.936+0000 7f665cc00640 1 -- 172.21.15.28:0/3065932961 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66600a1140 con 0x55c1cf55c130 2024-04-23T14:52:19.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.936+0000 7f665cc00640 1 -- 172.21.15.28:0/3065932961 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f6668004c10 con 0x55c1cf55c130 2024-04-23T14:52:19.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.936+0000 7f6678d86740 10 client.26522 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:52:19.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.936+0000 7f6678d86740 10 client.26522 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:52:19.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.936+0000 7f6678d86740 20 client.26522 populate_metadata read hostname 'smithi028' 2024-04-23T14:52:19.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.936+0000 7f6678d86740 1 --2- 172.21.15.28:0/3065932961 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c1cf5bf540 0x55c1cf5c1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:19.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.936+0000 7f6678d86740 4 client.26522 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:52:19.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.936+0000 7f6678d86740 1 -- 172.21.15.28:0/3065932961 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c1cf4aaa80 con 0x55c1cf5bf540 2024-04-23T14:52:19.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.936+0000 7f666ea00640 1 --2- 172.21.15.28:0/3065932961 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c1cf5bf540 0x55c1cf5c1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:19.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.937+0000 7f666ea00640 1 --2- 172.21.15.28:0/3065932961 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c1cf5bf540 0x55c1cf5c1960 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2793b02a0412113 server_cookie=cdf0185563e0ff2 in_seq=0 out_seq=0 2024-04-23T14:52:19.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.937+0000 7f665cc00640 10 client.26522 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:19.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.938+0000 7f665cc00640 1 -- 172.21.15.28:0/3065932961 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c1cf4aaa80 con 0x55c1cf5bf540 2024-04-23T14:52:19.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.938+0000 7f665cc00640 10 client.26522 handle_command_reply: tid=0 2024-04-23T14:52:19.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.938+0000 7f6678d86740 1 client.26522 shutdown 2024-04-23T14:52:19.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.938+0000 7f6678d86740 1 -- 172.21.15.28:0/3065932961 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1cf55c130 msgr2=0x55c1cf5bdbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:19.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.938+0000 7f6678d86740 1 --2- 172.21.15.28:0/3065932961 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1cf55c130 0x55c1cf5bdbe0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6668010780 tx=0x7f66680046c0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.938+0000 7f6678d86740 1 -- 172.21.15.28:0/3065932961 shutdown_connections 2024-04-23T14:52:19.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.938+0000 7f6678d86740 1 --2- 172.21.15.28:0/3065932961 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c1cf5bf540 0x55c1cf5c1960 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.938+0000 7f6678d86740 1 --2- 172.21.15.28:0/3065932961 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1cf55c130 0x55c1cf5bdbe0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.938+0000 7f6678d86740 1 --2- 172.21.15.28:0/3065932961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1cf548850 0x55c1cf55bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.938+0000 7f6678d86740 1 --2- 172.21.15.28:0/3065932961 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c1cf53e180 0x55c1cf55b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.938+0000 7f665cc00640 0 client.26522 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:19.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.938+0000 7f6678d86740 1 -- 172.21.15.28:0/3065932961 >> 172.21.15.28:0/3065932961 conn(0x55c1cf539b80 msgr2=0x55c1cf5542a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:19.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.938+0000 7f6678d86740 1 -- 172.21.15.28:0/3065932961 shutdown_connections 2024-04-23T14:52:19.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.938+0000 7f6678d86740 1 -- 172.21.15.28:0/3065932961 wait complete. 2024-04-23T14:52:19.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.938+0000 7f6678d86740 20 client.26522 trim_cache size 0 max 16384 2024-04-23T14:52:19.946 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:52:19.946 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-23T14:52:19.946 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-23T14:52:19.946 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:52:19.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.939+0000 7f6675e00640 1 -- 172.21.15.28:0/4027348836 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f66540a3d20 msgr2=0x7f66540a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:19.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.939+0000 7f6675e00640 1 --2- 172.21.15.28:0/4027348836 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f66540a3d20 0x7f66540a61e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6668009fd0 tx=0x7f6668004730 comp rx=0 tx=0).stop 2024-04-23T14:52:19.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.939+0000 7f6675e00640 1 -- 172.21.15.28:0/4027348836 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66701044b0 msgr2=0x7f6670076510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:19.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.939+0000 7f6675e00640 1 --2- 172.21.15.28:0/4027348836 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66701044b0 0x7f6670076510 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f666000ebc0 tx=0x7f666000c6a0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.939+0000 7f6675e00640 1 -- 172.21.15.28:0/4027348836 shutdown_connections 2024-04-23T14:52:19.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.939+0000 7f6675e00640 1 --2- 172.21.15.28:0/4027348836 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f66540a3d20 0x7f66540a61e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.939+0000 7f6675e00640 1 --2- 172.21.15.28:0/4027348836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6670076f90 0x7f66701ab190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.939+0000 7f6675e00640 1 --2- 172.21.15.28:0/4027348836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6670109580 0x7f6670076a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.939+0000 7f6675e00640 1 --2- 172.21.15.28:0/4027348836 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66701044b0 0x7f6670076510 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:19.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.939+0000 7f6675e00640 1 -- 172.21.15.28:0/4027348836 >> 172.21.15.28:0/4027348836 conn(0x7f66700ffd90 msgr2=0x7f6670101120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:19.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.939+0000 7f6675e00640 1 -- 172.21.15.28:0/4027348836 shutdown_connections 2024-04-23T14:52:19.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:19.939+0000 7f6675e00640 1 -- 172.21.15.28:0/4027348836 wait complete. 2024-04-23T14:52:19.959 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-23T14:52:19.959 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:52:20.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.215+0000 7f30d3e00640 1 -- 172.21.15.28:0/1359970018 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30d41084a0 msgr2=0x7f30d4108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:20.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.215+0000 7f30d3e00640 1 --2- 172.21.15.28:0/1359970018 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30d41084a0 0x7f30d4108900 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f30c0009900 tx=0x7f30c002a480 comp rx=0 tx=0).stop 2024-04-23T14:52:20.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.216+0000 7f30d3e00640 1 -- 172.21.15.28:0/1359970018 shutdown_connections 2024-04-23T14:52:20.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.216+0000 7f30d3e00640 1 --2- 172.21.15.28:0/1359970018 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30d41084a0 0x7f30d4108900 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.216+0000 7f30d3e00640 1 --2- 172.21.15.28:0/1359970018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30d4108e40 0x7f30d41152b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.216+0000 7f30d3e00640 1 --2- 172.21.15.28:0/1359970018 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f30d410d3c0 0x7f30d410d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.216+0000 7f30d3e00640 1 -- 172.21.15.28:0/1359970018 >> 172.21.15.28:0/1359970018 conn(0x7f30d4077b80 msgr2=0x7f30d4079fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:20.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.216+0000 7f30d3e00640 1 -- 172.21.15.28:0/1359970018 shutdown_connections 2024-04-23T14:52:20.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.216+0000 7f30d3e00640 1 -- 172.21.15.28:0/1359970018 wait complete. 2024-04-23T14:52:20.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.217+0000 7f30d3e00640 1 Processor -- start 2024-04-23T14:52:20.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.217+0000 7f30d3e00640 1 -- start start 2024-04-23T14:52:20.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.218+0000 7f30d3e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30d41084a0 0x7f30d41a4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:20.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.218+0000 7f30d3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30d4108e40 0x7f30d41a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:20.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.218+0000 7f30d3e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f30d410d3c0 0x7f30d419eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:20.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.218+0000 7f30d3e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f30d4118850 con 0x7f30d4108e40 2024-04-23T14:52:20.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.218+0000 7f30d3e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f30d41186d0 con 0x7f30d41084a0 2024-04-23T14:52:20.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.218+0000 7f30d3e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f30d41189d0 con 0x7f30d410d3c0 2024-04-23T14:52:20.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.218+0000 7f30d2a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30d41084a0 0x7f30d41a4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:20.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.218+0000 7f30d2a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30d41084a0 0x7f30d41a4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40402/0 (socket says 172.21.15.28:40402) 2024-04-23T14:52:20.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.218+0000 7f30d2a00640 1 -- 172.21.15.28:0/199379277 learned_addr learned my addr 172.21.15.28:0/199379277 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:20.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.218+0000 7f30d2000640 1 --2- 172.21.15.28:0/199379277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30d4108e40 0x7f30d41a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:20.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.218+0000 7f30d3400640 1 --2- 172.21.15.28:0/199379277 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f30d410d3c0 0x7f30d419eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:20.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.218+0000 7f30d2a00640 1 -- 172.21.15.28:0/199379277 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f30d410d3c0 msgr2=0x7f30d419eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:20.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.218+0000 7f30d2a00640 1 --2- 172.21.15.28:0/199379277 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f30d410d3c0 0x7f30d419eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.218+0000 7f30d2a00640 1 -- 172.21.15.28:0/199379277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30d4108e40 msgr2=0x7f30d41a4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:20.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.218+0000 7f30d2a00640 1 --2- 172.21.15.28:0/199379277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30d4108e40 0x7f30d41a4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.218+0000 7f30d2a00640 1 -- 172.21.15.28:0/199379277 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30d419f270 con 0x7f30d41084a0 2024-04-23T14:52:20.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.219+0000 7f30d2a00640 1 --2- 172.21.15.28:0/199379277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30d41084a0 0x7f30d41a4940 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f30c400e9f0 tx=0x7f30c400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:20.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.219+0000 7f30cb400640 1 -- 172.21.15.28:0/199379277 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c400cd80 con 0x7f30d41084a0 2024-04-23T14:52:20.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.219+0000 7f30cb400640 1 -- 172.21.15.28:0/199379277 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30c4005500 con 0x7f30d41084a0 2024-04-23T14:52:20.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.219+0000 7f30d3e00640 1 -- 172.21.15.28:0/199379277 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30d419f560 con 0x7f30d41084a0 2024-04-23T14:52:20.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.219+0000 7f30cb400640 1 -- 172.21.15.28:0/199379277 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c4010690 con 0x7f30d41084a0 2024-04-23T14:52:20.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.219+0000 7f30d3e00640 1 -- 172.21.15.28:0/199379277 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30d41ab800 con 0x7f30d41084a0 2024-04-23T14:52:20.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.220+0000 7f30d3e00640 1 -- 172.21.15.28:0/199379277 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30d4069190 con 0x7f30d41084a0 2024-04-23T14:52:20.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.222+0000 7f30cb400640 1 -- 172.21.15.28:0/199379277 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f30c40046c0 con 0x7f30d41084a0 2024-04-23T14:52:20.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.224+0000 7f30cb400640 1 --2- 172.21.15.28:0/199379277 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f30b80a3da0 0x7f30b80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:20.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.224+0000 7f30cb400640 1 -- 172.21.15.28:0/199379277 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7f30c40d0b90 con 0x7f30d41084a0 2024-04-23T14:52:20.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.224+0000 7f30d2000640 1 --2- 172.21.15.28:0/199379277 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f30b80a3da0 0x7f30b80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:20.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.225+0000 7f30d2000640 1 --2- 172.21.15.28:0/199379277 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f30b80a3da0 0x7f30b80a6260 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f30c0000c00 tx=0x7f30c0005b50 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:20.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.228+0000 7f30cb400640 1 -- 172.21.15.28:0/199379277 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f30c409d1c0 con 0x7f30d41084a0 2024-04-23T14:52:20.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.500+0000 7f30d3e00640 1 -- 172.21.15.28:0/199379277 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f30d402d620 con 0x7f30d41084a0 2024-04-23T14:52:20.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.501+0000 7f30cb400640 1 -- 172.21.15.28:0/199379277 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7f30c409cac0 con 0x7f30d41084a0 2024-04-23T14:52:20.501 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:52:20.501 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:52:20.502 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:52:20.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.503+0000 7f30d3e00640 1 -- 172.21.15.28:0/199379277 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f30b80a3da0 msgr2=0x7f30b80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:20.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.504+0000 7f30d3e00640 1 --2- 172.21.15.28:0/199379277 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f30b80a3da0 0x7f30b80a6260 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f30c0000c00 tx=0x7f30c0005b50 comp rx=0 tx=0).stop 2024-04-23T14:52:20.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.504+0000 7f30d3e00640 1 -- 172.21.15.28:0/199379277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30d41084a0 msgr2=0x7f30d41a4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:20.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.504+0000 7f30d3e00640 1 --2- 172.21.15.28:0/199379277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30d41084a0 0x7f30d41a4940 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f30c400e9f0 tx=0x7f30c400edf0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.504+0000 7f30d3e00640 1 -- 172.21.15.28:0/199379277 shutdown_connections 2024-04-23T14:52:20.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.504+0000 7f30d3e00640 1 --2- 172.21.15.28:0/199379277 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f30b80a3da0 0x7f30b80a6260 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.504+0000 7f30d3e00640 1 --2- 172.21.15.28:0/199379277 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f30d410d3c0 0x7f30d419eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.504+0000 7f30d3e00640 1 --2- 172.21.15.28:0/199379277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30d4108e40 0x7f30d41a4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.504+0000 7f30d3e00640 1 --2- 172.21.15.28:0/199379277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30d41084a0 0x7f30d41a4940 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.504+0000 7f30d3e00640 1 -- 172.21.15.28:0/199379277 >> 172.21.15.28:0/199379277 conn(0x7f30d4077b80 msgr2=0x7f30d40781a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:20.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.504+0000 7f30d3e00640 1 -- 172.21.15.28:0/199379277 shutdown_connections 2024-04-23T14:52:20.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.504+0000 7f30d3e00640 1 -- 172.21.15.28:0/199379277 wait complete. 2024-04-23T14:52:20.510 INFO:tasks.workunit.client.0.smithi028.stdout: 1 691149 8.97 MB/sec execute 276 sec latency 37.519 ms 2024-04-23T14:52:20.525 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-23T14:52:20.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.746+0000 7fe78fc00640 1 -- 172.21.15.28:0/833611011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe78806a910 msgr2=0x7fe78806acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:20.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.746+0000 7fe78fc00640 1 --2- 172.21.15.28:0/833611011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe78806a910 0x7fe78806acf0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe77c009900 tx=0x7fe77c02a3f0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.746+0000 7fe78fc00640 1 -- 172.21.15.28:0/833611011 shutdown_connections 2024-04-23T14:52:20.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.746+0000 7fe78fc00640 1 --2- 172.21.15.28:0/833611011 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe788107320 0x7fe78810eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.746+0000 7fe78fc00640 1 --2- 172.21.15.28:0/833611011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe78806b2c0 0x7fe788106c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.746+0000 7fe78fc00640 1 --2- 172.21.15.28:0/833611011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe78806a910 0x7fe78806acf0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.746+0000 7fe78fc00640 1 -- 172.21.15.28:0/833611011 >> 172.21.15.28:0/833611011 conn(0x7fe788077f60 msgr2=0x7fe788078370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:20.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.746+0000 7fe78fc00640 1 -- 172.21.15.28:0/833611011 shutdown_connections 2024-04-23T14:52:20.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.746+0000 7fe78fc00640 1 -- 172.21.15.28:0/833611011 wait complete. 2024-04-23T14:52:20.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.748+0000 7fe78fc00640 1 Processor -- start 2024-04-23T14:52:20.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.748+0000 7fe78fc00640 1 -- start start 2024-04-23T14:52:20.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.748+0000 7fe78fc00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe78806a910 0x7fe78819e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:20.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.749+0000 7fe78fc00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe788107320 0x7fe78819ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:20.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.749+0000 7fe78fc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe788198760 0x7fe788198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:20.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.749+0000 7fe78fc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe788112400 con 0x7fe788198760 2024-04-23T14:52:20.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.749+0000 7fe78fc00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe788112280 con 0x7fe788107320 2024-04-23T14:52:20.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.749+0000 7fe78fc00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe788112580 con 0x7fe78806a910 2024-04-23T14:52:20.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.749+0000 7fe787e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe788107320 0x7fe78819ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:20.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.749+0000 7fe787e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe788107320 0x7fe78819ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40432/0 (socket says 172.21.15.28:40432) 2024-04-23T14:52:20.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.749+0000 7fe787e00640 1 -- 172.21.15.28:0/1613908033 learned_addr learned my addr 172.21.15.28:0/1613908033 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:20.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.749+0000 7fe78d200640 1 --2- 172.21.15.28:0/1613908033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe78806a910 0x7fe78819e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:20.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.749+0000 7fe78dc00640 1 --2- 172.21.15.28:0/1613908033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe788198760 0x7fe788198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:20.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.749+0000 7fe787e00640 1 -- 172.21.15.28:0/1613908033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe78806a910 msgr2=0x7fe78819e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:20.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.749+0000 7fe787e00640 1 --2- 172.21.15.28:0/1613908033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe78806a910 0x7fe78819e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.749+0000 7fe787e00640 1 -- 172.21.15.28:0/1613908033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe788198760 msgr2=0x7fe788198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:20.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.749+0000 7fe787e00640 1 --2- 172.21.15.28:0/1613908033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe788198760 0x7fe788198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.749+0000 7fe787e00640 1 -- 172.21.15.28:0/1613908033 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe788199360 con 0x7fe788107320 2024-04-23T14:52:20.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.749+0000 7fe78dc00640 1 --2- 172.21.15.28:0/1613908033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe788198760 0x7fe788198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:52:20.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.750+0000 7fe787e00640 1 --2- 172.21.15.28:0/1613908033 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe788107320 0x7fe78819ebe0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe77400d770 tx=0x7fe77400dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:20.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.750+0000 7fe785600640 1 -- 172.21.15.28:0/1613908033 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe774017070 con 0x7fe788107320 2024-04-23T14:52:20.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.750+0000 7fe785600640 1 -- 172.21.15.28:0/1613908033 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe774009c30 con 0x7fe788107320 2024-04-23T14:52:20.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.750+0000 7fe785600640 1 -- 172.21.15.28:0/1613908033 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe77400f620 con 0x7fe788107320 2024-04-23T14:52:20.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.750+0000 7fe78fc00640 1 -- 172.21.15.28:0/1613908033 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7881a5440 con 0x7fe788107320 2024-04-23T14:52:20.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.750+0000 7fe78fc00640 1 -- 172.21.15.28:0/1613908033 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7881a57f0 con 0x7fe788107320 2024-04-23T14:52:20.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.753+0000 7fe792b92740 1 Processor -- start 2024-04-23T14:52:20.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.754+0000 7fe792b92740 1 -- start start 2024-04-23T14:52:20.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.754+0000 7fe792b92740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c84064e680 0x55c84064ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:20.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.754+0000 7fe792b92740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84064f030 0x55c84065d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:20.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.754+0000 7fe792b92740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c84065dd50 0x55c840660140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:20.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.754+0000 7fe792b92740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c8405b41d0 con 0x55c84064f030 2024-04-23T14:52:20.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.754+0000 7fe792b92740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c84055a450 con 0x55c84065dd50 2024-04-23T14:52:20.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.754+0000 7fe792b92740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c8405b0190 con 0x55c84064e680 2024-04-23T14:52:20.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.754+0000 7fe78d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84064f030 0x55c84065d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:20.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.754+0000 7fe78dc00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c84064e680 0x55c84064ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:20.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.754+0000 7fe785600640 1 -- 172.21.15.28:0/1613908033 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fe77401a070 con 0x7fe788107320 2024-04-23T14:52:20.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.754+0000 7fe78d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84064f030 0x55c84065d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53082/0 (socket says 172.21.15.28:53082) 2024-04-23T14:52:20.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.754+0000 7fe78d200640 1 -- 172.21.15.28:0/2753639700 learned_addr learned my addr 172.21.15.28:0/2753639700 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:20.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.755+0000 7fe787e00640 1 --2- 172.21.15.28:0/2753639700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c84065dd50 0x55c840660140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:20.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.755+0000 7fe78dc00640 1 -- 172.21.15.28:0/2753639700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c84065dd50 msgr2=0x55c840660140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:20.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.755+0000 7fe78dc00640 1 --2- 172.21.15.28:0/2753639700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c84065dd50 0x55c840660140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.755+0000 7fe78dc00640 1 -- 172.21.15.28:0/2753639700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84064f030 msgr2=0x55c84065d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:20.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.755+0000 7fe78dc00640 1 --2- 172.21.15.28:0/2753639700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84064f030 0x55c84065d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.755+0000 7fe78dc00640 1 -- 172.21.15.28:0/2753639700 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c840617310 con 0x55c84064e680 2024-04-23T14:52:20.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.755+0000 7fe787e00640 1 --2- 172.21.15.28:0/2753639700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c84065dd50 0x55c840660140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:52:20.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.755+0000 7fe78dc00640 1 --2- 172.21.15.28:0/2753639700 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c84064e680 0x55c84064ea60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe780009f40 tx=0x7fe780010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ed72c1e05c7c433 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:20.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.755+0000 7fe77a000640 1 -- 172.21.15.28:0/2753639700 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe780011700 con 0x55c84064e680 2024-04-23T14:52:20.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.756+0000 7fe77a000640 1 -- 172.21.15.28:0/2753639700 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe780011d80 con 0x55c84064e680 2024-04-23T14:52:20.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.756+0000 7fe77a000640 1 -- 172.21.15.28:0/2753639700 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe78001a380 con 0x55c84064e680 2024-04-23T14:52:20.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.756+0000 7fe785600640 1 --2- 172.21.15.28:0/1613908033 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe7700a3d20 0x7fe7700a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:20.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.756+0000 7fe785600640 1 -- 172.21.15.28:0/1613908033 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7fe7740d0330 con 0x7fe788107320 2024-04-23T14:52:20.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.756+0000 7fe78d200640 1 --2- 172.21.15.28:0/1613908033 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe7700a3d20 0x7fe7700a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:20.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.756+0000 7fe792b92740 1 -- 172.21.15.28:0/2753639700 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c84064e680 msgr2=0x55c84064ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:20.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.756+0000 7fe792b92740 1 --2- 172.21.15.28:0/2753639700 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c84064e680 0x55c84064ea60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe780009f40 tx=0x7fe780010860 comp rx=0 tx=0).stop 2024-04-23T14:52:20.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.756+0000 7fe792b92740 1 -- 172.21.15.28:0/2753639700 shutdown_connections 2024-04-23T14:52:20.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.756+0000 7fe792b92740 1 --2- 172.21.15.28:0/2753639700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c84065dd50 0x55c840660140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.756+0000 7fe792b92740 1 --2- 172.21.15.28:0/2753639700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84064f030 0x55c84065d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.756+0000 7fe792b92740 1 --2- 172.21.15.28:0/2753639700 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c84064e680 0x55c84064ea60 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.756+0000 7fe792b92740 1 -- 172.21.15.28:0/2753639700 >> 172.21.15.28:0/2753639700 conn(0x55c84064ac70 msgr2=0x55c84064b050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:20.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.757+0000 7fe792b92740 1 -- 172.21.15.28:0/2753639700 shutdown_connections 2024-04-23T14:52:20.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.757+0000 7fe792b92740 1 -- 172.21.15.28:0/2753639700 wait complete. 2024-04-23T14:52:20.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.757+0000 7fe792b92740 1 -- start start 2024-04-23T14:52:20.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.757+0000 7fe78d200640 1 --2- 172.21.15.28:0/1613908033 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe7700a3d20 0x7fe7700a61e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe77c005c40 tx=0x7fe77c005b50 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:20.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.757+0000 7fe792b92740 1 Processor -- start 2024-04-23T14:52:20.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.758+0000 7fe792b92740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c84064e680 0x55c84065d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:20.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.758+0000 7fe792b92740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c84064f030 0x55c84065aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:20.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.758+0000 7fe792b92740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84065dd50 0x55c84065b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:20.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.758+0000 7fe792b92740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c840660bb0 con 0x55c84065dd50 2024-04-23T14:52:20.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.758+0000 7fe792b92740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c84056d810 con 0x55c84064f030 2024-04-23T14:52:20.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.758+0000 7fe792b92740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c840660d30 con 0x55c84064e680 2024-04-23T14:52:20.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.758+0000 7fe792b92740 10 client.0 _ll_register_callbacks cb 0x55c84053b140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:52:20.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.758+0000 7fe787e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c84064f030 0x55c84065aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:20.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.758+0000 7fe787e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c84064f030 0x55c84065aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40448/0 (socket says 172.21.15.28:40448) 2024-04-23T14:52:20.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.758+0000 7fe78d200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c84064e680 0x55c84065d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:20.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.758+0000 7fe787e00640 1 -- 172.21.15.28:0/3136869470 learned_addr learned my addr 172.21.15.28:0/3136869470 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:20.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.758+0000 7fe78dc00640 1 --2- 172.21.15.28:0/3136869470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84065dd50 0x55c84065b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:20.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.758+0000 7fe787e00640 1 -- 172.21.15.28:0/3136869470 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c84064e680 msgr2=0x55c84065d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:20.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.758+0000 7fe787e00640 1 --2- 172.21.15.28:0/3136869470 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c84064e680 0x55c84065d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.758+0000 7fe787e00640 1 -- 172.21.15.28:0/3136869470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84065dd50 msgr2=0x55c84065b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:20.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.758+0000 7fe787e00640 1 --2- 172.21.15.28:0/3136869470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84065dd50 0x55c84065b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.758+0000 7fe787e00640 1 -- 172.21.15.28:0/3136869470 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c84057e3f0 con 0x55c84064f030 2024-04-23T14:52:20.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.759+0000 7fe787e00640 1 --2- 172.21.15.28:0/3136869470 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c84064f030 0x55c84065aae0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fe77409bfc0 tx=0x7fe77409bef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cf3224448d154a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:20.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.759+0000 7fe76ea00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:52:20.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.759+0000 7fe76ea00640 1 -- 172.21.15.28:0/3136869470 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe774021040 con 0x55c84064f030 2024-04-23T14:52:20.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.759+0000 7fe76ea00640 1 -- 172.21.15.28:0/3136869470 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7fe77409b710 con 0x55c84064f030 2024-04-23T14:52:20.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.759+0000 7fe792b92740 1 -- 172.21.15.28:0/3136869470 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe780011d80 con 0x55c84064f030 2024-04-23T14:52:20.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.759+0000 7fe76ea00640 1 -- 172.21.15.28:0/3136869470 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fe77409c2c0 con 0x55c84064f030 2024-04-23T14:52:20.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.759+0000 7fe792b92740 10 client.26537 fetch_fsmap learned FSMap version 20 2024-04-23T14:52:20.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.759+0000 7fe792b92740 1 -- 172.21.15.28:0/3136869470 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55c8402c63f0 con 0x55c84064f030 2024-04-23T14:52:20.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.759+0000 7fe76ea00640 1 -- 172.21.15.28:0/3136869470 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe780011d80 con 0x55c84064f030 2024-04-23T14:52:20.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.760+0000 7fe76ea00640 1 -- 172.21.15.28:0/3136869470 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fe774098130 con 0x55c84064f030 2024-04-23T14:52:20.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.760+0000 7fe792b92740 10 client.26537 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:52:20.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.760+0000 7fe792b92740 10 client.26537 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:52:20.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.760+0000 7fe792b92740 20 client.26537 populate_metadata read hostname 'smithi028' 2024-04-23T14:52:20.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.760+0000 7fe792b92740 1 --2- 172.21.15.28:0/3136869470 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c84066df30 0x55c8406703b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:20.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.760+0000 7fe792b92740 4 client.26537 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:52:20.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.760+0000 7fe792b92740 1 -- 172.21.15.28:0/3136869470 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8405c1bf0 con 0x55c84066df30 2024-04-23T14:52:20.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.760+0000 7fe78d200640 1 --2- 172.21.15.28:0/3136869470 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c84066df30 0x55c8406703b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:20.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.761+0000 7fe78d200640 1 --2- 172.21.15.28:0/3136869470 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c84066df30 0x55c8406703b0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3543b7510c967ab server_cookie=1e920bef125362be in_seq=0 out_seq=0 2024-04-23T14:52:20.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.761+0000 7fe76ea00640 10 client.26537 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:20.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.761+0000 7fe76ea00640 1 -- 172.21.15.28:0/3136869470 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c8405c1bf0 con 0x55c84066df30 2024-04-23T14:52:20.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.761+0000 7fe76ea00640 10 client.26537 handle_command_reply: tid=0 2024-04-23T14:52:20.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.762+0000 7fe792b92740 1 client.26537 shutdown 2024-04-23T14:52:20.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.762+0000 7fe792b92740 1 -- 172.21.15.28:0/3136869470 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c84064f030 msgr2=0x55c84065aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:20.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.762+0000 7fe792b92740 1 --2- 172.21.15.28:0/3136869470 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c84064f030 0x55c84065aae0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fe77409bfc0 tx=0x7fe77409bef0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.762+0000 7fe792b92740 1 -- 172.21.15.28:0/3136869470 shutdown_connections 2024-04-23T14:52:20.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.762+0000 7fe792b92740 1 --2- 172.21.15.28:0/3136869470 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c84066df30 0x55c8406703b0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.762+0000 7fe792b92740 1 --2- 172.21.15.28:0/3136869470 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c84064f030 0x55c84065aae0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.762+0000 7fe792b92740 1 --2- 172.21.15.28:0/3136869470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84065dd50 0x55c84065b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.762+0000 7fe792b92740 1 --2- 172.21.15.28:0/3136869470 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c84064e680 0x55c84065d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.762+0000 7fe76ea00640 0 client.26537 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:20.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.762+0000 7fe792b92740 1 -- 172.21.15.28:0/3136869470 >> 172.21.15.28:0/3136869470 conn(0x55c84064ac70 msgr2=0x55c840650ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:20.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.762+0000 7fe792b92740 1 -- 172.21.15.28:0/3136869470 shutdown_connections 2024-04-23T14:52:20.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.762+0000 7fe792b92740 1 -- 172.21.15.28:0/3136869470 wait complete. 2024-04-23T14:52:20.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.762+0000 7fe792b92740 20 client.26537 trim_cache size 0 max 16384 2024-04-23T14:52:20.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.779+0000 7fe792b92740 1 Processor -- start 2024-04-23T14:52:20.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.779+0000 7fe792b92740 1 -- start start 2024-04-23T14:52:20.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.779+0000 7fe792b92740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c84064e680 0x55c840669ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:20.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.779+0000 7fe792b92740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84064f030 0x55c840650940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:20.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.779+0000 7fe792b92740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c84065dd50 0x55c84065d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:20.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.779+0000 7fe792b92740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe7881a9bc0 con 0x55c84064f030 2024-04-23T14:52:20.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.779+0000 7fe792b92740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe7881ac420 con 0x55c84065dd50 2024-04-23T14:52:20.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.779+0000 7fe792b92740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe7881aa210 con 0x55c84064e680 2024-04-23T14:52:20.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.779+0000 7fe78dc00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c84065dd50 0x55c84065d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:20.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.779+0000 7fe787e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84064f030 0x55c840650940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:20.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.779+0000 7fe78dc00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c84065dd50 0x55c84065d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40458/0 (socket says 172.21.15.28:40458) 2024-04-23T14:52:20.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.779+0000 7fe78dc00640 1 -- 172.21.15.28:0/561070020 learned_addr learned my addr 172.21.15.28:0/561070020 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:20.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.779+0000 7fe78d200640 1 --2- 172.21.15.28:0/561070020 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c84064e680 0x55c840669ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:20.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.780+0000 7fe78dc00640 1 -- 172.21.15.28:0/561070020 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c84064e680 msgr2=0x55c840669ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:20.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.780+0000 7fe78dc00640 1 --2- 172.21.15.28:0/561070020 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c84064e680 0x55c840669ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.780+0000 7fe78dc00640 1 -- 172.21.15.28:0/561070020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84064f030 msgr2=0x55c840650940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:20.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.780+0000 7fe78dc00640 1 --2- 172.21.15.28:0/561070020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84064f030 0x55c840650940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.780+0000 7fe78dc00640 1 -- 172.21.15.28:0/561070020 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8403907d0 con 0x55c84065dd50 2024-04-23T14:52:20.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.780+0000 7fe78dc00640 1 --2- 172.21.15.28:0/561070020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c84065dd50 0x55c84065d320 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fe780009f10 tx=0x7fe780009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61b0c52731fd9eb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:20.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.780+0000 7fe779600640 1 -- 172.21.15.28:0/561070020 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe78002b020 con 0x55c84065dd50 2024-04-23T14:52:20.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.780+0000 7fe779600640 1 -- 172.21.15.28:0/561070020 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe78001c680 con 0x55c84065dd50 2024-04-23T14:52:20.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.780+0000 7fe779600640 1 -- 172.21.15.28:0/561070020 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe780005040 con 0x55c84065dd50 2024-04-23T14:52:20.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.781+0000 7fe792b92740 1 -- 172.21.15.28:0/561070020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c84065dd50 msgr2=0x55c84065d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:20.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.781+0000 7fe792b92740 1 --2- 172.21.15.28:0/561070020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c84065dd50 0x55c84065d320 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fe780009f10 tx=0x7fe780009c00 comp rx=0 tx=0).stop 2024-04-23T14:52:20.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.781+0000 7fe792b92740 1 -- 172.21.15.28:0/561070020 shutdown_connections 2024-04-23T14:52:20.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.781+0000 7fe792b92740 1 --2- 172.21.15.28:0/561070020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c84065dd50 0x55c84065d320 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.781+0000 7fe792b92740 1 --2- 172.21.15.28:0/561070020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84064f030 0x55c840650940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.781+0000 7fe792b92740 1 --2- 172.21.15.28:0/561070020 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c84064e680 0x55c840669ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.781+0000 7fe792b92740 1 -- 172.21.15.28:0/561070020 >> 172.21.15.28:0/561070020 conn(0x55c84064ac70 msgr2=0x55c840650230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:20.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.781+0000 7fe792b92740 1 -- 172.21.15.28:0/561070020 shutdown_connections 2024-04-23T14:52:20.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.781+0000 7fe792b92740 1 -- 172.21.15.28:0/561070020 wait complete. 2024-04-23T14:52:20.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.782+0000 7fe792b92740 1 -- start start 2024-04-23T14:52:20.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.782+0000 7fe792b92740 1 Processor -- start 2024-04-23T14:52:20.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.782+0000 7fe792b92740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c84064e680 0x55c84065efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:20.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.782+0000 7fe792b92740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c84064f030 0x55c84065f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:20.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.782+0000 7fe792b92740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84065dd50 0x55c840664590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:20.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.782+0000 7fe792b92740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe7881aa8e0 con 0x55c84065dd50 2024-04-23T14:52:20.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.782+0000 7fe792b92740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe7881abd50 con 0x55c84064f030 2024-04-23T14:52:20.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.782+0000 7fe792b92740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe7881aafb0 con 0x55c84064e680 2024-04-23T14:52:20.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.782+0000 7fe792b92740 10 client.0 _ll_register_callbacks cb 0x7fe77400d6c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:52:20.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.782+0000 7fe787e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c84064f030 0x55c84065f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:20.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.782+0000 7fe787e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c84064f030 0x55c84065f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40468/0 (socket says 172.21.15.28:40468) 2024-04-23T14:52:20.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.783+0000 7fe787e00640 1 -- 172.21.15.28:0/3131132371 learned_addr learned my addr 172.21.15.28:0/3131132371 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:20.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.783+0000 7fe78dc00640 1 --2- 172.21.15.28:0/3131132371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84065dd50 0x55c840664590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:20.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.783+0000 7fe78d200640 1 --2- 172.21.15.28:0/3131132371 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c84064e680 0x55c84065efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:20.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.783+0000 7fe787e00640 1 -- 172.21.15.28:0/3131132371 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c84064e680 msgr2=0x55c84065efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:20.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.783+0000 7fe787e00640 1 --2- 172.21.15.28:0/3131132371 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c84064e680 0x55c84065efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.783+0000 7fe787e00640 1 -- 172.21.15.28:0/3131132371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84065dd50 msgr2=0x55c840664590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:20.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.783+0000 7fe787e00640 1 --2- 172.21.15.28:0/3131132371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84065dd50 0x55c840664590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.783+0000 7fe787e00640 1 -- 172.21.15.28:0/3131132371 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c840664ad0 con 0x55c84064f030 2024-04-23T14:52:20.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.783+0000 7fe787e00640 1 --2- 172.21.15.28:0/3131132371 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c84064f030 0x55c84065f520 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe774005c60 tx=0x7fe77409b620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bebd805d9b7a0604 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:20.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.783+0000 7fe76ea00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:52:20.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.783+0000 7fe76ea00640 1 -- 172.21.15.28:0/3131132371 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe774014070 con 0x55c84064f030 2024-04-23T14:52:20.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.784+0000 7fe76ea00640 1 -- 172.21.15.28:0/3131132371 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7fe774098220 con 0x55c84064f030 2024-04-23T14:52:20.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.784+0000 7fe792b92740 1 -- 172.21.15.28:0/3131132371 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe78001c680 con 0x55c84064f030 2024-04-23T14:52:20.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.784+0000 7fe76ea00640 1 -- 172.21.15.28:0/3131132371 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fe774021040 con 0x55c84064f030 2024-04-23T14:52:20.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.784+0000 7fe792b92740 10 client.26543 fetch_fsmap learned FSMap version 20 2024-04-23T14:52:20.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.784+0000 7fe792b92740 1 -- 172.21.15.28:0/3131132371 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55c8406d4bf0 con 0x55c84064f030 2024-04-23T14:52:20.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.784+0000 7fe76ea00640 1 -- 172.21.15.28:0/3131132371 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe78001c680 con 0x55c84064f030 2024-04-23T14:52:20.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.784+0000 7fe76ea00640 1 -- 172.21.15.28:0/3131132371 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fe7740a9080 con 0x55c84064f030 2024-04-23T14:52:20.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.784+0000 7fe792b92740 10 client.26543 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:52:20.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.784+0000 7fe792b92740 10 client.26543 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:52:20.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.784+0000 7fe792b92740 20 client.26543 populate_metadata read hostname 'smithi028' 2024-04-23T14:52:20.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.784+0000 7fe792b92740 1 --2- 172.21.15.28:0/3131132371 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c8406d51d0 0x55c8406d75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:20.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.784+0000 7fe792b92740 4 client.26543 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-23T14:52:20.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.784+0000 7fe792b92740 1 -- 172.21.15.28:0/3131132371 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c8405bba60 con 0x55c8406d51d0 2024-04-23T14:52:20.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.785+0000 7fe78d200640 1 --2- 172.21.15.28:0/3131132371 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c8406d51d0 0x55c8406d75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:20.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.785+0000 7fe78d200640 1 --2- 172.21.15.28:0/3131132371 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c8406d51d0 0x55c8406d75f0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7eb87a878808cd8b server_cookie=a97ac574508a2a90 in_seq=0 out_seq=0 2024-04-23T14:52:20.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.785+0000 7fe76ea00640 10 client.26543 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:20.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.787+0000 7fe76ea00640 1 -- 172.21.15.28:0/3131132371 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c8405bba60 con 0x55c8406d51d0 2024-04-23T14:52:20.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.787+0000 7fe76ea00640 10 client.26543 handle_command_reply: tid=0 2024-04-23T14:52:20.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.787+0000 7fe792b92740 1 client.26543 shutdown 2024-04-23T14:52:20.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.787+0000 7fe792b92740 1 -- 172.21.15.28:0/3131132371 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c84064f030 msgr2=0x55c84065f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:20.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.787+0000 7fe792b92740 1 --2- 172.21.15.28:0/3131132371 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c84064f030 0x55c84065f520 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe774005c60 tx=0x7fe77409b620 comp rx=0 tx=0).stop 2024-04-23T14:52:20.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.787+0000 7fe792b92740 1 -- 172.21.15.28:0/3131132371 shutdown_connections 2024-04-23T14:52:20.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.787+0000 7fe792b92740 1 --2- 172.21.15.28:0/3131132371 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c8406d51d0 0x55c8406d75f0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.787+0000 7fe792b92740 1 --2- 172.21.15.28:0/3131132371 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c84064f030 0x55c84065f520 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.787+0000 7fe792b92740 1 --2- 172.21.15.28:0/3131132371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84065dd50 0x55c840664590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.787+0000 7fe76ea00640 0 client.26543 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:20.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.787+0000 7fe792b92740 1 --2- 172.21.15.28:0/3131132371 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c84064e680 0x55c84065efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.787+0000 7fe792b92740 1 -- 172.21.15.28:0/3131132371 >> 172.21.15.28:0/3131132371 conn(0x55c84064ac70 msgr2=0x55c840650ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:20.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.787+0000 7fe792b92740 1 -- 172.21.15.28:0/3131132371 shutdown_connections 2024-04-23T14:52:20.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.787+0000 7fe792b92740 1 -- 172.21.15.28:0/3131132371 wait complete. 2024-04-23T14:52:20.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.787+0000 7fe792b92740 20 client.26543 trim_cache size 0 max 16384 2024-04-23T14:52:20.795 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-23T14:52:20.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.788+0000 7fe78fc00640 1 -- 172.21.15.28:0/1613908033 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe7700a3d20 msgr2=0x7fe7700a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:20.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.788+0000 7fe78fc00640 1 --2- 172.21.15.28:0/1613908033 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe7700a3d20 0x7fe7700a61e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe77c005c40 tx=0x7fe77c005b50 comp rx=0 tx=0).stop 2024-04-23T14:52:20.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.788+0000 7fe78fc00640 1 -- 172.21.15.28:0/1613908033 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe788107320 msgr2=0x7fe78819ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:20.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.788+0000 7fe78fc00640 1 --2- 172.21.15.28:0/1613908033 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe788107320 0x7fe78819ebe0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe77400d770 tx=0x7fe77400dc30 comp rx=0 tx=0).stop 2024-04-23T14:52:20.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.788+0000 7fe78fc00640 1 -- 172.21.15.28:0/1613908033 shutdown_connections 2024-04-23T14:52:20.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.788+0000 7fe78fc00640 1 --2- 172.21.15.28:0/1613908033 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe7700a3d20 0x7fe7700a61e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.788+0000 7fe78fc00640 1 --2- 172.21.15.28:0/1613908033 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe788107320 0x7fe78819ebe0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.788+0000 7fe78fc00640 1 --2- 172.21.15.28:0/1613908033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe788198760 0x7fe788198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.788+0000 7fe78fc00640 1 --2- 172.21.15.28:0/1613908033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe78806a910 0x7fe78819e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:20.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.788+0000 7fe78fc00640 1 -- 172.21.15.28:0/1613908033 >> 172.21.15.28:0/1613908033 conn(0x7fe788077f60 msgr2=0x7fe7880fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:20.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.788+0000 7fe78fc00640 1 -- 172.21.15.28:0/1613908033 shutdown_connections 2024-04-23T14:52:20.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:20.788+0000 7fe78fc00640 1 -- 172.21.15.28:0/1613908033 wait complete. 2024-04-23T14:52:20.803 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:20 smithi028 ceph-mon[27810]: pgmap v256: 97 pgs: 97 active+clean; 436 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 154 op/s 2024-04-23T14:52:20.803 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:20 smithi028 ceph-mon[27810]: scrub queued for path: / 2024-04-23T14:52:20.803 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:20 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:52:20.804 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:20 smithi028 ceph-mon[27810]: scrub summary: active paths [/] 2024-04-23T14:52:20.804 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:20 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:52:20.804 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:20 smithi028 ceph-mon[27810]: scrub summary: idle 2024-04-23T14:52:20.804 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:20 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/199379277' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:52:21.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:20 smithi105 ceph-mon[32555]: pgmap v256: 97 pgs: 97 active+clean; 436 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 154 op/s 2024-04-23T14:52:21.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:20 smithi105 ceph-mon[32555]: scrub queued for path: / 2024-04-23T14:52:21.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:20 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:52:21.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:20 smithi105 ceph-mon[32555]: scrub summary: active paths [/] 2024-04-23T14:52:21.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:20 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:52:21.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:20 smithi105 ceph-mon[32555]: scrub summary: idle 2024-04-23T14:52:21.037 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:20 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/199379277' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:52:21.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:20 smithi038 ceph-mon[32348]: pgmap v256: 97 pgs: 97 active+clean; 436 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 154 op/s 2024-04-23T14:52:21.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:20 smithi038 ceph-mon[32348]: scrub queued for path: / 2024-04-23T14:52:21.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:20 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:52:21.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:20 smithi038 ceph-mon[32348]: scrub summary: active paths [/] 2024-04-23T14:52:21.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:20 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:52:21.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:20 smithi038 ceph-mon[32348]: scrub summary: idle 2024-04-23T14:52:21.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:20 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/199379277' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:52:21.511 INFO:tasks.workunit.client.0.smithi028.stdout: 1 693158 8.99 MB/sec execute 277 sec latency 12.836 ms 2024-04-23T14:52:21.806 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-23T14:52:21.806 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:52:21.806 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-23T14:52:22.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.044+0000 7f1191a00640 1 -- 172.21.15.28:0/1981535859 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f118c10c5c0 msgr2=0x7f118c10ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:22.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.044+0000 7f1191a00640 1 --2- 172.21.15.28:0/1981535859 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f118c10c5c0 0x7f118c10ea60 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f1184009900 tx=0x7f118402a510 comp rx=0 tx=0).stop 2024-04-23T14:52:22.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.044+0000 7f1191a00640 1 -- 172.21.15.28:0/1981535859 shutdown_connections 2024-04-23T14:52:22.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.044+0000 7f1191a00640 1 --2- 172.21.15.28:0/1981535859 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f118c10c5c0 0x7f118c10ea60 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.044+0000 7f1191a00640 1 --2- 172.21.15.28:0/1981535859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f118c103ba0 0x7f118c10c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.044+0000 7f1191a00640 1 --2- 172.21.15.28:0/1981535859 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f118c103280 0x7f118c103660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.044+0000 7f1191a00640 1 -- 172.21.15.28:0/1981535859 >> 172.21.15.28:0/1981535859 conn(0x7f118c078000 msgr2=0x7f118c078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:22.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.044+0000 7f1191a00640 1 -- 172.21.15.28:0/1981535859 shutdown_connections 2024-04-23T14:52:22.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.044+0000 7f1191a00640 1 -- 172.21.15.28:0/1981535859 wait complete. 2024-04-23T14:52:22.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.045+0000 7f1191a00640 1 Processor -- start 2024-04-23T14:52:22.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.045+0000 7f1191a00640 1 -- start start 2024-04-23T14:52:22.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.045+0000 7f1191a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f118c103280 0x7f118c1a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.046+0000 7f1191a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f118c103ba0 0x7f118c1a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.046+0000 7f1191a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f118c10c5c0 0x7f118c19a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.046+0000 7f1191a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f118c112380 con 0x7f118c10c5c0 2024-04-23T14:52:22.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.046+0000 7f1191a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f118c112200 con 0x7f118c103ba0 2024-04-23T14:52:22.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.046+0000 7f1191a00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f118c112500 con 0x7f118c103280 2024-04-23T14:52:22.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.046+0000 7f118aa00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f118c103280 0x7f118c1a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:22.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.046+0000 7f118a000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f118c103ba0 0x7f118c1a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:22.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.046+0000 7f118a000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f118c103ba0 0x7f118c1a0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40478/0 (socket says 172.21.15.28:40478) 2024-04-23T14:52:22.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.046+0000 7f118a000640 1 -- 172.21.15.28:0/1437732339 learned_addr learned my addr 172.21.15.28:0/1437732339 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:22.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.046+0000 7f118b400640 1 --2- 172.21.15.28:0/1437732339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f118c10c5c0 0x7f118c19a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:22.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.046+0000 7f118aa00640 1 -- 172.21.15.28:0/1437732339 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f118c103ba0 msgr2=0x7f118c1a0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:22.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.046+0000 7f118aa00640 1 --2- 172.21.15.28:0/1437732339 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f118c103ba0 0x7f118c1a0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.046+0000 7f118aa00640 1 -- 172.21.15.28:0/1437732339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f118c10c5c0 msgr2=0x7f118c19a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:22.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.046+0000 7f118aa00640 1 --2- 172.21.15.28:0/1437732339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f118c10c5c0 0x7f118c19a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.046+0000 7f118aa00640 1 -- 172.21.15.28:0/1437732339 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f118c19b100 con 0x7f118c103280 2024-04-23T14:52:22.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.046+0000 7f118a000640 1 --2- 172.21.15.28:0/1437732339 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f118c103ba0 0x7f118c1a0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:52:22.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.047+0000 7f118aa00640 1 --2- 172.21.15.28:0/1437732339 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f118c103280 0x7f118c1a07d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f117c00eb00 tx=0x7f117c00c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:22.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.047+0000 7f1183400640 1 -- 172.21.15.28:0/1437732339 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f117c010430 con 0x7f118c103280 2024-04-23T14:52:22.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.047+0000 7f1183400640 1 -- 172.21.15.28:0/1437732339 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f117c010ab0 con 0x7f118c103280 2024-04-23T14:52:22.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.047+0000 7f1183400640 1 -- 172.21.15.28:0/1437732339 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f117c017bb0 con 0x7f118c103280 2024-04-23T14:52:22.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.047+0000 7f1191a00640 1 -- 172.21.15.28:0/1437732339 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f118c19b3f0 con 0x7f118c103280 2024-04-23T14:52:22.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.047+0000 7f1191a00640 1 -- 172.21.15.28:0/1437732339 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f118c1a7690 con 0x7f118c103280 2024-04-23T14:52:22.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.051+0000 7f1183400640 1 -- 172.21.15.28:0/1437732339 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f117c010c60 con 0x7f118c103280 2024-04-23T14:52:22.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.051+0000 7f1194781740 1 Processor -- start 2024-04-23T14:52:22.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.051+0000 7f1194781740 1 -- start start 2024-04-23T14:52:22.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.052+0000 7f1194781740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636e9f0b090 0x5636e9f0b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.052+0000 7f1194781740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636e9f0b9b0 0x5636e9f15b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.052+0000 7f1194781740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636e9f16080 0x5636e9f18470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.052+0000 7f1194781740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5636e9e70e70 con 0x5636e9f16080 2024-04-23T14:52:22.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.052+0000 7f1194781740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5636e9e17380 con 0x5636e9f0b090 2024-04-23T14:52:22.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.052+0000 7f1194781740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5636e9e6ce30 con 0x5636e9f0b9b0 2024-04-23T14:52:22.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.052+0000 7f118b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636e9f0b090 0x5636e9f0b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:22.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.052+0000 7f118b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636e9f0b090 0x5636e9f0b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40490/0 (socket says 172.21.15.28:40490) 2024-04-23T14:52:22.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.052+0000 7f118b400640 1 -- 172.21.15.28:0/2358690776 learned_addr learned my addr 172.21.15.28:0/2358690776 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:22.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.052+0000 7f118a000640 1 --2- 172.21.15.28:0/2358690776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636e9f16080 0x5636e9f18470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:22.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.052+0000 7f118aa00640 1 --2- 172.21.15.28:0/2358690776 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636e9f0b9b0 0x5636e9f15b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:22.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.052+0000 7f118b400640 1 -- 172.21.15.28:0/2358690776 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636e9f0b9b0 msgr2=0x5636e9f15b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:22.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.052+0000 7f118b400640 1 --2- 172.21.15.28:0/2358690776 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636e9f0b9b0 0x5636e9f15b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.052+0000 7f118b400640 1 -- 172.21.15.28:0/2358690776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636e9f16080 msgr2=0x5636e9f18470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:22.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.052+0000 7f118b400640 1 --2- 172.21.15.28:0/2358690776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636e9f16080 0x5636e9f18470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.052+0000 7f118b400640 1 -- 172.21.15.28:0/2358690776 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636e9c4b7e0 con 0x5636e9f0b090 2024-04-23T14:52:22.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.053+0000 7f1183400640 1 --2- 172.21.15.28:0/1437732339 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f11700a3de0 0x7f11700a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.053+0000 7f118b400640 1 --2- 172.21.15.28:0/2358690776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636e9f0b090 0x5636e9f0b470 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f118402ab00 tx=0x7f1184031400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18eb54b82061420c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:22.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.053+0000 7f1183400640 1 -- 172.21.15.28:0/1437732339 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7f117c0d1150 con 0x7f118c103280 2024-04-23T14:52:22.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.053+0000 7f118aa00640 1 --2- 172.21.15.28:0/1437732339 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f11700a3de0 0x7f11700a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:22.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.053+0000 7f117be00640 1 -- 172.21.15.28:0/2358690776 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1184035040 con 0x5636e9f0b090 2024-04-23T14:52:22.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.053+0000 7f117be00640 1 -- 172.21.15.28:0/2358690776 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1184005520 con 0x5636e9f0b090 2024-04-23T14:52:22.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.053+0000 7f117be00640 1 -- 172.21.15.28:0/2358690776 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11840035f0 con 0x5636e9f0b090 2024-04-23T14:52:22.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.054+0000 7f1194781740 1 -- 172.21.15.28:0/2358690776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636e9f0b090 msgr2=0x5636e9f0b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:22.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.054+0000 7f1194781740 1 --2- 172.21.15.28:0/2358690776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636e9f0b090 0x5636e9f0b470 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f118402ab00 tx=0x7f1184031400 comp rx=0 tx=0).stop 2024-04-23T14:52:22.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.054+0000 7f118aa00640 1 --2- 172.21.15.28:0/1437732339 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f11700a3de0 0x7f11700a62a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x5636e9f0c650 tx=0x7f117c000f40 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:22.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.054+0000 7f1194781740 1 -- 172.21.15.28:0/2358690776 shutdown_connections 2024-04-23T14:52:22.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.054+0000 7f1194781740 1 --2- 172.21.15.28:0/2358690776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636e9f16080 0x5636e9f18470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.054+0000 7f1194781740 1 --2- 172.21.15.28:0/2358690776 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636e9f0b9b0 0x5636e9f15b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.054+0000 7f1194781740 1 --2- 172.21.15.28:0/2358690776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636e9f0b090 0x5636e9f0b470 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.054+0000 7f1194781740 1 -- 172.21.15.28:0/2358690776 >> 172.21.15.28:0/2358690776 conn(0x5636e9f073b0 msgr2=0x5636e9f07790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:22.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.054+0000 7f1194781740 1 -- 172.21.15.28:0/2358690776 shutdown_connections 2024-04-23T14:52:22.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.054+0000 7f1194781740 1 -- 172.21.15.28:0/2358690776 wait complete. 2024-04-23T14:52:22.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.055+0000 7f1194781740 1 -- start start 2024-04-23T14:52:22.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.055+0000 7f1194781740 1 Processor -- start 2024-04-23T14:52:22.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.056+0000 7f1194781740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636e9f0b090 0x5636e9f13030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.056+0000 7f1194781740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636e9f0b9b0 0x5636e9f13570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.056+0000 7f1194781740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636e9f16080 0x5636e9f13ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.056+0000 7f1194781740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5636e9f18e10 con 0x5636e9f0b090 2024-04-23T14:52:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.056+0000 7f1194781740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5636e9e2a660 con 0x5636e9f16080 2024-04-23T14:52:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.056+0000 7f1194781740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5636e9f18f90 con 0x5636e9f0b9b0 2024-04-23T14:52:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.056+0000 7f1194781740 10 client.0 _ll_register_callbacks cb 0x5636e9e18aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:52:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.056+0000 7f118a000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636e9f0b9b0 0x5636e9f13570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.056+0000 7f118b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636e9f0b090 0x5636e9f13030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.056+0000 7f118a000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636e9f0b9b0 0x5636e9f13570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:42324/0 (socket says 172.21.15.28:42324) 2024-04-23T14:52:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.056+0000 7f118a000640 1 -- 172.21.15.28:0/2629166411 learned_addr learned my addr 172.21.15.28:0/2629166411 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.056+0000 7f118b400640 1 --2- 172.21.15.28:0/2629166411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636e9f16080 0x5636e9f13ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.057+0000 7f118a000640 1 -- 172.21.15.28:0/2629166411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636e9f16080 msgr2=0x5636e9f13ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.057+0000 7f118a000640 1 --2- 172.21.15.28:0/2629166411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636e9f16080 0x5636e9f13ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.057+0000 7f118a000640 1 -- 172.21.15.28:0/2629166411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636e9f0b090 msgr2=0x5636e9f13030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:22.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.057+0000 7f118a000640 1 --2- 172.21.15.28:0/2629166411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636e9f0b090 0x5636e9f13030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.057+0000 7f118a000640 1 -- 172.21.15.28:0/2629166411 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636e9cabf70 con 0x5636e9f0b9b0 2024-04-23T14:52:22.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.057+0000 7f118a000640 1 --2- 172.21.15.28:0/2629166411 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636e9f0b9b0 0x5636e9f13570 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f1174012d50 tx=0x7f117400f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed36b1600e972b35 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:22.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.057+0000 7f1178c00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:52:22.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.057+0000 7f1178c00640 1 -- 172.21.15.28:0/2629166411 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1174012110 con 0x5636e9f0b9b0 2024-04-23T14:52:22.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.057+0000 7f1178c00640 1 -- 172.21.15.28:0/2629166411 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7f1174004050 con 0x5636e9f0b9b0 2024-04-23T14:52:22.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.057+0000 7f1194781740 1 -- 172.21.15.28:0/2629166411 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1184005520 con 0x5636e9f0b9b0 2024-04-23T14:52:22.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.058+0000 7f1178c00640 1 -- 172.21.15.28:0/2629166411 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f11740122b0 con 0x5636e9f0b9b0 2024-04-23T14:52:22.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.058+0000 7f1194781740 10 client.26539 fetch_fsmap learned FSMap version 20 2024-04-23T14:52:22.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.058+0000 7f1194781740 1 -- 172.21.15.28:0/2629166411 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5636e9b43170 con 0x5636e9f0b9b0 2024-04-23T14:52:22.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.058+0000 7f1178c00640 1 -- 172.21.15.28:0/2629166411 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1184005520 con 0x5636e9f0b9b0 2024-04-23T14:52:22.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.058+0000 7f1178c00640 1 -- 172.21.15.28:0/2629166411 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f1174010580 con 0x5636e9f0b9b0 2024-04-23T14:52:22.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.058+0000 7f1194781740 10 client.26539 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:52:22.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.058+0000 7f1194781740 10 client.26539 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:52:22.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.058+0000 7f1194781740 20 client.26539 populate_metadata read hostname 'smithi028' 2024-04-23T14:52:22.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.058+0000 7f1194781740 1 --2- 172.21.15.28:0/2629166411 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5636e9f26450 0x5636e9f28880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.058+0000 7f1194781740 4 client.26539 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:52:22.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.058+0000 7f1194781740 1 -- 172.21.15.28:0/2629166411 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5636e9e7e890 con 0x5636e9f26450 2024-04-23T14:52:22.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.059+0000 7f118b400640 1 --2- 172.21.15.28:0/2629166411 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5636e9f26450 0x5636e9f28880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:22.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.059+0000 7f118b400640 1 --2- 172.21.15.28:0/2629166411 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5636e9f26450 0x5636e9f28880 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ca26e6b257ff2 server_cookie=e9549351b9d98ee in_seq=0 out_seq=0 2024-04-23T14:52:22.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.059+0000 7f1178c00640 10 client.26539 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:22.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.060+0000 7f1178c00640 1 -- 172.21.15.28:0/2629166411 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5636e9e7e890 con 0x5636e9f26450 2024-04-23T14:52:22.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.060+0000 7f1178c00640 10 client.26539 handle_command_reply: tid=0 2024-04-23T14:52:22.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.060+0000 7f1194781740 1 client.26539 shutdown 2024-04-23T14:52:22.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.061+0000 7f1194781740 1 -- 172.21.15.28:0/2629166411 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636e9f0b9b0 msgr2=0x5636e9f13570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:22.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.061+0000 7f1194781740 1 --2- 172.21.15.28:0/2629166411 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636e9f0b9b0 0x5636e9f13570 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f1174012d50 tx=0x7f117400f790 comp rx=0 tx=0).stop 2024-04-23T14:52:22.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.061+0000 7f1194781740 1 -- 172.21.15.28:0/2629166411 shutdown_connections 2024-04-23T14:52:22.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.061+0000 7f1194781740 1 --2- 172.21.15.28:0/2629166411 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5636e9f26450 0x5636e9f28880 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.061+0000 7f1194781740 1 --2- 172.21.15.28:0/2629166411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636e9f16080 0x5636e9f13ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.061+0000 7f1194781740 1 --2- 172.21.15.28:0/2629166411 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636e9f0b9b0 0x5636e9f13570 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.061+0000 7f1194781740 1 --2- 172.21.15.28:0/2629166411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636e9f0b090 0x5636e9f13030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.061+0000 7f1194781740 1 -- 172.21.15.28:0/2629166411 >> 172.21.15.28:0/2629166411 conn(0x5636e9f073b0 msgr2=0x5636e9f0d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:22.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.061+0000 7f1178c00640 0 client.26539 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:22.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.062+0000 7f1194781740 1 -- 172.21.15.28:0/2629166411 shutdown_connections 2024-04-23T14:52:22.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.062+0000 7f1194781740 1 -- 172.21.15.28:0/2629166411 wait complete. 2024-04-23T14:52:22.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.062+0000 7f1194781740 20 client.26539 trim_cache size 0 max 16384 2024-04-23T14:52:22.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.085+0000 7f1194781740 1 Processor -- start 2024-04-23T14:52:22.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.086+0000 7f1194781740 1 -- start start 2024-04-23T14:52:22.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.086+0000 7f1194781740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636e9f0b090 0x5636e9f21d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.086+0000 7f1194781740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636e9f0b9b0 0x5636e9f11f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.086+0000 7f1194781740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636e9f16080 0x5636e9f2e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.086+0000 7f1194781740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f118c1adfc0 con 0x5636e9f16080 2024-04-23T14:52:22.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.086+0000 7f1194781740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f118c1b0820 con 0x5636e9f0b9b0 2024-04-23T14:52:22.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.086+0000 7f1194781740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f118c1ae610 con 0x5636e9f0b090 2024-04-23T14:52:22.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.086+0000 7f118a000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636e9f0b9b0 0x5636e9f11f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:22.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.086+0000 7f118a000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636e9f0b9b0 0x5636e9f11f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40508/0 (socket says 172.21.15.28:40508) 2024-04-23T14:52:22.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.086+0000 7f118a000640 1 -- 172.21.15.28:0/1715360329 learned_addr learned my addr 172.21.15.28:0/1715360329 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:22.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.086+0000 7f118b400640 1 --2- 172.21.15.28:0/1715360329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636e9f16080 0x5636e9f2e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:22.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.087+0000 7f118a000640 1 -- 172.21.15.28:0/1715360329 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636e9f0b090 msgr2=0x5636e9f21d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:22.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.087+0000 7f118a000640 1 --2- 172.21.15.28:0/1715360329 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636e9f0b090 0x5636e9f21d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.087+0000 7f118a000640 1 -- 172.21.15.28:0/1715360329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636e9f16080 msgr2=0x5636e9f2e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:22.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.087+0000 7f118a000640 1 --2- 172.21.15.28:0/1715360329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636e9f16080 0x5636e9f2e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.087+0000 7f118a000640 1 -- 172.21.15.28:0/1715360329 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636e9f18470 con 0x5636e9f0b9b0 2024-04-23T14:52:22.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.087+0000 7f118a000640 1 --2- 172.21.15.28:0/1715360329 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636e9f0b9b0 0x5636e9f11f70 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f1174000c00 tx=0x7f1174009c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b25549bf3166d253 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:22.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.087+0000 7f117b400640 1 -- 172.21.15.28:0/1715360329 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1174028070 con 0x5636e9f0b9b0 2024-04-23T14:52:22.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.087+0000 7f117b400640 1 -- 172.21.15.28:0/1715360329 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1174010c10 con 0x5636e9f0b9b0 2024-04-23T14:52:22.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.087+0000 7f117b400640 1 -- 172.21.15.28:0/1715360329 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11740046c0 con 0x5636e9f0b9b0 2024-04-23T14:52:22.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.088+0000 7f1194781740 1 -- 172.21.15.28:0/1715360329 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636e9f0b9b0 msgr2=0x5636e9f11f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:22.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.088+0000 7f1194781740 1 --2- 172.21.15.28:0/1715360329 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636e9f0b9b0 0x5636e9f11f70 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f1174000c00 tx=0x7f1174009c70 comp rx=0 tx=0).stop 2024-04-23T14:52:22.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.088+0000 7f1194781740 1 -- 172.21.15.28:0/1715360329 shutdown_connections 2024-04-23T14:52:22.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.088+0000 7f1194781740 1 --2- 172.21.15.28:0/1715360329 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636e9f0b9b0 0x5636e9f11f70 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.088+0000 7f1194781740 1 --2- 172.21.15.28:0/1715360329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636e9f16080 0x5636e9f2e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.088+0000 7f1194781740 1 --2- 172.21.15.28:0/1715360329 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636e9f0b090 0x5636e9f21d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.088+0000 7f1194781740 1 -- 172.21.15.28:0/1715360329 >> 172.21.15.28:0/1715360329 conn(0x5636e9f073b0 msgr2=0x5636e9f06510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:22.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.088+0000 7f1194781740 1 -- 172.21.15.28:0/1715360329 shutdown_connections 2024-04-23T14:52:22.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.088+0000 7f1194781740 1 -- 172.21.15.28:0/1715360329 wait complete. 2024-04-23T14:52:22.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.089+0000 7f1194781740 1 -- start start 2024-04-23T14:52:22.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.089+0000 7f1194781740 1 Processor -- start 2024-04-23T14:52:22.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.089+0000 7f1194781740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636e9f0b090 0x5636e9f0fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.089+0000 7f1194781740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636e9f0b9b0 0x5636e9f10350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.089+0000 7f1194781740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636e9f16080 0x5636e9f10890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.089+0000 7f1194781740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f118c1aece0 con 0x5636e9f16080 2024-04-23T14:52:22.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.089+0000 7f1194781740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f118c1b0150 con 0x5636e9f0b9b0 2024-04-23T14:52:22.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.089+0000 7f1194781740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f118c1af3b0 con 0x5636e9f0b090 2024-04-23T14:52:22.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.089+0000 7f1194781740 10 client.0 _ll_register_callbacks cb 0x7f1174009d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:52:22.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.090+0000 7f118a000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636e9f0b9b0 0x5636e9f10350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:22.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.090+0000 7f118a000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636e9f0b9b0 0x5636e9f10350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40518/0 (socket says 172.21.15.28:40518) 2024-04-23T14:52:22.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.090+0000 7f118a000640 1 -- 172.21.15.28:0/1161769906 learned_addr learned my addr 172.21.15.28:0/1161769906 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:22.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.090+0000 7f118b400640 1 --2- 172.21.15.28:0/1161769906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636e9f16080 0x5636e9f10890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:22.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.090+0000 7f118b400640 1 --2- 172.21.15.28:0/1161769906 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636e9f0b090 0x5636e9f0fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:22.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.090+0000 7f118a000640 1 -- 172.21.15.28:0/1161769906 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636e9f0b090 msgr2=0x5636e9f0fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:22.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.090+0000 7f118a000640 1 --2- 172.21.15.28:0/1161769906 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636e9f0b090 0x5636e9f0fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.090+0000 7f118a000640 1 -- 172.21.15.28:0/1161769906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636e9f16080 msgr2=0x5636e9f10890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:22.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.090+0000 7f118a000640 1 --2- 172.21.15.28:0/1161769906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636e9f16080 0x5636e9f10890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.090+0000 7f118a000640 1 -- 172.21.15.28:0/1161769906 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636e9f8bb30 con 0x5636e9f0b9b0 2024-04-23T14:52:22.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.090+0000 7f118b400640 1 --2- 172.21.15.28:0/1161769906 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636e9f0b090 0x5636e9f0fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:52:22.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.090+0000 7f118a000640 1 --2- 172.21.15.28:0/1161769906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636e9f0b9b0 0x5636e9f10350 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f1174009d00 tx=0x7f1174010640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=690fe346ffe6bb89 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:22.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.090+0000 7f1178c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:52:22.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.091+0000 7f1178c00640 1 -- 172.21.15.28:0/1161769906 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1174028070 con 0x5636e9f0b9b0 2024-04-23T14:52:22.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.091+0000 7f1178c00640 1 -- 172.21.15.28:0/1161769906 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7f1174018040 con 0x5636e9f0b9b0 2024-04-23T14:52:22.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.091+0000 7f1194781740 1 -- 172.21.15.28:0/1161769906 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1174010c10 con 0x5636e9f0b9b0 2024-04-23T14:52:22.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.091+0000 7f1178c00640 1 -- 172.21.15.28:0/1161769906 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f1174011e40 con 0x5636e9f0b9b0 2024-04-23T14:52:22.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.091+0000 7f1194781740 10 client.26561 fetch_fsmap learned FSMap version 20 2024-04-23T14:52:22.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.091+0000 7f1194781740 1 -- 172.21.15.28:0/1161769906 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5636e9f8ce50 con 0x5636e9f0b9b0 2024-04-23T14:52:22.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.091+0000 7f1178c00640 1 -- 172.21.15.28:0/1161769906 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1174010c10 con 0x5636e9f0b9b0 2024-04-23T14:52:22.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.092+0000 7f1178c00640 1 -- 172.21.15.28:0/1161769906 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f1174035810 con 0x5636e9f0b9b0 2024-04-23T14:52:22.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.092+0000 7f1194781740 10 client.26561 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:52:22.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.092+0000 7f1194781740 10 client.26561 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:52:22.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.092+0000 7f1194781740 20 client.26561 populate_metadata read hostname 'smithi028' 2024-04-23T14:52:22.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.092+0000 7f1194781740 1 --2- 172.21.15.28:0/1161769906 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5636e9f8d4e0 0x5636e9f8f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.092+0000 7f1194781740 4 client.26561 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-23T14:52:22.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.092+0000 7f1194781740 1 -- 172.21.15.28:0/1161769906 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5636e9e78700 con 0x5636e9f8d4e0 2024-04-23T14:52:22.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.092+0000 7f118b400640 1 --2- 172.21.15.28:0/1161769906 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5636e9f8d4e0 0x5636e9f8f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:22.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.093+0000 7f118b400640 1 --2- 172.21.15.28:0/1161769906 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5636e9f8d4e0 0x5636e9f8f900 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=286349877b544465 server_cookie=99fe645cb0e70363 in_seq=0 out_seq=0 2024-04-23T14:52:22.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.093+0000 7f1178c00640 10 client.26561 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:22.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.093+0000 7f1178c00640 1 -- 172.21.15.28:0/1161769906 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5636e9e78700 con 0x5636e9f8d4e0 2024-04-23T14:52:22.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.093+0000 7f1178c00640 10 client.26561 handle_command_reply: tid=0 2024-04-23T14:52:22.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.094+0000 7f1194781740 1 client.26561 shutdown 2024-04-23T14:52:22.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.094+0000 7f1194781740 1 -- 172.21.15.28:0/1161769906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636e9f0b9b0 msgr2=0x5636e9f10350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:22.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.094+0000 7f1194781740 1 --2- 172.21.15.28:0/1161769906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636e9f0b9b0 0x5636e9f10350 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f1174009d00 tx=0x7f1174010640 comp rx=0 tx=0).stop 2024-04-23T14:52:22.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.094+0000 7f1194781740 1 -- 172.21.15.28:0/1161769906 shutdown_connections 2024-04-23T14:52:22.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.094+0000 7f1194781740 1 --2- 172.21.15.28:0/1161769906 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5636e9f8d4e0 0x5636e9f8f900 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.094+0000 7f1178c00640 0 client.26561 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:22.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.094+0000 7f1194781740 1 --2- 172.21.15.28:0/1161769906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636e9f0b9b0 0x5636e9f10350 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.094+0000 7f1194781740 1 --2- 172.21.15.28:0/1161769906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636e9f16080 0x5636e9f10890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.094+0000 7f1194781740 1 --2- 172.21.15.28:0/1161769906 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5636e9f0b090 0x5636e9f0fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.094+0000 7f1194781740 1 -- 172.21.15.28:0/1161769906 >> 172.21.15.28:0/1161769906 conn(0x5636e9f073b0 msgr2=0x5636e9f0c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:22.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.094+0000 7f1194781740 1 -- 172.21.15.28:0/1161769906 shutdown_connections 2024-04-23T14:52:22.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.094+0000 7f1194781740 1 -- 172.21.15.28:0/1161769906 wait complete. 2024-04-23T14:52:22.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.094+0000 7f1194781740 20 client.26561 trim_cache size 0 max 16384 2024-04-23T14:52:22.099 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:52:22.099 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-23T14:52:22.099 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "ff7d27ee-2bd6-422e-9785-47c658f9258f", 2024-04-23T14:52:22.099 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-23T14:52:22.099 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:52:22.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.095+0000 7f1191a00640 1 -- 172.21.15.28:0/1437732339 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f11700a3de0 msgr2=0x7f11700a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:22.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.095+0000 7f1191a00640 1 --2- 172.21.15.28:0/1437732339 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f11700a3de0 0x7f11700a62a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x5636e9f0c650 tx=0x7f117c000f40 comp rx=0 tx=0).stop 2024-04-23T14:52:22.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.095+0000 7f1191a00640 1 -- 172.21.15.28:0/1437732339 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f118c103280 msgr2=0x7f118c1a07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:22.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.095+0000 7f1191a00640 1 --2- 172.21.15.28:0/1437732339 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f118c103280 0x7f118c1a07d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f117c00eb00 tx=0x7f117c00c2d0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.096+0000 7f1191a00640 1 -- 172.21.15.28:0/1437732339 shutdown_connections 2024-04-23T14:52:22.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.096+0000 7f1191a00640 1 --2- 172.21.15.28:0/1437732339 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f11700a3de0 0x7f11700a62a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.096+0000 7f1191a00640 1 --2- 172.21.15.28:0/1437732339 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f118c103ba0 0x7f118c1a0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.096+0000 7f1191a00640 1 --2- 172.21.15.28:0/1437732339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f118c10c5c0 0x7f118c19a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.096+0000 7f1191a00640 1 --2- 172.21.15.28:0/1437732339 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f118c103280 0x7f118c1a07d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.096+0000 7f1191a00640 1 -- 172.21.15.28:0/1437732339 >> 172.21.15.28:0/1437732339 conn(0x7f118c078000 msgr2=0x7f118c108570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.096+0000 7f1191a00640 1 -- 172.21.15.28:0/1437732339 shutdown_connections 2024-04-23T14:52:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.096+0000 7f1191a00640 1 -- 172.21.15.28:0/1437732339 wait complete. 2024-04-23T14:52:22.118 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:52:22.118 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:52:22.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.339+0000 7f5bcb600640 1 -- 172.21.15.28:0/2678370867 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bc41044b0 msgr2=0x7f5bc4104890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:22.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.339+0000 7f5bcb600640 1 --2- 172.21.15.28:0/2678370867 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bc41044b0 0x7f5bc4104890 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5bb4009ca0 tx=0x7f5bb402a8c0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.339+0000 7f5bcb600640 1 -- 172.21.15.28:0/2678370867 shutdown_connections 2024-04-23T14:52:22.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.339+0000 7f5bcb600640 1 --2- 172.21.15.28:0/2678370867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bc4109580 0x7f5bc4111150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.339+0000 7f5bcb600640 1 --2- 172.21.15.28:0/2678370867 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5bc4104e60 0x7f5bc4108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.339+0000 7f5bcb600640 1 --2- 172.21.15.28:0/2678370867 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bc41044b0 0x7f5bc4104890 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.339+0000 7f5bcb600640 1 -- 172.21.15.28:0/2678370867 >> 172.21.15.28:0/2678370867 conn(0x7f5bc40ffd90 msgr2=0x7f5bc41021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:22.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.340+0000 7f5bcb600640 1 -- 172.21.15.28:0/2678370867 shutdown_connections 2024-04-23T14:52:22.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.340+0000 7f5bcb600640 1 -- 172.21.15.28:0/2678370867 wait complete. 2024-04-23T14:52:22.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.341+0000 7f5bcb600640 1 Processor -- start 2024-04-23T14:52:22.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.341+0000 7f5bcb600640 1 -- start start 2024-04-23T14:52:22.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.341+0000 7f5bcb600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bc41044b0 0x7f5bc41a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.341+0000 7f5bcb600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5bc4109580 0x7f5bc41a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.341+0000 7f5bcb600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bc419a9e0 0x7f5bc419ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.341+0000 7f5bcb600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc4114660 con 0x7f5bc419a9e0 2024-04-23T14:52:22.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.341+0000 7f5bcb600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc41144e0 con 0x7f5bc41044b0 2024-04-23T14:52:22.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.341+0000 7f5bcb600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc41147e0 con 0x7f5bc4109580 2024-04-23T14:52:22.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.341+0000 7f5bc8c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bc41044b0 0x7f5bc41a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:22.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.341+0000 7f5bc8c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bc41044b0 0x7f5bc41a07e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40532/0 (socket says 172.21.15.28:40532) 2024-04-23T14:52:22.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.341+0000 7f5bc8c00640 1 -- 172.21.15.28:0/2923309936 learned_addr learned my addr 172.21.15.28:0/2923309936 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:22.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.342+0000 7f5bc9600640 1 --2- 172.21.15.28:0/2923309936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bc419a9e0 0x7f5bc419ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:22.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.342+0000 7f5bc3e00640 1 --2- 172.21.15.28:0/2923309936 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5bc4109580 0x7f5bc41a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:22.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.342+0000 7f5bc8c00640 1 -- 172.21.15.28:0/2923309936 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5bc4109580 msgr2=0x7f5bc41a0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:22.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.342+0000 7f5bc8c00640 1 --2- 172.21.15.28:0/2923309936 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5bc4109580 0x7f5bc41a0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.342+0000 7f5bc8c00640 1 -- 172.21.15.28:0/2923309936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bc419a9e0 msgr2=0x7f5bc419ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:22.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.342+0000 7f5bc8c00640 1 --2- 172.21.15.28:0/2923309936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bc419a9e0 0x7f5bc419ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.342+0000 7f5bc8c00640 1 -- 172.21.15.28:0/2923309936 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bc419b500 con 0x7f5bc41044b0 2024-04-23T14:52:22.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.342+0000 7f5bc8c00640 1 --2- 172.21.15.28:0/2923309936 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bc41044b0 0x7f5bc41a07e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f5bb4005e30 tx=0x7f5bb4003f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.343+0000 7f5bc1600640 1 -- 172.21.15.28:0/2923309936 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb4031650 con 0x7f5bc41044b0 2024-04-23T14:52:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.343+0000 7f5bc1600640 1 -- 172.21.15.28:0/2923309936 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bb40317f0 con 0x7f5bc41044b0 2024-04-23T14:52:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.343+0000 7f5bcb600640 1 -- 172.21.15.28:0/2923309936 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bc419b790 con 0x7f5bc41044b0 2024-04-23T14:52:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.343+0000 7f5bc1600640 1 -- 172.21.15.28:0/2923309936 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb402fa30 con 0x7f5bc41044b0 2024-04-23T14:52:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.343+0000 7f5bcb600640 1 -- 172.21.15.28:0/2923309936 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bc41a79e0 con 0x7f5bc41044b0 2024-04-23T14:52:22.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.346+0000 7f5bce5a5740 1 Processor -- start 2024-04-23T14:52:22.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.346+0000 7f5bce5a5740 1 -- start start 2024-04-23T14:52:22.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.346+0000 7f5bc1600640 1 -- 172.21.15.28:0/2923309936 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f5bb402fc30 con 0x7f5bc41044b0 2024-04-23T14:52:22.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.346+0000 7f5bce5a5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562599e4f860 0x562599e4fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.346+0000 7f5bce5a5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562599e50180 0x562599e5a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.346+0000 7f5bce5a5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562599e5a850 0x562599e5cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.346+0000 7f5bce5a5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562599db51f0 con 0x562599e4f860 2024-04-23T14:52:22.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.346+0000 7f5bce5a5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562599d5b350 con 0x562599e5a850 2024-04-23T14:52:22.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.346+0000 7f5bce5a5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562599db11b0 con 0x562599e50180 2024-04-23T14:52:22.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.347+0000 7f5bc9600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562599e4f860 0x562599e4fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:22.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.347+0000 7f5bc3e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562599e5a850 0x562599e5cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:22.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.347+0000 7f5bc8c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562599e50180 0x562599e5a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:22.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.347+0000 7f5bc3e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562599e5a850 0x562599e5cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40534/0 (socket says 172.21.15.28:40534) 2024-04-23T14:52:22.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.347+0000 7f5bc9600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562599e4f860 0x562599e4fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53200/0 (socket says 172.21.15.28:53200) 2024-04-23T14:52:22.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.347+0000 7f5bc3e00640 1 -- 172.21.15.28:0/3081233351 learned_addr learned my addr 172.21.15.28:0/3081233351 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:22.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.347+0000 7f5bc3e00640 1 -- 172.21.15.28:0/3081233351 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562599e50180 msgr2=0x562599e5a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:22.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.347+0000 7f5bc3e00640 1 --2- 172.21.15.28:0/3081233351 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562599e50180 0x562599e5a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.347+0000 7f5bc3e00640 1 -- 172.21.15.28:0/3081233351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562599e4f860 msgr2=0x562599e4fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:22.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.347+0000 7f5bc3e00640 1 --2- 172.21.15.28:0/3081233351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562599e4f860 0x562599e4fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.347+0000 7f5bc3e00640 1 -- 172.21.15.28:0/3081233351 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562599daff50 con 0x562599e5a850 2024-04-23T14:52:22.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.348+0000 7f5bc3e00640 1 --2- 172.21.15.28:0/3081233351 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562599e5a850 0x562599e5cc40 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f5bac009c20 tx=0x7f5bac011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdcf94e697360faa server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:22.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.348+0000 7f5bb9c00640 1 -- 172.21.15.28:0/3081233351 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bac004230 con 0x562599e5a850 2024-04-23T14:52:22.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.348+0000 7f5bb9c00640 1 -- 172.21.15.28:0/3081233351 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bac0043d0 con 0x562599e5a850 2024-04-23T14:52:22.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.348+0000 7f5bb9c00640 1 -- 172.21.15.28:0/3081233351 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bac004e60 con 0x562599e5a850 2024-04-23T14:52:22.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.348+0000 7f5bc1600640 1 --2- 172.21.15.28:0/2923309936 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5ba80a3da0 0x7f5ba80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.349+0000 7f5bc1600640 1 -- 172.21.15.28:0/2923309936 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7f5bb40f1910 con 0x7f5bc41044b0 2024-04-23T14:52:22.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.349+0000 7f5bce5a5740 1 -- 172.21.15.28:0/3081233351 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562599e5a850 msgr2=0x562599e5cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:22.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.349+0000 7f5bce5a5740 1 --2- 172.21.15.28:0/3081233351 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562599e5a850 0x562599e5cc40 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f5bac009c20 tx=0x7f5bac011810 comp rx=0 tx=0).stop 2024-04-23T14:52:22.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.349+0000 7f5bc9600640 1 --2- 172.21.15.28:0/2923309936 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5ba80a3da0 0x7f5ba80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:22.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.349+0000 7f5bce5a5740 1 -- 172.21.15.28:0/3081233351 shutdown_connections 2024-04-23T14:52:22.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.349+0000 7f5bce5a5740 1 --2- 172.21.15.28:0/3081233351 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562599e5a850 0x562599e5cc40 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.349+0000 7f5bce5a5740 1 --2- 172.21.15.28:0/3081233351 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562599e50180 0x562599e5a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.349+0000 7f5bce5a5740 1 --2- 172.21.15.28:0/3081233351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562599e4f860 0x562599e4fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.349+0000 7f5bce5a5740 1 -- 172.21.15.28:0/3081233351 >> 172.21.15.28:0/3081233351 conn(0x562599e4bb80 msgr2=0x562599e4bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:22.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.349+0000 7f5bce5a5740 1 -- 172.21.15.28:0/3081233351 shutdown_connections 2024-04-23T14:52:22.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.349+0000 7f5bce5a5740 1 -- 172.21.15.28:0/3081233351 wait complete. 2024-04-23T14:52:22.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.350+0000 7f5bc9600640 1 --2- 172.21.15.28:0/2923309936 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5ba80a3da0 0x7f5ba80a6260 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc009d20 tx=0x7f5bbc005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:22.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.350+0000 7f5bce5a5740 1 -- start start 2024-04-23T14:52:22.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.350+0000 7f5bce5a5740 1 Processor -- start 2024-04-23T14:52:22.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.351+0000 7f5bce5a5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562599e4f860 0x562599e576f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.351+0000 7f5bce5a5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562599e50180 0x562599e57c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.351+0000 7f5bce5a5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562599e5a850 0x562599e58170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.351+0000 7f5bce5a5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562599e5d640 con 0x562599e4f860 2024-04-23T14:52:22.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.351+0000 7f5bce5a5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562599d6e690 con 0x562599e50180 2024-04-23T14:52:22.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.351+0000 7f5bce5a5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562599e5d7c0 con 0x562599e5a850 2024-04-23T14:52:22.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.351+0000 7f5bce5a5740 10 client.0 _ll_register_callbacks cb 0x562599daa650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:52:22.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.351+0000 7f5bc3e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562599e50180 0x562599e57c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:22.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.351+0000 7f5bc3e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562599e50180 0x562599e57c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40538/0 (socket says 172.21.15.28:40538) 2024-04-23T14:52:22.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.351+0000 7f5bc3e00640 1 -- 172.21.15.28:0/1826975478 learned_addr learned my addr 172.21.15.28:0/1826975478 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:22.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.351+0000 7f5bc8c00640 1 --2- 172.21.15.28:0/1826975478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562599e4f860 0x562599e576f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:22.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.351+0000 7f5bc9600640 1 --2- 172.21.15.28:0/1826975478 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562599e5a850 0x562599e58170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:22.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.351+0000 7f5bc3e00640 1 -- 172.21.15.28:0/1826975478 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562599e5a850 msgr2=0x562599e58170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:22.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.351+0000 7f5bc3e00640 1 --2- 172.21.15.28:0/1826975478 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562599e5a850 0x562599e58170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.351+0000 7f5bc3e00640 1 -- 172.21.15.28:0/1826975478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562599e4f860 msgr2=0x562599e576f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:22.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.351+0000 7f5bc3e00640 1 --2- 172.21.15.28:0/1826975478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562599e4f860 0x562599e576f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.351+0000 7f5bc3e00640 1 -- 172.21.15.28:0/1826975478 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562599d6d920 con 0x562599e50180 2024-04-23T14:52:22.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.351+0000 7f5bc8c00640 1 --2- 172.21.15.28:0/1826975478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562599e4f860 0x562599e576f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:52:22.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.352+0000 7f5bc3e00640 1 --2- 172.21.15.28:0/1826975478 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562599e50180 0x562599e57c30 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f5bac004ac0 tx=0x7f5bac019b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ab9ab76dc6b2b1e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:22.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.352+0000 7f5bb2000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:52:22.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.352+0000 7f5bb2000640 1 -- 172.21.15.28:0/1826975478 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bac028020 con 0x562599e50180 2024-04-23T14:52:22.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.352+0000 7f5bb2000640 1 -- 172.21.15.28:0/1826975478 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7f5bac024420 con 0x562599e50180 2024-04-23T14:52:22.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.352+0000 7f5bce5a5740 1 -- 172.21.15.28:0/1826975478 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bac0043d0 con 0x562599e50180 2024-04-23T14:52:22.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.352+0000 7f5bb2000640 1 -- 172.21.15.28:0/1826975478 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f5bac033050 con 0x562599e50180 2024-04-23T14:52:22.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.352+0000 7f5bce5a5740 10 client.26573 fetch_fsmap learned FSMap version 20 2024-04-23T14:52:22.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.352+0000 7f5bce5a5740 1 -- 172.21.15.28:0/1826975478 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x562599b917d0 con 0x562599e50180 2024-04-23T14:52:22.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.353+0000 7f5bb2000640 1 -- 172.21.15.28:0/1826975478 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bac0043d0 con 0x562599e50180 2024-04-23T14:52:22.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.353+0000 7f5bb2000640 1 -- 172.21.15.28:0/1826975478 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f5bac018070 con 0x562599e50180 2024-04-23T14:52:22.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.353+0000 7f5bce5a5740 10 client.26573 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:52:22.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.353+0000 7f5bce5a5740 10 client.26573 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:52:22.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.353+0000 7f5bce5a5740 20 client.26573 populate_metadata read hostname 'smithi028' 2024-04-23T14:52:22.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.353+0000 7f5bce5a5740 1 --2- 172.21.15.28:0/1826975478 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562599e6aa30 0x562599e6ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.353+0000 7f5bce5a5740 4 client.26573 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:52:22.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.353+0000 7f5bce5a5740 1 -- 172.21.15.28:0/1826975478 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562599dc2c10 con 0x562599e6aa30 2024-04-23T14:52:22.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.353+0000 7f5bc8c00640 1 --2- 172.21.15.28:0/1826975478 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562599e6aa30 0x562599e6ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:22.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.354+0000 7f5bc8c00640 1 --2- 172.21.15.28:0/1826975478 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562599e6aa30 0x562599e6ce20 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c999ba8c70e93165 server_cookie=996b30b732ec6475 in_seq=0 out_seq=0 2024-04-23T14:52:22.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.354+0000 7f5bb2000640 10 client.26573 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:22.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.354+0000 7f5bb2000640 1 -- 172.21.15.28:0/1826975478 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562599dc2c10 con 0x562599e6aa30 2024-04-23T14:52:22.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.354+0000 7f5bb2000640 10 client.26573 handle_command_reply: tid=0 2024-04-23T14:52:22.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.355+0000 7f5bce5a5740 1 client.26573 shutdown 2024-04-23T14:52:22.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.355+0000 7f5bce5a5740 1 -- 172.21.15.28:0/1826975478 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562599e50180 msgr2=0x562599e57c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:22.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.355+0000 7f5bce5a5740 1 --2- 172.21.15.28:0/1826975478 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562599e50180 0x562599e57c30 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f5bac004ac0 tx=0x7f5bac019b00 comp rx=0 tx=0).stop 2024-04-23T14:52:22.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.355+0000 7f5bce5a5740 1 -- 172.21.15.28:0/1826975478 shutdown_connections 2024-04-23T14:52:22.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.355+0000 7f5bce5a5740 1 --2- 172.21.15.28:0/1826975478 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562599e6aa30 0x562599e6ce20 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.355+0000 7f5bce5a5740 1 --2- 172.21.15.28:0/1826975478 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562599e50180 0x562599e57c30 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.355+0000 7f5bce5a5740 1 --2- 172.21.15.28:0/1826975478 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562599e5a850 0x562599e58170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.355+0000 7f5bce5a5740 1 --2- 172.21.15.28:0/1826975478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562599e4f860 0x562599e576f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.355+0000 7f5bb2000640 0 client.26573 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:22.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.355+0000 7f5bce5a5740 1 -- 172.21.15.28:0/1826975478 >> 172.21.15.28:0/1826975478 conn(0x562599e4bb80 msgr2=0x562599e51980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:22.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.355+0000 7f5bce5a5740 1 -- 172.21.15.28:0/1826975478 shutdown_connections 2024-04-23T14:52:22.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.355+0000 7f5bce5a5740 1 -- 172.21.15.28:0/1826975478 wait complete. 2024-04-23T14:52:22.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.355+0000 7f5bce5a5740 20 client.26573 trim_cache size 0 max 16384 2024-04-23T14:52:22.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.377+0000 7f5bce5a5740 1 Processor -- start 2024-04-23T14:52:22.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.377+0000 7f5bce5a5740 1 -- start start 2024-04-23T14:52:22.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.377+0000 7f5bce5a5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562599e4f860 0x562599e66500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.377+0000 7f5bce5a5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562599e50180 0x562599e56850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.377+0000 7f5bce5a5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562599e5a850 0x562599e57dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.377+0000 7f5bce5a5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc41adfd0 con 0x562599e4f860 2024-04-23T14:52:22.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.377+0000 7f5bce5a5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc41b0830 con 0x562599e50180 2024-04-23T14:52:22.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.377+0000 7f5bce5a5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc41ae620 con 0x562599e5a850 2024-04-23T14:52:22.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.377+0000 7f5bc3e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562599e50180 0x562599e56850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:22.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.377+0000 7f5bc3e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562599e50180 0x562599e56850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40550/0 (socket says 172.21.15.28:40550) 2024-04-23T14:52:22.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.377+0000 7f5bc3e00640 1 -- 172.21.15.28:0/4120136858 learned_addr learned my addr 172.21.15.28:0/4120136858 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:22.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.378+0000 7f5bc8c00640 1 --2- 172.21.15.28:0/4120136858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562599e4f860 0x562599e66500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:22.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.378+0000 7f5bc9600640 1 --2- 172.21.15.28:0/4120136858 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562599e5a850 0x562599e57dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:22.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.378+0000 7f5bc3e00640 1 -- 172.21.15.28:0/4120136858 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562599e5a850 msgr2=0x562599e57dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:22.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.378+0000 7f5bc3e00640 1 --2- 172.21.15.28:0/4120136858 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562599e5a850 0x562599e57dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.378+0000 7f5bc3e00640 1 -- 172.21.15.28:0/4120136858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562599e4f860 msgr2=0x562599e66500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:22.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.378+0000 7f5bc3e00640 1 --2- 172.21.15.28:0/4120136858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562599e4f860 0x562599e66500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.378+0000 7f5bc3e00640 1 -- 172.21.15.28:0/4120136858 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562599ac73f0 con 0x562599e50180 2024-04-23T14:52:22.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.378+0000 7f5bc3e00640 1 --2- 172.21.15.28:0/4120136858 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562599e50180 0x562599e56850 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f5bac000c00 tx=0x7f5bac001e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28629fc3a443e5bc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:22.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.379+0000 7f5bb9200640 1 -- 172.21.15.28:0/4120136858 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bac03c070 con 0x562599e50180 2024-04-23T14:52:22.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.379+0000 7f5bb9200640 1 -- 172.21.15.28:0/4120136858 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bac01f040 con 0x562599e50180 2024-04-23T14:52:22.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.379+0000 7f5bb9200640 1 -- 172.21.15.28:0/4120136858 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bac028050 con 0x562599e50180 2024-04-23T14:52:22.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.379+0000 7f5bce5a5740 1 -- 172.21.15.28:0/4120136858 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562599e50180 msgr2=0x562599e56850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:22.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.379+0000 7f5bce5a5740 1 --2- 172.21.15.28:0/4120136858 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562599e50180 0x562599e56850 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f5bac000c00 tx=0x7f5bac001e50 comp rx=0 tx=0).stop 2024-04-23T14:52:22.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.379+0000 7f5bce5a5740 1 -- 172.21.15.28:0/4120136858 shutdown_connections 2024-04-23T14:52:22.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.379+0000 7f5bce5a5740 1 --2- 172.21.15.28:0/4120136858 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562599e50180 0x562599e56850 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.379+0000 7f5bce5a5740 1 --2- 172.21.15.28:0/4120136858 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562599e5a850 0x562599e57dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.379+0000 7f5bce5a5740 1 --2- 172.21.15.28:0/4120136858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562599e4f860 0x562599e66500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.379+0000 7f5bce5a5740 1 -- 172.21.15.28:0/4120136858 >> 172.21.15.28:0/4120136858 conn(0x562599e4bb80 msgr2=0x562599e576b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:22.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.380+0000 7f5bce5a5740 1 -- 172.21.15.28:0/4120136858 shutdown_connections 2024-04-23T14:52:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.380+0000 7f5bce5a5740 1 -- 172.21.15.28:0/4120136858 wait complete. 2024-04-23T14:52:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.380+0000 7f5bce5a5740 1 -- start start 2024-04-23T14:52:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.380+0000 7f5bce5a5740 1 Processor -- start 2024-04-23T14:52:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.381+0000 7f5bce5a5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562599e4f860 0x562599e62a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.381+0000 7f5bce5a5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562599e50180 0x562599e62f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.381+0000 7f5bce5a5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562599e5a850 0x562599e53b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.381+0000 7f5bce5a5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc41aecf0 con 0x562599e5a850 2024-04-23T14:52:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.381+0000 7f5bce5a5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc41b0160 con 0x562599e4f860 2024-04-23T14:52:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.381+0000 7f5bce5a5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc41af3c0 con 0x562599e50180 2024-04-23T14:52:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.381+0000 7f5bce5a5740 10 client.0 _ll_register_callbacks cb 0x7f5bbc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:52:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.381+0000 7f5bc8c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562599e4f860 0x562599e62a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.381+0000 7f5bc8c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562599e4f860 0x562599e62a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40558/0 (socket says 172.21.15.28:40558) 2024-04-23T14:52:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.381+0000 7f5bc8c00640 1 -- 172.21.15.28:0/20310909 learned_addr learned my addr 172.21.15.28:0/20310909 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.381+0000 7f5bc9600640 1 --2- 172.21.15.28:0/20310909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562599e5a850 0x562599e53b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.381+0000 7f5bc3e00640 1 --2- 172.21.15.28:0/20310909 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562599e50180 0x562599e62f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:22.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.381+0000 7f5bc8c00640 1 -- 172.21.15.28:0/20310909 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562599e50180 msgr2=0x562599e62f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:22.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.381+0000 7f5bc8c00640 1 --2- 172.21.15.28:0/20310909 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562599e50180 0x562599e62f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.381+0000 7f5bc8c00640 1 -- 172.21.15.28:0/20310909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562599e5a850 msgr2=0x562599e53b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:22.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.381+0000 7f5bc8c00640 1 --2- 172.21.15.28:0/20310909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562599e5a850 0x562599e53b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.381+0000 7f5bc8c00640 1 -- 172.21.15.28:0/20310909 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562599a870f0 con 0x562599e4f860 2024-04-23T14:52:22.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.382+0000 7f5bc8c00640 1 --2- 172.21.15.28:0/20310909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562599e4f860 0x562599e62a10 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f5bb40ef1c0 tx=0x7f5bb40f7550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e95e9a8ade718eb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:22.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.382+0000 7f5bb2000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:52:22.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.382+0000 7f5bb2000640 1 -- 172.21.15.28:0/20310909 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb40bd310 con 0x562599e4f860 2024-04-23T14:52:22.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.382+0000 7f5bb2000640 1 -- 172.21.15.28:0/20310909 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7f5bb40ba020 con 0x562599e4f860 2024-04-23T14:52:22.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.382+0000 7f5bce5a5740 1 -- 172.21.15.28:0/20310909 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bac01f040 con 0x562599e4f860 2024-04-23T14:52:22.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.382+0000 7f5bb2000640 1 -- 172.21.15.28:0/20310909 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f5bb40f73a0 con 0x562599e4f860 2024-04-23T14:52:22.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.382+0000 7f5bce5a5740 10 client.26579 fetch_fsmap learned FSMap version 20 2024-04-23T14:52:22.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.382+0000 7f5bce5a5740 1 -- 172.21.15.28:0/20310909 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x562599d7a810 con 0x562599e4f860 2024-04-23T14:52:22.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.382+0000 7f5bb2000640 1 -- 172.21.15.28:0/20310909 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bb40d9030 con 0x562599e4f860 2024-04-23T14:52:22.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.383+0000 7f5bb2000640 1 -- 172.21.15.28:0/20310909 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f5bb40bece0 con 0x562599e4f860 2024-04-23T14:52:22.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.383+0000 7f5bce5a5740 10 client.26579 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:52:22.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.383+0000 7f5bce5a5740 10 client.26579 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:52:22.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.383+0000 7f5bce5a5740 20 client.26579 populate_metadata read hostname 'smithi028' 2024-04-23T14:52:22.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.383+0000 7f5bce5a5740 1 --2- 172.21.15.28:0/20310909 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562599ed0d40 0x562599ed3160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:22.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.383+0000 7f5bce5a5740 4 client.26579 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:52:22.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.383+0000 7f5bce5a5740 1 -- 172.21.15.28:0/20310909 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562599dbca80 con 0x562599ed0d40 2024-04-23T14:52:22.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.383+0000 7f5bc3e00640 1 --2- 172.21.15.28:0/20310909 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562599ed0d40 0x562599ed3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:22.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.383+0000 7f5bc3e00640 1 --2- 172.21.15.28:0/20310909 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562599ed0d40 0x562599ed3160 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12a974679b8cf136 server_cookie=19cbf4d4d33e20af in_seq=0 out_seq=0 2024-04-23T14:52:22.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.383+0000 7f5bb2000640 10 client.26579 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:22.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.384+0000 7f5bb2000640 1 -- 172.21.15.28:0/20310909 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x562599dbca80 con 0x562599ed0d40 2024-04-23T14:52:22.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.384+0000 7f5bb2000640 10 client.26579 handle_command_reply: tid=0 2024-04-23T14:52:22.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.384+0000 7f5bce5a5740 1 client.26579 shutdown 2024-04-23T14:52:22.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.384+0000 7f5bce5a5740 1 -- 172.21.15.28:0/20310909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562599e4f860 msgr2=0x562599e62a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:22.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.384+0000 7f5bce5a5740 1 --2- 172.21.15.28:0/20310909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562599e4f860 0x562599e62a10 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f5bb40ef1c0 tx=0x7f5bb40f7550 comp rx=0 tx=0).stop 2024-04-23T14:52:22.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.384+0000 7f5bce5a5740 1 -- 172.21.15.28:0/20310909 shutdown_connections 2024-04-23T14:52:22.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.384+0000 7f5bce5a5740 1 --2- 172.21.15.28:0/20310909 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562599ed0d40 0x562599ed3160 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.384+0000 7f5bce5a5740 1 --2- 172.21.15.28:0/20310909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562599e5a850 0x562599e53b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.384+0000 7f5bce5a5740 1 --2- 172.21.15.28:0/20310909 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562599e50180 0x562599e62f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.384+0000 7f5bce5a5740 1 --2- 172.21.15.28:0/20310909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562599e4f860 0x562599e62a10 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.384+0000 7f5bb2000640 0 client.26579 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:22.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.384+0000 7f5bce5a5740 1 -- 172.21.15.28:0/20310909 >> 172.21.15.28:0/20310909 conn(0x562599e4bb80 msgr2=0x562599e4a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:22.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.384+0000 7f5bce5a5740 1 -- 172.21.15.28:0/20310909 shutdown_connections 2024-04-23T14:52:22.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.384+0000 7f5bce5a5740 1 -- 172.21.15.28:0/20310909 wait complete. 2024-04-23T14:52:22.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.384+0000 7f5bce5a5740 20 client.26579 trim_cache size 0 max 16384 2024-04-23T14:52:22.389 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:52:22.389 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-23T14:52:22.389 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-23T14:52:22.389 INFO:teuthology.orchestra.run.smithi028.stdout: "ff7d27ee-2bd6-422e-9785-47c658f9258f": { 2024-04-23T14:52:22.389 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-23T14:52:22.390 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "ff7d27ee-2bd6-422e-9785-47c658f9258f", 2024-04-23T14:52:22.390 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-23T14:52:22.390 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:52:22.390 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:52:22.390 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:52:22.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.385+0000 7f5bcb600640 1 -- 172.21.15.28:0/2923309936 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5ba80a3da0 msgr2=0x7f5ba80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:22.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.385+0000 7f5bcb600640 1 --2- 172.21.15.28:0/2923309936 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5ba80a3da0 0x7f5ba80a6260 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc009d20 tx=0x7f5bbc005f90 comp rx=0 tx=0).stop 2024-04-23T14:52:22.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.385+0000 7f5bcb600640 1 -- 172.21.15.28:0/2923309936 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bc41044b0 msgr2=0x7f5bc41a07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:22.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.385+0000 7f5bcb600640 1 --2- 172.21.15.28:0/2923309936 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bc41044b0 0x7f5bc41a07e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f5bb4005e30 tx=0x7f5bb4003f90 comp rx=0 tx=0).stop 2024-04-23T14:52:22.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.385+0000 7f5bcb600640 1 -- 172.21.15.28:0/2923309936 shutdown_connections 2024-04-23T14:52:22.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.385+0000 7f5bcb600640 1 --2- 172.21.15.28:0/2923309936 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5ba80a3da0 0x7f5ba80a6260 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.385+0000 7f5bcb600640 1 --2- 172.21.15.28:0/2923309936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bc419a9e0 0x7f5bc419ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.385+0000 7f5bcb600640 1 --2- 172.21.15.28:0/2923309936 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5bc4109580 0x7f5bc41a0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.385+0000 7f5bcb600640 1 --2- 172.21.15.28:0/2923309936 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bc41044b0 0x7f5bc41a07e0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:22.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.385+0000 7f5bcb600640 1 -- 172.21.15.28:0/2923309936 >> 172.21.15.28:0/2923309936 conn(0x7f5bc40ffd90 msgr2=0x7f5bc4102140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:22.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.385+0000 7f5bcb600640 1 -- 172.21.15.28:0/2923309936 shutdown_connections 2024-04-23T14:52:22.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:22.385+0000 7f5bcb600640 1 -- 172.21.15.28:0/2923309936 wait complete. 2024-04-23T14:52:22.406 INFO:tasks.cephfs.filesystem:scrub status for tag:ff7d27ee-2bd6-422e-9785-47c658f9258f - {'path': '/', 'tag': 'ff7d27ee-2bd6-422e-9785-47c658f9258f', 'options': 'recursive,force'} 2024-04-23T14:52:22.511 INFO:tasks.workunit.client.0.smithi028.stdout: 1 694909 8.98 MB/sec execute 278 sec latency 22.628 ms 2024-04-23T14:52:23.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:22 smithi105 ceph-mon[32555]: pgmap v257: 97 pgs: 97 active+clean; 436 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 154 op/s 2024-04-23T14:52:23.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:22 smithi038 ceph-mon[32348]: pgmap v257: 97 pgs: 97 active+clean; 436 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 154 op/s 2024-04-23T14:52:23.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:22 smithi028 ceph-mon[27810]: pgmap v257: 97 pgs: 97 active+clean; 436 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 154 op/s 2024-04-23T14:52:23.511 INFO:tasks.workunit.client.0.smithi028.stdout: 1 696443 8.99 MB/sec execute 279 sec latency 12.580 ms 2024-04-23T14:52:24.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:23 smithi105 ceph-mon[32555]: scrub queued for path: / 2024-04-23T14:52:24.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:23 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:52:24.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:23 smithi105 ceph-mon[32555]: scrub summary: active paths [/] 2024-04-23T14:52:24.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:23 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:52:24.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:23 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:52:24.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:23 smithi038 ceph-mon[32348]: scrub queued for path: / 2024-04-23T14:52:24.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:23 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:52:24.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:23 smithi038 ceph-mon[32348]: scrub summary: active paths [/] 2024-04-23T14:52:24.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:23 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:52:24.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:23 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:52:24.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:23 smithi028 ceph-mon[27810]: scrub queued for path: / 2024-04-23T14:52:24.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:23 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:52:24.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:23 smithi028 ceph-mon[27810]: scrub summary: active paths [/] 2024-04-23T14:52:24.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:23 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:52:24.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:23 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:52:24.511 INFO:tasks.workunit.client.0.smithi028.stdout: 1 697699 8.98 MB/sec execute 280 sec latency 21.878 ms 2024-04-23T14:52:25.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:24 smithi105 ceph-mon[32555]: pgmap v258: 97 pgs: 97 active+clean; 444 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.8 MiB/s wr, 210 op/s 2024-04-23T14:52:25.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:24 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:52:25.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:24 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:52:25.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:24 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:52:25.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:24 smithi038 ceph-mon[32348]: pgmap v258: 97 pgs: 97 active+clean; 444 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.8 MiB/s wr, 210 op/s 2024-04-23T14:52:25.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:24 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:52:25.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:24 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:52:25.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:24 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:52:25.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:24 smithi028 ceph-mon[27810]: pgmap v258: 97 pgs: 97 active+clean; 444 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.8 MiB/s wr, 210 op/s 2024-04-23T14:52:25.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:24 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:52:25.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:24 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:52:25.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:24 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:52:25.511 INFO:tasks.workunit.client.0.smithi028.stdout: 1 699337 8.98 MB/sec execute 281 sec latency 105.617 ms 2024-04-23T14:52:26.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:25 smithi105 ceph-mon[32555]: scrub summary: idle 2024-04-23T14:52:26.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:25 smithi038 ceph-mon[32348]: scrub summary: idle 2024-04-23T14:52:26.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:25 smithi028 ceph-mon[27810]: scrub summary: idle 2024-04-23T14:52:26.512 INFO:tasks.workunit.client.0.smithi028.stdout: 1 700737 8.97 MB/sec execute 282 sec latency 15.119 ms 2024-04-23T14:52:27.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:26 smithi105 ceph-mon[32555]: pgmap v259: 97 pgs: 97 active+clean; 444 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 144 op/s 2024-04-23T14:52:27.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:26 smithi038 ceph-mon[32348]: pgmap v259: 97 pgs: 97 active+clean; 444 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 144 op/s 2024-04-23T14:52:27.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:26 smithi028 ceph-mon[27810]: pgmap v259: 97 pgs: 97 active+clean; 444 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 144 op/s 2024-04-23T14:52:27.511 INFO:tasks.workunit.client.0.smithi028.stdout: 1 702789 8.98 MB/sec execute 283 sec latency 17.406 ms 2024-04-23T14:52:28.512 INFO:tasks.workunit.client.0.smithi028.stdout: 1 704068 8.97 MB/sec execute 284 sec latency 19.989 ms 2024-04-23T14:52:29.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:28 smithi105 ceph-mon[32555]: pgmap v260: 97 pgs: 97 active+clean; 451 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.9 MiB/s wr, 211 op/s 2024-04-23T14:52:29.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:28 smithi038 ceph-mon[32348]: pgmap v260: 97 pgs: 97 active+clean; 451 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.9 MiB/s wr, 211 op/s 2024-04-23T14:52:29.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:28 smithi028 ceph-mon[27810]: pgmap v260: 97 pgs: 97 active+clean; 451 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.9 MiB/s wr, 211 op/s 2024-04-23T14:52:29.512 INFO:tasks.workunit.client.0.smithi028.stdout: 1 706031 8.98 MB/sec execute 285 sec latency 18.072 ms 2024-04-23T14:52:30.512 INFO:tasks.workunit.client.0.smithi028.stdout: 1 707293 8.97 MB/sec execute 286 sec latency 44.209 ms 2024-04-23T14:52:31.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:30 smithi105 ceph-mon[32555]: pgmap v261: 97 pgs: 97 active+clean; 451 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 146 op/s 2024-04-23T14:52:31.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:30 smithi038 ceph-mon[32348]: pgmap v261: 97 pgs: 97 active+clean; 451 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 146 op/s 2024-04-23T14:52:31.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:30 smithi028 ceph-mon[27810]: pgmap v261: 97 pgs: 97 active+clean; 451 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 146 op/s 2024-04-23T14:52:31.512 INFO:tasks.workunit.client.0.smithi028.stdout: 1 709114 8.96 MB/sec execute 287 sec latency 16.147 ms 2024-04-23T14:52:32.512 INFO:tasks.workunit.client.0.smithi028.stdout: 1 710912 8.98 MB/sec execute 288 sec latency 14.135 ms 2024-04-23T14:52:33.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:32 smithi105 ceph-mon[32555]: pgmap v262: 97 pgs: 97 active+clean; 451 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 146 op/s 2024-04-23T14:52:33.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:32 smithi038 ceph-mon[32348]: pgmap v262: 97 pgs: 97 active+clean; 451 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 146 op/s 2024-04-23T14:52:33.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:32 smithi028 ceph-mon[27810]: pgmap v262: 97 pgs: 97 active+clean; 451 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 146 op/s 2024-04-23T14:52:33.512 INFO:tasks.workunit.client.0.smithi028.stdout: 1 712625 8.97 MB/sec execute 289 sec latency 39.693 ms 2024-04-23T14:52:34.512 INFO:tasks.workunit.client.0.smithi028.stdout: 1 714198 8.98 MB/sec execute 290 sec latency 10.348 ms 2024-04-23T14:52:35.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:34 smithi105 ceph-mon[32555]: pgmap v263: 97 pgs: 97 active+clean; 460 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.1 MiB/s wr, 217 op/s 2024-04-23T14:52:35.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:34 smithi038 ceph-mon[32348]: pgmap v263: 97 pgs: 97 active+clean; 460 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.1 MiB/s wr, 217 op/s 2024-04-23T14:52:35.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:34 smithi028 ceph-mon[27810]: pgmap v263: 97 pgs: 97 active+clean; 460 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.1 MiB/s wr, 217 op/s 2024-04-23T14:52:35.513 INFO:tasks.workunit.client.0.smithi028.stdout: 1 715460 8.97 MB/sec execute 291 sec latency 63.975 ms 2024-04-23T14:52:36.513 INFO:tasks.workunit.client.0.smithi028.stdout: 1 717598 8.98 MB/sec execute 292 sec latency 25.932 ms 2024-04-23T14:52:37.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:36 smithi105 ceph-mon[32555]: pgmap v264: 97 pgs: 97 active+clean; 460 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 137 op/s 2024-04-23T14:52:37.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:36 smithi038 ceph-mon[32348]: pgmap v264: 97 pgs: 97 active+clean; 460 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 137 op/s 2024-04-23T14:52:37.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:36 smithi028 ceph-mon[27810]: pgmap v264: 97 pgs: 97 active+clean; 460 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 137 op/s 2024-04-23T14:52:37.513 INFO:tasks.workunit.client.0.smithi028.stdout: 1 719005 8.97 MB/sec execute 293 sec latency 16.300 ms 2024-04-23T14:52:38.513 INFO:tasks.workunit.client.0.smithi028.stdout: 1 721167 8.99 MB/sec execute 294 sec latency 15.030 ms 2024-04-23T14:52:39.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:38 smithi105 ceph-mon[32555]: pgmap v265: 97 pgs: 97 active+clean; 468 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.1 MiB/s wr, 205 op/s 2024-04-23T14:52:39.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:38 smithi038 ceph-mon[32348]: pgmap v265: 97 pgs: 97 active+clean; 468 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.1 MiB/s wr, 205 op/s 2024-04-23T14:52:39.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:38 smithi028 ceph-mon[27810]: pgmap v265: 97 pgs: 97 active+clean; 468 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.1 MiB/s wr, 205 op/s 2024-04-23T14:52:39.513 INFO:tasks.workunit.client.0.smithi028.stdout: 1 722266 8.97 MB/sec execute 295 sec latency 71.635 ms 2024-04-23T14:52:40.513 INFO:tasks.workunit.client.0.smithi028.stdout: 1 723311 8.95 MB/sec execute 296 sec latency 144.078 ms 2024-04-23T14:52:41.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:40 smithi105 ceph-mon[32555]: pgmap v266: 97 pgs: 97 active+clean; 468 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 138 op/s 2024-04-23T14:52:41.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:40 smithi038 ceph-mon[32348]: pgmap v266: 97 pgs: 97 active+clean; 468 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 138 op/s 2024-04-23T14:52:41.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:40 smithi028 ceph-mon[27810]: pgmap v266: 97 pgs: 97 active+clean; 468 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 138 op/s 2024-04-23T14:52:41.513 INFO:tasks.workunit.client.0.smithi028.stdout: 1 724807 8.96 MB/sec execute 297 sec latency 12.405 ms 2024-04-23T14:52:42.513 INFO:tasks.workunit.client.0.smithi028.stdout: 1 726383 8.95 MB/sec execute 298 sec latency 17.042 ms 2024-04-23T14:52:43.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:42 smithi105 ceph-mon[32555]: pgmap v267: 97 pgs: 97 active+clean; 468 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 138 op/s 2024-04-23T14:52:43.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:42 smithi038 ceph-mon[32348]: pgmap v267: 97 pgs: 97 active+clean; 468 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 138 op/s 2024-04-23T14:52:43.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:42 smithi028 ceph-mon[27810]: pgmap v267: 97 pgs: 97 active+clean; 468 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 138 op/s 2024-04-23T14:52:43.514 INFO:tasks.workunit.client.0.smithi028.stdout: 1 728381 8.96 MB/sec execute 299 sec latency 17.937 ms 2024-04-23T14:52:44.514 INFO:tasks.workunit.client.0.smithi028.stdout: 1 729926 8.95 MB/sec execute 300 sec latency 23.727 ms 2024-04-23T14:52:45.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:44 smithi105 ceph-mon[32555]: pgmap v268: 97 pgs: 97 active+clean; 439 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.0 MiB/s wr, 202 op/s 2024-04-23T14:52:45.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:44 smithi038 ceph-mon[32348]: pgmap v268: 97 pgs: 97 active+clean; 439 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.0 MiB/s wr, 202 op/s 2024-04-23T14:52:45.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:44 smithi028 ceph-mon[27810]: pgmap v268: 97 pgs: 97 active+clean; 439 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.0 MiB/s wr, 202 op/s 2024-04-23T14:52:45.514 INFO:tasks.workunit.client.0.smithi028.stdout: 1 731703 8.97 MB/sec execute 301 sec latency 76.566 ms 2024-04-23T14:52:46.514 INFO:tasks.workunit.client.0.smithi028.stdout: 1 732898 8.96 MB/sec execute 302 sec latency 21.131 ms 2024-04-23T14:52:47.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:46 smithi105 ceph-mon[32555]: pgmap v269: 97 pgs: 97 active+clean; 439 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 132 op/s 2024-04-23T14:52:47.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:46 smithi038 ceph-mon[32348]: pgmap v269: 97 pgs: 97 active+clean; 439 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 132 op/s 2024-04-23T14:52:47.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:46 smithi028 ceph-mon[27810]: pgmap v269: 97 pgs: 97 active+clean; 439 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 132 op/s 2024-04-23T14:52:47.514 INFO:tasks.workunit.client.0.smithi028.stdout: 1 734669 8.95 MB/sec execute 303 sec latency 19.867 ms 2024-04-23T14:52:48.514 INFO:tasks.workunit.client.0.smithi028.stdout: 1 736106 8.95 MB/sec execute 304 sec latency 21.477 ms 2024-04-23T14:52:49.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:48 smithi105 ceph-mon[32555]: pgmap v270: 97 pgs: 97 active+clean; 446 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 196 op/s 2024-04-23T14:52:49.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:48 smithi038 ceph-mon[32348]: pgmap v270: 97 pgs: 97 active+clean; 446 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 196 op/s 2024-04-23T14:52:49.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:48 smithi028 ceph-mon[27810]: pgmap v270: 97 pgs: 97 active+clean; 446 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 196 op/s 2024-04-23T14:52:49.514 INFO:tasks.workunit.client.0.smithi028.stdout: 1 738244 8.96 MB/sec execute 305 sec latency 16.553 ms 2024-04-23T14:52:50.515 INFO:tasks.workunit.client.0.smithi028.stdout: 1 739487 8.95 MB/sec execute 306 sec latency 32.918 ms 2024-04-23T14:52:51.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:50 smithi105 ceph-mon[32555]: pgmap v271: 97 pgs: 97 active+clean; 446 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 129 op/s 2024-04-23T14:52:51.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:50 smithi038 ceph-mon[32348]: pgmap v271: 97 pgs: 97 active+clean; 446 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 129 op/s 2024-04-23T14:52:51.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:50 smithi028 ceph-mon[27810]: pgmap v271: 97 pgs: 97 active+clean; 446 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 129 op/s 2024-04-23T14:52:51.515 INFO:tasks.workunit.client.0.smithi028.stdout: 1 741200 8.94 MB/sec execute 307 sec latency 24.752 ms 2024-04-23T14:52:52.407 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:52:52.408 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:52:52.515 INFO:tasks.workunit.client.0.smithi028.stdout: 1 742897 8.95 MB/sec execute 308 sec latency 18.281 ms 2024-04-23T14:52:52.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.643+0000 7f27db200640 1 -- 172.21.15.28:0/789354014 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27dc0743b0 msgr2=0x7f27dc074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:52.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.643+0000 7f27db200640 1 --2- 172.21.15.28:0/789354014 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27dc0743b0 0x7f27dc074790 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f27cc009860 tx=0x7f27cc02a3d0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.643+0000 7f27db200640 1 -- 172.21.15.28:0/789354014 shutdown_connections 2024-04-23T14:52:52.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.643+0000 7f27db200640 1 --2- 172.21.15.28:0/789354014 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f27dc10e750 0x7f27dc110b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.643+0000 7f27db200640 1 --2- 172.21.15.28:0/789354014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27dc074cd0 0x7f27dc10e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.643+0000 7f27db200640 1 --2- 172.21.15.28:0/789354014 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27dc0743b0 0x7f27dc074790 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.643+0000 7f27db200640 1 -- 172.21.15.28:0/789354014 >> 172.21.15.28:0/789354014 conn(0x7f27dc0fd9a0 msgr2=0x7f27dc0ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:52.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.643+0000 7f27db200640 1 -- 172.21.15.28:0/789354014 shutdown_connections 2024-04-23T14:52:52.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.643+0000 7f27db200640 1 -- 172.21.15.28:0/789354014 wait complete. 2024-04-23T14:52:52.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.644+0000 7f27db200640 1 Processor -- start 2024-04-23T14:52:52.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.644+0000 7f27db200640 1 -- start start 2024-04-23T14:52:52.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.644+0000 7f27db200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27dc0743b0 0x7f27dc1a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:52.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.645+0000 7f27db200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f27dc074cd0 0x7f27dc1a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:52.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.645+0000 7f27db200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27dc10e750 0x7f27dc19a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:52.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.645+0000 7f27db200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f27dc1144a0 con 0x7f27dc0743b0 2024-04-23T14:52:52.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.645+0000 7f27db200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f27dc114320 con 0x7f27dc10e750 2024-04-23T14:52:52.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.645+0000 7f27db200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f27dc114620 con 0x7f27dc074cd0 2024-04-23T14:52:52.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.645+0000 7f27d9400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f27dc074cd0 0x7f27dc1a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:52.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.645+0000 7f27d9400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f27dc074cd0 0x7f27dc1a0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:45882/0 (socket says 172.21.15.28:45882) 2024-04-23T14:52:52.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.645+0000 7f27d9400640 1 -- 172.21.15.28:0/3212502962 learned_addr learned my addr 172.21.15.28:0/3212502962 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:52.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.645+0000 7f27d9e00640 1 --2- 172.21.15.28:0/3212502962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27dc0743b0 0x7f27dc1a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:52.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.645+0000 7f27da800640 1 --2- 172.21.15.28:0/3212502962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27dc10e750 0x7f27dc19a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:52.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.645+0000 7f27d9400640 1 -- 172.21.15.28:0/3212502962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27dc10e750 msgr2=0x7f27dc19a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:52.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.645+0000 7f27d9400640 1 --2- 172.21.15.28:0/3212502962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27dc10e750 0x7f27dc19a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.645+0000 7f27d9400640 1 -- 172.21.15.28:0/3212502962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27dc0743b0 msgr2=0x7f27dc1a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:52.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.645+0000 7f27d9400640 1 --2- 172.21.15.28:0/3212502962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27dc0743b0 0x7f27dc1a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.645+0000 7f27d9400640 1 -- 172.21.15.28:0/3212502962 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27dc19b010 con 0x7f27dc074cd0 2024-04-23T14:52:52.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.646+0000 7f27d9400640 1 --2- 172.21.15.28:0/3212502962 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f27dc074cd0 0x7f27dc1a0bc0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f27c800e9c0 tx=0x7f27c800edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:52.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.646+0000 7f27d2a00640 1 -- 172.21.15.28:0/3212502962 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c800cd80 con 0x7f27dc074cd0 2024-04-23T14:52:52.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.646+0000 7f27d2a00640 1 -- 172.21.15.28:0/3212502962 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27c80035f0 con 0x7f27dc074cd0 2024-04-23T14:52:52.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.646+0000 7f27db200640 1 -- 172.21.15.28:0/3212502962 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27dc19b300 con 0x7f27dc074cd0 2024-04-23T14:52:52.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.646+0000 7f27d2a00640 1 -- 172.21.15.28:0/3212502962 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c8010670 con 0x7f27dc074cd0 2024-04-23T14:52:52.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.646+0000 7f27db200640 1 -- 172.21.15.28:0/3212502962 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27dc1a74d0 con 0x7f27dc074cd0 2024-04-23T14:52:52.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.650+0000 7f27d2a00640 1 -- 172.21.15.28:0/3212502962 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f27c80037a0 con 0x7f27dc074cd0 2024-04-23T14:52:52.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.650+0000 7f27e381a740 1 Processor -- start 2024-04-23T14:52:52.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.651+0000 7f27e381a740 1 -- start start 2024-04-23T14:52:52.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.651+0000 7f27e381a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f5bf0ba30 0x562f5bf0be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:52.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.651+0000 7f27e381a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f5bf0c350 0x562f5bf164e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:52.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.651+0000 7f27e381a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f5bf16a20 0x562f5bf18e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:52.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.651+0000 7f27e381a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562f5be711f0 con 0x562f5bf0ba30 2024-04-23T14:52:52.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.651+0000 7f27e381a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562f5be17350 con 0x562f5bf16a20 2024-04-23T14:52:52.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.651+0000 7f27e381a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562f5be6d1b0 con 0x562f5bf0c350 2024-04-23T14:52:52.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.651+0000 7f27d9400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f5bf16a20 0x562f5bf18e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:52.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.651+0000 7f27d9e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f5bf0c350 0x562f5bf164e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:52.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.651+0000 7f27d9e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f5bf0c350 0x562f5bf164e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:45892/0 (socket says 172.21.15.28:45892) 2024-04-23T14:52:52.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.651+0000 7f27d9e00640 1 -- 172.21.15.28:0/4166854469 learned_addr learned my addr 172.21.15.28:0/4166854469 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:52.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.651+0000 7f27d9400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f5bf16a20 0x562f5bf18e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60164/0 (socket says 172.21.15.28:60164) 2024-04-23T14:52:52.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.652+0000 7f27da800640 1 --2- 172.21.15.28:0/4166854469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f5bf0ba30 0x562f5bf0be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:52.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.652+0000 7f27d2a00640 1 --2- 172.21.15.28:0/3212502962 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f27c00a3da0 0x7f27c00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:52.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.652+0000 7f27d9400640 1 -- 172.21.15.28:0/4166854469 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f5bf0c350 msgr2=0x562f5bf164e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:52.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.652+0000 7f27d9400640 1 --2- 172.21.15.28:0/4166854469 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f5bf0c350 0x562f5bf164e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.652+0000 7f27d9400640 1 -- 172.21.15.28:0/4166854469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f5bf0ba30 msgr2=0x562f5bf0be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:52.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.652+0000 7f27d9400640 1 --2- 172.21.15.28:0/4166854469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f5bf0ba30 0x562f5bf0be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.652+0000 7f27d9400640 1 -- 172.21.15.28:0/4166854469 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f5be6bf50 con 0x562f5bf16a20 2024-04-23T14:52:52.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.652+0000 7f27da800640 1 --2- 172.21.15.28:0/3212502962 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f27c00a3da0 0x7f27c00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:52.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.652+0000 7f27d2a00640 1 -- 172.21.15.28:0/3212502962 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7f27c80d0a10 con 0x7f27dc074cd0 2024-04-23T14:52:52.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.652+0000 7f27d9400640 1 --2- 172.21.15.28:0/4166854469 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f5bf16a20 0x562f5bf18e10 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f27c80ce0f0 tx=0x7f27c8098a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c0ba0200e4e0e43 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:52.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.653+0000 7f27c7000640 1 -- 172.21.15.28:0/4166854469 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c809c480 con 0x562f5bf16a20 2024-04-23T14:52:52.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.653+0000 7f27c7000640 1 -- 172.21.15.28:0/4166854469 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27c809d040 con 0x562f5bf16a20 2024-04-23T14:52:52.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.653+0000 7f27c7000640 1 -- 172.21.15.28:0/4166854469 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c809c2a0 con 0x562f5bf16a20 2024-04-23T14:52:52.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.654+0000 7f27e381a740 1 -- 172.21.15.28:0/4166854469 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f5bf16a20 msgr2=0x562f5bf18e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:52.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.654+0000 7f27e381a740 1 --2- 172.21.15.28:0/4166854469 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f5bf16a20 0x562f5bf18e10 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f27c80ce0f0 tx=0x7f27c8098a10 comp rx=0 tx=0).stop 2024-04-23T14:52:52.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.654+0000 7f27da800640 1 --2- 172.21.15.28:0/3212502962 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f27c00a3da0 0x7f27c00a6260 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f27d4005fd0 tx=0x7f27d4004a40 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:52.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.654+0000 7f27e381a740 1 -- 172.21.15.28:0/4166854469 shutdown_connections 2024-04-23T14:52:52.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.654+0000 7f27e381a740 1 --2- 172.21.15.28:0/4166854469 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f5bf16a20 0x562f5bf18e10 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.654+0000 7f27e381a740 1 --2- 172.21.15.28:0/4166854469 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f5bf0c350 0x562f5bf164e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.654+0000 7f27e381a740 1 --2- 172.21.15.28:0/4166854469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f5bf0ba30 0x562f5bf0be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.654+0000 7f27e381a740 1 -- 172.21.15.28:0/4166854469 >> 172.21.15.28:0/4166854469 conn(0x562f5bf07d50 msgr2=0x562f5bf08130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:52.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.654+0000 7f27e381a740 1 -- 172.21.15.28:0/4166854469 shutdown_connections 2024-04-23T14:52:52.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.654+0000 7f27e381a740 1 -- 172.21.15.28:0/4166854469 wait complete. 2024-04-23T14:52:52.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.654+0000 7f27e381a740 1 -- start start 2024-04-23T14:52:52.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.655+0000 7f27e381a740 1 Processor -- start 2024-04-23T14:52:52.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.655+0000 7f27e381a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f5bf0ba30 0x562f5bf13a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:52.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.655+0000 7f27e381a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f5bf0c350 0x562f5bf13f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:52.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.655+0000 7f27e381a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f5bf16a20 0x562f5bf14480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:52.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.655+0000 7f27e381a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562f5bf19810 con 0x562f5bf0c350 2024-04-23T14:52:52.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.655+0000 7f27e381a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562f5be2a690 con 0x562f5bf0ba30 2024-04-23T14:52:52.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.655+0000 7f27e381a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562f5bf19990 con 0x562f5bf16a20 2024-04-23T14:52:52.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.655+0000 7f27e381a740 10 client.0 _ll_register_callbacks cb 0x562f5be66650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:52:52.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.655+0000 7f27d9e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f5bf0ba30 0x562f5bf13a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:52.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.656+0000 7f27d9e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f5bf0ba30 0x562f5bf13a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60166/0 (socket says 172.21.15.28:60166) 2024-04-23T14:52:52.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.656+0000 7f27d9e00640 1 -- 172.21.15.28:0/4049173655 learned_addr learned my addr 172.21.15.28:0/4049173655 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:52.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.656+0000 7f27d9400640 1 --2- 172.21.15.28:0/4049173655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f5bf0c350 0x562f5bf13f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:52.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.656+0000 7f27da800640 1 --2- 172.21.15.28:0/4049173655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f5bf16a20 0x562f5bf14480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:52.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.656+0000 7f27d9e00640 1 -- 172.21.15.28:0/4049173655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f5bf16a20 msgr2=0x562f5bf14480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:52.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.656+0000 7f27d9e00640 1 --2- 172.21.15.28:0/4049173655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f5bf16a20 0x562f5bf14480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.656+0000 7f27d9e00640 1 -- 172.21.15.28:0/4049173655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f5bf0c350 msgr2=0x562f5bf13f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:52.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.656+0000 7f27d9e00640 1 --2- 172.21.15.28:0/4049173655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f5bf0c350 0x562f5bf13f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.656+0000 7f27d9e00640 1 -- 172.21.15.28:0/4049173655 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f5be29920 con 0x562f5bf0ba30 2024-04-23T14:52:52.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.656+0000 7f27d9e00640 1 --2- 172.21.15.28:0/4049173655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f5bf0ba30 0x562f5bf13a00 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f27cc030600 tx=0x7f27cc030630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0b1d44c930e4ca4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:52.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.656+0000 7f27bb400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:52:52.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.657+0000 7f27bb400640 1 -- 172.21.15.28:0/4049173655 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27cc004050 con 0x562f5bf0ba30 2024-04-23T14:52:52.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.657+0000 7f27bb400640 1 -- 172.21.15.28:0/4049173655 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7f27cc0041f0 con 0x562f5bf0ba30 2024-04-23T14:52:52.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.657+0000 7f27e381a740 1 -- 172.21.15.28:0/4049173655 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27c809d040 con 0x562f5bf0ba30 2024-04-23T14:52:52.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.657+0000 7f27bb400640 1 -- 172.21.15.28:0/4049173655 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f27cc04e050 con 0x562f5bf0ba30 2024-04-23T14:52:52.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.657+0000 7f27e381a740 10 client.26591 fetch_fsmap learned FSMap version 20 2024-04-23T14:52:52.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.657+0000 7f27e381a740 1 -- 172.21.15.28:0/4049173655 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x562f5bc4d7d0 con 0x562f5bf0ba30 2024-04-23T14:52:52.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.657+0000 7f27bb400640 1 -- 172.21.15.28:0/4049173655 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27c809d040 con 0x562f5bf0ba30 2024-04-23T14:52:52.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.657+0000 7f27bb400640 1 -- 172.21.15.28:0/4049173655 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f27cc004930 con 0x562f5bf0ba30 2024-04-23T14:52:52.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.658+0000 7f27e381a740 10 client.26591 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:52:52.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.658+0000 7f27e381a740 10 client.26591 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:52:52.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.658+0000 7f27e381a740 20 client.26591 populate_metadata read hostname 'smithi028' 2024-04-23T14:52:52.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.658+0000 7f27e381a740 1 --2- 172.21.15.28:0/4049173655 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562f5bf26d90 0x562f5bf291c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:52.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.658+0000 7f27e381a740 4 client.26591 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:52:52.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.658+0000 7f27e381a740 1 -- 172.21.15.28:0/4049173655 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562f5be7ec10 con 0x562f5bf26d90 2024-04-23T14:52:52.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.658+0000 7f27d9400640 1 --2- 172.21.15.28:0/4049173655 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562f5bf26d90 0x562f5bf291c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:52.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.658+0000 7f27d9400640 1 --2- 172.21.15.28:0/4049173655 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562f5bf26d90 0x562f5bf291c0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc65876fa9aae8f0 server_cookie=e7e7706e7774d8b in_seq=0 out_seq=0 2024-04-23T14:52:52.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.658+0000 7f27bb400640 10 client.26591 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:52.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.659+0000 7f27bb400640 1 -- 172.21.15.28:0/4049173655 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562f5be7ec10 con 0x562f5bf26d90 2024-04-23T14:52:52.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.659+0000 7f27bb400640 10 client.26591 handle_command_reply: tid=0 2024-04-23T14:52:52.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.660+0000 7f27e381a740 1 client.26591 shutdown 2024-04-23T14:52:52.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.660+0000 7f27e381a740 1 -- 172.21.15.28:0/4049173655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f5bf0ba30 msgr2=0x562f5bf13a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:52.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.660+0000 7f27e381a740 1 --2- 172.21.15.28:0/4049173655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f5bf0ba30 0x562f5bf13a00 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f27cc030600 tx=0x7f27cc030630 comp rx=0 tx=0).stop 2024-04-23T14:52:52.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.660+0000 7f27e381a740 1 -- 172.21.15.28:0/4049173655 shutdown_connections 2024-04-23T14:52:52.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.660+0000 7f27e381a740 1 --2- 172.21.15.28:0/4049173655 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562f5bf26d90 0x562f5bf291c0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.660+0000 7f27e381a740 1 --2- 172.21.15.28:0/4049173655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f5bf16a20 0x562f5bf14480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.660+0000 7f27e381a740 1 --2- 172.21.15.28:0/4049173655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f5bf0c350 0x562f5bf13f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.660+0000 7f27e381a740 1 --2- 172.21.15.28:0/4049173655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f5bf0ba30 0x562f5bf13a00 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.660+0000 7f27bb400640 0 client.26591 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:52.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.660+0000 7f27e381a740 1 -- 172.21.15.28:0/4049173655 >> 172.21.15.28:0/4049173655 conn(0x562f5bf07d50 msgr2=0x562f5bf0db50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:52.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.660+0000 7f27e381a740 1 -- 172.21.15.28:0/4049173655 shutdown_connections 2024-04-23T14:52:52.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.660+0000 7f27e381a740 1 -- 172.21.15.28:0/4049173655 wait complete. 2024-04-23T14:52:52.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.660+0000 7f27e381a740 20 client.26591 trim_cache size 0 max 16384 2024-04-23T14:52:52.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.683+0000 7f27e381a740 1 Processor -- start 2024-04-23T14:52:52.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.683+0000 7f27e381a740 1 -- start start 2024-04-23T14:52:52.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.683+0000 7f27e381a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f5bf0ba30 0x562f5bf226d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:52.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.683+0000 7f27e381a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f5bf0c350 0x562f5bf2ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:52.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.683+0000 7f27e381a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f5bf16a20 0x562f5bf2efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:52.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.683+0000 7f27e381a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f27dc1adea0 con 0x562f5bf16a20 2024-04-23T14:52:52.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.683+0000 7f27e381a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f27dc1b0700 con 0x562f5bf0ba30 2024-04-23T14:52:52.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.683+0000 7f27e381a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f27dc1ae4f0 con 0x562f5bf0c350 2024-04-23T14:52:52.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.683+0000 7f27d9e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f5bf0ba30 0x562f5bf226d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:52.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.683+0000 7f27d9400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f5bf0c350 0x562f5bf2ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:52.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.683+0000 7f27d9e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f5bf0ba30 0x562f5bf226d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60168/0 (socket says 172.21.15.28:60168) 2024-04-23T14:52:52.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.683+0000 7f27d9e00640 1 -- 172.21.15.28:0/249207697 learned_addr learned my addr 172.21.15.28:0/249207697 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:52.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.684+0000 7f27d9400640 1 -- 172.21.15.28:0/249207697 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f5bf0ba30 msgr2=0x562f5bf226d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:52.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.684+0000 7f27d9400640 1 --2- 172.21.15.28:0/249207697 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f5bf0ba30 0x562f5bf226d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.684+0000 7f27da800640 1 --2- 172.21.15.28:0/249207697 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f5bf16a20 0x562f5bf2efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:52.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.684+0000 7f27d9400640 1 -- 172.21.15.28:0/249207697 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f5bf16a20 msgr2=0x562f5bf2efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:52.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.684+0000 7f27d9400640 1 --2- 172.21.15.28:0/249207697 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f5bf16a20 0x562f5bf2efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.684+0000 7f27d9400640 1 -- 172.21.15.28:0/249207697 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f5bb833f0 con 0x562f5bf0c350 2024-04-23T14:52:52.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.684+0000 7f27d9e00640 1 --2- 172.21.15.28:0/249207697 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f5bf0ba30 0x562f5bf226d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:52:52.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.684+0000 7f27d9400640 1 --2- 172.21.15.28:0/249207697 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f5bf0c350 0x562f5bf2ea90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f27c809abf0 tx=0x7f27c8098b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=189c5f9da221d892 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:52.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.684+0000 7f27c6600640 1 -- 172.21.15.28:0/249207697 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c809c2a0 con 0x562f5bf0c350 2024-04-23T14:52:52.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.684+0000 7f27c6600640 1 -- 172.21.15.28:0/249207697 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27c809d040 con 0x562f5bf0c350 2024-04-23T14:52:52.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.685+0000 7f27c6600640 1 -- 172.21.15.28:0/249207697 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c809c480 con 0x562f5bf0c350 2024-04-23T14:52:52.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.685+0000 7f27e381a740 1 -- 172.21.15.28:0/249207697 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f5bf0c350 msgr2=0x562f5bf2ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:52.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.685+0000 7f27e381a740 1 --2- 172.21.15.28:0/249207697 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f5bf0c350 0x562f5bf2ea90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f27c809abf0 tx=0x7f27c8098b40 comp rx=0 tx=0).stop 2024-04-23T14:52:52.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.685+0000 7f27e381a740 1 -- 172.21.15.28:0/249207697 shutdown_connections 2024-04-23T14:52:52.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.685+0000 7f27e381a740 1 --2- 172.21.15.28:0/249207697 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f5bf16a20 0x562f5bf2efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.685+0000 7f27e381a740 1 --2- 172.21.15.28:0/249207697 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f5bf0c350 0x562f5bf2ea90 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.685+0000 7f27e381a740 1 --2- 172.21.15.28:0/249207697 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f5bf0ba30 0x562f5bf226d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.685+0000 7f27e381a740 1 -- 172.21.15.28:0/249207697 >> 172.21.15.28:0/249207697 conn(0x562f5bf07d50 msgr2=0x562f5bf02e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:52.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.685+0000 7f27e381a740 1 -- 172.21.15.28:0/249207697 shutdown_connections 2024-04-23T14:52:52.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.685+0000 7f27e381a740 1 -- 172.21.15.28:0/249207697 wait complete. 2024-04-23T14:52:52.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.686+0000 7f27e381a740 1 -- start start 2024-04-23T14:52:52.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.686+0000 7f27e381a740 1 Processor -- start 2024-04-23T14:52:52.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.687+0000 7f27e381a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f5bf0ba30 0x562f5bf1eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:52.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.687+0000 7f27e381a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f5bf0c350 0x562f5bf10350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:52.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.687+0000 7f27e381a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f5bf16a20 0x562f5bf10890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:52.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.687+0000 7f27e381a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f27dc1aebc0 con 0x562f5bf0ba30 2024-04-23T14:52:52.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.687+0000 7f27e381a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f27dc1b0030 con 0x562f5bf0c350 2024-04-23T14:52:52.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.687+0000 7f27e381a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f27dc1af290 con 0x562f5bf16a20 2024-04-23T14:52:52.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.687+0000 7f27e381a740 10 client.0 _ll_register_callbacks cb 0x7f27cc0096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:52:52.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.687+0000 7f27d9400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f5bf0c350 0x562f5bf10350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:52.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.687+0000 7f27d9400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f5bf0c350 0x562f5bf10350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60182/0 (socket says 172.21.15.28:60182) 2024-04-23T14:52:52.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.687+0000 7f27d9400640 1 -- 172.21.15.28:0/2973662098 learned_addr learned my addr 172.21.15.28:0/2973662098 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:52.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.687+0000 7f27d9e00640 1 --2- 172.21.15.28:0/2973662098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f5bf0ba30 0x562f5bf1eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:52.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.687+0000 7f27da800640 1 --2- 172.21.15.28:0/2973662098 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f5bf16a20 0x562f5bf10890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:52.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.687+0000 7f27d9400640 1 -- 172.21.15.28:0/2973662098 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f5bf16a20 msgr2=0x562f5bf10890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:52.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.687+0000 7f27d9400640 1 --2- 172.21.15.28:0/2973662098 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f5bf16a20 0x562f5bf10890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.687+0000 7f27d9400640 1 -- 172.21.15.28:0/2973662098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f5bf0ba30 msgr2=0x562f5bf1eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:52.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.687+0000 7f27d9400640 1 --2- 172.21.15.28:0/2973662098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f5bf0ba30 0x562f5bf1eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.687+0000 7f27d9400640 1 -- 172.21.15.28:0/2973662098 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f5bb430f0 con 0x562f5bf0c350 2024-04-23T14:52:52.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.687+0000 7f27d9e00640 1 --2- 172.21.15.28:0/2973662098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f5bf0ba30 0x562f5bf1eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:52:52.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.688+0000 7f27d9400640 1 --2- 172.21.15.28:0/2973662098 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f5bf0c350 0x562f5bf10350 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f27c800f010 tx=0x7f27c80d46b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43010bae6748b59f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:52.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.688+0000 7f27bb400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:52:52.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.688+0000 7f27bb400640 1 -- 172.21.15.28:0/2973662098 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c809c2a0 con 0x562f5bf0c350 2024-04-23T14:52:52.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.688+0000 7f27bb400640 1 -- 172.21.15.28:0/2973662098 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7f27c80d0e00 con 0x562f5bf0c350 2024-04-23T14:52:52.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.688+0000 7f27e381a740 1 -- 172.21.15.28:0/2973662098 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27c809d040 con 0x562f5bf0c350 2024-04-23T14:52:52.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.688+0000 7f27bb400640 1 -- 172.21.15.28:0/2973662098 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f27c80a0cf0 con 0x562f5bf0c350 2024-04-23T14:52:52.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.688+0000 7f27e381a740 10 client.26597 fetch_fsmap learned FSMap version 20 2024-04-23T14:52:52.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.688+0000 7f27e381a740 1 -- 172.21.15.28:0/2973662098 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x562f5be36810 con 0x562f5bf0c350 2024-04-23T14:52:52.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.688+0000 7f27bb400640 1 -- 172.21.15.28:0/2973662098 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27c809d040 con 0x562f5bf0c350 2024-04-23T14:52:52.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.689+0000 7f27bb400640 1 -- 172.21.15.28:0/2973662098 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f27c8098fb0 con 0x562f5bf0c350 2024-04-23T14:52:52.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.689+0000 7f27e381a740 10 client.26597 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:52:52.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.689+0000 7f27e381a740 10 client.26597 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:52:52.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.689+0000 7f27e381a740 20 client.26597 populate_metadata read hostname 'smithi028' 2024-04-23T14:52:52.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.689+0000 7f27e381a740 1 --2- 172.21.15.28:0/2973662098 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562f5bf8d0e0 0x562f5bf8f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:52.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.689+0000 7f27e381a740 4 client.26597 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:52:52.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.689+0000 7f27e381a740 1 -- 172.21.15.28:0/2973662098 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562f5be78a80 con 0x562f5bf8d0e0 2024-04-23T14:52:52.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.689+0000 7f27d9e00640 1 --2- 172.21.15.28:0/2973662098 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562f5bf8d0e0 0x562f5bf8f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:52.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.690+0000 7f27d9e00640 1 --2- 172.21.15.28:0/2973662098 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562f5bf8d0e0 0x562f5bf8f4d0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=205b7e3e2c0cdc69 server_cookie=327c1500969e0aba in_seq=0 out_seq=0 2024-04-23T14:52:52.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.690+0000 7f27bb400640 10 client.26597 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:52.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.691+0000 7f27bb400640 1 -- 172.21.15.28:0/2973662098 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x562f5be78a80 con 0x562f5bf8d0e0 2024-04-23T14:52:52.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.691+0000 7f27bb400640 10 client.26597 handle_command_reply: tid=0 2024-04-23T14:52:52.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.691+0000 7f27e381a740 1 client.26597 shutdown 2024-04-23T14:52:52.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.691+0000 7f27e381a740 1 -- 172.21.15.28:0/2973662098 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f5bf0c350 msgr2=0x562f5bf10350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:52.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.691+0000 7f27e381a740 1 --2- 172.21.15.28:0/2973662098 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f5bf0c350 0x562f5bf10350 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f27c800f010 tx=0x7f27c80d46b0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.692+0000 7f27e381a740 1 -- 172.21.15.28:0/2973662098 shutdown_connections 2024-04-23T14:52:52.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.692+0000 7f27e381a740 1 --2- 172.21.15.28:0/2973662098 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562f5bf8d0e0 0x562f5bf8f4d0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.692+0000 7f27e381a740 1 --2- 172.21.15.28:0/2973662098 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562f5bf0c350 0x562f5bf10350 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.692+0000 7f27e381a740 1 --2- 172.21.15.28:0/2973662098 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562f5bf16a20 0x562f5bf10890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.692+0000 7f27e381a740 1 --2- 172.21.15.28:0/2973662098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f5bf0ba30 0x562f5bf1eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.692+0000 7f27e381a740 1 -- 172.21.15.28:0/2973662098 >> 172.21.15.28:0/2973662098 conn(0x562f5bf07d50 msgr2=0x562f5bf06740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:52.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.692+0000 7f27bb400640 0 client.26597 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:52.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.692+0000 7f27e381a740 1 -- 172.21.15.28:0/2973662098 shutdown_connections 2024-04-23T14:52:52.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.692+0000 7f27e381a740 1 -- 172.21.15.28:0/2973662098 wait complete. 2024-04-23T14:52:52.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.692+0000 7f27e381a740 20 client.26597 trim_cache size 0 max 16384 2024-04-23T14:52:52.696 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:52:52.696 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-23T14:52:52.697 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-23T14:52:52.697 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:52:52.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.693+0000 7f27db200640 1 -- 172.21.15.28:0/3212502962 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f27c00a3da0 msgr2=0x7f27c00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:52.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.693+0000 7f27db200640 1 --2- 172.21.15.28:0/3212502962 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f27c00a3da0 0x7f27c00a6260 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f27d4005fd0 tx=0x7f27d4004a40 comp rx=0 tx=0).stop 2024-04-23T14:52:52.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.693+0000 7f27db200640 1 -- 172.21.15.28:0/3212502962 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f27dc074cd0 msgr2=0x7f27dc1a0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:52.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.693+0000 7f27db200640 1 --2- 172.21.15.28:0/3212502962 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f27dc074cd0 0x7f27dc1a0bc0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f27c800e9c0 tx=0x7f27c800edc0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.693+0000 7f27db200640 1 -- 172.21.15.28:0/3212502962 shutdown_connections 2024-04-23T14:52:52.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.693+0000 7f27db200640 1 --2- 172.21.15.28:0/3212502962 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f27c00a3da0 0x7f27c00a6260 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.693+0000 7f27db200640 1 --2- 172.21.15.28:0/3212502962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27dc10e750 0x7f27dc19a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.693+0000 7f27db200640 1 --2- 172.21.15.28:0/3212502962 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f27dc074cd0 0x7f27dc1a0bc0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.693+0000 7f27db200640 1 --2- 172.21.15.28:0/3212502962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27dc0743b0 0x7f27dc1a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.693+0000 7f27db200640 1 -- 172.21.15.28:0/3212502962 >> 172.21.15.28:0/3212502962 conn(0x7f27dc0fd9a0 msgr2=0x7f27dc0ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:52.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.693+0000 7f27db200640 1 -- 172.21.15.28:0/3212502962 shutdown_connections 2024-04-23T14:52:52.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.693+0000 7f27db200640 1 -- 172.21.15.28:0/3212502962 wait complete. 2024-04-23T14:52:52.710 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-23T14:52:52.710 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:52:52.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.912+0000 7f086a000640 1 -- 172.21.15.28:0/4212755144 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0864102a40 msgr2=0x7f0864106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.912+0000 7f086a000640 1 --2- 172.21.15.28:0/4212755144 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0864102a40 0x7f0864106a80 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f0850009860 tx=0x7f085002a340 comp rx=0 tx=0).stop 2024-04-23T14:52:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.912+0000 7f086a000640 1 -- 172.21.15.28:0/4212755144 shutdown_connections 2024-04-23T14:52:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.912+0000 7f086a000640 1 --2- 172.21.15.28:0/4212755144 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0864107160 0x7f086410ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.912+0000 7f086a000640 1 --2- 172.21.15.28:0/4212755144 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0864102a40 0x7f0864106a80 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.912+0000 7f086a000640 1 --2- 172.21.15.28:0/4212755144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0864102090 0x7f0864102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.912+0000 7f086a000640 1 -- 172.21.15.28:0/4212755144 >> 172.21.15.28:0/4212755144 conn(0x7f08640fd930 msgr2=0x7f08640ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.913+0000 7f086a000640 1 -- 172.21.15.28:0/4212755144 shutdown_connections 2024-04-23T14:52:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.913+0000 7f086a000640 1 -- 172.21.15.28:0/4212755144 wait complete. 2024-04-23T14:52:52.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.914+0000 7f086a000640 1 Processor -- start 2024-04-23T14:52:52.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.914+0000 7f086a000640 1 -- start start 2024-04-23T14:52:52.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.914+0000 7f086a000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0864102090 0x7f08641a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:52.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.914+0000 7f086a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0864107160 0x7f08641a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:52.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.914+0000 7f086a000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f086419a790 0x7f086419ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:52.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.914+0000 7f086a000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f08641121d0 con 0x7f0864107160 2024-04-23T14:52:52.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.914+0000 7f086a000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0864112050 con 0x7f0864102090 2024-04-23T14:52:52.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.914+0000 7f086a000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0864112350 con 0x7f086419a790 2024-04-23T14:52:52.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.914+0000 7f0868c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0864102090 0x7f08641a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:52.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.914+0000 7f0863e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0864107160 0x7f08641a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:52.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.914+0000 7f0869600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f086419a790 0x7f086419ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:52.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.915+0000 7f0868c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0864102090 0x7f08641a0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60196/0 (socket says 172.21.15.28:60196) 2024-04-23T14:52:52.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.915+0000 7f0868c00640 1 -- 172.21.15.28:0/1238719606 learned_addr learned my addr 172.21.15.28:0/1238719606 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:52.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.915+0000 7f0868c00640 1 -- 172.21.15.28:0/1238719606 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f086419a790 msgr2=0x7f086419ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:52.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.915+0000 7f0868c00640 1 --2- 172.21.15.28:0/1238719606 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f086419a790 0x7f086419ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.915+0000 7f0868c00640 1 -- 172.21.15.28:0/1238719606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0864107160 msgr2=0x7f08641a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:52.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.915+0000 7f0868c00640 1 --2- 172.21.15.28:0/1238719606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0864107160 0x7f08641a0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:52.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.915+0000 7f0868c00640 1 -- 172.21.15.28:0/1238719606 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f086419b390 con 0x7f0864102090 2024-04-23T14:52:52.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.915+0000 7f0868c00640 1 --2- 172.21.15.28:0/1238719606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0864102090 0x7f08641a0680 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f085800e9c0 tx=0x7f085800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:52.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.915+0000 7f0861600640 1 -- 172.21.15.28:0/1238719606 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085800ccc0 con 0x7f0864102090 2024-04-23T14:52:52.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.915+0000 7f0861600640 1 -- 172.21.15.28:0/1238719606 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0858005500 con 0x7f0864102090 2024-04-23T14:52:52.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.916+0000 7f086a000640 1 -- 172.21.15.28:0/1238719606 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08641a7420 con 0x7f0864102090 2024-04-23T14:52:52.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.916+0000 7f0861600640 1 -- 172.21.15.28:0/1238719606 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0858010430 con 0x7f0864102090 2024-04-23T14:52:52.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.916+0000 7f086a000640 1 -- 172.21.15.28:0/1238719606 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08641a77d0 con 0x7f0864102090 2024-04-23T14:52:52.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.917+0000 7f086a000640 1 -- 172.21.15.28:0/1238719606 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0864103560 con 0x7f0864102090 2024-04-23T14:52:52.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.919+0000 7f0861600640 1 -- 172.21.15.28:0/1238719606 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f08580046c0 con 0x7f0864102090 2024-04-23T14:52:52.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.920+0000 7f0861600640 1 --2- 172.21.15.28:0/1238719606 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f084c0a3d20 0x7f084c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:52.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.920+0000 7f0861600640 1 -- 172.21.15.28:0/1238719606 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7f08580d0b40 con 0x7f0864102090 2024-04-23T14:52:52.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.921+0000 7f0863e00640 1 --2- 172.21.15.28:0/1238719606 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f084c0a3d20 0x7f084c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:52.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.921+0000 7f0863e00640 1 --2- 172.21.15.28:0/1238719606 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f084c0a3d20 0x7f084c0a61e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0850009830 tx=0x7f08500056d0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:52.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:52.926+0000 7f0861600640 1 -- 172.21.15.28:0/1238719606 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f085809d170 con 0x7f0864102090 2024-04-23T14:52:53.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:52 smithi105 ceph-mon[32555]: pgmap v272: 97 pgs: 97 active+clean; 446 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 129 op/s 2024-04-23T14:52:53.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:52 smithi038 ceph-mon[32348]: pgmap v272: 97 pgs: 97 active+clean; 446 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 129 op/s 2024-04-23T14:52:53.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:52 smithi028 ceph-mon[27810]: pgmap v272: 97 pgs: 97 active+clean; 446 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 129 op/s 2024-04-23T14:52:53.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.190+0000 7f086a000640 1 -- 172.21.15.28:0/1238719606 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0864102aa0 con 0x7f0864102090 2024-04-23T14:52:53.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.191+0000 7f0861600640 1 -- 172.21.15.28:0/1238719606 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7f085809ca70 con 0x7f0864102090 2024-04-23T14:52:53.191 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:52:53.191 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:52:53.191 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:52:53.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.194+0000 7f086a000640 1 -- 172.21.15.28:0/1238719606 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f084c0a3d20 msgr2=0x7f084c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:53.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.194+0000 7f086a000640 1 --2- 172.21.15.28:0/1238719606 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f084c0a3d20 0x7f084c0a61e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0850009830 tx=0x7f08500056d0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.194+0000 7f086a000640 1 -- 172.21.15.28:0/1238719606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0864102090 msgr2=0x7f08641a0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:53.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.194+0000 7f086a000640 1 --2- 172.21.15.28:0/1238719606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0864102090 0x7f08641a0680 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f085800e9c0 tx=0x7f085800ee80 comp rx=0 tx=0).stop 2024-04-23T14:52:53.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.194+0000 7f086a000640 1 -- 172.21.15.28:0/1238719606 shutdown_connections 2024-04-23T14:52:53.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.194+0000 7f086a000640 1 --2- 172.21.15.28:0/1238719606 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f084c0a3d20 0x7f084c0a61e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.194+0000 7f086a000640 1 --2- 172.21.15.28:0/1238719606 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f086419a790 0x7f086419ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.194+0000 7f086a000640 1 --2- 172.21.15.28:0/1238719606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0864107160 0x7f08641a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.194+0000 7f086a000640 1 --2- 172.21.15.28:0/1238719606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0864102090 0x7f08641a0680 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.194+0000 7f086a000640 1 -- 172.21.15.28:0/1238719606 >> 172.21.15.28:0/1238719606 conn(0x7f08640fd930 msgr2=0x7f08640ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:53.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.194+0000 7f086a000640 1 -- 172.21.15.28:0/1238719606 shutdown_connections 2024-04-23T14:52:53.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.194+0000 7f086a000640 1 -- 172.21.15.28:0/1238719606 wait complete. 2024-04-23T14:52:53.216 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-23T14:52:53.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.428+0000 7fa855a00640 1 -- 172.21.15.28:0/733773135 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa850103980 msgr2=0x7fa85010be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:53.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.428+0000 7fa855a00640 1 --2- 172.21.15.28:0/733773135 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa850103980 0x7fa85010be60 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fa838009900 tx=0x7fa83802a480 comp rx=0 tx=0).stop 2024-04-23T14:52:53.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.428+0000 7fa855a00640 1 -- 172.21.15.28:0/733773135 shutdown_connections 2024-04-23T14:52:53.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.428+0000 7fa855a00640 1 --2- 172.21.15.28:0/733773135 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa850103980 0x7fa85010be60 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.428+0000 7fa855a00640 1 --2- 172.21.15.28:0/733773135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa85010c480 0x7fa85010e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.428+0000 7fa855a00640 1 --2- 172.21.15.28:0/733773135 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa850103060 0x7fa850103440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.428+0000 7fa855a00640 1 -- 172.21.15.28:0/733773135 >> 172.21.15.28:0/733773135 conn(0x7fa8500fc880 msgr2=0x7fa8500feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:53.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.429+0000 7fa855a00640 1 -- 172.21.15.28:0/733773135 shutdown_connections 2024-04-23T14:52:53.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.429+0000 7fa855a00640 1 -- 172.21.15.28:0/733773135 wait complete. 2024-04-23T14:52:53.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.430+0000 7fa855a00640 1 Processor -- start 2024-04-23T14:52:53.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.430+0000 7fa855a00640 1 -- start start 2024-04-23T14:52:53.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.430+0000 7fa855a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa850103060 0x7fa8501a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:53.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.430+0000 7fa855a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa850103980 0x7fa8501a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:53.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.430+0000 7fa855a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa85010c480 0x7fa85019a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:53.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.430+0000 7fa855a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa8501122b0 con 0x7fa85010c480 2024-04-23T14:52:53.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.430+0000 7fa855a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa850112130 con 0x7fa850103060 2024-04-23T14:52:53.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.430+0000 7fa855a00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa850112430 con 0x7fa850103980 2024-04-23T14:52:53.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.430+0000 7fa84fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa850103060 0x7fa8501a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:53.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.430+0000 7fa84fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa850103060 0x7fa8501a0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60218/0 (socket says 172.21.15.28:60218) 2024-04-23T14:52:53.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.431+0000 7fa84fe00640 1 -- 172.21.15.28:0/669387841 learned_addr learned my addr 172.21.15.28:0/669387841 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:53.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.431+0000 7fa855000640 1 --2- 172.21.15.28:0/669387841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa85010c480 0x7fa85019a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:53.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.431+0000 7fa847400640 1 --2- 172.21.15.28:0/669387841 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa850103980 0x7fa8501a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:53.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.431+0000 7fa84fe00640 1 -- 172.21.15.28:0/669387841 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa850103980 msgr2=0x7fa8501a0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:53.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.431+0000 7fa84fe00640 1 --2- 172.21.15.28:0/669387841 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa850103980 0x7fa8501a0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.431+0000 7fa84fe00640 1 -- 172.21.15.28:0/669387841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa85010c480 msgr2=0x7fa85019a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:53.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.431+0000 7fa84fe00640 1 --2- 172.21.15.28:0/669387841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa85010c480 0x7fa85019a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.431+0000 7fa84fe00640 1 -- 172.21.15.28:0/669387841 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa85019b020 con 0x7fa850103060 2024-04-23T14:52:53.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.431+0000 7fa84fe00640 1 --2- 172.21.15.28:0/669387841 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa850103060 0x7fa8501a0690 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fa84000e9f0 tx=0x7fa84000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:53.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.432+0000 7fa84d600640 1 -- 172.21.15.28:0/669387841 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa84000cd80 con 0x7fa850103060 2024-04-23T14:52:53.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.432+0000 7fa84d600640 1 -- 172.21.15.28:0/669387841 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa840005500 con 0x7fa850103060 2024-04-23T14:52:53.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.432+0000 7fa84d600640 1 -- 172.21.15.28:0/669387841 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa840010690 con 0x7fa850103060 2024-04-23T14:52:53.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.432+0000 7fa855a00640 1 -- 172.21.15.28:0/669387841 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa85019b310 con 0x7fa850103060 2024-04-23T14:52:53.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.432+0000 7fa855a00640 1 -- 172.21.15.28:0/669387841 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8501a74e0 con 0x7fa850103060 2024-04-23T14:52:53.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.436+0000 7fa84d600640 1 -- 172.21.15.28:0/669387841 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fa8400046c0 con 0x7fa850103060 2024-04-23T14:52:53.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.436+0000 7fa859e76740 1 Processor -- start 2024-04-23T14:52:53.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.436+0000 7fa859e76740 1 -- start start 2024-04-23T14:52:53.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.437+0000 7fa859e76740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574a7b79b40 0x5574a7b79f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:53.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.437+0000 7fa859e76740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574a7b7a460 0x5574a7b845f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:53.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.437+0000 7fa859e76740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574a7b84b30 0x5574a7b86f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:53.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.437+0000 7fa859e76740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5574a7adf1d0 con 0x5574a7b7a460 2024-04-23T14:52:53.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.437+0000 7fa859e76740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5574a7a85450 con 0x5574a7b84b30 2024-04-23T14:52:53.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.437+0000 7fa859e76740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5574a7adb190 con 0x5574a7b79b40 2024-04-23T14:52:53.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.437+0000 7fa84fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574a7b7a460 0x5574a7b845f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:53.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.437+0000 7fa855000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574a7b79b40 0x5574a7b79f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:53.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.437+0000 7fa855000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574a7b79b40 0x5574a7b79f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:45936/0 (socket says 172.21.15.28:45936) 2024-04-23T14:52:53.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.437+0000 7fa847400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574a7b84b30 0x5574a7b86f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:53.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.437+0000 7fa855000640 1 -- 172.21.15.28:0/508174743 learned_addr learned my addr 172.21.15.28:0/508174743 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:53.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.438+0000 7fa855000640 1 -- 172.21.15.28:0/508174743 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574a7b84b30 msgr2=0x5574a7b86f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:53.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.438+0000 7fa855000640 1 --2- 172.21.15.28:0/508174743 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574a7b84b30 0x5574a7b86f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.438+0000 7fa855000640 1 -- 172.21.15.28:0/508174743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574a7b7a460 msgr2=0x5574a7b845f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:53.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.438+0000 7fa855000640 1 --2- 172.21.15.28:0/508174743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574a7b7a460 0x5574a7b845f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.438+0000 7fa855000640 1 -- 172.21.15.28:0/508174743 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574a7b42310 con 0x5574a7b79b40 2024-04-23T14:52:53.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.438+0000 7fa84fe00640 1 --2- 172.21.15.28:0/508174743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574a7b7a460 0x5574a7b845f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:52:53.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.438+0000 7fa855000640 1 --2- 172.21.15.28:0/508174743 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574a7b79b40 0x5574a7b79f20 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa848009f90 tx=0x7fa848010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f684bf7cb0602023 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:53.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.438+0000 7fa84d600640 1 --2- 172.21.15.28:0/669387841 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa8340a3da0 0x7fa8340a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:53.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.438+0000 7fa84d600640 1 -- 172.21.15.28:0/669387841 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7fa8400d0c00 con 0x7fa850103060 2024-04-23T14:52:53.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.438+0000 7fa845600640 1 -- 172.21.15.28:0/508174743 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa848011700 con 0x5574a7b79b40 2024-04-23T14:52:53.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.438+0000 7fa845600640 1 -- 172.21.15.28:0/508174743 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa848011d80 con 0x5574a7b79b40 2024-04-23T14:52:53.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.438+0000 7fa845600640 1 -- 172.21.15.28:0/508174743 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa848018380 con 0x5574a7b79b40 2024-04-23T14:52:53.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.438+0000 7fa84fe00640 1 --2- 172.21.15.28:0/669387841 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa8340a3da0 0x7fa8340a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:53.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.439+0000 7fa859e76740 1 -- 172.21.15.28:0/508174743 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574a7b79b40 msgr2=0x5574a7b79f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:53.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.439+0000 7fa84fe00640 1 --2- 172.21.15.28:0/669387841 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa8340a3da0 0x7fa8340a6260 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x5574a7b7b100 tx=0x7fa840004a40 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:53.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.439+0000 7fa859e76740 1 --2- 172.21.15.28:0/508174743 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574a7b79b40 0x5574a7b79f20 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa848009f90 tx=0x7fa848010860 comp rx=0 tx=0).stop 2024-04-23T14:52:53.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.439+0000 7fa859e76740 1 -- 172.21.15.28:0/508174743 shutdown_connections 2024-04-23T14:52:53.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.439+0000 7fa859e76740 1 --2- 172.21.15.28:0/508174743 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574a7b84b30 0x5574a7b86f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.439+0000 7fa859e76740 1 --2- 172.21.15.28:0/508174743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574a7b7a460 0x5574a7b845f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.439+0000 7fa859e76740 1 --2- 172.21.15.28:0/508174743 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574a7b79b40 0x5574a7b79f20 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.439+0000 7fa859e76740 1 -- 172.21.15.28:0/508174743 >> 172.21.15.28:0/508174743 conn(0x5574a7b75e60 msgr2=0x5574a7b76240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:53.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.439+0000 7fa859e76740 1 -- 172.21.15.28:0/508174743 shutdown_connections 2024-04-23T14:52:53.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.439+0000 7fa859e76740 1 -- 172.21.15.28:0/508174743 wait complete. 2024-04-23T14:52:53.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.440+0000 7fa859e76740 1 -- start start 2024-04-23T14:52:53.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.440+0000 7fa859e76740 1 Processor -- start 2024-04-23T14:52:53.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.441+0000 7fa859e76740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574a7b79b40 0x5574a7b845e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:53.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.441+0000 7fa859e76740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574a7b7a460 0x5574a7b818d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:53.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.441+0000 7fa859e76740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574a7b84b30 0x5574a7b82010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:53.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.441+0000 7fa859e76740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5574a7b879c0 con 0x5574a7b79b40 2024-04-23T14:52:53.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.441+0000 7fa859e76740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5574a7a98810 con 0x5574a7b84b30 2024-04-23T14:52:53.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.441+0000 7fa859e76740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5574a7b87b40 con 0x5574a7b7a460 2024-04-23T14:52:53.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.441+0000 7fa859e76740 10 client.0 _ll_register_callbacks cb 0x5574a7a66140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:52:53.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.441+0000 7fa855000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574a7b79b40 0x5574a7b845e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:53.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.441+0000 7fa847400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574a7b7a460 0x5574a7b818d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:53.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.441+0000 7fa855000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574a7b79b40 0x5574a7b845e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44424/0 (socket says 172.21.15.28:44424) 2024-04-23T14:52:53.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.441+0000 7fa855000640 1 -- 172.21.15.28:0/1619515466 learned_addr learned my addr 172.21.15.28:0/1619515466 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:53.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.441+0000 7fa855000640 1 --2- 172.21.15.28:0/1619515466 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574a7b84b30 0x5574a7b82010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:53.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.442+0000 7fa847400640 1 -- 172.21.15.28:0/1619515466 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574a7b84b30 msgr2=0x5574a7b82010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:53.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.442+0000 7fa847400640 1 --2- 172.21.15.28:0/1619515466 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574a7b84b30 0x5574a7b82010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.442+0000 7fa847400640 1 -- 172.21.15.28:0/1619515466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574a7b79b40 msgr2=0x5574a7b845e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:53.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.442+0000 7fa847400640 1 --2- 172.21.15.28:0/1619515466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574a7b79b40 0x5574a7b845e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.442+0000 7fa847400640 1 -- 172.21.15.28:0/1619515466 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574a7aa93f0 con 0x5574a7b7a460 2024-04-23T14:52:53.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.442+0000 7fa855000640 1 --2- 172.21.15.28:0/1619515466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574a7b79b40 0x5574a7b845e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:52:53.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.442+0000 7fa847400640 1 --2- 172.21.15.28:0/1619515466 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574a7b7a460 0x5574a7b818d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa83802aa70 tx=0x7fa838004390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61ebfc04902ed37d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:53.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.442+0000 7fa83e000640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:52:53.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.442+0000 7fa83e000640 1 -- 172.21.15.28:0/1619515466 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa838035040 con 0x5574a7b7a460 2024-04-23T14:52:53.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.442+0000 7fa83e000640 1 -- 172.21.15.28:0/1619515466 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7fa838005450 con 0x5574a7b7a460 2024-04-23T14:52:53.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.442+0000 7fa859e76740 1 -- 172.21.15.28:0/1619515466 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa848011d80 con 0x5574a7b7a460 2024-04-23T14:52:53.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.443+0000 7fa83e000640 1 -- 172.21.15.28:0/1619515466 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fa838005650 con 0x5574a7b7a460 2024-04-23T14:52:53.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.443+0000 7fa859e76740 10 client.26593 fetch_fsmap learned FSMap version 20 2024-04-23T14:52:53.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.443+0000 7fa859e76740 1 -- 172.21.15.28:0/1619515466 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5574a77f13f0 con 0x5574a7b7a460 2024-04-23T14:52:53.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.443+0000 7fa83e000640 1 -- 172.21.15.28:0/1619515466 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa848011d80 con 0x5574a7b7a460 2024-04-23T14:52:53.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.443+0000 7fa83e000640 1 -- 172.21.15.28:0/1619515466 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fa838003a60 con 0x5574a7b7a460 2024-04-23T14:52:53.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.443+0000 7fa859e76740 10 client.26593 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:52:53.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.443+0000 7fa859e76740 10 client.26593 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:52:53.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.443+0000 7fa859e76740 20 client.26593 populate_metadata read hostname 'smithi028' 2024-04-23T14:52:53.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.443+0000 7fa859e76740 1 --2- 172.21.15.28:0/1619515466 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5574a7b94d60 0x5574a7b97170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:53.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.444+0000 7fa859e76740 4 client.26593 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:52:53.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.444+0000 7fa859e76740 1 -- 172.21.15.28:0/1619515466 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5574a7aecbf0 con 0x5574a7b94d60 2024-04-23T14:52:53.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.444+0000 7fa855000640 1 --2- 172.21.15.28:0/1619515466 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5574a7b94d60 0x5574a7b97170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:53.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.444+0000 7fa855000640 1 --2- 172.21.15.28:0/1619515466 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5574a7b94d60 0x5574a7b97170 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37c0d1ee492a76ee server_cookie=6382d196351205ad in_seq=0 out_seq=0 2024-04-23T14:52:53.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.444+0000 7fa83e000640 10 client.26593 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:53.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.446+0000 7fa83e000640 1 -- 172.21.15.28:0/1619515466 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5574a7aecbf0 con 0x5574a7b94d60 2024-04-23T14:52:53.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.446+0000 7fa83e000640 10 client.26593 handle_command_reply: tid=0 2024-04-23T14:52:53.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.446+0000 7fa859e76740 1 client.26593 shutdown 2024-04-23T14:52:53.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.446+0000 7fa859e76740 1 -- 172.21.15.28:0/1619515466 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574a7b7a460 msgr2=0x5574a7b818d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:53.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.446+0000 7fa859e76740 1 --2- 172.21.15.28:0/1619515466 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574a7b7a460 0x5574a7b818d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa83802aa70 tx=0x7fa838004390 comp rx=0 tx=0).stop 2024-04-23T14:52:53.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.446+0000 7fa859e76740 1 -- 172.21.15.28:0/1619515466 shutdown_connections 2024-04-23T14:52:53.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.446+0000 7fa859e76740 1 --2- 172.21.15.28:0/1619515466 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5574a7b94d60 0x5574a7b97170 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.446+0000 7fa859e76740 1 --2- 172.21.15.28:0/1619515466 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574a7b84b30 0x5574a7b82010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.446+0000 7fa859e76740 1 --2- 172.21.15.28:0/1619515466 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574a7b7a460 0x5574a7b818d0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.446+0000 7fa859e76740 1 --2- 172.21.15.28:0/1619515466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574a7b79b40 0x5574a7b845e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.446+0000 7fa83e000640 0 client.26593 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:53.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.446+0000 7fa859e76740 1 -- 172.21.15.28:0/1619515466 >> 172.21.15.28:0/1619515466 conn(0x5574a7b75e60 msgr2=0x5574a7b7bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:53.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.446+0000 7fa859e76740 1 -- 172.21.15.28:0/1619515466 shutdown_connections 2024-04-23T14:52:53.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.446+0000 7fa859e76740 1 -- 172.21.15.28:0/1619515466 wait complete. 2024-04-23T14:52:53.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.447+0000 7fa859e76740 20 client.26593 trim_cache size 0 max 16384 2024-04-23T14:52:53.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.463+0000 7fa859e76740 1 Processor -- start 2024-04-23T14:52:53.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.463+0000 7fa859e76740 1 -- start start 2024-04-23T14:52:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.463+0000 7fa859e76740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574a7b79b40 0x5574a7b90880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.463+0000 7fa859e76740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574a7b7a460 0x5574a7b84110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.463+0000 7fa859e76740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574a7b84b30 0x5574a7b9ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.463+0000 7fa859e76740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa8501ade20 con 0x5574a7b7a460 2024-04-23T14:52:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.463+0000 7fa859e76740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa8501b0680 con 0x5574a7b79b40 2024-04-23T14:52:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.463+0000 7fa859e76740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa8501ae470 con 0x5574a7b84b30 2024-04-23T14:52:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.464+0000 7fa855000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574a7b79b40 0x5574a7b90880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.464+0000 7fa855000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574a7b79b40 0x5574a7b90880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60250/0 (socket says 172.21.15.28:60250) 2024-04-23T14:52:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.464+0000 7fa855000640 1 -- 172.21.15.28:0/4194330743 learned_addr learned my addr 172.21.15.28:0/4194330743 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.464+0000 7fa847400640 1 --2- 172.21.15.28:0/4194330743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574a7b7a460 0x5574a7b84110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.464+0000 7fa855000640 1 --2- 172.21.15.28:0/4194330743 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574a7b84b30 0x5574a7b9ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.464+0000 7fa847400640 1 -- 172.21.15.28:0/4194330743 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574a7b84b30 msgr2=0x5574a7b9ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:53.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.464+0000 7fa847400640 1 --2- 172.21.15.28:0/4194330743 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574a7b84b30 0x5574a7b9ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.464+0000 7fa847400640 1 -- 172.21.15.28:0/4194330743 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574a7b79b40 msgr2=0x5574a7b90880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:53.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.464+0000 7fa847400640 1 --2- 172.21.15.28:0/4194330743 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574a7b79b40 0x5574a7b90880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.464+0000 7fa847400640 1 -- 172.21.15.28:0/4194330743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574a78bb7d0 con 0x5574a7b7a460 2024-04-23T14:52:53.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.465+0000 7fa847400640 1 --2- 172.21.15.28:0/4194330743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574a7b7a460 0x5574a7b84110 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa838000c00 tx=0x7fa83802a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82bd13188b6f603b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:53.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.465+0000 7fa844c00640 1 -- 172.21.15.28:0/4194330743 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa838035030 con 0x5574a7b7a460 2024-04-23T14:52:53.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.465+0000 7fa844c00640 1 -- 172.21.15.28:0/4194330743 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa838041070 con 0x5574a7b7a460 2024-04-23T14:52:53.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.465+0000 7fa844c00640 1 -- 172.21.15.28:0/4194330743 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa838053070 con 0x5574a7b7a460 2024-04-23T14:52:53.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.466+0000 7fa859e76740 1 -- 172.21.15.28:0/4194330743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574a7b7a460 msgr2=0x5574a7b84110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:53.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.466+0000 7fa859e76740 1 --2- 172.21.15.28:0/4194330743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574a7b7a460 0x5574a7b84110 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa838000c00 tx=0x7fa83802a480 comp rx=0 tx=0).stop 2024-04-23T14:52:53.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.466+0000 7fa859e76740 1 -- 172.21.15.28:0/4194330743 shutdown_connections 2024-04-23T14:52:53.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.466+0000 7fa859e76740 1 --2- 172.21.15.28:0/4194330743 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574a7b84b30 0x5574a7b9ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.466+0000 7fa859e76740 1 --2- 172.21.15.28:0/4194330743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574a7b7a460 0x5574a7b84110 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.466+0000 7fa859e76740 1 --2- 172.21.15.28:0/4194330743 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574a7b79b40 0x5574a7b90880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.466+0000 7fa859e76740 1 -- 172.21.15.28:0/4194330743 >> 172.21.15.28:0/4194330743 conn(0x5574a7b75e60 msgr2=0x5574a7b7cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:53.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.467+0000 7fa859e76740 1 -- 172.21.15.28:0/4194330743 shutdown_connections 2024-04-23T14:52:53.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.467+0000 7fa859e76740 1 -- 172.21.15.28:0/4194330743 wait complete. 2024-04-23T14:52:53.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.467+0000 7fa859e76740 1 -- start start 2024-04-23T14:52:53.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.467+0000 7fa859e76740 1 Processor -- start 2024-04-23T14:52:53.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.468+0000 7fa859e76740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574a7b79b40 0x5574a7b8d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.468+0000 7fa859e76740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574a7b7a460 0x5574a7b7ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.468+0000 7fa859e76740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574a7b84b30 0x5574a7b7f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.468+0000 7fa859e76740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa8501aeb40 con 0x5574a7b7a460 2024-04-23T14:52:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.468+0000 7fa859e76740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa8501affb0 con 0x5574a7b79b40 2024-04-23T14:52:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.468+0000 7fa859e76740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa8501af210 con 0x5574a7b84b30 2024-04-23T14:52:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.468+0000 7fa859e76740 10 client.0 _ll_register_callbacks cb 0x7fa838009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:52:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.468+0000 7fa855000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574a7b79b40 0x5574a7b8d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.468+0000 7fa855000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574a7b79b40 0x5574a7b8d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60258/0 (socket says 172.21.15.28:60258) 2024-04-23T14:52:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.468+0000 7fa855000640 1 -- 172.21.15.28:0/2936389581 learned_addr learned my addr 172.21.15.28:0/2936389581 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.468+0000 7fa847400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574a7b7a460 0x5574a7b7ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.468+0000 7fa855000640 1 --2- 172.21.15.28:0/2936389581 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574a7b84b30 0x5574a7b7f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.469+0000 7fa847400640 1 -- 172.21.15.28:0/2936389581 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574a7b84b30 msgr2=0x5574a7b7f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.469+0000 7fa847400640 1 --2- 172.21.15.28:0/2936389581 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574a7b84b30 0x5574a7b7f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.469+0000 7fa847400640 1 -- 172.21.15.28:0/2936389581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574a7b79b40 msgr2=0x5574a7b8d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:53.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.469+0000 7fa847400640 1 --2- 172.21.15.28:0/2936389581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574a7b79b40 0x5574a7b8d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.469+0000 7fa847400640 1 -- 172.21.15.28:0/2936389581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574a7b858e0 con 0x5574a7b7a460 2024-04-23T14:52:53.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.469+0000 7fa855000640 1 --2- 172.21.15.28:0/2936389581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574a7b79b40 0x5574a7b8d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:52:53.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.469+0000 7fa847400640 1 --2- 172.21.15.28:0/2936389581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574a7b7a460 0x5574a7b7ed10 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa838003eb0 tx=0x7fa838031460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=434e7dd39f14e546 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:53.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.469+0000 7fa83e000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-23T14:52:53.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.469+0000 7fa83e000640 1 -- 172.21.15.28:0/2936389581 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa838035070 con 0x5574a7b7a460 2024-04-23T14:52:53.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.469+0000 7fa83e000640 1 -- 172.21.15.28:0/2936389581 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7fa838003a00 con 0x5574a7b7a460 2024-04-23T14:52:53.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.469+0000 7fa859e76740 1 -- 172.21.15.28:0/2936389581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa838041070 con 0x5574a7b7a460 2024-04-23T14:52:53.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.470+0000 7fa83e000640 1 -- 172.21.15.28:0/2936389581 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fa838066050 con 0x5574a7b7a460 2024-04-23T14:52:53.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.470+0000 7fa859e76740 10 client.16449 fetch_fsmap learned FSMap version 20 2024-04-23T14:52:53.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.470+0000 7fa859e76740 1 -- 172.21.15.28:0/2936389581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5574a7bfb5d0 con 0x5574a7b7a460 2024-04-23T14:52:53.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.470+0000 7fa83e000640 1 -- 172.21.15.28:0/2936389581 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa838041070 con 0x5574a7b7a460 2024-04-23T14:52:53.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.470+0000 7fa83e000640 1 -- 172.21.15.28:0/2936389581 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fa838004ba0 con 0x5574a7b7a460 2024-04-23T14:52:53.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.470+0000 7fa859e76740 10 client.16449 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:52:53.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.470+0000 7fa859e76740 10 client.16449 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:52:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.470+0000 7fa859e76740 20 client.16449 populate_metadata read hostname 'smithi028' 2024-04-23T14:52:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.470+0000 7fa859e76740 1 --2- 172.21.15.28:0/2936389581 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5574a7bfbca0 0x5574a7bfe0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.470+0000 7fa859e76740 4 client.16449 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-23T14:52:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.470+0000 7fa859e76740 1 -- 172.21.15.28:0/2936389581 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5574a7ae6a60 con 0x5574a7bfbca0 2024-04-23T14:52:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.470+0000 7fa855000640 1 --2- 172.21.15.28:0/2936389581 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5574a7bfbca0 0x5574a7bfe0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.471+0000 7fa855000640 1 --2- 172.21.15.28:0/2936389581 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5574a7bfbca0 0x5574a7bfe0c0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d0861c72e75b99a server_cookie=8dea88db09d5fd6b in_seq=0 out_seq=0 2024-04-23T14:52:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.471+0000 7fa83e000640 10 client.16449 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.471+0000 7fa83e000640 1 -- 172.21.15.28:0/2936389581 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5574a7ae6a60 con 0x5574a7bfbca0 2024-04-23T14:52:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.471+0000 7fa83e000640 10 client.16449 handle_command_reply: tid=0 2024-04-23T14:52:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.471+0000 7fa859e76740 1 client.16449 shutdown 2024-04-23T14:52:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.471+0000 7fa859e76740 1 -- 172.21.15.28:0/2936389581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574a7b7a460 msgr2=0x5574a7b7ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.471+0000 7fa859e76740 1 --2- 172.21.15.28:0/2936389581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574a7b7a460 0x5574a7b7ed10 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa838003eb0 tx=0x7fa838031460 comp rx=0 tx=0).stop 2024-04-23T14:52:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.471+0000 7fa859e76740 1 -- 172.21.15.28:0/2936389581 shutdown_connections 2024-04-23T14:52:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.471+0000 7fa859e76740 1 --2- 172.21.15.28:0/2936389581 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5574a7bfbca0 0x5574a7bfe0c0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.471+0000 7fa859e76740 1 --2- 172.21.15.28:0/2936389581 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5574a7b84b30 0x5574a7b7f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.471+0000 7fa859e76740 1 --2- 172.21.15.28:0/2936389581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574a7b7a460 0x5574a7b7ed10 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.471+0000 7fa859e76740 1 --2- 172.21.15.28:0/2936389581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574a7b79b40 0x5574a7b8d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.471+0000 7fa83e000640 0 client.16449 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:53.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.471+0000 7fa859e76740 1 -- 172.21.15.28:0/2936389581 >> 172.21.15.28:0/2936389581 conn(0x5574a7b75e60 msgr2=0x5574a7b746c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:53.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.472+0000 7fa859e76740 1 -- 172.21.15.28:0/2936389581 shutdown_connections 2024-04-23T14:52:53.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.472+0000 7fa859e76740 1 -- 172.21.15.28:0/2936389581 wait complete. 2024-04-23T14:52:53.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.472+0000 7fa859e76740 20 client.16449 trim_cache size 0 max 16384 2024-04-23T14:52:53.474 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-23T14:52:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.473+0000 7fa855a00640 1 -- 172.21.15.28:0/669387841 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa8340a3da0 msgr2=0x7fa8340a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.473+0000 7fa855a00640 1 --2- 172.21.15.28:0/669387841 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa8340a3da0 0x7fa8340a6260 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x5574a7b7b100 tx=0x7fa840004a40 comp rx=0 tx=0).stop 2024-04-23T14:52:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.473+0000 7fa855a00640 1 -- 172.21.15.28:0/669387841 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa850103060 msgr2=0x7fa8501a0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.473+0000 7fa855a00640 1 --2- 172.21.15.28:0/669387841 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa850103060 0x7fa8501a0690 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fa84000e9f0 tx=0x7fa84000edf0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.473+0000 7fa855a00640 1 -- 172.21.15.28:0/669387841 shutdown_connections 2024-04-23T14:52:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.473+0000 7fa855a00640 1 --2- 172.21.15.28:0/669387841 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa8340a3da0 0x7fa8340a6260 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.473+0000 7fa855a00640 1 --2- 172.21.15.28:0/669387841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa85010c480 0x7fa85019a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.473+0000 7fa855a00640 1 --2- 172.21.15.28:0/669387841 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa850103980 0x7fa8501a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.473+0000 7fa855a00640 1 --2- 172.21.15.28:0/669387841 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa850103060 0x7fa8501a0690 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.473+0000 7fa855a00640 1 -- 172.21.15.28:0/669387841 >> 172.21.15.28:0/669387841 conn(0x7fa8500fc880 msgr2=0x7fa850100c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.473+0000 7fa855a00640 1 -- 172.21.15.28:0/669387841 shutdown_connections 2024-04-23T14:52:53.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:53.473+0000 7fa855a00640 1 -- 172.21.15.28:0/669387841 wait complete. 2024-04-23T14:52:53.515 INFO:tasks.workunit.client.0.smithi028.stdout: 1 744725 8.95 MB/sec execute 309 sec latency 17.915 ms 2024-04-23T14:52:54.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:53 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1238719606' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:52:54.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:53 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1238719606' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:52:54.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:53 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1238719606' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:52:54.493 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-23T14:52:54.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:52:54.493 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-23T14:52:54.515 INFO:tasks.workunit.client.0.smithi028.stdout: 1 746210 8.95 MB/sec execute 310 sec latency 13.474 ms 2024-04-23T14:52:54.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.743+0000 7f7173000640 1 -- 172.21.15.28:0/3790751584 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f716c104e60 msgr2=0x7f716c108ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:54.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.743+0000 7f7173000640 1 --2- 172.21.15.28:0/3790751584 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f716c104e60 0x7f716c108ea0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f7158009900 tx=0x7f715802a3f0 comp rx=0 tx=0).stop 2024-04-23T14:52:54.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.743+0000 7f7173000640 1 -- 172.21.15.28:0/3790751584 shutdown_connections 2024-04-23T14:52:54.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.743+0000 7f7173000640 1 --2- 172.21.15.28:0/3790751584 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f716c104e60 0x7f716c108ea0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:54.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.743+0000 7f7173000640 1 --2- 172.21.15.28:0/3790751584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f716c109580 0x7f716c111150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:54.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.743+0000 7f7173000640 1 --2- 172.21.15.28:0/3790751584 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f716c1044b0 0x7f716c104890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:54.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.743+0000 7f7173000640 1 -- 172.21.15.28:0/3790751584 >> 172.21.15.28:0/3790751584 conn(0x7f716c0ffd90 msgr2=0x7f716c1021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:54.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.743+0000 7f7173000640 1 -- 172.21.15.28:0/3790751584 shutdown_connections 2024-04-23T14:52:54.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.743+0000 7f7173000640 1 -- 172.21.15.28:0/3790751584 wait complete. 2024-04-23T14:52:54.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.744+0000 7f7173000640 1 Processor -- start 2024-04-23T14:52:54.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.744+0000 7f7173000640 1 -- start start 2024-04-23T14:52:54.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.745+0000 7f7173000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f716c1044b0 0x7f716c076510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:54.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.745+0000 7f7173000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f716c109580 0x7f716c076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:54.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.745+0000 7f7173000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f716c076f90 0x7f716c1ab1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:54.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.745+0000 7f7173000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f716c114660 con 0x7f716c1044b0 2024-04-23T14:52:54.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.745+0000 7f7173000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f716c1144e0 con 0x7f716c109580 2024-04-23T14:52:54.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.745+0000 7f7173000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f716c1147e0 con 0x7f716c076f90 2024-04-23T14:52:54.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.745+0000 7f716b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f716c109580 0x7f716c076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:54.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.745+0000 7f716be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f716c1044b0 0x7f716c076510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:54.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.745+0000 7f716b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f716c109580 0x7f716c076a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60276/0 (socket says 172.21.15.28:60276) 2024-04-23T14:52:54.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.745+0000 7f716b400640 1 -- 172.21.15.28:0/224246808 learned_addr learned my addr 172.21.15.28:0/224246808 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:54.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.745+0000 7f7171000640 1 --2- 172.21.15.28:0/224246808 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f716c076f90 0x7f716c1ab1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:54.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.746+0000 7f716b400640 1 -- 172.21.15.28:0/224246808 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f716c076f90 msgr2=0x7f716c1ab1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:54.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.746+0000 7f716b400640 1 --2- 172.21.15.28:0/224246808 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f716c076f90 0x7f716c1ab1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:54.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.746+0000 7f716b400640 1 -- 172.21.15.28:0/224246808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f716c1044b0 msgr2=0x7f716c076510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:54.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.746+0000 7f716b400640 1 --2- 172.21.15.28:0/224246808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f716c1044b0 0x7f716c076510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:54.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.746+0000 7f716b400640 1 -- 172.21.15.28:0/224246808 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f716c1ab710 con 0x7f716c109580 2024-04-23T14:52:54.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.746+0000 7f716be00640 1 --2- 172.21.15.28:0/224246808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f716c1044b0 0x7f716c076510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:52:54.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.746+0000 7f716b400640 1 --2- 172.21.15.28:0/224246808 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f716c109580 0x7f716c076a50 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f715802a910 tx=0x7f7158003b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:54.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.747+0000 7f7168c00640 1 -- 172.21.15.28:0/224246808 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715803f070 con 0x7f716c109580 2024-04-23T14:52:54.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.747+0000 7f7168c00640 1 -- 172.21.15.28:0/224246808 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7158031c90 con 0x7f716c109580 2024-04-23T14:52:54.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.747+0000 7f7173000640 1 -- 172.21.15.28:0/224246808 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f716c1ab940 con 0x7f716c109580 2024-04-23T14:52:54.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.747+0000 7f7168c00640 1 -- 172.21.15.28:0/224246808 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7158035030 con 0x7f716c109580 2024-04-23T14:52:54.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.747+0000 7f7173000640 1 -- 172.21.15.28:0/224246808 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f716c1abe50 con 0x7f716c109580 2024-04-23T14:52:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.752+0000 7f7175dd8740 1 Processor -- start 2024-04-23T14:52:54.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.752+0000 7f7168c00640 1 -- 172.21.15.28:0/224246808 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f7158042070 con 0x7f716c109580 2024-04-23T14:52:54.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.752+0000 7f7175dd8740 1 -- start start 2024-04-23T14:52:54.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.752+0000 7f7175dd8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdd543b090 0x55fdd543b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:54.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.752+0000 7f7175dd8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fdd543b9b0 0x55fdd5445b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:54.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.752+0000 7f7175dd8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdd5446080 0x55fdd5448470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:54.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.752+0000 7f7175dd8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fdd53a0e70 con 0x55fdd5446080 2024-04-23T14:52:54.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.752+0000 7f7175dd8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55fdd5347380 con 0x55fdd543b9b0 2024-04-23T14:52:54.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.752+0000 7f7175dd8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55fdd539ce30 con 0x55fdd543b090 2024-04-23T14:52:54.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.753+0000 7f7171000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdd543b090 0x55fdd543b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:54.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.753+0000 7f7171000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdd543b090 0x55fdd543b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:45994/0 (socket says 172.21.15.28:45994) 2024-04-23T14:52:54.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.753+0000 7f7171000640 1 -- 172.21.15.28:0/3067990893 learned_addr learned my addr 172.21.15.28:0/3067990893 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:54.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.753+0000 7f716be00640 1 --2- 172.21.15.28:0/3067990893 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fdd543b9b0 0x55fdd5445b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:54.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.753+0000 7f716b400640 1 --2- 172.21.15.28:0/3067990893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdd5446080 0x55fdd5448470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:54.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.754+0000 7f716be00640 1 -- 172.21.15.28:0/3067990893 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdd543b090 msgr2=0x55fdd543b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:54.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.754+0000 7f716be00640 1 --2- 172.21.15.28:0/3067990893 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdd543b090 0x55fdd543b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:54.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.754+0000 7f716be00640 1 -- 172.21.15.28:0/3067990893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdd5446080 msgr2=0x55fdd5448470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:54.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.754+0000 7f716be00640 1 --2- 172.21.15.28:0/3067990893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdd5446080 0x55fdd5448470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:54.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.754+0000 7f716be00640 1 -- 172.21.15.28:0/3067990893 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdd517b7e0 con 0x55fdd543b9b0 2024-04-23T14:52:54.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.754+0000 7f7171000640 1 --2- 172.21.15.28:0/3067990893 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdd543b090 0x55fdd543b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:52:54.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.754+0000 7f716be00640 1 --2- 172.21.15.28:0/3067990893 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fdd543b9b0 0x55fdd5445b40 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f715c009fd0 tx=0x7f715c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1954ebf04748a0a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:54.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.755+0000 7f7161600640 1 -- 172.21.15.28:0/3067990893 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715c004310 con 0x55fdd543b9b0 2024-04-23T14:52:54.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.755+0000 7f7161600640 1 -- 172.21.15.28:0/3067990893 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f715c0044b0 con 0x55fdd543b9b0 2024-04-23T14:52:54.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.755+0000 7f7161600640 1 -- 172.21.15.28:0/3067990893 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715c0035f0 con 0x55fdd543b9b0 2024-04-23T14:52:54.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.755+0000 7f7168c00640 1 --2- 172.21.15.28:0/224246808 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f71500a3da0 0x7f71500a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:54.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.756+0000 7f7171000640 1 --2- 172.21.15.28:0/224246808 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f71500a3da0 0x7f71500a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:54.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.756+0000 7f7175dd8740 1 -- 172.21.15.28:0/3067990893 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fdd543b9b0 msgr2=0x55fdd5445b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:54.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.756+0000 7f7175dd8740 1 --2- 172.21.15.28:0/3067990893 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fdd543b9b0 0x55fdd5445b40 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f715c009fd0 tx=0x7f715c0118d0 comp rx=0 tx=0).stop 2024-04-23T14:52:54.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.757+0000 7f7168c00640 1 -- 172.21.15.28:0/224246808 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7f71580f0db0 con 0x7f716c109580 2024-04-23T14:52:54.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.757+0000 7f7175dd8740 1 -- 172.21.15.28:0/3067990893 shutdown_connections 2024-04-23T14:52:54.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.757+0000 7f7175dd8740 1 --2- 172.21.15.28:0/3067990893 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fdd543b9b0 0x55fdd5445b40 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:54.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.757+0000 7f7175dd8740 1 --2- 172.21.15.28:0/3067990893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdd5446080 0x55fdd5448470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:54.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.757+0000 7f7175dd8740 1 --2- 172.21.15.28:0/3067990893 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdd543b090 0x55fdd543b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:54.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.757+0000 7f7175dd8740 1 -- 172.21.15.28:0/3067990893 >> 172.21.15.28:0/3067990893 conn(0x55fdd54373b0 msgr2=0x55fdd5437790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:54.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.757+0000 7f7171000640 1 --2- 172.21.15.28:0/224246808 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f71500a3da0 0x7f71500a6260 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7164009fd0 tx=0x7f7164009500 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:54.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.757+0000 7f7175dd8740 1 -- 172.21.15.28:0/3067990893 shutdown_connections 2024-04-23T14:52:54.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.758+0000 7f7175dd8740 1 -- 172.21.15.28:0/3067990893 wait complete. 2024-04-23T14:52:54.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.758+0000 7f7175dd8740 1 -- start start 2024-04-23T14:52:54.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.758+0000 7f7175dd8740 1 Processor -- start 2024-04-23T14:52:54.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.759+0000 7f7175dd8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fdd543b090 0x55fdd5442ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.759+0000 7f7175dd8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdd543b9b0 0x55fdd5443530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.759+0000 7f7175dd8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdd5446080 0x55fdd5443a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.759+0000 7f7175dd8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fdd5448e70 con 0x55fdd5446080 2024-04-23T14:52:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.759+0000 7f7175dd8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55fdd535a660 con 0x55fdd543b090 2024-04-23T14:52:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.759+0000 7f7175dd8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55fdd5448ff0 con 0x55fdd543b9b0 2024-04-23T14:52:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.759+0000 7f7175dd8740 10 client.0 _ll_register_callbacks cb 0x55fdd5348aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:52:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.759+0000 7f716be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fdd543b090 0x55fdd5442ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.759+0000 7f716be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fdd543b090 0x55fdd5442ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60300/0 (socket says 172.21.15.28:60300) 2024-04-23T14:52:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.759+0000 7f716be00640 1 -- 172.21.15.28:0/3571395832 learned_addr learned my addr 172.21.15.28:0/3571395832 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.759+0000 7f716b400640 1 --2- 172.21.15.28:0/3571395832 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdd543b9b0 0x55fdd5443530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.760+0000 7f716be00640 1 -- 172.21.15.28:0/3571395832 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdd543b9b0 msgr2=0x55fdd5443530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.760+0000 7f716be00640 1 --2- 172.21.15.28:0/3571395832 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdd543b9b0 0x55fdd5443530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.760+0000 7f716be00640 1 -- 172.21.15.28:0/3571395832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdd5446080 msgr2=0x55fdd5443a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.760+0000 7f716be00640 1 --2- 172.21.15.28:0/3571395832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdd5446080 0x55fdd5443a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.760+0000 7f716be00640 1 -- 172.21.15.28:0/3571395832 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdd51dbf70 con 0x55fdd543b090 2024-04-23T14:52:54.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.760+0000 7f716be00640 1 --2- 172.21.15.28:0/3571395832 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fdd543b090 0x55fdd5442ff0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f715c011dd0 tx=0x7f715c009d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4757d2be903b74c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.760+0000 7f7156000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:52:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.761+0000 7f7156000640 1 -- 172.21.15.28:0/3571395832 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715c029070 con 0x55fdd543b090 2024-04-23T14:52:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.761+0000 7f7156000640 1 -- 172.21.15.28:0/3571395832 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7f715c003a00 con 0x55fdd543b090 2024-04-23T14:52:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.761+0000 7f7175dd8740 1 -- 172.21.15.28:0/3571395832 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f715c0044b0 con 0x55fdd543b090 2024-04-23T14:52:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.761+0000 7f7156000640 1 -- 172.21.15.28:0/3571395832 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f715c035050 con 0x55fdd543b090 2024-04-23T14:52:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.761+0000 7f7175dd8740 10 client.26633 fetch_fsmap learned FSMap version 20 2024-04-23T14:52:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.761+0000 7f7175dd8740 1 -- 172.21.15.28:0/3571395832 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55fdd5073170 con 0x55fdd543b090 2024-04-23T14:52:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.762+0000 7f7156000640 1 -- 172.21.15.28:0/3571395832 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f715c0044b0 con 0x55fdd543b090 2024-04-23T14:52:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.762+0000 7f7156000640 1 -- 172.21.15.28:0/3571395832 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f715c00f4e0 con 0x55fdd543b090 2024-04-23T14:52:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.762+0000 7f7175dd8740 10 client.26633 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:52:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.762+0000 7f7175dd8740 10 client.26633 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:52:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.762+0000 7f7175dd8740 20 client.26633 populate_metadata read hostname 'smithi028' 2024-04-23T14:52:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.762+0000 7f7175dd8740 1 --2- 172.21.15.28:0/3571395832 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55fdd54562d0 0x55fdd54586c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.762+0000 7f7175dd8740 4 client.26633 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:52:54.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.762+0000 7f7175dd8740 1 -- 172.21.15.28:0/3571395832 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fdd53ae890 con 0x55fdd54562d0 2024-04-23T14:52:54.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.762+0000 7f716b400640 1 --2- 172.21.15.28:0/3571395832 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55fdd54562d0 0x55fdd54586c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:54.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.763+0000 7f716b400640 1 --2- 172.21.15.28:0/3571395832 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55fdd54562d0 0x55fdd54586c0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0dc9aa0a2e82a36 server_cookie=920c8e3124668e84 in_seq=0 out_seq=0 2024-04-23T14:52:54.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.763+0000 7f7156000640 10 client.26633 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.764+0000 7f7156000640 1 -- 172.21.15.28:0/3571395832 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fdd53ae890 con 0x55fdd54562d0 2024-04-23T14:52:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.764+0000 7f7156000640 10 client.26633 handle_command_reply: tid=0 2024-04-23T14:52:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.764+0000 7f7175dd8740 1 client.26633 shutdown 2024-04-23T14:52:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.764+0000 7f7175dd8740 1 -- 172.21.15.28:0/3571395832 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fdd543b090 msgr2=0x55fdd5442ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.764+0000 7f7175dd8740 1 --2- 172.21.15.28:0/3571395832 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fdd543b090 0x55fdd5442ff0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f715c011dd0 tx=0x7f715c009d20 comp rx=0 tx=0).stop 2024-04-23T14:52:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.764+0000 7f7175dd8740 1 -- 172.21.15.28:0/3571395832 shutdown_connections 2024-04-23T14:52:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.764+0000 7f7175dd8740 1 --2- 172.21.15.28:0/3571395832 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55fdd54562d0 0x55fdd54586c0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.764+0000 7f7175dd8740 1 --2- 172.21.15.28:0/3571395832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdd5446080 0x55fdd5443a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.764+0000 7f7175dd8740 1 --2- 172.21.15.28:0/3571395832 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdd543b9b0 0x55fdd5443530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.764+0000 7f7175dd8740 1 --2- 172.21.15.28:0/3571395832 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fdd543b090 0x55fdd5442ff0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.764+0000 7f7156000640 0 client.26633 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.764+0000 7f7175dd8740 1 -- 172.21.15.28:0/3571395832 >> 172.21.15.28:0/3571395832 conn(0x55fdd54373b0 msgr2=0x55fdd543d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:54.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.765+0000 7f7175dd8740 1 -- 172.21.15.28:0/3571395832 shutdown_connections 2024-04-23T14:52:54.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.765+0000 7f7175dd8740 1 -- 172.21.15.28:0/3571395832 wait complete. 2024-04-23T14:52:54.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.765+0000 7f7175dd8740 20 client.26633 trim_cache size 0 max 16384 2024-04-23T14:52:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.786+0000 7f7175dd8740 1 Processor -- start 2024-04-23T14:52:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.786+0000 7f7175dd8740 1 -- start start 2024-04-23T14:52:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.786+0000 7f7175dd8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdd543b090 0x55fdd5451d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.786+0000 7f7175dd8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdd543b9b0 0x55fdd543d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.786+0000 7f7175dd8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fdd5446080 0x55fdd5436390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.786+0000 7f7175dd8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f716c1b23a0 con 0x55fdd543b9b0 2024-04-23T14:52:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.786+0000 7f7175dd8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f716c1b4c00 con 0x55fdd5446080 2024-04-23T14:52:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.786+0000 7f7175dd8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f716c1b29f0 con 0x55fdd543b090 2024-04-23T14:52:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.786+0000 7f716be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdd543b090 0x55fdd5451d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.787+0000 7f716be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdd543b090 0x55fdd5451d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:46014/0 (socket says 172.21.15.28:46014) 2024-04-23T14:52:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.787+0000 7f716be00640 1 -- 172.21.15.28:0/2747740085 learned_addr learned my addr 172.21.15.28:0/2747740085 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.787+0000 7f716b400640 1 --2- 172.21.15.28:0/2747740085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdd543b9b0 0x55fdd543d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.787+0000 7f7171000640 1 --2- 172.21.15.28:0/2747740085 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fdd5446080 0x55fdd5436390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.787+0000 7f716be00640 1 -- 172.21.15.28:0/2747740085 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fdd5446080 msgr2=0x55fdd5436390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.787+0000 7f716be00640 1 --2- 172.21.15.28:0/2747740085 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fdd5446080 0x55fdd5436390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.787+0000 7f716be00640 1 -- 172.21.15.28:0/2747740085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdd543b9b0 msgr2=0x55fdd543d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.787+0000 7f716be00640 1 --2- 172.21.15.28:0/2747740085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdd543b9b0 0x55fdd543d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.787+0000 7f716be00640 1 -- 172.21.15.28:0/2747740085 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdd5458a30 con 0x55fdd543b090 2024-04-23T14:52:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.787+0000 7f7171000640 1 --2- 172.21.15.28:0/2747740085 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fdd5446080 0x55fdd5436390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:52:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.787+0000 7f716b400640 1 --2- 172.21.15.28:0/2747740085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdd543b9b0 0x55fdd543d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:52:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.787+0000 7f716be00640 1 --2- 172.21.15.28:0/2747740085 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdd543b090 0x55fdd5451d30 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f715c010010 tx=0x7f715c009e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55a64746d88b4dfa server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.787+0000 7f7160c00640 1 -- 172.21.15.28:0/2747740085 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715c032070 con 0x55fdd543b090 2024-04-23T14:52:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.787+0000 7f7160c00640 1 -- 172.21.15.28:0/2747740085 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f715c004510 con 0x55fdd543b090 2024-04-23T14:52:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.787+0000 7f7160c00640 1 -- 172.21.15.28:0/2747740085 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715c029070 con 0x55fdd543b090 2024-04-23T14:52:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.788+0000 7f7175dd8740 1 -- 172.21.15.28:0/2747740085 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdd543b090 msgr2=0x55fdd5451d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.788+0000 7f7175dd8740 1 --2- 172.21.15.28:0/2747740085 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdd543b090 0x55fdd5451d30 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f715c010010 tx=0x7f715c009e20 comp rx=0 tx=0).stop 2024-04-23T14:52:54.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.788+0000 7f7175dd8740 1 -- 172.21.15.28:0/2747740085 shutdown_connections 2024-04-23T14:52:54.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.788+0000 7f7175dd8740 1 --2- 172.21.15.28:0/2747740085 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fdd5446080 0x55fdd5436390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:54.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.788+0000 7f7175dd8740 1 --2- 172.21.15.28:0/2747740085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdd543b9b0 0x55fdd543d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:54.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.788+0000 7f7175dd8740 1 --2- 172.21.15.28:0/2747740085 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdd543b090 0x55fdd5451d30 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:54.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.788+0000 7f7175dd8740 1 -- 172.21.15.28:0/2747740085 >> 172.21.15.28:0/2747740085 conn(0x55fdd54373b0 msgr2=0x55fdd5435c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:54.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.789+0000 7f7175dd8740 1 -- 172.21.15.28:0/2747740085 shutdown_connections 2024-04-23T14:52:54.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.789+0000 7f7175dd8740 1 -- 172.21.15.28:0/2747740085 wait complete. 2024-04-23T14:52:54.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.789+0000 7f7175dd8740 1 -- start start 2024-04-23T14:52:54.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.789+0000 7f7175dd8740 1 Processor -- start 2024-04-23T14:52:54.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.790+0000 7f7175dd8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdd543b090 0x55fdd544e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:54.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.790+0000 7f7175dd8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fdd543b9b0 0x55fdd543f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:54.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.790+0000 7f7175dd8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdd5446080 0x55fdd543fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:54.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.790+0000 7f7175dd8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f716c1b30c0 con 0x55fdd5446080 2024-04-23T14:52:54.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.790+0000 7f7175dd8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f716c1b4530 con 0x55fdd543b9b0 2024-04-23T14:52:54.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.790+0000 7f7175dd8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f716c1b3790 con 0x55fdd543b090 2024-04-23T14:52:54.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.790+0000 7f7175dd8740 10 client.0 _ll_register_callbacks cb 0x7f7164009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:52:54.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.790+0000 7f716be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdd543b090 0x55fdd544e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:54.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.790+0000 7f716be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdd543b090 0x55fdd544e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:46020/0 (socket says 172.21.15.28:46020) 2024-04-23T14:52:54.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.790+0000 7f716b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fdd543b9b0 0x55fdd543f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:54.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.790+0000 7f716be00640 1 -- 172.21.15.28:0/3604722598 learned_addr learned my addr 172.21.15.28:0/3604722598 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:54.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.790+0000 7f7171000640 1 --2- 172.21.15.28:0/3604722598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdd5446080 0x55fdd543fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:54.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.790+0000 7f716be00640 1 -- 172.21.15.28:0/3604722598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fdd543b9b0 msgr2=0x55fdd543f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:54.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.790+0000 7f716be00640 1 --2- 172.21.15.28:0/3604722598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fdd543b9b0 0x55fdd543f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:54.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.790+0000 7f716be00640 1 -- 172.21.15.28:0/3604722598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdd5446080 msgr2=0x55fdd543fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:54.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.790+0000 7f716be00640 1 --2- 172.21.15.28:0/3604722598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdd5446080 0x55fdd543fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:54.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.791+0000 7f716be00640 1 -- 172.21.15.28:0/3604722598 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdd5440130 con 0x55fdd543b090 2024-04-23T14:52:54.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.791+0000 7f7171000640 1 --2- 172.21.15.28:0/3604722598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdd5446080 0x55fdd543fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:52:54.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.791+0000 7f716be00640 1 --2- 172.21.15.28:0/3604722598 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdd543b090 0x55fdd544e430 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f715c006010 tx=0x7f715c009f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97ee4579d86feaa7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.791+0000 7f7156000640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:52:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.791+0000 7f7156000640 1 -- 172.21.15.28:0/3604722598 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715c029030 con 0x55fdd543b090 2024-04-23T14:52:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.791+0000 7f7156000640 1 -- 172.21.15.28:0/3604722598 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7f715c003c10 con 0x55fdd543b090 2024-04-23T14:52:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.791+0000 7f7175dd8740 1 -- 172.21.15.28:0/3604722598 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f715c004510 con 0x55fdd543b090 2024-04-23T14:52:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.791+0000 7f7156000640 1 -- 172.21.15.28:0/3604722598 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f715c032040 con 0x55fdd543b090 2024-04-23T14:52:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.791+0000 7f7175dd8740 10 client.26617 fetch_fsmap learned FSMap version 20 2024-04-23T14:52:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.791+0000 7f7175dd8740 1 -- 172.21.15.28:0/3604722598 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55fdd54bc3d0 con 0x55fdd543b090 2024-04-23T14:52:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.792+0000 7f7156000640 1 -- 172.21.15.28:0/3604722598 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f715c004510 con 0x55fdd543b090 2024-04-23T14:52:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.792+0000 7f7156000640 1 -- 172.21.15.28:0/3604722598 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f715c023430 con 0x55fdd543b090 2024-04-23T14:52:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.792+0000 7f7175dd8740 10 client.26617 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:52:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.792+0000 7f7175dd8740 10 client.26617 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:52:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.792+0000 7f7175dd8740 20 client.26617 populate_metadata read hostname 'smithi028' 2024-04-23T14:52:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.792+0000 7f7175dd8740 1 --2- 172.21.15.28:0/3604722598 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55fdd54bc9e0 0x55fdd54bee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.792+0000 7f7175dd8740 4 client.26617 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-23T14:52:54.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.792+0000 7f7175dd8740 1 -- 172.21.15.28:0/3604722598 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55fdd53a8700 con 0x55fdd54bc9e0 2024-04-23T14:52:54.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.792+0000 7f716b400640 1 --2- 172.21.15.28:0/3604722598 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55fdd54bc9e0 0x55fdd54bee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:54.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.793+0000 7f7156000640 10 client.26617 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:54.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.793+0000 7f716b400640 1 --2- 172.21.15.28:0/3604722598 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55fdd54bc9e0 0x55fdd54bee00 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c2556195ddba70c server_cookie=42124e52775fe412 in_seq=0 out_seq=0 2024-04-23T14:52:54.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.798+0000 7f7156000640 1 -- 172.21.15.28:0/3604722598 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55fdd53a8700 con 0x55fdd54bc9e0 2024-04-23T14:52:54.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.798+0000 7f7156000640 10 client.26617 handle_command_reply: tid=0 2024-04-23T14:52:54.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.798+0000 7f7175dd8740 1 client.26617 shutdown 2024-04-23T14:52:54.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.798+0000 7f7175dd8740 1 -- 172.21.15.28:0/3604722598 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdd543b090 msgr2=0x55fdd544e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:54.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.798+0000 7f7175dd8740 1 --2- 172.21.15.28:0/3604722598 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdd543b090 0x55fdd544e430 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f715c006010 tx=0x7f715c009f10 comp rx=0 tx=0).stop 2024-04-23T14:52:54.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.798+0000 7f7175dd8740 1 -- 172.21.15.28:0/3604722598 shutdown_connections 2024-04-23T14:52:54.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.798+0000 7f7175dd8740 1 --2- 172.21.15.28:0/3604722598 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55fdd54bc9e0 0x55fdd54bee00 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:54.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.798+0000 7f7175dd8740 1 --2- 172.21.15.28:0/3604722598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fdd543b9b0 0x55fdd543f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:54.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.798+0000 7f7175dd8740 1 --2- 172.21.15.28:0/3604722598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fdd5446080 0x55fdd543fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:54.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.798+0000 7f7175dd8740 1 --2- 172.21.15.28:0/3604722598 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fdd543b090 0x55fdd544e430 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:54.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.798+0000 7f7156000640 0 client.26617 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:54.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.798+0000 7f7175dd8740 1 -- 172.21.15.28:0/3604722598 >> 172.21.15.28:0/3604722598 conn(0x55fdd54373b0 msgr2=0x55fdd54437b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:54.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.798+0000 7f7175dd8740 1 -- 172.21.15.28:0/3604722598 shutdown_connections 2024-04-23T14:52:54.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.799+0000 7f7175dd8740 1 -- 172.21.15.28:0/3604722598 wait complete. 2024-04-23T14:52:54.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.799+0000 7f7175dd8740 20 client.26617 trim_cache size 0 max 16384 2024-04-23T14:52:54.801 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:52:54.801 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-23T14:52:54.801 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "d0fa219f-ce75-41fa-bb72-95e0edc295ea", 2024-04-23T14:52:54.801 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-23T14:52:54.801 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:52:54.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.799+0000 7f7173000640 1 -- 172.21.15.28:0/224246808 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f71500a3da0 msgr2=0x7f71500a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:54.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.799+0000 7f7173000640 1 --2- 172.21.15.28:0/224246808 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f71500a3da0 0x7f71500a6260 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7164009fd0 tx=0x7f7164009500 comp rx=0 tx=0).stop 2024-04-23T14:52:54.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.799+0000 7f7173000640 1 -- 172.21.15.28:0/224246808 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f716c109580 msgr2=0x7f716c076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:54.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.799+0000 7f7173000640 1 --2- 172.21.15.28:0/224246808 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f716c109580 0x7f716c076a50 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f715802a910 tx=0x7f7158003b20 comp rx=0 tx=0).stop 2024-04-23T14:52:54.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.800+0000 7f7173000640 1 -- 172.21.15.28:0/224246808 shutdown_connections 2024-04-23T14:52:54.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.800+0000 7f7173000640 1 --2- 172.21.15.28:0/224246808 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f71500a3da0 0x7f71500a6260 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:54.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.800+0000 7f7173000640 1 --2- 172.21.15.28:0/224246808 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f716c109580 0x7f716c076a50 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:54.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.800+0000 7f7173000640 1 --2- 172.21.15.28:0/224246808 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f716c076f90 0x7f716c1ab1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:54.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.800+0000 7f7173000640 1 --2- 172.21.15.28:0/224246808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f716c1044b0 0x7f716c076510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:54.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.800+0000 7f7173000640 1 -- 172.21.15.28:0/224246808 >> 172.21.15.28:0/224246808 conn(0x7f716c0ffd90 msgr2=0x7f716c101160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:54.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.800+0000 7f7173000640 1 -- 172.21.15.28:0/224246808 shutdown_connections 2024-04-23T14:52:54.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:54.800+0000 7f7173000640 1 -- 172.21.15.28:0/224246808 wait complete. 2024-04-23T14:52:54.819 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:52:54.819 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:52:54.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:54 smithi028 ceph-mon[27810]: pgmap v273: 97 pgs: 97 active+clean; 454 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.8 MiB/s wr, 202 op/s 2024-04-23T14:52:55.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:54 smithi105 ceph-mon[32555]: pgmap v273: 97 pgs: 97 active+clean; 454 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.8 MiB/s wr, 202 op/s 2024-04-23T14:52:55.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.071+0000 7f80c4c00640 1 -- 172.21.15.28:0/705032863 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80c0104450 msgr2=0x7f80c0104830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:55.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.071+0000 7f80c4c00640 1 --2- 172.21.15.28:0/705032863 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80c0104450 0x7f80c0104830 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f80b0009900 tx=0x7f80b002a410 comp rx=0 tx=0).stop 2024-04-23T14:52:55.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.071+0000 7f80c4c00640 1 -- 172.21.15.28:0/705032863 shutdown_connections 2024-04-23T14:52:55.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.071+0000 7f80c4c00640 1 --2- 172.21.15.28:0/705032863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80c0109520 0x7f80c01110f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.071+0000 7f80c4c00640 1 --2- 172.21.15.28:0/705032863 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f80c0104e00 0x7f80c0108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.071+0000 7f80c4c00640 1 --2- 172.21.15.28:0/705032863 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80c0104450 0x7f80c0104830 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.071+0000 7f80c4c00640 1 -- 172.21.15.28:0/705032863 >> 172.21.15.28:0/705032863 conn(0x7f80c00ffc70 msgr2=0x7f80c0102090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:55.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.071+0000 7f80c4c00640 1 -- 172.21.15.28:0/705032863 shutdown_connections 2024-04-23T14:52:55.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.072+0000 7f80c4c00640 1 -- 172.21.15.28:0/705032863 wait complete. 2024-04-23T14:52:55.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.073+0000 7f80c4c00640 1 Processor -- start 2024-04-23T14:52:55.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.073+0000 7f80c4c00640 1 -- start start 2024-04-23T14:52:55.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.073+0000 7f80c4c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80c0104450 0x7f80c01a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:55.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.073+0000 7f80c4c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f80c0109520 0x7f80c01a0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:55.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.073+0000 7f80c4c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80c019a960 0x7f80c019ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:55.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.073+0000 7f80c4c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f80c0114620 con 0x7f80c0104450 2024-04-23T14:52:55.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.073+0000 7f80c4c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f80c01144a0 con 0x7f80c019a960 2024-04-23T14:52:55.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.073+0000 7f80c4c00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f80c01147a0 con 0x7f80c0109520 2024-04-23T14:52:55.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.074+0000 7f80be800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80c019a960 0x7f80c019ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:55.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.074+0000 7f80be800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80c019a960 0x7f80c019ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60338/0 (socket says 172.21.15.28:60338) 2024-04-23T14:52:55.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.074+0000 7f80be800640 1 -- 172.21.15.28:0/630379177 learned_addr learned my addr 172.21.15.28:0/630379177 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:55.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.074+0000 7f80bde00640 1 --2- 172.21.15.28:0/630379177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80c0104450 0x7f80c01a07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:55.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.074+0000 7f80b7e00640 1 --2- 172.21.15.28:0/630379177 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f80c0109520 0x7f80c01a0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:55.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.074+0000 7f80be800640 1 -- 172.21.15.28:0/630379177 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f80c0109520 msgr2=0x7f80c01a0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:55.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.074+0000 7f80be800640 1 --2- 172.21.15.28:0/630379177 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f80c0109520 0x7f80c01a0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.074+0000 7f80be800640 1 -- 172.21.15.28:0/630379177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80c0104450 msgr2=0x7f80c01a07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:55.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.074+0000 7f80be800640 1 --2- 172.21.15.28:0/630379177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80c0104450 0x7f80c01a07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.074+0000 7f80be800640 1 -- 172.21.15.28:0/630379177 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80c019b480 con 0x7f80c019a960 2024-04-23T14:52:55.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.074+0000 7f80be800640 1 --2- 172.21.15.28:0/630379177 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80c019a960 0x7f80c019ae10 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f80b800eb10 tx=0x7f80b800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:55.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.074+0000 7f80b6a00640 1 -- 172.21.15.28:0/630379177 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b8004b20 con 0x7f80c019a960 2024-04-23T14:52:55.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.075+0000 7f80b6a00640 1 -- 172.21.15.28:0/630379177 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80b8004cc0 con 0x7f80c019a960 2024-04-23T14:52:55.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.075+0000 7f80c4c00640 1 -- 172.21.15.28:0/630379177 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80c019b770 con 0x7f80c019a960 2024-04-23T14:52:55.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.075+0000 7f80b6a00640 1 -- 172.21.15.28:0/630379177 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b8010760 con 0x7f80c019a960 2024-04-23T14:52:55.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.075+0000 7f80c4c00640 1 -- 172.21.15.28:0/630379177 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80c01a79c0 con 0x7f80c019a960 2024-04-23T14:52:55.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.078+0000 7f80b6a00640 1 -- 172.21.15.28:0/630379177 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f80b8005020 con 0x7f80c019a960 2024-04-23T14:52:55.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.078+0000 7f80c7a16740 1 Processor -- start 2024-04-23T14:52:55.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.079+0000 7f80c7a16740 1 -- start start 2024-04-23T14:52:55.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.079+0000 7f80c7a16740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fa5b1cf5b0 0x55fa5b1cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:55.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.079+0000 7f80b6a00640 1 --2- 172.21.15.28:0/630379177 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f80a40a3da0 0x7f80a40a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:55.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.079+0000 7f80c7a16740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fa5b1cff60 0x55fa5b1e2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:55.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.079+0000 7f80c7a16740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa5b1e3060 0x55fa5b1e5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:55.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.079+0000 7f80c7a16740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fa5b1351f0 con 0x55fa5b1e3060 2024-04-23T14:52:55.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.079+0000 7f80c7a16740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55fa5b0db350 con 0x55fa5b1cf5b0 2024-04-23T14:52:55.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.079+0000 7f80c7a16740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55fa5b1311b0 con 0x55fa5b1cff60 2024-04-23T14:52:55.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.079+0000 7f80b7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fa5b1cf5b0 0x55fa5b1cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:55.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.080+0000 7f80b7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fa5b1cf5b0 0x55fa5b1cf990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60354/0 (socket says 172.21.15.28:60354) 2024-04-23T14:52:55.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.080+0000 7f80b7e00640 1 -- 172.21.15.28:0/476546347 learned_addr learned my addr 172.21.15.28:0/476546347 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:55.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.080+0000 7f80b6a00640 1 -- 172.21.15.28:0/630379177 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7f80b80d0d20 con 0x7f80c019a960 2024-04-23T14:52:55.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.080+0000 7f80bde00640 1 --2- 172.21.15.28:0/630379177 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f80a40a3da0 0x7f80a40a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:55.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.080+0000 7f80b7e00640 1 --2- 172.21.15.28:0/476546347 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fa5b1cff60 0x55fa5b1e2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:55.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.080+0000 7f80be800640 1 --2- 172.21.15.28:0/476546347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa5b1e3060 0x55fa5b1e5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:55.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.080+0000 7f80b7e00640 1 -- 172.21.15.28:0/476546347 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fa5b1cff60 msgr2=0x55fa5b1e2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:55.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.080+0000 7f80b7e00640 1 --2- 172.21.15.28:0/476546347 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fa5b1cff60 0x55fa5b1e2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.080+0000 7f80b7e00640 1 -- 172.21.15.28:0/476546347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa5b1e3060 msgr2=0x55fa5b1e5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:55.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.080+0000 7f80b7e00640 1 --2- 172.21.15.28:0/476546347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa5b1e3060 0x55fa5b1e5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.080+0000 7f80b7e00640 1 -- 172.21.15.28:0/476546347 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa5b12ff50 con 0x55fa5b1cf5b0 2024-04-23T14:52:55.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.080+0000 7f80b7e00640 1 --2- 172.21.15.28:0/476546347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fa5b1cf5b0 0x55fa5b1cf990 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f80a8013780 tx=0x7f80a8005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4f30ef90eee755d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:55.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.081+0000 7f80bde00640 1 --2- 172.21.15.28:0/630379177 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f80a40a3da0 0x7f80a40a6260 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f80b0006010 tx=0x7f80b0000bc0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:55.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.081+0000 7f80af000640 1 -- 172.21.15.28:0/476546347 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a8013c40 con 0x55fa5b1cf5b0 2024-04-23T14:52:55.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.081+0000 7f80af000640 1 -- 172.21.15.28:0/476546347 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80a8013de0 con 0x55fa5b1cf5b0 2024-04-23T14:52:55.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.081+0000 7f80af000640 1 -- 172.21.15.28:0/476546347 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a8011a40 con 0x55fa5b1cf5b0 2024-04-23T14:52:55.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.081+0000 7f80c7a16740 1 -- 172.21.15.28:0/476546347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fa5b1cf5b0 msgr2=0x55fa5b1cf990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:55.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.081+0000 7f80c7a16740 1 --2- 172.21.15.28:0/476546347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fa5b1cf5b0 0x55fa5b1cf990 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f80a8013780 tx=0x7f80a8005850 comp rx=0 tx=0).stop 2024-04-23T14:52:55.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.082+0000 7f80c7a16740 1 -- 172.21.15.28:0/476546347 shutdown_connections 2024-04-23T14:52:55.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.082+0000 7f80c7a16740 1 --2- 172.21.15.28:0/476546347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa5b1e3060 0x55fa5b1e5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.082+0000 7f80c7a16740 1 --2- 172.21.15.28:0/476546347 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fa5b1cff60 0x55fa5b1e2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.082+0000 7f80c7a16740 1 --2- 172.21.15.28:0/476546347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fa5b1cf5b0 0x55fa5b1cf990 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.082+0000 7f80c7a16740 1 -- 172.21.15.28:0/476546347 >> 172.21.15.28:0/476546347 conn(0x55fa5b1cbb80 msgr2=0x55fa5b1cbf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:55.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.082+0000 7f80c7a16740 1 -- 172.21.15.28:0/476546347 shutdown_connections 2024-04-23T14:52:55.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.082+0000 7f80c7a16740 1 -- 172.21.15.28:0/476546347 wait complete. 2024-04-23T14:52:55.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.082+0000 7f80c7a16740 1 -- start start 2024-04-23T14:52:55.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.082+0000 7f80c7a16740 1 Processor -- start 2024-04-23T14:52:55.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.083+0000 7f80c7a16740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fa5b1cf5b0 0x55fa5b1e0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:55.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.083+0000 7f80c7a16740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa5b1cff60 0x55fa5b1e05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:55.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.083+0000 7f80c7a16740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fa5b1e3060 0x55fa5b1e0b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:55.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.083+0000 7f80c7a16740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fa5b1e5e20 con 0x55fa5b1cff60 2024-04-23T14:52:55.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.083+0000 7f80c7a16740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55fa5b0ee690 con 0x55fa5b1cf5b0 2024-04-23T14:52:55.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.083+0000 7f80c7a16740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55fa5b1e5fa0 con 0x55fa5b1e3060 2024-04-23T14:52:55.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.083+0000 7f80c7a16740 10 client.0 _ll_register_callbacks cb 0x55fa5b12a650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:52:55.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.083+0000 7f80bde00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fa5b1cf5b0 0x55fa5b1e0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:55.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.083+0000 7f80bde00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fa5b1cf5b0 0x55fa5b1e0090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60370/0 (socket says 172.21.15.28:60370) 2024-04-23T14:52:55.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.083+0000 7f80b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa5b1cff60 0x55fa5b1e05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:55.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.083+0000 7f80bde00640 1 -- 172.21.15.28:0/3242158900 learned_addr learned my addr 172.21.15.28:0/3242158900 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:55.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.083+0000 7f80be800640 1 --2- 172.21.15.28:0/3242158900 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fa5b1e3060 0x55fa5b1e0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:55.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.084+0000 7f80bde00640 1 -- 172.21.15.28:0/3242158900 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fa5b1e3060 msgr2=0x55fa5b1e0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:55.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.084+0000 7f80bde00640 1 --2- 172.21.15.28:0/3242158900 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fa5b1e3060 0x55fa5b1e0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.084+0000 7f80bde00640 1 -- 172.21.15.28:0/3242158900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa5b1cff60 msgr2=0x55fa5b1e05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:55.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.084+0000 7f80bde00640 1 --2- 172.21.15.28:0/3242158900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa5b1cff60 0x55fa5b1e05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.084+0000 7f80bde00640 1 -- 172.21.15.28:0/3242158900 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa5b0ed920 con 0x55fa5b1cf5b0 2024-04-23T14:52:55.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.084+0000 7f80b7e00640 1 --2- 172.21.15.28:0/3242158900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa5b1cff60 0x55fa5b1e05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:52:55.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.084+0000 7f80bde00640 1 --2- 172.21.15.28:0/3242158900 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fa5b1cf5b0 0x55fa5b1e0090 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f80b00096c0 tx=0x7f80b0009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=730a486f840bbcfa server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:55.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.084+0000 7f80a3400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:52:55.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.084+0000 7f80a3400640 1 -- 172.21.15.28:0/3242158900 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b004d020 con 0x55fa5b1cf5b0 2024-04-23T14:52:55.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.084+0000 7f80a3400640 1 -- 172.21.15.28:0/3242158900 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7f80b0005450 con 0x55fa5b1cf5b0 2024-04-23T14:52:55.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.084+0000 7f80c7a16740 1 -- 172.21.15.28:0/3242158900 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80a8013de0 con 0x55fa5b1cf5b0 2024-04-23T14:52:55.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.085+0000 7f80a3400640 1 -- 172.21.15.28:0/3242158900 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f80b0058050 con 0x55fa5b1cf5b0 2024-04-23T14:52:55.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.085+0000 7f80c7a16740 10 client.26651 fetch_fsmap learned FSMap version 20 2024-04-23T14:52:55.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.085+0000 7f80c7a16740 1 -- 172.21.15.28:0/3242158900 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55fa5af117d0 con 0x55fa5b1cf5b0 2024-04-23T14:52:55.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.085+0000 7f80a3400640 1 -- 172.21.15.28:0/3242158900 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80a8013de0 con 0x55fa5b1cf5b0 2024-04-23T14:52:55.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.086+0000 7f80a3400640 1 -- 172.21.15.28:0/3242158900 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f80b00328c0 con 0x55fa5b1cf5b0 2024-04-23T14:52:55.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.086+0000 7f80c7a16740 10 client.26651 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:52:55.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.086+0000 7f80c7a16740 10 client.26651 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:52:55.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.086+0000 7f80c7a16740 20 client.26651 populate_metadata read hostname 'smithi028' 2024-04-23T14:52:55.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.086+0000 7f80c7a16740 1 --2- 172.21.15.28:0/3242158900 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55fa5b1f34d0 0x55fa5b1f5940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:55.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.086+0000 7f80c7a16740 4 client.26651 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:52:55.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.086+0000 7f80c7a16740 1 -- 172.21.15.28:0/3242158900 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fa5b142c10 con 0x55fa5b1f34d0 2024-04-23T14:52:55.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.086+0000 7f80b7e00640 1 --2- 172.21.15.28:0/3242158900 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55fa5b1f34d0 0x55fa5b1f5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:55.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.087+0000 7f80b7e00640 1 --2- 172.21.15.28:0/3242158900 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55fa5b1f34d0 0x55fa5b1f5940 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2b3283d7564727b server_cookie=be1d5f527c47958c in_seq=0 out_seq=0 2024-04-23T14:52:55.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.087+0000 7f80a3400640 10 client.26651 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:55.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.088+0000 7f80a3400640 1 -- 172.21.15.28:0/3242158900 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fa5b142c10 con 0x55fa5b1f34d0 2024-04-23T14:52:55.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.088+0000 7f80a3400640 10 client.26651 handle_command_reply: tid=0 2024-04-23T14:52:55.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.089+0000 7f80c7a16740 1 client.26651 shutdown 2024-04-23T14:52:55.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.089+0000 7f80c7a16740 1 -- 172.21.15.28:0/3242158900 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fa5b1cf5b0 msgr2=0x55fa5b1e0090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:55.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.089+0000 7f80c7a16740 1 --2- 172.21.15.28:0/3242158900 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fa5b1cf5b0 0x55fa5b1e0090 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f80b00096c0 tx=0x7f80b0009260 comp rx=0 tx=0).stop 2024-04-23T14:52:55.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.089+0000 7f80c7a16740 1 -- 172.21.15.28:0/3242158900 shutdown_connections 2024-04-23T14:52:55.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.089+0000 7f80c7a16740 1 --2- 172.21.15.28:0/3242158900 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55fa5b1f34d0 0x55fa5b1f5940 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.089+0000 7f80c7a16740 1 --2- 172.21.15.28:0/3242158900 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fa5b1e3060 0x55fa5b1e0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.089+0000 7f80c7a16740 1 --2- 172.21.15.28:0/3242158900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa5b1cff60 0x55fa5b1e05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.089+0000 7f80a3400640 0 client.26651 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:55.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.089+0000 7f80c7a16740 1 --2- 172.21.15.28:0/3242158900 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fa5b1cf5b0 0x55fa5b1e0090 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.089+0000 7f80c7a16740 1 -- 172.21.15.28:0/3242158900 >> 172.21.15.28:0/3242158900 conn(0x55fa5b1cbb80 msgr2=0x55fa5b1d1980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:55.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.089+0000 7f80c7a16740 1 -- 172.21.15.28:0/3242158900 shutdown_connections 2024-04-23T14:52:55.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.089+0000 7f80c7a16740 1 -- 172.21.15.28:0/3242158900 wait complete. 2024-04-23T14:52:55.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.089+0000 7f80c7a16740 20 client.26651 trim_cache size 0 max 16384 2024-04-23T14:52:55.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.109+0000 7f80c7a16740 1 Processor -- start 2024-04-23T14:52:55.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.109+0000 7f80c7a16740 1 -- start start 2024-04-23T14:52:55.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.110+0000 7f80c7a16740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa5b1cf5b0 0x55fa5b1eed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:55.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.110+0000 7f80c7a16740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fa5b1cff60 0x55fa5b1caaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:55.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.110+0000 7f80c7a16740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fa5b1e3060 0x55fa5b1fb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:55.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.110+0000 7f80c7a16740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f80c01a9c50 con 0x55fa5b1cf5b0 2024-04-23T14:52:55.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.110+0000 7f80c7a16740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f80c01ac4b0 con 0x55fa5b1e3060 2024-04-23T14:52:55.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.110+0000 7f80c7a16740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f80c01aa2a0 con 0x55fa5b1cff60 2024-04-23T14:52:55.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.110+0000 7f80be800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fa5b1e3060 0x55fa5b1fb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:55.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.110+0000 7f80be800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fa5b1e3060 0x55fa5b1fb210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60386/0 (socket says 172.21.15.28:60386) 2024-04-23T14:52:55.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.110+0000 7f80be800640 1 -- 172.21.15.28:0/423380572 learned_addr learned my addr 172.21.15.28:0/423380572 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:55.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.110+0000 7f80b7e00640 1 --2- 172.21.15.28:0/423380572 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fa5b1cff60 0x55fa5b1caaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:55.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.110+0000 7f80bde00640 1 --2- 172.21.15.28:0/423380572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa5b1cf5b0 0x55fa5b1eed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:55.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.110+0000 7f80be800640 1 -- 172.21.15.28:0/423380572 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fa5b1cff60 msgr2=0x55fa5b1caaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:55.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.110+0000 7f80be800640 1 --2- 172.21.15.28:0/423380572 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fa5b1cff60 0x55fa5b1caaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.110+0000 7f80be800640 1 -- 172.21.15.28:0/423380572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa5b1cf5b0 msgr2=0x55fa5b1eed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:55.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.111+0000 7f80be800640 1 --2- 172.21.15.28:0/423380572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa5b1cf5b0 0x55fa5b1eed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.111+0000 7f80be800640 1 -- 172.21.15.28:0/423380572 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa5ae473f0 con 0x55fa5b1e3060 2024-04-23T14:52:55.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.111+0000 7f80bde00640 1 --2- 172.21.15.28:0/423380572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa5b1cf5b0 0x55fa5b1eed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:52:55.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.111+0000 7f80be800640 1 --2- 172.21.15.28:0/423380572 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fa5b1e3060 0x55fa5b1fb210 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f80b8099620 tx=0x7f80b800efd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15ee72d35bafc3c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:55.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.111+0000 7f80ae600640 1 -- 172.21.15.28:0/423380572 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b80a1000 con 0x55fa5b1e3060 2024-04-23T14:52:55.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.111+0000 7f80ae600640 1 -- 172.21.15.28:0/423380572 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80b80a58c0 con 0x55fa5b1e3060 2024-04-23T14:52:55.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.111+0000 7f80ae600640 1 -- 172.21.15.28:0/423380572 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b80a5430 con 0x55fa5b1e3060 2024-04-23T14:52:55.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.112+0000 7f80c7a16740 1 -- 172.21.15.28:0/423380572 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fa5b1e3060 msgr2=0x55fa5b1fb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:55.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.112+0000 7f80c7a16740 1 --2- 172.21.15.28:0/423380572 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fa5b1e3060 0x55fa5b1fb210 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f80b8099620 tx=0x7f80b800efd0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.112+0000 7f80c7a16740 1 -- 172.21.15.28:0/423380572 shutdown_connections 2024-04-23T14:52:55.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.112+0000 7f80c7a16740 1 --2- 172.21.15.28:0/423380572 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fa5b1e3060 0x55fa5b1fb210 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.112+0000 7f80c7a16740 1 --2- 172.21.15.28:0/423380572 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fa5b1cff60 0x55fa5b1caaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.112+0000 7f80c7a16740 1 --2- 172.21.15.28:0/423380572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa5b1cf5b0 0x55fa5b1eed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.112+0000 7f80c7a16740 1 -- 172.21.15.28:0/423380572 >> 172.21.15.28:0/423380572 conn(0x55fa5b1cbb80 msgr2=0x55fa5b1ca390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:55.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.112+0000 7f80c7a16740 1 -- 172.21.15.28:0/423380572 shutdown_connections 2024-04-23T14:52:55.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.113+0000 7f80c7a16740 1 -- 172.21.15.28:0/423380572 wait complete. 2024-04-23T14:52:55.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.113+0000 7f80c7a16740 1 -- start start 2024-04-23T14:52:55.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.113+0000 7f80c7a16740 1 Processor -- start 2024-04-23T14:52:55.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.114+0000 7f80c7a16740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa5b1cf5b0 0x55fa5b1eb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:55.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.114+0000 7f80c7a16740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fa5b1cff60 0x55fa5b1dcad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:55.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.114+0000 7f80c7a16740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fa5b1e3060 0x55fa5b1dd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:55.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.114+0000 7f80c7a16740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f80c01aa970 con 0x55fa5b1cf5b0 2024-04-23T14:52:55.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.114+0000 7f80c7a16740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f80c01abde0 con 0x55fa5b1cff60 2024-04-23T14:52:55.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.114+0000 7f80c7a16740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f80c01ab040 con 0x55fa5b1e3060 2024-04-23T14:52:55.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.114+0000 7f80c7a16740 10 client.0 _ll_register_callbacks cb 0x7f80b0040070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:52:55.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.114+0000 7f80b7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fa5b1cff60 0x55fa5b1dcad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:55.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.114+0000 7f80bde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa5b1cf5b0 0x55fa5b1eb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:55.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.114+0000 7f80be800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fa5b1e3060 0x55fa5b1dd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:55.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.114+0000 7f80b7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fa5b1cff60 0x55fa5b1dcad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60398/0 (socket says 172.21.15.28:60398) 2024-04-23T14:52:55.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.114+0000 7f80bde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa5b1cf5b0 0x55fa5b1eb6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44542/0 (socket says 172.21.15.28:44542) 2024-04-23T14:52:55.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.114+0000 7f80b7e00640 1 -- 172.21.15.28:0/4163332895 learned_addr learned my addr 172.21.15.28:0/4163332895 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:55.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.114+0000 7f80b7e00640 1 -- 172.21.15.28:0/4163332895 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fa5b1e3060 msgr2=0x55fa5b1dd070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:55.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.114+0000 7f80b7e00640 1 --2- 172.21.15.28:0/4163332895 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fa5b1e3060 0x55fa5b1dd070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.114+0000 7f80b7e00640 1 -- 172.21.15.28:0/4163332895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa5b1cf5b0 msgr2=0x55fa5b1eb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:55.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.114+0000 7f80b7e00640 1 --2- 172.21.15.28:0/4163332895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa5b1cf5b0 0x55fa5b1eb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.114+0000 7f80b7e00640 1 -- 172.21.15.28:0/4163332895 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa5ae070f0 con 0x55fa5b1cff60 2024-04-23T14:52:55.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.115+0000 7f80be800640 1 --2- 172.21.15.28:0/4163332895 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fa5b1e3060 0x55fa5b1dd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:52:55.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.115+0000 7f80b7e00640 1 --2- 172.21.15.28:0/4163332895 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fa5b1cff60 0x55fa5b1dcad0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f80a8015010 tx=0x7f80a8005e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18d58b40f3fcdeea server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:55.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.115+0000 7f80a3400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:52:55.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.115+0000 7f80a3400640 1 -- 172.21.15.28:0/4163332895 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a802e070 con 0x55fa5b1cff60 2024-04-23T14:52:55.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.115+0000 7f80c7a16740 1 -- 172.21.15.28:0/4163332895 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80b80a58c0 con 0x55fa5b1cff60 2024-04-23T14:52:55.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.115+0000 7f80a3400640 1 -- 172.21.15.28:0/4163332895 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7f80a8004040 con 0x55fa5b1cff60 2024-04-23T14:52:55.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.115+0000 7f80a3400640 1 -- 172.21.15.28:0/4163332895 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f80a8004240 con 0x55fa5b1cff60 2024-04-23T14:52:55.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.115+0000 7f80c7a16740 10 client.26657 fetch_fsmap learned FSMap version 20 2024-04-23T14:52:55.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.115+0000 7f80c7a16740 1 -- 172.21.15.28:0/4163332895 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55fa5b0fa810 con 0x55fa5b1cff60 2024-04-23T14:52:55.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.116+0000 7f80a3400640 1 -- 172.21.15.28:0/4163332895 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80b80a58c0 con 0x55fa5b1cff60 2024-04-23T14:52:55.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.116+0000 7f80a3400640 1 -- 172.21.15.28:0/4163332895 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f80a8011940 con 0x55fa5b1cff60 2024-04-23T14:52:55.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.116+0000 7f80c7a16740 10 client.26657 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:52:55.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.116+0000 7f80c7a16740 10 client.26657 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:52:55.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.116+0000 7f80c7a16740 20 client.26657 populate_metadata read hostname 'smithi028' 2024-04-23T14:52:55.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.116+0000 7f80c7a16740 1 --2- 172.21.15.28:0/4163332895 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55fa5b259b50 0x55fa5b25bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:55.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.116+0000 7f80c7a16740 4 client.26657 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:52:55.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.116+0000 7f80c7a16740 1 -- 172.21.15.28:0/4163332895 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fa5b13ca80 con 0x55fa5b259b50 2024-04-23T14:52:55.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.116+0000 7f80bde00640 1 --2- 172.21.15.28:0/4163332895 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55fa5b259b50 0x55fa5b25bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:55.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.117+0000 7f80bde00640 1 --2- 172.21.15.28:0/4163332895 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55fa5b259b50 0x55fa5b25bf40 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e59615feaaa754b3 server_cookie=4970448a2d87bfd1 in_seq=0 out_seq=0 2024-04-23T14:52:55.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.117+0000 7f80a3400640 10 client.26657 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:55.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.118+0000 7f80a3400640 1 -- 172.21.15.28:0/4163332895 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55fa5b13ca80 con 0x55fa5b259b50 2024-04-23T14:52:55.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.118+0000 7f80a3400640 10 client.26657 handle_command_reply: tid=0 2024-04-23T14:52:55.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.118+0000 7f80c7a16740 1 client.26657 shutdown 2024-04-23T14:52:55.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.119+0000 7f80c7a16740 1 -- 172.21.15.28:0/4163332895 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fa5b1cff60 msgr2=0x55fa5b1dcad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:55.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.119+0000 7f80c7a16740 1 --2- 172.21.15.28:0/4163332895 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fa5b1cff60 0x55fa5b1dcad0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f80a8015010 tx=0x7f80a8005e20 comp rx=0 tx=0).stop 2024-04-23T14:52:55.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.119+0000 7f80c7a16740 1 -- 172.21.15.28:0/4163332895 shutdown_connections 2024-04-23T14:52:55.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.119+0000 7f80c7a16740 1 --2- 172.21.15.28:0/4163332895 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55fa5b259b50 0x55fa5b25bf40 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.119+0000 7f80c7a16740 1 --2- 172.21.15.28:0/4163332895 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fa5b1cff60 0x55fa5b1dcad0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.119+0000 7f80c7a16740 1 --2- 172.21.15.28:0/4163332895 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fa5b1e3060 0x55fa5b1dd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.119+0000 7f80a3400640 0 client.26657 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:55.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.119+0000 7f80c7a16740 1 --2- 172.21.15.28:0/4163332895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa5b1cf5b0 0x55fa5b1eb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.119+0000 7f80c7a16740 1 -- 172.21.15.28:0/4163332895 >> 172.21.15.28:0/4163332895 conn(0x55fa5b1cbb80 msgr2=0x55fa5b1de540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:55.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.119+0000 7f80c7a16740 1 -- 172.21.15.28:0/4163332895 shutdown_connections 2024-04-23T14:52:55.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.119+0000 7f80c7a16740 1 -- 172.21.15.28:0/4163332895 wait complete. 2024-04-23T14:52:55.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.119+0000 7f80c7a16740 20 client.26657 trim_cache size 0 max 16384 2024-04-23T14:52:55.121 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:52:55.121 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-23T14:52:55.122 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-23T14:52:55.122 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:52:55.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.120+0000 7f80c4c00640 1 -- 172.21.15.28:0/630379177 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f80a40a3da0 msgr2=0x7f80a40a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:55.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.120+0000 7f80c4c00640 1 --2- 172.21.15.28:0/630379177 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f80a40a3da0 0x7f80a40a6260 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f80b0006010 tx=0x7f80b0000bc0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.120+0000 7f80c4c00640 1 -- 172.21.15.28:0/630379177 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80c019a960 msgr2=0x7f80c019ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:55.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.120+0000 7f80c4c00640 1 --2- 172.21.15.28:0/630379177 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80c019a960 0x7f80c019ae10 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f80b800eb10 tx=0x7f80b800c2d0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.120+0000 7f80c4c00640 1 -- 172.21.15.28:0/630379177 shutdown_connections 2024-04-23T14:52:55.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.120+0000 7f80c4c00640 1 --2- 172.21.15.28:0/630379177 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f80a40a3da0 0x7f80a40a6260 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.120+0000 7f80c4c00640 1 --2- 172.21.15.28:0/630379177 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80c019a960 0x7f80c019ae10 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.120+0000 7f80c4c00640 1 --2- 172.21.15.28:0/630379177 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f80c0109520 0x7f80c01a0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.120+0000 7f80c4c00640 1 --2- 172.21.15.28:0/630379177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80c0104450 0x7f80c01a07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.120+0000 7f80c4c00640 1 -- 172.21.15.28:0/630379177 >> 172.21.15.28:0/630379177 conn(0x7f80c00ffc70 msgr2=0x7f80c0101fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:55.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.120+0000 7f80c4c00640 1 -- 172.21.15.28:0/630379177 shutdown_connections 2024-04-23T14:52:55.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.120+0000 7f80c4c00640 1 -- 172.21.15.28:0/630379177 wait complete. 2024-04-23T14:52:55.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:54 smithi038 ceph-mon[32348]: pgmap v273: 97 pgs: 97 active+clean; 454 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.8 MiB/s wr, 202 op/s 2024-04-23T14:52:55.138 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-23T14:52:55.139 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:52:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.414+0000 7f67c2400640 1 -- 172.21.15.28:0/1277813872 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67bc102230 msgr2=0x7f67bc102610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.414+0000 7f67c2400640 1 --2- 172.21.15.28:0/1277813872 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67bc102230 0x7f67bc102610 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f67ac009900 tx=0x7f67ac02a3f0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.414+0000 7f67c2400640 1 -- 172.21.15.28:0/1277813872 shutdown_connections 2024-04-23T14:52:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.414+0000 7f67c2400640 1 --2- 172.21.15.28:0/1277813872 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67bc107300 0x7f67bc10eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.414+0000 7f67c2400640 1 --2- 172.21.15.28:0/1277813872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67bc102be0 0x7f67bc106c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.414+0000 7f67c2400640 1 --2- 172.21.15.28:0/1277813872 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67bc102230 0x7f67bc102610 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.414+0000 7f67c2400640 1 -- 172.21.15.28:0/1277813872 >> 172.21.15.28:0/1277813872 conn(0x7f67bc079400 msgr2=0x7f67bc079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.414+0000 7f67c2400640 1 -- 172.21.15.28:0/1277813872 shutdown_connections 2024-04-23T14:52:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.414+0000 7f67c2400640 1 -- 172.21.15.28:0/1277813872 wait complete. 2024-04-23T14:52:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.415+0000 7f67c2400640 1 Processor -- start 2024-04-23T14:52:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.415+0000 7f67c2400640 1 -- start start 2024-04-23T14:52:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.415+0000 7f67c2400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67bc102230 0x7f67bc1a0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.415+0000 7f67c2400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67bc107300 0x7f67bc1a0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.415+0000 7f67c2400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67bc19a920 0x7f67bc19add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.415+0000 7f67c2400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc112410 con 0x7f67bc19a920 2024-04-23T14:52:55.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.416+0000 7f67c2400640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc112290 con 0x7f67bc107300 2024-04-23T14:52:55.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.416+0000 7f67c2400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc112590 con 0x7f67bc102230 2024-04-23T14:52:55.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.416+0000 7f67bb400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67bc102230 0x7f67bc1a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:55.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.416+0000 7f67b3e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67bc107300 0x7f67bc1a0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:55.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.416+0000 7f67bb400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67bc102230 0x7f67bc1a0770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:46112/0 (socket says 172.21.15.28:46112) 2024-04-23T14:52:55.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.416+0000 7f67bb400640 1 -- 172.21.15.28:0/959608954 learned_addr learned my addr 172.21.15.28:0/959608954 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:55.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.416+0000 7f67bbe00640 1 --2- 172.21.15.28:0/959608954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67bc19a920 0x7f67bc19add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:55.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.416+0000 7f67b3e00640 1 -- 172.21.15.28:0/959608954 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67bc102230 msgr2=0x7f67bc1a0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:55.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.416+0000 7f67b3e00640 1 --2- 172.21.15.28:0/959608954 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67bc102230 0x7f67bc1a0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.416+0000 7f67b3e00640 1 -- 172.21.15.28:0/959608954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67bc19a920 msgr2=0x7f67bc19add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:55.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.416+0000 7f67b3e00640 1 --2- 172.21.15.28:0/959608954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67bc19a920 0x7f67bc19add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.416+0000 7f67b3e00640 1 -- 172.21.15.28:0/959608954 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67bc19b440 con 0x7f67bc107300 2024-04-23T14:52:55.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.416+0000 7f67bb400640 1 --2- 172.21.15.28:0/959608954 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67bc102230 0x7f67bc1a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:52:55.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.417+0000 7f67b3e00640 1 --2- 172.21.15.28:0/959608954 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67bc107300 0x7f67bc1a0cb0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f67a400e9c0 tx=0x7f67a400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:55.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.417+0000 7f67b8c00640 1 -- 172.21.15.28:0/959608954 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a400cd80 con 0x7f67bc107300 2024-04-23T14:52:55.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.417+0000 7f67b8c00640 1 -- 172.21.15.28:0/959608954 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67a40035f0 con 0x7f67bc107300 2024-04-23T14:52:55.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.417+0000 7f67b8c00640 1 -- 172.21.15.28:0/959608954 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a4010690 con 0x7f67bc107300 2024-04-23T14:52:55.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.417+0000 7f67c2400640 1 -- 172.21.15.28:0/959608954 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67bc19b730 con 0x7f67bc107300 2024-04-23T14:52:55.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.417+0000 7f67c2400640 1 -- 172.21.15.28:0/959608954 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67bc1a7930 con 0x7f67bc107300 2024-04-23T14:52:55.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.418+0000 7f67c2400640 1 -- 172.21.15.28:0/959608954 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67bc069270 con 0x7f67bc107300 2024-04-23T14:52:55.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.420+0000 7f67b8c00640 1 -- 172.21.15.28:0/959608954 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f67a40037a0 con 0x7f67bc107300 2024-04-23T14:52:55.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.422+0000 7f67b8c00640 1 --2- 172.21.15.28:0/959608954 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f67a00a3da0 0x7f67a00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:55.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.422+0000 7f67b8c00640 1 -- 172.21.15.28:0/959608954 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7f67a40d0be0 con 0x7f67bc107300 2024-04-23T14:52:55.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.422+0000 7f67bb400640 1 --2- 172.21.15.28:0/959608954 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f67a00a3da0 0x7f67a00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:55.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.423+0000 7f67bb400640 1 --2- 172.21.15.28:0/959608954 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f67a00a3da0 0x7f67a00a6260 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f67ac0096c0 tx=0x7f67ac0046c0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:55.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.428+0000 7f67b8c00640 1 -- 172.21.15.28:0/959608954 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f67a409d210 con 0x7f67bc107300 2024-04-23T14:52:55.515 INFO:tasks.workunit.client.0.smithi028.stdout: 1 747452 8.94 MB/sec execute 311 sec latency 93.741 ms 2024-04-23T14:52:55.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.695+0000 7f67c2400640 1 -- 172.21.15.28:0/959608954 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f67bc19be50 con 0x7f67bc107300 2024-04-23T14:52:55.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.696+0000 7f67b8c00640 1 -- 172.21.15.28:0/959608954 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7f67a409cb10 con 0x7f67bc107300 2024-04-23T14:52:55.696 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:52:55.696 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:52:55.696 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:52:55.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.699+0000 7f67c2400640 1 -- 172.21.15.28:0/959608954 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f67a00a3da0 msgr2=0x7f67a00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:55.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.699+0000 7f67c2400640 1 --2- 172.21.15.28:0/959608954 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f67a00a3da0 0x7f67a00a6260 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f67ac0096c0 tx=0x7f67ac0046c0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.699+0000 7f67c2400640 1 -- 172.21.15.28:0/959608954 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67bc107300 msgr2=0x7f67bc1a0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:55.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.699+0000 7f67c2400640 1 --2- 172.21.15.28:0/959608954 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67bc107300 0x7f67bc1a0cb0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f67a400e9c0 tx=0x7f67a400ee80 comp rx=0 tx=0).stop 2024-04-23T14:52:55.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.699+0000 7f67c2400640 1 -- 172.21.15.28:0/959608954 shutdown_connections 2024-04-23T14:52:55.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.699+0000 7f67c2400640 1 --2- 172.21.15.28:0/959608954 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f67a00a3da0 0x7f67a00a6260 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.699+0000 7f67c2400640 1 --2- 172.21.15.28:0/959608954 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67bc107300 0x7f67bc1a0cb0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.699+0000 7f67c2400640 1 --2- 172.21.15.28:0/959608954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67bc19a920 0x7f67bc19add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.699+0000 7f67c2400640 1 --2- 172.21.15.28:0/959608954 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67bc102230 0x7f67bc1a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.699+0000 7f67c2400640 1 -- 172.21.15.28:0/959608954 >> 172.21.15.28:0/959608954 conn(0x7f67bc079400 msgr2=0x7f67bc0ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:55.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.699+0000 7f67c2400640 1 -- 172.21.15.28:0/959608954 shutdown_connections 2024-04-23T14:52:55.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.700+0000 7f67c2400640 1 -- 172.21.15.28:0/959608954 wait complete. 2024-04-23T14:52:55.721 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-23T14:52:55.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.954+0000 7f9a99a00640 1 -- 172.21.15.28:0/841533822 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a94103980 msgr2=0x7f9a9410be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:55.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.954+0000 7f9a99a00640 1 --2- 172.21.15.28:0/841533822 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a94103980 0x7f9a9410be60 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c009900 tx=0x7f9a7c02a480 comp rx=0 tx=0).stop 2024-04-23T14:52:55.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.955+0000 7f9a99a00640 1 -- 172.21.15.28:0/841533822 shutdown_connections 2024-04-23T14:52:55.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.955+0000 7f9a99a00640 1 --2- 172.21.15.28:0/841533822 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a94103980 0x7f9a9410be60 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.955+0000 7f9a99a00640 1 --2- 172.21.15.28:0/841533822 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a9410c480 0x7f9a9410e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.955+0000 7f9a99a00640 1 --2- 172.21.15.28:0/841533822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a94103060 0x7f9a94103440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.955+0000 7f9a99a00640 1 -- 172.21.15.28:0/841533822 >> 172.21.15.28:0/841533822 conn(0x7f9a940fc880 msgr2=0x7f9a940feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:55.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.955+0000 7f9a99a00640 1 -- 172.21.15.28:0/841533822 shutdown_connections 2024-04-23T14:52:55.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.955+0000 7f9a99a00640 1 -- 172.21.15.28:0/841533822 wait complete. 2024-04-23T14:52:55.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.956+0000 7f9a99a00640 1 Processor -- start 2024-04-23T14:52:55.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.956+0000 7f9a99a00640 1 -- start start 2024-04-23T14:52:55.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.956+0000 7f9a99a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a94103060 0x7f9a941a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:55.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.956+0000 7f9a99a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a94103980 0x7f9a941a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:55.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.956+0000 7f9a99a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a9410c480 0x7f9a9419a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:55.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.956+0000 7f9a99a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9a941122b0 con 0x7f9a9410c480 2024-04-23T14:52:55.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.956+0000 7f9a99a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9a94112130 con 0x7f9a94103980 2024-04-23T14:52:55.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.956+0000 7f9a99a00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9a94112430 con 0x7f9a94103060 2024-04-23T14:52:55.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.956+0000 7f9a8b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a94103980 0x7f9a941a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:55.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.957+0000 7f9a99000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a9410c480 0x7f9a9419a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:55.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.957+0000 7f9a8b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a94103980 0x7f9a941a0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60428/0 (socket says 172.21.15.28:60428) 2024-04-23T14:52:55.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.957+0000 7f9a93e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a94103060 0x7f9a941a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:55.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.957+0000 7f9a8b400640 1 -- 172.21.15.28:0/1731888645 learned_addr learned my addr 172.21.15.28:0/1731888645 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:55.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.957+0000 7f9a93e00640 1 -- 172.21.15.28:0/1731888645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a94103980 msgr2=0x7f9a941a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:55.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.957+0000 7f9a93e00640 1 --2- 172.21.15.28:0/1731888645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a94103980 0x7f9a941a0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.957+0000 7f9a93e00640 1 -- 172.21.15.28:0/1731888645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a9410c480 msgr2=0x7f9a9419a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:55.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.957+0000 7f9a93e00640 1 --2- 172.21.15.28:0/1731888645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a9410c480 0x7f9a9419a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.957+0000 7f9a93e00640 1 -- 172.21.15.28:0/1731888645 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a9419afc0 con 0x7f9a94103060 2024-04-23T14:52:55.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.957+0000 7f9a93e00640 1 --2- 172.21.15.28:0/1731888645 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a94103060 0x7f9a941a0680 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f9a8400eb00 tx=0x7f9a8400c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:55.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.958+0000 7f9a91600640 1 -- 172.21.15.28:0/1731888645 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a84010430 con 0x7f9a94103060 2024-04-23T14:52:55.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.958+0000 7f9a91600640 1 -- 172.21.15.28:0/1731888645 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a84010ab0 con 0x7f9a94103060 2024-04-23T14:52:55.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.958+0000 7f9a99a00640 1 -- 172.21.15.28:0/1731888645 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a9419b2b0 con 0x7f9a94103060 2024-04-23T14:52:55.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.958+0000 7f9a91600640 1 -- 172.21.15.28:0/1731888645 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a84017bb0 con 0x7f9a94103060 2024-04-23T14:52:55.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.958+0000 7f9a99a00640 1 -- 172.21.15.28:0/1731888645 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a941a74d0 con 0x7f9a94103060 2024-04-23T14:52:55.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.961+0000 7f9a9de0b740 1 Processor -- start 2024-04-23T14:52:55.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.962+0000 7f9a9de0b740 1 -- start start 2024-04-23T14:52:55.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.962+0000 7f9a9de0b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556569b58b40 0x556569b58f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:55.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.962+0000 7f9a9de0b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556569b59460 0x556569b635f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:55.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.962+0000 7f9a9de0b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556569b63b30 0x556569b65f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:55.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.962+0000 7f9a9de0b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556569abe1d0 con 0x556569b63b30 2024-04-23T14:52:55.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.962+0000 7f9a9de0b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x556569a64450 con 0x556569b59460 2024-04-23T14:52:55.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.962+0000 7f9a9de0b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556569aba190 con 0x556569b58b40 2024-04-23T14:52:55.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.963+0000 7f9a99000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556569b58b40 0x556569b58f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:55.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.963+0000 7f9a99000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556569b58b40 0x556569b58f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:46148/0 (socket says 172.21.15.28:46148) 2024-04-23T14:52:55.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.963+0000 7f9a99000640 1 -- 172.21.15.28:0/1192994017 learned_addr learned my addr 172.21.15.28:0/1192994017 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:55.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.963+0000 7f9a8b400640 1 --2- 172.21.15.28:0/1192994017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556569b63b30 0x556569b65f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:55.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.963+0000 7f9a91600640 1 -- 172.21.15.28:0/1731888645 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f9a84010c60 con 0x7f9a94103060 2024-04-23T14:52:55.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.963+0000 7f9a93e00640 1 --2- 172.21.15.28:0/1192994017 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556569b59460 0x556569b635f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:55.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.964+0000 7f9a8b400640 1 -- 172.21.15.28:0/1192994017 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556569b58b40 msgr2=0x556569b58f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:55.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.964+0000 7f9a8b400640 1 --2- 172.21.15.28:0/1192994017 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556569b58b40 0x556569b58f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.964+0000 7f9a8b400640 1 -- 172.21.15.28:0/1192994017 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556569b59460 msgr2=0x556569b635f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:55.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.964+0000 7f9a8b400640 1 --2- 172.21.15.28:0/1192994017 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556569b59460 0x556569b635f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.964+0000 7f9a8b400640 1 -- 172.21.15.28:0/1192994017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556569b21310 con 0x556569b63b30 2024-04-23T14:52:55.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.964+0000 7f9a93e00640 1 --2- 172.21.15.28:0/1192994017 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556569b59460 0x556569b635f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:52:55.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.964+0000 7f9a8b400640 1 --2- 172.21.15.28:0/1192994017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556569b63b30 0x556569b65f20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c000c00 tx=0x7f9a7c005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5b93df4b243c3f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:55.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.965+0000 7f9a89600640 1 -- 172.21.15.28:0/1192994017 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c040070 con 0x556569b63b30 2024-04-23T14:52:55.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.965+0000 7f9a89600640 1 -- 172.21.15.28:0/1192994017 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a7c005500 con 0x556569b63b30 2024-04-23T14:52:55.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.965+0000 7f9a91600640 1 --2- 172.21.15.28:0/1731888645 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f9a780a3e20 0x7f9a780a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:55.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.965+0000 7f9a89600640 1 -- 172.21.15.28:0/1192994017 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c035040 con 0x556569b63b30 2024-04-23T14:52:55.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.965+0000 7f9a91600640 1 -- 172.21.15.28:0/1731888645 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7f9a840d1150 con 0x7f9a94103060 2024-04-23T14:52:55.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.966+0000 7f9a99000640 1 --2- 172.21.15.28:0/1731888645 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f9a780a3e20 0x7f9a780a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:55.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.966+0000 7f9a9de0b740 1 -- 172.21.15.28:0/1192994017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556569b63b30 msgr2=0x556569b65f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:55.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.966+0000 7f9a9de0b740 1 --2- 172.21.15.28:0/1192994017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556569b63b30 0x556569b65f20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c000c00 tx=0x7f9a7c005c00 comp rx=0 tx=0).stop 2024-04-23T14:52:55.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.966+0000 7f9a9de0b740 1 -- 172.21.15.28:0/1192994017 shutdown_connections 2024-04-23T14:52:55.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.966+0000 7f9a9de0b740 1 --2- 172.21.15.28:0/1192994017 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556569b59460 0x556569b635f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.966+0000 7f9a9de0b740 1 --2- 172.21.15.28:0/1192994017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556569b63b30 0x556569b65f20 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.966+0000 7f9a9de0b740 1 --2- 172.21.15.28:0/1192994017 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556569b58b40 0x556569b58f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.966+0000 7f9a9de0b740 1 -- 172.21.15.28:0/1192994017 >> 172.21.15.28:0/1192994017 conn(0x556569b54e60 msgr2=0x556569b55240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:55.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.967+0000 7f9a9de0b740 1 -- 172.21.15.28:0/1192994017 shutdown_connections 2024-04-23T14:52:55.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.967+0000 7f9a9de0b740 1 -- 172.21.15.28:0/1192994017 wait complete. 2024-04-23T14:52:55.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.967+0000 7f9a99000640 1 --2- 172.21.15.28:0/1731888645 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f9a780a3e20 0x7f9a780a62e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c009e60 tx=0x7f9a8c005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:55.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.967+0000 7f9a9de0b740 1 -- start start 2024-04-23T14:52:55.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.968+0000 7f9a9de0b740 1 Processor -- start 2024-04-23T14:52:55.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.968+0000 7f9a9de0b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556569b58b40 0x556569b63440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:55.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.968+0000 7f9a9de0b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556569b59460 0x556569b60730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:55.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.968+0000 7f9a9de0b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556569b63b30 0x556569b60c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:55.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.969+0000 7f9a9de0b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556569b669b0 con 0x556569b63b30 2024-04-23T14:52:55.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.969+0000 7f9a9de0b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x556569a77810 con 0x556569b58b40 2024-04-23T14:52:55.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.969+0000 7f9a9de0b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556569b66b30 con 0x556569b59460 2024-04-23T14:52:55.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.969+0000 7f9a9de0b740 10 client.0 _ll_register_callbacks cb 0x556569a45140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:52:55.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.969+0000 7f9a93e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556569b58b40 0x556569b63440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:55.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.969+0000 7f9a8b400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556569b59460 0x556569b60730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:55.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.969+0000 7f9a93e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556569b58b40 0x556569b63440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60448/0 (socket says 172.21.15.28:60448) 2024-04-23T14:52:55.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.969+0000 7f9a8b400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556569b59460 0x556569b60730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:46164/0 (socket says 172.21.15.28:46164) 2024-04-23T14:52:55.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.969+0000 7f9a93e00640 1 -- 172.21.15.28:0/2701431605 learned_addr learned my addr 172.21.15.28:0/2701431605 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:55.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.969+0000 7f9a99000640 1 --2- 172.21.15.28:0/2701431605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556569b63b30 0x556569b60c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:55.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.969+0000 7f9a8b400640 1 -- 172.21.15.28:0/2701431605 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556569b58b40 msgr2=0x556569b63440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:55.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.969+0000 7f9a8b400640 1 --2- 172.21.15.28:0/2701431605 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556569b58b40 0x556569b63440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.969+0000 7f9a8b400640 1 -- 172.21.15.28:0/2701431605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556569b63b30 msgr2=0x556569b60c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:55.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.969+0000 7f9a8b400640 1 --2- 172.21.15.28:0/2701431605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556569b63b30 0x556569b60c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.969+0000 7f9a8b400640 1 -- 172.21.15.28:0/2701431605 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556569a883f0 con 0x556569b59460 2024-04-23T14:52:55.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.969+0000 7f9a99000640 1 --2- 172.21.15.28:0/2701431605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556569b63b30 0x556569b60c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:52:55.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.970+0000 7f9a8b400640 1 --2- 172.21.15.28:0/2701431605 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556569b59460 0x556569b60730 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x556569b55ff0 tx=0x7f9a7c0058b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75a822310c1ab618 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:55.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.970+0000 7f9a82000640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:52:55.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.970+0000 7f9a82000640 1 -- 172.21.15.28:0/2701431605 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c035070 con 0x556569b59460 2024-04-23T14:52:55.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.970+0000 7f9a82000640 1 -- 172.21.15.28:0/2701431605 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7f9a7c004380 con 0x556569b59460 2024-04-23T14:52:55.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.970+0000 7f9a9de0b740 1 -- 172.21.15.28:0/2701431605 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a7c005500 con 0x556569b59460 2024-04-23T14:52:55.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.970+0000 7f9a82000640 1 -- 172.21.15.28:0/2701431605 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f9a7c040040 con 0x556569b59460 2024-04-23T14:52:55.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.970+0000 7f9a9de0b740 10 client.26653 fetch_fsmap learned FSMap version 20 2024-04-23T14:52:55.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.970+0000 7f9a9de0b740 1 -- 172.21.15.28:0/2701431605 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5565697d03f0 con 0x556569b59460 2024-04-23T14:52:55.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.971+0000 7f9a82000640 1 -- 172.21.15.28:0/2701431605 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a7c005500 con 0x556569b59460 2024-04-23T14:52:55.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.971+0000 7f9a82000640 1 -- 172.21.15.28:0/2701431605 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f9a7c031ad0 con 0x556569b59460 2024-04-23T14:52:55.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.971+0000 7f9a9de0b740 10 client.26653 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:52:55.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.971+0000 7f9a9de0b740 10 client.26653 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:52:55.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.971+0000 7f9a9de0b740 20 client.26653 populate_metadata read hostname 'smithi028' 2024-04-23T14:52:55.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.971+0000 7f9a9de0b740 1 --2- 172.21.15.28:0/2701431605 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x556569b73aa0 0x556569b75e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:55.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.971+0000 7f9a9de0b740 4 client.26653 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:52:55.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.971+0000 7f9a9de0b740 1 -- 172.21.15.28:0/2701431605 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556569acbbf0 con 0x556569b73aa0 2024-04-23T14:52:55.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.971+0000 7f9a93e00640 1 --2- 172.21.15.28:0/2701431605 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x556569b73aa0 0x556569b75e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:55.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.972+0000 7f9a93e00640 1 --2- 172.21.15.28:0/2701431605 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x556569b73aa0 0x556569b75e90 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ae2a9c1c265c27f server_cookie=e5c9bfbba9721c8a in_seq=0 out_seq=0 2024-04-23T14:52:55.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.972+0000 7f9a82000640 10 client.26653 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:55.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.973+0000 7f9a82000640 1 -- 172.21.15.28:0/2701431605 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556569acbbf0 con 0x556569b73aa0 2024-04-23T14:52:55.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.973+0000 7f9a82000640 10 client.26653 handle_command_reply: tid=0 2024-04-23T14:52:55.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.973+0000 7f9a9de0b740 1 client.26653 shutdown 2024-04-23T14:52:55.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.974+0000 7f9a9de0b740 1 -- 172.21.15.28:0/2701431605 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556569b59460 msgr2=0x556569b60730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:55.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.974+0000 7f9a9de0b740 1 --2- 172.21.15.28:0/2701431605 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556569b59460 0x556569b60730 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x556569b55ff0 tx=0x7f9a7c0058b0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.974+0000 7f9a9de0b740 1 -- 172.21.15.28:0/2701431605 shutdown_connections 2024-04-23T14:52:55.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.974+0000 7f9a9de0b740 1 --2- 172.21.15.28:0/2701431605 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x556569b73aa0 0x556569b75e90 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.974+0000 7f9a9de0b740 1 --2- 172.21.15.28:0/2701431605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556569b63b30 0x556569b60c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.974+0000 7f9a9de0b740 1 --2- 172.21.15.28:0/2701431605 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556569b59460 0x556569b60730 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.974+0000 7f9a9de0b740 1 --2- 172.21.15.28:0/2701431605 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556569b58b40 0x556569b63440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:55.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.974+0000 7f9a9de0b740 1 -- 172.21.15.28:0/2701431605 >> 172.21.15.28:0/2701431605 conn(0x556569b54e60 msgr2=0x556569b5ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:55.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.974+0000 7f9a82000640 0 client.26653 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:55.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.974+0000 7f9a9de0b740 1 -- 172.21.15.28:0/2701431605 shutdown_connections 2024-04-23T14:52:55.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.974+0000 7f9a9de0b740 1 -- 172.21.15.28:0/2701431605 wait complete. 2024-04-23T14:52:55.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:55.975+0000 7f9a9de0b740 20 client.26653 trim_cache size 0 max 16384 2024-04-23T14:52:56.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.001+0000 7f9a9de0b740 1 Processor -- start 2024-04-23T14:52:56.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.002+0000 7f9a9de0b740 1 -- start start 2024-04-23T14:52:56.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.002+0000 7f9a9de0b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556569b58b40 0x556569b6f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:56.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.002+0000 7f9a9de0b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556569b59460 0x556569b5f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:56.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.002+0000 7f9a9de0b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556569b63b30 0x556569b62f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:56.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.002+0000 7f9a9de0b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9a941add50 con 0x556569b63b30 2024-04-23T14:52:56.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.002+0000 7f9a9de0b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9a941b05b0 con 0x556569b58b40 2024-04-23T14:52:56.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.002+0000 7f9a9de0b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9a941ae3a0 con 0x556569b59460 2024-04-23T14:52:56.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.002+0000 7f9a93e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556569b58b40 0x556569b6f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:56.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.002+0000 7f9a93e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556569b58b40 0x556569b6f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60450/0 (socket says 172.21.15.28:60450) 2024-04-23T14:52:56.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.002+0000 7f9a93e00640 1 -- 172.21.15.28:0/844675625 learned_addr learned my addr 172.21.15.28:0/844675625 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:56.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.002+0000 7f9a8b400640 1 --2- 172.21.15.28:0/844675625 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556569b59460 0x556569b5f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:56.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.002+0000 7f9a99000640 1 --2- 172.21.15.28:0/844675625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556569b63b30 0x556569b62f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:56.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.002+0000 7f9a93e00640 1 -- 172.21.15.28:0/844675625 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556569b59460 msgr2=0x556569b5f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:56.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.002+0000 7f9a93e00640 1 --2- 172.21.15.28:0/844675625 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556569b59460 0x556569b5f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:56.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.002+0000 7f9a93e00640 1 -- 172.21.15.28:0/844675625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556569b63b30 msgr2=0x556569b62f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:56.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.003+0000 7f9a93e00640 1 --2- 172.21.15.28:0/844675625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556569b63b30 0x556569b62f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:56.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.003+0000 7f9a93e00640 1 -- 172.21.15.28:0/844675625 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55656989a7d0 con 0x556569b58b40 2024-04-23T14:52:56.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.003+0000 7f9a93e00640 1 --2- 172.21.15.28:0/844675625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556569b58b40 0x556569b6f8c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9a840994e0 tx=0x7f9a8400cf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84ad531d0ba43105 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:56.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.003+0000 7f9a88c00640 1 -- 172.21.15.28:0/844675625 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8409d780 con 0x556569b58b40 2024-04-23T14:52:56.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.003+0000 7f9a88c00640 1 -- 172.21.15.28:0/844675625 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a8409d080 con 0x556569b58b40 2024-04-23T14:52:56.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.003+0000 7f9a88c00640 1 -- 172.21.15.28:0/844675625 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a84017e70 con 0x556569b58b40 2024-04-23T14:52:56.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.004+0000 7f9a9de0b740 1 -- 172.21.15.28:0/844675625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556569b58b40 msgr2=0x556569b6f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:56.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.004+0000 7f9a9de0b740 1 --2- 172.21.15.28:0/844675625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556569b58b40 0x556569b6f8c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9a840994e0 tx=0x7f9a8400cf10 comp rx=0 tx=0).stop 2024-04-23T14:52:56.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.004+0000 7f9a9de0b740 1 -- 172.21.15.28:0/844675625 shutdown_connections 2024-04-23T14:52:56.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.004+0000 7f9a9de0b740 1 --2- 172.21.15.28:0/844675625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556569b63b30 0x556569b62f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:56.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.004+0000 7f9a9de0b740 1 --2- 172.21.15.28:0/844675625 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556569b59460 0x556569b5f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:56.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.004+0000 7f9a9de0b740 1 --2- 172.21.15.28:0/844675625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556569b58b40 0x556569b6f8c0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:56.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.004+0000 7f9a9de0b740 1 -- 172.21.15.28:0/844675625 >> 172.21.15.28:0/844675625 conn(0x556569b54e60 msgr2=0x556569b5f000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:56.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.004+0000 7f9a9de0b740 1 -- 172.21.15.28:0/844675625 shutdown_connections 2024-04-23T14:52:56.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.004+0000 7f9a9de0b740 1 -- 172.21.15.28:0/844675625 wait complete. 2024-04-23T14:52:56.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.005+0000 7f9a9de0b740 1 -- start start 2024-04-23T14:52:56.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.005+0000 7f9a9de0b740 1 Processor -- start 2024-04-23T14:52:56.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.005+0000 7f9a9de0b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556569b58b40 0x556569b66f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:56.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.005+0000 7f9a9de0b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556569b59460 0x556569b674c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:56.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.005+0000 7f9a9de0b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556569b63b30 0x556569bd7de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:56.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.006+0000 7f9a9de0b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9a941aea70 con 0x556569b58b40 2024-04-23T14:52:56.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.006+0000 7f9a9de0b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9a941afee0 con 0x556569b59460 2024-04-23T14:52:56.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.006+0000 7f9a9de0b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9a941af140 con 0x556569b63b30 2024-04-23T14:52:56.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.006+0000 7f9a9de0b740 10 client.0 _ll_register_callbacks cb 0x7f9a8c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:52:56.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.006+0000 7f9a8b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556569b59460 0x556569b674c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:56.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.006+0000 7f9a8b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556569b59460 0x556569b674c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60452/0 (socket says 172.21.15.28:60452) 2024-04-23T14:52:56.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.006+0000 7f9a8b400640 1 -- 172.21.15.28:0/429371219 learned_addr learned my addr 172.21.15.28:0/429371219 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:56.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.006+0000 7f9a99000640 1 --2- 172.21.15.28:0/429371219 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556569b63b30 0x556569bd7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:56.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.006+0000 7f9a93e00640 1 --2- 172.21.15.28:0/429371219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556569b58b40 0x556569b66f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:56.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.006+0000 7f9a8b400640 1 -- 172.21.15.28:0/429371219 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556569b63b30 msgr2=0x556569bd7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:56.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.006+0000 7f9a8b400640 1 --2- 172.21.15.28:0/429371219 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556569b63b30 0x556569bd7de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:56.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.006+0000 7f9a8b400640 1 -- 172.21.15.28:0/429371219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556569b58b40 msgr2=0x556569b66f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:56.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.006+0000 7f9a8b400640 1 --2- 172.21.15.28:0/429371219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556569b58b40 0x556569b66f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:56.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.006+0000 7f9a8b400640 1 -- 172.21.15.28:0/429371219 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556569bd83a0 con 0x556569b59460 2024-04-23T14:52:56.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.006+0000 7f9a8b400640 1 --2- 172.21.15.28:0/429371219 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556569b59460 0x556569b674c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c006010 tx=0x7f9a7c004cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8dba60a3577d749 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:56.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.006+0000 7f9a82000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:52:56.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.007+0000 7f9a82000640 1 -- 172.21.15.28:0/429371219 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c035040 con 0x556569b59460 2024-04-23T14:52:56.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.007+0000 7f9a82000640 1 -- 172.21.15.28:0/429371219 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7f9a7c0051d0 con 0x556569b59460 2024-04-23T14:52:56.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.007+0000 7f9a9de0b740 1 -- 172.21.15.28:0/429371219 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a8409d080 con 0x556569b59460 2024-04-23T14:52:56.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.007+0000 7f9a82000640 1 -- 172.21.15.28:0/429371219 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f9a7c004490 con 0x556569b59460 2024-04-23T14:52:56.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.007+0000 7f9a9de0b740 10 client.26681 fetch_fsmap learned FSMap version 20 2024-04-23T14:52:56.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.007+0000 7f9a9de0b740 1 -- 172.21.15.28:0/429371219 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x556569bd9680 con 0x556569b59460 2024-04-23T14:52:56.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.007+0000 7f9a82000640 1 -- 172.21.15.28:0/429371219 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a8409d080 con 0x556569b59460 2024-04-23T14:52:56.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.007+0000 7f9a82000640 1 -- 172.21.15.28:0/429371219 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f9a7c003680 con 0x556569b59460 2024-04-23T14:52:56.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.007+0000 7f9a9de0b740 10 client.26681 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:52:56.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.007+0000 7f9a9de0b740 10 client.26681 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:52:56.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.007+0000 7f9a9de0b740 20 client.26681 populate_metadata read hostname 'smithi028' 2024-04-23T14:52:56.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.008+0000 7f9a9de0b740 1 --2- 172.21.15.28:0/429371219 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x556569bd9dd0 0x556569bdc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:56.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.008+0000 7f9a9de0b740 4 client.26681 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-23T14:52:56.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.008+0000 7f9a9de0b740 1 -- 172.21.15.28:0/429371219 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556569ac5a60 con 0x556569bd9dd0 2024-04-23T14:52:56.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.008+0000 7f9a93e00640 1 --2- 172.21.15.28:0/429371219 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x556569bd9dd0 0x556569bdc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:56.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.008+0000 7f9a93e00640 1 --2- 172.21.15.28:0/429371219 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x556569bd9dd0 0x556569bdc1f0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e02a377066a8c6d server_cookie=35714326681ad6bf in_seq=0 out_seq=0 2024-04-23T14:52:56.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.008+0000 7f9a82000640 10 client.26681 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:56.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.009+0000 7f9a82000640 1 -- 172.21.15.28:0/429371219 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556569ac5a60 con 0x556569bd9dd0 2024-04-23T14:52:56.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.009+0000 7f9a82000640 10 client.26681 handle_command_reply: tid=0 2024-04-23T14:52:56.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.009+0000 7f9a9de0b740 1 client.26681 shutdown 2024-04-23T14:52:56.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.009+0000 7f9a9de0b740 1 -- 172.21.15.28:0/429371219 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556569b59460 msgr2=0x556569b674c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:56.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.009+0000 7f9a9de0b740 1 --2- 172.21.15.28:0/429371219 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556569b59460 0x556569b674c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c006010 tx=0x7f9a7c004cb0 comp rx=0 tx=0).stop 2024-04-23T14:52:56.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.009+0000 7f9a9de0b740 1 -- 172.21.15.28:0/429371219 shutdown_connections 2024-04-23T14:52:56.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.009+0000 7f9a9de0b740 1 --2- 172.21.15.28:0/429371219 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x556569bd9dd0 0x556569bdc1f0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:56.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.009+0000 7f9a9de0b740 1 --2- 172.21.15.28:0/429371219 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556569b59460 0x556569b674c0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:56.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.009+0000 7f9a9de0b740 1 --2- 172.21.15.28:0/429371219 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556569b63b30 0x556569bd7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:56.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.009+0000 7f9a82000640 0 client.26681 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:56.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.009+0000 7f9a9de0b740 1 --2- 172.21.15.28:0/429371219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556569b58b40 0x556569b66f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:56.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.009+0000 7f9a9de0b740 1 -- 172.21.15.28:0/429371219 >> 172.21.15.28:0/429371219 conn(0x556569b54e60 msgr2=0x556569b536c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:56.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.009+0000 7f9a9de0b740 1 -- 172.21.15.28:0/429371219 shutdown_connections 2024-04-23T14:52:56.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.009+0000 7f9a9de0b740 1 -- 172.21.15.28:0/429371219 wait complete. 2024-04-23T14:52:56.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.010+0000 7f9a9de0b740 20 client.26681 trim_cache size 0 max 16384 2024-04-23T14:52:56.013 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-23T14:52:56.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.010+0000 7f9a99a00640 1 -- 172.21.15.28:0/1731888645 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f9a780a3e20 msgr2=0x7f9a780a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:56.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.010+0000 7f9a99a00640 1 --2- 172.21.15.28:0/1731888645 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f9a780a3e20 0x7f9a780a62e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c009e60 tx=0x7f9a8c005f90 comp rx=0 tx=0).stop 2024-04-23T14:52:56.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.010+0000 7f9a99a00640 1 -- 172.21.15.28:0/1731888645 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a94103060 msgr2=0x7f9a941a0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:56.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.010+0000 7f9a99a00640 1 --2- 172.21.15.28:0/1731888645 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a94103060 0x7f9a941a0680 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f9a8400eb00 tx=0x7f9a8400c2d0 comp rx=0 tx=0).stop 2024-04-23T14:52:56.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.010+0000 7f9a99a00640 1 -- 172.21.15.28:0/1731888645 shutdown_connections 2024-04-23T14:52:56.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.010+0000 7f9a99a00640 1 --2- 172.21.15.28:0/1731888645 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f9a780a3e20 0x7f9a780a62e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:56.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.010+0000 7f9a99a00640 1 --2- 172.21.15.28:0/1731888645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a94103980 0x7f9a941a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:56.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.010+0000 7f9a99a00640 1 --2- 172.21.15.28:0/1731888645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a9410c480 0x7f9a9419a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:56.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.010+0000 7f9a99a00640 1 --2- 172.21.15.28:0/1731888645 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a94103060 0x7f9a941a0680 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:56.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.010+0000 7f9a99a00640 1 -- 172.21.15.28:0/1731888645 >> 172.21.15.28:0/1731888645 conn(0x7f9a940fc880 msgr2=0x7f9a94100be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:56.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.010+0000 7f9a99a00640 1 -- 172.21.15.28:0/1731888645 shutdown_connections 2024-04-23T14:52:56.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:56.010+0000 7f9a99a00640 1 -- 172.21.15.28:0/1731888645 wait complete. 2024-04-23T14:52:56.026 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:55 smithi028 ceph-mon[27810]: scrub queued for path: / 2024-04-23T14:52:56.026 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:55 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:52:56.026 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:55 smithi028 ceph-mon[27810]: scrub summary: active paths [/] 2024-04-23T14:52:56.027 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:55 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:52:56.027 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:55 smithi028 ceph-mon[27810]: scrub summary: idle 2024-04-23T14:52:56.027 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:55 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/959608954' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:52:56.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:55 smithi105 ceph-mon[32555]: scrub queued for path: / 2024-04-23T14:52:56.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:55 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:52:56.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:55 smithi105 ceph-mon[32555]: scrub summary: active paths [/] 2024-04-23T14:52:56.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:55 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:52:56.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:55 smithi105 ceph-mon[32555]: scrub summary: idle 2024-04-23T14:52:56.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:55 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/959608954' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:52:56.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:55 smithi038 ceph-mon[32348]: scrub queued for path: / 2024-04-23T14:52:56.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:55 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:52:56.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:55 smithi038 ceph-mon[32348]: scrub summary: active paths [/] 2024-04-23T14:52:56.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:55 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:52:56.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:55 smithi038 ceph-mon[32348]: scrub summary: idle 2024-04-23T14:52:56.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:55 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/959608954' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:52:56.515 INFO:tasks.workunit.client.0.smithi028.stdout: 1 749590 8.96 MB/sec execute 312 sec latency 21.615 ms 2024-04-23T14:52:57.032 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-23T14:52:57.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:52:57.032 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-23T14:52:57.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:56 smithi105 ceph-mon[32555]: pgmap v274: 97 pgs: 97 active+clean; 454 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 138 op/s 2024-04-23T14:52:57.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:56 smithi028 ceph-mon[27810]: pgmap v274: 97 pgs: 97 active+clean; 454 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 138 op/s 2024-04-23T14:52:57.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:56 smithi038 ceph-mon[32348]: pgmap v274: 97 pgs: 97 active+clean; 454 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 138 op/s 2024-04-23T14:52:57.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.248+0000 7f522f000640 1 -- 172.21.15.28:0/795214956 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52281071e0 msgr2=0x7f522810edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:57.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.248+0000 7f522f000640 1 --2- 172.21.15.28:0/795214956 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52281071e0 0x7f522810edb0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f5220009ca0 tx=0x7f522002a8c0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.248+0000 7f522f000640 1 -- 172.21.15.28:0/795214956 shutdown_connections 2024-04-23T14:52:57.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.248+0000 7f522f000640 1 --2- 172.21.15.28:0/795214956 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52281071e0 0x7f522810edb0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.248+0000 7f522f000640 1 --2- 172.21.15.28:0/795214956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5228102ac0 0x7f5228106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.248+0000 7f522f000640 1 --2- 172.21.15.28:0/795214956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5228102110 0x7f52281024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.248+0000 7f522f000640 1 -- 172.21.15.28:0/795214956 >> 172.21.15.28:0/795214956 conn(0x7f52280fd9d0 msgr2=0x7f52280ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:57.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.248+0000 7f522f000640 1 -- 172.21.15.28:0/795214956 shutdown_connections 2024-04-23T14:52:57.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.248+0000 7f522f000640 1 -- 172.21.15.28:0/795214956 wait complete. 2024-04-23T14:52:57.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.249+0000 7f522f000640 1 Processor -- start 2024-04-23T14:52:57.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.250+0000 7f522f000640 1 -- start start 2024-04-23T14:52:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.250+0000 7f522f000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5228102110 0x7f52281a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.250+0000 7f522f000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52281071e0 0x7f52281a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.250+0000 7f522f000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f522819a780 0x7f522819ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.250+0000 7f522f000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f52281122a0 con 0x7f52281071e0 2024-04-23T14:52:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.250+0000 7f522f000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5228112120 con 0x7f5228102110 2024-04-23T14:52:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.250+0000 7f522f000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5228112420 con 0x7f522819a780 2024-04-23T14:52:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.250+0000 7f522d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52281071e0 0x7f52281a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.250+0000 7f522dc00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5228102110 0x7f52281a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.250+0000 7f522d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52281071e0 0x7f52281a0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44610/0 (socket says 172.21.15.28:44610) 2024-04-23T14:52:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.250+0000 7f522d200640 1 -- 172.21.15.28:0/3813658669 learned_addr learned my addr 172.21.15.28:0/3813658669 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.251+0000 7f522e600640 1 --2- 172.21.15.28:0/3813658669 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f522819a780 0x7f522819ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.251+0000 7f522d200640 1 -- 172.21.15.28:0/3813658669 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f522819a780 msgr2=0x7f522819ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.251+0000 7f522d200640 1 --2- 172.21.15.28:0/3813658669 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f522819a780 0x7f522819ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.251+0000 7f522d200640 1 -- 172.21.15.28:0/3813658669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5228102110 msgr2=0x7f52281a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:57.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.251+0000 7f522d200640 1 --2- 172.21.15.28:0/3813658669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5228102110 0x7f52281a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.251+0000 7f522d200640 1 -- 172.21.15.28:0/3813658669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f522819b310 con 0x7f52281071e0 2024-04-23T14:52:57.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.251+0000 7f522dc00640 1 --2- 172.21.15.28:0/3813658669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5228102110 0x7f52281a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:52:57.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.251+0000 7f522d200640 1 --2- 172.21.15.28:0/3813658669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52281071e0 0x7f52281a0b50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f521400e9f0 tx=0x7f521400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:57.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.252+0000 7f5226000640 1 -- 172.21.15.28:0/3813658669 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521400ccc0 con 0x7f52281071e0 2024-04-23T14:52:57.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.252+0000 7f5226000640 1 -- 172.21.15.28:0/3813658669 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5214005500 con 0x7f52281071e0 2024-04-23T14:52:57.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.252+0000 7f5226000640 1 -- 172.21.15.28:0/3813658669 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5214010690 con 0x7f52281071e0 2024-04-23T14:52:57.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.252+0000 7f522f000640 1 -- 172.21.15.28:0/3813658669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f522819b600 con 0x7f52281071e0 2024-04-23T14:52:57.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.252+0000 7f522f000640 1 -- 172.21.15.28:0/3813658669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52281a7790 con 0x7f52281071e0 2024-04-23T14:52:57.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.256+0000 7f523358d740 1 Processor -- start 2024-04-23T14:52:57.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.257+0000 7f523358d740 1 -- start start 2024-04-23T14:52:57.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.257+0000 7f5226000640 1 -- 172.21.15.28:0/3813658669 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f52140046c0 con 0x7f52281071e0 2024-04-23T14:52:57.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.257+0000 7f523358d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564de9822270 0x564de9822650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.257+0000 7f523358d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564de9822b90 0x564de982cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.257+0000 7f523358d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564de982d260 0x564de982f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.257+0000 7f523358d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564de9787e70 con 0x564de9822270 2024-04-23T14:52:57.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.257+0000 7f523358d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564de972e380 con 0x564de982d260 2024-04-23T14:52:57.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.257+0000 7f523358d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x564de9783e30 con 0x564de9822b90 2024-04-23T14:52:57.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.257+0000 7f522dc00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564de9822b90 0x564de982cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.257+0000 7f522e600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564de9822270 0x564de9822650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.257+0000 7f522dc00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564de9822b90 0x564de982cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:46222/0 (socket says 172.21.15.28:46222) 2024-04-23T14:52:57.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.257+0000 7f522dc00640 1 -- 172.21.15.28:0/1975012080 learned_addr learned my addr 172.21.15.28:0/1975012080 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:57.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.257+0000 7f522d200640 1 --2- 172.21.15.28:0/1975012080 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564de982d260 0x564de982f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.258+0000 7f522dc00640 1 -- 172.21.15.28:0/1975012080 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564de982d260 msgr2=0x564de982f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:57.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.258+0000 7f522dc00640 1 --2- 172.21.15.28:0/1975012080 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564de982d260 0x564de982f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.258+0000 7f522dc00640 1 -- 172.21.15.28:0/1975012080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564de9822270 msgr2=0x564de9822650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:57.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.258+0000 7f522dc00640 1 --2- 172.21.15.28:0/1975012080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564de9822270 0x564de9822650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.258+0000 7f522dc00640 1 -- 172.21.15.28:0/1975012080 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564de95627e0 con 0x564de9822b90 2024-04-23T14:52:57.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.258+0000 7f522d200640 1 --2- 172.21.15.28:0/1975012080 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564de982d260 0x564de982f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:52:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.258+0000 7f522dc00640 1 --2- 172.21.15.28:0/1975012080 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564de9822b90 0x564de982cd20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f521c010480 tx=0x7f521c010940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bd1a5af52651e9b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.259+0000 7f521aa00640 1 -- 172.21.15.28:0/1975012080 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521c011760 con 0x564de9822b90 2024-04-23T14:52:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.259+0000 7f521aa00640 1 -- 172.21.15.28:0/1975012080 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f521c011de0 con 0x564de9822b90 2024-04-23T14:52:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.259+0000 7f521aa00640 1 -- 172.21.15.28:0/1975012080 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521c00e430 con 0x564de9822b90 2024-04-23T14:52:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.259+0000 7f5226000640 1 --2- 172.21.15.28:0/3813658669 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f52100a3da0 0x7f52100a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.259+0000 7f5226000640 1 -- 172.21.15.28:0/3813658669 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7f52140d0ae0 con 0x7f52281071e0 2024-04-23T14:52:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.259+0000 7f522e600640 1 --2- 172.21.15.28:0/3813658669 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f52100a3da0 0x7f52100a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.260+0000 7f523358d740 1 -- 172.21.15.28:0/1975012080 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564de9822b90 msgr2=0x564de982cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.260+0000 7f523358d740 1 --2- 172.21.15.28:0/1975012080 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564de9822b90 0x564de982cd20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f521c010480 tx=0x7f521c010940 comp rx=0 tx=0).stop 2024-04-23T14:52:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.260+0000 7f523358d740 1 -- 172.21.15.28:0/1975012080 shutdown_connections 2024-04-23T14:52:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.260+0000 7f523358d740 1 --2- 172.21.15.28:0/1975012080 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564de982d260 0x564de982f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.260+0000 7f523358d740 1 --2- 172.21.15.28:0/1975012080 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564de9822b90 0x564de982cd20 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.260+0000 7f523358d740 1 --2- 172.21.15.28:0/1975012080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564de9822270 0x564de9822650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.260+0000 7f523358d740 1 -- 172.21.15.28:0/1975012080 >> 172.21.15.28:0/1975012080 conn(0x564de981e590 msgr2=0x564de981e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:57.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.260+0000 7f522e600640 1 --2- 172.21.15.28:0/3813658669 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f52100a3da0 0x7f52100a6260 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5220004af0 tx=0x7f5220004a80 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:57.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.260+0000 7f523358d740 1 -- 172.21.15.28:0/1975012080 shutdown_connections 2024-04-23T14:52:57.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.260+0000 7f523358d740 1 -- 172.21.15.28:0/1975012080 wait complete. 2024-04-23T14:52:57.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.261+0000 7f523358d740 1 -- start start 2024-04-23T14:52:57.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.261+0000 7f523358d740 1 Processor -- start 2024-04-23T14:52:57.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.261+0000 7f523358d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564de9822270 0x564de982a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.261+0000 7f523358d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564de9822b90 0x564de982a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.261+0000 7f523358d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564de982d260 0x564de982acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.261+0000 7f523358d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564de9830050 con 0x564de9822b90 2024-04-23T14:52:57.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.261+0000 7f523358d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564de9741660 con 0x564de982d260 2024-04-23T14:52:57.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.261+0000 7f523358d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x564de98301d0 con 0x564de9822270 2024-04-23T14:52:57.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.262+0000 7f522dc00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564de9822270 0x564de982a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.262+0000 7f523358d740 10 client.0 _ll_register_callbacks cb 0x564de972faa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:52:57.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.262+0000 7f522dc00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564de9822270 0x564de982a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:46234/0 (socket says 172.21.15.28:46234) 2024-04-23T14:52:57.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.262+0000 7f522dc00640 1 -- 172.21.15.28:0/1425820370 learned_addr learned my addr 172.21.15.28:0/1425820370 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:57.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.262+0000 7f522e600640 1 --2- 172.21.15.28:0/1425820370 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564de982d260 0x564de982acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.262+0000 7f522d200640 1 --2- 172.21.15.28:0/1425820370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564de9822b90 0x564de982a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.262+0000 7f522dc00640 1 -- 172.21.15.28:0/1425820370 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564de982d260 msgr2=0x564de982acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:57.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.262+0000 7f522dc00640 1 --2- 172.21.15.28:0/1425820370 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564de982d260 0x564de982acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.262+0000 7f522dc00640 1 -- 172.21.15.28:0/1425820370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564de9822b90 msgr2=0x564de982a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:57.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.262+0000 7f522dc00640 1 --2- 172.21.15.28:0/1425820370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564de9822b90 0x564de982a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.262+0000 7f522dc00640 1 -- 172.21.15.28:0/1425820370 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564de95c2f70 con 0x564de9822270 2024-04-23T14:52:57.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.262+0000 7f522dc00640 1 --2- 172.21.15.28:0/1425820370 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564de9822270 0x564de982a240 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f521c01c6b0 tx=0x7f521c01c6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91f634b8ae280818 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:57.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.262+0000 7f520f400640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:52:57.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.263+0000 7f520f400640 1 -- 172.21.15.28:0/1425820370 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521c011970 con 0x564de9822270 2024-04-23T14:52:57.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.263+0000 7f523358d740 1 -- 172.21.15.28:0/1425820370 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f521c011de0 con 0x564de9822270 2024-04-23T14:52:57.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.263+0000 7f520f400640 1 -- 172.21.15.28:0/1425820370 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7f521c011b10 con 0x564de9822270 2024-04-23T14:52:57.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.263+0000 7f520f400640 1 -- 172.21.15.28:0/1425820370 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f521c032050 con 0x564de9822270 2024-04-23T14:52:57.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.263+0000 7f523358d740 10 client.26671 fetch_fsmap learned FSMap version 20 2024-04-23T14:52:57.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.263+0000 7f523358d740 1 -- 172.21.15.28:0/1425820370 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x564de945a170 con 0x564de9822270 2024-04-23T14:52:57.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.263+0000 7f520f400640 1 -- 172.21.15.28:0/1425820370 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f521c011de0 con 0x564de9822270 2024-04-23T14:52:57.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.263+0000 7f520f400640 1 -- 172.21.15.28:0/1425820370 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f521c02d030 con 0x564de9822270 2024-04-23T14:52:57.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.263+0000 7f523358d740 10 client.26671 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:52:57.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.263+0000 7f523358d740 10 client.26671 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:52:57.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.264+0000 7f523358d740 20 client.26671 populate_metadata read hostname 'smithi028' 2024-04-23T14:52:57.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.264+0000 7f523358d740 1 --2- 172.21.15.28:0/1425820370 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564de9845de0 0x564de98481d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.264+0000 7f523358d740 4 client.26671 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:52:57.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.264+0000 7f523358d740 1 -- 172.21.15.28:0/1425820370 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564de9795890 con 0x564de9845de0 2024-04-23T14:52:57.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.264+0000 7f522d200640 1 --2- 172.21.15.28:0/1425820370 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564de9845de0 0x564de98481d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.264+0000 7f522d200640 1 --2- 172.21.15.28:0/1425820370 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564de9845de0 0x564de98481d0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddf667604023b790 server_cookie=9fbe98ed355c2c8e in_seq=0 out_seq=0 2024-04-23T14:52:57.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.264+0000 7f520f400640 10 client.26671 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:57.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.265+0000 7f520f400640 1 -- 172.21.15.28:0/1425820370 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564de9795890 con 0x564de9845de0 2024-04-23T14:52:57.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.265+0000 7f520f400640 10 client.26671 handle_command_reply: tid=0 2024-04-23T14:52:57.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.265+0000 7f523358d740 1 client.26671 shutdown 2024-04-23T14:52:57.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.266+0000 7f523358d740 1 -- 172.21.15.28:0/1425820370 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564de9822270 msgr2=0x564de982a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:57.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.266+0000 7f523358d740 1 --2- 172.21.15.28:0/1425820370 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564de9822270 0x564de982a240 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f521c01c6b0 tx=0x7f521c01c6e0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.266+0000 7f523358d740 1 -- 172.21.15.28:0/1425820370 shutdown_connections 2024-04-23T14:52:57.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.266+0000 7f523358d740 1 --2- 172.21.15.28:0/1425820370 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564de9845de0 0x564de98481d0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.266+0000 7f523358d740 1 --2- 172.21.15.28:0/1425820370 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564de982d260 0x564de982acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.266+0000 7f523358d740 1 --2- 172.21.15.28:0/1425820370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564de9822b90 0x564de982a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.266+0000 7f523358d740 1 --2- 172.21.15.28:0/1425820370 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564de9822270 0x564de982a240 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.266+0000 7f523358d740 1 -- 172.21.15.28:0/1425820370 >> 172.21.15.28:0/1425820370 conn(0x564de981e590 msgr2=0x564de9824390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:57.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.266+0000 7f520f400640 0 client.26671 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:57.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.266+0000 7f523358d740 1 -- 172.21.15.28:0/1425820370 shutdown_connections 2024-04-23T14:52:57.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.266+0000 7f523358d740 1 -- 172.21.15.28:0/1425820370 wait complete. 2024-04-23T14:52:57.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.266+0000 7f523358d740 20 client.26671 trim_cache size 0 max 16384 2024-04-23T14:52:57.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.286+0000 7f523358d740 1 Processor -- start 2024-04-23T14:52:57.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.286+0000 7f523358d740 1 -- start start 2024-04-23T14:52:57.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.286+0000 7f523358d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564de9822270 0x564de98417a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.286+0000 7f523358d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564de9822b90 0x564de9824a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.286+0000 7f523358d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564de982d260 0x564de984d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.286+0000 7f523358d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f52281a9a70 con 0x564de9822b90 2024-04-23T14:52:57.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.286+0000 7f523358d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f52281ac290 con 0x564de9822270 2024-04-23T14:52:57.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.286+0000 7f523358d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f52281aa0c0 con 0x564de982d260 2024-04-23T14:52:57.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.286+0000 7f522dc00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564de9822270 0x564de98417a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.286+0000 7f522d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564de9822b90 0x564de9824a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.286+0000 7f522dc00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564de9822270 0x564de98417a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60504/0 (socket says 172.21.15.28:60504) 2024-04-23T14:52:57.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.286+0000 7f522d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564de9822b90 0x564de9824a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44634/0 (socket says 172.21.15.28:44634) 2024-04-23T14:52:57.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.286+0000 7f522dc00640 1 -- 172.21.15.28:0/1047149491 learned_addr learned my addr 172.21.15.28:0/1047149491 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:57.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.286+0000 7f522e600640 1 --2- 172.21.15.28:0/1047149491 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564de982d260 0x564de984d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.287+0000 7f522d200640 1 -- 172.21.15.28:0/1047149491 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564de982d260 msgr2=0x564de984d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:57.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.287+0000 7f522d200640 1 --2- 172.21.15.28:0/1047149491 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564de982d260 0x564de984d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.287+0000 7f522d200640 1 -- 172.21.15.28:0/1047149491 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564de9822270 msgr2=0x564de98417a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:57.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.287+0000 7f522d200640 1 --2- 172.21.15.28:0/1047149491 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564de9822270 0x564de98417a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.287+0000 7f522d200640 1 -- 172.21.15.28:0/1047149491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564de981ef10 con 0x564de9822b90 2024-04-23T14:52:57.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.287+0000 7f522d200640 1 --2- 172.21.15.28:0/1047149491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564de9822b90 0x564de9824a10 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f52140ce1c0 tx=0x7f52140d2f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7387ffd67d3c7641 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:57.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.287+0000 7f521a000640 1 -- 172.21.15.28:0/1047149491 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521409c550 con 0x564de9822b90 2024-04-23T14:52:57.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.287+0000 7f521a000640 1 -- 172.21.15.28:0/1047149491 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f521409d110 con 0x564de9822b90 2024-04-23T14:52:57.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.287+0000 7f521a000640 1 -- 172.21.15.28:0/1047149491 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52140056b0 con 0x564de9822b90 2024-04-23T14:52:57.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.288+0000 7f523358d740 1 -- 172.21.15.28:0/1047149491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564de9822b90 msgr2=0x564de9824a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:57.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.288+0000 7f523358d740 1 --2- 172.21.15.28:0/1047149491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564de9822b90 0x564de9824a10 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f52140ce1c0 tx=0x7f52140d2f30 comp rx=0 tx=0).stop 2024-04-23T14:52:57.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.288+0000 7f523358d740 1 -- 172.21.15.28:0/1047149491 shutdown_connections 2024-04-23T14:52:57.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.288+0000 7f523358d740 1 --2- 172.21.15.28:0/1047149491 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564de982d260 0x564de984d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.288+0000 7f523358d740 1 --2- 172.21.15.28:0/1047149491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564de9822b90 0x564de9824a10 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.288+0000 7f523358d740 1 --2- 172.21.15.28:0/1047149491 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564de9822270 0x564de98417a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.288+0000 7f523358d740 1 -- 172.21.15.28:0/1047149491 >> 172.21.15.28:0/1047149491 conn(0x564de981e590 msgr2=0x564de981d6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:57.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.288+0000 7f523358d740 1 -- 172.21.15.28:0/1047149491 shutdown_connections 2024-04-23T14:52:57.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.288+0000 7f523358d740 1 -- 172.21.15.28:0/1047149491 wait complete. 2024-04-23T14:52:57.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.289+0000 7f523358d740 1 -- start start 2024-04-23T14:52:57.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.289+0000 7f523358d740 1 Processor -- start 2024-04-23T14:52:57.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.289+0000 7f523358d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564de9822270 0x564de983dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.289+0000 7f523358d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564de9822b90 0x564de983e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.289+0000 7f523358d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564de982d260 0x564de983ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.289+0000 7f523358d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f52281aa750 con 0x564de9822b90 2024-04-23T14:52:57.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.289+0000 7f523358d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f52281abbc0 con 0x564de982d260 2024-04-23T14:52:57.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.289+0000 7f523358d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f52281aae20 con 0x564de9822270 2024-04-23T14:52:57.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.289+0000 7f523358d740 10 client.0 _ll_register_callbacks cb 0x7f5220009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:52:57.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.290+0000 7f522dc00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564de9822270 0x564de983dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.290+0000 7f522dc00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564de9822270 0x564de983dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:46242/0 (socket says 172.21.15.28:46242) 2024-04-23T14:52:57.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.290+0000 7f522dc00640 1 -- 172.21.15.28:0/2147028626 learned_addr learned my addr 172.21.15.28:0/2147028626 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:57.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.290+0000 7f522d200640 1 --2- 172.21.15.28:0/2147028626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564de9822b90 0x564de983e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.290+0000 7f522e600640 1 --2- 172.21.15.28:0/2147028626 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564de982d260 0x564de983ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.290+0000 7f522dc00640 1 -- 172.21.15.28:0/2147028626 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564de982d260 msgr2=0x564de983ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:57.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.290+0000 7f522dc00640 1 --2- 172.21.15.28:0/2147028626 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564de982d260 0x564de983ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.290+0000 7f522dc00640 1 -- 172.21.15.28:0/2147028626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564de9822b90 msgr2=0x564de983e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:57.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.290+0000 7f522dc00640 1 --2- 172.21.15.28:0/2147028626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564de9822b90 0x564de983e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.290+0000 7f522dc00640 1 -- 172.21.15.28:0/2147028626 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564de983b2f0 con 0x564de9822270 2024-04-23T14:52:57.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.290+0000 7f522dc00640 1 --2- 172.21.15.28:0/2147028626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564de9822270 0x564de983dcb0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f521c005bc0 tx=0x7f521c00e530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec8437fe06e2ee2a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:57.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.290+0000 7f520f400640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:52:57.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.291+0000 7f520f400640 1 -- 172.21.15.28:0/2147028626 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521c01c950 con 0x564de9822270 2024-04-23T14:52:57.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.291+0000 7f520f400640 1 -- 172.21.15.28:0/2147028626 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7f521c00e930 con 0x564de9822270 2024-04-23T14:52:57.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.291+0000 7f523358d740 1 -- 172.21.15.28:0/2147028626 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f521409d110 con 0x564de9822270 2024-04-23T14:52:57.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.291+0000 7f520f400640 1 -- 172.21.15.28:0/2147028626 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f521c01caf0 con 0x564de9822270 2024-04-23T14:52:57.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.291+0000 7f523358d740 10 client.26677 fetch_fsmap learned FSMap version 20 2024-04-23T14:52:57.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.291+0000 7f523358d740 1 -- 172.21.15.28:0/2147028626 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x564de98ac990 con 0x564de9822270 2024-04-23T14:52:57.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.291+0000 7f520f400640 1 -- 172.21.15.28:0/2147028626 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f521409d110 con 0x564de9822270 2024-04-23T14:52:57.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.291+0000 7f520f400640 1 -- 172.21.15.28:0/2147028626 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f521c03d020 con 0x564de9822270 2024-04-23T14:52:57.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.291+0000 7f523358d740 10 client.26677 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:52:57.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.291+0000 7f523358d740 10 client.26677 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:52:57.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.291+0000 7f523358d740 20 client.26677 populate_metadata read hostname 'smithi028' 2024-04-23T14:52:57.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.292+0000 7f523358d740 1 --2- 172.21.15.28:0/2147028626 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564de98ad080 0x564de98af4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.292+0000 7f523358d740 4 client.26677 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-23T14:52:57.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.292+0000 7f523358d740 1 -- 172.21.15.28:0/2147028626 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564de978f700 con 0x564de98ad080 2024-04-23T14:52:57.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.292+0000 7f522d200640 1 --2- 172.21.15.28:0/2147028626 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564de98ad080 0x564de98af4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.292+0000 7f522d200640 1 --2- 172.21.15.28:0/2147028626 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564de98ad080 0x564de98af4a0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a7994871e2704f3 server_cookie=644cbd314300724a in_seq=0 out_seq=0 2024-04-23T14:52:57.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.292+0000 7f520f400640 10 client.26677 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:57.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.294+0000 7f520f400640 1 -- 172.21.15.28:0/2147028626 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564de978f700 con 0x564de98ad080 2024-04-23T14:52:57.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.294+0000 7f520f400640 10 client.26677 handle_command_reply: tid=0 2024-04-23T14:52:57.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.294+0000 7f523358d740 1 client.26677 shutdown 2024-04-23T14:52:57.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.294+0000 7f523358d740 1 -- 172.21.15.28:0/2147028626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564de9822270 msgr2=0x564de983dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:57.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.294+0000 7f523358d740 1 --2- 172.21.15.28:0/2147028626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564de9822270 0x564de983dcb0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f521c005bc0 tx=0x7f521c00e530 comp rx=0 tx=0).stop 2024-04-23T14:52:57.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.294+0000 7f523358d740 1 -- 172.21.15.28:0/2147028626 shutdown_connections 2024-04-23T14:52:57.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.294+0000 7f523358d740 1 --2- 172.21.15.28:0/2147028626 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564de98ad080 0x564de98af4a0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.294+0000 7f523358d740 1 --2- 172.21.15.28:0/2147028626 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564de982d260 0x564de983ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.294+0000 7f523358d740 1 --2- 172.21.15.28:0/2147028626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564de9822b90 0x564de983e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.294+0000 7f523358d740 1 --2- 172.21.15.28:0/2147028626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564de9822270 0x564de983dcb0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.294+0000 7f523358d740 1 -- 172.21.15.28:0/2147028626 >> 172.21.15.28:0/2147028626 conn(0x564de981e590 msgr2=0x564de982ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:57.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.294+0000 7f520f400640 0 client.26677 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:57.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.294+0000 7f523358d740 1 -- 172.21.15.28:0/2147028626 shutdown_connections 2024-04-23T14:52:57.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.294+0000 7f523358d740 1 -- 172.21.15.28:0/2147028626 wait complete. 2024-04-23T14:52:57.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.294+0000 7f523358d740 20 client.26677 trim_cache size 0 max 16384 2024-04-23T14:52:57.299 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:52:57.299 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-23T14:52:57.299 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "b8ffc275-4bd4-44c3-a226-fc17554a5e95", 2024-04-23T14:52:57.299 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-23T14:52:57.300 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:52:57.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.295+0000 7f522f000640 1 -- 172.21.15.28:0/3813658669 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f52100a3da0 msgr2=0x7f52100a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:57.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.295+0000 7f522f000640 1 --2- 172.21.15.28:0/3813658669 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f52100a3da0 0x7f52100a6260 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5220004af0 tx=0x7f5220004a80 comp rx=0 tx=0).stop 2024-04-23T14:52:57.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.295+0000 7f522f000640 1 -- 172.21.15.28:0/3813658669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52281071e0 msgr2=0x7f52281a0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:57.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.295+0000 7f522f000640 1 --2- 172.21.15.28:0/3813658669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52281071e0 0x7f52281a0b50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f521400e9f0 tx=0x7f521400eeb0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.295+0000 7f522f000640 1 -- 172.21.15.28:0/3813658669 shutdown_connections 2024-04-23T14:52:57.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.295+0000 7f522f000640 1 --2- 172.21.15.28:0/3813658669 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f52100a3da0 0x7f52100a6260 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.295+0000 7f522f000640 1 --2- 172.21.15.28:0/3813658669 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f522819a780 0x7f522819ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.295+0000 7f522f000640 1 --2- 172.21.15.28:0/3813658669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52281071e0 0x7f52281a0b50 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.295+0000 7f522f000640 1 --2- 172.21.15.28:0/3813658669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5228102110 0x7f52281a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.295+0000 7f522f000640 1 -- 172.21.15.28:0/3813658669 >> 172.21.15.28:0/3813658669 conn(0x7f52280fd9d0 msgr2=0x7f52280ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:57.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.295+0000 7f522f000640 1 -- 172.21.15.28:0/3813658669 shutdown_connections 2024-04-23T14:52:57.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.295+0000 7f522f000640 1 -- 172.21.15.28:0/3813658669 wait complete. 2024-04-23T14:52:57.315 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:52:57.315 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:52:57.516 INFO:tasks.workunit.client.0.smithi028.stdout: 1 751014 8.95 MB/sec execute 313 sec latency 17.266 ms 2024-04-23T14:52:57.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.522+0000 7fc3cda00640 1 -- 172.21.15.28:0/1537714841 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3c8108610 msgr2=0x7fc3c8108a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:57.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.522+0000 7fc3cda00640 1 --2- 172.21.15.28:0/1537714841 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3c8108610 0x7fc3c8108a90 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fc3b0009900 tx=0x7fc3b002a400 comp rx=0 tx=0).stop 2024-04-23T14:52:57.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.522+0000 7fc3cda00640 1 -- 172.21.15.28:0/1537714841 shutdown_connections 2024-04-23T14:52:57.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.522+0000 7fc3cda00640 1 --2- 172.21.15.28:0/1537714841 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3c8108610 0x7fc3c8108a90 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.522+0000 7fc3cda00640 1 --2- 172.21.15.28:0/1537714841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3c8108fd0 0x7fc3c8115490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.522+0000 7fc3cda00640 1 --2- 172.21.15.28:0/1537714841 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3c810e660 0x7fc3c810ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.522+0000 7fc3cda00640 1 -- 172.21.15.28:0/1537714841 >> 172.21.15.28:0/1537714841 conn(0x7fc3c8076540 msgr2=0x7fc3c8078960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:57.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.523+0000 7fc3cda00640 1 -- 172.21.15.28:0/1537714841 shutdown_connections 2024-04-23T14:52:57.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.523+0000 7fc3cda00640 1 -- 172.21.15.28:0/1537714841 wait complete. 2024-04-23T14:52:57.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.524+0000 7fc3cda00640 1 Processor -- start 2024-04-23T14:52:57.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.524+0000 7fc3cda00640 1 -- start start 2024-04-23T14:52:57.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.524+0000 7fc3cda00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3c8108610 0x7fc3c819ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.524+0000 7fc3cda00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3c8108fd0 0x7fc3c819f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.524+0000 7fc3cda00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3c810e660 0x7fc3c81a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.524+0000 7fc3cda00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c81189f0 con 0x7fc3c8108610 2024-04-23T14:52:57.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.524+0000 7fc3cda00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c8118870 con 0x7fc3c810e660 2024-04-23T14:52:57.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.524+0000 7fc3cda00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c8118b70 con 0x7fc3c8108fd0 2024-04-23T14:52:57.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.525+0000 7fc3bfe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3c8108fd0 0x7fc3c819f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.525+0000 7fc3c6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3c8108610 0x7fc3c819ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.525+0000 7fc3c7400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3c810e660 0x7fc3c81a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.525+0000 7fc3bfe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3c8108fd0 0x7fc3c819f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:46258/0 (socket says 172.21.15.28:46258) 2024-04-23T14:52:57.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.525+0000 7fc3bfe00640 1 -- 172.21.15.28:0/1167821927 learned_addr learned my addr 172.21.15.28:0/1167821927 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:57.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.525+0000 7fc3c7400640 1 -- 172.21.15.28:0/1167821927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3c8108fd0 msgr2=0x7fc3c819f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:57.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.525+0000 7fc3c7400640 1 --2- 172.21.15.28:0/1167821927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3c8108fd0 0x7fc3c819f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.525+0000 7fc3c7400640 1 -- 172.21.15.28:0/1167821927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3c8108610 msgr2=0x7fc3c819ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:57.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.525+0000 7fc3c7400640 1 --2- 172.21.15.28:0/1167821927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3c8108610 0x7fc3c819ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.525+0000 7fc3c7400640 1 -- 172.21.15.28:0/1167821927 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3c81a3e40 con 0x7fc3c810e660 2024-04-23T14:52:57.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.525+0000 7fc3c7400640 1 --2- 172.21.15.28:0/1167821927 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3c810e660 0x7fc3c81a3710 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc3c0009fd0 tx=0x7fc3c000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:57.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.526+0000 7fc3bf400640 1 -- 172.21.15.28:0/1167821927 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c000ce20 con 0x7fc3c810e660 2024-04-23T14:52:57.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.526+0000 7fc3bf400640 1 -- 172.21.15.28:0/1167821927 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3c0005020 con 0x7fc3c810e660 2024-04-23T14:52:57.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.526+0000 7fc3cda00640 1 -- 172.21.15.28:0/1167821927 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3c81a4130 con 0x7fc3c810e660 2024-04-23T14:52:57.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.526+0000 7fc3bf400640 1 -- 172.21.15.28:0/1167821927 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c0018760 con 0x7fc3c810e660 2024-04-23T14:52:57.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.526+0000 7fc3cda00640 1 -- 172.21.15.28:0/1167821927 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3c81ab9c0 con 0x7fc3c810e660 2024-04-23T14:52:57.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.529+0000 7fc3bf400640 1 -- 172.21.15.28:0/1167821927 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fc3c00051d0 con 0x7fc3c810e660 2024-04-23T14:52:57.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.529+0000 7fc3d0815740 1 Processor -- start 2024-04-23T14:52:57.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.530+0000 7fc3d0815740 1 -- start start 2024-04-23T14:52:57.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.530+0000 7fc3d0815740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564619fcc860 0x564619fccc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.530+0000 7fc3d0815740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564619fcd180 0x564619fd7310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.530+0000 7fc3d0815740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564619fd7850 0x564619fd9c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.530+0000 7fc3d0815740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564619f321f0 con 0x564619fd7850 2024-04-23T14:52:57.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.530+0000 7fc3d0815740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564619ed8350 con 0x564619fcd180 2024-04-23T14:52:57.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.530+0000 7fc3d0815740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x564619f2e1b0 con 0x564619fcc860 2024-04-23T14:52:57.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.530+0000 7fc3c6a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564619fcd180 0x564619fd7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.530+0000 7fc3bfe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564619fcc860 0x564619fccc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.530+0000 7fc3c6a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564619fcd180 0x564619fd7310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60540/0 (socket says 172.21.15.28:60540) 2024-04-23T14:52:57.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.530+0000 7fc3c6a00640 1 -- 172.21.15.28:0/3356352853 learned_addr learned my addr 172.21.15.28:0/3356352853 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:57.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.530+0000 7fc3bfe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564619fcc860 0x564619fccc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:46268/0 (socket says 172.21.15.28:46268) 2024-04-23T14:52:57.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.530+0000 7fc3bfe00640 1 --2- 172.21.15.28:0/3356352853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564619fd7850 0x564619fd9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.531+0000 7fc3c6a00640 1 -- 172.21.15.28:0/3356352853 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564619fcc860 msgr2=0x564619fccc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:57.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.531+0000 7fc3c6a00640 1 --2- 172.21.15.28:0/3356352853 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564619fcc860 0x564619fccc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.531+0000 7fc3c6a00640 1 -- 172.21.15.28:0/3356352853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564619fd7850 msgr2=0x564619fd9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:57.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.531+0000 7fc3c6a00640 1 --2- 172.21.15.28:0/3356352853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564619fd7850 0x564619fd9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.531+0000 7fc3c6a00640 1 -- 172.21.15.28:0/3356352853 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564619f2cf50 con 0x564619fcd180 2024-04-23T14:52:57.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.531+0000 7fc3bfe00640 1 --2- 172.21.15.28:0/3356352853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564619fd7850 0x564619fd9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:52:57.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.531+0000 7fc3bf400640 1 --2- 172.21.15.28:0/1167821927 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc3ac0a3d20 0x7fc3ac0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.531+0000 7fc3c6a00640 1 --2- 172.21.15.28:0/3356352853 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564619fcd180 0x564619fd7310 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc3b8009ca0 tx=0x7fc3b8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71850467e1b90cf server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:57.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.531+0000 7fc3bf400640 1 -- 172.21.15.28:0/1167821927 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7fc3c00d0f30 con 0x7fc3c810e660 2024-04-23T14:52:57.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.531+0000 7fc3b7e00640 1 -- 172.21.15.28:0/3356352853 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b8004220 con 0x564619fcd180 2024-04-23T14:52:57.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.531+0000 7fc3b7e00640 1 -- 172.21.15.28:0/3356352853 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3b80043c0 con 0x564619fcd180 2024-04-23T14:52:57.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.531+0000 7fc3b7e00640 1 -- 172.21.15.28:0/3356352853 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b80198d0 con 0x564619fcd180 2024-04-23T14:52:57.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.531+0000 7fc3bfe00640 1 --2- 172.21.15.28:0/1167821927 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc3ac0a3d20 0x7fc3ac0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.532+0000 7fc3bfe00640 1 --2- 172.21.15.28:0/1167821927 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc3ac0a3d20 0x7fc3ac0a61e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x564619fc9d10 tx=0x7fc3b002a400 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:57.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.532+0000 7fc3d0815740 1 -- 172.21.15.28:0/3356352853 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564619fcd180 msgr2=0x564619fd7310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:57.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.532+0000 7fc3d0815740 1 --2- 172.21.15.28:0/3356352853 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564619fcd180 0x564619fd7310 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc3b8009ca0 tx=0x7fc3b8011810 comp rx=0 tx=0).stop 2024-04-23T14:52:57.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.532+0000 7fc3d0815740 1 -- 172.21.15.28:0/3356352853 shutdown_connections 2024-04-23T14:52:57.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.532+0000 7fc3d0815740 1 --2- 172.21.15.28:0/3356352853 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564619fcd180 0x564619fd7310 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.532+0000 7fc3d0815740 1 --2- 172.21.15.28:0/3356352853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564619fd7850 0x564619fd9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.532+0000 7fc3d0815740 1 --2- 172.21.15.28:0/3356352853 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564619fcc860 0x564619fccc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.532+0000 7fc3d0815740 1 -- 172.21.15.28:0/3356352853 >> 172.21.15.28:0/3356352853 conn(0x564619fc8b80 msgr2=0x564619fc8f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:57.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.533+0000 7fc3d0815740 1 -- 172.21.15.28:0/3356352853 shutdown_connections 2024-04-23T14:52:57.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.533+0000 7fc3d0815740 1 -- 172.21.15.28:0/3356352853 wait complete. 2024-04-23T14:52:57.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.533+0000 7fc3d0815740 1 -- start start 2024-04-23T14:52:57.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.533+0000 7fc3d0815740 1 Processor -- start 2024-04-23T14:52:57.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.534+0000 7fc3d0815740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564619fcc860 0x564619fd47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.534+0000 7fc3d0815740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564619fcd180 0x564619fd4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.534+0000 7fc3d0815740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564619fd7850 0x564619fd5230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.534+0000 7fc3d0815740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564619fda640 con 0x564619fd7850 2024-04-23T14:52:57.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.534+0000 7fc3d0815740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564619eeb690 con 0x564619fcc860 2024-04-23T14:52:57.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.534+0000 7fc3d0815740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x564619fda7c0 con 0x564619fcd180 2024-04-23T14:52:57.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.534+0000 7fc3d0815740 10 client.0 _ll_register_callbacks cb 0x564619f27650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:52:57.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.534+0000 7fc3c6a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564619fcc860 0x564619fd47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.534+0000 7fc3c6a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564619fcc860 0x564619fd47b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60552/0 (socket says 172.21.15.28:60552) 2024-04-23T14:52:57.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.534+0000 7fc3c6a00640 1 -- 172.21.15.28:0/2752545626 learned_addr learned my addr 172.21.15.28:0/2752545626 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:57.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.534+0000 7fc3c7400640 1 --2- 172.21.15.28:0/2752545626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564619fd7850 0x564619fd5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.534+0000 7fc3bfe00640 1 --2- 172.21.15.28:0/2752545626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564619fcd180 0x564619fd4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.535+0000 7fc3c6a00640 1 -- 172.21.15.28:0/2752545626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564619fcd180 msgr2=0x564619fd4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:57.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.535+0000 7fc3c6a00640 1 --2- 172.21.15.28:0/2752545626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564619fcd180 0x564619fd4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.535+0000 7fc3c6a00640 1 -- 172.21.15.28:0/2752545626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564619fd7850 msgr2=0x564619fd5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:57.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.535+0000 7fc3c6a00640 1 --2- 172.21.15.28:0/2752545626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564619fd7850 0x564619fd5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.535+0000 7fc3c6a00640 1 -- 172.21.15.28:0/2752545626 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564619eea920 con 0x564619fcc860 2024-04-23T14:52:57.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.535+0000 7fc3c6a00640 1 --2- 172.21.15.28:0/2752545626 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564619fcc860 0x564619fd47b0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc3b8009700 tx=0x7fc3b80194e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8aef35b691cf594 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:57.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.535+0000 7fc3b4c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:52:57.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.535+0000 7fc3b4c00640 1 -- 172.21.15.28:0/2752545626 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b8028070 con 0x564619fcc860 2024-04-23T14:52:57.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.535+0000 7fc3b4c00640 1 -- 172.21.15.28:0/2752545626 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7fc3b80038c0 con 0x564619fcc860 2024-04-23T14:52:57.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.535+0000 7fc3d0815740 1 -- 172.21.15.28:0/2752545626 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3b80043c0 con 0x564619fcc860 2024-04-23T14:52:57.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.536+0000 7fc3b4c00640 1 -- 172.21.15.28:0/2752545626 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fc3b8034050 con 0x564619fcc860 2024-04-23T14:52:57.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.536+0000 7fc3d0815740 10 client.26711 fetch_fsmap learned FSMap version 20 2024-04-23T14:52:57.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.536+0000 7fc3d0815740 1 -- 172.21.15.28:0/2752545626 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x564619d0e7d0 con 0x564619fcc860 2024-04-23T14:52:57.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.536+0000 7fc3b4c00640 1 -- 172.21.15.28:0/2752545626 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3b80043c0 con 0x564619fcc860 2024-04-23T14:52:57.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.536+0000 7fc3b4c00640 1 -- 172.21.15.28:0/2752545626 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fc3b80199e0 con 0x564619fcc860 2024-04-23T14:52:57.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.536+0000 7fc3d0815740 10 client.26711 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:52:57.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.536+0000 7fc3d0815740 10 client.26711 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:52:57.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.536+0000 7fc3d0815740 20 client.26711 populate_metadata read hostname 'smithi028' 2024-04-23T14:52:57.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.536+0000 7fc3d0815740 1 --2- 172.21.15.28:0/2752545626 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564619fe7ae0 0x564619fe9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.536+0000 7fc3d0815740 4 client.26711 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:52:57.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.536+0000 7fc3d0815740 1 -- 172.21.15.28:0/2752545626 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564619f3fc10 con 0x564619fe7ae0 2024-04-23T14:52:57.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.537+0000 7fc3bfe00640 1 --2- 172.21.15.28:0/2752545626 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564619fe7ae0 0x564619fe9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.537+0000 7fc3bfe00640 1 --2- 172.21.15.28:0/2752545626 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564619fe7ae0 0x564619fe9ed0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b85df4885f4d5343 server_cookie=e4831e4fa66d2e03 in_seq=0 out_seq=0 2024-04-23T14:52:57.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.537+0000 7fc3b4c00640 10 client.26711 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:57.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.539+0000 7fc3b4c00640 1 -- 172.21.15.28:0/2752545626 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564619f3fc10 con 0x564619fe7ae0 2024-04-23T14:52:57.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.539+0000 7fc3b4c00640 10 client.26711 handle_command_reply: tid=0 2024-04-23T14:52:57.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.539+0000 7fc3d0815740 1 client.26711 shutdown 2024-04-23T14:52:57.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.539+0000 7fc3d0815740 1 -- 172.21.15.28:0/2752545626 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564619fcc860 msgr2=0x564619fd47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:57.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.539+0000 7fc3d0815740 1 --2- 172.21.15.28:0/2752545626 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564619fcc860 0x564619fd47b0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc3b8009700 tx=0x7fc3b80194e0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.539+0000 7fc3d0815740 1 -- 172.21.15.28:0/2752545626 shutdown_connections 2024-04-23T14:52:57.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.539+0000 7fc3d0815740 1 --2- 172.21.15.28:0/2752545626 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x564619fe7ae0 0x564619fe9ed0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.539+0000 7fc3d0815740 1 --2- 172.21.15.28:0/2752545626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564619fd7850 0x564619fd5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.539+0000 7fc3d0815740 1 --2- 172.21.15.28:0/2752545626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564619fcd180 0x564619fd4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.539+0000 7fc3b4c00640 0 client.26711 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:57.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.539+0000 7fc3d0815740 1 --2- 172.21.15.28:0/2752545626 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564619fcc860 0x564619fd47b0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.539+0000 7fc3d0815740 1 -- 172.21.15.28:0/2752545626 >> 172.21.15.28:0/2752545626 conn(0x564619fc8b80 msgr2=0x564619fce980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:57.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.539+0000 7fc3d0815740 1 -- 172.21.15.28:0/2752545626 shutdown_connections 2024-04-23T14:52:57.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.539+0000 7fc3d0815740 1 -- 172.21.15.28:0/2752545626 wait complete. 2024-04-23T14:52:57.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.540+0000 7fc3d0815740 20 client.26711 trim_cache size 0 max 16384 2024-04-23T14:52:57.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.561+0000 7fc3d0815740 1 Processor -- start 2024-04-23T14:52:57.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.561+0000 7fc3d0815740 1 -- start start 2024-04-23T14:52:57.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.561+0000 7fc3d0815740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564619fcc860 0x564619fe3500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.561+0000 7fc3d0815740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564619fcd180 0x564619fcf000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.561+0000 7fc3d0815740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564619fd7850 0x564619fd3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.561+0000 7fc3d0815740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c81b2350 con 0x564619fcc860 2024-04-23T14:52:57.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.561+0000 7fc3d0815740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c81b4bb0 con 0x564619fd7850 2024-04-23T14:52:57.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.561+0000 7fc3d0815740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c81b29a0 con 0x564619fcd180 2024-04-23T14:52:57.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.562+0000 7fc3c7400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564619fd7850 0x564619fd3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.562+0000 7fc3bfe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564619fcd180 0x564619fcf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.562+0000 7fc3c7400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564619fd7850 0x564619fd3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60564/0 (socket says 172.21.15.28:60564) 2024-04-23T14:52:57.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.562+0000 7fc3c6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564619fcc860 0x564619fe3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.562+0000 7fc3c7400640 1 -- 172.21.15.28:0/2317239628 learned_addr learned my addr 172.21.15.28:0/2317239628 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:57.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.562+0000 7fc3c7400640 1 -- 172.21.15.28:0/2317239628 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564619fcd180 msgr2=0x564619fcf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:57.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.562+0000 7fc3c7400640 1 --2- 172.21.15.28:0/2317239628 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564619fcd180 0x564619fcf000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.562+0000 7fc3c7400640 1 -- 172.21.15.28:0/2317239628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564619fcc860 msgr2=0x564619fe3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:57.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.562+0000 7fc3c7400640 1 --2- 172.21.15.28:0/2317239628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564619fcc860 0x564619fe3500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.562+0000 7fc3c7400640 1 -- 172.21.15.28:0/2317239628 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564619c443f0 con 0x564619fd7850 2024-04-23T14:52:57.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.562+0000 7fc3bfe00640 1 --2- 172.21.15.28:0/2317239628 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564619fcd180 0x564619fcf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:52:57.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.562+0000 7fc3c7400640 1 --2- 172.21.15.28:0/2317239628 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564619fd7850 0x564619fd3660 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fc3c00ce670 tx=0x7fc3c00a1210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=996dda8c90997b09 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:57.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.563+0000 7fc3b7400640 1 -- 172.21.15.28:0/2317239628 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c00ba070 con 0x564619fd7850 2024-04-23T14:52:57.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.563+0000 7fc3b7400640 1 -- 172.21.15.28:0/2317239628 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3c00994d0 con 0x564619fd7850 2024-04-23T14:52:57.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.563+0000 7fc3b7400640 1 -- 172.21.15.28:0/2317239628 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c009d0c0 con 0x564619fd7850 2024-04-23T14:52:57.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.564+0000 7fc3d0815740 1 -- 172.21.15.28:0/2317239628 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564619fd7850 msgr2=0x564619fd3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:57.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.564+0000 7fc3d0815740 1 --2- 172.21.15.28:0/2317239628 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564619fd7850 0x564619fd3660 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fc3c00ce670 tx=0x7fc3c00a1210 comp rx=0 tx=0).stop 2024-04-23T14:52:57.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.564+0000 7fc3d0815740 1 -- 172.21.15.28:0/2317239628 shutdown_connections 2024-04-23T14:52:57.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.564+0000 7fc3d0815740 1 --2- 172.21.15.28:0/2317239628 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564619fd7850 0x564619fd3660 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.564+0000 7fc3d0815740 1 --2- 172.21.15.28:0/2317239628 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564619fcd180 0x564619fcf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.564+0000 7fc3d0815740 1 --2- 172.21.15.28:0/2317239628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564619fcc860 0x564619fe3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.564+0000 7fc3d0815740 1 -- 172.21.15.28:0/2317239628 >> 172.21.15.28:0/2317239628 conn(0x564619fc8b80 msgr2=0x564619fd2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:57.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.564+0000 7fc3d0815740 1 -- 172.21.15.28:0/2317239628 shutdown_connections 2024-04-23T14:52:57.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.564+0000 7fc3d0815740 1 -- 172.21.15.28:0/2317239628 wait complete. 2024-04-23T14:52:57.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.564+0000 7fc3d0815740 1 -- start start 2024-04-23T14:52:57.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.565+0000 7fc3d0815740 1 Processor -- start 2024-04-23T14:52:57.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.565+0000 7fc3d0815740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564619fcc860 0x564619fdb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.565+0000 7fc3d0815740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564619fcd180 0x564619fdbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.565+0000 7fc3d0815740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564619fd7850 0x564619fdc170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.565+0000 7fc3d0815740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c81b3070 con 0x564619fd7850 2024-04-23T14:52:57.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.565+0000 7fc3d0815740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c81b44e0 con 0x564619fcd180 2024-04-23T14:52:57.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.565+0000 7fc3d0815740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c81b3740 con 0x564619fcc860 2024-04-23T14:52:57.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.565+0000 7fc3d0815740 10 client.0 _ll_register_callbacks cb 0x7fc3b8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:52:57.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.565+0000 7fc3c6a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564619fcc860 0x564619fdb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.565+0000 7fc3bfe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564619fcd180 0x564619fdbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.565+0000 7fc3c7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564619fd7850 0x564619fdc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.565+0000 7fc3c6a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564619fcc860 0x564619fdb6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:46278/0 (socket says 172.21.15.28:46278) 2024-04-23T14:52:57.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.565+0000 7fc3bfe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564619fcd180 0x564619fdbc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:60576/0 (socket says 172.21.15.28:60576) 2024-04-23T14:52:57.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.565+0000 7fc3c6a00640 1 -- 172.21.15.28:0/1511592939 learned_addr learned my addr 172.21.15.28:0/1511592939 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:52:57.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.566+0000 7fc3bfe00640 1 -- 172.21.15.28:0/1511592939 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564619fcc860 msgr2=0x564619fdb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:57.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.566+0000 7fc3bfe00640 1 --2- 172.21.15.28:0/1511592939 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564619fcc860 0x564619fdb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.566+0000 7fc3bfe00640 1 -- 172.21.15.28:0/1511592939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564619fd7850 msgr2=0x564619fdc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:52:57.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.566+0000 7fc3bfe00640 1 --2- 172.21.15.28:0/1511592939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564619fd7850 0x564619fdc170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.566+0000 7fc3bfe00640 1 -- 172.21.15.28:0/1511592939 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564619c040f0 con 0x564619fcd180 2024-04-23T14:52:57.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.566+0000 7fc3c6a00640 1 --2- 172.21.15.28:0/1511592939 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564619fcc860 0x564619fdb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:52:57.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.566+0000 7fc3bfe00640 1 --2- 172.21.15.28:0/1511592939 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564619fcd180 0x564619fdbc30 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc3c810dc60 tx=0x7fc3b0009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d217b2ff5b75e459 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:52:57.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.566+0000 7fc3b4c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:52:57.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.566+0000 7fc3b4c00640 1 -- 172.21.15.28:0/1511592939 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b0045040 con 0x564619fcd180 2024-04-23T14:52:57.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.567+0000 7fc3d0815740 1 -- 172.21.15.28:0/1511592939 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3c00994d0 con 0x564619fcd180 2024-04-23T14:52:57.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.567+0000 7fc3b4c00640 1 -- 172.21.15.28:0/1511592939 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8542+0+0 (secure 0 0 0) 0x7fc3b00543a0 con 0x564619fcd180 2024-04-23T14:52:57.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.567+0000 7fc3b4c00640 1 -- 172.21.15.28:0/1511592939 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fc3b0064050 con 0x564619fcd180 2024-04-23T14:52:57.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.567+0000 7fc3d0815740 10 client.26717 fetch_fsmap learned FSMap version 20 2024-04-23T14:52:57.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.567+0000 7fc3d0815740 1 -- 172.21.15.28:0/1511592939 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x564619ef7810 con 0x564619fcd180 2024-04-23T14:52:57.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.568+0000 7fc3b4c00640 1 -- 172.21.15.28:0/1511592939 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3c00994d0 con 0x564619fcd180 2024-04-23T14:52:57.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.568+0000 7fc3b4c00640 1 -- 172.21.15.28:0/1511592939 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fc3b0031cd0 con 0x564619fcd180 2024-04-23T14:52:57.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.568+0000 7fc3d0815740 10 client.26717 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:52:57.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.568+0000 7fc3d0815740 10 client.26717 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:52:57.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.568+0000 7fc3d0815740 20 client.26717 populate_metadata read hostname 'smithi028' 2024-04-23T14:52:57.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.568+0000 7fc3d0815740 1 --2- 172.21.15.28:0/1511592939 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56461a04e810 0x56461a050c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:52:57.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.568+0000 7fc3d0815740 4 client.26717 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:52:57.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.568+0000 7fc3d0815740 1 -- 172.21.15.28:0/1511592939 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564619f39a80 con 0x56461a04e810 2024-04-23T14:52:57.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.568+0000 7fc3c6a00640 1 --2- 172.21.15.28:0/1511592939 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56461a04e810 0x56461a050c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:52:57.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.569+0000 7fc3c6a00640 1 --2- 172.21.15.28:0/1511592939 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56461a04e810 0x56461a050c30 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca93264fdb276368 server_cookie=86f310d2e3e0ea50 in_seq=0 out_seq=0 2024-04-23T14:52:57.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.569+0000 7fc3b4c00640 10 client.26717 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:57.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.571+0000 7fc3b4c00640 1 -- 172.21.15.28:0/1511592939 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564619f39a80 con 0x56461a04e810 2024-04-23T14:52:57.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.571+0000 7fc3b4c00640 10 client.26717 handle_command_reply: tid=0 2024-04-23T14:52:57.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.572+0000 7fc3d0815740 1 client.26717 shutdown 2024-04-23T14:52:57.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.572+0000 7fc3d0815740 1 -- 172.21.15.28:0/1511592939 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564619fcd180 msgr2=0x564619fdbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:57.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.572+0000 7fc3d0815740 1 --2- 172.21.15.28:0/1511592939 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564619fcd180 0x564619fdbc30 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc3c810dc60 tx=0x7fc3b0009260 comp rx=0 tx=0).stop 2024-04-23T14:52:57.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.572+0000 7fc3d0815740 1 -- 172.21.15.28:0/1511592939 shutdown_connections 2024-04-23T14:52:57.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.572+0000 7fc3d0815740 1 --2- 172.21.15.28:0/1511592939 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56461a04e810 0x56461a050c30 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.572+0000 7fc3d0815740 1 --2- 172.21.15.28:0/1511592939 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564619fcd180 0x564619fdbc30 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.572+0000 7fc3d0815740 1 --2- 172.21.15.28:0/1511592939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564619fd7850 0x564619fdc170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.572+0000 7fc3d0815740 1 --2- 172.21.15.28:0/1511592939 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564619fcc860 0x564619fdb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.572+0000 7fc3b4c00640 0 client.26717 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:52:57.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.572+0000 7fc3d0815740 1 -- 172.21.15.28:0/1511592939 >> 172.21.15.28:0/1511592939 conn(0x564619fc8b80 msgr2=0x564619fd37e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:57.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.572+0000 7fc3d0815740 1 -- 172.21.15.28:0/1511592939 shutdown_connections 2024-04-23T14:52:57.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.572+0000 7fc3d0815740 1 -- 172.21.15.28:0/1511592939 wait complete. 2024-04-23T14:52:57.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.572+0000 7fc3d0815740 20 client.26717 trim_cache size 0 max 16384 2024-04-23T14:52:57.575 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:52:57.575 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-23T14:52:57.575 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-23T14:52:57.575 INFO:teuthology.orchestra.run.smithi028.stdout: "b8ffc275-4bd4-44c3-a226-fc17554a5e95": { 2024-04-23T14:52:57.575 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-23T14:52:57.575 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "b8ffc275-4bd4-44c3-a226-fc17554a5e95", 2024-04-23T14:52:57.575 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-23T14:52:57.575 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:52:57.575 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:52:57.575 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:52:57.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.573+0000 7fc3cda00640 1 -- 172.21.15.28:0/1167821927 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc3ac0a3d20 msgr2=0x7fc3ac0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:57.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.573+0000 7fc3cda00640 1 --2- 172.21.15.28:0/1167821927 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc3ac0a3d20 0x7fc3ac0a61e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x564619fc9d10 tx=0x7fc3b002a400 comp rx=0 tx=0).stop 2024-04-23T14:52:57.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.573+0000 7fc3cda00640 1 -- 172.21.15.28:0/1167821927 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3c810e660 msgr2=0x7fc3c81a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:52:57.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.573+0000 7fc3cda00640 1 --2- 172.21.15.28:0/1167821927 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3c810e660 0x7fc3c81a3710 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc3c0009fd0 tx=0x7fc3c000ef10 comp rx=0 tx=0).stop 2024-04-23T14:52:57.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.573+0000 7fc3cda00640 1 -- 172.21.15.28:0/1167821927 shutdown_connections 2024-04-23T14:52:57.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.573+0000 7fc3cda00640 1 --2- 172.21.15.28:0/1167821927 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc3ac0a3d20 0x7fc3ac0a61e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.573+0000 7fc3cda00640 1 --2- 172.21.15.28:0/1167821927 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3c810e660 0x7fc3c81a3710 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.573+0000 7fc3cda00640 1 --2- 172.21.15.28:0/1167821927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3c8108fd0 0x7fc3c819f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.573+0000 7fc3cda00640 1 --2- 172.21.15.28:0/1167821927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3c8108610 0x7fc3c819ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:52:57.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.573+0000 7fc3cda00640 1 -- 172.21.15.28:0/1167821927 >> 172.21.15.28:0/1167821927 conn(0x7fc3c8076540 msgr2=0x7fc3c8077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:52:57.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.573+0000 7fc3cda00640 1 -- 172.21.15.28:0/1167821927 shutdown_connections 2024-04-23T14:52:57.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:52:57.573+0000 7fc3cda00640 1 -- 172.21.15.28:0/1167821927 wait complete. 2024-04-23T14:52:57.593 INFO:tasks.cephfs.filesystem:scrub status for tag:b8ffc275-4bd4-44c3-a226-fc17554a5e95 - {'path': '/', 'tag': 'b8ffc275-4bd4-44c3-a226-fc17554a5e95', 'options': 'recursive,force'} 2024-04-23T14:52:58.515 INFO:tasks.workunit.client.0.smithi028.stdout: 1 753146 8.96 MB/sec execute 314 sec latency 17.795 ms 2024-04-23T14:52:59.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:58 smithi105 ceph-mon[32555]: scrub queued for path: / 2024-04-23T14:52:59.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:58 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:52:59.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:58 smithi105 ceph-mon[32555]: scrub summary: active paths [/] 2024-04-23T14:52:59.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:58 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:52:59.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:52:58 smithi105 ceph-mon[32555]: pgmap v275: 97 pgs: 97 active+clean; 461 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.9 MiB/s wr, 214 op/s 2024-04-23T14:52:59.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:58 smithi038 ceph-mon[32348]: scrub queued for path: / 2024-04-23T14:52:59.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:58 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:52:59.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:58 smithi038 ceph-mon[32348]: scrub summary: active paths [/] 2024-04-23T14:52:59.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:58 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:52:59.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:52:58 smithi038 ceph-mon[32348]: pgmap v275: 97 pgs: 97 active+clean; 461 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.9 MiB/s wr, 214 op/s 2024-04-23T14:52:59.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:58 smithi028 ceph-mon[27810]: scrub queued for path: / 2024-04-23T14:52:59.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:58 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:52:59.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:58 smithi028 ceph-mon[27810]: scrub summary: active paths [/] 2024-04-23T14:52:59.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:58 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:52:59.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:52:58 smithi028 ceph-mon[27810]: pgmap v275: 97 pgs: 97 active+clean; 461 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.9 MiB/s wr, 214 op/s 2024-04-23T14:52:59.515 INFO:tasks.workunit.client.0.smithi028.stdout: 1 754684 8.95 MB/sec execute 315 sec latency 18.214 ms 2024-04-23T14:53:00.516 INFO:tasks.workunit.client.0.smithi028.stdout: 1 756632 8.96 MB/sec execute 316 sec latency 29.286 ms 2024-04-23T14:53:01.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:00 smithi105 ceph-mon[32555]: pgmap v276: 97 pgs: 97 active+clean; 461 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 150 op/s 2024-04-23T14:53:01.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:00 smithi105 ceph-mon[32555]: scrub summary: idle 2024-04-23T14:53:01.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:00 smithi038 ceph-mon[32348]: pgmap v276: 97 pgs: 97 active+clean; 461 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 150 op/s 2024-04-23T14:53:01.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:00 smithi038 ceph-mon[32348]: scrub summary: idle 2024-04-23T14:53:01.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:00 smithi028 ceph-mon[27810]: pgmap v276: 97 pgs: 97 active+clean; 461 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 150 op/s 2024-04-23T14:53:01.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:00 smithi028 ceph-mon[27810]: scrub summary: idle 2024-04-23T14:53:01.516 INFO:tasks.workunit.client.0.smithi028.stdout: 1 757823 8.95 MB/sec execute 317 sec latency 79.325 ms 2024-04-23T14:53:02.516 INFO:tasks.workunit.client.0.smithi028.stdout: 1 759667 8.95 MB/sec execute 318 sec latency 16.936 ms 2024-04-23T14:53:03.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:02 smithi105 ceph-mon[32555]: pgmap v277: 97 pgs: 97 active+clean; 461 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 150 op/s 2024-04-23T14:53:03.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:02 smithi038 ceph-mon[32348]: pgmap v277: 97 pgs: 97 active+clean; 461 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 150 op/s 2024-04-23T14:53:03.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:02 smithi028 ceph-mon[27810]: pgmap v277: 97 pgs: 97 active+clean; 461 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 150 op/s 2024-04-23T14:53:03.516 INFO:tasks.workunit.client.0.smithi028.stdout: 1 760856 8.94 MB/sec execute 319 sec latency 18.315 ms 2024-04-23T14:53:04.516 INFO:tasks.workunit.client.0.smithi028.stdout: 1 762467 8.94 MB/sec execute 320 sec latency 23.796 ms 2024-04-23T14:53:05.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:04 smithi038 ceph-mon[32348]: pgmap v278: 97 pgs: 97 active+clean; 469 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.3 MiB/s wr, 238 op/s 2024-04-23T14:53:05.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:04 smithi028 ceph-mon[27810]: pgmap v278: 97 pgs: 97 active+clean; 469 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.3 MiB/s wr, 238 op/s 2024-04-23T14:53:05.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:04 smithi105 ceph-mon[32555]: pgmap v278: 97 pgs: 97 active+clean; 469 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.3 MiB/s wr, 238 op/s 2024-04-23T14:53:05.516 INFO:tasks.workunit.client.0.smithi028.stdout: 1 763962 8.95 MB/sec execute 321 sec latency 40.070 ms 2024-04-23T14:53:06.516 INFO:tasks.workunit.client.0.smithi028.stdout: 1 765101 8.93 MB/sec execute 322 sec latency 45.024 ms 2024-04-23T14:53:07.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:06 smithi038 ceph-mon[32348]: pgmap v279: 97 pgs: 97 active+clean; 469 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 164 op/s 2024-04-23T14:53:07.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:06 smithi038 ceph-mon[32348]: osdmap e93: 12 total, 12 up, 12 in 2024-04-23T14:53:07.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:06 smithi028 ceph-mon[27810]: pgmap v279: 97 pgs: 97 active+clean; 469 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 164 op/s 2024-04-23T14:53:07.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:06 smithi028 ceph-mon[27810]: osdmap e93: 12 total, 12 up, 12 in 2024-04-23T14:53:07.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:06 smithi105 ceph-mon[32555]: pgmap v279: 97 pgs: 97 active+clean; 469 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 164 op/s 2024-04-23T14:53:07.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:06 smithi105 ceph-mon[32555]: osdmap e93: 12 total, 12 up, 12 in 2024-04-23T14:53:07.516 INFO:tasks.workunit.client.0.smithi028.stdout: 1 766899 8.94 MB/sec execute 323 sec latency 27.330 ms 2024-04-23T14:53:08.516 INFO:tasks.workunit.client.0.smithi028.stdout: 1 767900 8.92 MB/sec execute 324 sec latency 13.181 ms 2024-04-23T14:53:09.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:08 smithi038 ceph-mon[32348]: pgmap v281: 97 pgs: 97 active+clean; 475 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 184 op/s 2024-04-23T14:53:09.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:08 smithi028 ceph-mon[27810]: pgmap v281: 97 pgs: 97 active+clean; 475 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 184 op/s 2024-04-23T14:53:09.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:08 smithi105 ceph-mon[32555]: pgmap v281: 97 pgs: 97 active+clean; 475 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 184 op/s 2024-04-23T14:53:09.517 INFO:tasks.workunit.client.0.smithi028.stdout: 1 769835 8.93 MB/sec execute 325 sec latency 18.628 ms 2024-04-23T14:53:10.517 INFO:tasks.workunit.client.0.smithi028.stdout: 1 771181 8.93 MB/sec execute 326 sec latency 24.126 ms 2024-04-23T14:53:11.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:10 smithi038 ceph-mon[32348]: pgmap v282: 97 pgs: 97 active+clean; 475 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 184 op/s 2024-04-23T14:53:11.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:10 smithi028 ceph-mon[27810]: pgmap v282: 97 pgs: 97 active+clean; 475 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 184 op/s 2024-04-23T14:53:11.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:10 smithi105 ceph-mon[32555]: pgmap v282: 97 pgs: 97 active+clean; 475 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 184 op/s 2024-04-23T14:53:11.517 INFO:tasks.workunit.client.0.smithi028.stdout: 1 772478 8.91 MB/sec execute 327 sec latency 45.176 ms 2024-04-23T14:53:12.517 INFO:tasks.workunit.client.0.smithi028.stdout: 1 774289 8.93 MB/sec execute 328 sec latency 16.142 ms 2024-04-23T14:53:12.926 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:12 smithi028 ceph-mon[27810]: pgmap v283: 97 pgs: 97 active+clean; 475 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 184 op/s 2024-04-23T14:53:12.926 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:12 smithi028 ceph-mon[27810]: osdmap e94: 12 total, 12 up, 12 in 2024-04-23T14:53:13.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:12 smithi105 ceph-mon[32555]: pgmap v283: 97 pgs: 97 active+clean; 475 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 184 op/s 2024-04-23T14:53:13.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:12 smithi105 ceph-mon[32555]: osdmap e94: 12 total, 12 up, 12 in 2024-04-23T14:53:13.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:12 smithi038 ceph-mon[32348]: pgmap v283: 97 pgs: 97 active+clean; 475 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 184 op/s 2024-04-23T14:53:13.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:12 smithi038 ceph-mon[32348]: osdmap e94: 12 total, 12 up, 12 in 2024-04-23T14:53:13.517 INFO:tasks.workunit.client.0.smithi028.stdout: 1 775473 8.92 MB/sec execute 329 sec latency 20.358 ms 2024-04-23T14:53:14.517 INFO:tasks.workunit.client.0.smithi028.stdout: 1 777383 8.92 MB/sec execute 330 sec latency 19.387 ms 2024-04-23T14:53:15.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:14 smithi105 ceph-mon[32555]: pgmap v285: 97 pgs: 97 active+clean; 483 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 201 op/s 2024-04-23T14:53:15.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:14 smithi038 ceph-mon[32348]: pgmap v285: 97 pgs: 97 active+clean; 483 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 201 op/s 2024-04-23T14:53:15.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:14 smithi028 ceph-mon[27810]: pgmap v285: 97 pgs: 97 active+clean; 483 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 201 op/s 2024-04-23T14:53:15.517 INFO:tasks.workunit.client.0.smithi028.stdout: 1 778514 8.91 MB/sec execute 331 sec latency 24.696 ms 2024-04-23T14:53:16.517 INFO:tasks.workunit.client.0.smithi028.stdout: 1 780045 8.90 MB/sec execute 332 sec latency 79.220 ms 2024-04-23T14:53:17.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:16 smithi105 ceph-mon[32555]: pgmap v286: 97 pgs: 97 active+clean; 483 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.2 MiB/s wr, 164 op/s 2024-04-23T14:53:17.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:16 smithi038 ceph-mon[32348]: pgmap v286: 97 pgs: 97 active+clean; 483 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.2 MiB/s wr, 164 op/s 2024-04-23T14:53:17.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:16 smithi028 ceph-mon[27810]: pgmap v286: 97 pgs: 97 active+clean; 483 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.2 MiB/s wr, 164 op/s 2024-04-23T14:53:17.518 INFO:tasks.workunit.client.0.smithi028.stdout: 1 781618 8.91 MB/sec execute 333 sec latency 12.329 ms 2024-04-23T14:53:18.518 INFO:tasks.workunit.client.0.smithi028.stdout: 1 783118 8.90 MB/sec execute 334 sec latency 19.241 ms 2024-04-23T14:53:19.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:18 smithi105 ceph-mon[32555]: pgmap v287: 97 pgs: 97 active+clean; 493 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 164 op/s 2024-04-23T14:53:19.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:18 smithi038 ceph-mon[32348]: pgmap v287: 97 pgs: 97 active+clean; 493 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 164 op/s 2024-04-23T14:53:19.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:18 smithi028 ceph-mon[27810]: pgmap v287: 97 pgs: 97 active+clean; 493 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 164 op/s 2024-04-23T14:53:19.518 INFO:tasks.workunit.client.0.smithi028.stdout: 1 784891 8.91 MB/sec execute 335 sec latency 22.254 ms 2024-04-23T14:53:20.518 INFO:tasks.workunit.client.0.smithi028.stdout: 1 785836 8.89 MB/sec execute 336 sec latency 29.336 ms 2024-04-23T14:53:21.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:20 smithi105 ceph-mon[32555]: pgmap v288: 97 pgs: 97 active+clean; 493 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 164 op/s 2024-04-23T14:53:21.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:20 smithi038 ceph-mon[32348]: pgmap v288: 97 pgs: 97 active+clean; 493 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 164 op/s 2024-04-23T14:53:21.155 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:20 smithi028 ceph-mon[27810]: pgmap v288: 97 pgs: 97 active+clean; 493 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 164 op/s 2024-04-23T14:53:21.518 INFO:tasks.workunit.client.0.smithi028.stdout: 1 787168 8.88 MB/sec execute 337 sec latency 115.101 ms 2024-04-23T14:53:22.518 INFO:tasks.workunit.client.0.smithi028.stdout: 1 788632 8.89 MB/sec execute 338 sec latency 16.217 ms 2024-04-23T14:53:23.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:22 smithi105 ceph-mon[32555]: pgmap v289: 97 pgs: 97 active+clean; 493 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 164 op/s 2024-04-23T14:53:23.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:22 smithi038 ceph-mon[32348]: pgmap v289: 97 pgs: 97 active+clean; 493 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 164 op/s 2024-04-23T14:53:23.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:22 smithi028 ceph-mon[27810]: pgmap v289: 97 pgs: 97 active+clean; 493 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 164 op/s 2024-04-23T14:53:23.518 INFO:tasks.workunit.client.0.smithi028.stdout: 1 789838 8.88 MB/sec execute 339 sec latency 22.149 ms 2024-04-23T14:53:24.519 INFO:tasks.workunit.client.0.smithi028.stdout: 1 792004 8.89 MB/sec execute 340 sec latency 18.174 ms 2024-04-23T14:53:25.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:24 smithi105 ceph-mon[32555]: pgmap v290: 97 pgs: 97 active+clean; 500 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.3 MiB/s wr, 205 op/s 2024-04-23T14:53:25.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:24 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:53:25.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:24 smithi038 ceph-mon[32348]: pgmap v290: 97 pgs: 97 active+clean; 500 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.3 MiB/s wr, 205 op/s 2024-04-23T14:53:25.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:24 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:53:25.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:24 smithi028 ceph-mon[27810]: pgmap v290: 97 pgs: 97 active+clean; 500 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.3 MiB/s wr, 205 op/s 2024-04-23T14:53:25.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:24 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:53:25.519 INFO:tasks.workunit.client.0.smithi028.stdout: 1 793281 8.88 MB/sec execute 341 sec latency 34.794 ms 2024-04-23T14:53:26.519 INFO:tasks.workunit.client.0.smithi028.stdout: 1 794792 8.88 MB/sec execute 342 sec latency 75.851 ms 2024-04-23T14:53:27.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:26 smithi105 ceph-mon[32555]: pgmap v291: 97 pgs: 97 active+clean; 500 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 135 op/s 2024-04-23T14:53:27.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:26 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:53:27.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:26 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:53:27.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:26 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:53:27.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:26 smithi038 ceph-mon[32348]: pgmap v291: 97 pgs: 97 active+clean; 500 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 135 op/s 2024-04-23T14:53:27.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:26 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:53:27.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:26 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:53:27.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:26 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:53:27.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:26 smithi028 ceph-mon[27810]: pgmap v291: 97 pgs: 97 active+clean; 500 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 135 op/s 2024-04-23T14:53:27.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:26 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:53:27.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:26 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:53:27.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:26 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:53:27.519 INFO:tasks.workunit.client.0.smithi028.stdout: 1 796111 8.87 MB/sec execute 343 sec latency 22.472 ms 2024-04-23T14:53:27.594 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:53:27.594 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:53:27.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.818+0000 7f4823e00640 1 -- 172.21.15.28:0/1422381112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4824102ac0 msgr2=0x7f4824106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:27.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.818+0000 7f4823e00640 1 --2- 172.21.15.28:0/1422381112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4824102ac0 0x7f4824106b00 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f4810009900 tx=0x7f481002a3f0 comp rx=0 tx=0).stop 2024-04-23T14:53:27.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.818+0000 7f4823e00640 1 -- 172.21.15.28:0/1422381112 shutdown_connections 2024-04-23T14:53:27.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.818+0000 7f4823e00640 1 --2- 172.21.15.28:0/1422381112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4824102ac0 0x7f4824106b00 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:27.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.818+0000 7f4823e00640 1 --2- 172.21.15.28:0/1422381112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48241071e0 0x7f482410edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:27.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.818+0000 7f4823e00640 1 --2- 172.21.15.28:0/1422381112 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4824102110 0x7f48241024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:27.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.818+0000 7f4823e00640 1 -- 172.21.15.28:0/1422381112 >> 172.21.15.28:0/1422381112 conn(0x7f48240fd9d0 msgr2=0x7f48240ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:53:27.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.818+0000 7f4823e00640 1 -- 172.21.15.28:0/1422381112 shutdown_connections 2024-04-23T14:53:27.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.818+0000 7f4823e00640 1 -- 172.21.15.28:0/1422381112 wait complete. 2024-04-23T14:53:27.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.819+0000 7f4823e00640 1 Processor -- start 2024-04-23T14:53:27.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.820+0000 7f4823e00640 1 -- start start 2024-04-23T14:53:27.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.820+0000 7f4823e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4824102110 0x7f482410e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:27.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.820+0000 7f4823e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f48241071e0 0x7f482410e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:27.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.820+0000 7f4823e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48241a3670 0x7f48241a3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:27.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.820+0000 7f4823e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f48241122a0 con 0x7f48241a3670 2024-04-23T14:53:27.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.820+0000 7f4823e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4824112120 con 0x7f4824102110 2024-04-23T14:53:27.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.820+0000 7f4823e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4824112420 con 0x7f48241071e0 2024-04-23T14:53:27.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.820+0000 7f4822a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4824102110 0x7f482410e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:27.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.820+0000 7f4822a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4824102110 0x7f482410e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:50842/0 (socket says 172.21.15.28:50842) 2024-04-23T14:53:27.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.820+0000 7f4822a00640 1 -- 172.21.15.28:0/1420593025 learned_addr learned my addr 172.21.15.28:0/1420593025 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:53:27.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.821+0000 7f4822000640 1 --2- 172.21.15.28:0/1420593025 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f48241071e0 0x7f482410e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:27.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.821+0000 7f4823400640 1 --2- 172.21.15.28:0/1420593025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48241a3670 0x7f48241a3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:27.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.821+0000 7f4822a00640 1 -- 172.21.15.28:0/1420593025 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f48241071e0 msgr2=0x7f482410e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:27.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.821+0000 7f4822a00640 1 --2- 172.21.15.28:0/1420593025 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f48241071e0 0x7f482410e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:27.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.821+0000 7f4822a00640 1 -- 172.21.15.28:0/1420593025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48241a3670 msgr2=0x7f48241a3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:27.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.821+0000 7f4822a00640 1 --2- 172.21.15.28:0/1420593025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48241a3670 0x7f48241a3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:27.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.821+0000 7f4822a00640 1 -- 172.21.15.28:0/1420593025 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48241a4160 con 0x7f4824102110 2024-04-23T14:53:27.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.821+0000 7f4823400640 1 --2- 172.21.15.28:0/1420593025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48241a3670 0x7f48241a3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:53:27.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.821+0000 7f4822a00640 1 --2- 172.21.15.28:0/1420593025 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4824102110 0x7f482410e170 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f481400d950 tx=0x7f481400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:53:27.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.821+0000 7f481b400640 1 -- 172.21.15.28:0/1420593025 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4814017070 con 0x7f4824102110 2024-04-23T14:53:27.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.822+0000 7f481b400640 1 -- 172.21.15.28:0/1420593025 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4814009c30 con 0x7f4824102110 2024-04-23T14:53:27.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.822+0000 7f4823e00640 1 -- 172.21.15.28:0/1420593025 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48241a4450 con 0x7f4824102110 2024-04-23T14:53:27.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.822+0000 7f481b400640 1 -- 172.21.15.28:0/1420593025 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f481400f6b0 con 0x7f4824102110 2024-04-23T14:53:27.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.822+0000 7f4823e00640 1 -- 172.21.15.28:0/1420593025 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48241abc60 con 0x7f4824102110 2024-04-23T14:53:27.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.825+0000 7f481b400640 1 -- 172.21.15.28:0/1420593025 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f48140109d0 con 0x7f4824102110 2024-04-23T14:53:27.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.825+0000 7f482c5c5740 1 Processor -- start 2024-04-23T14:53:27.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.825+0000 7f482c5c5740 1 -- start start 2024-04-23T14:53:27.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.826+0000 7f482c5c5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557658773a30 0x557658773e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:27.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.826+0000 7f482c5c5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557658774350 0x55765877e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:27.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.826+0000 7f482c5c5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55765877ea20 0x557658780e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:27.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.826+0000 7f482c5c5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5576586d91f0 con 0x55765877ea20 2024-04-23T14:53:27.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.826+0000 7f482c5c5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55765867f350 con 0x557658773a30 2024-04-23T14:53:27.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.826+0000 7f482c5c5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5576586d51b0 con 0x557658774350 2024-04-23T14:53:27.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.826+0000 7f4823400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557658773a30 0x557658773e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:27.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.826+0000 7f4823400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557658773a30 0x557658773e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:50844/0 (socket says 172.21.15.28:50844) 2024-04-23T14:53:27.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.826+0000 7f4823400640 1 -- 172.21.15.28:0/1836320039 learned_addr learned my addr 172.21.15.28:0/1836320039 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:53:27.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.826+0000 7f4822000640 1 --2- 172.21.15.28:0/1836320039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55765877ea20 0x557658780e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:27.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.826+0000 7f4822a00640 1 --2- 172.21.15.28:0/1836320039 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557658774350 0x55765877e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:27.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.827+0000 7f4823400640 1 -- 172.21.15.28:0/1836320039 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557658774350 msgr2=0x55765877e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:53:27.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.827+0000 7f4823400640 1 --2- 172.21.15.28:0/1836320039 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557658774350 0x55765877e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:27.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.827+0000 7f4823400640 1 -- 172.21.15.28:0/1836320039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55765877ea20 msgr2=0x557658780e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:53:27.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.827+0000 7f4823400640 1 --2- 172.21.15.28:0/1836320039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55765877ea20 0x557658780e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:27.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.827+0000 7f4823400640 1 -- 172.21.15.28:0/1836320039 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576586d3f50 con 0x557658773a30 2024-04-23T14:53:27.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.827+0000 7f4822a00640 1 --2- 172.21.15.28:0/1836320039 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557658774350 0x55765877e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:53:27.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.827+0000 7f4823400640 1 --2- 172.21.15.28:0/1836320039 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557658773a30 0x557658773e10 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f481c009fd0 tx=0x7f481c010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fe908045877f75e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:53:27.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.827+0000 7f481b400640 1 --2- 172.21.15.28:0/1420593025 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f48080a3da0 0x7f48080a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:27.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.827+0000 7f480f400640 1 -- 172.21.15.28:0/1836320039 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f481c011700 con 0x557658773a30 2024-04-23T14:53:27.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.827+0000 7f480f400640 1 -- 172.21.15.28:0/1836320039 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f481c011d80 con 0x557658773a30 2024-04-23T14:53:27.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.827+0000 7f480f400640 1 -- 172.21.15.28:0/1836320039 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f481c00e3c0 con 0x557658773a30 2024-04-23T14:53:27.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.827+0000 7f481b400640 1 -- 172.21.15.28:0/1420593025 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7f48140d0420 con 0x7f4824102110 2024-04-23T14:53:27.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.828+0000 7f4822a00640 1 --2- 172.21.15.28:0/1420593025 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f48080a3da0 0x7f48080a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:27.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.828+0000 7f482c5c5740 1 -- 172.21.15.28:0/1836320039 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557658773a30 msgr2=0x557658773e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:27.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.828+0000 7f482c5c5740 1 --2- 172.21.15.28:0/1836320039 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557658773a30 0x557658773e10 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f481c009fd0 tx=0x7f481c010860 comp rx=0 tx=0).stop 2024-04-23T14:53:27.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.828+0000 7f482c5c5740 1 -- 172.21.15.28:0/1836320039 shutdown_connections 2024-04-23T14:53:27.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.828+0000 7f482c5c5740 1 --2- 172.21.15.28:0/1836320039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55765877ea20 0x557658780e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:27.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.828+0000 7f482c5c5740 1 --2- 172.21.15.28:0/1836320039 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557658774350 0x55765877e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:27.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.828+0000 7f482c5c5740 1 --2- 172.21.15.28:0/1836320039 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557658773a30 0x557658773e10 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:27.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.828+0000 7f482c5c5740 1 -- 172.21.15.28:0/1836320039 >> 172.21.15.28:0/1836320039 conn(0x55765876fd50 msgr2=0x557658770130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:53:27.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.828+0000 7f4822a00640 1 --2- 172.21.15.28:0/1420593025 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f48080a3da0 0x7f48080a6260 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x557658774ff0 tx=0x7f4814005880 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:53:27.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.828+0000 7f482c5c5740 1 -- 172.21.15.28:0/1836320039 shutdown_connections 2024-04-23T14:53:27.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.828+0000 7f482c5c5740 1 -- 172.21.15.28:0/1836320039 wait complete. 2024-04-23T14:53:27.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.829+0000 7f482c5c5740 1 -- start start 2024-04-23T14:53:27.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.829+0000 7f482c5c5740 1 Processor -- start 2024-04-23T14:53:27.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.830+0000 7f482c5c5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557658773a30 0x55765877b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:27.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.830+0000 7f482c5c5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557658774350 0x55765877bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:27.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.830+0000 7f482c5c5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55765877ea20 0x55765877c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:27.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.830+0000 7f482c5c5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557658781810 con 0x557658774350 2024-04-23T14:53:27.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.830+0000 7f482c5c5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x557658692690 con 0x557658773a30 2024-04-23T14:53:27.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.830+0000 7f482c5c5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x557658781990 con 0x55765877ea20 2024-04-23T14:53:27.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.830+0000 7f482c5c5740 10 client.0 _ll_register_callbacks cb 0x5576586ce650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:53:27.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.830+0000 7f4822000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557658774350 0x55765877bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:27.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.830+0000 7f4823400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557658773a30 0x55765877b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:27.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.830+0000 7f4822000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557658774350 0x55765877bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53596/0 (socket says 172.21.15.28:53596) 2024-04-23T14:53:27.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.830+0000 7f4822000640 1 -- 172.21.15.28:0/307769791 learned_addr learned my addr 172.21.15.28:0/307769791 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:53:27.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.830+0000 7f4823400640 1 --2- 172.21.15.28:0/307769791 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55765877ea20 0x55765877c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:27.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.830+0000 7f4822000640 1 -- 172.21.15.28:0/307769791 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55765877ea20 msgr2=0x55765877c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:53:27.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.830+0000 7f4822000640 1 --2- 172.21.15.28:0/307769791 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55765877ea20 0x55765877c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:27.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.830+0000 7f4822000640 1 -- 172.21.15.28:0/307769791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557658773a30 msgr2=0x55765877b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:53:27.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.830+0000 7f4822000640 1 --2- 172.21.15.28:0/307769791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557658773a30 0x55765877b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:27.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.830+0000 7f4822000640 1 -- 172.21.15.28:0/307769791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557658691920 con 0x557658774350 2024-04-23T14:53:27.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.830+0000 7f4823400640 1 --2- 172.21.15.28:0/307769791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557658773a30 0x55765877b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:53:27.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.831+0000 7f4822000640 1 --2- 172.21.15.28:0/307769791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557658774350 0x55765877bee0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4810009900 tx=0x7f4810004440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27c53d756d77908f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:53:27.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.831+0000 7f4803e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-23T14:53:27.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.831+0000 7f4803e00640 1 -- 172.21.15.28:0/307769791 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4810035040 con 0x557658774350 2024-04-23T14:53:27.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.831+0000 7f482c5c5740 1 -- 172.21.15.28:0/307769791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f481c011d80 con 0x557658774350 2024-04-23T14:53:27.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.831+0000 7f4803e00640 1 -- 172.21.15.28:0/307769791 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7f4810005450 con 0x557658774350 2024-04-23T14:53:27.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.831+0000 7f4803e00640 1 -- 172.21.15.28:0/307769791 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f4810005650 con 0x557658774350 2024-04-23T14:53:27.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.831+0000 7f482c5c5740 10 client.16551 fetch_fsmap learned FSMap version 20 2024-04-23T14:53:27.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.831+0000 7f482c5c5740 1 -- 172.21.15.28:0/307769791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5576584b57d0 con 0x557658774350 2024-04-23T14:53:27.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.832+0000 7f4803e00640 1 -- 172.21.15.28:0/307769791 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f481c011d80 con 0x557658774350 2024-04-23T14:53:27.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.832+0000 7f4803e00640 1 -- 172.21.15.28:0/307769791 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f4810003ad0 con 0x557658774350 2024-04-23T14:53:27.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.832+0000 7f482c5c5740 10 client.16551 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:53:27.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.832+0000 7f482c5c5740 10 client.16551 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:53:27.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.832+0000 7f482c5c5740 20 client.16551 populate_metadata read hostname 'smithi028' 2024-04-23T14:53:27.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.832+0000 7f482c5c5740 1 --2- 172.21.15.28:0/307769791 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55765878edb0 0x5576587911a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:27.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.832+0000 7f482c5c5740 4 client.16551 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:53:27.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.832+0000 7f482c5c5740 1 -- 172.21.15.28:0/307769791 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5576586e6c10 con 0x55765878edb0 2024-04-23T14:53:27.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.833+0000 7f4823400640 1 --2- 172.21.15.28:0/307769791 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55765878edb0 0x5576587911a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:27.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.833+0000 7f4823400640 1 --2- 172.21.15.28:0/307769791 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55765878edb0 0x5576587911a0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6f31c9249e9094b server_cookie=a24affbfca8f9f84 in_seq=0 out_seq=0 2024-04-23T14:53:27.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.833+0000 7f4803e00640 10 client.16551 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:53:27.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.834+0000 7f4803e00640 1 -- 172.21.15.28:0/307769791 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5576586e6c10 con 0x55765878edb0 2024-04-23T14:53:27.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.834+0000 7f4803e00640 10 client.16551 handle_command_reply: tid=0 2024-04-23T14:53:27.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.834+0000 7f482c5c5740 1 client.16551 shutdown 2024-04-23T14:53:27.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.835+0000 7f482c5c5740 1 -- 172.21.15.28:0/307769791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557658774350 msgr2=0x55765877bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:27.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.835+0000 7f482c5c5740 1 --2- 172.21.15.28:0/307769791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557658774350 0x55765877bee0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4810009900 tx=0x7f4810004440 comp rx=0 tx=0).stop 2024-04-23T14:53:27.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.835+0000 7f482c5c5740 1 -- 172.21.15.28:0/307769791 shutdown_connections 2024-04-23T14:53:27.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.835+0000 7f482c5c5740 1 --2- 172.21.15.28:0/307769791 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55765878edb0 0x5576587911a0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:27.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.835+0000 7f482c5c5740 1 --2- 172.21.15.28:0/307769791 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55765877ea20 0x55765877c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:27.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.835+0000 7f482c5c5740 1 --2- 172.21.15.28:0/307769791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557658774350 0x55765877bee0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:27.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.835+0000 7f482c5c5740 1 --2- 172.21.15.28:0/307769791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557658773a30 0x55765877b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:27.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.835+0000 7f482c5c5740 1 -- 172.21.15.28:0/307769791 >> 172.21.15.28:0/307769791 conn(0x55765876fd50 msgr2=0x557658775b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:53:27.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.835+0000 7f4803e00640 0 client.16551 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:53:27.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.835+0000 7f482c5c5740 1 -- 172.21.15.28:0/307769791 shutdown_connections 2024-04-23T14:53:27.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.835+0000 7f482c5c5740 1 -- 172.21.15.28:0/307769791 wait complete. 2024-04-23T14:53:27.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.835+0000 7f482c5c5740 20 client.16551 trim_cache size 0 max 16384 2024-04-23T14:53:27.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.859+0000 7f482c5c5740 1 Processor -- start 2024-04-23T14:53:27.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.859+0000 7f482c5c5740 1 -- start start 2024-04-23T14:53:27.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.860+0000 7f482c5c5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55765876fd50 0x55765878a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:27.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.860+0000 7f482c5c5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557658773a30 0x5576587759a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:27.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.860+0000 7f482c5c5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557658774350 0x557658796a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:27.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.860+0000 7f482c5c5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f48241b2280 con 0x557658773a30 2024-04-23T14:53:27.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.860+0000 7f482c5c5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f48241b4ae0 con 0x55765876fd50 2024-04-23T14:53:27.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.860+0000 7f482c5c5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f48241b28d0 con 0x557658774350 2024-04-23T14:53:27.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.860+0000 7f4823400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55765876fd50 0x55765878a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:27.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.860+0000 7f4823400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55765876fd50 0x55765878a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:39004/0 (socket says 172.21.15.28:39004) 2024-04-23T14:53:27.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.860+0000 7f4823400640 1 -- 172.21.15.28:0/3246949168 learned_addr learned my addr 172.21.15.28:0/3246949168 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:53:27.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.860+0000 7f4822000640 1 --2- 172.21.15.28:0/3246949168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557658773a30 0x5576587759a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:27.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.860+0000 7f4823400640 1 --2- 172.21.15.28:0/3246949168 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557658774350 0x557658796a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:27.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.861+0000 7f4823400640 1 -- 172.21.15.28:0/3246949168 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557658774350 msgr2=0x557658796a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:53:27.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.861+0000 7f4823400640 1 --2- 172.21.15.28:0/3246949168 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557658774350 0x557658796a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:27.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.861+0000 7f4823400640 1 -- 172.21.15.28:0/3246949168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557658773a30 msgr2=0x5576587759a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:53:27.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.861+0000 7f4823400640 1 --2- 172.21.15.28:0/3246949168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557658773a30 0x5576587759a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:27.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.861+0000 7f4823400640 1 -- 172.21.15.28:0/3246949168 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576583eb3f0 con 0x55765876fd50 2024-04-23T14:53:27.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.861+0000 7f4823400640 1 --2- 172.21.15.28:0/3246949168 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55765876fd50 0x55765878a700 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f481c010d60 tx=0x7f481c013040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75ea2e7b6b7c6365 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:53:27.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.862+0000 7f480ea00640 1 -- 172.21.15.28:0/3246949168 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f481c032070 con 0x55765876fd50 2024-04-23T14:53:27.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.862+0000 7f480ea00640 1 -- 172.21.15.28:0/3246949168 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f481c024040 con 0x55765876fd50 2024-04-23T14:53:27.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.862+0000 7f480ea00640 1 -- 172.21.15.28:0/3246949168 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f481c02a660 con 0x55765876fd50 2024-04-23T14:53:27.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.863+0000 7f482c5c5740 1 -- 172.21.15.28:0/3246949168 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55765876fd50 msgr2=0x55765878a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:27.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.863+0000 7f482c5c5740 1 --2- 172.21.15.28:0/3246949168 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55765876fd50 0x55765878a700 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f481c010d60 tx=0x7f481c013040 comp rx=0 tx=0).stop 2024-04-23T14:53:27.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.863+0000 7f482c5c5740 1 -- 172.21.15.28:0/3246949168 shutdown_connections 2024-04-23T14:53:27.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.863+0000 7f482c5c5740 1 --2- 172.21.15.28:0/3246949168 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557658774350 0x557658796a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:27.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.863+0000 7f482c5c5740 1 --2- 172.21.15.28:0/3246949168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557658773a30 0x5576587759a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:27.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.863+0000 7f482c5c5740 1 --2- 172.21.15.28:0/3246949168 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55765876fd50 0x55765878a700 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:27.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.863+0000 7f482c5c5740 1 -- 172.21.15.28:0/3246949168 >> 172.21.15.28:0/3246949168 conn(0x55765877a5e0 msgr2=0x557658775290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:53:27.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.863+0000 7f482c5c5740 1 -- 172.21.15.28:0/3246949168 shutdown_connections 2024-04-23T14:53:27.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.863+0000 7f482c5c5740 1 -- 172.21.15.28:0/3246949168 wait complete. 2024-04-23T14:53:27.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.863+0000 7f482c5c5740 1 -- start start 2024-04-23T14:53:27.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.863+0000 7f482c5c5740 1 Processor -- start 2024-04-23T14:53:27.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.864+0000 7f482c5c5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55765876fd50 0x5576587826a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:27.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.864+0000 7f482c5c5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557658773a30 0x557658782be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:27.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.864+0000 7f482c5c5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557658774350 0x557658783120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:27.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.864+0000 7f482c5c5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f48241b2fa0 con 0x557658774350 2024-04-23T14:53:27.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.864+0000 7f482c5c5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f48241b4410 con 0x55765876fd50 2024-04-23T14:53:27.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.864+0000 7f482c5c5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f48241b3670 con 0x557658773a30 2024-04-23T14:53:27.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.864+0000 7f482c5c5740 10 client.0 _ll_register_callbacks cb 0x7f4810009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:53:27.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.864+0000 7f4822000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557658773a30 0x557658782be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:27.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.864+0000 7f4823400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55765876fd50 0x5576587826a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:27.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.864+0000 7f4822000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557658773a30 0x557658782be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:39088/0 (socket says 172.21.15.28:39088) 2024-04-23T14:53:27.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.864+0000 7f4823400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55765876fd50 0x5576587826a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:39014/0 (socket says 172.21.15.28:39014) 2024-04-23T14:53:27.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.864+0000 7f4822000640 1 -- 172.21.15.28:0/1124703046 learned_addr learned my addr 172.21.15.28:0/1124703046 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:53:27.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.864+0000 7f4823400640 1 --2- 172.21.15.28:0/1124703046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557658774350 0x557658783120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:27.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.865+0000 7f4822000640 1 -- 172.21.15.28:0/1124703046 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55765876fd50 msgr2=0x5576587826a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:53:27.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.865+0000 7f4822000640 1 --2- 172.21.15.28:0/1124703046 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55765876fd50 0x5576587826a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:27.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.865+0000 7f4822000640 1 -- 172.21.15.28:0/1124703046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557658774350 msgr2=0x557658783120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:53:27.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.865+0000 7f4822000640 1 --2- 172.21.15.28:0/1124703046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557658774350 0x557658783120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:27.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.865+0000 7f4822000640 1 -- 172.21.15.28:0/1124703046 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576583ab0f0 con 0x557658773a30 2024-04-23T14:53:27.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.865+0000 7f4823400640 1 --2- 172.21.15.28:0/1124703046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557658774350 0x557658783120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:53:27.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.865+0000 7f4822000640 1 --2- 172.21.15.28:0/1124703046 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557658773a30 0x557658782be0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f48100096c0 tx=0x7f4810005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20676bf04a21a7e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:53:27.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.865+0000 7f4803e00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:53:27.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.865+0000 7f4803e00640 1 -- 172.21.15.28:0/1124703046 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4810035070 con 0x557658773a30 2024-04-23T14:53:27.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.865+0000 7f4803e00640 1 -- 172.21.15.28:0/1124703046 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7f4810031a50 con 0x557658773a30 2024-04-23T14:53:27.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.865+0000 7f482c5c5740 1 -- 172.21.15.28:0/1124703046 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f481c024040 con 0x557658773a30 2024-04-23T14:53:27.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.866+0000 7f4803e00640 1 -- 172.21.15.28:0/1124703046 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f481005e050 con 0x557658773a30 2024-04-23T14:53:27.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.866+0000 7f482c5c5740 10 client.26713 fetch_fsmap learned FSMap version 20 2024-04-23T14:53:27.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.866+0000 7f482c5c5740 1 -- 172.21.15.28:0/1124703046 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55765869e810 con 0x557658773a30 2024-04-23T14:53:27.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.866+0000 7f4803e00640 1 -- 172.21.15.28:0/1124703046 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f481c024040 con 0x557658773a30 2024-04-23T14:53:27.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.867+0000 7f4803e00640 1 -- 172.21.15.28:0/1124703046 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f4810004860 con 0x557658773a30 2024-04-23T14:53:27.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.867+0000 7f482c5c5740 10 client.26713 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:53:27.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.867+0000 7f482c5c5740 10 client.26713 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:53:27.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.867+0000 7f482c5c5740 20 client.26713 populate_metadata read hostname 'smithi028' 2024-04-23T14:53:27.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.867+0000 7f482c5c5740 1 --2- 172.21.15.28:0/1124703046 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55765877ea20 0x5576587f7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:27.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.867+0000 7f482c5c5740 4 client.26713 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:53:27.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.867+0000 7f482c5c5740 1 -- 172.21.15.28:0/1124703046 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5576586e0a80 con 0x55765877ea20 2024-04-23T14:53:27.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.867+0000 7f4823400640 1 --2- 172.21.15.28:0/1124703046 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55765877ea20 0x5576587f7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:27.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.867+0000 7f4823400640 1 --2- 172.21.15.28:0/1124703046 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55765877ea20 0x5576587f7ad0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8351291c7d81c9fb server_cookie=94cbc8e7cab92f3c in_seq=0 out_seq=0 2024-04-23T14:53:27.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.867+0000 7f4803e00640 10 client.26713 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:53:27.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.868+0000 7f4803e00640 1 -- 172.21.15.28:0/1124703046 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5576586e0a80 con 0x55765877ea20 2024-04-23T14:53:27.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.868+0000 7f4803e00640 10 client.26713 handle_command_reply: tid=0 2024-04-23T14:53:27.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.868+0000 7f482c5c5740 1 client.26713 shutdown 2024-04-23T14:53:27.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.869+0000 7f482c5c5740 1 -- 172.21.15.28:0/1124703046 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557658773a30 msgr2=0x557658782be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:27.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.869+0000 7f482c5c5740 1 --2- 172.21.15.28:0/1124703046 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557658773a30 0x557658782be0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f48100096c0 tx=0x7f4810005850 comp rx=0 tx=0).stop 2024-04-23T14:53:27.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.869+0000 7f482c5c5740 1 -- 172.21.15.28:0/1124703046 shutdown_connections 2024-04-23T14:53:27.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.869+0000 7f482c5c5740 1 --2- 172.21.15.28:0/1124703046 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55765877ea20 0x5576587f7ad0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:27.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.869+0000 7f482c5c5740 1 --2- 172.21.15.28:0/1124703046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557658774350 0x557658783120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:27.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.869+0000 7f482c5c5740 1 --2- 172.21.15.28:0/1124703046 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557658773a30 0x557658782be0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:27.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.869+0000 7f482c5c5740 1 --2- 172.21.15.28:0/1124703046 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55765876fd50 0x5576587826a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:27.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.869+0000 7f4803e00640 0 client.26713 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:53:27.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.869+0000 7f482c5c5740 1 -- 172.21.15.28:0/1124703046 >> 172.21.15.28:0/1124703046 conn(0x55765877a5e0 msgr2=0x55765877c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:53:27.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.869+0000 7f482c5c5740 1 -- 172.21.15.28:0/1124703046 shutdown_connections 2024-04-23T14:53:27.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.869+0000 7f482c5c5740 1 -- 172.21.15.28:0/1124703046 wait complete. 2024-04-23T14:53:27.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.869+0000 7f482c5c5740 20 client.26713 trim_cache size 0 max 16384 2024-04-23T14:53:27.876 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:53:27.876 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-23T14:53:27.876 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-23T14:53:27.876 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:53:27.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.870+0000 7f4823e00640 1 -- 172.21.15.28:0/1420593025 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f48080a3da0 msgr2=0x7f48080a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:27.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.870+0000 7f4823e00640 1 --2- 172.21.15.28:0/1420593025 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f48080a3da0 0x7f48080a6260 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x557658774ff0 tx=0x7f4814005880 comp rx=0 tx=0).stop 2024-04-23T14:53:27.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.870+0000 7f4823e00640 1 -- 172.21.15.28:0/1420593025 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4824102110 msgr2=0x7f482410e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:27.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.870+0000 7f4823e00640 1 --2- 172.21.15.28:0/1420593025 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4824102110 0x7f482410e170 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f481400d950 tx=0x7f481400de10 comp rx=0 tx=0).stop 2024-04-23T14:53:27.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.870+0000 7f4823e00640 1 -- 172.21.15.28:0/1420593025 shutdown_connections 2024-04-23T14:53:27.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.870+0000 7f4823e00640 1 --2- 172.21.15.28:0/1420593025 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f48080a3da0 0x7f48080a6260 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:27.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.870+0000 7f4823e00640 1 --2- 172.21.15.28:0/1420593025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48241a3670 0x7f48241a3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:27.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.870+0000 7f4823e00640 1 --2- 172.21.15.28:0/1420593025 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f48241071e0 0x7f482410e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:27.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.870+0000 7f4823e00640 1 --2- 172.21.15.28:0/1420593025 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4824102110 0x7f482410e170 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:27.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.870+0000 7f4823e00640 1 -- 172.21.15.28:0/1420593025 >> 172.21.15.28:0/1420593025 conn(0x7f48240fd9d0 msgr2=0x7f48240fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:53:27.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.870+0000 7f4823e00640 1 -- 172.21.15.28:0/1420593025 shutdown_connections 2024-04-23T14:53:27.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:27.870+0000 7f4823e00640 1 -- 172.21.15.28:0/1420593025 wait complete. 2024-04-23T14:53:27.890 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-23T14:53:27.891 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:53:28.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.109+0000 7f1aa9600640 1 -- 172.21.15.28:0/1550514103 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1aa4069db0 msgr2=0x7f1aa4106a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:28.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.109+0000 7f1aa9600640 1 --2- 172.21.15.28:0/1550514103 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1aa4069db0 0x7f1aa4106a70 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c009900 tx=0x7f1a8c02a3f0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.109+0000 7f1aa9600640 1 -- 172.21.15.28:0/1550514103 shutdown_connections 2024-04-23T14:53:28.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.109+0000 7f1aa9600640 1 --2- 172.21.15.28:0/1550514103 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1aa4069db0 0x7f1aa4106a70 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.109+0000 7f1aa9600640 1 --2- 172.21.15.28:0/1550514103 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1aa41071a0 0x7f1aa410ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.109+0000 7f1aa9600640 1 --2- 172.21.15.28:0/1550514103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa4069400 0x7f1aa40697e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.109+0000 7f1aa9600640 1 -- 172.21.15.28:0/1550514103 >> 172.21.15.28:0/1550514103 conn(0x7f1aa4076840 msgr2=0x7f1aa4076c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:53:28.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.109+0000 7f1aa9600640 1 -- 172.21.15.28:0/1550514103 shutdown_connections 2024-04-23T14:53:28.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.110+0000 7f1aa9600640 1 -- 172.21.15.28:0/1550514103 wait complete. 2024-04-23T14:53:28.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.111+0000 7f1aa9600640 1 Processor -- start 2024-04-23T14:53:28.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.111+0000 7f1aa9600640 1 -- start start 2024-04-23T14:53:28.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.111+0000 7f1aa9600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa4069400 0x7f1aa4198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:28.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.111+0000 7f1aa9600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1aa41071a0 0x7f1aa4198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:28.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.111+0000 7f1aa9600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1aa419cfd0 0x7f1aa419d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:28.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.111+0000 7f1aa9600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa4112210 con 0x7f1aa4069400 2024-04-23T14:53:28.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.111+0000 7f1aa9600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa4112090 con 0x7f1aa41071a0 2024-04-23T14:53:28.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.111+0000 7f1aa9600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa4112390 con 0x7f1aa419cfd0 2024-04-23T14:53:28.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.111+0000 7f1a9b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1aa41071a0 0x7f1aa4198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:28.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.111+0000 7f1aa3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa4069400 0x7f1aa4198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:28.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.112+0000 7f1a9b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1aa41071a0 0x7f1aa4198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:39026/0 (socket says 172.21.15.28:39026) 2024-04-23T14:53:28.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.112+0000 7f1a9b400640 1 -- 172.21.15.28:0/2358886172 learned_addr learned my addr 172.21.15.28:0/2358886172 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:53:28.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.112+0000 7f1aa8c00640 1 --2- 172.21.15.28:0/2358886172 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1aa419cfd0 0x7f1aa419d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:28.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.112+0000 7f1a9b400640 1 -- 172.21.15.28:0/2358886172 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1aa419cfd0 msgr2=0x7f1aa419d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:28.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.112+0000 7f1a9b400640 1 --2- 172.21.15.28:0/2358886172 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1aa419cfd0 0x7f1aa419d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.112+0000 7f1a9b400640 1 -- 172.21.15.28:0/2358886172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa4069400 msgr2=0x7f1aa4198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:28.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.112+0000 7f1a9b400640 1 --2- 172.21.15.28:0/2358886172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa4069400 0x7f1aa4198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.112+0000 7f1a9b400640 1 -- 172.21.15.28:0/2358886172 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aa419dac0 con 0x7f1aa41071a0 2024-04-23T14:53:28.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.112+0000 7f1a9b400640 1 --2- 172.21.15.28:0/2358886172 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1aa41071a0 0x7f1aa4198c50 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c02a910 tx=0x7f1a8c003b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:53:28.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.113+0000 7f1aa1600640 1 -- 172.21.15.28:0/2358886172 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c03f070 con 0x7f1aa41071a0 2024-04-23T14:53:28.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.113+0000 7f1aa1600640 1 -- 172.21.15.28:0/2358886172 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a8c031c90 con 0x7f1aa41071a0 2024-04-23T14:53:28.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.113+0000 7f1aa1600640 1 -- 172.21.15.28:0/2358886172 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c035030 con 0x7f1aa41071a0 2024-04-23T14:53:28.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.113+0000 7f1aa9600640 1 -- 172.21.15.28:0/2358886172 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1aa419dd50 con 0x7f1aa41071a0 2024-04-23T14:53:28.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.113+0000 7f1aa9600640 1 -- 172.21.15.28:0/2358886172 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1aa41a54c0 con 0x7f1aa41071a0 2024-04-23T14:53:28.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.114+0000 7f1aa9600640 1 -- 172.21.15.28:0/2358886172 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1aa4068d40 con 0x7f1aa41071a0 2024-04-23T14:53:28.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.116+0000 7f1aa1600640 1 -- 172.21.15.28:0/2358886172 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f1a8c042070 con 0x7f1aa41071a0 2024-04-23T14:53:28.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.117+0000 7f1aa1600640 1 --2- 172.21.15.28:0/2358886172 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f1a880a3da0 0x7f1a880a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:28.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.118+0000 7f1aa1600640 1 -- 172.21.15.28:0/2358886172 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7f1a8c0f0d90 con 0x7f1aa41071a0 2024-04-23T14:53:28.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.118+0000 7f1aa3e00640 1 --2- 172.21.15.28:0/2358886172 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f1a880a3da0 0x7f1a880a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:28.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.118+0000 7f1aa3e00640 1 --2- 172.21.15.28:0/2358886172 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f1a880a3da0 0x7f1a880a6260 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1a94005fd0 tx=0x7f1a94009500 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:53:28.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.122+0000 7f1aa1600640 1 -- 172.21.15.28:0/2358886172 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1a8c0bcd70 con 0x7f1aa41071a0 2024-04-23T14:53:28.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.393+0000 7f1aa9600640 1 -- 172.21.15.28:0/2358886172 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1aa4069db0 con 0x7f1aa41071a0 2024-04-23T14:53:28.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.394+0000 7f1aa1600640 1 -- 172.21.15.28:0/2358886172 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7f1a8c0c1070 con 0x7f1aa41071a0 2024-04-23T14:53:28.394 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:53:28.394 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:53:28.394 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:53:28.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.396+0000 7f1aa9600640 1 -- 172.21.15.28:0/2358886172 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f1a880a3da0 msgr2=0x7f1a880a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:28.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.396+0000 7f1aa9600640 1 --2- 172.21.15.28:0/2358886172 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f1a880a3da0 0x7f1a880a6260 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1a94005fd0 tx=0x7f1a94009500 comp rx=0 tx=0).stop 2024-04-23T14:53:28.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.396+0000 7f1aa9600640 1 -- 172.21.15.28:0/2358886172 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1aa41071a0 msgr2=0x7f1aa4198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:28.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.396+0000 7f1aa9600640 1 --2- 172.21.15.28:0/2358886172 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1aa41071a0 0x7f1aa4198c50 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c02a910 tx=0x7f1a8c003b20 comp rx=0 tx=0).stop 2024-04-23T14:53:28.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.396+0000 7f1aa9600640 1 -- 172.21.15.28:0/2358886172 shutdown_connections 2024-04-23T14:53:28.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.396+0000 7f1aa9600640 1 --2- 172.21.15.28:0/2358886172 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f1a880a3da0 0x7f1a880a6260 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.396+0000 7f1aa9600640 1 --2- 172.21.15.28:0/2358886172 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1aa41071a0 0x7f1aa4198c50 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.396+0000 7f1aa9600640 1 --2- 172.21.15.28:0/2358886172 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1aa419cfd0 0x7f1aa419d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.396+0000 7f1aa9600640 1 --2- 172.21.15.28:0/2358886172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa4069400 0x7f1aa4198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.396+0000 7f1aa9600640 1 -- 172.21.15.28:0/2358886172 >> 172.21.15.28:0/2358886172 conn(0x7f1aa4076840 msgr2=0x7f1aa41082d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:53:28.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.397+0000 7f1aa9600640 1 -- 172.21.15.28:0/2358886172 shutdown_connections 2024-04-23T14:53:28.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.397+0000 7f1aa9600640 1 -- 172.21.15.28:0/2358886172 wait complete. 2024-04-23T14:53:28.418 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-23T14:53:28.519 INFO:tasks.workunit.client.0.smithi028.stdout: 1 797824 8.87 MB/sec execute 344 sec latency 17.787 ms 2024-04-23T14:53:28.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.645+0000 7f4707e00640 1 -- 172.21.15.28:0/2686999687 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4708103060 msgr2=0x7f4708103440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:28.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.645+0000 7f4707e00640 1 --2- 172.21.15.28:0/2686999687 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4708103060 0x7f4708103440 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f46f8009900 tx=0x7f46f802a480 comp rx=0 tx=0).stop 2024-04-23T14:53:28.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.645+0000 7f4707e00640 1 -- 172.21.15.28:0/2686999687 shutdown_connections 2024-04-23T14:53:28.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.645+0000 7f4707e00640 1 --2- 172.21.15.28:0/2686999687 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f470810c480 0x7f470810e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.645+0000 7f4707e00640 1 --2- 172.21.15.28:0/2686999687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4708103980 0x7f470810be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.645+0000 7f4707e00640 1 --2- 172.21.15.28:0/2686999687 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4708103060 0x7f4708103440 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.645+0000 7f4707e00640 1 -- 172.21.15.28:0/2686999687 >> 172.21.15.28:0/2686999687 conn(0x7f47080fc880 msgr2=0x7f47080feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:53:28.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.645+0000 7f4707e00640 1 -- 172.21.15.28:0/2686999687 shutdown_connections 2024-04-23T14:53:28.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.646+0000 7f4707e00640 1 -- 172.21.15.28:0/2686999687 wait complete. 2024-04-23T14:53:28.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.647+0000 7f4707e00640 1 Processor -- start 2024-04-23T14:53:28.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.647+0000 7f4707e00640 1 -- start start 2024-04-23T14:53:28.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.647+0000 7f4707e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4708103060 0x7f470819e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:28.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.647+0000 7f4707e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4708103980 0x7f470819e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:28.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.647+0000 7f4707e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f470810c480 0x7f47081985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:28.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.647+0000 7f4707e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4708112290 con 0x7f470810c480 2024-04-23T14:53:28.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.647+0000 7f4707e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4708112110 con 0x7f4708103060 2024-04-23T14:53:28.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.648+0000 7f4707e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4708112410 con 0x7f4708103980 2024-04-23T14:53:28.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.648+0000 7f4706a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4708103060 0x7f470819e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:28.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.648+0000 7f4706000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4708103980 0x7f470819e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:28.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.648+0000 7f4706a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4708103060 0x7f470819e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:39054/0 (socket says 172.21.15.28:39054) 2024-04-23T14:53:28.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.648+0000 7f4706a00640 1 -- 172.21.15.28:0/4223611715 learned_addr learned my addr 172.21.15.28:0/4223611715 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:53:28.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.648+0000 7f4707400640 1 --2- 172.21.15.28:0/4223611715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f470810c480 0x7f47081985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:28.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.648+0000 7f4706a00640 1 -- 172.21.15.28:0/4223611715 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4708103980 msgr2=0x7f470819e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:28.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.648+0000 7f4706a00640 1 --2- 172.21.15.28:0/4223611715 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4708103980 0x7f470819e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.648+0000 7f4706a00640 1 -- 172.21.15.28:0/4223611715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f470810c480 msgr2=0x7f47081985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:28.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.648+0000 7f4706a00640 1 --2- 172.21.15.28:0/4223611715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f470810c480 0x7f47081985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.648+0000 7f4706a00640 1 -- 172.21.15.28:0/4223611715 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4708198d30 con 0x7f4708103060 2024-04-23T14:53:28.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.649+0000 7f4706a00640 1 --2- 172.21.15.28:0/4223611715 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4708103060 0x7f470819e400 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f46f8004770 tx=0x7f46f8003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:53:28.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.649+0000 7f46ff400640 1 -- 172.21.15.28:0/4223611715 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f803f070 con 0x7f4708103060 2024-04-23T14:53:28.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.649+0000 7f46ff400640 1 -- 172.21.15.28:0/4223611715 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46f8031ca0 con 0x7f4708103060 2024-04-23T14:53:28.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.649+0000 7f46ff400640 1 -- 172.21.15.28:0/4223611715 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f8035030 con 0x7f4708103060 2024-04-23T14:53:28.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.649+0000 7f4707e00640 1 -- 172.21.15.28:0/4223611715 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4708198fc0 con 0x7f4708103060 2024-04-23T14:53:28.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.649+0000 7f4707e00640 1 -- 172.21.15.28:0/4223611715 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47081a52c0 con 0x7f4708103060 2024-04-23T14:53:28.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.652+0000 7f46ff400640 1 -- 172.21.15.28:0/4223611715 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f46f8042070 con 0x7f4708103060 2024-04-23T14:53:28.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.653+0000 7f471087b740 1 Processor -- start 2024-04-23T14:53:28.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.653+0000 7f471087b740 1 -- start start 2024-04-23T14:53:28.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.653+0000 7f471087b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556786752b40 0x556786752f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:28.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.653+0000 7f471087b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556786753460 0x55678675d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:28.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.653+0000 7f471087b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55678675db30 0x55678675ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:28.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.653+0000 7f471087b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5567866b81d0 con 0x556786753460 2024-04-23T14:53:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.653+0000 7f471087b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55678665e450 con 0x55678675db30 2024-04-23T14:53:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.653+0000 7f471087b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5567866b4190 con 0x556786752b40 2024-04-23T14:53:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.654+0000 7f4706000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55678675db30 0x55678675ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.654+0000 7f4707400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556786752b40 0x556786752f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.654+0000 7f4706000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55678675db30 0x55678675ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:39058/0 (socket says 172.21.15.28:39058) 2024-04-23T14:53:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.654+0000 7f4706000640 1 -- 172.21.15.28:0/476005560 learned_addr learned my addr 172.21.15.28:0/476005560 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:53:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.654+0000 7f4706a00640 1 --2- 172.21.15.28:0/476005560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556786753460 0x55678675d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:28.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.654+0000 7f4706000640 1 -- 172.21.15.28:0/476005560 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556786752b40 msgr2=0x556786752f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:53:28.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.654+0000 7f4706000640 1 --2- 172.21.15.28:0/476005560 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556786752b40 0x556786752f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.654+0000 7f4706000640 1 -- 172.21.15.28:0/476005560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556786753460 msgr2=0x55678675d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:53:28.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.654+0000 7f4706000640 1 --2- 172.21.15.28:0/476005560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556786753460 0x55678675d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.654+0000 7f4706000640 1 -- 172.21.15.28:0/476005560 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55678671b310 con 0x55678675db30 2024-04-23T14:53:28.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.654+0000 7f4706a00640 1 --2- 172.21.15.28:0/476005560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556786753460 0x55678675d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:53:28.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.654+0000 7f46ff400640 1 --2- 172.21.15.28:0/4223611715 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f46e80a3d20 0x7f46e80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:28.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.654+0000 7f4706000640 1 --2- 172.21.15.28:0/476005560 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55678675db30 0x55678675ff20 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f46f0009fc0 tx=0x7f46f00118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80390b2741347d57 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:53:28.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.655+0000 7f46ff400640 1 -- 172.21.15.28:0/4223611715 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7f46f80f0eb0 con 0x7f4708103060 2024-04-23T14:53:28.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.655+0000 7f46f7e00640 1 -- 172.21.15.28:0/476005560 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f0004d20 con 0x55678675db30 2024-04-23T14:53:28.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.655+0000 7f46f7e00640 1 -- 172.21.15.28:0/476005560 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46f0004050 con 0x55678675db30 2024-04-23T14:53:28.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.655+0000 7f4707400640 1 --2- 172.21.15.28:0/4223611715 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f46e80a3d20 0x7f46e80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:28.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.655+0000 7f46f7e00640 1 -- 172.21.15.28:0/476005560 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f0005260 con 0x55678675db30 2024-04-23T14:53:28.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.656+0000 7f471087b740 1 -- 172.21.15.28:0/476005560 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55678675db30 msgr2=0x55678675ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:28.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.656+0000 7f4707400640 1 --2- 172.21.15.28:0/4223611715 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f46e80a3d20 0x7f46e80a61e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4700005fd0 tx=0x7f4700005cc0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:53:28.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.656+0000 7f471087b740 1 --2- 172.21.15.28:0/476005560 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55678675db30 0x55678675ff20 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f46f0009fc0 tx=0x7f46f00118d0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.656+0000 7f471087b740 1 -- 172.21.15.28:0/476005560 shutdown_connections 2024-04-23T14:53:28.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.656+0000 7f471087b740 1 --2- 172.21.15.28:0/476005560 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55678675db30 0x55678675ff20 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.656+0000 7f471087b740 1 --2- 172.21.15.28:0/476005560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556786753460 0x55678675d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.656+0000 7f471087b740 1 --2- 172.21.15.28:0/476005560 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556786752b40 0x556786752f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.656+0000 7f471087b740 1 -- 172.21.15.28:0/476005560 >> 172.21.15.28:0/476005560 conn(0x55678674ee60 msgr2=0x55678674f240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:53:28.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.656+0000 7f471087b740 1 -- 172.21.15.28:0/476005560 shutdown_connections 2024-04-23T14:53:28.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.656+0000 7f471087b740 1 -- 172.21.15.28:0/476005560 wait complete. 2024-04-23T14:53:28.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.656+0000 7f471087b740 1 -- start start 2024-04-23T14:53:28.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.657+0000 7f471087b740 1 Processor -- start 2024-04-23T14:53:28.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.657+0000 7f471087b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556786752b40 0x55678675d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:28.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.657+0000 7f471087b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556786753460 0x55678675a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:28.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.657+0000 7f471087b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55678675db30 0x55678675ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:28.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.657+0000 7f471087b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5567867609c0 con 0x55678675db30 2024-04-23T14:53:28.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.657+0000 7f471087b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x556786671810 con 0x556786752b40 2024-04-23T14:53:28.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.657+0000 7f471087b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556786760b40 con 0x556786753460 2024-04-23T14:53:28.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.657+0000 7f471087b740 10 client.0 _ll_register_callbacks cb 0x55678663f140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:53:28.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.657+0000 7f4706000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556786753460 0x55678675a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:28.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.657+0000 7f4706a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556786752b40 0x55678675d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:28.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.658+0000 7f4706000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556786753460 0x55678675a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:39156/0 (socket says 172.21.15.28:39156) 2024-04-23T14:53:28.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.658+0000 7f4706a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556786752b40 0x55678675d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:39062/0 (socket says 172.21.15.28:39062) 2024-04-23T14:53:28.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.658+0000 7f4706000640 1 -- 172.21.15.28:0/1519469309 learned_addr learned my addr 172.21.15.28:0/1519469309 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:53:28.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.658+0000 7f4707400640 1 --2- 172.21.15.28:0/1519469309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55678675db30 0x55678675ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:28.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.658+0000 7f4706a00640 1 -- 172.21.15.28:0/1519469309 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556786753460 msgr2=0x55678675a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:53:28.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.658+0000 7f4706a00640 1 --2- 172.21.15.28:0/1519469309 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556786753460 0x55678675a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.658+0000 7f4706a00640 1 -- 172.21.15.28:0/1519469309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55678675db30 msgr2=0x55678675ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:53:28.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.658+0000 7f4706a00640 1 --2- 172.21.15.28:0/1519469309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55678675db30 0x55678675ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.658+0000 7f4706a00640 1 -- 172.21.15.28:0/1519469309 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567866823f0 con 0x556786752b40 2024-04-23T14:53:28.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.658+0000 7f4706a00640 1 --2- 172.21.15.28:0/1519469309 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556786752b40 0x55678675d4e0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f46f80ee590 tx=0x7f46f802a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91754d0312532c9d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:53:28.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.658+0000 7f46f4c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:53:28.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.658+0000 7f46f4c00640 1 -- 172.21.15.28:0/1519469309 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f8049050 con 0x556786752b40 2024-04-23T14:53:28.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.658+0000 7f46f4c00640 1 -- 172.21.15.28:0/1519469309 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7f46f803b4f0 con 0x556786752b40 2024-04-23T14:53:28.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.659+0000 7f471087b740 1 -- 172.21.15.28:0/1519469309 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46f0004050 con 0x556786752b40 2024-04-23T14:53:28.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.659+0000 7f46f4c00640 1 -- 172.21.15.28:0/1519469309 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f46f80bc9d0 con 0x556786752b40 2024-04-23T14:53:28.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.659+0000 7f471087b740 10 client.26753 fetch_fsmap learned FSMap version 20 2024-04-23T14:53:28.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.659+0000 7f471087b740 1 -- 172.21.15.28:0/1519469309 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5567863ca3f0 con 0x556786752b40 2024-04-23T14:53:28.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.659+0000 7f46f4c00640 1 -- 172.21.15.28:0/1519469309 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46f0004050 con 0x556786752b40 2024-04-23T14:53:28.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.659+0000 7f46f4c00640 1 -- 172.21.15.28:0/1519469309 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f46f80eece0 con 0x556786752b40 2024-04-23T14:53:28.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.659+0000 7f471087b740 10 client.26753 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:53:28.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.659+0000 7f471087b740 10 client.26753 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:53:28.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.659+0000 7f471087b740 20 client.26753 populate_metadata read hostname 'smithi028' 2024-04-23T14:53:28.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.659+0000 7f471087b740 1 --2- 172.21.15.28:0/1519469309 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55678676dc90 0x556786770150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:28.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.659+0000 7f471087b740 4 client.26753 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:53:28.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.659+0000 7f471087b740 1 -- 172.21.15.28:0/1519469309 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5567866c5bf0 con 0x55678676dc90 2024-04-23T14:53:28.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.660+0000 7f4706000640 1 --2- 172.21.15.28:0/1519469309 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55678676dc90 0x556786770150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:28.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.660+0000 7f4706000640 1 --2- 172.21.15.28:0/1519469309 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55678676dc90 0x556786770150 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=452542790dca6f80 server_cookie=8e75ee2586f66503 in_seq=0 out_seq=0 2024-04-23T14:53:28.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.660+0000 7f46f4c00640 10 client.26753 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:53:28.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.661+0000 7f46f4c00640 1 -- 172.21.15.28:0/1519469309 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5567866c5bf0 con 0x55678676dc90 2024-04-23T14:53:28.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.661+0000 7f46f4c00640 10 client.26753 handle_command_reply: tid=0 2024-04-23T14:53:28.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.661+0000 7f471087b740 1 client.26753 shutdown 2024-04-23T14:53:28.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.661+0000 7f471087b740 1 -- 172.21.15.28:0/1519469309 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556786752b40 msgr2=0x55678675d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:28.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.661+0000 7f471087b740 1 --2- 172.21.15.28:0/1519469309 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556786752b40 0x55678675d4e0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f46f80ee590 tx=0x7f46f802a480 comp rx=0 tx=0).stop 2024-04-23T14:53:28.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.661+0000 7f471087b740 1 -- 172.21.15.28:0/1519469309 shutdown_connections 2024-04-23T14:53:28.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.661+0000 7f471087b740 1 --2- 172.21.15.28:0/1519469309 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55678676dc90 0x556786770150 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.661+0000 7f471087b740 1 --2- 172.21.15.28:0/1519469309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55678675db30 0x55678675ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.661+0000 7f471087b740 1 --2- 172.21.15.28:0/1519469309 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556786753460 0x55678675a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.661+0000 7f471087b740 1 --2- 172.21.15.28:0/1519469309 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556786752b40 0x55678675d4e0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.661+0000 7f46f4c00640 0 client.26753 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:53:28.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.661+0000 7f471087b740 1 -- 172.21.15.28:0/1519469309 >> 172.21.15.28:0/1519469309 conn(0x55678674ee60 msgr2=0x556786754b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:53:28.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.661+0000 7f471087b740 1 -- 172.21.15.28:0/1519469309 shutdown_connections 2024-04-23T14:53:28.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.662+0000 7f471087b740 1 -- 172.21.15.28:0/1519469309 wait complete. 2024-04-23T14:53:28.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.662+0000 7f471087b740 20 client.26753 trim_cache size 0 max 16384 2024-04-23T14:53:28.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.680+0000 7f471087b740 1 Processor -- start 2024-04-23T14:53:28.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.680+0000 7f471087b740 1 -- start start 2024-04-23T14:53:28.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.680+0000 7f471087b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556786752b40 0x5567867698e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:28.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.680+0000 7f471087b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556786753460 0x55678675d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:28.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.680+0000 7f471087b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55678675db30 0x556786775a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:28.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.680+0000 7f471087b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f47081abbc0 con 0x55678675db30 2024-04-23T14:53:28.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.680+0000 7f471087b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f47081ae420 con 0x556786752b40 2024-04-23T14:53:28.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.680+0000 7f471087b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f47081ac210 con 0x556786753460 2024-04-23T14:53:28.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.681+0000 7f4706a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556786752b40 0x5567867698e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:28.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.681+0000 7f4707400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55678675db30 0x556786775a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:28.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.681+0000 7f4707400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55678675db30 0x556786775a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53670/0 (socket says 172.21.15.28:53670) 2024-04-23T14:53:28.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.681+0000 7f4706a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556786752b40 0x5567867698e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:39078/0 (socket says 172.21.15.28:39078) 2024-04-23T14:53:28.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.681+0000 7f4707400640 1 -- 172.21.15.28:0/555720604 learned_addr learned my addr 172.21.15.28:0/555720604 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:53:28.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.681+0000 7f4706000640 1 --2- 172.21.15.28:0/555720604 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556786753460 0x55678675d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:28.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.681+0000 7f4706a00640 1 -- 172.21.15.28:0/555720604 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556786753460 msgr2=0x55678675d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:53:28.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.681+0000 7f4706a00640 1 --2- 172.21.15.28:0/555720604 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556786753460 0x55678675d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.681+0000 7f4706a00640 1 -- 172.21.15.28:0/555720604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55678675db30 msgr2=0x556786775a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:53:28.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.681+0000 7f4706a00640 1 --2- 172.21.15.28:0/555720604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55678675db30 0x556786775a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.681+0000 7f4706a00640 1 -- 172.21.15.28:0/555720604 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567864947d0 con 0x556786752b40 2024-04-23T14:53:28.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.681+0000 7f4706000640 1 --2- 172.21.15.28:0/555720604 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556786753460 0x55678675d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:53:28.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.682+0000 7f4706a00640 1 --2- 172.21.15.28:0/555720604 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556786752b40 0x5567867698e0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f46f80b92f0 tx=0x7f46f80bb3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b1bb5bdc40382da server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:53:28.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.682+0000 7f46f7400640 1 -- 172.21.15.28:0/555720604 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f8049070 con 0x556786752b40 2024-04-23T14:53:28.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.682+0000 7f46f7400640 1 -- 172.21.15.28:0/555720604 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46f80bce90 con 0x556786752b40 2024-04-23T14:53:28.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.682+0000 7f46f7400640 1 -- 172.21.15.28:0/555720604 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f8005d50 con 0x556786752b40 2024-04-23T14:53:28.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.683+0000 7f471087b740 1 -- 172.21.15.28:0/555720604 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556786752b40 msgr2=0x5567867698e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:28.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.683+0000 7f471087b740 1 --2- 172.21.15.28:0/555720604 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556786752b40 0x5567867698e0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f46f80b92f0 tx=0x7f46f80bb3a0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.683+0000 7f471087b740 1 -- 172.21.15.28:0/555720604 shutdown_connections 2024-04-23T14:53:28.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.683+0000 7f471087b740 1 --2- 172.21.15.28:0/555720604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55678675db30 0x556786775a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.683+0000 7f471087b740 1 --2- 172.21.15.28:0/555720604 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556786753460 0x55678675d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.683+0000 7f471087b740 1 --2- 172.21.15.28:0/555720604 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556786752b40 0x5567867698e0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.683+0000 7f471087b740 1 -- 172.21.15.28:0/555720604 >> 172.21.15.28:0/555720604 conn(0x55678674ee60 msgr2=0x556786755ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:53:28.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.683+0000 7f471087b740 1 -- 172.21.15.28:0/555720604 shutdown_connections 2024-04-23T14:53:28.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.683+0000 7f471087b740 1 -- 172.21.15.28:0/555720604 wait complete. 2024-04-23T14:53:28.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.683+0000 7f471087b740 1 -- start start 2024-04-23T14:53:28.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.684+0000 7f471087b740 1 Processor -- start 2024-04-23T14:53:28.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.684+0000 7f471087b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556786752b40 0x556786766310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:28.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.684+0000 7f471087b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556786753460 0x556786757c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:28.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.684+0000 7f471087b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55678675db30 0x5567867583c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:28.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.684+0000 7f471087b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f47081ac8e0 con 0x55678675db30 2024-04-23T14:53:28.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.684+0000 7f471087b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f47081add50 con 0x556786753460 2024-04-23T14:53:28.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.684+0000 7f471087b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f47081acfb0 con 0x556786752b40 2024-04-23T14:53:28.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.684+0000 7f471087b740 10 client.0 _ll_register_callbacks cb 0x7f46f8005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:53:28.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.684+0000 7f4706000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556786753460 0x556786757c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:28.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.684+0000 7f4706a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556786752b40 0x556786766310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:28.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.684+0000 7f4706000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556786753460 0x556786757c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:39090/0 (socket says 172.21.15.28:39090) 2024-04-23T14:53:28.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.684+0000 7f4706a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556786752b40 0x556786766310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:39188/0 (socket says 172.21.15.28:39188) 2024-04-23T14:53:28.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.684+0000 7f4706000640 1 -- 172.21.15.28:0/2652981059 learned_addr learned my addr 172.21.15.28:0/2652981059 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:53:28.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.684+0000 7f4707400640 1 --2- 172.21.15.28:0/2652981059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55678675db30 0x5567867583c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:28.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.685+0000 7f4706000640 1 -- 172.21.15.28:0/2652981059 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556786752b40 msgr2=0x556786766310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:53:28.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.685+0000 7f4706000640 1 --2- 172.21.15.28:0/2652981059 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556786752b40 0x556786766310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.685+0000 7f4706000640 1 -- 172.21.15.28:0/2652981059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55678675db30 msgr2=0x5567867583c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:53:28.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.685+0000 7f4706000640 1 --2- 172.21.15.28:0/2652981059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55678675db30 0x5567867583c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.685+0000 7f4706000640 1 -- 172.21.15.28:0/2652981059 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55678675e990 con 0x556786753460 2024-04-23T14:53:28.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.685+0000 7f4706000640 1 --2- 172.21.15.28:0/2652981059 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556786753460 0x556786757c80 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f46f0011dd0 tx=0x7f46f00098e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bae621b9c9ec945 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:53:28.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.685+0000 7f46f4c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:53:28.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.685+0000 7f46f4c00640 1 -- 172.21.15.28:0/2652981059 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f002ab50 con 0x556786753460 2024-04-23T14:53:28.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.685+0000 7f46f4c00640 1 -- 172.21.15.28:0/2652981059 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7f46f002acf0 con 0x556786753460 2024-04-23T14:53:28.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.685+0000 7f471087b740 1 -- 172.21.15.28:0/2652981059 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46f80bce90 con 0x556786753460 2024-04-23T14:53:28.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.686+0000 7f46f4c00640 1 -- 172.21.15.28:0/2652981059 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f46f0004990 con 0x556786753460 2024-04-23T14:53:28.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.686+0000 7f471087b740 10 client.26759 fetch_fsmap learned FSMap version 20 2024-04-23T14:53:28.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.686+0000 7f471087b740 1 -- 172.21.15.28:0/2652981059 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5567867d8a10 con 0x556786753460 2024-04-23T14:53:28.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.686+0000 7f46f4c00640 1 -- 172.21.15.28:0/2652981059 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46f80bce90 con 0x556786753460 2024-04-23T14:53:28.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.686+0000 7f46f4c00640 1 -- 172.21.15.28:0/2652981059 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f46f002a1a0 con 0x556786753460 2024-04-23T14:53:28.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.686+0000 7f471087b740 10 client.26759 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:53:28.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.686+0000 7f471087b740 10 client.26759 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:53:28.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.686+0000 7f471087b740 20 client.26759 populate_metadata read hostname 'smithi028' 2024-04-23T14:53:28.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.686+0000 7f471087b740 1 --2- 172.21.15.28:0/2652981059 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5567867d9020 0x5567867db440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:28.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.686+0000 7f471087b740 4 client.26759 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-23T14:53:28.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.686+0000 7f471087b740 1 -- 172.21.15.28:0/2652981059 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5567866bfa60 con 0x5567867d9020 2024-04-23T14:53:28.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.687+0000 7f4706a00640 1 --2- 172.21.15.28:0/2652981059 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5567867d9020 0x5567867db440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:28.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.687+0000 7f4706a00640 1 --2- 172.21.15.28:0/2652981059 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5567867d9020 0x5567867db440 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4dd6cceb6af65d5 server_cookie=48ee4c5885f4ac7d in_seq=0 out_seq=0 2024-04-23T14:53:28.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.687+0000 7f46f4c00640 10 client.26759 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:53:28.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.688+0000 7f46f4c00640 1 -- 172.21.15.28:0/2652981059 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5567866bfa60 con 0x5567867d9020 2024-04-23T14:53:28.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.688+0000 7f46f4c00640 10 client.26759 handle_command_reply: tid=0 2024-04-23T14:53:28.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.688+0000 7f471087b740 1 client.26759 shutdown 2024-04-23T14:53:28.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.688+0000 7f471087b740 1 -- 172.21.15.28:0/2652981059 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556786753460 msgr2=0x556786757c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:28.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.688+0000 7f471087b740 1 --2- 172.21.15.28:0/2652981059 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556786753460 0x556786757c80 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f46f0011dd0 tx=0x7f46f00098e0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.688+0000 7f471087b740 1 -- 172.21.15.28:0/2652981059 shutdown_connections 2024-04-23T14:53:28.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.688+0000 7f471087b740 1 --2- 172.21.15.28:0/2652981059 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5567867d9020 0x5567867db440 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.688+0000 7f471087b740 1 --2- 172.21.15.28:0/2652981059 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556786753460 0x556786757c80 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.688+0000 7f471087b740 1 --2- 172.21.15.28:0/2652981059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55678675db30 0x5567867583c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.688+0000 7f471087b740 1 --2- 172.21.15.28:0/2652981059 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556786752b40 0x556786766310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.688+0000 7f471087b740 1 -- 172.21.15.28:0/2652981059 >> 172.21.15.28:0/2652981059 conn(0x55678674ee60 msgr2=0x55678674d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:53:28.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.688+0000 7f46f4c00640 0 client.26759 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:53:28.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.688+0000 7f471087b740 1 -- 172.21.15.28:0/2652981059 shutdown_connections 2024-04-23T14:53:28.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.688+0000 7f471087b740 1 -- 172.21.15.28:0/2652981059 wait complete. 2024-04-23T14:53:28.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.688+0000 7f471087b740 20 client.26759 trim_cache size 0 max 16384 2024-04-23T14:53:28.692 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-23T14:53:28.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.689+0000 7f4707e00640 1 -- 172.21.15.28:0/4223611715 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f46e80a3d20 msgr2=0x7f46e80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:28.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.689+0000 7f4707e00640 1 --2- 172.21.15.28:0/4223611715 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f46e80a3d20 0x7f46e80a61e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4700005fd0 tx=0x7f4700005cc0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.689+0000 7f4707e00640 1 -- 172.21.15.28:0/4223611715 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4708103060 msgr2=0x7f470819e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:28.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.689+0000 7f4707e00640 1 --2- 172.21.15.28:0/4223611715 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4708103060 0x7f470819e400 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f46f8004770 tx=0x7f46f8003c10 comp rx=0 tx=0).stop 2024-04-23T14:53:28.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.689+0000 7f4707e00640 1 -- 172.21.15.28:0/4223611715 shutdown_connections 2024-04-23T14:53:28.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.689+0000 7f4707e00640 1 --2- 172.21.15.28:0/4223611715 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f46e80a3d20 0x7f46e80a61e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.689+0000 7f4707e00640 1 --2- 172.21.15.28:0/4223611715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f470810c480 0x7f47081985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.689+0000 7f4707e00640 1 --2- 172.21.15.28:0/4223611715 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4708103980 0x7f470819e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.689+0000 7f4707e00640 1 --2- 172.21.15.28:0/4223611715 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4708103060 0x7f470819e400 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:28.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.689+0000 7f4707e00640 1 -- 172.21.15.28:0/4223611715 >> 172.21.15.28:0/4223611715 conn(0x7f47080fc880 msgr2=0x7f47080fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:53:28.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.690+0000 7f4707e00640 1 -- 172.21.15.28:0/4223611715 shutdown_connections 2024-04-23T14:53:28.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:28.690+0000 7f4707e00640 1 -- 172.21.15.28:0/4223611715 wait complete. 2024-04-23T14:53:29.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:28 smithi105 ceph-mon[32555]: pgmap v292: 97 pgs: 97 active+clean; 509 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.4 MiB/s wr, 203 op/s 2024-04-23T14:53:29.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:28 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2358886172' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:53:29.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:28 smithi038 ceph-mon[32348]: pgmap v292: 97 pgs: 97 active+clean; 509 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.4 MiB/s wr, 203 op/s 2024-04-23T14:53:29.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:28 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2358886172' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:53:29.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:28 smithi028 ceph-mon[27810]: pgmap v292: 97 pgs: 97 active+clean; 509 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.4 MiB/s wr, 203 op/s 2024-04-23T14:53:29.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:28 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2358886172' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:53:29.519 INFO:tasks.workunit.client.0.smithi028.stdout: 1 799369 8.88 MB/sec execute 345 sec latency 56.977 ms 2024-04-23T14:53:29.712 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-23T14:53:29.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:53:29.712 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-23T14:53:29.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.935+0000 7f6b7b000640 1 -- 172.21.15.28:0/145513352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b74109520 msgr2=0x7f6b741110f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:29.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.935+0000 7f6b7b000640 1 --2- 172.21.15.28:0/145513352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b74109520 0x7f6b741110f0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c009ca0 tx=0x7f6b6c02a8c0 comp rx=0 tx=0).stop 2024-04-23T14:53:29.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.935+0000 7f6b7b000640 1 -- 172.21.15.28:0/145513352 shutdown_connections 2024-04-23T14:53:29.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.935+0000 7f6b7b000640 1 --2- 172.21.15.28:0/145513352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b74109520 0x7f6b741110f0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:29.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.935+0000 7f6b7b000640 1 --2- 172.21.15.28:0/145513352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b74104e00 0x7f6b74108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:29.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.935+0000 7f6b7b000640 1 --2- 172.21.15.28:0/145513352 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6b74104450 0x7f6b74104830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:29.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.935+0000 7f6b7b000640 1 -- 172.21.15.28:0/145513352 >> 172.21.15.28:0/145513352 conn(0x7f6b740ffc70 msgr2=0x7f6b74102090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:53:29.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.935+0000 7f6b7b000640 1 -- 172.21.15.28:0/145513352 shutdown_connections 2024-04-23T14:53:29.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.935+0000 7f6b7b000640 1 -- 172.21.15.28:0/145513352 wait complete. 2024-04-23T14:53:29.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.936+0000 7f6b7b000640 1 Processor -- start 2024-04-23T14:53:29.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.936+0000 7f6b7b000640 1 -- start start 2024-04-23T14:53:29.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.937+0000 7f6b7b000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6b74104450 0x7f6b741a0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:29.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.937+0000 7f6b7b000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b74109520 0x7f6b741a0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:29.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.937+0000 7f6b7b000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b7419a8d0 0x7f6b7419ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:29.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.937+0000 7f6b7b000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6b74114630 con 0x7f6b74109520 2024-04-23T14:53:29.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.937+0000 7f6b7b000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6b741144b0 con 0x7f6b7419a8d0 2024-04-23T14:53:29.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.937+0000 7f6b7b000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6b741147b0 con 0x7f6b74104450 2024-04-23T14:53:29.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.937+0000 7f6b73e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6b74104450 0x7f6b741a0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:29.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.937+0000 7f6b73e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6b74104450 0x7f6b741a0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:39206/0 (socket says 172.21.15.28:39206) 2024-04-23T14:53:29.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.937+0000 7f6b73e00640 1 -- 172.21.15.28:0/1525633898 learned_addr learned my addr 172.21.15.28:0/1525633898 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:53:29.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.937+0000 7f6b79000640 1 --2- 172.21.15.28:0/1525633898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b7419a8d0 0x7f6b7419ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:29.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.937+0000 7f6b73400640 1 --2- 172.21.15.28:0/1525633898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b74109520 0x7f6b741a0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:29.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.937+0000 7f6b73e00640 1 -- 172.21.15.28:0/1525633898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b7419a8d0 msgr2=0x7f6b7419ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:29.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.937+0000 7f6b73e00640 1 --2- 172.21.15.28:0/1525633898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b7419a8d0 0x7f6b7419ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:29.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.937+0000 7f6b73e00640 1 -- 172.21.15.28:0/1525633898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b74109520 msgr2=0x7f6b741a0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:29.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.937+0000 7f6b73e00640 1 --2- 172.21.15.28:0/1525633898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b74109520 0x7f6b741a0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:29.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.937+0000 7f6b73e00640 1 -- 172.21.15.28:0/1525633898 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b7419b3f0 con 0x7f6b74104450 2024-04-23T14:53:29.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.938+0000 7f6b73e00640 1 --2- 172.21.15.28:0/1525633898 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6b74104450 0x7f6b741a0720 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6b6400e9f0 tx=0x7f6b6400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:53:29.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.938+0000 7f6b70c00640 1 -- 172.21.15.28:0/1525633898 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6400cd80 con 0x7f6b74104450 2024-04-23T14:53:29.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.938+0000 7f6b7b000640 1 -- 172.21.15.28:0/1525633898 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b7419b6e0 con 0x7f6b74104450 2024-04-23T14:53:29.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.938+0000 7f6b70c00640 1 -- 172.21.15.28:0/1525633898 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b64005500 con 0x7f6b74104450 2024-04-23T14:53:29.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.938+0000 7f6b70c00640 1 -- 172.21.15.28:0/1525633898 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b64010690 con 0x7f6b74104450 2024-04-23T14:53:29.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.938+0000 7f6b7b000640 1 -- 172.21.15.28:0/1525633898 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b741a78e0 con 0x7f6b74104450 2024-04-23T14:53:29.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.941+0000 7f6b70c00640 1 -- 172.21.15.28:0/1525633898 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f6b64004f60 con 0x7f6b74104450 2024-04-23T14:53:29.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.942+0000 7f6b7dfbd740 1 Processor -- start 2024-04-23T14:53:29.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.942+0000 7f6b7dfbd740 1 -- start start 2024-04-23T14:53:29.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.942+0000 7f6b7dfbd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9a2645090 0x55d9a2645470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:29.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.942+0000 7f6b7dfbd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9a26459b0 0x55d9a264fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:29.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.942+0000 7f6b7dfbd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9a2650080 0x55d9a2652470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:29.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.942+0000 7f6b7dfbd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d9a25aae70 con 0x55d9a2650080 2024-04-23T14:53:29.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.942+0000 7f6b7dfbd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d9a2551380 con 0x55d9a2645090 2024-04-23T14:53:29.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.942+0000 7f6b7dfbd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d9a25a6e30 con 0x55d9a26459b0 2024-04-23T14:53:29.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.943+0000 7f6b73e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9a26459b0 0x55d9a264fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:29.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.943+0000 7f6b73e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9a26459b0 0x55d9a264fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:39208/0 (socket says 172.21.15.28:39208) 2024-04-23T14:53:29.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.943+0000 7f6b73e00640 1 -- 172.21.15.28:0/649041370 learned_addr learned my addr 172.21.15.28:0/649041370 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:53:29.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.943+0000 7f6b73400640 1 --2- 172.21.15.28:0/649041370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9a2650080 0x55d9a2652470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.943+0000 7f6b79000640 1 --2- 172.21.15.28:0/649041370 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9a2645090 0x55d9a2645470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.943+0000 7f6b73e00640 1 -- 172.21.15.28:0/649041370 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9a2645090 msgr2=0x55d9a2645470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:53:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.943+0000 7f6b73e00640 1 --2- 172.21.15.28:0/649041370 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9a2645090 0x55d9a2645470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.943+0000 7f6b73e00640 1 -- 172.21.15.28:0/649041370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9a2650080 msgr2=0x55d9a2652470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:53:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.943+0000 7f6b73e00640 1 --2- 172.21.15.28:0/649041370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9a2650080 0x55d9a2652470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.943+0000 7f6b73e00640 1 -- 172.21.15.28:0/649041370 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9a23857e0 con 0x55d9a26459b0 2024-04-23T14:53:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.943+0000 7f6b79000640 1 --2- 172.21.15.28:0/649041370 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9a2645090 0x55d9a2645470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:53:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.944+0000 7f6b73e00640 1 --2- 172.21.15.28:0/649041370 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9a26459b0 0x55d9a264fb40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f6b640ce270 tx=0x7f6b640d4cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48813e4b183db3a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:53:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.944+0000 7f6b70c00640 1 --2- 172.21.15.28:0/1525633898 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f6b580a3d20 0x7f6b580a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.944+0000 7f6b69600640 1 -- 172.21.15.28:0/649041370 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b640d9d30 con 0x55d9a26459b0 2024-04-23T14:53:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.944+0000 7f6b69600640 1 -- 172.21.15.28:0/649041370 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b6409d280 con 0x55d9a26459b0 2024-04-23T14:53:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.944+0000 7f6b69600640 1 -- 172.21.15.28:0/649041370 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b640c7ef0 con 0x55d9a26459b0 2024-04-23T14:53:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.944+0000 7f6b70c00640 1 -- 172.21.15.28:0/1525633898 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7f6b640d0c50 con 0x7f6b74104450 2024-04-23T14:53:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.944+0000 7f6b79000640 1 --2- 172.21.15.28:0/1525633898 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f6b580a3d20 0x7f6b580a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:29.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.945+0000 7f6b7dfbd740 1 -- 172.21.15.28:0/649041370 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9a26459b0 msgr2=0x55d9a264fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:29.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.945+0000 7f6b7dfbd740 1 --2- 172.21.15.28:0/649041370 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9a26459b0 0x55d9a264fb40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f6b640ce270 tx=0x7f6b640d4cf0 comp rx=0 tx=0).stop 2024-04-23T14:53:29.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.945+0000 7f6b79000640 1 --2- 172.21.15.28:0/1525633898 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f6b580a3d20 0x7f6b580a61e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c006010 tx=0x7f6b6c005d80 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:53:29.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.945+0000 7f6b7dfbd740 1 -- 172.21.15.28:0/649041370 shutdown_connections 2024-04-23T14:53:29.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.945+0000 7f6b7dfbd740 1 --2- 172.21.15.28:0/649041370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9a2650080 0x55d9a2652470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:29.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.945+0000 7f6b7dfbd740 1 --2- 172.21.15.28:0/649041370 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9a26459b0 0x55d9a264fb40 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:29.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.945+0000 7f6b7dfbd740 1 --2- 172.21.15.28:0/649041370 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9a2645090 0x55d9a2645470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:29.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.945+0000 7f6b7dfbd740 1 -- 172.21.15.28:0/649041370 >> 172.21.15.28:0/649041370 conn(0x55d9a26413b0 msgr2=0x55d9a2641790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:53:29.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.945+0000 7f6b7dfbd740 1 -- 172.21.15.28:0/649041370 shutdown_connections 2024-04-23T14:53:29.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.945+0000 7f6b7dfbd740 1 -- 172.21.15.28:0/649041370 wait complete. 2024-04-23T14:53:29.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.946+0000 7f6b7dfbd740 1 -- start start 2024-04-23T14:53:29.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.946+0000 7f6b7dfbd740 1 Processor -- start 2024-04-23T14:53:29.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.946+0000 7f6b7dfbd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9a2645090 0x55d9a264cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:29.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.946+0000 7f6b7dfbd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9a26459b0 0x55d9a264d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:29.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.946+0000 7f6b7dfbd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9a2650080 0x55d9a264da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:29.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.946+0000 7f6b7dfbd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d9a2652e70 con 0x55d9a2645090 2024-04-23T14:53:29.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.946+0000 7f6b7dfbd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d9a2564660 con 0x55d9a26459b0 2024-04-23T14:53:29.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.946+0000 7f6b7dfbd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d9a2652ff0 con 0x55d9a2650080 2024-04-23T14:53:29.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.947+0000 7f6b7dfbd740 10 client.0 _ll_register_callbacks cb 0x55d9a2552aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:53:29.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.947+0000 7f6b73e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9a2645090 0x55d9a264cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:29.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.947+0000 7f6b73400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9a26459b0 0x55d9a264d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:29.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.947+0000 7f6b79000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9a2650080 0x55d9a264da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:29.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.947+0000 7f6b73e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9a2645090 0x55d9a264cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53722/0 (socket says 172.21.15.28:53722) 2024-04-23T14:53:29.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.947+0000 7f6b73e00640 1 -- 172.21.15.28:0/4167626996 learned_addr learned my addr 172.21.15.28:0/4167626996 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:53:29.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.947+0000 7f6b73400640 1 -- 172.21.15.28:0/4167626996 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9a2650080 msgr2=0x55d9a264da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:53:29.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.947+0000 7f6b73400640 1 --2- 172.21.15.28:0/4167626996 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9a2650080 0x55d9a264da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:29.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.947+0000 7f6b73400640 1 -- 172.21.15.28:0/4167626996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9a2645090 msgr2=0x55d9a264cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:53:29.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.947+0000 7f6b73400640 1 --2- 172.21.15.28:0/4167626996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9a2645090 0x55d9a264cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:29.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.947+0000 7f6b73400640 1 -- 172.21.15.28:0/4167626996 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9a23e5f70 con 0x55d9a26459b0 2024-04-23T14:53:29.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.947+0000 7f6b73e00640 1 --2- 172.21.15.28:0/4167626996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9a2645090 0x55d9a264cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:53:29.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.947+0000 7f6b73400640 1 --2- 172.21.15.28:0/4167626996 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9a26459b0 0x55d9a264d510 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f6b7410aa80 tx=0x7f6b60011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f6931efe8f6a13f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:53:29.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.947+0000 7f6b5e000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:53:29.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.948+0000 7f6b5e000640 1 -- 172.21.15.28:0/4167626996 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b60004020 con 0x55d9a26459b0 2024-04-23T14:53:29.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.948+0000 7f6b7dfbd740 1 -- 172.21.15.28:0/4167626996 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b6409d280 con 0x55d9a26459b0 2024-04-23T14:53:29.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.948+0000 7f6b5e000640 1 -- 172.21.15.28:0/4167626996 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7f6b600041c0 con 0x55d9a26459b0 2024-04-23T14:53:29.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.948+0000 7f6b5e000640 1 -- 172.21.15.28:0/4167626996 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f6b600043c0 con 0x55d9a26459b0 2024-04-23T14:53:29.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.948+0000 7f6b7dfbd740 10 client.26771 fetch_fsmap learned FSMap version 20 2024-04-23T14:53:29.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.948+0000 7f6b7dfbd740 1 -- 172.21.15.28:0/4167626996 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55d9a227d170 con 0x55d9a26459b0 2024-04-23T14:53:29.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.948+0000 7f6b5e000640 1 -- 172.21.15.28:0/4167626996 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b6409d280 con 0x55d9a26459b0 2024-04-23T14:53:29.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.948+0000 7f6b5e000640 1 -- 172.21.15.28:0/4167626996 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f6b60003ac0 con 0x55d9a26459b0 2024-04-23T14:53:29.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.949+0000 7f6b7dfbd740 10 client.26771 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:53:29.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.949+0000 7f6b7dfbd740 10 client.26771 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:53:29.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.949+0000 7f6b7dfbd740 20 client.26771 populate_metadata read hostname 'smithi028' 2024-04-23T14:53:29.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.949+0000 7f6b7dfbd740 1 --2- 172.21.15.28:0/4167626996 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d9a2660350 0x55d9a26627a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:29.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.949+0000 7f6b7dfbd740 4 client.26771 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:53:29.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.949+0000 7f6b7dfbd740 1 -- 172.21.15.28:0/4167626996 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9a25b8890 con 0x55d9a2660350 2024-04-23T14:53:29.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.949+0000 7f6b73e00640 1 --2- 172.21.15.28:0/4167626996 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d9a2660350 0x55d9a26627a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:29.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.949+0000 7f6b73e00640 1 --2- 172.21.15.28:0/4167626996 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d9a2660350 0x55d9a26627a0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c586181e371dadc5 server_cookie=e91867205dd73221 in_seq=0 out_seq=0 2024-04-23T14:53:29.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.949+0000 7f6b5e000640 10 client.26771 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:53:29.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.950+0000 7f6b5e000640 1 -- 172.21.15.28:0/4167626996 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d9a25b8890 con 0x55d9a2660350 2024-04-23T14:53:29.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.950+0000 7f6b5e000640 10 client.26771 handle_command_reply: tid=0 2024-04-23T14:53:29.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.950+0000 7f6b7dfbd740 1 client.26771 shutdown 2024-04-23T14:53:29.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.950+0000 7f6b7dfbd740 1 -- 172.21.15.28:0/4167626996 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9a26459b0 msgr2=0x55d9a264d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:29.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.950+0000 7f6b7dfbd740 1 --2- 172.21.15.28:0/4167626996 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9a26459b0 0x55d9a264d510 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f6b7410aa80 tx=0x7f6b60011810 comp rx=0 tx=0).stop 2024-04-23T14:53:29.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.950+0000 7f6b7dfbd740 1 -- 172.21.15.28:0/4167626996 shutdown_connections 2024-04-23T14:53:29.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.950+0000 7f6b7dfbd740 1 --2- 172.21.15.28:0/4167626996 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d9a2660350 0x55d9a26627a0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:29.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.950+0000 7f6b7dfbd740 1 --2- 172.21.15.28:0/4167626996 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9a26459b0 0x55d9a264d510 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:29.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.950+0000 7f6b7dfbd740 1 --2- 172.21.15.28:0/4167626996 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9a2650080 0x55d9a264da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:29.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.950+0000 7f6b5e000640 0 client.26771 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:53:29.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.950+0000 7f6b7dfbd740 1 --2- 172.21.15.28:0/4167626996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9a2645090 0x55d9a264cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:29.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.951+0000 7f6b7dfbd740 1 -- 172.21.15.28:0/4167626996 >> 172.21.15.28:0/4167626996 conn(0x55d9a26413b0 msgr2=0x55d9a26471b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:53:29.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.951+0000 7f6b7dfbd740 1 -- 172.21.15.28:0/4167626996 shutdown_connections 2024-04-23T14:53:29.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.951+0000 7f6b7dfbd740 1 -- 172.21.15.28:0/4167626996 wait complete. 2024-04-23T14:53:29.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.951+0000 7f6b7dfbd740 20 client.26771 trim_cache size 0 max 16384 2024-04-23T14:53:29.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.971+0000 7f6b7dfbd740 1 Processor -- start 2024-04-23T14:53:29.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.971+0000 7f6b7dfbd740 1 -- start start 2024-04-23T14:53:29.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.972+0000 7f6b7dfbd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9a2645090 0x55d9a265bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:29.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.972+0000 7f6b7dfbd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9a26459b0 0x55d9a2668070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:29.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.972+0000 7f6b7dfbd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9a2650080 0x55d9a26685b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:29.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.972+0000 7f6b7dfbd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6b741ae030 con 0x55d9a26459b0 2024-04-23T14:53:29.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.972+0000 7f6b7dfbd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6b741b0890 con 0x55d9a2650080 2024-04-23T14:53:29.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.972+0000 7f6b7dfbd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6b741ae680 con 0x55d9a2645090 2024-04-23T14:53:29.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.972+0000 7f6b73e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9a2645090 0x55d9a265bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:29.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.972+0000 7f6b73400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9a26459b0 0x55d9a2668070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:29.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.972+0000 7f6b73e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9a2645090 0x55d9a265bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:39214/0 (socket says 172.21.15.28:39214) 2024-04-23T14:53:29.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.972+0000 7f6b73e00640 1 -- 172.21.15.28:0/169387426 learned_addr learned my addr 172.21.15.28:0/169387426 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:53:29.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.972+0000 7f6b79000640 1 --2- 172.21.15.28:0/169387426 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9a2650080 0x55d9a26685b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:29.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.972+0000 7f6b73e00640 1 -- 172.21.15.28:0/169387426 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9a2650080 msgr2=0x55d9a26685b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:53:29.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.973+0000 7f6b73e00640 1 --2- 172.21.15.28:0/169387426 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9a2650080 0x55d9a26685b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:29.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.973+0000 7f6b73e00640 1 -- 172.21.15.28:0/169387426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9a26459b0 msgr2=0x55d9a2668070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:53:29.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.973+0000 7f6b73e00640 1 --2- 172.21.15.28:0/169387426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9a26459b0 0x55d9a2668070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:29.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.973+0000 7f6b73e00640 1 -- 172.21.15.28:0/169387426 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9a263cd10 con 0x55d9a2645090 2024-04-23T14:53:29.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.973+0000 7f6b73e00640 1 --2- 172.21.15.28:0/169387426 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9a2645090 0x55d9a265bd30 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6b6409c880 tx=0x7f6b640d4cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=914bb7403eab5aba server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:53:29.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.973+0000 7f6b68c00640 1 -- 172.21.15.28:0/169387426 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b640cba10 con 0x55d9a2645090 2024-04-23T14:53:29.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.973+0000 7f6b68c00640 1 -- 172.21.15.28:0/169387426 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b640d44d0 con 0x55d9a2645090 2024-04-23T14:53:29.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.973+0000 7f6b68c00640 1 -- 172.21.15.28:0/169387426 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b640c9e40 con 0x55d9a2645090 2024-04-23T14:53:29.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.974+0000 7f6b7dfbd740 1 -- 172.21.15.28:0/169387426 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9a2645090 msgr2=0x55d9a265bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:29.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.974+0000 7f6b7dfbd740 1 --2- 172.21.15.28:0/169387426 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9a2645090 0x55d9a265bd30 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6b6409c880 tx=0x7f6b640d4cf0 comp rx=0 tx=0).stop 2024-04-23T14:53:29.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.974+0000 7f6b7dfbd740 1 -- 172.21.15.28:0/169387426 shutdown_connections 2024-04-23T14:53:29.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.974+0000 7f6b7dfbd740 1 --2- 172.21.15.28:0/169387426 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9a2650080 0x55d9a26685b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:29.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.974+0000 7f6b7dfbd740 1 --2- 172.21.15.28:0/169387426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9a26459b0 0x55d9a2668070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:29.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.974+0000 7f6b7dfbd740 1 --2- 172.21.15.28:0/169387426 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9a2645090 0x55d9a265bd30 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:29.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.974+0000 7f6b7dfbd740 1 -- 172.21.15.28:0/169387426 >> 172.21.15.28:0/169387426 conn(0x55d9a26413b0 msgr2=0x55d9a264d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:53:29.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.974+0000 7f6b7dfbd740 1 -- 172.21.15.28:0/169387426 shutdown_connections 2024-04-23T14:53:29.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.974+0000 7f6b7dfbd740 1 -- 172.21.15.28:0/169387426 wait complete. 2024-04-23T14:53:29.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.974+0000 7f6b7dfbd740 1 -- start start 2024-04-23T14:53:29.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.975+0000 7f6b7dfbd740 1 Processor -- start 2024-04-23T14:53:29.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.975+0000 7f6b7dfbd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9a2645090 0x55d9a2649eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:29.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.975+0000 7f6b7dfbd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9a26459b0 0x55d9a264a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:29.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.975+0000 7f6b7dfbd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9a2650080 0x55d9a264a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:29.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.975+0000 7f6b7dfbd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6b741aed50 con 0x55d9a2650080 2024-04-23T14:53:29.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.975+0000 7f6b7dfbd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6b741b01c0 con 0x55d9a2645090 2024-04-23T14:53:29.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.975+0000 7f6b7dfbd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6b741af420 con 0x55d9a26459b0 2024-04-23T14:53:29.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.975+0000 7f6b7dfbd740 10 client.0 _ll_register_callbacks cb 0x7f6b6000bb10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:53:29.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.975+0000 7f6b73e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9a2645090 0x55d9a2649eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:29.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.975+0000 7f6b73e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9a2645090 0x55d9a2649eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:39130/0 (socket says 172.21.15.28:39130) 2024-04-23T14:53:29.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.975+0000 7f6b79000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9a2650080 0x55d9a264a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:29.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.975+0000 7f6b73400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9a26459b0 0x55d9a264a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:29.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.976+0000 7f6b73e00640 1 -- 172.21.15.28:0/187489733 learned_addr learned my addr 172.21.15.28:0/187489733 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:53:29.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.976+0000 7f6b73e00640 1 -- 172.21.15.28:0/187489733 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9a26459b0 msgr2=0x55d9a264a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:53:29.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.976+0000 7f6b73e00640 1 --2- 172.21.15.28:0/187489733 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9a26459b0 0x55d9a264a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:29.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.976+0000 7f6b73e00640 1 -- 172.21.15.28:0/187489733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9a2650080 msgr2=0x55d9a264a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:53:29.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.976+0000 7f6b73e00640 1 --2- 172.21.15.28:0/187489733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9a2650080 0x55d9a264a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:29.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.976+0000 7f6b73e00640 1 -- 172.21.15.28:0/187489733 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9a26c5ae0 con 0x55d9a2645090 2024-04-23T14:53:29.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.976+0000 7f6b73e00640 1 --2- 172.21.15.28:0/187489733 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9a2645090 0x55d9a2649eb0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f6b640d9710 tx=0x7f6b640c7c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d081566474a6508 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:53:29.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.976+0000 7f6b5e000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:53:29.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.976+0000 7f6b5e000640 1 -- 172.21.15.28:0/187489733 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b640ca1c0 con 0x55d9a2645090 2024-04-23T14:53:29.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.976+0000 7f6b5e000640 1 -- 172.21.15.28:0/187489733 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7f6b64005250 con 0x55d9a2645090 2024-04-23T14:53:29.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.976+0000 7f6b7dfbd740 1 -- 172.21.15.28:0/187489733 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b640d44d0 con 0x55d9a2645090 2024-04-23T14:53:29.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.977+0000 7f6b5e000640 1 -- 172.21.15.28:0/187489733 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f6b640ca390 con 0x55d9a2645090 2024-04-23T14:53:29.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.977+0000 7f6b7dfbd740 10 client.26777 fetch_fsmap learned FSMap version 20 2024-04-23T14:53:29.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.977+0000 7f6b7dfbd740 1 -- 172.21.15.28:0/187489733 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55d9a26c6e00 con 0x55d9a2645090 2024-04-23T14:53:29.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.977+0000 7f6b5e000640 1 -- 172.21.15.28:0/187489733 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b640d44d0 con 0x55d9a2645090 2024-04-23T14:53:29.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.977+0000 7f6b5e000640 1 -- 172.21.15.28:0/187489733 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f6b640046c0 con 0x55d9a2645090 2024-04-23T14:53:29.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.977+0000 7f6b7dfbd740 10 client.26777 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:53:29.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.977+0000 7f6b7dfbd740 10 client.26777 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:53:29.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.977+0000 7f6b7dfbd740 20 client.26777 populate_metadata read hostname 'smithi028' 2024-04-23T14:53:29.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.977+0000 7f6b7dfbd740 1 --2- 172.21.15.28:0/187489733 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d9a26c7070 0x55d9a26c9460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:29.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.977+0000 7f6b7dfbd740 4 client.26777 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-23T14:53:29.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.977+0000 7f6b7dfbd740 1 -- 172.21.15.28:0/187489733 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d9a25b2700 con 0x55d9a26c7070 2024-04-23T14:53:29.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.977+0000 7f6b73400640 1 --2- 172.21.15.28:0/187489733 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d9a26c7070 0x55d9a26c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:29.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.978+0000 7f6b73400640 1 --2- 172.21.15.28:0/187489733 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d9a26c7070 0x55d9a26c9460 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0531446bb67560e server_cookie=e747c034121bea67 in_seq=0 out_seq=0 2024-04-23T14:53:29.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.978+0000 7f6b5e000640 10 client.26777 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:53:29.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.979+0000 7f6b5e000640 1 -- 172.21.15.28:0/187489733 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d9a25b2700 con 0x55d9a26c7070 2024-04-23T14:53:29.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.979+0000 7f6b5e000640 10 client.26777 handle_command_reply: tid=0 2024-04-23T14:53:29.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.979+0000 7f6b7dfbd740 1 client.26777 shutdown 2024-04-23T14:53:29.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.980+0000 7f6b7dfbd740 1 -- 172.21.15.28:0/187489733 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9a2645090 msgr2=0x55d9a2649eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:29.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.980+0000 7f6b7dfbd740 1 --2- 172.21.15.28:0/187489733 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9a2645090 0x55d9a2649eb0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f6b640d9710 tx=0x7f6b640c7c90 comp rx=0 tx=0).stop 2024-04-23T14:53:29.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.980+0000 7f6b7dfbd740 1 -- 172.21.15.28:0/187489733 shutdown_connections 2024-04-23T14:53:29.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.980+0000 7f6b7dfbd740 1 --2- 172.21.15.28:0/187489733 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55d9a26c7070 0x55d9a26c9460 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:29.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.980+0000 7f6b7dfbd740 1 --2- 172.21.15.28:0/187489733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9a2650080 0x55d9a264a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:29.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.980+0000 7f6b7dfbd740 1 --2- 172.21.15.28:0/187489733 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d9a26459b0 0x55d9a264a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:29.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.980+0000 7f6b7dfbd740 1 --2- 172.21.15.28:0/187489733 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d9a2645090 0x55d9a2649eb0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:29.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.980+0000 7f6b7dfbd740 1 -- 172.21.15.28:0/187489733 >> 172.21.15.28:0/187489733 conn(0x55d9a26413b0 msgr2=0x55d9a26468b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:53:29.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.980+0000 7f6b5e000640 0 client.26777 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:53:29.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.980+0000 7f6b7dfbd740 1 -- 172.21.15.28:0/187489733 shutdown_connections 2024-04-23T14:53:29.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.980+0000 7f6b7dfbd740 1 -- 172.21.15.28:0/187489733 wait complete. 2024-04-23T14:53:29.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.980+0000 7f6b7dfbd740 20 client.26777 trim_cache size 0 max 16384 2024-04-23T14:53:29.983 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:53:29.983 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-23T14:53:29.983 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "9af96ddf-cc40-45b3-a594-2df1676f49a2", 2024-04-23T14:53:29.983 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-23T14:53:29.983 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:53:29.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.981+0000 7f6b7b000640 1 -- 172.21.15.28:0/1525633898 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f6b580a3d20 msgr2=0x7f6b580a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:29.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.981+0000 7f6b7b000640 1 --2- 172.21.15.28:0/1525633898 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f6b580a3d20 0x7f6b580a61e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c006010 tx=0x7f6b6c005d80 comp rx=0 tx=0).stop 2024-04-23T14:53:29.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.981+0000 7f6b7b000640 1 -- 172.21.15.28:0/1525633898 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6b74104450 msgr2=0x7f6b741a0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:29.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.981+0000 7f6b7b000640 1 --2- 172.21.15.28:0/1525633898 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6b74104450 0x7f6b741a0720 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6b6400e9f0 tx=0x7f6b6400edf0 comp rx=0 tx=0).stop 2024-04-23T14:53:29.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.981+0000 7f6b7b000640 1 -- 172.21.15.28:0/1525633898 shutdown_connections 2024-04-23T14:53:29.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.981+0000 7f6b7b000640 1 --2- 172.21.15.28:0/1525633898 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f6b580a3d20 0x7f6b580a61e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:29.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.981+0000 7f6b7b000640 1 --2- 172.21.15.28:0/1525633898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b7419a8d0 0x7f6b7419ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:29.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.981+0000 7f6b7b000640 1 --2- 172.21.15.28:0/1525633898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b74109520 0x7f6b741a0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:29.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.981+0000 7f6b7b000640 1 --2- 172.21.15.28:0/1525633898 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6b74104450 0x7f6b741a0720 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:29.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.981+0000 7f6b7b000640 1 -- 172.21.15.28:0/1525633898 >> 172.21.15.28:0/1525633898 conn(0x7f6b740ffc70 msgr2=0x7f6b741007b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:53:29.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.981+0000 7f6b7b000640 1 -- 172.21.15.28:0/1525633898 shutdown_connections 2024-04-23T14:53:29.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:29.981+0000 7f6b7b000640 1 -- 172.21.15.28:0/1525633898 wait complete. 2024-04-23T14:53:30.001 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:53:30.001 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:53:30.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.217+0000 7fa76d600640 1 -- 172.21.15.28:0/1758175394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa768074cc0 msgr2=0x7fa768112a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:30.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.217+0000 7fa76d600640 1 --2- 172.21.15.28:0/1758175394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa768074cc0 0x7fa768112a80 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fa754009900 tx=0x7fa75402a480 comp rx=0 tx=0).stop 2024-04-23T14:53:30.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.217+0000 7fa76d600640 1 -- 172.21.15.28:0/1758175394 shutdown_connections 2024-04-23T14:53:30.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.217+0000 7fa76d600640 1 --2- 172.21.15.28:0/1758175394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa768074cc0 0x7fa768112a80 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:30.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.217+0000 7fa76d600640 1 --2- 172.21.15.28:0/1758175394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa768112fc0 0x7fa7681153b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:30.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.217+0000 7fa76d600640 1 --2- 172.21.15.28:0/1758175394 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7680743a0 0x7fa768074780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:30.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.217+0000 7fa76d600640 1 -- 172.21.15.28:0/1758175394 >> 172.21.15.28:0/1758175394 conn(0x7fa7680fda80 msgr2=0x7fa7680ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:53:30.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.217+0000 7fa76d600640 1 -- 172.21.15.28:0/1758175394 shutdown_connections 2024-04-23T14:53:30.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.217+0000 7fa76d600640 1 -- 172.21.15.28:0/1758175394 wait complete. 2024-04-23T14:53:30.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.218+0000 7fa76d600640 1 Processor -- start 2024-04-23T14:53:30.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.218+0000 7fa76d600640 1 -- start start 2024-04-23T14:53:30.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.219+0000 7fa76d600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7680743a0 0x7fa76819ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:30.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.219+0000 7fa76d600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa768074cc0 0x7fa76819f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:30.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.219+0000 7fa76d600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa768112fc0 0x7fa7681a3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:30.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.219+0000 7fa76d600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa7681189c0 con 0x7fa768112fc0 2024-04-23T14:53:30.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.219+0000 7fa76d600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa768118840 con 0x7fa768074cc0 2024-04-23T14:53:30.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.219+0000 7fa76d600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa768118b40 con 0x7fa7680743a0 2024-04-23T14:53:30.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.219+0000 7fa766000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa768074cc0 0x7fa76819f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:30.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.219+0000 7fa766a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7680743a0 0x7fa76819ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:30.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.219+0000 7fa766a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7680743a0 0x7fa76819ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:39224/0 (socket says 172.21.15.28:39224) 2024-04-23T14:53:30.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.219+0000 7fa766000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa768074cc0 0x7fa76819f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:39152/0 (socket says 172.21.15.28:39152) 2024-04-23T14:53:30.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.219+0000 7fa766a00640 1 -- 172.21.15.28:0/1175830730 learned_addr learned my addr 172.21.15.28:0/1175830730 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:53:30.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.219+0000 7fa767400640 1 --2- 172.21.15.28:0/1175830730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa768112fc0 0x7fa7681a3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:30.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.219+0000 7fa766a00640 1 -- 172.21.15.28:0/1175830730 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa768074cc0 msgr2=0x7fa76819f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:30.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.219+0000 7fa766a00640 1 --2- 172.21.15.28:0/1175830730 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa768074cc0 0x7fa76819f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:30.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.219+0000 7fa766a00640 1 -- 172.21.15.28:0/1175830730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa768112fc0 msgr2=0x7fa7681a3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:30.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.219+0000 7fa766a00640 1 --2- 172.21.15.28:0/1175830730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa768112fc0 0x7fa7681a3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:30.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.220+0000 7fa766a00640 1 -- 172.21.15.28:0/1175830730 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7681a3f30 con 0x7fa7680743a0 2024-04-23T14:53:30.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.220+0000 7fa766000640 1 --2- 172.21.15.28:0/1175830730 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa768074cc0 0x7fa76819f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:53:30.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.220+0000 7fa766a00640 1 --2- 172.21.15.28:0/1175830730 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7680743a0 0x7fa76819ef40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa75800ebc0 tx=0x7fa75800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:53:30.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.220+0000 7fa75f400640 1 -- 172.21.15.28:0/1175830730 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa758010630 con 0x7fa7680743a0 2024-04-23T14:53:30.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.220+0000 7fa75f400640 1 -- 172.21.15.28:0/1175830730 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa758010cb0 con 0x7fa7680743a0 2024-04-23T14:53:30.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.220+0000 7fa76d600640 1 -- 172.21.15.28:0/1175830730 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7681a4220 con 0x7fa7680743a0 2024-04-23T14:53:30.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.220+0000 7fa75f400640 1 -- 172.21.15.28:0/1175830730 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa758018bd0 con 0x7fa7680743a0 2024-04-23T14:53:30.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.220+0000 7fa76d600640 1 -- 172.21.15.28:0/1175830730 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7681abab0 con 0x7fa7680743a0 2024-04-23T14:53:30.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.223+0000 7fa75f400640 1 -- 172.21.15.28:0/1175830730 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fa7580107d0 con 0x7fa7680743a0 2024-04-23T14:53:30.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.224+0000 7fa7703ea740 1 Processor -- start 2024-04-23T14:53:30.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.224+0000 7fa7703ea740 1 -- start start 2024-04-23T14:53:30.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.224+0000 7fa7703ea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412d384860 0x56412d384c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:30.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.224+0000 7fa7703ea740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56412d385180 0x56412d38f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:30.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.224+0000 7fa7703ea740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56412d38f850 0x56412d391c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:30.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.224+0000 7fa7703ea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56412d2ea1f0 con 0x56412d384860 2024-04-23T14:53:30.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.224+0000 7fa7703ea740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56412d290350 con 0x56412d385180 2024-04-23T14:53:30.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.224+0000 7fa7703ea740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56412d2e61b0 con 0x56412d38f850 2024-04-23T14:53:30.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.225+0000 7fa767400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412d384860 0x56412d384c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:30.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.225+0000 7fa766a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56412d385180 0x56412d38f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:30.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.225+0000 7fa767400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412d384860 0x56412d384c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53760/0 (socket says 172.21.15.28:53760) 2024-04-23T14:53:30.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.225+0000 7fa767400640 1 -- 172.21.15.28:0/436465293 learned_addr learned my addr 172.21.15.28:0/436465293 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:53:30.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.225+0000 7fa766000640 1 --2- 172.21.15.28:0/436465293 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56412d38f850 0x56412d391c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:30.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.225+0000 7fa766000640 1 -- 172.21.15.28:0/436465293 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56412d385180 msgr2=0x56412d38f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:53:30.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.225+0000 7fa766000640 1 --2- 172.21.15.28:0/436465293 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56412d385180 0x56412d38f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:30.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.225+0000 7fa766000640 1 -- 172.21.15.28:0/436465293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412d384860 msgr2=0x56412d384c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:53:30.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.225+0000 7fa766000640 1 --2- 172.21.15.28:0/436465293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412d384860 0x56412d384c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:30.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.225+0000 7fa766000640 1 -- 172.21.15.28:0/436465293 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56412d2e4f50 con 0x56412d38f850 2024-04-23T14:53:30.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.225+0000 7fa766a00640 1 --2- 172.21.15.28:0/436465293 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56412d385180 0x56412d38f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:53:30.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.225+0000 7fa766000640 1 --2- 172.21.15.28:0/436465293 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56412d38f850 0x56412d391c40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa754005940 tx=0x7fa754005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da0beffb23443fc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:53:30.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.226+0000 7fa75f400640 1 --2- 172.21.15.28:0/1175830730 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa74c0a3d20 0x7fa74c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:30.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.226+0000 7fa753400640 1 -- 172.21.15.28:0/436465293 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa754035040 con 0x56412d38f850 2024-04-23T14:53:30.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.226+0000 7fa753400640 1 -- 172.21.15.28:0/436465293 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa754031830 con 0x56412d38f850 2024-04-23T14:53:30.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.226+0000 7fa753400640 1 -- 172.21.15.28:0/436465293 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa754004750 con 0x56412d38f850 2024-04-23T14:53:30.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.226+0000 7fa75f400640 1 -- 172.21.15.28:0/1175830730 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7fa7580d0e60 con 0x7fa7680743a0 2024-04-23T14:53:30.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.226+0000 7fa767400640 1 --2- 172.21.15.28:0/1175830730 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa74c0a3d20 0x7fa74c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:30.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.227+0000 7fa7703ea740 1 -- 172.21.15.28:0/436465293 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56412d38f850 msgr2=0x56412d391c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:30.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.227+0000 7fa7703ea740 1 --2- 172.21.15.28:0/436465293 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56412d38f850 0x56412d391c40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa754005940 tx=0x7fa754005d50 comp rx=0 tx=0).stop 2024-04-23T14:53:30.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.227+0000 7fa7703ea740 1 -- 172.21.15.28:0/436465293 shutdown_connections 2024-04-23T14:53:30.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.227+0000 7fa7703ea740 1 --2- 172.21.15.28:0/436465293 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56412d385180 0x56412d38f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:30.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.227+0000 7fa7703ea740 1 --2- 172.21.15.28:0/436465293 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56412d38f850 0x56412d391c40 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:30.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.227+0000 7fa7703ea740 1 --2- 172.21.15.28:0/436465293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412d384860 0x56412d384c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:30.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.227+0000 7fa7703ea740 1 -- 172.21.15.28:0/436465293 >> 172.21.15.28:0/436465293 conn(0x56412d380b80 msgr2=0x56412d380f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:53:30.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.227+0000 7fa767400640 1 --2- 172.21.15.28:0/1175830730 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa74c0a3d20 0x7fa74c0a61e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa760009de0 tx=0x7fa760005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:53:30.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.227+0000 7fa7703ea740 1 -- 172.21.15.28:0/436465293 shutdown_connections 2024-04-23T14:53:30.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.227+0000 7fa7703ea740 1 -- 172.21.15.28:0/436465293 wait complete. 2024-04-23T14:53:30.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.227+0000 7fa7703ea740 1 -- start start 2024-04-23T14:53:30.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.228+0000 7fa7703ea740 1 Processor -- start 2024-04-23T14:53:30.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.228+0000 7fa7703ea740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56412d384860 0x56412d38c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:30.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.228+0000 7fa7703ea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412d385180 0x56412d38cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.228+0000 7fa7703ea740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56412d38f850 0x56412d38d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.228+0000 7fa7703ea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56412d3925e0 con 0x56412d385180 2024-04-23T14:53:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.228+0000 7fa7703ea740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56412d2a3690 con 0x56412d384860 2024-04-23T14:53:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.228+0000 7fa7703ea740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56412d392760 con 0x56412d38f850 2024-04-23T14:53:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.228+0000 7fa7703ea740 10 client.0 _ll_register_callbacks cb 0x56412d2df650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:53:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.228+0000 7fa766a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56412d384860 0x56412d38c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.228+0000 7fa766a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56412d384860 0x56412d38c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:39176/0 (socket says 172.21.15.28:39176) 2024-04-23T14:53:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.228+0000 7fa766a00640 1 -- 172.21.15.28:0/3840047283 learned_addr learned my addr 172.21.15.28:0/3840047283 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:53:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.229+0000 7fa766000640 1 --2- 172.21.15.28:0/3840047283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412d385180 0x56412d38cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.229+0000 7fa767400640 1 --2- 172.21.15.28:0/3840047283 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56412d38f850 0x56412d38d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.229+0000 7fa766a00640 1 -- 172.21.15.28:0/3840047283 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56412d38f850 msgr2=0x56412d38d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:53:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.229+0000 7fa766a00640 1 --2- 172.21.15.28:0/3840047283 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56412d38f850 0x56412d38d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.229+0000 7fa766a00640 1 -- 172.21.15.28:0/3840047283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412d385180 msgr2=0x56412d38cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:53:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.229+0000 7fa766a00640 1 --2- 172.21.15.28:0/3840047283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412d385180 0x56412d38cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.229+0000 7fa766a00640 1 -- 172.21.15.28:0/3840047283 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56412d2a2920 con 0x56412d384860 2024-04-23T14:53:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.229+0000 7fa766a00640 1 --2- 172.21.15.28:0/3840047283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56412d384860 0x56412d38c840 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa7580ce5a0 tx=0x7fa758018670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79a8346ebd75ba1f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:53:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.229+0000 7fa747e00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:53:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.229+0000 7fa747e00640 1 -- 172.21.15.28:0/3840047283 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa758010e60 con 0x56412d384860 2024-04-23T14:53:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.229+0000 7fa747e00640 1 -- 172.21.15.28:0/3840047283 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7fa75809c290 con 0x56412d384860 2024-04-23T14:53:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.229+0000 7fa7703ea740 1 -- 172.21.15.28:0/3840047283 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa754031830 con 0x56412d384860 2024-04-23T14:53:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.230+0000 7fa747e00640 1 -- 172.21.15.28:0/3840047283 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fa75809cc10 con 0x56412d384860 2024-04-23T14:53:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.230+0000 7fa7703ea740 10 client.26789 fetch_fsmap learned FSMap version 20 2024-04-23T14:53:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.230+0000 7fa7703ea740 1 -- 172.21.15.28:0/3840047283 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x56412d0c67d0 con 0x56412d384860 2024-04-23T14:53:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.230+0000 7fa747e00640 1 -- 172.21.15.28:0/3840047283 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa754031830 con 0x56412d384860 2024-04-23T14:53:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.230+0000 7fa747e00640 1 -- 172.21.15.28:0/3840047283 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fa758098b90 con 0x56412d384860 2024-04-23T14:53:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.230+0000 7fa7703ea740 10 client.26789 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:53:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.230+0000 7fa7703ea740 10 client.26789 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:53:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.230+0000 7fa7703ea740 20 client.26789 populate_metadata read hostname 'smithi028' 2024-04-23T14:53:30.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.230+0000 7fa7703ea740 1 --2- 172.21.15.28:0/3840047283 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56412d39fc30 0x56412d3a20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:30.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.230+0000 7fa7703ea740 4 client.26789 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:53:30.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.230+0000 7fa7703ea740 1 -- 172.21.15.28:0/3840047283 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56412d2f7c10 con 0x56412d39fc30 2024-04-23T14:53:30.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.231+0000 7fa766000640 1 --2- 172.21.15.28:0/3840047283 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56412d39fc30 0x56412d3a20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:30.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.231+0000 7fa766000640 1 --2- 172.21.15.28:0/3840047283 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56412d39fc30 0x56412d3a20c0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0e20471299ed479 server_cookie=a8d4c3ca2db3b0c2 in_seq=0 out_seq=0 2024-04-23T14:53:30.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.231+0000 7fa747e00640 10 client.26789 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:53:30.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.232+0000 7fa747e00640 1 -- 172.21.15.28:0/3840047283 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56412d2f7c10 con 0x56412d39fc30 2024-04-23T14:53:30.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.232+0000 7fa747e00640 10 client.26789 handle_command_reply: tid=0 2024-04-23T14:53:30.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.232+0000 7fa7703ea740 1 client.26789 shutdown 2024-04-23T14:53:30.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.232+0000 7fa7703ea740 1 -- 172.21.15.28:0/3840047283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56412d384860 msgr2=0x56412d38c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:30.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.232+0000 7fa7703ea740 1 --2- 172.21.15.28:0/3840047283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56412d384860 0x56412d38c840 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa7580ce5a0 tx=0x7fa758018670 comp rx=0 tx=0).stop 2024-04-23T14:53:30.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.233+0000 7fa7703ea740 1 -- 172.21.15.28:0/3840047283 shutdown_connections 2024-04-23T14:53:30.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.233+0000 7fa7703ea740 1 --2- 172.21.15.28:0/3840047283 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56412d39fc30 0x56412d3a20c0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:30.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.233+0000 7fa7703ea740 1 --2- 172.21.15.28:0/3840047283 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56412d38f850 0x56412d38d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:30.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.233+0000 7fa7703ea740 1 --2- 172.21.15.28:0/3840047283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412d385180 0x56412d38cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:30.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.233+0000 7fa7703ea740 1 --2- 172.21.15.28:0/3840047283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56412d384860 0x56412d38c840 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:30.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.233+0000 7fa7703ea740 1 -- 172.21.15.28:0/3840047283 >> 172.21.15.28:0/3840047283 conn(0x56412d380b80 msgr2=0x56412d386980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:53:30.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.233+0000 7fa747e00640 0 client.26789 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:53:30.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.233+0000 7fa7703ea740 1 -- 172.21.15.28:0/3840047283 shutdown_connections 2024-04-23T14:53:30.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.233+0000 7fa7703ea740 1 -- 172.21.15.28:0/3840047283 wait complete. 2024-04-23T14:53:30.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.233+0000 7fa7703ea740 20 client.26789 trim_cache size 0 max 16384 2024-04-23T14:53:30.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.253+0000 7fa7703ea740 1 Processor -- start 2024-04-23T14:53:30.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.253+0000 7fa7703ea740 1 -- start start 2024-04-23T14:53:30.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.254+0000 7fa7703ea740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56412d384860 0x56412d39b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:30.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.254+0000 7fa7703ea740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56412d385180 0x56412d38b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:30.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.254+0000 7fa7703ea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412d38f850 0x56412d37faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:30.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.254+0000 7fa7703ea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa7681b23b0 con 0x56412d38f850 2024-04-23T14:53:30.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.254+0000 7fa7703ea740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa7681b4c10 con 0x56412d385180 2024-04-23T14:53:30.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.254+0000 7fa7703ea740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa7681b2a00 con 0x56412d384860 2024-04-23T14:53:30.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.254+0000 7fa766000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56412d385180 0x56412d38b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:30.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.254+0000 7fa766a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56412d384860 0x56412d39b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:30.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.254+0000 7fa766000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56412d385180 0x56412d38b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:39190/0 (socket says 172.21.15.28:39190) 2024-04-23T14:53:30.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.254+0000 7fa766000640 1 -- 172.21.15.28:0/485631286 learned_addr learned my addr 172.21.15.28:0/485631286 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:53:30.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.254+0000 7fa767400640 1 --2- 172.21.15.28:0/485631286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412d38f850 0x56412d37faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:30.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.254+0000 7fa766000640 1 -- 172.21.15.28:0/485631286 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56412d384860 msgr2=0x56412d39b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:53:30.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.254+0000 7fa766000640 1 --2- 172.21.15.28:0/485631286 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56412d384860 0x56412d39b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:30.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.254+0000 7fa766000640 1 -- 172.21.15.28:0/485631286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412d38f850 msgr2=0x56412d37faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:53:30.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.254+0000 7fa766000640 1 --2- 172.21.15.28:0/485631286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412d38f850 0x56412d37faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:30.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.254+0000 7fa766000640 1 -- 172.21.15.28:0/485631286 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56412cffc3f0 con 0x56412d385180 2024-04-23T14:53:30.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.254+0000 7fa766a00640 1 --2- 172.21.15.28:0/485631286 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56412d384860 0x56412d39b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:53:30.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.255+0000 7fa766000640 1 --2- 172.21.15.28:0/485631286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56412d385180 0x56412d38b9a0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa754005940 tx=0x7fa7540053f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a64e18360912fe4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:53:30.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.255+0000 7fa752a00640 1 -- 172.21.15.28:0/485631286 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa754058070 con 0x56412d385180 2024-04-23T14:53:30.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.255+0000 7fa752a00640 1 -- 172.21.15.28:0/485631286 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa754031830 con 0x56412d385180 2024-04-23T14:53:30.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.255+0000 7fa752a00640 1 -- 172.21.15.28:0/485631286 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa754035040 con 0x56412d385180 2024-04-23T14:53:30.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.256+0000 7fa7703ea740 1 -- 172.21.15.28:0/485631286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56412d385180 msgr2=0x56412d38b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:30.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.256+0000 7fa7703ea740 1 --2- 172.21.15.28:0/485631286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56412d385180 0x56412d38b9a0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa754005940 tx=0x7fa7540053f0 comp rx=0 tx=0).stop 2024-04-23T14:53:30.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.256+0000 7fa7703ea740 1 -- 172.21.15.28:0/485631286 shutdown_connections 2024-04-23T14:53:30.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.256+0000 7fa7703ea740 1 --2- 172.21.15.28:0/485631286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56412d385180 0x56412d38b9a0 secure :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa754005940 tx=0x7fa7540053f0 comp rx=0 tx=0).stop 2024-04-23T14:53:30.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.256+0000 7fa7703ea740 1 --2- 172.21.15.28:0/485631286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412d38f850 0x56412d37faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:30.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.256+0000 7fa7703ea740 1 --2- 172.21.15.28:0/485631286 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56412d384860 0x56412d39b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:30.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.256+0000 7fa7703ea740 1 -- 172.21.15.28:0/485631286 >> 172.21.15.28:0/485631286 conn(0x56412d380b80 msgr2=0x56412d37f390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:53:30.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.256+0000 7fa7703ea740 1 -- 172.21.15.28:0/485631286 shutdown_connections 2024-04-23T14:53:30.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.256+0000 7fa7703ea740 1 -- 172.21.15.28:0/485631286 wait complete. 2024-04-23T14:53:30.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.256+0000 7fa7703ea740 1 -- start start 2024-04-23T14:53:30.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.257+0000 7fa7703ea740 1 Processor -- start 2024-04-23T14:53:30.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.257+0000 7fa7703ea740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56412d384860 0x56412d397b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:30.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.257+0000 7fa7703ea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412d38f850 0x56412d392f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:30.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.257+0000 7fa7703ea740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56412d3934b0 0x56412d393930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:30.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.257+0000 7fa7703ea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa7681b30d0 con 0x56412d38f850 2024-04-23T14:53:30.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.257+0000 7fa7703ea740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa7681b4540 con 0x56412d384860 2024-04-23T14:53:30.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.257+0000 7fa7703ea740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa7681b37a0 con 0x56412d3934b0 2024-04-23T14:53:30.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.257+0000 7fa7703ea740 10 client.0 _ll_register_callbacks cb 0x7fa7580d4600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:53:30.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.257+0000 7fa766a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56412d384860 0x56412d397b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:30.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.257+0000 7fa766a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56412d384860 0x56412d397b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:39204/0 (socket says 172.21.15.28:39204) 2024-04-23T14:53:30.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.257+0000 7fa766a00640 1 -- 172.21.15.28:0/30837405 learned_addr learned my addr 172.21.15.28:0/30837405 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:53:30.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.258+0000 7fa766000640 1 --2- 172.21.15.28:0/30837405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412d38f850 0x56412d392f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:30.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.258+0000 7fa767400640 1 --2- 172.21.15.28:0/30837405 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56412d3934b0 0x56412d393930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:30.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.258+0000 7fa766a00640 1 -- 172.21.15.28:0/30837405 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56412d3934b0 msgr2=0x56412d393930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:53:30.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.258+0000 7fa766a00640 1 --2- 172.21.15.28:0/30837405 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56412d3934b0 0x56412d393930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:30.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.258+0000 7fa766a00640 1 -- 172.21.15.28:0/30837405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412d38f850 msgr2=0x56412d392f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:53:30.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.258+0000 7fa766a00640 1 --2- 172.21.15.28:0/30837405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412d38f850 0x56412d392f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:30.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.258+0000 7fa766a00640 1 -- 172.21.15.28:0/30837405 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56412cfbc0f0 con 0x56412d384860 2024-04-23T14:53:30.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.258+0000 7fa767400640 1 --2- 172.21.15.28:0/30837405 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56412d3934b0 0x56412d393930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:53:30.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.258+0000 7fa766000640 1 --2- 172.21.15.28:0/30837405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412d38f850 0x56412d392f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:53:30.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.258+0000 7fa766a00640 1 --2- 172.21.15.28:0/30837405 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56412d384860 0x56412d397b90 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fa758098f50 tx=0x7fa7580d4720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4eb6568e0f01067 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:53:30.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.258+0000 7fa747e00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:53:30.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.258+0000 7fa747e00640 1 -- 172.21.15.28:0/30837405 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7580bb060 con 0x56412d384860 2024-04-23T14:53:30.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.258+0000 7fa747e00640 1 -- 172.21.15.28:0/30837405 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7fa7580bd2b0 con 0x56412d384860 2024-04-23T14:53:30.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.258+0000 7fa7703ea740 1 -- 172.21.15.28:0/30837405 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa754031830 con 0x56412d384860 2024-04-23T14:53:30.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.259+0000 7fa747e00640 1 -- 172.21.15.28:0/30837405 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fa758082050 con 0x56412d384860 2024-04-23T14:53:30.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.259+0000 7fa7703ea740 10 client.26795 fetch_fsmap learned FSMap version 20 2024-04-23T14:53:30.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.259+0000 7fa7703ea740 1 -- 172.21.15.28:0/30837405 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x56412d2af810 con 0x56412d384860 2024-04-23T14:53:30.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.259+0000 7fa747e00640 1 -- 172.21.15.28:0/30837405 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa754031830 con 0x56412d384860 2024-04-23T14:53:30.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.259+0000 7fa747e00640 1 -- 172.21.15.28:0/30837405 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fa7580d3040 con 0x56412d384860 2024-04-23T14:53:30.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.259+0000 7fa7703ea740 10 client.26795 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:53:30.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.259+0000 7fa7703ea740 10 client.26795 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:53:30.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.259+0000 7fa7703ea740 20 client.26795 populate_metadata read hostname 'smithi028' 2024-04-23T14:53:30.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.259+0000 7fa7703ea740 1 --2- 172.21.15.28:0/30837405 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56412d406620 0x56412d408a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:53:30.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.259+0000 7fa7703ea740 4 client.26795 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:53:30.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.259+0000 7fa7703ea740 1 -- 172.21.15.28:0/30837405 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56412d2f1a80 con 0x56412d406620 2024-04-23T14:53:30.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.259+0000 7fa766000640 1 --2- 172.21.15.28:0/30837405 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56412d406620 0x56412d408a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:53:30.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.260+0000 7fa766000640 1 --2- 172.21.15.28:0/30837405 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56412d406620 0x56412d408a40 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfc63afb1f7f353a server_cookie=7bc38a50d651848 in_seq=0 out_seq=0 2024-04-23T14:53:30.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.260+0000 7fa747e00640 10 client.26795 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:53:30.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.260+0000 7fa747e00640 1 -- 172.21.15.28:0/30837405 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56412d2f1a80 con 0x56412d406620 2024-04-23T14:53:30.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.260+0000 7fa747e00640 10 client.26795 handle_command_reply: tid=0 2024-04-23T14:53:30.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.260+0000 7fa7703ea740 1 client.26795 shutdown 2024-04-23T14:53:30.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.260+0000 7fa7703ea740 1 -- 172.21.15.28:0/30837405 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56412d384860 msgr2=0x56412d397b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:30.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.260+0000 7fa7703ea740 1 --2- 172.21.15.28:0/30837405 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56412d384860 0x56412d397b90 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fa758098f50 tx=0x7fa7580d4720 comp rx=0 tx=0).stop 2024-04-23T14:53:30.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.260+0000 7fa7703ea740 1 -- 172.21.15.28:0/30837405 shutdown_connections 2024-04-23T14:53:30.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.260+0000 7fa7703ea740 1 --2- 172.21.15.28:0/30837405 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56412d406620 0x56412d408a40 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:30.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.260+0000 7fa7703ea740 1 --2- 172.21.15.28:0/30837405 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56412d3934b0 0x56412d393930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:30.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.260+0000 7fa7703ea740 1 --2- 172.21.15.28:0/30837405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56412d38f850 0x56412d392f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:30.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.260+0000 7fa7703ea740 1 --2- 172.21.15.28:0/30837405 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56412d384860 0x56412d397b90 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:30.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.260+0000 7fa747e00640 0 client.26795 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:53:30.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.260+0000 7fa7703ea740 1 -- 172.21.15.28:0/30837405 >> 172.21.15.28:0/30837405 conn(0x56412d380b80 msgr2=0x56412d37f390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:53:30.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.261+0000 7fa7703ea740 1 -- 172.21.15.28:0/30837405 shutdown_connections 2024-04-23T14:53:30.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.261+0000 7fa7703ea740 1 -- 172.21.15.28:0/30837405 wait complete. 2024-04-23T14:53:30.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.261+0000 7fa7703ea740 20 client.26795 trim_cache size 0 max 16384 2024-04-23T14:53:30.265 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:53:30.265 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-23T14:53:30.265 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-23T14:53:30.265 INFO:teuthology.orchestra.run.smithi028.stdout: "9af96ddf-cc40-45b3-a594-2df1676f49a2": { 2024-04-23T14:53:30.265 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-23T14:53:30.265 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "9af96ddf-cc40-45b3-a594-2df1676f49a2", 2024-04-23T14:53:30.265 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-23T14:53:30.265 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:53:30.265 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:53:30.265 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:53:30.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.261+0000 7fa76d600640 1 -- 172.21.15.28:0/1175830730 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa74c0a3d20 msgr2=0x7fa74c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:30.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.261+0000 7fa76d600640 1 --2- 172.21.15.28:0/1175830730 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa74c0a3d20 0x7fa74c0a61e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa760009de0 tx=0x7fa760005f90 comp rx=0 tx=0).stop 2024-04-23T14:53:30.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.261+0000 7fa76d600640 1 -- 172.21.15.28:0/1175830730 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7680743a0 msgr2=0x7fa76819ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:53:30.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.261+0000 7fa76d600640 1 --2- 172.21.15.28:0/1175830730 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7680743a0 0x7fa76819ef40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa75800ebc0 tx=0x7fa75800c6a0 comp rx=0 tx=0).stop 2024-04-23T14:53:30.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.262+0000 7fa76d600640 1 -- 172.21.15.28:0/1175830730 shutdown_connections 2024-04-23T14:53:30.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.262+0000 7fa76d600640 1 --2- 172.21.15.28:0/1175830730 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fa74c0a3d20 0x7fa74c0a61e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:30.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.262+0000 7fa76d600640 1 --2- 172.21.15.28:0/1175830730 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa768074cc0 0x7fa76819f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:30.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.262+0000 7fa76d600640 1 --2- 172.21.15.28:0/1175830730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa768112fc0 0x7fa7681a3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:30.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.262+0000 7fa76d600640 1 --2- 172.21.15.28:0/1175830730 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7680743a0 0x7fa76819ef40 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:53:30.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.262+0000 7fa76d600640 1 -- 172.21.15.28:0/1175830730 >> 172.21.15.28:0/1175830730 conn(0x7fa7680fda80 msgr2=0x7fa7680ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:53:30.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.262+0000 7fa76d600640 1 -- 172.21.15.28:0/1175830730 shutdown_connections 2024-04-23T14:53:30.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:53:30.262+0000 7fa76d600640 1 -- 172.21.15.28:0/1175830730 wait complete. 2024-04-23T14:53:30.281 INFO:tasks.cephfs.filesystem:scrub status for tag:9af96ddf-cc40-45b3-a594-2df1676f49a2 - {'path': '/', 'tag': '9af96ddf-cc40-45b3-a594-2df1676f49a2', 'options': 'recursive,force'} 2024-04-23T14:53:30.519 INFO:tasks.workunit.client.0.smithi028.stdout: 1 801006 8.87 MB/sec execute 346 sec latency 18.934 ms 2024-04-23T14:53:31.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:30 smithi105 ceph-mon[32555]: pgmap v293: 97 pgs: 97 active+clean; 509 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 135 op/s 2024-04-23T14:53:31.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:30 smithi038 ceph-mon[32348]: pgmap v293: 97 pgs: 97 active+clean; 509 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 135 op/s 2024-04-23T14:53:31.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:30 smithi028 ceph-mon[27810]: pgmap v293: 97 pgs: 97 active+clean; 509 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 135 op/s 2024-04-23T14:53:31.520 INFO:tasks.workunit.client.0.smithi028.stdout: 1 802684 8.88 MB/sec execute 347 sec latency 37.747 ms 2024-04-23T14:53:32.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:31 smithi105 ceph-mon[32555]: scrub queued for path: / 2024-04-23T14:53:32.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:31 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:53:32.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:31 smithi105 ceph-mon[32555]: scrub summary: active paths [/] 2024-04-23T14:53:32.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:31 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:53:32.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:31 smithi038 ceph-mon[32348]: scrub queued for path: / 2024-04-23T14:53:32.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:31 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:53:32.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:31 smithi038 ceph-mon[32348]: scrub summary: active paths [/] 2024-04-23T14:53:32.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:31 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:53:32.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:31 smithi028 ceph-mon[27810]: scrub queued for path: / 2024-04-23T14:53:32.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:31 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:53:32.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:31 smithi028 ceph-mon[27810]: scrub summary: active paths [/] 2024-04-23T14:53:32.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:31 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:53:32.520 INFO:tasks.workunit.client.0.smithi028.stdout: 1 804057 8.87 MB/sec execute 348 sec latency 18.567 ms 2024-04-23T14:53:33.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:32 smithi105 ceph-mon[32555]: pgmap v294: 97 pgs: 97 active+clean; 509 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 135 op/s 2024-04-23T14:53:33.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:32 smithi038 ceph-mon[32348]: pgmap v294: 97 pgs: 97 active+clean; 509 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 135 op/s 2024-04-23T14:53:33.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:32 smithi028 ceph-mon[27810]: pgmap v294: 97 pgs: 97 active+clean; 509 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 135 op/s 2024-04-23T14:53:33.520 INFO:tasks.workunit.client.0.smithi028.stdout: 1 806228 8.88 MB/sec execute 349 sec latency 21.509 ms 2024-04-23T14:53:34.520 INFO:tasks.workunit.client.0.smithi028.stdout: 1 807646 8.87 MB/sec execute 350 sec latency 16.972 ms 2024-04-23T14:53:35.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:34 smithi105 ceph-mon[32555]: pgmap v295: 97 pgs: 97 active+clean; 477 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 205 op/s 2024-04-23T14:53:35.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:34 smithi038 ceph-mon[32348]: pgmap v295: 97 pgs: 97 active+clean; 477 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 205 op/s 2024-04-23T14:53:35.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:34 smithi028 ceph-mon[27810]: pgmap v295: 97 pgs: 97 active+clean; 477 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 205 op/s 2024-04-23T14:53:35.520 INFO:tasks.workunit.client.0.smithi028.stdout: 1 809826 8.89 MB/sec execute 351 sec latency 15.781 ms 2024-04-23T14:53:36.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:35 smithi105 ceph-mon[32555]: scrub summary: idle 2024-04-23T14:53:36.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:35 smithi038 ceph-mon[32348]: scrub summary: idle 2024-04-23T14:53:36.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:35 smithi028 ceph-mon[27810]: scrub summary: idle 2024-04-23T14:53:36.520 INFO:tasks.workunit.client.0.smithi028.stdout: 1 811061 8.88 MB/sec execute 352 sec latency 103.557 ms 2024-04-23T14:53:37.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:36 smithi105 ceph-mon[32555]: pgmap v296: 97 pgs: 97 active+clean; 477 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 138 op/s 2024-04-23T14:53:37.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:36 smithi038 ceph-mon[32348]: pgmap v296: 97 pgs: 97 active+clean; 477 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 138 op/s 2024-04-23T14:53:37.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:36 smithi028 ceph-mon[27810]: pgmap v296: 97 pgs: 97 active+clean; 477 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 138 op/s 2024-04-23T14:53:37.520 INFO:tasks.workunit.client.0.smithi028.stdout: 1 813316 8.89 MB/sec execute 353 sec latency 14.285 ms 2024-04-23T14:53:38.520 INFO:tasks.workunit.client.0.smithi028.stdout: 1 814606 8.88 MB/sec execute 354 sec latency 17.896 ms 2024-04-23T14:53:39.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:38 smithi105 ceph-mon[32555]: pgmap v297: 97 pgs: 97 active+clean; 486 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 208 op/s 2024-04-23T14:53:39.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:38 smithi038 ceph-mon[32348]: pgmap v297: 97 pgs: 97 active+clean; 486 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 208 op/s 2024-04-23T14:53:39.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:38 smithi028 ceph-mon[27810]: pgmap v297: 97 pgs: 97 active+clean; 486 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 208 op/s 2024-04-23T14:53:39.521 INFO:tasks.workunit.client.0.smithi028.stdout: 1 816555 8.89 MB/sec execute 355 sec latency 24.313 ms 2024-04-23T14:53:40.521 INFO:tasks.workunit.client.0.smithi028.stdout: 1 817508 8.87 MB/sec execute 356 sec latency 49.434 ms 2024-04-23T14:53:41.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:40 smithi105 ceph-mon[32555]: pgmap v298: 97 pgs: 97 active+clean; 486 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.6 MiB/s wr, 140 op/s 2024-04-23T14:53:41.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:40 smithi038 ceph-mon[32348]: pgmap v298: 97 pgs: 97 active+clean; 486 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.6 MiB/s wr, 140 op/s 2024-04-23T14:53:41.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:40 smithi028 ceph-mon[27810]: pgmap v298: 97 pgs: 97 active+clean; 486 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.6 MiB/s wr, 140 op/s 2024-04-23T14:53:41.521 INFO:tasks.workunit.client.0.smithi028.stdout: 1 819035 8.87 MB/sec execute 357 sec latency 62.962 ms 2024-04-23T14:53:42.521 INFO:tasks.workunit.client.0.smithi028.stdout: 1 820607 8.88 MB/sec execute 358 sec latency 23.988 ms 2024-04-23T14:53:43.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:42 smithi105 ceph-mon[32555]: pgmap v299: 97 pgs: 97 active+clean; 486 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.6 MiB/s wr, 140 op/s 2024-04-23T14:53:43.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:42 smithi038 ceph-mon[32348]: pgmap v299: 97 pgs: 97 active+clean; 486 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.6 MiB/s wr, 140 op/s 2024-04-23T14:53:43.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:42 smithi028 ceph-mon[27810]: pgmap v299: 97 pgs: 97 active+clean; 486 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.6 MiB/s wr, 140 op/s 2024-04-23T14:53:43.521 INFO:tasks.workunit.client.0.smithi028.stdout: 1 822029 8.87 MB/sec execute 359 sec latency 20.735 ms 2024-04-23T14:53:44.521 INFO:tasks.workunit.client.0.smithi028.stdout: 1 824107 8.88 MB/sec execute 360 sec latency 13.726 ms 2024-04-23T14:53:45.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:44 smithi105 ceph-mon[32555]: pgmap v300: 97 pgs: 97 active+clean; 494 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.9 MiB/s wr, 208 op/s 2024-04-23T14:53:45.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:44 smithi038 ceph-mon[32348]: pgmap v300: 97 pgs: 97 active+clean; 494 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.9 MiB/s wr, 208 op/s 2024-04-23T14:53:45.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:44 smithi028 ceph-mon[27810]: pgmap v300: 97 pgs: 97 active+clean; 494 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.9 MiB/s wr, 208 op/s 2024-04-23T14:53:45.522 INFO:tasks.workunit.client.0.smithi028.stdout: 1 825510 8.87 MB/sec execute 361 sec latency 26.249 ms 2024-04-23T14:53:46.522 INFO:tasks.workunit.client.0.smithi028.stdout: 1 827593 8.88 MB/sec execute 362 sec latency 21.699 ms 2024-04-23T14:53:47.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:46 smithi105 ceph-mon[32555]: pgmap v301: 97 pgs: 97 active+clean; 494 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 138 op/s 2024-04-23T14:53:47.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:46 smithi038 ceph-mon[32348]: pgmap v301: 97 pgs: 97 active+clean; 494 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 138 op/s 2024-04-23T14:53:47.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:46 smithi028 ceph-mon[27810]: pgmap v301: 97 pgs: 97 active+clean; 494 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 138 op/s 2024-04-23T14:53:47.522 INFO:tasks.workunit.client.0.smithi028.stdout: 1 828886 8.87 MB/sec execute 363 sec latency 53.623 ms 2024-04-23T14:53:48.522 INFO:tasks.workunit.client.0.smithi028.stdout: 1 831051 8.89 MB/sec execute 364 sec latency 13.634 ms 2024-04-23T14:53:49.038 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:48 smithi105 ceph-mon[32555]: pgmap v302: 97 pgs: 97 active+clean; 503 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.1 MiB/s wr, 210 op/s 2024-04-23T14:53:49.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:48 smithi038 ceph-mon[32348]: pgmap v302: 97 pgs: 97 active+clean; 503 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.1 MiB/s wr, 210 op/s 2024-04-23T14:53:49.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:48 smithi028 ceph-mon[27810]: pgmap v302: 97 pgs: 97 active+clean; 503 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.1 MiB/s wr, 210 op/s 2024-04-23T14:53:49.522 INFO:tasks.workunit.client.0.smithi028.stdout: 1 832470 8.88 MB/sec execute 365 sec latency 26.140 ms 2024-04-23T14:53:50.522 INFO:tasks.workunit.client.0.smithi028.stdout: 1 834606 8.89 MB/sec execute 366 sec latency 13.473 ms 2024-04-23T14:53:51.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:50 smithi105 ceph-mon[32555]: pgmap v303: 97 pgs: 97 active+clean; 503 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 139 op/s 2024-04-23T14:53:51.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:50 smithi038 ceph-mon[32348]: pgmap v303: 97 pgs: 97 active+clean; 503 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 139 op/s 2024-04-23T14:53:51.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:50 smithi028 ceph-mon[27810]: pgmap v303: 97 pgs: 97 active+clean; 503 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 139 op/s 2024-04-23T14:53:51.522 INFO:tasks.workunit.client.0.smithi028.stdout: 1 835824 8.88 MB/sec execute 367 sec latency 19.978 ms 2024-04-23T14:53:52.522 INFO:tasks.workunit.client.0.smithi028.stdout: 1 837524 8.88 MB/sec execute 368 sec latency 108.192 ms 2024-04-23T14:53:53.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:52 smithi105 ceph-mon[32555]: pgmap v304: 97 pgs: 97 active+clean; 503 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 139 op/s 2024-04-23T14:53:53.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:52 smithi038 ceph-mon[32348]: pgmap v304: 97 pgs: 97 active+clean; 503 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 139 op/s 2024-04-23T14:53:53.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:52 smithi028 ceph-mon[27810]: pgmap v304: 97 pgs: 97 active+clean; 503 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 139 op/s 2024-04-23T14:53:53.522 INFO:tasks.workunit.client.0.smithi028.stdout: 1 838830 8.87 MB/sec execute 369 sec latency 20.111 ms 2024-04-23T14:53:54.523 INFO:tasks.workunit.client.0.smithi028.stdout: 1 840807 8.88 MB/sec execute 370 sec latency 26.337 ms 2024-04-23T14:53:55.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:54 smithi105 ceph-mon[32555]: pgmap v305: 97 pgs: 97 active+clean; 511 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.1 MiB/s wr, 207 op/s 2024-04-23T14:53:55.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:54 smithi038 ceph-mon[32348]: pgmap v305: 97 pgs: 97 active+clean; 511 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.1 MiB/s wr, 207 op/s 2024-04-23T14:53:55.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:54 smithi028 ceph-mon[27810]: pgmap v305: 97 pgs: 97 active+clean; 511 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.1 MiB/s wr, 207 op/s 2024-04-23T14:53:55.523 INFO:tasks.workunit.client.0.smithi028.stdout: 1 842268 8.87 MB/sec execute 371 sec latency 18.498 ms 2024-04-23T14:53:56.523 INFO:tasks.workunit.client.0.smithi028.stdout: 1 843940 8.87 MB/sec execute 372 sec latency 19.010 ms 2024-04-23T14:53:57.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:56 smithi038 ceph-mon[32348]: pgmap v306: 97 pgs: 97 active+clean; 511 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 140 op/s 2024-04-23T14:53:57.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:56 smithi028 ceph-mon[27810]: pgmap v306: 97 pgs: 97 active+clean; 511 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 140 op/s 2024-04-23T14:53:57.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:56 smithi105 ceph-mon[32555]: pgmap v306: 97 pgs: 97 active+clean; 511 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 140 op/s 2024-04-23T14:53:57.523 INFO:tasks.workunit.client.0.smithi028.stdout: 1 845436 8.88 MB/sec execute 373 sec latency 53.600 ms 2024-04-23T14:53:58.523 INFO:tasks.workunit.client.0.smithi028.stdout: 1 846724 8.87 MB/sec execute 374 sec latency 21.371 ms 2024-04-23T14:53:59.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:53:58 smithi038 ceph-mon[32348]: pgmap v307: 97 pgs: 97 active+clean; 519 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.2 MiB/s wr, 208 op/s 2024-04-23T14:53:59.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:53:58 smithi028 ceph-mon[27810]: pgmap v307: 97 pgs: 97 active+clean; 519 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.2 MiB/s wr, 208 op/s 2024-04-23T14:53:59.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:53:58 smithi105 ceph-mon[32555]: pgmap v307: 97 pgs: 97 active+clean; 519 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.2 MiB/s wr, 208 op/s 2024-04-23T14:53:59.523 INFO:tasks.workunit.client.0.smithi028.stdout: 1 848703 8.88 MB/sec execute 375 sec latency 15.854 ms 2024-04-23T14:54:00.282 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:54:00.282 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:54:00.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.507+0000 7fe83c800640 1 -- 172.21.15.28:0/2744876032 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe83410e660 msgr2=0x7fe83410ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:00.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.507+0000 7fe83c800640 1 --2- 172.21.15.28:0/2744876032 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe83410e660 0x7fe83410ea40 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fe828009900 tx=0x7fe82802a3f0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.507+0000 7fe83c800640 1 -- 172.21.15.28:0/2744876032 shutdown_connections 2024-04-23T14:54:00.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.507+0000 7fe83c800640 1 --2- 172.21.15.28:0/2744876032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe834108fd0 0x7fe834115490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.507+0000 7fe83c800640 1 --2- 172.21.15.28:0/2744876032 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe834108610 0x7fe834108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.507+0000 7fe83c800640 1 --2- 172.21.15.28:0/2744876032 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe83410e660 0x7fe83410ea40 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.507+0000 7fe83c800640 1 -- 172.21.15.28:0/2744876032 >> 172.21.15.28:0/2744876032 conn(0x7fe834076540 msgr2=0x7fe834078960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:00.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.507+0000 7fe83c800640 1 -- 172.21.15.28:0/2744876032 shutdown_connections 2024-04-23T14:54:00.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.507+0000 7fe83c800640 1 -- 172.21.15.28:0/2744876032 wait complete. 2024-04-23T14:54:00.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.508+0000 7fe83c800640 1 Processor -- start 2024-04-23T14:54:00.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.508+0000 7fe83c800640 1 -- start start 2024-04-23T14:54:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.509+0000 7fe83c800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe834108610 0x7fe83419ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.509+0000 7fe83c800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe834108fd0 0x7fe83419f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.509+0000 7fe83c800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe83410e660 0x7fe8341a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.509+0000 7fe83c800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe8341189f0 con 0x7fe834108fd0 2024-04-23T14:54:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.509+0000 7fe83c800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe834118870 con 0x7fe834108610 2024-04-23T14:54:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.509+0000 7fe83c800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe834118b70 con 0x7fe83410e660 2024-04-23T14:54:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.509+0000 7fe839e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe834108610 0x7fe83419ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.509+0000 7fe839e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe834108610 0x7fe83419ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40040/0 (socket says 172.21.15.28:40040) 2024-04-23T14:54:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.509+0000 7fe839e00640 1 -- 172.21.15.28:0/2219582601 learned_addr learned my addr 172.21.15.28:0/2219582601 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.509+0000 7fe839400640 1 --2- 172.21.15.28:0/2219582601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe834108fd0 0x7fe83419f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.509+0000 7fe83a800640 1 --2- 172.21.15.28:0/2219582601 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe83410e660 0x7fe8341a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.510+0000 7fe839e00640 1 -- 172.21.15.28:0/2219582601 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe83410e660 msgr2=0x7fe8341a3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.510+0000 7fe839e00640 1 --2- 172.21.15.28:0/2219582601 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe83410e660 0x7fe8341a3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.510+0000 7fe839e00640 1 -- 172.21.15.28:0/2219582601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe834108fd0 msgr2=0x7fe83419f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.510+0000 7fe839e00640 1 --2- 172.21.15.28:0/2219582601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe834108fd0 0x7fe83419f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.510+0000 7fe839e00640 1 -- 172.21.15.28:0/2219582601 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8341a3e40 con 0x7fe834108610 2024-04-23T14:54:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.510+0000 7fe839e00640 1 --2- 172.21.15.28:0/2219582601 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe834108610 0x7fe83419ee50 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fe82802a960 tx=0x7fe828003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.510+0000 7fe82ea00640 1 -- 172.21.15.28:0/2219582601 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe82803f070 con 0x7fe834108610 2024-04-23T14:54:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.510+0000 7fe82ea00640 1 -- 172.21.15.28:0/2219582601 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe828031c20 con 0x7fe834108610 2024-04-23T14:54:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.510+0000 7fe82ea00640 1 -- 172.21.15.28:0/2219582601 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe828035030 con 0x7fe834108610 2024-04-23T14:54:00.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.510+0000 7fe83c800640 1 -- 172.21.15.28:0/2219582601 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8341a40d0 con 0x7fe834108610 2024-04-23T14:54:00.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.511+0000 7fe83c800640 1 -- 172.21.15.28:0/2219582601 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8341ab9c0 con 0x7fe834108610 2024-04-23T14:54:00.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.514+0000 7fe82ea00640 1 -- 172.21.15.28:0/2219582601 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fe828042070 con 0x7fe834108610 2024-04-23T14:54:00.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.514+0000 7fe83f67e740 1 Processor -- start 2024-04-23T14:54:00.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.514+0000 7fe83f67e740 1 -- start start 2024-04-23T14:54:00.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.514+0000 7fe83f67e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a37eb8e860 0x55a37eb8ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:00.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.514+0000 7fe83f67e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a37eb8f180 0x55a37eb99310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:00.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.515+0000 7fe83f67e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a37eb99850 0x55a37eb9bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:00.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.515+0000 7fe83f67e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a37eaf41f0 con 0x55a37eb8f180 2024-04-23T14:54:00.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.515+0000 7fe83f67e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a37ea9a350 con 0x55a37eb8e860 2024-04-23T14:54:00.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.515+0000 7fe83f67e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a37eaf01b0 con 0x55a37eb99850 2024-04-23T14:54:00.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.515+0000 7fe83a800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a37eb8e860 0x55a37eb8ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:00.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.515+0000 7fe83a800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a37eb8e860 0x55a37eb8ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40056/0 (socket says 172.21.15.28:40056) 2024-04-23T14:54:00.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.515+0000 7fe83a800640 1 -- 172.21.15.28:0/3097193124 learned_addr learned my addr 172.21.15.28:0/3097193124 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:00.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.515+0000 7fe839e00640 1 --2- 172.21.15.28:0/3097193124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a37eb8f180 0x55a37eb99310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:00.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.515+0000 7fe839400640 1 --2- 172.21.15.28:0/3097193124 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a37eb99850 0x55a37eb9bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:00.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.515+0000 7fe83a800640 1 -- 172.21.15.28:0/3097193124 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a37eb99850 msgr2=0x55a37eb9bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:00.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.515+0000 7fe83a800640 1 --2- 172.21.15.28:0/3097193124 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a37eb99850 0x55a37eb9bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.515+0000 7fe83a800640 1 -- 172.21.15.28:0/3097193124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a37eb8f180 msgr2=0x55a37eb99310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:00.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.515+0000 7fe83a800640 1 --2- 172.21.15.28:0/3097193124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a37eb8f180 0x55a37eb99310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.515+0000 7fe83a800640 1 -- 172.21.15.28:0/3097193124 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a37eaeef50 con 0x55a37eb8e860 2024-04-23T14:54:00.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.516+0000 7fe83a800640 1 --2- 172.21.15.28:0/3097193124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a37eb8e860 0x55a37eb8ec40 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fe830009fd0 tx=0x7fe8300118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbc013588f6fb573 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:00.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.516+0000 7fe82ea00640 1 --2- 172.21.15.28:0/2219582601 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe81c0a3de0 0x7fe81c0a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:00.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.516+0000 7fe823000640 1 -- 172.21.15.28:0/3097193124 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe830004d20 con 0x55a37eb8e860 2024-04-23T14:54:00.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.516+0000 7fe823000640 1 -- 172.21.15.28:0/3097193124 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe830004050 con 0x55a37eb8e860 2024-04-23T14:54:00.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.516+0000 7fe823000640 1 -- 172.21.15.28:0/3097193124 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe830005260 con 0x55a37eb8e860 2024-04-23T14:54:00.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.516+0000 7fe82ea00640 1 -- 172.21.15.28:0/2219582601 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7fe8280f0e90 con 0x7fe834108610 2024-04-23T14:54:00.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.516+0000 7fe839e00640 1 --2- 172.21.15.28:0/2219582601 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe81c0a3de0 0x7fe81c0a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:00.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.517+0000 7fe83f67e740 1 -- 172.21.15.28:0/3097193124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a37eb8e860 msgr2=0x55a37eb8ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:00.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.517+0000 7fe83f67e740 1 --2- 172.21.15.28:0/3097193124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a37eb8e860 0x55a37eb8ec40 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fe830009fd0 tx=0x7fe8300118d0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.517+0000 7fe839e00640 1 --2- 172.21.15.28:0/2219582601 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe81c0a3de0 0x7fe81c0a62a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x55a37eb8fe20 tx=0x7fe8280f4940 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:00.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.517+0000 7fe83f67e740 1 -- 172.21.15.28:0/3097193124 shutdown_connections 2024-04-23T14:54:00.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.517+0000 7fe83f67e740 1 --2- 172.21.15.28:0/3097193124 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a37eb99850 0x55a37eb9bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.517+0000 7fe83f67e740 1 --2- 172.21.15.28:0/3097193124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a37eb8f180 0x55a37eb99310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.517+0000 7fe83f67e740 1 --2- 172.21.15.28:0/3097193124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a37eb8e860 0x55a37eb8ec40 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.517+0000 7fe83f67e740 1 -- 172.21.15.28:0/3097193124 >> 172.21.15.28:0/3097193124 conn(0x55a37eb8ab80 msgr2=0x55a37eb8af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:00.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.517+0000 7fe83f67e740 1 -- 172.21.15.28:0/3097193124 shutdown_connections 2024-04-23T14:54:00.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.517+0000 7fe83f67e740 1 -- 172.21.15.28:0/3097193124 wait complete. 2024-04-23T14:54:00.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.518+0000 7fe83f67e740 1 -- start start 2024-04-23T14:54:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.518+0000 7fe83f67e740 1 Processor -- start 2024-04-23T14:54:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.518+0000 7fe83f67e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a37eb8e860 0x55a37eb96820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.518+0000 7fe83f67e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a37eb8f180 0x55a37eb96d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.518+0000 7fe83f67e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a37eb99850 0x55a37eb972a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.518+0000 7fe83f67e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a37eb9c640 con 0x55a37eb99850 2024-04-23T14:54:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.518+0000 7fe83f67e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a37eaad690 con 0x55a37eb8e860 2024-04-23T14:54:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.518+0000 7fe83f67e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a37eb9c7c0 con 0x55a37eb8f180 2024-04-23T14:54:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.518+0000 7fe83f67e740 10 client.0 _ll_register_callbacks cb 0x55a37eae9650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:54:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.519+0000 7fe83a800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a37eb8e860 0x55a37eb96820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.519+0000 7fe83a800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a37eb8e860 0x55a37eb96820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40058/0 (socket says 172.21.15.28:40058) 2024-04-23T14:54:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.519+0000 7fe83a800640 1 -- 172.21.15.28:0/56224711 learned_addr learned my addr 172.21.15.28:0/56224711 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.519+0000 7fe839400640 1 --2- 172.21.15.28:0/56224711 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a37eb8f180 0x55a37eb96d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.519+0000 7fe83a800640 1 --2- 172.21.15.28:0/56224711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a37eb99850 0x55a37eb972a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.519+0000 7fe839400640 1 -- 172.21.15.28:0/56224711 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a37eb8e860 msgr2=0x55a37eb96820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.519+0000 7fe839400640 1 --2- 172.21.15.28:0/56224711 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a37eb8e860 0x55a37eb96820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.519+0000 7fe839400640 1 -- 172.21.15.28:0/56224711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a37eb99850 msgr2=0x55a37eb972a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.519+0000 7fe839400640 1 --2- 172.21.15.28:0/56224711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a37eb99850 0x55a37eb972a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.519+0000 7fe839400640 1 -- 172.21.15.28:0/56224711 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a37eaac920 con 0x55a37eb8f180 2024-04-23T14:54:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.519+0000 7fe83a800640 1 --2- 172.21.15.28:0/56224711 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a37eb8e860 0x55a37eb96820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:54:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.520+0000 7fe839400640 1 --2- 172.21.15.28:0/56224711 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a37eb8f180 0x55a37eb96d60 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe8240126d0 tx=0x7fe824012b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d9ba1f412df0954 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.520+0000 7fe817400640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:54:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.520+0000 7fe817400640 1 -- 172.21.15.28:0/56224711 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe82400f920 con 0x55a37eb8f180 2024-04-23T14:54:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.520+0000 7fe83f67e740 1 -- 172.21.15.28:0/56224711 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe830004050 con 0x55a37eb8f180 2024-04-23T14:54:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.520+0000 7fe817400640 1 -- 172.21.15.28:0/56224711 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7fe82400fac0 con 0x55a37eb8f180 2024-04-23T14:54:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.520+0000 7fe817400640 1 -- 172.21.15.28:0/56224711 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fe82400fcc0 con 0x55a37eb8f180 2024-04-23T14:54:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.520+0000 7fe83f67e740 10 client.26785 fetch_fsmap learned FSMap version 20 2024-04-23T14:54:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.520+0000 7fe83f67e740 1 -- 172.21.15.28:0/56224711 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55a37e8d07d0 con 0x55a37eb8f180 2024-04-23T14:54:00.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.521+0000 7fe817400640 1 -- 172.21.15.28:0/56224711 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe830004050 con 0x55a37eb8f180 2024-04-23T14:54:00.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.521+0000 7fe817400640 1 -- 172.21.15.28:0/56224711 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fe82401c420 con 0x55a37eb8f180 2024-04-23T14:54:00.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.521+0000 7fe83f67e740 10 client.26785 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:54:00.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.521+0000 7fe83f67e740 10 client.26785 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:54:00.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.521+0000 7fe83f67e740 20 client.26785 populate_metadata read hostname 'smithi028' 2024-04-23T14:54:00.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.521+0000 7fe83f67e740 1 --2- 172.21.15.28:0/56224711 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55a37eba9d50 0x55a37ebac190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:00.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.521+0000 7fe83f67e740 4 client.26785 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:54:00.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.521+0000 7fe83f67e740 1 -- 172.21.15.28:0/56224711 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a37eb01c10 con 0x55a37eba9d50 2024-04-23T14:54:00.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.521+0000 7fe83a800640 1 --2- 172.21.15.28:0/56224711 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55a37eba9d50 0x55a37ebac190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:00.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.522+0000 7fe83a800640 1 --2- 172.21.15.28:0/56224711 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55a37eba9d50 0x55a37ebac190 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=574c806c61d60502 server_cookie=98c1fa4beb29a634 in_seq=0 out_seq=0 2024-04-23T14:54:00.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.522+0000 7fe817400640 10 client.26785 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:54:00.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.523+0000 7fe817400640 1 -- 172.21.15.28:0/56224711 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a37eb01c10 con 0x55a37eba9d50 2024-04-23T14:54:00.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.523+0000 7fe817400640 10 client.26785 handle_command_reply: tid=0 2024-04-23T14:54:00.527 INFO:tasks.workunit.client.0.smithi028.stdout: 1 849796 8.86 MB/sec execute 376 sec latency 22.162 ms 2024-04-23T14:54:00.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.523+0000 7fe83f67e740 1 client.26785 shutdown 2024-04-23T14:54:00.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.523+0000 7fe83f67e740 1 -- 172.21.15.28:0/56224711 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a37eb8f180 msgr2=0x55a37eb96d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:00.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.523+0000 7fe83f67e740 1 --2- 172.21.15.28:0/56224711 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a37eb8f180 0x55a37eb96d60 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe8240126d0 tx=0x7fe824012b90 comp rx=0 tx=0).stop 2024-04-23T14:54:00.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.523+0000 7fe83f67e740 1 -- 172.21.15.28:0/56224711 shutdown_connections 2024-04-23T14:54:00.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.523+0000 7fe83f67e740 1 --2- 172.21.15.28:0/56224711 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55a37eba9d50 0x55a37ebac190 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.523+0000 7fe83f67e740 1 --2- 172.21.15.28:0/56224711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a37eb99850 0x55a37eb972a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.523+0000 7fe83f67e740 1 --2- 172.21.15.28:0/56224711 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a37eb8f180 0x55a37eb96d60 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.523+0000 7fe83f67e740 1 --2- 172.21.15.28:0/56224711 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a37eb8e860 0x55a37eb96820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.523+0000 7fe83f67e740 1 -- 172.21.15.28:0/56224711 >> 172.21.15.28:0/56224711 conn(0x55a37eb8ab80 msgr2=0x55a37eb90840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:00.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.523+0000 7fe817400640 0 client.26785 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:54:00.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.523+0000 7fe83f67e740 1 -- 172.21.15.28:0/56224711 shutdown_connections 2024-04-23T14:54:00.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.523+0000 7fe83f67e740 1 -- 172.21.15.28:0/56224711 wait complete. 2024-04-23T14:54:00.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.523+0000 7fe83f67e740 20 client.26785 trim_cache size 0 max 16384 2024-04-23T14:54:00.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.544+0000 7fe83f67e740 1 Processor -- start 2024-04-23T14:54:00.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.544+0000 7fe83f67e740 1 -- start start 2024-04-23T14:54:00.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.544+0000 7fe83f67e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a37eb8e860 0x55a37eb95760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:00.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.544+0000 7fe83f67e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a37eb8f180 0x55a37eba5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:00.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.544+0000 7fe83f67e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a37eb99850 0x55a37eb90fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:00.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.544+0000 7fe83f67e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe8341b2370 con 0x55a37eb8f180 2024-04-23T14:54:00.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.544+0000 7fe83f67e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe8341b4bd0 con 0x55a37eb8e860 2024-04-23T14:54:00.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.544+0000 7fe83f67e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe8341b29c0 con 0x55a37eb99850 2024-04-23T14:54:00.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.545+0000 7fe83a800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a37eb8e860 0x55a37eb95760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:00.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.545+0000 7fe839400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a37eb8f180 0x55a37eba5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:00.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.545+0000 7fe83a800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a37eb8e860 0x55a37eb95760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40064/0 (socket says 172.21.15.28:40064) 2024-04-23T14:54:00.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.545+0000 7fe83a800640 1 -- 172.21.15.28:0/2249722009 learned_addr learned my addr 172.21.15.28:0/2249722009 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:00.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.545+0000 7fe83a800640 1 --2- 172.21.15.28:0/2249722009 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a37eb99850 0x55a37eb90fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:00.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.545+0000 7fe83a800640 1 -- 172.21.15.28:0/2249722009 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a37eb99850 msgr2=0x55a37eb90fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:00.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.545+0000 7fe83a800640 1 --2- 172.21.15.28:0/2249722009 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a37eb99850 0x55a37eb90fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.545+0000 7fe83a800640 1 -- 172.21.15.28:0/2249722009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a37eb8f180 msgr2=0x55a37eba5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:00.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.545+0000 7fe83a800640 1 --2- 172.21.15.28:0/2249722009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a37eb8f180 0x55a37eba5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.545+0000 7fe83a800640 1 -- 172.21.15.28:0/2249722009 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a37e8063f0 con 0x55a37eb8e860 2024-04-23T14:54:00.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.545+0000 7fe83a800640 1 --2- 172.21.15.28:0/2249722009 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a37eb8e860 0x55a37eb95760 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe83000a010 tx=0x7fe830009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27caa82cb4a40e7a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:00.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.546+0000 7fe822600640 1 -- 172.21.15.28:0/2249722009 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe830004520 con 0x55a37eb8e860 2024-04-23T14:54:00.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.546+0000 7fe822600640 1 -- 172.21.15.28:0/2249722009 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8300046c0 con 0x55a37eb8e860 2024-04-23T14:54:00.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.546+0000 7fe822600640 1 -- 172.21.15.28:0/2249722009 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8300286b0 con 0x55a37eb8e860 2024-04-23T14:54:00.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.546+0000 7fe83f67e740 1 -- 172.21.15.28:0/2249722009 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a37eb8e860 msgr2=0x55a37eb95760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:00.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.546+0000 7fe83f67e740 1 --2- 172.21.15.28:0/2249722009 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a37eb8e860 0x55a37eb95760 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe83000a010 tx=0x7fe830009500 comp rx=0 tx=0).stop 2024-04-23T14:54:00.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.547+0000 7fe83f67e740 1 -- 172.21.15.28:0/2249722009 shutdown_connections 2024-04-23T14:54:00.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.547+0000 7fe83f67e740 1 --2- 172.21.15.28:0/2249722009 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a37eb99850 0x55a37eb90fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.547+0000 7fe83f67e740 1 --2- 172.21.15.28:0/2249722009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a37eb8f180 0x55a37eba5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.547+0000 7fe83f67e740 1 --2- 172.21.15.28:0/2249722009 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a37eb8e860 0x55a37eb95760 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.547+0000 7fe83f67e740 1 -- 172.21.15.28:0/2249722009 >> 172.21.15.28:0/2249722009 conn(0x55a37eb8ab80 msgr2=0x55a37eb97070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:00.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.547+0000 7fe83f67e740 1 -- 172.21.15.28:0/2249722009 shutdown_connections 2024-04-23T14:54:00.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.547+0000 7fe83f67e740 1 -- 172.21.15.28:0/2249722009 wait complete. 2024-04-23T14:54:00.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.547+0000 7fe83f67e740 1 -- start start 2024-04-23T14:54:00.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.547+0000 7fe83f67e740 1 Processor -- start 2024-04-23T14:54:00.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.548+0000 7fe83f67e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a37eb8e860 0x55a37eb9d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:00.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.548+0000 7fe83f67e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a37eb8f180 0x55a37eb9d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:00.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.548+0000 7fe83f67e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a37eb99850 0x55a37eba14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:00.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.548+0000 7fe83f67e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe8341b3090 con 0x55a37eb99850 2024-04-23T14:54:00.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.548+0000 7fe83f67e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe8341b4500 con 0x55a37eb8e860 2024-04-23T14:54:00.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.548+0000 7fe83f67e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe8341b3760 con 0x55a37eb8f180 2024-04-23T14:54:00.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.548+0000 7fe83f67e740 10 client.0 _ll_register_callbacks cb 0x7fe824009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:54:00.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.548+0000 7fe83a800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a37eb8e860 0x55a37eb9d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:00.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.548+0000 7fe83a800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a37eb8e860 0x55a37eb9d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40078/0 (socket says 172.21.15.28:40078) 2024-04-23T14:54:00.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.548+0000 7fe83a800640 1 -- 172.21.15.28:0/269772499 learned_addr learned my addr 172.21.15.28:0/269772499 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:00.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.548+0000 7fe839400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a37eb8f180 0x55a37eb9d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:00.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.548+0000 7fe83a800640 1 --2- 172.21.15.28:0/269772499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a37eb99850 0x55a37eba14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:00.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.549+0000 7fe839400640 1 -- 172.21.15.28:0/269772499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a37eb8e860 msgr2=0x55a37eb9d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:00.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.549+0000 7fe839400640 1 --2- 172.21.15.28:0/269772499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a37eb8e860 0x55a37eb9d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.549+0000 7fe839400640 1 -- 172.21.15.28:0/269772499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a37eb99850 msgr2=0x55a37eba14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:00.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.549+0000 7fe839400640 1 --2- 172.21.15.28:0/269772499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a37eb99850 0x55a37eba14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.549+0000 7fe839400640 1 -- 172.21.15.28:0/269772499 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a37e7c60f0 con 0x55a37eb8f180 2024-04-23T14:54:00.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.549+0000 7fe83a800640 1 --2- 172.21.15.28:0/269772499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a37eb99850 0x55a37eba14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:54:00.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.549+0000 7fe839400640 1 --2- 172.21.15.28:0/269772499 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a37eb8f180 0x55a37eb9d560 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe824000c00 tx=0x7fe82400fe60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20a3db60c8a0702c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:00.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.549+0000 7fe817400640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:54:00.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.549+0000 7fe817400640 1 -- 172.21.15.28:0/269772499 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe82401c630 con 0x55a37eb8f180 2024-04-23T14:54:00.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.549+0000 7fe817400640 1 -- 172.21.15.28:0/269772499 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7fe82401cb00 con 0x55a37eb8f180 2024-04-23T14:54:00.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.549+0000 7fe83f67e740 1 -- 172.21.15.28:0/269772499 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8300046c0 con 0x55a37eb8f180 2024-04-23T14:54:00.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.550+0000 7fe817400640 1 -- 172.21.15.28:0/269772499 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fe82401cd20 con 0x55a37eb8f180 2024-04-23T14:54:00.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.550+0000 7fe83f67e740 10 client.26791 fetch_fsmap learned FSMap version 20 2024-04-23T14:54:00.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.550+0000 7fe83f67e740 1 -- 172.21.15.28:0/269772499 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55a37eab9810 con 0x55a37eb8f180 2024-04-23T14:54:00.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.550+0000 7fe817400640 1 -- 172.21.15.28:0/269772499 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8300046c0 con 0x55a37eb8f180 2024-04-23T14:54:00.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.550+0000 7fe817400640 1 -- 172.21.15.28:0/269772499 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fe824010730 con 0x55a37eb8f180 2024-04-23T14:54:00.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.550+0000 7fe83f67e740 10 client.26791 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:54:00.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.550+0000 7fe83f67e740 10 client.26791 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:54:00.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.550+0000 7fe83f67e740 20 client.26791 populate_metadata read hostname 'smithi028' 2024-04-23T14:54:00.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.550+0000 7fe83f67e740 1 --2- 172.21.15.28:0/269772499 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55a37ec10250 0x55a37ec12670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:00.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.550+0000 7fe83f67e740 4 client.26791 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:54:00.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.550+0000 7fe83f67e740 1 -- 172.21.15.28:0/269772499 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a37eafba80 con 0x55a37ec10250 2024-04-23T14:54:00.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.550+0000 7fe83a800640 1 --2- 172.21.15.28:0/269772499 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55a37ec10250 0x55a37ec12670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:00.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.551+0000 7fe83a800640 1 --2- 172.21.15.28:0/269772499 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55a37ec10250 0x55a37ec12670 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd5198d23acd57db server_cookie=2dbc6bb4a3967a3 in_seq=0 out_seq=0 2024-04-23T14:54:00.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.551+0000 7fe817400640 10 client.26791 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:54:00.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.552+0000 7fe817400640 1 -- 172.21.15.28:0/269772499 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a37eafba80 con 0x55a37ec10250 2024-04-23T14:54:00.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.552+0000 7fe817400640 10 client.26791 handle_command_reply: tid=0 2024-04-23T14:54:00.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.552+0000 7fe83f67e740 1 client.26791 shutdown 2024-04-23T14:54:00.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.552+0000 7fe83f67e740 1 -- 172.21.15.28:0/269772499 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a37eb8f180 msgr2=0x55a37eb9d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:00.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.552+0000 7fe83f67e740 1 --2- 172.21.15.28:0/269772499 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a37eb8f180 0x55a37eb9d560 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe824000c00 tx=0x7fe82400fe60 comp rx=0 tx=0).stop 2024-04-23T14:54:00.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.552+0000 7fe83f67e740 1 -- 172.21.15.28:0/269772499 shutdown_connections 2024-04-23T14:54:00.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.552+0000 7fe83f67e740 1 --2- 172.21.15.28:0/269772499 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55a37ec10250 0x55a37ec12670 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.552+0000 7fe83f67e740 1 --2- 172.21.15.28:0/269772499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a37eb99850 0x55a37eba14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.552+0000 7fe83f67e740 1 --2- 172.21.15.28:0/269772499 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a37eb8f180 0x55a37eb9d560 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.552+0000 7fe817400640 0 client.26791 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:54:00.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.552+0000 7fe83f67e740 1 --2- 172.21.15.28:0/269772499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a37eb8e860 0x55a37eb9d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.552+0000 7fe83f67e740 1 -- 172.21.15.28:0/269772499 >> 172.21.15.28:0/269772499 conn(0x55a37eb8ab80 msgr2=0x55a37eb89390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:00.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.552+0000 7fe83f67e740 1 -- 172.21.15.28:0/269772499 shutdown_connections 2024-04-23T14:54:00.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.552+0000 7fe83f67e740 1 -- 172.21.15.28:0/269772499 wait complete. 2024-04-23T14:54:00.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.552+0000 7fe83f67e740 20 client.26791 trim_cache size 0 max 16384 2024-04-23T14:54:00.556 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:54:00.556 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-23T14:54:00.556 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-23T14:54:00.556 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:54:00.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.553+0000 7fe83c800640 1 -- 172.21.15.28:0/2219582601 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe81c0a3de0 msgr2=0x7fe81c0a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:00.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.553+0000 7fe83c800640 1 --2- 172.21.15.28:0/2219582601 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe81c0a3de0 0x7fe81c0a62a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x55a37eb8fe20 tx=0x7fe8280f4940 comp rx=0 tx=0).stop 2024-04-23T14:54:00.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.553+0000 7fe83c800640 1 -- 172.21.15.28:0/2219582601 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe834108610 msgr2=0x7fe83419ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:00.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.553+0000 7fe83c800640 1 --2- 172.21.15.28:0/2219582601 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe834108610 0x7fe83419ee50 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fe82802a960 tx=0x7fe828003b90 comp rx=0 tx=0).stop 2024-04-23T14:54:00.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.553+0000 7fe83c800640 1 -- 172.21.15.28:0/2219582601 shutdown_connections 2024-04-23T14:54:00.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.553+0000 7fe83c800640 1 --2- 172.21.15.28:0/2219582601 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe81c0a3de0 0x7fe81c0a62a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.553+0000 7fe83c800640 1 --2- 172.21.15.28:0/2219582601 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe83410e660 0x7fe8341a3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.553+0000 7fe83c800640 1 --2- 172.21.15.28:0/2219582601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe834108fd0 0x7fe83419f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.553+0000 7fe83c800640 1 --2- 172.21.15.28:0/2219582601 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe834108610 0x7fe83419ee50 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.553+0000 7fe83c800640 1 -- 172.21.15.28:0/2219582601 >> 172.21.15.28:0/2219582601 conn(0x7fe834076540 msgr2=0x7fe834077ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:00.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.553+0000 7fe83c800640 1 -- 172.21.15.28:0/2219582601 shutdown_connections 2024-04-23T14:54:00.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.553+0000 7fe83c800640 1 -- 172.21.15.28:0/2219582601 wait complete. 2024-04-23T14:54:00.576 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-23T14:54:00.577 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:54:00.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.797+0000 7f7c09800640 1 -- 172.21.15.28:0/2421227221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c0410e680 msgr2=0x7f7c04110a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:00.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.797+0000 7f7c09800640 1 --2- 172.21.15.28:0/2421227221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c0410e680 0x7f7c04110a90 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc009be0 tx=0x7f7bfc02a750 comp rx=0 tx=0).stop 2024-04-23T14:54:00.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.797+0000 7f7c09800640 1 -- 172.21.15.28:0/2421227221 shutdown_connections 2024-04-23T14:54:00.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.797+0000 7f7c09800640 1 --2- 172.21.15.28:0/2421227221 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c04074cf0 0x7f7c0410e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.797+0000 7f7c09800640 1 --2- 172.21.15.28:0/2421227221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c0410e680 0x7f7c04110a90 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.797+0000 7f7c09800640 1 --2- 172.21.15.28:0/2421227221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c040743d0 0x7f7c040747b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.797+0000 7f7c09800640 1 -- 172.21.15.28:0/2421227221 >> 172.21.15.28:0/2421227221 conn(0x7f7c040fd930 msgr2=0x7f7c040ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:00.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.798+0000 7f7c09800640 1 -- 172.21.15.28:0/2421227221 shutdown_connections 2024-04-23T14:54:00.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.798+0000 7f7c09800640 1 -- 172.21.15.28:0/2421227221 wait complete. 2024-04-23T14:54:00.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.799+0000 7f7c09800640 1 Processor -- start 2024-04-23T14:54:00.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.799+0000 7f7c09800640 1 -- start start 2024-04-23T14:54:00.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.799+0000 7f7c09800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c040743d0 0x7f7c041a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:00.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.799+0000 7f7c09800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c04074cf0 0x7f7c041a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:00.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.799+0000 7f7c09800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c0410e680 0x7f7c0419a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:00.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.799+0000 7f7c09800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7c041143a0 con 0x7f7c040743d0 2024-04-23T14:54:00.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.799+0000 7f7c09800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7c04114220 con 0x7f7c0410e680 2024-04-23T14:54:00.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.799+0000 7f7c09800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7c04114520 con 0x7f7c04074cf0 2024-04-23T14:54:00.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.800+0000 7f7c08e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c0410e680 0x7f7c0419a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:00.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.800+0000 7f7c03400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c04074cf0 0x7f7c041a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:00.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.800+0000 7f7c08e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c0410e680 0x7f7c0419a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40090/0 (socket says 172.21.15.28:40090) 2024-04-23T14:54:00.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.800+0000 7f7c03e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c040743d0 0x7f7c041a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:00.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.800+0000 7f7c08e00640 1 -- 172.21.15.28:0/733161160 learned_addr learned my addr 172.21.15.28:0/733161160 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:00.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.800+0000 7f7c08e00640 1 -- 172.21.15.28:0/733161160 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c04074cf0 msgr2=0x7f7c041a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:00.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.800+0000 7f7c08e00640 1 --2- 172.21.15.28:0/733161160 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c04074cf0 0x7f7c041a0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.800+0000 7f7c08e00640 1 -- 172.21.15.28:0/733161160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c040743d0 msgr2=0x7f7c041a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:00.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.800+0000 7f7c08e00640 1 --2- 172.21.15.28:0/733161160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c040743d0 0x7f7c041a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:00.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.800+0000 7f7c08e00640 1 -- 172.21.15.28:0/733161160 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c0419afe0 con 0x7f7c0410e680 2024-04-23T14:54:00.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.800+0000 7f7c08e00640 1 --2- 172.21.15.28:0/733161160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c0410e680 0x7f7c0419a720 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc034040 tx=0x7f7bfc0305b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:00.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.801+0000 7f7c00c00640 1 -- 172.21.15.28:0/733161160 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc041020 con 0x7f7c0410e680 2024-04-23T14:54:00.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.801+0000 7f7c00c00640 1 -- 172.21.15.28:0/733161160 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bfc003e50 con 0x7f7c0410e680 2024-04-23T14:54:00.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.801+0000 7f7c00c00640 1 -- 172.21.15.28:0/733161160 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc036040 con 0x7f7c0410e680 2024-04-23T14:54:00.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.801+0000 7f7c09800640 1 -- 172.21.15.28:0/733161160 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c0419b270 con 0x7f7c0410e680 2024-04-23T14:54:00.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.801+0000 7f7c09800640 1 -- 172.21.15.28:0/733161160 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c041a7590 con 0x7f7c0410e680 2024-04-23T14:54:00.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.802+0000 7f7c09800640 1 -- 172.21.15.28:0/733161160 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c04075a80 con 0x7f7c0410e680 2024-04-23T14:54:00.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.804+0000 7f7c00c00640 1 -- 172.21.15.28:0/733161160 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f7bfc044020 con 0x7f7c0410e680 2024-04-23T14:54:00.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.806+0000 7f7c00c00640 1 --2- 172.21.15.28:0/733161160 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7be80a3de0 0x7f7be80a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:00.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.806+0000 7f7c00c00640 1 -- 172.21.15.28:0/733161160 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7f7bfc0f11a0 con 0x7f7c0410e680 2024-04-23T14:54:00.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.806+0000 7f7c03e00640 1 --2- 172.21.15.28:0/733161160 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7be80a3de0 0x7f7be80a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:00.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.808+0000 7f7c03e00640 1 --2- 172.21.15.28:0/733161160 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7be80a3de0 0x7f7be80a62a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7bf4005fd0 tx=0x7f7bf4009260 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:00.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:00.810+0000 7f7c00c00640 1 -- 172.21.15.28:0/733161160 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7bfc04b020 con 0x7f7c0410e680 2024-04-23T14:54:01.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.076+0000 7f7c09800640 1 -- 172.21.15.28:0/733161160 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c04109800 con 0x7f7c0410e680 2024-04-23T14:54:01.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.077+0000 7f7c00c00640 1 -- 172.21.15.28:0/733161160 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7f7bfc0bd830 con 0x7f7c0410e680 2024-04-23T14:54:01.078 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:54:01.078 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:54:01.078 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:54:01.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.080+0000 7f7c09800640 1 -- 172.21.15.28:0/733161160 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7be80a3de0 msgr2=0x7f7be80a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:01.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.080+0000 7f7c09800640 1 --2- 172.21.15.28:0/733161160 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7be80a3de0 0x7f7be80a62a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7bf4005fd0 tx=0x7f7bf4009260 comp rx=0 tx=0).stop 2024-04-23T14:54:01.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.080+0000 7f7c09800640 1 -- 172.21.15.28:0/733161160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c0410e680 msgr2=0x7f7c0419a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:01.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.080+0000 7f7c09800640 1 --2- 172.21.15.28:0/733161160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c0410e680 0x7f7c0419a720 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc034040 tx=0x7f7bfc0305b0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.080+0000 7f7c09800640 1 -- 172.21.15.28:0/733161160 shutdown_connections 2024-04-23T14:54:01.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.080+0000 7f7c09800640 1 --2- 172.21.15.28:0/733161160 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7be80a3de0 0x7f7be80a62a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.080+0000 7f7c09800640 1 --2- 172.21.15.28:0/733161160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c0410e680 0x7f7c0419a720 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.080+0000 7f7c09800640 1 --2- 172.21.15.28:0/733161160 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c04074cf0 0x7f7c041a0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.080+0000 7f7c09800640 1 --2- 172.21.15.28:0/733161160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c040743d0 0x7f7c041a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.080+0000 7f7c09800640 1 -- 172.21.15.28:0/733161160 >> 172.21.15.28:0/733161160 conn(0x7f7c040fd930 msgr2=0x7f7c040ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:01.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.080+0000 7f7c09800640 1 -- 172.21.15.28:0/733161160 shutdown_connections 2024-04-23T14:54:01.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.080+0000 7f7c09800640 1 -- 172.21.15.28:0/733161160 wait complete. 2024-04-23T14:54:01.098 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:00 smithi028 ceph-mon[27810]: pgmap v308: 97 pgs: 97 active+clean; 519 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 136 op/s 2024-04-23T14:54:01.102 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-23T14:54:01.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:00 smithi038 ceph-mon[32348]: pgmap v308: 97 pgs: 97 active+clean; 519 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 136 op/s 2024-04-23T14:54:01.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:00 smithi105 ceph-mon[32555]: pgmap v308: 97 pgs: 97 active+clean; 519 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 136 op/s 2024-04-23T14:54:01.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.355+0000 7f4500c00640 1 -- 172.21.15.28:0/592741514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44fc103060 msgr2=0x7f44fc103440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:01.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.355+0000 7f4500c00640 1 --2- 172.21.15.28:0/592741514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44fc103060 0x7f44fc103440 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f44ec009900 tx=0x7f44ec02a480 comp rx=0 tx=0).stop 2024-04-23T14:54:01.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.355+0000 7f4500c00640 1 -- 172.21.15.28:0/592741514 shutdown_connections 2024-04-23T14:54:01.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.355+0000 7f4500c00640 1 --2- 172.21.15.28:0/592741514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44fc10c480 0x7f44fc10e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.355+0000 7f4500c00640 1 --2- 172.21.15.28:0/592741514 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f44fc103980 0x7f44fc10be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.355+0000 7f4500c00640 1 --2- 172.21.15.28:0/592741514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44fc103060 0x7f44fc103440 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.355+0000 7f4500c00640 1 -- 172.21.15.28:0/592741514 >> 172.21.15.28:0/592741514 conn(0x7f44fc0fc880 msgr2=0x7f44fc0feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:01.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.355+0000 7f4500c00640 1 -- 172.21.15.28:0/592741514 shutdown_connections 2024-04-23T14:54:01.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.355+0000 7f4500c00640 1 -- 172.21.15.28:0/592741514 wait complete. 2024-04-23T14:54:01.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.356+0000 7f4500c00640 1 Processor -- start 2024-04-23T14:54:01.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.357+0000 7f4500c00640 1 -- start start 2024-04-23T14:54:01.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.357+0000 7f4500c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44fc103060 0x7f44fc1a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:01.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.357+0000 7f4500c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44fc103980 0x7f44fc1a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:01.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.357+0000 7f4500c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f44fc10c480 0x7f44fc19a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:01.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.357+0000 7f4500c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f44fc1122b0 con 0x7f44fc103060 2024-04-23T14:54:01.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.357+0000 7f4500c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f44fc112130 con 0x7f44fc103980 2024-04-23T14:54:01.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.357+0000 7f4500c00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f44fc112430 con 0x7f44fc10c480 2024-04-23T14:54:01.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.357+0000 7f44faa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44fc103980 0x7f44fc1a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:01.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.357+0000 7f44fb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44fc103060 0x7f44fc1a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:01.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.357+0000 7f44faa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44fc103980 0x7f44fc1a0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40114/0 (socket says 172.21.15.28:40114) 2024-04-23T14:54:01.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.357+0000 7f44faa00640 1 -- 172.21.15.28:0/3686549116 learned_addr learned my addr 172.21.15.28:0/3686549116 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:01.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.358+0000 7f44fbe00640 1 --2- 172.21.15.28:0/3686549116 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f44fc10c480 0x7f44fc19a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:01.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.358+0000 7f44faa00640 1 -- 172.21.15.28:0/3686549116 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f44fc10c480 msgr2=0x7f44fc19a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:01.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.358+0000 7f44faa00640 1 --2- 172.21.15.28:0/3686549116 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f44fc10c480 0x7f44fc19a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.358+0000 7f44faa00640 1 -- 172.21.15.28:0/3686549116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44fc103060 msgr2=0x7f44fc1a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:01.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.358+0000 7f44faa00640 1 --2- 172.21.15.28:0/3686549116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44fc103060 0x7f44fc1a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.358+0000 7f44faa00640 1 -- 172.21.15.28:0/3686549116 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44fc19af90 con 0x7f44fc103980 2024-04-23T14:54:01.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.358+0000 7f44faa00640 1 --2- 172.21.15.28:0/3686549116 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44fc103980 0x7f44fc1a0b90 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f44e800e9f0 tx=0x7f44e800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:01.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.358+0000 7f44f3e00640 1 -- 172.21.15.28:0/3686549116 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e800ccc0 con 0x7f44fc103980 2024-04-23T14:54:01.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.359+0000 7f44f3e00640 1 -- 172.21.15.28:0/3686549116 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44e800ce60 con 0x7f44fc103980 2024-04-23T14:54:01.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.359+0000 7f44f3e00640 1 -- 172.21.15.28:0/3686549116 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e8010690 con 0x7f44fc103980 2024-04-23T14:54:01.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.359+0000 7f4500c00640 1 -- 172.21.15.28:0/3686549116 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44fc19b280 con 0x7f44fc103980 2024-04-23T14:54:01.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.359+0000 7f4500c00640 1 -- 172.21.15.28:0/3686549116 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44fc1a74a0 con 0x7f44fc103980 2024-04-23T14:54:01.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.362+0000 7f4504ff7740 1 Processor -- start 2024-04-23T14:54:01.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.362+0000 7f4504ff7740 1 -- start start 2024-04-23T14:54:01.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.363+0000 7f4504ff7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4d1c4bb40 0x55b4d1c4bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:01.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.363+0000 7f4504ff7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d1c4c460 0x55b4d1c565f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:01.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.363+0000 7f4504ff7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b4d1c56b30 0x55b4d1c58f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:01.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.363+0000 7f4504ff7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b4d1bb11d0 con 0x55b4d1c4c460 2024-04-23T14:54:01.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.363+0000 7f4504ff7740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b4d1b57450 con 0x55b4d1c4bb40 2024-04-23T14:54:01.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.363+0000 7f4504ff7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b4d1bad190 con 0x55b4d1c56b30 2024-04-23T14:54:01.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.363+0000 7f44f3e00640 1 -- 172.21.15.28:0/3686549116 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f44e80046c0 con 0x7f44fc103980 2024-04-23T14:54:01.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.363+0000 7f44fbe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4d1c4bb40 0x55b4d1c4bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:01.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.363+0000 7f44fbe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4d1c4bb40 0x55b4d1c4bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40128/0 (socket says 172.21.15.28:40128) 2024-04-23T14:54:01.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.363+0000 7f44fbe00640 1 -- 172.21.15.28:0/3922775316 learned_addr learned my addr 172.21.15.28:0/3922775316 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:01.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.363+0000 7f44faa00640 1 --2- 172.21.15.28:0/3922775316 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b4d1c56b30 0x55b4d1c58f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:01.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.364+0000 7f44fbe00640 1 -- 172.21.15.28:0/3922775316 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b4d1c56b30 msgr2=0x55b4d1c58f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:01.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.364+0000 7f44fbe00640 1 --2- 172.21.15.28:0/3922775316 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b4d1c56b30 0x55b4d1c58f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.364+0000 7f44fbe00640 1 -- 172.21.15.28:0/3922775316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d1c4c460 msgr2=0x55b4d1c565f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:01.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.364+0000 7f44fbe00640 1 --2- 172.21.15.28:0/3922775316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d1c4c460 0x55b4d1c565f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.364+0000 7f44fbe00640 1 -- 172.21.15.28:0/3922775316 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4d1c14310 con 0x55b4d1c4bb40 2024-04-23T14:54:01.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.364+0000 7f44fbe00640 1 --2- 172.21.15.28:0/3922775316 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4d1c4bb40 0x55b4d1c4bf20 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f44f4009f50 tx=0x7f44f40118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a65cab7e028dc442 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:01.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.364+0000 7f44e7e00640 1 -- 172.21.15.28:0/3922775316 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f4004d20 con 0x55b4d1c4bb40 2024-04-23T14:54:01.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.364+0000 7f44e7e00640 1 -- 172.21.15.28:0/3922775316 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44f4004050 con 0x55b4d1c4bb40 2024-04-23T14:54:01.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.364+0000 7f44e7e00640 1 -- 172.21.15.28:0/3922775316 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f4005280 con 0x55b4d1c4bb40 2024-04-23T14:54:01.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.365+0000 7f44f3e00640 1 --2- 172.21.15.28:0/3686549116 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f44e00a3da0 0x7f44e00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:01.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.365+0000 7f44f3e00640 1 -- 172.21.15.28:0/3686549116 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7f44e80d0a20 con 0x7f44fc103980 2024-04-23T14:54:01.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.365+0000 7f44fb400640 1 --2- 172.21.15.28:0/3686549116 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f44e00a3da0 0x7f44e00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:01.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.365+0000 7f4504ff7740 1 -- 172.21.15.28:0/3922775316 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4d1c4bb40 msgr2=0x55b4d1c4bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:01.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.365+0000 7f4504ff7740 1 --2- 172.21.15.28:0/3922775316 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4d1c4bb40 0x55b4d1c4bf20 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f44f4009f50 tx=0x7f44f40118d0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.365+0000 7f4504ff7740 1 -- 172.21.15.28:0/3922775316 shutdown_connections 2024-04-23T14:54:01.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.365+0000 7f4504ff7740 1 --2- 172.21.15.28:0/3922775316 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b4d1c56b30 0x55b4d1c58f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.365+0000 7f4504ff7740 1 --2- 172.21.15.28:0/3922775316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d1c4c460 0x55b4d1c565f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.365+0000 7f4504ff7740 1 --2- 172.21.15.28:0/3922775316 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4d1c4bb40 0x55b4d1c4bf20 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.365+0000 7f4504ff7740 1 -- 172.21.15.28:0/3922775316 >> 172.21.15.28:0/3922775316 conn(0x55b4d1c47e60 msgr2=0x55b4d1c48240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:01.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.366+0000 7f4504ff7740 1 -- 172.21.15.28:0/3922775316 shutdown_connections 2024-04-23T14:54:01.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.366+0000 7f44fb400640 1 --2- 172.21.15.28:0/3686549116 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f44e00a3da0 0x7f44e00a6260 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f44ec000c00 tx=0x7f44ec005bc0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:01.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.366+0000 7f4504ff7740 1 -- 172.21.15.28:0/3922775316 wait complete. 2024-04-23T14:54:01.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.367+0000 7f4504ff7740 1 -- start start 2024-04-23T14:54:01.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.367+0000 7f4504ff7740 1 Processor -- start 2024-04-23T14:54:01.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.367+0000 7f4504ff7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b4d1c4bb40 0x55b4d1c564e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:01.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.367+0000 7f4504ff7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4d1c4c460 0x55b4d1c537d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:01.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.368+0000 7f4504ff7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d1c56b30 0x55b4d1c53d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:01.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.368+0000 7f4504ff7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b4d1c599c0 con 0x55b4d1c56b30 2024-04-23T14:54:01.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.368+0000 7f4504ff7740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b4d1b6a810 con 0x55b4d1c4c460 2024-04-23T14:54:01.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.368+0000 7f4504ff7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b4d1c59b40 con 0x55b4d1c4bb40 2024-04-23T14:54:01.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.368+0000 7f4504ff7740 10 client.0 _ll_register_callbacks cb 0x55b4d1b38140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:54:01.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.368+0000 7f44fb400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b4d1c4bb40 0x55b4d1c564e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:01.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.368+0000 7f44fb400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b4d1c4bb40 0x55b4d1c564e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:38576/0 (socket says 172.21.15.28:38576) 2024-04-23T14:54:01.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.368+0000 7f44fb400640 1 -- 172.21.15.28:0/2648984497 learned_addr learned my addr 172.21.15.28:0/2648984497 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:01.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.368+0000 7f44faa00640 1 --2- 172.21.15.28:0/2648984497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4d1c4c460 0x55b4d1c537d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:01.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.368+0000 7f44fbe00640 1 --2- 172.21.15.28:0/2648984497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d1c56b30 0x55b4d1c53d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:01.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.368+0000 7f44faa00640 1 -- 172.21.15.28:0/2648984497 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b4d1c4bb40 msgr2=0x55b4d1c564e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:01.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.368+0000 7f44faa00640 1 --2- 172.21.15.28:0/2648984497 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b4d1c4bb40 0x55b4d1c564e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.368+0000 7f44faa00640 1 -- 172.21.15.28:0/2648984497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d1c56b30 msgr2=0x55b4d1c53d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:01.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.368+0000 7f44faa00640 1 --2- 172.21.15.28:0/2648984497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d1c56b30 0x55b4d1c53d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.368+0000 7f44faa00640 1 -- 172.21.15.28:0/2648984497 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4d1b7b3f0 con 0x55b4d1c4c460 2024-04-23T14:54:01.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.368+0000 7f44fb400640 1 --2- 172.21.15.28:0/2648984497 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b4d1c4bb40 0x55b4d1c564e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:54:01.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.368+0000 7f44fbe00640 1 --2- 172.21.15.28:0/2648984497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d1c56b30 0x55b4d1c53d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:54:01.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.369+0000 7f44faa00640 1 --2- 172.21.15.28:0/2648984497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4d1c4c460 0x55b4d1c537d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f44e80ce160 tx=0x7f44e809c870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=503abd643fe68572 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:01.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.369+0000 7f44e4c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:54:01.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.369+0000 7f44e4c00640 1 -- 172.21.15.28:0/2648984497 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e80a5390 con 0x55b4d1c4c460 2024-04-23T14:54:01.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.369+0000 7f44e4c00640 1 -- 172.21.15.28:0/2648984497 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7f44e80870e0 con 0x55b4d1c4c460 2024-04-23T14:54:01.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.369+0000 7f4504ff7740 1 -- 172.21.15.28:0/2648984497 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44f4004050 con 0x55b4d1c4c460 2024-04-23T14:54:01.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.369+0000 7f44e4c00640 1 -- 172.21.15.28:0/2648984497 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f44e809d0b0 con 0x55b4d1c4c460 2024-04-23T14:54:01.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.369+0000 7f4504ff7740 10 client.26828 fetch_fsmap learned FSMap version 20 2024-04-23T14:54:01.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.369+0000 7f4504ff7740 1 -- 172.21.15.28:0/2648984497 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55b4d18c33f0 con 0x55b4d1c4c460 2024-04-23T14:54:01.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.370+0000 7f44e4c00640 1 -- 172.21.15.28:0/2648984497 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44f4004050 con 0x55b4d1c4c460 2024-04-23T14:54:01.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.370+0000 7f44e4c00640 1 -- 172.21.15.28:0/2648984497 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f44e809dce0 con 0x55b4d1c4c460 2024-04-23T14:54:01.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.371+0000 7f4504ff7740 10 client.26828 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:54:01.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.371+0000 7f4504ff7740 10 client.26828 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:54:01.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.371+0000 7f4504ff7740 20 client.26828 populate_metadata read hostname 'smithi028' 2024-04-23T14:54:01.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.371+0000 7f4504ff7740 1 --2- 172.21.15.28:0/2648984497 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55b4d1c66d30 0x55b4d1c69150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:01.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.371+0000 7f4504ff7740 4 client.26828 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:54:01.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.371+0000 7f4504ff7740 1 -- 172.21.15.28:0/2648984497 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b4d1bbebf0 con 0x55b4d1c66d30 2024-04-23T14:54:01.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.371+0000 7f44fb400640 1 --2- 172.21.15.28:0/2648984497 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55b4d1c66d30 0x55b4d1c69150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:01.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.372+0000 7f44fb400640 1 --2- 172.21.15.28:0/2648984497 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55b4d1c66d30 0x55b4d1c69150 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c415052e6fbda490 server_cookie=7802de0aa8759387 in_seq=0 out_seq=0 2024-04-23T14:54:01.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.372+0000 7f44e4c00640 10 client.26828 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:54:01.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.372+0000 7f44e4c00640 1 -- 172.21.15.28:0/2648984497 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b4d1bbebf0 con 0x55b4d1c66d30 2024-04-23T14:54:01.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.372+0000 7f44e4c00640 10 client.26828 handle_command_reply: tid=0 2024-04-23T14:54:01.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.372+0000 7f4504ff7740 1 client.26828 shutdown 2024-04-23T14:54:01.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.373+0000 7f4504ff7740 1 -- 172.21.15.28:0/2648984497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4d1c4c460 msgr2=0x55b4d1c537d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:01.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.373+0000 7f4504ff7740 1 --2- 172.21.15.28:0/2648984497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4d1c4c460 0x55b4d1c537d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f44e80ce160 tx=0x7f44e809c870 comp rx=0 tx=0).stop 2024-04-23T14:54:01.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.373+0000 7f4504ff7740 1 -- 172.21.15.28:0/2648984497 shutdown_connections 2024-04-23T14:54:01.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.373+0000 7f4504ff7740 1 --2- 172.21.15.28:0/2648984497 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55b4d1c66d30 0x55b4d1c69150 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.373+0000 7f4504ff7740 1 --2- 172.21.15.28:0/2648984497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4d1c4c460 0x55b4d1c537d0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.373+0000 7f4504ff7740 1 --2- 172.21.15.28:0/2648984497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d1c56b30 0x55b4d1c53d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.373+0000 7f4504ff7740 1 --2- 172.21.15.28:0/2648984497 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b4d1c4bb40 0x55b4d1c564e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.373+0000 7f44e4c00640 0 client.26828 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:54:01.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.373+0000 7f4504ff7740 1 -- 172.21.15.28:0/2648984497 >> 172.21.15.28:0/2648984497 conn(0x55b4d1c47e60 msgr2=0x55b4d1c4db60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:01.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.373+0000 7f4504ff7740 1 -- 172.21.15.28:0/2648984497 shutdown_connections 2024-04-23T14:54:01.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.373+0000 7f4504ff7740 1 -- 172.21.15.28:0/2648984497 wait complete. 2024-04-23T14:54:01.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.373+0000 7f4504ff7740 20 client.26828 trim_cache size 0 max 16384 2024-04-23T14:54:01.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.393+0000 7f4504ff7740 1 Processor -- start 2024-04-23T14:54:01.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.394+0000 7f4504ff7740 1 -- start start 2024-04-23T14:54:01.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.394+0000 7f4504ff7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d1c4bb40 0x55b4d1c628b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:01.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.394+0000 7f4504ff7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b4d1c4c460 0x55b4d1c56010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:01.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.394+0000 7f4504ff7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4d1c56b30 0x55b4d1c6ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:01.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.394+0000 7f4504ff7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f44dc00c7d0 con 0x55b4d1c4bb40 2024-04-23T14:54:01.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.394+0000 7f4504ff7740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f44dc00f030 con 0x55b4d1c56b30 2024-04-23T14:54:01.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.394+0000 7f4504ff7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f44dc00ce20 con 0x55b4d1c4c460 2024-04-23T14:54:01.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.394+0000 7f44fb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d1c4bb40 0x55b4d1c628b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:01.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.394+0000 7f44fb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d1c4bb40 0x55b4d1c628b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50452/0 (socket says 172.21.15.28:50452) 2024-04-23T14:54:01.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.394+0000 7f44fb400640 1 -- 172.21.15.28:0/2052164037 learned_addr learned my addr 172.21.15.28:0/2052164037 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:01.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.394+0000 7f44faa00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b4d1c4c460 0x55b4d1c56010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:01.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.395+0000 7f44fbe00640 1 --2- 172.21.15.28:0/2052164037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4d1c56b30 0x55b4d1c6ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:01.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.395+0000 7f44faa00640 1 -- 172.21.15.28:0/2052164037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4d1c56b30 msgr2=0x55b4d1c6ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:01.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.395+0000 7f44faa00640 1 --2- 172.21.15.28:0/2052164037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4d1c56b30 0x55b4d1c6ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.395+0000 7f44faa00640 1 -- 172.21.15.28:0/2052164037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d1c4bb40 msgr2=0x55b4d1c628b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:01.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.395+0000 7f44faa00640 1 --2- 172.21.15.28:0/2052164037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d1c4bb40 0x55b4d1c628b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.395+0000 7f44faa00640 1 -- 172.21.15.28:0/2052164037 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4d1c540e0 con 0x55b4d1c4c460 2024-04-23T14:54:01.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.395+0000 7f44fb400640 1 --2- 172.21.15.28:0/2052164037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d1c4bb40 0x55b4d1c628b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:54:01.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.395+0000 7f44faa00640 1 --2- 172.21.15.28:0/2052164037 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b4d1c4c460 0x55b4d1c56010 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x55b4d1c54da0 tx=0x7f44e80bd0b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab25665f9c3532d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:01.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.396+0000 7f44e7400640 1 -- 172.21.15.28:0/2052164037 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e80a5670 con 0x55b4d1c4c460 2024-04-23T14:54:01.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.396+0000 7f44e7400640 1 -- 172.21.15.28:0/2052164037 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44e80a0d60 con 0x55b4d1c4c460 2024-04-23T14:54:01.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.396+0000 7f44e7400640 1 -- 172.21.15.28:0/2052164037 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e80870b0 con 0x55b4d1c4c460 2024-04-23T14:54:01.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.396+0000 7f4504ff7740 1 -- 172.21.15.28:0/2052164037 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b4d1c4c460 msgr2=0x55b4d1c56010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:01.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.396+0000 7f4504ff7740 1 --2- 172.21.15.28:0/2052164037 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b4d1c4c460 0x55b4d1c56010 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x55b4d1c54da0 tx=0x7f44e80bd0b0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.397+0000 7f4504ff7740 1 -- 172.21.15.28:0/2052164037 shutdown_connections 2024-04-23T14:54:01.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.397+0000 7f4504ff7740 1 --2- 172.21.15.28:0/2052164037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4d1c56b30 0x55b4d1c6ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.397+0000 7f4504ff7740 1 --2- 172.21.15.28:0/2052164037 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b4d1c4c460 0x55b4d1c56010 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.397+0000 7f4504ff7740 1 --2- 172.21.15.28:0/2052164037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d1c4bb40 0x55b4d1c628b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.397+0000 7f4504ff7740 1 -- 172.21.15.28:0/2052164037 >> 172.21.15.28:0/2052164037 conn(0x55b4d1c47e60 msgr2=0x55b4d1c48240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.397+0000 7f4504ff7740 1 -- 172.21.15.28:0/2052164037 shutdown_connections 2024-04-23T14:54:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.397+0000 7f4504ff7740 1 -- 172.21.15.28:0/2052164037 wait complete. 2024-04-23T14:54:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.397+0000 7f4504ff7740 1 -- start start 2024-04-23T14:54:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.397+0000 7f4504ff7740 1 Processor -- start 2024-04-23T14:54:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.398+0000 7f4504ff7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4d1c4bb40 0x55b4d1c5f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.398+0000 7f4504ff7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b4d1c4c460 0x55b4d1c50a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.398+0000 7f4504ff7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d1c56b30 0x55b4d1c50f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.398+0000 7f4504ff7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f44dc00d4f0 con 0x55b4d1c56b30 2024-04-23T14:54:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.398+0000 7f4504ff7740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f44dc00e960 con 0x55b4d1c4bb40 2024-04-23T14:54:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.398+0000 7f4504ff7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f44dc00dbc0 con 0x55b4d1c4c460 2024-04-23T14:54:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.398+0000 7f4504ff7740 10 client.0 _ll_register_callbacks cb 0x7f44e80d5040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:54:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.398+0000 7f44fb400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4d1c4bb40 0x55b4d1c5f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.398+0000 7f44fb400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4d1c4bb40 0x55b4d1c5f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40166/0 (socket says 172.21.15.28:40166) 2024-04-23T14:54:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.398+0000 7f44fb400640 1 -- 172.21.15.28:0/3688487981 learned_addr learned my addr 172.21.15.28:0/3688487981 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.398+0000 7f44faa00640 1 --2- 172.21.15.28:0/3688487981 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b4d1c4c460 0x55b4d1c50a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.399+0000 7f44fbe00640 1 --2- 172.21.15.28:0/3688487981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d1c56b30 0x55b4d1c50f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.399+0000 7f44fb400640 1 -- 172.21.15.28:0/3688487981 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b4d1c4c460 msgr2=0x55b4d1c50a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.399+0000 7f44fb400640 1 --2- 172.21.15.28:0/3688487981 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b4d1c4c460 0x55b4d1c50a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.399+0000 7f44fb400640 1 -- 172.21.15.28:0/3688487981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d1c56b30 msgr2=0x55b4d1c50f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.399+0000 7f44fb400640 1 --2- 172.21.15.28:0/3688487981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d1c56b30 0x55b4d1c50f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.399+0000 7f44fb400640 1 -- 172.21.15.28:0/3688487981 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4d198d7d0 con 0x55b4d1c4bb40 2024-04-23T14:54:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.399+0000 7f44faa00640 1 --2- 172.21.15.28:0/3688487981 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b4d1c4c460 0x55b4d1c50a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:54:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.399+0000 7f44fbe00640 1 --2- 172.21.15.28:0/3688487981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d1c56b30 0x55b4d1c50f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:54:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.399+0000 7f44fb400640 1 --2- 172.21.15.28:0/3688487981 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4d1c4bb40 0x55b4d1c5f1b0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f44ec009900 tx=0x7f44ec03d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f89f8c6d8307296 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.399+0000 7f44e4c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:54:01.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.399+0000 7f44e4c00640 1 -- 172.21.15.28:0/3688487981 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44ec035040 con 0x55b4d1c4bb40 2024-04-23T14:54:01.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.399+0000 7f4504ff7740 1 -- 172.21.15.28:0/3688487981 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44e80a0d60 con 0x55b4d1c4bb40 2024-04-23T14:54:01.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.399+0000 7f44e4c00640 1 -- 172.21.15.28:0/3688487981 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7f44ec043040 con 0x55b4d1c4bb40 2024-04-23T14:54:01.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.400+0000 7f44e4c00640 1 -- 172.21.15.28:0/3688487981 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f44ec060050 con 0x55b4d1c4bb40 2024-04-23T14:54:01.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.400+0000 7f4504ff7740 10 client.26834 fetch_fsmap learned FSMap version 20 2024-04-23T14:54:01.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.400+0000 7f4504ff7740 1 -- 172.21.15.28:0/3688487981 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55b4d1cd5ae0 con 0x55b4d1c4bb40 2024-04-23T14:54:01.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.400+0000 7f44e4c00640 1 -- 172.21.15.28:0/3688487981 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44e80a0d60 con 0x55b4d1c4bb40 2024-04-23T14:54:01.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.400+0000 7f44e4c00640 1 -- 172.21.15.28:0/3688487981 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f44ec0046c0 con 0x55b4d1c4bb40 2024-04-23T14:54:01.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.400+0000 7f4504ff7740 10 client.26834 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:54:01.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.400+0000 7f4504ff7740 10 client.26834 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:54:01.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.400+0000 7f4504ff7740 20 client.26834 populate_metadata read hostname 'smithi028' 2024-04-23T14:54:01.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.400+0000 7f4504ff7740 1 --2- 172.21.15.28:0/3688487981 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55b4d1cd60c0 0x55b4d1cd8500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:01.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.400+0000 7f4504ff7740 4 client.26834 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-23T14:54:01.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.400+0000 7f4504ff7740 1 -- 172.21.15.28:0/3688487981 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b4d1bb8a60 con 0x55b4d1cd60c0 2024-04-23T14:54:01.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.400+0000 7f44faa00640 1 --2- 172.21.15.28:0/3688487981 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55b4d1cd60c0 0x55b4d1cd8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:01.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.401+0000 7f44faa00640 1 --2- 172.21.15.28:0/3688487981 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55b4d1cd60c0 0x55b4d1cd8500 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4768f718daca466d server_cookie=cdf4281e791e9810 in_seq=0 out_seq=0 2024-04-23T14:54:01.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.401+0000 7f44e4c00640 10 client.26834 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:54:01.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.401+0000 7f44e4c00640 1 -- 172.21.15.28:0/3688487981 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b4d1bb8a60 con 0x55b4d1cd60c0 2024-04-23T14:54:01.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.401+0000 7f44e4c00640 10 client.26834 handle_command_reply: tid=0 2024-04-23T14:54:01.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.401+0000 7f4504ff7740 1 client.26834 shutdown 2024-04-23T14:54:01.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.401+0000 7f4504ff7740 1 -- 172.21.15.28:0/3688487981 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4d1c4bb40 msgr2=0x55b4d1c5f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:01.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.401+0000 7f4504ff7740 1 --2- 172.21.15.28:0/3688487981 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4d1c4bb40 0x55b4d1c5f1b0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f44ec009900 tx=0x7f44ec03d000 comp rx=0 tx=0).stop 2024-04-23T14:54:01.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.402+0000 7f4504ff7740 1 -- 172.21.15.28:0/3688487981 shutdown_connections 2024-04-23T14:54:01.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.402+0000 7f4504ff7740 1 --2- 172.21.15.28:0/3688487981 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55b4d1cd60c0 0x55b4d1cd8500 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.402+0000 7f4504ff7740 1 --2- 172.21.15.28:0/3688487981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4d1c56b30 0x55b4d1c50f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.402+0000 7f4504ff7740 1 --2- 172.21.15.28:0/3688487981 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b4d1c4c460 0x55b4d1c50a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.402+0000 7f44e4c00640 0 client.26834 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:54:01.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.402+0000 7f4504ff7740 1 --2- 172.21.15.28:0/3688487981 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4d1c4bb40 0x55b4d1c5f1b0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.402+0000 7f4504ff7740 1 -- 172.21.15.28:0/3688487981 >> 172.21.15.28:0/3688487981 conn(0x55b4d1c47e60 msgr2=0x55b4d1c52000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:01.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.402+0000 7f4504ff7740 1 -- 172.21.15.28:0/3688487981 shutdown_connections 2024-04-23T14:54:01.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.402+0000 7f4504ff7740 1 -- 172.21.15.28:0/3688487981 wait complete. 2024-04-23T14:54:01.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.402+0000 7f4504ff7740 20 client.26834 trim_cache size 0 max 16384 2024-04-23T14:54:01.407 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-23T14:54:01.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.403+0000 7f4500c00640 1 -- 172.21.15.28:0/3686549116 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f44e00a3da0 msgr2=0x7f44e00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:01.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.403+0000 7f4500c00640 1 --2- 172.21.15.28:0/3686549116 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f44e00a3da0 0x7f44e00a6260 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f44ec000c00 tx=0x7f44ec005bc0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.403+0000 7f4500c00640 1 -- 172.21.15.28:0/3686549116 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44fc103980 msgr2=0x7f44fc1a0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:01.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.403+0000 7f4500c00640 1 --2- 172.21.15.28:0/3686549116 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44fc103980 0x7f44fc1a0b90 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f44e800e9f0 tx=0x7f44e800eeb0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.403+0000 7f4500c00640 1 -- 172.21.15.28:0/3686549116 shutdown_connections 2024-04-23T14:54:01.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.403+0000 7f4500c00640 1 --2- 172.21.15.28:0/3686549116 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f44e00a3da0 0x7f44e00a6260 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.403+0000 7f4500c00640 1 --2- 172.21.15.28:0/3686549116 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f44fc103980 0x7f44fc1a0b90 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.403+0000 7f4500c00640 1 --2- 172.21.15.28:0/3686549116 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f44fc10c480 0x7f44fc19a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.403+0000 7f4500c00640 1 --2- 172.21.15.28:0/3686549116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44fc103060 0x7f44fc1a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:01.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.403+0000 7f4500c00640 1 -- 172.21.15.28:0/3686549116 >> 172.21.15.28:0/3686549116 conn(0x7f44fc0fc880 msgr2=0x7f44fc100be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:01.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.403+0000 7f4500c00640 1 -- 172.21.15.28:0/3686549116 shutdown_connections 2024-04-23T14:54:01.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:01.403+0000 7f4500c00640 1 -- 172.21.15.28:0/3686549116 wait complete. 2024-04-23T14:54:01.524 INFO:tasks.workunit.client.0.smithi028.stdout: 1 850854 8.85 MB/sec execute 377 sec latency 61.684 ms 2024-04-23T14:54:02.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:01 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/733161160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:54:02.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:01 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/733161160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:54:02.187 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:01 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/733161160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:54:02.421 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-23T14:54:02.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:54:02.422 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-23T14:54:02.524 INFO:tasks.workunit.client.0.smithi028.stdout: 1 852288 8.86 MB/sec execute 378 sec latency 62.096 ms 2024-04-23T14:54:02.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.630+0000 7f894fe00640 1 -- 172.21.15.28:0/1737809099 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89501030d0 msgr2=0x7f89501034b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:02.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.630+0000 7f894fe00640 1 --2- 172.21.15.28:0/1737809099 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89501030d0 0x7f89501034b0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f8940009900 tx=0x7f894002a480 comp rx=0 tx=0).stop 2024-04-23T14:54:02.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.631+0000 7f894fe00640 1 -- 172.21.15.28:0/1737809099 shutdown_connections 2024-04-23T14:54:02.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.631+0000 7f894fe00640 1 --2- 172.21.15.28:0/1737809099 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f895010c4f0 0x7f895010e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.631+0000 7f894fe00640 1 --2- 172.21.15.28:0/1737809099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89501039f0 0x7f895010bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.631+0000 7f894fe00640 1 --2- 172.21.15.28:0/1737809099 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89501030d0 0x7f89501034b0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.631+0000 7f894fe00640 1 -- 172.21.15.28:0/1737809099 >> 172.21.15.28:0/1737809099 conn(0x7f89500fc910 msgr2=0x7f89500fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:02.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.631+0000 7f894fe00640 1 -- 172.21.15.28:0/1737809099 shutdown_connections 2024-04-23T14:54:02.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.631+0000 7f894fe00640 1 -- 172.21.15.28:0/1737809099 wait complete. 2024-04-23T14:54:02.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.632+0000 7f894fe00640 1 Processor -- start 2024-04-23T14:54:02.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.632+0000 7f894fe00640 1 -- start start 2024-04-23T14:54:02.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.633+0000 7f894fe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89501030d0 0x7f895019e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.633+0000 7f894fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89501039f0 0x7f895019eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.633+0000 7f894fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f895010c4f0 0x7f8950198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.633+0000 7f894fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8950112300 con 0x7f89501039f0 2024-04-23T14:54:02.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.633+0000 7f894fe00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8950112180 con 0x7f895010c4f0 2024-04-23T14:54:02.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.633+0000 7f894fe00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8950112480 con 0x7f89501030d0 2024-04-23T14:54:02.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.633+0000 7f894e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89501039f0 0x7f895019eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.633+0000 7f894ea00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89501030d0 0x7f895019e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.633+0000 7f894f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f895010c4f0 0x7f8950198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.633+0000 7f894e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89501039f0 0x7f895019eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50480/0 (socket says 172.21.15.28:50480) 2024-04-23T14:54:02.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.633+0000 7f894e000640 1 -- 172.21.15.28:0/1400470193 learned_addr learned my addr 172.21.15.28:0/1400470193 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:02.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.633+0000 7f894f400640 1 -- 172.21.15.28:0/1400470193 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89501030d0 msgr2=0x7f895019e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:02.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.633+0000 7f894f400640 1 --2- 172.21.15.28:0/1400470193 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89501030d0 0x7f895019e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.633+0000 7f894f400640 1 -- 172.21.15.28:0/1400470193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89501039f0 msgr2=0x7f895019eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:02.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.633+0000 7f894f400640 1 --2- 172.21.15.28:0/1400470193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89501039f0 0x7f895019eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.633+0000 7f894f400640 1 -- 172.21.15.28:0/1400470193 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8950198f40 con 0x7f895010c4f0 2024-04-23T14:54:02.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.634+0000 7f894f400640 1 --2- 172.21.15.28:0/1400470193 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f895010c4f0 0x7f8950198680 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f894800ec50 tx=0x7f894800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:02.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.634+0000 7f8947400640 1 -- 172.21.15.28:0/1400470193 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8948003b70 con 0x7f895010c4f0 2024-04-23T14:54:02.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.634+0000 7f8947400640 1 -- 172.21.15.28:0/1400470193 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8948003d10 con 0x7f895010c4f0 2024-04-23T14:54:02.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.634+0000 7f8947400640 1 -- 172.21.15.28:0/1400470193 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8948010890 con 0x7f895010c4f0 2024-04-23T14:54:02.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.634+0000 7f894fe00640 1 -- 172.21.15.28:0/1400470193 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8950199230 con 0x7f895010c4f0 2024-04-23T14:54:02.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.634+0000 7f894fe00640 1 -- 172.21.15.28:0/1400470193 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89501a54f0 con 0x7f895010c4f0 2024-04-23T14:54:02.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.638+0000 7f8947400640 1 -- 172.21.15.28:0/1400470193 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f8948003fc0 con 0x7f895010c4f0 2024-04-23T14:54:02.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.638+0000 7f895839f740 1 Processor -- start 2024-04-23T14:54:02.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.638+0000 7f895839f740 1 -- start start 2024-04-23T14:54:02.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.638+0000 7f895839f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557f9fa2e270 0x557f9fa2e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.638+0000 7f895839f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f9fa2eb90 0x557f9fa38d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.639+0000 7f895839f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f9fa39260 0x557f9fa3b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.639+0000 7f895839f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557f9f993e70 con 0x557f9fa2eb90 2024-04-23T14:54:02.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.639+0000 7f895839f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x557f9f93a380 con 0x557f9fa39260 2024-04-23T14:54:02.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.639+0000 7f895839f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x557f9f98fe30 con 0x557f9fa2e270 2024-04-23T14:54:02.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.639+0000 7f894e000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557f9fa2e270 0x557f9fa2e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.639+0000 7f894e000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557f9fa2e270 0x557f9fa2e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:38624/0 (socket says 172.21.15.28:38624) 2024-04-23T14:54:02.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.639+0000 7f894e000640 1 -- 172.21.15.28:0/3057757110 learned_addr learned my addr 172.21.15.28:0/3057757110 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:02.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.639+0000 7f894e000640 1 --2- 172.21.15.28:0/3057757110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f9fa39260 0x557f9fa3b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.639+0000 7f894ea00640 1 --2- 172.21.15.28:0/3057757110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f9fa2eb90 0x557f9fa38d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.639+0000 7f894e000640 1 -- 172.21.15.28:0/3057757110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f9fa39260 msgr2=0x557f9fa3b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:02.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.639+0000 7f894e000640 1 --2- 172.21.15.28:0/3057757110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f9fa39260 0x557f9fa3b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.639+0000 7f894e000640 1 -- 172.21.15.28:0/3057757110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f9fa2eb90 msgr2=0x557f9fa38d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:02.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.639+0000 7f8947400640 1 --2- 172.21.15.28:0/1400470193 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f89340a3da0 0x7f89340a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.639+0000 7f894e000640 1 --2- 172.21.15.28:0/3057757110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f9fa2eb90 0x557f9fa38d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.639+0000 7f894e000640 1 -- 172.21.15.28:0/3057757110 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f9f76e7e0 con 0x557f9fa2e270 2024-04-23T14:54:02.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.640+0000 7f8947400640 1 -- 172.21.15.28:0/1400470193 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7f89480d0d10 con 0x7f895010c4f0 2024-04-23T14:54:02.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.640+0000 7f894e000640 1 --2- 172.21.15.28:0/3057757110 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557f9fa2e270 0x557f9fa2e650 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f893c0095f0 tx=0x7f893c009540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f57f97d141bc5c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:02.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.640+0000 7f894f400640 1 --2- 172.21.15.28:0/1400470193 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f89340a3da0 0x7f89340a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.640+0000 7f893b400640 1 -- 172.21.15.28:0/3057757110 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f893c011e60 con 0x557f9fa2e270 2024-04-23T14:54:02.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.640+0000 7f893b400640 1 -- 172.21.15.28:0/3057757110 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f893c00fca0 con 0x557f9fa2e270 2024-04-23T14:54:02.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.640+0000 7f893b400640 1 -- 172.21.15.28:0/3057757110 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f893c01fce0 con 0x557f9fa2e270 2024-04-23T14:54:02.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.641+0000 7f895839f740 1 -- 172.21.15.28:0/3057757110 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557f9fa2e270 msgr2=0x557f9fa2e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:02.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.641+0000 7f894f400640 1 --2- 172.21.15.28:0/1400470193 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f89340a3da0 0x7f89340a6260 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f894809c940 tx=0x7f894800ef50 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:02.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.641+0000 7f895839f740 1 --2- 172.21.15.28:0/3057757110 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557f9fa2e270 0x557f9fa2e650 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f893c0095f0 tx=0x7f893c009540 comp rx=0 tx=0).stop 2024-04-23T14:54:02.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.641+0000 7f895839f740 1 -- 172.21.15.28:0/3057757110 shutdown_connections 2024-04-23T14:54:02.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.641+0000 7f895839f740 1 --2- 172.21.15.28:0/3057757110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f9fa39260 0x557f9fa3b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.641+0000 7f895839f740 1 --2- 172.21.15.28:0/3057757110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f9fa2eb90 0x557f9fa38d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.641+0000 7f895839f740 1 --2- 172.21.15.28:0/3057757110 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557f9fa2e270 0x557f9fa2e650 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.641+0000 7f895839f740 1 -- 172.21.15.28:0/3057757110 >> 172.21.15.28:0/3057757110 conn(0x557f9fa2a590 msgr2=0x557f9fa2a970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:02.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.641+0000 7f895839f740 1 -- 172.21.15.28:0/3057757110 shutdown_connections 2024-04-23T14:54:02.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.641+0000 7f895839f740 1 -- 172.21.15.28:0/3057757110 wait complete. 2024-04-23T14:54:02.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.642+0000 7f895839f740 1 -- start start 2024-04-23T14:54:02.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.642+0000 7f895839f740 1 Processor -- start 2024-04-23T14:54:02.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.643+0000 7f895839f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557f9fa2e270 0x557f9fa36210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.643+0000 7f895839f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f9fa2eb90 0x557f9fa36750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.643+0000 7f895839f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f9fa39260 0x557f9fa36c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.643+0000 7f895839f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557f9fa3c050 con 0x557f9fa2eb90 2024-04-23T14:54:02.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.643+0000 7f895839f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x557f9f94d660 con 0x557f9fa39260 2024-04-23T14:54:02.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.643+0000 7f895839f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x557f9fa3c1d0 con 0x557f9fa2e270 2024-04-23T14:54:02.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.643+0000 7f895839f740 10 client.0 _ll_register_callbacks cb 0x557f9f93baa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:54:02.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.643+0000 7f894ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f9fa2eb90 0x557f9fa36750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.643+0000 7f894e000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557f9fa2e270 0x557f9fa36210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.643+0000 7f894ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f9fa2eb90 0x557f9fa36750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50498/0 (socket says 172.21.15.28:50498) 2024-04-23T14:54:02.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.643+0000 7f894ea00640 1 -- 172.21.15.28:0/2589905018 learned_addr learned my addr 172.21.15.28:0/2589905018 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:02.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.643+0000 7f894e000640 1 --2- 172.21.15.28:0/2589905018 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f9fa39260 0x557f9fa36c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.644+0000 7f894ea00640 1 -- 172.21.15.28:0/2589905018 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557f9fa2e270 msgr2=0x557f9fa36210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:02.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.644+0000 7f894ea00640 1 --2- 172.21.15.28:0/2589905018 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557f9fa2e270 0x557f9fa36210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.644+0000 7f894ea00640 1 -- 172.21.15.28:0/2589905018 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f9fa39260 msgr2=0x557f9fa36c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:02.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.644+0000 7f894ea00640 1 --2- 172.21.15.28:0/2589905018 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f9fa39260 0x557f9fa36c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.644+0000 7f894ea00640 1 -- 172.21.15.28:0/2589905018 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f9f7cef70 con 0x557f9fa2eb90 2024-04-23T14:54:02.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.644+0000 7f894e000640 1 --2- 172.21.15.28:0/2589905018 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f9fa39260 0x557f9fa36c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:54:02.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.644+0000 7f894ea00640 1 --2- 172.21.15.28:0/2589905018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f9fa2eb90 0x557f9fa36750 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8940033000 tx=0x7f8940004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6db79768298db4d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:02.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.644+0000 7f892fe00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-23T14:54:02.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.644+0000 7f892fe00640 1 -- 172.21.15.28:0/2589905018 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8940035040 con 0x557f9fa2eb90 2024-04-23T14:54:02.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.645+0000 7f892fe00640 1 -- 172.21.15.28:0/2589905018 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7f8940005230 con 0x557f9fa2eb90 2024-04-23T14:54:02.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.645+0000 7f895839f740 1 -- 172.21.15.28:0/2589905018 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f893c00fca0 con 0x557f9fa2eb90 2024-04-23T14:54:02.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.645+0000 7f892fe00640 1 -- 172.21.15.28:0/2589905018 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f8940005430 con 0x557f9fa2eb90 2024-04-23T14:54:02.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.645+0000 7f895839f740 10 client.16665 fetch_fsmap learned FSMap version 20 2024-04-23T14:54:02.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.645+0000 7f895839f740 1 -- 172.21.15.28:0/2589905018 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x557f9f666170 con 0x557f9fa2eb90 2024-04-23T14:54:02.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.645+0000 7f892fe00640 1 -- 172.21.15.28:0/2589905018 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f893c00fca0 con 0x557f9fa2eb90 2024-04-23T14:54:02.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.645+0000 7f892fe00640 1 -- 172.21.15.28:0/2589905018 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f8940031810 con 0x557f9fa2eb90 2024-04-23T14:54:02.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.646+0000 7f895839f740 10 client.16665 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:54:02.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.646+0000 7f895839f740 10 client.16665 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:54:02.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.646+0000 7f895839f740 20 client.16665 populate_metadata read hostname 'smithi028' 2024-04-23T14:54:02.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.646+0000 7f895839f740 1 --2- 172.21.15.28:0/2589905018 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x557f9fa49600 0x557f9fa4ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.646+0000 7f895839f740 4 client.16665 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:54:02.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.646+0000 7f895839f740 1 -- 172.21.15.28:0/2589905018 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f9f9a1890 con 0x557f9fa49600 2024-04-23T14:54:02.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.646+0000 7f894e000640 1 --2- 172.21.15.28:0/2589905018 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x557f9fa49600 0x557f9fa4ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.646+0000 7f894e000640 1 --2- 172.21.15.28:0/2589905018 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x557f9fa49600 0x557f9fa4ba40 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff1c8dcf5ad6044c server_cookie=7bed075e1d4b442e in_seq=0 out_seq=0 2024-04-23T14:54:02.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.646+0000 7f892fe00640 10 client.16665 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:54:02.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.647+0000 7f892fe00640 1 -- 172.21.15.28:0/2589905018 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557f9f9a1890 con 0x557f9fa49600 2024-04-23T14:54:02.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.647+0000 7f892fe00640 10 client.16665 handle_command_reply: tid=0 2024-04-23T14:54:02.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.647+0000 7f895839f740 1 client.16665 shutdown 2024-04-23T14:54:02.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.647+0000 7f895839f740 1 -- 172.21.15.28:0/2589905018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f9fa2eb90 msgr2=0x557f9fa36750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:02.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.647+0000 7f895839f740 1 --2- 172.21.15.28:0/2589905018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f9fa2eb90 0x557f9fa36750 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8940033000 tx=0x7f8940004270 comp rx=0 tx=0).stop 2024-04-23T14:54:02.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.648+0000 7f895839f740 1 -- 172.21.15.28:0/2589905018 shutdown_connections 2024-04-23T14:54:02.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.648+0000 7f895839f740 1 --2- 172.21.15.28:0/2589905018 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x557f9fa49600 0x557f9fa4ba40 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.648+0000 7f895839f740 1 --2- 172.21.15.28:0/2589905018 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f9fa39260 0x557f9fa36c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.648+0000 7f895839f740 1 --2- 172.21.15.28:0/2589905018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f9fa2eb90 0x557f9fa36750 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.648+0000 7f895839f740 1 --2- 172.21.15.28:0/2589905018 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557f9fa2e270 0x557f9fa36210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.648+0000 7f895839f740 1 -- 172.21.15.28:0/2589905018 >> 172.21.15.28:0/2589905018 conn(0x557f9fa2a590 msgr2=0x557f9fa30390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:02.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.648+0000 7f892fe00640 0 client.16665 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:54:02.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.648+0000 7f895839f740 1 -- 172.21.15.28:0/2589905018 shutdown_connections 2024-04-23T14:54:02.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.648+0000 7f895839f740 1 -- 172.21.15.28:0/2589905018 wait complete. 2024-04-23T14:54:02.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.648+0000 7f895839f740 20 client.16665 trim_cache size 0 max 16384 2024-04-23T14:54:02.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.670+0000 7f895839f740 1 Processor -- start 2024-04-23T14:54:02.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.670+0000 7f895839f740 1 -- start start 2024-04-23T14:54:02.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.670+0000 7f895839f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f9fa2e270 0x557f9fa44f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.670+0000 7f895839f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557f9fa2eb90 0x557f9fa29570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.670+0000 7f895839f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f9fa39260 0x557f9fa51310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.670+0000 7f895839f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f89501a99f0 con 0x557f9fa2e270 2024-04-23T14:54:02.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.670+0000 7f895839f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f89501ac250 con 0x557f9fa39260 2024-04-23T14:54:02.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.670+0000 7f895839f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f89501aa040 con 0x557f9fa2eb90 2024-04-23T14:54:02.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.670+0000 7f894e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f9fa2e270 0x557f9fa44f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.670+0000 7f894e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f9fa2e270 0x557f9fa44f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50506/0 (socket says 172.21.15.28:50506) 2024-04-23T14:54:02.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.670+0000 7f894e000640 1 -- 172.21.15.28:0/1869177801 learned_addr learned my addr 172.21.15.28:0/1869177801 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:02.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.671+0000 7f894e000640 1 --2- 172.21.15.28:0/1869177801 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f9fa39260 0x557f9fa51310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.671+0000 7f894ea00640 1 --2- 172.21.15.28:0/1869177801 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557f9fa2eb90 0x557f9fa29570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.671+0000 7f894ea00640 1 -- 172.21.15.28:0/1869177801 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f9fa39260 msgr2=0x557f9fa51310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:02.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.671+0000 7f894ea00640 1 --2- 172.21.15.28:0/1869177801 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f9fa39260 0x557f9fa51310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.671+0000 7f894ea00640 1 -- 172.21.15.28:0/1869177801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f9fa2e270 msgr2=0x557f9fa44f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:02.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.671+0000 7f894ea00640 1 --2- 172.21.15.28:0/1869177801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f9fa2e270 0x557f9fa44f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.671+0000 7f894ea00640 1 -- 172.21.15.28:0/1869177801 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f9fa2f730 con 0x557f9fa2eb90 2024-04-23T14:54:02.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.671+0000 7f894ea00640 1 --2- 172.21.15.28:0/1869177801 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557f9fa2eb90 0x557f9fa29570 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f8940031dd0 tx=0x7f894002a480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=943b26e81e045aa7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:02.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.672+0000 7f893aa00640 1 -- 172.21.15.28:0/1869177801 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8940035030 con 0x557f9fa2eb90 2024-04-23T14:54:02.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.672+0000 7f893aa00640 1 -- 172.21.15.28:0/1869177801 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8940049070 con 0x557f9fa2eb90 2024-04-23T14:54:02.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.672+0000 7f893aa00640 1 -- 172.21.15.28:0/1869177801 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894005a070 con 0x557f9fa2eb90 2024-04-23T14:54:02.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.672+0000 7f895839f740 1 -- 172.21.15.28:0/1869177801 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557f9fa2eb90 msgr2=0x557f9fa29570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:02.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.672+0000 7f895839f740 1 --2- 172.21.15.28:0/1869177801 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557f9fa2eb90 0x557f9fa29570 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f8940031dd0 tx=0x7f894002a480 comp rx=0 tx=0).stop 2024-04-23T14:54:02.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.673+0000 7f895839f740 1 -- 172.21.15.28:0/1869177801 shutdown_connections 2024-04-23T14:54:02.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.673+0000 7f895839f740 1 --2- 172.21.15.28:0/1869177801 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f9fa39260 0x557f9fa51310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.673+0000 7f895839f740 1 --2- 172.21.15.28:0/1869177801 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557f9fa2eb90 0x557f9fa29570 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.673+0000 7f895839f740 1 --2- 172.21.15.28:0/1869177801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f9fa2e270 0x557f9fa44f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.673+0000 7f895839f740 1 -- 172.21.15.28:0/1869177801 >> 172.21.15.28:0/1869177801 conn(0x557f9fa2a590 msgr2=0x557f9fa28e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:02.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.673+0000 7f895839f740 1 -- 172.21.15.28:0/1869177801 shutdown_connections 2024-04-23T14:54:02.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.673+0000 7f895839f740 1 -- 172.21.15.28:0/1869177801 wait complete. 2024-04-23T14:54:02.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.673+0000 7f895839f740 1 -- start start 2024-04-23T14:54:02.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.673+0000 7f895839f740 1 Processor -- start 2024-04-23T14:54:02.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.674+0000 7f895839f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f9fa2e270 0x557f9fa3f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.674+0000 7f895839f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557f9fa2eb90 0x557f9fa3fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.674+0000 7f895839f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f9fa39260 0x557f9faa8b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.674+0000 7f895839f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f89501aa710 con 0x557f9fa39260 2024-04-23T14:54:02.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.674+0000 7f895839f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f89501abb80 con 0x557f9fa2e270 2024-04-23T14:54:02.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.674+0000 7f895839f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f89501aade0 con 0x557f9fa2eb90 2024-04-23T14:54:02.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.674+0000 7f895839f740 10 client.0 _ll_register_callbacks cb 0x7f8940009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:54:02.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.674+0000 7f894ea00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557f9fa2eb90 0x557f9fa3fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.674+0000 7f894ea00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557f9fa2eb90 0x557f9fa3fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:38642/0 (socket says 172.21.15.28:38642) 2024-04-23T14:54:02.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.674+0000 7f894ea00640 1 -- 172.21.15.28:0/2165426904 learned_addr learned my addr 172.21.15.28:0/2165426904 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:02.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.674+0000 7f894e000640 1 --2- 172.21.15.28:0/2165426904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f9fa2e270 0x557f9fa3f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.674+0000 7f894e000640 1 --2- 172.21.15.28:0/2165426904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f9fa39260 0x557f9faa8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.674+0000 7f894ea00640 1 -- 172.21.15.28:0/2165426904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f9fa2e270 msgr2=0x557f9fa3f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:02.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.674+0000 7f894ea00640 1 --2- 172.21.15.28:0/2165426904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f9fa2e270 0x557f9fa3f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.674+0000 7f894ea00640 1 -- 172.21.15.28:0/2165426904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f9fa39260 msgr2=0x557f9faa8b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:02.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.674+0000 7f894ea00640 1 --2- 172.21.15.28:0/2165426904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f9fa39260 0x557f9faa8b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.674+0000 7f894ea00640 1 -- 172.21.15.28:0/2165426904 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f9faa9150 con 0x557f9fa2eb90 2024-04-23T14:54:02.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.674+0000 7f894e000640 1 --2- 172.21.15.28:0/2165426904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f9fa2e270 0x557f9fa3f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:54:02.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.675+0000 7f894ea00640 1 --2- 172.21.15.28:0/2165426904 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557f9fa2eb90 0x557f9fa3fc10 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8940031be0 tx=0x7f8940031400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b954d39c7ad10c08 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:02.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.675+0000 7f892fe00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:54:02.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.675+0000 7f892fe00640 1 -- 172.21.15.28:0/2165426904 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8940035070 con 0x557f9fa2eb90 2024-04-23T14:54:02.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.675+0000 7f895839f740 1 -- 172.21.15.28:0/2165426904 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8940049070 con 0x557f9fa2eb90 2024-04-23T14:54:02.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.675+0000 7f892fe00640 1 -- 172.21.15.28:0/2165426904 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7f8940003960 con 0x557f9fa2eb90 2024-04-23T14:54:02.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.675+0000 7f892fe00640 1 -- 172.21.15.28:0/2165426904 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f894006c050 con 0x557f9fa2eb90 2024-04-23T14:54:02.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.675+0000 7f895839f740 10 client.26833 fetch_fsmap learned FSMap version 20 2024-04-23T14:54:02.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.675+0000 7f895839f740 1 -- 172.21.15.28:0/2165426904 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x557f9fab0510 con 0x557f9fa2eb90 2024-04-23T14:54:02.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.676+0000 7f892fe00640 1 -- 172.21.15.28:0/2165426904 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8940049070 con 0x557f9fa2eb90 2024-04-23T14:54:02.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.676+0000 7f892fe00640 1 -- 172.21.15.28:0/2165426904 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f8940056cc0 con 0x557f9fa2eb90 2024-04-23T14:54:02.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.676+0000 7f895839f740 10 client.26833 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:54:02.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.676+0000 7f895839f740 10 client.26833 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:54:02.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.676+0000 7f895839f740 20 client.26833 populate_metadata read hostname 'smithi028' 2024-04-23T14:54:02.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.676+0000 7f895839f740 1 --2- 172.21.15.28:0/2165426904 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x557f9fab0ac0 0x557f9fab2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.676+0000 7f895839f740 4 client.26833 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-23T14:54:02.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.676+0000 7f895839f740 1 -- 172.21.15.28:0/2165426904 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557f9f99b700 con 0x557f9fab0ac0 2024-04-23T14:54:02.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.677+0000 7f894e000640 1 --2- 172.21.15.28:0/2165426904 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x557f9fab0ac0 0x557f9fab2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.677+0000 7f894e000640 1 --2- 172.21.15.28:0/2165426904 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x557f9fab0ac0 0x557f9fab2eb0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11f4afa7669dd0fd server_cookie=d0db06f4f3e2c88b in_seq=0 out_seq=0 2024-04-23T14:54:02.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.677+0000 7f892fe00640 10 client.26833 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:54:02.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.680+0000 7f892fe00640 1 -- 172.21.15.28:0/2165426904 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557f9f99b700 con 0x557f9fab0ac0 2024-04-23T14:54:02.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.680+0000 7f892fe00640 10 client.26833 handle_command_reply: tid=0 2024-04-23T14:54:02.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.680+0000 7f895839f740 1 client.26833 shutdown 2024-04-23T14:54:02.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.680+0000 7f895839f740 1 -- 172.21.15.28:0/2165426904 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557f9fa2eb90 msgr2=0x557f9fa3fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:02.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.680+0000 7f895839f740 1 --2- 172.21.15.28:0/2165426904 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557f9fa2eb90 0x557f9fa3fc10 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8940031be0 tx=0x7f8940031400 comp rx=0 tx=0).stop 2024-04-23T14:54:02.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.680+0000 7f895839f740 1 -- 172.21.15.28:0/2165426904 shutdown_connections 2024-04-23T14:54:02.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.680+0000 7f895839f740 1 --2- 172.21.15.28:0/2165426904 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x557f9fab0ac0 0x557f9fab2eb0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.680+0000 7f895839f740 1 --2- 172.21.15.28:0/2165426904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f9fa39260 0x557f9faa8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.680+0000 7f895839f740 1 --2- 172.21.15.28:0/2165426904 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557f9fa2eb90 0x557f9fa3fc10 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.680+0000 7f895839f740 1 --2- 172.21.15.28:0/2165426904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557f9fa2e270 0x557f9fa3f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.680+0000 7f892fe00640 0 client.26833 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:54:02.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.680+0000 7f895839f740 1 -- 172.21.15.28:0/2165426904 >> 172.21.15.28:0/2165426904 conn(0x557f9fa2a590 msgr2=0x557f9fa296f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:02.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.680+0000 7f895839f740 1 -- 172.21.15.28:0/2165426904 shutdown_connections 2024-04-23T14:54:02.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.680+0000 7f895839f740 1 -- 172.21.15.28:0/2165426904 wait complete. 2024-04-23T14:54:02.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.681+0000 7f895839f740 20 client.26833 trim_cache size 0 max 16384 2024-04-23T14:54:02.682 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:54:02.682 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-23T14:54:02.682 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "1a5541d7-1e94-447e-a60e-a6750bad3281", 2024-04-23T14:54:02.682 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-23T14:54:02.682 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:54:02.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.681+0000 7f894fe00640 1 -- 172.21.15.28:0/1400470193 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f89340a3da0 msgr2=0x7f89340a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:02.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.681+0000 7f894fe00640 1 --2- 172.21.15.28:0/1400470193 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f89340a3da0 0x7f89340a6260 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f894809c940 tx=0x7f894800ef50 comp rx=0 tx=0).stop 2024-04-23T14:54:02.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.681+0000 7f894fe00640 1 -- 172.21.15.28:0/1400470193 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f895010c4f0 msgr2=0x7f8950198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:02.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.681+0000 7f894fe00640 1 --2- 172.21.15.28:0/1400470193 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f895010c4f0 0x7f8950198680 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f894800ec50 tx=0x7f894800c6a0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.681+0000 7f894fe00640 1 -- 172.21.15.28:0/1400470193 shutdown_connections 2024-04-23T14:54:02.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.681+0000 7f894fe00640 1 --2- 172.21.15.28:0/1400470193 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f89340a3da0 0x7f89340a6260 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.681+0000 7f894fe00640 1 --2- 172.21.15.28:0/1400470193 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f895010c4f0 0x7f8950198680 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.681+0000 7f894fe00640 1 --2- 172.21.15.28:0/1400470193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89501039f0 0x7f895019eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.681+0000 7f894fe00640 1 --2- 172.21.15.28:0/1400470193 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89501030d0 0x7f895019e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.681+0000 7f894fe00640 1 -- 172.21.15.28:0/1400470193 >> 172.21.15.28:0/1400470193 conn(0x7f89500fc910 msgr2=0x7f89500fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:02.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.682+0000 7f894fe00640 1 -- 172.21.15.28:0/1400470193 shutdown_connections 2024-04-23T14:54:02.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.682+0000 7f894fe00640 1 -- 172.21.15.28:0/1400470193 wait complete. 2024-04-23T14:54:02.700 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:54:02.700 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:54:02.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.897+0000 7f5e3a600640 1 -- 172.21.15.28:0/2639434703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e3410e740 msgr2=0x7f5e34110b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:02.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.897+0000 7f5e3a600640 1 --2- 172.21.15.28:0/2639434703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e3410e740 0x7f5e34110b50 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c009ca0 tx=0x7f5e2c02a950 comp rx=0 tx=0).stop 2024-04-23T14:54:02.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.897+0000 7f5e3a600640 1 -- 172.21.15.28:0/2639434703 shutdown_connections 2024-04-23T14:54:02.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.897+0000 7f5e3a600640 1 --2- 172.21.15.28:0/2639434703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e3410e740 0x7f5e34110b50 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.897+0000 7f5e3a600640 1 --2- 172.21.15.28:0/2639434703 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5e34074cf0 0x7f5e3410e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.897+0000 7f5e3a600640 1 --2- 172.21.15.28:0/2639434703 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e340743d0 0x7f5e340747b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.897+0000 7f5e3a600640 1 -- 172.21.15.28:0/2639434703 >> 172.21.15.28:0/2639434703 conn(0x7f5e340fd9d0 msgr2=0x7f5e340ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:02.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.897+0000 7f5e3a600640 1 -- 172.21.15.28:0/2639434703 shutdown_connections 2024-04-23T14:54:02.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.898+0000 7f5e3a600640 1 -- 172.21.15.28:0/2639434703 wait complete. 2024-04-23T14:54:02.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.899+0000 7f5e3a600640 1 Processor -- start 2024-04-23T14:54:02.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.899+0000 7f5e3a600640 1 -- start start 2024-04-23T14:54:02.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.899+0000 7f5e3a600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e340743d0 0x7f5e341a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.899+0000 7f5e3a600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e34074cf0 0x7f5e341a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.899+0000 7f5e3a600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5e3410e740 0x7f5e3419a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.899+0000 7f5e3a600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5e34114480 con 0x7f5e340743d0 2024-04-23T14:54:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.899+0000 7f5e3a600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5e34114300 con 0x7f5e34074cf0 2024-04-23T14:54:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.899+0000 7f5e3a600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5e34114600 con 0x7f5e3410e740 2024-04-23T14:54:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.899+0000 7f5e2be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e34074cf0 0x7f5e341a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.899+0000 7f5e2be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e34074cf0 0x7f5e341a0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40248/0 (socket says 172.21.15.28:40248) 2024-04-23T14:54:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.900+0000 7f5e2be00640 1 -- 172.21.15.28:0/3997343869 learned_addr learned my addr 172.21.15.28:0/3997343869 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.900+0000 7f5e39200640 1 --2- 172.21.15.28:0/3997343869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e340743d0 0x7f5e341a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.900+0000 7f5e39c00640 1 --2- 172.21.15.28:0/3997343869 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5e3410e740 0x7f5e3419a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.900+0000 7f5e2be00640 1 -- 172.21.15.28:0/3997343869 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5e3410e740 msgr2=0x7f5e3419a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.900+0000 7f5e2be00640 1 --2- 172.21.15.28:0/3997343869 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5e3410e740 0x7f5e3419a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.900+0000 7f5e2be00640 1 -- 172.21.15.28:0/3997343869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e340743d0 msgr2=0x7f5e341a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.900+0000 7f5e2be00640 1 --2- 172.21.15.28:0/3997343869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e340743d0 0x7f5e341a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.900+0000 7f5e2be00640 1 -- 172.21.15.28:0/3997343869 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e3419b080 con 0x7f5e34074cf0 2024-04-23T14:54:02.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.900+0000 7f5e39200640 1 --2- 172.21.15.28:0/3997343869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e340743d0 0x7f5e341a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:54:02.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.900+0000 7f5e2be00640 1 --2- 172.21.15.28:0/3997343869 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e34074cf0 0x7f5e341a0bf0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c00e9f0 tx=0x7f5e1c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:02.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.901+0000 7f5e32000640 1 -- 172.21.15.28:0/3997343869 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c00cd80 con 0x7f5e34074cf0 2024-04-23T14:54:02.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.901+0000 7f5e32000640 1 -- 172.21.15.28:0/3997343869 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e1c005500 con 0x7f5e34074cf0 2024-04-23T14:54:02.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.901+0000 7f5e32000640 1 -- 172.21.15.28:0/3997343869 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c010690 con 0x7f5e34074cf0 2024-04-23T14:54:02.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.901+0000 7f5e3a600640 1 -- 172.21.15.28:0/3997343869 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e3419b370 con 0x7f5e34074cf0 2024-04-23T14:54:02.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.901+0000 7f5e3a600640 1 -- 172.21.15.28:0/3997343869 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e341a7630 con 0x7f5e34074cf0 2024-04-23T14:54:02.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.904+0000 7f5e32000640 1 -- 172.21.15.28:0/3997343869 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f5e1c0046c0 con 0x7f5e34074cf0 2024-04-23T14:54:02.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.904+0000 7f5e3ea7a740 1 Processor -- start 2024-04-23T14:54:02.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.904+0000 7f5e3ea7a740 1 -- start start 2024-04-23T14:54:02.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.905+0000 7f5e3ea7a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557373a02a30 0x557373a02e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.905+0000 7f5e3ea7a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557373a03350 0x557373a0d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.905+0000 7f5e3ea7a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557373a0da20 0x557373a0fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.905+0000 7f5e3ea7a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5573739681f0 con 0x557373a0da20 2024-04-23T14:54:02.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.905+0000 7f5e3ea7a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55737390e350 con 0x557373a03350 2024-04-23T14:54:02.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.905+0000 7f5e3ea7a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5573739641b0 con 0x557373a02a30 2024-04-23T14:54:02.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.905+0000 7f5e39200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557373a03350 0x557373a0d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.905+0000 7f5e39200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557373a03350 0x557373a0d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40252/0 (socket says 172.21.15.28:40252) 2024-04-23T14:54:02.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.905+0000 7f5e39200640 1 -- 172.21.15.28:0/509786583 learned_addr learned my addr 172.21.15.28:0/509786583 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:02.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.905+0000 7f5e39c00640 1 --2- 172.21.15.28:0/509786583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557373a02a30 0x557373a02e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.905+0000 7f5e2be00640 1 --2- 172.21.15.28:0/509786583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557373a0da20 0x557373a0fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.905+0000 7f5e39200640 1 -- 172.21.15.28:0/509786583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557373a02a30 msgr2=0x557373a02e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:02.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.905+0000 7f5e39200640 1 --2- 172.21.15.28:0/509786583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557373a02a30 0x557373a02e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.905+0000 7f5e39200640 1 -- 172.21.15.28:0/509786583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557373a0da20 msgr2=0x557373a0fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:02.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.905+0000 7f5e39200640 1 --2- 172.21.15.28:0/509786583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557373a0da20 0x557373a0fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.905+0000 7f5e39200640 1 -- 172.21.15.28:0/509786583 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557373962f50 con 0x557373a03350 2024-04-23T14:54:02.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.906+0000 7f5e32000640 1 --2- 172.21.15.28:0/3997343869 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5e180a3da0 0x7f5e180a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.906+0000 7f5e32000640 1 -- 172.21.15.28:0/3997343869 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7f5e1c0d0bb0 con 0x7f5e34074cf0 2024-04-23T14:54:02.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.906+0000 7f5e39200640 1 --2- 172.21.15.28:0/509786583 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557373a03350 0x557373a0d4e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f5e24009d60 tx=0x7f5e240117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea4507798acb180f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:02.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.906+0000 7f5e39c00640 1 --2- 172.21.15.28:0/3997343869 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5e180a3da0 0x7f5e180a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.906+0000 7f5e2a000640 1 -- 172.21.15.28:0/509786583 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e24012650 con 0x557373a03350 2024-04-23T14:54:02.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.906+0000 7f5e2a000640 1 -- 172.21.15.28:0/509786583 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e24009e50 con 0x557373a03350 2024-04-23T14:54:02.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.906+0000 7f5e2a000640 1 -- 172.21.15.28:0/509786583 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2400fd10 con 0x557373a03350 2024-04-23T14:54:02.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.907+0000 7f5e3ea7a740 1 -- 172.21.15.28:0/509786583 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557373a03350 msgr2=0x557373a0d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:02.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.907+0000 7f5e39c00640 1 --2- 172.21.15.28:0/3997343869 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5e180a3da0 0x7f5e180a6260 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c005cc0 tx=0x7f5e2c005c50 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:02.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.907+0000 7f5e3ea7a740 1 --2- 172.21.15.28:0/509786583 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557373a03350 0x557373a0d4e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f5e24009d60 tx=0x7f5e240117b0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.907+0000 7f5e3ea7a740 1 -- 172.21.15.28:0/509786583 shutdown_connections 2024-04-23T14:54:02.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.907+0000 7f5e3ea7a740 1 --2- 172.21.15.28:0/509786583 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557373a03350 0x557373a0d4e0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.907+0000 7f5e3ea7a740 1 --2- 172.21.15.28:0/509786583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557373a0da20 0x557373a0fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.907+0000 7f5e3ea7a740 1 --2- 172.21.15.28:0/509786583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557373a02a30 0x557373a02e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.907+0000 7f5e3ea7a740 1 -- 172.21.15.28:0/509786583 >> 172.21.15.28:0/509786583 conn(0x5573739fed50 msgr2=0x5573739ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:02.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.907+0000 7f5e3ea7a740 1 -- 172.21.15.28:0/509786583 shutdown_connections 2024-04-23T14:54:02.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.907+0000 7f5e3ea7a740 1 -- 172.21.15.28:0/509786583 wait complete. 2024-04-23T14:54:02.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.908+0000 7f5e3ea7a740 1 -- start start 2024-04-23T14:54:02.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.908+0000 7f5e3ea7a740 1 Processor -- start 2024-04-23T14:54:02.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.908+0000 7f5e3ea7a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557373a02a30 0x557373a0a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.908+0000 7f5e3ea7a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557373a03350 0x557373a0aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.908+0000 7f5e3ea7a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557373a0da20 0x557373a0b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.908+0000 7f5e3ea7a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557373a10810 con 0x557373a0da20 2024-04-23T14:54:02.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.908+0000 7f5e3ea7a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x557373921690 con 0x557373a02a30 2024-04-23T14:54:02.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.908+0000 7f5e3ea7a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x557373a10990 con 0x557373a03350 2024-04-23T14:54:02.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.908+0000 7f5e3ea7a740 10 client.0 _ll_register_callbacks cb 0x55737395d650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:54:02.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.909+0000 7f5e39200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557373a02a30 0x557373a0a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.909+0000 7f5e39200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557373a02a30 0x557373a0a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40264/0 (socket says 172.21.15.28:40264) 2024-04-23T14:54:02.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.909+0000 7f5e39200640 1 -- 172.21.15.28:0/3265020054 learned_addr learned my addr 172.21.15.28:0/3265020054 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:02.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.909+0000 7f5e2be00640 1 --2- 172.21.15.28:0/3265020054 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557373a03350 0x557373a0aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.909+0000 7f5e39c00640 1 --2- 172.21.15.28:0/3265020054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557373a0da20 0x557373a0b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.909+0000 7f5e39200640 1 -- 172.21.15.28:0/3265020054 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557373a03350 msgr2=0x557373a0aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:02.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.909+0000 7f5e39200640 1 --2- 172.21.15.28:0/3265020054 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557373a03350 0x557373a0aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.909+0000 7f5e39200640 1 -- 172.21.15.28:0/3265020054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557373a0da20 msgr2=0x557373a0b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:02.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.909+0000 7f5e39200640 1 --2- 172.21.15.28:0/3265020054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557373a0da20 0x557373a0b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.909+0000 7f5e39200640 1 -- 172.21.15.28:0/3265020054 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557373920920 con 0x557373a02a30 2024-04-23T14:54:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.909+0000 7f5e39c00640 1 --2- 172.21.15.28:0/3265020054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557373a0da20 0x557373a0b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:54:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.909+0000 7f5e22a00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:54:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.909+0000 7f5e39200640 1 --2- 172.21.15.28:0/3265020054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557373a02a30 0x557373a0a980 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f5e2400f8f0 tx=0x7f5e2400fc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b6579d0d8ad945b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.910+0000 7f5e22a00640 1 -- 172.21.15.28:0/3265020054 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e24028030 con 0x557373a02a30 2024-04-23T14:54:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.910+0000 7f5e22a00640 1 -- 172.21.15.28:0/3265020054 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7f5e24005250 con 0x557373a02a30 2024-04-23T14:54:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.910+0000 7f5e3ea7a740 1 -- 172.21.15.28:0/3265020054 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e24009e50 con 0x557373a02a30 2024-04-23T14:54:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.910+0000 7f5e22a00640 1 -- 172.21.15.28:0/3265020054 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f5e24032050 con 0x557373a02a30 2024-04-23T14:54:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.910+0000 7f5e3ea7a740 10 client.26861 fetch_fsmap learned FSMap version 20 2024-04-23T14:54:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.910+0000 7f5e3ea7a740 1 -- 172.21.15.28:0/3265020054 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5573737447d0 con 0x557373a02a30 2024-04-23T14:54:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.910+0000 7f5e22a00640 1 -- 172.21.15.28:0/3265020054 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e24009e50 con 0x557373a02a30 2024-04-23T14:54:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.910+0000 7f5e22a00640 1 -- 172.21.15.28:0/3265020054 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f5e24012450 con 0x557373a02a30 2024-04-23T14:54:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.911+0000 7f5e3ea7a740 10 client.26861 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:54:02.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.911+0000 7f5e3ea7a740 10 client.26861 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:54:02.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.911+0000 7f5e3ea7a740 20 client.26861 populate_metadata read hostname 'smithi028' 2024-04-23T14:54:02.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.911+0000 7f5e3ea7a740 1 --2- 172.21.15.28:0/3265020054 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x557373a1dd90 0x557373a20180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.911+0000 7f5e3ea7a740 4 client.26861 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:54:02.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.911+0000 7f5e3ea7a740 1 -- 172.21.15.28:0/3265020054 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557373975c10 con 0x557373a1dd90 2024-04-23T14:54:02.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.911+0000 7f5e2be00640 1 --2- 172.21.15.28:0/3265020054 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x557373a1dd90 0x557373a20180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.911+0000 7f5e2be00640 1 --2- 172.21.15.28:0/3265020054 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x557373a1dd90 0x557373a20180 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c751cc22f91e480f server_cookie=4cdeda50c223a237 in_seq=0 out_seq=0 2024-04-23T14:54:02.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.911+0000 7f5e22a00640 10 client.26861 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:54:02.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.914+0000 7f5e22a00640 1 -- 172.21.15.28:0/3265020054 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557373975c10 con 0x557373a1dd90 2024-04-23T14:54:02.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.914+0000 7f5e22a00640 10 client.26861 handle_command_reply: tid=0 2024-04-23T14:54:02.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.914+0000 7f5e3ea7a740 1 client.26861 shutdown 2024-04-23T14:54:02.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.914+0000 7f5e3ea7a740 1 -- 172.21.15.28:0/3265020054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557373a02a30 msgr2=0x557373a0a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:02.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.914+0000 7f5e3ea7a740 1 --2- 172.21.15.28:0/3265020054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557373a02a30 0x557373a0a980 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f5e2400f8f0 tx=0x7f5e2400fc80 comp rx=0 tx=0).stop 2024-04-23T14:54:02.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.914+0000 7f5e3ea7a740 1 -- 172.21.15.28:0/3265020054 shutdown_connections 2024-04-23T14:54:02.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.914+0000 7f5e3ea7a740 1 --2- 172.21.15.28:0/3265020054 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x557373a1dd90 0x557373a20180 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.914+0000 7f5e3ea7a740 1 --2- 172.21.15.28:0/3265020054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557373a0da20 0x557373a0b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.914+0000 7f5e3ea7a740 1 --2- 172.21.15.28:0/3265020054 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557373a03350 0x557373a0aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.914+0000 7f5e3ea7a740 1 --2- 172.21.15.28:0/3265020054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557373a02a30 0x557373a0a980 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.914+0000 7f5e3ea7a740 1 -- 172.21.15.28:0/3265020054 >> 172.21.15.28:0/3265020054 conn(0x5573739fed50 msgr2=0x557373a04b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:02.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.914+0000 7f5e22a00640 0 client.26861 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:54:02.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.914+0000 7f5e3ea7a740 1 -- 172.21.15.28:0/3265020054 shutdown_connections 2024-04-23T14:54:02.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.914+0000 7f5e3ea7a740 1 -- 172.21.15.28:0/3265020054 wait complete. 2024-04-23T14:54:02.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.914+0000 7f5e3ea7a740 20 client.26861 trim_cache size 0 max 16384 2024-04-23T14:54:02.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.934+0000 7f5e3ea7a740 1 Processor -- start 2024-04-23T14:54:02.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.934+0000 7f5e3ea7a740 1 -- start start 2024-04-23T14:54:02.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.934+0000 7f5e3ea7a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557373a02a30 0x557373a19700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.934+0000 7f5e3ea7a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557373a03350 0x557373a05190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.934+0000 7f5e3ea7a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557373a0da20 0x557373a25830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.934+0000 7f5e3ea7a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5e341add00 con 0x557373a03350 2024-04-23T14:54:02.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.934+0000 7f5e3ea7a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5e341b0560 con 0x557373a02a30 2024-04-23T14:54:02.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.934+0000 7f5e3ea7a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5e341ae350 con 0x557373a0da20 2024-04-23T14:54:02.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.934+0000 7f5e39200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557373a02a30 0x557373a19700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.935+0000 7f5e2be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557373a03350 0x557373a05190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.935+0000 7f5e39200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557373a02a30 0x557373a19700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40274/0 (socket says 172.21.15.28:40274) 2024-04-23T14:54:02.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.935+0000 7f5e39200640 1 -- 172.21.15.28:0/2834381139 learned_addr learned my addr 172.21.15.28:0/2834381139 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:02.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.935+0000 7f5e39c00640 1 --2- 172.21.15.28:0/2834381139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557373a0da20 0x557373a25830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.935+0000 7f5e39200640 1 -- 172.21.15.28:0/2834381139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557373a0da20 msgr2=0x557373a25830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:02.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.935+0000 7f5e39200640 1 --2- 172.21.15.28:0/2834381139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557373a0da20 0x557373a25830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.935+0000 7f5e39200640 1 -- 172.21.15.28:0/2834381139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557373a03350 msgr2=0x557373a05190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:02.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.935+0000 7f5e39200640 1 --2- 172.21.15.28:0/2834381139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557373a03350 0x557373a05190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.935+0000 7f5e39200640 1 -- 172.21.15.28:0/2834381139 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55737367a3f0 con 0x557373a02a30 2024-04-23T14:54:02.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.935+0000 7f5e39200640 1 --2- 172.21.15.28:0/2834381139 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557373a02a30 0x557373a19700 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f5e24009af0 tx=0x7f5e240117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78c16dfde33447f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:02.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.935+0000 7f5e29600640 1 -- 172.21.15.28:0/2834381139 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e24028040 con 0x557373a02a30 2024-04-23T14:54:02.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.935+0000 7f5e29600640 1 -- 172.21.15.28:0/2834381139 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e2403b070 con 0x557373a02a30 2024-04-23T14:54:02.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.935+0000 7f5e29600640 1 -- 172.21.15.28:0/2834381139 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e24009280 con 0x557373a02a30 2024-04-23T14:54:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.936+0000 7f5e3ea7a740 1 -- 172.21.15.28:0/2834381139 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557373a02a30 msgr2=0x557373a19700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.936+0000 7f5e3ea7a740 1 --2- 172.21.15.28:0/2834381139 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557373a02a30 0x557373a19700 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f5e24009af0 tx=0x7f5e240117b0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.936+0000 7f5e3ea7a740 1 -- 172.21.15.28:0/2834381139 shutdown_connections 2024-04-23T14:54:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.936+0000 7f5e3ea7a740 1 --2- 172.21.15.28:0/2834381139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557373a0da20 0x557373a25830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.936+0000 7f5e3ea7a740 1 --2- 172.21.15.28:0/2834381139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557373a03350 0x557373a05190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.936+0000 7f5e3ea7a740 1 --2- 172.21.15.28:0/2834381139 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557373a02a30 0x557373a19700 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.936+0000 7f5e3ea7a740 1 -- 172.21.15.28:0/2834381139 >> 172.21.15.28:0/2834381139 conn(0x5573739fed50 msgr2=0x5573739fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.936+0000 7f5e3ea7a740 1 -- 172.21.15.28:0/2834381139 shutdown_connections 2024-04-23T14:54:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.936+0000 7f5e3ea7a740 1 -- 172.21.15.28:0/2834381139 wait complete. 2024-04-23T14:54:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.937+0000 7f5e3ea7a740 1 -- start start 2024-04-23T14:54:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.937+0000 7f5e3ea7a740 1 Processor -- start 2024-04-23T14:54:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.937+0000 7f5e3ea7a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557373a02a30 0x557373a11790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.937+0000 7f5e3ea7a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557373a03350 0x557373a11cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.937+0000 7f5e3ea7a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557373a0da20 0x557373a12210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.937+0000 7f5e3ea7a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5e341aea20 con 0x557373a02a30 2024-04-23T14:54:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.937+0000 7f5e3ea7a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5e341afe90 con 0x557373a03350 2024-04-23T14:54:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.937+0000 7f5e3ea7a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5e341af0f0 con 0x557373a0da20 2024-04-23T14:54:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.937+0000 7f5e3ea7a740 10 client.0 _ll_register_callbacks cb 0x7f5e1c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:54:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.938+0000 7f5e2be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557373a03350 0x557373a11cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.938+0000 7f5e39200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557373a02a30 0x557373a11790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.938+0000 7f5e2be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557373a03350 0x557373a11cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:40276/0 (socket says 172.21.15.28:40276) 2024-04-23T14:54:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.938+0000 7f5e2be00640 1 -- 172.21.15.28:0/2532196225 learned_addr learned my addr 172.21.15.28:0/2532196225 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.938+0000 7f5e39c00640 1 --2- 172.21.15.28:0/2532196225 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557373a0da20 0x557373a12210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.938+0000 7f5e2be00640 1 -- 172.21.15.28:0/2532196225 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557373a0da20 msgr2=0x557373a12210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.938+0000 7f5e2be00640 1 --2- 172.21.15.28:0/2532196225 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557373a0da20 0x557373a12210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.938+0000 7f5e2be00640 1 -- 172.21.15.28:0/2532196225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557373a02a30 msgr2=0x557373a11790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.938+0000 7f5e2be00640 1 --2- 172.21.15.28:0/2532196225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557373a02a30 0x557373a11790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.938+0000 7f5e2be00640 1 -- 172.21.15.28:0/2532196225 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55737363a0f0 con 0x557373a03350 2024-04-23T14:54:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.938+0000 7f5e39200640 1 --2- 172.21.15.28:0/2532196225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557373a02a30 0x557373a11790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:54:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.938+0000 7f5e2be00640 1 --2- 172.21.15.28:0/2532196225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557373a03350 0x557373a11cd0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c0ce350 tx=0x7f5e1c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69a97791c32db92f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.938+0000 7f5e22a00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:54:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.938+0000 7f5e22a00640 1 -- 172.21.15.28:0/2532196225 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c09d240 con 0x557373a03350 2024-04-23T14:54:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.939+0000 7f5e22a00640 1 -- 172.21.15.28:0/2532196225 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8542+0+0 (secure 0 0 0) 0x7f5e1c0ce570 con 0x557373a03350 2024-04-23T14:54:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.939+0000 7f5e3ea7a740 1 -- 172.21.15.28:0/2532196225 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e2403b070 con 0x557373a03350 2024-04-23T14:54:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.939+0000 7f5e22a00640 1 -- 172.21.15.28:0/2532196225 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f5e1c09cb40 con 0x557373a03350 2024-04-23T14:54:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.939+0000 7f5e3ea7a740 10 client.26867 fetch_fsmap learned FSMap version 20 2024-04-23T14:54:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.939+0000 7f5e3ea7a740 1 -- 172.21.15.28:0/2532196225 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55737392d810 con 0x557373a03350 2024-04-23T14:54:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.939+0000 7f5e22a00640 1 -- 172.21.15.28:0/2532196225 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e2403b070 con 0x557373a03350 2024-04-23T14:54:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.939+0000 7f5e22a00640 1 -- 172.21.15.28:0/2532196225 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f5e1c0d7040 con 0x557373a03350 2024-04-23T14:54:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.939+0000 7f5e3ea7a740 10 client.26867 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:54:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.939+0000 7f5e3ea7a740 10 client.26867 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:54:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.939+0000 7f5e3ea7a740 20 client.26867 populate_metadata read hostname 'smithi028' 2024-04-23T14:54:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.939+0000 7f5e3ea7a740 1 --2- 172.21.15.28:0/2532196225 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x557373a84870 0x557373a86c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.939+0000 7f5e3ea7a740 4 client.26867 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:54:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.939+0000 7f5e3ea7a740 1 -- 172.21.15.28:0/2532196225 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55737396fa80 con 0x557373a84870 2024-04-23T14:54:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.939+0000 7f5e39200640 1 --2- 172.21.15.28:0/2532196225 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x557373a84870 0x557373a86c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.940+0000 7f5e39200640 1 --2- 172.21.15.28:0/2532196225 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x557373a84870 0x557373a86c90 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f4ba76ee28ce6e6 server_cookie=5ea57014042e0976 in_seq=0 out_seq=0 2024-04-23T14:54:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.940+0000 7f5e22a00640 10 client.26867 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:54:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.940+0000 7f5e22a00640 1 -- 172.21.15.28:0/2532196225 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55737396fa80 con 0x557373a84870 2024-04-23T14:54:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.940+0000 7f5e22a00640 10 client.26867 handle_command_reply: tid=0 2024-04-23T14:54:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.940+0000 7f5e3ea7a740 1 client.26867 shutdown 2024-04-23T14:54:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.941+0000 7f5e3ea7a740 1 -- 172.21.15.28:0/2532196225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557373a03350 msgr2=0x557373a11cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.941+0000 7f5e3ea7a740 1 --2- 172.21.15.28:0/2532196225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557373a03350 0x557373a11cd0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c0ce350 tx=0x7f5e1c004a40 comp rx=0 tx=0).stop 2024-04-23T14:54:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.941+0000 7f5e3ea7a740 1 -- 172.21.15.28:0/2532196225 shutdown_connections 2024-04-23T14:54:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.941+0000 7f5e3ea7a740 1 --2- 172.21.15.28:0/2532196225 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x557373a84870 0x557373a86c90 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.941+0000 7f5e3ea7a740 1 --2- 172.21.15.28:0/2532196225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557373a03350 0x557373a11cd0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.941+0000 7f5e3ea7a740 1 --2- 172.21.15.28:0/2532196225 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557373a0da20 0x557373a12210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.941+0000 7f5e22a00640 0 client.26867 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:54:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.941+0000 7f5e3ea7a740 1 --2- 172.21.15.28:0/2532196225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557373a02a30 0x557373a11790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.941+0000 7f5e3ea7a740 1 -- 172.21.15.28:0/2532196225 >> 172.21.15.28:0/2532196225 conn(0x5573739fed50 msgr2=0x557373a0a940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.941+0000 7f5e3ea7a740 1 -- 172.21.15.28:0/2532196225 shutdown_connections 2024-04-23T14:54:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.941+0000 7f5e3ea7a740 1 -- 172.21.15.28:0/2532196225 wait complete. 2024-04-23T14:54:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.941+0000 7f5e3ea7a740 20 client.26867 trim_cache size 0 max 16384 2024-04-23T14:54:02.946 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:54:02.946 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-23T14:54:02.946 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-23T14:54:02.946 INFO:teuthology.orchestra.run.smithi028.stdout: "1a5541d7-1e94-447e-a60e-a6750bad3281": { 2024-04-23T14:54:02.946 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-23T14:54:02.946 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "1a5541d7-1e94-447e-a60e-a6750bad3281", 2024-04-23T14:54:02.946 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-23T14:54:02.946 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:54:02.946 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:54:02.946 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:54:02.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.942+0000 7f5e3a600640 1 -- 172.21.15.28:0/3997343869 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5e180a3da0 msgr2=0x7f5e180a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:02.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.942+0000 7f5e3a600640 1 --2- 172.21.15.28:0/3997343869 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5e180a3da0 0x7f5e180a6260 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c005cc0 tx=0x7f5e2c005c50 comp rx=0 tx=0).stop 2024-04-23T14:54:02.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.942+0000 7f5e3a600640 1 -- 172.21.15.28:0/3997343869 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e34074cf0 msgr2=0x7f5e341a0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:02.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.942+0000 7f5e3a600640 1 --2- 172.21.15.28:0/3997343869 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e34074cf0 0x7f5e341a0bf0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c00e9f0 tx=0x7f5e1c00edf0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.942+0000 7f5e3a600640 1 -- 172.21.15.28:0/3997343869 shutdown_connections 2024-04-23T14:54:02.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.942+0000 7f5e3a600640 1 --2- 172.21.15.28:0/3997343869 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5e180a3da0 0x7f5e180a6260 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.942+0000 7f5e3a600640 1 --2- 172.21.15.28:0/3997343869 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e34074cf0 0x7f5e341a0bf0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.942+0000 7f5e3a600640 1 --2- 172.21.15.28:0/3997343869 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5e3410e740 0x7f5e3419a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.942+0000 7f5e3a600640 1 --2- 172.21.15.28:0/3997343869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e340743d0 0x7f5e341a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:02.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.942+0000 7f5e3a600640 1 -- 172.21.15.28:0/3997343869 >> 172.21.15.28:0/3997343869 conn(0x7f5e340fd9d0 msgr2=0x7f5e340fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:02.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.942+0000 7f5e3a600640 1 -- 172.21.15.28:0/3997343869 shutdown_connections 2024-04-23T14:54:02.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:02.942+0000 7f5e3a600640 1 -- 172.21.15.28:0/3997343869 wait complete. 2024-04-23T14:54:02.962 INFO:tasks.cephfs.filesystem:scrub status for tag:1a5541d7-1e94-447e-a60e-a6750bad3281 - {'path': '/', 'tag': '1a5541d7-1e94-447e-a60e-a6750bad3281', 'options': 'recursive,force'} 2024-04-23T14:54:03.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:02 smithi038 ceph-mon[32348]: pgmap v309: 97 pgs: 97 active+clean; 519 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 136 op/s 2024-04-23T14:54:03.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:02 smithi028 ceph-mon[27810]: pgmap v309: 97 pgs: 97 active+clean; 519 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 136 op/s 2024-04-23T14:54:03.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:02 smithi105 ceph-mon[32555]: pgmap v309: 97 pgs: 97 active+clean; 519 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 136 op/s 2024-04-23T14:54:03.524 INFO:tasks.workunit.client.0.smithi028.stdout: 1 853641 8.85 MB/sec execute 379 sec latency 15.673 ms 2024-04-23T14:54:04.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:03 smithi038 ceph-mon[32348]: scrub queued for path: / 2024-04-23T14:54:04.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:03 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:54:04.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:03 smithi038 ceph-mon[32348]: scrub summary: active paths [/] 2024-04-23T14:54:04.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:03 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:54:04.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:03 smithi028 ceph-mon[27810]: scrub queued for path: / 2024-04-23T14:54:04.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:03 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:54:04.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:03 smithi028 ceph-mon[27810]: scrub summary: active paths [/] 2024-04-23T14:54:04.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:03 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:54:04.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:03 smithi105 ceph-mon[32555]: scrub queued for path: / 2024-04-23T14:54:04.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:03 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:54:04.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:03 smithi105 ceph-mon[32555]: scrub summary: active paths [/] 2024-04-23T14:54:04.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:03 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:54:04.524 INFO:tasks.workunit.client.0.smithi028.stdout: 1 855882 8.86 MB/sec execute 380 sec latency 16.042 ms 2024-04-23T14:54:05.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:04 smithi038 ceph-mon[32348]: pgmap v310: 97 pgs: 97 active+clean; 527 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.2 MiB/s wr, 235 op/s 2024-04-23T14:54:05.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:04 smithi028 ceph-mon[27810]: pgmap v310: 97 pgs: 97 active+clean; 527 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.2 MiB/s wr, 235 op/s 2024-04-23T14:54:05.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:04 smithi105 ceph-mon[32555]: pgmap v310: 97 pgs: 97 active+clean; 527 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.2 MiB/s wr, 235 op/s 2024-04-23T14:54:05.524 INFO:tasks.workunit.client.0.smithi028.stdout: 1 857184 8.85 MB/sec execute 381 sec latency 21.914 ms 2024-04-23T14:54:06.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:05 smithi038 ceph-mon[32348]: scrub summary: idle 2024-04-23T14:54:06.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:05 smithi028 ceph-mon[27810]: scrub summary: idle 2024-04-23T14:54:06.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:05 smithi105 ceph-mon[32555]: scrub summary: idle 2024-04-23T14:54:06.524 INFO:tasks.workunit.client.0.smithi028.stdout: 1 859023 8.86 MB/sec execute 382 sec latency 41.774 ms 2024-04-23T14:54:07.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:06 smithi038 ceph-mon[32348]: pgmap v311: 97 pgs: 97 active+clean; 527 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 167 op/s 2024-04-23T14:54:07.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:06 smithi038 ceph-mon[32348]: osdmap e95: 12 total, 12 up, 12 in 2024-04-23T14:54:07.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:06 smithi028 ceph-mon[27810]: pgmap v311: 97 pgs: 97 active+clean; 527 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 167 op/s 2024-04-23T14:54:07.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:06 smithi028 ceph-mon[27810]: osdmap e95: 12 total, 12 up, 12 in 2024-04-23T14:54:07.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:06 smithi105 ceph-mon[32555]: pgmap v311: 97 pgs: 97 active+clean; 527 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 167 op/s 2024-04-23T14:54:07.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:06 smithi105 ceph-mon[32555]: osdmap e95: 12 total, 12 up, 12 in 2024-04-23T14:54:07.524 INFO:tasks.workunit.client.0.smithi028.stdout: 1 860069 8.85 MB/sec execute 383 sec latency 59.106 ms 2024-04-23T14:54:08.524 INFO:tasks.workunit.client.0.smithi028.stdout: 1 861651 8.84 MB/sec execute 384 sec latency 18.961 ms 2024-04-23T14:54:09.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:08 smithi038 ceph-mon[32348]: pgmap v313: 97 pgs: 97 active+clean; 538 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 205 op/s 2024-04-23T14:54:09.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:08 smithi028 ceph-mon[27810]: pgmap v313: 97 pgs: 97 active+clean; 538 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 205 op/s 2024-04-23T14:54:09.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:08 smithi105 ceph-mon[32555]: pgmap v313: 97 pgs: 97 active+clean; 538 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 205 op/s 2024-04-23T14:54:09.525 INFO:tasks.workunit.client.0.smithi028.stdout: 1 863179 8.85 MB/sec execute 385 sec latency 13.518 ms 2024-04-23T14:54:10.525 INFO:tasks.workunit.client.0.smithi028.stdout: 1 864303 8.84 MB/sec execute 386 sec latency 35.121 ms 2024-04-23T14:54:11.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:10 smithi038 ceph-mon[32348]: pgmap v314: 97 pgs: 97 active+clean; 538 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 205 op/s 2024-04-23T14:54:11.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:10 smithi028 ceph-mon[27810]: pgmap v314: 97 pgs: 97 active+clean; 538 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 205 op/s 2024-04-23T14:54:11.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:10 smithi105 ceph-mon[32555]: pgmap v314: 97 pgs: 97 active+clean; 538 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 205 op/s 2024-04-23T14:54:11.525 INFO:tasks.workunit.client.0.smithi028.stdout: 1 865954 8.84 MB/sec execute 387 sec latency 24.435 ms 2024-04-23T14:54:12.525 INFO:tasks.workunit.client.0.smithi028.stdout: 1 866912 8.83 MB/sec execute 388 sec latency 85.526 ms 2024-04-23T14:54:13.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:12 smithi105 ceph-mon[32555]: pgmap v315: 97 pgs: 97 active+clean; 538 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 205 op/s 2024-04-23T14:54:13.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:12 smithi105 ceph-mon[32555]: osdmap e96: 12 total, 12 up, 12 in 2024-04-23T14:54:13.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:12 smithi038 ceph-mon[32348]: pgmap v315: 97 pgs: 97 active+clean; 538 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 205 op/s 2024-04-23T14:54:13.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:12 smithi038 ceph-mon[32348]: osdmap e96: 12 total, 12 up, 12 in 2024-04-23T14:54:13.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:12 smithi028 ceph-mon[27810]: pgmap v315: 97 pgs: 97 active+clean; 538 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.6 MiB/s wr, 205 op/s 2024-04-23T14:54:13.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:12 smithi028 ceph-mon[27810]: osdmap e96: 12 total, 12 up, 12 in 2024-04-23T14:54:13.525 INFO:tasks.workunit.client.0.smithi028.stdout: 1 868384 8.82 MB/sec execute 389 sec latency 22.696 ms 2024-04-23T14:54:14.525 INFO:tasks.workunit.client.0.smithi028.stdout: 1 870131 8.83 MB/sec execute 390 sec latency 21.174 ms 2024-04-23T14:54:15.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:14 smithi105 ceph-mon[32555]: pgmap v317: 97 pgs: 97 active+clean; 544 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.4 MiB/s wr, 206 op/s 2024-04-23T14:54:15.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:14 smithi038 ceph-mon[32348]: pgmap v317: 97 pgs: 97 active+clean; 544 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.4 MiB/s wr, 206 op/s 2024-04-23T14:54:15.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:14 smithi028 ceph-mon[27810]: pgmap v317: 97 pgs: 97 active+clean; 544 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.4 MiB/s wr, 206 op/s 2024-04-23T14:54:15.525 INFO:tasks.workunit.client.0.smithi028.stdout: 1 871455 8.82 MB/sec execute 391 sec latency 28.302 ms 2024-04-23T14:54:16.526 INFO:tasks.workunit.client.0.smithi028.stdout: 1 873617 8.84 MB/sec execute 392 sec latency 17.701 ms 2024-04-23T14:54:17.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:16 smithi105 ceph-mon[32555]: pgmap v318: 97 pgs: 97 active+clean; 544 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.6 MiB/s wr, 169 op/s 2024-04-23T14:54:17.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:16 smithi038 ceph-mon[32348]: pgmap v318: 97 pgs: 97 active+clean; 544 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.6 MiB/s wr, 169 op/s 2024-04-23T14:54:17.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:16 smithi028 ceph-mon[27810]: pgmap v318: 97 pgs: 97 active+clean; 544 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.6 MiB/s wr, 169 op/s 2024-04-23T14:54:17.526 INFO:tasks.workunit.client.0.smithi028.stdout: 1 874819 8.83 MB/sec execute 393 sec latency 49.350 ms 2024-04-23T14:54:18.526 INFO:tasks.workunit.client.0.smithi028.stdout: 1 876397 8.82 MB/sec execute 394 sec latency 24.300 ms 2024-04-23T14:54:19.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:18 smithi105 ceph-mon[32555]: pgmap v319: 97 pgs: 97 active+clean; 523 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.6 MiB/s wr, 158 op/s 2024-04-23T14:54:19.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:18 smithi038 ceph-mon[32348]: pgmap v319: 97 pgs: 97 active+clean; 523 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.6 MiB/s wr, 158 op/s 2024-04-23T14:54:19.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:18 smithi028 ceph-mon[27810]: pgmap v319: 97 pgs: 97 active+clean; 523 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.6 MiB/s wr, 158 op/s 2024-04-23T14:54:19.526 INFO:tasks.workunit.client.0.smithi028.stdout: 1 877660 8.82 MB/sec execute 395 sec latency 18.517 ms 2024-04-23T14:54:20.526 INFO:tasks.workunit.client.0.smithi028.stdout: 1 878753 8.81 MB/sec execute 396 sec latency 28.840 ms 2024-04-23T14:54:21.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:20 smithi105 ceph-mon[32555]: pgmap v320: 97 pgs: 97 active+clean; 523 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.6 MiB/s wr, 158 op/s 2024-04-23T14:54:21.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:20 smithi038 ceph-mon[32348]: pgmap v320: 97 pgs: 97 active+clean; 523 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.6 MiB/s wr, 158 op/s 2024-04-23T14:54:21.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:20 smithi028 ceph-mon[27810]: pgmap v320: 97 pgs: 97 active+clean; 523 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.6 MiB/s wr, 158 op/s 2024-04-23T14:54:21.526 INFO:tasks.workunit.client.0.smithi028.stdout: 1 880560 8.82 MB/sec execute 397 sec latency 18.255 ms 2024-04-23T14:54:22.526 INFO:tasks.workunit.client.0.smithi028.stdout: 1 881278 8.80 MB/sec execute 398 sec latency 78.333 ms 2024-04-23T14:54:23.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:22 smithi105 ceph-mon[32555]: pgmap v321: 97 pgs: 97 active+clean; 523 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.6 MiB/s wr, 158 op/s 2024-04-23T14:54:23.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:22 smithi038 ceph-mon[32348]: pgmap v321: 97 pgs: 97 active+clean; 523 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.6 MiB/s wr, 158 op/s 2024-04-23T14:54:23.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:22 smithi028 ceph-mon[27810]: pgmap v321: 97 pgs: 97 active+clean; 523 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.6 MiB/s wr, 158 op/s 2024-04-23T14:54:23.526 INFO:tasks.workunit.client.0.smithi028.stdout: 1 882906 8.80 MB/sec execute 399 sec latency 41.319 ms 2024-04-23T14:54:24.527 INFO:tasks.workunit.client.0.smithi028.stdout: 1 884393 8.80 MB/sec execute 400 sec latency 12.915 ms 2024-04-23T14:54:25.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:24 smithi105 ceph-mon[32555]: pgmap v322: 97 pgs: 97 active+clean; 526 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.1 MiB/s wr, 190 op/s 2024-04-23T14:54:25.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:24 smithi038 ceph-mon[32348]: pgmap v322: 97 pgs: 97 active+clean; 526 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.1 MiB/s wr, 190 op/s 2024-04-23T14:54:25.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:24 smithi028 ceph-mon[27810]: pgmap v322: 97 pgs: 97 active+clean; 526 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.1 MiB/s wr, 190 op/s 2024-04-23T14:54:25.526 INFO:tasks.workunit.client.0.smithi028.stdout: 1 885548 8.79 MB/sec execute 401 sec latency 35.351 ms 2024-04-23T14:54:26.527 INFO:tasks.workunit.client.0.smithi028.stdout: 1 887571 8.80 MB/sec execute 402 sec latency 17.162 ms 2024-04-23T14:54:27.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:26 smithi105 ceph-mon[32555]: pgmap v323: 97 pgs: 97 active+clean; 526 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 122 op/s 2024-04-23T14:54:27.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:26 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:54:27.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:26 smithi038 ceph-mon[32348]: pgmap v323: 97 pgs: 97 active+clean; 526 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 122 op/s 2024-04-23T14:54:27.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:26 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:54:27.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:26 smithi028 ceph-mon[27810]: pgmap v323: 97 pgs: 97 active+clean; 526 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 122 op/s 2024-04-23T14:54:27.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:26 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:54:27.527 INFO:tasks.workunit.client.0.smithi028.stdout: 1 888535 8.79 MB/sec execute 403 sec latency 46.455 ms 2024-04-23T14:54:28.026 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:27 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:54:28.026 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:27 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:54:28.026 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:27 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:54:28.026 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:27 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:54:28.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:27 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:54:28.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:27 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:54:28.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:27 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:54:28.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:27 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:54:28.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:27 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:54:28.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:27 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:54:28.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:27 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:54:28.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:27 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:54:28.526 INFO:tasks.workunit.client.0.smithi028.stdout: 1 890140 8.78 MB/sec execute 404 sec latency 45.998 ms 2024-04-23T14:54:28.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:28 smithi028 ceph-mon[27810]: pgmap v324: 97 pgs: 97 active+clean; 530 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.8 MiB/s wr, 186 op/s 2024-04-23T14:54:28.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:28 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:54:28.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:28 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:54:28.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:28 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:28.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:28 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:28.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:28 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:28.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:28 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:28.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:28 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:28.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:28 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:28.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:28 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:28.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:28 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:28.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:28 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:54:28.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:28 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:54:29.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:28 smithi038 ceph-mon[32348]: pgmap v324: 97 pgs: 97 active+clean; 530 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.8 MiB/s wr, 186 op/s 2024-04-23T14:54:29.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:28 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:54:29.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:28 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:54:29.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:28 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:29.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:28 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:29.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:28 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:29.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:28 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:29.124 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:28 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:29.124 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:28 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:29.124 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:28 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:29.124 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:28 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:29.124 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:28 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:54:29.124 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:28 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:54:29.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:28 smithi105 ceph-mon[32555]: pgmap v324: 97 pgs: 97 active+clean; 530 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.8 MiB/s wr, 186 op/s 2024-04-23T14:54:29.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:28 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:54:29.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:28 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:54:29.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:28 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:29.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:28 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:29.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:28 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:29.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:28 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:29.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:28 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:29.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:28 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:29.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:28 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:29.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:28 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:29.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:28 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:54:29.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:28 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:54:29.526 INFO:tasks.workunit.client.0.smithi028.stdout: 1 891774 8.79 MB/sec execute 405 sec latency 15.017 ms 2024-04-23T14:54:30.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:29 smithi038 ceph-mon[32348]: Adjusting osd_memory_target on smithi105 to 176.2M 2024-04-23T14:54:30.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:29 smithi038 ceph-mon[32348]: Unable to set osd_memory_target on smithi105 to 184783872: error parsing value: Value '184783872' is below minimum 939524096 2024-04-23T14:54:30.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:29 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:30.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:29 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:30.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:29 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:30.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:29 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:30.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:29 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:30.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:29 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:30.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:29 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:30.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:29 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:30.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:29 smithi028 ceph-mon[27810]: Adjusting osd_memory_target on smithi105 to 176.2M 2024-04-23T14:54:30.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:29 smithi028 ceph-mon[27810]: Unable to set osd_memory_target on smithi105 to 184783872: error parsing value: Value '184783872' is below minimum 939524096 2024-04-23T14:54:30.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:29 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:30.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:29 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:30.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:29 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:30.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:29 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:30.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:29 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:30.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:29 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:30.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:29 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:30.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:29 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:30.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:29 smithi105 ceph-mon[32555]: Adjusting osd_memory_target on smithi105 to 176.2M 2024-04-23T14:54:30.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:29 smithi105 ceph-mon[32555]: Unable to set osd_memory_target on smithi105 to 184783872: error parsing value: Value '184783872' is below minimum 939524096 2024-04-23T14:54:30.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:29 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:30.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:29 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:30.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:29 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:30.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:29 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:30.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:29 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:30.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:29 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:30.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:29 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:30.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:29 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:30.527 INFO:tasks.workunit.client.0.smithi028.stdout: 1 893588 8.79 MB/sec execute 406 sec latency 35.989 ms 2024-04-23T14:54:31.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:30 smithi038 ceph-mon[32348]: Adjusting osd_memory_target on smithi038 to 177.2M 2024-04-23T14:54:31.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:30 smithi038 ceph-mon[32348]: Unable to set osd_memory_target on smithi038 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-23T14:54:31.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:30 smithi038 ceph-mon[32348]: pgmap v325: 97 pgs: 97 active+clean; 530 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 120 op/s 2024-04-23T14:54:31.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:30 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:54:31.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:30 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:54:31.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:30 smithi028 ceph-mon[27810]: Adjusting osd_memory_target on smithi038 to 177.2M 2024-04-23T14:54:31.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:30 smithi028 ceph-mon[27810]: Unable to set osd_memory_target on smithi038 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-23T14:54:31.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:30 smithi028 ceph-mon[27810]: pgmap v325: 97 pgs: 97 active+clean; 530 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 120 op/s 2024-04-23T14:54:31.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:30 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:54:31.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:30 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:54:31.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:30 smithi105 ceph-mon[32555]: Adjusting osd_memory_target on smithi038 to 177.2M 2024-04-23T14:54:31.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:30 smithi105 ceph-mon[32555]: Unable to set osd_memory_target on smithi038 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-23T14:54:31.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:30 smithi105 ceph-mon[32555]: pgmap v325: 97 pgs: 97 active+clean; 530 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 120 op/s 2024-04-23T14:54:31.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:30 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:54:31.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:30 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:54:31.527 INFO:tasks.workunit.client.0.smithi028.stdout: 1 895323 8.79 MB/sec execute 407 sec latency 20.793 ms 2024-04-23T14:54:32.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:31 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:32.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:31 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:32.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:31 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:32.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:31 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:32.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:31 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:32.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:31 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:32.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:31 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:32.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:31 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:32.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:31 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:54:32.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:31 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:54:32.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:31 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:54:32.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:31 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:32.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:31 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:32.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:31 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:32.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:31 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:32.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:31 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:32.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:31 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:32.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:31 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:32.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:31 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:32.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:31 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:54:32.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:31 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:54:32.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:31 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:54:32.187 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:31 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:32.187 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:31 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:32.188 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:31 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:32.188 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:31 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:32.188 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:31 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:32.188 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:31 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:32.188 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:31 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:32.188 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:31 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T14:54:32.188 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:31 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:54:32.188 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:31 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:54:32.188 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:31 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:54:32.527 INFO:tasks.workunit.client.0.smithi028.stdout: 1 896636 8.79 MB/sec execute 408 sec latency 29.052 ms 2024-04-23T14:54:32.963 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:54:32.963 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:54:33.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:32 smithi038 ceph-mon[32348]: pgmap v326: 97 pgs: 97 active+clean; 530 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 120 op/s 2024-04-23T14:54:33.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:32 smithi038 ceph-mon[32348]: Adjusting osd_memory_target on smithi028 to 177.5M 2024-04-23T14:54:33.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:32 smithi038 ceph-mon[32348]: Unable to set osd_memory_target on smithi028 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-23T14:54:33.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:32 smithi028 ceph-mon[27810]: pgmap v326: 97 pgs: 97 active+clean; 530 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 120 op/s 2024-04-23T14:54:33.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:32 smithi028 ceph-mon[27810]: Adjusting osd_memory_target on smithi028 to 177.5M 2024-04-23T14:54:33.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:32 smithi028 ceph-mon[27810]: Unable to set osd_memory_target on smithi028 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-23T14:54:33.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.198+0000 7f9d87000640 1 -- 172.21.15.28:0/3893026017 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d80104320 msgr2=0x7f9d80104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:33.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.198+0000 7f9d87000640 1 --2- 172.21.15.28:0/3893026017 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d80104320 0x7f9d80104700 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f9d74009900 tx=0x7f9d7402a3f0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.198+0000 7f9d87000640 1 -- 172.21.15.28:0/3893026017 shutdown_connections 2024-04-23T14:54:33.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.198+0000 7f9d87000640 1 --2- 172.21.15.28:0/3893026017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d801093f0 0x7f9d80110fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.198+0000 7f9d87000640 1 --2- 172.21.15.28:0/3893026017 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9d80104cd0 0x7f9d80108d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.198+0000 7f9d87000640 1 --2- 172.21.15.28:0/3893026017 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d80104320 0x7f9d80104700 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.198+0000 7f9d87000640 1 -- 172.21.15.28:0/3893026017 >> 172.21.15.28:0/3893026017 conn(0x7f9d800ffb40 msgr2=0x7f9d80101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:33.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.199+0000 7f9d87000640 1 -- 172.21.15.28:0/3893026017 shutdown_connections 2024-04-23T14:54:33.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.199+0000 7f9d87000640 1 -- 172.21.15.28:0/3893026017 wait complete. 2024-04-23T14:54:33.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.200+0000 7f9d87000640 1 Processor -- start 2024-04-23T14:54:33.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.200+0000 7f9d87000640 1 -- start start 2024-04-23T14:54:33.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.200+0000 7f9d87000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9d80104320 0x7f9d801a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:33.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.200+0000 7f9d87000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d801093f0 0x7f9d801a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:33.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.200+0000 7f9d87000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d8019a7d0 0x7f9d8019ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:33.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.200+0000 7f9d87000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d801144d0 con 0x7f9d8019a7d0 2024-04-23T14:54:33.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.200+0000 7f9d87000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9d80114350 con 0x7f9d801093f0 2024-04-23T14:54:33.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.200+0000 7f9d87000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9d80114650 con 0x7f9d80104320 2024-04-23T14:54:33.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.201+0000 7f9d85200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d801093f0 0x7f9d801a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:33.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.201+0000 7f9d85200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d801093f0 0x7f9d801a0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48618/0 (socket says 172.21.15.28:48618) 2024-04-23T14:54:33.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.201+0000 7f9d85200640 1 -- 172.21.15.28:0/3339019394 learned_addr learned my addr 172.21.15.28:0/3339019394 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:33.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.201+0000 7f9d85c00640 1 --2- 172.21.15.28:0/3339019394 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9d80104320 0x7f9d801a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:33.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.201+0000 7f9d86600640 1 --2- 172.21.15.28:0/3339019394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d8019a7d0 0x7f9d8019ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:33.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.201+0000 7f9d85200640 1 -- 172.21.15.28:0/3339019394 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9d80104320 msgr2=0x7f9d801a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:33.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.201+0000 7f9d85200640 1 --2- 172.21.15.28:0/3339019394 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9d80104320 0x7f9d801a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.201+0000 7f9d85200640 1 -- 172.21.15.28:0/3339019394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d8019a7d0 msgr2=0x7f9d8019ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:33.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.201+0000 7f9d85200640 1 --2- 172.21.15.28:0/3339019394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d8019a7d0 0x7f9d8019ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.201+0000 7f9d85200640 1 -- 172.21.15.28:0/3339019394 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d8019b360 con 0x7f9d801093f0 2024-04-23T14:54:33.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.201+0000 7f9d86600640 1 --2- 172.21.15.28:0/3339019394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d8019a7d0 0x7f9d8019ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:54:33.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.201+0000 7f9d85200640 1 --2- 172.21.15.28:0/3339019394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d801093f0 0x7f9d801a0ba0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c00e9a0 tx=0x7f9d6c00ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:33.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.202+0000 7f9d7e000640 1 -- 172.21.15.28:0/3339019394 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c00cd80 con 0x7f9d801093f0 2024-04-23T14:54:33.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.202+0000 7f9d7e000640 1 -- 172.21.15.28:0/3339019394 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d6c0035f0 con 0x7f9d801093f0 2024-04-23T14:54:33.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.202+0000 7f9d7e000640 1 -- 172.21.15.28:0/3339019394 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c010670 con 0x7f9d801093f0 2024-04-23T14:54:33.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.202+0000 7f9d87000640 1 -- 172.21.15.28:0/3339019394 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d8019b650 con 0x7f9d801093f0 2024-04-23T14:54:33.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.202+0000 7f9d87000640 1 -- 172.21.15.28:0/3339019394 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d801a7880 con 0x7f9d801093f0 2024-04-23T14:54:33.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.205+0000 7f9d8b5db740 1 Processor -- start 2024-04-23T14:54:33.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.206+0000 7f9d8b5db740 1 -- start start 2024-04-23T14:54:33.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.206+0000 7f9d7e000640 1 -- 172.21.15.28:0/3339019394 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f9d6c0037a0 con 0x7f9d801093f0 2024-04-23T14:54:33.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.206+0000 7f9d8b5db740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2bebd6a30 0x55c2bebd6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:33.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.206+0000 7f9d8b5db740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c2bebd7350 0x55c2bebe14e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:33.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.206+0000 7f9d8b5db740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2bebe1a20 0x55c2bebe3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:33.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.206+0000 7f9d8b5db740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c2beb3c1f0 con 0x55c2bebe1a20 2024-04-23T14:54:33.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.206+0000 7f9d8b5db740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c2beae2350 con 0x55c2bebd6a30 2024-04-23T14:54:33.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.206+0000 7f9d8b5db740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c2beb381b0 con 0x55c2bebd7350 2024-04-23T14:54:33.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.206+0000 7f9d86600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2bebd6a30 0x55c2bebd6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:33.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.206+0000 7f9d86600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2bebd6a30 0x55c2bebd6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48630/0 (socket says 172.21.15.28:48630) 2024-04-23T14:54:33.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.206+0000 7f9d86600640 1 -- 172.21.15.28:0/556054336 learned_addr learned my addr 172.21.15.28:0/556054336 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:33.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.206+0000 7f9d85200640 1 --2- 172.21.15.28:0/556054336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2bebe1a20 0x55c2bebe3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:33.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.206+0000 7f9d85c00640 1 --2- 172.21.15.28:0/556054336 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c2bebd7350 0x55c2bebe14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.207+0000 7f9d86600640 1 -- 172.21.15.28:0/556054336 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c2bebd7350 msgr2=0x55c2bebe14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.207+0000 7f9d86600640 1 --2- 172.21.15.28:0/556054336 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c2bebd7350 0x55c2bebe14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.207+0000 7f9d86600640 1 -- 172.21.15.28:0/556054336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2bebe1a20 msgr2=0x55c2bebe3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.207+0000 7f9d86600640 1 --2- 172.21.15.28:0/556054336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2bebe1a20 0x55c2bebe3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.207+0000 7f9d86600640 1 -- 172.21.15.28:0/556054336 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2beb36f50 con 0x55c2bebd6a30 2024-04-23T14:54:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.207+0000 7f9d86600640 1 --2- 172.21.15.28:0/556054336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2bebd6a30 0x55c2bebd6e10 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f9d78009fd0 tx=0x7f9d78010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc5f39efee3001e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.207+0000 7f9d72a00640 1 -- 172.21.15.28:0/556054336 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d78011700 con 0x55c2bebd6a30 2024-04-23T14:54:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.208+0000 7f9d72a00640 1 -- 172.21.15.28:0/556054336 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d78011d80 con 0x55c2bebd6a30 2024-04-23T14:54:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.208+0000 7f9d72a00640 1 -- 172.21.15.28:0/556054336 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d7800e3c0 con 0x55c2bebd6a30 2024-04-23T14:54:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.208+0000 7f9d7e000640 1 --2- 172.21.15.28:0/3339019394 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f9d680a3da0 0x7f9d680a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.208+0000 7f9d7e000640 1 -- 172.21.15.28:0/3339019394 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8542+0+0 (secure 0 0 0) 0x7f9d6c0d0b00 con 0x7f9d801093f0 2024-04-23T14:54:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.208+0000 7f9d85c00640 1 --2- 172.21.15.28:0/3339019394 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f9d680a3da0 0x7f9d680a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.208+0000 7f9d8b5db740 1 -- 172.21.15.28:0/556054336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2bebd6a30 msgr2=0x55c2bebd6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.208+0000 7f9d8b5db740 1 --2- 172.21.15.28:0/556054336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2bebd6a30 0x55c2bebd6e10 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f9d78009fd0 tx=0x7f9d78010860 comp rx=0 tx=0).stop 2024-04-23T14:54:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.208+0000 7f9d8b5db740 1 -- 172.21.15.28:0/556054336 shutdown_connections 2024-04-23T14:54:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.208+0000 7f9d8b5db740 1 --2- 172.21.15.28:0/556054336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2bebe1a20 0x55c2bebe3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.208+0000 7f9d8b5db740 1 --2- 172.21.15.28:0/556054336 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c2bebd7350 0x55c2bebe14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.208+0000 7f9d8b5db740 1 --2- 172.21.15.28:0/556054336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2bebd6a30 0x55c2bebd6e10 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.208+0000 7f9d8b5db740 1 -- 172.21.15.28:0/556054336 >> 172.21.15.28:0/556054336 conn(0x55c2bebd2d50 msgr2=0x55c2bebd3130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.209+0000 7f9d8b5db740 1 -- 172.21.15.28:0/556054336 shutdown_connections 2024-04-23T14:54:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.209+0000 7f9d8b5db740 1 -- 172.21.15.28:0/556054336 wait complete. 2024-04-23T14:54:33.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.209+0000 7f9d8b5db740 1 -- start start 2024-04-23T14:54:33.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.209+0000 7f9d85c00640 1 --2- 172.21.15.28:0/3339019394 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f9d680a3da0 0x7f9d680a6260 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9d74000c00 tx=0x7f9d74005b50 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:33.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.209+0000 7f9d8b5db740 1 Processor -- start 2024-04-23T14:54:33.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.210+0000 7f9d8b5db740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c2bebd6a30 0x55c2bebdea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.210+0000 7f9d8b5db740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2bebd7350 0x55c2bebdef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.210+0000 7f9d8b5db740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2bebe1a20 0x55c2bebdf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.210+0000 7f9d8b5db740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c2bebe4810 con 0x55c2bebd7350 2024-04-23T14:54:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.210+0000 7f9d8b5db740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c2beaf5690 con 0x55c2bebe1a20 2024-04-23T14:54:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.210+0000 7f9d8b5db740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c2bebe4990 con 0x55c2bebd6a30 2024-04-23T14:54:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.210+0000 7f9d8b5db740 10 client.0 _ll_register_callbacks cb 0x55c2beb31650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:54:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.210+0000 7f9d85c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c2bebd6a30 0x55c2bebdea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.210+0000 7f9d85c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c2bebd6a30 0x55c2bebdea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:56332/0 (socket says 172.21.15.28:56332) 2024-04-23T14:54:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.210+0000 7f9d85c00640 1 -- 172.21.15.28:0/2577011781 learned_addr learned my addr 172.21.15.28:0/2577011781 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.210+0000 7f9d85200640 1 --2- 172.21.15.28:0/2577011781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2bebd7350 0x55c2bebdef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.210+0000 7f9d86600640 1 --2- 172.21.15.28:0/2577011781 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2bebe1a20 0x55c2bebdf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.211+0000 7f9d85c00640 1 -- 172.21.15.28:0/2577011781 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2bebe1a20 msgr2=0x55c2bebdf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.211+0000 7f9d85c00640 1 --2- 172.21.15.28:0/2577011781 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2bebe1a20 0x55c2bebdf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.211+0000 7f9d85c00640 1 -- 172.21.15.28:0/2577011781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2bebd7350 msgr2=0x55c2bebdef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:33.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.211+0000 7f9d85c00640 1 --2- 172.21.15.28:0/2577011781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2bebd7350 0x55c2bebdef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.211+0000 7f9d85c00640 1 -- 172.21.15.28:0/2577011781 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2beaf4920 con 0x55c2bebd6a30 2024-04-23T14:54:33.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.211+0000 7f9d86600640 1 --2- 172.21.15.28:0/2577011781 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2bebe1a20 0x55c2bebdf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:54:33.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.211+0000 7f9d85c00640 1 --2- 172.21.15.28:0/2577011781 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c2bebd6a30 0x55c2bebdea10 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9d74009900 tx=0x7f9d74009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fd80de2239001d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:33.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.211+0000 7f9d67400640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:54:33.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.211+0000 7f9d67400640 1 -- 172.21.15.28:0/2577011781 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d7404e070 con 0x55c2bebd6a30 2024-04-23T14:54:33.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.212+0000 7f9d67400640 1 -- 172.21.15.28:0/2577011781 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8542+0+0 (secure 0 0 0) 0x7f9d74004c50 con 0x55c2bebd6a30 2024-04-23T14:54:33.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.212+0000 7f9d8b5db740 1 -- 172.21.15.28:0/2577011781 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d78011d80 con 0x55c2bebd6a30 2024-04-23T14:54:33.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.212+0000 7f9d67400640 1 -- 172.21.15.28:0/2577011781 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f9d7405a050 con 0x55c2bebd6a30 2024-04-23T14:54:33.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.212+0000 7f9d8b5db740 10 client.26863 fetch_fsmap learned FSMap version 20 2024-04-23T14:54:33.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.212+0000 7f9d8b5db740 1 -- 172.21.15.28:0/2577011781 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55c2be9187d0 con 0x55c2bebd6a30 2024-04-23T14:54:33.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.212+0000 7f9d67400640 1 -- 172.21.15.28:0/2577011781 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d78011d80 con 0x55c2bebd6a30 2024-04-23T14:54:33.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.212+0000 7f9d67400640 1 -- 172.21.15.28:0/2577011781 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f9d74003850 con 0x55c2bebd6a30 2024-04-23T14:54:33.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.212+0000 7f9d8b5db740 10 client.26863 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:54:33.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.212+0000 7f9d8b5db740 10 client.26863 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:54:33.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.212+0000 7f9d8b5db740 20 client.26863 populate_metadata read hostname 'smithi028' 2024-04-23T14:54:33.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.213+0000 7f9d8b5db740 1 --2- 172.21.15.28:0/2577011781 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c2bebf1da0 0x55c2bebf4230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:33.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.213+0000 7f9d8b5db740 4 client.26863 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:54:33.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.213+0000 7f9d8b5db740 1 -- 172.21.15.28:0/2577011781 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c2beb49c10 con 0x55c2bebf1da0 2024-04-23T14:54:33.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.213+0000 7f9d85200640 1 --2- 172.21.15.28:0/2577011781 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c2bebf1da0 0x55c2bebf4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:33.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.213+0000 7f9d85200640 1 --2- 172.21.15.28:0/2577011781 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c2bebf1da0 0x55c2bebf4230 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14c24afcec897b14 server_cookie=d1c54bb718c9ef8f in_seq=0 out_seq=0 2024-04-23T14:54:33.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.213+0000 7f9d67400640 10 client.26863 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:54:33.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.215+0000 7f9d67400640 1 -- 172.21.15.28:0/2577011781 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c2beb49c10 con 0x55c2bebf1da0 2024-04-23T14:54:33.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.215+0000 7f9d67400640 10 client.26863 handle_command_reply: tid=0 2024-04-23T14:54:33.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.215+0000 7f9d8b5db740 1 client.26863 shutdown 2024-04-23T14:54:33.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.215+0000 7f9d8b5db740 1 -- 172.21.15.28:0/2577011781 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c2bebd6a30 msgr2=0x55c2bebdea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:33.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.215+0000 7f9d8b5db740 1 --2- 172.21.15.28:0/2577011781 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c2bebd6a30 0x55c2bebdea10 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9d74009900 tx=0x7f9d74009260 comp rx=0 tx=0).stop 2024-04-23T14:54:33.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.215+0000 7f9d8b5db740 1 -- 172.21.15.28:0/2577011781 shutdown_connections 2024-04-23T14:54:33.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.215+0000 7f9d8b5db740 1 --2- 172.21.15.28:0/2577011781 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c2bebf1da0 0x55c2bebf4230 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.215+0000 7f9d8b5db740 1 --2- 172.21.15.28:0/2577011781 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2bebe1a20 0x55c2bebdf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.215+0000 7f9d8b5db740 1 --2- 172.21.15.28:0/2577011781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2bebd7350 0x55c2bebdef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.215+0000 7f9d67400640 0 client.26863 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:54:33.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.215+0000 7f9d8b5db740 1 --2- 172.21.15.28:0/2577011781 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c2bebd6a30 0x55c2bebdea10 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.215+0000 7f9d8b5db740 1 -- 172.21.15.28:0/2577011781 >> 172.21.15.28:0/2577011781 conn(0x55c2bebd2d50 msgr2=0x55c2bebd8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:33.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.215+0000 7f9d8b5db740 1 -- 172.21.15.28:0/2577011781 shutdown_connections 2024-04-23T14:54:33.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.215+0000 7f9d8b5db740 1 -- 172.21.15.28:0/2577011781 wait complete. 2024-04-23T14:54:33.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.216+0000 7f9d8b5db740 20 client.26863 trim_cache size 0 max 16384 2024-04-23T14:54:33.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.238+0000 7f9d8b5db740 1 Processor -- start 2024-04-23T14:54:33.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.238+0000 7f9d8b5db740 1 -- start start 2024-04-23T14:54:33.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.238+0000 7f9d8b5db740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2bebd6a30 0x55c2bebdd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:33.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.238+0000 7f9d8b5db740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2bebd7350 0x55c2bebed6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:33.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.238+0000 7f9d8b5db740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c2bebe1a20 0x55c2bebf9b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:33.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.238+0000 7f9d8b5db740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d801adeb0 con 0x55c2bebd6a30 2024-04-23T14:54:33.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.238+0000 7f9d8b5db740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9d801b0710 con 0x55c2bebd7350 2024-04-23T14:54:33.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.238+0000 7f9d8b5db740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9d801ae500 con 0x55c2bebe1a20 2024-04-23T14:54:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.239+0000 7f9d85200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2bebd7350 0x55c2bebed6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.239+0000 7f9d85c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2bebd6a30 0x55c2bebdd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.239+0000 7f9d86600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c2bebe1a20 0x55c2bebf9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.239+0000 7f9d85200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2bebd7350 0x55c2bebed6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48652/0 (socket says 172.21.15.28:48652) 2024-04-23T14:54:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.239+0000 7f9d85200640 1 -- 172.21.15.28:0/1131397867 learned_addr learned my addr 172.21.15.28:0/1131397867 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.239+0000 7f9d85200640 1 -- 172.21.15.28:0/1131397867 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c2bebe1a20 msgr2=0x55c2bebf9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.239+0000 7f9d85200640 1 --2- 172.21.15.28:0/1131397867 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c2bebe1a20 0x55c2bebf9b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.239+0000 7f9d85200640 1 -- 172.21.15.28:0/1131397867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2bebd6a30 msgr2=0x55c2bebdd2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.239+0000 7f9d85200640 1 --2- 172.21.15.28:0/1131397867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2bebd6a30 0x55c2bebdd2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.239+0000 7f9d85200640 1 -- 172.21.15.28:0/1131397867 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2be84e3f0 con 0x55c2bebd7350 2024-04-23T14:54:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.239+0000 7f9d86600640 1 --2- 172.21.15.28:0/1131397867 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c2bebe1a20 0x55c2bebf9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:54:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.239+0000 7f9d85c00640 1 --2- 172.21.15.28:0/1131397867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2bebd6a30 0x55c2bebdd2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:54:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.240+0000 7f9d85200640 1 --2- 172.21.15.28:0/1131397867 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2bebd7350 0x55c2bebed6d0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c00c6a0 tx=0x7f9d6c087040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec231203345e8238 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:33.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.240+0000 7f9d72000640 1 -- 172.21.15.28:0/1131397867 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c0d8040 con 0x55c2bebd7350 2024-04-23T14:54:33.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.240+0000 7f9d72000640 1 -- 172.21.15.28:0/1131397867 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d6c09d130 con 0x55c2bebd7350 2024-04-23T14:54:33.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.240+0000 7f9d72000640 1 -- 172.21.15.28:0/1131397867 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c003b10 con 0x55c2bebd7350 2024-04-23T14:54:33.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.241+0000 7f9d8b5db740 1 -- 172.21.15.28:0/1131397867 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2bebd7350 msgr2=0x55c2bebed6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:33.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.241+0000 7f9d8b5db740 1 --2- 172.21.15.28:0/1131397867 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2bebd7350 0x55c2bebed6d0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c00c6a0 tx=0x7f9d6c087040 comp rx=0 tx=0).stop 2024-04-23T14:54:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.241+0000 7f9d8b5db740 1 -- 172.21.15.28:0/1131397867 shutdown_connections 2024-04-23T14:54:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.241+0000 7f9d8b5db740 1 --2- 172.21.15.28:0/1131397867 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2bebd7350 0x55c2bebed6d0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.241+0000 7f9d8b5db740 1 --2- 172.21.15.28:0/1131397867 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c2bebe1a20 0x55c2bebf9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.241+0000 7f9d8b5db740 1 --2- 172.21.15.28:0/1131397867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2bebd6a30 0x55c2bebdd2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.241+0000 7f9d8b5db740 1 -- 172.21.15.28:0/1131397867 >> 172.21.15.28:0/1131397867 conn(0x55c2bebd2d50 msgr2=0x55c2bebd8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.241+0000 7f9d8b5db740 1 -- 172.21.15.28:0/1131397867 shutdown_connections 2024-04-23T14:54:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.241+0000 7f9d8b5db740 1 -- 172.21.15.28:0/1131397867 wait complete. 2024-04-23T14:54:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.241+0000 7f9d8b5db740 1 -- start start 2024-04-23T14:54:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.241+0000 7f9d8b5db740 1 Processor -- start 2024-04-23T14:54:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.242+0000 7f9d8b5db740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2bebd6a30 0x55c2bebdba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.242+0000 7f9d8b5db740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2bebd7350 0x55c2bebdbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.242+0000 7f9d8b5db740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c2bebe1a20 0x55c2bec57370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.242+0000 7f9d8b5db740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d801aebd0 con 0x55c2bebd6a30 2024-04-23T14:54:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.242+0000 7f9d8b5db740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9d801b0040 con 0x55c2bebd7350 2024-04-23T14:54:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.242+0000 7f9d8b5db740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9d801af2a0 con 0x55c2bebe1a20 2024-04-23T14:54:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.242+0000 7f9d8b5db740 10 client.0 _ll_register_callbacks cb 0x7f9d74009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:54:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.242+0000 7f9d85c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2bebd6a30 0x55c2bebdba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.242+0000 7f9d85200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2bebd7350 0x55c2bebdbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.242+0000 7f9d85c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2bebd6a30 0x55c2bebdba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56714/0 (socket says 172.21.15.28:56714) 2024-04-23T14:54:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.242+0000 7f9d85c00640 1 -- 172.21.15.28:0/614696142 learned_addr learned my addr 172.21.15.28:0/614696142 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.242+0000 7f9d86600640 1 --2- 172.21.15.28:0/614696142 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c2bebe1a20 0x55c2bec57370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.243+0000 7f9d85200640 1 -- 172.21.15.28:0/614696142 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c2bebe1a20 msgr2=0x55c2bec57370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.243+0000 7f9d85200640 1 --2- 172.21.15.28:0/614696142 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c2bebe1a20 0x55c2bec57370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.243+0000 7f9d85200640 1 -- 172.21.15.28:0/614696142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2bebd6a30 msgr2=0x55c2bebdba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.243+0000 7f9d85200640 1 --2- 172.21.15.28:0/614696142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2bebd6a30 0x55c2bebdba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.243+0000 7f9d85200640 1 -- 172.21.15.28:0/614696142 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2be80e0f0 con 0x55c2bebd7350 2024-04-23T14:54:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.243+0000 7f9d85200640 1 --2- 172.21.15.28:0/614696142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2bebd7350 0x55c2bebdbfd0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c00c6a0 tx=0x7f9d6c0095b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=707dbe2edcacc238 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:33.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.243+0000 7f9d67400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:54:33.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.243+0000 7f9d67400640 1 -- 172.21.15.28:0/614696142 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c0b5020 con 0x55c2bebd7350 2024-04-23T14:54:33.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.243+0000 7f9d8b5db740 1 -- 172.21.15.28:0/614696142 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d6c09d130 con 0x55c2bebd7350 2024-04-23T14:54:33.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.243+0000 7f9d67400640 1 -- 172.21.15.28:0/614696142 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8542+0+0 (secure 0 0 0) 0x7f9d6c09ad30 con 0x55c2bebd7350 2024-04-23T14:54:33.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.243+0000 7f9d67400640 1 -- 172.21.15.28:0/614696142 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f9d6c09ca30 con 0x55c2bebd7350 2024-04-23T14:54:33.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.243+0000 7f9d8b5db740 10 client.26885 fetch_fsmap learned FSMap version 20 2024-04-23T14:54:33.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.243+0000 7f9d8b5db740 1 -- 172.21.15.28:0/614696142 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55c2beb01810 con 0x55c2bebd7350 2024-04-23T14:54:33.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.244+0000 7f9d67400640 1 -- 172.21.15.28:0/614696142 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d6c09d130 con 0x55c2bebd7350 2024-04-23T14:54:33.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.244+0000 7f9d67400640 1 -- 172.21.15.28:0/614696142 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f9d6c09be80 con 0x55c2bebd7350 2024-04-23T14:54:33.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.244+0000 7f9d8b5db740 10 client.26885 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:54:33.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.244+0000 7f9d8b5db740 10 client.26885 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:54:33.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.244+0000 7f9d8b5db740 20 client.26885 populate_metadata read hostname 'smithi028' 2024-04-23T14:54:33.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.244+0000 7f9d8b5db740 1 --2- 172.21.15.28:0/614696142 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c2bec58d10 0x55c2bec5b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:33.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.244+0000 7f9d8b5db740 4 client.26885 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:54:33.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.244+0000 7f9d8b5db740 1 -- 172.21.15.28:0/614696142 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c2beb43a80 con 0x55c2bec58d10 2024-04-23T14:54:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.244+0000 7f9d85c00640 1 --2- 172.21.15.28:0/614696142 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c2bec58d10 0x55c2bec5b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.245+0000 7f9d85c00640 1 --2- 172.21.15.28:0/614696142 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c2bec58d10 0x55c2bec5b130 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc159a66ac46d64 server_cookie=61047e7e5808b62e in_seq=0 out_seq=0 2024-04-23T14:54:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.245+0000 7f9d67400640 10 client.26885 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:54:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.245+0000 7f9d67400640 1 -- 172.21.15.28:0/614696142 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c2beb43a80 con 0x55c2bec58d10 2024-04-23T14:54:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.245+0000 7f9d67400640 10 client.26885 handle_command_reply: tid=0 2024-04-23T14:54:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.245+0000 7f9d8b5db740 1 client.26885 shutdown 2024-04-23T14:54:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.245+0000 7f9d8b5db740 1 -- 172.21.15.28:0/614696142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2bebd7350 msgr2=0x55c2bebdbfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.245+0000 7f9d8b5db740 1 --2- 172.21.15.28:0/614696142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2bebd7350 0x55c2bebdbfd0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c00c6a0 tx=0x7f9d6c0095b0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.245+0000 7f9d8b5db740 1 -- 172.21.15.28:0/614696142 shutdown_connections 2024-04-23T14:54:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.245+0000 7f9d8b5db740 1 --2- 172.21.15.28:0/614696142 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55c2bec58d10 0x55c2bec5b130 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.245+0000 7f9d8b5db740 1 --2- 172.21.15.28:0/614696142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c2bebd7350 0x55c2bebdbfd0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.245+0000 7f9d8b5db740 1 --2- 172.21.15.28:0/614696142 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c2bebe1a20 0x55c2bec57370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.245+0000 7f9d8b5db740 1 --2- 172.21.15.28:0/614696142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2bebd6a30 0x55c2bebdba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.245+0000 7f9d67400640 0 client.26885 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:54:33.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.245+0000 7f9d8b5db740 1 -- 172.21.15.28:0/614696142 >> 172.21.15.28:0/614696142 conn(0x55c2bebd2d50 msgr2=0x55c2bebd8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:33.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.246+0000 7f9d8b5db740 1 -- 172.21.15.28:0/614696142 shutdown_connections 2024-04-23T14:54:33.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.246+0000 7f9d8b5db740 1 -- 172.21.15.28:0/614696142 wait complete. 2024-04-23T14:54:33.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.246+0000 7f9d8b5db740 20 client.26885 trim_cache size 0 max 16384 2024-04-23T14:54:33.250 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:54:33.250 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-23T14:54:33.250 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-23T14:54:33.251 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:54:33.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.246+0000 7f9d87000640 1 -- 172.21.15.28:0/3339019394 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f9d680a3da0 msgr2=0x7f9d680a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:33.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.246+0000 7f9d87000640 1 --2- 172.21.15.28:0/3339019394 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f9d680a3da0 0x7f9d680a6260 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9d74000c00 tx=0x7f9d74005b50 comp rx=0 tx=0).stop 2024-04-23T14:54:33.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.246+0000 7f9d87000640 1 -- 172.21.15.28:0/3339019394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d801093f0 msgr2=0x7f9d801a0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:33.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.246+0000 7f9d87000640 1 --2- 172.21.15.28:0/3339019394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d801093f0 0x7f9d801a0ba0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c00e9a0 tx=0x7f9d6c00ee60 comp rx=0 tx=0).stop 2024-04-23T14:54:33.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.247+0000 7f9d87000640 1 -- 172.21.15.28:0/3339019394 shutdown_connections 2024-04-23T14:54:33.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.247+0000 7f9d87000640 1 --2- 172.21.15.28:0/3339019394 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f9d680a3da0 0x7f9d680a6260 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.247+0000 7f9d87000640 1 --2- 172.21.15.28:0/3339019394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d801093f0 0x7f9d801a0ba0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.247+0000 7f9d87000640 1 --2- 172.21.15.28:0/3339019394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d8019a7d0 0x7f9d8019ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.247+0000 7f9d87000640 1 --2- 172.21.15.28:0/3339019394 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9d80104320 0x7f9d801a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.247+0000 7f9d87000640 1 -- 172.21.15.28:0/3339019394 >> 172.21.15.28:0/3339019394 conn(0x7f9d800ffb40 msgr2=0x7f9d80101ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:33.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.247+0000 7f9d87000640 1 -- 172.21.15.28:0/3339019394 shutdown_connections 2024-04-23T14:54:33.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.247+0000 7f9d87000640 1 -- 172.21.15.28:0/3339019394 wait complete. 2024-04-23T14:54:33.269 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-23T14:54:33.270 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:54:33.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:32 smithi105 ceph-mon[32555]: pgmap v326: 97 pgs: 97 active+clean; 530 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 120 op/s 2024-04-23T14:54:33.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:32 smithi105 ceph-mon[32555]: Adjusting osd_memory_target on smithi028 to 177.5M 2024-04-23T14:54:33.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:32 smithi105 ceph-mon[32555]: Unable to set osd_memory_target on smithi028 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-23T14:54:33.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.512+0000 7f0da9800640 1 -- 172.21.15.28:0/134248009 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0da4102230 msgr2=0x7f0da4102610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:33.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.512+0000 7f0da9800640 1 --2- 172.21.15.28:0/134248009 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0da4102230 0x7f0da4102610 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f0d94009900 tx=0x7f0d9402a3f0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.512+0000 7f0da9800640 1 -- 172.21.15.28:0/134248009 shutdown_connections 2024-04-23T14:54:33.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.512+0000 7f0da9800640 1 --2- 172.21.15.28:0/134248009 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0da4107300 0x7f0da410eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.512+0000 7f0da9800640 1 --2- 172.21.15.28:0/134248009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0da4102be0 0x7f0da4106c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.512+0000 7f0da9800640 1 --2- 172.21.15.28:0/134248009 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0da4102230 0x7f0da4102610 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.512+0000 7f0da9800640 1 -- 172.21.15.28:0/134248009 >> 172.21.15.28:0/134248009 conn(0x7f0da4079400 msgr2=0x7f0da4079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:33.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.512+0000 7f0da9800640 1 -- 172.21.15.28:0/134248009 shutdown_connections 2024-04-23T14:54:33.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.513+0000 7f0da9800640 1 -- 172.21.15.28:0/134248009 wait complete. 2024-04-23T14:54:33.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.514+0000 7f0da9800640 1 Processor -- start 2024-04-23T14:54:33.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.514+0000 7f0da9800640 1 -- start start 2024-04-23T14:54:33.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.514+0000 7f0da9800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0da4102230 0x7f0da41a0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:33.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.514+0000 7f0da9800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0da4107300 0x7f0da41a0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:33.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.514+0000 7f0da9800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0da419a920 0x7f0da419add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:33.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.514+0000 7f0da9800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0da41123e0 con 0x7f0da4102230 2024-04-23T14:54:33.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.514+0000 7f0da9800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0da4112260 con 0x7f0da419a920 2024-04-23T14:54:33.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.514+0000 7f0da9800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0da4112560 con 0x7f0da4107300 2024-04-23T14:54:33.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.514+0000 7f0da3400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0da419a920 0x7f0da419add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:33.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.515+0000 7f0da2000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0da4107300 0x7f0da41a0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:33.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.515+0000 7f0da3400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0da419a920 0x7f0da419add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48696/0 (socket says 172.21.15.28:48696) 2024-04-23T14:54:33.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.515+0000 7f0da3400640 1 -- 172.21.15.28:0/3351910135 learned_addr learned my addr 172.21.15.28:0/3351910135 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:33.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.515+0000 7f0da2a00640 1 --2- 172.21.15.28:0/3351910135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0da4102230 0x7f0da41a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:33.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.515+0000 7f0da3400640 1 -- 172.21.15.28:0/3351910135 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0da4107300 msgr2=0x7f0da41a0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:33.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.515+0000 7f0da3400640 1 --2- 172.21.15.28:0/3351910135 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0da4107300 0x7f0da41a0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.515+0000 7f0da3400640 1 -- 172.21.15.28:0/3351910135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0da4102230 msgr2=0x7f0da41a0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:33.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.515+0000 7f0da3400640 1 --2- 172.21.15.28:0/3351910135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0da4102230 0x7f0da41a0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.515+0000 7f0da3400640 1 -- 172.21.15.28:0/3351910135 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0da419b440 con 0x7f0da419a920 2024-04-23T14:54:33.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.515+0000 7f0da2000640 1 --2- 172.21.15.28:0/3351910135 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0da4107300 0x7f0da41a0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:54:33.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.515+0000 7f0da2a00640 1 --2- 172.21.15.28:0/3351910135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0da4102230 0x7f0da41a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:54:33.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.515+0000 7f0da3400640 1 --2- 172.21.15.28:0/3351910135 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0da419a920 0x7f0da419add0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c00efd0 tx=0x7f0d9c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:33.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.516+0000 7f0d9b400640 1 -- 172.21.15.28:0/3351910135 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d9c003b70 con 0x7f0da419a920 2024-04-23T14:54:33.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.516+0000 7f0d9b400640 1 -- 172.21.15.28:0/3351910135 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d9c003d10 con 0x7f0da419a920 2024-04-23T14:54:33.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.516+0000 7f0d9b400640 1 -- 172.21.15.28:0/3351910135 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d9c0109f0 con 0x7f0da419a920 2024-04-23T14:54:33.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.516+0000 7f0da9800640 1 -- 172.21.15.28:0/3351910135 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0da419b730 con 0x7f0da419a920 2024-04-23T14:54:33.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.516+0000 7f0da9800640 1 -- 172.21.15.28:0/3351910135 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0da41a7980 con 0x7f0da419a920 2024-04-23T14:54:33.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.517+0000 7f0da9800640 1 -- 172.21.15.28:0/3351910135 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0da4069270 con 0x7f0da419a920 2024-04-23T14:54:33.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.519+0000 7f0d9b400640 1 -- 172.21.15.28:0/3351910135 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f0d9c010b90 con 0x7f0da419a920 2024-04-23T14:54:33.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.520+0000 7f0d9b400640 1 --2- 172.21.15.28:0/3351910135 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f0d880a3da0 0x7f0d880a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:33.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.521+0000 7f0d9b400640 1 -- 172.21.15.28:0/3351910135 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8542+0+0 (secure 0 0 0) 0x7f0d9c0d10b0 con 0x7f0da419a920 2024-04-23T14:54:33.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.521+0000 7f0da2a00640 1 --2- 172.21.15.28:0/3351910135 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f0d880a3da0 0x7f0d880a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:33.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.522+0000 7f0da2a00640 1 --2- 172.21.15.28:0/3351910135 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f0d880a3da0 0x7f0d880a6260 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0d94006010 tx=0x7f0d94005a40 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:33.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.525+0000 7f0d9b400640 1 -- 172.21.15.28:0/3351910135 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0d9c09d6e0 con 0x7f0da419a920 2024-04-23T14:54:33.527 INFO:tasks.workunit.client.0.smithi028.stdout: 1 898440 8.80 MB/sec execute 409 sec latency 122.229 ms 2024-04-23T14:54:33.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.812+0000 7f0da9800640 1 -- 172.21.15.28:0/3351910135 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0da419be50 con 0x7f0da419a920 2024-04-23T14:54:33.814 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:54:33.814 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:54:33.814 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:54:33.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.813+0000 7f0d9b400640 1 -- 172.21.15.28:0/3351910135 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7f0da419be50 con 0x7f0da419a920 2024-04-23T14:54:33.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.816+0000 7f0da9800640 1 -- 172.21.15.28:0/3351910135 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f0d880a3da0 msgr2=0x7f0d880a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:33.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.816+0000 7f0da9800640 1 --2- 172.21.15.28:0/3351910135 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f0d880a3da0 0x7f0d880a6260 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0d94006010 tx=0x7f0d94005a40 comp rx=0 tx=0).stop 2024-04-23T14:54:33.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.816+0000 7f0da9800640 1 -- 172.21.15.28:0/3351910135 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0da419a920 msgr2=0x7f0da419add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:33.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.816+0000 7f0da9800640 1 --2- 172.21.15.28:0/3351910135 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0da419a920 0x7f0da419add0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c00efd0 tx=0x7f0d9c00c6a0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.816+0000 7f0da9800640 1 -- 172.21.15.28:0/3351910135 shutdown_connections 2024-04-23T14:54:33.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.816+0000 7f0da9800640 1 --2- 172.21.15.28:0/3351910135 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f0d880a3da0 0x7f0d880a6260 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.816+0000 7f0da9800640 1 --2- 172.21.15.28:0/3351910135 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0da419a920 0x7f0da419add0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.816+0000 7f0da9800640 1 --2- 172.21.15.28:0/3351910135 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0da4107300 0x7f0da41a0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.816+0000 7f0da9800640 1 --2- 172.21.15.28:0/3351910135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0da4102230 0x7f0da41a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:33.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.816+0000 7f0da9800640 1 -- 172.21.15.28:0/3351910135 >> 172.21.15.28:0/3351910135 conn(0x7f0da4079400 msgr2=0x7f0da4100250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:33.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.817+0000 7f0da9800640 1 -- 172.21.15.28:0/3351910135 shutdown_connections 2024-04-23T14:54:33.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:33.817+0000 7f0da9800640 1 -- 172.21.15.28:0/3351910135 wait complete. 2024-04-23T14:54:33.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-23T14:54:34.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.067+0000 7f53f9000640 1 -- 172.21.15.28:0/1599405834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53f4101620 msgr2=0x7f53f410e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:34.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.067+0000 7f53f9000640 1 --2- 172.21.15.28:0/1599405834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53f4101620 0x7f53f410e250 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f53e0009860 tx=0x7f53e002a340 comp rx=0 tx=0).stop 2024-04-23T14:54:34.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.068+0000 7f53f9000640 1 -- 172.21.15.28:0/1599405834 shutdown_connections 2024-04-23T14:54:34.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.068+0000 7f53f9000640 1 --2- 172.21.15.28:0/1599405834 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53f410e820 0x7f53f4110c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:34.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.068+0000 7f53f9000640 1 --2- 172.21.15.28:0/1599405834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53f4101620 0x7f53f410e250 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:34.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.068+0000 7f53f9000640 1 --2- 172.21.15.28:0/1599405834 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53f4100c70 0x7f53f4101050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:34.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.068+0000 7f53f9000640 1 -- 172.21.15.28:0/1599405834 >> 172.21.15.28:0/1599405834 conn(0x7f53f40fc880 msgr2=0x7f53f40feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:34.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.068+0000 7f53f9000640 1 -- 172.21.15.28:0/1599405834 shutdown_connections 2024-04-23T14:54:34.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.068+0000 7f53f9000640 1 -- 172.21.15.28:0/1599405834 wait complete. 2024-04-23T14:54:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.070+0000 7f53f9000640 1 Processor -- start 2024-04-23T14:54:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.070+0000 7f53f9000640 1 -- start start 2024-04-23T14:54:34.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.070+0000 7f53f9000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53f4100c70 0x7f53f41a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:34.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.070+0000 7f53f9000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53f4101620 0x7f53f41a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:34.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.070+0000 7f53f9000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53f410e820 0x7f53f419a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:34.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.071+0000 7f53f9000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f53f41144a0 con 0x7f53f4100c70 2024-04-23T14:54:34.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.071+0000 7f53f9000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f53f4114320 con 0x7f53f410e820 2024-04-23T14:54:34.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.071+0000 7f53f9000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f53f4114620 con 0x7f53f4101620 2024-04-23T14:54:34.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.071+0000 7f53f3e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53f410e820 0x7f53f419a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:34.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.071+0000 7f53f3e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53f410e820 0x7f53f419a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48712/0 (socket says 172.21.15.28:48712) 2024-04-23T14:54:34.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.071+0000 7f53f3e00640 1 -- 172.21.15.28:0/1988722043 learned_addr learned my addr 172.21.15.28:0/1988722043 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:34.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.071+0000 7f53f3400640 1 --2- 172.21.15.28:0/1988722043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53f4100c70 0x7f53f41a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:34.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.071+0000 7f53f2a00640 1 --2- 172.21.15.28:0/1988722043 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53f4101620 0x7f53f41a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:34.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.071+0000 7f53f3e00640 1 -- 172.21.15.28:0/1988722043 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53f4101620 msgr2=0x7f53f41a0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:34.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.071+0000 7f53f3e00640 1 --2- 172.21.15.28:0/1988722043 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53f4101620 0x7f53f41a0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:34.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.071+0000 7f53f3e00640 1 -- 172.21.15.28:0/1988722043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53f4100c70 msgr2=0x7f53f41a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:34.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.071+0000 7f53f3e00640 1 --2- 172.21.15.28:0/1988722043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53f4100c70 0x7f53f41a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:34.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.071+0000 7f53f3e00640 1 -- 172.21.15.28:0/1988722043 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53f419afe0 con 0x7f53f410e820 2024-04-23T14:54:34.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.072+0000 7f53f3e00640 1 --2- 172.21.15.28:0/1988722043 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53f410e820 0x7f53f419a860 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f53ec00c3a0 tx=0x7f53ec00c860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:34.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.072+0000 7f53ebe00640 1 -- 172.21.15.28:0/1988722043 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ec003970 con 0x7f53f410e820 2024-04-23T14:54:34.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.072+0000 7f53ebe00640 1 -- 172.21.15.28:0/1988722043 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53ec004020 con 0x7f53f410e820 2024-04-23T14:54:34.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.072+0000 7f53ebe00640 1 -- 172.21.15.28:0/1988722043 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ec00bab0 con 0x7f53f410e820 2024-04-23T14:54:34.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.073+0000 7f53f9000640 1 -- 172.21.15.28:0/1988722043 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53f419b2d0 con 0x7f53f410e820 2024-04-23T14:54:34.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.074+0000 7f53f9000640 1 -- 172.21.15.28:0/1988722043 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53f4102df0 con 0x7f53f410e820 2024-04-23T14:54:34.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.081+0000 7f53ebe00640 1 -- 172.21.15.28:0/1988722043 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f53ec01d020 con 0x7f53f410e820 2024-04-23T14:54:34.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.081+0000 7f53fd3bd740 1 Processor -- start 2024-04-23T14:54:34.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.082+0000 7f53fd3bd740 1 -- start start 2024-04-23T14:54:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.082+0000 7f53fd3bd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a956dcb40 0x556a956dcf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.082+0000 7f53fd3bd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a956dd460 0x556a956e75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.082+0000 7f53fd3bd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a956e7b30 0x556a956e9f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.082+0000 7f53fd3bd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556a956421d0 con 0x556a956dd460 2024-04-23T14:54:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.082+0000 7f53fd3bd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x556a955e8450 con 0x556a956e7b30 2024-04-23T14:54:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.082+0000 7f53fd3bd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556a9563e190 con 0x556a956dcb40 2024-04-23T14:54:34.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.083+0000 7f53f3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a956dd460 0x556a956e75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:34.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.083+0000 7f53f2a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a956dcb40 0x556a956dcf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:34.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.083+0000 7f53f2a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a956e7b30 0x556a956e9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:34.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.083+0000 7f53f3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a956dd460 0x556a956e75f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56768/0 (socket says 172.21.15.28:56768) 2024-04-23T14:54:34.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.083+0000 7f53f3400640 1 -- 172.21.15.28:0/2136345348 learned_addr learned my addr 172.21.15.28:0/2136345348 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:34.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.083+0000 7f53f3400640 1 -- 172.21.15.28:0/2136345348 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a956dcb40 msgr2=0x556a956dcf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:34.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.083+0000 7f53f3400640 1 --2- 172.21.15.28:0/2136345348 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a956dcb40 0x556a956dcf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:34.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.083+0000 7f53f3400640 1 -- 172.21.15.28:0/2136345348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a956e7b30 msgr2=0x556a956e9f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:34.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.084+0000 7f53f3400640 1 --2- 172.21.15.28:0/2136345348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a956e7b30 0x556a956e9f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:34.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.084+0000 7f53f3400640 1 -- 172.21.15.28:0/2136345348 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a956a5310 con 0x556a956dd460 2024-04-23T14:54:34.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.084+0000 7f53f2a00640 1 --2- 172.21.15.28:0/2136345348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a956e7b30 0x556a956e9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:54:34.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.084+0000 7f53f3400640 1 --2- 172.21.15.28:0/2136345348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a956dd460 0x556a956e75f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f53e4009c90 tx=0x7f53e4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76ac21aa62fdbcc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:34.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.085+0000 7f53ebe00640 1 --2- 172.21.15.28:0/1988722043 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f53d80ac5b0 0x7f53d80aea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:34.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.086+0000 7f53ebe00640 1 -- 172.21.15.28:0/1988722043 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8542+0+0 (secure 0 0 0) 0x7f53ec0d0690 con 0x7f53f410e820 2024-04-23T14:54:34.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.086+0000 7f53e8c00640 1 -- 172.21.15.28:0/2136345348 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53e4005220 con 0x556a956dd460 2024-04-23T14:54:34.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.086+0000 7f53f2a00640 1 --2- 172.21.15.28:0/1988722043 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f53d80ac5b0 0x7f53d80aea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:34.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.086+0000 7f53e8c00640 1 -- 172.21.15.28:0/2136345348 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53e4004c80 con 0x556a956dd460 2024-04-23T14:54:34.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.086+0000 7f53e8c00640 1 -- 172.21.15.28:0/2136345348 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53e4019830 con 0x556a956dd460 2024-04-23T14:54:34.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.087+0000 7f53fd3bd740 1 -- 172.21.15.28:0/2136345348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a956dd460 msgr2=0x556a956e75f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:34.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.087+0000 7f53fd3bd740 1 --2- 172.21.15.28:0/2136345348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a956dd460 0x556a956e75f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f53e4009c90 tx=0x7f53e4011810 comp rx=0 tx=0).stop 2024-04-23T14:54:34.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.088+0000 7f53f2a00640 1 --2- 172.21.15.28:0/1988722043 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f53d80ac5b0 0x7f53d80aea70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f53e002a930 tx=0x7f53e002a340 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:34.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.088+0000 7f53fd3bd740 1 -- 172.21.15.28:0/2136345348 shutdown_connections 2024-04-23T14:54:34.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.088+0000 7f53fd3bd740 1 --2- 172.21.15.28:0/2136345348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a956e7b30 0x556a956e9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:34.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.088+0000 7f53fd3bd740 1 --2- 172.21.15.28:0/2136345348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a956dd460 0x556a956e75f0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:34.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.088+0000 7f53fd3bd740 1 --2- 172.21.15.28:0/2136345348 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a956dcb40 0x556a956dcf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:34.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.088+0000 7f53fd3bd740 1 -- 172.21.15.28:0/2136345348 >> 172.21.15.28:0/2136345348 conn(0x556a956d8e60 msgr2=0x556a956d9240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:34.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.088+0000 7f53fd3bd740 1 -- 172.21.15.28:0/2136345348 shutdown_connections 2024-04-23T14:54:34.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.089+0000 7f53fd3bd740 1 -- 172.21.15.28:0/2136345348 wait complete. 2024-04-23T14:54:34.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.089+0000 7f53fd3bd740 1 -- start start 2024-04-23T14:54:34.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.089+0000 7f53fd3bd740 1 Processor -- start 2024-04-23T14:54:34.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.090+0000 7f53fd3bd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a956dcb40 0x556a956e7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:34.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.090+0000 7f53fd3bd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a956dd460 0x556a956e4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:34.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.090+0000 7f53fd3bd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a956e7b30 0x556a956e4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:34.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.090+0000 7f53fd3bd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556a956ea9c0 con 0x556a956e7b30 2024-04-23T14:54:34.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.090+0000 7f53fd3bd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x556a955fb810 con 0x556a956dd460 2024-04-23T14:54:34.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.090+0000 7f53fd3bd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556a956eab40 con 0x556a956dcb40 2024-04-23T14:54:34.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.090+0000 7f53fd3bd740 10 client.0 _ll_register_callbacks cb 0x556a955c9140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:54:34.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.091+0000 7f53f3400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a956dcb40 0x556a956e7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:34.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.091+0000 7f53f3400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a956dcb40 0x556a956e7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:56390/0 (socket says 172.21.15.28:56390) 2024-04-23T14:54:34.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.091+0000 7f53f3400640 1 -- 172.21.15.28:0/3187273200 learned_addr learned my addr 172.21.15.28:0/3187273200 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:34.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.091+0000 7f53f2a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a956dd460 0x556a956e4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:34.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.091+0000 7f53f3e00640 1 --2- 172.21.15.28:0/3187273200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a956e7b30 0x556a956e4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:34.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.091+0000 7f53f3400640 1 -- 172.21.15.28:0/3187273200 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a956dd460 msgr2=0x556a956e4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:34.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.091+0000 7f53f3400640 1 --2- 172.21.15.28:0/3187273200 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a956dd460 0x556a956e4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:34.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.091+0000 7f53f3400640 1 -- 172.21.15.28:0/3187273200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a956e7b30 msgr2=0x556a956e4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:34.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.091+0000 7f53f3400640 1 --2- 172.21.15.28:0/3187273200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a956e7b30 0x556a956e4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:34.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.091+0000 7f53f3400640 1 -- 172.21.15.28:0/3187273200 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a9560c3f0 con 0x556a956dcb40 2024-04-23T14:54:34.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.091+0000 7f53f2a00640 1 --2- 172.21.15.28:0/3187273200 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a956dd460 0x556a956e4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:54:34.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.091+0000 7f53f3400640 1 --2- 172.21.15.28:0/3187273200 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a956dcb40 0x556a956e7540 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f53e40048d0 tx=0x7f53e4004900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a497e7bb4f9670e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:34.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.091+0000 7f53dd600640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:54:34.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.092+0000 7f53dd600640 1 -- 172.21.15.28:0/3187273200 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53e4029070 con 0x556a956dcb40 2024-04-23T14:54:34.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.092+0000 7f53dd600640 1 -- 172.21.15.28:0/3187273200 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8542+0+0 (secure 0 0 0) 0x7f53e4019940 con 0x556a956dcb40 2024-04-23T14:54:34.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.092+0000 7f53fd3bd740 1 -- 172.21.15.28:0/3187273200 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53e4004c80 con 0x556a956dcb40 2024-04-23T14:54:34.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.092+0000 7f53dd600640 1 -- 172.21.15.28:0/3187273200 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f53e4035050 con 0x556a956dcb40 2024-04-23T14:54:34.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.092+0000 7f53fd3bd740 10 client.26887 fetch_fsmap learned FSMap version 20 2024-04-23T14:54:34.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.092+0000 7f53fd3bd740 1 -- 172.21.15.28:0/3187273200 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x556a953543f0 con 0x556a956dcb40 2024-04-23T14:54:34.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.092+0000 7f53dd600640 1 -- 172.21.15.28:0/3187273200 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53e4004c80 con 0x556a956dcb40 2024-04-23T14:54:34.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.093+0000 7f53dd600640 1 -- 172.21.15.28:0/3187273200 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f53e4019bd0 con 0x556a956dcb40 2024-04-23T14:54:34.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.093+0000 7f53fd3bd740 10 client.26887 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:54:34.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.093+0000 7f53fd3bd740 10 client.26887 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:54:34.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.093+0000 7f53fd3bd740 20 client.26887 populate_metadata read hostname 'smithi028' 2024-04-23T14:54:34.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.093+0000 7f53fd3bd740 1 --2- 172.21.15.28:0/3187273200 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x556a956f7bd0 0x556a956f9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:34.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.093+0000 7f53fd3bd740 4 client.26887 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:54:34.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.093+0000 7f53fd3bd740 1 -- 172.21.15.28:0/3187273200 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a9564fbf0 con 0x556a956f7bd0 2024-04-23T14:54:34.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.094+0000 7f53f2a00640 1 --2- 172.21.15.28:0/3187273200 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x556a956f7bd0 0x556a956f9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:34.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.094+0000 7f53f2a00640 1 --2- 172.21.15.28:0/3187273200 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x556a956f7bd0 0x556a956f9fc0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=484e3de7f04cedd4 server_cookie=e8f7a735922a7711 in_seq=0 out_seq=0 2024-04-23T14:54:34.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.094+0000 7f53dd600640 10 client.26887 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:54:34.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.095+0000 7f53dd600640 1 -- 172.21.15.28:0/3187273200 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556a9564fbf0 con 0x556a956f7bd0 2024-04-23T14:54:34.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.095+0000 7f53dd600640 10 client.26887 handle_command_reply: tid=0 2024-04-23T14:54:34.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.095+0000 7f53fd3bd740 1 client.26887 shutdown 2024-04-23T14:54:34.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.095+0000 7f53fd3bd740 1 -- 172.21.15.28:0/3187273200 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a956dcb40 msgr2=0x556a956e7540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:34.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.095+0000 7f53fd3bd740 1 --2- 172.21.15.28:0/3187273200 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a956dcb40 0x556a956e7540 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f53e40048d0 tx=0x7f53e4004900 comp rx=0 tx=0).stop 2024-04-23T14:54:34.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.095+0000 7f53fd3bd740 1 -- 172.21.15.28:0/3187273200 shutdown_connections 2024-04-23T14:54:34.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.095+0000 7f53fd3bd740 1 --2- 172.21.15.28:0/3187273200 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x556a956f7bd0 0x556a956f9fc0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:34.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.095+0000 7f53fd3bd740 1 --2- 172.21.15.28:0/3187273200 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a956dd460 0x556a956e4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:34.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.095+0000 7f53fd3bd740 1 --2- 172.21.15.28:0/3187273200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a956e7b30 0x556a956e4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:34.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.095+0000 7f53dd600640 0 client.26887 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:54:34.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.095+0000 7f53fd3bd740 1 --2- 172.21.15.28:0/3187273200 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a956dcb40 0x556a956e7540 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:34.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.095+0000 7f53fd3bd740 1 -- 172.21.15.28:0/3187273200 >> 172.21.15.28:0/3187273200 conn(0x556a956d8e60 msgr2=0x556a956dec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:34.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.096+0000 7f53fd3bd740 1 -- 172.21.15.28:0/3187273200 shutdown_connections 2024-04-23T14:54:34.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.096+0000 7f53fd3bd740 1 -- 172.21.15.28:0/3187273200 wait complete. 2024-04-23T14:54:34.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.096+0000 7f53fd3bd740 20 client.26887 trim_cache size 0 max 16384 2024-04-23T14:54:34.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.114+0000 7f53fd3bd740 1 Processor -- start 2024-04-23T14:54:34.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.114+0000 7f53fd3bd740 1 -- start start 2024-04-23T14:54:34.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.114+0000 7f53fd3bd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a956dcb40 0x556a956f3880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:34.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.114+0000 7f53fd3bd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a956dd460 0x556a956e3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:34.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.114+0000 7f53fd3bd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a956e7b30 0x556a956e7070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:34.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.114+0000 7f53fd3bd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f53f41a9b50 con 0x556a956e7b30 2024-04-23T14:54:34.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.114+0000 7f53fd3bd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f53f41ac390 con 0x556a956dd460 2024-04-23T14:54:34.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.114+0000 7f53fd3bd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f53f41aa1a0 con 0x556a956dcb40 2024-04-23T14:54:34.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.115+0000 7f53f3400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a956dcb40 0x556a956f3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:34.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.115+0000 7f53f3400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a956dcb40 0x556a956f3880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:56396/0 (socket says 172.21.15.28:56396) 2024-04-23T14:54:34.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.115+0000 7f53f2a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a956dd460 0x556a956e3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:34.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.115+0000 7f53f3400640 1 -- 172.21.15.28:0/3599102384 learned_addr learned my addr 172.21.15.28:0/3599102384 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:34.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.115+0000 7f53f3e00640 1 --2- 172.21.15.28:0/3599102384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a956e7b30 0x556a956e7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:34.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.115+0000 7f53f3400640 1 -- 172.21.15.28:0/3599102384 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a956dd460 msgr2=0x556a956e3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:34.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.115+0000 7f53f3400640 1 --2- 172.21.15.28:0/3599102384 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a956dd460 0x556a956e3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:34.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.115+0000 7f53f3400640 1 -- 172.21.15.28:0/3599102384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a956e7b30 msgr2=0x556a956e7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:34.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.115+0000 7f53f3400640 1 --2- 172.21.15.28:0/3599102384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a956e7b30 0x556a956e7070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:34.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.115+0000 7f53f3400640 1 -- 172.21.15.28:0/3599102384 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a956e51b0 con 0x556a956dcb40 2024-04-23T14:54:34.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.115+0000 7f53f2a00640 1 --2- 172.21.15.28:0/3599102384 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a956dd460 0x556a956e3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:54:34.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.115+0000 7f53f3400640 1 --2- 172.21.15.28:0/3599102384 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a956dcb40 0x556a956f3880 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f53e4000c00 tx=0x7f53e4019430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31db07de7e57d841 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:34.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.116+0000 7f53dfe00640 1 -- 172.21.15.28:0/3599102384 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53e4029040 con 0x556a956dcb40 2024-04-23T14:54:34.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.116+0000 7f53dfe00640 1 -- 172.21.15.28:0/3599102384 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53e4005d30 con 0x556a956dcb40 2024-04-23T14:54:34.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.116+0000 7f53dfe00640 1 -- 172.21.15.28:0/3599102384 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53e4004bf0 con 0x556a956dcb40 2024-04-23T14:54:34.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.116+0000 7f53fd3bd740 1 -- 172.21.15.28:0/3599102384 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a956dcb40 msgr2=0x556a956f3880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:34.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.116+0000 7f53fd3bd740 1 --2- 172.21.15.28:0/3599102384 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a956dcb40 0x556a956f3880 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f53e4000c00 tx=0x7f53e4019430 comp rx=0 tx=0).stop 2024-04-23T14:54:34.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.116+0000 7f53fd3bd740 1 -- 172.21.15.28:0/3599102384 shutdown_connections 2024-04-23T14:54:34.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.116+0000 7f53fd3bd740 1 --2- 172.21.15.28:0/3599102384 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a956dd460 0x556a956e3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:34.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.116+0000 7f53fd3bd740 1 --2- 172.21.15.28:0/3599102384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a956e7b30 0x556a956e7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:34.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.116+0000 7f53fd3bd740 1 --2- 172.21.15.28:0/3599102384 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a956dcb40 0x556a956f3880 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:34.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.116+0000 7f53fd3bd740 1 -- 172.21.15.28:0/3599102384 >> 172.21.15.28:0/3599102384 conn(0x556a956d8e60 msgr2=0x556a956e9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:34.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.117+0000 7f53fd3bd740 1 -- 172.21.15.28:0/3599102384 shutdown_connections 2024-04-23T14:54:34.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.117+0000 7f53fd3bd740 1 -- 172.21.15.28:0/3599102384 wait complete. 2024-04-23T14:54:34.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.117+0000 7f53fd3bd740 1 -- start start 2024-04-23T14:54:34.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.117+0000 7f53fd3bd740 1 Processor -- start 2024-04-23T14:54:34.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.118+0000 7f53fd3bd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a956dcb40 0x556a956f0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.118+0000 7f53fd3bd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a956dd460 0x556a956e1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.118+0000 7f53fd3bd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a956e7b30 0x556a956e17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.118+0000 7f53fd3bd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f53f41aa850 con 0x556a956dd460 2024-04-23T14:54:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.118+0000 7f53fd3bd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f53f41abcc0 con 0x556a956dcb40 2024-04-23T14:54:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.118+0000 7f53fd3bd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f53f41aaf20 con 0x556a956e7b30 2024-04-23T14:54:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.118+0000 7f53fd3bd740 10 client.0 _ll_register_callbacks cb 0x7f53ec009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:54:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.118+0000 7f53f3400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a956dcb40 0x556a956f0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.118+0000 7f53f3e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a956e7b30 0x556a956e17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.118+0000 7f53f3400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a956dcb40 0x556a956f0210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48744/0 (socket says 172.21.15.28:48744) 2024-04-23T14:54:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.118+0000 7f53f3400640 1 -- 172.21.15.28:0/351271758 learned_addr learned my addr 172.21.15.28:0/351271758 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.118+0000 7f53f3400640 1 -- 172.21.15.28:0/351271758 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a956e7b30 msgr2=0x556a956e17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.118+0000 7f53f3400640 1 --2- 172.21.15.28:0/351271758 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a956e7b30 0x556a956e17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.118+0000 7f53f3400640 1 -- 172.21.15.28:0/351271758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a956dd460 msgr2=0x556a956e1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.119+0000 7f53f2a00640 1 --2- 172.21.15.28:0/351271758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a956dd460 0x556a956e1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.119+0000 7f53f3400640 1 --2- 172.21.15.28:0/351271758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a956dd460 0x556a956e1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.119+0000 7f53f3400640 1 -- 172.21.15.28:0/351271758 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a9541e7d0 con 0x556a956dcb40 2024-04-23T14:54:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.119+0000 7f53f3e00640 1 --2- 172.21.15.28:0/351271758 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a956e7b30 0x556a956e17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:54:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.119+0000 7f53f2a00640 1 --2- 172.21.15.28:0/351271758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a956dd460 0x556a956e1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:54:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.119+0000 7f53f3400640 1 --2- 172.21.15.28:0/351271758 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a956dcb40 0x556a956f0210 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f53e4009350 tx=0x7f53e4004b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3fa0ecb5bdcdd11 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.119+0000 7f53dd600640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:54:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.119+0000 7f53dd600640 1 -- 172.21.15.28:0/351271758 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53e4029040 con 0x556a956dcb40 2024-04-23T14:54:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.119+0000 7f53dd600640 1 -- 172.21.15.28:0/351271758 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8542+0+0 (secure 0 0 0) 0x7f53e4005380 con 0x556a956dcb40 2024-04-23T14:54:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.119+0000 7f53fd3bd740 1 -- 172.21.15.28:0/351271758 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53e4005d30 con 0x556a956dcb40 2024-04-23T14:54:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.120+0000 7f53dd600640 1 -- 172.21.15.28:0/351271758 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f53e4004b90 con 0x556a956dcb40 2024-04-23T14:54:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.120+0000 7f53fd3bd740 10 client.26909 fetch_fsmap learned FSMap version 20 2024-04-23T14:54:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.120+0000 7f53fd3bd740 1 -- 172.21.15.28:0/351271758 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x556a9575dfc0 con 0x556a956dcb40 2024-04-23T14:54:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.120+0000 7f53dd600640 1 -- 172.21.15.28:0/351271758 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53e4005d30 con 0x556a956dcb40 2024-04-23T14:54:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.120+0000 7f53dd600640 1 -- 172.21.15.28:0/351271758 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f53e4019740 con 0x556a956dcb40 2024-04-23T14:54:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.120+0000 7f53fd3bd740 10 client.26909 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:54:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.120+0000 7f53fd3bd740 10 client.26909 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:54:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.120+0000 7f53fd3bd740 20 client.26909 populate_metadata read hostname 'smithi028' 2024-04-23T14:54:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.120+0000 7f53fd3bd740 1 --2- 172.21.15.28:0/351271758 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x556a9575e430 0x556a95760850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:34.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.120+0000 7f53fd3bd740 4 client.26909 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-23T14:54:34.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.120+0000 7f53fd3bd740 1 -- 172.21.15.28:0/351271758 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556a95649a60 con 0x556a9575e430 2024-04-23T14:54:34.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.121+0000 7f53f2a00640 1 --2- 172.21.15.28:0/351271758 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x556a9575e430 0x556a95760850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:34.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.121+0000 7f53f2a00640 1 --2- 172.21.15.28:0/351271758 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x556a9575e430 0x556a95760850 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f63ec7b14e7ab2e server_cookie=397d6f17ce1c2335 in_seq=0 out_seq=0 2024-04-23T14:54:34.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.121+0000 7f53dd600640 10 client.26909 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:54:34.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.121+0000 7f53dd600640 1 -- 172.21.15.28:0/351271758 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556a95649a60 con 0x556a9575e430 2024-04-23T14:54:34.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.121+0000 7f53dd600640 10 client.26909 handle_command_reply: tid=0 2024-04-23T14:54:34.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.121+0000 7f53fd3bd740 1 client.26909 shutdown 2024-04-23T14:54:34.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.122+0000 7f53fd3bd740 1 -- 172.21.15.28:0/351271758 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a956dcb40 msgr2=0x556a956f0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:34.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.122+0000 7f53fd3bd740 1 --2- 172.21.15.28:0/351271758 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a956dcb40 0x556a956f0210 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f53e4009350 tx=0x7f53e4004b00 comp rx=0 tx=0).stop 2024-04-23T14:54:34.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.122+0000 7f53fd3bd740 1 -- 172.21.15.28:0/351271758 shutdown_connections 2024-04-23T14:54:34.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.122+0000 7f53fd3bd740 1 --2- 172.21.15.28:0/351271758 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x556a9575e430 0x556a95760850 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:34.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.122+0000 7f53fd3bd740 1 --2- 172.21.15.28:0/351271758 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a956e7b30 0x556a956e17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:34.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.122+0000 7f53fd3bd740 1 --2- 172.21.15.28:0/351271758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a956dd460 0x556a956e1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:34.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.122+0000 7f53dd600640 0 client.26909 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:54:34.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.122+0000 7f53fd3bd740 1 --2- 172.21.15.28:0/351271758 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a956dcb40 0x556a956f0210 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:34.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.122+0000 7f53fd3bd740 1 -- 172.21.15.28:0/351271758 >> 172.21.15.28:0/351271758 conn(0x556a956d8e60 msgr2=0x556a956e9f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:34.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.122+0000 7f53fd3bd740 1 -- 172.21.15.28:0/351271758 shutdown_connections 2024-04-23T14:54:34.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.122+0000 7f53fd3bd740 1 -- 172.21.15.28:0/351271758 wait complete. 2024-04-23T14:54:34.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.122+0000 7f53fd3bd740 20 client.26909 trim_cache size 0 max 16384 2024-04-23T14:54:34.127 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-23T14:54:34.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.123+0000 7f53f9000640 1 -- 172.21.15.28:0/1988722043 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f53d80ac5b0 msgr2=0x7f53d80aea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:34.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.123+0000 7f53f9000640 1 --2- 172.21.15.28:0/1988722043 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f53d80ac5b0 0x7f53d80aea70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f53e002a930 tx=0x7f53e002a340 comp rx=0 tx=0).stop 2024-04-23T14:54:34.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.123+0000 7f53f9000640 1 -- 172.21.15.28:0/1988722043 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53f410e820 msgr2=0x7f53f419a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:34.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.123+0000 7f53f9000640 1 --2- 172.21.15.28:0/1988722043 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53f410e820 0x7f53f419a860 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f53ec00c3a0 tx=0x7f53ec00c860 comp rx=0 tx=0).stop 2024-04-23T14:54:34.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.123+0000 7f53f9000640 1 -- 172.21.15.28:0/1988722043 shutdown_connections 2024-04-23T14:54:34.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.123+0000 7f53f9000640 1 --2- 172.21.15.28:0/1988722043 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f53d80ac5b0 0x7f53d80aea70 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:34.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.123+0000 7f53f9000640 1 --2- 172.21.15.28:0/1988722043 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53f410e820 0x7f53f419a860 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:34.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.123+0000 7f53f9000640 1 --2- 172.21.15.28:0/1988722043 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53f4101620 0x7f53f41a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:34.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.123+0000 7f53f9000640 1 --2- 172.21.15.28:0/1988722043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53f4100c70 0x7f53f41a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:34.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.123+0000 7f53f9000640 1 -- 172.21.15.28:0/1988722043 >> 172.21.15.28:0/1988722043 conn(0x7f53f40fc880 msgr2=0x7f53f40fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:34.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.123+0000 7f53f9000640 1 -- 172.21.15.28:0/1988722043 shutdown_connections 2024-04-23T14:54:34.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:34.123+0000 7f53f9000640 1 -- 172.21.15.28:0/1988722043 wait complete. 2024-04-23T14:54:34.527 INFO:tasks.workunit.client.0.smithi028.stdout: 1 899730 8.79 MB/sec execute 410 sec latency 22.822 ms 2024-04-23T14:54:35.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:34 smithi038 ceph-mon[32348]: pgmap v327: 97 pgs: 97 active+clean; 538 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.7 MiB/s wr, 190 op/s 2024-04-23T14:54:35.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:34 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3351910135' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:54:35.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:34 smithi028 ceph-mon[27810]: pgmap v327: 97 pgs: 97 active+clean; 538 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.7 MiB/s wr, 190 op/s 2024-04-23T14:54:35.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:34 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3351910135' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:54:35.147 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-23T14:54:35.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:54:35.148 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-23T14:54:35.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:34 smithi105 ceph-mon[32555]: pgmap v327: 97 pgs: 97 active+clean; 538 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.7 MiB/s wr, 190 op/s 2024-04-23T14:54:35.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:34 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3351910135' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:54:35.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.373+0000 7f160d600640 1 -- 172.21.15.28:0/1999384824 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f160810c7c0 msgr2=0x7f1608109000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:35.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.373+0000 7f160d600640 1 --2- 172.21.15.28:0/1999384824 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f160810c7c0 0x7f1608109000 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f15f8009900 tx=0x7f15f802a3f0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.373+0000 7f160d600640 1 -- 172.21.15.28:0/1999384824 shutdown_connections 2024-04-23T14:54:35.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.373+0000 7f160d600640 1 --2- 172.21.15.28:0/1999384824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16081095d0 0x7f1608109a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.373+0000 7f160d600640 1 --2- 172.21.15.28:0/1999384824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1608117130 0x7f1608119520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.373+0000 7f160d600640 1 --2- 172.21.15.28:0/1999384824 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f160810c7c0 0x7f1608109000 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.373+0000 7f160d600640 1 -- 172.21.15.28:0/1999384824 >> 172.21.15.28:0/1999384824 conn(0x7f160806d570 msgr2=0x7f160806d980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:35.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.373+0000 7f160d600640 1 -- 172.21.15.28:0/1999384824 shutdown_connections 2024-04-23T14:54:35.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.373+0000 7f160d600640 1 -- 172.21.15.28:0/1999384824 wait complete. 2024-04-23T14:54:35.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.374+0000 7f160d600640 1 Processor -- start 2024-04-23T14:54:35.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.374+0000 7f160d600640 1 -- start start 2024-04-23T14:54:35.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.375+0000 7f160d600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16081095d0 0x7f16081a3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.375+0000 7f160d600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1608117130 0x7f16081a38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.375+0000 7f160d600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16081a7c40 0x7f16081a80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.375+0000 7f160d600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f160811ce20 con 0x7f16081095d0 2024-04-23T14:54:35.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.375+0000 7f160d600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f160811cca0 con 0x7f16081a7c40 2024-04-23T14:54:35.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.375+0000 7f160d600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f160811cfa0 con 0x7f1608117130 2024-04-23T14:54:35.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.375+0000 7f1606a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16081095d0 0x7f16081a3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.375+0000 7f1607400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16081a7c40 0x7f16081a80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.375+0000 7f1606000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1608117130 0x7f16081a38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.375+0000 7f1607400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16081a7c40 0x7f16081a80f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48760/0 (socket says 172.21.15.28:48760) 2024-04-23T14:54:35.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.375+0000 7f1607400640 1 -- 172.21.15.28:0/3857524818 learned_addr learned my addr 172.21.15.28:0/3857524818 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:35.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.376+0000 7f1606000640 1 -- 172.21.15.28:0/3857524818 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16081a7c40 msgr2=0x7f16081a80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:35.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.376+0000 7f1606000640 1 --2- 172.21.15.28:0/3857524818 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16081a7c40 0x7f16081a80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.376+0000 7f1606000640 1 -- 172.21.15.28:0/3857524818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16081095d0 msgr2=0x7f16081a3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:35.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.376+0000 7f1606000640 1 --2- 172.21.15.28:0/3857524818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16081095d0 0x7f16081a3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.376+0000 7f1606000640 1 -- 172.21.15.28:0/3857524818 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16081a8730 con 0x7f1608117130 2024-04-23T14:54:35.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.376+0000 7f1606000640 1 --2- 172.21.15.28:0/3857524818 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1608117130 0x7f16081a38c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f15f400e9c0 tx=0x7f15f400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:35.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.376+0000 7f15ff400640 1 -- 172.21.15.28:0/3857524818 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f400ccc0 con 0x7f1608117130 2024-04-23T14:54:35.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.376+0000 7f15ff400640 1 -- 172.21.15.28:0/3857524818 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15f40035f0 con 0x7f1608117130 2024-04-23T14:54:35.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.376+0000 7f160d600640 1 -- 172.21.15.28:0/3857524818 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16081a8a20 con 0x7f1608117130 2024-04-23T14:54:35.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.377+0000 7f160d600640 1 -- 172.21.15.28:0/3857524818 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16081b01c0 con 0x7f1608117130 2024-04-23T14:54:35.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.377+0000 7f15ff400640 1 -- 172.21.15.28:0/3857524818 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f400ce60 con 0x7f1608117130 2024-04-23T14:54:35.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.380+0000 7f1610402740 1 Processor -- start 2024-04-23T14:54:35.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.380+0000 7f1610402740 1 -- start start 2024-04-23T14:54:35.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.380+0000 7f15ff400640 1 -- 172.21.15.28:0/3857524818 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f15f40037a0 con 0x7f1608117130 2024-04-23T14:54:35.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.381+0000 7f1610402740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559ff9764090 0x559ff9764470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.381+0000 7f1610402740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ff97649b0 0x559ff976eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.381+0000 7f1610402740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559ff976f080 0x559ff9771470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.381+0000 7f1610402740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559ff96c9e70 con 0x559ff97649b0 2024-04-23T14:54:35.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.381+0000 7f1610402740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559ff9670380 con 0x559ff976f080 2024-04-23T14:54:35.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.381+0000 7f1610402740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559ff96c5e30 con 0x559ff9764090 2024-04-23T14:54:35.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.381+0000 7f1606a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ff97649b0 0x559ff976eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.381+0000 7f1606000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559ff976f080 0x559ff9771470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.381+0000 7f1607400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559ff9764090 0x559ff9764470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.381+0000 7f1606a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ff97649b0 0x559ff976eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56822/0 (socket says 172.21.15.28:56822) 2024-04-23T14:54:35.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.381+0000 7f1606a00640 1 -- 172.21.15.28:0/3727505616 learned_addr learned my addr 172.21.15.28:0/3727505616 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:35.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.382+0000 7f1607400640 1 -- 172.21.15.28:0/3727505616 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559ff976f080 msgr2=0x559ff9771470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:35.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.382+0000 7f1607400640 1 --2- 172.21.15.28:0/3727505616 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559ff976f080 0x559ff9771470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.382+0000 7f1607400640 1 -- 172.21.15.28:0/3727505616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ff97649b0 msgr2=0x559ff976eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:35.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.382+0000 7f1607400640 1 --2- 172.21.15.28:0/3727505616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ff97649b0 0x559ff976eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.382+0000 7f1607400640 1 -- 172.21.15.28:0/3727505616 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ff94a47e0 con 0x559ff9764090 2024-04-23T14:54:35.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.382+0000 7f1606a00640 1 --2- 172.21.15.28:0/3727505616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ff97649b0 0x559ff976eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:54:35.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.382+0000 7f1607400640 1 --2- 172.21.15.28:0/3727505616 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559ff9764090 0x559ff9764470 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1600009f10 tx=0x7f1600010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4903b245ff840fd server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:35.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.382+0000 7f15f3400640 1 -- 172.21.15.28:0/3727505616 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1600016070 con 0x559ff9764090 2024-04-23T14:54:35.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.382+0000 7f15f3400640 1 -- 172.21.15.28:0/3727505616 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1600004d20 con 0x559ff9764090 2024-04-23T14:54:35.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.382+0000 7f15f3400640 1 -- 172.21.15.28:0/3727505616 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16000128f0 con 0x559ff9764090 2024-04-23T14:54:35.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.383+0000 7f15ff400640 1 --2- 172.21.15.28:0/3857524818 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f15ec0a3da0 0x7f15ec0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.383+0000 7f15ff400640 1 -- 172.21.15.28:0/3857524818 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8542+0+0 (secure 0 0 0) 0x7f15f40d0c80 con 0x7f1608117130 2024-04-23T14:54:35.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.383+0000 7f1606a00640 1 --2- 172.21.15.28:0/3857524818 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f15ec0a3da0 0x7f15ec0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.383+0000 7f1610402740 1 -- 172.21.15.28:0/3727505616 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559ff9764090 msgr2=0x559ff9764470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:35.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.383+0000 7f1610402740 1 --2- 172.21.15.28:0/3727505616 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559ff9764090 0x559ff9764470 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1600009f10 tx=0x7f1600010740 comp rx=0 tx=0).stop 2024-04-23T14:54:35.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.383+0000 7f1610402740 1 -- 172.21.15.28:0/3727505616 shutdown_connections 2024-04-23T14:54:35.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.383+0000 7f1610402740 1 --2- 172.21.15.28:0/3727505616 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559ff976f080 0x559ff9771470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.383+0000 7f1610402740 1 --2- 172.21.15.28:0/3727505616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ff97649b0 0x559ff976eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.383+0000 7f1610402740 1 --2- 172.21.15.28:0/3727505616 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559ff9764090 0x559ff9764470 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.383+0000 7f1610402740 1 -- 172.21.15.28:0/3727505616 >> 172.21.15.28:0/3727505616 conn(0x559ff97603b0 msgr2=0x559ff9760790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:35.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.383+0000 7f1610402740 1 -- 172.21.15.28:0/3727505616 shutdown_connections 2024-04-23T14:54:35.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.384+0000 7f1610402740 1 -- 172.21.15.28:0/3727505616 wait complete. 2024-04-23T14:54:35.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.384+0000 7f1606a00640 1 --2- 172.21.15.28:0/3857524818 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f15ec0a3da0 0x7f15ec0a6260 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f15f80096c0 tx=0x7f15f8000bc0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:35.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.384+0000 7f1610402740 1 -- start start 2024-04-23T14:54:35.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.385+0000 7f1610402740 1 Processor -- start 2024-04-23T14:54:35.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.385+0000 7f1610402740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ff9764090 0x559ff976c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.385+0000 7f1610402740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559ff97649b0 0x559ff976c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.385+0000 7f1610402740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559ff976f080 0x559ff976caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.385+0000 7f1610402740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559ff9771e70 con 0x559ff9764090 2024-04-23T14:54:35.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.385+0000 7f1610402740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559ff9683660 con 0x559ff97649b0 2024-04-23T14:54:35.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.385+0000 7f1610402740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559ff9771ff0 con 0x559ff976f080 2024-04-23T14:54:35.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.385+0000 7f1610402740 10 client.0 _ll_register_callbacks cb 0x559ff9671aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:54:35.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.385+0000 7f1606000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559ff97649b0 0x559ff976c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.385+0000 7f1607400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559ff976f080 0x559ff976caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.385+0000 7f1606000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559ff97649b0 0x559ff976c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48788/0 (socket says 172.21.15.28:48788) 2024-04-23T14:54:35.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.386+0000 7f1606000640 1 -- 172.21.15.28:0/1347322590 learned_addr learned my addr 172.21.15.28:0/1347322590 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:35.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.386+0000 7f1606a00640 1 --2- 172.21.15.28:0/1347322590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ff9764090 0x559ff976c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.386+0000 7f1606000640 1 -- 172.21.15.28:0/1347322590 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559ff976f080 msgr2=0x559ff976caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:35.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.386+0000 7f1606000640 1 --2- 172.21.15.28:0/1347322590 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559ff976f080 0x559ff976caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.386+0000 7f1606000640 1 -- 172.21.15.28:0/1347322590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ff9764090 msgr2=0x559ff976c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:35.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.386+0000 7f1606000640 1 --2- 172.21.15.28:0/1347322590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ff9764090 0x559ff976c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.386+0000 7f1606000640 1 -- 172.21.15.28:0/1347322590 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ff9504f70 con 0x559ff97649b0 2024-04-23T14:54:35.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.386+0000 7f1606a00640 1 --2- 172.21.15.28:0/1347322590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ff9764090 0x559ff976c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:54:35.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.386+0000 7f1607400640 1 --2- 172.21.15.28:0/1347322590 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559ff976f080 0x559ff976caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:54:35.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.386+0000 7f1606000640 1 --2- 172.21.15.28:0/1347322590 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559ff97649b0 0x559ff976c560 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f15f409cba0 tx=0x7f15f4098850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfed620473c974dc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:35.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.386+0000 7f15e7e00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:54:35.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.387+0000 7f15e7e00640 1 -- 172.21.15.28:0/1347322590 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f409c570 con 0x559ff97649b0 2024-04-23T14:54:35.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.387+0000 7f15e7e00640 1 -- 172.21.15.28:0/1347322590 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8542+0+0 (secure 0 0 0) 0x7f15f4099280 con 0x559ff97649b0 2024-04-23T14:54:35.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.387+0000 7f1610402740 1 -- 172.21.15.28:0/1347322590 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1600004d20 con 0x559ff97649b0 2024-04-23T14:54:35.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.387+0000 7f15e7e00640 1 -- 172.21.15.28:0/1347322590 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f15f409d310 con 0x559ff97649b0 2024-04-23T14:54:35.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.387+0000 7f1610402740 10 client.26921 fetch_fsmap learned FSMap version 20 2024-04-23T14:54:35.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.387+0000 7f1610402740 1 -- 172.21.15.28:0/1347322590 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x559ff939c170 con 0x559ff97649b0 2024-04-23T14:54:35.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.387+0000 7f15e7e00640 1 -- 172.21.15.28:0/1347322590 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1600004d20 con 0x559ff97649b0 2024-04-23T14:54:35.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.387+0000 7f15e7e00640 1 -- 172.21.15.28:0/1347322590 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f15f40a5480 con 0x559ff97649b0 2024-04-23T14:54:35.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.387+0000 7f1610402740 10 client.26921 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:54:35.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.387+0000 7f1610402740 10 client.26921 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:54:35.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.387+0000 7f1610402740 20 client.26921 populate_metadata read hostname 'smithi028' 2024-04-23T14:54:35.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.387+0000 7f1610402740 1 --2- 172.21.15.28:0/1347322590 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x559ff977f450 0x559ff9781840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.387+0000 7f1610402740 4 client.26921 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:54:35.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.387+0000 7f1610402740 1 -- 172.21.15.28:0/1347322590 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ff96d7890 con 0x559ff977f450 2024-04-23T14:54:35.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.388+0000 7f1606a00640 1 --2- 172.21.15.28:0/1347322590 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x559ff977f450 0x559ff9781840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.388+0000 7f1606a00640 1 --2- 172.21.15.28:0/1347322590 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x559ff977f450 0x559ff9781840 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4bdc192e0e8325b5 server_cookie=38478a79608af0a6 in_seq=0 out_seq=0 2024-04-23T14:54:35.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.388+0000 7f15e7e00640 10 client.26921 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:54:35.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.390+0000 7f15e7e00640 1 -- 172.21.15.28:0/1347322590 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559ff96d7890 con 0x559ff977f450 2024-04-23T14:54:35.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.390+0000 7f15e7e00640 10 client.26921 handle_command_reply: tid=0 2024-04-23T14:54:35.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.390+0000 7f1610402740 1 client.26921 shutdown 2024-04-23T14:54:35.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.390+0000 7f1610402740 1 -- 172.21.15.28:0/1347322590 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559ff97649b0 msgr2=0x559ff976c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:35.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.390+0000 7f1610402740 1 --2- 172.21.15.28:0/1347322590 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559ff97649b0 0x559ff976c560 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f15f409cba0 tx=0x7f15f4098850 comp rx=0 tx=0).stop 2024-04-23T14:54:35.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.390+0000 7f1610402740 1 -- 172.21.15.28:0/1347322590 shutdown_connections 2024-04-23T14:54:35.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.390+0000 7f1610402740 1 --2- 172.21.15.28:0/1347322590 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x559ff977f450 0x559ff9781840 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.390+0000 7f1610402740 1 --2- 172.21.15.28:0/1347322590 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559ff97649b0 0x559ff976c560 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.390+0000 7f1610402740 1 --2- 172.21.15.28:0/1347322590 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559ff976f080 0x559ff976caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.390+0000 7f1610402740 1 --2- 172.21.15.28:0/1347322590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ff9764090 0x559ff976c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.390+0000 7f15e7e00640 0 client.26921 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:54:35.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.390+0000 7f1610402740 1 -- 172.21.15.28:0/1347322590 >> 172.21.15.28:0/1347322590 conn(0x559ff97603b0 msgr2=0x559ff97661b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:35.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.391+0000 7f1610402740 1 -- 172.21.15.28:0/1347322590 shutdown_connections 2024-04-23T14:54:35.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.391+0000 7f1610402740 1 -- 172.21.15.28:0/1347322590 wait complete. 2024-04-23T14:54:35.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.391+0000 7f1610402740 20 client.26921 trim_cache size 0 max 16384 2024-04-23T14:54:35.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.412+0000 7f1610402740 1 Processor -- start 2024-04-23T14:54:35.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.412+0000 7f1610402740 1 -- start start 2024-04-23T14:54:35.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.413+0000 7f1610402740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559ff9764090 0x559ff977ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.413+0000 7f1610402740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ff97649b0 0x559ff9787090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.413+0000 7f1610402740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559ff976f080 0x559ff97875d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.413+0000 7f1610402740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16081b6820 con 0x559ff97649b0 2024-04-23T14:54:35.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.413+0000 7f1610402740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f16081b9080 con 0x559ff9764090 2024-04-23T14:54:35.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.413+0000 7f1610402740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f16081b6e70 con 0x559ff976f080 2024-04-23T14:54:35.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.413+0000 7f1606000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ff97649b0 0x559ff9787090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.413+0000 7f1606a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559ff9764090 0x559ff977ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.413+0000 7f1606000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ff97649b0 0x559ff9787090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56844/0 (socket says 172.21.15.28:56844) 2024-04-23T14:54:35.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.413+0000 7f1606000640 1 -- 172.21.15.28:0/3067378454 learned_addr learned my addr 172.21.15.28:0/3067378454 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:35.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.413+0000 7f1606a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559ff9764090 0x559ff977ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48800/0 (socket says 172.21.15.28:48800) 2024-04-23T14:54:35.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.413+0000 7f1607400640 1 --2- 172.21.15.28:0/3067378454 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559ff976f080 0x559ff97875d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.414+0000 7f1607400640 1 -- 172.21.15.28:0/3067378454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559ff9764090 msgr2=0x559ff977ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:35.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.414+0000 7f1607400640 1 --2- 172.21.15.28:0/3067378454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559ff9764090 0x559ff977ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.414+0000 7f1607400640 1 -- 172.21.15.28:0/3067378454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ff97649b0 msgr2=0x559ff9787090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:35.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.414+0000 7f1607400640 1 --2- 172.21.15.28:0/3067378454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ff97649b0 0x559ff9787090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.414+0000 7f1607400640 1 -- 172.21.15.28:0/3067378454 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ff976a570 con 0x559ff976f080 2024-04-23T14:54:35.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.414+0000 7f1606000640 1 --2- 172.21.15.28:0/3067378454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ff97649b0 0x559ff9787090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:54:35.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.414+0000 7f1606a00640 1 --2- 172.21.15.28:0/3067378454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559ff9764090 0x559ff977ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:54:35.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.414+0000 7f1607400640 1 --2- 172.21.15.28:0/3067378454 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559ff976f080 0x559ff97875d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1600000f80 tx=0x7f1600012510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=377c548da935d743 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:35.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.414+0000 7f15f2a00640 1 -- 172.21.15.28:0/3067378454 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1600029070 con 0x559ff976f080 2024-04-23T14:54:35.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.414+0000 7f15f2a00640 1 -- 172.21.15.28:0/3067378454 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1600012a60 con 0x559ff976f080 2024-04-23T14:54:35.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.414+0000 7f15f2a00640 1 -- 172.21.15.28:0/3067378454 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1600016040 con 0x559ff976f080 2024-04-23T14:54:35.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.415+0000 7f1610402740 1 -- 172.21.15.28:0/3067378454 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559ff976f080 msgr2=0x559ff97875d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:35.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.415+0000 7f1610402740 1 --2- 172.21.15.28:0/3067378454 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559ff976f080 0x559ff97875d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1600000f80 tx=0x7f1600012510 comp rx=0 tx=0).stop 2024-04-23T14:54:35.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.415+0000 7f1610402740 1 -- 172.21.15.28:0/3067378454 shutdown_connections 2024-04-23T14:54:35.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.415+0000 7f1610402740 1 --2- 172.21.15.28:0/3067378454 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559ff976f080 0x559ff97875d0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.415+0000 7f1610402740 1 --2- 172.21.15.28:0/3067378454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ff97649b0 0x559ff9787090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.415+0000 7f1610402740 1 --2- 172.21.15.28:0/3067378454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559ff9764090 0x559ff977ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.415+0000 7f1610402740 1 -- 172.21.15.28:0/3067378454 >> 172.21.15.28:0/3067378454 conn(0x559ff97603b0 msgr2=0x559ff976a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:35.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.415+0000 7f1610402740 1 -- 172.21.15.28:0/3067378454 shutdown_connections 2024-04-23T14:54:35.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.415+0000 7f1610402740 1 -- 172.21.15.28:0/3067378454 wait complete. 2024-04-23T14:54:35.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.416+0000 7f1610402740 1 -- start start 2024-04-23T14:54:35.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.416+0000 7f1610402740 1 Processor -- start 2024-04-23T14:54:35.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.417+0000 7f1610402740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559ff9764090 0x559ff9773310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.417+0000 7f1610402740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ff97649b0 0x559ff9773850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.417+0000 7f1610402740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559ff976f080 0x559ff97e4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.417+0000 7f1610402740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16081b7540 con 0x559ff97649b0 2024-04-23T14:54:35.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.417+0000 7f1610402740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f16081b89b0 con 0x559ff9764090 2024-04-23T14:54:35.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.417+0000 7f1610402740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f16081b7c10 con 0x559ff976f080 2024-04-23T14:54:35.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.417+0000 7f1610402740 10 client.0 _ll_register_callbacks cb 0x7f15f4003e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:54:35.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.417+0000 7f1607400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559ff976f080 0x559ff97e4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.417+0000 7f1606a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559ff9764090 0x559ff9773310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.417+0000 7f1607400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559ff976f080 0x559ff97e4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:56480/0 (socket says 172.21.15.28:56480) 2024-04-23T14:54:35.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.417+0000 7f1606000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ff97649b0 0x559ff9773850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.417+0000 7f1607400640 1 -- 172.21.15.28:0/3210790802 learned_addr learned my addr 172.21.15.28:0/3210790802 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:35.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.417+0000 7f1606a00640 1 -- 172.21.15.28:0/3210790802 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559ff976f080 msgr2=0x559ff97e4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:35.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.417+0000 7f1606a00640 1 --2- 172.21.15.28:0/3210790802 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559ff976f080 0x559ff97e4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.417+0000 7f1606a00640 1 -- 172.21.15.28:0/3210790802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ff97649b0 msgr2=0x559ff9773850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:35.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.417+0000 7f1606a00640 1 --2- 172.21.15.28:0/3210790802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ff97649b0 0x559ff9773850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.417+0000 7f1606a00640 1 -- 172.21.15.28:0/3210790802 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ff97e4e40 con 0x559ff9764090 2024-04-23T14:54:35.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.417+0000 7f1607400640 1 --2- 172.21.15.28:0/3210790802 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559ff976f080 0x559ff97e4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:54:35.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.418+0000 7f1606a00640 1 --2- 172.21.15.28:0/3210790802 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559ff9764090 0x559ff9773310 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f15f8031550 tx=0x7f15f8009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf5268517936ad7f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:35.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.418+0000 7f15e7e00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:54:35.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.418+0000 7f15e7e00640 1 -- 172.21.15.28:0/3210790802 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f8035040 con 0x559ff9764090 2024-04-23T14:54:35.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.418+0000 7f15e7e00640 1 -- 172.21.15.28:0/3210790802 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8542+0+0 (secure 0 0 0) 0x7f15f8053430 con 0x559ff9764090 2024-04-23T14:54:35.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.418+0000 7f1610402740 1 -- 172.21.15.28:0/3210790802 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1600012a60 con 0x559ff9764090 2024-04-23T14:54:35.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.419+0000 7f15e7e00640 1 -- 172.21.15.28:0/3210790802 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f15f8053630 con 0x559ff9764090 2024-04-23T14:54:35.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.419+0000 7f1610402740 10 client.26927 fetch_fsmap learned FSMap version 20 2024-04-23T14:54:35.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.419+0000 7f1610402740 1 -- 172.21.15.28:0/3210790802 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x559ff97e6150 con 0x559ff9764090 2024-04-23T14:54:35.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.419+0000 7f15e7e00640 1 -- 172.21.15.28:0/3210790802 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1600012a60 con 0x559ff9764090 2024-04-23T14:54:35.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.419+0000 7f15e7e00640 1 -- 172.21.15.28:0/3210790802 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f15f8032650 con 0x559ff9764090 2024-04-23T14:54:35.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.419+0000 7f1610402740 10 client.26927 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:54:35.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.419+0000 7f1610402740 10 client.26927 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:54:35.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.419+0000 7f1610402740 20 client.26927 populate_metadata read hostname 'smithi028' 2024-04-23T14:54:35.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.419+0000 7f1610402740 1 --2- 172.21.15.28:0/3210790802 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x559ff97e6850 0x559ff97e8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.419+0000 7f1610402740 4 client.26927 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-23T14:54:35.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.419+0000 7f1610402740 1 -- 172.21.15.28:0/3210790802 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559ff96d1700 con 0x559ff97e6850 2024-04-23T14:54:35.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.419+0000 7f1606000640 1 --2- 172.21.15.28:0/3210790802 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x559ff97e6850 0x559ff97e8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.420+0000 7f1606000640 1 --2- 172.21.15.28:0/3210790802 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x559ff97e6850 0x559ff97e8c70 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b91d91a5a431828 server_cookie=9dc699f53c338ed9 in_seq=0 out_seq=0 2024-04-23T14:54:35.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.420+0000 7f15e7e00640 10 client.26927 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:54:35.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.424+0000 7f15e7e00640 1 -- 172.21.15.28:0/3210790802 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559ff96d1700 con 0x559ff97e6850 2024-04-23T14:54:35.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.424+0000 7f15e7e00640 10 client.26927 handle_command_reply: tid=0 2024-04-23T14:54:35.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.424+0000 7f1610402740 1 client.26927 shutdown 2024-04-23T14:54:35.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.425+0000 7f1610402740 1 -- 172.21.15.28:0/3210790802 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559ff9764090 msgr2=0x559ff9773310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:35.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.425+0000 7f1610402740 1 --2- 172.21.15.28:0/3210790802 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559ff9764090 0x559ff9773310 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f15f8031550 tx=0x7f15f8009260 comp rx=0 tx=0).stop 2024-04-23T14:54:35.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.425+0000 7f1610402740 1 -- 172.21.15.28:0/3210790802 shutdown_connections 2024-04-23T14:54:35.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.425+0000 7f1610402740 1 --2- 172.21.15.28:0/3210790802 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x559ff97e6850 0x559ff97e8c70 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.425+0000 7f15e7e00640 0 client.26927 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:54:35.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.425+0000 7f1610402740 1 --2- 172.21.15.28:0/3210790802 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559ff976f080 0x559ff97e4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.425+0000 7f1610402740 1 --2- 172.21.15.28:0/3210790802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ff97649b0 0x559ff9773850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.425+0000 7f1610402740 1 --2- 172.21.15.28:0/3210790802 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559ff9764090 0x559ff9773310 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.425+0000 7f1610402740 1 -- 172.21.15.28:0/3210790802 >> 172.21.15.28:0/3210790802 conn(0x559ff97603b0 msgr2=0x559ff976bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:35.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.425+0000 7f1610402740 1 -- 172.21.15.28:0/3210790802 shutdown_connections 2024-04-23T14:54:35.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.425+0000 7f1610402740 1 -- 172.21.15.28:0/3210790802 wait complete. 2024-04-23T14:54:35.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.425+0000 7f1610402740 20 client.26927 trim_cache size 0 max 16384 2024-04-23T14:54:35.427 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:54:35.427 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-23T14:54:35.428 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "5a11b244-d076-455a-8232-0acdbd1219ef", 2024-04-23T14:54:35.428 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-23T14:54:35.428 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:54:35.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.426+0000 7f160d600640 1 -- 172.21.15.28:0/3857524818 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f15ec0a3da0 msgr2=0x7f15ec0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:35.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.426+0000 7f160d600640 1 --2- 172.21.15.28:0/3857524818 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f15ec0a3da0 0x7f15ec0a6260 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f15f80096c0 tx=0x7f15f8000bc0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.426+0000 7f160d600640 1 -- 172.21.15.28:0/3857524818 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1608117130 msgr2=0x7f16081a38c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:35.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.426+0000 7f160d600640 1 --2- 172.21.15.28:0/3857524818 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1608117130 0x7f16081a38c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f15f400e9c0 tx=0x7f15f400ee80 comp rx=0 tx=0).stop 2024-04-23T14:54:35.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.426+0000 7f160d600640 1 -- 172.21.15.28:0/3857524818 shutdown_connections 2024-04-23T14:54:35.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.426+0000 7f160d600640 1 --2- 172.21.15.28:0/3857524818 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f15ec0a3da0 0x7f15ec0a6260 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.426+0000 7f160d600640 1 --2- 172.21.15.28:0/3857524818 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16081a7c40 0x7f16081a80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.426+0000 7f160d600640 1 --2- 172.21.15.28:0/3857524818 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1608117130 0x7f16081a38c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.426+0000 7f160d600640 1 --2- 172.21.15.28:0/3857524818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16081095d0 0x7f16081a3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.426+0000 7f160d600640 1 -- 172.21.15.28:0/3857524818 >> 172.21.15.28:0/3857524818 conn(0x7f160806d570 msgr2=0x7f1608117510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:35.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.427+0000 7f160d600640 1 -- 172.21.15.28:0/3857524818 shutdown_connections 2024-04-23T14:54:35.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.427+0000 7f160d600640 1 -- 172.21.15.28:0/3857524818 wait complete. 2024-04-23T14:54:35.448 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:54:35.448 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:54:35.527 INFO:tasks.workunit.client.0.smithi028.stdout: 1 901539 8.79 MB/sec execute 411 sec latency 34.516 ms 2024-04-23T14:54:35.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.665+0000 7f3977e00640 1 -- 172.21.15.28:0/1350367802 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3978108580 msgr2=0x7f3978108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:35.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.665+0000 7f3977e00640 1 --2- 172.21.15.28:0/1350367802 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3978108580 0x7f3978108a00 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f3964009900 tx=0x7f396402a410 comp rx=0 tx=0).stop 2024-04-23T14:54:35.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.665+0000 7f3977e00640 1 -- 172.21.15.28:0/1350367802 shutdown_connections 2024-04-23T14:54:35.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.665+0000 7f3977e00640 1 --2- 172.21.15.28:0/1350367802 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3978108580 0x7f3978108a00 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.665+0000 7f3977e00640 1 --2- 172.21.15.28:0/1350367802 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3978108f40 0x7f39781153b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.665+0000 7f3977e00640 1 --2- 172.21.15.28:0/1350367802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f397810d500 0x7f397810d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.665+0000 7f3977e00640 1 -- 172.21.15.28:0/1350367802 >> 172.21.15.28:0/1350367802 conn(0x7f3978077e20 msgr2=0x7f3978106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:35.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.665+0000 7f3977e00640 1 -- 172.21.15.28:0/1350367802 shutdown_connections 2024-04-23T14:54:35.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.665+0000 7f3977e00640 1 -- 172.21.15.28:0/1350367802 wait complete. 2024-04-23T14:54:35.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.666+0000 7f3977e00640 1 Processor -- start 2024-04-23T14:54:35.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.666+0000 7f3977e00640 1 -- start start 2024-04-23T14:54:35.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.666+0000 7f3977e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3978108580 0x7f397819ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.666+0000 7f3977e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3978108f40 0x7f397819f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.666+0000 7f3977e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f397810d500 0x7f39781a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.666+0000 7f3977e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3978118900 con 0x7f3978108f40 2024-04-23T14:54:35.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.666+0000 7f3977e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3978118780 con 0x7f397810d500 2024-04-23T14:54:35.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.666+0000 7f3977e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3978118a80 con 0x7f3978108580 2024-04-23T14:54:35.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.667+0000 7f3976a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3978108580 0x7f397819ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.667+0000 7f3977400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f397810d500 0x7f39781a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.667+0000 7f3976000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3978108f40 0x7f397819f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.667+0000 7f3976a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3978108580 0x7f397819ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:56494/0 (socket says 172.21.15.28:56494) 2024-04-23T14:54:35.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.667+0000 7f3976a00640 1 -- 172.21.15.28:0/1193905597 learned_addr learned my addr 172.21.15.28:0/1193905597 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:35.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.667+0000 7f3976a00640 1 -- 172.21.15.28:0/1193905597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f397810d500 msgr2=0x7f39781a35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:35.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.667+0000 7f3976a00640 1 --2- 172.21.15.28:0/1193905597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f397810d500 0x7f39781a35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.667+0000 7f3976a00640 1 -- 172.21.15.28:0/1193905597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3978108f40 msgr2=0x7f397819f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:35.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.667+0000 7f3976a00640 1 --2- 172.21.15.28:0/1193905597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3978108f40 0x7f397819f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.667+0000 7f3976a00640 1 -- 172.21.15.28:0/1193905597 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39781a3d10 con 0x7f3978108580 2024-04-23T14:54:35.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.667+0000 7f3977400640 1 --2- 172.21.15.28:0/1193905597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f397810d500 0x7f39781a35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:54:35.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.668+0000 7f3976a00640 1 --2- 172.21.15.28:0/1193905597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3978108580 0x7f397819ed20 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f396800e9c0 tx=0x7f396800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:35.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.668+0000 7f397ca00640 1 -- 172.21.15.28:0/1193905597 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f396800cde0 con 0x7f3978108580 2024-04-23T14:54:35.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.668+0000 7f397ca00640 1 -- 172.21.15.28:0/1193905597 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3968010910 con 0x7f3978108580 2024-04-23T14:54:35.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.668+0000 7f397ca00640 1 -- 172.21.15.28:0/1193905597 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3968018a00 con 0x7f3978108580 2024-04-23T14:54:35.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.668+0000 7f3977e00640 1 -- 172.21.15.28:0/1193905597 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39781a4000 con 0x7f3978108580 2024-04-23T14:54:35.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.668+0000 7f3977e00640 1 -- 172.21.15.28:0/1193905597 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39781ab890 con 0x7f3978108580 2024-04-23T14:54:35.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.672+0000 7f3980b68740 1 Processor -- start 2024-04-23T14:54:35.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.672+0000 7f3980b68740 1 -- start start 2024-04-23T14:54:35.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.672+0000 7f397ca00640 1 -- 172.21.15.28:0/1193905597 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f396801f020 con 0x7f3978108580 2024-04-23T14:54:35.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.672+0000 7f3980b68740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623c85f2a30 0x5623c85f2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.672+0000 7f3980b68740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623c85f3350 0x5623c85fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.673+0000 7f3980b68740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623c85fda20 0x5623c85ffe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.673+0000 7f3980b68740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5623c85581f0 con 0x5623c85fda20 2024-04-23T14:54:35.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.673+0000 7f3980b68740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5623c84fe350 con 0x5623c85f2a30 2024-04-23T14:54:35.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.673+0000 7f3980b68740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5623c85541b0 con 0x5623c85f3350 2024-04-23T14:54:35.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.673+0000 7f3977400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623c85f2a30 0x5623c85f2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.673+0000 7f3976a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623c85f3350 0x5623c85fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.673+0000 7f3977400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623c85f2a30 0x5623c85f2e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48838/0 (socket says 172.21.15.28:48838) 2024-04-23T14:54:35.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.673+0000 7f3977400640 1 -- 172.21.15.28:0/3372036544 learned_addr learned my addr 172.21.15.28:0/3372036544 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:35.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.673+0000 7f3976000640 1 --2- 172.21.15.28:0/3372036544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623c85fda20 0x5623c85ffe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.673+0000 7f3977400640 1 -- 172.21.15.28:0/3372036544 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623c85f3350 msgr2=0x5623c85fd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:35.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.673+0000 7f3977400640 1 --2- 172.21.15.28:0/3372036544 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623c85f3350 0x5623c85fd4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.673+0000 7f3977400640 1 -- 172.21.15.28:0/3372036544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623c85fda20 msgr2=0x5623c85ffe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:35.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.673+0000 7f3977400640 1 --2- 172.21.15.28:0/3372036544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623c85fda20 0x5623c85ffe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.673+0000 7f3977400640 1 -- 172.21.15.28:0/3372036544 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623c8552f50 con 0x5623c85f2a30 2024-04-23T14:54:35.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.674+0000 7f3977400640 1 --2- 172.21.15.28:0/3372036544 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623c85f2a30 0x5623c85f2e10 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f39700117f0 tx=0x7f3970011cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=107512982a5b7a78 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:35.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.674+0000 7f3963e00640 1 -- 172.21.15.28:0/3372036544 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39700112f0 con 0x5623c85f2a30 2024-04-23T14:54:35.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.674+0000 7f3963e00640 1 -- 172.21.15.28:0/3372036544 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f397000fe80 con 0x5623c85f2a30 2024-04-23T14:54:35.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.674+0000 7f3963e00640 1 -- 172.21.15.28:0/3372036544 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39700193e0 con 0x5623c85f2a30 2024-04-23T14:54:35.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.674+0000 7f397ca00640 1 --2- 172.21.15.28:0/1193905597 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f395c0a3da0 0x7f395c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.675+0000 7f397ca00640 1 -- 172.21.15.28:0/1193905597 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8542+0+0 (secure 0 0 0) 0x7f39680d0950 con 0x7f3978108580 2024-04-23T14:54:35.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.675+0000 7f3976a00640 1 --2- 172.21.15.28:0/1193905597 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f395c0a3da0 0x7f395c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.675+0000 7f3980b68740 1 -- 172.21.15.28:0/3372036544 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623c85f2a30 msgr2=0x5623c85f2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:35.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.675+0000 7f3980b68740 1 --2- 172.21.15.28:0/3372036544 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623c85f2a30 0x5623c85f2e10 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f39700117f0 tx=0x7f3970011cb0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.675+0000 7f3980b68740 1 -- 172.21.15.28:0/3372036544 shutdown_connections 2024-04-23T14:54:35.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.675+0000 7f3980b68740 1 --2- 172.21.15.28:0/3372036544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623c85fda20 0x5623c85ffe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.675+0000 7f3980b68740 1 --2- 172.21.15.28:0/3372036544 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623c85f3350 0x5623c85fd4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.675+0000 7f3980b68740 1 --2- 172.21.15.28:0/3372036544 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623c85f2a30 0x5623c85f2e10 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.675+0000 7f3980b68740 1 -- 172.21.15.28:0/3372036544 >> 172.21.15.28:0/3372036544 conn(0x5623c85eed50 msgr2=0x5623c85ef130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:35.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.676+0000 7f3980b68740 1 -- 172.21.15.28:0/3372036544 shutdown_connections 2024-04-23T14:54:35.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.676+0000 7f3980b68740 1 -- 172.21.15.28:0/3372036544 wait complete. 2024-04-23T14:54:35.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.676+0000 7f3976a00640 1 --2- 172.21.15.28:0/1193905597 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f395c0a3da0 0x7f395c0a6260 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f39680ce140 tx=0x7f39680d6550 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:35.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.677+0000 7f3980b68740 1 -- start start 2024-04-23T14:54:35.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.677+0000 7f3980b68740 1 Processor -- start 2024-04-23T14:54:35.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.677+0000 7f3980b68740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623c85f2a30 0x5623c85faa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.677+0000 7f3980b68740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623c85f3350 0x5623c85faf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.677+0000 7f3980b68740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623c85fda20 0x5623c85fb480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.677+0000 7f3980b68740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5623c8600810 con 0x5623c85fda20 2024-04-23T14:54:35.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.677+0000 7f3980b68740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5623c8511690 con 0x5623c85f2a30 2024-04-23T14:54:35.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.677+0000 7f3980b68740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5623c8600990 con 0x5623c85f3350 2024-04-23T14:54:35.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.678+0000 7f3980b68740 10 client.0 _ll_register_callbacks cb 0x5623c854d650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:54:35.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.678+0000 7f3977400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623c85f2a30 0x5623c85faa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.678+0000 7f3976000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623c85f3350 0x5623c85faf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.678+0000 7f3977400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623c85f2a30 0x5623c85faa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48852/0 (socket says 172.21.15.28:48852) 2024-04-23T14:54:35.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.678+0000 7f3977400640 1 -- 172.21.15.28:0/414166244 learned_addr learned my addr 172.21.15.28:0/414166244 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:35.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.678+0000 7f3977400640 1 --2- 172.21.15.28:0/414166244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623c85fda20 0x5623c85fb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.678+0000 7f3976000640 1 -- 172.21.15.28:0/414166244 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623c85f2a30 msgr2=0x5623c85faa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:35.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.678+0000 7f3976000640 1 --2- 172.21.15.28:0/414166244 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623c85f2a30 0x5623c85faa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.678+0000 7f3976000640 1 -- 172.21.15.28:0/414166244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623c85fda20 msgr2=0x5623c85fb480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:35.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.678+0000 7f3976000640 1 --2- 172.21.15.28:0/414166244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623c85fda20 0x5623c85fb480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.678+0000 7f3976000640 1 -- 172.21.15.28:0/414166244 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623c8510920 con 0x5623c85f3350 2024-04-23T14:54:35.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.679+0000 7f3976000640 1 --2- 172.21.15.28:0/414166244 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623c85f3350 0x5623c85faf40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f396402a930 tx=0x7f39640043b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7826b31388d481a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:35.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.679+0000 7f3960c00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:54:35.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.679+0000 7f3960c00640 1 -- 172.21.15.28:0/414166244 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3964035040 con 0x5623c85f3350 2024-04-23T14:54:35.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.679+0000 7f3960c00640 1 -- 172.21.15.28:0/414166244 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8542+0+0 (secure 0 0 0) 0x7f3964005450 con 0x5623c85f3350 2024-04-23T14:54:35.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.679+0000 7f3980b68740 1 -- 172.21.15.28:0/414166244 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5623c8565c10 con 0x5623c85f3350 2024-04-23T14:54:35.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.679+0000 7f3960c00640 1 -- 172.21.15.28:0/414166244 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f3964005650 con 0x5623c85f3350 2024-04-23T14:54:35.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.679+0000 7f3980b68740 10 client.26923 fetch_fsmap learned FSMap version 20 2024-04-23T14:54:35.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.679+0000 7f3980b68740 1 -- 172.21.15.28:0/414166244 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5623c83347d0 con 0x5623c85f3350 2024-04-23T14:54:35.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.679+0000 7f3960c00640 1 -- 172.21.15.28:0/414166244 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5623c8565c10 con 0x5623c85f3350 2024-04-23T14:54:35.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.680+0000 7f3960c00640 1 -- 172.21.15.28:0/414166244 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f39640039d0 con 0x5623c85f3350 2024-04-23T14:54:35.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.680+0000 7f3980b68740 10 client.26923 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:54:35.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.680+0000 7f3980b68740 10 client.26923 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:54:35.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.680+0000 7f3980b68740 20 client.26923 populate_metadata read hostname 'smithi028' 2024-04-23T14:54:35.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.680+0000 7f3980b68740 1 --2- 172.21.15.28:0/414166244 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5623c860ddf0 0x5623c8610200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.680+0000 7f3980b68740 4 client.26923 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:54:35.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.680+0000 7f3980b68740 1 -- 172.21.15.28:0/414166244 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5623c855fa80 con 0x5623c860ddf0 2024-04-23T14:54:35.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.681+0000 7f3977400640 1 --2- 172.21.15.28:0/414166244 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5623c860ddf0 0x5623c8610200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.681+0000 7f3977400640 1 --2- 172.21.15.28:0/414166244 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5623c860ddf0 0x5623c8610200 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9fd4a376034a683 server_cookie=95022a862e6582c5 in_seq=0 out_seq=0 2024-04-23T14:54:35.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.681+0000 7f3960c00640 10 client.26923 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:54:35.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.682+0000 7f3960c00640 1 -- 172.21.15.28:0/414166244 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5623c855fa80 con 0x5623c860ddf0 2024-04-23T14:54:35.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.682+0000 7f3960c00640 10 client.26923 handle_command_reply: tid=0 2024-04-23T14:54:35.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.682+0000 7f3980b68740 1 client.26923 shutdown 2024-04-23T14:54:35.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.682+0000 7f3980b68740 1 -- 172.21.15.28:0/414166244 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623c85f3350 msgr2=0x5623c85faf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:35.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.682+0000 7f3980b68740 1 --2- 172.21.15.28:0/414166244 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623c85f3350 0x5623c85faf40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f396402a930 tx=0x7f39640043b0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.682+0000 7f3980b68740 1 -- 172.21.15.28:0/414166244 shutdown_connections 2024-04-23T14:54:35.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.682+0000 7f3980b68740 1 --2- 172.21.15.28:0/414166244 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5623c860ddf0 0x5623c8610200 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.682+0000 7f3980b68740 1 --2- 172.21.15.28:0/414166244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623c85fda20 0x5623c85fb480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.682+0000 7f3980b68740 1 --2- 172.21.15.28:0/414166244 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623c85f3350 0x5623c85faf40 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.682+0000 7f3980b68740 1 --2- 172.21.15.28:0/414166244 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623c85f2a30 0x5623c85faa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.682+0000 7f3960c00640 0 client.26923 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:54:35.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.682+0000 7f3980b68740 1 -- 172.21.15.28:0/414166244 >> 172.21.15.28:0/414166244 conn(0x5623c85eed50 msgr2=0x5623c85f4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:35.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.682+0000 7f3980b68740 1 -- 172.21.15.28:0/414166244 shutdown_connections 2024-04-23T14:54:35.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.682+0000 7f3980b68740 1 -- 172.21.15.28:0/414166244 wait complete. 2024-04-23T14:54:35.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.683+0000 7f3980b68740 20 client.26923 trim_cache size 0 max 16384 2024-04-23T14:54:35.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.706+0000 7f3980b68740 1 Processor -- start 2024-04-23T14:54:35.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.706+0000 7f3980b68740 1 -- start start 2024-04-23T14:54:35.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.706+0000 7f3980b68740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623c85f2a30 0x5623c86096d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.706+0000 7f3980b68740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623c85f3350 0x5623c85fb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.706+0000 7f3980b68740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623c85fda20 0x5623c8615ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.706+0000 7f3980b68740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f39781b2160 con 0x5623c85f3350 2024-04-23T14:54:35.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.706+0000 7f3980b68740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f39781b49c0 con 0x5623c85f2a30 2024-04-23T14:54:35.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.706+0000 7f3980b68740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f39781b27b0 con 0x5623c85fda20 2024-04-23T14:54:35.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.707+0000 7f3977400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623c85f2a30 0x5623c86096d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.707+0000 7f3977400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623c85f2a30 0x5623c86096d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48854/0 (socket says 172.21.15.28:48854) 2024-04-23T14:54:35.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.707+0000 7f3977400640 1 -- 172.21.15.28:0/943773645 learned_addr learned my addr 172.21.15.28:0/943773645 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:35.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.707+0000 7f3977400640 1 --2- 172.21.15.28:0/943773645 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623c85fda20 0x5623c8615ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.707+0000 7f3976000640 1 --2- 172.21.15.28:0/943773645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623c85f3350 0x5623c85fb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.707+0000 7f3977400640 1 -- 172.21.15.28:0/943773645 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623c85fda20 msgr2=0x5623c8615ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:35.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.707+0000 7f3977400640 1 --2- 172.21.15.28:0/943773645 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623c85fda20 0x5623c8615ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.707+0000 7f3977400640 1 -- 172.21.15.28:0/943773645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623c85f3350 msgr2=0x5623c85fb2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:35.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.707+0000 7f3977400640 1 --2- 172.21.15.28:0/943773645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623c85f3350 0x5623c85fb2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.707+0000 7f3977400640 1 -- 172.21.15.28:0/943773645 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623c826a3f0 con 0x5623c85f2a30 2024-04-23T14:54:35.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.707+0000 7f3976000640 1 --2- 172.21.15.28:0/943773645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623c85f3350 0x5623c85fb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:54:35.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.707+0000 7f3977400640 1 --2- 172.21.15.28:0/943773645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623c85f2a30 0x5623c86096d0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f3970009980 tx=0x7f3970000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fbd637796a6917e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:35.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.708+0000 7f3963400640 1 -- 172.21.15.28:0/943773645 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3970010430 con 0x5623c85f2a30 2024-04-23T14:54:35.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.708+0000 7f3963400640 1 -- 172.21.15.28:0/943773645 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39700105d0 con 0x5623c85f2a30 2024-04-23T14:54:35.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.708+0000 7f3963400640 1 -- 172.21.15.28:0/943773645 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3970004b20 con 0x5623c85f2a30 2024-04-23T14:54:35.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.708+0000 7f3980b68740 1 -- 172.21.15.28:0/943773645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623c85f2a30 msgr2=0x5623c86096d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:35.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.708+0000 7f3980b68740 1 --2- 172.21.15.28:0/943773645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623c85f2a30 0x5623c86096d0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f3970009980 tx=0x7f3970000f40 comp rx=0 tx=0).stop 2024-04-23T14:54:35.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.709+0000 7f3980b68740 1 -- 172.21.15.28:0/943773645 shutdown_connections 2024-04-23T14:54:35.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.709+0000 7f3980b68740 1 --2- 172.21.15.28:0/943773645 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623c85fda20 0x5623c8615ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.709+0000 7f3980b68740 1 --2- 172.21.15.28:0/943773645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623c85f3350 0x5623c85fb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.709+0000 7f3980b68740 1 --2- 172.21.15.28:0/943773645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623c85f2a30 0x5623c86096d0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.709+0000 7f3980b68740 1 -- 172.21.15.28:0/943773645 >> 172.21.15.28:0/943773645 conn(0x5623c85eed50 msgr2=0x5623c85f4a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:35.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.709+0000 7f3980b68740 1 -- 172.21.15.28:0/943773645 shutdown_connections 2024-04-23T14:54:35.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.709+0000 7f3980b68740 1 -- 172.21.15.28:0/943773645 wait complete. 2024-04-23T14:54:35.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.709+0000 7f3980b68740 1 -- start start 2024-04-23T14:54:35.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.709+0000 7f3980b68740 1 Processor -- start 2024-04-23T14:54:35.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.710+0000 7f3980b68740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623c85f2a30 0x5623c8605ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.710+0000 7f3980b68740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623c85f3350 0x5623c85f72f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.710+0000 7f3980b68740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623c85fda20 0x5623c85f7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.710+0000 7f3980b68740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f39781b2e80 con 0x5623c85fda20 2024-04-23T14:54:35.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.710+0000 7f3980b68740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f39781b42f0 con 0x5623c85f2a30 2024-04-23T14:54:35.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.710+0000 7f3980b68740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f39781b3550 con 0x5623c85f3350 2024-04-23T14:54:35.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.710+0000 7f3980b68740 10 client.0 _ll_register_callbacks cb 0x7f3964009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:54:35.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.710+0000 7f3976000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623c85f3350 0x5623c85f72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.710+0000 7f3976000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623c85f3350 0x5623c85f72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:56554/0 (socket says 172.21.15.28:56554) 2024-04-23T14:54:35.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.710+0000 7f3976000640 1 -- 172.21.15.28:0/1846175919 learned_addr learned my addr 172.21.15.28:0/1846175919 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:54:35.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.710+0000 7f3977400640 1 --2- 172.21.15.28:0/1846175919 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623c85f2a30 0x5623c8605ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.710+0000 7f3977400640 1 --2- 172.21.15.28:0/1846175919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623c85fda20 0x5623c85f7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.710+0000 7f3976000640 1 -- 172.21.15.28:0/1846175919 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623c85f2a30 msgr2=0x5623c8605ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:35.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.710+0000 7f3976000640 1 --2- 172.21.15.28:0/1846175919 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623c85f2a30 0x5623c8605ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.710+0000 7f3976000640 1 -- 172.21.15.28:0/1846175919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623c85fda20 msgr2=0x5623c85f7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:54:35.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.710+0000 7f3976000640 1 --2- 172.21.15.28:0/1846175919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623c85fda20 0x5623c85f7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.710+0000 7f3976000640 1 -- 172.21.15.28:0/1846175919 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623c822a0f0 con 0x5623c85f3350 2024-04-23T14:54:35.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.710+0000 7f3977400640 1 --2- 172.21.15.28:0/1846175919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623c85fda20 0x5623c85f7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:54:35.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.711+0000 7f3976000640 1 --2- 172.21.15.28:0/1846175919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623c85f3350 0x5623c85f72f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3964000c00 tx=0x7f39640054e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=478f8b88a3e67ef5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:54:35.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.711+0000 7f3960c00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:54:35.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.711+0000 7f3960c00640 1 -- 172.21.15.28:0/1846175919 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3964035070 con 0x5623c85f3350 2024-04-23T14:54:35.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.711+0000 7f3980b68740 1 -- 172.21.15.28:0/1846175919 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39700105d0 con 0x5623c85f3350 2024-04-23T14:54:35.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.711+0000 7f3960c00640 1 -- 172.21.15.28:0/1846175919 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8542+0+0 (secure 0 0 0) 0x7f3964031d40 con 0x5623c85f3350 2024-04-23T14:54:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.711+0000 7f3960c00640 1 -- 172.21.15.28:0/1846175919 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f3964052070 con 0x5623c85f3350 2024-04-23T14:54:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.711+0000 7f3980b68740 10 client.26929 fetch_fsmap learned FSMap version 20 2024-04-23T14:54:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.711+0000 7f3980b68740 1 -- 172.21.15.28:0/1846175919 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5623c851d810 con 0x5623c85f3350 2024-04-23T14:54:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.712+0000 7f3960c00640 1 -- 172.21.15.28:0/1846175919 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39700105d0 con 0x5623c85f3350 2024-04-23T14:54:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.712+0000 7f3960c00640 1 -- 172.21.15.28:0/1846175919 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f39640047e0 con 0x5623c85f3350 2024-04-23T14:54:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.712+0000 7f3980b68740 10 client.26929 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:54:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.712+0000 7f3980b68740 10 client.26929 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:54:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.712+0000 7f3980b68740 20 client.26929 populate_metadata read hostname 'smithi028' 2024-04-23T14:54:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.712+0000 7f3980b68740 1 --2- 172.21.15.28:0/1846175919 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5623c8674520 0x5623c8676910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:54:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.712+0000 7f3980b68740 4 client.26929 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:54:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.712+0000 7f3980b68740 1 -- 172.21.15.28:0/1846175919 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5623c84db360 con 0x5623c8674520 2024-04-23T14:54:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.712+0000 7f3977400640 1 --2- 172.21.15.28:0/1846175919 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5623c8674520 0x5623c8676910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:54:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.713+0000 7f3977400640 1 --2- 172.21.15.28:0/1846175919 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5623c8674520 0x5623c8676910 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8aecb70eca1bafbc server_cookie=1ca83931dde57d80 in_seq=0 out_seq=0 2024-04-23T14:54:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.713+0000 7f3960c00640 10 client.26929 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:54:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.714+0000 7f3960c00640 1 -- 172.21.15.28:0/1846175919 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5623c84db360 con 0x5623c8674520 2024-04-23T14:54:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.714+0000 7f3960c00640 10 client.26929 handle_command_reply: tid=0 2024-04-23T14:54:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.714+0000 7f3980b68740 1 client.26929 shutdown 2024-04-23T14:54:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.714+0000 7f3980b68740 1 -- 172.21.15.28:0/1846175919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623c85f3350 msgr2=0x5623c85f72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.714+0000 7f3980b68740 1 --2- 172.21.15.28:0/1846175919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623c85f3350 0x5623c85f72f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3964000c00 tx=0x7f39640054e0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.714+0000 7f3980b68740 1 -- 172.21.15.28:0/1846175919 shutdown_connections 2024-04-23T14:54:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.714+0000 7f3980b68740 1 --2- 172.21.15.28:0/1846175919 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5623c8674520 0x5623c8676910 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.714+0000 7f3980b68740 1 --2- 172.21.15.28:0/1846175919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623c85fda20 0x5623c85f7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.714+0000 7f3980b68740 1 --2- 172.21.15.28:0/1846175919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5623c85f3350 0x5623c85f72f0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.714+0000 7f3980b68740 1 --2- 172.21.15.28:0/1846175919 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623c85f2a30 0x5623c8605ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.714+0000 7f3980b68740 1 -- 172.21.15.28:0/1846175919 >> 172.21.15.28:0/1846175919 conn(0x5623c85eed50 msgr2=0x5623c85ed740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.714+0000 7f3960c00640 0 client.26929 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:54:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.714+0000 7f3980b68740 1 -- 172.21.15.28:0/1846175919 shutdown_connections 2024-04-23T14:54:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.714+0000 7f3980b68740 1 -- 172.21.15.28:0/1846175919 wait complete. 2024-04-23T14:54:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.715+0000 7f3980b68740 20 client.26929 trim_cache size 0 max 16384 2024-04-23T14:54:35.719 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:54:35.719 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-23T14:54:35.719 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-23T14:54:35.719 INFO:teuthology.orchestra.run.smithi028.stdout: "5a11b244-d076-455a-8232-0acdbd1219ef": { 2024-04-23T14:54:35.719 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-23T14:54:35.719 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "5a11b244-d076-455a-8232-0acdbd1219ef", 2024-04-23T14:54:35.719 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-23T14:54:35.719 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:54:35.719 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:54:35.719 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:54:35.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.715+0000 7f3977e00640 1 -- 172.21.15.28:0/1193905597 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f395c0a3da0 msgr2=0x7f395c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:35.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.715+0000 7f3977e00640 1 --2- 172.21.15.28:0/1193905597 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f395c0a3da0 0x7f395c0a6260 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f39680ce140 tx=0x7f39680d6550 comp rx=0 tx=0).stop 2024-04-23T14:54:35.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.715+0000 7f3977e00640 1 -- 172.21.15.28:0/1193905597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3978108580 msgr2=0x7f397819ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:54:35.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.715+0000 7f3977e00640 1 --2- 172.21.15.28:0/1193905597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3978108580 0x7f397819ed20 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f396800e9c0 tx=0x7f396800ee80 comp rx=0 tx=0).stop 2024-04-23T14:54:35.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.716+0000 7f3977e00640 1 -- 172.21.15.28:0/1193905597 shutdown_connections 2024-04-23T14:54:35.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.716+0000 7f3977e00640 1 --2- 172.21.15.28:0/1193905597 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f395c0a3da0 0x7f395c0a6260 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.716+0000 7f3977e00640 1 --2- 172.21.15.28:0/1193905597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f397810d500 0x7f39781a35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.716+0000 7f3977e00640 1 --2- 172.21.15.28:0/1193905597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3978108f40 0x7f397819f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.716+0000 7f3977e00640 1 --2- 172.21.15.28:0/1193905597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3978108580 0x7f397819ed20 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:54:35.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.716+0000 7f3977e00640 1 -- 172.21.15.28:0/1193905597 >> 172.21.15.28:0/1193905597 conn(0x7f3978077e20 msgr2=0x7f3978078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:54:35.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.716+0000 7f3977e00640 1 -- 172.21.15.28:0/1193905597 shutdown_connections 2024-04-23T14:54:35.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:54:35.716+0000 7f3977e00640 1 -- 172.21.15.28:0/1193905597 wait complete. 2024-04-23T14:54:35.736 INFO:tasks.cephfs.filesystem:scrub status for tag:5a11b244-d076-455a-8232-0acdbd1219ef - {'path': '/', 'tag': '5a11b244-d076-455a-8232-0acdbd1219ef', 'options': 'recursive,force'} 2024-04-23T14:54:36.527 INFO:tasks.workunit.client.0.smithi028.stdout: 1 902722 8.78 MB/sec execute 412 sec latency 22.196 ms 2024-04-23T14:54:37.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:36 smithi038 ceph-mon[32348]: scrub queued for path: / 2024-04-23T14:54:37.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:36 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:54:37.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:36 smithi038 ceph-mon[32348]: scrub summary: active paths [/] 2024-04-23T14:54:37.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:36 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:54:37.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:36 smithi038 ceph-mon[32348]: pgmap v328: 97 pgs: 97 active+clean; 538 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 133 op/s 2024-04-23T14:54:37.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:36 smithi028 ceph-mon[27810]: scrub queued for path: / 2024-04-23T14:54:37.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:36 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:54:37.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:36 smithi028 ceph-mon[27810]: scrub summary: active paths [/] 2024-04-23T14:54:37.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:36 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:54:37.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:36 smithi028 ceph-mon[27810]: pgmap v328: 97 pgs: 97 active+clean; 538 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 133 op/s 2024-04-23T14:54:37.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:36 smithi105 ceph-mon[32555]: scrub queued for path: / 2024-04-23T14:54:37.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:36 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:54:37.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:36 smithi105 ceph-mon[32555]: scrub summary: active paths [/] 2024-04-23T14:54:37.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:36 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:54:37.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:36 smithi105 ceph-mon[32555]: pgmap v328: 97 pgs: 97 active+clean; 538 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 133 op/s 2024-04-23T14:54:37.527 INFO:tasks.workunit.client.0.smithi028.stdout: 1 904227 8.78 MB/sec execute 413 sec latency 23.260 ms 2024-04-23T14:54:38.528 INFO:tasks.workunit.client.0.smithi028.stdout: 1 905653 8.78 MB/sec execute 414 sec latency 52.513 ms 2024-04-23T14:54:39.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:38 smithi038 ceph-mon[32348]: pgmap v329: 97 pgs: 97 active+clean; 547 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.9 MiB/s wr, 198 op/s 2024-04-23T14:54:39.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:38 smithi028 ceph-mon[27810]: pgmap v329: 97 pgs: 97 active+clean; 547 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.9 MiB/s wr, 198 op/s 2024-04-23T14:54:39.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:38 smithi105 ceph-mon[32555]: pgmap v329: 97 pgs: 97 active+clean; 547 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.9 MiB/s wr, 198 op/s 2024-04-23T14:54:39.528 INFO:tasks.workunit.client.0.smithi028.stdout: 1 906982 8.77 MB/sec execute 415 sec latency 23.821 ms 2024-04-23T14:54:40.528 INFO:tasks.workunit.client.0.smithi028.stdout: 1 908999 8.78 MB/sec execute 416 sec latency 20.627 ms 2024-04-23T14:54:41.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:40 smithi038 ceph-mon[32348]: pgmap v330: 97 pgs: 97 active+clean; 547 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.9 MiB/s wr, 135 op/s 2024-04-23T14:54:41.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:40 smithi038 ceph-mon[32348]: scrub summary: idle 2024-04-23T14:54:41.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:40 smithi028 ceph-mon[27810]: pgmap v330: 97 pgs: 97 active+clean; 547 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.9 MiB/s wr, 135 op/s 2024-04-23T14:54:41.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:40 smithi028 ceph-mon[27810]: scrub summary: idle 2024-04-23T14:54:41.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:40 smithi105 ceph-mon[32555]: pgmap v330: 97 pgs: 97 active+clean; 547 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.9 MiB/s wr, 135 op/s 2024-04-23T14:54:41.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:40 smithi105 ceph-mon[32555]: scrub summary: idle 2024-04-23T14:54:41.528 INFO:tasks.workunit.client.0.smithi028.stdout: 1 910287 8.78 MB/sec execute 417 sec latency 19.471 ms 2024-04-23T14:54:42.528 INFO:tasks.workunit.client.0.smithi028.stdout: 1 912209 8.78 MB/sec execute 418 sec latency 16.416 ms 2024-04-23T14:54:43.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:42 smithi038 ceph-mon[32348]: pgmap v331: 97 pgs: 97 active+clean; 547 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.9 MiB/s wr, 135 op/s 2024-04-23T14:54:43.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:42 smithi028 ceph-mon[27810]: pgmap v331: 97 pgs: 97 active+clean; 547 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.9 MiB/s wr, 135 op/s 2024-04-23T14:54:43.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:42 smithi105 ceph-mon[32555]: pgmap v331: 97 pgs: 97 active+clean; 547 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.9 MiB/s wr, 135 op/s 2024-04-23T14:54:43.528 INFO:tasks.workunit.client.0.smithi028.stdout: 1 913284 8.77 MB/sec execute 419 sec latency 45.497 ms 2024-04-23T14:54:44.528 INFO:tasks.workunit.client.0.smithi028.stdout: 1 914998 8.77 MB/sec execute 420 sec latency 23.881 ms 2024-04-23T14:54:45.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:44 smithi038 ceph-mon[32348]: pgmap v332: 97 pgs: 97 active+clean; 555 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.2 MiB/s wr, 201 op/s 2024-04-23T14:54:45.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:44 smithi028 ceph-mon[27810]: pgmap v332: 97 pgs: 97 active+clean; 555 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.2 MiB/s wr, 201 op/s 2024-04-23T14:54:45.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:44 smithi105 ceph-mon[32555]: pgmap v332: 97 pgs: 97 active+clean; 555 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.2 MiB/s wr, 201 op/s 2024-04-23T14:54:45.528 INFO:tasks.workunit.client.0.smithi028.stdout: 1 916435 8.77 MB/sec execute 421 sec latency 70.340 ms 2024-04-23T14:54:46.529 INFO:tasks.workunit.client.0.smithi028.stdout: 1 917335 8.80 MB/sec execute 422 sec latency 70.592 ms 2024-04-23T14:54:47.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:46 smithi038 ceph-mon[32348]: pgmap v333: 97 pgs: 97 active+clean; 555 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 132 op/s 2024-04-23T14:54:47.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:46 smithi028 ceph-mon[27810]: pgmap v333: 97 pgs: 97 active+clean; 555 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 132 op/s 2024-04-23T14:54:47.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:46 smithi105 ceph-mon[32555]: pgmap v333: 97 pgs: 97 active+clean; 555 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 132 op/s 2024-04-23T14:54:47.529 INFO:tasks.workunit.client.0.smithi028.stdout: 1 918886 8.80 MB/sec execute 423 sec latency 22.955 ms 2024-04-23T14:54:48.529 INFO:tasks.workunit.client.0.smithi028.stdout: 1 920361 8.80 MB/sec execute 424 sec latency 75.808 ms 2024-04-23T14:54:49.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:48 smithi038 ceph-mon[32348]: pgmap v334: 97 pgs: 97 active+clean; 563 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 195 op/s 2024-04-23T14:54:49.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:48 smithi028 ceph-mon[27810]: pgmap v334: 97 pgs: 97 active+clean; 563 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 195 op/s 2024-04-23T14:54:49.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:48 smithi105 ceph-mon[32555]: pgmap v334: 97 pgs: 97 active+clean; 563 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 195 op/s 2024-04-23T14:54:49.529 INFO:tasks.workunit.client.0.smithi028.stdout: 1 921250 8.79 MB/sec execute 425 sec latency 23.325 ms 2024-04-23T14:54:50.529 INFO:tasks.workunit.client.0.smithi028.stdout: 1 922837 8.78 MB/sec execute 426 sec latency 24.833 ms 2024-04-23T14:54:51.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:50 smithi038 ceph-mon[32348]: pgmap v335: 97 pgs: 97 active+clean; 563 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 130 op/s 2024-04-23T14:54:51.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:50 smithi028 ceph-mon[27810]: pgmap v335: 97 pgs: 97 active+clean; 563 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 130 op/s 2024-04-23T14:54:51.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:50 smithi105 ceph-mon[32555]: pgmap v335: 97 pgs: 97 active+clean; 563 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 130 op/s 2024-04-23T14:54:51.529 INFO:tasks.workunit.client.0.smithi028.stdout: 1 924531 8.79 MB/sec execute 427 sec latency 14.787 ms 2024-04-23T14:54:52.529 INFO:tasks.workunit.client.0.smithi028.stdout: 1 926387 8.79 MB/sec execute 428 sec latency 18.884 ms 2024-04-23T14:54:53.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:52 smithi038 ceph-mon[32348]: pgmap v336: 97 pgs: 97 active+clean; 563 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 130 op/s 2024-04-23T14:54:53.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:52 smithi028 ceph-mon[27810]: pgmap v336: 97 pgs: 97 active+clean; 563 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 130 op/s 2024-04-23T14:54:53.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:52 smithi105 ceph-mon[32555]: pgmap v336: 97 pgs: 97 active+clean; 563 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 130 op/s 2024-04-23T14:54:53.529 INFO:tasks.workunit.client.0.smithi028.stdout: 1 928003 8.79 MB/sec execute 429 sec latency 45.236 ms 2024-04-23T14:54:54.530 INFO:tasks.workunit.client.0.smithi028.stdout: 1 929487 8.79 MB/sec execute 430 sec latency 49.684 ms 2024-04-23T14:54:55.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:54 smithi038 ceph-mon[32348]: pgmap v337: 97 pgs: 97 active+clean; 573 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.1 MiB/s wr, 200 op/s 2024-04-23T14:54:55.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:54 smithi028 ceph-mon[27810]: pgmap v337: 97 pgs: 97 active+clean; 573 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.1 MiB/s wr, 200 op/s 2024-04-23T14:54:55.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:54 smithi105 ceph-mon[32555]: pgmap v337: 97 pgs: 97 active+clean; 573 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.1 MiB/s wr, 200 op/s 2024-04-23T14:54:55.530 INFO:tasks.workunit.client.0.smithi028.stdout: 1 931155 8.80 MB/sec execute 431 sec latency 22.555 ms 2024-04-23T14:54:56.530 INFO:tasks.workunit.client.0.smithi028.stdout: 1 932105 8.78 MB/sec execute 432 sec latency 20.731 ms 2024-04-23T14:54:57.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:56 smithi105 ceph-mon[32555]: pgmap v338: 97 pgs: 97 active+clean; 573 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 133 op/s 2024-04-23T14:54:57.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:56 smithi038 ceph-mon[32348]: pgmap v338: 97 pgs: 97 active+clean; 573 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 133 op/s 2024-04-23T14:54:57.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:56 smithi028 ceph-mon[27810]: pgmap v338: 97 pgs: 97 active+clean; 573 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 133 op/s 2024-04-23T14:54:57.530 INFO:tasks.workunit.client.0.smithi028.stdout: 1 933814 8.78 MB/sec execute 433 sec latency 19.914 ms 2024-04-23T14:54:58.532 INFO:tasks.workunit.client.0.smithi028.stdout: 1 935001 8.78 MB/sec execute 434 sec latency 76.664 ms 2024-04-23T14:54:59.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:54:58 smithi105 ceph-mon[32555]: pgmap v339: 97 pgs: 97 active+clean; 581 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.1 MiB/s wr, 195 op/s 2024-04-23T14:54:59.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:54:58 smithi038 ceph-mon[32348]: pgmap v339: 97 pgs: 97 active+clean; 581 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.1 MiB/s wr, 195 op/s 2024-04-23T14:54:59.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:54:58 smithi028 ceph-mon[27810]: pgmap v339: 97 pgs: 97 active+clean; 581 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.1 MiB/s wr, 195 op/s 2024-04-23T14:54:59.531 INFO:tasks.workunit.client.0.smithi028.stdout: 1 936306 8.77 MB/sec execute 435 sec latency 30.322 ms 2024-04-23T14:55:00.530 INFO:tasks.workunit.client.0.smithi028.stdout: 1 938109 8.78 MB/sec execute 436 sec latency 25.026 ms 2024-04-23T14:55:01.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:00 smithi105 ceph-mon[32555]: pgmap v340: 97 pgs: 97 active+clean; 581 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 132 op/s 2024-04-23T14:55:01.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:00 smithi038 ceph-mon[32348]: pgmap v340: 97 pgs: 97 active+clean; 581 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 132 op/s 2024-04-23T14:55:01.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:00 smithi028 ceph-mon[27810]: pgmap v340: 97 pgs: 97 active+clean; 581 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 132 op/s 2024-04-23T14:55:01.530 INFO:tasks.workunit.client.0.smithi028.stdout: 1 938939 8.77 MB/sec execute 437 sec latency 25.961 ms 2024-04-23T14:55:02.531 INFO:tasks.workunit.client.0.smithi028.stdout: 1 939267 8.75 MB/sec execute 438 sec latency 77.887 ms 2024-04-23T14:55:03.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:02 smithi105 ceph-mon[32555]: pgmap v341: 97 pgs: 97 active+clean; 581 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 132 op/s 2024-04-23T14:55:03.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:02 smithi038 ceph-mon[32348]: pgmap v341: 97 pgs: 97 active+clean; 581 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 132 op/s 2024-04-23T14:55:03.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:02 smithi028 ceph-mon[27810]: pgmap v341: 97 pgs: 97 active+clean; 581 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 132 op/s 2024-04-23T14:55:03.531 INFO:tasks.workunit.client.0.smithi028.stdout: 1 940430 8.74 MB/sec execute 439 sec latency 103.841 ms 2024-04-23T14:55:04.531 INFO:tasks.workunit.client.0.smithi028.stdout: 1 941962 8.75 MB/sec execute 440 sec latency 25.869 ms 2024-04-23T14:55:05.173 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:04 smithi038 ceph-mon[32348]: pgmap v342: 97 pgs: 97 active+clean; 589 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 227 op/s 2024-04-23T14:55:05.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:04 smithi105 ceph-mon[32555]: pgmap v342: 97 pgs: 97 active+clean; 589 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 227 op/s 2024-04-23T14:55:05.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:04 smithi028 ceph-mon[27810]: pgmap v342: 97 pgs: 97 active+clean; 589 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 227 op/s 2024-04-23T14:55:05.531 INFO:tasks.workunit.client.0.smithi028.stdout: 1 943453 8.74 MB/sec execute 441 sec latency 17.352 ms 2024-04-23T14:55:05.737 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:55:05.737 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:55:05.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.967+0000 7f2f42000640 1 -- 172.21.15.28:0/2808340156 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f3c074d50 msgr2=0x7f2f3c10e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:05.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.967+0000 7f2f42000640 1 --2- 172.21.15.28:0/2808340156 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f3c074d50 0x7f2f3c10e1a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f2f24009860 tx=0x7f2f2402a3d0 comp rx=0 tx=0).stop 2024-04-23T14:55:05.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.967+0000 7f2f42000640 1 -- 172.21.15.28:0/2808340156 shutdown_connections 2024-04-23T14:55:05.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.967+0000 7f2f42000640 1 --2- 172.21.15.28:0/2808340156 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f3c10e6e0 0x7f2f3c110b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:05.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.967+0000 7f2f42000640 1 --2- 172.21.15.28:0/2808340156 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f3c074d50 0x7f2f3c10e1a0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:05.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.967+0000 7f2f42000640 1 --2- 172.21.15.28:0/2808340156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f3c074430 0x7f2f3c074810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:05.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.967+0000 7f2f42000640 1 -- 172.21.15.28:0/2808340156 >> 172.21.15.28:0/2808340156 conn(0x7f2f3c0fd9d0 msgr2=0x7f2f3c0ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:05.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.967+0000 7f2f42000640 1 -- 172.21.15.28:0/2808340156 shutdown_connections 2024-04-23T14:55:05.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.967+0000 7f2f42000640 1 -- 172.21.15.28:0/2808340156 wait complete. 2024-04-23T14:55:05.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.968+0000 7f2f42000640 1 Processor -- start 2024-04-23T14:55:05.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.968+0000 7f2f42000640 1 -- start start 2024-04-23T14:55:05.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.968+0000 7f2f42000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f3c074430 0x7f2f3c1a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:05.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.968+0000 7f2f42000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f3c074d50 0x7f2f3c1a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:05.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.968+0000 7f2f42000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f3c10e6e0 0x7f2f3c19a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:05.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.969+0000 7f2f42000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2f3c1144a0 con 0x7f2f3c10e6e0 2024-04-23T14:55:05.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.969+0000 7f2f42000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2f3c114320 con 0x7f2f3c074d50 2024-04-23T14:55:05.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.969+0000 7f2f42000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2f3c114620 con 0x7f2f3c074430 2024-04-23T14:55:05.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.969+0000 7f2f40c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f3c074430 0x7f2f3c1a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:05.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.969+0000 7f2f40c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f3c074430 0x7f2f3c1a0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:51930/0 (socket says 172.21.15.28:51930) 2024-04-23T14:55:05.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.969+0000 7f2f40c00640 1 -- 172.21.15.28:0/3928081238 learned_addr learned my addr 172.21.15.28:0/3928081238 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:05.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.969+0000 7f2f41600640 1 --2- 172.21.15.28:0/3928081238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f3c10e6e0 0x7f2f3c19a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:05.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.969+0000 7f2f33e00640 1 --2- 172.21.15.28:0/3928081238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f3c074d50 0x7f2f3c1a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:05.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.969+0000 7f2f40c00640 1 -- 172.21.15.28:0/3928081238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f3c074d50 msgr2=0x7f2f3c1a0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:05.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.969+0000 7f2f40c00640 1 --2- 172.21.15.28:0/3928081238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f3c074d50 0x7f2f3c1a0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:05.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.969+0000 7f2f40c00640 1 -- 172.21.15.28:0/3928081238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f3c10e6e0 msgr2=0x7f2f3c19a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:05.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.969+0000 7f2f40c00640 1 --2- 172.21.15.28:0/3928081238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f3c10e6e0 0x7f2f3c19a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:05.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.969+0000 7f2f40c00640 1 -- 172.21.15.28:0/3928081238 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f3c19af90 con 0x7f2f3c074430 2024-04-23T14:55:05.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.970+0000 7f2f40c00640 1 --2- 172.21.15.28:0/3928081238 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f3c074430 0x7f2f3c1a0650 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c00ebc0 tx=0x7f2f2c00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:05.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.970+0000 7f2f3a000640 1 -- 172.21.15.28:0/3928081238 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c010630 con 0x7f2f3c074430 2024-04-23T14:55:05.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.970+0000 7f2f3a000640 1 -- 172.21.15.28:0/3928081238 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f2c010cb0 con 0x7f2f3c074430 2024-04-23T14:55:05.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.970+0000 7f2f3a000640 1 -- 172.21.15.28:0/3928081238 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c018bd0 con 0x7f2f3c074430 2024-04-23T14:55:05.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.970+0000 7f2f42000640 1 -- 172.21.15.28:0/3928081238 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f3c19b280 con 0x7f2f3c074430 2024-04-23T14:55:05.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.970+0000 7f2f42000640 1 -- 172.21.15.28:0/3928081238 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f3c1a74a0 con 0x7f2f3c074430 2024-04-23T14:55:05.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.974+0000 7f2f3a000640 1 -- 172.21.15.28:0/3928081238 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f2f2c0107d0 con 0x7f2f3c074430 2024-04-23T14:55:05.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.974+0000 7f2f463f4740 1 Processor -- start 2024-04-23T14:55:05.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.974+0000 7f2f463f4740 1 -- start start 2024-04-23T14:55:05.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.974+0000 7f2f463f4740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5649ba088a30 0x5649ba088e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:05.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.974+0000 7f2f463f4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649ba089350 0x5649ba0934e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:05.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.974+0000 7f2f463f4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ba093a20 0x5649ba095e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:05.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.974+0000 7f2f463f4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5649b9fee1f0 con 0x5649ba093a20 2024-04-23T14:55:05.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.974+0000 7f2f463f4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5649b9f94350 con 0x5649ba089350 2024-04-23T14:55:05.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.974+0000 7f2f463f4740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5649b9fea1b0 con 0x5649ba088a30 2024-04-23T14:55:05.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.975+0000 7f2f41600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5649ba088a30 0x5649ba088e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:05.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.975+0000 7f2f41600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5649ba088a30 0x5649ba088e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:51946/0 (socket says 172.21.15.28:51946) 2024-04-23T14:55:05.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.975+0000 7f2f40c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649ba089350 0x5649ba0934e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:05.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.975+0000 7f2f41600640 1 -- 172.21.15.28:0/3991180718 learned_addr learned my addr 172.21.15.28:0/3991180718 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:05.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.975+0000 7f2f33e00640 1 --2- 172.21.15.28:0/3991180718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ba093a20 0x5649ba095e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:05.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.975+0000 7f2f41600640 1 -- 172.21.15.28:0/3991180718 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649ba089350 msgr2=0x5649ba0934e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:05.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.975+0000 7f2f41600640 1 --2- 172.21.15.28:0/3991180718 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649ba089350 0x5649ba0934e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:05.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.975+0000 7f2f41600640 1 -- 172.21.15.28:0/3991180718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ba093a20 msgr2=0x5649ba095e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:05.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.975+0000 7f2f41600640 1 --2- 172.21.15.28:0/3991180718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ba093a20 0x5649ba095e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:05.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.975+0000 7f2f41600640 1 -- 172.21.15.28:0/3991180718 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649b9fe8f50 con 0x5649ba088a30 2024-04-23T14:55:05.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.975+0000 7f2f33e00640 1 --2- 172.21.15.28:0/3991180718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ba093a20 0x5649ba095e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:55:05.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.975+0000 7f2f41600640 1 --2- 172.21.15.28:0/3991180718 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5649ba088a30 0x5649ba088e10 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2f34009be0 tx=0x7f2f34011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c1b9869befab850 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:05.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.975+0000 7f2f3a000640 1 --2- 172.21.15.28:0/3928081238 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f2f200a3da0 0x7f2f200a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:05.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.976+0000 7f2f31c00640 1 -- 172.21.15.28:0/3991180718 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f34004230 con 0x5649ba088a30 2024-04-23T14:55:05.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.976+0000 7f2f31c00640 1 -- 172.21.15.28:0/3991180718 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f34009e50 con 0x5649ba088a30 2024-04-23T14:55:05.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.976+0000 7f2f3a000640 1 -- 172.21.15.28:0/3928081238 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8586+0+0 (secure 0 0 0) 0x7f2f2c0d0e50 con 0x7f2f3c074430 2024-04-23T14:55:05.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.976+0000 7f2f31c00640 1 -- 172.21.15.28:0/3991180718 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f340197e0 con 0x5649ba088a30 2024-04-23T14:55:05.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.976+0000 7f2f40c00640 1 --2- 172.21.15.28:0/3928081238 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f2f200a3da0 0x7f2f200a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:05.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.976+0000 7f2f463f4740 1 -- 172.21.15.28:0/3991180718 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5649ba088a30 msgr2=0x5649ba088e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:05.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.976+0000 7f2f463f4740 1 --2- 172.21.15.28:0/3991180718 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5649ba088a30 0x5649ba088e10 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2f34009be0 tx=0x7f2f34011810 comp rx=0 tx=0).stop 2024-04-23T14:55:05.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.976+0000 7f2f40c00640 1 --2- 172.21.15.28:0/3928081238 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f2f200a3da0 0x7f2f200a6260 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x5649ba089ff0 tx=0x7f2f2c0d4900 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:05.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.976+0000 7f2f463f4740 1 -- 172.21.15.28:0/3991180718 shutdown_connections 2024-04-23T14:55:05.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.976+0000 7f2f463f4740 1 --2- 172.21.15.28:0/3991180718 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649ba089350 0x5649ba0934e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:05.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.976+0000 7f2f463f4740 1 --2- 172.21.15.28:0/3991180718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ba093a20 0x5649ba095e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:05.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.977+0000 7f2f463f4740 1 --2- 172.21.15.28:0/3991180718 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5649ba088a30 0x5649ba088e10 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:05.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.977+0000 7f2f463f4740 1 -- 172.21.15.28:0/3991180718 >> 172.21.15.28:0/3991180718 conn(0x5649ba084d50 msgr2=0x5649ba085130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:05.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.977+0000 7f2f463f4740 1 -- 172.21.15.28:0/3991180718 shutdown_connections 2024-04-23T14:55:05.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.977+0000 7f2f463f4740 1 -- 172.21.15.28:0/3991180718 wait complete. 2024-04-23T14:55:05.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.977+0000 7f2f463f4740 1 -- start start 2024-04-23T14:55:05.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.977+0000 7f2f463f4740 1 Processor -- start 2024-04-23T14:55:05.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.978+0000 7f2f463f4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ba088a30 0x5649ba090a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:05.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.978+0000 7f2f463f4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649ba089350 0x5649ba090fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:05.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.978+0000 7f2f463f4740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5649ba093a20 0x5649ba091500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:05.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.978+0000 7f2f463f4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5649ba096810 con 0x5649ba088a30 2024-04-23T14:55:05.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.978+0000 7f2f463f4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5649b9fa7690 con 0x5649ba089350 2024-04-23T14:55:05.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.978+0000 7f2f463f4740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5649ba096990 con 0x5649ba093a20 2024-04-23T14:55:05.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.978+0000 7f2f463f4740 10 client.0 _ll_register_callbacks cb 0x5649b9fe3650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:55:05.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.978+0000 7f2f33e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649ba089350 0x5649ba090fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:05.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.978+0000 7f2f33e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649ba089350 0x5649ba090fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:39364/0 (socket says 172.21.15.28:39364) 2024-04-23T14:55:05.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.978+0000 7f2f33e00640 1 -- 172.21.15.28:0/110242177 learned_addr learned my addr 172.21.15.28:0/110242177 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:05.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.978+0000 7f2f41600640 1 --2- 172.21.15.28:0/110242177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ba088a30 0x5649ba090a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:05.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.978+0000 7f2f33e00640 1 -- 172.21.15.28:0/110242177 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5649ba093a20 msgr2=0x5649ba091500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:05.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.978+0000 7f2f33e00640 1 --2- 172.21.15.28:0/110242177 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5649ba093a20 0x5649ba091500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:05.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.978+0000 7f2f33e00640 1 -- 172.21.15.28:0/110242177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ba088a30 msgr2=0x5649ba090a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:05.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.978+0000 7f2f33e00640 1 --2- 172.21.15.28:0/110242177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ba088a30 0x5649ba090a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:05.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.978+0000 7f2f33e00640 1 -- 172.21.15.28:0/110242177 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649b9fa6920 con 0x5649ba089350 2024-04-23T14:55:05.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.979+0000 7f2f33e00640 1 --2- 172.21.15.28:0/110242177 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649ba089350 0x5649ba090fc0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f2f2402a8f0 tx=0x7f2f24030650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98651ec5cbfc9f19 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:05.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.979+0000 7f2f2a000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:55:05.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.979+0000 7f2f2a000640 1 -- 172.21.15.28:0/110242177 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f24004230 con 0x5649ba089350 2024-04-23T14:55:05.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.979+0000 7f2f463f4740 1 -- 172.21.15.28:0/110242177 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f34009e50 con 0x5649ba089350 2024-04-23T14:55:05.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.979+0000 7f2f2a000640 1 -- 172.21.15.28:0/110242177 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8586+0+0 (secure 0 0 0) 0x7f2f240043d0 con 0x5649ba089350 2024-04-23T14:55:05.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.979+0000 7f2f2a000640 1 -- 172.21.15.28:0/110242177 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f2f2404d050 con 0x5649ba089350 2024-04-23T14:55:05.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.979+0000 7f2f463f4740 10 client.26957 fetch_fsmap learned FSMap version 20 2024-04-23T14:55:05.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.979+0000 7f2f463f4740 1 -- 172.21.15.28:0/110242177 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5649b9dca7d0 con 0x5649ba089350 2024-04-23T14:55:05.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.980+0000 7f2f2a000640 1 -- 172.21.15.28:0/110242177 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f34009e50 con 0x5649ba089350 2024-04-23T14:55:05.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.980+0000 7f2f2a000640 1 -- 172.21.15.28:0/110242177 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f2f24044070 con 0x5649ba089350 2024-04-23T14:55:05.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.980+0000 7f2f463f4740 10 client.26957 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:55:05.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.980+0000 7f2f463f4740 10 client.26957 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:55:05.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.980+0000 7f2f463f4740 20 client.26957 populate_metadata read hostname 'smithi028' 2024-04-23T14:55:06.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.980+0000 7f2f463f4740 1 --2- 172.21.15.28:0/110242177 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5649ba0a3e20 0x5649ba0a6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:06.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.980+0000 7f2f463f4740 4 client.26957 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:55:06.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.980+0000 7f2f463f4740 1 -- 172.21.15.28:0/110242177 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649b9ffbc10 con 0x5649ba0a3e20 2024-04-23T14:55:06.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.980+0000 7f2f41600640 1 --2- 172.21.15.28:0/110242177 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5649ba0a3e20 0x5649ba0a6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:06.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.981+0000 7f2f41600640 1 --2- 172.21.15.28:0/110242177 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5649ba0a3e20 0x5649ba0a6230 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=641bb9ff2032efa4 server_cookie=79c733e523253600 in_seq=0 out_seq=0 2024-04-23T14:55:06.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.981+0000 7f2f2a000640 10 client.26957 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:55:06.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.981+0000 7f2f2a000640 1 -- 172.21.15.28:0/110242177 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5649b9ffbc10 con 0x5649ba0a3e20 2024-04-23T14:55:06.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.981+0000 7f2f2a000640 10 client.26957 handle_command_reply: tid=0 2024-04-23T14:55:06.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.982+0000 7f2f463f4740 1 client.26957 shutdown 2024-04-23T14:55:06.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.982+0000 7f2f463f4740 1 -- 172.21.15.28:0/110242177 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649ba089350 msgr2=0x5649ba090fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:06.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.982+0000 7f2f463f4740 1 --2- 172.21.15.28:0/110242177 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649ba089350 0x5649ba090fc0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f2f2402a8f0 tx=0x7f2f24030650 comp rx=0 tx=0).stop 2024-04-23T14:55:06.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.982+0000 7f2f463f4740 1 -- 172.21.15.28:0/110242177 shutdown_connections 2024-04-23T14:55:06.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.982+0000 7f2f463f4740 1 --2- 172.21.15.28:0/110242177 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5649ba0a3e20 0x5649ba0a6230 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.982+0000 7f2f463f4740 1 --2- 172.21.15.28:0/110242177 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649ba089350 0x5649ba090fc0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.982+0000 7f2f463f4740 1 --2- 172.21.15.28:0/110242177 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5649ba093a20 0x5649ba091500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.982+0000 7f2f463f4740 1 --2- 172.21.15.28:0/110242177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ba088a30 0x5649ba090a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.982+0000 7f2f2a000640 0 client.26957 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:55:06.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.982+0000 7f2f463f4740 1 -- 172.21.15.28:0/110242177 >> 172.21.15.28:0/110242177 conn(0x5649ba084d50 msgr2=0x5649ba08ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:06.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.982+0000 7f2f463f4740 1 -- 172.21.15.28:0/110242177 shutdown_connections 2024-04-23T14:55:06.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.982+0000 7f2f463f4740 1 -- 172.21.15.28:0/110242177 wait complete. 2024-04-23T14:55:06.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:05.982+0000 7f2f463f4740 20 client.26957 trim_cache size 0 max 16384 2024-04-23T14:55:06.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.004+0000 7f2f463f4740 1 Processor -- start 2024-04-23T14:55:06.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.004+0000 7f2f463f4740 1 -- start start 2024-04-23T14:55:06.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.004+0000 7f2f463f4740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5649ba088a30 0x5649ba09f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:06.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.004+0000 7f2f463f4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649ba089350 0x5649ba083e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:06.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.004+0000 7f2f463f4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ba093a20 0x5649ba08b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:06.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.004+0000 7f2f463f4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2f3c1adea0 con 0x5649ba093a20 2024-04-23T14:55:06.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.004+0000 7f2f463f4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2f3c1b0700 con 0x5649ba089350 2024-04-23T14:55:06.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.004+0000 7f2f463f4740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2f3c1ae4f0 con 0x5649ba088a30 2024-04-23T14:55:06.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.004+0000 7f2f41600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5649ba088a30 0x5649ba09f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:06.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.004+0000 7f2f41600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5649ba088a30 0x5649ba09f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:51964/0 (socket says 172.21.15.28:51964) 2024-04-23T14:55:06.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.004+0000 7f2f41600640 1 -- 172.21.15.28:0/2347397998 learned_addr learned my addr 172.21.15.28:0/2347397998 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:06.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.004+0000 7f2f33e00640 1 --2- 172.21.15.28:0/2347397998 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649ba089350 0x5649ba083e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:06.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.005+0000 7f2f41600640 1 --2- 172.21.15.28:0/2347397998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ba093a20 0x5649ba08b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:06.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.005+0000 7f2f33e00640 1 -- 172.21.15.28:0/2347397998 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5649ba088a30 msgr2=0x5649ba09f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:06.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.005+0000 7f2f33e00640 1 --2- 172.21.15.28:0/2347397998 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5649ba088a30 0x5649ba09f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.005+0000 7f2f33e00640 1 -- 172.21.15.28:0/2347397998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ba093a20 msgr2=0x5649ba08b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:06.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.005+0000 7f2f33e00640 1 --2- 172.21.15.28:0/2347397998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ba093a20 0x5649ba08b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.005+0000 7f2f33e00640 1 -- 172.21.15.28:0/2347397998 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649ba089180 con 0x5649ba089350 2024-04-23T14:55:06.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.005+0000 7f2f41600640 1 --2- 172.21.15.28:0/2347397998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ba093a20 0x5649ba08b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:55:06.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.005+0000 7f2f33e00640 1 --2- 172.21.15.28:0/2347397998 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649ba089350 0x5649ba083e50 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f2f24030a50 tx=0x7f2f240305b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf4eec594fc58382 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:06.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.006+0000 7f2f31200640 1 -- 172.21.15.28:0/2347397998 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f2404a020 con 0x5649ba089350 2024-04-23T14:55:06.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.006+0000 7f2f31200640 1 -- 172.21.15.28:0/2347397998 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f2403c420 con 0x5649ba089350 2024-04-23T14:55:06.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.006+0000 7f2f31200640 1 -- 172.21.15.28:0/2347397998 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f240054d0 con 0x5649ba089350 2024-04-23T14:55:06.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.006+0000 7f2f463f4740 1 -- 172.21.15.28:0/2347397998 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649ba089350 msgr2=0x5649ba083e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:06.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.006+0000 7f2f463f4740 1 --2- 172.21.15.28:0/2347397998 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649ba089350 0x5649ba083e50 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f2f24030a50 tx=0x7f2f240305b0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.006+0000 7f2f463f4740 1 -- 172.21.15.28:0/2347397998 shutdown_connections 2024-04-23T14:55:06.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.006+0000 7f2f463f4740 1 --2- 172.21.15.28:0/2347397998 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649ba089350 0x5649ba083e50 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.006+0000 7f2f463f4740 1 --2- 172.21.15.28:0/2347397998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ba093a20 0x5649ba08b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.006+0000 7f2f463f4740 1 --2- 172.21.15.28:0/2347397998 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5649ba088a30 0x5649ba09f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.006+0000 7f2f463f4740 1 -- 172.21.15.28:0/2347397998 >> 172.21.15.28:0/2347397998 conn(0x5649ba084d50 msgr2=0x5649ba083740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:06.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.007+0000 7f2f463f4740 1 -- 172.21.15.28:0/2347397998 shutdown_connections 2024-04-23T14:55:06.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.007+0000 7f2f463f4740 1 -- 172.21.15.28:0/2347397998 wait complete. 2024-04-23T14:55:06.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.007+0000 7f2f463f4740 1 -- start start 2024-04-23T14:55:06.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.007+0000 7f2f463f4740 1 Processor -- start 2024-04-23T14:55:06.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.008+0000 7f2f463f4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649ba088a30 0x5649ba09bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:06.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.008+0000 7f2f463f4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ba089350 0x5649ba08d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:06.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.008+0000 7f2f463f4740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5649ba093a20 0x5649ba08d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:06.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.008+0000 7f2f463f4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2f3c1aebc0 con 0x5649ba089350 2024-04-23T14:55:06.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.008+0000 7f2f463f4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2f3c1b0030 con 0x5649ba088a30 2024-04-23T14:55:06.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.008+0000 7f2f463f4740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2f3c1af290 con 0x5649ba093a20 2024-04-23T14:55:06.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.008+0000 7f2f463f4740 10 client.0 _ll_register_callbacks cb 0x7f2f34005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:55:06.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.008+0000 7f2f41600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649ba088a30 0x5649ba09bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:06.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.008+0000 7f2f41600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649ba088a30 0x5649ba09bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:39374/0 (socket says 172.21.15.28:39374) 2024-04-23T14:55:06.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.008+0000 7f2f41600640 1 -- 172.21.15.28:0/664707840 learned_addr learned my addr 172.21.15.28:0/664707840 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:06.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.008+0000 7f2f41600640 1 --2- 172.21.15.28:0/664707840 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5649ba093a20 0x5649ba08d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:06.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.008+0000 7f2f33e00640 1 --2- 172.21.15.28:0/664707840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ba089350 0x5649ba08d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:06.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.009+0000 7f2f41600640 1 -- 172.21.15.28:0/664707840 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5649ba093a20 msgr2=0x5649ba08d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:06.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.009+0000 7f2f41600640 1 --2- 172.21.15.28:0/664707840 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5649ba093a20 0x5649ba08d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.009+0000 7f2f41600640 1 -- 172.21.15.28:0/664707840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ba089350 msgr2=0x5649ba08d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:06.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.009+0000 7f2f41600640 1 --2- 172.21.15.28:0/664707840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ba089350 0x5649ba08d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.009+0000 7f2f41600640 1 -- 172.21.15.28:0/664707840 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649b9d003f0 con 0x5649ba088a30 2024-04-23T14:55:06.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.009+0000 7f2f41600640 1 --2- 172.21.15.28:0/664707840 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649ba088a30 0x5649ba09bea0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f2f34000c00 tx=0x7f2f340097b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1e7351ebd08023b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:06.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.009+0000 7f2f2a000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:55:06.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.009+0000 7f2f2a000640 1 -- 172.21.15.28:0/664707840 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f340050d0 con 0x5649ba088a30 2024-04-23T14:55:06.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.009+0000 7f2f2a000640 1 -- 172.21.15.28:0/664707840 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8586+0+0 (secure 0 0 0) 0x7f2f34005270 con 0x5649ba088a30 2024-04-23T14:55:06.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.009+0000 7f2f463f4740 1 -- 172.21.15.28:0/664707840 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f2403c420 con 0x5649ba088a30 2024-04-23T14:55:06.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.009+0000 7f2f2a000640 1 -- 172.21.15.28:0/664707840 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f2f34005470 con 0x5649ba088a30 2024-04-23T14:55:06.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.009+0000 7f2f463f4740 10 client.26963 fetch_fsmap learned FSMap version 20 2024-04-23T14:55:06.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.009+0000 7f2f463f4740 1 -- 172.21.15.28:0/664707840 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5649b9cc00f0 con 0x5649ba088a30 2024-04-23T14:55:06.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.010+0000 7f2f2a000640 1 -- 172.21.15.28:0/664707840 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f2403c420 con 0x5649ba088a30 2024-04-23T14:55:06.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.010+0000 7f2f2a000640 1 -- 172.21.15.28:0/664707840 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f2f34028440 con 0x5649ba088a30 2024-04-23T14:55:06.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.010+0000 7f2f463f4740 10 client.26963 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:55:06.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.010+0000 7f2f463f4740 10 client.26963 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:55:06.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.010+0000 7f2f463f4740 20 client.26963 populate_metadata read hostname 'smithi028' 2024-04-23T14:55:06.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.010+0000 7f2f463f4740 1 --2- 172.21.15.28:0/664707840 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5649ba112c50 0x5649ba115040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:06.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.010+0000 7f2f463f4740 4 client.26963 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:55:06.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.010+0000 7f2f463f4740 1 -- 172.21.15.28:0/664707840 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5649b9ff5a80 con 0x5649ba112c50 2024-04-23T14:55:06.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.010+0000 7f2f33e00640 1 --2- 172.21.15.28:0/664707840 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5649ba112c50 0x5649ba115040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:06.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.011+0000 7f2f33e00640 1 --2- 172.21.15.28:0/664707840 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5649ba112c50 0x5649ba115040 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84e80625a6394b36 server_cookie=d8c543bb9ab1b3f in_seq=0 out_seq=0 2024-04-23T14:55:06.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.011+0000 7f2f2a000640 10 client.26963 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:55:06.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.011+0000 7f2f2a000640 1 -- 172.21.15.28:0/664707840 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5649b9ff5a80 con 0x5649ba112c50 2024-04-23T14:55:06.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.011+0000 7f2f2a000640 10 client.26963 handle_command_reply: tid=0 2024-04-23T14:55:06.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.011+0000 7f2f463f4740 1 client.26963 shutdown 2024-04-23T14:55:06.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.011+0000 7f2f463f4740 1 -- 172.21.15.28:0/664707840 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649ba088a30 msgr2=0x5649ba09bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:06.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.011+0000 7f2f463f4740 1 --2- 172.21.15.28:0/664707840 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649ba088a30 0x5649ba09bea0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f2f34000c00 tx=0x7f2f340097b0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.011+0000 7f2f463f4740 1 -- 172.21.15.28:0/664707840 shutdown_connections 2024-04-23T14:55:06.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.011+0000 7f2f463f4740 1 --2- 172.21.15.28:0/664707840 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5649ba112c50 0x5649ba115040 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.011+0000 7f2f463f4740 1 --2- 172.21.15.28:0/664707840 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5649ba093a20 0x5649ba08d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.011+0000 7f2f2a000640 0 client.26963 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:55:06.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.011+0000 7f2f463f4740 1 --2- 172.21.15.28:0/664707840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649ba089350 0x5649ba08d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.011+0000 7f2f463f4740 1 --2- 172.21.15.28:0/664707840 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649ba088a30 0x5649ba09bea0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.011+0000 7f2f463f4740 1 -- 172.21.15.28:0/664707840 >> 172.21.15.28:0/664707840 conn(0x5649ba084d50 msgr2=0x5649ba083fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:06.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.012+0000 7f2f463f4740 1 -- 172.21.15.28:0/664707840 shutdown_connections 2024-04-23T14:55:06.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.012+0000 7f2f463f4740 1 -- 172.21.15.28:0/664707840 wait complete. 2024-04-23T14:55:06.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.012+0000 7f2f463f4740 20 client.26963 trim_cache size 0 max 16384 2024-04-23T14:55:06.023 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:55:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-23T14:55:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-23T14:55:06.024 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:55:06.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.012+0000 7f2f42000640 1 -- 172.21.15.28:0/3928081238 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f2f200a3da0 msgr2=0x7f2f200a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:06.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.012+0000 7f2f42000640 1 --2- 172.21.15.28:0/3928081238 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f2f200a3da0 0x7f2f200a6260 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x5649ba089ff0 tx=0x7f2f2c0d4900 comp rx=0 tx=0).stop 2024-04-23T14:55:06.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.012+0000 7f2f42000640 1 -- 172.21.15.28:0/3928081238 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f3c074430 msgr2=0x7f2f3c1a0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:06.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.012+0000 7f2f42000640 1 --2- 172.21.15.28:0/3928081238 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f3c074430 0x7f2f3c1a0650 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c00ebc0 tx=0x7f2f2c00c6a0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.012+0000 7f2f42000640 1 -- 172.21.15.28:0/3928081238 shutdown_connections 2024-04-23T14:55:06.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.012+0000 7f2f42000640 1 --2- 172.21.15.28:0/3928081238 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f2f200a3da0 0x7f2f200a6260 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.012+0000 7f2f42000640 1 --2- 172.21.15.28:0/3928081238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f3c074d50 0x7f2f3c1a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.012+0000 7f2f42000640 1 --2- 172.21.15.28:0/3928081238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f3c10e6e0 0x7f2f3c19a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.012+0000 7f2f42000640 1 --2- 172.21.15.28:0/3928081238 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f3c074430 0x7f2f3c1a0650 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.012+0000 7f2f42000640 1 -- 172.21.15.28:0/3928081238 >> 172.21.15.28:0/3928081238 conn(0x7f2f3c0fd9d0 msgr2=0x7f2f3c0fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:06.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.013+0000 7f2f42000640 1 -- 172.21.15.28:0/3928081238 shutdown_connections 2024-04-23T14:55:06.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.013+0000 7f2f42000640 1 -- 172.21.15.28:0/3928081238 wait complete. 2024-04-23T14:55:06.031 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-23T14:55:06.032 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:55:06.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:05 smithi028 ceph-mon[27810]: osdmap e97: 12 total, 12 up, 12 in 2024-04-23T14:55:06.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:05 smithi028 ceph-mon[27810]: pgmap v344: 97 pgs: 97 active+clean; 589 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.2 MiB/s wr, 189 op/s 2024-04-23T14:55:06.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.252+0000 7f3477200640 1 -- 172.21.15.28:0/2279170450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3478109360 msgr2=0x7f3478110f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:06.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.252+0000 7f3477200640 1 --2- 172.21.15.28:0/2279170450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3478109360 0x7f3478110f30 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f3470009ca0 tx=0x7f347002a8c0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.253+0000 7f3477200640 1 -- 172.21.15.28:0/2279170450 shutdown_connections 2024-04-23T14:55:06.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.253+0000 7f3477200640 1 --2- 172.21.15.28:0/2279170450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3478109360 0x7f3478110f30 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.253+0000 7f3477200640 1 --2- 172.21.15.28:0/2279170450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3478104c40 0x7f3478108c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.253+0000 7f3477200640 1 --2- 172.21.15.28:0/2279170450 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3478104290 0x7f3478104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.253+0000 7f3477200640 1 -- 172.21.15.28:0/2279170450 >> 172.21.15.28:0/2279170450 conn(0x7f34780ffad0 msgr2=0x7f3478101ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:06.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.253+0000 7f3477200640 1 -- 172.21.15.28:0/2279170450 shutdown_connections 2024-04-23T14:55:06.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.253+0000 7f3477200640 1 -- 172.21.15.28:0/2279170450 wait complete. 2024-04-23T14:55:06.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.254+0000 7f3477200640 1 Processor -- start 2024-04-23T14:55:06.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.254+0000 7f3477200640 1 -- start start 2024-04-23T14:55:06.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.255+0000 7f3477200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3478104290 0x7f34781a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:06.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.255+0000 7f3477200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3478109360 0x7f34781a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:06.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.255+0000 7f3477200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f347819a710 0x7f347819ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:06.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.255+0000 7f3477200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3478114450 con 0x7f3478109360 2024-04-23T14:55:06.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.255+0000 7f3477200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f34781142d0 con 0x7f347819a710 2024-04-23T14:55:06.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.255+0000 7f3477200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f34781145d0 con 0x7f3478104290 2024-04-23T14:55:06.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.255+0000 7f3475400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3478109360 0x7f34781a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:06.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.255+0000 7f3475400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3478109360 0x7f34781a0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41324/0 (socket says 172.21.15.28:41324) 2024-04-23T14:55:06.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.255+0000 7f3475400640 1 -- 172.21.15.28:0/4124052546 learned_addr learned my addr 172.21.15.28:0/4124052546 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:06.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.255+0000 7f3476800640 1 --2- 172.21.15.28:0/4124052546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f347819a710 0x7f347819ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:06.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.255+0000 7f3475e00640 1 --2- 172.21.15.28:0/4124052546 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3478104290 0x7f34781a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:06.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.255+0000 7f3475400640 1 -- 172.21.15.28:0/4124052546 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3478104290 msgr2=0x7f34781a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:06.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.255+0000 7f3475400640 1 --2- 172.21.15.28:0/4124052546 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3478104290 0x7f34781a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.255+0000 7f3475400640 1 -- 172.21.15.28:0/4124052546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f347819a710 msgr2=0x7f347819ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:06.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.255+0000 7f3475400640 1 --2- 172.21.15.28:0/4124052546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f347819a710 0x7f347819ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.255+0000 7f3475400640 1 -- 172.21.15.28:0/4124052546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f347819b310 con 0x7f3478109360 2024-04-23T14:55:06.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.256+0000 7f3475400640 1 --2- 172.21.15.28:0/4124052546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3478109360 0x7f34781a0b90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f346400e9c0 tx=0x7f346400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:06.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.256+0000 7f346ea00640 1 -- 172.21.15.28:0/4124052546 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f346400cd80 con 0x7f3478109360 2024-04-23T14:55:06.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.256+0000 7f3477200640 1 -- 172.21.15.28:0/4124052546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34781a73f0 con 0x7f3478109360 2024-04-23T14:55:06.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.256+0000 7f346ea00640 1 -- 172.21.15.28:0/4124052546 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3464005500 con 0x7f3478109360 2024-04-23T14:55:06.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.256+0000 7f346ea00640 1 -- 172.21.15.28:0/4124052546 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3464010690 con 0x7f3478109360 2024-04-23T14:55:06.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.256+0000 7f3477200640 1 -- 172.21.15.28:0/4124052546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34781a77f0 con 0x7f3478109360 2024-04-23T14:55:06.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.258+0000 7f3477200640 1 -- 172.21.15.28:0/4124052546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3478105760 con 0x7f3478109360 2024-04-23T14:55:06.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.261+0000 7f346ea00640 1 -- 172.21.15.28:0/4124052546 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f34640046c0 con 0x7f3478109360 2024-04-23T14:55:06.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.262+0000 7f346ea00640 1 --2- 172.21.15.28:0/4124052546 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f345c0a3d20 0x7f345c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:06.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.262+0000 7f346ea00640 1 -- 172.21.15.28:0/4124052546 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8586+0+0 (secure 0 0 0) 0x7f34640d0b70 con 0x7f3478109360 2024-04-23T14:55:06.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.262+0000 7f3475e00640 1 --2- 172.21.15.28:0/4124052546 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f345c0a3d20 0x7f345c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:06.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.263+0000 7f3475e00640 1 --2- 172.21.15.28:0/4124052546 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f345c0a3d20 0x7f345c0a61e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f34680095f0 tx=0x7f3468005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:06.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.267+0000 7f346ea00640 1 -- 172.21.15.28:0/4124052546 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f346409d170 con 0x7f3478109360 2024-04-23T14:55:06.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:05 smithi105 ceph-mon[32555]: osdmap e97: 12 total, 12 up, 12 in 2024-04-23T14:55:06.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:05 smithi105 ceph-mon[32555]: pgmap v344: 97 pgs: 97 active+clean; 589 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.2 MiB/s wr, 189 op/s 2024-04-23T14:55:06.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:05 smithi038 ceph-mon[32348]: osdmap e97: 12 total, 12 up, 12 in 2024-04-23T14:55:06.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:05 smithi038 ceph-mon[32348]: pgmap v344: 97 pgs: 97 active+clean; 589 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.2 MiB/s wr, 189 op/s 2024-04-23T14:55:06.531 INFO:tasks.workunit.client.0.smithi028.stdout: 1 945570 8.75 MB/sec execute 442 sec latency 13.515 ms 2024-04-23T14:55:06.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.542+0000 7f3477200640 1 -- 172.21.15.28:0/4124052546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3478068d40 con 0x7f3478109360 2024-04-23T14:55:06.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.544+0000 7f346ea00640 1 -- 172.21.15.28:0/4124052546 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7f346409ca70 con 0x7f3478109360 2024-04-23T14:55:06.545 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:55:06.546 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:55:06.546 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:55:06.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.547+0000 7f3477200640 1 -- 172.21.15.28:0/4124052546 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f345c0a3d20 msgr2=0x7f345c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:06.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.547+0000 7f3477200640 1 --2- 172.21.15.28:0/4124052546 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f345c0a3d20 0x7f345c0a61e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f34680095f0 tx=0x7f3468005f90 comp rx=0 tx=0).stop 2024-04-23T14:55:06.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.547+0000 7f3477200640 1 -- 172.21.15.28:0/4124052546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3478109360 msgr2=0x7f34781a0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:06.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.547+0000 7f3477200640 1 --2- 172.21.15.28:0/4124052546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3478109360 0x7f34781a0b90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f346400e9c0 tx=0x7f346400ee80 comp rx=0 tx=0).stop 2024-04-23T14:55:06.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.547+0000 7f3477200640 1 -- 172.21.15.28:0/4124052546 shutdown_connections 2024-04-23T14:55:06.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.547+0000 7f3477200640 1 --2- 172.21.15.28:0/4124052546 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f345c0a3d20 0x7f345c0a61e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.547+0000 7f3477200640 1 --2- 172.21.15.28:0/4124052546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f347819a710 0x7f347819ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.547+0000 7f3477200640 1 --2- 172.21.15.28:0/4124052546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3478109360 0x7f34781a0b90 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.547+0000 7f3477200640 1 --2- 172.21.15.28:0/4124052546 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3478104290 0x7f34781a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.547+0000 7f3477200640 1 -- 172.21.15.28:0/4124052546 >> 172.21.15.28:0/4124052546 conn(0x7f34780ffad0 msgr2=0x7f3478101d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:06.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.547+0000 7f3477200640 1 -- 172.21.15.28:0/4124052546 shutdown_connections 2024-04-23T14:55:06.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.547+0000 7f3477200640 1 -- 172.21.15.28:0/4124052546 wait complete. 2024-04-23T14:55:06.569 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-23T14:55:06.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.779+0000 7f43a0c00640 1 -- 172.21.15.28:0/2546676190 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f439c103060 msgr2=0x7f439c103440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:06.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.779+0000 7f43a0c00640 1 --2- 172.21.15.28:0/2546676190 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f439c103060 0x7f439c103440 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f438c009900 tx=0x7f438c02a480 comp rx=0 tx=0).stop 2024-04-23T14:55:06.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.780+0000 7f43a0c00640 1 -- 172.21.15.28:0/2546676190 shutdown_connections 2024-04-23T14:55:06.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.780+0000 7f43a0c00640 1 --2- 172.21.15.28:0/2546676190 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f439c10c480 0x7f439c10e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.780+0000 7f43a0c00640 1 --2- 172.21.15.28:0/2546676190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f439c103980 0x7f439c10be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.780+0000 7f43a0c00640 1 --2- 172.21.15.28:0/2546676190 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f439c103060 0x7f439c103440 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.780+0000 7f43a0c00640 1 -- 172.21.15.28:0/2546676190 >> 172.21.15.28:0/2546676190 conn(0x7f439c0fc880 msgr2=0x7f439c0feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:06.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.780+0000 7f43a0c00640 1 -- 172.21.15.28:0/2546676190 shutdown_connections 2024-04-23T14:55:06.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.780+0000 7f43a0c00640 1 -- 172.21.15.28:0/2546676190 wait complete. 2024-04-23T14:55:06.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.781+0000 7f43a0c00640 1 Processor -- start 2024-04-23T14:55:06.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.781+0000 7f43a0c00640 1 -- start start 2024-04-23T14:55:06.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.781+0000 7f43a0c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f439c103060 0x7f439c1a06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:06.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.781+0000 7f43a0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f439c103980 0x7f439c1a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:06.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.781+0000 7f43a0c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f439c10c480 0x7f439c19a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:06.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.781+0000 7f43a0c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f439c112310 con 0x7f439c103980 2024-04-23T14:55:06.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.781+0000 7f43a0c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f439c112190 con 0x7f439c10c480 2024-04-23T14:55:06.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.781+0000 7f43a0c00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f439c112490 con 0x7f439c103060 2024-04-23T14:55:06.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.782+0000 7f439b400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f439c103060 0x7f439c1a06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:06.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.782+0000 7f439b400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f439c103060 0x7f439c1a06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:52006/0 (socket says 172.21.15.28:52006) 2024-04-23T14:55:06.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.782+0000 7f439b400640 1 -- 172.21.15.28:0/2714664354 learned_addr learned my addr 172.21.15.28:0/2714664354 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:06.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.782+0000 7f439be00640 1 --2- 172.21.15.28:0/2714664354 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f439c10c480 0x7f439c19a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:06.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.782+0000 7f439aa00640 1 --2- 172.21.15.28:0/2714664354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f439c103980 0x7f439c1a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:06.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.782+0000 7f439b400640 1 -- 172.21.15.28:0/2714664354 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f439c10c480 msgr2=0x7f439c19a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:06.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.782+0000 7f439b400640 1 --2- 172.21.15.28:0/2714664354 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f439c10c480 0x7f439c19a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.782+0000 7f439b400640 1 -- 172.21.15.28:0/2714664354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f439c103980 msgr2=0x7f439c1a0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:06.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.782+0000 7f439b400640 1 --2- 172.21.15.28:0/2714664354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f439c103980 0x7f439c1a0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.782+0000 7f439b400640 1 -- 172.21.15.28:0/2714664354 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f439c19b020 con 0x7f439c103060 2024-04-23T14:55:06.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.782+0000 7f439aa00640 1 --2- 172.21.15.28:0/2714664354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f439c103980 0x7f439c1a0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:55:06.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.783+0000 7f439b400640 1 --2- 172.21.15.28:0/2714664354 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f439c103060 0x7f439c1a06e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f438c004770 tx=0x7f438c003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:06.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.783+0000 7f4393e00640 1 -- 172.21.15.28:0/2714664354 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f438c03f070 con 0x7f439c103060 2024-04-23T14:55:06.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.783+0000 7f4393e00640 1 -- 172.21.15.28:0/2714664354 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f438c031ca0 con 0x7f439c103060 2024-04-23T14:55:06.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.783+0000 7f43a0c00640 1 -- 172.21.15.28:0/2714664354 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f439c19b2b0 con 0x7f439c103060 2024-04-23T14:55:06.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.783+0000 7f4393e00640 1 -- 172.21.15.28:0/2714664354 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f438c035030 con 0x7f439c103060 2024-04-23T14:55:06.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.783+0000 7f43a0c00640 1 -- 172.21.15.28:0/2714664354 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f439c1a7530 con 0x7f439c103060 2024-04-23T14:55:06.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.787+0000 7f4393e00640 1 -- 172.21.15.28:0/2714664354 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f438c042070 con 0x7f439c103060 2024-04-23T14:55:06.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.787+0000 7f43a5188740 1 Processor -- start 2024-04-23T14:55:06.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.788+0000 7f43a5188740 1 -- start start 2024-04-23T14:55:06.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.788+0000 7f43a5188740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56430ec37890 0x56430ec37c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:06.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.788+0000 7f43a5188740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56430ec38240 0x56430ec4ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:06.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.788+0000 7f43a5188740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430ec4b340 0x56430ec4d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:06.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.788+0000 7f43a5188740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56430eb9d1d0 con 0x56430ec4b340 2024-04-23T14:55:06.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.788+0000 7f43a5188740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56430eb43450 con 0x56430ec38240 2024-04-23T14:55:06.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.788+0000 7f43a5188740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56430eb99190 con 0x56430ec37890 2024-04-23T14:55:06.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.788+0000 7f439be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56430ec37890 0x56430ec37c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:06.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.788+0000 7f439b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56430ec38240 0x56430ec4ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:06.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.788+0000 7f439be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56430ec37890 0x56430ec37c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:52018/0 (socket says 172.21.15.28:52018) 2024-04-23T14:55:06.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.789+0000 7f439be00640 1 -- 172.21.15.28:0/4239142873 learned_addr learned my addr 172.21.15.28:0/4239142873 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:06.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.789+0000 7f439aa00640 1 --2- 172.21.15.28:0/4239142873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430ec4b340 0x56430ec4d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:06.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.789+0000 7f4393e00640 1 --2- 172.21.15.28:0/2714664354 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f43800a3da0 0x7f43800a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:06.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.789+0000 7f439b400640 1 -- 172.21.15.28:0/4239142873 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56430ec37890 msgr2=0x56430ec37c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:06.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.789+0000 7f439b400640 1 --2- 172.21.15.28:0/4239142873 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56430ec37890 0x56430ec37c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.789+0000 7f439b400640 1 -- 172.21.15.28:0/4239142873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430ec4b340 msgr2=0x56430ec4d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:06.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.789+0000 7f439b400640 1 --2- 172.21.15.28:0/4239142873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430ec4b340 0x56430ec4d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.789+0000 7f439b400640 1 -- 172.21.15.28:0/4239142873 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56430ec00310 con 0x56430ec38240 2024-04-23T14:55:06.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.789+0000 7f439aa00640 1 --2- 172.21.15.28:0/4239142873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430ec4b340 0x56430ec4d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:55:06.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.789+0000 7f4393e00640 1 -- 172.21.15.28:0/2714664354 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8586+0+0 (secure 0 0 0) 0x7f438c0f0fe0 con 0x7f439c103060 2024-04-23T14:55:06.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.789+0000 7f439be00640 1 --2- 172.21.15.28:0/2714664354 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f43800a3da0 0x7f43800a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:06.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.790+0000 7f439b400640 1 --2- 172.21.15.28:0/4239142873 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56430ec38240 0x56430ec4ae00 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f438c0ee6f0 tx=0x7f438c04b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cef708853cc7ea3a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:06.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.790+0000 7f4387e00640 1 -- 172.21.15.28:0/4239142873 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f438c0bc8a0 con 0x56430ec38240 2024-04-23T14:55:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.790+0000 7f4387e00640 1 -- 172.21.15.28:0/4239142873 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f438c049e80 con 0x56430ec38240 2024-04-23T14:55:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.790+0000 7f4387e00640 1 -- 172.21.15.28:0/4239142873 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f438c0bcf40 con 0x56430ec38240 2024-04-23T14:55:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.791+0000 7f43a5188740 1 -- 172.21.15.28:0/4239142873 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56430ec38240 msgr2=0x56430ec4ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.791+0000 7f43a5188740 1 --2- 172.21.15.28:0/4239142873 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56430ec38240 0x56430ec4ae00 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f438c0ee6f0 tx=0x7f438c04b000 comp rx=0 tx=0).stop 2024-04-23T14:55:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.791+0000 7f439be00640 1 --2- 172.21.15.28:0/2714664354 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f43800a3da0 0x7f43800a6260 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4394005f40 tx=0x7f4394004a40 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.791+0000 7f43a5188740 1 -- 172.21.15.28:0/4239142873 shutdown_connections 2024-04-23T14:55:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.791+0000 7f43a5188740 1 --2- 172.21.15.28:0/4239142873 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56430ec38240 0x56430ec4ae00 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.791+0000 7f43a5188740 1 --2- 172.21.15.28:0/4239142873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430ec4b340 0x56430ec4d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.791+0000 7f43a5188740 1 --2- 172.21.15.28:0/4239142873 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56430ec37890 0x56430ec37c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.791+0000 7f43a5188740 1 -- 172.21.15.28:0/4239142873 >> 172.21.15.28:0/4239142873 conn(0x56430ec33e60 msgr2=0x56430ec34240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.791+0000 7f43a5188740 1 -- 172.21.15.28:0/4239142873 shutdown_connections 2024-04-23T14:55:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.791+0000 7f43a5188740 1 -- 172.21.15.28:0/4239142873 wait complete. 2024-04-23T14:55:06.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.792+0000 7f43a5188740 1 -- start start 2024-04-23T14:55:06.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.792+0000 7f43a5188740 1 Processor -- start 2024-04-23T14:55:06.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.792+0000 7f43a5188740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56430ec37890 0x56430ec4add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:06.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.792+0000 7f43a5188740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430ec38240 0x56430ec480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:06.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.793+0000 7f43a5188740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56430ec4b340 0x56430ec487b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:06.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.793+0000 7f43a5188740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56430ec4dec0 con 0x56430ec38240 2024-04-23T14:55:06.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.793+0000 7f43a5188740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56430eb56810 con 0x56430ec37890 2024-04-23T14:55:06.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.793+0000 7f43a5188740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56430ec4e040 con 0x56430ec4b340 2024-04-23T14:55:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.793+0000 7f43a5188740 10 client.0 _ll_register_callbacks cb 0x56430eb24140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:55:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.793+0000 7f439b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56430ec37890 0x56430ec4add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.793+0000 7f439b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56430ec37890 0x56430ec4add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:39438/0 (socket says 172.21.15.28:39438) 2024-04-23T14:55:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.793+0000 7f439b400640 1 -- 172.21.15.28:0/631915683 learned_addr learned my addr 172.21.15.28:0/631915683 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.793+0000 7f439aa00640 1 --2- 172.21.15.28:0/631915683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430ec38240 0x56430ec480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.793+0000 7f439be00640 1 --2- 172.21.15.28:0/631915683 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56430ec4b340 0x56430ec487b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.793+0000 7f439b400640 1 -- 172.21.15.28:0/631915683 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56430ec4b340 msgr2=0x56430ec487b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.793+0000 7f439b400640 1 --2- 172.21.15.28:0/631915683 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56430ec4b340 0x56430ec487b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.793+0000 7f439b400640 1 -- 172.21.15.28:0/631915683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430ec38240 msgr2=0x56430ec480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.793+0000 7f439b400640 1 --2- 172.21.15.28:0/631915683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430ec38240 0x56430ec480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.793+0000 7f439b400640 1 -- 172.21.15.28:0/631915683 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56430eb673f0 con 0x56430ec37890 2024-04-23T14:55:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.794+0000 7f439b400640 1 --2- 172.21.15.28:0/631915683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56430ec37890 0x56430ec4add0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f438c0c5680 tx=0x7f438c02a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6d14efc93ef66ea server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.794+0000 7f4384c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:55:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.794+0000 7f4384c00640 1 -- 172.21.15.28:0/631915683 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f438c0bc8a0 con 0x56430ec37890 2024-04-23T14:55:06.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.794+0000 7f4384c00640 1 -- 172.21.15.28:0/631915683 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8586+0+0 (secure 0 0 0) 0x7f438c0d16a0 con 0x56430ec37890 2024-04-23T14:55:06.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.794+0000 7f43a5188740 1 -- 172.21.15.28:0/631915683 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f438c049e80 con 0x56430ec37890 2024-04-23T14:55:06.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.794+0000 7f4384c00640 1 -- 172.21.15.28:0/631915683 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f438c0d6050 con 0x56430ec37890 2024-04-23T14:55:06.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.794+0000 7f43a5188740 10 client.26981 fetch_fsmap learned FSMap version 20 2024-04-23T14:55:06.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.794+0000 7f43a5188740 1 -- 172.21.15.28:0/631915683 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x56430e8af3f0 con 0x56430ec37890 2024-04-23T14:55:06.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.795+0000 7f4384c00640 1 -- 172.21.15.28:0/631915683 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f438c049e80 con 0x56430ec37890 2024-04-23T14:55:06.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.795+0000 7f4384c00640 1 -- 172.21.15.28:0/631915683 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f438c0ee970 con 0x56430ec37890 2024-04-23T14:55:06.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.795+0000 7f43a5188740 10 client.26981 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:55:06.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.795+0000 7f43a5188740 10 client.26981 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:55:06.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.795+0000 7f43a5188740 20 client.26981 populate_metadata read hostname 'smithi028' 2024-04-23T14:55:06.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.795+0000 7f43a5188740 1 --2- 172.21.15.28:0/631915683 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56430ec5b090 0x56430ec5d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:06.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.795+0000 7f43a5188740 4 client.26981 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:55:06.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.795+0000 7f43a5188740 1 -- 172.21.15.28:0/631915683 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56430ebaabf0 con 0x56430ec5b090 2024-04-23T14:55:06.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.795+0000 7f439aa00640 1 --2- 172.21.15.28:0/631915683 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56430ec5b090 0x56430ec5d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:06.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.796+0000 7f439aa00640 1 --2- 172.21.15.28:0/631915683 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56430ec5b090 0x56430ec5d480 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa5c7b24b1c35aa4 server_cookie=92ef85280a6257d3 in_seq=0 out_seq=0 2024-04-23T14:55:06.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.796+0000 7f4384c00640 10 client.26981 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:55:06.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.799+0000 7f4384c00640 1 -- 172.21.15.28:0/631915683 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56430ebaabf0 con 0x56430ec5b090 2024-04-23T14:55:06.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.799+0000 7f4384c00640 10 client.26981 handle_command_reply: tid=0 2024-04-23T14:55:06.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.799+0000 7f43a5188740 1 client.26981 shutdown 2024-04-23T14:55:06.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.799+0000 7f43a5188740 1 -- 172.21.15.28:0/631915683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56430ec37890 msgr2=0x56430ec4add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:06.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.799+0000 7f43a5188740 1 --2- 172.21.15.28:0/631915683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56430ec37890 0x56430ec4add0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f438c0c5680 tx=0x7f438c02a480 comp rx=0 tx=0).stop 2024-04-23T14:55:06.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.799+0000 7f43a5188740 1 -- 172.21.15.28:0/631915683 shutdown_connections 2024-04-23T14:55:06.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.799+0000 7f43a5188740 1 --2- 172.21.15.28:0/631915683 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56430ec5b090 0x56430ec5d480 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.799+0000 7f43a5188740 1 --2- 172.21.15.28:0/631915683 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56430ec4b340 0x56430ec487b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.799+0000 7f43a5188740 1 --2- 172.21.15.28:0/631915683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430ec38240 0x56430ec480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.799+0000 7f43a5188740 1 --2- 172.21.15.28:0/631915683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56430ec37890 0x56430ec4add0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.799+0000 7f4384c00640 0 client.26981 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:55:06.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.799+0000 7f43a5188740 1 -- 172.21.15.28:0/631915683 >> 172.21.15.28:0/631915683 conn(0x56430ec33e60 msgr2=0x56430ec39c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:06.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.799+0000 7f43a5188740 1 -- 172.21.15.28:0/631915683 shutdown_connections 2024-04-23T14:55:06.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.799+0000 7f43a5188740 1 -- 172.21.15.28:0/631915683 wait complete. 2024-04-23T14:55:06.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.799+0000 7f43a5188740 20 client.26981 trim_cache size 0 max 16384 2024-04-23T14:55:06.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.817+0000 7f43a5188740 1 Processor -- start 2024-04-23T14:55:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.817+0000 7f43a5188740 1 -- start start 2024-04-23T14:55:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.817+0000 7f43a5188740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56430ec37890 0x56430ec56d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.817+0000 7f43a5188740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56430ec38240 0x56430ec48060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.817+0000 7f43a5188740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430ec4b340 0x56430ec4a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.817+0000 7f43a5188740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f439c1a9b50 con 0x56430ec4b340 2024-04-23T14:55:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.817+0000 7f43a5188740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f439c1ac390 con 0x56430ec37890 2024-04-23T14:55:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.817+0000 7f43a5188740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f439c1aa1a0 con 0x56430ec38240 2024-04-23T14:55:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.817+0000 7f439b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56430ec37890 0x56430ec56d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.818+0000 7f439b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56430ec37890 0x56430ec56d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:39452/0 (socket says 172.21.15.28:39452) 2024-04-23T14:55:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.818+0000 7f439b400640 1 -- 172.21.15.28:0/3644830071 learned_addr learned my addr 172.21.15.28:0/3644830071 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.818+0000 7f439aa00640 1 --2- 172.21.15.28:0/3644830071 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56430ec38240 0x56430ec48060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.818+0000 7f439be00640 1 --2- 172.21.15.28:0/3644830071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430ec4b340 0x56430ec4a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.818+0000 7f439b400640 1 -- 172.21.15.28:0/3644830071 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56430ec38240 msgr2=0x56430ec48060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:06.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.818+0000 7f439b400640 1 --2- 172.21.15.28:0/3644830071 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56430ec38240 0x56430ec48060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.818+0000 7f439b400640 1 -- 172.21.15.28:0/3644830071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430ec4b340 msgr2=0x56430ec4a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:06.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.818+0000 7f439b400640 1 --2- 172.21.15.28:0/3644830071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430ec4b340 0x56430ec4a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.818+0000 7f439b400640 1 -- 172.21.15.28:0/3644830071 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56430ec48b20 con 0x56430ec37890 2024-04-23T14:55:06.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.818+0000 7f439be00640 1 --2- 172.21.15.28:0/3644830071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430ec4b340 0x56430ec4a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:55:06.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.818+0000 7f439b400640 1 --2- 172.21.15.28:0/3644830071 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56430ec37890 0x56430ec56d80 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f438c0b93a0 tx=0x7f438c0bb450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e6f176e2523ad server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:06.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.818+0000 7f4387400640 1 -- 172.21.15.28:0/3644830071 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f438c0bc8a0 con 0x56430ec37890 2024-04-23T14:55:06.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.818+0000 7f4387400640 1 -- 172.21.15.28:0/3644830071 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f438c031e50 con 0x56430ec37890 2024-04-23T14:55:06.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.818+0000 7f4387400640 1 -- 172.21.15.28:0/3644830071 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f438c0bcf40 con 0x56430ec37890 2024-04-23T14:55:06.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.819+0000 7f43a5188740 1 -- 172.21.15.28:0/3644830071 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56430ec37890 msgr2=0x56430ec56d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:06.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.819+0000 7f43a5188740 1 --2- 172.21.15.28:0/3644830071 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56430ec37890 0x56430ec56d80 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f438c0b93a0 tx=0x7f438c0bb450 comp rx=0 tx=0).stop 2024-04-23T14:55:06.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.819+0000 7f43a5188740 1 -- 172.21.15.28:0/3644830071 shutdown_connections 2024-04-23T14:55:06.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.819+0000 7f43a5188740 1 --2- 172.21.15.28:0/3644830071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430ec4b340 0x56430ec4a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.819+0000 7f43a5188740 1 --2- 172.21.15.28:0/3644830071 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56430ec38240 0x56430ec48060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.819+0000 7f43a5188740 1 --2- 172.21.15.28:0/3644830071 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56430ec37890 0x56430ec56d80 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.819+0000 7f43a5188740 1 -- 172.21.15.28:0/3644830071 >> 172.21.15.28:0/3644830071 conn(0x56430ec33e60 msgr2=0x56430ec47950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:06.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.819+0000 7f43a5188740 1 -- 172.21.15.28:0/3644830071 shutdown_connections 2024-04-23T14:55:06.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.819+0000 7f43a5188740 1 -- 172.21.15.28:0/3644830071 wait complete. 2024-04-23T14:55:06.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.820+0000 7f43a5188740 1 -- start start 2024-04-23T14:55:06.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.820+0000 7f43a5188740 1 Processor -- start 2024-04-23T14:55:06.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.820+0000 7f43a5188740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430ec37890 0x56430ec53820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:06.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.821+0000 7f43a5188740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56430ec38240 0x56430ec4bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:06.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.821+0000 7f43a5188740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56430ec4b340 0x56430ec4c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:06.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.821+0000 7f43a5188740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f439c1aa850 con 0x56430ec37890 2024-04-23T14:55:06.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.821+0000 7f43a5188740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f439c1abcc0 con 0x56430ec38240 2024-04-23T14:55:06.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.821+0000 7f43a5188740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f439c1aaf20 con 0x56430ec4b340 2024-04-23T14:55:06.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.821+0000 7f43a5188740 10 client.0 _ll_register_callbacks cb 0x7f4394009ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:55:06.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.821+0000 7f439aa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56430ec38240 0x56430ec4bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:06.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.821+0000 7f439b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430ec37890 0x56430ec53820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:06.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.821+0000 7f439aa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56430ec38240 0x56430ec4bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:39468/0 (socket says 172.21.15.28:39468) 2024-04-23T14:55:06.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.821+0000 7f439aa00640 1 -- 172.21.15.28:0/1262822632 learned_addr learned my addr 172.21.15.28:0/1262822632 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:06.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.821+0000 7f439b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430ec37890 0x56430ec53820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41380/0 (socket says 172.21.15.28:41380) 2024-04-23T14:55:06.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.821+0000 7f439be00640 1 --2- 172.21.15.28:0/1262822632 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56430ec4b340 0x56430ec4c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:06.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.821+0000 7f439aa00640 1 -- 172.21.15.28:0/1262822632 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56430ec4b340 msgr2=0x56430ec4c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:06.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.821+0000 7f439aa00640 1 --2- 172.21.15.28:0/1262822632 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56430ec4b340 0x56430ec4c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.821+0000 7f439aa00640 1 -- 172.21.15.28:0/1262822632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430ec37890 msgr2=0x56430ec53820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:06.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.821+0000 7f439aa00640 1 --2- 172.21.15.28:0/1262822632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430ec37890 0x56430ec53820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.821+0000 7f439aa00640 1 -- 172.21.15.28:0/1262822632 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56430e9797d0 con 0x56430ec38240 2024-04-23T14:55:06.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.821+0000 7f439be00640 1 --2- 172.21.15.28:0/1262822632 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56430ec4b340 0x56430ec4c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:55:06.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.822+0000 7f439aa00640 1 --2- 172.21.15.28:0/1262822632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56430ec38240 0x56430ec4bfd0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x56430ec3a470 tx=0x7f438801cf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98f30d63961e31b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:06.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.822+0000 7f4384c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:55:06.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.822+0000 7f4384c00640 1 -- 172.21.15.28:0/1262822632 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f438800be50 con 0x56430ec38240 2024-04-23T14:55:06.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.822+0000 7f43a5188740 1 -- 172.21.15.28:0/1262822632 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f438c031e50 con 0x56430ec38240 2024-04-23T14:55:06.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.822+0000 7f4384c00640 1 -- 172.21.15.28:0/1262822632 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8586+0+0 (secure 0 0 0) 0x7f4388015a60 con 0x56430ec38240 2024-04-23T14:55:06.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.822+0000 7f4384c00640 1 -- 172.21.15.28:0/1262822632 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f4388015ca0 con 0x56430ec38240 2024-04-23T14:55:06.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.822+0000 7f43a5188740 10 client.26987 fetch_fsmap learned FSMap version 20 2024-04-23T14:55:06.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.822+0000 7f43a5188740 1 -- 172.21.15.28:0/1262822632 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x56430ecc1500 con 0x56430ec38240 2024-04-23T14:55:06.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.823+0000 7f4384c00640 1 -- 172.21.15.28:0/1262822632 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f438c031e50 con 0x56430ec38240 2024-04-23T14:55:06.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.823+0000 7f4384c00640 1 -- 172.21.15.28:0/1262822632 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f4388024810 con 0x56430ec38240 2024-04-23T14:55:06.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.823+0000 7f43a5188740 10 client.26987 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:55:06.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.823+0000 7f43a5188740 10 client.26987 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:55:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.823+0000 7f43a5188740 20 client.26987 populate_metadata read hostname 'smithi028' 2024-04-23T14:55:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.823+0000 7f43a5188740 1 --2- 172.21.15.28:0/1262822632 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56430ecc1cb0 0x56430ecc40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.823+0000 7f43a5188740 4 client.26987 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-23T14:55:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.823+0000 7f43a5188740 1 -- 172.21.15.28:0/1262822632 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56430eba4a60 con 0x56430ecc1cb0 2024-04-23T14:55:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.823+0000 7f439b400640 1 --2- 172.21.15.28:0/1262822632 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56430ecc1cb0 0x56430ecc40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.824+0000 7f439b400640 1 --2- 172.21.15.28:0/1262822632 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56430ecc1cb0 0x56430ecc40d0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1027ab0cf838175 server_cookie=af6f89a9c9b28a49 in_seq=0 out_seq=0 2024-04-23T14:55:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.824+0000 7f4384c00640 10 client.26987 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:55:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.825+0000 7f4384c00640 1 -- 172.21.15.28:0/1262822632 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56430eba4a60 con 0x56430ecc1cb0 2024-04-23T14:55:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.825+0000 7f4384c00640 10 client.26987 handle_command_reply: tid=0 2024-04-23T14:55:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.825+0000 7f43a5188740 1 client.26987 shutdown 2024-04-23T14:55:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.825+0000 7f43a5188740 1 -- 172.21.15.28:0/1262822632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56430ec38240 msgr2=0x56430ec4bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.825+0000 7f43a5188740 1 --2- 172.21.15.28:0/1262822632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56430ec38240 0x56430ec4bfd0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x56430ec3a470 tx=0x7f438801cf20 comp rx=0 tx=0).stop 2024-04-23T14:55:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.826+0000 7f43a5188740 1 -- 172.21.15.28:0/1262822632 shutdown_connections 2024-04-23T14:55:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.826+0000 7f43a5188740 1 --2- 172.21.15.28:0/1262822632 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56430ecc1cb0 0x56430ecc40d0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.826+0000 7f43a5188740 1 --2- 172.21.15.28:0/1262822632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56430ec38240 0x56430ec4bfd0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.826+0000 7f4384c00640 0 client.26987 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:55:06.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.826+0000 7f43a5188740 1 --2- 172.21.15.28:0/1262822632 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56430ec4b340 0x56430ec4c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.826+0000 7f43a5188740 1 --2- 172.21.15.28:0/1262822632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430ec37890 0x56430ec53820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.826+0000 7f43a5188740 1 -- 172.21.15.28:0/1262822632 >> 172.21.15.28:0/1262822632 conn(0x56430ec33e60 msgr2=0x56430ec39c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:06.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.826+0000 7f43a5188740 1 -- 172.21.15.28:0/1262822632 shutdown_connections 2024-04-23T14:55:06.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.826+0000 7f43a5188740 1 -- 172.21.15.28:0/1262822632 wait complete. 2024-04-23T14:55:06.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.826+0000 7f43a5188740 20 client.26987 trim_cache size 0 max 16384 2024-04-23T14:55:06.829 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-23T14:55:06.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.826+0000 7f43a0c00640 1 -- 172.21.15.28:0/2714664354 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f43800a3da0 msgr2=0x7f43800a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:06.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.826+0000 7f43a0c00640 1 --2- 172.21.15.28:0/2714664354 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f43800a3da0 0x7f43800a6260 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4394005f40 tx=0x7f4394004a40 comp rx=0 tx=0).stop 2024-04-23T14:55:06.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.826+0000 7f43a0c00640 1 -- 172.21.15.28:0/2714664354 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f439c103060 msgr2=0x7f439c1a06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:06.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.826+0000 7f43a0c00640 1 --2- 172.21.15.28:0/2714664354 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f439c103060 0x7f439c1a06e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f438c004770 tx=0x7f438c003c10 comp rx=0 tx=0).stop 2024-04-23T14:55:06.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.827+0000 7f43a0c00640 1 -- 172.21.15.28:0/2714664354 shutdown_connections 2024-04-23T14:55:06.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.827+0000 7f43a0c00640 1 --2- 172.21.15.28:0/2714664354 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f43800a3da0 0x7f43800a6260 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.827+0000 7f43a0c00640 1 --2- 172.21.15.28:0/2714664354 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f439c10c480 0x7f439c19a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.827+0000 7f43a0c00640 1 --2- 172.21.15.28:0/2714664354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f439c103980 0x7f439c1a0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.827+0000 7f43a0c00640 1 --2- 172.21.15.28:0/2714664354 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f439c103060 0x7f439c1a06e0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:06.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.827+0000 7f43a0c00640 1 -- 172.21.15.28:0/2714664354 >> 172.21.15.28:0/2714664354 conn(0x7f439c0fc880 msgr2=0x7f439c100be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:06.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.827+0000 7f43a0c00640 1 -- 172.21.15.28:0/2714664354 shutdown_connections 2024-04-23T14:55:06.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:06.827+0000 7f43a0c00640 1 -- 172.21.15.28:0/2714664354 wait complete. 2024-04-23T14:55:07.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:06 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/4124052546' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:55:07.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:06 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/4124052546' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:55:07.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:06 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/4124052546' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:55:07.531 INFO:tasks.workunit.client.0.smithi028.stdout: 1 946999 8.74 MB/sec execute 443 sec latency 38.766 ms 2024-04-23T14:55:07.846 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-23T14:55:07.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:55:07.846 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-23T14:55:08.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.078+0000 7f7c7a400640 1 -- 172.21.15.28:0/4287165288 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c74102130 msgr2=0x7f7c74102510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:08.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.078+0000 7f7c7a400640 1 --2- 172.21.15.28:0/4287165288 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c74102130 0x7f7c74102510 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f7c64009900 tx=0x7f7c6402a3f0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.079+0000 7f7c7a400640 1 -- 172.21.15.28:0/4287165288 shutdown_connections 2024-04-23T14:55:08.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.079+0000 7f7c7a400640 1 --2- 172.21.15.28:0/4287165288 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c74107200 0x7f7c7410edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.079+0000 7f7c7a400640 1 --2- 172.21.15.28:0/4287165288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c74102ae0 0x7f7c74106b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.079+0000 7f7c7a400640 1 --2- 172.21.15.28:0/4287165288 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c74102130 0x7f7c74102510 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.079+0000 7f7c7a400640 1 -- 172.21.15.28:0/4287165288 >> 172.21.15.28:0/4287165288 conn(0x7f7c740fd9d0 msgr2=0x7f7c740ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:08.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.079+0000 7f7c7a400640 1 -- 172.21.15.28:0/4287165288 shutdown_connections 2024-04-23T14:55:08.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.079+0000 7f7c7a400640 1 -- 172.21.15.28:0/4287165288 wait complete. 2024-04-23T14:55:08.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.080+0000 7f7c7a400640 1 Processor -- start 2024-04-23T14:55:08.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.080+0000 7f7c7a400640 1 -- start start 2024-04-23T14:55:08.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.081+0000 7f7c7a400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c74102130 0x7f7c741a06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.081+0000 7f7c7a400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c74107200 0x7f7c741a0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.081+0000 7f7c7a400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c7419a8a0 0x7f7c7419ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.081+0000 7f7c7a400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7c741122e0 con 0x7f7c74102130 2024-04-23T14:55:08.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.081+0000 7f7c7a400640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7c74112160 con 0x7f7c7419a8a0 2024-04-23T14:55:08.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.081+0000 7f7c7a400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7c74112460 con 0x7f7c74107200 2024-04-23T14:55:08.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.081+0000 7f7c6be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c74107200 0x7f7c741a0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.081+0000 7f7c6be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c74107200 0x7f7c741a0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:43878/0 (socket says 172.21.15.28:43878) 2024-04-23T14:55:08.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.081+0000 7f7c6be00640 1 -- 172.21.15.28:0/3072928575 learned_addr learned my addr 172.21.15.28:0/3072928575 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:08.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.081+0000 7f7c79000640 1 --2- 172.21.15.28:0/3072928575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c74102130 0x7f7c741a06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.081+0000 7f7c79a00640 1 --2- 172.21.15.28:0/3072928575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c7419a8a0 0x7f7c7419ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.081+0000 7f7c6be00640 1 -- 172.21.15.28:0/3072928575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c7419a8a0 msgr2=0x7f7c7419ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:08.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.081+0000 7f7c6be00640 1 --2- 172.21.15.28:0/3072928575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c7419a8a0 0x7f7c7419ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.081+0000 7f7c6be00640 1 -- 172.21.15.28:0/3072928575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c74102130 msgr2=0x7f7c741a06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:08.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.081+0000 7f7c6be00640 1 --2- 172.21.15.28:0/3072928575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c74102130 0x7f7c741a06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.081+0000 7f7c6be00640 1 -- 172.21.15.28:0/3072928575 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c7419b3c0 con 0x7f7c74107200 2024-04-23T14:55:08.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.082+0000 7f7c79a00640 1 --2- 172.21.15.28:0/3072928575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c7419a8a0 0x7f7c7419ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:55:08.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.082+0000 7f7c6be00640 1 --2- 172.21.15.28:0/3072928575 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c74107200 0x7f7c741a0c30 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c00e9c0 tx=0x7f7c5c00edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:08.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.082+0000 7f7c72000640 1 -- 172.21.15.28:0/3072928575 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c00ccc0 con 0x7f7c74107200 2024-04-23T14:55:08.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.082+0000 7f7c7a400640 1 -- 172.21.15.28:0/3072928575 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c7419b6b0 con 0x7f7c74107200 2024-04-23T14:55:08.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.082+0000 7f7c72000640 1 -- 172.21.15.28:0/3072928575 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c5c005500 con 0x7f7c74107200 2024-04-23T14:55:08.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.082+0000 7f7c72000640 1 -- 172.21.15.28:0/3072928575 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c010430 con 0x7f7c74107200 2024-04-23T14:55:08.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.082+0000 7f7c7a400640 1 -- 172.21.15.28:0/3072928575 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c741a7950 con 0x7f7c74107200 2024-04-23T14:55:08.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.085+0000 7f7c72000640 1 -- 172.21.15.28:0/3072928575 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f7c5c005000 con 0x7f7c74107200 2024-04-23T14:55:08.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.086+0000 7f7c7e85e740 1 Processor -- start 2024-04-23T14:55:08.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.086+0000 7f7c7e85e740 1 -- start start 2024-04-23T14:55:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.086+0000 7f7c7e85e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a8dcb3270 0x558a8dcb3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.086+0000 7f7c7e85e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558a8dcb3b90 0x558a8dcbdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.086+0000 7f7c7e85e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a8dcbe260 0x558a8dcc0650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.086+0000 7f7c7e85e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558a8dc18e70 con 0x558a8dcb3270 2024-04-23T14:55:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.086+0000 7f7c7e85e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558a8dbbf380 con 0x558a8dcb3b90 2024-04-23T14:55:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.086+0000 7f7c7e85e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558a8dc14e30 con 0x558a8dcbe260 2024-04-23T14:55:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.087+0000 7f7c79000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558a8dcb3b90 0x558a8dcbdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.087+0000 7f7c79a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a8dcb3270 0x558a8dcb3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.087+0000 7f7c6be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a8dcbe260 0x558a8dcc0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.087+0000 7f7c79000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558a8dcb3b90 0x558a8dcbdd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:36394/0 (socket says 172.21.15.28:36394) 2024-04-23T14:55:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.087+0000 7f7c79000640 1 -- 172.21.15.28:0/942939089 learned_addr learned my addr 172.21.15.28:0/942939089 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.087+0000 7f7c79000640 1 -- 172.21.15.28:0/942939089 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a8dcbe260 msgr2=0x558a8dcc0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:08.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.087+0000 7f7c79000640 1 --2- 172.21.15.28:0/942939089 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a8dcbe260 0x558a8dcc0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.087+0000 7f7c79000640 1 -- 172.21.15.28:0/942939089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a8dcb3270 msgr2=0x558a8dcb3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:08.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.087+0000 7f7c79000640 1 --2- 172.21.15.28:0/942939089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a8dcb3270 0x558a8dcb3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.087+0000 7f7c79000640 1 -- 172.21.15.28:0/942939089 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a8d9f37e0 con 0x558a8dcb3b90 2024-04-23T14:55:08.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.087+0000 7f7c72000640 1 --2- 172.21.15.28:0/3072928575 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7c580a3da0 0x7f7c580a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.087+0000 7f7c79000640 1 --2- 172.21.15.28:0/942939089 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558a8dcb3b90 0x558a8dcbdd20 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f7c64009900 tx=0x7f7c64005de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8661a75dd0e18ee9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:08.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.088+0000 7f7c72000640 1 -- 172.21.15.28:0/3072928575 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8586+0+0 (secure 0 0 0) 0x7f7c5c0d0c40 con 0x7f7c74107200 2024-04-23T14:55:08.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.088+0000 7f7c6a000640 1 -- 172.21.15.28:0/942939089 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c64035040 con 0x558a8dcb3b90 2024-04-23T14:55:08.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.088+0000 7f7c6a000640 1 -- 172.21.15.28:0/942939089 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c64005520 con 0x558a8dcb3b90 2024-04-23T14:55:08.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.088+0000 7f7c79a00640 1 --2- 172.21.15.28:0/3072928575 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7c580a3da0 0x7f7c580a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.088+0000 7f7c6a000640 1 -- 172.21.15.28:0/942939089 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c640035f0 con 0x558a8dcb3b90 2024-04-23T14:55:08.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.088+0000 7f7c7e85e740 1 -- 172.21.15.28:0/942939089 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558a8dcb3b90 msgr2=0x558a8dcbdd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:08.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.088+0000 7f7c7e85e740 1 --2- 172.21.15.28:0/942939089 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558a8dcb3b90 0x558a8dcbdd20 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f7c64009900 tx=0x7f7c64005de0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.089+0000 7f7c79a00640 1 --2- 172.21.15.28:0/3072928575 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7c580a3da0 0x7f7c580a6260 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c005fd0 tx=0x7f7c6c009500 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:08.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.089+0000 7f7c7e85e740 1 -- 172.21.15.28:0/942939089 shutdown_connections 2024-04-23T14:55:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.089+0000 7f7c7e85e740 1 --2- 172.21.15.28:0/942939089 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558a8dcb3b90 0x558a8dcbdd20 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.089+0000 7f7c7e85e740 1 --2- 172.21.15.28:0/942939089 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a8dcbe260 0x558a8dcc0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.089+0000 7f7c7e85e740 1 --2- 172.21.15.28:0/942939089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a8dcb3270 0x558a8dcb3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.089+0000 7f7c7e85e740 1 -- 172.21.15.28:0/942939089 >> 172.21.15.28:0/942939089 conn(0x558a8dcaf590 msgr2=0x558a8dcaf970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.089+0000 7f7c7e85e740 1 -- 172.21.15.28:0/942939089 shutdown_connections 2024-04-23T14:55:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.089+0000 7f7c7e85e740 1 -- 172.21.15.28:0/942939089 wait complete. 2024-04-23T14:55:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.089+0000 7f7c7e85e740 1 -- start start 2024-04-23T14:55:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.089+0000 7f7c7e85e740 1 Processor -- start 2024-04-23T14:55:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.090+0000 7f7c7e85e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a8dcb3270 0x558a8dcbb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.090+0000 7f7c7e85e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a8dcb3b90 0x558a8dcbb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.090+0000 7f7c7e85e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558a8dcbe260 0x558a8dcbbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.090+0000 7f7c7e85e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558a8dcc0ff0 con 0x558a8dcb3270 2024-04-23T14:55:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.090+0000 7f7c7e85e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558a8dbd2660 con 0x558a8dcbe260 2024-04-23T14:55:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.090+0000 7f7c7e85e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558a8dcc1170 con 0x558a8dcb3b90 2024-04-23T14:55:08.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.090+0000 7f7c7e85e740 10 client.0 _ll_register_callbacks cb 0x558a8dbc0aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:55:08.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.090+0000 7f7c79a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558a8dcbe260 0x558a8dcbbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.090+0000 7f7c79000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a8dcb3270 0x558a8dcbb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.090+0000 7f7c6be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a8dcb3b90 0x558a8dcbb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.090+0000 7f7c79a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558a8dcbe260 0x558a8dcbbc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:36400/0 (socket says 172.21.15.28:36400) 2024-04-23T14:55:08.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.090+0000 7f7c79a00640 1 -- 172.21.15.28:0/3480006231 learned_addr learned my addr 172.21.15.28:0/3480006231 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:08.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.091+0000 7f7c6be00640 1 -- 172.21.15.28:0/3480006231 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558a8dcbe260 msgr2=0x558a8dcbbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:08.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.091+0000 7f7c6be00640 1 --2- 172.21.15.28:0/3480006231 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558a8dcbe260 0x558a8dcbbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.091+0000 7f7c6be00640 1 -- 172.21.15.28:0/3480006231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a8dcb3270 msgr2=0x558a8dcbb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:08.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.091+0000 7f7c6be00640 1 --2- 172.21.15.28:0/3480006231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a8dcb3270 0x558a8dcbb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.091+0000 7f7c6be00640 1 -- 172.21.15.28:0/3480006231 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a8da53f70 con 0x558a8dcb3b90 2024-04-23T14:55:08.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.091+0000 7f7c79a00640 1 --2- 172.21.15.28:0/3480006231 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558a8dcbe260 0x558a8dcbbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:55:08.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.091+0000 7f7c6be00640 1 --2- 172.21.15.28:0/3480006231 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a8dcb3b90 0x558a8dcbb6f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c0ce2f0 tx=0x7f7c5c00ce60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e953f77a69ecfa62 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:08.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.091+0000 7f7c62a00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:55:08.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.091+0000 7f7c62a00640 1 -- 172.21.15.28:0/3480006231 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c0d4020 con 0x558a8dcb3b90 2024-04-23T14:55:08.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.091+0000 7f7c7e85e740 1 -- 172.21.15.28:0/3480006231 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c64005520 con 0x558a8dcb3b90 2024-04-23T14:55:08.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.091+0000 7f7c62a00640 1 -- 172.21.15.28:0/3480006231 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8586+0+0 (secure 0 0 0) 0x7f7c5c0ce570 con 0x558a8dcb3b90 2024-04-23T14:55:08.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.091+0000 7f7c62a00640 1 -- 172.21.15.28:0/3480006231 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f7c5c09c680 con 0x558a8dcb3b90 2024-04-23T14:55:08.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.091+0000 7f7c7e85e740 10 client.26980 fetch_fsmap learned FSMap version 20 2024-04-23T14:55:08.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.091+0000 7f7c7e85e740 1 -- 172.21.15.28:0/3480006231 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x558a8d8eb170 con 0x558a8dcb3b90 2024-04-23T14:55:08.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.092+0000 7f7c62a00640 1 -- 172.21.15.28:0/3480006231 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c64005520 con 0x558a8dcb3b90 2024-04-23T14:55:08.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.092+0000 7f7c62a00640 1 -- 172.21.15.28:0/3480006231 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f7c5c0a5320 con 0x558a8dcb3b90 2024-04-23T14:55:08.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.092+0000 7f7c7e85e740 10 client.26980 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:55:08.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.092+0000 7f7c7e85e740 10 client.26980 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:55:08.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.092+0000 7f7c7e85e740 20 client.26980 populate_metadata read hostname 'smithi028' 2024-04-23T14:55:08.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.092+0000 7f7c7e85e740 1 --2- 172.21.15.28:0/3480006231 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558a8dcce530 0x558a8dcd0920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.092+0000 7f7c7e85e740 4 client.26980 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:55:08.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.092+0000 7f7c7e85e740 1 -- 172.21.15.28:0/3480006231 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558a8dc26890 con 0x558a8dcce530 2024-04-23T14:55:08.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.092+0000 7f7c79000640 1 --2- 172.21.15.28:0/3480006231 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558a8dcce530 0x558a8dcd0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.093+0000 7f7c79000640 1 --2- 172.21.15.28:0/3480006231 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558a8dcce530 0x558a8dcd0920 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a85eca512d13054 server_cookie=41d4b1b55e6c4771 in_seq=0 out_seq=0 2024-04-23T14:55:08.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.093+0000 7f7c62a00640 10 client.26980 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:55:08.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.094+0000 7f7c62a00640 1 -- 172.21.15.28:0/3480006231 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558a8dc26890 con 0x558a8dcce530 2024-04-23T14:55:08.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.094+0000 7f7c62a00640 10 client.26980 handle_command_reply: tid=0 2024-04-23T14:55:08.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.094+0000 7f7c7e85e740 1 client.26980 shutdown 2024-04-23T14:55:08.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.094+0000 7f7c7e85e740 1 -- 172.21.15.28:0/3480006231 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a8dcb3b90 msgr2=0x558a8dcbb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:08.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.094+0000 7f7c7e85e740 1 --2- 172.21.15.28:0/3480006231 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a8dcb3b90 0x558a8dcbb6f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c0ce2f0 tx=0x7f7c5c00ce60 comp rx=0 tx=0).stop 2024-04-23T14:55:08.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.094+0000 7f7c7e85e740 1 -- 172.21.15.28:0/3480006231 shutdown_connections 2024-04-23T14:55:08.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.094+0000 7f7c7e85e740 1 --2- 172.21.15.28:0/3480006231 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558a8dcce530 0x558a8dcd0920 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.094+0000 7f7c7e85e740 1 --2- 172.21.15.28:0/3480006231 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558a8dcbe260 0x558a8dcbbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.094+0000 7f7c7e85e740 1 --2- 172.21.15.28:0/3480006231 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a8dcb3b90 0x558a8dcbb6f0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.094+0000 7f7c7e85e740 1 --2- 172.21.15.28:0/3480006231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a8dcb3270 0x558a8dcbb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.094+0000 7f7c7e85e740 1 -- 172.21.15.28:0/3480006231 >> 172.21.15.28:0/3480006231 conn(0x558a8dcaf590 msgr2=0x558a8dcb5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:08.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.094+0000 7f7c62a00640 0 client.26980 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:55:08.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.094+0000 7f7c7e85e740 1 -- 172.21.15.28:0/3480006231 shutdown_connections 2024-04-23T14:55:08.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.094+0000 7f7c7e85e740 1 -- 172.21.15.28:0/3480006231 wait complete. 2024-04-23T14:55:08.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.094+0000 7f7c7e85e740 20 client.26980 trim_cache size 0 max 16384 2024-04-23T14:55:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.115+0000 7f7c7e85e740 1 Processor -- start 2024-04-23T14:55:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.115+0000 7f7c7e85e740 1 -- start start 2024-04-23T14:55:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.116+0000 7f7c7e85e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a8dcb3270 0x558a8dcc9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.116+0000 7f7c7e85e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558a8dcb3b90 0x558a8dcae570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.116+0000 7f7c7e85e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a8dcbe260 0x558a8dcd6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.116+0000 7f7c7e85e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7c741adf20 con 0x558a8dcbe260 2024-04-23T14:55:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.116+0000 7f7c7e85e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7c741b0780 con 0x558a8dcb3b90 2024-04-23T14:55:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.116+0000 7f7c7e85e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7c741ae570 con 0x558a8dcb3270 2024-04-23T14:55:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.116+0000 7f7c79a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a8dcbe260 0x558a8dcd6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.116+0000 7f7c6be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558a8dcb3b90 0x558a8dcae570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.116+0000 7f7c6be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558a8dcb3b90 0x558a8dcae570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:36404/0 (socket says 172.21.15.28:36404) 2024-04-23T14:55:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.116+0000 7f7c6be00640 1 -- 172.21.15.28:0/2943417647 learned_addr learned my addr 172.21.15.28:0/2943417647 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:08.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.116+0000 7f7c79000640 1 --2- 172.21.15.28:0/2943417647 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a8dcb3270 0x558a8dcc9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.117+0000 7f7c6be00640 1 -- 172.21.15.28:0/2943417647 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a8dcb3270 msgr2=0x558a8dcc9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:08.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.117+0000 7f7c6be00640 1 --2- 172.21.15.28:0/2943417647 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a8dcb3270 0x558a8dcc9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.117+0000 7f7c6be00640 1 -- 172.21.15.28:0/2943417647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a8dcbe260 msgr2=0x558a8dcd6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:08.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.117+0000 7f7c6be00640 1 --2- 172.21.15.28:0/2943417647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a8dcbe260 0x558a8dcd6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.117+0000 7f7c6be00640 1 -- 172.21.15.28:0/2943417647 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a8dcba160 con 0x558a8dcb3b90 2024-04-23T14:55:08.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.117+0000 7f7c6be00640 1 --2- 172.21.15.28:0/2943417647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558a8dcb3b90 0x558a8dcae570 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c099510 tx=0x7f7c5c09b050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a7338defb6ae705 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:08.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.117+0000 7f7c69600640 1 -- 172.21.15.28:0/2943417647 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c09c4a0 con 0x558a8dcb3b90 2024-04-23T14:55:08.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.117+0000 7f7c69600640 1 -- 172.21.15.28:0/2943417647 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c5c0d4070 con 0x558a8dcb3b90 2024-04-23T14:55:08.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.117+0000 7f7c69600640 1 -- 172.21.15.28:0/2943417647 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c0ce570 con 0x558a8dcb3b90 2024-04-23T14:55:08.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.118+0000 7f7c7e85e740 1 -- 172.21.15.28:0/2943417647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558a8dcb3b90 msgr2=0x558a8dcae570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:08.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.118+0000 7f7c7e85e740 1 --2- 172.21.15.28:0/2943417647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558a8dcb3b90 0x558a8dcae570 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c099510 tx=0x7f7c5c09b050 comp rx=0 tx=0).stop 2024-04-23T14:55:08.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.118+0000 7f7c7e85e740 1 -- 172.21.15.28:0/2943417647 shutdown_connections 2024-04-23T14:55:08.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.118+0000 7f7c7e85e740 1 --2- 172.21.15.28:0/2943417647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558a8dcb3b90 0x558a8dcae570 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.118+0000 7f7c7e85e740 1 --2- 172.21.15.28:0/2943417647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a8dcbe260 0x558a8dcd6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.118+0000 7f7c7e85e740 1 --2- 172.21.15.28:0/2943417647 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a8dcb3270 0x558a8dcc9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.118+0000 7f7c7e85e740 1 -- 172.21.15.28:0/2943417647 >> 172.21.15.28:0/2943417647 conn(0x558a8dcaf590 msgr2=0x558a8dcade60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:08.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.118+0000 7f7c7e85e740 1 -- 172.21.15.28:0/2943417647 shutdown_connections 2024-04-23T14:55:08.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.118+0000 7f7c7e85e740 1 -- 172.21.15.28:0/2943417647 wait complete. 2024-04-23T14:55:08.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.119+0000 7f7c7e85e740 1 -- start start 2024-04-23T14:55:08.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.119+0000 7f7c7e85e740 1 Processor -- start 2024-04-23T14:55:08.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.119+0000 7f7c7e85e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a8dcb3270 0x558a8dcc20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.119+0000 7f7c7e85e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a8dcb3b90 0x558a8dcc2610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.119+0000 7f7c7e85e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558a8dcbe260 0x558a8dcc2b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.119+0000 7f7c7e85e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7c741aec40 con 0x558a8dcb3b90 2024-04-23T14:55:08.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.119+0000 7f7c7e85e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7c741b00b0 con 0x558a8dcbe260 2024-04-23T14:55:08.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.119+0000 7f7c7e85e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7c741af310 con 0x558a8dcb3270 2024-04-23T14:55:08.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.120+0000 7f7c7e85e740 10 client.0 _ll_register_callbacks cb 0x7f7c5c0d43e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:55:08.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.120+0000 7f7c79000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a8dcb3270 0x558a8dcc20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.120+0000 7f7c79000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a8dcb3270 0x558a8dcc20d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:43912/0 (socket says 172.21.15.28:43912) 2024-04-23T14:55:08.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.120+0000 7f7c79000640 1 -- 172.21.15.28:0/2157729152 learned_addr learned my addr 172.21.15.28:0/2157729152 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:08.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.120+0000 7f7c79a00640 1 --2- 172.21.15.28:0/2157729152 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558a8dcbe260 0x558a8dcc2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.120+0000 7f7c6be00640 1 --2- 172.21.15.28:0/2157729152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a8dcb3b90 0x558a8dcc2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.120+0000 7f7c79000640 1 -- 172.21.15.28:0/2157729152 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558a8dcbe260 msgr2=0x558a8dcc2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:08.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.120+0000 7f7c79000640 1 --2- 172.21.15.28:0/2157729152 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558a8dcbe260 0x558a8dcc2b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.120+0000 7f7c79000640 1 -- 172.21.15.28:0/2157729152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a8dcb3b90 msgr2=0x558a8dcc2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:08.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.120+0000 7f7c79000640 1 --2- 172.21.15.28:0/2157729152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a8dcb3b90 0x558a8dcc2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.120+0000 7f7c79000640 1 -- 172.21.15.28:0/2157729152 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a8dd33ce0 con 0x558a8dcb3270 2024-04-23T14:55:08.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.120+0000 7f7c79000640 1 --2- 172.21.15.28:0/2157729152 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a8dcb3270 0x558a8dcc20d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7c6402a9e0 tx=0x7f7c6402aa70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc70176d94678267 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:08.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.120+0000 7f7c62a00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:55:08.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.121+0000 7f7c62a00640 1 -- 172.21.15.28:0/2157729152 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c64035040 con 0x558a8dcb3270 2024-04-23T14:55:08.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.121+0000 7f7c62a00640 1 -- 172.21.15.28:0/2157729152 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8586+0+0 (secure 0 0 0) 0x7f7c640042c0 con 0x558a8dcb3270 2024-04-23T14:55:08.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.121+0000 7f7c7e85e740 1 -- 172.21.15.28:0/2157729152 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c5c0d4070 con 0x558a8dcb3270 2024-04-23T14:55:08.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.121+0000 7f7c62a00640 1 -- 172.21.15.28:0/2157729152 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f7c640044e0 con 0x558a8dcb3270 2024-04-23T14:55:08.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.121+0000 7f7c7e85e740 10 client.26986 fetch_fsmap learned FSMap version 20 2024-04-23T14:55:08.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.121+0000 7f7c7e85e740 1 -- 172.21.15.28:0/2157729152 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x558a8dd35080 con 0x558a8dcb3270 2024-04-23T14:55:08.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.121+0000 7f7c62a00640 1 -- 172.21.15.28:0/2157729152 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c5c0d4070 con 0x558a8dcb3270 2024-04-23T14:55:08.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.121+0000 7f7c62a00640 1 -- 172.21.15.28:0/2157729152 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f7c6405b030 con 0x558a8dcb3270 2024-04-23T14:55:08.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.121+0000 7f7c7e85e740 10 client.26986 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:55:08.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.121+0000 7f7c7e85e740 10 client.26986 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:55:08.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.121+0000 7f7c7e85e740 20 client.26986 populate_metadata read hostname 'smithi028' 2024-04-23T14:55:08.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.121+0000 7f7c7e85e740 1 --2- 172.21.15.28:0/2157729152 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558a8dd353c0 0x558a8dd377b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.121+0000 7f7c7e85e740 4 client.26986 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-23T14:55:08.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.121+0000 7f7c7e85e740 1 -- 172.21.15.28:0/2157729152 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558a8dc20700 con 0x558a8dd353c0 2024-04-23T14:55:08.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.122+0000 7f7c6be00640 1 --2- 172.21.15.28:0/2157729152 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558a8dd353c0 0x558a8dd377b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.122+0000 7f7c6be00640 1 --2- 172.21.15.28:0/2157729152 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558a8dd353c0 0x558a8dd377b0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2ec0e60239aa02e server_cookie=d040945dbe9a21a1 in_seq=0 out_seq=0 2024-04-23T14:55:08.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.122+0000 7f7c62a00640 10 client.26986 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:55:08.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.124+0000 7f7c62a00640 1 -- 172.21.15.28:0/2157729152 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558a8dc20700 con 0x558a8dd353c0 2024-04-23T14:55:08.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.124+0000 7f7c62a00640 10 client.26986 handle_command_reply: tid=0 2024-04-23T14:55:08.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.124+0000 7f7c7e85e740 1 client.26986 shutdown 2024-04-23T14:55:08.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.125+0000 7f7c7e85e740 1 -- 172.21.15.28:0/2157729152 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a8dcb3270 msgr2=0x558a8dcc20d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:08.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.125+0000 7f7c7e85e740 1 --2- 172.21.15.28:0/2157729152 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a8dcb3270 0x558a8dcc20d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7c6402a9e0 tx=0x7f7c6402aa70 comp rx=0 tx=0).stop 2024-04-23T14:55:08.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.125+0000 7f7c7e85e740 1 -- 172.21.15.28:0/2157729152 shutdown_connections 2024-04-23T14:55:08.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.125+0000 7f7c7e85e740 1 --2- 172.21.15.28:0/2157729152 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558a8dd353c0 0x558a8dd377b0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.125+0000 7f7c7e85e740 1 --2- 172.21.15.28:0/2157729152 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558a8dcbe260 0x558a8dcc2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.125+0000 7f7c7e85e740 1 --2- 172.21.15.28:0/2157729152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a8dcb3b90 0x558a8dcc2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.125+0000 7f7c62a00640 0 client.26986 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:55:08.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.125+0000 7f7c7e85e740 1 --2- 172.21.15.28:0/2157729152 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a8dcb3270 0x558a8dcc20d0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.125+0000 7f7c7e85e740 1 -- 172.21.15.28:0/2157729152 >> 172.21.15.28:0/2157729152 conn(0x558a8dcaf590 msgr2=0x558a8dcae6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:08.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.125+0000 7f7c7e85e740 1 -- 172.21.15.28:0/2157729152 shutdown_connections 2024-04-23T14:55:08.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.125+0000 7f7c7e85e740 1 -- 172.21.15.28:0/2157729152 wait complete. 2024-04-23T14:55:08.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.125+0000 7f7c7e85e740 20 client.26986 trim_cache size 0 max 16384 2024-04-23T14:55:08.128 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:55:08.128 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-23T14:55:08.128 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "555e6dc6-5bc6-42d2-b13d-17deecc233a8", 2024-04-23T14:55:08.128 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-23T14:55:08.128 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:55:08.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.126+0000 7f7c7a400640 1 -- 172.21.15.28:0/3072928575 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7c580a3da0 msgr2=0x7f7c580a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:08.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.126+0000 7f7c7a400640 1 --2- 172.21.15.28:0/3072928575 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7c580a3da0 0x7f7c580a6260 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c005fd0 tx=0x7f7c6c009500 comp rx=0 tx=0).stop 2024-04-23T14:55:08.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.126+0000 7f7c7a400640 1 -- 172.21.15.28:0/3072928575 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c74107200 msgr2=0x7f7c741a0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:08.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.126+0000 7f7c7a400640 1 --2- 172.21.15.28:0/3072928575 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c74107200 0x7f7c741a0c30 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c00e9c0 tx=0x7f7c5c00edc0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.126+0000 7f7c7a400640 1 -- 172.21.15.28:0/3072928575 shutdown_connections 2024-04-23T14:55:08.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.126+0000 7f7c7a400640 1 --2- 172.21.15.28:0/3072928575 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f7c580a3da0 0x7f7c580a6260 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.126+0000 7f7c7a400640 1 --2- 172.21.15.28:0/3072928575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c7419a8a0 0x7f7c7419ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.126+0000 7f7c7a400640 1 --2- 172.21.15.28:0/3072928575 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c74107200 0x7f7c741a0c30 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.126+0000 7f7c7a400640 1 --2- 172.21.15.28:0/3072928575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c74102130 0x7f7c741a06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.126+0000 7f7c7a400640 1 -- 172.21.15.28:0/3072928575 >> 172.21.15.28:0/3072928575 conn(0x7f7c740fd9d0 msgr2=0x7f7c740fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:08.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.126+0000 7f7c7a400640 1 -- 172.21.15.28:0/3072928575 shutdown_connections 2024-04-23T14:55:08.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.126+0000 7f7c7a400640 1 -- 172.21.15.28:0/3072928575 wait complete. 2024-04-23T14:55:08.144 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:55:08.145 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:55:08.214 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:07 smithi028 ceph-mon[27810]: pgmap v345: 97 pgs: 97 active+clean; 527 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 195 op/s 2024-04-23T14:55:08.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:07 smithi105 ceph-mon[32555]: pgmap v345: 97 pgs: 97 active+clean; 527 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 195 op/s 2024-04-23T14:55:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.357+0000 7f0703e00640 1 -- 172.21.15.28:0/3297264765 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0704076530 msgr2=0x7f0704076910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.357+0000 7f0703e00640 1 --2- 172.21.15.28:0/3297264765 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0704076530 0x7f0704076910 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f06f4009cd0 tx=0x7f06f402a8f0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.357+0000 7f0703e00640 1 -- 172.21.15.28:0/3297264765 shutdown_connections 2024-04-23T14:55:08.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.357+0000 7f0703e00640 1 --2- 172.21.15.28:0/3297264765 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0704112d40 0x7f0704115130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.357+0000 7f0703e00640 1 --2- 172.21.15.28:0/3297264765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0704076e50 0x7f0704112800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.357+0000 7f0703e00640 1 --2- 172.21.15.28:0/3297264765 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0704076530 0x7f0704076910 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.357+0000 7f0703e00640 1 -- 172.21.15.28:0/3297264765 >> 172.21.15.28:0/3297264765 conn(0x7f07040ffb10 msgr2=0x7f0704101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:08.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.357+0000 7f0703e00640 1 -- 172.21.15.28:0/3297264765 shutdown_connections 2024-04-23T14:55:08.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.357+0000 7f0703e00640 1 -- 172.21.15.28:0/3297264765 wait complete. 2024-04-23T14:55:08.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.358+0000 7f0703e00640 1 Processor -- start 2024-04-23T14:55:08.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.359+0000 7f0703e00640 1 -- start start 2024-04-23T14:55:08.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.359+0000 7f0703e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0704076530 0x7f070419ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.359+0000 7f0703e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0704076e50 0x7f070419f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.359+0000 7f0703e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0704112d40 0x7f07041a36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.359+0000 7f0703e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f07041188b0 con 0x7f0704076530 2024-04-23T14:55:08.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.359+0000 7f0703e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0704118730 con 0x7f0704076e50 2024-04-23T14:55:08.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.359+0000 7f0703e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0704118a30 con 0x7f0704112d40 2024-04-23T14:55:08.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.359+0000 7f06fa000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0704076e50 0x7f070419f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.359+0000 7f0702a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0704076530 0x7f070419ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.359+0000 7f06fa000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0704076e50 0x7f070419f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:36438/0 (socket says 172.21.15.28:36438) 2024-04-23T14:55:08.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.359+0000 7f06fa000640 1 -- 172.21.15.28:0/10943031 learned_addr learned my addr 172.21.15.28:0/10943031 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:08.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.359+0000 7f0703400640 1 --2- 172.21.15.28:0/10943031 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0704112d40 0x7f07041a36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.360+0000 7f06fa000640 1 -- 172.21.15.28:0/10943031 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0704112d40 msgr2=0x7f07041a36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:08.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.360+0000 7f06fa000640 1 --2- 172.21.15.28:0/10943031 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0704112d40 0x7f07041a36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.360+0000 7f06fa000640 1 -- 172.21.15.28:0/10943031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0704076530 msgr2=0x7f070419ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:08.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.360+0000 7f06fa000640 1 --2- 172.21.15.28:0/10943031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0704076530 0x7f070419ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.360+0000 7f06fa000640 1 -- 172.21.15.28:0/10943031 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07041a3dd0 con 0x7f0704076e50 2024-04-23T14:55:08.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.360+0000 7f06fa000640 1 --2- 172.21.15.28:0/10943031 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0704076e50 0x7f070419f370 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f06ec00e9f0 tx=0x7f06ec00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:08.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.360+0000 7f06fbe00640 1 -- 172.21.15.28:0/10943031 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06ec00cd80 con 0x7f0704076e50 2024-04-23T14:55:08.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.360+0000 7f0703e00640 1 -- 172.21.15.28:0/10943031 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07041a40c0 con 0x7f0704076e50 2024-04-23T14:55:08.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.360+0000 7f06fbe00640 1 -- 172.21.15.28:0/10943031 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06ec010910 con 0x7f0704076e50 2024-04-23T14:55:08.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.360+0000 7f06fbe00640 1 -- 172.21.15.28:0/10943031 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06ec0188f0 con 0x7f0704076e50 2024-04-23T14:55:08.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.361+0000 7f0703e00640 1 -- 172.21.15.28:0/10943031 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07041ab950 con 0x7f0704076e50 2024-04-23T14:55:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.364+0000 7f070c834740 1 Processor -- start 2024-04-23T14:55:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.364+0000 7f06fbe00640 1 -- 172.21.15.28:0/10943031 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f06ec010ac0 con 0x7f0704076e50 2024-04-23T14:55:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.364+0000 7f070c834740 1 -- start start 2024-04-23T14:55:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.364+0000 7f070c834740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e52d3ea30 0x561e52d3ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.364+0000 7f070c834740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e52d3f350 0x561e52d494e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.364+0000 7f070c834740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561e52d49a20 0x561e52d4be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.364+0000 7f070c834740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561e52ca41f0 con 0x561e52d3f350 2024-04-23T14:55:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.364+0000 7f070c834740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x561e52c4a350 con 0x561e52d3ea30 2024-04-23T14:55:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.364+0000 7f070c834740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x561e52ca01b0 con 0x561e52d49a20 2024-04-23T14:55:08.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.365+0000 7f0703400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e52d3ea30 0x561e52d3ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.365+0000 7f0703400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e52d3ea30 0x561e52d3ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:36446/0 (socket says 172.21.15.28:36446) 2024-04-23T14:55:08.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.365+0000 7f0702a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e52d3f350 0x561e52d494e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.365+0000 7f0703400640 1 -- 172.21.15.28:0/93244775 learned_addr learned my addr 172.21.15.28:0/93244775 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:08.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.365+0000 7f06fa000640 1 --2- 172.21.15.28:0/93244775 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561e52d49a20 0x561e52d4be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.365+0000 7f0703400640 1 -- 172.21.15.28:0/93244775 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561e52d49a20 msgr2=0x561e52d4be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:08.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.365+0000 7f0703400640 1 --2- 172.21.15.28:0/93244775 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561e52d49a20 0x561e52d4be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.365+0000 7f0703400640 1 -- 172.21.15.28:0/93244775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e52d3f350 msgr2=0x561e52d494e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:08.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.365+0000 7f0703400640 1 --2- 172.21.15.28:0/93244775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e52d3f350 0x561e52d494e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.365+0000 7f0703400640 1 -- 172.21.15.28:0/93244775 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e52c9ef50 con 0x561e52d3ea30 2024-04-23T14:55:08.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.365+0000 7f06fa000640 1 --2- 172.21.15.28:0/93244775 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561e52d49a20 0x561e52d4be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:55:08.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.365+0000 7f0702a00640 1 --2- 172.21.15.28:0/93244775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e52d3f350 0x561e52d494e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:55:08.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.366+0000 7f0703400640 1 --2- 172.21.15.28:0/93244775 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e52d3ea30 0x561e52d3ee10 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f06fc009ba0 tx=0x7f06fc011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24d8a3fb8289c24b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:08.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.366+0000 7f06fbe00640 1 --2- 172.21.15.28:0/10943031 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f06e80a3d20 0x7f06e80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.366+0000 7f06f3e00640 1 -- 172.21.15.28:0/93244775 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06fc004230 con 0x561e52d3ea30 2024-04-23T14:55:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.366+0000 7f06f3e00640 1 -- 172.21.15.28:0/93244775 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06fc009e10 con 0x561e52d3ea30 2024-04-23T14:55:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.366+0000 7f06f3e00640 1 -- 172.21.15.28:0/93244775 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06fc004d70 con 0x561e52d3ea30 2024-04-23T14:55:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.366+0000 7f06fbe00640 1 -- 172.21.15.28:0/10943031 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8586+0+0 (secure 0 0 0) 0x7f06ec0d0c30 con 0x7f0704076e50 2024-04-23T14:55:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.366+0000 7f0702a00640 1 --2- 172.21.15.28:0/10943031 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f06e80a3d20 0x7f06e80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.367+0000 7f070c834740 1 -- 172.21.15.28:0/93244775 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e52d3ea30 msgr2=0x561e52d3ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.367+0000 7f070c834740 1 --2- 172.21.15.28:0/93244775 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e52d3ea30 0x561e52d3ee10 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f06fc009ba0 tx=0x7f06fc011810 comp rx=0 tx=0).stop 2024-04-23T14:55:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.367+0000 7f070c834740 1 -- 172.21.15.28:0/93244775 shutdown_connections 2024-04-23T14:55:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.367+0000 7f070c834740 1 --2- 172.21.15.28:0/93244775 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561e52d49a20 0x561e52d4be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.367+0000 7f070c834740 1 --2- 172.21.15.28:0/93244775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e52d3f350 0x561e52d494e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.367+0000 7f070c834740 1 --2- 172.21.15.28:0/93244775 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e52d3ea30 0x561e52d3ee10 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.367+0000 7f070c834740 1 -- 172.21.15.28:0/93244775 >> 172.21.15.28:0/93244775 conn(0x561e52d3ad50 msgr2=0x561e52d3b130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.367+0000 7f070c834740 1 -- 172.21.15.28:0/93244775 shutdown_connections 2024-04-23T14:55:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.367+0000 7f0702a00640 1 --2- 172.21.15.28:0/10943031 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f06e80a3d20 0x7f06e80a61e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x561e52d3fff0 tx=0x7f06f40099f0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.367+0000 7f070c834740 1 -- 172.21.15.28:0/93244775 wait complete. 2024-04-23T14:55:08.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.368+0000 7f070c834740 1 -- start start 2024-04-23T14:55:08.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.368+0000 7f070c834740 1 Processor -- start 2024-04-23T14:55:08.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.368+0000 7f070c834740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e52d3ea30 0x561e52d46a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.368+0000 7f070c834740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561e52d3f350 0x561e52d46f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.368+0000 7f070c834740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e52d49a20 0x561e52d474b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.368+0000 7f070c834740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561e52d4c810 con 0x561e52d49a20 2024-04-23T14:55:08.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.368+0000 7f070c834740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x561e52c5d690 con 0x561e52d3ea30 2024-04-23T14:55:08.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.368+0000 7f070c834740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x561e52d4c990 con 0x561e52d3f350 2024-04-23T14:55:08.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.368+0000 7f070c834740 10 client.0 _ll_register_callbacks cb 0x561e52c99650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:55:08.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.369+0000 7f0702a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e52d3ea30 0x561e52d46a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.369+0000 7f0703400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e52d49a20 0x561e52d474b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.369+0000 7f0702a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e52d3ea30 0x561e52d46a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:36450/0 (socket says 172.21.15.28:36450) 2024-04-23T14:55:08.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.369+0000 7f0702a00640 1 -- 172.21.15.28:0/3223959343 learned_addr learned my addr 172.21.15.28:0/3223959343 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:08.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.369+0000 7f06fa000640 1 --2- 172.21.15.28:0/3223959343 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561e52d3f350 0x561e52d46f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.369+0000 7f0702a00640 1 -- 172.21.15.28:0/3223959343 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561e52d3f350 msgr2=0x561e52d46f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:08.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.369+0000 7f0702a00640 1 --2- 172.21.15.28:0/3223959343 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561e52d3f350 0x561e52d46f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.369+0000 7f0702a00640 1 -- 172.21.15.28:0/3223959343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e52d49a20 msgr2=0x561e52d474b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:08.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.369+0000 7f0702a00640 1 --2- 172.21.15.28:0/3223959343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e52d49a20 0x561e52d474b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.369+0000 7f0702a00640 1 -- 172.21.15.28:0/3223959343 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e52c5c920 con 0x561e52d3ea30 2024-04-23T14:55:08.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.369+0000 7f06fa000640 1 --2- 172.21.15.28:0/3223959343 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561e52d3f350 0x561e52d46f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:55:08.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.369+0000 7f0702a00640 1 --2- 172.21.15.28:0/3223959343 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e52d3ea30 0x561e52d46a30 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f06f4009cd0 tx=0x7f06f402a8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d90e5e431866f780 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:08.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.369+0000 7f06f0c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:55:08.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.370+0000 7f06f0c00640 1 -- 172.21.15.28:0/3223959343 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f4053440 con 0x561e52d3ea30 2024-04-23T14:55:08.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.370+0000 7f06f0c00640 1 -- 172.21.15.28:0/3223959343 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8586+0+0 (secure 0 0 0) 0x7f06f4049420 con 0x561e52d3ea30 2024-04-23T14:55:08.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.370+0000 7f070c834740 1 -- 172.21.15.28:0/3223959343 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06fc009e10 con 0x561e52d3ea30 2024-04-23T14:55:08.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.370+0000 7f06f0c00640 1 -- 172.21.15.28:0/3223959343 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f06f4049620 con 0x561e52d3ea30 2024-04-23T14:55:08.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.370+0000 7f070c834740 10 client.27017 fetch_fsmap learned FSMap version 20 2024-04-23T14:55:08.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.370+0000 7f070c834740 1 -- 172.21.15.28:0/3223959343 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x561e52a807d0 con 0x561e52d3ea30 2024-04-23T14:55:08.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.370+0000 7f06f0c00640 1 -- 172.21.15.28:0/3223959343 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06fc009e10 con 0x561e52d3ea30 2024-04-23T14:55:08.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.370+0000 7f06f0c00640 1 -- 172.21.15.28:0/3223959343 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f06f4059030 con 0x561e52d3ea30 2024-04-23T14:55:08.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.371+0000 7f070c834740 10 client.27017 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:55:08.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.371+0000 7f070c834740 10 client.27017 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:55:08.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.371+0000 7f070c834740 20 client.27017 populate_metadata read hostname 'smithi028' 2024-04-23T14:55:08.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.371+0000 7f070c834740 1 --2- 172.21.15.28:0/3223959343 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x561e52d59eb0 0x561e52d5c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.371+0000 7f070c834740 4 client.27017 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:55:08.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.371+0000 7f070c834740 1 -- 172.21.15.28:0/3223959343 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561e52cb1c10 con 0x561e52d59eb0 2024-04-23T14:55:08.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.371+0000 7f06fa000640 1 --2- 172.21.15.28:0/3223959343 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x561e52d59eb0 0x561e52d5c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.371+0000 7f06fa000640 1 --2- 172.21.15.28:0/3223959343 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x561e52d59eb0 0x561e52d5c2e0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fdee86b475e52448 server_cookie=2aab75a192c25440 in_seq=0 out_seq=0 2024-04-23T14:55:08.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.371+0000 7f06f0c00640 10 client.27017 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:55:08.376 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:07 smithi038 ceph-mon[32348]: pgmap v345: 97 pgs: 97 active+clean; 527 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 195 op/s 2024-04-23T14:55:08.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.375+0000 7f06f0c00640 1 -- 172.21.15.28:0/3223959343 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561e52cb1c10 con 0x561e52d59eb0 2024-04-23T14:55:08.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.375+0000 7f06f0c00640 10 client.27017 handle_command_reply: tid=0 2024-04-23T14:55:08.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.375+0000 7f070c834740 1 client.27017 shutdown 2024-04-23T14:55:08.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.375+0000 7f070c834740 1 -- 172.21.15.28:0/3223959343 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e52d3ea30 msgr2=0x561e52d46a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:08.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.375+0000 7f070c834740 1 --2- 172.21.15.28:0/3223959343 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e52d3ea30 0x561e52d46a30 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f06f4009cd0 tx=0x7f06f402a8f0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.375+0000 7f070c834740 1 -- 172.21.15.28:0/3223959343 shutdown_connections 2024-04-23T14:55:08.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.375+0000 7f070c834740 1 --2- 172.21.15.28:0/3223959343 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x561e52d59eb0 0x561e52d5c2e0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.375+0000 7f070c834740 1 --2- 172.21.15.28:0/3223959343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e52d49a20 0x561e52d474b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.375+0000 7f070c834740 1 --2- 172.21.15.28:0/3223959343 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561e52d3f350 0x561e52d46f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.375+0000 7f06f0c00640 0 client.27017 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:55:08.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.375+0000 7f070c834740 1 --2- 172.21.15.28:0/3223959343 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e52d3ea30 0x561e52d46a30 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.375+0000 7f070c834740 1 -- 172.21.15.28:0/3223959343 >> 172.21.15.28:0/3223959343 conn(0x561e52d3ad50 msgr2=0x561e52d39fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:08.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.375+0000 7f070c834740 1 -- 172.21.15.28:0/3223959343 shutdown_connections 2024-04-23T14:55:08.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.376+0000 7f070c834740 1 -- 172.21.15.28:0/3223959343 wait complete. 2024-04-23T14:55:08.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.376+0000 7f070c834740 20 client.27017 trim_cache size 0 max 16384 2024-04-23T14:55:08.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.395+0000 7f070c834740 1 Processor -- start 2024-04-23T14:55:08.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.395+0000 7f070c834740 1 -- start start 2024-04-23T14:55:08.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.395+0000 7f070c834740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e52d3ea30 0x561e52d55700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.395+0000 7f070c834740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561e52d3f350 0x561e52d41190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.395+0000 7f070c834740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e52d49a20 0x561e52d47280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.395+0000 7f070c834740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f07041b2260 con 0x561e52d49a20 2024-04-23T14:55:08.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.395+0000 7f070c834740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f07041b4ac0 con 0x561e52d3ea30 2024-04-23T14:55:08.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.395+0000 7f070c834740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f07041b28b0 con 0x561e52d3f350 2024-04-23T14:55:08.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.395+0000 7f0702a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e52d3ea30 0x561e52d55700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.396+0000 7f06fa000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561e52d3f350 0x561e52d41190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.396+0000 7f06fa000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561e52d3f350 0x561e52d41190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:43948/0 (socket says 172.21.15.28:43948) 2024-04-23T14:55:08.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.396+0000 7f06fa000640 1 -- 172.21.15.28:0/3960790229 learned_addr learned my addr 172.21.15.28:0/3960790229 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:08.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.396+0000 7f0703400640 1 --2- 172.21.15.28:0/3960790229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e52d49a20 0x561e52d47280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.396+0000 7f06fa000640 1 -- 172.21.15.28:0/3960790229 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e52d3ea30 msgr2=0x561e52d55700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:08.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.396+0000 7f06fa000640 1 --2- 172.21.15.28:0/3960790229 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e52d3ea30 0x561e52d55700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.396+0000 7f06fa000640 1 -- 172.21.15.28:0/3960790229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e52d49a20 msgr2=0x561e52d47280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:08.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.396+0000 7f06fa000640 1 --2- 172.21.15.28:0/3960790229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e52d49a20 0x561e52d47280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.396+0000 7f06fa000640 1 -- 172.21.15.28:0/3960790229 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e529b63f0 con 0x561e52d3f350 2024-04-23T14:55:08.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.396+0000 7f0703400640 1 --2- 172.21.15.28:0/3960790229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e52d49a20 0x561e52d47280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:55:08.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.396+0000 7f06fa000640 1 --2- 172.21.15.28:0/3960790229 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561e52d3f350 0x561e52d41190 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f06ec0995b0 tx=0x7f06ec0d3080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=422ad99f8dc0d113 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:08.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.396+0000 7f06f3400640 1 -- 172.21.15.28:0/3960790229 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06ec09de60 con 0x561e52d3f350 2024-04-23T14:55:08.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.396+0000 7f06f3400640 1 -- 172.21.15.28:0/3960790229 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06ec09cbe0 con 0x561e52d3f350 2024-04-23T14:55:08.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.396+0000 7f06f3400640 1 -- 172.21.15.28:0/3960790229 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06ec0ce560 con 0x561e52d3f350 2024-04-23T14:55:08.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.397+0000 7f070c834740 1 -- 172.21.15.28:0/3960790229 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561e52d3f350 msgr2=0x561e52d41190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:08.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.397+0000 7f070c834740 1 --2- 172.21.15.28:0/3960790229 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561e52d3f350 0x561e52d41190 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f06ec0995b0 tx=0x7f06ec0d3080 comp rx=0 tx=0).stop 2024-04-23T14:55:08.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.397+0000 7f070c834740 1 -- 172.21.15.28:0/3960790229 shutdown_connections 2024-04-23T14:55:08.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.397+0000 7f070c834740 1 --2- 172.21.15.28:0/3960790229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e52d49a20 0x561e52d47280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.397+0000 7f070c834740 1 --2- 172.21.15.28:0/3960790229 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561e52d3f350 0x561e52d41190 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.397+0000 7f070c834740 1 --2- 172.21.15.28:0/3960790229 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e52d3ea30 0x561e52d55700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.397+0000 7f070c834740 1 -- 172.21.15.28:0/3960790229 >> 172.21.15.28:0/3960790229 conn(0x561e52d3ad50 msgr2=0x561e52d45970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:08.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.397+0000 7f070c834740 1 -- 172.21.15.28:0/3960790229 shutdown_connections 2024-04-23T14:55:08.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.397+0000 7f070c834740 1 -- 172.21.15.28:0/3960790229 wait complete. 2024-04-23T14:55:08.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.398+0000 7f070c834740 1 -- start start 2024-04-23T14:55:08.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.398+0000 7f070c834740 1 Processor -- start 2024-04-23T14:55:08.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.398+0000 7f070c834740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e52d3ea30 0x561e52d51f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.398+0000 7f070c834740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561e52d3f350 0x561e52d43320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.398+0000 7f070c834740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e52d49a20 0x561e52d43860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.398+0000 7f070c834740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f07041b2f80 con 0x561e52d3ea30 2024-04-23T14:55:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.398+0000 7f070c834740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f07041b43f0 con 0x561e52d49a20 2024-04-23T14:55:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.398+0000 7f070c834740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f07041b3650 con 0x561e52d3f350 2024-04-23T14:55:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.398+0000 7f070c834740 10 client.0 _ll_register_callbacks cb 0x7f06f4009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:55:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.398+0000 7f0702a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e52d3ea30 0x561e52d51f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.398+0000 7f0703400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e52d49a20 0x561e52d43860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.399+0000 7f0702a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e52d3ea30 0x561e52d51f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33594/0 (socket says 172.21.15.28:33594) 2024-04-23T14:55:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.399+0000 7f0702a00640 1 -- 172.21.15.28:0/1168386658 learned_addr learned my addr 172.21.15.28:0/1168386658 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.399+0000 7f06fa000640 1 --2- 172.21.15.28:0/1168386658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561e52d3f350 0x561e52d43320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.399+0000 7f0702a00640 1 -- 172.21.15.28:0/1168386658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561e52d3f350 msgr2=0x561e52d43320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.399+0000 7f0702a00640 1 --2- 172.21.15.28:0/1168386658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561e52d3f350 0x561e52d43320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.399+0000 7f0702a00640 1 -- 172.21.15.28:0/1168386658 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e52d49a20 msgr2=0x561e52d43860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.399+0000 7f0702a00640 1 --2- 172.21.15.28:0/1168386658 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e52d49a20 0x561e52d43860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.399+0000 7f0702a00640 1 -- 172.21.15.28:0/1168386658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e529760f0 con 0x561e52d3ea30 2024-04-23T14:55:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.399+0000 7f0703400640 1 --2- 172.21.15.28:0/1168386658 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e52d49a20 0x561e52d43860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:55:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.399+0000 7f0702a00640 1 --2- 172.21.15.28:0/1168386658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e52d3ea30 0x561e52d51f00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f06f4053a50 tx=0x7f06f4053670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddf08c0c3f365542 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:08.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.399+0000 7f06f0c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-23T14:55:08.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.400+0000 7f06f0c00640 1 -- 172.21.15.28:0/1168386658 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f4053860 con 0x561e52d3ea30 2024-04-23T14:55:08.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.400+0000 7f06f0c00640 1 -- 172.21.15.28:0/1168386658 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8586+0+0 (secure 0 0 0) 0x7f06f40499d0 con 0x561e52d3ea30 2024-04-23T14:55:08.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.400+0000 7f070c834740 1 -- 172.21.15.28:0/1168386658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06ec09cbe0 con 0x561e52d3ea30 2024-04-23T14:55:08.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.400+0000 7f06f0c00640 1 -- 172.21.15.28:0/1168386658 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f06f406b050 con 0x561e52d3ea30 2024-04-23T14:55:08.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.400+0000 7f070c834740 10 client.16833 fetch_fsmap learned FSMap version 20 2024-04-23T14:55:08.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.400+0000 7f070c834740 1 -- 172.21.15.28:0/1168386658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x561e52c69810 con 0x561e52d3ea30 2024-04-23T14:55:08.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.400+0000 7f06f0c00640 1 -- 172.21.15.28:0/1168386658 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06ec09cbe0 con 0x561e52d3ea30 2024-04-23T14:55:08.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.400+0000 7f06f0c00640 1 -- 172.21.15.28:0/1168386658 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f06f4062070 con 0x561e52d3ea30 2024-04-23T14:55:08.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.401+0000 7f070c834740 10 client.16833 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:55:08.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.401+0000 7f070c834740 10 client.16833 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:55:08.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.401+0000 7f070c834740 20 client.16833 populate_metadata read hostname 'smithi028' 2024-04-23T14:55:08.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.401+0000 7f070c834740 1 --2- 172.21.15.28:0/1168386658 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x561e52dc0770 0x561e52dc2b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:08.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.401+0000 7f070c834740 4 client.16833 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:55:08.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.401+0000 7f070c834740 1 -- 172.21.15.28:0/1168386658 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561e52caba80 con 0x561e52dc0770 2024-04-23T14:55:08.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.401+0000 7f06fa000640 1 --2- 172.21.15.28:0/1168386658 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x561e52dc0770 0x561e52dc2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:08.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.401+0000 7f06fa000640 1 --2- 172.21.15.28:0/1168386658 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x561e52dc0770 0x561e52dc2b90 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c11377dbc5f94e7c server_cookie=5c737b91d1a0e0e in_seq=0 out_seq=0 2024-04-23T14:55:08.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.401+0000 7f06f0c00640 10 client.16833 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:55:08.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.402+0000 7f06f0c00640 1 -- 172.21.15.28:0/1168386658 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x561e52caba80 con 0x561e52dc0770 2024-04-23T14:55:08.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.402+0000 7f06f0c00640 10 client.16833 handle_command_reply: tid=0 2024-04-23T14:55:08.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.402+0000 7f070c834740 1 client.16833 shutdown 2024-04-23T14:55:08.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.402+0000 7f070c834740 1 -- 172.21.15.28:0/1168386658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e52d3ea30 msgr2=0x561e52d51f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:08.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.402+0000 7f070c834740 1 --2- 172.21.15.28:0/1168386658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e52d3ea30 0x561e52d51f00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f06f4053a50 tx=0x7f06f4053670 comp rx=0 tx=0).stop 2024-04-23T14:55:08.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.402+0000 7f070c834740 1 -- 172.21.15.28:0/1168386658 shutdown_connections 2024-04-23T14:55:08.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.402+0000 7f070c834740 1 --2- 172.21.15.28:0/1168386658 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x561e52dc0770 0x561e52dc2b90 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.402+0000 7f070c834740 1 --2- 172.21.15.28:0/1168386658 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e52d49a20 0x561e52d43860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.402+0000 7f070c834740 1 --2- 172.21.15.28:0/1168386658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x561e52d3f350 0x561e52d43320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.402+0000 7f070c834740 1 --2- 172.21.15.28:0/1168386658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e52d3ea30 0x561e52d51f00 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.402+0000 7f070c834740 1 -- 172.21.15.28:0/1168386658 >> 172.21.15.28:0/1168386658 conn(0x561e52d3ad50 msgr2=0x561e52d47260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:08.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.402+0000 7f06f0c00640 0 client.16833 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:55:08.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.402+0000 7f070c834740 1 -- 172.21.15.28:0/1168386658 shutdown_connections 2024-04-23T14:55:08.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.402+0000 7f070c834740 1 -- 172.21.15.28:0/1168386658 wait complete. 2024-04-23T14:55:08.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.403+0000 7f070c834740 20 client.16833 trim_cache size 0 max 16384 2024-04-23T14:55:08.407 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:55:08.407 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-23T14:55:08.407 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-23T14:55:08.407 INFO:teuthology.orchestra.run.smithi028.stdout: "555e6dc6-5bc6-42d2-b13d-17deecc233a8": { 2024-04-23T14:55:08.407 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-23T14:55:08.407 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "555e6dc6-5bc6-42d2-b13d-17deecc233a8", 2024-04-23T14:55:08.407 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-23T14:55:08.408 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:55:08.408 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:55:08.408 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:55:08.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.403+0000 7f0703e00640 1 -- 172.21.15.28:0/10943031 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f06e80a3d20 msgr2=0x7f06e80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:08.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.403+0000 7f0703e00640 1 --2- 172.21.15.28:0/10943031 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f06e80a3d20 0x7f06e80a61e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x561e52d3fff0 tx=0x7f06f40099f0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.403+0000 7f0703e00640 1 -- 172.21.15.28:0/10943031 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0704076e50 msgr2=0x7f070419f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:08.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.403+0000 7f0703e00640 1 --2- 172.21.15.28:0/10943031 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0704076e50 0x7f070419f370 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f06ec00e9f0 tx=0x7f06ec00eeb0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.403+0000 7f0703e00640 1 -- 172.21.15.28:0/10943031 shutdown_connections 2024-04-23T14:55:08.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.403+0000 7f0703e00640 1 --2- 172.21.15.28:0/10943031 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f06e80a3d20 0x7f06e80a61e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.403+0000 7f0703e00640 1 --2- 172.21.15.28:0/10943031 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0704076e50 0x7f070419f370 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.403+0000 7f0703e00640 1 --2- 172.21.15.28:0/10943031 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0704112d40 0x7f07041a36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.403+0000 7f0703e00640 1 --2- 172.21.15.28:0/10943031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0704076530 0x7f070419ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:08.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.403+0000 7f0703e00640 1 -- 172.21.15.28:0/10943031 >> 172.21.15.28:0/10943031 conn(0x7f07040ffb10 msgr2=0x7f07041144d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:08.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.403+0000 7f0703e00640 1 -- 172.21.15.28:0/10943031 shutdown_connections 2024-04-23T14:55:08.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:08.404+0000 7f0703e00640 1 -- 172.21.15.28:0/10943031 wait complete. 2024-04-23T14:55:08.440 INFO:tasks.cephfs.filesystem:scrub status for tag:555e6dc6-5bc6-42d2-b13d-17deecc233a8 - {'path': '/', 'tag': '555e6dc6-5bc6-42d2-b13d-17deecc233a8', 'options': 'recursive,force'} 2024-04-23T14:55:08.532 INFO:tasks.workunit.client.0.smithi028.stdout: 1 949038 8.75 MB/sec execute 444 sec latency 53.706 ms 2024-04-23T14:55:09.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:08 smithi105 ceph-mon[32555]: scrub queued for path: / 2024-04-23T14:55:09.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:08 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:55:09.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:08 smithi105 ceph-mon[32555]: scrub summary: active paths [/] 2024-04-23T14:55:09.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:08 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:55:09.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:08 smithi038 ceph-mon[32348]: scrub queued for path: / 2024-04-23T14:55:09.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:08 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:55:09.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:08 smithi038 ceph-mon[32348]: scrub summary: active paths [/] 2024-04-23T14:55:09.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:08 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:55:09.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:08 smithi028 ceph-mon[27810]: scrub queued for path: / 2024-04-23T14:55:09.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:08 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:55:09.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:08 smithi028 ceph-mon[27810]: scrub summary: active paths [/] 2024-04-23T14:55:09.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:08 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:55:09.531 INFO:tasks.workunit.client.0.smithi028.stdout: 1 950290 8.75 MB/sec execute 445 sec latency 41.281 ms 2024-04-23T14:55:10.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:10 smithi105 ceph-mon[32555]: pgmap v346: 97 pgs: 97 active+clean; 527 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 195 op/s 2024-04-23T14:55:10.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:10 smithi105 ceph-mon[32555]: scrub summary: idle 2024-04-23T14:55:10.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:10 smithi038 ceph-mon[32348]: pgmap v346: 97 pgs: 97 active+clean; 527 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 195 op/s 2024-04-23T14:55:10.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:10 smithi038 ceph-mon[32348]: scrub summary: idle 2024-04-23T14:55:10.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:10 smithi028 ceph-mon[27810]: pgmap v346: 97 pgs: 97 active+clean; 527 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 195 op/s 2024-04-23T14:55:10.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:10 smithi028 ceph-mon[27810]: scrub summary: idle 2024-04-23T14:55:10.531 INFO:tasks.workunit.client.0.smithi028.stdout: 1 951959 8.75 MB/sec execute 446 sec latency 31.645 ms 2024-04-23T14:55:11.532 INFO:tasks.workunit.client.0.smithi028.stdout: 1 953001 8.74 MB/sec execute 447 sec latency 15.503 ms 2024-04-23T14:55:12.532 INFO:tasks.workunit.client.0.smithi028.stdout: 1 954585 8.73 MB/sec execute 448 sec latency 23.756 ms 2024-04-23T14:55:13.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:12 smithi105 ceph-mon[32555]: pgmap v347: 97 pgs: 97 active+clean; 527 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 195 op/s 2024-04-23T14:55:13.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:12 smithi105 ceph-mon[32555]: osdmap e98: 12 total, 12 up, 12 in 2024-04-23T14:55:13.077 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:12 smithi028 ceph-mon[27810]: pgmap v347: 97 pgs: 97 active+clean; 527 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 195 op/s 2024-04-23T14:55:13.077 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:12 smithi028 ceph-mon[27810]: osdmap e98: 12 total, 12 up, 12 in 2024-04-23T14:55:13.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:12 smithi038 ceph-mon[32348]: pgmap v347: 97 pgs: 97 active+clean; 527 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 195 op/s 2024-04-23T14:55:13.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:12 smithi038 ceph-mon[32348]: osdmap e98: 12 total, 12 up, 12 in 2024-04-23T14:55:13.532 INFO:tasks.workunit.client.0.smithi028.stdout: 1 956127 8.74 MB/sec execute 449 sec latency 12.573 ms 2024-04-23T14:55:14.532 INFO:tasks.workunit.client.0.smithi028.stdout: 1 957353 8.73 MB/sec execute 450 sec latency 41.098 ms 2024-04-23T14:55:15.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:14 smithi105 ceph-mon[32555]: pgmap v349: 97 pgs: 97 active+clean; 536 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.2 MiB/s wr, 187 op/s 2024-04-23T14:55:15.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:14 smithi038 ceph-mon[32348]: pgmap v349: 97 pgs: 97 active+clean; 536 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.2 MiB/s wr, 187 op/s 2024-04-23T14:55:15.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:14 smithi028 ceph-mon[27810]: pgmap v349: 97 pgs: 97 active+clean; 536 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.2 MiB/s wr, 187 op/s 2024-04-23T14:55:15.532 INFO:tasks.workunit.client.0.smithi028.stdout: 1 959446 8.74 MB/sec execute 451 sec latency 44.656 ms 2024-04-23T14:55:16.532 INFO:tasks.workunit.client.0.smithi028.stdout: 1 960847 8.74 MB/sec execute 452 sec latency 18.570 ms 2024-04-23T14:55:17.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:16 smithi105 ceph-mon[32555]: pgmap v350: 97 pgs: 97 active+clean; 536 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.6 MiB/s wr, 163 op/s 2024-04-23T14:55:17.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:16 smithi038 ceph-mon[32348]: pgmap v350: 97 pgs: 97 active+clean; 536 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.6 MiB/s wr, 163 op/s 2024-04-23T14:55:17.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:16 smithi028 ceph-mon[27810]: pgmap v350: 97 pgs: 97 active+clean; 536 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.6 MiB/s wr, 163 op/s 2024-04-23T14:55:17.532 INFO:tasks.workunit.client.0.smithi028.stdout: 1 963047 8.75 MB/sec execute 453 sec latency 15.917 ms 2024-04-23T14:55:18.532 INFO:tasks.workunit.client.0.smithi028.stdout: 1 964220 8.74 MB/sec execute 454 sec latency 18.099 ms 2024-04-23T14:55:19.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:18 smithi105 ceph-mon[32555]: pgmap v351: 97 pgs: 97 active+clean; 548 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 170 op/s 2024-04-23T14:55:19.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:18 smithi038 ceph-mon[32348]: pgmap v351: 97 pgs: 97 active+clean; 548 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 170 op/s 2024-04-23T14:55:19.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:18 smithi028 ceph-mon[27810]: pgmap v351: 97 pgs: 97 active+clean; 548 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 170 op/s 2024-04-23T14:55:19.532 INFO:tasks.workunit.client.0.smithi028.stdout: 1 965757 8.74 MB/sec execute 455 sec latency 141.530 ms 2024-04-23T14:55:20.532 INFO:tasks.workunit.client.0.smithi028.stdout: 1 967369 8.74 MB/sec execute 456 sec latency 13.052 ms 2024-04-23T14:55:21.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:20 smithi105 ceph-mon[32555]: pgmap v352: 97 pgs: 97 active+clean; 548 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 170 op/s 2024-04-23T14:55:21.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:20 smithi038 ceph-mon[32348]: pgmap v352: 97 pgs: 97 active+clean; 548 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 170 op/s 2024-04-23T14:55:21.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:20 smithi028 ceph-mon[27810]: pgmap v352: 97 pgs: 97 active+clean; 548 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 170 op/s 2024-04-23T14:55:21.532 INFO:tasks.workunit.client.0.smithi028.stdout: 1 969271 8.74 MB/sec execute 457 sec latency 18.058 ms 2024-04-23T14:55:22.533 INFO:tasks.workunit.client.0.smithi028.stdout: 1 970603 8.74 MB/sec execute 458 sec latency 27.038 ms 2024-04-23T14:55:23.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:22 smithi105 ceph-mon[32555]: pgmap v353: 97 pgs: 97 active+clean; 548 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 170 op/s 2024-04-23T14:55:23.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:22 smithi038 ceph-mon[32348]: pgmap v353: 97 pgs: 97 active+clean; 548 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 170 op/s 2024-04-23T14:55:23.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:22 smithi028 ceph-mon[27810]: pgmap v353: 97 pgs: 97 active+clean; 548 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 170 op/s 2024-04-23T14:55:23.532 INFO:tasks.workunit.client.0.smithi028.stdout: 1 972260 8.73 MB/sec execute 459 sec latency 20.835 ms 2024-04-23T14:55:24.533 INFO:tasks.workunit.client.0.smithi028.stdout: 1 973964 8.74 MB/sec execute 460 sec latency 40.639 ms 2024-04-23T14:55:25.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:24 smithi105 ceph-mon[32555]: pgmap v354: 97 pgs: 97 active+clean; 578 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.2 MiB/s wr, 212 op/s 2024-04-23T14:55:25.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:24 smithi038 ceph-mon[32348]: pgmap v354: 97 pgs: 97 active+clean; 578 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.2 MiB/s wr, 212 op/s 2024-04-23T14:55:25.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:24 smithi028 ceph-mon[27810]: pgmap v354: 97 pgs: 97 active+clean; 578 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.2 MiB/s wr, 212 op/s 2024-04-23T14:55:25.533 INFO:tasks.workunit.client.0.smithi028.stdout: 1 975609 8.73 MB/sec execute 461 sec latency 17.791 ms 2024-04-23T14:55:26.533 INFO:tasks.workunit.client.0.smithi028.stdout: 1 976957 8.74 MB/sec execute 462 sec latency 395.030 ms 2024-04-23T14:55:27.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:26 smithi105 ceph-mon[32555]: pgmap v355: 97 pgs: 97 active+clean; 578 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.8 MiB/s wr, 142 op/s 2024-04-23T14:55:27.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:26 smithi038 ceph-mon[32348]: pgmap v355: 97 pgs: 97 active+clean; 578 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.8 MiB/s wr, 142 op/s 2024-04-23T14:55:27.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:26 smithi028 ceph-mon[27810]: pgmap v355: 97 pgs: 97 active+clean; 578 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.8 MiB/s wr, 142 op/s 2024-04-23T14:55:27.533 INFO:tasks.workunit.client.0.smithi028.stdout: 1 976957 8.72 MB/sec execute 463 sec latency 1395.179 ms 2024-04-23T14:55:28.533 INFO:tasks.workunit.client.0.smithi028.stdout: 1 976957 8.70 MB/sec execute 464 sec latency 2395.335 ms 2024-04-23T14:55:29.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:28 smithi105 ceph-mon[32555]: pgmap v356: 97 pgs: 97 active+clean; 585 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.0 MiB/s wr, 200 op/s 2024-04-23T14:55:29.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:28 smithi038 ceph-mon[32348]: pgmap v356: 97 pgs: 97 active+clean; 585 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.0 MiB/s wr, 200 op/s 2024-04-23T14:55:29.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:28 smithi028 ceph-mon[27810]: pgmap v356: 97 pgs: 97 active+clean; 585 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.0 MiB/s wr, 200 op/s 2024-04-23T14:55:29.533 INFO:tasks.workunit.client.0.smithi028.stdout: 1 976957 8.68 MB/sec execute 465 sec latency 3395.456 ms 2024-04-23T14:55:30.533 INFO:tasks.workunit.client.0.smithi028.stdout: 1 977291 8.67 MB/sec execute 466 sec latency 3910.747 ms 2024-04-23T14:55:31.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:30 smithi105 ceph-mon[32555]: pgmap v357: 97 pgs: 97 active+clean; 585 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.3 MiB/s wr, 127 op/s 2024-04-23T14:55:31.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:30 smithi038 ceph-mon[32348]: pgmap v357: 97 pgs: 97 active+clean; 585 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.3 MiB/s wr, 127 op/s 2024-04-23T14:55:31.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:30 smithi028 ceph-mon[27810]: pgmap v357: 97 pgs: 97 active+clean; 585 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.3 MiB/s wr, 127 op/s 2024-04-23T14:55:31.533 INFO:tasks.workunit.client.0.smithi028.stdout: 1 978473 8.66 MB/sec execute 467 sec latency 26.685 ms 2024-04-23T14:55:32.534 INFO:tasks.workunit.client.0.smithi028.stdout: 1 980310 8.66 MB/sec execute 468 sec latency 17.270 ms 2024-04-23T14:55:33.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:32 smithi105 ceph-mon[32555]: pgmap v358: 97 pgs: 97 active+clean; 585 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.3 MiB/s wr, 127 op/s 2024-04-23T14:55:33.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:32 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:55:33.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:32 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:55:33.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:32 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:55:33.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:32 smithi038 ceph-mon[32348]: pgmap v358: 97 pgs: 97 active+clean; 585 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.3 MiB/s wr, 127 op/s 2024-04-23T14:55:33.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:32 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:55:33.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:32 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:55:33.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:32 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:55:33.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:32 smithi028 ceph-mon[27810]: pgmap v358: 97 pgs: 97 active+clean; 585 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.3 MiB/s wr, 127 op/s 2024-04-23T14:55:33.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:32 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:55:33.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:32 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:55:33.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:32 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:55:33.534 INFO:tasks.workunit.client.0.smithi028.stdout: 1 981465 8.66 MB/sec execute 469 sec latency 18.682 ms 2024-04-23T14:55:34.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:33 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:55:34.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:33 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:55:34.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:33 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:55:34.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:33 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:55:34.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:33 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:55:34.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:33 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:55:34.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:33 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:55:34.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:33 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:55:34.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:33 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:55:34.534 INFO:tasks.workunit.client.0.smithi028.stdout: 1 982894 8.65 MB/sec execute 470 sec latency 71.468 ms 2024-04-23T14:55:35.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:34 smithi105 ceph-mon[32555]: pgmap v359: 97 pgs: 97 active+clean; 591 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.4 MiB/s wr, 173 op/s 2024-04-23T14:55:35.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:34 smithi038 ceph-mon[32348]: pgmap v359: 97 pgs: 97 active+clean; 591 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.4 MiB/s wr, 173 op/s 2024-04-23T14:55:35.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:34 smithi028 ceph-mon[27810]: pgmap v359: 97 pgs: 97 active+clean; 591 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.4 MiB/s wr, 173 op/s 2024-04-23T14:55:35.534 INFO:tasks.workunit.client.0.smithi028.stdout: 1 984385 8.66 MB/sec execute 471 sec latency 39.167 ms 2024-04-23T14:55:36.534 INFO:tasks.workunit.client.0.smithi028.stdout: 1 985700 8.65 MB/sec execute 472 sec latency 22.994 ms 2024-04-23T14:55:37.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:36 smithi105 ceph-mon[32555]: pgmap v360: 97 pgs: 97 active+clean; 591 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 103 op/s 2024-04-23T14:55:37.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:36 smithi038 ceph-mon[32348]: pgmap v360: 97 pgs: 97 active+clean; 591 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 103 op/s 2024-04-23T14:55:37.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:36 smithi028 ceph-mon[27810]: pgmap v360: 97 pgs: 97 active+clean; 591 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 103 op/s 2024-04-23T14:55:37.534 INFO:tasks.workunit.client.0.smithi028.stdout: 1 987713 8.66 MB/sec execute 473 sec latency 24.950 ms 2024-04-23T14:55:38.441 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:55:38.442 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:55:38.534 INFO:tasks.workunit.client.0.smithi028.stdout: 1 988594 8.65 MB/sec execute 474 sec latency 57.034 ms 2024-04-23T14:55:38.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.666+0000 7ffa4e400640 1 -- 172.21.15.28:0/1074230470 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa48104320 msgr2=0x7ffa48104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:38.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.666+0000 7ffa4e400640 1 --2- 172.21.15.28:0/1074230470 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa48104320 0x7ffa48104700 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c009900 tx=0x7ffa3c02a400 comp rx=0 tx=0).stop 2024-04-23T14:55:38.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.666+0000 7ffa4e400640 1 -- 172.21.15.28:0/1074230470 shutdown_connections 2024-04-23T14:55:38.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.666+0000 7ffa4e400640 1 --2- 172.21.15.28:0/1074230470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa481093f0 0x7ffa48110fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.666+0000 7ffa4e400640 1 --2- 172.21.15.28:0/1074230470 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa48104cd0 0x7ffa48108d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.666+0000 7ffa4e400640 1 --2- 172.21.15.28:0/1074230470 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa48104320 0x7ffa48104700 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.666+0000 7ffa4e400640 1 -- 172.21.15.28:0/1074230470 >> 172.21.15.28:0/1074230470 conn(0x7ffa480ffb40 msgr2=0x7ffa48101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:38.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.666+0000 7ffa4e400640 1 -- 172.21.15.28:0/1074230470 shutdown_connections 2024-04-23T14:55:38.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.666+0000 7ffa4e400640 1 -- 172.21.15.28:0/1074230470 wait complete. 2024-04-23T14:55:38.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.668+0000 7ffa4e400640 1 Processor -- start 2024-04-23T14:55:38.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.668+0000 7ffa4e400640 1 -- start start 2024-04-23T14:55:38.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.668+0000 7ffa4e400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa48104320 0x7ffa481a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:38.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.668+0000 7ffa4e400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa481093f0 0x7ffa481a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:38.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.668+0000 7ffa4e400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa4819a830 0x7ffa4819ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:38.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.669+0000 7ffa4e400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa481144f0 con 0x7ffa48104320 2024-04-23T14:55:38.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.669+0000 7ffa4e400640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ffa48114370 con 0x7ffa481093f0 2024-04-23T14:55:38.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.669+0000 7ffa4e400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ffa48114670 con 0x7ffa4819a830 2024-04-23T14:55:38.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.669+0000 7ffa4da00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa4819a830 0x7ffa4819ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:38.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.669+0000 7ffa4da00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa4819a830 0x7ffa4819ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:53036/0 (socket says 172.21.15.28:53036) 2024-04-23T14:55:38.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.669+0000 7ffa4da00640 1 -- 172.21.15.28:0/3426576621 learned_addr learned my addr 172.21.15.28:0/3426576621 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:38.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.669+0000 7ffa47e00640 1 --2- 172.21.15.28:0/3426576621 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa481093f0 0x7ffa481a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:38.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.669+0000 7ffa4d000640 1 --2- 172.21.15.28:0/3426576621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa48104320 0x7ffa481a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:38.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.669+0000 7ffa47e00640 1 -- 172.21.15.28:0/3426576621 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa4819a830 msgr2=0x7ffa4819ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:38.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.669+0000 7ffa47e00640 1 --2- 172.21.15.28:0/3426576621 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa4819a830 0x7ffa4819ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.669+0000 7ffa47e00640 1 -- 172.21.15.28:0/3426576621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa48104320 msgr2=0x7ffa481a06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:38.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.669+0000 7ffa47e00640 1 --2- 172.21.15.28:0/3426576621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa48104320 0x7ffa481a06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.669+0000 7ffa47e00640 1 -- 172.21.15.28:0/3426576621 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa4819b3c0 con 0x7ffa481093f0 2024-04-23T14:55:38.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.670+0000 7ffa47e00640 1 --2- 172.21.15.28:0/3426576621 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa481093f0 0x7ffa481a0c00 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7ffa3400e9c0 tx=0x7ffa3400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.670+0000 7ffa45600640 1 -- 172.21.15.28:0/3426576621 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3400ccc0 con 0x7ffa481093f0 2024-04-23T14:55:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.670+0000 7ffa4e400640 1 -- 172.21.15.28:0/3426576621 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa4819b6b0 con 0x7ffa481093f0 2024-04-23T14:55:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.670+0000 7ffa45600640 1 -- 172.21.15.28:0/3426576621 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa34005020 con 0x7ffa481093f0 2024-04-23T14:55:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.670+0000 7ffa45600640 1 -- 172.21.15.28:0/3426576621 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa34004a00 con 0x7ffa481093f0 2024-04-23T14:55:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.670+0000 7ffa4e400640 1 -- 172.21.15.28:0/3426576621 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa481a7840 con 0x7ffa481093f0 2024-04-23T14:55:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.674+0000 7ffa45600640 1 -- 172.21.15.28:0/3426576621 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7ffa340051d0 con 0x7ffa481093f0 2024-04-23T14:55:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.674+0000 7ffa5278d740 1 Processor -- start 2024-04-23T14:55:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.674+0000 7ffa5278d740 1 -- start start 2024-04-23T14:55:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.674+0000 7ffa5278d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fa91bda30 0x560fa91bde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.674+0000 7ffa5278d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560fa91be350 0x560fa91c84e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.674+0000 7ffa5278d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fa91c8a20 0x560fa91cae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.674+0000 7ffa5278d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560fa91231f0 con 0x560fa91bda30 2024-04-23T14:55:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.674+0000 7ffa5278d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560fa90c9350 con 0x560fa91be350 2024-04-23T14:55:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.674+0000 7ffa5278d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560fa911f1b0 con 0x560fa91c8a20 2024-04-23T14:55:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.675+0000 7ffa4d000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560fa91be350 0x560fa91c84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.675+0000 7ffa4da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fa91bda30 0x560fa91bde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.675+0000 7ffa4d000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560fa91be350 0x560fa91c84e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:53078/0 (socket says 172.21.15.28:53078) 2024-04-23T14:55:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.675+0000 7ffa4d000640 1 -- 172.21.15.28:0/1693394827 learned_addr learned my addr 172.21.15.28:0/1693394827 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.675+0000 7ffa47e00640 1 --2- 172.21.15.28:0/1693394827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fa91c8a20 0x560fa91cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.675+0000 7ffa4d000640 1 -- 172.21.15.28:0/1693394827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fa91c8a20 msgr2=0x560fa91cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.675+0000 7ffa4d000640 1 --2- 172.21.15.28:0/1693394827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fa91c8a20 0x560fa91cae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.675+0000 7ffa4d000640 1 -- 172.21.15.28:0/1693394827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fa91bda30 msgr2=0x560fa91bde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.675+0000 7ffa4d000640 1 --2- 172.21.15.28:0/1693394827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fa91bda30 0x560fa91bde10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.675+0000 7ffa4d000640 1 -- 172.21.15.28:0/1693394827 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fa911df50 con 0x560fa91be350 2024-04-23T14:55:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.675+0000 7ffa4d000640 1 --2- 172.21.15.28:0/1693394827 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560fa91be350 0x560fa91c84e0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c009900 tx=0x7ffa3c005de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f780499a30d9b74 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.675+0000 7ffa45600640 1 --2- 172.21.15.28:0/3426576621 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7ffa300a3d20 0x7ffa300a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.676+0000 7ffa3a000640 1 -- 172.21.15.28:0/1693394827 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c035040 con 0x560fa91be350 2024-04-23T14:55:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.676+0000 7ffa3a000640 1 -- 172.21.15.28:0/1693394827 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa3c005520 con 0x560fa91be350 2024-04-23T14:55:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.676+0000 7ffa45600640 1 -- 172.21.15.28:0/3426576621 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8542+0+0 (secure 0 0 0) 0x7ffa340d0800 con 0x7ffa481093f0 2024-04-23T14:55:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.676+0000 7ffa3a000640 1 -- 172.21.15.28:0/1693394827 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c0035f0 con 0x560fa91be350 2024-04-23T14:55:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.676+0000 7ffa4da00640 1 --2- 172.21.15.28:0/3426576621 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7ffa300a3d20 0x7ffa300a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:38.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.677+0000 7ffa5278d740 1 -- 172.21.15.28:0/1693394827 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560fa91be350 msgr2=0x560fa91c84e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:38.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.677+0000 7ffa4da00640 1 --2- 172.21.15.28:0/3426576621 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7ffa300a3d20 0x7ffa300a61e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ffa400099a0 tx=0x7ffa40009500 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:38.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.677+0000 7ffa5278d740 1 --2- 172.21.15.28:0/1693394827 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560fa91be350 0x560fa91c84e0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c009900 tx=0x7ffa3c005de0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.677+0000 7ffa5278d740 1 -- 172.21.15.28:0/1693394827 shutdown_connections 2024-04-23T14:55:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.677+0000 7ffa5278d740 1 --2- 172.21.15.28:0/1693394827 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560fa91be350 0x560fa91c84e0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.677+0000 7ffa5278d740 1 --2- 172.21.15.28:0/1693394827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fa91c8a20 0x560fa91cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.677+0000 7ffa5278d740 1 --2- 172.21.15.28:0/1693394827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fa91bda30 0x560fa91bde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.677+0000 7ffa5278d740 1 -- 172.21.15.28:0/1693394827 >> 172.21.15.28:0/1693394827 conn(0x560fa91b9d50 msgr2=0x560fa91ba130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.677+0000 7ffa5278d740 1 -- 172.21.15.28:0/1693394827 shutdown_connections 2024-04-23T14:55:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.677+0000 7ffa5278d740 1 -- 172.21.15.28:0/1693394827 wait complete. 2024-04-23T14:55:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.677+0000 7ffa5278d740 1 -- start start 2024-04-23T14:55:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.678+0000 7ffa5278d740 1 Processor -- start 2024-04-23T14:55:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.678+0000 7ffa5278d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fa91bda30 0x560fa91c5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.678+0000 7ffa5278d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fa91be350 0x560fa91c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.678+0000 7ffa5278d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560fa91c8a20 0x560fa91c64b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.678+0000 7ffa5278d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560fa91cb7b0 con 0x560fa91be350 2024-04-23T14:55:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.678+0000 7ffa5278d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560fa90dc690 con 0x560fa91c8a20 2024-04-23T14:55:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.678+0000 7ffa5278d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560fa91cb930 con 0x560fa91bda30 2024-04-23T14:55:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.678+0000 7ffa5278d740 10 client.0 _ll_register_callbacks cb 0x560fa9118650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:55:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.678+0000 7ffa47e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fa91be350 0x560fa91c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.678+0000 7ffa4da00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560fa91c8a20 0x560fa91c64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.679+0000 7ffa47e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fa91be350 0x560fa91c5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39684/0 (socket says 172.21.15.28:39684) 2024-04-23T14:55:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.679+0000 7ffa47e00640 1 -- 172.21.15.28:0/2361110093 learned_addr learned my addr 172.21.15.28:0/2361110093 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.679+0000 7ffa4d000640 1 --2- 172.21.15.28:0/2361110093 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fa91bda30 0x560fa91c5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.679+0000 7ffa47e00640 1 -- 172.21.15.28:0/2361110093 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fa91bda30 msgr2=0x560fa91c5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.679+0000 7ffa47e00640 1 --2- 172.21.15.28:0/2361110093 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fa91bda30 0x560fa91c5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.679+0000 7ffa47e00640 1 -- 172.21.15.28:0/2361110093 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560fa91c8a20 msgr2=0x560fa91c64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.679+0000 7ffa47e00640 1 --2- 172.21.15.28:0/2361110093 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560fa91c8a20 0x560fa91c64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.679+0000 7ffa47e00640 1 -- 172.21.15.28:0/2361110093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fa90db920 con 0x560fa91be350 2024-04-23T14:55:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.679+0000 7ffa4da00640 1 --2- 172.21.15.28:0/2361110093 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560fa91c8a20 0x560fa91c64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:55:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.679+0000 7ffa47e00640 1 --2- 172.21.15.28:0/2361110093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fa91be350 0x560fa91c5f70 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ffa34004610 tx=0x7ffa3409c090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5be9686e01e9deac server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.679+0000 7ffa2ea00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-23T14:55:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.679+0000 7ffa2ea00640 1 -- 172.21.15.28:0/2361110093 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3401c070 con 0x560fa91be350 2024-04-23T14:55:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.679+0000 7ffa2ea00640 1 -- 172.21.15.28:0/2361110093 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8542+0+0 (secure 0 0 0) 0x7ffa340ce160 con 0x560fa91be350 2024-04-23T14:55:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.680+0000 7ffa5278d740 1 -- 172.21.15.28:0/2361110093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa3c005520 con 0x560fa91be350 2024-04-23T14:55:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.680+0000 7ffa2ea00640 1 -- 172.21.15.28:0/2361110093 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7ffa340ce360 con 0x560fa91be350 2024-04-23T14:55:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.680+0000 7ffa5278d740 10 client.16845 fetch_fsmap learned FSMap version 20 2024-04-23T14:55:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.680+0000 7ffa5278d740 1 -- 172.21.15.28:0/2361110093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x560fa8eff7d0 con 0x560fa91be350 2024-04-23T14:55:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.680+0000 7ffa2ea00640 1 -- 172.21.15.28:0/2361110093 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa3c005520 con 0x560fa91be350 2024-04-23T14:55:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.680+0000 7ffa2ea00640 1 -- 172.21.15.28:0/2361110093 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7ffa34004bc0 con 0x560fa91be350 2024-04-23T14:55:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.680+0000 7ffa5278d740 10 client.16845 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:55:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.680+0000 7ffa5278d740 10 client.16845 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:55:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.680+0000 7ffa5278d740 20 client.16845 populate_metadata read hostname 'smithi028' 2024-04-23T14:55:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.680+0000 7ffa5278d740 1 --2- 172.21.15.28:0/2361110093 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560fa91d8dd0 0x560fa91db1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.681+0000 7ffa5278d740 4 client.16845 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:55:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.681+0000 7ffa5278d740 1 -- 172.21.15.28:0/2361110093 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560fa9130c10 con 0x560fa91d8dd0 2024-04-23T14:55:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.681+0000 7ffa4d000640 1 --2- 172.21.15.28:0/2361110093 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560fa91d8dd0 0x560fa91db1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.681+0000 7ffa4d000640 1 --2- 172.21.15.28:0/2361110093 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560fa91d8dd0 0x560fa91db1e0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a47627480043e8b3 server_cookie=294712a6cd174544 in_seq=0 out_seq=0 2024-04-23T14:55:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.681+0000 7ffa2ea00640 10 client.16845 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:55:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.682+0000 7ffa2ea00640 1 -- 172.21.15.28:0/2361110093 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560fa9130c10 con 0x560fa91d8dd0 2024-04-23T14:55:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.682+0000 7ffa2ea00640 10 client.16845 handle_command_reply: tid=0 2024-04-23T14:55:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.682+0000 7ffa5278d740 1 client.16845 shutdown 2024-04-23T14:55:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.682+0000 7ffa5278d740 1 -- 172.21.15.28:0/2361110093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fa91be350 msgr2=0x560fa91c5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.682+0000 7ffa5278d740 1 --2- 172.21.15.28:0/2361110093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fa91be350 0x560fa91c5f70 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ffa34004610 tx=0x7ffa3409c090 comp rx=0 tx=0).stop 2024-04-23T14:55:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.682+0000 7ffa5278d740 1 -- 172.21.15.28:0/2361110093 shutdown_connections 2024-04-23T14:55:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.682+0000 7ffa5278d740 1 --2- 172.21.15.28:0/2361110093 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560fa91d8dd0 0x560fa91db1e0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.683+0000 7ffa5278d740 1 --2- 172.21.15.28:0/2361110093 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560fa91c8a20 0x560fa91c64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.683+0000 7ffa2ea00640 0 client.16845 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:55:38.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.683+0000 7ffa5278d740 1 --2- 172.21.15.28:0/2361110093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fa91be350 0x560fa91c5f70 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.683+0000 7ffa5278d740 1 --2- 172.21.15.28:0/2361110093 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fa91bda30 0x560fa91c5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.683+0000 7ffa5278d740 1 -- 172.21.15.28:0/2361110093 >> 172.21.15.28:0/2361110093 conn(0x560fa91b9d50 msgr2=0x560fa91b8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:38.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.683+0000 7ffa5278d740 1 -- 172.21.15.28:0/2361110093 shutdown_connections 2024-04-23T14:55:38.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.683+0000 7ffa5278d740 1 -- 172.21.15.28:0/2361110093 wait complete. 2024-04-23T14:55:38.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.683+0000 7ffa5278d740 20 client.16845 trim_cache size 0 max 16384 2024-04-23T14:55:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.703+0000 7ffa5278d740 1 Processor -- start 2024-04-23T14:55:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.703+0000 7ffa5278d740 1 -- start start 2024-04-23T14:55:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.703+0000 7ffa5278d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fa91bda30 0x560fa91b9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.703+0000 7ffa5278d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560fa91be350 0x560fa91c01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.703+0000 7ffa5278d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fa91c8a20 0x560fa91c6100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.703+0000 7ffa5278d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa481ade30 con 0x560fa91c8a20 2024-04-23T14:55:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.703+0000 7ffa5278d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ffa481b0690 con 0x560fa91be350 2024-04-23T14:55:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.703+0000 7ffa5278d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ffa481ae480 con 0x560fa91bda30 2024-04-23T14:55:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.704+0000 7ffa4da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fa91c8a20 0x560fa91c6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.704+0000 7ffa4d000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fa91bda30 0x560fa91b9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.704+0000 7ffa4d000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fa91bda30 0x560fa91b9750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:53070/0 (socket says 172.21.15.28:53070) 2024-04-23T14:55:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.704+0000 7ffa4da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fa91c8a20 0x560fa91c6100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39698/0 (socket says 172.21.15.28:39698) 2024-04-23T14:55:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.704+0000 7ffa4d000640 1 -- 172.21.15.28:0/2202411849 learned_addr learned my addr 172.21.15.28:0/2202411849 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.704+0000 7ffa47e00640 1 --2- 172.21.15.28:0/2202411849 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560fa91be350 0x560fa91c01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.704+0000 7ffa4d000640 1 -- 172.21.15.28:0/2202411849 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560fa91be350 msgr2=0x560fa91c01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.704+0000 7ffa4d000640 1 --2- 172.21.15.28:0/2202411849 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560fa91be350 0x560fa91c01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.704+0000 7ffa4d000640 1 -- 172.21.15.28:0/2202411849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fa91c8a20 msgr2=0x560fa91c6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:38.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.704+0000 7ffa4d000640 1 --2- 172.21.15.28:0/2202411849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fa91c8a20 0x560fa91c6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.704+0000 7ffa4d000640 1 -- 172.21.15.28:0/2202411849 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fa8e353f0 con 0x560fa91bda30 2024-04-23T14:55:38.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.704+0000 7ffa4d000640 1 --2- 172.21.15.28:0/2202411849 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fa91bda30 0x560fa91b9750 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c004a90 tx=0x7ffa3c004ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3990129b5d2bb1cb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:38.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.705+0000 7ffa39600640 1 -- 172.21.15.28:0/2202411849 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c035040 con 0x560fa91bda30 2024-04-23T14:55:38.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.705+0000 7ffa39600640 1 -- 172.21.15.28:0/2202411849 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa3c0042c0 con 0x560fa91bda30 2024-04-23T14:55:38.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.705+0000 7ffa39600640 1 -- 172.21.15.28:0/2202411849 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c003a10 con 0x560fa91bda30 2024-04-23T14:55:38.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.705+0000 7ffa5278d740 1 -- 172.21.15.28:0/2202411849 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fa91bda30 msgr2=0x560fa91b9750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:38.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.705+0000 7ffa5278d740 1 --2- 172.21.15.28:0/2202411849 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fa91bda30 0x560fa91b9750 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c004a90 tx=0x7ffa3c004ac0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.705+0000 7ffa5278d740 1 -- 172.21.15.28:0/2202411849 shutdown_connections 2024-04-23T14:55:38.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.705+0000 7ffa5278d740 1 --2- 172.21.15.28:0/2202411849 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560fa91be350 0x560fa91c01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.705+0000 7ffa5278d740 1 --2- 172.21.15.28:0/2202411849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fa91c8a20 0x560fa91c6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.705+0000 7ffa5278d740 1 --2- 172.21.15.28:0/2202411849 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fa91bda30 0x560fa91b9750 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.705+0000 7ffa5278d740 1 -- 172.21.15.28:0/2202411849 >> 172.21.15.28:0/2202411849 conn(0x560fa91b9d50 msgr2=0x560fa91c59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:38.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.706+0000 7ffa5278d740 1 -- 172.21.15.28:0/2202411849 shutdown_connections 2024-04-23T14:55:38.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.706+0000 7ffa5278d740 1 -- 172.21.15.28:0/2202411849 wait complete. 2024-04-23T14:55:38.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.706+0000 7ffa5278d740 1 -- start start 2024-04-23T14:55:38.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.706+0000 7ffa5278d740 1 Processor -- start 2024-04-23T14:55:38.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.707+0000 7ffa5278d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560fa91bda30 0x560fa91d0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:38.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.707+0000 7ffa5278d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fa91be350 0x560fa91d10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:38.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.707+0000 7ffa5278d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fa91c8a20 0x560fa91c1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:38.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.707+0000 7ffa5278d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa481aeb50 con 0x560fa91be350 2024-04-23T14:55:38.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.707+0000 7ffa5278d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ffa481affc0 con 0x560fa91bda30 2024-04-23T14:55:38.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.707+0000 7ffa5278d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ffa481af220 con 0x560fa91c8a20 2024-04-23T14:55:38.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.707+0000 7ffa5278d740 10 client.0 _ll_register_callbacks cb 0x7ffa340046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:55:38.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.707+0000 7ffa4d000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560fa91bda30 0x560fa91d0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:38.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.707+0000 7ffa47e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fa91be350 0x560fa91d10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:38.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.707+0000 7ffa4d000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560fa91bda30 0x560fa91d0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:53092/0 (socket says 172.21.15.28:53092) 2024-04-23T14:55:38.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.707+0000 7ffa4d000640 1 -- 172.21.15.28:0/3179492601 learned_addr learned my addr 172.21.15.28:0/3179492601 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:38.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.707+0000 7ffa4da00640 1 --2- 172.21.15.28:0/3179492601 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fa91c8a20 0x560fa91c1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:38.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.707+0000 7ffa47e00640 1 -- 172.21.15.28:0/3179492601 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fa91c8a20 msgr2=0x560fa91c1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:38.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.707+0000 7ffa47e00640 1 --2- 172.21.15.28:0/3179492601 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fa91c8a20 0x560fa91c1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.707+0000 7ffa47e00640 1 -- 172.21.15.28:0/3179492601 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560fa91bda30 msgr2=0x560fa91d0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:38.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.707+0000 7ffa47e00640 1 --2- 172.21.15.28:0/3179492601 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560fa91bda30 0x560fa91d0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.707+0000 7ffa47e00640 1 -- 172.21.15.28:0/3179492601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fa8df50f0 con 0x560fa91be350 2024-04-23T14:55:38.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.708+0000 7ffa4d000640 1 --2- 172.21.15.28:0/3179492601 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560fa91bda30 0x560fa91d0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:55:38.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.708+0000 7ffa47e00640 1 --2- 172.21.15.28:0/3179492601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fa91be350 0x560fa91d10c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ffa3409c6c0 tx=0x7ffa3409c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a0d1ca01d727f66 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:38.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.708+0000 7ffa2ea00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-23T14:55:38.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.708+0000 7ffa2ea00640 1 -- 172.21.15.28:0/3179492601 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3409c500 con 0x560fa91be350 2024-04-23T14:55:38.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.708+0000 7ffa2ea00640 1 -- 172.21.15.28:0/3179492601 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8542+0+0 (secure 0 0 0) 0x7ffa340ce160 con 0x560fa91be350 2024-04-23T14:55:38.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.708+0000 7ffa5278d740 1 -- 172.21.15.28:0/3179492601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa3c0042c0 con 0x560fa91be350 2024-04-23T14:55:38.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.708+0000 7ffa2ea00640 1 -- 172.21.15.28:0/3179492601 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7ffa340ce360 con 0x560fa91be350 2024-04-23T14:55:38.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.708+0000 7ffa5278d740 10 client.16851 fetch_fsmap learned FSMap version 20 2024-04-23T14:55:38.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.708+0000 7ffa5278d740 1 -- 172.21.15.28:0/3179492601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x560fa90e8810 con 0x560fa91be350 2024-04-23T14:55:38.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.709+0000 7ffa2ea00640 1 -- 172.21.15.28:0/3179492601 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa3c0042c0 con 0x560fa91be350 2024-04-23T14:55:38.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.709+0000 7ffa2ea00640 1 -- 172.21.15.28:0/3179492601 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7ffa3409b5f0 con 0x560fa91be350 2024-04-23T14:55:38.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.709+0000 7ffa5278d740 10 client.16851 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:55:38.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.709+0000 7ffa5278d740 10 client.16851 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:55:38.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.709+0000 7ffa5278d740 20 client.16851 populate_metadata read hostname 'smithi028' 2024-04-23T14:55:38.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.709+0000 7ffa5278d740 1 --2- 172.21.15.28:0/3179492601 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560fa923ede0 0x560fa9241200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:38.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.709+0000 7ffa5278d740 4 client.16851 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:55:38.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.709+0000 7ffa5278d740 1 -- 172.21.15.28:0/3179492601 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560fa912aa80 con 0x560fa923ede0 2024-04-23T14:55:38.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.709+0000 7ffa4d000640 1 --2- 172.21.15.28:0/3179492601 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560fa923ede0 0x560fa9241200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:38.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.710+0000 7ffa4d000640 1 --2- 172.21.15.28:0/3179492601 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560fa923ede0 0x560fa9241200 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dac44879add7558 server_cookie=2b4635316f8736b in_seq=0 out_seq=0 2024-04-23T14:55:38.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.710+0000 7ffa2ea00640 10 client.16851 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:55:38.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.712+0000 7ffa2ea00640 1 -- 172.21.15.28:0/3179492601 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560fa912aa80 con 0x560fa923ede0 2024-04-23T14:55:38.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.712+0000 7ffa2ea00640 10 client.16851 handle_command_reply: tid=0 2024-04-23T14:55:38.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.712+0000 7ffa5278d740 1 client.16851 shutdown 2024-04-23T14:55:38.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.712+0000 7ffa5278d740 1 -- 172.21.15.28:0/3179492601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fa91be350 msgr2=0x560fa91d10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:38.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.712+0000 7ffa5278d740 1 --2- 172.21.15.28:0/3179492601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fa91be350 0x560fa91d10c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ffa3409c6c0 tx=0x7ffa3409c8b0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.712+0000 7ffa5278d740 1 -- 172.21.15.28:0/3179492601 shutdown_connections 2024-04-23T14:55:38.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.712+0000 7ffa5278d740 1 --2- 172.21.15.28:0/3179492601 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560fa923ede0 0x560fa9241200 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.712+0000 7ffa5278d740 1 --2- 172.21.15.28:0/3179492601 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560fa91c8a20 0x560fa91c1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.712+0000 7ffa5278d740 1 --2- 172.21.15.28:0/3179492601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560fa91be350 0x560fa91d10c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.712+0000 7ffa5278d740 1 --2- 172.21.15.28:0/3179492601 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560fa91bda30 0x560fa91d0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.712+0000 7ffa2ea00640 0 client.16851 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:55:38.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.712+0000 7ffa5278d740 1 -- 172.21.15.28:0/3179492601 >> 172.21.15.28:0/3179492601 conn(0x560fa91b9d50 msgr2=0x560fa91c6280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:38.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.712+0000 7ffa5278d740 1 -- 172.21.15.28:0/3179492601 shutdown_connections 2024-04-23T14:55:38.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.712+0000 7ffa5278d740 1 -- 172.21.15.28:0/3179492601 wait complete. 2024-04-23T14:55:38.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.712+0000 7ffa5278d740 20 client.16851 trim_cache size 0 max 16384 2024-04-23T14:55:38.716 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:55:38.716 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-23T14:55:38.716 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-23T14:55:38.716 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:55:38.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.713+0000 7ffa4e400640 1 -- 172.21.15.28:0/3426576621 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7ffa300a3d20 msgr2=0x7ffa300a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:38.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.713+0000 7ffa4e400640 1 --2- 172.21.15.28:0/3426576621 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7ffa300a3d20 0x7ffa300a61e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ffa400099a0 tx=0x7ffa40009500 comp rx=0 tx=0).stop 2024-04-23T14:55:38.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.713+0000 7ffa4e400640 1 -- 172.21.15.28:0/3426576621 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa481093f0 msgr2=0x7ffa481a0c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:38.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.713+0000 7ffa4e400640 1 --2- 172.21.15.28:0/3426576621 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa481093f0 0x7ffa481a0c00 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7ffa3400e9c0 tx=0x7ffa3400ee80 comp rx=0 tx=0).stop 2024-04-23T14:55:38.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.713+0000 7ffa4e400640 1 -- 172.21.15.28:0/3426576621 shutdown_connections 2024-04-23T14:55:38.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.713+0000 7ffa4e400640 1 --2- 172.21.15.28:0/3426576621 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7ffa300a3d20 0x7ffa300a61e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.713+0000 7ffa4e400640 1 --2- 172.21.15.28:0/3426576621 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa481093f0 0x7ffa481a0c00 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.713+0000 7ffa4e400640 1 --2- 172.21.15.28:0/3426576621 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa4819a830 0x7ffa4819ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.713+0000 7ffa4e400640 1 --2- 172.21.15.28:0/3426576621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa48104320 0x7ffa481a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.713+0000 7ffa4e400640 1 -- 172.21.15.28:0/3426576621 >> 172.21.15.28:0/3426576621 conn(0x7ffa480ffb40 msgr2=0x7ffa48101ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:38.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.713+0000 7ffa4e400640 1 -- 172.21.15.28:0/3426576621 shutdown_connections 2024-04-23T14:55:38.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.713+0000 7ffa4e400640 1 -- 172.21.15.28:0/3426576621 wait complete. 2024-04-23T14:55:38.731 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-23T14:55:38.732 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:55:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.949+0000 7fae07200640 1 -- 172.21.15.28:0/1346973416 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae08109340 msgr2=0x7fae08110f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.949+0000 7fae07200640 1 --2- 172.21.15.28:0/1346973416 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae08109340 0x7fae08110f10 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fae00009ca0 tx=0x7fae0002a8c0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.949+0000 7fae07200640 1 -- 172.21.15.28:0/1346973416 shutdown_connections 2024-04-23T14:55:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.949+0000 7fae07200640 1 --2- 172.21.15.28:0/1346973416 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae08109340 0x7fae08110f10 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.949+0000 7fae07200640 1 --2- 172.21.15.28:0/1346973416 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae08104c20 0x7fae08108c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.949+0000 7fae07200640 1 --2- 172.21.15.28:0/1346973416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae08104270 0x7fae08104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.949+0000 7fae07200640 1 -- 172.21.15.28:0/1346973416 >> 172.21.15.28:0/1346973416 conn(0x7fae080ffad0 msgr2=0x7fae08101ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.949+0000 7fae07200640 1 -- 172.21.15.28:0/1346973416 shutdown_connections 2024-04-23T14:55:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.949+0000 7fae07200640 1 -- 172.21.15.28:0/1346973416 wait complete. 2024-04-23T14:55:38.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.950+0000 7fae07200640 1 Processor -- start 2024-04-23T14:55:38.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.950+0000 7fae07200640 1 -- start start 2024-04-23T14:55:38.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.951+0000 7fae07200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae08104270 0x7fae081a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:38.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.951+0000 7fae07200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae08109340 0x7fae081a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:38.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.951+0000 7fae07200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae0819a6e0 0x7fae0819ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:38.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.951+0000 7fae07200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fae08114440 con 0x7fae08109340 2024-04-23T14:55:38.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.951+0000 7fae07200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fae081142c0 con 0x7fae0819a6e0 2024-04-23T14:55:38.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.951+0000 7fae07200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fae081145c0 con 0x7fae08104270 2024-04-23T14:55:38.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.951+0000 7fae06800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae0819a6e0 0x7fae0819ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:38.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.951+0000 7fae06800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae0819a6e0 0x7fae0819ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:53116/0 (socket says 172.21.15.28:53116) 2024-04-23T14:55:38.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.951+0000 7fae05e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae08104270 0x7fae081a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:38.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.951+0000 7fae06800640 1 -- 172.21.15.28:0/1017078072 learned_addr learned my addr 172.21.15.28:0/1017078072 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:38.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.951+0000 7fae05400640 1 --2- 172.21.15.28:0/1017078072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae08109340 0x7fae081a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:38.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.951+0000 7fae05e00640 1 -- 172.21.15.28:0/1017078072 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae0819a6e0 msgr2=0x7fae0819ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:38.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.951+0000 7fae05e00640 1 --2- 172.21.15.28:0/1017078072 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae0819a6e0 0x7fae0819ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.951+0000 7fae05e00640 1 -- 172.21.15.28:0/1017078072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae08109340 msgr2=0x7fae081a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:38.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.951+0000 7fae05e00640 1 --2- 172.21.15.28:0/1017078072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae08109340 0x7fae081a0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:38.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.951+0000 7fae05e00640 1 -- 172.21.15.28:0/1017078072 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae0819b2e0 con 0x7fae08104270 2024-04-23T14:55:38.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.952+0000 7fae06800640 1 --2- 172.21.15.28:0/1017078072 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae0819a6e0 0x7fae0819ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:55:38.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.952+0000 7fae05e00640 1 --2- 172.21.15.28:0/1017078072 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae08104270 0x7fae081a0620 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fadf800e9f0 tx=0x7fadf800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:38.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.952+0000 7fadfea00640 1 -- 172.21.15.28:0/1017078072 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf800ccc0 con 0x7fae08104270 2024-04-23T14:55:38.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.952+0000 7fadfea00640 1 -- 172.21.15.28:0/1017078072 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadf8010910 con 0x7fae08104270 2024-04-23T14:55:38.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.952+0000 7fadfea00640 1 -- 172.21.15.28:0/1017078072 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf8018960 con 0x7fae08104270 2024-04-23T14:55:38.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.952+0000 7fae07200640 1 -- 172.21.15.28:0/1017078072 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae081a73c0 con 0x7fae08104270 2024-04-23T14:55:38.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.952+0000 7fae07200640 1 -- 172.21.15.28:0/1017078072 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae081a7770 con 0x7fae08104270 2024-04-23T14:55:38.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.954+0000 7fae07200640 1 -- 172.21.15.28:0/1017078072 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae08105740 con 0x7fae08104270 2024-04-23T14:55:38.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.955+0000 7fadfea00640 1 -- 172.21.15.28:0/1017078072 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fadf8010ac0 con 0x7fae08104270 2024-04-23T14:55:38.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.957+0000 7fadfea00640 1 --2- 172.21.15.28:0/1017078072 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fadec0a3da0 0x7fadec0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:38.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.957+0000 7fadfea00640 1 -- 172.21.15.28:0/1017078072 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8542+0+0 (secure 0 0 0) 0x7fadf80d0b50 con 0x7fae08104270 2024-04-23T14:55:38.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.957+0000 7fae05400640 1 --2- 172.21.15.28:0/1017078072 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fadec0a3da0 0x7fadec0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:38.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.958+0000 7fae05400640 1 --2- 172.21.15.28:0/1017078072 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fadec0a3da0 0x7fadec0a6260 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fae0819bf30 tx=0x7fadf4005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:38.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:38.961+0000 7fadfea00640 1 -- 172.21.15.28:0/1017078072 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fadf809cb30 con 0x7fae08104270 2024-04-23T14:55:39.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:38 smithi105 ceph-mon[32555]: pgmap v361: 97 pgs: 97 active+clean; 600 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.5 MiB/s wr, 160 op/s 2024-04-23T14:55:39.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:38 smithi038 ceph-mon[32348]: pgmap v361: 97 pgs: 97 active+clean; 600 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.5 MiB/s wr, 160 op/s 2024-04-23T14:55:39.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:38 smithi028 ceph-mon[27810]: pgmap v361: 97 pgs: 97 active+clean; 600 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.5 MiB/s wr, 160 op/s 2024-04-23T14:55:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.266+0000 7fae07200640 1 -- 172.21.15.28:0/1017078072 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fae08104c20 con 0x7fae08104270 2024-04-23T14:55:39.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.267+0000 7fadfea00640 1 -- 172.21.15.28:0/1017078072 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7fadf80bd0c0 con 0x7fae08104270 2024-04-23T14:55:39.268 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:55:39.268 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:55:39.268 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:55:39.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.270+0000 7fae07200640 1 -- 172.21.15.28:0/1017078072 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fadec0a3da0 msgr2=0x7fadec0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:39.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.270+0000 7fae07200640 1 --2- 172.21.15.28:0/1017078072 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fadec0a3da0 0x7fadec0a6260 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fae0819bf30 tx=0x7fadf4005f90 comp rx=0 tx=0).stop 2024-04-23T14:55:39.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.270+0000 7fae07200640 1 -- 172.21.15.28:0/1017078072 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae08104270 msgr2=0x7fae081a0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:39.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.270+0000 7fae07200640 1 --2- 172.21.15.28:0/1017078072 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae08104270 0x7fae081a0620 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fadf800e9f0 tx=0x7fadf800eeb0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.270+0000 7fae07200640 1 -- 172.21.15.28:0/1017078072 shutdown_connections 2024-04-23T14:55:39.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.270+0000 7fae07200640 1 --2- 172.21.15.28:0/1017078072 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fadec0a3da0 0x7fadec0a6260 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.270+0000 7fae07200640 1 --2- 172.21.15.28:0/1017078072 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae0819a6e0 0x7fae0819ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.270+0000 7fae07200640 1 --2- 172.21.15.28:0/1017078072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae08109340 0x7fae081a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.270+0000 7fae07200640 1 --2- 172.21.15.28:0/1017078072 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae08104270 0x7fae081a0620 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.270+0000 7fae07200640 1 -- 172.21.15.28:0/1017078072 >> 172.21.15.28:0/1017078072 conn(0x7fae080ffad0 msgr2=0x7fae08101d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:39.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.270+0000 7fae07200640 1 -- 172.21.15.28:0/1017078072 shutdown_connections 2024-04-23T14:55:39.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.270+0000 7fae07200640 1 -- 172.21.15.28:0/1017078072 wait complete. 2024-04-23T14:55:39.291 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-23T14:55:39.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.492+0000 7f34f8000640 1 -- 172.21.15.28:0/1147919049 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34f01044b0 msgr2=0x7f34f0104890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:39.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.492+0000 7f34f8000640 1 --2- 172.21.15.28:0/1147919049 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34f01044b0 0x7f34f0104890 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f34e4009900 tx=0x7f34e402a410 comp rx=0 tx=0).stop 2024-04-23T14:55:39.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.492+0000 7f34f8000640 1 -- 172.21.15.28:0/1147919049 shutdown_connections 2024-04-23T14:55:39.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.492+0000 7f34f8000640 1 --2- 172.21.15.28:0/1147919049 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f34f0109580 0x7f34f0111150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.492+0000 7f34f8000640 1 --2- 172.21.15.28:0/1147919049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34f0104e60 0x7f34f0108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.492+0000 7f34f8000640 1 --2- 172.21.15.28:0/1147919049 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34f01044b0 0x7f34f0104890 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.493+0000 7f34f8000640 1 -- 172.21.15.28:0/1147919049 >> 172.21.15.28:0/1147919049 conn(0x7f34f00ffd90 msgr2=0x7f34f01021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:39.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.493+0000 7f34f8000640 1 -- 172.21.15.28:0/1147919049 shutdown_connections 2024-04-23T14:55:39.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.493+0000 7f34f8000640 1 -- 172.21.15.28:0/1147919049 wait complete. 2024-04-23T14:55:39.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.494+0000 7f34f8000640 1 Processor -- start 2024-04-23T14:55:39.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.494+0000 7f34f8000640 1 -- start start 2024-04-23T14:55:39.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.495+0000 7f34f8000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f34f01044b0 0x7f34f01a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:39.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.495+0000 7f34f8000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34f0109580 0x7f34f01a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:39.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.495+0000 7f34f8000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34f019aa80 0x7f34f019af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:39.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.495+0000 7f34f8000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f34f0114660 con 0x7f34f019aa80 2024-04-23T14:55:39.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.495+0000 7f34f8000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f34f01144e0 con 0x7f34f0109580 2024-04-23T14:55:39.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.495+0000 7f34f8000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f34f01147e0 con 0x7f34f01044b0 2024-04-23T14:55:39.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.495+0000 7f34f5600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f34f01044b0 0x7f34f01a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:39.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.495+0000 7f34f5600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f34f01044b0 0x7f34f01a0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:53106/0 (socket says 172.21.15.28:53106) 2024-04-23T14:55:39.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.495+0000 7f34f5600640 1 -- 172.21.15.28:0/388398490 learned_addr learned my addr 172.21.15.28:0/388398490 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:39.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.495+0000 7f34f4c00640 1 --2- 172.21.15.28:0/388398490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34f0109580 0x7f34f01a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:39.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.495+0000 7f34f6000640 1 --2- 172.21.15.28:0/388398490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34f019aa80 0x7f34f019af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:39.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.495+0000 7f34f4c00640 1 -- 172.21.15.28:0/388398490 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f34f01044b0 msgr2=0x7f34f01a0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:39.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.496+0000 7f34f4c00640 1 --2- 172.21.15.28:0/388398490 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f34f01044b0 0x7f34f01a0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.496+0000 7f34f4c00640 1 -- 172.21.15.28:0/388398490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34f019aa80 msgr2=0x7f34f019af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:39.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.496+0000 7f34f4c00640 1 --2- 172.21.15.28:0/388398490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34f019aa80 0x7f34f019af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.496+0000 7f34f4c00640 1 -- 172.21.15.28:0/388398490 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34f019b5a0 con 0x7f34f0109580 2024-04-23T14:55:39.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.496+0000 7f34f6000640 1 --2- 172.21.15.28:0/388398490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34f019aa80 0x7f34f019af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:55:39.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.496+0000 7f34f4c00640 1 --2- 172.21.15.28:0/388398490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34f0109580 0x7f34f01a0dc0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f34dc00e9c0 tx=0x7f34dc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:39.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.496+0000 7f34ee000640 1 -- 172.21.15.28:0/388398490 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34dc00ccc0 con 0x7f34f0109580 2024-04-23T14:55:39.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.496+0000 7f34ee000640 1 -- 172.21.15.28:0/388398490 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34dc010910 con 0x7f34f0109580 2024-04-23T14:55:39.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.496+0000 7f34f8000640 1 -- 172.21.15.28:0/388398490 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34f019b890 con 0x7f34f0109580 2024-04-23T14:55:39.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.496+0000 7f34ee000640 1 -- 172.21.15.28:0/388398490 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34dc0188c0 con 0x7f34f0109580 2024-04-23T14:55:39.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.496+0000 7f34f8000640 1 -- 172.21.15.28:0/388398490 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34f01a7ae0 con 0x7f34f0109580 2024-04-23T14:55:39.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.501+0000 7f34fadf8740 1 Processor -- start 2024-04-23T14:55:39.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.501+0000 7f34fadf8740 1 -- start start 2024-04-23T14:55:39.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.502+0000 7f34ee000640 1 -- 172.21.15.28:0/388398490 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f34dc010ac0 con 0x7f34f0109580 2024-04-23T14:55:39.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.502+0000 7f34fadf8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55877b086950 0x55877b086d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:39.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.502+0000 7f34fadf8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55877b087270 0x55877b091400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:39.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.502+0000 7f34fadf8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55877b091940 0x55877b093d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:39.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.502+0000 7f34fadf8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55877afec1d0 con 0x55877b086950 2024-04-23T14:55:39.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.502+0000 7f34fadf8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55877af92450 con 0x55877b091940 2024-04-23T14:55:39.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.502+0000 7f34fadf8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55877afe8190 con 0x55877b087270 2024-04-23T14:55:39.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.502+0000 7f34f5600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55877b087270 0x55877b091400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:39.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.502+0000 7f34f5600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55877b087270 0x55877b091400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:53114/0 (socket says 172.21.15.28:53114) 2024-04-23T14:55:39.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.502+0000 7f34f5600640 1 -- 172.21.15.28:0/4153012286 learned_addr learned my addr 172.21.15.28:0/4153012286 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:39.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.502+0000 7f34f4c00640 1 --2- 172.21.15.28:0/4153012286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55877b091940 0x55877b093d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:39.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.503+0000 7f34f5600640 1 -- 172.21.15.28:0/4153012286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55877b091940 msgr2=0x55877b093d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:39.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.503+0000 7f34f5600640 1 --2- 172.21.15.28:0/4153012286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55877b091940 0x55877b093d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.503+0000 7f34f5600640 1 -- 172.21.15.28:0/4153012286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55877b086950 msgr2=0x55877b086d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:39.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.503+0000 7f34f5600640 1 --2- 172.21.15.28:0/4153012286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55877b086950 0x55877b086d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.503+0000 7f34f5600640 1 -- 172.21.15.28:0/4153012286 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55877b04f310 con 0x55877b087270 2024-04-23T14:55:39.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.504+0000 7f34f5600640 1 --2- 172.21.15.28:0/4153012286 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55877b087270 0x55877b091400 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f34e4006010 tx=0x7f34e4005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a97cc50856e6c75a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:39.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.504+0000 7f34e2600640 1 -- 172.21.15.28:0/4153012286 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e4040070 con 0x55877b087270 2024-04-23T14:55:39.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.504+0000 7f34e2600640 1 -- 172.21.15.28:0/4153012286 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34e4004480 con 0x55877b087270 2024-04-23T14:55:39.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.504+0000 7f34e2600640 1 -- 172.21.15.28:0/4153012286 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e4035040 con 0x55877b087270 2024-04-23T14:55:39.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.504+0000 7f34ee000640 1 --2- 172.21.15.28:0/388398490 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f34d80a3da0 0x7f34d80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:39.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.505+0000 7f34ee000640 1 -- 172.21.15.28:0/388398490 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8542+0+0 (secure 0 0 0) 0x7f34dc0d0c10 con 0x7f34f0109580 2024-04-23T14:55:39.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.505+0000 7f34f6000640 1 --2- 172.21.15.28:0/388398490 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f34d80a3da0 0x7f34d80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:39.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.505+0000 7f34fadf8740 1 -- 172.21.15.28:0/4153012286 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55877b087270 msgr2=0x55877b091400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:39.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.505+0000 7f34fadf8740 1 --2- 172.21.15.28:0/4153012286 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55877b087270 0x55877b091400 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f34e4006010 tx=0x7f34e4005e80 comp rx=0 tx=0).stop 2024-04-23T14:55:39.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.505+0000 7f34fadf8740 1 -- 172.21.15.28:0/4153012286 shutdown_connections 2024-04-23T14:55:39.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.505+0000 7f34fadf8740 1 --2- 172.21.15.28:0/4153012286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55877b091940 0x55877b093d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.505+0000 7f34fadf8740 1 --2- 172.21.15.28:0/4153012286 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55877b087270 0x55877b091400 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.505+0000 7f34fadf8740 1 --2- 172.21.15.28:0/4153012286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55877b086950 0x55877b086d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.505+0000 7f34fadf8740 1 -- 172.21.15.28:0/4153012286 >> 172.21.15.28:0/4153012286 conn(0x55877b082c70 msgr2=0x55877b083050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:39.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.505+0000 7f34fadf8740 1 -- 172.21.15.28:0/4153012286 shutdown_connections 2024-04-23T14:55:39.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.505+0000 7f34fadf8740 1 -- 172.21.15.28:0/4153012286 wait complete. 2024-04-23T14:55:39.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.506+0000 7f34f6000640 1 --2- 172.21.15.28:0/388398490 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f34d80a3da0 0x7f34d80a6260 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f34e8009370 tx=0x7f34e80092e0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:39.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.506+0000 7f34fadf8740 1 -- start start 2024-04-23T14:55:39.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.506+0000 7f34fadf8740 1 Processor -- start 2024-04-23T14:55:39.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.506+0000 7f34fadf8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55877b086950 0x55877b0913d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:39.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.507+0000 7f34fadf8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55877b087270 0x55877b08e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:39.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.507+0000 7f34fadf8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55877b091940 0x55877b08edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:39.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.507+0000 7f34fadf8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55877b094770 con 0x55877b087270 2024-04-23T14:55:39.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.507+0000 7f34fadf8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55877afa5810 con 0x55877b086950 2024-04-23T14:55:39.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.507+0000 7f34fadf8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55877b0948f0 con 0x55877b091940 2024-04-23T14:55:39.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.507+0000 7f34fadf8740 10 client.0 _ll_register_callbacks cb 0x55877af73140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:55:39.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.507+0000 7f34f5600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55877b086950 0x55877b0913d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:39.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.507+0000 7f34f5600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55877b086950 0x55877b0913d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:53154/0 (socket says 172.21.15.28:53154) 2024-04-23T14:55:39.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.507+0000 7f34f5600640 1 -- 172.21.15.28:0/903103188 learned_addr learned my addr 172.21.15.28:0/903103188 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:39.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.507+0000 7f34f6000640 1 --2- 172.21.15.28:0/903103188 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55877b091940 0x55877b08edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:39.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.507+0000 7f34f4c00640 1 --2- 172.21.15.28:0/903103188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55877b087270 0x55877b08e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:39.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.507+0000 7f34f5600640 1 -- 172.21.15.28:0/903103188 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55877b091940 msgr2=0x55877b08edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:39.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.507+0000 7f34f5600640 1 --2- 172.21.15.28:0/903103188 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55877b091940 0x55877b08edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.507+0000 7f34f5600640 1 -- 172.21.15.28:0/903103188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55877b087270 msgr2=0x55877b08e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:39.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.507+0000 7f34f5600640 1 --2- 172.21.15.28:0/903103188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55877b087270 0x55877b08e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.507+0000 7f34f5600640 1 -- 172.21.15.28:0/903103188 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55877afb63f0 con 0x55877b086950 2024-04-23T14:55:39.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.508+0000 7f34f5600640 1 --2- 172.21.15.28:0/903103188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55877b086950 0x55877b0913d0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f34e402a450 tx=0x7f34e4004b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3caa8803cce61eaf server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:39.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.508+0000 7f34d6a00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:55:39.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.508+0000 7f34d6a00640 1 -- 172.21.15.28:0/903103188 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e4040070 con 0x55877b086950 2024-04-23T14:55:39.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.508+0000 7f34fadf8740 1 -- 172.21.15.28:0/903103188 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34e4004480 con 0x55877b086950 2024-04-23T14:55:39.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.508+0000 7f34d6a00640 1 -- 172.21.15.28:0/903103188 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8542+0+0 (secure 0 0 0) 0x7f34e4005090 con 0x55877b086950 2024-04-23T14:55:39.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.508+0000 7f34d6a00640 1 -- 172.21.15.28:0/903103188 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f34e4056050 con 0x55877b086950 2024-04-23T14:55:39.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.508+0000 7f34fadf8740 10 client.27059 fetch_fsmap learned FSMap version 20 2024-04-23T14:55:39.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.508+0000 7f34fadf8740 1 -- 172.21.15.28:0/903103188 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55877acfe3f0 con 0x55877b086950 2024-04-23T14:55:39.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.509+0000 7f34d6a00640 1 -- 172.21.15.28:0/903103188 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34e4004480 con 0x55877b086950 2024-04-23T14:55:39.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.509+0000 7f34d6a00640 1 -- 172.21.15.28:0/903103188 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f34e40035f0 con 0x55877b086950 2024-04-23T14:55:39.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.509+0000 7f34fadf8740 10 client.27059 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:55:39.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.509+0000 7f34fadf8740 10 client.27059 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:55:39.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.509+0000 7f34fadf8740 20 client.27059 populate_metadata read hostname 'smithi028' 2024-04-23T14:55:39.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.509+0000 7f34fadf8740 1 --2- 172.21.15.28:0/903103188 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55877b0a1980 0x55877b0a3d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:39.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.509+0000 7f34fadf8740 4 client.27059 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:55:39.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.509+0000 7f34fadf8740 1 -- 172.21.15.28:0/903103188 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55877aff9bf0 con 0x55877b0a1980 2024-04-23T14:55:39.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.509+0000 7f34f4c00640 1 --2- 172.21.15.28:0/903103188 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55877b0a1980 0x55877b0a3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:39.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.510+0000 7f34f4c00640 1 --2- 172.21.15.28:0/903103188 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55877b0a1980 0x55877b0a3d70 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ab94b926b3d2318 server_cookie=d05ff7cc578eacd8 in_seq=0 out_seq=0 2024-04-23T14:55:39.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.510+0000 7f34d6a00640 10 client.27059 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:55:39.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.511+0000 7f34d6a00640 1 -- 172.21.15.28:0/903103188 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55877aff9bf0 con 0x55877b0a1980 2024-04-23T14:55:39.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.511+0000 7f34d6a00640 10 client.27059 handle_command_reply: tid=0 2024-04-23T14:55:39.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.511+0000 7f34fadf8740 1 client.27059 shutdown 2024-04-23T14:55:39.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.511+0000 7f34fadf8740 1 -- 172.21.15.28:0/903103188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55877b086950 msgr2=0x55877b0913d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:39.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.511+0000 7f34fadf8740 1 --2- 172.21.15.28:0/903103188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55877b086950 0x55877b0913d0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f34e402a450 tx=0x7f34e4004b20 comp rx=0 tx=0).stop 2024-04-23T14:55:39.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.512+0000 7f34fadf8740 1 -- 172.21.15.28:0/903103188 shutdown_connections 2024-04-23T14:55:39.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.512+0000 7f34fadf8740 1 --2- 172.21.15.28:0/903103188 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55877b0a1980 0x55877b0a3d70 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.512+0000 7f34fadf8740 1 --2- 172.21.15.28:0/903103188 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55877b091940 0x55877b08edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.512+0000 7f34fadf8740 1 --2- 172.21.15.28:0/903103188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55877b087270 0x55877b08e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.512+0000 7f34fadf8740 1 --2- 172.21.15.28:0/903103188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55877b086950 0x55877b0913d0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.512+0000 7f34d6a00640 0 client.27059 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:55:39.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.512+0000 7f34fadf8740 1 -- 172.21.15.28:0/903103188 >> 172.21.15.28:0/903103188 conn(0x55877b082c70 msgr2=0x55877b088a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:39.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.512+0000 7f34fadf8740 1 -- 172.21.15.28:0/903103188 shutdown_connections 2024-04-23T14:55:39.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.512+0000 7f34fadf8740 1 -- 172.21.15.28:0/903103188 wait complete. 2024-04-23T14:55:39.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.512+0000 7f34fadf8740 20 client.27059 trim_cache size 0 max 16384 2024-04-23T14:55:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.531+0000 7f34fadf8740 1 Processor -- start 2024-04-23T14:55:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.531+0000 7f34fadf8740 1 -- start start 2024-04-23T14:55:39.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.532+0000 7f34fadf8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55877b086950 0x55877b088e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:39.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.532+0000 7f34fadf8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55877b087270 0x55877b08d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:39.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.532+0000 7f34fadf8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55877b091940 0x55877b090f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:39.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.532+0000 7f34fadf8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f34f01adf60 con 0x55877b087270 2024-04-23T14:55:39.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.532+0000 7f34fadf8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f34f01b07c0 con 0x55877b091940 2024-04-23T14:55:39.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.532+0000 7f34fadf8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f34f01ae5b0 con 0x55877b086950 2024-04-23T14:55:39.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.533+0000 7f34f6000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55877b091940 0x55877b090f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:39.534 INFO:tasks.workunit.client.0.smithi028.stdout: 1 990266 8.64 MB/sec execute 475 sec latency 33.936 ms 2024-04-23T14:55:39.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.533+0000 7f34f6000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55877b091940 0x55877b090f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:53160/0 (socket says 172.21.15.28:53160) 2024-04-23T14:55:39.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.533+0000 7f34f6000640 1 -- 172.21.15.28:0/4231790868 learned_addr learned my addr 172.21.15.28:0/4231790868 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:39.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.534+0000 7f34f5600640 1 --2- 172.21.15.28:0/4231790868 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55877b086950 0x55877b088e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:39.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.534+0000 7f34f4c00640 1 --2- 172.21.15.28:0/4231790868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55877b087270 0x55877b08d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:39.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.534+0000 7f34f6000640 1 -- 172.21.15.28:0/4231790868 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55877b086950 msgr2=0x55877b088e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:39.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.534+0000 7f34f6000640 1 --2- 172.21.15.28:0/4231790868 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55877b086950 0x55877b088e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.534+0000 7f34f6000640 1 -- 172.21.15.28:0/4231790868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55877b087270 msgr2=0x55877b08d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:39.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.535+0000 7f34f6000640 1 --2- 172.21.15.28:0/4231790868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55877b087270 0x55877b08d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.535+0000 7f34f6000640 1 -- 172.21.15.28:0/4231790868 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55877b0870a0 con 0x55877b091940 2024-04-23T14:55:39.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.535+0000 7f34f4c00640 1 --2- 172.21.15.28:0/4231790868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55877b087270 0x55877b08d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:55:39.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.536+0000 7f34f6000640 1 --2- 172.21.15.28:0/4231790868 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55877b091940 0x55877b090f00 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f34e8011600 tx=0x7f34e8012000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=409aab4c525eb0d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:39.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.537+0000 7f34e1c00640 1 -- 172.21.15.28:0/4231790868 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e8010710 con 0x55877b091940 2024-04-23T14:55:39.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.537+0000 7f34e1c00640 1 -- 172.21.15.28:0/4231790868 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34e8023070 con 0x55877b091940 2024-04-23T14:55:39.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.537+0000 7f34e1c00640 1 -- 172.21.15.28:0/4231790868 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e801e430 con 0x55877b091940 2024-04-23T14:55:39.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.541+0000 7f34fadf8740 1 -- 172.21.15.28:0/4231790868 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55877b091940 msgr2=0x55877b090f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:39.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.542+0000 7f34fadf8740 1 --2- 172.21.15.28:0/4231790868 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55877b091940 0x55877b090f00 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f34e8011600 tx=0x7f34e8012000 comp rx=0 tx=0).stop 2024-04-23T14:55:39.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.542+0000 7f34fadf8740 1 -- 172.21.15.28:0/4231790868 shutdown_connections 2024-04-23T14:55:39.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.542+0000 7f34fadf8740 1 --2- 172.21.15.28:0/4231790868 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55877b091940 0x55877b090f00 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.542+0000 7f34fadf8740 1 --2- 172.21.15.28:0/4231790868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55877b087270 0x55877b08d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.542+0000 7f34fadf8740 1 --2- 172.21.15.28:0/4231790868 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55877b086950 0x55877b088e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.542+0000 7f34fadf8740 1 -- 172.21.15.28:0/4231790868 >> 172.21.15.28:0/4231790868 conn(0x55877b082c70 msgr2=0x55877b08ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:39.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.542+0000 7f34fadf8740 1 -- 172.21.15.28:0/4231790868 shutdown_connections 2024-04-23T14:55:39.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.543+0000 7f34fadf8740 1 -- 172.21.15.28:0/4231790868 wait complete. 2024-04-23T14:55:39.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.543+0000 7f34fadf8740 1 -- start start 2024-04-23T14:55:39.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.544+0000 7f34fadf8740 1 Processor -- start 2024-04-23T14:55:39.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.545+0000 7f34fadf8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55877b086950 0x55877b0924f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:39.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.545+0000 7f34fadf8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55877b087270 0x55877b092a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:39.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.545+0000 7f34fadf8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55877b091940 0x55877b098b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:39.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.545+0000 7f34fadf8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f34f01aec80 con 0x55877b086950 2024-04-23T14:55:39.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.545+0000 7f34fadf8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f34f01b00f0 con 0x55877b091940 2024-04-23T14:55:39.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.545+0000 7f34fadf8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f34f01af350 con 0x55877b087270 2024-04-23T14:55:39.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.545+0000 7f34fadf8740 10 client.0 _ll_register_callbacks cb 0x7f34e4005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:55:39.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.546+0000 7f34f4c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55877b087270 0x55877b092a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:39.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.546+0000 7f34f6000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55877b091940 0x55877b098b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:39.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.546+0000 7f34f4c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55877b087270 0x55877b092a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:53140/0 (socket says 172.21.15.28:53140) 2024-04-23T14:55:39.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.546+0000 7f34f4c00640 1 -- 172.21.15.28:0/2121503552 learned_addr learned my addr 172.21.15.28:0/2121503552 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:39.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.546+0000 7f34f5600640 1 --2- 172.21.15.28:0/2121503552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55877b086950 0x55877b0924f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:39.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.547+0000 7f34f4c00640 1 -- 172.21.15.28:0/2121503552 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55877b091940 msgr2=0x55877b098b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:39.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.547+0000 7f34f4c00640 1 --2- 172.21.15.28:0/2121503552 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55877b091940 0x55877b098b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.547+0000 7f34f4c00640 1 -- 172.21.15.28:0/2121503552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55877b086950 msgr2=0x55877b0924f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:39.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.547+0000 7f34f4c00640 1 --2- 172.21.15.28:0/2121503552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55877b086950 0x55877b0924f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.547+0000 7f34f4c00640 1 -- 172.21.15.28:0/2121503552 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55877adc87d0 con 0x55877b087270 2024-04-23T14:55:39.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.548+0000 7f34f4c00640 1 --2- 172.21.15.28:0/2121503552 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55877b087270 0x55877b092a30 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f34dc0ce400 tx=0x7f34dc00efb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1911e9f4e9e65b30 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:39.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.548+0000 7f34d6a00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:55:39.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.548+0000 7f34d6a00640 1 -- 172.21.15.28:0/2121503552 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34dc0a0f50 con 0x55877b087270 2024-04-23T14:55:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.549+0000 7f34d6a00640 1 -- 172.21.15.28:0/2121503552 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8542+0+0 (secure 0 0 0) 0x7f34dc0ce5d0 con 0x55877b087270 2024-04-23T14:55:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.549+0000 7f34fadf8740 1 -- 172.21.15.28:0/2121503552 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34e8023070 con 0x55877b087270 2024-04-23T14:55:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.550+0000 7f34d6a00640 1 -- 172.21.15.28:0/2121503552 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f34dc0a5810 con 0x55877b087270 2024-04-23T14:55:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.550+0000 7f34fadf8740 10 client.27046 fetch_fsmap learned FSMap version 20 2024-04-23T14:55:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.551+0000 7f34fadf8740 1 -- 172.21.15.28:0/2121503552 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55877b09a2d0 con 0x55877b087270 2024-04-23T14:55:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.551+0000 7f34d6a00640 1 -- 172.21.15.28:0/2121503552 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34e8023070 con 0x55877b087270 2024-04-23T14:55:39.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.553+0000 7f34d6a00640 1 -- 172.21.15.28:0/2121503552 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f34dc0988f0 con 0x55877b087270 2024-04-23T14:55:39.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.553+0000 7f34fadf8740 10 client.27046 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:55:39.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.553+0000 7f34fadf8740 10 client.27046 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:55:39.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.553+0000 7f34fadf8740 20 client.27046 populate_metadata read hostname 'smithi028' 2024-04-23T14:55:39.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.553+0000 7f34fadf8740 1 --2- 172.21.15.28:0/2121503552 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55877b107f50 0x55877b11b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:39.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.553+0000 7f34fadf8740 4 client.27046 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-23T14:55:39.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.553+0000 7f34fadf8740 1 -- 172.21.15.28:0/2121503552 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55877aff3a60 con 0x55877b107f50 2024-04-23T14:55:39.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.553+0000 7f34f5600640 1 --2- 172.21.15.28:0/2121503552 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55877b107f50 0x55877b11b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:39.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.554+0000 7f34f5600640 1 --2- 172.21.15.28:0/2121503552 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55877b107f50 0x55877b11b3a0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e675135186be0896 server_cookie=300f4f0939df2832 in_seq=0 out_seq=0 2024-04-23T14:55:39.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.554+0000 7f34d6a00640 10 client.27046 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:55:39.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.555+0000 7f34d6a00640 1 -- 172.21.15.28:0/2121503552 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55877aff3a60 con 0x55877b107f50 2024-04-23T14:55:39.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.555+0000 7f34d6a00640 10 client.27046 handle_command_reply: tid=0 2024-04-23T14:55:39.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.555+0000 7f34fadf8740 1 client.27046 shutdown 2024-04-23T14:55:39.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.556+0000 7f34fadf8740 1 -- 172.21.15.28:0/2121503552 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55877b087270 msgr2=0x55877b092a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:39.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.556+0000 7f34fadf8740 1 --2- 172.21.15.28:0/2121503552 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55877b087270 0x55877b092a30 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f34dc0ce400 tx=0x7f34dc00efb0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.556+0000 7f34fadf8740 1 -- 172.21.15.28:0/2121503552 shutdown_connections 2024-04-23T14:55:39.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.556+0000 7f34fadf8740 1 --2- 172.21.15.28:0/2121503552 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55877b107f50 0x55877b11b3a0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.556+0000 7f34fadf8740 1 --2- 172.21.15.28:0/2121503552 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55877b091940 0x55877b098b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.556+0000 7f34fadf8740 1 --2- 172.21.15.28:0/2121503552 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55877b087270 0x55877b092a30 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.556+0000 7f34fadf8740 1 --2- 172.21.15.28:0/2121503552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55877b086950 0x55877b0924f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.557+0000 7f34fadf8740 1 -- 172.21.15.28:0/2121503552 >> 172.21.15.28:0/2121503552 conn(0x55877b082c70 msgr2=0x55877b08d6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:39.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.557+0000 7f34d6a00640 0 client.27046 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:55:39.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.558+0000 7f34fadf8740 1 -- 172.21.15.28:0/2121503552 shutdown_connections 2024-04-23T14:55:39.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.558+0000 7f34fadf8740 1 -- 172.21.15.28:0/2121503552 wait complete. 2024-04-23T14:55:39.560 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-23T14:55:39.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.559+0000 7f34fadf8740 20 client.27046 trim_cache size 0 max 16384 2024-04-23T14:55:39.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.560+0000 7f34f8000640 1 -- 172.21.15.28:0/388398490 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f34d80a3da0 msgr2=0x7f34d80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:39.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.560+0000 7f34f8000640 1 --2- 172.21.15.28:0/388398490 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f34d80a3da0 0x7f34d80a6260 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f34e8009370 tx=0x7f34e80092e0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.560+0000 7f34f8000640 1 -- 172.21.15.28:0/388398490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34f0109580 msgr2=0x7f34f01a0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:39.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.560+0000 7f34f8000640 1 --2- 172.21.15.28:0/388398490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34f0109580 0x7f34f01a0dc0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f34dc00e9c0 tx=0x7f34dc00ee80 comp rx=0 tx=0).stop 2024-04-23T14:55:39.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.561+0000 7f34f8000640 1 -- 172.21.15.28:0/388398490 shutdown_connections 2024-04-23T14:55:39.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.561+0000 7f34f8000640 1 --2- 172.21.15.28:0/388398490 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f34d80a3da0 0x7f34d80a6260 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.561+0000 7f34f8000640 1 --2- 172.21.15.28:0/388398490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34f0109580 0x7f34f01a0dc0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.561+0000 7f34f8000640 1 --2- 172.21.15.28:0/388398490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34f019aa80 0x7f34f019af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.561+0000 7f34f8000640 1 --2- 172.21.15.28:0/388398490 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f34f01044b0 0x7f34f01a0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:39.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.561+0000 7f34f8000640 1 -- 172.21.15.28:0/388398490 >> 172.21.15.28:0/388398490 conn(0x7f34f00ffd90 msgr2=0x7f34f01010c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:39.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.561+0000 7f34f8000640 1 -- 172.21.15.28:0/388398490 shutdown_connections 2024-04-23T14:55:39.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:39.562+0000 7f34f8000640 1 -- 172.21.15.28:0/388398490 wait complete. 2024-04-23T14:55:39.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:39 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1017078072' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:55:40.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:39 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1017078072' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:55:40.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:39 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1017078072' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:55:40.534 INFO:tasks.workunit.client.0.smithi028.stdout: 1 991813 8.65 MB/sec execute 476 sec latency 22.978 ms 2024-04-23T14:55:40.590 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-23T14:55:40.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:55:40.590 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-23T14:55:40.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.822+0000 7fe3ae400640 1 -- 172.21.15.28:0/1297945971 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe3a8104320 msgr2=0x7fe3a8104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:40.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.822+0000 7fe3ae400640 1 --2- 172.21.15.28:0/1297945971 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe3a8104320 0x7fe3a8104700 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe39c009860 tx=0x7fe39c02a340 comp rx=0 tx=0).stop 2024-04-23T14:55:40.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.822+0000 7fe3ae400640 1 -- 172.21.15.28:0/1297945971 shutdown_connections 2024-04-23T14:55:40.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.822+0000 7fe3ae400640 1 --2- 172.21.15.28:0/1297945971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3a8104cd0 0x7fe3a8108d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:40.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.822+0000 7fe3ae400640 1 --2- 172.21.15.28:0/1297945971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3a81093f0 0x7fe3a8110fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:40.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.822+0000 7fe3ae400640 1 --2- 172.21.15.28:0/1297945971 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe3a8104320 0x7fe3a8104700 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:40.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.822+0000 7fe3ae400640 1 -- 172.21.15.28:0/1297945971 >> 172.21.15.28:0/1297945971 conn(0x7fe3a80ffb40 msgr2=0x7fe3a8101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:40.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.822+0000 7fe3ae400640 1 -- 172.21.15.28:0/1297945971 shutdown_connections 2024-04-23T14:55:40.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.822+0000 7fe3ae400640 1 -- 172.21.15.28:0/1297945971 wait complete. 2024-04-23T14:55:40.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.823+0000 7fe3ae400640 1 Processor -- start 2024-04-23T14:55:40.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.824+0000 7fe3ae400640 1 -- start start 2024-04-23T14:55:40.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.824+0000 7fe3ae400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3a8104320 0x7fe3a81a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:40.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.824+0000 7fe3ae400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe3a81093f0 0x7fe3a81a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:40.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.824+0000 7fe3ae400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3a819a870 0x7fe3a819ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:40.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.824+0000 7fe3ae400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a81144d0 con 0x7fe3a819a870 2024-04-23T14:55:40.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.824+0000 7fe3ae400640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a8114350 con 0x7fe3a8104320 2024-04-23T14:55:40.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.824+0000 7fe3ae400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a8114650 con 0x7fe3a81093f0 2024-04-23T14:55:40.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.824+0000 7fe3ad000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3a8104320 0x7fe3a81a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:40.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.825+0000 7fe3ad000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3a8104320 0x7fe3a81a06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:53184/0 (socket says 172.21.15.28:53184) 2024-04-23T14:55:40.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.825+0000 7fe3ad000640 1 -- 172.21.15.28:0/590816610 learned_addr learned my addr 172.21.15.28:0/590816610 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:40.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.825+0000 7fe3ada00640 1 --2- 172.21.15.28:0/590816610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3a819a870 0x7fe3a819ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:40.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.825+0000 7fe3a7e00640 1 --2- 172.21.15.28:0/590816610 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe3a81093f0 0x7fe3a81a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:40.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.825+0000 7fe3ad000640 1 -- 172.21.15.28:0/590816610 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe3a81093f0 msgr2=0x7fe3a81a0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:40.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.825+0000 7fe3ad000640 1 --2- 172.21.15.28:0/590816610 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe3a81093f0 0x7fe3a81a0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:40.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.825+0000 7fe3ad000640 1 -- 172.21.15.28:0/590816610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3a819a870 msgr2=0x7fe3a819ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:40.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.825+0000 7fe3ad000640 1 --2- 172.21.15.28:0/590816610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3a819a870 0x7fe3a819ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:40.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.825+0000 7fe3ad000640 1 -- 172.21.15.28:0/590816610 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3a819b390 con 0x7fe3a8104320 2024-04-23T14:55:40.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.825+0000 7fe3a7e00640 1 --2- 172.21.15.28:0/590816610 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe3a81093f0 0x7fe3a81a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:55:40.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.825+0000 7fe3ad000640 1 --2- 172.21.15.28:0/590816610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3a8104320 0x7fe3a81a06c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fe39c02a8b0 tx=0x7fe39c033d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:40.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.826+0000 7fe3a5600640 1 -- 172.21.15.28:0/590816610 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe39c037040 con 0x7fe3a8104320 2024-04-23T14:55:40.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.826+0000 7fe3a5600640 1 -- 172.21.15.28:0/590816610 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe39c030550 con 0x7fe3a8104320 2024-04-23T14:55:40.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.826+0000 7fe3a5600640 1 -- 172.21.15.28:0/590816610 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe39c003a80 con 0x7fe3a8104320 2024-04-23T14:55:40.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.826+0000 7fe3ae400640 1 -- 172.21.15.28:0/590816610 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3a819b620 con 0x7fe3a8104320 2024-04-23T14:55:40.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.826+0000 7fe3ae400640 1 -- 172.21.15.28:0/590816610 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3a81a77c0 con 0x7fe3a8104320 2024-04-23T14:55:40.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.829+0000 7fe3b2522740 1 Processor -- start 2024-04-23T14:55:40.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.829+0000 7fe3a5600640 1 -- 172.21.15.28:0/590816610 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fe39c042020 con 0x7fe3a8104320 2024-04-23T14:55:40.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.830+0000 7fe3b2522740 1 -- start start 2024-04-23T14:55:40.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.830+0000 7fe3b2522740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef3e107270 0x55ef3e107650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:40.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.830+0000 7fe3b2522740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef3e107b90 0x55ef3e111d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:40.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.830+0000 7fe3b2522740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3e112260 0x55ef3e114650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:40.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.830+0000 7fe3b2522740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ef3e06ce70 con 0x55ef3e107b90 2024-04-23T14:55:40.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.830+0000 7fe3b2522740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ef3e013380 con 0x55ef3e107270 2024-04-23T14:55:40.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.830+0000 7fe3b2522740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55ef3e068e30 con 0x55ef3e112260 2024-04-23T14:55:40.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.830+0000 7fe3a7e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3e112260 0x55ef3e114650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:40.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.830+0000 7fe3ada00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef3e107270 0x55ef3e107650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:40.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.830+0000 7fe3a7e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3e112260 0x55ef3e114650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:53158/0 (socket says 172.21.15.28:53158) 2024-04-23T14:55:40.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.830+0000 7fe3a7e00640 1 -- 172.21.15.28:0/2865128146 learned_addr learned my addr 172.21.15.28:0/2865128146 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:40.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.830+0000 7fe3ad000640 1 --2- 172.21.15.28:0/2865128146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef3e107b90 0x55ef3e111d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:40.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.831+0000 7fe3ada00640 1 -- 172.21.15.28:0/2865128146 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3e112260 msgr2=0x55ef3e114650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:40.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.831+0000 7fe3ada00640 1 --2- 172.21.15.28:0/2865128146 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3e112260 0x55ef3e114650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:40.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.831+0000 7fe3ada00640 1 -- 172.21.15.28:0/2865128146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef3e107b90 msgr2=0x55ef3e111d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:40.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.831+0000 7fe3ada00640 1 --2- 172.21.15.28:0/2865128146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef3e107b90 0x55ef3e111d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:40.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.831+0000 7fe3ada00640 1 -- 172.21.15.28:0/2865128146 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef3de477e0 con 0x55ef3e107270 2024-04-23T14:55:40.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.831+0000 7fe3a7e00640 1 --2- 172.21.15.28:0/2865128146 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3e112260 0x55ef3e114650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:55:40.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.831+0000 7fe3ada00640 1 --2- 172.21.15.28:0/2865128146 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef3e107270 0x55ef3e107650 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0009eb0 tx=0x7fe3a00107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=801254a1e69d6a18 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:40.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.831+0000 7fe39a000640 1 -- 172.21.15.28:0/2865128146 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a0003b70 con 0x55ef3e107270 2024-04-23T14:55:40.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.832+0000 7fe39a000640 1 -- 172.21.15.28:0/2865128146 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3a0015070 con 0x55ef3e107270 2024-04-23T14:55:40.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.832+0000 7fe39a000640 1 -- 172.21.15.28:0/2865128146 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a001a940 con 0x55ef3e107270 2024-04-23T14:55:40.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.832+0000 7fe3a5600640 1 --2- 172.21.15.28:0/590816610 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe3900a3d60 0x7fe3900a6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:40.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.832+0000 7fe3a5600640 1 -- 172.21.15.28:0/590816610 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8542+0+0 (secure 0 0 0) 0x7fe39c0efe10 con 0x7fe3a8104320 2024-04-23T14:55:40.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.832+0000 7fe3ad000640 1 --2- 172.21.15.28:0/590816610 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe3900a3d60 0x7fe3900a6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:40.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.832+0000 7fe3b2522740 1 -- 172.21.15.28:0/2865128146 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef3e107270 msgr2=0x55ef3e107650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:40.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.832+0000 7fe3b2522740 1 --2- 172.21.15.28:0/2865128146 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef3e107270 0x55ef3e107650 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0009eb0 tx=0x7fe3a00107a0 comp rx=0 tx=0).stop 2024-04-23T14:55:40.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.832+0000 7fe3b2522740 1 -- 172.21.15.28:0/2865128146 shutdown_connections 2024-04-23T14:55:40.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.832+0000 7fe3b2522740 1 --2- 172.21.15.28:0/2865128146 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3e112260 0x55ef3e114650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:40.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.832+0000 7fe3b2522740 1 --2- 172.21.15.28:0/2865128146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef3e107b90 0x55ef3e111d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:40.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.832+0000 7fe3b2522740 1 --2- 172.21.15.28:0/2865128146 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef3e107270 0x55ef3e107650 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:40.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.832+0000 7fe3b2522740 1 -- 172.21.15.28:0/2865128146 >> 172.21.15.28:0/2865128146 conn(0x55ef3e103590 msgr2=0x55ef3e103970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:40.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.833+0000 7fe3b2522740 1 -- 172.21.15.28:0/2865128146 shutdown_connections 2024-04-23T14:55:40.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.833+0000 7fe3b2522740 1 -- 172.21.15.28:0/2865128146 wait complete. 2024-04-23T14:55:40.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.833+0000 7fe3ad000640 1 --2- 172.21.15.28:0/590816610 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe3900a3d60 0x7fe3900a6220 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x55ef3e108830 tx=0x7fe39c0f38c0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:40.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.833+0000 7fe3b2522740 1 -- start start 2024-04-23T14:55:40.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.833+0000 7fe3b2522740 1 Processor -- start 2024-04-23T14:55:40.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.834+0000 7fe3b2522740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef3e107270 0x55ef3e10f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:40.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.834+0000 7fe3b2522740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3e107b90 0x55ef3e10f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:40.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.834+0000 7fe3b2522740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef3e112260 0x55ef3e10fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:40.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.834+0000 7fe3b2522740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ef3e115050 con 0x55ef3e112260 2024-04-23T14:55:40.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.834+0000 7fe3b2522740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ef3e026660 con 0x55ef3e107270 2024-04-23T14:55:40.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.834+0000 7fe3b2522740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55ef3e1151d0 con 0x55ef3e107b90 2024-04-23T14:55:40.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.834+0000 7fe3b2522740 10 client.0 _ll_register_callbacks cb 0x55ef3e014aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:55:40.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.834+0000 7fe3ada00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef3e107270 0x55ef3e10f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:40.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.834+0000 7fe3a7e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3e107b90 0x55ef3e10f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:40.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.834+0000 7fe3ada00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef3e107270 0x55ef3e10f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:53196/0 (socket says 172.21.15.28:53196) 2024-04-23T14:55:40.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.834+0000 7fe3ada00640 1 -- 172.21.15.28:0/1400147395 learned_addr learned my addr 172.21.15.28:0/1400147395 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:40.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.834+0000 7fe3ada00640 1 --2- 172.21.15.28:0/1400147395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef3e112260 0x55ef3e10fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:40.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.835+0000 7fe3a7e00640 1 -- 172.21.15.28:0/1400147395 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef3e107270 msgr2=0x55ef3e10f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:40.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.835+0000 7fe3a7e00640 1 --2- 172.21.15.28:0/1400147395 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef3e107270 0x55ef3e10f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:40.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.835+0000 7fe3a7e00640 1 -- 172.21.15.28:0/1400147395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef3e112260 msgr2=0x55ef3e10fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:40.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.835+0000 7fe3a7e00640 1 --2- 172.21.15.28:0/1400147395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef3e112260 0x55ef3e10fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:40.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.835+0000 7fe3a7e00640 1 -- 172.21.15.28:0/1400147395 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef3dea7f70 con 0x55ef3e107b90 2024-04-23T14:55:40.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.835+0000 7fe3ada00640 1 --2- 172.21.15.28:0/1400147395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef3e112260 0x55ef3e10fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:55:40.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.835+0000 7fe3a7e00640 1 --2- 172.21.15.28:0/1400147395 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3e107b90 0x55ef3e10f7b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x55ef3e104720 tx=0x7fe394013b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c44a1293a4ee32f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:40.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.835+0000 7fe38ea00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:55:40.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.835+0000 7fe38ea00640 1 -- 172.21.15.28:0/1400147395 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe394011e30 con 0x55ef3e107b90 2024-04-23T14:55:40.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.836+0000 7fe38ea00640 1 -- 172.21.15.28:0/1400147395 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8542+0+0 (secure 0 0 0) 0x7fe394004030 con 0x55ef3e107b90 2024-04-23T14:55:40.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.836+0000 7fe3b2522740 1 -- 172.21.15.28:0/1400147395 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3a0015070 con 0x55ef3e107b90 2024-04-23T14:55:40.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.836+0000 7fe38ea00640 1 -- 172.21.15.28:0/1400147395 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fe394004230 con 0x55ef3e107b90 2024-04-23T14:55:40.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.836+0000 7fe3b2522740 10 client.27058 fetch_fsmap learned FSMap version 20 2024-04-23T14:55:40.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.836+0000 7fe3b2522740 1 -- 172.21.15.28:0/1400147395 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55ef3dd3f170 con 0x55ef3e107b90 2024-04-23T14:55:40.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.836+0000 7fe38ea00640 1 -- 172.21.15.28:0/1400147395 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3a0015070 con 0x55ef3e107b90 2024-04-23T14:55:40.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.837+0000 7fe38ea00640 1 -- 172.21.15.28:0/1400147395 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fe39401a070 con 0x55ef3e107b90 2024-04-23T14:55:40.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.837+0000 7fe3b2522740 10 client.27058 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:55:40.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.837+0000 7fe3b2522740 10 client.27058 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:55:40.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.837+0000 7fe3b2522740 20 client.27058 populate_metadata read hostname 'smithi028' 2024-04-23T14:55:40.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.837+0000 7fe3b2522740 1 --2- 172.21.15.28:0/1400147395 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55ef3e1227a0 0x55ef3e124bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:40.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.837+0000 7fe3b2522740 4 client.27058 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:55:40.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.837+0000 7fe3b2522740 1 -- 172.21.15.28:0/1400147395 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef3e07a890 con 0x55ef3e1227a0 2024-04-23T14:55:40.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.837+0000 7fe3ada00640 1 --2- 172.21.15.28:0/1400147395 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55ef3e1227a0 0x55ef3e124bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:40.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.838+0000 7fe3ada00640 1 --2- 172.21.15.28:0/1400147395 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55ef3e1227a0 0x55ef3e124bc0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce4f8128d216b4f7 server_cookie=ea8b2983827c0d3b in_seq=0 out_seq=0 2024-04-23T14:55:40.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.838+0000 7fe38ea00640 10 client.27058 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:55:40.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.839+0000 7fe38ea00640 1 -- 172.21.15.28:0/1400147395 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ef3e07a890 con 0x55ef3e1227a0 2024-04-23T14:55:40.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.839+0000 7fe38ea00640 10 client.27058 handle_command_reply: tid=0 2024-04-23T14:55:40.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.839+0000 7fe3b2522740 1 client.27058 shutdown 2024-04-23T14:55:40.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.839+0000 7fe3b2522740 1 -- 172.21.15.28:0/1400147395 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3e107b90 msgr2=0x55ef3e10f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:40.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.839+0000 7fe3b2522740 1 --2- 172.21.15.28:0/1400147395 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3e107b90 0x55ef3e10f7b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x55ef3e104720 tx=0x7fe394013b70 comp rx=0 tx=0).stop 2024-04-23T14:55:40.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.839+0000 7fe3b2522740 1 -- 172.21.15.28:0/1400147395 shutdown_connections 2024-04-23T14:55:40.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.839+0000 7fe3b2522740 1 --2- 172.21.15.28:0/1400147395 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55ef3e1227a0 0x55ef3e124bc0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:40.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.839+0000 7fe3b2522740 1 --2- 172.21.15.28:0/1400147395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef3e112260 0x55ef3e10fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:40.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.839+0000 7fe3b2522740 1 --2- 172.21.15.28:0/1400147395 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3e107b90 0x55ef3e10f7b0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:40.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.839+0000 7fe38ea00640 0 client.27058 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:55:40.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.839+0000 7fe3b2522740 1 --2- 172.21.15.28:0/1400147395 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef3e107270 0x55ef3e10f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:40.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.839+0000 7fe3b2522740 1 -- 172.21.15.28:0/1400147395 >> 172.21.15.28:0/1400147395 conn(0x55ef3e103590 msgr2=0x55ef3e109290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:40.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.839+0000 7fe3b2522740 1 -- 172.21.15.28:0/1400147395 shutdown_connections 2024-04-23T14:55:40.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.839+0000 7fe3b2522740 1 -- 172.21.15.28:0/1400147395 wait complete. 2024-04-23T14:55:40.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.839+0000 7fe3b2522740 20 client.27058 trim_cache size 0 max 16384 2024-04-23T14:55:40.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.864+0000 7fe3b2522740 1 Processor -- start 2024-04-23T14:55:40.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.865+0000 7fe3b2522740 1 -- start start 2024-04-23T14:55:40.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.865+0000 7fe3b2522740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3e107270 0x55ef3e11df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:40.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.865+0000 7fe3b2522740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef3e107b90 0x55ef3e10fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:40.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.865+0000 7fe3b2522740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef3e112260 0x55ef3e12a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:40.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.865+0000 7fe3b2522740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a81ade20 con 0x55ef3e112260 2024-04-23T14:55:40.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.865+0000 7fe3b2522740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a81b0680 con 0x55ef3e107b90 2024-04-23T14:55:40.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.865+0000 7fe3b2522740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a81ae470 con 0x55ef3e107270 2024-04-23T14:55:40.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.865+0000 7fe3a7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef3e107b90 0x55ef3e10fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:40.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.865+0000 7fe3a7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef3e107b90 0x55ef3e10fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:53200/0 (socket says 172.21.15.28:53200) 2024-04-23T14:55:40.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.865+0000 7fe3a7e00640 1 -- 172.21.15.28:0/1640475942 learned_addr learned my addr 172.21.15.28:0/1640475942 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:40.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.865+0000 7fe3ada00640 1 --2- 172.21.15.28:0/1640475942 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3e107270 0x55ef3e11df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:40.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.865+0000 7fe3ada00640 1 --2- 172.21.15.28:0/1640475942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef3e112260 0x55ef3e12a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:40.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.865+0000 7fe3a7e00640 1 -- 172.21.15.28:0/1640475942 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3e107270 msgr2=0x55ef3e11df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:40.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.866+0000 7fe3a7e00640 1 --2- 172.21.15.28:0/1640475942 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3e107270 0x55ef3e11df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:40.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.866+0000 7fe3a7e00640 1 -- 172.21.15.28:0/1640475942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef3e112260 msgr2=0x55ef3e12a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:40.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.866+0000 7fe3a7e00640 1 --2- 172.21.15.28:0/1640475942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef3e112260 0x55ef3e12a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:40.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.866+0000 7fe3a7e00640 1 -- 172.21.15.28:0/1640475942 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef3e11e480 con 0x55ef3e107b90 2024-04-23T14:55:40.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.866+0000 7fe3a7e00640 1 --2- 172.21.15.28:0/1640475942 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef3e107b90 0x55ef3e10fb10 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fe3940093d0 tx=0x7fe394009680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5021e4ec01f24903 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:40.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.866+0000 7fe399600640 1 -- 172.21.15.28:0/1640475942 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe394031070 con 0x55ef3e107b90 2024-04-23T14:55:40.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.866+0000 7fe399600640 1 -- 172.21.15.28:0/1640475942 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe394004080 con 0x55ef3e107b90 2024-04-23T14:55:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.866+0000 7fe399600640 1 -- 172.21.15.28:0/1640475942 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe394003d30 con 0x55ef3e107b90 2024-04-23T14:55:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.867+0000 7fe3b2522740 1 -- 172.21.15.28:0/1640475942 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef3e107b90 msgr2=0x55ef3e10fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.867+0000 7fe3b2522740 1 --2- 172.21.15.28:0/1640475942 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef3e107b90 0x55ef3e10fb10 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fe3940093d0 tx=0x7fe394009680 comp rx=0 tx=0).stop 2024-04-23T14:55:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.867+0000 7fe3b2522740 1 -- 172.21.15.28:0/1640475942 shutdown_connections 2024-04-23T14:55:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.867+0000 7fe3b2522740 1 --2- 172.21.15.28:0/1640475942 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef3e107b90 0x55ef3e10fb10 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.867+0000 7fe3b2522740 1 --2- 172.21.15.28:0/1640475942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef3e112260 0x55ef3e12a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.867+0000 7fe3b2522740 1 --2- 172.21.15.28:0/1640475942 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3e107270 0x55ef3e11df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.867+0000 7fe3b2522740 1 -- 172.21.15.28:0/1640475942 >> 172.21.15.28:0/1640475942 conn(0x55ef3e103590 msgr2=0x55ef3e1026f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.867+0000 7fe3b2522740 1 -- 172.21.15.28:0/1640475942 shutdown_connections 2024-04-23T14:55:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.867+0000 7fe3b2522740 1 -- 172.21.15.28:0/1640475942 wait complete. 2024-04-23T14:55:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.868+0000 7fe3b2522740 1 -- start start 2024-04-23T14:55:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.868+0000 7fe3b2522740 1 Processor -- start 2024-04-23T14:55:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.868+0000 7fe3b2522740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef3e107270 0x55ef3e116020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.868+0000 7fe3b2522740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef3e107b90 0x55ef3e116560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.868+0000 7fe3b2522740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3e112260 0x55ef3e116aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.868+0000 7fe3b2522740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a81aeb40 con 0x55ef3e107b90 2024-04-23T14:55:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.868+0000 7fe3b2522740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a81affb0 con 0x55ef3e107270 2024-04-23T14:55:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.868+0000 7fe3b2522740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a81af210 con 0x55ef3e112260 2024-04-23T14:55:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.868+0000 7fe3b2522740 10 client.0 _ll_register_callbacks cb 0x7fe394009790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:55:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.869+0000 7fe3ada00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef3e107270 0x55ef3e116020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.869+0000 7fe3ada00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef3e107270 0x55ef3e116020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:53210/0 (socket says 172.21.15.28:53210) 2024-04-23T14:55:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.869+0000 7fe3ada00640 1 -- 172.21.15.28:0/2425875127 learned_addr learned my addr 172.21.15.28:0/2425875127 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:40.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.869+0000 7fe3ada00640 1 --2- 172.21.15.28:0/2425875127 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3e112260 0x55ef3e116aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:40.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.869+0000 7fe3a7e00640 1 --2- 172.21.15.28:0/2425875127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef3e107b90 0x55ef3e116560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:40.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.869+0000 7fe3ada00640 1 -- 172.21.15.28:0/2425875127 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3e112260 msgr2=0x55ef3e116aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:40.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.869+0000 7fe3ada00640 1 --2- 172.21.15.28:0/2425875127 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3e112260 0x55ef3e116aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:40.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.869+0000 7fe3ada00640 1 -- 172.21.15.28:0/2425875127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef3e107b90 msgr2=0x55ef3e116560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:40.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.869+0000 7fe3ada00640 1 --2- 172.21.15.28:0/2425875127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef3e107b90 0x55ef3e116560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:40.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.869+0000 7fe3ada00640 1 -- 172.21.15.28:0/2425875127 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef3e187f00 con 0x55ef3e107270 2024-04-23T14:55:40.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.869+0000 7fe3ada00640 1 --2- 172.21.15.28:0/2425875127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef3e107270 0x55ef3e116020 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0009960 tx=0x7fe3a0013040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b53b059216ec194a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:40.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.869+0000 7fe38ea00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:55:40.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.870+0000 7fe38ea00640 1 -- 172.21.15.28:0/2425875127 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a0004460 con 0x55ef3e107270 2024-04-23T14:55:40.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.870+0000 7fe38ea00640 1 -- 172.21.15.28:0/2425875127 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8542+0+0 (secure 0 0 0) 0x7fe3a001a610 con 0x55ef3e107270 2024-04-23T14:55:40.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.870+0000 7fe3b2522740 1 -- 172.21.15.28:0/2425875127 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe394004080 con 0x55ef3e107270 2024-04-23T14:55:40.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.870+0000 7fe38ea00640 1 -- 172.21.15.28:0/2425875127 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fe3a0003e70 con 0x55ef3e107270 2024-04-23T14:55:40.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.870+0000 7fe3b2522740 10 client.27080 fetch_fsmap learned FSMap version 20 2024-04-23T14:55:40.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.870+0000 7fe3b2522740 1 -- 172.21.15.28:0/2425875127 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55ef3e1891e0 con 0x55ef3e107270 2024-04-23T14:55:40.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.870+0000 7fe38ea00640 1 -- 172.21.15.28:0/2425875127 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe394004080 con 0x55ef3e107270 2024-04-23T14:55:40.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.870+0000 7fe38ea00640 1 -- 172.21.15.28:0/2425875127 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fe3a002c460 con 0x55ef3e107270 2024-04-23T14:55:40.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.870+0000 7fe3b2522740 10 client.27080 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:55:40.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.870+0000 7fe3b2522740 10 client.27080 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:55:40.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.870+0000 7fe3b2522740 20 client.27080 populate_metadata read hostname 'smithi028' 2024-04-23T14:55:40.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.870+0000 7fe3b2522740 1 --2- 172.21.15.28:0/2425875127 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55ef3e189630 0x55ef3e18ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:40.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.871+0000 7fe3b2522740 4 client.27080 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-23T14:55:40.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.871+0000 7fe3b2522740 1 -- 172.21.15.28:0/2425875127 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ef3e074700 con 0x55ef3e189630 2024-04-23T14:55:40.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.871+0000 7fe3a7e00640 1 --2- 172.21.15.28:0/2425875127 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55ef3e189630 0x55ef3e18ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:40.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.871+0000 7fe3a7e00640 1 --2- 172.21.15.28:0/2425875127 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55ef3e189630 0x55ef3e18ba20 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae8d3a8cb9cb9259 server_cookie=698a0e27a728026b in_seq=0 out_seq=0 2024-04-23T14:55:40.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.871+0000 7fe38ea00640 10 client.27080 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:55:40.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.873+0000 7fe38ea00640 1 -- 172.21.15.28:0/2425875127 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ef3e074700 con 0x55ef3e189630 2024-04-23T14:55:40.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.873+0000 7fe38ea00640 10 client.27080 handle_command_reply: tid=0 2024-04-23T14:55:40.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.873+0000 7fe3b2522740 1 client.27080 shutdown 2024-04-23T14:55:40.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.873+0000 7fe3b2522740 1 -- 172.21.15.28:0/2425875127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef3e107270 msgr2=0x55ef3e116020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:40.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.873+0000 7fe3b2522740 1 --2- 172.21.15.28:0/2425875127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef3e107270 0x55ef3e116020 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0009960 tx=0x7fe3a0013040 comp rx=0 tx=0).stop 2024-04-23T14:55:40.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.873+0000 7fe3b2522740 1 -- 172.21.15.28:0/2425875127 shutdown_connections 2024-04-23T14:55:40.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.873+0000 7fe3b2522740 1 --2- 172.21.15.28:0/2425875127 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55ef3e189630 0x55ef3e18ba20 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:40.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.873+0000 7fe3b2522740 1 --2- 172.21.15.28:0/2425875127 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ef3e112260 0x55ef3e116aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:40.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.873+0000 7fe3b2522740 1 --2- 172.21.15.28:0/2425875127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef3e107b90 0x55ef3e116560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:40.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.873+0000 7fe3b2522740 1 --2- 172.21.15.28:0/2425875127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef3e107270 0x55ef3e116020 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:40.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.873+0000 7fe38ea00640 0 client.27080 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:55:40.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.873+0000 7fe3b2522740 1 -- 172.21.15.28:0/2425875127 >> 172.21.15.28:0/2425875127 conn(0x55ef3e103590 msgr2=0x55ef3e1089c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:40.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.873+0000 7fe3b2522740 1 -- 172.21.15.28:0/2425875127 shutdown_connections 2024-04-23T14:55:40.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.873+0000 7fe3b2522740 1 -- 172.21.15.28:0/2425875127 wait complete. 2024-04-23T14:55:40.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.873+0000 7fe3b2522740 20 client.27080 trim_cache size 0 max 16384 2024-04-23T14:55:40.877 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:55:40.878 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-23T14:55:40.878 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "d6c38a2c-fb5e-45aa-a067-a0ca4e233bdd", 2024-04-23T14:55:40.878 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-23T14:55:40.878 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:55:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.874+0000 7fe3ae400640 1 -- 172.21.15.28:0/590816610 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe3900a3d60 msgr2=0x7fe3900a6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.874+0000 7fe3ae400640 1 --2- 172.21.15.28:0/590816610 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe3900a3d60 0x7fe3900a6220 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x55ef3e108830 tx=0x7fe39c0f38c0 comp rx=0 tx=0).stop 2024-04-23T14:55:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.874+0000 7fe3ae400640 1 -- 172.21.15.28:0/590816610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3a8104320 msgr2=0x7fe3a81a06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.874+0000 7fe3ae400640 1 --2- 172.21.15.28:0/590816610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3a8104320 0x7fe3a81a06c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fe39c02a8b0 tx=0x7fe39c033d10 comp rx=0 tx=0).stop 2024-04-23T14:55:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.874+0000 7fe3ae400640 1 -- 172.21.15.28:0/590816610 shutdown_connections 2024-04-23T14:55:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.874+0000 7fe3ae400640 1 --2- 172.21.15.28:0/590816610 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe3900a3d60 0x7fe3900a6220 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.874+0000 7fe3ae400640 1 --2- 172.21.15.28:0/590816610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3a819a870 0x7fe3a819ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.874+0000 7fe3ae400640 1 --2- 172.21.15.28:0/590816610 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe3a81093f0 0x7fe3a81a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.874+0000 7fe3ae400640 1 --2- 172.21.15.28:0/590816610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3a8104320 0x7fe3a81a06c0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.874+0000 7fe3ae400640 1 -- 172.21.15.28:0/590816610 >> 172.21.15.28:0/590816610 conn(0x7fe3a80ffb40 msgr2=0x7fe3a8100f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.874+0000 7fe3ae400640 1 -- 172.21.15.28:0/590816610 shutdown_connections 2024-04-23T14:55:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:40.874+0000 7fe3ae400640 1 -- 172.21.15.28:0/590816610 wait complete. 2024-04-23T14:55:40.895 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:55:40.895 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:55:40.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:40 smithi028 ceph-mon[27810]: pgmap v362: 97 pgs: 97 active+clean; 600 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 103 op/s 2024-04-23T14:55:41.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.108+0000 7f1230c00640 1 -- 172.21.15.28:0/1857382553 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f122c0743a0 msgr2=0x7f122c074780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:41.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.108+0000 7f1230c00640 1 --2- 172.21.15.28:0/1857382553 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f122c0743a0 0x7f122c074780 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f121c009900 tx=0x7f121c02a480 comp rx=0 tx=0).stop 2024-04-23T14:55:41.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.108+0000 7f1230c00640 1 -- 172.21.15.28:0/1857382553 shutdown_connections 2024-04-23T14:55:41.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.108+0000 7f1230c00640 1 --2- 172.21.15.28:0/1857382553 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f122c074cc0 0x7f122c112980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:41.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.108+0000 7f1230c00640 1 --2- 172.21.15.28:0/1857382553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f122c112ec0 0x7f122c1152b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:41.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.108+0000 7f1230c00640 1 --2- 172.21.15.28:0/1857382553 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f122c0743a0 0x7f122c074780 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:41.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.108+0000 7f1230c00640 1 -- 172.21.15.28:0/1857382553 >> 172.21.15.28:0/1857382553 conn(0x7f122c0fd9a0 msgr2=0x7f122c0ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:41.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.108+0000 7f1230c00640 1 -- 172.21.15.28:0/1857382553 shutdown_connections 2024-04-23T14:55:41.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.108+0000 7f1230c00640 1 -- 172.21.15.28:0/1857382553 wait complete. 2024-04-23T14:55:41.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.109+0000 7f1230c00640 1 Processor -- start 2024-04-23T14:55:41.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.109+0000 7f1230c00640 1 -- start start 2024-04-23T14:55:41.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.110+0000 7f1230c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f122c0743a0 0x7f122c19ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:41.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.110+0000 7f1230c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f122c074cc0 0x7f122c19f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:41.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.110+0000 7f1230c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f122c112ec0 0x7f122c1a36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:41.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.110+0000 7f1230c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f122c118900 con 0x7f122c074cc0 2024-04-23T14:55:41.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.110+0000 7f1230c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f122c118780 con 0x7f122c0743a0 2024-04-23T14:55:41.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.110+0000 7f1230c00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f122c118a80 con 0x7f122c112ec0 2024-04-23T14:55:41.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.110+0000 7f122b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f122c0743a0 0x7f122c19ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:41.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.110+0000 7f122b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f122c0743a0 0x7f122c19ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:53234/0 (socket says 172.21.15.28:53234) 2024-04-23T14:55:41.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.110+0000 7f122b400640 1 -- 172.21.15.28:0/2286895178 learned_addr learned my addr 172.21.15.28:0/2286895178 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:41.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.110+0000 7f122aa00640 1 --2- 172.21.15.28:0/2286895178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f122c074cc0 0x7f122c19f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:41.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.110+0000 7f122be00640 1 --2- 172.21.15.28:0/2286895178 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f122c112ec0 0x7f122c1a36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:41.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.111+0000 7f122b400640 1 -- 172.21.15.28:0/2286895178 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f122c112ec0 msgr2=0x7f122c1a36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:41.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.111+0000 7f122b400640 1 --2- 172.21.15.28:0/2286895178 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f122c112ec0 0x7f122c1a36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:41.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.111+0000 7f122b400640 1 -- 172.21.15.28:0/2286895178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f122c074cc0 msgr2=0x7f122c19f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:41.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.111+0000 7f122b400640 1 --2- 172.21.15.28:0/2286895178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f122c074cc0 0x7f122c19f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:41.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.111+0000 7f122b400640 1 -- 172.21.15.28:0/2286895178 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f122c1a3dd0 con 0x7f122c0743a0 2024-04-23T14:55:41.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.111+0000 7f122aa00640 1 --2- 172.21.15.28:0/2286895178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f122c074cc0 0x7f122c19f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:55:41.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.111+0000 7f122be00640 1 --2- 172.21.15.28:0/2286895178 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f122c112ec0 0x7f122c1a36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:55:41.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.111+0000 7f122b400640 1 --2- 172.21.15.28:0/2286895178 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f122c0743a0 0x7f122c19ede0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f121c004770 tx=0x7f121c003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:41.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.111+0000 7f1223e00640 1 -- 172.21.15.28:0/2286895178 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121c03f070 con 0x7f122c0743a0 2024-04-23T14:55:41.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.111+0000 7f1223e00640 1 -- 172.21.15.28:0/2286895178 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f121c031ca0 con 0x7f122c0743a0 2024-04-23T14:55:41.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.111+0000 7f1223e00640 1 -- 172.21.15.28:0/2286895178 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121c035030 con 0x7f122c0743a0 2024-04-23T14:55:41.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.111+0000 7f1230c00640 1 -- 172.21.15.28:0/2286895178 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f122c1a4060 con 0x7f122c0743a0 2024-04-23T14:55:41.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.111+0000 7f1230c00640 1 -- 172.21.15.28:0/2286895178 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f122c1ab950 con 0x7f122c0743a0 2024-04-23T14:55:41.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.115+0000 7f1234ffc740 1 Processor -- start 2024-04-23T14:55:41.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.115+0000 7f1234ffc740 1 -- start start 2024-04-23T14:55:41.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.115+0000 7f1223e00640 1 -- 172.21.15.28:0/2286895178 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f121c042070 con 0x7f122c0743a0 2024-04-23T14:55:41.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.115+0000 7f1234ffc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0cd366a30 0x55b0cd366e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:41.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.115+0000 7f1234ffc740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0cd367350 0x55b0cd3714e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:41.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.115+0000 7f1234ffc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0cd371a20 0x55b0cd373e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:41.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.115+0000 7f1234ffc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b0cd2cc1f0 con 0x55b0cd371a20 2024-04-23T14:55:41.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.115+0000 7f1234ffc740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b0cd272350 con 0x55b0cd367350 2024-04-23T14:55:41.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.115+0000 7f1234ffc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b0cd2c81b0 con 0x55b0cd366a30 2024-04-23T14:55:41.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.115+0000 7f122b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0cd367350 0x55b0cd3714e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:41.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.115+0000 7f122be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0cd366a30 0x55b0cd366e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:41.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.115+0000 7f122aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0cd371a20 0x55b0cd373e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:41.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.115+0000 7f122be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0cd366a30 0x55b0cd366e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:53212/0 (socket says 172.21.15.28:53212) 2024-04-23T14:55:41.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.115+0000 7f122b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0cd367350 0x55b0cd3714e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:53238/0 (socket says 172.21.15.28:53238) 2024-04-23T14:55:41.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.115+0000 7f122be00640 1 -- 172.21.15.28:0/2998433285 learned_addr learned my addr 172.21.15.28:0/2998433285 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:41.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.116+0000 7f122be00640 1 -- 172.21.15.28:0/2998433285 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0cd367350 msgr2=0x55b0cd3714e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:41.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.116+0000 7f122be00640 1 --2- 172.21.15.28:0/2998433285 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0cd367350 0x55b0cd3714e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:41.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.116+0000 7f122be00640 1 -- 172.21.15.28:0/2998433285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0cd371a20 msgr2=0x55b0cd373e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:41.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.116+0000 7f122be00640 1 --2- 172.21.15.28:0/2998433285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0cd371a20 0x55b0cd373e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:41.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.116+0000 7f122be00640 1 -- 172.21.15.28:0/2998433285 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0cd2c6f50 con 0x55b0cd366a30 2024-04-23T14:55:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.116+0000 7f122b400640 1 --2- 172.21.15.28:0/2998433285 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0cd367350 0x55b0cd3714e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:55:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.116+0000 7f122be00640 1 --2- 172.21.15.28:0/2998433285 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0cd366a30 0x55b0cd366e10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f12240106c0 tx=0x7f1224010b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41b607d324a085fe server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.116+0000 7f1217e00640 1 -- 172.21.15.28:0/2998433285 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f122400e8d0 con 0x55b0cd366a30 2024-04-23T14:55:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.116+0000 7f1217e00640 1 -- 172.21.15.28:0/2998433285 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1224012450 con 0x55b0cd366a30 2024-04-23T14:55:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.117+0000 7f1217e00640 1 -- 172.21.15.28:0/2998433285 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f122401c630 con 0x55b0cd366a30 2024-04-23T14:55:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.117+0000 7f1223e00640 1 --2- 172.21.15.28:0/2286895178 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f12100a3d20 0x7f12100a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.117+0000 7f1223e00640 1 -- 172.21.15.28:0/2286895178 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8542+0+0 (secure 0 0 0) 0x7f121c0f0f60 con 0x7f122c0743a0 2024-04-23T14:55:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.117+0000 7f122b400640 1 --2- 172.21.15.28:0/2286895178 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f12100a3d20 0x7f12100a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.117+0000 7f1234ffc740 1 -- 172.21.15.28:0/2998433285 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0cd366a30 msgr2=0x55b0cd366e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.117+0000 7f1234ffc740 1 --2- 172.21.15.28:0/2998433285 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0cd366a30 0x55b0cd366e10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f12240106c0 tx=0x7f1224010b80 comp rx=0 tx=0).stop 2024-04-23T14:55:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.117+0000 7f1234ffc740 1 -- 172.21.15.28:0/2998433285 shutdown_connections 2024-04-23T14:55:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.117+0000 7f1234ffc740 1 --2- 172.21.15.28:0/2998433285 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0cd367350 0x55b0cd3714e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.117+0000 7f1234ffc740 1 --2- 172.21.15.28:0/2998433285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0cd371a20 0x55b0cd373e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:41.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.117+0000 7f1234ffc740 1 --2- 172.21.15.28:0/2998433285 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0cd366a30 0x55b0cd366e10 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:41.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.117+0000 7f1234ffc740 1 -- 172.21.15.28:0/2998433285 >> 172.21.15.28:0/2998433285 conn(0x55b0cd362d50 msgr2=0x55b0cd363130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:41.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.118+0000 7f1234ffc740 1 -- 172.21.15.28:0/2998433285 shutdown_connections 2024-04-23T14:55:41.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.118+0000 7f1234ffc740 1 -- 172.21.15.28:0/2998433285 wait complete. 2024-04-23T14:55:41.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.118+0000 7f122b400640 1 --2- 172.21.15.28:0/2286895178 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f12100a3d20 0x7f12100a61e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x55b0cd367ff0 tx=0x7f121c0f4a10 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:41.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.118+0000 7f1234ffc740 1 -- start start 2024-04-23T14:55:41.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.119+0000 7f1234ffc740 1 Processor -- start 2024-04-23T14:55:41.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.119+0000 7f1234ffc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0cd366a30 0x55b0cd36e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:41.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.119+0000 7f1234ffc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0cd367350 0x55b0cd36ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:41.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.119+0000 7f1234ffc740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0cd371a20 0x55b0cd36f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:41.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.119+0000 7f1234ffc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b0cd374810 con 0x55b0cd366a30 2024-04-23T14:55:41.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.119+0000 7f1234ffc740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b0cd285690 con 0x55b0cd371a20 2024-04-23T14:55:41.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.119+0000 7f1234ffc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b0cd374990 con 0x55b0cd367350 2024-04-23T14:55:41.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.119+0000 7f1234ffc740 10 client.0 _ll_register_callbacks cb 0x55b0cd2c1650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:55:41.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.119+0000 7f122be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0cd366a30 0x55b0cd36e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:41.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.119+0000 7f122aa00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0cd367350 0x55b0cd36ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:41.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.119+0000 7f122be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0cd366a30 0x55b0cd36e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39884/0 (socket says 172.21.15.28:39884) 2024-04-23T14:55:41.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.119+0000 7f122aa00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0cd367350 0x55b0cd36ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:53228/0 (socket says 172.21.15.28:53228) 2024-04-23T14:55:41.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.119+0000 7f122be00640 1 -- 172.21.15.28:0/1735711372 learned_addr learned my addr 172.21.15.28:0/1735711372 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:41.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.120+0000 7f122be00640 1 --2- 172.21.15.28:0/1735711372 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0cd371a20 0x55b0cd36f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:41.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.120+0000 7f122aa00640 1 -- 172.21.15.28:0/1735711372 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0cd371a20 msgr2=0x55b0cd36f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:41.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.120+0000 7f122aa00640 1 --2- 172.21.15.28:0/1735711372 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0cd371a20 0x55b0cd36f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:41.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.120+0000 7f122aa00640 1 -- 172.21.15.28:0/1735711372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0cd366a30 msgr2=0x55b0cd36e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:41.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.120+0000 7f122aa00640 1 --2- 172.21.15.28:0/1735711372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0cd366a30 0x55b0cd36e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:41.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.120+0000 7f122aa00640 1 -- 172.21.15.28:0/1735711372 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0cd284920 con 0x55b0cd367350 2024-04-23T14:55:41.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.120+0000 7f122aa00640 1 --2- 172.21.15.28:0/1735711372 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0cd367350 0x55b0cd36ef20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f12180126d0 tx=0x7f1218012b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb2237cb3a746288 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:41.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.120+0000 7f1214c00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:55:41.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.120+0000 7f1214c00640 1 -- 172.21.15.28:0/1735711372 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1218011c70 con 0x55b0cd367350 2024-04-23T14:55:41.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.120+0000 7f1214c00640 1 -- 172.21.15.28:0/1735711372 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8542+0+0 (secure 0 0 0) 0x7f1218011e10 con 0x55b0cd367350 2024-04-23T14:55:41.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.120+0000 7f1234ffc740 1 -- 172.21.15.28:0/1735711372 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1224012450 con 0x55b0cd367350 2024-04-23T14:55:41.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.121+0000 7f1214c00640 1 -- 172.21.15.28:0/1735711372 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f1218005020 con 0x55b0cd367350 2024-04-23T14:55:41.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.121+0000 7f1234ffc740 10 client.27076 fetch_fsmap learned FSMap version 20 2024-04-23T14:55:41.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.121+0000 7f1234ffc740 1 -- 172.21.15.28:0/1735711372 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55b0cd0a87d0 con 0x55b0cd367350 2024-04-23T14:55:41.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.121+0000 7f1214c00640 1 -- 172.21.15.28:0/1735711372 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1224012450 con 0x55b0cd367350 2024-04-23T14:55:41.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.121+0000 7f1214c00640 1 -- 172.21.15.28:0/1735711372 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f121801c470 con 0x55b0cd367350 2024-04-23T14:55:41.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.122+0000 7f1234ffc740 10 client.27076 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:55:41.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.122+0000 7f1234ffc740 10 client.27076 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:55:41.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.122+0000 7f1234ffc740 20 client.27076 populate_metadata read hostname 'smithi028' 2024-04-23T14:55:41.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.122+0000 7f1234ffc740 1 --2- 172.21.15.28:0/1735711372 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55b0cd381e60 0x55b0cd384290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:41.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.122+0000 7f1234ffc740 4 client.27076 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:55:41.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.122+0000 7f1234ffc740 1 -- 172.21.15.28:0/1735711372 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b0cd2d9c10 con 0x55b0cd381e60 2024-04-23T14:55:41.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.122+0000 7f122be00640 1 --2- 172.21.15.28:0/1735711372 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55b0cd381e60 0x55b0cd384290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:41.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.122+0000 7f122be00640 1 --2- 172.21.15.28:0/1735711372 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55b0cd381e60 0x55b0cd384290 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d14c8bc4abc4489 server_cookie=e74f3617659768b2 in_seq=0 out_seq=0 2024-04-23T14:55:41.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.122+0000 7f1214c00640 10 client.27076 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:55:41.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.123+0000 7f1214c00640 1 -- 172.21.15.28:0/1735711372 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b0cd2d9c10 con 0x55b0cd381e60 2024-04-23T14:55:41.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.123+0000 7f1214c00640 10 client.27076 handle_command_reply: tid=0 2024-04-23T14:55:41.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.123+0000 7f1234ffc740 1 client.27076 shutdown 2024-04-23T14:55:41.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.124+0000 7f1234ffc740 1 -- 172.21.15.28:0/1735711372 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0cd367350 msgr2=0x55b0cd36ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:41.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.124+0000 7f1234ffc740 1 --2- 172.21.15.28:0/1735711372 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0cd367350 0x55b0cd36ef20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f12180126d0 tx=0x7f1218012b90 comp rx=0 tx=0).stop 2024-04-23T14:55:41.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.124+0000 7f1234ffc740 1 -- 172.21.15.28:0/1735711372 shutdown_connections 2024-04-23T14:55:41.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.124+0000 7f1234ffc740 1 --2- 172.21.15.28:0/1735711372 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55b0cd381e60 0x55b0cd384290 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:41.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.124+0000 7f1234ffc740 1 --2- 172.21.15.28:0/1735711372 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0cd371a20 0x55b0cd36f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:41.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.124+0000 7f1234ffc740 1 --2- 172.21.15.28:0/1735711372 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0cd367350 0x55b0cd36ef20 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:41.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.124+0000 7f1214c00640 0 client.27076 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:55:41.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.124+0000 7f1234ffc740 1 --2- 172.21.15.28:0/1735711372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0cd366a30 0x55b0cd36e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:41.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.124+0000 7f1234ffc740 1 -- 172.21.15.28:0/1735711372 >> 172.21.15.28:0/1735711372 conn(0x55b0cd362d50 msgr2=0x55b0cd368b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:41.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.124+0000 7f1234ffc740 1 -- 172.21.15.28:0/1735711372 shutdown_connections 2024-04-23T14:55:41.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.124+0000 7f1234ffc740 1 -- 172.21.15.28:0/1735711372 wait complete. 2024-04-23T14:55:41.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.124+0000 7f1234ffc740 20 client.27076 trim_cache size 0 max 16384 2024-04-23T14:55:41.129 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:40 smithi038 ceph-mon[32348]: pgmap v362: 97 pgs: 97 active+clean; 600 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 103 op/s 2024-04-23T14:55:41.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.143+0000 7f1234ffc740 1 Processor -- start 2024-04-23T14:55:41.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.144+0000 7f1234ffc740 1 -- start start 2024-04-23T14:55:41.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.144+0000 7f1234ffc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0cd366a30 0x55b0cd369280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:41.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.144+0000 7f1234ffc740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0cd367350 0x55b0cd36f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:41.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.144+0000 7f1234ffc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0cd371a20 0x55b0cd389ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:41.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.144+0000 7f1234ffc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f122c1b2260 con 0x55b0cd366a30 2024-04-23T14:55:41.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.144+0000 7f1234ffc740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f122c1b4ac0 con 0x55b0cd367350 2024-04-23T14:55:41.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.144+0000 7f1234ffc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f122c1b28b0 con 0x55b0cd371a20 2024-04-23T14:55:41.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.144+0000 7f122be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0cd366a30 0x55b0cd369280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:41.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.144+0000 7f122aa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0cd367350 0x55b0cd36f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:41.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.144+0000 7f122be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0cd366a30 0x55b0cd369280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39894/0 (socket says 172.21.15.28:39894) 2024-04-23T14:55:41.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.144+0000 7f122be00640 1 -- 172.21.15.28:0/1930950011 learned_addr learned my addr 172.21.15.28:0/1930950011 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:41.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.144+0000 7f122be00640 1 --2- 172.21.15.28:0/1930950011 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0cd371a20 0x55b0cd389ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:41.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.145+0000 7f122be00640 1 -- 172.21.15.28:0/1930950011 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0cd371a20 msgr2=0x55b0cd389ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:41.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.145+0000 7f122be00640 1 --2- 172.21.15.28:0/1930950011 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0cd371a20 0x55b0cd389ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:41.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.145+0000 7f122be00640 1 -- 172.21.15.28:0/1930950011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0cd367350 msgr2=0x55b0cd36f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:41.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.145+0000 7f122be00640 1 --2- 172.21.15.28:0/1930950011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0cd367350 0x55b0cd36f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:41.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.145+0000 7f122be00640 1 -- 172.21.15.28:0/1930950011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0ccfde3f0 con 0x55b0cd366a30 2024-04-23T14:55:41.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.145+0000 7f122aa00640 1 --2- 172.21.15.28:0/1930950011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0cd367350 0x55b0cd36f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:55:41.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.145+0000 7f122be00640 1 --2- 172.21.15.28:0/1930950011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0cd366a30 0x55b0cd369280 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f12240106c0 tx=0x7f1224012e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f0cb9e57cce8e57 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:41.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.146+0000 7f1217400640 1 -- 172.21.15.28:0/1930950011 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f122402fe60 con 0x55b0cd366a30 2024-04-23T14:55:41.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.146+0000 7f1217400640 1 -- 172.21.15.28:0/1930950011 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1224012930 con 0x55b0cd366a30 2024-04-23T14:55:41.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.146+0000 7f1217400640 1 -- 172.21.15.28:0/1930950011 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f122400eea0 con 0x55b0cd366a30 2024-04-23T14:55:41.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.146+0000 7f1234ffc740 1 -- 172.21.15.28:0/1930950011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0cd366a30 msgr2=0x55b0cd369280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:41.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.146+0000 7f1234ffc740 1 --2- 172.21.15.28:0/1930950011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0cd366a30 0x55b0cd369280 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f12240106c0 tx=0x7f1224012e20 comp rx=0 tx=0).stop 2024-04-23T14:55:41.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.146+0000 7f1234ffc740 1 -- 172.21.15.28:0/1930950011 shutdown_connections 2024-04-23T14:55:41.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.146+0000 7f1234ffc740 1 --2- 172.21.15.28:0/1930950011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0cd367350 0x55b0cd36f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:41.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.146+0000 7f1234ffc740 1 --2- 172.21.15.28:0/1930950011 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0cd371a20 0x55b0cd389ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:41.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.146+0000 7f1234ffc740 1 --2- 172.21.15.28:0/1930950011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0cd366a30 0x55b0cd369280 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:41.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.146+0000 7f1234ffc740 1 -- 172.21.15.28:0/1930950011 >> 172.21.15.28:0/1930950011 conn(0x55b0cd362d50 msgr2=0x55b0cd361740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:41.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.147+0000 7f1234ffc740 1 -- 172.21.15.28:0/1930950011 shutdown_connections 2024-04-23T14:55:41.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.147+0000 7f1234ffc740 1 -- 172.21.15.28:0/1930950011 wait complete. 2024-04-23T14:55:41.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.147+0000 7f1234ffc740 1 -- start start 2024-04-23T14:55:41.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.147+0000 7f1234ffc740 1 Processor -- start 2024-04-23T14:55:41.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.148+0000 7f1234ffc740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0cd366a30 0x55b0cd375c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:41.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.148+0000 7f1234ffc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0cd367350 0x55b0cd3761d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:41.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.148+0000 7f1234ffc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0cd371a20 0x55b0cd3e7410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:41.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.148+0000 7f1234ffc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f122c1b2f80 con 0x55b0cd371a20 2024-04-23T14:55:41.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.148+0000 7f1234ffc740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f122c1b43f0 con 0x55b0cd366a30 2024-04-23T14:55:41.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.148+0000 7f1234ffc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f122c1b3650 con 0x55b0cd367350 2024-04-23T14:55:41.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.148+0000 7f1234ffc740 10 client.0 _ll_register_callbacks cb 0x7f1218009860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:55:41.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.148+0000 7f122be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0cd366a30 0x55b0cd375c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:41.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.148+0000 7f122aa00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0cd367350 0x55b0cd3761d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:41.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.148+0000 7f122be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0cd366a30 0x55b0cd375c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:53258/0 (socket says 172.21.15.28:53258) 2024-04-23T14:55:41.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.148+0000 7f122aa00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0cd367350 0x55b0cd3761d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:53250/0 (socket says 172.21.15.28:53250) 2024-04-23T14:55:41.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.148+0000 7f122be00640 1 -- 172.21.15.28:0/1391794321 learned_addr learned my addr 172.21.15.28:0/1391794321 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:55:41.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.148+0000 7f122be00640 1 --2- 172.21.15.28:0/1391794321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0cd371a20 0x55b0cd3e7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:41.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.148+0000 7f122aa00640 1 -- 172.21.15.28:0/1391794321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0cd366a30 msgr2=0x55b0cd375c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:41.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.148+0000 7f122aa00640 1 --2- 172.21.15.28:0/1391794321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0cd366a30 0x55b0cd375c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:41.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.148+0000 7f122aa00640 1 -- 172.21.15.28:0/1391794321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0cd371a20 msgr2=0x55b0cd3e7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:55:41.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.148+0000 7f122aa00640 1 --2- 172.21.15.28:0/1391794321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0cd371a20 0x55b0cd3e7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.148+0000 7f122aa00640 1 -- 172.21.15.28:0/1391794321 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0ccf9e0f0 con 0x55b0cd367350 2024-04-23T14:55:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.148+0000 7f122be00640 1 --2- 172.21.15.28:0/1391794321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0cd371a20 0x55b0cd3e7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:55:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.149+0000 7f122aa00640 1 --2- 172.21.15.28:0/1391794321 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0cd367350 0x55b0cd3761d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f121801c8e0 tx=0x7f121801c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=573b98e6b679c7da server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:55:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.149+0000 7f1214c00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:55:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.149+0000 7f1214c00640 1 -- 172.21.15.28:0/1391794321 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1218031020 con 0x55b0cd367350 2024-04-23T14:55:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.149+0000 7f1214c00640 1 -- 172.21.15.28:0/1391794321 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8542+0+0 (secure 0 0 0) 0x7f1218011c70 con 0x55b0cd367350 2024-04-23T14:55:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.149+0000 7f1234ffc740 1 -- 172.21.15.28:0/1391794321 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1224012930 con 0x55b0cd367350 2024-04-23T14:55:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.149+0000 7f1214c00640 1 -- 172.21.15.28:0/1391794321 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f121803c050 con 0x55b0cd367350 2024-04-23T14:55:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.149+0000 7f1234ffc740 10 client.27082 fetch_fsmap learned FSMap version 20 2024-04-23T14:55:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.149+0000 7f1234ffc740 1 -- 172.21.15.28:0/1391794321 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55b0cd291810 con 0x55b0cd367350 2024-04-23T14:55:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.150+0000 7f1214c00640 1 -- 172.21.15.28:0/1391794321 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1224012930 con 0x55b0cd367350 2024-04-23T14:55:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.150+0000 7f1214c00640 1 -- 172.21.15.28:0/1391794321 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f1218004820 con 0x55b0cd367350 2024-04-23T14:55:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.150+0000 7f1234ffc740 10 client.27082 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:55:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.150+0000 7f1234ffc740 10 client.27082 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:55:41.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.150+0000 7f1234ffc740 20 client.27082 populate_metadata read hostname 'smithi028' 2024-04-23T14:55:41.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.150+0000 7f1234ffc740 1 --2- 172.21.15.28:0/1391794321 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55b0cd3e8ff0 0x55b0cd3eb410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:55:41.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.150+0000 7f1234ffc740 4 client.27082 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:55:41.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.150+0000 7f1234ffc740 1 -- 172.21.15.28:0/1391794321 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b0cd2d3a80 con 0x55b0cd3e8ff0 2024-04-23T14:55:41.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.150+0000 7f122be00640 1 --2- 172.21.15.28:0/1391794321 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55b0cd3e8ff0 0x55b0cd3eb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:55:41.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.151+0000 7f122be00640 1 --2- 172.21.15.28:0/1391794321 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55b0cd3e8ff0 0x55b0cd3eb410 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a26f8a0007de97cf server_cookie=8802aec1b035df7d in_seq=0 out_seq=0 2024-04-23T14:55:41.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.151+0000 7f1214c00640 10 client.27082 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:55:41.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.153+0000 7f1214c00640 1 -- 172.21.15.28:0/1391794321 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55b0cd2d3a80 con 0x55b0cd3e8ff0 2024-04-23T14:55:41.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.153+0000 7f1214c00640 10 client.27082 handle_command_reply: tid=0 2024-04-23T14:55:41.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.153+0000 7f1234ffc740 1 client.27082 shutdown 2024-04-23T14:55:41.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.153+0000 7f1234ffc740 1 -- 172.21.15.28:0/1391794321 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0cd367350 msgr2=0x55b0cd3761d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:41.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.153+0000 7f1234ffc740 1 --2- 172.21.15.28:0/1391794321 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0cd367350 0x55b0cd3761d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f121801c8e0 tx=0x7f121801c6a0 comp rx=0 tx=0).stop 2024-04-23T14:55:41.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.153+0000 7f1234ffc740 1 -- 172.21.15.28:0/1391794321 shutdown_connections 2024-04-23T14:55:41.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.153+0000 7f1234ffc740 1 --2- 172.21.15.28:0/1391794321 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55b0cd3e8ff0 0x55b0cd3eb410 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:41.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.153+0000 7f1234ffc740 1 --2- 172.21.15.28:0/1391794321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0cd371a20 0x55b0cd3e7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:41.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.153+0000 7f1234ffc740 1 --2- 172.21.15.28:0/1391794321 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b0cd367350 0x55b0cd3761d0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:41.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.153+0000 7f1234ffc740 1 --2- 172.21.15.28:0/1391794321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b0cd366a30 0x55b0cd375c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:41.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.153+0000 7f1234ffc740 1 -- 172.21.15.28:0/1391794321 >> 172.21.15.28:0/1391794321 conn(0x55b0cd362d50 msgr2=0x55b0cd36f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:41.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.153+0000 7f1214c00640 0 client.27082 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:55:41.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.154+0000 7f1234ffc740 1 -- 172.21.15.28:0/1391794321 shutdown_connections 2024-04-23T14:55:41.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.154+0000 7f1234ffc740 1 -- 172.21.15.28:0/1391794321 wait complete. 2024-04-23T14:55:41.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.154+0000 7f1234ffc740 20 client.27082 trim_cache size 0 max 16384 2024-04-23T14:55:41.157 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:55:41.157 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-23T14:55:41.157 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-23T14:55:41.157 INFO:teuthology.orchestra.run.smithi028.stdout: "d6c38a2c-fb5e-45aa-a067-a0ca4e233bdd": { 2024-04-23T14:55:41.157 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-23T14:55:41.158 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "d6c38a2c-fb5e-45aa-a067-a0ca4e233bdd", 2024-04-23T14:55:41.158 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-23T14:55:41.158 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:55:41.158 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:55:41.158 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:55:41.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.155+0000 7f1230c00640 1 -- 172.21.15.28:0/2286895178 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f12100a3d20 msgr2=0x7f12100a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:41.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.155+0000 7f1230c00640 1 --2- 172.21.15.28:0/2286895178 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f12100a3d20 0x7f12100a61e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x55b0cd367ff0 tx=0x7f121c0f4a10 comp rx=0 tx=0).stop 2024-04-23T14:55:41.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.155+0000 7f1230c00640 1 -- 172.21.15.28:0/2286895178 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f122c0743a0 msgr2=0x7f122c19ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:55:41.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.155+0000 7f1230c00640 1 --2- 172.21.15.28:0/2286895178 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f122c0743a0 0x7f122c19ede0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f121c004770 tx=0x7f121c003c10 comp rx=0 tx=0).stop 2024-04-23T14:55:41.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.155+0000 7f1230c00640 1 -- 172.21.15.28:0/2286895178 shutdown_connections 2024-04-23T14:55:41.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.155+0000 7f1230c00640 1 --2- 172.21.15.28:0/2286895178 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f12100a3d20 0x7f12100a61e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:41.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.155+0000 7f1230c00640 1 --2- 172.21.15.28:0/2286895178 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f122c112ec0 0x7f122c1a36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:41.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.155+0000 7f1230c00640 1 --2- 172.21.15.28:0/2286895178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f122c074cc0 0x7f122c19f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:41.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.155+0000 7f1230c00640 1 --2- 172.21.15.28:0/2286895178 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f122c0743a0 0x7f122c19ede0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:55:41.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.155+0000 7f1230c00640 1 -- 172.21.15.28:0/2286895178 >> 172.21.15.28:0/2286895178 conn(0x7f122c0fd9a0 msgr2=0x7f122c0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:55:41.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.155+0000 7f1230c00640 1 -- 172.21.15.28:0/2286895178 shutdown_connections 2024-04-23T14:55:41.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:55:41.155+0000 7f1230c00640 1 -- 172.21.15.28:0/2286895178 wait complete. 2024-04-23T14:55:41.175 INFO:tasks.cephfs.filesystem:scrub status for tag:d6c38a2c-fb5e-45aa-a067-a0ca4e233bdd - {'path': '/', 'tag': 'd6c38a2c-fb5e-45aa-a067-a0ca4e233bdd', 'options': 'recursive,force'} 2024-04-23T14:55:41.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:40 smithi105 ceph-mon[32555]: pgmap v362: 97 pgs: 97 active+clean; 600 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 103 op/s 2024-04-23T14:55:41.535 INFO:tasks.workunit.client.0.smithi028.stdout: 1 993375 8.64 MB/sec execute 477 sec latency 22.228 ms 2024-04-23T14:55:42.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:41 smithi038 ceph-mon[32348]: scrub queued for path: / 2024-04-23T14:55:42.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:41 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:55:42.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:41 smithi038 ceph-mon[32348]: scrub summary: active paths [/] 2024-04-23T14:55:42.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:41 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:55:42.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:41 smithi028 ceph-mon[27810]: scrub queued for path: / 2024-04-23T14:55:42.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:41 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:55:42.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:41 smithi028 ceph-mon[27810]: scrub summary: active paths [/] 2024-04-23T14:55:42.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:41 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:55:42.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:41 smithi105 ceph-mon[32555]: scrub queued for path: / 2024-04-23T14:55:42.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:41 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:55:42.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:41 smithi105 ceph-mon[32555]: scrub summary: active paths [/] 2024-04-23T14:55:42.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:41 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:55:42.535 INFO:tasks.workunit.client.0.smithi028.stdout: 1 995119 8.65 MB/sec execute 478 sec latency 14.545 ms 2024-04-23T14:55:43.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:42 smithi038 ceph-mon[32348]: pgmap v363: 97 pgs: 97 active+clean; 600 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 103 op/s 2024-04-23T14:55:43.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:42 smithi028 ceph-mon[27810]: pgmap v363: 97 pgs: 97 active+clean; 600 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 103 op/s 2024-04-23T14:55:43.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:42 smithi105 ceph-mon[32555]: pgmap v363: 97 pgs: 97 active+clean; 600 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 103 op/s 2024-04-23T14:55:43.535 INFO:tasks.workunit.client.0.smithi028.stdout: 1 996442 8.64 MB/sec execute 479 sec latency 28.668 ms 2024-04-23T14:55:44.536 INFO:tasks.workunit.client.0.smithi028.stdout: 1 998110 8.65 MB/sec execute 480 sec latency 60.903 ms 2024-04-23T14:55:45.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:44 smithi038 ceph-mon[32348]: pgmap v364: 97 pgs: 97 active+clean; 557 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.6 MiB/s wr, 171 op/s 2024-04-23T14:55:45.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:44 smithi028 ceph-mon[27810]: pgmap v364: 97 pgs: 97 active+clean; 557 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.6 MiB/s wr, 171 op/s 2024-04-23T14:55:45.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:44 smithi105 ceph-mon[32555]: pgmap v364: 97 pgs: 97 active+clean; 557 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.6 MiB/s wr, 171 op/s 2024-04-23T14:55:45.536 INFO:tasks.workunit.client.0.smithi028.stdout: 1 999131 8.64 MB/sec execute 481 sec latency 17.943 ms 2024-04-23T14:55:46.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:45 smithi038 ceph-mon[32348]: scrub summary: idle 2024-04-23T14:55:46.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:45 smithi028 ceph-mon[27810]: scrub summary: idle 2024-04-23T14:55:46.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:45 smithi105 ceph-mon[32555]: scrub summary: idle 2024-04-23T14:55:46.537 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1000753 8.63 MB/sec execute 482 sec latency 23.275 ms 2024-04-23T14:55:47.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:46 smithi038 ceph-mon[32348]: pgmap v365: 97 pgs: 97 active+clean; 557 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 125 op/s 2024-04-23T14:55:47.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:46 smithi028 ceph-mon[27810]: pgmap v365: 97 pgs: 97 active+clean; 557 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 125 op/s 2024-04-23T14:55:47.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:46 smithi105 ceph-mon[32555]: pgmap v365: 97 pgs: 97 active+clean; 557 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 125 op/s 2024-04-23T14:55:47.537 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1002257 8.64 MB/sec execute 483 sec latency 19.184 ms 2024-04-23T14:55:48.537 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1003504 8.63 MB/sec execute 484 sec latency 18.696 ms 2024-04-23T14:55:49.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:48 smithi038 ceph-mon[32348]: pgmap v366: 97 pgs: 97 active+clean; 566 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.9 MiB/s wr, 189 op/s 2024-04-23T14:55:49.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:48 smithi028 ceph-mon[27810]: pgmap v366: 97 pgs: 97 active+clean; 566 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.9 MiB/s wr, 189 op/s 2024-04-23T14:55:49.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:48 smithi105 ceph-mon[32555]: pgmap v366: 97 pgs: 97 active+clean; 566 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.9 MiB/s wr, 189 op/s 2024-04-23T14:55:49.537 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1005072 8.63 MB/sec execute 485 sec latency 45.702 ms 2024-04-23T14:55:50.537 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1005980 8.63 MB/sec execute 486 sec latency 37.129 ms 2024-04-23T14:55:51.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:50 smithi038 ceph-mon[32348]: pgmap v367: 97 pgs: 97 active+clean; 566 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 132 op/s 2024-04-23T14:55:51.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:50 smithi028 ceph-mon[27810]: pgmap v367: 97 pgs: 97 active+clean; 566 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 132 op/s 2024-04-23T14:55:51.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:50 smithi105 ceph-mon[32555]: pgmap v367: 97 pgs: 97 active+clean; 566 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 132 op/s 2024-04-23T14:55:51.538 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1007195 8.62 MB/sec execute 487 sec latency 27.391 ms 2024-04-23T14:55:52.538 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1008939 8.63 MB/sec execute 488 sec latency 24.487 ms 2024-04-23T14:55:53.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:52 smithi038 ceph-mon[32348]: pgmap v368: 97 pgs: 97 active+clean; 566 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 132 op/s 2024-04-23T14:55:53.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:52 smithi028 ceph-mon[27810]: pgmap v368: 97 pgs: 97 active+clean; 566 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 132 op/s 2024-04-23T14:55:53.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:52 smithi105 ceph-mon[32555]: pgmap v368: 97 pgs: 97 active+clean; 566 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 132 op/s 2024-04-23T14:55:53.538 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1009870 8.61 MB/sec execute 489 sec latency 22.883 ms 2024-04-23T14:55:54.538 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1011554 8.61 MB/sec execute 490 sec latency 31.776 ms 2024-04-23T14:55:55.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:54 smithi038 ceph-mon[32348]: pgmap v369: 97 pgs: 97 active+clean; 574 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.3 MiB/s wr, 201 op/s 2024-04-23T14:55:55.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:54 smithi028 ceph-mon[27810]: pgmap v369: 97 pgs: 97 active+clean; 574 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.3 MiB/s wr, 201 op/s 2024-04-23T14:55:55.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:54 smithi105 ceph-mon[32555]: pgmap v369: 97 pgs: 97 active+clean; 574 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.3 MiB/s wr, 201 op/s 2024-04-23T14:55:55.538 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1013003 8.62 MB/sec execute 491 sec latency 38.258 ms 2024-04-23T14:55:56.538 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1014365 8.61 MB/sec execute 492 sec latency 27.974 ms 2024-04-23T14:55:57.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:56 smithi038 ceph-mon[32348]: pgmap v370: 97 pgs: 97 active+clean; 574 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.9 MiB/s wr, 133 op/s 2024-04-23T14:55:57.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:56 smithi028 ceph-mon[27810]: pgmap v370: 97 pgs: 97 active+clean; 574 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.9 MiB/s wr, 133 op/s 2024-04-23T14:55:57.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:56 smithi105 ceph-mon[32555]: pgmap v370: 97 pgs: 97 active+clean; 574 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.9 MiB/s wr, 133 op/s 2024-04-23T14:55:57.539 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1016129 8.62 MB/sec execute 493 sec latency 21.798 ms 2024-04-23T14:55:58.539 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1017185 8.61 MB/sec execute 494 sec latency 24.556 ms 2024-04-23T14:55:59.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:55:58 smithi038 ceph-mon[32348]: pgmap v371: 97 pgs: 97 active+clean; 583 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.4 MiB/s wr, 195 op/s 2024-04-23T14:55:59.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:55:58 smithi028 ceph-mon[27810]: pgmap v371: 97 pgs: 97 active+clean; 583 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.4 MiB/s wr, 195 op/s 2024-04-23T14:55:59.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:55:58 smithi105 ceph-mon[32555]: pgmap v371: 97 pgs: 97 active+clean; 583 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.4 MiB/s wr, 195 op/s 2024-04-23T14:55:59.539 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1019242 8.61 MB/sec execute 495 sec latency 16.761 ms 2024-04-23T14:56:00.539 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1020481 8.61 MB/sec execute 496 sec latency 26.172 ms 2024-04-23T14:56:01.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:00 smithi038 ceph-mon[32348]: pgmap v372: 97 pgs: 97 active+clean; 583 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 131 op/s 2024-04-23T14:56:01.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:00 smithi028 ceph-mon[27810]: pgmap v372: 97 pgs: 97 active+clean; 583 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 131 op/s 2024-04-23T14:56:01.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:00 smithi105 ceph-mon[32555]: pgmap v372: 97 pgs: 97 active+clean; 583 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 131 op/s 2024-04-23T14:56:01.539 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1021870 8.60 MB/sec execute 497 sec latency 64.070 ms 2024-04-23T14:56:02.539 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1022595 8.60 MB/sec execute 498 sec latency 70.755 ms 2024-04-23T14:56:03.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:02 smithi038 ceph-mon[32348]: pgmap v373: 97 pgs: 97 active+clean; 583 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 131 op/s 2024-04-23T14:56:03.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:02 smithi028 ceph-mon[27810]: pgmap v373: 97 pgs: 97 active+clean; 583 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 131 op/s 2024-04-23T14:56:03.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:02 smithi105 ceph-mon[32555]: pgmap v373: 97 pgs: 97 active+clean; 583 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 131 op/s 2024-04-23T14:56:03.540 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1023713 8.59 MB/sec execute 499 sec latency 16.329 ms 2024-04-23T14:56:04.540 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1024964 8.59 MB/sec execute 500 sec latency 29.829 ms 2024-04-23T14:56:05.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:04 smithi105 ceph-mon[32555]: pgmap v374: 97 pgs: 97 active+clean; 591 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.2 MiB/s wr, 217 op/s 2024-04-23T14:56:05.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:04 smithi038 ceph-mon[32348]: pgmap v374: 97 pgs: 97 active+clean; 591 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.2 MiB/s wr, 217 op/s 2024-04-23T14:56:05.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:04 smithi028 ceph-mon[27810]: pgmap v374: 97 pgs: 97 active+clean; 591 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.2 MiB/s wr, 217 op/s 2024-04-23T14:56:05.540 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1026803 8.60 MB/sec execute 501 sec latency 46.595 ms 2024-04-23T14:56:06.540 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1028073 8.59 MB/sec execute 502 sec latency 39.013 ms 2024-04-23T14:56:07.173 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:06 smithi038 ceph-mon[32348]: pgmap v375: 97 pgs: 97 active+clean; 591 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 149 op/s 2024-04-23T14:56:07.173 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:06 smithi038 ceph-mon[32348]: osdmap e99: 12 total, 12 up, 12 in 2024-04-23T14:56:07.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:06 smithi105 ceph-mon[32555]: pgmap v375: 97 pgs: 97 active+clean; 591 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 149 op/s 2024-04-23T14:56:07.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:06 smithi105 ceph-mon[32555]: osdmap e99: 12 total, 12 up, 12 in 2024-04-23T14:56:07.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:06 smithi028 ceph-mon[27810]: pgmap v375: 97 pgs: 97 active+clean; 591 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 149 op/s 2024-04-23T14:56:07.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:06 smithi028 ceph-mon[27810]: osdmap e99: 12 total, 12 up, 12 in 2024-04-23T14:56:07.540 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1029581 8.59 MB/sec execute 503 sec latency 19.215 ms 2024-04-23T14:56:08.540 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1030671 8.58 MB/sec execute 504 sec latency 22.633 ms 2024-04-23T14:56:09.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:08 smithi105 ceph-mon[32555]: pgmap v377: 97 pgs: 97 active+clean; 599 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 176 op/s 2024-04-23T14:56:09.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:08 smithi038 ceph-mon[32348]: pgmap v377: 97 pgs: 97 active+clean; 599 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 176 op/s 2024-04-23T14:56:09.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:08 smithi028 ceph-mon[27810]: pgmap v377: 97 pgs: 97 active+clean; 599 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 176 op/s 2024-04-23T14:56:09.540 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1032008 8.58 MB/sec execute 505 sec latency 22.198 ms 2024-04-23T14:56:10.540 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1033908 8.59 MB/sec execute 506 sec latency 48.947 ms 2024-04-23T14:56:11.177 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:56:11.177 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:56:11.251 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:10 smithi028 ceph-mon[27810]: pgmap v378: 97 pgs: 97 active+clean; 599 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 176 op/s 2024-04-23T14:56:11.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:10 smithi105 ceph-mon[32555]: pgmap v378: 97 pgs: 97 active+clean; 599 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 176 op/s 2024-04-23T14:56:11.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:10 smithi038 ceph-mon[32348]: pgmap v378: 97 pgs: 97 active+clean; 599 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 176 op/s 2024-04-23T14:56:11.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.403+0000 7ff432800640 1 -- 172.21.15.28:0/3462877999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff42c102c70 msgr2=0x7ff42c106cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:11.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.403+0000 7ff432800640 1 --2- 172.21.15.28:0/3462877999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff42c102c70 0x7ff42c106cb0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff418009900 tx=0x7ff41802a3f0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.403+0000 7ff432800640 1 -- 172.21.15.28:0/3462877999 shutdown_connections 2024-04-23T14:56:11.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.403+0000 7ff432800640 1 --2- 172.21.15.28:0/3462877999 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff42c107390 0x7ff42c10ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.403+0000 7ff432800640 1 --2- 172.21.15.28:0/3462877999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff42c102c70 0x7ff42c106cb0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.403+0000 7ff432800640 1 --2- 172.21.15.28:0/3462877999 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff42c1022c0 0x7ff42c1026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.403+0000 7ff432800640 1 -- 172.21.15.28:0/3462877999 >> 172.21.15.28:0/3462877999 conn(0x7ff42c0fdb60 msgr2=0x7ff42c0fff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:11.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.404+0000 7ff432800640 1 -- 172.21.15.28:0/3462877999 shutdown_connections 2024-04-23T14:56:11.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.404+0000 7ff432800640 1 -- 172.21.15.28:0/3462877999 wait complete. 2024-04-23T14:56:11.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.405+0000 7ff432800640 1 Processor -- start 2024-04-23T14:56:11.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.405+0000 7ff432800640 1 -- start start 2024-04-23T14:56:11.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.405+0000 7ff432800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff42c1022c0 0x7ff42c1a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:11.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.405+0000 7ff432800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff42c107390 0x7ff42c1a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:11.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.405+0000 7ff432800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff42c19aa40 0x7ff42c19aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:11.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.405+0000 7ff432800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff42c112400 con 0x7ff42c107390 2024-04-23T14:56:11.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.405+0000 7ff432800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff42c112280 con 0x7ff42c19aa40 2024-04-23T14:56:11.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.405+0000 7ff432800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff42c112580 con 0x7ff42c1022c0 2024-04-23T14:56:11.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.406+0000 7ff42be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff42c19aa40 0x7ff42c19aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:11.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.406+0000 7ff42be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff42c19aa40 0x7ff42c19aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:47670/0 (socket says 172.21.15.28:47670) 2024-04-23T14:56:11.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.406+0000 7ff42b400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff42c1022c0 0x7ff42c1a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:11.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.406+0000 7ff42be00640 1 -- 172.21.15.28:0/1070165611 learned_addr learned my addr 172.21.15.28:0/1070165611 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:11.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.406+0000 7ff42aa00640 1 --2- 172.21.15.28:0/1070165611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff42c107390 0x7ff42c1a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:11.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.406+0000 7ff42be00640 1 -- 172.21.15.28:0/1070165611 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff42c1022c0 msgr2=0x7ff42c1a0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:11.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.406+0000 7ff42be00640 1 --2- 172.21.15.28:0/1070165611 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff42c1022c0 0x7ff42c1a0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.406+0000 7ff42be00640 1 -- 172.21.15.28:0/1070165611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff42c107390 msgr2=0x7ff42c1a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:11.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.406+0000 7ff42be00640 1 --2- 172.21.15.28:0/1070165611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff42c107390 0x7ff42c1a0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.406+0000 7ff42be00640 1 -- 172.21.15.28:0/1070165611 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff42c19b560 con 0x7ff42c19aa40 2024-04-23T14:56:11.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.406+0000 7ff42be00640 1 --2- 172.21.15.28:0/1070165611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff42c19aa40 0x7ff42c19aef0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7ff42400efd0 tx=0x7ff42400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:11.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.407+0000 7ff423e00640 1 -- 172.21.15.28:0/1070165611 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff424003b70 con 0x7ff42c19aa40 2024-04-23T14:56:11.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.407+0000 7ff423e00640 1 -- 172.21.15.28:0/1070165611 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff424003d10 con 0x7ff42c19aa40 2024-04-23T14:56:11.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.407+0000 7ff432800640 1 -- 172.21.15.28:0/1070165611 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff42c19b850 con 0x7ff42c19aa40 2024-04-23T14:56:11.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.407+0000 7ff423e00640 1 -- 172.21.15.28:0/1070165611 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4240109f0 con 0x7ff42c19aa40 2024-04-23T14:56:11.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.407+0000 7ff432800640 1 -- 172.21.15.28:0/1070165611 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff42c1a7a50 con 0x7ff42c19aa40 2024-04-23T14:56:11.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.410+0000 7ff4357c3740 1 Processor -- start 2024-04-23T14:56:11.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.410+0000 7ff423e00640 1 -- 172.21.15.28:0/1070165611 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7ff424004020 con 0x7ff42c19aa40 2024-04-23T14:56:11.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.411+0000 7ff4357c3740 1 -- start start 2024-04-23T14:56:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.411+0000 7ff4357c3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ad6c8f860 0x562ad6c8fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.411+0000 7ff4357c3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ad6c90180 0x562ad6c9a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.411+0000 7ff4357c3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ad6c9a850 0x562ad6c9cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.411+0000 7ff4357c3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562ad6bf51f0 con 0x562ad6c8f860 2024-04-23T14:56:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.411+0000 7ff4357c3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562ad6b9b350 con 0x562ad6c9a850 2024-04-23T14:56:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.411+0000 7ff4357c3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562ad6bf11b0 con 0x562ad6c90180 2024-04-23T14:56:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.411+0000 7ff42aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ad6c8f860 0x562ad6c8fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.411+0000 7ff42b400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ad6c90180 0x562ad6c9a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.411+0000 7ff42b400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ad6c90180 0x562ad6c9a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:47256/0 (socket says 172.21.15.28:47256) 2024-04-23T14:56:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.412+0000 7ff42b400640 1 -- 172.21.15.28:0/3385342555 learned_addr learned my addr 172.21.15.28:0/3385342555 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.412+0000 7ff42aa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ad6c9a850 0x562ad6c9cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:11.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.412+0000 7ff42b400640 1 -- 172.21.15.28:0/3385342555 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ad6c9a850 msgr2=0x562ad6c9cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:11.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.412+0000 7ff42b400640 1 --2- 172.21.15.28:0/3385342555 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ad6c9a850 0x562ad6c9cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.412+0000 7ff42b400640 1 -- 172.21.15.28:0/3385342555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ad6c8f860 msgr2=0x562ad6c8fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:11.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.412+0000 7ff42b400640 1 --2- 172.21.15.28:0/3385342555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ad6c8f860 0x562ad6c8fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.412+0000 7ff42b400640 1 -- 172.21.15.28:0/3385342555 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ad6beff50 con 0x562ad6c90180 2024-04-23T14:56:11.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.412+0000 7ff42aa00640 1 --2- 172.21.15.28:0/3385342555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ad6c8f860 0x562ad6c8fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:56:11.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.412+0000 7ff42b400640 1 --2- 172.21.15.28:0/3385342555 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ad6c90180 0x562ad6c9a310 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff41c009c90 tx=0x7ff41c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48a0b36eba46ba89 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:11.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.412+0000 7ff423e00640 1 --2- 172.21.15.28:0/1070165611 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7ff4100a3d20 0x7ff4100a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:11.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.413+0000 7ff420c00640 1 -- 172.21.15.28:0/3385342555 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff41c004230 con 0x562ad6c90180 2024-04-23T14:56:11.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.413+0000 7ff423e00640 1 -- 172.21.15.28:0/1070165611 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8586+0+0 (secure 0 0 0) 0x7ff4240d0c60 con 0x7ff42c19aa40 2024-04-23T14:56:11.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.413+0000 7ff420c00640 1 -- 172.21.15.28:0/3385342555 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff41c0043d0 con 0x562ad6c90180 2024-04-23T14:56:11.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.413+0000 7ff420c00640 1 -- 172.21.15.28:0/3385342555 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff41c0197e0 con 0x562ad6c90180 2024-04-23T14:56:11.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.413+0000 7ff42aa00640 1 --2- 172.21.15.28:0/1070165611 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7ff4100a3d20 0x7ff4100a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:11.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.414+0000 7ff4357c3740 1 -- 172.21.15.28:0/3385342555 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ad6c90180 msgr2=0x562ad6c9a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:11.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.414+0000 7ff4357c3740 1 --2- 172.21.15.28:0/3385342555 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ad6c90180 0x562ad6c9a310 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff41c009c90 tx=0x7ff41c011810 comp rx=0 tx=0).stop 2024-04-23T14:56:11.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.414+0000 7ff42aa00640 1 --2- 172.21.15.28:0/1070165611 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7ff4100a3d20 0x7ff4100a61e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff418004770 tx=0x7ff418000bc0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:11.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.414+0000 7ff4357c3740 1 -- 172.21.15.28:0/3385342555 shutdown_connections 2024-04-23T14:56:11.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.414+0000 7ff4357c3740 1 --2- 172.21.15.28:0/3385342555 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ad6c9a850 0x562ad6c9cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.414+0000 7ff4357c3740 1 --2- 172.21.15.28:0/3385342555 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ad6c90180 0x562ad6c9a310 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.414+0000 7ff4357c3740 1 --2- 172.21.15.28:0/3385342555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ad6c8f860 0x562ad6c8fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.414+0000 7ff4357c3740 1 -- 172.21.15.28:0/3385342555 >> 172.21.15.28:0/3385342555 conn(0x562ad6c8bb80 msgr2=0x562ad6c8bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:11.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.414+0000 7ff4357c3740 1 -- 172.21.15.28:0/3385342555 shutdown_connections 2024-04-23T14:56:11.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.414+0000 7ff4357c3740 1 -- 172.21.15.28:0/3385342555 wait complete. 2024-04-23T14:56:11.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.415+0000 7ff4357c3740 1 -- start start 2024-04-23T14:56:11.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.415+0000 7ff4357c3740 1 Processor -- start 2024-04-23T14:56:11.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.415+0000 7ff4357c3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ad6c8f860 0x562ad6c97780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:11.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.415+0000 7ff4357c3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ad6c90180 0x562ad6c97cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:11.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.415+0000 7ff4357c3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ad6c9a850 0x562ad6c98200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:11.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.415+0000 7ff4357c3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562ad6c9d640 con 0x562ad6c8f860 2024-04-23T14:56:11.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.415+0000 7ff4357c3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562ad6bae690 con 0x562ad6c90180 2024-04-23T14:56:11.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.415+0000 7ff4357c3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562ad6c9d7c0 con 0x562ad6c9a850 2024-04-23T14:56:11.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.416+0000 7ff4357c3740 10 client.0 _ll_register_callbacks cb 0x562ad6bea650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:56:11.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.416+0000 7ff42aa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ad6c90180 0x562ad6c97cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:11.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.416+0000 7ff42aa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ad6c90180 0x562ad6c97cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:47692/0 (socket says 172.21.15.28:47692) 2024-04-23T14:56:11.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.416+0000 7ff42aa00640 1 -- 172.21.15.28:0/3746319646 learned_addr learned my addr 172.21.15.28:0/3746319646 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:11.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.416+0000 7ff42b400640 1 --2- 172.21.15.28:0/3746319646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ad6c8f860 0x562ad6c97780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:11.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.416+0000 7ff42be00640 1 --2- 172.21.15.28:0/3746319646 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ad6c9a850 0x562ad6c98200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:11.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.416+0000 7ff42aa00640 1 -- 172.21.15.28:0/3746319646 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ad6c9a850 msgr2=0x562ad6c98200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:11.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.416+0000 7ff42aa00640 1 --2- 172.21.15.28:0/3746319646 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ad6c9a850 0x562ad6c98200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.416+0000 7ff42aa00640 1 -- 172.21.15.28:0/3746319646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ad6c8f860 msgr2=0x562ad6c97780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:11.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.416+0000 7ff42aa00640 1 --2- 172.21.15.28:0/3746319646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ad6c8f860 0x562ad6c97780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.416+0000 7ff42aa00640 1 -- 172.21.15.28:0/3746319646 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ad6bad920 con 0x562ad6c90180 2024-04-23T14:56:11.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.416+0000 7ff42be00640 1 --2- 172.21.15.28:0/3746319646 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ad6c9a850 0x562ad6c98200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:56:11.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.416+0000 7ff42aa00640 1 --2- 172.21.15.28:0/3746319646 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ad6c90180 0x562ad6c97cc0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7ff418009900 tx=0x7ff41802a3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bf8002c5dc92d7b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:11.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.417+0000 7ff415600640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:56:11.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.417+0000 7ff415600640 1 -- 172.21.15.28:0/3746319646 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff41804e030 con 0x562ad6c90180 2024-04-23T14:56:11.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.417+0000 7ff4357c3740 1 -- 172.21.15.28:0/3746319646 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff41c0043d0 con 0x562ad6c90180 2024-04-23T14:56:11.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.417+0000 7ff415600640 1 -- 172.21.15.28:0/3746319646 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8586+0+0 (secure 0 0 0) 0x7ff418004240 con 0x562ad6c90180 2024-04-23T14:56:11.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.417+0000 7ff415600640 1 -- 172.21.15.28:0/3746319646 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7ff418035040 con 0x562ad6c90180 2024-04-23T14:56:11.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.417+0000 7ff4357c3740 10 client.27110 fetch_fsmap learned FSMap version 20 2024-04-23T14:56:11.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.417+0000 7ff4357c3740 1 -- 172.21.15.28:0/3746319646 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x562ad69d17d0 con 0x562ad6c90180 2024-04-23T14:56:11.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.417+0000 7ff415600640 1 -- 172.21.15.28:0/3746319646 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff41c0043d0 con 0x562ad6c90180 2024-04-23T14:56:11.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.418+0000 7ff415600640 1 -- 172.21.15.28:0/3746319646 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7ff418047050 con 0x562ad6c90180 2024-04-23T14:56:11.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.418+0000 7ff4357c3740 10 client.27110 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:56:11.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.418+0000 7ff4357c3740 10 client.27110 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:56:11.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.418+0000 7ff4357c3740 20 client.27110 populate_metadata read hostname 'smithi028' 2024-04-23T14:56:11.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.418+0000 7ff4357c3740 1 --2- 172.21.15.28:0/3746319646 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562ad6caabb0 0x562ad6cad010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:11.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.418+0000 7ff4357c3740 4 client.27110 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:56:11.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.418+0000 7ff4357c3740 1 -- 172.21.15.28:0/3746319646 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562ad6c02c10 con 0x562ad6caabb0 2024-04-23T14:56:11.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.418+0000 7ff42b400640 1 --2- 172.21.15.28:0/3746319646 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562ad6caabb0 0x562ad6cad010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:11.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.419+0000 7ff42b400640 1 --2- 172.21.15.28:0/3746319646 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562ad6caabb0 0x562ad6cad010 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8e407cb01e9dc1c server_cookie=8d4aaf8d0f6e423b in_seq=0 out_seq=0 2024-04-23T14:56:11.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.419+0000 7ff415600640 10 client.27110 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:56:11.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.419+0000 7ff415600640 1 -- 172.21.15.28:0/3746319646 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562ad6c02c10 con 0x562ad6caabb0 2024-04-23T14:56:11.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.419+0000 7ff415600640 10 client.27110 handle_command_reply: tid=0 2024-04-23T14:56:11.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.419+0000 7ff4357c3740 1 client.27110 shutdown 2024-04-23T14:56:11.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.420+0000 7ff4357c3740 1 -- 172.21.15.28:0/3746319646 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ad6c90180 msgr2=0x562ad6c97cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:11.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.420+0000 7ff4357c3740 1 --2- 172.21.15.28:0/3746319646 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ad6c90180 0x562ad6c97cc0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7ff418009900 tx=0x7ff41802a3f0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.420+0000 7ff4357c3740 1 -- 172.21.15.28:0/3746319646 shutdown_connections 2024-04-23T14:56:11.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.420+0000 7ff4357c3740 1 --2- 172.21.15.28:0/3746319646 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562ad6caabb0 0x562ad6cad010 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.420+0000 7ff4357c3740 1 --2- 172.21.15.28:0/3746319646 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ad6c90180 0x562ad6c97cc0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.420+0000 7ff4357c3740 1 --2- 172.21.15.28:0/3746319646 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ad6c9a850 0x562ad6c98200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.420+0000 7ff4357c3740 1 --2- 172.21.15.28:0/3746319646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ad6c8f860 0x562ad6c97780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.420+0000 7ff4357c3740 1 -- 172.21.15.28:0/3746319646 >> 172.21.15.28:0/3746319646 conn(0x562ad6c8bb80 msgr2=0x562ad6c91980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:11.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.420+0000 7ff415600640 0 client.27110 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:56:11.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.420+0000 7ff4357c3740 1 -- 172.21.15.28:0/3746319646 shutdown_connections 2024-04-23T14:56:11.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.420+0000 7ff4357c3740 1 -- 172.21.15.28:0/3746319646 wait complete. 2024-04-23T14:56:11.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.420+0000 7ff4357c3740 20 client.27110 trim_cache size 0 max 16384 2024-04-23T14:56:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.441+0000 7ff4357c3740 1 Processor -- start 2024-04-23T14:56:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.442+0000 7ff4357c3740 1 -- start start 2024-04-23T14:56:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.442+0000 7ff4357c3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ad6c8f860 0x562ad6ca6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.442+0000 7ff4357c3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ad6c90180 0x562ad6c92190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.442+0000 7ff4357c3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ad6c9a850 0x562ad6cb28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.442+0000 7ff4357c3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff42c1ae040 con 0x562ad6c9a850 2024-04-23T14:56:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.442+0000 7ff4357c3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff42c1b08a0 con 0x562ad6c90180 2024-04-23T14:56:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.442+0000 7ff4357c3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff42c1ae690 con 0x562ad6c8f860 2024-04-23T14:56:11.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.442+0000 7ff42aa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ad6c90180 0x562ad6c92190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:11.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.442+0000 7ff42aa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ad6c90180 0x562ad6c92190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:47696/0 (socket says 172.21.15.28:47696) 2024-04-23T14:56:11.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.442+0000 7ff42aa00640 1 -- 172.21.15.28:0/2288685699 learned_addr learned my addr 172.21.15.28:0/2288685699 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:11.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.442+0000 7ff42b400640 1 --2- 172.21.15.28:0/2288685699 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ad6c8f860 0x562ad6ca6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:11.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.442+0000 7ff42be00640 1 --2- 172.21.15.28:0/2288685699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ad6c9a850 0x562ad6cb28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:11.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.443+0000 7ff42aa00640 1 -- 172.21.15.28:0/2288685699 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ad6c8f860 msgr2=0x562ad6ca6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:11.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.443+0000 7ff42aa00640 1 --2- 172.21.15.28:0/2288685699 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ad6c8f860 0x562ad6ca6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.443+0000 7ff42aa00640 1 -- 172.21.15.28:0/2288685699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ad6c9a850 msgr2=0x562ad6cb28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:11.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.443+0000 7ff42aa00640 1 --2- 172.21.15.28:0/2288685699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ad6c9a850 0x562ad6cb28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.443+0000 7ff42aa00640 1 -- 172.21.15.28:0/2288685699 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ad69073f0 con 0x562ad6c90180 2024-04-23T14:56:11.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.443+0000 7ff42aa00640 1 --2- 172.21.15.28:0/2288685699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ad6c90180 0x562ad6c92190 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ff41802a910 tx=0x7ff418031a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9aec4607b05e73b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:11.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.444+0000 7ff417e00640 1 -- 172.21.15.28:0/2288685699 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff41805f070 con 0x562ad6c90180 2024-04-23T14:56:11.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.444+0000 7ff417e00640 1 -- 172.21.15.28:0/2288685699 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff418032d70 con 0x562ad6c90180 2024-04-23T14:56:11.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.444+0000 7ff417e00640 1 -- 172.21.15.28:0/2288685699 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff41804e030 con 0x562ad6c90180 2024-04-23T14:56:11.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.445+0000 7ff4357c3740 1 -- 172.21.15.28:0/2288685699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ad6c90180 msgr2=0x562ad6c92190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:11.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.445+0000 7ff4357c3740 1 --2- 172.21.15.28:0/2288685699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ad6c90180 0x562ad6c92190 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ff41802a910 tx=0x7ff418031a90 comp rx=0 tx=0).stop 2024-04-23T14:56:11.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.445+0000 7ff4357c3740 1 -- 172.21.15.28:0/2288685699 shutdown_connections 2024-04-23T14:56:11.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.445+0000 7ff4357c3740 1 --2- 172.21.15.28:0/2288685699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ad6c90180 0x562ad6c92190 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.445+0000 7ff4357c3740 1 --2- 172.21.15.28:0/2288685699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ad6c9a850 0x562ad6cb28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.445+0000 7ff4357c3740 1 --2- 172.21.15.28:0/2288685699 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ad6c8f860 0x562ad6ca6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.445+0000 7ff4357c3740 1 -- 172.21.15.28:0/2288685699 >> 172.21.15.28:0/2288685699 conn(0x562ad6c8bb80 msgr2=0x562ad6c96050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:11.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.445+0000 7ff4357c3740 1 -- 172.21.15.28:0/2288685699 shutdown_connections 2024-04-23T14:56:11.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.445+0000 7ff4357c3740 1 -- 172.21.15.28:0/2288685699 wait complete. 2024-04-23T14:56:11.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.445+0000 7ff4357c3740 1 -- start start 2024-04-23T14:56:11.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.446+0000 7ff4357c3740 1 Processor -- start 2024-04-23T14:56:11.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.446+0000 7ff4357c3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ad6c8f860 0x562ad6ca2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.446+0000 7ff4357c3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ad6c90180 0x562ad6c9e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.446+0000 7ff4357c3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ad6c9a850 0x562ad6c9e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.446+0000 7ff4357c3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff42c1aed60 con 0x562ad6c90180 2024-04-23T14:56:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.446+0000 7ff4357c3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff42c1b01d0 con 0x562ad6c9a850 2024-04-23T14:56:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.446+0000 7ff4357c3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff42c1af430 con 0x562ad6c8f860 2024-04-23T14:56:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.446+0000 7ff4357c3740 10 client.0 _ll_register_callbacks cb 0x7ff418009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:56:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.446+0000 7ff42be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ad6c9a850 0x562ad6c9e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.446+0000 7ff42be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ad6c9a850 0x562ad6c9e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:47700/0 (socket says 172.21.15.28:47700) 2024-04-23T14:56:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.446+0000 7ff42be00640 1 -- 172.21.15.28:0/1517889943 learned_addr learned my addr 172.21.15.28:0/1517889943 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.446+0000 7ff42aa00640 1 --2- 172.21.15.28:0/1517889943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ad6c90180 0x562ad6c9e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.447+0000 7ff42b400640 1 --2- 172.21.15.28:0/1517889943 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ad6c8f860 0x562ad6ca2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.447+0000 7ff42be00640 1 -- 172.21.15.28:0/1517889943 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ad6c8f860 msgr2=0x562ad6ca2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.447+0000 7ff42be00640 1 --2- 172.21.15.28:0/1517889943 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ad6c8f860 0x562ad6ca2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.447+0000 7ff42be00640 1 -- 172.21.15.28:0/1517889943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ad6c90180 msgr2=0x562ad6c9e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.447+0000 7ff42be00640 1 --2- 172.21.15.28:0/1517889943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ad6c90180 0x562ad6c9e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.447+0000 7ff42be00640 1 -- 172.21.15.28:0/1517889943 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ad68c70f0 con 0x562ad6c9a850 2024-04-23T14:56:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.447+0000 7ff42be00640 1 --2- 172.21.15.28:0/1517889943 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ad6c9a850 0x562ad6c9e590 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7ff4240ce370 tx=0x7ff4240098e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1390fdc4d9edc28 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.447+0000 7ff415600640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:56:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.447+0000 7ff415600640 1 -- 172.21.15.28:0/1517889943 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4240a0f10 con 0x562ad6c9a850 2024-04-23T14:56:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.447+0000 7ff415600640 1 -- 172.21.15.28:0/1517889943 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8586+0+0 (secure 0 0 0) 0x7ff424004310 con 0x562ad6c9a850 2024-04-23T14:56:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.447+0000 7ff4357c3740 1 -- 172.21.15.28:0/1517889943 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff418032d70 con 0x562ad6c9a850 2024-04-23T14:56:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.447+0000 7ff415600640 1 -- 172.21.15.28:0/1517889943 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7ff4240a57d0 con 0x562ad6c9a850 2024-04-23T14:56:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.448+0000 7ff4357c3740 10 client.27116 fetch_fsmap learned FSMap version 20 2024-04-23T14:56:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.448+0000 7ff4357c3740 1 -- 172.21.15.28:0/1517889943 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x562ad6bba810 con 0x562ad6c9a850 2024-04-23T14:56:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.448+0000 7ff415600640 1 -- 172.21.15.28:0/1517889943 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff418032d70 con 0x562ad6c9a850 2024-04-23T14:56:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.448+0000 7ff415600640 1 -- 172.21.15.28:0/1517889943 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7ff42401f020 con 0x562ad6c9a850 2024-04-23T14:56:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.448+0000 7ff4357c3740 10 client.27116 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:56:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.448+0000 7ff4357c3740 10 client.27116 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:56:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.448+0000 7ff4357c3740 20 client.27116 populate_metadata read hostname 'smithi028' 2024-04-23T14:56:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.448+0000 7ff4357c3740 1 --2- 172.21.15.28:0/1517889943 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562ad6d11380 0x562ad6d137a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.448+0000 7ff4357c3740 4 client.27116 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:56:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.448+0000 7ff4357c3740 1 -- 172.21.15.28:0/1517889943 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562ad6bfca80 con 0x562ad6d11380 2024-04-23T14:56:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.448+0000 7ff42b400640 1 --2- 172.21.15.28:0/1517889943 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562ad6d11380 0x562ad6d137a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.449+0000 7ff42b400640 1 --2- 172.21.15.28:0/1517889943 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562ad6d11380 0x562ad6d137a0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3dcb77d17b2199be server_cookie=d522abbe59edda9 in_seq=0 out_seq=0 2024-04-23T14:56:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.449+0000 7ff415600640 10 client.27116 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:56:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.450+0000 7ff415600640 1 -- 172.21.15.28:0/1517889943 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x562ad6bfca80 con 0x562ad6d11380 2024-04-23T14:56:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.450+0000 7ff415600640 10 client.27116 handle_command_reply: tid=0 2024-04-23T14:56:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.450+0000 7ff4357c3740 1 client.27116 shutdown 2024-04-23T14:56:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.451+0000 7ff4357c3740 1 -- 172.21.15.28:0/1517889943 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ad6c9a850 msgr2=0x562ad6c9e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.451+0000 7ff4357c3740 1 --2- 172.21.15.28:0/1517889943 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ad6c9a850 0x562ad6c9e590 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7ff4240ce370 tx=0x7ff4240098e0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.451+0000 7ff4357c3740 1 -- 172.21.15.28:0/1517889943 shutdown_connections 2024-04-23T14:56:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.451+0000 7ff4357c3740 1 --2- 172.21.15.28:0/1517889943 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562ad6d11380 0x562ad6d137a0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.451+0000 7ff4357c3740 1 --2- 172.21.15.28:0/1517889943 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ad6c9a850 0x562ad6c9e590 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.451+0000 7ff4357c3740 1 --2- 172.21.15.28:0/1517889943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ad6c90180 0x562ad6c9e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.451+0000 7ff415600640 0 client.27116 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:56:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.451+0000 7ff4357c3740 1 --2- 172.21.15.28:0/1517889943 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562ad6c8f860 0x562ad6ca2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.451+0000 7ff4357c3740 1 -- 172.21.15.28:0/1517889943 >> 172.21.15.28:0/1517889943 conn(0x562ad6c8bb80 msgr2=0x562ad6c97fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.451+0000 7ff4357c3740 1 -- 172.21.15.28:0/1517889943 shutdown_connections 2024-04-23T14:56:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.451+0000 7ff4357c3740 1 -- 172.21.15.28:0/1517889943 wait complete. 2024-04-23T14:56:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.451+0000 7ff4357c3740 20 client.27116 trim_cache size 0 max 16384 2024-04-23T14:56:11.457 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:56:11.458 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-23T14:56:11.458 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-23T14:56:11.458 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:56:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.452+0000 7ff432800640 1 -- 172.21.15.28:0/1070165611 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7ff4100a3d20 msgr2=0x7ff4100a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.452+0000 7ff432800640 1 --2- 172.21.15.28:0/1070165611 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7ff4100a3d20 0x7ff4100a61e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff418004770 tx=0x7ff418000bc0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.452+0000 7ff432800640 1 -- 172.21.15.28:0/1070165611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff42c19aa40 msgr2=0x7ff42c19aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.452+0000 7ff432800640 1 --2- 172.21.15.28:0/1070165611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff42c19aa40 0x7ff42c19aef0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7ff42400efd0 tx=0x7ff42400c6a0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.452+0000 7ff432800640 1 -- 172.21.15.28:0/1070165611 shutdown_connections 2024-04-23T14:56:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.452+0000 7ff432800640 1 --2- 172.21.15.28:0/1070165611 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7ff4100a3d20 0x7ff4100a61e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.452+0000 7ff432800640 1 --2- 172.21.15.28:0/1070165611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff42c19aa40 0x7ff42c19aef0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.452+0000 7ff432800640 1 --2- 172.21.15.28:0/1070165611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff42c107390 0x7ff42c1a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.452+0000 7ff432800640 1 --2- 172.21.15.28:0/1070165611 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff42c1022c0 0x7ff42c1a0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.452+0000 7ff432800640 1 -- 172.21.15.28:0/1070165611 >> 172.21.15.28:0/1070165611 conn(0x7ff42c0fdb60 msgr2=0x7ff42c0fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.452+0000 7ff432800640 1 -- 172.21.15.28:0/1070165611 shutdown_connections 2024-04-23T14:56:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.453+0000 7ff432800640 1 -- 172.21.15.28:0/1070165611 wait complete. 2024-04-23T14:56:11.473 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-23T14:56:11.473 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:56:11.540 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1035213 8.58 MB/sec execute 507 sec latency 40.700 ms 2024-04-23T14:56:11.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.679+0000 7f42fe800640 1 -- 172.21.15.28:0/2717588912 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42f810e6b0 msgr2=0x7f42f810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:11.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.679+0000 7f42fe800640 1 --2- 172.21.15.28:0/2717588912 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42f810e6b0 0x7f42f810ea90 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f42e8009900 tx=0x7f42e802a3f0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.680+0000 7f42fe800640 1 -- 172.21.15.28:0/2717588912 shutdown_connections 2024-04-23T14:56:11.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.680+0000 7f42fe800640 1 --2- 172.21.15.28:0/2717588912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42f8108fd0 0x7f42f81154e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.680+0000 7f42fe800640 1 --2- 172.21.15.28:0/2717588912 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f42f8108610 0x7f42f8108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.680+0000 7f42fe800640 1 --2- 172.21.15.28:0/2717588912 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42f810e6b0 0x7f42f810ea90 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.680+0000 7f42fe800640 1 -- 172.21.15.28:0/2717588912 >> 172.21.15.28:0/2717588912 conn(0x7f42f8075800 msgr2=0x7f42f8106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:11.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.680+0000 7f42fe800640 1 -- 172.21.15.28:0/2717588912 shutdown_connections 2024-04-23T14:56:11.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.680+0000 7f42fe800640 1 -- 172.21.15.28:0/2717588912 wait complete. 2024-04-23T14:56:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.681+0000 7f42fe800640 1 Processor -- start 2024-04-23T14:56:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.681+0000 7f42fe800640 1 -- start start 2024-04-23T14:56:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.681+0000 7f42fe800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42f8108610 0x7f42f819eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.681+0000 7f42fe800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42f8108fd0 0x7f42f819f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.681+0000 7f42fe800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f42f810e6b0 0x7f42f81a3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.681+0000 7f42fe800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f42f81189f0 con 0x7f42f8108610 2024-04-23T14:56:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.681+0000 7f42fe800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f42f8118870 con 0x7f42f8108fd0 2024-04-23T14:56:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.681+0000 7f42fe800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f42f8118b70 con 0x7f42f810e6b0 2024-04-23T14:56:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.681+0000 7f42f6a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42f8108fd0 0x7f42f819f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.681+0000 7f42f6a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42f8108fd0 0x7f42f819f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:47732/0 (socket says 172.21.15.28:47732) 2024-04-23T14:56:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.681+0000 7f42f6a00640 1 -- 172.21.15.28:0/2006177424 learned_addr learned my addr 172.21.15.28:0/2006177424 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.681+0000 7f42f7400640 1 --2- 172.21.15.28:0/2006177424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42f8108610 0x7f42f819eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.682+0000 7f42f7e00640 1 --2- 172.21.15.28:0/2006177424 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f42f810e6b0 0x7f42f81a3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.682+0000 7f42f6a00640 1 -- 172.21.15.28:0/2006177424 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f42f810e6b0 msgr2=0x7f42f81a3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.682+0000 7f42f6a00640 1 --2- 172.21.15.28:0/2006177424 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f42f810e6b0 0x7f42f81a3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.682+0000 7f42f6a00640 1 -- 172.21.15.28:0/2006177424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42f8108610 msgr2=0x7f42f819eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.682+0000 7f42f6a00640 1 --2- 172.21.15.28:0/2006177424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42f8108610 0x7f42f819eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.682+0000 7f42f6a00640 1 -- 172.21.15.28:0/2006177424 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42f81a3e70 con 0x7f42f8108fd0 2024-04-23T14:56:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.682+0000 7f42f6a00640 1 --2- 172.21.15.28:0/2006177424 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42f8108fd0 0x7f42f819f410 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f42e400e9f0 tx=0x7f42e400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.683+0000 7f42efe00640 1 -- 172.21.15.28:0/2006177424 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e400cd80 con 0x7f42f8108fd0 2024-04-23T14:56:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.683+0000 7f42efe00640 1 -- 172.21.15.28:0/2006177424 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42e4010910 con 0x7f42f8108fd0 2024-04-23T14:56:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.683+0000 7f42fe800640 1 -- 172.21.15.28:0/2006177424 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42f81a4160 con 0x7f42f8108fd0 2024-04-23T14:56:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.683+0000 7f42efe00640 1 -- 172.21.15.28:0/2006177424 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e4018990 con 0x7f42f8108fd0 2024-04-23T14:56:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.683+0000 7f42fe800640 1 -- 172.21.15.28:0/2006177424 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42f81ab9f0 con 0x7f42f8108fd0 2024-04-23T14:56:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.684+0000 7f42fe800640 1 -- 172.21.15.28:0/2006177424 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42f8069270 con 0x7f42f8108fd0 2024-04-23T14:56:11.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.687+0000 7f42efe00640 1 -- 172.21.15.28:0/2006177424 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f42e4010ac0 con 0x7f42f8108fd0 2024-04-23T14:56:11.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.688+0000 7f42efe00640 1 --2- 172.21.15.28:0/2006177424 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f42dc0a3da0 0x7f42dc0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:11.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.688+0000 7f42efe00640 1 -- 172.21.15.28:0/2006177424 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8586+0+0 (secure 0 0 0) 0x7f42e40d0f00 con 0x7f42f8108fd0 2024-04-23T14:56:11.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.689+0000 7f42f7400640 1 --2- 172.21.15.28:0/2006177424 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f42dc0a3da0 0x7f42dc0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:11.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.689+0000 7f42f7400640 1 --2- 172.21.15.28:0/2006177424 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f42dc0a3da0 0x7f42dc0a6260 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f42e8004770 tx=0x7f42e80058d0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:11.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.694+0000 7f42efe00640 1 -- 172.21.15.28:0/2006177424 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f42e409ceb0 con 0x7f42f8108fd0 2024-04-23T14:56:11.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.988+0000 7f42fe800640 1 -- 172.21.15.28:0/2006177424 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f42f81a01d0 con 0x7f42f8108fd0 2024-04-23T14:56:11.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.989+0000 7f42efe00640 1 -- 172.21.15.28:0/2006177424 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7f42f81a01d0 con 0x7f42f8108fd0 2024-04-23T14:56:11.991 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:56:11.991 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:56:11.991 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:56:11.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.993+0000 7f42fe800640 1 -- 172.21.15.28:0/2006177424 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f42dc0a3da0 msgr2=0x7f42dc0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:11.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.993+0000 7f42fe800640 1 --2- 172.21.15.28:0/2006177424 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f42dc0a3da0 0x7f42dc0a6260 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f42e8004770 tx=0x7f42e80058d0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.993+0000 7f42fe800640 1 -- 172.21.15.28:0/2006177424 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42f8108fd0 msgr2=0x7f42f819f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:11.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.993+0000 7f42fe800640 1 --2- 172.21.15.28:0/2006177424 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42f8108fd0 0x7f42f819f410 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f42e400e9f0 tx=0x7f42e400eeb0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.995+0000 7f42fe800640 1 -- 172.21.15.28:0/2006177424 shutdown_connections 2024-04-23T14:56:11.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.995+0000 7f42fe800640 1 --2- 172.21.15.28:0/2006177424 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f42dc0a3da0 0x7f42dc0a6260 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.995+0000 7f42fe800640 1 --2- 172.21.15.28:0/2006177424 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42f8108fd0 0x7f42f819f410 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.995+0000 7f42fe800640 1 --2- 172.21.15.28:0/2006177424 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f42f810e6b0 0x7f42f81a3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.995+0000 7f42fe800640 1 --2- 172.21.15.28:0/2006177424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42f8108610 0x7f42f819eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:11.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.995+0000 7f42fe800640 1 -- 172.21.15.28:0/2006177424 >> 172.21.15.28:0/2006177424 conn(0x7f42f8075800 msgr2=0x7f42f8076820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:11.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.996+0000 7f42fe800640 1 -- 172.21.15.28:0/2006177424 shutdown_connections 2024-04-23T14:56:11.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:11.996+0000 7f42fe800640 1 -- 172.21.15.28:0/2006177424 wait complete. 2024-04-23T14:56:12.019 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-23T14:56:12.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.244+0000 7fc50fe00640 1 -- 172.21.15.28:0/2960407369 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc510076530 msgr2=0x7fc510076910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:12.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.244+0000 7fc50fe00640 1 --2- 172.21.15.28:0/2960407369 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc510076530 0x7fc510076910 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc500009930 tx=0x7fc50002a420 comp rx=0 tx=0).stop 2024-04-23T14:56:12.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.244+0000 7fc50fe00640 1 -- 172.21.15.28:0/2960407369 shutdown_connections 2024-04-23T14:56:12.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.244+0000 7fc50fe00640 1 --2- 172.21.15.28:0/2960407369 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc510076e50 0x7fc510112810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:12.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.244+0000 7fc50fe00640 1 --2- 172.21.15.28:0/2960407369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc510112d50 0x7fc510115140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:12.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.244+0000 7fc50fe00640 1 --2- 172.21.15.28:0/2960407369 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc510076530 0x7fc510076910 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:12.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.244+0000 7fc50fe00640 1 -- 172.21.15.28:0/2960407369 >> 172.21.15.28:0/2960407369 conn(0x7fc5100ffb40 msgr2=0x7fc510101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:12.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.244+0000 7fc50fe00640 1 -- 172.21.15.28:0/2960407369 shutdown_connections 2024-04-23T14:56:12.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.244+0000 7fc50fe00640 1 -- 172.21.15.28:0/2960407369 wait complete. 2024-04-23T14:56:12.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.245+0000 7fc50fe00640 1 Processor -- start 2024-04-23T14:56:12.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.246+0000 7fc50fe00640 1 -- start start 2024-04-23T14:56:12.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.246+0000 7fc50fe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc510076530 0x7fc51019ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:12.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.246+0000 7fc50fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc510076e50 0x7fc51019f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:12.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.246+0000 7fc50fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc510112d50 0x7fc5101a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:12.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.246+0000 7fc50fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc510118910 con 0x7fc510112d50 2024-04-23T14:56:12.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.246+0000 7fc50fe00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc510118790 con 0x7fc510076e50 2024-04-23T14:56:12.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.246+0000 7fc50fe00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc510118a90 con 0x7fc510076530 2024-04-23T14:56:12.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.246+0000 7fc50ea00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc510076530 0x7fc51019ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:12.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.246+0000 7fc50e000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc510076e50 0x7fc51019f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:12.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.246+0000 7fc50ea00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc510076530 0x7fc51019ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:47310/0 (socket says 172.21.15.28:47310) 2024-04-23T14:56:12.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.246+0000 7fc50ea00640 1 -- 172.21.15.28:0/3848527927 learned_addr learned my addr 172.21.15.28:0/3848527927 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:12.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.246+0000 7fc50f400640 1 --2- 172.21.15.28:0/3848527927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc510112d50 0x7fc5101a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:12.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.247+0000 7fc50e000640 1 -- 172.21.15.28:0/3848527927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc510076530 msgr2=0x7fc51019ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:12.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.247+0000 7fc50e000640 1 --2- 172.21.15.28:0/3848527927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc510076530 0x7fc51019ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:12.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.247+0000 7fc50e000640 1 -- 172.21.15.28:0/3848527927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc510112d50 msgr2=0x7fc5101a36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:12.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.247+0000 7fc50e000640 1 --2- 172.21.15.28:0/3848527927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc510112d50 0x7fc5101a36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:12.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.247+0000 7fc50e000640 1 -- 172.21.15.28:0/3848527927 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5101a3e00 con 0x7fc510076e50 2024-04-23T14:56:12.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.247+0000 7fc50e000640 1 --2- 172.21.15.28:0/3848527927 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc510076e50 0x7fc51019f350 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fc4f800e9f0 tx=0x7fc4f800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:12.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.247+0000 7fc507400640 1 -- 172.21.15.28:0/3848527927 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f800ccc0 con 0x7fc510076e50 2024-04-23T14:56:12.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.247+0000 7fc507400640 1 -- 172.21.15.28:0/3848527927 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4f8005020 con 0x7fc510076e50 2024-04-23T14:56:12.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.247+0000 7fc50fe00640 1 -- 172.21.15.28:0/3848527927 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5101a40f0 con 0x7fc510076e50 2024-04-23T14:56:12.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.247+0000 7fc507400640 1 -- 172.21.15.28:0/3848527927 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f80049e0 con 0x7fc510076e50 2024-04-23T14:56:12.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.247+0000 7fc50fe00640 1 -- 172.21.15.28:0/3848527927 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5101ab980 con 0x7fc510076e50 2024-04-23T14:56:12.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.251+0000 7fc518b98740 1 Processor -- start 2024-04-23T14:56:12.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.251+0000 7fc518b98740 1 -- start start 2024-04-23T14:56:12.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.251+0000 7fc507400640 1 -- 172.21.15.28:0/3848527927 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fc4f8005560 con 0x7fc510076e50 2024-04-23T14:56:12.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.252+0000 7fc518b98740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d5b38ab40 0x560d5b38af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:12.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.252+0000 7fc518b98740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d5b38b460 0x560d5b3955f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:12.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.252+0000 7fc518b98740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d5b395b30 0x560d5b397f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:12.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.252+0000 7fc518b98740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560d5b2f01d0 con 0x560d5b395b30 2024-04-23T14:56:12.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.252+0000 7fc518b98740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560d5b296450 con 0x560d5b38b460 2024-04-23T14:56:12.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.252+0000 7fc518b98740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560d5b2ec190 con 0x560d5b38ab40 2024-04-23T14:56:12.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.252+0000 7fc50ea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d5b38b460 0x560d5b3955f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:12.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.252+0000 7fc50ea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d5b38b460 0x560d5b3955f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:47754/0 (socket says 172.21.15.28:47754) 2024-04-23T14:56:12.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.252+0000 7fc50f400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d5b38ab40 0x560d5b38af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:12.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.252+0000 7fc50ea00640 1 -- 172.21.15.28:0/125716189 learned_addr learned my addr 172.21.15.28:0/125716189 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:12.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.252+0000 7fc50e000640 1 --2- 172.21.15.28:0/125716189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d5b395b30 0x560d5b397f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:12.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.253+0000 7fc50ea00640 1 -- 172.21.15.28:0/125716189 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d5b38ab40 msgr2=0x560d5b38af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:12.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.253+0000 7fc50ea00640 1 --2- 172.21.15.28:0/125716189 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d5b38ab40 0x560d5b38af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:12.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.253+0000 7fc50ea00640 1 -- 172.21.15.28:0/125716189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d5b395b30 msgr2=0x560d5b397f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:12.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.253+0000 7fc50ea00640 1 --2- 172.21.15.28:0/125716189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d5b395b30 0x560d5b397f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:12.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.253+0000 7fc50ea00640 1 -- 172.21.15.28:0/125716189 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d5b353310 con 0x560d5b38b460 2024-04-23T14:56:12.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.253+0000 7fc50ea00640 1 --2- 172.21.15.28:0/125716189 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d5b38b460 0x560d5b3955f0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fc500005ec0 tx=0x7fc500004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d951ec59bdc004c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:12.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.253+0000 7fc4ffe00640 1 -- 172.21.15.28:0/125716189 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc500035040 con 0x560d5b38b460 2024-04-23T14:56:12.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.253+0000 7fc4ffe00640 1 -- 172.21.15.28:0/125716189 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc500005520 con 0x560d5b38b460 2024-04-23T14:56:12.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.253+0000 7fc4ffe00640 1 -- 172.21.15.28:0/125716189 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc500003850 con 0x560d5b38b460 2024-04-23T14:56:12.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.254+0000 7fc507400640 1 --2- 172.21.15.28:0/3848527927 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc4f40a3da0 0x7fc4f40a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:12.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.254+0000 7fc507400640 1 -- 172.21.15.28:0/3848527927 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8542+0+0 (secure 0 0 0) 0x7fc4f80d07a0 con 0x7fc510076e50 2024-04-23T14:56:12.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.254+0000 7fc50f400640 1 --2- 172.21.15.28:0/3848527927 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc4f40a3da0 0x7fc4f40a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:12.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.254+0000 7fc518b98740 1 -- 172.21.15.28:0/125716189 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d5b38b460 msgr2=0x560d5b3955f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:12.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.254+0000 7fc518b98740 1 --2- 172.21.15.28:0/125716189 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d5b38b460 0x560d5b3955f0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fc500005ec0 tx=0x7fc500004040 comp rx=0 tx=0).stop 2024-04-23T14:56:12.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.255+0000 7fc518b98740 1 -- 172.21.15.28:0/125716189 shutdown_connections 2024-04-23T14:56:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.255+0000 7fc518b98740 1 --2- 172.21.15.28:0/125716189 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d5b38b460 0x560d5b3955f0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.255+0000 7fc518b98740 1 --2- 172.21.15.28:0/125716189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d5b395b30 0x560d5b397f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.255+0000 7fc518b98740 1 --2- 172.21.15.28:0/125716189 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d5b38ab40 0x560d5b38af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.255+0000 7fc518b98740 1 -- 172.21.15.28:0/125716189 >> 172.21.15.28:0/125716189 conn(0x560d5b386e60 msgr2=0x560d5b387240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.255+0000 7fc518b98740 1 -- 172.21.15.28:0/125716189 shutdown_connections 2024-04-23T14:56:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.255+0000 7fc50f400640 1 --2- 172.21.15.28:0/3848527927 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc4f40a3da0 0x7fc4f40a6260 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc51006b2d0 tx=0x7fc508009760 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.255+0000 7fc518b98740 1 -- 172.21.15.28:0/125716189 wait complete. 2024-04-23T14:56:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.255+0000 7fc518b98740 1 -- start start 2024-04-23T14:56:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.255+0000 7fc518b98740 1 Processor -- start 2024-04-23T14:56:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.256+0000 7fc518b98740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d5b38ab40 0x560d5b3954d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.256+0000 7fc518b98740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d5b38b460 0x560d5b3927c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.256+0000 7fc518b98740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d5b395b30 0x560d5b392d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.256+0000 7fc518b98740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560d5b398960 con 0x560d5b38b460 2024-04-23T14:56:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.256+0000 7fc518b98740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560d5b2a9810 con 0x560d5b38ab40 2024-04-23T14:56:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.256+0000 7fc518b98740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560d5b398ae0 con 0x560d5b395b30 2024-04-23T14:56:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.256+0000 7fc518b98740 10 client.0 _ll_register_callbacks cb 0x560d5b277140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:56:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.256+0000 7fc50ea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d5b38ab40 0x560d5b3954d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.256+0000 7fc50ea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d5b38ab40 0x560d5b3954d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:47770/0 (socket says 172.21.15.28:47770) 2024-04-23T14:56:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.256+0000 7fc50ea00640 1 -- 172.21.15.28:0/3686048483 learned_addr learned my addr 172.21.15.28:0/3686048483 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.256+0000 7fc50e000640 1 --2- 172.21.15.28:0/3686048483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d5b38b460 0x560d5b3927c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.256+0000 7fc50f400640 1 --2- 172.21.15.28:0/3686048483 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d5b395b30 0x560d5b392d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.257+0000 7fc50ea00640 1 -- 172.21.15.28:0/3686048483 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d5b395b30 msgr2=0x560d5b392d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.257+0000 7fc50ea00640 1 --2- 172.21.15.28:0/3686048483 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d5b395b30 0x560d5b392d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.257+0000 7fc50ea00640 1 -- 172.21.15.28:0/3686048483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d5b38b460 msgr2=0x560d5b3927c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.257+0000 7fc50ea00640 1 --2- 172.21.15.28:0/3686048483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d5b38b460 0x560d5b3927c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.257+0000 7fc50ea00640 1 -- 172.21.15.28:0/3686048483 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d5b2ba3f0 con 0x560d5b38ab40 2024-04-23T14:56:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.257+0000 7fc50e000640 1 --2- 172.21.15.28:0/3686048483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d5b38b460 0x560d5b3927c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:56:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.257+0000 7fc50ea00640 1 --2- 172.21.15.28:0/3686048483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d5b38ab40 0x560d5b3954d0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc500004550 tx=0x7fc500009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ae9220313cbbb1f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.257+0000 7fc4fcc00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:56:12.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.257+0000 7fc4fcc00640 1 -- 172.21.15.28:0/3686048483 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc500052060 con 0x560d5b38ab40 2024-04-23T14:56:12.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.257+0000 7fc4fcc00640 1 -- 172.21.15.28:0/3686048483 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8542+0+0 (secure 0 0 0) 0x7fc500005060 con 0x560d5b38ab40 2024-04-23T14:56:12.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.257+0000 7fc518b98740 1 -- 172.21.15.28:0/3686048483 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc500005520 con 0x560d5b38ab40 2024-04-23T14:56:12.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.257+0000 7fc4fcc00640 1 -- 172.21.15.28:0/3686048483 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fc500035040 con 0x560d5b38ab40 2024-04-23T14:56:12.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.258+0000 7fc518b98740 10 client.27134 fetch_fsmap learned FSMap version 20 2024-04-23T14:56:12.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.258+0000 7fc518b98740 1 -- 172.21.15.28:0/3686048483 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x560d5b0023f0 con 0x560d5b38ab40 2024-04-23T14:56:12.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.258+0000 7fc4fcc00640 1 -- 172.21.15.28:0/3686048483 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc500005520 con 0x560d5b38ab40 2024-04-23T14:56:12.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.258+0000 7fc4fcc00640 1 -- 172.21.15.28:0/3686048483 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fc50004c070 con 0x560d5b38ab40 2024-04-23T14:56:12.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.258+0000 7fc518b98740 10 client.27134 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:56:12.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.258+0000 7fc518b98740 10 client.27134 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:56:12.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.258+0000 7fc518b98740 20 client.27134 populate_metadata read hostname 'smithi028' 2024-04-23T14:56:12.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.258+0000 7fc518b98740 1 --2- 172.21.15.28:0/3686048483 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560d5b3a5ac0 0x560d5b3a7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:12.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.258+0000 7fc518b98740 4 client.27134 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:56:12.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.258+0000 7fc518b98740 1 -- 172.21.15.28:0/3686048483 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d5b2fdbf0 con 0x560d5b3a5ac0 2024-04-23T14:56:12.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.258+0000 7fc50e000640 1 --2- 172.21.15.28:0/3686048483 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560d5b3a5ac0 0x560d5b3a7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:12.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.259+0000 7fc50e000640 1 --2- 172.21.15.28:0/3686048483 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560d5b3a5ac0 0x560d5b3a7eb0 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38740c55b3dc5011 server_cookie=c62d1d45df08625b in_seq=0 out_seq=0 2024-04-23T14:56:12.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.259+0000 7fc4fcc00640 10 client.27134 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:56:12.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.260+0000 7fc4fcc00640 1 -- 172.21.15.28:0/3686048483 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560d5b2fdbf0 con 0x560d5b3a5ac0 2024-04-23T14:56:12.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.260+0000 7fc4fcc00640 10 client.27134 handle_command_reply: tid=0 2024-04-23T14:56:12.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.260+0000 7fc518b98740 1 client.27134 shutdown 2024-04-23T14:56:12.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.260+0000 7fc518b98740 1 -- 172.21.15.28:0/3686048483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d5b38ab40 msgr2=0x560d5b3954d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:12.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.260+0000 7fc518b98740 1 --2- 172.21.15.28:0/3686048483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d5b38ab40 0x560d5b3954d0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc500004550 tx=0x7fc500009260 comp rx=0 tx=0).stop 2024-04-23T14:56:12.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.260+0000 7fc518b98740 1 -- 172.21.15.28:0/3686048483 shutdown_connections 2024-04-23T14:56:12.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.260+0000 7fc518b98740 1 --2- 172.21.15.28:0/3686048483 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560d5b3a5ac0 0x560d5b3a7eb0 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:12.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.260+0000 7fc518b98740 1 --2- 172.21.15.28:0/3686048483 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d5b395b30 0x560d5b392d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:12.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.260+0000 7fc518b98740 1 --2- 172.21.15.28:0/3686048483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d5b38b460 0x560d5b3927c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:12.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.260+0000 7fc518b98740 1 --2- 172.21.15.28:0/3686048483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d5b38ab40 0x560d5b3954d0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:12.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.260+0000 7fc4fcc00640 0 client.27134 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:56:12.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.260+0000 7fc518b98740 1 -- 172.21.15.28:0/3686048483 >> 172.21.15.28:0/3686048483 conn(0x560d5b386e60 msgr2=0x560d5b38cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:12.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.260+0000 7fc518b98740 1 -- 172.21.15.28:0/3686048483 shutdown_connections 2024-04-23T14:56:12.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.261+0000 7fc518b98740 1 -- 172.21.15.28:0/3686048483 wait complete. 2024-04-23T14:56:12.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.261+0000 7fc518b98740 20 client.27134 trim_cache size 0 max 16384 2024-04-23T14:56:12.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.280+0000 7fc518b98740 1 Processor -- start 2024-04-23T14:56:12.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.280+0000 7fc518b98740 1 -- start start 2024-04-23T14:56:12.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.280+0000 7fc518b98740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d5b38ab40 0x560d5b3a1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:12.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.280+0000 7fc518b98740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d5b38b460 0x560d5b395000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:12.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.280+0000 7fc518b98740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d5b395b30 0x560d5b3ad480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:12.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.280+0000 7fc518b98740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc5101b21a0 con 0x560d5b395b30 2024-04-23T14:56:12.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.280+0000 7fc518b98740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc5101b4a00 con 0x560d5b38b460 2024-04-23T14:56:12.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.280+0000 7fc518b98740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc5101b27f0 con 0x560d5b38ab40 2024-04-23T14:56:12.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.281+0000 7fc50e000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d5b38b460 0x560d5b395000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:12.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.281+0000 7fc50ea00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d5b38ab40 0x560d5b3a1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:12.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.281+0000 7fc50f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d5b395b30 0x560d5b3ad480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:12.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.281+0000 7fc50e000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d5b38b460 0x560d5b395000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:47784/0 (socket says 172.21.15.28:47784) 2024-04-23T14:56:12.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.281+0000 7fc50ea00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d5b38ab40 0x560d5b3a1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:47330/0 (socket says 172.21.15.28:47330) 2024-04-23T14:56:12.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.281+0000 7fc50e000640 1 -- 172.21.15.28:0/2913947396 learned_addr learned my addr 172.21.15.28:0/2913947396 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:12.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.281+0000 7fc50e000640 1 -- 172.21.15.28:0/2913947396 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d5b38ab40 msgr2=0x560d5b3a1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:12.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.281+0000 7fc50e000640 1 --2- 172.21.15.28:0/2913947396 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d5b38ab40 0x560d5b3a1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:12.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.281+0000 7fc50e000640 1 -- 172.21.15.28:0/2913947396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d5b395b30 msgr2=0x560d5b3ad480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:12.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.281+0000 7fc50e000640 1 --2- 172.21.15.28:0/2913947396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d5b395b30 0x560d5b3ad480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:12.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.281+0000 7fc50e000640 1 -- 172.21.15.28:0/2913947396 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d5b3930d0 con 0x560d5b38b460 2024-04-23T14:56:12.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.282+0000 7fc50e000640 1 --2- 172.21.15.28:0/2913947396 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d5b38b460 0x560d5b395000 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8099100 tx=0x7fc4f8087040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8994c330e518e68 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:12.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.282+0000 7fc4ff400640 1 -- 172.21.15.28:0/2913947396 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f801c040 con 0x560d5b38b460 2024-04-23T14:56:12.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.282+0000 7fc4ff400640 1 -- 172.21.15.28:0/2913947396 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4f80051d0 con 0x560d5b38b460 2024-04-23T14:56:12.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.282+0000 7fc4ff400640 1 -- 172.21.15.28:0/2913947396 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f809a7d0 con 0x560d5b38b460 2024-04-23T14:56:12.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.283+0000 7fc518b98740 1 -- 172.21.15.28:0/2913947396 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d5b38b460 msgr2=0x560d5b395000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:12.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.283+0000 7fc518b98740 1 --2- 172.21.15.28:0/2913947396 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d5b38b460 0x560d5b395000 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8099100 tx=0x7fc4f8087040 comp rx=0 tx=0).stop 2024-04-23T14:56:12.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.283+0000 7fc518b98740 1 -- 172.21.15.28:0/2913947396 shutdown_connections 2024-04-23T14:56:12.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.283+0000 7fc518b98740 1 --2- 172.21.15.28:0/2913947396 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d5b38b460 0x560d5b395000 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:12.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.283+0000 7fc518b98740 1 --2- 172.21.15.28:0/2913947396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d5b395b30 0x560d5b3ad480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:12.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.283+0000 7fc518b98740 1 --2- 172.21.15.28:0/2913947396 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d5b38ab40 0x560d5b3a1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:12.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.283+0000 7fc518b98740 1 -- 172.21.15.28:0/2913947396 >> 172.21.15.28:0/2913947396 conn(0x560d5b386e60 msgr2=0x560d5b397e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:12.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.283+0000 7fc518b98740 1 -- 172.21.15.28:0/2913947396 shutdown_connections 2024-04-23T14:56:12.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.283+0000 7fc518b98740 1 -- 172.21.15.28:0/2913947396 wait complete. 2024-04-23T14:56:12.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.283+0000 7fc518b98740 1 -- start start 2024-04-23T14:56:12.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.283+0000 7fc518b98740 1 Processor -- start 2024-04-23T14:56:12.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.284+0000 7fc518b98740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d5b38ab40 0x560d5b38f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:12.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.284+0000 7fc518b98740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d5b38b460 0x560d5b38f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:12.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.284+0000 7fc518b98740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d5b395b30 0x560d5b39cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:12.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.284+0000 7fc518b98740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc5101b2ec0 con 0x560d5b38b460 2024-04-23T14:56:12.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.284+0000 7fc518b98740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc5101b4330 con 0x560d5b38ab40 2024-04-23T14:56:12.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.284+0000 7fc518b98740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc5101b3590 con 0x560d5b395b30 2024-04-23T14:56:12.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.284+0000 7fc518b98740 10 client.0 _ll_register_callbacks cb 0x7fc4f80d6420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:56:12.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.284+0000 7fc50ea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d5b38ab40 0x560d5b38f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:12.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.284+0000 7fc50ea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d5b38ab40 0x560d5b38f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:47796/0 (socket says 172.21.15.28:47796) 2024-04-23T14:56:12.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.284+0000 7fc50ea00640 1 -- 172.21.15.28:0/4180493997 learned_addr learned my addr 172.21.15.28:0/4180493997 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:12.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.284+0000 7fc50e000640 1 --2- 172.21.15.28:0/4180493997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d5b38b460 0x560d5b38f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:12.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.284+0000 7fc50f400640 1 --2- 172.21.15.28:0/4180493997 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d5b395b30 0x560d5b39cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:12.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.285+0000 7fc50ea00640 1 -- 172.21.15.28:0/4180493997 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d5b395b30 msgr2=0x560d5b39cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:12.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.285+0000 7fc50ea00640 1 --2- 172.21.15.28:0/4180493997 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d5b395b30 0x560d5b39cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:12.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.285+0000 7fc50ea00640 1 -- 172.21.15.28:0/4180493997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d5b38b460 msgr2=0x560d5b38f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:12.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.285+0000 7fc50ea00640 1 --2- 172.21.15.28:0/4180493997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d5b38b460 0x560d5b38f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:12.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.285+0000 7fc50ea00640 1 -- 172.21.15.28:0/4180493997 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d5b0cc7d0 con 0x560d5b38ab40 2024-04-23T14:56:12.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.285+0000 7fc50ea00640 1 --2- 172.21.15.28:0/4180493997 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d5b38ab40 0x560d5b38f220 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fc500031f00 tx=0x7fc500005750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60dccec7fda5c96a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:12.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.285+0000 7fc4fcc00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:56:12.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.285+0000 7fc4fcc00640 1 -- 172.21.15.28:0/4180493997 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc500052040 con 0x560d5b38ab40 2024-04-23T14:56:12.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.285+0000 7fc4fcc00640 1 -- 172.21.15.28:0/4180493997 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8542+0+0 (secure 0 0 0) 0x7fc500003e20 con 0x560d5b38ab40 2024-04-23T14:56:12.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.285+0000 7fc518b98740 1 -- 172.21.15.28:0/4180493997 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4f80051d0 con 0x560d5b38ab40 2024-04-23T14:56:12.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.285+0000 7fc4fcc00640 1 -- 172.21.15.28:0/4180493997 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fc500065050 con 0x560d5b38ab40 2024-04-23T14:56:12.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.285+0000 7fc518b98740 10 client.27140 fetch_fsmap learned FSMap version 20 2024-04-23T14:56:12.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.285+0000 7fc518b98740 1 -- 172.21.15.28:0/4180493997 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x560d5b39e460 con 0x560d5b38ab40 2024-04-23T14:56:12.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.286+0000 7fc4fcc00640 1 -- 172.21.15.28:0/4180493997 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4f80051d0 con 0x560d5b38ab40 2024-04-23T14:56:12.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.286+0000 7fc4fcc00640 1 -- 172.21.15.28:0/4180493997 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fc50004c040 con 0x560d5b38ab40 2024-04-23T14:56:12.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.286+0000 7fc518b98740 10 client.27140 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:56:12.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.286+0000 7fc518b98740 10 client.27140 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:56:12.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.286+0000 7fc518b98740 20 client.27140 populate_metadata read hostname 'smithi028' 2024-04-23T14:56:12.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.286+0000 7fc518b98740 1 --2- 172.21.15.28:0/4180493997 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560d5b40c030 0x560d5b40e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:12.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.286+0000 7fc518b98740 4 client.27140 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-23T14:56:12.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.286+0000 7fc518b98740 1 -- 172.21.15.28:0/4180493997 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560d5b2f7a60 con 0x560d5b40c030 2024-04-23T14:56:12.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.286+0000 7fc50e000640 1 --2- 172.21.15.28:0/4180493997 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560d5b40c030 0x560d5b40e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:12.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.287+0000 7fc50e000640 1 --2- 172.21.15.28:0/4180493997 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560d5b40c030 0x560d5b40e450 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9cea29ecaf26d202 server_cookie=1eb983b9d00b752 in_seq=0 out_seq=0 2024-04-23T14:56:12.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.287+0000 7fc4fcc00640 10 client.27140 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:56:12.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.287+0000 7fc4fcc00640 1 -- 172.21.15.28:0/4180493997 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560d5b2f7a60 con 0x560d5b40c030 2024-04-23T14:56:12.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.287+0000 7fc4fcc00640 10 client.27140 handle_command_reply: tid=0 2024-04-23T14:56:12.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.287+0000 7fc518b98740 1 client.27140 shutdown 2024-04-23T14:56:12.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.287+0000 7fc518b98740 1 -- 172.21.15.28:0/4180493997 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d5b38ab40 msgr2=0x560d5b38f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:12.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.287+0000 7fc518b98740 1 --2- 172.21.15.28:0/4180493997 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d5b38ab40 0x560d5b38f220 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fc500031f00 tx=0x7fc500005750 comp rx=0 tx=0).stop 2024-04-23T14:56:12.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.288+0000 7fc518b98740 1 -- 172.21.15.28:0/4180493997 shutdown_connections 2024-04-23T14:56:12.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.288+0000 7fc518b98740 1 --2- 172.21.15.28:0/4180493997 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x560d5b40c030 0x560d5b40e450 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:12.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.288+0000 7fc518b98740 1 --2- 172.21.15.28:0/4180493997 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560d5b395b30 0x560d5b39cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:12.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.288+0000 7fc518b98740 1 --2- 172.21.15.28:0/4180493997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d5b38b460 0x560d5b38f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:12.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.288+0000 7fc518b98740 1 --2- 172.21.15.28:0/4180493997 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d5b38ab40 0x560d5b38f220 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:12.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.288+0000 7fc4fcc00640 0 client.27140 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:56:12.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.288+0000 7fc518b98740 1 -- 172.21.15.28:0/4180493997 >> 172.21.15.28:0/4180493997 conn(0x560d5b386e60 msgr2=0x560d5b391920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:12.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.288+0000 7fc518b98740 1 -- 172.21.15.28:0/4180493997 shutdown_connections 2024-04-23T14:56:12.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.288+0000 7fc518b98740 1 -- 172.21.15.28:0/4180493997 wait complete. 2024-04-23T14:56:12.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.288+0000 7fc518b98740 20 client.27140 trim_cache size 0 max 16384 2024-04-23T14:56:12.293 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-23T14:56:12.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.289+0000 7fc50fe00640 1 -- 172.21.15.28:0/3848527927 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc4f40a3da0 msgr2=0x7fc4f40a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:12.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.289+0000 7fc50fe00640 1 --2- 172.21.15.28:0/3848527927 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc4f40a3da0 0x7fc4f40a6260 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc51006b2d0 tx=0x7fc508009760 comp rx=0 tx=0).stop 2024-04-23T14:56:12.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.289+0000 7fc50fe00640 1 -- 172.21.15.28:0/3848527927 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc510076e50 msgr2=0x7fc51019f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:12.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.289+0000 7fc50fe00640 1 --2- 172.21.15.28:0/3848527927 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc510076e50 0x7fc51019f350 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fc4f800e9f0 tx=0x7fc4f800eeb0 comp rx=0 tx=0).stop 2024-04-23T14:56:12.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.289+0000 7fc50fe00640 1 -- 172.21.15.28:0/3848527927 shutdown_connections 2024-04-23T14:56:12.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.289+0000 7fc50fe00640 1 --2- 172.21.15.28:0/3848527927 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc4f40a3da0 0x7fc4f40a6260 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:12.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.289+0000 7fc50fe00640 1 --2- 172.21.15.28:0/3848527927 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc510076e50 0x7fc51019f350 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:12.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.289+0000 7fc50fe00640 1 --2- 172.21.15.28:0/3848527927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc510112d50 0x7fc5101a36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:12.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.289+0000 7fc50fe00640 1 --2- 172.21.15.28:0/3848527927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc510076530 0x7fc51019ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:12.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.289+0000 7fc50fe00640 1 -- 172.21.15.28:0/3848527927 >> 172.21.15.28:0/3848527927 conn(0x7fc5100ffb40 msgr2=0x7fc510101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:12.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.289+0000 7fc50fe00640 1 -- 172.21.15.28:0/3848527927 shutdown_connections 2024-04-23T14:56:12.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:12.289+0000 7fc50fe00640 1 -- 172.21.15.28:0/3848527927 wait complete. 2024-04-23T14:56:12.541 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1037073 8.58 MB/sec execute 508 sec latency 18.566 ms 2024-04-23T14:56:13.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:12 smithi105 ceph-mon[32555]: pgmap v379: 97 pgs: 97 active+clean; 599 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 176 op/s 2024-04-23T14:56:13.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:12 smithi105 ceph-mon[32555]: osdmap e100: 12 total, 12 up, 12 in 2024-04-23T14:56:13.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:12 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2006177424' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:56:13.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:12 smithi038 ceph-mon[32348]: pgmap v379: 97 pgs: 97 active+clean; 599 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 176 op/s 2024-04-23T14:56:13.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:12 smithi038 ceph-mon[32348]: osdmap e100: 12 total, 12 up, 12 in 2024-04-23T14:56:13.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:12 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2006177424' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:56:13.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:12 smithi028 ceph-mon[27810]: pgmap v379: 97 pgs: 97 active+clean; 599 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 176 op/s 2024-04-23T14:56:13.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:12 smithi028 ceph-mon[27810]: osdmap e100: 12 total, 12 up, 12 in 2024-04-23T14:56:13.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:12 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2006177424' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:56:13.310 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-23T14:56:13.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:56:13.311 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-23T14:56:13.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.535+0000 7f9848e00640 1 -- 172.21.15.28:0/3890996099 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f984410d500 msgr2=0x7f984410d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:13.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.535+0000 7f9848e00640 1 --2- 172.21.15.28:0/3890996099 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f984410d500 0x7f984410d8e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9834009930 tx=0x7f983402a420 comp rx=0 tx=0).stop 2024-04-23T14:56:13.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.535+0000 7f9848e00640 1 -- 172.21.15.28:0/3890996099 shutdown_connections 2024-04-23T14:56:13.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.535+0000 7f9848e00640 1 --2- 172.21.15.28:0/3890996099 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9844108580 0x7f9844108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.535+0000 7f9848e00640 1 --2- 172.21.15.28:0/3890996099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9844108f40 0x7f98441153b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.535+0000 7f9848e00640 1 --2- 172.21.15.28:0/3890996099 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f984410d500 0x7f984410d8e0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.535+0000 7f9848e00640 1 -- 172.21.15.28:0/3890996099 >> 172.21.15.28:0/3890996099 conn(0x7f9844077e20 msgr2=0x7f9844106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:13.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.535+0000 7f9848e00640 1 -- 172.21.15.28:0/3890996099 shutdown_connections 2024-04-23T14:56:13.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.535+0000 7f9848e00640 1 -- 172.21.15.28:0/3890996099 wait complete. 2024-04-23T14:56:13.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.536+0000 7f9848e00640 1 Processor -- start 2024-04-23T14:56:13.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.536+0000 7f9848e00640 1 -- start start 2024-04-23T14:56:13.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.537+0000 7f9848e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9844108580 0x7f984419edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.537+0000 7f9848e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9844108f40 0x7f984419f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.537+0000 7f9848e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f984410d500 0x7f98441a3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.537+0000 7f9848e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9844118910 con 0x7f9844108f40 2024-04-23T14:56:13.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.537+0000 7f9848e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9844118790 con 0x7f9844108580 2024-04-23T14:56:13.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.537+0000 7f9848e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9844118a90 con 0x7f984410d500 2024-04-23T14:56:13.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.537+0000 7f9843400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9844108580 0x7f984419edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.537+0000 7f9843400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9844108580 0x7f984419edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:47824/0 (socket says 172.21.15.28:47824) 2024-04-23T14:56:13.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.537+0000 7f9843400640 1 -- 172.21.15.28:0/1670090391 learned_addr learned my addr 172.21.15.28:0/1670090391 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:13.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.537+0000 7f9842a00640 1 --2- 172.21.15.28:0/1670090391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9844108f40 0x7f984419f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.537+0000 7f9843e00640 1 --2- 172.21.15.28:0/1670090391 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f984410d500 0x7f98441a3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.538+0000 7f9843400640 1 -- 172.21.15.28:0/1670090391 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f984410d500 msgr2=0x7f98441a3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:13.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.538+0000 7f9843400640 1 --2- 172.21.15.28:0/1670090391 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f984410d500 0x7f98441a3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.538+0000 7f9843400640 1 -- 172.21.15.28:0/1670090391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9844108f40 msgr2=0x7f984419f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:13.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.538+0000 7f9843400640 1 --2- 172.21.15.28:0/1670090391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9844108f40 0x7f984419f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.538+0000 7f9843400640 1 -- 172.21.15.28:0/1670090391 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98441a3d50 con 0x7f9844108580 2024-04-23T14:56:13.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.538+0000 7f9843400640 1 --2- 172.21.15.28:0/1670090391 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9844108580 0x7f984419edb0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f983402a940 tx=0x7f9834003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:13.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.538+0000 7f983be00640 1 -- 172.21.15.28:0/1670090391 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f983403f070 con 0x7f9844108580 2024-04-23T14:56:13.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.538+0000 7f983be00640 1 -- 172.21.15.28:0/1670090391 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9834031c90 con 0x7f9844108580 2024-04-23T14:56:13.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.538+0000 7f983be00640 1 -- 172.21.15.28:0/1670090391 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9834035030 con 0x7f9844108580 2024-04-23T14:56:13.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.538+0000 7f9848e00640 1 -- 172.21.15.28:0/1670090391 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98441a3fe0 con 0x7f9844108580 2024-04-23T14:56:13.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.538+0000 7f9848e00640 1 -- 172.21.15.28:0/1670090391 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98441ab8d0 con 0x7f9844108580 2024-04-23T14:56:13.541 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1038045 8.58 MB/sec execute 509 sec latency 18.894 ms 2024-04-23T14:56:13.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.542+0000 7f983be00640 1 -- 172.21.15.28:0/1670090391 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f9834042070 con 0x7f9844108580 2024-04-23T14:56:13.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.542+0000 7f984d3ee740 1 Processor -- start 2024-04-23T14:56:13.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.543+0000 7f984d3ee740 1 -- start start 2024-04-23T14:56:13.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.543+0000 7f984d3ee740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55acca9e1270 0x55acca9e1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.543+0000 7f984d3ee740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acca9e1b90 0x55acca9ebd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.543+0000 7f984d3ee740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acca9ec260 0x55acca9ee650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.543+0000 7f984d3ee740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55acca946e70 con 0x55acca9e1b90 2024-04-23T14:56:13.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.543+0000 7f984d3ee740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55acca8ed380 con 0x55acca9e1270 2024-04-23T14:56:13.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.543+0000 7f984d3ee740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55acca942e30 con 0x55acca9ec260 2024-04-23T14:56:13.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.544+0000 7f9842a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acca9ec260 0x55acca9ee650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.543+0000 7f9843e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55acca9e1270 0x55acca9e1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.544+0000 7f9843e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55acca9e1270 0x55acca9e1650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:47832/0 (socket says 172.21.15.28:47832) 2024-04-23T14:56:13.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.544+0000 7f9843e00640 1 -- 172.21.15.28:0/2619773063 learned_addr learned my addr 172.21.15.28:0/2619773063 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:13.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.544+0000 7f9843400640 1 --2- 172.21.15.28:0/2619773063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acca9e1b90 0x55acca9ebd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.544+0000 7f983be00640 1 --2- 172.21.15.28:0/1670090391 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f98280a3da0 0x7f98280a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.544+0000 7f9842a00640 1 -- 172.21.15.28:0/2619773063 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55acca9e1270 msgr2=0x55acca9e1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:13.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.544+0000 7f9842a00640 1 --2- 172.21.15.28:0/2619773063 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55acca9e1270 0x55acca9e1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.544+0000 7f9842a00640 1 -- 172.21.15.28:0/2619773063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acca9e1b90 msgr2=0x55acca9ebd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:13.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.544+0000 7f9842a00640 1 --2- 172.21.15.28:0/2619773063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acca9e1b90 0x55acca9ebd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.544+0000 7f9842a00640 1 -- 172.21.15.28:0/2619773063 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55acca7217e0 con 0x55acca9ec260 2024-04-23T14:56:13.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.544+0000 7f9843400640 1 --2- 172.21.15.28:0/2619773063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acca9e1b90 0x55acca9ebd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:56:13.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.544+0000 7f983be00640 1 -- 172.21.15.28:0/1670090391 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8542+0+0 (secure 0 0 0) 0x7f98340f0e60 con 0x7f9844108580 2024-04-23T14:56:13.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.544+0000 7f9843e00640 1 --2- 172.21.15.28:0/1670090391 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f98280a3da0 0x7f98280a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.545+0000 7f9842a00640 1 --2- 172.21.15.28:0/2619773063 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acca9ec260 0x55acca9ee650 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9830009bd0 tx=0x7f9830011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ddb3fd153090756 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:13.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.545+0000 7f9838c00640 1 -- 172.21.15.28:0/2619773063 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9830004230 con 0x55acca9ec260 2024-04-23T14:56:13.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.545+0000 7f9838c00640 1 -- 172.21.15.28:0/2619773063 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9830009e40 con 0x55acca9ec260 2024-04-23T14:56:13.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.545+0000 7f9838c00640 1 -- 172.21.15.28:0/2619773063 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9830004db0 con 0x55acca9ec260 2024-04-23T14:56:13.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.546+0000 7f984d3ee740 1 -- 172.21.15.28:0/2619773063 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acca9ec260 msgr2=0x55acca9ee650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:13.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.546+0000 7f9843e00640 1 --2- 172.21.15.28:0/1670090391 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f98280a3da0 0x7f98280a6260 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f983c009fd0 tx=0x7f983c004a40 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:13.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.546+0000 7f984d3ee740 1 --2- 172.21.15.28:0/2619773063 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acca9ec260 0x55acca9ee650 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9830009bd0 tx=0x7f9830011810 comp rx=0 tx=0).stop 2024-04-23T14:56:13.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.546+0000 7f984d3ee740 1 -- 172.21.15.28:0/2619773063 shutdown_connections 2024-04-23T14:56:13.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.546+0000 7f984d3ee740 1 --2- 172.21.15.28:0/2619773063 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acca9ec260 0x55acca9ee650 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.546+0000 7f984d3ee740 1 --2- 172.21.15.28:0/2619773063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acca9e1b90 0x55acca9ebd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.546+0000 7f984d3ee740 1 --2- 172.21.15.28:0/2619773063 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55acca9e1270 0x55acca9e1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.546+0000 7f984d3ee740 1 -- 172.21.15.28:0/2619773063 >> 172.21.15.28:0/2619773063 conn(0x55acca9dd590 msgr2=0x55acca9dd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:13.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.546+0000 7f984d3ee740 1 -- 172.21.15.28:0/2619773063 shutdown_connections 2024-04-23T14:56:13.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.546+0000 7f984d3ee740 1 -- 172.21.15.28:0/2619773063 wait complete. 2024-04-23T14:56:13.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.547+0000 7f984d3ee740 1 -- start start 2024-04-23T14:56:13.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.547+0000 7f984d3ee740 1 Processor -- start 2024-04-23T14:56:13.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.547+0000 7f984d3ee740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acca9e1270 0x55acca9e9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.547+0000 7f984d3ee740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55acca9e1b90 0x55acca9e9690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.548+0000 7f984d3ee740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acca9ec260 0x55acca9e9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.548+0000 7f984d3ee740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55acca9ef050 con 0x55acca9e1270 2024-04-23T14:56:13.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.548+0000 7f984d3ee740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55acca900660 con 0x55acca9e1b90 2024-04-23T14:56:13.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.548+0000 7f984d3ee740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55acca9ef1d0 con 0x55acca9ec260 2024-04-23T14:56:13.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.548+0000 7f984d3ee740 10 client.0 _ll_register_callbacks cb 0x55acca8eeaa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:56:13.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.548+0000 7f9842a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55acca9e1b90 0x55acca9e9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.548+0000 7f9843400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acca9e1270 0x55acca9e9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.548+0000 7f9842a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55acca9e1b90 0x55acca9e9690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:47844/0 (socket says 172.21.15.28:47844) 2024-04-23T14:56:13.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.548+0000 7f9842a00640 1 -- 172.21.15.28:0/480595144 learned_addr learned my addr 172.21.15.28:0/480595144 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:13.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.548+0000 7f9843e00640 1 --2- 172.21.15.28:0/480595144 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acca9ec260 0x55acca9e9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.548+0000 7f9842a00640 1 -- 172.21.15.28:0/480595144 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acca9ec260 msgr2=0x55acca9e9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:13.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.548+0000 7f9842a00640 1 --2- 172.21.15.28:0/480595144 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acca9ec260 0x55acca9e9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.548+0000 7f9842a00640 1 -- 172.21.15.28:0/480595144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acca9e1270 msgr2=0x55acca9e9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:13.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.548+0000 7f9842a00640 1 --2- 172.21.15.28:0/480595144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acca9e1270 0x55acca9e9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.548+0000 7f9842a00640 1 -- 172.21.15.28:0/480595144 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55acca781f70 con 0x55acca9e1b90 2024-04-23T14:56:13.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.548+0000 7f9843400640 1 --2- 172.21.15.28:0/480595144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acca9e1270 0x55acca9e9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:56:13.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.549+0000 7f9842a00640 1 --2- 172.21.15.28:0/480595144 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55acca9e1b90 0x55acca9e9690 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f9830003820 tx=0x7f9830003850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5183f4ee16ad2d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:13.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.549+0000 7f982d600640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:56:13.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.549+0000 7f982d600640 1 -- 172.21.15.28:0/480595144 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9830027020 con 0x55acca9e1b90 2024-04-23T14:56:13.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.549+0000 7f982d600640 1 -- 172.21.15.28:0/480595144 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8542+0+0 (secure 0 0 0) 0x7f9830003bf0 con 0x55acca9e1b90 2024-04-23T14:56:13.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.549+0000 7f984d3ee740 1 -- 172.21.15.28:0/480595144 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55acca954890 con 0x55acca9e1b90 2024-04-23T14:56:13.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.549+0000 7f982d600640 1 -- 172.21.15.28:0/480595144 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f9830032050 con 0x55acca9e1b90 2024-04-23T14:56:13.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.549+0000 7f984d3ee740 10 client.27152 fetch_fsmap learned FSMap version 20 2024-04-23T14:56:13.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.549+0000 7f984d3ee740 1 -- 172.21.15.28:0/480595144 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55acca619170 con 0x55acca9e1b90 2024-04-23T14:56:13.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.550+0000 7f982d600640 1 -- 172.21.15.28:0/480595144 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55acca954890 con 0x55acca9e1b90 2024-04-23T14:56:13.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.550+0000 7f982d600640 1 -- 172.21.15.28:0/480595144 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f9830018070 con 0x55acca9e1b90 2024-04-23T14:56:13.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.550+0000 7f984d3ee740 10 client.27152 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:56:13.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.550+0000 7f984d3ee740 10 client.27152 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:56:13.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.550+0000 7f984d3ee740 20 client.27152 populate_metadata read hostname 'smithi028' 2024-04-23T14:56:13.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.550+0000 7f984d3ee740 1 --2- 172.21.15.28:0/480595144 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55acca9fc430 0x55acca9fe820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.550+0000 7f984d3ee740 4 client.27152 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:56:13.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.550+0000 7f984d3ee740 1 -- 172.21.15.28:0/480595144 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55acca94e700 con 0x55acca9fc430 2024-04-23T14:56:13.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.550+0000 7f9843400640 1 --2- 172.21.15.28:0/480595144 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55acca9fc430 0x55acca9fe820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.551+0000 7f9843400640 1 --2- 172.21.15.28:0/480595144 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55acca9fc430 0x55acca9fe820 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95b4de44d407f40d server_cookie=ff53ce6cb0df4a8c in_seq=0 out_seq=0 2024-04-23T14:56:13.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.551+0000 7f982d600640 10 client.27152 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:56:13.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.551+0000 7f982d600640 1 -- 172.21.15.28:0/480595144 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55acca94e700 con 0x55acca9fc430 2024-04-23T14:56:13.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.551+0000 7f982d600640 10 client.27152 handle_command_reply: tid=0 2024-04-23T14:56:13.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.552+0000 7f984d3ee740 1 client.27152 shutdown 2024-04-23T14:56:13.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.552+0000 7f984d3ee740 1 -- 172.21.15.28:0/480595144 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55acca9e1b90 msgr2=0x55acca9e9690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:13.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.552+0000 7f984d3ee740 1 --2- 172.21.15.28:0/480595144 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55acca9e1b90 0x55acca9e9690 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f9830003820 tx=0x7f9830003850 comp rx=0 tx=0).stop 2024-04-23T14:56:13.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.552+0000 7f984d3ee740 1 -- 172.21.15.28:0/480595144 shutdown_connections 2024-04-23T14:56:13.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.552+0000 7f984d3ee740 1 --2- 172.21.15.28:0/480595144 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55acca9fc430 0x55acca9fe820 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.552+0000 7f984d3ee740 1 --2- 172.21.15.28:0/480595144 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55acca9e1b90 0x55acca9e9690 secure :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f9830003820 tx=0x7f9830003850 comp rx=0 tx=0).stop 2024-04-23T14:56:13.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.552+0000 7f984d3ee740 1 --2- 172.21.15.28:0/480595144 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acca9ec260 0x55acca9e9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.552+0000 7f984d3ee740 1 --2- 172.21.15.28:0/480595144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acca9e1270 0x55acca9e9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.552+0000 7f982d600640 0 client.27152 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:56:13.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.552+0000 7f984d3ee740 1 -- 172.21.15.28:0/480595144 >> 172.21.15.28:0/480595144 conn(0x55acca9dd590 msgr2=0x55acca9e3290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:13.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.552+0000 7f984d3ee740 1 -- 172.21.15.28:0/480595144 shutdown_connections 2024-04-23T14:56:13.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.552+0000 7f984d3ee740 1 -- 172.21.15.28:0/480595144 wait complete. 2024-04-23T14:56:13.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.552+0000 7f984d3ee740 20 client.27152 trim_cache size 0 max 16384 2024-04-23T14:56:13.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.583+0000 7f984d3ee740 1 Processor -- start 2024-04-23T14:56:13.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.584+0000 7f984d3ee740 1 -- start start 2024-04-23T14:56:13.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.584+0000 7f984d3ee740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acca9e1270 0x55acca9e8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.584+0000 7f984d3ee740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55acca9ec260 0x55acca9f7f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.584+0000 7f984d3ee740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acca9eb990 0x55accaa03f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.584+0000 7f984d3ee740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f98441b2170 con 0x55acca9e1270 2024-04-23T14:56:13.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.584+0000 7f984d3ee740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f98441b49d0 con 0x55acca9ec260 2024-04-23T14:56:13.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.584+0000 7f984d3ee740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f98441b27c0 con 0x55acca9eb990 2024-04-23T14:56:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.585+0000 7f9842a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55acca9ec260 0x55acca9f7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.585+0000 7f9843e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acca9eb990 0x55accaa03f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.585+0000 7f9843e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acca9eb990 0x55accaa03f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:47400/0 (socket says 172.21.15.28:47400) 2024-04-23T14:56:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.585+0000 7f9843e00640 1 -- 172.21.15.28:0/2349038863 learned_addr learned my addr 172.21.15.28:0/2349038863 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.585+0000 7f9843400640 1 --2- 172.21.15.28:0/2349038863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acca9e1270 0x55acca9e8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.586+0000 7f9843e00640 1 -- 172.21.15.28:0/2349038863 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55acca9ec260 msgr2=0x55acca9f7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.586+0000 7f9843e00640 1 --2- 172.21.15.28:0/2349038863 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55acca9ec260 0x55acca9f7f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.586+0000 7f9843e00640 1 -- 172.21.15.28:0/2349038863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acca9e1270 msgr2=0x55acca9e8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:13.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.586+0000 7f9843e00640 1 --2- 172.21.15.28:0/2349038863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acca9e1270 0x55acca9e8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.586+0000 7f9843e00640 1 -- 172.21.15.28:0/2349038863 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55acca9e3a10 con 0x55acca9eb990 2024-04-23T14:56:13.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.588+0000 7f9843e00640 1 --2- 172.21.15.28:0/2349038863 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acca9eb990 0x55accaa03f60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f983c010780 tx=0x7f983c016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a318413e0e684a68 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:13.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.589+0000 7f982fe00640 1 -- 172.21.15.28:0/2349038863 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f983c011800 con 0x55acca9eb990 2024-04-23T14:56:13.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.589+0000 7f982fe00640 1 -- 172.21.15.28:0/2349038863 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f983c020400 con 0x55acca9eb990 2024-04-23T14:56:13.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.589+0000 7f982fe00640 1 -- 172.21.15.28:0/2349038863 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f983c00f620 con 0x55acca9eb990 2024-04-23T14:56:13.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.591+0000 7f984d3ee740 1 -- 172.21.15.28:0/2349038863 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acca9eb990 msgr2=0x55accaa03f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:13.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.591+0000 7f984d3ee740 1 --2- 172.21.15.28:0/2349038863 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acca9eb990 0x55accaa03f60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f983c010780 tx=0x7f983c016000 comp rx=0 tx=0).stop 2024-04-23T14:56:13.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.591+0000 7f984d3ee740 1 -- 172.21.15.28:0/2349038863 shutdown_connections 2024-04-23T14:56:13.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.591+0000 7f984d3ee740 1 --2- 172.21.15.28:0/2349038863 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55acca9ec260 0x55acca9f7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.591+0000 7f984d3ee740 1 --2- 172.21.15.28:0/2349038863 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acca9eb990 0x55accaa03f60 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.591+0000 7f984d3ee740 1 --2- 172.21.15.28:0/2349038863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acca9e1270 0x55acca9e8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.591+0000 7f984d3ee740 1 -- 172.21.15.28:0/2349038863 >> 172.21.15.28:0/2349038863 conn(0x55acca9dd590 msgr2=0x55acca9e99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:13.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.592+0000 7f984d3ee740 1 -- 172.21.15.28:0/2349038863 shutdown_connections 2024-04-23T14:56:13.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.592+0000 7f984d3ee740 1 -- 172.21.15.28:0/2349038863 wait complete. 2024-04-23T14:56:13.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.592+0000 7f984d3ee740 1 -- start start 2024-04-23T14:56:13.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.593+0000 7f984d3ee740 1 Processor -- start 2024-04-23T14:56:13.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.594+0000 7f984d3ee740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55acca9e1270 0x55accaa5e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.594+0000 7f984d3ee740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acca9eb990 0x55acca9f4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.594+0000 7f984d3ee740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acca9ec260 0x55accaa5b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.594+0000 7f984d3ee740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f98441b2e90 con 0x55acca9ec260 2024-04-23T14:56:13.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.594+0000 7f984d3ee740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f98441b4300 con 0x55acca9e1270 2024-04-23T14:56:13.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.594+0000 7f984d3ee740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f98441b3560 con 0x55acca9eb990 2024-04-23T14:56:13.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.594+0000 7f984d3ee740 10 client.0 _ll_register_callbacks cb 0x7f983c009ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:56:13.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.594+0000 7f9843400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55acca9e1270 0x55accaa5e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.594+0000 7f9843400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55acca9e1270 0x55accaa5e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:47858/0 (socket says 172.21.15.28:47858) 2024-04-23T14:56:13.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.594+0000 7f9843400640 1 -- 172.21.15.28:0/615599116 learned_addr learned my addr 172.21.15.28:0/615599116 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:13.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.595+0000 7f9842a00640 1 --2- 172.21.15.28:0/615599116 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acca9eb990 0x55acca9f4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.595+0000 7f9843e00640 1 --2- 172.21.15.28:0/615599116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acca9ec260 0x55accaa5b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.595+0000 7f9842a00640 1 -- 172.21.15.28:0/615599116 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55acca9e1270 msgr2=0x55accaa5e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:13.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.595+0000 7f9842a00640 1 --2- 172.21.15.28:0/615599116 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55acca9e1270 0x55accaa5e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.595+0000 7f9842a00640 1 -- 172.21.15.28:0/615599116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acca9ec260 msgr2=0x55accaa5b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:13.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.595+0000 7f9842a00640 1 --2- 172.21.15.28:0/615599116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acca9ec260 0x55accaa5b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.595+0000 7f9842a00640 1 -- 172.21.15.28:0/615599116 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55acca9f4cf0 con 0x55acca9eb990 2024-04-23T14:56:13.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.595+0000 7f9843e00640 1 --2- 172.21.15.28:0/615599116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acca9ec260 0x55accaa5b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:56:13.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.595+0000 7f9842a00640 1 --2- 172.21.15.28:0/615599116 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acca9eb990 0x55acca9f4420 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x55acca9ea1e0 tx=0x7f9830004350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9fb7015cd0c2243 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:13.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.596+0000 7f982d600640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:56:13.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.597+0000 7f982d600640 1 -- 172.21.15.28:0/615599116 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9830015070 con 0x55acca9eb990 2024-04-23T14:56:13.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.597+0000 7f982d600640 1 -- 172.21.15.28:0/615599116 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8542+0+0 (secure 0 0 0) 0x7f98300038c0 con 0x55acca9eb990 2024-04-23T14:56:13.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.597+0000 7f984d3ee740 1 -- 172.21.15.28:0/615599116 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f983c020400 con 0x55acca9eb990 2024-04-23T14:56:13.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.598+0000 7f982d600640 1 -- 172.21.15.28:0/615599116 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f9830027030 con 0x55acca9eb990 2024-04-23T14:56:13.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.598+0000 7f984d3ee740 10 client.27142 fetch_fsmap learned FSMap version 20 2024-04-23T14:56:13.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.598+0000 7f984d3ee740 1 -- 172.21.15.28:0/615599116 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55accaa73e20 con 0x55acca9eb990 2024-04-23T14:56:13.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.598+0000 7f982d600640 1 -- 172.21.15.28:0/615599116 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f983c020400 con 0x55acca9eb990 2024-04-23T14:56:13.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.598+0000 7f982d600640 1 -- 172.21.15.28:0/615599116 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f9830018040 con 0x55acca9eb990 2024-04-23T14:56:13.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.599+0000 7f984d3ee740 10 client.27142 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:56:13.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.599+0000 7f984d3ee740 10 client.27142 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:56:13.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.599+0000 7f984d3ee740 20 client.27142 populate_metadata read hostname 'smithi028' 2024-04-23T14:56:13.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.599+0000 7f984d3ee740 1 --2- 172.21.15.28:0/615599116 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55accaa744b0 0x55accaa768d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.599+0000 7f984d3ee740 4 client.27142 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-23T14:56:13.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.599+0000 7f984d3ee740 1 -- 172.21.15.28:0/615599116 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55acca8ca980 con 0x55accaa744b0 2024-04-23T14:56:13.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.599+0000 7f9843400640 1 --2- 172.21.15.28:0/615599116 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55accaa744b0 0x55accaa768d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.599+0000 7f9843400640 1 --2- 172.21.15.28:0/615599116 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55accaa744b0 0x55accaa768d0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff11fa98ee729300 server_cookie=c714e16ebb681223 in_seq=0 out_seq=0 2024-04-23T14:56:13.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.599+0000 7f982d600640 10 client.27142 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:56:13.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.601+0000 7f982d600640 1 -- 172.21.15.28:0/615599116 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55acca8ca980 con 0x55accaa744b0 2024-04-23T14:56:13.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.601+0000 7f982d600640 10 client.27142 handle_command_reply: tid=0 2024-04-23T14:56:13.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.602+0000 7f984d3ee740 1 client.27142 shutdown 2024-04-23T14:56:13.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.602+0000 7f984d3ee740 1 -- 172.21.15.28:0/615599116 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acca9eb990 msgr2=0x55acca9f4420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:13.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.602+0000 7f984d3ee740 1 --2- 172.21.15.28:0/615599116 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acca9eb990 0x55acca9f4420 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x55acca9ea1e0 tx=0x7f9830004350 comp rx=0 tx=0).stop 2024-04-23T14:56:13.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.602+0000 7f984d3ee740 1 -- 172.21.15.28:0/615599116 shutdown_connections 2024-04-23T14:56:13.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.602+0000 7f984d3ee740 1 --2- 172.21.15.28:0/615599116 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55accaa744b0 0x55accaa768d0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.602+0000 7f984d3ee740 1 --2- 172.21.15.28:0/615599116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acca9ec260 0x55accaa5b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.602+0000 7f982d600640 0 client.27142 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:56:13.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.602+0000 7f984d3ee740 1 --2- 172.21.15.28:0/615599116 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55acca9eb990 0x55acca9f4420 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.602+0000 7f984d3ee740 1 --2- 172.21.15.28:0/615599116 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55acca9e1270 0x55accaa5e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.602+0000 7f984d3ee740 1 -- 172.21.15.28:0/615599116 >> 172.21.15.28:0/615599116 conn(0x55acca9dd590 msgr2=0x55acca9ea980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:13.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.602+0000 7f984d3ee740 1 -- 172.21.15.28:0/615599116 shutdown_connections 2024-04-23T14:56:13.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.602+0000 7f984d3ee740 1 -- 172.21.15.28:0/615599116 wait complete. 2024-04-23T14:56:13.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.602+0000 7f984d3ee740 20 client.27142 trim_cache size 0 max 16384 2024-04-23T14:56:13.604 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:56:13.604 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-23T14:56:13.604 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "445eefb4-d9e5-4d3a-a5d6-aab2909624a4", 2024-04-23T14:56:13.604 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-23T14:56:13.604 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:56:13.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.603+0000 7f9848e00640 1 -- 172.21.15.28:0/1670090391 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f98280a3da0 msgr2=0x7f98280a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:13.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.603+0000 7f9848e00640 1 --2- 172.21.15.28:0/1670090391 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f98280a3da0 0x7f98280a6260 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f983c009fd0 tx=0x7f983c004a40 comp rx=0 tx=0).stop 2024-04-23T14:56:13.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.603+0000 7f9848e00640 1 -- 172.21.15.28:0/1670090391 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9844108580 msgr2=0x7f984419edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:13.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.603+0000 7f9848e00640 1 --2- 172.21.15.28:0/1670090391 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9844108580 0x7f984419edb0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f983402a940 tx=0x7f9834003b90 comp rx=0 tx=0).stop 2024-04-23T14:56:13.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.603+0000 7f9848e00640 1 -- 172.21.15.28:0/1670090391 shutdown_connections 2024-04-23T14:56:13.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.603+0000 7f9848e00640 1 --2- 172.21.15.28:0/1670090391 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f98280a3da0 0x7f98280a6260 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.603+0000 7f9848e00640 1 --2- 172.21.15.28:0/1670090391 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f984410d500 0x7f98441a3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.603+0000 7f9848e00640 1 --2- 172.21.15.28:0/1670090391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9844108f40 0x7f984419f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.603+0000 7f9848e00640 1 --2- 172.21.15.28:0/1670090391 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9844108580 0x7f984419edb0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.603+0000 7f9848e00640 1 -- 172.21.15.28:0/1670090391 >> 172.21.15.28:0/1670090391 conn(0x7f9844077e20 msgr2=0x7f9844078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:13.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.603+0000 7f9848e00640 1 -- 172.21.15.28:0/1670090391 shutdown_connections 2024-04-23T14:56:13.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.603+0000 7f9848e00640 1 -- 172.21.15.28:0/1670090391 wait complete. 2024-04-23T14:56:13.623 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:56:13.623 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:56:13.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.813+0000 7f372be00640 1 -- 172.21.15.28:0/1017662230 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f372c076530 msgr2=0x7f372c076910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:13.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.813+0000 7f372be00640 1 --2- 172.21.15.28:0/1017662230 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f372c076530 0x7f372c076910 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f371c009930 tx=0x7f371c02a420 comp rx=0 tx=0).stop 2024-04-23T14:56:13.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.813+0000 7f372be00640 1 -- 172.21.15.28:0/1017662230 shutdown_connections 2024-04-23T14:56:13.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.813+0000 7f372be00640 1 --2- 172.21.15.28:0/1017662230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f372c112d40 0x7f372c115130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.813+0000 7f372be00640 1 --2- 172.21.15.28:0/1017662230 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f372c076e50 0x7f372c112800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.813+0000 7f372be00640 1 --2- 172.21.15.28:0/1017662230 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f372c076530 0x7f372c076910 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.813+0000 7f372be00640 1 -- 172.21.15.28:0/1017662230 >> 172.21.15.28:0/1017662230 conn(0x7f372c0ffb70 msgr2=0x7f372c101f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:13.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.813+0000 7f372be00640 1 -- 172.21.15.28:0/1017662230 shutdown_connections 2024-04-23T14:56:13.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.813+0000 7f372be00640 1 -- 172.21.15.28:0/1017662230 wait complete. 2024-04-23T14:56:13.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.814+0000 7f372be00640 1 Processor -- start 2024-04-23T14:56:13.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.814+0000 7f372be00640 1 -- start start 2024-04-23T14:56:13.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.814+0000 7f372be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f372c076530 0x7f372c19edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.815+0000 7f372be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f372c076e50 0x7f372c19f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.815+0000 7f372be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f372c112d40 0x7f372c1a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.815+0000 7f372be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f372c1188e0 con 0x7f372c112d40 2024-04-23T14:56:13.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.815+0000 7f372be00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f372c118760 con 0x7f372c076e50 2024-04-23T14:56:13.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.815+0000 7f372be00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f372c118a60 con 0x7f372c076530 2024-04-23T14:56:13.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.815+0000 7f372aa00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f372c076530 0x7f372c19edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.815+0000 7f372aa00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f372c076530 0x7f372c19edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:47430/0 (socket says 172.21.15.28:47430) 2024-04-23T14:56:13.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.815+0000 7f372aa00640 1 -- 172.21.15.28:0/3497848827 learned_addr learned my addr 172.21.15.28:0/3497848827 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:13.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.815+0000 7f372b400640 1 --2- 172.21.15.28:0/3497848827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f372c112d40 0x7f372c1a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.815+0000 7f3723e00640 1 --2- 172.21.15.28:0/3497848827 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f372c076e50 0x7f372c19f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.815+0000 7f372aa00640 1 -- 172.21.15.28:0/3497848827 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f372c076e50 msgr2=0x7f372c19f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:13.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.815+0000 7f372aa00640 1 --2- 172.21.15.28:0/3497848827 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f372c076e50 0x7f372c19f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.815+0000 7f372aa00640 1 -- 172.21.15.28:0/3497848827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f372c112d40 msgr2=0x7f372c1a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:13.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.815+0000 7f372aa00640 1 --2- 172.21.15.28:0/3497848827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f372c112d40 0x7f372c1a3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.815+0000 7f372aa00640 1 -- 172.21.15.28:0/3497848827 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f372c1a3db0 con 0x7f372c076530 2024-04-23T14:56:13.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.816+0000 7f372aa00640 1 --2- 172.21.15.28:0/3497848827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f372c076530 0x7f372c19edc0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f371c02a940 tx=0x7f371c003b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:13.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.816+0000 7f3730a00640 1 -- 172.21.15.28:0/3497848827 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f371c03f070 con 0x7f372c076530 2024-04-23T14:56:13.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.816+0000 7f3730a00640 1 -- 172.21.15.28:0/3497848827 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f371c031c90 con 0x7f372c076530 2024-04-23T14:56:13.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.816+0000 7f3730a00640 1 -- 172.21.15.28:0/3497848827 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f371c035030 con 0x7f372c076530 2024-04-23T14:56:13.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.816+0000 7f372be00640 1 -- 172.21.15.28:0/3497848827 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f372c1a4040 con 0x7f372c076530 2024-04-23T14:56:13.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.816+0000 7f372be00640 1 -- 172.21.15.28:0/3497848827 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f372c1ab930 con 0x7f372c076530 2024-04-23T14:56:13.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.819+0000 7f3734db5740 1 Processor -- start 2024-04-23T14:56:13.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.819+0000 7f3730a00640 1 -- 172.21.15.28:0/3497848827 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f371c042070 con 0x7f372c076530 2024-04-23T14:56:13.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.820+0000 7f3734db5740 1 -- start start 2024-04-23T14:56:13.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.820+0000 7f3734db5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e57d43a30 0x555e57d43e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.820+0000 7f3734db5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e57d44350 0x555e57d4e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.820+0000 7f3734db5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e57d4ea20 0x555e57d50e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.820+0000 7f3734db5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555e57ca91f0 con 0x555e57d43a30 2024-04-23T14:56:13.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.820+0000 7f3734db5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555e57c4f350 con 0x555e57d4ea20 2024-04-23T14:56:13.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.820+0000 7f3734db5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x555e57ca51b0 con 0x555e57d44350 2024-04-23T14:56:13.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.820+0000 7f3723e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e57d4ea20 0x555e57d50e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.820+0000 7f372b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e57d43a30 0x555e57d43e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.820+0000 7f3723e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e57d4ea20 0x555e57d50e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:47896/0 (socket says 172.21.15.28:47896) 2024-04-23T14:56:13.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.820+0000 7f3723e00640 1 -- 172.21.15.28:0/4266461535 learned_addr learned my addr 172.21.15.28:0/4266461535 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:13.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.820+0000 7f372aa00640 1 --2- 172.21.15.28:0/4266461535 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e57d44350 0x555e57d4e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.821+0000 7f3723e00640 1 -- 172.21.15.28:0/4266461535 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e57d44350 msgr2=0x555e57d4e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:13.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.821+0000 7f3723e00640 1 --2- 172.21.15.28:0/4266461535 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e57d44350 0x555e57d4e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.821+0000 7f3723e00640 1 -- 172.21.15.28:0/4266461535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e57d43a30 msgr2=0x555e57d43e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:13.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.821+0000 7f3723e00640 1 --2- 172.21.15.28:0/4266461535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e57d43a30 0x555e57d43e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.821+0000 7f3723e00640 1 -- 172.21.15.28:0/4266461535 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e57ca3f50 con 0x555e57d4ea20 2024-04-23T14:56:13.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.821+0000 7f3723e00640 1 --2- 172.21.15.28:0/4266461535 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e57d4ea20 0x555e57d50e10 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f3714009c90 tx=0x7f3714011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0b0e7e7b9d1d58f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:13.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.821+0000 7f3730a00640 1 --2- 172.21.15.28:0/3497848827 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f37100a3da0 0x7f37100a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.821+0000 7f371be00640 1 -- 172.21.15.28:0/4266461535 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f371401e070 con 0x555e57d4ea20 2024-04-23T14:56:13.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.821+0000 7f371be00640 1 -- 172.21.15.28:0/4266461535 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3714019b10 con 0x555e57d4ea20 2024-04-23T14:56:13.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.821+0000 7f371be00640 1 -- 172.21.15.28:0/4266461535 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f371400eaf0 con 0x555e57d4ea20 2024-04-23T14:56:13.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.821+0000 7f3730a00640 1 -- 172.21.15.28:0/3497848827 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8542+0+0 (secure 0 0 0) 0x7f371c0f0de0 con 0x7f372c076530 2024-04-23T14:56:13.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.822+0000 7f372b400640 1 --2- 172.21.15.28:0/3497848827 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f37100a3da0 0x7f37100a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.822+0000 7f3734db5740 1 -- 172.21.15.28:0/4266461535 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e57d4ea20 msgr2=0x555e57d50e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:13.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.822+0000 7f3734db5740 1 --2- 172.21.15.28:0/4266461535 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e57d4ea20 0x555e57d50e10 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f3714009c90 tx=0x7f3714011810 comp rx=0 tx=0).stop 2024-04-23T14:56:13.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.822+0000 7f3734db5740 1 -- 172.21.15.28:0/4266461535 shutdown_connections 2024-04-23T14:56:13.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.822+0000 7f3734db5740 1 --2- 172.21.15.28:0/4266461535 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e57d4ea20 0x555e57d50e10 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.822+0000 7f3734db5740 1 --2- 172.21.15.28:0/4266461535 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e57d44350 0x555e57d4e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.822+0000 7f3734db5740 1 --2- 172.21.15.28:0/4266461535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e57d43a30 0x555e57d43e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.822+0000 7f3734db5740 1 -- 172.21.15.28:0/4266461535 >> 172.21.15.28:0/4266461535 conn(0x555e57d3fd50 msgr2=0x555e57d40130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:13.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.822+0000 7f372b400640 1 --2- 172.21.15.28:0/3497848827 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f37100a3da0 0x7f37100a6260 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f372c06b460 tx=0x7f3724009500 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:13.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.822+0000 7f3734db5740 1 -- 172.21.15.28:0/4266461535 shutdown_connections 2024-04-23T14:56:13.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.822+0000 7f3734db5740 1 -- 172.21.15.28:0/4266461535 wait complete. 2024-04-23T14:56:13.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.823+0000 7f3734db5740 1 -- start start 2024-04-23T14:56:13.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.823+0000 7f3734db5740 1 Processor -- start 2024-04-23T14:56:13.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.823+0000 7f3734db5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e57d43a30 0x555e57d4b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.823+0000 7f3734db5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e57d44350 0x555e57d4bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.823+0000 7f3734db5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e57d4ea20 0x555e57d4c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.823+0000 7f3734db5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555e57d51810 con 0x555e57d4ea20 2024-04-23T14:56:13.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.823+0000 7f3734db5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555e57c62690 con 0x555e57d44350 2024-04-23T14:56:13.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.823+0000 7f3734db5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x555e57d51990 con 0x555e57d43a30 2024-04-23T14:56:13.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.823+0000 7f3734db5740 10 client.0 _ll_register_callbacks cb 0x555e57c9e650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:56:13.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.824+0000 7f372aa00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e57d43a30 0x555e57d4b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.824+0000 7f3723e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e57d44350 0x555e57d4bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.824+0000 7f372aa00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e57d43a30 0x555e57d4b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:47452/0 (socket says 172.21.15.28:47452) 2024-04-23T14:56:13.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.824+0000 7f372aa00640 1 -- 172.21.15.28:0/1544313842 learned_addr learned my addr 172.21.15.28:0/1544313842 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:13.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.824+0000 7f372b400640 1 --2- 172.21.15.28:0/1544313842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e57d4ea20 0x555e57d4c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.824+0000 7f3723e00640 1 -- 172.21.15.28:0/1544313842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e57d43a30 msgr2=0x555e57d4b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:13.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.824+0000 7f3723e00640 1 --2- 172.21.15.28:0/1544313842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e57d43a30 0x555e57d4b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.824+0000 7f3723e00640 1 -- 172.21.15.28:0/1544313842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e57d4ea20 msgr2=0x555e57d4c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:13.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.824+0000 7f3723e00640 1 --2- 172.21.15.28:0/1544313842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e57d4ea20 0x555e57d4c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.824+0000 7f3723e00640 1 -- 172.21.15.28:0/1544313842 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e57c61920 con 0x555e57d44350 2024-04-23T14:56:13.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.824+0000 7f372aa00640 1 --2- 172.21.15.28:0/1544313842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e57d43a30 0x555e57d4b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:56:13.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.824+0000 7f3723e00640 1 --2- 172.21.15.28:0/1544313842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e57d44350 0x555e57d4bec0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f3714000c00 tx=0x7f3714019640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=540b9481b858d063 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:13.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.824+0000 7f3718c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:56:13.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.825+0000 7f3718c00640 1 -- 172.21.15.28:0/1544313842 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3714029070 con 0x555e57d44350 2024-04-23T14:56:13.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.825+0000 7f3734db5740 1 -- 172.21.15.28:0/1544313842 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3714019b10 con 0x555e57d44350 2024-04-23T14:56:13.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.825+0000 7f3718c00640 1 -- 172.21.15.28:0/1544313842 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8542+0+0 (secure 0 0 0) 0x7f3714004020 con 0x555e57d44350 2024-04-23T14:56:13.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.825+0000 7f3718c00640 1 -- 172.21.15.28:0/1544313842 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f3714032050 con 0x555e57d44350 2024-04-23T14:56:13.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.825+0000 7f3734db5740 10 client.27170 fetch_fsmap learned FSMap version 20 2024-04-23T14:56:13.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.825+0000 7f3734db5740 1 -- 172.21.15.28:0/1544313842 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x555e57a857d0 con 0x555e57d44350 2024-04-23T14:56:13.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.825+0000 7f3718c00640 1 -- 172.21.15.28:0/1544313842 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3714019b10 con 0x555e57d44350 2024-04-23T14:56:13.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.826+0000 7f3718c00640 1 -- 172.21.15.28:0/1544313842 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f371402c020 con 0x555e57d44350 2024-04-23T14:56:13.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.826+0000 7f3734db5740 10 client.27170 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:56:13.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.826+0000 7f3734db5740 10 client.27170 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:56:13.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.826+0000 7f3734db5740 20 client.27170 populate_metadata read hostname 'smithi028' 2024-04-23T14:56:13.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.826+0000 7f3734db5740 1 --2- 172.21.15.28:0/1544313842 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x555e57d5ecb0 0x555e57d610a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.826+0000 7f3734db5740 4 client.27170 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:56:13.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.826+0000 7f3734db5740 1 -- 172.21.15.28:0/1544313842 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555e57cb6c10 con 0x555e57d5ecb0 2024-04-23T14:56:13.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.826+0000 7f372aa00640 1 --2- 172.21.15.28:0/1544313842 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x555e57d5ecb0 0x555e57d610a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.827+0000 7f372aa00640 1 --2- 172.21.15.28:0/1544313842 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x555e57d5ecb0 0x555e57d610a0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50ccd819adccdd59 server_cookie=d82a8aa84143bc6e in_seq=0 out_seq=0 2024-04-23T14:56:13.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.827+0000 7f3718c00640 10 client.27170 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:56:13.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.827+0000 7f3718c00640 1 -- 172.21.15.28:0/1544313842 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555e57cb6c10 con 0x555e57d5ecb0 2024-04-23T14:56:13.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.827+0000 7f3718c00640 10 client.27170 handle_command_reply: tid=0 2024-04-23T14:56:13.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.827+0000 7f3734db5740 1 client.27170 shutdown 2024-04-23T14:56:13.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.828+0000 7f3734db5740 1 -- 172.21.15.28:0/1544313842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e57d44350 msgr2=0x555e57d4bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:13.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.828+0000 7f3734db5740 1 --2- 172.21.15.28:0/1544313842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e57d44350 0x555e57d4bec0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f3714000c00 tx=0x7f3714019640 comp rx=0 tx=0).stop 2024-04-23T14:56:13.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.828+0000 7f3734db5740 1 -- 172.21.15.28:0/1544313842 shutdown_connections 2024-04-23T14:56:13.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.828+0000 7f3734db5740 1 --2- 172.21.15.28:0/1544313842 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x555e57d5ecb0 0x555e57d610a0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.828+0000 7f3734db5740 1 --2- 172.21.15.28:0/1544313842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e57d44350 0x555e57d4bec0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.828+0000 7f3734db5740 1 --2- 172.21.15.28:0/1544313842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e57d4ea20 0x555e57d4c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.828+0000 7f3734db5740 1 --2- 172.21.15.28:0/1544313842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e57d43a30 0x555e57d4b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.828+0000 7f3718c00640 0 client.27170 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:56:13.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.828+0000 7f3734db5740 1 -- 172.21.15.28:0/1544313842 >> 172.21.15.28:0/1544313842 conn(0x555e57d3fd50 msgr2=0x555e57d3efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:13.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.828+0000 7f3734db5740 1 -- 172.21.15.28:0/1544313842 shutdown_connections 2024-04-23T14:56:13.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.828+0000 7f3734db5740 1 -- 172.21.15.28:0/1544313842 wait complete. 2024-04-23T14:56:13.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.828+0000 7f3734db5740 20 client.27170 trim_cache size 0 max 16384 2024-04-23T14:56:13.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.849+0000 7f3734db5740 1 Processor -- start 2024-04-23T14:56:13.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.849+0000 7f3734db5740 1 -- start start 2024-04-23T14:56:13.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.849+0000 7f3734db5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e57d43a30 0x555e57d5a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.849+0000 7f3734db5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e57d44350 0x555e57d3f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.849+0000 7f3734db5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e57d4ea20 0x555e57d668a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.849+0000 7f3734db5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f372c1b21d0 con 0x555e57d43a30 2024-04-23T14:56:13.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.849+0000 7f3734db5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f372c1b4a30 con 0x555e57d44350 2024-04-23T14:56:13.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.849+0000 7f3734db5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f372c1b2820 con 0x555e57d4ea20 2024-04-23T14:56:13.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.850+0000 7f372aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e57d43a30 0x555e57d5a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.850+0000 7f3723e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e57d44350 0x555e57d3f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.850+0000 7f372aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e57d43a30 0x555e57d5a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57734/0 (socket says 172.21.15.28:57734) 2024-04-23T14:56:13.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.850+0000 7f372aa00640 1 -- 172.21.15.28:0/1237348142 learned_addr learned my addr 172.21.15.28:0/1237348142 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:13.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.850+0000 7f372b400640 1 --2- 172.21.15.28:0/1237348142 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e57d4ea20 0x555e57d668a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.850+0000 7f372aa00640 1 -- 172.21.15.28:0/1237348142 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e57d4ea20 msgr2=0x555e57d668a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:13.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.850+0000 7f372aa00640 1 --2- 172.21.15.28:0/1237348142 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e57d4ea20 0x555e57d668a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.850+0000 7f372aa00640 1 -- 172.21.15.28:0/1237348142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e57d44350 msgr2=0x555e57d3f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:13.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.850+0000 7f372aa00640 1 --2- 172.21.15.28:0/1237348142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e57d44350 0x555e57d3f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.850+0000 7f372aa00640 1 -- 172.21.15.28:0/1237348142 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e57d4a930 con 0x555e57d43a30 2024-04-23T14:56:13.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.850+0000 7f3723e00640 1 --2- 172.21.15.28:0/1237348142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e57d44350 0x555e57d3f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:56:13.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.850+0000 7f372aa00640 1 --2- 172.21.15.28:0/1237348142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e57d43a30 0x555e57d5a730 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f371c0bcac0 tx=0x7f371c03b720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=746663740a7edb2c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:13.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.851+0000 7f371b400640 1 -- 172.21.15.28:0/1237348142 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f371c049040 con 0x555e57d43a30 2024-04-23T14:56:13.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.851+0000 7f371b400640 1 -- 172.21.15.28:0/1237348142 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f371c0bcdc0 con 0x555e57d43a30 2024-04-23T14:56:13.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.851+0000 7f371b400640 1 -- 172.21.15.28:0/1237348142 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f371c0b9430 con 0x555e57d43a30 2024-04-23T14:56:13.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.851+0000 7f3734db5740 1 -- 172.21.15.28:0/1237348142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e57d43a30 msgr2=0x555e57d5a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:13.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.851+0000 7f3734db5740 1 --2- 172.21.15.28:0/1237348142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e57d43a30 0x555e57d5a730 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f371c0bcac0 tx=0x7f371c03b720 comp rx=0 tx=0).stop 2024-04-23T14:56:13.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.851+0000 7f3734db5740 1 -- 172.21.15.28:0/1237348142 shutdown_connections 2024-04-23T14:56:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.851+0000 7f3734db5740 1 --2- 172.21.15.28:0/1237348142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e57d44350 0x555e57d3f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.851+0000 7f3734db5740 1 --2- 172.21.15.28:0/1237348142 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e57d4ea20 0x555e57d668a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.851+0000 7f3734db5740 1 --2- 172.21.15.28:0/1237348142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e57d43a30 0x555e57d5a730 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.852+0000 7f3734db5740 1 -- 172.21.15.28:0/1237348142 >> 172.21.15.28:0/1237348142 conn(0x555e57d3fd50 msgr2=0x555e57d4c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.852+0000 7f3734db5740 1 -- 172.21.15.28:0/1237348142 shutdown_connections 2024-04-23T14:56:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.852+0000 7f3734db5740 1 -- 172.21.15.28:0/1237348142 wait complete. 2024-04-23T14:56:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.852+0000 7f3734db5740 1 -- start start 2024-04-23T14:56:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.852+0000 7f3734db5740 1 Processor -- start 2024-04-23T14:56:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.853+0000 7f3734db5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e57d43a30 0x555e57d527a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.853+0000 7f3734db5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e57d44350 0x555e57d52ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.853+0000 7f3734db5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e57d4ea20 0x555e57d53220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.853+0000 7f3734db5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f372c1b2ef0 con 0x555e57d43a30 2024-04-23T14:56:13.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.853+0000 7f3734db5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f372c1b4360 con 0x555e57d4ea20 2024-04-23T14:56:13.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.853+0000 7f3734db5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f372c1b35c0 con 0x555e57d44350 2024-04-23T14:56:13.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.853+0000 7f3734db5740 10 client.0 _ll_register_callbacks cb 0x7f371401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:56:13.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.853+0000 7f372aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e57d43a30 0x555e57d527a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.853+0000 7f372b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e57d4ea20 0x555e57d53220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.853+0000 7f372aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e57d43a30 0x555e57d527a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57744/0 (socket says 172.21.15.28:57744) 2024-04-23T14:56:13.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.853+0000 7f372aa00640 1 -- 172.21.15.28:0/2607256613 learned_addr learned my addr 172.21.15.28:0/2607256613 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:13.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.853+0000 7f3723e00640 1 --2- 172.21.15.28:0/2607256613 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e57d44350 0x555e57d52ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.854+0000 7f372b400640 1 -- 172.21.15.28:0/2607256613 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e57d44350 msgr2=0x555e57d52ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:13.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.854+0000 7f372b400640 1 --2- 172.21.15.28:0/2607256613 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e57d44350 0x555e57d52ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.854+0000 7f372b400640 1 -- 172.21.15.28:0/2607256613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e57d43a30 msgr2=0x555e57d527a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:13.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.854+0000 7f372b400640 1 --2- 172.21.15.28:0/2607256613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e57d43a30 0x555e57d527a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.854+0000 7f372b400640 1 -- 172.21.15.28:0/2607256613 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e57d61410 con 0x555e57d4ea20 2024-04-23T14:56:13.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.854+0000 7f372aa00640 1 --2- 172.21.15.28:0/2607256613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e57d43a30 0x555e57d527a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:56:13.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.854+0000 7f372b400640 1 --2- 172.21.15.28:0/2607256613 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e57d4ea20 0x555e57d53220 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f3724009c30 tx=0x7f3724004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b7f5929bd70b269 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:13.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.854+0000 7f3718c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:56:13.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.854+0000 7f3718c00640 1 -- 172.21.15.28:0/2607256613 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f372402e060 con 0x555e57d4ea20 2024-04-23T14:56:13.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.854+0000 7f3718c00640 1 -- 172.21.15.28:0/2607256613 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8542+0+0 (secure 0 0 0) 0x7f372400f990 con 0x555e57d4ea20 2024-04-23T14:56:13.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.854+0000 7f3734db5740 1 -- 172.21.15.28:0/2607256613 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f371c0bcdc0 con 0x555e57d4ea20 2024-04-23T14:56:13.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.854+0000 7f3718c00640 1 -- 172.21.15.28:0/2607256613 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f372400fb90 con 0x555e57d4ea20 2024-04-23T14:56:13.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.854+0000 7f3734db5740 10 client.27176 fetch_fsmap learned FSMap version 20 2024-04-23T14:56:13.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.854+0000 7f3734db5740 1 -- 172.21.15.28:0/2607256613 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x555e579bb3f0 con 0x555e57d4ea20 2024-04-23T14:56:13.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.855+0000 7f3718c00640 1 -- 172.21.15.28:0/2607256613 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f371c0bcdc0 con 0x555e57d4ea20 2024-04-23T14:56:13.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.855+0000 7f3718c00640 1 -- 172.21.15.28:0/2607256613 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f372401d420 con 0x555e57d4ea20 2024-04-23T14:56:13.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.855+0000 7f3734db5740 10 client.27176 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:56:13.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.855+0000 7f3734db5740 10 client.27176 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:56:13.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.855+0000 7f3734db5740 20 client.27176 populate_metadata read hostname 'smithi028' 2024-04-23T14:56:13.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.855+0000 7f3734db5740 1 --2- 172.21.15.28:0/2607256613 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x555e57dc5940 0x555e57dc7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:13.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.855+0000 7f3734db5740 4 client.27176 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:56:13.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.855+0000 7f3734db5740 1 -- 172.21.15.28:0/2607256613 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555e57cb0a80 con 0x555e57dc5940 2024-04-23T14:56:13.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.855+0000 7f372aa00640 1 --2- 172.21.15.28:0/2607256613 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x555e57dc5940 0x555e57dc7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:13.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.856+0000 7f372aa00640 1 --2- 172.21.15.28:0/2607256613 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x555e57dc5940 0x555e57dc7d60 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44f43f14eae306c0 server_cookie=2698b7cdce33f4ca in_seq=0 out_seq=0 2024-04-23T14:56:13.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.856+0000 7f3718c00640 10 client.27176 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:56:13.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.859+0000 7f3718c00640 1 -- 172.21.15.28:0/2607256613 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x555e57cb0a80 con 0x555e57dc5940 2024-04-23T14:56:13.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.859+0000 7f3718c00640 10 client.27176 handle_command_reply: tid=0 2024-04-23T14:56:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.860+0000 7f3734db5740 1 client.27176 shutdown 2024-04-23T14:56:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.860+0000 7f3734db5740 1 -- 172.21.15.28:0/2607256613 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e57d4ea20 msgr2=0x555e57d53220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.860+0000 7f3734db5740 1 --2- 172.21.15.28:0/2607256613 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e57d4ea20 0x555e57d53220 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f3724009c30 tx=0x7f3724004a40 comp rx=0 tx=0).stop 2024-04-23T14:56:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.860+0000 7f3734db5740 1 -- 172.21.15.28:0/2607256613 shutdown_connections 2024-04-23T14:56:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.860+0000 7f3734db5740 1 --2- 172.21.15.28:0/2607256613 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x555e57dc5940 0x555e57dc7d60 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.860+0000 7f3734db5740 1 --2- 172.21.15.28:0/2607256613 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e57d4ea20 0x555e57d53220 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.860+0000 7f3734db5740 1 --2- 172.21.15.28:0/2607256613 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e57d44350 0x555e57d52ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.860+0000 7f3734db5740 1 --2- 172.21.15.28:0/2607256613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e57d43a30 0x555e57d527a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.860+0000 7f3718c00640 0 client.27176 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:56:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.860+0000 7f3734db5740 1 -- 172.21.15.28:0/2607256613 >> 172.21.15.28:0/2607256613 conn(0x555e57d3fd50 msgr2=0x555e57d45120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.860+0000 7f3734db5740 1 -- 172.21.15.28:0/2607256613 shutdown_connections 2024-04-23T14:56:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.860+0000 7f3734db5740 1 -- 172.21.15.28:0/2607256613 wait complete. 2024-04-23T14:56:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.860+0000 7f3734db5740 20 client.27176 trim_cache size 0 max 16384 2024-04-23T14:56:13.862 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:56:13.862 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-23T14:56:13.862 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-23T14:56:13.862 INFO:teuthology.orchestra.run.smithi028.stdout: "445eefb4-d9e5-4d3a-a5d6-aab2909624a4": { 2024-04-23T14:56:13.863 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-23T14:56:13.863 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "445eefb4-d9e5-4d3a-a5d6-aab2909624a4", 2024-04-23T14:56:13.863 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-23T14:56:13.863 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:56:13.863 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:56:13.864 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:56:13.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.861+0000 7f372be00640 1 -- 172.21.15.28:0/3497848827 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f37100a3da0 msgr2=0x7f37100a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:13.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.861+0000 7f372be00640 1 --2- 172.21.15.28:0/3497848827 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f37100a3da0 0x7f37100a6260 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f372c06b460 tx=0x7f3724009500 comp rx=0 tx=0).stop 2024-04-23T14:56:13.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.861+0000 7f372be00640 1 -- 172.21.15.28:0/3497848827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f372c076530 msgr2=0x7f372c19edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:13.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.861+0000 7f372be00640 1 --2- 172.21.15.28:0/3497848827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f372c076530 0x7f372c19edc0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f371c02a940 tx=0x7f371c003b20 comp rx=0 tx=0).stop 2024-04-23T14:56:13.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.861+0000 7f372be00640 1 -- 172.21.15.28:0/3497848827 shutdown_connections 2024-04-23T14:56:13.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.861+0000 7f372be00640 1 --2- 172.21.15.28:0/3497848827 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f37100a3da0 0x7f37100a6260 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.861+0000 7f372be00640 1 --2- 172.21.15.28:0/3497848827 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f372c076e50 0x7f372c19f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.861+0000 7f372be00640 1 --2- 172.21.15.28:0/3497848827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f372c112d40 0x7f372c1a3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.861+0000 7f372be00640 1 --2- 172.21.15.28:0/3497848827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f372c076530 0x7f372c19edc0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:13.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.861+0000 7f372be00640 1 -- 172.21.15.28:0/3497848827 >> 172.21.15.28:0/3497848827 conn(0x7f372c0ffb70 msgr2=0x7f372c101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:13.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.861+0000 7f372be00640 1 -- 172.21.15.28:0/3497848827 shutdown_connections 2024-04-23T14:56:13.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:13.861+0000 7f372be00640 1 -- 172.21.15.28:0/3497848827 wait complete. 2024-04-23T14:56:13.880 INFO:tasks.cephfs.filesystem:scrub status for tag:445eefb4-d9e5-4d3a-a5d6-aab2909624a4 - {'path': '/', 'tag': '445eefb4-d9e5-4d3a-a5d6-aab2909624a4', 'options': 'recursive,force'} 2024-04-23T14:56:14.541 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1039757 8.57 MB/sec execute 510 sec latency 23.763 ms 2024-04-23T14:56:15.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:14 smithi105 ceph-mon[32555]: scrub queued for path: / 2024-04-23T14:56:15.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:14 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:56:15.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:14 smithi105 ceph-mon[32555]: scrub summary: active paths [/] 2024-04-23T14:56:15.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:14 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:56:15.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:14 smithi105 ceph-mon[32555]: pgmap v381: 97 pgs: 97 active+clean; 553 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.3 MiB/s wr, 188 op/s 2024-04-23T14:56:15.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:14 smithi038 ceph-mon[32348]: scrub queued for path: / 2024-04-23T14:56:15.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:14 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:56:15.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:14 smithi038 ceph-mon[32348]: scrub summary: active paths [/] 2024-04-23T14:56:15.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:14 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:56:15.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:14 smithi038 ceph-mon[32348]: pgmap v381: 97 pgs: 97 active+clean; 553 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.3 MiB/s wr, 188 op/s 2024-04-23T14:56:15.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:14 smithi028 ceph-mon[27810]: scrub queued for path: / 2024-04-23T14:56:15.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:14 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:56:15.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:14 smithi028 ceph-mon[27810]: scrub summary: active paths [/] 2024-04-23T14:56:15.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:14 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:56:15.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:14 smithi028 ceph-mon[27810]: pgmap v381: 97 pgs: 97 active+clean; 553 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.3 MiB/s wr, 188 op/s 2024-04-23T14:56:15.541 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1041125 8.58 MB/sec execute 511 sec latency 62.349 ms 2024-04-23T14:56:16.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:15 smithi105 ceph-mon[32555]: scrub summary: idle 2024-04-23T14:56:16.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:15 smithi038 ceph-mon[32348]: scrub summary: idle 2024-04-23T14:56:16.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:15 smithi028 ceph-mon[27810]: scrub summary: idle 2024-04-23T14:56:16.541 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1042393 8.57 MB/sec execute 512 sec latency 30.890 ms 2024-04-23T14:56:17.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:16 smithi105 ceph-mon[32555]: pgmap v382: 97 pgs: 97 active+clean; 553 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.5 MiB/s wr, 154 op/s 2024-04-23T14:56:17.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:16 smithi038 ceph-mon[32348]: pgmap v382: 97 pgs: 97 active+clean; 553 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.5 MiB/s wr, 154 op/s 2024-04-23T14:56:17.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:16 smithi028 ceph-mon[27810]: pgmap v382: 97 pgs: 97 active+clean; 553 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.5 MiB/s wr, 154 op/s 2024-04-23T14:56:17.541 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1044222 8.57 MB/sec execute 513 sec latency 21.170 ms 2024-04-23T14:56:18.541 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1044887 8.57 MB/sec execute 514 sec latency 25.457 ms 2024-04-23T14:56:19.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:18 smithi105 ceph-mon[32555]: pgmap v383: 97 pgs: 97 active+clean; 563 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.6 MiB/s wr, 157 op/s 2024-04-23T14:56:19.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:18 smithi038 ceph-mon[32348]: pgmap v383: 97 pgs: 97 active+clean; 563 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.6 MiB/s wr, 157 op/s 2024-04-23T14:56:19.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:18 smithi028 ceph-mon[27810]: pgmap v383: 97 pgs: 97 active+clean; 563 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.6 MiB/s wr, 157 op/s 2024-04-23T14:56:19.541 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1045977 8.56 MB/sec execute 515 sec latency 72.581 ms 2024-04-23T14:56:20.541 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1047331 8.56 MB/sec execute 516 sec latency 46.946 ms 2024-04-23T14:56:21.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:20 smithi105 ceph-mon[32555]: pgmap v384: 97 pgs: 97 active+clean; 563 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.6 MiB/s wr, 157 op/s 2024-04-23T14:56:21.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:20 smithi038 ceph-mon[32348]: pgmap v384: 97 pgs: 97 active+clean; 563 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.6 MiB/s wr, 157 op/s 2024-04-23T14:56:21.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:20 smithi028 ceph-mon[27810]: pgmap v384: 97 pgs: 97 active+clean; 563 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.6 MiB/s wr, 157 op/s 2024-04-23T14:56:21.542 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1048411 8.55 MB/sec execute 517 sec latency 32.307 ms 2024-04-23T14:56:22.542 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1049613 8.55 MB/sec execute 518 sec latency 26.049 ms 2024-04-23T14:56:23.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:22 smithi105 ceph-mon[32555]: pgmap v385: 97 pgs: 97 active+clean; 563 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.6 MiB/s wr, 157 op/s 2024-04-23T14:56:23.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:22 smithi038 ceph-mon[32348]: pgmap v385: 97 pgs: 97 active+clean; 563 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.6 MiB/s wr, 157 op/s 2024-04-23T14:56:23.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:22 smithi028 ceph-mon[27810]: pgmap v385: 97 pgs: 97 active+clean; 563 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.6 MiB/s wr, 157 op/s 2024-04-23T14:56:23.542 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1051339 8.55 MB/sec execute 519 sec latency 20.908 ms 2024-04-23T14:56:24.542 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1052065 8.54 MB/sec execute 520 sec latency 41.396 ms 2024-04-23T14:56:25.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:24 smithi105 ceph-mon[32555]: pgmap v386: 97 pgs: 97 active+clean; 572 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.4 MiB/s wr, 189 op/s 2024-04-23T14:56:25.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:24 smithi038 ceph-mon[32348]: pgmap v386: 97 pgs: 97 active+clean; 572 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.4 MiB/s wr, 189 op/s 2024-04-23T14:56:25.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:24 smithi028 ceph-mon[27810]: pgmap v386: 97 pgs: 97 active+clean; 572 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.4 MiB/s wr, 189 op/s 2024-04-23T14:56:25.542 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1053488 8.54 MB/sec execute 521 sec latency 46.119 ms 2024-04-23T14:56:26.542 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1055268 8.54 MB/sec execute 522 sec latency 46.501 ms 2024-04-23T14:56:27.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:26 smithi105 ceph-mon[32555]: pgmap v387: 97 pgs: 97 active+clean; 572 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 122 op/s 2024-04-23T14:56:27.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:26 smithi038 ceph-mon[32348]: pgmap v387: 97 pgs: 97 active+clean; 572 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 122 op/s 2024-04-23T14:56:27.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:26 smithi028 ceph-mon[27810]: pgmap v387: 97 pgs: 97 active+clean; 572 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 122 op/s 2024-04-23T14:56:27.542 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1056306 8.53 MB/sec execute 523 sec latency 19.708 ms 2024-04-23T14:56:28.542 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1057955 8.54 MB/sec execute 524 sec latency 25.478 ms 2024-04-23T14:56:29.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:28 smithi105 ceph-mon[32555]: pgmap v388: 97 pgs: 97 active+clean; 581 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.2 MiB/s wr, 183 op/s 2024-04-23T14:56:29.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:28 smithi038 ceph-mon[32348]: pgmap v388: 97 pgs: 97 active+clean; 581 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.2 MiB/s wr, 183 op/s 2024-04-23T14:56:29.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:28 smithi028 ceph-mon[27810]: pgmap v388: 97 pgs: 97 active+clean; 581 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.2 MiB/s wr, 183 op/s 2024-04-23T14:56:29.543 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1059068 8.53 MB/sec execute 525 sec latency 18.465 ms 2024-04-23T14:56:30.543 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1060258 8.53 MB/sec execute 526 sec latency 93.162 ms 2024-04-23T14:56:31.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:30 smithi105 ceph-mon[32555]: pgmap v389: 97 pgs: 97 active+clean; 581 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 117 op/s 2024-04-23T14:56:31.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:30 smithi038 ceph-mon[32348]: pgmap v389: 97 pgs: 97 active+clean; 581 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 117 op/s 2024-04-23T14:56:31.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:30 smithi028 ceph-mon[27810]: pgmap v389: 97 pgs: 97 active+clean; 581 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 117 op/s 2024-04-23T14:56:31.543 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1062145 8.53 MB/sec execute 527 sec latency 31.745 ms 2024-04-23T14:56:32.543 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1063031 8.52 MB/sec execute 528 sec latency 24.717 ms 2024-04-23T14:56:33.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:32 smithi038 ceph-mon[32348]: pgmap v390: 97 pgs: 97 active+clean; 581 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 117 op/s 2024-04-23T14:56:33.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:32 smithi028 ceph-mon[27810]: pgmap v390: 97 pgs: 97 active+clean; 581 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 117 op/s 2024-04-23T14:56:33.285 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:32 smithi105 ceph-mon[32555]: pgmap v390: 97 pgs: 97 active+clean; 581 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 117 op/s 2024-04-23T14:56:33.543 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1064556 8.52 MB/sec execute 529 sec latency 20.004 ms 2024-04-23T14:56:34.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:33 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:56:34.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:33 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:56:34.186 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:33 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:56:34.543 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1065849 8.52 MB/sec execute 530 sec latency 20.049 ms 2024-04-23T14:56:35.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:34 smithi038 ceph-mon[32348]: pgmap v391: 97 pgs: 97 active+clean; 589 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.1 MiB/s wr, 177 op/s 2024-04-23T14:56:35.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:34 smithi028 ceph-mon[27810]: pgmap v391: 97 pgs: 97 active+clean; 589 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.1 MiB/s wr, 177 op/s 2024-04-23T14:56:35.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:34 smithi105 ceph-mon[32555]: pgmap v391: 97 pgs: 97 active+clean; 589 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.1 MiB/s wr, 177 op/s 2024-04-23T14:56:35.543 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1066909 8.51 MB/sec execute 531 sec latency 34.034 ms 2024-04-23T14:56:36.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:35 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:56:36.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:35 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:56:36.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:35 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:56:36.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:35 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:56:36.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:35 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:56:36.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:35 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:56:36.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:35 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:56:36.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:35 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:56:36.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:35 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:56:36.543 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1068957 8.52 MB/sec execute 532 sec latency 39.791 ms 2024-04-23T14:56:37.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:36 smithi038 ceph-mon[32348]: pgmap v392: 97 pgs: 97 active+clean; 589 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 120 op/s 2024-04-23T14:56:37.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:36 smithi028 ceph-mon[27810]: pgmap v392: 97 pgs: 97 active+clean; 589 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 120 op/s 2024-04-23T14:56:37.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:36 smithi105 ceph-mon[32555]: pgmap v392: 97 pgs: 97 active+clean; 589 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 120 op/s 2024-04-23T14:56:37.544 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1069971 8.51 MB/sec execute 533 sec latency 18.864 ms 2024-04-23T14:56:38.544 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1071217 8.51 MB/sec execute 534 sec latency 26.401 ms 2024-04-23T14:56:39.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:38 smithi038 ceph-mon[32348]: pgmap v393: 97 pgs: 97 active+clean; 598 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.1 MiB/s wr, 180 op/s 2024-04-23T14:56:39.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:38 smithi028 ceph-mon[27810]: pgmap v393: 97 pgs: 97 active+clean; 598 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.1 MiB/s wr, 180 op/s 2024-04-23T14:56:39.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:38 smithi105 ceph-mon[32555]: pgmap v393: 97 pgs: 97 active+clean; 598 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.1 MiB/s wr, 180 op/s 2024-04-23T14:56:39.544 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1072919 8.51 MB/sec execute 535 sec latency 17.998 ms 2024-04-23T14:56:40.544 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1073961 8.50 MB/sec execute 536 sec latency 37.428 ms 2024-04-23T14:56:41.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:40 smithi038 ceph-mon[32348]: pgmap v394: 97 pgs: 97 active+clean; 598 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 118 op/s 2024-04-23T14:56:41.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:40 smithi028 ceph-mon[27810]: pgmap v394: 97 pgs: 97 active+clean; 598 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 118 op/s 2024-04-23T14:56:41.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:40 smithi105 ceph-mon[32555]: pgmap v394: 97 pgs: 97 active+clean; 598 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 118 op/s 2024-04-23T14:56:41.544 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1075352 8.50 MB/sec execute 537 sec latency 66.770 ms 2024-04-23T14:56:42.544 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1076791 8.50 MB/sec execute 538 sec latency 15.079 ms 2024-04-23T14:56:43.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:42 smithi038 ceph-mon[32348]: pgmap v395: 97 pgs: 97 active+clean; 598 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 118 op/s 2024-04-23T14:56:43.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:42 smithi028 ceph-mon[27810]: pgmap v395: 97 pgs: 97 active+clean; 598 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 118 op/s 2024-04-23T14:56:43.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:42 smithi105 ceph-mon[32555]: pgmap v395: 97 pgs: 97 active+clean; 598 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 118 op/s 2024-04-23T14:56:43.544 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1078103 8.50 MB/sec execute 539 sec latency 25.316 ms 2024-04-23T14:56:43.881 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:56:43.881 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:56:44.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.108+0000 7fb793000640 1 -- 172.21.15.28:0/2191155623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb78c104e60 msgr2=0x7fb78c108ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:44.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.108+0000 7fb793000640 1 --2- 172.21.15.28:0/2191155623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb78c104e60 0x7fb78c108ea0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fb774009900 tx=0x7fb77402a410 comp rx=0 tx=0).stop 2024-04-23T14:56:44.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.108+0000 7fb793000640 1 -- 172.21.15.28:0/2191155623 shutdown_connections 2024-04-23T14:56:44.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.108+0000 7fb793000640 1 --2- 172.21.15.28:0/2191155623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb78c104e60 0x7fb78c108ea0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.108+0000 7fb793000640 1 --2- 172.21.15.28:0/2191155623 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb78c109580 0x7fb78c111150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.108+0000 7fb793000640 1 --2- 172.21.15.28:0/2191155623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb78c1044b0 0x7fb78c104890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.108+0000 7fb793000640 1 -- 172.21.15.28:0/2191155623 >> 172.21.15.28:0/2191155623 conn(0x7fb78c0ffd90 msgr2=0x7fb78c1021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:44.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.108+0000 7fb793000640 1 -- 172.21.15.28:0/2191155623 shutdown_connections 2024-04-23T14:56:44.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.108+0000 7fb793000640 1 -- 172.21.15.28:0/2191155623 wait complete. 2024-04-23T14:56:44.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.109+0000 7fb793000640 1 Processor -- start 2024-04-23T14:56:44.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.109+0000 7fb793000640 1 -- start start 2024-04-23T14:56:44.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.110+0000 7fb793000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb78c1044b0 0x7fb78c1a0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.110+0000 7fb793000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb78c109580 0x7fb78c1a0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.110+0000 7fb793000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb78c19aa70 0x7fb78c19af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.110+0000 7fb793000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb78c114610 con 0x7fb78c19aa70 2024-04-23T14:56:44.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.110+0000 7fb793000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb78c114490 con 0x7fb78c109580 2024-04-23T14:56:44.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.110+0000 7fb793000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb78c114790 con 0x7fb78c1044b0 2024-04-23T14:56:44.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.110+0000 7fb783400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb78c109580 0x7fb78c1a0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.110+0000 7fb78be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb78c1044b0 0x7fb78c1a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.110+0000 7fb783400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb78c109580 0x7fb78c1a0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:47888/0 (socket says 172.21.15.28:47888) 2024-04-23T14:56:44.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.110+0000 7fb783400640 1 -- 172.21.15.28:0/1971173634 learned_addr learned my addr 172.21.15.28:0/1971173634 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:44.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.110+0000 7fb791000640 1 --2- 172.21.15.28:0/1971173634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb78c19aa70 0x7fb78c19af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.111+0000 7fb783400640 1 -- 172.21.15.28:0/1971173634 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb78c1044b0 msgr2=0x7fb78c1a0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:44.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.111+0000 7fb783400640 1 --2- 172.21.15.28:0/1971173634 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb78c1044b0 0x7fb78c1a0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.111+0000 7fb783400640 1 -- 172.21.15.28:0/1971173634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb78c19aa70 msgr2=0x7fb78c19af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:44.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.111+0000 7fb783400640 1 --2- 172.21.15.28:0/1971173634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb78c19aa70 0x7fb78c19af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.111+0000 7fb783400640 1 -- 172.21.15.28:0/1971173634 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb78c19b590 con 0x7fb78c109580 2024-04-23T14:56:44.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.111+0000 7fb78be00640 1 --2- 172.21.15.28:0/1971173634 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb78c1044b0 0x7fb78c1a0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:56:44.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.111+0000 7fb783400640 1 --2- 172.21.15.28:0/1971173634 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb78c109580 0x7fb78c1a0db0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fb774005940 tx=0x7fb7740040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:44.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.111+0000 7fb789600640 1 -- 172.21.15.28:0/1971173634 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77403f030 con 0x7fb78c109580 2024-04-23T14:56:44.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.111+0000 7fb789600640 1 -- 172.21.15.28:0/1971173634 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7740044c0 con 0x7fb78c109580 2024-04-23T14:56:44.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.111+0000 7fb789600640 1 -- 172.21.15.28:0/1971173634 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb774035070 con 0x7fb78c109580 2024-04-23T14:56:44.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.111+0000 7fb793000640 1 -- 172.21.15.28:0/1971173634 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb78c19b820 con 0x7fb78c109580 2024-04-23T14:56:44.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.111+0000 7fb793000640 1 -- 172.21.15.28:0/1971173634 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb78c1a7970 con 0x7fb78c109580 2024-04-23T14:56:44.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.115+0000 7fb789600640 1 -- 172.21.15.28:0/1971173634 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fb774042030 con 0x7fb78c109580 2024-04-23T14:56:44.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.115+0000 7fb795fad740 1 Processor -- start 2024-04-23T14:56:44.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.115+0000 7fb795fad740 1 -- start start 2024-04-23T14:56:44.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.115+0000 7fb795fad740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3daa68860 0x55a3daa68c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.115+0000 7fb795fad740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3daa69180 0x55a3daa73310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.115+0000 7fb795fad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3daa73850 0x55a3daa75c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.115+0000 7fb795fad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a3da9ce1f0 con 0x55a3daa73850 2024-04-23T14:56:44.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.115+0000 7fb795fad740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a3da974350 con 0x55a3daa68860 2024-04-23T14:56:44.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.115+0000 7fb795fad740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a3da9ca1b0 con 0x55a3daa69180 2024-04-23T14:56:44.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.116+0000 7fb791000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3daa68860 0x55a3daa68c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.116+0000 7fb78be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3daa69180 0x55a3daa73310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.116+0000 7fb791000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3daa68860 0x55a3daa68c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:47900/0 (socket says 172.21.15.28:47900) 2024-04-23T14:56:44.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.116+0000 7fb791000640 1 -- 172.21.15.28:0/3107425342 learned_addr learned my addr 172.21.15.28:0/3107425342 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:44.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.116+0000 7fb783400640 1 --2- 172.21.15.28:0/3107425342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3daa73850 0x55a3daa75c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.116+0000 7fb791000640 1 -- 172.21.15.28:0/3107425342 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3daa69180 msgr2=0x55a3daa73310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:44.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.116+0000 7fb791000640 1 --2- 172.21.15.28:0/3107425342 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3daa69180 0x55a3daa73310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.116+0000 7fb791000640 1 -- 172.21.15.28:0/3107425342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3daa73850 msgr2=0x55a3daa75c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:44.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.116+0000 7fb791000640 1 --2- 172.21.15.28:0/3107425342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3daa73850 0x55a3daa75c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.116+0000 7fb791000640 1 -- 172.21.15.28:0/3107425342 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3da9c8f50 con 0x55a3daa68860 2024-04-23T14:56:44.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.116+0000 7fb783400640 1 --2- 172.21.15.28:0/3107425342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3daa73850 0x55a3daa75c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T14:56:44.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.116+0000 7fb791000640 1 --2- 172.21.15.28:0/3107425342 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3daa68860 0x55a3daa68c40 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fb784009f50 tx=0x7fb7840118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d72cbdd4182acfb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:44.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.117+0000 7fb789600640 1 --2- 172.21.15.28:0/1971173634 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb7700a3da0 0x7fb7700a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.117+0000 7fb781600640 1 -- 172.21.15.28:0/3107425342 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb784004b20 con 0x55a3daa68860 2024-04-23T14:56:44.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.117+0000 7fb781600640 1 -- 172.21.15.28:0/3107425342 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb784003e50 con 0x55a3daa68860 2024-04-23T14:56:44.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.117+0000 7fb781600640 1 -- 172.21.15.28:0/3107425342 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb78400f540 con 0x55a3daa68860 2024-04-23T14:56:44.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.117+0000 7fb789600640 1 -- 172.21.15.28:0/1971173634 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8542+0+0 (secure 0 0 0) 0x7fb7740f0b60 con 0x7fb78c109580 2024-04-23T14:56:44.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.117+0000 7fb78be00640 1 --2- 172.21.15.28:0/1971173634 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb7700a3da0 0x7fb7700a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.118+0000 7fb795fad740 1 -- 172.21.15.28:0/3107425342 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3daa68860 msgr2=0x55a3daa68c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:44.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.118+0000 7fb795fad740 1 --2- 172.21.15.28:0/3107425342 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3daa68860 0x55a3daa68c40 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fb784009f50 tx=0x7fb7840118d0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.118+0000 7fb795fad740 1 -- 172.21.15.28:0/3107425342 shutdown_connections 2024-04-23T14:56:44.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.118+0000 7fb795fad740 1 --2- 172.21.15.28:0/3107425342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3daa73850 0x55a3daa75c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.118+0000 7fb795fad740 1 --2- 172.21.15.28:0/3107425342 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3daa69180 0x55a3daa73310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.118+0000 7fb795fad740 1 --2- 172.21.15.28:0/3107425342 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3daa68860 0x55a3daa68c40 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.118+0000 7fb795fad740 1 -- 172.21.15.28:0/3107425342 >> 172.21.15.28:0/3107425342 conn(0x55a3daa64b80 msgr2=0x55a3daa64f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:44.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.118+0000 7fb78be00640 1 --2- 172.21.15.28:0/1971173634 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb7700a3da0 0x7fb7700a6260 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x55a3daa69e20 tx=0x7fb77c005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:44.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.118+0000 7fb795fad740 1 -- 172.21.15.28:0/3107425342 shutdown_connections 2024-04-23T14:56:44.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.118+0000 7fb795fad740 1 -- 172.21.15.28:0/3107425342 wait complete. 2024-04-23T14:56:44.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.118+0000 7fb795fad740 1 -- start start 2024-04-23T14:56:44.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.119+0000 7fb795fad740 1 Processor -- start 2024-04-23T14:56:44.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.119+0000 7fb795fad740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3daa68860 0x55a3daa707e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.119+0000 7fb795fad740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3daa69180 0x55a3daa70d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.119+0000 7fb795fad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3daa73850 0x55a3daa71260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.119+0000 7fb795fad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a3daa76640 con 0x55a3daa73850 2024-04-23T14:56:44.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.119+0000 7fb795fad740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a3da987690 con 0x55a3daa68860 2024-04-23T14:56:44.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.119+0000 7fb795fad740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a3daa767c0 con 0x55a3daa69180 2024-04-23T14:56:44.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.119+0000 7fb795fad740 10 client.0 _ll_register_callbacks cb 0x55a3da9c3650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:56:44.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.119+0000 7fb78be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3daa68860 0x55a3daa707e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.119+0000 7fb783400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3daa69180 0x55a3daa70d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.119+0000 7fb78be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3daa68860 0x55a3daa707e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:47902/0 (socket says 172.21.15.28:47902) 2024-04-23T14:56:44.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.119+0000 7fb783400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3daa69180 0x55a3daa70d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:34998/0 (socket says 172.21.15.28:34998) 2024-04-23T14:56:44.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.119+0000 7fb791000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3daa73850 0x55a3daa71260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.119+0000 7fb78be00640 1 -- 172.21.15.28:0/3065023476 learned_addr learned my addr 172.21.15.28:0/3065023476 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:44.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.120+0000 7fb783400640 1 -- 172.21.15.28:0/3065023476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3daa68860 msgr2=0x55a3daa707e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:44.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.120+0000 7fb783400640 1 --2- 172.21.15.28:0/3065023476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3daa68860 0x55a3daa707e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.120+0000 7fb783400640 1 -- 172.21.15.28:0/3065023476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3daa73850 msgr2=0x55a3daa71260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:44.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.120+0000 7fb783400640 1 --2- 172.21.15.28:0/3065023476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3daa73850 0x55a3daa71260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.120+0000 7fb783400640 1 -- 172.21.15.28:0/3065023476 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3da986920 con 0x55a3daa69180 2024-04-23T14:56:44.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.120+0000 7fb783400640 1 --2- 172.21.15.28:0/3065023476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3daa69180 0x55a3daa70d20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fb7740ee240 tx=0x7fb7740bc4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a5ca01404b2eba9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:44.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.120+0000 7fb77a000640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:56:44.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.120+0000 7fb77a000640 1 -- 172.21.15.28:0/3065023476 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb774049040 con 0x55a3daa69180 2024-04-23T14:56:44.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.120+0000 7fb77a000640 1 -- 172.21.15.28:0/3065023476 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8542+0+0 (secure 0 0 0) 0x7fb7740b91b0 con 0x55a3daa69180 2024-04-23T14:56:44.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.120+0000 7fb795fad740 1 -- 172.21.15.28:0/3065023476 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb784003e50 con 0x55a3daa69180 2024-04-23T14:56:44.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.121+0000 7fb77a000640 1 -- 172.21.15.28:0/3065023476 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fb7740c0e40 con 0x55a3daa69180 2024-04-23T14:56:44.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.121+0000 7fb795fad740 10 client.27172 fetch_fsmap learned FSMap version 20 2024-04-23T14:56:44.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.121+0000 7fb795fad740 1 -- 172.21.15.28:0/3065023476 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55a3da7aa7d0 con 0x55a3daa69180 2024-04-23T14:56:44.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.121+0000 7fb77a000640 1 -- 172.21.15.28:0/3065023476 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb784003e50 con 0x55a3daa69180 2024-04-23T14:56:44.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.121+0000 7fb77a000640 1 -- 172.21.15.28:0/3065023476 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fb7740bce30 con 0x55a3daa69180 2024-04-23T14:56:44.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.121+0000 7fb795fad740 10 client.27172 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:56:44.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.121+0000 7fb795fad740 10 client.27172 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:56:44.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.121+0000 7fb795fad740 20 client.27172 populate_metadata read hostname 'smithi028' 2024-04-23T14:56:44.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.121+0000 7fb795fad740 1 --2- 172.21.15.28:0/3065023476 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55a3daa83a50 0x55a3daa85ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.121+0000 7fb795fad740 4 client.27172 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:56:44.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.121+0000 7fb795fad740 1 -- 172.21.15.28:0/3065023476 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a3da9dbc10 con 0x55a3daa83a50 2024-04-23T14:56:44.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.121+0000 7fb78be00640 1 --2- 172.21.15.28:0/3065023476 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55a3daa83a50 0x55a3daa85ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.122+0000 7fb78be00640 1 --2- 172.21.15.28:0/3065023476 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55a3daa83a50 0x55a3daa85ed0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82b18806a02b7d server_cookie=4b9f85a87d86029 in_seq=0 out_seq=0 2024-04-23T14:56:44.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.122+0000 7fb77a000640 10 client.27172 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:56:44.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.123+0000 7fb77a000640 1 -- 172.21.15.28:0/3065023476 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a3da9dbc10 con 0x55a3daa83a50 2024-04-23T14:56:44.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.123+0000 7fb77a000640 10 client.27172 handle_command_reply: tid=0 2024-04-23T14:56:44.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.123+0000 7fb795fad740 1 client.27172 shutdown 2024-04-23T14:56:44.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.123+0000 7fb795fad740 1 -- 172.21.15.28:0/3065023476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3daa69180 msgr2=0x55a3daa70d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:44.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.123+0000 7fb795fad740 1 --2- 172.21.15.28:0/3065023476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3daa69180 0x55a3daa70d20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fb7740ee240 tx=0x7fb7740bc4a0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.123+0000 7fb795fad740 1 -- 172.21.15.28:0/3065023476 shutdown_connections 2024-04-23T14:56:44.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.123+0000 7fb795fad740 1 --2- 172.21.15.28:0/3065023476 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55a3daa83a50 0x55a3daa85ed0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.123+0000 7fb795fad740 1 --2- 172.21.15.28:0/3065023476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3daa73850 0x55a3daa71260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.123+0000 7fb795fad740 1 --2- 172.21.15.28:0/3065023476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3daa69180 0x55a3daa70d20 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.123+0000 7fb795fad740 1 --2- 172.21.15.28:0/3065023476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3daa68860 0x55a3daa707e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.123+0000 7fb77a000640 0 client.27172 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:56:44.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.123+0000 7fb795fad740 1 -- 172.21.15.28:0/3065023476 >> 172.21.15.28:0/3065023476 conn(0x55a3daa64b80 msgr2=0x55a3daa6a980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:44.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.123+0000 7fb795fad740 1 -- 172.21.15.28:0/3065023476 shutdown_connections 2024-04-23T14:56:44.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.123+0000 7fb795fad740 1 -- 172.21.15.28:0/3065023476 wait complete. 2024-04-23T14:56:44.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.124+0000 7fb795fad740 20 client.27172 trim_cache size 0 max 16384 2024-04-23T14:56:44.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.145+0000 7fb795fad740 1 Processor -- start 2024-04-23T14:56:44.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.145+0000 7fb795fad740 1 -- start start 2024-04-23T14:56:44.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.145+0000 7fb795fad740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3daa68860 0x55a3daa6f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.145+0000 7fb795fad740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3daa69180 0x55a3daa7f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.146+0000 7fb795fad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3daa73850 0x55a3daa6b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.146+0000 7fb795fad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb78c1ae020 con 0x55a3daa73850 2024-04-23T14:56:44.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.146+0000 7fb795fad740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb78c1b0880 con 0x55a3daa69180 2024-04-23T14:56:44.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.146+0000 7fb795fad740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb78c1ae670 con 0x55a3daa68860 2024-04-23T14:56:44.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.146+0000 7fb78be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3daa68860 0x55a3daa6f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.146+0000 7fb78be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3daa68860 0x55a3daa6f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:35012/0 (socket says 172.21.15.28:35012) 2024-04-23T14:56:44.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.146+0000 7fb78be00640 1 -- 172.21.15.28:0/109643699 learned_addr learned my addr 172.21.15.28:0/109643699 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:44.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.146+0000 7fb783400640 1 --2- 172.21.15.28:0/109643699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3daa69180 0x55a3daa7f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.146+0000 7fb791000640 1 --2- 172.21.15.28:0/109643699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3daa73850 0x55a3daa6b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.146+0000 7fb783400640 1 -- 172.21.15.28:0/109643699 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3daa68860 msgr2=0x55a3daa6f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:44.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.146+0000 7fb783400640 1 --2- 172.21.15.28:0/109643699 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3daa68860 0x55a3daa6f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.146+0000 7fb783400640 1 -- 172.21.15.28:0/109643699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3daa73850 msgr2=0x55a3daa6b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:44.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.146+0000 7fb783400640 1 --2- 172.21.15.28:0/109643699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3daa73850 0x55a3daa6b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.146+0000 7fb783400640 1 -- 172.21.15.28:0/109643699 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3da6e03f0 con 0x55a3daa69180 2024-04-23T14:56:44.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.146+0000 7fb78be00640 1 --2- 172.21.15.28:0/109643699 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3daa68860 0x55a3daa6f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:56:44.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.147+0000 7fb783400640 1 --2- 172.21.15.28:0/109643699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3daa69180 0x55a3daa7f530 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fb774009340 tx=0x7fb774005b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d0a9c8c732ede35 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:44.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.147+0000 7fb780c00640 1 -- 172.21.15.28:0/109643699 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb774049070 con 0x55a3daa69180 2024-04-23T14:56:44.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.147+0000 7fb780c00640 1 -- 172.21.15.28:0/109643699 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7740c5700 con 0x55a3daa69180 2024-04-23T14:56:44.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.147+0000 7fb780c00640 1 -- 172.21.15.28:0/109643699 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7740ee4c0 con 0x55a3daa69180 2024-04-23T14:56:44.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.148+0000 7fb795fad740 1 -- 172.21.15.28:0/109643699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3daa69180 msgr2=0x55a3daa7f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:44.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.148+0000 7fb795fad740 1 --2- 172.21.15.28:0/109643699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3daa69180 0x55a3daa7f530 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fb774009340 tx=0x7fb774005b50 comp rx=0 tx=0).stop 2024-04-23T14:56:44.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.148+0000 7fb795fad740 1 -- 172.21.15.28:0/109643699 shutdown_connections 2024-04-23T14:56:44.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.148+0000 7fb795fad740 1 --2- 172.21.15.28:0/109643699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3daa69180 0x55a3daa7f530 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.148+0000 7fb795fad740 1 --2- 172.21.15.28:0/109643699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3daa73850 0x55a3daa6b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.148+0000 7fb795fad740 1 --2- 172.21.15.28:0/109643699 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3daa68860 0x55a3daa6f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.148+0000 7fb795fad740 1 -- 172.21.15.28:0/109643699 >> 172.21.15.28:0/109643699 conn(0x55a3daa64b80 msgr2=0x55a3daa71030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:44.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.148+0000 7fb795fad740 1 -- 172.21.15.28:0/109643699 shutdown_connections 2024-04-23T14:56:44.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.148+0000 7fb795fad740 1 -- 172.21.15.28:0/109643699 wait complete. 2024-04-23T14:56:44.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.148+0000 7fb795fad740 1 -- start start 2024-04-23T14:56:44.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.148+0000 7fb795fad740 1 Processor -- start 2024-04-23T14:56:44.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.149+0000 7fb795fad740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3daa68860 0x55a3daa77420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.149+0000 7fb795fad740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3daa69180 0x55a3daa77960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.149+0000 7fb795fad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3daa73850 0x55a3daa77ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.149+0000 7fb795fad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb78c1aed40 con 0x55a3daa73850 2024-04-23T14:56:44.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.149+0000 7fb795fad740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb78c1b01b0 con 0x55a3daa69180 2024-04-23T14:56:44.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.149+0000 7fb795fad740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb78c1af410 con 0x55a3daa68860 2024-04-23T14:56:44.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.149+0000 7fb795fad740 10 client.0 _ll_register_callbacks cb 0x7fb774009290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:56:44.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.149+0000 7fb78be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3daa68860 0x55a3daa77420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.149+0000 7fb783400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3daa69180 0x55a3daa77960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.149+0000 7fb78be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3daa68860 0x55a3daa77420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:35016/0 (socket says 172.21.15.28:35016) 2024-04-23T14:56:44.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.149+0000 7fb78be00640 1 -- 172.21.15.28:0/1619084393 learned_addr learned my addr 172.21.15.28:0/1619084393 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:44.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.149+0000 7fb791000640 1 --2- 172.21.15.28:0/1619084393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3daa73850 0x55a3daa77ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.150+0000 7fb783400640 1 -- 172.21.15.28:0/1619084393 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3daa68860 msgr2=0x55a3daa77420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:44.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.150+0000 7fb783400640 1 --2- 172.21.15.28:0/1619084393 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3daa68860 0x55a3daa77420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.150+0000 7fb783400640 1 -- 172.21.15.28:0/1619084393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3daa73850 msgr2=0x55a3daa77ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:44.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.150+0000 7fb783400640 1 --2- 172.21.15.28:0/1619084393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3daa73850 0x55a3daa77ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.150+0000 7fb783400640 1 -- 172.21.15.28:0/1619084393 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3da6a00f0 con 0x55a3daa69180 2024-04-23T14:56:44.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.150+0000 7fb791000640 1 --2- 172.21.15.28:0/1619084393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3daa73850 0x55a3daa77ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:56:44.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.150+0000 7fb78be00640 1 --2- 172.21.15.28:0/1619084393 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3daa68860 0x55a3daa77420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:56:44.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.150+0000 7fb783400640 1 --2- 172.21.15.28:0/1619084393 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3daa69180 0x55a3daa77960 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x55a3daa861b0 tx=0x7fb7740f0e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0d63ed2201650df server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:44.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.150+0000 7fb77a000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:56:44.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.150+0000 7fb77a000640 1 -- 172.21.15.28:0/1619084393 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb774049070 con 0x55a3daa69180 2024-04-23T14:56:44.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.150+0000 7fb77a000640 1 -- 172.21.15.28:0/1619084393 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8542+0+0 (secure 0 0 0) 0x7fb77403bba0 con 0x55a3daa69180 2024-04-23T14:56:44.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.150+0000 7fb795fad740 1 -- 172.21.15.28:0/1619084393 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7740c5700 con 0x55a3daa69180 2024-04-23T14:56:44.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.150+0000 7fb77a000640 1 -- 172.21.15.28:0/1619084393 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fb7740baf00 con 0x55a3daa69180 2024-04-23T14:56:44.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.150+0000 7fb795fad740 10 client.27194 fetch_fsmap learned FSMap version 20 2024-04-23T14:56:44.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.150+0000 7fb795fad740 1 -- 172.21.15.28:0/1619084393 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55a3da993810 con 0x55a3daa69180 2024-04-23T14:56:44.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.151+0000 7fb77a000640 1 -- 172.21.15.28:0/1619084393 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7740c5700 con 0x55a3daa69180 2024-04-23T14:56:44.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.151+0000 7fb77a000640 1 -- 172.21.15.28:0/1619084393 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fb7740bba00 con 0x55a3daa69180 2024-04-23T14:56:44.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.151+0000 7fb795fad740 10 client.27194 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:56:44.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.151+0000 7fb795fad740 10 client.27194 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:56:44.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.151+0000 7fb795fad740 20 client.27194 populate_metadata read hostname 'smithi028' 2024-04-23T14:56:44.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.151+0000 7fb795fad740 1 --2- 172.21.15.28:0/1619084393 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55a3daaea210 0x55a3daaec600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.151+0000 7fb795fad740 4 client.27194 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:56:44.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.151+0000 7fb795fad740 1 -- 172.21.15.28:0/1619084393 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a3da9d5a80 con 0x55a3daaea210 2024-04-23T14:56:44.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.151+0000 7fb78be00640 1 --2- 172.21.15.28:0/1619084393 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55a3daaea210 0x55a3daaec600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.152+0000 7fb78be00640 1 --2- 172.21.15.28:0/1619084393 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55a3daaea210 0x55a3daaec600 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6cdce402829ee server_cookie=a11feba568d23f28 in_seq=0 out_seq=0 2024-04-23T14:56:44.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.152+0000 7fb77a000640 10 client.27194 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:56:44.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.152+0000 7fb77a000640 1 -- 172.21.15.28:0/1619084393 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a3da9d5a80 con 0x55a3daaea210 2024-04-23T14:56:44.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.152+0000 7fb77a000640 10 client.27194 handle_command_reply: tid=0 2024-04-23T14:56:44.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.152+0000 7fb795fad740 1 client.27194 shutdown 2024-04-23T14:56:44.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.152+0000 7fb795fad740 1 -- 172.21.15.28:0/1619084393 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3daa69180 msgr2=0x55a3daa77960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:44.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.152+0000 7fb795fad740 1 --2- 172.21.15.28:0/1619084393 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3daa69180 0x55a3daa77960 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x55a3daa861b0 tx=0x7fb7740f0e60 comp rx=0 tx=0).stop 2024-04-23T14:56:44.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.152+0000 7fb795fad740 1 -- 172.21.15.28:0/1619084393 shutdown_connections 2024-04-23T14:56:44.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.152+0000 7fb795fad740 1 --2- 172.21.15.28:0/1619084393 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55a3daaea210 0x55a3daaec600 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.152+0000 7fb795fad740 1 --2- 172.21.15.28:0/1619084393 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3daa69180 0x55a3daa77960 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.152+0000 7fb795fad740 1 --2- 172.21.15.28:0/1619084393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3daa73850 0x55a3daa77ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.152+0000 7fb795fad740 1 --2- 172.21.15.28:0/1619084393 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a3daa68860 0x55a3daa77420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.152+0000 7fb795fad740 1 -- 172.21.15.28:0/1619084393 >> 172.21.15.28:0/1619084393 conn(0x55a3daa64b80 msgr2=0x55a3daa707a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:44.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.152+0000 7fb77a000640 0 client.27194 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:56:44.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.152+0000 7fb795fad740 1 -- 172.21.15.28:0/1619084393 shutdown_connections 2024-04-23T14:56:44.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.152+0000 7fb795fad740 1 -- 172.21.15.28:0/1619084393 wait complete. 2024-04-23T14:56:44.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.153+0000 7fb795fad740 20 client.27194 trim_cache size 0 max 16384 2024-04-23T14:56:44.161 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:56:44.161 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-23T14:56:44.161 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-23T14:56:44.161 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:56:44.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.153+0000 7fb793000640 1 -- 172.21.15.28:0/1971173634 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb7700a3da0 msgr2=0x7fb7700a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:44.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.153+0000 7fb793000640 1 --2- 172.21.15.28:0/1971173634 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb7700a3da0 0x7fb7700a6260 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x55a3daa69e20 tx=0x7fb77c005f90 comp rx=0 tx=0).stop 2024-04-23T14:56:44.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.153+0000 7fb793000640 1 -- 172.21.15.28:0/1971173634 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb78c109580 msgr2=0x7fb78c1a0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:44.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.153+0000 7fb793000640 1 --2- 172.21.15.28:0/1971173634 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb78c109580 0x7fb78c1a0db0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fb774005940 tx=0x7fb7740040c0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.153+0000 7fb793000640 1 -- 172.21.15.28:0/1971173634 shutdown_connections 2024-04-23T14:56:44.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.153+0000 7fb793000640 1 --2- 172.21.15.28:0/1971173634 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb7700a3da0 0x7fb7700a6260 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.153+0000 7fb793000640 1 --2- 172.21.15.28:0/1971173634 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb78c109580 0x7fb78c1a0db0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.153+0000 7fb793000640 1 --2- 172.21.15.28:0/1971173634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb78c19aa70 0x7fb78c19af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.153+0000 7fb793000640 1 --2- 172.21.15.28:0/1971173634 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb78c1044b0 0x7fb78c1a0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.153+0000 7fb793000640 1 -- 172.21.15.28:0/1971173634 >> 172.21.15.28:0/1971173634 conn(0x7fb78c0ffd90 msgr2=0x7fb78c101120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:44.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.154+0000 7fb793000640 1 -- 172.21.15.28:0/1971173634 shutdown_connections 2024-04-23T14:56:44.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.154+0000 7fb793000640 1 -- 172.21.15.28:0/1971173634 wait complete. 2024-04-23T14:56:44.172 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-23T14:56:44.172 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:56:44.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.387+0000 7fb571000640 1 -- 172.21.15.28:0/1873136815 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb56c104df0 msgr2=0x7fb56c108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:44.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.387+0000 7fb571000640 1 --2- 172.21.15.28:0/1873136815 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb56c104df0 0x7fb56c108e30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb558009900 tx=0x7fb55802a410 comp rx=0 tx=0).stop 2024-04-23T14:56:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.387+0000 7fb571000640 1 -- 172.21.15.28:0/1873136815 shutdown_connections 2024-04-23T14:56:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.387+0000 7fb571000640 1 --2- 172.21.15.28:0/1873136815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb56c109510 0x7fb56c1110e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.387+0000 7fb571000640 1 --2- 172.21.15.28:0/1873136815 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb56c104df0 0x7fb56c108e30 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.387+0000 7fb571000640 1 --2- 172.21.15.28:0/1873136815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb56c104440 0x7fb56c104820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.387+0000 7fb571000640 1 -- 172.21.15.28:0/1873136815 >> 172.21.15.28:0/1873136815 conn(0x7fb56c0ffd00 msgr2=0x7fb56c102120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.388+0000 7fb571000640 1 -- 172.21.15.28:0/1873136815 shutdown_connections 2024-04-23T14:56:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.388+0000 7fb571000640 1 -- 172.21.15.28:0/1873136815 wait complete. 2024-04-23T14:56:44.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.388+0000 7fb571000640 1 Processor -- start 2024-04-23T14:56:44.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.389+0000 7fb571000640 1 -- start start 2024-04-23T14:56:44.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.389+0000 7fb571000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb56c104440 0x7fb56c1a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.389+0000 7fb571000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb56c109510 0x7fb56c1a0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.389+0000 7fb571000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb56c19a950 0x7fb56c19ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.389+0000 7fb571000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb56c1145d0 con 0x7fb56c19a950 2024-04-23T14:56:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.389+0000 7fb571000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb56c114450 con 0x7fb56c104440 2024-04-23T14:56:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.389+0000 7fb571000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb56c114750 con 0x7fb56c109510 2024-04-23T14:56:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.389+0000 7fb569e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb56c104440 0x7fb56c1a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.389+0000 7fb56a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb56c19a950 0x7fb56c19ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.389+0000 7fb569e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb56c104440 0x7fb56c1a0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:47942/0 (socket says 172.21.15.28:47942) 2024-04-23T14:56:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.389+0000 7fb569e00640 1 -- 172.21.15.28:0/2270749489 learned_addr learned my addr 172.21.15.28:0/2270749489 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.390+0000 7fb569400640 1 --2- 172.21.15.28:0/2270749489 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb56c109510 0x7fb56c1a0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.390+0000 7fb569e00640 1 -- 172.21.15.28:0/2270749489 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb56c109510 msgr2=0x7fb56c1a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.390+0000 7fb569e00640 1 --2- 172.21.15.28:0/2270749489 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb56c109510 0x7fb56c1a0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.390+0000 7fb569e00640 1 -- 172.21.15.28:0/2270749489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb56c19a950 msgr2=0x7fb56c19ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.390+0000 7fb569e00640 1 --2- 172.21.15.28:0/2270749489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb56c19a950 0x7fb56c19ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.390+0000 7fb569e00640 1 -- 172.21.15.28:0/2270749489 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb56c19b470 con 0x7fb56c104440 2024-04-23T14:56:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.390+0000 7fb56a800640 1 --2- 172.21.15.28:0/2270749489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb56c19a950 0x7fb56c19ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:56:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.390+0000 7fb569e00640 1 --2- 172.21.15.28:0/2270749489 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb56c104440 0x7fb56c1a0750 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fb55c00e9f0 tx=0x7fb55c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.390+0000 7fb562a00640 1 -- 172.21.15.28:0/2270749489 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55c00cd80 con 0x7fb56c104440 2024-04-23T14:56:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.391+0000 7fb571000640 1 -- 172.21.15.28:0/2270749489 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb56c19b760 con 0x7fb56c104440 2024-04-23T14:56:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.391+0000 7fb562a00640 1 -- 172.21.15.28:0/2270749489 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb55c010910 con 0x7fb56c104440 2024-04-23T14:56:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.391+0000 7fb562a00640 1 -- 172.21.15.28:0/2270749489 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55c0188f0 con 0x7fb56c104440 2024-04-23T14:56:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.391+0000 7fb571000640 1 -- 172.21.15.28:0/2270749489 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb56c1a79b0 con 0x7fb56c104440 2024-04-23T14:56:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.392+0000 7fb571000640 1 -- 172.21.15.28:0/2270749489 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb56c069270 con 0x7fb56c104440 2024-04-23T14:56:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.394+0000 7fb562a00640 1 -- 172.21.15.28:0/2270749489 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fb55c010ac0 con 0x7fb56c104440 2024-04-23T14:56:44.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.395+0000 7fb562a00640 1 --2- 172.21.15.28:0/2270749489 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb5500a3da0 0x7fb5500a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.396+0000 7fb562a00640 1 -- 172.21.15.28:0/2270749489 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8542+0+0 (secure 0 0 0) 0x7fb55c0d0b90 con 0x7fb56c104440 2024-04-23T14:56:44.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.396+0000 7fb569400640 1 --2- 172.21.15.28:0/2270749489 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb5500a3da0 0x7fb5500a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.397+0000 7fb569400640 1 --2- 172.21.15.28:0/2270749489 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb5500a3da0 0x7fb5500a6260 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb558006010 tx=0x7fb558000bc0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:44.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.401+0000 7fb562a00640 1 -- 172.21.15.28:0/2270749489 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb55c09cb70 con 0x7fb56c104440 2024-04-23T14:56:44.544 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1080018 8.51 MB/sec execute 540 sec latency 20.113 ms 2024-04-23T14:56:44.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.664+0000 7fb571000640 1 -- 172.21.15.28:0/2270749489 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb56c19be30 con 0x7fb56c104440 2024-04-23T14:56:44.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.665+0000 7fb562a00640 1 -- 172.21.15.28:0/2270749489 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7fb55c0a0e70 con 0x7fb56c104440 2024-04-23T14:56:44.666 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:56:44.666 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:56:44.667 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:56:44.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.668+0000 7fb571000640 1 -- 172.21.15.28:0/2270749489 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb5500a3da0 msgr2=0x7fb5500a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:44.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.668+0000 7fb571000640 1 --2- 172.21.15.28:0/2270749489 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb5500a3da0 0x7fb5500a6260 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb558006010 tx=0x7fb558000bc0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.668+0000 7fb571000640 1 -- 172.21.15.28:0/2270749489 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb56c104440 msgr2=0x7fb56c1a0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:44.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.668+0000 7fb571000640 1 --2- 172.21.15.28:0/2270749489 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb56c104440 0x7fb56c1a0750 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fb55c00e9f0 tx=0x7fb55c00eeb0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.668+0000 7fb571000640 1 -- 172.21.15.28:0/2270749489 shutdown_connections 2024-04-23T14:56:44.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.668+0000 7fb571000640 1 --2- 172.21.15.28:0/2270749489 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb5500a3da0 0x7fb5500a6260 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.668+0000 7fb571000640 1 --2- 172.21.15.28:0/2270749489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb56c19a950 0x7fb56c19ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.669+0000 7fb571000640 1 --2- 172.21.15.28:0/2270749489 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb56c109510 0x7fb56c1a0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.669+0000 7fb571000640 1 --2- 172.21.15.28:0/2270749489 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb56c104440 0x7fb56c1a0750 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.669+0000 7fb571000640 1 -- 172.21.15.28:0/2270749489 >> 172.21.15.28:0/2270749489 conn(0x7fb56c0ffd00 msgr2=0x7fb56c101070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:44.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.669+0000 7fb571000640 1 -- 172.21.15.28:0/2270749489 shutdown_connections 2024-04-23T14:56:44.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.669+0000 7fb571000640 1 -- 172.21.15.28:0/2270749489 wait complete. 2024-04-23T14:56:44.690 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-23T14:56:44.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.909+0000 7ff467000640 1 -- 172.21.15.28:0/1163572549 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff460076530 msgr2=0x7ff460076910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:44.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.909+0000 7ff467000640 1 --2- 172.21.15.28:0/1163572549 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff460076530 0x7ff460076910 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7ff450009cd0 tx=0x7ff45002a8f0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.909+0000 7ff467000640 1 -- 172.21.15.28:0/1163572549 shutdown_connections 2024-04-23T14:56:44.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.909+0000 7ff467000640 1 --2- 172.21.15.28:0/1163572549 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff460112ee0 0x7ff4601152d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.909+0000 7ff467000640 1 --2- 172.21.15.28:0/1163572549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff460076e50 0x7ff4601129a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.909+0000 7ff467000640 1 --2- 172.21.15.28:0/1163572549 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff460076530 0x7ff460076910 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.909+0000 7ff467000640 1 -- 172.21.15.28:0/1163572549 >> 172.21.15.28:0/1163572549 conn(0x7ff4600ffd90 msgr2=0x7ff4601021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:44.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.909+0000 7ff467000640 1 -- 172.21.15.28:0/1163572549 shutdown_connections 2024-04-23T14:56:44.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.909+0000 7ff467000640 1 -- 172.21.15.28:0/1163572549 wait complete. 2024-04-23T14:56:44.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.910+0000 7ff467000640 1 Processor -- start 2024-04-23T14:56:44.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.910+0000 7ff467000640 1 -- start start 2024-04-23T14:56:44.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.911+0000 7ff467000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff460076530 0x7ff46019eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.911+0000 7ff467000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff460076e50 0x7ff46019f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.911+0000 7ff467000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff460112ee0 0x7ff4601a38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.911+0000 7ff467000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff460118a90 con 0x7ff460112ee0 2024-04-23T14:56:44.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.911+0000 7ff467000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff460118910 con 0x7ff460076530 2024-04-23T14:56:44.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.911+0000 7ff467000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff460118c10 con 0x7ff460076e50 2024-04-23T14:56:44.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.911+0000 7ff45fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff460076530 0x7ff46019eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.911+0000 7ff45fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff460076530 0x7ff46019eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:47968/0 (socket says 172.21.15.28:47968) 2024-04-23T14:56:44.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.911+0000 7ff45fe00640 1 -- 172.21.15.28:0/2161258637 learned_addr learned my addr 172.21.15.28:0/2161258637 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:44.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.911+0000 7ff465000640 1 --2- 172.21.15.28:0/2161258637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff460112ee0 0x7ff4601a38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.911+0000 7ff457400640 1 --2- 172.21.15.28:0/2161258637 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff460076e50 0x7ff46019f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.912+0000 7ff45fe00640 1 -- 172.21.15.28:0/2161258637 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff460076e50 msgr2=0x7ff46019f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:44.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.912+0000 7ff45fe00640 1 --2- 172.21.15.28:0/2161258637 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff460076e50 0x7ff46019f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.912+0000 7ff45fe00640 1 -- 172.21.15.28:0/2161258637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff460112ee0 msgr2=0x7ff4601a38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:44.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.912+0000 7ff45fe00640 1 --2- 172.21.15.28:0/2161258637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff460112ee0 0x7ff4601a38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.912+0000 7ff45fe00640 1 -- 172.21.15.28:0/2161258637 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4601a3fe0 con 0x7ff460076530 2024-04-23T14:56:44.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.912+0000 7ff45fe00640 1 --2- 172.21.15.28:0/2161258637 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff460076530 0x7ff46019eff0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7ff450000f80 tx=0x7ff450003ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:44.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.912+0000 7ff45d600640 1 -- 172.21.15.28:0/2161258637 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff450030e40 con 0x7ff460076530 2024-04-23T14:56:44.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.912+0000 7ff45d600640 1 -- 172.21.15.28:0/2161258637 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff450040070 con 0x7ff460076530 2024-04-23T14:56:44.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.912+0000 7ff45d600640 1 -- 172.21.15.28:0/2161258637 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff450045a60 con 0x7ff460076530 2024-04-23T14:56:44.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.912+0000 7ff467000640 1 -- 172.21.15.28:0/2161258637 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4601a4270 con 0x7ff460076530 2024-04-23T14:56:44.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.913+0000 7ff467000640 1 -- 172.21.15.28:0/2161258637 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4601abb60 con 0x7ff460076530 2024-04-23T14:56:44.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.916+0000 7ff45d600640 1 -- 172.21.15.28:0/2161258637 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7ff450045c00 con 0x7ff460076530 2024-04-23T14:56:44.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.916+0000 7ff469e66740 1 Processor -- start 2024-04-23T14:56:44.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.916+0000 7ff469e66740 1 -- start start 2024-04-23T14:56:44.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.917+0000 7ff469e66740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5582d15e9950 0x5582d15e9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.917+0000 7ff469e66740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d15ea270 0x5582d15f4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.917+0000 7ff469e66740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582d15f4940 0x5582d15f6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.917+0000 7ff469e66740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5582d154f1d0 con 0x5582d15ea270 2024-04-23T14:56:44.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.917+0000 7ff469e66740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5582d14f5450 con 0x5582d15e9950 2024-04-23T14:56:44.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.917+0000 7ff469e66740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5582d154b190 con 0x5582d15f4940 2024-04-23T14:56:44.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.917+0000 7ff465000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5582d15e9950 0x5582d15e9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.917+0000 7ff465000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5582d15e9950 0x5582d15e9d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:47972/0 (socket says 172.21.15.28:47972) 2024-04-23T14:56:44.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.917+0000 7ff465000640 1 -- 172.21.15.28:0/1315000766 learned_addr learned my addr 172.21.15.28:0/1315000766 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:44.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.917+0000 7ff45fe00640 1 --2- 172.21.15.28:0/1315000766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d15ea270 0x5582d15f4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.917+0000 7ff457400640 1 --2- 172.21.15.28:0/1315000766 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582d15f4940 0x5582d15f6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.917+0000 7ff465000640 1 -- 172.21.15.28:0/1315000766 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582d15f4940 msgr2=0x5582d15f6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:44.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.917+0000 7ff465000640 1 --2- 172.21.15.28:0/1315000766 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582d15f4940 0x5582d15f6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.917+0000 7ff465000640 1 -- 172.21.15.28:0/1315000766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d15ea270 msgr2=0x5582d15f4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:44.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.917+0000 7ff465000640 1 --2- 172.21.15.28:0/1315000766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d15ea270 0x5582d15f4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.917+0000 7ff465000640 1 -- 172.21.15.28:0/1315000766 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582d15b2310 con 0x5582d15e9950 2024-04-23T14:56:44.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.918+0000 7ff465000640 1 --2- 172.21.15.28:0/1315000766 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5582d15e9950 0x5582d15e9d30 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7ff458009bb0 tx=0x7ff458011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f1944d70b889ec2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:44.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.918+0000 7ff45d600640 1 --2- 172.21.15.28:0/2161258637 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7ff4440a3da0 0x7ff4440a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.918+0000 7ff455600640 1 -- 172.21.15.28:0/1315000766 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff458004230 con 0x5582d15e9950 2024-04-23T14:56:44.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.918+0000 7ff455600640 1 -- 172.21.15.28:0/1315000766 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff458009e20 con 0x5582d15e9950 2024-04-23T14:56:44.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.918+0000 7ff455600640 1 -- 172.21.15.28:0/1315000766 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff458019780 con 0x5582d15e9950 2024-04-23T14:56:44.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.918+0000 7ff45d600640 1 -- 172.21.15.28:0/2161258637 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8542+0+0 (secure 0 0 0) 0x7ff4500f1650 con 0x7ff460076530 2024-04-23T14:56:44.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.918+0000 7ff45fe00640 1 --2- 172.21.15.28:0/2161258637 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7ff4440a3da0 0x7ff4440a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.919+0000 7ff469e66740 1 -- 172.21.15.28:0/1315000766 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5582d15e9950 msgr2=0x5582d15e9d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:44.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.919+0000 7ff469e66740 1 --2- 172.21.15.28:0/1315000766 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5582d15e9950 0x5582d15e9d30 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7ff458009bb0 tx=0x7ff458011810 comp rx=0 tx=0).stop 2024-04-23T14:56:44.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.919+0000 7ff45fe00640 1 --2- 172.21.15.28:0/2161258637 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7ff4440a3da0 0x7ff4440a6260 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x5582d15eaf10 tx=0x7ff450003f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:44.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.919+0000 7ff469e66740 1 -- 172.21.15.28:0/1315000766 shutdown_connections 2024-04-23T14:56:44.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.919+0000 7ff469e66740 1 --2- 172.21.15.28:0/1315000766 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582d15f4940 0x5582d15f6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.919+0000 7ff469e66740 1 --2- 172.21.15.28:0/1315000766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d15ea270 0x5582d15f4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.919+0000 7ff469e66740 1 --2- 172.21.15.28:0/1315000766 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5582d15e9950 0x5582d15e9d30 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.919+0000 7ff469e66740 1 -- 172.21.15.28:0/1315000766 >> 172.21.15.28:0/1315000766 conn(0x5582d15e5c70 msgr2=0x5582d15e6050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:44.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.919+0000 7ff469e66740 1 -- 172.21.15.28:0/1315000766 shutdown_connections 2024-04-23T14:56:44.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.919+0000 7ff469e66740 1 -- 172.21.15.28:0/1315000766 wait complete. 2024-04-23T14:56:44.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.920+0000 7ff469e66740 1 -- start start 2024-04-23T14:56:44.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.920+0000 7ff469e66740 1 Processor -- start 2024-04-23T14:56:44.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.920+0000 7ff469e66740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582d15e9950 0x5582d15f4370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.920+0000 7ff469e66740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5582d15ea270 0x5582d15f1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.921+0000 7ff469e66740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d15f4940 0x5582d15f1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.921+0000 7ff469e66740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5582d15f77d0 con 0x5582d15f4940 2024-04-23T14:56:44.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.921+0000 7ff469e66740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5582d1508810 con 0x5582d15ea270 2024-04-23T14:56:44.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.921+0000 7ff469e66740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5582d15f7950 con 0x5582d15e9950 2024-04-23T14:56:44.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.921+0000 7ff469e66740 10 client.0 _ll_register_callbacks cb 0x5582d14d6140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:56:44.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.921+0000 7ff457400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5582d15ea270 0x5582d15f1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.921+0000 7ff457400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5582d15ea270 0x5582d15f1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:47984/0 (socket says 172.21.15.28:47984) 2024-04-23T14:56:44.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.921+0000 7ff457400640 1 -- 172.21.15.28:0/1936323696 learned_addr learned my addr 172.21.15.28:0/1936323696 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:44.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.921+0000 7ff465000640 1 --2- 172.21.15.28:0/1936323696 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582d15e9950 0x5582d15f4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.921+0000 7ff465000640 1 --2- 172.21.15.28:0/1936323696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d15f4940 0x5582d15f1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.921+0000 7ff457400640 1 -- 172.21.15.28:0/1936323696 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582d15e9950 msgr2=0x5582d15f4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:44.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.921+0000 7ff457400640 1 --2- 172.21.15.28:0/1936323696 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582d15e9950 0x5582d15f4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.921+0000 7ff457400640 1 -- 172.21.15.28:0/1936323696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d15f4940 msgr2=0x5582d15f1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:44.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.921+0000 7ff457400640 1 --2- 172.21.15.28:0/1936323696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d15f4940 0x5582d15f1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.921+0000 7ff457400640 1 -- 172.21.15.28:0/1936323696 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582d15193f0 con 0x5582d15ea270 2024-04-23T14:56:44.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.921+0000 7ff465000640 1 --2- 172.21.15.28:0/1936323696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d15f4940 0x5582d15f1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:56:44.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.921+0000 7ff457400640 1 --2- 172.21.15.28:0/1936323696 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5582d15ea270 0x5582d15f1660 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7ff448012700 tx=0x7ff448012bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=900ea86ea46f9ac0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:44.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.922+0000 7ff44e000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:56:44.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.922+0000 7ff44e000640 1 -- 172.21.15.28:0/1936323696 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff448004050 con 0x5582d15ea270 2024-04-23T14:56:44.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.922+0000 7ff44e000640 1 -- 172.21.15.28:0/1936323696 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8542+0+0 (secure 0 0 0) 0x7ff4480041f0 con 0x5582d15ea270 2024-04-23T14:56:44.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.922+0000 7ff469e66740 1 -- 172.21.15.28:0/1936323696 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff458009e20 con 0x5582d15ea270 2024-04-23T14:56:44.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.922+0000 7ff44e000640 1 -- 172.21.15.28:0/1936323696 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7ff4480043f0 con 0x5582d15ea270 2024-04-23T14:56:44.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.922+0000 7ff469e66740 10 client.27212 fetch_fsmap learned FSMap version 20 2024-04-23T14:56:44.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.922+0000 7ff469e66740 1 -- 172.21.15.28:0/1936323696 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5582d12613f0 con 0x5582d15ea270 2024-04-23T14:56:44.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.923+0000 7ff44e000640 1 -- 172.21.15.28:0/1936323696 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff458009e20 con 0x5582d15ea270 2024-04-23T14:56:44.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.923+0000 7ff44e000640 1 -- 172.21.15.28:0/1936323696 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7ff448003be0 con 0x5582d15ea270 2024-04-23T14:56:44.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.923+0000 7ff469e66740 10 client.27212 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:56:44.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.923+0000 7ff469e66740 10 client.27212 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:56:44.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.923+0000 7ff469e66740 20 client.27212 populate_metadata read hostname 'smithi028' 2024-04-23T14:56:44.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.923+0000 7ff469e66740 1 --2- 172.21.15.28:0/1936323696 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5582d1604c80 0x5582d1607070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.923+0000 7ff469e66740 4 client.27212 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:56:44.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.923+0000 7ff469e66740 1 -- 172.21.15.28:0/1936323696 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5582d155cbf0 con 0x5582d1604c80 2024-04-23T14:56:44.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.923+0000 7ff465000640 1 --2- 172.21.15.28:0/1936323696 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5582d1604c80 0x5582d1607070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.924+0000 7ff465000640 1 --2- 172.21.15.28:0/1936323696 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5582d1604c80 0x5582d1607070 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80ab047034b6e7e6 server_cookie=55f27434e98e1514 in_seq=0 out_seq=0 2024-04-23T14:56:44.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.924+0000 7ff44e000640 10 client.27212 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:56:44.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.925+0000 7ff44e000640 1 -- 172.21.15.28:0/1936323696 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5582d155cbf0 con 0x5582d1604c80 2024-04-23T14:56:44.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.925+0000 7ff44e000640 10 client.27212 handle_command_reply: tid=0 2024-04-23T14:56:44.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.925+0000 7ff469e66740 1 client.27212 shutdown 2024-04-23T14:56:44.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.925+0000 7ff469e66740 1 -- 172.21.15.28:0/1936323696 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5582d15ea270 msgr2=0x5582d15f1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:44.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.925+0000 7ff469e66740 1 --2- 172.21.15.28:0/1936323696 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5582d15ea270 0x5582d15f1660 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7ff448012700 tx=0x7ff448012bc0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.925+0000 7ff469e66740 1 -- 172.21.15.28:0/1936323696 shutdown_connections 2024-04-23T14:56:44.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.925+0000 7ff469e66740 1 --2- 172.21.15.28:0/1936323696 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5582d1604c80 0x5582d1607070 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.925+0000 7ff469e66740 1 --2- 172.21.15.28:0/1936323696 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5582d15ea270 0x5582d15f1660 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.925+0000 7ff469e66740 1 --2- 172.21.15.28:0/1936323696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d15f4940 0x5582d15f1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.925+0000 7ff469e66740 1 --2- 172.21.15.28:0/1936323696 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582d15e9950 0x5582d15f4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.925+0000 7ff44e000640 0 client.27212 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:56:44.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.925+0000 7ff469e66740 1 -- 172.21.15.28:0/1936323696 >> 172.21.15.28:0/1936323696 conn(0x5582d15e5c70 msgr2=0x5582d15eb930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:44.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.925+0000 7ff469e66740 1 -- 172.21.15.28:0/1936323696 shutdown_connections 2024-04-23T14:56:44.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.925+0000 7ff469e66740 1 -- 172.21.15.28:0/1936323696 wait complete. 2024-04-23T14:56:44.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.926+0000 7ff469e66740 20 client.27212 trim_cache size 0 max 16384 2024-04-23T14:56:44.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.942+0000 7ff469e66740 1 Processor -- start 2024-04-23T14:56:44.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.943+0000 7ff469e66740 1 -- start start 2024-04-23T14:56:44.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.943+0000 7ff469e66740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582d15e9950 0x5582d1600770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.943+0000 7ff469e66740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d15ea270 0x5582d15f3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.943+0000 7ff469e66740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5582d15f4940 0x5582d15f43e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.943+0000 7ff469e66740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff4601b2420 con 0x5582d15ea270 2024-04-23T14:56:44.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.943+0000 7ff469e66740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff4601b4c80 con 0x5582d15f4940 2024-04-23T14:56:44.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.943+0000 7ff469e66740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff4601b2a70 con 0x5582d15e9950 2024-04-23T14:56:44.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.943+0000 7ff465000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5582d15f4940 0x5582d15f43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.943+0000 7ff465000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5582d15f4940 0x5582d15f43e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:47996/0 (socket says 172.21.15.28:47996) 2024-04-23T14:56:44.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.943+0000 7ff465000640 1 -- 172.21.15.28:0/3605081588 learned_addr learned my addr 172.21.15.28:0/3605081588 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:44.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.943+0000 7ff457400640 1 --2- 172.21.15.28:0/3605081588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d15ea270 0x5582d15f3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.943+0000 7ff465000640 1 --2- 172.21.15.28:0/3605081588 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582d15e9950 0x5582d1600770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.944+0000 7ff465000640 1 -- 172.21.15.28:0/3605081588 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582d15e9950 msgr2=0x5582d1600770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:44.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.944+0000 7ff465000640 1 --2- 172.21.15.28:0/3605081588 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582d15e9950 0x5582d1600770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.944+0000 7ff465000640 1 -- 172.21.15.28:0/3605081588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d15ea270 msgr2=0x5582d15f3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:44.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.944+0000 7ff465000640 1 --2- 172.21.15.28:0/3605081588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d15ea270 0x5582d15f3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.944+0000 7ff465000640 1 -- 172.21.15.28:0/3605081588 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582d15f1fe0 con 0x5582d15f4940 2024-04-23T14:56:44.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.944+0000 7ff457400640 1 --2- 172.21.15.28:0/3605081588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d15ea270 0x5582d15f3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:56:44.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.944+0000 7ff465000640 1 --2- 172.21.15.28:0/3605081588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5582d15f4940 0x5582d15f43e0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ff458003f10 tx=0x7ff458005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=309d39e402131d12 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:44.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.944+0000 7ff454c00640 1 -- 172.21.15.28:0/3605081588 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff458005040 con 0x5582d15f4940 2024-04-23T14:56:44.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.944+0000 7ff454c00640 1 -- 172.21.15.28:0/3605081588 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4580035f0 con 0x5582d15f4940 2024-04-23T14:56:44.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.944+0000 7ff454c00640 1 -- 172.21.15.28:0/3605081588 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff45802c2a0 con 0x5582d15f4940 2024-04-23T14:56:44.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.945+0000 7ff469e66740 1 -- 172.21.15.28:0/3605081588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5582d15f4940 msgr2=0x5582d15f43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:44.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.945+0000 7ff469e66740 1 --2- 172.21.15.28:0/3605081588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5582d15f4940 0x5582d15f43e0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ff458003f10 tx=0x7ff458005f90 comp rx=0 tx=0).stop 2024-04-23T14:56:44.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.945+0000 7ff469e66740 1 -- 172.21.15.28:0/3605081588 shutdown_connections 2024-04-23T14:56:44.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.945+0000 7ff469e66740 1 --2- 172.21.15.28:0/3605081588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5582d15f4940 0x5582d15f43e0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.945+0000 7ff469e66740 1 --2- 172.21.15.28:0/3605081588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d15ea270 0x5582d15f3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.945+0000 7ff469e66740 1 --2- 172.21.15.28:0/3605081588 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582d15e9950 0x5582d1600770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.945+0000 7ff469e66740 1 -- 172.21.15.28:0/3605081588 >> 172.21.15.28:0/3605081588 conn(0x5582d15e5c70 msgr2=0x5582d15ec8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:44.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.945+0000 7ff469e66740 1 -- 172.21.15.28:0/3605081588 shutdown_connections 2024-04-23T14:56:44.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.945+0000 7ff469e66740 1 -- 172.21.15.28:0/3605081588 wait complete. 2024-04-23T14:56:44.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.945+0000 7ff469e66740 1 -- start start 2024-04-23T14:56:44.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.946+0000 7ff469e66740 1 Processor -- start 2024-04-23T14:56:44.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.946+0000 7ff469e66740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5582d15e9950 0x5582d15f5a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.946+0000 7ff469e66740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d15ea270 0x5582d15f5f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.946+0000 7ff469e66740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582d15f4940 0x5582d15fb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.946+0000 7ff469e66740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff4601b3140 con 0x5582d15ea270 2024-04-23T14:56:44.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.946+0000 7ff469e66740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff4601b45b0 con 0x5582d15e9950 2024-04-23T14:56:44.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.946+0000 7ff469e66740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff4601b3810 con 0x5582d15f4940 2024-04-23T14:56:44.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.946+0000 7ff469e66740 10 client.0 _ll_register_callbacks cb 0x7ff4480097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:56:44.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.946+0000 7ff457400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d15ea270 0x5582d15f5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.946+0000 7ff457400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d15ea270 0x5582d15f5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49218/0 (socket says 172.21.15.28:49218) 2024-04-23T14:56:44.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.946+0000 7ff457400640 1 -- 172.21.15.28:0/628627454 learned_addr learned my addr 172.21.15.28:0/628627454 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:44.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.946+0000 7ff465000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5582d15e9950 0x5582d15f5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.947+0000 7ff465000640 1 --2- 172.21.15.28:0/628627454 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582d15f4940 0x5582d15fb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.947+0000 7ff457400640 1 -- 172.21.15.28:0/628627454 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582d15f4940 msgr2=0x5582d15fb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:44.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.947+0000 7ff457400640 1 --2- 172.21.15.28:0/628627454 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582d15f4940 0x5582d15fb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.947+0000 7ff457400640 1 -- 172.21.15.28:0/628627454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5582d15e9950 msgr2=0x5582d15f5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:44.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.947+0000 7ff457400640 1 --2- 172.21.15.28:0/628627454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5582d15e9950 0x5582d15f5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.947+0000 7ff457400640 1 -- 172.21.15.28:0/628627454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582d132b7d0 con 0x5582d15ea270 2024-04-23T14:56:44.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.947+0000 7ff465000640 1 --2- 172.21.15.28:0/628627454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5582d15e9950 0x5582d15f5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:56:44.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.947+0000 7ff457400640 1 --2- 172.21.15.28:0/628627454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d15ea270 0x5582d15f5f80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x5582d15f2c30 tx=0x7ff4480108d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cfb055bf44e5208 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:44.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.947+0000 7ff44e000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-23T14:56:44.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.948+0000 7ff44e000640 1 -- 172.21.15.28:0/628627454 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff448029040 con 0x5582d15ea270 2024-04-23T14:56:44.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.948+0000 7ff44e000640 1 -- 172.21.15.28:0/628627454 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8542+0+0 (secure 0 0 0) 0x7ff448004000 con 0x5582d15ea270 2024-04-23T14:56:44.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.948+0000 7ff469e66740 1 -- 172.21.15.28:0/628627454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4580035f0 con 0x5582d15ea270 2024-04-23T14:56:44.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.948+0000 7ff44e000640 1 -- 172.21.15.28:0/628627454 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7ff44803b050 con 0x5582d15ea270 2024-04-23T14:56:44.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.948+0000 7ff469e66740 10 client.17019 fetch_fsmap learned FSMap version 20 2024-04-23T14:56:44.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.948+0000 7ff469e66740 1 -- 172.21.15.28:0/628627454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5582d166b780 con 0x5582d15ea270 2024-04-23T14:56:44.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.948+0000 7ff44e000640 1 -- 172.21.15.28:0/628627454 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4580035f0 con 0x5582d15ea270 2024-04-23T14:56:44.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.949+0000 7ff44e000640 1 -- 172.21.15.28:0/628627454 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7ff44802c070 con 0x5582d15ea270 2024-04-23T14:56:44.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.949+0000 7ff469e66740 10 client.17019 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:56:44.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.949+0000 7ff469e66740 10 client.17019 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:56:44.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.949+0000 7ff469e66740 20 client.17019 populate_metadata read hostname 'smithi028' 2024-04-23T14:56:44.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.949+0000 7ff469e66740 1 --2- 172.21.15.28:0/628627454 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5582d166bec0 0x5582d166e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:44.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.949+0000 7ff469e66740 4 client.17019 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-23T14:56:44.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.949+0000 7ff469e66740 1 -- 172.21.15.28:0/628627454 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5582d1556a60 con 0x5582d166bec0 2024-04-23T14:56:44.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.949+0000 7ff465000640 1 --2- 172.21.15.28:0/628627454 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5582d166bec0 0x5582d166e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:44.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.949+0000 7ff465000640 1 --2- 172.21.15.28:0/628627454 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5582d166bec0 0x5582d166e2e0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50253c06387bc48a server_cookie=d426a9369bc7375b in_seq=0 out_seq=0 2024-04-23T14:56:44.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.949+0000 7ff44e000640 10 client.17019 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:56:44.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.951+0000 7ff44e000640 1 -- 172.21.15.28:0/628627454 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5582d1556a60 con 0x5582d166bec0 2024-04-23T14:56:44.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.951+0000 7ff44e000640 10 client.17019 handle_command_reply: tid=0 2024-04-23T14:56:44.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.952+0000 7ff469e66740 1 client.17019 shutdown 2024-04-23T14:56:44.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.952+0000 7ff469e66740 1 -- 172.21.15.28:0/628627454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d15ea270 msgr2=0x5582d15f5f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:44.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.952+0000 7ff469e66740 1 --2- 172.21.15.28:0/628627454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d15ea270 0x5582d15f5f80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x5582d15f2c30 tx=0x7ff4480108d0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.952+0000 7ff469e66740 1 -- 172.21.15.28:0/628627454 shutdown_connections 2024-04-23T14:56:44.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.952+0000 7ff469e66740 1 --2- 172.21.15.28:0/628627454 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5582d166bec0 0x5582d166e2e0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.952+0000 7ff469e66740 1 --2- 172.21.15.28:0/628627454 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5582d15f4940 0x5582d15fb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.952+0000 7ff469e66740 1 --2- 172.21.15.28:0/628627454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d15ea270 0x5582d15f5f80 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.952+0000 7ff469e66740 1 --2- 172.21.15.28:0/628627454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5582d15e9950 0x5582d15f5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.952+0000 7ff469e66740 1 -- 172.21.15.28:0/628627454 >> 172.21.15.28:0/628627454 conn(0x5582d15e5c70 msgr2=0x5582d15e4560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:44.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.952+0000 7ff44e000640 0 client.17019 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:56:44.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.952+0000 7ff469e66740 1 -- 172.21.15.28:0/628627454 shutdown_connections 2024-04-23T14:56:44.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.952+0000 7ff469e66740 1 -- 172.21.15.28:0/628627454 wait complete. 2024-04-23T14:56:44.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.952+0000 7ff469e66740 20 client.17019 trim_cache size 0 max 16384 2024-04-23T14:56:44.955 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-23T14:56:44.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.953+0000 7ff467000640 1 -- 172.21.15.28:0/2161258637 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7ff4440a3da0 msgr2=0x7ff4440a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:44.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.953+0000 7ff467000640 1 --2- 172.21.15.28:0/2161258637 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7ff4440a3da0 0x7ff4440a6260 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x5582d15eaf10 tx=0x7ff450003f90 comp rx=0 tx=0).stop 2024-04-23T14:56:44.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.953+0000 7ff467000640 1 -- 172.21.15.28:0/2161258637 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff460076530 msgr2=0x7ff46019eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:44.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.953+0000 7ff467000640 1 --2- 172.21.15.28:0/2161258637 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff460076530 0x7ff46019eff0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7ff450000f80 tx=0x7ff450003ea0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.953+0000 7ff467000640 1 -- 172.21.15.28:0/2161258637 shutdown_connections 2024-04-23T14:56:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.953+0000 7ff467000640 1 --2- 172.21.15.28:0/2161258637 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7ff4440a3da0 0x7ff4440a6260 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.953+0000 7ff467000640 1 --2- 172.21.15.28:0/2161258637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff460112ee0 0x7ff4601a38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.953+0000 7ff467000640 1 --2- 172.21.15.28:0/2161258637 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff460076e50 0x7ff46019f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.953+0000 7ff467000640 1 --2- 172.21.15.28:0/2161258637 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff460076530 0x7ff46019eff0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.953+0000 7ff467000640 1 -- 172.21.15.28:0/2161258637 >> 172.21.15.28:0/2161258637 conn(0x7ff4600ffd90 msgr2=0x7ff460102180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.953+0000 7ff467000640 1 -- 172.21.15.28:0/2161258637 shutdown_connections 2024-04-23T14:56:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:44.953+0000 7ff467000640 1 -- 172.21.15.28:0/2161258637 wait complete. 2024-04-23T14:56:44.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:44 smithi028 ceph-mon[27810]: pgmap v396: 97 pgs: 97 active+clean; 607 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.2 MiB/s wr, 181 op/s 2024-04-23T14:56:44.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:44 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2270749489' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:56:45.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:44 smithi038 ceph-mon[32348]: pgmap v396: 97 pgs: 97 active+clean; 607 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.2 MiB/s wr, 181 op/s 2024-04-23T14:56:45.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:44 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2270749489' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:56:45.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:44 smithi105 ceph-mon[32555]: pgmap v396: 97 pgs: 97 active+clean; 607 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.2 MiB/s wr, 181 op/s 2024-04-23T14:56:45.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:44 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2270749489' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:56:45.545 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1081160 8.50 MB/sec execute 541 sec latency 26.192 ms 2024-04-23T14:56:45.974 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-23T14:56:45.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:56:45.974 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-23T14:56:46.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.194+0000 7fc7fde00640 1 -- 172.21.15.28:0/2095477000 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7f810c480 msgr2=0x7fc7f810e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:46.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.194+0000 7fc7fde00640 1 --2- 172.21.15.28:0/2095477000 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7f810c480 0x7fc7f810e920 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0009900 tx=0x7fc7f002a480 comp rx=0 tx=0).stop 2024-04-23T14:56:46.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.194+0000 7fc7fde00640 1 -- 172.21.15.28:0/2095477000 shutdown_connections 2024-04-23T14:56:46.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.194+0000 7fc7fde00640 1 --2- 172.21.15.28:0/2095477000 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7f810c480 0x7fc7f810e920 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.194+0000 7fc7fde00640 1 --2- 172.21.15.28:0/2095477000 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7f8103980 0x7fc7f810be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.194+0000 7fc7fde00640 1 --2- 172.21.15.28:0/2095477000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7f8103060 0x7fc7f8103440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.194+0000 7fc7fde00640 1 -- 172.21.15.28:0/2095477000 >> 172.21.15.28:0/2095477000 conn(0x7fc7f80fc880 msgr2=0x7fc7f80feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:46.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.194+0000 7fc7fde00640 1 -- 172.21.15.28:0/2095477000 shutdown_connections 2024-04-23T14:56:46.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.194+0000 7fc7fde00640 1 -- 172.21.15.28:0/2095477000 wait complete. 2024-04-23T14:56:46.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.195+0000 7fc7fde00640 1 Processor -- start 2024-04-23T14:56:46.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.195+0000 7fc7fde00640 1 -- start start 2024-04-23T14:56:46.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.196+0000 7fc7fde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7f8103060 0x7fc7f8198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.196+0000 7fc7fde00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7f8103980 0x7fc7f8198cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.196+0000 7fc7fde00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7f810c480 0x7fc7f819cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.196+0000 7fc7fde00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f8112260 con 0x7fc7f8103060 2024-04-23T14:56:46.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.196+0000 7fc7fde00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f81120e0 con 0x7fc7f8103980 2024-04-23T14:56:46.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.196+0000 7fc7fde00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f81123e0 con 0x7fc7f810c480 2024-04-23T14:56:46.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.196+0000 7fc7fca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7f8103060 0x7fc7f8198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.196+0000 7fc7fca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7f8103060 0x7fc7f8198780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49230/0 (socket says 172.21.15.28:49230) 2024-04-23T14:56:46.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.196+0000 7fc7fca00640 1 -- 172.21.15.28:0/3157801002 learned_addr learned my addr 172.21.15.28:0/3157801002 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:46.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.196+0000 7fc7f7e00640 1 --2- 172.21.15.28:0/3157801002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7f8103980 0x7fc7f8198cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.196+0000 7fc7fd400640 1 --2- 172.21.15.28:0/3157801002 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7f810c480 0x7fc7f819cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.197+0000 7fc7fca00640 1 -- 172.21.15.28:0/3157801002 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7f810c480 msgr2=0x7fc7f819cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:46.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.197+0000 7fc7fca00640 1 --2- 172.21.15.28:0/3157801002 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7f810c480 0x7fc7f819cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.197+0000 7fc7fca00640 1 -- 172.21.15.28:0/3157801002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7f8103980 msgr2=0x7fc7f8198cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:46.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.197+0000 7fc7fca00640 1 --2- 172.21.15.28:0/3157801002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7f8103980 0x7fc7f8198cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.197+0000 7fc7fca00640 1 -- 172.21.15.28:0/3157801002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7f819d720 con 0x7fc7f8103060 2024-04-23T14:56:46.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.197+0000 7fc7f7e00640 1 --2- 172.21.15.28:0/3157801002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7f8103980 0x7fc7f8198cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:56:46.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.197+0000 7fc7fd400640 1 --2- 172.21.15.28:0/3157801002 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7f810c480 0x7fc7f819cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:56:46.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.197+0000 7fc7fca00640 1 --2- 172.21.15.28:0/3157801002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7f8103060 0x7fc7f8198780 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc7e000e9f0 tx=0x7fc7e000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:46.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.197+0000 7fc7f5600640 1 -- 172.21.15.28:0/3157801002 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e000cd80 con 0x7fc7f8103060 2024-04-23T14:56:46.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.197+0000 7fc7f5600640 1 -- 172.21.15.28:0/3157801002 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7e0010910 con 0x7fc7f8103060 2024-04-23T14:56:46.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.197+0000 7fc7fde00640 1 -- 172.21.15.28:0/3157801002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7f819da10 con 0x7fc7f8103060 2024-04-23T14:56:46.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.197+0000 7fc7f5600640 1 -- 172.21.15.28:0/3157801002 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e0018990 con 0x7fc7f8103060 2024-04-23T14:56:46.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.197+0000 7fc7fde00640 1 -- 172.21.15.28:0/3157801002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7f81a52a0 con 0x7fc7f8103060 2024-04-23T14:56:46.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.201+0000 7fc802235740 1 Processor -- start 2024-04-23T14:56:46.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.201+0000 7fc7f5600640 1 -- 172.21.15.28:0/3157801002 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fc7e0010ac0 con 0x7fc7f8103060 2024-04-23T14:56:46.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.201+0000 7fc802235740 1 -- start start 2024-04-23T14:56:46.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.201+0000 7fc802235740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562e31354270 0x562e31354650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.201+0000 7fc802235740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e31354b90 0x562e3135ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.201+0000 7fc802235740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e3135f260 0x562e31361650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.201+0000 7fc802235740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562e312b9e70 con 0x562e31354b90 2024-04-23T14:56:46.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.201+0000 7fc802235740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562e31260380 con 0x562e3135f260 2024-04-23T14:56:46.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.201+0000 7fc802235740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562e312b5e30 con 0x562e31354270 2024-04-23T14:56:46.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.202+0000 7fc7f7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e3135f260 0x562e31361650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.202+0000 7fc7f7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e3135f260 0x562e31361650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48036/0 (socket says 172.21.15.28:48036) 2024-04-23T14:56:46.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.202+0000 7fc7f7e00640 1 -- 172.21.15.28:0/3352505079 learned_addr learned my addr 172.21.15.28:0/3352505079 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:46.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.202+0000 7fc7fd400640 1 --2- 172.21.15.28:0/3352505079 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562e31354270 0x562e31354650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.202+0000 7fc7fca00640 1 --2- 172.21.15.28:0/3352505079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e31354b90 0x562e3135ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.202+0000 7fc7f7e00640 1 -- 172.21.15.28:0/3352505079 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562e31354270 msgr2=0x562e31354650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:46.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.202+0000 7fc7f7e00640 1 --2- 172.21.15.28:0/3352505079 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562e31354270 0x562e31354650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.202+0000 7fc7f7e00640 1 -- 172.21.15.28:0/3352505079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e31354b90 msgr2=0x562e3135ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:46.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.202+0000 7fc7f7e00640 1 --2- 172.21.15.28:0/3352505079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e31354b90 0x562e3135ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.202+0000 7fc7f7e00640 1 -- 172.21.15.28:0/3352505079 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e310947e0 con 0x562e3135f260 2024-04-23T14:56:46.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.202+0000 7fc7f7e00640 1 --2- 172.21.15.28:0/3352505079 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e3135f260 0x562e31361650 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fc7e8011810 tx=0x7fc7e8011cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4047d335d5d8b67f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:46.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.203+0000 7fc7edc00640 1 -- 172.21.15.28:0/3352505079 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e8004d20 con 0x562e3135f260 2024-04-23T14:56:46.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.203+0000 7fc7edc00640 1 -- 172.21.15.28:0/3352505079 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7e800f940 con 0x562e3135f260 2024-04-23T14:56:46.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.203+0000 7fc7edc00640 1 -- 172.21.15.28:0/3352505079 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e80044c0 con 0x562e3135f260 2024-04-23T14:56:46.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.203+0000 7fc7f5600640 1 --2- 172.21.15.28:0/3157801002 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc7dc0a3d20 0x7fc7dc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.203+0000 7fc7f5600640 1 -- 172.21.15.28:0/3157801002 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8542+0+0 (secure 0 0 0) 0x7fc7e00d0c40 con 0x7fc7f8103060 2024-04-23T14:56:46.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.203+0000 7fc7fd400640 1 --2- 172.21.15.28:0/3157801002 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc7dc0a3d20 0x7fc7dc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.204+0000 7fc802235740 1 -- 172.21.15.28:0/3352505079 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e3135f260 msgr2=0x562e31361650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:46.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.204+0000 7fc802235740 1 --2- 172.21.15.28:0/3352505079 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e3135f260 0x562e31361650 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fc7e8011810 tx=0x7fc7e8011cd0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.204+0000 7fc802235740 1 -- 172.21.15.28:0/3352505079 shutdown_connections 2024-04-23T14:56:46.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.204+0000 7fc802235740 1 --2- 172.21.15.28:0/3352505079 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e3135f260 0x562e31361650 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.204+0000 7fc802235740 1 --2- 172.21.15.28:0/3352505079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e31354b90 0x562e3135ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.204+0000 7fc802235740 1 --2- 172.21.15.28:0/3352505079 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562e31354270 0x562e31354650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.204+0000 7fc802235740 1 -- 172.21.15.28:0/3352505079 >> 172.21.15.28:0/3352505079 conn(0x562e31350590 msgr2=0x562e31350970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:46.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.204+0000 7fc802235740 1 -- 172.21.15.28:0/3352505079 shutdown_connections 2024-04-23T14:56:46.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.204+0000 7fc7fd400640 1 --2- 172.21.15.28:0/3157801002 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc7dc0a3d20 0x7fc7dc0a61e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0005940 tx=0x7fc7f00058d0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:46.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.204+0000 7fc802235740 1 -- 172.21.15.28:0/3352505079 wait complete. 2024-04-23T14:56:46.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.205+0000 7fc802235740 1 -- start start 2024-04-23T14:56:46.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.205+0000 7fc802235740 1 Processor -- start 2024-04-23T14:56:46.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.205+0000 7fc802235740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562e31354270 0x562e3135c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.205+0000 7fc802235740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e31354b90 0x562e3135c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.205+0000 7fc802235740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3135f260 0x562e3135cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.205+0000 7fc802235740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562e31362050 con 0x562e3135f260 2024-04-23T14:56:46.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.205+0000 7fc802235740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562e31273660 con 0x562e31354b90 2024-04-23T14:56:46.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.205+0000 7fc802235740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562e313621d0 con 0x562e31354270 2024-04-23T14:56:46.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.205+0000 7fc802235740 10 client.0 _ll_register_callbacks cb 0x562e31261aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:56:46.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.206+0000 7fc7f7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e31354b90 0x562e3135c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.206+0000 7fc7f7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e31354b90 0x562e3135c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48044/0 (socket says 172.21.15.28:48044) 2024-04-23T14:56:46.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.206+0000 7fc7f7e00640 1 -- 172.21.15.28:0/764201470 learned_addr learned my addr 172.21.15.28:0/764201470 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:46.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.206+0000 7fc7fca00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562e31354270 0x562e3135c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.206+0000 7fc7fd400640 1 --2- 172.21.15.28:0/764201470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3135f260 0x562e3135cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.206+0000 7fc7f7e00640 1 -- 172.21.15.28:0/764201470 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562e31354270 msgr2=0x562e3135c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:46.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.206+0000 7fc7f7e00640 1 --2- 172.21.15.28:0/764201470 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562e31354270 0x562e3135c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.206+0000 7fc7f7e00640 1 -- 172.21.15.28:0/764201470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3135f260 msgr2=0x562e3135cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:46.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.206+0000 7fc7f7e00640 1 --2- 172.21.15.28:0/764201470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3135f260 0x562e3135cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.206+0000 7fc7f7e00640 1 -- 172.21.15.28:0/764201470 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e310f4f70 con 0x562e31354b90 2024-04-23T14:56:46.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.206+0000 7fc7fd400640 1 --2- 172.21.15.28:0/764201470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3135f260 0x562e3135cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:56:46.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.206+0000 7fc7f7e00640 1 --2- 172.21.15.28:0/764201470 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e31354b90 0x562e3135c700 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fc7e8006010 tx=0x7fc7e8003f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=355a6dcb3253995a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:46.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.206+0000 7fc7e6000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:56:46.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.207+0000 7fc7e6000640 1 -- 172.21.15.28:0/764201470 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e8004fd0 con 0x562e31354b90 2024-04-23T14:56:46.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.207+0000 7fc7e6000640 1 -- 172.21.15.28:0/764201470 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8542+0+0 (secure 0 0 0) 0x7fc7e8005170 con 0x562e31354b90 2024-04-23T14:56:46.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.207+0000 7fc802235740 1 -- 172.21.15.28:0/764201470 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7e800f940 con 0x562e31354b90 2024-04-23T14:56:46.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.207+0000 7fc7e6000640 1 -- 172.21.15.28:0/764201470 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fc7e8032050 con 0x562e31354b90 2024-04-23T14:56:46.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.207+0000 7fc802235740 10 client.27230 fetch_fsmap learned FSMap version 20 2024-04-23T14:56:46.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.207+0000 7fc802235740 1 -- 172.21.15.28:0/764201470 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x562e30f8c170 con 0x562e31354b90 2024-04-23T14:56:46.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.207+0000 7fc7e6000640 1 -- 172.21.15.28:0/764201470 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7e800f940 con 0x562e31354b90 2024-04-23T14:56:46.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.207+0000 7fc7e6000640 1 -- 172.21.15.28:0/764201470 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fc7e8018070 con 0x562e31354b90 2024-04-23T14:56:46.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.208+0000 7fc802235740 10 client.27230 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:56:46.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.208+0000 7fc802235740 10 client.27230 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:56:46.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.208+0000 7fc802235740 20 client.27230 populate_metadata read hostname 'smithi028' 2024-04-23T14:56:46.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.208+0000 7fc802235740 1 --2- 172.21.15.28:0/764201470 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562e31373980 0x562e31375d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.208+0000 7fc802235740 4 client.27230 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:56:46.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.208+0000 7fc802235740 1 -- 172.21.15.28:0/764201470 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e312c7890 con 0x562e31373980 2024-04-23T14:56:46.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.208+0000 7fc7fca00640 1 --2- 172.21.15.28:0/764201470 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562e31373980 0x562e31375d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.208+0000 7fc7fca00640 1 --2- 172.21.15.28:0/764201470 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562e31373980 0x562e31375d70 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8bbb47cab9e1bae8 server_cookie=384f5a1d620de216 in_seq=0 out_seq=0 2024-04-23T14:56:46.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.209+0000 7fc7e6000640 10 client.27230 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:56:46.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.209+0000 7fc7e6000640 1 -- 172.21.15.28:0/764201470 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562e312c7890 con 0x562e31373980 2024-04-23T14:56:46.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.209+0000 7fc7e6000640 10 client.27230 handle_command_reply: tid=0 2024-04-23T14:56:46.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.210+0000 7fc802235740 1 client.27230 shutdown 2024-04-23T14:56:46.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.210+0000 7fc802235740 1 -- 172.21.15.28:0/764201470 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e31354b90 msgr2=0x562e3135c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:46.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.210+0000 7fc802235740 1 --2- 172.21.15.28:0/764201470 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e31354b90 0x562e3135c700 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fc7e8006010 tx=0x7fc7e8003f10 comp rx=0 tx=0).stop 2024-04-23T14:56:46.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.210+0000 7fc802235740 1 -- 172.21.15.28:0/764201470 shutdown_connections 2024-04-23T14:56:46.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.210+0000 7fc802235740 1 --2- 172.21.15.28:0/764201470 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562e31373980 0x562e31375d70 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.210+0000 7fc802235740 1 --2- 172.21.15.28:0/764201470 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e31354b90 0x562e3135c700 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.210+0000 7fc802235740 1 --2- 172.21.15.28:0/764201470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3135f260 0x562e3135cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.210+0000 7fc7e6000640 0 client.27230 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:56:46.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.210+0000 7fc802235740 1 --2- 172.21.15.28:0/764201470 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562e31354270 0x562e3135c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.210+0000 7fc802235740 1 -- 172.21.15.28:0/764201470 >> 172.21.15.28:0/764201470 conn(0x562e31350590 msgr2=0x562e3134f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:46.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.210+0000 7fc802235740 1 -- 172.21.15.28:0/764201470 shutdown_connections 2024-04-23T14:56:46.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.210+0000 7fc802235740 1 -- 172.21.15.28:0/764201470 wait complete. 2024-04-23T14:56:46.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.210+0000 7fc802235740 20 client.27230 trim_cache size 0 max 16384 2024-04-23T14:56:46.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.231+0000 7fc802235740 1 Processor -- start 2024-04-23T14:56:46.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.231+0000 7fc802235740 1 -- start start 2024-04-23T14:56:46.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.231+0000 7fc802235740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e31354270 0x562e3136f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.232+0000 7fc802235740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562e31354b90 0x562e31356a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.232+0000 7fc802235740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e3135f260 0x562e3137b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.232+0000 7fc802235740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f81a99a0 con 0x562e31354270 2024-04-23T14:56:46.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.232+0000 7fc802235740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f81ac200 con 0x562e3135f260 2024-04-23T14:56:46.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.232+0000 7fc802235740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f81a9ff0 con 0x562e31354b90 2024-04-23T14:56:46.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.232+0000 7fc7fd400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e3135f260 0x562e3137b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.232+0000 7fc7fd400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e3135f260 0x562e3137b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48046/0 (socket says 172.21.15.28:48046) 2024-04-23T14:56:46.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.232+0000 7fc7f7e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562e31354b90 0x562e31356a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.232+0000 7fc7fca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e31354270 0x562e3136f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.232+0000 7fc7fd400640 1 -- 172.21.15.28:0/2969526453 learned_addr learned my addr 172.21.15.28:0/2969526453 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:46.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.232+0000 7fc7f7e00640 1 -- 172.21.15.28:0/2969526453 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e3135f260 msgr2=0x562e3137b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:46.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.232+0000 7fc7f7e00640 1 --2- 172.21.15.28:0/2969526453 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e3135f260 0x562e3137b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.232+0000 7fc7f7e00640 1 -- 172.21.15.28:0/2969526453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e31354270 msgr2=0x562e3136f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:46.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.232+0000 7fc7f7e00640 1 --2- 172.21.15.28:0/2969526453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e31354270 0x562e3136f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.232+0000 7fc7f7e00640 1 -- 172.21.15.28:0/2969526453 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e313550b0 con 0x562e31354b90 2024-04-23T14:56:46.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.233+0000 7fc7f7e00640 1 --2- 172.21.15.28:0/2969526453 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562e31354b90 0x562e31356a10 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc7e8010a70 tx=0x7fc7e8011cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec9f90300cec02ba server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:46.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.233+0000 7fc7ed200640 1 -- 172.21.15.28:0/2969526453 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e800fd00 con 0x562e31354b90 2024-04-23T14:56:46.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.233+0000 7fc7ed200640 1 -- 172.21.15.28:0/2969526453 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7e8029070 con 0x562e31354b90 2024-04-23T14:56:46.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.233+0000 7fc7ed200640 1 -- 172.21.15.28:0/2969526453 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e8005150 con 0x562e31354b90 2024-04-23T14:56:46.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.234+0000 7fc802235740 1 -- 172.21.15.28:0/2969526453 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562e31354b90 msgr2=0x562e31356a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:46.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.234+0000 7fc802235740 1 --2- 172.21.15.28:0/2969526453 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562e31354b90 0x562e31356a10 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc7e8010a70 tx=0x7fc7e8011cd0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.234+0000 7fc802235740 1 -- 172.21.15.28:0/2969526453 shutdown_connections 2024-04-23T14:56:46.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.234+0000 7fc802235740 1 --2- 172.21.15.28:0/2969526453 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e3135f260 0x562e3137b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.234+0000 7fc802235740 1 --2- 172.21.15.28:0/2969526453 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562e31354b90 0x562e31356a10 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.234+0000 7fc802235740 1 --2- 172.21.15.28:0/2969526453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e31354270 0x562e3136f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.234+0000 7fc802235740 1 -- 172.21.15.28:0/2969526453 >> 172.21.15.28:0/2969526453 conn(0x562e31350590 msgr2=0x562e3134f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:46.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.234+0000 7fc802235740 1 -- 172.21.15.28:0/2969526453 shutdown_connections 2024-04-23T14:56:46.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.234+0000 7fc802235740 1 -- 172.21.15.28:0/2969526453 wait complete. 2024-04-23T14:56:46.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.234+0000 7fc802235740 1 -- start start 2024-04-23T14:56:46.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.234+0000 7fc802235740 1 Processor -- start 2024-04-23T14:56:46.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.235+0000 7fc802235740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e31354270 0x562e31363480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.235+0000 7fc802235740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562e31354b90 0x562e313639c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.235+0000 7fc802235740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3135f260 0x562e31363f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.235+0000 7fc802235740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f81aa6c0 con 0x562e3135f260 2024-04-23T14:56:46.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.235+0000 7fc802235740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f81abb30 con 0x562e31354270 2024-04-23T14:56:46.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.235+0000 7fc802235740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f81aad90 con 0x562e31354b90 2024-04-23T14:56:46.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.235+0000 7fc802235740 10 client.0 _ll_register_callbacks cb 0x7fc7e0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:56:46.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.235+0000 7fc7fca00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e31354270 0x562e31363480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.235+0000 7fc7fca00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e31354270 0x562e31363480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48062/0 (socket says 172.21.15.28:48062) 2024-04-23T14:56:46.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.235+0000 7fc7fca00640 1 -- 172.21.15.28:0/4287455270 learned_addr learned my addr 172.21.15.28:0/4287455270 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:46.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.235+0000 7fc7fd400640 1 --2- 172.21.15.28:0/4287455270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3135f260 0x562e31363f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.235+0000 7fc7f7e00640 1 --2- 172.21.15.28:0/4287455270 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562e31354b90 0x562e313639c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.235+0000 7fc7fca00640 1 -- 172.21.15.28:0/4287455270 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562e31354b90 msgr2=0x562e313639c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:46.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.235+0000 7fc7fca00640 1 --2- 172.21.15.28:0/4287455270 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562e31354b90 0x562e313639c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.235+0000 7fc7fca00640 1 -- 172.21.15.28:0/4287455270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3135f260 msgr2=0x562e31363f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:46.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.236+0000 7fc7fca00640 1 --2- 172.21.15.28:0/4287455270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3135f260 0x562e31363f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.236+0000 7fc7fca00640 1 -- 172.21.15.28:0/4287455270 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e31367ed0 con 0x562e31354270 2024-04-23T14:56:46.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.236+0000 7fc7fd400640 1 --2- 172.21.15.28:0/4287455270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3135f260 0x562e31363f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:56:46.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.236+0000 7fc7fca00640 1 --2- 172.21.15.28:0/4287455270 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e31354270 0x562e31363480 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fc7e000c6a0 tx=0x7fc7e00bd040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fa7ea4a336e4b92 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:46.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.236+0000 7fc7e6000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:56:46.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.236+0000 7fc7e6000640 1 -- 172.21.15.28:0/4287455270 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e00d4020 con 0x562e31354270 2024-04-23T14:56:46.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.236+0000 7fc7e6000640 1 -- 172.21.15.28:0/4287455270 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8542+0+0 (secure 0 0 0) 0x7fc7e00991e0 con 0x562e31354270 2024-04-23T14:56:46.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.236+0000 7fc802235740 1 -- 172.21.15.28:0/4287455270 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7e8029070 con 0x562e31354270 2024-04-23T14:56:46.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.236+0000 7fc7e6000640 1 -- 172.21.15.28:0/4287455270 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fc7e00a57e0 con 0x562e31354270 2024-04-23T14:56:46.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.236+0000 7fc802235740 10 client.27236 fetch_fsmap learned FSMap version 20 2024-04-23T14:56:46.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.236+0000 7fc802235740 1 -- 172.21.15.28:0/4287455270 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x562e313deb00 con 0x562e31354270 2024-04-23T14:56:46.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.237+0000 7fc7e6000640 1 -- 172.21.15.28:0/4287455270 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7e8029070 con 0x562e31354270 2024-04-23T14:56:46.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.237+0000 7fc7e6000640 1 -- 172.21.15.28:0/4287455270 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fc7e00b7030 con 0x562e31354270 2024-04-23T14:56:46.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.237+0000 7fc802235740 10 client.27236 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:56:46.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.237+0000 7fc802235740 10 client.27236 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:56:46.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.237+0000 7fc802235740 20 client.27236 populate_metadata read hostname 'smithi028' 2024-04-23T14:56:46.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.237+0000 7fc802235740 1 --2- 172.21.15.28:0/4287455270 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562e313df180 0x562e313e15c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.237+0000 7fc802235740 4 client.27236 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-23T14:56:46.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.237+0000 7fc802235740 1 -- 172.21.15.28:0/4287455270 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x562e312c1700 con 0x562e313df180 2024-04-23T14:56:46.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.237+0000 7fc7f7e00640 1 --2- 172.21.15.28:0/4287455270 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562e313df180 0x562e313e15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.238+0000 7fc7e6000640 10 client.27236 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:56:46.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.238+0000 7fc7f7e00640 1 --2- 172.21.15.28:0/4287455270 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562e313df180 0x562e313e15c0 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e12ee72397d9b3e1 server_cookie=1229b2ee169dd714 in_seq=0 out_seq=0 2024-04-23T14:56:46.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.239+0000 7fc7e6000640 1 -- 172.21.15.28:0/4287455270 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x562e312c1700 con 0x562e313df180 2024-04-23T14:56:46.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.239+0000 7fc7e6000640 10 client.27236 handle_command_reply: tid=0 2024-04-23T14:56:46.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.239+0000 7fc802235740 1 client.27236 shutdown 2024-04-23T14:56:46.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.239+0000 7fc802235740 1 -- 172.21.15.28:0/4287455270 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e31354270 msgr2=0x562e31363480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:46.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.239+0000 7fc802235740 1 --2- 172.21.15.28:0/4287455270 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e31354270 0x562e31363480 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fc7e000c6a0 tx=0x7fc7e00bd040 comp rx=0 tx=0).stop 2024-04-23T14:56:46.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.239+0000 7fc802235740 1 -- 172.21.15.28:0/4287455270 shutdown_connections 2024-04-23T14:56:46.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.239+0000 7fc802235740 1 --2- 172.21.15.28:0/4287455270 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x562e313df180 0x562e313e15c0 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.239+0000 7fc802235740 1 --2- 172.21.15.28:0/4287455270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3135f260 0x562e31363f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.239+0000 7fc802235740 1 --2- 172.21.15.28:0/4287455270 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562e31354b90 0x562e313639c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.239+0000 7fc802235740 1 --2- 172.21.15.28:0/4287455270 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e31354270 0x562e31363480 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.239+0000 7fc7e6000640 0 client.27236 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:56:46.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.239+0000 7fc802235740 1 -- 172.21.15.28:0/4287455270 >> 172.21.15.28:0/4287455270 conn(0x562e31350590 msgr2=0x562e3135c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:46.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.239+0000 7fc802235740 1 -- 172.21.15.28:0/4287455270 shutdown_connections 2024-04-23T14:56:46.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.239+0000 7fc802235740 1 -- 172.21.15.28:0/4287455270 wait complete. 2024-04-23T14:56:46.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.240+0000 7fc802235740 20 client.27236 trim_cache size 0 max 16384 2024-04-23T14:56:46.244 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:56:46.244 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-23T14:56:46.244 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "715791da-4d97-489b-9f42-f65c5f4df880", 2024-04-23T14:56:46.244 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-23T14:56:46.244 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:56:46.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.240+0000 7fc7fde00640 1 -- 172.21.15.28:0/3157801002 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc7dc0a3d20 msgr2=0x7fc7dc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:46.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.240+0000 7fc7fde00640 1 --2- 172.21.15.28:0/3157801002 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc7dc0a3d20 0x7fc7dc0a61e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0005940 tx=0x7fc7f00058d0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.240+0000 7fc7fde00640 1 -- 172.21.15.28:0/3157801002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7f8103060 msgr2=0x7fc7f8198780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:46.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.240+0000 7fc7fde00640 1 --2- 172.21.15.28:0/3157801002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7f8103060 0x7fc7f8198780 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc7e000e9f0 tx=0x7fc7e000eeb0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.241+0000 7fc7fde00640 1 -- 172.21.15.28:0/3157801002 shutdown_connections 2024-04-23T14:56:46.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.241+0000 7fc7fde00640 1 --2- 172.21.15.28:0/3157801002 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc7dc0a3d20 0x7fc7dc0a61e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.241+0000 7fc7fde00640 1 --2- 172.21.15.28:0/3157801002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7f8103980 0x7fc7f8198cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.241+0000 7fc7fde00640 1 --2- 172.21.15.28:0/3157801002 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7f810c480 0x7fc7f819cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.241+0000 7fc7fde00640 1 --2- 172.21.15.28:0/3157801002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7f8103060 0x7fc7f8198780 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.241+0000 7fc7fde00640 1 -- 172.21.15.28:0/3157801002 >> 172.21.15.28:0/3157801002 conn(0x7fc7f80fc880 msgr2=0x7fc7f80fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:46.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.241+0000 7fc7fde00640 1 -- 172.21.15.28:0/3157801002 shutdown_connections 2024-04-23T14:56:46.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.241+0000 7fc7fde00640 1 -- 172.21.15.28:0/3157801002 wait complete. 2024-04-23T14:56:46.262 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:56:46.262 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:56:46.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.473+0000 7f02b4e00640 1 -- 172.21.15.28:0/4090471361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02b0102110 msgr2=0x7f02b01024f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:46.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.473+0000 7f02b4e00640 1 --2- 172.21.15.28:0/4090471361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02b0102110 0x7f02b01024f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f02a0009900 tx=0x7f02a002a3f0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.473+0000 7f02b4e00640 1 -- 172.21.15.28:0/4090471361 shutdown_connections 2024-04-23T14:56:46.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.473+0000 7f02b4e00640 1 --2- 172.21.15.28:0/4090471361 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02b01071e0 0x7f02b010edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.473+0000 7f02b4e00640 1 --2- 172.21.15.28:0/4090471361 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02b0102ac0 0x7f02b0106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.473+0000 7f02b4e00640 1 --2- 172.21.15.28:0/4090471361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02b0102110 0x7f02b01024f0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.473+0000 7f02b4e00640 1 -- 172.21.15.28:0/4090471361 >> 172.21.15.28:0/4090471361 conn(0x7f02b00fd9d0 msgr2=0x7f02b00ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:46.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.473+0000 7f02b4e00640 1 -- 172.21.15.28:0/4090471361 shutdown_connections 2024-04-23T14:56:46.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.473+0000 7f02b4e00640 1 -- 172.21.15.28:0/4090471361 wait complete. 2024-04-23T14:56:46.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.474+0000 7f02b4e00640 1 Processor -- start 2024-04-23T14:56:46.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.474+0000 7f02b4e00640 1 -- start start 2024-04-23T14:56:46.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.474+0000 7f02b4e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02b0102110 0x7f02b01a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.474+0000 7f02b4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02b01071e0 0x7f02b01a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.474+0000 7f02b4e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02b019a740 0x7f02b019abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.474+0000 7f02b4e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f02b01122e0 con 0x7f02b01071e0 2024-04-23T14:56:46.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.474+0000 7f02b4e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f02b0112160 con 0x7f02b019a740 2024-04-23T14:56:46.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.474+0000 7f02b4e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f02b0112460 con 0x7f02b0102110 2024-04-23T14:56:46.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.475+0000 7f02afe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02b019a740 0x7f02b019abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.475+0000 7f02af400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02b0102110 0x7f02b01a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.475+0000 7f02afe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02b019a740 0x7f02b019abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48092/0 (socket says 172.21.15.28:48092) 2024-04-23T14:56:46.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.475+0000 7f02afe00640 1 -- 172.21.15.28:0/1268512591 learned_addr learned my addr 172.21.15.28:0/1268512591 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:46.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.475+0000 7f02aea00640 1 --2- 172.21.15.28:0/1268512591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02b01071e0 0x7f02b01a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.475+0000 7f02afe00640 1 -- 172.21.15.28:0/1268512591 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02b0102110 msgr2=0x7f02b01a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:46.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.475+0000 7f02afe00640 1 --2- 172.21.15.28:0/1268512591 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02b0102110 0x7f02b01a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.475+0000 7f02afe00640 1 -- 172.21.15.28:0/1268512591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02b01071e0 msgr2=0x7f02b01a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:46.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.475+0000 7f02afe00640 1 --2- 172.21.15.28:0/1268512591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02b01071e0 0x7f02b01a0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.475+0000 7f02afe00640 1 -- 172.21.15.28:0/1268512591 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02b019b340 con 0x7f02b019a740 2024-04-23T14:56:46.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.476+0000 7f02afe00640 1 --2- 172.21.15.28:0/1268512591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02b019a740 0x7f02b019abc0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f02a800cce0 tx=0x7f02a800a620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:46.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.476+0000 7f02a7e00640 1 -- 172.21.15.28:0/1268512591 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a800f450 con 0x7f02b019a740 2024-04-23T14:56:46.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.476+0000 7f02a7e00640 1 -- 172.21.15.28:0/1268512591 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02a800fad0 con 0x7f02b019a740 2024-04-23T14:56:46.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.476+0000 7f02b4e00640 1 -- 172.21.15.28:0/1268512591 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02b01a7420 con 0x7f02b019a740 2024-04-23T14:56:46.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.476+0000 7f02a7e00640 1 -- 172.21.15.28:0/1268512591 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a8018c10 con 0x7f02b019a740 2024-04-23T14:56:46.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.476+0000 7f02b4e00640 1 -- 172.21.15.28:0/1268512591 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02b01a77d0 con 0x7f02b019a740 2024-04-23T14:56:46.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.481+0000 7f02b91fc740 1 Processor -- start 2024-04-23T14:56:46.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.481+0000 7f02a7e00640 1 -- 172.21.15.28:0/1268512591 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f02a800f5f0 con 0x7f02b019a740 2024-04-23T14:56:46.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.481+0000 7f02b91fc740 1 -- start start 2024-04-23T14:56:46.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.481+0000 7f02b91fc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d0bfaca30 0x558d0bface10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.481+0000 7f02b91fc740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d0bfad350 0x558d0bfb74e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.481+0000 7f02b91fc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d0bfb7a20 0x558d0bfb9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.481+0000 7f02b91fc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558d0bf121f0 con 0x558d0bfb7a20 2024-04-23T14:56:46.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.481+0000 7f02b91fc740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558d0beb8350 con 0x558d0bfad350 2024-04-23T14:56:46.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.481+0000 7f02b91fc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558d0bf0e1b0 con 0x558d0bfaca30 2024-04-23T14:56:46.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.482+0000 7f02af400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d0bfad350 0x558d0bfb74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.482+0000 7f02af400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d0bfad350 0x558d0bfb74e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48098/0 (socket says 172.21.15.28:48098) 2024-04-23T14:56:46.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.482+0000 7f02af400640 1 -- 172.21.15.28:0/164351624 learned_addr learned my addr 172.21.15.28:0/164351624 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:46.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.482+0000 7f02aea00640 1 --2- 172.21.15.28:0/164351624 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d0bfaca30 0x558d0bface10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.482+0000 7f02aea00640 1 --2- 172.21.15.28:0/164351624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d0bfb7a20 0x558d0bfb9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.482+0000 7f02af400640 1 -- 172.21.15.28:0/164351624 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d0bfaca30 msgr2=0x558d0bface10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:46.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.482+0000 7f02af400640 1 --2- 172.21.15.28:0/164351624 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d0bfaca30 0x558d0bface10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.482+0000 7f02af400640 1 -- 172.21.15.28:0/164351624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d0bfb7a20 msgr2=0x558d0bfb9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:46.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.482+0000 7f02af400640 1 --2- 172.21.15.28:0/164351624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d0bfb7a20 0x558d0bfb9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.482+0000 7f02af400640 1 -- 172.21.15.28:0/164351624 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d0bf0cf50 con 0x558d0bfad350 2024-04-23T14:56:46.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.482+0000 7f02aea00640 1 --2- 172.21.15.28:0/164351624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d0bfb7a20 0x558d0bfb9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:56:46.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.483+0000 7f02af400640 1 --2- 172.21.15.28:0/164351624 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d0bfad350 0x558d0bfb74e0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f02a0009230 tx=0x7f02a00040f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c86a2ffa45422a54 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:46.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.483+0000 7f02a7e00640 1 --2- 172.21.15.28:0/1268512591 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f02940a3d20 0x7f02940a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.483+0000 7f02a4c00640 1 -- 172.21.15.28:0/164351624 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a0035040 con 0x558d0bfad350 2024-04-23T14:56:46.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.483+0000 7f02a7e00640 1 -- 172.21.15.28:0/1268512591 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8542+0+0 (secure 0 0 0) 0x7f02a80d0e50 con 0x7f02b019a740 2024-04-23T14:56:46.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.483+0000 7f02a4c00640 1 -- 172.21.15.28:0/164351624 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02a0005520 con 0x558d0bfad350 2024-04-23T14:56:46.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.483+0000 7f02a4c00640 1 -- 172.21.15.28:0/164351624 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a0003850 con 0x558d0bfad350 2024-04-23T14:56:46.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.483+0000 7f02aea00640 1 --2- 172.21.15.28:0/1268512591 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f02940a3d20 0x7f02940a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.484+0000 7f02b91fc740 1 -- 172.21.15.28:0/164351624 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d0bfad350 msgr2=0x558d0bfb74e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:46.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.484+0000 7f02aea00640 1 --2- 172.21.15.28:0/1268512591 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f02940a3d20 0x7f02940a61e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x558d0bfa9ee0 tx=0x7f029c013040 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:46.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.484+0000 7f02b91fc740 1 --2- 172.21.15.28:0/164351624 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d0bfad350 0x558d0bfb74e0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f02a0009230 tx=0x7f02a00040f0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.484+0000 7f02b91fc740 1 -- 172.21.15.28:0/164351624 shutdown_connections 2024-04-23T14:56:46.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.484+0000 7f02b91fc740 1 --2- 172.21.15.28:0/164351624 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d0bfad350 0x558d0bfb74e0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.484+0000 7f02b91fc740 1 --2- 172.21.15.28:0/164351624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d0bfb7a20 0x558d0bfb9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.484+0000 7f02b91fc740 1 --2- 172.21.15.28:0/164351624 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d0bfaca30 0x558d0bface10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.484+0000 7f02b91fc740 1 -- 172.21.15.28:0/164351624 >> 172.21.15.28:0/164351624 conn(0x558d0bfa8d50 msgr2=0x558d0bfa9130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:46.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.484+0000 7f02b91fc740 1 -- 172.21.15.28:0/164351624 shutdown_connections 2024-04-23T14:56:46.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.484+0000 7f02b91fc740 1 -- 172.21.15.28:0/164351624 wait complete. 2024-04-23T14:56:46.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.485+0000 7f02b91fc740 1 -- start start 2024-04-23T14:56:46.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.485+0000 7f02b91fc740 1 Processor -- start 2024-04-23T14:56:46.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.486+0000 7f02b91fc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d0bfaca30 0x558d0bfb49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.486+0000 7f02b91fc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d0bfad350 0x558d0bfb4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.486+0000 7f02b91fc740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d0bfb7a20 0x558d0bfb5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.486+0000 7f02b91fc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558d0bfba7b0 con 0x558d0bfaca30 2024-04-23T14:56:46.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.486+0000 7f02b91fc740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558d0becb690 con 0x558d0bfb7a20 2024-04-23T14:56:46.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.486+0000 7f02b91fc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558d0bfba930 con 0x558d0bfad350 2024-04-23T14:56:46.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.486+0000 7f02b91fc740 10 client.0 _ll_register_callbacks cb 0x558d0bf07650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:56:46.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.486+0000 7f02af400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d0bfaca30 0x558d0bfb49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.486+0000 7f02afe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d0bfb7a20 0x558d0bfb5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.486+0000 7f02aea00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d0bfad350 0x558d0bfb4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.486+0000 7f02af400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d0bfaca30 0x558d0bfb49e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49334/0 (socket says 172.21.15.28:49334) 2024-04-23T14:56:46.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.486+0000 7f02af400640 1 -- 172.21.15.28:0/3191332811 learned_addr learned my addr 172.21.15.28:0/3191332811 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:46.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.486+0000 7f02afe00640 1 -- 172.21.15.28:0/3191332811 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d0bfad350 msgr2=0x558d0bfb4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:46.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.486+0000 7f02afe00640 1 --2- 172.21.15.28:0/3191332811 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d0bfad350 0x558d0bfb4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.486+0000 7f02afe00640 1 -- 172.21.15.28:0/3191332811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d0bfaca30 msgr2=0x558d0bfb49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:46.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.486+0000 7f02afe00640 1 --2- 172.21.15.28:0/3191332811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d0bfaca30 0x558d0bfb49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.487+0000 7f02afe00640 1 -- 172.21.15.28:0/3191332811 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d0beca920 con 0x558d0bfb7a20 2024-04-23T14:56:46.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.487+0000 7f02afe00640 1 --2- 172.21.15.28:0/3191332811 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d0bfb7a20 0x558d0bfb5460 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f02a80ce500 tx=0x7f02a809c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec3215a1e78feb3e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:46.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.487+0000 7f0299600640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:56:46.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.487+0000 7f0299600640 1 -- 172.21.15.28:0/3191332811 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a80b6060 con 0x558d0bfb7a20 2024-04-23T14:56:46.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.487+0000 7f0299600640 1 -- 172.21.15.28:0/3191332811 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8542+0+0 (secure 0 0 0) 0x7f02a80993f0 con 0x558d0bfb7a20 2024-04-23T14:56:46.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.487+0000 7f02b91fc740 1 -- 172.21.15.28:0/3191332811 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02a0005520 con 0x558d0bfb7a20 2024-04-23T14:56:46.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.487+0000 7f0299600640 1 -- 172.21.15.28:0/3191332811 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f02a80a1130 con 0x558d0bfb7a20 2024-04-23T14:56:46.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.487+0000 7f02b91fc740 10 client.27245 fetch_fsmap learned FSMap version 20 2024-04-23T14:56:46.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.487+0000 7f02b91fc740 1 -- 172.21.15.28:0/3191332811 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x558d0bcee7d0 con 0x558d0bfb7a20 2024-04-23T14:56:46.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.487+0000 7f0299600640 1 -- 172.21.15.28:0/3191332811 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02a0005520 con 0x558d0bfb7a20 2024-04-23T14:56:46.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.488+0000 7f0299600640 1 -- 172.21.15.28:0/3191332811 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f02a807bba0 con 0x558d0bfb7a20 2024-04-23T14:56:46.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.488+0000 7f02b91fc740 10 client.27245 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:56:46.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.488+0000 7f02b91fc740 10 client.27245 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:56:46.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.488+0000 7f02b91fc740 20 client.27245 populate_metadata read hostname 'smithi028' 2024-04-23T14:56:46.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.488+0000 7f02b91fc740 1 --2- 172.21.15.28:0/3191332811 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558d0bfc7dd0 0x558d0bfca1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.488+0000 7f02b91fc740 4 client.27245 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:56:46.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.488+0000 7f02b91fc740 1 -- 172.21.15.28:0/3191332811 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558d0bf1fc10 con 0x558d0bfc7dd0 2024-04-23T14:56:46.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.488+0000 7f02af400640 1 --2- 172.21.15.28:0/3191332811 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558d0bfc7dd0 0x558d0bfca1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.489+0000 7f02af400640 1 --2- 172.21.15.28:0/3191332811 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558d0bfc7dd0 0x558d0bfca1c0 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2fb09b3f287ed16d server_cookie=fc7c08147403bc6 in_seq=0 out_seq=0 2024-04-23T14:56:46.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.489+0000 7f0299600640 10 client.27245 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:56:46.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.489+0000 7f0299600640 1 -- 172.21.15.28:0/3191332811 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558d0bf1fc10 con 0x558d0bfc7dd0 2024-04-23T14:56:46.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.489+0000 7f0299600640 10 client.27245 handle_command_reply: tid=0 2024-04-23T14:56:46.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.490+0000 7f02b91fc740 1 client.27245 shutdown 2024-04-23T14:56:46.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.490+0000 7f02b91fc740 1 -- 172.21.15.28:0/3191332811 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d0bfb7a20 msgr2=0x558d0bfb5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:46.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.490+0000 7f02b91fc740 1 --2- 172.21.15.28:0/3191332811 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d0bfb7a20 0x558d0bfb5460 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f02a80ce500 tx=0x7f02a809c940 comp rx=0 tx=0).stop 2024-04-23T14:56:46.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.490+0000 7f02b91fc740 1 -- 172.21.15.28:0/3191332811 shutdown_connections 2024-04-23T14:56:46.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.490+0000 7f02b91fc740 1 --2- 172.21.15.28:0/3191332811 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558d0bfc7dd0 0x558d0bfca1c0 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.490+0000 7f02b91fc740 1 --2- 172.21.15.28:0/3191332811 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d0bfb7a20 0x558d0bfb5460 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.490+0000 7f02b91fc740 1 --2- 172.21.15.28:0/3191332811 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d0bfad350 0x558d0bfb4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.490+0000 7f02b91fc740 1 --2- 172.21.15.28:0/3191332811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d0bfaca30 0x558d0bfb49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.490+0000 7f02b91fc740 1 -- 172.21.15.28:0/3191332811 >> 172.21.15.28:0/3191332811 conn(0x558d0bfa8d50 msgr2=0x558d0bfa7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:46.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.490+0000 7f0299600640 0 client.27245 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:56:46.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.490+0000 7f02b91fc740 1 -- 172.21.15.28:0/3191332811 shutdown_connections 2024-04-23T14:56:46.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.490+0000 7f02b91fc740 1 -- 172.21.15.28:0/3191332811 wait complete. 2024-04-23T14:56:46.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.490+0000 7f02b91fc740 20 client.27245 trim_cache size 0 max 16384 2024-04-23T14:56:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.513+0000 7f02b91fc740 1 Processor -- start 2024-04-23T14:56:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.513+0000 7f02b91fc740 1 -- start start 2024-04-23T14:56:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.513+0000 7f02b91fc740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d0bfaca30 0x558d0bfc36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.513+0000 7f02b91fc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d0bfad350 0x558d0bfa9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.513+0000 7f02b91fc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d0bfb7a20 0x558d0bfaf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.513+0000 7f02b91fc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f02b01adc50 con 0x558d0bfb7a20 2024-04-23T14:56:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.513+0000 7f02b91fc740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f02b01b04b0 con 0x558d0bfaca30 2024-04-23T14:56:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.513+0000 7f02b91fc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f02b01ae2a0 con 0x558d0bfad350 2024-04-23T14:56:46.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.513+0000 7f02af400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d0bfaca30 0x558d0bfc36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.514+0000 7f02af400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d0bfaca30 0x558d0bfc36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48106/0 (socket says 172.21.15.28:48106) 2024-04-23T14:56:46.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.514+0000 7f02afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d0bfb7a20 0x558d0bfaf1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.514+0000 7f02af400640 1 -- 172.21.15.28:0/1332342057 learned_addr learned my addr 172.21.15.28:0/1332342057 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:46.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.514+0000 7f02aea00640 1 --2- 172.21.15.28:0/1332342057 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d0bfad350 0x558d0bfa9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.514+0000 7f02af400640 1 -- 172.21.15.28:0/1332342057 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d0bfad350 msgr2=0x558d0bfa9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:46.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.514+0000 7f02af400640 1 --2- 172.21.15.28:0/1332342057 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d0bfad350 0x558d0bfa9130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.514+0000 7f02af400640 1 -- 172.21.15.28:0/1332342057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d0bfb7a20 msgr2=0x558d0bfaf1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:46.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.514+0000 7f02af400640 1 --2- 172.21.15.28:0/1332342057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d0bfb7a20 0x558d0bfaf1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.514+0000 7f02af400640 1 -- 172.21.15.28:0/1332342057 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d0bc243f0 con 0x558d0bfaca30 2024-04-23T14:56:46.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.514+0000 7f02af400640 1 --2- 172.21.15.28:0/1332342057 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d0bfaca30 0x558d0bfc36d0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f02a002a9e0 tx=0x7f02a0005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc1efe713d2c6f77 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:46.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.514+0000 7f029be00640 1 -- 172.21.15.28:0/1332342057 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a0035040 con 0x558d0bfaca30 2024-04-23T14:56:46.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.514+0000 7f029be00640 1 -- 172.21.15.28:0/1332342057 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02a0005400 con 0x558d0bfaca30 2024-04-23T14:56:46.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.515+0000 7f029be00640 1 -- 172.21.15.28:0/1332342057 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a0054630 con 0x558d0bfaca30 2024-04-23T14:56:46.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.515+0000 7f02b91fc740 1 -- 172.21.15.28:0/1332342057 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d0bfaca30 msgr2=0x558d0bfc36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:46.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.515+0000 7f02b91fc740 1 --2- 172.21.15.28:0/1332342057 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d0bfaca30 0x558d0bfc36d0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f02a002a9e0 tx=0x7f02a0005fd0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.515+0000 7f02b91fc740 1 -- 172.21.15.28:0/1332342057 shutdown_connections 2024-04-23T14:56:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.515+0000 7f02b91fc740 1 --2- 172.21.15.28:0/1332342057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d0bfb7a20 0x558d0bfaf1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.515+0000 7f02b91fc740 1 --2- 172.21.15.28:0/1332342057 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d0bfad350 0x558d0bfa9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.515+0000 7f02b91fc740 1 --2- 172.21.15.28:0/1332342057 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d0bfaca30 0x558d0bfc36d0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.515+0000 7f02b91fc740 1 -- 172.21.15.28:0/1332342057 >> 172.21.15.28:0/1332342057 conn(0x558d0bfa8d50 msgr2=0x558d0bfb49a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.516+0000 7f02b91fc740 1 -- 172.21.15.28:0/1332342057 shutdown_connections 2024-04-23T14:56:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.516+0000 7f02b91fc740 1 -- 172.21.15.28:0/1332342057 wait complete. 2024-04-23T14:56:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.516+0000 7f02b91fc740 1 -- start start 2024-04-23T14:56:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.516+0000 7f02b91fc740 1 Processor -- start 2024-04-23T14:56:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.517+0000 7f02b91fc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d0bfaca30 0x558d0c026550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.517+0000 7f02b91fc740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d0bfad350 0x558d0c026a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.517+0000 7f02b91fc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d0bfb7a20 0x558d0bfbafd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.517+0000 7f02b91fc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f02b01ae970 con 0x558d0bfb7a20 2024-04-23T14:56:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.517+0000 7f02b91fc740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f02b01afde0 con 0x558d0bfad350 2024-04-23T14:56:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.517+0000 7f02b91fc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f02b01af040 con 0x558d0bfaca30 2024-04-23T14:56:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.517+0000 7f02b91fc740 10 client.0 _ll_register_callbacks cb 0x7f02a80077a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:56:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.517+0000 7f02aea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d0bfad350 0x558d0c026a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.517+0000 7f02aea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d0bfad350 0x558d0c026a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48122/0 (socket says 172.21.15.28:48122) 2024-04-23T14:56:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.517+0000 7f02afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d0bfb7a20 0x558d0bfbafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.517+0000 7f02aea00640 1 -- 172.21.15.28:0/2022542584 learned_addr learned my addr 172.21.15.28:0/2022542584 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:56:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.517+0000 7f02af400640 1 --2- 172.21.15.28:0/2022542584 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d0bfaca30 0x558d0c026550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.518+0000 7f02aea00640 1 -- 172.21.15.28:0/2022542584 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d0bfaca30 msgr2=0x558d0c026550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.518+0000 7f02aea00640 1 --2- 172.21.15.28:0/2022542584 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d0bfaca30 0x558d0c026550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.518+0000 7f02aea00640 1 -- 172.21.15.28:0/2022542584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d0bfb7a20 msgr2=0x558d0bfbafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:56:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.518+0000 7f02aea00640 1 --2- 172.21.15.28:0/2022542584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d0bfb7a20 0x558d0bfbafd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.518+0000 7f02aea00640 1 -- 172.21.15.28:0/2022542584 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d0bbe40f0 con 0x558d0bfad350 2024-04-23T14:56:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.518+0000 7f02af400640 1 --2- 172.21.15.28:0/2022542584 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d0bfaca30 0x558d0c026550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:56:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.518+0000 7f02afe00640 1 --2- 172.21.15.28:0/2022542584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d0bfb7a20 0x558d0bfbafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:56:46.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.518+0000 7f02aea00640 1 --2- 172.21.15.28:0/2022542584 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d0bfad350 0x558d0c026a90 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f02b019bf90 tx=0x7f029c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd702d20c988101c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:56:46.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.518+0000 7f0299600640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:56:46.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.518+0000 7f0299600640 1 -- 172.21.15.28:0/2022542584 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f029c003e50 con 0x558d0bfad350 2024-04-23T14:56:46.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.518+0000 7f02b91fc740 1 -- 172.21.15.28:0/2022542584 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02a0005400 con 0x558d0bfad350 2024-04-23T14:56:46.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.518+0000 7f0299600640 1 -- 172.21.15.28:0/2022542584 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8542+0+0 (secure 0 0 0) 0x7f029c01a070 con 0x558d0bfad350 2024-04-23T14:56:46.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.519+0000 7f0299600640 1 -- 172.21.15.28:0/2022542584 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f029c0219a0 con 0x558d0bfad350 2024-04-23T14:56:46.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.519+0000 7f02b91fc740 10 client.27251 fetch_fsmap learned FSMap version 20 2024-04-23T14:56:46.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.519+0000 7f02b91fc740 1 -- 172.21.15.28:0/2022542584 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x558d0bed7810 con 0x558d0bfad350 2024-04-23T14:56:46.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.519+0000 7f0299600640 1 -- 172.21.15.28:0/2022542584 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02a0005400 con 0x558d0bfad350 2024-04-23T14:56:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.519+0000 7f0299600640 1 -- 172.21.15.28:0/2022542584 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f029c00f480 con 0x558d0bfad350 2024-04-23T14:56:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.519+0000 7f02b91fc740 10 client.27251 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:56:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.519+0000 7f02b91fc740 10 client.27251 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:56:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.519+0000 7f02b91fc740 20 client.27251 populate_metadata read hostname 'smithi028' 2024-04-23T14:56:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.519+0000 7f02b91fc740 1 --2- 172.21.15.28:0/2022542584 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558d0c02e2c0 0x558d0c0306e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:56:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.519+0000 7f02b91fc740 4 client.27251 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:56:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.519+0000 7f02b91fc740 1 -- 172.21.15.28:0/2022542584 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558d0bf19a80 con 0x558d0c02e2c0 2024-04-23T14:56:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.520+0000 7f02af400640 1 --2- 172.21.15.28:0/2022542584 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558d0c02e2c0 0x558d0c0306e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:56:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.520+0000 7f02af400640 1 --2- 172.21.15.28:0/2022542584 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558d0c02e2c0 0x558d0c0306e0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6054caa93c7168 server_cookie=fe83fb701bd5173a in_seq=0 out_seq=0 2024-04-23T14:56:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.520+0000 7f0299600640 10 client.27251 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:56:46.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.524+0000 7f0299600640 1 -- 172.21.15.28:0/2022542584 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x558d0bf19a80 con 0x558d0c02e2c0 2024-04-23T14:56:46.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.524+0000 7f0299600640 10 client.27251 handle_command_reply: tid=0 2024-04-23T14:56:46.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.524+0000 7f02b91fc740 1 client.27251 shutdown 2024-04-23T14:56:46.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.524+0000 7f02b91fc740 1 -- 172.21.15.28:0/2022542584 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d0bfad350 msgr2=0x558d0c026a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:46.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.524+0000 7f02b91fc740 1 --2- 172.21.15.28:0/2022542584 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d0bfad350 0x558d0c026a90 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f02b019bf90 tx=0x7f029c004730 comp rx=0 tx=0).stop 2024-04-23T14:56:46.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.524+0000 7f02b91fc740 1 -- 172.21.15.28:0/2022542584 shutdown_connections 2024-04-23T14:56:46.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.524+0000 7f02b91fc740 1 --2- 172.21.15.28:0/2022542584 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x558d0c02e2c0 0x558d0c0306e0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.524+0000 7f02b91fc740 1 --2- 172.21.15.28:0/2022542584 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558d0bfad350 0x558d0c026a90 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.524+0000 7f02b91fc740 1 --2- 172.21.15.28:0/2022542584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d0bfb7a20 0x558d0bfbafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.524+0000 7f02b91fc740 1 --2- 172.21.15.28:0/2022542584 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558d0bfaca30 0x558d0c026550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.524+0000 7f0299600640 0 client.27251 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:56:46.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.524+0000 7f02b91fc740 1 -- 172.21.15.28:0/2022542584 >> 172.21.15.28:0/2022542584 conn(0x558d0bfa8d50 msgr2=0x558d0bfb5230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:46.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.524+0000 7f02b91fc740 1 -- 172.21.15.28:0/2022542584 shutdown_connections 2024-04-23T14:56:46.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.524+0000 7f02b91fc740 1 -- 172.21.15.28:0/2022542584 wait complete. 2024-04-23T14:56:46.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.524+0000 7f02b91fc740 20 client.27251 trim_cache size 0 max 16384 2024-04-23T14:56:46.526 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:56:46.526 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-23T14:56:46.526 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-23T14:56:46.526 INFO:teuthology.orchestra.run.smithi028.stdout: "715791da-4d97-489b-9f42-f65c5f4df880": { 2024-04-23T14:56:46.526 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-23T14:56:46.527 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "715791da-4d97-489b-9f42-f65c5f4df880", 2024-04-23T14:56:46.527 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-23T14:56:46.527 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:56:46.527 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:56:46.527 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:56:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.525+0000 7f02b4e00640 1 -- 172.21.15.28:0/1268512591 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f02940a3d20 msgr2=0x7f02940a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.525+0000 7f02b4e00640 1 --2- 172.21.15.28:0/1268512591 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f02940a3d20 0x7f02940a61e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x558d0bfa9ee0 tx=0x7f029c013040 comp rx=0 tx=0).stop 2024-04-23T14:56:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.525+0000 7f02b4e00640 1 -- 172.21.15.28:0/1268512591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02b019a740 msgr2=0x7f02b019abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:56:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.525+0000 7f02b4e00640 1 --2- 172.21.15.28:0/1268512591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02b019a740 0x7f02b019abc0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f02a800cce0 tx=0x7f02a800a620 comp rx=0 tx=0).stop 2024-04-23T14:56:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.525+0000 7f02b4e00640 1 -- 172.21.15.28:0/1268512591 shutdown_connections 2024-04-23T14:56:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.525+0000 7f02b4e00640 1 --2- 172.21.15.28:0/1268512591 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f02940a3d20 0x7f02940a61e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.525+0000 7f02b4e00640 1 --2- 172.21.15.28:0/1268512591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02b019a740 0x7f02b019abc0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.525+0000 7f02b4e00640 1 --2- 172.21.15.28:0/1268512591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02b01071e0 0x7f02b01a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.525+0000 7f02b4e00640 1 --2- 172.21.15.28:0/1268512591 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02b0102110 0x7f02b01a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:56:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.525+0000 7f02b4e00640 1 -- 172.21.15.28:0/1268512591 >> 172.21.15.28:0/1268512591 conn(0x7f02b00fd9d0 msgr2=0x7f02b00fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:56:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.525+0000 7f02b4e00640 1 -- 172.21.15.28:0/1268512591 shutdown_connections 2024-04-23T14:56:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:56:46.526+0000 7f02b4e00640 1 -- 172.21.15.28:0/1268512591 wait complete. 2024-04-23T14:56:46.545 INFO:tasks.cephfs.filesystem:scrub status for tag:715791da-4d97-489b-9f42-f65c5f4df880 - {'path': '/', 'tag': '715791da-4d97-489b-9f42-f65c5f4df880', 'options': 'recursive,force'} 2024-04-23T14:56:46.545 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1082842 8.50 MB/sec execute 542 sec latency 58.423 ms 2024-04-23T14:56:47.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:46 smithi038 ceph-mon[32348]: pgmap v397: 97 pgs: 97 active+clean; 607 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 121 op/s 2024-04-23T14:56:47.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:46 smithi028 ceph-mon[27810]: pgmap v397: 97 pgs: 97 active+clean; 607 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 121 op/s 2024-04-23T14:56:47.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:46 smithi105 ceph-mon[32555]: pgmap v397: 97 pgs: 97 active+clean; 607 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 121 op/s 2024-04-23T14:56:47.545 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1083865 8.50 MB/sec execute 543 sec latency 62.727 ms 2024-04-23T14:56:48.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:47 smithi038 ceph-mon[32348]: scrub queued for path: / 2024-04-23T14:56:48.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:47 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:56:48.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:47 smithi038 ceph-mon[32348]: scrub summary: active paths [/] 2024-04-23T14:56:48.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:47 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:56:48.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:47 smithi028 ceph-mon[27810]: scrub queued for path: / 2024-04-23T14:56:48.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:47 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:56:48.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:47 smithi028 ceph-mon[27810]: scrub summary: active paths [/] 2024-04-23T14:56:48.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:47 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:56:48.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:47 smithi105 ceph-mon[32555]: scrub queued for path: / 2024-04-23T14:56:48.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:47 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:56:48.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:47 smithi105 ceph-mon[32555]: scrub summary: active paths [/] 2024-04-23T14:56:48.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:47 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:56:48.545 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1085079 8.49 MB/sec execute 544 sec latency 32.322 ms 2024-04-23T14:56:49.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:48 smithi038 ceph-mon[32348]: pgmap v398: 97 pgs: 97 active+clean; 553 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.2 MiB/s wr, 187 op/s 2024-04-23T14:56:49.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:48 smithi028 ceph-mon[27810]: pgmap v398: 97 pgs: 97 active+clean; 553 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.2 MiB/s wr, 187 op/s 2024-04-23T14:56:49.285 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:48 smithi105 ceph-mon[32555]: pgmap v398: 97 pgs: 97 active+clean; 553 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.2 MiB/s wr, 187 op/s 2024-04-23T14:56:49.545 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1086794 8.49 MB/sec execute 545 sec latency 19.425 ms 2024-04-23T14:56:50.545 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1087629 8.48 MB/sec execute 546 sec latency 22.036 ms 2024-04-23T14:56:51.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:50 smithi038 ceph-mon[32348]: pgmap v399: 97 pgs: 97 active+clean; 553 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 128 op/s 2024-04-23T14:56:51.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:50 smithi038 ceph-mon[32348]: scrub summary: idle 2024-04-23T14:56:51.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:50 smithi028 ceph-mon[27810]: pgmap v399: 97 pgs: 97 active+clean; 553 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 128 op/s 2024-04-23T14:56:51.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:50 smithi028 ceph-mon[27810]: scrub summary: idle 2024-04-23T14:56:51.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:50 smithi105 ceph-mon[32555]: pgmap v399: 97 pgs: 97 active+clean; 553 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 128 op/s 2024-04-23T14:56:51.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:50 smithi105 ceph-mon[32555]: scrub summary: idle 2024-04-23T14:56:51.545 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1088656 8.48 MB/sec execute 547 sec latency 66.216 ms 2024-04-23T14:56:52.546 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1090505 8.49 MB/sec execute 548 sec latency 33.948 ms 2024-04-23T14:56:53.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:52 smithi038 ceph-mon[32348]: pgmap v400: 97 pgs: 97 active+clean; 553 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 128 op/s 2024-04-23T14:56:53.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:52 smithi028 ceph-mon[27810]: pgmap v400: 97 pgs: 97 active+clean; 553 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 128 op/s 2024-04-23T14:56:53.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:52 smithi105 ceph-mon[32555]: pgmap v400: 97 pgs: 97 active+clean; 553 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 128 op/s 2024-04-23T14:56:53.546 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1091424 8.47 MB/sec execute 549 sec latency 18.442 ms 2024-04-23T14:56:54.546 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1093287 8.48 MB/sec execute 550 sec latency 51.356 ms 2024-04-23T14:56:55.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:54 smithi038 ceph-mon[32348]: pgmap v401: 97 pgs: 97 active+clean; 561 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.2 MiB/s wr, 183 op/s 2024-04-23T14:56:55.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:54 smithi028 ceph-mon[27810]: pgmap v401: 97 pgs: 97 active+clean; 561 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.2 MiB/s wr, 183 op/s 2024-04-23T14:56:55.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:54 smithi105 ceph-mon[32555]: pgmap v401: 97 pgs: 97 active+clean; 561 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.2 MiB/s wr, 183 op/s 2024-04-23T14:56:55.546 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1094645 8.48 MB/sec execute 551 sec latency 18.623 ms 2024-04-23T14:56:56.546 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1096198 8.47 MB/sec execute 552 sec latency 36.582 ms 2024-04-23T14:56:57.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:56 smithi038 ceph-mon[32348]: pgmap v402: 97 pgs: 97 active+clean; 561 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 120 op/s 2024-04-23T14:56:57.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:56 smithi028 ceph-mon[27810]: pgmap v402: 97 pgs: 97 active+clean; 561 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 120 op/s 2024-04-23T14:56:57.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:56 smithi105 ceph-mon[32555]: pgmap v402: 97 pgs: 97 active+clean; 561 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 120 op/s 2024-04-23T14:56:57.546 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1097847 8.48 MB/sec execute 553 sec latency 91.094 ms 2024-04-23T14:56:58.546 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1098755 8.47 MB/sec execute 554 sec latency 26.237 ms 2024-04-23T14:56:59.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:56:58 smithi038 ceph-mon[32348]: pgmap v403: 97 pgs: 97 active+clean; 571 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.1 MiB/s wr, 183 op/s 2024-04-23T14:56:59.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:56:58 smithi028 ceph-mon[27810]: pgmap v403: 97 pgs: 97 active+clean; 571 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.1 MiB/s wr, 183 op/s 2024-04-23T14:56:59.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:56:58 smithi105 ceph-mon[32555]: pgmap v403: 97 pgs: 97 active+clean; 571 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.1 MiB/s wr, 183 op/s 2024-04-23T14:56:59.546 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1100154 8.46 MB/sec execute 555 sec latency 27.423 ms 2024-04-23T14:57:00.546 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1101322 8.47 MB/sec execute 556 sec latency 59.364 ms 2024-04-23T14:57:01.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:00 smithi105 ceph-mon[32555]: pgmap v404: 97 pgs: 97 active+clean; 571 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 117 op/s 2024-04-23T14:57:01.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:00 smithi038 ceph-mon[32348]: pgmap v404: 97 pgs: 97 active+clean; 571 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 117 op/s 2024-04-23T14:57:01.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:00 smithi028 ceph-mon[27810]: pgmap v404: 97 pgs: 97 active+clean; 571 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 117 op/s 2024-04-23T14:57:01.547 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1101510 8.45 MB/sec execute 557 sec latency 111.230 ms 2024-04-23T14:57:02.547 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1102637 8.45 MB/sec execute 558 sec latency 67.473 ms 2024-04-23T14:57:03.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:02 smithi105 ceph-mon[32555]: pgmap v405: 97 pgs: 97 active+clean; 571 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 117 op/s 2024-04-23T14:57:03.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:02 smithi038 ceph-mon[32348]: pgmap v405: 97 pgs: 97 active+clean; 571 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 117 op/s 2024-04-23T14:57:03.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:02 smithi028 ceph-mon[27810]: pgmap v405: 97 pgs: 97 active+clean; 571 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 117 op/s 2024-04-23T14:57:03.547 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1104672 8.46 MB/sec execute 559 sec latency 19.425 ms 2024-04-23T14:57:04.547 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1105762 8.45 MB/sec execute 560 sec latency 19.738 ms 2024-04-23T14:57:05.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:04 smithi105 ceph-mon[32555]: pgmap v406: 97 pgs: 97 active+clean; 579 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.0 MiB/s wr, 206 op/s 2024-04-23T14:57:05.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:04 smithi038 ceph-mon[32348]: pgmap v406: 97 pgs: 97 active+clean; 579 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.0 MiB/s wr, 206 op/s 2024-04-23T14:57:05.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:04 smithi028 ceph-mon[27810]: pgmap v406: 97 pgs: 97 active+clean; 579 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.0 MiB/s wr, 206 op/s 2024-04-23T14:57:05.547 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1107265 8.44 MB/sec execute 561 sec latency 20.535 ms 2024-04-23T14:57:06.547 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1108555 8.45 MB/sec execute 562 sec latency 80.578 ms 2024-04-23T14:57:07.173 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:06 smithi038 ceph-mon[32348]: pgmap v407: 97 pgs: 97 active+clean; 579 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 151 op/s 2024-04-23T14:57:07.173 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:06 smithi038 ceph-mon[32348]: osdmap e101: 12 total, 12 up, 12 in 2024-04-23T14:57:07.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:06 smithi105 ceph-mon[32555]: pgmap v407: 97 pgs: 97 active+clean; 579 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 151 op/s 2024-04-23T14:57:07.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:06 smithi105 ceph-mon[32555]: osdmap e101: 12 total, 12 up, 12 in 2024-04-23T14:57:07.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:06 smithi028 ceph-mon[27810]: pgmap v407: 97 pgs: 97 active+clean; 579 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 151 op/s 2024-04-23T14:57:07.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:06 smithi028 ceph-mon[27810]: osdmap e101: 12 total, 12 up, 12 in 2024-04-23T14:57:07.547 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1109688 8.44 MB/sec execute 563 sec latency 50.976 ms 2024-04-23T14:57:08.547 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1111215 8.44 MB/sec execute 564 sec latency 19.866 ms 2024-04-23T14:57:09.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:08 smithi105 ceph-mon[32555]: pgmap v409: 97 pgs: 97 active+clean; 588 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.4 MiB/s wr, 180 op/s 2024-04-23T14:57:09.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:08 smithi038 ceph-mon[32348]: pgmap v409: 97 pgs: 97 active+clean; 588 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.4 MiB/s wr, 180 op/s 2024-04-23T14:57:09.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:08 smithi028 ceph-mon[27810]: pgmap v409: 97 pgs: 97 active+clean; 588 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.4 MiB/s wr, 180 op/s 2024-04-23T14:57:09.548 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1112251 8.44 MB/sec execute 565 sec latency 20.454 ms 2024-04-23T14:57:10.548 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1113875 8.43 MB/sec execute 566 sec latency 19.229 ms 2024-04-23T14:57:11.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:10 smithi105 ceph-mon[32555]: pgmap v410: 97 pgs: 97 active+clean; 588 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.4 MiB/s wr, 180 op/s 2024-04-23T14:57:11.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:10 smithi028 ceph-mon[27810]: pgmap v410: 97 pgs: 97 active+clean; 588 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.4 MiB/s wr, 180 op/s 2024-04-23T14:57:11.375 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:10 smithi038 ceph-mon[32348]: pgmap v410: 97 pgs: 97 active+clean; 588 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.4 MiB/s wr, 180 op/s 2024-04-23T14:57:11.548 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1115459 8.44 MB/sec execute 567 sec latency 61.612 ms 2024-04-23T14:57:12.548 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1116514 8.43 MB/sec execute 568 sec latency 49.802 ms 2024-04-23T14:57:13.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:12 smithi105 ceph-mon[32555]: pgmap v411: 97 pgs: 97 active+clean; 588 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.4 MiB/s wr, 180 op/s 2024-04-23T14:57:13.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:12 smithi105 ceph-mon[32555]: osdmap e102: 12 total, 12 up, 12 in 2024-04-23T14:57:13.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:12 smithi038 ceph-mon[32348]: pgmap v411: 97 pgs: 97 active+clean; 588 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.4 MiB/s wr, 180 op/s 2024-04-23T14:57:13.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:12 smithi038 ceph-mon[32348]: osdmap e102: 12 total, 12 up, 12 in 2024-04-23T14:57:13.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:12 smithi028 ceph-mon[27810]: pgmap v411: 97 pgs: 97 active+clean; 588 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.4 MiB/s wr, 180 op/s 2024-04-23T14:57:13.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:12 smithi028 ceph-mon[27810]: osdmap e102: 12 total, 12 up, 12 in 2024-04-23T14:57:13.548 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1118621 8.44 MB/sec execute 569 sec latency 19.721 ms 2024-04-23T14:57:14.548 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1119757 8.43 MB/sec execute 570 sec latency 16.529 ms 2024-04-23T14:57:15.035 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:14 smithi105 ceph-mon[32555]: pgmap v413: 97 pgs: 97 active+clean; 598 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.3 MiB/s wr, 183 op/s 2024-04-23T14:57:15.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:14 smithi038 ceph-mon[32348]: pgmap v413: 97 pgs: 97 active+clean; 598 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.3 MiB/s wr, 183 op/s 2024-04-23T14:57:15.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:14 smithi028 ceph-mon[27810]: pgmap v413: 97 pgs: 97 active+clean; 598 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.3 MiB/s wr, 183 op/s 2024-04-23T14:57:15.548 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1121311 8.43 MB/sec execute 571 sec latency 54.799 ms 2024-04-23T14:57:16.546 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:57:16.546 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:57:16.548 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1122689 8.43 MB/sec execute 572 sec latency 43.073 ms 2024-04-23T14:57:16.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.756+0000 7fb143a00640 1 -- 172.21.15.28:0/643070356 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb13c104e60 msgr2=0x7fb13c108ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:16.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.756+0000 7fb143a00640 1 --2- 172.21.15.28:0/643070356 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb13c104e60 0x7fb13c108ea0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fb124009860 tx=0x7fb12402a340 comp rx=0 tx=0).stop 2024-04-23T14:57:16.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.757+0000 7fb143a00640 1 -- 172.21.15.28:0/643070356 shutdown_connections 2024-04-23T14:57:16.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.757+0000 7fb143a00640 1 --2- 172.21.15.28:0/643070356 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb13c104e60 0x7fb13c108ea0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:16.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.757+0000 7fb143a00640 1 --2- 172.21.15.28:0/643070356 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb13c109580 0x7fb13c111150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:16.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.757+0000 7fb143a00640 1 --2- 172.21.15.28:0/643070356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb13c1044b0 0x7fb13c104890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:16.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.757+0000 7fb143a00640 1 -- 172.21.15.28:0/643070356 >> 172.21.15.28:0/643070356 conn(0x7fb13c0ffd90 msgr2=0x7fb13c1021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:16.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.757+0000 7fb143a00640 1 -- 172.21.15.28:0/643070356 shutdown_connections 2024-04-23T14:57:16.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.757+0000 7fb143a00640 1 -- 172.21.15.28:0/643070356 wait complete. 2024-04-23T14:57:16.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.759+0000 7fb143a00640 1 Processor -- start 2024-04-23T14:57:16.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.759+0000 7fb143a00640 1 -- start start 2024-04-23T14:57:16.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.760+0000 7fb143a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb13c1044b0 0x7fb13c111150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:16.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.760+0000 7fb143a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb13c109580 0x7fb13c10e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:16.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.760+0000 7fb143a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb13c10e890 0x7fb13c1a2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:16.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.761+0000 7fb141a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb13c10e890 0x7fb13c1a2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:16.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.761+0000 7fb141a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb13c10e890 0x7fb13c1a2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42162/0 (socket says 172.21.15.28:42162) 2024-04-23T14:57:16.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.761+0000 7fb141a00640 1 -- 172.21.15.28:0/3686339358 learned_addr learned my addr 172.21.15.28:0/3686339358 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:16.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.761+0000 7fb143a00640 1 -- 172.21.15.28:0/3686339358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb13c114640 con 0x7fb13c10e890 2024-04-23T14:57:16.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.761+0000 7fb143a00640 1 -- 172.21.15.28:0/3686339358 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb13c1144c0 con 0x7fb13c109580 2024-04-23T14:57:16.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.761+0000 7fb13be00640 1 --2- 172.21.15.28:0/3686339358 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb13c109580 0x7fb13c10e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:16.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.761+0000 7fb141000640 1 --2- 172.21.15.28:0/3686339358 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb13c1044b0 0x7fb13c111150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:16.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.761+0000 7fb143a00640 1 -- 172.21.15.28:0/3686339358 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb13c1147c0 con 0x7fb13c1044b0 2024-04-23T14:57:16.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.762+0000 7fb141000640 1 -- 172.21.15.28:0/3686339358 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb13c109580 msgr2=0x7fb13c10e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:16.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.762+0000 7fb141000640 1 --2- 172.21.15.28:0/3686339358 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb13c109580 0x7fb13c10e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:16.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.762+0000 7fb141000640 1 -- 172.21.15.28:0/3686339358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb13c10e890 msgr2=0x7fb13c1a2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:16.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.763+0000 7fb141000640 1 --2- 172.21.15.28:0/3686339358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb13c10e890 0x7fb13c1a2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:16.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.763+0000 7fb141000640 1 -- 172.21.15.28:0/3686339358 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb13c1a30b0 con 0x7fb13c1044b0 2024-04-23T14:57:16.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.763+0000 7fb141a00640 1 --2- 172.21.15.28:0/3686339358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb13c10e890 0x7fb13c1a2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:57:16.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.763+0000 7fb141000640 1 --2- 172.21.15.28:0/3686339358 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb13c1044b0 0x7fb13c111150 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb12c00e790 tx=0x7fb12c00ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:16.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.767+0000 7fb139600640 1 -- 172.21.15.28:0/3686339358 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12c00eea0 con 0x7fb13c1044b0 2024-04-23T14:57:16.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.768+0000 7fb143a00640 1 -- 172.21.15.28:0/3686339358 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb13c1a33a0 con 0x7fb13c1044b0 2024-04-23T14:57:16.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.769+0000 7fb143a00640 1 -- 172.21.15.28:0/3686339358 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb13c1a38c0 con 0x7fb13c1044b0 2024-04-23T14:57:16.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.769+0000 7fb139600640 1 -- 172.21.15.28:0/3686339358 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb12c00ccc0 con 0x7fb13c1044b0 2024-04-23T14:57:16.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.769+0000 7fb139600640 1 -- 172.21.15.28:0/3686339358 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12c018780 con 0x7fb13c1044b0 2024-04-23T14:57:16.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.774+0000 7fb1469eb740 1 Processor -- start 2024-04-23T14:57:16.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.774+0000 7fb139600640 1 -- 172.21.15.28:0/3686339358 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fb12c018920 con 0x7fb13c1044b0 2024-04-23T14:57:16.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.775+0000 7fb1469eb740 1 -- start start 2024-04-23T14:57:16.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.775+0000 7fb1469eb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5635356a8860 0x5635356a8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:16.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.775+0000 7fb1469eb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635356a9180 0x5635356b3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:16.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.775+0000 7fb1469eb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635356b3850 0x5635356b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:16.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.775+0000 7fb1469eb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56353560e1f0 con 0x5635356b3850 2024-04-23T14:57:16.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.775+0000 7fb1469eb740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5635355b4350 con 0x5635356a9180 2024-04-23T14:57:16.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.775+0000 7fb1469eb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56353560a1b0 con 0x5635356a8860 2024-04-23T14:57:16.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.776+0000 7fb141000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635356a9180 0x5635356b3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:16.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.776+0000 7fb141000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635356a9180 0x5635356b3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:51982/0 (socket says 172.21.15.28:51982) 2024-04-23T14:57:16.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.776+0000 7fb141000640 1 -- 172.21.15.28:0/2935619532 learned_addr learned my addr 172.21.15.28:0/2935619532 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:16.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.776+0000 7fb13be00640 1 --2- 172.21.15.28:0/2935619532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635356b3850 0x5635356b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:16.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.776+0000 7fb141a00640 1 --2- 172.21.15.28:0/2935619532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5635356a8860 0x5635356a8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:16.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.776+0000 7fb141000640 1 -- 172.21.15.28:0/2935619532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5635356a8860 msgr2=0x5635356a8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:16.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.776+0000 7fb141000640 1 --2- 172.21.15.28:0/2935619532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5635356a8860 0x5635356a8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:16.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.776+0000 7fb141000640 1 -- 172.21.15.28:0/2935619532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635356b3850 msgr2=0x5635356b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:16.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.776+0000 7fb141000640 1 --2- 172.21.15.28:0/2935619532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635356b3850 0x5635356b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:16.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.776+0000 7fb141000640 1 -- 172.21.15.28:0/2935619532 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563535608f50 con 0x5635356a9180 2024-04-23T14:57:16.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.777+0000 7fb141000640 1 --2- 172.21.15.28:0/2935619532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635356a9180 0x5635356b3310 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fb12c0cdd60 tx=0x7fb12c0d82c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=307f1fd9120bf4c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:16.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.777+0000 7fb139600640 1 --2- 172.21.15.28:0/3686339358 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb1200a3ea0 0x7fb1200a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:16.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.777+0000 7fb132000640 1 -- 172.21.15.28:0/2935619532 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12c0df070 con 0x5635356a9180 2024-04-23T14:57:16.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.777+0000 7fb141a00640 1 --2- 172.21.15.28:0/3686339358 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb1200a3ea0 0x7fb1200a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:16.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.777+0000 7fb139600640 1 -- 172.21.15.28:0/3686339358 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8542+0+0 (secure 0 0 0) 0x7fb12c0d06b0 con 0x7fb13c1044b0 2024-04-23T14:57:16.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.777+0000 7fb132000640 1 -- 172.21.15.28:0/2935619532 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb12c0a52b0 con 0x5635356a9180 2024-04-23T14:57:16.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.777+0000 7fb132000640 1 -- 172.21.15.28:0/2935619532 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12c09c050 con 0x5635356a9180 2024-04-23T14:57:16.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.778+0000 7fb1469eb740 1 -- 172.21.15.28:0/2935619532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635356a9180 msgr2=0x5635356b3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:16.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.778+0000 7fb141a00640 1 --2- 172.21.15.28:0/3686339358 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb1200a3ea0 0x7fb1200a6360 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb1340049d0 tx=0x7fb134009500 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:16.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.778+0000 7fb1469eb740 1 --2- 172.21.15.28:0/2935619532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635356a9180 0x5635356b3310 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fb12c0cdd60 tx=0x7fb12c0d82c0 comp rx=0 tx=0).stop 2024-04-23T14:57:16.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.778+0000 7fb1469eb740 1 -- 172.21.15.28:0/2935619532 shutdown_connections 2024-04-23T14:57:16.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.778+0000 7fb1469eb740 1 --2- 172.21.15.28:0/2935619532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635356a9180 0x5635356b3310 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:16.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.778+0000 7fb1469eb740 1 --2- 172.21.15.28:0/2935619532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635356b3850 0x5635356b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:16.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.778+0000 7fb1469eb740 1 --2- 172.21.15.28:0/2935619532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5635356a8860 0x5635356a8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:16.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.778+0000 7fb1469eb740 1 -- 172.21.15.28:0/2935619532 >> 172.21.15.28:0/2935619532 conn(0x5635356a4b80 msgr2=0x5635356a4f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:16.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.778+0000 7fb1469eb740 1 -- 172.21.15.28:0/2935619532 shutdown_connections 2024-04-23T14:57:16.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.779+0000 7fb1469eb740 1 -- 172.21.15.28:0/2935619532 wait complete. 2024-04-23T14:57:16.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.779+0000 7fb1469eb740 1 -- start start 2024-04-23T14:57:16.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.779+0000 7fb1469eb740 1 Processor -- start 2024-04-23T14:57:16.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.780+0000 7fb1469eb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635356a8860 0x5635356b06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:16.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.780+0000 7fb1469eb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635356a9180 0x5635356b0c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:16.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.780+0000 7fb1469eb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5635356b3850 0x5635356b1150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:16.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.780+0000 7fb1469eb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5635356b6640 con 0x5635356a8860 2024-04-23T14:57:16.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.780+0000 7fb1469eb740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5635355c7690 con 0x5635356a9180 2024-04-23T14:57:16.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.780+0000 7fb1469eb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5635356b67c0 con 0x5635356b3850 2024-04-23T14:57:16.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.780+0000 7fb1469eb740 10 client.0 _ll_register_callbacks cb 0x563535603650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:57:16.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.780+0000 7fb13be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635356a9180 0x5635356b0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:16.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.780+0000 7fb13be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635356a9180 0x5635356b0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:51990/0 (socket says 172.21.15.28:51990) 2024-04-23T14:57:16.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.780+0000 7fb13be00640 1 -- 172.21.15.28:0/1540172256 learned_addr learned my addr 172.21.15.28:0/1540172256 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:16.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.780+0000 7fb141a00640 1 --2- 172.21.15.28:0/1540172256 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5635356b3850 0x5635356b1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:16.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.780+0000 7fb141000640 1 --2- 172.21.15.28:0/1540172256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635356a8860 0x5635356b06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:16.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.780+0000 7fb13be00640 1 -- 172.21.15.28:0/1540172256 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5635356b3850 msgr2=0x5635356b1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:16.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.780+0000 7fb13be00640 1 --2- 172.21.15.28:0/1540172256 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5635356b3850 0x5635356b1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:16.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.780+0000 7fb13be00640 1 -- 172.21.15.28:0/1540172256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635356a8860 msgr2=0x5635356b06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:16.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.780+0000 7fb13be00640 1 --2- 172.21.15.28:0/1540172256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635356a8860 0x5635356b06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:16.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.780+0000 7fb13be00640 1 -- 172.21.15.28:0/1540172256 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635355c6920 con 0x5635356a9180 2024-04-23T14:57:16.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.780+0000 7fb141a00640 1 --2- 172.21.15.28:0/1540172256 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5635356b3850 0x5635356b1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:57:16.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.781+0000 7fb13be00640 1 --2- 172.21.15.28:0/1540172256 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635356a9180 0x5635356b0c10 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fb124000c00 tx=0x7fb124009520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cd919006e8cdae3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:16.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.781+0000 7fb12aa00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:57:16.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.781+0000 7fb12aa00640 1 -- 172.21.15.28:0/1540172256 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb124037040 con 0x5635356a9180 2024-04-23T14:57:16.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.781+0000 7fb12aa00640 1 -- 172.21.15.28:0/1540172256 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8542+0+0 (secure 0 0 0) 0x7fb124030550 con 0x5635356a9180 2024-04-23T14:57:16.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.781+0000 7fb1469eb740 1 -- 172.21.15.28:0/1540172256 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb12c0a52b0 con 0x5635356a9180 2024-04-23T14:57:16.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.782+0000 7fb12aa00640 1 -- 172.21.15.28:0/1540172256 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fb12404c050 con 0x5635356a9180 2024-04-23T14:57:16.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.782+0000 7fb1469eb740 10 client.27263 fetch_fsmap learned FSMap version 20 2024-04-23T14:57:16.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.782+0000 7fb1469eb740 1 -- 172.21.15.28:0/1540172256 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5635353ea7d0 con 0x5635356a9180 2024-04-23T14:57:16.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.782+0000 7fb12aa00640 1 -- 172.21.15.28:0/1540172256 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb12c0a52b0 con 0x5635356a9180 2024-04-23T14:57:16.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.782+0000 7fb12aa00640 1 -- 172.21.15.28:0/1540172256 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fb12403d450 con 0x5635356a9180 2024-04-23T14:57:16.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.782+0000 7fb1469eb740 10 client.27263 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:57:16.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.782+0000 7fb1469eb740 10 client.27263 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:57:16.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.782+0000 7fb1469eb740 20 client.27263 populate_metadata read hostname 'smithi028' 2024-04-23T14:57:16.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.782+0000 7fb1469eb740 1 --2- 172.21.15.28:0/1540172256 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5635356c3ba0 0x5635356c6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:16.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.782+0000 7fb1469eb740 4 client.27263 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:57:16.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.782+0000 7fb1469eb740 1 -- 172.21.15.28:0/1540172256 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56353561bc10 con 0x5635356c3ba0 2024-04-23T14:57:16.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.783+0000 7fb141000640 1 --2- 172.21.15.28:0/1540172256 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5635356c3ba0 0x5635356c6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:16.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.783+0000 7fb141000640 1 --2- 172.21.15.28:0/1540172256 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5635356c3ba0 0x5635356c6020 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f12e74e37d41fdca server_cookie=d8f604e69ba2cd2a in_seq=0 out_seq=0 2024-04-23T14:57:16.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.783+0000 7fb12aa00640 10 client.27263 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:57:16.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.785+0000 7fb12aa00640 1 -- 172.21.15.28:0/1540172256 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56353561bc10 con 0x5635356c3ba0 2024-04-23T14:57:16.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.785+0000 7fb12aa00640 10 client.27263 handle_command_reply: tid=0 2024-04-23T14:57:16.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.785+0000 7fb1469eb740 1 client.27263 shutdown 2024-04-23T14:57:16.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.785+0000 7fb1469eb740 1 -- 172.21.15.28:0/1540172256 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635356a9180 msgr2=0x5635356b0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:16.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.785+0000 7fb1469eb740 1 --2- 172.21.15.28:0/1540172256 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635356a9180 0x5635356b0c10 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fb124000c00 tx=0x7fb124009520 comp rx=0 tx=0).stop 2024-04-23T14:57:16.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.786+0000 7fb1469eb740 1 -- 172.21.15.28:0/1540172256 shutdown_connections 2024-04-23T14:57:16.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.786+0000 7fb1469eb740 1 --2- 172.21.15.28:0/1540172256 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5635356c3ba0 0x5635356c6020 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:16.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.786+0000 7fb1469eb740 1 --2- 172.21.15.28:0/1540172256 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635356a9180 0x5635356b0c10 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:16.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.786+0000 7fb1469eb740 1 --2- 172.21.15.28:0/1540172256 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5635356b3850 0x5635356b1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:16.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.786+0000 7fb1469eb740 1 --2- 172.21.15.28:0/1540172256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635356a8860 0x5635356b06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:16.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.786+0000 7fb12aa00640 0 client.27263 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:57:16.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.786+0000 7fb1469eb740 1 -- 172.21.15.28:0/1540172256 >> 172.21.15.28:0/1540172256 conn(0x5635356a4b80 msgr2=0x5635356aa860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:16.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.786+0000 7fb1469eb740 1 -- 172.21.15.28:0/1540172256 shutdown_connections 2024-04-23T14:57:16.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.786+0000 7fb1469eb740 1 -- 172.21.15.28:0/1540172256 wait complete. 2024-04-23T14:57:16.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.786+0000 7fb1469eb740 20 client.27263 trim_cache size 0 max 16384 2024-04-23T14:57:16.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.808+0000 7fb1469eb740 1 Processor -- start 2024-04-23T14:57:16.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.808+0000 7fb1469eb740 1 -- start start 2024-04-23T14:57:16.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.808+0000 7fb1469eb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635356a8860 0x5635356bf560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:16.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.808+0000 7fb1469eb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5635356a9180 0x5635356af850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:16.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.808+0000 7fb1469eb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635356b3850 0x5635356aafc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:16.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.808+0000 7fb1469eb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb11800c7e0 con 0x5635356a8860 2024-04-23T14:57:16.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.808+0000 7fb1469eb740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb11800f040 con 0x5635356b3850 2024-04-23T14:57:16.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.808+0000 7fb1469eb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb11800ce30 con 0x5635356a9180 2024-04-23T14:57:16.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.809+0000 7fb141a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635356b3850 0x5635356aafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:16.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.809+0000 7fb141000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635356a8860 0x5635356bf560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:16.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.809+0000 7fb141a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635356b3850 0x5635356aafc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:51996/0 (socket says 172.21.15.28:51996) 2024-04-23T14:57:16.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.809+0000 7fb141a00640 1 -- 172.21.15.28:0/793895849 learned_addr learned my addr 172.21.15.28:0/793895849 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:16.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.809+0000 7fb13be00640 1 --2- 172.21.15.28:0/793895849 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5635356a9180 0x5635356af850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:16.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.809+0000 7fb141a00640 1 -- 172.21.15.28:0/793895849 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5635356a9180 msgr2=0x5635356af850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:16.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.809+0000 7fb141a00640 1 --2- 172.21.15.28:0/793895849 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5635356a9180 0x5635356af850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:16.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.809+0000 7fb141a00640 1 -- 172.21.15.28:0/793895849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635356a8860 msgr2=0x5635356bf560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:16.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.809+0000 7fb141a00640 1 --2- 172.21.15.28:0/793895849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635356a8860 0x5635356bf560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:16.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.809+0000 7fb141a00640 1 -- 172.21.15.28:0/793895849 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635353203f0 con 0x5635356b3850 2024-04-23T14:57:16.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.809+0000 7fb141000640 1 --2- 172.21.15.28:0/793895849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635356a8860 0x5635356bf560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:57:16.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.809+0000 7fb141a00640 1 --2- 172.21.15.28:0/793895849 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635356b3850 0x5635356aafc0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fb13400fd80 tx=0x7fb13401a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69f1e69e68f17a3c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:16.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.810+0000 7fb131600640 1 -- 172.21.15.28:0/793895849 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13400fdb0 con 0x5635356b3850 2024-04-23T14:57:16.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.810+0000 7fb131600640 1 -- 172.21.15.28:0/793895849 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb13401f900 con 0x5635356b3850 2024-04-23T14:57:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.810+0000 7fb131600640 1 -- 172.21.15.28:0/793895849 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13401d930 con 0x5635356b3850 2024-04-23T14:57:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.810+0000 7fb1469eb740 1 -- 172.21.15.28:0/793895849 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635356b3850 msgr2=0x5635356aafc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.810+0000 7fb1469eb740 1 --2- 172.21.15.28:0/793895849 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635356b3850 0x5635356aafc0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fb13400fd80 tx=0x7fb13401a000 comp rx=0 tx=0).stop 2024-04-23T14:57:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.810+0000 7fb1469eb740 1 -- 172.21.15.28:0/793895849 shutdown_connections 2024-04-23T14:57:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.810+0000 7fb1469eb740 1 --2- 172.21.15.28:0/793895849 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635356b3850 0x5635356aafc0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.810+0000 7fb1469eb740 1 --2- 172.21.15.28:0/793895849 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5635356a9180 0x5635356af850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.810+0000 7fb1469eb740 1 --2- 172.21.15.28:0/793895849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635356a8860 0x5635356bf560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.810+0000 7fb1469eb740 1 -- 172.21.15.28:0/793895849 >> 172.21.15.28:0/793895849 conn(0x5635356a4b80 msgr2=0x5635356b0690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.811+0000 7fb1469eb740 1 -- 172.21.15.28:0/793895849 shutdown_connections 2024-04-23T14:57:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.811+0000 7fb1469eb740 1 -- 172.21.15.28:0/793895849 wait complete. 2024-04-23T14:57:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.811+0000 7fb1469eb740 1 -- start start 2024-04-23T14:57:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.811+0000 7fb1469eb740 1 Processor -- start 2024-04-23T14:57:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.812+0000 7fb1469eb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635356a8860 0x5635356bbe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.812+0000 7fb1469eb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5635356a9180 0x5635356ad120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.812+0000 7fb1469eb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635356b3850 0x5635356ad6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.812+0000 7fb1469eb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb11800d500 con 0x5635356b3850 2024-04-23T14:57:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.812+0000 7fb1469eb740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb11800e970 con 0x5635356a8860 2024-04-23T14:57:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.812+0000 7fb1469eb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb11800dbd0 con 0x5635356a9180 2024-04-23T14:57:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.812+0000 7fb1469eb740 10 client.0 _ll_register_callbacks cb 0x7fb1340093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:57:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.812+0000 7fb141000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635356a8860 0x5635356bbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.812+0000 7fb141000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635356a8860 0x5635356bbe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:52010/0 (socket says 172.21.15.28:52010) 2024-04-23T14:57:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.812+0000 7fb141000640 1 -- 172.21.15.28:0/3138087595 learned_addr learned my addr 172.21.15.28:0/3138087595 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.812+0000 7fb141a00640 1 --2- 172.21.15.28:0/3138087595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635356b3850 0x5635356ad6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.812+0000 7fb13be00640 1 --2- 172.21.15.28:0/3138087595 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5635356a9180 0x5635356ad120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:16.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.812+0000 7fb141000640 1 -- 172.21.15.28:0/3138087595 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5635356a9180 msgr2=0x5635356ad120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:16.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.812+0000 7fb141000640 1 --2- 172.21.15.28:0/3138087595 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5635356a9180 0x5635356ad120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:16.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.812+0000 7fb141000640 1 -- 172.21.15.28:0/3138087595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635356b3850 msgr2=0x5635356ad6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:16.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.812+0000 7fb141000640 1 --2- 172.21.15.28:0/3138087595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635356b3850 0x5635356ad6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:16.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.812+0000 7fb141000640 1 -- 172.21.15.28:0/3138087595 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635352e00f0 con 0x5635356a8860 2024-04-23T14:57:16.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.812+0000 7fb141a00640 1 --2- 172.21.15.28:0/3138087595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635356b3850 0x5635356ad6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:57:16.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.813+0000 7fb141000640 1 --2- 172.21.15.28:0/3138087595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635356a8860 0x5635356bbe90 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fb12c0058c0 tx=0x7fb12c009690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a841e9725da7646 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:16.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.813+0000 7fb12aa00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:57:16.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.813+0000 7fb12aa00640 1 -- 172.21.15.28:0/3138087595 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12c0df040 con 0x5635356a8860 2024-04-23T14:57:16.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.813+0000 7fb12aa00640 1 -- 172.21.15.28:0/3138087595 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8542+0+0 (secure 0 0 0) 0x7fb12c0d6730 con 0x5635356a8860 2024-04-23T14:57:16.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.813+0000 7fb1469eb740 1 -- 172.21.15.28:0/3138087595 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb13401f900 con 0x5635356a8860 2024-04-23T14:57:16.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.813+0000 7fb12aa00640 1 -- 172.21.15.28:0/3138087595 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fb12c098990 con 0x5635356a8860 2024-04-23T14:57:16.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.813+0000 7fb1469eb740 10 client.27269 fetch_fsmap learned FSMap version 20 2024-04-23T14:57:16.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.813+0000 7fb1469eb740 1 -- 172.21.15.28:0/3138087595 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5635355d3810 con 0x5635356a8860 2024-04-23T14:57:16.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.813+0000 7fb12aa00640 1 -- 172.21.15.28:0/3138087595 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb13401f900 con 0x5635356a8860 2024-04-23T14:57:16.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.814+0000 7fb12aa00640 1 -- 172.21.15.28:0/3138087595 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fb12c0b5020 con 0x5635356a8860 2024-04-23T14:57:16.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.814+0000 7fb1469eb740 10 client.27269 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:57:16.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.814+0000 7fb1469eb740 10 client.27269 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:57:16.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.814+0000 7fb1469eb740 20 client.27269 populate_metadata read hostname 'smithi028' 2024-04-23T14:57:16.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.814+0000 7fb1469eb740 1 --2- 172.21.15.28:0/3138087595 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56353572a1e0 0x56353572c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:16.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.814+0000 7fb1469eb740 4 client.27269 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:57:16.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.814+0000 7fb1469eb740 1 -- 172.21.15.28:0/3138087595 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563535615a80 con 0x56353572a1e0 2024-04-23T14:57:16.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.814+0000 7fb13be00640 1 --2- 172.21.15.28:0/3138087595 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56353572a1e0 0x56353572c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:16.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.815+0000 7fb13be00640 1 --2- 172.21.15.28:0/3138087595 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56353572a1e0 0x56353572c5d0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9e5a3d79f3c64d9 server_cookie=ec20bdff82b46eb3 in_seq=0 out_seq=0 2024-04-23T14:57:16.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.815+0000 7fb12aa00640 10 client.27269 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:57:16.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.816+0000 7fb12aa00640 1 -- 172.21.15.28:0/3138087595 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563535615a80 con 0x56353572a1e0 2024-04-23T14:57:16.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.816+0000 7fb12aa00640 10 client.27269 handle_command_reply: tid=0 2024-04-23T14:57:16.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.816+0000 7fb1469eb740 1 client.27269 shutdown 2024-04-23T14:57:16.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.816+0000 7fb1469eb740 1 -- 172.21.15.28:0/3138087595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635356a8860 msgr2=0x5635356bbe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:16.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.816+0000 7fb1469eb740 1 --2- 172.21.15.28:0/3138087595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635356a8860 0x5635356bbe90 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fb12c0058c0 tx=0x7fb12c009690 comp rx=0 tx=0).stop 2024-04-23T14:57:16.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.816+0000 7fb1469eb740 1 -- 172.21.15.28:0/3138087595 shutdown_connections 2024-04-23T14:57:16.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.816+0000 7fb1469eb740 1 --2- 172.21.15.28:0/3138087595 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x56353572a1e0 0x56353572c5d0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:16.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.816+0000 7fb1469eb740 1 --2- 172.21.15.28:0/3138087595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635356b3850 0x5635356ad6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:16.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.816+0000 7fb1469eb740 1 --2- 172.21.15.28:0/3138087595 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5635356a9180 0x5635356ad120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:16.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.816+0000 7fb1469eb740 1 --2- 172.21.15.28:0/3138087595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635356a8860 0x5635356bbe90 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:16.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.816+0000 7fb12aa00640 0 client.27269 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:57:16.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.816+0000 7fb1469eb740 1 -- 172.21.15.28:0/3138087595 >> 172.21.15.28:0/3138087595 conn(0x5635356a4b80 msgr2=0x5635356a3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:16.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.816+0000 7fb1469eb740 1 -- 172.21.15.28:0/3138087595 shutdown_connections 2024-04-23T14:57:16.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.817+0000 7fb1469eb740 1 -- 172.21.15.28:0/3138087595 wait complete. 2024-04-23T14:57:16.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.817+0000 7fb1469eb740 20 client.27269 trim_cache size 0 max 16384 2024-04-23T14:57:16.823 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:57:16.823 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-23T14:57:16.823 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-23T14:57:16.823 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:57:16.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.817+0000 7fb143a00640 1 -- 172.21.15.28:0/3686339358 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb1200a3ea0 msgr2=0x7fb1200a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:16.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.817+0000 7fb143a00640 1 --2- 172.21.15.28:0/3686339358 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb1200a3ea0 0x7fb1200a6360 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb1340049d0 tx=0x7fb134009500 comp rx=0 tx=0).stop 2024-04-23T14:57:16.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.817+0000 7fb143a00640 1 -- 172.21.15.28:0/3686339358 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb13c1044b0 msgr2=0x7fb13c111150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:16.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.817+0000 7fb143a00640 1 --2- 172.21.15.28:0/3686339358 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb13c1044b0 0x7fb13c111150 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb12c00e790 tx=0x7fb12c00ec50 comp rx=0 tx=0).stop 2024-04-23T14:57:16.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.817+0000 7fb143a00640 1 -- 172.21.15.28:0/3686339358 shutdown_connections 2024-04-23T14:57:16.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.817+0000 7fb143a00640 1 --2- 172.21.15.28:0/3686339358 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fb1200a3ea0 0x7fb1200a6360 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:16.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.817+0000 7fb143a00640 1 --2- 172.21.15.28:0/3686339358 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb13c109580 0x7fb13c10e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:16.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.817+0000 7fb143a00640 1 --2- 172.21.15.28:0/3686339358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb13c10e890 0x7fb13c1a2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:16.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.817+0000 7fb143a00640 1 --2- 172.21.15.28:0/3686339358 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb13c1044b0 0x7fb13c111150 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:16.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.817+0000 7fb143a00640 1 -- 172.21.15.28:0/3686339358 >> 172.21.15.28:0/3686339358 conn(0x7fb13c0ffd90 msgr2=0x7fb13c1010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:16.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.817+0000 7fb143a00640 1 -- 172.21.15.28:0/3686339358 shutdown_connections 2024-04-23T14:57:16.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:16.818+0000 7fb143a00640 1 -- 172.21.15.28:0/3686339358 wait complete. 2024-04-23T14:57:16.836 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-23T14:57:16.857 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:57:16.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:16 smithi028 ceph-mon[27810]: pgmap v414: 97 pgs: 97 active+clean; 598 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.5 MiB/s wr, 150 op/s 2024-04-23T14:57:17.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:16 smithi105 ceph-mon[32555]: pgmap v414: 97 pgs: 97 active+clean; 598 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.5 MiB/s wr, 150 op/s 2024-04-23T14:57:17.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.066+0000 7f271d800640 1 -- 172.21.15.28:0/350246210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2718109510 msgr2=0x7f27181110e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:17.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.066+0000 7f271d800640 1 --2- 172.21.15.28:0/350246210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2718109510 0x7f27181110e0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f2710009ca0 tx=0x7f271002a8b0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.066+0000 7f271d800640 1 -- 172.21.15.28:0/350246210 shutdown_connections 2024-04-23T14:57:17.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.066+0000 7f271d800640 1 --2- 172.21.15.28:0/350246210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2718109510 0x7f27181110e0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.066+0000 7f271d800640 1 --2- 172.21.15.28:0/350246210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2718104df0 0x7f2718108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.066+0000 7f271d800640 1 --2- 172.21.15.28:0/350246210 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2718104440 0x7f2718104820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.066+0000 7f271d800640 1 -- 172.21.15.28:0/350246210 >> 172.21.15.28:0/350246210 conn(0x7f27180ffcc0 msgr2=0x7f27181020e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:17.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.066+0000 7f271d800640 1 -- 172.21.15.28:0/350246210 shutdown_connections 2024-04-23T14:57:17.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.066+0000 7f271d800640 1 -- 172.21.15.28:0/350246210 wait complete. 2024-04-23T14:57:17.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.068+0000 7f271d800640 1 Processor -- start 2024-04-23T14:57:17.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.068+0000 7f271d800640 1 -- start start 2024-04-23T14:57:17.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.068+0000 7f271d800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2718104440 0x7f27181a0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:17.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.068+0000 7f271d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2718109510 0x7f27181a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:17.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.068+0000 7f271d800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f271819a8c0 0x7f271819ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:17.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.068+0000 7f271d800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2718114620 con 0x7f2718109510 2024-04-23T14:57:17.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.068+0000 7f271d800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f27181144a0 con 0x7f271819a8c0 2024-04-23T14:57:17.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.068+0000 7f271d800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f27181147a0 con 0x7f2718104440 2024-04-23T14:57:17.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.069+0000 7f2716a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2718104440 0x7f27181a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:17.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.069+0000 7f2716a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2718104440 0x7f27181a0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:47220/0 (socket says 172.21.15.28:47220) 2024-04-23T14:57:17.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.069+0000 7f2717400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f271819a8c0 0x7f271819ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:17.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.069+0000 7f2716a00640 1 -- 172.21.15.28:0/1141967562 learned_addr learned my addr 172.21.15.28:0/1141967562 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:17.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.069+0000 7f2716000640 1 --2- 172.21.15.28:0/1141967562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2718109510 0x7f27181a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:17.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.069+0000 7f2716a00640 1 -- 172.21.15.28:0/1141967562 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f271819a8c0 msgr2=0x7f271819ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:17.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.069+0000 7f2716a00640 1 --2- 172.21.15.28:0/1141967562 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f271819a8c0 0x7f271819ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.069+0000 7f2716a00640 1 -- 172.21.15.28:0/1141967562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2718109510 msgr2=0x7f27181a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:17.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.069+0000 7f2716a00640 1 --2- 172.21.15.28:0/1141967562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2718109510 0x7f27181a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.069+0000 7f2716a00640 1 -- 172.21.15.28:0/1141967562 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f271819b3e0 con 0x7f2718104440 2024-04-23T14:57:17.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.070+0000 7f2716a00640 1 --2- 172.21.15.28:0/1141967562 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2718104440 0x7f27181a0710 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f270800e9f0 tx=0x7f270800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:17.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.070+0000 7f270f400640 1 -- 172.21.15.28:0/1141967562 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f270800cd80 con 0x7f2718104440 2024-04-23T14:57:17.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.070+0000 7f271d800640 1 -- 172.21.15.28:0/1141967562 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f271819b6d0 con 0x7f2718104440 2024-04-23T14:57:17.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.070+0000 7f270f400640 1 -- 172.21.15.28:0/1141967562 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2708005500 con 0x7f2718104440 2024-04-23T14:57:17.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.070+0000 7f270f400640 1 -- 172.21.15.28:0/1141967562 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2708010690 con 0x7f2718104440 2024-04-23T14:57:17.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.070+0000 7f271d800640 1 -- 172.21.15.28:0/1141967562 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27181a78d0 con 0x7f2718104440 2024-04-23T14:57:17.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.072+0000 7f271d800640 1 -- 172.21.15.28:0/1141967562 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2718069270 con 0x7f2718104440 2024-04-23T14:57:17.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.074+0000 7f270f400640 1 -- 172.21.15.28:0/1141967562 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f27080046c0 con 0x7f2718104440 2024-04-23T14:57:17.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.076+0000 7f270f400640 1 --2- 172.21.15.28:0/1141967562 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f26fc0a3d20 0x7f26fc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:17.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.076+0000 7f270f400640 1 -- 172.21.15.28:0/1141967562 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8542+0+0 (secure 0 0 0) 0x7f27080d0c30 con 0x7f2718104440 2024-04-23T14:57:17.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.076+0000 7f2716000640 1 --2- 172.21.15.28:0/1141967562 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f26fc0a3d20 0x7f26fc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:17.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.077+0000 7f2716000640 1 --2- 172.21.15.28:0/1141967562 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f26fc0a3d20 0x7f26fc0a61e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2718103390 tx=0x7f2704005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:17.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.077+0000 7f270f400640 1 -- 172.21.15.28:0/1141967562 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f270809d260 con 0x7f2718104440 2024-04-23T14:57:17.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:16 smithi038 ceph-mon[32348]: pgmap v414: 97 pgs: 97 active+clean; 598 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.5 MiB/s wr, 150 op/s 2024-04-23T14:57:17.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.343+0000 7f271d800640 1 -- 172.21.15.28:0/1141967562 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f271819bdf0 con 0x7f2718104440 2024-04-23T14:57:17.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.344+0000 7f270f400640 1 -- 172.21.15.28:0/1141967562 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7f270809cb60 con 0x7f2718104440 2024-04-23T14:57:17.345 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:57:17.345 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:57:17.345 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:57:17.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.347+0000 7f271d800640 1 -- 172.21.15.28:0/1141967562 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f26fc0a3d20 msgr2=0x7f26fc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:17.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.347+0000 7f271d800640 1 --2- 172.21.15.28:0/1141967562 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f26fc0a3d20 0x7f26fc0a61e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2718103390 tx=0x7f2704005f90 comp rx=0 tx=0).stop 2024-04-23T14:57:17.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.347+0000 7f271d800640 1 -- 172.21.15.28:0/1141967562 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2718104440 msgr2=0x7f27181a0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:17.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.347+0000 7f271d800640 1 --2- 172.21.15.28:0/1141967562 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2718104440 0x7f27181a0710 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f270800e9f0 tx=0x7f270800edf0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.347+0000 7f271d800640 1 -- 172.21.15.28:0/1141967562 shutdown_connections 2024-04-23T14:57:17.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.347+0000 7f271d800640 1 --2- 172.21.15.28:0/1141967562 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f26fc0a3d20 0x7f26fc0a61e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.347+0000 7f271d800640 1 --2- 172.21.15.28:0/1141967562 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f271819a8c0 0x7f271819ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.347+0000 7f271d800640 1 --2- 172.21.15.28:0/1141967562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2718109510 0x7f27181a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.347+0000 7f271d800640 1 --2- 172.21.15.28:0/1141967562 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2718104440 0x7f27181a0710 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.347+0000 7f271d800640 1 -- 172.21.15.28:0/1141967562 >> 172.21.15.28:0/1141967562 conn(0x7f27180ffcc0 msgr2=0x7f27181007d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:17.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.347+0000 7f271d800640 1 -- 172.21.15.28:0/1141967562 shutdown_connections 2024-04-23T14:57:17.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.347+0000 7f271d800640 1 -- 172.21.15.28:0/1141967562 wait complete. 2024-04-23T14:57:17.369 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-23T14:57:17.548 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1123947 8.43 MB/sec execute 573 sec latency 34.943 ms 2024-04-23T14:57:17.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.626+0000 7f22d2c00640 1 -- 172.21.15.28:0/1669068607 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f22cc0743b0 msgr2=0x7f22cc074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:17.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.626+0000 7f22d2c00640 1 --2- 172.21.15.28:0/1669068607 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f22cc0743b0 0x7f22cc074790 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f22bc009900 tx=0x7f22bc02a490 comp rx=0 tx=0).stop 2024-04-23T14:57:17.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.627+0000 7f22d2c00640 1 -- 172.21.15.28:0/1669068607 shutdown_connections 2024-04-23T14:57:17.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.627+0000 7f22d2c00640 1 --2- 172.21.15.28:0/1669068607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22cc10e760 0x7f22cc110b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.627+0000 7f22d2c00640 1 --2- 172.21.15.28:0/1669068607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22cc074cd0 0x7f22cc10e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.627+0000 7f22d2c00640 1 --2- 172.21.15.28:0/1669068607 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f22cc0743b0 0x7f22cc074790 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.627+0000 7f22d2c00640 1 -- 172.21.15.28:0/1669068607 >> 172.21.15.28:0/1669068607 conn(0x7f22cc0fd9d0 msgr2=0x7f22cc0ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:17.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.627+0000 7f22d2c00640 1 -- 172.21.15.28:0/1669068607 shutdown_connections 2024-04-23T14:57:17.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.627+0000 7f22d2c00640 1 -- 172.21.15.28:0/1669068607 wait complete. 2024-04-23T14:57:17.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.628+0000 7f22d2c00640 1 Processor -- start 2024-04-23T14:57:17.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.628+0000 7f22d2c00640 1 -- start start 2024-04-23T14:57:17.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.629+0000 7f22d2c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f22cc0743b0 0x7f22cc1a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:17.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.629+0000 7f22d2c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22cc074cd0 0x7f22cc1a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:17.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.629+0000 7f22d2c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22cc10e760 0x7f22cc19a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:17.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.629+0000 7f22d2c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f22cc1144b0 con 0x7f22cc10e760 2024-04-23T14:57:17.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.629+0000 7f22d2c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f22cc114330 con 0x7f22cc074cd0 2024-04-23T14:57:17.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.629+0000 7f22d2c00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f22cc114630 con 0x7f22cc0743b0 2024-04-23T14:57:17.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.629+0000 7f22d1800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f22cc0743b0 0x7f22cc1a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:17.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.629+0000 7f22d0e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22cc074cd0 0x7f22cc1a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:17.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.629+0000 7f22d1800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f22cc0743b0 0x7f22cc1a0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:47230/0 (socket says 172.21.15.28:47230) 2024-04-23T14:57:17.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.629+0000 7f22d1800640 1 -- 172.21.15.28:0/3859178 learned_addr learned my addr 172.21.15.28:0/3859178 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:17.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.629+0000 7f22d2200640 1 --2- 172.21.15.28:0/3859178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22cc10e760 0x7f22cc19a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:17.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.629+0000 7f22d0e00640 1 -- 172.21.15.28:0/3859178 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f22cc0743b0 msgr2=0x7f22cc1a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:17.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.629+0000 7f22d0e00640 1 --2- 172.21.15.28:0/3859178 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f22cc0743b0 0x7f22cc1a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.630+0000 7f22d0e00640 1 -- 172.21.15.28:0/3859178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22cc10e760 msgr2=0x7f22cc19a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:17.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.630+0000 7f22d0e00640 1 --2- 172.21.15.28:0/3859178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22cc10e760 0x7f22cc19a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.630+0000 7f22d0e00640 1 -- 172.21.15.28:0/3859178 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22cc19afc0 con 0x7f22cc074cd0 2024-04-23T14:57:17.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.630+0000 7f22d0e00640 1 --2- 172.21.15.28:0/3859178 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22cc074cd0 0x7f22cc1a0b70 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f22b800eb00 tx=0x7f22b800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:17.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.630+0000 7f22ca000640 1 -- 172.21.15.28:0/3859178 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b8010430 con 0x7f22cc074cd0 2024-04-23T14:57:17.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.630+0000 7f22d2c00640 1 -- 172.21.15.28:0/3859178 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22cc19b2b0 con 0x7f22cc074cd0 2024-04-23T14:57:17.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.630+0000 7f22ca000640 1 -- 172.21.15.28:0/3859178 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22b8010ab0 con 0x7f22cc074cd0 2024-04-23T14:57:17.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.630+0000 7f22ca000640 1 -- 172.21.15.28:0/3859178 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b8017bb0 con 0x7f22cc074cd0 2024-04-23T14:57:17.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.630+0000 7f22d2c00640 1 -- 172.21.15.28:0/3859178 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22cc1a7480 con 0x7f22cc074cd0 2024-04-23T14:57:17.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.634+0000 7f22d7172740 1 Processor -- start 2024-04-23T14:57:17.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.634+0000 7f22d7172740 1 -- start start 2024-04-23T14:57:17.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.634+0000 7f22ca000640 1 -- 172.21.15.28:0/3859178 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f22b8010c60 con 0x7f22cc074cd0 2024-04-23T14:57:17.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.635+0000 7f22d7172740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9a5791b40 0x55a9a5791f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:17.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.635+0000 7f22d7172740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a5792460 0x55a9a579c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:17.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.635+0000 7f22d7172740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9a579cb30 0x55a9a579ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:17.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.635+0000 7f22d7172740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a9a56f71d0 con 0x55a9a5791b40 2024-04-23T14:57:17.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.635+0000 7f22d7172740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a9a569d450 con 0x55a9a579cb30 2024-04-23T14:57:17.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.635+0000 7f22d7172740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a9a56f3190 con 0x55a9a5792460 2024-04-23T14:57:17.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.635+0000 7f22d2200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9a5791b40 0x55a9a5791f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:17.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.635+0000 7f22d2200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9a5791b40 0x55a9a5791f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42232/0 (socket says 172.21.15.28:42232) 2024-04-23T14:57:17.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.635+0000 7f22d2200640 1 -- 172.21.15.28:0/384183449 learned_addr learned my addr 172.21.15.28:0/384183449 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:17.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.635+0000 7f22d1800640 1 --2- 172.21.15.28:0/384183449 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a5792460 0x55a9a579c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:17.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.635+0000 7f22d0e00640 1 --2- 172.21.15.28:0/384183449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9a579cb30 0x55a9a579ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:17.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.635+0000 7f22d2200640 1 -- 172.21.15.28:0/384183449 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a5792460 msgr2=0x55a9a579c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:17.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.635+0000 7f22d2200640 1 --2- 172.21.15.28:0/384183449 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a5792460 0x55a9a579c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.635+0000 7f22d2200640 1 -- 172.21.15.28:0/384183449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9a579cb30 msgr2=0x55a9a579ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:17.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.635+0000 7f22d2200640 1 --2- 172.21.15.28:0/384183449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9a579cb30 0x55a9a579ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.635+0000 7f22d2200640 1 -- 172.21.15.28:0/384183449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9a575a310 con 0x55a9a5791b40 2024-04-23T14:57:17.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.635+0000 7f22d1800640 1 --2- 172.21.15.28:0/384183449 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a5792460 0x55a9a579c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:57:17.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.636+0000 7f22d2200640 1 --2- 172.21.15.28:0/384183449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9a5791b40 0x55a9a5791f20 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f22c4009c20 tx=0x7f22c4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a6c5466f2826db6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:17.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.636+0000 7f22c2a00640 1 -- 172.21.15.28:0/384183449 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c401e070 con 0x55a9a5791b40 2024-04-23T14:57:17.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.636+0000 7f22c2a00640 1 -- 172.21.15.28:0/384183449 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22c4019b10 con 0x55a9a5791b40 2024-04-23T14:57:17.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.636+0000 7f22c2a00640 1 -- 172.21.15.28:0/384183449 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c400e9e0 con 0x55a9a5791b40 2024-04-23T14:57:17.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.637+0000 7f22ca000640 1 --2- 172.21.15.28:0/3859178 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f22b00a3da0 0x7f22b00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:17.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.637+0000 7f22ca000640 1 -- 172.21.15.28:0/3859178 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8542+0+0 (secure 0 0 0) 0x7f22b80d1150 con 0x7f22cc074cd0 2024-04-23T14:57:17.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.637+0000 7f22d1800640 1 --2- 172.21.15.28:0/3859178 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f22b00a3da0 0x7f22b00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:17.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.637+0000 7f22d7172740 1 -- 172.21.15.28:0/384183449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9a5791b40 msgr2=0x55a9a5791f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:17.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.637+0000 7f22d7172740 1 --2- 172.21.15.28:0/384183449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9a5791b40 0x55a9a5791f20 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f22c4009c20 tx=0x7f22c4011810 comp rx=0 tx=0).stop 2024-04-23T14:57:17.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.637+0000 7f22d7172740 1 -- 172.21.15.28:0/384183449 shutdown_connections 2024-04-23T14:57:17.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.637+0000 7f22d7172740 1 --2- 172.21.15.28:0/384183449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9a579cb30 0x55a9a579ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.637+0000 7f22d7172740 1 --2- 172.21.15.28:0/384183449 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a5792460 0x55a9a579c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.637+0000 7f22d7172740 1 --2- 172.21.15.28:0/384183449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9a5791b40 0x55a9a5791f20 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.637+0000 7f22d7172740 1 -- 172.21.15.28:0/384183449 >> 172.21.15.28:0/384183449 conn(0x55a9a578de60 msgr2=0x55a9a578e240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:17.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.638+0000 7f22d7172740 1 -- 172.21.15.28:0/384183449 shutdown_connections 2024-04-23T14:57:17.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.638+0000 7f22d7172740 1 -- 172.21.15.28:0/384183449 wait complete. 2024-04-23T14:57:17.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.638+0000 7f22d1800640 1 --2- 172.21.15.28:0/3859178 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f22b00a3da0 0x7f22b00a6260 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x55a9a5793100 tx=0x7f22bc0046c0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:17.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.638+0000 7f22d7172740 1 -- start start 2024-04-23T14:57:17.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.639+0000 7f22d7172740 1 Processor -- start 2024-04-23T14:57:17.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.639+0000 7f22d7172740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a5791b40 0x55a9a579c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:17.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.639+0000 7f22d7172740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9a5792460 0x55a9a57998c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:17.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.639+0000 7f22d7172740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9a579cb30 0x55a9a5799fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:17.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.639+0000 7f22d7172740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a9a579f9c0 con 0x55a9a5792460 2024-04-23T14:57:17.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.639+0000 7f22d7172740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a9a56b0810 con 0x55a9a579cb30 2024-04-23T14:57:17.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.639+0000 7f22d7172740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a9a579fb40 con 0x55a9a5791b40 2024-04-23T14:57:17.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.639+0000 7f22d7172740 10 client.0 _ll_register_callbacks cb 0x55a9a567e140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:57:17.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.639+0000 7f22d2200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9a579cb30 0x55a9a5799fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:17.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.639+0000 7f22d0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9a5792460 0x55a9a57998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:17.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.639+0000 7f22d2200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9a579cb30 0x55a9a5799fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:52080/0 (socket says 172.21.15.28:52080) 2024-04-23T14:57:17.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.639+0000 7f22d2200640 1 -- 172.21.15.28:0/3688677980 learned_addr learned my addr 172.21.15.28:0/3688677980 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:17.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.639+0000 7f22d1800640 1 --2- 172.21.15.28:0/3688677980 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a5791b40 0x55a9a579c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:17.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.640+0000 7f22d0e00640 1 -- 172.21.15.28:0/3688677980 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a5791b40 msgr2=0x55a9a579c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:17.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.640+0000 7f22d0e00640 1 --2- 172.21.15.28:0/3688677980 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a5791b40 0x55a9a579c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.640+0000 7f22d0e00640 1 -- 172.21.15.28:0/3688677980 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9a579cb30 msgr2=0x55a9a5799fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:17.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.640+0000 7f22d0e00640 1 --2- 172.21.15.28:0/3688677980 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9a579cb30 0x55a9a5799fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.640+0000 7f22d0e00640 1 -- 172.21.15.28:0/3688677980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9a56c13f0 con 0x55a9a5792460 2024-04-23T14:57:17.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.640+0000 7f22d2200640 1 --2- 172.21.15.28:0/3688677980 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9a579cb30 0x55a9a5799fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:57:17.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.640+0000 7f22d0e00640 1 --2- 172.21.15.28:0/3688677980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9a5792460 0x55a9a57998c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f22b80ce830 tx=0x7f22b8099320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34923880852ecfde server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:17.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.640+0000 7f22b7400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-23T14:57:17.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.640+0000 7f22b7400640 1 -- 172.21.15.28:0/3688677980 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b809c9e0 con 0x55a9a5792460 2024-04-23T14:57:17.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.640+0000 7f22b7400640 1 -- 172.21.15.28:0/3688677980 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8542+0+0 (secure 0 0 0) 0x7f22b80996f0 con 0x55a9a5792460 2024-04-23T14:57:17.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.640+0000 7f22d7172740 1 -- 172.21.15.28:0/3688677980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22c4019b10 con 0x55a9a5792460 2024-04-23T14:57:17.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.641+0000 7f22b7400640 1 -- 172.21.15.28:0/3688677980 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f22b809d780 con 0x55a9a5792460 2024-04-23T14:57:17.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.641+0000 7f22d7172740 10 client.17091 fetch_fsmap learned FSMap version 20 2024-04-23T14:57:17.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.641+0000 7f22d7172740 1 -- 172.21.15.28:0/3688677980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55a9a54093f0 con 0x55a9a5792460 2024-04-23T14:57:17.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.641+0000 7f22b7400640 1 -- 172.21.15.28:0/3688677980 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22c4019b10 con 0x55a9a5792460 2024-04-23T14:57:17.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.641+0000 7f22b7400640 1 -- 172.21.15.28:0/3688677980 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f22b80ceab0 con 0x55a9a5792460 2024-04-23T14:57:17.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.641+0000 7f22d7172740 10 client.17091 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:57:17.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.641+0000 7f22d7172740 10 client.17091 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:57:17.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.641+0000 7f22d7172740 20 client.17091 populate_metadata read hostname 'smithi028' 2024-04-23T14:57:17.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.641+0000 7f22d7172740 1 --2- 172.21.15.28:0/3688677980 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55a9a57ace50 0x55a9a57af260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:17.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.641+0000 7f22d7172740 4 client.17091 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:57:17.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.641+0000 7f22d7172740 1 -- 172.21.15.28:0/3688677980 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a9a5704bf0 con 0x55a9a57ace50 2024-04-23T14:57:17.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.642+0000 7f22d1800640 1 --2- 172.21.15.28:0/3688677980 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55a9a57ace50 0x55a9a57af260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:17.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.642+0000 7f22d1800640 1 --2- 172.21.15.28:0/3688677980 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55a9a57ace50 0x55a9a57af260 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=467dcc2e3fd9de33 server_cookie=c1382d86998c5310 in_seq=0 out_seq=0 2024-04-23T14:57:17.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.642+0000 7f22b7400640 10 client.17091 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:57:17.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.643+0000 7f22b7400640 1 -- 172.21.15.28:0/3688677980 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a9a5704bf0 con 0x55a9a57ace50 2024-04-23T14:57:17.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.643+0000 7f22b7400640 10 client.17091 handle_command_reply: tid=0 2024-04-23T14:57:17.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.643+0000 7f22d7172740 1 client.17091 shutdown 2024-04-23T14:57:17.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.644+0000 7f22d7172740 1 -- 172.21.15.28:0/3688677980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9a5792460 msgr2=0x55a9a57998c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:17.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.644+0000 7f22d7172740 1 --2- 172.21.15.28:0/3688677980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9a5792460 0x55a9a57998c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f22b80ce830 tx=0x7f22b8099320 comp rx=0 tx=0).stop 2024-04-23T14:57:17.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.644+0000 7f22d7172740 1 -- 172.21.15.28:0/3688677980 shutdown_connections 2024-04-23T14:57:17.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.644+0000 7f22d7172740 1 --2- 172.21.15.28:0/3688677980 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55a9a57ace50 0x55a9a57af260 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.644+0000 7f22d7172740 1 --2- 172.21.15.28:0/3688677980 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9a579cb30 0x55a9a5799fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.644+0000 7f22d7172740 1 --2- 172.21.15.28:0/3688677980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9a5792460 0x55a9a57998c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.644+0000 7f22d7172740 1 --2- 172.21.15.28:0/3688677980 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a5791b40 0x55a9a579c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.644+0000 7f22b7400640 0 client.17091 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:57:17.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.644+0000 7f22d7172740 1 -- 172.21.15.28:0/3688677980 >> 172.21.15.28:0/3688677980 conn(0x55a9a578de60 msgr2=0x55a9a5793b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:17.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.644+0000 7f22d7172740 1 -- 172.21.15.28:0/3688677980 shutdown_connections 2024-04-23T14:57:17.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.644+0000 7f22d7172740 1 -- 172.21.15.28:0/3688677980 wait complete. 2024-04-23T14:57:17.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.644+0000 7f22d7172740 20 client.17091 trim_cache size 0 max 16384 2024-04-23T14:57:17.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.662+0000 7f22d7172740 1 Processor -- start 2024-04-23T14:57:17.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.662+0000 7f22d7172740 1 -- start start 2024-04-23T14:57:17.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.662+0000 7f22d7172740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a5791b40 0x55a9a57a88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:17.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.662+0000 7f22d7172740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9a5792460 0x55a9a57a9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:17.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.662+0000 7f22d7172740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9a579cb30 0x55a9a578e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:17.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.662+0000 7f22d7172740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f22cc1ade30 con 0x55a9a5792460 2024-04-23T14:57:17.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.662+0000 7f22d7172740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f22cc1b0690 con 0x55a9a579cb30 2024-04-23T14:57:17.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.662+0000 7f22d7172740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f22cc1ae480 con 0x55a9a5791b40 2024-04-23T14:57:17.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.662+0000 7f22d0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9a5792460 0x55a9a57a9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:17.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.662+0000 7f22d2200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9a579cb30 0x55a9a578e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:17.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.662+0000 7f22d2200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9a579cb30 0x55a9a578e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:52092/0 (socket says 172.21.15.28:52092) 2024-04-23T14:57:17.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.662+0000 7f22d0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9a5792460 0x55a9a57a9220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42256/0 (socket says 172.21.15.28:42256) 2024-04-23T14:57:17.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.662+0000 7f22d2200640 1 -- 172.21.15.28:0/1393629843 learned_addr learned my addr 172.21.15.28:0/1393629843 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:17.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.663+0000 7f22d1800640 1 --2- 172.21.15.28:0/1393629843 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a5791b40 0x55a9a57a88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:17.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.663+0000 7f22d2200640 1 -- 172.21.15.28:0/1393629843 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a5791b40 msgr2=0x55a9a57a88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:17.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.663+0000 7f22d2200640 1 --2- 172.21.15.28:0/1393629843 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a5791b40 0x55a9a57a88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.663+0000 7f22d2200640 1 -- 172.21.15.28:0/1393629843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9a5792460 msgr2=0x55a9a57a9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:17.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.663+0000 7f22d2200640 1 --2- 172.21.15.28:0/1393629843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9a5792460 0x55a9a57a9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.663+0000 7f22d2200640 1 -- 172.21.15.28:0/1393629843 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9a579a320 con 0x55a9a579cb30 2024-04-23T14:57:17.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.663+0000 7f22d1800640 1 --2- 172.21.15.28:0/1393629843 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a5791b40 0x55a9a57a88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:57:17.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.663+0000 7f22d0e00640 1 --2- 172.21.15.28:0/1393629843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9a5792460 0x55a9a57a9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:57:17.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.663+0000 7f22d2200640 1 --2- 172.21.15.28:0/1393629843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9a579cb30 0x55a9a578e240 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f22c4009870 tx=0x7f22c400e3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6128c4f9f2c056e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:17.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.664+0000 7f22c2000640 1 -- 172.21.15.28:0/1393629843 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c4027020 con 0x55a9a579cb30 2024-04-23T14:57:17.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.664+0000 7f22c2000640 1 -- 172.21.15.28:0/1393629843 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22c40098a0 con 0x55a9a579cb30 2024-04-23T14:57:17.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.664+0000 7f22c2000640 1 -- 172.21.15.28:0/1393629843 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c401e040 con 0x55a9a579cb30 2024-04-23T14:57:17.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.664+0000 7f22d7172740 1 -- 172.21.15.28:0/1393629843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9a579cb30 msgr2=0x55a9a578e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:17.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.664+0000 7f22d7172740 1 --2- 172.21.15.28:0/1393629843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9a579cb30 0x55a9a578e240 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f22c4009870 tx=0x7f22c400e3b0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.665+0000 7f22d7172740 1 -- 172.21.15.28:0/1393629843 shutdown_connections 2024-04-23T14:57:17.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.665+0000 7f22d7172740 1 --2- 172.21.15.28:0/1393629843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9a579cb30 0x55a9a578e240 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.665+0000 7f22d7172740 1 --2- 172.21.15.28:0/1393629843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9a5792460 0x55a9a57a9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.665+0000 7f22d7172740 1 --2- 172.21.15.28:0/1393629843 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a5791b40 0x55a9a57a88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.665+0000 7f22d7172740 1 -- 172.21.15.28:0/1393629843 >> 172.21.15.28:0/1393629843 conn(0x55a9a578de60 msgr2=0x55a9a5799150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.665+0000 7f22d7172740 1 -- 172.21.15.28:0/1393629843 shutdown_connections 2024-04-23T14:57:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.665+0000 7f22d7172740 1 -- 172.21.15.28:0/1393629843 wait complete. 2024-04-23T14:57:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.665+0000 7f22d7172740 1 -- start start 2024-04-23T14:57:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.665+0000 7f22d7172740 1 Processor -- start 2024-04-23T14:57:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.666+0000 7f22d7172740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a5791b40 0x55a9a57a5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.666+0000 7f22d7172740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9a5792460 0x55a9a579d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.666+0000 7f22d7172740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9a579cb30 0x55a9a579dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.666+0000 7f22d7172740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f22cc1aeb50 con 0x55a9a579cb30 2024-04-23T14:57:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.666+0000 7f22d7172740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f22cc1affc0 con 0x55a9a5792460 2024-04-23T14:57:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.666+0000 7f22d7172740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f22cc1af220 con 0x55a9a5791b40 2024-04-23T14:57:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.666+0000 7f22d7172740 10 client.0 _ll_register_callbacks cb 0x7f22b80d4750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:57:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.666+0000 7f22d0e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9a5792460 0x55a9a579d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.666+0000 7f22d2200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9a579cb30 0x55a9a579dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.666+0000 7f22d0e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9a5792460 0x55a9a579d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:52108/0 (socket says 172.21.15.28:52108) 2024-04-23T14:57:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.666+0000 7f22d0e00640 1 -- 172.21.15.28:0/2162508821 learned_addr learned my addr 172.21.15.28:0/2162508821 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.666+0000 7f22d1800640 1 --2- 172.21.15.28:0/2162508821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a5791b40 0x55a9a57a5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.667+0000 7f22d0e00640 1 -- 172.21.15.28:0/2162508821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a5791b40 msgr2=0x55a9a57a5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.667+0000 7f22d0e00640 1 --2- 172.21.15.28:0/2162508821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a5791b40 0x55a9a57a5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.667+0000 7f22d0e00640 1 -- 172.21.15.28:0/2162508821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9a579cb30 msgr2=0x55a9a579dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.667+0000 7f22d0e00640 1 --2- 172.21.15.28:0/2162508821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9a579cb30 0x55a9a579dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.667+0000 7f22d0e00640 1 -- 172.21.15.28:0/2162508821 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9a54d37d0 con 0x55a9a5792460 2024-04-23T14:57:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.667+0000 7f22d0e00640 1 --2- 172.21.15.28:0/2162508821 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9a5792460 0x55a9a579d7b0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f22b8017470 tx=0x7f22b80cece0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c128b24d290447e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.667+0000 7f22b7400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:57:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.667+0000 7f22b7400640 1 -- 172.21.15.28:0/2162508821 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b8005e30 con 0x55a9a5792460 2024-04-23T14:57:17.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.667+0000 7f22b7400640 1 -- 172.21.15.28:0/2162508821 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8542+0+0 (secure 0 0 0) 0x7f22b80b6370 con 0x55a9a5792460 2024-04-23T14:57:17.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.667+0000 7f22d7172740 1 -- 172.21.15.28:0/2162508821 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a9a56fea60 con 0x55a9a5792460 2024-04-23T14:57:17.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.667+0000 7f22b7400640 1 -- 172.21.15.28:0/2162508821 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f22b809d080 con 0x55a9a5792460 2024-04-23T14:57:17.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.667+0000 7f22d7172740 10 client.27293 fetch_fsmap learned FSMap version 20 2024-04-23T14:57:17.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.667+0000 7f22d7172740 1 -- 172.21.15.28:0/2162508821 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55a9a57a4aa0 con 0x55a9a5792460 2024-04-23T14:57:17.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.668+0000 7f22b7400640 1 -- 172.21.15.28:0/2162508821 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55a9a56fea60 con 0x55a9a5792460 2024-04-23T14:57:17.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.668+0000 7f22b7400640 1 -- 172.21.15.28:0/2162508821 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f22b80b1b20 con 0x55a9a5792460 2024-04-23T14:57:17.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.668+0000 7f22d7172740 10 client.27293 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:57:17.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.668+0000 7f22d7172740 10 client.27293 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:57:17.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.668+0000 7f22d7172740 20 client.27293 populate_metadata read hostname 'smithi028' 2024-04-23T14:57:17.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.668+0000 7f22d7172740 1 --2- 172.21.15.28:0/2162508821 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55a9a58137c0 0x55a9a5815be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:17.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.668+0000 7f22d7172740 4 client.27293 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-23T14:57:17.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.668+0000 7f22d7172740 1 -- 172.21.15.28:0/2162508821 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a9a567a4d0 con 0x55a9a58137c0 2024-04-23T14:57:17.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.668+0000 7f22d1800640 1 --2- 172.21.15.28:0/2162508821 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55a9a58137c0 0x55a9a5815be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:17.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.668+0000 7f22d1800640 1 --2- 172.21.15.28:0/2162508821 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55a9a58137c0 0x55a9a5815be0 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2316565192a14f59 server_cookie=37f03aa419739c15 in_seq=0 out_seq=0 2024-04-23T14:57:17.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.669+0000 7f22b7400640 10 client.27293 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:57:17.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.669+0000 7f22b7400640 1 -- 172.21.15.28:0/2162508821 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a9a567a4d0 con 0x55a9a58137c0 2024-04-23T14:57:17.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.670+0000 7f22b7400640 10 client.27293 handle_command_reply: tid=0 2024-04-23T14:57:17.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.670+0000 7f22d7172740 1 client.27293 shutdown 2024-04-23T14:57:17.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.670+0000 7f22d7172740 1 -- 172.21.15.28:0/2162508821 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9a5792460 msgr2=0x55a9a579d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:17.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.670+0000 7f22d7172740 1 --2- 172.21.15.28:0/2162508821 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9a5792460 0x55a9a579d7b0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f22b8017470 tx=0x7f22b80cece0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.670+0000 7f22d7172740 1 -- 172.21.15.28:0/2162508821 shutdown_connections 2024-04-23T14:57:17.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.670+0000 7f22d7172740 1 --2- 172.21.15.28:0/2162508821 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55a9a58137c0 0x55a9a5815be0 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.670+0000 7f22d7172740 1 --2- 172.21.15.28:0/2162508821 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a9a5792460 0x55a9a579d7b0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.670+0000 7f22b7400640 0 client.27293 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:57:17.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.670+0000 7f22d7172740 1 --2- 172.21.15.28:0/2162508821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9a579cb30 0x55a9a579dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.670+0000 7f22d7172740 1 --2- 172.21.15.28:0/2162508821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a9a5791b40 0x55a9a57a5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.670+0000 7f22d7172740 1 -- 172.21.15.28:0/2162508821 >> 172.21.15.28:0/2162508821 conn(0x55a9a578de60 msgr2=0x55a9a578e240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:17.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.670+0000 7f22d7172740 1 -- 172.21.15.28:0/2162508821 shutdown_connections 2024-04-23T14:57:17.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.670+0000 7f22d7172740 1 -- 172.21.15.28:0/2162508821 wait complete. 2024-04-23T14:57:17.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.670+0000 7f22d7172740 20 client.27293 trim_cache size 0 max 16384 2024-04-23T14:57:17.676 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-23T14:57:17.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.671+0000 7f22d2c00640 1 -- 172.21.15.28:0/3859178 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f22b00a3da0 msgr2=0x7f22b00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:17.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.671+0000 7f22d2c00640 1 --2- 172.21.15.28:0/3859178 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f22b00a3da0 0x7f22b00a6260 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x55a9a5793100 tx=0x7f22bc0046c0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.671+0000 7f22d2c00640 1 -- 172.21.15.28:0/3859178 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22cc074cd0 msgr2=0x7f22cc1a0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:17.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.671+0000 7f22d2c00640 1 --2- 172.21.15.28:0/3859178 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22cc074cd0 0x7f22cc1a0b70 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f22b800eb00 tx=0x7f22b800c2d0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.671+0000 7f22d2c00640 1 -- 172.21.15.28:0/3859178 shutdown_connections 2024-04-23T14:57:17.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.671+0000 7f22d2c00640 1 --2- 172.21.15.28:0/3859178 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f22b00a3da0 0x7f22b00a6260 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.671+0000 7f22d2c00640 1 --2- 172.21.15.28:0/3859178 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22cc074cd0 0x7f22cc1a0b70 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.671+0000 7f22d2c00640 1 --2- 172.21.15.28:0/3859178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22cc10e760 0x7f22cc19a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.671+0000 7f22d2c00640 1 --2- 172.21.15.28:0/3859178 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f22cc0743b0 0x7f22cc1a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:17.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.671+0000 7f22d2c00640 1 -- 172.21.15.28:0/3859178 >> 172.21.15.28:0/3859178 conn(0x7f22cc0fd9d0 msgr2=0x7f22cc0fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:17.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.672+0000 7f22d2c00640 1 -- 172.21.15.28:0/3859178 shutdown_connections 2024-04-23T14:57:17.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:17.672+0000 7f22d2c00640 1 -- 172.21.15.28:0/3859178 wait complete. 2024-04-23T14:57:18.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:17 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1141967562' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:57:18.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:17 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1141967562' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:57:18.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:17 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1141967562' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:57:18.549 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1125766 8.43 MB/sec execute 574 sec latency 20.711 ms 2024-04-23T14:57:18.695 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-23T14:57:18.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:57:18.695 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-23T14:57:18.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.927+0000 7f6427e00640 1 -- 172.21.15.28:0/678726269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6428102ad0 msgr2=0x7f6428106b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:18.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.927+0000 7f6427e00640 1 --2- 172.21.15.28:0/678726269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6428102ad0 0x7f6428106b10 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f6410009ca0 tx=0x7f641002a8c0 comp rx=0 tx=0).stop 2024-04-23T14:57:18.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.927+0000 7f6427e00640 1 -- 172.21.15.28:0/678726269 shutdown_connections 2024-04-23T14:57:18.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.927+0000 7f6427e00640 1 --2- 172.21.15.28:0/678726269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6428102ad0 0x7f6428106b10 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:18.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.927+0000 7f6427e00640 1 --2- 172.21.15.28:0/678726269 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64281071f0 0x7f642810edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:18.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.927+0000 7f6427e00640 1 --2- 172.21.15.28:0/678726269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6428102120 0x7f6428102500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:18.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.927+0000 7f6427e00640 1 -- 172.21.15.28:0/678726269 >> 172.21.15.28:0/678726269 conn(0x7f64280fd9a0 msgr2=0x7f64280ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:18.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.927+0000 7f6427e00640 1 -- 172.21.15.28:0/678726269 shutdown_connections 2024-04-23T14:57:18.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.927+0000 7f6427e00640 1 -- 172.21.15.28:0/678726269 wait complete. 2024-04-23T14:57:18.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.928+0000 7f6427e00640 1 Processor -- start 2024-04-23T14:57:18.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.929+0000 7f6427e00640 1 -- start start 2024-04-23T14:57:18.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.929+0000 7f6427e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6428102120 0x7f642819e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:18.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.929+0000 7f6427e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64281071f0 0x7f642819eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:18.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.929+0000 7f6427e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6428198630 0x7f6428198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:18.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.929+0000 7f6427e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f64281122d0 con 0x7f64281071f0 2024-04-23T14:57:18.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.929+0000 7f6427e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6428112150 con 0x7f6428198630 2024-04-23T14:57:18.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.929+0000 7f6427e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6428112450 con 0x7f6428102120 2024-04-23T14:57:18.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.929+0000 7f6426a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6428102120 0x7f642819e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:18.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.930+0000 7f6427400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6428198630 0x7f6428198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:18.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.930+0000 7f6426a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6428102120 0x7f642819e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:55562/0 (socket says 172.21.15.28:55562) 2024-04-23T14:57:18.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.930+0000 7f6426a00640 1 -- 172.21.15.28:0/4223083414 learned_addr learned my addr 172.21.15.28:0/4223083414 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:18.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.930+0000 7f641e000640 1 --2- 172.21.15.28:0/4223083414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64281071f0 0x7f642819eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:18.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.930+0000 7f6426a00640 1 -- 172.21.15.28:0/4223083414 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6428198630 msgr2=0x7f6428198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:18.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.930+0000 7f6426a00640 1 --2- 172.21.15.28:0/4223083414 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6428198630 0x7f6428198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:18.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.930+0000 7f6426a00640 1 -- 172.21.15.28:0/4223083414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64281071f0 msgr2=0x7f642819eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:18.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.930+0000 7f6426a00640 1 --2- 172.21.15.28:0/4223083414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64281071f0 0x7f642819eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:18.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.930+0000 7f6426a00640 1 -- 172.21.15.28:0/4223083414 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6428199230 con 0x7f6428102120 2024-04-23T14:57:18.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.930+0000 7f6426a00640 1 --2- 172.21.15.28:0/4223083414 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6428102120 0x7f642819e570 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f641800e9f0 tx=0x7f641800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:18.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.930+0000 7f641fe00640 1 -- 172.21.15.28:0/4223083414 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f641800cd80 con 0x7f6428102120 2024-04-23T14:57:18.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.931+0000 7f6427e00640 1 -- 172.21.15.28:0/4223083414 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64281a5310 con 0x7f6428102120 2024-04-23T14:57:18.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.931+0000 7f641fe00640 1 -- 172.21.15.28:0/4223083414 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6418010910 con 0x7f6428102120 2024-04-23T14:57:18.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.931+0000 7f641fe00640 1 -- 172.21.15.28:0/4223083414 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64180188f0 con 0x7f6428102120 2024-04-23T14:57:18.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.931+0000 7f6427e00640 1 -- 172.21.15.28:0/4223083414 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64281a5710 con 0x7f6428102120 2024-04-23T14:57:18.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.934+0000 7f641fe00640 1 -- 172.21.15.28:0/4223083414 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f6418010ac0 con 0x7f6428102120 2024-04-23T14:57:18.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.934+0000 7f6430623740 1 Processor -- start 2024-04-23T14:57:18.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.934+0000 7f6430623740 1 -- start start 2024-04-23T14:57:18.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.935+0000 7f6430623740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f1ee0e680 0x555f1ee0ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:18.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.935+0000 7f6430623740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f1ee0efa0 0x555f1ee19130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:18.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.935+0000 7f6430623740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f1ee19670 0x555f1ee1ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:18.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.935+0000 7f6430623740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555f1ed6fe70 con 0x555f1ee0efa0 2024-04-23T14:57:18.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.935+0000 7f6430623740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555f1ed16380 con 0x555f1ee19670 2024-04-23T14:57:18.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.935+0000 7f6430623740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x555f1ed6be30 con 0x555f1ee0e680 2024-04-23T14:57:18.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.935+0000 7f6427400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f1ee0e680 0x555f1ee0ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:18.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.935+0000 7f6427400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f1ee0e680 0x555f1ee0ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:55572/0 (socket says 172.21.15.28:55572) 2024-04-23T14:57:18.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.935+0000 7f6427400640 1 -- 172.21.15.28:0/2030997736 learned_addr learned my addr 172.21.15.28:0/2030997736 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:18.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.935+0000 7f641e000640 1 --2- 172.21.15.28:0/2030997736 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f1ee19670 0x555f1ee1ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:18.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.935+0000 7f6426a00640 1 --2- 172.21.15.28:0/2030997736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f1ee0efa0 0x555f1ee19130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:18.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.936+0000 7f6427400640 1 -- 172.21.15.28:0/2030997736 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f1ee19670 msgr2=0x555f1ee1ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:18.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.936+0000 7f6427400640 1 --2- 172.21.15.28:0/2030997736 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f1ee19670 0x555f1ee1ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:18.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.936+0000 7f6427400640 1 -- 172.21.15.28:0/2030997736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f1ee0efa0 msgr2=0x555f1ee19130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:18.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.936+0000 7f6427400640 1 --2- 172.21.15.28:0/2030997736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f1ee0efa0 0x555f1ee19130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:18.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.936+0000 7f6427400640 1 -- 172.21.15.28:0/2030997736 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f1eb4a7e0 con 0x555f1ee0e680 2024-04-23T14:57:18.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.936+0000 7f641fe00640 1 --2- 172.21.15.28:0/4223083414 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f640c0a3d20 0x7f640c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:18.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.936+0000 7f641fe00640 1 -- 172.21.15.28:0/4223083414 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8542+0+0 (secure 0 0 0) 0x7f64180d0cb0 con 0x7f6428102120 2024-04-23T14:57:18.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.936+0000 7f6427400640 1 --2- 172.21.15.28:0/2030997736 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f1ee0e680 0x555f1ee0ea60 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6420009c20 tx=0x7f6420011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=177cf6d1bc6d2873 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:18.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.936+0000 7f6426a00640 1 --2- 172.21.15.28:0/4223083414 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f640c0a3d20 0x7f640c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:18.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.936+0000 7f6417e00640 1 -- 172.21.15.28:0/2030997736 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f642001e070 con 0x555f1ee0e680 2024-04-23T14:57:18.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.936+0000 7f6417e00640 1 -- 172.21.15.28:0/2030997736 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6420019b10 con 0x555f1ee0e680 2024-04-23T14:57:18.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.936+0000 7f6417e00640 1 -- 172.21.15.28:0/2030997736 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f642000ea80 con 0x555f1ee0e680 2024-04-23T14:57:18.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.937+0000 7f6430623740 1 -- 172.21.15.28:0/2030997736 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f1ee0e680 msgr2=0x555f1ee0ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:18.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.937+0000 7f6426a00640 1 --2- 172.21.15.28:0/4223083414 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f640c0a3d20 0x7f640c0a61e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x555f1ee0fc40 tx=0x7f64180d4490 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:18.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.937+0000 7f6430623740 1 --2- 172.21.15.28:0/2030997736 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f1ee0e680 0x555f1ee0ea60 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6420009c20 tx=0x7f6420011810 comp rx=0 tx=0).stop 2024-04-23T14:57:18.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.937+0000 7f6430623740 1 -- 172.21.15.28:0/2030997736 shutdown_connections 2024-04-23T14:57:18.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.937+0000 7f6430623740 1 --2- 172.21.15.28:0/2030997736 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f1ee19670 0x555f1ee1ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:18.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.937+0000 7f6430623740 1 --2- 172.21.15.28:0/2030997736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f1ee0efa0 0x555f1ee19130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:18.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.937+0000 7f6430623740 1 --2- 172.21.15.28:0/2030997736 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f1ee0e680 0x555f1ee0ea60 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:18.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.937+0000 7f6430623740 1 -- 172.21.15.28:0/2030997736 >> 172.21.15.28:0/2030997736 conn(0x555f1ee0a9a0 msgr2=0x555f1ee0ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:18.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.937+0000 7f6430623740 1 -- 172.21.15.28:0/2030997736 shutdown_connections 2024-04-23T14:57:18.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.937+0000 7f6430623740 1 -- 172.21.15.28:0/2030997736 wait complete. 2024-04-23T14:57:18.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.938+0000 7f6430623740 1 -- start start 2024-04-23T14:57:18.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.938+0000 7f6430623740 1 Processor -- start 2024-04-23T14:57:18.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.938+0000 7f6430623740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f1ee0e680 0x555f1ee16630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:18.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.939+0000 7f6430623740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f1ee0efa0 0x555f1ee16b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:18.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.939+0000 7f6430623740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f1ee19670 0x555f1ee170b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:18.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.939+0000 7f6430623740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555f1ee1c460 con 0x555f1ee0e680 2024-04-23T14:57:18.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.939+0000 7f6430623740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555f1ed29660 con 0x555f1ee0efa0 2024-04-23T14:57:18.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.939+0000 7f6430623740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x555f1ee1c5e0 con 0x555f1ee19670 2024-04-23T14:57:18.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.939+0000 7f6430623740 10 client.0 _ll_register_callbacks cb 0x555f1ed17aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:57:18.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.939+0000 7f641e000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f1ee0efa0 0x555f1ee16b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:18.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.939+0000 7f6427400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f1ee0e680 0x555f1ee16630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:18.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.939+0000 7f641e000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f1ee0efa0 0x555f1ee16b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:45092/0 (socket says 172.21.15.28:45092) 2024-04-23T14:57:18.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.939+0000 7f641e000640 1 -- 172.21.15.28:0/3003096508 learned_addr learned my addr 172.21.15.28:0/3003096508 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:18.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.939+0000 7f6427400640 1 --2- 172.21.15.28:0/3003096508 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f1ee19670 0x555f1ee170b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:18.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.939+0000 7f641e000640 1 -- 172.21.15.28:0/3003096508 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f1ee19670 msgr2=0x555f1ee170b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:18.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.939+0000 7f641e000640 1 --2- 172.21.15.28:0/3003096508 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f1ee19670 0x555f1ee170b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:18.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.939+0000 7f641e000640 1 -- 172.21.15.28:0/3003096508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f1ee0e680 msgr2=0x555f1ee16630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:18.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.939+0000 7f641e000640 1 --2- 172.21.15.28:0/3003096508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f1ee0e680 0x555f1ee16630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:18.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.939+0000 7f641e000640 1 -- 172.21.15.28:0/3003096508 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f1ebaaf70 con 0x555f1ee0efa0 2024-04-23T14:57:18.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.940+0000 7f641e000640 1 --2- 172.21.15.28:0/3003096508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f1ee0efa0 0x555f1ee16b70 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f642806b5e0 tx=0x7f641002f380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6716e44e03739a89 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:18.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.940+0000 7f6414c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:57:18.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.940+0000 7f6414c00640 1 -- 172.21.15.28:0/3003096508 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6410003c40 con 0x555f1ee0efa0 2024-04-23T14:57:18.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.940+0000 7f6414c00640 1 -- 172.21.15.28:0/3003096508 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8542+0+0 (secure 0 0 0) 0x7f6410003de0 con 0x555f1ee0efa0 2024-04-23T14:57:18.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.940+0000 7f6430623740 1 -- 172.21.15.28:0/3003096508 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6420019b10 con 0x555f1ee0efa0 2024-04-23T14:57:18.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.940+0000 7f6414c00640 1 -- 172.21.15.28:0/3003096508 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f6410004b40 con 0x555f1ee0efa0 2024-04-23T14:57:18.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.940+0000 7f6430623740 10 client.27305 fetch_fsmap learned FSMap version 20 2024-04-23T14:57:18.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.940+0000 7f6430623740 1 -- 172.21.15.28:0/3003096508 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x555f1ea42170 con 0x555f1ee0efa0 2024-04-23T14:57:18.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.940+0000 7f6414c00640 1 -- 172.21.15.28:0/3003096508 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6420019b10 con 0x555f1ee0efa0 2024-04-23T14:57:18.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.941+0000 7f6414c00640 1 -- 172.21.15.28:0/3003096508 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f6410004340 con 0x555f1ee0efa0 2024-04-23T14:57:18.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.941+0000 7f6430623740 10 client.27305 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:57:18.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.941+0000 7f6430623740 10 client.27305 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:57:18.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.941+0000 7f6430623740 20 client.27305 populate_metadata read hostname 'smithi028' 2024-04-23T14:57:18.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.941+0000 7f6430623740 1 --2- 172.21.15.28:0/3003096508 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x555f1ee29a20 0x555f1ee2be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:18.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.941+0000 7f6430623740 4 client.27305 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:57:18.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.941+0000 7f6430623740 1 -- 172.21.15.28:0/3003096508 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f1ed7d890 con 0x555f1ee29a20 2024-04-23T14:57:18.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.941+0000 7f6427400640 1 --2- 172.21.15.28:0/3003096508 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x555f1ee29a20 0x555f1ee2be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:18.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.942+0000 7f6427400640 1 --2- 172.21.15.28:0/3003096508 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x555f1ee29a20 0x555f1ee2be70 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83e8c8933fef8657 server_cookie=8d4c141646476cd6 in_seq=0 out_seq=0 2024-04-23T14:57:18.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.942+0000 7f6414c00640 10 client.27305 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:57:18.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.942+0000 7f6414c00640 1 -- 172.21.15.28:0/3003096508 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555f1ed7d890 con 0x555f1ee29a20 2024-04-23T14:57:18.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.942+0000 7f6414c00640 10 client.27305 handle_command_reply: tid=0 2024-04-23T14:57:18.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.942+0000 7f6430623740 1 client.27305 shutdown 2024-04-23T14:57:18.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.943+0000 7f6430623740 1 -- 172.21.15.28:0/3003096508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f1ee0efa0 msgr2=0x555f1ee16b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:18.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.943+0000 7f6430623740 1 --2- 172.21.15.28:0/3003096508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f1ee0efa0 0x555f1ee16b70 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f642806b5e0 tx=0x7f641002f380 comp rx=0 tx=0).stop 2024-04-23T14:57:18.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.943+0000 7f6430623740 1 -- 172.21.15.28:0/3003096508 shutdown_connections 2024-04-23T14:57:18.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.943+0000 7f6430623740 1 --2- 172.21.15.28:0/3003096508 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x555f1ee29a20 0x555f1ee2be70 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:18.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.943+0000 7f6430623740 1 --2- 172.21.15.28:0/3003096508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f1ee0efa0 0x555f1ee16b70 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:18.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.943+0000 7f6430623740 1 --2- 172.21.15.28:0/3003096508 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f1ee19670 0x555f1ee170b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:18.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.943+0000 7f6414c00640 0 client.27305 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:57:18.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.943+0000 7f6430623740 1 --2- 172.21.15.28:0/3003096508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f1ee0e680 0x555f1ee16630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:18.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.943+0000 7f6430623740 1 -- 172.21.15.28:0/3003096508 >> 172.21.15.28:0/3003096508 conn(0x555f1ee0a9a0 msgr2=0x555f1ee107a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:18.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.943+0000 7f6430623740 1 -- 172.21.15.28:0/3003096508 shutdown_connections 2024-04-23T14:57:18.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.943+0000 7f6430623740 1 -- 172.21.15.28:0/3003096508 wait complete. 2024-04-23T14:57:18.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.943+0000 7f6430623740 20 client.27305 trim_cache size 0 max 16384 2024-04-23T14:57:18.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.963+0000 7f6430623740 1 Processor -- start 2024-04-23T14:57:18.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.963+0000 7f6430623740 1 -- start start 2024-04-23T14:57:18.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.963+0000 7f6430623740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f1ee0e680 0x555f1ee10de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:18.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.963+0000 7f6430623740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f1ee0efa0 0x555f1ee16d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:18.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.964+0000 7f6430623740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f1ee19670 0x555f1ee31740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:18.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.964+0000 7f6430623740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f64281a9a80 con 0x555f1ee19670 2024-04-23T14:57:18.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.964+0000 7f6430623740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f64281ac2e0 con 0x555f1ee0efa0 2024-04-23T14:57:18.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.964+0000 7f6430623740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f64281aa0d0 con 0x555f1ee0e680 2024-04-23T14:57:18.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.964+0000 7f641e000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f1ee0efa0 0x555f1ee16d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:18.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.964+0000 7f6427400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f1ee0e680 0x555f1ee10de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:18.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.964+0000 7f641e000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f1ee0efa0 0x555f1ee16d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:45098/0 (socket says 172.21.15.28:45098) 2024-04-23T14:57:18.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.964+0000 7f641e000640 1 -- 172.21.15.28:0/2525525272 learned_addr learned my addr 172.21.15.28:0/2525525272 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:18.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.964+0000 7f6427400640 1 --2- 172.21.15.28:0/2525525272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f1ee19670 0x555f1ee31740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:18.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.964+0000 7f641e000640 1 -- 172.21.15.28:0/2525525272 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f1ee0e680 msgr2=0x555f1ee10de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:18.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.964+0000 7f641e000640 1 --2- 172.21.15.28:0/2525525272 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f1ee0e680 0x555f1ee10de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:18.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.964+0000 7f641e000640 1 -- 172.21.15.28:0/2525525272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f1ee19670 msgr2=0x555f1ee31740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:18.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.964+0000 7f641e000640 1 --2- 172.21.15.28:0/2525525272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f1ee19670 0x555f1ee31740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:18.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.964+0000 7f641e000640 1 -- 172.21.15.28:0/2525525272 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f1ee01e80 con 0x555f1ee0efa0 2024-04-23T14:57:18.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.964+0000 7f6427400640 1 --2- 172.21.15.28:0/2525525272 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f1ee0e680 0x555f1ee10de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:57:18.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.965+0000 7f641e000640 1 --2- 172.21.15.28:0/2525525272 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f1ee0efa0 0x555f1ee16d00 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f6410000f80 tx=0x7f64100021f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c4731bfbb0e2b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:18.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.965+0000 7f6417400640 1 -- 172.21.15.28:0/2525525272 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f641002f9b0 con 0x555f1ee0efa0 2024-04-23T14:57:18.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.965+0000 7f6417400640 1 -- 172.21.15.28:0/2525525272 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6410036040 con 0x555f1ee0efa0 2024-04-23T14:57:18.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.965+0000 7f6417400640 1 -- 172.21.15.28:0/2525525272 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6410003c40 con 0x555f1ee0efa0 2024-04-23T14:57:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.966+0000 7f6430623740 1 -- 172.21.15.28:0/2525525272 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f1ee0efa0 msgr2=0x555f1ee16d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.966+0000 7f6430623740 1 --2- 172.21.15.28:0/2525525272 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f1ee0efa0 0x555f1ee16d00 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f6410000f80 tx=0x7f64100021f0 comp rx=0 tx=0).stop 2024-04-23T14:57:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.966+0000 7f6430623740 1 -- 172.21.15.28:0/2525525272 shutdown_connections 2024-04-23T14:57:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.966+0000 7f6430623740 1 --2- 172.21.15.28:0/2525525272 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f1ee0efa0 0x555f1ee16d00 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.966+0000 7f6430623740 1 --2- 172.21.15.28:0/2525525272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f1ee19670 0x555f1ee31740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.966+0000 7f6430623740 1 --2- 172.21.15.28:0/2525525272 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f1ee0e680 0x555f1ee10de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.966+0000 7f6430623740 1 -- 172.21.15.28:0/2525525272 >> 172.21.15.28:0/2525525272 conn(0x555f1ee0a9a0 msgr2=0x555f1ee165f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.966+0000 7f6430623740 1 -- 172.21.15.28:0/2525525272 shutdown_connections 2024-04-23T14:57:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.966+0000 7f6430623740 1 -- 172.21.15.28:0/2525525272 wait complete. 2024-04-23T14:57:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.966+0000 7f6430623740 1 -- start start 2024-04-23T14:57:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.967+0000 7f6430623740 1 Processor -- start 2024-04-23T14:57:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.967+0000 7f6430623740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f1ee0e680 0x555f1ee21b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.967+0000 7f6430623740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f1ee0efa0 0x555f1ee1e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.967+0000 7f6430623740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f1ee19670 0x555f1ee1ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.967+0000 7f6430623740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f64281aa7a0 con 0x555f1ee0efa0 2024-04-23T14:57:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.967+0000 7f6430623740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f64281abc10 con 0x555f1ee19670 2024-04-23T14:57:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.967+0000 7f6430623740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f64281aae70 con 0x555f1ee0e680 2024-04-23T14:57:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.967+0000 7f6430623740 10 client.0 _ll_register_callbacks cb 0x7f6420005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:57:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.967+0000 7f6427400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f1ee0e680 0x555f1ee21b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.967+0000 7f6427400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f1ee0e680 0x555f1ee21b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:55602/0 (socket says 172.21.15.28:55602) 2024-04-23T14:57:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.967+0000 7f6427400640 1 -- 172.21.15.28:0/1228407190 learned_addr learned my addr 172.21.15.28:0/1228407190 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.967+0000 7f641e000640 1 --2- 172.21.15.28:0/1228407190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f1ee0efa0 0x555f1ee1e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.967+0000 7f6427400640 1 --2- 172.21.15.28:0/1228407190 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f1ee19670 0x555f1ee1ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.968+0000 7f6427400640 1 -- 172.21.15.28:0/1228407190 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f1ee19670 msgr2=0x555f1ee1ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.968+0000 7f6427400640 1 --2- 172.21.15.28:0/1228407190 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f1ee19670 0x555f1ee1ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.968+0000 7f6427400640 1 -- 172.21.15.28:0/1228407190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f1ee0efa0 msgr2=0x555f1ee1e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.968+0000 7f6427400640 1 --2- 172.21.15.28:0/1228407190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f1ee0efa0 0x555f1ee1e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.968+0000 7f6427400640 1 -- 172.21.15.28:0/1228407190 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f1ee1f480 con 0x555f1ee0e680 2024-04-23T14:57:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.968+0000 7f641e000640 1 --2- 172.21.15.28:0/1228407190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f1ee0efa0 0x555f1ee1e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:57:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.968+0000 7f6427400640 1 --2- 172.21.15.28:0/1228407190 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f1ee0e680 0x555f1ee21b20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f64200098a0 tx=0x7f6420011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4653b8d32bc8b4ba server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.968+0000 7f6414c00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:57:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.968+0000 7f6414c00640 1 -- 172.21.15.28:0/1228407190 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f642001e040 con 0x555f1ee0e680 2024-04-23T14:57:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.968+0000 7f6414c00640 1 -- 172.21.15.28:0/1228407190 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8542+0+0 (secure 0 0 0) 0x7f6420005450 con 0x555f1ee0e680 2024-04-23T14:57:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.968+0000 7f6430623740 1 -- 172.21.15.28:0/1228407190 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6410036040 con 0x555f1ee0e680 2024-04-23T14:57:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.969+0000 7f6414c00640 1 -- 172.21.15.28:0/1228407190 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f6420005650 con 0x555f1ee0e680 2024-04-23T14:57:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.969+0000 7f6430623740 10 client.27298 fetch_fsmap learned FSMap version 20 2024-04-23T14:57:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.969+0000 7f6430623740 1 -- 172.21.15.28:0/1228407190 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x555f1ee90f30 con 0x555f1ee0e680 2024-04-23T14:57:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.969+0000 7f6414c00640 1 -- 172.21.15.28:0/1228407190 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6410036040 con 0x555f1ee0e680 2024-04-23T14:57:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.969+0000 7f6414c00640 1 -- 172.21.15.28:0/1228407190 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f642002d530 con 0x555f1ee0e680 2024-04-23T14:57:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.969+0000 7f6430623740 10 client.27298 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:57:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.969+0000 7f6430623740 10 client.27298 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:57:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.969+0000 7f6430623740 20 client.27298 populate_metadata read hostname 'smithi028' 2024-04-23T14:57:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.969+0000 7f6430623740 1 --2- 172.21.15.28:0/1228407190 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x555f1ee91490 0x555f1ee938b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.969+0000 7f6430623740 4 client.27298 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-23T14:57:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.969+0000 7f6430623740 1 -- 172.21.15.28:0/1228407190 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x555f1ed77700 con 0x555f1ee91490 2024-04-23T14:57:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.969+0000 7f641e000640 1 --2- 172.21.15.28:0/1228407190 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x555f1ee91490 0x555f1ee938b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.970+0000 7f641e000640 1 --2- 172.21.15.28:0/1228407190 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x555f1ee91490 0x555f1ee938b0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f0cd01828322917 server_cookie=6f7c4e1955e75a79 in_seq=0 out_seq=0 2024-04-23T14:57:18.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.970+0000 7f6414c00640 10 client.27298 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:57:18.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.973+0000 7f6414c00640 1 -- 172.21.15.28:0/1228407190 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x555f1ed77700 con 0x555f1ee91490 2024-04-23T14:57:18.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.973+0000 7f6414c00640 10 client.27298 handle_command_reply: tid=0 2024-04-23T14:57:18.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.973+0000 7f6430623740 1 client.27298 shutdown 2024-04-23T14:57:18.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.973+0000 7f6430623740 1 -- 172.21.15.28:0/1228407190 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f1ee0e680 msgr2=0x555f1ee21b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:18.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.973+0000 7f6430623740 1 --2- 172.21.15.28:0/1228407190 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f1ee0e680 0x555f1ee21b20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f64200098a0 tx=0x7f6420011810 comp rx=0 tx=0).stop 2024-04-23T14:57:18.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.973+0000 7f6430623740 1 -- 172.21.15.28:0/1228407190 shutdown_connections 2024-04-23T14:57:18.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.973+0000 7f6430623740 1 --2- 172.21.15.28:0/1228407190 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x555f1ee91490 0x555f1ee938b0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:18.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.973+0000 7f6430623740 1 --2- 172.21.15.28:0/1228407190 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555f1ee19670 0x555f1ee1ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:18.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.973+0000 7f6414c00640 0 client.27298 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:57:18.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.973+0000 7f6430623740 1 --2- 172.21.15.28:0/1228407190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f1ee0efa0 0x555f1ee1e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:18.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.973+0000 7f6430623740 1 --2- 172.21.15.28:0/1228407190 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555f1ee0e680 0x555f1ee21b20 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:18.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.973+0000 7f6430623740 1 -- 172.21.15.28:0/1228407190 >> 172.21.15.28:0/1228407190 conn(0x555f1ee0a9a0 msgr2=0x555f1ee16e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:18.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.973+0000 7f6430623740 1 -- 172.21.15.28:0/1228407190 shutdown_connections 2024-04-23T14:57:18.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.973+0000 7f6430623740 1 -- 172.21.15.28:0/1228407190 wait complete. 2024-04-23T14:57:18.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.973+0000 7f6430623740 20 client.27298 trim_cache size 0 max 16384 2024-04-23T14:57:18.976 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:57:18.976 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-23T14:57:18.976 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "0291f42f-b83b-41bf-b6a0-42153561996e", 2024-04-23T14:57:18.976 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-23T14:57:18.976 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:57:18.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.974+0000 7f6427e00640 1 -- 172.21.15.28:0/4223083414 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f640c0a3d20 msgr2=0x7f640c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:18.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.974+0000 7f6427e00640 1 --2- 172.21.15.28:0/4223083414 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f640c0a3d20 0x7f640c0a61e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x555f1ee0fc40 tx=0x7f64180d4490 comp rx=0 tx=0).stop 2024-04-23T14:57:18.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.974+0000 7f6427e00640 1 -- 172.21.15.28:0/4223083414 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6428102120 msgr2=0x7f642819e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:18.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.974+0000 7f6427e00640 1 --2- 172.21.15.28:0/4223083414 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6428102120 0x7f642819e570 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f641800e9f0 tx=0x7f641800eeb0 comp rx=0 tx=0).stop 2024-04-23T14:57:18.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.974+0000 7f6427e00640 1 -- 172.21.15.28:0/4223083414 shutdown_connections 2024-04-23T14:57:18.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.974+0000 7f6427e00640 1 --2- 172.21.15.28:0/4223083414 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f640c0a3d20 0x7f640c0a61e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:18.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.974+0000 7f6427e00640 1 --2- 172.21.15.28:0/4223083414 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6428198630 0x7f6428198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:18.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.974+0000 7f6427e00640 1 --2- 172.21.15.28:0/4223083414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64281071f0 0x7f642819eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:18.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.974+0000 7f6427e00640 1 --2- 172.21.15.28:0/4223083414 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6428102120 0x7f642819e570 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:18.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.974+0000 7f6427e00640 1 -- 172.21.15.28:0/4223083414 >> 172.21.15.28:0/4223083414 conn(0x7f64280fd9a0 msgr2=0x7f64280ffd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:18.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.974+0000 7f6427e00640 1 -- 172.21.15.28:0/4223083414 shutdown_connections 2024-04-23T14:57:18.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:18.974+0000 7f6427e00640 1 -- 172.21.15.28:0/4223083414 wait complete. 2024-04-23T14:57:18.997 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:57:18.997 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:57:19.035 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:18 smithi105 ceph-mon[32555]: pgmap v415: 97 pgs: 97 active+clean; 611 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.8 MiB/s wr, 155 op/s 2024-04-23T14:57:19.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:18 smithi028 ceph-mon[27810]: pgmap v415: 97 pgs: 97 active+clean; 611 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.8 MiB/s wr, 155 op/s 2024-04-23T14:57:19.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:18 smithi038 ceph-mon[32348]: pgmap v415: 97 pgs: 97 active+clean; 611 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.8 MiB/s wr, 155 op/s 2024-04-23T14:57:19.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.206+0000 7fe358400640 1 -- 172.21.15.28:0/107286348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe350102240 msgr2=0x7fe350102620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:19.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.206+0000 7fe358400640 1 --2- 172.21.15.28:0/107286348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe350102240 0x7fe350102620 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fe344009900 tx=0x7fe34402a3f0 comp rx=0 tx=0).stop 2024-04-23T14:57:19.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.206+0000 7fe358400640 1 -- 172.21.15.28:0/107286348 shutdown_connections 2024-04-23T14:57:19.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.206+0000 7fe358400640 1 --2- 172.21.15.28:0/107286348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe350107310 0x7fe35010eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:19.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.206+0000 7fe358400640 1 --2- 172.21.15.28:0/107286348 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe350102bf0 0x7fe350106c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:19.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.206+0000 7fe358400640 1 --2- 172.21.15.28:0/107286348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe350102240 0x7fe350102620 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:19.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.206+0000 7fe358400640 1 -- 172.21.15.28:0/107286348 >> 172.21.15.28:0/107286348 conn(0x7fe350079450 msgr2=0x7fe350079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:19.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.206+0000 7fe358400640 1 -- 172.21.15.28:0/107286348 shutdown_connections 2024-04-23T14:57:19.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.206+0000 7fe358400640 1 -- 172.21.15.28:0/107286348 wait complete. 2024-04-23T14:57:19.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.208+0000 7fe358400640 1 Processor -- start 2024-04-23T14:57:19.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.208+0000 7fe358400640 1 -- start start 2024-04-23T14:57:19.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.208+0000 7fe358400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe350102240 0x7fe3501a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:19.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.208+0000 7fe358400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe350107310 0x7fe3501a0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:19.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.208+0000 7fe358400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe35019a930 0x7fe35019ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:19.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.208+0000 7fe358400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe3501123f0 con 0x7fe350107310 2024-04-23T14:57:19.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.208+0000 7fe358400640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe350112270 con 0x7fe350102240 2024-04-23T14:57:19.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.208+0000 7fe358400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe350112570 con 0x7fe35019a930 2024-04-23T14:57:19.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.209+0000 7fe355a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe350102240 0x7fe3501a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:19.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.209+0000 7fe355a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe350102240 0x7fe3501a0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:45124/0 (socket says 172.21.15.28:45124) 2024-04-23T14:57:19.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.209+0000 7fe355a00640 1 -- 172.21.15.28:0/2081328164 learned_addr learned my addr 172.21.15.28:0/2081328164 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:19.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.209+0000 7fe355000640 1 --2- 172.21.15.28:0/2081328164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe350107310 0x7fe3501a0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:19.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.209+0000 7fe356400640 1 --2- 172.21.15.28:0/2081328164 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe35019a930 0x7fe35019ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:19.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.209+0000 7fe355a00640 1 -- 172.21.15.28:0/2081328164 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe35019a930 msgr2=0x7fe35019ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:19.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.209+0000 7fe355a00640 1 --2- 172.21.15.28:0/2081328164 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe35019a930 0x7fe35019ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:19.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.209+0000 7fe355a00640 1 -- 172.21.15.28:0/2081328164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe350107310 msgr2=0x7fe3501a0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:19.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.209+0000 7fe355a00640 1 --2- 172.21.15.28:0/2081328164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe350107310 0x7fe3501a0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:19.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.209+0000 7fe355a00640 1 -- 172.21.15.28:0/2081328164 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe35019b450 con 0x7fe350102240 2024-04-23T14:57:19.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.209+0000 7fe356400640 1 --2- 172.21.15.28:0/2081328164 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe35019a930 0x7fe35019ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:57:19.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.209+0000 7fe355a00640 1 --2- 172.21.15.28:0/2081328164 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe350102240 0x7fe3501a0780 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fe344004770 tx=0x7fe344003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:19.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.210+0000 7fe34e000640 1 -- 172.21.15.28:0/2081328164 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe34403f070 con 0x7fe350102240 2024-04-23T14:57:19.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.210+0000 7fe358400640 1 -- 172.21.15.28:0/2081328164 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe35019b6e0 con 0x7fe350102240 2024-04-23T14:57:19.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.210+0000 7fe34e000640 1 -- 172.21.15.28:0/2081328164 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe344031ca0 con 0x7fe350102240 2024-04-23T14:57:19.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.210+0000 7fe34e000640 1 -- 172.21.15.28:0/2081328164 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe344035030 con 0x7fe350102240 2024-04-23T14:57:19.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.210+0000 7fe358400640 1 -- 172.21.15.28:0/2081328164 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3501a78d0 con 0x7fe350102240 2024-04-23T14:57:19.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.213+0000 7fe35b223740 1 Processor -- start 2024-04-23T14:57:19.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.213+0000 7fe34e000640 1 -- 172.21.15.28:0/2081328164 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fe344042070 con 0x7fe350102240 2024-04-23T14:57:19.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.214+0000 7fe35b223740 1 -- start start 2024-04-23T14:57:19.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.214+0000 7fe35b223740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565149657860 0x565149657c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:19.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.214+0000 7fe35b223740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565149658180 0x565149662310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:19.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.214+0000 7fe35b223740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565149662850 0x565149664c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:19.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.214+0000 7fe35b223740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5651495bd1f0 con 0x565149658180 2024-04-23T14:57:19.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.214+0000 7fe35b223740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x565149563350 con 0x565149662850 2024-04-23T14:57:19.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.214+0000 7fe35b223740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5651495b91b0 con 0x565149657860 2024-04-23T14:57:19.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.214+0000 7fe355000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565149662850 0x565149664c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:19.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.214+0000 7fe355a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565149658180 0x565149662310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:19.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.214+0000 7fe356400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565149657860 0x565149657c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:19.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.214+0000 7fe355000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565149662850 0x565149664c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:45128/0 (socket says 172.21.15.28:45128) 2024-04-23T14:57:19.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.214+0000 7fe355000640 1 -- 172.21.15.28:0/1149511932 learned_addr learned my addr 172.21.15.28:0/1149511932 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:19.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.215+0000 7fe355000640 1 -- 172.21.15.28:0/1149511932 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565149657860 msgr2=0x565149657c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:19.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.215+0000 7fe355000640 1 --2- 172.21.15.28:0/1149511932 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565149657860 0x565149657c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:19.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.215+0000 7fe355000640 1 -- 172.21.15.28:0/1149511932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565149658180 msgr2=0x565149662310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:19.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.215+0000 7fe355000640 1 --2- 172.21.15.28:0/1149511932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565149658180 0x565149662310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:19.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.215+0000 7fe355000640 1 -- 172.21.15.28:0/1149511932 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651495b7f50 con 0x565149662850 2024-04-23T14:57:19.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.215+0000 7fe355000640 1 --2- 172.21.15.28:0/1149511932 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565149662850 0x565149664c40 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fe33c00bc60 tx=0x7fe33c0107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f38d2acda3c3128 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:19.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.215+0000 7fe342a00640 1 -- 172.21.15.28:0/1149511932 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe33c01c480 con 0x565149662850 2024-04-23T14:57:19.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.215+0000 7fe342a00640 1 -- 172.21.15.28:0/1149511932 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe33c01cb00 con 0x565149662850 2024-04-23T14:57:19.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.215+0000 7fe342a00640 1 -- 172.21.15.28:0/1149511932 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe33c011c70 con 0x565149662850 2024-04-23T14:57:19.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.216+0000 7fe34e000640 1 --2- 172.21.15.28:0/2081328164 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe3380a3d20 0x7fe3380a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:19.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.216+0000 7fe34e000640 1 -- 172.21.15.28:0/2081328164 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8542+0+0 (secure 0 0 0) 0x7fe3440f1020 con 0x7fe350102240 2024-04-23T14:57:19.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.216+0000 7fe356400640 1 --2- 172.21.15.28:0/2081328164 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe3380a3d20 0x7fe3380a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:19.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.216+0000 7fe35b223740 1 -- 172.21.15.28:0/1149511932 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565149662850 msgr2=0x565149664c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:19.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.216+0000 7fe35b223740 1 --2- 172.21.15.28:0/1149511932 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565149662850 0x565149664c40 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fe33c00bc60 tx=0x7fe33c0107a0 comp rx=0 tx=0).stop 2024-04-23T14:57:19.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.216+0000 7fe35b223740 1 -- 172.21.15.28:0/1149511932 shutdown_connections 2024-04-23T14:57:19.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.216+0000 7fe35b223740 1 --2- 172.21.15.28:0/1149511932 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565149662850 0x565149664c40 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:19.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.216+0000 7fe35b223740 1 --2- 172.21.15.28:0/1149511932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565149658180 0x565149662310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:19.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.216+0000 7fe35b223740 1 --2- 172.21.15.28:0/1149511932 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565149657860 0x565149657c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:19.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.216+0000 7fe35b223740 1 -- 172.21.15.28:0/1149511932 >> 172.21.15.28:0/1149511932 conn(0x565149653b80 msgr2=0x565149653f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:19.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.217+0000 7fe35b223740 1 -- 172.21.15.28:0/1149511932 shutdown_connections 2024-04-23T14:57:19.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.217+0000 7fe35b223740 1 -- 172.21.15.28:0/1149511932 wait complete. 2024-04-23T14:57:19.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.217+0000 7fe356400640 1 --2- 172.21.15.28:0/2081328164 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe3380a3d20 0x7fe3380a61e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe348005fd0 tx=0x7fe348009780 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:19.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.217+0000 7fe35b223740 1 -- start start 2024-04-23T14:57:19.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.217+0000 7fe35b223740 1 Processor -- start 2024-04-23T14:57:19.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.218+0000 7fe35b223740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565149657860 0x56514965f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:19.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.218+0000 7fe35b223740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565149658180 0x56514965fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:19.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.218+0000 7fe35b223740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565149662850 0x5651496602c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:19.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.218+0000 7fe35b223740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x565149665640 con 0x565149662850 2024-04-23T14:57:19.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.218+0000 7fe35b223740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x565149576690 con 0x565149657860 2024-04-23T14:57:19.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.218+0000 7fe35b223740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5651496657c0 con 0x565149658180 2024-04-23T14:57:19.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.218+0000 7fe35b223740 10 client.0 _ll_register_callbacks cb 0x5651495b2650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:57:19.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.218+0000 7fe355a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565149657860 0x56514965f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:19.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.218+0000 7fe355a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565149657860 0x56514965f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:45144/0 (socket says 172.21.15.28:45144) 2024-04-23T14:57:19.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.218+0000 7fe355a00640 1 -- 172.21.15.28:0/932515960 learned_addr learned my addr 172.21.15.28:0/932515960 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:19.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.218+0000 7fe356400640 1 --2- 172.21.15.28:0/932515960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565149662850 0x5651496602c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:19.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.218+0000 7fe355000640 1 --2- 172.21.15.28:0/932515960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565149658180 0x56514965fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:19.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.219+0000 7fe355a00640 1 -- 172.21.15.28:0/932515960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565149658180 msgr2=0x56514965fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:19.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.219+0000 7fe355a00640 1 --2- 172.21.15.28:0/932515960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565149658180 0x56514965fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:19.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.219+0000 7fe355a00640 1 -- 172.21.15.28:0/932515960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565149662850 msgr2=0x5651496602c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:19.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.219+0000 7fe355a00640 1 --2- 172.21.15.28:0/932515960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565149662850 0x5651496602c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:19.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.219+0000 7fe355a00640 1 -- 172.21.15.28:0/932515960 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565149575920 con 0x565149657860 2024-04-23T14:57:19.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.219+0000 7fe355000640 1 --2- 172.21.15.28:0/932515960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565149658180 0x56514965fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:57:19.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.219+0000 7fe355a00640 1 --2- 172.21.15.28:0/932515960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565149657860 0x56514965f840 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fe344031550 tx=0x7fe3440f4760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ae612a7220e8f2b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:19.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.219+0000 7fe337400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:57:19.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.219+0000 7fe337400640 1 -- 172.21.15.28:0/932515960 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe344049070 con 0x565149657860 2024-04-23T14:57:19.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.219+0000 7fe337400640 1 -- 172.21.15.28:0/932515960 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8542+0+0 (secure 0 0 0) 0x7fe3440b9620 con 0x565149657860 2024-04-23T14:57:19.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.219+0000 7fe35b223740 1 -- 172.21.15.28:0/932515960 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe33c01cb00 con 0x565149657860 2024-04-23T14:57:19.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.219+0000 7fe337400640 1 -- 172.21.15.28:0/932515960 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fe3440dd040 con 0x565149657860 2024-04-23T14:57:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.219+0000 7fe35b223740 10 client.27323 fetch_fsmap learned FSMap version 20 2024-04-23T14:57:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.219+0000 7fe35b223740 1 -- 172.21.15.28:0/932515960 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5651493997d0 con 0x565149657860 2024-04-23T14:57:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.220+0000 7fe337400640 1 -- 172.21.15.28:0/932515960 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe33c01cb00 con 0x565149657860 2024-04-23T14:57:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.220+0000 7fe337400640 1 -- 172.21.15.28:0/932515960 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fe34403b5f0 con 0x565149657860 2024-04-23T14:57:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.220+0000 7fe35b223740 10 client.27323 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:57:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.220+0000 7fe35b223740 10 client.27323 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:57:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.220+0000 7fe35b223740 20 client.27323 populate_metadata read hostname 'smithi028' 2024-04-23T14:57:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.220+0000 7fe35b223740 1 --2- 172.21.15.28:0/932515960 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x565149672bd0 0x565149675070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.220+0000 7fe35b223740 4 client.27323 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:57:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.220+0000 7fe35b223740 1 -- 172.21.15.28:0/932515960 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5651495cac10 con 0x565149672bd0 2024-04-23T14:57:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.220+0000 7fe355000640 1 --2- 172.21.15.28:0/932515960 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x565149672bd0 0x565149675070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.221+0000 7fe355000640 1 --2- 172.21.15.28:0/932515960 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x565149672bd0 0x565149675070 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ceded416d2f47856 server_cookie=97b7051d8287fa31 in_seq=0 out_seq=0 2024-04-23T14:57:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.221+0000 7fe337400640 10 client.27323 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:57:19.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.223+0000 7fe337400640 1 -- 172.21.15.28:0/932515960 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5651495cac10 con 0x565149672bd0 2024-04-23T14:57:19.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.223+0000 7fe337400640 10 client.27323 handle_command_reply: tid=0 2024-04-23T14:57:19.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.223+0000 7fe35b223740 1 client.27323 shutdown 2024-04-23T14:57:19.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.223+0000 7fe35b223740 1 -- 172.21.15.28:0/932515960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565149657860 msgr2=0x56514965f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:19.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.223+0000 7fe35b223740 1 --2- 172.21.15.28:0/932515960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565149657860 0x56514965f840 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fe344031550 tx=0x7fe3440f4760 comp rx=0 tx=0).stop 2024-04-23T14:57:19.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.223+0000 7fe35b223740 1 -- 172.21.15.28:0/932515960 shutdown_connections 2024-04-23T14:57:19.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.223+0000 7fe35b223740 1 --2- 172.21.15.28:0/932515960 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x565149672bd0 0x565149675070 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:19.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.223+0000 7fe35b223740 1 --2- 172.21.15.28:0/932515960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565149662850 0x5651496602c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:19.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.223+0000 7fe35b223740 1 --2- 172.21.15.28:0/932515960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565149658180 0x56514965fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:19.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.223+0000 7fe337400640 0 client.27323 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:57:19.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.223+0000 7fe35b223740 1 --2- 172.21.15.28:0/932515960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565149657860 0x56514965f840 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:19.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.223+0000 7fe35b223740 1 -- 172.21.15.28:0/932515960 >> 172.21.15.28:0/932515960 conn(0x565149653b80 msgr2=0x565149659980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:19.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.223+0000 7fe35b223740 1 -- 172.21.15.28:0/932515960 shutdown_connections 2024-04-23T14:57:19.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.223+0000 7fe35b223740 1 -- 172.21.15.28:0/932515960 wait complete. 2024-04-23T14:57:19.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.223+0000 7fe35b223740 20 client.27323 trim_cache size 0 max 16384 2024-04-23T14:57:19.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.244+0000 7fe35b223740 1 Processor -- start 2024-04-23T14:57:19.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.244+0000 7fe35b223740 1 -- start start 2024-04-23T14:57:19.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.245+0000 7fe35b223740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565149657860 0x56514965f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:19.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.245+0000 7fe35b223740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565149658180 0x56514965a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:19.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.245+0000 7fe35b223740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565149662850 0x56514967a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:19.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.245+0000 7fe35b223740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe3501adf80 con 0x565149657860 2024-04-23T14:57:19.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.245+0000 7fe35b223740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe3501b07e0 con 0x565149662850 2024-04-23T14:57:19.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.245+0000 7fe35b223740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe3501ae5d0 con 0x565149658180 2024-04-23T14:57:19.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.245+0000 7fe356400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565149662850 0x56514967a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.245+0000 7fe356400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565149662850 0x56514967a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:45148/0 (socket says 172.21.15.28:45148) 2024-04-23T14:57:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.245+0000 7fe356400640 1 -- 172.21.15.28:0/4072917645 learned_addr learned my addr 172.21.15.28:0/4072917645 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.245+0000 7fe355a00640 1 --2- 172.21.15.28:0/4072917645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565149657860 0x56514965f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.245+0000 7fe355000640 1 --2- 172.21.15.28:0/4072917645 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565149658180 0x56514965a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.245+0000 7fe356400640 1 -- 172.21.15.28:0/4072917645 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565149658180 msgr2=0x56514965a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.245+0000 7fe356400640 1 --2- 172.21.15.28:0/4072917645 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565149658180 0x56514965a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.245+0000 7fe356400640 1 -- 172.21.15.28:0/4072917645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565149657860 msgr2=0x56514965f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.245+0000 7fe356400640 1 --2- 172.21.15.28:0/4072917645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565149657860 0x56514965f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.245+0000 7fe356400640 1 -- 172.21.15.28:0/4072917645 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651492cf3f0 con 0x565149662850 2024-04-23T14:57:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.246+0000 7fe356400640 1 --2- 172.21.15.28:0/4072917645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565149662850 0x56514967a980 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fe34801e630 tx=0x7fe348016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=618351cce2d2f6e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:19.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.246+0000 7fe342000640 1 -- 172.21.15.28:0/4072917645 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe34802d070 con 0x565149662850 2024-04-23T14:57:19.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.246+0000 7fe342000640 1 -- 172.21.15.28:0/4072917645 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe34801ece0 con 0x565149662850 2024-04-23T14:57:19.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.246+0000 7fe342000640 1 -- 172.21.15.28:0/4072917645 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3480287f0 con 0x565149662850 2024-04-23T14:57:19.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.247+0000 7fe35b223740 1 -- 172.21.15.28:0/4072917645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565149662850 msgr2=0x56514967a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:19.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.247+0000 7fe35b223740 1 --2- 172.21.15.28:0/4072917645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565149662850 0x56514967a980 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fe34801e630 tx=0x7fe348016000 comp rx=0 tx=0).stop 2024-04-23T14:57:19.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.247+0000 7fe35b223740 1 -- 172.21.15.28:0/4072917645 shutdown_connections 2024-04-23T14:57:19.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.247+0000 7fe35b223740 1 --2- 172.21.15.28:0/4072917645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565149662850 0x56514967a980 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:19.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.247+0000 7fe35b223740 1 --2- 172.21.15.28:0/4072917645 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565149658180 0x56514965a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:19.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.247+0000 7fe35b223740 1 --2- 172.21.15.28:0/4072917645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565149657860 0x56514965f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:19.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.247+0000 7fe35b223740 1 -- 172.21.15.28:0/4072917645 >> 172.21.15.28:0/4072917645 conn(0x565149653b80 msgr2=0x565149652c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:19.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.247+0000 7fe35b223740 1 -- 172.21.15.28:0/4072917645 shutdown_connections 2024-04-23T14:57:19.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.247+0000 7fe35b223740 1 -- 172.21.15.28:0/4072917645 wait complete. 2024-04-23T14:57:19.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.247+0000 7fe35b223740 1 -- start start 2024-04-23T14:57:19.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.247+0000 7fe35b223740 1 Processor -- start 2024-04-23T14:57:19.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.248+0000 7fe35b223740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565149657860 0x565149666420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:19.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.248+0000 7fe35b223740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565149658180 0x565149666960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:19.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.248+0000 7fe35b223740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565149662850 0x565149666ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:19.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.248+0000 7fe35b223740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe3501aeca0 con 0x565149662850 2024-04-23T14:57:19.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.248+0000 7fe35b223740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe3501b0110 con 0x565149658180 2024-04-23T14:57:19.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.248+0000 7fe35b223740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe3501af370 con 0x565149657860 2024-04-23T14:57:19.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.248+0000 7fe35b223740 10 client.0 _ll_register_callbacks cb 0x7fe3440f4860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:57:19.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.248+0000 7fe355000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565149658180 0x565149666960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:19.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.248+0000 7fe356400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565149662850 0x565149666ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:19.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.248+0000 7fe355a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565149657860 0x565149666420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:19.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.248+0000 7fe355000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565149658180 0x565149666960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:45156/0 (socket says 172.21.15.28:45156) 2024-04-23T14:57:19.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.248+0000 7fe355000640 1 -- 172.21.15.28:0/1724127024 learned_addr learned my addr 172.21.15.28:0/1724127024 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:19.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.249+0000 7fe355a00640 1 -- 172.21.15.28:0/1724127024 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565149658180 msgr2=0x565149666960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:19.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.249+0000 7fe355a00640 1 --2- 172.21.15.28:0/1724127024 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565149658180 0x565149666960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:19.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.249+0000 7fe355a00640 1 -- 172.21.15.28:0/1724127024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565149662850 msgr2=0x565149666ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:19.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.249+0000 7fe355a00640 1 --2- 172.21.15.28:0/1724127024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565149662850 0x565149666ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:19.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.249+0000 7fe355a00640 1 -- 172.21.15.28:0/1724127024 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56514928f0f0 con 0x565149657860 2024-04-23T14:57:19.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.249+0000 7fe355000640 1 --2- 172.21.15.28:0/1724127024 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565149658180 0x565149666960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:57:19.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.249+0000 7fe355a00640 1 --2- 172.21.15.28:0/1724127024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565149657860 0x565149666420 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe3440bcf40 tx=0x7fe3440eec10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68262da3a4658ed6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:19.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.249+0000 7fe337400640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:57:19.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.249+0000 7fe337400640 1 -- 172.21.15.28:0/1724127024 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe344049040 con 0x565149657860 2024-04-23T14:57:19.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.249+0000 7fe35b223740 1 -- 172.21.15.28:0/1724127024 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe34801ece0 con 0x565149657860 2024-04-23T14:57:19.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.249+0000 7fe337400640 1 -- 172.21.15.28:0/1724127024 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8542+0+0 (secure 0 0 0) 0x7fe3440c5790 con 0x565149657860 2024-04-23T14:57:19.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.249+0000 7fe337400640 1 -- 172.21.15.28:0/1724127024 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fe3440d5040 con 0x565149657860 2024-04-23T14:57:19.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.250+0000 7fe35b223740 10 client.27316 fetch_fsmap learned FSMap version 20 2024-04-23T14:57:19.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.250+0000 7fe35b223740 1 -- 172.21.15.28:0/1724127024 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x565149582810 con 0x565149657860 2024-04-23T14:57:19.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.250+0000 7fe337400640 1 -- 172.21.15.28:0/1724127024 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe34801ece0 con 0x565149657860 2024-04-23T14:57:19.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.250+0000 7fe337400640 1 -- 172.21.15.28:0/1724127024 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fe3440bc400 con 0x565149657860 2024-04-23T14:57:19.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.250+0000 7fe35b223740 10 client.27316 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:57:19.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.250+0000 7fe35b223740 10 client.27316 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:57:19.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.250+0000 7fe35b223740 20 client.27316 populate_metadata read hostname 'smithi028' 2024-04-23T14:57:19.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.250+0000 7fe35b223740 1 --2- 172.21.15.28:0/1724127024 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5651496d96a0 0x5651496dbac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:19.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.250+0000 7fe35b223740 4 client.27316 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:57:19.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.250+0000 7fe35b223740 1 -- 172.21.15.28:0/1724127024 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56514965b440 con 0x5651496d96a0 2024-04-23T14:57:19.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.250+0000 7fe355000640 1 --2- 172.21.15.28:0/1724127024 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5651496d96a0 0x5651496dbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:19.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.251+0000 7fe355000640 1 --2- 172.21.15.28:0/1724127024 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5651496d96a0 0x5651496dbac0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54a9604ca53e590d server_cookie=6fc05a0abbaf2168 in_seq=0 out_seq=0 2024-04-23T14:57:19.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.251+0000 7fe337400640 10 client.27316 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:57:19.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.251+0000 7fe337400640 1 -- 172.21.15.28:0/1724127024 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56514965b440 con 0x5651496d96a0 2024-04-23T14:57:19.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.251+0000 7fe337400640 10 client.27316 handle_command_reply: tid=0 2024-04-23T14:57:19.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.251+0000 7fe35b223740 1 client.27316 shutdown 2024-04-23T14:57:19.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.252+0000 7fe35b223740 1 -- 172.21.15.28:0/1724127024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565149657860 msgr2=0x565149666420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:19.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.252+0000 7fe35b223740 1 --2- 172.21.15.28:0/1724127024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565149657860 0x565149666420 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe3440bcf40 tx=0x7fe3440eec10 comp rx=0 tx=0).stop 2024-04-23T14:57:19.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.252+0000 7fe35b223740 1 -- 172.21.15.28:0/1724127024 shutdown_connections 2024-04-23T14:57:19.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.252+0000 7fe35b223740 1 --2- 172.21.15.28:0/1724127024 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5651496d96a0 0x5651496dbac0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:19.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.252+0000 7fe35b223740 1 --2- 172.21.15.28:0/1724127024 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565149658180 0x565149666960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:19.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.252+0000 7fe35b223740 1 --2- 172.21.15.28:0/1724127024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565149662850 0x565149666ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:19.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.252+0000 7fe35b223740 1 --2- 172.21.15.28:0/1724127024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565149657860 0x565149666420 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:19.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.252+0000 7fe35b223740 1 -- 172.21.15.28:0/1724127024 >> 172.21.15.28:0/1724127024 conn(0x565149653b80 msgr2=0x565149661070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:19.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.252+0000 7fe337400640 0 client.27316 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:57:19.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.252+0000 7fe35b223740 1 -- 172.21.15.28:0/1724127024 shutdown_connections 2024-04-23T14:57:19.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.252+0000 7fe35b223740 1 -- 172.21.15.28:0/1724127024 wait complete. 2024-04-23T14:57:19.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.252+0000 7fe35b223740 20 client.27316 trim_cache size 0 max 16384 2024-04-23T14:57:19.256 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:57:19.256 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-23T14:57:19.256 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-23T14:57:19.257 INFO:teuthology.orchestra.run.smithi028.stdout: "0291f42f-b83b-41bf-b6a0-42153561996e": { 2024-04-23T14:57:19.257 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-23T14:57:19.257 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "0291f42f-b83b-41bf-b6a0-42153561996e", 2024-04-23T14:57:19.257 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-23T14:57:19.257 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:57:19.257 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:57:19.257 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:57:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.253+0000 7fe358400640 1 -- 172.21.15.28:0/2081328164 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe3380a3d20 msgr2=0x7fe3380a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.253+0000 7fe358400640 1 --2- 172.21.15.28:0/2081328164 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe3380a3d20 0x7fe3380a61e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe348005fd0 tx=0x7fe348009780 comp rx=0 tx=0).stop 2024-04-23T14:57:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.253+0000 7fe358400640 1 -- 172.21.15.28:0/2081328164 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe350102240 msgr2=0x7fe3501a0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.253+0000 7fe358400640 1 --2- 172.21.15.28:0/2081328164 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe350102240 0x7fe3501a0780 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fe344004770 tx=0x7fe344003c10 comp rx=0 tx=0).stop 2024-04-23T14:57:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.253+0000 7fe358400640 1 -- 172.21.15.28:0/2081328164 shutdown_connections 2024-04-23T14:57:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.253+0000 7fe358400640 1 --2- 172.21.15.28:0/2081328164 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe3380a3d20 0x7fe3380a61e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.253+0000 7fe358400640 1 --2- 172.21.15.28:0/2081328164 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe35019a930 0x7fe35019ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.253+0000 7fe358400640 1 --2- 172.21.15.28:0/2081328164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe350107310 0x7fe3501a0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.253+0000 7fe358400640 1 --2- 172.21.15.28:0/2081328164 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe350102240 0x7fe3501a0780 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.253+0000 7fe358400640 1 -- 172.21.15.28:0/2081328164 >> 172.21.15.28:0/2081328164 conn(0x7fe350079450 msgr2=0x7fe3500fffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.253+0000 7fe358400640 1 -- 172.21.15.28:0/2081328164 shutdown_connections 2024-04-23T14:57:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:19.253+0000 7fe358400640 1 -- 172.21.15.28:0/2081328164 wait complete. 2024-04-23T14:57:19.271 INFO:tasks.cephfs.filesystem:scrub status for tag:0291f42f-b83b-41bf-b6a0-42153561996e - {'path': '/', 'tag': '0291f42f-b83b-41bf-b6a0-42153561996e', 'options': 'recursive,force'} 2024-04-23T14:57:19.549 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1127012 8.43 MB/sec execute 575 sec latency 21.308 ms 2024-04-23T14:57:20.549 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1128957 8.43 MB/sec execute 576 sec latency 25.859 ms 2024-04-23T14:57:21.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:20 smithi105 ceph-mon[32555]: scrub queued for path: / 2024-04-23T14:57:21.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:20 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:57:21.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:20 smithi105 ceph-mon[32555]: scrub summary: active paths [/] 2024-04-23T14:57:21.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:20 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:57:21.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:20 smithi105 ceph-mon[32555]: pgmap v416: 97 pgs: 97 active+clean; 611 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.8 MiB/s wr, 155 op/s 2024-04-23T14:57:21.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:20 smithi105 ceph-mon[32555]: scrub summary: idle 2024-04-23T14:57:21.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:20 smithi038 ceph-mon[32348]: scrub queued for path: / 2024-04-23T14:57:21.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:20 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:57:21.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:20 smithi038 ceph-mon[32348]: scrub summary: active paths [/] 2024-04-23T14:57:21.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:20 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:57:21.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:20 smithi038 ceph-mon[32348]: pgmap v416: 97 pgs: 97 active+clean; 611 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.8 MiB/s wr, 155 op/s 2024-04-23T14:57:21.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:20 smithi038 ceph-mon[32348]: scrub summary: idle 2024-04-23T14:57:21.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:20 smithi028 ceph-mon[27810]: scrub queued for path: / 2024-04-23T14:57:21.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:20 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:57:21.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:20 smithi028 ceph-mon[27810]: scrub summary: active paths [/] 2024-04-23T14:57:21.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:20 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:57:21.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:20 smithi028 ceph-mon[27810]: pgmap v416: 97 pgs: 97 active+clean; 611 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.8 MiB/s wr, 155 op/s 2024-04-23T14:57:21.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:20 smithi028 ceph-mon[27810]: scrub summary: idle 2024-04-23T14:57:21.549 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1130297 8.43 MB/sec execute 577 sec latency 19.916 ms 2024-04-23T14:57:22.549 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1131703 8.42 MB/sec execute 578 sec latency 51.492 ms 2024-04-23T14:57:23.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:22 smithi105 ceph-mon[32555]: pgmap v417: 97 pgs: 97 active+clean; 611 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.8 MiB/s wr, 155 op/s 2024-04-23T14:57:23.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:22 smithi038 ceph-mon[32348]: pgmap v417: 97 pgs: 97 active+clean; 611 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.8 MiB/s wr, 155 op/s 2024-04-23T14:57:23.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:22 smithi028 ceph-mon[27810]: pgmap v417: 97 pgs: 97 active+clean; 611 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.8 MiB/s wr, 155 op/s 2024-04-23T14:57:23.549 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1133248 8.43 MB/sec execute 579 sec latency 17.473 ms 2024-04-23T14:57:24.549 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1134182 8.42 MB/sec execute 580 sec latency 25.135 ms 2024-04-23T14:57:25.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:24 smithi038 ceph-mon[32348]: pgmap v418: 97 pgs: 97 active+clean; 555 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.8 MiB/s wr, 197 op/s 2024-04-23T14:57:25.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:24 smithi028 ceph-mon[27810]: pgmap v418: 97 pgs: 97 active+clean; 555 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.8 MiB/s wr, 197 op/s 2024-04-23T14:57:25.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:24 smithi105 ceph-mon[32555]: pgmap v418: 97 pgs: 97 active+clean; 555 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.8 MiB/s wr, 197 op/s 2024-04-23T14:57:25.550 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1135706 8.42 MB/sec execute 581 sec latency 29.175 ms 2024-04-23T14:57:26.550 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1137022 8.42 MB/sec execute 582 sec latency 23.608 ms 2024-04-23T14:57:27.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:26 smithi038 ceph-mon[32348]: pgmap v419: 97 pgs: 97 active+clean; 555 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.3 MiB/s wr, 136 op/s 2024-04-23T14:57:27.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:26 smithi028 ceph-mon[27810]: pgmap v419: 97 pgs: 97 active+clean; 555 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.3 MiB/s wr, 136 op/s 2024-04-23T14:57:27.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:26 smithi105 ceph-mon[32555]: pgmap v419: 97 pgs: 97 active+clean; 555 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.3 MiB/s wr, 136 op/s 2024-04-23T14:57:27.550 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1138302 8.41 MB/sec execute 583 sec latency 40.371 ms 2024-04-23T14:57:28.550 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1140003 8.42 MB/sec execute 584 sec latency 26.815 ms 2024-04-23T14:57:29.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:28 smithi038 ceph-mon[32348]: pgmap v420: 97 pgs: 97 active+clean; 570 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.8 MiB/s wr, 188 op/s 2024-04-23T14:57:29.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:28 smithi028 ceph-mon[27810]: pgmap v420: 97 pgs: 97 active+clean; 570 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.8 MiB/s wr, 188 op/s 2024-04-23T14:57:29.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:28 smithi105 ceph-mon[32555]: pgmap v420: 97 pgs: 97 active+clean; 570 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.8 MiB/s wr, 188 op/s 2024-04-23T14:57:29.550 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1140821 8.41 MB/sec execute 585 sec latency 23.628 ms 2024-04-23T14:57:30.550 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1141810 8.40 MB/sec execute 586 sec latency 35.999 ms 2024-04-23T14:57:31.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:30 smithi038 ceph-mon[32348]: pgmap v421: 97 pgs: 97 active+clean; 570 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 118 op/s 2024-04-23T14:57:31.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:30 smithi028 ceph-mon[27810]: pgmap v421: 97 pgs: 97 active+clean; 570 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 118 op/s 2024-04-23T14:57:31.285 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:30 smithi105 ceph-mon[32555]: pgmap v421: 97 pgs: 97 active+clean; 570 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 118 op/s 2024-04-23T14:57:31.550 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1143852 8.41 MB/sec execute 587 sec latency 30.206 ms 2024-04-23T14:57:32.550 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1144952 8.40 MB/sec execute 588 sec latency 32.061 ms 2024-04-23T14:57:33.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:32 smithi038 ceph-mon[32348]: pgmap v422: 97 pgs: 97 active+clean; 570 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 118 op/s 2024-04-23T14:57:33.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:32 smithi028 ceph-mon[27810]: pgmap v422: 97 pgs: 97 active+clean; 570 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 118 op/s 2024-04-23T14:57:33.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:32 smithi105 ceph-mon[32555]: pgmap v422: 97 pgs: 97 active+clean; 570 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 118 op/s 2024-04-23T14:57:33.550 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1146663 8.41 MB/sec execute 589 sec latency 36.605 ms 2024-04-23T14:57:34.551 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1147881 8.40 MB/sec execute 590 sec latency 17.182 ms 2024-04-23T14:57:35.087 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:34 smithi105 ceph-mon[32555]: pgmap v423: 97 pgs: 97 active+clean; 578 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.2 MiB/s wr, 176 op/s 2024-04-23T14:57:35.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:34 smithi038 ceph-mon[32348]: pgmap v423: 97 pgs: 97 active+clean; 578 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.2 MiB/s wr, 176 op/s 2024-04-23T14:57:35.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:34 smithi028 ceph-mon[27810]: pgmap v423: 97 pgs: 97 active+clean; 578 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.2 MiB/s wr, 176 op/s 2024-04-23T14:57:35.551 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1149493 8.40 MB/sec execute 591 sec latency 22.405 ms 2024-04-23T14:57:36.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:35 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:57:36.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:35 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:57:36.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:35 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:57:36.551 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1151227 8.41 MB/sec execute 592 sec latency 18.658 ms 2024-04-23T14:57:37.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:36 smithi038 ceph-mon[32348]: pgmap v424: 97 pgs: 97 active+clean; 578 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 110 op/s 2024-04-23T14:57:37.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:36 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:57:37.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:36 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:57:37.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:36 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:57:37.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:36 smithi028 ceph-mon[27810]: pgmap v424: 97 pgs: 97 active+clean; 578 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 110 op/s 2024-04-23T14:57:37.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:36 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:57:37.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:36 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:57:37.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:36 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:57:37.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:36 smithi105 ceph-mon[32555]: pgmap v424: 97 pgs: 97 active+clean; 578 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 110 op/s 2024-04-23T14:57:37.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:36 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:57:37.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:36 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:57:37.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:36 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:57:37.551 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1152397 8.40 MB/sec execute 593 sec latency 42.987 ms 2024-04-23T14:57:38.551 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1154453 8.41 MB/sec execute 594 sec latency 49.260 ms 2024-04-23T14:57:39.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:38 smithi038 ceph-mon[32348]: pgmap v425: 97 pgs: 97 active+clean; 591 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.2 MiB/s wr, 170 op/s 2024-04-23T14:57:39.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:38 smithi028 ceph-mon[27810]: pgmap v425: 97 pgs: 97 active+clean; 591 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.2 MiB/s wr, 170 op/s 2024-04-23T14:57:39.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:38 smithi105 ceph-mon[32555]: pgmap v425: 97 pgs: 97 active+clean; 591 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.2 MiB/s wr, 170 op/s 2024-04-23T14:57:39.551 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1155809 8.41 MB/sec execute 595 sec latency 21.663 ms 2024-04-23T14:57:40.552 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1157739 8.41 MB/sec execute 596 sec latency 23.593 ms 2024-04-23T14:57:41.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:40 smithi038 ceph-mon[32348]: pgmap v426: 97 pgs: 97 active+clean; 591 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 118 op/s 2024-04-23T14:57:41.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:40 smithi028 ceph-mon[27810]: pgmap v426: 97 pgs: 97 active+clean; 591 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 118 op/s 2024-04-23T14:57:41.285 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:40 smithi105 ceph-mon[32555]: pgmap v426: 97 pgs: 97 active+clean; 591 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 118 op/s 2024-04-23T14:57:41.552 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1158916 8.40 MB/sec execute 597 sec latency 14.560 ms 2024-04-23T14:57:42.552 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1160345 8.40 MB/sec execute 598 sec latency 81.854 ms 2024-04-23T14:57:43.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:42 smithi038 ceph-mon[32348]: pgmap v427: 97 pgs: 97 active+clean; 591 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 118 op/s 2024-04-23T14:57:43.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:42 smithi028 ceph-mon[27810]: pgmap v427: 97 pgs: 97 active+clean; 591 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 118 op/s 2024-04-23T14:57:43.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:42 smithi105 ceph-mon[32555]: pgmap v427: 97 pgs: 97 active+clean; 591 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 118 op/s 2024-04-23T14:57:43.552 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1161735 8.40 MB/sec execute 599 sec latency 37.213 ms 2024-04-23T14:57:44.552 INFO:tasks.workunit.client.0.smithi028.stdout: 1 cleanup 600 sec 2024-04-23T14:57:44.712 INFO:tasks.workunit.client.0.smithi028.stdout: 0 cleanup 600 sec 2024-04-23T14:57:44.713 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-23T14:57:44.713 INFO:tasks.workunit.client.0.smithi028.stdout: Operation Count AvgLat MaxLat 2024-04-23T14:57:44.713 INFO:tasks.workunit.client.0.smithi028.stdout: ---------------------------------------- 2024-04-23T14:57:44.713 INFO:tasks.workunit.client.0.smithi028.stdout: NTCreateX 160554 1.296 3910.731 2024-04-23T14:57:44.713 INFO:tasks.workunit.client.0.smithi028.stdout: Close 117907 0.009 0.648 2024-04-23T14:57:44.713 INFO:tasks.workunit.client.0.smithi028.stdout: Rename 6800 5.357 97.707 2024-04-23T14:57:44.713 INFO:tasks.workunit.client.0.smithi028.stdout: Unlink 32452 2.068 75.783 2024-04-23T14:57:44.713 INFO:tasks.workunit.client.0.smithi028.stdout: Deltree 4 98.107 141.184 2024-04-23T14:57:44.713 INFO:tasks.workunit.client.0.smithi028.stdout: Mkdir 2 0.043 0.045 2024-04-23T14:57:44.713 INFO:tasks.workunit.client.0.smithi028.stdout: Qpathinfo 145500 1.140 144.161 2024-04-23T14:57:44.713 INFO:tasks.workunit.client.0.smithi028.stdout: Qfileinfo 25479 0.005 0.077 2024-04-23T14:57:44.713 INFO:tasks.workunit.client.0.smithi028.stdout: Qfsinfo 26684 0.475 70.322 2024-04-23T14:57:44.713 INFO:tasks.workunit.client.0.smithi028.stdout: Sfileinfo 13088 0.076 1.724 2024-04-23T14:57:44.713 INFO:tasks.workunit.client.0.smithi028.stdout: Find 56238 0.312 47.794 2024-04-23T14:57:44.713 INFO:tasks.workunit.client.0.smithi028.stdout: WriteX 80022 0.162 111.216 2024-04-23T14:57:44.714 INFO:tasks.workunit.client.0.smithi028.stdout: ReadX 251437 0.015 38.809 2024-04-23T14:57:44.714 INFO:tasks.workunit.client.0.smithi028.stdout: LockX 522 0.852 4.427 2024-04-23T14:57:44.714 INFO:tasks.workunit.client.0.smithi028.stdout: UnlockX 522 0.733 2.958 2024-04-23T14:57:44.714 INFO:tasks.workunit.client.0.smithi028.stdout: Flush 11259 5.717 156.560 2024-04-23T14:57:44.714 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-23T14:57:44.714 INFO:tasks.workunit.client.0.smithi028.stdout:Throughput 8.40379 MB/sec 1 clients 1 procs max_latency=3910.747 ms 2024-04-23T14:57:44.714 INFO:tasks.workunit.client.0.smithi028.stderr:+ dbench 10 2024-04-23T14:57:44.716 INFO:tasks.workunit.client.0.smithi028.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-23T14:57:44.716 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-23T14:57:44.716 INFO:tasks.workunit.client.0.smithi028.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-04-23T14:57:44.721 INFO:tasks.workunit.client.0.smithi028.stdout:9 of 10 processes prepared for launch 0 sec 2024-04-23T14:57:44.821 INFO:tasks.workunit.client.0.smithi028.stdout:10 of 10 processes prepared for launch 0 sec 2024-04-23T14:57:44.821 INFO:tasks.workunit.client.0.smithi028.stdout:releasing clients 2024-04-23T14:57:45.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:44 smithi038 ceph-mon[32348]: pgmap v428: 97 pgs: 97 active+clean; 603 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.4 MiB/s wr, 185 op/s 2024-04-23T14:57:45.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:44 smithi028 ceph-mon[27810]: pgmap v428: 97 pgs: 97 active+clean; 603 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.4 MiB/s wr, 185 op/s 2024-04-23T14:57:45.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:44 smithi105 ceph-mon[32555]: pgmap v428: 97 pgs: 97 active+clean; 603 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.4 MiB/s wr, 185 op/s 2024-04-23T14:57:45.822 INFO:tasks.workunit.client.0.smithi028.stdout: 10 394 140.05 MB/sec warmup 1 sec latency 149.941 ms 2024-04-23T14:57:46.822 INFO:tasks.workunit.client.0.smithi028.stdout: 10 848 104.08 MB/sec warmup 2 sec latency 183.909 ms 2024-04-23T14:57:47.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:46 smithi038 ceph-mon[32348]: pgmap v429: 97 pgs: 97 active+clean; 603 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 127 op/s 2024-04-23T14:57:47.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:46 smithi028 ceph-mon[27810]: pgmap v429: 97 pgs: 97 active+clean; 603 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 127 op/s 2024-04-23T14:57:47.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:46 smithi105 ceph-mon[32555]: pgmap v429: 97 pgs: 97 active+clean; 603 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 127 op/s 2024-04-23T14:57:47.822 INFO:tasks.workunit.client.0.smithi028.stdout: 10 1364 83.13 MB/sec warmup 3 sec latency 178.422 ms 2024-04-23T14:57:48.822 INFO:tasks.workunit.client.0.smithi028.stdout: 10 2170 67.05 MB/sec warmup 4 sec latency 164.641 ms 2024-04-23T14:57:49.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:48 smithi038 ceph-mon[32348]: pgmap v430: 97 pgs: 97 active+clean; 616 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.0 MiB/s wr, 230 op/s 2024-04-23T14:57:49.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:48 smithi028 ceph-mon[27810]: pgmap v430: 97 pgs: 97 active+clean; 616 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.0 MiB/s wr, 230 op/s 2024-04-23T14:57:49.273 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:57:49.273 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:57:49.285 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:48 smithi105 ceph-mon[32555]: pgmap v430: 97 pgs: 97 active+clean; 616 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.0 MiB/s wr, 230 op/s 2024-04-23T14:57:49.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.503+0000 7fd973000640 1 -- 172.21.15.28:0/135220604 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd96c104320 msgr2=0x7fd96c104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:49.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.503+0000 7fd973000640 1 --2- 172.21.15.28:0/135220604 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd96c104320 0x7fd96c104700 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fd960009900 tx=0x7fd96002a3f0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.503+0000 7fd973000640 1 -- 172.21.15.28:0/135220604 shutdown_connections 2024-04-23T14:57:49.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.503+0000 7fd973000640 1 --2- 172.21.15.28:0/135220604 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd96c1093f0 0x7fd96c110fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.503+0000 7fd973000640 1 --2- 172.21.15.28:0/135220604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd96c104cd0 0x7fd96c108d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.503+0000 7fd973000640 1 --2- 172.21.15.28:0/135220604 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd96c104320 0x7fd96c104700 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.503+0000 7fd973000640 1 -- 172.21.15.28:0/135220604 >> 172.21.15.28:0/135220604 conn(0x7fd96c0ffb40 msgr2=0x7fd96c101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:49.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.504+0000 7fd973000640 1 -- 172.21.15.28:0/135220604 shutdown_connections 2024-04-23T14:57:49.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.504+0000 7fd973000640 1 -- 172.21.15.28:0/135220604 wait complete. 2024-04-23T14:57:49.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.505+0000 7fd973000640 1 Processor -- start 2024-04-23T14:57:49.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.505+0000 7fd973000640 1 -- start start 2024-04-23T14:57:49.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.505+0000 7fd973000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd96c104320 0x7fd96c1a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:49.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.505+0000 7fd973000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd96c1093f0 0x7fd96c1a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:49.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.505+0000 7fd973000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd96c19a7f0 0x7fd96c19aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:49.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.505+0000 7fd973000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c1144d0 con 0x7fd96c19a7f0 2024-04-23T14:57:49.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.505+0000 7fd973000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c114350 con 0x7fd96c1093f0 2024-04-23T14:57:49.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.505+0000 7fd973000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c114650 con 0x7fd96c104320 2024-04-23T14:57:49.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.505+0000 7fd971200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd96c1093f0 0x7fd96c1a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:49.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.506+0000 7fd971200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd96c1093f0 0x7fd96c1a0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:41864/0 (socket says 172.21.15.28:41864) 2024-04-23T14:57:49.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.506+0000 7fd971c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd96c104320 0x7fd96c1a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:49.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.506+0000 7fd971200640 1 -- 172.21.15.28:0/493830643 learned_addr learned my addr 172.21.15.28:0/493830643 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:49.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.506+0000 7fd972600640 1 --2- 172.21.15.28:0/493830643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd96c19a7f0 0x7fd96c19aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:49.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.506+0000 7fd971200640 1 -- 172.21.15.28:0/493830643 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd96c104320 msgr2=0x7fd96c1a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:49.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.506+0000 7fd971200640 1 --2- 172.21.15.28:0/493830643 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd96c104320 0x7fd96c1a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.506+0000 7fd971200640 1 -- 172.21.15.28:0/493830643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd96c19a7f0 msgr2=0x7fd96c19aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:49.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.506+0000 7fd971200640 1 --2- 172.21.15.28:0/493830643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd96c19a7f0 0x7fd96c19aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.506+0000 7fd971200640 1 -- 172.21.15.28:0/493830643 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd96c19b380 con 0x7fd96c1093f0 2024-04-23T14:57:49.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.506+0000 7fd971200640 1 --2- 172.21.15.28:0/493830643 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd96c1093f0 0x7fd96c1a0bc0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fd95c00e9a0 tx=0x7fd95c00ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:49.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.506+0000 7fd96a000640 1 -- 172.21.15.28:0/493830643 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95c00cd80 con 0x7fd96c1093f0 2024-04-23T14:57:49.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.507+0000 7fd96a000640 1 -- 172.21.15.28:0/493830643 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd95c0035f0 con 0x7fd96c1093f0 2024-04-23T14:57:49.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.507+0000 7fd96a000640 1 -- 172.21.15.28:0/493830643 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95c010670 con 0x7fd96c1093f0 2024-04-23T14:57:49.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.507+0000 7fd973000640 1 -- 172.21.15.28:0/493830643 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd96c19b670 con 0x7fd96c1093f0 2024-04-23T14:57:49.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.507+0000 7fd973000640 1 -- 172.21.15.28:0/493830643 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd96c1a78a0 con 0x7fd96c1093f0 2024-04-23T14:57:49.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.510+0000 7fd96a000640 1 -- 172.21.15.28:0/493830643 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fd95c0037a0 con 0x7fd96c1093f0 2024-04-23T14:57:49.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.510+0000 7fd977410740 1 Processor -- start 2024-04-23T14:57:49.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.510+0000 7fd977410740 1 -- start start 2024-04-23T14:57:49.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.511+0000 7fd977410740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5565475b6a30 0x5565475b6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:49.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.511+0000 7fd977410740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565475b7350 0x5565475c14e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:49.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.511+0000 7fd977410740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565475c1a20 0x5565475c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:49.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.511+0000 7fd977410740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55654751c1f0 con 0x5565475c1a20 2024-04-23T14:57:49.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.511+0000 7fd977410740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5565474c2350 con 0x5565475b7350 2024-04-23T14:57:49.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.511+0000 7fd977410740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5565475181b0 con 0x5565475b6a30 2024-04-23T14:57:49.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.511+0000 7fd971c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565475b7350 0x5565475c14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:49.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.511+0000 7fd972600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5565475b6a30 0x5565475b6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:49.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.511+0000 7fd972600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5565475b6a30 0x5565475b6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:34440/0 (socket says 172.21.15.28:34440) 2024-04-23T14:57:49.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.511+0000 7fd971c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565475b7350 0x5565475c14e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:41880/0 (socket says 172.21.15.28:41880) 2024-04-23T14:57:49.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.511+0000 7fd972600640 1 -- 172.21.15.28:0/359982325 learned_addr learned my addr 172.21.15.28:0/359982325 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:49.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.511+0000 7fd971200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565475c1a20 0x5565475c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:49.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.512+0000 7fd972600640 1 -- 172.21.15.28:0/359982325 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565475b7350 msgr2=0x5565475c14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:49.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.512+0000 7fd972600640 1 --2- 172.21.15.28:0/359982325 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565475b7350 0x5565475c14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.512+0000 7fd972600640 1 -- 172.21.15.28:0/359982325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565475c1a20 msgr2=0x5565475c3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:49.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.512+0000 7fd972600640 1 --2- 172.21.15.28:0/359982325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565475c1a20 0x5565475c3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.512+0000 7fd972600640 1 -- 172.21.15.28:0/359982325 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556547516f50 con 0x5565475b6a30 2024-04-23T14:57:49.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.512+0000 7fd972600640 1 --2- 172.21.15.28:0/359982325 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5565475b6a30 0x5565475b6e10 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd964009f40 tx=0x7fd964010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e1209dcac43b3da server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:49.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.512+0000 7fd96a000640 1 --2- 172.21.15.28:0/493830643 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd9540a3da0 0x7fd9540a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:49.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.512+0000 7fd95aa00640 1 -- 172.21.15.28:0/359982325 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd964003b70 con 0x5565475b6a30 2024-04-23T14:57:49.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.513+0000 7fd95aa00640 1 -- 172.21.15.28:0/359982325 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd964003d10 con 0x5565475b6a30 2024-04-23T14:57:49.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.513+0000 7fd95aa00640 1 -- 172.21.15.28:0/359982325 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd964011af0 con 0x5565475b6a30 2024-04-23T14:57:49.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.513+0000 7fd96a000640 1 -- 172.21.15.28:0/493830643 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8542+0+0 (secure 0 0 0) 0x7fd95c0d0b00 con 0x7fd96c1093f0 2024-04-23T14:57:49.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.513+0000 7fd971c00640 1 --2- 172.21.15.28:0/493830643 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd9540a3da0 0x7fd9540a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:49.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.513+0000 7fd977410740 1 -- 172.21.15.28:0/359982325 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5565475b6a30 msgr2=0x5565475b6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:49.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.513+0000 7fd977410740 1 --2- 172.21.15.28:0/359982325 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5565475b6a30 0x5565475b6e10 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd964009f40 tx=0x7fd964010860 comp rx=0 tx=0).stop 2024-04-23T14:57:49.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.514+0000 7fd971c00640 1 --2- 172.21.15.28:0/493830643 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd9540a3da0 0x7fd9540a6260 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd960000c00 tx=0x7fd960005b50 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:49.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.514+0000 7fd977410740 1 -- 172.21.15.28:0/359982325 shutdown_connections 2024-04-23T14:57:49.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.514+0000 7fd977410740 1 --2- 172.21.15.28:0/359982325 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565475b7350 0x5565475c14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.514+0000 7fd977410740 1 --2- 172.21.15.28:0/359982325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565475c1a20 0x5565475c3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.514+0000 7fd977410740 1 --2- 172.21.15.28:0/359982325 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5565475b6a30 0x5565475b6e10 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.514+0000 7fd977410740 1 -- 172.21.15.28:0/359982325 >> 172.21.15.28:0/359982325 conn(0x5565475b2d50 msgr2=0x5565475b3130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:49.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.514+0000 7fd977410740 1 -- 172.21.15.28:0/359982325 shutdown_connections 2024-04-23T14:57:49.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.514+0000 7fd977410740 1 -- 172.21.15.28:0/359982325 wait complete. 2024-04-23T14:57:49.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.514+0000 7fd977410740 1 -- start start 2024-04-23T14:57:49.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.514+0000 7fd977410740 1 Processor -- start 2024-04-23T14:57:49.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.515+0000 7fd977410740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565475b6a30 0x5565475bea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:49.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.515+0000 7fd977410740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565475b7350 0x5565475bef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:49.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.515+0000 7fd977410740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5565475c1a20 0x5565475bf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:49.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.515+0000 7fd977410740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5565475c4810 con 0x5565475b7350 2024-04-23T14:57:49.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.515+0000 7fd977410740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5565474d5690 con 0x5565475b6a30 2024-04-23T14:57:49.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.515+0000 7fd977410740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5565475c4990 con 0x5565475c1a20 2024-04-23T14:57:49.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.515+0000 7fd977410740 10 client.0 _ll_register_callbacks cb 0x556547511650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:57:49.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.515+0000 7fd971c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565475b6a30 0x5565475bea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:49.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.515+0000 7fd971c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565475b6a30 0x5565475bea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:41894/0 (socket says 172.21.15.28:41894) 2024-04-23T14:57:49.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.515+0000 7fd971c00640 1 -- 172.21.15.28:0/1810914940 learned_addr learned my addr 172.21.15.28:0/1810914940 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:49.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.516+0000 7fd972600640 1 --2- 172.21.15.28:0/1810914940 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5565475c1a20 0x5565475bf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:49.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.516+0000 7fd971200640 1 --2- 172.21.15.28:0/1810914940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565475b7350 0x5565475bef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:49.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.516+0000 7fd971c00640 1 -- 172.21.15.28:0/1810914940 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5565475c1a20 msgr2=0x5565475bf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:49.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.516+0000 7fd971c00640 1 --2- 172.21.15.28:0/1810914940 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5565475c1a20 0x5565475bf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.516+0000 7fd971c00640 1 -- 172.21.15.28:0/1810914940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565475b7350 msgr2=0x5565475bef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:49.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.516+0000 7fd971c00640 1 --2- 172.21.15.28:0/1810914940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565475b7350 0x5565475bef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.516+0000 7fd971c00640 1 -- 172.21.15.28:0/1810914940 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565474d4920 con 0x5565475b6a30 2024-04-23T14:57:49.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.516+0000 7fd971200640 1 --2- 172.21.15.28:0/1810914940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565475b7350 0x5565475bef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:57:49.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.516+0000 7fd971c00640 1 --2- 172.21.15.28:0/1810914940 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565475b6a30 0x5565475bea10 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fd960031550 tx=0x7fd960009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ad02c78e6ef7531 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:49.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.516+0000 7fd94f400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:57:49.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.517+0000 7fd94f400640 1 -- 172.21.15.28:0/1810914940 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd96004e070 con 0x5565475b6a30 2024-04-23T14:57:49.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.517+0000 7fd94f400640 1 -- 172.21.15.28:0/1810914940 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8542+0+0 (secure 0 0 0) 0x7fd960004c10 con 0x5565475b6a30 2024-04-23T14:57:49.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.517+0000 7fd977410740 1 -- 172.21.15.28:0/1810914940 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd964003d10 con 0x5565475b6a30 2024-04-23T14:57:49.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.517+0000 7fd94f400640 1 -- 172.21.15.28:0/1810914940 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fd96005a050 con 0x5565475b6a30 2024-04-23T14:57:49.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.517+0000 7fd977410740 10 client.27341 fetch_fsmap learned FSMap version 20 2024-04-23T14:57:49.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.517+0000 7fd977410740 1 -- 172.21.15.28:0/1810914940 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5565472f87d0 con 0x5565475b6a30 2024-04-23T14:57:49.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.517+0000 7fd94f400640 1 -- 172.21.15.28:0/1810914940 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd964003d10 con 0x5565475b6a30 2024-04-23T14:57:49.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.517+0000 7fd94f400640 1 -- 172.21.15.28:0/1810914940 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fd960003850 con 0x5565475b6a30 2024-04-23T14:57:49.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.518+0000 7fd977410740 10 client.27341 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:57:49.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.518+0000 7fd977410740 10 client.27341 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:57:49.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.518+0000 7fd977410740 20 client.27341 populate_metadata read hostname 'smithi028' 2024-04-23T14:57:49.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.518+0000 7fd977410740 1 --2- 172.21.15.28:0/1810914940 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5565475d1da0 0x5565475d4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:49.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.518+0000 7fd977410740 4 client.27341 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:57:49.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.518+0000 7fd977410740 1 -- 172.21.15.28:0/1810914940 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556547529c10 con 0x5565475d1da0 2024-04-23T14:57:49.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.518+0000 7fd971200640 1 --2- 172.21.15.28:0/1810914940 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5565475d1da0 0x5565475d4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:49.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.518+0000 7fd971200640 1 --2- 172.21.15.28:0/1810914940 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5565475d1da0 0x5565475d4210 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8923fdc40f44fedf server_cookie=ca48b071ab50754a in_seq=0 out_seq=0 2024-04-23T14:57:49.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.518+0000 7fd94f400640 10 client.27341 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:57:49.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.523+0000 7fd94f400640 1 -- 172.21.15.28:0/1810914940 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556547529c10 con 0x5565475d1da0 2024-04-23T14:57:49.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.523+0000 7fd94f400640 10 client.27341 handle_command_reply: tid=0 2024-04-23T14:57:49.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.523+0000 7fd977410740 1 client.27341 shutdown 2024-04-23T14:57:49.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.523+0000 7fd977410740 1 -- 172.21.15.28:0/1810914940 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565475b6a30 msgr2=0x5565475bea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:49.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.523+0000 7fd977410740 1 --2- 172.21.15.28:0/1810914940 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565475b6a30 0x5565475bea10 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fd960031550 tx=0x7fd960009260 comp rx=0 tx=0).stop 2024-04-23T14:57:49.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.523+0000 7fd977410740 1 -- 172.21.15.28:0/1810914940 shutdown_connections 2024-04-23T14:57:49.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.523+0000 7fd977410740 1 --2- 172.21.15.28:0/1810914940 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5565475d1da0 0x5565475d4210 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.523+0000 7fd977410740 1 --2- 172.21.15.28:0/1810914940 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5565475c1a20 0x5565475bf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.523+0000 7fd977410740 1 --2- 172.21.15.28:0/1810914940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565475b7350 0x5565475bef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.523+0000 7fd977410740 1 --2- 172.21.15.28:0/1810914940 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565475b6a30 0x5565475bea10 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.523+0000 7fd94f400640 0 client.27341 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:57:49.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.523+0000 7fd977410740 1 -- 172.21.15.28:0/1810914940 >> 172.21.15.28:0/1810914940 conn(0x5565475b2d50 msgr2=0x5565475b8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:49.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.523+0000 7fd977410740 1 -- 172.21.15.28:0/1810914940 shutdown_connections 2024-04-23T14:57:49.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.523+0000 7fd977410740 1 -- 172.21.15.28:0/1810914940 wait complete. 2024-04-23T14:57:49.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.523+0000 7fd977410740 20 client.27341 trim_cache size 0 max 16384 2024-04-23T14:57:49.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.545+0000 7fd977410740 1 Processor -- start 2024-04-23T14:57:49.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.546+0000 7fd977410740 1 -- start start 2024-04-23T14:57:49.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.546+0000 7fd977410740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5565475b6a30 0x5565475bd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:49.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.546+0000 7fd977410740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565475b7350 0x5565475b9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:49.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.546+0000 7fd977410740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565475c1a20 0x5565475d9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:49.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.546+0000 7fd977410740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c1ade60 con 0x5565475c1a20 2024-04-23T14:57:49.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.546+0000 7fd977410740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c1b06c0 con 0x5565475b7350 2024-04-23T14:57:49.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.546+0000 7fd977410740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c1ae4b0 con 0x5565475b6a30 2024-04-23T14:57:49.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.546+0000 7fd971200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565475b7350 0x5565475b9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:49.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.546+0000 7fd971c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5565475b6a30 0x5565475bd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:49.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.546+0000 7fd971200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565475b7350 0x5565475b9190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:41896/0 (socket says 172.21.15.28:41896) 2024-04-23T14:57:49.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.546+0000 7fd971200640 1 -- 172.21.15.28:0/2422414715 learned_addr learned my addr 172.21.15.28:0/2422414715 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:49.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.547+0000 7fd972600640 1 --2- 172.21.15.28:0/2422414715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565475c1a20 0x5565475d9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:49.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.547+0000 7fd971c00640 1 -- 172.21.15.28:0/2422414715 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565475b7350 msgr2=0x5565475b9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:49.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.547+0000 7fd971c00640 1 --2- 172.21.15.28:0/2422414715 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565475b7350 0x5565475b9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.547+0000 7fd971c00640 1 -- 172.21.15.28:0/2422414715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565475c1a20 msgr2=0x5565475d9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:49.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.547+0000 7fd971c00640 1 --2- 172.21.15.28:0/2422414715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565475c1a20 0x5565475d9ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.547+0000 7fd971c00640 1 -- 172.21.15.28:0/2422414715 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55654722e3f0 con 0x5565475b6a30 2024-04-23T14:57:49.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.547+0000 7fd971200640 1 --2- 172.21.15.28:0/2422414715 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565475b7350 0x5565475b9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:57:49.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.547+0000 7fd971c00640 1 --2- 172.21.15.28:0/2422414715 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5565475b6a30 0x5565475bd2e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fd960003c70 tx=0x7fd960005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64be77f71673c1fb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:49.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.547+0000 7fd95a000640 1 -- 172.21.15.28:0/2422414715 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd96004e050 con 0x5565475b6a30 2024-04-23T14:57:49.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.547+0000 7fd95a000640 1 -- 172.21.15.28:0/2422414715 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd960005850 con 0x5565475b6a30 2024-04-23T14:57:49.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.547+0000 7fd95a000640 1 -- 172.21.15.28:0/2422414715 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd960035040 con 0x5565475b6a30 2024-04-23T14:57:49.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.549+0000 7fd977410740 1 -- 172.21.15.28:0/2422414715 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5565475b6a30 msgr2=0x5565475bd2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:49.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.549+0000 7fd977410740 1 --2- 172.21.15.28:0/2422414715 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5565475b6a30 0x5565475bd2e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fd960003c70 tx=0x7fd960005c00 comp rx=0 tx=0).stop 2024-04-23T14:57:49.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.549+0000 7fd977410740 1 -- 172.21.15.28:0/2422414715 shutdown_connections 2024-04-23T14:57:49.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.549+0000 7fd977410740 1 --2- 172.21.15.28:0/2422414715 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565475b7350 0x5565475b9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.549+0000 7fd977410740 1 --2- 172.21.15.28:0/2422414715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565475c1a20 0x5565475d9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.549+0000 7fd977410740 1 --2- 172.21.15.28:0/2422414715 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5565475b6a30 0x5565475bd2e0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.549+0000 7fd977410740 1 -- 172.21.15.28:0/2422414715 >> 172.21.15.28:0/2422414715 conn(0x5565475b2d50 msgr2=0x5565475b1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:49.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.549+0000 7fd977410740 1 -- 172.21.15.28:0/2422414715 shutdown_connections 2024-04-23T14:57:49.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.549+0000 7fd977410740 1 -- 172.21.15.28:0/2422414715 wait complete. 2024-04-23T14:57:49.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.549+0000 7fd977410740 1 -- start start 2024-04-23T14:57:49.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.550+0000 7fd977410740 1 Processor -- start 2024-04-23T14:57:49.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.550+0000 7fd977410740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565475b6a30 0x5565475c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:49.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.550+0000 7fd977410740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5565475b7350 0x5565475c5250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:49.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.550+0000 7fd977410740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565475c1a20 0x5565475c5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:49.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.550+0000 7fd977410740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c1aeb80 con 0x5565475b6a30 2024-04-23T14:57:49.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.550+0000 7fd977410740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c1afff0 con 0x5565475c1a20 2024-04-23T14:57:49.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.550+0000 7fd977410740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c1af250 con 0x5565475b7350 2024-04-23T14:57:49.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.550+0000 7fd977410740 10 client.0 _ll_register_callbacks cb 0x7fd960009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:57:49.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.551+0000 7fd971c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565475b6a30 0x5565475c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:49.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.551+0000 7fd971c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565475b6a30 0x5565475c9ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36918/0 (socket says 172.21.15.28:36918) 2024-04-23T14:57:49.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.551+0000 7fd971c00640 1 -- 172.21.15.28:0/1516883561 learned_addr learned my addr 172.21.15.28:0/1516883561 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:49.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.551+0000 7fd972600640 1 --2- 172.21.15.28:0/1516883561 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565475c1a20 0x5565475c5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:49.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.551+0000 7fd971200640 1 --2- 172.21.15.28:0/1516883561 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5565475b7350 0x5565475c5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:49.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.551+0000 7fd972600640 1 -- 172.21.15.28:0/1516883561 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5565475b7350 msgr2=0x5565475c5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:49.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.551+0000 7fd972600640 1 --2- 172.21.15.28:0/1516883561 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5565475b7350 0x5565475c5250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.551+0000 7fd972600640 1 -- 172.21.15.28:0/1516883561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565475b6a30 msgr2=0x5565475c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:49.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.551+0000 7fd972600640 1 --2- 172.21.15.28:0/1516883561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565475b6a30 0x5565475c9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.551+0000 7fd972600640 1 -- 172.21.15.28:0/1516883561 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565471ee0f0 con 0x5565475c1a20 2024-04-23T14:57:49.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.551+0000 7fd971c00640 1 --2- 172.21.15.28:0/1516883561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565475b6a30 0x5565475c9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:57:49.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.551+0000 7fd971200640 1 --2- 172.21.15.28:0/1516883561 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5565475b7350 0x5565475c5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:57:49.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.551+0000 7fd972600640 1 --2- 172.21.15.28:0/1516883561 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565475c1a20 0x5565475c5790 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fd964009f10 tx=0x7fd964003e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb4a068164bde96b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:49.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.551+0000 7fd94f400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:57:49.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.552+0000 7fd94f400640 1 -- 172.21.15.28:0/1516883561 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd964011910 con 0x5565475c1a20 2024-04-23T14:57:49.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.552+0000 7fd977410740 1 -- 172.21.15.28:0/1516883561 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd960005850 con 0x5565475c1a20 2024-04-23T14:57:49.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.552+0000 7fd94f400640 1 -- 172.21.15.28:0/1516883561 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8542+0+0 (secure 0 0 0) 0x7fd964011ab0 con 0x5565475c1a20 2024-04-23T14:57:49.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.552+0000 7fd94f400640 1 -- 172.21.15.28:0/1516883561 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fd964033050 con 0x5565475c1a20 2024-04-23T14:57:49.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.552+0000 7fd977410740 10 client.27347 fetch_fsmap learned FSMap version 20 2024-04-23T14:57:49.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.552+0000 7fd977410740 1 -- 172.21.15.28:0/1516883561 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5565474e1810 con 0x5565475c1a20 2024-04-23T14:57:49.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.552+0000 7fd94f400640 1 -- 172.21.15.28:0/1516883561 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd960005850 con 0x5565475c1a20 2024-04-23T14:57:49.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.553+0000 7fd94f400640 1 -- 172.21.15.28:0/1516883561 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fd9640254b0 con 0x5565475c1a20 2024-04-23T14:57:49.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.553+0000 7fd977410740 10 client.27347 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:57:49.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.553+0000 7fd977410740 10 client.27347 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:57:49.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.553+0000 7fd977410740 20 client.27347 populate_metadata read hostname 'smithi028' 2024-04-23T14:57:49.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.553+0000 7fd977410740 1 --2- 172.21.15.28:0/1516883561 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5565476382f0 0x55654763a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:49.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.553+0000 7fd977410740 4 client.27347 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:57:49.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.553+0000 7fd977410740 1 -- 172.21.15.28:0/1516883561 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556547523a80 con 0x5565476382f0 2024-04-23T14:57:49.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.553+0000 7fd971c00640 1 --2- 172.21.15.28:0/1516883561 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5565476382f0 0x55654763a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:49.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.554+0000 7fd971c00640 1 --2- 172.21.15.28:0/1516883561 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5565476382f0 0x55654763a710 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16e041495e3bd4dd server_cookie=fbbb7bd9b4962f5e in_seq=0 out_seq=0 2024-04-23T14:57:49.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.554+0000 7fd94f400640 10 client.27347 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:57:49.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.558+0000 7fd94f400640 1 -- 172.21.15.28:0/1516883561 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x556547523a80 con 0x5565476382f0 2024-04-23T14:57:49.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.558+0000 7fd94f400640 10 client.27347 handle_command_reply: tid=0 2024-04-23T14:57:49.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.558+0000 7fd977410740 1 client.27347 shutdown 2024-04-23T14:57:49.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.558+0000 7fd977410740 1 -- 172.21.15.28:0/1516883561 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565475c1a20 msgr2=0x5565475c5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:49.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.558+0000 7fd977410740 1 --2- 172.21.15.28:0/1516883561 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565475c1a20 0x5565475c5790 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fd964009f10 tx=0x7fd964003e10 comp rx=0 tx=0).stop 2024-04-23T14:57:49.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.558+0000 7fd977410740 1 -- 172.21.15.28:0/1516883561 shutdown_connections 2024-04-23T14:57:49.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.558+0000 7fd977410740 1 --2- 172.21.15.28:0/1516883561 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5565476382f0 0x55654763a710 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.558+0000 7fd977410740 1 --2- 172.21.15.28:0/1516883561 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565475c1a20 0x5565475c5790 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.558+0000 7fd977410740 1 --2- 172.21.15.28:0/1516883561 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5565475b7350 0x5565475c5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.558+0000 7fd94f400640 0 client.27347 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:57:49.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.558+0000 7fd977410740 1 --2- 172.21.15.28:0/1516883561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565475b6a30 0x5565475c9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.558+0000 7fd977410740 1 -- 172.21.15.28:0/1516883561 >> 172.21.15.28:0/1516883561 conn(0x5565475b2d50 msgr2=0x5565475b1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:49.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.559+0000 7fd977410740 1 -- 172.21.15.28:0/1516883561 shutdown_connections 2024-04-23T14:57:49.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.559+0000 7fd977410740 1 -- 172.21.15.28:0/1516883561 wait complete. 2024-04-23T14:57:49.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.559+0000 7fd977410740 20 client.27347 trim_cache size 0 max 16384 2024-04-23T14:57:49.560 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:57:49.561 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-23T14:57:49.561 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-23T14:57:49.561 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:57:49.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.559+0000 7fd973000640 1 -- 172.21.15.28:0/493830643 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd9540a3da0 msgr2=0x7fd9540a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:49.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.559+0000 7fd973000640 1 --2- 172.21.15.28:0/493830643 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd9540a3da0 0x7fd9540a6260 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd960000c00 tx=0x7fd960005b50 comp rx=0 tx=0).stop 2024-04-23T14:57:49.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.559+0000 7fd973000640 1 -- 172.21.15.28:0/493830643 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd96c1093f0 msgr2=0x7fd96c1a0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:49.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.559+0000 7fd973000640 1 --2- 172.21.15.28:0/493830643 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd96c1093f0 0x7fd96c1a0bc0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fd95c00e9a0 tx=0x7fd95c00ee60 comp rx=0 tx=0).stop 2024-04-23T14:57:49.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.560+0000 7fd973000640 1 -- 172.21.15.28:0/493830643 shutdown_connections 2024-04-23T14:57:49.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.560+0000 7fd973000640 1 --2- 172.21.15.28:0/493830643 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd9540a3da0 0x7fd9540a6260 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.560+0000 7fd973000640 1 --2- 172.21.15.28:0/493830643 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd96c1093f0 0x7fd96c1a0bc0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.560+0000 7fd973000640 1 --2- 172.21.15.28:0/493830643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd96c19a7f0 0x7fd96c19aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.560+0000 7fd973000640 1 --2- 172.21.15.28:0/493830643 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd96c104320 0x7fd96c1a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.560+0000 7fd973000640 1 -- 172.21.15.28:0/493830643 >> 172.21.15.28:0/493830643 conn(0x7fd96c0ffb40 msgr2=0x7fd96c101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:49.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.560+0000 7fd973000640 1 -- 172.21.15.28:0/493830643 shutdown_connections 2024-04-23T14:57:49.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.560+0000 7fd973000640 1 -- 172.21.15.28:0/493830643 wait complete. 2024-04-23T14:57:49.580 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-23T14:57:49.581 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:57:49.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.800+0000 7f8598e00640 1 -- 172.21.15.28:0/681905521 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85941019b0 msgr2=0x7f8594105e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:49.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.800+0000 7f8598e00640 1 --2- 172.21.15.28:0/681905521 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85941019b0 0x7f8594105e70 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f857c009900 tx=0x7f857c02a510 comp rx=0 tx=0).stop 2024-04-23T14:57:49.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.800+0000 7f8598e00640 1 -- 172.21.15.28:0/681905521 shutdown_connections 2024-04-23T14:57:49.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.800+0000 7f8598e00640 1 --2- 172.21.15.28:0/681905521 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85941019b0 0x7f8594105e70 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.800+0000 7f8598e00640 1 --2- 172.21.15.28:0/681905521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85941063b0 0x7f8594106840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.800+0000 7f8598e00640 1 --2- 172.21.15.28:0/681905521 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8594101090 0x7f8594101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.800+0000 7f8598e00640 1 -- 172.21.15.28:0/681905521 >> 172.21.15.28:0/681905521 conn(0x7f8594078000 msgr2=0x7f8594078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:49.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.801+0000 7f8598e00640 1 -- 172.21.15.28:0/681905521 shutdown_connections 2024-04-23T14:57:49.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.801+0000 7f8598e00640 1 -- 172.21.15.28:0/681905521 wait complete. 2024-04-23T14:57:49.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.802+0000 7f8598e00640 1 Processor -- start 2024-04-23T14:57:49.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.802+0000 7f8598e00640 1 -- start start 2024-04-23T14:57:49.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.803+0000 7f8598e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8594101090 0x7f859419eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:49.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.803+0000 7f8598e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85941019b0 0x7f859419f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:49.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.803+0000 7f8598e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f85941063b0 0x7f85941a3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:49.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.803+0000 7f8598e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8594118a30 con 0x7f8594101090 2024-04-23T14:57:49.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.803+0000 7f8598e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f85941188b0 con 0x7f85941019b0 2024-04-23T14:57:49.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.803+0000 7f8598e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8594118bb0 con 0x7f85941063b0 2024-04-23T14:57:49.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.803+0000 7f8591400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85941019b0 0x7f859419f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:49.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.803+0000 7f8591e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8594101090 0x7f859419eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:49.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.803+0000 7f8591400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85941019b0 0x7f859419f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:41912/0 (socket says 172.21.15.28:41912) 2024-04-23T14:57:49.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.803+0000 7f8591400640 1 -- 172.21.15.28:0/313901293 learned_addr learned my addr 172.21.15.28:0/313901293 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:49.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.803+0000 7f8592800640 1 --2- 172.21.15.28:0/313901293 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f85941063b0 0x7f85941a3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:49.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.803+0000 7f8591400640 1 -- 172.21.15.28:0/313901293 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f85941063b0 msgr2=0x7f85941a3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:49.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.803+0000 7f8591400640 1 --2- 172.21.15.28:0/313901293 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f85941063b0 0x7f85941a3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.803+0000 7f8591400640 1 -- 172.21.15.28:0/313901293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8594101090 msgr2=0x7f859419eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:49.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.803+0000 7f8591400640 1 --2- 172.21.15.28:0/313901293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8594101090 0x7f859419eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:49.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.803+0000 7f8591400640 1 -- 172.21.15.28:0/313901293 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85941a3ec0 con 0x7f85941019b0 2024-04-23T14:57:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.803+0000 7f8592800640 1 --2- 172.21.15.28:0/313901293 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f85941063b0 0x7f85941a3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:57:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.804+0000 7f8591400640 1 --2- 172.21.15.28:0/313901293 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85941019b0 0x7f859419f410 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f857c004770 tx=0x7f857c003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.804+0000 7f858aa00640 1 -- 172.21.15.28:0/313901293 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f857c03f070 con 0x7f85941019b0 2024-04-23T14:57:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.804+0000 7f858aa00640 1 -- 172.21.15.28:0/313901293 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f857c031ca0 con 0x7f85941019b0 2024-04-23T14:57:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.804+0000 7f8598e00640 1 -- 172.21.15.28:0/313901293 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85941a4150 con 0x7f85941019b0 2024-04-23T14:57:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.804+0000 7f858aa00640 1 -- 172.21.15.28:0/313901293 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f857c035030 con 0x7f85941019b0 2024-04-23T14:57:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.804+0000 7f8598e00640 1 -- 172.21.15.28:0/313901293 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85941aba40 con 0x7f85941019b0 2024-04-23T14:57:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.806+0000 7f8598e00640 1 -- 172.21.15.28:0/313901293 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8594069270 con 0x7f85941019b0 2024-04-23T14:57:49.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.808+0000 7f858aa00640 1 -- 172.21.15.28:0/313901293 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f857c042070 con 0x7f85941019b0 2024-04-23T14:57:49.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.809+0000 7f858aa00640 1 --2- 172.21.15.28:0/313901293 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f85780a3da0 0x7f85780a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:49.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.809+0000 7f858aa00640 1 -- 172.21.15.28:0/313901293 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8542+0+0 (secure 0 0 0) 0x7f857c0f0d70 con 0x7f85941019b0 2024-04-23T14:57:49.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.809+0000 7f8591e00640 1 --2- 172.21.15.28:0/313901293 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f85780a3da0 0x7f85780a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:49.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.810+0000 7f8591e00640 1 --2- 172.21.15.28:0/313901293 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f85780a3da0 0x7f85780a6260 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8584009bd0 tx=0x7f8584005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:49.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:49.816+0000 7f858aa00640 1 -- 172.21.15.28:0/313901293 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f857c0bce10 con 0x7f85941019b0 2024-04-23T14:57:49.822 INFO:tasks.workunit.client.0.smithi028.stdout: 10 2730 64.38 MB/sec warmup 5 sec latency 88.971 ms 2024-04-23T14:57:50.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.128+0000 7f8598e00640 1 -- 172.21.15.28:0/313901293 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f85941abc10 con 0x7f85941019b0 2024-04-23T14:57:50.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.129+0000 7f858aa00640 1 -- 172.21.15.28:0/313901293 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7f857c0c1110 con 0x7f85941019b0 2024-04-23T14:57:50.130 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:57:50.131 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:57:50.131 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:57:50.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.132+0000 7f8598e00640 1 -- 172.21.15.28:0/313901293 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f85780a3da0 msgr2=0x7f85780a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:50.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.132+0000 7f8598e00640 1 --2- 172.21.15.28:0/313901293 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f85780a3da0 0x7f85780a6260 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8584009bd0 tx=0x7f8584005f90 comp rx=0 tx=0).stop 2024-04-23T14:57:50.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.132+0000 7f8598e00640 1 -- 172.21.15.28:0/313901293 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85941019b0 msgr2=0x7f859419f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:50.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.132+0000 7f8598e00640 1 --2- 172.21.15.28:0/313901293 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85941019b0 0x7f859419f410 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f857c004770 tx=0x7f857c003c10 comp rx=0 tx=0).stop 2024-04-23T14:57:50.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.132+0000 7f8598e00640 1 -- 172.21.15.28:0/313901293 shutdown_connections 2024-04-23T14:57:50.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.132+0000 7f8598e00640 1 --2- 172.21.15.28:0/313901293 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f85780a3da0 0x7f85780a6260 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:50.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.132+0000 7f8598e00640 1 --2- 172.21.15.28:0/313901293 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85941019b0 0x7f859419f410 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:50.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.132+0000 7f8598e00640 1 --2- 172.21.15.28:0/313901293 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f85941063b0 0x7f85941a3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:50.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.132+0000 7f8598e00640 1 --2- 172.21.15.28:0/313901293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8594101090 0x7f859419eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:50.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.132+0000 7f8598e00640 1 -- 172.21.15.28:0/313901293 >> 172.21.15.28:0/313901293 conn(0x7f8594078000 msgr2=0x7f8594103e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:50.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.133+0000 7f8598e00640 1 -- 172.21.15.28:0/313901293 shutdown_connections 2024-04-23T14:57:50.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.133+0000 7f8598e00640 1 -- 172.21.15.28:0/313901293 wait complete. 2024-04-23T14:57:50.154 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-23T14:57:50.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.395+0000 7f5910c00640 1 -- 172.21.15.28:0/1120653610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f590c1003b0 msgr2=0x7f590c100790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:50.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.395+0000 7f5910c00640 1 --2- 172.21.15.28:0/1120653610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f590c1003b0 0x7f590c100790 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f58fc009930 tx=0x7f58fc02a420 comp rx=0 tx=0).stop 2024-04-23T14:57:50.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.395+0000 7f5910c00640 1 -- 172.21.15.28:0/1120653610 shutdown_connections 2024-04-23T14:57:50.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.395+0000 7f5910c00640 1 --2- 172.21.15.28:0/1120653610 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f590c107470 0x7f590c10ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:50.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.395+0000 7f5910c00640 1 --2- 172.21.15.28:0/1120653610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f590c100d60 0x7f590c106d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:50.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.395+0000 7f5910c00640 1 --2- 172.21.15.28:0/1120653610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f590c1003b0 0x7f590c100790 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:50.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.395+0000 7f5910c00640 1 -- 172.21.15.28:0/1120653610 >> 172.21.15.28:0/1120653610 conn(0x7f590c077060 msgr2=0x7f590c0fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:50.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.395+0000 7f5910c00640 1 -- 172.21.15.28:0/1120653610 shutdown_connections 2024-04-23T14:57:50.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.395+0000 7f5910c00640 1 -- 172.21.15.28:0/1120653610 wait complete. 2024-04-23T14:57:50.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.396+0000 7f5910c00640 1 Processor -- start 2024-04-23T14:57:50.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.396+0000 7f5910c00640 1 -- start start 2024-04-23T14:57:50.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.397+0000 7f5910c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f590c1003b0 0x7f590c19e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:50.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.397+0000 7f5910c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f590c107470 0x7f590c19ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:50.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.397+0000 7f5910c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f590c198860 0x7f590c198d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:50.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.397+0000 7f5910c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f590c1124a0 con 0x7f590c1003b0 2024-04-23T14:57:50.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.397+0000 7f5910c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f590c112320 con 0x7f590c107470 2024-04-23T14:57:50.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.397+0000 7f5910c00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f590c112620 con 0x7f590c198860 2024-04-23T14:57:50.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.397+0000 7f5909e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f590c1003b0 0x7f590c19e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:50.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.397+0000 7f5909e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f590c1003b0 0x7f590c19e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36960/0 (socket says 172.21.15.28:36960) 2024-04-23T14:57:50.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.397+0000 7f5909e00640 1 -- 172.21.15.28:0/1019201335 learned_addr learned my addr 172.21.15.28:0/1019201335 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:50.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.397+0000 7f5909400640 1 --2- 172.21.15.28:0/1019201335 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f590c107470 0x7f590c19ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:50.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.398+0000 7f590a800640 1 --2- 172.21.15.28:0/1019201335 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f590c198860 0x7f590c198d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:50.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.398+0000 7f5909e00640 1 -- 172.21.15.28:0/1019201335 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f590c198860 msgr2=0x7f590c198d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:50.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.398+0000 7f5909e00640 1 --2- 172.21.15.28:0/1019201335 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f590c198860 0x7f590c198d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:50.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.398+0000 7f5909e00640 1 -- 172.21.15.28:0/1019201335 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f590c107470 msgr2=0x7f590c19ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:50.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.398+0000 7f5909e00640 1 --2- 172.21.15.28:0/1019201335 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f590c107470 0x7f590c19ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:50.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.398+0000 7f5909e00640 1 -- 172.21.15.28:0/1019201335 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f590c199380 con 0x7f590c1003b0 2024-04-23T14:57:50.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.398+0000 7f5909400640 1 --2- 172.21.15.28:0/1019201335 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f590c107470 0x7f590c19ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:57:50.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.398+0000 7f5909e00640 1 --2- 172.21.15.28:0/1019201335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f590c1003b0 0x7f590c19e6b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f58fc009230 tx=0x7f58fc004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:50.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.399+0000 7f5902a00640 1 -- 172.21.15.28:0/1019201335 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58fc03f030 con 0x7f590c1003b0 2024-04-23T14:57:50.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.399+0000 7f5902a00640 1 -- 172.21.15.28:0/1019201335 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58fc004440 con 0x7f590c1003b0 2024-04-23T14:57:50.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.399+0000 7f5910c00640 1 -- 172.21.15.28:0/1019201335 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f590c199610 con 0x7f590c1003b0 2024-04-23T14:57:50.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.399+0000 7f5902a00640 1 -- 172.21.15.28:0/1019201335 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58fc035070 con 0x7f590c1003b0 2024-04-23T14:57:50.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.399+0000 7f5910c00640 1 -- 172.21.15.28:0/1019201335 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f590c1a5800 con 0x7f590c1003b0 2024-04-23T14:57:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.403+0000 7f5913a1e740 1 Processor -- start 2024-04-23T14:57:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.404+0000 7f5913a1e740 1 -- start start 2024-04-23T14:57:50.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.404+0000 7f5913a1e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621bcdf8d60 0x5621bcdf9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:50.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.404+0000 7f5913a1e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621bcdf9680 0x5621bce03810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.404+0000 7f5913a1e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621bce03d50 0x5621bce06140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.404+0000 7f5913a1e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5621bcd5a1d0 con 0x5621bcdf8d60 2024-04-23T14:57:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.404+0000 7f5913a1e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5621bcd00450 con 0x5621bcdf9680 2024-04-23T14:57:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.404+0000 7f5913a1e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5621bcd56190 con 0x5621bce03d50 2024-04-23T14:57:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.404+0000 7f5902a00640 1 -- 172.21.15.28:0/1019201335 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f58fc042030 con 0x7f590c1003b0 2024-04-23T14:57:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.405+0000 7f5909400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621bce03d50 0x5621bce06140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.405+0000 7f590a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621bcdf8d60 0x5621bcdf9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.405+0000 7f5909400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621bce03d50 0x5621bce06140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:34524/0 (socket says 172.21.15.28:34524) 2024-04-23T14:57:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.405+0000 7f5909400640 1 -- 172.21.15.28:0/1609574076 learned_addr learned my addr 172.21.15.28:0/1609574076 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.405+0000 7f5909e00640 1 --2- 172.21.15.28:0/1609574076 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621bcdf9680 0x5621bce03810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.405+0000 7f5909400640 1 -- 172.21.15.28:0/1609574076 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621bcdf9680 msgr2=0x5621bce03810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.405+0000 7f5909400640 1 --2- 172.21.15.28:0/1609574076 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621bcdf9680 0x5621bce03810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.405+0000 7f5909400640 1 -- 172.21.15.28:0/1609574076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621bcdf8d60 msgr2=0x5621bcdf9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.405+0000 7f5909400640 1 --2- 172.21.15.28:0/1609574076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621bcdf8d60 0x5621bcdf9140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:50.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.405+0000 7f5909400640 1 -- 172.21.15.28:0/1609574076 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621bcdbd310 con 0x5621bce03d50 2024-04-23T14:57:50.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.406+0000 7f5909400640 1 --2- 172.21.15.28:0/1609574076 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621bce03d50 0x5621bce06140 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f58f8011470 tx=0x7f58f8011930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af532c6cbdc91de4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:50.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.406+0000 7f58f7000640 1 -- 172.21.15.28:0/1609574076 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f801e070 con 0x5621bce03d50 2024-04-23T14:57:50.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.406+0000 7f58f7000640 1 -- 172.21.15.28:0/1609574076 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58f800fae0 con 0x5621bce03d50 2024-04-23T14:57:50.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.406+0000 7f58f7000640 1 -- 172.21.15.28:0/1609574076 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f8019bb0 con 0x5621bce03d50 2024-04-23T14:57:50.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.406+0000 7f5902a00640 1 --2- 172.21.15.28:0/1019201335 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f58f00a3d20 0x7f58f00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:50.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.406+0000 7f5902a00640 1 -- 172.21.15.28:0/1019201335 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8542+0+0 (secure 0 0 0) 0x7f58fc0f07f0 con 0x7f590c1003b0 2024-04-23T14:57:50.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.407+0000 7f590a800640 1 --2- 172.21.15.28:0/1019201335 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f58f00a3d20 0x7f58f00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:50.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.407+0000 7f5913a1e740 1 -- 172.21.15.28:0/1609574076 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621bce03d50 msgr2=0x5621bce06140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:50.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.407+0000 7f5913a1e740 1 --2- 172.21.15.28:0/1609574076 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621bce03d50 0x5621bce06140 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f58f8011470 tx=0x7f58f8011930 comp rx=0 tx=0).stop 2024-04-23T14:57:50.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.407+0000 7f5913a1e740 1 -- 172.21.15.28:0/1609574076 shutdown_connections 2024-04-23T14:57:50.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.407+0000 7f5913a1e740 1 --2- 172.21.15.28:0/1609574076 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621bcdf9680 0x5621bce03810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:50.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.407+0000 7f5913a1e740 1 --2- 172.21.15.28:0/1609574076 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621bce03d50 0x5621bce06140 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:50.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.407+0000 7f5913a1e740 1 --2- 172.21.15.28:0/1609574076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621bcdf8d60 0x5621bcdf9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:50.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.407+0000 7f5913a1e740 1 -- 172.21.15.28:0/1609574076 >> 172.21.15.28:0/1609574076 conn(0x5621bcdf5080 msgr2=0x5621bcdf5460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:50.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.407+0000 7f5913a1e740 1 -- 172.21.15.28:0/1609574076 shutdown_connections 2024-04-23T14:57:50.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.407+0000 7f590a800640 1 --2- 172.21.15.28:0/1019201335 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f58f00a3d20 0x7f58f00a61e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5904005fd0 tx=0x7f5904014040 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:50.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.408+0000 7f5913a1e740 1 -- 172.21.15.28:0/1609574076 wait complete. 2024-04-23T14:57:50.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.408+0000 7f5913a1e740 1 -- start start 2024-04-23T14:57:50.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.409+0000 7f5913a1e740 1 Processor -- start 2024-04-23T14:57:50.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.409+0000 7f5913a1e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621bcdf8d60 0x5621bce037b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:50.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.409+0000 7f5913a1e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621bcdf9680 0x5621bce00aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:50.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.409+0000 7f5913a1e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621bce03d50 0x5621bce011c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:50.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.409+0000 7f5913a1e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5621bce06be0 con 0x5621bce03d50 2024-04-23T14:57:50.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.409+0000 7f5913a1e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5621bcd13810 con 0x5621bcdf8d60 2024-04-23T14:57:50.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.409+0000 7f5913a1e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5621bce06d60 con 0x5621bcdf9680 2024-04-23T14:57:50.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.410+0000 7f5909e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621bcdf8d60 0x5621bce037b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:50.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.410+0000 7f5913a1e740 10 client.0 _ll_register_callbacks cb 0x5621bcce1140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:57:50.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.410+0000 7f5909e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621bcdf8d60 0x5621bce037b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:41954/0 (socket says 172.21.15.28:41954) 2024-04-23T14:57:50.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.410+0000 7f5909e00640 1 -- 172.21.15.28:0/3729534162 learned_addr learned my addr 172.21.15.28:0/3729534162 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:50.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.410+0000 7f5909400640 1 --2- 172.21.15.28:0/3729534162 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621bcdf9680 0x5621bce00aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:50.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.410+0000 7f590a800640 1 --2- 172.21.15.28:0/3729534162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621bce03d50 0x5621bce011c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:50.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.410+0000 7f5909e00640 1 -- 172.21.15.28:0/3729534162 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621bcdf9680 msgr2=0x5621bce00aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:50.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.410+0000 7f5909e00640 1 --2- 172.21.15.28:0/3729534162 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621bcdf9680 0x5621bce00aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:50.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.410+0000 7f5909e00640 1 -- 172.21.15.28:0/3729534162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621bce03d50 msgr2=0x5621bce011c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:50.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.410+0000 7f5909e00640 1 --2- 172.21.15.28:0/3729534162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621bce03d50 0x5621bce011c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:50.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.410+0000 7f5909e00640 1 -- 172.21.15.28:0/3729534162 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621bcd243f0 con 0x5621bcdf8d60 2024-04-23T14:57:50.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.411+0000 7f5909e00640 1 --2- 172.21.15.28:0/3729534162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621bcdf8d60 0x5621bce037b0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f58fc0edf30 tx=0x7f58fc0b8b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ec73a85d88f40a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:50.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.411+0000 7f58eb400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:57:50.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.411+0000 7f58eb400640 1 -- 172.21.15.28:0/3729534162 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58fc049050 con 0x5621bcdf8d60 2024-04-23T14:57:50.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.411+0000 7f58eb400640 1 -- 172.21.15.28:0/3729534162 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8542+0+0 (secure 0 0 0) 0x7f58fc0ee150 con 0x5621bcdf8d60 2024-04-23T14:57:50.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.411+0000 7f5913a1e740 1 -- 172.21.15.28:0/3729534162 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58f800fae0 con 0x5621bcdf8d60 2024-04-23T14:57:50.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.411+0000 7f58eb400640 1 -- 172.21.15.28:0/3729534162 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f58fc0ee350 con 0x5621bcdf8d60 2024-04-23T14:57:50.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.411+0000 7f5913a1e740 10 client.27365 fetch_fsmap learned FSMap version 20 2024-04-23T14:57:50.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.411+0000 7f5913a1e740 1 -- 172.21.15.28:0/3729534162 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5621bca6c3f0 con 0x5621bcdf8d60 2024-04-23T14:57:50.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.411+0000 7f58eb400640 1 -- 172.21.15.28:0/3729534162 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58f800fae0 con 0x5621bcdf8d60 2024-04-23T14:57:50.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.412+0000 7f58eb400640 1 -- 172.21.15.28:0/3729534162 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f58fc0f0be0 con 0x5621bcdf8d60 2024-04-23T14:57:50.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.412+0000 7f5913a1e740 10 client.27365 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:57:50.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.412+0000 7f5913a1e740 10 client.27365 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:57:50.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.412+0000 7f5913a1e740 20 client.27365 populate_metadata read hostname 'smithi028' 2024-04-23T14:57:50.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.412+0000 7f5913a1e740 1 --2- 172.21.15.28:0/3729534162 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5621bce13ea0 0x5621bce162b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:50.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.412+0000 7f5913a1e740 4 client.27365 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:57:50.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.412+0000 7f5913a1e740 1 -- 172.21.15.28:0/3729534162 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5621bcd67bf0 con 0x5621bce13ea0 2024-04-23T14:57:50.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.412+0000 7f5909400640 1 --2- 172.21.15.28:0/3729534162 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5621bce13ea0 0x5621bce162b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:50.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.413+0000 7f5909400640 1 --2- 172.21.15.28:0/3729534162 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5621bce13ea0 0x5621bce162b0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3be2673a2a6ed09 server_cookie=b4208150b3949a5f in_seq=0 out_seq=0 2024-04-23T14:57:50.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.413+0000 7f58eb400640 10 client.27365 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:57:50.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.416+0000 7f58eb400640 1 -- 172.21.15.28:0/3729534162 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5621bcd67bf0 con 0x5621bce13ea0 2024-04-23T14:57:50.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.416+0000 7f58eb400640 10 client.27365 handle_command_reply: tid=0 2024-04-23T14:57:50.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.416+0000 7f5913a1e740 1 client.27365 shutdown 2024-04-23T14:57:50.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.416+0000 7f5913a1e740 1 -- 172.21.15.28:0/3729534162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621bcdf8d60 msgr2=0x5621bce037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:50.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.416+0000 7f5913a1e740 1 --2- 172.21.15.28:0/3729534162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621bcdf8d60 0x5621bce037b0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f58fc0edf30 tx=0x7f58fc0b8b30 comp rx=0 tx=0).stop 2024-04-23T14:57:50.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.416+0000 7f5913a1e740 1 -- 172.21.15.28:0/3729534162 shutdown_connections 2024-04-23T14:57:50.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.416+0000 7f5913a1e740 1 --2- 172.21.15.28:0/3729534162 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5621bce13ea0 0x5621bce162b0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:50.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.416+0000 7f5913a1e740 1 --2- 172.21.15.28:0/3729534162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621bce03d50 0x5621bce011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:50.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.417+0000 7f5913a1e740 1 --2- 172.21.15.28:0/3729534162 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621bcdf9680 0x5621bce00aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:50.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.417+0000 7f5913a1e740 1 --2- 172.21.15.28:0/3729534162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621bcdf8d60 0x5621bce037b0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:50.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.417+0000 7f5913a1e740 1 -- 172.21.15.28:0/3729534162 >> 172.21.15.28:0/3729534162 conn(0x5621bcdf5080 msgr2=0x5621bcdefdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:50.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.417+0000 7f58eb400640 0 client.27365 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:57:50.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.417+0000 7f5913a1e740 1 -- 172.21.15.28:0/3729534162 shutdown_connections 2024-04-23T14:57:50.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.417+0000 7f5913a1e740 1 -- 172.21.15.28:0/3729534162 wait complete. 2024-04-23T14:57:50.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.417+0000 7f5913a1e740 20 client.27365 trim_cache size 0 max 16384 2024-04-23T14:57:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.434+0000 7f5913a1e740 1 Processor -- start 2024-04-23T14:57:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.434+0000 7f5913a1e740 1 -- start start 2024-04-23T14:57:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.434+0000 7f5913a1e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621bcdf8d60 0x5621bce0fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.434+0000 7f5913a1e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621bcdf9680 0x5621bcdffa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.434+0000 7f5913a1e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621bce03d50 0x5621bce032e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.434+0000 7f5913a1e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f590c1a9b80 con 0x5621bcdf9680 2024-04-23T14:57:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.434+0000 7f5913a1e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f590c1ac3c0 con 0x5621bce03d50 2024-04-23T14:57:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.434+0000 7f5913a1e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f590c1aa1d0 con 0x5621bcdf8d60 2024-04-23T14:57:50.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.435+0000 7f590a800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621bce03d50 0x5621bce032e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:50.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.435+0000 7f5909400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621bcdf9680 0x5621bcdffa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:50.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.435+0000 7f590a800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621bce03d50 0x5621bce032e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:41956/0 (socket says 172.21.15.28:41956) 2024-04-23T14:57:50.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.435+0000 7f590a800640 1 -- 172.21.15.28:0/3921640137 learned_addr learned my addr 172.21.15.28:0/3921640137 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:50.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.435+0000 7f5909e00640 1 --2- 172.21.15.28:0/3921640137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621bcdf8d60 0x5621bce0fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.435+0000 7f590a800640 1 -- 172.21.15.28:0/3921640137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621bcdf8d60 msgr2=0x5621bce0fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.435+0000 7f590a800640 1 --2- 172.21.15.28:0/3921640137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621bcdf8d60 0x5621bce0fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.435+0000 7f590a800640 1 -- 172.21.15.28:0/3921640137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621bcdf9680 msgr2=0x5621bcdffa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.435+0000 7f590a800640 1 --2- 172.21.15.28:0/3921640137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621bcdf9680 0x5621bcdffa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.435+0000 7f590a800640 1 -- 172.21.15.28:0/3921640137 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621bcdec450 con 0x5621bce03d50 2024-04-23T14:57:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.435+0000 7f5909400640 1 --2- 172.21.15.28:0/3921640137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621bcdf9680 0x5621bcdffa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:57:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.435+0000 7f5909e00640 1 --2- 172.21.15.28:0/3921640137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621bcdf8d60 0x5621bce0fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:57:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.436+0000 7f590a800640 1 --2- 172.21.15.28:0/3921640137 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621bce03d50 0x5621bce032e0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f5904012dc0 tx=0x7f5904016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f972931a3c68e72b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.436+0000 7f58f6600640 1 -- 172.21.15.28:0/3921640137 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f590402d070 con 0x5621bce03d50 2024-04-23T14:57:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.436+0000 7f58f6600640 1 -- 172.21.15.28:0/3921640137 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5904004680 con 0x5621bce03d50 2024-04-23T14:57:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.436+0000 7f58f6600640 1 -- 172.21.15.28:0/3921640137 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f590400f7b0 con 0x5621bce03d50 2024-04-23T14:57:50.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.437+0000 7f5913a1e740 1 -- 172.21.15.28:0/3921640137 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621bce03d50 msgr2=0x5621bce032e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:50.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.437+0000 7f5913a1e740 1 --2- 172.21.15.28:0/3921640137 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621bce03d50 0x5621bce032e0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f5904012dc0 tx=0x7f5904016000 comp rx=0 tx=0).stop 2024-04-23T14:57:50.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.437+0000 7f5913a1e740 1 -- 172.21.15.28:0/3921640137 shutdown_connections 2024-04-23T14:57:50.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.437+0000 7f5913a1e740 1 --2- 172.21.15.28:0/3921640137 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621bce03d50 0x5621bce032e0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:50.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.437+0000 7f5913a1e740 1 --2- 172.21.15.28:0/3921640137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621bcdf9680 0x5621bcdffa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:50.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.437+0000 7f5913a1e740 1 --2- 172.21.15.28:0/3921640137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621bcdf8d60 0x5621bce0fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:50.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.437+0000 7f5913a1e740 1 -- 172.21.15.28:0/3921640137 >> 172.21.15.28:0/3921640137 conn(0x5621bcdf5080 msgr2=0x5621bcdff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:50.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.437+0000 7f5913a1e740 1 -- 172.21.15.28:0/3921640137 shutdown_connections 2024-04-23T14:57:50.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.437+0000 7f5913a1e740 1 -- 172.21.15.28:0/3921640137 wait complete. 2024-04-23T14:57:50.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.437+0000 7f5913a1e740 1 -- start start 2024-04-23T14:57:50.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.438+0000 7f5913a1e740 1 Processor -- start 2024-04-23T14:57:50.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.438+0000 7f5913a1e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621bcdf8d60 0x5621bce0c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:50.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.438+0000 7f5913a1e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621bcdf9680 0x5621bce04730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:50.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.438+0000 7f5913a1e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621bce03d50 0x5621bce04c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:50.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.438+0000 7f5913a1e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f590c1aa880 con 0x5621bce03d50 2024-04-23T14:57:50.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.438+0000 7f5913a1e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f590c1abcf0 con 0x5621bcdf9680 2024-04-23T14:57:50.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.438+0000 7f5913a1e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f590c1aaf50 con 0x5621bcdf8d60 2024-04-23T14:57:50.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.438+0000 7f5913a1e740 10 client.0 _ll_register_callbacks cb 0x7f58fc009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:57:50.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.438+0000 7f5909400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621bcdf9680 0x5621bce04730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:50.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.438+0000 7f590a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621bce03d50 0x5621bce04c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:50.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.438+0000 7f5909400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621bcdf9680 0x5621bce04730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:41958/0 (socket says 172.21.15.28:41958) 2024-04-23T14:57:50.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.438+0000 7f5909400640 1 -- 172.21.15.28:0/2065067513 learned_addr learned my addr 172.21.15.28:0/2065067513 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:50.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.439+0000 7f5909e00640 1 --2- 172.21.15.28:0/2065067513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621bcdf8d60 0x5621bce0c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:50.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.439+0000 7f5909400640 1 -- 172.21.15.28:0/2065067513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621bcdf8d60 msgr2=0x5621bce0c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:50.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.439+0000 7f5909400640 1 --2- 172.21.15.28:0/2065067513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621bcdf8d60 0x5621bce0c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:50.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.439+0000 7f5909400640 1 -- 172.21.15.28:0/2065067513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621bce03d50 msgr2=0x5621bce04c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:50.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.439+0000 7f5909400640 1 --2- 172.21.15.28:0/2065067513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621bce03d50 0x5621bce04c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:50.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.439+0000 7f5909400640 1 -- 172.21.15.28:0/2065067513 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621bcb367d0 con 0x5621bcdf9680 2024-04-23T14:57:50.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.439+0000 7f5909400640 1 --2- 172.21.15.28:0/2065067513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621bcdf9680 0x5621bce04730 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f58f8009230 tx=0x7f58f8005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b7d5b0d17d6a571 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.439+0000 7f58eb400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:57:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.439+0000 7f58eb400640 1 -- 172.21.15.28:0/2065067513 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f801e040 con 0x5621bcdf9680 2024-04-23T14:57:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.439+0000 7f5913a1e740 1 -- 172.21.15.28:0/2065067513 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5904004680 con 0x5621bcdf9680 2024-04-23T14:57:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.439+0000 7f58eb400640 1 -- 172.21.15.28:0/2065067513 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8542+0+0 (secure 0 0 0) 0x7f58f8005270 con 0x5621bcdf9680 2024-04-23T14:57:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.440+0000 7f58eb400640 1 -- 172.21.15.28:0/2065067513 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f58f8005470 con 0x5621bcdf9680 2024-04-23T14:57:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.440+0000 7f5913a1e740 10 client.27371 fetch_fsmap learned FSMap version 20 2024-04-23T14:57:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.440+0000 7f5913a1e740 1 -- 172.21.15.28:0/2065067513 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x5621bce0bcf0 con 0x5621bcdf9680 2024-04-23T14:57:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.440+0000 7f58eb400640 1 -- 172.21.15.28:0/2065067513 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5904004680 con 0x5621bcdf9680 2024-04-23T14:57:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.440+0000 7f58eb400640 1 -- 172.21.15.28:0/2065067513 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f58f80039f0 con 0x5621bcdf9680 2024-04-23T14:57:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.440+0000 7f5913a1e740 10 client.27371 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:57:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.440+0000 7f5913a1e740 10 client.27371 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:57:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.440+0000 7f5913a1e740 20 client.27371 populate_metadata read hostname 'smithi028' 2024-04-23T14:57:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.440+0000 7f5913a1e740 1 --2- 172.21.15.28:0/2065067513 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5621bce7a650 0x5621bce7ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.440+0000 7f5913a1e740 4 client.27371 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-23T14:57:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.440+0000 7f5913a1e740 1 -- 172.21.15.28:0/2065067513 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5621bcd61a60 con 0x5621bce7a650 2024-04-23T14:57:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.440+0000 7f5909e00640 1 --2- 172.21.15.28:0/2065067513 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5621bce7a650 0x5621bce7ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.441+0000 7f5909e00640 1 --2- 172.21.15.28:0/2065067513 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5621bce7a650 0x5621bce7ca70 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d84704775bd3df61 server_cookie=6a31ab8b9d25adff in_seq=0 out_seq=0 2024-04-23T14:57:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.441+0000 7f58eb400640 10 client.27371 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:57:50.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.443+0000 7f58eb400640 1 -- 172.21.15.28:0/2065067513 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5621bcd61a60 con 0x5621bce7a650 2024-04-23T14:57:50.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.443+0000 7f58eb400640 10 client.27371 handle_command_reply: tid=0 2024-04-23T14:57:50.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.443+0000 7f5913a1e740 1 client.27371 shutdown 2024-04-23T14:57:50.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.444+0000 7f5913a1e740 1 -- 172.21.15.28:0/2065067513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621bcdf9680 msgr2=0x5621bce04730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:50.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.444+0000 7f5913a1e740 1 --2- 172.21.15.28:0/2065067513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621bcdf9680 0x5621bce04730 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f58f8009230 tx=0x7f58f8005850 comp rx=0 tx=0).stop 2024-04-23T14:57:50.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.444+0000 7f5913a1e740 1 -- 172.21.15.28:0/2065067513 shutdown_connections 2024-04-23T14:57:50.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.444+0000 7f5913a1e740 1 --2- 172.21.15.28:0/2065067513 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x5621bce7a650 0x5621bce7ca70 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:50.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.444+0000 7f5913a1e740 1 --2- 172.21.15.28:0/2065067513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621bcdf9680 0x5621bce04730 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:50.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.444+0000 7f58eb400640 0 client.27371 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:57:50.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.444+0000 7f5913a1e740 1 --2- 172.21.15.28:0/2065067513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621bce03d50 0x5621bce04c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:50.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.444+0000 7f5913a1e740 1 --2- 172.21.15.28:0/2065067513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5621bcdf8d60 0x5621bce0c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:50.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.444+0000 7f5913a1e740 1 -- 172.21.15.28:0/2065067513 >> 172.21.15.28:0/2065067513 conn(0x5621bcdf5080 msgr2=0x5621bcdfa4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:50.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.444+0000 7f5913a1e740 1 -- 172.21.15.28:0/2065067513 shutdown_connections 2024-04-23T14:57:50.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.444+0000 7f5913a1e740 1 -- 172.21.15.28:0/2065067513 wait complete. 2024-04-23T14:57:50.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.444+0000 7f5913a1e740 20 client.27371 trim_cache size 0 max 16384 2024-04-23T14:57:50.447 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-23T14:57:50.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.445+0000 7f5910c00640 1 -- 172.21.15.28:0/1019201335 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f58f00a3d20 msgr2=0x7f58f00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:50.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.445+0000 7f5910c00640 1 --2- 172.21.15.28:0/1019201335 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f58f00a3d20 0x7f58f00a61e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5904005fd0 tx=0x7f5904014040 comp rx=0 tx=0).stop 2024-04-23T14:57:50.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.445+0000 7f5910c00640 1 -- 172.21.15.28:0/1019201335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f590c1003b0 msgr2=0x7f590c19e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:50.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.445+0000 7f5910c00640 1 --2- 172.21.15.28:0/1019201335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f590c1003b0 0x7f590c19e6b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f58fc009230 tx=0x7f58fc004040 comp rx=0 tx=0).stop 2024-04-23T14:57:50.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.445+0000 7f5910c00640 1 -- 172.21.15.28:0/1019201335 shutdown_connections 2024-04-23T14:57:50.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.445+0000 7f5910c00640 1 --2- 172.21.15.28:0/1019201335 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f58f00a3d20 0x7f58f00a61e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:50.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.445+0000 7f5910c00640 1 --2- 172.21.15.28:0/1019201335 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f590c107470 0x7f590c19ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:50.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.445+0000 7f5910c00640 1 --2- 172.21.15.28:0/1019201335 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f590c198860 0x7f590c198d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:50.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.445+0000 7f5910c00640 1 --2- 172.21.15.28:0/1019201335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f590c1003b0 0x7f590c19e6b0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:50.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.445+0000 7f5910c00640 1 -- 172.21.15.28:0/1019201335 >> 172.21.15.28:0/1019201335 conn(0x7f590c077060 msgr2=0x7f590c105750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:50.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.445+0000 7f5910c00640 1 -- 172.21.15.28:0/1019201335 shutdown_connections 2024-04-23T14:57:50.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:50.445+0000 7f5910c00640 1 -- 172.21.15.28:0/1019201335 wait complete. 2024-04-23T14:57:50.822 INFO:tasks.workunit.client.0.smithi028.stdout: 10 3381 62.14 MB/sec warmup 6 sec latency 193.101 ms 2024-04-23T14:57:51.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:50 smithi038 ceph-mon[32348]: pgmap v431: 97 pgs: 97 active+clean; 616 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.4 MiB/s wr, 170 op/s 2024-04-23T14:57:51.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:50 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/313901293' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:57:51.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:50 smithi028 ceph-mon[27810]: pgmap v431: 97 pgs: 97 active+clean; 616 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.4 MiB/s wr, 170 op/s 2024-04-23T14:57:51.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:50 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/313901293' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:57:51.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:50 smithi105 ceph-mon[32555]: pgmap v431: 97 pgs: 97 active+clean; 616 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.4 MiB/s wr, 170 op/s 2024-04-23T14:57:51.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:50 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/313901293' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:57:51.465 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-23T14:57:51.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:57:51.465 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-23T14:57:51.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.694+0000 7f6603200640 1 -- 172.21.15.28:0/92236658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6604103980 msgr2=0x7f660410be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:51.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.694+0000 7f6603200640 1 --2- 172.21.15.28:0/92236658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6604103980 0x7f660410be60 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f65f4009be0 tx=0x7f65f402a750 comp rx=0 tx=0).stop 2024-04-23T14:57:51.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.694+0000 7f6603200640 1 -- 172.21.15.28:0/92236658 shutdown_connections 2024-04-23T14:57:51.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.694+0000 7f6603200640 1 --2- 172.21.15.28:0/92236658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f660410c480 0x7f660410e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:51.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.694+0000 7f6603200640 1 --2- 172.21.15.28:0/92236658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6604103980 0x7f660410be60 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:51.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.694+0000 7f6603200640 1 --2- 172.21.15.28:0/92236658 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6604103060 0x7f6604103440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:51.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.694+0000 7f6603200640 1 -- 172.21.15.28:0/92236658 >> 172.21.15.28:0/92236658 conn(0x7f66040fc880 msgr2=0x7f66040feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:51.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.694+0000 7f6603200640 1 -- 172.21.15.28:0/92236658 shutdown_connections 2024-04-23T14:57:51.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.695+0000 7f6603200640 1 -- 172.21.15.28:0/92236658 wait complete. 2024-04-23T14:57:51.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.696+0000 7f6603200640 1 Processor -- start 2024-04-23T14:57:51.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.696+0000 7f6603200640 1 -- start start 2024-04-23T14:57:51.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.696+0000 7f6603200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6604103060 0x7f66041a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:51.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.696+0000 7f6603200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6604103980 0x7f66041a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:51.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.696+0000 7f6603200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f660410c480 0x7f660419a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:51.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.696+0000 7f6603200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6604112240 con 0x7f660410c480 2024-04-23T14:57:51.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.696+0000 7f6603200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f66041120c0 con 0x7f6604103980 2024-04-23T14:57:51.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.696+0000 7f6603200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f66041123c0 con 0x7f6604103060 2024-04-23T14:57:51.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.696+0000 7f65f9400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6604103980 0x7f66041a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:51.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.696+0000 7f65f9400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6604103980 0x7f66041a0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:41976/0 (socket says 172.21.15.28:41976) 2024-04-23T14:57:51.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.697+0000 7f65f9400640 1 -- 172.21.15.28:0/2427150168 learned_addr learned my addr 172.21.15.28:0/2427150168 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:51.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.697+0000 7f6601e00640 1 --2- 172.21.15.28:0/2427150168 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6604103060 0x7f66041a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:51.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.697+0000 7f6602800640 1 --2- 172.21.15.28:0/2427150168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f660410c480 0x7f660419a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:51.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.697+0000 7f65f9400640 1 -- 172.21.15.28:0/2427150168 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6604103060 msgr2=0x7f66041a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:51.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.697+0000 7f65f9400640 1 --2- 172.21.15.28:0/2427150168 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6604103060 0x7f66041a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:51.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.697+0000 7f65f9400640 1 -- 172.21.15.28:0/2427150168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f660410c480 msgr2=0x7f660419a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:51.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.697+0000 7f65f9400640 1 --2- 172.21.15.28:0/2427150168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f660410c480 0x7f660419a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:51.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.697+0000 7f65f9400640 1 -- 172.21.15.28:0/2427150168 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f660419aff0 con 0x7f6604103980 2024-04-23T14:57:51.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.697+0000 7f6602800640 1 --2- 172.21.15.28:0/2427150168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f660410c480 0x7f660419a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:57:51.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.697+0000 7f65f9400640 1 --2- 172.21.15.28:0/2427150168 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6604103980 0x7f66041a0bf0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f65f4034040 tx=0x7f65f40305d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:51.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.698+0000 7f65fb400640 1 -- 172.21.15.28:0/2427150168 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f4041020 con 0x7f6604103980 2024-04-23T14:57:51.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.698+0000 7f65fb400640 1 -- 172.21.15.28:0/2427150168 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65f4003e50 con 0x7f6604103980 2024-04-23T14:57:51.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.698+0000 7f65fb400640 1 -- 172.21.15.28:0/2427150168 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f4036040 con 0x7f6604103980 2024-04-23T14:57:51.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.698+0000 7f6603200640 1 -- 172.21.15.28:0/2427150168 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f660419b280 con 0x7f6604103980 2024-04-23T14:57:51.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.698+0000 7f6603200640 1 -- 172.21.15.28:0/2427150168 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66041a7500 con 0x7f6604103980 2024-04-23T14:57:51.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.701+0000 7f660bb9e740 1 Processor -- start 2024-04-23T14:57:51.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.701+0000 7f660bb9e740 1 -- start start 2024-04-23T14:57:51.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.701+0000 7f65fb400640 1 -- 172.21.15.28:0/2427150168 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f65f4044020 con 0x7f6604103980 2024-04-23T14:57:51.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.701+0000 7f660bb9e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de162b6270 0x55de162b6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:51.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.701+0000 7f660bb9e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de162b6b90 0x55de162c0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:51.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.702+0000 7f660bb9e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de162c1260 0x55de162c3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:51.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.702+0000 7f660bb9e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55de1621be70 con 0x55de162b6b90 2024-04-23T14:57:51.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.702+0000 7f660bb9e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55de161c2380 con 0x55de162c1260 2024-04-23T14:57:51.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.702+0000 7f660bb9e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55de16217e30 con 0x55de162b6270 2024-04-23T14:57:51.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.702+0000 7f6602800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de162b6270 0x55de162b6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:51.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.702+0000 7f6601e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de162b6b90 0x55de162c0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:51.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.702+0000 7f65f9400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de162c1260 0x55de162c3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:51.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.702+0000 7f6602800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de162b6270 0x55de162b6650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:34588/0 (socket says 172.21.15.28:34588) 2024-04-23T14:57:51.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.702+0000 7f6602800640 1 -- 172.21.15.28:0/4098647910 learned_addr learned my addr 172.21.15.28:0/4098647910 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:51.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.702+0000 7f6602800640 1 -- 172.21.15.28:0/4098647910 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de162c1260 msgr2=0x55de162c3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:51.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.702+0000 7f6602800640 1 --2- 172.21.15.28:0/4098647910 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de162c1260 0x55de162c3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:51.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.702+0000 7f6602800640 1 -- 172.21.15.28:0/4098647910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de162b6b90 msgr2=0x55de162c0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:51.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.702+0000 7f6602800640 1 --2- 172.21.15.28:0/4098647910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de162b6b90 0x55de162c0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:51.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.702+0000 7f6602800640 1 -- 172.21.15.28:0/4098647910 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de15ff67e0 con 0x55de162b6270 2024-04-23T14:57:51.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.703+0000 7f6602800640 1 --2- 172.21.15.28:0/4098647910 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de162b6270 0x55de162b6650 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f65fc009c10 tx=0x7f65fc0117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4451d2519e7b582 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:51.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.703+0000 7f65f3400640 1 -- 172.21.15.28:0/4098647910 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65fc012450 con 0x55de162b6270 2024-04-23T14:57:51.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.703+0000 7f65f3400640 1 -- 172.21.15.28:0/4098647910 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65fc012ad0 con 0x55de162b6270 2024-04-23T14:57:51.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.703+0000 7f65f3400640 1 -- 172.21.15.28:0/4098647910 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65fc00fc10 con 0x55de162b6270 2024-04-23T14:57:51.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.703+0000 7f65fb400640 1 --2- 172.21.15.28:0/2427150168 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f65e80a3d20 0x7f65e80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:51.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.703+0000 7f65fb400640 1 -- 172.21.15.28:0/2427150168 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8542+0+0 (secure 0 0 0) 0x7f65f40f1400 con 0x7f6604103980 2024-04-23T14:57:51.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.703+0000 7f6601e00640 1 --2- 172.21.15.28:0/2427150168 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f65e80a3d20 0x7f65e80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:51.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.704+0000 7f660bb9e740 1 -- 172.21.15.28:0/4098647910 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de162b6270 msgr2=0x55de162b6650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:51.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.704+0000 7f660bb9e740 1 --2- 172.21.15.28:0/4098647910 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de162b6270 0x55de162b6650 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f65fc009c10 tx=0x7f65fc0117b0 comp rx=0 tx=0).stop 2024-04-23T14:57:51.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.704+0000 7f660bb9e740 1 -- 172.21.15.28:0/4098647910 shutdown_connections 2024-04-23T14:57:51.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.704+0000 7f660bb9e740 1 --2- 172.21.15.28:0/4098647910 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de162c1260 0x55de162c3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:51.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.704+0000 7f660bb9e740 1 --2- 172.21.15.28:0/4098647910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de162b6b90 0x55de162c0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:51.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.704+0000 7f660bb9e740 1 --2- 172.21.15.28:0/4098647910 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de162b6270 0x55de162b6650 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:51.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.704+0000 7f660bb9e740 1 -- 172.21.15.28:0/4098647910 >> 172.21.15.28:0/4098647910 conn(0x55de162b2590 msgr2=0x55de162b2970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:51.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.704+0000 7f660bb9e740 1 -- 172.21.15.28:0/4098647910 shutdown_connections 2024-04-23T14:57:51.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.704+0000 7f6601e00640 1 --2- 172.21.15.28:0/2427150168 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f65e80a3d20 0x7f65e80a61e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x55de162b7830 tx=0x7f65ec009260 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:51.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.704+0000 7f660bb9e740 1 -- 172.21.15.28:0/4098647910 wait complete. 2024-04-23T14:57:51.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.705+0000 7f660bb9e740 1 -- start start 2024-04-23T14:57:51.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.705+0000 7f660bb9e740 1 Processor -- start 2024-04-23T14:57:51.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.705+0000 7f660bb9e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de162b6270 0x55de162be200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:51.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.705+0000 7f660bb9e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de162b6b90 0x55de162be740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:51.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.705+0000 7f660bb9e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de162c1260 0x55de162bec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:51.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.705+0000 7f660bb9e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55de162c4050 con 0x55de162b6b90 2024-04-23T14:57:51.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.705+0000 7f660bb9e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55de161d5660 con 0x55de162c1260 2024-04-23T14:57:51.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.705+0000 7f660bb9e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55de162c41d0 con 0x55de162b6270 2024-04-23T14:57:51.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.705+0000 7f660bb9e740 10 client.0 _ll_register_callbacks cb 0x55de161c3aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:57:51.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.706+0000 7f65f9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de162b6b90 0x55de162be740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:51.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.706+0000 7f6601e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de162b6270 0x55de162be200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:51.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.706+0000 7f6601e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de162b6270 0x55de162be200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:34604/0 (socket says 172.21.15.28:34604) 2024-04-23T14:57:51.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.706+0000 7f6601e00640 1 -- 172.21.15.28:0/3290558460 learned_addr learned my addr 172.21.15.28:0/3290558460 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:51.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.706+0000 7f6602800640 1 --2- 172.21.15.28:0/3290558460 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de162c1260 0x55de162bec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:51.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.706+0000 7f65f9400640 1 -- 172.21.15.28:0/3290558460 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de162b6270 msgr2=0x55de162be200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:51.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.706+0000 7f65f9400640 1 --2- 172.21.15.28:0/3290558460 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de162b6270 0x55de162be200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:51.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.706+0000 7f65f9400640 1 -- 172.21.15.28:0/3290558460 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de162c1260 msgr2=0x55de162bec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:51.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.706+0000 7f65f9400640 1 --2- 172.21.15.28:0/3290558460 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de162c1260 0x55de162bec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:51.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.706+0000 7f65f9400640 1 -- 172.21.15.28:0/3290558460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de16056f70 con 0x55de162b6b90 2024-04-23T14:57:51.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.706+0000 7f6601e00640 1 --2- 172.21.15.28:0/3290558460 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de162b6270 0x55de162be200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:57:51.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.706+0000 7f6602800640 1 --2- 172.21.15.28:0/3290558460 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de162c1260 0x55de162bec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:57:51.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.706+0000 7f65f9400640 1 --2- 172.21.15.28:0/3290558460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de162b6b90 0x55de162be740 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f65f40eeb10 tx=0x7f65f40f4b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37afe4311a9dc595 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:51.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.706+0000 7f65e7e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-23T14:57:51.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.707+0000 7f65e7e00640 1 -- 172.21.15.28:0/3290558460 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f40b9630 con 0x55de162b6b90 2024-04-23T14:57:51.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.707+0000 7f660bb9e740 1 -- 172.21.15.28:0/3290558460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65fc012ad0 con 0x55de162b6b90 2024-04-23T14:57:51.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.707+0000 7f65e7e00640 1 -- 172.21.15.28:0/3290558460 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8542+0+0 (secure 0 0 0) 0x7f65f40c5b70 con 0x55de162b6b90 2024-04-23T14:57:51.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.707+0000 7f65e7e00640 1 -- 172.21.15.28:0/3290558460 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f65f40bccf0 con 0x55de162b6b90 2024-04-23T14:57:51.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.707+0000 7f660bb9e740 10 client.17187 fetch_fsmap learned FSMap version 20 2024-04-23T14:57:51.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.707+0000 7f660bb9e740 1 -- 172.21.15.28:0/3290558460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55de15eee170 con 0x55de162b6b90 2024-04-23T14:57:51.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.707+0000 7f65e7e00640 1 -- 172.21.15.28:0/3290558460 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65fc012ad0 con 0x55de162b6b90 2024-04-23T14:57:51.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.707+0000 7f65e7e00640 1 -- 172.21.15.28:0/3290558460 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f65f4004b40 con 0x55de162b6b90 2024-04-23T14:57:51.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.707+0000 7f660bb9e740 10 client.17187 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:57:51.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.707+0000 7f660bb9e740 10 client.17187 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:57:51.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.708+0000 7f660bb9e740 20 client.17187 populate_metadata read hostname 'smithi028' 2024-04-23T14:57:51.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.708+0000 7f660bb9e740 1 --2- 172.21.15.28:0/3290558460 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55de162d14e0 0x55de162d3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:51.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.708+0000 7f660bb9e740 4 client.17187 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:57:51.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.708+0000 7f660bb9e740 1 -- 172.21.15.28:0/3290558460 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55de16229890 con 0x55de162d14e0 2024-04-23T14:57:51.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.708+0000 7f6601e00640 1 --2- 172.21.15.28:0/3290558460 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55de162d14e0 0x55de162d3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:51.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.708+0000 7f6601e00640 1 --2- 172.21.15.28:0/3290558460 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55de162d14e0 0x55de162d3930 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a25e3ad12770c22 server_cookie=561d767078212cf3 in_seq=0 out_seq=0 2024-04-23T14:57:51.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.708+0000 7f65e7e00640 10 client.17187 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:57:51.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.711+0000 7f65e7e00640 1 -- 172.21.15.28:0/3290558460 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55de16229890 con 0x55de162d14e0 2024-04-23T14:57:51.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.711+0000 7f65e7e00640 10 client.17187 handle_command_reply: tid=0 2024-04-23T14:57:51.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.711+0000 7f660bb9e740 1 client.17187 shutdown 2024-04-23T14:57:51.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.711+0000 7f660bb9e740 1 -- 172.21.15.28:0/3290558460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de162b6b90 msgr2=0x55de162be740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:51.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.711+0000 7f660bb9e740 1 --2- 172.21.15.28:0/3290558460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de162b6b90 0x55de162be740 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f65f40eeb10 tx=0x7f65f40f4b40 comp rx=0 tx=0).stop 2024-04-23T14:57:51.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.711+0000 7f660bb9e740 1 -- 172.21.15.28:0/3290558460 shutdown_connections 2024-04-23T14:57:51.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.711+0000 7f660bb9e740 1 --2- 172.21.15.28:0/3290558460 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55de162d14e0 0x55de162d3930 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:51.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.711+0000 7f660bb9e740 1 --2- 172.21.15.28:0/3290558460 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de162c1260 0x55de162bec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:51.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.711+0000 7f660bb9e740 1 --2- 172.21.15.28:0/3290558460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de162b6b90 0x55de162be740 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:51.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.711+0000 7f660bb9e740 1 --2- 172.21.15.28:0/3290558460 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de162b6270 0x55de162be200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:51.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.711+0000 7f65e7e00640 0 client.17187 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:57:51.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.711+0000 7f660bb9e740 1 -- 172.21.15.28:0/3290558460 >> 172.21.15.28:0/3290558460 conn(0x55de162b2590 msgr2=0x55de162b8390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:51.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.711+0000 7f660bb9e740 1 -- 172.21.15.28:0/3290558460 shutdown_connections 2024-04-23T14:57:51.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.711+0000 7f660bb9e740 1 -- 172.21.15.28:0/3290558460 wait complete. 2024-04-23T14:57:51.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.711+0000 7f660bb9e740 20 client.17187 trim_cache size 0 max 16384 2024-04-23T14:57:51.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.731+0000 7f660bb9e740 1 Processor -- start 2024-04-23T14:57:51.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.732+0000 7f660bb9e740 1 -- start start 2024-04-23T14:57:51.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.732+0000 7f660bb9e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de162b6270 0x55de162ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:51.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.732+0000 7f660bb9e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de162b6b90 0x55de162b81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:51.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.732+0000 7f660bb9e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de162c1260 0x55de162d9240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:51.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.732+0000 7f660bb9e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f66041add80 con 0x55de162c1260 2024-04-23T14:57:51.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.732+0000 7f660bb9e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f66041b05e0 con 0x55de162b6b90 2024-04-23T14:57:51.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.732+0000 7f660bb9e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f66041ae3d0 con 0x55de162b6270 2024-04-23T14:57:51.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.732+0000 7f6601e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de162b6270 0x55de162ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:51.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.732+0000 7f6601e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de162b6270 0x55de162ccf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:34618/0 (socket says 172.21.15.28:34618) 2024-04-23T14:57:51.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.732+0000 7f6601e00640 1 -- 172.21.15.28:0/3905629826 learned_addr learned my addr 172.21.15.28:0/3905629826 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:51.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.732+0000 7f65f9400640 1 --2- 172.21.15.28:0/3905629826 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de162b6b90 0x55de162b81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:51.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.732+0000 7f6602800640 1 --2- 172.21.15.28:0/3905629826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de162c1260 0x55de162d9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:51.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.733+0000 7f6601e00640 1 -- 172.21.15.28:0/3905629826 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de162b6b90 msgr2=0x55de162b81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:51.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.733+0000 7f6601e00640 1 --2- 172.21.15.28:0/3905629826 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de162b6b90 0x55de162b81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:51.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.733+0000 7f6601e00640 1 -- 172.21.15.28:0/3905629826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de162c1260 msgr2=0x55de162d9240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:51.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.733+0000 7f6601e00640 1 --2- 172.21.15.28:0/3905629826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de162c1260 0x55de162d9240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:51.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.733+0000 7f6601e00640 1 -- 172.21.15.28:0/3905629826 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de162d3fb0 con 0x55de162b6270 2024-04-23T14:57:51.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.733+0000 7f65f9400640 1 --2- 172.21.15.28:0/3905629826 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de162b6b90 0x55de162b81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:57:51.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.733+0000 7f6601e00640 1 --2- 172.21.15.28:0/3905629826 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de162b6270 0x55de162ccf10 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f65ec005dc0 tx=0x7f65ec01b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0d84d190dfd7af0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:51.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.733+0000 7f65f2a00640 1 -- 172.21.15.28:0/3905629826 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ec004020 con 0x55de162b6270 2024-04-23T14:57:51.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.733+0000 7f65f2a00640 1 -- 172.21.15.28:0/3905629826 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65ec005500 con 0x55de162b6270 2024-04-23T14:57:51.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.733+0000 7f65f2a00640 1 -- 172.21.15.28:0/3905629826 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ec004e00 con 0x55de162b6270 2024-04-23T14:57:51.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.734+0000 7f660bb9e740 1 -- 172.21.15.28:0/3905629826 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de162b6270 msgr2=0x55de162ccf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:51.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.734+0000 7f660bb9e740 1 --2- 172.21.15.28:0/3905629826 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de162b6270 0x55de162ccf10 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f65ec005dc0 tx=0x7f65ec01b000 comp rx=0 tx=0).stop 2024-04-23T14:57:51.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.734+0000 7f660bb9e740 1 -- 172.21.15.28:0/3905629826 shutdown_connections 2024-04-23T14:57:51.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.734+0000 7f660bb9e740 1 --2- 172.21.15.28:0/3905629826 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de162b6b90 0x55de162b81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:51.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.734+0000 7f660bb9e740 1 --2- 172.21.15.28:0/3905629826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de162c1260 0x55de162d9240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:51.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.734+0000 7f660bb9e740 1 --2- 172.21.15.28:0/3905629826 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de162b6270 0x55de162ccf10 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:51.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.734+0000 7f660bb9e740 1 -- 172.21.15.28:0/3905629826 >> 172.21.15.28:0/3905629826 conn(0x55de162b2590 msgr2=0x55de162b7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:51.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.734+0000 7f660bb9e740 1 -- 172.21.15.28:0/3905629826 shutdown_connections 2024-04-23T14:57:51.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.734+0000 7f660bb9e740 1 -- 172.21.15.28:0/3905629826 wait complete. 2024-04-23T14:57:51.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.735+0000 7f660bb9e740 1 -- start start 2024-04-23T14:57:51.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.735+0000 7f660bb9e740 1 Processor -- start 2024-04-23T14:57:51.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.735+0000 7f660bb9e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de162b6270 0x55de162c5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:51.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.736+0000 7f660bb9e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de162b6b90 0x55de162c5590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:51.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.736+0000 7f660bb9e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de162c1260 0x55de162c5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:51.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.736+0000 7f660bb9e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f66041aeaa0 con 0x55de162b6b90 2024-04-23T14:57:51.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.736+0000 7f660bb9e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f66041aff10 con 0x55de162b6270 2024-04-23T14:57:51.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.736+0000 7f660bb9e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f66041af170 con 0x55de162c1260 2024-04-23T14:57:51.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.736+0000 7f660bb9e740 10 client.0 _ll_register_callbacks cb 0x7f65f4005f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:57:51.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.736+0000 7f6601e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de162b6270 0x55de162c5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:51.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.736+0000 7f6601e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de162b6270 0x55de162c5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:42020/0 (socket says 172.21.15.28:42020) 2024-04-23T14:57:51.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.736+0000 7f65f9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de162b6b90 0x55de162c5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:51.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.736+0000 7f6601e00640 1 -- 172.21.15.28:0/3827772769 learned_addr learned my addr 172.21.15.28:0/3827772769 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:51.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.736+0000 7f6602800640 1 --2- 172.21.15.28:0/3827772769 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de162c1260 0x55de162c5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:51.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.736+0000 7f6601e00640 1 -- 172.21.15.28:0/3827772769 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de162c1260 msgr2=0x55de162c5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:51.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.736+0000 7f6601e00640 1 --2- 172.21.15.28:0/3827772769 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de162c1260 0x55de162c5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:51.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.736+0000 7f6601e00640 1 -- 172.21.15.28:0/3827772769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de162b6b90 msgr2=0x55de162c5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:51.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.736+0000 7f6601e00640 1 --2- 172.21.15.28:0/3827772769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de162b6b90 0x55de162c5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:51.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.736+0000 7f6601e00640 1 -- 172.21.15.28:0/3827772769 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de16336d00 con 0x55de162b6270 2024-04-23T14:57:51.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.737+0000 7f6601e00640 1 --2- 172.21.15.28:0/3827772769 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de162b6270 0x55de162c5050 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f65ec019040 tx=0x7f65ec009e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f164b666508a743 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:51.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.737+0000 7f65e7e00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-23T14:57:51.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.737+0000 7f65e7e00640 1 -- 172.21.15.28:0/3827772769 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ec032040 con 0x55de162b6270 2024-04-23T14:57:51.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.737+0000 7f65e7e00640 1 -- 172.21.15.28:0/3827772769 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8542+0+0 (secure 0 0 0) 0x7f65ec02d110 con 0x55de162b6270 2024-04-23T14:57:51.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.737+0000 7f660bb9e740 1 -- 172.21.15.28:0/3827772769 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65ec005500 con 0x55de162b6270 2024-04-23T14:57:51.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.737+0000 7f65e7e00640 1 -- 172.21.15.28:0/3827772769 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f65ec046050 con 0x55de162b6270 2024-04-23T14:57:51.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.737+0000 7f660bb9e740 10 client.27389 fetch_fsmap learned FSMap version 20 2024-04-23T14:57:51.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.737+0000 7f660bb9e740 1 -- 172.21.15.28:0/3827772769 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55de16338000 con 0x55de162b6270 2024-04-23T14:57:51.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.737+0000 7f65e7e00640 1 -- 172.21.15.28:0/3827772769 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65ec005500 con 0x55de162b6270 2024-04-23T14:57:51.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.738+0000 7f65e7e00640 1 -- 172.21.15.28:0/3827772769 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f65ec005080 con 0x55de162b6270 2024-04-23T14:57:51.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.738+0000 7f660bb9e740 10 client.27389 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:57:51.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.738+0000 7f660bb9e740 10 client.27389 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:57:51.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.738+0000 7f660bb9e740 20 client.27389 populate_metadata read hostname 'smithi028' 2024-04-23T14:57:51.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.738+0000 7f660bb9e740 1 --2- 172.21.15.28:0/3827772769 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55de163386b0 0x55de1633aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:51.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.738+0000 7f660bb9e740 4 client.27389 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-23T14:57:51.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.738+0000 7f660bb9e740 1 -- 172.21.15.28:0/3827772769 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55de16223700 con 0x55de163386b0 2024-04-23T14:57:51.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.738+0000 7f65f9400640 1 --2- 172.21.15.28:0/3827772769 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55de163386b0 0x55de1633aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:51.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.739+0000 7f65f9400640 1 --2- 172.21.15.28:0/3827772769 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55de163386b0 0x55de1633aaa0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=671a506544c09e32 server_cookie=8e6d994ac121ccac in_seq=0 out_seq=0 2024-04-23T14:57:51.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.739+0000 7f65e7e00640 10 client.27389 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:57:51.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.748+0000 7f65e7e00640 1 -- 172.21.15.28:0/3827772769 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55de16223700 con 0x55de163386b0 2024-04-23T14:57:51.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.748+0000 7f65e7e00640 10 client.27389 handle_command_reply: tid=0 2024-04-23T14:57:51.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.748+0000 7f660bb9e740 1 client.27389 shutdown 2024-04-23T14:57:51.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.748+0000 7f660bb9e740 1 -- 172.21.15.28:0/3827772769 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de162b6270 msgr2=0x55de162c5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:51.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.748+0000 7f660bb9e740 1 --2- 172.21.15.28:0/3827772769 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de162b6270 0x55de162c5050 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f65ec019040 tx=0x7f65ec009e80 comp rx=0 tx=0).stop 2024-04-23T14:57:51.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.748+0000 7f660bb9e740 1 -- 172.21.15.28:0/3827772769 shutdown_connections 2024-04-23T14:57:51.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.748+0000 7f660bb9e740 1 --2- 172.21.15.28:0/3827772769 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55de163386b0 0x55de1633aaa0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:51.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.748+0000 7f660bb9e740 1 --2- 172.21.15.28:0/3827772769 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55de162c1260 0x55de162c5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:51.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.748+0000 7f660bb9e740 1 --2- 172.21.15.28:0/3827772769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de162b6b90 0x55de162c5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:51.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.748+0000 7f660bb9e740 1 --2- 172.21.15.28:0/3827772769 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de162b6270 0x55de162c5050 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:51.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.748+0000 7f660bb9e740 1 -- 172.21.15.28:0/3827772769 >> 172.21.15.28:0/3827772769 conn(0x55de162b2590 msgr2=0x55de162b8360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:51.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.748+0000 7f65e7e00640 0 client.27389 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:57:51.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.748+0000 7f660bb9e740 1 -- 172.21.15.28:0/3827772769 shutdown_connections 2024-04-23T14:57:51.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.748+0000 7f660bb9e740 1 -- 172.21.15.28:0/3827772769 wait complete. 2024-04-23T14:57:51.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.748+0000 7f660bb9e740 20 client.27389 trim_cache size 0 max 16384 2024-04-23T14:57:51.750 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:57:51.750 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-23T14:57:51.750 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "8d378d34-36a1-4f00-8fb3-4c63c1510ab4", 2024-04-23T14:57:51.750 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-23T14:57:51.750 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:57:51.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.749+0000 7f6603200640 1 -- 172.21.15.28:0/2427150168 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f65e80a3d20 msgr2=0x7f65e80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:51.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.749+0000 7f6603200640 1 --2- 172.21.15.28:0/2427150168 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f65e80a3d20 0x7f65e80a61e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x55de162b7830 tx=0x7f65ec009260 comp rx=0 tx=0).stop 2024-04-23T14:57:51.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.749+0000 7f6603200640 1 -- 172.21.15.28:0/2427150168 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6604103980 msgr2=0x7f66041a0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:51.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.749+0000 7f6603200640 1 --2- 172.21.15.28:0/2427150168 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6604103980 0x7f66041a0bf0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f65f4034040 tx=0x7f65f40305d0 comp rx=0 tx=0).stop 2024-04-23T14:57:51.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.749+0000 7f6603200640 1 -- 172.21.15.28:0/2427150168 shutdown_connections 2024-04-23T14:57:51.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.749+0000 7f6603200640 1 --2- 172.21.15.28:0/2427150168 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f65e80a3d20 0x7f65e80a61e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:51.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.749+0000 7f6603200640 1 --2- 172.21.15.28:0/2427150168 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6604103980 0x7f66041a0bf0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:51.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.749+0000 7f6603200640 1 --2- 172.21.15.28:0/2427150168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f660410c480 0x7f660419a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:51.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.749+0000 7f6603200640 1 --2- 172.21.15.28:0/2427150168 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6604103060 0x7f66041a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:51.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.749+0000 7f6603200640 1 -- 172.21.15.28:0/2427150168 >> 172.21.15.28:0/2427150168 conn(0x7f66040fc880 msgr2=0x7f6604101e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:51.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.749+0000 7f6603200640 1 -- 172.21.15.28:0/2427150168 shutdown_connections 2024-04-23T14:57:51.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:51.749+0000 7f6603200640 1 -- 172.21.15.28:0/2427150168 wait complete. 2024-04-23T14:57:51.769 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:57:51.769 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:57:51.822 INFO:tasks.workunit.client.0.smithi028.stdout: 10 3631 54.86 MB/sec warmup 7 sec latency 139.092 ms 2024-04-23T14:57:52.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.018+0000 7f052e200640 1 -- 172.21.15.28:0/4077013851 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0528104310 msgr2=0x7f05281046f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:52.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.018+0000 7f052e200640 1 --2- 172.21.15.28:0/4077013851 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0528104310 0x7f05281046f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f051c009900 tx=0x7f051c02a3f0 comp rx=0 tx=0).stop 2024-04-23T14:57:52.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.018+0000 7f052e200640 1 -- 172.21.15.28:0/4077013851 shutdown_connections 2024-04-23T14:57:52.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.018+0000 7f052e200640 1 --2- 172.21.15.28:0/4077013851 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0528104cc0 0x7f0528108d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:52.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.018+0000 7f052e200640 1 --2- 172.21.15.28:0/4077013851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05281093e0 0x7f0528110fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:52.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.018+0000 7f052e200640 1 --2- 172.21.15.28:0/4077013851 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0528104310 0x7f05281046f0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:52.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.018+0000 7f052e200640 1 -- 172.21.15.28:0/4077013851 >> 172.21.15.28:0/4077013851 conn(0x7f05280ffb10 msgr2=0x7f0528101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:52.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.019+0000 7f052e200640 1 -- 172.21.15.28:0/4077013851 shutdown_connections 2024-04-23T14:57:52.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.019+0000 7f052e200640 1 -- 172.21.15.28:0/4077013851 wait complete. 2024-04-23T14:57:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.020+0000 7f052e200640 1 Processor -- start 2024-04-23T14:57:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.020+0000 7f052e200640 1 -- start start 2024-04-23T14:57:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.020+0000 7f052e200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0528104310 0x7f05281a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.020+0000 7f052e200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05281093e0 0x7f05281a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.020+0000 7f052e200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f052819a7c0 0x7f052819ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.020+0000 7f052e200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f05281144c0 con 0x7f052819a7c0 2024-04-23T14:57:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.020+0000 7f052e200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0528114340 con 0x7f05281093e0 2024-04-23T14:57:52.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.020+0000 7f052e200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0528114640 con 0x7f0528104310 2024-04-23T14:57:52.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.020+0000 7f052ce00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0528104310 0x7f05281a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:52.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.020+0000 7f052ce00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0528104310 0x7f05281a06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:34638/0 (socket says 172.21.15.28:34638) 2024-04-23T14:57:52.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.020+0000 7f052ce00640 1 -- 172.21.15.28:0/454519700 learned_addr learned my addr 172.21.15.28:0/454519700 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:52.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.021+0000 7f0527e00640 1 --2- 172.21.15.28:0/454519700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05281093e0 0x7f05281a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:52.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.021+0000 7f052d800640 1 --2- 172.21.15.28:0/454519700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f052819a7c0 0x7f052819ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:52.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.021+0000 7f052ce00640 1 -- 172.21.15.28:0/454519700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05281093e0 msgr2=0x7f05281a0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:52.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.021+0000 7f052ce00640 1 --2- 172.21.15.28:0/454519700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05281093e0 0x7f05281a0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:52.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.021+0000 7f052ce00640 1 -- 172.21.15.28:0/454519700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f052819a7c0 msgr2=0x7f052819ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:52.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.021+0000 7f052ce00640 1 --2- 172.21.15.28:0/454519700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f052819a7c0 0x7f052819ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:52.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.021+0000 7f052ce00640 1 -- 172.21.15.28:0/454519700 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f052819b3c0 con 0x7f0528104310 2024-04-23T14:57:52.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.021+0000 7f052ce00640 1 --2- 172.21.15.28:0/454519700 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0528104310 0x7f05281a06b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f051c004770 tx=0x7f051c003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:52.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.022+0000 7f0525600640 1 -- 172.21.15.28:0/454519700 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f051c03f070 con 0x7f0528104310 2024-04-23T14:57:52.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.022+0000 7f0525600640 1 -- 172.21.15.28:0/454519700 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f051c031ca0 con 0x7f0528104310 2024-04-23T14:57:52.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.022+0000 7f0525600640 1 -- 172.21.15.28:0/454519700 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f051c035030 con 0x7f0528104310 2024-04-23T14:57:52.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.022+0000 7f052e200640 1 -- 172.21.15.28:0/454519700 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f052819b650 con 0x7f0528104310 2024-04-23T14:57:52.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.022+0000 7f052e200640 1 -- 172.21.15.28:0/454519700 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05281a7790 con 0x7f0528104310 2024-04-23T14:57:52.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.025+0000 7f0525600640 1 -- 172.21.15.28:0/454519700 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f051c042070 con 0x7f0528104310 2024-04-23T14:57:52.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.025+0000 7f0532601740 1 Processor -- start 2024-04-23T14:57:52.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.025+0000 7f0532601740 1 -- start start 2024-04-23T14:57:52.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.026+0000 7f0532601740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f954d56a30 0x55f954d56e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:52.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.026+0000 7f0532601740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f954d57350 0x55f954d614e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:52.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.026+0000 7f0532601740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f954d61a20 0x55f954d63e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:52.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.026+0000 7f0532601740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f954cbc1f0 con 0x55f954d61a20 2024-04-23T14:57:52.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.026+0000 7f0532601740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f954c62350 con 0x55f954d57350 2024-04-23T14:57:52.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.026+0000 7f0532601740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f954cb81b0 con 0x55f954d56a30 2024-04-23T14:57:52.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.026+0000 7f052d800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f954d56a30 0x55f954d56e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:52.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.026+0000 7f052ce00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f954d57350 0x55f954d614e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:52.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.026+0000 7f052d800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f954d56a30 0x55f954d56e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:34650/0 (socket says 172.21.15.28:34650) 2024-04-23T14:57:52.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.026+0000 7f052d800640 1 -- 172.21.15.28:0/3858099725 learned_addr learned my addr 172.21.15.28:0/3858099725 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:52.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.026+0000 7f0527e00640 1 --2- 172.21.15.28:0/3858099725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f954d61a20 0x55f954d63e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:52.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.026+0000 7f052d800640 1 -- 172.21.15.28:0/3858099725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f954d57350 msgr2=0x55f954d614e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:52.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.026+0000 7f052d800640 1 --2- 172.21.15.28:0/3858099725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f954d57350 0x55f954d614e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:52.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.026+0000 7f052d800640 1 -- 172.21.15.28:0/3858099725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f954d61a20 msgr2=0x55f954d63e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:52.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.026+0000 7f052d800640 1 --2- 172.21.15.28:0/3858099725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f954d61a20 0x55f954d63e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:52.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.026+0000 7f052d800640 1 -- 172.21.15.28:0/3858099725 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f954cb6f50 con 0x55f954d56a30 2024-04-23T14:57:52.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.027+0000 7f052d800640 1 --2- 172.21.15.28:0/3858099725 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f954d56a30 0x55f954d56e10 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f0520009f50 tx=0x7f05200118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7a3bfbac0759b26 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:52.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.027+0000 7f0525600640 1 --2- 172.21.15.28:0/454519700 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f05100a3da0 0x7f05100a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:52.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.027+0000 7f051a000640 1 -- 172.21.15.28:0/3858099725 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0520004d20 con 0x55f954d56a30 2024-04-23T14:57:52.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.027+0000 7f051a000640 1 -- 172.21.15.28:0/3858099725 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0520004050 con 0x55f954d56a30 2024-04-23T14:57:52.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.027+0000 7f051a000640 1 -- 172.21.15.28:0/3858099725 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0520005280 con 0x55f954d56a30 2024-04-23T14:57:52.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.027+0000 7f0525600640 1 -- 172.21.15.28:0/454519700 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8542+0+0 (secure 0 0 0) 0x7f051c0f0d90 con 0x7f0528104310 2024-04-23T14:57:52.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.027+0000 7f052ce00640 1 --2- 172.21.15.28:0/454519700 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f05100a3da0 0x7f05100a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:52.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.028+0000 7f0532601740 1 -- 172.21.15.28:0/3858099725 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f954d56a30 msgr2=0x55f954d56e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:52.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.028+0000 7f052ce00640 1 --2- 172.21.15.28:0/454519700 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f05100a3da0 0x7f05100a6260 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x55f954d57ff0 tx=0x7f051c0f4840 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:52.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.028+0000 7f0532601740 1 --2- 172.21.15.28:0/3858099725 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f954d56a30 0x55f954d56e10 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f0520009f50 tx=0x7f05200118d0 comp rx=0 tx=0).stop 2024-04-23T14:57:52.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.028+0000 7f0532601740 1 -- 172.21.15.28:0/3858099725 shutdown_connections 2024-04-23T14:57:52.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.028+0000 7f0532601740 1 --2- 172.21.15.28:0/3858099725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f954d57350 0x55f954d614e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:52.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.028+0000 7f0532601740 1 --2- 172.21.15.28:0/3858099725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f954d61a20 0x55f954d63e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:52.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.028+0000 7f0532601740 1 --2- 172.21.15.28:0/3858099725 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f954d56a30 0x55f954d56e10 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:52.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.028+0000 7f0532601740 1 -- 172.21.15.28:0/3858099725 >> 172.21.15.28:0/3858099725 conn(0x55f954d52d50 msgr2=0x55f954d53130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:52.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.028+0000 7f0532601740 1 -- 172.21.15.28:0/3858099725 shutdown_connections 2024-04-23T14:57:52.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.028+0000 7f0532601740 1 -- 172.21.15.28:0/3858099725 wait complete. 2024-04-23T14:57:52.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.029+0000 7f0532601740 1 -- start start 2024-04-23T14:57:52.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.029+0000 7f0532601740 1 Processor -- start 2024-04-23T14:57:52.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.029+0000 7f0532601740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f954d56a30 0x55f954d5e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:52.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.029+0000 7f0532601740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f954d57350 0x55f954d5ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:52.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.030+0000 7f0532601740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f954d61a20 0x55f954d5f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:52.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.030+0000 7f0532601740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f954d64810 con 0x55f954d57350 2024-04-23T14:57:52.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.030+0000 7f0532601740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f954c75690 con 0x55f954d56a30 2024-04-23T14:57:52.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.030+0000 7f0532601740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f954d64990 con 0x55f954d61a20 2024-04-23T14:57:52.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.030+0000 7f0532601740 10 client.0 _ll_register_callbacks cb 0x55f954cb1650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:57:52.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.030+0000 7f052d800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f954d56a30 0x55f954d5e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:52.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.030+0000 7f0527e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f954d57350 0x55f954d5ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:52.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.030+0000 7f0527e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f954d57350 0x55f954d5ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37064/0 (socket says 172.21.15.28:37064) 2024-04-23T14:57:52.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.030+0000 7f0527e00640 1 -- 172.21.15.28:0/934400754 learned_addr learned my addr 172.21.15.28:0/934400754 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:52.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.030+0000 7f052d800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f954d56a30 0x55f954d5e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:42064/0 (socket says 172.21.15.28:42064) 2024-04-23T14:57:52.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.030+0000 7f052d800640 1 --2- 172.21.15.28:0/934400754 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f954d61a20 0x55f954d5f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:52.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.030+0000 7f0527e00640 1 -- 172.21.15.28:0/934400754 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f954d61a20 msgr2=0x55f954d5f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:52.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.030+0000 7f0527e00640 1 --2- 172.21.15.28:0/934400754 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f954d61a20 0x55f954d5f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:52.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.030+0000 7f0527e00640 1 -- 172.21.15.28:0/934400754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f954d56a30 msgr2=0x55f954d5e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:52.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.030+0000 7f0527e00640 1 --2- 172.21.15.28:0/934400754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f954d56a30 0x55f954d5e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:52.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.030+0000 7f0527e00640 1 -- 172.21.15.28:0/934400754 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f954c74920 con 0x55f954d57350 2024-04-23T14:57:52.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.030+0000 7f052d800640 1 --2- 172.21.15.28:0/934400754 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f954d61a20 0x55f954d5f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:57:52.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.031+0000 7f0527e00640 1 --2- 172.21.15.28:0/934400754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f954d57350 0x55f954d5ee80 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f0514012730 tx=0x7f0514012bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fed8df0e24d8bd3d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:52.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.031+0000 7f050ea00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-23T14:57:52.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.031+0000 7f050ea00640 1 -- 172.21.15.28:0/934400754 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0514004050 con 0x55f954d57350 2024-04-23T14:57:52.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.031+0000 7f050ea00640 1 -- 172.21.15.28:0/934400754 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8542+0+0 (secure 0 0 0) 0x7f05140041f0 con 0x55f954d57350 2024-04-23T14:57:52.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.031+0000 7f0532601740 1 -- 172.21.15.28:0/934400754 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0520004050 con 0x55f954d57350 2024-04-23T14:57:52.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.032+0000 7f050ea00640 1 -- 172.21.15.28:0/934400754 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f05140043f0 con 0x55f954d57350 2024-04-23T14:57:52.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.032+0000 7f0532601740 10 client.17205 fetch_fsmap learned FSMap version 20 2024-04-23T14:57:52.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.032+0000 7f0532601740 1 -- 172.21.15.28:0/934400754 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55f954a987d0 con 0x55f954d57350 2024-04-23T14:57:52.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.032+0000 7f050ea00640 1 -- 172.21.15.28:0/934400754 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0520004050 con 0x55f954d57350 2024-04-23T14:57:52.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.032+0000 7f050ea00640 1 -- 172.21.15.28:0/934400754 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f0514003c00 con 0x55f954d57350 2024-04-23T14:57:52.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.032+0000 7f0532601740 10 client.17205 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:57:52.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.032+0000 7f0532601740 10 client.17205 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:57:52.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.032+0000 7f0532601740 20 client.17205 populate_metadata read hostname 'smithi028' 2024-04-23T14:57:52.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.032+0000 7f0532601740 1 --2- 172.21.15.28:0/934400754 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f954d71da0 0x55f954d741d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:52.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.032+0000 7f0532601740 4 client.17205 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:57:52.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.032+0000 7f0532601740 1 -- 172.21.15.28:0/934400754 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f954cc9c10 con 0x55f954d71da0 2024-04-23T14:57:52.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.033+0000 7f052d800640 1 --2- 172.21.15.28:0/934400754 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f954d71da0 0x55f954d741d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:52.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.033+0000 7f052d800640 1 --2- 172.21.15.28:0/934400754 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f954d71da0 0x55f954d741d0 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a6f87873699eda0 server_cookie=2047ef1e24281d15 in_seq=0 out_seq=0 2024-04-23T14:57:52.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.033+0000 7f050ea00640 10 client.17205 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:57:52.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.035+0000 7f050ea00640 1 -- 172.21.15.28:0/934400754 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f954cc9c10 con 0x55f954d71da0 2024-04-23T14:57:52.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.035+0000 7f050ea00640 10 client.17205 handle_command_reply: tid=0 2024-04-23T14:57:52.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.035+0000 7f0532601740 1 client.17205 shutdown 2024-04-23T14:57:52.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.036+0000 7f0532601740 1 -- 172.21.15.28:0/934400754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f954d57350 msgr2=0x55f954d5ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:52.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.036+0000 7f0532601740 1 --2- 172.21.15.28:0/934400754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f954d57350 0x55f954d5ee80 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f0514012730 tx=0x7f0514012bf0 comp rx=0 tx=0).stop 2024-04-23T14:57:52.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.036+0000 7f0532601740 1 -- 172.21.15.28:0/934400754 shutdown_connections 2024-04-23T14:57:52.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.036+0000 7f0532601740 1 --2- 172.21.15.28:0/934400754 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f954d71da0 0x55f954d741d0 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:52.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.036+0000 7f0532601740 1 --2- 172.21.15.28:0/934400754 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f954d61a20 0x55f954d5f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:52.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.036+0000 7f0532601740 1 --2- 172.21.15.28:0/934400754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f954d57350 0x55f954d5ee80 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:52.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.036+0000 7f0532601740 1 --2- 172.21.15.28:0/934400754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f954d56a30 0x55f954d5e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:52.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.036+0000 7f050ea00640 0 client.17205 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:57:52.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.036+0000 7f0532601740 1 -- 172.21.15.28:0/934400754 >> 172.21.15.28:0/934400754 conn(0x55f954d52d50 msgr2=0x55f954d58b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:52.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.036+0000 7f0532601740 1 -- 172.21.15.28:0/934400754 shutdown_connections 2024-04-23T14:57:52.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.036+0000 7f0532601740 1 -- 172.21.15.28:0/934400754 wait complete. 2024-04-23T14:57:52.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.036+0000 7f0532601740 20 client.17205 trim_cache size 0 max 16384 2024-04-23T14:57:52.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.057+0000 7f0532601740 1 Processor -- start 2024-04-23T14:57:52.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.057+0000 7f0532601740 1 -- start start 2024-04-23T14:57:52.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.058+0000 7f0532601740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f954d52d50 0x55f954d6d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:52.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.058+0000 7f0532601740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f954d56a30 0x55f954d79ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:52.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.058+0000 7f0532601740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f954d57350 0x55f954d7a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:52.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.058+0000 7f0532601740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f05281add90 con 0x55f954d57350 2024-04-23T14:57:52.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.058+0000 7f0532601740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f05281b05f0 con 0x55f954d52d50 2024-04-23T14:57:52.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.058+0000 7f0532601740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f05281ae3e0 con 0x55f954d56a30 2024-04-23T14:57:52.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.058+0000 7f052d800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f954d52d50 0x55f954d6d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:52.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.058+0000 7f0527e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f954d56a30 0x55f954d79ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:52.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.058+0000 7f052d800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f954d52d50 0x55f954d6d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:42076/0 (socket says 172.21.15.28:42076) 2024-04-23T14:57:52.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.058+0000 7f052d800640 1 -- 172.21.15.28:0/588928791 learned_addr learned my addr 172.21.15.28:0/588928791 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:52.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.058+0000 7f052d800640 1 --2- 172.21.15.28:0/588928791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f954d57350 0x55f954d7a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:52.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.059+0000 7f052d800640 1 -- 172.21.15.28:0/588928791 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f954d56a30 msgr2=0x55f954d79ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:52.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.059+0000 7f052d800640 1 --2- 172.21.15.28:0/588928791 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f954d56a30 0x55f954d79ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:52.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.059+0000 7f052d800640 1 -- 172.21.15.28:0/588928791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f954d57350 msgr2=0x55f954d7a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:52.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.059+0000 7f052d800640 1 --2- 172.21.15.28:0/588928791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f954d57350 0x55f954d7a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:52.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.059+0000 7f052d800640 1 -- 172.21.15.28:0/588928791 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9549ce3f0 con 0x55f954d52d50 2024-04-23T14:57:52.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.059+0000 7f052d800640 1 --2- 172.21.15.28:0/588928791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f954d52d50 0x55f954d6d6d0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f0520011dd0 tx=0x7f05200046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=809aa58ecf6e1626 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:52.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.059+0000 7f0519600640 1 -- 172.21.15.28:0/588928791 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0520004050 con 0x55f954d52d50 2024-04-23T14:57:52.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.059+0000 7f0519600640 1 -- 172.21.15.28:0/588928791 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0520004990 con 0x55f954d52d50 2024-04-23T14:57:52.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.059+0000 7f0519600640 1 -- 172.21.15.28:0/588928791 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0520005470 con 0x55f954d52d50 2024-04-23T14:57:52.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.060+0000 7f0532601740 1 -- 172.21.15.28:0/588928791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f954d52d50 msgr2=0x55f954d6d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:52.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.060+0000 7f0532601740 1 --2- 172.21.15.28:0/588928791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f954d52d50 0x55f954d6d6d0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f0520011dd0 tx=0x7f05200046d0 comp rx=0 tx=0).stop 2024-04-23T14:57:52.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.060+0000 7f0532601740 1 -- 172.21.15.28:0/588928791 shutdown_connections 2024-04-23T14:57:52.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.060+0000 7f0532601740 1 --2- 172.21.15.28:0/588928791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f954d57350 0x55f954d7a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:52.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.060+0000 7f0532601740 1 --2- 172.21.15.28:0/588928791 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f954d56a30 0x55f954d79ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:52.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.060+0000 7f0532601740 1 --2- 172.21.15.28:0/588928791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f954d52d50 0x55f954d6d6d0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:52.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.060+0000 7f0532601740 1 -- 172.21.15.28:0/588928791 >> 172.21.15.28:0/588928791 conn(0x55f954d5d580 msgr2=0x55f954d5f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:52.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.060+0000 7f0532601740 1 -- 172.21.15.28:0/588928791 shutdown_connections 2024-04-23T14:57:52.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.060+0000 7f0532601740 1 -- 172.21.15.28:0/588928791 wait complete. 2024-04-23T14:57:52.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.060+0000 7f0532601740 1 -- start start 2024-04-23T14:57:52.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.061+0000 7f0532601740 1 Processor -- start 2024-04-23T14:57:52.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.061+0000 7f0532601740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f954d52d50 0x55f954d66220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:52.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.061+0000 7f0532601740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f954d56a30 0x55f954d66760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:52.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.061+0000 7f0532601740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f954d57350 0x55f954dd0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:52.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.061+0000 7f0532601740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f05281aeab0 con 0x55f954d56a30 2024-04-23T14:57:52.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.061+0000 7f0532601740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f05281aff20 con 0x55f954d57350 2024-04-23T14:57:52.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.061+0000 7f0532601740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f05281af180 con 0x55f954d52d50 2024-04-23T14:57:52.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.061+0000 7f0532601740 10 client.0 _ll_register_callbacks cb 0x7f0514009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:57:52.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.062+0000 7f0527e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f954d56a30 0x55f954d66760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:52.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.062+0000 7f052d800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f954d52d50 0x55f954d66220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:52.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.062+0000 7f0527e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f954d56a30 0x55f954d66760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37086/0 (socket says 172.21.15.28:37086) 2024-04-23T14:57:52.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.062+0000 7f0527e00640 1 -- 172.21.15.28:0/777981242 learned_addr learned my addr 172.21.15.28:0/777981242 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:57:52.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.062+0000 7f052d800640 1 --2- 172.21.15.28:0/777981242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f954d57350 0x55f954dd0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:52.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.062+0000 7f0527e00640 1 -- 172.21.15.28:0/777981242 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f954d52d50 msgr2=0x55f954d66220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:52.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.062+0000 7f0527e00640 1 --2- 172.21.15.28:0/777981242 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f954d52d50 0x55f954d66220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:52.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.062+0000 7f0527e00640 1 -- 172.21.15.28:0/777981242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f954d57350 msgr2=0x55f954dd0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:57:52.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.062+0000 7f0527e00640 1 --2- 172.21.15.28:0/777981242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f954d57350 0x55f954dd0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:52.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.062+0000 7f0527e00640 1 -- 172.21.15.28:0/777981242 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f95498e0f0 con 0x55f954d56a30 2024-04-23T14:57:52.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.062+0000 7f052d800640 1 --2- 172.21.15.28:0/777981242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f954d57350 0x55f954dd0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:57:52.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.062+0000 7f0527e00640 1 --2- 172.21.15.28:0/777981242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f954d56a30 0x55f954d66760 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f0514000c00 tx=0x7f0514005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6ac56557e028460 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:57:52.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.062+0000 7f050ea00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-23T14:57:52.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.063+0000 7f050ea00640 1 -- 172.21.15.28:0/777981242 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0514028040 con 0x55f954d56a30 2024-04-23T14:57:52.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.063+0000 7f050ea00640 1 -- 172.21.15.28:0/777981242 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8542+0+0 (secure 0 0 0) 0x7f05140052f0 con 0x55f954d56a30 2024-04-23T14:57:52.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.063+0000 7f0532601740 1 -- 172.21.15.28:0/777981242 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0520004990 con 0x55f954d56a30 2024-04-23T14:57:52.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.063+0000 7f050ea00640 1 -- 172.21.15.28:0/777981242 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f051403a050 con 0x55f954d56a30 2024-04-23T14:57:52.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.063+0000 7f0532601740 10 client.17211 fetch_fsmap learned FSMap version 20 2024-04-23T14:57:52.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.063+0000 7f0532601740 1 -- 172.21.15.28:0/777981242 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55f954c81810 con 0x55f954d56a30 2024-04-23T14:57:52.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.063+0000 7f050ea00640 1 -- 172.21.15.28:0/777981242 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0520004990 con 0x55f954d56a30 2024-04-23T14:57:52.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.063+0000 7f050ea00640 1 -- 172.21.15.28:0/777981242 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f051402b070 con 0x55f954d56a30 2024-04-23T14:57:52.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.064+0000 7f0532601740 10 client.17211 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:57:52.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.064+0000 7f0532601740 10 client.17211 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:57:52.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.064+0000 7f0532601740 20 client.17211 populate_metadata read hostname 'smithi028' 2024-04-23T14:57:52.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.064+0000 7f0532601740 1 --2- 172.21.15.28:0/777981242 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f954d61a20 0x55f954ddb310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:57:52.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.064+0000 7f0532601740 4 client.17211 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:57:52.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.064+0000 7f0532601740 1 -- 172.21.15.28:0/777981242 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f954cc3a80 con 0x55f954d61a20 2024-04-23T14:57:52.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.064+0000 7f052d800640 1 --2- 172.21.15.28:0/777981242 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f954d61a20 0x55f954ddb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:57:52.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.064+0000 7f052d800640 1 --2- 172.21.15.28:0/777981242 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f954d61a20 0x55f954ddb310 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80d6a54a94a336bc server_cookie=192fcb0cbcb8b43f in_seq=0 out_seq=0 2024-04-23T14:57:52.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.064+0000 7f050ea00640 10 client.17211 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:57:52.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.069+0000 7f050ea00640 1 -- 172.21.15.28:0/777981242 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55f954cc3a80 con 0x55f954d61a20 2024-04-23T14:57:52.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.069+0000 7f050ea00640 10 client.17211 handle_command_reply: tid=0 2024-04-23T14:57:52.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.069+0000 7f0532601740 1 client.17211 shutdown 2024-04-23T14:57:52.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.069+0000 7f0532601740 1 -- 172.21.15.28:0/777981242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f954d56a30 msgr2=0x55f954d66760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:52.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.069+0000 7f0532601740 1 --2- 172.21.15.28:0/777981242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f954d56a30 0x55f954d66760 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f0514000c00 tx=0x7f0514005040 comp rx=0 tx=0).stop 2024-04-23T14:57:52.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.069+0000 7f0532601740 1 -- 172.21.15.28:0/777981242 shutdown_connections 2024-04-23T14:57:52.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.069+0000 7f0532601740 1 --2- 172.21.15.28:0/777981242 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55f954d61a20 0x55f954ddb310 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:52.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.069+0000 7f0532601740 1 --2- 172.21.15.28:0/777981242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f954d57350 0x55f954dd0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:52.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.069+0000 7f0532601740 1 --2- 172.21.15.28:0/777981242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f954d56a30 0x55f954d66760 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:52.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.069+0000 7f0532601740 1 --2- 172.21.15.28:0/777981242 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f954d52d50 0x55f954d66220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:52.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.069+0000 7f050ea00640 0 client.17211 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:57:52.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.069+0000 7f0532601740 1 -- 172.21.15.28:0/777981242 >> 172.21.15.28:0/777981242 conn(0x55f954d5d580 msgr2=0x55f954d5e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:52.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.069+0000 7f0532601740 1 -- 172.21.15.28:0/777981242 shutdown_connections 2024-04-23T14:57:52.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.069+0000 7f0532601740 1 -- 172.21.15.28:0/777981242 wait complete. 2024-04-23T14:57:52.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.070+0000 7f0532601740 20 client.17211 trim_cache size 0 max 16384 2024-04-23T14:57:52.071 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:57:52.071 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (115 inodes in the stack)", 2024-04-23T14:57:52.071 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-23T14:57:52.072 INFO:teuthology.orchestra.run.smithi028.stdout: "8d378d34-36a1-4f00-8fb3-4c63c1510ab4": { 2024-04-23T14:57:52.072 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-23T14:57:52.072 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "8d378d34-36a1-4f00-8fb3-4c63c1510ab4", 2024-04-23T14:57:52.072 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-23T14:57:52.072 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:57:52.072 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:57:52.072 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:57:52.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.070+0000 7f052e200640 1 -- 172.21.15.28:0/454519700 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f05100a3da0 msgr2=0x7f05100a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:52.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.070+0000 7f052e200640 1 --2- 172.21.15.28:0/454519700 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f05100a3da0 0x7f05100a6260 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x55f954d57ff0 tx=0x7f051c0f4840 comp rx=0 tx=0).stop 2024-04-23T14:57:52.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.070+0000 7f052e200640 1 -- 172.21.15.28:0/454519700 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0528104310 msgr2=0x7f05281a06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:57:52.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.070+0000 7f052e200640 1 --2- 172.21.15.28:0/454519700 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0528104310 0x7f05281a06b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f051c004770 tx=0x7f051c003c10 comp rx=0 tx=0).stop 2024-04-23T14:57:52.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.070+0000 7f052e200640 1 -- 172.21.15.28:0/454519700 shutdown_connections 2024-04-23T14:57:52.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.070+0000 7f052e200640 1 --2- 172.21.15.28:0/454519700 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f05100a3da0 0x7f05100a6260 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:52.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.070+0000 7f052e200640 1 --2- 172.21.15.28:0/454519700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05281093e0 0x7f05281a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:52.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.070+0000 7f052e200640 1 --2- 172.21.15.28:0/454519700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f052819a7c0 0x7f052819ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:52.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.070+0000 7f052e200640 1 --2- 172.21.15.28:0/454519700 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0528104310 0x7f05281a06b0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:57:52.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.070+0000 7f052e200640 1 -- 172.21.15.28:0/454519700 >> 172.21.15.28:0/454519700 conn(0x7f05280ffb10 msgr2=0x7f0528101e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:57:52.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.071+0000 7f052e200640 1 -- 172.21.15.28:0/454519700 shutdown_connections 2024-04-23T14:57:52.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:57:52.071+0000 7f052e200640 1 -- 172.21.15.28:0/454519700 wait complete. 2024-04-23T14:57:52.090 INFO:tasks.cephfs.filesystem:scrub status for tag:8d378d34-36a1-4f00-8fb3-4c63c1510ab4 - {'path': '/', 'tag': '8d378d34-36a1-4f00-8fb3-4c63c1510ab4', 'options': 'recursive,force'} 2024-04-23T14:57:52.822 INFO:tasks.workunit.client.0.smithi028.stdout: 10 3780 48.36 MB/sec warmup 8 sec latency 232.890 ms 2024-04-23T14:57:53.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:52 smithi038 ceph-mon[32348]: pgmap v432: 97 pgs: 97 active+clean; 616 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.4 MiB/s wr, 170 op/s 2024-04-23T14:57:53.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:52 smithi038 ceph-mon[32348]: scrub queued for path: / 2024-04-23T14:57:53.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:52 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:57:53.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:52 smithi038 ceph-mon[32348]: scrub summary: active paths [/] 2024-04-23T14:57:53.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:52 smithi028 ceph-mon[27810]: pgmap v432: 97 pgs: 97 active+clean; 616 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.4 MiB/s wr, 170 op/s 2024-04-23T14:57:53.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:52 smithi028 ceph-mon[27810]: scrub queued for path: / 2024-04-23T14:57:53.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:52 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:57:53.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:52 smithi028 ceph-mon[27810]: scrub summary: active paths [/] 2024-04-23T14:57:53.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:52 smithi105 ceph-mon[32555]: pgmap v432: 97 pgs: 97 active+clean; 616 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.4 MiB/s wr, 170 op/s 2024-04-23T14:57:53.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:52 smithi105 ceph-mon[32555]: scrub queued for path: / 2024-04-23T14:57:53.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:52 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:57:53.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:52 smithi105 ceph-mon[32555]: scrub summary: active paths [/] 2024-04-23T14:57:53.822 INFO:tasks.workunit.client.0.smithi028.stdout: 10 4007 43.12 MB/sec warmup 9 sec latency 254.323 ms 2024-04-23T14:57:54.822 INFO:tasks.workunit.client.0.smithi028.stdout: 10 4334 39.73 MB/sec warmup 10 sec latency 255.554 ms 2024-04-23T14:57:55.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:54 smithi038 ceph-mon[32348]: scrub summary: idle+waiting paths [/] 2024-04-23T14:57:55.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:54 smithi038 ceph-mon[32348]: pgmap v433: 97 pgs: 97 active+clean; 633 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.5 MiB/s wr, 277 op/s 2024-04-23T14:57:55.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:54 smithi028 ceph-mon[27810]: scrub summary: idle+waiting paths [/] 2024-04-23T14:57:55.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:54 smithi028 ceph-mon[27810]: pgmap v433: 97 pgs: 97 active+clean; 633 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.5 MiB/s wr, 277 op/s 2024-04-23T14:57:55.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:54 smithi105 ceph-mon[32555]: scrub summary: idle+waiting paths [/] 2024-04-23T14:57:55.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:54 smithi105 ceph-mon[32555]: pgmap v433: 97 pgs: 97 active+clean; 633 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.5 MiB/s wr, 277 op/s 2024-04-23T14:57:55.822 INFO:tasks.workunit.client.0.smithi028.stdout: 10 4887 40.04 MB/sec warmup 11 sec latency 82.598 ms 2024-04-23T14:57:56.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:55 smithi105 ceph-mon[32555]: scrub summary: idle 2024-04-23T14:57:56.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:55 smithi038 ceph-mon[32348]: scrub summary: idle 2024-04-23T14:57:56.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:55 smithi028 ceph-mon[27810]: scrub summary: idle 2024-04-23T14:57:56.822 INFO:tasks.workunit.client.0.smithi028.stdout: 10 5714 38.52 MB/sec warmup 12 sec latency 116.036 ms 2024-04-23T14:57:57.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:56 smithi105 ceph-mon[32555]: pgmap v434: 97 pgs: 97 active+clean; 633 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 210 op/s 2024-04-23T14:57:57.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:56 smithi038 ceph-mon[32348]: pgmap v434: 97 pgs: 97 active+clean; 633 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 210 op/s 2024-04-23T14:57:57.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:56 smithi028 ceph-mon[27810]: pgmap v434: 97 pgs: 97 active+clean; 633 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 210 op/s 2024-04-23T14:57:57.823 INFO:tasks.workunit.client.0.smithi028.stdout: 10 6167 38.83 MB/sec warmup 13 sec latency 121.475 ms 2024-04-23T14:57:58.823 INFO:tasks.workunit.client.0.smithi028.stdout: 10 6750 39.62 MB/sec warmup 14 sec latency 61.936 ms 2024-04-23T14:57:59.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:57:58 smithi105 ceph-mon[32555]: pgmap v435: 97 pgs: 97 active+clean; 572 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.1 MiB/s wr, 325 op/s 2024-04-23T14:57:59.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:57:58 smithi038 ceph-mon[32348]: pgmap v435: 97 pgs: 97 active+clean; 572 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.1 MiB/s wr, 325 op/s 2024-04-23T14:57:59.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:57:58 smithi028 ceph-mon[27810]: pgmap v435: 97 pgs: 97 active+clean; 572 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 6.1 MiB/s wr, 325 op/s 2024-04-23T14:57:59.823 INFO:tasks.workunit.client.0.smithi028.stdout: 10 7046 38.00 MB/sec warmup 15 sec latency 47.394 ms 2024-04-23T14:58:00.823 INFO:tasks.workunit.client.0.smithi028.stdout: 10 7213 35.84 MB/sec warmup 16 sec latency 137.375 ms 2024-04-23T14:58:01.285 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:00 smithi105 ceph-mon[32555]: pgmap v436: 97 pgs: 97 active+clean; 572 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.3 MiB/s wr, 222 op/s 2024-04-23T14:58:01.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:00 smithi038 ceph-mon[32348]: pgmap v436: 97 pgs: 97 active+clean; 572 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.3 MiB/s wr, 222 op/s 2024-04-23T14:58:01.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:00 smithi028 ceph-mon[27810]: pgmap v436: 97 pgs: 97 active+clean; 572 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.3 MiB/s wr, 222 op/s 2024-04-23T14:58:01.823 INFO:tasks.workunit.client.0.smithi028.stdout: 10 7389 33.83 MB/sec warmup 17 sec latency 134.171 ms 2024-04-23T14:58:02.823 INFO:tasks.workunit.client.0.smithi028.stdout: 10 7577 32.15 MB/sec warmup 18 sec latency 349.407 ms 2024-04-23T14:58:03.285 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:02 smithi105 ceph-mon[32555]: pgmap v437: 97 pgs: 97 active+clean; 572 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.3 MiB/s wr, 222 op/s 2024-04-23T14:58:03.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:02 smithi038 ceph-mon[32348]: pgmap v437: 97 pgs: 97 active+clean; 572 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.3 MiB/s wr, 222 op/s 2024-04-23T14:58:03.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:02 smithi028 ceph-mon[27810]: pgmap v437: 97 pgs: 97 active+clean; 572 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.3 MiB/s wr, 222 op/s 2024-04-23T14:58:03.823 INFO:tasks.workunit.client.0.smithi028.stdout: 10 7838 31.11 MB/sec warmup 19 sec latency 112.868 ms 2024-04-23T14:58:04.824 INFO:tasks.workunit.client.0.smithi028.stdout: 10 8230 30.41 MB/sec warmup 20 sec latency 106.510 ms 2024-04-23T14:58:05.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:04 smithi105 ceph-mon[32555]: pgmap v438: 97 pgs: 97 active+clean; 583 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 6.2 MiB/s wr, 342 op/s 2024-04-23T14:58:05.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:04 smithi038 ceph-mon[32348]: pgmap v438: 97 pgs: 97 active+clean; 583 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 6.2 MiB/s wr, 342 op/s 2024-04-23T14:58:05.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:04 smithi028 ceph-mon[27810]: pgmap v438: 97 pgs: 97 active+clean; 583 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 6.2 MiB/s wr, 342 op/s 2024-04-23T14:58:05.823 INFO:tasks.workunit.client.0.smithi028.stdout: 10 8956 30.41 MB/sec warmup 21 sec latency 55.654 ms 2024-04-23T14:58:06.824 INFO:tasks.workunit.client.0.smithi028.stdout: 10 9506 30.64 MB/sec warmup 22 sec latency 48.036 ms 2024-04-23T14:58:07.174 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:06 smithi038 ceph-mon[32348]: pgmap v439: 97 pgs: 97 active+clean; 583 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.1 MiB/s wr, 235 op/s 2024-04-23T14:58:07.174 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:06 smithi038 ceph-mon[32348]: osdmap e103: 12 total, 12 up, 12 in 2024-04-23T14:58:07.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:06 smithi105 ceph-mon[32555]: pgmap v439: 97 pgs: 97 active+clean; 583 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.1 MiB/s wr, 235 op/s 2024-04-23T14:58:07.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:06 smithi105 ceph-mon[32555]: osdmap e103: 12 total, 12 up, 12 in 2024-04-23T14:58:07.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:06 smithi028 ceph-mon[27810]: pgmap v439: 97 pgs: 97 active+clean; 583 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.1 MiB/s wr, 235 op/s 2024-04-23T14:58:07.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:06 smithi028 ceph-mon[27810]: osdmap e103: 12 total, 12 up, 12 in 2024-04-23T14:58:07.824 INFO:tasks.workunit.client.0.smithi028.stdout: 10 9988 31.14 MB/sec warmup 23 sec latency 210.237 ms 2024-04-23T14:58:08.824 INFO:tasks.workunit.client.0.smithi028.stdout: 10 10448 31.43 MB/sec warmup 24 sec latency 207.041 ms 2024-04-23T14:58:09.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:08 smithi105 ceph-mon[32555]: pgmap v441: 97 pgs: 97 active+clean; 606 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.6 MiB/s wr, 496 op/s 2024-04-23T14:58:09.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:08 smithi038 ceph-mon[32348]: pgmap v441: 97 pgs: 97 active+clean; 606 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.6 MiB/s wr, 496 op/s 2024-04-23T14:58:09.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:08 smithi028 ceph-mon[27810]: pgmap v441: 97 pgs: 97 active+clean; 606 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.6 MiB/s wr, 496 op/s 2024-04-23T14:58:09.824 INFO:tasks.workunit.client.0.smithi028.stdout: 10 10649 30.49 MB/sec warmup 25 sec latency 45.734 ms 2024-04-23T14:58:10.824 INFO:tasks.workunit.client.0.smithi028.stdout: 10 10828 29.41 MB/sec warmup 26 sec latency 146.292 ms 2024-04-23T14:58:11.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:10 smithi105 ceph-mon[32555]: pgmap v442: 97 pgs: 97 active+clean; 606 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.6 MiB/s wr, 496 op/s 2024-04-23T14:58:11.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:10 smithi038 ceph-mon[32348]: pgmap v442: 97 pgs: 97 active+clean; 606 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.6 MiB/s wr, 496 op/s 2024-04-23T14:58:11.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:10 smithi028 ceph-mon[27810]: pgmap v442: 97 pgs: 97 active+clean; 606 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.6 MiB/s wr, 496 op/s 2024-04-23T14:58:11.824 INFO:tasks.workunit.client.0.smithi028.stdout: 10 11034 28.46 MB/sec warmup 27 sec latency 241.826 ms 2024-04-23T14:58:12.052 INFO:tasks.workunit.client.0.smithi028.stdout:[7958] write failed on handle 11572 (Resource temporarily unavailable) 2024-04-23T14:58:12.052 INFO:tasks.workunit.client.0.smithi028.stdout:Child failed with status 1 2024-04-23T14:58:12.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T14:58:12.074 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-04-23T14:58:12.075 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-23T14:58:13.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:12 smithi105 ceph-mon[32555]: pgmap v443: 97 pgs: 97 active+clean; 606 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.6 MiB/s wr, 496 op/s 2024-04-23T14:58:13.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:12 smithi105 ceph-mon[32555]: osdmap e104: 12 total, 12 up, 12 in 2024-04-23T14:58:13.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:12 smithi038 ceph-mon[32348]: pgmap v443: 97 pgs: 97 active+clean; 606 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.6 MiB/s wr, 496 op/s 2024-04-23T14:58:13.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:12 smithi038 ceph-mon[32348]: osdmap e104: 12 total, 12 up, 12 in 2024-04-23T14:58:13.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:12 smithi028 ceph-mon[27810]: pgmap v443: 97 pgs: 97 active+clean; 606 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.6 MiB/s wr, 496 op/s 2024-04-23T14:58:13.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:12 smithi028 ceph-mon[27810]: osdmap e104: 12 total, 12 up, 12 in 2024-04-23T14:58:13.217 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d44fddd910b5e94da0f2357af4be37d7bd495ae0/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_d44fddd910b5e94da0f2357af4be37d7bd495ae0/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d44fddd910b5e94da0f2357af4be37d7bd495ae0/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi028 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d44fddd910b5e94da0f2357af4be37d7bd495ae0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-04-23T14:58:13.464 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=5efad306a3fc46e281c6b2e55223662a Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d44fddd910b5e94da0f2357af4be37d7bd495ae0/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_d44fddd910b5e94da0f2357af4be37d7bd495ae0/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d44fddd910b5e94da0f2357af4be37d7bd495ae0/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi028 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d44fddd910b5e94da0f2357af4be37d7bd495ae0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-04-23T14:58:13.467 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-04-23T14:58:13.477 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-04-23T14:58:13.478 INFO:teuthology.task.exec:Executing custom commands... 2024-04-23T14:58:13.478 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-04-23T14:58:13.478 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-04-23T14:58:13.573 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-04-23T14:58:13.669 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-23T14:58:13.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.888+0000 7f36c1600640 1 -- 172.21.15.28:0/3541128004 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f36bc06e2f0 msgr2=0x7f36bc109000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:13.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.888+0000 7f36c1600640 1 --2- 172.21.15.28:0/3541128004 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f36bc06e2f0 0x7f36bc109000 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f36ac009ca0 tx=0x7f36ac02a8d0 comp rx=0 tx=0).stop 2024-04-23T14:58:13.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.889+0000 7f36c1600640 1 -- 172.21.15.28:0/3541128004 shutdown_connections 2024-04-23T14:58:13.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.889+0000 7f36c1600640 1 --2- 172.21.15.28:0/3541128004 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36bc1095d0 0x7f36bc109a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:13.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.889+0000 7f36c1600640 1 --2- 172.21.15.28:0/3541128004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36bc114d40 0x7f36bc117130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:13.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.889+0000 7f36c1600640 1 --2- 172.21.15.28:0/3541128004 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f36bc06e2f0 0x7f36bc109000 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:13.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.889+0000 7f36c1600640 1 -- 172.21.15.28:0/3541128004 >> 172.21.15.28:0/3541128004 conn(0x7f36bc06cf40 msgr2=0x7f36bc06d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:58:13.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.889+0000 7f36c1600640 1 -- 172.21.15.28:0/3541128004 shutdown_connections 2024-04-23T14:58:13.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.890+0000 7f36c1600640 1 -- 172.21.15.28:0/3541128004 wait complete. 2024-04-23T14:58:13.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.892+0000 7f36c1600640 1 Processor -- start 2024-04-23T14:58:13.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.893+0000 7f36c1600640 1 -- start start 2024-04-23T14:58:13.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.893+0000 7f36c1600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f36bc06e2f0 0x7f36bc1a3220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:13.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.893+0000 7f36c1600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36bc1095d0 0x7f36bc1a3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:13.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.893+0000 7f36c1600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36bc114d40 0x7f36bc1a7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:13.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.893+0000 7f36c1600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f36bc11aa60 con 0x7f36bc1095d0 2024-04-23T14:58:13.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.893+0000 7f36c1600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f36bc11a8e0 con 0x7f36bc114d40 2024-04-23T14:58:13.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.893+0000 7f36c1600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f36bc11abe0 con 0x7f36bc06e2f0 2024-04-23T14:58:13.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.894+0000 7f36bbe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f36bc06e2f0 0x7f36bc1a3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:13.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.894+0000 7f36bbe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f36bc06e2f0 0x7f36bc1a3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:46386/0 (socket says 172.21.15.28:46386) 2024-04-23T14:58:13.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.894+0000 7f36bbe00640 1 -- 172.21.15.28:0/754695194 learned_addr learned my addr 172.21.15.28:0/754695194 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:58:13.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.894+0000 7f36c0c00640 1 --2- 172.21.15.28:0/754695194 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36bc114d40 0x7f36bc1a7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:13.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.894+0000 7f36bbe00640 1 -- 172.21.15.28:0/754695194 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36bc114d40 msgr2=0x7f36bc1a7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:13.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.894+0000 7f36bbe00640 1 --2- 172.21.15.28:0/754695194 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36bc114d40 0x7f36bc1a7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:13.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.894+0000 7f36bbe00640 1 -- 172.21.15.28:0/754695194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36bc1095d0 msgr2=0x7f36bc1a3760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T14:58:13.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.895+0000 7f36bbe00640 1 --2- 172.21.15.28:0/754695194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36bc1095d0 0x7f36bc1a3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:13.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.895+0000 7f36bbe00640 1 -- 172.21.15.28:0/754695194 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36bc1a8210 con 0x7f36bc06e2f0 2024-04-23T14:58:13.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.895+0000 7f36c0c00640 1 --2- 172.21.15.28:0/754695194 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36bc114d40 0x7f36bc1a7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:58:13.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.895+0000 7f36bbe00640 1 --2- 172.21.15.28:0/754695194 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f36bc06e2f0 0x7f36bc1a3220 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f36ac02ae40 tx=0x7f36ac0317d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:58:13.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.896+0000 7f36b9600640 1 -- 172.21.15.28:0/754695194 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ac003b70 con 0x7f36bc06e2f0 2024-04-23T14:58:13.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.896+0000 7f36b9600640 1 -- 172.21.15.28:0/754695194 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36ac003d10 con 0x7f36bc06e2f0 2024-04-23T14:58:13.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.896+0000 7f36b9600640 1 -- 172.21.15.28:0/754695194 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ac02fae0 con 0x7f36bc06e2f0 2024-04-23T14:58:13.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.896+0000 7f36c1600640 1 -- 172.21.15.28:0/754695194 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36bc1a84a0 con 0x7f36bc06e2f0 2024-04-23T14:58:13.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.896+0000 7f36c1600640 1 -- 172.21.15.28:0/754695194 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36bc1afd90 con 0x7f36bc06e2f0 2024-04-23T14:58:13.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.898+0000 7f36c1600640 1 -- 172.21.15.28:0/754695194 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36bc0505e0 con 0x7f36bc06e2f0 2024-04-23T14:58:13.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.902+0000 7f36b9600640 1 -- 172.21.15.28:0/754695194 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f36ac02fc80 con 0x7f36bc06e2f0 2024-04-23T14:58:13.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.904+0000 7f36b9600640 1 --2- 172.21.15.28:0/754695194 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f36a00a3f20 0x7f36a00a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:13.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.904+0000 7f36b9600640 1 -- 172.21.15.28:0/754695194 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8542+0+0 (secure 0 0 0) 0x7f36ac0f1c30 con 0x7f36bc06e2f0 2024-04-23T14:58:13.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.905+0000 7f36b3400640 1 --2- 172.21.15.28:0/754695194 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f36a00a3f20 0x7f36a00a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:13.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.906+0000 7f36b3400640 1 --2- 172.21.15.28:0/754695194 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f36a00a3f20 0x7f36a00a63e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f36bc10bf60 tx=0x7f36a4009630 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:58:13.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:13.908+0000 7f36b9600640 1 -- 172.21.15.28:0/754695194 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f36ac0be370 con 0x7f36bc06e2f0 2024-04-23T14:58:14.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.137+0000 7f36c1600640 1 -- 172.21.15.28:0/754695194 --> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f36bc073010 con 0x7f36a00a3f20 2024-04-23T14:58:14.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.153+0000 7f36b9600640 1 -- 172.21.15.28:0/754695194 <== mgr.25153 v2:172.21.15.28:6800/1615546338 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7f36bc073010 con 0x7f36a00a3f20 2024-04-23T14:58:14.153 INFO:teuthology.orchestra.run.smithi028.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-23T00:00:00", "created": "2024-04-23T14:44:09", "first": "2024-04-23T14:45:00", "last": "2024-04-23T14:58:00", "last_pruned": "2024-04-23T14:58:00", "created_count": 14, "pruned_count": 8, "active": true} 2024-04-23T14:58:14.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.156+0000 7f36c1600640 1 -- 172.21.15.28:0/754695194 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f36a00a3f20 msgr2=0x7f36a00a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:14.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.156+0000 7f36c1600640 1 --2- 172.21.15.28:0/754695194 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f36a00a3f20 0x7f36a00a63e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f36bc10bf60 tx=0x7f36a4009630 comp rx=0 tx=0).stop 2024-04-23T14:58:14.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.156+0000 7f36c1600640 1 -- 172.21.15.28:0/754695194 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f36bc06e2f0 msgr2=0x7f36bc1a3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:14.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.156+0000 7f36c1600640 1 --2- 172.21.15.28:0/754695194 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f36bc06e2f0 0x7f36bc1a3220 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f36ac02ae40 tx=0x7f36ac0317d0 comp rx=0 tx=0).stop 2024-04-23T14:58:14.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.156+0000 7f36c1600640 1 -- 172.21.15.28:0/754695194 shutdown_connections 2024-04-23T14:58:14.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.156+0000 7f36c1600640 1 --2- 172.21.15.28:0/754695194 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f36a00a3f20 0x7f36a00a63e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:14.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.157+0000 7f36c1600640 1 --2- 172.21.15.28:0/754695194 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36bc114d40 0x7f36bc1a7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:14.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.157+0000 7f36c1600640 1 --2- 172.21.15.28:0/754695194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36bc1095d0 0x7f36bc1a3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:14.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.157+0000 7f36c1600640 1 --2- 172.21.15.28:0/754695194 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f36bc06e2f0 0x7f36bc1a3220 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:14.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.157+0000 7f36c1600640 1 -- 172.21.15.28:0/754695194 >> 172.21.15.28:0/754695194 conn(0x7f36bc06cf40 msgr2=0x7f36bc072900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:58:14.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.157+0000 7f36c1600640 1 -- 172.21.15.28:0/754695194 shutdown_connections 2024-04-23T14:58:14.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.157+0000 7f36c1600640 1 -- 172.21.15.28:0/754695194 wait complete. 2024-04-23T14:58:14.176 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-23T14:58:14.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.380+0000 7f2cff200640 1 -- 172.21.15.28:0/3303137980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d00076530 msgr2=0x7f2d00076910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:14.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.380+0000 7f2cff200640 1 --2- 172.21.15.28:0/3303137980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d00076530 0x7f2d00076910 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0009900 tx=0x7f2cf002a3f0 comp rx=0 tx=0).stop 2024-04-23T14:58:14.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.380+0000 7f2cff200640 1 -- 172.21.15.28:0/3303137980 shutdown_connections 2024-04-23T14:58:14.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.380+0000 7f2cff200640 1 --2- 172.21.15.28:0/3303137980 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d00112cb0 0x7f2d001150a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:14.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.380+0000 7f2cff200640 1 --2- 172.21.15.28:0/3303137980 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d00076e50 0x7f2d00112770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:14.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.380+0000 7f2cff200640 1 --2- 172.21.15.28:0/3303137980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d00076530 0x7f2d00076910 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:14.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.380+0000 7f2cff200640 1 -- 172.21.15.28:0/3303137980 >> 172.21.15.28:0/3303137980 conn(0x7f2d000ffb20 msgr2=0x7f2d00101f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:58:14.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.380+0000 7f2cff200640 1 -- 172.21.15.28:0/3303137980 shutdown_connections 2024-04-23T14:58:14.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.380+0000 7f2cff200640 1 -- 172.21.15.28:0/3303137980 wait complete. 2024-04-23T14:58:14.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.381+0000 7f2cff200640 1 Processor -- start 2024-04-23T14:58:14.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.381+0000 7f2cff200640 1 -- start start 2024-04-23T14:58:14.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.382+0000 7f2cff200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d00076530 0x7f2d001a4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:14.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.382+0000 7f2cff200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d00076e50 0x7f2d001a4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:14.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.382+0000 7f2cff200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d00112cb0 0x7f2d0019eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:14.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.382+0000 7f2cff200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2d00118800 con 0x7f2d00112cb0 2024-04-23T14:58:14.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.382+0000 7f2cff200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2d00118680 con 0x7f2d00076e50 2024-04-23T14:58:14.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.382+0000 7f2cff200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2d00118980 con 0x7f2d00076530 2024-04-23T14:58:14.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.382+0000 7f2cfe800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d00112cb0 0x7f2d0019eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:14.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.382+0000 7f2cfe800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d00112cb0 0x7f2d0019eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48118/0 (socket says 172.21.15.28:48118) 2024-04-23T14:58:14.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.382+0000 7f2cfe800640 1 -- 172.21.15.28:0/689111784 learned_addr learned my addr 172.21.15.28:0/689111784 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:58:14.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.382+0000 7f2cfd400640 1 --2- 172.21.15.28:0/689111784 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d00076e50 0x7f2d001a4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:14.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.382+0000 7f2cfde00640 1 --2- 172.21.15.28:0/689111784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d00076530 0x7f2d001a4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:14.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.383+0000 7f2cfe800640 1 -- 172.21.15.28:0/689111784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d00076530 msgr2=0x7f2d001a4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:14.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.383+0000 7f2cfe800640 1 --2- 172.21.15.28:0/689111784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d00076530 0x7f2d001a4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:14.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.383+0000 7f2cfe800640 1 -- 172.21.15.28:0/689111784 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d00076e50 msgr2=0x7f2d001a4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:14.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.383+0000 7f2cfe800640 1 --2- 172.21.15.28:0/689111784 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d00076e50 0x7f2d001a4f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:14.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.383+0000 7f2cfe800640 1 -- 172.21.15.28:0/689111784 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d0019f420 con 0x7f2d00112cb0 2024-04-23T14:58:14.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.383+0000 7f2cfe800640 1 --2- 172.21.15.28:0/689111784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d00112cb0 0x7f2d0019eb60 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2cf800eb40 tx=0x7f2cf800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:58:14.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.383+0000 7f2cf6a00640 1 -- 172.21.15.28:0/689111784 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf8004b20 con 0x7f2d00112cb0 2024-04-23T14:58:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.383+0000 7f2cf6a00640 1 -- 172.21.15.28:0/689111784 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cf8004cc0 con 0x7f2d00112cb0 2024-04-23T14:58:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.383+0000 7f2cff200640 1 -- 172.21.15.28:0/689111784 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d0019f710 con 0x7f2d00112cb0 2024-04-23T14:58:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.383+0000 7f2cf6a00640 1 -- 172.21.15.28:0/689111784 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf8010850 con 0x7f2d00112cb0 2024-04-23T14:58:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.383+0000 7f2cff200640 1 -- 172.21.15.28:0/689111784 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d001ab9d0 con 0x7f2d00112cb0 2024-04-23T14:58:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.385+0000 7f2cff200640 1 -- 172.21.15.28:0/689111784 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d00077c70 con 0x7f2d00112cb0 2024-04-23T14:58:14.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.387+0000 7f2cf6a00640 1 -- 172.21.15.28:0/689111784 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f2cf8005020 con 0x7f2d00112cb0 2024-04-23T14:58:14.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.388+0000 7f2cf6a00640 1 --2- 172.21.15.28:0/689111784 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f2ce40a3d90 0x7f2ce40a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:14.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.388+0000 7f2cf6a00640 1 -- 172.21.15.28:0/689111784 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8542+0+0 (secure 0 0 0) 0x7f2cf80d0f90 con 0x7f2d00112cb0 2024-04-23T14:58:14.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.389+0000 7f2cfde00640 1 --2- 172.21.15.28:0/689111784 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f2ce40a3d90 0x7f2ce40a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:14.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.389+0000 7f2cfde00640 1 --2- 172.21.15.28:0/689111784 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f2ce40a3d90 0x7f2ce40a6250 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0009230 tx=0x7f2cf00058d0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:58:14.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.393+0000 7f2cf6a00640 1 -- 172.21.15.28:0/689111784 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2cf809d5c0 con 0x7f2d00112cb0 2024-04-23T14:58:14.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.624+0000 7f2cff200640 1 -- 172.21.15.28:0/689111784 --> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f2d00062440 con 0x7f2ce40a3d90 2024-04-23T14:58:14.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.639+0000 7f2cf6a00640 1 -- 172.21.15.28:0/689111784 <== mgr.25153 v2:172.21.15.28:6800/1615546338 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f2d00062440 con 0x7f2ce40a3d90 2024-04-23T14:58:14.640 INFO:teuthology.orchestra.run.smithi028.stdout:/ 1m 6m3h 2024-04-23T14:58:14.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.642+0000 7f2cff200640 1 -- 172.21.15.28:0/689111784 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f2ce40a3d90 msgr2=0x7f2ce40a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:14.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.642+0000 7f2cff200640 1 --2- 172.21.15.28:0/689111784 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f2ce40a3d90 0x7f2ce40a6250 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0009230 tx=0x7f2cf00058d0 comp rx=0 tx=0).stop 2024-04-23T14:58:14.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.642+0000 7f2cff200640 1 -- 172.21.15.28:0/689111784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d00112cb0 msgr2=0x7f2d0019eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:14.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.642+0000 7f2cff200640 1 --2- 172.21.15.28:0/689111784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d00112cb0 0x7f2d0019eb60 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2cf800eb40 tx=0x7f2cf800c2d0 comp rx=0 tx=0).stop 2024-04-23T14:58:14.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.643+0000 7f2cff200640 1 -- 172.21.15.28:0/689111784 shutdown_connections 2024-04-23T14:58:14.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.643+0000 7f2cff200640 1 --2- 172.21.15.28:0/689111784 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f2ce40a3d90 0x7f2ce40a6250 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:14.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.643+0000 7f2cff200640 1 --2- 172.21.15.28:0/689111784 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d00076e50 0x7f2d001a4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:14.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.643+0000 7f2cff200640 1 --2- 172.21.15.28:0/689111784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d00112cb0 0x7f2d0019eb60 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:14.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.643+0000 7f2cff200640 1 --2- 172.21.15.28:0/689111784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d00076530 0x7f2d001a4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:14.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.643+0000 7f2cff200640 1 -- 172.21.15.28:0/689111784 >> 172.21.15.28:0/689111784 conn(0x7f2d000ffb20 msgr2=0x7f2d00101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:58:14.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.643+0000 7f2cff200640 1 -- 172.21.15.28:0/689111784 shutdown_connections 2024-04-23T14:58:14.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:14.643+0000 7f2cff200640 1 -- 172.21.15.28:0/689111784 wait complete. 2024-04-23T14:58:14.663 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-04-23T14:58:14.674 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-04-23T14:58:15.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:14 smithi038 ceph-mon[32348]: pgmap v445: 97 pgs: 97 active+clean; 622 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 7.3 MiB/s wr, 592 op/s 2024-04-23T14:58:15.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:14 smithi028 ceph-mon[27810]: pgmap v445: 97 pgs: 97 active+clean; 622 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 7.3 MiB/s wr, 592 op/s 2024-04-23T14:58:15.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:14 smithi105 ceph-mon[32555]: pgmap v445: 97 pgs: 97 active+clean; 622 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 7.3 MiB/s wr, 592 op/s 2024-04-23T14:58:16.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:15 smithi038 ceph-mon[32348]: from='client.27400 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:58:16.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:15 smithi038 ceph-mon[32348]: from='client.17220 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:58:16.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:15 smithi028 ceph-mon[27810]: from='client.27400 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:58:16.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:15 smithi028 ceph-mon[27810]: from='client.17220 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:58:16.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:15 smithi105 ceph-mon[32555]: from='client.27400 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:58:16.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:15 smithi105 ceph-mon[32555]: from='client.17220 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T14:58:17.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:16 smithi038 ceph-mon[32348]: pgmap v446: 97 pgs: 97 active+clean; 622 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.9 MiB/s wr, 483 op/s 2024-04-23T14:58:17.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:16 smithi028 ceph-mon[27810]: pgmap v446: 97 pgs: 97 active+clean; 622 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.9 MiB/s wr, 483 op/s 2024-04-23T14:58:17.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:16 smithi105 ceph-mon[32555]: pgmap v446: 97 pgs: 97 active+clean; 622 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.9 MiB/s wr, 483 op/s 2024-04-23T14:58:19.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:18 smithi038 ceph-mon[32348]: pgmap v447: 97 pgs: 97 active+clean; 645 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.6 MiB/s wr, 178 op/s 2024-04-23T14:58:19.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:18 smithi028 ceph-mon[27810]: pgmap v447: 97 pgs: 97 active+clean; 645 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.6 MiB/s wr, 178 op/s 2024-04-23T14:58:19.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:18 smithi105 ceph-mon[32555]: pgmap v447: 97 pgs: 97 active+clean; 645 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.6 MiB/s wr, 178 op/s 2024-04-23T14:58:21.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:20 smithi038 ceph-mon[32348]: pgmap v448: 97 pgs: 97 active+clean; 645 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.6 MiB/s wr, 178 op/s 2024-04-23T14:58:21.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:20 smithi028 ceph-mon[27810]: pgmap v448: 97 pgs: 97 active+clean; 645 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.6 MiB/s wr, 178 op/s 2024-04-23T14:58:21.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:20 smithi105 ceph-mon[32555]: pgmap v448: 97 pgs: 97 active+clean; 645 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.6 MiB/s wr, 178 op/s 2024-04-23T14:58:22.091 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-23T14:58:22.091 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-23T14:58:22.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.318+0000 7f3a35c00640 1 -- 172.21.15.28:0/15261143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a3010e9b0 msgr2=0x7f3a30110da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:22.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.318+0000 7f3a35c00640 1 --2- 172.21.15.28:0/15261143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a3010e9b0 0x7f3a30110da0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3a28009900 tx=0x7f3a2802a3f0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.318+0000 7f3a35c00640 1 -- 172.21.15.28:0/15261143 shutdown_connections 2024-04-23T14:58:22.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.318+0000 7f3a35c00640 1 --2- 172.21.15.28:0/15261143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a3010e9b0 0x7f3a30110da0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.318+0000 7f3a35c00640 1 --2- 172.21.15.28:0/15261143 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a30101720 0x7f3a3010e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.318+0000 7f3a35c00640 1 --2- 172.21.15.28:0/15261143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a30100e00 0x7f3a301011e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.318+0000 7f3a35c00640 1 -- 172.21.15.28:0/15261143 >> 172.21.15.28:0/15261143 conn(0x7f3a30078000 msgr2=0x7f3a30078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:58:22.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.319+0000 7f3a35c00640 1 -- 172.21.15.28:0/15261143 shutdown_connections 2024-04-23T14:58:22.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.319+0000 7f3a35c00640 1 -- 172.21.15.28:0/15261143 wait complete. 2024-04-23T14:58:22.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.320+0000 7f3a35c00640 1 Processor -- start 2024-04-23T14:58:22.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.320+0000 7f3a35c00640 1 -- start start 2024-04-23T14:58:22.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.320+0000 7f3a35c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a30100e00 0x7f3a301a06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:22.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.320+0000 7f3a35c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a30101720 0x7f3a301a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:22.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.320+0000 7f3a35c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a3010e9b0 0x7f3a3019a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:22.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.320+0000 7f3a35c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3a301145c0 con 0x7f3a30100e00 2024-04-23T14:58:22.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.320+0000 7f3a35c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3a30114440 con 0x7f3a30101720 2024-04-23T14:58:22.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.320+0000 7f3a35c00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3a30114740 con 0x7f3a3010e9b0 2024-04-23T14:58:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.320+0000 7f3a2ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a30100e00 0x7f3a301a06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.321+0000 7f3a2f400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a3010e9b0 0x7f3a3019a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.321+0000 7f3a2e000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a30101720 0x7f3a301a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.321+0000 7f3a2ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a30100e00 0x7f3a301a06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34574/0 (socket says 172.21.15.28:34574) 2024-04-23T14:58:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.321+0000 7f3a2ea00640 1 -- 172.21.15.28:0/2925664789 learned_addr learned my addr 172.21.15.28:0/2925664789 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:58:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.321+0000 7f3a2ea00640 1 -- 172.21.15.28:0/2925664789 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a3010e9b0 msgr2=0x7f3a3019a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.321+0000 7f3a2ea00640 1 --2- 172.21.15.28:0/2925664789 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a3010e9b0 0x7f3a3019a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.321+0000 7f3a2ea00640 1 -- 172.21.15.28:0/2925664789 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a30101720 msgr2=0x7f3a301a0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.321+0000 7f3a2ea00640 1 --2- 172.21.15.28:0/2925664789 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a30101720 0x7f3a301a0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.321+0000 7f3a2ea00640 1 -- 172.21.15.28:0/2925664789 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a3019b060 con 0x7f3a30100e00 2024-04-23T14:58:22.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.321+0000 7f3a2f400640 1 --2- 172.21.15.28:0/2925664789 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a3010e9b0 0x7f3a3019a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:58:22.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.321+0000 7f3a2ea00640 1 --2- 172.21.15.28:0/2925664789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a30100e00 0x7f3a301a06e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f3a2000ebc0 tx=0x7f3a2000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:58:22.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.322+0000 7f3a27400640 1 -- 172.21.15.28:0/2925664789 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a20010630 con 0x7f3a30100e00 2024-04-23T14:58:22.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.322+0000 7f3a27400640 1 -- 172.21.15.28:0/2925664789 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a20010cb0 con 0x7f3a30100e00 2024-04-23T14:58:22.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.322+0000 7f3a27400640 1 -- 172.21.15.28:0/2925664789 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a20018bd0 con 0x7f3a30100e00 2024-04-23T14:58:22.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.322+0000 7f3a35c00640 1 -- 172.21.15.28:0/2925664789 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a3019b350 con 0x7f3a30100e00 2024-04-23T14:58:22.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.322+0000 7f3a35c00640 1 -- 172.21.15.28:0/2925664789 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a301a75a0 con 0x7f3a30100e00 2024-04-23T14:58:22.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.325+0000 7f3a389f9740 1 Processor -- start 2024-04-23T14:58:22.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.325+0000 7f3a27400640 1 -- 172.21.15.28:0/2925664789 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f3a200107d0 con 0x7f3a30100e00 2024-04-23T14:58:22.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.325+0000 7f3a389f9740 1 -- start start 2024-04-23T14:58:22.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.326+0000 7f3a389f9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb4b755b0 0x55ecb4b75990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:22.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.326+0000 7f3a389f9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ecb4b75f60 0x55ecb4b88b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:22.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.326+0000 7f3a389f9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ecb4b89060 0x55ecb4b8b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:22.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.326+0000 7f3a389f9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ecb4adb1f0 con 0x55ecb4b755b0 2024-04-23T14:58:22.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.326+0000 7f3a389f9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ecb4a81350 con 0x55ecb4b89060 2024-04-23T14:58:22.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.326+0000 7f3a389f9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55ecb4ad71b0 con 0x55ecb4b75f60 2024-04-23T14:58:22.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.326+0000 7f3a2ea00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ecb4b75f60 0x55ecb4b88b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:22.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.326+0000 7f3a2f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb4b755b0 0x55ecb4b75990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:22.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.326+0000 7f3a2f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb4b755b0 0x55ecb4b75990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34576/0 (socket says 172.21.15.28:34576) 2024-04-23T14:58:22.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.326+0000 7f3a2f400640 1 -- 172.21.15.28:0/3244795398 learned_addr learned my addr 172.21.15.28:0/3244795398 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:58:22.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.326+0000 7f3a2e000640 1 --2- 172.21.15.28:0/3244795398 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ecb4b89060 0x55ecb4b8b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.327+0000 7f3a2ea00640 1 -- 172.21.15.28:0/3244795398 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ecb4b89060 msgr2=0x55ecb4b8b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:58:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.327+0000 7f3a2ea00640 1 --2- 172.21.15.28:0/3244795398 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ecb4b89060 0x55ecb4b8b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.327+0000 7f3a2ea00640 1 -- 172.21.15.28:0/3244795398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb4b755b0 msgr2=0x55ecb4b75990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:58:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.327+0000 7f3a2ea00640 1 --2- 172.21.15.28:0/3244795398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb4b755b0 0x55ecb4b75990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.327+0000 7f3a2ea00640 1 -- 172.21.15.28:0/3244795398 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecb4ad5f50 con 0x55ecb4b75f60 2024-04-23T14:58:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.327+0000 7f3a2ea00640 1 --2- 172.21.15.28:0/3244795398 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ecb4b75f60 0x55ecb4b88b20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3a200ce540 tx=0x7f3a200d94b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=449d4d0ad3e5f183 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:58:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.327+0000 7f3a1be00640 1 -- 172.21.15.28:0/3244795398 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a200e4070 con 0x55ecb4b75f60 2024-04-23T14:58:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.327+0000 7f3a1be00640 1 -- 172.21.15.28:0/3244795398 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a200dfae0 con 0x55ecb4b75f60 2024-04-23T14:58:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.327+0000 7f3a1be00640 1 -- 172.21.15.28:0/3244795398 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a200dea80 con 0x55ecb4b75f60 2024-04-23T14:58:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.328+0000 7f3a27400640 1 --2- 172.21.15.28:0/2925664789 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f3a140a3d20 0x7f3a140a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.328+0000 7f3a27400640 1 -- 172.21.15.28:0/2925664789 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8542+0+0 (secure 0 0 0) 0x7f3a200d0e60 con 0x7f3a30100e00 2024-04-23T14:58:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.328+0000 7f3a2f400640 1 --2- 172.21.15.28:0/2925664789 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f3a140a3d20 0x7f3a140a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.328+0000 7f3a389f9740 1 -- 172.21.15.28:0/3244795398 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ecb4b75f60 msgr2=0x55ecb4b88b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.328+0000 7f3a389f9740 1 --2- 172.21.15.28:0/3244795398 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ecb4b75f60 0x55ecb4b88b20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3a200ce540 tx=0x7f3a200d94b0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.328+0000 7f3a389f9740 1 -- 172.21.15.28:0/3244795398 shutdown_connections 2024-04-23T14:58:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.328+0000 7f3a389f9740 1 --2- 172.21.15.28:0/3244795398 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ecb4b89060 0x55ecb4b8b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.328+0000 7f3a389f9740 1 --2- 172.21.15.28:0/3244795398 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ecb4b75f60 0x55ecb4b88b20 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.328+0000 7f3a389f9740 1 --2- 172.21.15.28:0/3244795398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb4b755b0 0x55ecb4b75990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.328+0000 7f3a389f9740 1 -- 172.21.15.28:0/3244795398 >> 172.21.15.28:0/3244795398 conn(0x55ecb4b71b80 msgr2=0x55ecb4b71f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:58:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.329+0000 7f3a389f9740 1 -- 172.21.15.28:0/3244795398 shutdown_connections 2024-04-23T14:58:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.329+0000 7f3a389f9740 1 -- 172.21.15.28:0/3244795398 wait complete. 2024-04-23T14:58:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.329+0000 7f3a2f400640 1 --2- 172.21.15.28:0/2925664789 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f3a140a3d20 0x7f3a140a61e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3a28009900 tx=0x7f3a28009560 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:58:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.329+0000 7f3a389f9740 1 -- start start 2024-04-23T14:58:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.329+0000 7f3a389f9740 1 Processor -- start 2024-04-23T14:58:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.330+0000 7f3a389f9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb4b755b0 0x55ecb4b86020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.330+0000 7f3a389f9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ecb4b75f60 0x55ecb4b86560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.330+0000 7f3a389f9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ecb4b89060 0x55ecb4b86aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.330+0000 7f3a389f9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ecb4b8be20 con 0x55ecb4b755b0 2024-04-23T14:58:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.330+0000 7f3a389f9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ecb4a94690 con 0x55ecb4b75f60 2024-04-23T14:58:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.330+0000 7f3a389f9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55ecb4b8bfa0 con 0x55ecb4b89060 2024-04-23T14:58:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.330+0000 7f3a389f9740 10 client.0 _ll_register_callbacks cb 0x55ecb4ad0650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:58:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.330+0000 7f3a2ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb4b755b0 0x55ecb4b86020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.330+0000 7f3a2ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb4b755b0 0x55ecb4b86020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34584/0 (socket says 172.21.15.28:34584) 2024-04-23T14:58:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.330+0000 7f3a2ea00640 1 -- 172.21.15.28:0/3450357378 learned_addr learned my addr 172.21.15.28:0/3450357378 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:58:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.330+0000 7f3a2f400640 1 --2- 172.21.15.28:0/3450357378 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ecb4b89060 0x55ecb4b86aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:22.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.330+0000 7f3a2e000640 1 --2- 172.21.15.28:0/3450357378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ecb4b75f60 0x55ecb4b86560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:22.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.331+0000 7f3a2ea00640 1 -- 172.21.15.28:0/3450357378 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ecb4b89060 msgr2=0x55ecb4b86aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:58:22.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.331+0000 7f3a2ea00640 1 --2- 172.21.15.28:0/3450357378 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ecb4b89060 0x55ecb4b86aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.331+0000 7f3a2ea00640 1 -- 172.21.15.28:0/3450357378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ecb4b75f60 msgr2=0x55ecb4b86560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:58:22.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.331+0000 7f3a2ea00640 1 --2- 172.21.15.28:0/3450357378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ecb4b75f60 0x55ecb4b86560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.331+0000 7f3a2ea00640 1 -- 172.21.15.28:0/3450357378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecb4a93920 con 0x55ecb4b755b0 2024-04-23T14:58:22.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.331+0000 7f3a2ea00640 1 --2- 172.21.15.28:0/3450357378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb4b755b0 0x55ecb4b86020 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f3a200ce5a0 tx=0x7f3a20000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84288838f16ede95 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:58:22.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.331+0000 7f3a18c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-23T14:58:22.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.331+0000 7f3a18c00640 1 -- 172.21.15.28:0/3450357378 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a20010e60 con 0x55ecb4b755b0 2024-04-23T14:58:22.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.331+0000 7f3a18c00640 1 -- 172.21.15.28:0/3450357378 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8542+0+0 (secure 0 0 0) 0x7f3a200ce7c0 con 0x55ecb4b755b0 2024-04-23T14:58:22.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.331+0000 7f3a389f9740 1 -- 172.21.15.28:0/3450357378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a200dfae0 con 0x55ecb4b755b0 2024-04-23T14:58:22.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.332+0000 7f3a18c00640 1 -- 172.21.15.28:0/3450357378 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f3a200990e0 con 0x55ecb4b755b0 2024-04-23T14:58:22.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.332+0000 7f3a389f9740 10 client.17232 fetch_fsmap learned FSMap version 20 2024-04-23T14:58:22.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.332+0000 7f3a389f9740 1 -- 172.21.15.28:0/3450357378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55ecb48b77d0 con 0x55ecb4b755b0 2024-04-23T14:58:22.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.332+0000 7f3a18c00640 1 -- 172.21.15.28:0/3450357378 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a200dfae0 con 0x55ecb4b755b0 2024-04-23T14:58:22.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.332+0000 7f3a18c00640 1 -- 172.21.15.28:0/3450357378 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f3a2009d0a0 con 0x55ecb4b755b0 2024-04-23T14:58:22.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.332+0000 7f3a389f9740 10 client.17232 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:58:22.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.332+0000 7f3a389f9740 10 client.17232 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:58:22.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.332+0000 7f3a389f9740 20 client.17232 populate_metadata read hostname 'smithi028' 2024-04-23T14:58:22.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.332+0000 7f3a389f9740 1 --2- 172.21.15.28:0/3450357378 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55ecb4b992b0 0x55ecb4b9b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:22.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.332+0000 7f3a389f9740 4 client.17232 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:58:22.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.332+0000 7f3a389f9740 1 -- 172.21.15.28:0/3450357378 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ecb4ae8c10 con 0x55ecb4b992b0 2024-04-23T14:58:22.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.333+0000 7f3a2e000640 1 --2- 172.21.15.28:0/3450357378 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55ecb4b992b0 0x55ecb4b9b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:22.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.333+0000 7f3a2e000640 1 --2- 172.21.15.28:0/3450357378 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55ecb4b992b0 0x55ecb4b9b6a0 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7b58eea029fd745 server_cookie=bf9ead0d166ec026 in_seq=0 out_seq=0 2024-04-23T14:58:22.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.333+0000 7f3a18c00640 10 client.17232 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:58:22.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.334+0000 7f3a18c00640 1 -- 172.21.15.28:0/3450357378 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ecb4ae8c10 con 0x55ecb4b992b0 2024-04-23T14:58:22.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.334+0000 7f3a18c00640 10 client.17232 handle_command_reply: tid=0 2024-04-23T14:58:22.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.334+0000 7f3a389f9740 1 client.17232 shutdown 2024-04-23T14:58:22.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.334+0000 7f3a389f9740 1 -- 172.21.15.28:0/3450357378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb4b755b0 msgr2=0x55ecb4b86020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:22.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.334+0000 7f3a389f9740 1 --2- 172.21.15.28:0/3450357378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb4b755b0 0x55ecb4b86020 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f3a200ce5a0 tx=0x7f3a20000f40 comp rx=0 tx=0).stop 2024-04-23T14:58:22.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.334+0000 7f3a389f9740 1 -- 172.21.15.28:0/3450357378 shutdown_connections 2024-04-23T14:58:22.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.334+0000 7f3a389f9740 1 --2- 172.21.15.28:0/3450357378 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55ecb4b992b0 0x55ecb4b9b6a0 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.334+0000 7f3a389f9740 1 --2- 172.21.15.28:0/3450357378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ecb4b75f60 0x55ecb4b86560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.334+0000 7f3a389f9740 1 --2- 172.21.15.28:0/3450357378 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ecb4b89060 0x55ecb4b86aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.334+0000 7f3a389f9740 1 --2- 172.21.15.28:0/3450357378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb4b755b0 0x55ecb4b86020 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.334+0000 7f3a18c00640 0 client.17232 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:58:22.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.334+0000 7f3a389f9740 1 -- 172.21.15.28:0/3450357378 >> 172.21.15.28:0/3450357378 conn(0x55ecb4b71b80 msgr2=0x55ecb4b77980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:58:22.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.334+0000 7f3a389f9740 1 -- 172.21.15.28:0/3450357378 shutdown_connections 2024-04-23T14:58:22.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.335+0000 7f3a389f9740 1 -- 172.21.15.28:0/3450357378 wait complete. 2024-04-23T14:58:22.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.335+0000 7f3a389f9740 20 client.17232 trim_cache size 0 max 16384 2024-04-23T14:58:22.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.354+0000 7f3a389f9740 1 Processor -- start 2024-04-23T14:58:22.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.355+0000 7f3a389f9740 1 -- start start 2024-04-23T14:58:22.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.355+0000 7f3a389f9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ecb4b755b0 0x55ecb4b77980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:22.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.355+0000 7f3a389f9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ecb4b75f60 0x55ecb4b9e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:22.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.355+0000 7f3a389f9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb4b89060 0x55ecb4b9ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:22.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.355+0000 7f3a389f9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3a301a9cf0 con 0x55ecb4b89060 2024-04-23T14:58:22.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.355+0000 7f3a389f9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3a301ac550 con 0x55ecb4b75f60 2024-04-23T14:58:22.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.355+0000 7f3a389f9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3a301aa340 con 0x55ecb4b755b0 2024-04-23T14:58:22.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.355+0000 7f3a2f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb4b89060 0x55ecb4b9ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:22.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.355+0000 7f3a2ea00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ecb4b755b0 0x55ecb4b77980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:22.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.355+0000 7f3a2e000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ecb4b75f60 0x55ecb4b9e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:22.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.355+0000 7f3a2ea00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ecb4b755b0 0x55ecb4b77980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:42854/0 (socket says 172.21.15.28:42854) 2024-04-23T14:58:22.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.355+0000 7f3a2ea00640 1 -- 172.21.15.28:0/2056473771 learned_addr learned my addr 172.21.15.28:0/2056473771 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:58:22.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.356+0000 7f3a2e000640 1 -- 172.21.15.28:0/2056473771 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ecb4b755b0 msgr2=0x55ecb4b77980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:58:22.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.356+0000 7f3a2e000640 1 --2- 172.21.15.28:0/2056473771 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ecb4b755b0 0x55ecb4b77980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.356+0000 7f3a2e000640 1 -- 172.21.15.28:0/2056473771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb4b89060 msgr2=0x55ecb4b9ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:58:22.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.356+0000 7f3a2e000640 1 --2- 172.21.15.28:0/2056473771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb4b89060 0x55ecb4b9ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.356+0000 7f3a2e000640 1 -- 172.21.15.28:0/2056473771 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecb47ed3f0 con 0x55ecb4b75f60 2024-04-23T14:58:22.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.356+0000 7f3a2ea00640 1 --2- 172.21.15.28:0/2056473771 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ecb4b755b0 0x55ecb4b77980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:58:22.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.356+0000 7f3a2e000640 1 --2- 172.21.15.28:0/2056473771 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ecb4b75f60 0x55ecb4b9e6c0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x55ecb4b85880 tx=0x7f3a1c005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b5be3b60e16a462 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:58:22.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.356+0000 7f3a1b400640 1 -- 172.21.15.28:0/2056473771 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c026070 con 0x55ecb4b75f60 2024-04-23T14:58:22.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.357+0000 7f3a1b400640 1 -- 172.21.15.28:0/2056473771 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a1c011e40 con 0x55ecb4b75f60 2024-04-23T14:58:22.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.357+0000 7f3a1b400640 1 -- 172.21.15.28:0/2056473771 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c003850 con 0x55ecb4b75f60 2024-04-23T14:58:22.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.357+0000 7f3a389f9740 1 -- 172.21.15.28:0/2056473771 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ecb4b75f60 msgr2=0x55ecb4b9e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:22.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.357+0000 7f3a389f9740 1 --2- 172.21.15.28:0/2056473771 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ecb4b75f60 0x55ecb4b9e6c0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x55ecb4b85880 tx=0x7f3a1c005f90 comp rx=0 tx=0).stop 2024-04-23T14:58:22.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.357+0000 7f3a389f9740 1 -- 172.21.15.28:0/2056473771 shutdown_connections 2024-04-23T14:58:22.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.357+0000 7f3a389f9740 1 --2- 172.21.15.28:0/2056473771 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ecb4b75f60 0x55ecb4b9e6c0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.357+0000 7f3a389f9740 1 --2- 172.21.15.28:0/2056473771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb4b89060 0x55ecb4b9ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.357+0000 7f3a389f9740 1 --2- 172.21.15.28:0/2056473771 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ecb4b755b0 0x55ecb4b77980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.357+0000 7f3a389f9740 1 -- 172.21.15.28:0/2056473771 >> 172.21.15.28:0/2056473771 conn(0x55ecb4b71b80 msgr2=0x55ecb4b84f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:58:22.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.357+0000 7f3a389f9740 1 -- 172.21.15.28:0/2056473771 shutdown_connections 2024-04-23T14:58:22.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.358+0000 7f3a389f9740 1 -- 172.21.15.28:0/2056473771 wait complete. 2024-04-23T14:58:22.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.358+0000 7f3a389f9740 1 -- start start 2024-04-23T14:58:22.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.358+0000 7f3a389f9740 1 Processor -- start 2024-04-23T14:58:22.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.359+0000 7f3a389f9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ecb4b755b0 0x55ecb4b9e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:22.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.359+0000 7f3a389f9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb4b75f60 0x55ecb4b9bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:22.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.359+0000 7f3a389f9740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ecb4b89060 0x55ecb4b9c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:22.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.359+0000 7f3a389f9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3a301aaa10 con 0x55ecb4b75f60 2024-04-23T14:58:22.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.359+0000 7f3a389f9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3a301abe80 con 0x55ecb4b755b0 2024-04-23T14:58:22.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.359+0000 7f3a389f9740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3a301ab0e0 con 0x55ecb4b89060 2024-04-23T14:58:22.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.359+0000 7f3a389f9740 10 client.0 _ll_register_callbacks cb 0x7f3a200e2070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:58:22.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.359+0000 7f3a2e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb4b75f60 0x55ecb4b9bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:22.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.359+0000 7f3a2e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb4b75f60 0x55ecb4b9bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34608/0 (socket says 172.21.15.28:34608) 2024-04-23T14:58:22.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.359+0000 7f3a2e000640 1 -- 172.21.15.28:0/1596902061 learned_addr learned my addr 172.21.15.28:0/1596902061 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:58:22.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.359+0000 7f3a2ea00640 1 --2- 172.21.15.28:0/1596902061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ecb4b755b0 0x55ecb4b9e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:22.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.359+0000 7f3a2f400640 1 --2- 172.21.15.28:0/1596902061 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ecb4b89060 0x55ecb4b9c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:22.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.359+0000 7f3a2e000640 1 -- 172.21.15.28:0/1596902061 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ecb4b89060 msgr2=0x55ecb4b9c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:58:22.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.359+0000 7f3a2e000640 1 --2- 172.21.15.28:0/1596902061 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ecb4b89060 0x55ecb4b9c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.359+0000 7f3a2e000640 1 -- 172.21.15.28:0/1596902061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ecb4b755b0 msgr2=0x55ecb4b9e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:58:22.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.359+0000 7f3a2e000640 1 --2- 172.21.15.28:0/1596902061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ecb4b755b0 0x55ecb4b9e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.359+0000 7f3a2e000640 1 -- 172.21.15.28:0/1596902061 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecb47ad0f0 con 0x55ecb4b75f60 2024-04-23T14:58:22.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.359+0000 7f3a2f400640 1 --2- 172.21.15.28:0/1596902061 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ecb4b89060 0x55ecb4b9c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:58:22.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.360+0000 7f3a2e000640 1 --2- 172.21.15.28:0/1596902061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb4b75f60 0x55ecb4b9bc70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c020ab0 tx=0x7f3a1c0040d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51a053ad232b1155 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:58:22.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.360+0000 7f3a18c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-23T14:58:22.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.360+0000 7f3a18c00640 1 -- 172.21.15.28:0/1596902061 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c02e020 con 0x55ecb4b75f60 2024-04-23T14:58:22.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.360+0000 7f3a18c00640 1 -- 172.21.15.28:0/1596902061 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8542+0+0 (secure 0 0 0) 0x7f3a1c005040 con 0x55ecb4b75f60 2024-04-23T14:58:22.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.360+0000 7f3a389f9740 1 -- 172.21.15.28:0/1596902061 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ecb4ae2a80 con 0x55ecb4b75f60 2024-04-23T14:58:22.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.360+0000 7f3a18c00640 1 -- 172.21.15.28:0/1596902061 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f3a1c039050 con 0x55ecb4b75f60 2024-04-23T14:58:22.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.360+0000 7f3a389f9740 10 client.17238 fetch_fsmap learned FSMap version 20 2024-04-23T14:58:22.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.360+0000 7f3a389f9740 1 -- 172.21.15.28:0/1596902061 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55ecb4aa0810 con 0x55ecb4b75f60 2024-04-23T14:58:22.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.361+0000 7f3a18c00640 1 -- 172.21.15.28:0/1596902061 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55ecb4ae2a80 con 0x55ecb4b75f60 2024-04-23T14:58:22.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.361+0000 7f3a18c00640 1 -- 172.21.15.28:0/1596902061 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f3a1c0046c0 con 0x55ecb4b75f60 2024-04-23T14:58:22.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.361+0000 7f3a389f9740 10 client.17238 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:58:22.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.361+0000 7f3a389f9740 10 client.17238 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:58:22.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.361+0000 7f3a389f9740 20 client.17238 populate_metadata read hostname 'smithi028' 2024-04-23T14:58:22.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.361+0000 7f3a389f9740 1 --2- 172.21.15.28:0/1596902061 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55ecb4bff700 0x55ecb4c01b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:22.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.361+0000 7f3a389f9740 4 client.17238 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-23T14:58:22.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.361+0000 7f3a389f9740 1 -- 172.21.15.28:0/1596902061 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ecb4a5e360 con 0x55ecb4bff700 2024-04-23T14:58:22.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.361+0000 7f3a2ea00640 1 --2- 172.21.15.28:0/1596902061 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55ecb4bff700 0x55ecb4c01b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:22.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.362+0000 7f3a2ea00640 1 --2- 172.21.15.28:0/1596902061 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55ecb4bff700 0x55ecb4c01b20 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45d2949af0eb9406 server_cookie=28bb9ec9c87dafd3 in_seq=0 out_seq=0 2024-04-23T14:58:22.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.362+0000 7f3a18c00640 10 client.17238 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:58:22.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.362+0000 7f3a18c00640 1 -- 172.21.15.28:0/1596902061 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ecb4a5e360 con 0x55ecb4bff700 2024-04-23T14:58:22.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.362+0000 7f3a18c00640 10 client.17238 handle_command_reply: tid=0 2024-04-23T14:58:22.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.362+0000 7f3a389f9740 1 client.17238 shutdown 2024-04-23T14:58:22.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.362+0000 7f3a389f9740 1 -- 172.21.15.28:0/1596902061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb4b75f60 msgr2=0x55ecb4b9bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:22.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.362+0000 7f3a389f9740 1 --2- 172.21.15.28:0/1596902061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb4b75f60 0x55ecb4b9bc70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c020ab0 tx=0x7f3a1c0040d0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.362+0000 7f3a389f9740 1 -- 172.21.15.28:0/1596902061 shutdown_connections 2024-04-23T14:58:22.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.362+0000 7f3a389f9740 1 --2- 172.21.15.28:0/1596902061 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x55ecb4bff700 0x55ecb4c01b20 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.362+0000 7f3a389f9740 1 --2- 172.21.15.28:0/1596902061 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ecb4b89060 0x55ecb4b9c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.362+0000 7f3a389f9740 1 --2- 172.21.15.28:0/1596902061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb4b75f60 0x55ecb4b9bc70 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.362+0000 7f3a18c00640 0 client.17238 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:58:22.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.362+0000 7f3a389f9740 1 --2- 172.21.15.28:0/1596902061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ecb4b755b0 0x55ecb4b9e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.362+0000 7f3a389f9740 1 -- 172.21.15.28:0/1596902061 >> 172.21.15.28:0/1596902061 conn(0x55ecb4b71b80 msgr2=0x55ecb4b94ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:58:22.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.363+0000 7f3a389f9740 1 -- 172.21.15.28:0/1596902061 shutdown_connections 2024-04-23T14:58:22.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.363+0000 7f3a389f9740 1 -- 172.21.15.28:0/1596902061 wait complete. 2024-04-23T14:58:22.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.363+0000 7f3a389f9740 20 client.17238 trim_cache size 0 max 16384 2024-04-23T14:58:22.370 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:58:22.370 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-23T14:58:22.370 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-23T14:58:22.371 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:58:22.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.363+0000 7f3a35c00640 1 -- 172.21.15.28:0/2925664789 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f3a140a3d20 msgr2=0x7f3a140a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:22.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.363+0000 7f3a35c00640 1 --2- 172.21.15.28:0/2925664789 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f3a140a3d20 0x7f3a140a61e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3a28009900 tx=0x7f3a28009560 comp rx=0 tx=0).stop 2024-04-23T14:58:22.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.363+0000 7f3a35c00640 1 -- 172.21.15.28:0/2925664789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a30100e00 msgr2=0x7f3a301a06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:22.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.363+0000 7f3a35c00640 1 --2- 172.21.15.28:0/2925664789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a30100e00 0x7f3a301a06e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f3a2000ebc0 tx=0x7f3a2000c6a0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.364+0000 7f3a35c00640 1 -- 172.21.15.28:0/2925664789 shutdown_connections 2024-04-23T14:58:22.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.364+0000 7f3a35c00640 1 --2- 172.21.15.28:0/2925664789 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f3a140a3d20 0x7f3a140a61e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.364+0000 7f3a35c00640 1 --2- 172.21.15.28:0/2925664789 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a30101720 0x7f3a301a0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.364+0000 7f3a35c00640 1 --2- 172.21.15.28:0/2925664789 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a3010e9b0 0x7f3a3019a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.364+0000 7f3a35c00640 1 --2- 172.21.15.28:0/2925664789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a30100e00 0x7f3a301a06e0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.364+0000 7f3a35c00640 1 -- 172.21.15.28:0/2925664789 >> 172.21.15.28:0/2925664789 conn(0x7f3a30078000 msgr2=0x7f3a3010feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:58:22.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.364+0000 7f3a35c00640 1 -- 172.21.15.28:0/2925664789 shutdown_connections 2024-04-23T14:58:22.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.364+0000 7f3a35c00640 1 -- 172.21.15.28:0/2925664789 wait complete. 2024-04-23T14:58:22.382 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-23T14:58:22.382 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:58:22.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.587+0000 7f85aa000640 1 -- 172.21.15.28:0/1753525153 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85a4104260 msgr2=0x7f85a4104640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:22.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.587+0000 7f85aa000640 1 --2- 172.21.15.28:0/1753525153 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85a4104260 0x7f85a4104640 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f8598009900 tx=0x7f859802a3f0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.587+0000 7f85aa000640 1 -- 172.21.15.28:0/1753525153 shutdown_connections 2024-04-23T14:58:22.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.587+0000 7f85aa000640 1 --2- 172.21.15.28:0/1753525153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85a4109330 0x7f85a4110f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.587+0000 7f85aa000640 1 --2- 172.21.15.28:0/1753525153 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f85a4104c10 0x7f85a4108c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.587+0000 7f85aa000640 1 --2- 172.21.15.28:0/1753525153 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85a4104260 0x7f85a4104640 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.587+0000 7f85aa000640 1 -- 172.21.15.28:0/1753525153 >> 172.21.15.28:0/1753525153 conn(0x7f85a40ffaa0 msgr2=0x7f85a4101ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:58:22.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.587+0000 7f85aa000640 1 -- 172.21.15.28:0/1753525153 shutdown_connections 2024-04-23T14:58:22.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.587+0000 7f85aa000640 1 -- 172.21.15.28:0/1753525153 wait complete. 2024-04-23T14:58:22.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.588+0000 7f85aa000640 1 Processor -- start 2024-04-23T14:58:22.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.588+0000 7f85aa000640 1 -- start start 2024-04-23T14:58:22.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.589+0000 7f85aa000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85a4104260 0x7f85a41a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:22.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.589+0000 7f85aa000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f85a4109330 0x7f85a41a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:22.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.589+0000 7f85aa000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85a419a6f0 0x7f85a419ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:22.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.589+0000 7f85aa000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f85a4114410 con 0x7f85a419a6f0 2024-04-23T14:58:22.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.589+0000 7f85aa000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f85a4114290 con 0x7f85a4104260 2024-04-23T14:58:22.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.589+0000 7f85aa000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f85a4114590 con 0x7f85a4109330 2024-04-23T14:58:22.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.589+0000 7f85a9600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85a419a6f0 0x7f85a419ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:22.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.589+0000 7f85a9600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85a419a6f0 0x7f85a419ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34620/0 (socket says 172.21.15.28:34620) 2024-04-23T14:58:22.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.589+0000 7f85a3e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f85a4109330 0x7f85a41a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:22.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.589+0000 7f85a8c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85a4104260 0x7f85a41a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:22.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.589+0000 7f85a9600640 1 -- 172.21.15.28:0/1403609735 learned_addr learned my addr 172.21.15.28:0/1403609735 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:58:22.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.590+0000 7f85a9600640 1 -- 172.21.15.28:0/1403609735 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f85a4109330 msgr2=0x7f85a41a0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:22.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.590+0000 7f85a9600640 1 --2- 172.21.15.28:0/1403609735 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f85a4109330 0x7f85a41a0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.590+0000 7f85a9600640 1 -- 172.21.15.28:0/1403609735 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85a4104260 msgr2=0x7f85a41a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:22.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.590+0000 7f85a9600640 1 --2- 172.21.15.28:0/1403609735 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85a4104260 0x7f85a41a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.590+0000 7f85a9600640 1 -- 172.21.15.28:0/1403609735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85a419b2f0 con 0x7f85a419a6f0 2024-04-23T14:58:22.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.590+0000 7f85a9600640 1 --2- 172.21.15.28:0/1403609735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85a419a6f0 0x7f85a419ab70 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f859c00eb10 tx=0x7f859c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:58:22.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.590+0000 7f85a1600640 1 -- 172.21.15.28:0/1403609735 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f859c004b20 con 0x7f85a419a6f0 2024-04-23T14:58:22.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.590+0000 7f85aa000640 1 -- 172.21.15.28:0/1403609735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85a41a73d0 con 0x7f85a419a6f0 2024-04-23T14:58:22.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.590+0000 7f85a1600640 1 -- 172.21.15.28:0/1403609735 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f859c004cc0 con 0x7f85a419a6f0 2024-04-23T14:58:22.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.590+0000 7f85a1600640 1 -- 172.21.15.28:0/1403609735 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f859c010760 con 0x7f85a419a6f0 2024-04-23T14:58:22.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.591+0000 7f85aa000640 1 -- 172.21.15.28:0/1403609735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85a41a7780 con 0x7f85a419a6f0 2024-04-23T14:58:22.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.592+0000 7f85aa000640 1 -- 172.21.15.28:0/1403609735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85a4105730 con 0x7f85a419a6f0 2024-04-23T14:58:22.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.594+0000 7f85a1600640 1 -- 172.21.15.28:0/1403609735 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f859c005020 con 0x7f85a419a6f0 2024-04-23T14:58:22.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.595+0000 7f85a1600640 1 --2- 172.21.15.28:0/1403609735 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f858c0a3d20 0x7f858c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:22.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.595+0000 7f85a1600640 1 -- 172.21.15.28:0/1403609735 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8542+0+0 (secure 0 0 0) 0x7f859c0d0d70 con 0x7f85a419a6f0 2024-04-23T14:58:22.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.595+0000 7f85a8c00640 1 --2- 172.21.15.28:0/1403609735 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f858c0a3d20 0x7f858c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:22.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.596+0000 7f85a8c00640 1 --2- 172.21.15.28:0/1403609735 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f858c0a3d20 0x7f858c0a61e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8598000c00 tx=0x7f8598005b50 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:58:22.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.600+0000 7f85a1600640 1 -- 172.21.15.28:0/1403609735 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f859c09d3a0 con 0x7f85a419a6f0 2024-04-23T14:58:22.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.846+0000 7f85aa000640 1 -- 172.21.15.28:0/1403609735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f85a410c560 con 0x7f85a419a6f0 2024-04-23T14:58:22.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.848+0000 7f85a1600640 1 -- 172.21.15.28:0/1403609735 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7f859c09cca0 con 0x7f85a419a6f0 2024-04-23T14:58:22.849 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:58:22.849 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:58:22.849 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:58:22.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.850+0000 7f85aa000640 1 -- 172.21.15.28:0/1403609735 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f858c0a3d20 msgr2=0x7f858c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:22.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.850+0000 7f85aa000640 1 --2- 172.21.15.28:0/1403609735 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f858c0a3d20 0x7f858c0a61e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8598000c00 tx=0x7f8598005b50 comp rx=0 tx=0).stop 2024-04-23T14:58:22.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.850+0000 7f85aa000640 1 -- 172.21.15.28:0/1403609735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85a419a6f0 msgr2=0x7f85a419ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:22.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.850+0000 7f85aa000640 1 --2- 172.21.15.28:0/1403609735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85a419a6f0 0x7f85a419ab70 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f859c00eb10 tx=0x7f859c00c2d0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.851+0000 7f85aa000640 1 -- 172.21.15.28:0/1403609735 shutdown_connections 2024-04-23T14:58:22.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.851+0000 7f85aa000640 1 --2- 172.21.15.28:0/1403609735 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f858c0a3d20 0x7f858c0a61e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.851+0000 7f85aa000640 1 --2- 172.21.15.28:0/1403609735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85a419a6f0 0x7f85a419ab70 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.851+0000 7f85aa000640 1 --2- 172.21.15.28:0/1403609735 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f85a4109330 0x7f85a41a0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.851+0000 7f85aa000640 1 --2- 172.21.15.28:0/1403609735 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85a4104260 0x7f85a41a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:22.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.851+0000 7f85aa000640 1 -- 172.21.15.28:0/1403609735 >> 172.21.15.28:0/1403609735 conn(0x7f85a40ffaa0 msgr2=0x7f85a4101d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:58:22.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.851+0000 7f85aa000640 1 -- 172.21.15.28:0/1403609735 shutdown_connections 2024-04-23T14:58:22.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:22.851+0000 7f85aa000640 1 -- 172.21.15.28:0/1403609735 wait complete. 2024-04-23T14:58:22.871 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-23T14:58:23.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.074+0000 7fd5c2e00640 1 -- 172.21.15.28:0/3474883307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5bc104ca0 msgr2=0x7fd5bc108ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:23.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.074+0000 7fd5c2e00640 1 --2- 172.21.15.28:0/3474883307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5bc104ca0 0x7fd5bc108ce0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8009900 tx=0x7fd5a802a3f0 comp rx=0 tx=0).stop 2024-04-23T14:58:23.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.074+0000 7fd5c2e00640 1 -- 172.21.15.28:0/3474883307 shutdown_connections 2024-04-23T14:58:23.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.074+0000 7fd5c2e00640 1 --2- 172.21.15.28:0/3474883307 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5bc1093c0 0x7fd5bc110f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:23.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.074+0000 7fd5c2e00640 1 --2- 172.21.15.28:0/3474883307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5bc104ca0 0x7fd5bc108ce0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:23.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.074+0000 7fd5c2e00640 1 --2- 172.21.15.28:0/3474883307 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd5bc1042f0 0x7fd5bc1046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:23.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.074+0000 7fd5c2e00640 1 -- 172.21.15.28:0/3474883307 >> 172.21.15.28:0/3474883307 conn(0x7fd5bc0ffb10 msgr2=0x7fd5bc101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:58:23.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.075+0000 7fd5c2e00640 1 -- 172.21.15.28:0/3474883307 shutdown_connections 2024-04-23T14:58:23.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.075+0000 7fd5c2e00640 1 -- 172.21.15.28:0/3474883307 wait complete. 2024-04-23T14:58:23.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.075+0000 7fd5c2e00640 1 Processor -- start 2024-04-23T14:58:23.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.076+0000 7fd5c2e00640 1 -- start start 2024-04-23T14:58:23.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.076+0000 7fd5c2e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd5bc1042f0 0x7fd5bc19aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:23.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.076+0000 7fd5c2e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5bc1093c0 0x7fd5bc19afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:23.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.076+0000 7fd5c2e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5bc19f370 0x7fd5bc19f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:23.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.076+0000 7fd5c2e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd5bc114480 con 0x7fd5bc19f370 2024-04-23T14:58:23.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.076+0000 7fd5c2e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd5bc114300 con 0x7fd5bc1093c0 2024-04-23T14:58:23.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.076+0000 7fd5c2e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd5bc114600 con 0x7fd5bc1042f0 2024-04-23T14:58:23.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.076+0000 7fd5c2400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5bc19f370 0x7fd5bc19f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:23.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.076+0000 7fd5c2400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5bc19f370 0x7fd5bc19f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34644/0 (socket says 172.21.15.28:34644) 2024-04-23T14:58:23.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.076+0000 7fd5c2400640 1 -- 172.21.15.28:0/1093201452 learned_addr learned my addr 172.21.15.28:0/1093201452 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:58:23.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.076+0000 7fd5c1000640 1 --2- 172.21.15.28:0/1093201452 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5bc1093c0 0x7fd5bc19afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:23.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.077+0000 7fd5c1a00640 1 --2- 172.21.15.28:0/1093201452 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd5bc1042f0 0x7fd5bc19aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:23.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.077+0000 7fd5c2400640 1 -- 172.21.15.28:0/1093201452 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd5bc1042f0 msgr2=0x7fd5bc19aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:23.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.077+0000 7fd5c2400640 1 --2- 172.21.15.28:0/1093201452 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd5bc1042f0 0x7fd5bc19aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:23.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.077+0000 7fd5c2400640 1 -- 172.21.15.28:0/1093201452 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5bc1093c0 msgr2=0x7fd5bc19afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:23.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.077+0000 7fd5c2400640 1 --2- 172.21.15.28:0/1093201452 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5bc1093c0 0x7fd5bc19afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:23.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.077+0000 7fd5c2400640 1 -- 172.21.15.28:0/1093201452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5bc19fe60 con 0x7fd5bc19f370 2024-04-23T14:58:23.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.077+0000 7fd5c1a00640 1 --2- 172.21.15.28:0/1093201452 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd5bc1042f0 0x7fd5bc19aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:58:23.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.077+0000 7fd5c2400640 1 --2- 172.21.15.28:0/1093201452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5bc19f370 0x7fd5bc19f820 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd5b400eb10 tx=0x7fd5b400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:58:23.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.078+0000 7fd5ba000640 1 -- 172.21.15.28:0/1093201452 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b4004b20 con 0x7fd5bc19f370 2024-04-23T14:58:23.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.078+0000 7fd5ba000640 1 -- 172.21.15.28:0/1093201452 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5b4004cc0 con 0x7fd5bc19f370 2024-04-23T14:58:23.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.078+0000 7fd5c2e00640 1 -- 172.21.15.28:0/1093201452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5bc1a0150 con 0x7fd5bc19f370 2024-04-23T14:58:23.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.078+0000 7fd5ba000640 1 -- 172.21.15.28:0/1093201452 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b4010760 con 0x7fd5bc19f370 2024-04-23T14:58:23.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.078+0000 7fd5c2e00640 1 -- 172.21.15.28:0/1093201452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5bc1a78c0 con 0x7fd5bc19f370 2024-04-23T14:58:23.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.081+0000 7fd5c7284740 1 Processor -- start 2024-04-23T14:58:23.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.081+0000 7fd5ba000640 1 -- 172.21.15.28:0/1093201452 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fd5b4005020 con 0x7fd5bc19f370 2024-04-23T14:58:23.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.081+0000 7fd5c7284740 1 -- start start 2024-04-23T14:58:23.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.082+0000 7fd5c7284740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559c89776b40 0x559c89776f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:23.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.082+0000 7fd5c7284740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c89777460 0x559c897815f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:23.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.082+0000 7fd5c7284740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c89781b30 0x559c89783f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:23.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.082+0000 7fd5c7284740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559c896dc1d0 con 0x559c89777460 2024-04-23T14:58:23.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.082+0000 7fd5c7284740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559c89682450 con 0x559c89776b40 2024-04-23T14:58:23.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.082+0000 7fd5c7284740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559c896d8190 con 0x559c89781b30 2024-04-23T14:58:23.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.082+0000 7fd5c1a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c89777460 0x559c897815f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:23.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.082+0000 7fd5c1000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559c89776b40 0x559c89776f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:23.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.082+0000 7fd5c1a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c89777460 0x559c897815f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34660/0 (socket says 172.21.15.28:34660) 2024-04-23T14:58:23.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.082+0000 7fd5c1a00640 1 -- 172.21.15.28:0/3342272777 learned_addr learned my addr 172.21.15.28:0/3342272777 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:58:23.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.082+0000 7fd5c1000640 1 --2- 172.21.15.28:0/3342272777 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c89781b30 0x559c89783f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:23.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.083+0000 7fd5c1a00640 1 -- 172.21.15.28:0/3342272777 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c89781b30 msgr2=0x559c89783f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:58:23.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.083+0000 7fd5c1a00640 1 --2- 172.21.15.28:0/3342272777 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c89781b30 0x559c89783f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:23.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.083+0000 7fd5c1a00640 1 -- 172.21.15.28:0/3342272777 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559c89776b40 msgr2=0x559c89776f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:58:23.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.083+0000 7fd5c1a00640 1 --2- 172.21.15.28:0/3342272777 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559c89776b40 0x559c89776f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:23.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.083+0000 7fd5c1a00640 1 -- 172.21.15.28:0/3342272777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c8973f310 con 0x559c89777460 2024-04-23T14:58:23.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.083+0000 7fd5c1000640 1 --2- 172.21.15.28:0/3342272777 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c89781b30 0x559c89783f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:58:23.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.083+0000 7fd5ba000640 1 --2- 172.21.15.28:0/1093201452 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd5a40a3d20 0x7fd5a40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:23.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.083+0000 7fd5c1a00640 1 --2- 172.21.15.28:0/3342272777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c89777460 0x559c897815f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd5b0011540 tx=0x7fd5b0011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25261deb3e549055 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:58:23.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.083+0000 7fd5ba000640 1 -- 172.21.15.28:0/1093201452 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8542+0+0 (secure 0 0 0) 0x7fd5b40d0d70 con 0x7fd5bc19f370 2024-04-23T14:58:23.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.083+0000 7fd5c1000640 1 --2- 172.21.15.28:0/1093201452 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd5a40a3d20 0x7fd5a40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:23.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.083+0000 7fd5aea00640 1 -- 172.21.15.28:0/3342272777 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b000f640 con 0x559c89777460 2024-04-23T14:58:23.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.083+0000 7fd5aea00640 1 -- 172.21.15.28:0/3342272777 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5b000fcc0 con 0x559c89777460 2024-04-23T14:58:23.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.083+0000 7fd5aea00640 1 -- 172.21.15.28:0/3342272777 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b0019d30 con 0x559c89777460 2024-04-23T14:58:23.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.084+0000 7fd5c7284740 1 -- 172.21.15.28:0/3342272777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c89777460 msgr2=0x559c897815f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:23.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.084+0000 7fd5c1000640 1 --2- 172.21.15.28:0/1093201452 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd5a40a3d20 0x7fd5a40a61e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8006010 tx=0x7fd5a8041040 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:58:23.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.084+0000 7fd5c7284740 1 --2- 172.21.15.28:0/3342272777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c89777460 0x559c897815f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd5b0011540 tx=0x7fd5b0011a00 comp rx=0 tx=0).stop 2024-04-23T14:58:23.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.084+0000 7fd5c7284740 1 -- 172.21.15.28:0/3342272777 shutdown_connections 2024-04-23T14:58:23.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.084+0000 7fd5c7284740 1 --2- 172.21.15.28:0/3342272777 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c89781b30 0x559c89783f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:23.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.084+0000 7fd5c7284740 1 --2- 172.21.15.28:0/3342272777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c89777460 0x559c897815f0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:23.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.084+0000 7fd5c7284740 1 --2- 172.21.15.28:0/3342272777 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559c89776b40 0x559c89776f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:23.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.084+0000 7fd5c7284740 1 -- 172.21.15.28:0/3342272777 >> 172.21.15.28:0/3342272777 conn(0x559c89772e60 msgr2=0x559c89773240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:58:23.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.084+0000 7fd5c7284740 1 -- 172.21.15.28:0/3342272777 shutdown_connections 2024-04-23T14:58:23.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.084+0000 7fd5c7284740 1 -- 172.21.15.28:0/3342272777 wait complete. 2024-04-23T14:58:23.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.085+0000 7fd5c7284740 1 -- start start 2024-04-23T14:58:23.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.085+0000 7fd5c7284740 1 Processor -- start 2024-04-23T14:58:23.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.086+0000 7fd5c7284740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559c89776b40 0x559c89781590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:23.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.086+0000 7fd5c7284740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c89777460 0x559c8977e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:23.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.086+0000 7fd5c7284740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c89781b30 0x559c8977efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:23.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.086+0000 7fd5c7284740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559c897849c0 con 0x559c89777460 2024-04-23T14:58:23.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.086+0000 7fd5c7284740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559c89695810 con 0x559c89776b40 2024-04-23T14:58:23.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.086+0000 7fd5c7284740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559c89784b40 con 0x559c89781b30 2024-04-23T14:58:23.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.086+0000 7fd5c7284740 10 client.0 _ll_register_callbacks cb 0x559c89663140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:58:23.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.086+0000 7fd5c1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c89777460 0x559c8977e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:23.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.086+0000 7fd5c2400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c89781b30 0x559c8977efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:23.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.086+0000 7fd5c2400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c89781b30 0x559c8977efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:42932/0 (socket says 172.21.15.28:42932) 2024-04-23T14:58:23.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.086+0000 7fd5c2400640 1 -- 172.21.15.28:0/418101352 learned_addr learned my addr 172.21.15.28:0/418101352 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:58:23.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.086+0000 7fd5c1a00640 1 --2- 172.21.15.28:0/418101352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559c89776b40 0x559c89781590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:23.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.086+0000 7fd5c2400640 1 -- 172.21.15.28:0/418101352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559c89776b40 msgr2=0x559c89781590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:58:23.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.086+0000 7fd5c2400640 1 --2- 172.21.15.28:0/418101352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559c89776b40 0x559c89781590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:23.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.086+0000 7fd5c2400640 1 -- 172.21.15.28:0/418101352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c89777460 msgr2=0x559c8977e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:58:23.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.086+0000 7fd5c2400640 1 --2- 172.21.15.28:0/418101352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c89777460 0x559c8977e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:23.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.086+0000 7fd5c2400640 1 -- 172.21.15.28:0/418101352 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c896a63f0 con 0x559c89781b30 2024-04-23T14:58:23.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.086+0000 7fd5c1000640 1 --2- 172.21.15.28:0/418101352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c89777460 0x559c8977e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:58:23.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.087+0000 7fd5c2400640 1 --2- 172.21.15.28:0/418101352 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c89781b30 0x559c8977efa0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fd5b40a53e0 tx=0x7fd5b409c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a03358999530da63 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:58:23.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.087+0000 7fd5a3400640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-23T14:58:23.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.087+0000 7fd5a3400640 1 -- 172.21.15.28:0/418101352 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b40bbe60 con 0x559c89781b30 2024-04-23T14:58:23.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.087+0000 7fd5a3400640 1 -- 172.21.15.28:0/418101352 <== mon.1 v2:172.21.15.105:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8542+0+0 (secure 0 0 0) 0x7fd5b40a5480 con 0x559c89781b30 2024-04-23T14:58:23.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.087+0000 7fd5c7284740 1 -- 172.21.15.28:0/418101352 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5b000fcc0 con 0x559c89781b30 2024-04-23T14:58:23.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.087+0000 7fd5a3400640 1 -- 172.21.15.28:0/418101352 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fd5b40b6200 con 0x559c89781b30 2024-04-23T14:58:23.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.087+0000 7fd5c7284740 10 client.27442 fetch_fsmap learned FSMap version 20 2024-04-23T14:58:23.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.087+0000 7fd5c7284740 1 -- 172.21.15.28:0/418101352 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x559c893ee3f0 con 0x559c89781b30 2024-04-23T14:58:23.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.088+0000 7fd5a3400640 1 -- 172.21.15.28:0/418101352 <== mon.1 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5b000fcc0 con 0x559c89781b30 2024-04-23T14:58:23.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.088+0000 7fd5a3400640 1 -- 172.21.15.28:0/418101352 <== mon.1 v2:172.21.15.105:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fd5b40bd070 con 0x559c89781b30 2024-04-23T14:58:23.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.088+0000 7fd5c7284740 10 client.27442 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:58:23.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.088+0000 7fd5c7284740 10 client.27442 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:58:23.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.088+0000 7fd5c7284740 20 client.27442 populate_metadata read hostname 'smithi028' 2024-04-23T14:58:23.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.088+0000 7fd5c7284740 1 --2- 172.21.15.28:0/418101352 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x559c89791c70 0x559c897940a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:23.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.088+0000 7fd5c7284740 4 client.27442 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T14:58:23.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.088+0000 7fd5c7284740 1 -- 172.21.15.28:0/418101352 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559c896e9bf0 con 0x559c89791c70 2024-04-23T14:58:23.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.089+0000 7fd5c1a00640 1 --2- 172.21.15.28:0/418101352 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x559c89791c70 0x559c897940a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:23.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.089+0000 7fd5c1a00640 1 --2- 172.21.15.28:0/418101352 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x559c89791c70 0x559c897940a0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86ecac2f9c886a2c server_cookie=274251fbc92010c4 in_seq=0 out_seq=0 2024-04-23T14:58:23.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.089+0000 7fd5a3400640 10 client.27442 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:58:23.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.090+0000 7fd5a3400640 1 -- 172.21.15.28:0/418101352 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559c896e9bf0 con 0x559c89791c70 2024-04-23T14:58:23.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.090+0000 7fd5a3400640 10 client.27442 handle_command_reply: tid=0 2024-04-23T14:58:23.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.090+0000 7fd5c7284740 1 client.27442 shutdown 2024-04-23T14:58:23.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.090+0000 7fd5c7284740 1 -- 172.21.15.28:0/418101352 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c89781b30 msgr2=0x559c8977efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:23.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.090+0000 7fd5c7284740 1 --2- 172.21.15.28:0/418101352 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c89781b30 0x559c8977efa0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fd5b40a53e0 tx=0x7fd5b409c6c0 comp rx=0 tx=0).stop 2024-04-23T14:58:23.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.090+0000 7fd5c7284740 1 -- 172.21.15.28:0/418101352 shutdown_connections 2024-04-23T14:58:23.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.090+0000 7fd5c7284740 1 --2- 172.21.15.28:0/418101352 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x559c89791c70 0x559c897940a0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:23.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.090+0000 7fd5c7284740 1 --2- 172.21.15.28:0/418101352 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c89781b30 0x559c8977efa0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:23.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.090+0000 7fd5c7284740 1 --2- 172.21.15.28:0/418101352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c89777460 0x559c8977e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:23.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.090+0000 7fd5a3400640 0 client.27442 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:58:23.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.090+0000 7fd5c7284740 1 --2- 172.21.15.28:0/418101352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559c89776b40 0x559c89781590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:23.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.090+0000 7fd5c7284740 1 -- 172.21.15.28:0/418101352 >> 172.21.15.28:0/418101352 conn(0x559c89772e60 msgr2=0x559c89778c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:58:23.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.091+0000 7fd5c7284740 1 -- 172.21.15.28:0/418101352 shutdown_connections 2024-04-23T14:58:23.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.091+0000 7fd5c7284740 1 -- 172.21.15.28:0/418101352 wait complete. 2024-04-23T14:58:23.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.091+0000 7fd5c7284740 20 client.27442 trim_cache size 0 max 16384 2024-04-23T14:58:23.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.105+0000 7fd5c7284740 1 Processor -- start 2024-04-23T14:58:23.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.105+0000 7fd5c7284740 1 -- start start 2024-04-23T14:58:23.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.105+0000 7fd5c7284740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c89776b40 0x559c8978d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:23.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.105+0000 7fd5c7284740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c89777460 0x559c8977ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:23.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.105+0000 7fd5c7284740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559c89781b30 0x559c8977d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:23.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.106+0000 7fd5c7284740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd5bc1add70 con 0x559c89777460 2024-04-23T14:58:23.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.106+0000 7fd5c7284740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd5bc1b05d0 con 0x559c89781b30 2024-04-23T14:58:23.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.106+0000 7fd5c7284740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd5bc1ae3c0 con 0x559c89776b40 2024-04-23T14:58:23.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.106+0000 7fd5c1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c89777460 0x559c8977ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:23.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.106+0000 7fd5c1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c89777460 0x559c8977ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34678/0 (socket says 172.21.15.28:34678) 2024-04-23T14:58:23.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.106+0000 7fd5c1000640 1 -- 172.21.15.28:0/975084642 learned_addr learned my addr 172.21.15.28:0/975084642 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:58:23.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.106+0000 7fd5c2400640 1 --2- 172.21.15.28:0/975084642 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559c89781b30 0x559c8977d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:23.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.106+0000 7fd5c1a00640 1 --2- 172.21.15.28:0/975084642 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c89776b40 0x559c8978d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:23.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.106+0000 7fd5c1000640 1 -- 172.21.15.28:0/975084642 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c89776b40 msgr2=0x559c8978d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:58:23.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.106+0000 7fd5c1000640 1 --2- 172.21.15.28:0/975084642 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c89776b40 0x559c8978d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:23.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.106+0000 7fd5c1000640 1 -- 172.21.15.28:0/975084642 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559c89781b30 msgr2=0x559c8977d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:58:23.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.106+0000 7fd5c1000640 1 --2- 172.21.15.28:0/975084642 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559c89781b30 0x559c8977d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:23.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.106+0000 7fd5c1000640 1 -- 172.21.15.28:0/975084642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c894b87d0 con 0x559c89777460 2024-04-23T14:58:23.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.107+0000 7fd5c1000640 1 --2- 172.21.15.28:0/975084642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c89777460 0x559c8977ef20 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8009900 tx=0x7fd5a8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62dcab4eff749f7d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:58:23.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.107+0000 7fd5ae000640 1 -- 172.21.15.28:0/975084642 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8046070 con 0x559c89777460 2024-04-23T14:58:23.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.107+0000 7fd5ae000640 1 -- 172.21.15.28:0/975084642 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5a8051070 con 0x559c89777460 2024-04-23T14:58:23.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.107+0000 7fd5ae000640 1 -- 172.21.15.28:0/975084642 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8003850 con 0x559c89777460 2024-04-23T14:58:23.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.108+0000 7fd5c7284740 1 -- 172.21.15.28:0/975084642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c89777460 msgr2=0x559c8977ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:23.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.108+0000 7fd5c7284740 1 --2- 172.21.15.28:0/975084642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c89777460 0x559c8977ef20 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8009900 tx=0x7fd5a8009260 comp rx=0 tx=0).stop 2024-04-23T14:58:23.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.108+0000 7fd5c7284740 1 -- 172.21.15.28:0/975084642 shutdown_connections 2024-04-23T14:58:23.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.108+0000 7fd5c7284740 1 --2- 172.21.15.28:0/975084642 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559c89781b30 0x559c8977d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:23.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.108+0000 7fd5c7284740 1 --2- 172.21.15.28:0/975084642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c89777460 0x559c8977ef20 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:23.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.108+0000 7fd5c7284740 1 --2- 172.21.15.28:0/975084642 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c89776b40 0x559c8978d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:23.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.108+0000 7fd5c7284740 1 -- 172.21.15.28:0/975084642 >> 172.21.15.28:0/975084642 conn(0x559c89772e60 msgr2=0x559c8977d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:58:23.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.108+0000 7fd5c7284740 1 -- 172.21.15.28:0/975084642 shutdown_connections 2024-04-23T14:58:23.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.108+0000 7fd5c7284740 1 -- 172.21.15.28:0/975084642 wait complete. 2024-04-23T14:58:23.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.108+0000 7fd5c7284740 1 -- start start 2024-04-23T14:58:23.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.108+0000 7fd5c7284740 1 Processor -- start 2024-04-23T14:58:23.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.109+0000 7fd5c7284740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c89776b40 0x559c89782320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:23.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.109+0000 7fd5c7284740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559c89777460 0x559c89782860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:23.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.109+0000 7fd5c7284740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c89781b30 0x559c89788d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:23.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.109+0000 7fd5c7284740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd5bc1aea90 con 0x559c89776b40 2024-04-23T14:58:23.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.109+0000 7fd5c7284740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd5bc1aff00 con 0x559c89777460 2024-04-23T14:58:23.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.109+0000 7fd5c7284740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd5bc1af160 con 0x559c89781b30 2024-04-23T14:58:23.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.109+0000 7fd5c7284740 10 client.0 _ll_register_callbacks cb 0x7fd5b409cae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T14:58:23.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.109+0000 7fd5c1a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c89776b40 0x559c89782320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:23.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.109+0000 7fd5c1a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c89776b40 0x559c89782320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34682/0 (socket says 172.21.15.28:34682) 2024-04-23T14:58:23.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.109+0000 7fd5c1a00640 1 -- 172.21.15.28:0/3333338510 learned_addr learned my addr 172.21.15.28:0/3333338510 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:58:23.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.109+0000 7fd5c2400640 1 --2- 172.21.15.28:0/3333338510 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c89781b30 0x559c89788d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:23.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.109+0000 7fd5c1000640 1 --2- 172.21.15.28:0/3333338510 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559c89777460 0x559c89782860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:23.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.109+0000 7fd5c1a00640 1 -- 172.21.15.28:0/3333338510 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c89781b30 msgr2=0x559c89788d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:58:23.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.109+0000 7fd5c1a00640 1 --2- 172.21.15.28:0/3333338510 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c89781b30 0x559c89788d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:23.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.110+0000 7fd5c1a00640 1 -- 172.21.15.28:0/3333338510 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559c89777460 msgr2=0x559c89782860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T14:58:23.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.110+0000 7fd5c1a00640 1 --2- 172.21.15.28:0/3333338510 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559c89777460 0x559c89782860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:23.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.110+0000 7fd5c1a00640 1 -- 172.21.15.28:0/3333338510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c897892d0 con 0x559c89776b40 2024-04-23T14:58:23.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.110+0000 7fd5c1000640 1 --2- 172.21.15.28:0/3333338510 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559c89777460 0x559c89782860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:58:23.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.110+0000 7fd5c1a00640 1 --2- 172.21.15.28:0/3333338510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c89776b40 0x559c89782320 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd5b0011540 tx=0x7fd5b0009cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99ba1e942d753da8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:58:23.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.110+0000 7fd5a3400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-23T14:58:23.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.110+0000 7fd5a3400640 1 -- 172.21.15.28:0/3333338510 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b0019900 con 0x559c89776b40 2024-04-23T14:58:23.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.110+0000 7fd5a3400640 1 -- 172.21.15.28:0/3333338510 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8542+0+0 (secure 0 0 0) 0x7fd5b0016040 con 0x559c89776b40 2024-04-23T14:58:23.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.110+0000 7fd5c7284740 1 -- 172.21.15.28:0/3333338510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5a8051070 con 0x559c89776b40 2024-04-23T14:58:23.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.110+0000 7fd5a3400640 1 -- 172.21.15.28:0/3333338510 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fd5b0019aa0 con 0x559c89776b40 2024-04-23T14:58:23.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.110+0000 7fd5c7284740 10 client.17262 fetch_fsmap learned FSMap version 20 2024-04-23T14:58:23.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.111+0000 7fd5c7284740 1 -- 172.21.15.28:0/3333338510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x559c8978a530 con 0x559c89776b40 2024-04-23T14:58:23.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.111+0000 7fd5a3400640 1 -- 172.21.15.28:0/3333338510 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5a8051070 con 0x559c89776b40 2024-04-23T14:58:23.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.111+0000 7fd5a3400640 1 -- 172.21.15.28:0/3333338510 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fd5b000f7f0 con 0x559c89776b40 2024-04-23T14:58:23.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.111+0000 7fd5c7284740 10 client.17262 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T14:58:23.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.111+0000 7fd5c7284740 10 client.17262 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-23T14:58:23.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.111+0000 7fd5c7284740 20 client.17262 populate_metadata read hostname 'smithi028' 2024-04-23T14:58:23.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.111+0000 7fd5c7284740 1 --2- 172.21.15.28:0/3333338510 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x559c897f8430 0x559c897fa820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:23.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.111+0000 7fd5c7284740 4 client.17262 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-23T14:58:23.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.111+0000 7fd5c7284740 1 -- 172.21.15.28:0/3333338510 --> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559c89772cb0 con 0x559c897f8430 2024-04-23T14:58:23.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.111+0000 7fd5c1000640 1 --2- 172.21.15.28:0/3333338510 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x559c897f8430 0x559c897fa820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:23.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.112+0000 7fd5c1000640 1 --2- 172.21.15.28:0/3333338510 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x559c897f8430 0x559c897fa820 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e21dd78921ef22ad server_cookie=7f76b85718a7dac9 in_seq=0 out_seq=0 2024-04-23T14:58:23.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.112+0000 7fd5a3400640 10 client.17262 ms_handle_connect on v2:172.21.15.38:6838/377787688 2024-04-23T14:58:23.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.112+0000 7fd5a3400640 1 -- 172.21.15.28:0/3333338510 <== mds.0 v2:172.21.15.38:6838/377787688 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559c89772cb0 con 0x559c897f8430 2024-04-23T14:58:23.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.112+0000 7fd5a3400640 10 client.17262 handle_command_reply: tid=0 2024-04-23T14:58:23.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.112+0000 7fd5c7284740 1 client.17262 shutdown 2024-04-23T14:58:23.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.112+0000 7fd5c7284740 1 -- 172.21.15.28:0/3333338510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c89776b40 msgr2=0x559c89782320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:23.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.112+0000 7fd5c7284740 1 --2- 172.21.15.28:0/3333338510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c89776b40 0x559c89782320 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd5b0011540 tx=0x7fd5b0009cc0 comp rx=0 tx=0).stop 2024-04-23T14:58:23.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.112+0000 7fd5c7284740 1 -- 172.21.15.28:0/3333338510 shutdown_connections 2024-04-23T14:58:23.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.113+0000 7fd5c7284740 1 --2- 172.21.15.28:0/3333338510 >> [v2:172.21.15.38:6838/377787688,v1:172.21.15.38:6839/377787688] conn(0x559c897f8430 0x559c897fa820 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:23.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.113+0000 7fd5c7284740 1 --2- 172.21.15.28:0/3333338510 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559c89777460 0x559c89782860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:23.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.113+0000 7fd5c7284740 1 --2- 172.21.15.28:0/3333338510 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559c89781b30 0x559c89788d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:23.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.113+0000 7fd5c7284740 1 --2- 172.21.15.28:0/3333338510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c89776b40 0x559c89782320 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:23.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.113+0000 7fd5a3400640 0 client.17262 ms_handle_reset on v2:172.21.15.38:6838/377787688 2024-04-23T14:58:23.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.113+0000 7fd5c7284740 1 -- 172.21.15.28:0/3333338510 >> 172.21.15.28:0/3333338510 conn(0x559c89772e60 msgr2=0x559c89778290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:58:23.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.113+0000 7fd5c7284740 1 -- 172.21.15.28:0/3333338510 shutdown_connections 2024-04-23T14:58:23.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.113+0000 7fd5c7284740 1 -- 172.21.15.28:0/3333338510 wait complete. 2024-04-23T14:58:23.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.113+0000 7fd5c7284740 20 client.17262 trim_cache size 0 max 16384 2024-04-23T14:58:23.120 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-23T14:58:23.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.113+0000 7fd5c2e00640 1 -- 172.21.15.28:0/1093201452 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd5a40a3d20 msgr2=0x7fd5a40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:23.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.113+0000 7fd5c2e00640 1 --2- 172.21.15.28:0/1093201452 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd5a40a3d20 0x7fd5a40a61e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8006010 tx=0x7fd5a8041040 comp rx=0 tx=0).stop 2024-04-23T14:58:23.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.113+0000 7fd5c2e00640 1 -- 172.21.15.28:0/1093201452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5bc19f370 msgr2=0x7fd5bc19f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:23.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.113+0000 7fd5c2e00640 1 --2- 172.21.15.28:0/1093201452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5bc19f370 0x7fd5bc19f820 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd5b400eb10 tx=0x7fd5b400c2d0 comp rx=0 tx=0).stop 2024-04-23T14:58:23.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.114+0000 7fd5c2e00640 1 -- 172.21.15.28:0/1093201452 shutdown_connections 2024-04-23T14:58:23.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.114+0000 7fd5c2e00640 1 --2- 172.21.15.28:0/1093201452 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd5a40a3d20 0x7fd5a40a61e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:23.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.114+0000 7fd5c2e00640 1 --2- 172.21.15.28:0/1093201452 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5bc1093c0 0x7fd5bc19afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:23.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.114+0000 7fd5c2e00640 1 --2- 172.21.15.28:0/1093201452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5bc19f370 0x7fd5bc19f820 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:23.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.114+0000 7fd5c2e00640 1 --2- 172.21.15.28:0/1093201452 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd5bc1042f0 0x7fd5bc19aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:23.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.114+0000 7fd5c2e00640 1 -- 172.21.15.28:0/1093201452 >> 172.21.15.28:0/1093201452 conn(0x7fd5bc0ffb10 msgr2=0x7fd5bc101e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:58:23.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.114+0000 7fd5c2e00640 1 -- 172.21.15.28:0/1093201452 shutdown_connections 2024-04-23T14:58:23.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:23.114+0000 7fd5c2e00640 1 -- 172.21.15.28:0/1093201452 wait complete. 2024-04-23T14:58:23.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:22 smithi038 ceph-mon[32348]: pgmap v449: 97 pgs: 97 active+clean; 645 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.6 MiB/s wr, 178 op/s 2024-04-23T14:58:23.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:22 smithi028 ceph-mon[27810]: pgmap v449: 97 pgs: 97 active+clean; 645 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.6 MiB/s wr, 178 op/s 2024-04-23T14:58:23.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:22 smithi105 ceph-mon[32555]: pgmap v449: 97 pgs: 97 active+clean; 645 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.6 MiB/s wr, 178 op/s 2024-04-23T14:58:24.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:23 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1403609735' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:58:24.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:23 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1403609735' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:58:24.132 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-04-23T14:58:24.132 INFO:tasks.fwd_scrub:done joining 2024-04-23T14:58:24.133 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-23T14:58:24.144 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:58:24.285 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:23 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1403609735' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:58:24.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.323+0000 7f3d41800640 1 -- 172.21.15.28:0/3280770851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3c1020d0 msgr2=0x7f3d3c1024b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:24.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.323+0000 7f3d41800640 1 --2- 172.21.15.28:0/3280770851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3c1020d0 0x7f3d3c1024b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c009900 tx=0x7f3d2c02a3f0 comp rx=0 tx=0).stop 2024-04-23T14:58:24.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.324+0000 7f3d41800640 1 -- 172.21.15.28:0/3280770851 shutdown_connections 2024-04-23T14:58:24.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.324+0000 7f3d41800640 1 --2- 172.21.15.28:0/3280770851 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d3c102a80 0x7f3d3c106ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:24.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.324+0000 7f3d41800640 1 --2- 172.21.15.28:0/3280770851 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d3c1071a0 0x7f3d3c10ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:24.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.324+0000 7f3d41800640 1 --2- 172.21.15.28:0/3280770851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3c1020d0 0x7f3d3c1024b0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:24.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.324+0000 7f3d41800640 1 -- 172.21.15.28:0/3280770851 >> 172.21.15.28:0/3280770851 conn(0x7f3d3c0fd970 msgr2=0x7f3d3c0ffd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:58:24.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.324+0000 7f3d41800640 1 -- 172.21.15.28:0/3280770851 shutdown_connections 2024-04-23T14:58:24.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.324+0000 7f3d41800640 1 -- 172.21.15.28:0/3280770851 wait complete. 2024-04-23T14:58:24.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.325+0000 7f3d41800640 1 Processor -- start 2024-04-23T14:58:24.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.325+0000 7f3d41800640 1 -- start start 2024-04-23T14:58:24.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.325+0000 7f3d41800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d3c1020d0 0x7f3d3c1a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:24.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.325+0000 7f3d41800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d3c1071a0 0x7f3d3c1a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:24.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.325+0000 7f3d41800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3c19a7a0 0x7f3d3c19ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:24.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.325+0000 7f3d41800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3c112260 con 0x7f3d3c19a7a0 2024-04-23T14:58:24.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.325+0000 7f3d41800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3c1120e0 con 0x7f3d3c1020d0 2024-04-23T14:58:24.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.325+0000 7f3d41800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3c1123e0 con 0x7f3d3c1071a0 2024-04-23T14:58:24.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.326+0000 7f3d33e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d3c1071a0 0x7f3d3c1a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:24.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.326+0000 7f3d40e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3c19a7a0 0x7f3d3c19ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:24.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.326+0000 7f3d33e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d3c1071a0 0x7f3d3c1a0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:42984/0 (socket says 172.21.15.28:42984) 2024-04-23T14:58:24.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.326+0000 7f3d33e00640 1 -- 172.21.15.28:0/777120918 learned_addr learned my addr 172.21.15.28:0/777120918 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:58:24.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.326+0000 7f3d3be00640 1 --2- 172.21.15.28:0/777120918 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d3c1020d0 0x7f3d3c1a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:24.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.326+0000 7f3d33e00640 1 -- 172.21.15.28:0/777120918 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d3c1020d0 msgr2=0x7f3d3c1a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:24.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.326+0000 7f3d33e00640 1 --2- 172.21.15.28:0/777120918 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d3c1020d0 0x7f3d3c1a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:24.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.326+0000 7f3d33e00640 1 -- 172.21.15.28:0/777120918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3c19a7a0 msgr2=0x7f3d3c19ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:24.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.326+0000 7f3d33e00640 1 --2- 172.21.15.28:0/777120918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3c19a7a0 0x7f3d3c19ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:24.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.326+0000 7f3d33e00640 1 -- 172.21.15.28:0/777120918 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d3c19b330 con 0x7f3d3c1071a0 2024-04-23T14:58:24.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.326+0000 7f3d33e00640 1 --2- 172.21.15.28:0/777120918 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d3c1071a0 0x7f3d3c1a0b70 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400e9c0 tx=0x7f3d2400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:58:24.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.327+0000 7f3d39600640 1 -- 172.21.15.28:0/777120918 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d2400cd80 con 0x7f3d3c1071a0 2024-04-23T14:58:24.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.327+0000 7f3d39600640 1 -- 172.21.15.28:0/777120918 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d24005500 con 0x7f3d3c1071a0 2024-04-23T14:58:24.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.327+0000 7f3d39600640 1 -- 172.21.15.28:0/777120918 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d24010690 con 0x7f3d3c1071a0 2024-04-23T14:58:24.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.327+0000 7f3d41800640 1 -- 172.21.15.28:0/777120918 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d3c19b620 con 0x7f3d3c1071a0 2024-04-23T14:58:24.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.327+0000 7f3d41800640 1 -- 172.21.15.28:0/777120918 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d3c1a77b0 con 0x7f3d3c1071a0 2024-04-23T14:58:24.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.328+0000 7f3d41800640 1 -- 172.21.15.28:0/777120918 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d3c1035a0 con 0x7f3d3c1071a0 2024-04-23T14:58:24.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.330+0000 7f3d39600640 1 -- 172.21.15.28:0/777120918 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f3d240046c0 con 0x7f3d3c1071a0 2024-04-23T14:58:24.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.331+0000 7f3d39600640 1 --2- 172.21.15.28:0/777120918 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f3d200a3da0 0x7f3d200a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:24.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.332+0000 7f3d39600640 1 -- 172.21.15.28:0/777120918 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8542+0+0 (secure 0 0 0) 0x7f3d240d0b60 con 0x7f3d3c1071a0 2024-04-23T14:58:24.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.332+0000 7f3d3be00640 1 --2- 172.21.15.28:0/777120918 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f3d200a3da0 0x7f3d200a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:24.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.333+0000 7f3d3be00640 1 --2- 172.21.15.28:0/777120918 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f3d200a3da0 0x7f3d200a6260 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c009230 tx=0x7f3d2c0058d0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:58:24.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.336+0000 7f3d39600640 1 -- 172.21.15.28:0/777120918 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3d3c19b620 con 0x7f3d3c1071a0 2024-04-23T14:58:24.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.544+0000 7f3d41800640 1 -- 172.21.15.28:0/777120918 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3d3c102b20 con 0x7f3d3c1071a0 2024-04-23T14:58:24.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.547+0000 7f3d39600640 1 -- 172.21.15.28:0/777120918 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) ==== 74+0+23120 (secure 0 0 0) 0x7f3d2409d190 con 0x7f3d3c1071a0 2024-04-23T14:58:24.548 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:58:24.548 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":104,"fsid":"1ee6e996-017e-11ef-bc93-c7b262605968","created":"2024-04-23T14:32:34.581082+0000","modified":"2024-04-23T14:58:11.777123+0000","last_up_change":"2024-04-23T14:40:43.375846+0000","last_in_change":"2024-04-23T14:40:23.576939+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:36:27.698112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:41:37.030341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:41:38.444742+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":23,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e0b427bc-be65-4543-99cc-2f53ad98ddad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6803","nonce":2993445336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6805","nonce":2993445336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6809","nonce":2993445336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6807","nonce":2993445336}]},"public_addr":"172.21.15.28:6803/2993445336","cluster_addr":"172.21.15.28:6805/2993445336","heartbeat_back_addr":"172.21.15.28:6809/2993445336","heartbeat_front_addr":"172.21.15.28:6807/2993445336","state":["exists","up"]},{"osd":1,"uuid":"ef527ec9-1bf3-4bb0-8df3-f5a015af557c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6801","nonce":4024093383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6803","nonce":4024093383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6807","nonce":4024093383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6805","nonce":4024093383}]},"public_addr":"172.21.15.38:6801/4024093383","cluster_addr":"172.21.15.38:6803/4024093383","heartbeat_back_addr":"172.21.15.38:6807/4024093383","heartbeat_front_addr":"172.21.15.38:6805/4024093383","state":["exists","up"]},{"osd":2,"uuid":"b33b23d9-adce-4c11-b304-2e117bdf6bfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6801","nonce":570739029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6803","nonce":570739029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6807","nonce":570739029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6805","nonce":570739029}]},"public_addr":"172.21.15.105:6801/570739029","cluster_addr":"172.21.15.105:6803/570739029","heartbeat_back_addr":"172.21.15.105:6807/570739029","heartbeat_front_addr":"172.21.15.105:6805/570739029","state":["exists","up"]},{"osd":3,"uuid":"28bd5608-b426-434e-9db1-bdaa9828bdb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6811","nonce":2480400247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6813","nonce":2480400247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6817","nonce":2480400247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6815","nonce":2480400247}]},"public_addr":"172.21.15.28:6811/2480400247","cluster_addr":"172.21.15.28:6813/2480400247","heartbeat_back_addr":"172.21.15.28:6817/2480400247","heartbeat_front_addr":"172.21.15.28:6815/2480400247","state":["exists","up"]},{"osd":4,"uuid":"e5d1c89b-025f-4d57-8807-59d75c787b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6809","nonce":3772189246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6811","nonce":3772189246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6815","nonce":3772189246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6813","nonce":3772189246}]},"public_addr":"172.21.15.38:6809/3772189246","cluster_addr":"172.21.15.38:6811/3772189246","heartbeat_back_addr":"172.21.15.38:6815/3772189246","heartbeat_front_addr":"172.21.15.38:6813/3772189246","state":["exists","up"]},{"osd":5,"uuid":"9cf9cf4c-aafd-4ed3-bfb1-0527710a0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6809","nonce":928755894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6811","nonce":928755894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6815","nonce":928755894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6813","nonce":928755894}]},"public_addr":"172.21.15.105:6809/928755894","cluster_addr":"172.21.15.105:6811/928755894","heartbeat_back_addr":"172.21.15.105:6815/928755894","heartbeat_front_addr":"172.21.15.105:6813/928755894","state":["exists","up"]},{"osd":6,"uuid":"304db525-d24d-4687-bfa0-f92bdf66d1e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6819","nonce":897979441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6821","nonce":897979441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6825","nonce":897979441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6823","nonce":897979441}]},"public_addr":"172.21.15.28:6819/897979441","cluster_addr":"172.21.15.28:6821/897979441","heartbeat_back_addr":"172.21.15.28:6825/897979441","heartbeat_front_addr":"172.21.15.28:6823/897979441","state":["exists","up"]},{"osd":7,"uuid":"08e01d59-03f3-47b1-b45f-35ec5a50992c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6817","nonce":2400251786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6819","nonce":2400251786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6823","nonce":2400251786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6821","nonce":2400251786}]},"public_addr":"172.21.15.38:6817/2400251786","cluster_addr":"172.21.15.38:6819/2400251786","heartbeat_back_addr":"172.21.15.38:6823/2400251786","heartbeat_front_addr":"172.21.15.38:6821/2400251786","state":["exists","up"]},{"osd":8,"uuid":"64ee1e95-0756-48b1-9c00-472d0c3cbd29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6817","nonce":1682312269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6819","nonce":1682312269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6823","nonce":1682312269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6821","nonce":1682312269}]},"public_addr":"172.21.15.105:6817/1682312269","cluster_addr":"172.21.15.105:6819/1682312269","heartbeat_back_addr":"172.21.15.105:6823/1682312269","heartbeat_front_addr":"172.21.15.105:6821/1682312269","state":["exists","up"]},{"osd":9,"uuid":"18f60893-3ffd-48c2-9db5-a67d35611d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6827","nonce":1292942516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6829","nonce":1292942516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6833","nonce":1292942516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6831","nonce":1292942516}]},"public_addr":"172.21.15.28:6827/1292942516","cluster_addr":"172.21.15.28:6829/1292942516","heartbeat_back_addr":"172.21.15.28:6833/1292942516","heartbeat_front_addr":"172.21.15.28:6831/1292942516","state":["exists","up"]},{"osd":10,"uuid":"76c06725-7040-4118-8894-fe808a87f999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6825","nonce":2444616468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6827","nonce":2444616468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6831","nonce":2444616468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6829","nonce":2444616468}]},"public_addr":"172.21.15.38:6825/2444616468","cluster_addr":"172.21.15.38:6827/2444616468","heartbeat_back_addr":"172.21.15.38:6831/2444616468","heartbeat_front_addr":"172.21.15.38:6829/2444616468","state":["exists","up"]},{"osd":11,"uuid":"e8fc9454-5540-4e2e-91bf-3ba251dbd1a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6825","nonce":3400299562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6827","nonce":3400299562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6831","nonce":3400299562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6829","nonce":3400299562}]},"public_addr":"172.21.15.105:6825/3400299562","cluster_addr":"172.21.15.105:6827/3400299562","heartbeat_back_addr":"172.21.15.105:6831/3400299562","heartbeat_front_addr":"172.21.15.105:6829/3400299562","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:35:33.615395+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:00.545293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:24.940211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:57.684887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:24.519061+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:49.010804+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:22.638478+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:49.559104+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:14.011771+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:47.859950+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:16.298445+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:41.619069+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.2d","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/38494154":"2024-04-24T14:44:01.286264+0000","172.21.15.28:0/905362079":"2024-04-24T14:44:01.286264+0000","172.21.15.28:6801/1997136173":"2024-04-24T14:44:01.286264+0000","172.21.15.28:6800/1997136173":"2024-04-24T14:44:01.286264+0000","172.21.15.28:0/590495740":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/1163172512":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/4290328460":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/4145020172":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/1997097543":"2024-04-24T14:44:01.286264+0000","172.21.15.28:0/3847772687":"2024-04-24T14:44:01.286264+0000","172.21.15.28:6800/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/720626958":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/948450262":"2024-04-24T14:33:07.576363+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":10,"length":1},{"begin":23,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:58:24.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.549+0000 7f3d41800640 1 -- 172.21.15.28:0/777120918 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f3d200a3da0 msgr2=0x7f3d200a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:24.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.549+0000 7f3d41800640 1 --2- 172.21.15.28:0/777120918 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f3d200a3da0 0x7f3d200a6260 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c009230 tx=0x7f3d2c0058d0 comp rx=0 tx=0).stop 2024-04-23T14:58:24.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.549+0000 7f3d41800640 1 -- 172.21.15.28:0/777120918 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d3c1071a0 msgr2=0x7f3d3c1a0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:24.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.549+0000 7f3d41800640 1 --2- 172.21.15.28:0/777120918 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d3c1071a0 0x7f3d3c1a0b70 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400e9c0 tx=0x7f3d2400ee80 comp rx=0 tx=0).stop 2024-04-23T14:58:24.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.550+0000 7f3d41800640 1 -- 172.21.15.28:0/777120918 shutdown_connections 2024-04-23T14:58:24.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.550+0000 7f3d41800640 1 --2- 172.21.15.28:0/777120918 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f3d200a3da0 0x7f3d200a6260 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:24.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.550+0000 7f3d41800640 1 --2- 172.21.15.28:0/777120918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3c19a7a0 0x7f3d3c19ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:24.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.550+0000 7f3d41800640 1 --2- 172.21.15.28:0/777120918 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d3c1071a0 0x7f3d3c1a0b70 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:24.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.550+0000 7f3d41800640 1 --2- 172.21.15.28:0/777120918 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d3c1020d0 0x7f3d3c1a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:24.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.550+0000 7f3d41800640 1 -- 172.21.15.28:0/777120918 >> 172.21.15.28:0/777120918 conn(0x7f3d3c0fd970 msgr2=0x7f3d3c0ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:58:24.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.550+0000 7f3d41800640 1 -- 172.21.15.28:0/777120918 shutdown_connections 2024-04-23T14:58:24.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.550+0000 7f3d41800640 1 -- 172.21.15.28:0/777120918 wait complete. 2024-04-23T14:58:24.570 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T14:36:27.698112+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T14:41:37.030341+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T14:41:38.444742+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 23, 'snap_epoch': 103, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T14:58:24.571 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T14:58:24.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.786+0000 7f6e42800640 1 -- 172.21.15.28:0/4229949170 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6e3c102ac0 msgr2=0x7f6e3c106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:24.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.786+0000 7f6e42800640 1 --2- 172.21.15.28:0/4229949170 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6e3c102ac0 0x7f6e3c106b00 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f6e28009900 tx=0x7f6e2802a400 comp rx=0 tx=0).stop 2024-04-23T14:58:24.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.787+0000 7f6e42800640 1 -- 172.21.15.28:0/4229949170 shutdown_connections 2024-04-23T14:58:24.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.787+0000 7f6e42800640 1 --2- 172.21.15.28:0/4229949170 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e3c1071e0 0x7f6e3c10edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:24.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.787+0000 7f6e42800640 1 --2- 172.21.15.28:0/4229949170 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6e3c102ac0 0x7f6e3c106b00 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:24.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.787+0000 7f6e42800640 1 --2- 172.21.15.28:0/4229949170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e3c102110 0x7f6e3c1024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:24.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.787+0000 7f6e42800640 1 -- 172.21.15.28:0/4229949170 >> 172.21.15.28:0/4229949170 conn(0x7f6e3c0fd9d0 msgr2=0x7f6e3c0ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:58:24.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.787+0000 7f6e42800640 1 -- 172.21.15.28:0/4229949170 shutdown_connections 2024-04-23T14:58:24.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.787+0000 7f6e42800640 1 -- 172.21.15.28:0/4229949170 wait complete. 2024-04-23T14:58:24.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.788+0000 7f6e42800640 1 Processor -- start 2024-04-23T14:58:24.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.788+0000 7f6e42800640 1 -- start start 2024-04-23T14:58:24.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.789+0000 7f6e42800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e3c102110 0x7f6e3c19e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:24.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.789+0000 7f6e42800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6e3c1071e0 0x7f6e3c19e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:24.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.789+0000 7f6e42800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e3c1985d0 0x7f6e3c198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:24.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.789+0000 7f6e42800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6e3c112270 con 0x7f6e3c102110 2024-04-23T14:58:24.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.789+0000 7f6e42800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6e3c1120f0 con 0x7f6e3c1985d0 2024-04-23T14:58:24.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.789+0000 7f6e42800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6e3c1123f0 con 0x7f6e3c1071e0 2024-04-23T14:58:24.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.789+0000 7f6e41400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e3c102110 0x7f6e3c19e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:24.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.789+0000 7f6e40a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6e3c1071e0 0x7f6e3c19e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:24.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.789+0000 7f6e41400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e3c102110 0x7f6e3c19e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34726/0 (socket says 172.21.15.28:34726) 2024-04-23T14:58:24.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.789+0000 7f6e41400640 1 -- 172.21.15.28:0/3101056787 learned_addr learned my addr 172.21.15.28:0/3101056787 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:58:24.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.789+0000 7f6e41e00640 1 --2- 172.21.15.28:0/3101056787 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e3c1985d0 0x7f6e3c198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:24.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.789+0000 7f6e41400640 1 -- 172.21.15.28:0/3101056787 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6e3c1071e0 msgr2=0x7f6e3c19e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:24.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.789+0000 7f6e41400640 1 --2- 172.21.15.28:0/3101056787 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6e3c1071e0 0x7f6e3c19e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:24.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.789+0000 7f6e41400640 1 -- 172.21.15.28:0/3101056787 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e3c1985d0 msgr2=0x7f6e3c198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:24.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.789+0000 7f6e41400640 1 --2- 172.21.15.28:0/3101056787 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e3c1985d0 0x7f6e3c198a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:24.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.790+0000 7f6e41400640 1 -- 172.21.15.28:0/3101056787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e3c1990f0 con 0x7f6e3c102110 2024-04-23T14:58:24.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.790+0000 7f6e41400640 1 --2- 172.21.15.28:0/3101056787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e3c102110 0x7f6e3c19e420 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c00e9f0 tx=0x7f6e2c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:58:24.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.790+0000 7f6e3a000640 1 -- 172.21.15.28:0/3101056787 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e2c00cd80 con 0x7f6e3c102110 2024-04-23T14:58:24.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.790+0000 7f6e42800640 1 -- 172.21.15.28:0/3101056787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e3c1993e0 con 0x7f6e3c102110 2024-04-23T14:58:24.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.790+0000 7f6e3a000640 1 -- 172.21.15.28:0/3101056787 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e2c010910 con 0x7f6e3c102110 2024-04-23T14:58:24.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.790+0000 7f6e3a000640 1 -- 172.21.15.28:0/3101056787 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e2c018990 con 0x7f6e3c102110 2024-04-23T14:58:24.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.790+0000 7f6e42800640 1 -- 172.21.15.28:0/3101056787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e3c1a5630 con 0x7f6e3c102110 2024-04-23T14:58:24.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.792+0000 7f6e42800640 1 -- 172.21.15.28:0/3101056787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e3c068d40 con 0x7f6e3c102110 2024-04-23T14:58:24.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.794+0000 7f6e3a000640 1 -- 172.21.15.28:0/3101056787 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f6e2c010ac0 con 0x7f6e3c102110 2024-04-23T14:58:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.796+0000 7f6e3a000640 1 --2- 172.21.15.28:0/3101056787 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f6e200a3d20 0x7f6e200a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.796+0000 7f6e3a000640 1 -- 172.21.15.28:0/3101056787 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8542+0+0 (secure 0 0 0) 0x7f6e2c0d0ba0 con 0x7f6e3c102110 2024-04-23T14:58:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.796+0000 7f6e40a00640 1 --2- 172.21.15.28:0/3101056787 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f6e200a3d20 0x7f6e200a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:24.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.797+0000 7f6e40a00640 1 --2- 172.21.15.28:0/3101056787 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f6e200a3d20 0x7f6e200a61e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6e28004770 tx=0x7f6e280058d0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:58:24.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:24.800+0000 7f6e3a000640 1 -- 172.21.15.28:0/3101056787 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6e2c09cb80 con 0x7f6e3c102110 2024-04-23T14:58:25.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.011+0000 7f6e42800640 1 -- 172.21.15.28:0/3101056787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6e3c10a410 con 0x7f6e3c102110 2024-04-23T14:58:25.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.012+0000 7f6e3a000640 1 -- 172.21.15.28:0/3101056787 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v104) ==== 93+0+10 (secure 0 0 0) 0x7f6e2c0a0e80 con 0x7f6e3c102110 2024-04-23T14:58:25.013 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-23T14:58:25.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.015+0000 7f6e42800640 1 -- 172.21.15.28:0/3101056787 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f6e200a3d20 msgr2=0x7f6e200a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:25.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.015+0000 7f6e42800640 1 --2- 172.21.15.28:0/3101056787 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f6e200a3d20 0x7f6e200a61e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6e28004770 tx=0x7f6e280058d0 comp rx=0 tx=0).stop 2024-04-23T14:58:25.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.015+0000 7f6e42800640 1 -- 172.21.15.28:0/3101056787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e3c102110 msgr2=0x7f6e3c19e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:25.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.015+0000 7f6e42800640 1 --2- 172.21.15.28:0/3101056787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e3c102110 0x7f6e3c19e420 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c00e9f0 tx=0x7f6e2c00eeb0 comp rx=0 tx=0).stop 2024-04-23T14:58:25.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.016+0000 7f6e42800640 1 -- 172.21.15.28:0/3101056787 shutdown_connections 2024-04-23T14:58:25.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.016+0000 7f6e42800640 1 --2- 172.21.15.28:0/3101056787 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f6e200a3d20 0x7f6e200a61e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:25.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.016+0000 7f6e42800640 1 --2- 172.21.15.28:0/3101056787 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e3c1985d0 0x7f6e3c198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:25.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.016+0000 7f6e42800640 1 --2- 172.21.15.28:0/3101056787 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6e3c1071e0 0x7f6e3c19e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:25.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.016+0000 7f6e42800640 1 --2- 172.21.15.28:0/3101056787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e3c102110 0x7f6e3c19e420 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:25.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.016+0000 7f6e42800640 1 -- 172.21.15.28:0/3101056787 >> 172.21.15.28:0/3101056787 conn(0x7f6e3c0fd9d0 msgr2=0x7f6e3c0feda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:58:25.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.016+0000 7f6e42800640 1 -- 172.21.15.28:0/3101056787 shutdown_connections 2024-04-23T14:58:25.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.016+0000 7f6e42800640 1 -- 172.21.15.28:0/3101056787 wait complete. 2024-04-23T14:58:25.036 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T14:58:25.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:24 smithi028 ceph-mon[27810]: pgmap v450: 97 pgs: 97 active+clean; 835 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 21 MiB/s wr, 261 op/s 2024-04-23T14:58:25.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:24 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/777120918' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:58:25.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:24 smithi038 ceph-mon[32348]: pgmap v450: 97 pgs: 97 active+clean; 835 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 21 MiB/s wr, 261 op/s 2024-04-23T14:58:25.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:24 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/777120918' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:58:25.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.250+0000 7f6b0b000640 1 -- 172.21.15.28:0/1311827522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b041030d0 msgr2=0x7f6b041034b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:25.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.250+0000 7f6b0b000640 1 --2- 172.21.15.28:0/1311827522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b041030d0 0x7f6b041034b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6af8009900 tx=0x7f6af802a490 comp rx=0 tx=0).stop 2024-04-23T14:58:25.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.250+0000 7f6b0b000640 1 -- 172.21.15.28:0/1311827522 shutdown_connections 2024-04-23T14:58:25.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.250+0000 7f6b0b000640 1 --2- 172.21.15.28:0/1311827522 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b0410c4f0 0x7f6b0410e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:25.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.250+0000 7f6b0b000640 1 --2- 172.21.15.28:0/1311827522 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6b041039f0 0x7f6b0410bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:25.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.250+0000 7f6b0b000640 1 --2- 172.21.15.28:0/1311827522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b041030d0 0x7f6b041034b0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:25.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.250+0000 7f6b0b000640 1 -- 172.21.15.28:0/1311827522 >> 172.21.15.28:0/1311827522 conn(0x7f6b040fc910 msgr2=0x7f6b040fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:58:25.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.250+0000 7f6b0b000640 1 -- 172.21.15.28:0/1311827522 shutdown_connections 2024-04-23T14:58:25.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.250+0000 7f6b0b000640 1 -- 172.21.15.28:0/1311827522 wait complete. 2024-04-23T14:58:25.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.251+0000 7f6b0b000640 1 Processor -- start 2024-04-23T14:58:25.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.251+0000 7f6b0b000640 1 -- start start 2024-04-23T14:58:25.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.252+0000 7f6b0b000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6b041030d0 0x7f6b04100d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:25.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.252+0000 7f6b0b000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b041039f0 0x7f6b04101260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:25.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.252+0000 7f6b0b000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b0410c4f0 0x7f6b041017a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:25.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.252+0000 7f6b0b000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6b04112300 con 0x7f6b0410c4f0 2024-04-23T14:58:25.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.252+0000 7f6b0b000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6b04112180 con 0x7f6b041039f0 2024-04-23T14:58:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.252+0000 7f6b0b000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6b04112480 con 0x7f6b041030d0 2024-04-23T14:58:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.252+0000 7f6b0a600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b0410c4f0 0x7f6b041017a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.252+0000 7f6b09c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6b041030d0 0x7f6b04100d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.252+0000 7f6b09200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b041039f0 0x7f6b04101260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.252+0000 7f6b09c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6b041030d0 0x7f6b04100d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:43014/0 (socket says 172.21.15.28:43014) 2024-04-23T14:58:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.252+0000 7f6b09c00640 1 -- 172.21.15.28:0/3965645383 learned_addr learned my addr 172.21.15.28:0/3965645383 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:58:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.252+0000 7f6b09c00640 1 -- 172.21.15.28:0/3965645383 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b041039f0 msgr2=0x7f6b04101260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.252+0000 7f6b09c00640 1 --2- 172.21.15.28:0/3965645383 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b041039f0 0x7f6b04101260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.252+0000 7f6b09c00640 1 -- 172.21.15.28:0/3965645383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b0410c4f0 msgr2=0x7f6b041017a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.252+0000 7f6b09c00640 1 --2- 172.21.15.28:0/3965645383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b0410c4f0 0x7f6b041017a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:25.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.252+0000 7f6b09c00640 1 -- 172.21.15.28:0/3965645383 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b041a6ce0 con 0x7f6b041030d0 2024-04-23T14:58:25.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.253+0000 7f6b09c00640 1 --2- 172.21.15.28:0/3965645383 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6b041030d0 0x7f6b04100d20 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f6af8004770 tx=0x7f6af8003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:58:25.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.253+0000 7f6b02000640 1 -- 172.21.15.28:0/3965645383 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6af803f070 con 0x7f6b041030d0 2024-04-23T14:58:25.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.253+0000 7f6b02000640 1 -- 172.21.15.28:0/3965645383 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6af8031ca0 con 0x7f6b041030d0 2024-04-23T14:58:25.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.253+0000 7f6b0b000640 1 -- 172.21.15.28:0/3965645383 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b041a6f70 con 0x7f6b041030d0 2024-04-23T14:58:25.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.253+0000 7f6b02000640 1 -- 172.21.15.28:0/3965645383 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6af8035030 con 0x7f6b041030d0 2024-04-23T14:58:25.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.253+0000 7f6b0b000640 1 -- 172.21.15.28:0/3965645383 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b041a73d0 con 0x7f6b041030d0 2024-04-23T14:58:25.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.254+0000 7f6b0b000640 1 -- 172.21.15.28:0/3965645383 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b0406aee0 con 0x7f6b041030d0 2024-04-23T14:58:25.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.257+0000 7f6b02000640 1 -- 172.21.15.28:0/3965645383 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f6af8042070 con 0x7f6b041030d0 2024-04-23T14:58:25.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.258+0000 7f6b02000640 1 --2- 172.21.15.28:0/3965645383 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f6aec0a3d20 0x7f6aec0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:25.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.258+0000 7f6b02000640 1 -- 172.21.15.28:0/3965645383 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8542+0+0 (secure 0 0 0) 0x7f6af80f1120 con 0x7f6b041030d0 2024-04-23T14:58:25.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.258+0000 7f6b09200640 1 --2- 172.21.15.28:0/3965645383 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f6aec0a3d20 0x7f6aec0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:25.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.259+0000 7f6b09200640 1 --2- 172.21.15.28:0/3965645383 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f6aec0a3d20 0x7f6aec0a61e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6b041027b0 tx=0x7f6af000b2e0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:58:25.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.260+0000 7f6b02000640 1 -- 172.21.15.28:0/3965645383 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6af80bd210 con 0x7f6b041030d0 2024-04-23T14:58:25.285 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:24 smithi105 ceph-mon[32555]: pgmap v450: 97 pgs: 97 active+clean; 835 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 21 MiB/s wr, 261 op/s 2024-04-23T14:58:25.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:24 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/777120918' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:58:25.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.486+0000 7f6b0b000640 1 -- 172.21.15.28:0/3965645383 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6b04107590 con 0x7f6b041030d0 2024-04-23T14:58:25.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.487+0000 7f6b02000640 1 -- 172.21.15.28:0/3965645383 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v104) ==== 104+0+11 (secure 0 0 0) 0x7f6af80c1510 con 0x7f6b041030d0 2024-04-23T14:58:25.488 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-23T14:58:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.490+0000 7f6b0b000640 1 -- 172.21.15.28:0/3965645383 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f6aec0a3d20 msgr2=0x7f6aec0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.490+0000 7f6b0b000640 1 --2- 172.21.15.28:0/3965645383 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f6aec0a3d20 0x7f6aec0a61e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6b041027b0 tx=0x7f6af000b2e0 comp rx=0 tx=0).stop 2024-04-23T14:58:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.490+0000 7f6b0b000640 1 -- 172.21.15.28:0/3965645383 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6b041030d0 msgr2=0x7f6b04100d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.490+0000 7f6b0b000640 1 --2- 172.21.15.28:0/3965645383 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6b041030d0 0x7f6b04100d20 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f6af8004770 tx=0x7f6af8003c10 comp rx=0 tx=0).stop 2024-04-23T14:58:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.490+0000 7f6b0b000640 1 -- 172.21.15.28:0/3965645383 shutdown_connections 2024-04-23T14:58:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.490+0000 7f6b0b000640 1 --2- 172.21.15.28:0/3965645383 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f6aec0a3d20 0x7f6aec0a61e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.490+0000 7f6b0b000640 1 --2- 172.21.15.28:0/3965645383 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b041039f0 0x7f6b04101260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:25.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.490+0000 7f6b0b000640 1 --2- 172.21.15.28:0/3965645383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b0410c4f0 0x7f6b041017a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:25.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.490+0000 7f6b0b000640 1 --2- 172.21.15.28:0/3965645383 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6b041030d0 0x7f6b04100d20 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:25.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.490+0000 7f6b0b000640 1 -- 172.21.15.28:0/3965645383 >> 172.21.15.28:0/3965645383 conn(0x7f6b040fc910 msgr2=0x7f6b040fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:58:25.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.490+0000 7f6b0b000640 1 -- 172.21.15.28:0/3965645383 shutdown_connections 2024-04-23T14:58:25.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.490+0000 7f6b0b000640 1 -- 172.21.15.28:0/3965645383 wait complete. 2024-04-23T14:58:25.510 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T14:58:25.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.726+0000 7f942e400640 1 -- 172.21.15.28:0/3795909389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94281044b0 msgr2=0x7f9428104890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:25.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.726+0000 7f942e400640 1 --2- 172.21.15.28:0/3795909389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94281044b0 0x7f9428104890 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f9418009900 tx=0x7f941802a410 comp rx=0 tx=0).stop 2024-04-23T14:58:25.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.726+0000 7f942e400640 1 -- 172.21.15.28:0/3795909389 shutdown_connections 2024-04-23T14:58:25.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.726+0000 7f942e400640 1 --2- 172.21.15.28:0/3795909389 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9428104e60 0x7f9428108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:25.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.726+0000 7f942e400640 1 --2- 172.21.15.28:0/3795909389 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9428109580 0x7f9428111150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:25.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.726+0000 7f942e400640 1 --2- 172.21.15.28:0/3795909389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94281044b0 0x7f9428104890 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:25.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.726+0000 7f942e400640 1 -- 172.21.15.28:0/3795909389 >> 172.21.15.28:0/3795909389 conn(0x7f94280ffd90 msgr2=0x7f94281021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:58:25.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.726+0000 7f942e400640 1 -- 172.21.15.28:0/3795909389 shutdown_connections 2024-04-23T14:58:25.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.727+0000 7f942e400640 1 -- 172.21.15.28:0/3795909389 wait complete. 2024-04-23T14:58:25.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.728+0000 7f942e400640 1 Processor -- start 2024-04-23T14:58:25.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.728+0000 7f942e400640 1 -- start start 2024-04-23T14:58:25.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.728+0000 7f942e400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94281044b0 0x7f94281a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:25.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.728+0000 7f942e400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9428109580 0x7f94281a0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:25.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.728+0000 7f942e400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f942819aa20 0x7f942819aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:25.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.728+0000 7f942e400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9428114640 con 0x7f94281044b0 2024-04-23T14:58:25.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.728+0000 7f942e400640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f94281144c0 con 0x7f942819aa20 2024-04-23T14:58:25.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.728+0000 7f942e400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f94281147c0 con 0x7f9428109580 2024-04-23T14:58:25.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.728+0000 7f9427400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94281044b0 0x7f94281a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:25.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.728+0000 7f9427400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94281044b0 0x7f94281a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34778/0 (socket says 172.21.15.28:34778) 2024-04-23T14:58:25.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.728+0000 7f9427e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f942819aa20 0x7f942819aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:25.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.728+0000 7f9427400640 1 -- 172.21.15.28:0/3549654427 learned_addr learned my addr 172.21.15.28:0/3549654427 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:58:25.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.729+0000 7f9426a00640 1 --2- 172.21.15.28:0/3549654427 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9428109580 0x7f94281a0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:25.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.729+0000 7f9427e00640 1 -- 172.21.15.28:0/3549654427 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9428109580 msgr2=0x7f94281a0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:25.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.729+0000 7f9427e00640 1 --2- 172.21.15.28:0/3549654427 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9428109580 0x7f94281a0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:25.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.729+0000 7f9427e00640 1 -- 172.21.15.28:0/3549654427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94281044b0 msgr2=0x7f94281a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.729+0000 7f9427e00640 1 --2- 172.21.15.28:0/3549654427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94281044b0 0x7f94281a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.729+0000 7f9427e00640 1 -- 172.21.15.28:0/3549654427 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f942819b540 con 0x7f942819aa20 2024-04-23T14:58:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.729+0000 7f9427e00640 1 --2- 172.21.15.28:0/3549654427 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f942819aa20 0x7f942819aed0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f9420009fd0 tx=0x7f942000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:58:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.729+0000 7f941fe00640 1 -- 172.21.15.28:0/3549654427 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f942000ce20 con 0x7f942819aa20 2024-04-23T14:58:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.730+0000 7f942e400640 1 -- 172.21.15.28:0/3549654427 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f942819b830 con 0x7f942819aa20 2024-04-23T14:58:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.730+0000 7f941fe00640 1 -- 172.21.15.28:0/3549654427 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9420005020 con 0x7f942819aa20 2024-04-23T14:58:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.730+0000 7f941fe00640 1 -- 172.21.15.28:0/3549654427 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9420010720 con 0x7f942819aa20 2024-04-23T14:58:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.730+0000 7f942e400640 1 -- 172.21.15.28:0/3549654427 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94281a7a30 con 0x7f942819aa20 2024-04-23T14:58:25.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.731+0000 7f942e400640 1 -- 172.21.15.28:0/3549654427 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9428069270 con 0x7f942819aa20 2024-04-23T14:58:25.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.733+0000 7f941fe00640 1 -- 172.21.15.28:0/3549654427 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f9420005540 con 0x7f942819aa20 2024-04-23T14:58:25.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.734+0000 7f941fe00640 1 --2- 172.21.15.28:0/3549654427 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f940c0a3d20 0x7f940c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:25.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.734+0000 7f941fe00640 1 -- 172.21.15.28:0/3549654427 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8542+0+0 (secure 0 0 0) 0x7f94200d0e00 con 0x7f942819aa20 2024-04-23T14:58:25.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.735+0000 7f9427400640 1 --2- 172.21.15.28:0/3549654427 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f940c0a3d20 0x7f940c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:25.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.735+0000 7f9427400640 1 --2- 172.21.15.28:0/3549654427 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f940c0a3d20 0x7f940c0a61e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f94180096c0 tx=0x7f9418005a20 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:58:25.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.739+0000 7f941fe00640 1 -- 172.21.15.28:0/3549654427 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f942009d430 con 0x7f942819aa20 2024-04-23T14:58:25.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.971+0000 7f942e400640 1 -- 172.21.15.28:0/3549654427 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9428104e60 con 0x7f942819aa20 2024-04-23T14:58:25.976 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-23T14:58:25.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.975+0000 7f941fe00640 1 -- 172.21.15.28:0/3549654427 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v104) ==== 100+0+11 (secure 0 0 0) 0x7f942009cd30 con 0x7f942819aa20 2024-04-23T14:58:25.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.979+0000 7f942e400640 1 -- 172.21.15.28:0/3549654427 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f940c0a3d20 msgr2=0x7f940c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:25.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.980+0000 7f942e400640 1 --2- 172.21.15.28:0/3549654427 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f940c0a3d20 0x7f940c0a61e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f94180096c0 tx=0x7f9418005a20 comp rx=0 tx=0).stop 2024-04-23T14:58:25.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.980+0000 7f942e400640 1 -- 172.21.15.28:0/3549654427 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f942819aa20 msgr2=0x7f942819aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:25.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.980+0000 7f942e400640 1 --2- 172.21.15.28:0/3549654427 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f942819aa20 0x7f942819aed0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f9420009fd0 tx=0x7f942000ef10 comp rx=0 tx=0).stop 2024-04-23T14:58:25.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.980+0000 7f942e400640 1 -- 172.21.15.28:0/3549654427 shutdown_connections 2024-04-23T14:58:25.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.980+0000 7f942e400640 1 --2- 172.21.15.28:0/3549654427 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f940c0a3d20 0x7f940c0a61e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:25.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.980+0000 7f942e400640 1 --2- 172.21.15.28:0/3549654427 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f942819aa20 0x7f942819aed0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:25.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.980+0000 7f942e400640 1 --2- 172.21.15.28:0/3549654427 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9428109580 0x7f94281a0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:25.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.980+0000 7f942e400640 1 --2- 172.21.15.28:0/3549654427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94281044b0 0x7f94281a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:25.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.980+0000 7f942e400640 1 -- 172.21.15.28:0/3549654427 >> 172.21.15.28:0/3549654427 conn(0x7f94280ffd90 msgr2=0x7f94281010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:58:25.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.982+0000 7f942e400640 1 -- 172.21.15.28:0/3549654427 shutdown_connections 2024-04-23T14:58:25.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:25.982+0000 7f942e400640 1 -- 172.21.15.28:0/3549654427 wait complete. 2024-04-23T14:58:26.011 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-23T14:58:26.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:25 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3101056787' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:58:26.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:25 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3965645383' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T14:58:26.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:25 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3101056787' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:58:26.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:25 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3965645383' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T14:58:26.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.193+0000 7f5f8d200640 1 -- 172.21.15.28:0/1351396270 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5f88102240 msgr2=0x7f5f88102620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:26.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.193+0000 7f5f8d200640 1 --2- 172.21.15.28:0/1351396270 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5f88102240 0x7f5f88102620 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f5f78009900 tx=0x7f5f7802a3f0 comp rx=0 tx=0).stop 2024-04-23T14:58:26.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.193+0000 7f5f8d200640 1 -- 172.21.15.28:0/1351396270 shutdown_connections 2024-04-23T14:58:26.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.193+0000 7f5f8d200640 1 --2- 172.21.15.28:0/1351396270 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f88107310 0x7f5f8810eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:26.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.193+0000 7f5f8d200640 1 --2- 172.21.15.28:0/1351396270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f88102bf0 0x7f5f88106c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:26.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.193+0000 7f5f8d200640 1 --2- 172.21.15.28:0/1351396270 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5f88102240 0x7f5f88102620 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:26.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.193+0000 7f5f8d200640 1 -- 172.21.15.28:0/1351396270 >> 172.21.15.28:0/1351396270 conn(0x7f5f880793c0 msgr2=0x7f5f880797d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:58:26.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.193+0000 7f5f8d200640 1 -- 172.21.15.28:0/1351396270 shutdown_connections 2024-04-23T14:58:26.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.193+0000 7f5f8d200640 1 -- 172.21.15.28:0/1351396270 wait complete. 2024-04-23T14:58:26.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.194+0000 7f5f8d200640 1 Processor -- start 2024-04-23T14:58:26.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.195+0000 7f5f8d200640 1 -- start start 2024-04-23T14:58:26.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.195+0000 7f5f8d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f88102240 0x7f5f881a0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:26.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.195+0000 7f5f8d200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5f88107310 0x7f5f881a0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:26.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.195+0000 7f5f8d200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f8819a910 0x7f5f8819adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:26.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.195+0000 7f5f8d200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5f881123f0 con 0x7f5f88102240 2024-04-23T14:58:26.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.195+0000 7f5f8d200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5f88112270 con 0x7f5f8819a910 2024-04-23T14:58:26.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.195+0000 7f5f8d200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5f88112570 con 0x7f5f88107310 2024-04-23T14:58:26.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.195+0000 7f5f85e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f88102240 0x7f5f881a0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:26.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.195+0000 7f5f85e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f88102240 0x7f5f881a0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34796/0 (socket says 172.21.15.28:34796) 2024-04-23T14:58:26.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.195+0000 7f5f85e00640 1 -- 172.21.15.28:0/3211144062 learned_addr learned my addr 172.21.15.28:0/3211144062 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:58:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.195+0000 7f5f86800640 1 --2- 172.21.15.28:0/3211144062 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f8819a910 0x7f5f8819adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.195+0000 7f5f85400640 1 --2- 172.21.15.28:0/3211144062 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5f88107310 0x7f5f881a0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.196+0000 7f5f85e00640 1 -- 172.21.15.28:0/3211144062 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5f88107310 msgr2=0x7f5f881a0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.196+0000 7f5f85e00640 1 --2- 172.21.15.28:0/3211144062 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5f88107310 0x7f5f881a0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.196+0000 7f5f85e00640 1 -- 172.21.15.28:0/3211144062 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f8819a910 msgr2=0x7f5f8819adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.196+0000 7f5f85e00640 1 --2- 172.21.15.28:0/3211144062 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f8819a910 0x7f5f8819adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.196+0000 7f5f85e00640 1 -- 172.21.15.28:0/3211144062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f8819b430 con 0x7f5f88102240 2024-04-23T14:58:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.196+0000 7f5f86800640 1 --2- 172.21.15.28:0/3211144062 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f8819a910 0x7f5f8819adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:58:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.196+0000 7f5f85e00640 1 --2- 172.21.15.28:0/3211144062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f88102240 0x7f5f881a0760 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f5f78004770 tx=0x7f5f78003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:58:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.196+0000 7f5f7ea00640 1 -- 172.21.15.28:0/3211144062 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f7803f070 con 0x7f5f88102240 2024-04-23T14:58:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.196+0000 7f5f7ea00640 1 -- 172.21.15.28:0/3211144062 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f78031ca0 con 0x7f5f88102240 2024-04-23T14:58:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.196+0000 7f5f7ea00640 1 -- 172.21.15.28:0/3211144062 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f78035030 con 0x7f5f88102240 2024-04-23T14:58:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.196+0000 7f5f8d200640 1 -- 172.21.15.28:0/3211144062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f8819b6c0 con 0x7f5f88102240 2024-04-23T14:58:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.197+0000 7f5f8d200640 1 -- 172.21.15.28:0/3211144062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f881a78b0 con 0x7f5f88102240 2024-04-23T14:58:26.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.198+0000 7f5f8d200640 1 -- 172.21.15.28:0/3211144062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f88069270 con 0x7f5f88102240 2024-04-23T14:58:26.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.200+0000 7f5f7ea00640 1 -- 172.21.15.28:0/3211144062 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f5f78042070 con 0x7f5f88102240 2024-04-23T14:58:26.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.201+0000 7f5f7ea00640 1 --2- 172.21.15.28:0/3211144062 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5f6c0a3d20 0x7f5f6c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:58:26.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.201+0000 7f5f7ea00640 1 -- 172.21.15.28:0/3211144062 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8542+0+0 (secure 0 0 0) 0x7f5f780f0e10 con 0x7f5f88102240 2024-04-23T14:58:26.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.201+0000 7f5f85400640 1 --2- 172.21.15.28:0/3211144062 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5f6c0a3d20 0x7f5f6c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:58:26.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.202+0000 7f5f85400640 1 --2- 172.21.15.28:0/3211144062 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5f6c0a3d20 0x7f5f6c0a61e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5f8810f330 tx=0x7f5f74005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:58:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.205+0000 7f5f7ea00640 1 -- 172.21.15.28:0/3211144062 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5f780f4030 con 0x7f5f88102240 2024-04-23T14:58:26.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:25 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3101056787' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:58:26.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:25 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3965645383' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T14:58:26.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.472+0000 7f5f8d200640 1 -- 172.21.15.28:0/3211144062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f5f88102cc0 con 0x7f5f88102240 2024-04-23T14:58:26.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.506+0000 7f5f7ea00640 1 -- 172.21.15.28:0/3211144062 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v27) ==== 81+0+746062 (secure 0 0 0) 0x7f5f780bceb0 con 0x7f5f88102240 2024-04-23T14:58:26.512 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:58:26.512 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:58:26.512 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 27, 2024-04-23T14:58:26.512 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.512 INFO:teuthology.orchestra.run.smithi028.stdout: "active_gid": 25153, 2024-04-23T14:58:26.512 INFO:teuthology.orchestra.run.smithi028.stdout: "active_name": "x", 2024-04-23T14:58:26.513 INFO:teuthology.orchestra.run.smithi028.stdout: "active_addrs": { 2024-04-23T14:58:26.513 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-04-23T14:58:26.513 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:26.513 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-04-23T14:58:26.513 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6800", 2024-04-23T14:58:26.513 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1615546338 2024-04-23T14:58:26.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.513 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:26.513 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-04-23T14:58:26.514 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-04-23T14:58:26.514 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1615546338 2024-04-23T14:58:26.514 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.514 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-23T14:58:26.514 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.514 INFO:teuthology.orchestra.run.smithi028.stdout: "active_addr": "172.21.15.28:6801/1615546338", 2024-04-23T14:58:26.514 INFO:teuthology.orchestra.run.smithi028.stdout: "active_change": "2024-04-23T14:44:01.286709+0000", 2024-04-23T14:58:26.514 INFO:teuthology.orchestra.run.smithi028.stdout: "active_mgr_features": 4540701547738038271, 2024-04-23T14:58:26.515 INFO:teuthology.orchestra.run.smithi028.stdout: "available": true, 2024-04-23T14:58:26.515 INFO:teuthology.orchestra.run.smithi028.stdout: "standbys": [ 2024-04-23T14:58:26.515 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:26.515 INFO:teuthology.orchestra.run.smithi028.stdout: "gid": 25144, 2024-04-23T14:58:26.515 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "z", 2024-04-23T14:58:26.515 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_features": 4540701547738038271, 2024-04-23T14:58:26.515 INFO:teuthology.orchestra.run.smithi028.stdout: "available_modules": [ 2024-04-23T14:58:26.515 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:26.516 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "alerts", 2024-04-23T14:58:26.516 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:26.516 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:26.516 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:26.516 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-23T14:58:26.516 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-23T14:58:26.516 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:26.516 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.517 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.517 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-23T14:58:26.517 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.517 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.517 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.517 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How frequently to reexamine health status", 2024-04-23T14:58:26.517 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.517 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.518 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.518 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:26.518 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:26.518 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.518 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.518 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.518 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.519 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.519 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.519 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.519 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.519 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.519 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.519 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.519 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.519 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.520 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.520 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.520 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.520 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.520 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.520 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:26.520 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:26.520 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.521 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.521 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.521 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.521 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.521 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.521 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.521 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.521 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.521 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.522 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.522 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:26.522 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:26.522 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.522 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.522 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.522 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:26.523 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.523 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.523 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.523 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.523 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.523 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.523 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.523 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.523 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.524 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.524 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.524 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.524 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.524 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.524 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.524 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:26.524 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:26.525 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.525 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.525 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.525 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.525 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.525 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.525 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.525 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.526 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.526 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.526 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.526 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_destination": { 2024-04-23T14:58:26.526 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_destination", 2024-04-23T14:58:26.526 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.526 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.527 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.527 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.527 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.527 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.527 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.527 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email address to send alerts to", 2024-04-23T14:58:26.527 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.527 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.528 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.528 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_from_name": { 2024-04-23T14:58:26.528 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_from_name", 2024-04-23T14:58:26.528 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.528 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.528 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.528 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "Ceph", 2024-04-23T14:58:26.529 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.529 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.529 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.529 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email From: name", 2024-04-23T14:58:26.529 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.529 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.529 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.529 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.529 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_host": { 2024-04-23T14:58:26.530 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_host", 2024-04-23T14:58:26.530 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.530 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.530 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.530 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.530 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.530 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.530 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.530 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP server", 2024-04-23T14:58:26.531 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.531 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.531 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.531 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_password": { 2024-04-23T14:58:26.531 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_password", 2024-04-23T14:58:26.531 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.531 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.531 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.532 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.532 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.532 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.532 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.532 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Password to authenticate with", 2024-04-23T14:58:26.532 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.532 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.532 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.532 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.532 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_port": { 2024-04-23T14:58:26.533 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_port", 2024-04-23T14:58:26.533 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:26.533 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.533 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.533 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "465", 2024-04-23T14:58:26.533 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.533 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.533 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.534 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP port", 2024-04-23T14:58:26.534 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.534 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.534 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.534 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_sender": { 2024-04-23T14:58:26.534 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_sender", 2024-04-23T14:58:26.534 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.534 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.535 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.535 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.535 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.535 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.535 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.535 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP envelope sender", 2024-04-23T14:58:26.535 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.535 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.535 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.536 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_ssl": { 2024-04-23T14:58:26.536 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_ssl", 2024-04-23T14:58:26.536 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.536 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.536 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.536 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:26.536 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.537 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.537 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.537 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-23T14:58:26.537 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.537 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.537 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.537 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_user": { 2024-04-23T14:58:26.537 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_user", 2024-04-23T14:58:26.538 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.538 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.538 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.538 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.538 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.538 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.538 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.538 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "User to authenticate as", 2024-04-23T14:58:26.539 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.539 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.539 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.539 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.539 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.539 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.539 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:26.539 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "balancer", 2024-04-23T14:58:26.539 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:26.540 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:26.540 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:26.540 INFO:teuthology.orchestra.run.smithi028.stdout: "active": { 2024-04-23T14:58:26.540 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "active", 2024-04-23T14:58:26.540 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.540 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.540 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.540 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:26.540 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.540 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.541 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.541 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically balance PGs across cluster", 2024-04-23T14:58:26.541 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.541 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.541 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.541 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_time": { 2024-04-23T14:58:26.541 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_time", 2024-04-23T14:58:26.542 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.542 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.542 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.542 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0000", 2024-04-23T14:58:26.542 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.542 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.542 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.542 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "beginning time of day to automatically balance", 2024-04-23T14:58:26.542 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-23T14:58:26.543 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.543 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.543 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.543 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_weekday": { 2024-04-23T14:58:26.543 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_weekday", 2024-04-23T14:58:26.543 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-23T14:58:26.543 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.543 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.543 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-23T14:58:26.544 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-04-23T14:58:26.544 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-04-23T14:58:26.544 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.544 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-23T14:58:26.544 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-23T14:58:26.544 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.544 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.544 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.544 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_max_iterations": { 2024-04-23T14:58:26.544 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_max_iterations", 2024-04-23T14:58:26.545 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-23T14:58:26.545 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.545 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.545 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "25", 2024-04-23T14:58:26.545 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-23T14:58:26.545 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "250", 2024-04-23T14:58:26.545 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.545 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-23T14:58:26.545 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.546 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.546 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.546 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_metrics": { 2024-04-23T14:58:26.546 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_metrics", 2024-04-23T14:58:26.546 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.546 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.546 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.546 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "pgs,objects,bytes", 2024-04-23T14:58:26.547 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.547 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.547 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.547 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-23T14:58:26.547 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-23T14:58:26.547 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.547 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.547 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.547 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_step": { 2024-04-23T14:58:26.548 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_step", 2024-04-23T14:58:26.548 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-23T14:58:26.548 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.548 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.548 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.5", 2024-04-23T14:58:26.548 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0.001", 2024-04-23T14:58:26.548 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "0.999", 2024-04-23T14:58:26.548 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.548 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "aggressiveness of optimization", 2024-04-23T14:58:26.548 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-23T14:58:26.549 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.549 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.549 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.549 INFO:teuthology.orchestra.run.smithi028.stdout: "end_time": { 2024-04-23T14:58:26.549 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_time", 2024-04-23T14:58:26.549 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.550 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.550 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.550 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2359", 2024-04-23T14:58:26.550 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.550 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.550 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.550 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ending time of day to automatically balance", 2024-04-23T14:58:26.550 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-23T14:58:26.550 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.551 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.551 INFO:teuthology.orchestra.run.smithi028.stdout: "end_weekday": { 2024-04-23T14:58:26.551 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_weekday", 2024-04-23T14:58:26.551 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-23T14:58:26.551 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.551 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.551 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-23T14:58:26.551 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-04-23T14:58:26.552 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-04-23T14:58:26.552 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.552 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-23T14:58:26.552 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-23T14:58:26.552 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.552 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.552 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:26.553 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:26.553 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.553 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.553 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.553 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.553 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.553 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.553 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.553 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.554 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.554 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.554 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.554 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.554 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.554 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.554 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.555 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.555 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.555 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.555 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:26.555 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:26.555 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.555 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.555 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.555 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.556 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.556 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.556 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.556 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.556 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.556 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.556 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.556 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.556 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:26.557 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:26.557 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.557 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.557 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.557 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:26.557 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.557 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.557 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.557 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.557 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.558 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.558 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.558 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.558 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.558 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.558 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.558 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.558 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.559 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.559 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.559 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:26.559 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:26.559 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.559 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.559 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.559 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.559 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.559 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.560 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.560 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.560 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.560 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.560 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.560 INFO:teuthology.orchestra.run.smithi028.stdout: "min_score": { 2024-04-23T14:58:26.560 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_score", 2024-04-23T14:58:26.561 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-23T14:58:26.561 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.561 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.561 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-23T14:58:26.561 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.561 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.561 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.562 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-23T14:58:26.562 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.562 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.562 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.562 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.562 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-04-23T14:58:26.562 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-04-23T14:58:26.562 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.562 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.563 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.563 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "upmap", 2024-04-23T14:58:26.563 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.563 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.563 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.563 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-compat", 2024-04-23T14:58:26.563 INFO:teuthology.orchestra.run.smithi028.stdout: "none", 2024-04-23T14:58:26.563 INFO:teuthology.orchestra.run.smithi028.stdout: "read", 2024-04-23T14:58:26.564 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap", 2024-04-23T14:58:26.564 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap-read" 2024-04-23T14:58:26.564 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.564 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Balancer mode", 2024-04-23T14:58:26.564 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.564 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.564 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.564 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.564 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_ids": { 2024-04-23T14:58:26.565 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_ids", 2024-04-23T14:58:26.565 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.565 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.565 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.565 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.565 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.565 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.565 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.565 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-23T14:58:26.566 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.566 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.566 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.566 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.566 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-23T14:58:26.566 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-23T14:58:26.566 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:26.566 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.566 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.567 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-23T14:58:26.567 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.567 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.567 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.567 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-23T14:58:26.567 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.567 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.567 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.567 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.568 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_deviation": { 2024-04-23T14:58:26.568 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_deviation", 2024-04-23T14:58:26.568 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:26.568 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.568 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.568 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-23T14:58:26.568 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-23T14:58:26.568 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.568 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.569 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-23T14:58:26.569 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-23T14:58:26.569 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.569 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.569 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.569 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_optimizations": { 2024-04-23T14:58:26.569 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_optimizations", 2024-04-23T14:58:26.569 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-23T14:58:26.569 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.570 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.570 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-23T14:58:26.570 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.570 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.570 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.570 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-23T14:58:26.570 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.570 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.571 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.571 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.571 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.571 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.571 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:26.571 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm", 2024-04-23T14:58:26.571 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:26.571 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:26.571 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:26.572 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_down_multiplier": { 2024-04-23T14:58:26.572 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_down_multiplier", 2024-04-23T14:58:26.572 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-23T14:58:26.572 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.572 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.572 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-04-23T14:58:26.572 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.573 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.573 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.573 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-23T14:58:26.573 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.573 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.573 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.573 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_refresh_rate": { 2024-04-23T14:58:26.574 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_refresh_rate", 2024-04-23T14:58:26.574 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:26.574 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.574 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.574 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "20", 2024-04-23T14:58:26.574 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.574 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.574 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.574 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-23T14:58:26.575 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.575 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.575 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.575 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.575 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_starting_port": { 2024-04-23T14:58:26.575 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_starting_port", 2024-04-23T14:58:26.575 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:26.575 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.575 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.576 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4721", 2024-04-23T14:58:26.576 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.576 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.576 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.576 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-23T14:58:26.576 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.576 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.576 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.577 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.577 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_ptrace": { 2024-04-23T14:58:26.577 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_ptrace", 2024-04-23T14:58:26.577 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.577 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.577 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.577 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.577 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.578 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.578 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.578 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-23T14:58:26.578 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-23T14:58:26.578 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.578 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.578 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.578 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_interval": { 2024-04-23T14:58:26.579 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_interval", 2024-04-23T14:58:26.579 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:26.579 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.579 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.579 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-23T14:58:26.579 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.579 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.579 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.580 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-23T14:58:26.580 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.580 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.580 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.580 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_memory_target_ratio": { 2024-04-23T14:58:26.580 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_memory_target_ratio", 2024-04-23T14:58:26.581 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-23T14:58:26.581 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.581 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.581 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.7", 2024-04-23T14:58:26.581 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.581 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.581 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.582 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-23T14:58:26.582 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.582 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.582 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.582 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.582 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_log_destination": { 2024-04-23T14:58:26.582 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm_log_destination", 2024-04-23T14:58:26.582 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.583 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.583 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.583 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.583 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.583 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.583 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.583 INFO:teuthology.orchestra.run.smithi028.stdout: "file", 2024-04-23T14:58:26.583 INFO:teuthology.orchestra.run.smithi028.stdout: "file,syslog", 2024-04-23T14:58:26.584 INFO:teuthology.orchestra.run.smithi028.stdout: "syslog" 2024-04-23T14:58:26.584 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.584 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-23T14:58:26.584 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.584 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.584 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.584 INFO:teuthology.orchestra.run.smithi028.stdout: "cgroups_split": { 2024-04-23T14:58:26.585 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cgroups_split", 2024-04-23T14:58:26.585 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.585 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.585 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.585 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:26.585 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.585 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.585 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.586 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-23T14:58:26.586 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.586 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.586 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.586 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.586 INFO:teuthology.orchestra.run.smithi028.stdout: "config_checks_enabled": { 2024-04-23T14:58:26.586 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_checks_enabled", 2024-04-23T14:58:26.586 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.587 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.587 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.587 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.587 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.587 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.587 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.587 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-23T14:58:26.588 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.588 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.588 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.588 INFO:teuthology.orchestra.run.smithi028.stdout: "config_dashboard": { 2024-04-23T14:58:26.588 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_dashboard", 2024-04-23T14:58:26.588 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.589 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.589 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.589 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:26.589 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.589 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.589 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.589 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-23T14:58:26.589 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.589 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.590 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.590 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_alertmanager": { 2024-04-23T14:58:26.590 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_alertmanager", 2024-04-23T14:58:26.590 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.590 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.590 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.590 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-23T14:58:26.591 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.591 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.591 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.591 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-23T14:58:26.591 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.591 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.591 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.592 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_base": { 2024-04-23T14:58:26.592 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_base", 2024-04-23T14:58:26.592 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.592 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.592 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.592 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-23T14:58:26.592 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.592 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.592 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.593 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Container image name, without the tag", 2024-04-23T14:58:26.593 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.593 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.593 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.593 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_elasticsearch": { 2024-04-23T14:58:26.593 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_elasticsearch", 2024-04-23T14:58:26.593 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.593 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.593 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.594 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-23T14:58:26.594 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.594 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.594 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.594 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "elasticsearch container image", 2024-04-23T14:58:26.594 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.594 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.594 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.595 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_grafana": { 2024-04-23T14:58:26.595 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_grafana", 2024-04-23T14:58:26.595 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.595 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.595 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.595 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-23T14:58:26.595 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.595 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.596 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.596 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-23T14:58:26.596 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.596 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.596 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.596 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_haproxy": { 2024-04-23T14:58:26.596 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_haproxy", 2024-04-23T14:58:26.597 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.597 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.597 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.597 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-23T14:58:26.597 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.597 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.597 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.598 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "HAproxy container image", 2024-04-23T14:58:26.598 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.598 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.598 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.598 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_agent": { 2024-04-23T14:58:26.598 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_agent", 2024-04-23T14:58:26.599 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.599 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.599 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.599 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-23T14:58:26.599 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.599 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.599 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.599 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger agent container image", 2024-04-23T14:58:26.600 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.600 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.600 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.600 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_collector": { 2024-04-23T14:58:26.600 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_collector", 2024-04-23T14:58:26.600 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.600 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.601 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.601 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-23T14:58:26.601 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.601 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.601 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.601 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger collector container image", 2024-04-23T14:58:26.601 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.601 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.601 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.602 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_query": { 2024-04-23T14:58:26.602 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_query", 2024-04-23T14:58:26.602 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.602 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.602 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.602 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-23T14:58:26.603 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.603 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.603 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.603 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger query container image", 2024-04-23T14:58:26.603 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.603 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.603 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.603 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_keepalived": { 2024-04-23T14:58:26.603 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_keepalived", 2024-04-23T14:58:26.604 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.604 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.604 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.604 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-23T14:58:26.604 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.604 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.604 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.604 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Keepalived container image", 2024-04-23T14:58:26.604 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.605 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.605 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.605 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.605 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_loki": { 2024-04-23T14:58:26.605 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_loki", 2024-04-23T14:58:26.605 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.605 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.605 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.605 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-23T14:58:26.605 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.606 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.606 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.606 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Loki container image", 2024-04-23T14:58:26.606 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.606 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.606 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.606 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.606 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_node_exporter": { 2024-04-23T14:58:26.607 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_node_exporter", 2024-04-23T14:58:26.607 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.607 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.607 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.607 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-23T14:58:26.607 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.607 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.607 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.607 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-23T14:58:26.608 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.608 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.608 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.608 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nvmeof": { 2024-04-23T14:58:26.608 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nvmeof", 2024-04-23T14:58:26.608 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.608 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.608 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.608 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-23T14:58:26.609 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.609 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.609 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.609 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nvme-of container image", 2024-04-23T14:58:26.609 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.609 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.609 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.609 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_prometheus": { 2024-04-23T14:58:26.609 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_prometheus", 2024-04-23T14:58:26.610 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.610 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.610 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.610 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-23T14:58:26.610 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.610 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.610 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.610 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-23T14:58:26.611 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.611 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.611 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.611 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_promtail": { 2024-04-23T14:58:26.611 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_promtail", 2024-04-23T14:58:26.611 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.611 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.611 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.612 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-23T14:58:26.612 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.612 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.612 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.612 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Promtail container image", 2024-04-23T14:58:26.612 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.612 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.612 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.613 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_samba": { 2024-04-23T14:58:26.613 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_samba", 2024-04-23T14:58:26.613 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.613 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.613 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.613 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-23T14:58:26.613 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.613 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.613 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.613 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Samba/SMB container image", 2024-04-23T14:58:26.614 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.614 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.614 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.614 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_snmp_gateway": { 2024-04-23T14:58:26.614 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_snmp_gateway", 2024-04-23T14:58:26.614 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.614 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.615 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.615 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-23T14:58:26.615 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.615 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.615 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.615 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SNMP Gateway container image", 2024-04-23T14:58:26.615 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.615 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.615 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.616 INFO:teuthology.orchestra.run.smithi028.stdout: "container_init": { 2024-04-23T14:58:26.616 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_init", 2024-04-23T14:58:26.616 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.616 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.616 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.616 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:26.616 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.616 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.616 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.617 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Run podman/docker with `--init`", 2024-04-23T14:58:26.617 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.617 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.617 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.617 INFO:teuthology.orchestra.run.smithi028.stdout: "daemon_cache_timeout": { 2024-04-23T14:58:26.617 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "daemon_cache_timeout", 2024-04-23T14:58:26.617 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:26.617 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.618 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.618 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-23T14:58:26.618 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.618 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.618 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.618 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-23T14:58:26.618 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.618 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.619 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.619 INFO:teuthology.orchestra.run.smithi028.stdout: "default_cephadm_command_timeout": { 2024-04-23T14:58:26.619 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_cephadm_command_timeout", 2024-04-23T14:58:26.619 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:26.619 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.619 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.619 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "900", 2024-04-23T14:58:26.619 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.620 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.620 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.620 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-23T14:58:26.620 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.620 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.620 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.620 INFO:teuthology.orchestra.run.smithi028.stdout: "default_registry": { 2024-04-23T14:58:26.620 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_registry", 2024-04-23T14:58:26.621 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.621 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.621 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.621 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io", 2024-04-23T14:58:26.621 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.621 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.621 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.621 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-23T14:58:26.621 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.621 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.622 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.622 INFO:teuthology.orchestra.run.smithi028.stdout: "device_cache_timeout": { 2024-04-23T14:58:26.622 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_cache_timeout", 2024-04-23T14:58:26.622 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:26.622 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.622 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.622 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1800", 2024-04-23T14:58:26.623 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.623 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.623 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.623 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache device inventory", 2024-04-23T14:58:26.623 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.623 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.623 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.623 INFO:teuthology.orchestra.run.smithi028.stdout: "device_enhanced_scan": { 2024-04-23T14:58:26.624 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_enhanced_scan", 2024-04-23T14:58:26.624 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.624 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.624 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.624 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.624 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.624 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.624 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.625 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-23T14:58:26.625 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.625 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.625 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.625 INFO:teuthology.orchestra.run.smithi028.stdout: "facts_cache_timeout": { 2024-04-23T14:58:26.625 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "facts_cache_timeout", 2024-04-23T14:58:26.625 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:26.625 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.626 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.626 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-23T14:58:26.626 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.626 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.626 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.626 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache host facts data", 2024-04-23T14:58:26.626 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.626 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.627 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.627 INFO:teuthology.orchestra.run.smithi028.stdout: "grafana_dashboards_path": { 2024-04-23T14:58:26.627 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "grafana_dashboards_path", 2024-04-23T14:58:26.627 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.627 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.627 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.627 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-23T14:58:26.627 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.628 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.628 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.628 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-23T14:58:26.628 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.628 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.628 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.628 INFO:teuthology.orchestra.run.smithi028.stdout: "host_check_interval": { 2024-04-23T14:58:26.628 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "host_check_interval", 2024-04-23T14:58:26.629 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:26.629 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.629 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.629 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-23T14:58:26.629 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.629 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.629 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.629 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to perform a host check", 2024-04-23T14:58:26.629 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.629 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.630 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.630 INFO:teuthology.orchestra.run.smithi028.stdout: "hw_monitoring": { 2024-04-23T14:58:26.630 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hw_monitoring", 2024-04-23T14:58:26.630 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.630 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.630 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.630 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.631 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.631 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.631 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.631 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-23T14:58:26.631 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.631 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.631 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.631 INFO:teuthology.orchestra.run.smithi028.stdout: "inventory_list_all": { 2024-04-23T14:58:26.631 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "inventory_list_all", 2024-04-23T14:58:26.632 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.632 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.632 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.632 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.632 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.632 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.632 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.632 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-23T14:58:26.632 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.633 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.633 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.633 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:26.633 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:26.633 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.633 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.633 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.634 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.634 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.634 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.634 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.634 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.634 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.634 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.634 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.634 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.635 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.635 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.635 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.635 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.635 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.635 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.635 INFO:teuthology.orchestra.run.smithi028.stdout: "log_refresh_metadata": { 2024-04-23T14:58:26.636 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_refresh_metadata", 2024-04-23T14:58:26.636 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.636 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.636 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.636 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.636 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.636 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.636 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.637 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-23T14:58:26.637 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.637 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.637 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.637 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:26.637 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:26.637 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.637 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.638 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.638 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:26.638 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.638 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.638 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.638 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-23T14:58:26.638 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.638 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.639 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.639 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:26.639 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:26.639 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.639 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.639 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.639 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:26.639 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.640 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.640 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.640 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.640 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.640 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.640 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.640 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.640 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.640 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.641 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.641 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.641 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.641 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.641 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:26.641 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:26.641 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.641 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.642 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.642 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.642 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.642 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.642 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.642 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.642 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.643 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.643 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.643 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-23T14:58:26.643 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-23T14:58:26.643 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.643 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.643 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.643 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.644 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.644 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.644 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.644 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-23T14:58:26.644 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.644 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.644 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.644 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-23T14:58:26.645 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-23T14:58:26.645 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.645 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.645 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.645 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "*", 2024-04-23T14:58:26.645 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.645 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.645 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.645 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-23T14:58:26.646 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.646 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.646 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.646 INFO:teuthology.orchestra.run.smithi028.stdout: "max_count_per_host": { 2024-04-23T14:58:26.646 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_count_per_host", 2024-04-23T14:58:26.646 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:26.646 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.647 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.647 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-23T14:58:26.647 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.647 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.647 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.647 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of daemons per service per host", 2024-04-23T14:58:26.647 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.647 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.648 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.648 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd_draining_count": { 2024-04-23T14:58:26.648 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_osd_draining_count", 2024-04-23T14:58:26.648 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:26.648 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.648 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.648 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-23T14:58:26.648 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.649 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.649 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.649 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-23T14:58:26.649 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.649 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.649 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.649 INFO:teuthology.orchestra.run.smithi028.stdout: "migration_current": { 2024-04-23T14:58:26.650 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "migration_current", 2024-04-23T14:58:26.650 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:26.650 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.650 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.650 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.650 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.650 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.650 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.650 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "internal - do not modify", 2024-04-23T14:58:26.651 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.651 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.651 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.651 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-04-23T14:58:26.651 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-04-23T14:58:26.651 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.651 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.651 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.652 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "root", 2024-04-23T14:58:26.652 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.652 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.652 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.652 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm-package", 2024-04-23T14:58:26.652 INFO:teuthology.orchestra.run.smithi028.stdout: "root" 2024-04-23T14:58:26.652 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.653 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "mode for remote execution of cephadm", 2024-04-23T14:58:26.653 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.653 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.653 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.653 INFO:teuthology.orchestra.run.smithi028.stdout: "oob_default_addr": { 2024-04-23T14:58:26.653 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "oob_default_addr", 2024-04-23T14:58:26.653 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.653 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.654 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.654 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "169.254.1.1", 2024-04-23T14:58:26.654 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.654 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.654 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.654 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-23T14:58:26.654 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.654 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.654 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.655 INFO:teuthology.orchestra.run.smithi028.stdout: "prometheus_alerts_path": { 2024-04-23T14:58:26.655 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus_alerts_path", 2024-04-23T14:58:26.655 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.655 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.655 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.655 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-23T14:58:26.655 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.655 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.655 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.656 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-23T14:58:26.656 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.656 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.656 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.656 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_insecure": { 2024-04-23T14:58:26.656 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_insecure", 2024-04-23T14:58:26.656 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.656 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.657 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.657 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.657 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.657 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.657 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.657 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-23T14:58:26.657 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.657 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.657 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.658 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_password": { 2024-04-23T14:58:26.658 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_password", 2024-04-23T14:58:26.658 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.658 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.658 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.658 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.658 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.659 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.659 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.659 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-23T14:58:26.659 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.659 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.659 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.660 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_url": { 2024-04-23T14:58:26.660 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_url", 2024-04-23T14:58:26.660 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.660 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.660 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.660 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.660 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.660 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.661 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.661 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-23T14:58:26.661 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.661 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.661 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.661 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_username": { 2024-04-23T14:58:26.662 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_username", 2024-04-23T14:58:26.662 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.662 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.662 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.662 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.662 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.662 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.662 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.663 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-23T14:58:26.663 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.663 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.663 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.663 INFO:teuthology.orchestra.run.smithi028.stdout: "secure_monitoring_stack": { 2024-04-23T14:58:26.663 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secure_monitoring_stack", 2024-04-23T14:58:26.663 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.663 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.664 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.664 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.664 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.664 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.664 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.664 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-23T14:58:26.664 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.665 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.665 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.665 INFO:teuthology.orchestra.run.smithi028.stdout: "service_discovery_port": { 2024-04-23T14:58:26.665 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "service_discovery_port", 2024-04-23T14:58:26.665 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:26.665 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.665 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.666 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8765", 2024-04-23T14:58:26.666 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.666 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.666 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.666 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "cephadm service discovery port", 2024-04-23T14:58:26.666 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.666 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.667 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.667 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_config_file": { 2024-04-23T14:58:26.667 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_config_file", 2024-04-23T14:58:26.667 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.667 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.667 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.667 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.667 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.668 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.668 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.668 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-23T14:58:26.668 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.668 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.668 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.669 INFO:teuthology.orchestra.run.smithi028.stdout: "use_agent": { 2024-04-23T14:58:26.669 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_agent", 2024-04-23T14:58:26.669 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.669 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.669 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.669 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.669 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.669 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.670 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.670 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-23T14:58:26.670 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.670 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.670 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.670 INFO:teuthology.orchestra.run.smithi028.stdout: "use_repo_digest": { 2024-04-23T14:58:26.670 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_repo_digest", 2024-04-23T14:58:26.671 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.671 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.671 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.671 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:26.671 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.671 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.671 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.671 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-23T14:58:26.672 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.672 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.672 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.672 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_failed_host_check": { 2024-04-23T14:58:26.672 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_failed_host_check", 2024-04-23T14:58:26.672 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.672 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.673 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.673 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:26.673 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.673 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.673 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.673 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if the host check fails", 2024-04-23T14:58:26.673 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.673 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.674 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.674 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_daemons": { 2024-04-23T14:58:26.674 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_daemons", 2024-04-23T14:58:26.674 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.674 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.674 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.675 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:26.675 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.675 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.675 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.675 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-23T14:58:26.675 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.676 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.676 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.676 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_hosts": { 2024-04-23T14:58:26.676 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_hosts", 2024-04-23T14:58:26.676 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.676 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.676 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.677 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:26.677 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.677 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.677 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.677 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-23T14:58:26.677 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.677 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.678 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.678 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.678 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:26.678 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crash", 2024-04-23T14:58:26.678 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:26.678 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:26.679 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:26.679 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:26.679 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:26.679 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.679 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.679 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.679 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.679 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.680 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.680 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.680 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.680 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.680 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.680 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.680 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.680 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.681 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.681 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.681 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.681 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.681 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.681 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:26.681 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:26.682 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.682 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.682 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.682 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.682 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.682 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.682 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.683 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.683 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.683 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.683 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.683 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:26.683 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:26.683 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.684 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.684 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.684 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:26.684 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.684 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.684 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.684 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.684 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.684 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.685 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.685 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.685 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.685 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.685 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.685 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.685 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.685 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.686 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:26.686 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:26.686 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.686 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.686 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.686 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.686 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.687 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.687 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.687 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.687 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.687 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.687 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.687 INFO:teuthology.orchestra.run.smithi028.stdout: "retain_interval": { 2024-04-23T14:58:26.688 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retain_interval", 2024-04-23T14:58:26.688 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:26.688 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.688 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.688 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "31536000", 2024-04-23T14:58:26.688 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.688 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.688 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.688 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-23T14:58:26.689 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.689 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.689 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.689 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_recent_interval": { 2024-04-23T14:58:26.689 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_recent_interval", 2024-04-23T14:58:26.689 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:26.689 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.689 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.690 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1209600", 2024-04-23T14:58:26.690 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.690 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.690 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.690 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-23T14:58:26.690 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.690 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.690 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.691 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.691 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:26.691 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dashboard", 2024-04-23T14:58:26.691 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:26.691 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:26.691 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:26.691 INFO:teuthology.orchestra.run.smithi028.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-23T14:58:26.692 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-23T14:58:26.692 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:26.692 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.692 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.692 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-23T14:58:26.692 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.692 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.692 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.692 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.693 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.693 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.693 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.693 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_HOST": { 2024-04-23T14:58:26.693 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-23T14:58:26.693 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.693 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.693 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.694 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.694 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.694 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.694 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.694 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.694 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.694 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.694 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.695 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-23T14:58:26.695 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-23T14:58:26.695 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.695 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.695 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.695 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:26.695 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.695 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.696 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.696 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.696 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.696 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.696 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.696 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_ENABLED": { 2024-04-23T14:58:26.697 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_ENABLED", 2024-04-23T14:58:26.697 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.697 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.697 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.697 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.697 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.697 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.697 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.697 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.698 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.698 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.698 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.698 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-23T14:58:26.698 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-23T14:58:26.698 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.698 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.698 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.699 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:26.699 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.699 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.699 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.699 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.699 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.701 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.701 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.701 INFO:teuthology.orchestra.run.smithi028.stdout: "ENABLE_BROWSABLE_API": { 2024-04-23T14:58:26.701 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-23T14:58:26.701 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.702 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.702 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.702 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:26.702 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.702 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.702 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.702 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.702 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.703 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.703 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.703 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-23T14:58:26.703 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-23T14:58:26.703 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.703 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.703 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.703 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:26.704 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.704 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.704 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.704 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.704 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.704 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.704 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.704 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-23T14:58:26.705 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-23T14:58:26.705 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.705 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.705 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.705 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:26.705 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.705 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.705 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.705 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.705 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.706 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.706 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.706 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-23T14:58:26.706 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-23T14:58:26.706 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.706 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.706 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.706 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:26.707 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.707 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.707 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.707 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.707 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.707 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.707 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.707 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-23T14:58:26.707 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-23T14:58:26.708 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.708 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.708 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.708 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:26.708 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.708 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.708 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.708 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.708 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.709 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.709 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.709 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-23T14:58:26.709 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-23T14:58:26.709 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.709 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.709 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.709 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:26.710 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.710 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.710 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.710 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.710 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.710 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.710 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.710 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-23T14:58:26.711 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-23T14:58:26.711 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.711 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.711 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.711 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:26.711 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.711 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.711 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.711 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.712 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.712 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.712 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.712 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-23T14:58:26.712 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-23T14:58:26.712 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.712 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.712 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.712 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:26.713 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.713 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.713 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.713 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.713 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.713 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.713 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.713 INFO:teuthology.orchestra.run.smithi028.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-23T14:58:26.713 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-23T14:58:26.714 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.714 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.714 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.714 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.714 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.714 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.714 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.714 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.715 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.715 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.715 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.715 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_PASSWORD": { 2024-04-23T14:58:26.715 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-23T14:58:26.715 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.715 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.715 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.716 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-23T14:58:26.716 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.716 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.716 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.716 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.716 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.716 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.716 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.717 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-23T14:58:26.717 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-23T14:58:26.717 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.717 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.717 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.717 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:26.717 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.717 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.717 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.718 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.718 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.718 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.718 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.718 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_URL": { 2024-04-23T14:58:26.718 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_URL", 2024-04-23T14:58:26.718 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.718 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.718 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.719 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.719 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.719 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.719 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.719 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.719 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.719 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.719 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.720 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_USERNAME": { 2024-04-23T14:58:26.720 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-23T14:58:26.720 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.720 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.720 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.720 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-23T14:58:26.720 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.720 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.720 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.721 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.721 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.721 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.721 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.721 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-23T14:58:26.721 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-23T14:58:26.721 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.721 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.721 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.722 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.722 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.722 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.722 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.722 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.722 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.722 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.722 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.723 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-23T14:58:26.723 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-23T14:58:26.723 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.723 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.723 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.723 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.723 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.723 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.724 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.724 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.724 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.724 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.724 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.724 INFO:teuthology.orchestra.run.smithi028.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-23T14:58:26.724 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-23T14:58:26.725 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.725 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.725 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.725 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:26.725 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.725 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.725 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.725 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.725 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.726 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.726 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.726 INFO:teuthology.orchestra.run.smithi028.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-23T14:58:26.726 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-23T14:58:26.726 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.726 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.726 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.726 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.727 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.727 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.727 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.727 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.727 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.727 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.727 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.727 INFO:teuthology.orchestra.run.smithi028.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-23T14:58:26.728 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-23T14:58:26.728 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.728 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.728 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.728 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "[]", 2024-04-23T14:58:26.728 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.728 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.728 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.729 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.729 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.729 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.729 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.729 INFO:teuthology.orchestra.run.smithi028.stdout: "MULTICLUSTER_CONFIG": { 2024-04-23T14:58:26.729 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-23T14:58:26.729 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.729 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.730 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.730 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "{}", 2024-04-23T14:58:26.730 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.730 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.730 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.730 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.730 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.730 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.730 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.731 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_HOST": { 2024-04-23T14:58:26.731 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-23T14:58:26.731 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.731 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.731 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.731 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.731 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.732 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.732 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.732 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.732 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.732 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.732 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.732 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-23T14:58:26.732 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-23T14:58:26.733 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.733 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.733 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.733 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:26.733 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.733 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.733 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.733 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.733 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.734 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.734 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.734 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-23T14:58:26.734 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-23T14:58:26.734 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.734 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.734 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.735 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.735 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.735 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.735 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.735 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.735 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.735 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.735 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.736 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-23T14:58:26.736 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-23T14:58:26.736 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.736 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.736 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.736 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.736 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.736 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.736 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.737 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.737 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.737 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.737 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.737 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-23T14:58:26.737 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-23T14:58:26.737 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.737 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.737 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.738 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:26.738 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.738 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.738 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.738 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.738 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.738 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.738 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.739 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-23T14:58:26.739 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-23T14:58:26.739 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.739 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.739 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.739 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:26.739 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.739 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.739 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.740 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.740 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.740 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.740 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.740 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-23T14:58:26.740 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-23T14:58:26.740 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.740 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.740 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.741 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.741 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.741 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.741 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.741 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.741 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.741 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.741 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.741 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-23T14:58:26.742 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-23T14:58:26.742 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.742 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.742 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.742 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.742 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.742 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.742 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.743 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.743 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.743 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.743 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.743 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-23T14:58:26.743 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-23T14:58:26.743 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.743 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.744 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.744 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.744 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.744 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.744 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.744 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.744 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.744 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.744 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.745 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_ENABLED": { 2024-04-23T14:58:26.745 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-23T14:58:26.745 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.745 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.745 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.745 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:26.745 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.745 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.745 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.746 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.746 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.746 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.746 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.746 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-23T14:58:26.746 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-23T14:58:26.746 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.746 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.747 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.747 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-23T14:58:26.747 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.747 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.747 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.747 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.747 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.747 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.747 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.748 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-23T14:58:26.748 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-23T14:58:26.748 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:26.748 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.748 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.748 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-23T14:58:26.748 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.749 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.749 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.749 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.749 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.749 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.749 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.749 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-23T14:58:26.749 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-23T14:58:26.750 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:26.750 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.750 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.750 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8", 2024-04-23T14:58:26.750 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.750 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.750 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.750 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.750 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.751 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.751 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.751 INFO:teuthology.orchestra.run.smithi028.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-23T14:58:26.751 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-23T14:58:26.751 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:26.751 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.751 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.751 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "45", 2024-04-23T14:58:26.752 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.752 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.752 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.752 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.752 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.752 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.752 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.752 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ACCESS_KEY": { 2024-04-23T14:58:26.753 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-23T14:58:26.753 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.753 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.753 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.753 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.753 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.753 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.753 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.753 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.754 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.754 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.754 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.754 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.754 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-23T14:58:26.754 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-23T14:58:26.754 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.754 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.754 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.755 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-23T14:58:26.755 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.755 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.755 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.755 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.755 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.755 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.755 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.755 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SECRET_KEY": { 2024-04-23T14:58:26.756 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-23T14:58:26.756 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.756 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.756 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.756 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.756 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.756 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.756 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.756 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.756 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.757 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.757 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.757 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SSL_VERIFY": { 2024-04-23T14:58:26.757 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-23T14:58:26.757 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.757 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.757 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.757 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:26.758 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.758 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.758 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.758 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.758 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.758 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.758 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.758 INFO:teuthology.orchestra.run.smithi028.stdout: "UNSAFE_TLS_v1_2": { 2024-04-23T14:58:26.759 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-23T14:58:26.759 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.759 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.759 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.759 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.759 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.759 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.759 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.759 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.759 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-23T14:58:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-23T14:58:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-23T14:58:26.761 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.761 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.761 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.761 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.761 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.761 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.761 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.761 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-23T14:58:26.761 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-23T14:58:26.762 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:26.762 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.762 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.762 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-23T14:58:26.762 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.762 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.762 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.762 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.763 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.763 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.763 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.763 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-23T14:58:26.763 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-23T14:58:26.763 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:26.763 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.763 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.763 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-23T14:58:26.764 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.764 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.764 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.764 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.764 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.764 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.764 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.764 INFO:teuthology.orchestra.run.smithi028.stdout: "cross_origin_url": { 2024-04-23T14:58:26.765 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cross_origin_url", 2024-04-23T14:58:26.765 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.765 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.765 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.765 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.765 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.765 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.765 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.765 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.765 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.766 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.766 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.766 INFO:teuthology.orchestra.run.smithi028.stdout: "crt_file": { 2024-04-23T14:58:26.766 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crt_file", 2024-04-23T14:58:26.766 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.766 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.766 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.767 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.767 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.767 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.767 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.767 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.767 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.767 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.767 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.768 INFO:teuthology.orchestra.run.smithi028.stdout: "debug": { 2024-04-23T14:58:26.768 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "debug", 2024-04-23T14:58:26.768 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.768 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.768 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.768 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.768 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.768 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.768 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.769 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable/disable debug options", 2024-04-23T14:58:26.769 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.769 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.769 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.769 INFO:teuthology.orchestra.run.smithi028.stdout: "jwt_token_ttl": { 2024-04-23T14:58:26.769 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "jwt_token_ttl", 2024-04-23T14:58:26.769 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:26.769 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.770 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.770 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "28800", 2024-04-23T14:58:26.770 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.770 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.770 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.770 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.770 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.770 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.770 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.771 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-04-23T14:58:26.771 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-04-23T14:58:26.771 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.771 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.771 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.771 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.771 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.771 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.772 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.772 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.772 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.772 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.772 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.772 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:26.772 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:26.772 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.772 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.773 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.773 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.773 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.773 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.773 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.773 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.773 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.773 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.773 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.774 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.774 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.774 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.774 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.774 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.774 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.774 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.774 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.774 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:26.775 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:26.775 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.775 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.775 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.775 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.775 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.775 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.775 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.775 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.776 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.776 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.776 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.776 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.776 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:26.776 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:26.776 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.776 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.776 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.776 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:26.777 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.777 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.777 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.777 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.777 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.777 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.777 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.777 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.777 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.777 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.778 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.778 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.778 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.778 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.778 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:26.778 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:26.778 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.778 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.779 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.779 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.779 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.779 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.779 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.779 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.779 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.779 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.779 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.780 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.780 INFO:teuthology.orchestra.run.smithi028.stdout: "motd": { 2024-04-23T14:58:26.780 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "motd", 2024-04-23T14:58:26.780 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.780 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.780 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.780 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.780 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.780 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.781 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.781 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "The message of the day", 2024-04-23T14:58:26.781 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.781 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.781 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.781 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect_resolve_ip_addr": { 2024-04-23T14:58:26.781 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "redirect_resolve_ip_addr", 2024-04-23T14:58:26.781 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.782 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.782 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.782 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.782 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.782 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.782 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.782 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.782 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.782 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.782 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.783 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-23T14:58:26.783 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-23T14:58:26.783 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.783 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.783 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.783 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-04-23T14:58:26.783 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.783 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.784 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.784 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.784 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.784 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.784 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.784 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-23T14:58:26.784 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-23T14:58:26.784 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:26.784 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.785 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.785 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8080", 2024-04-23T14:58:26.785 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.785 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.785 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.785 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.785 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.785 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.785 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.786 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-04-23T14:58:26.786 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-04-23T14:58:26.786 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.786 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.786 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.786 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:26.786 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.786 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.787 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.787 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.787 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.787 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.787 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.787 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl_server_port": { 2024-04-23T14:58:26.787 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl_server_port", 2024-04-23T14:58:26.787 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:26.787 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.788 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.788 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8443", 2024-04-23T14:58:26.788 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.788 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.788 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.788 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.788 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.788 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.788 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.788 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.789 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-04-23T14:58:26.789 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-04-23T14:58:26.789 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.789 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.789 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.789 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "redirect", 2024-04-23T14:58:26.789 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.789 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.789 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.789 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.790 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect" 2024-04-23T14:58:26.790 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.790 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.790 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.790 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.790 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.790 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-04-23T14:58:26.791 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-04-23T14:58:26.791 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:26.791 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.791 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.791 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-04-23T14:58:26.791 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-04-23T14:58:26.791 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-04-23T14:58:26.791 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.791 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.792 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.792 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.792 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.792 INFO:teuthology.orchestra.run.smithi028.stdout: "url_prefix": { 2024-04-23T14:58:26.792 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url_prefix", 2024-04-23T14:58:26.792 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.792 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.792 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.792 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.793 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.793 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.793 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.793 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.793 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.793 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.793 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.793 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.793 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.794 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:26.794 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-04-23T14:58:26.794 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:26.794 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:26.794 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:26.794 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_monitoring": { 2024-04-23T14:58:26.794 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_monitoring", 2024-04-23T14:58:26.794 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.794 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.795 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.795 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:26.795 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.795 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.795 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.795 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "monitor device health metrics", 2024-04-23T14:58:26.795 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.795 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.795 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.796 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:26.796 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:26.796 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.796 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.796 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.796 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.796 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.796 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.796 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.797 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.797 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.797 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.797 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.797 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.797 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.797 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.797 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.797 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.798 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.798 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.798 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:26.798 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:26.798 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.798 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.798 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.799 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.799 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.799 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.799 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.799 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.799 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.799 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.799 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.799 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:26.800 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:26.800 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.800 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.800 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.800 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:26.800 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.800 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.800 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.800 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.801 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.801 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.801 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.801 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.801 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.801 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.801 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.801 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.801 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.801 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.802 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:26.802 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:26.802 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.802 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.802 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.802 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.802 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.802 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.803 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.803 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.803 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.803 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.803 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.803 INFO:teuthology.orchestra.run.smithi028.stdout: "mark_out_threshold": { 2024-04-23T14:58:26.803 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mark_out_threshold", 2024-04-23T14:58:26.803 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:26.803 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.804 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.804 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2419200", 2024-04-23T14:58:26.804 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.804 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.804 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.804 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-23T14:58:26.804 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.804 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.804 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.805 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": { 2024-04-23T14:58:26.805 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_name", 2024-04-23T14:58:26.805 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.805 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.805 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.805 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "device_health_metrics", 2024-04-23T14:58:26.805 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.805 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.806 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.806 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-23T14:58:26.806 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.806 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.806 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.806 INFO:teuthology.orchestra.run.smithi028.stdout: "retention_period": { 2024-04-23T14:58:26.806 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retention_period", 2024-04-23T14:58:26.806 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:26.807 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.807 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.807 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15552000", 2024-04-23T14:58:26.807 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.807 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.807 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.807 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain device health metrics", 2024-04-23T14:58:26.807 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.807 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.808 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.808 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_frequency": { 2024-04-23T14:58:26.808 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_frequency", 2024-04-23T14:58:26.808 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:26.808 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.808 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.808 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-04-23T14:58:26.809 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.809 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.809 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.809 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-23T14:58:26.809 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.809 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.809 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.809 INFO:teuthology.orchestra.run.smithi028.stdout: "self_heal": { 2024-04-23T14:58:26.810 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "self_heal", 2024-04-23T14:58:26.810 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.810 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.810 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.810 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:26.810 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.810 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.810 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.811 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-23T14:58:26.811 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.811 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.811 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.811 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-23T14:58:26.811 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-23T14:58:26.811 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:26.811 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.812 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.812 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-23T14:58:26.812 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.812 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.812 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.812 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and check device health", 2024-04-23T14:58:26.812 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.812 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.813 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.813 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_threshold": { 2024-04-23T14:58:26.813 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_threshold", 2024-04-23T14:58:26.813 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:26.813 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.813 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.813 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7257600", 2024-04-23T14:58:26.813 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.814 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.814 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.814 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-23T14:58:26.814 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.814 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.814 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.814 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.814 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:26.815 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "diskprediction_local", 2024-04-23T14:58:26.815 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:26.815 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:26.815 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:26.815 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:26.815 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:26.815 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.816 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.816 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.816 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.816 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.816 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.816 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.816 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.816 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.816 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.817 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.817 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.817 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.817 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.817 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.817 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.817 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.817 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.818 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:26.818 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:26.818 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.818 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.818 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.818 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.818 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.818 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.819 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.819 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.819 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.819 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.819 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.819 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:26.819 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:26.819 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.820 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.820 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.820 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:26.820 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.820 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.820 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.820 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.820 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.820 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.821 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.821 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.821 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.821 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.821 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.821 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.821 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.821 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.821 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:26.822 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:26.822 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.822 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.822 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.822 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.822 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.822 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.822 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.823 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.823 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.823 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.823 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.823 INFO:teuthology.orchestra.run.smithi028.stdout: "predict_interval": { 2024-04-23T14:58:26.823 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predict_interval", 2024-04-23T14:58:26.823 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.823 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.824 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.824 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-04-23T14:58:26.824 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.824 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.824 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.824 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.824 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.824 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.825 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.825 INFO:teuthology.orchestra.run.smithi028.stdout: "predictor_model": { 2024-04-23T14:58:26.825 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predictor_model", 2024-04-23T14:58:26.825 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.825 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.825 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.825 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "prophetstor", 2024-04-23T14:58:26.825 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.826 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.826 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.826 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.826 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.826 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.826 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.826 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-23T14:58:26.827 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-23T14:58:26.827 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.827 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.827 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.827 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-23T14:58:26.827 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.827 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.827 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.827 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.828 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.828 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.828 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.828 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.828 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:26.828 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "influx", 2024-04-23T14:58:26.828 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": false, 2024-04-23T14:58:26.829 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "influxdb python module not found", 2024-04-23T14:58:26.829 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:26.829 INFO:teuthology.orchestra.run.smithi028.stdout: "batch_size": { 2024-04-23T14:58:26.829 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "batch_size", 2024-04-23T14:58:26.829 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:26.829 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.829 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.829 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5000", 2024-04-23T14:58:26.830 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.830 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.830 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.830 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-23T14:58:26.830 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.830 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.830 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.830 INFO:teuthology.orchestra.run.smithi028.stdout: "database": { 2024-04-23T14:58:26.831 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "database", 2024-04-23T14:58:26.831 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.831 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.831 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.831 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "ceph", 2024-04-23T14:58:26.831 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.831 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.831 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.831 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-23T14:58:26.832 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.832 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.832 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.832 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": { 2024-04-23T14:58:26.832 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hostname", 2024-04-23T14:58:26.832 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.832 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.833 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.833 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.833 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.833 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.833 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.833 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server hostname", 2024-04-23T14:58:26.833 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.833 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.833 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.834 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-23T14:58:26.834 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-23T14:58:26.834 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:26.834 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.834 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.834 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "30", 2024-04-23T14:58:26.834 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "5", 2024-04-23T14:58:26.835 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.835 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.835 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-23T14:58:26.835 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.835 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.835 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.835 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:26.835 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:26.835 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.836 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.836 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.836 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.836 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.836 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.836 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.836 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.836 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.836 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.836 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.837 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.837 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.837 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.837 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.837 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.837 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.837 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.837 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:26.838 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:26.838 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.838 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.838 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.838 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.838 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.838 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.838 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.838 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.839 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.839 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.839 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.839 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:26.839 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:26.839 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.839 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.839 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.839 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:26.840 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.840 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.840 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.840 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.840 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.840 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.840 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.840 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.840 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.841 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.841 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.841 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.841 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.841 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.841 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:26.841 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:26.841 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.841 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.842 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.842 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.842 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.842 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.842 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.842 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.842 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.842 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.843 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.843 INFO:teuthology.orchestra.run.smithi028.stdout: "password": { 2024-04-23T14:58:26.843 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "password", 2024-04-23T14:58:26.843 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.843 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.843 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.843 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.843 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.843 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.844 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.844 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "password of InfluxDB server user", 2024-04-23T14:58:26.844 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.844 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.844 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.844 INFO:teuthology.orchestra.run.smithi028.stdout: "port": { 2024-04-23T14:58:26.844 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "port", 2024-04-23T14:58:26.844 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:26.845 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.845 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.845 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8086", 2024-04-23T14:58:26.845 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.845 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.845 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.845 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server port", 2024-04-23T14:58:26.845 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.845 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.846 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.846 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-04-23T14:58:26.846 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-04-23T14:58:26.846 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.846 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.846 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.846 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "false", 2024-04-23T14:58:26.847 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.847 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.847 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.847 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-23T14:58:26.847 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.847 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.847 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.847 INFO:teuthology.orchestra.run.smithi028.stdout: "threads": { 2024-04-23T14:58:26.847 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threads", 2024-04-23T14:58:26.848 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:26.848 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.848 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.848 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-23T14:58:26.848 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-23T14:58:26.848 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "32", 2024-04-23T14:58:26.848 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.848 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-23T14:58:26.848 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.849 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.849 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.849 INFO:teuthology.orchestra.run.smithi028.stdout: "username": { 2024-04-23T14:58:26.849 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "username", 2024-04-23T14:58:26.849 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.849 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.849 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.849 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.849 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.850 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.850 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.850 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "username of InfluxDB server user", 2024-04-23T14:58:26.850 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.850 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.850 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.850 INFO:teuthology.orchestra.run.smithi028.stdout: "verify_ssl": { 2024-04-23T14:58:26.851 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "verify_ssl", 2024-04-23T14:58:26.851 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.851 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.851 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.851 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "true", 2024-04-23T14:58:26.851 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.851 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.851 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.851 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-23T14:58:26.852 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.852 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.852 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.852 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.852 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:26.852 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "insights", 2024-04-23T14:58:26.852 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:26.853 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:26.853 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:26.853 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:26.853 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:26.853 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.853 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.853 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.853 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.853 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.854 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.854 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.854 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.854 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.854 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.854 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.854 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.854 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.854 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.855 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.855 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.855 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.855 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.855 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:26.855 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:26.855 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.855 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.856 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.856 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.856 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.856 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.856 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.856 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.856 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.856 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.857 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.857 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:26.857 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:26.857 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.857 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.857 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.857 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:26.857 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.858 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.858 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.858 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.858 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.858 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.858 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.858 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.858 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.859 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.859 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.859 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.859 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.859 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.859 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:26.859 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:26.860 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.860 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.860 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.860 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.860 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.860 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.860 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.860 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.860 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.861 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.861 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.861 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.861 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:26.861 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "iostat", 2024-04-23T14:58:26.861 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:26.862 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:26.862 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:26.862 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:26.862 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:26.862 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.862 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.862 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.862 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.863 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.863 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.863 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.863 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.863 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.863 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.863 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.863 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.863 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.864 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.864 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.864 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.864 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.864 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.864 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:26.864 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:26.864 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.865 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.865 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.865 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.865 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.865 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.865 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.865 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.865 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.865 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.866 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.866 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:26.866 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:26.866 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.866 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.866 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.866 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:26.867 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.867 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.867 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.867 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.867 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.867 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.867 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.867 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.867 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.868 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.868 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.868 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.868 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.868 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.868 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:26.868 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:26.868 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.869 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.869 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.869 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.869 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.869 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.869 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.869 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.869 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.869 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.870 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.870 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.870 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:26.870 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "k8sevents", 2024-04-23T14:58:26.870 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:26.870 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:26.871 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:26.871 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_event_retention_days": { 2024-04-23T14:58:26.871 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ceph_event_retention_days", 2024-04-23T14:58:26.871 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:26.871 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.871 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.871 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7", 2024-04-23T14:58:26.871 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.871 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.871 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.872 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-23T14:58:26.872 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.872 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.872 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.872 INFO:teuthology.orchestra.run.smithi028.stdout: "config_check_secs": { 2024-04-23T14:58:26.872 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_check_secs", 2024-04-23T14:58:26.872 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:26.872 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.873 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.873 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-23T14:58:26.873 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "10", 2024-04-23T14:58:26.873 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.873 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.873 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-23T14:58:26.873 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.873 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.873 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.874 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:26.874 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:26.874 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.874 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.874 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.875 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.875 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.875 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.875 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.875 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.875 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.875 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.875 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.875 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.876 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.876 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.876 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.876 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.876 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.876 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.876 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:26.876 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:26.877 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.877 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.877 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.877 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.877 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.877 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.877 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.877 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.877 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.877 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.878 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.878 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:26.878 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:26.878 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.878 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.878 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.878 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:26.879 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.879 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.879 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.879 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.879 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.879 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.879 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.879 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.880 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.880 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.880 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.880 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.880 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.880 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.880 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:26.880 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:26.881 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.881 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.881 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.881 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.881 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.881 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.881 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.881 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.882 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.882 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.882 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.882 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.882 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.882 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:26.882 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "localpool", 2024-04-23T14:58:26.883 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:26.883 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:26.883 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:26.883 INFO:teuthology.orchestra.run.smithi028.stdout: "failure_domain": { 2024-04-23T14:58:26.883 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "failure_domain", 2024-04-23T14:58:26.883 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.883 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.883 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.884 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "host", 2024-04-23T14:58:26.884 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.884 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.884 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.884 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "failure domain for any created local pool", 2024-04-23T14:58:26.884 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-23T14:58:26.884 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.884 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.884 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:26.885 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:26.885 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.885 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.885 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.885 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.885 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.885 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.885 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.885 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.886 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.886 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.886 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.886 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.886 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.886 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.886 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.886 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.887 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.887 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.887 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:26.887 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:26.887 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.887 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.887 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.887 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.887 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.888 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.888 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.888 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.888 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.888 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.888 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.888 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:26.888 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:26.889 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.889 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.889 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.889 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:26.889 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.889 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.889 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.889 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.889 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.890 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.890 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.890 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.890 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.890 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.890 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.890 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.890 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.891 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.891 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:26.891 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:26.891 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.891 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.891 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.891 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.891 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.892 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.892 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.892 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.892 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.892 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.892 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.892 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": { 2024-04-23T14:58:26.892 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_size", 2024-04-23T14:58:26.893 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:26.893 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.893 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.893 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.893 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.893 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.893 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.893 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default min_size for any created local pool", 2024-04-23T14:58:26.893 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-23T14:58:26.894 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.894 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.894 INFO:teuthology.orchestra.run.smithi028.stdout: "num_rep": { 2024-04-23T14:58:26.894 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "num_rep", 2024-04-23T14:58:26.894 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:26.894 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.894 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.895 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3", 2024-04-23T14:58:26.895 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.895 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.895 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.895 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default replica count for any created local pool", 2024-04-23T14:58:26.895 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.895 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.895 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.895 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": { 2024-04-23T14:58:26.896 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_num", 2024-04-23T14:58:26.896 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:26.896 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.896 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.896 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "128", 2024-04-23T14:58:26.896 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.896 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.896 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.896 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default pg_num for any created local pool", 2024-04-23T14:58:26.897 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.897 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.897 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.897 INFO:teuthology.orchestra.run.smithi028.stdout: "prefix": { 2024-04-23T14:58:26.897 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prefix", 2024-04-23T14:58:26.897 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.897 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.897 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.898 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.898 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.898 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.898 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.898 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name prefix for any created local pool", 2024-04-23T14:58:26.898 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.898 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.898 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.899 INFO:teuthology.orchestra.run.smithi028.stdout: "subtree": { 2024-04-23T14:58:26.899 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "subtree", 2024-04-23T14:58:26.899 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.899 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.899 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.899 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "rack", 2024-04-23T14:58:26.899 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.899 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.899 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.900 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-23T14:58:26.900 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-23T14:58:26.900 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.900 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.900 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.900 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:26.900 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mds_autoscaler", 2024-04-23T14:58:26.900 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:26.901 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:26.901 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:26.901 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:26.901 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:26.901 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.901 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.901 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.901 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.902 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.902 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.902 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.902 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.902 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.902 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.902 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.902 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.902 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.903 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.903 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.903 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.903 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.903 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.903 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:26.903 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:26.904 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.904 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.904 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.904 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.904 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.904 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.904 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.904 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.904 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.905 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.905 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.905 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:26.905 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:26.905 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.905 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.905 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.905 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:26.906 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.906 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.906 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.906 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.906 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.906 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.906 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.906 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.906 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.907 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.907 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.907 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.907 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.907 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.907 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:26.907 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:26.907 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.907 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.908 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.908 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.908 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.908 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.908 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.908 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.908 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.908 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.908 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.909 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.909 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.909 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:26.909 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirroring", 2024-04-23T14:58:26.909 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:26.909 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:26.909 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:26.909 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:26.910 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:26.910 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.910 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.910 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.910 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.910 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.910 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.910 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.911 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.911 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.911 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.911 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.911 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.911 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.911 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.911 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.911 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.912 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.912 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.912 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:26.912 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:26.912 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.912 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.912 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.912 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.913 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.913 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.913 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.913 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.913 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.913 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.913 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.913 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:26.914 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:26.914 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.914 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.914 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.914 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:26.914 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.914 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.914 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.914 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.915 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.915 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.915 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.915 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.915 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.915 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.915 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.915 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.915 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.916 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.916 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:26.916 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:26.916 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.916 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.916 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.916 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.916 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.917 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.917 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.917 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.917 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.917 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.917 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.917 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.917 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.918 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:26.918 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "nfs", 2024-04-23T14:58:26.918 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:26.918 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:26.918 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:26.918 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:26.918 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:26.918 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.918 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.919 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.919 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.919 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.919 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.919 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.919 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.919 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.919 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.919 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.920 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.920 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.920 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.920 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.920 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.920 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.920 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.920 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:26.921 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:26.921 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.921 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.921 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.921 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.921 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.921 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.921 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.922 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.922 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.922 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.922 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.922 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:26.922 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:26.922 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.922 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.923 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.923 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:26.923 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.923 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.923 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.923 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.923 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.923 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.924 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.924 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.924 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.924 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.924 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.924 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.924 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.924 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.925 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:26.925 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:26.925 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.925 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.925 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.925 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.925 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.925 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.926 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.926 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.926 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.926 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.926 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.926 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.926 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.926 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:26.927 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-04-23T14:58:26.927 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:26.927 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:26.927 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:26.927 INFO:teuthology.orchestra.run.smithi028.stdout: "fail_fs": { 2024-04-23T14:58:26.927 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "fail_fs", 2024-04-23T14:58:26.927 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.927 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.928 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.928 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.928 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.928 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.928 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.928 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-23T14:58:26.928 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.928 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.928 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.929 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:26.929 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:26.929 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.929 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.929 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.929 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.929 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.929 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.930 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.930 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.930 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.930 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.930 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.930 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.930 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.930 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.931 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.931 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.931 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.931 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.931 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:26.931 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:26.931 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.931 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.932 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.932 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.932 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.932 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.932 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.932 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.932 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.932 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.933 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.933 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:26.933 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:26.933 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.933 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.933 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.933 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:26.933 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.934 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.934 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.934 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.934 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.934 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.934 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.934 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.934 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.935 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.935 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.935 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.935 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.935 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.935 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:26.935 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:26.935 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.936 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.936 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.936 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.936 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.936 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.936 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.936 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.936 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.936 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.937 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.937 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator": { 2024-04-23T14:58:26.937 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-04-23T14:58:26.937 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.937 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.937 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.937 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.937 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.938 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.938 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.938 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-04-23T14:58:26.938 INFO:teuthology.orchestra.run.smithi028.stdout: "rook", 2024-04-23T14:58:26.938 INFO:teuthology.orchestra.run.smithi028.stdout: "test_orchestrator" 2024-04-23T14:58:26.938 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.938 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Orchestrator backend", 2024-04-23T14:58:26.938 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.939 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.939 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.939 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.939 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.939 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:26.939 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_perf_query", 2024-04-23T14:58:26.939 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:26.939 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:26.940 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:26.940 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:26.940 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:26.940 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.940 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.940 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.940 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.940 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.940 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.941 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.941 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.941 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.941 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.941 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.941 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.941 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.941 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.941 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.942 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.942 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.942 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.942 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:26.942 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:26.942 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.942 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.943 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.943 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.943 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.943 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.943 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.943 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.943 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.943 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.943 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.944 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:26.944 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:26.944 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.944 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.944 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.944 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:26.944 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.944 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.945 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.945 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.945 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.945 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.945 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.945 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.945 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.945 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.946 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.946 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.946 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.946 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.946 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:26.946 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:26.946 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.947 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.947 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.947 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.947 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.947 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.947 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.947 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.947 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.947 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.948 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.948 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.948 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:26.948 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_support", 2024-04-23T14:58:26.948 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:26.948 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:26.948 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:26.949 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:26.949 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:26.949 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.949 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.949 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.949 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.949 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.949 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.949 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.950 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.950 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.950 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.950 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.950 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.950 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.950 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.950 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.950 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.951 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.951 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.951 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:26.951 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:26.951 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.951 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.951 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.952 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.952 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.952 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.952 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.952 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.952 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.952 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.952 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.953 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:26.953 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:26.953 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.953 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.953 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.953 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:26.953 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.953 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.953 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.954 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.954 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.954 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.954 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.954 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.954 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.954 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.954 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.955 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.955 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.955 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.955 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:26.955 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:26.955 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.955 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.955 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.956 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.956 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.956 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.956 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.956 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.956 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.956 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.956 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.957 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.957 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:26.957 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_autoscaler", 2024-04-23T14:58:26.957 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:26.957 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:26.957 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:26.957 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:26.958 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:26.958 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.958 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.958 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.958 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.958 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.958 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.958 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.958 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.959 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.959 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.959 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.959 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.959 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.959 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.959 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.959 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.960 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.960 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.960 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:26.960 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:26.960 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.960 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.960 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.960 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.961 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.961 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.961 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.961 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.961 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.961 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.961 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.961 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:26.962 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:26.962 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.962 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.962 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.962 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:26.962 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.962 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.962 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.962 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.963 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.963 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.963 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.963 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.963 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.963 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.963 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.963 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.963 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.964 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.964 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:26.964 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:26.964 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.964 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.964 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.964 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.964 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.965 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.965 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.965 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.965 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.965 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.965 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.965 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-23T14:58:26.966 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-23T14:58:26.966 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:26.966 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.966 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.966 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-23T14:58:26.966 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.966 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.966 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.966 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.967 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.967 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.967 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.967 INFO:teuthology.orchestra.run.smithi028.stdout: "threshold": { 2024-04-23T14:58:26.967 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threshold", 2024-04-23T14:58:26.967 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-23T14:58:26.967 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.968 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.968 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-04-23T14:58:26.968 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1.0", 2024-04-23T14:58:26.968 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.968 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.968 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "scaling threshold", 2024-04-23T14:58:26.968 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-23T14:58:26.968 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.968 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.969 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.969 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:26.969 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "progress", 2024-04-23T14:58:26.969 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:26.969 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:26.969 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:26.969 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_pg_recovery_event": { 2024-04-23T14:58:26.970 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_pg_recovery_event", 2024-04-23T14:58:26.970 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.970 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.970 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.970 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.970 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.970 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.970 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.970 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-23T14:58:26.971 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.971 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.971 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.971 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-04-23T14:58:26.971 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-04-23T14:58:26.971 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.971 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.971 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.972 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:26.972 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.972 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.972 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.972 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.972 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.972 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.972 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.973 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:26.973 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:26.973 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.973 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.973 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.973 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.973 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.973 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.974 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.974 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.974 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.974 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.974 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.974 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.974 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.974 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.974 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.975 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.975 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.975 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.975 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:26.975 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:26.975 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.975 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.976 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.976 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.976 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.976 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.976 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.976 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.976 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.976 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.976 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.977 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:26.977 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:26.977 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.977 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.977 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.977 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:26.977 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.977 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.978 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.978 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.978 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.978 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.978 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.978 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.978 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.978 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.979 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.982 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.982 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.982 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.983 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:26.983 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:26.983 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.983 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.983 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.983 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.983 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.984 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.984 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.984 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.984 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.984 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.984 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.984 INFO:teuthology.orchestra.run.smithi028.stdout: "max_completed_events": { 2024-04-23T14:58:26.984 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_completed_events", 2024-04-23T14:58:26.985 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:26.985 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.985 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.985 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "50", 2024-04-23T14:58:26.985 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.985 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.985 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.985 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "number of past completed events to remember", 2024-04-23T14:58:26.985 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.986 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.986 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.986 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-23T14:58:26.986 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-23T14:58:26.986 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:26.986 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.986 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.987 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-23T14:58:26.987 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.987 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.987 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.987 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long the module is going to sleep", 2024-04-23T14:58:26.987 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.987 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.987 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.987 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.988 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:26.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:26.988 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus", 2024-04-23T14:58:26.988 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:26.988 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:26.988 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:26.988 INFO:teuthology.orchestra.run.smithi028.stdout: "cache": { 2024-04-23T14:58:26.988 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cache", 2024-04-23T14:58:26.989 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.989 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.989 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:26.989 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:26.989 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.989 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.989 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.989 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.989 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.990 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.990 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.990 INFO:teuthology.orchestra.run.smithi028.stdout: "exclude_perf_counters": { 2024-04-23T14:58:26.990 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "exclude_perf_counters", 2024-04-23T14:58:26.990 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.990 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.990 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.991 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:26.991 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.991 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.991 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.991 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-23T14:58:26.991 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-23T14:58:26.991 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.991 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.992 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:26.992 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:26.992 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.992 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.992 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.992 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:26.992 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.992 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.992 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.993 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.993 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.993 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.993 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.993 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.993 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.993 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.993 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.993 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.994 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.994 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.994 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:26.994 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:26.994 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.994 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.995 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.995 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.995 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.995 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.995 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.995 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.995 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.995 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.995 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.996 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:26.996 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:26.996 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:26.996 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.996 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.996 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:26.996 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.996 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.996 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:26.997 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:26.997 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:26.997 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:26.997 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:26.997 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:26.997 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:26.997 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:26.997 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.998 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.998 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.998 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:26.998 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:26.998 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:26.998 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:26.998 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:26.998 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:26.999 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:26.999 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:26.999 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:26.999 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:26.999 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:26.999 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:26.999 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:26.999 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:26.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.000 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools": { 2024-04-23T14:58:27.000 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools", 2024-04-23T14:58:27.000 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.000 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.000 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.000 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.000 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.000 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.000 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.000 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.001 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.001 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.001 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.001 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-23T14:58:27.001 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-23T14:58:27.001 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.001 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.001 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.001 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "300", 2024-04-23T14:58:27.002 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.002 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.002 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.002 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.002 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.002 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.002 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.003 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_interval": { 2024-04-23T14:58:27.003 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_interval", 2024-04-23T14:58:27.003 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-23T14:58:27.003 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.003 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.003 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15.0", 2024-04-23T14:58:27.003 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.003 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.003 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.004 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.004 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.004 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.004 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.004 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-23T14:58:27.004 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-23T14:58:27.004 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.004 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.005 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.005 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-04-23T14:58:27.005 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.005 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.005 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.005 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-23T14:58:27.005 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.005 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.005 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.006 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-23T14:58:27.006 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-23T14:58:27.006 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.006 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.006 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.006 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "9283", 2024-04-23T14:58:27.006 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.006 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.007 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.007 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-23T14:58:27.007 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.007 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.007 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.007 INFO:teuthology.orchestra.run.smithi028.stdout: "stale_cache_strategy": { 2024-04-23T14:58:27.007 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stale_cache_strategy", 2024-04-23T14:58:27.008 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.008 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.008 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.008 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "log", 2024-04-23T14:58:27.008 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.008 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.008 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.008 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.008 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.009 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.009 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.009 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-04-23T14:58:27.009 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-04-23T14:58:27.009 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.009 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.009 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.010 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "default", 2024-04-23T14:58:27.010 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.010 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.010 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.010 INFO:teuthology.orchestra.run.smithi028.stdout: "default", 2024-04-23T14:58:27.010 INFO:teuthology.orchestra.run.smithi028.stdout: "error" 2024-04-23T14:58:27.010 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.010 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.011 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.011 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.011 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.011 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-04-23T14:58:27.011 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-04-23T14:58:27.011 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.011 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.011 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.012 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-04-23T14:58:27.012 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-04-23T14:58:27.012 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-04-23T14:58:27.012 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.012 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.012 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.012 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.012 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.013 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.013 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.013 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-04-23T14:58:27.013 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.013 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.013 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.013 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.014 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.014 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.014 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.014 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.014 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.014 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.014 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.014 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.015 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.015 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.015 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.015 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.015 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.015 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.015 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.015 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.015 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.016 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.016 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.016 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.016 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.016 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.016 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.016 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.016 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.017 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.017 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.017 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.017 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.017 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.017 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.017 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.017 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.018 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.018 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.018 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.018 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.018 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.018 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.018 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.018 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.019 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.019 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.019 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.019 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.019 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.019 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.019 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.019 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.019 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.020 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.020 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.020 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.020 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.020 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.020 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.020 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.021 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.021 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.021 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.021 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.021 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.021 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.021 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.021 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.022 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_snap_create": { 2024-04-23T14:58:27.022 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_snap_create", 2024-04-23T14:58:27.022 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.022 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.022 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.022 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-23T14:58:27.022 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.022 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.022 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.023 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.023 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.023 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.023 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.023 INFO:teuthology.orchestra.run.smithi028.stdout: "mirror_snapshot_schedule": { 2024-04-23T14:58:27.023 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirror_snapshot_schedule", 2024-04-23T14:58:27.023 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.024 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.024 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.024 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.024 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.024 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.024 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.024 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.024 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.024 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.025 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.025 INFO:teuthology.orchestra.run.smithi028.stdout: "trash_purge_schedule": { 2024-04-23T14:58:27.025 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "trash_purge_schedule", 2024-04-23T14:58:27.025 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.025 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.025 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.026 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.026 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.026 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.026 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.026 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.026 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.026 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.026 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.027 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.027 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.027 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.027 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "restful", 2024-04-23T14:58:27.027 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.027 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.027 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.027 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_auth": { 2024-04-23T14:58:27.028 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_auth", 2024-04-23T14:58:27.028 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.028 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.028 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.028 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.028 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.028 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.028 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.029 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.029 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.029 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.029 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.029 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-04-23T14:58:27.029 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-04-23T14:58:27.029 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.029 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.030 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.030 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.030 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.030 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.030 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.030 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.030 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.030 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.031 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.031 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.031 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.031 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.031 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.031 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.031 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.031 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.032 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.032 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.032 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.032 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.032 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.032 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.032 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.032 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.032 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.033 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.033 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.033 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.033 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.033 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.033 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.033 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.034 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.034 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.034 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.034 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.034 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.034 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.034 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.034 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.034 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.035 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.035 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.035 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.035 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.035 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.035 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.035 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.035 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.036 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.036 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.036 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.036 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.036 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.036 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.036 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.036 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.036 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.037 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.037 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.037 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.037 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.037 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.037 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.037 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.038 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.038 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.038 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.038 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.038 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.038 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.038 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.038 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.039 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.039 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.039 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-23T14:58:27.039 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-23T14:58:27.039 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.039 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.039 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.039 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.040 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.040 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.040 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.040 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.040 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.040 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.040 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.040 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-23T14:58:27.041 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-23T14:58:27.041 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.041 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.041 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.041 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.041 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.041 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.041 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.042 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.042 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.042 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.042 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.042 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.042 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.042 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rgw", 2024-04-23T14:58:27.043 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.043 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.043 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.043 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.043 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.043 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.043 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.043 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.044 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.044 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.044 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.044 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.044 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.044 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.044 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.044 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.044 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.045 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.045 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.045 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.045 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.045 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.045 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.045 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.045 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.046 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.046 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.046 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.046 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.046 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.046 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.046 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.046 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.047 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.047 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.047 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.047 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.047 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.047 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.047 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.047 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.048 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.048 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.048 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.048 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.048 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.048 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.048 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.048 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.048 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.049 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.049 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.049 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.049 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.049 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.049 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.049 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.050 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.050 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.050 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.050 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.050 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.050 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.050 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.050 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.051 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.051 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.051 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.051 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.051 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.051 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.051 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.051 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rook", 2024-04-23T14:58:27.052 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.052 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.052 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.052 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.052 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.052 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.052 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.052 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.053 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.053 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.053 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.053 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.053 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.053 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.053 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.053 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.053 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.054 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.054 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.054 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.054 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.054 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.054 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.054 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.054 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.055 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.055 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.055 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.055 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.055 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.055 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.055 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.055 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.056 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.056 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.056 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.056 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.056 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.056 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.056 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.056 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.057 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.057 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.057 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.057 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.057 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.057 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.057 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.057 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.057 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.058 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.058 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.058 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.058 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.058 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.058 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.058 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.059 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.059 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.059 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.059 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.059 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.059 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.059 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.059 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.059 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.060 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.060 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.060 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.060 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.060 INFO:teuthology.orchestra.run.smithi028.stdout: "storage_class": { 2024-04-23T14:58:27.060 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "storage_class", 2024-04-23T14:58:27.060 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.060 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.061 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.061 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "local", 2024-04-23T14:58:27.061 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.061 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.061 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.061 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-23T14:58:27.061 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.061 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.062 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.062 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.062 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.062 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "selftest", 2024-04-23T14:58:27.062 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.062 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.062 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.063 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.063 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.063 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.063 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.063 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.063 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.063 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.063 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.064 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.064 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.064 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.064 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.064 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.064 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.064 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.064 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.064 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.065 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.065 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.065 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.065 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.065 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.065 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.065 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.065 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.066 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.066 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.066 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.066 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.066 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.066 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.066 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.066 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.067 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.067 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.067 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.067 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.067 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.067 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.067 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.067 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.067 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.068 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.068 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.068 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.068 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.068 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.068 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.068 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.068 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.069 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.069 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.069 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.069 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.069 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.069 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.069 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.069 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.069 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.070 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.070 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.070 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.070 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.070 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.070 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.070 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.071 INFO:teuthology.orchestra.run.smithi028.stdout: "roption1": { 2024-04-23T14:58:27.071 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption1", 2024-04-23T14:58:27.071 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.071 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.071 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.071 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.071 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.071 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.072 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.072 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.072 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.072 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.072 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.072 INFO:teuthology.orchestra.run.smithi028.stdout: "roption2": { 2024-04-23T14:58:27.072 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption2", 2024-04-23T14:58:27.072 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.073 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.073 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.073 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "xyz", 2024-04-23T14:58:27.073 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.073 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.073 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.073 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.073 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.074 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.074 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.074 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption1": { 2024-04-23T14:58:27.074 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption1", 2024-04-23T14:58:27.074 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.074 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.074 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.074 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.075 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.075 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.075 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.075 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.075 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.075 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.075 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.076 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption2": { 2024-04-23T14:58:27.076 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption2", 2024-04-23T14:58:27.076 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.076 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.076 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.076 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.076 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.076 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.076 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.077 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.077 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.077 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.077 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.077 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption3": { 2024-04-23T14:58:27.077 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption3", 2024-04-23T14:58:27.077 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-23T14:58:27.077 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.078 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.078 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.078 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.078 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.078 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.078 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.078 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.078 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.079 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.079 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption4": { 2024-04-23T14:58:27.079 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption4", 2024-04-23T14:58:27.079 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.079 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.079 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.079 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.079 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.080 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.080 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.080 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.080 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.080 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.080 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.080 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption5": { 2024-04-23T14:58:27.080 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption5", 2024-04-23T14:58:27.081 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.081 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.081 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.081 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.081 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.081 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.081 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.081 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.082 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.082 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.082 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.082 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption6": { 2024-04-23T14:58:27.082 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption6", 2024-04-23T14:58:27.082 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.082 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.082 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.083 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.083 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.083 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.083 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.083 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.083 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.083 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.083 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.083 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption7": { 2024-04-23T14:58:27.084 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption7", 2024-04-23T14:58:27.084 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.084 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.084 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.084 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.084 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-23T14:58:27.084 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "42", 2024-04-23T14:58:27.084 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.084 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.085 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.085 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.085 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.085 INFO:teuthology.orchestra.run.smithi028.stdout: "testkey": { 2024-04-23T14:58:27.085 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testkey", 2024-04-23T14:58:27.085 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.085 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.086 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.086 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.086 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.086 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.086 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.086 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.086 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.086 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.086 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.087 INFO:teuthology.orchestra.run.smithi028.stdout: "testlkey": { 2024-04-23T14:58:27.087 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testlkey", 2024-04-23T14:58:27.087 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.087 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.087 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.087 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.087 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.087 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.088 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.088 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.088 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.088 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.088 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.088 INFO:teuthology.orchestra.run.smithi028.stdout: "testnewline": { 2024-04-23T14:58:27.088 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testnewline", 2024-04-23T14:58:27.088 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.088 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.089 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.089 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.089 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.089 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.089 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.089 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.089 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.089 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.089 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.090 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.090 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.090 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-04-23T14:58:27.090 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.090 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.090 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.090 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_m_granularity": { 2024-04-23T14:58:27.091 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_m_granularity", 2024-04-23T14:58:27.091 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.091 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.091 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.091 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.091 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.091 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.091 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.091 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow minute scheduled snapshots", 2024-04-23T14:58:27.092 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.092 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.092 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.092 INFO:teuthology.orchestra.run.smithi028.stdout: "dump_on_update": { 2024-04-23T14:58:27.092 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dump_on_update", 2024-04-23T14:58:27.092 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.092 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.092 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.093 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.093 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.093 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.093 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.093 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "dump database to debug log on update", 2024-04-23T14:58:27.093 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.093 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.093 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.094 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.094 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.094 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.094 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.094 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.094 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.094 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.094 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.095 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.095 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.095 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.095 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.095 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.095 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.095 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.095 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.095 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.096 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.096 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.096 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.096 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.096 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.096 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.096 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.096 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.097 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.097 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.097 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.097 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.097 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.097 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.097 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.097 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.098 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.098 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.098 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.098 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.098 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.098 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.098 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.099 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.099 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.099 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.099 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.099 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.099 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.099 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.099 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.100 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.100 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.100 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.100 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.100 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.100 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.100 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.100 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.100 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.101 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.101 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.101 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.101 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.101 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.101 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.101 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.101 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.102 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.102 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.102 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.102 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stats", 2024-04-23T14:58:27.102 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.102 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.103 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.103 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.103 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.103 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.103 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.103 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.103 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.103 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.103 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.104 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.104 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.104 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.104 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.104 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.104 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.104 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.104 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.104 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.105 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.105 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.105 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.105 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.105 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.105 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.105 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.105 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.106 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.106 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.106 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.106 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.106 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.106 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.106 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.106 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.107 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.107 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.107 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.107 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.107 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.107 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.107 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.107 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.107 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.107 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.108 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.108 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.108 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.108 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.108 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.108 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.108 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.108 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.108 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.109 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.109 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.109 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.109 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.109 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.109 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.109 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.109 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.109 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.110 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.110 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.110 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.110 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.110 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.110 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.110 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.111 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "status", 2024-04-23T14:58:27.111 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.111 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.111 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.111 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.111 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.111 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.111 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.112 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.112 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.112 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.112 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.112 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.112 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.112 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.112 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.112 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.113 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.113 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.113 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.113 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.113 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.113 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.113 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.113 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.113 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.114 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.114 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.114 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.114 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.114 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.114 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.114 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.114 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.114 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.115 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.115 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.115 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.115 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.115 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.115 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.115 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.115 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.116 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.116 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.116 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.116 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.116 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.116 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.116 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.116 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.116 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.117 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.117 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.117 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.117 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.117 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.117 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.117 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.117 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.118 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.118 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.118 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.118 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.118 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.118 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.118 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.118 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.118 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.119 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.119 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.119 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.119 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telegraf", 2024-04-23T14:58:27.119 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.119 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.119 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.120 INFO:teuthology.orchestra.run.smithi028.stdout: "address": { 2024-04-23T14:58:27.120 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "address", 2024-04-23T14:58:27.120 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.120 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.120 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.120 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-23T14:58:27.120 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.120 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.121 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.121 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.121 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.121 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.121 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.121 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-23T14:58:27.121 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-23T14:58:27.121 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:27.122 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.122 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.122 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15", 2024-04-23T14:58:27.122 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.122 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.122 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.122 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.122 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.122 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.123 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.123 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.123 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.123 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.123 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.123 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.123 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.123 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.124 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.124 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.124 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.124 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.124 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.124 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.124 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.124 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.125 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.125 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.125 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.125 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.125 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.125 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.125 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.125 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.125 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.126 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.126 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.126 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.126 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.126 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.126 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.126 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.126 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.127 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.127 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.127 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.127 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.127 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.127 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.127 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.127 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.128 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.128 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.128 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.128 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.128 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.128 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.128 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.128 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.128 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.129 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.129 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.129 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.129 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.129 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.129 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.129 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.129 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.130 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.130 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.130 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.130 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.130 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.130 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.130 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.130 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.131 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.131 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.131 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.131 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telemetry", 2024-04-23T14:58:27.131 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.131 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.131 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.132 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_basic": { 2024-04-23T14:58:27.132 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_basic", 2024-04-23T14:58:27.132 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.132 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.132 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.132 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.132 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.132 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.132 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.133 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-23T14:58:27.133 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.133 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.133 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.133 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_crash": { 2024-04-23T14:58:27.133 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_crash", 2024-04-23T14:58:27.133 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.134 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.134 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.134 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.134 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.134 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.134 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.134 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-23T14:58:27.134 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.135 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.135 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.135 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_device": { 2024-04-23T14:58:27.135 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_device", 2024-04-23T14:58:27.135 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.135 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.135 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.135 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.136 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.136 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.136 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.136 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-23T14:58:27.136 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.136 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.136 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.136 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_ident": { 2024-04-23T14:58:27.137 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_ident", 2024-04-23T14:58:27.137 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.137 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.137 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.137 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.137 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.137 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.137 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.138 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-23T14:58:27.138 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.138 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.138 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.138 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_perf": { 2024-04-23T14:58:27.138 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_perf", 2024-04-23T14:58:27.138 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.139 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.139 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.139 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.139 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.139 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.139 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.139 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-23T14:58:27.139 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.139 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.140 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.140 INFO:teuthology.orchestra.run.smithi028.stdout: "contact": { 2024-04-23T14:58:27.140 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "contact", 2024-04-23T14:58:27.140 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.140 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.140 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.140 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.141 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.141 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.141 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.141 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.141 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.141 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.141 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.142 INFO:teuthology.orchestra.run.smithi028.stdout: "description": { 2024-04-23T14:58:27.142 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "description", 2024-04-23T14:58:27.142 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.142 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.142 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.142 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.142 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.142 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.143 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.143 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.143 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.143 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.143 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.143 INFO:teuthology.orchestra.run.smithi028.stdout: "device_url": { 2024-04-23T14:58:27.143 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_url", 2024-04-23T14:58:27.143 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.144 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.144 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.144 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-23T14:58:27.144 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.144 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.144 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.144 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.144 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.144 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.145 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.145 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-04-23T14:58:27.145 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-04-23T14:58:27.145 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.145 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.145 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.145 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.145 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.146 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.146 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.146 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.146 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.146 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.146 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.146 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-23T14:58:27.147 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-23T14:58:27.147 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.147 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.147 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.147 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "24", 2024-04-23T14:58:27.147 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "8", 2024-04-23T14:58:27.147 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.147 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.147 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.148 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.148 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.148 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.148 INFO:teuthology.orchestra.run.smithi028.stdout: "last_opt_revision": { 2024-04-23T14:58:27.148 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "last_opt_revision", 2024-04-23T14:58:27.148 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.148 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.149 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.149 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1", 2024-04-23T14:58:27.149 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.149 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.149 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.149 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.149 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.149 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.149 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.150 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard": { 2024-04-23T14:58:27.150 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard", 2024-04-23T14:58:27.150 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.150 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.150 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.150 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.150 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.150 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.151 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.151 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.151 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.151 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.151 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.151 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard_description": { 2024-04-23T14:58:27.151 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard_description", 2024-04-23T14:58:27.151 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.152 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.152 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.152 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.152 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.152 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.152 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.152 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.152 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.153 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.153 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.153 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.153 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.153 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.153 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.153 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.153 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.154 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.154 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.154 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.154 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.154 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.154 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.154 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.154 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.155 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.155 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.155 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.155 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.155 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.155 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.155 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.155 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.156 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.156 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.156 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.156 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.156 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.156 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.156 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.156 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.157 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.157 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.157 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.157 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.157 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.157 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.157 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.157 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.158 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.158 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.158 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.158 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.158 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.158 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.158 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.158 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.158 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.159 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.159 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.159 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.159 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.159 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.159 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.159 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.160 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.160 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.160 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.160 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.160 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.160 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.160 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.160 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.160 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.161 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.161 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.161 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.161 INFO:teuthology.orchestra.run.smithi028.stdout: "organization": { 2024-04-23T14:58:27.161 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "organization", 2024-04-23T14:58:27.161 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.161 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.161 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.162 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.162 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.162 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.162 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.162 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.162 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.162 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.162 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.163 INFO:teuthology.orchestra.run.smithi028.stdout: "proxy": { 2024-04-23T14:58:27.163 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "proxy", 2024-04-23T14:58:27.163 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.163 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.163 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.163 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.163 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.163 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.164 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.164 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.164 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.164 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.164 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.164 INFO:teuthology.orchestra.run.smithi028.stdout: "url": { 2024-04-23T14:58:27.164 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url", 2024-04-23T14:58:27.164 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.165 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.165 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.165 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-23T14:58:27.165 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.165 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.165 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.165 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.165 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.165 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.166 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.166 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.166 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.166 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "test_orchestrator", 2024-04-23T14:58:27.166 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.167 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.167 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.167 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.167 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.167 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.167 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.167 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.167 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.167 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.168 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.168 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.168 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.168 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.168 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.168 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.168 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.168 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.168 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.169 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.169 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.169 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.169 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.169 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.169 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.169 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.169 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.170 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.170 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.170 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.170 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.170 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.170 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.170 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.171 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.171 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.171 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.171 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.171 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.171 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.171 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.171 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.172 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.172 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.172 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.172 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.172 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.172 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.172 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.172 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.172 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.173 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.173 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.173 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.173 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.173 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.173 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.173 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.173 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.174 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.174 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.174 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.174 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.174 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.174 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.174 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.174 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.175 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.175 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.175 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.175 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.175 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.175 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-04-23T14:58:27.175 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.175 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.176 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.176 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.176 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.176 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.176 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.176 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.176 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.176 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.177 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.177 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.177 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.177 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.177 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.177 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.177 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.177 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.177 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.178 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.178 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.178 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.178 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.178 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.178 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.178 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.179 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.179 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.179 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.179 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.179 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.179 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.179 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.179 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.179 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.180 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.180 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.180 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.180 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.180 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.180 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.180 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.180 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.181 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.181 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.181 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.181 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.181 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.181 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.181 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.181 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.182 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.182 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.182 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.182 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.182 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.182 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.182 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.183 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.183 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.183 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.183 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.183 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.183 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.183 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.183 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.183 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.184 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.184 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.184 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_clones": { 2024-04-23T14:58:27.184 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_clones", 2024-04-23T14:58:27.184 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.184 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.184 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.184 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4", 2024-04-23T14:58:27.185 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.185 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.185 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.185 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-23T14:58:27.185 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.185 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.185 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.185 INFO:teuthology.orchestra.run.smithi028.stdout: "periodic_async_work": { 2024-04-23T14:58:27.186 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "periodic_async_work", 2024-04-23T14:58:27.186 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.186 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.186 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.186 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.186 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.186 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.186 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.187 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Periodically check for async work", 2024-04-23T14:58:27.187 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.187 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.187 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.187 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_delay": { 2024-04-23T14:58:27.187 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_delay", 2024-04-23T14:58:27.187 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.187 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.188 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.188 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-23T14:58:27.188 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.188 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.188 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.188 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-23T14:58:27.188 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.188 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.188 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.189 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_no_wait": { 2024-04-23T14:58:27.189 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_no_wait", 2024-04-23T14:58:27.189 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.189 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.189 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.189 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.189 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.190 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.190 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.190 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-23T14:58:27.190 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.190 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.190 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.190 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.190 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.191 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix", 2024-04-23T14:58:27.191 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.191 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.191 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.191 INFO:teuthology.orchestra.run.smithi028.stdout: "discovery_interval": { 2024-04-23T14:58:27.191 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "discovery_interval", 2024-04-23T14:58:27.191 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-23T14:58:27.192 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.192 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.192 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "100", 2024-04-23T14:58:27.192 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.192 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.192 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.192 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.192 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.192 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.193 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.193 INFO:teuthology.orchestra.run.smithi028.stdout: "identifier": { 2024-04-23T14:58:27.193 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "identifier", 2024-04-23T14:58:27.193 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.193 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.193 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.193 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.194 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.194 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.194 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.194 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.194 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.194 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.194 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.194 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-23T14:58:27.195 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-23T14:58:27.195 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:27.195 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.195 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.195 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-23T14:58:27.195 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.195 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.195 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.195 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.196 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.196 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.196 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.196 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.196 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.196 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.196 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.196 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.197 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.197 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.197 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.197 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.197 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.197 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.197 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.197 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.197 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.198 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.198 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.198 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.198 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.198 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.198 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.198 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.199 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.199 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.199 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.199 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.199 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.199 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.199 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.199 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.199 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.200 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.200 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.200 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.200 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.200 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.200 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.200 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.201 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.201 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.201 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.201 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.201 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.201 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.201 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.201 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.201 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.202 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.202 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.202 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.202 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.202 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.202 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.202 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.202 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.203 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.203 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.203 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.203 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.203 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.203 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.203 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.203 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.204 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.204 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.204 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.204 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.204 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_host": { 2024-04-23T14:58:27.204 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_host", 2024-04-23T14:58:27.204 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.204 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.205 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.205 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.205 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.205 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.205 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.205 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.205 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.205 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.205 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.206 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_port": { 2024-04-23T14:58:27.206 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_port", 2024-04-23T14:58:27.206 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.206 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.206 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.206 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10051", 2024-04-23T14:58:27.206 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.207 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.207 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.207 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.207 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.207 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.207 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.207 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_sender": { 2024-04-23T14:58:27.207 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_sender", 2024-04-23T14:58:27.208 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.208 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.208 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.208 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-23T14:58:27.208 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.208 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.208 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.208 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.208 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.208 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.209 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.209 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.209 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.209 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.209 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-23T14:58:27.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.209 INFO:teuthology.orchestra.run.smithi028.stdout: "gid": 25150, 2024-04-23T14:58:27.209 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "y", 2024-04-23T14:58:27.210 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_features": 4540701547738038271, 2024-04-23T14:58:27.210 INFO:teuthology.orchestra.run.smithi028.stdout: "available_modules": [ 2024-04-23T14:58:27.210 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.210 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "alerts", 2024-04-23T14:58:27.210 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.210 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.210 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.210 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-23T14:58:27.211 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-23T14:58:27.211 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:27.211 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.211 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.211 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-23T14:58:27.211 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.211 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.211 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.211 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How frequently to reexamine health status", 2024-04-23T14:58:27.212 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.212 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.212 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.212 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.212 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.212 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.212 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.212 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.213 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.213 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.213 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.213 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.213 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.213 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.213 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.213 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.213 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.213 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.214 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.214 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.214 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.214 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.214 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.214 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.214 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.215 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.215 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.215 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.215 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.215 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.215 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.215 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.215 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.216 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.216 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.216 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.216 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.216 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.216 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.216 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.216 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.217 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.217 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.217 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.217 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.217 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.217 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.217 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.217 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.217 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.218 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.218 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.218 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.218 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.218 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.218 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.218 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.219 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.219 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.219 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.219 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.219 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.219 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.219 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.219 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.219 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.219 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.220 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.220 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.220 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_destination": { 2024-04-23T14:58:27.220 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_destination", 2024-04-23T14:58:27.220 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.220 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.220 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.221 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.221 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.221 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.221 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.221 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email address to send alerts to", 2024-04-23T14:58:27.221 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.221 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.221 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.222 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_from_name": { 2024-04-23T14:58:27.222 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_from_name", 2024-04-23T14:58:27.222 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.222 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.222 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.222 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "Ceph", 2024-04-23T14:58:27.222 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.222 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.223 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.223 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email From: name", 2024-04-23T14:58:27.223 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.223 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.223 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.223 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_host": { 2024-04-23T14:58:27.223 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_host", 2024-04-23T14:58:27.223 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.223 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.224 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.224 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.224 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.224 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.224 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.224 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP server", 2024-04-23T14:58:27.224 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.224 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.224 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.225 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_password": { 2024-04-23T14:58:27.225 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_password", 2024-04-23T14:58:27.225 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.225 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.225 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.225 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.225 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.225 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.226 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.226 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Password to authenticate with", 2024-04-23T14:58:27.226 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.226 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.226 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.226 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_port": { 2024-04-23T14:58:27.226 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_port", 2024-04-23T14:58:27.227 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.227 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.227 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.227 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "465", 2024-04-23T14:58:27.227 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.227 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.227 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.227 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP port", 2024-04-23T14:58:27.227 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.228 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.228 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.228 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_sender": { 2024-04-23T14:58:27.228 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_sender", 2024-04-23T14:58:27.228 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.228 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.228 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.228 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.229 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.229 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.229 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.229 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP envelope sender", 2024-04-23T14:58:27.229 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.229 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.229 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.230 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_ssl": { 2024-04-23T14:58:27.230 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_ssl", 2024-04-23T14:58:27.230 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.230 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.230 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.230 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.230 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.230 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.231 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.231 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-23T14:58:27.231 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.231 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.231 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.231 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_user": { 2024-04-23T14:58:27.231 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_user", 2024-04-23T14:58:27.231 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.232 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.232 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.232 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.232 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.232 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.232 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.232 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "User to authenticate as", 2024-04-23T14:58:27.232 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.233 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.233 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.233 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.233 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.233 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "balancer", 2024-04-23T14:58:27.233 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.233 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.233 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.234 INFO:teuthology.orchestra.run.smithi028.stdout: "active": { 2024-04-23T14:58:27.234 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "active", 2024-04-23T14:58:27.234 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.234 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.234 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.234 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.234 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.234 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.235 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.235 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically balance PGs across cluster", 2024-04-23T14:58:27.235 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.235 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.235 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.235 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_time": { 2024-04-23T14:58:27.235 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_time", 2024-04-23T14:58:27.235 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.236 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.236 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.236 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0000", 2024-04-23T14:58:27.236 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.236 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.236 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.236 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "beginning time of day to automatically balance", 2024-04-23T14:58:27.236 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-23T14:58:27.236 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.237 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.237 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_weekday": { 2024-04-23T14:58:27.237 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_weekday", 2024-04-23T14:58:27.237 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-23T14:58:27.237 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.237 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.237 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-23T14:58:27.237 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-04-23T14:58:27.237 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-04-23T14:58:27.238 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.238 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-23T14:58:27.238 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-23T14:58:27.238 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.238 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.238 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_max_iterations": { 2024-04-23T14:58:27.238 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_max_iterations", 2024-04-23T14:58:27.239 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-23T14:58:27.239 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.239 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.239 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "25", 2024-04-23T14:58:27.239 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-23T14:58:27.239 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "250", 2024-04-23T14:58:27.239 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.239 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-23T14:58:27.239 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.240 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.240 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.240 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_metrics": { 2024-04-23T14:58:27.240 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_metrics", 2024-04-23T14:58:27.240 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.240 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.240 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.240 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "pgs,objects,bytes", 2024-04-23T14:58:27.241 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.241 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.241 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.241 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-23T14:58:27.241 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-23T14:58:27.241 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.241 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.241 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_step": { 2024-04-23T14:58:27.242 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_step", 2024-04-23T14:58:27.242 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-23T14:58:27.242 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.242 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.242 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.5", 2024-04-23T14:58:27.242 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0.001", 2024-04-23T14:58:27.242 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "0.999", 2024-04-23T14:58:27.242 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.243 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "aggressiveness of optimization", 2024-04-23T14:58:27.243 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-23T14:58:27.243 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.243 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.243 INFO:teuthology.orchestra.run.smithi028.stdout: "end_time": { 2024-04-23T14:58:27.243 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_time", 2024-04-23T14:58:27.243 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.244 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.244 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.244 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2359", 2024-04-23T14:58:27.244 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.244 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.244 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.244 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ending time of day to automatically balance", 2024-04-23T14:58:27.244 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-23T14:58:27.244 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.245 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.245 INFO:teuthology.orchestra.run.smithi028.stdout: "end_weekday": { 2024-04-23T14:58:27.245 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_weekday", 2024-04-23T14:58:27.245 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-23T14:58:27.245 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.245 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.245 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-23T14:58:27.245 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-04-23T14:58:27.246 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-04-23T14:58:27.246 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.246 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-23T14:58:27.246 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-23T14:58:27.246 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.246 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.246 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.247 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.247 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.247 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.247 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.247 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.247 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.247 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.247 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.247 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.247 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.248 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.248 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.248 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.248 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.248 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.248 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.248 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.248 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.248 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.249 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.249 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.249 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.249 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.249 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.249 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.249 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.249 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.250 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.250 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.250 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.250 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.250 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.250 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.250 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.250 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.250 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.251 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.251 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.251 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.251 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.251 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.251 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.251 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.251 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.251 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.252 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.252 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.252 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.252 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.252 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.252 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.252 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.253 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.253 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.253 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.253 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.253 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.253 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.253 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.253 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.253 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.254 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.254 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.254 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.254 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.254 INFO:teuthology.orchestra.run.smithi028.stdout: "min_score": { 2024-04-23T14:58:27.254 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_score", 2024-04-23T14:58:27.254 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-23T14:58:27.254 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.255 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.255 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-23T14:58:27.255 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.255 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.255 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.255 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-23T14:58:27.255 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.255 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.255 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.256 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-04-23T14:58:27.256 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-04-23T14:58:27.256 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.256 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.256 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.256 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "upmap", 2024-04-23T14:58:27.256 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.256 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.257 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.257 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-compat", 2024-04-23T14:58:27.257 INFO:teuthology.orchestra.run.smithi028.stdout: "none", 2024-04-23T14:58:27.257 INFO:teuthology.orchestra.run.smithi028.stdout: "read", 2024-04-23T14:58:27.257 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap", 2024-04-23T14:58:27.257 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap-read" 2024-04-23T14:58:27.257 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.257 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Balancer mode", 2024-04-23T14:58:27.257 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.258 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.258 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.258 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_ids": { 2024-04-23T14:58:27.258 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_ids", 2024-04-23T14:58:27.258 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.258 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.258 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.258 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.259 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.259 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.259 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.259 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-23T14:58:27.259 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.259 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.259 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.260 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-23T14:58:27.260 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-23T14:58:27.260 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:27.260 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.260 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.260 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-23T14:58:27.260 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.260 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.260 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.261 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-23T14:58:27.261 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.261 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.261 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.261 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_deviation": { 2024-04-23T14:58:27.261 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_deviation", 2024-04-23T14:58:27.261 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.261 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.261 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.262 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-23T14:58:27.262 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-23T14:58:27.262 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.262 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.262 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-23T14:58:27.262 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-23T14:58:27.262 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.262 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.263 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_optimizations": { 2024-04-23T14:58:27.263 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_optimizations", 2024-04-23T14:58:27.263 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-23T14:58:27.263 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.263 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.263 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-23T14:58:27.263 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.263 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.264 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.264 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-23T14:58:27.264 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.264 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.264 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.264 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.264 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.265 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm", 2024-04-23T14:58:27.265 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.265 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.265 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.265 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_down_multiplier": { 2024-04-23T14:58:27.265 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_down_multiplier", 2024-04-23T14:58:27.265 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-23T14:58:27.265 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.266 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.266 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-04-23T14:58:27.266 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.266 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.266 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.266 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-23T14:58:27.266 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.266 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.266 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.267 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_refresh_rate": { 2024-04-23T14:58:27.267 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_refresh_rate", 2024-04-23T14:58:27.267 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:27.267 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.267 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.267 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "20", 2024-04-23T14:58:27.267 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.267 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.268 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.268 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-23T14:58:27.268 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.268 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.268 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.268 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_starting_port": { 2024-04-23T14:58:27.268 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_starting_port", 2024-04-23T14:58:27.268 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.269 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.269 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.269 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4721", 2024-04-23T14:58:27.269 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.269 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.269 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.269 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-23T14:58:27.269 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.269 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.270 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.270 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_ptrace": { 2024-04-23T14:58:27.270 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_ptrace", 2024-04-23T14:58:27.270 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.270 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.270 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.270 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.270 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.271 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.271 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.271 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-23T14:58:27.271 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-23T14:58:27.271 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.271 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.271 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_interval": { 2024-04-23T14:58:27.272 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_interval", 2024-04-23T14:58:27.272 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:27.272 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.272 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.272 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-23T14:58:27.272 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.272 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.272 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.272 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-23T14:58:27.273 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.273 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.273 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.273 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_memory_target_ratio": { 2024-04-23T14:58:27.273 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_memory_target_ratio", 2024-04-23T14:58:27.273 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-23T14:58:27.273 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.273 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.274 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.7", 2024-04-23T14:58:27.274 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.274 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.274 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.274 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-23T14:58:27.274 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.274 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.274 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.275 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.275 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_log_destination": { 2024-04-23T14:58:27.275 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm_log_destination", 2024-04-23T14:58:27.275 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.275 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.275 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.275 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.275 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.275 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.276 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.276 INFO:teuthology.orchestra.run.smithi028.stdout: "file", 2024-04-23T14:58:27.276 INFO:teuthology.orchestra.run.smithi028.stdout: "file,syslog", 2024-04-23T14:58:27.276 INFO:teuthology.orchestra.run.smithi028.stdout: "syslog" 2024-04-23T14:58:27.276 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.276 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-23T14:58:27.276 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.276 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.277 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.277 INFO:teuthology.orchestra.run.smithi028.stdout: "cgroups_split": { 2024-04-23T14:58:27.277 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cgroups_split", 2024-04-23T14:58:27.277 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.277 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.277 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.277 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.278 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.278 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.278 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.278 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-23T14:58:27.278 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.278 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.278 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.278 INFO:teuthology.orchestra.run.smithi028.stdout: "config_checks_enabled": { 2024-04-23T14:58:27.279 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_checks_enabled", 2024-04-23T14:58:27.279 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.279 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.279 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.279 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.279 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.279 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.279 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.279 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-23T14:58:27.280 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.280 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.280 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.280 INFO:teuthology.orchestra.run.smithi028.stdout: "config_dashboard": { 2024-04-23T14:58:27.280 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_dashboard", 2024-04-23T14:58:27.280 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.280 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.280 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.281 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.281 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.281 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.281 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.281 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-23T14:58:27.281 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.281 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.281 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.282 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_alertmanager": { 2024-04-23T14:58:27.282 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_alertmanager", 2024-04-23T14:58:27.282 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.282 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.282 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.282 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-23T14:58:27.282 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.282 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.283 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.283 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-23T14:58:27.283 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.283 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.283 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.283 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_base": { 2024-04-23T14:58:27.283 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_base", 2024-04-23T14:58:27.284 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.284 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.284 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.284 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-23T14:58:27.284 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.284 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.284 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.284 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Container image name, without the tag", 2024-04-23T14:58:27.284 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.285 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.285 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.285 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_elasticsearch": { 2024-04-23T14:58:27.285 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_elasticsearch", 2024-04-23T14:58:27.285 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.285 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.285 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.285 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-23T14:58:27.286 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.286 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.286 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.286 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "elasticsearch container image", 2024-04-23T14:58:27.286 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.286 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.286 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.287 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_grafana": { 2024-04-23T14:58:27.287 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_grafana", 2024-04-23T14:58:27.287 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.287 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.287 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.287 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-23T14:58:27.287 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.287 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.288 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.288 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-23T14:58:27.288 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.288 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.288 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.288 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_haproxy": { 2024-04-23T14:58:27.288 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_haproxy", 2024-04-23T14:58:27.288 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.289 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.289 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.289 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-23T14:58:27.289 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.289 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.289 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.289 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "HAproxy container image", 2024-04-23T14:58:27.289 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.289 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.290 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.290 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_agent": { 2024-04-23T14:58:27.290 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_agent", 2024-04-23T14:58:27.290 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.290 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.290 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.290 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-23T14:58:27.291 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.291 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.291 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.291 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger agent container image", 2024-04-23T14:58:27.291 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.291 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.291 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.291 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.291 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_collector": { 2024-04-23T14:58:27.292 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_collector", 2024-04-23T14:58:27.292 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.292 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.292 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.292 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-23T14:58:27.292 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.292 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.292 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.293 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger collector container image", 2024-04-23T14:58:27.293 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.293 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.293 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.293 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_query": { 2024-04-23T14:58:27.293 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_query", 2024-04-23T14:58:27.293 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.293 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.294 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.294 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-23T14:58:27.294 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.294 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.294 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.294 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger query container image", 2024-04-23T14:58:27.294 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.294 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.295 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.295 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_keepalived": { 2024-04-23T14:58:27.295 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_keepalived", 2024-04-23T14:58:27.295 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.295 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.295 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.295 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-23T14:58:27.295 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.296 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.296 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.296 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Keepalived container image", 2024-04-23T14:58:27.296 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.296 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.296 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.296 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_loki": { 2024-04-23T14:58:27.296 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_loki", 2024-04-23T14:58:27.297 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.297 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.297 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.297 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-23T14:58:27.297 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.297 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.297 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.297 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Loki container image", 2024-04-23T14:58:27.297 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.298 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.298 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.298 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_node_exporter": { 2024-04-23T14:58:27.298 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_node_exporter", 2024-04-23T14:58:27.298 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.298 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.298 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.299 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-23T14:58:27.299 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.299 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.299 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.299 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-23T14:58:27.299 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.299 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.299 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.300 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nvmeof": { 2024-04-23T14:58:27.300 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nvmeof", 2024-04-23T14:58:27.300 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.300 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.300 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.300 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-23T14:58:27.300 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.300 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.300 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.301 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nvme-of container image", 2024-04-23T14:58:27.301 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.301 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.301 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.301 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_prometheus": { 2024-04-23T14:58:27.301 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_prometheus", 2024-04-23T14:58:27.301 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.301 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.302 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.302 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-23T14:58:27.302 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.302 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.302 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.302 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-23T14:58:27.302 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.302 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.303 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.303 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.303 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_promtail": { 2024-04-23T14:58:27.303 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_promtail", 2024-04-23T14:58:27.303 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.303 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.303 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.303 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-23T14:58:27.303 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.304 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.304 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.304 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Promtail container image", 2024-04-23T14:58:27.304 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.304 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.304 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.304 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_samba": { 2024-04-23T14:58:27.305 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_samba", 2024-04-23T14:58:27.305 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.305 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.305 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.305 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-23T14:58:27.305 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.305 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.305 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.305 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Samba/SMB container image", 2024-04-23T14:58:27.306 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.306 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.306 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.306 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_snmp_gateway": { 2024-04-23T14:58:27.306 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_snmp_gateway", 2024-04-23T14:58:27.306 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.306 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.306 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.307 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-23T14:58:27.307 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.307 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.307 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.307 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SNMP Gateway container image", 2024-04-23T14:58:27.307 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.307 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.307 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.308 INFO:teuthology.orchestra.run.smithi028.stdout: "container_init": { 2024-04-23T14:58:27.308 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_init", 2024-04-23T14:58:27.308 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.308 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.308 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.308 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.308 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.308 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.309 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.309 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Run podman/docker with `--init`", 2024-04-23T14:58:27.309 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.309 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.309 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.309 INFO:teuthology.orchestra.run.smithi028.stdout: "daemon_cache_timeout": { 2024-04-23T14:58:27.309 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "daemon_cache_timeout", 2024-04-23T14:58:27.309 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:27.310 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.310 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.310 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-23T14:58:27.310 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.310 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.310 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.310 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-23T14:58:27.310 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.310 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.311 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.311 INFO:teuthology.orchestra.run.smithi028.stdout: "default_cephadm_command_timeout": { 2024-04-23T14:58:27.311 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_cephadm_command_timeout", 2024-04-23T14:58:27.311 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.311 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.311 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.311 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "900", 2024-04-23T14:58:27.311 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.312 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.312 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.312 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-23T14:58:27.312 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.312 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.312 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.312 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.312 INFO:teuthology.orchestra.run.smithi028.stdout: "default_registry": { 2024-04-23T14:58:27.312 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_registry", 2024-04-23T14:58:27.313 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.313 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.313 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.313 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io", 2024-04-23T14:58:27.313 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.313 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.313 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.313 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-23T14:58:27.313 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.314 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.314 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.314 INFO:teuthology.orchestra.run.smithi028.stdout: "device_cache_timeout": { 2024-04-23T14:58:27.314 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_cache_timeout", 2024-04-23T14:58:27.314 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:27.314 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.314 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.315 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1800", 2024-04-23T14:58:27.315 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.315 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.315 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.315 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache device inventory", 2024-04-23T14:58:27.315 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.315 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.315 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.316 INFO:teuthology.orchestra.run.smithi028.stdout: "device_enhanced_scan": { 2024-04-23T14:58:27.316 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_enhanced_scan", 2024-04-23T14:58:27.316 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.316 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.316 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.316 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.316 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.316 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.316 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.316 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-23T14:58:27.317 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.317 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.317 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.317 INFO:teuthology.orchestra.run.smithi028.stdout: "facts_cache_timeout": { 2024-04-23T14:58:27.317 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "facts_cache_timeout", 2024-04-23T14:58:27.317 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:27.317 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.317 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.318 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-23T14:58:27.318 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.318 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.318 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.318 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache host facts data", 2024-04-23T14:58:27.318 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.318 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.318 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.318 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.319 INFO:teuthology.orchestra.run.smithi028.stdout: "grafana_dashboards_path": { 2024-04-23T14:58:27.319 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "grafana_dashboards_path", 2024-04-23T14:58:27.319 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.319 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.319 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.319 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-23T14:58:27.319 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.319 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.319 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.320 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-23T14:58:27.320 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.320 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.320 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.320 INFO:teuthology.orchestra.run.smithi028.stdout: "host_check_interval": { 2024-04-23T14:58:27.320 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "host_check_interval", 2024-04-23T14:58:27.320 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:27.320 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.320 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.321 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-23T14:58:27.321 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.321 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.321 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.321 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to perform a host check", 2024-04-23T14:58:27.321 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.321 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.321 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.322 INFO:teuthology.orchestra.run.smithi028.stdout: "hw_monitoring": { 2024-04-23T14:58:27.322 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hw_monitoring", 2024-04-23T14:58:27.322 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.322 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.322 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.322 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.322 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.322 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.323 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.323 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-23T14:58:27.323 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.323 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.323 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.323 INFO:teuthology.orchestra.run.smithi028.stdout: "inventory_list_all": { 2024-04-23T14:58:27.323 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "inventory_list_all", 2024-04-23T14:58:27.323 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.324 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.324 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.324 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.324 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.324 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.324 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.324 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-23T14:58:27.324 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.324 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.325 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.325 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.325 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.325 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.325 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.325 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.325 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.325 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.326 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.326 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.326 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.326 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.326 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.326 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.326 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.326 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.327 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.327 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.327 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.327 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.327 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.327 INFO:teuthology.orchestra.run.smithi028.stdout: "log_refresh_metadata": { 2024-04-23T14:58:27.327 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_refresh_metadata", 2024-04-23T14:58:27.327 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.328 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.328 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.328 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.328 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.328 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.328 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.328 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-23T14:58:27.328 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.329 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.329 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.329 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.329 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.329 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.329 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.329 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.329 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.330 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.330 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.330 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.330 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-23T14:58:27.330 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.330 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.330 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.331 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.331 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.331 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.331 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.331 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.331 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.331 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.331 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.331 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.332 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.332 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.332 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.332 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.332 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.332 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.332 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.332 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.332 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.333 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.333 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.333 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.333 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.333 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.333 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.333 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.333 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.334 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.334 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.334 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.334 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.334 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.334 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.334 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.335 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-23T14:58:27.335 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-23T14:58:27.335 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.335 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.335 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.335 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.335 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.335 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.335 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.336 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-23T14:58:27.336 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.336 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.336 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.336 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-23T14:58:27.336 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-23T14:58:27.336 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.336 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.337 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.337 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "*", 2024-04-23T14:58:27.337 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.337 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.337 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.337 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-23T14:58:27.337 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.337 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.337 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.337 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.338 INFO:teuthology.orchestra.run.smithi028.stdout: "max_count_per_host": { 2024-04-23T14:58:27.338 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_count_per_host", 2024-04-23T14:58:27.338 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.338 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.338 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.338 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-23T14:58:27.338 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.338 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.339 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.339 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of daemons per service per host", 2024-04-23T14:58:27.339 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.339 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.339 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.339 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd_draining_count": { 2024-04-23T14:58:27.339 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_osd_draining_count", 2024-04-23T14:58:27.339 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.339 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.340 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.340 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-23T14:58:27.340 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.340 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.340 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.340 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-23T14:58:27.340 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.340 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.340 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.341 INFO:teuthology.orchestra.run.smithi028.stdout: "migration_current": { 2024-04-23T14:58:27.341 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "migration_current", 2024-04-23T14:58:27.341 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.341 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.341 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.341 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.341 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.341 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.342 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.342 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "internal - do not modify", 2024-04-23T14:58:27.342 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.342 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.342 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.342 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.342 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-04-23T14:58:27.342 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-04-23T14:58:27.343 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.343 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.343 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.343 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "root", 2024-04-23T14:58:27.343 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.343 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.343 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.343 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm-package", 2024-04-23T14:58:27.343 INFO:teuthology.orchestra.run.smithi028.stdout: "root" 2024-04-23T14:58:27.344 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.344 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "mode for remote execution of cephadm", 2024-04-23T14:58:27.344 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.344 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.344 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.344 INFO:teuthology.orchestra.run.smithi028.stdout: "oob_default_addr": { 2024-04-23T14:58:27.344 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "oob_default_addr", 2024-04-23T14:58:27.344 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.345 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.345 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.345 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "169.254.1.1", 2024-04-23T14:58:27.345 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.345 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.345 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.345 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-23T14:58:27.345 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.346 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.346 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.346 INFO:teuthology.orchestra.run.smithi028.stdout: "prometheus_alerts_path": { 2024-04-23T14:58:27.346 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus_alerts_path", 2024-04-23T14:58:27.346 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.346 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.346 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.347 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-23T14:58:27.347 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.347 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.347 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.347 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-23T14:58:27.347 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.347 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.347 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.347 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_insecure": { 2024-04-23T14:58:27.348 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_insecure", 2024-04-23T14:58:27.348 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.348 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.348 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.348 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.348 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.348 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.348 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.349 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-23T14:58:27.349 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.349 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.349 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.349 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_password": { 2024-04-23T14:58:27.349 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_password", 2024-04-23T14:58:27.349 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.349 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.350 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.350 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.350 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.350 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.350 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.350 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-23T14:58:27.350 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.350 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.351 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.351 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_url": { 2024-04-23T14:58:27.351 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_url", 2024-04-23T14:58:27.351 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.351 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.351 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.351 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.351 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.352 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.352 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.352 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-23T14:58:27.352 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.352 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.352 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.352 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_username": { 2024-04-23T14:58:27.352 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_username", 2024-04-23T14:58:27.352 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.353 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.353 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.353 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.353 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.353 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.353 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.353 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-23T14:58:27.353 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.353 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.354 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.354 INFO:teuthology.orchestra.run.smithi028.stdout: "secure_monitoring_stack": { 2024-04-23T14:58:27.354 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secure_monitoring_stack", 2024-04-23T14:58:27.354 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.354 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.354 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.354 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.355 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.355 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.355 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.355 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-23T14:58:27.355 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.355 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.355 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.355 INFO:teuthology.orchestra.run.smithi028.stdout: "service_discovery_port": { 2024-04-23T14:58:27.355 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "service_discovery_port", 2024-04-23T14:58:27.356 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.356 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.356 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.356 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8765", 2024-04-23T14:58:27.356 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.356 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.356 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.356 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "cephadm service discovery port", 2024-04-23T14:58:27.356 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.357 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.357 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.357 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_config_file": { 2024-04-23T14:58:27.357 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_config_file", 2024-04-23T14:58:27.357 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.357 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.357 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.357 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.358 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.358 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.358 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.358 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-23T14:58:27.358 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.358 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.358 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.359 INFO:teuthology.orchestra.run.smithi028.stdout: "use_agent": { 2024-04-23T14:58:27.359 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_agent", 2024-04-23T14:58:27.359 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.359 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.359 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.359 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.359 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.359 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.359 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.360 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-23T14:58:27.360 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.360 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.360 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.360 INFO:teuthology.orchestra.run.smithi028.stdout: "use_repo_digest": { 2024-04-23T14:58:27.360 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_repo_digest", 2024-04-23T14:58:27.360 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.360 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.360 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.361 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.361 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.361 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.361 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.361 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-23T14:58:27.361 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.361 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.361 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.362 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_failed_host_check": { 2024-04-23T14:58:27.362 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_failed_host_check", 2024-04-23T14:58:27.362 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.362 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.362 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.362 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.362 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.362 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.363 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.363 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if the host check fails", 2024-04-23T14:58:27.363 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.363 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.363 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.363 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_daemons": { 2024-04-23T14:58:27.363 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_daemons", 2024-04-23T14:58:27.363 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.364 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.364 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.364 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.364 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.364 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.364 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.364 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-23T14:58:27.364 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.365 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.365 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.365 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_hosts": { 2024-04-23T14:58:27.365 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_hosts", 2024-04-23T14:58:27.365 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.365 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.365 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.365 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.366 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.366 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.366 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.366 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-23T14:58:27.366 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.366 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.366 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.366 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.367 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.367 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.367 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crash", 2024-04-23T14:58:27.367 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.367 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.367 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.367 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.367 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.368 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.368 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.368 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.368 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.368 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.368 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.368 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.368 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.368 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.369 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.369 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.369 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.369 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.369 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.369 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.369 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.369 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.369 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.370 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.370 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.370 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.370 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.370 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.370 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.370 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.370 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.370 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.371 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.371 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.371 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.371 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.371 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.371 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.371 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.371 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.372 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.372 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.372 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.372 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.372 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.372 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.372 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.372 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.372 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.372 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.373 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.373 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.373 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.373 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.373 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.373 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.373 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.373 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.374 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.374 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.374 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.374 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.374 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.374 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.374 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.374 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.374 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.375 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.375 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.375 INFO:teuthology.orchestra.run.smithi028.stdout: "retain_interval": { 2024-04-23T14:58:27.375 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retain_interval", 2024-04-23T14:58:27.375 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:27.375 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.375 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.376 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "31536000", 2024-04-23T14:58:27.376 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.376 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.376 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.376 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-23T14:58:27.376 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.376 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.376 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.377 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_recent_interval": { 2024-04-23T14:58:27.377 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_recent_interval", 2024-04-23T14:58:27.377 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:27.377 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.377 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.377 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1209600", 2024-04-23T14:58:27.377 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.377 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.377 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.378 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-23T14:58:27.378 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.378 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.378 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.378 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.378 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.378 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dashboard", 2024-04-23T14:58:27.379 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.379 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.379 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.379 INFO:teuthology.orchestra.run.smithi028.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-23T14:58:27.379 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-23T14:58:27.379 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.379 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.379 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.379 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-23T14:58:27.380 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.380 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.380 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.380 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.380 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.380 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.380 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.380 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_HOST": { 2024-04-23T14:58:27.380 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-23T14:58:27.381 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.381 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.381 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.381 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.381 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.381 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.381 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.381 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.381 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.382 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.382 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.382 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-23T14:58:27.382 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-23T14:58:27.382 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.382 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.382 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.382 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.383 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.383 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.383 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.383 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.383 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.383 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.383 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.384 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_ENABLED": { 2024-04-23T14:58:27.384 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_ENABLED", 2024-04-23T14:58:27.384 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.384 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.384 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.384 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.384 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.384 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.384 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.385 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.385 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.385 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.385 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.385 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-23T14:58:27.385 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-23T14:58:27.385 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.385 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.385 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.386 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.386 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.386 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.386 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.386 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.386 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.386 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.386 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.387 INFO:teuthology.orchestra.run.smithi028.stdout: "ENABLE_BROWSABLE_API": { 2024-04-23T14:58:27.387 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-23T14:58:27.387 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.387 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.387 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.387 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.387 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.387 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.388 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.388 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.388 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.388 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.388 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.388 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-23T14:58:27.388 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-23T14:58:27.388 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.389 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.389 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.389 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.389 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.389 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.389 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.389 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.389 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.390 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.390 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.390 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-23T14:58:27.390 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-23T14:58:27.390 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.390 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.390 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.390 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.391 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.391 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.391 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.391 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.391 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.391 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.391 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.391 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-23T14:58:27.392 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-23T14:58:27.392 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.392 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.392 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.392 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.392 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.392 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.392 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.392 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.393 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.393 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.393 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.393 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-23T14:58:27.393 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-23T14:58:27.393 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.393 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.393 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.394 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.394 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.394 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.394 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.394 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.394 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.394 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.394 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.395 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-23T14:58:27.395 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-23T14:58:27.395 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.395 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.395 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.395 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.395 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.395 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.395 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.396 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.396 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.396 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.396 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.396 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-23T14:58:27.396 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-23T14:58:27.396 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.396 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.397 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.397 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.397 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.397 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.397 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.397 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.397 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.397 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.398 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.398 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-23T14:58:27.398 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-23T14:58:27.398 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.398 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.398 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.398 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.399 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.399 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.399 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.399 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.399 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.399 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.399 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.399 INFO:teuthology.orchestra.run.smithi028.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-23T14:58:27.400 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-23T14:58:27.400 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.400 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.400 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.400 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.400 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.400 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.400 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.400 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.401 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.401 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.401 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.401 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_PASSWORD": { 2024-04-23T14:58:27.401 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-23T14:58:27.401 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.401 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.401 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.402 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-23T14:58:27.402 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.402 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.402 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.402 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.402 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.402 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.402 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.403 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-23T14:58:27.403 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-23T14:58:27.403 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.403 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.403 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.403 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.403 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.403 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.404 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.404 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.404 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.404 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.404 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.404 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_URL": { 2024-04-23T14:58:27.404 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_URL", 2024-04-23T14:58:27.404 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.405 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.405 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.405 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.405 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.405 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.405 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.405 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.405 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.405 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.406 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.406 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_USERNAME": { 2024-04-23T14:58:27.406 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-23T14:58:27.406 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.406 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.406 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.406 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-23T14:58:27.407 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.407 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.407 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.407 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.407 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.407 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.407 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.407 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-23T14:58:27.408 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-23T14:58:27.408 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.408 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.408 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.408 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.408 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.408 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.408 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.408 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.409 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.409 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.409 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.409 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-23T14:58:27.409 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-23T14:58:27.409 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.409 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.409 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.410 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.410 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.410 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.410 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.410 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.410 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.410 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.410 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.411 INFO:teuthology.orchestra.run.smithi028.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-23T14:58:27.411 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-23T14:58:27.411 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.411 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.411 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.411 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.411 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.411 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.412 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.412 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.412 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.412 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.412 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.412 INFO:teuthology.orchestra.run.smithi028.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-23T14:58:27.412 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-23T14:58:27.412 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.413 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.413 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.413 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.413 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.413 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.413 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.413 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.413 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.413 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.414 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.414 INFO:teuthology.orchestra.run.smithi028.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-23T14:58:27.414 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-23T14:58:27.414 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.414 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.414 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.414 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "[]", 2024-04-23T14:58:27.414 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.415 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.415 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.415 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.415 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.415 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.415 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.415 INFO:teuthology.orchestra.run.smithi028.stdout: "MULTICLUSTER_CONFIG": { 2024-04-23T14:58:27.416 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-23T14:58:27.416 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.416 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.416 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.416 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "{}", 2024-04-23T14:58:27.416 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.416 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.416 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.416 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.417 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.417 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.417 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.417 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_HOST": { 2024-04-23T14:58:27.417 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-23T14:58:27.417 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.417 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.417 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.418 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.418 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.418 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.418 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.418 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.418 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.418 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.418 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.419 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-23T14:58:27.419 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-23T14:58:27.419 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.419 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.419 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.419 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.419 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.419 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.420 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.420 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.420 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.420 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.420 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.420 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-23T14:58:27.420 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-23T14:58:27.420 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.421 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.421 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.421 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.421 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.421 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.421 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.421 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.421 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.421 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.422 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.422 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-23T14:58:27.422 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-23T14:58:27.422 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.422 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.422 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.422 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.422 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.423 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.423 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.423 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.423 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.423 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.423 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.423 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-23T14:58:27.423 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-23T14:58:27.424 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.424 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.424 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.424 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.424 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.424 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.424 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.424 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.424 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.425 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.425 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.425 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-23T14:58:27.425 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-23T14:58:27.425 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.425 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.425 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.425 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.426 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.426 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.426 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.426 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.426 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.426 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.426 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.427 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-23T14:58:27.427 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-23T14:58:27.427 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.427 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.427 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.427 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.427 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.427 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.428 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.428 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.428 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.428 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.428 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.428 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-23T14:58:27.428 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-23T14:58:27.428 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.429 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.429 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.429 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.429 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.429 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.429 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.429 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.429 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.429 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.430 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.430 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-23T14:58:27.430 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-23T14:58:27.430 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.430 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.430 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.430 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.431 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.431 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.431 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.431 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.431 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.431 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.431 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.431 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_ENABLED": { 2024-04-23T14:58:27.432 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-23T14:58:27.432 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.432 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.432 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.432 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.432 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.432 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.432 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.432 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.433 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.433 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.433 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.433 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-23T14:58:27.433 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-23T14:58:27.433 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.433 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.433 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.434 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-23T14:58:27.434 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.434 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.434 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.434 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.434 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.434 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.434 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.435 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-23T14:58:27.435 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-23T14:58:27.435 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.435 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.435 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.435 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-23T14:58:27.435 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.436 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.436 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.436 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.436 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.436 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.436 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.436 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-23T14:58:27.437 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-23T14:58:27.437 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.437 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.437 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.437 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8", 2024-04-23T14:58:27.437 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.437 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.437 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.437 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.438 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.438 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.438 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.438 INFO:teuthology.orchestra.run.smithi028.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-23T14:58:27.438 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-23T14:58:27.438 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.438 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.438 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.439 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "45", 2024-04-23T14:58:27.439 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.439 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.439 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.439 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.439 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.439 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.439 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.440 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ACCESS_KEY": { 2024-04-23T14:58:27.440 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-23T14:58:27.440 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.440 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.440 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.440 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.440 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.440 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.441 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.441 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.441 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.441 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.441 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.441 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-23T14:58:27.441 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-23T14:58:27.441 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.442 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.442 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.442 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-23T14:58:27.442 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.442 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.442 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.442 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.442 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.443 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.443 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.443 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SECRET_KEY": { 2024-04-23T14:58:27.443 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-23T14:58:27.443 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.443 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.443 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.443 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.444 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.444 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.444 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.444 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.444 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.444 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.444 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.445 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SSL_VERIFY": { 2024-04-23T14:58:27.445 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-23T14:58:27.445 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.445 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.445 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.445 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.445 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.445 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.446 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.446 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.446 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.446 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.446 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.446 INFO:teuthology.orchestra.run.smithi028.stdout: "UNSAFE_TLS_v1_2": { 2024-04-23T14:58:27.446 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-23T14:58:27.446 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.447 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.447 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.447 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.447 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.447 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.447 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.447 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.447 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.448 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.448 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.448 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-23T14:58:27.448 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-23T14:58:27.448 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.448 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.448 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.448 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-23T14:58:27.449 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.449 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.449 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.449 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.449 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.449 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.449 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.449 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-23T14:58:27.450 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-23T14:58:27.450 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.450 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.450 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.450 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-23T14:58:27.450 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.450 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.450 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.451 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.451 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.451 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.451 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.451 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-23T14:58:27.451 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-23T14:58:27.451 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.451 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.452 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.452 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-23T14:58:27.452 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.452 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.452 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.452 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.452 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.452 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.452 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.453 INFO:teuthology.orchestra.run.smithi028.stdout: "cross_origin_url": { 2024-04-23T14:58:27.453 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cross_origin_url", 2024-04-23T14:58:27.453 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.453 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.453 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.453 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.453 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.453 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.454 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.454 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.454 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.454 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.454 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.454 INFO:teuthology.orchestra.run.smithi028.stdout: "crt_file": { 2024-04-23T14:58:27.454 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crt_file", 2024-04-23T14:58:27.455 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.455 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.455 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.455 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.455 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.455 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.455 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.455 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.455 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.455 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.456 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.456 INFO:teuthology.orchestra.run.smithi028.stdout: "debug": { 2024-04-23T14:58:27.456 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "debug", 2024-04-23T14:58:27.456 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.456 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.456 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.456 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.456 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.457 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.457 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.457 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable/disable debug options", 2024-04-23T14:58:27.457 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.457 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.457 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.457 INFO:teuthology.orchestra.run.smithi028.stdout: "jwt_token_ttl": { 2024-04-23T14:58:27.457 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "jwt_token_ttl", 2024-04-23T14:58:27.458 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.458 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.458 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.458 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "28800", 2024-04-23T14:58:27.458 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.458 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.458 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.458 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.459 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.459 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.459 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.459 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-04-23T14:58:27.459 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-04-23T14:58:27.459 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.459 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.460 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.460 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.460 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.460 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.460 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.460 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.460 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.460 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.460 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.461 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.461 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.461 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.461 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.461 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.461 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.461 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.461 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.462 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.462 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.462 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.462 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.462 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.462 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.462 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.462 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.463 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.463 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.463 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.463 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.463 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.463 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.463 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.463 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.464 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.464 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.464 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.464 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.464 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.464 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.464 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.464 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.464 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.465 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.465 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.465 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.465 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.465 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.465 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.465 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.465 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.466 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.466 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.466 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.466 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.466 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.466 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.466 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.466 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.469 INFO:teuthology.orchestra.run.smithi028.stdout: "motd": { 2024-04-23T14:58:27.469 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "motd", 2024-04-23T14:58:27.469 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.469 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.469 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.469 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.469 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.469 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.470 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.470 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "The message of the day", 2024-04-23T14:58:27.470 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.470 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.470 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.470 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect_resolve_ip_addr": { 2024-04-23T14:58:27.470 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "redirect_resolve_ip_addr", 2024-04-23T14:58:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.471 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-23T14:58:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-23T14:58:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.472 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-04-23T14:58:27.473 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.473 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.473 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.473 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.473 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.473 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.473 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.474 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-23T14:58:27.474 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-23T14:58:27.474 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.474 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.474 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.474 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8080", 2024-04-23T14:58:27.474 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.474 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-04-23T14:58:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-04-23T14:58:27.475 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.476 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl_server_port": { 2024-04-23T14:58:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl_server_port", 2024-04-23T14:58:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.477 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8443", 2024-04-23T14:58:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.478 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-04-23T14:58:27.479 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-04-23T14:58:27.479 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.479 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.479 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.479 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "redirect", 2024-04-23T14:58:27.479 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.479 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.479 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.480 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.480 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect" 2024-04-23T14:58:27.480 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.480 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.480 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.480 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.480 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.480 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-04-23T14:58:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-04-23T14:58:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-04-23T14:58:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-04-23T14:58:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-04-23T14:58:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: "url_prefix": { 2024-04-23T14:58:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url_prefix", 2024-04-23T14:58:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-04-23T14:58:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_monitoring": { 2024-04-23T14:58:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_monitoring", 2024-04-23T14:58:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "monitor device health metrics", 2024-04-23T14:58:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.493 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: "mark_out_threshold": { 2024-04-23T14:58:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mark_out_threshold", 2024-04-23T14:58:27.494 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2419200", 2024-04-23T14:58:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-23T14:58:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.495 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": { 2024-04-23T14:58:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_name", 2024-04-23T14:58:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "device_health_metrics", 2024-04-23T14:58:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.496 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-23T14:58:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: "retention_period": { 2024-04-23T14:58:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retention_period", 2024-04-23T14:58:27.497 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15552000", 2024-04-23T14:58:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain device health metrics", 2024-04-23T14:58:27.498 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.499 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.499 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.499 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_frequency": { 2024-04-23T14:58:27.499 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_frequency", 2024-04-23T14:58:27.499 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:27.499 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.499 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.499 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-04-23T14:58:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-23T14:58:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.500 INFO:teuthology.orchestra.run.smithi028.stdout: "self_heal": { 2024-04-23T14:58:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "self_heal", 2024-04-23T14:58:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.501 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-23T14:58:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-23T14:58:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-23T14:58:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.502 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.503 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-23T14:58:27.503 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.503 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.503 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.503 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and check device health", 2024-04-23T14:58:27.503 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.503 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.503 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.504 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.504 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_threshold": { 2024-04-23T14:58:27.504 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_threshold", 2024-04-23T14:58:27.504 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:27.504 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.504 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.504 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7257600", 2024-04-23T14:58:27.504 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.505 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.505 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.505 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-23T14:58:27.505 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.505 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.505 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.505 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.505 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.506 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.506 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.506 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "diskprediction_local", 2024-04-23T14:58:27.506 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.506 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.506 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.506 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.506 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.507 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.507 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.507 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.507 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.507 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.507 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.507 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.507 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.508 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.508 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.508 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.508 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.508 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.508 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.508 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.508 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.509 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.509 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.509 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.509 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.509 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.509 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.509 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.509 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.509 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.510 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.510 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.510 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.510 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.510 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.510 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.510 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.510 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.511 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.511 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.511 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.511 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.511 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.511 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.511 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.511 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.511 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.512 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.512 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.512 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.512 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.512 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.512 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.512 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.512 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.512 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.513 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.513 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.513 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.513 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.513 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.513 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.513 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.514 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.514 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.514 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.514 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.514 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.514 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.514 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.514 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.515 INFO:teuthology.orchestra.run.smithi028.stdout: "predict_interval": { 2024-04-23T14:58:27.515 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predict_interval", 2024-04-23T14:58:27.515 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.515 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.515 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.515 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-04-23T14:58:27.515 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.515 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.516 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.516 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.516 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.516 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.516 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.516 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.516 INFO:teuthology.orchestra.run.smithi028.stdout: "predictor_model": { 2024-04-23T14:58:27.516 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predictor_model", 2024-04-23T14:58:27.516 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.517 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.517 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.517 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "prophetstor", 2024-04-23T14:58:27.517 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.517 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.517 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.517 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.517 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.517 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.518 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.518 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-23T14:58:27.518 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-23T14:58:27.518 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.518 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.518 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.518 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-23T14:58:27.519 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.519 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.519 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.519 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.519 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.519 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.519 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.519 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.519 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.520 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.520 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "influx", 2024-04-23T14:58:27.520 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": false, 2024-04-23T14:58:27.520 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "influxdb python module not found", 2024-04-23T14:58:27.520 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.520 INFO:teuthology.orchestra.run.smithi028.stdout: "batch_size": { 2024-04-23T14:58:27.520 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "batch_size", 2024-04-23T14:58:27.521 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.521 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.521 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.521 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5000", 2024-04-23T14:58:27.521 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.521 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.521 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.521 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-23T14:58:27.521 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.522 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.522 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.522 INFO:teuthology.orchestra.run.smithi028.stdout: "database": { 2024-04-23T14:58:27.522 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "database", 2024-04-23T14:58:27.522 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.522 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.522 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.523 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "ceph", 2024-04-23T14:58:27.523 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.523 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.523 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.523 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-23T14:58:27.523 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.523 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.523 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.523 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.524 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": { 2024-04-23T14:58:27.524 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hostname", 2024-04-23T14:58:27.524 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.524 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.524 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.524 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.524 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.524 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.524 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.524 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server hostname", 2024-04-23T14:58:27.525 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.525 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.525 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.525 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.525 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-23T14:58:27.525 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-23T14:58:27.525 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:27.525 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.526 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.526 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "30", 2024-04-23T14:58:27.526 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "5", 2024-04-23T14:58:27.526 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.526 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.526 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-23T14:58:27.526 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.526 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.527 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.527 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.527 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.527 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.527 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.527 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.527 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.527 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.527 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.528 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.528 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.528 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.528 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.528 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.528 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.528 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.528 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.528 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.529 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.529 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.529 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.529 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.529 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.529 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.529 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.529 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.529 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.530 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.530 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.530 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.530 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.530 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.530 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.530 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.530 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.531 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.531 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.531 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.531 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.531 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.531 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.531 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.532 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.532 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.532 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.532 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.532 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.532 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.532 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.532 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.532 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.533 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.533 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.533 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.533 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.533 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.533 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.533 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.533 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.534 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.534 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.534 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.534 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.534 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.534 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.534 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.534 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.535 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.535 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.535 INFO:teuthology.orchestra.run.smithi028.stdout: "password": { 2024-04-23T14:58:27.535 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "password", 2024-04-23T14:58:27.535 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.535 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.535 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.536 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.536 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.536 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.536 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.536 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "password of InfluxDB server user", 2024-04-23T14:58:27.536 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.536 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.536 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.537 INFO:teuthology.orchestra.run.smithi028.stdout: "port": { 2024-04-23T14:58:27.537 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "port", 2024-04-23T14:58:27.537 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.537 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.537 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.537 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8086", 2024-04-23T14:58:27.537 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.537 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.538 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.538 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server port", 2024-04-23T14:58:27.538 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.538 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.538 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.538 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-04-23T14:58:27.538 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-04-23T14:58:27.538 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.539 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.539 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.539 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "false", 2024-04-23T14:58:27.539 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.539 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.539 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.539 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-23T14:58:27.539 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.539 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.540 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.540 INFO:teuthology.orchestra.run.smithi028.stdout: "threads": { 2024-04-23T14:58:27.540 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threads", 2024-04-23T14:58:27.540 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.540 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.540 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.540 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-23T14:58:27.540 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-23T14:58:27.541 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "32", 2024-04-23T14:58:27.541 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.541 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-23T14:58:27.541 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.541 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.541 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.541 INFO:teuthology.orchestra.run.smithi028.stdout: "username": { 2024-04-23T14:58:27.541 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "username", 2024-04-23T14:58:27.542 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.542 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.542 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.542 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.542 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.542 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.542 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.542 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "username of InfluxDB server user", 2024-04-23T14:58:27.542 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.543 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.543 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.543 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.543 INFO:teuthology.orchestra.run.smithi028.stdout: "verify_ssl": { 2024-04-23T14:58:27.543 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "verify_ssl", 2024-04-23T14:58:27.543 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.543 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.543 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.544 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "true", 2024-04-23T14:58:27.544 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.544 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.544 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.544 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-23T14:58:27.544 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.544 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.544 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.544 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.545 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.545 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.545 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "insights", 2024-04-23T14:58:27.545 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.545 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.545 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.545 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.546 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.546 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.546 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.546 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.546 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.546 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.546 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.546 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.547 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.547 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.547 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.547 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.547 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.547 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.547 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.547 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.547 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.548 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.548 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.548 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.548 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.548 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.548 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.548 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.548 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.548 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.548 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.549 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.549 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.549 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.549 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.549 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.549 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.549 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.549 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.549 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.550 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.550 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.550 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.550 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.550 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.550 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.550 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.550 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.551 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.551 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.551 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.551 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.551 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.551 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.551 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.551 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.551 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.552 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.552 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.552 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.552 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.552 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.552 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.552 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.552 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.552 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.553 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.553 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.553 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.553 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.553 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.553 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.553 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.554 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.554 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "iostat", 2024-04-23T14:58:27.554 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.554 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.554 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.554 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.554 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.554 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.554 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.555 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.555 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.555 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.555 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.555 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.555 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.555 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.555 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.555 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.555 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.556 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.556 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.556 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.556 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.556 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.556 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.556 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.556 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.556 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.557 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.557 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.557 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.557 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.557 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.557 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.557 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.557 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.557 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.558 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.558 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.558 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.558 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.558 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.558 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.558 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.558 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.559 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.559 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.559 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.559 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.559 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.559 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.559 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.559 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.559 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.559 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.560 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.560 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.560 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.560 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.560 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.560 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.560 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.560 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.561 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.561 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.561 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.561 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.561 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.561 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.561 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.561 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.562 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.562 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.562 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.562 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.562 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.562 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.562 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "k8sevents", 2024-04-23T14:58:27.562 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.563 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.563 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.563 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_event_retention_days": { 2024-04-23T14:58:27.563 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ceph_event_retention_days", 2024-04-23T14:58:27.563 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.563 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.563 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.563 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7", 2024-04-23T14:58:27.563 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.564 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.564 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.564 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-23T14:58:27.564 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.564 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.564 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.564 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.564 INFO:teuthology.orchestra.run.smithi028.stdout: "config_check_secs": { 2024-04-23T14:58:27.564 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_check_secs", 2024-04-23T14:58:27.565 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.565 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.565 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.565 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-23T14:58:27.565 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "10", 2024-04-23T14:58:27.565 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.565 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.565 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-23T14:58:27.565 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.566 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.566 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.566 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.566 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.566 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.566 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.566 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.566 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.566 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.567 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.567 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.567 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.567 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.567 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.567 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.567 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.567 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.567 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.568 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.568 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.568 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.568 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.568 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.568 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.568 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.568 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.568 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.568 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.569 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.569 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.569 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.569 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.569 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.569 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.569 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.569 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.569 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.570 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.570 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.570 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.570 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.570 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.570 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.570 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.570 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.570 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.571 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.571 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.571 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.571 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.571 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.571 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.571 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.571 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.571 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.572 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.572 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.572 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.572 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.572 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.572 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.572 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.572 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.572 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.573 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.573 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.573 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.573 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.573 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.573 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.573 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.573 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.573 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.573 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.574 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.574 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "localpool", 2024-04-23T14:58:27.574 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.574 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.574 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.574 INFO:teuthology.orchestra.run.smithi028.stdout: "failure_domain": { 2024-04-23T14:58:27.574 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "failure_domain", 2024-04-23T14:58:27.575 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.575 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.575 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.575 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "host", 2024-04-23T14:58:27.575 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.575 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.575 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.575 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "failure domain for any created local pool", 2024-04-23T14:58:27.575 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-23T14:58:27.576 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.576 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.576 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.576 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.576 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.576 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.576 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.576 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.577 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.577 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.577 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.577 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.577 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.577 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.577 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.577 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.578 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.578 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.578 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.578 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.578 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.578 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.578 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.579 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.579 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.579 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.579 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.579 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.579 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.579 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.579 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.580 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.580 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.580 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.580 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.580 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.580 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.580 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.580 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.581 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.581 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.581 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.581 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.581 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.581 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.581 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.581 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.582 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.582 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.582 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.582 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.582 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.582 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.582 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.582 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.583 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.583 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.583 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.583 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.583 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.583 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.583 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.583 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.584 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.584 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.584 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.584 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.584 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.584 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.584 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.584 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": { 2024-04-23T14:58:27.585 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_size", 2024-04-23T14:58:27.585 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.585 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.585 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.585 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.585 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.585 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.585 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.586 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default min_size for any created local pool", 2024-04-23T14:58:27.586 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-23T14:58:27.586 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.586 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.586 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.586 INFO:teuthology.orchestra.run.smithi028.stdout: "num_rep": { 2024-04-23T14:58:27.586 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "num_rep", 2024-04-23T14:58:27.586 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.586 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.587 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.587 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3", 2024-04-23T14:58:27.587 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.587 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.587 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.587 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default replica count for any created local pool", 2024-04-23T14:58:27.587 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.587 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.588 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.588 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": { 2024-04-23T14:58:27.588 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_num", 2024-04-23T14:58:27.588 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.588 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.588 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.588 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "128", 2024-04-23T14:58:27.588 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.589 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.589 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.589 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default pg_num for any created local pool", 2024-04-23T14:58:27.589 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.589 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.589 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.589 INFO:teuthology.orchestra.run.smithi028.stdout: "prefix": { 2024-04-23T14:58:27.590 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prefix", 2024-04-23T14:58:27.590 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.590 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.590 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.590 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.590 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.590 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.590 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.591 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name prefix for any created local pool", 2024-04-23T14:58:27.591 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.591 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.591 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.591 INFO:teuthology.orchestra.run.smithi028.stdout: "subtree": { 2024-04-23T14:58:27.591 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "subtree", 2024-04-23T14:58:27.591 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.591 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.592 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.592 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "rack", 2024-04-23T14:58:27.592 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.592 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.592 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.592 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-23T14:58:27.592 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-23T14:58:27.592 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.592 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.593 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.593 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.593 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.593 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mds_autoscaler", 2024-04-23T14:58:27.593 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.593 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.593 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.594 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.594 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.594 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.594 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.594 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.594 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.594 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.594 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.594 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.595 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.595 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.595 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.595 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.595 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.595 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.595 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.595 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.596 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.596 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.596 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.596 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.596 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.596 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.596 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.596 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.596 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.597 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.597 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.597 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.597 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.597 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.597 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.597 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.597 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.598 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.598 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.598 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.598 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.598 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.598 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.598 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.598 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.599 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.599 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.599 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.599 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.599 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.599 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.599 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.599 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.599 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.599 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.600 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.600 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.600 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.600 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.600 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.600 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.600 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.600 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.601 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.601 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.601 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.601 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.601 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.601 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.601 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.601 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.602 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.602 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirroring", 2024-04-23T14:58:27.602 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.602 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.602 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.602 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.602 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.602 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.603 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.603 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.603 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.603 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.603 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.603 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.603 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.603 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.603 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.603 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.604 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.604 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.604 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.604 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.604 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.604 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.604 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.604 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.605 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.605 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.605 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.605 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.605 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.605 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.605 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.605 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.605 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.606 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.606 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.606 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.606 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.606 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.606 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.606 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.606 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.606 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.607 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.607 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.607 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.607 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.607 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.607 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.607 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.607 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.607 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.608 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.608 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.608 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.608 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.608 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.608 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.608 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.608 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.609 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.609 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.609 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.609 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.609 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.609 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.609 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.609 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.609 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.610 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.610 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.610 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.610 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.610 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.610 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "nfs", 2024-04-23T14:58:27.610 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.610 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.611 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.611 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.611 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.611 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.611 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.611 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.611 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.611 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.611 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.612 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.612 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.612 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.612 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.612 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.612 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.612 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.612 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.612 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.613 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.613 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.613 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.613 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.613 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.613 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.613 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.613 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.614 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.614 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.614 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.614 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.614 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.614 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.614 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.614 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.615 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.615 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.615 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.615 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.615 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.615 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.615 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.615 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.615 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.616 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.616 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.616 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.616 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.616 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.616 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.616 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.616 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.617 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.617 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.617 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.617 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.617 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.617 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.617 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.617 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.618 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.618 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.618 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.618 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.618 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.618 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.618 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.618 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.618 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.619 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.619 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-04-23T14:58:27.619 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.619 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.619 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.619 INFO:teuthology.orchestra.run.smithi028.stdout: "fail_fs": { 2024-04-23T14:58:27.619 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "fail_fs", 2024-04-23T14:58:27.620 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.620 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.620 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.620 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.620 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.620 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.620 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.620 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-23T14:58:27.620 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.621 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.621 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.621 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.621 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.621 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.621 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.621 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.621 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.621 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.622 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.622 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.622 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.622 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.622 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.622 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.622 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.622 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.623 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.623 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.623 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.623 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.623 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.623 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.623 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.623 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.624 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.624 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.624 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.624 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.624 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.624 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.624 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.624 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.624 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.625 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.625 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.625 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.625 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.625 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.625 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.625 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.625 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.626 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.626 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.626 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.626 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.626 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.626 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.626 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.626 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.626 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.627 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.627 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.627 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.627 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.627 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.627 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.627 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.627 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.628 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.628 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.628 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.628 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.628 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.628 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.628 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.628 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.628 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.629 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator": { 2024-04-23T14:58:27.629 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-04-23T14:58:27.629 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.629 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.629 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.629 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.629 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.629 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.630 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.630 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-04-23T14:58:27.630 INFO:teuthology.orchestra.run.smithi028.stdout: "rook", 2024-04-23T14:58:27.630 INFO:teuthology.orchestra.run.smithi028.stdout: "test_orchestrator" 2024-04-23T14:58:27.630 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.630 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Orchestrator backend", 2024-04-23T14:58:27.630 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.630 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.630 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.631 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.631 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.631 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_perf_query", 2024-04-23T14:58:27.631 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.631 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.631 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.631 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.632 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.632 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.632 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.632 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.632 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.632 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.632 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.632 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.632 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.633 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.633 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.633 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.633 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.633 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.633 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.633 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.633 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.633 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.634 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.634 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.634 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.634 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.634 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.634 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.634 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.634 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.635 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.635 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.635 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.635 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.635 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.635 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.635 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.636 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.636 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.636 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.636 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.636 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.636 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.636 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.636 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.636 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.637 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.637 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.637 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.637 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.637 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.637 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.637 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.637 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.637 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.638 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.638 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.638 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.638 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.638 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.638 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.638 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.639 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.639 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.639 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.639 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.639 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.639 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.639 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.639 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.639 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.640 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_support", 2024-04-23T14:58:27.640 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.640 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.640 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.640 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.640 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.640 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.641 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.641 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.641 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.641 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.641 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.641 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.641 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.641 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.641 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.642 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.642 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.642 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.642 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.642 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.642 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.642 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.642 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.643 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.643 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.643 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.643 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.643 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.643 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.643 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.643 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.644 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.644 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.644 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.644 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.644 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.644 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.644 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.644 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.645 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.645 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.645 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.645 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.645 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.645 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.645 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.645 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.645 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.646 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.646 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.646 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.646 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.646 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.646 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.646 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.646 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.647 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.647 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.647 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.647 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.647 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.647 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.647 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.647 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.648 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.648 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.648 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.648 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.648 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.648 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.648 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.649 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_autoscaler", 2024-04-23T14:58:27.649 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.649 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.649 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.649 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.649 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.649 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.649 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.650 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.650 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.650 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.650 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.650 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.650 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.650 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.650 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.650 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.651 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.651 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.651 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.651 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.651 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.651 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.651 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.651 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.651 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.652 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.652 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.652 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.652 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.652 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.652 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.652 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.652 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.652 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.653 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.653 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.653 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.653 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.653 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.653 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.653 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.653 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.654 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.654 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.654 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.654 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.654 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.654 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.654 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.654 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.655 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.655 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.655 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.655 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.655 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.655 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.655 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.655 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.655 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.656 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.656 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.656 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.656 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.656 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.656 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.656 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.656 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.656 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.657 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.657 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-23T14:58:27.657 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-23T14:58:27.657 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:27.657 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.657 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.657 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-23T14:58:27.657 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.658 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.658 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.658 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.658 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.658 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.658 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.658 INFO:teuthology.orchestra.run.smithi028.stdout: "threshold": { 2024-04-23T14:58:27.659 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threshold", 2024-04-23T14:58:27.659 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-23T14:58:27.659 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.659 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.659 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-04-23T14:58:27.659 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1.0", 2024-04-23T14:58:27.659 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.659 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.659 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "scaling threshold", 2024-04-23T14:58:27.659 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-23T14:58:27.660 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.660 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.660 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.660 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.660 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "progress", 2024-04-23T14:58:27.660 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.660 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.661 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.661 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_pg_recovery_event": { 2024-04-23T14:58:27.661 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_pg_recovery_event", 2024-04-23T14:58:27.661 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.661 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.661 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.661 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.661 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.661 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.662 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.662 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-23T14:58:27.662 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.662 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.662 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.662 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-04-23T14:58:27.662 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-04-23T14:58:27.663 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.663 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.663 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.663 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.663 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.663 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.663 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.663 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.663 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.664 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.664 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.664 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.664 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.664 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.664 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.664 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.664 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.665 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.665 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.665 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.665 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.665 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.665 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.665 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.665 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.665 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.666 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.666 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.666 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.666 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.666 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.666 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.525+0000 7f5f8d200640 1 -- 172.21.15.28:0/3211144062 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5f6c0a3d20 msgr2=0x7f5f6c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:27.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.525+0000 7f5f8d200640 1 --2- 172.21.15.28:0/3211144062 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5f6c0a3d20 0x7f5f6c0a61e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5f8810f330 tx=0x7f5f74005f90 comp rx=0 tx=0).stop 2024-04-23T14:58:27.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.525+0000 7f5f8d200640 1 -- 172.21.15.28:0/3211144062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f88102240 msgr2=0x7f5f881a0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:58:27.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.525+0000 7f5f8d200640 1 --2- 172.21.15.28:0/3211144062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f88102240 0x7f5f881a0760 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f5f78004770 tx=0x7f5f78003c10 comp rx=0 tx=0).stop 2024-04-23T14:58:27.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.525+0000 7f5f8d200640 1 -- 172.21.15.28:0/3211144062 shutdown_connections 2024-04-23T14:58:27.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.525+0000 7f5f8d200640 1 --2- 172.21.15.28:0/3211144062 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f5f6c0a3d20 0x7f5f6c0a61e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:27.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.525+0000 7f5f8d200640 1 --2- 172.21.15.28:0/3211144062 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f8819a910 0x7f5f8819adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:27.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.525+0000 7f5f8d200640 1 --2- 172.21.15.28:0/3211144062 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5f88107310 0x7f5f881a0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:27.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.525+0000 7f5f8d200640 1 --2- 172.21.15.28:0/3211144062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f88102240 0x7f5f881a0760 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:58:27.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.525+0000 7f5f8d200640 1 -- 172.21.15.28:0/3211144062 >> 172.21.15.28:0/3211144062 conn(0x7f5f880793c0 msgr2=0x7f5f880fff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:58:27.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.525+0000 7f5f8d200640 1 -- 172.21.15.28:0/3211144062 shutdown_connections 2024-04-23T14:58:27.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:58:26.525+0000 7f5f8d200640 1 -- 172.21.15.28:0/3211144062 wait complete. 2024-04-23T14:58:27.671 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:26 smithi105 ceph-mon[32555]: pgmap v451: 97 pgs: 97 active+clean; 835 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 19 MiB/s wr, 159 op/s 2024-04-23T14:58:27.671 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:26 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3549654427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T14:58:27.671 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:26 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3211144062' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-23T14:58:27.671 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:26 smithi038 ceph-mon[32348]: pgmap v451: 97 pgs: 97 active+clean; 835 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 19 MiB/s wr, 159 op/s 2024-04-23T14:58:27.671 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:26 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3549654427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T14:58:27.671 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:26 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3211144062' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-23T14:58:27.672 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.672 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.672 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.672 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.672 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.673 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.673 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.673 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.673 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.673 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.673 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.673 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.673 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.673 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.674 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.674 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.674 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.674 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.674 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.674 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.674 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.674 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.674 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.675 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.675 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.675 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.675 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.675 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.675 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.675 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.675 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.675 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.676 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.676 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.676 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.676 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.676 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.676 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.676 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.676 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.677 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.677 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.677 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.677 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.677 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.677 INFO:teuthology.orchestra.run.smithi028.stdout: "max_completed_events": { 2024-04-23T14:58:27.677 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_completed_events", 2024-04-23T14:58:27.677 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.678 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.678 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.678 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "50", 2024-04-23T14:58:27.678 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.678 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.678 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.678 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "number of past completed events to remember", 2024-04-23T14:58:27.678 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.678 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.679 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.679 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-23T14:58:27.679 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-23T14:58:27.679 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:27.679 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.679 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.679 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-23T14:58:27.679 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.680 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.680 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.680 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long the module is going to sleep", 2024-04-23T14:58:27.680 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.680 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.680 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.680 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.680 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.681 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus", 2024-04-23T14:58:27.681 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.681 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.681 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.681 INFO:teuthology.orchestra.run.smithi028.stdout: "cache": { 2024-04-23T14:58:27.681 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cache", 2024-04-23T14:58:27.681 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.681 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.682 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.682 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.682 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.682 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.682 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.682 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.682 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.682 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.682 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.683 INFO:teuthology.orchestra.run.smithi028.stdout: "exclude_perf_counters": { 2024-04-23T14:58:27.683 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "exclude_perf_counters", 2024-04-23T14:58:27.683 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.683 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.683 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.683 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.683 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.683 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.683 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.684 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-23T14:58:27.684 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-23T14:58:27.684 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.684 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.684 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.684 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.684 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.684 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.685 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.685 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.685 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.685 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.685 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.685 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.685 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.685 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.685 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.686 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.686 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.686 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.686 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.686 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.686 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.686 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.686 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.686 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.687 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.687 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.687 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.687 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.687 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.687 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.687 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.687 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.687 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.688 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.688 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.688 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.688 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.688 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.688 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.688 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.688 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.689 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.689 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.689 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.689 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.689 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.689 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.689 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.689 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.689 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.689 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.690 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.690 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.690 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.690 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.690 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.690 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.690 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.690 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.691 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.691 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.691 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.691 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.691 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.691 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.691 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.691 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.691 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.692 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools": { 2024-04-23T14:58:27.692 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools", 2024-04-23T14:58:27.692 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.692 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.692 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.692 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.692 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.692 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.692 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.693 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.693 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.693 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.693 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.693 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-23T14:58:27.693 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-23T14:58:27.693 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.693 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.694 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.694 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "300", 2024-04-23T14:58:27.694 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.694 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.694 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.694 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.694 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.694 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.694 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.695 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_interval": { 2024-04-23T14:58:27.695 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_interval", 2024-04-23T14:58:27.695 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-23T14:58:27.695 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.695 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.695 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15.0", 2024-04-23T14:58:27.695 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.695 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.696 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.696 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.696 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.696 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.696 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.696 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-23T14:58:27.696 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-23T14:58:27.696 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.696 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.697 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.697 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-04-23T14:58:27.697 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.697 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.697 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.697 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-23T14:58:27.697 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.697 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.697 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.698 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-23T14:58:27.698 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-23T14:58:27.698 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.698 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.698 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.698 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "9283", 2024-04-23T14:58:27.698 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.698 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.699 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.699 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-23T14:58:27.699 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.699 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.699 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.699 INFO:teuthology.orchestra.run.smithi028.stdout: "stale_cache_strategy": { 2024-04-23T14:58:27.699 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stale_cache_strategy", 2024-04-23T14:58:27.699 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.700 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.700 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.700 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "log", 2024-04-23T14:58:27.700 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.700 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.700 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.700 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.700 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.700 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.701 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.701 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-04-23T14:58:27.701 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-04-23T14:58:27.701 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.701 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.701 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.701 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "default", 2024-04-23T14:58:27.701 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.702 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.702 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.702 INFO:teuthology.orchestra.run.smithi028.stdout: "default", 2024-04-23T14:58:27.702 INFO:teuthology.orchestra.run.smithi028.stdout: "error" 2024-04-23T14:58:27.702 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.702 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.702 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.702 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.702 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.703 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-04-23T14:58:27.703 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-04-23T14:58:27.703 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.703 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.703 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.703 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-04-23T14:58:27.703 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-04-23T14:58:27.703 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-04-23T14:58:27.704 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.704 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.704 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.704 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.704 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.704 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.704 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.705 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-04-23T14:58:27.705 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.705 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.705 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.705 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.705 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.705 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.705 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.705 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.706 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.706 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.706 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.706 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.706 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.706 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.706 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.706 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.706 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.707 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.707 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.707 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.707 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.707 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.707 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.707 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.707 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.708 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.708 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.708 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.708 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.708 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.708 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.708 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.708 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.708 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.709 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.709 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.709 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.709 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.709 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.709 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.709 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.709 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.709 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.710 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.710 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.710 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.710 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.710 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.710 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.710 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.710 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.710 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.710 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.711 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.711 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.711 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.711 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.711 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.711 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.711 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.711 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.712 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.712 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.712 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.712 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.712 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.712 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.712 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.712 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.712 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_snap_create": { 2024-04-23T14:58:27.713 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_snap_create", 2024-04-23T14:58:27.713 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.713 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.713 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.713 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-23T14:58:27.713 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.713 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.713 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.714 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.714 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.714 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.714 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.714 INFO:teuthology.orchestra.run.smithi028.stdout: "mirror_snapshot_schedule": { 2024-04-23T14:58:27.714 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirror_snapshot_schedule", 2024-04-23T14:58:27.714 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.714 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.715 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.715 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.715 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.715 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.715 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.715 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.715 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.715 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.715 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.716 INFO:teuthology.orchestra.run.smithi028.stdout: "trash_purge_schedule": { 2024-04-23T14:58:27.716 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "trash_purge_schedule", 2024-04-23T14:58:27.716 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.716 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.716 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.716 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.716 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.716 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.717 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.717 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.717 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.717 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.717 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.717 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.717 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.718 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "restful", 2024-04-23T14:58:27.718 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.718 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.718 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.718 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_auth": { 2024-04-23T14:58:27.718 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_auth", 2024-04-23T14:58:27.718 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.718 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.719 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.719 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.719 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.719 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.719 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.719 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.719 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.719 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.720 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.720 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-04-23T14:58:27.720 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-04-23T14:58:27.720 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.720 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.720 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.720 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.720 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.720 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.721 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.721 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.721 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.721 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.721 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.721 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.721 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.721 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.722 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.722 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.722 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.722 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.722 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.722 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.722 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.722 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.722 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.722 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.723 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.723 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.723 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.723 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.723 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.723 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.723 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.723 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.724 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.724 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.724 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.724 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.724 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.724 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.724 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.724 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.724 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.724 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.725 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.725 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.725 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.725 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.725 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.725 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.725 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.725 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.726 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.726 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.726 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.726 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.726 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.726 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.726 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.726 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.726 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.727 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.727 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.727 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.727 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.727 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.727 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.727 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.727 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.728 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.728 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.728 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.728 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.728 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.728 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.728 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.728 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.728 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.729 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.729 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-23T14:58:27.729 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-23T14:58:27.729 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.729 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.729 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.729 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.730 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.730 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.730 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.730 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.730 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.730 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.730 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.730 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-23T14:58:27.730 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-23T14:58:27.731 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.731 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.731 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.731 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.731 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.731 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.731 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.731 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.732 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.732 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.732 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.732 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.732 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.732 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rgw", 2024-04-23T14:58:27.732 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.732 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.733 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.733 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.733 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.733 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.733 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.733 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.733 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.733 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.733 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.734 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.734 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.734 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.734 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.734 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.734 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.734 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.734 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.735 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.735 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.735 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.735 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.735 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.735 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.735 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.735 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.736 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.736 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.736 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.736 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.736 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.736 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.736 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.736 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.736 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.737 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.737 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.737 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.737 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.737 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.737 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.737 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.738 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.738 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.738 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.738 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.738 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.738 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.738 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.738 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.738 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.739 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.739 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.739 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.739 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.739 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.739 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.739 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.739 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.739 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.740 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.740 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.740 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.740 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.740 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.740 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.740 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.740 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.740 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.741 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.741 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.741 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rook", 2024-04-23T14:58:27.741 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.741 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.741 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.741 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.741 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.741 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.742 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.742 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.742 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.742 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.742 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.742 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.742 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.742 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.743 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.743 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.743 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.743 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.743 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.743 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.743 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.743 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.743 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.744 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.744 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.744 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.744 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.744 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.744 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.744 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.744 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.744 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.745 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.745 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.745 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.745 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.745 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.745 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.745 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.745 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.746 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.746 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.746 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.746 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.746 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.746 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.746 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.746 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.747 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.747 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.747 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.747 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.747 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.747 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.747 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.747 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.748 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.748 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.748 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.748 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.748 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.748 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.748 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.748 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.748 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.749 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.749 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.749 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.749 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.749 INFO:teuthology.orchestra.run.smithi028.stdout: "storage_class": { 2024-04-23T14:58:27.749 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "storage_class", 2024-04-23T14:58:27.749 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.749 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.750 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.750 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "local", 2024-04-23T14:58:27.750 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.750 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.750 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.750 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-23T14:58:27.750 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.750 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.751 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.751 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.751 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.751 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.751 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "selftest", 2024-04-23T14:58:27.751 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.751 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.751 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.751 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.752 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.752 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.752 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.752 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.752 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.752 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.752 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.752 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.753 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.753 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.753 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.753 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.753 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.753 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.753 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.753 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.753 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.754 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.754 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.754 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.754 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.754 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.754 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.754 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.754 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.755 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.755 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.755 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.755 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.755 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.755 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.755 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.755 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.755 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.756 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.756 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.756 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.756 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.756 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.756 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.756 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.756 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.756 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.757 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.757 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.757 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.757 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.757 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.757 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.757 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.757 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.757 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.758 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.758 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.758 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.758 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.758 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.758 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.758 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.759 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.759 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.759 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.759 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.759 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.759 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.759 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.759 INFO:teuthology.orchestra.run.smithi028.stdout: "roption1": { 2024-04-23T14:58:27.760 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption1", 2024-04-23T14:58:27.760 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.760 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.760 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.760 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.760 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.760 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.760 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.760 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.761 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.761 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.761 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.761 INFO:teuthology.orchestra.run.smithi028.stdout: "roption2": { 2024-04-23T14:58:27.761 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption2", 2024-04-23T14:58:27.761 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.761 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.761 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.762 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "xyz", 2024-04-23T14:58:27.762 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.762 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.762 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.762 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.762 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.762 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.762 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.763 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption1": { 2024-04-23T14:58:27.763 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption1", 2024-04-23T14:58:27.763 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.763 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.763 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.763 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.763 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.763 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.764 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.764 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.764 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.764 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.764 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.764 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption2": { 2024-04-23T14:58:27.764 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption2", 2024-04-23T14:58:27.764 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.765 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.765 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.765 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.765 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.765 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.765 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.765 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.765 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.765 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.766 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.766 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption3": { 2024-04-23T14:58:27.766 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption3", 2024-04-23T14:58:27.766 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-23T14:58:27.766 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.766 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.766 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.766 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.767 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.767 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.767 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.767 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.767 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.767 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.767 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption4": { 2024-04-23T14:58:27.767 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption4", 2024-04-23T14:58:27.768 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.768 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.768 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.768 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.768 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.768 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.768 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.768 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.769 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.769 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.769 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.769 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption5": { 2024-04-23T14:58:27.769 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption5", 2024-04-23T14:58:27.769 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.769 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.770 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.770 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.770 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.770 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.770 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.770 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.770 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.770 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.771 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.771 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption6": { 2024-04-23T14:58:27.771 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption6", 2024-04-23T14:58:27.771 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.771 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.771 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.771 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.771 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.772 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.772 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.772 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.772 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.772 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.772 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.772 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption7": { 2024-04-23T14:58:27.773 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption7", 2024-04-23T14:58:27.773 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.773 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.773 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.773 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.773 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-23T14:58:27.773 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "42", 2024-04-23T14:58:27.773 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.773 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.774 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.774 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.774 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.774 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.774 INFO:teuthology.orchestra.run.smithi028.stdout: "testkey": { 2024-04-23T14:58:27.774 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testkey", 2024-04-23T14:58:27.774 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.774 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.775 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.775 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.775 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.775 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.775 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.775 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.775 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.775 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.775 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.776 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.776 INFO:teuthology.orchestra.run.smithi028.stdout: "testlkey": { 2024-04-23T14:58:27.776 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testlkey", 2024-04-23T14:58:27.776 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.776 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.776 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.776 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.776 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.776 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.777 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.777 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.777 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.777 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.777 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.777 INFO:teuthology.orchestra.run.smithi028.stdout: "testnewline": { 2024-04-23T14:58:27.777 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testnewline", 2024-04-23T14:58:27.777 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.778 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.778 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.778 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.778 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.778 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.778 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.778 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.778 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.778 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.779 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.779 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.779 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.779 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.779 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.779 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-04-23T14:58:27.779 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.779 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.779 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.780 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_m_granularity": { 2024-04-23T14:58:27.780 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_m_granularity", 2024-04-23T14:58:27.780 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.780 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.780 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.780 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.780 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.780 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.780 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.781 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow minute scheduled snapshots", 2024-04-23T14:58:27.781 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.781 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.781 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.781 INFO:teuthology.orchestra.run.smithi028.stdout: "dump_on_update": { 2024-04-23T14:58:27.781 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dump_on_update", 2024-04-23T14:58:27.781 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.781 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.782 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.782 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.782 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.782 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.782 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.782 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "dump database to debug log on update", 2024-04-23T14:58:27.782 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.782 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.782 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.783 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.783 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.783 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.783 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.783 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.783 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.783 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.783 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.784 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.784 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.784 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.784 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.784 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.784 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.784 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.784 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.785 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.785 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.785 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.785 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.785 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.785 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.785 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.785 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.785 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.786 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.786 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.786 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.786 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.786 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.786 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.786 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.786 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.787 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.787 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.787 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.787 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.787 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.787 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.787 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.787 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.788 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.788 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.788 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.788 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.788 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.788 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.788 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.788 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.788 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.789 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.789 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.789 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.789 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.789 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.789 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.789 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.789 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.790 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.790 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.790 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.790 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.790 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.790 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.790 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.790 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.791 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.791 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.791 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stats", 2024-04-23T14:58:27.791 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.791 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.791 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.791 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.792 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.792 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.792 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.792 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.792 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.792 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.792 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.792 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.792 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.793 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.793 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.793 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.793 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.793 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.793 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.793 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.793 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.793 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.793 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.794 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.794 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.794 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.794 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.794 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.794 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.794 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.795 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.795 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.795 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.795 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.795 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.795 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.795 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.795 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.796 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.796 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.796 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.796 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.796 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.796 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.796 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.796 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.796 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.797 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.797 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.797 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.797 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.797 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.797 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.797 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.797 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.797 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.798 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.798 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.798 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.798 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.798 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.798 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.798 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.799 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.799 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.799 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.799 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.799 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.799 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.799 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.799 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.800 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.800 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "status", 2024-04-23T14:58:27.800 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.800 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.800 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.800 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.800 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.800 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.800 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.801 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.801 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.801 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.801 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.801 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.801 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.801 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.801 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.802 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.802 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.802 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.802 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.802 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.802 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.802 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.802 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.803 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.803 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.803 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.803 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.803 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.803 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.803 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.803 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.804 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.804 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.804 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.804 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.804 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.804 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.804 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.804 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.805 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.805 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.805 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.805 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.805 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.805 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.805 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.805 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.805 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.806 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.806 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.806 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.806 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.806 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.806 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.806 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.806 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.807 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.807 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.807 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.807 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.807 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.807 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.807 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.808 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.808 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.808 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.808 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.808 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.808 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.808 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.808 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.809 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telegraf", 2024-04-23T14:58:27.809 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.809 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.809 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.809 INFO:teuthology.orchestra.run.smithi028.stdout: "address": { 2024-04-23T14:58:27.809 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "address", 2024-04-23T14:58:27.809 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.809 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.810 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.810 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-23T14:58:27.810 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.810 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.810 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.810 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.810 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.810 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.811 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.811 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-23T14:58:27.811 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-23T14:58:27.811 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:27.811 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.811 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.811 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15", 2024-04-23T14:58:27.811 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.812 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.812 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.812 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.812 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.812 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.812 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.812 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.812 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.813 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.813 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.813 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.813 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.813 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.813 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.813 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.813 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.814 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.814 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.814 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.814 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.814 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.814 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.814 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.814 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.814 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.815 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.815 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.815 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.815 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.815 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.815 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.815 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.815 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.816 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.816 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.816 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.816 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.816 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.816 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.816 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.816 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.817 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.817 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.817 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.817 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.817 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.817 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.817 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.817 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.817 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.818 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.818 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.818 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.818 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.818 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.818 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.818 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.818 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.818 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.819 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.819 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.819 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.819 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.819 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.819 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.819 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.819 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.820 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.820 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.820 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.820 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.820 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.820 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.820 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.821 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telemetry", 2024-04-23T14:58:27.821 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.821 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.821 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.821 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_basic": { 2024-04-23T14:58:27.821 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_basic", 2024-04-23T14:58:27.821 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.821 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.822 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.822 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.822 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.822 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.822 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.822 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-23T14:58:27.822 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.822 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.822 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.823 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_crash": { 2024-04-23T14:58:27.823 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_crash", 2024-04-23T14:58:27.823 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.823 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.823 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.823 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.823 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.824 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.824 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.824 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-23T14:58:27.824 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.824 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.824 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.824 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_device": { 2024-04-23T14:58:27.824 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_device", 2024-04-23T14:58:27.825 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.825 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.825 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.825 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.825 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.825 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.825 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.825 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-23T14:58:27.825 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.826 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.826 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.826 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_ident": { 2024-04-23T14:58:27.826 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_ident", 2024-04-23T14:58:27.826 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.826 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.826 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.827 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.827 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.827 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.827 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.827 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-23T14:58:27.827 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.827 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.827 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.828 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_perf": { 2024-04-23T14:58:27.828 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_perf", 2024-04-23T14:58:27.828 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.828 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.828 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.828 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.828 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.828 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.828 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.829 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-23T14:58:27.829 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.829 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.829 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.829 INFO:teuthology.orchestra.run.smithi028.stdout: "contact": { 2024-04-23T14:58:27.829 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "contact", 2024-04-23T14:58:27.829 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.830 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.830 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.830 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.830 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.830 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.830 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.830 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.830 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.831 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.831 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.831 INFO:teuthology.orchestra.run.smithi028.stdout: "description": { 2024-04-23T14:58:27.831 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "description", 2024-04-23T14:58:27.831 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.831 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.831 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.831 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.832 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.832 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.832 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.832 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.832 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.832 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.832 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.833 INFO:teuthology.orchestra.run.smithi028.stdout: "device_url": { 2024-04-23T14:58:27.833 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_url", 2024-04-23T14:58:27.833 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.833 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.833 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.833 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-23T14:58:27.833 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.833 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.833 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.834 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.834 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.834 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.834 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.834 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-04-23T14:58:27.834 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-04-23T14:58:27.834 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.835 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.835 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.835 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.835 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.835 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.835 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.835 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.835 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.835 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.836 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.836 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-23T14:58:27.836 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-23T14:58:27.836 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.836 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.836 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.836 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "24", 2024-04-23T14:58:27.837 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "8", 2024-04-23T14:58:27.837 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.837 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.837 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.837 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.837 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.837 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.837 INFO:teuthology.orchestra.run.smithi028.stdout: "last_opt_revision": { 2024-04-23T14:58:27.838 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "last_opt_revision", 2024-04-23T14:58:27.838 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.838 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.838 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.838 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1", 2024-04-23T14:58:27.838 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.838 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.838 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.839 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.839 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.839 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.839 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.839 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard": { 2024-04-23T14:58:27.839 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard", 2024-04-23T14:58:27.839 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.839 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.840 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.840 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.840 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.840 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.840 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.840 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.840 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.840 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.840 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.841 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard_description": { 2024-04-23T14:58:27.841 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard_description", 2024-04-23T14:58:27.841 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.841 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.841 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.841 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.841 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.842 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.842 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.842 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.842 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.842 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.842 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.842 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.843 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.843 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.843 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.843 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.843 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.843 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.843 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.843 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.843 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.844 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.844 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.844 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.844 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.844 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.844 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.844 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.844 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.844 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.845 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.845 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.845 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.845 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.845 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.845 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.845 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.846 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.846 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.846 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.846 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.846 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.846 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.846 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.847 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.847 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.847 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.847 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.847 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.847 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.847 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.847 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.847 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.848 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.848 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.848 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.848 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.848 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.848 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.848 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.848 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.848 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.849 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.849 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.849 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.849 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.849 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.849 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.849 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.850 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.850 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.850 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.850 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.850 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.850 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.850 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.850 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.851 INFO:teuthology.orchestra.run.smithi028.stdout: "organization": { 2024-04-23T14:58:27.851 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "organization", 2024-04-23T14:58:27.851 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.851 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.851 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.851 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.851 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.851 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.852 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.852 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.852 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.852 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.852 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.852 INFO:teuthology.orchestra.run.smithi028.stdout: "proxy": { 2024-04-23T14:58:27.852 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "proxy", 2024-04-23T14:58:27.852 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.853 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.853 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.853 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.853 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.853 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.853 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.853 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.853 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.853 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.854 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.854 INFO:teuthology.orchestra.run.smithi028.stdout: "url": { 2024-04-23T14:58:27.854 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url", 2024-04-23T14:58:27.854 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.854 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.854 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.854 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-23T14:58:27.855 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.855 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.855 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.855 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.855 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.855 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.855 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.855 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.855 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.856 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "test_orchestrator", 2024-04-23T14:58:27.856 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.856 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.856 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.856 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.856 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.856 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.857 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.857 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.857 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.857 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.857 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.857 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.857 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.857 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.858 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.858 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.858 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.858 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.858 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.858 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.858 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.858 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.858 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.859 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.859 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.859 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.859 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.859 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.859 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.859 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.859 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.860 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.860 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.860 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.860 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.860 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.860 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.860 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.861 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.861 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.861 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.861 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.861 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.861 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.861 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.861 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.861 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.862 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.862 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.862 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.862 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.862 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.862 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.862 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.862 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.863 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.863 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.863 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.863 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.863 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.863 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.863 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.863 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.864 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.864 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.864 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.864 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.864 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.864 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.864 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.864 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.865 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-04-23T14:58:27.865 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.865 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.865 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.865 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.865 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.865 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.866 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.866 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.866 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.866 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.866 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.866 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.866 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.866 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.866 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.867 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.867 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.867 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.867 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.867 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.867 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.867 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.867 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.868 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.868 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.868 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.868 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.868 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.868 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.868 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.868 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.869 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.869 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.869 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.869 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.869 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.869 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.869 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.869 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.870 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.870 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.870 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.870 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.870 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.870 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.870 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.870 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.871 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.871 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.871 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.871 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.871 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.871 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.871 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.871 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.871 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.872 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.872 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.872 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.872 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.872 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.872 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.872 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.872 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.873 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.873 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.873 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.873 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.873 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.873 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_clones": { 2024-04-23T14:58:27.873 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_clones", 2024-04-23T14:58:27.874 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.874 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.874 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.874 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4", 2024-04-23T14:58:27.874 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.874 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.874 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.874 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-23T14:58:27.874 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.875 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.875 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.875 INFO:teuthology.orchestra.run.smithi028.stdout: "periodic_async_work": { 2024-04-23T14:58:27.875 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "periodic_async_work", 2024-04-23T14:58:27.875 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.875 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.875 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.876 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.876 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.876 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.876 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.876 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Periodically check for async work", 2024-04-23T14:58:27.876 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.876 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.876 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.877 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_delay": { 2024-04-23T14:58:27.877 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_delay", 2024-04-23T14:58:27.877 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.877 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.877 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.877 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-23T14:58:27.877 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.877 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.877 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.878 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-23T14:58:27.878 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.878 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.878 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.878 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_no_wait": { 2024-04-23T14:58:27.878 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_no_wait", 2024-04-23T14:58:27.878 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.878 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.879 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.879 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.879 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.879 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.879 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.879 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-23T14:58:27.879 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.879 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.880 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.880 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.880 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.880 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix", 2024-04-23T14:58:27.880 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.880 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.880 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.881 INFO:teuthology.orchestra.run.smithi028.stdout: "discovery_interval": { 2024-04-23T14:58:27.881 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "discovery_interval", 2024-04-23T14:58:27.881 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-23T14:58:27.881 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.881 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.881 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "100", 2024-04-23T14:58:27.881 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.881 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.881 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.882 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.882 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.882 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.882 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.882 INFO:teuthology.orchestra.run.smithi028.stdout: "identifier": { 2024-04-23T14:58:27.882 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "identifier", 2024-04-23T14:58:27.882 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.883 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.883 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.883 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.883 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.883 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.883 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.883 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.883 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.883 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.884 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.884 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-23T14:58:27.884 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-23T14:58:27.884 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:27.884 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.884 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.884 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-23T14:58:27.885 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.885 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.885 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.885 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.885 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.885 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.885 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.885 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.886 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.886 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.886 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.886 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.886 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.886 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.886 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.886 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.886 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.887 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.887 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.887 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.887 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.887 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.887 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.887 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.887 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.887 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.888 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.888 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.888 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.888 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.888 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.888 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.888 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.888 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.889 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.889 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.889 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.889 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.889 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.889 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.889 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.889 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.890 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.890 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.890 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.890 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.890 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.890 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.890 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.890 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.891 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.891 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.891 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.891 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.891 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.891 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.891 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.891 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.891 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.892 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.892 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.892 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.892 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.892 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.892 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.892 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.892 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.893 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.893 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.893 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.893 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.893 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.893 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.893 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_host": { 2024-04-23T14:58:27.893 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_host", 2024-04-23T14:58:27.894 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.894 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.894 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.894 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.894 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.894 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.894 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.894 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.895 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.895 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.895 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.895 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_port": { 2024-04-23T14:58:27.895 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_port", 2024-04-23T14:58:27.895 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.895 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.895 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.896 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10051", 2024-04-23T14:58:27.896 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.896 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.896 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.896 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.896 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.896 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.896 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.897 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_sender": { 2024-04-23T14:58:27.897 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_sender", 2024-04-23T14:58:27.897 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.897 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.897 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.897 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-23T14:58:27.897 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.897 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.897 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.898 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.898 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.898 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.898 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.898 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.898 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.898 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.898 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-23T14:58:27.899 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.899 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.899 INFO:teuthology.orchestra.run.smithi028.stdout: "modules": [ 2024-04-23T14:58:27.899 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-04-23T14:58:27.899 INFO:teuthology.orchestra.run.smithi028.stdout: "dashboard", 2024-04-23T14:58:27.899 INFO:teuthology.orchestra.run.smithi028.stdout: "iostat", 2024-04-23T14:58:27.899 INFO:teuthology.orchestra.run.smithi028.stdout: "nfs", 2024-04-23T14:58:27.899 INFO:teuthology.orchestra.run.smithi028.stdout: "restful", 2024-04-23T14:58:27.899 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_schedule" 2024-04-23T14:58:27.900 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.900 INFO:teuthology.orchestra.run.smithi028.stdout: "available_modules": [ 2024-04-23T14:58:27.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.900 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "alerts", 2024-04-23T14:58:27.900 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.900 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.900 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.900 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-23T14:58:27.900 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-23T14:58:27.901 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:27.901 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.901 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.901 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-23T14:58:27.901 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.901 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.901 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.901 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How frequently to reexamine health status", 2024-04-23T14:58:27.901 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.902 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.902 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.902 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.902 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.902 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.902 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.902 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.903 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.903 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.903 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.903 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.903 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.903 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.903 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.903 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.903 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.904 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.904 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.904 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.904 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.904 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.904 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.904 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.905 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.905 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.905 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.905 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.905 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.905 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.905 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.905 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.905 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.906 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.906 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.906 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.906 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.906 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.906 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.906 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.906 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.907 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.907 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.907 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.907 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.907 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.907 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.907 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.907 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.907 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.908 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.908 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.908 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.908 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.908 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.908 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.908 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.908 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.909 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.909 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.909 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.909 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.909 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.909 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.909 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.909 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.909 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.910 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.910 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.910 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_destination": { 2024-04-23T14:58:27.910 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_destination", 2024-04-23T14:58:27.910 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.910 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.910 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.911 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.911 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.911 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.911 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.911 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email address to send alerts to", 2024-04-23T14:58:27.911 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.911 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.911 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.912 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_from_name": { 2024-04-23T14:58:27.912 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_from_name", 2024-04-23T14:58:27.912 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.912 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.912 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.912 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "Ceph", 2024-04-23T14:58:27.912 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.912 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.912 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.913 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email From: name", 2024-04-23T14:58:27.913 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.913 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.913 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.913 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_host": { 2024-04-23T14:58:27.913 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_host", 2024-04-23T14:58:27.913 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.913 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.914 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.914 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.914 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.914 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.914 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.914 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP server", 2024-04-23T14:58:27.914 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.914 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.915 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.915 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_password": { 2024-04-23T14:58:27.915 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_password", 2024-04-23T14:58:27.915 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.915 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.915 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.915 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.916 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.916 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.916 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.916 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Password to authenticate with", 2024-04-23T14:58:27.916 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.916 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.916 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.916 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_port": { 2024-04-23T14:58:27.917 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_port", 2024-04-23T14:58:27.917 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.917 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.917 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.917 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "465", 2024-04-23T14:58:27.917 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.917 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.917 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.917 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP port", 2024-04-23T14:58:27.918 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.918 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.918 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.918 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.918 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_sender": { 2024-04-23T14:58:27.918 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_sender", 2024-04-23T14:58:27.918 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.918 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.919 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.919 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.919 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.919 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.919 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.919 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP envelope sender", 2024-04-23T14:58:27.919 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.919 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.919 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.920 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_ssl": { 2024-04-23T14:58:27.920 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_ssl", 2024-04-23T14:58:27.920 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.920 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.920 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.920 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.920 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.920 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.921 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.921 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-23T14:58:27.921 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.921 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.921 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.921 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_user": { 2024-04-23T14:58:27.921 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_user", 2024-04-23T14:58:27.921 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.922 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.922 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.922 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.922 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.922 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.922 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.922 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "User to authenticate as", 2024-04-23T14:58:27.922 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.923 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.923 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.923 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.923 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.923 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.923 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.923 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "balancer", 2024-04-23T14:58:27.923 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.923 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.924 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.924 INFO:teuthology.orchestra.run.smithi028.stdout: "active": { 2024-04-23T14:58:27.924 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "active", 2024-04-23T14:58:27.924 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.924 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.924 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.924 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.924 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.924 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.925 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.925 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically balance PGs across cluster", 2024-04-23T14:58:27.925 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.925 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.925 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.925 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_time": { 2024-04-23T14:58:27.925 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_time", 2024-04-23T14:58:27.926 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.926 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.926 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.926 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0000", 2024-04-23T14:58:27.926 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.926 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.926 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.926 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "beginning time of day to automatically balance", 2024-04-23T14:58:27.929 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-23T14:58:27.929 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.930 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.930 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_weekday": { 2024-04-23T14:58:27.930 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_weekday", 2024-04-23T14:58:27.930 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-23T14:58:27.930 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.930 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.930 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-23T14:58:27.931 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-04-23T14:58:27.931 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-04-23T14:58:27.931 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.931 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-23T14:58:27.931 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-23T14:58:27.931 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.931 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.932 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_max_iterations": { 2024-04-23T14:58:27.932 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_max_iterations", 2024-04-23T14:58:27.932 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-23T14:58:27.932 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.932 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.932 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "25", 2024-04-23T14:58:27.932 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-23T14:58:27.932 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "250", 2024-04-23T14:58:27.932 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.933 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-23T14:58:27.933 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.933 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.933 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.933 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_metrics": { 2024-04-23T14:58:27.933 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_metrics", 2024-04-23T14:58:27.933 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.934 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.934 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.934 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "pgs,objects,bytes", 2024-04-23T14:58:27.934 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.934 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.934 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.934 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-23T14:58:27.934 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-23T14:58:27.935 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.935 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.935 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_step": { 2024-04-23T14:58:27.935 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_step", 2024-04-23T14:58:27.935 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-23T14:58:27.935 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.936 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.936 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.5", 2024-04-23T14:58:27.936 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0.001", 2024-04-23T14:58:27.936 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "0.999", 2024-04-23T14:58:27.936 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.936 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "aggressiveness of optimization", 2024-04-23T14:58:27.936 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-23T14:58:27.936 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.937 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.937 INFO:teuthology.orchestra.run.smithi028.stdout: "end_time": { 2024-04-23T14:58:27.937 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_time", 2024-04-23T14:58:27.937 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.937 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.937 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.937 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2359", 2024-04-23T14:58:27.938 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.938 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.938 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.938 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ending time of day to automatically balance", 2024-04-23T14:58:27.938 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-23T14:58:27.938 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.938 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.938 INFO:teuthology.orchestra.run.smithi028.stdout: "end_weekday": { 2024-04-23T14:58:27.939 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_weekday", 2024-04-23T14:58:27.939 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-23T14:58:27.939 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.939 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.939 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-23T14:58:27.939 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-04-23T14:58:27.939 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-04-23T14:58:27.939 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.939 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-23T14:58:27.940 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-23T14:58:27.940 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.940 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.940 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:27.940 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:27.940 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.940 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.941 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.941 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.941 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.941 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.941 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.941 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.941 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.941 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.941 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.942 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.942 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.942 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.942 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.942 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.942 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.942 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.942 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:27.943 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:27.943 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.943 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.943 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.943 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.943 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.943 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.943 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.943 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.944 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.944 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.944 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.944 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:27.944 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:27.944 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.944 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.944 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.945 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:27.945 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.945 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.945 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.945 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:27.945 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:27.945 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:27.945 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:27.945 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:27.946 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:27.946 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.946 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.946 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.946 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.946 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.946 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:27.947 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:27.947 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.947 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.947 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.947 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.947 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.947 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.947 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.947 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:27.948 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.948 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.948 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.948 INFO:teuthology.orchestra.run.smithi028.stdout: "min_score": { 2024-04-23T14:58:27.948 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_score", 2024-04-23T14:58:27.948 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-23T14:58:27.948 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.949 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.949 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-23T14:58:27.949 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.949 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.949 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.949 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-23T14:58:27.949 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.949 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.949 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.950 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-04-23T14:58:27.950 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-04-23T14:58:27.950 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.950 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.950 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.950 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "upmap", 2024-04-23T14:58:27.950 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.950 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.951 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.951 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-compat", 2024-04-23T14:58:27.951 INFO:teuthology.orchestra.run.smithi028.stdout: "none", 2024-04-23T14:58:27.951 INFO:teuthology.orchestra.run.smithi028.stdout: "read", 2024-04-23T14:58:27.951 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap", 2024-04-23T14:58:27.951 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap-read" 2024-04-23T14:58:27.951 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.951 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Balancer mode", 2024-04-23T14:58:27.952 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.952 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.952 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.952 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_ids": { 2024-04-23T14:58:27.952 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_ids", 2024-04-23T14:58:27.952 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.952 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.952 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.953 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.953 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.953 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.953 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.953 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-23T14:58:27.953 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.953 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.953 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.954 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-23T14:58:27.954 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-23T14:58:27.954 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:27.954 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.954 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.954 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-23T14:58:27.955 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.955 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.955 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.955 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-23T14:58:27.955 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.955 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.955 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.955 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_deviation": { 2024-04-23T14:58:27.956 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_deviation", 2024-04-23T14:58:27.956 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.956 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.956 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.956 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-23T14:58:27.956 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-23T14:58:27.956 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.956 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.956 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-23T14:58:27.957 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-23T14:58:27.957 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.957 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.957 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_optimizations": { 2024-04-23T14:58:27.957 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_optimizations", 2024-04-23T14:58:27.957 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-23T14:58:27.957 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.958 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.958 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-23T14:58:27.958 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.958 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.958 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.958 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-23T14:58:27.958 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.958 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.959 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.959 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.959 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:27.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:27.959 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm", 2024-04-23T14:58:27.959 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:27.959 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:27.959 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:27.960 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_down_multiplier": { 2024-04-23T14:58:27.960 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_down_multiplier", 2024-04-23T14:58:27.960 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-23T14:58:27.960 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.960 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.960 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-04-23T14:58:27.960 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.960 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.961 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.961 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-23T14:58:27.961 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.961 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.961 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.961 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_refresh_rate": { 2024-04-23T14:58:27.961 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_refresh_rate", 2024-04-23T14:58:27.962 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:27.962 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.962 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.962 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "20", 2024-04-23T14:58:27.962 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.962 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.962 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.963 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-23T14:58:27.963 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.963 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.963 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.963 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_starting_port": { 2024-04-23T14:58:27.963 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_starting_port", 2024-04-23T14:58:27.963 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:27.964 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.964 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.964 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4721", 2024-04-23T14:58:27.964 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.964 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.964 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.964 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-23T14:58:27.964 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.965 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.965 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.965 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_ptrace": { 2024-04-23T14:58:27.965 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_ptrace", 2024-04-23T14:58:27.965 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.965 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.965 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.966 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.966 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.966 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.966 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.966 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-23T14:58:27.966 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-23T14:58:27.966 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.966 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.967 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_interval": { 2024-04-23T14:58:27.967 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_interval", 2024-04-23T14:58:27.967 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:27.967 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.967 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.967 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-23T14:58:27.967 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.968 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.968 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.968 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-23T14:58:27.968 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.968 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.968 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.968 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_memory_target_ratio": { 2024-04-23T14:58:27.968 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_memory_target_ratio", 2024-04-23T14:58:27.969 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-23T14:58:27.969 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.969 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.969 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.7", 2024-04-23T14:58:27.969 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.969 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.969 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.969 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-23T14:58:27.970 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.970 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.970 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.970 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_log_destination": { 2024-04-23T14:58:27.970 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm_log_destination", 2024-04-23T14:58:27.970 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.970 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.971 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.971 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:27.971 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.971 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.971 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:27.971 INFO:teuthology.orchestra.run.smithi028.stdout: "file", 2024-04-23T14:58:27.971 INFO:teuthology.orchestra.run.smithi028.stdout: "file,syslog", 2024-04-23T14:58:27.971 INFO:teuthology.orchestra.run.smithi028.stdout: "syslog" 2024-04-23T14:58:27.972 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:27.972 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-23T14:58:27.972 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.972 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.972 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.972 INFO:teuthology.orchestra.run.smithi028.stdout: "cgroups_split": { 2024-04-23T14:58:27.972 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cgroups_split", 2024-04-23T14:58:27.973 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.973 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.973 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.973 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.973 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.973 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.973 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.973 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-23T14:58:27.973 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.974 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.974 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.974 INFO:teuthology.orchestra.run.smithi028.stdout: "config_checks_enabled": { 2024-04-23T14:58:27.974 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_checks_enabled", 2024-04-23T14:58:27.974 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.974 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.974 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.975 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:27.975 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.975 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.975 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.975 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-23T14:58:27.975 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.975 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.975 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.976 INFO:teuthology.orchestra.run.smithi028.stdout: "config_dashboard": { 2024-04-23T14:58:27.976 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_dashboard", 2024-04-23T14:58:27.976 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:27.976 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.976 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.976 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:27.976 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.976 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.977 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.977 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-23T14:58:27.977 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.977 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.977 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.977 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_alertmanager": { 2024-04-23T14:58:27.977 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_alertmanager", 2024-04-23T14:58:27.977 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.978 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.978 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.978 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-23T14:58:27.978 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.978 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.978 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.978 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-23T14:58:27.978 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.979 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.979 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.979 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_base": { 2024-04-23T14:58:27.979 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_base", 2024-04-23T14:58:27.979 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.979 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.979 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:27.979 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-23T14:58:27.980 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.980 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.980 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.980 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Container image name, without the tag", 2024-04-23T14:58:27.980 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.980 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.980 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.981 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_elasticsearch": { 2024-04-23T14:58:27.981 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_elasticsearch", 2024-04-23T14:58:27.981 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.981 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.981 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.981 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-23T14:58:27.981 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.981 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.981 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.982 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "elasticsearch container image", 2024-04-23T14:58:27.982 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.982 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.982 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.982 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_grafana": { 2024-04-23T14:58:27.982 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_grafana", 2024-04-23T14:58:27.982 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.983 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.983 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.983 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-23T14:58:27.983 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.983 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.983 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.983 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-23T14:58:27.983 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.983 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.984 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.984 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_haproxy": { 2024-04-23T14:58:27.984 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_haproxy", 2024-04-23T14:58:27.984 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.984 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.984 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.984 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-23T14:58:27.984 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.985 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.985 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.985 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "HAproxy container image", 2024-04-23T14:58:27.985 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.985 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.985 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.985 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_agent": { 2024-04-23T14:58:27.986 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_agent", 2024-04-23T14:58:27.986 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.986 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.986 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.986 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-23T14:58:27.986 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.986 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.986 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.987 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger agent container image", 2024-04-23T14:58:27.987 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.987 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.987 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.987 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_collector": { 2024-04-23T14:58:27.987 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_collector", 2024-04-23T14:58:27.987 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.988 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.988 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.988 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-23T14:58:27.988 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.988 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.988 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.988 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger collector container image", 2024-04-23T14:58:27.988 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.988 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.989 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.989 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_query": { 2024-04-23T14:58:27.989 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_query", 2024-04-23T14:58:27.989 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.989 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.989 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.989 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-23T14:58:27.989 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.990 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.990 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.990 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger query container image", 2024-04-23T14:58:27.990 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.990 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.990 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.990 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_keepalived": { 2024-04-23T14:58:27.991 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_keepalived", 2024-04-23T14:58:27.991 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.991 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.991 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.991 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-23T14:58:27.991 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.991 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.991 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.992 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Keepalived container image", 2024-04-23T14:58:27.992 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.992 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.992 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.992 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_loki": { 2024-04-23T14:58:27.992 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_loki", 2024-04-23T14:58:27.992 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.992 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.993 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.993 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-23T14:58:27.993 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.993 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.993 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.993 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Loki container image", 2024-04-23T14:58:27.993 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.994 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.994 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.994 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_node_exporter": { 2024-04-23T14:58:27.994 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_node_exporter", 2024-04-23T14:58:27.994 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.994 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.994 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.995 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-23T14:58:27.995 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.995 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.995 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.995 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-23T14:58:27.995 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.995 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.995 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.996 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nvmeof": { 2024-04-23T14:58:27.996 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nvmeof", 2024-04-23T14:58:27.996 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.996 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.996 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.996 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-23T14:58:27.996 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.996 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.997 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.997 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nvme-of container image", 2024-04-23T14:58:27.997 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.997 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.997 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.997 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_prometheus": { 2024-04-23T14:58:27.997 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_prometheus", 2024-04-23T14:58:27.998 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.998 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:27.998 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:27.998 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-23T14:58:27.998 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:27.998 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:27.998 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:27.998 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-23T14:58:27.999 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:27.999 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:27.999 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:27.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:27.999 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_promtail": { 2024-04-23T14:58:27.999 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_promtail", 2024-04-23T14:58:27.999 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:27.999 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.000 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.000 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-23T14:58:28.000 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.000 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.000 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.000 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Promtail container image", 2024-04-23T14:58:28.000 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.000 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.000 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.001 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_samba": { 2024-04-23T14:58:28.001 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_samba", 2024-04-23T14:58:28.001 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.001 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.001 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.001 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-23T14:58:28.001 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.001 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.002 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.002 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Samba/SMB container image", 2024-04-23T14:58:28.002 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.002 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.002 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.002 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_snmp_gateway": { 2024-04-23T14:58:28.002 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_snmp_gateway", 2024-04-23T14:58:28.003 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.003 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.003 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.003 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-23T14:58:28.003 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.003 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.003 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.003 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SNMP Gateway container image", 2024-04-23T14:58:28.004 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.004 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.004 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.004 INFO:teuthology.orchestra.run.smithi028.stdout: "container_init": { 2024-04-23T14:58:28.004 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_init", 2024-04-23T14:58:28.004 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.004 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.005 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.005 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:28.005 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.005 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.005 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.005 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Run podman/docker with `--init`", 2024-04-23T14:58:28.005 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.005 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.006 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.006 INFO:teuthology.orchestra.run.smithi028.stdout: "daemon_cache_timeout": { 2024-04-23T14:58:28.006 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "daemon_cache_timeout", 2024-04-23T14:58:28.006 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:28.006 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.006 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.007 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-23T14:58:28.007 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.007 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.007 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.007 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-23T14:58:28.007 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.007 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.007 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.008 INFO:teuthology.orchestra.run.smithi028.stdout: "default_cephadm_command_timeout": { 2024-04-23T14:58:28.008 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_cephadm_command_timeout", 2024-04-23T14:58:28.008 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.008 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.008 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.008 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "900", 2024-04-23T14:58:28.008 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.008 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.009 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.009 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-23T14:58:28.009 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.009 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.009 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.009 INFO:teuthology.orchestra.run.smithi028.stdout: "default_registry": { 2024-04-23T14:58:28.009 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_registry", 2024-04-23T14:58:28.010 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.010 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.010 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.010 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io", 2024-04-23T14:58:28.010 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.010 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.010 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.010 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-23T14:58:28.010 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.011 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.011 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.011 INFO:teuthology.orchestra.run.smithi028.stdout: "device_cache_timeout": { 2024-04-23T14:58:28.011 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_cache_timeout", 2024-04-23T14:58:28.011 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:28.011 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.011 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.011 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1800", 2024-04-23T14:58:28.012 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.012 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.012 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.012 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache device inventory", 2024-04-23T14:58:28.012 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.012 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.012 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.012 INFO:teuthology.orchestra.run.smithi028.stdout: "device_enhanced_scan": { 2024-04-23T14:58:28.013 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_enhanced_scan", 2024-04-23T14:58:28.013 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.013 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.013 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.013 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.013 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.013 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.013 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.013 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-23T14:58:28.014 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.014 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.014 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.014 INFO:teuthology.orchestra.run.smithi028.stdout: "facts_cache_timeout": { 2024-04-23T14:58:28.014 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "facts_cache_timeout", 2024-04-23T14:58:28.014 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:28.014 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.015 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.015 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-23T14:58:28.015 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.015 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.015 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.015 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache host facts data", 2024-04-23T14:58:28.015 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.015 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.015 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.016 INFO:teuthology.orchestra.run.smithi028.stdout: "grafana_dashboards_path": { 2024-04-23T14:58:28.016 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "grafana_dashboards_path", 2024-04-23T14:58:28.016 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.016 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.016 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.016 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-23T14:58:28.016 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.017 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.017 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.017 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-23T14:58:28.017 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.017 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.017 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.018 INFO:teuthology.orchestra.run.smithi028.stdout: "host_check_interval": { 2024-04-23T14:58:28.018 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "host_check_interval", 2024-04-23T14:58:28.018 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:28.018 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.018 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.018 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-23T14:58:28.018 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.018 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.019 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.019 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to perform a host check", 2024-04-23T14:58:28.019 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.019 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.019 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.019 INFO:teuthology.orchestra.run.smithi028.stdout: "hw_monitoring": { 2024-04-23T14:58:28.019 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hw_monitoring", 2024-04-23T14:58:28.019 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.020 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.020 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.020 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.020 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.020 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.020 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.020 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-23T14:58:28.020 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.020 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.021 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.021 INFO:teuthology.orchestra.run.smithi028.stdout: "inventory_list_all": { 2024-04-23T14:58:28.021 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "inventory_list_all", 2024-04-23T14:58:28.021 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.021 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.021 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.021 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.022 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.022 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.022 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.022 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-23T14:58:28.022 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.022 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.022 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.022 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:28.023 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:28.023 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.023 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.023 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.023 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.023 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.023 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.023 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.023 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.024 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.024 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.024 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.024 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.024 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.024 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.024 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.024 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.025 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.025 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.025 INFO:teuthology.orchestra.run.smithi028.stdout: "log_refresh_metadata": { 2024-04-23T14:58:28.025 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_refresh_metadata", 2024-04-23T14:58:28.025 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.025 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.025 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.025 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.026 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.026 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.026 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.026 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-23T14:58:28.026 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.026 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.026 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.027 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:28.027 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:28.027 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.027 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.027 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.027 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:28.027 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.027 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.028 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.028 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-23T14:58:28.028 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.028 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.028 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.028 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:28.028 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:28.029 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.029 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.029 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.029 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:28.029 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.029 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.029 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.029 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.029 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.030 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.030 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.030 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.030 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.030 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.030 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.030 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.030 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.030 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.031 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:28.031 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:28.031 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.031 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.031 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.031 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.031 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.031 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.032 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.032 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.032 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.032 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.032 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.032 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-23T14:58:28.032 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-23T14:58:28.032 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.033 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.033 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.033 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.033 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.033 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.033 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.033 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-23T14:58:28.033 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.033 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.034 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.034 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-23T14:58:28.034 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-23T14:58:28.034 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.034 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.034 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.034 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "*", 2024-04-23T14:58:28.034 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.035 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.035 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.035 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-23T14:58:28.035 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.035 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.035 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.035 INFO:teuthology.orchestra.run.smithi028.stdout: "max_count_per_host": { 2024-04-23T14:58:28.036 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_count_per_host", 2024-04-23T14:58:28.036 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.036 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.036 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.036 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-23T14:58:28.036 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.036 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.036 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.037 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of daemons per service per host", 2024-04-23T14:58:28.037 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.037 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.037 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.037 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd_draining_count": { 2024-04-23T14:58:28.037 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_osd_draining_count", 2024-04-23T14:58:28.037 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.038 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.038 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.038 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-23T14:58:28.038 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.038 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.038 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.038 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-23T14:58:28.038 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.039 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.039 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.039 INFO:teuthology.orchestra.run.smithi028.stdout: "migration_current": { 2024-04-23T14:58:28.039 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "migration_current", 2024-04-23T14:58:28.039 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.039 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.039 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.039 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.040 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.040 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.040 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.040 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "internal - do not modify", 2024-04-23T14:58:28.040 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.040 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.040 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.041 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-04-23T14:58:28.041 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-04-23T14:58:28.041 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.041 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.041 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.041 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "root", 2024-04-23T14:58:28.041 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.041 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.041 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.042 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm-package", 2024-04-23T14:58:28.042 INFO:teuthology.orchestra.run.smithi028.stdout: "root" 2024-04-23T14:58:28.042 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.042 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "mode for remote execution of cephadm", 2024-04-23T14:58:28.042 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.042 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.042 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.043 INFO:teuthology.orchestra.run.smithi028.stdout: "oob_default_addr": { 2024-04-23T14:58:28.043 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "oob_default_addr", 2024-04-23T14:58:28.043 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.043 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.043 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.043 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "169.254.1.1", 2024-04-23T14:58:28.043 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.043 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.044 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.044 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-23T14:58:28.044 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.044 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.044 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.044 INFO:teuthology.orchestra.run.smithi028.stdout: "prometheus_alerts_path": { 2024-04-23T14:58:28.044 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus_alerts_path", 2024-04-23T14:58:28.044 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.045 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.045 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.045 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-23T14:58:28.045 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.045 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.045 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.045 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-23T14:58:28.045 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.045 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.046 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.046 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_insecure": { 2024-04-23T14:58:28.046 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_insecure", 2024-04-23T14:58:28.046 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.046 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.046 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.046 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.047 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.047 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.047 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.047 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-23T14:58:28.047 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.047 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.047 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.047 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_password": { 2024-04-23T14:58:28.048 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_password", 2024-04-23T14:58:28.048 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.048 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.048 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.048 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.048 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.048 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.048 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.049 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-23T14:58:28.049 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.049 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.049 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.049 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_url": { 2024-04-23T14:58:28.049 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_url", 2024-04-23T14:58:28.049 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.050 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.050 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.050 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.050 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.050 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.050 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.050 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-23T14:58:28.050 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.051 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.051 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.051 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_username": { 2024-04-23T14:58:28.051 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_username", 2024-04-23T14:58:28.051 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.051 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.051 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.051 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.052 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.052 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.052 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.052 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-23T14:58:28.052 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.052 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.052 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.052 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.052 INFO:teuthology.orchestra.run.smithi028.stdout: "secure_monitoring_stack": { 2024-04-23T14:58:28.053 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secure_monitoring_stack", 2024-04-23T14:58:28.053 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.053 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.053 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.053 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.053 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.053 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.053 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.053 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-23T14:58:28.054 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.054 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.054 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.054 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.054 INFO:teuthology.orchestra.run.smithi028.stdout: "service_discovery_port": { 2024-04-23T14:58:28.054 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "service_discovery_port", 2024-04-23T14:58:28.054 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.054 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.054 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.055 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8765", 2024-04-23T14:58:28.055 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.055 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.055 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.055 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "cephadm service discovery port", 2024-04-23T14:58:28.055 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.055 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.055 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.056 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_config_file": { 2024-04-23T14:58:28.056 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_config_file", 2024-04-23T14:58:28.056 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.056 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.056 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.056 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.056 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.056 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.057 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.057 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-23T14:58:28.057 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.057 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.057 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.057 INFO:teuthology.orchestra.run.smithi028.stdout: "use_agent": { 2024-04-23T14:58:28.057 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_agent", 2024-04-23T14:58:28.057 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.058 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.058 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.058 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.058 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.058 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.058 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.058 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-23T14:58:28.058 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.059 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.059 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.059 INFO:teuthology.orchestra.run.smithi028.stdout: "use_repo_digest": { 2024-04-23T14:58:28.059 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_repo_digest", 2024-04-23T14:58:28.059 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.059 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.059 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.060 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:28.060 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.060 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.060 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.060 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-23T14:58:28.060 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.060 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.060 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.061 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_failed_host_check": { 2024-04-23T14:58:28.061 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_failed_host_check", 2024-04-23T14:58:28.061 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.061 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.061 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.061 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:28.061 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.062 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.062 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.062 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if the host check fails", 2024-04-23T14:58:28.062 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.062 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.062 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.062 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_daemons": { 2024-04-23T14:58:28.063 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_daemons", 2024-04-23T14:58:28.063 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.063 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.063 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.063 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:28.063 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.063 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.063 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.064 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-23T14:58:28.064 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.064 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.064 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.064 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_hosts": { 2024-04-23T14:58:28.064 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_hosts", 2024-04-23T14:58:28.064 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.064 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.065 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.065 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:28.065 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.065 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.065 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.065 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-23T14:58:28.065 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.065 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.066 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.066 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.066 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.066 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crash", 2024-04-23T14:58:28.066 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:28.067 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:28.067 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:28.067 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:28.067 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:28.067 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.067 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.067 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.067 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.068 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.068 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.068 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.068 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.068 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.068 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.068 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.068 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.068 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.069 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.069 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.069 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.069 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.069 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.069 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:28.069 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:28.070 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.070 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.070 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.070 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.070 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.070 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.070 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.071 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.071 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.071 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.071 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.071 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:28.071 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:28.071 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.072 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.072 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.072 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:28.072 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.072 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.072 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.072 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.072 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.072 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.073 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.073 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.073 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.073 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.073 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.073 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.073 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.073 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.074 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:28.074 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:28.074 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.074 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.074 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.074 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.074 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.075 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.075 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.075 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.075 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.075 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.075 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.075 INFO:teuthology.orchestra.run.smithi028.stdout: "retain_interval": { 2024-04-23T14:58:28.075 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retain_interval", 2024-04-23T14:58:28.076 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:28.076 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.076 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.076 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "31536000", 2024-04-23T14:58:28.076 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.076 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.076 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.076 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-23T14:58:28.077 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.077 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.077 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.077 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_recent_interval": { 2024-04-23T14:58:28.077 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_recent_interval", 2024-04-23T14:58:28.077 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:28.077 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.077 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.078 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1209600", 2024-04-23T14:58:28.078 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.078 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.078 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.078 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-23T14:58:28.078 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.078 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.078 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.079 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.079 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.079 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.079 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dashboard", 2024-04-23T14:58:28.079 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:28.079 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:28.079 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:28.080 INFO:teuthology.orchestra.run.smithi028.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-23T14:58:28.080 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-23T14:58:28.080 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.080 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.080 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.080 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-23T14:58:28.080 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.080 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.081 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.081 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.081 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.081 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.081 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.081 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_HOST": { 2024-04-23T14:58:28.081 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-23T14:58:28.081 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.082 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.082 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.082 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.082 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.082 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.082 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.082 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.083 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.083 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.083 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.083 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-23T14:58:28.083 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-23T14:58:28.083 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.083 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.084 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.084 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:28.084 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.084 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.084 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.084 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.084 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.084 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.085 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.085 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_ENABLED": { 2024-04-23T14:58:28.085 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_ENABLED", 2024-04-23T14:58:28.085 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.085 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.085 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.085 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.086 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.086 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.086 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.086 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.086 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.086 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.086 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.087 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-23T14:58:28.087 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-23T14:58:28.087 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.087 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.087 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.087 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:28.087 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.087 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.087 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.088 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.088 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.088 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.088 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.088 INFO:teuthology.orchestra.run.smithi028.stdout: "ENABLE_BROWSABLE_API": { 2024-04-23T14:58:28.088 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-23T14:58:28.088 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.089 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.089 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.089 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:28.089 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.089 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.089 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.089 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.089 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.089 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.090 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.090 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-23T14:58:28.090 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-23T14:58:28.090 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.090 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.090 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.090 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:28.091 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.091 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.091 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.091 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.091 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.091 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.091 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.092 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-23T14:58:28.092 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-23T14:58:28.092 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.092 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.092 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.092 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:28.092 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.092 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.093 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.093 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.093 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.093 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.093 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.093 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-23T14:58:28.093 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-23T14:58:28.094 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.094 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.094 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.094 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:28.094 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.094 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.094 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.094 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.095 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.095 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.095 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.095 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-23T14:58:28.095 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-23T14:58:28.095 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.095 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.096 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.096 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:28.096 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.096 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.096 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.096 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.096 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.096 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.097 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.097 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-23T14:58:28.097 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-23T14:58:28.097 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.097 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.097 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.097 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:28.098 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.098 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.098 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.098 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.098 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.098 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.098 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.099 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-23T14:58:28.099 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-23T14:58:28.099 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.099 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.099 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.099 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:28.099 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.099 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.099 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.100 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.100 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.100 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.100 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.100 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-23T14:58:28.100 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-23T14:58:28.100 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.101 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.101 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.101 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:28.101 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.101 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.101 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.101 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.101 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.102 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.102 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.102 INFO:teuthology.orchestra.run.smithi028.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-23T14:58:28.102 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-23T14:58:28.102 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.102 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.102 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.102 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.103 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.103 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.103 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.103 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.103 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.103 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.103 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.104 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_PASSWORD": { 2024-04-23T14:58:28.104 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-23T14:58:28.104 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.104 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.104 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.104 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-23T14:58:28.104 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.104 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.104 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.105 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.105 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.105 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.105 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.105 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-23T14:58:28.105 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-23T14:58:28.105 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.105 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.106 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.106 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:28.106 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.106 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.106 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.106 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.106 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.106 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.107 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.107 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_URL": { 2024-04-23T14:58:28.107 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_URL", 2024-04-23T14:58:28.107 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.107 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.107 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.107 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.108 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.108 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.108 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.108 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.108 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.108 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.108 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.108 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_USERNAME": { 2024-04-23T14:58:28.109 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-23T14:58:28.109 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.109 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.109 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.109 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-23T14:58:28.109 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.109 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.109 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.109 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.110 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.110 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.110 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.110 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-23T14:58:28.110 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-23T14:58:28.110 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.110 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.111 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.111 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.111 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.111 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.111 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.111 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.111 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.111 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.111 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.112 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-23T14:58:28.112 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-23T14:58:28.112 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.112 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.112 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.112 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.112 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.112 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.113 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.113 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.113 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.113 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.113 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.113 INFO:teuthology.orchestra.run.smithi028.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-23T14:58:28.113 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-23T14:58:28.114 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.114 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.114 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.114 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:28.114 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.114 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.114 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.114 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.115 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.115 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.115 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.115 INFO:teuthology.orchestra.run.smithi028.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-23T14:58:28.115 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-23T14:58:28.115 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.115 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.116 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.116 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.116 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.116 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.116 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.116 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.116 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.116 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.117 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.117 INFO:teuthology.orchestra.run.smithi028.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-23T14:58:28.117 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-23T14:58:28.117 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.117 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.117 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.117 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "[]", 2024-04-23T14:58:28.118 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.118 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.118 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.118 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.118 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.118 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.118 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.119 INFO:teuthology.orchestra.run.smithi028.stdout: "MULTICLUSTER_CONFIG": { 2024-04-23T14:58:28.119 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-23T14:58:28.119 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.119 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.119 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.119 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "{}", 2024-04-23T14:58:28.119 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.120 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.120 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.120 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.120 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.120 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.120 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.120 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_HOST": { 2024-04-23T14:58:28.120 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-23T14:58:28.120 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.121 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.121 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.121 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.121 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.121 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.121 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.121 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.121 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.122 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.122 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.122 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-23T14:58:28.122 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-23T14:58:28.122 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.122 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.122 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.123 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:28.123 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.123 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.123 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.123 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.123 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.123 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.123 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.124 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-23T14:58:28.124 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-23T14:58:28.124 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.124 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.124 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.124 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.124 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.124 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.125 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.125 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.125 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.125 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.125 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.125 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-23T14:58:28.125 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-23T14:58:28.126 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.126 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.126 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.126 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.126 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.126 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.126 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.126 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.127 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.127 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.127 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.127 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-23T14:58:28.127 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-23T14:58:28.127 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.127 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.128 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.128 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:28.128 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.128 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.128 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.128 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.128 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.128 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.129 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.129 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-23T14:58:28.129 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-23T14:58:28.129 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.129 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.129 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.129 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:28.129 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.130 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.130 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.130 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.130 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.130 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.130 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.131 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-23T14:58:28.131 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-23T14:58:28.131 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.131 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.131 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.131 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.131 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.131 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.131 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.132 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.132 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.132 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.132 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.132 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-23T14:58:28.132 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-23T14:58:28.132 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.133 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.133 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.133 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.133 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.133 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.133 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.133 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.133 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.133 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.134 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.134 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-23T14:58:28.134 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-23T14:58:28.134 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.134 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.134 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.134 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.135 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.135 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.135 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.135 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.137 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.137 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.137 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.137 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_ENABLED": { 2024-04-23T14:58:28.138 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-23T14:58:28.138 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.138 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.138 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.138 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:28.138 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.138 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.138 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.138 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.139 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.139 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.139 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.139 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-23T14:58:28.139 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-23T14:58:28.139 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.139 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.140 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.140 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-23T14:58:28.140 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.140 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.140 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.140 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.140 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.140 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.140 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.141 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-23T14:58:28.141 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-23T14:58:28.141 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.141 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.141 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.141 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-23T14:58:28.141 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.142 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.142 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.142 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.142 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.142 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.142 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.142 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-23T14:58:28.143 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-23T14:58:28.143 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.143 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.143 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.143 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8", 2024-04-23T14:58:28.143 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.143 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.143 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.143 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.144 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.144 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.144 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.144 INFO:teuthology.orchestra.run.smithi028.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-23T14:58:28.144 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-23T14:58:28.144 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.144 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.145 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.145 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "45", 2024-04-23T14:58:28.145 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.145 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.145 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.145 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.145 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.145 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.145 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.146 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ACCESS_KEY": { 2024-04-23T14:58:28.146 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-23T14:58:28.146 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.146 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.146 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.146 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.146 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.147 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.147 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.147 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.147 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.147 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.147 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.147 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-23T14:58:28.147 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-23T14:58:28.148 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.148 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.148 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.148 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-23T14:58:28.148 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.148 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.148 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.148 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.149 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.149 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.149 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.149 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SECRET_KEY": { 2024-04-23T14:58:28.149 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-23T14:58:28.149 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.149 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.150 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.150 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.150 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.150 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.150 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.150 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.150 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.150 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.151 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.151 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SSL_VERIFY": { 2024-04-23T14:58:28.151 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-23T14:58:28.151 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.151 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.151 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.151 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:28.151 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.152 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.152 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.152 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.152 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.152 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.152 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.152 INFO:teuthology.orchestra.run.smithi028.stdout: "UNSAFE_TLS_v1_2": { 2024-04-23T14:58:28.152 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-23T14:58:28.153 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.153 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.153 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.153 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.153 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.153 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.153 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.153 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.154 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.154 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.154 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.154 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-23T14:58:28.154 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-23T14:58:28.154 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.154 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.155 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.155 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-23T14:58:28.155 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.155 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.155 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.155 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.155 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.155 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.155 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.156 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-23T14:58:28.156 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-23T14:58:28.156 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.156 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.156 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.156 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-23T14:58:28.156 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.157 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.157 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.157 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.157 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.157 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.157 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.157 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-23T14:58:28.157 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-23T14:58:28.158 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.158 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.158 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.158 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-23T14:58:28.158 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.158 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.158 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.158 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.159 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.159 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.159 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.159 INFO:teuthology.orchestra.run.smithi028.stdout: "cross_origin_url": { 2024-04-23T14:58:28.159 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cross_origin_url", 2024-04-23T14:58:28.159 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.159 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.159 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.160 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.160 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.160 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.160 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.160 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.160 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.160 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.160 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.161 INFO:teuthology.orchestra.run.smithi028.stdout: "crt_file": { 2024-04-23T14:58:28.161 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crt_file", 2024-04-23T14:58:28.161 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.161 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.161 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.161 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.161 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.161 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.162 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.162 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.162 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.162 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.162 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.162 INFO:teuthology.orchestra.run.smithi028.stdout: "debug": { 2024-04-23T14:58:28.162 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "debug", 2024-04-23T14:58:28.163 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.163 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.163 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.163 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.163 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.163 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.163 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.163 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable/disable debug options", 2024-04-23T14:58:28.163 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.164 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.164 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.164 INFO:teuthology.orchestra.run.smithi028.stdout: "jwt_token_ttl": { 2024-04-23T14:58:28.164 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "jwt_token_ttl", 2024-04-23T14:58:28.164 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.164 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.164 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.164 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "28800", 2024-04-23T14:58:28.165 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.165 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.165 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.165 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.165 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.165 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.165 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.166 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-04-23T14:58:28.166 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-04-23T14:58:28.166 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.166 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.166 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.166 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.166 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.166 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.166 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.167 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.167 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.167 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.167 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.167 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:28.168 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:28.168 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.168 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.168 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.168 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.168 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.168 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.168 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.168 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.169 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.169 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.169 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.169 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.169 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.169 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.169 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.169 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.169 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.170 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.170 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:28.170 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:28.170 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.170 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.170 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.170 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.171 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.171 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.171 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.171 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.171 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.171 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.171 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.171 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:28.172 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:28.172 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.172 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.172 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.172 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:28.172 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.172 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.172 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.172 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.173 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.173 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.173 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.173 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.173 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.173 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.173 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.173 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.173 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.174 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.174 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:28.174 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:28.174 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.174 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.174 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.174 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.174 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.175 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.175 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.175 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.175 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.175 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.175 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.175 INFO:teuthology.orchestra.run.smithi028.stdout: "motd": { 2024-04-23T14:58:28.175 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "motd", 2024-04-23T14:58:28.176 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.176 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.176 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.176 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.176 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.176 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.176 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.176 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "The message of the day", 2024-04-23T14:58:28.176 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.177 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.177 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.177 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect_resolve_ip_addr": { 2024-04-23T14:58:28.177 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "redirect_resolve_ip_addr", 2024-04-23T14:58:28.177 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.177 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.177 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.177 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.178 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.178 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.178 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.178 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.178 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.178 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.178 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.179 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-23T14:58:28.179 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-23T14:58:28.179 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.179 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.179 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.179 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-04-23T14:58:28.179 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.179 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.179 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.179 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.180 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.180 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.180 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.180 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-23T14:58:28.180 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-23T14:58:28.180 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.180 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.180 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.181 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8080", 2024-04-23T14:58:28.181 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.181 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.181 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.181 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.181 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.181 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.181 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.182 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-04-23T14:58:28.182 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-04-23T14:58:28.182 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.182 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.182 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.182 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:28.182 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.182 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.183 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.183 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.183 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.183 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.183 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.183 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl_server_port": { 2024-04-23T14:58:28.183 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl_server_port", 2024-04-23T14:58:28.183 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.184 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.184 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.184 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8443", 2024-04-23T14:58:28.184 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.184 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.184 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.184 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.184 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.184 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.185 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.185 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-04-23T14:58:28.185 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-04-23T14:58:28.185 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.185 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.185 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.185 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "redirect", 2024-04-23T14:58:28.185 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.186 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.186 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.186 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.186 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect" 2024-04-23T14:58:28.186 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.186 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.186 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.186 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.186 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.187 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-04-23T14:58:28.187 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-04-23T14:58:28.187 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.187 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.187 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.187 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-04-23T14:58:28.187 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-04-23T14:58:28.187 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-04-23T14:58:28.188 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.188 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.188 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.188 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.188 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.188 INFO:teuthology.orchestra.run.smithi028.stdout: "url_prefix": { 2024-04-23T14:58:28.188 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url_prefix", 2024-04-23T14:58:28.189 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.189 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.189 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.189 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.189 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.189 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.189 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.189 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.189 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.189 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.190 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.190 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.190 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.190 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-04-23T14:58:28.190 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:28.191 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:28.191 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:28.191 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_monitoring": { 2024-04-23T14:58:28.191 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_monitoring", 2024-04-23T14:58:28.191 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.191 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.191 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.191 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:28.191 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.192 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.192 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.192 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "monitor device health metrics", 2024-04-23T14:58:28.192 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.192 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.192 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.192 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:28.193 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:28.193 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.193 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.193 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.193 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.193 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.193 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.193 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.193 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.194 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.194 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.194 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.194 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.194 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.194 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.194 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.194 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.194 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.195 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.195 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:28.195 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:28.195 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.195 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.195 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.195 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.195 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.196 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.196 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.196 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.196 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.196 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.196 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.196 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:28.196 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:28.197 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.197 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.197 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.197 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:28.197 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.197 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.197 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.197 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.197 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.198 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.198 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.198 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.198 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.198 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.198 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.198 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.198 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.199 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.199 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:28.199 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:28.199 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.199 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.199 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.199 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.199 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.200 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.200 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.200 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.200 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.200 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.200 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.200 INFO:teuthology.orchestra.run.smithi028.stdout: "mark_out_threshold": { 2024-04-23T14:58:28.200 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mark_out_threshold", 2024-04-23T14:58:28.201 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:28.201 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.201 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.201 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2419200", 2024-04-23T14:58:28.201 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.201 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.201 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.201 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-23T14:58:28.202 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.202 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.202 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.202 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": { 2024-04-23T14:58:28.202 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_name", 2024-04-23T14:58:28.202 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.202 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.202 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.203 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "device_health_metrics", 2024-04-23T14:58:28.203 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.203 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.203 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.203 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-23T14:58:28.203 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.203 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.203 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.204 INFO:teuthology.orchestra.run.smithi028.stdout: "retention_period": { 2024-04-23T14:58:28.204 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retention_period", 2024-04-23T14:58:28.204 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:28.204 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.204 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.204 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15552000", 2024-04-23T14:58:28.204 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.204 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.204 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.205 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain device health metrics", 2024-04-23T14:58:28.205 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.205 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.205 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.205 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_frequency": { 2024-04-23T14:58:28.205 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_frequency", 2024-04-23T14:58:28.205 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:28.205 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.206 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.206 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-04-23T14:58:28.206 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.206 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.206 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.206 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-23T14:58:28.206 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.206 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.207 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.207 INFO:teuthology.orchestra.run.smithi028.stdout: "self_heal": { 2024-04-23T14:58:28.207 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "self_heal", 2024-04-23T14:58:28.207 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.207 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.207 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.207 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:28.207 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.208 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.208 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.208 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-23T14:58:28.208 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.208 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.208 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.208 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-23T14:58:28.208 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-23T14:58:28.209 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:28.209 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.209 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.209 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-23T14:58:28.209 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.209 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.209 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.209 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and check device health", 2024-04-23T14:58:28.209 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.210 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.210 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.210 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_threshold": { 2024-04-23T14:58:28.210 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_threshold", 2024-04-23T14:58:28.210 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:28.210 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.210 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.210 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7257600", 2024-04-23T14:58:28.211 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.211 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.211 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.211 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-23T14:58:28.211 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.211 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.211 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.211 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.212 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.212 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.212 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "diskprediction_local", 2024-04-23T14:58:28.212 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:28.212 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:28.212 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:28.212 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:28.212 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:28.213 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.213 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.213 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.213 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.213 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.213 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.213 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.213 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.214 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.214 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.214 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.214 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.214 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.214 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.214 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.214 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.215 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.215 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.215 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:28.215 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:28.215 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.215 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.215 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.215 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.216 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.216 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.216 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.216 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.216 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.216 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.216 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.217 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:28.217 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:28.217 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.217 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.217 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.217 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:28.217 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.217 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.217 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.218 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.218 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.218 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.218 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.218 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.218 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.218 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.218 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.219 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.219 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.219 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.219 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:28.219 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:28.219 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.219 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.219 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.220 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.220 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.220 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.220 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.220 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.220 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.220 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.220 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.221 INFO:teuthology.orchestra.run.smithi028.stdout: "predict_interval": { 2024-04-23T14:58:28.221 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predict_interval", 2024-04-23T14:58:28.221 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.221 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.221 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.221 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-04-23T14:58:28.221 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.221 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.222 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.222 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.222 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.222 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.222 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.222 INFO:teuthology.orchestra.run.smithi028.stdout: "predictor_model": { 2024-04-23T14:58:28.222 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predictor_model", 2024-04-23T14:58:28.223 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.223 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.223 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.223 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "prophetstor", 2024-04-23T14:58:28.223 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.223 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.223 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.223 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.223 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.224 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.224 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.224 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-23T14:58:28.224 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-23T14:58:28.224 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.224 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.224 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.225 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-23T14:58:28.225 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.225 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.225 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.225 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.225 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.225 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.225 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.225 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.226 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.226 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "influx", 2024-04-23T14:58:28.226 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": false, 2024-04-23T14:58:28.226 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "influxdb python module not found", 2024-04-23T14:58:28.226 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:28.226 INFO:teuthology.orchestra.run.smithi028.stdout: "batch_size": { 2024-04-23T14:58:28.227 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "batch_size", 2024-04-23T14:58:28.227 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.227 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.227 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.227 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5000", 2024-04-23T14:58:28.227 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.227 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.227 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.227 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-23T14:58:28.228 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.228 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.228 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.228 INFO:teuthology.orchestra.run.smithi028.stdout: "database": { 2024-04-23T14:58:28.228 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "database", 2024-04-23T14:58:28.228 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.228 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.229 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.229 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "ceph", 2024-04-23T14:58:28.229 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.229 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.229 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.229 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-23T14:58:28.229 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.229 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.229 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.230 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": { 2024-04-23T14:58:28.230 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hostname", 2024-04-23T14:58:28.230 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.230 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.230 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.230 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.230 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.231 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.231 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.231 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server hostname", 2024-04-23T14:58:28.231 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.231 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.231 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.231 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-23T14:58:28.231 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-23T14:58:28.232 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:28.232 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.232 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.232 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "30", 2024-04-23T14:58:28.232 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "5", 2024-04-23T14:58:28.232 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.232 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.232 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-23T14:58:28.232 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.233 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.233 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.233 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:28.233 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:28.233 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.233 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.233 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.233 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.234 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.234 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.234 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.234 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.234 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.234 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.234 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.234 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.235 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.235 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.235 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.235 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.235 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.235 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.235 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:28.235 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:28.236 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.236 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.236 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.236 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.236 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.236 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.236 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.236 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.236 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.237 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.237 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.237 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:28.237 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:28.237 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.237 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.237 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.237 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:28.238 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.238 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.238 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.238 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.238 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.238 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.238 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.238 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.239 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.239 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.239 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.239 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.239 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.239 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.239 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:28.239 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:28.239 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.240 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.240 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.240 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.240 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.240 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.240 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.240 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.240 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.240 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.241 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.241 INFO:teuthology.orchestra.run.smithi028.stdout: "password": { 2024-04-23T14:58:28.241 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "password", 2024-04-23T14:58:28.241 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.241 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.241 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.241 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.242 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.242 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.242 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.242 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "password of InfluxDB server user", 2024-04-23T14:58:28.242 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.242 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.242 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.243 INFO:teuthology.orchestra.run.smithi028.stdout: "port": { 2024-04-23T14:58:28.243 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "port", 2024-04-23T14:58:28.243 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.243 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.243 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.243 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8086", 2024-04-23T14:58:28.243 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.243 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.243 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.244 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server port", 2024-04-23T14:58:28.244 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.244 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.244 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.244 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-04-23T14:58:28.244 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-04-23T14:58:28.244 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.245 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.245 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.245 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "false", 2024-04-23T14:58:28.245 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.245 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.245 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.245 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-23T14:58:28.245 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.246 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.246 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.246 INFO:teuthology.orchestra.run.smithi028.stdout: "threads": { 2024-04-23T14:58:28.246 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threads", 2024-04-23T14:58:28.246 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.246 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.246 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.246 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-23T14:58:28.247 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-23T14:58:28.247 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "32", 2024-04-23T14:58:28.247 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.247 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-23T14:58:28.247 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.247 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.247 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.248 INFO:teuthology.orchestra.run.smithi028.stdout: "username": { 2024-04-23T14:58:28.248 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "username", 2024-04-23T14:58:28.248 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.248 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.248 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.248 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.248 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.248 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.249 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.249 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "username of InfluxDB server user", 2024-04-23T14:58:28.249 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.249 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.249 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.249 INFO:teuthology.orchestra.run.smithi028.stdout: "verify_ssl": { 2024-04-23T14:58:28.249 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "verify_ssl", 2024-04-23T14:58:28.249 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.250 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.250 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.250 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "true", 2024-04-23T14:58:28.250 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.250 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.250 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.250 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-23T14:58:28.250 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.251 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.251 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.251 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.251 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.251 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "insights", 2024-04-23T14:58:28.251 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:28.252 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:28.252 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:28.252 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:28.252 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:28.252 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.252 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.252 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.252 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.253 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.253 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.253 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.253 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.253 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.253 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.253 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.253 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.253 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.254 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.254 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.254 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.254 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.254 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.254 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:28.254 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:28.255 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.255 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.255 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.255 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.255 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.255 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.255 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.255 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.256 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.256 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.256 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.256 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:28.256 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:28.256 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.256 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.256 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.257 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:28.257 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.257 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.257 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.257 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.257 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.257 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.257 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.257 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.258 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.258 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.258 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.258 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.258 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.258 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.258 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:28.259 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:28.259 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.259 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.259 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.259 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.259 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.259 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.259 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.259 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.260 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.260 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.260 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.260 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.260 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.260 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.260 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "iostat", 2024-04-23T14:58:28.260 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:28.261 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:28.261 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:28.261 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:28.261 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:28.261 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.261 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.261 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.261 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.261 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.262 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.262 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.262 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.262 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.262 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.262 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.262 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.262 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.263 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.263 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.263 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.263 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.263 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.263 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:28.263 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:28.264 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.264 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.264 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.264 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.264 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.264 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.264 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.264 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.264 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.265 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.265 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.265 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:28.265 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:28.265 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.265 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.265 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.266 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:28.266 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.266 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.266 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.266 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.266 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.266 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.266 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.267 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.267 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.267 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.267 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.267 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.267 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.267 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.268 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:28.268 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:28.268 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.268 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.268 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.268 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.268 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.268 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.268 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.269 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.269 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.269 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.269 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.269 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.269 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.269 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.269 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "k8sevents", 2024-04-23T14:58:28.270 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:28.270 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:28.270 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:28.270 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_event_retention_days": { 2024-04-23T14:58:28.270 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ceph_event_retention_days", 2024-04-23T14:58:28.270 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.270 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.270 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.271 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7", 2024-04-23T14:58:28.271 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.271 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.271 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.271 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-23T14:58:28.271 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.271 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.271 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.272 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.272 INFO:teuthology.orchestra.run.smithi028.stdout: "config_check_secs": { 2024-04-23T14:58:28.272 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_check_secs", 2024-04-23T14:58:28.272 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.272 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.272 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.272 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-23T14:58:28.272 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "10", 2024-04-23T14:58:28.272 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.273 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.273 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-23T14:58:28.273 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.273 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.273 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.273 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:28.273 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:28.273 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.274 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.274 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.274 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.274 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.274 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.274 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.274 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.274 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.275 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.275 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.275 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.275 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.275 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.275 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.275 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.275 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.276 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.276 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:28.276 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:28.276 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.276 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.276 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.276 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.276 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.277 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.277 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.277 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.277 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.277 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.277 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.277 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:28.277 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:28.278 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.278 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.278 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.278 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:28.278 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.278 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.278 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.278 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.279 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.279 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.279 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.279 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.279 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.279 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.279 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.279 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.279 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.280 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.280 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:28.280 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:28.280 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.280 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.280 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.280 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.281 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.281 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.281 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.281 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.281 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.281 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.281 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.281 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.282 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.282 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.282 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "localpool", 2024-04-23T14:58:28.282 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:28.282 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:28.282 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:28.282 INFO:teuthology.orchestra.run.smithi028.stdout: "failure_domain": { 2024-04-23T14:58:28.283 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "failure_domain", 2024-04-23T14:58:28.283 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.283 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.283 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.283 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "host", 2024-04-23T14:58:28.283 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.283 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.283 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.283 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "failure domain for any created local pool", 2024-04-23T14:58:28.284 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-23T14:58:28.284 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.284 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.284 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:28.284 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:28.284 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.284 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.285 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.285 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.285 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.285 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.285 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.285 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.285 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.285 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.286 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.286 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.286 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.286 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.286 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.286 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.286 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.286 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.287 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:28.287 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:28.287 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.287 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.287 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.287 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.287 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.287 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.288 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.288 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.288 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.288 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.288 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.288 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:28.288 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:28.288 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.289 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.289 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.289 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:28.289 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.289 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.289 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.289 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.289 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.289 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.290 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.290 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.290 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.290 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.290 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.290 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.290 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.290 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.291 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.291 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:28.291 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:28.291 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.291 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.291 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.291 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.291 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.292 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.292 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.292 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.292 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.292 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.292 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.292 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.292 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": { 2024-04-23T14:58:28.292 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_size", 2024-04-23T14:58:28.293 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.293 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.293 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.293 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.293 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.293 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.293 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.293 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default min_size for any created local pool", 2024-04-23T14:58:28.293 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-23T14:58:28.294 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.294 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.294 INFO:teuthology.orchestra.run.smithi028.stdout: "num_rep": { 2024-04-23T14:58:28.294 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "num_rep", 2024-04-23T14:58:28.294 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.294 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.294 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.295 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3", 2024-04-23T14:58:28.295 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.295 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.295 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.295 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default replica count for any created local pool", 2024-04-23T14:58:28.295 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.295 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.295 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.296 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": { 2024-04-23T14:58:28.296 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_num", 2024-04-23T14:58:28.296 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.296 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.296 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.296 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "128", 2024-04-23T14:58:28.296 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.296 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.297 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.297 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default pg_num for any created local pool", 2024-04-23T14:58:28.297 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.297 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.297 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.297 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.297 INFO:teuthology.orchestra.run.smithi028.stdout: "prefix": { 2024-04-23T14:58:28.297 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prefix", 2024-04-23T14:58:28.297 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name prefix for any created local pool", 2024-04-23T14:58:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.299 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.299 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.299 INFO:teuthology.orchestra.run.smithi028.stdout: "subtree": { 2024-04-23T14:58:28.299 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "subtree", 2024-04-23T14:58:28.299 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.299 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.299 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.300 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "rack", 2024-04-23T14:58:28.300 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.300 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.300 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.300 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-23T14:58:28.300 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-23T14:58:28.300 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.300 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.300 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.301 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.301 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.301 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mds_autoscaler", 2024-04-23T14:58:28.301 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:28.301 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:28.301 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:28.301 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:28.302 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:28.302 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.302 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.302 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.302 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.302 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.302 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.302 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.302 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.303 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.303 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.303 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.303 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.303 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.303 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.303 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.303 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.304 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.304 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.304 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:28.304 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:28.304 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.304 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.304 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.304 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.306 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:28.306 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:28.306 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.306 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.306 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.306 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:28.306 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.306 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.307 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.307 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.307 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.307 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.307 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.307 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.307 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.307 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.307 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.308 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.308 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.308 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.308 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:28.308 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:28.308 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.308 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.309 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.309 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.309 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.309 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.309 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.309 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.309 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.309 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.309 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.310 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.310 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.310 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.310 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirroring", 2024-04-23T14:58:28.310 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:28.310 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:28.310 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:28.311 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:28.311 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:28.311 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.311 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.311 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.311 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.311 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.311 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.312 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.312 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.312 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.312 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.312 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.312 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.312 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.312 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.312 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.313 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.313 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.313 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.313 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:28.313 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:28.313 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.313 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.313 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.314 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.314 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.314 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.314 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.314 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.314 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.314 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.314 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.315 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:28.315 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:28.315 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.315 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.315 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.315 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:28.315 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.316 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.316 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.316 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.316 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.316 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.316 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.316 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.316 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.316 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.317 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.317 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.317 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.317 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.317 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:28.317 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:28.317 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.318 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.318 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.318 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.318 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.318 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.318 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.318 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.318 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.319 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.319 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.319 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.319 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.319 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.319 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "nfs", 2024-04-23T14:58:28.319 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:28.319 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:28.320 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:28.320 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:28.320 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:28.320 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.320 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.320 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.320 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.320 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.321 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.321 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.321 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.321 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.321 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.321 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.321 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.321 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.321 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.322 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.322 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.322 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.322 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.322 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.322 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:28.322 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:28.322 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.323 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.323 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.323 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.323 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.323 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.323 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.323 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.323 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.323 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.324 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.324 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:28.324 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:28.324 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.324 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.324 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.324 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:28.325 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.325 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.325 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.325 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.325 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.325 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.325 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.325 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.325 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.326 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.326 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.326 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.326 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.326 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.326 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:28.326 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:28.327 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.327 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.327 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.327 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.327 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.327 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.327 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.327 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.328 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.328 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.328 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.328 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.328 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.328 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.328 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.328 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-04-23T14:58:28.328 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:28.329 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:28.329 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:28.329 INFO:teuthology.orchestra.run.smithi028.stdout: "fail_fs": { 2024-04-23T14:58:28.329 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "fail_fs", 2024-04-23T14:58:28.329 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.329 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.329 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.329 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.330 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.330 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.330 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.330 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-23T14:58:28.330 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.330 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.330 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.330 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:28.331 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:28.331 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.331 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.331 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.331 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.331 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.331 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.331 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.332 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.332 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.332 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.332 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.332 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.332 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.332 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.332 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.332 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.333 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.333 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.333 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:28.333 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:28.333 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.333 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.333 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.333 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.334 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.334 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.334 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.334 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.334 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.334 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.334 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.335 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:28.335 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:28.335 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.335 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.335 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.335 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:28.335 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.335 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.335 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.336 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.336 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.336 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.336 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.336 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.336 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.336 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.336 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.337 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.337 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.337 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.337 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.337 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:28.337 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:28.337 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.337 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.337 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.338 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.338 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.338 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.338 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.338 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.338 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.338 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.338 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.339 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator": { 2024-04-23T14:58:28.339 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-04-23T14:58:28.339 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.339 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.339 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.339 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.339 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.340 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.340 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.341 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-04-23T14:58:28.341 INFO:teuthology.orchestra.run.smithi028.stdout: "rook", 2024-04-23T14:58:28.341 INFO:teuthology.orchestra.run.smithi028.stdout: "test_orchestrator" 2024-04-23T14:58:28.341 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.341 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Orchestrator backend", 2024-04-23T14:58:28.341 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.342 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.342 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.342 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.342 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.342 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.342 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.342 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_perf_query", 2024-04-23T14:58:28.342 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:28.343 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:28.343 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:28.343 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:28.343 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:28.343 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.343 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.343 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.343 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.343 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.344 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.344 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.344 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.344 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.344 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.344 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.344 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.345 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.345 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.345 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.345 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.345 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.345 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.345 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:28.345 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:28.346 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.346 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.346 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.346 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.346 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.346 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.346 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.347 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.347 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.347 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.347 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.347 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:28.347 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:28.347 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.347 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.348 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.348 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:28.348 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.348 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.348 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.348 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.348 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.348 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.348 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.349 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.349 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.349 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.349 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.349 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.349 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.349 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.349 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:28.350 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:28.350 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.350 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.350 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.350 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.350 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.350 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.350 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.351 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.351 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.351 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.351 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.351 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.351 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.351 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_support", 2024-04-23T14:58:28.352 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:28.352 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:28.352 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:28.352 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:28.352 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:28.352 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.352 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.352 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.352 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.353 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.353 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.353 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.353 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.353 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.353 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.353 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.353 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.354 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.354 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.354 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.354 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.354 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.354 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.354 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:28.355 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:28.355 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.355 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.355 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.355 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.355 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.355 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.355 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.355 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.356 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.356 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.356 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.356 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:28.356 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:28.356 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.357 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.357 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.357 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:28.357 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.357 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.357 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.357 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.357 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.358 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.358 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.358 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.358 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.358 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.358 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.358 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.358 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.359 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.359 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:28.359 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:28.359 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.359 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.359 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.359 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.359 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.360 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.360 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.360 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.360 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.360 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.360 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.360 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.360 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.361 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.361 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_autoscaler", 2024-04-23T14:58:28.361 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:28.361 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:28.361 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:28.361 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:28.361 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:28.361 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.362 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.362 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.362 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.362 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.362 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.362 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.362 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.362 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.363 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.363 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.363 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.363 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.363 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.363 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.363 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.363 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.364 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.364 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:28.364 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:28.364 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.364 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.364 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.364 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.364 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.365 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.365 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.365 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.365 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.365 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.365 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.365 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:28.366 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:28.366 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.366 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.366 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.366 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:28.366 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.366 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.366 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.367 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.367 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.367 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.367 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.367 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.367 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.367 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.367 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.368 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.368 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.368 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.368 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:28.368 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:28.368 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.368 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.368 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.369 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.369 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.369 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.369 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.369 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.369 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.369 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.369 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.370 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-23T14:58:28.370 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-23T14:58:28.370 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:28.370 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.370 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.370 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-23T14:58:28.370 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.371 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.371 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.371 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.371 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.371 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.371 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.371 INFO:teuthology.orchestra.run.smithi028.stdout: "threshold": { 2024-04-23T14:58:28.371 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threshold", 2024-04-23T14:58:28.372 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-23T14:58:28.372 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.372 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.372 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-04-23T14:58:28.372 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1.0", 2024-04-23T14:58:28.372 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.372 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.372 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "scaling threshold", 2024-04-23T14:58:28.372 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-23T14:58:28.373 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.373 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.373 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.373 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.373 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "progress", 2024-04-23T14:58:28.373 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:28.374 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:28.374 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:28.374 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_pg_recovery_event": { 2024-04-23T14:58:28.374 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_pg_recovery_event", 2024-04-23T14:58:28.374 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.374 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.374 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.374 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.375 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.375 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.375 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.375 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-23T14:58:28.375 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.375 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.375 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.376 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-04-23T14:58:28.376 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-04-23T14:58:28.376 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.376 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.376 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.376 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:28.376 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.376 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.376 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.377 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.377 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.377 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.377 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.377 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:28.377 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:28.377 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.377 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.378 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.378 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.378 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.378 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.378 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.378 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.378 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.379 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.379 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.379 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.379 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.379 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.379 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.379 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.379 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.379 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.380 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:28.380 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:28.380 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.380 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.380 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.380 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.380 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.380 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.381 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.381 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.381 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.381 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.381 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.381 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:28.381 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:28.381 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.382 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.382 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.382 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:28.382 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.382 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.382 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.382 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.382 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.383 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.383 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.383 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.383 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.383 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.383 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.383 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.383 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.384 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.384 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:28.384 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:28.384 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.384 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.384 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.384 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.384 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.385 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.385 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.385 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.385 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.385 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.385 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.385 INFO:teuthology.orchestra.run.smithi028.stdout: "max_completed_events": { 2024-04-23T14:58:28.386 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_completed_events", 2024-04-23T14:58:28.386 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.386 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.386 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.386 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "50", 2024-04-23T14:58:28.386 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.386 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.386 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.386 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "number of past completed events to remember", 2024-04-23T14:58:28.387 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.387 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.387 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.387 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-23T14:58:28.387 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-23T14:58:28.387 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:28.387 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.388 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.388 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-23T14:58:28.388 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.388 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.388 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.388 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long the module is going to sleep", 2024-04-23T14:58:28.388 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.388 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.389 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.389 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.389 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.389 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus", 2024-04-23T14:58:28.389 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:28.389 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:28.389 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:28.390 INFO:teuthology.orchestra.run.smithi028.stdout: "cache": { 2024-04-23T14:58:28.390 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cache", 2024-04-23T14:58:28.390 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.390 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.390 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.390 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:28.390 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.390 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.391 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.391 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.391 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.391 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.391 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.391 INFO:teuthology.orchestra.run.smithi028.stdout: "exclude_perf_counters": { 2024-04-23T14:58:28.391 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "exclude_perf_counters", 2024-04-23T14:58:28.391 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.392 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.392 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.392 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:28.392 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.392 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.392 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.392 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-23T14:58:28.392 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-23T14:58:28.392 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.393 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.393 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:28.393 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:28.393 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.393 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.393 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.393 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.394 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.394 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.394 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.394 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.394 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.394 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.394 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.394 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.395 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.395 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.395 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.395 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.395 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.395 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.395 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:28.396 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:28.396 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.396 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.396 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.396 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.396 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.396 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.396 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.396 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.397 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.397 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.397 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.397 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:28.397 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:28.397 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.397 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.397 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.398 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:28.398 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.398 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.398 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.398 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.398 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.398 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.398 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.399 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.399 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.399 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.399 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.399 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.399 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.399 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.400 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:28.400 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:28.400 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.400 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.400 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.400 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.400 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.400 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.400 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.401 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.401 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.401 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.401 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.401 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools": { 2024-04-23T14:58:28.401 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools", 2024-04-23T14:58:28.401 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.402 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.402 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.402 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.402 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.402 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.402 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.402 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.402 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.402 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.403 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.403 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-23T14:58:28.403 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-23T14:58:28.403 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.403 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.403 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.403 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "300", 2024-04-23T14:58:28.404 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.404 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.404 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.404 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.404 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.404 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.404 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.405 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_interval": { 2024-04-23T14:58:28.405 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_interval", 2024-04-23T14:58:28.405 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-23T14:58:28.405 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.405 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.405 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15.0", 2024-04-23T14:58:28.405 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.405 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.405 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.406 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.406 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.406 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.406 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.406 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-23T14:58:28.406 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-23T14:58:28.406 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.407 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.407 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.407 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-04-23T14:58:28.407 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.407 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.407 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.407 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-23T14:58:28.407 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.407 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.408 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.408 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-23T14:58:28.408 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-23T14:58:28.408 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.408 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.408 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.408 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "9283", 2024-04-23T14:58:28.409 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.409 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.409 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.409 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-23T14:58:28.409 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.409 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.409 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.409 INFO:teuthology.orchestra.run.smithi028.stdout: "stale_cache_strategy": { 2024-04-23T14:58:28.410 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stale_cache_strategy", 2024-04-23T14:58:28.410 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.410 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.410 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.410 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "log", 2024-04-23T14:58:28.410 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.410 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.410 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.411 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.411 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.411 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.411 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.411 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-04-23T14:58:28.411 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-04-23T14:58:28.411 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.412 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.412 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.412 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "default", 2024-04-23T14:58:28.412 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.412 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.412 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.412 INFO:teuthology.orchestra.run.smithi028.stdout: "default", 2024-04-23T14:58:28.412 INFO:teuthology.orchestra.run.smithi028.stdout: "error" 2024-04-23T14:58:28.412 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.413 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.413 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.413 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.413 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.413 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-04-23T14:58:28.413 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-04-23T14:58:28.413 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.413 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.414 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.414 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-04-23T14:58:28.414 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-04-23T14:58:28.414 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-04-23T14:58:28.414 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.414 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.414 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.414 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.415 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.415 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.415 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.415 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-04-23T14:58:28.415 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:28.415 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:28.416 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:28.416 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:28.416 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:28.416 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.416 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.416 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.416 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.416 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.416 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.417 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.417 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.417 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.417 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.417 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.417 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.417 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.417 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.418 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.418 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.418 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.418 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.418 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:28.418 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:28.418 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.419 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.419 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.419 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.419 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.419 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.419 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.419 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.419 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.419 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.420 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.420 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:28.420 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:28.420 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.420 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.420 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.420 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:28.421 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.421 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.421 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.421 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.421 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.421 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.421 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.421 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.421 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.422 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.422 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.422 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.422 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.422 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.422 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:28.422 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:28.423 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.423 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.423 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.423 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.423 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.423 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.423 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.423 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.424 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.424 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.424 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.424 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_snap_create": { 2024-04-23T14:58:28.424 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_snap_create", 2024-04-23T14:58:28.424 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.424 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.425 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.425 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-23T14:58:28.425 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.425 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.425 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.425 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.425 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.425 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.425 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.426 INFO:teuthology.orchestra.run.smithi028.stdout: "mirror_snapshot_schedule": { 2024-04-23T14:58:28.426 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirror_snapshot_schedule", 2024-04-23T14:58:28.426 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.426 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.426 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.426 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.426 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.427 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.427 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.427 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.427 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.427 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.427 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.427 INFO:teuthology.orchestra.run.smithi028.stdout: "trash_purge_schedule": { 2024-04-23T14:58:28.428 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "trash_purge_schedule", 2024-04-23T14:58:28.428 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.428 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.428 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.428 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.428 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.428 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.428 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.428 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.429 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.429 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.429 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.429 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.429 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.429 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "restful", 2024-04-23T14:58:28.430 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:28.430 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:28.430 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:28.430 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_auth": { 2024-04-23T14:58:28.430 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_auth", 2024-04-23T14:58:28.430 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.430 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.430 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.431 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:28.431 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.431 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.431 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.431 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.431 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.431 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.431 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.432 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-04-23T14:58:28.432 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-04-23T14:58:28.432 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.432 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.432 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.432 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.432 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.432 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.433 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.433 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.433 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.433 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.433 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.433 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:28.433 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:28.433 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.434 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.434 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.434 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.434 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.434 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.434 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.434 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.434 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.435 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.435 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.435 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.435 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.435 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.435 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.435 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.435 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.435 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.436 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:28.436 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:28.436 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.436 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.436 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.436 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.436 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.437 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.437 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.437 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.437 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.437 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.437 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.437 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:28.437 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:28.438 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.438 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.438 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.438 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:28.438 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.438 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.438 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.438 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.439 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.439 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.439 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.439 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.439 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.439 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.439 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.439 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.439 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.440 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.440 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:28.440 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:28.440 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.440 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.440 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.440 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.441 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.441 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.441 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.441 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.441 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.441 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.441 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.441 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-23T14:58:28.442 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-23T14:58:28.442 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.442 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.442 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.442 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.442 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.442 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.442 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.443 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.443 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.443 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.443 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.443 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-23T14:58:28.443 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-23T14:58:28.443 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.443 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.444 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.444 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.444 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.444 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.444 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.444 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.444 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.444 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.444 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.445 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.445 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.445 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.445 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rgw", 2024-04-23T14:58:28.445 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:28.445 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:28.445 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:28.446 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:28.446 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:28.446 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.446 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.446 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.446 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.446 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.446 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.446 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.447 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.447 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.447 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.447 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.447 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.447 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.447 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.447 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.448 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.448 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.448 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.448 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:28.448 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:28.448 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.448 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.448 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.449 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.449 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.449 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.449 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.449 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.449 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.449 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.449 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.450 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:28.450 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:28.450 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.450 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.450 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.450 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:28.450 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.450 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.451 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.451 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.451 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.451 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.451 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.451 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.451 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.451 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.452 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.452 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.452 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.452 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.452 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:28.452 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:28.452 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.452 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.453 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.453 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.453 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.453 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.453 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.453 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.453 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.453 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.453 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.454 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.454 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.454 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rook", 2024-04-23T14:58:28.454 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:28.454 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:28.454 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:28.455 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:28.455 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:28.455 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.455 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.455 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.455 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.455 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.455 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.455 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.456 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.456 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.456 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.456 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.456 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.456 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.456 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.456 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.457 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.457 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.457 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.457 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:28.457 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:28.457 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.457 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.458 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.458 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.458 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.458 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.458 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.458 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.458 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.458 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.459 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.459 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:28.459 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:28.459 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.459 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.459 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.459 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:28.459 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.460 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.460 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.460 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.460 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.460 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.460 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.460 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.460 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.461 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.461 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.461 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.461 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.461 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.461 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:28.461 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:28.461 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.462 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.462 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.462 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.462 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.462 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.462 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.462 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.462 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.463 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.463 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.463 INFO:teuthology.orchestra.run.smithi028.stdout: "storage_class": { 2024-04-23T14:58:28.463 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "storage_class", 2024-04-23T14:58:28.463 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.463 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.463 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.463 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "local", 2024-04-23T14:58:28.464 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.464 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.464 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.464 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-23T14:58:28.464 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.464 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.464 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.464 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.465 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.465 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.465 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "selftest", 2024-04-23T14:58:28.465 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:28.465 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:28.465 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:28.465 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:28.465 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:28.466 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.466 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.466 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.466 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.466 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.466 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.466 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.466 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.467 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.467 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.467 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.467 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.467 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.467 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.467 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.467 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.467 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.468 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.468 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:28.468 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:28.468 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.468 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.468 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.468 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.469 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.469 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.469 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.469 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.469 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.469 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.469 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.469 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:28.470 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:28.470 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.470 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.470 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.470 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:28.470 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.470 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.470 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.471 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.471 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.471 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.471 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.471 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.471 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.471 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.471 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.472 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.472 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.472 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.472 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:28.472 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:28.472 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.472 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.472 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.473 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.473 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.473 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.473 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.473 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.473 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.473 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.473 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.474 INFO:teuthology.orchestra.run.smithi028.stdout: "roption1": { 2024-04-23T14:58:28.474 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption1", 2024-04-23T14:58:28.474 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.474 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.474 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.474 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.474 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.474 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.475 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.475 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.475 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.475 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.475 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.475 INFO:teuthology.orchestra.run.smithi028.stdout: "roption2": { 2024-04-23T14:58:28.475 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption2", 2024-04-23T14:58:28.476 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.476 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.476 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.476 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "xyz", 2024-04-23T14:58:28.476 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.476 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.476 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.476 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.476 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.477 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.477 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.477 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption1": { 2024-04-23T14:58:28.477 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption1", 2024-04-23T14:58:28.477 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.477 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.477 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.477 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.478 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.478 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.478 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.478 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.478 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.478 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.478 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.479 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption2": { 2024-04-23T14:58:28.479 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption2", 2024-04-23T14:58:28.479 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.479 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.479 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.479 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.479 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.479 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.479 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.480 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.480 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.480 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.480 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.480 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption3": { 2024-04-23T14:58:28.480 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption3", 2024-04-23T14:58:28.480 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-23T14:58:28.480 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.481 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.481 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.481 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.481 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.481 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.481 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.481 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.481 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.482 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.482 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption4": { 2024-04-23T14:58:28.482 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption4", 2024-04-23T14:58:28.482 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.482 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.482 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.482 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.482 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.483 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.483 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.483 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.483 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.483 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.483 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.483 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption5": { 2024-04-23T14:58:28.484 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption5", 2024-04-23T14:58:28.484 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.484 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.484 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.484 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.484 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.484 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.484 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.485 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.485 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.485 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.485 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.485 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption6": { 2024-04-23T14:58:28.485 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption6", 2024-04-23T14:58:28.485 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.486 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.486 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.486 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:28.486 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.486 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.486 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.486 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.486 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.486 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.487 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.487 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption7": { 2024-04-23T14:58:28.487 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption7", 2024-04-23T14:58:28.487 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.487 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.487 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.487 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.488 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-23T14:58:28.488 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "42", 2024-04-23T14:58:28.488 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.488 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.488 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.488 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.488 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.488 INFO:teuthology.orchestra.run.smithi028.stdout: "testkey": { 2024-04-23T14:58:28.489 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testkey", 2024-04-23T14:58:28.489 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.489 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.489 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.489 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.489 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.489 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.489 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.490 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.490 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.490 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.490 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.490 INFO:teuthology.orchestra.run.smithi028.stdout: "testlkey": { 2024-04-23T14:58:28.490 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testlkey", 2024-04-23T14:58:28.490 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.491 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.491 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.491 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.491 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.491 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.491 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.491 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.491 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: "testnewline": { 2024-04-23T14:58:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testnewline", 2024-04-23T14:58:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.493 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.493 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.493 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.493 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.493 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.493 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.493 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.493 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.493 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.494 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.494 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-04-23T14:58:28.494 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:28.494 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:28.494 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:28.494 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_m_granularity": { 2024-04-23T14:58:28.494 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_m_granularity", 2024-04-23T14:58:28.495 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.495 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.495 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.495 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.495 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.495 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.495 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.495 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow minute scheduled snapshots", 2024-04-23T14:58:28.495 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.495 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: "dump_on_update": { 2024-04-23T14:58:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dump_on_update", 2024-04-23T14:58:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.497 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.497 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.497 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "dump database to debug log on update", 2024-04-23T14:58:28.497 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.497 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.497 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.497 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.497 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:28.497 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:28.498 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.498 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.498 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.498 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.498 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.498 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.498 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.498 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.498 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.499 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.499 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.499 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.499 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.499 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.499 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.499 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.499 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.499 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.500 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:28.500 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:28.500 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.500 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.500 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.500 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.500 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.500 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.501 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.501 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.501 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.501 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.501 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.501 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:28.501 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:28.502 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.502 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.502 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.502 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:28.502 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.502 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.502 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.502 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.502 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.503 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.503 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.503 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.503 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.503 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.503 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.503 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.503 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.504 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.504 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.504 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:28.504 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:28.504 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.504 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.504 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.504 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.504 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.505 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.505 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.505 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.505 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.505 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.505 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.505 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.505 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.505 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.506 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.506 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stats", 2024-04-23T14:58:28.506 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:28.506 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:28.506 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:28.506 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:28.506 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:28.506 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.506 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.507 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.507 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.507 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.507 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.507 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.507 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.507 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.507 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.507 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.507 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.508 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.508 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.508 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.508 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.508 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.508 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.508 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:28.508 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:28.509 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.509 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.509 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.509 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.509 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.509 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.509 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.509 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.509 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.510 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.510 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.510 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.510 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:28.510 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:28.510 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.510 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.510 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.510 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:28.511 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.511 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.511 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.511 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.511 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.511 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.511 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.511 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.512 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.512 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.513 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.513 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.513 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.513 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.513 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:28.514 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:28.514 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.514 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.514 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.514 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.514 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.514 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.514 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.515 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.515 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.515 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.515 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.515 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.515 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.515 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.515 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "status", 2024-04-23T14:58:28.516 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:28.516 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:28.516 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:28.516 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:28.516 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:28.516 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.516 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.516 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.516 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.517 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.517 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.517 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.517 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.517 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.517 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.517 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.517 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.518 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.518 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.518 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.518 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.518 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.518 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.518 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:28.519 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:28.519 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.519 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.519 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.519 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.519 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.519 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.519 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.520 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.520 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.520 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.520 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.520 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:28.520 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:28.520 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.520 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.521 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.521 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:28.521 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.521 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.521 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.521 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.521 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.521 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.521 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.522 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.522 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.522 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.522 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.522 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.522 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.522 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.523 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:28.523 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:28.523 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.523 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.523 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.523 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.523 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.523 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.524 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.524 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.524 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.524 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.524 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.524 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.524 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.524 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.524 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.525 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telegraf", 2024-04-23T14:58:28.525 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:28.525 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:28.525 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:28.525 INFO:teuthology.orchestra.run.smithi028.stdout: "address": { 2024-04-23T14:58:28.525 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "address", 2024-04-23T14:58:28.525 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.525 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.525 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.526 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-23T14:58:28.526 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.526 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.526 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.526 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.526 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.526 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.526 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.527 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-23T14:58:28.527 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-23T14:58:28.527 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:28.527 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.527 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.527 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15", 2024-04-23T14:58:28.527 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.527 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.527 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.528 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.528 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.528 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.528 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.528 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:28.528 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:28.528 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.529 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.529 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.529 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.529 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.529 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.529 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.529 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.529 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.529 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.529 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.530 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.530 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.530 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.530 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.530 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.530 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.530 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.530 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.530 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:28.531 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:28.531 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.531 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.531 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.531 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.531 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.531 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.531 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.531 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.532 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.532 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.532 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.532 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.532 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:28.532 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:28.532 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.532 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.532 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.533 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:28.533 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.533 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.533 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.533 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.533 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.533 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.533 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.533 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.534 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.534 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.534 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.534 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.534 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.534 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.534 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:28.535 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:28.535 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.535 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.535 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.535 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.535 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.535 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.535 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.535 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.536 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.536 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.536 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.536 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.536 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.536 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.536 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telemetry", 2024-04-23T14:58:28.536 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:28.537 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:28.537 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:28.537 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_basic": { 2024-04-23T14:58:28.537 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_basic", 2024-04-23T14:58:28.537 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.537 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.537 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.537 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:28.537 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.538 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.538 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.538 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-23T14:58:28.538 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.538 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.538 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.538 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_crash": { 2024-04-23T14:58:28.539 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_crash", 2024-04-23T14:58:28.539 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.539 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.539 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.539 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:28.539 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.539 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.539 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.539 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-23T14:58:28.540 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.540 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.540 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.540 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_device": { 2024-04-23T14:58:28.540 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_device", 2024-04-23T14:58:28.540 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.540 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.541 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.541 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:28.541 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.541 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.541 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.541 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-23T14:58:28.541 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.541 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.541 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.542 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_ident": { 2024-04-23T14:58:28.542 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_ident", 2024-04-23T14:58:28.542 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.542 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.542 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.542 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.542 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.543 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.543 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.543 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-23T14:58:28.543 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.543 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.543 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.543 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.543 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_perf": { 2024-04-23T14:58:28.544 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_perf", 2024-04-23T14:58:28.544 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.544 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.544 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.544 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.544 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.544 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.544 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.544 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-23T14:58:28.545 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.545 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.545 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.545 INFO:teuthology.orchestra.run.smithi028.stdout: "contact": { 2024-04-23T14:58:28.545 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "contact", 2024-04-23T14:58:28.545 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.545 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.545 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.546 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.546 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.546 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.546 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.546 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.546 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.546 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.546 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.547 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.547 INFO:teuthology.orchestra.run.smithi028.stdout: "description": { 2024-04-23T14:58:28.547 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "description", 2024-04-23T14:58:28.547 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.547 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.547 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.547 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.547 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.547 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.548 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.548 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.548 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.548 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.548 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.548 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.548 INFO:teuthology.orchestra.run.smithi028.stdout: "device_url": { 2024-04-23T14:58:28.548 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_url", 2024-04-23T14:58:28.548 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.548 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.549 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.549 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-23T14:58:28.549 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.549 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.549 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.549 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.549 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.549 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.549 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.550 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-04-23T14:58:28.550 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-04-23T14:58:28.550 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.550 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.550 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.550 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.550 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.550 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.551 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.551 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.551 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.551 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.551 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.551 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-23T14:58:28.551 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-23T14:58:28.551 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.552 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.552 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.552 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "24", 2024-04-23T14:58:28.552 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "8", 2024-04-23T14:58:28.552 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.552 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.552 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.552 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.552 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.553 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.553 INFO:teuthology.orchestra.run.smithi028.stdout: "last_opt_revision": { 2024-04-23T14:58:28.553 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "last_opt_revision", 2024-04-23T14:58:28.553 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.553 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.553 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.553 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1", 2024-04-23T14:58:28.553 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.554 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.554 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.554 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.554 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.554 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.554 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.554 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.554 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard": { 2024-04-23T14:58:28.555 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard", 2024-04-23T14:58:28.555 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.555 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.555 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.555 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.555 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.555 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.555 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.555 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.556 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.556 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.556 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.556 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.556 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard_description": { 2024-04-23T14:58:28.556 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard_description", 2024-04-23T14:58:28.556 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.556 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.556 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.556 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.557 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.557 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.557 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.557 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.557 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.557 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.557 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.557 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.558 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:28.558 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:28.558 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.558 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.558 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.558 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.558 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.558 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.558 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.559 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.559 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.559 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.559 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.559 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.559 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.559 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.559 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.559 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.560 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.560 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.560 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:28.560 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:28.560 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.560 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.560 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.560 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.561 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.561 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.561 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.561 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.561 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.561 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.561 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.561 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.561 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:28.561 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:28.562 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.562 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.562 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.562 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:28.562 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.562 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.562 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.562 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.563 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.563 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.563 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.563 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.563 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.563 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.563 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.563 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.563 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.564 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.564 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.564 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:28.564 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:28.564 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.564 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.564 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.564 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.564 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.565 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.565 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.565 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.565 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.565 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.565 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.565 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.565 INFO:teuthology.orchestra.run.smithi028.stdout: "organization": { 2024-04-23T14:58:28.566 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "organization", 2024-04-23T14:58:28.566 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.566 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.566 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.566 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.566 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.566 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.566 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.567 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.567 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.567 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.567 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.567 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.567 INFO:teuthology.orchestra.run.smithi028.stdout: "proxy": { 2024-04-23T14:58:28.567 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "proxy", 2024-04-23T14:58:28.567 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.568 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.568 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.568 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.568 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.568 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.568 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.568 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.568 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.568 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.569 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.569 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.569 INFO:teuthology.orchestra.run.smithi028.stdout: "url": { 2024-04-23T14:58:28.569 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url", 2024-04-23T14:58:28.569 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.569 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.569 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.569 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-23T14:58:28.569 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.570 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.570 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.570 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.570 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.570 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.570 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.570 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.570 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.570 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.571 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.571 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "test_orchestrator", 2024-04-23T14:58:28.571 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:28.571 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:28.571 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:28.571 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:28.571 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:28.571 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.572 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.572 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.572 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.572 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.572 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.572 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.572 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.572 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.572 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.573 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.573 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.573 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.573 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.573 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.573 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.573 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.573 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.574 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:28.574 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:28.574 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.574 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.574 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.574 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.574 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.574 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.574 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.575 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.575 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.575 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.575 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.575 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.575 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:28.575 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:28.575 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.576 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.576 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.576 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:28.576 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.576 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.576 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.576 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.576 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.576 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.577 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.577 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.577 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.577 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.577 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.577 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.577 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.577 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.577 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.577 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:28.578 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:28.578 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.578 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.578 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.578 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.578 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.578 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.578 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.578 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.579 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.579 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.579 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.579 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.579 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.579 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.579 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.579 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-04-23T14:58:28.579 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:28.580 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:28.580 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:28.580 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:28.580 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:28.580 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.580 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.580 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.580 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.580 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.581 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.581 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.581 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.581 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.581 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.581 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.581 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.581 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.581 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.582 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.582 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.582 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.582 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.582 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.582 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:28.582 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:28.582 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.583 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.583 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.583 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.583 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.583 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.583 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.583 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.583 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.583 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.584 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.584 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:28.584 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:28.584 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.584 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.584 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.584 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:28.584 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.585 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.585 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.585 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.585 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.585 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.585 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.585 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.585 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.586 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.586 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.586 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.586 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.586 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.586 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.586 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:28.586 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:28.587 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.587 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.587 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.587 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.587 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.587 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.587 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.587 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.587 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.588 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.588 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.588 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_clones": { 2024-04-23T14:58:28.588 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_clones", 2024-04-23T14:58:28.588 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.588 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.588 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.589 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4", 2024-04-23T14:58:28.589 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.589 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.589 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.589 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-23T14:58:28.589 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.589 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.589 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.590 INFO:teuthology.orchestra.run.smithi028.stdout: "periodic_async_work": { 2024-04-23T14:58:28.590 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "periodic_async_work", 2024-04-23T14:58:28.590 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.590 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.590 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.590 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.590 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.591 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.591 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.591 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Periodically check for async work", 2024-04-23T14:58:28.591 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.591 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.591 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.591 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_delay": { 2024-04-23T14:58:28.591 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_delay", 2024-04-23T14:58:28.592 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.592 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.592 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.592 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-23T14:58:28.592 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.592 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.592 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.592 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-23T14:58:28.592 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.593 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.593 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.593 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_no_wait": { 2024-04-23T14:58:28.593 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_no_wait", 2024-04-23T14:58:28.593 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.593 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.593 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.593 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-23T14:58:28.594 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.594 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.594 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.594 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-23T14:58:28.594 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.594 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.594 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.594 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.595 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.595 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.595 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix", 2024-04-23T14:58:28.595 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-23T14:58:28.595 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-23T14:58:28.595 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-23T14:58:28.595 INFO:teuthology.orchestra.run.smithi028.stdout: "discovery_interval": { 2024-04-23T14:58:28.596 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "discovery_interval", 2024-04-23T14:58:28.596 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-23T14:58:28.596 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.596 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.596 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "100", 2024-04-23T14:58:28.596 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.596 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.596 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.596 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.597 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.597 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.597 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.597 INFO:teuthology.orchestra.run.smithi028.stdout: "identifier": { 2024-04-23T14:58:28.597 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "identifier", 2024-04-23T14:58:28.597 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.597 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.597 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.598 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.598 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.598 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.598 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.598 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.598 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.598 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.598 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.599 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-23T14:58:28.599 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-23T14:58:28.599 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-23T14:58:28.599 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.599 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.599 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-23T14:58:28.599 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.600 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.600 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.600 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.600 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.600 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.600 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.600 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-23T14:58:28.600 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-23T14:58:28.601 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.601 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.601 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.601 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.601 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.601 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.601 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.601 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.602 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.602 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.602 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.602 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.602 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.602 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.602 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.602 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.603 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.603 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.603 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-23T14:58:28.603 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-23T14:58:28.603 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.603 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.603 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.603 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.604 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.604 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.604 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.604 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.604 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.604 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.604 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-23T14:58:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-23T14:58:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-23T14:58:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-23T14:58:28.606 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-23T14:58:28.606 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-23T14:58:28.606 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-23T14:58:28.606 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-23T14:58:28.606 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-23T14:58:28.606 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-23T14:58:28.606 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.606 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-23T14:58:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-23T14:58:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-23T14:58:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-23T14:58:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-23T14:58:28.608 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.608 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.608 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.608 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.608 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.608 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.608 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.608 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_host": { 2024-04-23T14:58:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_host", 2024-04-23T14:58:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-23T14:58:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.610 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.610 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.610 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_port": { 2024-04-23T14:58:28.610 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_port", 2024-04-23T14:58:28.610 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-23T14:58:28.610 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.610 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.611 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10051", 2024-04-23T14:58:28.611 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.611 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.611 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.611 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.611 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.611 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.611 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.611 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_sender": { 2024-04-23T14:58:28.612 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_sender", 2024-04-23T14:58:28.612 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-23T14:58:28.612 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-23T14:58:28.612 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-23T14:58:28.612 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-23T14:58:28.612 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-23T14:58:28.612 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-23T14:58:28.612 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-23T14:58:28.612 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-23T14:58:28.613 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-23T14:58:28.613 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-23T14:58:28.613 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-23T14:58:28.613 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.613 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.613 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.613 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.613 INFO:teuthology.orchestra.run.smithi028.stdout: "services": { 2024-04-23T14:58:28.613 INFO:teuthology.orchestra.run.smithi028.stdout: "dashboard": "https://172.21.15.28:8443/" 2024-04-23T14:58:28.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.614 INFO:teuthology.orchestra.run.smithi028.stdout: "always_on_modules": { 2024-04-23T14:58:28.614 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus": [ 2024-04-23T14:58:28.614 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-04-23T14:58:28.614 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-04-23T14:58:28.614 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-04-23T14:58:28.614 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-04-23T14:58:28.614 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-04-23T14:58:28.615 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-04-23T14:58:28.615 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-04-23T14:58:28.615 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-04-23T14:58:28.615 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-04-23T14:58:28.615 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-04-23T14:58:28.615 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.615 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific": [ 2024-04-23T14:58:28.615 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-04-23T14:58:28.615 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-04-23T14:58:28.615 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-04-23T14:58:28.616 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-04-23T14:58:28.616 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-04-23T14:58:28.616 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-04-23T14:58:28.616 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-04-23T14:58:28.616 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-04-23T14:58:28.616 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-04-23T14:58:28.616 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-04-23T14:58:28.616 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.616 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy": [ 2024-04-23T14:58:28.617 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-04-23T14:58:28.617 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-04-23T14:58:28.617 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-04-23T14:58:28.617 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-04-23T14:58:28.617 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-04-23T14:58:28.617 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-04-23T14:58:28.617 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-04-23T14:58:28.617 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-04-23T14:58:28.617 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-04-23T14:58:28.618 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-04-23T14:58:28.618 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.618 INFO:teuthology.orchestra.run.smithi028.stdout: "reef": [ 2024-04-23T14:58:28.618 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-04-23T14:58:28.618 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-04-23T14:58:28.618 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-04-23T14:58:28.618 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-04-23T14:58:28.618 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-04-23T14:58:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-04-23T14:58:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-04-23T14:58:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-04-23T14:58:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-04-23T14:58:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-04-23T14:58:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-23T14:58:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: "squid": [ 2024-04-23T14:58:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-04-23T14:58:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-04-23T14:58:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-04-23T14:58:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-04-23T14:58:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-04-23T14:58:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-04-23T14:58:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-04-23T14:58:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-04-23T14:58:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-04-23T14:58:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-04-23T14:58:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-23T14:58:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.621 INFO:teuthology.orchestra.run.smithi028.stdout: "last_failure_osd_epoch": 88, 2024-04-23T14:58:28.621 INFO:teuthology.orchestra.run.smithi028.stdout: "active_clients": [ 2024-04-23T14:58:28.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.621 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-04-23T14:58:28.621 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-04-23T14:58:28.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.621 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-04-23T14:58:28.621 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-04-23T14:58:28.621 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3924253050 2024-04-23T14:58:28.622 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.622 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-23T14:58:28.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.622 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "libcephsqlite", 2024-04-23T14:58:28.622 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-04-23T14:58:28.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.622 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-04-23T14:58:28.622 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-04-23T14:58:28.623 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2375371582 2024-04-23T14:58:28.623 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.623 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-23T14:58:28.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.623 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-04-23T14:58:28.623 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-04-23T14:58:28.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.623 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-04-23T14:58:28.624 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-04-23T14:58:28.624 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 353714578 2024-04-23T14:58:28.624 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.624 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-23T14:58:28.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.624 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-04-23T14:58:28.624 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-04-23T14:58:28.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.625 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-04-23T14:58:28.625 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-04-23T14:58:28.625 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4031919296 2024-04-23T14:58:28.625 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.625 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-23T14:58:28.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.625 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.625 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-04-23T14:58:28.625 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-04-23T14:58:28.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.626 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-04-23T14:58:28.626 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-04-23T14:58:28.626 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1941510443 2024-04-23T14:58:28.626 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.626 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-23T14:58:28.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:28.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.627 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-04-23T14:58:28.627 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-04-23T14:58:28.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-23T14:58:28.627 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-04-23T14:58:28.627 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-04-23T14:58:28.627 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3082498491 2024-04-23T14:58:28.627 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.627 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-23T14:58:28.627 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:28.628 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-23T14:58:28.628 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:58:28.634 DEBUG:tasks.check_counter:Getting stats from a 2024-04-23T14:58:28.634 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-23T14:58:28.635 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:26 smithi028 ceph-mon[27810]: pgmap v451: 97 pgs: 97 active+clean; 835 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 19 MiB/s wr, 159 op/s 2024-04-23T14:58:28.635 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:26 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3549654427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T14:58:28.635 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:26 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3211144062' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-23T14:58:29.110 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:58:29.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:28 smithi038 ceph-mon[32348]: pgmap v452: 97 pgs: 97 active+clean; 838 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 21 MiB/s wr, 209 op/s 2024-04-23T14:58:29.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:28 smithi028 ceph-mon[27810]: pgmap v452: 97 pgs: 97 active+clean; 838 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 21 MiB/s wr, 209 op/s 2024-04-23T14:58:29.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:28 smithi105 ceph-mon[32555]: pgmap v452: 97 pgs: 97 active+clean; 838 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 21 MiB/s wr, 209 op/s 2024-04-23T14:58:31.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:30 smithi038 ceph-mon[32348]: pgmap v453: 97 pgs: 97 active+clean; 838 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 19 MiB/s wr, 162 op/s 2024-04-23T14:58:31.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:30 smithi028 ceph-mon[27810]: pgmap v453: 97 pgs: 97 active+clean; 838 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 19 MiB/s wr, 162 op/s 2024-04-23T14:58:31.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:30 smithi105 ceph-mon[32555]: pgmap v453: 97 pgs: 97 active+clean; 838 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 19 MiB/s wr, 162 op/s 2024-04-23T14:58:31.353 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:58:31.353 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T14:58:31.354 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 0, 2024-04-23T14:58:31.354 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 2, 2024-04-23T14:58:31.354 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 0, 2024-04-23T14:58:31.354 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 0, 2024-04-23T14:58:31.354 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 4, 2024-04-23T14:58:31.354 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 0, 2024-04-23T14:58:31.354 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.004457477, 2024-04-23T14:58:31.354 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.000000000, 2024-04-23T14:58:31.355 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.000867573, 2024-04-23T14:58:31.355 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T14:58:31.355 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:58:31.355 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:31.355 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:31.355 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:31.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:31.355 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:58:31.355 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:31.355 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:31.356 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:31.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:31.356 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-23T14:58:31.356 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 0 2024-04-23T14:58:31.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:31.356 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T14:58:31.356 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 2, 2024-04-23T14:58:31.356 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 207, 2024-04-23T14:58:31.356 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 320, 2024-04-23T14:58:31.357 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 192608, 2024-04-23T14:58:31.357 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 5, 2024-04-23T14:58:31.357 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 3, 2024-04-23T14:58:31.357 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.043523889, 2024-04-23T14:58:31.357 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.032497482, 2024-04-23T14:58:31.357 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.004381538, 2024-04-23T14:58:31.357 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T14:58:31.357 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:58:31.357 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 204, 2024-04-23T14:58:31.358 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.034392738, 2024-04-23T14:58:31.358 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000168591 2024-04-23T14:58:31.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:31.358 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:58:31.358 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:31.358 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:31.358 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:31.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:31.358 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-23T14:58:31.359 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 192608 2024-04-23T14:58:31.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:31.359 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T14:58:31.359 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 288, 2024-04-23T14:58:31.359 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 274, 2024-04-23T14:58:31.359 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 1748864, 2024-04-23T14:58:31.359 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 168192, 2024-04-23T14:58:31.359 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 2, 2024-04-23T14:58:31.359 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 1, 2024-04-23T14:58:31.360 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.119086448, 2024-04-23T14:58:31.360 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.030040499, 2024-04-23T14:58:31.360 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.099534587, 2024-04-23T14:58:31.360 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.015702543, 2024-04-23T14:58:31.360 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:58:31.360 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 274, 2024-04-23T14:58:31.360 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.030640827, 2024-04-23T14:58:31.360 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000111827 2024-04-23T14:58:31.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:31.361 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:58:31.361 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:31.361 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:31.361 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:31.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:31.361 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1748864, 2024-04-23T14:58:31.361 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 168192 2024-04-23T14:58:31.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:31.361 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-04-23T14:58:31.361 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 0, 2024-04-23T14:58:31.362 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-04-23T14:58:31.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:31.362 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-04-23T14:58:31.362 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-04-23T14:58:31.362 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-04-23T14:58:31.362 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T14:58:31.362 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-04-23T14:58:31.362 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T14:58:31.362 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-04-23T14:58:31.363 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T14:58:31.363 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T14:58:31.363 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T14:58:31.363 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T14:58:31.363 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T14:58:31.363 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-04-23T14:58:31.363 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T14:58:31.363 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T14:58:31.363 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-04-23T14:58:31.363 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-04-23T14:58:31.364 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-04-23T14:58:31.364 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-04-23T14:58:31.364 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T14:58:31.364 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T14:58:31.364 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T14:58:31.364 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T14:58:31.364 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T14:58:31.364 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-04-23T14:58:31.364 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-04-23T14:58:31.365 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-04-23T14:58:31.365 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T14:58:31.365 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T14:58:31.365 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-04-23T14:58:31.365 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-04-23T14:58:31.365 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-04-23T14:58:31.365 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-04-23T14:58:31.365 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T14:58:31.366 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-04-23T14:58:31.366 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T14:58:31.366 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-04-23T14:58:31.366 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 150025, 2024-04-23T14:58:31.366 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 32, 2024-04-23T14:58:31.366 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-04-23T14:58:31.366 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-04-23T14:58:31.366 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-04-23T14:58:31.367 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-04-23T14:58:31.367 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-04-23T14:58:31.367 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-04-23T14:58:31.367 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-04-23T14:58:31.367 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-04-23T14:58:31.367 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 0, 2024-04-23T14:58:31.367 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 0, 2024-04-23T14:58:31.367 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T14:58:31.367 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-04-23T14:58:31.368 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-04-23T14:58:31.368 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-04-23T14:58:31.368 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 928, 2024-04-23T14:58:31.368 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 2, 2024-04-23T14:58:31.368 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-04-23T14:58:31.368 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-04-23T14:58:31.368 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-04-23T14:58:31.368 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-04-23T14:58:31.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:31.369 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T14:58:31.369 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-23T14:58:31.369 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-04-23T14:58:31.369 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-23T14:58:31.369 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 286, 2024-04-23T14:58:31.369 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 1705796, 2024-04-23T14:58:31.369 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-23T14:58:31.369 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 286, 2024-04-23T14:58:31.369 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-04-23T14:58:31.370 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-04-23T14:58:31.370 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 286, 2024-04-23T14:58:31.370 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 1705796, 2024-04-23T14:58:31.370 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-23T14:58:31.370 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:31.370 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:31.370 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:31.370 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:31.370 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:31.371 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:58:32.165 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-04-23T14:58:32.165 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-04-23T14:58:32.165 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-04-23T14:58:32.165 DEBUG:tasks.check_counter:Getting stats from d 2024-04-23T14:58:32.165 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-23T14:58:32.620 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:58:32.985 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:32 smithi028 ceph-mon[27810]: pgmap v454: 97 pgs: 97 active+clean; 838 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 19 MiB/s wr, 162 op/s 2024-04-23T14:58:33.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:32 smithi038 ceph-mon[32348]: pgmap v454: 97 pgs: 97 active+clean; 838 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 19 MiB/s wr, 162 op/s 2024-04-23T14:58:33.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:32 smithi105 ceph-mon[32555]: pgmap v454: 97 pgs: 97 active+clean; 838 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 19 MiB/s wr, 162 op/s 2024-04-23T14:58:34.961 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:58:34.962 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T14:58:34.962 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 4, 2024-04-23T14:58:34.962 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 4, 2024-04-23T14:58:34.962 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 2992, 2024-04-23T14:58:34.962 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 448, 2024-04-23T14:58:34.963 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 4, 2024-04-23T14:58:34.963 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 0, 2024-04-23T14:58:34.963 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.004376315, 2024-04-23T14:58:34.963 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.000053619, 2024-04-23T14:58:34.963 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.001149968, 2024-04-23T14:58:34.963 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T14:58:34.963 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:58:34.963 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 3, 2024-04-23T14:58:34.964 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.001564319, 2024-04-23T14:58:34.964 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000521439 2024-04-23T14:58:34.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:34.964 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:58:34.964 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:34.964 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:34.964 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:34.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:34.964 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-23T14:58:34.965 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 448 2024-04-23T14:58:34.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:34.965 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T14:58:34.965 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 291, 2024-04-23T14:58:34.965 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 279, 2024-04-23T14:58:34.965 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 1747776, 2024-04-23T14:58:34.965 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 172224, 2024-04-23T14:58:34.965 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 2, 2024-04-23T14:58:34.965 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 1, 2024-04-23T14:58:34.966 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.122237419, 2024-04-23T14:58:34.966 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.047052455, 2024-04-23T14:58:34.966 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.079842421, 2024-04-23T14:58:34.966 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.013509010, 2024-04-23T14:58:34.966 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:58:34.966 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 278, 2024-04-23T14:58:34.966 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.029501820, 2024-04-23T14:58:34.966 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000106121 2024-04-23T14:58:34.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:34.967 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:58:34.967 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:34.967 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:34.967 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:34.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:34.967 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1747776, 2024-04-23T14:58:34.967 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 172224 2024-04-23T14:58:34.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:34.968 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T14:58:34.968 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 2, 2024-04-23T14:58:34.968 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 213, 2024-04-23T14:58:34.968 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 320, 2024-04-23T14:58:34.968 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 193664, 2024-04-23T14:58:34.968 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 5, 2024-04-23T14:58:34.968 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 3, 2024-04-23T14:58:34.968 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.045470967, 2024-04-23T14:58:34.968 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.033565468, 2024-04-23T14:58:34.969 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.004681130, 2024-04-23T14:58:34.969 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T14:58:34.969 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:58:34.969 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 210, 2024-04-23T14:58:34.969 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.029353946, 2024-04-23T14:58:34.969 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000139780 2024-04-23T14:58:34.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:34.969 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:58:34.969 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:34.970 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:34.970 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:34.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:34.970 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-23T14:58:34.970 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 193664 2024-04-23T14:58:34.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:34.970 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-04-23T14:58:34.970 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 0, 2024-04-23T14:58:34.970 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-04-23T14:58:34.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:34.971 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-04-23T14:58:34.971 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-04-23T14:58:34.971 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-04-23T14:58:34.971 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T14:58:34.971 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-04-23T14:58:34.971 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T14:58:34.971 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-04-23T14:58:34.972 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T14:58:34.972 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T14:58:34.972 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T14:58:34.972 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T14:58:34.972 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T14:58:34.972 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-04-23T14:58:34.972 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T14:58:34.972 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T14:58:34.972 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-04-23T14:58:34.973 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-04-23T14:58:34.973 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-04-23T14:58:34.973 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-04-23T14:58:34.973 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T14:58:34.973 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T14:58:34.973 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T14:58:34.973 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T14:58:34.973 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T14:58:34.973 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-04-23T14:58:34.974 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-04-23T14:58:34.974 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-04-23T14:58:34.974 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T14:58:34.974 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T14:58:34.974 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-04-23T14:58:34.974 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-04-23T14:58:34.974 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-04-23T14:58:34.974 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-04-23T14:58:34.975 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T14:58:34.975 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-04-23T14:58:34.975 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T14:58:34.975 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-04-23T14:58:34.975 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 151039, 2024-04-23T14:58:34.975 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 32, 2024-04-23T14:58:34.975 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-04-23T14:58:34.975 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-04-23T14:58:34.975 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-04-23T14:58:34.976 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-04-23T14:58:34.976 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-04-23T14:58:34.976 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-04-23T14:58:34.976 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-04-23T14:58:34.976 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-04-23T14:58:34.976 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 0, 2024-04-23T14:58:34.976 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 0, 2024-04-23T14:58:34.976 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T14:58:34.976 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-04-23T14:58:34.977 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-04-23T14:58:34.977 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-04-23T14:58:34.977 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 928, 2024-04-23T14:58:34.977 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 2, 2024-04-23T14:58:34.977 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-04-23T14:58:34.977 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-04-23T14:58:34.977 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-04-23T14:58:34.977 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-04-23T14:58:34.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:34.978 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T14:58:34.978 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-23T14:58:34.978 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-04-23T14:58:34.978 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-23T14:58:34.978 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 293, 2024-04-23T14:58:34.978 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 1706706, 2024-04-23T14:58:34.978 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-23T14:58:34.978 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 293, 2024-04-23T14:58:34.979 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-04-23T14:58:34.979 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-04-23T14:58:34.979 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 293, 2024-04-23T14:58:34.979 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 1706706, 2024-04-23T14:58:34.979 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-23T14:58:34.979 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:34.979 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:34.979 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:34.979 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:34.980 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:34.980 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:58:35.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:34 smithi038 ceph-mon[32348]: pgmap v455: 97 pgs: 97 active+clean; 836 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 19 MiB/s wr, 209 op/s 2024-04-23T14:58:35.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:34 smithi028 ceph-mon[27810]: pgmap v455: 97 pgs: 97 active+clean; 836 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 19 MiB/s wr, 209 op/s 2024-04-23T14:58:35.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:34 smithi105 ceph-mon[32555]: pgmap v455: 97 pgs: 97 active+clean; 836 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 19 MiB/s wr, 209 op/s 2024-04-23T14:58:35.764 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-04-23T14:58:35.764 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-04-23T14:58:35.764 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-04-23T14:58:35.765 DEBUG:tasks.check_counter:Getting stats from g 2024-04-23T14:58:35.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-23T14:58:36.240 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:58:37.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:36 smithi038 ceph-mon[32348]: pgmap v456: 97 pgs: 97 active+clean; 836 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 3.0 MiB/s wr, 97 op/s 2024-04-23T14:58:37.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:36 smithi028 ceph-mon[27810]: pgmap v456: 97 pgs: 97 active+clean; 836 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 3.0 MiB/s wr, 97 op/s 2024-04-23T14:58:37.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:36 smithi105 ceph-mon[32555]: pgmap v456: 97 pgs: 97 active+clean; 836 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 3.0 MiB/s wr, 97 op/s 2024-04-23T14:58:38.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:37 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:58:38.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:37 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:58:38.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:37 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T14:58:38.586 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:58:38.586 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T14:58:38.587 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 2630, 2024-04-23T14:58:38.587 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 2615, 2024-04-23T14:58:38.587 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 346398105, 2024-04-23T14:58:38.587 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 909986, 2024-04-23T14:58:38.587 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 7, 2024-04-23T14:58:38.587 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 3, 2024-04-23T14:58:38.587 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 1.713237490, 2024-04-23T14:58:38.587 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.214568961, 2024-04-23T14:58:38.587 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 1.312820494, 2024-04-23T14:58:38.588 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.147203235, 2024-04-23T14:58:38.588 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:58:38.588 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 2615, 2024-04-23T14:58:38.588 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.235325479, 2024-04-23T14:58:38.588 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000089990 2024-04-23T14:58:38.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.588 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:58:38.588 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.588 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.589 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.589 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1774800, 2024-04-23T14:58:38.589 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 174448 2024-04-23T14:58:38.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.589 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T14:58:38.589 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 1399, 2024-04-23T14:58:38.589 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 1607, 2024-04-23T14:58:38.589 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 401543752, 2024-04-23T14:58:38.589 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 982440, 2024-04-23T14:58:38.590 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 9, 2024-04-23T14:58:38.590 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 7, 2024-04-23T14:58:38.590 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 1.689163520, 2024-04-23T14:58:38.590 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.142869869, 2024-04-23T14:58:38.590 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 1.429115799, 2024-04-23T14:58:38.590 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.080716836, 2024-04-23T14:58:38.590 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:58:38.590 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1604, 2024-04-23T14:58:38.591 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.161190625, 2024-04-23T14:58:38.591 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000100492 2024-04-23T14:58:38.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.591 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:58:38.591 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.591 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.591 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.591 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-23T14:58:38.593 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 538128 2024-04-23T14:58:38.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.593 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T14:58:38.593 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 1333, 2024-04-23T14:58:38.593 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 1335, 2024-04-23T14:58:38.593 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 365244456, 2024-04-23T14:58:38.593 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 423901, 2024-04-23T14:58:38.593 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 6, 2024-04-23T14:58:38.593 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 5, 2024-04-23T14:58:38.594 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 1.440036042, 2024-04-23T14:58:38.594 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.095525980, 2024-04-23T14:58:38.594 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 1.241427271, 2024-04-23T14:58:38.594 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.076120725, 2024-04-23T14:58:38.594 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:58:38.594 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1333, 2024-04-23T14:58:38.594 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.117933482, 2024-04-23T14:58:38.594 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000088472 2024-04-23T14:58:38.594 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.595 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:58:38.595 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.595 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.595 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.595 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-23T14:58:38.595 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 0 2024-04-23T14:58:38.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.595 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-04-23T14:58:38.596 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 1, 2024-04-23T14:58:38.596 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-04-23T14:58:38.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.596 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-MDSRank": { 2024-04-23T14:58:38.596 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-04-23T14:58:38.596 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-04-23T14:58:38.596 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 7637, 2024-04-23T14:58:38.596 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.598170444, 2024-04-23T14:58:38.596 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000078325 2024-04-23T14:58:38.596 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:38.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.597 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-PurgeQueue": { 2024-04-23T14:58:38.597 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-04-23T14:58:38.597 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-04-23T14:58:38.597 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.597 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.597 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.597 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:38.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.597 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-04-23T14:58:38.598 INFO:teuthology.orchestra.run.smithi028.stdout: "request": 0, 2024-04-23T14:58:38.598 INFO:teuthology.orchestra.run.smithi028.stdout: "reply": 0, 2024-04-23T14:58:38.598 INFO:teuthology.orchestra.run.smithi028.stdout: "reply_latency": { 2024-04-23T14:58:38.598 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.598 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.598 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.598 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_reply": 0, 2024-04-23T14:58:38.598 INFO:teuthology.orchestra.run.smithi028.stdout: "forward": 0, 2024-04-23T14:58:38.599 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_complete": 0, 2024-04-23T14:58:38.599 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_keys": 0, 2024-04-23T14:58:38.599 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_commit": 0, 2024-04-23T14:58:38.599 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_split": 0, 2024-04-23T14:58:38.599 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_merge": 0, 2024-04-23T14:58:38.599 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes": 47194, 2024-04-23T14:58:38.599 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_top": 19322, 2024-04-23T14:58:38.599 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_bottom": 27872, 2024-04-23T14:58:38.599 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pin_tail": 0, 2024-04-23T14:58:38.599 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pinned": 19591, 2024-04-23T14:58:38.600 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_expired": 0, 2024-04-23T14:58:38.600 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_with_caps": 0, 2024-04-23T14:58:38.600 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": 0, 2024-04-23T14:58:38.600 INFO:teuthology.orchestra.run.smithi028.stdout: "subtrees": 2, 2024-04-23T14:58:38.600 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse": 0, 2024-04-23T14:58:38.600 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_hit": 0, 2024-04-23T14:58:38.600 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_forward": 0, 2024-04-23T14:58:38.600 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_discover": 0, 2024-04-23T14:58:38.600 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_dir_fetch": 0, 2024-04-23T14:58:38.600 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_remote_ino": 0, 2024-04-23T14:58:38.601 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_lock": 0, 2024-04-23T14:58:38.601 INFO:teuthology.orchestra.run.smithi028.stdout: "load_cent": 0, 2024-04-23T14:58:38.601 INFO:teuthology.orchestra.run.smithi028.stdout: "q": 0, 2024-04-23T14:58:38.601 INFO:teuthology.orchestra.run.smithi028.stdout: "exported": 0, 2024-04-23T14:58:38.601 INFO:teuthology.orchestra.run.smithi028.stdout: "exported_inodes": 0, 2024-04-23T14:58:38.601 INFO:teuthology.orchestra.run.smithi028.stdout: "imported": 0, 2024-04-23T14:58:38.601 INFO:teuthology.orchestra.run.smithi028.stdout: "imported_inodes": 0, 2024-04-23T14:58:38.601 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_dir_fetch": 0, 2024-04-23T14:58:38.601 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_backtrace_fetch": 0, 2024-04-23T14:58:38.601 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_peer_discover": 0, 2024-04-23T14:58:38.602 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rfiles": 615, 2024-04-23T14:58:38.602 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rbytes": 68840109, 2024-04-23T14:58:38.602 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rsnaps": 6, 2024-04-23T14:58:38.602 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_fetch": 0, 2024-04-23T14:58:38.602 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_set_tag": 0, 2024-04-23T14:58:38.602 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_repaired": 0, 2024-04-23T14:58:38.602 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_inotable_repaired": 0, 2024-04-23T14:58:38.602 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_inodes": 0, 2024-04-23T14:58:38.602 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_base_inodes": 0, 2024-04-23T14:58:38.603 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dirfrag_rstats": 0, 2024-04-23T14:58:38.603 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_file_inodes": 0, 2024-04-23T14:58:38.603 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_inode_file_caps": 0, 2024-04-23T14:58:38.603 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_revoke": 0, 2024-04-23T14:58:38.603 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_grant": 0, 2024-04-23T14:58:38.603 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_trunc": 0, 2024-04-23T14:58:38.603 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-23T14:58:38.603 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-23T14:58:38.603 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps": 0, 2024-04-23T14:58:38.604 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps_dirty": 0, 2024-04-23T14:58:38.604 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_cap_release": 0, 2024-04-23T14:58:38.604 INFO:teuthology.orchestra.run.smithi028.stdout: "process_request_cap_release": 0 2024-04-23T14:58:38.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.604 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache": { 2024-04-23T14:58:38.604 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update": 0, 2024-04-23T14:58:38.604 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update_receipt": 0, 2024-04-23T14:58:38.604 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_try_discover": 0, 2024-04-23T14:58:38.604 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_send_discover": 0, 2024-04-23T14:58:38.605 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_handle_discover": 0, 2024-04-23T14:58:38.605 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays": 0, 2024-04-23T14:58:38.605 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_delayed": 0, 2024-04-23T14:58:38.605 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_enqueuing": 0, 2024-04-23T14:58:38.605 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_created": 0, 2024-04-23T14:58:38.605 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_enqueued": 0, 2024-04-23T14:58:38.605 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_reintegrated": 0, 2024-04-23T14:58:38.605 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_migrated": 0, 2024-04-23T14:58:38.605 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_processing": 0, 2024-04-23T14:58:38.605 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_enqueued": 0, 2024-04-23T14:58:38.606 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_prioritized": 0, 2024-04-23T14:58:38.606 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_started": 0, 2024-04-23T14:58:38.606 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_completed": 0, 2024-04-23T14:58:38.606 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_path": 0, 2024-04-23T14:58:38.606 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_inode": 0, 2024-04-23T14:58:38.606 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_enqueue_scrub": 0, 2024-04-23T14:58:38.606 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_exportdir": 0, 2024-04-23T14:58:38.606 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_flush": 0, 2024-04-23T14:58:38.606 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragmentdir": 0, 2024-04-23T14:58:38.606 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragstats": 0, 2024-04-23T14:58:38.607 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_inodestats": 0 2024-04-23T14:58:38.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.607 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log": { 2024-04-23T14:58:38.607 INFO:teuthology.orchestra.run.smithi028.stdout: "evlrg": 0, 2024-04-23T14:58:38.607 INFO:teuthology.orchestra.run.smithi028.stdout: "evadd": 0, 2024-04-23T14:58:38.607 INFO:teuthology.orchestra.run.smithi028.stdout: "evex": 0, 2024-04-23T14:58:38.607 INFO:teuthology.orchestra.run.smithi028.stdout: "evtrm": 0, 2024-04-23T14:58:38.607 INFO:teuthology.orchestra.run.smithi028.stdout: "ev": 199338, 2024-04-23T14:58:38.607 INFO:teuthology.orchestra.run.smithi028.stdout: "evexg": 0, 2024-04-23T14:58:38.607 INFO:teuthology.orchestra.run.smithi028.stdout: "evexd": 0, 2024-04-23T14:58:38.608 INFO:teuthology.orchestra.run.smithi028.stdout: "segadd": 0, 2024-04-23T14:58:38.608 INFO:teuthology.orchestra.run.smithi028.stdout: "segex": 0, 2024-04-23T14:58:38.608 INFO:teuthology.orchestra.run.smithi028.stdout: "segtrm": 0, 2024-04-23T14:58:38.608 INFO:teuthology.orchestra.run.smithi028.stdout: "seg": 147, 2024-04-23T14:58:38.608 INFO:teuthology.orchestra.run.smithi028.stdout: "segmjr": 18, 2024-04-23T14:58:38.608 INFO:teuthology.orchestra.run.smithi028.stdout: "segexg": 0, 2024-04-23T14:58:38.608 INFO:teuthology.orchestra.run.smithi028.stdout: "segexd": 0, 2024-04-23T14:58:38.608 INFO:teuthology.orchestra.run.smithi028.stdout: "expos": 539245793, 2024-04-23T14:58:38.608 INFO:teuthology.orchestra.run.smithi028.stdout: "wrpos": 1114214842, 2024-04-23T14:58:38.608 INFO:teuthology.orchestra.run.smithi028.stdout: "rdpos": 1114206605, 2024-04-23T14:58:38.609 INFO:teuthology.orchestra.run.smithi028.stdout: "jlat": { 2024-04-23T14:58:38.609 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.609 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.609 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.609 INFO:teuthology.orchestra.run.smithi028.stdout: "replayed": 199338 2024-04-23T14:58:38.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.609 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_mem": { 2024-04-23T14:58:38.609 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 2581, 2024-04-23T14:58:38.610 INFO:teuthology.orchestra.run.smithi028.stdout: "ino+": 46445, 2024-04-23T14:58:38.610 INFO:teuthology.orchestra.run.smithi028.stdout: "ino-": 43864, 2024-04-23T14:58:38.610 INFO:teuthology.orchestra.run.smithi028.stdout: "dir": 182, 2024-04-23T14:58:38.610 INFO:teuthology.orchestra.run.smithi028.stdout: "dir+": 300, 2024-04-23T14:58:38.610 INFO:teuthology.orchestra.run.smithi028.stdout: "dir-": 118, 2024-04-23T14:58:38.610 INFO:teuthology.orchestra.run.smithi028.stdout: "dn": 47194, 2024-04-23T14:58:38.610 INFO:teuthology.orchestra.run.smithi028.stdout: "dn+": 47194, 2024-04-23T14:58:38.610 INFO:teuthology.orchestra.run.smithi028.stdout: "dn-": 0, 2024-04-23T14:58:38.610 INFO:teuthology.orchestra.run.smithi028.stdout: "cap": 0, 2024-04-23T14:58:38.611 INFO:teuthology.orchestra.run.smithi028.stdout: "cap+": 0, 2024-04-23T14:58:38.611 INFO:teuthology.orchestra.run.smithi028.stdout: "cap-": 0, 2024-04-23T14:58:38.611 INFO:teuthology.orchestra.run.smithi028.stdout: "rss": 102768, 2024-04-23T14:58:38.611 INFO:teuthology.orchestra.run.smithi028.stdout: "heap": 215416 2024-04-23T14:58:38.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.611 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_server": { 2024-04-23T14:58:38.611 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_client_request": 0, 2024-04-23T14:58:38.611 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_server_request": 0, 2024-04-23T14:58:38.611 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_request": 0, 2024-04-23T14:58:38.611 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_session": 0, 2024-04-23T14:58:38.612 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_peer_request": 0, 2024-04-23T14:58:38.612 INFO:teuthology.orchestra.run.smithi028.stdout: "req_create_latency": { 2024-04-23T14:58:38.612 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.612 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.612 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.612 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getattr_latency": { 2024-04-23T14:58:38.612 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.612 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.612 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.613 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getfilelock_latency": { 2024-04-23T14:58:38.613 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.613 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.613 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.613 INFO:teuthology.orchestra.run.smithi028.stdout: "req_link_latency": { 2024-04-23T14:58:38.613 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.613 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.614 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.614 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookup_latency": { 2024-04-23T14:58:38.614 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.614 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.614 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.614 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookuphash_latency": { 2024-04-23T14:58:38.614 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.615 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.615 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.615 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupino_latency": { 2024-04-23T14:58:38.615 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.615 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.615 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.615 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupname_latency": { 2024-04-23T14:58:38.615 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.616 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.616 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.616 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupparent_latency": { 2024-04-23T14:58:38.616 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.616 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.616 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.616 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupsnap_latency": { 2024-04-23T14:58:38.616 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.617 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.617 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.617 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lssnap_latency": { 2024-04-23T14:58:38.617 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.617 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.617 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.617 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mkdir_latency": { 2024-04-23T14:58:38.618 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.618 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.618 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.618 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mknod_latency": { 2024-04-23T14:58:38.618 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.618 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.618 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.618 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mksnap_latency": { 2024-04-23T14:58:38.619 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.619 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.619 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.619 INFO:teuthology.orchestra.run.smithi028.stdout: "req_open_latency": { 2024-04-23T14:58:38.619 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.619 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.619 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.620 INFO:teuthology.orchestra.run.smithi028.stdout: "req_readdir_latency": { 2024-04-23T14:58:38.620 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.620 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.620 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.620 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rename_latency": { 2024-04-23T14:58:38.620 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.620 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.620 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.621 INFO:teuthology.orchestra.run.smithi028.stdout: "req_renamesnap_latency": { 2024-04-23T14:58:38.621 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.621 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.621 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.621 INFO:teuthology.orchestra.run.smithi028.stdout: "req_snapdiff_latency": { 2024-04-23T14:58:38.621 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.621 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.621 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.622 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmdir_latency": { 2024-04-23T14:58:38.622 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.622 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.622 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.622 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmsnap_latency": { 2024-04-23T14:58:38.622 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.623 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.623 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.623 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmxattr_latency": { 2024-04-23T14:58:38.623 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.623 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.623 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.623 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setattr_latency": { 2024-04-23T14:58:38.624 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.624 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.624 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.624 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setdirlayout_latency": { 2024-04-23T14:58:38.624 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.624 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.624 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.625 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setfilelock_latency": { 2024-04-23T14:58:38.625 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.625 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.625 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.625 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setlayout_latency": { 2024-04-23T14:58:38.625 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.625 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.625 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.626 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setxattr_latency": { 2024-04-23T14:58:38.626 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.626 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.626 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.626 INFO:teuthology.orchestra.run.smithi028.stdout: "req_symlink_latency": { 2024-04-23T14:58:38.626 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.627 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.627 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.627 INFO:teuthology.orchestra.run.smithi028.stdout: "req_unlink_latency": { 2024-04-23T14:58:38.627 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.627 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.627 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.628 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_revoke_eviction": 0, 2024-04-23T14:58:38.628 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_acquisition_throttle": 0, 2024-04-23T14:58:38.628 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getvxattr_latency": { 2024-04-23T14:58:38.628 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.628 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.628 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.628 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:38.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.628 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_sessions": { 2024-04-23T14:58:38.629 INFO:teuthology.orchestra.run.smithi028.stdout: "session_count": 2, 2024-04-23T14:58:38.629 INFO:teuthology.orchestra.run.smithi028.stdout: "session_add": 5, 2024-04-23T14:58:38.629 INFO:teuthology.orchestra.run.smithi028.stdout: "session_remove": 3, 2024-04-23T14:58:38.629 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_open": 4, 2024-04-23T14:58:38.629 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_stale": 0, 2024-04-23T14:58:38.629 INFO:teuthology.orchestra.run.smithi028.stdout: "total_load": 0, 2024-04-23T14:58:38.629 INFO:teuthology.orchestra.run.smithi028.stdout: "average_load": 0, 2024-04-23T14:58:38.629 INFO:teuthology.orchestra.run.smithi028.stdout: "avg_session_uptime": 3411, 2024-04-23T14:58:38.629 INFO:teuthology.orchestra.run.smithi028.stdout: "mdthresh_evicted": 0 2024-04-23T14:58:38.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.630 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-04-23T14:58:38.630 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-04-23T14:58:38.630 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-04-23T14:58:38.630 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T14:58:38.630 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-04-23T14:58:38.630 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T14:58:38.630 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-04-23T14:58:38.630 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T14:58:38.631 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T14:58:38.631 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T14:58:38.631 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T14:58:38.631 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T14:58:38.631 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-04-23T14:58:38.631 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T14:58:38.631 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T14:58:38.631 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-04-23T14:58:38.631 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-04-23T14:58:38.632 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-04-23T14:58:38.632 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-04-23T14:58:38.632 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T14:58:38.632 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T14:58:38.632 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T14:58:38.632 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T14:58:38.632 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T14:58:38.632 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-04-23T14:58:38.632 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-04-23T14:58:38.632 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-04-23T14:58:38.633 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T14:58:38.633 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T14:58:38.633 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-04-23T14:58:38.633 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-04-23T14:58:38.633 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-04-23T14:58:38.633 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-04-23T14:58:38.633 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T14:58:38.633 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-04-23T14:58:38.633 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T14:58:38.633 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-04-23T14:58:38.634 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 347234, 2024-04-23T14:58:38.634 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 2966, 2024-04-23T14:58:38.634 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-04-23T14:58:38.634 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-04-23T14:58:38.634 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-04-23T14:58:38.634 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-04-23T14:58:38.634 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-04-23T14:58:38.634 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-04-23T14:58:38.634 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-04-23T14:58:38.635 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-04-23T14:58:38.635 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 8128, 2024-04-23T14:58:38.635 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 169, 2024-04-23T14:58:38.635 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T14:58:38.635 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-04-23T14:58:38.635 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-04-23T14:58:38.635 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-04-23T14:58:38.635 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 39966908, 2024-04-23T14:58:38.635 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 308484, 2024-04-23T14:58:38.636 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-04-23T14:58:38.636 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-04-23T14:58:38.636 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-04-23T14:58:38.636 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-04-23T14:58:38.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.636 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter": { 2024-04-23T14:58:38.636 INFO:teuthology.orchestra.run.smithi028.stdout: "op_active": 0, 2024-04-23T14:58:38.636 INFO:teuthology.orchestra.run.smithi028.stdout: "op_laggy": 0, 2024-04-23T14:58:38.636 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send": 5043, 2024-04-23T14:58:38.636 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send_bytes": 26, 2024-04-23T14:58:38.637 INFO:teuthology.orchestra.run.smithi028.stdout: "op_resend": 4, 2024-04-23T14:58:38.637 INFO:teuthology.orchestra.run.smithi028.stdout: "op_reply": 5043, 2024-04-23T14:58:38.637 INFO:teuthology.orchestra.run.smithi028.stdout: "op_latency": { 2024-04-23T14:58:38.637 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 5043, 2024-04-23T14:58:38.637 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 10.743695988, 2024-04-23T14:58:38.637 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.002130417 2024-04-23T14:58:38.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.637 INFO:teuthology.orchestra.run.smithi028.stdout: "op_inflight": 0, 2024-04-23T14:58:38.637 INFO:teuthology.orchestra.run.smithi028.stdout: "oplen_avg": { 2024-04-23T14:58:38.637 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 5043, 2024-04-23T14:58:38.638 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 5044 2024-04-23T14:58:38.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.638 INFO:teuthology.orchestra.run.smithi028.stdout: "op": 5043, 2024-04-23T14:58:38.638 INFO:teuthology.orchestra.run.smithi028.stdout: "op_r": 5043, 2024-04-23T14:58:38.638 INFO:teuthology.orchestra.run.smithi028.stdout: "op_w": 0, 2024-04-23T14:58:38.638 INFO:teuthology.orchestra.run.smithi028.stdout: "op_rmw": 0, 2024-04-23T14:58:38.638 INFO:teuthology.orchestra.run.smithi028.stdout: "op_pg": 0, 2024-04-23T14:58:38.638 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_stat": 3063, 2024-04-23T14:58:38.638 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_create": 0, 2024-04-23T14:58:38.639 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_read": 1977, 2024-04-23T14:58:38.639 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_write": 0, 2024-04-23T14:58:38.639 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writefull": 0, 2024-04-23T14:58:38.639 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writesame": 0, 2024-04-23T14:58:38.639 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_append": 0, 2024-04-23T14:58:38.639 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_zero": 0, 2024-04-23T14:58:38.639 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_truncate": 0, 2024-04-23T14:58:38.639 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_delete": 0, 2024-04-23T14:58:38.639 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_mapext": 0, 2024-04-23T14:58:38.640 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_sparse_read": 0, 2024-04-23T14:58:38.640 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_clonerange": 0, 2024-04-23T14:58:38.640 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_getxattr": 2, 2024-04-23T14:58:38.640 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_setxattr": 0, 2024-04-23T14:58:38.640 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_cmpxattr": 0, 2024-04-23T14:58:38.640 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_rmxattr": 0, 2024-04-23T14:58:38.640 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_resetxattrs": 0, 2024-04-23T14:58:38.640 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_call": 0, 2024-04-23T14:58:38.640 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_watch": 0, 2024-04-23T14:58:38.640 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_notify": 0, 2024-04-23T14:58:38.641 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_src_cmpxattr": 0, 2024-04-23T14:58:38.641 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls": 0, 2024-04-23T14:58:38.641 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls_filter": 0, 2024-04-23T14:58:38.641 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_other": 0, 2024-04-23T14:58:38.641 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_active": 0, 2024-04-23T14:58:38.641 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_send": 0, 2024-04-23T14:58:38.641 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_resend": 0, 2024-04-23T14:58:38.641 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_ping": 0, 2024-04-23T14:58:38.641 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_active": 0, 2024-04-23T14:58:38.642 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_send": 0, 2024-04-23T14:58:38.642 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_resend": 0, 2024-04-23T14:58:38.642 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_active": 0, 2024-04-23T14:58:38.642 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_send": 0, 2024-04-23T14:58:38.642 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_resend": 0, 2024-04-23T14:58:38.642 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_active": 0, 2024-04-23T14:58:38.642 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_send": 0, 2024-04-23T14:58:38.642 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_resend": 0, 2024-04-23T14:58:38.643 INFO:teuthology.orchestra.run.smithi028.stdout: "command_active": 0, 2024-04-23T14:58:38.643 INFO:teuthology.orchestra.run.smithi028.stdout: "command_send": 0, 2024-04-23T14:58:38.643 INFO:teuthology.orchestra.run.smithi028.stdout: "command_resend": 0, 2024-04-23T14:58:38.643 INFO:teuthology.orchestra.run.smithi028.stdout: "map_epoch": 104, 2024-04-23T14:58:38.643 INFO:teuthology.orchestra.run.smithi028.stdout: "map_full": 0, 2024-04-23T14:58:38.643 INFO:teuthology.orchestra.run.smithi028.stdout: "map_inc": 19, 2024-04-23T14:58:38.643 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_sessions": 11, 2024-04-23T14:58:38.643 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_open": 11, 2024-04-23T14:58:38.643 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_close": 0, 2024-04-23T14:58:38.644 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_laggy": 0, 2024-04-23T14:58:38.644 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_wr": 0, 2024-04-23T14:58:38.644 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_rd": 2, 2024-04-23T14:58:38.644 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_del": 0 2024-04-23T14:58:38.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.644 INFO:teuthology.orchestra.run.smithi028.stdout: "oft": { 2024-04-23T14:58:38.644 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_objs": 0, 2024-04-23T14:58:38.644 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_kv_pairs": 0, 2024-04-23T14:58:38.644 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_updates": 0, 2024-04-23T14:58:38.644 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_removes": 0 2024-04-23T14:58:38.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.645 INFO:teuthology.orchestra.run.smithi028.stdout: "osd-slow-ops": { 2024-04-23T14:58:38.645 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_ops_count": 0 2024-04-23T14:58:38.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.645 INFO:teuthology.orchestra.run.smithi028.stdout: "purge_queue": { 2024-04-23T14:58:38.645 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops": 0, 2024-04-23T14:58:38.645 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops_high_water": 0, 2024-04-23T14:58:38.645 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing": 0, 2024-04-23T14:58:38.645 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_high_water": 0, 2024-04-23T14:58:38.645 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed_ops": 0, 2024-04-23T14:58:38.646 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed": 0, 2024-04-23T14:58:38.646 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_item_in_journal": 0 2024-04-23T14:58:38.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.646 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T14:58:38.646 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-23T14:58:38.646 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-04-23T14:58:38.646 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-23T14:58:38.646 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 5358, 2024-04-23T14:58:38.646 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 1112685097, 2024-04-23T14:58:38.647 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-23T14:58:38.647 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 5358, 2024-04-23T14:58:38.647 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-04-23T14:58:38.647 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-04-23T14:58:38.647 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 5358, 2024-04-23T14:58:38.647 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 1112685097, 2024-04-23T14:58:38.647 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-23T14:58:38.647 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.647 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.647 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.648 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:38.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.648 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_bytes": { 2024-04-23T14:58:38.648 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-23T14:58:38.648 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-04-23T14:58:38.648 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-23T14:58:38.648 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-04-23T14:58:38.648 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-04-23T14:58:38.648 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-23T14:58:38.648 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-04-23T14:58:38.649 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 5043, 2024-04-23T14:58:38.649 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 1110037729, 2024-04-23T14:58:38.649 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 5043, 2024-04-23T14:58:38.649 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 1110037729, 2024-04-23T14:58:38.649 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-23T14:58:38.649 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.649 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.649 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.649 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:38.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.650 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_ops": { 2024-04-23T14:58:38.650 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-23T14:58:38.650 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 1024, 2024-04-23T14:58:38.650 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-23T14:58:38.650 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-04-23T14:58:38.650 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-04-23T14:58:38.650 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-23T14:58:38.650 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-04-23T14:58:38.650 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 5043, 2024-04-23T14:58:38.651 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 5043, 2024-04-23T14:58:38.651 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 5043, 2024-04-23T14:58:38.651 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 5043, 2024-04-23T14:58:38.651 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-23T14:58:38.651 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.651 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.651 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.651 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:38.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.651 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle": { 2024-04-23T14:58:38.652 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-23T14:58:38.652 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-04-23T14:58:38.652 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-23T14:58:38.652 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-04-23T14:58:38.652 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-04-23T14:58:38.652 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-23T14:58:38.652 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-04-23T14:58:38.652 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-04-23T14:58:38.652 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-04-23T14:58:38.652 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 0, 2024-04-23T14:58:38.653 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 0, 2024-04-23T14:58:38.653 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-23T14:58:38.653 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.653 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.653 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.653 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:38.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:38.653 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle-0x5609b6c1c780": { 2024-04-23T14:58:38.653 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-23T14:58:38.653 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-04-23T14:58:38.654 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-23T14:58:38.654 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-04-23T14:58:38.654 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-04-23T14:58:38.654 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-23T14:58:38.654 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-04-23T14:58:38.654 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-04-23T14:58:38.654 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-04-23T14:58:38.654 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 0, 2024-04-23T14:58:38.654 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 0, 2024-04-23T14:58:38.655 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-23T14:58:38.655 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:38.655 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:38.655 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:38.655 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:38.655 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:38.655 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:58:39.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:38 smithi105 ceph-mon[32555]: pgmap v457: 97 pgs: 97 active+clean; 834 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 3.1 MiB/s wr, 112 op/s 2024-04-23T14:58:39.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:38 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:58:39.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:38 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:58:39.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:38 smithi105 ceph-mon[32555]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:58:39.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:38 smithi038 ceph-mon[32348]: pgmap v457: 97 pgs: 97 active+clean; 834 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 3.1 MiB/s wr, 112 op/s 2024-04-23T14:58:39.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:38 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:58:39.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:38 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:58:39.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:38 smithi038 ceph-mon[32348]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:58:39.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:38 smithi028 ceph-mon[27810]: pgmap v457: 97 pgs: 97 active+clean; 834 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 3.1 MiB/s wr, 112 op/s 2024-04-23T14:58:39.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:38 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T14:58:39.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:38 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T14:58:39.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:38 smithi028 ceph-mon[27810]: from='mgr.25153 ' entity='mgr.x' 2024-04-23T14:58:39.598 INFO:tasks.check_counter:Daemon mds.g mds.root_rsnaps=6 2024-04-23T14:58:39.599 INFO:tasks.check_counter:Daemon mds.g mds_server.req_mksnap_latency.avgcount=0 2024-04-23T14:58:39.599 INFO:tasks.check_counter:Daemon mds.g mds_server.req_rmsnap_latency.avgcount=0 2024-04-23T14:58:39.599 DEBUG:tasks.check_counter:Getting stats from j 2024-04-23T14:58:39.599 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-23T14:58:40.060 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T14:58:41.163 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:40 smithi028 ceph-mon[27810]: pgmap v458: 97 pgs: 97 active+clean; 834 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 629 KiB/s wr, 61 op/s 2024-04-23T14:58:41.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:40 smithi105 ceph-mon[32555]: pgmap v458: 97 pgs: 97 active+clean; 834 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 629 KiB/s wr, 61 op/s 2024-04-23T14:58:41.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:40 smithi038 ceph-mon[32348]: pgmap v458: 97 pgs: 97 active+clean; 834 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 629 KiB/s wr, 61 op/s 2024-04-23T14:58:42.426 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-23T14:58:42.426 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T14:58:42.426 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 0, 2024-04-23T14:58:42.426 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 2, 2024-04-23T14:58:42.426 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 0, 2024-04-23T14:58:42.427 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 0, 2024-04-23T14:58:42.427 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 4, 2024-04-23T14:58:42.427 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 0, 2024-04-23T14:58:42.427 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.003781029, 2024-04-23T14:58:42.427 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.000000000, 2024-04-23T14:58:42.427 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.000711612, 2024-04-23T14:58:42.427 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T14:58:42.427 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:58:42.427 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:42.427 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:42.428 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:42.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:42.428 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:58:42.428 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:42.428 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:42.428 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:42.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:42.428 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-23T14:58:42.428 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 0 2024-04-23T14:58:42.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:42.429 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T14:58:42.429 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 294, 2024-04-23T14:58:42.429 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 282, 2024-04-23T14:58:42.429 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 1748592, 2024-04-23T14:58:42.429 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 174144, 2024-04-23T14:58:42.429 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 2, 2024-04-23T14:58:42.429 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 1, 2024-04-23T14:58:42.429 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.129016824, 2024-04-23T14:58:42.430 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.048727345, 2024-04-23T14:58:42.430 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.084443759, 2024-04-23T14:58:42.430 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.014345633, 2024-04-23T14:58:42.430 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:58:42.430 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 281, 2024-04-23T14:58:42.430 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.030312820, 2024-04-23T14:58:42.430 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000107874 2024-04-23T14:58:42.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:42.430 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:58:42.431 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:42.431 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:42.431 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:42.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:42.431 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1748592, 2024-04-23T14:58:42.431 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 174144 2024-04-23T14:58:42.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:42.431 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T14:58:42.431 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 6, 2024-04-23T14:58:42.432 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 218, 2024-04-23T14:58:42.432 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 3312, 2024-04-23T14:58:42.432 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 194640, 2024-04-23T14:58:42.432 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 5, 2024-04-23T14:58:42.432 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 3, 2024-04-23T14:58:42.432 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.046183263, 2024-04-23T14:58:42.432 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.034668803, 2024-04-23T14:58:42.432 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.004587968, 2024-04-23T14:58:42.432 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T14:58:42.433 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:58:42.433 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 216, 2024-04-23T14:58:42.433 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.027239132, 2024-04-23T14:58:42.433 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000126107 2024-04-23T14:58:42.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:42.433 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:58:42.433 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:42.433 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:42.433 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:42.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:42.434 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-23T14:58:42.434 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 194640 2024-04-23T14:58:42.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:42.434 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-04-23T14:58:42.434 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 0, 2024-04-23T14:58:42.434 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-04-23T14:58:42.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:42.435 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-04-23T14:58:42.435 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-04-23T14:58:42.435 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-04-23T14:58:42.435 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T14:58:42.435 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-04-23T14:58:42.435 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T14:58:42.435 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-04-23T14:58:42.435 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T14:58:42.435 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T14:58:42.436 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T14:58:42.436 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T14:58:42.436 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T14:58:42.436 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-04-23T14:58:42.436 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T14:58:42.436 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T14:58:42.436 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-04-23T14:58:42.436 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-04-23T14:58:42.436 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-04-23T14:58:42.436 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-04-23T14:58:42.437 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T14:58:42.437 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T14:58:42.437 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T14:58:42.437 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T14:58:42.437 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T14:58:42.437 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-04-23T14:58:42.437 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-04-23T14:58:42.437 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-04-23T14:58:42.437 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T14:58:42.438 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T14:58:42.438 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-04-23T14:58:42.438 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-04-23T14:58:42.438 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-04-23T14:58:42.438 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-04-23T14:58:42.438 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T14:58:42.438 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-04-23T14:58:42.438 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T14:58:42.439 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-04-23T14:58:42.439 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 150008, 2024-04-23T14:58:42.439 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 31, 2024-04-23T14:58:42.439 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-04-23T14:58:42.439 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-04-23T14:58:42.439 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-04-23T14:58:42.439 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-04-23T14:58:42.439 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-04-23T14:58:42.439 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-04-23T14:58:42.440 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-04-23T14:58:42.440 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-04-23T14:58:42.440 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 0, 2024-04-23T14:58:42.440 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 0, 2024-04-23T14:58:42.440 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T14:58:42.440 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-04-23T14:58:42.440 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-04-23T14:58:42.440 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-04-23T14:58:42.440 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 928, 2024-04-23T14:58:42.440 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 2, 2024-04-23T14:58:42.441 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-04-23T14:58:42.441 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-04-23T14:58:42.441 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-04-23T14:58:42.441 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-04-23T14:58:42.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-23T14:58:42.441 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T14:58:42.441 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-23T14:58:42.441 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-04-23T14:58:42.442 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-23T14:58:42.442 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 296, 2024-04-23T14:58:42.442 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 1707096, 2024-04-23T14:58:42.442 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-23T14:58:42.442 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 296, 2024-04-23T14:58:42.442 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-04-23T14:58:42.442 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-04-23T14:58:42.442 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 296, 2024-04-23T14:58:42.442 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 1707096, 2024-04-23T14:58:42.443 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-23T14:58:42.443 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-23T14:58:42.443 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-23T14:58:42.443 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-23T14:58:42.443 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:42.443 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-23T14:58:42.443 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-23T14:58:43.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:42 smithi028 ceph-mon[27810]: pgmap v459: 97 pgs: 97 active+clean; 834 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 629 KiB/s wr, 61 op/s 2024-04-23T14:58:43.210 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-04-23T14:58:43.210 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-04-23T14:58:43.210 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-04-23T14:58:43.211 DEBUG:tasks.check_counter:Getting stats from b 2024-04-23T14:58:43.211 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-23T14:58:43.268 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:42 smithi038 ceph-mon[32348]: pgmap v459: 97 pgs: 97 active+clean; 834 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 629 KiB/s wr, 61 op/s 2024-04-23T14:58:43.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:42 smithi105 ceph-mon[32555]: pgmap v459: 97 pgs: 97 active+clean; 834 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 629 KiB/s wr, 61 op/s 2024-04-23T14:58:43.505 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T14:58:44.960 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:44 smithi038 ceph-mon[32348]: pgmap v460: 97 pgs: 97 active+clean; 840 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 3.1 MiB/s wr, 133 op/s 2024-04-23T14:58:45.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:44 smithi105 ceph-mon[32555]: pgmap v460: 97 pgs: 97 active+clean; 840 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 3.1 MiB/s wr, 133 op/s 2024-04-23T14:58:45.342 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-23T14:58:45.343 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T14:58:45.343 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 55636, 2024-04-23T14:58:45.343 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 54856, 2024-04-23T14:58:45.343 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 14390629, 2024-04-23T14:58:45.343 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 563277538, 2024-04-23T14:58:45.343 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 22, 2024-04-23T14:58:45.343 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 5, 2024-04-23T14:58:45.344 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 5.038963090, 2024-04-23T14:58:45.344 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 2.112340509, 2024-04-23T14:58:45.344 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 1.012380086, 2024-04-23T14:58:45.344 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 1.564243993, 2024-04-23T14:58:45.344 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:58:45.344 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 54855, 2024-04-23T14:58:45.344 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 4.000542017, 2024-04-23T14:58:45.344 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000072929 2024-04-23T14:58:45.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.345 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:58:45.345 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 1211, 2024-04-23T14:58:45.345 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000161447, 2024-04-23T14:58:45.345 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000133 2024-04-23T14:58:45.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.345 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 440176, 2024-04-23T14:58:45.345 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 134608 2024-04-23T14:58:45.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.345 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T14:58:45.346 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 46761, 2024-04-23T14:58:45.346 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 46619, 2024-04-23T14:58:45.346 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 15919449, 2024-04-23T14:58:45.346 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 434508250, 2024-04-23T14:58:45.346 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 219, 2024-04-23T14:58:45.346 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 4, 2024-04-23T14:58:45.346 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 4.184658523, 2024-04-23T14:58:45.346 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 1.667693655, 2024-04-23T14:58:45.346 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.904640457, 2024-04-23T14:58:45.347 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 1.309889170, 2024-04-23T14:58:45.347 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:58:45.347 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 46616, 2024-04-23T14:58:45.347 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 3.434828892, 2024-04-23T14:58:45.347 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000073683 2024-04-23T14:58:45.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.347 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:58:45.347 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 415, 2024-04-23T14:58:45.348 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000050785, 2024-04-23T14:58:45.348 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000122 2024-04-23T14:58:45.348 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.348 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 86448, 2024-04-23T14:58:45.348 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 221104 2024-04-23T14:58:45.348 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.348 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T14:58:45.348 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 361457, 2024-04-23T14:58:45.348 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 641057, 2024-04-23T14:58:45.348 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 120910910, 2024-04-23T14:58:45.349 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 656846283, 2024-04-23T14:58:45.349 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 12, 2024-04-23T14:58:45.349 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 6, 2024-04-23T14:58:45.349 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 22.942292821, 2024-04-23T14:58:45.349 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 10.113686489, 2024-04-23T14:58:45.349 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 15.567544662, 2024-04-23T14:58:45.349 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.637486757, 2024-04-23T14:58:45.349 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:58:45.349 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 641056, 2024-04-23T14:58:45.350 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 13.719952773, 2024-04-23T14:58:45.350 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000021402 2024-04-23T14:58:45.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.350 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:58:45.350 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 700977, 2024-04-23T14:58:45.350 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.126414156, 2024-04-23T14:58:45.350 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000180 2024-04-23T14:58:45.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.351 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 1850208, 2024-04-23T14:58:45.351 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 652992 2024-04-23T14:58:45.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.351 INFO:teuthology.orchestra.run.smithi038.stdout: "cct": { 2024-04-23T14:58:45.351 INFO:teuthology.orchestra.run.smithi038.stdout: "total_workers": 1, 2024-04-23T14:58:45.351 INFO:teuthology.orchestra.run.smithi038.stdout: "unhealthy_workers": 0 2024-04-23T14:58:45.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.351 INFO:teuthology.orchestra.run.smithi038.stdout: "finisher-MDSRank": { 2024-04-23T14:58:45.351 INFO:teuthology.orchestra.run.smithi038.stdout: "queue_len": 0, 2024-04-23T14:58:45.351 INFO:teuthology.orchestra.run.smithi038.stdout: "complete_latency": { 2024-04-23T14:58:45.352 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 130038, 2024-04-23T14:58:45.352 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 132.411888087, 2024-04-23T14:58:45.352 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.001018255 2024-04-23T14:58:45.352 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-23T14:58:45.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.352 INFO:teuthology.orchestra.run.smithi038.stdout: "finisher-PurgeQueue": { 2024-04-23T14:58:45.352 INFO:teuthology.orchestra.run.smithi038.stdout: "queue_len": 0, 2024-04-23T14:58:45.352 INFO:teuthology.orchestra.run.smithi038.stdout: "complete_latency": { 2024-04-23T14:58:45.353 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 55864, 2024-04-23T14:58:45.353 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 9.324882269, 2024-04-23T14:58:45.353 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000166921 2024-04-23T14:58:45.353 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-23T14:58:45.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.353 INFO:teuthology.orchestra.run.smithi038.stdout: "mds": { 2024-04-23T14:58:45.353 INFO:teuthology.orchestra.run.smithi038.stdout: "request": 241407, 2024-04-23T14:58:45.353 INFO:teuthology.orchestra.run.smithi038.stdout: "reply": 241407, 2024-04-23T14:58:45.353 INFO:teuthology.orchestra.run.smithi038.stdout: "reply_latency": { 2024-04-23T14:58:45.354 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 241407, 2024-04-23T14:58:45.354 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 616.367193470, 2024-04-23T14:58:45.354 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.002553228 2024-04-23T14:58:45.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.354 INFO:teuthology.orchestra.run.smithi038.stdout: "slow_reply": 0, 2024-04-23T14:58:45.354 INFO:teuthology.orchestra.run.smithi038.stdout: "forward": 0, 2024-04-23T14:58:45.354 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_fetch_complete": 0, 2024-04-23T14:58:45.354 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_fetch_keys": 0, 2024-04-23T14:58:45.354 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_commit": 246, 2024-04-23T14:58:45.355 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_split": 18, 2024-04-23T14:58:45.355 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_merge": 25, 2024-04-23T14:58:45.355 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes": 5424, 2024-04-23T14:58:45.355 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_top": 1586, 2024-04-23T14:58:45.355 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_bottom": 3838, 2024-04-23T14:58:45.355 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_pin_tail": 0, 2024-04-23T14:58:45.355 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_pinned": 3157, 2024-04-23T14:58:45.355 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_expired": 0, 2024-04-23T14:58:45.355 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_with_caps": 725, 2024-04-23T14:58:45.356 INFO:teuthology.orchestra.run.smithi038.stdout: "caps": 726, 2024-04-23T14:58:45.356 INFO:teuthology.orchestra.run.smithi038.stdout: "subtrees": 2, 2024-04-23T14:58:45.356 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse": 277333, 2024-04-23T14:58:45.356 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_hit": 227161, 2024-04-23T14:58:45.356 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_forward": 0, 2024-04-23T14:58:45.356 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_discover": 0, 2024-04-23T14:58:45.356 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_dir_fetch": 0, 2024-04-23T14:58:45.356 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_remote_ino": 0, 2024-04-23T14:58:45.356 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_lock": 432, 2024-04-23T14:58:45.356 INFO:teuthology.orchestra.run.smithi038.stdout: "load_cent": 0, 2024-04-23T14:58:45.357 INFO:teuthology.orchestra.run.smithi038.stdout: "q": 0, 2024-04-23T14:58:45.357 INFO:teuthology.orchestra.run.smithi038.stdout: "exported": 0, 2024-04-23T14:58:45.357 INFO:teuthology.orchestra.run.smithi038.stdout: "exported_inodes": 0, 2024-04-23T14:58:45.357 INFO:teuthology.orchestra.run.smithi038.stdout: "imported": 0, 2024-04-23T14:58:45.357 INFO:teuthology.orchestra.run.smithi038.stdout: "imported_inodes": 0, 2024-04-23T14:58:45.358 INFO:teuthology.orchestra.run.smithi038.stdout: "openino_dir_fetch": 0, 2024-04-23T14:58:45.358 INFO:teuthology.orchestra.run.smithi038.stdout: "openino_backtrace_fetch": 3180, 2024-04-23T14:58:45.358 INFO:teuthology.orchestra.run.smithi038.stdout: "openino_peer_discover": 0, 2024-04-23T14:58:45.358 INFO:teuthology.orchestra.run.smithi038.stdout: "root_rfiles": 615, 2024-04-23T14:58:45.359 INFO:teuthology.orchestra.run.smithi038.stdout: "root_rbytes": 207020717, 2024-04-23T14:58:45.359 INFO:teuthology.orchestra.run.smithi038.stdout: "root_rsnaps": 6, 2024-04-23T14:58:45.359 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_backtrace_fetch": 3180, 2024-04-23T14:58:45.359 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_set_tag": 0, 2024-04-23T14:58:45.359 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_backtrace_repaired": 0, 2024-04-23T14:58:45.359 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_inotable_repaired": 0, 2024-04-23T14:58:45.359 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_dir_inodes": 647, 2024-04-23T14:58:45.359 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_dir_base_inodes": 28, 2024-04-23T14:58:45.359 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_dirfrag_rstats": 619, 2024-04-23T14:58:45.360 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_file_inodes": 2533, 2024-04-23T14:58:45.360 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_inode_file_caps": 0, 2024-04-23T14:58:45.360 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_revoke": 23492, 2024-04-23T14:58:45.360 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_grant": 222946, 2024-04-23T14:58:45.360 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_trunc": 0, 2024-04-23T14:58:45.360 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-23T14:58:45.360 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-23T14:58:45.360 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_caps": 78164, 2024-04-23T14:58:45.360 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_caps_dirty": 36320, 2024-04-23T14:58:45.361 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_cap_release": 16833, 2024-04-23T14:58:45.361 INFO:teuthology.orchestra.run.smithi038.stdout: "process_request_cap_release": 171323 2024-04-23T14:58:45.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.361 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache": { 2024-04-23T14:58:45.361 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_update": 0, 2024-04-23T14:58:45.361 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_update_receipt": 0, 2024-04-23T14:58:45.361 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_try_discover": 0, 2024-04-23T14:58:45.361 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_send_discover": 0, 2024-04-23T14:58:45.361 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_handle_discover": 0, 2024-04-23T14:58:45.362 INFO:teuthology.orchestra.run.smithi038.stdout: "num_strays": 0, 2024-04-23T14:58:45.362 INFO:teuthology.orchestra.run.smithi038.stdout: "num_strays_delayed": 0, 2024-04-23T14:58:45.362 INFO:teuthology.orchestra.run.smithi038.stdout: "num_strays_enqueuing": 0, 2024-04-23T14:58:45.362 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_created": 44348, 2024-04-23T14:58:45.362 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_enqueued": 44348, 2024-04-23T14:58:45.362 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_reintegrated": 0, 2024-04-23T14:58:45.362 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_migrated": 0, 2024-04-23T14:58:45.362 INFO:teuthology.orchestra.run.smithi038.stdout: "num_recovering_processing": 0, 2024-04-23T14:58:45.362 INFO:teuthology.orchestra.run.smithi038.stdout: "num_recovering_enqueued": 0, 2024-04-23T14:58:45.363 INFO:teuthology.orchestra.run.smithi038.stdout: "num_recovering_prioritized": 0, 2024-04-23T14:58:45.363 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_started": 0, 2024-04-23T14:58:45.363 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_completed": 0, 2024-04-23T14:58:45.363 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_quiesce_path": 0, 2024-04-23T14:58:45.363 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_quiesce_inode": 0, 2024-04-23T14:58:45.363 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_enqueue_scrub": 28, 2024-04-23T14:58:45.363 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_exportdir": 0, 2024-04-23T14:58:45.363 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_flush": 0, 2024-04-23T14:58:45.363 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_fragmentdir": 43, 2024-04-23T14:58:45.364 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_fragstats": 0, 2024-04-23T14:58:45.364 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_inodestats": 0 2024-04-23T14:58:45.364 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.364 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log": { 2024-04-23T14:58:45.364 INFO:teuthology.orchestra.run.smithi038.stdout: "evlrg": 0, 2024-04-23T14:58:45.364 INFO:teuthology.orchestra.run.smithi038.stdout: "evadd": 200292, 2024-04-23T14:58:45.364 INFO:teuthology.orchestra.run.smithi038.stdout: "evex": 134302, 2024-04-23T14:58:45.364 INFO:teuthology.orchestra.run.smithi038.stdout: "evtrm": 122901, 2024-04-23T14:58:45.364 INFO:teuthology.orchestra.run.smithi038.stdout: "ev": 77391, 2024-04-23T14:58:45.365 INFO:teuthology.orchestra.run.smithi038.stdout: "evexg": 0, 2024-04-23T14:58:45.365 INFO:teuthology.orchestra.run.smithi038.stdout: "evexd": 11401, 2024-04-23T14:58:45.365 INFO:teuthology.orchestra.run.smithi038.stdout: "segadd": 329, 2024-04-23T14:58:45.365 INFO:teuthology.orchestra.run.smithi038.stdout: "segex": 201, 2024-04-23T14:58:45.365 INFO:teuthology.orchestra.run.smithi038.stdout: "segtrm": 181, 2024-04-23T14:58:45.365 INFO:teuthology.orchestra.run.smithi038.stdout: "seg": 148, 2024-04-23T14:58:45.365 INFO:teuthology.orchestra.run.smithi038.stdout: "segmjr": 7, 2024-04-23T14:58:45.365 INFO:teuthology.orchestra.run.smithi038.stdout: "segexg": 0, 2024-04-23T14:58:45.365 INFO:teuthology.orchestra.run.smithi038.stdout: "segexd": 20, 2024-04-23T14:58:45.366 INFO:teuthology.orchestra.run.smithi038.stdout: "expos": 539245793, 2024-04-23T14:58:45.366 INFO:teuthology.orchestra.run.smithi038.stdout: "wrpos": 1118678703, 2024-04-23T14:58:45.366 INFO:teuthology.orchestra.run.smithi038.stdout: "rdpos": 0, 2024-04-23T14:58:45.366 INFO:teuthology.orchestra.run.smithi038.stdout: "jlat": { 2024-04-23T14:58:45.366 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 67088, 2024-04-23T14:58:45.366 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 285.939161141, 2024-04-23T14:58:45.366 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.004262150 2024-04-23T14:58:45.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.366 INFO:teuthology.orchestra.run.smithi038.stdout: "replayed": 0 2024-04-23T14:58:45.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.367 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_mem": { 2024-04-23T14:58:45.367 INFO:teuthology.orchestra.run.smithi038.stdout: "ino": 2119, 2024-04-23T14:58:45.367 INFO:teuthology.orchestra.run.smithi038.stdout: "ino+": 46473, 2024-04-23T14:58:45.367 INFO:teuthology.orchestra.run.smithi038.stdout: "ino-": 44354, 2024-04-23T14:58:45.367 INFO:teuthology.orchestra.run.smithi038.stdout: "dir": 182, 2024-04-23T14:58:45.367 INFO:teuthology.orchestra.run.smithi038.stdout: "dir+": 300, 2024-04-23T14:58:45.367 INFO:teuthology.orchestra.run.smithi038.stdout: "dir-": 118, 2024-04-23T14:58:45.367 INFO:teuthology.orchestra.run.smithi038.stdout: "dn": 5350, 2024-04-23T14:58:45.368 INFO:teuthology.orchestra.run.smithi038.stdout: "dn+": 69915, 2024-04-23T14:58:45.368 INFO:teuthology.orchestra.run.smithi038.stdout: "dn-": 64565, 2024-04-23T14:58:45.368 INFO:teuthology.orchestra.run.smithi038.stdout: "cap": 726, 2024-04-23T14:58:45.368 INFO:teuthology.orchestra.run.smithi038.stdout: "cap+": 45245, 2024-04-23T14:58:45.368 INFO:teuthology.orchestra.run.smithi038.stdout: "cap-": 44519, 2024-04-23T14:58:45.368 INFO:teuthology.orchestra.run.smithi038.stdout: "rss": 97672, 2024-04-23T14:58:45.368 INFO:teuthology.orchestra.run.smithi038.stdout: "heap": 231800 2024-04-23T14:58:45.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.368 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_server": { 2024-04-23T14:58:45.369 INFO:teuthology.orchestra.run.smithi038.stdout: "dispatch_client_request": 315074, 2024-04-23T14:58:45.369 INFO:teuthology.orchestra.run.smithi038.stdout: "dispatch_server_request": 0, 2024-04-23T14:58:45.369 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_request": 241407, 2024-04-23T14:58:45.369 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_session": 6157, 2024-04-23T14:58:45.369 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_peer_request": 0, 2024-04-23T14:58:45.369 INFO:teuthology.orchestra.run.smithi038.stdout: "req_create_latency": { 2024-04-23T14:58:45.369 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 44963, 2024-04-23T14:58:45.369 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 79.601296376, 2024-04-23T14:58:45.369 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.001770373 2024-04-23T14:58:45.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.370 INFO:teuthology.orchestra.run.smithi038.stdout: "req_getattr_latency": { 2024-04-23T14:58:45.370 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 164, 2024-04-23T14:58:45.370 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.147368874, 2024-04-23T14:58:45.370 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000898590 2024-04-23T14:58:45.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.370 INFO:teuthology.orchestra.run.smithi038.stdout: "req_getfilelock_latency": { 2024-04-23T14:58:45.370 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-23T14:58:45.370 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-23T14:58:45.371 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-23T14:58:45.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.371 INFO:teuthology.orchestra.run.smithi038.stdout: "req_link_latency": { 2024-04-23T14:58:45.371 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-23T14:58:45.371 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-23T14:58:45.371 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-23T14:58:45.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.371 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookup_latency": { 2024-04-23T14:58:45.371 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 26054, 2024-04-23T14:58:45.371 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 23.259801247, 2024-04-23T14:58:45.372 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000892753 2024-04-23T14:58:45.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.372 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookuphash_latency": { 2024-04-23T14:58:45.372 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-23T14:58:45.372 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-23T14:58:45.372 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-23T14:58:45.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.372 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupino_latency": { 2024-04-23T14:58:45.372 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 2, 2024-04-23T14:58:45.373 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000279679, 2024-04-23T14:58:45.373 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000139839 2024-04-23T14:58:45.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.373 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupname_latency": { 2024-04-23T14:58:45.373 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-23T14:58:45.373 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-23T14:58:45.373 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-23T14:58:45.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.373 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupparent_latency": { 2024-04-23T14:58:45.374 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-23T14:58:45.374 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-23T14:58:45.374 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-23T14:58:45.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.374 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupsnap_latency": { 2024-04-23T14:58:45.374 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-23T14:58:45.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:44 smithi028 ceph-mon[27810]: pgmap v460: 97 pgs: 97 active+clean; 840 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 3.1 MiB/s wr, 133 op/s 2024-04-23T14:58:45.375 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-23T14:58:45.375 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-23T14:58:45.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.376 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lssnap_latency": { 2024-04-23T14:58:45.376 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 14, 2024-04-23T14:58:45.376 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.011591846, 2024-04-23T14:58:45.376 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000827989 2024-04-23T14:58:45.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.376 INFO:teuthology.orchestra.run.smithi038.stdout: "req_mkdir_latency": { 2024-04-23T14:58:45.376 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 119, 2024-04-23T14:58:45.376 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.632261930, 2024-04-23T14:58:45.377 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.005313125 2024-04-23T14:58:45.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.377 INFO:teuthology.orchestra.run.smithi038.stdout: "req_mknod_latency": { 2024-04-23T14:58:45.377 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-23T14:58:45.377 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-23T14:58:45.377 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-23T14:58:45.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.377 INFO:teuthology.orchestra.run.smithi038.stdout: "req_mksnap_latency": { 2024-04-23T14:58:45.377 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 14, 2024-04-23T14:58:45.378 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.173694206, 2024-04-23T14:58:45.378 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.012406729 2024-04-23T14:58:45.378 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.378 INFO:teuthology.orchestra.run.smithi038.stdout: "req_open_latency": { 2024-04-23T14:58:45.378 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 10, 2024-04-23T14:58:45.378 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.011072527, 2024-04-23T14:58:45.378 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.001107252 2024-04-23T14:58:45.378 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.379 INFO:teuthology.orchestra.run.smithi038.stdout: "req_readdir_latency": { 2024-04-23T14:58:45.379 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 114989, 2024-04-23T14:58:45.379 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 392.409186120, 2024-04-23T14:58:45.379 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.003412580 2024-04-23T14:58:45.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.379 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rename_latency": { 2024-04-23T14:58:45.379 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 9289, 2024-04-23T14:58:45.379 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 17.990518808, 2024-04-23T14:58:45.379 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.001936755 2024-04-23T14:58:45.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.380 INFO:teuthology.orchestra.run.smithi038.stdout: "req_renamesnap_latency": { 2024-04-23T14:58:45.380 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-23T14:58:45.380 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-23T14:58:45.380 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-23T14:58:45.380 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.380 INFO:teuthology.orchestra.run.smithi038.stdout: "req_snapdiff_latency": { 2024-04-23T14:58:45.380 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-23T14:58:45.380 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-23T14:58:45.380 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-23T14:58:45.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.381 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rmdir_latency": { 2024-04-23T14:58:45.381 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 1, 2024-04-23T14:58:45.381 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000392438, 2024-04-23T14:58:45.381 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000392438 2024-04-23T14:58:45.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.381 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rmsnap_latency": { 2024-04-23T14:58:45.381 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 8, 2024-04-23T14:58:45.381 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.091221626, 2024-04-23T14:58:45.381 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.011402703 2024-04-23T14:58:45.382 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.382 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rmxattr_latency": { 2024-04-23T14:58:45.382 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-23T14:58:45.382 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-23T14:58:45.382 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-23T14:58:45.382 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.382 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setattr_latency": { 2024-04-23T14:58:45.382 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 1, 2024-04-23T14:58:45.382 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.001083681, 2024-04-23T14:58:45.383 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.001083681 2024-04-23T14:58:45.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.383 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setdirlayout_latency": { 2024-04-23T14:58:45.383 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-23T14:58:45.383 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-23T14:58:45.383 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-23T14:58:45.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.383 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setfilelock_latency": { 2024-04-23T14:58:45.383 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 1424, 2024-04-23T14:58:45.384 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 1.245505129, 2024-04-23T14:58:45.384 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000874652 2024-04-23T14:58:45.384 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.384 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setlayout_latency": { 2024-04-23T14:58:45.384 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-23T14:58:45.384 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-23T14:58:45.384 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-23T14:58:45.384 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.385 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setxattr_latency": { 2024-04-23T14:58:45.385 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 7, 2024-04-23T14:58:45.385 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.020364951, 2024-04-23T14:58:45.385 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.002909278 2024-04-23T14:58:45.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.385 INFO:teuthology.orchestra.run.smithi038.stdout: "req_symlink_latency": { 2024-04-23T14:58:45.385 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-23T14:58:45.385 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-23T14:58:45.385 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-23T14:58:45.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.386 INFO:teuthology.orchestra.run.smithi038.stdout: "req_unlink_latency": { 2024-04-23T14:58:45.386 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 44348, 2024-04-23T14:58:45.386 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 100.771554032, 2024-04-23T14:58:45.386 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.002272290 2024-04-23T14:58:45.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.386 INFO:teuthology.orchestra.run.smithi038.stdout: "cap_revoke_eviction": 0, 2024-04-23T14:58:45.386 INFO:teuthology.orchestra.run.smithi038.stdout: "cap_acquisition_throttle": 0, 2024-04-23T14:58:45.386 INFO:teuthology.orchestra.run.smithi038.stdout: "req_getvxattr_latency": { 2024-04-23T14:58:45.387 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-23T14:58:45.387 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-23T14:58:45.387 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-23T14:58:45.387 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-23T14:58:45.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.387 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_sessions": { 2024-04-23T14:58:45.387 INFO:teuthology.orchestra.run.smithi038.stdout: "session_count": 2, 2024-04-23T14:58:45.387 INFO:teuthology.orchestra.run.smithi038.stdout: "session_add": 5, 2024-04-23T14:58:45.387 INFO:teuthology.orchestra.run.smithi038.stdout: "session_remove": 3, 2024-04-23T14:58:45.388 INFO:teuthology.orchestra.run.smithi038.stdout: "sessions_open": 2, 2024-04-23T14:58:45.388 INFO:teuthology.orchestra.run.smithi038.stdout: "sessions_stale": 0, 2024-04-23T14:58:45.388 INFO:teuthology.orchestra.run.smithi038.stdout: "total_load": 35910, 2024-04-23T14:58:45.388 INFO:teuthology.orchestra.run.smithi038.stdout: "average_load": 17955, 2024-04-23T14:58:45.388 INFO:teuthology.orchestra.run.smithi038.stdout: "avg_session_uptime": 854, 2024-04-23T14:58:45.388 INFO:teuthology.orchestra.run.smithi038.stdout: "mdthresh_evicted": 0 2024-04-23T14:58:45.388 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.388 INFO:teuthology.orchestra.run.smithi038.stdout: "mempool": { 2024-04-23T14:58:45.389 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_bytes": 0, 2024-04-23T14:58:45.389 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_items": 0, 2024-04-23T14:58:45.389 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T14:58:45.389 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_items": 0, 2024-04-23T14:58:45.389 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T14:58:45.389 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_items": 0, 2024-04-23T14:58:45.389 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T14:58:45.389 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T14:58:45.389 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T14:58:45.390 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T14:58:45.390 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T14:58:45.390 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_items": 0, 2024-04-23T14:58:45.390 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T14:58:45.390 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T14:58:45.390 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_bytes": 0, 2024-04-23T14:58:45.390 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_items": 0, 2024-04-23T14:58:45.390 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_bytes": 0, 2024-04-23T14:58:45.390 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_items": 0, 2024-04-23T14:58:45.391 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T14:58:45.391 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T14:58:45.391 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T14:58:45.391 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T14:58:45.391 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T14:58:45.391 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_items": 0, 2024-04-23T14:58:45.391 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_bytes": 0, 2024-04-23T14:58:45.391 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_items": 0, 2024-04-23T14:58:45.391 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T14:58:45.392 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T14:58:45.392 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_bytes": 0, 2024-04-23T14:58:45.392 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_items": 0, 2024-04-23T14:58:45.392 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_bytes": 0, 2024-04-23T14:58:45.392 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_items": 0, 2024-04-23T14:58:45.392 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T14:58:45.392 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_items": 0, 2024-04-23T14:58:45.392 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T14:58:45.392 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_items": 0, 2024-04-23T14:58:45.393 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_bytes": 2503101, 2024-04-23T14:58:45.393 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_items": 2164, 2024-04-23T14:58:45.393 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_bytes": 616, 2024-04-23T14:58:45.393 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_items": 7, 2024-04-23T14:58:45.393 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bytes": 0, 2024-04-23T14:58:45.393 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_items": 0, 2024-04-23T14:58:45.393 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_bytes": 0, 2024-04-23T14:58:45.393 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_items": 0, 2024-04-23T14:58:45.393 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_bytes": 0, 2024-04-23T14:58:45.394 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_items": 0, 2024-04-23T14:58:45.394 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_bytes": 8128, 2024-04-23T14:58:45.394 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_items": 169, 2024-04-23T14:58:45.394 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T14:58:45.394 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_items": 0, 2024-04-23T14:58:45.394 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_bytes": 0, 2024-04-23T14:58:45.394 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_items": 0, 2024-04-23T14:58:45.394 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_bytes": 10390944, 2024-04-23T14:58:45.395 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_items": 128838, 2024-04-23T14:58:45.395 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_bytes": 0, 2024-04-23T14:58:45.395 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_items": 0, 2024-04-23T14:58:45.395 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_bytes": 0, 2024-04-23T14:58:45.395 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_items": 0 2024-04-23T14:58:45.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.395 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter": { 2024-04-23T14:58:45.395 INFO:teuthology.orchestra.run.smithi038.stdout: "op_active": 0, 2024-04-23T14:58:45.395 INFO:teuthology.orchestra.run.smithi038.stdout: "op_laggy": 0, 2024-04-23T14:58:45.396 INFO:teuthology.orchestra.run.smithi038.stdout: "op_send": 118678, 2024-04-23T14:58:45.396 INFO:teuthology.orchestra.run.smithi038.stdout: "op_send_bytes": 1126025837, 2024-04-23T14:58:45.396 INFO:teuthology.orchestra.run.smithi038.stdout: "op_resend": 37, 2024-04-23T14:58:45.396 INFO:teuthology.orchestra.run.smithi038.stdout: "op_reply": 118678, 2024-04-23T14:58:45.396 INFO:teuthology.orchestra.run.smithi038.stdout: "op_latency": { 2024-04-23T14:58:45.396 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 118678, 2024-04-23T14:58:45.396 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 703.424544012, 2024-04-23T14:58:45.396 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.005927168 2024-04-23T14:58:45.396 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.397 INFO:teuthology.orchestra.run.smithi038.stdout: "op_inflight": 0, 2024-04-23T14:58:45.397 INFO:teuthology.orchestra.run.smithi038.stdout: "oplen_avg": { 2024-04-23T14:58:45.397 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 118678, 2024-04-23T14:58:45.397 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 120286 2024-04-23T14:58:45.397 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.397 INFO:teuthology.orchestra.run.smithi038.stdout: "op": 118678, 2024-04-23T14:58:45.397 INFO:teuthology.orchestra.run.smithi038.stdout: "op_r": 3981, 2024-04-23T14:58:45.397 INFO:teuthology.orchestra.run.smithi038.stdout: "op_w": 114697, 2024-04-23T14:58:45.397 INFO:teuthology.orchestra.run.smithi038.stdout: "op_rmw": 0, 2024-04-23T14:58:45.398 INFO:teuthology.orchestra.run.smithi038.stdout: "op_pg": 0, 2024-04-23T14:58:45.398 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_stat": 45, 2024-04-23T14:58:45.398 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_create": 47, 2024-04-23T14:58:45.398 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_read": 773, 2024-04-23T14:58:45.398 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_write": 67833, 2024-04-23T14:58:45.398 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_writefull": 417, 2024-04-23T14:58:45.398 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_writesame": 0, 2024-04-23T14:58:45.398 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_append": 0, 2024-04-23T14:58:45.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_zero": 0, 2024-04-23T14:58:45.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_truncate": 18, 2024-04-23T14:58:45.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_delete": 45354, 2024-04-23T14:58:45.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_mapext": 0, 2024-04-23T14:58:45.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_sparse_read": 0, 2024-04-23T14:58:45.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_clonerange": 0, 2024-04-23T14:58:45.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_getxattr": 3208, 2024-04-23T14:58:45.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_setxattr": 94, 2024-04-23T14:58:45.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_cmpxattr": 0, 2024-04-23T14:58:45.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_rmxattr": 0, 2024-04-23T14:58:45.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_resetxattrs": 0, 2024-04-23T14:58:45.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_call": 0, 2024-04-23T14:58:45.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_watch": 0, 2024-04-23T14:58:45.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_notify": 0, 2024-04-23T14:58:45.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_src_cmpxattr": 0, 2024-04-23T14:58:45.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_pgls": 0, 2024-04-23T14:58:45.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_pgls_filter": 0, 2024-04-23T14:58:45.401 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_other": 0, 2024-04-23T14:58:45.401 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_active": 0, 2024-04-23T14:58:45.401 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_send": 0, 2024-04-23T14:58:45.401 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_resend": 0, 2024-04-23T14:58:45.401 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_ping": 0, 2024-04-23T14:58:45.401 INFO:teuthology.orchestra.run.smithi038.stdout: "poolop_active": 0, 2024-04-23T14:58:45.401 INFO:teuthology.orchestra.run.smithi038.stdout: "poolop_send": 0, 2024-04-23T14:58:45.401 INFO:teuthology.orchestra.run.smithi038.stdout: "poolop_resend": 0, 2024-04-23T14:58:45.401 INFO:teuthology.orchestra.run.smithi038.stdout: "poolstat_active": 0, 2024-04-23T14:58:45.402 INFO:teuthology.orchestra.run.smithi038.stdout: "poolstat_send": 0, 2024-04-23T14:58:45.402 INFO:teuthology.orchestra.run.smithi038.stdout: "poolstat_resend": 0, 2024-04-23T14:58:45.402 INFO:teuthology.orchestra.run.smithi038.stdout: "statfs_active": 0, 2024-04-23T14:58:45.402 INFO:teuthology.orchestra.run.smithi038.stdout: "statfs_send": 0, 2024-04-23T14:58:45.402 INFO:teuthology.orchestra.run.smithi038.stdout: "statfs_resend": 0, 2024-04-23T14:58:45.402 INFO:teuthology.orchestra.run.smithi038.stdout: "command_active": 0, 2024-04-23T14:58:45.402 INFO:teuthology.orchestra.run.smithi038.stdout: "command_send": 0, 2024-04-23T14:58:45.402 INFO:teuthology.orchestra.run.smithi038.stdout: "command_resend": 0, 2024-04-23T14:58:45.402 INFO:teuthology.orchestra.run.smithi038.stdout: "map_epoch": 104, 2024-04-23T14:58:45.403 INFO:teuthology.orchestra.run.smithi038.stdout: "map_full": 0, 2024-04-23T14:58:45.403 INFO:teuthology.orchestra.run.smithi038.stdout: "map_inc": 20, 2024-04-23T14:58:45.403 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_sessions": 12, 2024-04-23T14:58:45.403 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_session_open": 12, 2024-04-23T14:58:45.403 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_session_close": 0, 2024-04-23T14:58:45.403 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_laggy": 0, 2024-04-23T14:58:45.403 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_wr": 2017, 2024-04-23T14:58:45.403 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_rd": 0, 2024-04-23T14:58:45.404 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_del": 480 2024-04-23T14:58:45.404 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.404 INFO:teuthology.orchestra.run.smithi038.stdout: "oft": { 2024-04-23T14:58:45.404 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_objs": 1, 2024-04-23T14:58:45.404 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_kv_pairs": 0, 2024-04-23T14:58:45.404 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_updates": 0, 2024-04-23T14:58:45.404 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_removes": 0 2024-04-23T14:58:45.404 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.404 INFO:teuthology.orchestra.run.smithi038.stdout: "osd-slow-ops": { 2024-04-23T14:58:45.404 INFO:teuthology.orchestra.run.smithi038.stdout: "slow_ops_count": 0 2024-04-23T14:58:45.405 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.405 INFO:teuthology.orchestra.run.smithi038.stdout: "purge_queue": { 2024-04-23T14:58:45.405 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing_ops": 0, 2024-04-23T14:58:45.405 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing_ops_high_water": 32, 2024-04-23T14:58:45.405 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing": 0, 2024-04-23T14:58:45.405 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing_high_water": 32, 2024-04-23T14:58:45.405 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executed_ops": 44348, 2024-04-23T14:58:45.405 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executed": 44348, 2024-04-23T14:58:45.406 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_item_in_journal": 0 2024-04-23T14:58:45.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.406 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T14:58:45.406 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-23T14:58:45.406 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 104857600, 2024-04-23T14:58:45.406 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-23T14:58:45.406 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 463850, 2024-04-23T14:58:45.406 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 109390272, 2024-04-23T14:58:45.407 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-23T14:58:45.407 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 463850, 2024-04-23T14:58:45.407 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-23T14:58:45.407 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-23T14:58:45.407 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 463850, 2024-04-23T14:58:45.407 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 109390272, 2024-04-23T14:58:45.407 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-23T14:58:45.407 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-23T14:58:45.407 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-23T14:58:45.408 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-23T14:58:45.408 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-23T14:58:45.408 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.408 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-objecter_bytes": { 2024-04-23T14:58:45.408 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-23T14:58:45.408 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 104857600, 2024-04-23T14:58:45.408 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-23T14:58:45.408 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 0, 2024-04-23T14:58:45.409 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 0, 2024-04-23T14:58:45.409 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-23T14:58:45.409 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 0, 2024-04-23T14:58:45.409 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 118678, 2024-04-23T14:58:45.409 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 1130513897, 2024-04-23T14:58:45.409 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 118678, 2024-04-23T14:58:45.409 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 1130513897, 2024-04-23T14:58:45.409 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-23T14:58:45.409 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-23T14:58:45.410 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-23T14:58:45.410 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-23T14:58:45.410 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-23T14:58:45.410 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.410 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-objecter_ops": { 2024-04-23T14:58:45.410 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-23T14:58:45.410 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 1024, 2024-04-23T14:58:45.410 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-23T14:58:45.411 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 0, 2024-04-23T14:58:45.411 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 0, 2024-04-23T14:58:45.411 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-23T14:58:45.411 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 0, 2024-04-23T14:58:45.411 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 118678, 2024-04-23T14:58:45.411 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 118678, 2024-04-23T14:58:45.411 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 118678, 2024-04-23T14:58:45.411 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 118678, 2024-04-23T14:58:45.411 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-23T14:58:45.411 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-23T14:58:45.412 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-23T14:58:45.412 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-23T14:58:45.412 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-23T14:58:45.412 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.412 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-write_buf_throttle": { 2024-04-23T14:58:45.412 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-23T14:58:45.412 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 3758096384, 2024-04-23T14:58:45.412 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-23T14:58:45.413 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 44348, 2024-04-23T14:58:45.413 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 4488060, 2024-04-23T14:58:45.413 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-23T14:58:45.413 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 44348, 2024-04-23T14:58:45.413 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-23T14:58:45.413 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-23T14:58:45.413 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 745, 2024-04-23T14:58:45.413 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 4488060, 2024-04-23T14:58:45.413 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-23T14:58:45.414 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-23T14:58:45.414 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-23T14:58:45.414 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-23T14:58:45.414 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-23T14:58:45.414 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:45.414 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-write_buf_throttle-0x556256fb6500": { 2024-04-23T14:58:45.414 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 13178, 2024-04-23T14:58:45.414 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 3758096384, 2024-04-23T14:58:45.415 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-23T14:58:45.415 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 200292, 2024-04-23T14:58:45.415 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 1114484399, 2024-04-23T14:58:45.415 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-23T14:58:45.415 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 200292, 2024-04-23T14:58:45.415 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-23T14:58:45.415 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-23T14:58:45.415 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 67088, 2024-04-23T14:58:45.415 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 1114471221, 2024-04-23T14:58:45.416 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-23T14:58:45.416 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-23T14:58:45.416 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-23T14:58:45.416 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-23T14:58:45.416 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-23T14:58:45.416 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-23T14:58:45.416 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-23T14:58:46.194 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-04-23T14:58:46.194 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=14 2024-04-23T14:58:46.195 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=8 2024-04-23T14:58:46.195 DEBUG:tasks.check_counter:Getting stats from e 2024-04-23T14:58:46.195 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-23T14:58:46.443 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T14:58:47.098 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:46 smithi028 ceph-mon[27810]: pgmap v461: 97 pgs: 97 active+clean; 840 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 2.6 MiB/s wr, 86 op/s 2024-04-23T14:58:47.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:46 smithi038 ceph-mon[32348]: pgmap v461: 97 pgs: 97 active+clean; 840 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 2.6 MiB/s wr, 86 op/s 2024-04-23T14:58:47.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:46 smithi105 ceph-mon[32555]: pgmap v461: 97 pgs: 97 active+clean; 840 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 2.6 MiB/s wr, 86 op/s 2024-04-23T14:58:48.404 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-23T14:58:48.405 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T14:58:48.405 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 293, 2024-04-23T14:58:48.405 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 281, 2024-04-23T14:58:48.405 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 1748320, 2024-04-23T14:58:48.405 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 173504, 2024-04-23T14:58:48.405 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 4, 2024-04-23T14:58:48.405 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 0, 2024-04-23T14:58:48.406 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.049394582, 2024-04-23T14:58:48.406 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.013246855, 2024-04-23T14:58:48.406 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.040941350, 2024-04-23T14:58:48.406 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.005756540, 2024-04-23T14:58:48.406 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:58:48.406 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 280, 2024-04-23T14:58:48.406 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.010846580, 2024-04-23T14:58:48.407 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000038737 2024-04-23T14:58:48.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:48.407 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:58:48.407 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-23T14:58:48.407 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-23T14:58:48.407 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-23T14:58:48.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:48.409 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 1748320, 2024-04-23T14:58:48.409 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 173504 2024-04-23T14:58:48.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:48.410 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T14:58:48.410 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 7, 2024-04-23T14:58:48.410 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 219, 2024-04-23T14:58:48.410 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 3472, 2024-04-23T14:58:48.410 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 272992, 2024-04-23T14:58:48.410 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 6, 2024-04-23T14:58:48.410 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 3, 2024-04-23T14:58:48.410 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.012083283, 2024-04-23T14:58:48.411 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.007586379, 2024-04-23T14:58:48.411 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.002186390, 2024-04-23T14:58:48.411 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T14:58:48.411 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:58:48.411 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 216, 2024-04-23T14:58:48.411 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.026341727, 2024-04-23T14:58:48.411 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000121952 2024-04-23T14:58:48.411 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:48.411 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:58:48.412 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-23T14:58:48.412 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-23T14:58:48.412 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-23T14:58:48.412 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:48.412 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 3472, 2024-04-23T14:58:48.412 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 272992 2024-04-23T14:58:48.412 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:48.412 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T14:58:48.412 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 0, 2024-04-23T14:58:48.412 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 2, 2024-04-23T14:58:48.413 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 0, 2024-04-23T14:58:48.413 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 0, 2024-04-23T14:58:48.413 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 2, 2024-04-23T14:58:48.413 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 1, 2024-04-23T14:58:48.413 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.000270969, 2024-04-23T14:58:48.413 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.000000000, 2024-04-23T14:58:48.413 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.000196556, 2024-04-23T14:58:48.413 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T14:58:48.413 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:58:48.414 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-23T14:58:48.414 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-23T14:58:48.414 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-23T14:58:48.414 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:48.414 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:58:48.414 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-23T14:58:48.414 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-23T14:58:48.414 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-23T14:58:48.414 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:48.415 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-23T14:58:48.415 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 0 2024-04-23T14:58:48.415 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:48.415 INFO:teuthology.orchestra.run.smithi038.stdout: "cct": { 2024-04-23T14:58:48.415 INFO:teuthology.orchestra.run.smithi038.stdout: "total_workers": 0, 2024-04-23T14:58:48.415 INFO:teuthology.orchestra.run.smithi038.stdout: "unhealthy_workers": 0 2024-04-23T14:58:48.415 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:48.415 INFO:teuthology.orchestra.run.smithi038.stdout: "mempool": { 2024-04-23T14:58:48.415 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_bytes": 0, 2024-04-23T14:58:48.415 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_items": 0, 2024-04-23T14:58:48.416 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T14:58:48.416 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_items": 0, 2024-04-23T14:58:48.416 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T14:58:48.416 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_items": 0, 2024-04-23T14:58:48.416 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T14:58:48.416 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T14:58:48.416 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T14:58:48.416 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T14:58:48.417 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T14:58:48.417 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_items": 0, 2024-04-23T14:58:48.417 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T14:58:48.417 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T14:58:48.417 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_bytes": 0, 2024-04-23T14:58:48.417 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_items": 0, 2024-04-23T14:58:48.417 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_bytes": 0, 2024-04-23T14:58:48.417 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_items": 0, 2024-04-23T14:58:48.417 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T14:58:48.418 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T14:58:48.418 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T14:58:48.418 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T14:58:48.418 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T14:58:48.418 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_items": 0, 2024-04-23T14:58:48.418 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_bytes": 0, 2024-04-23T14:58:48.418 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_items": 0, 2024-04-23T14:58:48.418 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T14:58:48.419 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T14:58:48.419 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_bytes": 0, 2024-04-23T14:58:48.419 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_items": 0, 2024-04-23T14:58:48.419 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_bytes": 0, 2024-04-23T14:58:48.419 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_items": 0, 2024-04-23T14:58:48.419 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T14:58:48.419 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_items": 0, 2024-04-23T14:58:48.419 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T14:58:48.420 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_items": 0, 2024-04-23T14:58:48.420 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_bytes": 148977, 2024-04-23T14:58:48.420 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_items": 30, 2024-04-23T14:58:48.420 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_bytes": 0, 2024-04-23T14:58:48.420 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_items": 0, 2024-04-23T14:58:48.420 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bytes": 0, 2024-04-23T14:58:48.420 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_items": 0, 2024-04-23T14:58:48.420 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_bytes": 0, 2024-04-23T14:58:48.421 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_items": 0, 2024-04-23T14:58:48.421 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_bytes": 0, 2024-04-23T14:58:48.421 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_items": 0, 2024-04-23T14:58:48.421 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_bytes": 0, 2024-04-23T14:58:48.421 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_items": 0, 2024-04-23T14:58:48.421 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T14:58:48.421 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_items": 0, 2024-04-23T14:58:48.421 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_bytes": 0, 2024-04-23T14:58:48.421 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_items": 0, 2024-04-23T14:58:48.422 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_bytes": 928, 2024-04-23T14:58:48.422 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_items": 2, 2024-04-23T14:58:48.422 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_bytes": 0, 2024-04-23T14:58:48.422 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_items": 0, 2024-04-23T14:58:48.422 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_bytes": 0, 2024-04-23T14:58:48.422 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_items": 0 2024-04-23T14:58:48.422 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:48.422 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T14:58:48.422 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-23T14:58:48.422 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 104857600, 2024-04-23T14:58:48.423 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-23T14:58:48.423 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 296, 2024-04-23T14:58:48.423 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 1706993, 2024-04-23T14:58:48.423 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-23T14:58:48.423 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 296, 2024-04-23T14:58:48.423 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-23T14:58:48.423 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-23T14:58:48.423 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 296, 2024-04-23T14:58:48.423 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 1706993, 2024-04-23T14:58:48.424 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-23T14:58:48.424 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-23T14:58:48.424 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-23T14:58:48.424 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-23T14:58:48.424 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-23T14:58:48.424 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-23T14:58:48.424 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-23T14:58:49.108 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.e 2024-04-23T14:58:49.108 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.e 2024-04-23T14:58:49.108 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.e 2024-04-23T14:58:49.108 DEBUG:tasks.check_counter:Getting stats from h 2024-04-23T14:58:49.109 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-23T14:58:49.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:48 smithi105 ceph-mon[32555]: pgmap v462: 97 pgs: 97 active+clean; 839 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 2.8 MiB/s wr, 92 op/s 2024-04-23T14:58:49.364 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T14:58:49.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:48 smithi038 ceph-mon[32348]: pgmap v462: 97 pgs: 97 active+clean; 839 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 2.8 MiB/s wr, 92 op/s 2024-04-23T14:58:49.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:48 smithi028 ceph-mon[27810]: pgmap v462: 97 pgs: 97 active+clean; 839 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 2.8 MiB/s wr, 92 op/s 2024-04-23T14:58:50.917 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-23T14:58:50.918 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T14:58:50.918 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 0, 2024-04-23T14:58:50.918 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 2, 2024-04-23T14:58:50.918 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 0, 2024-04-23T14:58:50.918 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 0, 2024-04-23T14:58:50.918 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 4, 2024-04-23T14:58:50.918 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 0, 2024-04-23T14:58:50.919 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.001321696, 2024-04-23T14:58:50.919 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.000000000, 2024-04-23T14:58:50.919 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.000246275, 2024-04-23T14:58:50.919 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T14:58:50.919 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:58:50.919 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-23T14:58:50.919 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-23T14:58:50.919 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-23T14:58:50.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:50.920 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:58:50.920 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-23T14:58:50.920 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-23T14:58:50.920 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-23T14:58:50.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:50.920 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-23T14:58:50.920 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 0 2024-04-23T14:58:50.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:50.921 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T14:58:50.921 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 6, 2024-04-23T14:58:50.921 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 216, 2024-04-23T14:58:50.921 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 3312, 2024-04-23T14:58:50.921 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 194288, 2024-04-23T14:58:50.921 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 5, 2024-04-23T14:58:50.922 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 3, 2024-04-23T14:58:50.922 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.011110507, 2024-04-23T14:58:50.922 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.007336300, 2024-04-23T14:58:50.922 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.001668096, 2024-04-23T14:58:50.922 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T14:58:50.922 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:58:50.922 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 214, 2024-04-23T14:58:50.922 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.021504333, 2024-04-23T14:58:50.923 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000100487 2024-04-23T14:58:50.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:50.923 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:58:50.923 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-23T14:58:50.923 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-23T14:58:50.923 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-23T14:58:50.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:50.923 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-23T14:58:50.924 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 194288 2024-04-23T14:58:50.924 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:50.924 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T14:58:50.924 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 292, 2024-04-23T14:58:50.924 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 280, 2024-04-23T14:58:50.924 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 1748048, 2024-04-23T14:58:50.924 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 172864, 2024-04-23T14:58:50.925 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 2, 2024-04-23T14:58:50.925 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 1, 2024-04-23T14:58:50.925 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.048876053, 2024-04-23T14:58:50.925 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.013882957, 2024-04-23T14:58:50.925 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.040184131, 2024-04-23T14:58:50.925 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.005806214, 2024-04-23T14:58:50.925 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:58:50.925 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 279, 2024-04-23T14:58:50.926 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.011438565, 2024-04-23T14:58:50.926 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000040998 2024-04-23T14:58:50.926 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:50.926 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:58:50.926 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-23T14:58:50.926 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-23T14:58:50.926 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-23T14:58:50.926 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:50.927 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 1748048, 2024-04-23T14:58:50.927 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 172864 2024-04-23T14:58:50.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:50.927 INFO:teuthology.orchestra.run.smithi038.stdout: "cct": { 2024-04-23T14:58:50.927 INFO:teuthology.orchestra.run.smithi038.stdout: "total_workers": 0, 2024-04-23T14:58:50.927 INFO:teuthology.orchestra.run.smithi038.stdout: "unhealthy_workers": 0 2024-04-23T14:58:50.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:50.928 INFO:teuthology.orchestra.run.smithi038.stdout: "mempool": { 2024-04-23T14:58:50.928 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_bytes": 0, 2024-04-23T14:58:50.928 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_items": 0, 2024-04-23T14:58:50.928 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T14:58:50.928 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_items": 0, 2024-04-23T14:58:50.928 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T14:58:50.928 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_items": 0, 2024-04-23T14:58:50.928 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T14:58:50.928 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T14:58:50.929 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T14:58:50.929 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T14:58:50.929 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T14:58:50.929 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_items": 0, 2024-04-23T14:58:50.929 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T14:58:50.929 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T14:58:50.929 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_bytes": 0, 2024-04-23T14:58:50.929 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_items": 0, 2024-04-23T14:58:50.930 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_bytes": 0, 2024-04-23T14:58:50.930 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_items": 0, 2024-04-23T14:58:50.930 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T14:58:50.930 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T14:58:50.930 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T14:58:50.930 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T14:58:50.930 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T14:58:50.931 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_items": 0, 2024-04-23T14:58:50.931 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_bytes": 0, 2024-04-23T14:58:50.931 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_items": 0, 2024-04-23T14:58:50.931 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T14:58:50.931 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T14:58:50.931 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_bytes": 0, 2024-04-23T14:58:50.931 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_items": 0, 2024-04-23T14:58:50.931 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_bytes": 0, 2024-04-23T14:58:50.932 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_items": 0, 2024-04-23T14:58:50.932 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T14:58:50.932 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_items": 0, 2024-04-23T14:58:50.932 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T14:58:50.932 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_items": 0, 2024-04-23T14:58:50.932 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_bytes": 148955, 2024-04-23T14:58:50.932 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_items": 30, 2024-04-23T14:58:50.933 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_bytes": 0, 2024-04-23T14:58:50.933 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_items": 0, 2024-04-23T14:58:50.933 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bytes": 0, 2024-04-23T14:58:50.933 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_items": 0, 2024-04-23T14:58:50.933 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_bytes": 0, 2024-04-23T14:58:50.933 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_items": 0, 2024-04-23T14:58:50.933 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_bytes": 0, 2024-04-23T14:58:50.934 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_items": 0, 2024-04-23T14:58:50.934 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_bytes": 0, 2024-04-23T14:58:50.934 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_items": 0, 2024-04-23T14:58:50.934 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T14:58:50.934 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_items": 0, 2024-04-23T14:58:50.934 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_bytes": 0, 2024-04-23T14:58:50.934 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_items": 0, 2024-04-23T14:58:50.934 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_bytes": 928, 2024-04-23T14:58:50.935 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_items": 2, 2024-04-23T14:58:50.935 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_bytes": 0, 2024-04-23T14:58:50.935 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_items": 0, 2024-04-23T14:58:50.935 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_bytes": 0, 2024-04-23T14:58:50.935 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_items": 0 2024-04-23T14:58:50.935 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:50.935 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T14:58:50.936 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-23T14:58:50.936 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 104857600, 2024-04-23T14:58:50.936 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-23T14:58:50.936 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 294, 2024-04-23T14:58:50.936 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 1706836, 2024-04-23T14:58:50.936 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-23T14:58:50.936 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 294, 2024-04-23T14:58:50.936 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-23T14:58:50.937 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-23T14:58:50.937 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 294, 2024-04-23T14:58:50.937 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 1706836, 2024-04-23T14:58:50.937 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-23T14:58:50.937 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-23T14:58:50.937 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-23T14:58:50.937 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-23T14:58:50.937 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-23T14:58:50.938 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-23T14:58:50.938 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-23T14:58:50.938 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:50 smithi038 ceph-mon[32348]: pgmap v463: 97 pgs: 97 active+clean; 839 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 2.7 MiB/s wr, 78 op/s 2024-04-23T14:58:51.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:50 smithi105 ceph-mon[32555]: pgmap v463: 97 pgs: 97 active+clean; 839 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 2.7 MiB/s wr, 78 op/s 2024-04-23T14:58:51.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:50 smithi028 ceph-mon[27810]: pgmap v463: 97 pgs: 97 active+clean; 839 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 2.7 MiB/s wr, 78 op/s 2024-04-23T14:58:51.637 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.h 2024-04-23T14:58:51.638 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.h 2024-04-23T14:58:51.638 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.h 2024-04-23T14:58:51.638 DEBUG:tasks.check_counter:Getting stats from k 2024-04-23T14:58:51.638 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-23T14:58:51.889 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T14:58:53.180 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:52 smithi038 ceph-mon[32348]: pgmap v464: 97 pgs: 97 active+clean; 839 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 2.7 MiB/s wr, 78 op/s 2024-04-23T14:58:53.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:52 smithi105 ceph-mon[32555]: pgmap v464: 97 pgs: 97 active+clean; 839 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 2.7 MiB/s wr, 78 op/s 2024-04-23T14:58:53.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:52 smithi028 ceph-mon[27810]: pgmap v464: 97 pgs: 97 active+clean; 839 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 2.7 MiB/s wr, 78 op/s 2024-04-23T14:58:53.598 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-23T14:58:53.598 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T14:58:53.598 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 297, 2024-04-23T14:58:53.598 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 285, 2024-04-23T14:58:53.598 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 1749408, 2024-04-23T14:58:53.599 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 176064, 2024-04-23T14:58:53.599 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 4, 2024-04-23T14:58:53.599 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 0, 2024-04-23T14:58:53.599 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.049435625, 2024-04-23T14:58:53.599 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.013648959, 2024-04-23T14:58:53.599 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.039090982, 2024-04-23T14:58:53.599 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.006134098, 2024-04-23T14:58:53.599 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:58:53.599 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 284, 2024-04-23T14:58:53.599 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.011667912, 2024-04-23T14:58:53.600 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000041084 2024-04-23T14:58:53.600 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:53.600 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:58:53.600 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-23T14:58:53.600 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-23T14:58:53.600 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-23T14:58:53.600 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:53.600 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 1749408, 2024-04-23T14:58:53.600 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 176064 2024-04-23T14:58:53.601 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:53.601 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T14:58:53.601 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 6, 2024-04-23T14:58:53.601 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 220, 2024-04-23T14:58:53.601 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 3312, 2024-04-23T14:58:53.601 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 194992, 2024-04-23T14:58:53.601 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 5, 2024-04-23T14:58:53.601 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 3, 2024-04-23T14:58:53.601 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.012209705, 2024-04-23T14:58:53.601 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.008042166, 2024-04-23T14:58:53.602 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.001952230, 2024-04-23T14:58:53.602 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T14:58:53.602 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:58:53.602 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 218, 2024-04-23T14:58:53.602 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.021374432, 2024-04-23T14:58:53.602 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000098047 2024-04-23T14:58:53.602 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:53.602 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:58:53.602 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-23T14:58:53.603 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-23T14:58:53.603 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-23T14:58:53.603 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:53.603 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-23T14:58:53.603 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 194992 2024-04-23T14:58:53.603 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:53.603 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T14:58:53.603 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 0, 2024-04-23T14:58:53.603 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 2, 2024-04-23T14:58:53.603 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 0, 2024-04-23T14:58:53.604 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 0, 2024-04-23T14:58:53.604 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 2, 2024-04-23T14:58:53.604 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 1, 2024-04-23T14:58:53.604 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.000293395, 2024-04-23T14:58:53.604 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.000000000, 2024-04-23T14:58:53.604 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.000225082, 2024-04-23T14:58:53.604 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T14:58:53.604 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:58:53.604 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-23T14:58:53.604 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-23T14:58:53.605 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-23T14:58:53.605 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:53.605 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:58:53.605 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-23T14:58:53.605 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-23T14:58:53.605 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-23T14:58:53.605 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:53.605 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-23T14:58:53.605 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 0 2024-04-23T14:58:53.605 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:53.606 INFO:teuthology.orchestra.run.smithi038.stdout: "cct": { 2024-04-23T14:58:53.606 INFO:teuthology.orchestra.run.smithi038.stdout: "total_workers": 0, 2024-04-23T14:58:53.606 INFO:teuthology.orchestra.run.smithi038.stdout: "unhealthy_workers": 0 2024-04-23T14:58:53.606 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:53.606 INFO:teuthology.orchestra.run.smithi038.stdout: "mempool": { 2024-04-23T14:58:53.606 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_bytes": 0, 2024-04-23T14:58:53.606 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_items": 0, 2024-04-23T14:58:53.606 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T14:58:53.606 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_items": 0, 2024-04-23T14:58:53.607 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T14:58:53.607 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_items": 0, 2024-04-23T14:58:53.607 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T14:58:53.607 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T14:58:53.607 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T14:58:53.607 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T14:58:53.607 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T14:58:53.607 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_items": 0, 2024-04-23T14:58:53.607 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T14:58:53.608 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T14:58:53.608 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_bytes": 0, 2024-04-23T14:58:53.608 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_items": 0, 2024-04-23T14:58:53.608 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_bytes": 0, 2024-04-23T14:58:53.608 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_items": 0, 2024-04-23T14:58:53.608 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T14:58:53.608 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T14:58:53.608 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T14:58:53.608 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T14:58:53.609 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T14:58:53.609 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_items": 0, 2024-04-23T14:58:53.609 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_bytes": 0, 2024-04-23T14:58:53.609 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_items": 0, 2024-04-23T14:58:53.609 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T14:58:53.609 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T14:58:53.609 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_bytes": 0, 2024-04-23T14:58:53.609 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_items": 0, 2024-04-23T14:58:53.609 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_bytes": 0, 2024-04-23T14:58:53.609 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_items": 0, 2024-04-23T14:58:53.610 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T14:58:53.610 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_items": 0, 2024-04-23T14:58:53.610 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T14:58:53.610 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_items": 0, 2024-04-23T14:58:53.610 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_bytes": 148754, 2024-04-23T14:58:53.610 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_items": 31, 2024-04-23T14:58:53.610 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_bytes": 0, 2024-04-23T14:58:53.610 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_items": 0, 2024-04-23T14:58:53.610 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bytes": 0, 2024-04-23T14:58:53.611 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_items": 0, 2024-04-23T14:58:53.611 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_bytes": 0, 2024-04-23T14:58:53.611 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_items": 0, 2024-04-23T14:58:53.611 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_bytes": 0, 2024-04-23T14:58:53.611 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_items": 0, 2024-04-23T14:58:53.611 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_bytes": 0, 2024-04-23T14:58:53.611 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_items": 0, 2024-04-23T14:58:53.611 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T14:58:53.611 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_items": 0, 2024-04-23T14:58:53.611 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_bytes": 0, 2024-04-23T14:58:53.612 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_items": 0, 2024-04-23T14:58:53.612 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_bytes": 928, 2024-04-23T14:58:53.612 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_items": 2, 2024-04-23T14:58:53.612 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_bytes": 0, 2024-04-23T14:58:53.612 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_items": 0, 2024-04-23T14:58:53.612 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_bytes": 0, 2024-04-23T14:58:53.612 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_items": 0 2024-04-23T14:58:53.612 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-23T14:58:53.612 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T14:58:53.612 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-23T14:58:53.613 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 104857600, 2024-04-23T14:58:53.613 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-23T14:58:53.613 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 299, 2024-04-23T14:58:53.613 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 1707486, 2024-04-23T14:58:53.613 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-23T14:58:53.613 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 299, 2024-04-23T14:58:53.613 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-23T14:58:53.613 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-23T14:58:53.613 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 299, 2024-04-23T14:58:53.613 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 1707486, 2024-04-23T14:58:53.614 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-23T14:58:53.614 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-23T14:58:53.614 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-23T14:58:53.614 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-23T14:58:53.614 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-23T14:58:53.614 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-23T14:58:53.614 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-23T14:58:54.275 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-04-23T14:58:54.275 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-04-23T14:58:54.276 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-04-23T14:58:54.276 DEBUG:tasks.check_counter:Getting stats from c 2024-04-23T14:58:54.276 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-23T14:58:54.525 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T14:58:55.226 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:54 smithi105 ceph-mon[32555]: pgmap v465: 97 pgs: 97 active+clean; 849 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 3.5 MiB/s wr, 93 op/s 2024-04-23T14:58:55.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:54 smithi038 ceph-mon[32348]: pgmap v465: 97 pgs: 97 active+clean; 849 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 3.5 MiB/s wr, 93 op/s 2024-04-23T14:58:55.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:54 smithi028 ceph-mon[27810]: pgmap v465: 97 pgs: 97 active+clean; 849 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 3.5 MiB/s wr, 93 op/s 2024-04-23T14:58:56.553 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-23T14:58:56.553 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T14:58:56.553 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 0, 2024-04-23T14:58:56.553 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 2, 2024-04-23T14:58:56.554 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 0, 2024-04-23T14:58:56.554 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 0, 2024-04-23T14:58:56.554 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 4, 2024-04-23T14:58:56.554 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 0, 2024-04-23T14:58:56.554 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.001001767, 2024-04-23T14:58:56.554 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.000000000, 2024-04-23T14:58:56.555 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.000144074, 2024-04-23T14:58:56.555 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T14:58:56.555 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:58:56.555 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-23T14:58:56.555 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-23T14:58:56.555 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-23T14:58:56.555 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:58:56.555 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:58:56.556 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-23T14:58:56.556 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-23T14:58:56.556 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-23T14:58:56.556 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:58:56.556 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-23T14:58:56.556 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 0 2024-04-23T14:58:56.556 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:58:56.556 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T14:58:56.556 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 298, 2024-04-23T14:58:56.557 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 284, 2024-04-23T14:58:56.557 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 1751584, 2024-04-23T14:58:56.557 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 174592, 2024-04-23T14:58:56.557 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 2, 2024-04-23T14:58:56.557 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 1, 2024-04-23T14:58:56.557 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.053470731, 2024-04-23T14:58:56.557 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.016316763, 2024-04-23T14:58:56.557 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.040431670, 2024-04-23T14:58:56.557 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.006278688, 2024-04-23T14:58:56.557 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:58:56.558 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 284, 2024-04-23T14:58:56.558 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.013733507, 2024-04-23T14:58:56.558 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000048357 2024-04-23T14:58:56.558 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:58:56.558 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:58:56.558 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-23T14:58:56.558 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-23T14:58:56.558 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-23T14:58:56.558 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:58:56.559 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 1751584, 2024-04-23T14:58:56.559 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 174592 2024-04-23T14:58:56.559 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:58:56.559 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T14:58:56.559 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 2, 2024-04-23T14:58:56.559 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 215, 2024-04-23T14:58:56.559 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 320, 2024-04-23T14:58:56.559 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 194016, 2024-04-23T14:58:56.559 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 5, 2024-04-23T14:58:56.560 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 3, 2024-04-23T14:58:56.560 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.012032994, 2024-04-23T14:58:56.560 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.008144768, 2024-04-23T14:58:56.560 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.001359295, 2024-04-23T14:58:56.560 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T14:58:56.560 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:58:56.560 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 212, 2024-04-23T14:58:56.560 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.024163712, 2024-04-23T14:58:56.560 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000113979 2024-04-23T14:58:56.561 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:58:56.561 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:58:56.561 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-23T14:58:56.561 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-23T14:58:56.561 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-23T14:58:56.561 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:58:56.561 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-23T14:58:56.561 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 194016 2024-04-23T14:58:56.561 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:58:56.561 INFO:teuthology.orchestra.run.smithi105.stdout: "cct": { 2024-04-23T14:58:56.562 INFO:teuthology.orchestra.run.smithi105.stdout: "total_workers": 0, 2024-04-23T14:58:56.562 INFO:teuthology.orchestra.run.smithi105.stdout: "unhealthy_workers": 0 2024-04-23T14:58:56.562 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:58:56.562 INFO:teuthology.orchestra.run.smithi105.stdout: "mempool": { 2024-04-23T14:58:56.562 INFO:teuthology.orchestra.run.smithi105.stdout: "bloom_filter_bytes": 0, 2024-04-23T14:58:56.562 INFO:teuthology.orchestra.run.smithi105.stdout: "bloom_filter_items": 0, 2024-04-23T14:58:56.562 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T14:58:56.562 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_alloc_items": 0, 2024-04-23T14:58:56.562 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T14:58:56.563 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_data_items": 0, 2024-04-23T14:58:56.563 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T14:58:56.563 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T14:58:56.563 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T14:58:56.563 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T14:58:56.563 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T14:58:56.563 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_other_items": 0, 2024-04-23T14:58:56.563 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T14:58:56.563 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T14:58:56.564 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_bytes": 0, 2024-04-23T14:58:56.564 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_items": 0, 2024-04-23T14:58:56.564 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_blob_bytes": 0, 2024-04-23T14:58:56.564 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_blob_items": 0, 2024-04-23T14:58:56.564 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T14:58:56.564 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T14:58:56.564 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T14:58:56.564 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T14:58:56.564 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T14:58:56.565 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_items": 0, 2024-04-23T14:58:56.565 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_txc_bytes": 0, 2024-04-23T14:58:56.565 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_txc_items": 0, 2024-04-23T14:58:56.565 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T14:58:56.565 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T14:58:56.565 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_bytes": 0, 2024-04-23T14:58:56.565 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_items": 0, 2024-04-23T14:58:56.565 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_bytes": 0, 2024-04-23T14:58:56.565 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_items": 0, 2024-04-23T14:58:56.566 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T14:58:56.566 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_reader_items": 0, 2024-04-23T14:58:56.566 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T14:58:56.566 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_writer_items": 0, 2024-04-23T14:58:56.566 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_anon_bytes": 148955, 2024-04-23T14:58:56.566 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_anon_items": 30, 2024-04-23T14:58:56.566 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_meta_bytes": 0, 2024-04-23T14:58:56.566 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_meta_items": 0, 2024-04-23T14:58:56.566 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_bytes": 0, 2024-04-23T14:58:56.567 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_items": 0, 2024-04-23T14:58:56.567 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mapbl_bytes": 0, 2024-04-23T14:58:56.567 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mapbl_items": 0, 2024-04-23T14:58:56.567 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pglog_bytes": 0, 2024-04-23T14:58:56.567 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pglog_items": 0, 2024-04-23T14:58:56.567 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_bytes": 0, 2024-04-23T14:58:56.567 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_items": 0, 2024-04-23T14:58:56.567 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T14:58:56.567 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_mapping_items": 0, 2024-04-23T14:58:56.568 INFO:teuthology.orchestra.run.smithi105.stdout: "pgmap_bytes": 0, 2024-04-23T14:58:56.568 INFO:teuthology.orchestra.run.smithi105.stdout: "pgmap_items": 0, 2024-04-23T14:58:56.568 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_co_bytes": 928, 2024-04-23T14:58:56.568 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_co_items": 2, 2024-04-23T14:58:56.568 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_1_bytes": 0, 2024-04-23T14:58:56.568 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_1_items": 0, 2024-04-23T14:58:56.568 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_2_bytes": 0, 2024-04-23T14:58:56.568 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_2_items": 0 2024-04-23T14:58:56.568 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:58:56.569 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T14:58:56.569 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-04-23T14:58:56.569 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 104857600, 2024-04-23T14:58:56.569 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-04-23T14:58:56.569 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 296, 2024-04-23T14:58:56.569 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 1707096, 2024-04-23T14:58:56.569 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-04-23T14:58:56.569 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 296, 2024-04-23T14:58:56.569 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 0, 2024-04-23T14:58:56.570 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 0, 2024-04-23T14:58:56.570 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 296, 2024-04-23T14:58:56.570 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 1707096, 2024-04-23T14:58:56.570 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-04-23T14:58:56.570 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-23T14:58:56.570 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-23T14:58:56.570 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-23T14:58:56.570 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-23T14:58:56.570 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-23T14:58:56.571 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-23T14:58:57.237 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:56 smithi105 ceph-mon[32555]: pgmap v466: 97 pgs: 97 active+clean; 849 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 1.0 MiB/s wr, 21 op/s 2024-04-23T14:58:57.265 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.c 2024-04-23T14:58:57.265 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.c 2024-04-23T14:58:57.265 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.c 2024-04-23T14:58:57.265 DEBUG:tasks.check_counter:Getting stats from f 2024-04-23T14:58:57.265 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-23T14:58:57.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:56 smithi038 ceph-mon[32348]: pgmap v466: 97 pgs: 97 active+clean; 849 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 1.0 MiB/s wr, 21 op/s 2024-04-23T14:58:57.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:56 smithi028 ceph-mon[27810]: pgmap v466: 97 pgs: 97 active+clean; 849 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 1.0 MiB/s wr, 21 op/s 2024-04-23T14:58:57.510 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T14:58:58.985 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:58:58 smithi105 ceph-mon[32555]: pgmap v467: 97 pgs: 97 active+clean; 849 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 1.0 MiB/s wr, 23 op/s 2024-04-23T14:58:59.273 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-23T14:58:59.273 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T14:58:59.273 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 295, 2024-04-23T14:58:59.273 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 283, 2024-04-23T14:58:59.274 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 1748864, 2024-04-23T14:58:59.274 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 174784, 2024-04-23T14:58:59.274 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 4, 2024-04-23T14:58:59.274 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 0, 2024-04-23T14:58:59.274 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.054872406, 2024-04-23T14:58:59.274 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.016303418, 2024-04-23T14:58:59.274 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.041555629, 2024-04-23T14:58:59.274 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.006413334, 2024-04-23T14:58:59.275 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:58:59.275 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 282, 2024-04-23T14:58:59.275 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.015116862, 2024-04-23T14:58:59.275 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000053605 2024-04-23T14:58:59.275 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:58:59.275 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:58:59.275 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-23T14:58:59.275 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-23T14:58:59.275 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-23T14:58:59.276 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:58:59.276 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 1748864, 2024-04-23T14:58:59.276 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 174784 2024-04-23T14:58:59.276 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:58:59.276 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T14:58:59.276 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 2, 2024-04-23T14:58:59.276 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 217, 2024-04-23T14:58:59.276 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 320, 2024-04-23T14:58:59.276 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 194368, 2024-04-23T14:58:59.276 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 5, 2024-04-23T14:58:59.277 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 3, 2024-04-23T14:58:59.277 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.012799878, 2024-04-23T14:58:59.277 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.009121411, 2024-04-23T14:58:59.277 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.001153882, 2024-04-23T14:58:59.277 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T14:58:59.277 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:58:59.277 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 214, 2024-04-23T14:58:59.277 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.023114390, 2024-04-23T14:58:59.277 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000108011 2024-04-23T14:58:59.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:58:59.278 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:58:59.278 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-23T14:58:59.278 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-23T14:58:59.278 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-23T14:58:59.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:58:59.278 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-23T14:58:59.278 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 194368 2024-04-23T14:58:59.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:58:59.279 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T14:58:59.279 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 4, 2024-04-23T14:58:59.279 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 4, 2024-04-23T14:58:59.279 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 2992, 2024-04-23T14:58:59.279 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 448, 2024-04-23T14:58:59.279 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 2, 2024-04-23T14:58:59.279 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 1, 2024-04-23T14:58:59.279 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.000653681, 2024-04-23T14:58:59.279 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.000033947, 2024-04-23T14:58:59.280 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.000600168, 2024-04-23T14:58:59.280 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T14:58:59.280 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:58:59.280 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 3, 2024-04-23T14:58:59.280 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000688857, 2024-04-23T14:58:59.280 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000229619 2024-04-23T14:58:59.280 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:58:59.280 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:58:59.280 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-23T14:58:59.281 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-23T14:58:59.281 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-23T14:58:59.281 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:58:59.281 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-23T14:58:59.281 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 448 2024-04-23T14:58:59.281 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:58:59.281 INFO:teuthology.orchestra.run.smithi105.stdout: "cct": { 2024-04-23T14:58:59.281 INFO:teuthology.orchestra.run.smithi105.stdout: "total_workers": 0, 2024-04-23T14:58:59.281 INFO:teuthology.orchestra.run.smithi105.stdout: "unhealthy_workers": 0 2024-04-23T14:58:59.281 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:58:59.282 INFO:teuthology.orchestra.run.smithi105.stdout: "mempool": { 2024-04-23T14:58:59.282 INFO:teuthology.orchestra.run.smithi105.stdout: "bloom_filter_bytes": 0, 2024-04-23T14:58:59.282 INFO:teuthology.orchestra.run.smithi105.stdout: "bloom_filter_items": 0, 2024-04-23T14:58:59.282 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T14:58:59.282 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_alloc_items": 0, 2024-04-23T14:58:59.282 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T14:58:59.282 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_data_items": 0, 2024-04-23T14:58:59.282 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T14:58:59.282 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T14:58:59.283 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T14:58:59.283 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T14:58:59.283 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T14:58:59.283 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_other_items": 0, 2024-04-23T14:58:59.283 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T14:58:59.283 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T14:58:59.283 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_bytes": 0, 2024-04-23T14:58:59.283 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_items": 0, 2024-04-23T14:58:59.284 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_blob_bytes": 0, 2024-04-23T14:58:59.284 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_blob_items": 0, 2024-04-23T14:58:59.284 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T14:58:59.284 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T14:58:59.284 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T14:58:59.284 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T14:58:59.284 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T14:58:59.284 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_items": 0, 2024-04-23T14:58:59.284 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_txc_bytes": 0, 2024-04-23T14:58:59.284 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_txc_items": 0, 2024-04-23T14:58:59.285 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T14:58:59.285 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T14:58:59.285 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_bytes": 0, 2024-04-23T14:58:59.285 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_items": 0, 2024-04-23T14:58:59.285 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_bytes": 0, 2024-04-23T14:58:59.285 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_items": 0, 2024-04-23T14:58:59.285 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T14:58:59.285 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_reader_items": 0, 2024-04-23T14:58:59.285 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T14:58:59.286 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_writer_items": 0, 2024-04-23T14:58:59.286 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_anon_bytes": 148955, 2024-04-23T14:58:59.286 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_anon_items": 30, 2024-04-23T14:58:59.286 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_meta_bytes": 0, 2024-04-23T14:58:59.286 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_meta_items": 0, 2024-04-23T14:58:59.286 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_bytes": 0, 2024-04-23T14:58:59.286 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_items": 0, 2024-04-23T14:58:59.286 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mapbl_bytes": 0, 2024-04-23T14:58:59.286 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mapbl_items": 0, 2024-04-23T14:58:59.286 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pglog_bytes": 0, 2024-04-23T14:58:59.287 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pglog_items": 0, 2024-04-23T14:58:59.287 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_bytes": 0, 2024-04-23T14:58:59.287 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_items": 0, 2024-04-23T14:58:59.287 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T14:58:59.287 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_mapping_items": 0, 2024-04-23T14:58:59.287 INFO:teuthology.orchestra.run.smithi105.stdout: "pgmap_bytes": 0, 2024-04-23T14:58:59.287 INFO:teuthology.orchestra.run.smithi105.stdout: "pgmap_items": 0, 2024-04-23T14:58:59.287 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_co_bytes": 928, 2024-04-23T14:58:59.287 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_co_items": 2, 2024-04-23T14:58:59.287 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_1_bytes": 0, 2024-04-23T14:58:59.288 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_1_items": 0, 2024-04-23T14:58:59.288 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_2_bytes": 0, 2024-04-23T14:58:59.288 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_2_items": 0 2024-04-23T14:58:59.288 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:58:59.288 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T14:58:59.288 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-04-23T14:58:59.288 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 104857600, 2024-04-23T14:58:59.288 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-04-23T14:58:59.288 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 297, 2024-04-23T14:58:59.289 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 1707226, 2024-04-23T14:58:59.289 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-04-23T14:58:59.289 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 297, 2024-04-23T14:58:59.289 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 0, 2024-04-23T14:58:59.289 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 0, 2024-04-23T14:58:59.289 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 297, 2024-04-23T14:58:59.289 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 1707226, 2024-04-23T14:58:59.289 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-04-23T14:58:59.289 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-23T14:58:59.289 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-23T14:58:59.290 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-23T14:58:59.290 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-23T14:58:59.290 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-23T14:58:59.290 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-23T14:58:59.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:58:58 smithi038 ceph-mon[32348]: pgmap v467: 97 pgs: 97 active+clean; 849 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 1.0 MiB/s wr, 23 op/s 2024-04-23T14:58:59.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:58:58 smithi028 ceph-mon[27810]: pgmap v467: 97 pgs: 97 active+clean; 849 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 1.0 MiB/s wr, 23 op/s 2024-04-23T14:58:59.970 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.f 2024-04-23T14:58:59.971 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.f 2024-04-23T14:58:59.971 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.f 2024-04-23T14:58:59.971 DEBUG:tasks.check_counter:Getting stats from i 2024-04-23T14:58:59.971 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-23T14:59:00.212 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T14:59:01.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:00 smithi105 ceph-mon[32555]: pgmap v468: 97 pgs: 97 active+clean; 849 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 809 KiB/s wr, 16 op/s 2024-04-23T14:59:01.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:00 smithi038 ceph-mon[32348]: pgmap v468: 97 pgs: 97 active+clean; 849 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 809 KiB/s wr, 16 op/s 2024-04-23T14:59:01.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:00 smithi028 ceph-mon[27810]: pgmap v468: 97 pgs: 97 active+clean; 849 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 809 KiB/s wr, 16 op/s 2024-04-23T14:59:01.902 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-23T14:59:01.902 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T14:59:01.902 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 294, 2024-04-23T14:59:01.902 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 282, 2024-04-23T14:59:01.902 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 1748592, 2024-04-23T14:59:01.903 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 174144, 2024-04-23T14:59:01.903 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 4, 2024-04-23T14:59:01.903 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 0, 2024-04-23T14:59:01.903 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.050707471, 2024-04-23T14:59:01.903 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.015172857, 2024-04-23T14:59:01.903 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.037081800, 2024-04-23T14:59:01.903 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.006359473, 2024-04-23T14:59:01.903 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:59:01.903 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 281, 2024-04-23T14:59:01.904 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.013290261, 2024-04-23T14:59:01.904 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000047296 2024-04-23T14:59:01.904 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:59:01.904 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:59:01.904 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-23T14:59:01.904 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-23T14:59:01.904 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-23T14:59:01.904 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:59:01.904 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 1748592, 2024-04-23T14:59:01.905 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 174144 2024-04-23T14:59:01.905 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:59:01.905 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T14:59:01.905 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 6, 2024-04-23T14:59:01.905 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 217, 2024-04-23T14:59:01.905 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 3312, 2024-04-23T14:59:01.905 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 194464, 2024-04-23T14:59:01.905 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 5, 2024-04-23T14:59:01.905 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 3, 2024-04-23T14:59:01.906 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.012178519, 2024-04-23T14:59:01.906 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.008146177, 2024-04-23T14:59:01.906 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.001503105, 2024-04-23T14:59:01.906 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T14:59:01.906 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:59:01.906 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 215, 2024-04-23T14:59:01.906 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.025968391, 2024-04-23T14:59:01.906 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000120783 2024-04-23T14:59:01.906 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:59:01.907 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:59:01.907 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-23T14:59:01.907 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-23T14:59:01.907 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-23T14:59:01.907 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:59:01.907 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-23T14:59:01.907 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 194464 2024-04-23T14:59:01.907 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:59:01.907 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T14:59:01.908 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 0, 2024-04-23T14:59:01.908 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 2, 2024-04-23T14:59:01.908 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 0, 2024-04-23T14:59:01.908 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 0, 2024-04-23T14:59:01.908 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 2, 2024-04-23T14:59:01.908 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 1, 2024-04-23T14:59:01.908 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.000206594, 2024-04-23T14:59:01.908 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.000000000, 2024-04-23T14:59:01.908 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.000131443, 2024-04-23T14:59:01.908 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T14:59:01.909 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:59:01.909 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-23T14:59:01.909 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-23T14:59:01.909 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-23T14:59:01.909 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:59:01.909 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:59:01.909 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-23T14:59:01.909 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-23T14:59:01.909 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-23T14:59:01.910 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:59:01.910 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-23T14:59:01.910 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 0 2024-04-23T14:59:01.910 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:59:01.910 INFO:teuthology.orchestra.run.smithi105.stdout: "cct": { 2024-04-23T14:59:01.910 INFO:teuthology.orchestra.run.smithi105.stdout: "total_workers": 0, 2024-04-23T14:59:01.910 INFO:teuthology.orchestra.run.smithi105.stdout: "unhealthy_workers": 0 2024-04-23T14:59:01.910 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:59:01.911 INFO:teuthology.orchestra.run.smithi105.stdout: "mempool": { 2024-04-23T14:59:01.911 INFO:teuthology.orchestra.run.smithi105.stdout: "bloom_filter_bytes": 0, 2024-04-23T14:59:01.911 INFO:teuthology.orchestra.run.smithi105.stdout: "bloom_filter_items": 0, 2024-04-23T14:59:01.911 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T14:59:01.911 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_alloc_items": 0, 2024-04-23T14:59:01.911 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T14:59:01.911 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_data_items": 0, 2024-04-23T14:59:01.911 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T14:59:01.912 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T14:59:01.912 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T14:59:01.912 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T14:59:01.912 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T14:59:01.912 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_other_items": 0, 2024-04-23T14:59:01.912 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T14:59:01.912 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T14:59:01.912 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_bytes": 0, 2024-04-23T14:59:01.912 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_items": 0, 2024-04-23T14:59:01.913 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_blob_bytes": 0, 2024-04-23T14:59:01.913 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_blob_items": 0, 2024-04-23T14:59:01.913 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T14:59:01.913 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T14:59:01.913 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T14:59:01.913 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T14:59:01.913 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T14:59:01.913 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_items": 0, 2024-04-23T14:59:01.913 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_txc_bytes": 0, 2024-04-23T14:59:01.914 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_txc_items": 0, 2024-04-23T14:59:01.914 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T14:59:01.914 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T14:59:01.914 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_bytes": 0, 2024-04-23T14:59:01.914 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_items": 0, 2024-04-23T14:59:01.914 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_bytes": 0, 2024-04-23T14:59:01.914 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_items": 0, 2024-04-23T14:59:01.914 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T14:59:01.915 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_reader_items": 0, 2024-04-23T14:59:01.915 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T14:59:01.915 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_writer_items": 0, 2024-04-23T14:59:01.915 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_anon_bytes": 148972, 2024-04-23T14:59:01.915 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_anon_items": 31, 2024-04-23T14:59:01.915 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_meta_bytes": 0, 2024-04-23T14:59:01.915 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_meta_items": 0, 2024-04-23T14:59:01.915 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_bytes": 0, 2024-04-23T14:59:01.916 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_items": 0, 2024-04-23T14:59:01.916 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mapbl_bytes": 0, 2024-04-23T14:59:01.916 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mapbl_items": 0, 2024-04-23T14:59:01.916 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pglog_bytes": 0, 2024-04-23T14:59:01.916 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pglog_items": 0, 2024-04-23T14:59:01.916 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_bytes": 0, 2024-04-23T14:59:01.916 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_items": 0, 2024-04-23T14:59:01.916 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T14:59:01.916 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_mapping_items": 0, 2024-04-23T14:59:01.917 INFO:teuthology.orchestra.run.smithi105.stdout: "pgmap_bytes": 0, 2024-04-23T14:59:01.917 INFO:teuthology.orchestra.run.smithi105.stdout: "pgmap_items": 0, 2024-04-23T14:59:01.917 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_co_bytes": 928, 2024-04-23T14:59:01.917 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_co_items": 2, 2024-04-23T14:59:01.917 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_1_bytes": 0, 2024-04-23T14:59:01.917 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_1_items": 0, 2024-04-23T14:59:01.917 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_2_bytes": 0, 2024-04-23T14:59:01.917 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_2_items": 0 2024-04-23T14:59:01.917 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:59:01.918 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T14:59:01.918 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-04-23T14:59:01.918 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 104857600, 2024-04-23T14:59:01.918 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-04-23T14:59:01.918 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 296, 2024-04-23T14:59:01.918 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 1707096, 2024-04-23T14:59:01.918 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-04-23T14:59:01.918 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 296, 2024-04-23T14:59:01.919 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 0, 2024-04-23T14:59:01.919 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 0, 2024-04-23T14:59:01.919 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 296, 2024-04-23T14:59:01.919 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 1707096, 2024-04-23T14:59:01.919 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-04-23T14:59:01.919 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-23T14:59:01.919 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-23T14:59:01.919 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-23T14:59:01.919 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-23T14:59:01.920 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-23T14:59:01.920 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-23T14:59:02.749 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.i 2024-04-23T14:59:02.749 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.i 2024-04-23T14:59:02.749 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.i 2024-04-23T14:59:02.749 DEBUG:tasks.check_counter:Getting stats from l 2024-04-23T14:59:02.750 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0 shell --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-23T14:59:02.992 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T14:59:03.036 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:02 smithi105 ceph-mon[32555]: pgmap v469: 97 pgs: 97 active+clean; 849 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 809 KiB/s wr, 16 op/s 2024-04-23T14:59:03.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:02 smithi038 ceph-mon[32348]: pgmap v469: 97 pgs: 97 active+clean; 849 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 809 KiB/s wr, 16 op/s 2024-04-23T14:59:03.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:02 smithi028 ceph-mon[27810]: pgmap v469: 97 pgs: 97 active+clean; 849 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 809 KiB/s wr, 16 op/s 2024-04-23T14:59:04.949 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-23T14:59:04.950 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T14:59:04.950 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 0, 2024-04-23T14:59:04.950 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 2, 2024-04-23T14:59:04.950 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 0, 2024-04-23T14:59:04.950 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 0, 2024-04-23T14:59:04.950 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 4, 2024-04-23T14:59:04.950 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 0, 2024-04-23T14:59:04.951 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.000975239, 2024-04-23T14:59:04.951 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.000000000, 2024-04-23T14:59:04.951 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.000208407, 2024-04-23T14:59:04.951 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T14:59:04.951 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:59:04.951 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-23T14:59:04.951 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-23T14:59:04.951 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-23T14:59:04.952 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:59:04.952 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:59:04.952 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-23T14:59:04.952 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-23T14:59:04.952 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-23T14:59:04.952 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:59:04.952 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-23T14:59:04.952 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 0 2024-04-23T14:59:04.952 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:59:04.952 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T14:59:04.953 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 2, 2024-04-23T14:59:04.953 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 221, 2024-04-23T14:59:04.953 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 320, 2024-04-23T14:59:04.953 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 195072, 2024-04-23T14:59:04.953 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 5, 2024-04-23T14:59:04.953 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 3, 2024-04-23T14:59:04.953 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.012228811, 2024-04-23T14:59:04.953 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.008390237, 2024-04-23T14:59:04.953 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.001279209, 2024-04-23T14:59:04.954 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T14:59:04.954 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:59:04.954 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 218, 2024-04-23T14:59:04.954 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.024909264, 2024-04-23T14:59:04.954 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000114262 2024-04-23T14:59:04.954 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:59:04.954 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:59:04.954 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-23T14:59:04.954 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-23T14:59:04.955 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-23T14:59:04.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:59:04.955 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-23T14:59:04.955 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 195072 2024-04-23T14:59:04.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:59:04.955 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T14:59:04.955 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 306, 2024-04-23T14:59:04.955 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 292, 2024-04-23T14:59:04.955 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 1754384, 2024-04-23T14:59:04.955 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 179408, 2024-04-23T14:59:04.956 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 2, 2024-04-23T14:59:04.956 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 1, 2024-04-23T14:59:04.956 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.054648820, 2024-04-23T14:59:04.956 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.016304750, 2024-04-23T14:59:04.956 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.042004044, 2024-04-23T14:59:04.956 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.006486331, 2024-04-23T14:59:04.956 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T14:59:04.956 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 292, 2024-04-23T14:59:04.956 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.014535210, 2024-04-23T14:59:04.956 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000049778 2024-04-23T14:59:04.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:59:04.957 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-04-23T14:59:04.957 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-23T14:59:04.957 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-23T14:59:04.957 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-23T14:59:04.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:59:04.957 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 1754384, 2024-04-23T14:59:04.957 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 179408 2024-04-23T14:59:04.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:59:04.957 INFO:teuthology.orchestra.run.smithi105.stdout: "cct": { 2024-04-23T14:59:04.958 INFO:teuthology.orchestra.run.smithi105.stdout: "total_workers": 0, 2024-04-23T14:59:04.958 INFO:teuthology.orchestra.run.smithi105.stdout: "unhealthy_workers": 0 2024-04-23T14:59:04.958 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:59:04.958 INFO:teuthology.orchestra.run.smithi105.stdout: "mempool": { 2024-04-23T14:59:04.958 INFO:teuthology.orchestra.run.smithi105.stdout: "bloom_filter_bytes": 0, 2024-04-23T14:59:04.958 INFO:teuthology.orchestra.run.smithi105.stdout: "bloom_filter_items": 0, 2024-04-23T14:59:04.958 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T14:59:04.958 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_alloc_items": 0, 2024-04-23T14:59:04.959 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T14:59:04.959 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_data_items": 0, 2024-04-23T14:59:04.959 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T14:59:04.959 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T14:59:04.959 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T14:59:04.959 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T14:59:04.959 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T14:59:04.959 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_other_items": 0, 2024-04-23T14:59:04.959 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T14:59:04.959 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T14:59:04.960 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_bytes": 0, 2024-04-23T14:59:04.960 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_items": 0, 2024-04-23T14:59:04.960 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_blob_bytes": 0, 2024-04-23T14:59:04.960 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_blob_items": 0, 2024-04-23T14:59:04.960 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T14:59:04.960 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T14:59:04.960 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T14:59:04.960 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T14:59:04.960 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T14:59:04.961 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_items": 0, 2024-04-23T14:59:04.961 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_txc_bytes": 0, 2024-04-23T14:59:04.961 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_txc_items": 0, 2024-04-23T14:59:04.961 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T14:59:04.961 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T14:59:04.961 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_bytes": 0, 2024-04-23T14:59:04.961 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_items": 0, 2024-04-23T14:59:04.961 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_bytes": 0, 2024-04-23T14:59:04.961 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_items": 0, 2024-04-23T14:59:04.961 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T14:59:04.962 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_reader_items": 0, 2024-04-23T14:59:04.962 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T14:59:04.962 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_writer_items": 0, 2024-04-23T14:59:04.962 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_anon_bytes": 148709, 2024-04-23T14:59:04.962 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_anon_items": 31, 2024-04-23T14:59:04.962 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_meta_bytes": 0, 2024-04-23T14:59:04.962 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_meta_items": 0, 2024-04-23T14:59:04.962 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_bytes": 0, 2024-04-23T14:59:04.962 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_items": 0, 2024-04-23T14:59:04.963 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mapbl_bytes": 0, 2024-04-23T14:59:04.963 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mapbl_items": 0, 2024-04-23T14:59:04.963 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pglog_bytes": 0, 2024-04-23T14:59:04.963 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pglog_items": 0, 2024-04-23T14:59:04.963 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_bytes": 0, 2024-04-23T14:59:04.963 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_items": 0, 2024-04-23T14:59:04.963 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T14:59:04.963 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_mapping_items": 0, 2024-04-23T14:59:04.963 INFO:teuthology.orchestra.run.smithi105.stdout: "pgmap_bytes": 0, 2024-04-23T14:59:04.964 INFO:teuthology.orchestra.run.smithi105.stdout: "pgmap_items": 0, 2024-04-23T14:59:04.964 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_co_bytes": 928, 2024-04-23T14:59:04.964 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_co_items": 2, 2024-04-23T14:59:04.964 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_1_bytes": 0, 2024-04-23T14:59:04.964 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_1_items": 0, 2024-04-23T14:59:04.964 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_2_bytes": 0, 2024-04-23T14:59:04.964 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_2_items": 0 2024-04-23T14:59:04.964 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-23T14:59:04.964 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T14:59:04.964 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-04-23T14:59:04.965 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 104857600, 2024-04-23T14:59:04.965 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-04-23T14:59:04.965 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 304, 2024-04-23T14:59:04.965 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 1708773, 2024-04-23T14:59:04.965 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-04-23T14:59:04.965 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 304, 2024-04-23T14:59:04.965 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 0, 2024-04-23T14:59:04.965 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 0, 2024-04-23T14:59:04.965 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 304, 2024-04-23T14:59:04.965 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 1708773, 2024-04-23T14:59:04.966 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-04-23T14:59:04.966 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-23T14:59:04.966 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-23T14:59:04.966 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-23T14:59:04.966 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-23T14:59:04.966 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-23T14:59:04.966 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-23T14:59:05.245 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:04 smithi105 ceph-mon[32555]: pgmap v470: 97 pgs: 97 active+clean; 849 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 866 KiB/s wr, 35 op/s 2024-04-23T14:59:05.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:04 smithi038 ceph-mon[32348]: pgmap v470: 97 pgs: 97 active+clean; 849 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 866 KiB/s wr, 35 op/s 2024-04-23T14:59:05.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:04 smithi028 ceph-mon[27810]: pgmap v470: 97 pgs: 97 active+clean; 849 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 866 KiB/s wr, 35 op/s 2024-04-23T14:59:05.659 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-04-23T14:59:05.659 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-04-23T14:59:05.659 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-04-23T14:59:05.660 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-23T14:59:05.670 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-23T14:59:05.671 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-23T14:59:05.671 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-23T14:59:05.735 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-23T14:59:05.735 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-23T14:59:05.792 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:59:06.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.015+0000 7fe927200640 1 -- 172.21.15.28:0/3851028754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9281071b0 msgr2=0x7fe9280696d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:06.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.015+0000 7fe927200640 1 --2- 172.21.15.28:0/3851028754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9281071b0 0x7fe9280696d0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fe910009860 tx=0x7fe91002a340 comp rx=0 tx=0).stop 2024-04-23T14:59:06.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.015+0000 7fe927200640 1 -- 172.21.15.28:0/3851028754 shutdown_connections 2024-04-23T14:59:06.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.015+0000 7fe927200640 1 --2- 172.21.15.28:0/3851028754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9281071b0 0x7fe9280696d0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:06.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.015+0000 7fe927200640 1 --2- 172.21.15.28:0/3851028754 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe928104890 0x7fe928104d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:06.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.015+0000 7fe927200640 1 --2- 172.21.15.28:0/3851028754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe928106d40 0x7fe928069190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:06.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.015+0000 7fe927200640 1 -- 172.21.15.28:0/3851028754 >> 172.21.15.28:0/3851028754 conn(0x7fe928076840 msgr2=0x7fe928076c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:59:06.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.016+0000 7fe927200640 1 -- 172.21.15.28:0/3851028754 shutdown_connections 2024-04-23T14:59:06.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.016+0000 7fe927200640 1 -- 172.21.15.28:0/3851028754 wait complete. 2024-04-23T14:59:06.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.017+0000 7fe927200640 1 Processor -- start 2024-04-23T14:59:06.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.017+0000 7fe927200640 1 -- start start 2024-04-23T14:59:06.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.017+0000 7fe927200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe928104890 0x7fe9281a48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:06.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.017+0000 7fe927200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe928106d40 0x7fe9281a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:06.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.017+0000 7fe927200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9281071b0 0x7fe92819ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:06.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.017+0000 7fe927200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe928118730 con 0x7fe928104890 2024-04-23T14:59:06.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.017+0000 7fe927200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe9281185b0 con 0x7fe928106d40 2024-04-23T14:59:06.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.017+0000 7fe927200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe9281188b0 con 0x7fe9281071b0 2024-04-23T14:59:06.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.017+0000 7fe925e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe928104890 0x7fe9281a48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:06.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.017+0000 7fe925e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe928104890 0x7fe9281a48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51694/0 (socket says 172.21.15.28:51694) 2024-04-23T14:59:06.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.017+0000 7fe925e00640 1 -- 172.21.15.28:0/526970855 learned_addr learned my addr 172.21.15.28:0/526970855 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:59:06.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.018+0000 7fe925400640 1 --2- 172.21.15.28:0/526970855 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe928106d40 0x7fe9281a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:06.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.018+0000 7fe926800640 1 --2- 172.21.15.28:0/526970855 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9281071b0 0x7fe92819ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:06.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.018+0000 7fe925e00640 1 -- 172.21.15.28:0/526970855 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9281071b0 msgr2=0x7fe92819ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:06.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.018+0000 7fe925e00640 1 --2- 172.21.15.28:0/526970855 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9281071b0 0x7fe92819ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:06.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.018+0000 7fe925e00640 1 -- 172.21.15.28:0/526970855 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe928106d40 msgr2=0x7fe9281a4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:06.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.018+0000 7fe925e00640 1 --2- 172.21.15.28:0/526970855 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe928106d40 0x7fe9281a4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:06.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.018+0000 7fe925e00640 1 -- 172.21.15.28:0/526970855 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe92819f1d0 con 0x7fe928104890 2024-04-23T14:59:06.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.018+0000 7fe925400640 1 --2- 172.21.15.28:0/526970855 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe928106d40 0x7fe9281a4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:59:06.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.018+0000 7fe925e00640 1 --2- 172.21.15.28:0/526970855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe928104890 0x7fe9281a48a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe91800efb0 tx=0x7fe91800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:59:06.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.019+0000 7fe91ea00640 1 -- 172.21.15.28:0/526970855 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe91801b070 con 0x7fe928104890 2024-04-23T14:59:06.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.019+0000 7fe91ea00640 1 -- 172.21.15.28:0/526970855 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe91800ba50 con 0x7fe928104890 2024-04-23T14:59:06.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.019+0000 7fe91ea00640 1 -- 172.21.15.28:0/526970855 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe918016c00 con 0x7fe928104890 2024-04-23T14:59:06.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.019+0000 7fe927200640 1 -- 172.21.15.28:0/526970855 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe92819f4c0 con 0x7fe928104890 2024-04-23T14:59:06.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.019+0000 7fe927200640 1 -- 172.21.15.28:0/526970855 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9281ab760 con 0x7fe928104890 2024-04-23T14:59:06.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.020+0000 7fe927200640 1 -- 172.21.15.28:0/526970855 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe928068d40 con 0x7fe928104890 2024-04-23T14:59:06.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.022+0000 7fe91ea00640 1 -- 172.21.15.28:0/526970855 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fe91801e070 con 0x7fe928104890 2024-04-23T14:59:06.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.023+0000 7fe91ea00640 1 --2- 172.21.15.28:0/526970855 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe90c0a3d20 0x7fe90c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:06.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.023+0000 7fe91ea00640 1 -- 172.21.15.28:0/526970855 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8542+0+0 (secure 0 0 0) 0x7fe9180d0720 con 0x7fe928104890 2024-04-23T14:59:06.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.024+0000 7fe925400640 1 --2- 172.21.15.28:0/526970855 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe90c0a3d20 0x7fe90c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:06.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.024+0000 7fe925400640 1 --2- 172.21.15.28:0/526970855 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe90c0a3d20 0x7fe90c0a61e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe91002a930 tx=0x7fe9100056d0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:59:06.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.028+0000 7fe91ea00640 1 -- 172.21.15.28:0/526970855 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe91809c700 con 0x7fe928104890 2024-04-23T14:59:06.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.240+0000 7fe927200640 1 -- 172.21.15.28:0/526970855 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe92819ff80 con 0x7fe928104890 2024-04-23T14:59:06.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.243+0000 7fe91ea00640 1 -- 172.21.15.28:0/526970855 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) ==== 74+0+23120 (secure 0 0 0) 0x7fe9180a0a00 con 0x7fe928104890 2024-04-23T14:59:06.244 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:59:06.244 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":104,"fsid":"1ee6e996-017e-11ef-bc93-c7b262605968","created":"2024-04-23T14:32:34.581082+0000","modified":"2024-04-23T14:58:11.777123+0000","last_up_change":"2024-04-23T14:40:43.375846+0000","last_in_change":"2024-04-23T14:40:23.576939+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:36:27.698112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:41:37.030341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:41:38.444742+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":23,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e0b427bc-be65-4543-99cc-2f53ad98ddad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6803","nonce":2993445336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6805","nonce":2993445336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6809","nonce":2993445336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6807","nonce":2993445336}]},"public_addr":"172.21.15.28:6803/2993445336","cluster_addr":"172.21.15.28:6805/2993445336","heartbeat_back_addr":"172.21.15.28:6809/2993445336","heartbeat_front_addr":"172.21.15.28:6807/2993445336","state":["exists","up"]},{"osd":1,"uuid":"ef527ec9-1bf3-4bb0-8df3-f5a015af557c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6801","nonce":4024093383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6803","nonce":4024093383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6807","nonce":4024093383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6805","nonce":4024093383}]},"public_addr":"172.21.15.38:6801/4024093383","cluster_addr":"172.21.15.38:6803/4024093383","heartbeat_back_addr":"172.21.15.38:6807/4024093383","heartbeat_front_addr":"172.21.15.38:6805/4024093383","state":["exists","up"]},{"osd":2,"uuid":"b33b23d9-adce-4c11-b304-2e117bdf6bfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6801","nonce":570739029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6803","nonce":570739029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6807","nonce":570739029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6805","nonce":570739029}]},"public_addr":"172.21.15.105:6801/570739029","cluster_addr":"172.21.15.105:6803/570739029","heartbeat_back_addr":"172.21.15.105:6807/570739029","heartbeat_front_addr":"172.21.15.105:6805/570739029","state":["exists","up"]},{"osd":3,"uuid":"28bd5608-b426-434e-9db1-bdaa9828bdb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6811","nonce":2480400247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6813","nonce":2480400247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6817","nonce":2480400247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6815","nonce":2480400247}]},"public_addr":"172.21.15.28:6811/2480400247","cluster_addr":"172.21.15.28:6813/2480400247","heartbeat_back_addr":"172.21.15.28:6817/2480400247","heartbeat_front_addr":"172.21.15.28:6815/2480400247","state":["exists","up"]},{"osd":4,"uuid":"e5d1c89b-025f-4d57-8807-59d75c787b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6809","nonce":3772189246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6811","nonce":3772189246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6815","nonce":3772189246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6813","nonce":3772189246}]},"public_addr":"172.21.15.38:6809/3772189246","cluster_addr":"172.21.15.38:6811/3772189246","heartbeat_back_addr":"172.21.15.38:6815/3772189246","heartbeat_front_addr":"172.21.15.38:6813/3772189246","state":["exists","up"]},{"osd":5,"uuid":"9cf9cf4c-aafd-4ed3-bfb1-0527710a0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6809","nonce":928755894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6811","nonce":928755894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6815","nonce":928755894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6813","nonce":928755894}]},"public_addr":"172.21.15.105:6809/928755894","cluster_addr":"172.21.15.105:6811/928755894","heartbeat_back_addr":"172.21.15.105:6815/928755894","heartbeat_front_addr":"172.21.15.105:6813/928755894","state":["exists","up"]},{"osd":6,"uuid":"304db525-d24d-4687-bfa0-f92bdf66d1e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6819","nonce":897979441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6821","nonce":897979441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6825","nonce":897979441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6823","nonce":897979441}]},"public_addr":"172.21.15.28:6819/897979441","cluster_addr":"172.21.15.28:6821/897979441","heartbeat_back_addr":"172.21.15.28:6825/897979441","heartbeat_front_addr":"172.21.15.28:6823/897979441","state":["exists","up"]},{"osd":7,"uuid":"08e01d59-03f3-47b1-b45f-35ec5a50992c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6817","nonce":2400251786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6819","nonce":2400251786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6823","nonce":2400251786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6821","nonce":2400251786}]},"public_addr":"172.21.15.38:6817/2400251786","cluster_addr":"172.21.15.38:6819/2400251786","heartbeat_back_addr":"172.21.15.38:6823/2400251786","heartbeat_front_addr":"172.21.15.38:6821/2400251786","state":["exists","up"]},{"osd":8,"uuid":"64ee1e95-0756-48b1-9c00-472d0c3cbd29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6817","nonce":1682312269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6819","nonce":1682312269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6823","nonce":1682312269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6821","nonce":1682312269}]},"public_addr":"172.21.15.105:6817/1682312269","cluster_addr":"172.21.15.105:6819/1682312269","heartbeat_back_addr":"172.21.15.105:6823/1682312269","heartbeat_front_addr":"172.21.15.105:6821/1682312269","state":["exists","up"]},{"osd":9,"uuid":"18f60893-3ffd-48c2-9db5-a67d35611d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6827","nonce":1292942516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6829","nonce":1292942516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6833","nonce":1292942516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6831","nonce":1292942516}]},"public_addr":"172.21.15.28:6827/1292942516","cluster_addr":"172.21.15.28:6829/1292942516","heartbeat_back_addr":"172.21.15.28:6833/1292942516","heartbeat_front_addr":"172.21.15.28:6831/1292942516","state":["exists","up"]},{"osd":10,"uuid":"76c06725-7040-4118-8894-fe808a87f999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6825","nonce":2444616468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6827","nonce":2444616468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6831","nonce":2444616468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6829","nonce":2444616468}]},"public_addr":"172.21.15.38:6825/2444616468","cluster_addr":"172.21.15.38:6827/2444616468","heartbeat_back_addr":"172.21.15.38:6831/2444616468","heartbeat_front_addr":"172.21.15.38:6829/2444616468","state":["exists","up"]},{"osd":11,"uuid":"e8fc9454-5540-4e2e-91bf-3ba251dbd1a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6825","nonce":3400299562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6827","nonce":3400299562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6831","nonce":3400299562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6829","nonce":3400299562}]},"public_addr":"172.21.15.105:6825/3400299562","cluster_addr":"172.21.15.105:6827/3400299562","heartbeat_back_addr":"172.21.15.105:6831/3400299562","heartbeat_front_addr":"172.21.15.105:6829/3400299562","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:35:33.615395+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:00.545293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:24.940211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:57.684887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:24.519061+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:49.010804+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:22.638478+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:49.559104+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:14.011771+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:47.859950+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:16.298445+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:41.619069+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.2d","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/38494154":"2024-04-24T14:44:01.286264+0000","172.21.15.28:0/905362079":"2024-04-24T14:44:01.286264+0000","172.21.15.28:6801/1997136173":"2024-04-24T14:44:01.286264+0000","172.21.15.28:6800/1997136173":"2024-04-24T14:44:01.286264+0000","172.21.15.28:0/590495740":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/1163172512":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/4290328460":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/4145020172":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/1997097543":"2024-04-24T14:44:01.286264+0000","172.21.15.28:0/3847772687":"2024-04-24T14:44:01.286264+0000","172.21.15.28:6800/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/720626958":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/948450262":"2024-04-24T14:33:07.576363+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":10,"length":1},{"begin":23,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:59:06.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.246+0000 7fe927200640 1 -- 172.21.15.28:0/526970855 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe90c0a3d20 msgr2=0x7fe90c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:06.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.246+0000 7fe927200640 1 --2- 172.21.15.28:0/526970855 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe90c0a3d20 0x7fe90c0a61e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe91002a930 tx=0x7fe9100056d0 comp rx=0 tx=0).stop 2024-04-23T14:59:06.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.246+0000 7fe927200640 1 -- 172.21.15.28:0/526970855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe928104890 msgr2=0x7fe9281a48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:06.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.246+0000 7fe927200640 1 --2- 172.21.15.28:0/526970855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe928104890 0x7fe9281a48a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe91800efb0 tx=0x7fe91800c800 comp rx=0 tx=0).stop 2024-04-23T14:59:06.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.246+0000 7fe927200640 1 -- 172.21.15.28:0/526970855 shutdown_connections 2024-04-23T14:59:06.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.246+0000 7fe927200640 1 --2- 172.21.15.28:0/526970855 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fe90c0a3d20 0x7fe90c0a61e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:06.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.246+0000 7fe927200640 1 --2- 172.21.15.28:0/526970855 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe928106d40 0x7fe9281a4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:06.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.246+0000 7fe927200640 1 --2- 172.21.15.28:0/526970855 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9281071b0 0x7fe92819ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:06.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.246+0000 7fe927200640 1 --2- 172.21.15.28:0/526970855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe928104890 0x7fe9281a48a0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:06.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.246+0000 7fe927200640 1 -- 172.21.15.28:0/526970855 >> 172.21.15.28:0/526970855 conn(0x7fe928076840 msgr2=0x7fe928101d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:59:06.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.246+0000 7fe927200640 1 -- 172.21.15.28:0/526970855 shutdown_connections 2024-04-23T14:59:06.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.246+0000 7fe927200640 1 -- 172.21.15.28:0/526970855 wait complete. 2024-04-23T14:59:06.267 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T14:36:27.698112+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T14:41:37.030341+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T14:41:38.444742+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 23, 'snap_epoch': 103, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T14:59:06.268 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T14:59:06.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.476+0000 7f272e000640 1 -- 172.21.15.28:0/1496314316 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2728102ac0 msgr2=0x7f2728106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:06.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.476+0000 7f272e000640 1 --2- 172.21.15.28:0/1496314316 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2728102ac0 0x7f2728106b00 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f2710009900 tx=0x7f271002a3f0 comp rx=0 tx=0).stop 2024-04-23T14:59:06.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.476+0000 7f272e000640 1 -- 172.21.15.28:0/1496314316 shutdown_connections 2024-04-23T14:59:06.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.476+0000 7f272e000640 1 --2- 172.21.15.28:0/1496314316 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2728102ac0 0x7f2728106b00 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:06.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.476+0000 7f272e000640 1 --2- 172.21.15.28:0/1496314316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27281071e0 0x7f272810edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:06.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.476+0000 7f272e000640 1 --2- 172.21.15.28:0/1496314316 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2728102110 0x7f27281024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:06.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.476+0000 7f272e000640 1 -- 172.21.15.28:0/1496314316 >> 172.21.15.28:0/1496314316 conn(0x7f27280fd9d0 msgr2=0x7f27280ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:59:06.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.476+0000 7f272e000640 1 -- 172.21.15.28:0/1496314316 shutdown_connections 2024-04-23T14:59:06.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.476+0000 7f272e000640 1 -- 172.21.15.28:0/1496314316 wait complete. 2024-04-23T14:59:06.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.477+0000 7f272e000640 1 Processor -- start 2024-04-23T14:59:06.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.477+0000 7f272e000640 1 -- start start 2024-04-23T14:59:06.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.478+0000 7f272e000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2728102110 0x7f272819e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:06.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.478+0000 7f272e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27281071e0 0x7f272819eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:06.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.478+0000 7f272e000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2728198660 0x7f2728198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:06.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.478+0000 7f272e000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f27281122c0 con 0x7f27281071e0 2024-04-23T14:59:06.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.478+0000 7f272e000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2728112140 con 0x7f2728198660 2024-04-23T14:59:06.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.478+0000 7f272e000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2728112440 con 0x7f2728102110 2024-04-23T14:59:06.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.478+0000 7f2727e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27281071e0 0x7f272819eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:06.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.478+0000 7f2727e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27281071e0 0x7f272819eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51726/0 (socket says 172.21.15.28:51726) 2024-04-23T14:59:06.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.478+0000 7f2727e00640 1 -- 172.21.15.28:0/2770613702 learned_addr learned my addr 172.21.15.28:0/2770613702 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:59:06.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.478+0000 7f272cc00640 1 --2- 172.21.15.28:0/2770613702 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2728102110 0x7f272819e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:06.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.478+0000 7f272d600640 1 --2- 172.21.15.28:0/2770613702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2728198660 0x7f2728198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:06.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.479+0000 7f272cc00640 1 -- 172.21.15.28:0/2770613702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2728198660 msgr2=0x7f2728198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:06.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.479+0000 7f272cc00640 1 --2- 172.21.15.28:0/2770613702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2728198660 0x7f2728198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:06.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.479+0000 7f272cc00640 1 -- 172.21.15.28:0/2770613702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27281071e0 msgr2=0x7f272819eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:06.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.479+0000 7f272cc00640 1 --2- 172.21.15.28:0/2770613702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27281071e0 0x7f272819eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:06.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.479+0000 7f272cc00640 1 -- 172.21.15.28:0/2770613702 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2728199260 con 0x7f2728102110 2024-04-23T14:59:06.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.479+0000 7f272cc00640 1 --2- 172.21.15.28:0/2770613702 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2728102110 0x7f272819e5a0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2718009fd0 tx=0x7f271800ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:59:06.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.479+0000 7f2725600640 1 -- 172.21.15.28:0/2770613702 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2718004b20 con 0x7f2728102110 2024-04-23T14:59:06.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.479+0000 7f2725600640 1 -- 172.21.15.28:0/2770613702 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2718004cc0 con 0x7f2728102110 2024-04-23T14:59:06.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.479+0000 7f272e000640 1 -- 172.21.15.28:0/2770613702 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27281a5340 con 0x7f2728102110 2024-04-23T14:59:06.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.479+0000 7f2725600640 1 -- 172.21.15.28:0/2770613702 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27180107b0 con 0x7f2728102110 2024-04-23T14:59:06.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.479+0000 7f272e000640 1 -- 172.21.15.28:0/2770613702 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27281a56f0 con 0x7f2728102110 2024-04-23T14:59:06.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.480+0000 7f272e000640 1 -- 172.21.15.28:0/2770613702 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27281035e0 con 0x7f2728102110 2024-04-23T14:59:06.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.483+0000 7f2725600640 1 -- 172.21.15.28:0/2770613702 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f2718005020 con 0x7f2728102110 2024-04-23T14:59:06.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.484+0000 7f2725600640 1 --2- 172.21.15.28:0/2770613702 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f270c0a3d20 0x7f270c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:06.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.484+0000 7f2725600640 1 -- 172.21.15.28:0/2770613702 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8542+0+0 (secure 0 0 0) 0x7f27180d0f50 con 0x7f2728102110 2024-04-23T14:59:06.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.484+0000 7f2727e00640 1 --2- 172.21.15.28:0/2770613702 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f270c0a3d20 0x7f270c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:06.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.485+0000 7f2727e00640 1 --2- 172.21.15.28:0/2770613702 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f270c0a3d20 0x7f270c0a61e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2710005c40 tx=0x7f2710005b50 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:59:06.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.486+0000 7f2725600640 1 -- 172.21.15.28:0/2770613702 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f271809d580 con 0x7f2728102110 2024-04-23T14:59:06.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.695+0000 7f272e000640 1 -- 172.21.15.28:0/2770613702 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f272810a410 con 0x7f2728102110 2024-04-23T14:59:06.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.696+0000 7f2725600640 1 -- 172.21.15.28:0/2770613702 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v104) ==== 93+0+10 (secure 0 0 0) 0x7f271809ce80 con 0x7f2728102110 2024-04-23T14:59:06.696 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-23T14:59:06.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.698+0000 7f272e000640 1 -- 172.21.15.28:0/2770613702 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f270c0a3d20 msgr2=0x7f270c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:06.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.698+0000 7f272e000640 1 --2- 172.21.15.28:0/2770613702 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f270c0a3d20 0x7f270c0a61e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2710005c40 tx=0x7f2710005b50 comp rx=0 tx=0).stop 2024-04-23T14:59:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.698+0000 7f272e000640 1 -- 172.21.15.28:0/2770613702 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2728102110 msgr2=0x7f272819e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.698+0000 7f272e000640 1 --2- 172.21.15.28:0/2770613702 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2728102110 0x7f272819e5a0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2718009fd0 tx=0x7f271800ef90 comp rx=0 tx=0).stop 2024-04-23T14:59:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.698+0000 7f272e000640 1 -- 172.21.15.28:0/2770613702 shutdown_connections 2024-04-23T14:59:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.698+0000 7f272e000640 1 --2- 172.21.15.28:0/2770613702 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f270c0a3d20 0x7f270c0a61e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.698+0000 7f272e000640 1 --2- 172.21.15.28:0/2770613702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2728198660 0x7f2728198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.698+0000 7f272e000640 1 --2- 172.21.15.28:0/2770613702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27281071e0 0x7f272819eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.698+0000 7f272e000640 1 --2- 172.21.15.28:0/2770613702 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2728102110 0x7f272819e5a0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.698+0000 7f272e000640 1 -- 172.21.15.28:0/2770613702 >> 172.21.15.28:0/2770613702 conn(0x7f27280fd9d0 msgr2=0x7f27280fece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:59:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.699+0000 7f272e000640 1 -- 172.21.15.28:0/2770613702 shutdown_connections 2024-04-23T14:59:06.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.699+0000 7f272e000640 1 -- 172.21.15.28:0/2770613702 wait complete. 2024-04-23T14:59:06.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T14:59:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.932+0000 7f3200a00640 1 -- 172.21.15.28:0/4163551220 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31fc076530 msgr2=0x7f31fc076910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.932+0000 7f3200a00640 1 --2- 172.21.15.28:0/4163551220 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31fc076530 0x7f31fc076910 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f31ec009cd0 tx=0x7f31ec02a8e0 comp rx=0 tx=0).stop 2024-04-23T14:59:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.932+0000 7f3200a00640 1 -- 172.21.15.28:0/4163551220 shutdown_connections 2024-04-23T14:59:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.932+0000 7f3200a00640 1 --2- 172.21.15.28:0/4163551220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31fc112d40 0x7f31fc115130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.932+0000 7f3200a00640 1 --2- 172.21.15.28:0/4163551220 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31fc076e50 0x7f31fc112800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.932+0000 7f3200a00640 1 --2- 172.21.15.28:0/4163551220 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31fc076530 0x7f31fc076910 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:06.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.932+0000 7f3200a00640 1 -- 172.21.15.28:0/4163551220 >> 172.21.15.28:0/4163551220 conn(0x7f31fc0ffb70 msgr2=0x7f31fc101f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:59:06.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.932+0000 7f3200a00640 1 -- 172.21.15.28:0/4163551220 shutdown_connections 2024-04-23T14:59:06.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.932+0000 7f3200a00640 1 -- 172.21.15.28:0/4163551220 wait complete. 2024-04-23T14:59:06.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.933+0000 7f3200a00640 1 Processor -- start 2024-04-23T14:59:06.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.933+0000 7f3200a00640 1 -- start start 2024-04-23T14:59:06.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.934+0000 7f3200a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31fc076530 0x7f31fc19ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:06.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.934+0000 7f3200a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31fc076e50 0x7f31fc19f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:06.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.934+0000 7f3200a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31fc112d40 0x7f31fc1a36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:06.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.934+0000 7f3200a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f31fc118900 con 0x7f31fc076530 2024-04-23T14:59:06.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.934+0000 7f3200a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f31fc118780 con 0x7f31fc076e50 2024-04-23T14:59:06.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.934+0000 7f3200a00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f31fc118a80 con 0x7f31fc112d40 2024-04-23T14:59:06.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.934+0000 7f31fb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31fc076530 0x7f31fc19ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:06.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.934+0000 7f31fb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31fc076530 0x7f31fc19ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51738/0 (socket says 172.21.15.28:51738) 2024-04-23T14:59:06.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.934+0000 7f31fb400640 1 -- 172.21.15.28:0/4102252012 learned_addr learned my addr 172.21.15.28:0/4102252012 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:59:06.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.934+0000 7f31faa00640 1 --2- 172.21.15.28:0/4102252012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31fc076e50 0x7f31fc19f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:06.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.935+0000 7f31fbe00640 1 --2- 172.21.15.28:0/4102252012 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31fc112d40 0x7f31fc1a36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:06.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.935+0000 7f31fb400640 1 -- 172.21.15.28:0/4102252012 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31fc112d40 msgr2=0x7f31fc1a36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:06.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.935+0000 7f31fb400640 1 --2- 172.21.15.28:0/4102252012 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31fc112d40 0x7f31fc1a36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:06.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.935+0000 7f31fb400640 1 -- 172.21.15.28:0/4102252012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31fc076e50 msgr2=0x7f31fc19f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:06.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.935+0000 7f31fb400640 1 --2- 172.21.15.28:0/4102252012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31fc076e50 0x7f31fc19f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:06.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.935+0000 7f31fb400640 1 -- 172.21.15.28:0/4102252012 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31fc1a3df0 con 0x7f31fc076530 2024-04-23T14:59:06.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.935+0000 7f31faa00640 1 --2- 172.21.15.28:0/4102252012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31fc076e50 0x7f31fc19f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:59:06.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.935+0000 7f31fb400640 1 --2- 172.21.15.28:0/4102252012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31fc076530 0x7f31fc19ee00 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f31ec033040 tx=0x7f31ec0093d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:59:06.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.936+0000 7f31f3e00640 1 -- 172.21.15.28:0/4102252012 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31ec004040 con 0x7f31fc076530 2024-04-23T14:59:06.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.936+0000 7f31f3e00640 1 -- 172.21.15.28:0/4102252012 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31ec0046c0 con 0x7f31fc076530 2024-04-23T14:59:06.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.936+0000 7f3200a00640 1 -- 172.21.15.28:0/4102252012 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31fc1a4080 con 0x7f31fc076530 2024-04-23T14:59:06.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.936+0000 7f31f3e00640 1 -- 172.21.15.28:0/4102252012 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31ec030ec0 con 0x7f31fc076530 2024-04-23T14:59:06.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.936+0000 7f3200a00640 1 -- 172.21.15.28:0/4102252012 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31fc1ab970 con 0x7f31fc076530 2024-04-23T14:59:06.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.937+0000 7f3200a00640 1 -- 172.21.15.28:0/4102252012 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31fc069190 con 0x7f31fc076530 2024-04-23T14:59:06.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.939+0000 7f31f3e00640 1 -- 172.21.15.28:0/4102252012 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f31ec0305a0 con 0x7f31fc076530 2024-04-23T14:59:06.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.940+0000 7f31f3e00640 1 --2- 172.21.15.28:0/4102252012 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f31e00a3ec0 0x7f31e00a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:06.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.941+0000 7f31f3e00640 1 -- 172.21.15.28:0/4102252012 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8542+0+0 (secure 0 0 0) 0x7f31ec02e030 con 0x7f31fc076530 2024-04-23T14:59:06.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.941+0000 7f31faa00640 1 --2- 172.21.15.28:0/4102252012 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f31e00a3ec0 0x7f31e00a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:06.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.941+0000 7f31faa00640 1 --2- 172.21.15.28:0/4102252012 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f31e00a3ec0 0x7f31e00a6380 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f31fc0794b0 tx=0x7f31e4005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:59:06.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:06.947+0000 7f31f3e00640 1 -- 172.21.15.28:0/4102252012 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f31ec04b060 con 0x7f31fc076530 2024-04-23T14:59:07.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:06 smithi028 ceph-mon[27810]: pgmap v471: 97 pgs: 97 active+clean; 849 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 65 KiB/s wr, 20 op/s 2024-04-23T14:59:07.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:06 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/526970855' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:59:07.124 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:06 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2770613702' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:59:07.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.177+0000 7f3200a00640 1 -- 172.21.15.28:0/4102252012 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f31fc10ded0 con 0x7f31fc076530 2024-04-23T14:59:07.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.178+0000 7f31f3e00640 1 -- 172.21.15.28:0/4102252012 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v104) ==== 104+0+11 (secure 0 0 0) 0x7f31ec0be450 con 0x7f31fc076530 2024-04-23T14:59:07.179 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-23T14:59:07.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.180+0000 7f3200a00640 1 -- 172.21.15.28:0/4102252012 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f31e00a3ec0 msgr2=0x7f31e00a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:07.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.180+0000 7f3200a00640 1 --2- 172.21.15.28:0/4102252012 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f31e00a3ec0 0x7f31e00a6380 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f31fc0794b0 tx=0x7f31e4005f90 comp rx=0 tx=0).stop 2024-04-23T14:59:07.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.181+0000 7f3200a00640 1 -- 172.21.15.28:0/4102252012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31fc076530 msgr2=0x7f31fc19ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:07.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.181+0000 7f3200a00640 1 --2- 172.21.15.28:0/4102252012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31fc076530 0x7f31fc19ee00 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f31ec033040 tx=0x7f31ec0093d0 comp rx=0 tx=0).stop 2024-04-23T14:59:07.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.181+0000 7f3200a00640 1 -- 172.21.15.28:0/4102252012 shutdown_connections 2024-04-23T14:59:07.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.181+0000 7f3200a00640 1 --2- 172.21.15.28:0/4102252012 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f31e00a3ec0 0x7f31e00a6380 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:07.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.181+0000 7f3200a00640 1 --2- 172.21.15.28:0/4102252012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31fc076e50 0x7f31fc19f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:07.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.181+0000 7f3200a00640 1 --2- 172.21.15.28:0/4102252012 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31fc112d40 0x7f31fc1a36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:07.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.181+0000 7f3200a00640 1 --2- 172.21.15.28:0/4102252012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31fc076530 0x7f31fc19ee00 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:07.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.181+0000 7f3200a00640 1 -- 172.21.15.28:0/4102252012 >> 172.21.15.28:0/4102252012 conn(0x7f31fc0ffb70 msgr2=0x7f31fc1139b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:59:07.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.181+0000 7f3200a00640 1 -- 172.21.15.28:0/4102252012 shutdown_connections 2024-04-23T14:59:07.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.181+0000 7f3200a00640 1 -- 172.21.15.28:0/4102252012 wait complete. 2024-04-23T14:59:07.202 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T14:59:07.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:06 smithi105 ceph-mon[32555]: pgmap v471: 97 pgs: 97 active+clean; 849 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 65 KiB/s wr, 20 op/s 2024-04-23T14:59:07.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:06 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/526970855' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:59:07.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:06 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2770613702' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:59:07.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:06 smithi038 ceph-mon[32348]: pgmap v471: 97 pgs: 97 active+clean; 849 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 65 KiB/s wr, 20 op/s 2024-04-23T14:59:07.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:06 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/526970855' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:59:07.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:06 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2770613702' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T14:59:07.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.419+0000 7f950f200640 1 -- 172.21.15.28:0/443165485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9510102ac0 msgr2=0x7f9510106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:07.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.419+0000 7f950f200640 1 --2- 172.21.15.28:0/443165485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9510102ac0 0x7f9510106b00 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f94f8009ca0 tx=0x7f94f802a8c0 comp rx=0 tx=0).stop 2024-04-23T14:59:07.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.420+0000 7f950f200640 1 -- 172.21.15.28:0/443165485 shutdown_connections 2024-04-23T14:59:07.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.420+0000 7f950f200640 1 --2- 172.21.15.28:0/443165485 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95101071e0 0x7f951010edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:07.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.420+0000 7f950f200640 1 --2- 172.21.15.28:0/443165485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9510102ac0 0x7f9510106b00 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:07.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.420+0000 7f950f200640 1 --2- 172.21.15.28:0/443165485 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9510102110 0x7f95101024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:07.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.420+0000 7f950f200640 1 -- 172.21.15.28:0/443165485 >> 172.21.15.28:0/443165485 conn(0x7f95100fd9d0 msgr2=0x7f95100ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:59:07.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.420+0000 7f950f200640 1 -- 172.21.15.28:0/443165485 shutdown_connections 2024-04-23T14:59:07.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.420+0000 7f950f200640 1 -- 172.21.15.28:0/443165485 wait complete. 2024-04-23T14:59:07.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.421+0000 7f950f200640 1 Processor -- start 2024-04-23T14:59:07.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.421+0000 7f950f200640 1 -- start start 2024-04-23T14:59:07.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.422+0000 7f950f200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9510102110 0x7f951019e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:07.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.422+0000 7f950f200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95101071e0 0x7f951019e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:07.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.422+0000 7f950f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95101985d0 0x7f9510198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:07.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.422+0000 7f950f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9510112270 con 0x7f95101985d0 2024-04-23T14:59:07.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.422+0000 7f950f200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f95101120f0 con 0x7f95101071e0 2024-04-23T14:59:07.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.422+0000 7f950f200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f95101123f0 con 0x7f9510102110 2024-04-23T14:59:07.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.422+0000 7f950e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95101985d0 0x7f9510198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:07.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.422+0000 7f950e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95101985d0 0x7f9510198a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51764/0 (socket says 172.21.15.28:51764) 2024-04-23T14:59:07.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.422+0000 7f950e800640 1 -- 172.21.15.28:0/1075177761 learned_addr learned my addr 172.21.15.28:0/1075177761 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:59:07.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.422+0000 7f950de00640 1 --2- 172.21.15.28:0/1075177761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9510102110 0x7f951019e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:07.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.422+0000 7f950d400640 1 --2- 172.21.15.28:0/1075177761 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95101071e0 0x7f951019e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:07.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.422+0000 7f950e800640 1 -- 172.21.15.28:0/1075177761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9510102110 msgr2=0x7f951019e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:07.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.422+0000 7f950e800640 1 --2- 172.21.15.28:0/1075177761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9510102110 0x7f951019e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:07.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.422+0000 7f950e800640 1 -- 172.21.15.28:0/1075177761 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95101071e0 msgr2=0x7f951019e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:07.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.422+0000 7f950e800640 1 --2- 172.21.15.28:0/1075177761 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95101071e0 0x7f951019e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:07.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.422+0000 7f950e800640 1 -- 172.21.15.28:0/1075177761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95101990f0 con 0x7f95101985d0 2024-04-23T14:59:07.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.423+0000 7f950de00640 1 --2- 172.21.15.28:0/1075177761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9510102110 0x7f951019e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:59:07.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.423+0000 7f950e800640 1 --2- 172.21.15.28:0/1075177761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95101985d0 0x7f9510198a80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f950800e970 tx=0x7f950800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:59:07.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.423+0000 7f9506a00640 1 -- 172.21.15.28:0/1075177761 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f950800ccc0 con 0x7f95101985d0 2024-04-23T14:59:07.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.423+0000 7f9506a00640 1 -- 172.21.15.28:0/1075177761 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f950800ce60 con 0x7f95101985d0 2024-04-23T14:59:07.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.423+0000 7f9506a00640 1 -- 172.21.15.28:0/1075177761 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9508010490 con 0x7f95101985d0 2024-04-23T14:59:07.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.423+0000 7f950f200640 1 -- 172.21.15.28:0/1075177761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95101993e0 con 0x7f95101985d0 2024-04-23T14:59:07.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.423+0000 7f950f200640 1 -- 172.21.15.28:0/1075177761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95101a55e0 con 0x7f95101985d0 2024-04-23T14:59:07.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.425+0000 7f950f200640 1 -- 172.21.15.28:0/1075177761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9510068d40 con 0x7f95101985d0 2024-04-23T14:59:07.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.427+0000 7f9506a00640 1 -- 172.21.15.28:0/1075177761 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f9508005020 con 0x7f95101985d0 2024-04-23T14:59:07.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.428+0000 7f9506a00640 1 --2- 172.21.15.28:0/1075177761 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f94f00a3da0 0x7f94f00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:07.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.428+0000 7f9506a00640 1 -- 172.21.15.28:0/1075177761 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8542+0+0 (secure 0 0 0) 0x7f95080d0ba0 con 0x7f95101985d0 2024-04-23T14:59:07.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.428+0000 7f950de00640 1 --2- 172.21.15.28:0/1075177761 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f94f00a3da0 0x7f94f00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:07.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.429+0000 7f950de00640 1 --2- 172.21.15.28:0/1075177761 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f94f00a3da0 0x7f94f00a6260 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f95000095f0 tx=0x7f9500005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:59:07.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.432+0000 7f9506a00640 1 -- 172.21.15.28:0/1075177761 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f950809d1d0 con 0x7f95101985d0 2024-04-23T14:59:07.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.641+0000 7f950f200640 1 -- 172.21.15.28:0/1075177761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9510102ac0 con 0x7f95101985d0 2024-04-23T14:59:07.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.642+0000 7f9506a00640 1 -- 172.21.15.28:0/1075177761 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v104) ==== 100+0+11 (secure 0 0 0) 0x7f950809cad0 con 0x7f95101985d0 2024-04-23T14:59:07.643 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-23T14:59:07.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.645+0000 7f950f200640 1 -- 172.21.15.28:0/1075177761 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f94f00a3da0 msgr2=0x7f94f00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:07.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.645+0000 7f950f200640 1 --2- 172.21.15.28:0/1075177761 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f94f00a3da0 0x7f94f00a6260 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f95000095f0 tx=0x7f9500005f90 comp rx=0 tx=0).stop 2024-04-23T14:59:07.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.645+0000 7f950f200640 1 -- 172.21.15.28:0/1075177761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95101985d0 msgr2=0x7f9510198a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:07.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.645+0000 7f950f200640 1 --2- 172.21.15.28:0/1075177761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95101985d0 0x7f9510198a80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f950800e970 tx=0x7f950800ed70 comp rx=0 tx=0).stop 2024-04-23T14:59:07.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.645+0000 7f950f200640 1 -- 172.21.15.28:0/1075177761 shutdown_connections 2024-04-23T14:59:07.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.645+0000 7f950f200640 1 --2- 172.21.15.28:0/1075177761 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f94f00a3da0 0x7f94f00a6260 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:07.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.645+0000 7f950f200640 1 --2- 172.21.15.28:0/1075177761 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95101071e0 0x7f951019e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:07.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.645+0000 7f950f200640 1 --2- 172.21.15.28:0/1075177761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95101985d0 0x7f9510198a80 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:07.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.645+0000 7f950f200640 1 --2- 172.21.15.28:0/1075177761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9510102110 0x7f951019e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:07.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.645+0000 7f950f200640 1 -- 172.21.15.28:0/1075177761 >> 172.21.15.28:0/1075177761 conn(0x7f95100fd9d0 msgr2=0x7f95100fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:59:07.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.646+0000 7f950f200640 1 -- 172.21.15.28:0/1075177761 shutdown_connections 2024-04-23T14:59:07.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.646+0000 7f950f200640 1 -- 172.21.15.28:0/1075177761 wait complete. 2024-04-23T14:59:07.667 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-23T14:59:07.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.881+0000 7fc2a8a00640 1 -- 172.21.15.28:0/379696788 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc298031940 con 0x7fc2ac076e50 2024-04-23T14:59:07.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.882+0000 7fc2ab200640 1 -- 172.21.15.28:0/379696788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2ac076e50 msgr2=0x7fc2ac112800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:07.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.882+0000 7fc2ab200640 1 --2- 172.21.15.28:0/379696788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2ac076e50 0x7fc2ac112800 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc298009900 tx=0x7fc29802a410 comp rx=0 tx=0).stop 2024-04-23T14:59:07.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.882+0000 7fc2ab200640 1 -- 172.21.15.28:0/379696788 shutdown_connections 2024-04-23T14:59:07.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.882+0000 7fc2ab200640 1 --2- 172.21.15.28:0/379696788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2ac112d40 0x7fc2ac115130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:07.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.882+0000 7fc2ab200640 1 --2- 172.21.15.28:0/379696788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2ac076e50 0x7fc2ac112800 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:07.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.882+0000 7fc2ab200640 1 --2- 172.21.15.28:0/379696788 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2ac076530 0x7fc2ac076910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:07.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.882+0000 7fc2ab200640 1 -- 172.21.15.28:0/379696788 >> 172.21.15.28:0/379696788 conn(0x7fc2ac0ffb30 msgr2=0x7fc2ac101f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:59:07.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.882+0000 7fc2ab200640 1 -- 172.21.15.28:0/379696788 shutdown_connections 2024-04-23T14:59:07.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.882+0000 7fc2ab200640 1 -- 172.21.15.28:0/379696788 wait complete. 2024-04-23T14:59:07.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.883+0000 7fc2ab200640 1 Processor -- start 2024-04-23T14:59:07.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.883+0000 7fc2ab200640 1 -- start start 2024-04-23T14:59:07.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.883+0000 7fc2ab200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2ac076530 0x7fc2ac19ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:07.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.883+0000 7fc2ab200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2ac076e50 0x7fc2ac19f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:07.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.883+0000 7fc2ab200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2ac112d40 0x7fc2ac1a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:07.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.883+0000 7fc2ab200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc2ac118890 con 0x7fc2ac076e50 2024-04-23T14:59:07.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.883+0000 7fc2ab200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc2ac118710 con 0x7fc2ac076530 2024-04-23T14:59:07.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.883+0000 7fc2ab200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc2ac118a10 con 0x7fc2ac112d40 2024-04-23T14:59:07.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.884+0000 7fc2aa800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2ac112d40 0x7fc2ac1a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.884+0000 7fc2a9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2ac076e50 0x7fc2ac19f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.884+0000 7fc2aa800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2ac112d40 0x7fc2ac1a3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:37908/0 (socket says 172.21.15.28:37908) 2024-04-23T14:59:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.884+0000 7fc2aa800640 1 -- 172.21.15.28:0/546931274 learned_addr learned my addr 172.21.15.28:0/546931274 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:59:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.884+0000 7fc2a9e00640 1 --2- 172.21.15.28:0/546931274 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2ac076530 0x7fc2ac19ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.884+0000 7fc2aa800640 1 -- 172.21.15.28:0/546931274 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2ac076530 msgr2=0x7fc2ac19ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.884+0000 7fc2aa800640 1 --2- 172.21.15.28:0/546931274 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2ac076530 0x7fc2ac19ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.884+0000 7fc2aa800640 1 -- 172.21.15.28:0/546931274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2ac076e50 msgr2=0x7fc2ac19f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.884+0000 7fc2aa800640 1 --2- 172.21.15.28:0/546931274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2ac076e50 0x7fc2ac19f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.884+0000 7fc2aa800640 1 -- 172.21.15.28:0/546931274 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2ac1a3db0 con 0x7fc2ac112d40 2024-04-23T14:59:07.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.884+0000 7fc2aa800640 1 --2- 172.21.15.28:0/546931274 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2ac112d40 0x7fc2ac1a3680 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fc2a400e970 tx=0x7fc2a400ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:59:07.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.885+0000 7fc2a2a00640 1 -- 172.21.15.28:0/546931274 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a400ccc0 con 0x7fc2ac112d40 2024-04-23T14:59:07.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.885+0000 7fc2ab200640 1 -- 172.21.15.28:0/546931274 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2ac1a40a0 con 0x7fc2ac112d40 2024-04-23T14:59:07.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.885+0000 7fc2a2a00640 1 -- 172.21.15.28:0/546931274 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2a4005500 con 0x7fc2ac112d40 2024-04-23T14:59:07.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.885+0000 7fc2a2a00640 1 -- 172.21.15.28:0/546931274 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a4010690 con 0x7fc2ac112d40 2024-04-23T14:59:07.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.885+0000 7fc2ab200640 1 -- 172.21.15.28:0/546931274 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2ac1ab930 con 0x7fc2ac112d40 2024-04-23T14:59:07.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.886+0000 7fc2ab200640 1 -- 172.21.15.28:0/546931274 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2ac0684b0 con 0x7fc2ac112d40 2024-04-23T14:59:07.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.888+0000 7fc2a2a00640 1 -- 172.21.15.28:0/546931274 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fc2a40046c0 con 0x7fc2ac112d40 2024-04-23T14:59:07.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.890+0000 7fc2a2a00640 1 --2- 172.21.15.28:0/546931274 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc2900a3da0 0x7fc2900a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:07.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.890+0000 7fc2a2a00640 1 -- 172.21.15.28:0/546931274 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8542+0+0 (secure 0 0 0) 0x7fc2a40d0d20 con 0x7fc2ac112d40 2024-04-23T14:59:07.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.890+0000 7fc2a9e00640 1 --2- 172.21.15.28:0/546931274 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc2900a3da0 0x7fc2900a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:07.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.891+0000 7fc2a9e00640 1 --2- 172.21.15.28:0/546931274 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc2900a3da0 0x7fc2900a6260 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc29c0099a0 tx=0x7fc29c009500 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:59:07.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:07.891+0000 7fc2a2a00640 1 -- 172.21.15.28:0/546931274 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc2a4053ff0 con 0x7fc2ac112d40 2024-04-23T14:59:07.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:07 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/4102252012' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T14:59:07.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:07 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1075177761' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T14:59:07.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:07 smithi028 ceph-mon[27810]: pgmap v472: 97 pgs: 97 active+clean; 849 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 70 KiB/s wr, 22 op/s 2024-04-23T14:59:08.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.123+0000 7fc2ab200640 1 -- 172.21.15.28:0/546931274 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fc2ac1a0110 con 0x7fc2ac112d40 2024-04-23T14:59:08.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.124+0000 7fc2a2a00640 1 -- 172.21.15.28:0/546931274 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v104) ==== 81+0+940 (secure 0 0 0) 0x7fc2ac1a0110 con 0x7fc2ac112d40 2024-04-23T14:59:08.125 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/38494154 2024-04-24T14:44:01.286264+0000 2024-04-23T14:59:08.125 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/905362079 2024-04-24T14:44:01.286264+0000 2024-04-23T14:59:08.125 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/1997136173 2024-04-24T14:44:01.286264+0000 2024-04-23T14:59:08.125 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/1997136173 2024-04-24T14:44:01.286264+0000 2024-04-23T14:59:08.125 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/590495740 2024-04-24T14:33:41.530295+0000 2024-04-23T14:59:08.125 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/1163172512 2024-04-24T14:33:41.530295+0000 2024-04-23T14:59:08.126 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/2222038555 2024-04-24T14:33:41.530295+0000 2024-04-23T14:59:08.126 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/4290328460 2024-04-24T14:33:07.576363+0000 2024-04-23T14:59:08.126 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/4145020172 2024-04-24T14:33:07.576363+0000 2024-04-23T14:59:08.126 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/1997097543 2024-04-24T14:44:01.286264+0000 2024-04-23T14:59:08.126 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/3847772687 2024-04-24T14:44:01.286264+0000 2024-04-23T14:59:08.126 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/1307025091 2024-04-24T14:33:07.576363+0000 2024-04-23T14:59:08.126 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/2222038555 2024-04-24T14:33:41.530295+0000 2024-04-23T14:59:08.127 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/720626958 2024-04-24T14:33:41.530295+0000 2024-04-23T14:59:08.127 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/1307025091 2024-04-24T14:33:07.576363+0000 2024-04-23T14:59:08.127 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/948450262 2024-04-24T14:33:07.576363+0000 2024-04-23T14:59:08.128 INFO:teuthology.orchestra.run.smithi028.stderr:listed 16 entries 2024-04-23T14:59:08.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.126+0000 7fc2ab200640 1 -- 172.21.15.28:0/546931274 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc2900a3da0 msgr2=0x7fc2900a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:08.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.126+0000 7fc2ab200640 1 --2- 172.21.15.28:0/546931274 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc2900a3da0 0x7fc2900a6260 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc29c0099a0 tx=0x7fc29c009500 comp rx=0 tx=0).stop 2024-04-23T14:59:08.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.126+0000 7fc2ab200640 1 -- 172.21.15.28:0/546931274 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2ac112d40 msgr2=0x7fc2ac1a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:08.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.126+0000 7fc2ab200640 1 --2- 172.21.15.28:0/546931274 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2ac112d40 0x7fc2ac1a3680 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fc2a400e970 tx=0x7fc2a400ee30 comp rx=0 tx=0).stop 2024-04-23T14:59:08.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.126+0000 7fc2ab200640 1 -- 172.21.15.28:0/546931274 shutdown_connections 2024-04-23T14:59:08.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.126+0000 7fc2ab200640 1 --2- 172.21.15.28:0/546931274 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fc2900a3da0 0x7fc2900a6260 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:08.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.126+0000 7fc2ab200640 1 --2- 172.21.15.28:0/546931274 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2ac112d40 0x7fc2ac1a3680 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:08.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.126+0000 7fc2ab200640 1 --2- 172.21.15.28:0/546931274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2ac076e50 0x7fc2ac19f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:08.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.126+0000 7fc2ab200640 1 --2- 172.21.15.28:0/546931274 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2ac076530 0x7fc2ac19ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:08.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.126+0000 7fc2ab200640 1 -- 172.21.15.28:0/546931274 >> 172.21.15.28:0/546931274 conn(0x7fc2ac0ffb30 msgr2=0x7fc2ac114400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:59:08.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.127+0000 7fc2ab200640 1 -- 172.21.15.28:0/546931274 shutdown_connections 2024-04-23T14:59:08.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.127+0000 7fc2ab200640 1 -- 172.21.15.28:0/546931274 wait complete. 2024-04-23T14:59:08.148 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-23T14:59:08.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:59:08.148 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-23T14:59:08.273 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:07 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/4102252012' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T14:59:08.273 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:07 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1075177761' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T14:59:08.274 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:07 smithi038 ceph-mon[32348]: pgmap v472: 97 pgs: 97 active+clean; 849 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 70 KiB/s wr, 22 op/s 2024-04-23T14:59:08.285 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:07 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/4102252012' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T14:59:08.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:07 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1075177761' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T14:59:08.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:07 smithi105 ceph-mon[32555]: pgmap v472: 97 pgs: 97 active+clean; 849 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 70 KiB/s wr, 22 op/s 2024-04-23T14:59:08.361 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-04-23T14:59:08.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:59:08.362 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-23T14:59:08.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:59:08.423 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-04-23T14:59:08.459 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-23T14:59:08.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:59:08.460 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-23T14:59:08.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T14:59:08.568 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-04-23T14:59:08.660 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-23T14:59:08.671 INFO:tasks.cephadm:Teardown begin 2024-04-23T14:59:08.671 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-23T14:59:08.671 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:59:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.848+0000 7fd3e3000640 1 -- 172.21.15.28:0/3503556410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3dc104c10 msgr2=0x7fd3dc108c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:08.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.848+0000 7fd3e3000640 1 --2- 172.21.15.28:0/3503556410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3dc104c10 0x7fd3dc108c50 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8009900 tx=0x7fd3c802a410 comp rx=0 tx=0).stop 2024-04-23T14:59:08.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.848+0000 7fd3e3000640 1 -- 172.21.15.28:0/3503556410 shutdown_connections 2024-04-23T14:59:08.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.848+0000 7fd3e3000640 1 --2- 172.21.15.28:0/3503556410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3dc104c10 0x7fd3dc108c50 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:08.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.848+0000 7fd3e3000640 1 --2- 172.21.15.28:0/3503556410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3dc109330 0x7fd3dc110f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:08.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.848+0000 7fd3e3000640 1 --2- 172.21.15.28:0/3503556410 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd3dc104260 0x7fd3dc104640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:08.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.848+0000 7fd3e3000640 1 -- 172.21.15.28:0/3503556410 >> 172.21.15.28:0/3503556410 conn(0x7fd3dc0ffaa0 msgr2=0x7fd3dc101ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:59:08.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.848+0000 7fd3e3000640 1 -- 172.21.15.28:0/3503556410 shutdown_connections 2024-04-23T14:59:08.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.849+0000 7fd3e3000640 1 -- 172.21.15.28:0/3503556410 wait complete. 2024-04-23T14:59:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.850+0000 7fd3e3000640 1 Processor -- start 2024-04-23T14:59:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.850+0000 7fd3e3000640 1 -- start start 2024-04-23T14:59:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.850+0000 7fd3e3000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd3dc104260 0x7fd3dc1a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.850+0000 7fd3e3000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3dc109330 0x7fd3dc1a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.850+0000 7fd3e3000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3dc19a790 0x7fd3dc19ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.850+0000 7fd3e3000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd3dc114410 con 0x7fd3dc19a790 2024-04-23T14:59:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.850+0000 7fd3e3000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd3dc114290 con 0x7fd3dc109330 2024-04-23T14:59:08.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.850+0000 7fd3e3000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd3dc114590 con 0x7fd3dc104260 2024-04-23T14:59:08.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.851+0000 7fd3e2600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3dc19a790 0x7fd3dc19ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:08.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.851+0000 7fd3e1200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3dc109330 0x7fd3dc1a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:08.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.851+0000 7fd3e1c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd3dc104260 0x7fd3dc1a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:08.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.851+0000 7fd3e1200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3dc109330 0x7fd3dc1a0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48030/0 (socket says 172.21.15.28:48030) 2024-04-23T14:59:08.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.851+0000 7fd3e1200640 1 -- 172.21.15.28:0/3611828537 learned_addr learned my addr 172.21.15.28:0/3611828537 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:59:08.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.851+0000 7fd3e1c00640 1 -- 172.21.15.28:0/3611828537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3dc109330 msgr2=0x7fd3dc1a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:08.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.851+0000 7fd3e1c00640 1 --2- 172.21.15.28:0/3611828537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3dc109330 0x7fd3dc1a0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:08.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.851+0000 7fd3e1c00640 1 -- 172.21.15.28:0/3611828537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3dc19a790 msgr2=0x7fd3dc19ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:08.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.851+0000 7fd3e1c00640 1 --2- 172.21.15.28:0/3611828537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3dc19a790 0x7fd3dc19ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:08.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.851+0000 7fd3e1c00640 1 -- 172.21.15.28:0/3611828537 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3dc19b390 con 0x7fd3dc104260 2024-04-23T14:59:08.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.851+0000 7fd3e1200640 1 --2- 172.21.15.28:0/3611828537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3dc109330 0x7fd3dc1a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:59:08.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.851+0000 7fd3e1c00640 1 --2- 172.21.15.28:0/3611828537 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd3dc104260 0x7fd3dc1a0680 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd3d000e9f0 tx=0x7fd3d000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:59:08.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.852+0000 7fd3da000640 1 -- 172.21.15.28:0/3611828537 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d000ccc0 con 0x7fd3dc104260 2024-04-23T14:59:08.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.852+0000 7fd3da000640 1 -- 172.21.15.28:0/3611828537 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3d0005020 con 0x7fd3dc104260 2024-04-23T14:59:08.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.852+0000 7fd3e3000640 1 -- 172.21.15.28:0/3611828537 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3dc1a7420 con 0x7fd3dc104260 2024-04-23T14:59:08.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.852+0000 7fd3da000640 1 -- 172.21.15.28:0/3611828537 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d00049e0 con 0x7fd3dc104260 2024-04-23T14:59:08.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.852+0000 7fd3e3000640 1 -- 172.21.15.28:0/3611828537 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3dc1a77d0 con 0x7fd3dc104260 2024-04-23T14:59:08.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.853+0000 7fd3e3000640 1 -- 172.21.15.28:0/3611828537 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3dc105730 con 0x7fd3dc104260 2024-04-23T14:59:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.855+0000 7fd3da000640 1 -- 172.21.15.28:0/3611828537 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fd3d0005560 con 0x7fd3dc104260 2024-04-23T14:59:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.856+0000 7fd3da000640 1 --2- 172.21.15.28:0/3611828537 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd3c40a3d20 0x7fd3c40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.856+0000 7fd3da000640 1 -- 172.21.15.28:0/3611828537 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8542+0+0 (secure 0 0 0) 0x7fd3d00d08e0 con 0x7fd3dc104260 2024-04-23T14:59:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.856+0000 7fd3e1200640 1 --2- 172.21.15.28:0/3611828537 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd3c40a3d20 0x7fd3c40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.857+0000 7fd3e1200640 1 --2- 172.21.15.28:0/3611828537 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd3c40a3d20 0x7fd3c40a61e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc19bf90 tx=0x7fd3c8031510 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:59:08.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:08.858+0000 7fd3da000640 1 -- 172.21.15.28:0/3611828537 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd3d00051d0 con 0x7fd3dc104260 2024-04-23T14:59:09.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.106+0000 7fd3e3000640 1 -- 172.21.15.28:0/3611828537 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd3dc069190 con 0x7fd3dc104260 2024-04-23T14:59:09.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.107+0000 7fd3da000640 1 -- 172.21.15.28:0/3611828537 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7fd3d009cf10 con 0x7fd3dc104260 2024-04-23T14:59:09.108 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:59:09.108 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:59:09.108 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:59:09.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.109+0000 7fd3e3000640 1 -- 172.21.15.28:0/3611828537 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd3c40a3d20 msgr2=0x7fd3c40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:09.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.109+0000 7fd3e3000640 1 --2- 172.21.15.28:0/3611828537 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd3c40a3d20 0x7fd3c40a61e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc19bf90 tx=0x7fd3c8031510 comp rx=0 tx=0).stop 2024-04-23T14:59:09.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.110+0000 7fd3e3000640 1 -- 172.21.15.28:0/3611828537 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd3dc104260 msgr2=0x7fd3dc1a0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:09.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.110+0000 7fd3e3000640 1 --2- 172.21.15.28:0/3611828537 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd3dc104260 0x7fd3dc1a0680 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd3d000e9f0 tx=0x7fd3d000eeb0 comp rx=0 tx=0).stop 2024-04-23T14:59:09.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.110+0000 7fd3e3000640 1 -- 172.21.15.28:0/3611828537 shutdown_connections 2024-04-23T14:59:09.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.110+0000 7fd3e3000640 1 --2- 172.21.15.28:0/3611828537 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd3c40a3d20 0x7fd3c40a61e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:09.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.110+0000 7fd3e3000640 1 --2- 172.21.15.28:0/3611828537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3dc109330 0x7fd3dc1a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:09.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.110+0000 7fd3e3000640 1 --2- 172.21.15.28:0/3611828537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3dc19a790 0x7fd3dc19ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:09.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.110+0000 7fd3e3000640 1 --2- 172.21.15.28:0/3611828537 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd3dc104260 0x7fd3dc1a0680 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:09.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.110+0000 7fd3e3000640 1 -- 172.21.15.28:0/3611828537 >> 172.21.15.28:0/3611828537 conn(0x7fd3dc0ffaa0 msgr2=0x7fd3dc101db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:59:09.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.110+0000 7fd3e3000640 1 -- 172.21.15.28:0/3611828537 shutdown_connections 2024-04-23T14:59:09.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.110+0000 7fd3e3000640 1 -- 172.21.15.28:0/3611828537 wait complete. 2024-04-23T14:59:09.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:08 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/546931274' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-23T14:59:09.129 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T14:59:09.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:08 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/546931274' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-23T14:59:09.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.328+0000 7fbb3f200640 1 -- 172.21.15.28:0/1794780970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb4010d3c0 msgr2=0x7fbb4010d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:09.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.328+0000 7fbb3f200640 1 --2- 172.21.15.28:0/1794780970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb4010d3c0 0x7fbb4010d7a0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fbb30009900 tx=0x7fbb3002a480 comp rx=0 tx=0).stop 2024-04-23T14:59:09.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.329+0000 7fbb3f200640 1 -- 172.21.15.28:0/1794780970 shutdown_connections 2024-04-23T14:59:09.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.329+0000 7fbb3f200640 1 --2- 172.21.15.28:0/1794780970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb40108e40 0x7fbb401152b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:09.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.329+0000 7fbb3f200640 1 --2- 172.21.15.28:0/1794780970 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb401084a0 0x7fbb40108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:09.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.329+0000 7fbb3f200640 1 --2- 172.21.15.28:0/1794780970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb4010d3c0 0x7fbb4010d7a0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:09.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.329+0000 7fbb3f200640 1 -- 172.21.15.28:0/1794780970 >> 172.21.15.28:0/1794780970 conn(0x7fbb40077b80 msgr2=0x7fbb40079fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:59:09.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.329+0000 7fbb3f200640 1 -- 172.21.15.28:0/1794780970 shutdown_connections 2024-04-23T14:59:09.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.329+0000 7fbb3f200640 1 -- 172.21.15.28:0/1794780970 wait complete. 2024-04-23T14:59:09.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.330+0000 7fbb3f200640 1 Processor -- start 2024-04-23T14:59:09.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.330+0000 7fbb3f200640 1 -- start start 2024-04-23T14:59:09.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.331+0000 7fbb3f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb401084a0 0x7fbb401a4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:09.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.331+0000 7fbb3f200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb40108e40 0x7fbb401a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:09.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.331+0000 7fbb3f200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb4010d3c0 0x7fbb4019eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:09.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.331+0000 7fbb3f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbb40118830 con 0x7fbb401084a0 2024-04-23T14:59:09.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.331+0000 7fbb3f200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbb401186b0 con 0x7fbb4010d3c0 2024-04-23T14:59:09.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.331+0000 7fbb3f200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fbb401189b0 con 0x7fbb40108e40 2024-04-23T14:59:09.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.331+0000 7fbb3de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb401084a0 0x7fbb401a4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:09.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.331+0000 7fbb3de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb401084a0 0x7fbb401a4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40688/0 (socket says 172.21.15.28:40688) 2024-04-23T14:59:09.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.331+0000 7fbb3de00640 1 -- 172.21.15.28:0/3329267624 learned_addr learned my addr 172.21.15.28:0/3329267624 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:59:09.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.331+0000 7fbb3d400640 1 --2- 172.21.15.28:0/3329267624 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb40108e40 0x7fbb401a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:09.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.331+0000 7fbb3e800640 1 --2- 172.21.15.28:0/3329267624 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb4010d3c0 0x7fbb4019eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:09.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.331+0000 7fbb3d400640 1 -- 172.21.15.28:0/3329267624 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb4010d3c0 msgr2=0x7fbb4019eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:09.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.331+0000 7fbb3d400640 1 --2- 172.21.15.28:0/3329267624 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb4010d3c0 0x7fbb4019eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:09.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.331+0000 7fbb3d400640 1 -- 172.21.15.28:0/3329267624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb401084a0 msgr2=0x7fbb401a4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:09.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.331+0000 7fbb3d400640 1 --2- 172.21.15.28:0/3329267624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb401084a0 0x7fbb401a4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:09.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.331+0000 7fbb3d400640 1 -- 172.21.15.28:0/3329267624 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb4019f270 con 0x7fbb40108e40 2024-04-23T14:59:09.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.332+0000 7fbb3e800640 1 --2- 172.21.15.28:0/3329267624 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb4010d3c0 0x7fbb4019eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:59:09.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.332+0000 7fbb3d400640 1 --2- 172.21.15.28:0/3329267624 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb40108e40 0x7fbb401a4e80 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fbb2800e9f0 tx=0x7fbb2800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:59:09.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.332+0000 7fbb36a00640 1 -- 172.21.15.28:0/3329267624 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2800cd80 con 0x7fbb40108e40 2024-04-23T14:59:09.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.332+0000 7fbb36a00640 1 -- 172.21.15.28:0/3329267624 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb280035f0 con 0x7fbb40108e40 2024-04-23T14:59:09.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.332+0000 7fbb3f200640 1 -- 172.21.15.28:0/3329267624 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb4019f560 con 0x7fbb40108e40 2024-04-23T14:59:09.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.332+0000 7fbb36a00640 1 -- 172.21.15.28:0/3329267624 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb28010690 con 0x7fbb40108e40 2024-04-23T14:59:09.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.332+0000 7fbb3f200640 1 -- 172.21.15.28:0/3329267624 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb401ab800 con 0x7fbb40108e40 2024-04-23T14:59:09.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.333+0000 7fbb3f200640 1 -- 172.21.15.28:0/3329267624 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb40069190 con 0x7fbb40108e40 2024-04-23T14:59:09.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.336+0000 7fbb36a00640 1 -- 172.21.15.28:0/3329267624 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fbb28003b30 con 0x7fbb40108e40 2024-04-23T14:59:09.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.337+0000 7fbb36a00640 1 --2- 172.21.15.28:0/3329267624 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fbb240a3d20 0x7fbb240a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:09.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.338+0000 7fbb36a00640 1 -- 172.21.15.28:0/3329267624 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8542+0+0 (secure 0 0 0) 0x7fbb280d0b20 con 0x7fbb40108e40 2024-04-23T14:59:09.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.338+0000 7fbb3de00640 1 --2- 172.21.15.28:0/3329267624 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fbb240a3d20 0x7fbb240a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:09.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.338+0000 7fbb3de00640 1 --2- 172.21.15.28:0/3329267624 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fbb240a3d20 0x7fbb240a61e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbb300096c0 tx=0x7fbb300046c0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:59:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.341+0000 7fbb36a00640 1 -- 172.21.15.28:0/3329267624 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbb2809d150 con 0x7fbb40108e40 2024-04-23T14:59:09.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:08 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/546931274' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-23T14:59:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.586+0000 7fbb3f200640 1 -- 172.21.15.28:0/3329267624 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb40110960 con 0x7fbb40108e40 2024-04-23T14:59:09.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.587+0000 7fbb36a00640 1 -- 172.21.15.28:0/3329267624 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11798 (secure 0 0 0) 0x7fbb2809ca50 con 0x7fbb40108e40 2024-04-23T14:59:09.588 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:59:09.588 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/760880137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":760880137},{"type":"v1","addr":"172.21.15.28:6835","nonce":760880137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2963041370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2963041370},{"type":"v1","addr":"172.21.15.28:6837","nonce":2963041370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1673432987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1673432987},{"type":"v1","addr":"172.21.15.38:6833","nonce":1673432987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/573683071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":573683071},{"type":"v1","addr":"172.21.15.105:6833","nonce":573683071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/2393540699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":2393540699},{"type":"v1","addr":"172.21.15.38:6835","nonce":2393540699}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24529,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1788458050","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1788458050},{"type":"v1","addr":"172.21.15.105:6835","nonce":1788458050}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/4078619599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":4078619599},{"type":"v1","addr":"172.21.15.38:6837","nonce":4078619599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/1691671537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":1691671537},{"type":"v1","addr":"172.21.15.105:6837","nonce":1691671537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/1987159450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":1987159450},{"type":"v1","addr":"172.21.15.105:6839","nonce":1987159450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1569158142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1569158142},{"type":"v1","addr":"172.21.15.28:6841","nonce":1569158142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T14:41:39.495067+0000","modified":"2024-04-23T14:41:43.869888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"g","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/940220693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":940220693},{"type":"v1","addr":"172.21.15.28:6839","nonce":940220693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/377787688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":377787688},{"type":"v1","addr":"172.21.15.38:6839","nonce":377787688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-23T14:59:09.589 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 20 2024-04-23T14:59:09.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.589+0000 7fbb3f200640 1 -- 172.21.15.28:0/3329267624 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fbb240a3d20 msgr2=0x7fbb240a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:09.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.590+0000 7fbb3f200640 1 --2- 172.21.15.28:0/3329267624 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fbb240a3d20 0x7fbb240a61e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbb300096c0 tx=0x7fbb300046c0 comp rx=0 tx=0).stop 2024-04-23T14:59:09.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.590+0000 7fbb3f200640 1 -- 172.21.15.28:0/3329267624 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb40108e40 msgr2=0x7fbb401a4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:09.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.590+0000 7fbb3f200640 1 --2- 172.21.15.28:0/3329267624 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb40108e40 0x7fbb401a4e80 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fbb2800e9f0 tx=0x7fbb2800eeb0 comp rx=0 tx=0).stop 2024-04-23T14:59:09.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.590+0000 7fbb3f200640 1 -- 172.21.15.28:0/3329267624 shutdown_connections 2024-04-23T14:59:09.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.590+0000 7fbb3f200640 1 --2- 172.21.15.28:0/3329267624 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fbb240a3d20 0x7fbb240a61e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:09.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.590+0000 7fbb3f200640 1 --2- 172.21.15.28:0/3329267624 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb4010d3c0 0x7fbb4019eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:09.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.590+0000 7fbb3f200640 1 --2- 172.21.15.28:0/3329267624 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb40108e40 0x7fbb401a4e80 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:09.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.590+0000 7fbb3f200640 1 --2- 172.21.15.28:0/3329267624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb401084a0 0x7fbb401a4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:09.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.590+0000 7fbb3f200640 1 -- 172.21.15.28:0/3329267624 >> 172.21.15.28:0/3329267624 conn(0x7fbb40077b80 msgr2=0x7fbb400784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:59:09.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.590+0000 7fbb3f200640 1 -- 172.21.15.28:0/3329267624 shutdown_connections 2024-04-23T14:59:09.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.590+0000 7fbb3f200640 1 -- 172.21.15.28:0/3329267624 wait complete. 2024-04-23T14:59:09.610 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T14:59:09.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.819+0000 7f05e7e00640 1 -- 172.21.15.28:0/1074281497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05e8109340 msgr2=0x7f05e8110f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:09.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.819+0000 7f05e7e00640 1 --2- 172.21.15.28:0/1074281497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05e8109340 0x7f05e8110f10 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f05e0009900 tx=0x7f05e002a3f0 comp rx=0 tx=0).stop 2024-04-23T14:59:09.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.819+0000 7f05e7e00640 1 -- 172.21.15.28:0/1074281497 shutdown_connections 2024-04-23T14:59:09.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.819+0000 7f05e7e00640 1 --2- 172.21.15.28:0/1074281497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05e8109340 0x7f05e8110f10 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:09.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.819+0000 7f05e7e00640 1 --2- 172.21.15.28:0/1074281497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05e8104c20 0x7f05e8108c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:09.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.819+0000 7f05e7e00640 1 --2- 172.21.15.28:0/1074281497 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05e8104270 0x7f05e8104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:09.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.819+0000 7f05e7e00640 1 -- 172.21.15.28:0/1074281497 >> 172.21.15.28:0/1074281497 conn(0x7f05e80ffa90 msgr2=0x7f05e8101eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:59:09.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.819+0000 7f05e7e00640 1 -- 172.21.15.28:0/1074281497 shutdown_connections 2024-04-23T14:59:09.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.819+0000 7f05e7e00640 1 -- 172.21.15.28:0/1074281497 wait complete. 2024-04-23T14:59:09.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.820+0000 7f05e7e00640 1 Processor -- start 2024-04-23T14:59:09.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.820+0000 7f05e7e00640 1 -- start start 2024-04-23T14:59:09.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.820+0000 7f05e7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05e8104270 0x7f05e81a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:09.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.820+0000 7f05e7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05e8109340 0x7f05e81a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:09.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.820+0000 7f05e7e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05e819a6f0 0x7f05e819ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:09.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.820+0000 7f05e7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f05e8114450 con 0x7f05e8109340 2024-04-23T14:59:09.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.820+0000 7f05e7e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f05e81142d0 con 0x7f05e8104270 2024-04-23T14:59:09.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.820+0000 7f05e7e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f05e81145d0 con 0x7f05e819a6f0 2024-04-23T14:59:09.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.821+0000 7f05e6a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05e8104270 0x7f05e81a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:09.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.821+0000 7f05e6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05e8109340 0x7f05e81a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:09.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.821+0000 7f05e6a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05e8104270 0x7f05e81a0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48056/0 (socket says 172.21.15.28:48056) 2024-04-23T14:59:09.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.821+0000 7f05e6a00640 1 -- 172.21.15.28:0/2866392013 learned_addr learned my addr 172.21.15.28:0/2866392013 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:59:09.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.821+0000 7f05e7400640 1 --2- 172.21.15.28:0/2866392013 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05e819a6f0 0x7f05e819ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:09.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.821+0000 7f05e6a00640 1 -- 172.21.15.28:0/2866392013 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05e819a6f0 msgr2=0x7f05e819ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:09.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.821+0000 7f05e6a00640 1 --2- 172.21.15.28:0/2866392013 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05e819a6f0 0x7f05e819ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:09.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.821+0000 7f05e6a00640 1 -- 172.21.15.28:0/2866392013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05e8109340 msgr2=0x7f05e81a0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:09.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.821+0000 7f05e6a00640 1 --2- 172.21.15.28:0/2866392013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05e8109340 0x7f05e81a0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:09.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.821+0000 7f05e6a00640 1 -- 172.21.15.28:0/2866392013 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05e819b2f0 con 0x7f05e8104270 2024-04-23T14:59:09.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.821+0000 7f05e7400640 1 --2- 172.21.15.28:0/2866392013 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05e819a6f0 0x7f05e819ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T14:59:09.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.821+0000 7f05e6a00640 1 --2- 172.21.15.28:0/2866392013 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05e8104270 0x7f05e81a0630 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f05d800eb90 tx=0x7f05d800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:59:09.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.822+0000 7f05df400640 1 -- 172.21.15.28:0/2866392013 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d8004d20 con 0x7f05e8104270 2024-04-23T14:59:09.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.822+0000 7f05df400640 1 -- 172.21.15.28:0/2866392013 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05d800ee90 con 0x7f05e8104270 2024-04-23T14:59:09.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.822+0000 7f05df400640 1 -- 172.21.15.28:0/2866392013 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d8010830 con 0x7f05e8104270 2024-04-23T14:59:09.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.822+0000 7f05e7e00640 1 -- 172.21.15.28:0/2866392013 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05e81a73d0 con 0x7f05e8104270 2024-04-23T14:59:09.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.822+0000 7f05e7e00640 1 -- 172.21.15.28:0/2866392013 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05e81a7780 con 0x7f05e8104270 2024-04-23T14:59:09.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.823+0000 7f05e7e00640 1 -- 172.21.15.28:0/2866392013 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05e8105740 con 0x7f05e8104270 2024-04-23T14:59:09.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.825+0000 7f05df400640 1 -- 172.21.15.28:0/2866392013 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f05d8005020 con 0x7f05e8104270 2024-04-23T14:59:09.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.826+0000 7f05df400640 1 --2- 172.21.15.28:0/2866392013 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f05cc0a3da0 0x7f05cc0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:09.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.826+0000 7f05df400640 1 -- 172.21.15.28:0/2866392013 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8542+0+0 (secure 0 0 0) 0x7f05d80d0ef0 con 0x7f05e8104270 2024-04-23T14:59:09.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.826+0000 7f05e6000640 1 --2- 172.21.15.28:0/2866392013 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f05cc0a3da0 0x7f05cc0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:09.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.827+0000 7f05e6000640 1 --2- 172.21.15.28:0/2866392013 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f05cc0a3da0 0x7f05cc0a6260 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f05e819bf40 tx=0x7f05d4005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:59:09.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:09.830+0000 7f05df400640 1 -- 172.21.15.28:0/2866392013 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f05d809d520 con 0x7f05e8104270 2024-04-23T14:59:10.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.069+0000 7f05e7e00640 1 -- 172.21.15.28:0/2866392013 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f05e8104c80 con 0x7f05e8104270 2024-04-23T14:59:10.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.071+0000 7f05df400640 1 -- 172.21.15.28:0/2866392013 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) ==== 74+0+23188 (secure 0 0 0) 0x7f05d809ce20 con 0x7f05e8104270 2024-04-23T14:59:10.072 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-23T14:59:10.072 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":105,"fsid":"1ee6e996-017e-11ef-bc93-c7b262605968","created":"2024-04-23T14:32:34.581082+0000","modified":"2024-04-23T14:59:09.979139+0000","last_up_change":"2024-04-23T14:40:43.375846+0000","last_in_change":"2024-04-23T14:40:23.576939+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T14:36:27.698112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T14:41:37.030341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T14:41:38.444742+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":25,"snap_epoch":105,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e0b427bc-be65-4543-99cc-2f53ad98ddad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6803","nonce":2993445336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6805","nonce":2993445336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6809","nonce":2993445336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2993445336},{"type":"v1","addr":"172.21.15.28:6807","nonce":2993445336}]},"public_addr":"172.21.15.28:6803/2993445336","cluster_addr":"172.21.15.28:6805/2993445336","heartbeat_back_addr":"172.21.15.28:6809/2993445336","heartbeat_front_addr":"172.21.15.28:6807/2993445336","state":["exists","up"]},{"osd":1,"uuid":"ef527ec9-1bf3-4bb0-8df3-f5a015af557c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6801","nonce":4024093383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6803","nonce":4024093383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6807","nonce":4024093383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":4024093383},{"type":"v1","addr":"172.21.15.38:6805","nonce":4024093383}]},"public_addr":"172.21.15.38:6801/4024093383","cluster_addr":"172.21.15.38:6803/4024093383","heartbeat_back_addr":"172.21.15.38:6807/4024093383","heartbeat_front_addr":"172.21.15.38:6805/4024093383","state":["exists","up"]},{"osd":2,"uuid":"b33b23d9-adce-4c11-b304-2e117bdf6bfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6801","nonce":570739029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6803","nonce":570739029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6807","nonce":570739029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":570739029},{"type":"v1","addr":"172.21.15.105:6805","nonce":570739029}]},"public_addr":"172.21.15.105:6801/570739029","cluster_addr":"172.21.15.105:6803/570739029","heartbeat_back_addr":"172.21.15.105:6807/570739029","heartbeat_front_addr":"172.21.15.105:6805/570739029","state":["exists","up"]},{"osd":3,"uuid":"28bd5608-b426-434e-9db1-bdaa9828bdb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6811","nonce":2480400247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6813","nonce":2480400247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6817","nonce":2480400247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2480400247},{"type":"v1","addr":"172.21.15.28:6815","nonce":2480400247}]},"public_addr":"172.21.15.28:6811/2480400247","cluster_addr":"172.21.15.28:6813/2480400247","heartbeat_back_addr":"172.21.15.28:6817/2480400247","heartbeat_front_addr":"172.21.15.28:6815/2480400247","state":["exists","up"]},{"osd":4,"uuid":"e5d1c89b-025f-4d57-8807-59d75c787b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6809","nonce":3772189246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6811","nonce":3772189246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6815","nonce":3772189246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3772189246},{"type":"v1","addr":"172.21.15.38:6813","nonce":3772189246}]},"public_addr":"172.21.15.38:6809/3772189246","cluster_addr":"172.21.15.38:6811/3772189246","heartbeat_back_addr":"172.21.15.38:6815/3772189246","heartbeat_front_addr":"172.21.15.38:6813/3772189246","state":["exists","up"]},{"osd":5,"uuid":"9cf9cf4c-aafd-4ed3-bfb1-0527710a0292","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6809","nonce":928755894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6811","nonce":928755894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6815","nonce":928755894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":928755894},{"type":"v1","addr":"172.21.15.105:6813","nonce":928755894}]},"public_addr":"172.21.15.105:6809/928755894","cluster_addr":"172.21.15.105:6811/928755894","heartbeat_back_addr":"172.21.15.105:6815/928755894","heartbeat_front_addr":"172.21.15.105:6813/928755894","state":["exists","up"]},{"osd":6,"uuid":"304db525-d24d-4687-bfa0-f92bdf66d1e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6819","nonce":897979441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6821","nonce":897979441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6825","nonce":897979441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":897979441},{"type":"v1","addr":"172.21.15.28:6823","nonce":897979441}]},"public_addr":"172.21.15.28:6819/897979441","cluster_addr":"172.21.15.28:6821/897979441","heartbeat_back_addr":"172.21.15.28:6825/897979441","heartbeat_front_addr":"172.21.15.28:6823/897979441","state":["exists","up"]},{"osd":7,"uuid":"08e01d59-03f3-47b1-b45f-35ec5a50992c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6817","nonce":2400251786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6819","nonce":2400251786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6823","nonce":2400251786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2400251786},{"type":"v1","addr":"172.21.15.38:6821","nonce":2400251786}]},"public_addr":"172.21.15.38:6817/2400251786","cluster_addr":"172.21.15.38:6819/2400251786","heartbeat_back_addr":"172.21.15.38:6823/2400251786","heartbeat_front_addr":"172.21.15.38:6821/2400251786","state":["exists","up"]},{"osd":8,"uuid":"64ee1e95-0756-48b1-9c00-472d0c3cbd29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6817","nonce":1682312269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6819","nonce":1682312269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6823","nonce":1682312269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":1682312269},{"type":"v1","addr":"172.21.15.105:6821","nonce":1682312269}]},"public_addr":"172.21.15.105:6817/1682312269","cluster_addr":"172.21.15.105:6819/1682312269","heartbeat_back_addr":"172.21.15.105:6823/1682312269","heartbeat_front_addr":"172.21.15.105:6821/1682312269","state":["exists","up"]},{"osd":9,"uuid":"18f60893-3ffd-48c2-9db5-a67d35611d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6827","nonce":1292942516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6829","nonce":1292942516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6833","nonce":1292942516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1292942516},{"type":"v1","addr":"172.21.15.28:6831","nonce":1292942516}]},"public_addr":"172.21.15.28:6827/1292942516","cluster_addr":"172.21.15.28:6829/1292942516","heartbeat_back_addr":"172.21.15.28:6833/1292942516","heartbeat_front_addr":"172.21.15.28:6831/1292942516","state":["exists","up"]},{"osd":10,"uuid":"76c06725-7040-4118-8894-fe808a87f999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6825","nonce":2444616468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6827","nonce":2444616468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6831","nonce":2444616468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2444616468},{"type":"v1","addr":"172.21.15.38:6829","nonce":2444616468}]},"public_addr":"172.21.15.38:6825/2444616468","cluster_addr":"172.21.15.38:6827/2444616468","heartbeat_back_addr":"172.21.15.38:6831/2444616468","heartbeat_front_addr":"172.21.15.38:6829/2444616468","state":["exists","up"]},{"osd":11,"uuid":"e8fc9454-5540-4e2e-91bf-3ba251dbd1a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6825","nonce":3400299562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6827","nonce":3400299562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6831","nonce":3400299562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3400299562},{"type":"v1","addr":"172.21.15.105:6829","nonce":3400299562}]},"public_addr":"172.21.15.105:6825/3400299562","cluster_addr":"172.21.15.105:6827/3400299562","heartbeat_back_addr":"172.21.15.105:6831/3400299562","heartbeat_front_addr":"172.21.15.105:6829/3400299562","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:35:33.615395+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:00.545293+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:24.940211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:36:57.684887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:24.519061+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:37:49.010804+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:22.638478+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:38:49.559104+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:14.011771+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:39:47.859950+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:16.298445+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T14:40:41.619069+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.2d","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/38494154":"2024-04-24T14:44:01.286264+0000","172.21.15.28:0/905362079":"2024-04-24T14:44:01.286264+0000","172.21.15.28:6801/1997136173":"2024-04-24T14:44:01.286264+0000","172.21.15.28:6800/1997136173":"2024-04-24T14:44:01.286264+0000","172.21.15.28:0/590495740":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/1163172512":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/4290328460":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/4145020172":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/1997097543":"2024-04-24T14:44:01.286264+0000","172.21.15.28:0/3847772687":"2024-04-24T14:44:01.286264+0000","172.21.15.28:6800/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:6800/2222038555":"2024-04-24T14:33:41.530295+0000","172.21.15.28:0/720626958":"2024-04-24T14:33:41.530295+0000","172.21.15.28:6801/1307025091":"2024-04-24T14:33:07.576363+0000","172.21.15.28:0/948450262":"2024-04-24T14:33:07.576363+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":25,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":25,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T14:59:10.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.073+0000 7f05e7e00640 1 -- 172.21.15.28:0/2866392013 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f05cc0a3da0 msgr2=0x7f05cc0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:10.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.073+0000 7f05e7e00640 1 --2- 172.21.15.28:0/2866392013 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f05cc0a3da0 0x7f05cc0a6260 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f05e819bf40 tx=0x7f05d4005f90 comp rx=0 tx=0).stop 2024-04-23T14:59:10.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.073+0000 7f05e7e00640 1 -- 172.21.15.28:0/2866392013 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05e8104270 msgr2=0x7f05e81a0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:10.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.073+0000 7f05e7e00640 1 --2- 172.21.15.28:0/2866392013 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05e8104270 0x7f05e81a0630 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f05d800eb90 tx=0x7f05d800c6a0 comp rx=0 tx=0).stop 2024-04-23T14:59:10.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.073+0000 7f05e7e00640 1 -- 172.21.15.28:0/2866392013 shutdown_connections 2024-04-23T14:59:10.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.073+0000 7f05e7e00640 1 --2- 172.21.15.28:0/2866392013 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f05cc0a3da0 0x7f05cc0a6260 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:10.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.073+0000 7f05e7e00640 1 --2- 172.21.15.28:0/2866392013 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05e819a6f0 0x7f05e819ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:10.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.073+0000 7f05e7e00640 1 --2- 172.21.15.28:0/2866392013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05e8109340 0x7f05e81a0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:10.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.073+0000 7f05e7e00640 1 --2- 172.21.15.28:0/2866392013 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05e8104270 0x7f05e81a0630 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:10.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.073+0000 7f05e7e00640 1 -- 172.21.15.28:0/2866392013 >> 172.21.15.28:0/2866392013 conn(0x7f05e80ffa90 msgr2=0x7f05e8100520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:59:10.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.073+0000 7f05e7e00640 1 -- 172.21.15.28:0/2866392013 shutdown_connections 2024-04-23T14:59:10.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.073+0000 7f05e7e00640 1 -- 172.21.15.28:0/2866392013 wait complete. 2024-04-23T14:59:10.091 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-23T14:59:10.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.269+0000 7fcd0d200640 1 -- 172.21.15.28:0/1237252582 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd08109580 msgr2=0x7fcd08111150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:10.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.269+0000 7fcd0d200640 1 --2- 172.21.15.28:0/1237252582 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd08109580 0x7fcd08111150 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fcd00009860 tx=0x7fcd0002a340 comp rx=0 tx=0).stop 2024-04-23T14:59:10.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.269+0000 7fcd0d200640 1 -- 172.21.15.28:0/1237252582 shutdown_connections 2024-04-23T14:59:10.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.269+0000 7fcd0d200640 1 --2- 172.21.15.28:0/1237252582 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd08109580 0x7fcd08111150 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:10.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.269+0000 7fcd0d200640 1 --2- 172.21.15.28:0/1237252582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd08104e60 0x7fcd08108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:10.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.269+0000 7fcd0d200640 1 --2- 172.21.15.28:0/1237252582 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd081044b0 0x7fcd08104890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:10.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.269+0000 7fcd0d200640 1 -- 172.21.15.28:0/1237252582 >> 172.21.15.28:0/1237252582 conn(0x7fcd080ffd90 msgr2=0x7fcd081021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:59:10.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.269+0000 7fcd0d200640 1 -- 172.21.15.28:0/1237252582 shutdown_connections 2024-04-23T14:59:10.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.269+0000 7fcd0d200640 1 -- 172.21.15.28:0/1237252582 wait complete. 2024-04-23T14:59:10.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.270+0000 7fcd0d200640 1 Processor -- start 2024-04-23T14:59:10.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.270+0000 7fcd0d200640 1 -- start start 2024-04-23T14:59:10.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.270+0000 7fcd0d200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd081044b0 0x7fcd081a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:10.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.270+0000 7fcd0d200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd08109580 0x7fcd081a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:10.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.270+0000 7fcd0d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd0819a980 0x7fcd0819ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:10.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.270+0000 7fcd0d200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcd08114620 con 0x7fcd0819a980 2024-04-23T14:59:10.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.270+0000 7fcd0d200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcd081144a0 con 0x7fcd081044b0 2024-04-23T14:59:10.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.270+0000 7fcd0d200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcd081147a0 con 0x7fcd08109580 2024-04-23T14:59:10.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.271+0000 7fcd05e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd081044b0 0x7fcd081a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:10.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.271+0000 7fcd05e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd081044b0 0x7fcd081a07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.28:48088/0 (socket says 172.21.15.28:48088) 2024-04-23T14:59:10.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.271+0000 7fcd05e00640 1 -- 172.21.15.28:0/2797513101 learned_addr learned my addr 172.21.15.28:0/2797513101 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:59:10.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.271+0000 7fcd05400640 1 --2- 172.21.15.28:0/2797513101 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd08109580 0x7fcd081a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:10.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.271+0000 7fcd06800640 1 --2- 172.21.15.28:0/2797513101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd0819a980 0x7fcd0819ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:10.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.271+0000 7fcd05e00640 1 -- 172.21.15.28:0/2797513101 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd08109580 msgr2=0x7fcd081a0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:10.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.271+0000 7fcd05e00640 1 --2- 172.21.15.28:0/2797513101 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd08109580 0x7fcd081a0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:10.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.271+0000 7fcd05e00640 1 -- 172.21.15.28:0/2797513101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd0819a980 msgr2=0x7fcd0819ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:10.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.271+0000 7fcd05e00640 1 --2- 172.21.15.28:0/2797513101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd0819a980 0x7fcd0819ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:10.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.271+0000 7fcd05e00640 1 -- 172.21.15.28:0/2797513101 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd0819b4a0 con 0x7fcd081044b0 2024-04-23T14:59:10.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.271+0000 7fcd05e00640 1 --2- 172.21.15.28:0/2797513101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd081044b0 0x7fcd081a07d0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fccf800ebc0 tx=0x7fccf800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:59:10.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.271+0000 7fccfea00640 1 -- 172.21.15.28:0/2797513101 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf8010430 con 0x7fcd081044b0 2024-04-23T14:59:10.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.272+0000 7fcd0d200640 1 -- 172.21.15.28:0/2797513101 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd0819b790 con 0x7fcd081044b0 2024-04-23T14:59:10.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.272+0000 7fccfea00640 1 -- 172.21.15.28:0/2797513101 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fccf8010ab0 con 0x7fcd081044b0 2024-04-23T14:59:10.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.272+0000 7fccfea00640 1 -- 172.21.15.28:0/2797513101 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf8018b30 con 0x7fcd081044b0 2024-04-23T14:59:10.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.272+0000 7fcd0d200640 1 -- 172.21.15.28:0/2797513101 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd081a7a30 con 0x7fcd081044b0 2024-04-23T14:59:10.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.273+0000 7fcd0d200640 1 -- 172.21.15.28:0/2797513101 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd08069270 con 0x7fcd081044b0 2024-04-23T14:59:10.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.275+0000 7fccfea00640 1 -- 172.21.15.28:0/2797513101 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fccf8018cd0 con 0x7fcd081044b0 2024-04-23T14:59:10.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.276+0000 7fccfea00640 1 --2- 172.21.15.28:0/2797513101 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fccec0a3da0 0x7fccec0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:10.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.276+0000 7fccfea00640 1 -- 172.21.15.28:0/2797513101 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8586+0+0 (secure 0 0 0) 0x7fccf80d0f10 con 0x7fcd081044b0 2024-04-23T14:59:10.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.276+0000 7fcd05400640 1 --2- 172.21.15.28:0/2797513101 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fccec0a3da0 0x7fccec0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:10.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.277+0000 7fcd05400640 1 --2- 172.21.15.28:0/2797513101 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fccec0a3da0 0x7fccec0a6260 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fcd081034c0 tx=0x7fccf0009260 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:59:10.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.280+0000 7fccfea00640 1 -- 172.21.15.28:0/2797513101 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fccf809d510 con 0x7fcd081044b0 2024-04-23T14:59:10.285 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:09 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3611828537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:59:10.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:09 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/3329267624' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:59:10.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:09 smithi105 ceph-mon[32555]: pgmap v473: 97 pgs: 97 active+clean; 849 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 62 KiB/s wr, 20 op/s 2024-04-23T14:59:10.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:09 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3611828537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:59:10.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:09 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/3329267624' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:59:10.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:09 smithi038 ceph-mon[32348]: pgmap v473: 97 pgs: 97 active+clean; 849 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 62 KiB/s wr, 20 op/s 2024-04-23T14:59:10.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:09 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3611828537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:59:10.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:09 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/3329267624' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T14:59:10.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:09 smithi028 ceph-mon[27810]: pgmap v473: 97 pgs: 97 active+clean; 849 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 62 KiB/s wr, 20 op/s 2024-04-23T14:59:10.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.517+0000 7fcd0d200640 1 -- 172.21.15.28:0/2797513101 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fcd0819beb0 con 0x7fcd081044b0 2024-04-23T14:59:10.530 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-23T14:59:10.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.530+0000 7fccfea00640 1 -- 172.21.15.28:0/2797513101 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v21) ==== 161+0+0 (secure 0 0 0) 0x7fccf809ce10 con 0x7fcd081044b0 2024-04-23T14:59:10.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.533+0000 7fcd0d200640 1 -- 172.21.15.28:0/2797513101 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fccec0a3da0 msgr2=0x7fccec0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:10.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.533+0000 7fcd0d200640 1 --2- 172.21.15.28:0/2797513101 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fccec0a3da0 0x7fccec0a6260 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fcd081034c0 tx=0x7fccf0009260 comp rx=0 tx=0).stop 2024-04-23T14:59:10.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.533+0000 7fcd0d200640 1 -- 172.21.15.28:0/2797513101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd081044b0 msgr2=0x7fcd081a07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:10.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.533+0000 7fcd0d200640 1 --2- 172.21.15.28:0/2797513101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd081044b0 0x7fcd081a07d0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fccf800ebc0 tx=0x7fccf800c2d0 comp rx=0 tx=0).stop 2024-04-23T14:59:10.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.533+0000 7fcd0d200640 1 -- 172.21.15.28:0/2797513101 shutdown_connections 2024-04-23T14:59:10.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.533+0000 7fcd0d200640 1 --2- 172.21.15.28:0/2797513101 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fccec0a3da0 0x7fccec0a6260 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:10.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.533+0000 7fcd0d200640 1 --2- 172.21.15.28:0/2797513101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd0819a980 0x7fcd0819ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:10.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.533+0000 7fcd0d200640 1 --2- 172.21.15.28:0/2797513101 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd08109580 0x7fcd081a0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:10.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.533+0000 7fcd0d200640 1 --2- 172.21.15.28:0/2797513101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd081044b0 0x7fcd081a07d0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:10.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.533+0000 7fcd0d200640 1 -- 172.21.15.28:0/2797513101 >> 172.21.15.28:0/2797513101 conn(0x7fcd080ffd90 msgr2=0x7fcd081020e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:59:10.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.533+0000 7fcd0d200640 1 -- 172.21.15.28:0/2797513101 shutdown_connections 2024-04-23T14:59:10.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.533+0000 7fcd0d200640 1 -- 172.21.15.28:0/2797513101 wait complete. 2024-04-23T14:59:10.551 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-23T14:59:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.732+0000 7fdc13200640 1 -- 172.21.15.28:0/483989743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc14103080 msgr2=0x7fdc14103460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.732+0000 7fdc13200640 1 --2- 172.21.15.28:0/483989743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc14103080 0x7fdc14103460 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fdc04009900 tx=0x7fdc0402a480 comp rx=0 tx=0).stop 2024-04-23T14:59:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.732+0000 7fdc13200640 1 -- 172.21.15.28:0/483989743 shutdown_connections 2024-04-23T14:59:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.732+0000 7fdc13200640 1 --2- 172.21.15.28:0/483989743 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdc1410c4a0 0x7fdc1410e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.732+0000 7fdc13200640 1 --2- 172.21.15.28:0/483989743 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc141039a0 0x7fdc1410be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:10.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.732+0000 7fdc13200640 1 --2- 172.21.15.28:0/483989743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc14103080 0x7fdc14103460 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:10.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.732+0000 7fdc13200640 1 -- 172.21.15.28:0/483989743 >> 172.21.15.28:0/483989743 conn(0x7fdc140fc8c0 msgr2=0x7fdc140fece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:59:10.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.732+0000 7fdc13200640 1 -- 172.21.15.28:0/483989743 shutdown_connections 2024-04-23T14:59:10.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.732+0000 7fdc13200640 1 -- 172.21.15.28:0/483989743 wait complete. 2024-04-23T14:59:10.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.733+0000 7fdc13200640 1 Processor -- start 2024-04-23T14:59:10.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.733+0000 7fdc13200640 1 -- start start 2024-04-23T14:59:10.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.733+0000 7fdc13200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdc14103080 0x7fdc14100cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:10.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.734+0000 7fdc13200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc141039a0 0x7fdc14101210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:10.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.734+0000 7fdc13200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc1410c4a0 0x7fdc14101750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:10.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.734+0000 7fdc13200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdc14112260 con 0x7fdc141039a0 2024-04-23T14:59:10.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.734+0000 7fdc13200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdc141120e0 con 0x7fdc14103080 2024-04-23T14:59:10.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.734+0000 7fdc13200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdc141123e0 con 0x7fdc1410c4a0 2024-04-23T14:59:10.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.734+0000 7fdc09400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc141039a0 0x7fdc14101210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:10.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.734+0000 7fdc11e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdc14103080 0x7fdc14100cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:10.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.734+0000 7fdc09400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc141039a0 0x7fdc14101210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40740/0 (socket says 172.21.15.28:40740) 2024-04-23T14:59:10.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.734+0000 7fdc09400640 1 -- 172.21.15.28:0/4172562896 learned_addr learned my addr 172.21.15.28:0/4172562896 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:59:10.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.734+0000 7fdc12800640 1 --2- 172.21.15.28:0/4172562896 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc1410c4a0 0x7fdc14101750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:10.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.734+0000 7fdc09400640 1 -- 172.21.15.28:0/4172562896 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc1410c4a0 msgr2=0x7fdc14101750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.734+0000 7fdc09400640 1 --2- 172.21.15.28:0/4172562896 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc1410c4a0 0x7fdc14101750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.734+0000 7fdc09400640 1 -- 172.21.15.28:0/4172562896 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdc14103080 msgr2=0x7fdc14100cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.734+0000 7fdc09400640 1 --2- 172.21.15.28:0/4172562896 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdc14103080 0x7fdc14100cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.734+0000 7fdc09400640 1 -- 172.21.15.28:0/4172562896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc141a6ca0 con 0x7fdc141039a0 2024-04-23T14:59:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.734+0000 7fdc09400640 1 --2- 172.21.15.28:0/4172562896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc141039a0 0x7fdc14101210 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc00e9f0 tx=0x7fdbfc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:59:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.735+0000 7fdc0b400640 1 -- 172.21.15.28:0/4172562896 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbfc00ccc0 con 0x7fdc141039a0 2024-04-23T14:59:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.735+0000 7fdc0b400640 1 -- 172.21.15.28:0/4172562896 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbfc010910 con 0x7fdc141039a0 2024-04-23T14:59:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.735+0000 7fdc13200640 1 -- 172.21.15.28:0/4172562896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc141a6f90 con 0x7fdc141039a0 2024-04-23T14:59:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.735+0000 7fdc0b400640 1 -- 172.21.15.28:0/4172562896 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbfc018920 con 0x7fdc141039a0 2024-04-23T14:59:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.735+0000 7fdc13200640 1 -- 172.21.15.28:0/4172562896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc141a7450 con 0x7fdc141039a0 2024-04-23T14:59:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.736+0000 7fdc13200640 1 -- 172.21.15.28:0/4172562896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc14069190 con 0x7fdc141039a0 2024-04-23T14:59:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.738+0000 7fdc0b400640 1 -- 172.21.15.28:0/4172562896 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fdbfc010430 con 0x7fdc141039a0 2024-04-23T14:59:10.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.739+0000 7fdc0b400640 1 --2- 172.21.15.28:0/4172562896 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fdbf40a3d20 0x7fdbf40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:10.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.739+0000 7fdc0b400640 1 -- 172.21.15.28:0/4172562896 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8628+0+0 (secure 0 0 0) 0x7fdbfc0d0b90 con 0x7fdc141039a0 2024-04-23T14:59:10.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.740+0000 7fdc11e00640 1 --2- 172.21.15.28:0/4172562896 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fdbf40a3d20 0x7fdbf40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.740+0000 7fdc11e00640 1 --2- 172.21.15.28:0/4172562896 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fdbf40a3d20 0x7fdbf40a61e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fdc04009230 tx=0x7fdc040058d0 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:59:10.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.744+0000 7fdc0b400640 1 -- 172.21.15.28:0/4172562896 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdbfc09d160 con 0x7fdc141039a0 2024-04-23T14:59:10.875 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 23 14:59:10 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-b[63031]: ignoring --setuser ceph since I am not root 2024-04-23T14:59:10.875 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 23 14:59:10 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-b[63031]: ignoring --setgroup ceph since I am not root 2024-04-23T14:59:10.875 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 23 14:59:10 smithi038 ceph-mds[63035]: -- 172.21.15.38:0/361192566 <== mon.2 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55eed8ef1a40 con 0x55eed8ec4c00 2024-04-23T14:59:10.875 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 23 14:59:10 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-b[63031]: starting mds.b at 2024-04-23T14:59:10.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:10 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mon-a[27806]: 2024-04-23T14:59:10.519+0000 7f19f5600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-23T14:59:10.876 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 23 14:59:10 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-g[67278]: ignoring --setuser ceph since I am not root 2024-04-23T14:59:10.876 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 23 14:59:10 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-g[67278]: ignoring --setgroup ceph since I am not root 2024-04-23T14:59:10.877 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 23 14:59:10 smithi028 ceph-mds[67287]: -- 172.21.15.28:0/1976339779 <== mon.1 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5645b7e6da40 con 0x5645b7e40800 2024-04-23T14:59:10.877 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 23 14:59:10 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-g[67278]: starting mds.g at 2024-04-23T14:59:10.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:10.984+0000 7fdc13200640 1 -- 172.21.15.28:0/4172562896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fdc14107540 con 0x7fdc141039a0 2024-04-23T14:59:11.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:10 smithi105 ceph-mon[32555]: osdmap e105: 12 total, 12 up, 12 in 2024-04-23T14:59:11.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:10 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2866392013' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:59:11.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:10 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/2797513101' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-23T14:59:11.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:10 smithi105 ceph-mon[32555]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-23T14:59:11.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:10 smithi105 ceph-mon[32555]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-23T14:59:11.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:10 smithi105 ceph-mon[32555]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-23T14:59:11.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:10 smithi105 ceph-mon[32555]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-23T14:59:11.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:10 smithi105 ceph-mon[32555]: osdmap e106: 12 total, 12 up, 12 in 2024-04-23T14:59:11.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:10 smithi105 ceph-mon[32555]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-04-23T14:59:11.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:10 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/4172562896' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T14:59:11.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:10 smithi038 ceph-mon[32348]: osdmap e105: 12 total, 12 up, 12 in 2024-04-23T14:59:11.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:10 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2866392013' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:59:11.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:10 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/2797513101' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-23T14:59:11.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:10 smithi038 ceph-mon[32348]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-23T14:59:11.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:10 smithi038 ceph-mon[32348]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-23T14:59:11.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:10 smithi038 ceph-mon[32348]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-23T14:59:11.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:10 smithi038 ceph-mon[32348]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-23T14:59:11.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:10 smithi038 ceph-mon[32348]: osdmap e106: 12 total, 12 up, 12 in 2024-04-23T14:59:11.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:10 smithi038 ceph-mon[32348]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-04-23T14:59:11.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:10 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/4172562896' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T14:59:11.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:10 smithi028 ceph-mon[27810]: osdmap e105: 12 total, 12 up, 12 in 2024-04-23T14:59:11.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:10 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2866392013' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T14:59:11.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:10 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/2797513101' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-23T14:59:11.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:10 smithi028 ceph-mon[27810]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-23T14:59:11.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:10 smithi028 ceph-mon[27810]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-23T14:59:11.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:10 smithi028 ceph-mon[27810]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-23T14:59:11.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:10 smithi028 ceph-mon[27810]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-23T14:59:11.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:10 smithi028 ceph-mon[27810]: osdmap e106: 12 total, 12 up, 12 in 2024-04-23T14:59:11.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:10 smithi028 ceph-mon[27810]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-04-23T14:59:11.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:10 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/4172562896' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T14:59:11.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.525+0000 7fdc0b400640 1 -- 172.21.15.28:0/4172562896 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v22) ==== 104+0+0 (secure 0 0 0) 0x7fdbfc09ca60 con 0x7fdc141039a0 2024-04-23T14:59:11.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.529+0000 7fdc13200640 1 -- 172.21.15.28:0/4172562896 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fdbf40a3d20 msgr2=0x7fdbf40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:11.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.529+0000 7fdc13200640 1 --2- 172.21.15.28:0/4172562896 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fdbf40a3d20 0x7fdbf40a61e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fdc04009230 tx=0x7fdc040058d0 comp rx=0 tx=0).stop 2024-04-23T14:59:11.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.530+0000 7fdc13200640 1 -- 172.21.15.28:0/4172562896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc141039a0 msgr2=0x7fdc14101210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:11.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.530+0000 7fdc13200640 1 --2- 172.21.15.28:0/4172562896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc141039a0 0x7fdc14101210 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc00e9f0 tx=0x7fdbfc00eeb0 comp rx=0 tx=0).stop 2024-04-23T14:59:11.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.530+0000 7fdc13200640 1 -- 172.21.15.28:0/4172562896 shutdown_connections 2024-04-23T14:59:11.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.530+0000 7fdc13200640 1 --2- 172.21.15.28:0/4172562896 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fdbf40a3d20 0x7fdbf40a61e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:11.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.530+0000 7fdc13200640 1 --2- 172.21.15.28:0/4172562896 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc1410c4a0 0x7fdc14101750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:11.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.530+0000 7fdc13200640 1 --2- 172.21.15.28:0/4172562896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc141039a0 0x7fdc14101210 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:11.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.530+0000 7fdc13200640 1 --2- 172.21.15.28:0/4172562896 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdc14103080 0x7fdc14100cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:11.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.530+0000 7fdc13200640 1 -- 172.21.15.28:0/4172562896 >> 172.21.15.28:0/4172562896 conn(0x7fdc140fc8c0 msgr2=0x7fdc140fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:59:11.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.530+0000 7fdc13200640 1 -- 172.21.15.28:0/4172562896 shutdown_connections 2024-04-23T14:59:11.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.530+0000 7fdc13200640 1 -- 172.21.15.28:0/4172562896 wait complete. 2024-04-23T14:59:11.557 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-23T14:59:11.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.777+0000 7fd5ffc00640 1 -- 172.21.15.28:0/3115815356 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5f8103270 msgr2=0x7fd5f8103650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:11.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.777+0000 7fd5ffc00640 1 --2- 172.21.15.28:0/3115815356 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5f8103270 0x7fd5f8103650 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8009900 tx=0x7fd5e802a510 comp rx=0 tx=0).stop 2024-04-23T14:59:11.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.777+0000 7fd5ffc00640 1 -- 172.21.15.28:0/3115815356 shutdown_connections 2024-04-23T14:59:11.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.777+0000 7fd5ffc00640 1 --2- 172.21.15.28:0/3115815356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5f810c5b0 0x7fd5f810ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:11.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.777+0000 7fd5ffc00640 1 --2- 172.21.15.28:0/3115815356 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd5f8103b90 0x7fd5f810c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:11.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.777+0000 7fd5ffc00640 1 --2- 172.21.15.28:0/3115815356 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5f8103270 0x7fd5f8103650 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:11.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.777+0000 7fd5ffc00640 1 -- 172.21.15.28:0/3115815356 >> 172.21.15.28:0/3115815356 conn(0x7fd5f80780b0 msgr2=0x7fd5f80784c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:59:11.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.777+0000 7fd5ffc00640 1 -- 172.21.15.28:0/3115815356 shutdown_connections 2024-04-23T14:59:11.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.777+0000 7fd5ffc00640 1 -- 172.21.15.28:0/3115815356 wait complete. 2024-04-23T14:59:11.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.778+0000 7fd5ffc00640 1 Processor -- start 2024-04-23T14:59:11.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.778+0000 7fd5ffc00640 1 -- start start 2024-04-23T14:59:11.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.778+0000 7fd5ffc00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd5f8103270 0x7fd5f819e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:11.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.779+0000 7fd5ffc00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5f8103b90 0x7fd5f819ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:11.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.779+0000 7fd5ffc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5f810c5b0 0x7fd5f8198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:11.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.779+0000 7fd5ffc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f81123c0 con 0x7fd5f810c5b0 2024-04-23T14:59:11.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.779+0000 7fd5ffc00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f8112240 con 0x7fd5f8103b90 2024-04-23T14:59:11.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.779+0000 7fd5ffc00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f8112540 con 0x7fd5f8103270 2024-04-23T14:59:11.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.779+0000 7fd5fd200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd5f8103270 0x7fd5f819e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.779+0000 7fd5fdc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5f810c5b0 0x7fd5f8198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.779+0000 7fd5fd200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd5f8103270 0x7fd5f819e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.28:38038/0 (socket says 172.21.15.28:38038) 2024-04-23T14:59:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.779+0000 7fd5fd200640 1 -- 172.21.15.28:0/1415092112 learned_addr learned my addr 172.21.15.28:0/1415092112 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:59:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.779+0000 7fd5f7e00640 1 --2- 172.21.15.28:0/1415092112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5f8103b90 0x7fd5f819ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.779+0000 7fd5fd200640 1 -- 172.21.15.28:0/1415092112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5f8103b90 msgr2=0x7fd5f819ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.779+0000 7fd5fd200640 1 --2- 172.21.15.28:0/1415092112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5f8103b90 0x7fd5f819ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.779+0000 7fd5fd200640 1 -- 172.21.15.28:0/1415092112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5f810c5b0 msgr2=0x7fd5f8198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.779+0000 7fd5fd200640 1 --2- 172.21.15.28:0/1415092112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5f810c5b0 0x7fd5f8198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:11.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.779+0000 7fd5fd200640 1 -- 172.21.15.28:0/1415092112 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5f8199020 con 0x7fd5f8103270 2024-04-23T14:59:11.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.780+0000 7fd5fdc00640 1 --2- 172.21.15.28:0/1415092112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5f810c5b0 0x7fd5f8198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:59:11.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.780+0000 7fd5fd200640 1 --2- 172.21.15.28:0/1415092112 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd5f8103270 0x7fd5f819e6a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8004770 tx=0x7fd5e8003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:59:11.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.780+0000 7fd5f5600640 1 -- 172.21.15.28:0/1415092112 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e803f070 con 0x7fd5f8103270 2024-04-23T14:59:11.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.780+0000 7fd5ffc00640 1 -- 172.21.15.28:0/1415092112 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5f81992b0 con 0x7fd5f8103270 2024-04-23T14:59:11.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.780+0000 7fd5f5600640 1 -- 172.21.15.28:0/1415092112 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5e8031ca0 con 0x7fd5f8103270 2024-04-23T14:59:11.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.780+0000 7fd5f5600640 1 -- 172.21.15.28:0/1415092112 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e8035030 con 0x7fd5f8103270 2024-04-23T14:59:11.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.780+0000 7fd5ffc00640 1 -- 172.21.15.28:0/1415092112 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5f81a5440 con 0x7fd5f8103270 2024-04-23T14:59:11.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.781+0000 7fd5ffc00640 1 -- 172.21.15.28:0/1415092112 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5f8069990 con 0x7fd5f8103270 2024-04-23T14:59:11.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.784+0000 7fd5f5600640 1 -- 172.21.15.28:0/1415092112 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fd5e8042070 con 0x7fd5f8103270 2024-04-23T14:59:11.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.785+0000 7fd5f5600640 1 --2- 172.21.15.28:0/1415092112 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd5dc0a3d20 0x7fd5dc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:11.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.785+0000 7fd5f5600640 1 -- 172.21.15.28:0/1415092112 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8628+0+0 (secure 0 0 0) 0x7fd5e80f1290 con 0x7fd5f8103270 2024-04-23T14:59:11.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.785+0000 7fd5f7e00640 1 --2- 172.21.15.28:0/1415092112 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd5dc0a3d20 0x7fd5dc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:11.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.786+0000 7fd5f7e00640 1 --2- 172.21.15.28:0/1415092112 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd5dc0a3d20 0x7fd5dc0a61e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8199fb0 tx=0x7fd5e4009500 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:59:11.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.788+0000 7fd5f5600640 1 -- 172.21.15.28:0/1415092112 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd5e80bd2d0 con 0x7fd5f8103270 2024-04-23T14:59:11.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:11.996+0000 7fd5ffc00640 1 -- 172.21.15.28:0/1415092112 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd5f8107730 con 0x7fd5f8103270 2024-04-23T14:59:12.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:11 smithi105 ceph-mon[32555]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-23T14:59:12.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:11 smithi105 ceph-mon[32555]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-23T14:59:12.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:11 smithi105 ceph-mon[32555]: Cluster is now healthy 2024-04-23T14:59:12.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:11 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/4172562896' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-23T14:59:12.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:11 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-23T14:59:12.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:11 smithi105 ceph-mon[32555]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T14:59:12.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:11 smithi105 ceph-mon[32555]: mds.? [v2:172.21.15.38:6838/1922475164,v1:172.21.15.38:6839/1922475164] up:boot 2024-04-23T14:59:12.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:11 smithi105 ceph-mon[32555]: mds.? [v2:172.21.15.28:6838/2601332415,v1:172.21.15.28:6839/2601332415] up:boot 2024-04-23T14:59:12.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:11 smithi105 ceph-mon[32555]: fsmap 12 up:standby 2024-04-23T14:59:12.287 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:11 smithi105 ceph-mon[32555]: pgmap v476: 97 pgs: 97 active+clean; 849 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 7.7 KiB/s wr, 2 op/s 2024-04-23T14:59:12.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:11 smithi038 ceph-mon[32348]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-23T14:59:12.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:11 smithi038 ceph-mon[32348]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-23T14:59:12.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:11 smithi038 ceph-mon[32348]: Cluster is now healthy 2024-04-23T14:59:12.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:11 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/4172562896' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-23T14:59:12.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:11 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-23T14:59:12.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:11 smithi038 ceph-mon[32348]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T14:59:12.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:11 smithi038 ceph-mon[32348]: mds.? [v2:172.21.15.38:6838/1922475164,v1:172.21.15.38:6839/1922475164] up:boot 2024-04-23T14:59:12.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:11 smithi038 ceph-mon[32348]: mds.? [v2:172.21.15.28:6838/2601332415,v1:172.21.15.28:6839/2601332415] up:boot 2024-04-23T14:59:12.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:11 smithi038 ceph-mon[32348]: fsmap 12 up:standby 2024-04-23T14:59:12.373 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:11 smithi038 ceph-mon[32348]: pgmap v476: 97 pgs: 97 active+clean; 849 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 7.7 KiB/s wr, 2 op/s 2024-04-23T14:59:12.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:11 smithi028 ceph-mon[27810]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-23T14:59:12.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:11 smithi028 ceph-mon[27810]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-23T14:59:12.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:11 smithi028 ceph-mon[27810]: Cluster is now healthy 2024-04-23T14:59:12.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:11 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/4172562896' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-23T14:59:12.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:11 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-23T14:59:12.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:11 smithi028 ceph-mon[27810]: from='mgr.25153 172.21.15.28:0/4057174827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T14:59:12.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:11 smithi028 ceph-mon[27810]: mds.? [v2:172.21.15.38:6838/1922475164,v1:172.21.15.38:6839/1922475164] up:boot 2024-04-23T14:59:12.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:11 smithi028 ceph-mon[27810]: mds.? [v2:172.21.15.28:6838/2601332415,v1:172.21.15.28:6839/2601332415] up:boot 2024-04-23T14:59:12.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:11 smithi028 ceph-mon[27810]: fsmap 12 up:standby 2024-04-23T14:59:12.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:11 smithi028 ceph-mon[27810]: pgmap v476: 97 pgs: 97 active+clean; 849 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 7.7 KiB/s wr, 2 op/s 2024-04-23T14:59:13.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.011+0000 7fd5f5600640 1 -- 172.21.15.28:0/1415092112 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v107) ==== 181+0+0 (secure 0 0 0) 0x7fd5e80c15d0 con 0x7fd5f8103270 2024-04-23T14:59:13.013 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-04-23T14:59:13.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.016+0000 7fd5ffc00640 1 -- 172.21.15.28:0/1415092112 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd5dc0a3d20 msgr2=0x7fd5dc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:13.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.016+0000 7fd5ffc00640 1 --2- 172.21.15.28:0/1415092112 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd5dc0a3d20 0x7fd5dc0a61e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8199fb0 tx=0x7fd5e4009500 comp rx=0 tx=0).stop 2024-04-23T14:59:13.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.016+0000 7fd5ffc00640 1 -- 172.21.15.28:0/1415092112 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd5f8103270 msgr2=0x7fd5f819e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:13.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.016+0000 7fd5ffc00640 1 --2- 172.21.15.28:0/1415092112 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd5f8103270 0x7fd5f819e6a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8004770 tx=0x7fd5e8003c10 comp rx=0 tx=0).stop 2024-04-23T14:59:13.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.017+0000 7fd5ffc00640 1 -- 172.21.15.28:0/1415092112 shutdown_connections 2024-04-23T14:59:13.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.017+0000 7fd5ffc00640 1 --2- 172.21.15.28:0/1415092112 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7fd5dc0a3d20 0x7fd5dc0a61e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:13.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.017+0000 7fd5ffc00640 1 --2- 172.21.15.28:0/1415092112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5f8103b90 0x7fd5f819ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:13.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.017+0000 7fd5ffc00640 1 --2- 172.21.15.28:0/1415092112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5f810c5b0 0x7fd5f8198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:13.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.017+0000 7fd5ffc00640 1 --2- 172.21.15.28:0/1415092112 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd5f8103270 0x7fd5f819e6a0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:13.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.018+0000 7fd5ffc00640 1 -- 172.21.15.28:0/1415092112 >> 172.21.15.28:0/1415092112 conn(0x7fd5f80780b0 msgr2=0x7fd5f80ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:59:13.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.018+0000 7fd5ffc00640 1 -- 172.21.15.28:0/1415092112 shutdown_connections 2024-04-23T14:59:13.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.018+0000 7fd5ffc00640 1 -- 172.21.15.28:0/1415092112 wait complete. 2024-04-23T14:59:13.044 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-23T14:59:13.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.240+0000 7f0ec1e00640 1 -- 172.21.15.28:0/1371656002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc1072f0 msgr2=0x7f0ebc10eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:13.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.240+0000 7f0ec1e00640 1 --2- 172.21.15.28:0/1371656002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc1072f0 0x7f0ebc10eec0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0eb4009ca0 tx=0x7f0eb402a8b0 comp rx=0 tx=0).stop 2024-04-23T14:59:13.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.240+0000 7f0ec1e00640 1 -- 172.21.15.28:0/1371656002 shutdown_connections 2024-04-23T14:59:13.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.240+0000 7f0ec1e00640 1 --2- 172.21.15.28:0/1371656002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc1072f0 0x7f0ebc10eec0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:13.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.240+0000 7f0ec1e00640 1 --2- 172.21.15.28:0/1371656002 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ebc102bd0 0x7f0ebc106c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:13.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.240+0000 7f0ec1e00640 1 --2- 172.21.15.28:0/1371656002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ebc102220 0x7f0ebc102600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:13.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.240+0000 7f0ec1e00640 1 -- 172.21.15.28:0/1371656002 >> 172.21.15.28:0/1371656002 conn(0x7f0ebc079450 msgr2=0x7f0ebc079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:59:13.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.241+0000 7f0ec1e00640 1 -- 172.21.15.28:0/1371656002 shutdown_connections 2024-04-23T14:59:13.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.241+0000 7f0ec1e00640 1 -- 172.21.15.28:0/1371656002 wait complete. 2024-04-23T14:59:13.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.242+0000 7f0ec1e00640 1 Processor -- start 2024-04-23T14:59:13.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.242+0000 7f0ec1e00640 1 -- start start 2024-04-23T14:59:13.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.242+0000 7f0ec1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc102220 0x7f0ebc108f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:13.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.242+0000 7f0ec1e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ebc1072f0 0x7f0ebc109480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:13.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.242+0000 7f0ec1e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ebc1099c0 0x7f0ebc1ab0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:13.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.242+0000 7f0ec1e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc1123b0 con 0x7f0ebc102220 2024-04-23T14:59:13.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.242+0000 7f0ec1e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc112230 con 0x7f0ebc1072f0 2024-04-23T14:59:13.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.242+0000 7f0ec1e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc112530 con 0x7f0ebc1099c0 2024-04-23T14:59:13.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.242+0000 7f0ebaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc102220 0x7f0ebc108f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:13.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.243+0000 7f0ebaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc102220 0x7f0ebc108f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40778/0 (socket says 172.21.15.28:40778) 2024-04-23T14:59:13.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.243+0000 7f0ebaa00640 1 -- 172.21.15.28:0/1666249577 learned_addr learned my addr 172.21.15.28:0/1666249577 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-23T14:59:13.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.243+0000 7f0eba000640 1 --2- 172.21.15.28:0/1666249577 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ebc1072f0 0x7f0ebc109480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:13.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.243+0000 7f0ebb400640 1 --2- 172.21.15.28:0/1666249577 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ebc1099c0 0x7f0ebc1ab0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:13.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.243+0000 7f0ebaa00640 1 -- 172.21.15.28:0/1666249577 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ebc1099c0 msgr2=0x7f0ebc1ab0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:13.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.243+0000 7f0ebaa00640 1 --2- 172.21.15.28:0/1666249577 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ebc1099c0 0x7f0ebc1ab0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:13.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.243+0000 7f0ebaa00640 1 -- 172.21.15.28:0/1666249577 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ebc1072f0 msgr2=0x7f0ebc109480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:13.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.243+0000 7f0ebaa00640 1 --2- 172.21.15.28:0/1666249577 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ebc1072f0 0x7f0ebc109480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:13.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.243+0000 7f0ebaa00640 1 -- 172.21.15.28:0/1666249577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ebc1ab620 con 0x7f0ebc102220 2024-04-23T14:59:13.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.243+0000 7f0eba000640 1 --2- 172.21.15.28:0/1666249577 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ebc1072f0 0x7f0ebc109480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T14:59:13.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.243+0000 7f0ebaa00640 1 --2- 172.21.15.28:0/1666249577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc102220 0x7f0ebc108f40 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0eac00e9f0 tx=0x7f0eac00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:59:13.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.244+0000 7f0ec0a00640 1 -- 172.21.15.28:0/1666249577 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eac00cd80 con 0x7f0ebc102220 2024-04-23T14:59:13.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.244+0000 7f0ec0a00640 1 -- 172.21.15.28:0/1666249577 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0eac010910 con 0x7f0ebc102220 2024-04-23T14:59:13.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.244+0000 7f0ec0a00640 1 -- 172.21.15.28:0/1666249577 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eac018990 con 0x7f0ebc102220 2024-04-23T14:59:13.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.244+0000 7f0ec1e00640 1 -- 172.21.15.28:0/1666249577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ebc1ab910 con 0x7f0ebc102220 2024-04-23T14:59:13.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.244+0000 7f0ec1e00640 1 -- 172.21.15.28:0/1666249577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ebc1abe80 con 0x7f0ebc102220 2024-04-23T14:59:13.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.245+0000 7f0ec1e00640 1 -- 172.21.15.28:0/1666249577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ebc1036f0 con 0x7f0ebc102220 2024-04-23T14:59:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.247+0000 7f0ec0a00640 1 -- 172.21.15.28:0/1666249577 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f0eac010ac0 con 0x7f0ebc102220 2024-04-23T14:59:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.249+0000 7f0ec0a00640 1 --2- 172.21.15.28:0/1666249577 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f0ea00a3da0 0x7f0ea00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T14:59:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.249+0000 7f0ec0a00640 1 -- 172.21.15.28:0/1666249577 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8074+0+0 (secure 0 0 0) 0x7f0eac0d0910 con 0x7f0ebc102220 2024-04-23T14:59:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.249+0000 7f0eba000640 1 --2- 172.21.15.28:0/1666249577 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f0ea00a3da0 0x7f0ea00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T14:59:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.250+0000 7f0eba000640 1 --2- 172.21.15.28:0/1666249577 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f0ea00a3da0 0x7f0ea00a6260 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc10a700 tx=0x7f0ea8005f90 comp rx=0 tx=0).ready entity=mgr.25153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T14:59:13.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.253+0000 7f0ec0a00640 1 -- 172.21.15.28:0/1666249577 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0eac09cad0 con 0x7f0ebc102220 2024-04-23T14:59:13.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:12 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1415092112' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T14:59:13.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:12 smithi105 ceph-mon[32555]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T14:59:13.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:12 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1415092112' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T14:59:13.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:12 smithi038 ceph-mon[32348]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T14:59:13.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:12 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1415092112' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T14:59:13.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:12 smithi028 ceph-mon[27810]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T14:59:13.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:13.478+0000 7f0ec1e00640 1 -- 172.21.15.28:0/1666249577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f0ebc069270 con 0x7f0ebc102220 2024-04-23T14:59:14.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:14.011+0000 7f0ec0a00640 1 -- 172.21.15.28:0/1666249577 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v108) ==== 169+0+0 (secure 0 0 0) 0x7f0eac0a0dd0 con 0x7f0ebc102220 2024-04-23T14:59:14.012 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-04-23T14:59:14.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:14.015+0000 7f0ec1e00640 1 -- 172.21.15.28:0/1666249577 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f0ea00a3da0 msgr2=0x7f0ea00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:14.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:14.015+0000 7f0ec1e00640 1 --2- 172.21.15.28:0/1666249577 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f0ea00a3da0 0x7f0ea00a6260 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc10a700 tx=0x7f0ea8005f90 comp rx=0 tx=0).stop 2024-04-23T14:59:14.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:14.015+0000 7f0ec1e00640 1 -- 172.21.15.28:0/1666249577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc102220 msgr2=0x7f0ebc108f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T14:59:14.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:14.015+0000 7f0ec1e00640 1 --2- 172.21.15.28:0/1666249577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc102220 0x7f0ebc108f40 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0eac00e9f0 tx=0x7f0eac00eeb0 comp rx=0 tx=0).stop 2024-04-23T14:59:14.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:14.017+0000 7f0ec1e00640 1 -- 172.21.15.28:0/1666249577 shutdown_connections 2024-04-23T14:59:14.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:14.017+0000 7f0ec1e00640 1 --2- 172.21.15.28:0/1666249577 >> [v2:172.21.15.28:6800/1615546338,v1:172.21.15.28:6801/1615546338] conn(0x7f0ea00a3da0 0x7f0ea00a6260 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:14.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:14.017+0000 7f0ec1e00640 1 --2- 172.21.15.28:0/1666249577 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ebc1072f0 0x7f0ebc109480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:14.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:14.017+0000 7f0ec1e00640 1 --2- 172.21.15.28:0/1666249577 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ebc1099c0 0x7f0ebc1ab0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:14.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:14.017+0000 7f0ec1e00640 1 --2- 172.21.15.28:0/1666249577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc102220 0x7f0ebc108f40 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T14:59:14.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:14.017+0000 7f0ec1e00640 1 -- 172.21.15.28:0/1666249577 >> 172.21.15.28:0/1666249577 conn(0x7f0ebc079450 msgr2=0x7f0ebc0ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T14:59:14.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:14.018+0000 7f0ec1e00640 1 -- 172.21.15.28:0/1666249577 shutdown_connections 2024-04-23T14:59:14.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-23T14:59:14.019+0000 7f0ec1e00640 1 -- 172.21.15.28:0/1666249577 wait complete. 2024-04-23T14:59:14.053 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T14:59:14.134 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T14:59:14.190 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T14:59:14.216 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-23T14:59:14.216 DEBUG:teuthology.orchestra.run.smithi028:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-23T14:59:14.270 DEBUG:teuthology.orchestra.run.smithi038:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-23T14:59:14.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:14 smithi105 ceph-mon[32555]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T14:59:14.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:14 smithi105 ceph-mon[32555]: osdmap e107: 12 total, 12 up, 12 in 2024-04-23T14:59:14.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:14 smithi105 ceph-mon[32555]: from='client.? 172.21.15.28:0/1666249577' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T14:59:14.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:14 smithi105 ceph-mon[32555]: pgmap v478: 65 pgs: 65 active+clean; 288 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:59:14.287 DEBUG:teuthology.orchestra.run.smithi105:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-23T14:59:14.301 INFO:tasks.cephadm:Stopping all daemons... 2024-04-23T14:59:14.302 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-23T14:59:14.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mon.a 2024-04-23T14:59:14.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:14 smithi038 ceph-mon[32348]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T14:59:14.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:14 smithi038 ceph-mon[32348]: osdmap e107: 12 total, 12 up, 12 in 2024-04-23T14:59:14.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:14 smithi038 ceph-mon[32348]: from='client.? 172.21.15.28:0/1666249577' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T14:59:14.372 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:14 smithi038 ceph-mon[32348]: pgmap v478: 65 pgs: 65 active+clean; 288 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:59:14.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:14 smithi028 ceph-mon[27810]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T14:59:14.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:14 smithi028 ceph-mon[27810]: osdmap e107: 12 total, 12 up, 12 in 2024-04-23T14:59:14.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:14 smithi028 ceph-mon[27810]: from='client.? 172.21.15.28:0/1666249577' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T14:59:14.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:14 smithi028 ceph-mon[27810]: pgmap v478: 65 pgs: 65 active+clean; 288 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T14:59:14.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:14 smithi028 systemd[1]: Stopping Ceph mon.a for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T14:59:15.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:14 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mon-a[27806]: 2024-04-23T14:59:14.893+0000 7f19fc400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T14:59:15.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:14 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mon-a[27806]: 2024-04-23T14:59:14.894+0000 7f19fc400700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-23T14:59:15.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:15 smithi028 podman[93448]: 2024-04-23 14:59:15.121022152 +0000 UTC m=+0.354383691 container died 4e1b4802bb2e60eb95ec341069b34f30d2320e0e2905d0a1c27842cec27769e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-23T14:59:15.819 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:15 smithi028 podman[93448]: 2024-04-23 14:59:15.553962239 +0000 UTC m=+0.787323801 container cleanup 4e1b4802bb2e60eb95ec341069b34f30d2320e0e2905d0a1c27842cec27769e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mon-a, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, io.buildah.version=1.33.5) 2024-04-23T14:59:15.819 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:15 smithi028 bash[93448]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mon-a 2024-04-23T14:59:16.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 23 14:59:15 smithi028 podman[93462]: 2024-04-23 14:59:15.818638555 +0000 UTC m=+0.679957107 container remove 4e1b4802bb2e60eb95ec341069b34f30d2320e0e2905d0a1c27842cec27769e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mon-a, ceph=True, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-23T14:59:16.389 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mon.a.service' 2024-04-23T14:59:16.506 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T14:59:16.506 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-23T14:59:16.507 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-23T14:59:16.507 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mon.b 2024-04-23T14:59:16.842 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:16 smithi038 systemd[1]: Stopping Ceph mon.b for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T14:59:17.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:16 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mon-b[32344]: 2024-04-23T14:59:16.841+0000 7ff550cb7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T14:59:17.122 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:16 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mon-b[32344]: 2024-04-23T14:59:16.841+0000 7ff550cb7700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-23T14:59:17.123 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:16 smithi038 podman[68900]: 2024-04-23 14:59:16.892839737 +0000 UTC m=+0.160512190 container died 8a2678fe19a19ff9da076bfa60308e44a1bd96ed446890180ff163e85e29f1c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mon-b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True) 2024-04-23T14:59:17.567 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:17 smithi038 podman[68900]: 2024-04-23 14:59:17.250183267 +0000 UTC m=+0.517855719 container cleanup 8a2678fe19a19ff9da076bfa60308e44a1bd96ed446890180ff163e85e29f1c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mon-b, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-23T14:59:17.568 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 23 14:59:17 smithi038 bash[68900]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mon-b 2024-04-23T14:59:17.669 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mon.b.service' 2024-04-23T14:59:17.716 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T14:59:17.716 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-23T14:59:17.717 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-23T14:59:17.717 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mon.c 2024-04-23T14:59:18.010 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:17 smithi105 systemd[1]: Stopping Ceph mon.c for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T14:59:18.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:18 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mon-c[32538]: 2024-04-23T14:59:18.008+0000 7f4eaea57700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T14:59:18.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:18 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mon-c[32538]: 2024-04-23T14:59:18.008+0000 7f4eaea57700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-23T14:59:18.286 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:18 smithi105 podman[69534]: 2024-04-23 14:59:18.029988333 +0000 UTC m=+0.126727803 container died 4684ef5a77bf23e76c8c169c169f128211ff4fa4bd86f95bb0d4e0bf9ae042ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mon-c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-23T14:59:18.685 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:18 smithi105 podman[69534]: 2024-04-23 14:59:18.392884947 +0000 UTC m=+0.489624408 container cleanup 4684ef5a77bf23e76c8c169c169f128211ff4fa4bd86f95bb0d4e0bf9ae042ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mon-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416) 2024-04-23T14:59:18.686 INFO:journalctl@ceph.mon.c.smithi105.stdout:Apr 23 14:59:18 smithi105 bash[69534]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mon-c 2024-04-23T14:59:18.811 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mon.c.service' 2024-04-23T14:59:18.853 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T14:59:18.853 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-23T14:59:18.854 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-23T14:59:18.854 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mgr.x 2024-04-23T14:59:19.227 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:59:18 smithi028 systemd[1]: Stopping Ceph mgr.x for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T14:59:19.525 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:59:19 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:59:19.440+0000 7f0b05200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T14:59:19.525 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:59:19 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x[28049]: 2024-04-23T14:59:19.440+0000 7f0b05200700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-23T14:59:19.780 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:59:19 smithi028 podman[93579]: 2024-04-23 14:59:19.532454892 +0000 UTC m=+0.281990875 container died 7f8a92032a8ce13a50ae5482bf17653809d0862599c39f2bed9d08ef99aa24b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T14:59:20.123 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:59:19 smithi028 podman[93579]: 2024-04-23 14:59:19.957881021 +0000 UTC m=+0.707417037 container cleanup 7f8a92032a8ce13a50ae5482bf17653809d0862599c39f2bed9d08ef99aa24b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T14:59:20.123 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:59:19 smithi028 bash[93579]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x 2024-04-23T14:59:20.612 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 23 14:59:20 smithi028 podman[93592]: 2024-04-23 14:59:20.176523605 +0000 UTC m=+0.635427496 container remove 7f8a92032a8ce13a50ae5482bf17653809d0862599c39f2bed9d08ef99aa24b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d44fddd910b5e94da0f2357af4be37d7bd495ae0, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd) 2024-04-23T14:59:20.656 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-23T14:59:20.779 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T14:59:20.780 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-23T14:59:20.780 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-23T14:59:20.780 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mgr.y 2024-04-23T14:59:21.122 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:59:20 smithi038 systemd[1]: Stopping Ceph mgr.y for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T14:59:21.437 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:59:21 smithi038 podman[69022]: 2024-04-23 14:59:21.13342756 +0000 UTC m=+0.145839078 container died bf6cd51d0f02f4c6094c6e5dc35c0aa41807ce5e6d44e08e677dc2d374b64c2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd) 2024-04-23T14:59:21.688 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:59:21 smithi038 podman[69022]: 2024-04-23 14:59:21.43777562 +0000 UTC m=+0.450187192 container cleanup bf6cd51d0f02f4c6094c6e5dc35c0aa41807ce5e6d44e08e677dc2d374b64c2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y, org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T14:59:21.688 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 23 14:59:21 smithi038 bash[69022]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-y 2024-04-23T14:59:21.810 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-23T14:59:21.845 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T14:59:21.845 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-23T14:59:21.845 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-23T14:59:21.845 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mgr.z 2024-04-23T14:59:22.176 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:59:21 smithi105 systemd[1]: Stopping Ceph mgr.z for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T14:59:22.456 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:59:22 smithi105 podman[69657]: 2024-04-23 14:59:22.17632869 +0000 UTC m=+0.134437268 container died afe82dbc72f2d6902a5c7b41314b235c4ba6c1369ca8c767617b3463dfc4bd29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.vendor=CentOS) 2024-04-23T14:59:22.786 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:59:22 smithi105 podman[69657]: 2024-04-23 14:59:22.555557363 +0000 UTC m=+0.513665940 container cleanup afe82dbc72f2d6902a5c7b41314b235c4ba6c1369ca8c767617b3463dfc4bd29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-23T14:59:22.786 INFO:journalctl@ceph.mgr.z.smithi105.stdout:Apr 23 14:59:22 smithi105 bash[69657]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mgr-z 2024-04-23T14:59:22.910 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-23T14:59:22.942 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T14:59:22.943 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-23T14:59:22.943 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-23T14:59:22.943 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.0 2024-04-23T14:59:23.373 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 23 14:59:23 smithi028 systemd[1]: Stopping Ceph osd.0 for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T14:59:23.874 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 23 14:59:23 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-0[40196]: 2024-04-23T14:59:23.595+0000 7f7680e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T14:59:23.874 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 23 14:59:23 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-0[40196]: 2024-04-23T14:59:23.595+0000 7f7680e00700 -1 osd.0 108 *** Got signal Terminated *** 2024-04-23T14:59:23.874 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 23 14:59:23 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-0[40196]: 2024-04-23T14:59:23.595+0000 7f7680e00700 -1 osd.0 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T14:59:29.488 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 23 14:59:29 smithi028 podman[93697]: 2024-04-23 14:59:29.19398534 +0000 UTC m=+5.791453821 container died 8a704935cdbc4be5392902ad247034cb3ca4a698a3527fea5f8f05cc20b1441c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T14:59:29.805 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 23 14:59:29 smithi028 podman[93697]: 2024-04-23 14:59:29.596150253 +0000 UTC m=+6.193618732 container cleanup 8a704935cdbc4be5392902ad247034cb3ca4a698a3527fea5f8f05cc20b1441c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-23T14:59:29.805 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 23 14:59:29 smithi028 bash[93697]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-0 2024-04-23T14:59:30.124 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 23 14:59:29 smithi028 podman[93718]: 2024-04-23 14:59:29.805518832 +0000 UTC m=+0.593844462 container remove 8a704935cdbc4be5392902ad247034cb3ca4a698a3527fea5f8f05cc20b1441c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-0, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-23T14:59:30.874 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 23 14:59:30 smithi028 podman[93808]: 2024-04-23 14:59:30.503085297 +0000 UTC m=+0.032956119 image pull 76e77f280c74312ca1add9b6019a4a921a516f91795d85bae00a0c8ca9b1c62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd 2024-04-23T14:59:30.874 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 23 14:59:30 smithi028 podman[93808]: 2024-04-23 14:59:30.639988095 +0000 UTC m=+0.169858958 container create 53e8bb2da1ccd64c0f4f557081883b009bb26fc92729f3f30dfe782596822bd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240416, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-23T14:59:31.374 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 23 14:59:30 smithi028 podman[93808]: 2024-04-23 14:59:30.934854979 +0000 UTC m=+0.464725862 container init 53e8bb2da1ccd64c0f4f557081883b009bb26fc92729f3f30dfe782596822bd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416) 2024-04-23T14:59:31.374 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 23 14:59:30 smithi028 podman[93808]: 2024-04-23 14:59:30.947289987 +0000 UTC m=+0.477160815 container start 53e8bb2da1ccd64c0f4f557081883b009bb26fc92729f3f30dfe782596822bd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-23T14:59:31.374 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 23 14:59:31 smithi028 podman[93808]: 2024-04-23 14:59:31.041794242 +0000 UTC m=+0.571665109 container attach 53e8bb2da1ccd64c0f4f557081883b009bb26fc92729f3f30dfe782596822bd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd) 2024-04-23T14:59:31.760 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 23 14:59:31 smithi028 podman[93808]: 2024-04-23 14:59:31.417080963 +0000 UTC m=+0.946951866 container died 53e8bb2da1ccd64c0f4f557081883b009bb26fc92729f3f30dfe782596822bd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416) 2024-04-23T14:59:32.198 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.0.service' 2024-04-23T14:59:32.317 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T14:59:32.317 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-23T14:59:32.317 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-23T14:59:32.318 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.1 2024-04-23T14:59:32.622 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 23 14:59:32 smithi038 systemd[1]: Stopping Ceph osd.1 for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T14:59:33.122 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 23 14:59:32 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-1[36932]: 2024-04-23T14:59:32.641+0000 7fb79f54f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T14:59:33.122 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 23 14:59:32 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-1[36932]: 2024-04-23T14:59:32.641+0000 7fb79f54f700 -1 osd.1 108 *** Got signal Terminated *** 2024-04-23T14:59:33.123 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 23 14:59:32 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-1[36932]: 2024-04-23T14:59:32.641+0000 7fb79f54f700 -1 osd.1 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T14:59:38.319 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 23 14:59:38 smithi038 podman[69144]: 2024-04-23 14:59:38.048827006 +0000 UTC m=+5.531290031 container died eaaea8b5b82077731b87f69f522c87d0b92e84cbabf7550a2a62a478f27cda3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-1, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T14:59:38.622 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 23 14:59:38 smithi038 podman[69144]: 2024-04-23 14:59:38.434215571 +0000 UTC m=+5.916678599 container cleanup eaaea8b5b82077731b87f69f522c87d0b92e84cbabf7550a2a62a478f27cda3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-1, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd) 2024-04-23T14:59:38.622 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 23 14:59:38 smithi038 bash[69144]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-1 2024-04-23T14:59:39.062 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 23 14:59:38 smithi038 podman[69161]: 2024-04-23 14:59:38.702331157 +0000 UTC m=+0.647098466 container remove eaaea8b5b82077731b87f69f522c87d0b92e84cbabf7550a2a62a478f27cda3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-1, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-23T14:59:39.314 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 23 14:59:39 smithi038 podman[69252]: 2024-04-23 14:59:38.965630078 +0000 UTC m=+0.018228306 image pull 76e77f280c74312ca1add9b6019a4a921a516f91795d85bae00a0c8ca9b1c62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd 2024-04-23T14:59:39.314 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 23 14:59:39 smithi038 podman[69252]: 2024-04-23 14:59:39.069920904 +0000 UTC m=+0.122519162 container create 515aeb8bc78507c7356e9b23c046d7ea4a06bc5a719f0c48c1a3c429d6ff294d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-23T14:59:39.314 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 23 14:59:39 smithi038 podman[69252]: 2024-04-23 14:59:39.314217947 +0000 UTC m=+0.366816160 container init 515aeb8bc78507c7356e9b23c046d7ea4a06bc5a719f0c48c1a3c429d6ff294d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T14:59:39.622 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 23 14:59:39 smithi038 podman[69252]: 2024-04-23 14:59:39.318026659 +0000 UTC m=+0.370624873 container start 515aeb8bc78507c7356e9b23c046d7ea4a06bc5a719f0c48c1a3c429d6ff294d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-23T14:59:39.622 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 23 14:59:39 smithi038 podman[69252]: 2024-04-23 14:59:39.406087962 +0000 UTC m=+0.458686192 container attach 515aeb8bc78507c7356e9b23c046d7ea4a06bc5a719f0c48c1a3c429d6ff294d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-23T14:59:39.967 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 23 14:59:39 smithi038 podman[69252]: 2024-04-23 14:59:39.641411495 +0000 UTC m=+0.694009709 container died 515aeb8bc78507c7356e9b23c046d7ea4a06bc5a719f0c48c1a3c429d6ff294d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-1-deactivate, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T14:59:40.373 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.1.service' 2024-04-23T14:59:40.415 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T14:59:40.415 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-23T14:59:40.415 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-23T14:59:40.416 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.2 2024-04-23T14:59:40.718 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 23 14:59:40 smithi105 systemd[1]: Stopping Ceph osd.2 for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T14:59:40.975 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 23 14:59:40 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-2[37737]: 2024-04-23T14:59:40.717+0000 7f9111bfd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T14:59:40.975 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 23 14:59:40 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-2[37737]: 2024-04-23T14:59:40.717+0000 7f9111bfd700 -1 osd.2 108 *** Got signal Terminated *** 2024-04-23T14:59:40.975 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 23 14:59:40 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-2[37737]: 2024-04-23T14:59:40.717+0000 7f9111bfd700 -1 osd.2 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T14:59:46.453 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 23 14:59:46 smithi105 podman[69780]: 2024-04-23 14:59:46.19546186 +0000 UTC m=+5.585073432 container died 90577d471727b664b6ace26d0c2153802dc01e6f73bf05b1e08df227411e6e7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T14:59:46.785 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 23 14:59:46 smithi105 podman[69780]: 2024-04-23 14:59:46.544480175 +0000 UTC m=+5.934091733 container cleanup 90577d471727b664b6ace26d0c2153802dc01e6f73bf05b1e08df227411e6e7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-23T14:59:46.786 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 23 14:59:46 smithi105 bash[69780]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-2 2024-04-23T14:59:47.214 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 23 14:59:46 smithi105 podman[69809]: 2024-04-23 14:59:46.896274987 +0000 UTC m=+0.698127340 container remove 90577d471727b664b6ace26d0c2153802dc01e6f73bf05b1e08df227411e6e7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-2, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-23T14:59:47.352 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 23 14:59:47 smithi105 podman[69907]: 2024-04-23 14:59:47.118450505 +0000 UTC m=+0.018065945 image pull 76e77f280c74312ca1add9b6019a4a921a516f91795d85bae00a0c8ca9b1c62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd 2024-04-23T14:59:47.352 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 23 14:59:47 smithi105 podman[69907]: 2024-04-23 14:59:47.238767244 +0000 UTC m=+0.138382685 container create 5196fb0e264ad4df285dfee104a8f974a47f02da143310fefc4c4a4a5477d288 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T14:59:47.786 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 23 14:59:47 smithi105 podman[69907]: 2024-04-23 14:59:47.491281422 +0000 UTC m=+0.390896854 container init 5196fb0e264ad4df285dfee104a8f974a47f02da143310fefc4c4a4a5477d288 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-23T14:59:47.786 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 23 14:59:47 smithi105 podman[69907]: 2024-04-23 14:59:47.494311905 +0000 UTC m=+0.393927337 container start 5196fb0e264ad4df285dfee104a8f974a47f02da143310fefc4c4a4a5477d288 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-2-deactivate, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-23T14:59:47.786 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 23 14:59:47 smithi105 podman[69907]: 2024-04-23 14:59:47.574551541 +0000 UTC m=+0.474166984 container attach 5196fb0e264ad4df285dfee104a8f974a47f02da143310fefc4c4a4a5477d288 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240416) 2024-04-23T14:59:48.127 INFO:journalctl@ceph.osd.2.smithi105.stdout:Apr 23 14:59:47 smithi105 podman[69907]: 2024-04-23 14:59:47.791251562 +0000 UTC m=+0.690866989 container died 5196fb0e264ad4df285dfee104a8f974a47f02da143310fefc4c4a4a5477d288 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T14:59:48.418 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.2.service' 2024-04-23T14:59:48.459 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T14:59:48.459 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-23T14:59:48.459 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-23T14:59:48.459 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.3 2024-04-23T14:59:48.843 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 23 14:59:48 smithi028 systemd[1]: Stopping Ceph osd.3 for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T14:59:49.124 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 23 14:59:49 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-3[46706]: 2024-04-23T14:59:49.080+0000 7fb37a000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T14:59:49.124 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 23 14:59:49 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-3[46706]: 2024-04-23T14:59:49.080+0000 7fb37a000700 -1 osd.3 108 *** Got signal Terminated *** 2024-04-23T14:59:49.124 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 23 14:59:49 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-3[46706]: 2024-04-23T14:59:49.080+0000 7fb37a000700 -1 osd.3 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T14:59:50.238 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 14:59:49 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T14:59:49.911+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.937927+0000 front 2024-04-23T14:59:23.938084+0000 (oldest deadline 2024-04-23T14:59:49.838393+0000) 2024-04-23T14:59:50.536 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 14:59:50 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T14:59:50.105+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:24.128787+0000 front 2024-04-23T14:59:24.128867+0000 (oldest deadline 2024-04-23T14:59:49.428639+0000) 2024-04-23T14:59:50.536 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 14:59:50 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T14:59:50.296+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T14:59:50.570 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 14:59:50 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T14:59:50.373+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T14:59:51.286 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 14:59:51 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T14:59:51.056+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:24.128787+0000 front 2024-04-23T14:59:24.128867+0000 (oldest deadline 2024-04-23T14:59:49.428639+0000) 2024-04-23T14:59:51.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 14:59:50 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T14:59:50.929+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T14:59:51.364 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 14:59:50 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T14:59:50.944+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.937927+0000 front 2024-04-23T14:59:23.938084+0000 (oldest deadline 2024-04-23T14:59:49.838393+0000) 2024-04-23T14:59:51.623 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 14:59:51 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T14:59:51.363+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T14:59:51.785 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 14:59:51 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T14:59:51.315+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T14:59:52.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 14:59:51 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T14:59:51.933+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T14:59:52.286 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 14:59:52 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T14:59:52.079+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:24.128787+0000 front 2024-04-23T14:59:24.128867+0000 (oldest deadline 2024-04-23T14:59:49.428639+0000) 2024-04-23T14:59:52.373 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 14:59:51 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T14:59:51.983+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.937927+0000 front 2024-04-23T14:59:23.938084+0000 (oldest deadline 2024-04-23T14:59:49.838393+0000) 2024-04-23T14:59:52.373 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 23 14:59:52 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-3[46706]: 2024-04-23T14:59:52.176+0000 7fb375800700 -1 osd.3 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.974412+0000 front 2024-04-23T14:59:27.974507+0000 (oldest deadline 2024-04-23T14:59:51.474534+0000) 2024-04-23T14:59:52.785 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 14:59:52 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T14:59:52.342+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T14:59:52.873 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 14:59:52 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T14:59:52.383+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T14:59:53.216 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 14:59:52 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T14:59:52.942+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.937927+0000 front 2024-04-23T14:59:23.938084+0000 (oldest deadline 2024-04-23T14:59:49.838393+0000) 2024-04-23T14:59:53.285 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 14:59:53 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T14:59:53.112+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:24.128787+0000 front 2024-04-23T14:59:24.128867+0000 (oldest deadline 2024-04-23T14:59:49.428639+0000) 2024-04-23T14:59:53.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 14:59:52 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T14:59:52.923+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T14:59:53.623 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 23 14:59:53 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-3[46706]: 2024-04-23T14:59:53.216+0000 7fb375800700 -1 osd.3 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.974412+0000 front 2024-04-23T14:59:27.974507+0000 (oldest deadline 2024-04-23T14:59:51.474534+0000) 2024-04-23T14:59:53.624 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 14:59:53 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T14:59:53.414+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T14:59:53.785 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 14:59:53 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T14:59:53.371+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T14:59:54.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 14:59:53 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T14:59:53.724+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T14:59:54.285 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 14:59:54 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T14:59:54.154+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:24.128787+0000 front 2024-04-23T14:59:24.128867+0000 (oldest deadline 2024-04-23T14:59:49.428639+0000) 2024-04-23T14:59:54.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 14:59:53 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T14:59:53.966+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T14:59:54.373 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 14:59:53 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T14:59:53.904+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.937927+0000 front 2024-04-23T14:59:23.938084+0000 (oldest deadline 2024-04-23T14:59:49.838393+0000) 2024-04-23T14:59:54.785 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 14:59:54 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T14:59:54.339+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T14:59:54.873 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 14:59:54 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T14:59:54.415+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T14:59:54.874 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 23 14:59:54 smithi028 podman[93914]: 2024-04-23 14:59:54.654392508 +0000 UTC m=+5.725098697 container died 3b027f125e877a9f04ae5ce1e37e21793fb0ad2cfd1286858c23878150f60a1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-3, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0) 2024-04-23T14:59:55.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 14:59:54 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T14:59:54.695+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T14:59:55.188 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 14:59:54 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T14:59:54.923+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T14:59:55.274 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 14:59:54 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T14:59:54.907+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.937927+0000 front 2024-04-23T14:59:23.938084+0000 (oldest deadline 2024-04-23T14:59:49.838393+0000) 2024-04-23T14:59:55.274 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 23 14:59:55 smithi028 podman[93914]: 2024-04-23 14:59:55.072535734 +0000 UTC m=+6.143241949 container cleanup 3b027f125e877a9f04ae5ce1e37e21793fb0ad2cfd1286858c23878150f60a1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-3, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-23T14:59:55.274 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 23 14:59:55 smithi028 bash[93914]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-3 2024-04-23T14:59:55.536 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 14:59:55 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T14:59:55.187+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:24.128787+0000 front 2024-04-23T14:59:24.128867+0000 (oldest deadline 2024-04-23T14:59:49.428639+0000) 2024-04-23T14:59:55.536 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 14:59:55 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T14:59:55.349+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T14:59:55.623 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 23 14:59:55 smithi028 podman[93945]: 2024-04-23 14:59:55.273636643 +0000 UTC m=+0.606382141 container remove 3b027f125e877a9f04ae5ce1e37e21793fb0ad2cfd1286858c23878150f60a1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-3, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-23T14:59:55.624 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 14:59:55 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T14:59:55.426+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T14:59:56.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 14:59:55 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T14:59:55.700+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T14:59:56.286 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 14:59:56 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T14:59:56.184+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:24.128787+0000 front 2024-04-23T14:59:24.128867+0000 (oldest deadline 2024-04-23T14:59:49.428639+0000) 2024-04-23T14:59:56.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 14:59:55 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T14:59:55.953+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T14:59:56.333 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 14:59:55 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T14:59:55.954+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.937927+0000 front 2024-04-23T14:59:23.938084+0000 (oldest deadline 2024-04-23T14:59:49.838393+0000) 2024-04-23T14:59:56.334 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 23 14:59:56 smithi028 podman[94032]: 2024-04-23 14:59:56.021222281 +0000 UTC m=+0.033894195 image pull 76e77f280c74312ca1add9b6019a4a921a516f91795d85bae00a0c8ca9b1c62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd 2024-04-23T14:59:56.334 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 23 14:59:56 smithi028 podman[94032]: 2024-04-23 14:59:56.157872418 +0000 UTC m=+0.170544373 container create 2e7601986419753c85715dc5e52013f8c20662b3b7b0614a54dd92c071b97552 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416) 2024-04-23T14:59:56.623 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 14:59:56 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T14:59:56.387+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T14:59:56.624 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 23 14:59:56 smithi028 podman[94032]: 2024-04-23 14:59:56.461960168 +0000 UTC m=+0.474632126 container init 2e7601986419753c85715dc5e52013f8c20662b3b7b0614a54dd92c071b97552 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T14:59:56.624 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 23 14:59:56 smithi028 podman[94032]: 2024-04-23 14:59:56.474143084 +0000 UTC m=+0.486815038 container start 2e7601986419753c85715dc5e52013f8c20662b3b7b0614a54dd92c071b97552 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS) 2024-04-23T14:59:56.624 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 23 14:59:56 smithi028 podman[94032]: 2024-04-23 14:59:56.568333174 +0000 UTC m=+0.581005088 container attach 2e7601986419753c85715dc5e52013f8c20662b3b7b0614a54dd92c071b97552 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-23T14:59:56.785 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 14:59:56 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T14:59:56.310+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T14:59:57.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 14:59:56 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T14:59:56.666+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T14:59:57.242 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 23 14:59:56 smithi028 podman[94032]: 2024-04-23 14:59:56.950418532 +0000 UTC m=+0.963090446 container died 2e7601986419753c85715dc5e52013f8c20662b3b7b0614a54dd92c071b97552 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2) 2024-04-23T14:59:57.242 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 14:59:56 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T14:59:56.965+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.937927+0000 front 2024-04-23T14:59:23.938084+0000 (oldest deadline 2024-04-23T14:59:49.838393+0000) 2024-04-23T14:59:57.285 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 14:59:57 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T14:59:57.162+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:24.128787+0000 front 2024-04-23T14:59:24.128867+0000 (oldest deadline 2024-04-23T14:59:49.428639+0000) 2024-04-23T14:59:57.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 14:59:56 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T14:59:56.925+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T14:59:57.623 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 14:59:57 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T14:59:57.353+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T14:59:57.781 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.3.service' 2024-04-23T14:59:57.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 14:59:57 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T14:59:57.359+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T14:59:57.905 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T14:59:57.905 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-23T14:59:57.906 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-23T14:59:57.906 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.4 2024-04-23T14:59:57.933 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 14:59:57 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T14:59:57.681+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T14:59:58.123 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 14:59:57 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T14:59:57.935+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.937927+0000 front 2024-04-23T14:59:23.938084+0000 (oldest deadline 2024-04-23T14:59:49.838393+0000) 2024-04-23T14:59:58.195 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 14:59:57 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T14:59:57.923+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T14:59:58.220 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 23 14:59:57 smithi038 systemd[1]: Stopping Ceph osd.4 for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T14:59:58.464 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 14:59:58 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T14:59:58.195+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:24.128787+0000 front 2024-04-23T14:59:24.128867+0000 (oldest deadline 2024-04-23T14:59:49.428639+0000) 2024-04-23T14:59:58.464 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 14:59:58 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T14:59:58.195+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:35.830387+0000 front 2024-04-23T14:59:35.830403+0000 (oldest deadline 2024-04-23T14:59:58.130400+0000) 2024-04-23T14:59:58.464 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 14:59:58 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T14:59:58.316+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T14:59:58.622 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 23 14:59:58 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-4[43599]: 2024-04-23T14:59:58.219+0000 7f5261b3e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T14:59:58.622 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 23 14:59:58 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-4[43599]: 2024-04-23T14:59:58.219+0000 7f5261b3e700 -1 osd.4 108 *** Got signal Terminated *** 2024-04-23T14:59:58.622 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 23 14:59:58 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-4[43599]: 2024-04-23T14:59:58.219+0000 7f5261b3e700 -1 osd.4 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T14:59:58.623 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 14:59:58 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T14:59:58.364+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T14:59:59.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 14:59:58 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T14:59:58.687+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T14:59:59.123 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 14:59:58 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T14:59:58.693+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.474272+0000 front 2024-04-23T14:59:34.474414+0000 (oldest deadline 2024-04-23T14:59:58.574412+0000) 2024-04-23T14:59:59.184 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 14:59:58 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T14:59:58.914+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T14:59:59.185 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 14:59:58 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T14:59:58.914+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T14:59:59.318 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 14:59:58 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T14:59:58.915+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.937927+0000 front 2024-04-23T14:59:23.938084+0000 (oldest deadline 2024-04-23T14:59:49.838393+0000) 2024-04-23T14:59:59.535 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 14:59:59 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T14:59:59.293+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T14:59:59.536 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 14:59:59 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T14:59:59.184+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:24.128787+0000 front 2024-04-23T14:59:24.128867+0000 (oldest deadline 2024-04-23T14:59:49.428639+0000) 2024-04-23T14:59:59.536 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 14:59:59 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T14:59:59.184+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:35.830387+0000 front 2024-04-23T14:59:35.830403+0000 (oldest deadline 2024-04-23T14:59:58.130400+0000) 2024-04-23T14:59:59.623 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 14:59:59 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T14:59:59.318+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T14:59:59.623 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 14:59:59 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T14:59:59.318+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:00.054 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 14:59:59 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T14:59:59.684+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.474272+0000 front 2024-04-23T14:59:34.474414+0000 (oldest deadline 2024-04-23T14:59:58.574412+0000) 2024-04-23T15:00:00.055 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 14:59:59 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T14:59:59.660+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:00.274 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 14:59:59 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T14:59:59.930+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.937927+0000 front 2024-04-23T14:59:23.938084+0000 (oldest deadline 2024-04-23T14:59:49.838393+0000) 2024-04-23T15:00:00.274 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 14:59:59 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T14:59:59.930+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:35.142219+0000 front 2024-04-23T14:59:35.142323+0000 (oldest deadline 2024-04-23T14:59:59.842444+0000) 2024-04-23T15:00:00.286 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:00 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T15:00:00.155+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:24.128787+0000 front 2024-04-23T14:59:24.128867+0000 (oldest deadline 2024-04-23T14:59:49.428639+0000) 2024-04-23T15:00:00.286 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:00 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T15:00:00.155+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:35.830387+0000 front 2024-04-23T14:59:35.830403+0000 (oldest deadline 2024-04-23T14:59:58.130400+0000) 2024-04-23T15:00:00.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 14:59:59 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T14:59:59.954+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:00.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 14:59:59 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T14:59:59.954+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:00.372 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 23 15:00:00 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-4[43599]: 2024-04-23T15:00:00.054+0000 7f525a12a700 -1 osd.4 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.505977+0000 front 2024-04-23T14:59:34.506000+0000 (oldest deadline 2024-04-23T14:59:59.206090+0000) 2024-04-23T15:00:00.623 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:00 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:00.274+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:00.623 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:00 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:00.274+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:00.785 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:00 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:00.303+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:00.872 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:00 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:00.722+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.474272+0000 front 2024-04-23T14:59:34.474414+0000 (oldest deadline 2024-04-23T14:59:58.574412+0000) 2024-04-23T15:00:00.872 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:00 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:00.611+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:00.872 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:00 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:00.611+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:01.168 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:00 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:00.913+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:01.168 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:00 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:00.913+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:01.293 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:00 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:00.946+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.937927+0000 front 2024-04-23T14:59:23.938084+0000 (oldest deadline 2024-04-23T14:59:49.838393+0000) 2024-04-23T15:00:01.293 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:00 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:00.946+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:35.142219+0000 front 2024-04-23T14:59:35.142323+0000 (oldest deadline 2024-04-23T14:59:59.842444+0000) 2024-04-23T15:00:01.372 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 23 15:00:01 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-4[43599]: 2024-04-23T15:00:01.090+0000 7f525a12a700 -1 osd.4 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.505977+0000 front 2024-04-23T14:59:34.506000+0000 (oldest deadline 2024-04-23T14:59:59.206090+0000) 2024-04-23T15:00:01.536 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:01 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:01.328+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:01.536 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:01 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T15:00:01.168+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:24.128787+0000 front 2024-04-23T14:59:24.128867+0000 (oldest deadline 2024-04-23T14:59:49.428639+0000) 2024-04-23T15:00:01.536 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:01 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T15:00:01.168+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:35.830387+0000 front 2024-04-23T14:59:35.830403+0000 (oldest deadline 2024-04-23T14:59:58.130400+0000) 2024-04-23T15:00:01.623 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:01 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:01.291+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:01.623 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:01 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:01.291+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:02.089 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:01 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:01.674+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.474272+0000 front 2024-04-23T14:59:34.474414+0000 (oldest deadline 2024-04-23T14:59:58.574412+0000) 2024-04-23T15:00:02.089 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:01 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:01.626+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:02.089 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:01 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:01.626+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:02.139 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:01 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:01.881+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:02.140 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:01 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:01.881+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:02.338 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:01 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:01.991+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.937927+0000 front 2024-04-23T14:59:23.938084+0000 (oldest deadline 2024-04-23T14:59:49.838393+0000) 2024-04-23T15:00:02.338 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:01 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:01.991+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:35.142219+0000 front 2024-04-23T14:59:35.142323+0000 (oldest deadline 2024-04-23T14:59:59.842444+0000) 2024-04-23T15:00:02.372 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 23 15:00:02 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-4[43599]: 2024-04-23T15:00:02.088+0000 7f525a12a700 -1 osd.4 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.505977+0000 front 2024-04-23T14:59:34.506000+0000 (oldest deadline 2024-04-23T14:59:59.206090+0000) 2024-04-23T15:00:02.536 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:02 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:02.358+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:02.536 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:02 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:02.358+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.214584+0000 front 2024-04-23T14:59:33.214522+0000 (oldest deadline 2024-04-23T15:00:02.015036+0000) 2024-04-23T15:00:02.536 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:02 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T15:00:02.139+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:24.128787+0000 front 2024-04-23T14:59:24.128867+0000 (oldest deadline 2024-04-23T14:59:49.428639+0000) 2024-04-23T15:00:02.536 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:02 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T15:00:02.139+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:35.830387+0000 front 2024-04-23T14:59:35.830403+0000 (oldest deadline 2024-04-23T14:59:58.130400+0000) 2024-04-23T15:00:02.623 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:02 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:02.337+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:02.623 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:02 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:02.337+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:03.047 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:02 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:02.667+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.474272+0000 front 2024-04-23T14:59:34.474414+0000 (oldest deadline 2024-04-23T14:59:58.574412+0000) 2024-04-23T15:00:03.047 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:02 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:02.628+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:03.047 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:02 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:02.628+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:03.164 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:02 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:02.910+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:03.164 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:02 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:02.910+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:03.372 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 23 15:00:03 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-4[43599]: 2024-04-23T15:00:03.046+0000 7f525a12a700 -1 osd.4 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.505977+0000 front 2024-04-23T14:59:34.506000+0000 (oldest deadline 2024-04-23T14:59:59.206090+0000) 2024-04-23T15:00:03.373 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:02 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:02.944+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.937927+0000 front 2024-04-23T14:59:23.938084+0000 (oldest deadline 2024-04-23T14:59:49.838393+0000) 2024-04-23T15:00:03.374 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:02 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:02.944+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:35.142219+0000 front 2024-04-23T14:59:35.142323+0000 (oldest deadline 2024-04-23T14:59:59.842444+0000) 2024-04-23T15:00:03.535 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:03 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:03.387+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:03.536 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:03 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:03.387+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.214584+0000 front 2024-04-23T14:59:33.214522+0000 (oldest deadline 2024-04-23T15:00:02.015036+0000) 2024-04-23T15:00:03.536 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:03 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T15:00:03.164+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:24.128787+0000 front 2024-04-23T14:59:24.128867+0000 (oldest deadline 2024-04-23T14:59:49.428639+0000) 2024-04-23T15:00:03.536 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:03 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T15:00:03.164+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:35.830387+0000 front 2024-04-23T14:59:35.830403+0000 (oldest deadline 2024-04-23T14:59:58.130400+0000) 2024-04-23T15:00:03.872 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:03 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:03.649+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.474272+0000 front 2024-04-23T14:59:34.474414+0000 (oldest deadline 2024-04-23T14:59:58.574412+0000) 2024-04-23T15:00:03.873 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 23 15:00:03 smithi038 podman[69371]: 2024-04-23 15:00:03.70613276 +0000 UTC m=+5.591512703 container died 2b2f2acf861278517b54344463057f15ad32963eb2634a662fde419bf37fafda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd) 2024-04-23T15:00:03.873 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:03 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:03.596+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:03.873 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:03 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:03.596+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:03.873 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:03 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:03.387+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:03.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:03 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:03.387+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:04.184 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:03 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:03.867+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:04.184 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:03 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:03.867+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:04.279 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 23 15:00:04 smithi038 podman[69371]: 2024-04-23 15:00:04.004295265 +0000 UTC m=+5.889675201 container cleanup 2b2f2acf861278517b54344463057f15ad32963eb2634a662fde419bf37fafda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-4, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd) 2024-04-23T15:00:04.280 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 23 15:00:04 smithi038 bash[69371]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-4 2024-04-23T15:00:04.374 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:03 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:03.984+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.937927+0000 front 2024-04-23T14:59:23.938084+0000 (oldest deadline 2024-04-23T14:59:49.838393+0000) 2024-04-23T15:00:04.374 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:03 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:03.984+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:35.142219+0000 front 2024-04-23T14:59:35.142323+0000 (oldest deadline 2024-04-23T14:59:59.842444+0000) 2024-04-23T15:00:04.536 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:04 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:04.346+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:04.536 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:04 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:04.346+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.214584+0000 front 2024-04-23T14:59:33.214522+0000 (oldest deadline 2024-04-23T15:00:02.015036+0000) 2024-04-23T15:00:04.536 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:04 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T15:00:04.183+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:24.128787+0000 front 2024-04-23T14:59:24.128867+0000 (oldest deadline 2024-04-23T14:59:49.428639+0000) 2024-04-23T15:00:04.536 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:04 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T15:00:04.183+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:35.830387+0000 front 2024-04-23T14:59:35.830403+0000 (oldest deadline 2024-04-23T14:59:58.130400+0000) 2024-04-23T15:00:04.555 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 23 15:00:04 smithi038 podman[69389]: 2024-04-23 15:00:04.280338406 +0000 UTC m=+0.575205896 container remove 2b2f2acf861278517b54344463057f15ad32963eb2634a662fde419bf37fafda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-4, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T15:00:04.825 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:04 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:04.624+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.474272+0000 front 2024-04-23T14:59:34.474414+0000 (oldest deadline 2024-04-23T14:59:58.574412+0000) 2024-04-23T15:00:04.825 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:04 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:04.576+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:04.825 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:04 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:04.576+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:04.825 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 23 15:00:04 smithi038 podman[69479]: 2024-04-23 15:00:04.556017854 +0000 UTC m=+0.092730839 container create dab4104f19dfcb079b68a072d51da34710b6173421c04f478ca6526631cebd62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd) 2024-04-23T15:00:04.826 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 23 15:00:04 smithi038 podman[69479]: 2024-04-23 15:00:04.481757028 +0000 UTC m=+0.018470008 image pull 76e77f280c74312ca1add9b6019a4a921a516f91795d85bae00a0c8ca9b1c62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd 2024-04-23T15:00:04.873 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:04 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:04.416+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:04.873 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:04 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:04.416+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:05.122 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 23 15:00:04 smithi038 podman[69479]: 2024-04-23 15:00:04.825391115 +0000 UTC m=+0.362104095 container init dab4104f19dfcb079b68a072d51da34710b6173421c04f478ca6526631cebd62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, ceph=True, io.buildah.version=1.33.5) 2024-04-23T15:00:05.123 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 23 15:00:04 smithi038 podman[69479]: 2024-04-23 15:00:04.829568048 +0000 UTC m=+0.366281034 container start dab4104f19dfcb079b68a072d51da34710b6173421c04f478ca6526631cebd62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T15:00:05.123 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 23 15:00:04 smithi038 podman[69479]: 2024-04-23 15:00:04.908584819 +0000 UTC m=+0.445297804 container attach dab4104f19dfcb079b68a072d51da34710b6173421c04f478ca6526631cebd62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd) 2024-04-23T15:00:05.182 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:04 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:04.912+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:05.183 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:04 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:04.912+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:05.373 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:04 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:04.951+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.937927+0000 front 2024-04-23T14:59:23.938084+0000 (oldest deadline 2024-04-23T14:59:49.838393+0000) 2024-04-23T15:00:05.373 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:04 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:04.951+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:35.142219+0000 front 2024-04-23T14:59:35.142323+0000 (oldest deadline 2024-04-23T14:59:59.842444+0000) 2024-04-23T15:00:05.428 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 23 15:00:05 smithi038 podman[69479]: 2024-04-23 15:00:05.13133094 +0000 UTC m=+0.668043911 container died dab4104f19dfcb079b68a072d51da34710b6173421c04f478ca6526631cebd62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-4-deactivate, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-23T15:00:05.536 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:05 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T15:00:05.182+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:24.128787+0000 front 2024-04-23T14:59:24.128867+0000 (oldest deadline 2024-04-23T14:59:49.428639+0000) 2024-04-23T15:00:05.536 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:05 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T15:00:05.182+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:35.830387+0000 front 2024-04-23T14:59:35.830403+0000 (oldest deadline 2024-04-23T14:59:58.130400+0000) 2024-04-23T15:00:05.536 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:05 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:05.340+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:05.536 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:05 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:05.340+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.214584+0000 front 2024-04-23T14:59:33.214522+0000 (oldest deadline 2024-04-23T15:00:02.015036+0000) 2024-04-23T15:00:05.828 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:05 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:05.661+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.474272+0000 front 2024-04-23T14:59:34.474414+0000 (oldest deadline 2024-04-23T14:59:58.574412+0000) 2024-04-23T15:00:05.828 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:05 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:05.606+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:05.829 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:05 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:05.606+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:05.860 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.4.service' 2024-04-23T15:00:05.873 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:05 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:05.399+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:05.873 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:05 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:05.399+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:05.947 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T15:00:05.948 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-23T15:00:05.948 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-23T15:00:05.948 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.5 2024-04-23T15:00:06.232 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:05 smithi105 systemd[1]: Stopping Ceph osd.5 for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T15:00:06.232 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:05 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:05.879+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:06.232 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:05 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:05.879+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:06.373 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:05 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:05.930+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.937927+0000 front 2024-04-23T14:59:23.938084+0000 (oldest deadline 2024-04-23T14:59:49.838393+0000) 2024-04-23T15:00:06.374 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:05 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:05.930+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:35.142219+0000 front 2024-04-23T14:59:35.142323+0000 (oldest deadline 2024-04-23T14:59:59.842444+0000) 2024-04-23T15:00:06.536 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:06 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:06.383+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:06.536 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:06 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:06.383+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.214584+0000 front 2024-04-23T14:59:33.214522+0000 (oldest deadline 2024-04-23T15:00:02.015036+0000) 2024-04-23T15:00:06.536 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:06 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T15:00:06.231+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:24.128787+0000 front 2024-04-23T14:59:24.128867+0000 (oldest deadline 2024-04-23T14:59:49.428639+0000) 2024-04-23T15:00:06.536 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:06 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T15:00:06.231+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:35.830387+0000 front 2024-04-23T14:59:35.830403+0000 (oldest deadline 2024-04-23T14:59:58.130400+0000) 2024-04-23T15:00:06.536 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:06 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T15:00:06.262+0000 7f7dc4a22700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T15:00:06.536 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:06 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T15:00:06.262+0000 7f7dc4a22700 -1 osd.5 108 *** Got signal Terminated *** 2024-04-23T15:00:06.536 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:06 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T15:00:06.262+0000 7f7dc4a22700 -1 osd.5 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T15:00:06.873 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:06 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:06.416+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:06.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:06 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:06.416+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:07.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:06 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:06.708+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.474272+0000 front 2024-04-23T14:59:34.474414+0000 (oldest deadline 2024-04-23T14:59:58.574412+0000) 2024-04-23T15:00:07.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:06 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:06.654+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:07.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:06 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:06.654+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:07.209 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:06 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:06.884+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:07.209 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:06 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:06.884+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:07.373 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:06 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:06.958+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.937927+0000 front 2024-04-23T14:59:23.938084+0000 (oldest deadline 2024-04-23T14:59:49.838393+0000) 2024-04-23T15:00:07.374 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:06 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:06.958+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:35.142219+0000 front 2024-04-23T14:59:35.142323+0000 (oldest deadline 2024-04-23T14:59:59.842444+0000) 2024-04-23T15:00:07.536 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:07 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:07.408+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:07.536 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:07 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:07.408+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.214584+0000 front 2024-04-23T14:59:33.214522+0000 (oldest deadline 2024-04-23T15:00:02.015036+0000) 2024-04-23T15:00:07.536 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:07 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T15:00:07.209+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:24.128787+0000 front 2024-04-23T14:59:24.128867+0000 (oldest deadline 2024-04-23T14:59:49.428639+0000) 2024-04-23T15:00:07.536 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:07 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T15:00:07.209+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:35.830387+0000 front 2024-04-23T14:59:35.830403+0000 (oldest deadline 2024-04-23T14:59:58.130400+0000) 2024-04-23T15:00:07.536 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:07 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T15:00:07.209+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:42.831187+0000 front 2024-04-23T14:59:42.831168+0000 (oldest deadline 2024-04-23T15:00:06.931318+0000) 2024-04-23T15:00:07.873 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:07 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:07.380+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:07.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:07 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:07.380+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:08.023 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:07 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:07.706+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.474272+0000 front 2024-04-23T14:59:34.474414+0000 (oldest deadline 2024-04-23T14:59:58.574412+0000) 2024-04-23T15:00:08.023 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:07 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:07.706+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.875719+0000 front 2024-04-23T14:59:43.875508+0000 (oldest deadline 2024-04-23T15:00:06.775587+0000) 2024-04-23T15:00:08.024 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:07 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:07.664+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:08.024 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:07 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:07.664+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:08.230 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:07 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:07.902+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:08.231 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:07 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:07.902+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:08.231 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:07 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:07.902+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:08.373 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:07 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:07.961+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.937927+0000 front 2024-04-23T14:59:23.938084+0000 (oldest deadline 2024-04-23T14:59:49.838393+0000) 2024-04-23T15:00:08.374 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:07 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:07.961+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:35.142219+0000 front 2024-04-23T14:59:35.142323+0000 (oldest deadline 2024-04-23T14:59:59.842444+0000) 2024-04-23T15:00:08.536 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:08 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T15:00:08.230+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:24.128787+0000 front 2024-04-23T14:59:24.128867+0000 (oldest deadline 2024-04-23T14:59:49.428639+0000) 2024-04-23T15:00:08.536 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:08 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T15:00:08.230+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:35.830387+0000 front 2024-04-23T14:59:35.830403+0000 (oldest deadline 2024-04-23T14:59:58.130400+0000) 2024-04-23T15:00:08.536 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:08 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T15:00:08.230+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:42.831187+0000 front 2024-04-23T14:59:42.831168+0000 (oldest deadline 2024-04-23T15:00:06.931318+0000) 2024-04-23T15:00:08.536 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:08 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:08.429+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:08.536 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:08 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:08.429+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.214584+0000 front 2024-04-23T14:59:33.214522+0000 (oldest deadline 2024-04-23T15:00:02.015036+0000) 2024-04-23T15:00:08.873 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:08 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:08.391+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:08.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:08 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:08.391+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:09.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:08 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:08.719+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.474272+0000 front 2024-04-23T14:59:34.474414+0000 (oldest deadline 2024-04-23T14:59:58.574412+0000) 2024-04-23T15:00:09.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:08 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:08.719+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.875719+0000 front 2024-04-23T14:59:43.875508+0000 (oldest deadline 2024-04-23T15:00:06.775587+0000) 2024-04-23T15:00:09.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:08 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:08.623+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:09.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:08 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:08.623+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:09.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:08 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:08.623+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:09.214 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:08 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:08.908+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:09.214 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:08 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:08.908+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:09.215 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:08 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:08.908+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:09.369 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:08 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:08.949+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.937927+0000 front 2024-04-23T14:59:23.938084+0000 (oldest deadline 2024-04-23T14:59:49.838393+0000) 2024-04-23T15:00:09.369 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:08 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:08.949+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:35.142219+0000 front 2024-04-23T14:59:35.142323+0000 (oldest deadline 2024-04-23T14:59:59.842444+0000) 2024-04-23T15:00:09.369 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:08 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:08.949+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.545809+0000 front 2024-04-23T14:59:44.545911+0000 (oldest deadline 2024-04-23T15:00:08.645986+0000) 2024-04-23T15:00:09.468 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:09 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T15:00:09.214+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:24.128787+0000 front 2024-04-23T14:59:24.128867+0000 (oldest deadline 2024-04-23T14:59:49.428639+0000) 2024-04-23T15:00:09.468 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:09 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T15:00:09.214+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:35.830387+0000 front 2024-04-23T14:59:35.830403+0000 (oldest deadline 2024-04-23T14:59:58.130400+0000) 2024-04-23T15:00:09.468 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:09 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T15:00:09.214+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:42.831187+0000 front 2024-04-23T14:59:42.831168+0000 (oldest deadline 2024-04-23T15:00:06.931318+0000) 2024-04-23T15:00:09.623 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:09 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:09.368+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:09.624 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:09 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:09.368+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:09.624 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:09 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:09.368+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:09.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:09 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:09.467+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:09.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:09 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:09.467+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.214584+0000 front 2024-04-23T14:59:33.214522+0000 (oldest deadline 2024-04-23T15:00:02.015036+0000) 2024-04-23T15:00:10.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:09 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:09.702+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.474272+0000 front 2024-04-23T14:59:34.474414+0000 (oldest deadline 2024-04-23T14:59:58.574412+0000) 2024-04-23T15:00:10.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:09 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:09.702+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.875719+0000 front 2024-04-23T14:59:43.875508+0000 (oldest deadline 2024-04-23T15:00:06.775587+0000) 2024-04-23T15:00:10.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:09 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:09.641+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:10.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:09 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:09.641+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:10.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:09 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:09.641+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:10.260 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:09 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:09.940+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:10.260 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:09 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:09.940+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:10.260 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:09 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:09.940+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:10.363 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:09 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:09.915+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.937927+0000 front 2024-04-23T14:59:23.938084+0000 (oldest deadline 2024-04-23T14:59:49.838393+0000) 2024-04-23T15:00:10.363 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:09 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:09.915+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:35.142219+0000 front 2024-04-23T14:59:35.142323+0000 (oldest deadline 2024-04-23T14:59:59.842444+0000) 2024-04-23T15:00:10.363 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:09 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:09.915+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.545809+0000 front 2024-04-23T14:59:44.545911+0000 (oldest deadline 2024-04-23T15:00:08.645986+0000) 2024-04-23T15:00:10.536 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:10 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T15:00:10.260+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:24.128787+0000 front 2024-04-23T14:59:24.128867+0000 (oldest deadline 2024-04-23T14:59:49.428639+0000) 2024-04-23T15:00:10.536 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:10 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T15:00:10.260+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:35.830387+0000 front 2024-04-23T14:59:35.830403+0000 (oldest deadline 2024-04-23T14:59:58.130400+0000) 2024-04-23T15:00:10.536 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:10 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T15:00:10.260+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:42.831187+0000 front 2024-04-23T14:59:42.831168+0000 (oldest deadline 2024-04-23T15:00:06.931318+0000) 2024-04-23T15:00:10.536 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:10 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:10.418+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:10.537 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:10 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:10.418+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.214584+0000 front 2024-04-23T14:59:33.214522+0000 (oldest deadline 2024-04-23T15:00:02.015036+0000) 2024-04-23T15:00:10.623 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:10 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:10.362+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:10.623 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:10 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:10.362+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:10.624 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:10 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:10.362+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:11.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:10 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:10.733+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.474272+0000 front 2024-04-23T14:59:34.474414+0000 (oldest deadline 2024-04-23T14:59:58.574412+0000) 2024-04-23T15:00:11.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:10 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:10.733+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.875719+0000 front 2024-04-23T14:59:43.875508+0000 (oldest deadline 2024-04-23T15:00:06.775587+0000) 2024-04-23T15:00:11.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:10 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:10.663+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:11.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:10 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:10.663+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:11.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:10 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:10.663+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:11.242 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:10 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:10.898+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:11.243 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:10 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:10.898+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:11.243 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:10 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:10.898+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:11.368 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:10 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:10.898+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.937927+0000 front 2024-04-23T14:59:23.938084+0000 (oldest deadline 2024-04-23T14:59:49.838393+0000) 2024-04-23T15:00:11.368 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:10 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:10.898+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:35.142219+0000 front 2024-04-23T14:59:35.142323+0000 (oldest deadline 2024-04-23T14:59:59.842444+0000) 2024-04-23T15:00:11.368 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:10 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:10.898+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.545809+0000 front 2024-04-23T14:59:44.545911+0000 (oldest deadline 2024-04-23T15:00:08.645986+0000) 2024-04-23T15:00:11.536 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:11 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T15:00:11.242+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:24.128787+0000 front 2024-04-23T14:59:24.128867+0000 (oldest deadline 2024-04-23T14:59:49.428639+0000) 2024-04-23T15:00:11.536 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:11 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T15:00:11.242+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:35.830387+0000 front 2024-04-23T14:59:35.830403+0000 (oldest deadline 2024-04-23T14:59:58.130400+0000) 2024-04-23T15:00:11.536 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:11 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5[44662]: 2024-04-23T15:00:11.242+0000 7f7dc1016700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:42.831187+0000 front 2024-04-23T14:59:42.831168+0000 (oldest deadline 2024-04-23T15:00:06.931318+0000) 2024-04-23T15:00:11.536 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:11 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:11.409+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:11.536 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:11 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:11.409+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.214584+0000 front 2024-04-23T14:59:33.214522+0000 (oldest deadline 2024-04-23T15:00:02.015036+0000) 2024-04-23T15:00:11.623 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:11 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:11.367+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:11.624 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:11 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:11.367+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:11.624 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:11 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:11.367+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:11.874 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:11 smithi105 podman[70014]: 2024-04-23 15:00:11.618499055 +0000 UTC m=+5.475989935 container died 703b42a945a9037d2d67d3654ec4f1080b789820450a36abd7c084f83fb5354f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-23T15:00:12.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:11 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:11.749+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.474272+0000 front 2024-04-23T14:59:34.474414+0000 (oldest deadline 2024-04-23T14:59:58.574412+0000) 2024-04-23T15:00:12.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:11 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:11.749+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.875719+0000 front 2024-04-23T14:59:43.875508+0000 (oldest deadline 2024-04-23T15:00:06.775587+0000) 2024-04-23T15:00:12.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:11 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:11.683+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:12.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:11 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:11.683+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:12.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:11 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:11.683+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:12.141 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:11 smithi105 podman[70014]: 2024-04-23 15:00:11.957043862 +0000 UTC m=+5.814534734 container cleanup 703b42a945a9037d2d67d3654ec4f1080b789820450a36abd7c084f83fb5354f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-23T15:00:12.141 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:11 smithi105 bash[70014]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5 2024-04-23T15:00:12.142 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:11 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:11.901+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:12.142 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:11 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:11.901+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:12.142 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:11 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:11.901+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:12.374 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:11 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:11.891+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.937927+0000 front 2024-04-23T14:59:23.938084+0000 (oldest deadline 2024-04-23T14:59:49.838393+0000) 2024-04-23T15:00:12.374 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:11 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:11.891+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:35.142219+0000 front 2024-04-23T14:59:35.142323+0000 (oldest deadline 2024-04-23T14:59:59.842444+0000) 2024-04-23T15:00:12.374 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:11 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:11.891+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.545809+0000 front 2024-04-23T14:59:44.545911+0000 (oldest deadline 2024-04-23T15:00:08.645986+0000) 2024-04-23T15:00:12.400 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:12 smithi105 podman[70031]: 2024-04-23 15:00:12.141456716 +0000 UTC m=+0.515834996 container remove 703b42a945a9037d2d67d3654ec4f1080b789820450a36abd7c084f83fb5354f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True) 2024-04-23T15:00:12.692 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:12 smithi105 podman[70122]: 2024-04-23 15:00:12.471387818 +0000 UTC m=+0.018077820 image pull 76e77f280c74312ca1add9b6019a4a921a516f91795d85bae00a0c8ca9b1c62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd 2024-04-23T15:00:12.692 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:12 smithi105 podman[70122]: 2024-04-23 15:00:12.584638106 +0000 UTC m=+0.131328151 container create c6cfad5eaf0241f426b7a9c215ead5d6bf26bcf141ee43048dda01d8ee8b34d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T15:00:12.692 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:12 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:12.400+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:12.692 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:12 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:12.400+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.214584+0000 front 2024-04-23T14:59:33.214522+0000 (oldest deadline 2024-04-23T15:00:02.015036+0000) 2024-04-23T15:00:12.873 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:12 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:12.376+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:12.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:12 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:12.376+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:12.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:12 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:12.376+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:13.036 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:12 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:12.889+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:13.036 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:12 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:12.889+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:13.036 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:12 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:12.889+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:13.036 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:12 smithi105 podman[70122]: 2024-04-23 15:00:12.820861426 +0000 UTC m=+0.367551428 container init c6cfad5eaf0241f426b7a9c215ead5d6bf26bcf141ee43048dda01d8ee8b34d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T15:00:13.037 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:12 smithi105 podman[70122]: 2024-04-23 15:00:12.823881857 +0000 UTC m=+0.370571861 container start c6cfad5eaf0241f426b7a9c215ead5d6bf26bcf141ee43048dda01d8ee8b34d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.schema-version=1.0) 2024-04-23T15:00:13.037 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:12 smithi105 podman[70122]: 2024-04-23 15:00:12.90408318 +0000 UTC m=+0.450773184 container attach c6cfad5eaf0241f426b7a9c215ead5d6bf26bcf141ee43048dda01d8ee8b34d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd) 2024-04-23T15:00:13.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:12 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:12.708+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.474272+0000 front 2024-04-23T14:59:34.474414+0000 (oldest deadline 2024-04-23T14:59:58.574412+0000) 2024-04-23T15:00:13.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:12 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:12.708+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.875719+0000 front 2024-04-23T14:59:43.875508+0000 (oldest deadline 2024-04-23T15:00:06.775587+0000) 2024-04-23T15:00:13.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:12 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:12.717+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:13.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:12 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:12.717+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:13.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:12 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:12.717+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:13.374 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:12 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:12.932+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.937927+0000 front 2024-04-23T14:59:23.938084+0000 (oldest deadline 2024-04-23T14:59:49.838393+0000) 2024-04-23T15:00:13.374 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:12 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:12.932+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:35.142219+0000 front 2024-04-23T14:59:35.142323+0000 (oldest deadline 2024-04-23T14:59:59.842444+0000) 2024-04-23T15:00:13.374 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:12 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:12.932+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.545809+0000 front 2024-04-23T14:59:44.545911+0000 (oldest deadline 2024-04-23T15:00:08.645986+0000) 2024-04-23T15:00:13.415 INFO:journalctl@ceph.osd.5.smithi105.stdout:Apr 23 15:00:13 smithi105 podman[70122]: 2024-04-23 15:00:13.121980895 +0000 UTC m=+0.668670900 container died c6cfad5eaf0241f426b7a9c215ead5d6bf26bcf141ee43048dda01d8ee8b34d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-5-deactivate, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-23T15:00:13.707 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:13 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:13.424+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:13.707 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:13 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:13.424+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.214584+0000 front 2024-04-23T14:59:33.214522+0000 (oldest deadline 2024-04-23T15:00:02.015036+0000) 2024-04-23T15:00:13.733 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.5.service' 2024-04-23T15:00:13.769 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T15:00:13.770 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-23T15:00:13.770 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-23T15:00:13.770 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.6 2024-04-23T15:00:13.855 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:13 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:13.410+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:13.856 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:13 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:13.410+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:13.856 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:13 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:13.410+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:14.036 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:13 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:13.843+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:14.036 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:13 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:13.843+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:14.036 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:13 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:13.843+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:14.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:13 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:13.694+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.474272+0000 front 2024-04-23T14:59:34.474414+0000 (oldest deadline 2024-04-23T14:59:58.574412+0000) 2024-04-23T15:00:14.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:13 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:13.694+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.875719+0000 front 2024-04-23T14:59:43.875508+0000 (oldest deadline 2024-04-23T15:00:06.775587+0000) 2024-04-23T15:00:14.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:13 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:13.735+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:14.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:13 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:13.735+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:14.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:13 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:13.735+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:14.124 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:13 smithi028 systemd[1]: Stopping Ceph osd.6 for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T15:00:14.124 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:13 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:13.972+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.937927+0000 front 2024-04-23T14:59:23.938084+0000 (oldest deadline 2024-04-23T14:59:49.838393+0000) 2024-04-23T15:00:14.124 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:13 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:13.972+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:35.142219+0000 front 2024-04-23T14:59:35.142323+0000 (oldest deadline 2024-04-23T14:59:59.842444+0000) 2024-04-23T15:00:14.124 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:13 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:13.972+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.545809+0000 front 2024-04-23T14:59:44.545911+0000 (oldest deadline 2024-04-23T15:00:08.645986+0000) 2024-04-23T15:00:14.623 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:14 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:14.424+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:14.624 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:14 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:14.424+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:14.624 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:14 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:14.424+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:14.624 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:14 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:14.355+0000 7fe636800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T15:00:14.624 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:14 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:14.355+0000 7fe636800700 -1 osd.6 108 *** Got signal Terminated *** 2024-04-23T15:00:14.624 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:14 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:14.355+0000 7fe636800700 -1 osd.6 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T15:00:14.785 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:14 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:14.434+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:14.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:14 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:14.434+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.214584+0000 front 2024-04-23T14:59:33.214522+0000 (oldest deadline 2024-04-23T15:00:02.015036+0000) 2024-04-23T15:00:15.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:14 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:14.734+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.474272+0000 front 2024-04-23T14:59:34.474414+0000 (oldest deadline 2024-04-23T14:59:58.574412+0000) 2024-04-23T15:00:15.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:14 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:14.734+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.875719+0000 front 2024-04-23T14:59:43.875508+0000 (oldest deadline 2024-04-23T15:00:06.775587+0000) 2024-04-23T15:00:15.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:14 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:14.775+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:15.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:14 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:14.775+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:15.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:14 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:14.775+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:15.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:14 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:14.874+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:15.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:14 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:14.874+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:15.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:14 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:14.874+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:15.374 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:15 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:15.008+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.937927+0000 front 2024-04-23T14:59:23.938084+0000 (oldest deadline 2024-04-23T14:59:49.838393+0000) 2024-04-23T15:00:15.374 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:15 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:15.008+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:35.142219+0000 front 2024-04-23T14:59:35.142323+0000 (oldest deadline 2024-04-23T14:59:59.842444+0000) 2024-04-23T15:00:15.374 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:15 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:15.008+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.545809+0000 front 2024-04-23T14:59:44.545911+0000 (oldest deadline 2024-04-23T15:00:08.645986+0000) 2024-04-23T15:00:15.785 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:15 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:15.432+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:15.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:15 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:15.432+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.214584+0000 front 2024-04-23T14:59:33.214522+0000 (oldest deadline 2024-04-23T15:00:02.015036+0000) 2024-04-23T15:00:15.873 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:15 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:15.377+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:15.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:15 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:15.377+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:15.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:15 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:15.377+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:15.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:15 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:15.377+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.079477+0000 front 2024-04-23T14:59:53.079548+0000 (oldest deadline 2024-04-23T15:00:14.779829+0000) 2024-04-23T15:00:16.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:15 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:15.710+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.474272+0000 front 2024-04-23T14:59:34.474414+0000 (oldest deadline 2024-04-23T14:59:58.574412+0000) 2024-04-23T15:00:16.123 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:15 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:15.710+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.875719+0000 front 2024-04-23T14:59:43.875508+0000 (oldest deadline 2024-04-23T15:00:06.775587+0000) 2024-04-23T15:00:16.123 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:15 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:15.710+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:49.577162+0000 front 2024-04-23T14:59:49.577074+0000 (oldest deadline 2024-04-23T15:00:15.476943+0000) 2024-04-23T15:00:16.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:15 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:15.742+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:16.124 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:15 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:15.742+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:16.124 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:15 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:15.742+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:16.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:15 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:15.880+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:16.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:15 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:15.880+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:16.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:15 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:15.880+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:16.374 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:15 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:15.991+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.937927+0000 front 2024-04-23T14:59:23.938084+0000 (oldest deadline 2024-04-23T14:59:49.838393+0000) 2024-04-23T15:00:16.374 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:15 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:15.991+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:35.142219+0000 front 2024-04-23T14:59:35.142323+0000 (oldest deadline 2024-04-23T14:59:59.842444+0000) 2024-04-23T15:00:16.374 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:15 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:15.991+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.545809+0000 front 2024-04-23T14:59:44.545911+0000 (oldest deadline 2024-04-23T15:00:08.645986+0000) 2024-04-23T15:00:16.785 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:16 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:16.414+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:16.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:16 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:16.414+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.214584+0000 front 2024-04-23T14:59:33.214522+0000 (oldest deadline 2024-04-23T15:00:02.015036+0000) 2024-04-23T15:00:16.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:16 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:16.414+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:51.416760+0000 front 2024-04-23T14:59:51.416700+0000 (oldest deadline 2024-04-23T15:00:15.516780+0000) 2024-04-23T15:00:16.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:16 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:16.411+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:16.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:16 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:16.411+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:16.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:16 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:16.411+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:16.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:16 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:16.411+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.079477+0000 front 2024-04-23T14:59:53.079548+0000 (oldest deadline 2024-04-23T15:00:14.779829+0000) 2024-04-23T15:00:17.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:16 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:16.672+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.474272+0000 front 2024-04-23T14:59:34.474414+0000 (oldest deadline 2024-04-23T14:59:58.574412+0000) 2024-04-23T15:00:17.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:16 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:16.672+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.875719+0000 front 2024-04-23T14:59:43.875508+0000 (oldest deadline 2024-04-23T15:00:06.775587+0000) 2024-04-23T15:00:17.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:16 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:16.672+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:49.577162+0000 front 2024-04-23T14:59:49.577074+0000 (oldest deadline 2024-04-23T15:00:15.476943+0000) 2024-04-23T15:00:17.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:16 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:16.757+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:17.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:16 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:16.757+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:17.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:16 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:16.757+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:17.285 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:16 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:16.920+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:17.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:16 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:16.920+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:17.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:16 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:16.920+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:17.374 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:17 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:17.032+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.937927+0000 front 2024-04-23T14:59:23.938084+0000 (oldest deadline 2024-04-23T14:59:49.838393+0000) 2024-04-23T15:00:17.374 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:17 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:17.032+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:35.142219+0000 front 2024-04-23T14:59:35.142323+0000 (oldest deadline 2024-04-23T14:59:59.842444+0000) 2024-04-23T15:00:17.374 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:17 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:17.032+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.545809+0000 front 2024-04-23T14:59:44.545911+0000 (oldest deadline 2024-04-23T15:00:08.645986+0000) 2024-04-23T15:00:17.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:17 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:17.418+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:17.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:17 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:17.418+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.214584+0000 front 2024-04-23T14:59:33.214522+0000 (oldest deadline 2024-04-23T15:00:02.015036+0000) 2024-04-23T15:00:17.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:17 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:17.418+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:51.416760+0000 front 2024-04-23T14:59:51.416700+0000 (oldest deadline 2024-04-23T15:00:15.516780+0000) 2024-04-23T15:00:17.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:17 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:17.457+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:17.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:17 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:17.457+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:17.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:17 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:17.457+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:17.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:17 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:17.457+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.079477+0000 front 2024-04-23T14:59:53.079548+0000 (oldest deadline 2024-04-23T15:00:14.779829+0000) 2024-04-23T15:00:18.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:17 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:17.637+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.474272+0000 front 2024-04-23T14:59:34.474414+0000 (oldest deadline 2024-04-23T14:59:58.574412+0000) 2024-04-23T15:00:18.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:17 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:17.637+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.875719+0000 front 2024-04-23T14:59:43.875508+0000 (oldest deadline 2024-04-23T15:00:06.775587+0000) 2024-04-23T15:00:18.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:17 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:17.637+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:49.577162+0000 front 2024-04-23T14:59:49.577074+0000 (oldest deadline 2024-04-23T15:00:15.476943+0000) 2024-04-23T15:00:18.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:17 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:17.734+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:18.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:17 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:17.734+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:18.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:17 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:17.734+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:18.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:17 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:17.957+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:18.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:17 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:17.957+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:18.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:17 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:17.957+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:18.373 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:18 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:18.052+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.937927+0000 front 2024-04-23T14:59:23.938084+0000 (oldest deadline 2024-04-23T14:59:49.838393+0000) 2024-04-23T15:00:18.374 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:18 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:18.052+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:35.142219+0000 front 2024-04-23T14:59:35.142323+0000 (oldest deadline 2024-04-23T14:59:59.842444+0000) 2024-04-23T15:00:18.374 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:18 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:18.052+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.545809+0000 front 2024-04-23T14:59:44.545911+0000 (oldest deadline 2024-04-23T15:00:08.645986+0000) 2024-04-23T15:00:18.768 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:18 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:18.450+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:18.768 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:18 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:18.450+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:18.769 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:18 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:18.450+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:18.769 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:18 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:18.450+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.079477+0000 front 2024-04-23T14:59:53.079548+0000 (oldest deadline 2024-04-23T15:00:14.779829+0000) 2024-04-23T15:00:18.785 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:18 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:18.431+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:18.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:18 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:18.431+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.214584+0000 front 2024-04-23T14:59:33.214522+0000 (oldest deadline 2024-04-23T15:00:02.015036+0000) 2024-04-23T15:00:18.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:18 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:18.431+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:51.416760+0000 front 2024-04-23T14:59:51.416700+0000 (oldest deadline 2024-04-23T15:00:15.516780+0000) 2024-04-23T15:00:19.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:18 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:18.647+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.474272+0000 front 2024-04-23T14:59:34.474414+0000 (oldest deadline 2024-04-23T14:59:58.574412+0000) 2024-04-23T15:00:19.123 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:18 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:18.647+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.875719+0000 front 2024-04-23T14:59:43.875508+0000 (oldest deadline 2024-04-23T15:00:06.775587+0000) 2024-04-23T15:00:19.123 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:18 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:18.647+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:49.577162+0000 front 2024-04-23T14:59:49.577074+0000 (oldest deadline 2024-04-23T15:00:15.476943+0000) 2024-04-23T15:00:19.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:18 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:18.763+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:19.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:18 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:18.763+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:19.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:18 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:18.763+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:19.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:18 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:18.997+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:19.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:18 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:18.997+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:19.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:18 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:18.997+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:19.374 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:19 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:19.090+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.937927+0000 front 2024-04-23T14:59:23.938084+0000 (oldest deadline 2024-04-23T14:59:49.838393+0000) 2024-04-23T15:00:19.374 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:19 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:19.090+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:35.142219+0000 front 2024-04-23T14:59:35.142323+0000 (oldest deadline 2024-04-23T14:59:59.842444+0000) 2024-04-23T15:00:19.374 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:19 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:19.090+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.545809+0000 front 2024-04-23T14:59:44.545911+0000 (oldest deadline 2024-04-23T15:00:08.645986+0000) 2024-04-23T15:00:19.374 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:19 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6[53689]: 2024-04-23T15:00:19.090+0000 7fe632000700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:52.748635+0000 front 2024-04-23T14:59:52.748557+0000 (oldest deadline 2024-04-23T15:00:18.550539+0000) 2024-04-23T15:00:19.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:19 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:19.469+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:19.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:19 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:19.469+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.214584+0000 front 2024-04-23T14:59:33.214522+0000 (oldest deadline 2024-04-23T15:00:02.015036+0000) 2024-04-23T15:00:19.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:19 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:19.469+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:51.416760+0000 front 2024-04-23T14:59:51.416700+0000 (oldest deadline 2024-04-23T15:00:15.516780+0000) 2024-04-23T15:00:19.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:19 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:19.442+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:19.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:19 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:19.442+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:19.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:19 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:19.442+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:19.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:19 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:19.442+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.079477+0000 front 2024-04-23T14:59:53.079548+0000 (oldest deadline 2024-04-23T15:00:14.779829+0000) 2024-04-23T15:00:20.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:19 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:19.654+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.474272+0000 front 2024-04-23T14:59:34.474414+0000 (oldest deadline 2024-04-23T14:59:58.574412+0000) 2024-04-23T15:00:20.125 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:19 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:19.654+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.875719+0000 front 2024-04-23T14:59:43.875508+0000 (oldest deadline 2024-04-23T15:00:06.775587+0000) 2024-04-23T15:00:20.125 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:19 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:19.654+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:49.577162+0000 front 2024-04-23T14:59:49.577074+0000 (oldest deadline 2024-04-23T15:00:15.476943+0000) 2024-04-23T15:00:20.126 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:19 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:19.724+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:20.126 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:19 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:19.724+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:20.126 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:19 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:19.724+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:20.264 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:19 smithi028 podman[94137]: 2024-04-23 15:00:19.962000232 +0000 UTC m=+5.733092692 container died 4027f1fa6f0915a5e4ba54c597736fb67d394909ac4582ff57aa678f1387a2ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-23T15:00:20.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:19 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:19.995+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:20.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:19 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:19.995+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:20.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:19 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:19.995+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:20.624 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:20 smithi028 podman[94137]: 2024-04-23 15:00:20.374870245 +0000 UTC m=+6.145962714 container cleanup 4027f1fa6f0915a5e4ba54c597736fb67d394909ac4582ff57aa678f1387a2ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True) 2024-04-23T15:00:20.624 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:20 smithi028 bash[94137]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6 2024-04-23T15:00:20.625 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:20 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:20.403+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:20.625 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:20 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:20.403+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:20.625 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:20 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:20.403+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:20.625 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:20 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:20.403+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.079477+0000 front 2024-04-23T14:59:53.079548+0000 (oldest deadline 2024-04-23T15:00:14.779829+0000) 2024-04-23T15:00:20.785 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:20 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:20.505+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:20.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:20 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:20.505+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.214584+0000 front 2024-04-23T14:59:33.214522+0000 (oldest deadline 2024-04-23T15:00:02.015036+0000) 2024-04-23T15:00:20.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:20 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:20.505+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:51.416760+0000 front 2024-04-23T14:59:51.416700+0000 (oldest deadline 2024-04-23T15:00:15.516780+0000) 2024-04-23T15:00:21.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:20 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:20.699+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.474272+0000 front 2024-04-23T14:59:34.474414+0000 (oldest deadline 2024-04-23T14:59:58.574412+0000) 2024-04-23T15:00:21.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:20 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:20.699+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.875719+0000 front 2024-04-23T14:59:43.875508+0000 (oldest deadline 2024-04-23T15:00:06.775587+0000) 2024-04-23T15:00:21.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:20 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:20.699+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:49.577162+0000 front 2024-04-23T14:59:49.577074+0000 (oldest deadline 2024-04-23T15:00:15.476943+0000) 2024-04-23T15:00:21.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:20 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:20.712+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:21.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:20 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:20.712+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:21.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:20 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:20.712+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:21.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:20 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:20.712+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.654392+0000 front 2024-04-23T14:59:53.654228+0000 (oldest deadline 2024-04-23T15:00:20.054780+0000) 2024-04-23T15:00:21.124 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:20 smithi028 podman[94169]: 2024-04-23 15:00:20.684399819 +0000 UTC m=+0.705405440 container remove 4027f1fa6f0915a5e4ba54c597736fb67d394909ac4582ff57aa678f1387a2ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-23T15:00:21.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:21 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:21.016+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:21.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:21 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:21.016+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:21.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:21 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:21.016+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:21.743 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:21 smithi028 podman[94257]: 2024-04-23 15:00:21.436593893 +0000 UTC m=+0.035387010 image pull 76e77f280c74312ca1add9b6019a4a921a516f91795d85bae00a0c8ca9b1c62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd 2024-04-23T15:00:21.743 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:21 smithi028 podman[94257]: 2024-04-23 15:00:21.568439475 +0000 UTC m=+0.167232629 container create 7286a022e3347c0e7a6cf5bdcecd49cd3b6495b4c30511cd5c76a966bad1abcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T15:00:21.743 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:21 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:21.442+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:21.743 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:21 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:21.442+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:21.743 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:21 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:21.442+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:21.743 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:21 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:21.442+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.079477+0000 front 2024-04-23T14:59:53.079548+0000 (oldest deadline 2024-04-23T15:00:14.779829+0000) 2024-04-23T15:00:21.785 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:21 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:21.468+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:21.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:21 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:21.468+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.214584+0000 front 2024-04-23T14:59:33.214522+0000 (oldest deadline 2024-04-23T15:00:02.015036+0000) 2024-04-23T15:00:21.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:21 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:21.468+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:51.416760+0000 front 2024-04-23T14:59:51.416700+0000 (oldest deadline 2024-04-23T15:00:15.516780+0000) 2024-04-23T15:00:22.080 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:21 smithi028 podman[94257]: 2024-04-23 15:00:21.847337749 +0000 UTC m=+0.446130909 container init 7286a022e3347c0e7a6cf5bdcecd49cd3b6495b4c30511cd5c76a966bad1abcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6-deactivate, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-23T15:00:22.081 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:21 smithi028 podman[94257]: 2024-04-23 15:00:21.85924518 +0000 UTC m=+0.458038340 container start 7286a022e3347c0e7a6cf5bdcecd49cd3b6495b4c30511cd5c76a966bad1abcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6-deactivate, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T15:00:22.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:21 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:21.681+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.474272+0000 front 2024-04-23T14:59:34.474414+0000 (oldest deadline 2024-04-23T14:59:58.574412+0000) 2024-04-23T15:00:22.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:21 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:21.681+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.875719+0000 front 2024-04-23T14:59:43.875508+0000 (oldest deadline 2024-04-23T15:00:06.775587+0000) 2024-04-23T15:00:22.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:21 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:21.681+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:49.577162+0000 front 2024-04-23T14:59:49.577074+0000 (oldest deadline 2024-04-23T15:00:15.476943+0000) 2024-04-23T15:00:22.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:21 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:21.678+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:22.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:21 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:21.678+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:22.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:21 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:21.678+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:22.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:21 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:21.678+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.654392+0000 front 2024-04-23T14:59:53.654228+0000 (oldest deadline 2024-04-23T15:00:20.054780+0000) 2024-04-23T15:00:22.335 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:22 smithi028 podman[94257]: 2024-04-23 15:00:22.079345502 +0000 UTC m=+0.678138706 container attach 7286a022e3347c0e7a6cf5bdcecd49cd3b6495b4c30511cd5c76a966bad1abcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6-deactivate, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T15:00:22.463 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:22 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:22.043+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:22.463 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:22 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:22.043+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:22.463 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:22 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:22.043+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:22.623 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 23 15:00:22 smithi028 podman[94257]: 2024-04-23 15:00:22.3369226 +0000 UTC m=+0.935715794 container died 7286a022e3347c0e7a6cf5bdcecd49cd3b6495b4c30511cd5c76a966bad1abcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-23T15:00:22.624 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:22 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:22.403+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:22.624 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:22 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:22.403+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:22.624 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:22 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:22.403+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:22.624 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:22 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:22.403+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.079477+0000 front 2024-04-23T14:59:53.079548+0000 (oldest deadline 2024-04-23T15:00:14.779829+0000) 2024-04-23T15:00:22.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:22 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:22.463+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:22.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:22 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:22.463+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.214584+0000 front 2024-04-23T14:59:33.214522+0000 (oldest deadline 2024-04-23T15:00:02.015036+0000) 2024-04-23T15:00:22.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:22 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:22.463+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:51.416760+0000 front 2024-04-23T14:59:51.416700+0000 (oldest deadline 2024-04-23T15:00:15.516780+0000) 2024-04-23T15:00:23.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:22 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:22.709+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.474272+0000 front 2024-04-23T14:59:34.474414+0000 (oldest deadline 2024-04-23T14:59:58.574412+0000) 2024-04-23T15:00:23.123 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:22 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:22.709+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.875719+0000 front 2024-04-23T14:59:43.875508+0000 (oldest deadline 2024-04-23T15:00:06.775587+0000) 2024-04-23T15:00:23.123 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:22 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:22.709+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:49.577162+0000 front 2024-04-23T14:59:49.577074+0000 (oldest deadline 2024-04-23T15:00:15.476943+0000) 2024-04-23T15:00:23.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:22 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:22.685+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:23.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:22 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:22.685+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:23.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:22 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:22.685+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:23.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:22 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:22.685+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.654392+0000 front 2024-04-23T14:59:53.654228+0000 (oldest deadline 2024-04-23T15:00:20.054780+0000) 2024-04-23T15:00:23.227 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.6.service' 2024-04-23T15:00:23.350 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T15:00:23.350 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-23T15:00:23.351 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-23T15:00:23.351 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.7 2024-04-23T15:00:23.485 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:23 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:23.044+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:23.486 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:23 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:23.044+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:23.486 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:23 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:23.044+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:23.656 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:23 smithi038 systemd[1]: Stopping Ceph osd.7 for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T15:00:23.657 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:23 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:23.656+0000 7f705b3c4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T15:00:23.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:23 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:23.486+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:23.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:23 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:23.486+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.214584+0000 front 2024-04-23T14:59:33.214522+0000 (oldest deadline 2024-04-23T15:00:02.015036+0000) 2024-04-23T15:00:23.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:23 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:23.486+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:51.416760+0000 front 2024-04-23T14:59:51.416700+0000 (oldest deadline 2024-04-23T15:00:15.516780+0000) 2024-04-23T15:00:23.873 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:23 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:23.440+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:23.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:23 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:23.440+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:23.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:23 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:23.440+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:23.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:23 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:23.440+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.079477+0000 front 2024-04-23T14:59:53.079548+0000 (oldest deadline 2024-04-23T15:00:14.779829+0000) 2024-04-23T15:00:24.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:23 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:23.666+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:24.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:23 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:23.666+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:24.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:23 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:23.666+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:24.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:23 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:23.666+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.654392+0000 front 2024-04-23T14:59:53.654228+0000 (oldest deadline 2024-04-23T15:00:20.054780+0000) 2024-04-23T15:00:24.123 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:23 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:23.656+0000 7f705b3c4700 -1 osd.7 108 *** Got signal Terminated *** 2024-04-23T15:00:24.123 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:23 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:23.656+0000 7f705b3c4700 -1 osd.7 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T15:00:24.123 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:23 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:23.674+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.474272+0000 front 2024-04-23T14:59:34.474414+0000 (oldest deadline 2024-04-23T14:59:58.574412+0000) 2024-04-23T15:00:24.123 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:23 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:23.674+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.875719+0000 front 2024-04-23T14:59:43.875508+0000 (oldest deadline 2024-04-23T15:00:06.775587+0000) 2024-04-23T15:00:24.123 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:23 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:23.674+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:49.577162+0000 front 2024-04-23T14:59:49.577074+0000 (oldest deadline 2024-04-23T15:00:15.476943+0000) 2024-04-23T15:00:24.490 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:24 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:24.075+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:24.490 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:24 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:24.075+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:24.490 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:24 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:24.075+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:24.491 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:24 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:24.075+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:24.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:24 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:24.490+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:24.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:24 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:24.490+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.214584+0000 front 2024-04-23T14:59:33.214522+0000 (oldest deadline 2024-04-23T15:00:02.015036+0000) 2024-04-23T15:00:24.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:24 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:24.490+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:51.416760+0000 front 2024-04-23T14:59:51.416700+0000 (oldest deadline 2024-04-23T15:00:15.516780+0000) 2024-04-23T15:00:24.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:24 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:24.490+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:01.418013+0000 front 2024-04-23T15:00:01.417867+0000 (oldest deadline 2024-04-23T15:00:23.717817+0000) 2024-04-23T15:00:24.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:24 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:24.456+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:24.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:24 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:24.456+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:24.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:24 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:24.456+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:24.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:24 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:24.456+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.079477+0000 front 2024-04-23T14:59:53.079548+0000 (oldest deadline 2024-04-23T15:00:14.779829+0000) 2024-04-23T15:00:25.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:24 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:24.656+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.474272+0000 front 2024-04-23T14:59:34.474414+0000 (oldest deadline 2024-04-23T14:59:58.574412+0000) 2024-04-23T15:00:25.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:24 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:24.656+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.875719+0000 front 2024-04-23T14:59:43.875508+0000 (oldest deadline 2024-04-23T15:00:06.775587+0000) 2024-04-23T15:00:25.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:24 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:24.656+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:49.577162+0000 front 2024-04-23T14:59:49.577074+0000 (oldest deadline 2024-04-23T15:00:15.476943+0000) 2024-04-23T15:00:25.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:24 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:24.674+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:25.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:24 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:24.674+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:25.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:24 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:24.674+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:25.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:24 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:24.674+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.654392+0000 front 2024-04-23T14:59:53.654228+0000 (oldest deadline 2024-04-23T15:00:20.054780+0000) 2024-04-23T15:00:25.441 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:25 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:25.079+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:25.441 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:25 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:25.079+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:25.441 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:25 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:25.079+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:25.441 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:25 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:25.079+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:25.785 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:25 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:25.441+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:25.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:25 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:25.441+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.214584+0000 front 2024-04-23T14:59:33.214522+0000 (oldest deadline 2024-04-23T15:00:02.015036+0000) 2024-04-23T15:00:25.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:25 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:25.441+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:51.416760+0000 front 2024-04-23T14:59:51.416700+0000 (oldest deadline 2024-04-23T15:00:15.516780+0000) 2024-04-23T15:00:25.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:25 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:25.441+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:01.418013+0000 front 2024-04-23T15:00:01.417867+0000 (oldest deadline 2024-04-23T15:00:23.717817+0000) 2024-04-23T15:00:25.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:25 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:25.484+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:25.875 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:25 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:25.484+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:25.875 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:25 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:25.484+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:25.875 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:25 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:25.484+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.079477+0000 front 2024-04-23T14:59:53.079548+0000 (oldest deadline 2024-04-23T15:00:14.779829+0000) 2024-04-23T15:00:25.875 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:25 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:25.484+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.482358+0000 front 2024-04-23T14:59:59.482241+0000 (oldest deadline 2024-04-23T15:00:24.683768+0000) 2024-04-23T15:00:26.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:25 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:25.675+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.474272+0000 front 2024-04-23T14:59:34.474414+0000 (oldest deadline 2024-04-23T14:59:58.574412+0000) 2024-04-23T15:00:26.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:25 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:25.675+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.875719+0000 front 2024-04-23T14:59:43.875508+0000 (oldest deadline 2024-04-23T15:00:06.775587+0000) 2024-04-23T15:00:26.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:25 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:25.675+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:49.577162+0000 front 2024-04-23T14:59:49.577074+0000 (oldest deadline 2024-04-23T15:00:15.476943+0000) 2024-04-23T15:00:26.123 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:25 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:25.675+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:02.478557+0000 front 2024-04-23T15:00:02.478590+0000 (oldest deadline 2024-04-23T15:00:24.778773+0000) 2024-04-23T15:00:26.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:25 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:25.676+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:26.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:25 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:25.676+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:26.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:25 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:25.676+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:26.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:25 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:25.676+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.654392+0000 front 2024-04-23T14:59:53.654228+0000 (oldest deadline 2024-04-23T15:00:20.054780+0000) 2024-04-23T15:00:26.418 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:26 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:26.047+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:26.419 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:26 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:26.047+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:26.419 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:26 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:26.047+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:26.419 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:26 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:26.047+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:26.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:26 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:26.418+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:26.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:26 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:26.418+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.214584+0000 front 2024-04-23T14:59:33.214522+0000 (oldest deadline 2024-04-23T15:00:02.015036+0000) 2024-04-23T15:00:26.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:26 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:26.418+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:51.416760+0000 front 2024-04-23T14:59:51.416700+0000 (oldest deadline 2024-04-23T15:00:15.516780+0000) 2024-04-23T15:00:26.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:26 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:26.418+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:01.418013+0000 front 2024-04-23T15:00:01.417867+0000 (oldest deadline 2024-04-23T15:00:23.717817+0000) 2024-04-23T15:00:26.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:26 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:26.463+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:26.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:26 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:26.463+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:26.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:26 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:26.463+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:26.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:26 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:26.463+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.079477+0000 front 2024-04-23T14:59:53.079548+0000 (oldest deadline 2024-04-23T15:00:14.779829+0000) 2024-04-23T15:00:26.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:26 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:26.463+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.482358+0000 front 2024-04-23T14:59:59.482241+0000 (oldest deadline 2024-04-23T15:00:24.683768+0000) 2024-04-23T15:00:27.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:26 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:26.701+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.474272+0000 front 2024-04-23T14:59:34.474414+0000 (oldest deadline 2024-04-23T14:59:58.574412+0000) 2024-04-23T15:00:27.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:26 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:26.701+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.875719+0000 front 2024-04-23T14:59:43.875508+0000 (oldest deadline 2024-04-23T15:00:06.775587+0000) 2024-04-23T15:00:27.123 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:26 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:26.701+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:49.577162+0000 front 2024-04-23T14:59:49.577074+0000 (oldest deadline 2024-04-23T15:00:15.476943+0000) 2024-04-23T15:00:27.123 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:26 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:26.701+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:02.478557+0000 front 2024-04-23T15:00:02.478590+0000 (oldest deadline 2024-04-23T15:00:24.778773+0000) 2024-04-23T15:00:27.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:26 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:26.636+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:27.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:26 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:26.636+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:27.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:26 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:26.636+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:27.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:26 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:26.636+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.654392+0000 front 2024-04-23T14:59:53.654228+0000 (oldest deadline 2024-04-23T15:00:20.054780+0000) 2024-04-23T15:00:27.425 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:27 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:27.056+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:27.426 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:27 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:27.056+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:27.426 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:27 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:27.056+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:27.426 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:27 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:27.056+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:27.739 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:27 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:27.737+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.474272+0000 front 2024-04-23T14:59:34.474414+0000 (oldest deadline 2024-04-23T14:59:58.574412+0000) 2024-04-23T15:00:27.739 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:27 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:27.737+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.875719+0000 front 2024-04-23T14:59:43.875508+0000 (oldest deadline 2024-04-23T15:00:06.775587+0000) 2024-04-23T15:00:27.739 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:27 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:27.737+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:49.577162+0000 front 2024-04-23T14:59:49.577074+0000 (oldest deadline 2024-04-23T15:00:15.476943+0000) 2024-04-23T15:00:27.739 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:27 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7[50632]: 2024-04-23T15:00:27.737+0000 7f70539b0700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:02.478557+0000 front 2024-04-23T15:00:02.478590+0000 (oldest deadline 2024-04-23T15:00:24.778773+0000) 2024-04-23T15:00:27.739 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:27 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:27.645+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:27.740 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:27 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:27.645+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:27.740 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:27 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:27.645+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:27.740 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:27 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:27.645+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.654392+0000 front 2024-04-23T14:59:53.654228+0000 (oldest deadline 2024-04-23T15:00:20.054780+0000) 2024-04-23T15:00:27.740 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:27 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:27.645+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:02.256240+0000 front 2024-04-23T15:00:02.256270+0000 (oldest deadline 2024-04-23T15:00:27.556454+0000) 2024-04-23T15:00:27.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:27 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:27.426+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:27.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:27 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:27.426+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.214584+0000 front 2024-04-23T14:59:33.214522+0000 (oldest deadline 2024-04-23T15:00:02.015036+0000) 2024-04-23T15:00:27.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:27 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:27.426+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:51.416760+0000 front 2024-04-23T14:59:51.416700+0000 (oldest deadline 2024-04-23T15:00:15.516780+0000) 2024-04-23T15:00:27.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:27 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:27.426+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:01.418013+0000 front 2024-04-23T15:00:01.417867+0000 (oldest deadline 2024-04-23T15:00:23.717817+0000) 2024-04-23T15:00:27.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:27 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:27.480+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:27.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:27 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:27.480+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:27.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:27 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:27.480+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:27.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:27 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:27.480+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.079477+0000 front 2024-04-23T14:59:53.079548+0000 (oldest deadline 2024-04-23T15:00:14.779829+0000) 2024-04-23T15:00:27.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:27 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:27.480+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.482358+0000 front 2024-04-23T14:59:59.482241+0000 (oldest deadline 2024-04-23T15:00:24.683768+0000) 2024-04-23T15:00:28.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:28 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:28.032+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:28.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:28 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:28.032+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:28.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:28 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:28.032+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:28.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:28 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:28.032+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:28.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:28 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:28.451+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:28.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:28 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:28.451+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.214584+0000 front 2024-04-23T14:59:33.214522+0000 (oldest deadline 2024-04-23T15:00:02.015036+0000) 2024-04-23T15:00:28.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:28 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:28.451+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:51.416760+0000 front 2024-04-23T14:59:51.416700+0000 (oldest deadline 2024-04-23T15:00:15.516780+0000) 2024-04-23T15:00:28.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:28 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:28.451+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:01.418013+0000 front 2024-04-23T15:00:01.417867+0000 (oldest deadline 2024-04-23T15:00:23.717817+0000) 2024-04-23T15:00:28.873 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:28 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:28.501+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:28.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:28 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:28.501+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:28.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:28 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:28.501+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:28.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:28 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:28.501+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.079477+0000 front 2024-04-23T14:59:53.079548+0000 (oldest deadline 2024-04-23T15:00:14.779829+0000) 2024-04-23T15:00:28.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:28 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:28.501+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.482358+0000 front 2024-04-23T14:59:59.482241+0000 (oldest deadline 2024-04-23T15:00:24.683768+0000) 2024-04-23T15:00:29.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:28 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:28.623+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:29.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:28 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:28.623+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:29.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:28 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:28.623+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:29.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:28 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:28.623+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.654392+0000 front 2024-04-23T14:59:53.654228+0000 (oldest deadline 2024-04-23T15:00:20.054780+0000) 2024-04-23T15:00:29.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:28 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:28.623+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:02.256240+0000 front 2024-04-23T15:00:02.256270+0000 (oldest deadline 2024-04-23T15:00:27.556454+0000) 2024-04-23T15:00:29.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:29 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:29.026+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:29.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:29 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:29.026+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:29.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:29 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:29.026+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:29.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:29 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:29.026+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:29.483 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:29 smithi038 podman[69595]: 2024-04-23 15:00:29.191180151 +0000 UTC m=+5.641536850 container died aa1a593aa4472affb6faf355ba3e75d5527572452698e0a590abc77552840bcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-23T15:00:29.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:29 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:29.440+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:29.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:29 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:29.440+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.214584+0000 front 2024-04-23T14:59:33.214522+0000 (oldest deadline 2024-04-23T15:00:02.015036+0000) 2024-04-23T15:00:29.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:29 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:29.440+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:51.416760+0000 front 2024-04-23T14:59:51.416700+0000 (oldest deadline 2024-04-23T15:00:15.516780+0000) 2024-04-23T15:00:29.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:29 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:29.440+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:01.418013+0000 front 2024-04-23T15:00:01.417867+0000 (oldest deadline 2024-04-23T15:00:23.717817+0000) 2024-04-23T15:00:29.823 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:29 smithi038 podman[69595]: 2024-04-23 15:00:29.581208652 +0000 UTC m=+6.031565341 container cleanup aa1a593aa4472affb6faf355ba3e75d5527572452698e0a590abc77552840bcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD) 2024-04-23T15:00:29.824 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:29 smithi038 bash[69595]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7 2024-04-23T15:00:29.824 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:29 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:29.648+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:29.824 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:29 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:29.648+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:29.824 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:29 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:29.648+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:29.824 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:29 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:29.648+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.654392+0000 front 2024-04-23T14:59:53.654228+0000 (oldest deadline 2024-04-23T15:00:20.054780+0000) 2024-04-23T15:00:29.824 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:29 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:29.648+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:02.256240+0000 front 2024-04-23T15:00:02.256270+0000 (oldest deadline 2024-04-23T15:00:27.556454+0000) 2024-04-23T15:00:29.873 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:29 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:29.457+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:29.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:29 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:29.457+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:29.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:29 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:29.457+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:29.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:29 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:29.457+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.079477+0000 front 2024-04-23T14:59:53.079548+0000 (oldest deadline 2024-04-23T15:00:14.779829+0000) 2024-04-23T15:00:29.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:29 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:29.457+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.482358+0000 front 2024-04-23T14:59:59.482241+0000 (oldest deadline 2024-04-23T15:00:24.683768+0000) 2024-04-23T15:00:30.122 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:29 smithi038 podman[69613]: 2024-04-23 15:00:29.824269457 +0000 UTC m=+0.624809242 container remove aa1a593aa4472affb6faf355ba3e75d5527572452698e0a590abc77552840bcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T15:00:30.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:29 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:29.994+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:30.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:29 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:29.994+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:30.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:29 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:29.994+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:30.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:29 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:29.994+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:30.460 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:30 smithi038 podman[69705]: 2024-04-23 15:00:30.14175185 +0000 UTC m=+0.114930042 container create bf084b253bd6afbac0321c0065c14da54f26dab666c5a77c756cdb1137fac3db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-23T15:00:30.460 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:30 smithi038 podman[69705]: 2024-04-23 15:00:30.045180864 +0000 UTC m=+0.018359068 image pull 76e77f280c74312ca1add9b6019a4a921a516f91795d85bae00a0c8ca9b1c62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd 2024-04-23T15:00:30.460 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:30 smithi038 podman[69705]: 2024-04-23 15:00:30.386006343 +0000 UTC m=+0.359184536 container init bf084b253bd6afbac0321c0065c14da54f26dab666c5a77c756cdb1137fac3db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-23T15:00:30.461 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:30 smithi038 podman[69705]: 2024-04-23 15:00:30.389101435 +0000 UTC m=+0.362279628 container start bf084b253bd6afbac0321c0065c14da54f26dab666c5a77c756cdb1137fac3db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T15:00:30.461 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:30 smithi038 podman[69705]: 2024-04-23 15:00:30.460725129 +0000 UTC m=+0.433903335 container attach bf084b253bd6afbac0321c0065c14da54f26dab666c5a77c756cdb1137fac3db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-23T15:00:30.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:30 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:30.400+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:30.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:30 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:30.400+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.214584+0000 front 2024-04-23T14:59:33.214522+0000 (oldest deadline 2024-04-23T15:00:02.015036+0000) 2024-04-23T15:00:30.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:30 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:30.400+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:51.416760+0000 front 2024-04-23T14:59:51.416700+0000 (oldest deadline 2024-04-23T15:00:15.516780+0000) 2024-04-23T15:00:30.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:30 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:30.400+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:01.418013+0000 front 2024-04-23T15:00:01.417867+0000 (oldest deadline 2024-04-23T15:00:23.717817+0000) 2024-04-23T15:00:30.872 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 23 15:00:30 smithi038 podman[69705]: 2024-04-23 15:00:30.693580888 +0000 UTC m=+0.666759083 container died bf084b253bd6afbac0321c0065c14da54f26dab666c5a77c756cdb1137fac3db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True) 2024-04-23T15:00:30.872 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:30 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:30.616+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:30.873 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:30 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:30.616+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:30.873 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:30 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:30.616+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:30.873 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:30 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:30.616+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.654392+0000 front 2024-04-23T14:59:53.654228+0000 (oldest deadline 2024-04-23T15:00:20.054780+0000) 2024-04-23T15:00:30.873 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:30 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:30.616+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:02.256240+0000 front 2024-04-23T15:00:02.256270+0000 (oldest deadline 2024-04-23T15:00:27.556454+0000) 2024-04-23T15:00:30.873 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:30 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:30.486+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:30.873 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:30 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:30.486+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:30.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:30 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:30.486+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:30.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:30 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:30.486+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.079477+0000 front 2024-04-23T14:59:53.079548+0000 (oldest deadline 2024-04-23T15:00:14.779829+0000) 2024-04-23T15:00:30.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:30 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:30.486+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.482358+0000 front 2024-04-23T14:59:59.482241+0000 (oldest deadline 2024-04-23T15:00:24.683768+0000) 2024-04-23T15:00:31.258 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.7.service' 2024-04-23T15:00:31.299 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T15:00:31.300 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-23T15:00:31.300 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-23T15:00:31.300 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.8 2024-04-23T15:00:31.325 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:31 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:31.038+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:31.326 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:31 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:31.038+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:31.326 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:31 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:31.038+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:31.326 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:31 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:31.038+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:31.593 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:31 smithi105 systemd[1]: Stopping Ceph osd.8 for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T15:00:31.594 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:31 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:31.350+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:31.594 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:31 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:31.350+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.214584+0000 front 2024-04-23T14:59:33.214522+0000 (oldest deadline 2024-04-23T15:00:02.015036+0000) 2024-04-23T15:00:31.594 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:31 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:31.350+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:51.416760+0000 front 2024-04-23T14:59:51.416700+0000 (oldest deadline 2024-04-23T15:00:15.516780+0000) 2024-04-23T15:00:31.594 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:31 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:31.350+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:01.418013+0000 front 2024-04-23T15:00:01.417867+0000 (oldest deadline 2024-04-23T15:00:23.717817+0000) 2024-04-23T15:00:31.872 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:31 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:31.647+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:31.872 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:31 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:31.647+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:31.872 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:31 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:31.647+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:31.872 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:31 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:31.647+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.654392+0000 front 2024-04-23T14:59:53.654228+0000 (oldest deadline 2024-04-23T15:00:20.054780+0000) 2024-04-23T15:00:31.872 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:31 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:31.647+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:02.256240+0000 front 2024-04-23T15:00:02.256270+0000 (oldest deadline 2024-04-23T15:00:27.556454+0000) 2024-04-23T15:00:31.873 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:31 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:31.527+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:31.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:31 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:31.527+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:31.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:31 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:31.527+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:31.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:31 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:31.527+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.079477+0000 front 2024-04-23T14:59:53.079548+0000 (oldest deadline 2024-04-23T15:00:14.779829+0000) 2024-04-23T15:00:31.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:31 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:31.527+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.482358+0000 front 2024-04-23T14:59:59.482241+0000 (oldest deadline 2024-04-23T15:00:24.683768+0000) 2024-04-23T15:00:32.005 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:31 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:31.593+0000 7feb26d70700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T15:00:32.006 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:31 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:31.593+0000 7feb26d70700 -1 osd.8 108 *** Got signal Terminated *** 2024-04-23T15:00:32.006 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:31 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:31.593+0000 7feb26d70700 -1 osd.8 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T15:00:32.285 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:32 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:32.006+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:32.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:32 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:32.006+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:32.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:32 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:32.006+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:32.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:32 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:32.006+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:32.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:32 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:32.345+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:32.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:32 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:32.345+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.214584+0000 front 2024-04-23T14:59:33.214522+0000 (oldest deadline 2024-04-23T15:00:02.015036+0000) 2024-04-23T15:00:32.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:32 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:32.345+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:51.416760+0000 front 2024-04-23T14:59:51.416700+0000 (oldest deadline 2024-04-23T15:00:15.516780+0000) 2024-04-23T15:00:32.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:32 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:32.345+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:01.418013+0000 front 2024-04-23T15:00:01.417867+0000 (oldest deadline 2024-04-23T15:00:23.717817+0000) 2024-04-23T15:00:32.872 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:32 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:32.659+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:32.872 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:32 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:32.659+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:32.872 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:32 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:32.659+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:32.872 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:32 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:32.659+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.654392+0000 front 2024-04-23T14:59:53.654228+0000 (oldest deadline 2024-04-23T15:00:20.054780+0000) 2024-04-23T15:00:32.872 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:32 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:32.659+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:02.256240+0000 front 2024-04-23T15:00:02.256270+0000 (oldest deadline 2024-04-23T15:00:27.556454+0000) 2024-04-23T15:00:32.873 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:32 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:32.659+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:07.557037+0000 front 2024-04-23T15:00:07.557132+0000 (oldest deadline 2024-04-23T15:00:31.656999+0000) 2024-04-23T15:00:32.873 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:32 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:32.508+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:32.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:32 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:32.508+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:32.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:32 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:32.508+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:32.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:32 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:32.508+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.079477+0000 front 2024-04-23T14:59:53.079548+0000 (oldest deadline 2024-04-23T15:00:14.779829+0000) 2024-04-23T15:00:32.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:32 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:32.508+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.482358+0000 front 2024-04-23T14:59:59.482241+0000 (oldest deadline 2024-04-23T15:00:24.683768+0000) 2024-04-23T15:00:33.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:32 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:32.964+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:33.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:32 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:32.964+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:33.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:32 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:32.964+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:33.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:32 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:32.964+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:33.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:32 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:32.964+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:09.809136+0000 front 2024-04-23T15:00:09.808985+0000 (oldest deadline 2024-04-23T15:00:32.709048+0000) 2024-04-23T15:00:33.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:33 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:33.323+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:33.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:33 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:33.323+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.214584+0000 front 2024-04-23T14:59:33.214522+0000 (oldest deadline 2024-04-23T15:00:02.015036+0000) 2024-04-23T15:00:33.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:33 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:33.323+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:51.416760+0000 front 2024-04-23T14:59:51.416700+0000 (oldest deadline 2024-04-23T15:00:15.516780+0000) 2024-04-23T15:00:33.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:33 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:33.323+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:01.418013+0000 front 2024-04-23T15:00:01.417867+0000 (oldest deadline 2024-04-23T15:00:23.717817+0000) 2024-04-23T15:00:33.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:33 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:33.469+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:33.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:33 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:33.469+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:33.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:33 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:33.469+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:33.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:33 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:33.469+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.079477+0000 front 2024-04-23T14:59:53.079548+0000 (oldest deadline 2024-04-23T15:00:14.779829+0000) 2024-04-23T15:00:33.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:33 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:33.469+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.482358+0000 front 2024-04-23T14:59:59.482241+0000 (oldest deadline 2024-04-23T15:00:24.683768+0000) 2024-04-23T15:00:33.875 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:33 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:33.469+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:08.686898+0000 front 2024-04-23T15:00:08.687026+0000 (oldest deadline 2024-04-23T15:00:33.387132+0000) 2024-04-23T15:00:34.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:33 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:33.647+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:34.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:33 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:33.647+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:34.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:33 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:33.647+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:34.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:33 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:33.647+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.654392+0000 front 2024-04-23T14:59:53.654228+0000 (oldest deadline 2024-04-23T15:00:20.054780+0000) 2024-04-23T15:00:34.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:33 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:33.647+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:02.256240+0000 front 2024-04-23T15:00:02.256270+0000 (oldest deadline 2024-04-23T15:00:27.556454+0000) 2024-04-23T15:00:34.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:33 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:33.647+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:07.557037+0000 front 2024-04-23T15:00:07.557132+0000 (oldest deadline 2024-04-23T15:00:31.656999+0000) 2024-04-23T15:00:34.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:33 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:33.975+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:34.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:33 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:33.975+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:34.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:33 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:33.975+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:34.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:33 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:33.975+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:34.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:33 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:33.975+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:09.809136+0000 front 2024-04-23T15:00:09.808985+0000 (oldest deadline 2024-04-23T15:00:32.709048+0000) 2024-04-23T15:00:34.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:34 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:34.288+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:34.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:34 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:34.288+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.214584+0000 front 2024-04-23T14:59:33.214522+0000 (oldest deadline 2024-04-23T15:00:02.015036+0000) 2024-04-23T15:00:34.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:34 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:34.288+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:51.416760+0000 front 2024-04-23T14:59:51.416700+0000 (oldest deadline 2024-04-23T15:00:15.516780+0000) 2024-04-23T15:00:34.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:34 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:34.288+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:01.418013+0000 front 2024-04-23T15:00:01.417867+0000 (oldest deadline 2024-04-23T15:00:23.717817+0000) 2024-04-23T15:00:34.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:34 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:34.497+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:34.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:34 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:34.497+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:34.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:34 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:34.497+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:34.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:34 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:34.497+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.079477+0000 front 2024-04-23T14:59:53.079548+0000 (oldest deadline 2024-04-23T15:00:14.779829+0000) 2024-04-23T15:00:34.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:34 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:34.497+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.482358+0000 front 2024-04-23T14:59:59.482241+0000 (oldest deadline 2024-04-23T15:00:24.683768+0000) 2024-04-23T15:00:34.875 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:34 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:34.497+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:08.686898+0000 front 2024-04-23T15:00:08.687026+0000 (oldest deadline 2024-04-23T15:00:33.387132+0000) 2024-04-23T15:00:35.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:34 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:34.686+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:35.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:34 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:34.686+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:35.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:34 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:34.686+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:35.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:34 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:34.686+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.654392+0000 front 2024-04-23T14:59:53.654228+0000 (oldest deadline 2024-04-23T15:00:20.054780+0000) 2024-04-23T15:00:35.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:34 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:34.686+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:02.256240+0000 front 2024-04-23T15:00:02.256270+0000 (oldest deadline 2024-04-23T15:00:27.556454+0000) 2024-04-23T15:00:35.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:34 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:34.686+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:07.557037+0000 front 2024-04-23T15:00:07.557132+0000 (oldest deadline 2024-04-23T15:00:31.656999+0000) 2024-04-23T15:00:35.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:35 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:35.019+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:35.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:35 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:35.019+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:35.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:35 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:35.019+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:35.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:35 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:35.019+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:35.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:35 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:35.019+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:09.809136+0000 front 2024-04-23T15:00:09.808985+0000 (oldest deadline 2024-04-23T15:00:32.709048+0000) 2024-04-23T15:00:35.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:35 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:35.319+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:35.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:35 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:35.319+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.214584+0000 front 2024-04-23T14:59:33.214522+0000 (oldest deadline 2024-04-23T15:00:02.015036+0000) 2024-04-23T15:00:35.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:35 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:35.319+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:51.416760+0000 front 2024-04-23T14:59:51.416700+0000 (oldest deadline 2024-04-23T15:00:15.516780+0000) 2024-04-23T15:00:35.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:35 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:35.319+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:01.418013+0000 front 2024-04-23T15:00:01.417867+0000 (oldest deadline 2024-04-23T15:00:23.717817+0000) 2024-04-23T15:00:35.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:35 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:35.319+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:09.320257+0000 front 2024-04-23T15:00:09.320282+0000 (oldest deadline 2024-04-23T15:00:34.620366+0000) 2024-04-23T15:00:35.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:35 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:35.466+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:35.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:35 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:35.466+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:35.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:35 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:35.466+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:35.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:35 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:35.466+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.079477+0000 front 2024-04-23T14:59:53.079548+0000 (oldest deadline 2024-04-23T15:00:14.779829+0000) 2024-04-23T15:00:35.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:35 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:35.466+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.482358+0000 front 2024-04-23T14:59:59.482241+0000 (oldest deadline 2024-04-23T15:00:24.683768+0000) 2024-04-23T15:00:35.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:35 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:35.466+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:08.686898+0000 front 2024-04-23T15:00:08.687026+0000 (oldest deadline 2024-04-23T15:00:33.387132+0000) 2024-04-23T15:00:36.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:35 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:35.715+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:36.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:35 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:35.715+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:36.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:35 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:35.715+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:36.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:35 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:35.715+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.654392+0000 front 2024-04-23T14:59:53.654228+0000 (oldest deadline 2024-04-23T15:00:20.054780+0000) 2024-04-23T15:00:36.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:35 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:35.715+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:02.256240+0000 front 2024-04-23T15:00:02.256270+0000 (oldest deadline 2024-04-23T15:00:27.556454+0000) 2024-04-23T15:00:36.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:35 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:35.715+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:07.557037+0000 front 2024-04-23T15:00:07.557132+0000 (oldest deadline 2024-04-23T15:00:31.656999+0000) 2024-04-23T15:00:36.367 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:36 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:36.052+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:36.367 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:36 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:36.052+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:36.367 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:36 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:36.052+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:36.367 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:36 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:36.052+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:36.368 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:36 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:36.052+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:09.809136+0000 front 2024-04-23T15:00:09.808985+0000 (oldest deadline 2024-04-23T15:00:32.709048+0000) 2024-04-23T15:00:36.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:36 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:36.366+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:23.813538+0000 front 2024-04-23T14:59:23.813943+0000 (oldest deadline 2024-04-23T14:59:49.713583+0000) 2024-04-23T15:00:36.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:36 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:36.366+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.214584+0000 front 2024-04-23T14:59:33.214522+0000 (oldest deadline 2024-04-23T15:00:02.015036+0000) 2024-04-23T15:00:36.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:36 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:36.366+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:51.416760+0000 front 2024-04-23T14:59:51.416700+0000 (oldest deadline 2024-04-23T15:00:15.516780+0000) 2024-04-23T15:00:36.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:36 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:36.366+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:01.418013+0000 front 2024-04-23T15:00:01.417867+0000 (oldest deadline 2024-04-23T15:00:23.717817+0000) 2024-04-23T15:00:36.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:36 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8[51213]: 2024-04-23T15:00:36.366+0000 7feb23364700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:09.320257+0000 front 2024-04-23T15:00:09.320282+0000 (oldest deadline 2024-04-23T15:00:34.620366+0000) 2024-04-23T15:00:36.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:36 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:36.453+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:36.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:36 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:36.453+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:36.875 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:36 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:36.453+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:36.875 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:36 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:36.453+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.079477+0000 front 2024-04-23T14:59:53.079548+0000 (oldest deadline 2024-04-23T15:00:14.779829+0000) 2024-04-23T15:00:36.875 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:36 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:36.453+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.482358+0000 front 2024-04-23T14:59:59.482241+0000 (oldest deadline 2024-04-23T15:00:24.683768+0000) 2024-04-23T15:00:36.875 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:36 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:36.453+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:08.686898+0000 front 2024-04-23T15:00:08.687026+0000 (oldest deadline 2024-04-23T15:00:33.387132+0000) 2024-04-23T15:00:37.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:36 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:36.741+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:37.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:36 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:36.741+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:37.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:36 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:36.741+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:37.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:36 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:36.741+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.654392+0000 front 2024-04-23T14:59:53.654228+0000 (oldest deadline 2024-04-23T15:00:20.054780+0000) 2024-04-23T15:00:37.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:36 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:36.741+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:02.256240+0000 front 2024-04-23T15:00:02.256270+0000 (oldest deadline 2024-04-23T15:00:27.556454+0000) 2024-04-23T15:00:37.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:36 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:36.741+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:07.557037+0000 front 2024-04-23T15:00:07.557132+0000 (oldest deadline 2024-04-23T15:00:31.656999+0000) 2024-04-23T15:00:37.452 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:37 smithi105 podman[70229]: 2024-04-23 15:00:37.165088398 +0000 UTC m=+5.671037425 container died 19c61452163f2c0776eb1f994f5bdadf04fa2eb8f2da147ac8e443dd433e2512 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T15:00:37.453 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:37 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:37.048+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:37.453 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:37 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:37.048+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:37.453 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:37 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:37.048+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:37.453 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:37 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:37.048+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:37.453 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:37 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:37.048+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:09.809136+0000 front 2024-04-23T15:00:09.808985+0000 (oldest deadline 2024-04-23T15:00:32.709048+0000) 2024-04-23T15:00:37.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:37 smithi105 podman[70229]: 2024-04-23 15:00:37.585352376 +0000 UTC m=+6.091301411 container cleanup 19c61452163f2c0776eb1f994f5bdadf04fa2eb8f2da147ac8e443dd433e2512 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8, org.label-schema.build-date=20240416, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-23T15:00:37.786 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:37 smithi105 bash[70229]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8 2024-04-23T15:00:37.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:37 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:37.475+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:37.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:37 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:37.475+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:37.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:37 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:37.475+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:37.875 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:37 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:37.475+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.079477+0000 front 2024-04-23T14:59:53.079548+0000 (oldest deadline 2024-04-23T15:00:14.779829+0000) 2024-04-23T15:00:37.875 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:37 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:37.475+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.482358+0000 front 2024-04-23T14:59:59.482241+0000 (oldest deadline 2024-04-23T15:00:24.683768+0000) 2024-04-23T15:00:37.875 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:37 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:37.475+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:08.686898+0000 front 2024-04-23T15:00:08.687026+0000 (oldest deadline 2024-04-23T15:00:33.387132+0000) 2024-04-23T15:00:38.045 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:37 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:37.788+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:38.046 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:37 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:37.788+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:38.046 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:37 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:37.788+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:38.046 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:37 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:37.788+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.654392+0000 front 2024-04-23T14:59:53.654228+0000 (oldest deadline 2024-04-23T15:00:20.054780+0000) 2024-04-23T15:00:38.046 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:37 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:37.788+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:02.256240+0000 front 2024-04-23T15:00:02.256270+0000 (oldest deadline 2024-04-23T15:00:27.556454+0000) 2024-04-23T15:00:38.046 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:37 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:37.788+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:07.557037+0000 front 2024-04-23T15:00:07.557132+0000 (oldest deadline 2024-04-23T15:00:31.656999+0000) 2024-04-23T15:00:38.055 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:37 smithi105 podman[70246]: 2024-04-23 15:00:37.786811517 +0000 UTC m=+0.620959607 container remove 19c61452163f2c0776eb1f994f5bdadf04fa2eb8f2da147ac8e443dd433e2512 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-23T15:00:38.313 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:38 smithi105 podman[70339]: 2024-04-23 15:00:38.102347249 +0000 UTC m=+0.023710444 image pull 76e77f280c74312ca1add9b6019a4a921a516f91795d85bae00a0c8ca9b1c62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd 2024-04-23T15:00:38.313 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:38 smithi105 podman[70339]: 2024-04-23 15:00:38.204253711 +0000 UTC m=+0.125616901 container create 224b3347bbab0f3adf893c79f6e1b5868e0c92f9f67099213ea09d3b0299a8e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.build-date=20240416, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T15:00:38.313 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:38 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:38.054+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:38.313 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:38 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:38.054+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:38.314 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:38 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:38.054+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:38.314 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:38 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:38.054+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:38.314 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:38 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:38.054+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:09.809136+0000 front 2024-04-23T15:00:09.808985+0000 (oldest deadline 2024-04-23T15:00:32.709048+0000) 2024-04-23T15:00:38.725 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:38 smithi105 podman[70339]: 2024-04-23 15:00:38.432710068 +0000 UTC m=+0.354073265 container init 224b3347bbab0f3adf893c79f6e1b5868e0c92f9f67099213ea09d3b0299a8e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8-deactivate, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-23T15:00:38.725 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:38 smithi105 podman[70339]: 2024-04-23 15:00:38.435994717 +0000 UTC m=+0.357357910 container start 224b3347bbab0f3adf893c79f6e1b5868e0c92f9f67099213ea09d3b0299a8e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.vendor=CentOS) 2024-04-23T15:00:38.725 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:38 smithi105 podman[70339]: 2024-04-23 15:00:38.515662741 +0000 UTC m=+0.437025947 container attach 224b3347bbab0f3adf893c79f6e1b5868e0c92f9f67099213ea09d3b0299a8e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240416, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd) 2024-04-23T15:00:38.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:38 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:38.463+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:38.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:38 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:38.463+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:38.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:38 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:38.463+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:38.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:38 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:38.463+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.079477+0000 front 2024-04-23T14:59:53.079548+0000 (oldest deadline 2024-04-23T15:00:14.779829+0000) 2024-04-23T15:00:38.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:38 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:38.463+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.482358+0000 front 2024-04-23T14:59:59.482241+0000 (oldest deadline 2024-04-23T15:00:24.683768+0000) 2024-04-23T15:00:38.875 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:38 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:38.463+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:08.686898+0000 front 2024-04-23T15:00:08.687026+0000 (oldest deadline 2024-04-23T15:00:33.387132+0000) 2024-04-23T15:00:39.018 INFO:journalctl@ceph.osd.8.smithi105.stdout:Apr 23 15:00:38 smithi105 podman[70339]: 2024-04-23 15:00:38.725348523 +0000 UTC m=+0.646711719 container died 224b3347bbab0f3adf893c79f6e1b5868e0c92f9f67099213ea09d3b0299a8e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-8-deactivate, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T15:00:39.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:38 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:38.794+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:39.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:38 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:38.794+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:39.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:38 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:38.794+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:39.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:38 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:38.794+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.654392+0000 front 2024-04-23T14:59:53.654228+0000 (oldest deadline 2024-04-23T15:00:20.054780+0000) 2024-04-23T15:00:39.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:38 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:38.794+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:02.256240+0000 front 2024-04-23T15:00:02.256270+0000 (oldest deadline 2024-04-23T15:00:27.556454+0000) 2024-04-23T15:00:39.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:38 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:38.794+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:07.557037+0000 front 2024-04-23T15:00:07.557132+0000 (oldest deadline 2024-04-23T15:00:31.656999+0000) 2024-04-23T15:00:39.285 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:39 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:39.058+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:39.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:39 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:39.058+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:39.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:39 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:39.058+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:39.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:39 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:39.058+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:39.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:39 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:39.058+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:09.809136+0000 front 2024-04-23T15:00:09.808985+0000 (oldest deadline 2024-04-23T15:00:32.709048+0000) 2024-04-23T15:00:39.304 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.8.service' 2024-04-23T15:00:39.382 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T15:00:39.382 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-23T15:00:39.382 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-23T15:00:39.383 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.9 2024-04-23T15:00:39.873 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:39 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:39.430+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:39.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:39 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:39.430+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:39.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:39 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:39.430+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:39.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:39 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:39.430+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.079477+0000 front 2024-04-23T14:59:53.079548+0000 (oldest deadline 2024-04-23T15:00:14.779829+0000) 2024-04-23T15:00:39.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:39 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:39.430+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.482358+0000 front 2024-04-23T14:59:59.482241+0000 (oldest deadline 2024-04-23T15:00:24.683768+0000) 2024-04-23T15:00:39.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:39 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:39.430+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:08.686898+0000 front 2024-04-23T15:00:08.687026+0000 (oldest deadline 2024-04-23T15:00:33.387132+0000) 2024-04-23T15:00:39.875 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:39 smithi028 systemd[1]: Stopping Ceph osd.9 for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T15:00:40.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:39 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:39.752+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:40.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:39 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:39.752+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:40.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:39 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:39.752+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:40.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:39 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:39.752+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.654392+0000 front 2024-04-23T14:59:53.654228+0000 (oldest deadline 2024-04-23T15:00:20.054780+0000) 2024-04-23T15:00:40.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:39 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:39.752+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:02.256240+0000 front 2024-04-23T15:00:02.256270+0000 (oldest deadline 2024-04-23T15:00:27.556454+0000) 2024-04-23T15:00:40.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:39 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:39.752+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:07.557037+0000 front 2024-04-23T15:00:07.557132+0000 (oldest deadline 2024-04-23T15:00:31.656999+0000) 2024-04-23T15:00:40.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:40 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:40.022+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:40.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:40 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:40.022+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:40.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:40 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:40.022+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:40.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:40 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:40.022+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:40.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:40 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:40.022+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:09.809136+0000 front 2024-04-23T15:00:09.808985+0000 (oldest deadline 2024-04-23T15:00:32.709048+0000) 2024-04-23T15:00:40.374 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:39 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:39.965+0000 7f4211a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T15:00:40.374 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:39 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:39.965+0000 7f4211a00700 -1 osd.9 108 *** Got signal Terminated *** 2024-04-23T15:00:40.374 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:39 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:39.965+0000 7f4211a00700 -1 osd.9 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T15:00:40.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:40 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:40.394+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:40.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:40 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:40.394+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:40.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:40 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:40.394+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:40.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:40 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:40.394+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.079477+0000 front 2024-04-23T14:59:53.079548+0000 (oldest deadline 2024-04-23T15:00:14.779829+0000) 2024-04-23T15:00:40.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:40 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:40.394+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.482358+0000 front 2024-04-23T14:59:59.482241+0000 (oldest deadline 2024-04-23T15:00:24.683768+0000) 2024-04-23T15:00:40.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:40 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:40.394+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:08.686898+0000 front 2024-04-23T15:00:08.687026+0000 (oldest deadline 2024-04-23T15:00:33.387132+0000) 2024-04-23T15:00:41.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:40 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:40.793+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:41.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:40 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:40.793+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:41.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:40 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:40.793+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:41.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:40 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:40.793+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.654392+0000 front 2024-04-23T14:59:53.654228+0000 (oldest deadline 2024-04-23T15:00:20.054780+0000) 2024-04-23T15:00:41.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:40 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:40.793+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:02.256240+0000 front 2024-04-23T15:00:02.256270+0000 (oldest deadline 2024-04-23T15:00:27.556454+0000) 2024-04-23T15:00:41.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:40 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:40.793+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:07.557037+0000 front 2024-04-23T15:00:07.557132+0000 (oldest deadline 2024-04-23T15:00:31.656999+0000) 2024-04-23T15:00:41.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:41 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:41.020+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:41.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:41 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:41.020+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:41.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:41 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:41.020+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:41.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:41 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:41.020+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:41.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:41 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:41.021+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:09.809136+0000 front 2024-04-23T15:00:09.808985+0000 (oldest deadline 2024-04-23T15:00:32.709048+0000) 2024-04-23T15:00:41.287 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:41 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:41.021+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:17.910940+0000 front 2024-04-23T15:00:17.910821+0000 (oldest deadline 2024-04-23T15:00:40.810518+0000) 2024-04-23T15:00:41.624 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:41 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:41.363+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:41.624 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:41 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:41.363+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:41.625 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:41 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:41.363+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:41.625 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:41 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:41.363+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.079477+0000 front 2024-04-23T14:59:53.079548+0000 (oldest deadline 2024-04-23T15:00:14.779829+0000) 2024-04-23T15:00:41.625 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:41 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:41.363+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.482358+0000 front 2024-04-23T14:59:59.482241+0000 (oldest deadline 2024-04-23T15:00:24.683768+0000) 2024-04-23T15:00:41.625 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:41 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:41.363+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:08.686898+0000 front 2024-04-23T15:00:08.687026+0000 (oldest deadline 2024-04-23T15:00:33.387132+0000) 2024-04-23T15:00:42.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:41 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:41.792+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:42.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:41 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:41.792+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:42.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:41 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:41.792+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:42.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:41 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:41.792+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.654392+0000 front 2024-04-23T14:59:53.654228+0000 (oldest deadline 2024-04-23T15:00:20.054780+0000) 2024-04-23T15:00:42.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:41 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:41.792+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:02.256240+0000 front 2024-04-23T15:00:02.256270+0000 (oldest deadline 2024-04-23T15:00:27.556454+0000) 2024-04-23T15:00:42.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:41 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:41.792+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:07.557037+0000 front 2024-04-23T15:00:07.557132+0000 (oldest deadline 2024-04-23T15:00:31.656999+0000) 2024-04-23T15:00:42.536 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:42 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:42.036+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:42.536 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:42 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:42.036+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:42.536 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:42 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:42.036+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:42.536 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:42 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:42.036+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:42.536 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:42 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:42.036+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:09.809136+0000 front 2024-04-23T15:00:09.808985+0000 (oldest deadline 2024-04-23T15:00:32.709048+0000) 2024-04-23T15:00:42.536 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:42 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:42.036+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:17.910940+0000 front 2024-04-23T15:00:17.910821+0000 (oldest deadline 2024-04-23T15:00:40.810518+0000) 2024-04-23T15:00:42.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:42 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:42.393+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:42.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:42 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:42.393+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:42.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:42 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:42.393+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:42.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:42 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:42.393+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.079477+0000 front 2024-04-23T14:59:53.079548+0000 (oldest deadline 2024-04-23T15:00:14.779829+0000) 2024-04-23T15:00:42.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:42 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:42.393+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.482358+0000 front 2024-04-23T14:59:59.482241+0000 (oldest deadline 2024-04-23T15:00:24.683768+0000) 2024-04-23T15:00:42.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:42 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:42.393+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:08.686898+0000 front 2024-04-23T15:00:08.687026+0000 (oldest deadline 2024-04-23T15:00:33.387132+0000) 2024-04-23T15:00:43.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:42 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:42.814+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:43.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:42 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:42.814+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:43.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:42 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:42.814+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:43.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:42 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:42.814+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.654392+0000 front 2024-04-23T14:59:53.654228+0000 (oldest deadline 2024-04-23T15:00:20.054780+0000) 2024-04-23T15:00:43.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:42 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:42.814+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:02.256240+0000 front 2024-04-23T15:00:02.256270+0000 (oldest deadline 2024-04-23T15:00:27.556454+0000) 2024-04-23T15:00:43.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:42 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:42.814+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:07.557037+0000 front 2024-04-23T15:00:07.557132+0000 (oldest deadline 2024-04-23T15:00:31.656999+0000) 2024-04-23T15:00:43.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:42 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:42.814+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:17.959113+0000 front 2024-04-23T15:00:17.959181+0000 (oldest deadline 2024-04-23T15:00:42.058823+0000) 2024-04-23T15:00:43.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:43 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:43.034+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:43.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:43 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:43.034+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:43.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:43 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:43.034+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:43.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:43 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:43.034+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:43.287 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:43 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:43.034+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:09.809136+0000 front 2024-04-23T15:00:09.808985+0000 (oldest deadline 2024-04-23T15:00:32.709048+0000) 2024-04-23T15:00:43.287 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:43 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:43.034+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:17.910940+0000 front 2024-04-23T15:00:17.910821+0000 (oldest deadline 2024-04-23T15:00:40.810518+0000) 2024-04-23T15:00:43.624 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:43 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:43.364+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:43.624 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:43 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:43.364+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:43.625 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:43 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:43.364+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:43.625 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:43 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:43.364+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.079477+0000 front 2024-04-23T14:59:53.079548+0000 (oldest deadline 2024-04-23T15:00:14.779829+0000) 2024-04-23T15:00:43.625 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:43 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:43.364+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.482358+0000 front 2024-04-23T14:59:59.482241+0000 (oldest deadline 2024-04-23T15:00:24.683768+0000) 2024-04-23T15:00:43.625 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:43 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:43.364+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:08.686898+0000 front 2024-04-23T15:00:08.687026+0000 (oldest deadline 2024-04-23T15:00:33.387132+0000) 2024-04-23T15:00:43.625 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:43 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:43.364+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:16.789892+0000 front 2024-04-23T15:00:16.790144+0000 (oldest deadline 2024-04-23T15:00:42.689920+0000) 2024-04-23T15:00:44.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:43 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:43.862+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:44.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:43 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:43.862+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:44.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:43 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:43.862+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:44.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:43 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:43.862+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.654392+0000 front 2024-04-23T14:59:53.654228+0000 (oldest deadline 2024-04-23T15:00:20.054780+0000) 2024-04-23T15:00:44.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:43 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:43.862+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:02.256240+0000 front 2024-04-23T15:00:02.256270+0000 (oldest deadline 2024-04-23T15:00:27.556454+0000) 2024-04-23T15:00:44.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:43 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:43.862+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:07.557037+0000 front 2024-04-23T15:00:07.557132+0000 (oldest deadline 2024-04-23T15:00:31.656999+0000) 2024-04-23T15:00:44.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:43 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:43.862+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:17.959113+0000 front 2024-04-23T15:00:17.959181+0000 (oldest deadline 2024-04-23T15:00:42.058823+0000) 2024-04-23T15:00:44.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:44 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:44.004+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:44.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:44 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:44.004+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:44.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:44 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:44.004+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:44.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:44 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:44.004+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:44.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:44 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:44.004+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:09.809136+0000 front 2024-04-23T15:00:09.808985+0000 (oldest deadline 2024-04-23T15:00:32.709048+0000) 2024-04-23T15:00:44.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:44 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:44.004+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:17.910940+0000 front 2024-04-23T15:00:17.910821+0000 (oldest deadline 2024-04-23T15:00:40.810518+0000) 2024-04-23T15:00:44.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:44 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:44.377+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:28.071413+0000 front 2024-04-23T14:59:28.072113+0000 (oldest deadline 2024-04-23T14:59:50.370728+0000) 2024-04-23T15:00:44.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:44 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:44.377+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:34.472420+0000 front 2024-04-23T14:59:34.472477+0000 (oldest deadline 2024-04-23T14:59:59.172668+0000) 2024-04-23T15:00:44.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:44 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:44.377+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:41.375719+0000 front 2024-04-23T14:59:41.375901+0000 (oldest deadline 2024-04-23T15:00:08.977724+0000) 2024-04-23T15:00:44.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:44 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:44.377+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.079477+0000 front 2024-04-23T14:59:53.079548+0000 (oldest deadline 2024-04-23T15:00:14.779829+0000) 2024-04-23T15:00:44.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:44 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:44.377+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.482358+0000 front 2024-04-23T14:59:59.482241+0000 (oldest deadline 2024-04-23T15:00:24.683768+0000) 2024-04-23T15:00:44.875 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:44 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:44.377+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:08.686898+0000 front 2024-04-23T15:00:08.687026+0000 (oldest deadline 2024-04-23T15:00:33.387132+0000) 2024-04-23T15:00:44.875 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:44 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9[60710]: 2024-04-23T15:00:44.377+0000 7f4208200700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:16.789892+0000 front 2024-04-23T15:00:16.790144+0000 (oldest deadline 2024-04-23T15:00:42.689920+0000) 2024-04-23T15:00:45.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:44 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:44.863+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:45.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:44 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:44.863+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:45.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:44 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:44.863+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:45.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:44 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:44.863+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.654392+0000 front 2024-04-23T14:59:53.654228+0000 (oldest deadline 2024-04-23T15:00:20.054780+0000) 2024-04-23T15:00:45.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:44 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:44.863+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:02.256240+0000 front 2024-04-23T15:00:02.256270+0000 (oldest deadline 2024-04-23T15:00:27.556454+0000) 2024-04-23T15:00:45.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:44 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:44.863+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:07.557037+0000 front 2024-04-23T15:00:07.557132+0000 (oldest deadline 2024-04-23T15:00:31.656999+0000) 2024-04-23T15:00:45.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:44 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:44.863+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:17.959113+0000 front 2024-04-23T15:00:17.959181+0000 (oldest deadline 2024-04-23T15:00:42.058823+0000) 2024-04-23T15:00:45.285 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:44 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:44.960+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:45.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:44 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:44.960+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:45.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:44 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:44.960+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:45.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:44 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:44.960+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:45.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:44 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:44.960+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:09.809136+0000 front 2024-04-23T15:00:09.808985+0000 (oldest deadline 2024-04-23T15:00:32.709048+0000) 2024-04-23T15:00:45.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:44 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:44.960+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:17.910940+0000 front 2024-04-23T15:00:17.910821+0000 (oldest deadline 2024-04-23T15:00:40.810518+0000) 2024-04-23T15:00:45.874 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:45 smithi028 podman[94361]: 2024-04-23 15:00:45.522738964 +0000 UTC m=+5.672074845 container died 60a654a97fd86f1c2b29968ec8ab20284c2bf4973f3185802f2ebcbede4f17d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T15:00:46.261 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:46 smithi028 podman[94361]: 2024-04-23 15:00:46.084876656 +0000 UTC m=+6.234212577 container cleanup 60a654a97fd86f1c2b29968ec8ab20284c2bf4973f3185802f2ebcbede4f17d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-23T15:00:46.261 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:46 smithi028 bash[94361]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9 2024-04-23T15:00:46.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:46 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:46.002+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:46.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:46 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:46.002+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:46.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:46 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:46.002+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:46.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:46 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:46.002+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:46.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:46 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:46.002+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:09.809136+0000 front 2024-04-23T15:00:09.808985+0000 (oldest deadline 2024-04-23T15:00:32.709048+0000) 2024-04-23T15:00:46.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:46 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:46.002+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:17.910940+0000 front 2024-04-23T15:00:17.910821+0000 (oldest deadline 2024-04-23T15:00:40.810518+0000) 2024-04-23T15:00:46.372 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:45 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:45.889+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:46.372 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:45 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:45.889+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:46.372 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:45 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:45.889+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:46.373 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:45 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:45.889+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.654392+0000 front 2024-04-23T14:59:53.654228+0000 (oldest deadline 2024-04-23T15:00:20.054780+0000) 2024-04-23T15:00:46.373 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:45 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:45.889+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:02.256240+0000 front 2024-04-23T15:00:02.256270+0000 (oldest deadline 2024-04-23T15:00:27.556454+0000) 2024-04-23T15:00:46.373 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:45 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:45.889+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:07.557037+0000 front 2024-04-23T15:00:07.557132+0000 (oldest deadline 2024-04-23T15:00:31.656999+0000) 2024-04-23T15:00:46.373 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:45 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:45.889+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:17.959113+0000 front 2024-04-23T15:00:17.959181+0000 (oldest deadline 2024-04-23T15:00:42.058823+0000) 2024-04-23T15:00:46.623 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:46 smithi028 podman[94380]: 2024-04-23 15:00:46.260575605 +0000 UTC m=+0.718829670 container remove 60a654a97fd86f1c2b29968ec8ab20284c2bf4973f3185802f2ebcbede4f17d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-23T15:00:47.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:46 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:46.850+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:47.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:46 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:46.850+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:47.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:46 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:46.850+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:47.122 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:46 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:46.850+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.654392+0000 front 2024-04-23T14:59:53.654228+0000 (oldest deadline 2024-04-23T15:00:20.054780+0000) 2024-04-23T15:00:47.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:46 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:46.850+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:02.256240+0000 front 2024-04-23T15:00:02.256270+0000 (oldest deadline 2024-04-23T15:00:27.556454+0000) 2024-04-23T15:00:47.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:46 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:46.850+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:07.557037+0000 front 2024-04-23T15:00:07.557132+0000 (oldest deadline 2024-04-23T15:00:31.656999+0000) 2024-04-23T15:00:47.123 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:46 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:46.850+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:17.959113+0000 front 2024-04-23T15:00:17.959181+0000 (oldest deadline 2024-04-23T15:00:42.058823+0000) 2024-04-23T15:00:47.285 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:47 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:47.003+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:47.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:47 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:47.003+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:47.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:47 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:47.003+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:47.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:47 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:47.003+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:47.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:47 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:47.003+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:09.809136+0000 front 2024-04-23T15:00:09.808985+0000 (oldest deadline 2024-04-23T15:00:32.709048+0000) 2024-04-23T15:00:47.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:47 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:47.003+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:17.910940+0000 front 2024-04-23T15:00:17.910821+0000 (oldest deadline 2024-04-23T15:00:40.810518+0000) 2024-04-23T15:00:47.373 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:47 smithi028 podman[94471]: 2024-04-23 15:00:47.029289866 +0000 UTC m=+0.039859419 image pull 76e77f280c74312ca1add9b6019a4a921a516f91795d85bae00a0c8ca9b1c62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd 2024-04-23T15:00:47.373 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:47 smithi028 podman[94471]: 2024-04-23 15:00:47.145001277 +0000 UTC m=+0.155570870 container create f4e9bad5c8e622080d2f772844983c93c64c307905292fdb43b4a6ddf8ebaf91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T15:00:47.658 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:47 smithi028 podman[94471]: 2024-04-23 15:00:47.406402713 +0000 UTC m=+0.416972291 container init f4e9bad5c8e622080d2f772844983c93c64c307905292fdb43b4a6ddf8ebaf91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-23T15:00:47.658 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:47 smithi028 podman[94471]: 2024-04-23 15:00:47.418643486 +0000 UTC m=+0.429213055 container start f4e9bad5c8e622080d2f772844983c93c64c307905292fdb43b4a6ddf8ebaf91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9-deactivate, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-23T15:00:47.659 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:47 smithi028 podman[94471]: 2024-04-23 15:00:47.522145413 +0000 UTC m=+0.532714997 container attach f4e9bad5c8e622080d2f772844983c93c64c307905292fdb43b4a6ddf8ebaf91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416) 2024-04-23T15:00:47.973 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 23 15:00:47 smithi028 podman[94471]: 2024-04-23 15:00:47.894745118 +0000 UTC m=+0.905314689 container died f4e9bad5c8e622080d2f772844983c93c64c307905292fdb43b4a6ddf8ebaf91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-23T15:00:48.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:47 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:47.977+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:48.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:47 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:47.977+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:48.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:47 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:47.977+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:48.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:47 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:47.977+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:48.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:47 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:47.977+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:09.809136+0000 front 2024-04-23T15:00:09.808985+0000 (oldest deadline 2024-04-23T15:00:32.709048+0000) 2024-04-23T15:00:48.287 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:47 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:47.977+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:17.910940+0000 front 2024-04-23T15:00:17.910821+0000 (oldest deadline 2024-04-23T15:00:40.810518+0000) 2024-04-23T15:00:48.372 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:47 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:47.889+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:48.372 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:47 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:47.889+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:48.373 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:47 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:47.889+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:48.373 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:47 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:47.889+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.654392+0000 front 2024-04-23T14:59:53.654228+0000 (oldest deadline 2024-04-23T15:00:20.054780+0000) 2024-04-23T15:00:48.373 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:47 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:47.889+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:02.256240+0000 front 2024-04-23T15:00:02.256270+0000 (oldest deadline 2024-04-23T15:00:27.556454+0000) 2024-04-23T15:00:48.373 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:47 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:47.889+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:07.557037+0000 front 2024-04-23T15:00:07.557132+0000 (oldest deadline 2024-04-23T15:00:31.656999+0000) 2024-04-23T15:00:48.373 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:47 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:47.889+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:17.959113+0000 front 2024-04-23T15:00:17.959181+0000 (oldest deadline 2024-04-23T15:00:42.058823+0000) 2024-04-23T15:00:48.558 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.9.service' 2024-04-23T15:00:48.647 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T15:00:48.648 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-23T15:00:48.648 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-23T15:00:48.648 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.10 2024-04-23T15:00:48.967 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:48 smithi038 systemd[1]: Stopping Ceph osd.10 for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T15:00:48.968 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:48 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:48.922+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:48.968 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:48 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:48.922+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:48.968 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:48 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:48.922+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:48.968 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:48 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:48.922+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.654392+0000 front 2024-04-23T14:59:53.654228+0000 (oldest deadline 2024-04-23T15:00:20.054780+0000) 2024-04-23T15:00:48.968 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:48 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:48.922+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:02.256240+0000 front 2024-04-23T15:00:02.256270+0000 (oldest deadline 2024-04-23T15:00:27.556454+0000) 2024-04-23T15:00:48.968 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:48 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:48.922+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:07.557037+0000 front 2024-04-23T15:00:07.557132+0000 (oldest deadline 2024-04-23T15:00:31.656999+0000) 2024-04-23T15:00:48.968 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:48 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:48.922+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:17.959113+0000 front 2024-04-23T15:00:17.959181+0000 (oldest deadline 2024-04-23T15:00:42.058823+0000) 2024-04-23T15:00:48.968 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:48 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:48.965+0000 7f7090ad7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T15:00:48.969 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:48 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:48.965+0000 7f7090ad7700 -1 osd.10 108 *** Got signal Terminated *** 2024-04-23T15:00:48.969 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:48 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:48.965+0000 7f7090ad7700 -1 osd.10 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T15:00:49.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:49 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:49.001+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:49.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:49 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:49.002+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:49.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:49 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:49.002+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:49.287 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:49 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:49.002+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:49.287 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:49 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:49.002+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:09.809136+0000 front 2024-04-23T15:00:09.808985+0000 (oldest deadline 2024-04-23T15:00:32.709048+0000) 2024-04-23T15:00:49.287 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:49 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:49.002+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:17.910940+0000 front 2024-04-23T15:00:17.910821+0000 (oldest deadline 2024-04-23T15:00:40.810518+0000) 2024-04-23T15:00:50.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:49 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:49.990+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:50.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:49 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:49.990+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:50.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:49 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:49.990+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:50.287 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:49 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:49.990+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:50.287 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:49 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:49.990+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:09.809136+0000 front 2024-04-23T15:00:09.808985+0000 (oldest deadline 2024-04-23T15:00:32.709048+0000) 2024-04-23T15:00:50.287 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:49 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:49.990+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:17.910940+0000 front 2024-04-23T15:00:17.910821+0000 (oldest deadline 2024-04-23T15:00:40.810518+0000) 2024-04-23T15:00:50.372 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:49 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:49.911+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:50.372 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:49 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:49.911+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:50.372 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:49 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:49.911+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:50.372 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:49 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:49.911+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.654392+0000 front 2024-04-23T14:59:53.654228+0000 (oldest deadline 2024-04-23T15:00:20.054780+0000) 2024-04-23T15:00:50.373 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:49 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:49.911+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:02.256240+0000 front 2024-04-23T15:00:02.256270+0000 (oldest deadline 2024-04-23T15:00:27.556454+0000) 2024-04-23T15:00:50.373 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:49 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:49.911+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:07.557037+0000 front 2024-04-23T15:00:07.557132+0000 (oldest deadline 2024-04-23T15:00:31.656999+0000) 2024-04-23T15:00:50.373 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:49 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:49.911+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:17.959113+0000 front 2024-04-23T15:00:17.959181+0000 (oldest deadline 2024-04-23T15:00:42.058823+0000) 2024-04-23T15:00:51.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:50 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:50.959+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:51.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:50 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:50.959+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:51.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:50 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:50.959+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:51.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:50 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:50.959+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:51.287 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:50 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:50.959+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:09.809136+0000 front 2024-04-23T15:00:09.808985+0000 (oldest deadline 2024-04-23T15:00:32.709048+0000) 2024-04-23T15:00:51.287 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:50 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:50.959+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:17.910940+0000 front 2024-04-23T15:00:17.910821+0000 (oldest deadline 2024-04-23T15:00:40.810518+0000) 2024-04-23T15:00:51.372 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:50 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:50.878+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:51.372 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:50 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:50.878+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:51.372 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:50 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:50.878+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:51.372 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:50 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:50.878+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.654392+0000 front 2024-04-23T14:59:53.654228+0000 (oldest deadline 2024-04-23T15:00:20.054780+0000) 2024-04-23T15:00:51.372 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:50 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:50.878+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:02.256240+0000 front 2024-04-23T15:00:02.256270+0000 (oldest deadline 2024-04-23T15:00:27.556454+0000) 2024-04-23T15:00:51.373 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:50 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:50.878+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:07.557037+0000 front 2024-04-23T15:00:07.557132+0000 (oldest deadline 2024-04-23T15:00:31.656999+0000) 2024-04-23T15:00:51.373 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:50 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:50.878+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:17.959113+0000 front 2024-04-23T15:00:17.959181+0000 (oldest deadline 2024-04-23T15:00:42.058823+0000) 2024-04-23T15:00:52.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:51 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:51.946+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:52.287 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:51 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:51.947+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:52.287 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:51 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:51.947+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:52.287 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:51 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:51.947+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:52.287 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:51 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:51.947+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:09.809136+0000 front 2024-04-23T15:00:09.808985+0000 (oldest deadline 2024-04-23T15:00:32.709048+0000) 2024-04-23T15:00:52.287 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:51 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:51.947+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:17.910940+0000 front 2024-04-23T15:00:17.910821+0000 (oldest deadline 2024-04-23T15:00:40.810518+0000) 2024-04-23T15:00:52.287 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:51 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:51.947+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-23T15:00:28.412161+0000 front 2024-04-23T15:00:28.412048+0000 (oldest deadline 2024-04-23T15:00:51.311801+0000) 2024-04-23T15:00:52.372 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:51 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:51.925+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:52.372 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:51 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:51.925+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:52.372 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:51 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:51.925+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:52.373 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:51 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:51.925+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.654392+0000 front 2024-04-23T14:59:53.654228+0000 (oldest deadline 2024-04-23T15:00:20.054780+0000) 2024-04-23T15:00:52.373 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:51 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:51.925+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:02.256240+0000 front 2024-04-23T15:00:02.256270+0000 (oldest deadline 2024-04-23T15:00:27.556454+0000) 2024-04-23T15:00:52.373 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:51 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:51.925+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:07.557037+0000 front 2024-04-23T15:00:07.557132+0000 (oldest deadline 2024-04-23T15:00:31.656999+0000) 2024-04-23T15:00:52.373 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:51 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:51.925+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:17.959113+0000 front 2024-04-23T15:00:17.959181+0000 (oldest deadline 2024-04-23T15:00:42.058823+0000) 2024-04-23T15:00:53.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:52 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:52.964+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:53.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:52 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:52.964+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:53.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:52 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:52.964+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:53.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:52 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:52.964+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:53.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:52 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:52.964+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:09.809136+0000 front 2024-04-23T15:00:09.808985+0000 (oldest deadline 2024-04-23T15:00:32.709048+0000) 2024-04-23T15:00:53.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:52 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:52.964+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:17.910940+0000 front 2024-04-23T15:00:17.910821+0000 (oldest deadline 2024-04-23T15:00:40.810518+0000) 2024-04-23T15:00:53.287 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:52 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:52.964+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-23T15:00:28.412161+0000 front 2024-04-23T15:00:28.412048+0000 (oldest deadline 2024-04-23T15:00:51.311801+0000) 2024-04-23T15:00:53.372 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:52 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:52.880+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:53.372 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:52 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:52.880+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:53.372 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:52 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:52.880+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:53.372 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:52 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:52.880+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.654392+0000 front 2024-04-23T14:59:53.654228+0000 (oldest deadline 2024-04-23T15:00:20.054780+0000) 2024-04-23T15:00:53.372 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:52 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:52.880+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:02.256240+0000 front 2024-04-23T15:00:02.256270+0000 (oldest deadline 2024-04-23T15:00:27.556454+0000) 2024-04-23T15:00:53.373 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:52 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:52.880+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:07.557037+0000 front 2024-04-23T15:00:07.557132+0000 (oldest deadline 2024-04-23T15:00:31.656999+0000) 2024-04-23T15:00:53.373 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:52 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:52.880+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:17.959113+0000 front 2024-04-23T15:00:17.959181+0000 (oldest deadline 2024-04-23T15:00:42.058823+0000) 2024-04-23T15:00:54.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:53 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:53.987+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:54.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:53 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:53.987+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:54.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:53 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:53.987+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:54.287 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:53 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:53.987+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:54.287 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:53 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:53.987+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:09.809136+0000 front 2024-04-23T15:00:09.808985+0000 (oldest deadline 2024-04-23T15:00:32.709048+0000) 2024-04-23T15:00:54.287 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:53 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:53.987+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:17.910940+0000 front 2024-04-23T15:00:17.910821+0000 (oldest deadline 2024-04-23T15:00:40.810518+0000) 2024-04-23T15:00:54.287 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:53 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:53.987+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-23T15:00:28.412161+0000 front 2024-04-23T15:00:28.412048+0000 (oldest deadline 2024-04-23T15:00:51.311801+0000) 2024-04-23T15:00:54.354 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:53 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:53.905+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:25.747853+0000 front 2024-04-23T14:59:25.748019+0000 (oldest deadline 2024-04-23T14:59:53.348530+0000) 2024-04-23T15:00:54.355 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:53 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:53.905+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:36.849829+0000 front 2024-04-23T14:59:36.849855+0000 (oldest deadline 2024-04-23T14:59:59.749991+0000) 2024-04-23T15:00:54.355 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:53 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:53.905+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:43.851513+0000 front 2024-04-23T14:59:43.851448+0000 (oldest deadline 2024-04-23T15:00:07.951579+0000) 2024-04-23T15:00:54.355 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:53 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:53.905+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-23T14:59:53.654392+0000 front 2024-04-23T14:59:53.654228+0000 (oldest deadline 2024-04-23T15:00:20.054780+0000) 2024-04-23T15:00:54.355 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:53 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:53.905+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T15:00:02.256240+0000 front 2024-04-23T15:00:02.256270+0000 (oldest deadline 2024-04-23T15:00:27.556454+0000) 2024-04-23T15:00:54.355 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:53 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:53.905+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:07.557037+0000 front 2024-04-23T15:00:07.557132+0000 (oldest deadline 2024-04-23T15:00:31.656999+0000) 2024-04-23T15:00:54.355 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:53 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-23T15:00:53.905+0000 7f708d0cb700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:17.959113+0000 front 2024-04-23T15:00:17.959181+0000 (oldest deadline 2024-04-23T15:00:42.058823+0000) 2024-04-23T15:00:54.622 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:54 smithi038 podman[69822]: 2024-04-23 15:00:54.355075867 +0000 UTC m=+5.501161076 container died 1f16cac69cb9d8fff53e8ee4897cb9af101bdeac3358aadc76352f74dfabcdba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T15:00:54.968 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:54 smithi038 podman[69822]: 2024-04-23 15:00:54.766733558 +0000 UTC m=+5.912818747 container cleanup 1f16cac69cb9d8fff53e8ee4897cb9af101bdeac3358aadc76352f74dfabcdba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-23T15:00:54.968 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:54 smithi038 bash[69822]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10 2024-04-23T15:00:55.276 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:54 smithi038 podman[69841]: 2024-04-23 15:00:54.967807858 +0000 UTC m=+0.607233588 container remove 1f16cac69cb9d8fff53e8ee4897cb9af101bdeac3358aadc76352f74dfabcdba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-23T15:00:55.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:54 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:54.937+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:55.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:54 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:54.937+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:55.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:54 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:54.937+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:55.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:54 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:54.937+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:55.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:54 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:54.937+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:09.809136+0000 front 2024-04-23T15:00:09.808985+0000 (oldest deadline 2024-04-23T15:00:32.709048+0000) 2024-04-23T15:00:55.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:54 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:54.937+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:17.910940+0000 front 2024-04-23T15:00:17.910821+0000 (oldest deadline 2024-04-23T15:00:40.810518+0000) 2024-04-23T15:00:55.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:54 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:54.937+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-23T15:00:28.412161+0000 front 2024-04-23T15:00:28.412048+0000 (oldest deadline 2024-04-23T15:00:51.311801+0000) 2024-04-23T15:00:55.562 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:55 smithi038 podman[69934]: 2024-04-23 15:00:55.276724963 +0000 UTC m=+0.093707090 container create b18d2b1c8676790d30ad9a04108f8f5c6695246ebc06c795ea5af53e37717551 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-23T15:00:55.562 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:55 smithi038 podman[69934]: 2024-04-23 15:00:55.201843212 +0000 UTC m=+0.018825348 image pull 76e77f280c74312ca1add9b6019a4a921a516f91795d85bae00a0c8ca9b1c62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd 2024-04-23T15:00:55.562 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:55 smithi038 podman[69934]: 2024-04-23 15:00:55.487578021 +0000 UTC m=+0.304560149 container init b18d2b1c8676790d30ad9a04108f8f5c6695246ebc06c795ea5af53e37717551 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-23T15:00:55.563 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:55 smithi038 podman[69934]: 2024-04-23 15:00:55.490816456 +0000 UTC m=+0.307798592 container start b18d2b1c8676790d30ad9a04108f8f5c6695246ebc06c795ea5af53e37717551 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd) 2024-04-23T15:00:55.563 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:55 smithi038 podman[69934]: 2024-04-23 15:00:55.562560757 +0000 UTC m=+0.379542885 container attach b18d2b1c8676790d30ad9a04108f8f5c6695246ebc06c795ea5af53e37717551 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-23T15:00:55.872 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 23 15:00:55 smithi038 podman[69934]: 2024-04-23 15:00:55.790835557 +0000 UTC m=+0.607817692 container died b18d2b1c8676790d30ad9a04108f8f5c6695246ebc06c795ea5af53e37717551 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-23T15:00:56.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:55 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:55.901+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:56.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:55 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:55.901+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:56.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:55 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:55.901+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:56.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:55 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:55.901+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:56.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:55 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:55.901+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:09.809136+0000 front 2024-04-23T15:00:09.808985+0000 (oldest deadline 2024-04-23T15:00:32.709048+0000) 2024-04-23T15:00:56.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:55 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:55.901+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:17.910940+0000 front 2024-04-23T15:00:17.910821+0000 (oldest deadline 2024-04-23T15:00:40.810518+0000) 2024-04-23T15:00:56.287 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:55 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:55.901+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-23T15:00:28.412161+0000 front 2024-04-23T15:00:28.412048+0000 (oldest deadline 2024-04-23T15:00:51.311801+0000) 2024-04-23T15:00:56.409 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.10.service' 2024-04-23T15:00:56.450 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T15:00:56.450 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-23T15:00:56.450 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-23T15:00:56.450 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.11 2024-04-23T15:00:56.747 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:56 smithi105 systemd[1]: Stopping Ceph osd.11 for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T15:00:57.036 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:56 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:56.747+0000 7f67b5d78700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T15:00:57.036 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:56 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:56.747+0000 7f67b5d78700 -1 osd.11 108 *** Got signal Terminated *** 2024-04-23T15:00:57.036 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:56 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:56.747+0000 7f67b5d78700 -1 osd.11 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T15:00:57.036 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:56 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:56.895+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:57.036 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:56 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:56.895+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:57.037 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:56 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:56.895+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:57.037 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:56 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:56.895+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:57.037 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:56 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:56.895+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:09.809136+0000 front 2024-04-23T15:00:09.808985+0000 (oldest deadline 2024-04-23T15:00:32.709048+0000) 2024-04-23T15:00:57.037 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:56 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:56.895+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:17.910940+0000 front 2024-04-23T15:00:17.910821+0000 (oldest deadline 2024-04-23T15:00:40.810518+0000) 2024-04-23T15:00:57.037 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:56 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:56.895+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-23T15:00:28.412161+0000 front 2024-04-23T15:00:28.412048+0000 (oldest deadline 2024-04-23T15:00:51.311801+0000) 2024-04-23T15:00:58.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:57 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:57.861+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:58.287 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:57 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:57.861+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:58.287 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:57 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:57.861+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:58.287 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:57 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:57.861+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:58.287 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:57 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:57.861+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:09.809136+0000 front 2024-04-23T15:00:09.808985+0000 (oldest deadline 2024-04-23T15:00:32.709048+0000) 2024-04-23T15:00:58.288 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:57 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:57.861+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:17.910940+0000 front 2024-04-23T15:00:17.910821+0000 (oldest deadline 2024-04-23T15:00:40.810518+0000) 2024-04-23T15:00:58.288 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:57 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:57.861+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-23T15:00:28.412161+0000 front 2024-04-23T15:00:28.412048+0000 (oldest deadline 2024-04-23T15:00:51.311801+0000) 2024-04-23T15:00:59.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:58 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:58.814+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:00:59.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:58 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:58.814+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:00:59.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:58 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:58.814+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:00:59.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:58 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:58.814+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:00:59.286 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:58 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:58.814+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:09.809136+0000 front 2024-04-23T15:00:09.808985+0000 (oldest deadline 2024-04-23T15:00:32.709048+0000) 2024-04-23T15:00:59.287 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:58 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:58.814+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:17.910940+0000 front 2024-04-23T15:00:17.910821+0000 (oldest deadline 2024-04-23T15:00:40.810518+0000) 2024-04-23T15:00:59.287 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:58 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:58.814+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-23T15:00:28.412161+0000 front 2024-04-23T15:00:28.412048+0000 (oldest deadline 2024-04-23T15:00:51.311801+0000) 2024-04-23T15:01:00.036 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:59 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:59.774+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:01:00.036 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:59 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:59.774+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:01:00.036 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:59 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:59.774+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:01:00.036 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:59 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:59.774+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:01:00.036 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:59 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:59.774+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:09.809136+0000 front 2024-04-23T15:00:09.808985+0000 (oldest deadline 2024-04-23T15:00:32.709048+0000) 2024-04-23T15:01:00.036 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:59 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:59.774+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:17.910940+0000 front 2024-04-23T15:00:17.910821+0000 (oldest deadline 2024-04-23T15:00:40.810518+0000) 2024-04-23T15:01:00.037 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:59 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:59.774+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-23T15:00:28.412161+0000 front 2024-04-23T15:00:28.412048+0000 (oldest deadline 2024-04-23T15:00:51.311801+0000) 2024-04-23T15:01:00.037 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:00:59 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:00:59.774+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6820 osd.8 since back 2024-04-23T15:00:36.012755+0000 front 2024-04-23T15:00:36.012702+0000 (oldest deadline 2024-04-23T15:00:59.512683+0000) 2024-04-23T15:01:01.036 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:01:00 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:01:00.737+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:01:01.036 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:01:00 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:01:00.737+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:01:01.036 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:01:00 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:01:00.737+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:01:01.037 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:01:00 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:01:00.737+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:01:01.037 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:01:00 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:01:00.737+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:09.809136+0000 front 2024-04-23T15:00:09.808985+0000 (oldest deadline 2024-04-23T15:00:32.709048+0000) 2024-04-23T15:01:01.037 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:01:00 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:01:00.737+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:17.910940+0000 front 2024-04-23T15:00:17.910821+0000 (oldest deadline 2024-04-23T15:00:40.810518+0000) 2024-04-23T15:01:01.037 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:01:00 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:01:00.737+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-23T15:00:28.412161+0000 front 2024-04-23T15:00:28.412048+0000 (oldest deadline 2024-04-23T15:00:51.311801+0000) 2024-04-23T15:01:01.037 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:01:00 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:01:00.737+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6820 osd.8 since back 2024-04-23T15:00:36.012755+0000 front 2024-04-23T15:00:36.012702+0000 (oldest deadline 2024-04-23T15:00:59.512683+0000) 2024-04-23T15:01:01.786 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:01:01 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:01:01.691+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-23T14:59:27.000417+0000 front 2024-04-23T14:59:27.000358+0000 (oldest deadline 2024-04-23T14:59:50.500172+0000) 2024-04-23T15:01:01.786 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:01:01 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:01:01.691+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-23T14:59:33.902103+0000 front 2024-04-23T14:59:33.902074+0000 (oldest deadline 2024-04-23T14:59:58.602275+0000) 2024-04-23T15:01:01.787 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:01:01 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:01:01.691+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6804 osd.2 since back 2024-04-23T14:59:44.403898+0000 front 2024-04-23T14:59:44.404039+0000 (oldest deadline 2024-04-23T15:00:07.304016+0000) 2024-04-23T15:01:01.787 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:01:01 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:01:01.691+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-23T14:59:59.606005+0000 front 2024-04-23T14:59:59.606103+0000 (oldest deadline 2024-04-23T15:00:23.705909+0000) 2024-04-23T15:01:01.787 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:01:01 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:01:01.691+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6812 osd.5 since back 2024-04-23T15:00:09.809136+0000 front 2024-04-23T15:00:09.808985+0000 (oldest deadline 2024-04-23T15:00:32.709048+0000) 2024-04-23T15:01:01.787 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:01:01 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:01:01.691+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-23T15:00:17.910940+0000 front 2024-04-23T15:00:17.910821+0000 (oldest deadline 2024-04-23T15:00:40.810518+0000) 2024-04-23T15:01:01.787 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:01:01 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:01:01.691+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-23T15:00:28.412161+0000 front 2024-04-23T15:00:28.412048+0000 (oldest deadline 2024-04-23T15:00:51.311801+0000) 2024-04-23T15:01:01.787 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:01:01 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11[58240]: 2024-04-23T15:01:01.691+0000 7f67b236c700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.105:6820 osd.8 since back 2024-04-23T15:00:36.012755+0000 front 2024-04-23T15:00:36.012702+0000 (oldest deadline 2024-04-23T15:00:59.512683+0000) 2024-04-23T15:01:02.523 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:01:02 smithi105 podman[70456]: 2024-04-23 15:01:02.254925137 +0000 UTC m=+5.616905407 container died 4859788ddf790c8ef080167df89cdf32f5077227533f0bfb0b2df9be09a8ea2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd) 2024-04-23T15:01:02.785 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:01:02 smithi105 podman[70456]: 2024-04-23 15:01:02.622330586 +0000 UTC m=+5.984310856 container cleanup 4859788ddf790c8ef080167df89cdf32f5077227533f0bfb0b2df9be09a8ea2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-23T15:01:02.786 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:01:02 smithi105 bash[70456]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11 2024-04-23T15:01:03.243 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:01:02 smithi105 podman[70491]: 2024-04-23 15:01:02.894576989 +0000 UTC m=+0.638095597 container remove 4859788ddf790c8ef080167df89cdf32f5077227533f0bfb0b2df9be09a8ea2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, ceph=True) 2024-04-23T15:01:03.536 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:01:03 smithi105 podman[70580]: 2024-04-23 15:01:03.146489637 +0000 UTC m=+0.017916584 image pull 76e77f280c74312ca1add9b6019a4a921a516f91795d85bae00a0c8ca9b1c62e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd 2024-04-23T15:01:03.536 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:01:03 smithi105 podman[70580]: 2024-04-23 15:01:03.265853141 +0000 UTC m=+0.137280088 container create 55bfbaa4b7b6843392684e7e7fb53f2a53a26c1d0460c153fd0918599c6def25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11-deactivate, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-23T15:01:03.536 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:01:03 smithi105 podman[70580]: 2024-04-23 15:01:03.484692095 +0000 UTC m=+0.356119042 container init 55bfbaa4b7b6843392684e7e7fb53f2a53a26c1d0460c153fd0918599c6def25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240416) 2024-04-23T15:01:03.536 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:01:03 smithi105 podman[70580]: 2024-04-23 15:01:03.487922771 +0000 UTC m=+0.359349719 container start 55bfbaa4b7b6843392684e7e7fb53f2a53a26c1d0460c153fd0918599c6def25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-23T15:01:04.036 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:01:03 smithi105 podman[70580]: 2024-04-23 15:01:03.555169656 +0000 UTC m=+0.426596614 container attach 55bfbaa4b7b6843392684e7e7fb53f2a53a26c1d0460c153fd0918599c6def25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T15:01:04.036 INFO:journalctl@ceph.osd.11.smithi105.stdout:Apr 23 15:01:03 smithi105 podman[70580]: 2024-04-23 15:01:03.772499796 +0000 UTC m=+0.643926744 container died 55bfbaa4b7b6843392684e7e7fb53f2a53a26c1d0460c153fd0918599c6def25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-osd-11-deactivate, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-23T15:01:04.356 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@osd.11.service' 2024-04-23T15:01:04.393 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T15:01:04.393 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-23T15:01:04.394 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-23T15:01:04.394 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.a 2024-04-23T15:01:04.873 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 23 15:01:04 smithi028 systemd[1]: Stopping Ceph mds.a for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T15:01:05.374 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 23 15:01:05 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-a[67625]: 2024-04-23T15:01:05.003+0000 7f338ce00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T15:01:05.374 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 23 15:01:05 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-a[67625]: 2024-04-23T15:01:05.003+0000 7f338ce00700 -1 mds.a *** got signal Terminated *** 2024-04-23T15:01:08.465 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 23 15:01:08 smithi028 podman[94604]: 2024-04-23 15:01:08.205833443 +0000 UTC m=+3.336942295 container died cd19ff722a9bcea3344a05d273e392c6b10884e2858cf63ea37a6d32ff818e44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-a, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-23T15:01:08.771 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 23 15:01:08 smithi028 podman[94604]: 2024-04-23 15:01:08.610539702 +0000 UTC m=+3.741648607 container cleanup cd19ff722a9bcea3344a05d273e392c6b10884e2858cf63ea37a6d32ff818e44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-a, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T15:01:08.771 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 23 15:01:08 smithi028 bash[94604]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-a 2024-04-23T15:01:09.123 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 23 15:01:08 smithi028 podman[94618]: 2024-04-23 15:01:08.769440365 +0000 UTC m=+0.545227660 container remove cd19ff722a9bcea3344a05d273e392c6b10884e2858cf63ea37a6d32ff818e44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-a, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-23T15:01:09.303 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.a.service' 2024-04-23T15:01:09.411 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T15:01:09.412 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-23T15:01:09.412 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-23T15:01:09.412 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.d 2024-04-23T15:01:09.623 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 23 15:01:09 smithi028 systemd[1]: Stopping Ceph mds.d for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T15:01:10.374 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 23 15:01:10 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-d[66636]: 2024-04-23T15:01:10.072+0000 7f2730200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T15:01:10.374 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 23 15:01:10 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-d[66636]: 2024-04-23T15:01:10.072+0000 7f2730200700 -1 mds.d *** got signal Terminated *** 2024-04-23T15:01:13.874 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 23 15:01:13 smithi028 podman[94723]: 2024-04-23 15:01:13.600107487 +0000 UTC m=+3.704197194 container stop 8bb57b364e12a222139c87969c10087984175a81d955a599be2665ac9b72296b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-d, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS) 2024-04-23T15:01:13.874 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 23 15:01:13 smithi028 podman[94723]: 2024-04-23 15:01:13.630465795 +0000 UTC m=+3.734555455 container died 8bb57b364e12a222139c87969c10087984175a81d955a599be2665ac9b72296b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-d, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-23T15:01:14.316 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 23 15:01:14 smithi028 podman[94723]: 2024-04-23 15:01:14.00656609 +0000 UTC m=+4.110655782 container cleanup 8bb57b364e12a222139c87969c10087984175a81d955a599be2665ac9b72296b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-d, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-23T15:01:14.316 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 23 15:01:14 smithi028 bash[94723]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-d 2024-04-23T15:01:14.623 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 23 15:01:14 smithi028 podman[94737]: 2024-04-23 15:01:14.315683064 +0000 UTC m=+0.695668325 container remove 8bb57b364e12a222139c87969c10087984175a81d955a599be2665ac9b72296b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-d, ceph=True, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-23T15:01:14.874 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.d.service' 2024-04-23T15:01:14.984 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T15:01:14.984 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-23T15:01:14.984 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-23T15:01:14.985 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.g 2024-04-23T15:01:15.623 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 23 15:01:15 smithi028 systemd[1]: Stopping Ceph mds.g for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T15:01:16.123 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 23 15:01:15 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-g[67278]: 2024-04-23T15:01:15.641+0000 7f36a4e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T15:01:16.124 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 23 15:01:15 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-g[67278]: 2024-04-23T15:01:15.641+0000 7f36a4e00700 -1 mds.g *** got signal Terminated *** 2024-04-23T15:01:18.973 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 23 15:01:18 smithi028 podman[94843]: 2024-04-23 15:01:18.68868065 +0000 UTC m=+3.232382076 container died b54a6fd11f60c2887535c14e998fec3ddb5cd36fcfb2e165e57a8bf7651e0c49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-g, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-23T15:01:19.304 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 23 15:01:19 smithi028 podman[94843]: 2024-04-23 15:01:19.084731542 +0000 UTC m=+3.628432961 container cleanup b54a6fd11f60c2887535c14e998fec3ddb5cd36fcfb2e165e57a8bf7651e0c49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-g, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=) 2024-04-23T15:01:19.304 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 23 15:01:19 smithi028 bash[94843]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-g 2024-04-23T15:01:19.623 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 23 15:01:19 smithi028 podman[94857]: 2024-04-23 15:01:19.402436299 +0000 UTC m=+0.697225456 container remove b54a6fd11f60c2887535c14e998fec3ddb5cd36fcfb2e165e57a8bf7651e0c49 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-g, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416) 2024-04-23T15:01:19.657 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.g.service' 2024-04-23T15:01:19.752 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T15:01:19.752 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-23T15:01:19.753 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-23T15:01:19.753 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.j 2024-04-23T15:01:20.123 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 23 15:01:19 smithi028 systemd[1]: Stopping Ceph mds.j for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T15:01:20.614 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 23 15:01:20 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-j[66293]: 2024-04-23T15:01:20.244+0000 7f4b2ca00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T15:01:20.614 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 23 15:01:20 smithi028 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-j[66293]: 2024-04-23T15:01:20.244+0000 7f4b2ca00700 -1 mds.j *** got signal Terminated *** 2024-04-23T15:01:21.777 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 23 15:01:21 smithi028 podman[94974]: 2024-04-23 15:01:21.471898961 +0000 UTC m=+1.351714587 container died 77263f305569c253505cab66459ecf25e16a4c73ebe4dafda38166895f0db2a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-j, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True) 2024-04-23T15:01:22.123 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 23 15:01:21 smithi028 podman[94974]: 2024-04-23 15:01:21.889056503 +0000 UTC m=+1.768872110 container cleanup 77263f305569c253505cab66459ecf25e16a4c73ebe4dafda38166895f0db2a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-j, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-23T15:01:22.123 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 23 15:01:21 smithi028 bash[94974]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-j 2024-04-23T15:01:22.623 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 23 15:01:22 smithi028 podman[94989]: 2024-04-23 15:01:22.156745093 +0000 UTC m=+0.661467587 container remove 77263f305569c253505cab66459ecf25e16a4c73ebe4dafda38166895f0db2a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-j, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-23T15:01:22.729 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.j.service' 2024-04-23T15:01:22.836 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T15:01:22.836 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-23T15:01:22.837 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-23T15:01:22.837 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.b 2024-04-23T15:01:23.122 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 23 15:01:22 smithi038 systemd[1]: Stopping Ceph mds.b for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T15:01:23.553 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 23 15:01:23 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-b[63031]: 2024-04-23T15:01:23.167+0000 7fb7a0893700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T15:01:23.553 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 23 15:01:23 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-b[63031]: 2024-04-23T15:01:23.167+0000 7fb7a0893700 -1 mds.b *** got signal Terminated *** 2024-04-23T15:01:26.868 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 23 15:01:26 smithi038 podman[70055]: 2024-04-23 15:01:26.598368168 +0000 UTC m=+3.556124302 container died bb8d937f28efb51feeb5f2a30227da9e0c491a9d9d221e6573f936989de13ced (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd) 2024-04-23T15:01:27.122 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 23 15:01:26 smithi038 podman[70055]: 2024-04-23 15:01:26.958730054 +0000 UTC m=+3.916486195 container cleanup bb8d937f28efb51feeb5f2a30227da9e0c491a9d9d221e6573f936989de13ced (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-23T15:01:27.122 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 23 15:01:26 smithi038 bash[70055]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-b 2024-04-23T15:01:27.342 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.b.service' 2024-04-23T15:01:27.378 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T15:01:27.379 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-23T15:01:27.379 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-23T15:01:27.379 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.e 2024-04-23T15:01:27.622 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 23 15:01:27 smithi038 systemd[1]: Stopping Ceph mds.e for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T15:01:28.122 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 23 15:01:27 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-e[62398]: 2024-04-23T15:01:27.704+0000 7ffbe3fae700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T15:01:28.122 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 23 15:01:27 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-e[62398]: 2024-04-23T15:01:27.704+0000 7ffbe3fae700 -1 mds.e *** got signal Terminated *** 2024-04-23T15:01:32.017 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 23 15:01:31 smithi038 podman[70188]: 2024-04-23 15:01:31.718473363 +0000 UTC m=+4.110474112 container died e2530c3f5fe27a6fa2349fdb565134628e1e28c85c6c6e7c3ef2cb767fad4032 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-e, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_BRANCH=HEAD) 2024-04-23T15:01:32.337 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 23 15:01:32 smithi038 podman[70188]: 2024-04-23 15:01:32.09547674 +0000 UTC m=+4.487477541 container cleanup e2530c3f5fe27a6fa2349fdb565134628e1e28c85c6c6e7c3ef2cb767fad4032 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-e, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416) 2024-04-23T15:01:32.338 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 23 15:01:32 smithi038 bash[70188]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-e 2024-04-23T15:01:32.461 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.e.service' 2024-04-23T15:01:32.494 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T15:01:32.494 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-23T15:01:32.495 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-23T15:01:32.495 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.h 2024-04-23T15:01:32.622 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 23 15:01:32 smithi038 systemd[1]: Stopping Ceph mds.h for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T15:01:33.122 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 23 15:01:32 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-h[62708]: 2024-04-23T15:01:32.823+0000 7f2b05b06700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T15:01:33.122 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 23 15:01:32 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-h[62708]: 2024-04-23T15:01:32.824+0000 7f2b05b06700 -1 mds.h *** got signal Terminated *** 2024-04-23T15:01:37.108 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 23 15:01:36 smithi038 podman[70307]: 2024-04-23 15:01:36.838047893 +0000 UTC m=+4.122419219 container died eea5e724a98ba815a6dbdb256e4546b4a03f65e8382b837e494d15a13aebec35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-23T15:01:37.372 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 23 15:01:37 smithi038 podman[70307]: 2024-04-23 15:01:37.265882705 +0000 UTC m=+4.550254031 container cleanup eea5e724a98ba815a6dbdb256e4546b4a03f65e8382b837e494d15a13aebec35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-h, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416) 2024-04-23T15:01:37.372 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 23 15:01:37 smithi038 bash[70307]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-h 2024-04-23T15:01:37.703 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.h.service' 2024-04-23T15:01:37.726 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 23 15:01:37 smithi038 podman[70321]: 2024-04-23 15:01:37.475598571 +0000 UTC m=+0.638670353 container remove eea5e724a98ba815a6dbdb256e4546b4a03f65e8382b837e494d15a13aebec35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-h, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-23T15:01:37.726 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 23 15:01:37 smithi038 systemd[1]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.h.service: Deactivated successfully. 2024-04-23T15:01:37.726 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 23 15:01:37 smithi038 systemd[1]: Stopped Ceph mds.h for 1ee6e996-017e-11ef-bc93-c7b262605968. 2024-04-23T15:01:37.726 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 23 15:01:37 smithi038 systemd[1]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.h.service: Consumed 1.118s CPU time. 2024-04-23T15:01:37.735 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T15:01:37.735 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-23T15:01:37.736 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-23T15:01:37.736 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.k 2024-04-23T15:01:38.069 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 23 15:01:37 smithi038 systemd[1]: Stopping Ceph mds.k for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T15:01:38.372 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 23 15:01:38 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-k[62073]: 2024-04-23T15:01:38.068+0000 7efdd4424700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T15:01:38.372 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 23 15:01:38 smithi038 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-k[62073]: 2024-04-23T15:01:38.068+0000 7efdd4424700 -1 mds.k *** got signal Terminated *** 2024-04-23T15:01:42.054 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 23 15:01:41 smithi038 podman[70428]: 2024-04-23 15:01:41.723244992 +0000 UTC m=+3.764420274 container died 2f2cbd0307e49a235c87f8e811dc6da2c3fb513619cf503739da08d58b198ed0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-k, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T15:01:42.372 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 23 15:01:42 smithi038 podman[70428]: 2024-04-23 15:01:42.145407195 +0000 UTC m=+4.186582465 container cleanup 2f2cbd0307e49a235c87f8e811dc6da2c3fb513619cf503739da08d58b198ed0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd) 2024-04-23T15:01:42.372 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 23 15:01:42 smithi038 bash[70428]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-k 2024-04-23T15:01:42.640 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.k.service' 2024-04-23T15:01:42.671 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T15:01:42.672 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-23T15:01:42.672 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-23T15:01:42.672 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.c 2024-04-23T15:01:43.035 INFO:journalctl@ceph.mds.c.smithi105.stdout:Apr 23 15:01:42 smithi105 systemd[1]: Stopping Ceph mds.c for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T15:01:43.296 INFO:journalctl@ceph.mds.c.smithi105.stdout:Apr 23 15:01:43 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-c[63357]: 2024-04-23T15:01:43.045+0000 7f091da4d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T15:01:43.296 INFO:journalctl@ceph.mds.c.smithi105.stdout:Apr 23 15:01:43 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-c[63357]: 2024-04-23T15:01:43.045+0000 7f091da4d700 -1 mds.c *** got signal Terminated *** 2024-04-23T15:01:46.680 INFO:journalctl@ceph.mds.c.smithi105.stdout:Apr 23 15:01:46 smithi105 podman[70685]: 2024-04-23 15:01:46.322646414 +0000 UTC m=+3.459553049 container died bca81fe91c8f14bd618e87d442ecb4de5f1b46088fbea53f242af2c1f621dcf8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-23T15:01:46.939 INFO:journalctl@ceph.mds.c.smithi105.stdout:Apr 23 15:01:46 smithi105 podman[70685]: 2024-04-23 15:01:46.754333579 +0000 UTC m=+3.891240206 container cleanup bca81fe91c8f14bd618e87d442ecb4de5f1b46088fbea53f242af2c1f621dcf8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-c, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS) 2024-04-23T15:01:46.939 INFO:journalctl@ceph.mds.c.smithi105.stdout:Apr 23 15:01:46 smithi105 bash[70685]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-c 2024-04-23T15:01:47.173 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.c.service' 2024-04-23T15:01:47.194 INFO:journalctl@ceph.mds.c.smithi105.stdout:Apr 23 15:01:46 smithi105 podman[70711]: 2024-04-23 15:01:46.938806953 +0000 UTC m=+0.618698722 container remove bca81fe91c8f14bd618e87d442ecb4de5f1b46088fbea53f242af2c1f621dcf8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-23T15:01:47.194 INFO:journalctl@ceph.mds.c.smithi105.stdout:Apr 23 15:01:47 smithi105 systemd[1]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.c.service: Deactivated successfully. 2024-04-23T15:01:47.194 INFO:journalctl@ceph.mds.c.smithi105.stdout:Apr 23 15:01:47 smithi105 systemd[1]: Stopped Ceph mds.c for 1ee6e996-017e-11ef-bc93-c7b262605968. 2024-04-23T15:01:47.195 INFO:journalctl@ceph.mds.c.smithi105.stdout:Apr 23 15:01:47 smithi105 systemd[1]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.c.service: Consumed 1.121s CPU time. 2024-04-23T15:01:47.208 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T15:01:47.208 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-23T15:01:47.209 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-23T15:01:47.209 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.f 2024-04-23T15:01:47.524 INFO:journalctl@ceph.mds.f.smithi105.stdout:Apr 23 15:01:47 smithi105 systemd[1]: Stopping Ceph mds.f for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T15:01:47.786 INFO:journalctl@ceph.mds.f.smithi105.stdout:Apr 23 15:01:47 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-f[63049]: 2024-04-23T15:01:47.524+0000 7fc450a4e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T15:01:47.786 INFO:journalctl@ceph.mds.f.smithi105.stdout:Apr 23 15:01:47 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-f[63049]: 2024-04-23T15:01:47.524+0000 7fc450a4e700 -1 mds.f *** got signal Terminated *** 2024-04-23T15:01:51.733 INFO:journalctl@ceph.mds.f.smithi105.stdout:Apr 23 15:01:51 smithi105 podman[70818]: 2024-04-23 15:01:51.389489819 +0000 UTC m=+3.971879940 container died 75e2c6354570a72ab416acf11d511b0c2ee8b6c47d0d094a4fd75b82a137f82f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-f, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, maintainer=Guillaume Abrioux ) 2024-04-23T15:01:52.036 INFO:journalctl@ceph.mds.f.smithi105.stdout:Apr 23 15:01:51 smithi105 podman[70818]: 2024-04-23 15:01:51.841597689 +0000 UTC m=+4.423987799 container cleanup 75e2c6354570a72ab416acf11d511b0c2ee8b6c47d0d094a4fd75b82a137f82f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416) 2024-04-23T15:01:52.036 INFO:journalctl@ceph.mds.f.smithi105.stdout:Apr 23 15:01:51 smithi105 bash[70818]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-f 2024-04-23T15:01:52.265 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.f.service' 2024-04-23T15:01:52.297 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T15:01:52.297 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-23T15:01:52.297 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-23T15:01:52.297 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.i 2024-04-23T15:01:52.620 INFO:journalctl@ceph.mds.i.smithi105.stdout:Apr 23 15:01:52 smithi105 systemd[1]: Stopping Ceph mds.i for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T15:01:53.035 INFO:journalctl@ceph.mds.i.smithi105.stdout:Apr 23 15:01:52 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-i[63670]: 2024-04-23T15:01:52.619+0000 7f416a5e6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T15:01:53.036 INFO:journalctl@ceph.mds.i.smithi105.stdout:Apr 23 15:01:52 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-i[63670]: 2024-04-23T15:01:52.619+0000 7f416a5e6700 -1 mds.i *** got signal Terminated *** 2024-04-23T15:01:56.244 INFO:journalctl@ceph.mds.i.smithi105.stdout:Apr 23 15:01:55 smithi105 podman[70936]: 2024-04-23 15:01:55.929504812 +0000 UTC m=+3.417133637 container died 56f412a338a7b1168f6e83a087604a0c7db5d59f6eec17879a9be93cf0801bcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-i, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-23T15:01:56.245 INFO:journalctl@ceph.mds.i.smithi105.stdout:Apr 23 15:01:56 smithi105 podman[70936]: 2024-04-23 15:01:56.24484633 +0000 UTC m=+3.732475150 container cleanup 56f412a338a7b1168f6e83a087604a0c7db5d59f6eec17879a9be93cf0801bcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-i, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5) 2024-04-23T15:01:56.536 INFO:journalctl@ceph.mds.i.smithi105.stdout:Apr 23 15:01:56 smithi105 bash[70936]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-i 2024-04-23T15:01:56.536 INFO:journalctl@ceph.mds.i.smithi105.stdout:Apr 23 15:01:56 smithi105 podman[70950]: 2024-04-23 15:01:56.446145662 +0000 UTC m=+0.517776555 container remove 56f412a338a7b1168f6e83a087604a0c7db5d59f6eec17879a9be93cf0801bcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-i, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-23T15:01:56.583 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.i.service' 2024-04-23T15:01:56.614 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T15:01:56.614 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-23T15:01:56.615 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-23T15:01:56.615 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.l 2024-04-23T15:01:56.931 INFO:journalctl@ceph.mds.l.smithi105.stdout:Apr 23 15:01:56 smithi105 systemd[1]: Stopping Ceph mds.l for 1ee6e996-017e-11ef-bc93-c7b262605968... 2024-04-23T15:01:57.285 INFO:journalctl@ceph.mds.l.smithi105.stdout:Apr 23 15:01:56 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-l[62742]: 2024-04-23T15:01:56.931+0000 7ff6b89ce700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T15:01:57.286 INFO:journalctl@ceph.mds.l.smithi105.stdout:Apr 23 15:01:56 smithi105 ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-l[62742]: 2024-04-23T15:01:56.931+0000 7ff6b89ce700 -1 mds.l *** got signal Terminated *** 2024-04-23T15:01:59.174 INFO:journalctl@ceph.mds.l.smithi105.stdout:Apr 23 15:01:58 smithi105 podman[71056]: 2024-04-23 15:01:58.909155811 +0000 UTC m=+2.076116299 container died 9622fdb37276d39eec47032f74056b2dec4a0f554781c92a6b8a7678e2d66a82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-l, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-23T15:01:59.536 INFO:journalctl@ceph.mds.l.smithi105.stdout:Apr 23 15:01:59 smithi105 podman[71056]: 2024-04-23 15:01:59.257110978 +0000 UTC m=+2.424071466 container cleanup 9622fdb37276d39eec47032f74056b2dec4a0f554781c92a6b8a7678e2d66a82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:af2350ee15fec7863cae8207e1de3b5c3baf4b211768c50629a9f2fdb9a2f3cd, name=ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240416.193735-3-d44fddd, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-23T15:01:59.536 INFO:journalctl@ceph.mds.l.smithi105.stdout:Apr 23 15:01:59 smithi105 bash[71056]: ceph-1ee6e996-017e-11ef-bc93-c7b262605968-mds-l 2024-04-23T15:01:59.725 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1ee6e996-017e-11ef-bc93-c7b262605968@mds.l.service' 2024-04-23T15:01:59.755 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T15:01:59.755 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-23T15:01:59.755 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-23T15:02:00.090 INFO:teuthology.orchestra.run.smithi028.stdout:Deleting cluster with fsid: 1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T15:02:05.500 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-23T15:02:05.700 INFO:teuthology.orchestra.run.smithi038.stdout:Deleting cluster with fsid: 1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T15:02:08.405 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-23T15:02:08.593 INFO:teuthology.orchestra.run.smithi105.stdout:Deleting cluster with fsid: 1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T15:02:10.862 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T15:02:10.949 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T15:02:10.981 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T15:02:11.006 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-23T15:02:11.008 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi/7669575/remote/smithi028/crash 2024-04-23T15:02:11.009 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/crash -- . 2024-04-23T15:02:11.110 INFO:teuthology.orchestra.run.smithi028.stderr:tar: /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-23T15:02:11.110 INFO:teuthology.orchestra.run.smithi028.stderr:tar: Error is not recoverable: exiting now 2024-04-23T15:02:11.115 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi/7669575/remote/smithi038/crash 2024-04-23T15:02:11.116 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/crash -- . 2024-04-23T15:02:11.155 INFO:teuthology.orchestra.run.smithi038.stderr:tar: /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-23T15:02:11.155 INFO:teuthology.orchestra.run.smithi038.stderr:tar: Error is not recoverable: exiting now 2024-04-23T15:02:11.157 DEBUG:teuthology.misc:Transferring archived files from smithi105:/var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi/7669575/remote/smithi105/crash 2024-04-23T15:02:11.158 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/crash -- . 2024-04-23T15:02:11.204 INFO:teuthology.orchestra.run.smithi105.stderr:tar: /var/lib/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-23T15:02:11.204 INFO:teuthology.orchestra.run.smithi105.stderr:tar: Error is not recoverable: exiting now 2024-04-23T15:02:11.205 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-23T15:02:11.205 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-23T15:02:11.318 INFO:tasks.cephadm:Compressing logs... 2024-04-23T15:02:11.319 DEBUG:teuthology.orchestra.run.smithi028:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-23T15:02:11.323 DEBUG:teuthology.orchestra.run.smithi038:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-23T15:02:11.325 DEBUG:teuthology.orchestra.run.smithi105:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-23T15:02:11.349 INFO:teuthology.orchestra.run.smithi105.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph.log 2024-04-23T15:02:11.349 INFO:teuthology.orchestra.run.smithi105.stderr:: No such file or directory 2024-04-23T15:02:11.350 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-23T15:02:11.350 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-23T15:02:11.350 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-23T15:02:11.350 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-23T15:02:11.350 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-23T15:02:11.350 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph.audit.log: /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-23T15:02:11.353 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-23T15:02:11.353 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mon.c.log: /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-23T15:02:11.353 INFO:teuthology.orchestra.run.smithi105.stderr: 87.6% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-23T15:02:11.353 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-23T15:02:11.353 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.i.log: 90.4% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-23T15:02:11.354 INFO:teuthology.orchestra.run.smithi105.stderr: 94.5% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-23T15:02:11.357 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-23T15:02:11.357 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-23T15:02:11.357 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.11.log: 83.5% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-23T15:02:11.357 INFO:teuthology.orchestra.run.smithi105.stderr: 94.5% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-23T15:02:11.357 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-23T15:02:11.357 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-23T15:02:11.360 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-volume.log: 94.5% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-23T15:02:11.360 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-23T15:02:11.362 INFO:teuthology.orchestra.run.smithi038.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-23T15:02:11.362 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph.log 2024-04-23T15:02:11.362 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-23T15:02:11.362 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-23T15:02:11.363 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-23T15:02:11.363 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-23T15:02:11.363 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.7.log: /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-23T15:02:11.363 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-23T15:02:11.363 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-23T15:02:11.363 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-23T15:02:11.363 INFO:teuthology.orchestra.run.smithi038.stderr:gzip/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.h.log: -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-23T15:02:11.363 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-23T15:02:11.363 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-23T15:02:11.363 INFO:teuthology.orchestra.run.smithi038.stderr:gzip/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.e.log: -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-23T15:02:11.364 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-23T15:02:11.365 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-23T15:02:11.365 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.l.log: 92.3% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-23T15:02:11.366 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/cephadm.log: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-23T15:02:11.366 INFO:teuthology.orchestra.run.smithi105.stderr: 94.5% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-23T15:02:11.370 INFO:teuthology.orchestra.run.smithi105.stderr: 91.0% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-23T15:02:11.378 INFO:teuthology.orchestra.run.smithi028.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-23T15:02:11.381 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph.log 2024-04-23T15:02:11.382 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-23T15:02:11.383 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-23T15:02:11.384 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-23T15:02:11.385 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-23T15:02:11.386 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.j.log: 87.6% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-23T15:02:11.387 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-23T15:02:11.387 INFO:teuthology.orchestra.run.smithi028.stderr: 90.2% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-23T15:02:11.388 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-23T15:02:11.388 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-23T15:02:11.390 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-23T15:02:11.391 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mon.a.log: 94.3% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-23T15:02:11.391 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-23T15:02:11.392 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-23T15:02:11.394 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph.cephadm.log: 83.9% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-23T15:02:11.395 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-23T15:02:11.397 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.a.log: /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-23T15:02:11.402 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.d.log: 94.3% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-23T15:02:11.402 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-23T15:02:11.404 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-23T15:02:11.416 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.9.log: /var/log/ceph/cephadm.log: 94.4% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-23T15:02:11.421 INFO:teuthology.orchestra.run.smithi028.stderr: 93.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-23T15:02:11.427 INFO:teuthology.orchestra.run.smithi028.stderr: 90.9% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-23T15:02:11.447 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.k.log: /var/log/ceph/cephadm.log: 83.5% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-23T15:02:11.490 INFO:teuthology.orchestra.run.smithi038.stderr: 92.2% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-23T15:02:11.517 INFO:teuthology.orchestra.run.smithi038.stderr: 94.5% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-23T15:02:11.530 INFO:teuthology.orchestra.run.smithi038.stderr: 87.6% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-23T15:02:11.531 INFO:teuthology.orchestra.run.smithi038.stderr: 90.4% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-23T15:02:11.536 INFO:teuthology.orchestra.run.smithi038.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-23T15:02:11.549 INFO:teuthology.orchestra.run.smithi038.stderr: 94.5% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-23T15:02:11.559 INFO:teuthology.orchestra.run.smithi038.stderr: 91.1% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-23T15:02:11.621 INFO:teuthology.orchestra.run.smithi038.stderr: 94.4% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-23T15:02:11.865 INFO:teuthology.orchestra.run.smithi028.stderr: 90.1% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-23T15:02:11.960 INFO:teuthology.orchestra.run.smithi105.stderr: 91.8% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-23T15:02:12.920 INFO:teuthology.orchestra.run.smithi028.stderr: 90.4% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-23T15:02:13.575 INFO:teuthology.orchestra.run.smithi038.stderr: 92.4% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-23T15:02:15.597 INFO:teuthology.orchestra.run.smithi028.stderr: 94.1% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-23T15:02:16.166 INFO:teuthology.orchestra.run.smithi105.stderr: 94.2% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-23T15:02:16.585 INFO:teuthology.orchestra.run.smithi105.stderr: 94.3% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-23T15:02:16.867 INFO:teuthology.orchestra.run.smithi028.stderr: 93.7% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-23T15:02:17.011 INFO:teuthology.orchestra.run.smithi105.stderr: 94.4% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-23T15:02:17.650 INFO:teuthology.orchestra.run.smithi028.stderr: 93.9% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-23T15:02:17.926 INFO:teuthology.orchestra.run.smithi105.stderr: 94.4% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-23T15:02:17.928 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-04-23T15:02:17.929 INFO:teuthology.orchestra.run.smithi105.stderr:real 0m6.589s 2024-04-23T15:02:17.929 INFO:teuthology.orchestra.run.smithi105.stderr:user 0m22.112s 2024-04-23T15:02:17.929 INFO:teuthology.orchestra.run.smithi105.stderr:sys 0m0.819s 2024-04-23T15:02:18.505 INFO:teuthology.orchestra.run.smithi028.stderr: 94.1% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-23T15:02:21.105 INFO:teuthology.orchestra.run.smithi038.stderr: 94.2% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-23T15:02:21.904 INFO:teuthology.orchestra.run.smithi038.stderr: 94.4% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-23T15:02:23.442 INFO:teuthology.orchestra.run.smithi028.stderr: 95.3% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-23T15:02:23.446 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-04-23T15:02:23.446 INFO:teuthology.orchestra.run.smithi028.stderr:real 0m12.091s 2024-04-23T15:02:23.446 INFO:teuthology.orchestra.run.smithi028.stderr:user 0m34.481s 2024-04-23T15:02:23.446 INFO:teuthology.orchestra.run.smithi028.stderr:sys 0m2.755s 2024-04-23T15:02:23.641 INFO:teuthology.orchestra.run.smithi038.stderr: 94.0% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-23T15:02:24.942 INFO:teuthology.orchestra.run.smithi038.stderr: 94.4% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-23T15:05:51.181 INFO:teuthology.orchestra.run.smithi038.stderr: 93.5% -- replaced with /var/log/ceph/1ee6e996-017e-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-23T15:05:51.183 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-04-23T15:05:51.183 INFO:teuthology.orchestra.run.smithi038.stderr:real 3m39.841s 2024-04-23T15:05:51.183 INFO:teuthology.orchestra.run.smithi038.stderr:user 3m48.659s 2024-04-23T15:05:51.183 INFO:teuthology.orchestra.run.smithi038.stderr:sys 0m6.793s 2024-04-23T15:05:51.184 INFO:tasks.cephadm:Archiving logs... 2024-04-23T15:05:51.185 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi/7669575/remote/smithi028/log 2024-04-23T15:05:51.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-23T15:05:55.377 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi/7669575/remote/smithi038/log 2024-04-23T15:05:55.378 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-23T15:06:27.990 DEBUG:teuthology.misc:Transferring archived files from smithi105:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi/7669575/remote/smithi105/log 2024-04-23T15:06:27.991 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-23T15:06:31.593 INFO:tasks.cephadm:Removing cluster... 2024-04-23T15:06:31.594 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 --force 2024-04-23T15:06:31.923 INFO:teuthology.orchestra.run.smithi028.stdout:Deleting cluster with fsid: 1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T15:06:32.658 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 --force 2024-04-23T15:06:32.859 INFO:teuthology.orchestra.run.smithi038.stdout:Deleting cluster with fsid: 1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T15:06:33.358 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1ee6e996-017e-11ef-bc93-c7b262605968 --force 2024-04-23T15:06:33.549 INFO:teuthology.orchestra.run.smithi105.stdout:Deleting cluster with fsid: 1ee6e996-017e-11ef-bc93-c7b262605968 2024-04-23T15:06:33.828 INFO:tasks.cephadm:Removing cephadm ... 2024-04-23T15:06:33.828 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-23T15:06:33.891 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-23T15:06:33.908 DEBUG:teuthology.orchestra.run.smithi105:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-23T15:06:33.924 INFO:tasks.cephadm:Teardown complete 2024-04-23T15:06:33.924 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-23T15:06:33.936 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-23T15:06:33.936 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-23T15:06:33.943 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-23T15:06:33.950 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-23T15:06:34.116 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-23T15:06:34.132 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-23T15:06:34.132 DEBUG:teuthology.orchestra.run.smithi028:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T15:06:34.141 DEBUG:teuthology.orchestra.run.smithi038:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T15:06:34.144 DEBUG:teuthology.orchestra.run.smithi105:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T15:06:34.158 INFO:teuthology.orchestra.run.smithi038.stderr:bash: line 1: ntpq: command not found 2024-04-23T15:06:34.159 INFO:teuthology.orchestra.run.smithi105.stderr:bash: line 1: ntpq: command not found 2024-04-23T15:06:34.163 INFO:teuthology.orchestra.run.smithi105.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T15:06:34.164 INFO:teuthology.orchestra.run.smithi105.stdout:=============================================================================== 2024-04-23T15:06:34.164 INFO:teuthology.orchestra.run.smithi105.stdout:^- hv01.front.sepia.ceph.com 2 7 377 114 +792us[ +817us] +/- 590ms 2024-04-23T15:06:34.164 INFO:teuthology.orchestra.run.smithi105.stdout:^* hv02.front.sepia.ceph.com 3 6 377 48 -590us[ -580us] +/- 56ms 2024-04-23T15:06:34.165 INFO:teuthology.orchestra.run.smithi105.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 47 +700us[ +700us] +/- 68ms 2024-04-23T15:06:34.165 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T15:06:34.180 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T15:06:34.181 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2024-04-23T15:06:34.181 INFO:teuthology.orchestra.run.smithi038.stdout:^- hv01.front.sepia.ceph.com 2 6 377 52 +786us[ +792us] +/- 591ms 2024-04-23T15:06:34.182 INFO:teuthology.orchestra.run.smithi038.stdout:^* hv02.front.sepia.ceph.com 3 6 377 47 -579us[ -573us] +/- 56ms 2024-04-23T15:06:34.182 INFO:teuthology.orchestra.run.smithi038.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 46 +709us[ +709us] +/- 68ms 2024-04-23T15:06:34.182 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T15:06:34.189 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-04-23T15:06:34.202 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T15:06:34.203 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-04-23T15:06:34.203 INFO:teuthology.orchestra.run.smithi028.stdout:^- hv01.front.sepia.ceph.com 2 7 377 113 +833us[ +834us] +/- 590ms 2024-04-23T15:06:34.203 INFO:teuthology.orchestra.run.smithi028.stdout:^* hv02.front.sepia.ceph.com 3 6 377 51 -572us[ -570us] +/- 56ms 2024-04-23T15:06:34.203 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 49 +718us[ +718us] +/- 68ms 2024-04-23T15:06:34.203 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T15:06:34.203 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-23T15:06:34.258 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-23T15:06:34.259 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-23T15:06:34.272 DEBUG:teuthology.orchestra.run.smithi028:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-23T15:06:34.505 DEBUG:teuthology.orchestra.run.smithi038:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-23T15:06:34.610 DEBUG:teuthology.orchestra.run.smithi105:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-23T15:06:34.674 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T15:06:34.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T15:06:34.775 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T15:06:34.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T15:06:34.807 DEBUG:teuthology.orchestra.run.smithi105:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T15:06:34.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T15:06:34.834 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-23T15:06:34.846 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-23T15:06:34.856 INFO:teuthology.task.internal:Duration was 2579.869989 seconds 2024-04-23T15:06:34.857 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-23T15:06:34.867 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-23T15:06:34.868 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-23T15:06:34.876 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-23T15:06:34.878 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-23T15:06:34.914 INFO:teuthology.orchestra.run.smithi105.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T15:06:34.943 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T15:06:35.008 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T15:06:35.323 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-23T15:06:35.323 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi028.front.sepia.ceph.com 2024-04-23T15:06:35.323 DEBUG:teuthology.orchestra.run.smithi028:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-23T15:06:35.405 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi038.front.sepia.ceph.com 2024-04-23T15:06:35.405 DEBUG:teuthology.orchestra.run.smithi038:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-23T15:06:35.439 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi105.front.sepia.ceph.com 2024-04-23T15:06:35.439 DEBUG:teuthology.orchestra.run.smithi105:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-23T15:06:35.464 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-23T15:06:35.465 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-23T15:06:35.472 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-23T15:06:35.482 DEBUG:teuthology.orchestra.run.smithi105:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-23T15:06:35.563 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-23T15:06:35.564 DEBUG:teuthology.orchestra.run.smithi028:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-23T15:06:35.572 DEBUG:teuthology.orchestra.run.smithi038:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-23T15:06:35.574 DEBUG:teuthology.orchestra.run.smithi105:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-23T15:06:36.091 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-23T15:06:36.102 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-23T15:06:36.102 DEBUG:teuthology.orchestra.run.smithi028:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-23T15:06:36.194 DEBUG:teuthology.orchestra.run.smithi038:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-23T15:06:36.228 DEBUG:teuthology.orchestra.run.smithi105:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-23T15:06:36.253 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-23T15:06:36.265 DEBUG:teuthology.orchestra.run.smithi028:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-23T15:06:36.272 DEBUG:teuthology.orchestra.run.smithi038:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-23T15:06:36.275 DEBUG:teuthology.orchestra.run.smithi105:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-23T15:06:36.300 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = core 2024-04-23T15:06:36.315 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern = core 2024-04-23T15:06:36.371 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = core 2024-04-23T15:06:36.418 DEBUG:teuthology.orchestra.run.smithi028:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-23T15:06:36.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T15:06:36.471 DEBUG:teuthology.orchestra.run.smithi038:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-23T15:06:36.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T15:06:36.486 DEBUG:teuthology.orchestra.run.smithi105:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-23T15:06:36.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T15:06:36.501 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-23T15:06:36.512 INFO:teuthology.task.internal:Transferring archived files... 2024-04-23T15:06:36.513 DEBUG:teuthology.misc:Transferring archived files from smithi028:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi/7669575/remote/smithi028 2024-04-23T15:06:36.513 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-23T15:06:36.626 DEBUG:teuthology.misc:Transferring archived files from smithi038:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi/7669575/remote/smithi038 2024-04-23T15:06:36.626 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-23T15:06:36.666 DEBUG:teuthology.misc:Transferring archived files from smithi105:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi/7669575/remote/smithi105 2024-04-23T15:06:36.667 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-23T15:06:36.702 INFO:teuthology.task.internal:Removing archive directory... 2024-04-23T15:06:36.703 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-23T15:06:36.710 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-23T15:06:36.713 DEBUG:teuthology.orchestra.run.smithi105:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-23T15:06:36.762 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-23T15:06:36.776 INFO:teuthology.task.internal:Not uploading archives. 2024-04-23T15:06:36.776 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-23T15:06:36.787 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-23T15:06:36.787 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-23T15:06:36.793 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-23T15:06:36.795 DEBUG:teuthology.orchestra.run.smithi105:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-23T15:06:36.809 INFO:teuthology.orchestra.run.smithi038.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 23 15:06 /home/ubuntu/cephtest 2024-04-23T15:06:36.809 INFO:teuthology.orchestra.run.smithi105.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 23 15:06 /home/ubuntu/cephtest 2024-04-23T15:06:36.845 INFO:teuthology.orchestra.run.smithi028.stdout: 393949 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 23 15:06 /home/ubuntu/cephtest 2024-04-23T15:06:36.848 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-23T15:06:36.861 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-23T15:06:36.885 INFO:teuthology.nuke:Checking targets against current locks 2024-04-23T15:06:36.912 DEBUG:teuthology.nuke:shortname: smithi028 2024-04-23T15:06:36.913 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-23T15:06:36.937 DEBUG:teuthology.nuke:shortname: smithi038 2024-04-23T15:06:36.938 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-23T15:06:36.943 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi/7669575', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 14:05:38.351647', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBUehiPSkozqQnJ9UaPSqzBxGc+A6onb13HqwF8SaFKzIJuHG8vAfiA2efvG52A4du+l9t0tmP/+d8LC/GLF1ls='} 2024-04-23T15:06:36.963 DEBUG:teuthology.nuke:shortname: smithi105 2024-04-23T15:06:36.963 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-23T15:06:36.968 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi/7669575', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 14:05:38.350474', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDEaaRjD218Zv1QKJfZeOdXOvqJctxdXM3ACEaWxTdK0xC2fsrPp874KZQJzxpfucl0yuqsl++K1Sho4lgDl4Mk='} 2024-04-23T15:06:36.998 INFO:teuthology.orchestra.console.smithi028:Power off 2024-04-23T15:06:36.999 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-23T15:06:37.011 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi105.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-23_14:04:26-fs-wip-rishabh-testing-20240416.193735-3-testing-default-smithi/7669575', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 14:05:38.348737', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKmkDL7EOfk19pPMyDZ6wHoDyMlR57e1ijQcjvaYXN9Cue9hzd2b5xKz07ICDhkEW9gQSsNFfw6uBT8hS6bjaVA='} 2024-04-23T15:06:37.025 DEBUG:teuthology.orchestra.console.smithi028:power off output: Chassis Power Control: Down/Off 2024-04-23T15:06:37.025 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T15:06:37.050 INFO:teuthology.orchestra.console.smithi038:Power off 2024-04-23T15:06:37.050 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-23T15:06:37.061 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is on 2024-04-23T15:06:37.075 DEBUG:teuthology.orchestra.console.smithi038:power off output: Chassis Power Control: Down/Off 2024-04-23T15:06:37.075 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T15:06:37.092 INFO:teuthology.orchestra.console.smithi105:Power off 2024-04-23T15:06:37.093 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-23T15:06:37.103 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is on 2024-04-23T15:06:37.116 DEBUG:teuthology.orchestra.console.smithi105:power off output: Chassis Power Control: Down/Off 2024-04-23T15:06:37.117 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T15:06:37.141 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is on 2024-04-23T15:06:41.063 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T15:06:41.105 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T15:06:41.142 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T15:06:41.175 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is on 2024-04-23T15:06:41.219 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is on 2024-04-23T15:06:41.258 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is on 2024-04-23T15:06:45.180 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T15:06:45.219 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T15:06:45.259 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T15:06:45.291 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is on 2024-04-23T15:06:45.330 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is off 2024-04-23T15:06:45.369 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is on 2024-04-23T15:06:45.432 INFO:teuthology.orchestra.console.smithi038:Power off completed 2024-04-23T15:06:49.295 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T15:06:49.370 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T15:06:49.407 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is off 2024-04-23T15:06:49.481 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is off 2024-04-23T15:06:49.507 INFO:teuthology.orchestra.console.smithi028:Power off completed 2024-04-23T15:06:49.582 INFO:teuthology.orchestra.console.smithi105:Power off completed 2024-04-23T15:06:49.689 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 806ec36b21129f113ce356c007dd59694d340b37 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} duration: 2579.8699889183044 failure_reason: 'Command failed (workunit test suites/dbench.sh) on smithi028 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d44fddd910b5e94da0f2357af4be37d7bd495ae0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh''' flavor: default owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=5efad306a3fc46e281c6b2e55223662a status: fail success: false 2024-04-23T15:06:49.690 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-23T15:06:49.761 INFO:teuthology.run:FAIL